Interrupting Python Threads
In my most recent project, rgc, I have been using the python threading library for concurrent operations. Python Threads are often overlooked because the python GIL forces them to share a single CPU core, but they are great for scaling I/O or subprocess calls without worrying about communication.
Things were running fine, but whenever I wanted to kill and restart my script, a simple
Ctrl+C would not force it to exit.
I had to background (
Ctrl+Z) the process, and send a kill signal to the specific process ID (
kill -9 [PID]) after looking it up.
Below is a simple script to demonstrate this issue. The main function spawns a (sleepy) thread that loops through work (sleep).
#!/usr/bin/env python from threading import Thread from time import sleep import sys def main(): # Spawn a new thread that runs sleepy t = Thread(target=sleepy, args=(0,)) # Start the thread t.start() # Join the child thread back to parent t.join() def sleepy(t_id, n_loops=5): ''' Thread function ''' for i in range(n_loops): print("Thread-%i sleeping %i/%i"%(t_id, i+1, n_loops)) sleep(2) if __name__ == "__main__": main()
If you run the script and try to interrupt it with
Ctrl+C before it is done executing, you’ll see output similar to this.
$ python thread_stuck.py Thread-0 sleeping 1/5 Thread-0 sleeping 2/5 ^C^C^C^C^C^C^C^CThread-0 sleeping 3/5 Thread-0 sleeping 4/5 Thread-0 sleeping 5/5 Traceback (most recent call last): File "thread_stuck.py", line 24, in <module> main() File "thread_stuck.py", line 13, in main t.join() File "/Users/gzynda/miniconda3/envs/github-pages/lib/python2.7/threading.py", line 940, in join self.__block.wait() File "/Users/gzynda/miniconda3/envs/github-pages/lib/python2.7/threading.py", line 340, in wait waiter.acquire() KeyboardInterrupt
Making a functional example
I discovered that the
Thread.join() function not only blocks until the thread finishes, but it also ignores the
To get the program to exit, we need to modify our code to do two things:
- Catch and communicate the interrupt signal to the child thread with an attribute
- While the child thread is still active, try to join with a set timeout
First, when the sleepy thread is spawned, it uses the threading.current_thread() to retrieve a pointer to its own Thread object.
With this object, Threads can read and modify their own attributes, so we can initialize an
Then, whenever the thread starts another unit of work, have it exit if
In the main function, we first need to catch the
KeyboardInterrupt with try and except statements.
In the try section, we want to try to join the child thread every half a second until it is no longer alive with a while loop.
This half-second timeout allows for our interrupt signal to be processed.
KeyboardInterrupt signal is received, the thread’s
alive attribute is set to
False, signaling that work needs to stop.
After the thread stops working it is joined back and the main process can exit.
#!/usr/bin/env python from threading import Thread, current_thread from time import sleep import sys def main(): # Spawn a new thread that runs sleepy t = Thread(target=sleepy, args=(0,)) try: # Start the thread t.start() # If the child thread is still running while t.is_alive(): # Try to join the child thread back to parent for 0.5 seconds t.join(0.5) # When ctrl+c is received except KeyboardInterrupt as e: # Set the alive attribute to false t.alive = False # Block until child thread is joined back to the parent t.join() # Exit with error code sys.exit(e) def sleepy(t_id, n_loops=5): ''' Thread function ''' # Local reference of THIS thread object t = current_thread() # Thread is alive by default t.alive = True for i in range(n_loops): # If alive is set to false if not t.alive: print("Thread-%i detected alive=False"%(t_id)) # Break out of for loop break print("Thread-%i sleeping %i/%i"%(t_id, i+1, n_loops)) sleep(2) print("Thread-%i finished sleeping"%(t_id)) # Thread then stops running print("Thread-%i broke out of loop"%(t_id)) if __name__ == "__main__": main()
If you run this code, you can see that it is now possible cleanly exit with
$ python thread_killable.py Thread-0 sleeping 1/5 Thread-0 finished sleeping Thread-0 sleeping 2/5 ^CThread-0 finished sleeping Thread-0 detected alive=False Thread-0 broke out of loop
I recommend utilizing this functionality for interruptible python threads in your own code. Just note that this only works when each thread works through multiple chunks of work either through a Queue or a loop.
21 Dec 2018