Interrupting Python Threads

In my most recent project, rgc, I have been using the python threading library for concurrent operations. Python Threads are often overlooked because the python GIL forces them to share a single CPU core, but they are great for scaling I/O or subprocess calls without worrying about communication.

Things were running fine, but whenever I wanted to kill and restart my script, a simple Ctrl+C would not force it to exit. I had to background (Ctrl+Z) the process, and send a kill signal to the specific process ID (kill -9 [PID]) after looking it up.

Below is a simple script to demonstrate this issue. The main function spawns a (sleepy) thread that loops through work (sleep).

#!/usr/bin/env python

from threading import Thread
from time import sleep
import sys

def main():
	# Spawn a new thread that runs sleepy
	t = Thread(target=sleepy, args=(0,))
	# Start the thread
	t.start()
	# Join the child thread back to parent
	t.join()

def sleepy(t_id, n_loops=5):
	'''
	Thread function
	'''
	for i in range(n_loops):
		print("Thread-%i sleeping %i/%i"%(t_id, i+1, n_loops))
		sleep(2)

if __name__ == "__main__":
	main()

If you run the script and try to interrupt it with Ctrl+C before it is done executing, you’ll see output similar to this.

$ python thread_stuck.py 

Thread-0 sleeping 1/5
Thread-0 sleeping 2/5
^C^C^C^C^C^C^C^CThread-0 sleeping 3/5
Thread-0 sleeping 4/5
Thread-0 sleeping 5/5
Traceback (most recent call last):
  File "thread_stuck.py", line 24, in <module>
    main()
  File "thread_stuck.py", line 13, in main
    t.join()
  File "/Users/gzynda/miniconda3/envs/github-pages/lib/python2.7/threading.py", line 940, in join
    self.__block.wait()
  File "/Users/gzynda/miniconda3/envs/github-pages/lib/python2.7/threading.py", line 340, in wait
    waiter.acquire()
KeyboardInterrupt

Making a functional example

I discovered that the Thread.join() function not only blocks until the thread finishes, but it also ignores the KeyboardInterrupt signal. To get the program to exit, we need to modify our code to do two things:

  1. Catch and communicate the interrupt signal to the child thread with an attribute
  2. While the child thread is still active, try to join with a set timeout

First, when the sleepy thread is spawned, it uses the threading.current_thread() to retrieve a pointer to its own Thread object. With this object, Threads can read and modify their own attributes, so we can initialize an alive=True attribute. Then, whenever the thread starts another unit of work, have it exit if alive is False.

In the main function, we first need to catch the KeyboardInterrupt with try and except statements. In the try section, we want to try to join the child thread every half a second until it is no longer alive with a while loop. This half-second timeout allows for our interrupt signal to be processed. If the KeyboardInterrupt signal is received, the thread’s alive attribute is set to False, signaling that work needs to stop. After the thread stops working it is joined back and the main process can exit.

#!/usr/bin/env python

from threading import Thread, current_thread
from time import sleep
import sys

def main():
	# Spawn a new thread that runs sleepy
	t = Thread(target=sleepy, args=(0,))
	try:
		# Start the thread
		t.start()
		# If the child thread is still running
		while t.is_alive():
			# Try to join the child thread back to parent for 0.5 seconds
			t.join(0.5)
	# When ctrl+c is received
	except KeyboardInterrupt as e:
		# Set the alive attribute to false
		t.alive = False
		# Block until child thread is joined back to the parent
		t.join()
		# Exit with error code
		sys.exit(e)

def sleepy(t_id, n_loops=5):
	'''
	Thread function
	'''
	# Local reference of THIS thread object
	t = current_thread()
	# Thread is alive by default
	t.alive = True
	for i in range(n_loops):
		# If alive is set to false
		if not t.alive:
			print("Thread-%i detected alive=False"%(t_id))
			# Break out of for loop
			break
		print("Thread-%i sleeping %i/%i"%(t_id, i+1, n_loops))
		sleep(2)
		print("Thread-%i finished sleeping"%(t_id))
	# Thread then stops running
	print("Thread-%i broke out of loop"%(t_id))


if __name__ == "__main__":
	main()

If you run this code, you can see that it is now possible cleanly exit with Ctrl+C.

$ python thread_killable.py 

Thread-0 sleeping 1/5
Thread-0 finished sleeping
Thread-0 sleeping 2/5
^CThread-0 finished sleeping
Thread-0 detected alive=False
Thread-0 broke out of loop

I recommend utilizing this functionality for interruptible python threads in your own code. Just note that this only works when each thread works through multiple chunks of work either through a Queue or a loop.

21 Dec 2018

©2017- Gregory J Zynda.