Catch Ctrl+C / SIGINT and exit multiprocesses gracefully in python

How do I catch a Ctrl+C in multiprocess python program and exit all processes gracefully, I need the solution to work both on unix and windows. I’ve tried the following:

import multiprocessing
import time
import signal
import sys

jobs = []

def worker():
    signal.signal(signal.SIGINT, signal_handler)
    while(True):
        time.sleep(1.1234)
        print "Working..."

def signal_handler(signal, frame):
    print 'You pressed Ctrl+C!'
    # for p in jobs:
    #     p.terminate()
    sys.exit(0)

if __name__ == "__main__":
    for i in range(50):
        p = multiprocessing.Process(target=worker)
        jobs.append(p)
        p.start()

And it’s kind of working, but I don’t think it’s the right solution.

Answers:

Thank you for visiting the Q&A section on Magenaut. Please note that all the answers may not help you solve the issue immediately. So please treat them as advisements. If you found the post helpful (or not), leave a comment & I’ll get back to you as soon as possible.

Method 1

The previously accepted solution has race conditions and it does not work with map and async functions.

The correct way to handle Ctrl+C/SIGINT with multiprocessing.Pool is to:

1. Make the process ignore SIGINT before a process Pool is created. This way created child processes inherit SIGINT handler.
2. Restore the original SIGINT handler in the parent process after a Pool has been created.
3. Use map_async and apply_async instead of blocking map and apply.
4. Wait on the results with timeout because the default blocking waits to ignore all signals. This is Python bug https://bugs.python.org/issue8296.

Putting it together:

#!/bin/env python
from __future__ import print_function

import multiprocessing
import os
import signal
import time

def run_worker(delay):
    print("In a worker process", os.getpid())
    time.sleep(delay)

def main():
    print("Initializng 2 workers")
    original_sigint_handler = signal.signal(signal.SIGINT, signal.SIG_IGN)
    pool = multiprocessing.Pool(2)
    signal.signal(signal.SIGINT, original_sigint_handler)
    try:
        print("Starting 2 jobs of 5 seconds each")
        res = pool.map_async(run_worker, [5, 5])
        print("Waiting for results")
        res.get(60) # Without the timeout this blocking call ignores all signals.
    except KeyboardInterrupt:
        print("Caught KeyboardInterrupt, terminating workers")
        pool.terminate()
    else:
        print("Normal termination")
        pool.close()
    pool.join()

if __name__ == "__main__":
    main()

As @YakovShklarov noted, there is a window of time between ignoring the signal and unignoring it in the parent process, during which the signal can be lost. Using pthread_sigmask instead to temporarily block the delivery of the signal in the parent process would prevent the signal from being lost, however, it is not available in Python-2.

Method 2

The solution is based on this link and this link and it solved the problem, I had to moved to Pool though:

import multiprocessing
import time
import signal
import sys

def init_worker():
    signal.signal(signal.SIGINT, signal.SIG_IGN)

def worker():
    while(True):
        time.sleep(1.1234)
        print "Working..."

if __name__ == "__main__":
    pool = multiprocessing.Pool(50, init_worker)
    try:
        for i in range(50):
            pool.apply_async(worker)

        time.sleep(10)
        pool.close()
        pool.join()

    except KeyboardInterrupt:
        print "Caught KeyboardInterrupt, terminating workers"
        pool.terminate()
        pool.join()

Method 3

Just handle KeyboardInterrupt-SystemExit exceptions in your worker process:

def worker():
    while(True):
        try:
            msg = self.msg_queue.get()
        except (KeyboardInterrupt, SystemExit):
            print("Exiting...")
            break

All methods was sourced from stackoverflow.com or stackexchange.com, is licensed under cc by-sa 2.5, cc by-sa 3.0 and cc by-sa 4.0