How do I run another script in Python without waiting for it to finish?

I am creating a little dashboard for a user that will allow him to run specific jobs. I am using Django so I want him to be able to click a link to start the job and then return the page back to him with a message that the job is running. The results of the job will be emailed to him later.

I believe I am supposed to use subprocess.Popen but I’m not sure of that. So in pseudocode, here is what I want to do:

if job == 1:
    run script in background: /path/to/script.py
    return 'Job is running'

Answers:

Thank you for visiting the Q&A section on Magenaut. Please note that all the answers may not help you solve the issue immediately. So please treat them as advisements. If you found the post helpful (or not), leave a comment & I’ll get back to you as soon as possible.

Method 1

p = subprocess.Popen([sys.executable, '/path/to/script.py'], 
                                    stdout=subprocess.PIPE, 
                                    stderr=subprocess.STDOUT)

That will start the subprocess in background. Your script will keep running normally.

Read the documentation here.

Method 2

Running this through a message queue is definitely the way to go if you’re thinking about long-term scaling. Send a message to the queue who’s running constantly in the background, and write job handlers to deal with the different sorts of messages.

Since you’re using Django, I think Beanstalkd is a pretty good fit. Here’s a pretty nice tutorial on the subject. The first comment in that article also has some good tips.

Personally I’ve rolled with a custom in-memory queue server written in Erlang, with Python-bindings written in C. But redis looks like it might work out as a great contender for future queuing/messaging-needs. Hope this helps!

Method 3

subprocess.Popen is indeed what you are looking for.

Method 4

Although if you find that you want to start communicating a bunch of information between the subprocess and the parent, you may want to consider a thread, or RPC framework like Twisted.

But most likely those are too heavy for your application.

All methods was sourced from stackoverflow.com or stackexchange.com, is licensed under cc by-sa 2.5, cc by-sa 3.0 and cc by-sa 4.0