Store output of subprocess.Popen call in a string

I’m trying to make a system call in Python and store the output to a string that I can manipulate in the Python program.

#!/usr/bin/python
import subprocess
p2 = subprocess.Popen("ntpq -p")

I’ve tried a few things including some of the suggestions here:

Retrieving the output of subprocess.call()

but without any luck.

Answers:

Thank you for visiting the Q&A section on Magenaut. Please note that all the answers may not help you solve the issue immediately. So please treat them as advisements. If you found the post helpful (or not), leave a comment & I’ll get back to you as soon as possible.

Method 1

In Python 2.7 or Python 3

Instead of making a Popen object directly, you can use the subprocess.check_output() function to store output of a command in a string:

from subprocess import check_output
out = check_output(["ntpq", "-p"])

In Python 2.4-2.6

Use the communicate method.

import subprocess
p = subprocess.Popen(["ntpq", "-p"], stdout=subprocess.PIPE)
out, err = p.communicate()

out is what you want.

Important note about the other answers

Note how I passed in the command. The "ntpq -p" example brings up another matter. Since Popen does not invoke the shell, you would use a list of the command and options—["ntpq", "-p"].

Method 2

This worked for me for redirecting stdout (stderr can be handled similarly):

from subprocess import Popen, PIPE
pipe = Popen(path, stdout=PIPE)
text = pipe.communicate()[0]

If it doesn’t work for you, please specify exactly the problem you’re having.

Method 3

Python 2: http://docs.python.org/2/library/subprocess.html#subprocess.Popen

from subprocess import PIPE, Popen

command = "ntpq -p"
process = Popen(command, stdout=PIPE, stderr=None, shell=True)
output = process.communicate()[0]
print output

In the Popen constructor, if shell is True, you should pass the command as a string rather than as a sequence. Otherwise, just split the command into a list:

command = ["ntpq", "-p"]
process = Popen(command, stdout=PIPE, stderr=None)

If you need to read also the standard error, into the Popen initialization, you should set stderr to PIPE or STDOUT:

command = "ntpq -p"
process = subprocess.Popen(command, stdout=PIPE, stderr=PIPE, shell=True)
output, error = process.communicate()

NOTE: Starting from Python 2.7, you could/should take advantage of subprocess.check_output (https://docs.python.org/2/library/subprocess.html#subprocess.check_output).

Python 3: https://docs.python.org/3/library/subprocess.html#subprocess.Popen

from subprocess import PIPE, Popen

command = "ntpq -p"
with Popen(command, stdout=PIPE, stderr=None, shell=True) as process:
    output = process.communicate()[0].decode("utf-8")
    print(output)

NOTE: If you’re targeting only versions of Python higher or equal than 3.5, then you could/should take advantage of subprocess.run (https://docs.python.org/3/library/subprocess.html#subprocess.run).

Method 4

In Python 3.7+ you can use the new capture_output= keyword argument for subprocess.run:

import subprocess

p = subprocess.run(["echo", "hello world!"], capture_output=True, text=True)
assert p.stdout == 'hello world!n'

Method 5

Assuming that pwd is just an example, this is how you can do it:

import subprocess

p = subprocess.Popen("pwd", stdout=subprocess.PIPE)
result = p.communicate()[0]
print result

See the subprocess documentation for another example and more information.

Method 6

for Python 2.7+ the idiomatic answer is to use subprocess.check_output()

You should also note the handling of arguments when invoking a subprocess, as it can be a little confusing….

If args is just single command with no args of its own (or you have shell=True set), it can be a string. Otherwise it must be a list.

for example… to invoke the ls command, this is fine:

from subprocess import check_call
check_call('ls')

so is this:

from subprocess import check_call
check_call(['ls',])

however, if you want to pass some args to the shell command, you can’t do this:

from subprocess import check_call
check_call('ls -al')

instead, you must pass it as a list:

from subprocess import check_call
check_call(['ls', '-al'])

the shlex.split() function can sometimes be useful to split a string into shell-like syntax before creating a subprocesses…
like this:

from subprocess import check_call
import shlex
check_call(shlex.split('ls -al'))

Method 7

This works perfectly for me:

import subprocess
try:
    #prints results and merges stdout and std
    result = subprocess.check_output("echo %USERNAME%", stderr=subprocess.STDOUT, shell=True)
    print result
    #causes error and merges stdout and stderr
    result = subprocess.check_output("copy testfds", stderr=subprocess.STDOUT, shell=True)
except subprocess.CalledProcessError, ex: # error code <> 0 
    print "--------error------"
    print ex.cmd
    print ex.message
    print ex.returncode
    print ex.output # contains stdout and stderr together

Method 8

This was perfect for me.
You will get the return code, stdout and stderr in a tuple.

from subprocess import Popen, PIPE

def console(cmd):
    p = Popen(cmd, shell=True, stdout=PIPE)
    out, err = p.communicate()
    return (p.returncode, out, err)

For Example:

result = console('ls -l')
print 'returncode: %s' % result[0]
print 'output: %s' % result[1]
print 'error: %s' % result[2]

Method 9

The accepted answer is still good, just a few remarks on newer features. Since python 3.6, you can handle encoding directly in check_output, see documentation. This returns a string object now:

import subprocess 
out = subprocess.check_output(["ls", "-l"], encoding="utf-8")

In python 3.7, a parameter capture_output was added to subprocess.run(), which does some of the Popen/PIPE handling for us, see the python docs :

import subprocess 
p2 = subprocess.run(["ls", "-l"], capture_output=True, encoding="utf-8")
p2.stdout

Method 10

I wrote a little function based on the other answers here:

def pexec(*args):
    return subprocess.Popen(args, stdout=subprocess.PIPE).communicate()[0].rstrip()

Usage:

changeset = pexec('hg','id','--id')
branch = pexec('hg','id','--branch')
revnum = pexec('hg','id','--num')
print('%s : %s (%s)' % (revnum, changeset, branch))

Method 11

 import os   
 list = os.popen('pwd').read()

In this case you will only have one element in the list.

Method 12

import subprocess
output = str(subprocess.Popen("ntpq -p",shell = True,stdout = subprocess.PIPE, 
stderr = subprocess.STDOUT).communicate()[0])

This is one line solution

Method 13

The following captures stdout and stderr of the process in a single variable. It is Python 2 and 3 compatible:

from subprocess import check_output, CalledProcessError, STDOUT

command = ["ls", "-l"]
try:
    output = check_output(command, stderr=STDOUT).decode()
    success = True 
except CalledProcessError as e:
    output = e.output.decode()
    success = False

If your command is a string rather than an array, prefix this with:

import shlex
command = shlex.split(command)

Method 14

Use check_output method of subprocess module

import subprocess

address = '192.168.x.x'
res = subprocess.check_output(['ping', address, '-c', '3'])

Finally parse the string

for line in res.splitlines():

Hope it helps, happy coding

Method 15

For python 3.5 I put up function based on previous answer. Log may be removed, thought it’s nice to have

import shlex
from subprocess import check_output, CalledProcessError, STDOUT


def cmdline(command):
    log("cmdline:{}".format(command))
    cmdArr = shlex.split(command)
    try:
        output = check_output(cmdArr,  stderr=STDOUT).decode()
        log("Success:{}".format(output))
    except (CalledProcessError) as e:
        output = e.output.decode()
        log("Fail:{}".format(output))
    except (Exception) as e:
        output = str(e);
        log("Fail:{}".format(e))
    return str(output)


def log(msg):
    msg = str(msg)
    d_date = datetime.datetime.now()
    now = str(d_date.strftime("%Y-%m-%d %H:%M:%S"))
    print(now + " " + msg)
    if ("LOG_FILE" in globals()):
        with open(LOG_FILE, "a") as myfile:
            myfile.write(now + " " + msg + "n")

All methods was sourced from stackoverflow.com or stackexchange.com, is licensed under cc by-sa 2.5, cc by-sa 3.0 and cc by-sa 4.0