How to use valgrind with python?

I am trying to memcheck a C python extension I am writing, but I’m having trouble setting up valgrind to work with python. I would really appreciate some advice. Just for context, this is Ubuntu 13.10, python 2.7.5+, and valgrind 3.8.1.

As per recommendation from Readme.valgrind I did the following.

1) Downloaded the python source with

sudo apt-get build-dep python2.7
apt-get source python2.7

2) Applied the code patch, i.e. “Uncomment Py_USING_MEMORY_DEBUGGER in Objects/obmalloc.c”.

3) Applied the suppression patch, i.e. “Uncomment the lines in Misc/valgrind-python.supp that suppress the warnings for PyObject_Free and PyObject_Realloc”

4) Compiled python with

./configure --prefix=/home/dejan/workspace/python --without-pymalloc
make -j4 install

Note that I did both 2 and 3, while README.valgrind says to do 2 or 3… more can’t hurt.

Now, let’s test this on some sample python code in test.py

print "Test"

Let’s run valgrind on python with this script

valgrind --tool=memcheck --leak-check=full --suppressions=python2.7-2.7.5/Misc/valgrind-python.supp bin/python test.py

Unexpectedly, there is still loads of reports from valgrind, with the first one being (and many more following)

==27944== HEAP SUMMARY:
==27944==     in use at exit: 857,932 bytes in 5,144 blocks  
==27944==   total heap usage: 22,766 allocs, 17,622 frees, 4,276,934 bytes allocated
==27944== 
==27944== 38 bytes in 1 blocks are possibly lost in loss record 24 of 1,343
==27944==    at 0x4C2A2DB: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==27944==    by 0x46B8DD: PyString_FromString (stringobject.c:143)
==27944==    by 0x439631: PyFile_FromFile (fileobject.c:157)
==27944==    by 0x4E9B4A: _PySys_Init (sysmodule.c:1383)
==27944==    by 0x4E29E9: Py_InitializeEx (pythonrun.c:222)
==27944==    by 0x4154B4: Py_Main (main.c:546)
==27944==    by 0x577DDE4: (below main) (libc-start.c:260)

Am I doing something wrong? Is there a way to valgrind a python script that doesn’t leak and get clean valgrind output?

Contents hide

Answers:

Method 1

Method 2

Answers:

Thank you for visiting the Q&A section on Magenaut. Please note that all the answers may not help you solve the issue immediately. So please treat them as advisements. If you found the post helpful (or not), leave a comment & I’ll get back to you as soon as possible.

Method 1

I found the answer here.

Python also needs to be compiled in debug mode, i.e.

./configure --prefix=/home/dejan/workspace/python --without-pymalloc --with-pydebug --with-valgrind

In addition, numpy has a suppresion file that gets rid of the extra valgrind warnings.

Method 2

Since python 3.6, there’s a PYTHONMALLOC environment variable which is available in release builds, without needing to recompile.

PYTHONMALLOC=malloc python3 foobar.py

This will disable pymalloc and just use the libc malloc directly, making it valgrind-friendly. This is equivalent to --without-pymalloc (and it is just as slow)

If valgrind is too slow, other values can be helpful. PYTHONMALLOC=debug and PYTHONMALLOC=malloc_debug add debug hooks on top of the default and the libc allocators respectively. Their effects, from the docs:

Newly allocated memory is filled with the byte 0xCB

Freed memory is filled with the byte 0xDB

Detect violations of the Python memory allocator API. For example, PyObject_Free() called on a memory block allocated by PyMem_Malloc().

Detect writes before the start of a buffer (buffer underflows)

Detect writes after the end of a buffer (buffer overflows)

Check that the GIL is held when allocator functions of PYMEM_DOMAIN_OBJ (ex: PyObject_Malloc()) and PYMEM_DOMAIN_MEM (ex: PyMem_Malloc()) domains are called.

This will catch some uninitialized reads, some use after free, some buffer under/overflows, etc, but won’t report leaks and won’t touch memory that isn’t allocated through python (When using glibc, the MALLOC_PERTURB_ and MALLOC_CHECK_ environment variables might help there)