What’s the difference between dist-packages and site-packages?

I’m a bit miffed by the python package installation process. Specifically, what’s the difference between packages installed in the dist-packages directory and the site-packages directory?

Answers:

Thank you for visiting the Q&A section on Magenaut. Please note that all the answers may not help you solve the issue immediately. So please treat them as advisements. If you found the post helpful (or not), leave a comment & I’ll get back to you as soon as possible.

Method 1

dist-packages is a Debian-specific convention that is also present in its derivatives, like Ubuntu. Modules are installed to dist-packages when they come from the Debian package manager into this location:

/usr/lib/python2.7/dist-packages

Since easy_install and pip are installed from the package manager, they also use dist-packages, but they put packages here:

/usr/local/lib/python2.7/dist-packages

From the Debian Python Wiki:

dist-packages instead of site-packages. Third party Python software
installed from Debian packages goes into dist-packages, not
site-packages. This is to reduce conflict between the system Python,
and any from-source Python build you might install manually.

This means that if you manually compile and install Python interpreter from source, it uses the site-packages directory. This allows you to keep the two installations separate, especially since Debian and Ubuntu rely on the system version of Python for many system utilities.

Method 2

dist-packages is the debian-specific directory where apt and friends install their stuff, and site-packages is the standard pip directory.

The problem is — what happens when different versions of the same package are present in different directories?

My solution to the problem is to make dist-packages a symlink to site-packages:

for d in $(find $WORKON_HOME -type d -name dist-packages); do
  pushd $d
  cd ..
  if test -d dist-packages/__pycache__; then
    mv -v dist-packages/__pycache__/* site-packages/__pycache__/
    rmdir -v dist-packages/__pycache__
  fi
  mv -v dist-packages/* site-packages/
  rmdir -v dist-packages
  ln -sv site-packages dist-packages
  popd
done

(if you are not using gnu tools, remove the -v option).

Method 3

Debian (and Ubuntu) has introduced its own convention

# python3 -m site

on Ubuntu Focal gives

sys.path = [
    '/qpid-dispatch',
    '/usr/lib/python38.zip',
    '/usr/lib/python3.8',
    '/usr/lib/python3.8/lib-dynload',
    '/usr/local/lib/python3.8/dist-packages',
    '/usr/lib/python3/dist-packages',
]
USER_BASE: '/root/.local' (doesn't exist)
USER_SITE: '/root/.local/lib/python3.8/site-packages' (doesn't exist)
ENABLE_USER_SITE: True

The convention, as described in the linked mailing list, is that python deb packages installed by the distribution package manager go into /usr/lib/python3/dist-packages and packages installed using sudo pip3 go into /usr/local/lib/python3.8/dist-packages.

If you compile and install your own Python interpreter, it will default to placing itself in /usr/local, with /usr/local/bin/pip3 installs going into /usr/local/lib/pythonX.Y/site-packages.

The point of the Debian convention is to keep the three sets of packages separate:

1. python packages installed by apt
2. packages installed by root user with /usr/bin/pip3
3. packages installed by root user with their own /usr/local/bin/pip3

All methods was sourced from stackoverflow.com or stackexchange.com, is licensed under cc by-sa 2.5, cc by-sa 3.0 and cc by-sa 4.0