distutils-r1 legacy concepts

This section describes concepts specific to the legacy mode of the distutils-r1 eclass. When using the modern PEP 517 mode, none of the features described here are available.

Different build system variations

The commonly used build systems specific to Python packages can be classified for eclass support into following groups:

  1. Plain distutils (built-in in Python).

  2. Setuptools and its direct derivatives (e.g. pbr).

  3. pyproject.toml-based build systems (Flit, Poetry).

The eclass supports the first two directly. Support for Flit and Poetry is provided through the dev-python/pyproject2setuppy package that converts the package’s metadata to setuptools call.

In addition to being a build system, setuptools provides runtime facilities via the pkg_resources module. If these facilities are used, the package needs to have a runtime dependency on dev-python/setuptools. Otherwise, a build-time dependency is sufficient.

DISTUTILS_USE_SETUPTOOLS

The most common case right now is a package using setuptools as a build system, and therefore needing a build-time dependency only. This is the eclass’ default. If your package does not fit this profile, you can set DISTUTILS_USE_SETUPTOOLS variable to one of the supported values:

  • no — pure distutils use (no extra dependencies).

  • bdepend — build-time use of setuptools (BDEPEND on dev-python/setuptools).

  • rdepend — build- and runtime use of setuptools (BDEPEND and RDEPEND on dev-python/setuptools).

  • pyproject.toml — use of Flit or Poetry (BDEPEND on dev-python/pyproject2toml and dev-python/setuptools).

  • manual — special value indicating that the eclass logic misbehaves and the dependency needs to be specified manually. Strongly discouraged, please report a bug and/or fix the package instead.

The cases for particular values are explained in subsequent sections.

The Gentoo repository includes a post-install QA check that verifies whether the value of DISTUTILS_USE_SETUPTOOLS is correct, and reports if it is most likely incorrect. This is why it is important to use the variable rather than specifying the dependency directly. An example report is:

* DISTUTILS_USE_SETUPTOOLS value is probably incorrect
*   have:     DISTUTILS_USE_SETUPTOOLS=bdepend (or unset)
*   expected: DISTUTILS_USE_SETUPTOOLS=rdepend

The value needs to be set before inheriting the eclass:

 # Copyright 1999-2020 Gentoo Authors
 # Distributed under the terms of the GNU General Public License v2

 EAPI=7

 PYTHON_COMPAT=( python2_7 python3_{6,7,8} pypy3 )
 DISTUTILS_USE_SETUPTOOLS=rdepend

 inherit distutils-r1

 DESCRIPTION="A configurable sidebar-enabled Sphinx theme"
 HOMEPAGE="https://github.com/bitprophet/alabaster"
 SRC_URI="mirror://pypi/${PN:0:1}/${PN}/${P}.tar.gz"

 LICENSE="BSD"
 KEYWORDS="~alpha ~amd64 ~arm ~arm64 ~hppa ~ia64 ~m68k ~mips ~ppc ~ppc64 ~s390 ~sh ~sparc ~x86 ~x64-solaris"
 SLOT="0"

distutils and setuptools build systems

Distutils and setuptools are the two most common build systems for Python packages right now. Their common feature is that they use a setup.py script that interfaces with the build system. Generally, you can determine which of the two build systems are being used by looking at the imports in setup.py, in particular from which module the setup function is imported.

Distutils-based packages (DISTUTILS_USE_SETUPTOOLS=no) use e.g.:

from distutils import setup

Setuptools-based package (DISTUTILS_USE_SETUPTOOLS=bdepend, unset or possibly rdepend as indicated by the subsequent sections) use:

from setuptools import setup

In some cases, upstreams find it convenient to alternatively support both setuptools and distutils. A commonly used snippet is:

try:
    from setuptools import setup
except ImportError:
    from distutils import setup

However, non-fixed build system choice can be problematic to Gentoo users. This is because pure distutils installs egg-info data as a single file, while setuptools install the same data as a directory (using the same path). Therefore, if you rebuild the same version of the package with a different build system than before, you end up trying to replace a file with a directory or the other way around. This is not permitted by the PMS and not handled cleanly by the package managers.

You must always ensure that a single build system will be used unconditionally. In the case of the condition presented above, it is sufficient to leave DISTUTILS_USE_SETUPTOOLS at its default value as that will ensure that setuptools is installed and therefore the fallback will never take place. However, patching setup.py may be necessary if you want to force distutils (e.g. to enable clean bootstrap) or the upstream condition requiers that.

Setuptools’ entry points

Important

With removal of Python 3.7, the correct DISTUTILS_USE_SETUPTOOLS value for packages using entry points changed to bdepend.

Entry points provide the ability to expose some of the package’s Python functions to other packages. They are commonly used to implement plugin systems and by setuptools themselves to implement wrapper scripts for starting programs.

Entry points are defined as entry_points argument to the setup() function, or entry_points section in setup.cfg. They are installed in the package’s egg-info as entry_points.txt. In both cases, they are grouped by entry point type, and defined as a dictionary mapping entry points names to the relevant functions.

For our purposes, we are only interested in entry points used to define wrapper scripts, the console_scripts and gui_scripts groups, as they are installed with the package itself. As for plugin systems, it is reasonable to assume that the installed plugins are only meaningful to the package using them, and therefore that the package using them will depend on the appropriate metadata provider.

Old versions of setuptools used to implement the script wrappers using pkg_resources package. Modern versions of setuptools use the following logic:

  1. If importlib.metadata module is available (Python 3.8+), use it. In this case, no external dependencies are necessary.

  2. If importlib_metadata backport is available, use it. It is provided by dev-python/importlib_metadata.

  3. Otherwise, fall back to pkg_resources. It is provided by dev-python/setuptools.

Since Python 3.7 is no longer present in Gentoo (we are not considering PyPy3.7 correctness important for the time being), new ebuilds do not need any additional dependencies for entry points and should use the default value (i.e. remove DISTUTILS_USE_SETUPTOOLS).

For the time being, the QA check for incorrect values is accepting both the new value and the old rdepend value. If you wish to be reminded about the update, you can add the following variable to your make.conf:

DISTUTILS_STRICT_ENTRY_POINTS=1

Please note that in some cases rdepend can still be the correct value, if there are other runtime uses of setuptools. In some cases the QA check will also trigger the wrong value because of leftover explicit dependencies on setuptools.

Other runtime uses of setuptools

Besides the generated wrapper scripts, the package code itself may use the setuptools or pkg_resources packages. The common cases for this include getting package metadata and resource files. This could also be a case for plugin managers and derived build systems.

As a rule of thumb, if any installed Python file imports setuptools or pkg_resources, the package needs to use the value of rdepend.

The QA check determines that this is the case by looking at the upstream dependencies (install_requires) installed by the package. It is quite common for packages to miss the dependency, or have a leftover dependency. If install_requires does not match actual imports in the installed modules, please submit a patch upstream.

pyproject.toml-based projects

The newer build systems used for Python packages avoid supplying setup.py and instead declare package’s metadata and build system information in pyproject.toml. Examples of these build systems are Flit and Poetry.

These build systems are generally very heavy and do not support plain installation to a directory. For this reason, Gentoo is using dev-python/pyproject2setuppy to provide a thin wrapper for installing these packages using setuptools.

To enable the necessary eclass logic and add appropriate build-time dependencies, specify the value of pyproject.toml to DISTUTILS_USE_SETUPTOOLS.

Strictly speaking, both Flit and Poetry do support entry points, and therefore some packages actually need a runtime dependency on setuptools. This is a known limitation, and it will probably not be addressed for the same reason as the logic for setuptools’ entry points is not updated.

In-source vs out-of-source builds

In the general definition, an out-of-source build is a build where output files are placed in a directory separate from source files. By default, distutils and its derivatives always do out-of-source builds and place output files in subdirectories of build directory.

Conversely, an in-source build happens when the output files are interspersed with source files. The closest distutils equivalent of an in-source build is the --inplace option of build_ext that places compiled C extensions alongside Python module sources.

distutils-r1 shifts this concept a little. When performing an out-of-source build (the default), it creates a dedicated output directory for every Python interpreter enabled, and then uses it throughout all build and install steps.

It should be noted that unlike build systems such as autotools or CMake, out-of-source builds in distutils are not executed from the build directory. Instead, the setup script is executed from source directory and passed path to build directory.

Sometimes out-of-source builds are incompatible with custom hacks used upstream. This could be a case if the setup script is writing implementation-specific changes to the source files (e.g. using 2to3 to convert them to Python 3) or relying on specific build paths. For better compatibility with those cases, the eclass provides an in-source build mode enabled via DISTUTILS_IN_SOURCE_BUILD.

In this mode, the eclass creates a separate copy of the source directory for each Python implementation, and then runs the build and install steps inside that copy. As a result, any changes done to the source files are contained within the copy used for the current interpreter.

 # Copyright 1999-2020 Gentoo Authors
 # Distributed under the terms of the GNU General Public License v2

 EAPI=7
 DISTUTILS_USE_SETUPTOOLS=no
 PYTHON_COMPAT=( python3_{6,7,8} pypy3 )
 PYTHON_REQ_USE="xml(+)"

 inherit distutils-r1

 DESCRIPTION="Collection of extensions to Distutils"
 HOMEPAGE="https://github.com/pypa/setuptools https://pypi.org/project/setuptools/"
 SRC_URI="mirror://pypi/${PN:0:1}/${PN}/${P}.zip"

 LICENSE="MIT"
 SLOT="0"
 KEYWORDS="~alpha ~amd64 ~arm ~arm64 ~hppa ~ia64 ~m68k ~mips ~ppc ~ppc64 ~riscv ~s390 ~sh ~sparc ~x86 ~x64-cygwin ~amd64-linux ~x86-linux ~ppc-macos ~x64-macos ~x86-macos ~sparc-solaris ~sparc64-solaris ~x64-solaris ~x86-solaris"

 # Force in-source build because build system modifies sources.
 DISTUTILS_IN_SOURCE_BUILD=1

Installing the package before testing

The tests are executed in src_test phase, after src_compile installed package files into the build directory. The eclass automatically adds appropriate PYTHONPATH so that the installed Python modules and extensions are used during testing. This works for the majority of packages.

However, some test suites will not work correctly unless the package has been properly installed via setup.py install. This may apply specifically to packages calling their executables that are created via entry points, various plugin systems or the use of package metadata.

The distutils_install_for_testing function runs setup.py install into a temporary directory, and adds the appropriate paths to PATH and PYTHONPATH.

This function currently supports two install layouts:

  • the standard root directory layout that is enabled via --via-root,

  • a virtualenv-alike venv layout that is enabled via --via-venv.

The eclass defaults to the root directory layout that is consistent with the layout used for the actual install. This ensures that the package’s scripts are found on PATH, and the package metadata is found via importlib.metadata / pkg_resources. It should be sufficient to resolve the most common test problems.

In some cases, particularly packages that do not preserve PYTHONPATH correctly, the virtualenv-alike layout (--via-venv) is better. Through wrapping the Python interpreter itself, it guarantees that the packages installed in the test environment are found independently of PYTHONPATH (just like a true venv). It should cover the few extreme cases.

In EAPIs prior to 8, an additional legacy --via-home layout used to be supported. It historically used to be necessary to fix problems with some packages. However, the underlying issues probably went away along with old versions of Python, and the removal of site.py hack has broken it for most of the consumers.

python_test() {
    distutils_install_for_testing
    epytest --no-network
}