diff options
| author |   Michał Górny <mgorny@gentoo.org> | 2022-07-18 21:29:37 +0200 |
|---|---|---|
| committer | Michał Górny <mgorny@gentoo.org> | 2022-07-18 21:29:37 +0200 |
| commit | a110da536315c4478fcfab23b4b8d724805d0e1c (patch) | |
| tree | 45d74df294ed381b90a056af4e89a855e3a4fe07 /guide/qawarn.html | |
| parent | Update Guide to 56cd973 (diff) | |
| download | python-a110da536315c4478fcfab23b4b8d724805d0e1c.tar.gz python-a110da536315c4478fcfab23b4b8d724805d0e1c.tar.bz2 python-a110da536315c4478fcfab23b4b8d724805d0e1c.zip | |
Update Guide to 8137ef6
Signed-off-by: Michał Górny <mgorny@gentoo.org>
Diffstat (limited to 'guide/qawarn.html')
| -rw-r--r-- | guide/qawarn.html | 261 |
1 files changed, 261 insertions, 0 deletions
diff --git a/guide/qawarn.html b/guide/qawarn.html new file mode 100644 index 0000000..931911f --- /dev/null +++ b/guide/qawarn.html @@ -0,0 +1,261 @@ + +<!DOCTYPE html> + +<html lang="en"> + <head> + <meta charset="utf-8" /> + <meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="generator" content="Docutils 0.18.1: http://docutils.sourceforge.net/" /> + + <title>QA checks and warnings — Gentoo Python Guide documentation</title> + <link rel="stylesheet" type="text/css" href="_static/pygments.css" /> + <link rel="stylesheet" type="text/css" href="_static/alabaster.css" /> + <script data-url_root="./" id="documentation_options" src="_static/documentation_options.js"></script> + <script src="_static/jquery.js"></script> + <script src="_static/underscore.js"></script> + <script src="_static/_sphinx_javascript_frameworks_compat.js"></script> + <script src="_static/doctools.js"></script> + <link rel="index" title="Index" href="genindex.html" /> + <link rel="search" title="Search" href="search.html" /> + <link rel="next" title="Python package maintenance" href="package-maintenance.html" /> + <link rel="prev" title="Migration guides" href="migration.html" /> + + <link rel="stylesheet" href="_static/custom.css" type="text/css" /> + + + <meta name="viewport" content="width=device-width, initial-scale=0.9, maximum-scale=0.9" /> + + </head><body> + + + <div class="document"> + <div class="documentwrapper"> + <div class="bodywrapper"> + + + <div class="body" role="main"> + + <section id="qa-checks-and-warnings"> +<h1>QA checks and warnings<a class="headerlink" href="#qa-checks-and-warnings" title="Permalink to this heading">¶</a></h1> +<p>This section explains Python-related QA checks and the resulting QA +warnings that can be output while running the package manager or related +tooling.</p> +<section id="improved-qa-warning-reporting-in-portage"> +<h2>Improved QA warning reporting in Portage<a class="headerlink" href="#improved-qa-warning-reporting-in-portage" title="Permalink to this heading">¶</a></h2> +<p>Normally, Portage outputs QA warnings at specific phases of the build +process. They are usually interspersed with other verbose output, +and they are easy to miss, especially when building multiple packages +in a single batch.</p> +<p>To make them easier to catch, Portage’s elog system can be used +to repeat all the QA warnings once emerge exits. The required “echo” +module is already enabled by default, however it skips QA warnings +by default. To change that, set in your <code class="docutils literal notranslate"><span class="pre">make.conf</span></code>:</p> +<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span><span class="nv">PORTAGE_ELOG_CLASSES</span><span class="o">=</span><span class="s2">"log warn error qa"</span> +</pre></div> +</div> +<p>For more information on using Portage’s elog system, please refer +to <a class="reference external" href="https://gitweb.gentoo.org/proj/portage.git/tree/cnf/make.conf.example#n330">make.conf.example</a> included in the Portage distribution.</p> +</section> +<section id="compiled-bytecode-related-warnings"> +<h2>Compiled bytecode-related warnings<a class="headerlink" href="#compiled-bytecode-related-warnings" title="Permalink to this heading">¶</a></h2> +<p>To improve performance, the Python interpreter compiles Python sources +into bytecode. CPython and PyPy3 feature three optimization levels +that impact the bytecode size:</p> +<ol class="arabic simple"> +<li><p>no optimizations (the default)</p></li> +<li><p><code class="docutils literal notranslate"><span class="pre">-O</span></code> that removes assert statements from code</p></li> +<li><p><code class="docutils literal notranslate"><span class="pre">-OO</span></code> that removes assert statements and docstrings</p></li> +</ol> +<p>Normally, the compiled bytecode is stored on disk in <code class="docutils literal notranslate"><span class="pre">.pyc</span></code> files +(historically, Python 2 has also used <code class="docutils literal notranslate"><span class="pre">.pyo</span></code> files). +When these files are present and up-to-date, Python loads +the precompiled bytecode from them rather than creating it from +the source code, improving module loading time. When they are missing, +Python normally creates them automatically if it has write permissions +to the respective directory.</p> +<p>In Gentoo, we aim for packages to always byte-compile all their Python +files to all supported optimization levels. Besides improving module +loading performance, it ensures that the Python interpreters will not +attempt to write compiled bytecode themselves, effectively creating +files that are not monitored by the package manager (and therefore e.g. +are not removed when the respective package is uninstalled) or causing +sandbox violations while building other packages.</p> +<p>The Gentoo repository features a QA check to ensure that all installed +Python modules are byte-compiled for all optimization levels supported +by the respective Python interpreter, and that no stray compiled files +exist. This check is implemented using an auxiliary command found +in <code class="docutils literal notranslate"><span class="pre">app-portage/gpep517-7</span></code> and newer. This check can also be run +manually (with machine-readable output) using e.g.:</p> +<div class="highlight-text notranslate"><div class="highlight"><pre><span></span>$ python3.10 -m gpep517 verify-pyc --destdir=/tmp/portage/dev-python/trimesh-3.12.9/image +missing:/usr/lib/python3.10/site-packages/trimesh/resources/templates/__pycache__/blender_boolean.cpython-310.opt-1.pyc:/usr/lib/python3.10/site-packages/trimesh/resources/templates/blender_boolean.py +missing:/usr/lib/python3.10/site-packages/trimesh/resources/templates/__pycache__/blender_boolean.cpython-310.opt-2.pyc:/usr/lib/python3.10/site-packages/trimesh/resources/templates/blender_boolean.py +missing:/usr/lib/python3.10/site-packages/trimesh/resources/templates/__pycache__/blender_boolean.cpython-310.pyc:/usr/lib/python3.10/site-packages/trimesh/resources/templates/blender_boolean.py +</pre></div> +</div> +<section id="modules-are-not-byte-compiled"> +<h3>Modules are not byte-compiled<a class="headerlink" href="#modules-are-not-byte-compiled" title="Permalink to this heading">¶</a></h3> +<p>The most common QA warning that can be noticed while building packages +indicates that at least some of the expected <code class="docutils literal notranslate"><span class="pre">.pyc</span></code> files are missing. +For example:</p> +<div class="highlight-text notranslate"><div class="highlight"><pre><span></span>* QA Notice: This package installs one or more Python modules that are +* not byte-compiled. +* The following files are missing: +* +* /usr/lib/python3.10/site-packages/trimesh/resources/templates/__pycache__/blender_boolean.cpython-310.opt-1.pyc +* /usr/lib/python3.10/site-packages/trimesh/resources/templates/__pycache__/blender_boolean.cpython-310.opt-2.pyc +* /usr/lib/python3.10/site-packages/trimesh/resources/templates/__pycache__/blender_boolean.cpython-310.pyc + +* QA Notice: This package installs one or more Python modules that are +* not byte-compiled. +* The following files are missing: +* +* /usr/lib/python3.10/site-packages/blueman/__pycache__/Constants.cpython-310.opt-2.pyc +* /usr/lib/python3.10/site-packages/blueman/__pycache__/DeviceClass.cpython-310.opt-2.pyc +* /usr/lib/python3.10/site-packages/blueman/__pycache__/Functions.cpython-310.opt-2.pyc +* /usr/lib/python3.10/site-packages/blueman/__pycache__/Sdp.cpython-310.opt-2.pyc +[...] +</pre></div> +</div> +<p>There are three common causes for these warnings:</p> +<ol class="arabic simple"> +<li><p>The package’s build system nor the ebuild do not byte-compile +installed Python modules — the warning lists all optimization levels +for all installed modules.</p></li> +<li><p>The package’s build system byte-compiles installed modules only for +a subset of optimization levels — the warning lists all modules +but only for a subset of levels (the second example in the example +snippet).</p></li> +<li><p>The package installs <code class="docutils literal notranslate"><span class="pre">.py</span></code> files with incorrect syntax that can not +be byte-compiled and usually trigger syntax errors during the install +phase (the first example in the above snippet).</p></li> +</ol> +<p>For the first two cases, the better solution is to patch the respective +build system to perform byte-compilation for all optimization levels. +An acceptable workaround is to call <code class="docutils literal notranslate"><span class="pre">python_optimize</span></code> from the ebuild +(note that in PEP517 mode, distutils-r1 does that unconditionally).</p> +<p>For the third case, the only real solution is to submit a fix upstream +that renames files that do not contain valid Python modules to use +another suffix. For example, the template triggering the QA warning +in trimesh package could be renamed from <code class="docutils literal notranslate"><span class="pre">.py</span></code> to <code class="docutils literal notranslate"><span class="pre">.py.tmpl</span></code>.</p> +</section> +<section id="stray-compiled-bytecode"> +<h3>Stray compiled bytecode<a class="headerlink" href="#stray-compiled-bytecode" title="Permalink to this heading">¶</a></h3> +<p>The following QA warning indicates that there are stray <code class="docutils literal notranslate"><span class="pre">.pyc</span></code> files +that are not clearly matching any installed Python module-implementation +pair:</p> +<div class="highlight-text notranslate"><div class="highlight"><pre><span></span>* QA Notice: This package installs one or more compiled Python modules +* that do not match installed modules (or their implementation). +* The following files are stray: +* +* /usr/lib/python3.10/site-packages/SCons/Tool/docbook/__pycache__/__init__.cpython-35.pyc +* /usr/lib/python3.10/site-packages/SCons/Tool/docbook/__pycache__/__init__.cpython-36.pyc +* /usr/lib/python3.10/site-packages/SCons/Tool/docbook/__pycache__/__init__.cpython-38.pyc +</pre></div> +</div> +<p>There are two common causes for this:</p> +<ol class="arabic simple"> +<li><p>The package is shipping precompiled <code class="docutils literal notranslate"><span class="pre">.pyc</span></code> files and installing +them along with <code class="docutils literal notranslate"><span class="pre">.py</span></code> modules. The ebuild should remove the stray +files in <code class="docutils literal notranslate"><span class="pre">src_prepare</span></code> then.</p></li> +<li><p>The ebuild is attempting to remove some <code class="docutils literal notranslate"><span class="pre">.py</span></code> files after they have +been byte-compiled. It needs to be modified to either remove them +prior to the byte-compilation stage, or to fix the build system +not to install them in the first place.</p></li> +</ol> +</section> +</section> +</section> + + + </div> + + </div> + </div> + <div class="sphinxsidebar" role="navigation" aria-label="main navigation"> + <div class="sphinxsidebarwrapper"> +<h1 class="logo"><a href="index.html">Gentoo Python Guide</a></h1> + + + + + + + + +<h3>Navigation</h3> +<p class="caption" role="heading"><span class="caption-text">Contents:</span></p> +<ul class="current"> +<li class="toctree-l1"><a class="reference internal" href="preface.html">Preface</a></li> +<li class="toctree-l1"><a class="reference internal" href="interpreter.html">Python interpreters</a></li> +<li class="toctree-l1"><a class="reference internal" href="eclass.html">Choosing between Python eclasses</a></li> +<li class="toctree-l1"><a class="reference internal" href="basic.html">Common basics</a></li> +<li class="toctree-l1"><a class="reference internal" href="any.html">python-any-r1 — build-time dependency</a></li> +<li class="toctree-l1"><a class="reference internal" href="single.html">python-single-r1 — single-impl packages</a></li> +<li class="toctree-l1"><a class="reference internal" href="multi.html">python-r1 — multi-impl packages</a></li> +<li class="toctree-l1"><a class="reference internal" href="distutils.html">distutils-r1 — standard Python build systems</a></li> +<li class="toctree-l1"><a class="reference internal" href="distutils-legacy.html">distutils-r1 legacy concepts</a></li> +<li class="toctree-l1"><a class="reference internal" href="helper.html">Common helper functions</a></li> +<li class="toctree-l1"><a class="reference internal" href="depend.html">Advanced dependencies</a></li> +<li class="toctree-l1"><a class="reference internal" href="test.html">Resolving test suite problems</a></li> +<li class="toctree-l1"><a class="reference internal" href="pytest.html">pytest recipes</a></li> +<li class="toctree-l1"><a class="reference internal" href="concept.html">Advanced concepts</a></li> +<li class="toctree-l1"><a class="reference internal" href="expert-multi.html">Expert python-r1 usage</a></li> +<li class="toctree-l1"><a class="reference internal" href="buildsys.html">Integration with build systems written in Python</a></li> +<li class="toctree-l1"><a class="reference internal" href="porting.html">Porting tips</a></li> +<li class="toctree-l1"><a class="reference internal" href="migration.html">Migration guides</a></li> +<li class="toctree-l1 current"><a class="current reference internal" href="#">QA checks and warnings</a><ul> +<li class="toctree-l2"><a class="reference internal" href="#improved-qa-warning-reporting-in-portage">Improved QA warning reporting in Portage</a></li> +<li class="toctree-l2"><a class="reference internal" href="#compiled-bytecode-related-warnings">Compiled bytecode-related warnings</a></li> +</ul> +</li> +<li class="toctree-l1"><a class="reference internal" href="package-maintenance.html">Python package maintenance</a></li> +<li class="toctree-l1"><a class="reference internal" href="interpreter-maintenance.html">Maintenance of Python implementations</a></li> +</ul> + +<div class="relations"> +<h3>Related Topics</h3> +<ul> + <li><a href="index.html">Documentation overview</a><ul> + <li>Previous: <a href="migration.html" title="previous chapter">Migration guides</a></li> + <li>Next: <a href="package-maintenance.html" title="next chapter">Python package maintenance</a></li> + </ul></li> +</ul> +</div> +<div id="searchbox" style="display: none" role="search"> + <h3 id="searchlabel">Quick search</h3> + <div class="searchformwrapper"> + <form class="search" action="search.html" method="get"> + <input type="text" name="q" aria-labelledby="searchlabel" autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false"/> + <input type="submit" value="Go" /> + </form> + </div> +</div> +<script>document.getElementById('searchbox').style.display = "block"</script> + + + + + + + + + </div> + </div> + <div class="clearer"></div> + </div> + <div class="footer"> + ©2020, Michał Górny, license: CC BY 4.0. + + | + Powered by <a href="http://sphinx-doc.org/">Sphinx 5.0.2</a> + & <a href="https://github.com/bitprophet/alabaster">Alabaster 0.7.12</a> + + | + <a href="_sources/qawarn.rst.txt" + rel="nofollow">Page source</a> + </div> + + + + + </body> +</html>
\ No newline at end of file |