1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
|
<!DOCTYPE html>
<html lang="en" data-content_root="./">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="viewport" content="width=device-width, initial-scale=1" />
<title>Advanced concepts — Gentoo Python Guide documentation</title>
<link rel="stylesheet" type="text/css" href="_static/pygments.css?v=d1102ebc" />
<link rel="stylesheet" type="text/css" href="_static/alabaster.css?v=12dfc556" />
<script src="_static/documentation_options.js?v=5929fcd5"></script>
<script src="_static/doctools.js?v=9a2dae69"></script>
<script src="_static/sphinx_highlight.js?v=dc90522c"></script>
<link rel="index" title="Index" href="genindex.html" />
<link rel="search" title="Search" href="search.html" />
<link rel="next" title="Expert python-r1 usage" href="expert-multi.html" />
<link rel="prev" title="pytest recipes" href="pytest.html" />
<link rel="stylesheet" href="_static/custom.css" type="text/css" />
</head><body>
<div class="document">
<div class="documentwrapper">
<div class="bodywrapper">
<div class="body" role="main">
<section id="advanced-concepts">
<h1>Advanced concepts<a class="headerlink" href="#advanced-concepts" title="Link to this heading">¶</a></h1>
<section id="namespace-packages">
<h2>Namespace packages<a class="headerlink" href="#namespace-packages" title="Link to this heading">¶</a></h2>
<section id="hierarchical-package-structure">
<h3>Hierarchical package structure<a class="headerlink" href="#hierarchical-package-structure" title="Link to this heading">¶</a></h3>
<p>Traditionally, Python packages were organized into a hierarchical
structure with modules and subpackages being located inside the parent
package directory. When submodules are imported, they are represented
as attributes on the parent module. Consider the following session:</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">import</span> <span class="nn">sphinx.addnodes</span>
<span class="gp">>>> </span><span class="n">sphinx</span>
<span class="go"><module 'sphinx' from '/usr/lib/python3.8/site-packages/sphinx/__init__.py'></span>
<span class="gp">>>> </span><span class="n">sphinx</span><span class="o">.</span><span class="n">addnodes</span>
<span class="go"><module 'sphinx.addnodes' from '/usr/lib/python3.8/site-packages/sphinx/addnodes.py'></span>
</pre></div>
</div>
<p>This works fine most of the time. However, it start being problematic
when multiple Gentoo packages install parts of the same top-level
package. This may happen e.g. with some plugin layouts where plugins
are installed inside the package. More commonly, it happens when
upstream wishes all their packages to start with a common component.</p>
<p>This is the case with Zope framework. Different Zope packages share
common <code class="docutils literal notranslate"><span class="pre">zope</span></code> top-level package. <code class="docutils literal notranslate"><span class="pre">dev-python/zope-interface</span></code>
installs into <code class="docutils literal notranslate"><span class="pre">zope.interface</span></code>, <code class="docutils literal notranslate"><span class="pre">dev-python/zope-event</span></code>
into <code class="docutils literal notranslate"><span class="pre">zope.event</span></code>. For this to work using the hierarchical layout,
a common package has to install <code class="docutils literal notranslate"><span class="pre">zope/__init__.py</span></code>, then other Zope
packages have to depend on it and install sub-packages inside that
directory. As far as installed packages are concerned, this is entirely
doable.</p>
<p>The real problem happens when we wish to test a freshly built package
that depends on an installed package. In that case, Python imports
<code class="docutils literal notranslate"><span class="pre">zope</span></code> from build directory that contains only <code class="docutils literal notranslate"><span class="pre">zope.interface</span></code>.
It will not be able to import <code class="docutils literal notranslate"><span class="pre">zope.event</span></code> that is installed in system
package directory:</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">import</span> <span class="nn">zope.interface</span>
<span class="gp">>>> </span><span class="n">zope</span>
<span class="go"><module 'zope' from '/tmp/portage/dev-python/zope-interface-4.7.1/work/zope.interface-4.7.1-python3_8/lib/zope/__init__.py'></span>
<span class="gp">>>> </span><span class="n">zope</span><span class="o">.</span><span class="n">interface</span>
<span class="go"><module 'zope.interface' from '/tmp/portage/dev-python/zope-interface-4.7.1/work/zope.interface-4.7.1-python3_8/lib/zope/interface/__init__.py'></span>
<span class="gp">>>> </span><span class="kn">import</span> <span class="nn">zope.event</span>
<span class="gt">Traceback (most recent call last):</span>
File <span class="nb">"<stdin>"</span>, line <span class="m">1</span>, in <span class="n"><module></span>
<span class="gr">ModuleNotFoundError</span>: <span class="n">No module named 'zope.event'</span>
</pre></div>
</div>
<p>Now, this could be worked around by copying all other subpackages back
to the build directory. However, there is a better solution.</p>
</section>
<section id="namespace-package-structure">
<h3>Namespace package structure<a class="headerlink" href="#namespace-package-structure" title="Link to this heading">¶</a></h3>
<p>Unlike traditional packages, namespace packages act as a kind of proxy.
They are not strictly bound to the containing directory, and instead
permit loading subpackages from all directories found in module search
path. If we make <code class="docutils literal notranslate"><span class="pre">zope</span></code> a namespace package, we can import both
the locally built <code class="docutils literal notranslate"><span class="pre">zope.interface</span></code> and system <code class="docutils literal notranslate"><span class="pre">zope.event</span></code>
packages:</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">import</span> <span class="nn">zope.interface</span>
<span class="gp">>>> </span><span class="kn">import</span> <span class="nn">zope.event</span>
<span class="gp">>>> </span><span class="n">zope</span>
<span class="go"><module 'zope' (namespace)></span>
<span class="gp">>>> </span><span class="n">zope</span><span class="o">.</span><span class="n">interface</span>
<span class="go"><module 'zope.interface' from '/tmp/portage/dev-python/zope-interface-4.7.1/work/zope.interface-4.7.1-python3_8/lib/zope/interface/__init__.py'></span>
<span class="gp">>>> </span><span class="n">zope</span><span class="o">.</span><span class="n">event</span>
<span class="go"><module 'zope.event' from '/usr/lib/python3.8/site-packages/zope/event/__init__.py'></span>
</pre></div>
</div>
<p>There are three common methods of creating namespace packages:</p>
<ol class="arabic simple">
<li><p><a class="reference external" href="https://www.python.org/dev/peps/pep-0420/">PEP 420</a> namespaces implemented in Python 3.3 and newer,</p></li>
<li><p>Using <a class="reference external" href="https://docs.python.org/3/library/pkgutil.html">pkgutil</a> standard library module,</p></li>
<li><p>Using <a class="reference external" href="https://setuptools.readthedocs.io/en/latest/setuptools.html#namespace-packages">namespace package support in setuptools</a> (discouraged).</p></li>
</ol>
<p>PEP 420 namespaces are created implicitly when a package directory
does not contain <code class="docutils literal notranslate"><span class="pre">__init__.py</span></code> file. While earlier versions
of Python (including Python 2.7) ignored such directories and did not
permit importing Python modules within them, Python 3.3 imports such
directories as namespace packages.</p>
<p>pkgutil namespaces use <code class="docutils literal notranslate"><span class="pre">__init__.py</span></code> with the following content:</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="n">__path__</span> <span class="o">=</span> <span class="nb">__import__</span><span class="p">(</span><span class="s1">'pkgutil'</span><span class="p">)</span><span class="o">.</span><span class="n">extend_path</span><span class="p">(</span><span class="n">__path__</span><span class="p">,</span> <span class="vm">__name__</span><span class="p">)</span>
</pre></div>
</div>
<p>setuptools namespace can use <code class="docutils literal notranslate"><span class="pre">__init__.py</span></code> with the following
content:</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="nb">__import__</span><span class="p">(</span><span class="s1">'pkg_resources'</span><span class="p">)</span><span class="o">.</span><span class="n">declare_namespace</span><span class="p">(</span><span class="vm">__name__</span><span class="p">)</span>
</pre></div>
</div>
<p>Alternatively, setuptools normally installs a <code class="docutils literal notranslate"><span class="pre">.pth</span></code> file that is
automatically loaded by Python and implicitly injects the namespace
into Python.</p>
<p>Both pkgutil and setuptools namespaces are portable to all versions
of Python.</p>
<p>More general information on the topic can be found under <a class="reference external" href="https://packaging.python.org/guides/packaging-namespace-packages/">packaging
namespace packages</a> in Python Packaging User Guide.</p>
</section>
<section id="determining-whether-namespaces-are-used">
<h3>Determining whether namespaces are used<a class="headerlink" href="#determining-whether-namespaces-are-used" title="Link to this heading">¶</a></h3>
<p>The exact method of detecting namespace packages depends on the type
of namespace used.</p>
<p>PEP 420 namespaces can generally be recognized by the lack
of <code class="docutils literal notranslate"><span class="pre">__init__.py</span></code> in an installed package directory. However, since
they do not require any specific action, distinguishing them is not very
important.</p>
<p>pkgutil namespaces can be recognized through the content of their
<code class="docutils literal notranslate"><span class="pre">__init__.py</span></code>. Generally, you should find it suspicious if it is
the only file in a top-level package directory, and if the name of this
directory is less specific than the package name (e.g. <code class="docutils literal notranslate"><span class="pre">zope</span></code> for
<code class="docutils literal notranslate"><span class="pre">zope.interface</span></code>, <code class="docutils literal notranslate"><span class="pre">ruamel</span></code> for <code class="docutils literal notranslate"><span class="pre">ruamel.yaml</span></code>). If you miss this,
then you will learn about it when the <code class="docutils literal notranslate"><span class="pre">__init__.py</span></code> file collides
between multiple packages.</p>
<p>setuptools namespaces usually do not install <code class="docutils literal notranslate"><span class="pre">__init__.py</span></code> but
do install a <code class="docutils literal notranslate"><span class="pre">.pth</span></code> file instead. Prior to installation, they can
also be recognized by <code class="docutils literal notranslate"><span class="pre">namespace_packages</span></code> option in <code class="docutils literal notranslate"><span class="pre">setup.py</span></code>
or <code class="docutils literal notranslate"><span class="pre">setup.cfg</span></code>. However, some packages use a custom <code class="docutils literal notranslate"><span class="pre">__init__.py</span></code>
file that does enable setuptools namespaces.</p>
</section>
<section id="adding-new-namespace-packages-to-gentoo">
<h3>Adding new namespace packages to Gentoo<a class="headerlink" href="#adding-new-namespace-packages-to-gentoo" title="Link to this heading">¶</a></h3>
<p>If the package uses PEP 420 namespaces, no special action is required.
Per PEP 420 layout, the package must not install <code class="docutils literal notranslate"><span class="pre">__init__.py</span></code> files
for namespaces.</p>
<p>If the package uses the regular setuptools namespace install method
(i.e. <code class="docutils literal notranslate"><span class="pre">namespace_packages</span></code> option), then the eclass detects that
and strips the namespaces automatically, e.g.:</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="o">*</span> <span class="n">python3_11</span><span class="p">:</span> <span class="n">running</span> <span class="n">distutils</span><span class="o">-</span><span class="n">r1_run_phase</span> <span class="n">distutils</span><span class="o">-</span><span class="n">r1_python_install</span>
<span class="o">*</span> <span class="n">Stripping</span> <span class="n">pkg_resources</span><span class="o">-</span><span class="n">style</span> <span class="n">namespace</span> <span class="n">ruamel</span>
<span class="o">*</span> <span class="n">Stripping</span> <span class="n">pkg_resources</span><span class="o">-</span><span class="n">style</span> <span class="n">namespace</span> <span class="n">ruamel</span><span class="o">.</span><span class="n">std</span>
</pre></div>
</div>
<p>If the package uses pkgutil-style or setuptools-style namespaces
via <code class="docutils literal notranslate"><span class="pre">__init__.py</span></code> files, these files need to be removed manually.
This is done after the PEP 517 build phase:</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>python_compile<span class="o">()</span><span class="w"> </span><span class="o">{</span>
<span class="w"> </span>distutils-r1_python_compile
<span class="w"> </span>rm<span class="w"> </span><span class="s2">"</span><span class="si">${</span><span class="nv">BUILD_DIR</span><span class="si">}</span><span class="s2">/install</span><span class="k">$(</span>python_get_sitedir<span class="k">)</span><span class="s2">"</span>/jaraco/__init__.py<span class="w"> </span><span class="o">||</span><span class="w"> </span>die
<span class="o">}</span>
</pre></div>
</div>
<p>Note that in some extreme cases, upstream combines namespace support
and other code in the <code class="docutils literal notranslate"><span class="pre">__init__.py</span></code> file. Naturally, this file cannot
be removed. No good solution has been found for this problem yet.</p>
<p>Some packages include an explicit <code class="docutils literal notranslate"><span class="pre">setuptools</span></code> runtime dependency
(<code class="docutils literal notranslate"><span class="pre">install_requires</span></code>) when using namespaces. If this is the only
use of <code class="docutils literal notranslate"><span class="pre">pkg_resources</span></code> and <code class="docutils literal notranslate"><span class="pre">setuptools</span></code> in installed package
sources, this dependency needs to be stripped, e.g.:</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>src_prepare<span class="o">()</span><span class="w"> </span><span class="o">{</span>
<span class="w"> </span><span class="c1"># strip rdep specific to namespaces</span>
<span class="w"> </span>sed<span class="w"> </span>-i<span class="w"> </span>-e<span class="w"> </span><span class="s2">"/'setuptools'/d"</span><span class="w"> </span>setup.py<span class="w"> </span><span class="o">||</span><span class="w"> </span>die
<span class="w"> </span>distutils-r1_src_prepare
<span class="o">}</span>
</pre></div>
</div>
</section>
<section id="legacy-namespace-packages-in-gentoo">
<h3>Legacy namespace packages in Gentoo<a class="headerlink" href="#legacy-namespace-packages-in-gentoo" title="Link to this heading">¶</a></h3>
<p>Historically, Gentoo has used <code class="docutils literal notranslate"><span class="pre">dev-python/namespace-*</span></code> packages
to support namespaces. This method is deprecated and it is in process
of being retired.</p>
</section>
</section>
</section>
</div>
</div>
</div>
<div class="sphinxsidebar" role="navigation" aria-label="main navigation">
<div class="sphinxsidebarwrapper">
<h1 class="logo"><a href="index.html">Gentoo Python Guide</a></h1>
<h3>Navigation</h3>
<p class="caption" role="heading"><span class="caption-text">Contents:</span></p>
<ul class="current">
<li class="toctree-l1"><a class="reference internal" href="preface.html">Preface</a></li>
<li class="toctree-l1"><a class="reference internal" href="interpreter.html">Python interpreters</a></li>
<li class="toctree-l1"><a class="reference internal" href="eclass.html">Choosing between Python eclasses</a></li>
<li class="toctree-l1"><a class="reference internal" href="basic.html">Common basics</a></li>
<li class="toctree-l1"><a class="reference internal" href="any.html">python-any-r1 — build-time dependency</a></li>
<li class="toctree-l1"><a class="reference internal" href="single.html">python-single-r1 — single-impl packages</a></li>
<li class="toctree-l1"><a class="reference internal" href="multi.html">python-r1 — multi-impl packages</a></li>
<li class="toctree-l1"><a class="reference internal" href="distutils.html">distutils-r1 — standard Python build systems</a></li>
<li class="toctree-l1"><a class="reference internal" href="test.html">Tests in Python packages</a></li>
<li class="toctree-l1"><a class="reference internal" href="distutils-legacy.html">distutils-r1 legacy concepts</a></li>
<li class="toctree-l1"><a class="reference internal" href="pypi.html">pypi — helper eclass for PyPI archives</a></li>
<li class="toctree-l1"><a class="reference internal" href="helper.html">Common helper functions</a></li>
<li class="toctree-l1"><a class="reference internal" href="depend.html">Advanced dependencies</a></li>
<li class="toctree-l1"><a class="reference internal" href="pytest.html">pytest recipes</a></li>
<li class="toctree-l1 current"><a class="current reference internal" href="#">Advanced concepts</a><ul>
<li class="toctree-l2"><a class="reference internal" href="#namespace-packages">Namespace packages</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="expert-multi.html">Expert python-r1 usage</a></li>
<li class="toctree-l1"><a class="reference internal" href="buildsys.html">Integration with build systems written in Python</a></li>
<li class="toctree-l1"><a class="reference internal" href="porting.html">Porting tips</a></li>
<li class="toctree-l1"><a class="reference internal" href="migration.html">Migration guides</a></li>
<li class="toctree-l1"><a class="reference internal" href="qawarn.html">QA checks and warnings</a></li>
<li class="toctree-l1"><a class="reference internal" href="package-maintenance.html">Python package maintenance</a></li>
<li class="toctree-l1"><a class="reference internal" href="interpreter-maintenance.html">Maintenance of Python implementations</a></li>
</ul>
<div class="relations">
<h3>Related Topics</h3>
<ul>
<li><a href="index.html">Documentation overview</a><ul>
<li>Previous: <a href="pytest.html" title="previous chapter">pytest recipes</a></li>
<li>Next: <a href="expert-multi.html" title="next chapter">Expert python-r1 usage</a></li>
</ul></li>
</ul>
</div>
<search id="searchbox" style="display: none" role="search">
<h3 id="searchlabel">Quick search</h3>
<div class="searchformwrapper">
<form class="search" action="search.html" method="get">
<input type="text" name="q" aria-labelledby="searchlabel" autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false"/>
<input type="submit" value="Go" />
</form>
</div>
</search>
<script>document.getElementById('searchbox').style.display = "block"</script>
</div>
</div>
<div class="clearer"></div>
</div>
<div class="footer">
©2020, Michał Górny, license: CC BY 4.0.
|
Powered by <a href="https://www.sphinx-doc.org/">Sphinx 7.3.7</a>
& <a href="https://alabaster.readthedocs.io">Alabaster 0.7.16</a>
|
<a href="_sources/concept.rst.txt"
rel="nofollow">Page source</a>
</div>
</body>
</html>
|
GitWeb