1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
|
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="generator" content="Docutils 0.19: https://docutils.sourceforge.io/" />
<title>Porting tips — Gentoo Python Guide documentation</title>
<link rel="stylesheet" type="text/css" href="_static/pygments.css" />
<link rel="stylesheet" type="text/css" href="_static/alabaster.css" />
<script data-url_root="./" id="documentation_options" src="_static/documentation_options.js"></script>
<script src="_static/jquery.js"></script>
<script src="_static/underscore.js"></script>
<script src="_static/_sphinx_javascript_frameworks_compat.js"></script>
<script src="_static/doctools.js"></script>
<script src="_static/sphinx_highlight.js"></script>
<link rel="index" title="Index" href="genindex.html" />
<link rel="search" title="Search" href="search.html" />
<link rel="next" title="Migration guides" href="migration.html" />
<link rel="prev" title="Integration with build systems written in Python" href="buildsys.html" />
<link rel="stylesheet" href="_static/custom.css" type="text/css" />
<meta name="viewport" content="width=device-width, initial-scale=0.9, maximum-scale=0.9" />
</head><body>
<div class="document">
<div class="documentwrapper">
<div class="bodywrapper">
<div class="body" role="main">
<section id="porting-tips">
<h1>Porting tips<a class="headerlink" href="#porting-tips" title="Permalink to this heading">¶</a></h1>
<p>This section highlights some of the known incompatible changes made
in Python that could break Python scripts and modules that used to work
in prior versions. The sections are split into retroactive changes made
to all Python releases, and information specific to every Python branch
(compared to the previous one).</p>
<p>This guide is by no means considered complete. If you can think
of other problems you’ve hit while porting your packages, please let me
know and I will update it.</p>
<section id="retroactive-changes">
<h2>Retroactive changes<a class="headerlink" href="#retroactive-changes" title="Permalink to this heading">¶</a></h2>
<section id="bpo43882-urlsplit-now-strips-lf-cr-and-ht-characters">
<h3><a class="reference external" href="https://bugs.python.org/issue43882">bpo43882</a>: urlsplit now strips LF, CR and HT characters<a class="headerlink" href="#bpo43882-urlsplit-now-strips-lf-cr-and-ht-characters" title="Permalink to this heading">¶</a></h3>
<p>Changed in: 2.7.18_p9, 3.6.13_p3, 3.7.10_p3, 3.8.9_p2, 3.9.4_p1</p>
<p>Historically, various <a class="reference external" href="https://docs.python.org/3/library/urllib.parse.html">urllib.parse</a> methods have passed special
characters such as LF, CR and HT through into the split URL components.
This could have resulted in various exploits if Python programs did not
validate the resulting components and used them verbatim.</p>
<p><a class="reference external" href="https://bugs.python.org/issue43882">bpo43882</a> attempted to address the issue by making <a class="reference external" href="https://docs.python.org/3/library/urllib.parse.html">urllib.parse</a> strip
the three aforementioned characters from the output of its functions.
This fixed one class of potential issues but at the same time opened
another can of worms. For example, URL validators that used to check
for dangerous special characters in the split URL components stopped
working correctly. In the best case, the URL were now sanitized instead
of being rejected. In the worst, the original unparsed URL with
dangerous characters started being passed through. See e.g. <a class="reference external" href="https://github.com/django/django/pull/14349">django
PR#14349</a> for an example of impact and a fix.</p>
<p>Behavior before:</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">urllib</span><span class="o">.</span><span class="n">parse</span><span class="o">.</span><span class="n">urlparse</span><span class="p">(</span><span class="s1">'https://example.com/bad</span><span class="se">\n</span><span class="s1">url'</span><span class="p">)</span>
<span class="go">ParseResult(scheme='https', netloc='example.com', path='/bad\nurl', params='', query='', fragment='')</span>
</pre></div>
</div>
<p>Behavior after:</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">urllib</span><span class="o">.</span><span class="n">parse</span><span class="o">.</span><span class="n">urlparse</span><span class="p">(</span><span class="s1">'https://example.com/bad</span><span class="se">\n</span><span class="s1">url'</span><span class="p">)</span>
<span class="go">ParseResult(scheme='https', netloc='example.com', path='/badurl', params='', query='', fragment='')</span>
</pre></div>
</div>
</section>
</section>
<section id="python-3-11">
<h2>Python 3.11<a class="headerlink" href="#python-3-11" title="Permalink to this heading">¶</a></h2>
<p>See also: <a class="reference external" href="https://docs.python.org/3.11/whatsnew/3.11.html">what’s new in Python 3.11</a></p>
<section id="generator-based-coroutine-removal-asyncio-coroutine">
<h3>Generator-based coroutine removal (asyncio.coroutine)<a class="headerlink" href="#generator-based-coroutine-removal-asyncio-coroutine" title="Permalink to this heading">¶</a></h3>
<p>Support for <a class="reference external" href="https://docs.python.org/3.10/library/asyncio-task.html#generator-based-coroutines">generator-based coroutines</a> has been deprecated since
Python 3.8, and is finally removed in 3.11. This usually results
in the following error:</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="ne">AttributeError</span><span class="p">:</span> <span class="n">module</span> <span class="s1">'asyncio'</span> <span class="n">has</span> <span class="n">no</span> <span class="n">attribute</span> <span class="s1">'coroutine'</span>
</pre></div>
</div>
<p>The recommended solution is to use <a class="reference external" href="https://docs.python.org/3.10/library/asyncio-task.html#coroutines">PEP 492 coroutines</a>. They are
available since Python 3.5. This means replacing
the <code class="docutils literal notranslate"><span class="pre">@asyncio.coroutine</span></code> decorator with <code class="docutils literal notranslate"><span class="pre">async</span> <span class="pre">def</span></code> keyword,
and <code class="docutils literal notranslate"><span class="pre">yield</span> <span class="pre">from</span></code> with <code class="docutils literal notranslate"><span class="pre">await</span></code>.</p>
<p>For example, the following snippet:</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="nd">@asyncio</span><span class="o">.</span><span class="n">coroutine</span>
<span class="k">def</span> <span class="nf">foo</span><span class="p">():</span>
<span class="k">yield from</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">5</span><span class="p">)</span>
</pre></div>
</div>
<p>would become:</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="k">async</span> <span class="k">def</span> <span class="nf">foo</span><span class="p">():</span>
<span class="k">await</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">5</span><span class="p">)</span>
</pre></div>
</div>
</section>
<section id="inspect-getargspec-and-inspect-formatargspec-removal">
<h3>inspect.getargspec() and inspect.formatargspec() removal<a class="headerlink" href="#inspect-getargspec-and-inspect-formatargspec-removal" title="Permalink to this heading">¶</a></h3>
<p>The <a class="reference external" href="https://docs.python.org/3.10/library/inspect.html#inspect.getargspec">inspect.getargspec()</a> (deprecated since Python 3.0)
and <a class="reference external" href="https://docs.python.org/3.10/library/inspect.html#inspect.formatargspec">inspect.formatargspec()</a> (deprecated since Python 3.5) functions
are both removed in Python 3.11.</p>
<p>The <a class="reference external" href="https://docs.python.org/3.10/library/inspect.html#inspect.getargspec">inspect.getargspec()</a> function provides a legacy interface
to inspect the signature of callables. It is replaced
by the object-oriented <a class="reference external" href="https://docs.python.org/3.10/library/inspect.html#inspect.signature">inspect.signature()</a> API (available since
Python 3.3), or a mostly compatible <a class="reference external" href="https://docs.python.org/3.10/library/inspect.html#inspect.getfullargspec">inspect.getfullargspec()</a> function
(available since Python 3.0).</p>
<p>For example, a trivial function would yield the following results:</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="k">def</span> <span class="nf">foo</span><span class="p">(</span><span class="n">p1</span><span class="p">,</span> <span class="n">p2</span><span class="p">,</span> <span class="o">/</span><span class="p">,</span> <span class="n">kp3</span><span class="p">,</span> <span class="n">kp4</span> <span class="o">=</span> <span class="mi">10</span><span class="p">,</span> <span class="n">kp5</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
<span class="gp">... </span> <span class="k">pass</span>
<span class="gp">...</span>
<span class="gp">>>> </span><span class="n">inspect</span><span class="o">.</span><span class="n">getargspec</span><span class="p">(</span><span class="n">foo</span><span class="p">)</span>
<span class="go">ArgSpec(args=['p1', 'p2', 'kp3', 'kp4', 'kp5'],</span>
<span class="go"> varargs='args',</span>
<span class="go"> keywords='kwargs',</span>
<span class="go"> defaults=(10, None))</span>
<span class="gp">>>> </span><span class="n">inspect</span><span class="o">.</span><span class="n">getfullargspec</span><span class="p">(</span><span class="n">foo</span><span class="p">)</span>
<span class="go">FullArgSpec(args=['p1', 'p2', 'kp3', 'kp4', 'kp5'],</span>
<span class="go"> varargs='args',</span>
<span class="go"> varkw='kwargs',</span>
<span class="go"> defaults=(10, None),</span>
<span class="go"> kwonlyargs=[],</span>
<span class="go"> kwonlydefaults=None,</span>
<span class="go"> annotations={})</span>
<span class="gp">>>> </span><span class="n">inspect</span><span class="o">.</span><span class="n">signature</span><span class="p">(</span><span class="n">foo</span><span class="p">)</span>
<span class="go"><Signature (p1, p2, /, kp3, kp4=10, kp5=None, *args, **kwargs)></span>
</pre></div>
</div>
<p>The named tuple returned by <a class="reference external" href="https://docs.python.org/3.10/library/inspect.html#inspect.getfullargspec">inspect.getfullargspec()</a> starts with
the same information, except that the key used to hold the name
of <code class="docutils literal notranslate"><span class="pre">**</span></code> parameter is <code class="docutils literal notranslate"><span class="pre">varkw</span></code> rather than <code class="docutils literal notranslate"><span class="pre">keywords</span></code>.
<a class="reference external" href="https://docs.python.org/3.10/library/inspect.html#inspect.signature">inspect.signature()</a> returns a <code class="docutils literal notranslate"><span class="pre">Signature</span></code> object.</p>
<p>Both of the newer functions support keyword-only arguments and type
annotations:</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="k">def</span> <span class="nf">foo</span><span class="p">(</span><span class="n">p1</span><span class="p">:</span> <span class="nb">int</span><span class="p">,</span> <span class="n">p2</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="o">/</span><span class="p">,</span> <span class="n">kp3</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">kp4</span><span class="p">:</span> <span class="nb">int</span> <span class="o">=</span> <span class="mi">10</span><span class="p">,</span>
<span class="gp">... </span> <span class="n">kp5</span><span class="p">:</span> <span class="nb">float</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="n">k6</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">k7</span><span class="p">:</span> <span class="nb">int</span> <span class="o">=</span> <span class="mi">12</span><span class="p">,</span>
<span class="gp">... </span> <span class="n">k8</span><span class="p">:</span> <span class="nb">float</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span> <span class="o">-></span> <span class="nb">float</span><span class="p">:</span>
<span class="gp">... </span> <span class="k">pass</span>
<span class="gp">...</span>
<span class="gp">>>> </span><span class="n">inspect</span><span class="o">.</span><span class="n">getfullargspec</span><span class="p">(</span><span class="n">foo</span><span class="p">)</span>
<span class="go">FullArgSpec(args=['p1', 'p2', 'kp3', 'kp4', 'kp5'],</span>
<span class="go"> varargs='args',</span>
<span class="go"> varkw='kwargs',</span>
<span class="go"> defaults=(10, None),</span>
<span class="go"> kwonlyargs=['k6', 'k7', 'k8'],</span>
<span class="go"> kwonlydefaults={'k7': 12},</span>
<span class="go"> annotations={'return': <class 'float'>,</span>
<span class="go"> 'p1': <class 'int'>,</span>
<span class="go"> 'p2': <class 'str'>,</span>
<span class="go"> 'kp3': <class 'str'>,</span>
<span class="go"> 'kp4': <class 'int'>,</span>
<span class="go"> 'kp5': <class 'float'>,</span>
<span class="go"> 'k6': <class 'str'>,</span>
<span class="go"> 'k7': <class 'int'>,</span>
<span class="go"> 'k8': <class 'float'>})</span>
<span class="gp">>>> </span><span class="n">inspect</span><span class="o">.</span><span class="n">signature</span><span class="p">(</span><span class="n">foo</span><span class="p">)</span>
<span class="go"><Signature (p1: int, p2: str, /, kp3: str, kp4: int = 10,</span>
<span class="go"> kp5: float = None, *args, k6: str, k7: int = 12,</span>
<span class="go"> k8: float, **kwargs) -> float></span>
</pre></div>
</div>
<p>One notable difference between <a class="reference external" href="https://docs.python.org/3.10/library/inspect.html#inspect.signature">inspect.signature()</a> and the two other
functions is that the latter always include the ‘self’ argument
of method prototypes, while the former skips it if the method is bound
to an object. That is:</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="k">class</span> <span class="nc">foo</span><span class="p">:</span>
<span class="gp">... </span> <span class="k">def</span> <span class="nf">x</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">bar</span><span class="p">):</span>
<span class="gp">... </span> <span class="k">pass</span>
<span class="gp">...</span>
<span class="gp">>>> </span><span class="n">inspect</span><span class="o">.</span><span class="n">getargspec</span><span class="p">(</span><span class="n">foo</span><span class="o">.</span><span class="n">x</span><span class="p">)</span>
<span class="go">ArgSpec(args=['self', 'bar'], varargs=None, keywords=None, defaults=None)</span>
<span class="gp">>>> </span><span class="n">inspect</span><span class="o">.</span><span class="n">getargspec</span><span class="p">(</span><span class="n">foo</span><span class="p">()</span><span class="o">.</span><span class="n">x</span><span class="p">)</span>
<span class="go">ArgSpec(args=['self', 'bar'], varargs=None, keywords=None, defaults=None)</span>
<span class="gp">>>> </span><span class="n">inspect</span><span class="o">.</span><span class="n">signature</span><span class="p">(</span><span class="n">foo</span><span class="o">.</span><span class="n">x</span><span class="p">)</span>
<span class="go"><Signature (self, bar)></span>
<span class="gp">>>> </span><span class="n">inspect</span><span class="o">.</span><span class="n">signature</span><span class="p">(</span><span class="n">foo</span><span class="p">()</span><span class="o">.</span><span class="n">x</span><span class="p">)</span>
<span class="go"><Signature (bar)></span>
</pre></div>
</div>
<p>The <a class="reference external" href="https://docs.python.org/3.10/library/inspect.html#inspect.formatargspec">inspect.formatargspec()</a> function provides a pretty-formatted
argument spec from the tuple returned by <a class="reference external" href="https://docs.python.org/3.10/library/inspect.html#inspect.getfullargspec">inspect.getfullargspec()</a>
(or <a class="reference external" href="https://docs.python.org/3.10/library/inspect.html#inspect.getargspec">inspect.getargspec()</a>). It is replaced by stringification
of <code class="docutils literal notranslate"><span class="pre">Signature</span></code> objects:</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="k">def</span> <span class="nf">foo</span><span class="p">(</span><span class="n">p1</span><span class="p">:</span> <span class="nb">int</span><span class="p">,</span> <span class="n">p2</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="o">/</span><span class="p">,</span> <span class="n">kp3</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">kp4</span><span class="p">:</span> <span class="nb">int</span> <span class="o">=</span> <span class="mi">10</span><span class="p">,</span>
<span class="gp">... </span> <span class="n">kp5</span><span class="p">:</span> <span class="nb">float</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="n">k6</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">k7</span><span class="p">:</span> <span class="nb">int</span> <span class="o">=</span> <span class="mi">12</span><span class="p">,</span>
<span class="gp">... </span> <span class="n">k8</span><span class="p">:</span> <span class="nb">float</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span> <span class="o">-></span> <span class="nb">float</span><span class="p">:</span>
<span class="gp">... </span> <span class="k">pass</span>
<span class="gp">...</span>
<span class="gp">>>> </span><span class="n">inspect</span><span class="o">.</span><span class="n">formatargspec</span><span class="p">(</span><span class="o">*</span><span class="n">inspect</span><span class="o">.</span><span class="n">getfullargspec</span><span class="p">(</span><span class="n">foo</span><span class="p">))</span>
<span class="go">'(p1: int, p2: str, kp3: str, kp4: int=10, kp5: float=None, '</span>
<span class="go">'*args, k6: str, k7: int=12, k8: float, **kwargs) -> float'</span>
<span class="gp">>>> </span><span class="nb">str</span><span class="p">(</span><span class="n">inspect</span><span class="o">.</span><span class="n">signature</span><span class="p">(</span><span class="n">foo</span><span class="p">))</span>
<span class="go">'(p1: int, p2: str, /, kp3: str, kp4: int = 10, kp5: float = None, '</span>
<span class="go">'*args, k6: str, k7: int = 12, k8: float, **kwargs) -> float'</span>
</pre></div>
</div>
</section>
</section>
<section id="python-3-10">
<h2>Python 3.10<a class="headerlink" href="#python-3-10" title="Permalink to this heading">¶</a></h2>
<p>See also: <a class="reference external" href="https://docs.python.org/3/whatsnew/3.10.html">what’s new in Python 3.10</a></p>
<section id="configure-no-package-python-3-1-found">
<h3>configure: No package ‘python-3.1’ found<a class="headerlink" href="#configure-no-package-python-3-1-found" title="Permalink to this heading">¶</a></h3>
<p>automake prior to 1.16.3 wrongly recognized Python 3.10 as 3.1.
As a result, build with Python 3.10 fails:</p>
<div class="highlight-console notranslate"><div class="highlight"><pre><span></span><span class="go">checking for python version... 3.1</span>
<span class="go">checking for python platform... linux</span>
<span class="go">checking for python script directory... ${prefix}/lib/python3.10/site-packages</span>
<span class="go">checking for python extension module directory... ${exec_prefix}/lib/python3.10/site-packages</span>
<span class="go">checking for PYTHON... no</span>
<span class="go">configure: error: Package requirements (python-3.1) were not met:</span>
<span class="go">No package 'python-3.1' found</span>
<span class="go">Consider adjusting the PKG_CONFIG_PATH environment variable if you</span>
<span class="go">installed software in a non-standard prefix.</span>
<span class="go">Alternatively, you may set the environment variables PYTHON_CFLAGS</span>
<span class="go">and PYTHON_LIBS to avoid the need to call pkg-config.</span>
<span class="go">See the pkg-config man page for more details.</span>
<span class="go">Error: Process completed with exit code 1.</span>
</pre></div>
</div>
<p>To resolve this in ebuild, you need to autoreconf with the Gentoo
distribution of automake:</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="n">inherit</span> <span class="n">autotools</span>
<span class="c1"># ...</span>
<span class="n">src_prepare</span><span class="p">()</span> <span class="p">{</span>
<span class="n">default</span>
<span class="n">eautoreconf</span>
<span class="p">}</span>
</pre></div>
</div>
<p>The upstream fix is to create new distfiles using automake-1.16.3+.</p>
</section>
<section id="distutils-sysconfig-deprecation">
<h3>distutils.sysconfig deprecation<a class="headerlink" href="#distutils-sysconfig-deprecation" title="Permalink to this heading">¶</a></h3>
<p>Upstream intends to remove distutils by Python 3.12. Python 3.10 starts
throwing deprecation warnings for various distutils modules.
The distutils.sysconfig is usually easy to port.</p>
<p>The following table summarizes replacements for common path getters.</p>
<blockquote>
<div><table class="docutils align-default">
<thead>
<tr class="row-odd"><th class="head"><p>distutils.sysconfig call</p></th>
<th class="head"><p>sysconfig replacement</p></th>
</tr>
</thead>
<tbody>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">get_python_inc(False)</span></code></p></td>
<td><p><code class="docutils literal notranslate"><span class="pre">get_path("include")</span></code></p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">get_python_inc(True)</span></code></p></td>
<td><p><code class="docutils literal notranslate"><span class="pre">get_path("platinclude")</span></code></p></td>
</tr>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">get_python_lib(False,</span> <span class="pre">False)</span></code></p></td>
<td><p><code class="docutils literal notranslate"><span class="pre">get_path("purelib")</span></code></p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">get_python_lib(True,</span> <span class="pre">False)</span></code></p></td>
<td><p><code class="docutils literal notranslate"><span class="pre">get_path("platlib")</span></code></p></td>
</tr>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">get_python_lib(False,</span> <span class="pre">True)</span></code></p></td>
<td><p><code class="docutils literal notranslate"><span class="pre">get_path("stdlib")</span></code></p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">get_python_lib(True,</span> <span class="pre">True)</span></code></p></td>
<td><p><code class="docutils literal notranslate"><span class="pre">get_path("platstdlib")</span></code></p></td>
</tr>
</tbody>
</table>
</div></blockquote>
<p>For both functions, omitted parameters default to <code class="docutils literal notranslate"><span class="pre">False</span></code>. There is
no trivial replacement for the variants with <code class="docutils literal notranslate"><span class="pre">prefix</span></code> argument.</p>
</section>
</section>
<section id="python-3-9">
<h2>Python 3.9<a class="headerlink" href="#python-3-9" title="Permalink to this heading">¶</a></h2>
<p>See also: <a class="reference external" href="https://docs.python.org/3/whatsnew/3.9.html">what’s new in Python 3.9</a></p>
<section id="base64-encodestring-base64-decodestring-removal">
<h3>base64.encodestring / base64.decodestring removal<a class="headerlink" href="#base64-encodestring-base64-decodestring-removal" title="Permalink to this heading">¶</a></h3>
<p>Python 3.9 removes the deprecated <code class="docutils literal notranslate"><span class="pre">base64.encodestring()</span></code>
and <code class="docutils literal notranslate"><span class="pre">base64.decodestring()</span></code> functions. While they were deprecated
since Python 3.1, many packages still use them today.</p>
<p>The drop-in Python 3.1+ replacements are <code class="docutils literal notranslate"><span class="pre">base64.encodebytes()</span></code>
and <code class="docutils literal notranslate"><span class="pre">base64.decodebytes()</span></code>. Note that contrary to the names, the old
functions were simply aliases to the byte variants in Python 3
and <em>required</em> the arguments to be <code class="docutils literal notranslate"><span class="pre">bytes</span></code> anyway.</p>
<p>If compatibility with Python 2 is still desired, then the byte variants
ought to be called on 3.1+ and string variants before that. The old
variants accept both byte and unicode strings on Python 2.</p>
<p>Example compatibility import:</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">sys</span>
<span class="k">if</span> <span class="n">sys</span><span class="o">.</span><span class="n">version_info</span> <span class="o">>=</span> <span class="p">(</span><span class="mi">3</span><span class="p">,</span> <span class="mi">1</span><span class="p">):</span>
<span class="kn">from</span> <span class="nn">base64</span> <span class="kn">import</span> <span class="n">encodebytes</span> <span class="k">as</span> <span class="n">b64_encodebytes</span>
<span class="k">else</span><span class="p">:</span>
<span class="kn">from</span> <span class="nn">base64</span> <span class="kn">import</span> <span class="n">encodestring</span> <span class="k">as</span> <span class="n">b64_encodebytes</span>
</pre></div>
</div>
<p>Note that the <code class="docutils literal notranslate"><span class="pre">base64</span></code> module also provides <code class="docutils literal notranslate"><span class="pre">b64encode()</span></code>
and <code class="docutils literal notranslate"><span class="pre">b64decode()</span></code> functions that were not renamed. <code class="docutils literal notranslate"><span class="pre">b64decode()</span></code>
can be used as a drop-in replacement for <code class="docutils literal notranslate"><span class="pre">decodebytes()</span></code>. However,
<code class="docutils literal notranslate"><span class="pre">b64encode()</span></code> does not insert newlines to split the output
like <code class="docutils literal notranslate"><span class="pre">encodebytes()</span></code> does, and instead returns a single line
of base64-encoded data for any length of output.</p>
</section>
</section>
<section id="python-3-8">
<h2>Python 3.8<a class="headerlink" href="#python-3-8" title="Permalink to this heading">¶</a></h2>
<p>See also: <a class="reference external" href="https://docs.python.org/3/whatsnew/3.8.html">what’s new in Python 3.8</a></p>
<section id="python-config-and-pkg-config-no-longer-list-python-library-by-default">
<h3>python-config and pkg-config no longer list Python library by default<a class="headerlink" href="#python-config-and-pkg-config-no-longer-list-python-library-by-default" title="Permalink to this heading">¶</a></h3>
<p>Until Python 3.7, the <code class="docutils literal notranslate"><span class="pre">python-X.Y</span></code> pkg-config file and python-config
tool listed the Python library. Starting with 3.8, this is no longer
the case. If you are building Python extensions, this is fine (they
are not supposed to link directly to libpython).</p>
<p>If you are building programs that need to embed the Python interpreter,
new <code class="docutils literal notranslate"><span class="pre">python-X.Y-embed</span></code> pkg-config file and <code class="docutils literal notranslate"><span class="pre">--embed</span></code> parameter
are provided for the purpose.</p>
<div class="highlight-console notranslate"><div class="highlight"><pre><span></span><span class="gp">$ </span>pkg-config --libs python-3.7
<span class="go">-lpython3.7m</span>
<span class="gp">$ </span>pkg-config --libs python-3.8
<span class="gp">$ </span>pkg-config --libs python-3.8-embed
<span class="go">-lpython3.8</span>
</pre></div>
</div>
<p>To achieve backwards compatibility, you should query
<code class="docutils literal notranslate"><span class="pre">python-X.Y-embed</span></code> first and fall back to <code class="docutils literal notranslate"><span class="pre">python-X.Y</span></code>.</p>
</section>
</section>
<section id="replacing-the-toml-package">
<h2>Replacing the toml package<a class="headerlink" href="#replacing-the-toml-package" title="Permalink to this heading">¶</a></h2>
<p>The old <a class="reference external" href="https://pypi.org/project/toml/">toml</a> package is no longer maintained. It was last released
in November 2020 and it was never updated to implement TOML 1.0.
The recommended alternatives are:</p>
<ul class="simple">
<li><p>the built-in <a class="reference external" href="https://docs.python.org/3.11/library/tomllib.html">tomllib</a> module (since Python 3.11) with fallback to
<a class="reference external" href="https://pypi.org/project/tomli/">tomli</a> package for reading TOML files</p></li>
<li><p>the <a class="reference external" href="https://pypi.org/project/tomli-w/">tomli-w</a> package for writing TOML files</p></li>
<li><p>the <a class="reference external" href="https://pypi.org/project/tomlkit/">tomlkit</a> package for editing already existing TOML files
while preserving style</p></li>
</ul>
<section id="porting-to-tomllib-tomli-without-toml-fallback">
<h3>Porting to tomllib/tomli without toml fallback<a class="headerlink" href="#porting-to-tomllib-tomli-without-toml-fallback" title="Permalink to this heading">¶</a></h3>
<p>Using a combination of <a class="reference external" href="https://docs.python.org/3.11/library/tomllib.html">tomllib</a> and <a class="reference external" href="https://pypi.org/project/tomli/">tomli</a> is the recommended approach
for packages that only read TOML files, or both read and write them
but do not need to preserve style. The tomllib module is available
since Python 3.11, while tomli versions providing a compatible API
are compatible with Python 3.6 and newer.</p>
<p>The key differences between <a class="reference external" href="https://pypi.org/project/toml/">toml</a> and tomllib/tomli are:</p>
<ul class="simple">
<li><p>the <code class="docutils literal notranslate"><span class="pre">load()</span></code> function accepts only a file object open for reading
in binary mode whereas toml expects a path or a file object open
for reading in text mode</p></li>
<li><p>the exception raised for invalid input is named <code class="docutils literal notranslate"><span class="pre">TOMLDecodeError</span></code>
where it is named <code class="docutils literal notranslate"><span class="pre">TomlDecodeError</span></code> in toml</p></li>
</ul>
<p>For example, the following code:</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">toml</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">d1</span> <span class="o">=</span> <span class="n">toml</span><span class="o">.</span><span class="n">load</span><span class="p">(</span><span class="s2">"in1.toml"</span><span class="p">)</span>
<span class="k">except</span> <span class="n">toml</span><span class="o">.</span><span class="n">TomlDecodeError</span><span class="p">:</span>
<span class="n">d1</span> <span class="o">=</span> <span class="kc">None</span>
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="s2">"in2.toml"</span><span class="p">,</span> <span class="s2">"r"</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
<span class="n">d2</span> <span class="o">=</span> <span class="n">toml</span><span class="o">.</span><span class="n">load</span><span class="p">(</span><span class="n">f</span><span class="p">)</span>
<span class="n">d3</span> <span class="o">=</span> <span class="n">toml</span><span class="o">.</span><span class="n">loads</span><span class="p">(</span><span class="s1">'test = "foo"</span><span class="se">\n</span><span class="s1">'</span><span class="p">)</span>
</pre></div>
</div>
<p>would normally be written as:</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">sys</span>
<span class="k">if</span> <span class="n">sys</span><span class="o">.</span><span class="n">version_info</span> <span class="o">>=</span> <span class="p">(</span><span class="mi">3</span><span class="p">,</span> <span class="mi">11</span><span class="p">):</span>
<span class="kn">import</span> <span class="nn">tomllib</span>
<span class="k">else</span><span class="p">:</span>
<span class="kn">import</span> <span class="nn">tomli</span> <span class="k">as</span> <span class="nn">tomllib</span>
<span class="k">try</span><span class="p">:</span>
<span class="c1"># tomllib does not accept paths</span>
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="s2">"in1.toml"</span><span class="p">,</span> <span class="s2">"rb"</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
<span class="n">d1</span> <span class="o">=</span> <span class="n">tomllib</span><span class="o">.</span><span class="n">load</span><span class="p">(</span><span class="n">f</span><span class="p">)</span>
<span class="c1"># the exception uses uppercase "TOML"</span>
<span class="k">except</span> <span class="n">tomllib</span><span class="o">.</span><span class="n">TOMLDecodeError</span><span class="p">:</span>
<span class="n">d1</span> <span class="o">=</span> <span class="kc">None</span>
<span class="c1"># the file must be open in binary mode</span>
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="s2">"in2.toml"</span><span class="p">,</span> <span class="s2">"rb"</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
<span class="n">d2</span> <span class="o">=</span> <span class="n">tomllib</span><span class="o">.</span><span class="n">load</span><span class="p">(</span><span class="n">f</span><span class="p">)</span>
<span class="n">d3</span> <span class="o">=</span> <span class="n">tomllib</span><span class="o">.</span><span class="n">loads</span><span class="p">(</span><span class="s1">'test = "foo"</span><span class="se">\n</span><span class="s1">'</span><span class="p">)</span>
</pre></div>
</div>
<p>The following dependency string:</p>
<div class="highlight-toml notranslate"><div class="highlight"><pre><span></span><span class="n">dependencies</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">[</span><span class="w"></span>
<span class="w"> </span><span class="s">"toml"</span><span class="p">,</span><span class="w"></span>
<span class="p">]</span><span class="w"></span>
</pre></div>
</div>
<p>would be replaced by:</p>
<div class="highlight-toml notranslate"><div class="highlight"><pre><span></span><span class="n">dependencies</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">[</span><span class="w"></span>
<span class="w"> </span><span class="s">"tomli >= 1.2.3; python_version < '3.11'"</span><span class="p">,</span><span class="w"></span>
<span class="p">]</span><span class="w"></span>
</pre></div>
</div>
</section>
<section id="porting-to-tomllib-tomli-with-toml-fallback">
<h3>Porting to tomllib/tomli with toml fallback<a class="headerlink" href="#porting-to-tomllib-tomli-with-toml-fallback" title="Permalink to this heading">¶</a></h3>
<p>If upstream insists on preserving compatibility with EOL versions
of Python, it is possible to use a combination of <a class="reference external" href="https://docs.python.org/3.11/library/tomllib.html">tomllib</a>, <a class="reference external" href="https://pypi.org/project/tomli/">tomli</a>
and <a class="reference external" href="https://pypi.org/project/toml/">toml</a>. Unfortunately, the incompatibilites in API need to be taken
into consideration.</p>
<p>For example, a backwards compatible code for loading a TOML file could
look like the following:</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">sys</span>
<span class="k">try</span><span class="p">:</span>
<span class="k">if</span> <span class="n">sys</span><span class="o">.</span><span class="n">version_info</span> <span class="o">>=</span> <span class="p">(</span><span class="mi">3</span><span class="p">,</span> <span class="mi">11</span><span class="p">):</span>
<span class="kn">import</span> <span class="nn">tomllib</span>
<span class="k">else</span><span class="p">:</span>
<span class="kn">import</span> <span class="nn">tomli</span> <span class="k">as</span> <span class="nn">tomllib</span>
<span class="k">try</span><span class="p">:</span>
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="s2">"in1.toml"</span><span class="p">,</span> <span class="s2">"rb"</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
<span class="n">d1</span> <span class="o">=</span> <span class="n">tomllib</span><span class="o">.</span><span class="n">load</span><span class="p">(</span><span class="n">f</span><span class="p">)</span>
<span class="k">except</span> <span class="n">tomllib</span><span class="o">.</span><span class="n">TOMLDecodeError</span><span class="p">:</span>
<span class="n">d1</span> <span class="o">=</span> <span class="kc">None</span>
<span class="k">except</span> <span class="ne">ImportError</span><span class="p">:</span>
<span class="kn">import</span> <span class="nn">toml</span>
<span class="k">try</span><span class="p">:</span>
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="s2">"in1.toml"</span><span class="p">,</span> <span class="s2">"r"</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
<span class="n">d1</span> <span class="o">=</span> <span class="n">toml</span><span class="o">.</span><span class="n">load</span><span class="p">(</span><span class="n">f</span><span class="p">)</span>
<span class="k">except</span> <span class="n">toml</span><span class="o">.</span><span class="n">TomlDecodeError</span><span class="p">:</span>
<span class="n">d1</span> <span class="o">=</span> <span class="kc">None</span>
</pre></div>
</div>
<p>In this case, the dependency string becomes more complex:</p>
<div class="highlight-toml notranslate"><div class="highlight"><pre><span></span><span class="n">dependencies</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">[</span><span class="w"></span>
<span class="w"> </span><span class="s">"tomli >= 1.2.3; python_version >= '3.6' and python_version < '3.11'"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="s">"toml; python_version < '3.6'"</span><span class="p">,</span><span class="w"></span>
<span class="p">]</span><span class="w"></span>
</pre></div>
</div>
</section>
<section id="porting-to-tomli-w">
<h3>Porting to tomli-w<a class="headerlink" href="#porting-to-tomli-w" title="Permalink to this heading">¶</a></h3>
<p><a class="reference external" href="https://pypi.org/project/tomli-w/">tomli-w</a> provides a minimal module for dumping TOML files.</p>
<p>The key differences between <a class="reference external" href="https://pypi.org/project/toml/">toml</a> and tomli-w are:</p>
<ul class="simple">
<li><p>the <code class="docutils literal notranslate"><span class="pre">dump()</span></code> function takes a file object open for writing in binary
mode whereas toml expected a file object open for writing in text mode</p></li>
<li><p>providing a custom encoder instance is not supported</p></li>
</ul>
<p>For example, the following code:</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">toml</span>
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="s2">"out.toml"</span><span class="p">,</span> <span class="s2">"w"</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
<span class="n">toml</span><span class="o">.</span><span class="n">dump</span><span class="p">({</span><span class="s2">"test"</span><span class="p">:</span> <span class="s2">"data"</span><span class="p">},</span> <span class="n">f</span><span class="p">)</span>
</pre></div>
</div>
<p>would be replaced by:</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">tomli_w</span>
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="s2">"out.toml"</span><span class="p">,</span> <span class="s2">"wb"</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
<span class="n">tomli_w</span><span class="o">.</span><span class="n">dump</span><span class="p">({</span><span class="s2">"test"</span><span class="p">:</span> <span class="s2">"data"</span><span class="p">},</span> <span class="n">f</span><span class="p">)</span>
</pre></div>
</div>
<p>Note that when both reading and writing TOML files is necessary, two
modules need to be imported and used separately rather than one.</p>
</section>
</section>
</section>
</div>
</div>
</div>
<div class="sphinxsidebar" role="navigation" aria-label="main navigation">
<div class="sphinxsidebarwrapper">
<h1 class="logo"><a href="index.html">Gentoo Python Guide</a></h1>
<h3>Navigation</h3>
<p class="caption" role="heading"><span class="caption-text">Contents:</span></p>
<ul class="current">
<li class="toctree-l1"><a class="reference internal" href="preface.html">Preface</a></li>
<li class="toctree-l1"><a class="reference internal" href="interpreter.html">Python interpreters</a></li>
<li class="toctree-l1"><a class="reference internal" href="eclass.html">Choosing between Python eclasses</a></li>
<li class="toctree-l1"><a class="reference internal" href="basic.html">Common basics</a></li>
<li class="toctree-l1"><a class="reference internal" href="any.html">python-any-r1 — build-time dependency</a></li>
<li class="toctree-l1"><a class="reference internal" href="single.html">python-single-r1 — single-impl packages</a></li>
<li class="toctree-l1"><a class="reference internal" href="multi.html">python-r1 — multi-impl packages</a></li>
<li class="toctree-l1"><a class="reference internal" href="distutils.html">distutils-r1 — standard Python build systems</a></li>
<li class="toctree-l1"><a class="reference internal" href="distutils-legacy.html">distutils-r1 legacy concepts</a></li>
<li class="toctree-l1"><a class="reference internal" href="helper.html">Common helper functions</a></li>
<li class="toctree-l1"><a class="reference internal" href="depend.html">Advanced dependencies</a></li>
<li class="toctree-l1"><a class="reference internal" href="test.html">Resolving test suite problems</a></li>
<li class="toctree-l1"><a class="reference internal" href="pytest.html">pytest recipes</a></li>
<li class="toctree-l1"><a class="reference internal" href="concept.html">Advanced concepts</a></li>
<li class="toctree-l1"><a class="reference internal" href="expert-multi.html">Expert python-r1 usage</a></li>
<li class="toctree-l1"><a class="reference internal" href="buildsys.html">Integration with build systems written in Python</a></li>
<li class="toctree-l1 current"><a class="current reference internal" href="#">Porting tips</a><ul>
<li class="toctree-l2"><a class="reference internal" href="#retroactive-changes">Retroactive changes</a></li>
<li class="toctree-l2"><a class="reference internal" href="#python-3-11">Python 3.11</a></li>
<li class="toctree-l2"><a class="reference internal" href="#python-3-10">Python 3.10</a></li>
<li class="toctree-l2"><a class="reference internal" href="#python-3-9">Python 3.9</a></li>
<li class="toctree-l2"><a class="reference internal" href="#python-3-8">Python 3.8</a></li>
<li class="toctree-l2"><a class="reference internal" href="#replacing-the-toml-package">Replacing the toml package</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="migration.html">Migration guides</a></li>
<li class="toctree-l1"><a class="reference internal" href="qawarn.html">QA checks and warnings</a></li>
<li class="toctree-l1"><a class="reference internal" href="package-maintenance.html">Python package maintenance</a></li>
<li class="toctree-l1"><a class="reference internal" href="interpreter-maintenance.html">Maintenance of Python implementations</a></li>
</ul>
<div class="relations">
<h3>Related Topics</h3>
<ul>
<li><a href="index.html">Documentation overview</a><ul>
<li>Previous: <a href="buildsys.html" title="previous chapter">Integration with build systems written in Python</a></li>
<li>Next: <a href="migration.html" title="next chapter">Migration guides</a></li>
</ul></li>
</ul>
</div>
<div id="searchbox" style="display: none" role="search">
<h3 id="searchlabel">Quick search</h3>
<div class="searchformwrapper">
<form class="search" action="search.html" method="get">
<input type="text" name="q" aria-labelledby="searchlabel" autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false"/>
<input type="submit" value="Go" />
</form>
</div>
</div>
<script>document.getElementById('searchbox').style.display = "block"</script>
</div>
</div>
<div class="clearer"></div>
</div>
<div class="footer">
©2020, Michał Górny, license: CC BY 4.0.
|
Powered by <a href="http://sphinx-doc.org/">Sphinx 5.3.0</a>
& <a href="https://github.com/bitprophet/alabaster">Alabaster 0.7.12</a>
|
<a href="_sources/porting.rst.txt"
rel="nofollow">Page source</a>
</div>
</body>
</html>
|
GitWeb