Merge lp:~barry/python/pep3147 into lp:python/py3k

Proposed by Barry Warsaw on 2010-04-01
Status: Merged
Merge reported by: Barry Warsaw
Merged at revision: not available
Proposed branch: lp:~barry/python/pep3147
Merge into: lp:python/py3k
Diff against target: 2808 lines (+1209/-293) 39 files modified
To merge this branch: bzr merge lp:~barry/python/pep3147
Reviewer Review Type Date Requested Status
Python Development 2010-04-01 Pending
Review via email: mp+22648@code.launchpad.net

Description of the Change

Work-in-progress implementation of PEP 3147

http://www.python.org/dev/peps/pep-3147/

To post a comment you must log in.
lp:~barry/python/pep3147 updated on 2010-04-16
41150. By raymond.hettinger on 2010-04-01

Document sorting HowTo

41151. By raymond.hettinger on 2010-04-02

Add and update itertools recipes.

41152. By raymond.hettinger on 2010-04-02

Fix nits in itertools recipes.

41153. By georg.brandl on 2010-04-02

#8213: document behavior of -u on py3k better.

41154. By larry.hastings on 2010-04-02

Blocked revisions 79590 via svnmerge

........
  r79590 | larry.hastings | 2010-04-02 04:01:35 -0700 (Fri, 02 Apr 2010) | 10 lines

  Capsule-related changes:
  * PyCObject_AsVoidPtr() can now open capsules. This addresses
    most of the remaining backwards-compatibility concerns about
    the conversion of Python 2.7 from CObjects to capsules.
  * CObjects were marked Pending Deprecation.
  * Documentation about this pending deprecation was added to
    cobject.h.
  * The capsule source files were added to the legacy PC build
    processes.
........

41155. By larry.hastings on 2010-04-02

Blocked revisions 79592 via svnmerge

........
  r79592 | larry.hastings | 2010-04-02 04:18:17 -0700 (Fri, 02 Apr 2010) | 3 lines

  Issue #8235: _socket: Add the constant ``SO_SETFIB``. SO_SETFIB is
  a socket option available on FreeBSD 7.1 and newer.
........

41156. By larry.hastings on 2010-04-02

- Issue #8235: _socket: Add the constant ``SO_SETFIB``. SO_SETFIB is
  a socket option available on FreeBSD 7.1 and newer.

41157. By ezio.melotti on 2010-04-02

Merged revisions 79540-79541 via svnmerge from
svn+ssh://<email address hidden>/python/trunk

Add -Wd to the flags used to run the tests (-3 is not necessary here).

........
  r79540 | ezio.melotti | 2010-04-01 01:18:09 +0300 (Thu, 01 Apr 2010) | 1 line

  Add -Wd and -3 to the flags used to run the tests.
........
  r79541 | ezio.melotti | 2010-04-01 01:43:31 +0300 (Thu, 01 Apr 2010) | 1 line

  Add -Wd and -3 to the flags used to run the tests on Windows.
........

41158. By raymond.hettinger on 2010-04-02

Issue 8257: Decimal constructor to accept float argument.

41159. By andrew.kuchling on 2010-04-02

Merged revisions 79603 via svnmerge from
svn+ssh://<email address hidden>/python/trunk

........
  r79603 | andrew.kuchling | 2010-04-02 12:59:16 -0400 (Fri, 02 Apr 2010) | 1 line

  #4440: modernize a use of filter(), making it compatible with 3.x
........

41160. By antoine.pitrou on 2010-04-02

Furniture is not very reliable these days (buildbot failures).

41161. By florent.xicluna on 2010-04-02

(partially)

Merged revisions 79534,79537,79539,79558,79606 via svnmerge from
svn+ssh://<email address hidden>/python/trunk

........
  r79534 | florent.xicluna | 2010-03-31 23:21:54 +0200 (mer, 31 mar 2010) | 2 lines

  Fix test for xml.etree when using a non-ascii path. And use check_warnings instead of catch_warnings.
........
  r79537 | florent.xicluna | 2010-03-31 23:40:32 +0200 (mer, 31 mar 2010) | 2 lines

  Fix typo
........
  r79539 | florent.xicluna | 2010-04-01 00:01:03 +0200 (jeu, 01 avr 2010) | 2 lines

  Replace catch_warnings with check_warnings when it makes sense. Use assertRaises context manager to simplify some tests.
........
  r79558 | florent.xicluna | 2010-04-01 20:17:09 +0200 (jeu, 01 avr 2010) | 2 lines

  #7092: Fix some -3 warnings, and fix Lib/platform.py when the path contains a double-quote.
........
  r79606 | florent.xicluna | 2010-04-02 19:26:42 +0200 (ven, 02 avr 2010) | 2 lines

  Backport some robotparser test and skip the test if the external resource is not available.
........

41162. By raymond.hettinger on 2010-04-02

Add nice docstrings to namedtuples.

41163. By florent.xicluna on 2010-04-02

An oversight in previous merge, r79610: test_doctest is sensible to the line numbering.

41164. By georg.brandl on 2010-04-02

#8288: fix next -> __next__.

41165. By tarek.ziade on 2010-04-02

Merged revisions 79618 via svnmerge from
svn+ssh://<email address hidden>/python/trunk

........
  r79618 | tarek.ziade | 2010-04-02 23:14:04 +0200 (Fri, 02 Apr 2010) | 1 line

  removed the local copy of xxmodule, and skip only test_build_ext when xxmodule is not found, not the whole unittest
........

41166. By ezio.melotti on 2010-04-02

Blocked revisions 79582 via svnmerge

........
  r79582 | georg.brandl | 2010-04-02 11:51:31 +0300 (Fri, 02 Apr 2010) | 1 line

  Fix typo in unicode character name.
........

41167. By brian.curtin on 2010-04-02

Change test_support to support. Fixes a failing test on Windows.

41168. By benjamin.peterson on 2010-04-03

Blocked revisions 79636,79639 via svnmerge

........
  r79636 | benjamin.peterson | 2010-04-02 18:59:41 -0500 (Fri, 02 Apr 2010) | 4 lines

  always check _PyString_Resize for error

  also normalize how this error is checked
........
  r79639 | benjamin.peterson | 2010-04-02 19:57:33 -0500 (Fri, 02 Apr 2010) | 1 line

  more _PyString_Resize error checking
........

41169. By benjamin.peterson on 2010-04-03

Merged revisions 79642,79644 via svnmerge from
svn+ssh://<email address hidden>/python/trunk

........
  r79642 | benjamin.peterson | 2010-04-02 20:08:34 -0500 (Fri, 02 Apr 2010) | 1 line

  split out large test function
........
  r79644 | benjamin.peterson | 2010-04-02 20:28:57 -0500 (Fri, 02 Apr 2010) | 1 line

  give TypeError when trying to set T_STRING_INPLACE
........

41170. By victor.stinner on 2010-04-03

Blocked revisions 79654 via svnmerge

........
  r79654 | victor.stinner | 2010-04-03 10:40:16 +0200 (sam., 03 avril 2010) | 5 lines

  Issue #8227: Fix C API documentation, argument parsing

   * 'z', 'z#', 'z*' does also accept Unicode
   * unify types name: replace "string or Unicode objet" by "string or Unicode"
........

41171. By tarek.ziade on 2010-04-03

Merged revisions 79652 via svnmerge from
svn+ssh://<email address hidden>/python/trunk

........
  r79652 | tarek.ziade | 2010-04-03 10:37:59 +0200 (Sat, 03 Apr 2010) | 1 line

  Issue #1222585: Added LDCXXSHARED for C++ support
........

41172. By raymond.hettinger on 2010-04-03

Add a subtract() method to collections.Counter()

41173. By mark.dickinson on 2010-04-03

Merged revisions 79661 via svnmerge from
svn+ssh://<email address hidden>/python/trunk

........
  r79661 | mark.dickinson | 2010-04-03 11:27:05 +0100 (Sat, 03 Apr 2010) | 14 lines

  Fix a couple of issues with the test_structmembersType class in _testcapimodule

   - rename to _test_structmembersType to avoid the class being automatically
     called by test_capi

   - allow space for trailing NUL in inplace_member field of all_structmembers

   - use T_STRING_INPLACE instead of T_INPLACE_STRING as keyword argument
     to _test_structmembersType initializer

   - don't attempt to initialize inplace_member field if T_STRING_INPLACE
     argument wasn't supplied.
........

41174. By mark.dickinson on 2010-04-03

Merged revisions 79583,79588-79589 via svnmerge from
svn+ssh://<email address hidden>/python/trunk

........
  r79583 | mark.dickinson | 2010-04-02 09:53:22 +0100 (Fri, 02 Apr 2010) | 7 lines

  Issue #2531: Make float-to-decimal comparisons return correct results.

  Float to decimal comparison operations now return a result based on
  the numeric values of the operands. Decimal.__hash__ has also been
  fixed so that Decimal and float values that compare equal have equal
  hash value.
........
  r79588 | mark.dickinson | 2010-04-02 11:17:07 +0100 (Fri, 02 Apr 2010) | 2 lines

  Issue #7279: Make comparisons involving a Decimal sNaN signal InvalidOperation.
........
  r79589 | mark.dickinson | 2010-04-02 11:35:12 +0100 (Fri, 02 Apr 2010) | 6 lines

  Issue #7279: Make Decimal('nan') hashable. Decimal('snan') remains unhashable.

  Also rewrite the Decimal __hash__ method so that it doesn't rely on
  float('inf') being valid: float('inf') could raise an exception on
  platforms not using IEEE 754 arithmetic.
........

41175. By mark.dickinson on 2010-04-03

Merged revisions 79629 via svnmerge from
svn+ssh://<email address hidden>/python/trunk

........
  r79629 | mark.dickinson | 2010-04-02 23:27:36 +0100 (Fri, 02 Apr 2010) | 2 lines

  Issue #8294: Allow float and Decimal arguments in Fraction constructor.
........

41176. By mark.dickinson on 2010-04-03

Remove unused define.

41177. By ezio.melotti on 2010-04-03

Merged revisions 79678 via svnmerge from
svn+ssh://<email address hidden>/python/trunk

........
  r79678 | ezio.melotti | 2010-04-03 17:51:00 +0300 (Sat, 03 Apr 2010) | 1 line

  Use more specific assert* methods in test_decimal.
........

41178. By mark.dickinson on 2010-04-03

Clean up integer tests in test_struct, and bring them more in line with the trunk tests.

41179. By mark.dickinson on 2010-04-03

Internal refactoring in struct.pack: make all integer conversions go through get_pylong.

41180. By ezio.melotti on 2010-04-03

Add -bb to rt.bat to do some extra checks during the tests (it is already used by the non-Windows buildbots).

41181. By ezio.melotti on 2010-04-03

Blocked revisions 79624 via svnmerge

........
  r79624 | ezio.melotti | 2010-04-03 00:43:10 +0300 (Sat, 03 Apr 2010) | 1 line

  Fix test_compiler.py that was using unittest.__file__ to find Lib/ (unittest is now a package).
........

41182. By benjamin.peterson on 2010-04-03

Blocked revisions 79693,79695 via svnmerge

........
  r79693 | benjamin.peterson | 2010-04-03 10:38:38 -0500 (Sat, 03 Apr 2010) | 1 line

  wrap
........
  r79695 | benjamin.peterson | 2010-04-03 10:40:29 -0500 (Sat, 03 Apr 2010) | 1 line

  remove unneeded argument
........

41183. By mark.dickinson on 2010-04-03

Merged revisions 79699 via svnmerge from
svn+ssh://<email address hidden>/python/trunk

........
  r79699 | mark.dickinson | 2010-04-03 16:47:46 +0100 (Sat, 03 Apr 2010) | 1 line

  Add Misc/NEWS entry for r79609.
........

41184. By mark.dickinson on 2010-04-03

Merged revisions 79674 via svnmerge from
svn+ssh://<email address hidden>/python/trunk

........
  r79674 | mark.dickinson | 2010-04-03 15:05:10 +0100 (Sat, 03 Apr 2010) | 3 lines

  Issue #8300: Let struct.pack use __index__ to convert and pack non-integers.
  Based on a patch by Meador Inge.
........

41185. By benjamin.peterson on 2010-04-03

Blocked revisions 79697-79698,79704 via svnmerge

........
  r79697 | benjamin.peterson | 2010-04-03 10:44:56 -0500 (Sat, 03 Apr 2010) | 1 line

  silence PyCObject warnings in bsddb
........
  r79698 | benjamin.peterson | 2010-04-03 10:45:59 -0500 (Sat, 03 Apr 2010) | 1 line

  spelling
........
  r79704 | benjamin.peterson | 2010-04-03 10:58:15 -0500 (Sat, 03 Apr 2010) | 1 line

  remove deprecation warnings silence attempting
........

41186. By mark.dickinson on 2010-04-03

Blocked revisions 79707 via svnmerge

........
  r79707 | mark.dickinson | 2010-04-03 17:41:20 +0100 (Sat, 03 Apr 2010) | 5 lines

  Ensure 'module removed' warning messages contain the word 'module' or 'package'.

  This should fix the test_py3kwarn failure on OS X. test_support.import_module
  also requires this.
........

41187. By mark.dickinson on 2010-04-03

Blocked revisions 79714 via svnmerge

........
  r79714 | mark.dickinson | 2010-04-03 19:17:54 +0100 (Sat, 03 Apr 2010) | 1 line

  Silence DeprecationWarnings from uses of has_key and <> in plat-mac.
........

41188. By benjamin.peterson on 2010-04-03

Blocked revisions 79712 via svnmerge

........
  r79712 | raymond.hettinger | 2010-04-03 12:10:05 -0500 (Sat, 03 Apr 2010) | 1 line

  Silence a compiler warning.
........

41189. By benjamin.peterson on 2010-04-03

Blocked revisions 79718-79719 via svnmerge

........
  r79718 | antoine.pitrou | 2010-04-03 16:42:18 -0500 (Sat, 03 Apr 2010) | 3 lines

  Remove useless (?) import from r79706
........
  r79719 | benjamin.peterson | 2010-04-03 16:50:40 -0500 (Sat, 03 Apr 2010) | 1 line

  import bsddb more robustly
........

41190. By benjamin.peterson on 2010-04-03

Merged revisions 79723 via svnmerge from
svn+ssh://<email address hidden>/python/trunk

........
  r79723 | benjamin.peterson | 2010-04-03 17:48:51 -0500 (Sat, 03 Apr 2010) | 1 line

  ensure that the locale does not affect the tokenization of identifiers
........

41191. By benjamin.peterson on 2010-04-03

Merged revisions 79725 via svnmerge from
svn+ssh://<email address hidden>/python/trunk

........
  r79725 | benjamin.peterson | 2010-04-03 18:03:35 -0500 (Sat, 03 Apr 2010) | 4 lines

  use our own locale independent ctype macros

  requires building pyctype.o into pgen
........

41192. By raymond.hettinger on 2010-04-03

Add count() method to collections.deque().

41193. By benjamin.peterson on 2010-04-03

add missing decrefs

41194. By ezio.melotti on 2010-04-04

Fix versionchanged from 2.7 to 3.2 in decimal.rst.

41195. By ezio.melotti on 2010-04-04

Merged revisions 79733,79735 via svnmerge from
svn+ssh://<email address hidden>/python/trunk

........
  r79733 | ezio.melotti | 2010-04-04 09:41:27 +0300 (Sun, 04 Apr 2010) | 1 line

  Add a versionadded to Counter.subtract() doc.
........
  r79735 | ezio.melotti | 2010-04-04 09:50:58 +0300 (Sun, 04 Apr 2010) | 1 line

  Revert 79733 - Counter is new in 2.7, the versionadded is not necessary - and fix a typo.
........

41196. By ezio.melotti on 2010-04-04

Recorded merge of revisions 79660 via svnmerge from
svn+ssh://<email address hidden>/python/trunk

........
  r79660 | raymond.hettinger | 2010-04-03 13:22:00 +0300 (Sat, 03 Apr 2010) | 1 line

  Add subtract() method to collections.Counter() objects.
........

41197. By ezio.melotti on 2010-04-04

Merged revisions 79740 via svnmerge from
svn+ssh://<email address hidden>/python/trunk

........
  r79740 | ezio.melotti | 2010-04-04 10:00:02 +0300 (Sun, 04 Apr 2010) | 1 line

  Use more specific assert* methods in test_struct.
........

41198. By mark.dickinson on 2010-04-04

Merged revisions 79745 via svnmerge from
svn+ssh://<email address hidden>/python/trunk

........
  r79745 | mark.dickinson | 2010-04-04 09:43:04 +0100 (Sun, 04 Apr 2010) | 3 lines

  Issue #8300 (__index__ handling in struct.pack): Remove redundant check
  and improve test coverage. Thanks Meador Inge for the patch.
........

41199. By mark.dickinson on 2010-04-04

Merged revisions 79751 via svnmerge from
svn+ssh://<email address hidden>/python/trunk

........
  r79751 | mark.dickinson | 2010-04-04 22:19:35 +0100 (Sun, 04 Apr 2010) | 1 line

  A handful of whitespace fixes in Modules/_struct.c.
........

41200. By mark.dickinson on 2010-04-04

Merged revisions 79756 via svnmerge from
svn+ssh://<email address hidden>/python/trunk

........
  r79756 | mark.dickinson | 2010-04-04 23:09:21 +0100 (Sun, 04 Apr 2010) | 6 lines

  Add versionchanged entry for Decimal(float) construction.

  Also add an example of constructing a Decimal directly from a float,
  update the quickstart tutorial, and tweak another couple of
  sentences.
........

41201. By benjamin.peterson on 2010-04-04

Merged revisions 79762 via svnmerge from
svn+ssh://<email address hidden>/python/trunk

........
  r79762 | benjamin.peterson | 2010-04-04 18:01:03 -0500 (Sun, 04 Apr 2010) | 1 line

  tabify
........

41202. By benjamin.peterson on 2010-04-04

Merged revisions 79763 via svnmerge from
svn+ssh://<email address hidden>/python/trunk

........
  r79763 | benjamin.peterson | 2010-04-04 18:03:22 -0500 (Sun, 04 Apr 2010) | 1 line

  fix tabs
........

41203. By benjamin.peterson on 2010-04-04

Blocked revisions 79769 via svnmerge

........
  r79769 | benjamin.peterson | 2010-04-04 18:23:22 -0500 (Sun, 04 Apr 2010) | 1 line

  fix dis on new style classes #8310
........

41204. By benjamin.peterson on 2010-04-04

factor out constant

41205. By ezio.melotti on 2010-04-04

Merged revisions 79768 via svnmerge from
svn+ssh://<email address hidden>/python/trunk

........
  r79768 | ezio.melotti | 2010-04-05 02:21:53 +0300 (Mon, 05 Apr 2010) | 1 line

  Fix markup in decimal.rst.
........

41206. By philip.jenvey on 2010-04-05

Merged revisions 79779 via svnmerge from
svn+ssh://<email address hidden>/python/trunk

........
  r79779 | philip.jenvey | 2010-04-04 19:51:51 -0700 (Sun, 04 Apr 2010) | 2 lines

  fix escape_encode to return the correct consumed size
........

41207. By ezio.melotti on 2010-04-05

Merged revisions 79781 via svnmerge from
svn+ssh://<email address hidden>/python/trunk

........
  r79781 | ezio.melotti | 2010-04-05 06:51:38 +0300 (Mon, 05 Apr 2010) | 1 line

  #8212: rephrase the webservers howto and fix some mistakes.
........

41208. By ezio.melotti on 2010-04-05

Merged revisions 79788 via svnmerge from
svn+ssh://<email address hidden>/python/trunk

........
  r79788 | ezio.melotti | 2010-04-05 11:16:41 +0300 (Mon, 05 Apr 2010) | 1 line

  Fix markup errors.
........

41209. By Barry Warsaw on 2010-04-05

branch merge

41210. By Barry Warsaw on 2010-04-05

Ensure that no .pyc file outside of __pycache__ is written by the test suite.

41211. By Barry Warsaw on 2010-04-06

merge trunk

41212. By Barry Warsaw on 2010-04-06

Many updates based on Rietveld review by Antoine Pitrou.

41213. By Barry Warsaw on 2010-04-06

trunk merge

41214. By Barry Warsaw on 2010-04-07

trunk merge

41215. By Barry Warsaw on 2010-04-08

trunk merge

41216. By Barry Warsaw on 2010-04-08

Second round of Rietveld comments.

* Added documentation for cache_from_source(), source_from_cache(), and
  magic_tags.
* Use addCleanup() where appropriate.
* Other style cleanup.

41217. By Barry Warsaw on 2010-04-08

Test repairs.

41218. By Barry Warsaw on 2010-04-08

trunk merge

41219. By Barry Warsaw on 2010-04-09

Fixes for Windows compatibility.

41220. By Barry Warsaw on 2010-04-09

Be more aggressive for Windows.

41221. By Barry Warsaw on 2010-04-09

trunk merge

41222. By Barry Warsaw on 2010-04-09

Benjamin makes a good point that debug_override should accept true-ish and
false-ish values (when not None) to more closely mirror a pure-Python
implementation.

41223. By Barry Warsaw on 2010-04-09

* Add __cached__ attribute to modules.
* PyImport_ExecCodeModuleEx() gets a new parameter.

41224. By Barry Warsaw on 2010-04-09

Remove some unnecessary code.

41225. By Barry Warsaw on 2010-04-09

Based on Guido's recommendation, add PyImport_ExecCodeModuleExEx()

41226. By Barry Warsaw on 2010-04-11

trunk merge

41227. By Barry Warsaw on 2010-04-12

* PyImport_ExecCodeModuleExEx() -> PyImport_ExecCodeModuleWithPathnames()
* Document the above.
* Added PyImport_GetMagicTag() and document.
* Removed imp.magic_tags and replaced with imp.get_tag().

41228. By Barry Warsaw on 2010-04-12

More responses to Rietveld and mailing list comments. The most important
change here is an additional guard against buffer overflows in
make_compiled_pathname().

41229. By Barry Warsaw on 2010-04-12

trunk merge

41230. By Barry Warsaw on 2010-04-12

Patches from Brian Curtin for Windows.

41231. By Barry Warsaw on 2010-04-13

Support -b flag for compile to create legacy .pyc files. This does not delete
the source files.

41232. By Barry Warsaw on 2010-04-13

Document extended compileall options.

41233. By Barry Warsaw on 2010-04-13

trunk merge

41234. By Barry Warsaw on 2010-04-13

Make sure __cache__ shows up in modules executed via command line and -m
(runpy). Update runpy documentation.

41235. By Barry Warsaw on 2010-04-13

Typo.

41236. By Barry Warsaw on 2010-04-13

trunk merge

41237. By Barry Warsaw on 2010-04-15

* abs__file__() -> abs_paths() since this now also modifies m.__cached__
* actually make test_abs_path() not suck.

41238. By Barry Warsaw on 2010-04-15

trunk merge

41239. By Barry Warsaw on 2010-04-16

trunk merge

Preview Diff

1=== modified file '.bzrignore'
2--- .bzrignore 2010-04-11 20:49:28 +0000
3+++ .bzrignore 2010-04-16 18:55:41 +0000
4@@ -33,3 +33,4 @@
5 Lib/test/data/*
6 Lib/lib2to3/Grammar*.pickle
7 Lib/lib2to3/PatternGrammar*.pickle
8+__pycache__
9
10=== modified file '.hgignore'
11--- .hgignore 2010-04-13 16:37:01 +0000
12+++ .hgignore 2010-04-16 18:55:41 +0000
13@@ -54,3 +54,4 @@
14 PCbuild/*.ncb
15 PCbuild/*.bsc
16 PCbuild/Win32-temp-*
17+__pycache__
18
19=== modified file 'Doc/c-api/import.rst'
20--- Doc/c-api/import.rst 2010-04-11 15:12:57 +0000
21+++ Doc/c-api/import.rst 2010-04-16 18:55:41 +0000
22@@ -124,12 +124,24 @@
23 If *name* points to a dotted name of the form ``package.module``, any package
24 structures not already created will still not be created.
25
26+ See also :func:`PyImport_ExecCodeModuleEx` and
27+ :func:`PyImport_ExecCodeModuleWithPathnames`.
28+
29
30 .. cfunction:: PyObject* PyImport_ExecCodeModuleEx(char *name, PyObject *co, char *pathname)
31
32 Like :cfunc:`PyImport_ExecCodeModule`, but the :attr:`__file__` attribute of
33 the module object is set to *pathname* if it is non-``NULL``.
34
35+ See also :func:`PyImport_ExecCodeModuleWithPathnames`.
36+
37+
38+.. cfunction:: PyObject* PyImport_ExecCodeModuleWithPathnames(char *name, PyObject *co, char *pathname, char *cpathname)
39+
40+ Like :cfunc:`PyImport_ExecCodeModuleEx`, but the :attr:`__cached__`
41+ attribute of the module object is set to *cpathname* if it is
42+ non-``NULL``. Of the three functions, this is the preferred one to use.
43+
44
45 .. cfunction:: long PyImport_GetMagicNumber()
46
47@@ -138,6 +150,11 @@
48 of the bytecode file, in little-endian byte order.
49
50
51+.. cfunction:: const char * PyImport_GetMagicTag()
52+
53+ Return the magic tag string for :pep:`3147` format Python bytecode file
54+ names.
55+
56 .. cfunction:: PyObject* PyImport_GetModuleDict()
57
58 Return the dictionary used for the module administration (a.k.a.
59
60=== modified file 'Doc/library/compileall.rst'
61--- Doc/library/compileall.rst 2009-05-17 11:29:12 +0000
62+++ Doc/library/compileall.rst 2010-04-16 18:55:41 +0000
63@@ -17,9 +17,11 @@
64 sys.path``. Printing lists of the files compiled can be disabled with the
65 :option:`-q` flag. In addition, the :option:`-x` option takes a regular
66 expression argument. All files that match the expression will be skipped.
67-
68-
69-.. function:: compile_dir(dir, maxlevels=10, ddir=None, force=False, rx=None, quiet=False)
70+The :option:`-b` flag may be given to write legacy ``.pyc`` file path names,
71+otherwise :pep:`3147` style byte-compiled path names are written.
72+
73+
74+.. function:: compile_dir(dir, maxlevels=10, ddir=None, force=False, rx=None, quiet=False, legacy=False)
75
76 Recursively descend the directory tree named by *dir*, compiling all :file:`.py`
77 files along the way. The *maxlevels* parameter is used to limit the depth of
78@@ -34,12 +36,16 @@
79 If *quiet* is true, nothing is printed to the standard output in normal
80 operation.
81
82-
83-.. function:: compile_path(skip_curdir=True, maxlevels=0, force=False)
84+ If *legacy* is true, old-style ``.pyc`` file path names are written,
85+ otherwise (the default), :pep:`3147` style path names are written.
86+
87+
88+.. function:: compile_path(skip_curdir=True, maxlevels=0, force=False, legacy=False)
89
90 Byte-compile all the :file:`.py` files found along ``sys.path``. If
91 *skip_curdir* is true (the default), the current directory is not included in
92- the search. The *maxlevels* and *force* parameters default to ``0`` and are
93+ the search. The *maxlevels* parameter defaults to ``0``, and the *force*
94+ and *legacy* parameters default to ``False``. All are
95 passed to the :func:`compile_dir` function.
96
97 To force a recompile of all the :file:`.py` files in the :file:`Lib/`
98
99=== modified file 'Doc/library/imp.rst'
100--- Doc/library/imp.rst 2009-07-17 09:42:05 +0000
101+++ Doc/library/imp.rst 2010-04-16 18:55:41 +0000
102@@ -204,8 +204,41 @@
103 function does nothing.
104
105
106-The following constants with integer values, defined in this module, are used to
107-indicate the search result of :func:`find_module`.
108+The following functions and data provide conveniences for handling :pep:`3147`
109+byte-compiled file paths.
110+
111+.. versionadded:: 3.2
112+
113+.. function:: cache_from_source(path, debug_override=None)
114+
115+ Return the PEP 3147 path to the byte-compiled file associated with the
116+ source *path*. For example, if *path* is ``/foo/bar/baz.py`` the return
117+ value would be ``/foo/bar/__pycache__/baz.cpython-32.pyc`` for Python 3.2.
118+ The ``cpython-32`` string comes from the current magic tag (see
119+ :func:`get_tag`). The returned path will end in ``.pyc`` when
120+ ``__debug__`` is True or ``.pyo`` for an optimized Python
121+ (i.e. ``__debug__`` is False). By passing in True or False for
122+ *debug_override* you can override the system's value for ``__debug__`` for
123+ extension selection.
124+
125+ *path* need not exist.
126+
127+.. function:: source_from_cache(path)
128+
129+ Given the *path* to a PEP 3147 file name, return the associated source code
130+ file path. For example, if *path* is
131+ ``/foo/bar/__pycache__/baz.cpython-32.pyc`` the returned path would be
132+ ``/foo/bar/baz.py``. *path* need not exist, however if it does not conform
133+ to PEP 3147 format, a ``ValueError`` is raised.
134+
135+.. function:: get_tag()
136+
137+ Return the PEP 3147 magic tag string matching this version of Python's
138+ magic number, as returned by :func:`get_magic`.
139+
140+
141+The following constants with integer values, defined in this module, are used
142+to indicate the search result of :func:`find_module`.
143
144
145 .. data:: PY_SOURCE
146
147=== modified file 'Doc/library/py_compile.rst'
148--- Doc/library/py_compile.rst 2009-09-02 19:34:52 +0000
149+++ Doc/library/py_compile.rst 2010-04-16 18:55:41 +0000
150@@ -26,12 +26,16 @@
151
152 Compile a source file to byte-code and write out the byte-code cache file. The
153 source code is loaded from the file name *file*. The byte-code is written to
154- *cfile*, which defaults to *file* ``+`` ``'c'`` (``'o'`` if optimization is
155- enabled in the current interpreter). If *dfile* is specified, it is used as the
156+ *cfile*, which defaults to the :PEP:`3147` path, ending in ``.pyc``
157+ (``'.pyo`` if optimization is enabled in the current interpreter). For
158+ example, if *file* is ``/foo/bar/baz.py`` *cfile* will default to
159+ ``/foo/bar/__pycache__/baz.cpython-32.pyc`` for Python 3.2. If *dfile* is specified, it is used as the
160 name of the source file in error messages instead of *file*. If *doraise* is
161 true, a :exc:`PyCompileError` is raised when an error is encountered while
162 compiling *file*. If *doraise* is false (the default), an error string is
163- written to ``sys.stderr``, but no exception is raised.
164+ written to ``sys.stderr``, but no exception is raised. This function
165+ returns the path to byte-compiled file, i.e. whatever *cfile* value was
166+ used.
167
168
169 .. function:: main(args=None)
170
171=== modified file 'Doc/library/runpy.rst'
172--- Doc/library/runpy.rst 2010-01-04 09:29:10 +0000
173+++ Doc/library/runpy.rst 2010-04-16 18:55:41 +0000
174@@ -32,7 +32,8 @@
175 below are defined in the supplied dictionary, those definitions are
176 overridden by :func:`run_module`.
177
178- The special global variables ``__name__``, ``__file__``, ``__loader__``
179+ The special global variables ``__name__``, ``__file__``, ``__cached__``,
180+ ``__loader__``
181 and ``__package__`` are set in the globals dictionary before the module
182 code is executed (Note that this is a minimal set of variables - other
183 variables may be set implicitly as an interpreter implementation detail).
184@@ -45,6 +46,8 @@
185 loader does not make filename information available, this variable is set
186 to :const:`None`.
187
188+ ``__cached__`` will be set to ``None``.
189+
190 ``__loader__`` is set to the PEP 302 module loader used to retrieve the
191 code for the module (This loader may be a wrapper around the standard
192 import mechanism).
193
194=== modified file 'Include/import.h'
195--- Include/import.h 2009-10-04 19:32:25 +0000
196+++ Include/import.h 2010-04-16 18:55:41 +0000
197@@ -8,9 +8,12 @@
198 #endif
199
200 PyAPI_FUNC(long) PyImport_GetMagicNumber(void);
201+PyAPI_FUNC(const char *) PyImport_GetMagicTag(void);
202 PyAPI_FUNC(PyObject *) PyImport_ExecCodeModule(char *name, PyObject *co);
203 PyAPI_FUNC(PyObject *) PyImport_ExecCodeModuleEx(
204 char *name, PyObject *co, char *pathname);
205+PyAPI_FUNC(PyObject *) PyImport_ExecCodeModuleWithPathnames(
206+ char *name, PyObject *co, char *pathname, char *cpathname);
207 PyAPI_FUNC(PyObject *) PyImport_GetModuleDict(void);
208 PyAPI_FUNC(PyObject *) PyImport_AddModule(const char *name);
209 PyAPI_FUNC(PyObject *) PyImport_ImportModule(const char *name);
210
211=== modified file 'Lib/compileall.py'
212--- Lib/compileall.py 2010-03-21 23:13:07 +0000
213+++ Lib/compileall.py 2010-04-16 18:55:41 +0000
214@@ -12,6 +12,7 @@
215
216 """
217 import os
218+import errno
219 import sys
220 import py_compile
221 import struct
222@@ -20,7 +21,7 @@
223 __all__ = ["compile_dir","compile_file","compile_path"]
224
225 def compile_dir(dir, maxlevels=10, ddir=None,
226- force=0, rx=None, quiet=0):
227+ force=False, rx=None, quiet=False, legacy=False):
228 """Byte-compile all modules in the given directory tree.
229
230 Arguments (only dir is required):
231@@ -29,8 +30,9 @@
232 maxlevels: maximum recursion level (default 10)
233 ddir: if given, purported directory name (this is the
234 directory name that will show up in error messages)
235- force: if 1, force compilation, even if timestamps are up-to-date
236- quiet: if 1, be quiet during compilation
237+ force: if True, force compilation, even if timestamps are up-to-date
238+ quiet: if True, be quiet during compilation
239+ legacy: if True, produce legacy pyc paths instead of PEP 3147 paths
240
241 """
242 if not quiet:
243@@ -49,24 +51,26 @@
244 else:
245 dfile = None
246 if not os.path.isdir(fullname):
247- if not compile_file(fullname, ddir, force, rx, quiet):
248+ if not compile_file(fullname, ddir, force, rx, quiet, legacy):
249 success = 0
250 elif maxlevels > 0 and \
251 name != os.curdir and name != os.pardir and \
252 os.path.isdir(fullname) and \
253 not os.path.islink(fullname):
254 if not compile_dir(fullname, maxlevels - 1, dfile, force, rx,
255- quiet):
256+ quiet, legacy):
257 success = 0
258 return success
259
260-def compile_file(fullname, ddir=None, force=0, rx=None, quiet=0):
261+def compile_file(fullname, ddir=None, force=0, rx=None, quiet=False,
262+ legacy=False):
263 """Byte-compile file.
264- file: the file to byte-compile
265+ fullname: the file to byte-compile
266 ddir: if given, purported directory name (this is the
267 directory name that will show up in error messages)
268- force: if 1, force compilation, even if timestamps are up-to-date
269- quiet: if 1, be quiet during compilation
270+ force: if True, force compilation, even if timestamps are up-to-date
271+ quiet: if True, be quiet during compilation
272+ legacy: if True, produce legacy pyc paths instead of PEP 3147 paths
273
274 """
275 success = 1
276@@ -80,13 +84,22 @@
277 if mo:
278 return success
279 if os.path.isfile(fullname):
280+ if legacy:
281+ cfile = fullname + ('c' if __debug__ else 'o')
282+ else:
283+ cfile = imp.cache_from_source(fullname)
284+ cache_dir = os.path.dirname(cfile)
285+ try:
286+ os.mkdir(cache_dir)
287+ except OSError as error:
288+ if error.errno != errno.EEXIST:
289+ raise
290 head, tail = name[:-3], name[-3:]
291 if tail == '.py':
292 if not force:
293 try:
294 mtime = int(os.stat(fullname).st_mtime)
295 expect = struct.pack('<4sl', imp.get_magic(), mtime)
296- cfile = fullname + (__debug__ and 'c' or 'o')
297 with open(cfile, 'rb') as chandle:
298 actual = chandle.read(8)
299 if expect == actual:
300@@ -96,14 +109,15 @@
301 if not quiet:
302 print('Compiling', fullname, '...')
303 try:
304- ok = py_compile.compile(fullname, None, dfile, True)
305+ ok = py_compile.compile(fullname, cfile, dfile, True)
306 except py_compile.PyCompileError as err:
307 if quiet:
308 print('*** Error compiling', fullname, '...')
309 else:
310 print('*** ', end='')
311 # escape non-printable characters in msg
312- msg = err.msg.encode(sys.stdout.encoding, errors='backslashreplace')
313+ msg = err.msg.encode(sys.stdout.encoding,
314+ errors='backslashreplace')
315 msg = msg.decode(sys.stdout.encoding)
316 print(msg)
317 success = 0
318@@ -119,15 +133,17 @@
319 success = 0
320 return success
321
322-def compile_path(skip_curdir=1, maxlevels=0, force=0, quiet=0):
323+def compile_path(skip_curdir=1, maxlevels=0, force=False, quiet=False,
324+ legacy=False):
325 """Byte-compile all module on sys.path.
326
327 Arguments (all optional):
328
329 skip_curdir: if true, skip current directory (default true)
330 maxlevels: max recursion level (default 0)
331- force: as for compile_dir() (default 0)
332- quiet: as for compile_dir() (default 0)
333+ force: as for compile_dir() (default False)
334+ quiet: as for compile_dir() (default False)
335+ legacy: as for compile_dir() (default False)
336
337 """
338 success = 1
339@@ -136,7 +152,8 @@
340 print('Skipping current directory')
341 else:
342 success = success and compile_dir(dir, maxlevels, None,
343- force, quiet=quiet)
344+ force, quiet=quiet,
345+ legacy=legacy)
346 return success
347
348 def expand_args(args, flist):
349@@ -162,10 +179,10 @@
350 """Script main program."""
351 import getopt
352 try:
353- opts, args = getopt.getopt(sys.argv[1:], 'lfqd:x:i:')
354+ opts, args = getopt.getopt(sys.argv[1:], 'lfqd:x:i:b')
355 except getopt.error as msg:
356 print(msg)
357- print("usage: python compileall.py [-l] [-f] [-q] [-d destdir] " \
358+ print("usage: python compileall.py [-l] [-f] [-q] [-d destdir] "
359 "[-x regexp] [-i list] [directory|file ...]")
360 print("-l: don't recurse down")
361 print("-f: force rebuild even if timestamps are up-to-date")
362@@ -174,23 +191,27 @@
363 print(" if no directory arguments, -l sys.path is assumed")
364 print("-x regexp: skip files matching the regular expression regexp")
365 print(" the regexp is searched for in the full path of the file")
366- print("-i list: expand list with its content (file and directory names)")
367+ print("-i list: expand list with its content "
368+ "(file and directory names)")
369+ print("-b: Produce legacy byte-compile file paths")
370 sys.exit(2)
371 maxlevels = 10
372 ddir = None
373- force = 0
374- quiet = 0
375+ force = False
376+ quiet = False
377 rx = None
378 flist = None
379+ legacy = False
380 for o, a in opts:
381 if o == '-l': maxlevels = 0
382 if o == '-d': ddir = a
383- if o == '-f': force = 1
384- if o == '-q': quiet = 1
385+ if o == '-f': force = True
386+ if o == '-q': quiet = True
387 if o == '-x':
388 import re
389 rx = re.compile(a)
390 if o == '-i': flist = a
391+ if o == '-b': legacy = True
392 if ddir:
393 if len(args) != 1 and not os.path.isdir(args[0]):
394 print("-d destdir require exactly one directory argument")
395@@ -207,13 +228,14 @@
396 for arg in args:
397 if os.path.isdir(arg):
398 if not compile_dir(arg, maxlevels, ddir,
399- force, rx, quiet):
400+ force, rx, quiet, legacy):
401 success = 0
402 else:
403- if not compile_file(arg, ddir, force, rx, quiet):
404+ if not compile_file(arg, ddir, force, rx,
405+ quiet, legacy):
406 success = 0
407 else:
408- success = compile_path()
409+ success = compile_path(legacy=legacy)
410 except KeyboardInterrupt:
411 print("\n[interrupt]")
412 success = 0
413
414=== modified file 'Lib/importlib/_bootstrap.py'
415--- Lib/importlib/_bootstrap.py 2010-03-19 20:44:28 +0000
416+++ Lib/importlib/_bootstrap.py 2010-04-16 18:55:41 +0000
417@@ -488,6 +488,16 @@
418
419 """Load a module from a source or bytecode file."""
420
421+ def _find_path(self, ext_type):
422+ """Return PEP 3147 path if ext_type is PY_COMPILED, otherwise
423+ super()._find_path() is called."""
424+ if ext_type == imp.PY_COMPILED:
425+ # We don't really care what the extension on self._base_path is,
426+ # as long as it has exactly one dot.
427+ bytecode_path = imp.cache_from_source(self._base_path + '.py')
428+ return (bytecode_path if _path_exists(bytecode_path) else None)
429+ return super()._find_path(ext_type)
430+
431 @_check_name
432 def source_mtime(self, name):
433 """Return the modification time of the source for the specified
434@@ -515,7 +525,16 @@
435 """
436 bytecode_path = self.bytecode_path(name)
437 if not bytecode_path:
438- bytecode_path = self._base_path + _suffix_list(imp.PY_COMPILED)[0]
439+ source_path = self.source_path(name)
440+ bytecode_path = imp.cache_from_source(source_path)
441+ # Ensure that the __pycache__ directory exists. We can't use
442+ # os.path.dirname() here.
443+ dirname, sep, basename = bytecode_path.rpartition(path_sep)
444+ try:
445+ _os.mkdir(dirname)
446+ except OSError as error:
447+ if error.errno != errno.EEXIST:
448+ raise
449 try:
450 # Assuming bytes.
451 with _closing(_io.FileIO(bytecode_path, 'w')) as bytecode_file:
452
453=== modified file 'Lib/importlib/test/__main__.py'
454--- Lib/importlib/test/__main__.py 2010-03-19 20:44:28 +0000
455+++ Lib/importlib/test/__main__.py 2010-04-16 18:55:41 +0000
456@@ -13,7 +13,12 @@
457
458
459 def test_main():
460- start_dir = os.path.dirname(__file__)
461+ if '__pycache__' in __file__:
462+ parts = __file__.split(os.path.sep)
463+ start_dir = sep.join(parts[:-2])
464+ else:
465+ start_dir = os.path.dirname(__file__)
466+ # XXX 2010-03-18 barry: Fix __file__
467 top_dir = os.path.dirname(os.path.dirname(start_dir))
468 test_loader = unittest.TestLoader()
469 if '--builtin' in sys.argv:
470
471=== modified file 'Lib/importlib/test/source/test_file_loader.py'
472--- Lib/importlib/test/source/test_file_loader.py 2010-03-19 20:44:28 +0000
473+++ Lib/importlib/test/source/test_file_loader.py 2010-04-16 18:55:41 +0000
474@@ -127,7 +127,7 @@
475 except KeyError:
476 pass
477 py_compile.compile(mapping[name])
478- bytecode_path = source_util.bytecode_path(mapping[name])
479+ bytecode_path = imp.cache_from_source(mapping[name])
480 with open(bytecode_path, 'rb') as file:
481 bc = file.read()
482 new_bc = manipulator(bc)
483@@ -226,7 +226,7 @@
484 zeros = b'\x00\x00\x00\x00'
485 with source_util.create_modules('_temp') as mapping:
486 py_compile.compile(mapping['_temp'])
487- bytecode_path = source_util.bytecode_path(mapping['_temp'])
488+ bytecode_path = imp.cache_from_source(mapping['_temp'])
489 with open(bytecode_path, 'r+b') as bytecode_file:
490 bytecode_file.seek(4)
491 bytecode_file.write(zeros)
492@@ -242,9 +242,10 @@
493 def test_bad_marshal(self):
494 # Bad marshal data should raise a ValueError.
495 with source_util.create_modules('_temp') as mapping:
496- bytecode_path = source_util.bytecode_path(mapping['_temp'])
497+ bytecode_path = imp.cache_from_source(mapping['_temp'])
498 source_mtime = os.path.getmtime(mapping['_temp'])
499 source_timestamp = importlib._w_long(source_mtime)
500+ source_util.ensure_bytecode_path(bytecode_path)
501 with open(bytecode_path, 'wb') as bytecode_file:
502 bytecode_file.write(imp.get_magic())
503 bytecode_file.write(source_timestamp)
504@@ -260,7 +261,7 @@
505 with source_util.create_modules('_temp') as mapping:
506 # Create bytecode that will need to be re-created.
507 py_compile.compile(mapping['_temp'])
508- bytecode_path = source_util.bytecode_path(mapping['_temp'])
509+ bytecode_path = imp.cache_from_source(mapping['_temp'])
510 with open(bytecode_path, 'r+b') as bytecode_file:
511 bytecode_file.seek(0)
512 bytecode_file.write(b'\x00\x00\x00\x00')
513
514=== modified file 'Lib/importlib/test/source/test_finder.py'
515--- Lib/importlib/test/source/test_finder.py 2010-03-19 20:44:28 +0000
516+++ Lib/importlib/test/source/test_finder.py 2010-04-16 18:55:41 +0000
517@@ -1,7 +1,9 @@
518 from importlib import _bootstrap
519 from .. import abc
520 from . import util as source_util
521+from test.support import make_legacy_pyc
522 import os
523+import errno
524 import py_compile
525 import unittest
526 import warnings
527@@ -52,6 +54,14 @@
528 if unlink:
529 for name in unlink:
530 os.unlink(mapping[name])
531+ try:
532+ make_legacy_pyc(mapping[name])
533+ except OSError as error:
534+ # Some tests do not set compile_=True so the source
535+ # module will not get compiled and there will be no
536+ # PEP 3147 pyc file to rename.
537+ if error.errno != errno.ENOENT:
538+ raise
539 loader = self.import_(mapping['.root'], test)
540 self.assertTrue(hasattr(loader, 'load_module'))
541 return loader
542@@ -60,7 +70,8 @@
543 # [top-level source]
544 self.run_test('top_level')
545 # [top-level bc]
546- self.run_test('top_level', compile_={'top_level'}, unlink={'top_level'})
547+ self.run_test('top_level', compile_={'top_level'},
548+ unlink={'top_level'})
549 # [top-level both]
550 self.run_test('top_level', compile_={'top_level'})
551
552
553=== modified file 'Lib/importlib/test/source/test_source_encoding.py'
554--- Lib/importlib/test/source/test_source_encoding.py 2010-03-19 20:44:28 +0000
555+++ Lib/importlib/test/source/test_source_encoding.py 2010-04-16 18:55:41 +0000
556@@ -33,7 +33,7 @@
557
558 def run_test(self, source):
559 with source_util.create_modules(self.module_name) as mapping:
560- with open(mapping[self.module_name], 'wb')as file:
561+ with open(mapping[self.module_name], 'wb') as file:
562 file.write(source)
563 loader = _bootstrap._PyPycFileLoader(self.module_name,
564 mapping[self.module_name], False)
565
566=== modified file 'Lib/importlib/test/source/util.py'
567--- Lib/importlib/test/source/util.py 2010-03-19 20:44:28 +0000
568+++ Lib/importlib/test/source/util.py 2010-04-16 18:55:41 +0000
569@@ -1,5 +1,6 @@
570 from .. import util
571 import contextlib
572+import errno
573 import functools
574 import imp
575 import os
576@@ -26,14 +27,16 @@
577 return wrapper
578
579
580-def bytecode_path(source_path):
581- for suffix, _, type_ in imp.get_suffixes():
582- if type_ == imp.PY_COMPILED:
583- bc_suffix = suffix
584- break
585- else:
586- raise ValueError("no bytecode suffix is defined")
587- return os.path.splitext(source_path)[0] + bc_suffix
588+def ensure_bytecode_path(bytecode_path):
589+ """Ensure that the __pycache__ directory for PEP 3147 pyc file exists.
590+
591+ :param bytecode_path: File system path to PEP 3147 pyc file.
592+ """
593+ try:
594+ os.mkdir(os.path.dirname(bytecode_path))
595+ except OSError as error:
596+ if error.errno != errno.EEXIST:
597+ raise
598
599
600 @contextlib.contextmanager
601
602=== modified file 'Lib/importlib/util.py'
603--- Lib/importlib/util.py 2010-03-19 20:44:28 +0000
604+++ Lib/importlib/util.py 2010-04-16 18:55:41 +0000
605@@ -1,4 +1,5 @@
606 """Utility code for constructing importers, etc."""
607+
608 from ._bootstrap import module_for_loader
609 from ._bootstrap import set_loader
610 from ._bootstrap import set_package
611
612=== modified file 'Lib/inspect.py'
613--- Lib/inspect.py 2010-03-30 17:42:32 +0000
614+++ Lib/inspect.py 2010-04-16 18:55:41 +0000
615@@ -54,6 +54,7 @@
616 """Return true if the object is a module.
617
618 Module objects provide these attributes:
619+ __cached__ pathname to byte compiled file
620 __doc__ documentation string
621 __file__ filename (missing for built-in modules)"""
622 return isinstance(object, types.ModuleType)
623
624=== modified file 'Lib/py_compile.py'
625--- Lib/py_compile.py 2010-03-31 20:36:22 +0000
626+++ Lib/py_compile.py 2010-04-16 18:55:41 +0000
627@@ -4,6 +4,7 @@
628 """
629
630 import builtins
631+import errno
632 import imp
633 import marshal
634 import os
635@@ -37,16 +38,18 @@
636 can be accesses as class variable 'file'
637
638 msg: string message to be written as error message
639- If no value is given, a default exception message will be given,
640- consistent with 'standard' py_compile output.
641- message (or default) can be accesses as class variable 'msg'
642+ If no value is given, a default exception message will be
643+ given, consistent with 'standard' py_compile output.
644+ message (or default) can be accesses as class variable
645+ 'msg'
646
647 """
648
649 def __init__(self, exc_type, exc_value, file, msg=''):
650 exc_type_name = exc_type.__name__
651 if exc_type is SyntaxError:
652- tbtext = ''.join(traceback.format_exception_only(exc_type, exc_value))
653+ tbtext = ''.join(traceback.format_exception_only(
654+ exc_type, exc_value))
655 errmsg = tbtext.replace('File "<string>"', 'File "%s"' % file)
656 else:
657 errmsg = "Sorry: %s: %s" % (exc_type_name,exc_value)
658@@ -64,7 +67,7 @@
659
660 def wr_long(f, x):
661 """Internal; write a 32-bit int to a file in little-endian order."""
662- f.write(bytes([x & 0xff,
663+ f.write(bytes([x & 0xff,
664 (x >> 8) & 0xff,
665 (x >> 16) & 0xff,
666 (x >> 24) & 0xff]))
667@@ -72,20 +75,18 @@
668 def compile(file, cfile=None, dfile=None, doraise=False):
669 """Byte-compile one Python source file to Python bytecode.
670
671- Arguments:
672-
673- file: source filename
674- cfile: target filename; defaults to source with 'c' or 'o' appended
675- ('c' normally, 'o' in optimizing mode, giving .pyc or .pyo)
676- dfile: purported filename; defaults to source (this is the filename
677- that will show up in error messages)
678- doraise: flag indicating whether or not an exception should be
679- raised when a compile error is found. If an exception
680- occurs and this flag is set to False, a string
681- indicating the nature of the exception will be printed,
682- and the function will return to the caller. If an
683- exception occurs and this flag is set to True, a
684- PyCompileError exception will be raised.
685+ :param file: The source file name.
686+ :param cfile: The target byte compiled file name. When not given, this
687+ defaults to the PEP 3147 location.
688+ :param dfile: Purported file name, i.e. the file name that shows up in
689+ error messages. Defaults to the source file name.
690+ :param doraise: Flag indicating whether or not an exception should be
691+ raised when a compile error is found. If an exception occurs and this
692+ flag is set to False, a string indicating the nature of the exception
693+ will be printed, and the function will return to the caller. If an
694+ exception occurs and this flag is set to True, a PyCompileError
695+ exception will be raised.
696+ :return: Path to the resulting byte compiled file.
697
698 Note that it isn't necessary to byte-compile Python modules for
699 execution efficiency -- Python itself byte-compiles a module when
700@@ -102,7 +103,6 @@
701 See compileall.py for a script/module that uses this module to
702 byte-compile all installed files (or all files in selected
703 directories).
704-
705 """
706 with open(file, "rb") as f:
707 encoding = tokenize.detect_encoding(f.readline)[0]
708@@ -122,7 +122,12 @@
709 sys.stderr.write(py_exc.msg + '\n')
710 return
711 if cfile is None:
712- cfile = file + (__debug__ and 'c' or 'o')
713+ cfile = imp.cache_from_source(file)
714+ try:
715+ os.mkdir(os.path.dirname(cfile))
716+ except OSError as error:
717+ if error.errno != errno.EEXIST:
718+ raise
719 with open(cfile, 'wb') as fc:
720 fc.write(b'\0\0\0\0')
721 wr_long(fc, timestamp)
722@@ -130,6 +135,7 @@
723 fc.flush()
724 fc.seek(0, 0)
725 fc.write(MAGIC)
726+ return cfile
727
728 def main(args=None):
729 """Compile several source files.
730
731=== modified file 'Lib/pydoc.py'
732--- Lib/pydoc.py 2010-04-01 03:05:25 +0000
733+++ Lib/pydoc.py 2010-04-16 18:55:41 +0000
734@@ -159,7 +159,8 @@
735 """Decide whether to show documentation on a variable."""
736 # Certain special names are redundant.
737 _hidden_names = ('__builtins__', '__doc__', '__file__', '__path__',
738- '__module__', '__name__', '__slots__', '__package__')
739+ '__module__', '__name__', '__slots__', '__package__',
740+ '__cached__')
741 if name in _hidden_names: return 0
742 # Private names are hidden, but special names are displayed.
743 if name.startswith('__') and name.endswith('__'): return 1
744
745=== modified file 'Lib/runpy.py'
746--- Lib/runpy.py 2009-12-20 17:28:31 +0000
747+++ Lib/runpy.py 2010-04-16 18:55:41 +0000
748@@ -67,6 +67,7 @@
749 run_globals.update(init_globals)
750 run_globals.update(__name__ = mod_name,
751 __file__ = mod_fname,
752+ __cached__ = None,
753 __loader__ = mod_loader,
754 __package__ = pkg_name)
755 exec(code, run_globals)
756@@ -130,6 +131,7 @@
757 At the very least, these variables in __main__ will be overwritten:
758 __name__
759 __file__
760+ __cached__
761 __loader__
762 __package__
763 """
764
765=== modified file 'Lib/site.py'
766--- Lib/site.py 2010-04-12 22:33:42 +0000
767+++ Lib/site.py 2010-04-16 18:55:41 +0000
768@@ -74,15 +74,19 @@
769 return dir, os.path.normcase(dir)
770
771
772-def abs__file__():
773- """Set all module' __file__ attribute to an absolute path"""
774+def abs_paths():
775+ """Set all module __file__ and __cached__ attributes to an absolute path"""
776 for m in set(sys.modules.values()):
777 if hasattr(m, '__loader__'):
778 continue # don't mess with a PEP 302-supplied __file__
779 try:
780 m.__file__ = os.path.abspath(m.__file__)
781 except AttributeError:
782- continue
783+ pass
784+ try:
785+ m.__cached__ = os.path.abspath(m.__cached__)
786+ except AttributeError:
787+ pass
788
789
790 def removeduppaths():
791@@ -518,7 +522,7 @@
792 def main():
793 global ENABLE_USER_SITE
794
795- abs__file__()
796+ abs_paths()
797 known_paths = removeduppaths()
798 if (os.name == "posix" and sys.path and
799 os.path.basename(sys.path[-1]) == "Modules"):
800
801=== modified file 'Lib/test/script_helper.py'
802--- Lib/test/script_helper.py 2010-02-27 16:12:22 +0000
803+++ Lib/test/script_helper.py 2010-04-16 18:55:41 +0000
804@@ -11,6 +11,9 @@
805 import shutil
806 import zipfile
807
808+from imp import source_from_cache
809+from test.support import make_legacy_pyc
810+
811 # Executing the interpreter in a subprocess
812 def python_exit_code(*args):
813 cmd_line = [sys.executable, '-E']
814@@ -62,20 +65,18 @@
815 script_file.close()
816 return script_name
817
818-def compile_script(script_name):
819- py_compile.compile(script_name, doraise=True)
820- if __debug__:
821- compiled_name = script_name + 'c'
822- else:
823- compiled_name = script_name + 'o'
824- return compiled_name
825-
826 def make_zip_script(zip_dir, zip_basename, script_name, name_in_zip=None):
827 zip_filename = zip_basename+os.extsep+'zip'
828 zip_name = os.path.join(zip_dir, zip_filename)
829 zip_file = zipfile.ZipFile(zip_name, 'w')
830 if name_in_zip is None:
831- name_in_zip = os.path.basename(script_name)
832+ parts = script_name.split(os.sep)
833+ if len(parts) >= 2 and parts[-2] == '__pycache__':
834+ legacy_pyc = make_legacy_pyc(source_from_cache(script_name))
835+ name_in_zip = os.path.basename(legacy_pyc)
836+ script_name = legacy_pyc
837+ else:
838+ name_in_zip = os.path.basename(script_name)
839 zip_file.write(script_name, name_in_zip)
840 zip_file.close()
841 #if test.test_support.verbose:
842@@ -98,8 +99,8 @@
843 script_name = make_script(zip_dir, script_basename, source)
844 unlink.append(script_name)
845 if compiled:
846- init_name = compile_script(init_name)
847- script_name = compile_script(script_name)
848+ init_name = py_compile(init_name, doraise=True)
849+ script_name = py_compile(script_name, doraise=True)
850 unlink.extend((init_name, script_name))
851 pkg_names = [os.sep.join([pkg_name]*i) for i in range(1, depth+1)]
852 script_name_in_zip = os.path.join(pkg_names[-1], os.path.basename(script_name))
853
854=== modified file 'Lib/test/support.py'
855--- Lib/test/support.py 2010-04-11 20:22:10 +0000
856+++ Lib/test/support.py 2010-04-16 18:55:41 +0000
857@@ -17,22 +17,25 @@
858 import importlib
859 import collections
860 import re
861+import imp
862 import time
863
864-__all__ = ["Error", "TestFailed", "ResourceDenied", "import_module",
865- "verbose", "use_resources", "max_memuse", "record_original_stdout",
866- "get_original_stdout", "unload", "unlink", "rmtree", "forget",
867- "is_resource_enabled", "requires", "find_unused_port", "bind_port",
868- "fcmp", "is_jython", "TESTFN", "HOST", "FUZZ", "SAVEDCWD", "temp_cwd",
869- "findfile", "sortdict", "check_syntax_error", "open_urlresource",
870- "check_warnings", "CleanImport", "EnvironmentVarGuard",
871- "TransientResource", "captured_output", "captured_stdout",
872- "time_out", "socket_peer_reset", "ioerror_peer_reset",
873- "run_with_locale",
874- "set_memlimit", "bigmemtest", "bigaddrspacetest", "BasicTestRunner",
875- "run_unittest", "run_doctest", "threading_setup", "threading_cleanup",
876- "reap_children", "cpython_only", "check_impl_detail", "get_attribute",
877- "swap_item", "swap_attr"]
878+__all__ = [
879+ "Error", "TestFailed", "ResourceDenied", "import_module",
880+ "verbose", "use_resources", "max_memuse", "record_original_stdout",
881+ "get_original_stdout", "unload", "unlink", "rmtree", "forget",
882+ "is_resource_enabled", "requires", "find_unused_port", "bind_port",
883+ "fcmp", "is_jython", "TESTFN", "HOST", "FUZZ", "SAVEDCWD", "temp_cwd",
884+ "findfile", "sortdict", "check_syntax_error", "open_urlresource",
885+ "check_warnings", "CleanImport", "EnvironmentVarGuard",
886+ "TransientResource", "captured_output", "captured_stdout",
887+ "time_out", "socket_peer_reset", "ioerror_peer_reset",
888+ "run_with_locale", 'temp_umask',
889+ "set_memlimit", "bigmemtest", "bigaddrspacetest", "BasicTestRunner",
890+ "run_unittest", "run_doctest", "threading_setup", "threading_cleanup",
891+ "reap_children", "cpython_only", "check_impl_detail", "get_attribute",
892+ "swap_item", "swap_attr",
893+ ]
894
895
896 class Error(Exception):
897@@ -177,27 +180,50 @@
898 def unlink(filename):
899 try:
900 os.unlink(filename)
901- except OSError:
902- pass
903+ except OSError as error:
904+ # The filename need not exist.
905+ if error.errno != errno.ENOENT:
906+ raise
907
908 def rmtree(path):
909 try:
910 shutil.rmtree(path)
911- except OSError as e:
912+ except OSError as error:
913 # Unix returns ENOENT, Windows returns ESRCH.
914- if e.errno not in (errno.ENOENT, errno.ESRCH):
915+ if error.errno not in (errno.ENOENT, errno.ESRCH):
916 raise
917
918+def make_legacy_pyc(source):
919+ """Move a PEP 3147 pyc/pyo file to its legacy pyc/pyo location.
920+
921+ The choice of .pyc or .pyo extension is done based on the __debug__ flag
922+ value.
923+
924+ :param source: The file system path to the source file. The source file
925+ does not need to exist, however the PEP 3147 pyc file must exist.
926+ :return: The file system path to the legacy pyc file.
927+ """
928+ pyc_file = imp.cache_from_source(source)
929+ up_one = os.path.dirname(os.path.abspath(source))
930+ legacy_pyc = os.path.join(up_one, source + ('c' if __debug__ else 'o'))
931+ os.rename(pyc_file, legacy_pyc)
932+ return legacy_pyc
933+
934 def forget(modname):
935- '''"Forget" a module was ever imported by removing it from sys.modules and
936- deleting any .pyc and .pyo files.'''
937+ """'Forget' a module was ever imported.
938+
939+ This removes the module from sys.modules and deletes any PEP 3147 or
940+ legacy .pyc and .pyo files.
941+ """
942 unload(modname)
943 for dirname in sys.path:
944- unlink(os.path.join(dirname, modname + '.pyc'))
945- # Deleting the .pyo file cannot be within the 'try' for the .pyc since
946- # the chance exists that there is no .pyc (and thus the 'try' statement
947- # is exited) but there is a .pyo file.
948- unlink(os.path.join(dirname, modname + '.pyo'))
949+ source = os.path.join(dirname, modname + '.py')
950+ # It doesn't matter if they exist or not, unlink all possible
951+ # combinations of PEP 3147 and legacy pyc and pyo files.
952+ unlink(source + 'c')
953+ unlink(source + 'o')
954+ unlink(imp.cache_from_source(source, debug_override=True))
955+ unlink(imp.cache_from_source(source, debug_override=False))
956
957 def is_resource_enabled(resource):
958 """Test whether a resource is enabled. Known resources are set by
959@@ -208,7 +234,9 @@
960 """Raise ResourceDenied if the specified resource is not available.
961
962 If the caller's module is __main__ then automatically return True. The
963- possibility of False being returned occurs when regrtest.py is executing."""
964+ possibility of False being returned occurs when regrtest.py is
965+ executing.
966+ """
967 # see if the caller's module is __main__ - if so, treat as if
968 # the resource was set
969 if sys._getframe(1).f_globals.get("__name__") == "__main__":
970@@ -405,6 +433,16 @@
971 rmtree(name)
972
973
974+@contextlib.contextmanager
975+def temp_umask(umask):
976+ """Context manager that temporarily sets the process umask."""
977+ oldmask = os.umask(umask)
978+ try:
979+ yield
980+ finally:
981+ os.umask(oldmask)
982+
983+
984 def findfile(file, here=__file__, subdir=None):
985 """Try to find a file on sys.path and the working directory. If it is not
986 found the argument passed to the function is returned (this does not
987
988=== modified file 'Lib/test/test_cmd_line_script.py'
989--- Lib/test/test_cmd_line_script.py 2010-03-14 10:45:50 +0000
990+++ Lib/test/test_cmd_line_script.py 2010-04-16 18:55:41 +0000
991@@ -1,12 +1,14 @@
992-# Tests command line execution of scripts
993+# tests command line execution of scripts
994
995 import unittest
996 import os
997 import os.path
998+import py_compile
999+
1000 import test.support
1001-from test.script_helper import (run_python,
1002- temp_dir, make_script, compile_script,
1003- make_pkg, make_zip_script, make_zip_pkg)
1004+from test.script_helper import (
1005+ make_pkg, make_script, make_zip_pkg, make_zip_script, run_python,
1006+ temp_dir)
1007
1008 verbose = test.support.verbose
1009
1010@@ -28,6 +30,7 @@
1011 # Check population of magic variables
1012 assertEqual(__name__, '__main__')
1013 print('__file__==%r' % __file__)
1014+assertEqual(__cached__, None)
1015 print('__package__==%r' % __package__)
1016 # Check the sys module
1017 import sys
1018@@ -101,9 +104,10 @@
1019 def test_script_compiled(self):
1020 with temp_dir() as script_dir:
1021 script_name = _make_test_script(script_dir, 'script')
1022- compiled_name = compile_script(script_name)
1023+ compiled_name = py_compile.compile(script_name, doraise=True)
1024 os.remove(script_name)
1025- self._check_script(compiled_name, compiled_name, compiled_name, None)
1026+ self._check_script(compiled_name, compiled_name,
1027+ compiled_name, None)
1028
1029 def test_directory(self):
1030 with temp_dir() as script_dir:
1031@@ -113,9 +117,10 @@
1032 def test_directory_compiled(self):
1033 with temp_dir() as script_dir:
1034 script_name = _make_test_script(script_dir, '__main__')
1035- compiled_name = compile_script(script_name)
1036+ compiled_name = py_compile.compile(script_name, doraise=True)
1037 os.remove(script_name)
1038- self._check_script(script_dir, compiled_name, script_dir, '')
1039+ pyc_file = test.support.make_legacy_pyc(script_name)
1040+ self._check_script(script_dir, pyc_file, script_dir, '')
1041
1042 def test_directory_error(self):
1043 with temp_dir() as script_dir:
1044@@ -131,7 +136,7 @@
1045 def test_zipfile_compiled(self):
1046 with temp_dir() as script_dir:
1047 script_name = _make_test_script(script_dir, '__main__')
1048- compiled_name = compile_script(script_name)
1049+ compiled_name = py_compile.compile(script_name, doraise=True)
1050 zip_name, run_name = make_zip_script(script_dir, 'test_zip', compiled_name)
1051 self._check_script(zip_name, run_name, zip_name, '')
1052
1053@@ -176,11 +181,12 @@
1054 pkg_dir = os.path.join(script_dir, 'test_pkg')
1055 make_pkg(pkg_dir)
1056 script_name = _make_test_script(pkg_dir, '__main__')
1057- compiled_name = compile_script(script_name)
1058+ compiled_name = py_compile.compile(script_name, doraise=True)
1059 os.remove(script_name)
1060+ pyc_file = test.support.make_legacy_pyc(script_name)
1061 launch_name = _make_launch_script(script_dir, 'launch', 'test_pkg')
1062- self._check_script(launch_name, compiled_name,
1063- compiled_name, 'test_pkg')
1064+ self._check_script(launch_name, pyc_file,
1065+ pyc_file, 'test_pkg')
1066
1067 def test_package_error(self):
1068 with temp_dir() as script_dir:
1069
1070=== modified file 'Lib/test/test_compileall.py'
1071--- Lib/test/test_compileall.py 2010-03-16 13:19:21 +0000
1072+++ Lib/test/test_compileall.py 2010-04-16 18:55:41 +0000
1073@@ -5,22 +5,23 @@
1074 import py_compile
1075 import shutil
1076 import struct
1077+import subprocess
1078 import tempfile
1079-from test import support
1080 import unittest
1081 import io
1082
1083+from test import support
1084
1085 class CompileallTests(unittest.TestCase):
1086
1087 def setUp(self):
1088 self.directory = tempfile.mkdtemp()
1089 self.source_path = os.path.join(self.directory, '_test.py')
1090- self.bc_path = self.source_path + ('c' if __debug__ else 'o')
1091+ self.bc_path = imp.cache_from_source(self.source_path)
1092 with open(self.source_path, 'w') as file:
1093 file.write('x = 123\n')
1094 self.source_path2 = os.path.join(self.directory, '_test2.py')
1095- self.bc_path2 = self.source_path2 + ('c' if __debug__ else 'o')
1096+ self.bc_path2 = imp.cache_from_source(self.source_path2)
1097 shutil.copyfile(self.source_path, self.source_path2)
1098
1099 def tearDown(self):
1100@@ -65,17 +66,19 @@
1101 except:
1102 pass
1103 compileall.compile_file(self.source_path, force=False, quiet=True)
1104- self.assertTrue(os.path.isfile(self.bc_path) \
1105- and not os.path.isfile(self.bc_path2))
1106+ self.assertTrue(os.path.isfile(self.bc_path) and
1107+ not os.path.isfile(self.bc_path2))
1108 os.unlink(self.bc_path)
1109 compileall.compile_dir(self.directory, force=False, quiet=True)
1110- self.assertTrue(os.path.isfile(self.bc_path) \
1111- and os.path.isfile(self.bc_path2))
1112+ self.assertTrue(os.path.isfile(self.bc_path) and
1113+ os.path.isfile(self.bc_path2))
1114 os.unlink(self.bc_path)
1115 os.unlink(self.bc_path2)
1116
1117+
1118 class EncodingTest(unittest.TestCase):
1119- 'Issue 6716: compileall should escape source code when printing errors to stdout.'
1120+ """Issue 6716: compileall should escape source code when printing errors
1121+ to stdout."""
1122
1123 def setUp(self):
1124 self.directory = tempfile.mkdtemp()
1125@@ -95,9 +98,65 @@
1126 finally:
1127 sys.stdout = orig_stdout
1128
1129+class CommandLineTests(unittest.TestCase):
1130+ """Test some aspects of compileall's CLI."""
1131+
1132+ def setUp(self):
1133+ self.addCleanup(self._cleanup)
1134+ self.directory = tempfile.mkdtemp()
1135+ self.pkgdir = os.path.join(self.directory, 'foo')
1136+ os.mkdir(self.pkgdir)
1137+ # Touch the __init__.py and a package module.
1138+ with open(os.path.join(self.pkgdir, '__init__.py'), 'w'):
1139+ pass
1140+ with open(os.path.join(self.pkgdir, 'bar.py'), 'w'):
1141+ pass
1142+ sys.path.insert(0, self.directory)
1143+
1144+ def _cleanup(self):
1145+ support.rmtree(self.directory)
1146+ assert sys.path[0] == self.directory, 'Missing path'
1147+ del sys.path[0]
1148+
1149+ def test_pep3147_paths(self):
1150+ # Ensure that the default behavior of compileall's CLI is to create
1151+ # PEP 3147 pyc/pyo files.
1152+ retcode = subprocess.call(
1153+ (sys.executable, '-m', 'compileall', '-q', self.pkgdir))
1154+ self.assertEqual(retcode, 0)
1155+ # Verify the __pycache__ directory contents.
1156+ cachedir = os.path.join(self.pkgdir, '__pycache__')
1157+ self.assertTrue(os.path.exists(cachedir))
1158+ ext = ('pyc' if __debug__ else 'pyo')
1159+ expected = sorted(base.format(imp.get_tag(), ext) for base in
1160+ ('__init__.{}.{}', 'bar.{}.{}'))
1161+ self.assertEqual(sorted(os.listdir(cachedir)), expected)
1162+ # Make sure there are no .pyc files in the source directory.
1163+ self.assertFalse([pyc_file for pyc_file in os.listdir(self.pkgdir)
1164+ if pyc_file.endswith(ext)])
1165+
1166+ def test_legacy_paths(self):
1167+ # Ensure that with the proper switch, compileall leaves legacy
1168+ # pyc/pyo files, and no __pycache__ directory.
1169+ retcode = subprocess.call(
1170+ (sys.executable, '-m', 'compileall', '-b', '-q', self.pkgdir))
1171+ self.assertEqual(retcode, 0)
1172+ # Verify the __pycache__ directory contents.
1173+ cachedir = os.path.join(self.pkgdir, '__pycache__')
1174+ self.assertFalse(os.path.exists(cachedir))
1175+ ext = ('pyc' if __debug__ else 'pyo')
1176+ expected = [base.format(ext) for base in ('__init__.{}', 'bar.{}')]
1177+ expected.extend(['__init__.py', 'bar.py'])
1178+ expected.sort()
1179+ self.assertEqual(sorted(os.listdir(self.pkgdir)), expected)
1180+
1181+
1182 def test_main():
1183- support.run_unittest(CompileallTests,
1184- EncodingTest)
1185+ support.run_unittest(
1186+ CommandLineTests,
1187+ CompileallTests,
1188+ EncodingTest,
1189+ )
1190
1191
1192 if __name__ == "__main__":
1193
1194=== modified file 'Lib/test/test_frozen.py'
1195--- Lib/test/test_frozen.py 2010-03-14 10:45:50 +0000
1196+++ Lib/test/test_frozen.py 2010-04-16 18:55:41 +0000
1197@@ -11,7 +11,7 @@
1198 except ImportError as x:
1199 self.fail("import __hello__ failed:" + str(x))
1200 self.assertEqual(__hello__.initialized, True)
1201- self.assertEqual(len(dir(__hello__)), 6, dir(__hello__))
1202+ self.assertEqual(len(dir(__hello__)), 7, dir(__hello__))
1203
1204 try:
1205 import __phello__
1206@@ -19,9 +19,9 @@
1207 self.fail("import __phello__ failed:" + str(x))
1208 self.assertEqual(__phello__.initialized, True)
1209 if not "__phello__.spam" in sys.modules:
1210- self.assertEqual(len(dir(__phello__)), 7, dir(__phello__))
1211- else:
1212 self.assertEqual(len(dir(__phello__)), 8, dir(__phello__))
1213+ else:
1214+ self.assertEqual(len(dir(__phello__)), 9, dir(__phello__))
1215 self.assertEquals(__phello__.__path__, [__phello__.__name__])
1216
1217 try:
1218@@ -29,8 +29,8 @@
1219 except ImportError as x:
1220 self.fail("import __phello__.spam failed:" + str(x))
1221 self.assertEqual(__phello__.spam.initialized, True)
1222- self.assertEqual(len(dir(__phello__.spam)), 6)
1223- self.assertEqual(len(dir(__phello__)), 8)
1224+ self.assertEqual(len(dir(__phello__.spam)), 7)
1225+ self.assertEqual(len(dir(__phello__)), 9)
1226
1227 try:
1228 import __phello__.foo
1229
1230=== modified file 'Lib/test/test_imp.py'
1231--- Lib/test/test_imp.py 2010-03-20 20:31:34 +0000
1232+++ Lib/test/test_imp.py 2010-04-16 18:55:41 +0000
1233@@ -1,6 +1,7 @@
1234 import imp
1235 import os
1236 import os.path
1237+import shutil
1238 import sys
1239 import unittest
1240 from test import support
1241@@ -139,7 +140,8 @@
1242 mod = imp.load_source(temp_mod_name, temp_mod_name + '.py')
1243 self.assertEqual(mod.a, 1)
1244
1245- mod = imp.load_compiled(temp_mod_name, temp_mod_name + '.pyc')
1246+ mod = imp.load_compiled(
1247+ temp_mod_name, imp.cache_from_source(temp_mod_name + '.py'))
1248 self.assertEqual(mod.a, 1)
1249
1250 if not os.path.exists(test_package_name):
1251@@ -184,11 +186,132 @@
1252 imp.reload(marshal)
1253
1254
1255+class PEP3147Tests(unittest.TestCase):
1256+ """Tests of PEP 3147."""
1257+
1258+ tag = imp.get_tag()
1259+
1260+ def test_cache_from_source(self):
1261+ # Given the path to a .py file, return the path to its PEP 3147
1262+ # defined .pyc file (i.e. under __pycache__).
1263+ self.assertEqual(
1264+ imp.cache_from_source('/foo/bar/baz/qux.py', True),
1265+ '/foo/bar/baz/__pycache__/qux.{}.pyc'.format(self.tag))
1266+
1267+ def test_cache_from_source_optimized(self):
1268+ # Given the path to a .py file, return the path to its PEP 3147
1269+ # defined .pyo file (i.e. under __pycache__).
1270+ self.assertEqual(
1271+ imp.cache_from_source('/foo/bar/baz/qux.py', False),
1272+ '/foo/bar/baz/__pycache__/qux.{}.pyo'.format(self.tag))
1273+
1274+ def test_cache_from_source_cwd(self):
1275+ self.assertEqual(imp.cache_from_source('foo.py', True),
1276+ os.sep.join(('__pycache__',
1277+ 'foo.{}.pyc'.format(self.tag))))
1278+
1279+ def test_cache_from_source_override(self):
1280+ # When debug_override is not None, it can be any true-ish or false-ish
1281+ # value.
1282+ self.assertEqual(
1283+ imp.cache_from_source('/foo/bar/baz.py', []),
1284+ '/foo/bar/__pycache__/baz.{}.pyo'.format(self.tag))
1285+ self.assertEqual(
1286+ imp.cache_from_source('/foo/bar/baz.py', [17]),
1287+ '/foo/bar/__pycache__/baz.{}.pyc'.format(self.tag))
1288+ # However if the bool-ishness can't be determined, the exception
1289+ # propagates.
1290+ class Bearish:
1291+ def __bool__(self): raise RuntimeError
1292+ self.assertRaises(
1293+ RuntimeError,
1294+ imp.cache_from_source, '/foo/bar/baz.py', Bearish())
1295+
1296+ @unittest.skipIf(os.altsep is None,
1297+ 'test meaningful only where os.altsep is defined')
1298+ def test_altsep_cache_from_source(self):
1299+ # Windows path and PEP 3147.
1300+ self.assertEqual(
1301+ imp.cache_from_source('\\foo\\bar\\baz\\qux.py', True),
1302+ '\\foo\\bar\\baz\\__pycache__\\qux.{}.pyc'.format(self.tag))
1303+
1304+ @unittest.skipIf(os.altsep is None,
1305+ 'test meaningful only where os.altsep is defined')
1306+ def test_altsep_and_sep_cache_from_source(self):
1307+ # Windows path and PEP 3147 where altsep is right of sep.
1308+ self.assertEqual(
1309+ imp.cache_from_source('\\foo\\bar/baz\\qux.py', True),
1310+ '\\foo\\bar/baz\\__pycache__\\qux.{}.pyc'.format(self.tag))
1311+
1312+ @unittest.skipIf(os.altsep is None,
1313+ 'test meaningful only where os.altsep is defined')
1314+ def test_sep_altsep_and_sep_cache_from_source(self):
1315+ # Windows path and PEP 3147 where sep is right of altsep.
1316+ self.assertEqual(
1317+ imp.cache_from_source('\\foo\\bar\\baz/qux.py', True),
1318+ '\\foo\\bar\\baz/__pycache__/qux.{}.pyc'.format(self.tag))
1319+
1320+ def test_source_from_cache(self):
1321+ # Given the path to a PEP 3147 defined .pyc file, return the path to
1322+ # its source. This tests the good path.
1323+ self.assertEqual(imp.source_from_cache(
1324+ '/foo/bar/baz/__pycache__/qux.{}.pyc'.format(self.tag)),
1325+ '/foo/bar/baz/qux.py')
1326+
1327+ def test_source_from_cache_bad_path(self):
1328+ # When the path to a pyc file is not in PEP 3147 format, a ValueError
1329+ # is raised.
1330+ self.assertRaises(
1331+ ValueError, imp.source_from_cache, '/foo/bar/bazqux.pyc')
1332+
1333+ def test_source_from_cache_no_slash(self):
1334+ # No slashes at all in path -> ValueError
1335+ self.assertRaises(
1336+ ValueError, imp.source_from_cache, 'foo.cpython-32.pyc')
1337+
1338+ def test_source_from_cache_too_few_dots(self):
1339+ # Too few dots in final path component -> ValueError
1340+ self.assertRaises(
1341+ ValueError, imp.source_from_cache, '__pycache__/foo.pyc')
1342+
1343+ def test_source_from_cache_too_many_dots(self):
1344+ # Too many dots in final path component -> ValueError
1345+ self.assertRaises(
1346+ ValueError, imp.source_from_cache,
1347+ '__pycache__/foo.cpython-32.foo.pyc')
1348+
1349+ def test_source_from_cache_no__pycache__(self):
1350+ # Another problem with the path -> ValueError
1351+ self.assertRaises(
1352+ ValueError, imp.source_from_cache,
1353+ '/foo/bar/foo.cpython-32.foo.pyc')
1354+
1355+ def test_package___file__(self):
1356+ # Test that a package's __file__ points to the right source directory.
1357+ os.mkdir('pep3147')
1358+ sys.path.insert(0, os.curdir)
1359+ def cleanup():
1360+ if sys.path[0] == os.curdir:
1361+ del sys.path[0]
1362+ shutil.rmtree('pep3147')
1363+ self.addCleanup(cleanup)
1364+ # Touch the __init__.py file.
1365+ with open('pep3147/__init__.py', 'w'):
1366+ pass
1367+ m = __import__('pep3147')
1368+ # Ensure we load the pyc file.
1369+ support.forget('pep3147')
1370+ m = __import__('pep3147')
1371+ self.assertEqual(m.__file__,
1372+ os.sep.join(('.', 'pep3147', '__init__.py')))
1373+
1374+
1375 def test_main():
1376 tests = [
1377 ImportTests,
1378+ PEP3147Tests,
1379 ReloadTests,
1380- ]
1381+ ]
1382 try:
1383 import _thread
1384 except ImportError:
1385
1386=== modified file 'Lib/test/test_import.py'
1387--- Lib/test/test_import.py 2010-03-17 20:29:51 +0000
1388+++ Lib/test/test_import.py 2010-04-16 18:55:41 +0000
1389@@ -1,4 +1,5 @@
1390 import builtins
1391+import errno
1392 import imp
1393 import marshal
1394 import os
1395@@ -8,8 +9,11 @@
1396 import stat
1397 import sys
1398 import unittest
1399-from test.support import (unlink, TESTFN, unload, run_unittest, is_jython,
1400- check_warnings, EnvironmentVarGuard, swap_attr, swap_item)
1401+
1402+from test.support import (
1403+ EnvironmentVarGuard, TESTFN, check_warnings, forget, is_jython,
1404+ make_legacy_pyc, rmtree, run_unittest, swap_attr, swap_item, temp_umask,
1405+ unlink, unload)
1406
1407
1408 def remove_files(name):
1409@@ -19,12 +23,18 @@
1410 name + ".pyw",
1411 name + "$py.class"):
1412 unlink(f)
1413+ try:
1414+ shutil.rmtree('__pycache__')
1415+ except OSError as error:
1416+ if error.errno != errno.ENOENT:
1417+ raise
1418
1419
1420 class ImportTests(unittest.TestCase):
1421
1422 def tearDown(self):
1423 unload(TESTFN)
1424+
1425 setUp = tearDown
1426
1427 def test_case_sensitivity(self):
1428@@ -53,8 +63,8 @@
1429 pyc = TESTFN + ".pyc"
1430
1431 with open(source, "w") as f:
1432- print("# This tests Python's ability to import a", ext, "file.",
1433- file=f)
1434+ print("# This tests Python's ability to import a",
1435+ ext, "file.", file=f)
1436 a = random.randrange(1000)
1437 b = random.randrange(1000)
1438 print("a =", a, file=f)
1439@@ -73,10 +83,10 @@
1440 self.assertEqual(mod.b, b,
1441 "module loaded (%s) but contents invalid" % mod)
1442 finally:
1443+ forget(TESTFN)
1444 unlink(source)
1445 unlink(pyc)
1446 unlink(pyo)
1447- unload(TESTFN)
1448
1449 sys.path.insert(0, os.curdir)
1450 try:
1451@@ -87,32 +97,31 @@
1452 finally:
1453 del sys.path[0]
1454
1455- @unittest.skipUnless(os.name == 'posix', "test meaningful only on posix systems")
1456+ @unittest.skipUnless(os.name == 'posix',
1457+ "test meaningful only on posix systems")
1458 def test_execute_bit_not_copied(self):
1459 # Issue 6070: under posix .pyc files got their execute bit set if
1460 # the .py file had the execute bit set, but they aren't executable.
1461- oldmask = os.umask(0o022)
1462- sys.path.insert(0, os.curdir)
1463- try:
1464- fname = TESTFN + os.extsep + "py"
1465- f = open(fname, 'w').close()
1466- os.chmod(fname, (stat.S_IRUSR | stat.S_IRGRP | stat.S_IROTH |
1467- stat.S_IXUSR | stat.S_IXGRP | stat.S_IXOTH))
1468- __import__(TESTFN)
1469- fn = fname + 'c'
1470- if not os.path.exists(fn):
1471- fn = fname + 'o'
1472+ with temp_umask(0o022):
1473+ sys.path.insert(0, os.curdir)
1474+ try:
1475+ fname = TESTFN + os.extsep + "py"
1476+ f = open(fname, 'w').close()
1477+ os.chmod(fname, (stat.S_IRUSR | stat.S_IRGRP | stat.S_IROTH |
1478+ stat.S_IXUSR | stat.S_IXGRP | stat.S_IXOTH))
1479+ __import__(TESTFN)
1480+ fn = imp.cache_from_source(fname)
1481 if not os.path.exists(fn):
1482 self.fail("__import__ did not result in creation of "
1483 "either a .pyc or .pyo file")
1484- s = os.stat(fn)
1485- self.assertEqual(stat.S_IMODE(s.st_mode),
1486- stat.S_IRUSR | stat.S_IRGRP | stat.S_IROTH)
1487- finally:
1488- os.umask(oldmask)
1489- remove_files(TESTFN)
1490- unload(TESTFN)
1491- del sys.path[0]
1492+ s = os.stat(fn)
1493+ self.assertEqual(
1494+ stat.S_IMODE(s.st_mode),
1495+ stat.S_IRUSR | stat.S_IRGRP | stat.S_IROTH)
1496+ finally:
1497+ del sys.path[0]
1498+ remove_files(TESTFN)
1499+ unload(TESTFN)
1500
1501 def test_imp_module(self):
1502 # Verify that the imp module can correctly load and find .py files
1503@@ -144,10 +153,12 @@
1504 f.write('"",\n')
1505 f.write(']')
1506
1507- # Compile & remove .py file, we only need .pyc (or .pyo).
1508+ # Compile & remove .py file, we only need .pyc (or .pyo), but that
1509+ # must be relocated to the PEP 3147 bytecode-only location.
1510 with open(filename, 'r') as f:
1511 py_compile.compile(filename)
1512 unlink(filename)
1513+ make_legacy_pyc(filename)
1514
1515 # Need to be able to load from current dir.
1516 sys.path.append('')
1517@@ -247,8 +258,9 @@
1518 self.assertTrue(mod.__file__.endswith('.py'))
1519 os.remove(source)
1520 del sys.modules[TESTFN]
1521+ make_legacy_pyc(source)
1522 mod = __import__(TESTFN)
1523- ext = mod.__file__[-4:]
1524+ base, ext = os.path.splitext(mod.__file__)
1525 self.assertIn(ext, ('.pyc', '.pyo'))
1526 finally:
1527 del sys.path[0]
1528@@ -298,7 +310,7 @@
1529 """
1530 dir_name = os.path.abspath(TESTFN)
1531 file_name = os.path.join(dir_name, module_name) + os.extsep + "py"
1532- compiled_name = file_name + ("c" if __debug__ else "o")
1533+ compiled_name = imp.cache_from_source(file_name)
1534
1535 def setUp(self):
1536 self.sys_path = sys.path[:]
1537@@ -346,8 +358,9 @@
1538 target = "another_module.py"
1539 py_compile.compile(self.file_name, dfile=target)
1540 os.remove(self.file_name)
1541+ pyc_file = make_legacy_pyc(self.file_name)
1542 mod = self.import_module()
1543- self.assertEqual(mod.module_filename, self.compiled_name)
1544+ self.assertEqual(mod.module_filename, pyc_file)
1545 self.assertEqual(mod.code_filename, target)
1546 self.assertEqual(mod.func_filename, target)
1547
1548@@ -476,10 +489,143 @@
1549 self.assertEqual(foo(), os)
1550
1551
1552+class PycacheTests(unittest.TestCase):
1553+ # Test the various PEP 3147 related behaviors.
1554+
1555+ tag = imp.get_tag()
1556+
1557+ def _clean(self):
1558+ forget(TESTFN)
1559+ rmtree('__pycache__')
1560+ unlink(self.source)
1561+
1562+ def setUp(self):
1563+ self.source = TESTFN + '.py'
1564+ self._clean()
1565+ with open(self.source, 'w') as fp:
1566+ print('# This is a test file written by test_import.py', file=fp)
1567+ sys.path.insert(0, os.curdir)
1568+
1569+ def tearDown(self):
1570+ assert sys.path[0] == os.curdir, 'Unexpected sys.path[0]'
1571+ del sys.path[0]
1572+ self._clean()
1573+
1574+ def test_import_pyc_path(self):
1575+ self.assertFalse(os.path.exists('__pycache__'))
1576+ __import__(TESTFN)
1577+ self.assertTrue(os.path.exists('__pycache__'))
1578+ self.assertTrue(os.path.exists(os.path.join(
1579+ '__pycache__', '{}.{}.pyc'.format(TESTFN, self.tag))))
1580+
1581+ @unittest.skipUnless(os.name == 'posix',
1582+ "test meaningful only on posix systems")
1583+ def test_unwritable_directory(self):
1584+ # When the umask causes the new __pycache__ directory to be
1585+ # unwritable, the import still succeeds but no .pyc file is written.
1586+ with temp_umask(0o222):
1587+ __import__(TESTFN)
1588+ self.assertTrue(os.path.exists('__pycache__'))
1589+ self.assertFalse(os.path.exists(os.path.join(
1590+ '__pycache__', '{}.{}.pyc'.format(TESTFN, self.tag))))
1591+
1592+ def test_missing_source(self):
1593+ # With PEP 3147 cache layout, removing the source but leaving the pyc
1594+ # file does not satisfy the import.
1595+ __import__(TESTFN)
1596+ pyc_file = imp.cache_from_source(self.source)
1597+ self.assertTrue(os.path.exists(pyc_file))
1598+ os.remove(self.source)
1599+ forget(TESTFN)
1600+ self.assertRaises(ImportError, __import__, TESTFN)
1601+
1602+ def test_missing_source_legacy(self):
1603+ # Like test_missing_source() except that for backward compatibility,
1604+ # when the pyc file lives where the py file would have been (and named
1605+ # without the tag), it is importable. The __file__ of the imported
1606+ # module is the pyc location.
1607+ __import__(TESTFN)
1608+ # pyc_file gets removed in _clean() via tearDown().
1609+ pyc_file = make_legacy_pyc(self.source)
1610+ os.remove(self.source)
1611+ unload(TESTFN)
1612+ m = __import__(TESTFN)
1613+ self.assertEqual(m.__file__,
1614+ os.path.join(os.curdir, os.path.relpath(pyc_file)))
1615+
1616+ def test___cached__(self):
1617+ # Modules now also have an __cached__ that points to the pyc file.
1618+ m = __import__(TESTFN)
1619+ pyc_file = imp.cache_from_source(TESTFN + '.py')
1620+ self.assertEqual(m.__cached__, os.path.join(os.curdir, pyc_file))
1621+
1622+ def test___cached___legacy_pyc(self):
1623+ # Like test___cached__() except that for backward compatibility,
1624+ # when the pyc file lives where the py file would have been (and named
1625+ # without the tag), it is importable. The __cached__ of the imported
1626+ # module is the pyc location.
1627+ __import__(TESTFN)
1628+ # pyc_file gets removed in _clean() via tearDown().
1629+ pyc_file = make_legacy_pyc(self.source)
1630+ os.remove(self.source)
1631+ unload(TESTFN)
1632+ m = __import__(TESTFN)
1633+ self.assertEqual(m.__cached__,
1634+ os.path.join(os.curdir, os.path.relpath(pyc_file)))
1635+
1636+ def test_package___cached__(self):
1637+ # Like test___cached__ but for packages.
1638+ def cleanup():
1639+ shutil.rmtree('pep3147')
1640+ os.mkdir('pep3147')
1641+ self.addCleanup(cleanup)
1642+ # Touch the __init__.py
1643+ with open(os.path.join('pep3147', '__init__.py'), 'w'):
1644+ pass
1645+ with open(os.path.join('pep3147', 'foo.py'), 'w'):
1646+ pass
1647+ unload('pep3147.foo')
1648+ unload('pep3147')
1649+ m = __import__('pep3147.foo')
1650+ init_pyc = imp.cache_from_source(
1651+ os.path.join('pep3147', '__init__.py'))
1652+ self.assertEqual(m.__cached__, os.path.join(os.curdir, init_pyc))
1653+ foo_pyc = imp.cache_from_source(os.path.join('pep3147', 'foo.py'))
1654+ self.assertEqual(sys.modules['pep3147.foo'].__cached__,
1655+ os.path.join(os.curdir, foo_pyc))
1656+
1657+ def test_package___cached___from_pyc(self):
1658+ # Like test___cached__ but ensuring __cached__ when imported from a
1659+ # PEP 3147 pyc file.
1660+ def cleanup():
1661+ shutil.rmtree('pep3147')
1662+ os.mkdir('pep3147')
1663+ self.addCleanup(cleanup)
1664+ unload('pep3147.foo')
1665+ unload('pep3147')
1666+ # Touch the __init__.py
1667+ with open(os.path.join('pep3147', '__init__.py'), 'w'):
1668+ pass
1669+ with open(os.path.join('pep3147', 'foo.py'), 'w'):
1670+ pass
1671+ m = __import__('pep3147.foo')
1672+ unload('pep3147.foo')
1673+ unload('pep3147')
1674+ m = __import__('pep3147.foo')
1675+ init_pyc = imp.cache_from_source(
1676+ os.path.join('pep3147', '__init__.py'))
1677+ self.assertEqual(m.__cached__, os.path.join(os.curdir, init_pyc))
1678+ foo_pyc = imp.cache_from_source(os.path.join('pep3147', 'foo.py'))
1679+ self.assertEqual(sys.modules['pep3147.foo'].__cached__,
1680+ os.path.join(os.curdir, foo_pyc))
1681+
1682+
1683 def test_main(verbose=None):
1684- run_unittest(ImportTests, PycRewritingTests, PathsTests, RelativeImportTests,
1685+ run_unittest(ImportTests, PycacheTests,
1686+ PycRewritingTests, PathsTests, RelativeImportTests,
1687 OverridingImportBuiltinTests)
1688
1689+
1690 if __name__ == '__main__':
1691 # Test needs to be a package, so we can do relative imports.
1692 from test.test_import import test_main
1693
1694=== modified file 'Lib/test/test_pkg.py'
1695--- Lib/test/test_pkg.py 2010-02-24 01:46:21 +0000
1696+++ Lib/test/test_pkg.py 2010-04-16 18:55:41 +0000
1697@@ -196,14 +196,14 @@
1698
1699 import t5
1700 self.assertEqual(fixdir(dir(t5)),
1701- ['__doc__', '__file__', '__name__',
1702+ ['__cached__', '__doc__', '__file__', '__name__',
1703 '__package__', '__path__', 'foo', 'string', 't5'])
1704 self.assertEqual(fixdir(dir(t5.foo)),
1705- ['__doc__', '__file__', '__name__', '__package__',
1706- 'string'])
1707+ ['__cached__', '__doc__', '__file__', '__name__',
1708+ '__package__', 'string'])
1709 self.assertEqual(fixdir(dir(t5.string)),
1710- ['__doc__', '__file__', '__name__','__package__',
1711- 'spam'])
1712+ ['__cached__', '__doc__', '__file__', '__name__',
1713+ '__package__', 'spam'])
1714
1715 def test_6(self):
1716 hier = [
1717@@ -218,13 +218,13 @@
1718
1719 import t6
1720 self.assertEqual(fixdir(dir(t6)),
1721- ['__all__', '__doc__', '__file__',
1722+ ['__all__', '__cached__', '__doc__', '__file__',
1723 '__name__', '__package__', '__path__'])
1724 s = """
1725 import t6
1726 from t6 import *
1727 self.assertEqual(fixdir(dir(t6)),
1728- ['__all__', '__doc__', '__file__',
1729+ ['__all__', '__cached__', '__doc__', '__file__',
1730 '__name__', '__package__', '__path__',
1731 'eggs', 'ham', 'spam'])
1732 self.assertEqual(dir(), ['eggs', 'ham', 'self', 'spam', 't6'])
1733@@ -252,18 +252,18 @@
1734 t7, sub, subsub = None, None, None
1735 import t7 as tas
1736 self.assertEqual(fixdir(dir(tas)),
1737- ['__doc__', '__file__', '__name__',
1738+ ['__cached__', '__doc__', '__file__', '__name__',
1739 '__package__', '__path__'])
1740 self.assertFalse(t7)
1741 from t7 import sub as subpar
1742 self.assertEqual(fixdir(dir(subpar)),
1743- ['__doc__', '__file__', '__name__',
1744+ ['__cached__', '__doc__', '__file__', '__name__',
1745 '__package__', '__path__'])
1746 self.assertFalse(t7)
1747 self.assertFalse(sub)
1748 from t7.sub import subsub as subsubsub
1749 self.assertEqual(fixdir(dir(subsubsub)),
1750- ['__doc__', '__file__', '__name__',
1751+ ['__cached__', '__doc__', '__file__', '__name__',
1752 '__package__', '__path__', 'spam'])
1753 self.assertFalse(t7)
1754 self.assertFalse(sub)
1755
1756=== modified file 'Lib/test/test_pkgimport.py'
1757--- Lib/test/test_pkgimport.py 2010-01-23 15:40:09 +0000
1758+++ Lib/test/test_pkgimport.py 2010-04-16 18:55:41 +0000
1759@@ -1,5 +1,12 @@
1760-import os, sys, string, random, tempfile, unittest
1761+import os
1762+import sys
1763+import shutil
1764+import string
1765+import random
1766+import tempfile
1767+import unittest
1768
1769+from imp import cache_from_source
1770 from test.support import run_unittest
1771
1772 class TestImport(unittest.TestCase):
1773@@ -26,22 +33,17 @@
1774 self.module_path = os.path.join(self.package_dir, 'foo.py')
1775
1776 def tearDown(self):
1777- for file in os.listdir(self.package_dir):
1778- os.remove(os.path.join(self.package_dir, file))
1779- os.rmdir(self.package_dir)
1780- os.rmdir(self.test_dir)
1781+ shutil.rmtree(self.test_dir)
1782 self.assertNotEqual(sys.path.count(self.test_dir), 0)
1783 sys.path.remove(self.test_dir)
1784 self.remove_modules()
1785
1786 def rewrite_file(self, contents):
1787- for extension in "co":
1788- compiled_path = self.module_path + extension
1789- if os.path.exists(compiled_path):
1790- os.remove(compiled_path)
1791- f = open(self.module_path, 'w')
1792- f.write(contents)
1793- f.close()
1794+ compiled_path = cache_from_source(self.module_path)
1795+ if os.path.exists(compiled_path):
1796+ os.remove(compiled_path)
1797+ with open(self.module_path, 'w') as f:
1798+ f.write(contents)
1799
1800 def test_package_import__semantics(self):
1801
1802
1803=== modified file 'Lib/test/test_pydoc.py'
1804--- Lib/test/test_pydoc.py 2010-03-31 02:19:28 +0000
1805+++ Lib/test/test_pydoc.py 2010-04-16 18:55:41 +0000
1806@@ -19,8 +19,7 @@
1807 if hasattr(pydoc_mod, "__loader__"):
1808 del pydoc_mod.__loader__
1809
1810-expected_text_pattern = \
1811-"""
1812+expected_text_pattern = """
1813 NAME
1814 test.pydoc_mod - This is a test module for test_pydoc
1815
1816@@ -87,8 +86,7 @@
1817 Nobody
1818 """.strip()
1819
1820-expected_html_pattern = \
1821-"""
1822+expected_html_pattern = """
1823 <table width="100%%" cellspacing=0 cellpadding=2 border=0 summary="heading">
1824 <tr bgcolor="#7799ee">
1825 <td valign=bottom>&nbsp;<br>
1826@@ -186,7 +184,7 @@
1827 \x20\x20\x20\x20
1828 <tr><td bgcolor="#7799ee"><tt>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</tt></td><td>&nbsp;</td>
1829 <td width="100%%">Nobody</td></tr></table>
1830-""".strip()
1831+""".strip() # ' <- emacs turd
1832
1833
1834 # output pattern for missing module
1835@@ -287,7 +285,8 @@
1836 ('i_am_not_here', 'i_am_not_here'),
1837 ('test.i_am_not_here_either', 'i_am_not_here_either'),
1838 ('test.i_am_not_here.neither_am_i', 'i_am_not_here.neither_am_i'),
1839- ('i_am_not_here.{}'.format(modname), 'i_am_not_here.{}'.format(modname)),
1840+ ('i_am_not_here.{}'.format(modname),
1841+ 'i_am_not_here.{}'.format(modname)),
1842 ('test.{}'.format(modname), modname),
1843 )
1844
1845@@ -304,9 +303,8 @@
1846 fullmodname = os.path.join(TESTFN, modname)
1847 sourcefn = fullmodname + os.extsep + "py"
1848 for importstring, expectedinmsg in testpairs:
1849- f = open(sourcefn, 'w')
1850- f.write("import {}\n".format(importstring))
1851- f.close()
1852+ with open(sourcefn, 'w') as f:
1853+ f.write("import {}\n".format(importstring))
1854 try:
1855 result = run_pydoc(modname).decode("ascii")
1856 finally:
1857
1858=== modified file 'Lib/test/test_runpy.py'
1859--- Lib/test/test_runpy.py 2010-01-19 00:09:57 +0000
1860+++ Lib/test/test_runpy.py 2010-04-16 18:55:41 +0000
1861@@ -5,9 +5,10 @@
1862 import sys
1863 import re
1864 import tempfile
1865-from test.support import verbose, run_unittest, forget
1866-from test.script_helper import (temp_dir, make_script, compile_script,
1867- make_pkg, make_zip_script, make_zip_pkg)
1868+import py_compile
1869+from test.support import forget, make_legacy_pyc, run_unittest, verbose
1870+from test.script_helper import (
1871+ make_pkg, make_script, make_zip_pkg, make_zip_script, temp_dir)
1872
1873
1874 from runpy import _run_code, _run_module_code, run_module, run_path
1875@@ -45,6 +46,7 @@
1876 self.assertEqual(d["result"], self.expected_result)
1877 self.assertIs(d["__name__"], None)
1878 self.assertIs(d["__file__"], None)
1879+ self.assertIs(d["__cached__"], None)
1880 self.assertIs(d["__loader__"], None)
1881 self.assertIs(d["__package__"], None)
1882 self.assertIs(d["run_argv0"], saved_argv0)
1883@@ -73,6 +75,7 @@
1884 self.assertTrue(d2["run_name_in_sys_modules"])
1885 self.assertTrue(d2["module_in_sys_modules"])
1886 self.assertIs(d2["__file__"], file)
1887+ self.assertIs(d2["__cached__"], None)
1888 self.assertIs(d2["run_argv0"], file)
1889 self.assertIs(d2["__loader__"], loader)
1890 self.assertIs(d2["__package__"], package)
1891@@ -170,6 +173,7 @@
1892 del d1 # Ensure __loader__ entry doesn't keep file open
1893 __import__(mod_name)
1894 os.remove(mod_fname)
1895+ make_legacy_pyc(mod_fname)
1896 if verbose: print("Running from compiled:", mod_name)
1897 d2 = run_module(mod_name) # Read from bytecode
1898 self.assertIn("x", d2)
1899@@ -192,6 +196,7 @@
1900 del d1 # Ensure __loader__ entry doesn't keep file open
1901 __import__(mod_name)
1902 os.remove(mod_fname)
1903+ make_legacy_pyc(mod_fname)
1904 if verbose: print("Running from compiled:", pkg_name)
1905 d2 = run_module(pkg_name) # Read from bytecode
1906 self.assertIn("x", d2)
1907@@ -246,6 +251,7 @@
1908 del d1 # Ensure __loader__ entry doesn't keep file open
1909 __import__(mod_name)
1910 os.remove(mod_fname)
1911+ make_legacy_pyc(mod_fname)
1912 if verbose: print("Running from compiled:", mod_name)
1913 d2 = run_module(mod_name, run_name=run_name) # Read from bytecode
1914 self.assertIn("__package__", d2)
1915@@ -313,6 +319,7 @@
1916 result = run_path(script_name)
1917 self.assertEqual(result["__name__"], expected_name)
1918 self.assertEqual(result["__file__"], expected_file)
1919+ self.assertEqual(result["__cached__"], None)
1920 self.assertIn("argv0", result)
1921 self.assertEqual(result["argv0"], expected_argv0)
1922 self.assertEqual(result["__package__"], expected_package)
1923@@ -332,7 +339,7 @@
1924 with temp_dir() as script_dir:
1925 mod_name = 'script'
1926 script_name = self._make_test_script(script_dir, mod_name)
1927- compiled_name = compile_script(script_name)
1928+ compiled_name = py_compile.compile(script_name, doraise=True)
1929 os.remove(script_name)
1930 self._check_script(compiled_name, "<run_path>", compiled_name,
1931 compiled_name, None)
1932@@ -348,9 +355,10 @@
1933 with temp_dir() as script_dir:
1934 mod_name = '__main__'
1935 script_name = self._make_test_script(script_dir, mod_name)
1936- compiled_name = compile_script(script_name)
1937+ compiled_name = py_compile.compile(script_name, doraise=True)
1938 os.remove(script_name)
1939- self._check_script(script_dir, "<run_path>", compiled_name,
1940+ legacy_pyc = make_legacy_pyc(script_name)
1941+ self._check_script(script_dir, "<run_path>", legacy_pyc,
1942 script_dir, '')
1943
1944 def test_directory_error(self):
1945@@ -371,8 +379,9 @@
1946 with temp_dir() as script_dir:
1947 mod_name = '__main__'
1948 script_name = self._make_test_script(script_dir, mod_name)
1949- compiled_name = compile_script(script_name)
1950- zip_name, fname = make_zip_script(script_dir, 'test_zip', compiled_name)
1951+ compiled_name = py_compile.compile(script_name, doraise=True)
1952+ zip_name, fname = make_zip_script(script_dir, 'test_zip',
1953+ compiled_name)
1954 self._check_script(zip_name, "<run_path>", fname, zip_name, '')
1955
1956 def test_zipfile_error(self):
1957
1958=== modified file 'Lib/test/test_site.py'
1959--- Lib/test/test_site.py 2010-01-29 11:41:03 +0000
1960+++ Lib/test/test_site.py 2010-04-16 18:55:41 +0000
1961@@ -258,19 +258,38 @@
1962 """Restore sys.path"""
1963 sys.path[:] = self.sys_path
1964
1965- def test_abs__file__(self):
1966- # Make sure all imported modules have their __file__ attribute
1967- # as an absolute path.
1968- # Handled by abs__file__()
1969- site.abs__file__()
1970- for module in (sys, os, builtins):
1971- try:
1972- self.assertTrue(os.path.isabs(module.__file__), repr(module))
1973- except AttributeError:
1974- continue
1975- # We could try everything in sys.modules; however, when regrtest.py
1976- # runs something like test_frozen before test_site, then we will
1977- # be testing things loaded *after* test_site did path normalization
1978+ def test_abs_paths(self):
1979+ # Make sure all imported modules have their __file__ and __cached__
1980+ # attributes as absolute paths. Arranging to put the Lib directory on
1981+ # PYTHONPATH would cause the os module to have a relative path for
1982+ # __file__ if abs_paths() does not get run. sys and builtins (the
1983+ # only other modules imported before site.py runs) do not have
1984+ # __file__ or __cached__ because they are built-in.
1985+ parent = os.path.relpath(os.path.dirname(os.__file__))
1986+ env = os.environ.copy()
1987+ env['PYTHONPATH'] = parent
1988+ command = 'import os; print(os.__file__, os.__cached__)'
1989+ # First, prove that with -S (no 'import site'), the paths are
1990+ # relative.
1991+ proc = subprocess.Popen([sys.executable, '-S', '-c', command],
1992+ env=env,
1993+ stdout=subprocess.PIPE,
1994+ stderr=subprocess.PIPE)
1995+ stdout, stderr = proc.communicate()
1996+ self.assertEqual(proc.returncode, 0)
1997+ os__file__, os__cached__ = stdout.split()
1998+ self.assertFalse(os.path.isabs(os__file__))
1999+ self.assertFalse(os.path.isabs(os__cached__))
2000+ # Now, with 'import site', it works.
2001+ proc = subprocess.Popen([sys.executable, '-c', command],
2002+ env=env,
2003+ stdout=subprocess.PIPE,
2004+ stderr=subprocess.PIPE)
2005+ stdout, stderr = proc.communicate()
2006+ self.assertEqual(proc.returncode, 0)
2007+ os__file__, os__cached__ = stdout.split()
2008+ self.assertTrue(os.path.isabs(os__file__))
2009+ self.assertTrue(os.path.isabs(os__cached__))
2010
2011 def test_no_duplicate_paths(self):
2012 # No duplicate paths should exist in sys.path
2013
2014=== modified file 'Lib/test/test_zipfile.py'
2015--- Lib/test/test_zipfile.py 2010-02-07 20:24:02 +0000
2016+++ Lib/test/test_zipfile.py 2010-04-16 18:55:41 +0000
2017@@ -6,6 +6,7 @@
2018
2019 import io
2020 import os
2021+import imp
2022 import time
2023 import shutil
2024 import struct
2025@@ -587,7 +588,13 @@
2026 with zipfile.PyZipFile(TemporaryFile(), "w") as zipfp:
2027 fn = __file__
2028 if fn.endswith('.pyc') or fn.endswith('.pyo'):
2029- fn = fn[:-1]
2030+ path_split = fn.split(os.sep)
2031+ if os.altsep is not None:
2032+ path_split.extend(fn.split(os.altsep))
2033+ if '__pycache__' in path_split:
2034+ fn = imp.source_from_cache(fn)
2035+ else:
2036+ fn = fn[:-1]
2037
2038 zipfp.writepy(fn)
2039
2040
2041=== modified file 'Lib/test/test_zipimport.py'
2042--- Lib/test/test_zipimport.py 2010-03-14 10:23:39 +0000
2043+++ Lib/test/test_zipimport.py 2010-04-16 18:55:41 +0000
2044@@ -48,17 +48,14 @@
2045 test_pyc = make_pyc(test_co, NOW)
2046
2047
2048-if __debug__:
2049- pyc_ext = ".pyc"
2050-else:
2051- pyc_ext = ".pyo"
2052-
2053-
2054 TESTMOD = "ziptestmodule"
2055 TESTPACK = "ziptestpackage"
2056 TESTPACK2 = "ziptestpackage2"
2057 TEMP_ZIP = os.path.abspath("junk95142.zip")
2058
2059+pyc_file = imp.cache_from_source(TESTMOD + '.py')
2060+pyc_ext = ('.pyc' if __debug__ else '.pyo')
2061+
2062
2063 class UncompressedZipImportTestCase(ImportHooksBaseTestCase):
2064
2065@@ -83,14 +80,11 @@
2066 stuff = kw.get("stuff", None)
2067 if stuff is not None:
2068 # Prepend 'stuff' to the start of the zipfile
2069- f = open(TEMP_ZIP, "rb")
2070- data = f.read()
2071- f.close()
2072-
2073- f = open(TEMP_ZIP, "wb")
2074- f.write(stuff)
2075- f.write(data)
2076- f.close()
2077+ with open(TEMP_ZIP, "rb") as f:
2078+ data = f.read()
2079+ with open(TEMP_ZIP, "wb") as f:
2080+ f.write(stuff)
2081+ f.write(data)
2082
2083 sys.path.insert(0, TEMP_ZIP)
2084
2085@@ -180,8 +174,9 @@
2086
2087 def testBadMTime(self):
2088 badtime_pyc = bytearray(test_pyc)
2089- badtime_pyc[7] ^= 0x02 # flip the second bit -- not the first as that one
2090- # isn't stored in the .py's mtime in the zip archive.
2091+ # flip the second bit -- not the first as that one isn't stored in the
2092+ # .py's mtime in the zip archive.
2093+ badtime_pyc[7] ^= 0x02
2094 files = {TESTMOD + ".py": (NOW, test_src),
2095 TESTMOD + pyc_ext: (NOW, badtime_pyc)}
2096 self.doTest(".py", files, TESTMOD)
2097@@ -232,7 +227,8 @@
2098 self.assertEquals(zi.get_source(TESTPACK), None)
2099 self.assertEquals(zi.get_source(mod_path), None)
2100 self.assertEquals(zi.get_filename(mod_path), mod.__file__)
2101- # To pass in the module name instead of the path, we must use the right importer
2102+ # To pass in the module name instead of the path, we must use the
2103+ # right importer
2104 loader = mod.__loader__
2105 self.assertEquals(loader.get_source(mod_name), None)
2106 self.assertEquals(loader.get_filename(mod_name), mod.__file__)
2107@@ -266,8 +262,10 @@
2108 mod = zi.load_module(TESTPACK2)
2109 self.assertEquals(zi.get_filename(TESTPACK2), mod.__file__)
2110
2111- self.assertEquals(zi.is_package(TESTPACK2 + os.sep + '__init__'), False)
2112- self.assertEquals(zi.is_package(TESTPACK2 + os.sep + TESTMOD), False)
2113+ self.assertEquals(
2114+ zi.is_package(TESTPACK2 + os.sep + '__init__'), False)
2115+ self.assertEquals(
2116+ zi.is_package(TESTPACK2 + os.sep + TESTMOD), False)
2117
2118 mod_path = TESTPACK2 + os.sep + TESTMOD
2119 mod_name = module_path_to_dotted_name(mod_path)
2120@@ -276,7 +274,8 @@
2121 self.assertEquals(zi.get_source(TESTPACK2), None)
2122 self.assertEquals(zi.get_source(mod_path), None)
2123 self.assertEquals(zi.get_filename(mod_path), mod.__file__)
2124- # To pass in the module name instead of the path, we must use the right importer
2125+ # To pass in the module name instead of the path, we must use the
2126+ # right importer
2127 loader = mod.__loader__
2128 self.assertEquals(loader.get_source(mod_name), None)
2129 self.assertEquals(loader.get_filename(mod_name), mod.__file__)
2130
2131=== modified file 'Lib/zipfile.py'
2132--- Lib/zipfile.py 2010-02-07 20:24:02 +0000
2133+++ Lib/zipfile.py 2010-04-16 18:55:41 +0000
2134@@ -3,10 +3,17 @@
2135
2136 XXX references to utf-8 need further investigation.
2137 """
2138-import struct, os, time, sys, shutil
2139-import binascii, io, stat
2140 import io
2141+import os
2142 import re
2143+import imp
2144+import sys
2145+import time
2146+import stat
2147+import shutil
2148+import struct
2149+import binascii
2150+
2151
2152 try:
2153 import zlib # We may need its compression method
2154@@ -1303,22 +1310,42 @@
2155 file_py = pathname + ".py"
2156 file_pyc = pathname + ".pyc"
2157 file_pyo = pathname + ".pyo"
2158- if os.path.isfile(file_pyo) and \
2159- os.stat(file_pyo).st_mtime >= os.stat(file_py).st_mtime:
2160- fname = file_pyo # Use .pyo file
2161- elif not os.path.isfile(file_pyc) or \
2162- os.stat(file_pyc).st_mtime < os.stat(file_py).st_mtime:
2163+ pycache_pyc = imp.cache_from_source(file_py, True)
2164+ pycache_pyo = imp.cache_from_source(file_py, False)
2165+ if (os.path.isfile(file_pyo) and
2166+ os.stat(file_pyo).st_mtime >= os.stat(file_py).st_mtime):
2167+ # Use .pyo file.
2168+ arcname = fname = file_pyo
2169+ elif (os.path.isfile(file_pyc) and
2170+ os.stat(file_pyc).st_mtime >= os.stat(file_py).st_mtime):
2171+ # Use .pyc file.
2172+ arcname = fname = file_pyc
2173+ elif (os.path.isfile(pycache_pyc) and
2174+ os.stat(pycache_pyc).st_mtime >= os.stat(file_py).st_mtime):
2175+ # Use the __pycache__/*.pyc file, but write it to the legacy pyc
2176+ # file name in the archive.
2177+ fname = pycache_pyc
2178+ arcname = file_pyc
2179+ elif (os.path.isfile(pycache_pyo) and
2180+ os.stat(pycache_pyo).st_mtime >= os.stat(file_py).st_mtime):
2181+ # Use the __pycache__/*.pyo file, but write it to the legacy pyo
2182+ # file name in the archive.
2183+ fname = pycache_pyo
2184+ arcname = file_pyo
2185+ else:
2186+ # Compile py into PEP 3147 pyc file.
2187 import py_compile
2188 if self.debug:
2189 print("Compiling", file_py)
2190 try:
2191- py_compile.compile(file_py, file_pyc, None, True)
2192- except py_compile.PyCompileError as err:
2193+ py_compile.compile(file_py, doraise=True)
2194+ except py_compile.PyCompileError as error:
2195 print(err.msg)
2196- fname = file_pyc
2197- else:
2198- fname = file_pyc
2199- archivename = os.path.split(fname)[1]
2200+ fname = file_py
2201+ else:
2202+ fname = (pycache_pyc if __debug__ else pycache_pyo)
2203+ arcname = (file_pyc if __debug__ else file_pyo)
2204+ archivename = os.path.split(arcname)[1]
2205 if basename:
2206 archivename = "%s/%s" % (basename, archivename)
2207 return (fname, archivename)
2208
2209=== modified file 'Makefile.pre.in'
2210--- Makefile.pre.in 2010-04-11 23:16:50 +0000
2211+++ Makefile.pre.in 2010-04-16 18:55:41 +0000
2212@@ -1161,6 +1161,7 @@
2213 # files, which clobber removes as well
2214 pycremoval:
2215 find $(srcdir) -name '*.py[co]' -exec rm -f {} ';'
2216+ find $(srcdir) -name '__pycache__' | xargs rmdir
2217
2218 rmtestturds:
2219 -rm -f *BAD *GOOD *SKIPPED
2220
2221=== modified file 'Python/import.c'
2222--- Python/import.c 2010-03-25 23:30:20 +0000
2223+++ Python/import.c 2010-04-16 18:55:41 +0000
2224@@ -43,6 +43,15 @@
2225 The current working scheme is to increment the previous value by
2226 10.
2227
2228+ Starting with the adoption of PEP 3147 in Python 3.2, every bump in magic
2229+ number also includes a new "magic tag", i.e. a human readable string used
2230+ to represent the magic number in __pycache__ directories. When you change
2231+ the magic number, you must also set a new unique magic tag. Generally this
2232+ can be named after the Python major version of the magic number bump, but
2233+ it can really be anything, as long as it's different than anything else
2234+ that's come before. The tags are included in the following table, starting
2235+ with Python 3.2a0.
2236+
2237 Known values:
2238 Python 1.5: 20121
2239 Python 1.5.1: 20121
2240@@ -91,11 +100,18 @@
2241 Python 3.1a0: 3151 (optimize conditional branches:
2242 introduce POP_JUMP_IF_FALSE and POP_JUMP_IF_TRUE)
2243 Python 3.2a0: 3160 (add SETUP_WITH)
2244+ tag: cpython-32
2245 */
2246
2247+/* If you change MAGIC, you must change TAG and you must insert the old value
2248+ into _PyMagicNumberTags below.
2249+*/
2250 #define MAGIC (3160 | ((long)'\r'<<16) | ((long)'\n'<<24))
2251-/* Magic word as global */
2252+#define TAG "cpython-32"
2253+#define CACHEDIR "__pycache__"
2254+/* Current magic word and string tag as globals. */
2255 static long pyc_magic = MAGIC;
2256+static const char *pyc_tag = TAG;
2257
2258 /* See _PyImport_FixupExtension() below */
2259 static PyObject *extensions = NULL;
2260@@ -517,7 +533,7 @@
2261 }
2262
2263
2264-/* Helper for pythonrun.c -- return magic number */
2265+/* Helper for pythonrun.c -- return magic number and tag. */
2266
2267 long
2268 PyImport_GetMagicNumber(void)
2269@@ -526,6 +542,12 @@
2270 }
2271
2272
2273+const char *
2274+PyImport_GetMagicTag(void)
2275+{
2276+ return pyc_tag;
2277+}
2278+
2279 /* Magic for extension modules (built-in as well as dynamically
2280 loaded). To prevent initializing an extension module more than
2281 once, we keep a static dictionary 'extensions' keyed by module name
2282@@ -671,7 +693,10 @@
2283 "sys.modules failed");
2284 }
2285
2286-static PyObject * get_sourcefile(const char *file);
2287+static PyObject * get_sourcefile(char *file);
2288+static char *make_source_pathname(char *pathname, char *buf);
2289+static char *make_compiled_pathname(char *pathname, char *buf, size_t buflen,
2290+ int debug);
2291
2292 /* Execute a code object in a module and return the module object
2293 * WITH INCREMENTED REFERENCE COUNT. If an error occurs, name is
2294@@ -679,16 +704,28 @@
2295 * in sys.modules. The caller may wish to restore the original
2296 * module object (if any) in this case; PyImport_ReloadModule is an
2297 * example.
2298+ *
2299+ * Note that PyImport_ExecCodeModuleWithPathnames() is the preferred, richer
2300+ * interface. The other two exist primarily for backward compatibility.
2301 */
2302 PyObject *
2303 PyImport_ExecCodeModule(char *name, PyObject *co)
2304 {
2305- return PyImport_ExecCodeModuleEx(name, co, (char *)NULL);
2306+ return PyImport_ExecCodeModuleWithPathnames(
2307+ name, co, (char *)NULL, (char *)NULL);
2308 }
2309
2310 PyObject *
2311 PyImport_ExecCodeModuleEx(char *name, PyObject *co, char *pathname)
2312 {
2313+ return PyImport_ExecCodeModuleWithPathnames(
2314+ name, co, pathname, (char *)NULL);
2315+}
2316+
2317+PyObject *
2318+PyImport_ExecCodeModuleWithPathnames(char *name, PyObject *co, char *pathname,
2319+ char *cpathname)
2320+{
2321 PyObject *modules = PyImport_GetModuleDict();
2322 PyObject *m, *d, *v;
2323
2324@@ -718,6 +755,20 @@
2325 PyErr_Clear(); /* Not important enough to report */
2326 Py_DECREF(v);
2327
2328+ /* Remember the pyc path name as the __cached__ attribute. */
2329+ if (cpathname == NULL) {
2330+ v = Py_None;
2331+ Py_INCREF(v);
2332+ }
2333+ else if ((v = PyUnicode_FromString(cpathname)) == NULL) {
2334+ PyErr_Clear(); /* Not important enough to report */
2335+ v = Py_None;
2336+ Py_INCREF(v);
2337+ }
2338+ if (PyDict_SetItemString(d, "__cached__", v) != 0)
2339+ PyErr_Clear(); /* Not important enough to report */
2340+ Py_DECREF(v);
2341+
2342 v = PyEval_EvalCode((PyCodeObject *)co, d, d);
2343 if (v == NULL)
2344 goto error;
2345@@ -740,31 +791,188 @@
2346 }
2347
2348
2349+/* Like strrchr(string, '/') but searches for the rightmost of either SEP
2350+ or ALTSEP, if the latter is defined.
2351+*/
2352+static char *
2353+rightmost_sep(char *s)
2354+{
2355+ char *found, c;
2356+ for (found = NULL; (c = *s); s++) {
2357+ if (c == SEP
2358+#ifdef ALTSEP
2359+ || c == ALTSEP
2360+#endif
2361+ )
2362+ {
2363+ found = s;
2364+ }
2365+ }
2366+ return found;
2367+}
2368+
2369+
2370 /* Given a pathname for a Python source file, fill a buffer with the
2371 pathname for the corresponding compiled file. Return the pathname
2372 for the compiled file, or NULL if there's no space in the buffer.
2373 Doesn't set an exception. */
2374
2375 static char *
2376-make_compiled_pathname(char *pathname, char *buf, size_t buflen)
2377+make_compiled_pathname(char *pathname, char *buf, size_t buflen, int debug)
2378 {
2379+ /* foo.py -> __pycache__/foo.<tag>.pyc */
2380 size_t len = strlen(pathname);
2381- if (len+2 > buflen)
2382+ size_t i, save;
2383+ char *pos;
2384+ int sep = SEP;
2385+
2386+ /* Sanity check that the buffer has roughly enough space to hold what
2387+ will eventually be the full path to the compiled file. The 5 extra
2388+ bytes include the slash afer __pycache__, the two extra dots, the
2389+ extra trailing character ('c' or 'o') and null. This isn't exact
2390+ because the contents of the buffer can affect how many actual
2391+ characters of the string get into the buffer. We'll do a final
2392+ sanity check before writing the extension to ensure we do not
2393+ overflow the buffer.
2394+ */
2395+ if (len + strlen(CACHEDIR) + strlen(pyc_tag) + 5 > buflen)
2396 return NULL;
2397
2398-#ifdef MS_WINDOWS
2399- /* Treat .pyw as if it were .py. The case of ".pyw" must match
2400- that used in _PyImport_StandardFiletab. */
2401- if (len >= 4 && strcmp(&pathname[len-4], ".pyw") == 0)
2402- --len; /* pretend 'w' isn't there */
2403+ /* Find the last path separator and copy everything from the start of
2404+ the source string up to and including the separator.
2405+ */
2406+ if ((pos = rightmost_sep(pathname)) == NULL) {
2407+ i = 0;
2408+ }
2409+ else {
2410+ sep = *pos;
2411+ i = pos - pathname + 1;
2412+ strncpy(buf, pathname, i);
2413+ }
2414+
2415+ save = i;
2416+ buf[i++] = '\0';
2417+ /* Add __pycache__/ */
2418+ strcat(buf, CACHEDIR);
2419+ i += strlen(CACHEDIR) - 1;
2420+ buf[i++] = sep;
2421+ buf[i++] = '\0';
2422+ /* Add the base filename, but remove the .py or .pyw extension, since
2423+ the tag name must go before the extension.
2424+ */
2425+ strcat(buf, pathname + save);
2426+ if ((pos = strrchr(buf, '.')) != NULL)
2427+ *++pos = '\0';
2428+ strcat(buf, pyc_tag);
2429+ /* The length test above assumes that we're only adding one character
2430+ to the end of what would normally be the extension. What if there
2431+ is no extension, or the string ends in '.' or '.p', and otherwise
2432+ fills the buffer? By appending 4 more characters onto the string
2433+ here, we could overrun the buffer.
2434+
2435+ As a simple example, let's say buflen=32 and the input string is
2436+ 'xxx.py'. strlen() would be 6 and the test above would yield:
2437+
2438+ (6 + 11 + 10 + 5 == 32) > 32
2439+
2440+ which is false and so the name mangling would continue. This would
2441+ be fine because we'd end up with this string in buf:
2442+
2443+ __pycache__/xxx.cpython-32.pyc\0
2444+
2445+ strlen(of that) == 30 + the nul fits inside a 32 character buffer.
2446+ We can even handle an input string of say 'xxxxx' above because
2447+ that's (5 + 11 + 10 + 5 == 31) > 32 which is also false. Name
2448+ mangling that yields:
2449+
2450+ __pycache__/xxxxxcpython-32.pyc\0
2451+
2452+ which is 32 characters including the nul, and thus fits in the
2453+ buffer. However, an input string of 'xxxxxx' would yield a result
2454+ string of:
2455+
2456+ __pycache__/xxxxxxcpython-32.pyc\0
2457+
2458+ which is 33 characters long (including the nul), thus overflowing
2459+ the buffer, even though the first test would fail, i.e.: the input
2460+ string is also 6 characters long, so 32 > 32 is false.
2461+
2462+ The reason the first test fails but we still overflow the buffer is
2463+ that the test above only expects to add one extra character to be
2464+ added to the extension, and here we're adding three (pyc). We
2465+ don't add the first dot, so that reclaims one of expected
2466+ positions, leaving us overflowing by 1 byte (3 extra - 1 reclaimed
2467+ dot - 1 expected extra == 1 overflowed).
2468+
2469+ The best we can do is ensure that we still have enough room in the
2470+ target buffer before we write the extension. Because it's always
2471+ only the extension that can cause the overflow, and never the other
2472+ path bytes we've written, it's sufficient to just do one more test
2473+ here. Still, the assertion that follows can't hurt.
2474+ */
2475+#if 0
2476+ printf("strlen(buf): %d; buflen: %d\n", (int)strlen(buf), (int)buflen);
2477 #endif
2478- memcpy(buf, pathname, len);
2479- buf[len] = Py_OptimizeFlag ? 'o' : 'c';
2480- buf[len+1] = '\0';
2481-
2482- return buf;
2483-}
2484-
2485+ if (strlen(buf) + 5 > buflen)
2486+ return NULL;
2487+ strcat(buf, debug ? ".pyc" : ".pyo");
2488+ assert(strlen(buf) < buflen);
2489+ return buf;
2490+}
2491+
2492+
2493+/* Given a pathname to a Python byte compiled file, return the path to the
2494+ source file, if the path matches the PEP 3147 format. This does not check
2495+ for any file existence, however, if the pyc file name does not match PEP
2496+ 3147 style, NULL is returned. buf must be at least as big as pathname;
2497+ the resulting path will always be shorter. */
2498+
2499+static char *
2500+make_source_pathname(char *pathname, char *buf)
2501+{
2502+ /* __pycache__/foo.<tag>.pyc -> foo.py */
2503+ size_t i, j;
2504+ char *left, *right, *dot0, *dot1, sep;
2505+
2506+ /* Look back two slashes from the end. In between these two slashes
2507+ must be the string __pycache__ or this is not a PEP 3147 style
2508+ path. It's possible for there to be only one slash.
2509+ */
2510+ if ((right = rightmost_sep(pathname)) == NULL)
2511+ return NULL;
2512+ sep = *right;
2513+ *right = '\0';
2514+ left = rightmost_sep(pathname);
2515+ *right = sep;
2516+ if (left == NULL)
2517+ left = pathname;
2518+ else
2519+ left++;
2520+ if (right-left != strlen(CACHEDIR) ||
2521+ strncmp(left, CACHEDIR, right-left) != 0)
2522+ return NULL;
2523+
2524+ /* Now verify that the path component to the right of the last slash
2525+ has two dots in it.
2526+ */
2527+ if ((dot0 = strchr(right + 1, '.')) == NULL)
2528+ return NULL;
2529+ if ((dot1 = strchr(dot0 + 1, '.')) == NULL)
2530+ return NULL;
2531+ /* Too many dots? */
2532+ if (strchr(dot1 + 1, '.') != NULL)
2533+ return NULL;
2534+
2535+ /* This is a PEP 3147 path. Start by copying everything from the
2536+ start of pathname up to and including the leftmost slash. Then
2537+ copy the file's basename, removing the magic tag and adding a .py
2538+ suffix.
2539+ */
2540+ strncpy(buf, pathname, (i=left-pathname));
2541+ strncpy(buf+i, right+1, (j=dot0-right));
2542+ strcpy(buf+i+j, "py");
2543+ return buf;
2544+}
2545
2546 /* Given a pathname for a Python source file, its time of last
2547 modification, and a pathname for a compiled file, check whether the
2548@@ -846,7 +1054,8 @@
2549 if (Py_VerboseFlag)
2550 PySys_WriteStderr("import %s # precompiled from %s\n",
2551 name, cpathname);
2552- m = PyImport_ExecCodeModuleEx(name, (PyObject *)co, cpathname);
2553+ m = PyImport_ExecCodeModuleWithPathnames(
2554+ name, (PyObject *)co, cpathname, cpathname);
2555 Py_DECREF(co);
2556
2557 return m;
2558@@ -919,12 +1128,41 @@
2559 write_compiled_module(PyCodeObject *co, char *cpathname, struct stat *srcstat)
2560 {
2561 FILE *fp;
2562+ char *dirpath;
2563 time_t mtime = srcstat->st_mtime;
2564 #ifdef MS_WINDOWS /* since Windows uses different permissions */
2565 mode_t mode = srcstat->st_mode & ~S_IEXEC;
2566+ mode_t dirmode = srcstat->st_mode | S_IEXEC; /* XXX Is this correct
2567+ for Windows?
2568+ 2010-04-07 BAW */
2569 #else
2570 mode_t mode = srcstat->st_mode & ~S_IXUSR & ~S_IXGRP & ~S_IXOTH;
2571+ mode_t dirmode = (srcstat->st_mode |
2572+ S_IXUSR | S_IXGRP | S_IXOTH |
2573+ S_IWUSR | S_IWGRP | S_IWOTH);
2574 #endif
2575+ int saved;
2576+
2577+ /* Ensure that the __pycache__ directory exists. */
2578+ dirpath = rightmost_sep(cpathname);
2579+ if (dirpath == NULL) {
2580+ if (Py_VerboseFlag)
2581+ PySys_WriteStderr(
2582+ "# no %s path found %s\n",
2583+ CACHEDIR, cpathname);
2584+ return;
2585+ }
2586+ saved = *dirpath;
2587+ *dirpath = '\0';
2588+ /* XXX call os.mkdir() or maybe CreateDirectoryA() on Windows? */
2589+ if (mkdir(cpathname, dirmode) < 0 && errno != EEXIST) {
2590+ *dirpath = saved;
2591+ if (Py_VerboseFlag)
2592+ PySys_WriteStderr(
2593+ "# cannot create cache dir %s\n", cpathname);
2594+ return;
2595+ }
2596+ *dirpath = saved;
2597
2598 fp = open_exclusive(cpathname, mode);
2599 if (fp == NULL) {
2600@@ -1032,8 +1270,8 @@
2601 return NULL;
2602 }
2603 #endif
2604- cpathname = make_compiled_pathname(pathname, buf,
2605- (size_t)MAXPATHLEN + 1);
2606+ cpathname = make_compiled_pathname(
2607+ pathname, buf, (size_t)MAXPATHLEN + 1, !Py_OptimizeFlag);
2608 if (cpathname != NULL &&
2609 (fpc = check_compiled_module(pathname, st.st_mtime, cpathname))) {
2610 co = read_compiled_module(cpathname, fpc);
2611@@ -1060,7 +1298,8 @@
2612 write_compiled_module(co, cpathname, &st);
2613 }
2614 }
2615- m = PyImport_ExecCodeModuleEx(name, (PyObject *)co, pathname);
2616+ m = PyImport_ExecCodeModuleWithPathnames(
2617+ name, (PyObject *)co, pathname, cpathname);
2618 Py_DECREF(co);
2619
2620 return m;
2621@@ -1070,7 +1309,7 @@
2622 * Returns the path to the py file if available, else the given path
2623 */
2624 static PyObject *
2625-get_sourcefile(const char *file)
2626+get_sourcefile(char *file)
2627 {
2628 char py[MAXPATHLEN + 1];
2629 Py_ssize_t len;
2630@@ -1087,8 +1326,15 @@
2631 return PyUnicode_DecodeFSDefault(file);
2632 }
2633
2634- strncpy(py, file, len-1);
2635- py[len-1] = '\0';
2636+ /* Start by trying to turn PEP 3147 path into source path. If that
2637+ * fails, just chop off the trailing character, i.e. legacy pyc path
2638+ * to py.
2639+ */
2640+ if (make_source_pathname(file, py) == NULL) {
2641+ strncpy(py, file, len-1);
2642+ py[len-1] = '\0';
2643+ }
2644+
2645 if (stat(py, &statbuf) == 0 &&
2646 S_ISREG(statbuf.st_mode)) {
2647 u = PyUnicode_DecodeFSDefault(py);
2648@@ -2813,16 +3059,28 @@
2649 */
2650
2651 static PyObject *
2652-imp_get_magic(PyObject *self, PyObject *noargs)
2653+imp_make_magic(long magic)
2654 {
2655 char buf[4];
2656
2657- buf[0] = (char) ((pyc_magic >> 0) & 0xff);
2658- buf[1] = (char) ((pyc_magic >> 8) & 0xff);
2659- buf[2] = (char) ((pyc_magic >> 16) & 0xff);
2660- buf[3] = (char) ((pyc_magic >> 24) & 0xff);
2661+ buf[0] = (char) ((magic >> 0) & 0xff);
2662+ buf[1] = (char) ((magic >> 8) & 0xff);
2663+ buf[2] = (char) ((magic >> 16) & 0xff);
2664+ buf[3] = (char) ((magic >> 24) & 0xff);
2665
2666 return PyBytes_FromStringAndSize(buf, 4);
2667+};
2668+
2669+static PyObject *
2670+imp_get_magic(PyObject *self, PyObject *noargs)
2671+{
2672+ return imp_make_magic(pyc_magic);
2673+}
2674+
2675+static PyObject *
2676+imp_get_tag(PyObject *self, PyObject *noargs)
2677+{
2678+ return PyUnicode_FromString(pyc_tag);
2679 }
2680
2681 static PyObject *
2682@@ -3190,6 +3448,75 @@
2683 \n\
2684 Reload the module. The module must have been successfully imported before.");
2685
2686+static PyObject *
2687+imp_cache_from_source(PyObject *self, PyObject *args, PyObject *kws)
2688+{
2689+ static char *kwlist[] = {"path", "debug_override", NULL};
2690+
2691+ char buf[MAXPATHLEN+1];
2692+ char *pathname, *cpathname;
2693+ PyObject *debug_override = Py_None;
2694+ int debug = !Py_OptimizeFlag;
2695+
2696+ if (!PyArg_ParseTupleAndKeywords(
2697+ args, kws, "es|O", kwlist,
2698+ Py_FileSystemDefaultEncoding, &pathname, &debug_override))
2699+ return NULL;
2700+
2701+ if (debug_override != Py_None)
2702+ if ((debug = PyObject_IsTrue(debug_override)) < 0)
2703+ return NULL;
2704+
2705+ cpathname = make_compiled_pathname(pathname, buf, MAXPATHLEN+1, debug);
2706+ PyMem_Free(pathname);
2707+
2708+ if (cpathname == NULL) {
2709+ PyErr_Format(PyExc_SystemError, "path buffer too short");
2710+ return NULL;
2711+ }
2712+ return PyUnicode_FromString(buf);
2713+}
2714+
2715+PyDoc_STRVAR(doc_cache_from_source,
2716+"Given the path to a .py file, return the path to its .pyc/.pyo file.\n\
2717+\n\
2718+The .py file does not need to exist; this simply returns the path to the\n\
2719+.pyc/.pyo file calculated as if the .py file were imported. The extension\n\
2720+will be .pyc unless __debug__ is not defined, then it will be .pyo.\n\
2721+\n\
2722+If debug_override is not None, then it must be a boolean and is taken as\n\
2723+the value of __debug__ instead.");
2724+
2725+static PyObject *
2726+imp_source_from_cache(PyObject *self, PyObject *args, PyObject *kws)
2727+{
2728+ static char *kwlist[] = {"path", NULL};
2729+
2730+ char *pathname;
2731+ char buf[MAXPATHLEN+1];
2732+
2733+ if (!PyArg_ParseTupleAndKeywords(
2734+ args, kws, "es", kwlist,
2735+ Py_FileSystemDefaultEncoding, &pathname))
2736+ return NULL;
2737+
2738+ if (make_source_pathname(pathname, buf) == NULL) {
2739+ PyErr_Format(PyExc_ValueError, "Not a PEP 3147 pyc path: %s",
2740+ pathname);
2741+ PyMem_Free(pathname);
2742+ return NULL;
2743+ }
2744+ PyMem_Free(pathname);
2745+ return PyUnicode_FromString(buf);
2746+}
2747+
2748+PyDoc_STRVAR(doc_source_from_cache,
2749+"Given the path to a .pyc./.pyo file, return the path to its .py file.\n\
2750+\n\
2751+The .pyc/.pyo file does not need to exist; this simply returns the path to\n\
2752+the .py file calculated to correspond to the .pyc/.pyo file. If path\n\
2753+does not conform to PEP 3147 format, ValueError will be raised.");
2754+
2755 /* Doc strings */
2756
2757 PyDoc_STRVAR(doc_imp,
2758@@ -3212,6 +3539,10 @@
2759 "get_magic() -> string\n\
2760 Return the magic number for .pyc or .pyo files.");
2761
2762+PyDoc_STRVAR(doc_get_tag,
2763+"get_tag() -> string\n\
2764+Return the magic tag for .pyc or .pyo files.");
2765+
2766 PyDoc_STRVAR(doc_get_suffixes,
2767 "get_suffixes() -> [(suffix, mode, type), ...]\n\
2768 Return a list of (suffix, mode, type) tuples describing the files\n\
2769@@ -3242,6 +3573,7 @@
2770 static PyMethodDef imp_methods[] = {
2771 {"find_module", imp_find_module, METH_VARARGS, doc_find_module},
2772 {"get_magic", imp_get_magic, METH_NOARGS, doc_get_magic},
2773+ {"get_tag", imp_get_tag, METH_NOARGS, doc_get_tag},
2774 {"get_suffixes", imp_get_suffixes, METH_NOARGS, doc_get_suffixes},
2775 {"load_module", imp_load_module, METH_VARARGS, doc_load_module},
2776 {"new_module", imp_new_module, METH_VARARGS, doc_new_module},
2777@@ -3249,6 +3581,10 @@
2778 {"acquire_lock", imp_acquire_lock, METH_NOARGS, doc_acquire_lock},
2779 {"release_lock", imp_release_lock, METH_NOARGS, doc_release_lock},
2780 {"reload", imp_reload, METH_O, doc_reload},
2781+ {"cache_from_source", (PyCFunction)imp_cache_from_source,
2782+ METH_VARARGS | METH_KEYWORDS, doc_cache_from_source},
2783+ {"source_from_cache", (PyCFunction)imp_source_from_cache,
2784+ METH_VARARGS | METH_KEYWORDS, doc_source_from_cache},
2785 /* The rest are obsolete */
2786 {"get_frozen_object", imp_get_frozen_object, METH_VARARGS},
2787 {"is_frozen_package", imp_is_frozen_package, METH_VARARGS},
2788@@ -3436,7 +3772,6 @@
2789 failure:
2790 Py_XDECREF(m);
2791 return NULL;
2792-
2793 }
2794
2795
2796
2797=== modified file 'Python/pythonrun.c'
2798--- Python/pythonrun.c 2010-03-14 06:49:55 +0000
2799+++ Python/pythonrun.c 2010-04-16 18:55:41 +0000
2800@@ -1155,6 +1155,8 @@
2801 Py_DECREF(f);
2802 return -1;
2803 }
2804+ if (PyDict_SetItemString(d, "__cached__", Py_None) < 0)
2805+ return -1;
2806 set_file_name = 1;
2807 Py_DECREF(f);
2808 }

Subscribers

People subscribed via source and target branches

to all changes: