From 03323f5c8b8ebefd85649cde3a96b84841871ad6 Mon Sep 17 00:00:00 2001 From: William S Fulton Date: Sun, 16 Dec 2018 16:34:04 +0000 Subject: [PATCH] The Python module import logic has changed to stop obfuscating real ImportError problems. Only one import of the low-level C/C++ module from the pure Python module is attempted now. Previously a second import of the low-level C/C++ module was attempted after an ImportError occurred and was done to support 'split modules'. A 'split module' is a configuration where the pure Python module is a module within a Python package and the low-level C/C++ module is a global Python module. Now a 'split module' configuration is no longer supported by default. This configuration can be supported with a simple customization, such as: %module(package="mypackage", moduleimport="import $module") foo or if using -builtin: %module(package="mypackage", moduleimport="from $module import *") foo instead of %module(package="mypackage") foo See the updated Python chapter titled "Location of modules" in the documentation. Closes #848 #1343 --- CHANGES.current | 22 ++ Doc/Manual/Contents.html | 7 +- Doc/Manual/Python.html | 331 +++++++++++++----- .../import_packages/split_modules/README | 3 + .../split_modules/vanilla_split/foo.i | 7 +- Source/Modules/python.cxx | 31 +- 6 files changed, 275 insertions(+), 126 deletions(-) diff --git a/CHANGES.current b/CHANGES.current index f41db17ca..290408409 100644 --- a/CHANGES.current +++ b/CHANGES.current @@ -7,6 +7,28 @@ the issue number to the end of the URL: https://github.com/swig/swig/issues/ Version 4.0.0 (in progress) =========================== +2018-12-16: wsfulton + [Python] #848 #1343 The module import logic has changed to stop obfuscating real ImportError + problems. Only one import of the low-level C/C++ module from the pure Python module is + attempted now. Previously a second import of the low-level C/C++ module was attempted + after an ImportError occurred and was done to support 'split modules'. A 'split module' is + a configuration where the pure Python module is a module within a Python package and the + low-level C/C++ module is a global Python module. Now a 'split module' configuration is + no longer supported by default. This configuration can be supported with a simple + customization, such as: + + %module(package="mypackage", moduleimport="import $module") foo + + or if using -builtin: + + %module(package="mypackage", moduleimport="from $module import *") foo + + instead of + + %module(package="mypackage") foo + + See the updated Python chapter titled "Location of modules" in the documentation. + 2018-12-06: vadz #1359 #1364 Add missing nested class destructor wrapper when the nested class is inside a template. Removes associated bogus 'Illegal destructor name' warning. Only diff --git a/Doc/Manual/Contents.html b/Doc/Manual/Contents.html index a3179bfd7..4919d2c6a 100644 --- a/Doc/Manual/Contents.html +++ b/Doc/Manual/Contents.html @@ -1690,12 +1690,13 @@
  • Absolute and relative imports
  • Enforcing absolute import semantics
  • Importing from __init__.py -
  • Implicit Namespace Packages -
  • Searching for the wrapper module +
  • Implicit namespace packages +
  • Location of modules diff --git a/Doc/Manual/Python.html b/Doc/Manual/Python.html index ff5ef2993..2b2b6e1b9 100644 --- a/Doc/Manual/Python.html +++ b/Doc/Manual/Python.html @@ -119,12 +119,13 @@
  • Absolute and relative imports
  • Enforcing absolute import semantics
  • Importing from __init__.py -
  • Implicit Namespace Packages -
  • Searching for the wrapper module +
  • Implicit namespace packages +
  • Location of modules @@ -5686,15 +5687,14 @@ Python3 adds another option for packages with namespace packages). Implicit namespace packages no longer use __init__.py files. SWIG generated Python modules support implicit namespace packages. See -36.11.5 Implicit Namespace -Packages for more information. +Implicit namespace +packages for more information.

    -If you place a SWIG generated module into a Python package then there -are details concerning the way SWIG -searches for the wrapper module -that you may want to familiarize yourself with. +You can place a SWIG generated module into a Python package or keep as a global module, +details are covered a little later in +Location of modules.

    The way Python defines its modules and packages impacts SWIG users. Some @@ -6040,7 +6040,7 @@ class Bar(pkg3.foo.Foo): pass effect (note, that the Python 2 case also needs the -relativeimport workaround).

    -

    38.11.5 Implicit Namespace Packages

    +

    38.11.5 Implicit namespace packages

    Python 3.3 introduced @@ -6118,14 +6118,14 @@ zipimporter requires python-3.5.1 or newer to work with subpackages.

    -

    38.11.6 Searching for the wrapper module

    +

    38.11.6 Location of modules

    When SWIG creates wrappers from an interface file, say foo.i, two Python modules are created. There is a pure Python module (foo.py) and C/C++ code which is -built and linked into a dynamically (or statically) loaded low-level module _foo -(see the Preliminaries section for details). So, the interface +compiled and linked into a dynamically (or statically) loaded low-level module _foo +(see the Preliminaries section for details). So, the interface file really defines two Python modules. How these two modules are loaded is covered next.

    @@ -6133,28 +6133,168 @@ covered next.

    The pure Python module needs to load the C/C++ module in order to call the wrapped C/C++ methods. To do this it must make some assumptions -about what package the C/C++ module may be located in. The approach the -pure Python module uses to find the C/C++ module is as follows: +about the location of the C/C++ module. +There are two configurations that are supported by default.

      -

    1. The pure Python module, foo.py, tries to load the C/C++ module, _foo, from the same package foo.py is - located in. The package name is determined from the __package__ - attribute if available, see PEP 366, otherwise it is derived from the __name__ - attribute given to foo.py by the Python loader that imported - foo.py. If foo.py is not in a package then _foo is loaded - as a global module.

      +

    2. Both modules in the same package

      3. -

    3. If the above import of _foo results in an ImportError - being thrown, then foo.py makes a final attempt to load _foo - as a global module.

      +

    4. Both modules are global

    -The Python code implementing the loading logic described above is quite complex to handle multiple -versions of Python, but it can be replaced with custom code. -This is not recommended unless you understand the full intricacies of importing Python modules. +Additional configurations are supported but require custom import code. +

    + + +

    +The following sub-sections look more closely at the two default configurations as well as some customized configurations. +An input interface file, foo.i, results in the two modules foo.py and _foo.so for each of the configurations. +

    + +

    38.11.6.1 Both modules in the same package

    + + +

    +In this configuration, the pure Python module, foo.py, tries to load the C/C++ module, _foo, from the same package foo.py is +located in. The package name is determined from the __package__ +attribute if available, see PEP 366. Otherwise it is derived from the __name__ +attribute given to foo.py by the Python loader that imported foo.py. +The interface file for this configuration would contain: +

    + +
    +
    +%module(package="mypackage") foo
+
    +
    + +

    The location of the files could be as follows:

    +
    +
    +/dir/mypackage/foo.py
+/dir/mypackage/__init__.py
+/dir/mypackage/_foo.so
+
    +
    + +

    Assuming /dir/ is in PYTHONPATH, the module can be imported using

    + +
    +
    +from mypackage import foo
+
    +
    + + +

    38.11.6.2 Both modules are global

    + + +

    +In this configuration, there are no packages. +If foo.py is not in a package, that is, it is a global module, then _foo is loaded +as a global module. +The interface file for this configuration would contain: +

    + +
    +
    +%module foo
+
    +
    + +

    The location of the files could be as follows:

    +
    +
    +/dir/foo.py
+/dir/_foo.so
+
    +
    + +

    Assuming /dir/ is in PYTHONPATH, the module can be imported using

    + +
    +
    +import foo
+
    +
    + +

    38.11.6.3 Split modules custom configuration

    + + +

    In this non-standard 'split module' configuration, the pure Python module is in a package and the low level C/C++ module is global. +This configuration is not generally recommended and is not supported by default as it needs a custom configuration. +The module import code customization required is via the moduleimport attribute in the %module directive. +The next sub-section elaborates further on this. +The interface file for this split module configuration would contain: +

    + +
    +
    +%module(package="mypackage", moduleimport="import _foo") foo
+
    +
    + +

    +When using -builtin, use the following instead (the reasons are also covered shortly in the next sub-section): +

    + +
    +
    +%module(package="mypackage", moduleimport="from _foo import *") foo
+
    +
    + +

    The location of the files could be as follows:

    +
    +
    +/dir/mypackage/foo.py
+/dir/mypackage/__init__.py
+/dir/_foo.so
+
    +
    + +

    Assuming /dir/ is in PYTHONPATH, the module can be imported using

    + +
    +
    +from mypackage import foo
+
    +
    + +

    +Compatibility Note: Versions of SWIG prior to SWIG-4.0.0 supported split modules without the above customization. +However, this had to be removed as the default import code often led to confusion due to obfuscation of genuine Python ImportError problems. +Using one of the two default configurations is the recommended approach now. +

    + + +

    38.11.6.4 More on customizing the module import code

    + + +

    +The Python code implementing the default import logic is shown below. It supports the two configurations described earlier, that is, +either both modules are in a package or loading both as global modules. +The code is generated into the pure Python module, foo.py, and merely imports the low-level _foo module. +

    + +
    +
    +def swig_import_helper():
+    import importlib
+    pkg = __package__ if __package__ else __name__.rpartition('.')[0]
+    mname = '.'.join((pkg, '_foo')).lstrip('.')
+    return importlib.import_module(mname)
+_foo = swig_import_helper()
+del swig_import_helper
+
    +
    + +

    +This import code implementation is non-trivial but it can be replaced with custom code providing opportunities to make it simpler and/or more flexible. +This is not normally recommended though unless you have a good understanding of the intricacies of importing Python modules. The custom code can be specified by setting the moduleimport option of the %module directive with the appropriate import code. For example:

    @@ -6165,8 +6305,33 @@ The custom code can be specified by setting the moduleimport option of

    -The special variable $module will also be expanded into the low-level C/C++ module name, _foo in the case above. -When you have more than just a line or so then you can retain the easy +This will replace the default import logic above and generate the following into the pure Python module, foo.py: +

    + +
    +
    +import _foo
+
    +
    + +

    +In fact the above is a simplification customization for the configuration where both modules are global; +it removes the logic for also handling the modules being in a package. +

    + +

    +There is a special variable, $module, which is expanded into the low-level C/C++ module name, _foo in the case above. +The identical output would be generated if instead the following had been used: +

    + +
    +
    +%module(moduleimport="import $module") foo
+
    +
    + +

    +When you have many lines you can retain the easy readability of the %module directive by using a macro. For example:

    @@ -6185,79 +6350,51 @@ print 'Module has loaded' -

    -Now let's consider an example using the SWIG default loading logic. -Suppose foo.i is compiled into foo.py and _foo.so. Assuming -/dir is on PYTHONPATH, then the two modules can be installed and used in the -following ways: +This will of course generate the following into the pure Python module:

    - -

    38.11.6.1 Both modules in the same package

    - - -

    Both modules are in one package:

    -
    +
    -/dir/package/foo.py
-/dir/package/__init__.py
-/dir/package/_foo.so
-
    -
    -

    And imported with

    -
    -
    -from package import foo
-
    -
    - - -

    38.11.6.2 Split modules

    - - -

    The pure Python module is in a package and the C/C++ module is global:

    -
    -
    -/dir/package/foo.py
-/dir/package/__init__.py
-/dir/_foo.so
-
    -
    -

    And imported with

    -
    -
    -from package import foo
-
    -
    - - -

    38.11.6.3 Both modules are global

    - - -

    Both modules are global:

    -
    -
    -/dir/foo.py
-/dir/_foo.so
-
    -
    -

    And imported with

    -
    -
    -import foo
+print 'Loading low-level module $module'
+import _foo
+print 'Module has loaded'

    -If _foo is statically linked into an embedded Python interpreter, then it may or -may not be in a Python package. This depends in the exact way the module was -loaded statically. The above search order will still be used for statically -loaded modules. So, one may place the module either globally or in a package -as desired. +When using the -builtin option, the link between the pure Python module and the low-level C/C++ module is slightly different as +all the objects from the low-level module are imported directly into the pure Python module. +The default import loading code is thus different:

    -

    38.11.6.4 Statically linked C modules

    +
    +
    +if __package__ or __name__.rpartition('.')[0]:
+    from ._foo import *
+else:
+    from _foo import *
+
    +
    + +

    +Any customizations must import the code in a similar manner. +The best way to support both with and without -builtin is to make use of the SWIGPYTHON_BUILTIN macro which is defined when -builtin is specified. +The following will do this for the split modules case above. +

    + + +
    +
    +#if defined(SWIGPYTHON_BUILTIN) /* defined when using -builtin */
+%module(package="mypackage", moduleimport="from $module import *") foo
+#else
+%module(package="mypackage", moduleimport="import $module") foo
+#endif
+
    +
    + +

    38.11.6.5 Statically linked C modules

    It is strongly recommended to use dynamically linked modules for the C @@ -6266,7 +6403,7 @@ If for some reason you still need to link the C module of the pair of Python modules generated by SWIG into your interpreter, then this section provides some details on how this impacts the pure Python modules ability to locate the other part of the pair. -Please also see the Static Linking section. +Please also see the Static Linking section.

    When Python is extended with C code the Python interpreter needs to be @@ -6283,7 +6420,7 @@ new SWIG C module exists. which would have normally been called when the shared object was dynamically loaded. The specific name of this method is not given here because statically linked modules are not encouraged with SWIG -(Static Linking). However one can find this +(Static Linking). However one can find this init function in the C file generated by SWIG.

    @@ -6306,21 +6443,21 @@ for Python itself. Links to the relevant sections follow:

    There are two keys things to understand. The first is that in Python 2 the init() function returns void. In Python 3 the init() function -returns a PyObject * which points to the new module. Secondly, when +returns a PyObject * which points to the new module. Secondly, when you call the init() method manually, you are the Python importer. So, you determine which package the C module will be located in.

    So, if you are using Python 3 it is important that you follow what is described in the Python documentation linked above. In particular, you can't -simply call the init() function generated by SWIG and cast the PyObject +simply call the init() function generated by SWIG and cast the PyObject pointer it returns over the side. If you do then Python 3 will have no idea that your C module exists and the pure Python half of your wrapper will not be able to find it. You need to register your module with the Python interpreter as described in the Python docs.

    -

    With Python 2 things are somewhat more simple. In this case the init function +

    With Python 2 things are somewhat more simple. In this case the init() function returns void. Calling it will register your new C module as a global module. The pure Python part of the SWIG wrapper will be able to find it because it tries both the pure Python module it is part of and the global diff --git a/Examples/python/import_packages/split_modules/README b/Examples/python/import_packages/split_modules/README index 0cb543e8a..d2ca15e7a 100644 --- a/Examples/python/import_packages/split_modules/README +++ b/Examples/python/import_packages/split_modules/README @@ -2,6 +2,9 @@ between two packages. Specifically the pure python part is part of a package and the C/C++ part is not in any package at all. Historically SWIG has supported this sort of thing. +From SWIG 4.0.0 onwards, split modules are not supported by default. +The %module directive needs to be customised with the moduleimport attribute +in order to import the a global C/C++ module. vanilla # "plane Jane" module both halves in pkg1 vanilla_split # python 1/2 in pkg1 C 1/2 in global namespace diff --git a/Examples/python/import_packages/split_modules/vanilla_split/foo.i b/Examples/python/import_packages/split_modules/vanilla_split/foo.i index 60ce16ec3..81ad43a5b 100644 --- a/Examples/python/import_packages/split_modules/vanilla_split/foo.i +++ b/Examples/python/import_packages/split_modules/vanilla_split/foo.i @@ -1,4 +1,9 @@ -%module(package="pkg1") foo +#if defined(SWIGPYTHON_BUILTIN) /* defined when using -builtin */ +%module(package="pkg1", moduleimport="from $module import *") foo +#else +%module(package="pkg1", moduleimport="import $module") foo +#endif + %{ static unsigned count(void) { diff --git a/Source/Modules/python.cxx b/Source/Modules/python.cxx index 8764fd97f..e7bc76757 100755 --- a/Source/Modules/python.cxx +++ b/Source/Modules/python.cxx @@ -697,35 +697,16 @@ public: Printv(default_import_code, "def swig_import_helper():\n", NULL); Printv(default_import_code, tab4, "import importlib\n", NULL); Printv(default_import_code, tab4, "pkg = __package__ if __package__ else __name__.rpartition('.')[0]\n", NULL); - Printf(default_import_code, tab4 "mname = '.'.join((pkg, '%s')).lstrip('.')\n", module); - Printv(default_import_code, tab4, "try:\n", NULL); - Printv(default_import_code, tab8, "return importlib.import_module(mname)\n", NULL); - Printv(default_import_code, tab4, "except ImportError:\n", NULL); - Printf(default_import_code, tab8 "return importlib.import_module('%s')\n", module); - Printf(default_import_code, "%s = swig_import_helper()\n", module); + Printv(default_import_code, tab4, "mname = '.'.join((pkg, '", module, "')).lstrip('.')\n", NULL); + Printv(default_import_code, tab4, "return importlib.import_module(mname)\n", NULL); + Printv(default_import_code, module, " = swig_import_helper()\n", NULL); Printv(default_import_code, "del swig_import_helper\n", NULL); } else { - /* - * Pull in all the attributes from the C module. - * - * An alternative approach to doing this if/else chain was - * proposed by Michael Thon at https://github.com/swig/swig/issues/691. - * Someone braver than I may try it out. - * I fear some current swig user may depend on some side effect - * of from _foo import * - * - * for attr in _foo.__all__: - * globals()[attr] = getattr(_foo, attr) - * - */ - Printf(default_import_code, "\n# Pull in all the attributes from %s\n", module); + Printv(default_import_code, "# Pull in all the attributes from the low-level C/C++ module\n", NULL); Printv(default_import_code, "if __package__ or __name__.rpartition('.')[0]:\n", NULL); - Printv(default_import_code, tab4, "try:\n", NULL); - Printf(default_import_code, tab4 tab4 "from .%s import *\n", module); - Printv(default_import_code, tab4 "except ImportError:\n", NULL); - Printf(default_import_code, tab4 tab4 "from %s import *\n", module); + Printv(default_import_code, tab4, "from .", module, " import *\n", NULL); Printv(default_import_code, "else:\n", NULL); - Printf(default_import_code, tab4 "from %s import *\n", module); + Printv(default_import_code, tab4, "from ", module, " import *\n", NULL); } /* Need builtins to qualify names like Exception that might also be