如何使用 sys.path_hooks 自定义加载模块？

Question

我希望下面的问题不会太长。但除此之外，我无法通过问题和我想要的来解释：

从如何使用 importlib 从任意来源导入模块中学习？（我昨天的问题）我为新文件类型 (.xxx) 编写了一个特定的加载程序。（实际上 xxx 是 pyc 的加密版本，以防止代码被盗）。

我只想为新文件类型“xxx”添加一个导入挂钩，而不会以任何方式影响其他类型（.py、.pyc、.pyd）。

现在，加载器是ModuleLoader，继承自mportlib.machinery.SourcelessFileLoader.

使用sys.path_hooksloader 应添加一个钩子：

myFinder = importlib.machinery.FileFinder
loader_details = (ModuleLoader, ['.xxx'])
sys.path_hooks.append(myFinder.path_hook(loader_details))

注意：这是通过调用激活一次 modloader.activateLoader()

加载名为test（这是一个test.xxx）的模块后，我得到：

myFinder = importlib.machinery.FileFinder
loader_details = (ModuleLoader, ['.xxx'])
sys.path_hooks.append(myFinder.path_hook(loader_details))

但是，当我sys.path_hooks在添加钩子之前删除内容时：

sys.path_hooks = []
sys.path.insert(0, '.') # current directory
sys.path_hooks.append(myFinder.path_hook(loader_details))

有用：

>>> modloader.activateLoader()
>>> import test
using xxx class

in xxxLoader exec_module
in xxxLoader get_code: .\test.xxx
ANALYZING ...

GENERATE CODE OBJECT ...

  2           0 LOAD_CONST               0
              3 LOAD_CONST               1 ('foo2')
              6 MAKE_FUNCTION            0
              9 STORE_NAME               0 (foo2)
             12 LOAD_CONST               2 (None)
             15 RETURN_VALUE
>>>>>> test
<module 'test' from '.\\test.xxx'>

将文件内容转换为代码对象后，模块会正确导入。

但是我无法从包中加载相同的模块： import pack.test

注意：__init__.py当然是包目录下的空文件。

>>> import modloader
>>> modloader.activateLoader()
>>> import test
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named 'test'
>>>

还不够，我不能再从那个包中加载普通的 *.py 模块了：我得到了与上面相同的错误：

sys.path_hooks = []
sys.path.insert(0, '.') # current directory
sys.path_hooks.append(myFinder.path_hook(loader_details))

因为我的理解sys.path_hooks是遍历直到最后一个条目被尝试。那么为什么第一个变体（不删除sys.path_hooks）不能识别新的扩展名“xxx”而第二个变体（删除sys.path_hooks）呢？当 的条目sys.path_hooks无法识别“xxx”时，看起来机器正在抛出异常而不是进一步遍历到下一个条目。

为什么是第二个版本的工作对于P，PYC和XXX模块在当前目录中，但不工作的包pack？我希望 py 和 pyc 甚至不在当前目录中工作，因为sys.path_hooks只包含一个“xxx”的钩子......

Answer 1

简短的回答是，默认的 PathFindersys.meta_path并不意味着在它已经支持的相同路径中添加新的文件扩展名和导入器。但还是有希望的！

快速分解

sys.path_hooks被importlib._bootstrap_external.PathFinder类消耗。

当导入发生时，sys.meta_path要求输入的每个条目为请求的模块找到匹配的规范。特别是路径查找器将获取 的内容sys.path并将其传递给 中的工厂函数sys.path_hooks。每个工厂函数都有机会引发 ImportError （基本上工厂说“不，我不支持此路径条目”）或返回该路径的查找程序实例。然后将第一个成功返回的查找器缓存在sys.path_importer_cache. 从那时起，PathFinder 将只询问那些缓存的 finder 实例是否可以提供请求的模块。

如果您查看 的内容sys.path_importer_cache，您将看到 中的所有目录条目sys.path都已映射到 FileFinder 实例。非目录条目（zip 文件等）将映射到其他查找器。

因此，如果您附加通过FileFinder.path_hookto创建的新工厂sys.path_hooks，则只有在先前的 FileFinder 挂钩不接受路径时才会调用您的工厂。这不太可能，因为 FileFinder 可以在任何现有目录上工作。

或者，如果您将新工厂插入到现有工厂之前的 sys.path_hooks 中，则仅当您的新工厂不接受路径时才会使用默认挂钩。同样，由于 FileFinder 对它所接受的内容非常自由，这将导致仅使用您的加载程序，正如您已经观察到的那样。

让它发挥作用

因此，您可以尝试调整现有工厂以支持您的文件扩展名和导入器（这很困难，因为导入器和扩展字符串元组保存在闭包中），或者做我最终做的事情，即添加新元路径查找器。

所以例如。从我自己的项目，

import sys

from importlib.abc import FileLoader
from importlib.machinery import FileFinder, PathFinder
from os import getcwd
from os.path import basename

from sibilant.module import prep_module, exec_module


SOURCE_SUFFIXES = [".lspy", ".sibilant"]


_path_importer_cache = {}
_path_hooks = []


class SibilantPathFinder(PathFinder):
    """
    An overridden PathFinder which will hunt for sibilant files in
    sys.path. Uses storage in this module to avoid conflicts with the
    original PathFinder
    """


    @classmethod
    def invalidate_caches(cls):
        for finder in _path_importer_cache.values():
            if hasattr(finder, 'invalidate_caches'):
                finder.invalidate_caches()


    @classmethod
    def _path_hooks(cls, path):
        for hook in _path_hooks:
            try:
                return hook(path)
            except ImportError:
                continue
        else:
            return None


    @classmethod
    def _path_importer_cache(cls, path):
        if path == '':
            try:
                path = getcwd()
            except FileNotFoundError:
                # Don't cache the failure as the cwd can easily change to
                # a valid directory later on.
                return None
        try:
            finder = _path_importer_cache[path]
        except KeyError:
            finder = cls._path_hooks(path)
            _path_importer_cache[path] = finder
        return finder


class SibilantSourceFileLoader(FileLoader):


    def create_module(self, spec):
        return None


    def get_source(self, fullname):
        return self.get_data(self.get_filename(fullname)).decode("utf8")


    def exec_module(self, module):
        name = module.__name__
        source = self.get_source(name)
        filename = basename(self.get_filename(name))

        prep_module(module)
        exec_module(module, source, filename=filename)


def _get_lspy_file_loader():
    return (SibilantSourceFileLoader, SOURCE_SUFFIXES)


def _get_lspy_path_hook():
    return FileFinder.path_hook(_get_lspy_file_loader())


def _install():
    done = False

    def install():
        nonlocal done
        if not done:
            _path_hooks.append(_get_lspy_path_hook())
            sys.meta_path.append(SibilantPathFinder)
            done = True

    return install


_install = _install()
_install()

SibilantPathFinder 覆盖 PathFinder 并仅替换那些引用sys.path_hook和sys.path_importer_cache具有类似实现的方法，这些实现相反在 a 中查找_path_hook并且_path_importer_cache对于该模块是本地的。

在导入期间，现有的 PathFinder 将尝试查找匹配的模块。如果不能，那么我注入的 SibilantPathFinder 将重新遍历sys.path并尝试找到与我自己的文件扩展名之一匹配的文件。

想出更多办法

我最终深入研究了 _bootstrap_external 模块的源代码 https://github.com/python/cpython/blob/master/Lib/importlib/_bootstrap_external.py

该_install函数和PathFinder.find_spec方法是最好的出发点看到为什么事情工作，他们的方式。

Answer 2

@obriencj 对情况的分析是正确的。但是我想出了一个不同的解决方案来解决这个问题，不需要在sys.meta_path. 相反，它安装了一个特殊的钩子 in sys.path_hooks，它几乎充当PathFinderinsys.meta_path和sys.path_hookswhere钩子之间的一种中间件，而不是仅仅使用第一个说“我可以处理这条路径！”的钩子。它按顺序尝试所有匹配的钩子，直到找到一个实际上ModuleSpec从其find_spec方法返回一个有用的钩子：

@PathEntryFinder.register
class MetaFileFinder:
    """
    A 'middleware', if you will, between the PathFinder sys.meta_path hook,
    and sys.path_hooks hooks--particularly FileFinder.

    The hook returned by FileFinder.path_hook is rather 'promiscuous' in that
    it will handle *any* directory.  So if one wants to insert another
    FileFinder.path_hook into sys.path_hooks, that will totally take over
    importing for any directory, and previous path hooks will be ignored.

    This class provides its own sys.path_hooks hook as follows: If inserted
    on sys.path_hooks (it should be inserted early so that it can supersede
    anything else).  Its find_spec method then calls each hook on
    sys.path_hooks after itself and, for each hook that can handle the given
    sys.path entry, it calls the hook to create a finder, and calls that
    finder's find_spec.  So each sys.path_hooks entry is tried until a spec is
    found or all finders are exhausted.
    """

    class hook:
        """
        Use this little internal class rather than a function with a closure
        or a classmethod or anything like that so that it's easier to
        identify our hook and skip over it while processing sys.path_hooks.
        """

        def __init__(self, basepath=None):
            self.basepath = os.path.abspath(basepath)

        def __call__(self, path):
            if not os.path.isdir(path):
                raise ImportError('only directories are supported', path=path)
            elif not self.handles(path):
                raise ImportError(
                    'only directories under {} are supported'.format(
                        self.basepath), path=path)

            return MetaFileFinder(path)

        def handles(self, path):
            """
            Return whether this hook will handle the given path, depending on
            what its basepath is.
            """

            path = os.path.abspath(path)

            return (self.basepath is None or
                    os.path.commonpath([self.basepath, path]) == self.basepath)

    def __init__(self, path):
        self.path = path
        self._finder_cache = {}

    def __repr__(self):
        return '{}({!r})'.format(self.__class__.__name__, self.path)

    def find_spec(self, fullname, target=None):
        if not sys.path_hooks:
            return None

        last = len(sys.path_hooks) - 1

        for idx, hook in enumerate(sys.path_hooks):
            if isinstance(hook, self.__class__.hook):
                continue

            finder = None
            try:
                if hook in self._finder_cache:
                    finder = self._finder_cache[hook]
                    if finder is None:
                        # We've tried this finder before and got an ImportError
                        continue
            except TypeError:
                # The hook is unhashable
                pass

            if finder is None:
                try:
                    finder = hook(self.path)
                except ImportError:
                    pass

            try:
                self._finder_cache[hook] = finder
            except TypeError:
                # The hook is unhashable for some reason so we don't bother
                # caching it
                pass

            if finder is not None:
                spec = finder.find_spec(fullname, target)
                if (spec is not None and
                        (spec.loader is not None or idx == last)):
                    # If no __init__.<suffix> was found by any Finder,
                    # we may be importing a namespace package (which
                    # FileFinder.find_spec returns in this case).  But we
                    # only want to return the namespace ModuleSpec if we've
                    # exhausted every other finder first.
                    return spec

        # Module spec not found through any of the finders
        return None

    def invalidate_caches(self):
        for finder in self._finder_cache.values():
            finder.invalidate_caches()

    @classmethod
    def install(cls, basepath=None):
        """
        Install the MetaFileFinder in the front sys.path_hooks, so that
        it can support any existing sys.path_hooks and any that might
        be appended later.

        If given, only support paths under and including basepath.  In this
        case it's not necessary to invalidate the entire
        sys.path_importer_cache, but only any existing entries under basepath.
        """

        if basepath is not None:
            basepath = os.path.abspath(basepath)

        hook = cls.hook(basepath)
        sys.path_hooks.insert(0, hook)
        if basepath is None:
            sys.path_importer_cache.clear()
        else:
            for path in list(sys.path_importer_cache):
                if hook.handles(path):
                    del sys.path_importer_cache[path]

这仍然是令人沮丧的，远比必要的复杂。我觉得在 Python 2 上，在导入系统重写之前，这样做要简单得多，因为对内置模块类型（.py等）的支持较少建立在导入钩子本身之上，所以它是通过添加钩子来导入新的模块类型，更难打破导入普通模块的过程。我将开始讨论 python-ideas，看看是否有任何方法无法改善这种情况。