制作整个命名空间的副本？

Question

我想用动态构造的版本替换一些函数来复制整个命名空间.

换句话说,从namespace(import tensorflow as tf)开始,我想复制它,用我自己的版本替换一些函数,并更新__globals__所有符号以保持在新的命名空间内.这需要以依赖的拓扑顺序完成.

我开始在这里做类似的事情,但现在我开始怀疑我是否重新发明轮子.需要注意处理系统模块中的循环依赖,需要以不同方式更新函数/类型/对象等.

任何人都可以指向解决类似任务的现有代码吗？

Answer 1

要在导入一组函数的第二个实例时修补一组函数,可以覆盖标准Python导入挂钩并在导入时直接应用补丁.这将确保没有其他模块可以看到任何模块的未修补版本,因此即使它们直接通过名称从另一个模块导入函数,它们也只能看到修补的函数.这是一个概念验证实现:

import __builtin__
import collections
import contextlib
import sys


@contextlib.contextmanager
def replace_import_hook(new_import_hook):
    original_import = __builtin__.__import__
    __builtin__.__import__ = new_import_hook
    yield original_import
    __builtin__.__import__ = original_import


def clone_modules(patches, additional_module_names=None):
    """Import new instances of a set of modules with some objects replaced.

    Arguments:
      patches - a dictionary mapping `full.module.name.symbol` to the new object.
      additional_module_names - a list of the additional modules you want new instances of, without
          replacing any objects in them.

    Returns:
      A dictionary mapping module names to the new patched module instances.
    """

    def import_hook(module_name, *args):
        result = original_import(module_name, *args)
        if module_name not in old_modules or module_name in new_modules:
            return result
        # The semantics for the return value of __import__() are a bit weird, so we need some logic
        # to determine the actual imported module object.
        if len(args) >= 3 and args[2]:
            module = result
        else:
            module = reduce(getattr, module_name.split('.')[1:], result)
        for symbol, obj in patches_by_module[module_name].items():
            setattr(module, symbol, obj)
        new_modules[module_name] = module
        return result

    # Group patches by module name
    patches_by_module = collections.defaultdict(dict)
    for dotted_name, obj in patches.items():
        module_name, symbol = dotted_name.rsplit('.', 1)  # Only allows patching top-level objects
        patches_by_module[module_name][symbol] = obj

    try:
        # Remove the old module instances from sys.modules and store them in old_modules
        all_module_names = list(patches_by_module)
        if additional_module_names is not None:
            all_module_names.extend(additional_module_names)
        old_modules = {}
        for name in all_module_names:
            old_modules[name] = sys.modules.pop(name)

        # Re-import modules to create new patched versions
        with replace_import_hook(import_hook) as original_import:
            new_modules = {}
            for module_name in all_module_names:
                import_hook(module_name)
    finally:
        sys.modules.update(old_modules)
    return new_modules

这里有一些测试代码:

from __future__ import print_function

import math
import random

def patched_log(x):
    print('Computing log({:g})'.format(x))
    return math.log(x)

patches = {'math.log': patched_log}
cloned_modules = clone_modules(patches, ['random'])
new_math = cloned_modules['math']
new_random = cloned_modules['random']
print('Original log:         ', math.log(2.0))
print('Patched log:          ', new_math.log(2.0))
print('Original expovariate: ', random.expovariate(2.0))
print('Patched expovariate:  ', new_random.expovariate(2.0))

测试代码有这个输出:

Computing log(4)
Computing log(4.5)
Original log:          0.69314718056
Computing log(2)
Patched log:           0.69314718056
Original expovariate:  0.00638038735379
Computing log(0.887611)
Patched expovariate:   0.0596108277801

前两行输出来自这两行random,它们在导入时执行.这表明立即random看到修补功能.输出其余表明,原来的math和random仍然使用的未打补丁的版本log,而克隆的模块都使用补丁版本.

覆盖导入钩子的一种更简洁的方法可能是使用PEP 302中定义的元导入钩子,但是提供该方法的完整实现超出了StackOverflow的范围.