Python可以腌制lambda函数吗？

Question

我已经阅读了许多线程,Python pickle/ cPickle不能pickle lambda函数.但是,以下代码使用Python 2.7.6:

import cPickle as pickle

if __name__ == "__main__":
    s = pickle.dumps(lambda x, y: x+y)
    f = pickle.loads(s)
    assert f(3,4) == 7

那么发生了什么？或者说,酸洗lambdas的限制是什么？

[编辑]我想我知道为什么这个代码运行.我忘了(抱歉!)我正在运行无堆栈python,它有一种称为tasklet执行函数的微线程形式.这些tasklet可以暂停,腌制,打开并继续,所以我猜(在无堆栈邮件列表上询问)它还提供了一种方法来挑选函数体.

Answer 1

是的,python可以腌制lambda函数......但只有当你有一些copy_reg用于注册如何 pickle lambda函数的东西时 - 当你运行时,软件包dill会将copy_reg你需要的东西加载到pickle注册表中import dill.

Python 2.7.8 (default, Jul 13 2014, 02:29:54) 
[GCC 4.2.1 Compatible Apple Clang 4.1 ((tags/Apple/clang-421.11.66))] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> 
>>> import dill  # the code below will fail without this line
>>> 
>>> import pickle
>>> s = pickle.dumps(lambda x, y: x+y)
>>> f = pickle.loads(s)
>>> assert f(3,4) == 7
>>> f
<function <lambda> at 0x10aebdaa0>

在这里得到莳萝:https://github.com/uqfoundation

Answer 2

不,Python不能腌制lambda函数:

>>> import cPickle as pickle
>>> s = pickle.dumps(lambda x,y: x+y)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy_reg.py", line 70, in _reduce_ex
    raise TypeError, "can't pickle %s objects" % base.__name__
TypeError: can't pickle function objects

不确定你做了什么成功...

Answer 3

Python可以腌制lambdas.我们将分别介绍Python 2和3,因为pickle的实现在不同的Python版本中是不同的.

• Python 2.7

pickle使用pickle注册表,它只是从type用于序列化(酸洗)该类型对象的函数的映射.您可以看到pickle注册表:

>> pickle.Pickler.dispatch

{bool: <function pickle.save_bool>,
 instance: <function pickle.save_inst>,
 classobj: <function pickle.save_global>,
 float: <function pickle.save_float>,
 function: <function pickle.save_global>,
 int: <function pickle.save_int>,
 list: <function pickle.save_list>,
 long: <function pickle.save_long>,
 dict: <function pickle.save_dict>,
 builtin_function_or_method: <function pickle.save_global>,
 NoneType: <function pickle.save_none>,
 str: <function pickle.save_string>,
 tuple: <function pickle.save_tuple>,
 type: <function pickle.save_global>,
 unicode: <function pickle.save_unicode>}

为了挑选自定义类型,Python提供copy_reg了注册我们函数的模块.你可以在这里阅读更多相关信息.默认情况下,copy_reg模块支持以下其他类型的pickle:

>> import copy_reg
>> copy_reg.dispatch_table

{code: <function ipykernel.codeutil.reduce_code>,
 complex: <function copy_reg.pickle_complex>,
 _sre.SRE_Pattern: <function re._pickle>,
 posix.statvfs_result: <function os._pickle_statvfs_result>,
 posix.stat_result: <function os._pickle_stat_result>}

现在,lambda函数的类型是types.FunctionType.但是,此类型的内置函数function: <function pickle.save_global>无法序列化lambda函数.因此,所有第三方库(如dill,cloudpickle等)都会覆盖内置方法,以使用一些额外的逻辑来序列化lambda函数.让我们导入dill并查看它的功能.

>> import dill
>> pickle.Pickler.dispatch

{_pyio.BufferedReader: <function dill.dill.save_file>,
 _pyio.TextIOWrapper: <function dill.dill.save_file>,
 _pyio.BufferedWriter: <function dill.dill.save_file>,
 _pyio.BufferedRandom: <function dill.dill.save_file>,
 functools.partial: <function dill.dill.save_functor>,
 operator.attrgetter: <function dill.dill.save_attrgetter>,
 operator.itemgetter: <function dill.dill.save_itemgetter>,
 cStringIO.StringI: <function dill.dill.save_stringi>,
 cStringIO.StringO: <function dill.dill.save_stringo>,
 bool: <function pickle.save_bool>,
 cell: <function dill.dill.save_cell>,
 instancemethod: <function dill.dill.save_instancemethod0>,
 instance: <function pickle.save_inst>,
 classobj: <function dill.dill.save_classobj>,
 code: <function dill.dill.save_code>,
 property: <function dill.dill.save_property>,
 method-wrapper: <function dill.dill.save_instancemethod>,
 dictproxy: <function dill.dill.save_dictproxy>,
 wrapper_descriptor: <function dill.dill.save_wrapper_descriptor>,
 getset_descriptor: <function dill.dill.save_wrapper_descriptor>,
 member_descriptor: <function dill.dill.save_wrapper_descriptor>,
 method_descriptor: <function dill.dill.save_wrapper_descriptor>,
 file: <function dill.dill.save_file>,
 float: <function pickle.save_float>,
 staticmethod: <function dill.dill.save_classmethod>,
 classmethod: <function dill.dill.save_classmethod>,
 function: <function dill.dill.save_function>,
 int: <function pickle.save_int>,
 list: <function pickle.save_list>,
 long: <function pickle.save_long>,
 dict: <function dill.dill.save_module_dict>,
 builtin_function_or_method: <function dill.dill.save_builtin_method>,
 module: <function dill.dill.save_module>,
 NotImplementedType: <function dill.dill.save_singleton>,
 NoneType: <function pickle.save_none>,
 xrange: <function dill.dill.save_singleton>,
 slice: <function dill.dill.save_slice>,
 ellipsis: <function dill.dill.save_singleton>,
 str: <function pickle.save_string>,
 tuple: <function pickle.save_tuple>,
 super: <function dill.dill.save_functor>,
 type: <function dill.dill.save_type>,
 weakcallableproxy: <function dill.dill.save_weakproxy>,
 weakproxy: <function dill.dill.save_weakproxy>,
 weakref: <function dill.dill.save_weakref>,
 unicode: <function pickle.save_unicode>,
 thread.lock: <function dill.dill.save_lock>}

现在,让我们尝试腌制lambda函数.

>> pickle.loads(pickle.dumps(lambda x:x))
<function __main__.<lambda>>

有用!!

在Python 2中我们有两个版本pickle-

import pickle # pure Python version
pickle.__file__ # <install directory>/python-2.7/lib64/python2.7/pickle.py

import cPickle # C extension
cPickle.__file__ # <install directory>/python-2.7/lib64/python2.7/lib-dynload/cPickle.so

现在,让我们尝试用C实现pickle lambda cPickle.

>> import cPickle
>> cPickle.loads(cPickle.dumps(lambda x:x))
TypeError: can't pickle function objects

什么地方出了错？我们来看看调度表cPickle.

>> cPickle.Pickler.dispatch_table
AttributeError: 'builtin_function_or_method' object has no attribute 'dispatch_table'

执行pickle和cPickle不同.Importingdill只做Python版本的pickle工作.使用pickle代替的缺点cPickle是它可能比cPickle慢1000倍.

• Python 3.6

在Python 3中,没有名为的模块cPickle.pickle相反,我们lambda默认也不支持酸洗功能.让我们看看它的调度表:

>> import pickle
>> pickle.Pickler.dispatch_table
<member 'dispatch_table' of '_pickle.Pickler' objects>

等待.我试图查找dispatch_table的pickle不是_pickle._pickle是pickle的替代和更快的C实现.但是我们还没有进口它!如果可用,则在纯Python pickle模块的末尾自动导入此C实现.

# Use the faster _pickle if possible
try:
    from _pickle import (
        PickleError,
        PicklingError,
        UnpicklingError,
        Pickler,
        Unpickler,
        dump,
        dumps,
        load,
        loads
    )
except ImportError:
    Pickler, Unpickler = _Pickler, _Unpickler
    dump, dumps, load, loads = _dump, _dumps, _load, _loads

我们仍然留下在Python 3中腌制lambdas的问题.答案是你不能用本机pickle或者_pickle.您将需要导入dill或cloudpickle并使用它而不是原生pickle模块.

>> import dill
>> dill.loads(dill.dumps(lambda x:x))
<function __main__.<lambda>>

我希望这能解决所有疑虑.