我们需要腌制任何类型的可调用

Question

最近提出了一些关于试图通过使用pickle进程来促进分布式计算的Python代码的问题.显然,该功能在历史上是可行的,但出于安全原因,相同的功能被禁用.在第二次尝试通过套接字传输功能对象时,仅传输了引用.如果我错了,请纠正我,但我不相信这个问题与Python的后期绑定有关.鉴于假设进程和线程对象无法被pickle,有没有办法传输可调用对象？我们希望避免为每个作业传输压缩源代码,因为这可能会使整个尝试毫无意义.出于可移植性原因,只能使用Python核心库.

Answer 1

你可以编组字节码并腌制其他功能:

import marshal
import pickle

marshaled_bytecode = marshal.dumps(your_function.func_code)
# In this process, other function things are lost, so they have to be sent separated.
pickled_name = pickle.dumps(your_function.func_name)
pickled_arguments = pickle.dumps(your_function.func_defaults)
pickled_closure = pickle.dumps(your_function.func_closure)
# Send the marshaled bytecode and the other function things through a socket (they are byte strings).
send_through_a_socket((marshaled_bytecode, pickled_name, pickled_arguments, pickled_closure))

在另一个python程序中:

import marshal
import pickle
import types

# Receive the marshaled bytecode and the other function things.
marshaled_bytecode, pickled_name, pickled_arguments, pickled_closure = receive_from_a_socket()
your_function = types.FunctionType(marshal.loads(marshaled_bytecode), globals(), pickle.loads(pickled_name), pickle.loads(pickled_arguments), pickle.loads(pickled_closure))

并且必须在接收函数的脚本中重新创建函数内对全局变量的任何引用.

在Python 3,所使用的函数的属性是__code__,__name__,__defaults__和__closure__.

请注意send_through_a_socket并且receive_from_a_socket实际上并不存在,您应该用通过套接字传输数据的实际代码替换它们.