RecursionError:使用pickle.load()时调用Python对象时超出了最大递归深度

Question

首先我知道已经有很多关于这个特殊错误的问题,但我找不到任何解决它的确切背景的问题.我也尝试过为其他类似错误提供的解决方案,但没有任何区别.

我正在使用python模块pickle将对象保存到文件并使用以下代码重新加载它:

with open('test_file.pkl', 'wb') as a: 
    pickle.dump(object1, a, pickle.HIGHEST_PROTOCOL)

这不会抛出任何错误,但是当我尝试使用以下代码打开文件时:

with open('test_file.pkl', 'rb') as a:
    object2 = pickle.load(a)

我收到此错误:

---------------------------------------------------------------------------

RecursionError                            Traceback (most recent call last)
<ipython-input-3-8c5a70d147f7> in <module>()
      1 with open('2test_bolfi_results.pkl', 'rb') as a:
----> 2     results = pickle.load(a)
      3

~/.local/lib/python3.5/site-packages/elfi/methods/results.py in __getattr__(self, item)
     95     def __getattr__(self, item):
     96         """Allow more convenient access to items under self.meta."""
---> 97         if item in self.meta.keys():
     98             return self.meta[item]
     99         else:

... last 1 frames repeated, from the frame below ...

~/.local/lib/python3.5/site-packages/elfi/methods/results.py in __getattr__(self, item)
     95     def __getattr__(self, item):
     96         """Allow more convenient access to items under self.meta."""
---> 97         if item in self.meta.keys():
     98             return self.meta[item]
     99         else:

RecursionError: maximum recursion depth exceeded while calling a Python object

我知道其他人在做的时候已经看到了同样的错误(使用Pickle/cPickle达到最大递归深度)pickle.dump并且我尝试通过这样做来增加最大递归深度sys.setrecursionlimit()但是这不起作用,我得到与上面相同的错误或者我进一步增加它和python崩溃的消息:Segmentation fault (core dumped).

我怀疑问题的根源实际上是我保存对象pickle.load()但我真的不知道如何诊断它.

有什么建议？

(我在Windows 10机器上运行python3)

Answer 1

这是一个相当小的派生类,collections.UserDict它执行与问题对象相同的技巧.它是一个字典,允许您通过普通的dict语法或属性访问其项目.我已经抛出一些print调用,所以我们可以看到主要方法何时被调用.

import collections

class AttrDict(collections.UserDict):
    ''' A dictionary that can be accessed via attributes '''
    def __setattr__(self, key, value):
        print('SA', key, value)
        if key == 'data':
            super().__setattr__('data', value)
        else:
            self.data[key] = value

    def __getattr__(self, key):
        print('GA', key)
        if key in self.data:
            return self.data[key]
        else: 
            print('NOKEY')
            raise AttributeError

    def __delattr__(self, key):
        del self.data[key]

# test

keys = 'zero', 'one', 'two', 'three'
data = {k: i for i, k in enumerate(keys)}
d = AttrDict(data)
print(d)
print(d.zero, d.one, d.two, d['three'])

产量

SA data {}
{'zero': 0, 'one': 1, 'two': 2, 'three': 3}
GA zero
GA one
GA two
0 1 2 3

到现在为止还挺好.但是如果我们试图挑选我们的d实例,那么我们就会得到这样做RecursionError,__getattr__它可以实现对键查找的属性访问的神奇转换.我们可以通过提供类__getstate__和__setstate__方法来克服这个问题.

import pickle
import collections

class AttrDict(collections.UserDict):
    ''' A dictionary that can be accessed via attributes '''
    def __setattr__(self, key, value):
        print('SA', key, value)
        if key == 'data':
            super().__setattr__('data', value)
        else:
            self.data[key] = value

    def __getattr__(self, key):
        print('GA', key)
        if key in self.data:
            return self.data[key]
        else: 
            print('NOKEY')
            raise AttributeError

    def __delattr__(self, key):
        del self.data[key]

    def __getstate__(self):
        print('GS')
        return self.data

    def __setstate__(self, state):
        print('SS')
        self.data = state

# tests

keys = 'zero', 'one', 'two', 'three'
data = {k: i for i, k in enumerate(keys)}
d = AttrDict(data)
print(d)
print(d.zero, d.one, d.two, d['three'])

print('Pickling')
s = pickle.dumps(d, pickle.HIGHEST_PROTOCOL)
print(s)

print('Unpickling')
obj = pickle.loads(s)
print(obj)

产量

SA data {}
{'zero': 0, 'one': 1, 'two': 2, 'three': 3}
GA zero
GA one
GA two
0 1 2 3
Pickling
GS
b'\x80\x04\x95D\x00\x00\x00\x00\x00\x00\x00\x8c\x08__main__\x94\x8c\x08AttrDict\x94\x93\x94)\x81\x94}\x94(\x8c\x04zero\x94K\x00\x8c\x03one\x94K\x01\x8c\x03two\x94K\x02\x8c\x05three\x94K\x03ub.'
Unpickling
SS
SA data {'zero': 0, 'one': 1, 'two': 2, 'three': 3}
{'zero': 0, 'one': 1, 'two': 2, 'three': 3}

但是我们可以做些什么来修复这种行为的现有类？幸运的是,Python允许我们轻松地向现有类添加新方法,甚至是我们通过导入获得的方法.

import pickle
import collections

class AttrDict(collections.UserDict):
    ''' A dictionary that can be accessed via attributes '''
    def __setattr__(self, key, value):
        print('SA', key, value)
        if key == 'data':
            super().__setattr__('data', value)
        else:
            self.data[key] = value

    def __getattr__(self, key):
        print('GA', key)
        if key in self.data:
            return self.data[key]
        else: 
            print('NOKEY')
            raise AttributeError

    def __delattr__(self, key):
        del self.data[key]

# Patch the existing AttrDict class with __getstate__ & __setstate__ methods

def getstate(self):
    print('GS')
    return self.data

def setstate(self, state):
    print('SS')
    self.data = state

AttrDict.__getstate__ = getstate
AttrDict.__setstate__ = setstate

# tests

keys = 'zero', 'one', 'two', 'three'
data = {k: i for i, k in enumerate(keys)}
d = AttrDict(data)
print(d)
print(d.zero, d.one, d.two, d['three'])

print('Pickling')
s = pickle.dumps(d, pickle.HIGHEST_PROTOCOL)
print(s)

print('Unpickling')
obj = pickle.loads(s)
print(obj)

此代码生成与先前版本相同的输出,因此我不在此重复.

希望这能为您提供足够的信息来修复您的错误对象.我__getstate__和__setstate__方法只保存和恢复.data字典中的内容.为了正确地腌制你的物体,我们可能需要更加激烈.例如,我们可能需要保存和恢复实例的.__dict__属性,而不仅仅是.data属性,它与.meta问题对象中的属性相对应.