将pickle py2移植到py3字符串变为字节

Question

我有一个用python 2.7创建的pickle文件,我正在尝试移植到python 3.6.该文件保存在py 2.7 viapickle.dumps(self.saved_objects, -1)

并通过loads(data, encoding="bytes")(从rb模式打开的文件)加载到python 3.6 中.如果我尝试打开r模式并传递encoding=latin1给loads我得到UnicodeDecode错误.当我打开它作为字节流时,它会加载,但字面上每个字符串现在都是一个字节字符串.每个对象的__dict__密钥都会b"a_variable_name"在调用时生成属性错误,an_object.a_variable_name因为它__getattr__传递一个字符串并且__dict__只包含字节.我觉得我已经尝试了各种参数和pickle协议的组合.除了强行将所有对象的__dict__键转换为字符串之外,我很茫然.有任何想法吗？

**跳到4/28/17更新以获得更好的示例

-------------------------------------------------- -------------------------------------------------- ---------

**更新4/27/17

这最小的例子说明了我的问题:

从py 2.7.13

import pickle

class test(object):
    def __init__(self):
        self.x = u"test ¢" # including a unicode str breaks things

t = test()
dumpstr = pickle.dumps(t)

>>> dumpstr
"ccopy_reg\n_reconstructor\np0\n(c__main__\ntest\np1\nc__builtin__\nobject\np2\nNtp3\nRp4\n(dp5\nS'x'\np6\nVtest \xa2\np7\nsb."

从py 3.6.1

import pickle

class test(object):
    def __init__(self):
        self.x = "xyz"

dumpstr = b"ccopy_reg\n_reconstructor\np0\n(c__main__\ntest\np1\nc__builtin__\nobject\np2\nNtp3\nRp4\n(dp5\nS'x'\np6\nVtest \xa2\np7\nsb."

t = pickle.loads(dumpstr, encoding="bytes")

>>> t
<__main__.test object at 0x040E3DF0>
>>> t.x
Traceback (most recent call last):
  File "<pyshell#15>", line 1, in <module>
    t.x
AttributeError: 'test' object has no attribute 'x'
>>> t.__dict__
{b'x': 'test ¢'} 
>>> 

-------------------------------------------------- -------------------------------------------------- ---------

更新4/28/17

要重新创建我的问题,我在这里发布我的实际原始泡菜数据

pickle文件是在python 2.7.13中创建的,windows 10使用的

with open("raw_data.pkl", "wb") as fileobj:
    pickle.dump(library, fileobj, protocol=0)

(协议0所以它是人类可读的)

要运行它你需要 classes.py

# classes.py

class Library(object): pass


class Book(object): pass


class Student(object): pass


class RentalDetails(object): pass

这里的测试脚本:

# load_pickle.py
import pickle, sys, itertools, os

raw_pkl = "raw_data.pkl"
is_py3 = sys.version_info.major == 3

read_modes = ["rb"]
encodings = ["bytes", "utf-8", "latin-1"]
fix_imports_choices = [True, False]
files = ["raw_data_%s.pkl" % x for x in range(3)]


def py2_test():
    with open(raw_pkl, "rb") as fileobj:
        loaded_object = pickle.load(fileobj)
        print("library dict: %s" % (loaded_object.__dict__.keys()))
        return loaded_object


def py2_dumps():
    library = py2_test()
    for protcol, path in enumerate(files):
        print("dumping library to %s, protocol=%s" % (path, protcol))
        with open(path, "wb") as writeobj:
            pickle.dump(library, writeobj, protocol=protcol)


def py3_test():
    # this test iterates over the different options trying to load
    # the data pickled with py2 into a py3 environment
    print("starting py3 test")
    for (read_mode, encoding, fix_import, path) in itertools.product(read_modes, encodings, fix_imports_choices, files):
        py3_load(path, read_mode=read_mode, fix_imports=fix_import, encoding=encoding)


def py3_load(path, read_mode, fix_imports, encoding):
    from traceback import print_exc
    print("-" * 50)
    print("path=%s, read_mode = %s fix_imports = %s, encoding = %s" % (path, read_mode, fix_imports, encoding))
    if not os.path.exists(path):
        print("start this file with py2 first")
        return
    try:
        with open(path, read_mode) as fileobj:
            loaded_object = pickle.load(fileobj, fix_imports=fix_imports, encoding=encoding)
            # print the object's __dict__
            print("library dict: %s" % (loaded_object.__dict__.keys()))
            # consider the test a failure if any member attributes are saved as bytes
            test_passed = not any((isinstance(k, bytes) for k in loaded_object.__dict__.keys()))
            print("Test %s" % ("Passed!" if test_passed else "Failed"))
    except Exception:
        print_exc()
        print("Test Failed")
    input("Press Enter to continue...")
    print("-" * 50)


if is_py3:
    py3_test()
else:
    # py2_test()
    py2_dumps()

将所有3放在同一目录中并c:\python27\python load_pickle.py首先运行,这将为3个协议中的每一个创建1个pickle文件.然后使用python 3运行相同的命令,并注意它的版本将__dict__ 键转换为字节.我让它工作了大约6个小时,但对于我的生活,我无法弄清楚我是如何再次打破它的.

Answer 1

简而言之,您正在使用对象中的对象来访问错误22005.datetime.dateRentalDetails

这可以使用encoding='bytes'参数解决,但这会使您的类__dict__包含字节:

>>> library = pickle.loads(pickle_data, encoding='bytes')
>>> dir(library)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: '<' not supported between instances of 'str' and 'bytes'

可以根据您的特定数据手动修复它:

def fix_object(obj):
    """Decode obj.__dict__ containing bytes keys"""
    obj.__dict__ = dict((k.decode("ascii"), v) for k, v in obj.__dict__.items())


def fix_library(library):
    """Walk all library objects and decode __dict__ keys"""
    fix_object(library)
    for student in library.students:
            fix_object(student)
    for book in library.books:
            fix_object(book)
            for rental in book.rentals:
                    fix_object(rental)

但这很脆弱,你应该寻找更好的选择.

1)实现__getstate__/__setstate__将datetime对象映射到非破坏表示,例如:

class Event(object):
    """Example class working around datetime pickling bug"""

    def __init__(self):
            self.date = datetime.date.today()

    def __getstate__(self):
            state = self.__dict__.copy()
            state["date"] = state["date"].toordinal()
            return state

    def __setstate__(self, state):
            self.__dict__.update(state)
            self.date = datetime.date.fromordinal(self.date)

2)根本不要使用泡菜.除了__getstate__/之外__setstate__,您可以在类中实现to_dict/ from_dict方法或类似方法,以将其内容保存为json或其他纯文本格式.

最后要注意的是,不应要求在每个对象中对库进行反向引用.