在大文件上,pickle是否会随OSError随机失败？

Question

问题陈述

我正在使用python3并试图挑选一个IntervalTrees字典,重量为2到3 GB.这是我的控制台输出:

10:39:25 - project: INFO - Checking if motifs file was generated by pickle...
10:39:25 - project: INFO -   - Motifs file does not seem to have been generated by pickle, proceeding to parse...
10:39:38 - project: INFO -   - Parse complete, constructing IntervalTrees...
11:04:05 - project: INFO -   - IntervalTree construction complete, saving pickle file for next time.
Traceback (most recent call last):
  File "/Users/alex/Documents/project/src/project.py", line 522, in dict_of_IntervalTree_from_motifs_file
    save_as_pickled_object(motifs, output_dir + 'motifs_IntervalTree_dictionary.pickle')
  File "/Users/alex/Documents/project/src/project.py", line 269, in save_as_pickled_object
    def save_as_pickled_object(object, filepath): return pickle.dump(object, open(filepath, "wb"))
OSError: [Errno 22] Invalid argument

我尝试保存的行是

def save_as_pickled_object(object, filepath): return pickle.dump(object, open(filepath, "wb"))

save_as_pickled_object调用错误可能是在调用后15分钟(在11:20).

我尝试使用motifs文件的一个小得多的子部分,它运行良好,所有完全相同的代码,所以它必须是规模问题.在python 3.6中是否有任何已知的错误与你试图腌制的规模有关？一般来说,是否存在酸洗大文件的已知错误？有没有任何已知的方法？

谢谢!

更新:这个问题可能与Python 3重复- 可以处理大于4GB的字节对象吗？

解

这是我用的代码.

def save_as_pickled_object(obj, filepath):
    """
    This is a defensive way to write pickle.write, allowing for very large files on all platforms
    """
    max_bytes = 2**31 - 1
    bytes_out = pickle.dumps(obj)
    n_bytes = sys.getsizeof(bytes_out)
    with open(filepath, 'wb') as f_out:
        for idx in range(0, n_bytes, max_bytes):
            f_out.write(bytes_out[idx:idx+max_bytes])


def try_to_load_as_pickled_object_or_None(filepath):
    """
    This is a defensive way to write pickle.load, allowing for very large files on all platforms
    """
    max_bytes = 2**31 - 1
    try:
        input_size = os.path.getsize(filepath)
        bytes_in = bytearray(0)
        with open(filepath, 'rb') as f_in:
            for _ in range(0, input_size, max_bytes):
                bytes_in += f_in.read(max_bytes)
        obj = pickle.loads(bytes_in)
    except:
        return None
    return obj

Answer 1

亚历克斯，如果我没记错的话，这个错误报告完美地描述了您的问题。

http://bugs.python.org/issue24658

作为一种解决方法，我认为您可以pickle.dumps先以pickle.dump小于 2**31 的块大小写入文件，然后再写入文件。