使用chunksize迭代地获取推断的数据帧类型

Question

如何使用pd.read_csv()迭代地浏览文件并保留dtype和其他元信息,就好像我一次读入整个数据集一样？

我需要读入一个太大而无法放入内存的数据集.我想使用pd.read_csv导入文件,然后立即将块附加到HDFStore中.但是,数据类型推断对后续块没有任何了解.

如果存储在表中的第一个块仅包含int,并且后续块包含float,则会引发异常.所以我需要首先使用read_csv遍历数据帧并保留最高的推断类型.另外,对于对象类型,我需要保留最大长度,因为它们将作为字符串存储在表中.

是否存在仅在不读取整个数据集的情况下仅保留此信息的Pandonic方式？

Answer 1

我不认为这会是直观的,否则我不会发布这个问题.但是,熊猫再一次让事情变得轻而易举.但是,将问题保留为此信息可能对使用大数据的其他人有用:

In [1]: chunker = pd.read_csv('DATASET.csv', chunksize=500, header=0)

# Store the dtypes of each chunk into a list and convert it to a dataframe:

In [2]: dtypes = pd.DataFrame([chunk.dtypes for chunk in chunker])

In [3]: dtypes.values[:5]
Out[3]:
array([[int64, int64, int64, object, int64, int64, int64, int64],
       [int64, int64, int64, int64, int64, int64, int64, int64],
       [int64, int64, int64, int64, int64, int64, int64, int64],
       [int64, int64, int64, int64, int64, int64, int64, int64],
       [int64, int64, int64, int64, int64, int64, int64, int64]], dtype=object)

# Very cool that I can take the max of these data types and it will preserve the hierarchy:

In [4]: dtypes.max().values
Out[4]: array([int64, int64, int64, object, int64, int64, int64, int64], dtype=object)

# I can now store the above into a dictionary:

types = dtypes.max().to_dict()

# And pass it into pd.read_csv fo the second run:

chunker = pd.read_csv('tree_prop_dset.csv', dtype=types, chunksize=500)