熊猫:如何在使用read_csv时获取读取行的状态？

Question

我正在加载一个非常巨大的csv文件,如1000万条记录,使用pandas和read_csv方法,我想知道是否有办法显示加载的进度,如:

100,000 lines read
150,000 lines read

谢谢.

Answer 1

要显示这样的进度：

Completed 1 %
Completed 2 % 
... 
Completed 99 % 
Completed 100 %

你可以试试这个：

import os, pandas
filename = "VeryLong.csv"
lines_number = sum(1 for line in open(filename))
lines_in_chunk = 500 # I don't know what size is better
counter = 0
completed = 0
reader = pandas.read_csv(filename, chunksize=lines_in_chunk)
for chunk in reader:
    # < ... reading the chunk somehow... >
    # showing progress:
    counter += lines_in_chunk
    new_completed = int(round(float(counter)/lines_number * 100))
    if (new_completed > completed): 
        completed = new_completed
        print "Completed", completed, "%"