与HDF5相比,为什么从CSV导入时pandas和dask的性能更好？

Question

与HDF5相比,为什么从CSV导入时pandas和dask的性能更好？

我正在使用当前使用大(> 5GB).csv文件操作的系统.为了提高性能,我正在测试(A)从磁盘创建数据帧的不同方法(pandas VS dask)以及(B)将结果存储到磁盘的不同方法(.csv VS hdf5文件).

为了衡量绩效,我做了以下几点:

def dask_read_from_hdf():
    results_dd_hdf = dd.read_hdf('store.h5', key='period1', columns = ['Security'])
    analyzed_stocks_dd_hdf =  results_dd_hdf.Security.unique()
    hdf.close()

def pandas_read_from_hdf():
    results_pd_hdf = pd.read_hdf('store.h5', key='period1', columns = ['Security'])
    analyzed_stocks_pd_hdf =  results_pd_hdf.Security.unique()
    hdf.close()

def dask_read_from_csv():
    results_dd_csv = dd.read_csv(results_path, sep = ",", usecols = [0], header = 1, names = ["Security"])
    analyzed_stocks_dd_csv =  results_dd_csv.Security.unique()

def pandas_read_from_csv():
    results_pd_csv = pd.read_csv(results_path, sep = ",", usecols = [0], header = 1, names = ["Security"])
    analyzed_stocks_pd_csv =  results_pd_csv.Security.unique()

print "dask hdf performance"
%timeit dask_read_from_hdf()
gc.collect()
print""
print "pandas hdf performance"
%timeit pandas_read_from_hdf()
gc.collect()
print""
print "dask csv performance"
%timeit dask_read_from_csv()
gc.collect()
print""
print "pandas csv performance"
%timeit pandas_read_from_csv()
gc.collect()

Run Code Online (Sandbox Code Playgroud)

我的发现是:

dask hdf performance
10 loops, best of 3: 133 ms per loop

pandas hdf performance
1 loop, best of 3: 1.42 s per loop

dask csv performance
1 loop, best of 3: 7.88 ms per loop

pandas csv performance
1 loop, best of 3: 827 ms per loop

Run Code Online (Sandbox Code Playgroud)

当hdf5存储的访问速度比.csv快时,并且当dask创建数据帧的速度比pandas快时,为什么hdf5的dask比csv的dask慢？难道我做错了什么？

何时从HDF5存储对象创建dask数据帧的性能是否有意义？