使用datetime索引对大熊猫read_csv进行速度提升

Question

我有大量的文件,如下所示:

5月31日/ 2012,15:30:00.029,130​​6.25,1,E,0,...,1306.25

5月31日/ 2012,15:30:00.029,130​​6.25,8,E,0,...,1306.25

我可以使用以下内容轻松阅读它们:

  pd.read_csv(gzip.open("myfile.gz"), header=None,names=
  ["date","time","price","size","type","zero","empty","last"], parse_dates=[[0,1]])

有没有办法有效地将这样的日期解析成熊猫时间戳？如果没有,是否有任何编写可以传递给date_parser =的cython函数的指南？

我尝试编写自己的解析器函数,但我正在处理的项目仍然需要很长时间.

Answer 1

我使用以下cython代码获得了令人难以置信的加速(50X):

从python调用:timestamps = convert_date_cython(df ["date"].values,df ["time"].values)

cimport numpy as np
import pandas as pd
import datetime
import numpy as np
def convert_date_cython(np.ndarray date_vec, np.ndarray time_vec):
    cdef int i
    cdef int N = len(date_vec)
    cdef out_ar = np.empty(N, dtype=np.object)
    date = None
    for i in range(N):
        if date is None or date_vec[i] != date_vec[i - 1]:
            dt_ar = map(int, date_vec[i].split("/"))
            date = datetime.date(dt_ar[2], dt_ar[0], dt_ar[1])
        time_ar = map(int, time_vec[i].split(".")[0].split(":"))
        time = datetime.time(time_ar[0], time_ar[1], time_ar[2])
        out_ar[i] = pd.Timestamp(datetime.datetime.combine(date, time))
    return out_ar

Answer 2

对以前的Michael WS解决方案的改进:

• 转换pandas.Timestamp为更好地在Cython代码之外执行
• atoi 并且处理native-c字符串比python funcs快一点
• datetime-lib调用的数量从2减少到1(偶尔为+1)
• 微秒也被处理

NB!此代码中的日期顺序是日/月/年.

总而言之,代码似乎比原始代码快大约10倍convert_date_cython.然而,如果read_csv在SSD硬盘驱动器之后调用此差异,则由于读取开销,总时间仅为几个百分点.我猜想在常规硬盘上,差异会更小.

cimport numpy as np
import datetime
import numpy as np
import pandas as pd
from libc.stdlib cimport atoi, malloc, free 
from libc.string cimport strcpy

### Modified code from Michael WS:
### https://stackoverflow.com/a/15812787/2447082

def convert_date_fast(np.ndarray date_vec, np.ndarray time_vec):
    cdef int i, d_year, d_month, d_day, t_hour, t_min, t_sec, t_ms
    cdef int N = len(date_vec)
    cdef np.ndarray out_ar = np.empty(N, dtype=np.object)  
    cdef bytes prev_date = <bytes> 'xx/xx/xxxx'
    cdef char *date_str = <char *> malloc(20)
    cdef char *time_str = <char *> malloc(20)

    for i in range(N):
        if date_vec[i] != prev_date:
            prev_date = date_vec[i] 
            strcpy(date_str, prev_date) ### xx/xx/xxxx
            date_str[2] = 0 
            date_str[5] = 0 
            d_year = atoi(date_str+6)
            d_month = atoi(date_str+3)
            d_day = atoi(date_str)

        strcpy(time_str, time_vec[i])   ### xx:xx:xx:xxxxxx
        time_str[2] = 0
        time_str[5] = 0
        time_str[8] = 0
        t_hour = atoi(time_str)
        t_min = atoi(time_str+3)
        t_sec = atoi(time_str+6)
        t_ms = atoi(time_str+9)

        out_ar[i] = datetime.datetime(d_year, d_month, d_day, t_hour, t_min, t_sec, t_ms)
    free(date_str)
    free(time_str)
    return pd.to_datetime(out_ar)