熊猫:缓慢的日期转换

Question

我正在阅读CSV格式中带有日期字段的巨大内容,YYYYMMDD并且我在阅读时使用以下lambda进行转换:

import pandas as pd

df = pd.read_csv(filen,
                 index_col=None,
                 header=None,
                 parse_dates=[0],
                 date_parser=lambda t:pd.to_datetime(str(t),
                                            format='%Y%m%d', coerce=True))

这个功能虽然很慢.

有什么建议可以改善吗？

Answer 1

尝试使用此函数解析日期:

def lookup(s):
    """
    This is an extremely fast approach to datetime parsing.
    For large data, the same dates are often repeated. Rather than
    re-parse these, we store all unique dates, parse them, and
    use a lookup to convert all dates.
    """
    dates = {date:pd.to_datetime(date) for date in s.unique()}
    return s.map(dates)

使用它像:

df['date-column'] = lookup(df['date-column'])

基准:

$ python date-parse.py
to_datetime: 5799 ms
dateutil:    5162 ms
strptime:    1651 ms
manual:       242 ms
lookup:        32 ms

资料来源:https://github.com/sanand0/benchmarks/tree/master/date-parse

Answer 2

很棒的建议@EdChum!正如@EdChum建议的那样,使用infer_datetime_format=True可以明显更快.以下是我的例子.

我有一个来自传感器日志的温度数据文件,如下所示:

RecNum,Date,LocationID,Unused
1,11/7/2013 20:53:01,13.60,"117","1",
2,11/7/2013 21:08:01,13.60,"117","1",
3,11/7/2013 21:23:01,13.60,"117","1",
4,11/7/2013 21:38:01,13.60,"117","1",
...

我的代码读取csv并解析date(parse_dates=['Date']).用infer_datetime_format=False,需要8分8秒:

Tue Jan 24 12:18:27 2017 - Loading the Temperature data file.
Tue Jan 24 12:18:27 2017 - Temperature file is 88.172 MB.
Tue Jan 24 12:18:27 2017 - Loading into memory. Please be patient.
Tue Jan 24 12:26:35 2017 - Success: loaded 2,169,903 records.

使用infer_datetime_format=True,需要13秒:

Tue Jan 24 13:19:58 2017 - Loading the Temperature data file.
Tue Jan 24 13:19:58 2017 - Temperature file is 88.172 MB.
Tue Jan 24 13:19:58 2017 - Loading into memory. Please be patient.
Tue Jan 24 13:20:11 2017 - Success: loaded 2,169,903 records.

Answer 3

通过缓存简化日期解析

读取所有数据然后转换它总是比读取CSV时转换慢.如果您立即执行此操作,则不需要对所有数据进行两次迭代.您也不必将其作为字符串存储在内存中.

我们可以定义我们自己的日期解析器,它使用缓存来查看它已经看到的日期.

import pandas as pd

cache = {}

def cached_date_parser(s):
    if s in cache:
        return cache[s]
    dt = pd.to_datetime(s, format='%Y%m%d', coerce=True)
    cache[s] = dt
    return dt

df = pd.read_csv(filen,
                 index_col=None,
                 header=None,
                 parse_dates=[0],
                 date_parser=cached_date_parser)

与@fixxxer的答案具有相同的优点,只需解析每个字符串一次,额外增加的好处是不必读取所有数据,然后解析它.节省您的记忆和处理时间.

Answer 4

从pandas 版本 0.25开始，该函数pandas.read_csv接受cache_dates=boolean（默认为True）关键字参数。因此，无需像接受的答案中那样编写自己的缓存函数。