性能 - 在文本文件中搜索字符串 - Python

Question

我有一组日期:

dates1 = {'21/5/2015', '4/4/2015', '15/6/2015', '30/1/2015', '19/3/2015', '25/2/2015', '25/5/2015', '8/2/2015', '6/6/2015', '15/3/2015', '15/1/2015', '30/5/2015'}

相同的日期出现在文本中(从现在起'数据').这是一个很长的文字.我想循环文本并获得每个日期出现在文本中的次数,然后我打印出更多出现的5个日期.

我现在拥有的是:

def dates(data, dates1):
    lines = data.split("\n")
    dict_days = {}
    for day in dates1:
        count = 0
        for line in lines:
            if day in line:
                count += 1
        dict_days[day] = count

    newA = heapq.nlargest(5, dict_days, key=dict_days.get)

    print(newA)

我在行中分割tex,创建一个dict,对于列表中的每个日期,它在每一行中查找它,如果它发现它将count加1.

这个工作正常,但是这个方法花了很多时间.

所以我要问的是,如果有人知道一种更有效的方法来做同样的事情

任何帮助将非常感激

编辑

我会尝试每一个答案并让你知道,提前谢谢

Answer 1

循环一次,提取任何日期,检查日期是否在集合中,如果是这样,使用Counter dict 计数增加计数,在结束时调用Counter.most_common以获得5个最常见的日期:

dates1 = {'21/5/2015', '4/4/2015', '15/6/2015', '30/1/2015', '19/3/2015', '25/2/2015', '25/5/2015', '8/2/2015', '6/6/2015', '15/3/2015', '15/1/2015', '30/5/2015'}


from collections import Counter
import re

def dates(data, dates1):
    lines = data.split("\n")
    dict_days = Counter()
    r = re.compile("\d+/\d+/\d+")
    for line in lines:
        match = r.search(line)
        if match:
            dte = match.group()
            if dte in dates1:
                dict_days[dte] += 1
    return dict_days.most_common(5)

这会对行列表进行一次传递,而不是对date1中的每个日期进行一次传递.

对于100k行,日期字符串位于包含200多个字符的字符串末尾:

In [9]: from random import choice

In [10]: dates1 = {'21/5/2015', '4/4/2015', '15/6/2015', '30/1/2015', '19/3/2015', '25/2/2015', '25/5/2015', '8/2/2015', '6/6/2015', '15/3/2015', '15/1/2015', '30/5/2015'}

In [11]: dtes = list(dates1)

In [12]: s = "the same dates appear in a text ('data' from now on). It's a pretty long text. I want to loop over the text and get the number of times each date appear in the text, then i print the 5 dates with more occurances. "

In [13]: data = "\n".join([s+ choice(dtes) for _ in range(100000)])

In [14]: timeit dates(data,dates1)
1 loops, best of 3: 662 ms per loop

如果每行可以显示多个日期,则可以使用findall:

def dates(data, dates1):
    lines = data.split("\n")
    r = re.compile("\d+/\d+/\d+")
    dict_days = Counter(dt for line in lines
                        for dt in r.findall(line) if dt in dates1)
    return dict_days.most_common(5)

如果数据实际上不是像对象这样的文件并且是单个字符串,那么只需搜索字符串本身:

def dates(data, dates1):
    r = re.compile("\d+/\d+/\d+")
    dict_days = Counter((dt for dt in r.findall(data) if dt in dates1))
    return dict_days.most_common(5)

编译测试数据上的日期似乎是最快的方法,拆分每个子字符串非常接近搜索实现:

def dates_split(data, dates1):
    lines = data.split("\n")
    dict_days = Counter(dt for line in lines
                        for dt in line.split() if dt in dates1)
    return dict_days.most_common(5)

def dates_comp_date1(data, dates1):
    lines = data.split("\n")
    r = re.compile("|".join(dates1))
    dict_days = Counter(dt for line in lines for dt in r.findall(line))
    return dict_days.most_common(5)

使用上述功能:

In [63]: timeit dates(data, dates1)
1 loops, best of 3: 640 ms per loop

In [64]: timeit dates_split(data, dates1)
1 loops, best of 3: 535 ms per loop

In [65]: timeit dates_comp_date1(data, dates1)
1 loops, best of 3: 368 ms per loop