提高性能:删除仅出现一次的(大)列表中的所有字符串

Question

我有一个很大的清单(25000项,14000字),像这样:

之前(请看右下方列表):

texts = ['Lorem hello ipsum', 'Lorem ipsum generator machine', ... 'hello Lorem ipsum']

我想删除整个列表中只出现一次的所有单词.

后:

texts = ['Lorem generator ipsum', 'Lorem ipsum generator machine', ..., 'Machine Lorem ipsum']

我现在已经这样做了,但它真的很慢(大约2个小时).

all_tokens = sum(texts, [])
tokens_once = set(word for word in set(all_tokens) if all_tokens.count(word) == 1)
texts = [[word for word in text if word not in tokens_once] for text in texts]

如何提高性能？

---

编辑:

@DSM是对的,我的输入列表看起来像这样.抱歉,抱歉:

texts = [['Lorem', 'hello', 'ipsum'], ['Lorem', 'ipsum', 'generator', 'machine'], ... ['hello, 'Lorem', 'ipsum']]

Answer 1

您可以使用collections.Counter此处存储每个单词的计数,然后根据此计数过滤掉单词.使用collection.Counter您可以及时获得项目数O(N),而您当前的方法(list.count)需要O(N**2)时间.

并且永远不会sum用于展平列表列表,它非常慢(在您的实际代码中,您已经列出了字符串,并且sum()会引发错误).我在答案中使用了嵌套列表理解,如果你实际上有一个列表列表,那么最好在itertools.chain.from_iterable这里使用.

>>> from collections import Counter
>>> texts = ['Lorem hello ipsum', 'Lorem ipsum generator machine', 'hello Lorem ipsum']
>>> c = Counter(word for x in texts for word in x.split())
>>> [' '.join(y for y in x.split() if c[y] > 1) for x in texts]
['Lorem hello ipsum', 'Lorem ipsum', 'hello Lorem ipsum']

时间比较:

In [8]: texts = ['Lorem hello ipsum', 'Lorem ipsum generator machine', 'hello Lorem ipsum']

In [9]: huge_texts = [x.split()*100 for x in texts]*1000  #list of lists                            

In [10]: %%timeit
from collections import Counter
from itertools import chain
c = Counter(chain.from_iterable(huge_texts))
texts = [[word for word in x if c[word]>1] for x in huge_texts]

1 loops, best of 3: 791 ms per loop

In [11]: %%timeit 
all_tokens = sum(huge_texts, [])
tokens_once = set(word for word in set(all_tokens) if all_tokens.count(word) == 1)
texts = [[word for word in text if word not in tokens_once] for text in huge_texts]                                                        

1 loops, best of 3: 20.4 s per loop