将语料库字典排序为OrderedDict的最快方法 - python

Question

鉴于语料库/文本本身:

Resumption of the session
I declare resumed the session of the European Parliament adjourned on Friday 17 December 1999 , and I would like once again to wish you a happy new year in the hope that you enjoyed a pleasant festive period .
Although , as you will have seen , the dreaded ' millennium bug ' failed to materialise , still the people in a number of countries suffered a series of natural disasters that truly were dreadful .
You have requested a debate on this subject in the course of the next few days , during this part @-@ session .
In the meantime , I should like to observe a minute ' s silence , as a number of Members have requested , on behalf of all the victims concerned , particularly those of the terrible storms , in the various countries of the European Union .

我可以简单地这样做以获得一个字频率的字典:

>>> word_freq = Counter()
>>> for line in text.split('\n'):
...     for word in line.split():
...             word_freq[word]+=1
... 

但如果目标是从最高频率到最低频率实现有序字典,我将不得不这样做:

>>> from collections import OrderedDict
>>> sorted_word_freq = OrderedDict()
>>> for word, freq in word_freq.most_common():
...     sorted_word_freq[word] = freq
... 

想象一下,我在Counter对象中有10亿个键,迭代通过most_common()一个语料库(非唯一实例)和词汇表(唯一键)的复杂性.

注意:Counter.most_common()会调用ad-hoc sorted(),请参阅https://hg.python.org/cpython/file/e38470b49d3c/Lib/collections.py#l472

鉴于此,我看到以下代码使用numpy.argsort():

>>> import numpy as np
>>> words = word_freq.keys()
>>> freqs = word_freq.values()
>>> sorted_word_index = np.argsort(freqs) # lowest to highest
>>> sorted_word_freq_with_numpy = OrderedDict()
>>> for idx in reversed(sorted_word_index):
...     sorted_word_freq_with_numpy[words[idx]] = freqs[idx]
... 

哪个更快？

是否有其他更快捷的方式得到这样一个OrderedDict从Counter？

除此之外OrderedDict,还有其他python对象可以实现相同的排序键值对吗？

假设内存不是问题.鉴于120 GB的内存,保持10亿个键值对不应该有太多问题吗？假设10亿个密钥每个密钥平均有20个字符,每个值都有一个整数.

Answer 1

Pandas 中的对象Series是一个可能令人感兴趣的键值对数组（可以有非唯一的键）。它有一个sort按值排序的方法，并在 Cython 中实现。下面是对长度为 100 万的数组进行排序的示例：

In [39]:
import pandas as pd
import numpy as np

arr = np.arange(1e6)
np.random.shuffle(arr)
s = pd.Series(arr, index=np.arange(1e6))
%timeit s.sort()
%timeit sorted(arr)

1 loops, best of 3: 85.8 ms per loop
1 loops, best of 3: 1.15 s per loop

给定一个普通的Python，dict你可以Series通过调用来构造一个：

my_series = pd.Series(my_dict)

然后按值排序

my_series.sort()