如何在具有大量不同计数的pandas中绘制value_counts,而不是均匀分布

Question

可以说我有以下数据:

s2 = pd.Series([1,2,3,4,5,2,3,333,2,123,434,1,2,3,1,11,11,432,3,2,4,3,3,3,54,34,24,2,223,2535334,3,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,30000, 2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2])
s2.value_counts(normalize=True).plot()

我想在图中显示的是,有一些数字构成了大多数情况.问题是这将在图的最左侧看到,然后将有一条直线为所有其他类别.在实际数据中,x轴将是大约18000个类别的分类,4%的计数将在10000左右,然后其余的将下降并且大约为50.

我想向"普通"商务人士展示这一点,因此无法通过一些难以阅读的解决方案.

更新:请参阅@unutbu answere更新的代码,并qcut在尝试使用元组时出现错误.

TypeError: unsupported operand type(s) for -: 'tuple' and 'tuple'

df = pd.DataFrame({'s1':[1,0,1,0], 's2':[1,0,1,1], 's3':[1,0,1,1], 's4':[0,0,0,1]})
perms = df.apply(tuple, axis=1)
prob = perms.value_counts(normalize=True).reset_index(drop='True')
category_classes = pd.qcut(prob, q=[0, .25, 0.95, 1.], 
                 labels=['bottom 25%', 'mid 70%', 'top 5%'])
prob_groups = prob.groupby(category_classes).sum()
prob_groups.plot(kind='bar')
plt.xticks(rotation=0)
plt.show()

Answer 1

您可以将标准化值计数保持在某个值以上threshold.然后将下面的值加在一起,threshold并将它们聚集在一个可以被称为"其他"的类别中.

通过选择threshold足够高,您将能够显示整体概率分布的最重要贡献者,同时仍然在标记为"其他"的栏中显示尾部的大小:

import matplotlib.pyplot as plt
import pandas as pd
s2 = pd.Series([1,2,3,4,5,2,3,333,2,123,434,1,2,3,1,11,11,432,3,2,4,3,3,3,54,34,24,2,223,2535334,3,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,30000, 2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2])
prob = s2.value_counts(normalize=True)
threshold = 0.02
mask = prob > threshold
tail_prob = prob.loc[~mask].sum()
prob = prob.loc[mask]
prob['other'] = tail_prob
prob.plot(kind='bar')
plt.xticks(rotation=25)
plt.show()

您可以在条形图上明显显示的类别标签数量有限制.对于正常大小的图表来说,3000太多了.此外,期望观众从阅读3000个标签中收集任何意义可能是不合理的.

该图应总结数据.主要观点似乎是4%或5%的类别占绝大多数案例.所以,开车回家这一点,也许使用pd.qcut的情况下,分类为简单的分类,例如bottom 25%,mid 70%和top 5%:

import numpy as np
import matplotlib.pyplot as plt
import pandas as pd

N = 18000
categories = np.arange(N)
np.random.shuffle(categories)
M = int(N*0.04)
prob = pd.Series(np.concatenate([np.random.randint(9000, 11000, size=M),
                      np.random.randint(0, 100, size=N-M), ]), index=categories)
prob /= prob.sum()
category_classes = pd.qcut(prob, q=[0, .25, 0.95, 1.], 
                 labels=['bottom 25%', 'mid 70%', 'top 5%'])
prob_groups = prob.groupby(category_classes).sum()
prob_groups.plot(kind='bar')
plt.xticks(rotation=0)
plt.show()