熊猫 - 过滤所有列

Question

我在pandas中有一个方形相关矩阵,我试图用最有效的方法来返回值(总是一个浮点数-1 <= x <= 1)高于某个阈值的所有值.

该pandas.DataFrame.filter方法请求列的列表或一个正则表达式,但我总是想传递中的所有列.是否有一个最佳实践呢？

Answer 1

有两种方法可以解决这个问题:

假设:

In [7]: c = np.array([-1,-2,-2,-3,-4,-6,-7,-8])

In [8]: a = np.array([1,2,3,4,6,7,8,9])

In [9]: b = np.array([2,4,6,8,10,12,13,15])

In [10]: c = np.array([-1,-2,-2,-3,-4,-6,-7,-8])

In [11]: corr = np.corrcoef([a,b,c])

In [12]: df = pd.DataFrame(corr)

In [13]: df
Out[13]:
          0         1         2
0  1.000000  0.995350 -0.980521
1  0.995350  1.000000 -0.971724
2 -0.980521 -0.971724  1.000000

然后你可以简单地说:

In [14]: df > 0.5
Out[14]:
       0      1      2
0   True   True  False
1   True   True  False
2  False  False   True

In [15]: df[df > 0.5]
Out[15]:
         0        1    2
0  1.00000  0.99535  NaN
1  0.99535  1.00000  NaN
2      NaN      NaN  1.0

如果只想要这些值,那么最简单的方法是使用values属性处理底层的numpy数据结构:

In [17]: df.values
Out[17]:
array([[ 1.        ,  0.99535001, -0.9805214 ],
       [ 0.99535001,  1.        , -0.97172394],
       [-0.9805214 , -0.97172394,  1.        ]])

In [18]: df.values[(df > 0.5).values]
Out[18]: array([ 1.        ,  0.99535001,  0.99535001,  1.        ,  1.        ])

而不是.values像ayhan所指出的那样,你可以使用stack它自动掉落NaN并保留标签......

In [22]: df.index = ['a','b','c']

In [23]: df.columns=['a','b','c']

In [24]: df
Out[24]:
          a         b         c
a  1.000000  0.995350 -0.980521
b  0.995350  1.000000 -0.971724
c -0.980521 -0.971724  1.000000


In [25]: df.stack() > 0.5
Out[25]:
a  a     True
   b     True
   c    False
b  a     True
   b     True
   c    False
c  a    False
   b    False
   c     True
dtype: bool

In [26]: df.stack()[df.stack() > 0.5]
Out[26]:
a  a    1.00000
   b    0.99535
b  a    0.99535
   b    1.00000
c  c    1.00000
dtype: float64

你总能回去......

In [29]: (df.stack()[df.stack() > 0.5]).unstack()
Out[29]:
         a        b    c
a  1.00000  0.99535  NaN
b  0.99535  1.00000  NaN
c      NaN      NaN  1.0

Answer 2

不确定你想要的输出是什么,因为你没有提供样品,但我会给你我的两分钱我会做什么:

In[1]:
import pandas as pd
import numpy as np

df = pd.DataFrame(np.random.rand(10,5))  
corr = df.corr()
corr.shape

Out[1]: (5, 5)

现在,让我们提取相关矩阵的上三角形(它是对称的),不包括对角线.为此,我们将使用np.tril,将其转换为布尔值,并使用~运算符与其相反.

In [2]: corr_triu = corr.where(~np.tril(np.ones(corr.shape)).astype(np.bool))
         corr_triu
Out[2]: 
    0         1         2         3         4
0 NaN  0.228763 -0.276406  0.286771 -0.050825
1 NaN       NaN -0.562459 -0.596057  0.540656
2 NaN       NaN       NaN  0.402752  0.042400
3 NaN       NaN       NaN       NaN -0.642285
4 NaN       NaN       NaN       NaN       NaN

现在让我们堆叠它并过滤上面的所有值,0.3例如:

In [3]: corr_triu = corr_triu.stack()
        corr_triu[corr_triu > 0.3]
Out[3]: 
1  4    0.540656
2  3    0.402752
dtype: float64

如果你想让它更漂亮一点:

In [4]: corr_triu.name = 'Pearson Correlation Coefficient'
        corr_triu.index.names = ['Col1', 'Col2']

In [5]: corr_triu[corr_triu > 0.3].to_frame()
Out[5]: 
           Pearson Correlation Coefficient
Col1 Col2                   
1    4              0.540656
2    3              0.402752