对具有 NaN 的列进行缩尾处理不会更改最大值

Question

请注意，不久前有人提出了类似的问题，但从未得到解答（请参阅Winsorizing does not change the max value）。

我正在尝试使用fromwinsorize数据框中的一列。如果列中没有 NaN 值，则该过程正常运行。winsorizescipy.stats.mstats

然而，NaN 值似乎阻止该过程在分布的顶部（而不是底部）工作。无论我设置什么值nan_policy，NaN 值都会设置为分布中的最大值。我觉得一定是错误地设置了该选项。

下面是一个示例，可用于重现没有 NaN 值时正确的缩尾处理以及存在 NaN 值时我遇到的问题行为。任何有关解决此问题的帮助将不胜感激。

#Import
import pandas as pd
import numpy as np
from scipy.stats.mstats import winsorize

# initialise data of lists.
data = {'Name':['A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K', 'L', 'M', 'N', 'O', 'P', 'Q', 'R', 'S', 'T'], 'Age':[1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0, 10.0, 11.0, 12.0, 13.0, 14.0, 15.0, 16.0, 17.0, 18.0, 19.0, 20.0]}
 
# Create 2 DataFrames
df = pd.DataFrame(data)
df2 = pd.DataFrame(data)

# Replace two values in 2nd DataFrame with np.nan
df2.loc[5,'Age'] = np.nan
df2.loc[8,'Age'] = np.nan

# Winsorize Age in both DataFrames
winsorize(df['Age'], limits=[0.1, 0.1], inplace = True, nan_policy='omit')
winsorize(df2['Age'], limits=[0.1, 0.1], inplace = True, nan_policy='omit')

# Check min and max values of Age in both DataFrames
print('Max/min value of Age from dataframe without NaN values')
print(df['Age'].max())
print(df['Age'].min())

print()

print('Max/min value of Age from dataframe with NaN values')
print(df2['Age'].max())
print(df2['Age'].min())

Answer 1

看起来好像nan_policy被忽略了。但缩尾化只是剪裁，所以你可以用 pandas 来处理这个问题。

def winsorize_with_pandas(s, limits):
    """
    s : pd.Series
        Series to winsorize
    limits : tuple of float
        Tuple of the percentages to cut on each side of the array, 
        with respect to the number of unmasked data, as floats between 0. and 1
    """
    return s.clip(lower=s.quantile(limits[0], interpolation='lower'), 
                  upper=s.quantile(1-limits[1], interpolation='higher'))


winsorize_with_pandas(df['Age'], limits=(0.1, 0.1))
0      3.0
1      3.0
2      3.0
3      4.0
4      5.0
5      6.0
6      7.0
7      8.0
8      9.0
9     10.0
10    11.0
11    12.0
12    13.0
13    14.0
14    15.0
15    16.0
16    17.0
17    18.0
18    18.0
19    18.0
Name: Age, dtype: float64

winsorize_with_pandas(df2['Age'], limits=(0.1, 0.1))
0      2.0
1      2.0
2      3.0
3      4.0
4      5.0
5      NaN
6      7.0
7      8.0
8      NaN
9     10.0
10    11.0
11    12.0
12    13.0
13    14.0
14    15.0
15    16.0
16    17.0
17    18.0
18    19.0
19    19.0
Name: Age, dtype: float64