熊猫的输出不一致，应用了功能，参数为np.std

Question

我sklearn.preprocessing.StandardScaler用来重新缩放数据。我想用来np.std做同样的事情StandardScaler。

但是，我发现一个有趣的事情，没有传入其他参数pandas.apply(fun = np.std)，输出在样本std和总体std之间变化。（请参阅2问题）

我知道有一个参数叫来ddof在计算样本方差时控制除数，不更改默认参数ddof = 0，如何获得不同的输出！

1个数据集：

首先，以虹膜数据集为例。我按如下方式缩放数据的第一列。

from sklearn import datasets
import numpy as np
from sklearn.preprocessing import StandardScaler
iris = datasets.load_iris()
X_train = iris.data[:,[1]] # my X_train is the first column if iris data
sc = StandardScaler() 
sc.fit(X_train) # Using StandardScaler to scale it!

2问题：在没有更改默认值的情况下，ddof = 0我得到了np.std的不同输出！

import pandas as pd
import sys
print("The mean and std(sample std) of X_train is :")
print(pd.DataFrame(X_train).apply([np.mean,np.std],axis = 0),"\n")

print("The std(population std) of X_train is :")
print(pd.DataFrame(X_train).apply(np.std,axis = 0),"\n") 

print("The std(population std) of X_train is :","{0:.6f}".format(sc.scale_[0]),'\n') 

print("Python version:",sys.version,
      "\npandas version:",pd.__version__,
      "\nsklearn version:",sklearn.__version__)

出：

The mean and std(sample std) of X_train is :
             0
mean  3.057333
std   0.435866 

The std(population std) of X_train is :
0    0.434411
dtype: float64 

The std(population std) of X_train is : 0.434411 

Python version: 3.7.1 (default, Dec 10 2018, 22:54:23) [MSC v.1915 64 bit (AMD64)] 
pandas version: 0.23.4 
sklearn version: 0.20.1

根据以上结果，pd.DataFrame(X_train).apply([np.mean,np.std],axis = 0)样本标准为0.435866，pd.DataFrame(X_train).apply(np.std,axis = 0)人口总体标准为0.434411。

3我的问题：

1. 为什么使用pandas.apply返回不同的结果？

2. 如何将附加参数传递给np.std，以提供总体std？

pd.DataFrame(X_train).apply(np.std,ddof = 1)可以做到。但我想知道pd.DataFrame(X_train).apply([np.mean,np.std],**args)

Answer 1

.apply()这种行为的原因可以在系列的（也许不优雅）评估中找到。如果你查看源代码，你会发现以下几行：

if isinstance(func, (list, dict)):
    return self.aggregate(func, *args, **kwds)

这意味着：如果您调用apply([func])，结果可能与不同apply(func)！关于np.std，我建议使用内置df.std()方法或者df.describe().

您可以尝试以下代码以了解哪些有效，哪些无效：

import numpy as np
import pandas as pd

print(10*"-","Showing ddof impact",10*"-")

print(np.std([4,5], ddof=0)) # 0.5      ## N   (population's standard deviation)
print(np.std([4,5], ddof=1)) # 0.707... # N-1 (unbiased sample variance)

x = pd.Series([4,5])

print(10*"-","calling builtin .std() on Series",10*"-")
print(x.std(ddof=0)) # 0.5
print(x.std()) # 0.707

df=pd.DataFrame([[4,5],[5,6]], columns=['A', 'B'])

print(10*"-","calling builtin .std() on DF",10*"-")

print(df["A"].std(ddof=0))# 0.5
print(df["B"].std(ddof=0))# 0.5
print(df["A"].std())# 0.707
print(df["B"].std())# 0.707

print(10*"-","applying np.std to whole DF",10*"-")
print(df.apply(np.std,ddof=0)) # A = 0.5,  B = 0.5
print(df.apply(np.std,ddof=1)) # A = 0.707 B = 0.707

# print(10*"-","applying [np.std] to whole DF WONT work",10*"-")
# print(df.apply([np.std],axis=0,ddof=0)) ## this WONT Work
# print(df.apply([np.std],axis=0,ddof=1)) ## this WONT Work

print(10*"-","applying [np.std] to DF columns",10*"-")
print(df["A"].apply([np.std])) # 0.707
print(df["A"].apply([np.std],ddof=1)) # 0.707

print(10*"-","applying np.std to DF columns",10*"-")
print(df["A"].apply(np.std)) # 0: 0 1: 0 WHOOPS !! #<---------------------
print(30*"-")

apply您还可以通过ing 自己的函数来了解发生了什么：

def myFun(a):
    print(type(a))
    return np.std(a,ddof=0)

print("> 0",20*"-")    
print(x.apply(myFun))
print("> 1",20*"-","## <- only this will be applied to the Series!")
print(df.apply(myFun))
print("> 2",20*"-","## <- this will be applied to each Int!")
print(df.apply([myFun]))
print("> 3",20*"-")
print(df["A"].apply(myFun))
print("> 4",20*"-")
print(df["A"].apply([myFun]))