使用高斯核密度(Python)计算值与值的平均值的差异

Question

我使用此代码计算此值的高斯核密度

from random import randint
x_grid=[]
for i in range(1000):
    x_grid.append(randint(0,4))
print (x_grid)

这是计算高斯核密度的代码

from statsmodels.nonparametric.kde import KDEUnivariate
import matplotlib.pyplot as plt

def kde_statsmodels_u(x, x_grid, bandwidth=0.2, **kwargs):
    """Univariate Kernel Density Estimation with Statsmodels"""
    kde = KDEUnivariate(x)
    kde.fit(bw=bandwidth, **kwargs)
    return kde.evaluate(x_grid)

import numpy as np
from scipy.stats.distributions import norm

# The grid we'll use for plotting
from random import randint
x_grid=[]
for i in range(1000):
    x_grid.append(randint(0,4))
print (x_grid)

# Draw points from a bimodal distribution in 1D
np.random.seed(0)
x = np.concatenate([norm(-1, 1.).rvs(400),
                    norm(1, 0.3).rvs(100)])

pdf_true = (0.8 * norm(-1, 1).pdf(x_grid) +
            0.2 * norm(1, 0.3).pdf(x_grid))

# Plot the three kernel density estimates
fig, ax = plt.subplots(1, 2, sharey=True, figsize=(13, 8))
fig.subplots_adjust(wspace=0)

pdf=kde_statsmodels_u(x, x_grid, bandwidth=0.2)
ax[0].plot(x_grid, pdf, color='blue', alpha=0.5, lw=3)
ax[0].fill(x_grid, pdf_true, ec='gray', fc='gray', alpha=0.4)
ax[0].set_title("kde_statsmodels_u")
ax[0].set_xlim(-4.5, 3.5)

plt.show()

网格中的所有值都在0 e 4之间.如果我收到新值5,我想计算该值与平均值的差异,并为其分配0到1之间的分数.(设置阈值)

因此,如果我收到新值5,其得分必须接近0.90,而如果我收到新值500则其得分必须接近0.0.

我怎样才能做到这一点？我的函数是计算高斯核密度正确还是有更好的方法/库来做到这一点？

*更新* 我在一篇论文中读到了一个例子.洗衣机的重量通常为100千克.通常供应商使用kg单位也参考其容量(例如9 kg).对于人类来说很容易理解,9 gk是洗衣机的容量而不是总重量.我们可以在没有深入语言理解的情况下"伪造"这种形式的智能,而是通过对每个属性的训练数据的值分布进行建模.

对于给定的属性a(例如洗衣机的重量),设Va = {va1,va2,... ..van}(| Va | = n)是对应于训练数据中的产品的属性a的值的集合.如果我发现了一个新值v直观地说它是"接近"(估计的分布)Va,那么我们应该更自信地将这个值分配给(例如洗衣机的重量).

一个想法可能是测量新值v与Va中的平均值不同的标准偏差的数量,但是更好的可以是对Va上的(高斯)核密度建模,然后以新值表示支持v作为该点的密度:

哪里？^(2)ak是第k个高斯的方差,Z是一个常数来确定S(csv,Va)？[0,1].如何使用statsmodels库在Python中获取它？

*更新2* 数据示例......但我认为这不是很重要...由此代码生成...

from random import randint
x_grid=[]
for i in range(1000):
    x_grid.append(randint(1,3))
print (x_grid)

[2,2,1,2,2,3,1,1,2,2,2,1,1,3,3,1,2,1,3,2,3,3,1,2] ,3,1,1,3,2,2,1,1,2,3,2,1,2,3,3,2,2,3,3,2,2,1,2,1 ,2,2,3,1,1,2,3,3,2,1,2,3,3,3,3,2,1,3,2,2,1,3,3,1 ,2,1,3,2,3,3,1,2,3,3,2,1,2,3,2,1,1,2,1,1,2,3,2,1,2 ,2,2,3,3,3,1,1,3,2,1,1,3,3,3,2,1,2,2,1,3,2,3,1,3 ,1,2,3,1,3,2,2,1,1,2,3,3,1,1,3,2,2,1,2,1,2,3,1,3,3 ,1,2,1,2,1,3,1,3,3,2,1,1,3,2,2,2,3,2,1,3,2,1,1,3,3 ,3,2,1,1,3,2,1,2,2,2,1,3,1,3,2,3,1,2,1,1,2,2,3,3,3 ,3,3,2,2,2,3,1,1,2,2,1,1,1,3,3,3,3,1,3,1,3,1,1,1,2 ,1,2,1,1,2,1,3,1,2,3,1,3,2,2,2,2,1,1,2,3,1,1,1,3 ,1,3,2,2,3,1,3,3,2,2,3,2,1,2,1,1,1,2,2,3,2,1,1,3,1 ,2,1,3,3,3,1,2,2,2,1,1,2,2,1,2,3,1,3,2,2,2,2,2,1,3,1,3,3,2,3,2,1,3,3,3, 3,3,1,2,2,2,1,1,3,2,3,1,2,3,2,3,2,1,1,3,3,1,1,2,3, 2,3,3,2,3,3,3,3,3,3,3,3,3,2,1,1,2,3,2,3,1,1,1,1, 2,2,2,1,1,2,2,1,3,1,1,2,3,1,1,2,3,1,2,3,1,2,1,3, 3,2,3,3,3,3,1,1,2,2,3,2,3,2,1,1,1,1,2,3,1,3,3,3, 2,1,2,3,1,2,1,1,2,3,3,1,1,3,2,1,3,3,2,1,1,3,1,3,1, 2,2,1,3,3,2,3,1,1,3,1,2,2,1,3,2,3,1,1,3,1,3,1,2,1, 3,2,2,2,2,1,3,2,1,3,3,2,3,2,1,3,1,2,1,2,3,2,3,2,3,3,2,2,2,2,3,3,2,1,3,3,2,3,2,1,3,3,2,3 3,2,3,3,1,1,3,2,3,2,2,3,3,3,3,2,3,3,3,3,3,2,2,3,3,3,3,3,2,3,2,3,3,3,3,2,3,3,3,3,3,2,2,3,3 3,1,3,2,3,1,1,2,1,3,1,2,2,3,3,1,3,1,1,2,2,1,3,3,3, 1,2,2,2,3,3,2,2,2,3,3,3,1,1,2,3,3,1,1,2,3,2,3,3, 2,2,1,3,3,3,3,2,3,1,3,3,2,1,3,2,1,1,3,3,2,2,2,1,1,1,1,2,3,3,3,2,1,3,1, 1,1,1,3,1,2,3,3,3,2,3,1,2,2,2,3,2,1,2,3,3,2,3,3,1, 2,3,3,3,3,2,3,3,2,1,1,1,2,3,1,3,3,2,1,3,3,3,2,2,1, 2,3,2,3,3,3,3,3,3,2,1,2,1,1,3,3,3,2,2,3,1,3,2,1,3, 1,1,3,3,1,2,2,3,3,1,2,1,2,1,3,2,3,3,3,3,3,3,3,1, 2,3,1,3,3,2,2,1,3,1,1,3,2,1,2,3,2,1,3,3,3,2,3,1,2, 3,3,1,2,2,3,3,2,1,1,1,3,1,3,1,3,3,2,3,1,3,2,3,3, 1,2,1,3,2,2,2,2,2,2,1,2,2,3,2,2,3,2,2,3,3,1,3,3,3,2,3,2,2,2,2,2,2,2,3,2,3,2,3 1,3,1,2,1,2,1,3,2,2,1,3,1,3,3,1,3,1,1,1,1,3,2,1,2, 3,1,1,3,1,1,3,1,3,3,3,1,1,3,1,3,2,2,2,1,1,2,3,3,2, 3,3,1,2,3,2,2,3,1,2,2,2,1,1,3,1,2,2,2,1,1,2,3,1,3, 1,1,3,2,2,3,2,2,3,3,1,1,2,3,3,1,2,3,2,2,3,1,2,2,1,1,3,2,3,1,1,3,1,3,2, 3,3,3,3,3,2,2,3,2,1,1,1,3,3,1,2,1,3,2,3,2,2,1,3,3, 3,1,1,1,1,3,3,1,3,3,1,1,3,1,3,1,3,2,3,1,3,3,3,1,1, 2,2,3,2,3,2,2,1,2,1,2,1,2,2,3,1,1,3,2,2,3,2,3,3,2, 2,2,2,2,3,3,3,2,2,1,1,2,3,3,1,3,3,1,3,3,1,3,2, 2,2,1,1,2,1,3,1,1,1,2,3,3,2,3,1,3]

这个阵列代表了市场上新智能手机的内存......通常它们有1,2,3 GB的内存.

那就是内核密度

***更新

我尝试使用此值的代码

[1024,1,1010,1000,1024,128,1536,16,192,2048,2000,2048,24,250,256,278,288,290,3072,3,3000,3072,32,384,4096 ,4,4096,448,45,512,576,64,768,8,96]

这些价值都是mb ......你觉得它运作良好吗？我认为我必须设定一个门槛

      100%      cdfv      kdev
1       42  0.210097  0.499734
1024    96  0.479597  0.499983
5000     0  0.000359  0.498885
2048    36  0.181609  0.499700
3048     8  0.040299  0.499424

*更新3*

[256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 512, 512, 512, 256, 256, 256, 512, 512, 512, 128, 128, 128, 512, 512, 512, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 2048, 2048, 2048, 1024, 1024, 1024, 512, 512, 512, 512, 512, 512, 1024, 1024, 1024, 2048, 2048, 2048, 512, 512, 512, 512, 512, 512, 512, 512, 512, 512, 512, 512, 512, 512, 512, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 512, 512, 512, 128, 128, 128, 512, 512, 512, 256, 256, 256, 256, 256, 256, 1024, 1024, 1024, 512, 512, 512, 128, 128, 128, 512, 512, 512, 512, 512, 512, 1024, 1024, 1024, 1024, 1024, 1024, 4, 4, 4, 3, 3, 3, 24, 24, 24, 8, 8, 8, 16, 16, 16, 16, 16, 16, 256, 256, 256, 512, 512, 512, 512, 512, 512, 512, 512, 512, 512, 512, 512, 512, 512, 512, 512, 512, 512, 512, 512, 512, 512, 512, 512, 1024, 1024, 1024, 1024, 1024, 1024, 512, 512, 512, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 2048, 2048, 2048, 512, 512, 512, 1024, 1024, 1024, 512, 512, 512, 1024, 1024, 1024, 2048, 2048, 2048, 2048, 2048, 2048, 512, 512, 512, 512, 512, 512, 256, 256, 256, 256, 256, 256, 256, 256, 256, 512, 512, 512, 512, 512, 512, 1024, 1024, 1024, 512, 512, 512, 512, 512, 512, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 2048, 2048, 2048, 2048, 2048, 2048, 2048, 2048, 2048, 2048, 2048, 2048, 2048, 2048, 2048, 2048, 2048, 2048, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 2048, 2048, 2048, 2048, 2048, 2048, 4096, 4096, 4096, 2048, 2048, 2048, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 2048, 2048, 2048, 2048, 2048, 2048, 2048, 2048, 2048, 2048, 2048, 2048, 2048, 2048, 2048, 512, 512, 512, 512, 512, 512, 512, 512, 512, 512, 512, 512, 512, 512, 512, 768, 768, 768, 768, 768, 768, 2048, 2048, 2048, 2048, 2048, 2048, 3072, 3072, 3072, 2048, 2048, 2048, 2048, 2048, 2048, 2048, 2048, 2048, 2048, 2048, 2048, 1024, 1024, 1024, 512, 512, 512, 256, 256, 256, 512, 512, 512, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 2048, 2048, 2048, 1024, 1024, 1024, 1024, 1024, 1024, 2048, 2048, 2048, 1024, 1024, 1024, 3072, 3072, 3072, 1024, 1024, 1024, 512, 512, 512, 1024, 1024, 1024, 1024, 1024, 1024, 512, 512, 512, 2048, 2048, 2048, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 2048, 2048, 2048, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 2048, 2048, 2048, 1024, 1024, 1024, 2048, 2048, 2048, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 512, 512, 512, 1024, 1024, 1024, 512, 512, 512, 512, 512, 512, 512, 512, 512, 1024, 1024, 1024, 1024, 1024, 1024, 512, 512, 512, 1024, 1024, 1024, 512, 512, 512, 1024, 1024, 1024, 512, 512, 512, 512, 512, 512, 512, 512, 512, 256, 256, 256, 1024, 1024, 1024, 2048, 2048, 2048, 1024, 1024, 1024, 1024, 1024, 1024, 512, 512, 512, 512, 512, 512, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 2048, 2048, 2048, 1024, 1024, 1024, 2048, 2048, 2048, 1024, 1024, 1024, 512, 512, 512, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 2048, 2048, 2048, 2048, 2048, 2048, 2048, 2048, 2048, 1024, 1024, 1024, 2048, 2048, 2048, 512, 512, 512, 512, 512, 512, 1024, 1024, 1024, 1024, 1024, 1024, 512, 512, 512, 64, 64, 64, 1024, 1024, 1024, 1024, 1024, 1024, 256, 256, 256, 512, 512, 512, 512, 512, 512, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 64, 64, 64, 64, 64, 64, 128, 128, 128, 128, 128, 128, 128, 128, 128, 128, 128, 128, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 128, 128, 128, 576, 576, 576, 512, 512, 512, 1024, 1024, 1024, 512, 512, 512, 576, 576, 576, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 2048, 2048, 2048, 512, 512, 512, 2048, 2048, 2048, 768, 768, 768, 768, 768, 768, 768, 768, 768, 512, 512, 512, 192, 192, 192, 1024, 1024, 1024, 512, 512, 512, 512, 512, 512, 384, 384, 384, 448, 448, 448, 576, 576, 576, 384, 384, 384, 288, 288, 288, 768, 768, 768, 384, 384, 384, 288, 288, 288, 64, 64, 64, 2048, 2048, 2048, 2048, 2048, 2048, 2048, 2048, 2048, 3072, 3072, 3072, 2048, 2048, 2048, 2048, 2048, 2048, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 512, 512, 512, 1024, 1024, 1024, 64, 64, 64, 128, 128, 128, 128, 128, 128, 128, 128, 128, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 256, 256, 256, 768, 768, 768, 768, 768, 768, 768, 768, 768, 256, 256, 256, 192, 192, 192, 256, 256, 256, 64, 64, 64, 256, 256, 256, 192, 192, 192, 128, 128, 128, 256, 256, 256, 192, 192, 192, 288, 288, 288, 288, 288, 288, 288, 288, 288, 288, 288, 288, 128, 128, 128, 128, 128, 128, 384, 384, 384, 512, 512, 512, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 2048, 2048, 2048, 2048, 2048, 2048, 2048, 2048, 2048, 3072, 3072, 3072, 1024, 1024, 1024, 2048, 2048, 2048, 2048, 2048, 2048, 3072, 3072, 3072, 512, 512, 512, 512, 512, 512, 512, 512, 512, 1024, 1024, 1024, 1024, 1024, 1024, 512, 512, 512, 1024, 1024, 1024, 512, 512, 512, 512, 512, 512, 512, 512, 512, 1024, 1024, 1024, 1024, 1024, 1024, 32, 32, 32, 768, 768, 768, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 2048, 2048, 2048, 1024, 1024, 1024, 2048, 2048, 2048, 3072, 3072, 3072, 2048, 2048, 2048, 1024, 1024, 1024, 2048, 2048, 2048, 1024, 1024, 1024, 2048, 2048, 2048, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 256, 512, 512, 512, 512, 512, 512, 256, 256, 256, 512, 512, 512, 512, 512, 512, 512, 512, 512, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 128, 128, 128, 128, 128, 128, 1024, 1024, 1024, 1024, 1024, 1024, 128, 128, 128, 1024, 1024, 1024, 2048, 2048, 2048, 1024, 1024, 1024, 1024, 1024, 1024, 2048, 2048, 2048, 3072, 3072, 3072, 1024, 1024, 1024, 1024, 1024, 1024, 512, 512, 512, 512, 512, 512, 512, 512, 512, 512, 512, 512, 2048, 2048, 2048, 1024, 1024, 1024, 2048, 2048, 2048, 1024, 1024, 1024, 1024, 1024, 1024, 512, 512, 512, 512, 512, 512, 1024, 1024, 1024, 512, 512, 512, 1024, 1024, 1024, 512, 512, 512, 512, 512, 512, 512, 512, 512, 1024, 1024, 1024, 2048, 2048, 2048, 2048, 2048, 2048, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 2048, 2048, 2048, 2048, 2048, 2048, 512, 512, 512, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 256, 256, 256, 256, 256, 256, 512, 512, 512, 512, 512, 512, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 2048, 2048, 2048, 2048, 2048, 2048, 2048, 2048, 2048, 3072, 3072, 3072, 2048, 2048, 2048, 384, 384, 384, 512, 512, 512, 512, 512, 512, 512, 512, 512, 512, 512, 512, 512, 512, 512, 512, 512, 512, 512, 512, 512, 512, 512, 512, 2048, 2048, 2048, 2048, 2048, 2048, 2048, 2048, 2048, 1024, 1024, 1024, 2048, 2048, 2048, 1024, 1024, 1024, 3072, 3072, 3072, 3072, 3072, 3072, 3072, 3072, 3072, 128, 128, 128, 256, 256, 256, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 2048, 2048, 2048, 512, 512, 512, 1024, 1024, 1024, 1024, 1024, 1024, 2048, 2048, 2048, 512, 512, 512, 512, 512, 512, 512, 512, 512, 512, 512, 512, 768, 768, 768, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 512, 512, 512, 1024, 1024, 1024, 128, 128, 128, 512, 512, 512, 1024, 1024, 1024, 512, 512, 512, 1024, 1024, 1024, 512, 512, 512, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 2048, 2048, 2048, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 2048, 2048, 2048, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 512, 512, 512, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 64, 64, 64, 64, 64, 64, 256, 256, 256, 512, 512, 512, 512, 512, 512, 512, 512, 512, 16, 16, 16, 3072, 3072, 3072, 3072, 3072, 3072, 256, 256, 256, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 2048, 2048, 2048, 2048, 2048, 2048, 2048, 2048, 2048, 512, 512, 512, 32, 32, 32, 1024, 1024, 1024, 1024, 1024, 1024, 256, 256, 256, 256, 256, 256, 512, 512, 512, 512, 512, 512, 512, 512, 512, 512, 512, 512, 512, 512, 512, 512, 512, 512, 1024, 1024, 1024, 1024, 1024, 1024, 512, 512, 512, 512, 512, 512, 512, 512, 512, 512, 512, 512, 512, 512, 512, 1024, 1024, 1024, 512, 512, 512, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 32, 32, 32, 2048, 2048, 2048, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 2048, 2048, 2048, 512, 512, 512, 1, 1, 1, 1024, 1024, 1024, 32, 32, 32, 32, 32, 32, 45, 45, 45, 8, 8, 8, 512, 512, 512, 256, 256, 256, 512, 512, 512, 512, 512, 512, 512, 512, 512, 512, 512, 512, 1024, 1024, 1024, 512, 512, 512, 512, 512, 512, 512, 512, 512, 512, 512, 512, 512, 512, 512, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 512, 512, 512, 16, 16, 16, 4, 4, 4, 4, 4, 4, 4, 4, 4, 16, 16, 16, 16, 16, 16, 16, 16, 16, 64, 64, 64, 8, 8, 8, 8, 8, 8, 8, 8, 8, 64, 64, 64, 64, 64, 64, 256, 256, 256, 64, 64, 64, 64, 64, 64, 512, 512, 512, 512, 512, 512, 512, 512, 512, 32, 32, 32, 32, 32, 32, 32, 32, 32, 128, 128, 128, 128, 128, 128, 128, 128, 128, 32, 32, 32, 128, 128, 128, 64, 64, 64, 64, 64, 64, 16, 16, 16, 256, 256, 256, 2048, 2048, 2048, 1024, 1024, 1024, 2048, 2048, 2048, 256, 256, 256, 512, 512, 512, 1024, 1024, 1024, 512, 512, 512, 256, 256, 256, 512, 512, 512, 512, 512, 512, 512, 512, 512, 512, 512, 512, 1024, 1024, 1024, 512, 512, 512, 512, 512, 512, 1024, 1024, 1024, 1024, 1024, 1024, 512, 512, 512, 1024, 1024, 1024, 1024, 1024, 1024, 512, 512, 512, 1024, 1024, 1024, 1024, 1024, 1024, 2048, 2048, 2048, 256, 256, 256, 256, 256, 256, 1024, 1024, 1024, 1024, 1024, 1024, 256, 256, 256, 3072, 3072, 3072, 3072, 3072, 3072, 128, 128, 128, 1024, 1024, 1024, 512, 512, 512, 512, 512, 512, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 128, 128, 128, 128, 128, 128, 64, 64, 64, 256, 256, 256, 256, 256, 256, 512, 512, 512, 768, 768, 768, 768, 768, 768, 16, 16, 16, 32, 32, 32, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 2048, 2048, 2048, 2048, 2048, 2048, 1024, 1024, 1024, 2048, 2048, 2048, 1024, 1024, 1024, 512, 512, 512, 2048, 2048, 2048, 1024, 1024, 1024, 3072, 3072, 3072, 3072, 3072, 3072, 2048, 2048, 2048, 1024, 1024, 1024, 1024, 1024, 1024, 3072, 3072, 3072, 3072, 3072, 3072, 3072, 3072, 3072, 3072, 3072, 3072, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 512, 512, 512, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 2048, 2048, 2048, 1024, 1024, 1024, 2048, 2048, 2048, 2048, 2048, 2048, 2048, 2048, 2048, 2048, 2048, 2048, 2048, 2048, 2048, 2048, 2048, 2048, 2048, 2048, 2048, 2048, 2048, 2048, 3072, 3072, 3072, 3072, 3072, 3072, 512, 512, 512, 512, 512, 512, 512, 512, 512, 512, 512, 512, 512, 512, 512, 1024, 1024, 1024, 512, 512, 512, 64, 64, 64, 96, 96, 96, 512, 512, 512, 64, 64, 64, 64, 64, 64, 512, 512, 512, 512, 512, 512, 512, 512, 512, 512, 512, 512, 512, 512, 512, 512, 512, 512, 1024, 1024, 1024, 512, 512, 512, 512, 512, 512, 1024, 1024, 1024, 512, 512, 512, 512, 512, 512, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 512, 512, 512, 512, 512, 512, 1024, 1024, 1024, 1024, 1024, 1024, 512, 512, 512, 512, 512, 512, 1024, 1024, 1024, 512, 512, 512, 1024, 1024, 1024, 1024, 1024, 1024, 512, 512, 512, 512, 512, 512, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 512, 512, 512, 512, 512, 512, 1024, 1024, 1024, 2048, 2048, 2048, 2048, 2048, 2048, 2048, 2048, 2048, 3072, 3072, 3072, 3072, 3072, 3072, 2048, 2048, 2048, 2048, 2048, 2048, 2048, 2048, 2048, 512, 512, 512, 1024, 1024, 1024, 2048, 2048, 2048, 1024, 1024, 1024, 1024, 1024, 1024, 512, 512, 512, 1024, 1024, 1024, 1024, 1024, 1024, 512, 512, 512, 1024, 1024, 1024, 512, 512, 512, 1024, 1024, 1024, 1024, 1024, 1024, 2048, 2048, 2048, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 2048, 2048, 2048, 64, 64, 64, 64, 64, 64, 256, 256, 256, 1024, 1024, 1024, 512, 512, 512, 256, 256, 256, 512, 512, 512, 1024, 1024, 1024, 512, 512, 512, 512, 512, 512, 1024, 1024, 1024, 1024, 1024, 1024, 2048, 2048, 2048, 2048, 2048, 2048, 512, 512, 512, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 2048, 2048, 2048, 2048, 2048, 2048, 2048, 2048, 2048, 2048, 2048, 2048, 2048, 2048, 2048, 2048, 2048, 2048, 3072, 3072, 3072, 3072, 3072, 3072, 2048, 2048, 2048, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 2048, 2048, 2048, 2048, 2048, 2048, 1024, 1024, 1024, 2048, 2048, 2048, 3072, 3072, 3072, 2048, 2048, 2048]

如果我尝试将此数字作为新值,则使用此数据

# new values
x = np.asarray([128,512,1024,2048,3072,2800])

3072出现问题(所有值均以MB为单位).

这是结果:

      100%      cdfv      kdev
128     26  0.129688  0.499376
512     55  0.275874  0.499671
1024    91  0.454159  0.499936
2048    12  0.062298  0.499150
3072     0  0.001556  0.498364
2800     1  0.004954  0.498573

我无法理解为什么会发生这种情况...... 3072值在数据中出现了很多时间......这是我数据的直方图...这很奇怪,因为有一些值为3072,也有4096 .

Answer 1

一些一般性评论，但不涉及统计模型的详细信息。

statsmodels 也有 cdf 内核，但我不记得它们的工作效果如何，而且我不认为它有自动带宽选择。

与 ali_m 在评论中链接到的 glen_b 的答案相关：

随着样本的增长，cdf 估计值比密度估计值更快地收敛到真实分布。为了平衡偏差 - 方差权衡，我们应该对 cdf 核使用较小的带宽，这相对于密度估计来说不够平滑。估计值应该比相应的密度估计值更准确。

尾部观察次数：

如果样本中最大的观测值是 4 并且您想知道 5 处的 cdf，那么您的数据没有任何相关信息。对于只有很少观测值的尾部，非参数估计量（如核分布估计量）的方差相对而言会很大（是 1e-5 还是 1e-20？）。

作为核密度或核分布估计的替代方法，我们可以估计尾部的帕累托分布。例如，取最大的 10% 或 20% 的观测值并拟合 Pareto 分布，并使用它来推断尾部密度。有几个用于幂律估计的 Python 包可以用于此目的。

更新

下面展示了如何使用参数正态分布假设和固定带宽的高斯核密度估计来计算“异常度”。

仅当样本来自连续分布或可以通过连续分布近似时，这才是真正正确的。在这里，我们假设只有 3 个不同值的样本来自正态分布。本质上，计算出的 cdf 值就像距离度量，而不是离散随机变量的概率。

这使用 scipy.stats 中具有固定带宽的 kde，而不是 statsmodels 版本。

我不确定 scipy 的 gaussian_kde 中的带宽是如何设置的，所以，我的固定带宽选择等于scale可能是错误的。如果只有三个不同的值，我不知道如何选择带宽，数据中没有足够的信息。默认带宽适用于近似正态或至少单峰的分布。

import numpy as np
from scipy import stats

# data
ram = np.array([2, <truncated from data in description>, 3])

loc = ram.mean()
scale = ram.std()

# new values
x = np.asarray([-1, 0, 2, 3, 4, 5, 100])

# assume normal distribution
cdf_val = stats.norm.cdf(x, loc=loc, scale=scale)
cdfv = np.minimum(cdf_val, 1 - cdf_val)

# use gaussian kde but fix bandwidth
kde = stats.gaussian_kde(ram, bw_method=scale)
kde_val = np.asarray([kde.integrate_box_1d(-np.inf, xx) for xx in  x])
kdev = np.minimum(kde_val, 1 - kde_val)


#print(np.column_stack((x, cdfv, kdev)))
# use pandas for prettier table
import pandas as pd
print(pd.DataFrame({'cdfv': cdfv, 'kdev': kdev}, index=x))

'''
          cdfv      kdev
-1    0.000096  0.000417
 0    0.006171  0.021262
 2    0.479955  0.482227
 3    0.119854  0.199565
 5    0.000143  0.000472
 100  0.000000  0.000000
 '''