向量化A列B列的百分位值(对于组)

Question

向量化A列B列的百分位值(对于组)

cod*_*nob 12 python apply scipy percentile pandas

对于每对src和dest机场城市,我想在a给定列值的情况下返回列的百分位数b.

我可以手动执行此操作:

示例df只有2对src/dest(我的实际df中有数千个):

dt  src dest    a   b
0   2016-01-01  YYZ SFO 548.12  279.28
1   2016-01-01  DFW PDX 111.35  -65.50
2   2016-02-01  YYZ SFO 64.84   342.35
3   2016-02-01  DFW PDX 63.81   61.64
4   2016-03-01  YYZ SFO 614.29  262.83

{'a': {0: 548.12,
  1: 111.34999999999999,
  2: 64.840000000000003,
  3: 63.810000000000002,
  4: 614.28999999999996,
  5: -207.49000000000001,
  6: 151.31999999999999,
  7: -56.43,
  8: 611.37,
  9: -296.62,
  10: 6417.5699999999997,
  11: -376.25999999999999,
  12: 465.12,
  13: -821.73000000000002,
  14: 1270.6700000000001,
  15: -1410.0899999999999,
  16: 1312.6600000000001,
  17: -326.25999999999999,
  18: 1683.3699999999999,
  19: -24.440000000000001,
  20: 583.60000000000002,
  21: -5.2400000000000002,
  22: 1122.74,
  23: 195.21000000000001,
  24: 97.040000000000006,
  25: 133.94},
 'b': {0: 279.27999999999997,
  1: -65.5,
  2: 342.35000000000002,
  3: 61.640000000000001,
  4: 262.82999999999998,
  5: 115.89,
  6: 268.63999999999999,
  7: 2.3500000000000001,
  8: 91.849999999999994,
  9: 62.119999999999997,
  10: 778.33000000000004,
  11: -142.78,
  12: 1675.53,
  13: -214.36000000000001,
  14: 983.80999999999995,
  15: -207.62,
  16: 632.13999999999999,
  17: -132.53,
  18: 422.36000000000001,
  19: 13.470000000000001,
  20: 642.73000000000002,
  21: -144.59999999999999,
  22: 213.15000000000001,
  23: -50.200000000000003,
  24: 338.27999999999997,
  25: -129.69},
 'dest': {0: 'SFO',
  1: 'PDX',
  2: 'SFO',
  3: 'PDX',
  4: 'SFO',
  5: 'PDX',
  6: 'SFO',
  7: 'PDX',
  8: 'SFO',
  9: 'PDX',
  10: 'SFO',
  11: 'PDX',
  12: 'SFO',
  13: 'PDX',
  14: 'SFO',
  15: 'PDX',
  16: 'SFO',
  17: 'PDX',
  18: 'SFO',
  19: 'PDX',
  20: 'SFO',
  21: 'PDX',
  22: 'SFO',
  23: 'PDX',
  24: 'SFO',
  25: 'PDX'},
 'dt': {0: Timestamp('2016-01-01 00:00:00'),
  1: Timestamp('2016-01-01 00:00:00'),
  2: Timestamp('2016-02-01 00:00:00'),
  3: Timestamp('2016-02-01 00:00:00'),
  4: Timestamp('2016-03-01 00:00:00'),
  5: Timestamp('2016-03-01 00:00:00'),
  6: Timestamp('2016-04-01 00:00:00'),
  7: Timestamp('2016-04-01 00:00:00'),
  8: Timestamp('2016-05-01 00:00:00'),
  9: Timestamp('2016-05-01 00:00:00'),
  10: Timestamp('2016-06-01 00:00:00'),
  11: Timestamp('2016-06-01 00:00:00'),
  12: Timestamp('2016-07-01 00:00:00'),
  13: Timestamp('2016-07-01 00:00:00'),
  14: Timestamp('2016-08-01 00:00:00'),
  15: Timestamp('2016-08-01 00:00:00'),
  16: Timestamp('2016-09-01 00:00:00'),
  17: Timestamp('2016-09-01 00:00:00'),
  18: Timestamp('2016-10-01 00:00:00'),
  19: Timestamp('2016-10-01 00:00:00'),
  20: Timestamp('2016-11-01 00:00:00'),
  21: Timestamp('2016-11-01 00:00:00'),
  22: Timestamp('2016-12-01 00:00:00'),
  23: Timestamp('2016-12-01 00:00:00'),
  24: Timestamp('2017-01-01 00:00:00'),
  25: Timestamp('2017-01-01 00:00:00')},
 'src': {0: 'YYZ',
  1: 'DFW',
  2: 'YYZ',
  3: 'DFW',
  4: 'YYZ',
  5: 'DFW',
  6: 'YYZ',
  7: 'DFW',
  8: 'YYZ',
  9: 'DFW',
  10: 'YYZ',
  11: 'DFW',
  12: 'YYZ',
  13: 'DFW',
  14: 'YYZ',
  15: 'DFW',
  16: 'YYZ',
  17: 'DFW',
  18: 'YYZ',
  19: 'DFW',
  20: 'YYZ',
  21: 'DFW',
  22: 'YYZ',
  23: 'DFW',
  24: 'YYZ',
  25: 'DFW'}}

Run Code Online (Sandbox Code Playgroud)

我想要每组src和dest对的百分位数.因此,每对只应有1百分位数值.我只是想给执行百分b其中date = 2017-01-01每个src和dest对整个列a的一对.合理？

我可以手动执行此操作,例如针对特定对i.e. src=YYZ and dest=SFT:

from scipy import stats
import datetime as dt
import pandas as pd

p0 = dt.datetime(2017,1,1)

# lets slice df for src=YYZ and dest = SFO
x = df[(df.src =='YYZ') &
(df.dest =='SFO') &
(df.dt ==p0)].b.values[0]

# given B, what percentile does it fall in for the entire column A for YYZ, SFO
stats.percentileofscore(df['a'],x)
61.53846153846154

Run Code Online (Sandbox Code Playgroud)

在上面的例子中,我手动完成了对YYZ和SFO.但是,我的df中有成千上万对.

我如何vectorize使用pandas features而不是循环遍历每一对？

必须有一种方法来使用groupby和使用apply功能吗？

我想要的df应该是这样的:

    src dest  percentile
0   YYZ SFO   61.54
1   DFW PDX   23.07
2   XXX YYY   blahblah1
3   AAA BBB   blahblah2
...

Run Code Online (Sandbox Code Playgroud)

更新:

我实现了以下内容:

def b_percentile_a(df,x,y,b):
    z = df[(df['src'] == x ) & (df['dest'] == y)].a
    r = stats.percentileofscore(z,b)
    return r

b_vector_df = df[df.dt == p0]

b_vector_df['p0_a_percentile_b'] = \
    b_vector_df.apply(lambda x: b_percentile_a(df,x.src,x.dest,x.b), axis=1)

Run Code Online (Sandbox Code Playgroud)

成对需要5.16几秒钟100.我有55,000双.所以这需要~50几分钟.我需要运行这个36时间才能several days运行时间.

一定有更快的方法吗？

Answer 1

sto*_*vfl 6

获得了令人难以置信的节省时间!

输出:
a_list的大小:49998随机化唯一值
percentile_1(您的给定df - scipy)
计算百分位数104次 - 在0:00中的104条记录:07.777022

percentile_9(使用给定df的
PercentileOfScore (rank_searchsorted_list)类别计算百分位数104次 - 在0:00:00.000609中的104条记录
_ dt src dest a b pct scipy _ 0: 2016-01-01 YYZ SFO 54812 279.28 74.81299251970079 74.8129925197 1: 2016-01-01 DFW PDX 111.35 -65.5 24.66698667946718 24.6669866795 2: 2016-02-01 YYZ SFO 64.84 342.35 76.4810592423697 76.4810592424 3: 2016-02-01 DFW PDX 63.81 61.64 63.84655386215449 63.8465538622 ... 24: 2017-01-01 YYZ SFO 97.04 338.28 76.3570542821712 76.3570542822 25: 2017-01-01 DFW PDX 133.94 -129.69 21.4668586743469 21.4668586743

看看scipy.percentileofscore我的实现,发现整个list( a ) 是 - 复制,插入,排序,搜索 - 每次调用percentileofscore.

我实现了自己的 class PercentileOfScore

import numpy as np
class PercentileOfScore(object):

    def __init__(self, aList):
        self.a = np.array( aList )
        self.a.sort()
        self.n = float(len(self.a))
        self.pct = self.__rank_searchsorted_list
    # end def __init__

    def __rank_searchsorted_list(self, score_list):
        adx = np.searchsorted(self.a, score_list, side='right')
        pct = []
        for idx in adx:
            # Python 2.x needs explicit type casting float(int)
            pct.append( (float(idx) / self.n) * 100.0 )

        return pct
    # end def _rank_searchsorted_list
# end class PercentileOfScore

Run Code Online (Sandbox Code Playgroud)

我认为这不def percentile_7符合您的需求.dt不会考虑.

PctOS = None
def percentile_7(df_flat):
    global PctOS
    result = {}
    for k in df_flat.pair_dict.keys():
        # df_flat.pair_dict = { 'src.dst': [b,b,...bn] }
        result[k] = PctOS.pct( df_flat.pair_dict[k] )

    return result
# end def percentile_7

Run Code Online (Sandbox Code Playgroud)

在您的手动示例中,您可以使用整体df.a.在这个样本中dt_flat.a_list,但我不确定这是否是你想要的？

from PercentileData import DF_flat
def main():
    # DF_flat.data = {'dt.src.dest':[a,b]}
    df_flat = DF_flat()

    # Instantiate Global PctOS
    global PctOS
    # df_flat.a_list = [a,a,...an]
    PctOS = PercentileOfScore(df_flat.a_list)

    result = percentile_7(df_flat)
    # result = dict{'src.dst':[pct,pct...pctn]}

Run Code Online (Sandbox Code Playgroud)

用Python测试:3.4.2和2.7.9 - numpy:1.8.2

Answer 2

Ash*_*hum 4

假设您有一个对列表，例如pairs = [[a,b], [c,d], ...]df 已定义，

    r = stats.percentileofscore(z,b)
    return r

for pair in pairs:
    # get the corresponding rows for each pair
    bvalues = df.loc[(df['src']==pair[0])&(df['dest']==pair[1])][['a', 'b']]
    # apply the percentileofscore map
    b_vector_df['p0_a_percentile_b'] = bvalues.b.apply(lambda x: stats.percentileofscore(bvalues.a, x))

Run Code Online (Sandbox Code Playgroud)

我不完全确定目标是什么。我的理解是，您读取每个src、dest对的b值，并查找相应的a值，然后计算该a值的百分位数。让我知道这是否有帮助:)

编辑：假设您仅使用五列date, src, dest, a, and b，您可以考虑使用仅包含这 5 列的数据框的副本。它减少了每个提取步骤所需的工作量。我觉得只使用您需要的数据量会更有效率。根据 pandas 中多列中的值从数据框中选择行是一个可能与您相关的讨论。

归档时间：	8 年，10 月前
查看次数：	608 次
最近记录：	6 年，1 月前