计算用于卡方检验的先前机会数字

Question

所以,我使用脚本来计算一个人在行中指定的日期之前的日期出现的次数,并且在第6列中出现1,并且还计算一个人的次数(列7)在行中指定的日期之前的日期列表中显示(注意它们按时间顺序排序.)(使用基于零的列引用)

示例数据集

02/01/2005,Data,Class xpv,4,11yo+,4,1,George Smith
02/01/2005,Data,Class xpv,4,11yo+,4,2,Ted James
02/01/2005,Data,Class xpv,4,11yo+,4,3,Emma Lilly
02/01/2005,Data,Class xpv,4,11yo+,4,5,George Smith
02/01/2005,Data,Class xpv,4,11yo+,6,4,Tom Phillips
03/01/2005,Data,Class tn2,4,10yo+,6,2,Tom Phillips
03/01/2005,Data,Class tn2,4,10yo+,6,5,George Smith
03/01/2005,Data,Class tn2,4,10yo+,6,3,Tom Phillips
03/01/2005,Data,Class tn2,4,10yo+,6,1,Emma Lilly
03/01/2005,Data,Class tn2,4,10yo+,6,6,George Smith
04/01/2005,Data,Class tn2,4,10yo+,6,6,Ted James
04/01/2005,Data,Class tn2,4,10yo+,6,3,Tom Phillips
04/01/2005,Data,Class tn2,4,10yo+,6,2,George Smith
04/01/2005,Data,Class tn2,4,10yo+,6,4,George Smith
04/01/2005,Data,Class tn2,4,10yo+,6,1,George Smith
04/01/2005,Data,Class tn2,4,10yo+,6,5,Tom Phillips
05/01/2005,Data,Class 22zn,2,10yo+,5,3,Emma Lilly
05/01/2005,Data,Class 22zn,2,10yo+,5,1,Ted James
05/01/2005,Data,Class 22zn,2,10yo+,5,2,George Smith
05/01/2005,Data,Class 22zn,2,10yo+,5,4,Emma Lilly
05/01/2005,Data,Class 22zn,2,10yo+,5,5,Tom Phillips

我正在使用的代码

import csv
import datetime
import copy
from collections import defaultdict

with open(r"C:\Temp\test.csv") as i, open(r"C:\Temp\resuls.csv", "wb") as o:
    rdr = csv.reader(i)
    wrt = csv.writer(o)

    data, currdate = defaultdict(lambda:[0, 0, 0, 0]), None
    for line in rdr:
        date, name = datetime.datetime.strptime(line[0], '%d/%m/%Y'), line[7]

        if date != currdate or not currdate:
            for v in data.itervalues(): v[:2] = v[2:]
            currdate = date

        wrt.writerow(line + data[name][:2])

        data[name][3] += 1
        if line[6] == "1": data[name][2] += 1

返回:

02/01/2005,Data,Class xpv,4,11yo+,4,1,George Smith,0,0
02/01/2005,Data,Class xpv,4,11yo+,4,2,Ted James,0,0
02/01/2005,Data,Class xpv,4,11yo+,4,3,Emma Lilly,0,0
02/01/2005,Data,Class xpv,4,11yo+,4,5,George Smith,0,0
02/01/2005,Data,Class xpv,4,11yo+,6,4,Tom Phillips,0,0
03/01/2005,Data,Class tn2,4,10yo+,6,2,Tom Phillips,0,1
03/01/2005,Data,Class tn2,4,10yo+,6,5,George Smith,1,2
03/01/2005,Data,Class tn2,4,10yo+,6,3,Tom Phillips,0,1
03/01/2005,Data,Class tn2,4,10yo+,6,1,Emma Lilly,0,1
03/01/2005,Data,Class tn2,4,10yo+,6,6,George Smith,1,2
04/01/2005,Data,Class tn2,4,10yo+,6,6,Ted James,0,1
04/01/2005,Data,Class tn2,4,10yo+,6,3,Tom Phillips,0,3
04/01/2005,Data,Class tn2,4,10yo+,6,2,George Smith,1,4
04/01/2005,Data,Class tn2,4,10yo+,6,4,George Smith,1,4
04/01/2005,Data,Class tn2,4,10yo+,6,1,George Smith,1,4
04/01/2005,Data,Class tn2,4,10yo+,6,5,Tom Phillips,0,3
05/01/2005,Data,Class 22zn,2,10yo+,5,3,Emma Lilly,1,2
05/01/2005,Data,Class 22zn,2,10yo+,5,1,Ted James,0,2
05/01/2005,Data,Class 22zn,2,10yo+,5,2,George Smith,2,7
05/01/2005,Data,Class 22zn,2,10yo+,5,4,Emma Lilly,1,2
05/01/2005,Data,Class 22zn,2,10yo+,5,5,Tom Phillips,0,5

最终,我会想要对我生成的百分比数据执行chi平方.但是现在我希望能够实现的是能够计算并总结唯一数据类(第2列)中任何一个人的分数几率,并将其作为新列附加到csv.我不确定如果我使用的代码可以编辑,以实现这一个代码.任何有关如何做到最好的建设性建议或意见将不胜感激.

我想要的输出如下:

02/01/2005,Data,Class xpv,4,11yo+,5,1,George Smith,0,0,0
02/01/2005,Data,Class xpv,4,11yo+,5,2,Ted James,0,0,0
02/01/2005,Data,Class xpv,4,11yo+,5,3,Emma Lilly,0,0,0
02/01/2005,Data,Class xpv,4,11yo+,5,5,George Smith,0,0,0
02/01/2005,Data,Class xpv,4,11yo+,5,4,Tom Phillips,0,0,0
03/01/2005,Data,Class tn2,4,10yo+,5,2,Tom Phillips,0,1,0.2, He gets 0.2 because there was a 1 in 5 chance for previous occurrences on dates prior to today. 1/5
03/01/2005,Data,Class tn2,4,10yo+,5,5,George Smith,1,2,0.4, He gets 0.4 because there was a 2 in 5 chance for previous occurrences on dates prior to today. 2/5
03/01/2005,Data,Class tn2,4,10yo+,5,3,Tom Phillips,0,1,0.2
03/01/2005,Data,Class tn2,4,10yo+,5,1,Emma Lilly,0,1,0.2
03/01/2005,Data,Class tn2,4,10yo+,5,6,George Smith,1,2,0.4
04/01/2005,Data,Class tn2,4,10yo+,6,6,Ted James,0,1,0.2
04/01/2005,Data,Class tn2,4,10yo+,6,3,Tom Phillips,0,3,0.6
04/01/2005,Data,Class tn2,4,10yo+,6,2,George Smith,1,4,0.8
04/01/2005,Data,Class tn2,4,10yo+,6,4,George Smith,1,4,0.8
04/01/2005,Data,Class tn2,4,10yo+,6,1,George Smith,1,4,0.8
04/01/2005,Data,Class tn2,4,10yo+,6,5,Tom Phillips,0,3,0.4
05/01/2005,Data,Class 22zn,2,10yo+,5,3,Emma Lilly,1,2,0.4
05/01/2005,Data,Class 22zn,2,10yo+,5,1,Ted James,0,2,0.366666667
05/01/2005,Data,Class 22zn,2,10yo+,5,2,George Smith,2,7,1.3
05/01/2005,Data,Class 22zn,2,10yo+,5,4,Emma Lilly,1,2,0.4
05/01/2005,Data,Class 22zn,2,10yo+,5,5,Tom Phillips,0,5,0.733333333

Answer 1

这不应该是你问题的完整答案(因为你想要做的事情有点含糊不清),只是为了向你展示大熊猫如何在这种计算中自然地适应; 您还可以通过名称而不是索引来调用列.

假设您有一个test.csv这样的文件:

date,x0,cls,x1,x2,x3,tag,name
02/01/2005,Data,Class xpv,4,11yo+,4,1,George Smith
02/01/2005,Data,Class xpv,4,11yo+,4,2,Ted James
02/01/2005,Data,Class xpv,4,11yo+,4,3,Emma Lilly
02/01/2005,Data,Class xpv,4,11yo+,4,5,George Smith
...

我已经为每列分配了名称.您可以将此文件读入pandas数据帧

import pandas as pd
df = pd.DataFrame.from_csv( 'test.csv', index_col=None )

df 将如下所示:

          date    x0         cls  x1     x2  x3  tag          name
0   02/01/2005  Data   Class xpv   4  11yo+   4    1  George Smith
1   02/01/2005  Data   Class xpv   4  11yo+   4    2     Ted James
2   02/01/2005  Data   Class xpv   4  11yo+   4    3    Emma Lilly
3   02/01/2005  Data   Class xpv   4  11yo+   4    5  George Smith
...

我删除了你没有使用的列(这只是为了演示的目的,你不必删除这些列)

df.drop( labels=['x0', 'x1', 'x2', 'x3'], axis=1, inplace=True )

现在df看起来如下:

          date         cls  tag          name
0   02/01/2005   Class xpv    1  George Smith
1   02/01/2005   Class xpv    2     Ted James
2   02/01/2005   Class xpv    3    Emma Lilly
3   02/01/2005   Class xpv    5  George Smith
...

假设您要查找每个人在每天之前的日期中出现的累计次数:

pv = df.pivot_table( cols='name',
                     rows='date',
                     values='tag',
                     aggfunc=len ).shift( 1 ).fillna( 0 ).cumsum( )

api文档(参见此处)包含每种方法正在执行的操作的详细说明.现在你有透视表pv看起来像这样

date        Emma Lilly  George Smith  Ted James  Tom Phillips
02/01/2005           0             0          0             0
03/01/2005           1             2          1             1
04/01/2005           2             4          1             3
05/01/2005           2             7          2             5

或者可以使用groupby:

df.groupby(['date', 'name'])['name'].aggregate(len).unstack( ).shift( 1 ).fillna( 0 ).cumsum( )

要进行相同的计算,但仅限于此tag == 1,您可以这样做

idx = df.tag == 1
pv1 = df[ idx ].pivot_table( cols='name',
                             rows='date',
                             values='tag',
                             aggfunc=len ).shift( 1 ).fillna( 0 ).cumsum( )

或使用groupby语法:

df[ df.tag == 1 ].groupby(['date', 'name'])['name'].aggregate(len).unstack( ).shift( 1 ).fillna( 0 ).cumsum( )

这将是:

date        Emma Lilly  George Smith  Ted James
02/01/2005           0             0          0
03/01/2005           0             1          0
04/01/2005           1             1          0
05/01/2005           1             2          0

为了填写两个新列,如果缺少值,我们编写一个辅助函数以回退到0:

def lookup( pivot_table, col, idx, fall_back=0 ):
    try:
        return pivot_table[ col ][ idx ]
    except KeyError:
        return fall_back

df[ 'cnt1' ] = [ lookup( pv1, row[ 'name' ], row[ 'date' ] ) for idx, row in df.iterrows( ) ]
df[ 'cnt' ] = [ lookup( pv, row[ 'name' ], row[ 'date' ] ) for idx, row in df.iterrows( ) ]

我们得到:

          date         cls  tag          name  cnt1  cnt
0   02/01/2005   Class xpv    1  George Smith     0    0
1   02/01/2005   Class xpv    2     Ted James     0    0
2   02/01/2005   Class xpv    3    Emma Lilly     0    0
3   02/01/2005   Class xpv    5  George Smith     0    0
4   02/01/2005   Class tn2    4  Tom Phillips     0    0
5   03/01/2005   Class tn2    2  Tom Phillips     0    1
6   03/01/2005   Class tn2    5  George Smith     1    2
7   03/01/2005   Class tn2    3  Tom Phillips     0    1
8   03/01/2005   Class tn2    1    Emma Lilly     0    1
9   03/01/2005   Class tn2    6  George Smith     1    2
10  04/01/2005   Class tn2    6     Ted James     0    1
11  04/01/2005   Class tn2    3  Tom Phillips     0    3
12  04/01/2005   Class tn2    2  George Smith     1    4
13  04/01/2005   Class tn2    4  George Smith     1    4
14  04/01/2005   Class tn2    1  George Smith     1    4
15  04/01/2005   Class tn2    5  Tom Phillips     0    3
16  05/01/2005  Class 22zn    3    Emma Lilly     1    2
17  05/01/2005  Class 22zn    1     Ted James     0    2
18  05/01/2005  Class 22zn    2  George Smith     2    7
19  05/01/2005  Class 22zn    4    Emma Lilly     1    2
20  05/01/2005  Class 22zn    5  Tom Phillips     0    5

如果我知道你是如何计算最后一栏的,我可以继续前进.例如为什么"汤姆菲利普斯"在第6排获得0.2？

编辑:好的,让我们继续吧.我们需要找出每个人在每个日期出现的次数; 这是另一个数据透视表:

appr = df.pivot_table( cols='name',
                       rows='date',
                       values='tag',
                       aggfunc=len ).fillna( 0 )

要么

df.groupby( ['date', 'name'] )['name'].aggregate(len).unstack( ).fillna( 0 )

输出:

date        Emma Lilly  George Smith  Ted James  Tom Phillips
02/01/2005           1             2          1             1
03/01/2005           1             2          0             2
04/01/2005           0             3          1             2
05/01/2005           2             1          1             1

以及每个日期出现的人数:

total_appr = appr.sum( axis=1 )

输出:

date
02/01/2005    5
03/01/2005    5
04/01/2005    6
05/01/2005    5

计算累积分数你可以简单地将每一行除以总数,换一个(因为我们查找以前的日期)并计算累计和:

frac = appr.apply( lambda x: x / total_appr ).shift( 1 ).fillna( 0 ).cumsum( )
df[ 'frac' ] = [ frac[ row[ 'name' ] ][ row[ 'date' ] ] for idx, row in df.iterrows( ) ]

现在df看起来如下:

          date         cls  tag          name  cnt1  cnt      frac
0   02/01/2005   Class xpv    1  George Smith     0    0  0.000000
1   02/01/2005   Class xpv    2     Ted James     0    0  0.000000
2   02/01/2005   Class xpv    3    Emma Lilly     0    0  0.000000
3   02/01/2005   Class xpv    5  George Smith     0    0  0.000000
4   02/01/2005   Class tn2    4  Tom Phillips     0    0  0.000000
5   03/01/2005   Class tn2    2  Tom Phillips     0    1  0.200000
6   03/01/2005   Class tn2    5  George Smith     1    2  0.400000
7   03/01/2005   Class tn2    3  Tom Phillips     0    1  0.200000
8   03/01/2005   Class tn2    1    Emma Lilly     0    1  0.200000
9   03/01/2005   Class tn2    6  George Smith     1    2  0.400000
10  04/01/2005   Class tn2    6     Ted James     0    1  0.200000
11  04/01/2005   Class tn2    3  Tom Phillips     0    3  0.600000
12  04/01/2005   Class tn2    2  George Smith     1    4  0.800000
13  04/01/2005   Class tn2    4  George Smith     1    4  0.800000
14  04/01/2005   Class tn2    1  George Smith     1    4  0.800000
15  04/01/2005   Class tn2    5  Tom Phillips     0    3  0.600000
16  05/01/2005  Class 22zn    3    Emma Lilly     1    2  0.400000
17  05/01/2005  Class 22zn    1     Ted James     0    2  0.366667
18  05/01/2005  Class 22zn    2  George Smith     2    7  1.300000
19  05/01/2005  Class 22zn    4    Emma Lilly     1    2  0.400000
20  05/01/2005  Class 22zn    5  Tom Phillips     0    5  0.933333

我的数字与你的数字在最后一列中的两行不同.所以要么你的计算错了,要么你错误地计算了这两个数字.