Python中的评估者协议(Cohen的Kappa)

Question

我有3个评级者对60个案例的评分.这些是按文档组织的列表 - 第一个元素是指第一个文档的评级,第二个文档的第二个,依此类推:

rater1 = [-8,-7,8,6,2,-5,...]
rater2 = [-3,-5,3,3,2,-2,...]
rater3 = [-4,-2,1,0,0,-2,...]

在某个地方是否有Cohen的Kappa的python实现？我在numpy或scipy中找不到任何东西,而且stackoverflow上没有任何东西,但也许我错过了它？这是一个非常常见的统计数据,所以我很惊讶我找不到像Python这样的语言.

Answer 1

您还可以使用nltk.metrics.agreement.以下是相同的代码段

from nltk import agreement
rater1 = [1,1,1]
rater2 = [1,1,0]
rater3 = [0,1,1]

taskdata=[[0,str(i),str(rater1[i])] for i in range(0,len(rater1))]+[[1,str(i),str(rater2[i])] for i in range(0,len(rater2))]+[[2,str(i),str(rater3[i])] for i in range(0,len(rater3))]
ratingtask = agreement.AnnotationTask(data=taskdata)
print("kappa " +str(ratingtask.kappa()))
print("fleiss " + str(ratingtask.multi_kappa()))
print("alpha " +str(ratingtask.alpha()))
print("scotts " + str(ratingtask.pi()))

有关其他示例,另请参阅http://courses.washington.edu/cmling/lab7.html

Answer 2

Cohen的kappa是在scikit-learn 0.17中引入的:

sklearn.metrics.cohen_kappa_score(y1,y2,labels = None,weights = None)

示例:

from sklearn.metrics import cohen_kappa_score
labeler1 = [2, 0, 2, 2, 0, 1]
labeler2 = [0, 0, 2, 2, 0, 2]
cohen_kappa_score(labeler1, labeler2)

提醒一下,从{1}开始:

参考文献:

• {1} Viera,Anthony J.和Joanne M. Garrett."了解观察者之间的协议:kappa统计数据." Fam Med 37,没有.5(2005):360-363.https://www.ncbi.nlm.nih.gov/pubmed/15883903:

Answer 3

为了扩展Franck Dernoncourt 的答案并解决 skjerns 评论，这里是为两个以上评估者创建矩阵的代码：

import itertools

from sklearn.metrics import cohen_kappa_score
import numpy as np

# Note that I updated the numbers so all Cohen kappa scores are different.
rater1 = [-8, -7, 8, 6, 2, -5]
rater2 = [-3, -5, 3, 3, 2, -2]
rater3 = [-4, -2, 1, 3, 0, -2]

raters = [rater1, rater2, rater3]

data = np.zeros((len(raters), len(raters)))
# Calculate cohen_kappa_score for every combination of raters
# Combinations are only calculated j -> k, but not k -> j, which are equal
# So not all places in the matrix are filled.
for j, k in list(itertools.combinations(range(len(raters)), r=2)):
    data[j, k] = cohen_kappa_score(raters[j], raters[k])

# [[0.        , 0.11764706, 0.        ],
#  [0.        , 0.        , 0.25      ],
#  [0.        , 0.        , 0.        ]]

这是一个情节data：

import seaborn as sns
import matplotlib.pyplot as plt

sns.heatmap(
    data, 
    mask=np.tri(len(raters)),
    annot=True, linewidths=5,
    vmin=0, vmax=1,
    xticklabels=[f"Rater {k + 1}" for k in range(len(raters))],
    yticklabels=[f"Rater {k + 1}" for k in range(len(raters))],
)
plt.show()

Answer 4

我还没有发现它包含在任何主要的库中，但是如果你用谷歌搜索，你可以在各种“cookbook”类型的网站等上找到实现。以下是Cohen 的 kappa、Fleiss 的 kappa和Krippendorff 的 alpha实现的页面