使用pandas数据帧获取tfidf的最简单方法是什么？

Question

我想从下面的文档中计算tf-idf.我正在使用python和pandas.

import pandas as pd
df = pd.DataFrame({'docId': [1,2,3], 
               'sent': ['This is the first sentence','This is the second sentence', 'This is the third sentence']})

首先,我想我需要为每一行获取word_count.所以我写了一个简单的函数:

def word_count(sent):
    word2cnt = dict()
    for word in sent.split():
        if word in word2cnt: word2cnt[word] += 1
        else: word2cnt[word] = 1
return word2cnt

然后,我将它应用于每一行.

df['word_count'] = df['sent'].apply(word_count)

但现在我迷路了.我知道如果我使用Graphlab,有一种简单的方法来计算tf-idf,但我想坚持使用开源选项.Sklearn和gensim都看起来势不可挡.获得tf-idf的最简单的解决方案是什么？

Answer 1

Scikit-learn实现非常简单:

from sklearn.feature_extraction.text import TfidfVectorizer
v = TfidfVectorizer()
x = v.fit_transform(df['sent'])

您可以指定大量参数.请参阅此处的文档

fit_transform的输出将是一个稀疏矩阵,如果您想要可视化它,您可以这样做 x.toarray()

In [44]: x.toarray()
Out[44]: 
array([[ 0.64612892,  0.38161415,  0.        ,  0.38161415,  0.38161415,
         0.        ,  0.38161415],
       [ 0.        ,  0.38161415,  0.64612892,  0.38161415,  0.38161415,
         0.        ,  0.38161415],
       [ 0.        ,  0.38161415,  0.        ,  0.38161415,  0.38161415,
         0.64612892,  0.38161415]])

Answer 2

一个简单的解决方案是使用texthero：

import texthero as hero
df['tfidf'] = hero.tfidf(df['sent'])

In [5]: df.head()
Out[5]:
   docId                         sent                                              tfidf
0      1   This is the first sentence  [0.3816141458138271, 0.6461289150464732, 0.381...
1      2  This is the second sentence  [0.3816141458138271, 0.0, 0.3816141458138271, ...
2      3   This is the third sentence  [0.3816141458138271, 0.0, 0.3816141458138271, ...