我有这个代码用于计算与tf-idf的文本相似性.
from sklearn.feature_extraction.text import TfidfVectorizer
documents = [doc1,doc2]
tfidf = TfidfVectorizer().fit_transform(documents)
pairwise_similarity = tfidf * tfidf.T
print pairwise_similarity.A
问题是这个代码作为输入普通字符串,我想通过删除停用词,词干和tokkenize来准备文档.所以输入将是一个列表.如果我documents = [doc1,doc2]用tokkenized文件调用该错误是:
Traceback (most recent call last):
File "C:\Users\tasos\Desktop\my thesis\beta\similarity.py", line 18, in <module>
tfidf = TfidfVectorizer().fit_transform(documents)
File "C:\Python27\lib\site-packages\scikit_learn-0.14.1-py2.7-win32.egg\sklearn\feature_extraction\text.py", line 1219, in fit_transform
X = super(TfidfVectorizer, self).fit_transform(raw_documents)
File "C:\Python27\lib\site-packages\scikit_learn-0.14.1-py2.7-win32.egg\sklearn\feature_extraction\text.py", line 780, in fit_transform
vocabulary, X = self._count_vocab(raw_documents, self.fixed_vocabulary)
File "C:\Python27\lib\site-packages\scikit_learn-0.14.1-py2.7-win32.egg\sklearn\feature_extraction\text.py", line 715, in _count_vocab
for feature in analyze(doc):
File "C:\Python27\lib\site-packages\scikit_learn-0.14.1-py2.7-win32.egg\sklearn\feature_extraction\text.py", line 229, in <lambda>
tokenize(preprocess(self.decode(doc))), stop_words)
File "C:\Python27\lib\site-packages\scikit_learn-0.14.1-py2.7-win32.egg\sklearn\feature_extraction\text.py", line …假设我的数据框包含以下数据:
>>> df = pd.DataFrame({'a':['l1','l2','l1','l2','l1','l2'],
'b':['1','2','2','1','2','2']})
>>> df
a b
0 l1 1
1 l2 2
2 l1 2
3 l2 1
4 l1 2
5 l2 2
l1应该对应,1而l2应该对应2.我想创建一个新列' c',对于每一行,c = 1if a = l1和b = 1(或a = l2和b = 2).如果a = l1和b = 2(或a = l2和b = 1)然后c = 0.
生成的数据框应如下所示:
a …