从Python:tf-idf-cosine:为了找到文档相似性,可以使用tf-idf余弦计算文档相似度.没有导入外部库,是否有任何方法可以计算2个字符串之间的余弦相似度？
s1 = "This is a foo bar sentence ."
s2 = "This sentence is similar to a foo bar sentence ."
s3 = "What is this string ? Totally not related to the other two lines ."
cosine_sim(s1, s2) # Should give high cosine similarity
cosine_sim(s1, s3) # Shouldn't give high cosine similarity value
cosine_sim(s2, s3) # Shouldn't give high cosine similarity value
python中有哪些标准的tf-idf实现/ api？我在nltk遇到过那个.我想知道提供此功能的其他库.
好的,所以我一直关注TF*IDF上的这两篇文章,但我很困惑:http://css.dzone.com/articles/machine-learning-text-feature
基本上,我想创建一个包含多个文档搜索的搜索查询.我想使用scikit-learn工具包以及Python的NLTK库
问题是我没有看到两个TF*IDF向量来自哪里.我需要一个搜索查询和多个文档来搜索.我想我会针对每个查询计算每个文档的TF*IDF分数,并找出它们之间的余弦相似度,然后通过按降序对分数进行排序来对它们进行排名.但是,代码似乎没有提出正确的向量.
每当我将查询减少到只有一个搜索时,它返回一个巨大的0列表,这真的很奇怪.
这是代码:
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.feature_extraction.text import TfidfTransformer
from nltk.corpus import stopwords
train_set = ("The sky is blue.", "The sun is bright.") #Documents
test_set = ("The sun in the sky is bright.") #Query
stopWords = stopwords.words('english')
vectorizer = CountVectorizer(stop_words = stopWords)
transformer = TfidfTransformer()
trainVectorizerArray = vectorizer.fit_transform(train_set).toarray()
testVectorizerArray = vectorizer.transform(test_set).toarray()
print 'Fit Vectorizer to train set', trainVectorizerArray
print 'Transform Vectorizer to test set', testVectorizerArray
transformer.fit(trainVectorizerArray)
print transformer.transform(trainVectorizerArray).toarray()
transformer.fit(testVectorizerArray)
tfidf = transformer.transform(testVectorizerArray)
print …受到这个答案的启发,我试图在经过训练的训练有素的tf-idf矢量化器和新文档之间找到余弦相似性,并返回类似的文档.
下面的代码找到第一个向量的余弦相似度,而不是新的查询
>>> from sklearn.metrics.pairwise import linear_kernel
>>> cosine_similarities = linear_kernel(tfidf[0:1], tfidf).flatten()
>>> cosine_similarities
array([ 1. , 0.04405952, 0.11016969, ..., 0.04433602,
0.04457106, 0.03293218])
由于我的火车数据很大,循环遍历整个训练过的矢量器听起来像个坏主意.如何推断新文档的向量,并找到相关文档,与下面的代码相同？
>>> related_docs_indices = cosine_similarities.argsort()[:-5:-1]
>>> related_docs_indices
array([ 0, 958, 10576, 3277])
>>> cosine_similarities[related_docs_indices]
array([ 1. , 0.54967926, 0.32902194, 0.2825788 ])
python machine-learning tf-idf document-classification scikit-learn
我知道公式tfidf vectorizer是
Count of word/Total count * log(Number of documents / no.of documents where word is present)
我在 scikit learn 中看到了 tfidf 转换器，我只是想区分它们。我找不到任何有用的东西。