标签: tf-idf

我正在研究关键字提取问题.考虑一般情况

tfidf = TfidfVectorizer(tokenizer=tokenize, stop_words='english')

t = """Two Travellers, walking in the noonday sun, sought the shade of a widespreading tree to rest. As they lay looking up among the pleasant leaves, they saw that it was a Plane Tree.

"How useless is the Plane!" said one of them. "It bears no fruit whatever, and only serves to litter the ground with leaves."

"Ungrateful creatures!" said a voice from the Plane Tree. "You lie here in my cooling shade, and …

本页:http://scikit-learn.org/stable/modules/feature_extraction.html提及:

由于tf-idf经常用于文本特征,因此还有另一个名为TfidfVectorizer的类,它将CountVectorizer和TfidfTransformer的所有选项组合在一个模型中.

然后我按照代码在我的语料库上使用fit_transform().如何获得fit_transform()计算的每个特征的权重？

我试过了:

In [39]: vectorizer.idf_
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-39-5475eefe04c0> in <module>()
----> 1 vectorizer.idf_

AttributeError: 'TfidfVectorizer' object has no attribute 'idf_'

但是这个属性丢失了.

谢谢

我想从下面的文档中计算tf-idf.我正在使用python和pandas.

import pandas as pd
df = pd.DataFrame({'docId': [1,2,3], 
               'sent': ['This is the first sentence','This is the second sentence', 'This is the third sentence']})

首先,我想我需要为每一行获取word_count.所以我写了一个简单的函数:

def word_count(sent):
    word2cnt = dict()
    for word in sent.split():
        if word in word2cnt: word2cnt[word] += 1
        else: word2cnt[word] = 1
return word2cnt

然后,我将它应用于每一行.

df['word_count'] = df['sent'].apply(word_count)

但现在我迷路了.我知道如果我使用Graphlab,有一种简单的方法来计算tf-idf,但我想坚持使用开源选项.Sklearn和gensim都看起来势不可挡.获得tf-idf的最简单的解决方案是什么？

我在Python上使用sklearn来做一些聚类.我已经训练了200,000个数据,下面的代码效果很好.

corpus = open("token_from_xml.txt")
vectorizer = CountVectorizer(decode_error="replace")
transformer = TfidfTransformer()
tfidf = transformer.fit_transform(vectorizer.fit_transform(corpus))
km = KMeans(30)
kmresult = km.fit(tfidf).predict(tfidf)

但是当我有新的测试内容时,我想将它集中到我训练过的现有集群中.所以我想知道如何保存IDF结果,这样我就可以为新的测试内容做TFIDF,并确保新测试内容的结果具有相同的数组长度.

提前致谢.

UPDATE

我可能需要将"transformer"或"tfidf"变量保存到文件(txt或其他),如果其中一个包含经过训练的IDF结果.

UPDATE

例如.我有训练数据:

["a", "b", "c"]
["a", "b", "d"]

并且做TFIDF,结果将包含4个特征(a,b,c,d)

当我测试时:

["a", "c", "d"]

查看它所属的集群(已由k-means制作).TFIDF只会给出3个特征(a,c,d)的结果,因此k-means中的聚类将会下降.(如果我测试["a", "b", "e"],可能还有其他问题.)

那么如何存储用于测试数据的功能列表(甚至更多,将其存储在文件中)？

UPDATE

解决了,见下面的答案.

首先,让我们每个文档每个术语提取TF-IDF分数:

from gensim import corpora, models, similarities
documents = ["Human machine interface for lab abc computer applications",
              "A survey of user opinion of computer system response time",
              "The EPS user interface management system",
              "System and human system engineering testing of EPS",
              "Relation of user perceived response time to error measurement",
              "The generation of random binary unordered trees",
              "The intersection graph of paths in trees",
              "Graph minors IV Widths of trees and well quasi ordering",
              "Graph minors A survey"]
stoplist = …

python中有哪些标准的tf-idf实现/ api？我在nltk遇到过那个.我想知道提供此功能的其他库.

我HashingTF在Spark中创建了Term Frequency .我已经tf.transform为每个单词使用了术语频率.

但结果以这种格式显示.

[<hashIndexofHashBucketofWord1>,<hashIndexofHashBucketofWord2> ...]
,[termFrequencyofWord1, termFrequencyOfWord2 ....]

例如:

(1048576,[105,3116],[1.0,2.0])

我能够使用哈希桶获取索引tf.indexOf("word").

但是,我怎样才能使用索引得到这个词？

我正在努力研究如何提高solr搜索结果的得分.我的应用程序需要从solr结果中获取分数,并根据查询的结果有多好来显示一些"星星".5星=差不多/精确到0星,意味着不能很好地匹配搜索,例如只有一个元素命中.然而,我得到的分数从1.4到0.8660254都返回结果,我会给5星.我需要做的是以某种方式将这些结果转换为百分比,以便我可以用正确的星数来标记这些结果.

我运行的查询给出了1.4分:

euallowed:true AND(等级:"2:1")

给我0.8660254分数的查询是:

euallowed:true AND(等级:"2:1"或等级:"1st")

我已经更新了Similarity,以便tf和idf返回1.0,因为我只对文档中有一个术语而不是文档中该术语的编号感兴趣.这就是我的相似性代码:

import org.apache.lucene.search.Similarity;

public class StudentSearchSimilarity extends Similarity {

    @Override
    public float lengthNorm(String fieldName, int numTerms) {
        return (float) (1.0 / Math.sqrt(numTerms));
    }

    @Override
    public float queryNorm(float sumOfSquaredWeights) {

        return (float) (1.0 / Math.sqrt(sumOfSquaredWeights));

    }

    @Override
    public float sloppyFreq(int distance) {
        return 1.0f / (distance + 1);
    }

    @Override
    public float tf(float freq) {
        return (float) 1.0;
    }

    @Override
    public float idf(int docFreq, int numDocs) {

        //return (float) (Math.log(numDocs / (double) (docFreq + …

好的,所以我一直关注TF*IDF上的这两篇文章,但我很困惑:http://css.dzone.com/articles/machine-learning-text-feature

基本上,我想创建一个包含多个文档搜索的搜索查询.我想使用scikit-learn工具包以及Python的NLTK库

问题是我没有看到两个TF*IDF向量来自哪里.我需要一个搜索查询和多个文档来搜索.我想我会针对每个查询计算每个文档的TF*IDF分数,并找出它们之间的余弦相似度,然后通过按降序对分数进行排序来对它们进行排名.但是,代码似乎没有提出正确的向量.

每当我将查询减少到只有一个搜索时,它返回一个巨大的0列表,这真的很奇怪.

这是代码:

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.feature_extraction.text import TfidfTransformer
from nltk.corpus import stopwords

train_set = ("The sky is blue.", "The sun is bright.") #Documents
test_set = ("The sun in the sky is bright.") #Query
stopWords = stopwords.words('english')

vectorizer = CountVectorizer(stop_words = stopWords)
transformer = TfidfTransformer()

trainVectorizerArray = vectorizer.fit_transform(train_set).toarray()
testVectorizerArray = vectorizer.transform(test_set).toarray()
print 'Fit Vectorizer to train set', trainVectorizerArray
print 'Transform Vectorizer to test set', testVectorizerArray

transformer.fit(trainVectorizerArray)
print transformer.transform(trainVectorizerArray).toarray()

transformer.fit(testVectorizerArray)

tfidf = transformer.transform(testVectorizerArray)
print …

我运行以下代码将文本矩阵转换为TF-IDF矩阵.

text = ['This is a string','This is another string','TFIDF computation calculation','TfIDF is the product of TF and IDF']

from sklearn.feature_extraction.text import TfidfVectorizer
vectorizer = TfidfVectorizer(max_df=1.0, min_df=1, stop_words='english',norm = None)

X = vectorizer.fit_transform(text)
X_vovab = vectorizer.get_feature_names()
X_mat = X.todense()
X_idf = vectorizer.idf_

我得到以下输出

X_vovab =

[u'calculation',
 u'computation',
 u'idf',
 u'product',
 u'string',
 u'tf',
 u'tfidf']

和X_mat =

  ([[ 0.        ,  0.        ,  0.        ,  0.        ,  1.51082562,
      0.        ,  0.        ],
    [ 0.        ,  0.        ,  0.        ,  0.        ,  1.51082562,
      0.        , …