我是scikit-learn的新手,我TfidfVectorizer用来在一组文档中找到术语的tfidf值.我用下面的代码来获得相同的代码.
vectorizer = TfidfVectorizer(stop_words=u'english',ngram_range=(1,5),lowercase=True)
X = vectorizer.fit_transform(lectures)
Run Code Online (Sandbox Code Playgroud)
现在如果我打印X,我能够看到矩阵中的所有条目,但我如何根据tfidf分数找到前n个条目.除此之外,是否有任何方法可以帮助我找到基于每个ngram的tfidf得分的前n个条目,即unigram,bigram,trigram等中的顶级条目?
我正在研究关键字提取问题.考虑一般情况
tfidf = TfidfVectorizer(tokenizer=tokenize, stop_words='english')
t = """Two Travellers, walking in the noonday sun, sought the shade of a widespreading tree to rest. As they lay looking up among the pleasant leaves, they saw that it was a Plane Tree.
"How useless is the Plane!" said one of them. "It bears no fruit whatever, and only serves to litter the ground with leaves."
"Ungrateful creatures!" said a voice from the Plane Tree. "You lie here in my cooling shade, and …Run Code Online (Sandbox Code Playgroud) 有这个代码:
feature_array = np.array(tfidf.get_feature_names())
tfidf_sorting = np.argsort(response.toarray()).flatten()[::-1]
n = 3
top_n = feature_array[tfidf_sorting][:n]
Run Code Online (Sandbox Code Playgroud)
来自这个答案。
我的问题是在我的稀疏矩阵太大而无法立即转换为密集矩阵(with response.toarray())的情况下,我如何有效地做到这一点?
显然,一般的答案是将稀疏矩阵分割成块,在 for 循环中对每个块进行转换,然后将所有块的结果组合起来。
但我想具体查看执行此操作的代码。