使用Lucene计算TFIDF分数

Question

这是我计算文档集合中文档的TF-IDF值的程序.这工作正常,但在计算"IDF"值(查找包含特定术语的文档的数量)时需要花费大量时间.

是否有更有效的方法来查找包含特定术语的文档？

freq = termsFreq.getTermFrequencies();

terms = termsFreq.getTerms();

int noOfTerms = terms.length;
score = new float[noOfTerms];
DefaultSimilarity simi = new DefaultSimilarity();

        for (i = 0; i < noOfTerms; i++) {

            int noofDocsContainTerm = noOfDocsContainTerm(terms[i]);
            float tf = simi.tf(freq[i]);
            float idf = simi.idf(noofDocsContainTerm, noOfDocs);  
            score[i] = tf * idf ;

        }

////

public int noOfDocsContainTerm(String querystr) throws CorruptIndexException, IOException, ParseException{

QueryParser qp=new QueryParser(Version.LUCENE_35, "docuemnt", new StandardAnalyzer(Version.LUCENE_35));  

Query q=qp.parse(querystr);

int hitsPerPage = docNames.length; //minumum number or search results
IndexSearcher searcher = new IndexSearcher(ramMemDir, true);
TopScoreDocCollector collector = TopScoreDocCollector.create(hitsPerPage, true);

searcher.search(q, collector);

ScoreDoc[] hits = collector.topDocs().scoreDocs;

    return hits.length;
}

Answer 1

如果您有一个术语并希望其文档频率,即包含该术语的文档数:调用IndexReader.termEnum(Term)方法.它为您提供了TermEnum对象.然后,调用TermEnum.docFreq()方法.它为您提供索引中术语的文档频率.