用Lucene扼杀英语单词

Question

我正在Java应用程序中处理一些英文文本,我需要阻止它们.例如,从文字"amenities/amenity"我需要得到"amenit".

该功能如下:

String stemTerm(String term){
   ...
}

我找到了Lucene分析仪,但它看起来太复杂了我需要的东西. http://lucene.apache.org/java/2_2_0/api/org/apache/lucene/analysis/PorterStemFilter.html

有没有办法用它来阻止单词而无需构建分析器？我不了解所有分析仪业务......

编辑:我实际上需要一个词干+词形还原.Lucene可以这样做吗？

Answer 1

SnowballAnalyzer已被弃用,您可以使用Lucene Porter Stemmer代替:

 PorterStemmer stem = new PorterStemmer();
 stem.setCurrent(word);
 stem.stem();
 String result = stem.getCurrent();

希望这有帮助!

Answer 2

import org.apache.lucene.analysis.PorterStemmer;
...
String stemTerm (String term) {
    PorterStemmer stemmer = new PorterStemmer();
    return stemmer.stem(term);
}

有关详细信息,请参见此处 如果你想做什么,那么你应该使用它而不是Lucene.

编辑:term在传递给你之前你应该小写stem().

Answer 3

你为什么不使用"EnglishAnalyzer"？它使用起来很简单,我认为它可以解决您的问题:

EnglishAnalyzer en_an = new EnglishAnalyzer(Version.LUCENE_34);
QueryParser parser = new QueryParser(Version.LUCENE_34, "your_field", en_an);
String str = "amenities";
System.out.println("result: " + parser.parse(str)); //amenit

希望它能帮到你!

Answer 4

前面的示例将词干应用于搜索查询,因此如果您想要阻止全文,则可以尝试以下操作:

import java.io.*;
import org.apache.lucene.analysis.*;
import org.apache.lucene.analysis.tokenattributes.*;
import org.apache.lucene.analysis.snowball.*;
import org.apache.lucene.util.*;
...
public class Stemmer{
    public static String Stem(String text, String language){
        StringBuffer result = new StringBuffer();
        if (text!=null && text.trim().length()>0){
            StringReader tReader = new StringReader(text);
            Analyzer analyzer = new SnowballAnalyzer(Version.LUCENE_35,language);
            TokenStream tStream = analyzer.tokenStream("contents", tReader);
            TermAttribute term = tStream.addAttribute(TermAttribute.class);

            try {
                while (tStream.incrementToken()){
                    result.append(term.term());
                    result.append(" ");
                }
            } catch (IOException ioe){
                System.out.println("Error: "+ioe.getMessage());
            }
        }

        // If, for some reason, the stemming did not happen, return the original text
        if (result.length()==0)
            result.append(text);
        return result.toString().trim();
    }

    public static void main (String[] args){
        Stemmer.Stem("Michele Bachmann amenities pressed her allegations that the former head of her Iowa presidential bid was bribed by the campaign of rival Ron Paul to endorse him, even as one of her own aides denied the charge.", "English");
    }
}

TermAttribute类已被弃用,在Lucene 4中将不再受支持,但文档中并未明确说明在何处使用.

同样在第一个示例中,PorterStemmer不可用作类(隐藏),因此您无法直接使用它.

希望这可以帮助.