引理化java

Question

我正在寻找Java中英语的lemmatisation实现.我已经发现了一些,但我需要一些不需要太多内存来运行的东西(1 GB顶部).谢谢.我不需要一个词干.

Answer 1

在斯坦福CoreNLP Java库包含lemmatizer就是有点资源密集型的,但我已经在我的笔记本电脑<512MB的RAM运行.

要使用它:

1. 下载jar文件 ;
2. 在您选择的编辑器中创建一个新项目/制作一个包含您刚刚下载的存档中包含的所有jar文件的ant脚本;
3. 创建一个新的Java,如下所示(基于斯坦福大学网站的片段);

import java.util.Properties;

public class StanfordLemmatizer {

    protected StanfordCoreNLP pipeline;

    public StanfordLemmatizer() {
        // Create StanfordCoreNLP object properties, with POS tagging
        // (required for lemmatization), and lemmatization
        Properties props;
        props = new Properties();
        props.put("annotators", "tokenize, ssplit, pos, lemma");

        // StanfordCoreNLP loads a lot of models, so you probably
        // only want to do this once per execution
        this.pipeline = new StanfordCoreNLP(props);
    }

    public List<String> lemmatize(String documentText)
    {
        List<String> lemmas = new LinkedList<String>();

        // create an empty Annotation just with the given text
        Annotation document = new Annotation(documentText);

        // run all Annotators on this text
        this.pipeline.annotate(document);

        // Iterate over all of the sentences found
        List<CoreMap> sentences = document.get(SentencesAnnotation.class);
        for(CoreMap sentence: sentences) {
            // Iterate over all tokens in a sentence
            for (CoreLabel token: sentence.get(TokensAnnotation.class)) {
                // Retrieve and add the lemma for each word into the list of lemmas
                lemmas.add(token.get(LemmaAnnotation.class));
            }
        }

        return lemmas;
    }
}

Answer 2

Chris对Standford Lemmatizer的回答很棒!简直美极了.他甚至包括一个指向jar文件的指针,所以我没有谷歌为它.

但他的一行代码有一个语法错误(他以某种方式切换了以"lemmas.add ..."开头的行中的结束括号和分号,并且他忘了包含导入.

就NoSuchMethodError错误而言,它通常是由于该方法不是公共静态引起的,但是如果你看一下代码本身(在http://grepcode.com/file/repo1.maven.org/maven2/com.guokr) /stan-cn-nlp/0.0.2/edu/stanford/nlp/util/Generics.java?av=h)这不是问题所在.我怀疑问题是在构建路径中的某个地方(我正在使用Eclipse Kepler,因此配置我在项目中使用的33个jar文件没有问题).

下面是我对Chris的代码的一些小修正,以及一个例子(我对Evanescence的屠宰他们完美的歌词表示道歉):

import java.util.LinkedList;
import java.util.List;
import java.util.Properties;

import edu.stanford.nlp.ling.CoreAnnotations.LemmaAnnotation;
import edu.stanford.nlp.ling.CoreAnnotations.SentencesAnnotation;
import edu.stanford.nlp.ling.CoreAnnotations.TokensAnnotation;
import edu.stanford.nlp.ling.CoreLabel;
import edu.stanford.nlp.pipeline.Annotation;
import edu.stanford.nlp.pipeline.StanfordCoreNLP;
import edu.stanford.nlp.util.CoreMap;

public class StanfordLemmatizer {

    protected StanfordCoreNLP pipeline;

    public StanfordLemmatizer() {
        // Create StanfordCoreNLP object properties, with POS tagging
        // (required for lemmatization), and lemmatization
        Properties props;
        props = new Properties();
        props.put("annotators", "tokenize, ssplit, pos, lemma");

        /*
         * This is a pipeline that takes in a string and returns various analyzed linguistic forms. 
         * The String is tokenized via a tokenizer (such as PTBTokenizerAnnotator), 
         * and then other sequence model style annotation can be used to add things like lemmas, 
         * POS tags, and named entities. These are returned as a list of CoreLabels. 
         * Other analysis components build and store parse trees, dependency graphs, etc. 
         * 
         * This class is designed to apply multiple Annotators to an Annotation. 
         * The idea is that you first build up the pipeline by adding Annotators, 
         * and then you take the objects you wish to annotate and pass them in and 
         * get in return a fully annotated object.
         * 
         *  StanfordCoreNLP loads a lot of models, so you probably
         *  only want to do this once per execution
         */
        this.pipeline = new StanfordCoreNLP(props);
    }

    public List<String> lemmatize(String documentText)
    {
        List<String> lemmas = new LinkedList<String>();
        // Create an empty Annotation just with the given text
        Annotation document = new Annotation(documentText);
        // run all Annotators on this text
        this.pipeline.annotate(document);
        // Iterate over all of the sentences found
        List<CoreMap> sentences = document.get(SentencesAnnotation.class);
        for(CoreMap sentence: sentences) {
            // Iterate over all tokens in a sentence
            for (CoreLabel token: sentence.get(TokensAnnotation.class)) {
                // Retrieve and add the lemma for each word into the
                // list of lemmas
                lemmas.add(token.get(LemmaAnnotation.class));
            }
        }
        return lemmas;
    }


    public static void main(String[] args) {
        System.out.println("Starting Stanford Lemmatizer");
        String text = "How could you be seeing into my eyes like open doors? \n"+
                "You led me down into my core where I've became so numb \n"+
                "Without a soul my spirit's sleeping somewhere cold \n"+
                "Until you find it there and led it back home \n"+
                "You woke me up inside \n"+
                "Called my name and saved me from the dark \n"+
                "You have bidden my blood and it ran \n"+
                "Before I would become undone \n"+
                "You saved me from the nothing I've almost become \n"+
                "You were bringing me to life \n"+
                "Now that I knew what I'm without \n"+
                "You can've just left me \n"+
                "You breathed into me and made me real \n"+
                "Frozen inside without your touch \n"+
                "Without your love, darling \n"+
                "Only you are the life among the dead \n"+
                "I've been living a lie, there's nothing inside \n"+
                "You were bringing me to life.";
        StanfordLemmatizer slem = new StanfordLemmatizer();
        System.out.println(slem.lemmatize(text));
    }

}

这是我的结果(我留下了非常深刻的印象;它抓住了"s"作为"是"(有时候),几乎完成其他所有事情):

启动斯坦福Lemmatizer

添加注释器标记化

添加注释器ssplit

添加注释器位置

从edu/stanford读取POS标记模型/ nlp/models/pos-tagger/english-left3words/english-left3words-distsim.tagger ...完成[1.7秒].

添加注释引理

[怎么样,可能,你,是,看,进入,我的,眼睛,像,开,门,？,你,领导,我,下来,进入,我的,核心,在哪里,我,有,成为,所以,麻木没有,一个,灵魂,我的,精神,睡觉,某个地方,冷,直到,你,找到,它,那里,然后,领导,它,回来,回家,你,唤醒,我,向上,在里面,打电话,我的,名字,以及,保存,我,从,黑暗,你,有,出价,我的,血,和,它,运行,之前,我,将,成为,撤消,你,保存,我,来自,没有,我,有,几乎,成为,你,成为,带来,我,生活,现在,那,我,知道,什么,我,是,没有,你,可以,拥有,只是,离开,我,你,呼吸,进入,我,和,使,我,真实,冷冻,内在,没有,你,触摸,没有,你,爱,,亲爱的,只有,你,是,生命,其中,死了,我,有,是,活着,一个,谎言,那里,没有,没有,里面,你,是,带来,我,对,生活,.

Answer 3

查看Lucene Snowball。