如何增加lucene索引中的位置偏移以对应<p>标签？

Question

我正在使用Lucene 3.0.3.为了准备使用SpanQuery和PhraseQuery,我想在一种方式中标记索引中的段落边界,以阻止这些查询跨段落边界匹配.我知道我需要PositionIncrementAttribute在处理文本时标记段落边界时将位置增加一些适当大的值.我们假设在源文档中,我的段落边界是<p>...</p>成对标记的.

如何设置令牌流以检测标签？另外,我实际上并不想自己索引标记.出于索引的目的,我宁愿增加下一个合法令牌的位置,而不是发出与该标签对应的令牌,因为我不希望它影响搜索.

Answer 1

添加间隙(= PositionIncrement> 1)的最简单方法是提供自定义TokenStream.您无需为此更改分析仪.但是,HTML解析应该在上游完成(即,在将输入文本提供给Lucene之前,应该相应地对输入文本进行分段和清理).

这是一个完整的工作示例(导入省略):

public class GapTest {

    public static void main(String[] args) throws Exception {
        final Directory dir = new RAMDirectory();
        final IndexWriterConfig iwConfig = new IndexWriterConfig(Version.LUCENE_4_10_1, new SimpleAnalyzer());
        final IndexWriter iw = new IndexWriter(dir, iwConfig);

        Document doc = new Document();
        doc.add(new TextField("body", "A B C", Store.YES));
        doc.add(new TextField("body", new PositionIncrementTokenStream(10)));
        doc.add(new TextField("body", "D E F", Store.YES));

        System.out.println(doc);
        iw.addDocument(doc);
        iw.close();

        final IndexReader ir = DirectoryReader.open(dir);
        IndexSearcher is = new IndexSearcher(ir);

        QueryParser qp = new QueryParser("body", new SimpleAnalyzer());

        for (String q : new String[] { "\"A B C\"", "\"A B C D\"",
                "\"A B C D\"", "\"A B C D\"~10", "\"A B C D E F\"~10",
                "\"A B C D F E\"~10", "\"A B C D F E\"~11" }) {
            Query query = qp.parse(q);
            TopDocs docs = is.search(query, 10);
            System.out.println(docs.totalHits + "\t" + q);
        }
        ir.close();
    }

    /**
     * A gaps-only TokenStream (uses {@link PositionIncrementAttribute}
     * 
     * @author Christian Kohlschuetter
     */
    private static final class PositionIncrementTokenStream extends TokenStream {
    private boolean first = true;
    private PositionIncrementAttribute attribute;
    private final int positionIncrement;

    public PositionIncrementTokenStream(final int positionIncrement) {
        super();
        this.positionIncrement = positionIncrement;
        attribute = addAttribute(PositionIncrementAttribute.class);
    }

    @Override
    public boolean incrementToken() throws IOException {
        if (first) {
            first = false;
            attribute.setPositionIncrement(positionIncrement);
            return true;
        } else {
            return false;
        }
    }

    @Override
    public void reset() throws IOException {
        super.reset();
        first = true;
    }
}

}