python中的模糊文本搜索

Question

我想知道是否有任何Python库可以进行模糊文本搜索.例如:

• 我有三个关键字"letter","stamp"和"mail".
• 我想有一个功能来检查这三个单词是否在同一段(或一定距离,一页)内.
• 另外,这些词必须保持相同的顺序.在这三个单词之间出现其他单词很好.

我试过fuzzywuzzy哪个没能解决我的问题.另一个库Whoosh看起来很强大,但我找不到合适的功能......

Answer 1

{1} 你可以这样做Whoosh 2.7.它通过添加插件进行模糊搜索whoosh.qparser.FuzzyTermPlugin:

whoosh.qparser.FuzzyTermPlugin让您搜索"模糊"术语,即不必完全匹配的术语.模糊术语将匹配特定数量的"编辑"(字符插入,删除和/或转置)中的任何类似术语 - 这称为"Damerau-Levenshtein编辑距离").

要添加模糊插件:

parser = qparser.QueryParser("fieldname", my_index.schema)
parser.add_plugin(qparser.FuzzyTermPlugin())

将模糊插件添加到解析器后,可以通过添加a ~后跟可选的最大编辑距离来指定模糊术语.如果未指定编辑距离,则默认值为1.

例如,以下"模糊"术语查询:

letter~
letter~2
letter~2/3

{2}要保持单词顺序,请使用查询,whoosh.query.Phrase但您应该替换Phrase插件whoosh.qparser.SequencePlugin,这样您就可以在短语中使用模糊术语:

"letter~ stamp~ mail~"

要使用序列插件替换默认短语插件:

parser = qparser.QueryParser("fieldname", my_index.schema)
parser.remove_plugin_class(qparser.PhrasePlugin)
parser.add_plugin(qparser.SequencePlugin())

{3}要允许两者之间的单词,slop请将您的短语查询中的arg 初始化为更大的数字:

whoosh.query.Phrase(fieldname, words, slop=1, boost=1.0, char_ranges=None)

slop - 短语中每个"单词"之间允许的单词数; 默认值为1表示短语必须完全匹配.

您还可以在Query中定义slop,如下所示:

"letter~ stamp~ mail~"~10

{4}整体解决方案:

{4.a} 索引器就像:

from whoosh.index import create_in
from whoosh.fields import *

schema = Schema(title=TEXT(stored=True), content=TEXT)
ix = create_in("indexdir", schema)
writer = ix.writer()
writer.add_document(title=u"First document", content=u"This is the first document we've added!")
writer.add_document(title=u"Second document", content=u"The second one is even more interesting!")
writer.add_document(title=u"Third document", content=u"letter first, stamp second, mail third")
writer.add_document(title=u"Fourth document", content=u"stamp first, mail third")
writer.add_document(title=u"Fivth document", content=u"letter first,  mail third")
writer.add_document(title=u"Sixth document", content=u"letters first, stamps second, mial third wrong")
writer.add_document(title=u"Seventh document", content=u"stamp first, letters second, mail third")
writer.commit()

{4.b} Searcher会像:

from whoosh.qparser import QueryParser, FuzzyTermPlugin, PhrasePlugin, SequencePlugin

with ix.searcher() as searcher:
    parser = QueryParser(u"content", ix.schema)
    parser.add_plugin(FuzzyTermPlugin())
    parser.remove_plugin_class(PhrasePlugin)
    parser.add_plugin(SequencePlugin())
    query = parser.parse(u"\"letter~2 stamp~2 mail~2\"~10")
    results = searcher.search(query)
    print "nb of results =", len(results)
    for r in results:
        print r

结果如下:

nb of results = 2
<Hit {'title': u'Sixth document'}>
<Hit {'title': u'Third document'}>

{5}如果要将模糊搜索设置为默认值而不使用word~n查询的每个单词中的语法,则可以QueryParser像这样初始化:

 from whoosh.query import FuzzyTerm
 parser = QueryParser(u"content", ix.schema, termclass = FuzzyTerm)

现在您可以使用查询,"letter stamp mail"~10但请记住,它FuzzyTerm具有默认编辑距离maxdist = 1.如果您想要更大的编辑距离,请个性化课程:

class MyFuzzyTerm(FuzzyTerm):
     def __init__(self, fieldname, text, boost=1.0, maxdist=2, prefixlength=1, constantscore=True):
         super(D, self).__init__(fieldname, text, boost, maxdist, prefixlength, constantscore) 
         # super().__init__() for Python 3 I think

参考文献:

1. whoosh.query.Phrase
2. 添加模糊术语查询
3. 允许复杂的短语查询
4. 班级whoosh.query.FuzzyTerm
5. qparser模块