使用PyParsing进行增量但完整的解析？

Question

我正在使用PyParsing来解析一些类似C格式的大文本文件(大括号和分号以及所有这些).

PyParsing效果很好,但由于我的文件大小,它很慢并且消耗了大量内存.

因此,我想尝试实现一种增量解析方法,其中我将逐个解析源文件的顶级元素.scanStringpyparsing 的方法似乎是明显的方法.但是,我想确保在解析的部分之间没有无效/不可解析的文本scanString,并且无法找到执行此操作的好方法.

这是一个简化的例子,显示了我遇到的问题:

sample="""f1(1,2,3); f2_no_args( );
# comment out: foo(4,5,6);
bar(7,8);
this should be an error;
baz(9,10);
"""

from pyparsing import *

COMMENT=Suppress('#' + restOfLine())
SEMI,COMMA,LPAREN,RPAREN = map(Suppress,';,()')

ident = Word(alphas, alphanums+"_")
integer = Word(nums+"+-",nums)

statement = ident("fn") + LPAREN + Group(Optional(delimitedList(integer)))("arguments") + RPAREN + SEMI

p = statement.ignore(COMMENT)

for res, start, end in p.scanString(sample):
    print "***** (%d,%d)" % (start, end)
    print res.dump()

输出:

***** (0,10)
['f1', ['1', '2', '3']]
- arguments: ['1', '2', '3']
- fn: f1
***** (11,25)
['f2_no_args', []]
- arguments: []
- fn: f2_no_args
***** (53,62)
['bar', ['7', '8']]
- arguments: ['7', '8']
- fn: bar
***** (88,98)
['baz', ['9', '10']]
- arguments: ['9', '10']
- fn: baz

返回的范围scanString由于它们之间的未解析文本而具有间隙((0,10),(11,25),(53,62),(88,98)).其中两个空白是空格或注释,不应该触发错误,但其中一个(this should be an error;)包含我要捕获的不可解析的文本.

有没有办法使用pyparsing以递增方式解析文件,同时仍然确保可以使用指定的解析器语法解析整个输入？

Answer 1

在对PyParsing用户的邮件列表进行简短讨论之后,我想出了一个看起来相当不错的解决方案.

我ParserElement.parseString稍微修改了这个方法parseConsumeString,这就是我想要的.这个版本ParserElement._parse后来ParserElement.preParse反复调用.

以下是ParserElement使用该parseConsumeString方法进行猴子补丁的代码:

from pyparsing import ParseBaseException, ParserElement

def parseConsumeString(self, instring, parseAll=True, yieldLoc=False):
    '''Generator version of parseString which does not try to parse
    the whole string at once.

    Should be called with a top-level parser that could parse the
    entire string if called repeatedly on the remaining pieces.
    Instead of:

        ZeroOrMore(TopLevel)).parseString(s ...)

    Use:

        TopLevel.parseConsumeString(s ...)

    If yieldLoc==True, it will yield a tuple of (tokens, startloc, endloc).
    If False, it will yield only tokens (like parseString).

    If parseAll==True, it will raise an error as soon as a parse
    error is encountered. If False, it will return as soon as a parse
    error is encountered (possibly before yielding any tokens).'''

    if not self.streamlined:
        self.streamline()
        #~ self.saveAsList = True
    for e in self.ignoreExprs:
        e.streamline()
    if not self.keepTabs:
        instring = instring.expandtabs()
    try:
        sloc = loc = 0
        while loc<len(instring):
            # keeping the cache (if in use) across loop iterations wastes memory (can't backtrack outside of loop)
            ParserElement.resetCache()
            loc, tokens = self._parse(instring, loc)
            if yieldLoc:
                yield tokens, sloc, loc
            else:
                yield tokens
            sloc = loc = self.preParse(instring, loc)
    except ParseBaseException as exc:
        if not parseAll:
            return
        elif ParserElement.verbose_stacktrace:
            raise
        else:
            # catch and re-raise exception from here, clears out pyparsing internal stack trace
            raise exc

def monkey_patch():
    ParserElement.parseConsumeString = parseConsumeString

请注意,我还将调用移动ParserElement.resetCache到每个循环迭代中.因为不可能从每个循环中回溯,所以不需要跨迭代保留缓存.使用PyParsing的packrat缓存功能时,这大大减少了内存消耗.在我使用10 MiB输入文件的测试中,峰值内存消耗从~6G下降到~100M峰值,而运行速度提高约15-20%.