Zea*_*rin 6 python testing parsing pyparsing
所以,我正在重构一个项目,我正在分离出一堆解析代码.我关心的代码是pyparsing.
即使花了很多时间阅读官方文档,我对pyparsing的理解也很差.我遇到了麻烦,因为(1)pyparsing采用(故意)非正统的方法进行解析,(2)我正在编写我没写的代码,评论很差,以及现有语法的非基本集合.
(我无法与原作者取得联系.)
我正在使用PyVows来测试我的代码.我的一个测试如下(我认为即使你不熟悉PyVows也很清楚;如果不是,请告诉我):
def test_multiline_command_ends(self, topic):
output = parsed_input('multiline command ends\n\n',topic)
expect(output).to_equal(
r'''['multiline', 'command ends', '\n', '\n']
- args: command ends
- multiline_command: multiline
- statement: ['multiline', 'command ends', '\n', '\n']
- args: command ends
- multiline_command: multiline
- terminator: ['\n', '\n']
- terminator: ['\n', '\n']''')
但是当我运行测试时,我在终端中得到以下信息:
Expected topic("['multiline', 'command ends']\n- args: command ends\n- command: multiline\n- statement: ['multiline', 'command ends']\n - args: command ends\n - command: multiline")
to equal "['multiline', 'command ends', '\\n', '\\n']\n- args: command ends\n- multiline_command: multiline\n- statement: ['multiline', 'command ends', '\\n', '\\n']\n - args: command ends\n - multiline_command: multiline\n - terminator: ['\\n', '\\n']\n- terminator: ['\\n', '\\n']"
注意:
由于输出是终端,预期输出(第二个)有额外的反斜杠.这个是正常的.在重构开始之前,测试没有问题.
第一行输出应与第二行匹配,但事实并非如此.具体来说,它不包括第一个列表对象中的两个换行符.
所以我得到了这个:
"['multiline', 'command ends']\n- args: command ends\n- command: multiline\n- statement: ['multiline', 'command ends']\n - args: command ends\n - command: multiline"
什么时候我应该得到这个:
"['multiline', 'command ends', '\\n', '\\n']\n- args: command ends\n- multiline_command: multiline\n- statement: ['multiline', 'command ends', '\\n', '\\n']\n - args: command ends\n - multiline_command: multiline\n - terminator: ['\\n', '\\n']\n- terminator: ['\\n', '\\n']"
在代码的早期,还有这样的声明:
pyparsing.ParserElement.setDefaultWhitespaceChars(' \t')
...我认为应该防止这种错误.但我不确定.
即使无法确定问题,只需缩小问题的位置将是一个巨大的帮助.
请告诉我如何解决这个问题.
编辑:那么,呃,我应该为此发布解析器代码,不应该吗？(感谢小费,@ andrew cooke!)
这是__init__我的解析器对象.
我知道这是一场噩梦.这就是我重构项目的原因.☺
def __init__(self, Cmd_object=None, *args, **kwargs):
# @NOTE
# This is one of the biggest pain points of the existing code.
# To aid in readability, I CAPITALIZED all variables that are
# not set on `self`.
#
# That means that CAPITALIZED variables aren't
# used outside of this method.
#
# Doing this has allowed me to more easily read what
# variables become a part of other variables during the
# building-up of the various parsers.
#
# I realize the capitalized variables is unorthodox
# and potentially anti-convention. But after reaching out
# to the project's creator several times over roughly 5
# months, I'm still working on this project alone...
# And without help, this is the only way I can move forward.
#
# I have a very poor understanding of the parser's
# control flow when the user types a command and hits ENTER,
# and until the author (or another pyparsing expert)
# explains what's happening to me, I have to do silly
# things like this. :-|
#
# Of course, if the impossible happens and this code
# gets cleaned up, then the variables will be restored to
# proper capitalization.
#
# —Zearin
# http://github.com/zearin/
# 2012 Mar 26
if Cmd_object is not None:
self.Cmd_object = Cmd_object
else:
raise Exception('Cmd_object be provided to Parser.__init__().')
# @FIXME
# Refactor methods into this class later
preparse = self.Cmd_object.preparse
postparse = self.Cmd_object.postparse
self._allow_blank_lines = False
self.abbrev = True # Recognize abbreviated commands
self.case_insensitive = True # Commands recognized regardless of case
# make sure your terminators are not in legal_chars!
self.legal_chars = u'!#$%.:?@_' + PYP.alphanums + PYP.alphas8bit
self.multiln_commands = [] if 'multiline_commands' not in kwargs else kwargs['multiln_commands']
self.no_special_parse = {'ed','edit','exit','set'}
self.redirector = '>' # for sending output to file
self.reserved_words = []
self.shortcuts = { '?' : 'help' ,
'!' : 'shell',
'@' : 'load' ,
'@@': '_relative_load'
}
# self._init_grammars()
#
# def _init_grammars(self):
# @FIXME
# Add Docstring
# ----------------------------
# Tell PYP how to parse
# file input from '< filename'
# ----------------------------
FILENAME = PYP.Word(self.legal_chars + '/\\')
INPUT_MARK = PYP.Literal('<')
INPUT_MARK.setParseAction(lambda x: '')
INPUT_FROM = FILENAME('INPUT_FROM')
INPUT_FROM.setParseAction( self.Cmd_object.replace_with_file_contents )
# ----------------------------
#OUTPUT_PARSER = (PYP.Literal('>>') | (PYP.WordStart() + '>') | PYP.Regex('[^=]>'))('output')
OUTPUT_PARSER = (PYP.Literal( 2 * self.redirector) | \
(PYP.WordStart() + self.redirector) | \
PYP.Regex('[^=]' + self.redirector))('output')
PIPE = PYP.Keyword('|', identChars='|')
STRING_END = PYP.stringEnd ^ '\nEOF'
TERMINATORS = [';']
TERMINATOR_PARSER = PYP.Or([
(hasattr(t, 'parseString') and t)
or
PYP.Literal(t) for t in TERMINATORS
])('terminator')
self.comment_grammars = PYP.Or([ PYP.pythonStyleComment,
PYP.cStyleComment ])
self.comment_grammars.ignore(PYP.quotedString)
self.comment_grammars.setParseAction(lambda x: '')
self.comment_grammars.addParseAction(lambda x: '')
self.comment_in_progress = '/*' + PYP.SkipTo(PYP.stringEnd ^ '*/')
# QuickRef: Pyparsing Operators
# ----------------------------
# ~ creates NotAny using the expression after the operator
#
# + creates And using the expressions before and after the operator
#
# | creates MatchFirst (first left-to-right match) using the
# expressions before and after the operator
#
# ^ creates Or (longest match) using the expressions before and
# after the operator
#
# & creates Each using the expressions before and after the operator
#
# * creates And by multiplying the expression by the integer operand;
# if expression is multiplied by a 2-tuple, creates an And of
# (min,max) expressions (similar to "{min,max}" form in
# regular expressions); if min is None, intepret as (0,max);
# if max is None, interpret as expr*min + ZeroOrMore(expr)
#
# - like + but with no backup and retry of alternatives
#
# * repetition of expression
#
# == matching expression to string; returns True if the string
# matches the given expression
#
# << inserts the expression following the operator as the body of the
# Forward expression before the operator
# ----------------------------
DO_NOT_PARSE = self.comment_grammars | \
self.comment_in_progress | \
PYP.quotedString
# moved here from class-level variable
self.URLRE = re.compile('(https?://[-\\w\\./]+)')
self.keywords = self.reserved_words + [fname[3:] for fname in dir( self.Cmd_object ) if fname.startswith('do_')]
# not to be confused with `multiln_parser` (below)
self.multiln_command = PYP.Or([
PYP.Keyword(c, caseless=self.case_insensitive)
for c in self.multiln_commands
])('multiline_command')
ONELN_COMMAND = ( ~self.multiln_command +
PYP.Word(self.legal_chars)
)('command')
#self.multiln_command.setDebug(True)
# Configure according to `allow_blank_lines` setting
if self._allow_blank_lines:
self.blankln_termination_parser = PYP.NoMatch
else:
BLANKLN_TERMINATOR = (2 * PYP.lineEnd)('terminator')
#BLANKLN_TERMINATOR('terminator')
self.blankln_termination_parser = (
(self.multiln_command ^ ONELN_COMMAND)
+ PYP.SkipTo(
BLANKLN_TERMINATOR,
ignore=DO_NOT_PARSE
).setParseAction(lambda x: x[0].strip())('args')
+ BLANKLN_TERMINATOR
)('statement')
# CASE SENSITIVITY for
# ONELN_COMMAND and self.multiln_command
if self.case_insensitive:
# Set parsers to account for case insensitivity (if appropriate)
self.multiln_command.setParseAction(lambda x: x[0].lower())
ONELN_COMMAND.setParseAction(lambda x: x[0].lower())
self.save_parser = ( PYP.Optional(PYP.Word(PYP.nums)^'*')('idx')
+ PYP.Optional(PYP.Word(self.legal_chars + '/\\'))('fname')
+ PYP.stringEnd)
AFTER_ELEMENTS = PYP.Optional(PIPE +
PYP.SkipTo(
OUTPUT_PARSER ^ STRING_END,
ignore=DO_NOT_PARSE
)('pipeTo')
) + \
PYP.Optional(OUTPUT_PARSER +
PYP.SkipTo(
STRING_END,
ignore=DO_NOT_PARSE
).setParseAction(lambda x: x[0].strip())('outputTo')
)
self.multiln_parser = (((self.multiln_command ^ ONELN_COMMAND)
+ PYP.SkipTo(
TERMINATOR_PARSER,
ignore=DO_NOT_PARSE
).setParseAction(lambda x: x[0].strip())('args')
+ TERMINATOR_PARSER)('statement')
+ PYP.SkipTo(
OUTPUT_PARSER ^ PIPE ^ STRING_END,
ignore=DO_NOT_PARSE
).setParseAction(lambda x: x[0].strip())('suffix')
+ AFTER_ELEMENTS
)
#self.multiln_parser.setDebug(True)
self.multiln_parser.ignore(self.comment_in_progress)
self.singleln_parser = (
( ONELN_COMMAND + PYP.SkipTo(
TERMINATOR_PARSER
^ STRING_END
^ PIPE
^ OUTPUT_PARSER,
ignore=DO_NOT_PARSE
).setParseAction(lambda x:x[0].strip())('args'))('statement')
+ PYP.Optional(TERMINATOR_PARSER)
+ AFTER_ELEMENTS)
#self.multiln_parser = self.multiln_parser('multiln_parser')
#self.singleln_parser = self.singleln_parser('singleln_parser')
self.prefix_parser = PYP.Empty()
self.parser = self.prefix_parser + (STRING_END |
self.multiln_parser |
self.singleln_parser |
self.blankln_termination_parser |
self.multiln_command +
PYP.SkipTo(
STRING_END,
ignore=DO_NOT_PARSE)
)
self.parser.ignore(self.comment_grammars)
# a not-entirely-satisfactory way of distinguishing
# '<' as in "import from" from
# '<' as in "lesser than"
self.input_parser = INPUT_MARK + \
PYP.Optional(INPUT_FROM) + \
PYP.Optional('>') + \
PYP.Optional(FILENAME) + \
(PYP.stringEnd | '|')
self.input_parser.ignore(self.comment_in_progress)
我怀疑问题是pyparsing的内置空白跳过,默认会跳过换行符.尽管setDefaultWhitespaceChars用于告诉pyparsing新行很重要,但此设置仅影响调用后创建的所有表达式setDefaultWhitespaceChars.问题在于,pyparsing尝试通过在导入时定义一些便利表达来帮助,例如emptyfor Empty(),lineEndfor LineEnd()等.但由于这些都是在导入时创建的,因此它们使用原始的默认空白字符定义,其中包括'\n'.
我应该这样做setDefaultWhitespaceChars,但你也可以自己清理一下.在调用之后setDefaultWhitespaceChars,在pyparsing中重新定义这些模块级表达式:
PYP.ParserElement.setDefaultWhitespaceChars(' \t')
# redefine module-level constants to use new default whitespace chars
PYP.empty = PYP.Empty()
PYP.lineEnd = PYP.LineEnd()
PYP.stringEnd = PYP.StringEnd()
我认为这有助于恢复嵌入式换行的重要性.
解析器代码上的其他一些位:
self.blankln_termination_parser = PYP.NoMatch
应该
self.blankln_termination_parser = PYP.NoMatch()
您的原始作者可能过于积极地使用'^'而不是'|'.只有当你真的解析了一个较长的表达式后面的一个表达式时,才会使用'^',而后者会在备选列表中稍后解析.例如,在:
self.save_parser = ( PYP.Optional(PYP.Word(PYP.nums)^'*')('idx')
数字或单独的数字之间不可能混淆'*'. Or(或'^'运算符)告诉pyparsing尝试评估所有替代方案,然后选择最长的匹配方 - 如果出现平局,请选择列表中最左侧的替代方案.如果解析'*',则无需查看是否也可能匹配较长的整数,或者如果解析整数,则无需查看它是否也可以作为单独传递'*'.所以改成这个:
self.save_parser = ( PYP.Optional(PYP.Word(PYP.nums)|'*')('idx')
使用解析操作将字符串替换为''更简单地使用PYP.Suppress包装器编写,或者如果您愿意,则调用expr.suppress()哪个返回Suppress(expr).加上对'|'的偏好 在'^',这个:
self.comment_grammars = PYP.Or([ PYP.pythonStyleComment,
PYP.cStyleComment ])
self.comment_grammars.ignore(PYP.quotedString)
self.comment_grammars.setParseAction(lambda x: '')
becomse:
self.comment_grammars = (PYP.pythonStyleComment | PYP.cStyleComment
).ignore(PYP.quotedString).suppress()
关键字具有内置逻辑以自动避免歧义,因此或完全不需要它们:
self.multiln_command = PYP.Or([
PYP.Keyword(c, caseless=self.case_insensitive)
for c in self.multiln_commands
])('multiline_command')
应该:
self.multiln_command = PYP.MatchFirst([
PYP.Keyword(c, caseless=self.case_insensitive)
for c in self.multiln_commands
])('multiline_command')
(在下一个版本中,我将放松那些初始值设定项以接受生成器表达式,以便[]将变得不必要.)
这就是我现在所能看到的.希望这可以帮助.
Pyparsing 没有错！
\n我曾是。\xe2\x98\xb9
\n通过将解析代码分离到不同的对象中，我产生了问题。最初，属性用于根据第二个属性的内容\xe2\x80\x9c 更新自身\xe2\x80\x9d。由于这一切过去都包含在一个 \xe2\x80\x9cgod 类\xe2\x80\x9d 中，所以它工作得很好。
\n只需将代码分离到另一个对象中，第一个属性就在实例化时设置，但如果它所依赖的第二个属性发生更改，则不再 \xe2\x80\x9c 更新自身\xe2\x80\x9d 。
\n该属性multiln_command（不要与multiln_commands\xe2\x80\x94aargh 混淆，命名多么令人困惑！）是一个 pyparsing 语法定义。如果出现以下情况，该multiln_command属性应该已更新其语法：multiln_commands。
尽管我知道这两个属性具有相似的名称，但用途却截然不同，但相似性无疑使追踪问题变得更加困难。我没有重命名multiln_command为multiln_grammar.
我很感谢@Paul McGuire\xe2\x80\x99s 很棒的答案，我希望它能在未来为我（和其他人）节省一些悲伤。虽然我觉得自己引起了这个问题有点愚蠢（并将其误诊为 pyparsing 问题），但我很高兴提出这个问题得到了一些好的建议（以 Paul\xe2\x80\x99s 的形式） 。
\n祝大家解析愉快。:)
\n