导入后ANTLR4语法标记识别错误

Question

我正在使用GitHub中的antlr4 解析器语法和词法分析器语法来解析Python3中的PHP.

当我直接使用这些语法时,我的PoC代码可以工作:

antlr-test.py

from antlr4 import *
# from PHPParentLexer import PHPParentLexer
# from PHPParentParser import PHPParentParser
# from PHPParentParser import PHPParentListener

from PHPLexer import PHPLexer as PHPParentLexer
from PHPParser import PHPParser as PHPParentParser
from PHPParser import PHPParserListener as PHPParentListener


class PhpGrammarListener(PHPParentListener):
    def enterFunctionInvocation(self, ctx):
        print("enterFunctionInvocation " + ctx.getText())


if __name__ == "__main__":
    scanner_input = FileStream('test.php')
    lexer = PHPParentLexer(scanner_input)
    stream = CommonTokenStream(lexer)
    parser = PHPParentParser(stream)
    tree = parser.htmlDocument()
    walker = ParseTreeWalker()
    printer = PhpGrammarListener()
    walker.walk(printer, tree)

它给出了输出

/opt/local/bin/python3.4 /Users/d/PycharmProjects/name/antlr-test.py
enterFunctionInvocation echo("hi") 
enterFunctionInvocation another_method("String")
enterFunctionInvocation print("print statement")

Process finished with exit code 0

当我使用以下PHPParent.g4语法时,我收到很多错误:

grammar PHPParent;
options { tokenVocab=PHPLexer; }
import PHPParser;

在交换关于pythons导入的注释后,我收到此错误

/opt/local/bin/python3.4 /Users/d/PycharmProjects/name/antlr-test.py
line 1:1 token recognition error at: '?'
line 1:2 token recognition error at: 'p'
line 1:3 token recognition error at: 'h'
line 1:4 token recognition error at: 'p'
line 1:5 token recognition error at: '\n'
...
line 2:8 no viable alternative at input '<('
line 2:14 mismatched input ';' expecting {<EOF>, '<', '{', '}', ')', '?>', 'list', 'global', 'continue', 'return', 'class', 'do', 'switch', 'function', 'break', 'if', 'for', 'foreach', 'while', 'new', 'clone', '&', '!', '-', '~', '@', '$', <INVALID>, 'Interface', 'abstract', 'static', Array, RequireOperator, DecimalNumber, HexNumber, OctalNumber, Float, Boolean, SingleQuotedString, DoubleQuotedString_Start, Identifier, IncrementOperator}
line 3:28 mismatched input ';' expecting {<EOF>, '<', '{', '}', ')', '?>', 'list', 'global', 'continue', 'return', 'class', 'do', 'switch', 'function', 'break', 'if', 'for', 'foreach', 'while', 'new', 'clone', '&', '!', '-', '~', '@', '$', <INVALID>, 'Interface', 'abstract', 'static', Array, RequireOperator, DecimalNumber, HexNumber, OctalNumber, Float, Boolean, SingleQuotedString, DoubleQuotedString_Start, Identifier, IncrementOperator}
line 4:28 mismatched input ';' expecting {<EOF>, '<', '{', '}', ')', '?>', 'list', 'global', 'continue', 'return', 'class', 'do', 'switch', 'function', 'break', 'if', 'for', 'foreach', 'while', 'new', 'clone', '&', '!', '-', '~', '@', '$', <INVALID>, 'Interface', 'abstract', 'static', Array, RequireOperator, DecimalNumber, HexNumber, OctalNumber, Float, Boolean, SingleQuotedString, DoubleQuotedString_Start, Identifier, IncrementOperator}

但是,在语法上运行antlr4工具时,我没有遇到任何错误.我被困在这里 - 可能导致这个问题的原因是什么？

$ a4p PHPLexer.g4
warning(146): PHPLexer.g4:363:0: non-fragment lexer rule DoubleQuotedStringBody can match the empty string
$ a4p PHPParser.g4
warning(154): PHPParser.g4:523:0: rule doubleQuotedString contains an optional block with at least one alternative that can match an empty string
$ a4p PHPParent.g4
warning(154): PHPParent.g4:523:0: rule doubleQuotedString contains an optional block with at least one alternative that can match an empty string

Answer 1

导入是ANTLR4有点乱.

首先,tokenVocab无法生成所需的词法分析器.它只是意味着这个语法使用的是令牌PHPLexer.如果你删除PHPLexer.tokens,它甚至不会编译!

看看PHPParser.g4我们也在使用的地方options { tokenVocab=PHPLexer; }.然而在python脚本中,我们仍然需要使用lexer PHPLexer来使其工作.嗯,这PHPParentLexer根本不可用.这就是你得到所有错误的原因.

要从组合语法生成新的词法分析器,您需要像这样导入它:

grammar PHPParent;
import PHPLexer;

但是,mode导入时不支持.PHPLexer本身使用mode了很多.所以它也不是一个选择.

我们能简单地替换PHPParentLexer用PHPLexer？可悲的是没有.因为PHPParentParser是生成的PHPParentLexer,所以它们是紧密耦合的,不能单独使用.如果你使用PHPLexer,PHPParentParser也将无法正常工作.至于这个语法,由于错误恢复,它实际上工作,但给出一些错误.

似乎没有更好的方法,但重写一些语法.在importANTLR4的这一部分肯定存在一些设计问题.