ANTLR4-动态注入令牌

Question

所以我正在编写一个python解析器,我需要根据python语法规范动态生成INDENT和DEDENT令牌(因为python不使用显式分隔符).

基本上我有一堆表示缩进级别的整数.在INDENT令牌中的嵌入式Java操作中,我检查当前缩进级别是否高于堆栈顶部的级别; 如果是的话,我就推开它; 如果没有,我打电话skip().

问题是,如果当前缩进级别与堆栈中的多个级别匹配,我必须生成多个DEDENT令牌,而我无法弄清楚如何执行此操作.

我目前的代码:(注意within_indent_block并current_indent_level在其他地方管理)

fragment DENT: {within_indent_block}? (SPACE|TAB)+;

INDENT: {within_indent_block}? DENT
        {if(current_indent_level > whitespace_stack.peek().intValue()){
                 whitespace_stack.push(new Integer(current_indent_level));
                 within_indent_block = false;
         }else{
                 skip();
         }
         }
         ;    

DEDENT: {within_indent_block}? DENT
        {if(current_indent_level < whitespace_stack.peek().intValue()){
            while(current_indent_level < whitespace_stack.peek().intValue()){
                      whitespace_stack.pop();
                      <<injectDedentToken()>>; //how do I do this
            }
         }else{
               skip();
         }
         }
         ;

我该怎么做和/或有更好的方法吗？

Answer 1

您发布的代码存在一些问题.

1. 该INDENT和DEDENT规则是相同的语义(考虑谓词和规则引用,但忽略动作).既然INDENT首先出现,这意味着你永远不会有DEDENT规则产生的令牌是这个语法.
2. 在{within_indent_block}?您引用前谓词出现DENT以及内部DENT片段规则本身.这种复制没有任何意义,但会减慢你的词法分析器.

匹配后动作的实际处理最好放在覆盖中Lexer.nextToken().例如,您可以从以下内容开始.

private final Deque<Token> pendingTokens = new ArrayDeque<>();

@Override
public Token nextToken() {
    while (pendingTokens.isEmpty()) {
        Token token = super.nextToken();
        switch (token.getType()) {
        case INDENT:
            // handle indent here. to skip this token, simply don't add
            // anything to the pendingTokens queue and super.nextToken()
            // will be called again.
            break;

        case DEDENT:
            // handle indent here. to skip this token, simply don't add
            // anything to the pendingTokens queue and super.nextToken()
            // will be called again.
            break;

        default:
            pendingTokens.add(token);
            break;
        }
    }

    return pendingTokens.poll();
}