Cam*_*ron 3 python parsing file-format
对于模糊的标题感到抱歉,但我真的不知道如何简洁地描述这个问题.
我创建了一个(或多或少)简单的特定于域的语言,我将用它来指定要应用于不同实体的验证规则(通常是从网页提交的表单).我在这篇文章的底部包含了一个示例语言的示例.
我的问题是我不知道如何开始将这种语言解析为我可以使用的形式(我将使用Python进行解析).我的目标是结束与规则/过滤器列表(如字符串,包括参数,如'cocoa(99)')应该应用(按顺序)到每个对象/实体(也是一个字符串,如'chocolate','chocolate.lindt'等).
我不确定使用什么技术开始,甚至不存在这样的问题的技术.您认为最好的解决方法是什么？我不是在寻找一个完整的解决方案,只是在正确的方向上进行一般性的推动.
谢谢.
语言示例文件:
# Comments start with the '#' character and last until the end of the line
# Indentation is significant (as in Python)
constant NINETY_NINE = 99 # Defines the constant `NINETY_NINE` to have the value `99`
*: # Applies to all data
isYummy # Everything must be yummy
chocolate: # To validate, say `validate("chocolate", object)`
sweet # chocolate must be sweet (but not necessarily chocolate.*)
lindt: # To validate, say `validate("chocolate.lindt", object)`
tasty # Applies only to chocolate.lindt (and not to chocolate.lindt.dark, for e.g.)
*: # Applies to all data under chocolate.lindt
smooth # Could also be written smooth()
creamy(1) # Level 1 creamy
dark: # dark has no special validation rules
extraDark:
melt # Filter that modifies the object being examined
c:bitter # Must be bitter, but only validated on client
s:cocoa(NINETY_NINE) # Must contain 99% cocoa, but only validated on server. Note constant
milk:
creamy(2) # Level 2 creamy, overrides creamy(1) of chocolate.lindt.* for chocolate.lindt.milk
creamy(3) # Overrides creamy(2) of previous line (all but the last specification of a given rule are ignored)
ruleset food: # To define a chunk of validation rules that can be expanded from the placeholder `food` (think macro)
caloriesWithin(10, 2000) # Unlimited parameters allowed
edible
leftovers: # Nested rules allowed in rulesets
stale
# Rulesets may be nested and/or include other rulesets in their definition
chocolate: # Previously defined groups can be re-opened and expanded later
ferrero:
hasHazelnut
cake:
tasty # Same rule used for different data (see chocolate.lindt)
isLie
ruleset food # Substitutes with rules defined for food; cake.leftovers must now be stale
pasta:
ruleset food # pasta.leftovers must also be stale
# Sample use (in JavaScript):
# var choc = {
# lindt: {
# cocoa: {
# percent: 67,
# mass: '27g'
# }
# }
# // Objects/groups that are ommitted (e.g. ferrro in this example) are not validated and raise no errors
# // Objects that are not defined in the validation rules do not raise any errors (e.g. cocoa in this example)
# };
# validate('chocolate', choc);
# `validate` called isYummy(choc), sweet(choc), isYummy(choc.lindt), smooth(choc.lindt), creamy(choc.lindt, 1), and tasty(choc.lindt) in that order
# `validate` returned an array of any validation errors that were found
# Order of rule validation for objects:
# The current object is initially the object passed in to the validation function (second argument).
# The entry point in the rule group hierarchy is given by the first argument to the validation function.
# 1. First all rules that apply to all objects (defined using '*') are applied to the current object,
# starting with the most global rules and ending with the most local ones.
# 2. Then all specific rules for the current object are applied.
# 3. Then a depth-first traversal of the current object is done, repeating steps 1 and 2 with each object found as the current object
# When two rules have equal priority, they are applied in the order they were defined in the file.
# No need to end on blank line
首先,如果你想学习解析,那么编写你自己的递归下降解析器.您定义的语言只需要少量制作.我建议使用Python的tokenize库来节省将字节流转换为令牌流的无聊任务.
对于实用的解析选项,请继续阅读...
一个快速而肮脏的解决方案是使用python本身:
NINETY_NINE = 99 # Defines the constant `NINETY_NINE` to have the value `99`
rules = {
'*': { # Applies to all data
'isYummy': {}, # Everything must be yummy
'chocolate': { # To validate, say `validate("chocolate", object)`
'sweet': {}, # chocolate must be sweet (but not necessarily chocolate.*)
'lindt': { # To validate, say `validate("chocolate.lindt", object)`
'tasty':{} # Applies only to chocolate.lindt (and not to chocolate.lindt.dark, for e.g.)
'*': { # Applies to all data under chocolate.lindt
'smooth': {} # Could also be written smooth()
'creamy': 1 # Level 1 creamy
},
# ...
}
}
}
有几种方法可以解决这个问题,例如,这是一种使用类的更清洁(虽然有点不寻常)的方法:
class _:
class isYummy: pass
class chocolate:
class sweet: pass
class lindt:
class tasty: pass
class _:
class smooth: pass
class creamy: level = 1
# ...
作为完整解析器的中间步骤,您可以使用"包含电池"的Python解析器,它解析Python语法并返回AST.AST非常深,有很多(IMO)不必要的级别.您可以通过剔除只有一个子节点的任何节点将这些结构过滤到更简单的结构.使用这种方法,您可以执行以下操作:
import parser, token, symbol, pprint
_map = dict(token.tok_name.items() + symbol.sym_name.items())
def clean_ast(ast):
if not isinstance(ast, list):
return ast
elif len(ast) == 2: # Elide single-child nodes.
return clean_ast(ast[1])
else:
return [_map[ast[0]]] + [clean_ast(a) for a in ast[1:]]
ast = parser.expr('''{
'*': { # Applies to all data
isYummy: _, # Everything must be yummy
chocolate: { # To validate, say `validate("chocolate", object)`
sweet: _, # chocolate must be sweet (but not necessarily chocolate.*)
lindt: { # To validate, say `validate("chocolate.lindt", object)`
tasty: _, # Applies only to chocolate.lindt (and not to chocolate.lindt.dark, for e.g.)
'*': { # Applies to all data under chocolate.lindt
smooth: _, # Could also be written smooth()
creamy: 1 # Level 1 creamy
}
# ...
}
}
}
}''').tolist()
pprint.pprint(clean_ast(ast))
这种方法确实有其局限性.最终的AST仍然有点嘈杂,你定义的语言必须可以解释为有效的python代码.例如,你不能支持这个......
*:
isYummy
...因为此语法不解析为python代码.然而,它的最大优点是你可以控制AST转换,因此不可能注入任意Python代码.