Skip to content
Hyperlark

Transformers

Transformers and visitors process the parse tree: you write code for each rule you wish to process, keyed by its name. There are four ways to run it:

PatternPython / TypeScriptRustC
Bottom-up, a value per nodeTransformerfold / fold_fnlark_fold_*
During the parse, straight to a valuetransformer= / { transformer }parse_fold_fnlark_reduce_fold_*, lark_parse_fold
Top-down, evaluate on demandInterpreterevalnone, recurse yourself
Side effects onlyVisitorCursor walkcursor walk

Run your code during the parse when all you need is the value. This needs LALR and has other limits. Run it on the tree when you also need the tree.

The grammar

Section titled “The grammar”

Every example uses this calculator grammar as GRAMMAR:

?start: sum
?sum: product
    | sum "+" product   -> add
    | sum "-" product   -> sub
?product: atom
    | product "*" atom  -> mul
    | product "/" atom  -> div
?atom: NUMBER           -> number
     | "-" atom         -> neg
     | "(" sum ")"
NUMBER: /[0-9]+/
%ignore " "

-> add names the node, and so the code that handles it. ? inlines a rule with one child, so an add node holds its operands (shaping the tree).

Bottom-up

Section titled “Bottom-up”

A transformer works bottom-up: each node’s callback receives its children, already transformed, and its return value replaces the node. The root’s value is the result.

from hyperlark import Lark, Transformer, v_args


@v_args(inline=True)                 # children arrive as positional args
class Calc(Transformer):
    def number(self, tok): return float(tok)
    def add(self, a, b):   return a + b
    def sub(self, a, b):   return a - b
    def mul(self, a, b):   return a * b
    def div(self, a, b):   return a / b
    def neg(self, a):      return -a


parser = Lark(GRAMMAR, parser="lalr")
tree = parser.parse("2 + 3 * 4")
print(Calc().transform(tree))        # 14.0

In Python and TypeScript:

  • A rule with no method calls __default__, which by default copies the node. A token with no method is returned as is.
  • A method named after a terminal (NUMBER) receives the token.
  • An exception in a method is wrapped in VisitError: obj is the node, orig_exc the exception.
  • Return Discard and the node won’t appear in the parent.
  • Transformer is recursive. For huge trees use Transformer_NonRecursive.
  • Python also has merge_transformers, for imported grammars (feature matrix).

v_args (Python and TypeScript)

Section titled “v_args (Python and TypeScript)”

By default, a rule method accepts one argument: a list of the node’s children. v_args modifies that, for one method or a whole class:

  • inline: positional arguments, add(a, b), as above.
  • meta: two arguments, meta and children. meta holds the node’s positions only with propagate_positions; without it, TypeScript passes undefined.
  • tree: the whole tree, add(tree).

In TypeScript a single method is wrapped as add = v_args({ inline: true })(function (a: number, b: number) { return a + b; }).

During the parse

Section titled “During the parse”

Give the parser your callbacks up front, and it returns the value instead of a tree. It is faster than transforming a tree; Python, Rust and C build no tree at all.

parser = Lark(GRAMMAR, parser="lalr", transformer=Calc())
print(parser.parse("2 + 3 * 4"))     # 14.0, no Tree built

Limits of this form:

  • With Earley, Python and TypeScript raise ConfigurationError, and Rust’s parse_fold_fn returns ParseError::Unsupported, as it also does with a custom lexer.
  • Python and TypeScript:
    • Discard is disabled; it reaches the parent as a value.
    • An exception in a callback propagates unchanged, with no VisitError wrapper.
  • Python:
    • A rule method with v_args(meta=True) raises NotImplementedError at construction.
    • __default__ receives meta=None, even with propagate_positions.
    • __default_token__ is not called.
    • Terminal methods run for every terminal the parser reads, including ones the tree drops (such as a "," named COMMA), even with visit_tokens=False. A %declared terminal, such as one a postlex inserts, gets no method call.
    • __default__ also runs for _-prefixed and internal rules (such as __start_plus_0 for item+). A method named after a _-prefixed rule raises ConfigurationError at construction, unless the transformer also overrides __default__. Except at the root, both must return a tree for these rules, or the parse raises AttributeError.
    • A chain of transformers (T1() * T2()) is ignored: parse() returns the tree.
  • TypeScript:
    • A callback named after a terminal (NUMBER) is never called.
    • By default the callbacks run after the whole input has parsed, so a parse error means none of them ran. batch_size on parse() (not the constructor) runs them during the parse, in groups of that many completed rules; 1 runs each as its rule completes. A custom lexer ignores batch_size and runs them over the finished tree.

Top-down

Section titled “Top-down”

An Interpreter (Python, TypeScript) walks the tree starting at the root (top-down), calling the method for each node according to tree.data. Unlike Transformer and Visitor, it doesn’t automatically visit its sub-branches: you call visit or visit_children explicitly, which allows branching and loops. A node with no method visits its children.

In Rust, one eval closure handles rules and tokens, and runs only for the root and the nodes you reach with n.eval(i). Both Interpreter and eval recurse, so a deep tree can overflow the stack. C has no interpreter: recurse with lark_node_child (reading a tree). The Examples page has Python and Rust interpreters.

from hyperlark.visitors import Interpreter


class Eval(Interpreter):
    def number(self, t): return float(t.children[0])
    def add(self, t): a, b = self.visit_children(t); return a + b
    def mul(self, t): a, b = self.visit_children(t); return a * b
    # sub, div, neg likewise


print(Eval().visit(tree))            # 14.0

Side effects

Section titled “Side effects”

A Visitor (Python, TypeScript) visits each node and runs the method for its data, then returns the tree. visit goes bottom-up, visit_topdown top-down. In Rust and C, walk the tree with a cursor.

from hyperlark import Visitor


class CountOps(Visitor):
    def __init__(self): self.ops = 0
    def add(self, tree): self.ops += 1
    def mul(self, tree): self.ops += 1


v = CountOps(); v.visit(tree); print(v.ops)   # 2