Skip to content
Hyperlark

Examples

Each section is one task, with a complete program for each language the table marks. Grammar and input are inline, so a file runs on its own, and its header comment has the run command. To install, see getting started for Python, TypeScript, Rust or C.

Hyperlark parses with LALR or Earley. Python, TypeScript and Rust default to Earley, so their programs select LALR where they use it. C has only LALR for now.

TaskPythonTypeScriptRustC
Calculator: compute the value instead of a tree✅✅✅✅
JSON: transform, query or streamtransformstream, queryquery
Interpreter: choose which children to visit✅✅✗
Indentation: INDENT and DEDENT tokens✅✗
Ambiguity: keep every reading✅✅✅✗
Interactive: ask what comes next✅✅✅✅
Error reporting: classify a failure by example✅✗✗✗

An empty cell means no program yet; ✗ means the language lacks the feature (feature matrix). The JSON cells name what each program does.

Calculator

Section titled “Calculator”

Evaluate arithmetic: each program’s transformer (in Rust and C, its fold function) computes the value instead of a tree (During the parse). This needs LALR (LALR and Earley).

docs/examples/python/calc.py
"""Calculator: parse arithmetic, print the tree, then evaluate during the parse
with transformer=, so no tree is built. Adapted from Lark's calculator example,
without the variables and the REPL.


Run:
    pip install hyperlark
    python calc.py
"""


import hyperlark
from hyperlark import Transformer, v_args


GRAMMAR = r"""
?start: sum
?sum: product
    | sum "+" product  -> add
    | sum "-" product  -> sub
?product: atom
    | product "*" atom -> mul
    | product "/" atom -> div
?atom: NUMBER          -> number
     | "-" atom        -> neg
     | "(" sum ")"


%import common.NUMBER
%import common.WS_INLINE
%ignore WS_INLINE
"""




@v_args(inline=True)  # children arrive as positional arguments
class Calc(Transformer):
    # Each rule name is bound to the matching operator function.
    from operator import add, mul, neg, sub
    from operator import truediv as div


    number = float




# A plain parser first, to look at the tree: aliases (`-> add`) label the nodes.
parser = hyperlark.Lark(GRAMMAR, parser="lalr")
print(parser.parse("2 + 3 * 4").pretty())


# transformer= runs Calc during the parse: no tree is built, and parse()
# returns the finished number.
calc = hyperlark.Lark(GRAMMAR, parser="lalr", transformer=Calc())
for expr in ["2 + 3 * 4", "(2 + 3) * 4", "-(10 / 4) + 0.5"]:
    print(f"{expr} = {calc.parse(expr)}")


# Parse errors carry a position and the caret context.
try:
    calc.parse("1 + * 2")
except hyperlark.exceptions.UnexpectedInput as e:
    print(f"as expected, '1 + * 2' fails at line {e.line}, column {e.column}:")
    print(e.get_context("1 + * 2"))

JSON

Section titled “JSON”

Get data out of a parsed document: convert it to native values, query the tree for the nodes you want, or stream the subtrees you choose to a callback as each completes. C has no program here yet; it can query and transform a tree, but not stream subtrees.

The program’s transformer turns the document into a dict, run both ways: on the finished tree, then during the parse (Transformers).

docs/examples/python/json_to_dict.py
"""Parse JSON into plain Python objects with a Transformer, run two ways:
  1. on the finished tree:  ToDict().transform(parser.parse(text))
  2. during the parse:      Lark(..., transformer=ToDict()), which builds
     no tree and is faster on large inputs


Adapted from Lark's JSON tutorial.


Run:
    pip install hyperlark
    python json_to_dict.py
"""


import json


import hyperlark
from hyperlark import Transformer, v_args


GRAMMAR = r"""
?start: value
?value: object
      | array
      | string
      | SIGNED_NUMBER    -> number
      | "true"           -> true
      | "false"          -> false
      | "null"           -> null
array  : "[" [value ("," value)*] "]"
object : "{" [pair ("," pair)*] "}"
pair   : string ":" value
string : ESCAPED_STRING


%import common.ESCAPED_STRING
%import common.SIGNED_NUMBER
%import common.WS
%ignore WS
"""


DOC = """
{
  "name": "hyperlark",
  "engines": ["lalr", "earley"],
  "bindings": {"python": true, "js": true, "c": true},
  "empty": [],
  "holes": [null, 1],
  "version": 0.1
}
"""




class ToDict(Transformer):
    """Each method replaces the rule of the same name with a Python value;
    rules whose children are already final map straight onto builtins."""


    @v_args(inline=True)
    def string(self, s):
        # Tokens are FastToken by default, not a str subclass, so json.loads
        # needs str() first (or pass fast_tokens=False to Lark). json.loads
        # then unquotes and unescapes.
        return json.loads(str(s))


    @v_args(inline=True)
    def number(self, n):
        return float(n)


    array = list
    pair = tuple
    object = dict


    def true(self, _):
        return True


    def false(self, _):
        return False


    def null(self, _):
        return None




# maybe_placeholders=False: otherwise an empty `[...]` optional inserts a None
# child, indistinguishable from a JSON null after the transformer runs.
OPTS = dict(parser="lalr", maybe_placeholders=False)


# 1. Post-parse: parse to a tree, then transform it.
tree_parser = hyperlark.Lark(GRAMMAR, **OPTS)
via_tree = ToDict().transform(tree_parser.parse(DOC))


# 2. During the parse: no tree is built.
folding_parser = hyperlark.Lark(GRAMMAR, transformer=ToDict(), **OPTS)
via_fold = folding_parser.parse(DOC)


assert via_tree == via_fold == json.loads(DOC)
print(json.dumps(via_fold, indent=2))
print("matches json.loads:", via_fold == json.loads(DOC))

Interpreter

Section titled “Interpreter”

Run a program from its tree. A loop or an if must decide which children to visit and how often. An Interpreter walks the tree starting at the root (top-down), calling the method for each node according to tree.data. Unlike Transformer and Visitor, it doesn’t automatically visit its sub-branches: you call visit or visit_children explicitly, which allows branching and loops. In Rust, eval does the same; C has none. See Top-down.

Python’s program interprets a turtle language whose repeat visits its block N times; Rust’s evaluates only the branch an if takes. TypeScript has the same Interpreter class as Python.

docs/examples/python/dsl_interpreter.py
"""A small turtle language, run with an Interpreter. Adapted from Lark's turtle
example, without the graphics window: it computes the pen's path instead.


An Interpreter visits top-down, and each method decides which children to
visit and how often, so `repeat N { ... }` runs its block N times. A
Transformer visits every node once, bottom-up: by the time `repeat` runs, its
block has already been transformed. (Compare rust/src/bin/interpreter.rs.)


Run:
    pip install hyperlark
    python dsl_interpreter.py
"""


import math


import hyperlark
from hyperlark.visitors import Interpreter


# f/b = forward/back, l/r = turn left/right (degrees). `repeat N { ... }` takes
# a nested block.
GRAMMAR = r"""
start: instruction+


?instruction: MOVE NUMBER              -> move
            | TURN NUMBER              -> turn
            | "repeat" NUMBER block    -> repeat


block: "{" instruction+ "}"


MOVE: "f" | "b"
TURN: "l" | "r"


%import common.NUMBER
%import common.WS
%ignore WS
"""


PROGRAM = """
repeat 4 {
    f 100
    r 90
}
f 50
"""




class Turtle(Interpreter):
    """Walks the tree top-down, keeping pen state as it goes. Each method is
    named after the rule, or the `-> alias`, it handles. `repeat` re-visits its
    block. `start` and `block` only recurse, as the inherited __default__ would."""


    def __init__(self):
        self.x = self.y = 0.0
        self.heading = 0.0            # degrees; 0 = +x axis
        self.segments = []            # (x0, y0, x1, y1) for each 'f'/'b' move


    def start(self, tree):
        self.visit_children(tree)     # run each top-level instruction in order


    def move(self, tree):
        direction, dist = tree.children          # MOVE token, NUMBER token
        step = float(dist) * (1 if direction == "f" else -1)
        rad = math.radians(self.heading)
        x1 = self.x + step * math.cos(rad)
        y1 = self.y + step * math.sin(rad)
        self.segments.append((self.x, self.y, x1, y1))
        self.x, self.y = x1, y1


    def turn(self, tree):
        direction, degrees = tree.children
        self.heading += float(degrees) * (1 if direction == "l" else -1)


    def repeat(self, tree):
        count, block = tree.children             # NUMBER token, block subtree
        for _ in range(int(count)):
            self.visit(block)                    # visit the same block N times


    def block(self, tree):
        self.visit_children(tree)




parser = hyperlark.Lark(GRAMMAR, parser="lalr")
turtle = Turtle()
turtle.visit(parser.parse(PROGRAM))


# `+ 0.0` normalises the -0.0 that trig at right angles produces, so the closing
# corner reads (0.0, 0.0) rather than (-0.0, -0.0).
def _z(v, places=1):
    return round(v, places) + 0.0




print(f"{len(turtle.segments)} pen strokes:")
for x0, y0, x1, y1 in turtle.segments:
    print(f"  ({_z(x0):6.1f}, {_z(y0):6.1f}) -> ({_z(x1):6.1f}, {_z(y1):6.1f})")


xs = [c for seg in turtle.segments for c in (seg[0], seg[2])]
ys = [c for seg in turtle.segments for c in (seg[1], seg[3])]
print(f"bounding box: x [{_z(min(xs), 0):.0f}, {_z(max(xs), 0):.0f}]  "
      f"y [{_z(min(ys), 0):.0f}, {_z(max(ys), 0):.0f}]")

Indentation

Section titled “Indentation”

Languages like Python and YAML carry structure in their leading whitespace. The grammar declares _INDENT / _DEDENT; the bundled Indenter postlexer generates these tokens based on the indentation it sees.

Rust has the same Indenter. TypeScript has only a fixed preset, whose newline terminal is _NEWLINE, not this program’s _NL. C has no postlex. See the lexers guide.

docs/examples/python/indented_language.py
"""Parse an indentation-based language with the Indenter postlex: the grammar
declares _INDENT and _DEDENT, and the Indenter emits them from each line's
leading whitespace.


Run:
    pip install hyperlark
    python indented_language.py
"""


import hyperlark
from hyperlark.indenter import Indenter


GRAMMAR = r"""
?start: _NL* tree
tree: NAME _NL [_INDENT tree+ _DEDENT]


NAME: /\w+/
_NL: /(\r?\n[\t ]*)+/


%declare _INDENT _DEDENT


// _NL matches the newline and the next line's indentation. WS_INLINE
// matches spaces inside a line and is ignored.
%import common.WS_INLINE
%ignore WS_INLINE
"""


DOC = """\
root
    branch
        leaf
        leaf2
    branch2
"""




class TreeIndenter(Indenter):
    NL_type = "_NL"
    OPEN_PAREN_types = []   # brackets that suspend indentation, e.g. ["LPAR"]
    CLOSE_PAREN_types = []
    INDENT_type = "_INDENT"
    DEDENT_type = "_DEDENT"
    tab_len = 8




parser = hyperlark.Lark(GRAMMAR, parser="lalr", postlex=TreeIndenter())
print(parser.parse(DOC).pretty())

Ambiguity

Section titled “Ambiguity”

Some grammars can be interpreted in more than one way, and for some inputs can result in many possible derivations. Hyperlark’s Earley can store each derivation in an efficiently packed forest.

With the explicit ambiguity option, the parser returns all derivations, wrapped in _ambig tree nodes; the default, resolve, chooses one. Python’s program runs CollapseAmbiguities, which transforms a tree that contains any number of _ambig nodes into a list of trees, each one unambiguous. TypeScript and Rust have no CollapseAmbiguities: their programs read the one _ambig node this grammar produces, and with nested ones you combine the readings yourself.

This needs Earley (LALR and Earley), which C does not support yet.

docs/examples/python/earley_ambiguity.py
"""Some inputs parse more than one way, and the Earley parser can keep every
reading. Adapted from Lark's "fruit flies like bananas" example.


  ambiguity="explicit"   the tree keeps an `_ambig` node wherever more than one
                         derivation exists, with each reading as a child
  CollapseAmbiguities    expands that tree into the list of plain trees it
                         stands for


Only Earley can report ambiguity. LALR parses this grammar too, committing to
one reading with no way to ask for the others.


Run:
    pip install hyperlark
    python earley_ambiguity.py
"""


import hyperlark
from hyperlark.visitors import CollapseAmbiguities


# "fruit flies like bananas" reads two ways: fruit flies (the insect) that like
# bananas, or fruit that flies the way bananas do. NOUN, VERB and ADJ overlap
# on purpose.
GRAMMAR = r"""
sentence: noun verb noun        -> simple
        | noun verb "like" noun -> comparative


noun: adj? NOUN
verb: VERB
adj: ADJ


NOUN: "flies" | "bananas" | "fruit"
VERB: "like" | "flies"
ADJ: "fruit"


%import common.WS
%ignore WS
"""


# Earley is the default engine, so no parser= is needed.
parser = hyperlark.Lark(GRAMMAR, start="sentence", ambiguity="explicit")


tree = parser.parse("fruit flies like bananas")


# The `_ambig` node at the root holds both readings side by side.
print("packed tree (both readings under one _ambig node):")
print(tree.pretty())


# CollapseAmbiguities expands the tree into one plain tree per reading, for a
# Transformer or any tree code that does not expect `_ambig`.
readings = CollapseAmbiguities().transform(tree)
print(f"{len(readings)} distinct readings:")
for i, reading in enumerate(readings, 1):
    # reading.data is the alias of the top alternative: 'simple' or 'comparative'
    print(f"  {i}. {reading.data}")


# The default, ambiguity="resolve", returns one tree. This grammar declares no
# priorities, so the alternative written first wins: swap the two `sentence`
# lines and the answer flips to `comparative`.
resolving = hyperlark.Lark(GRAMMAR, start="sentence")  # ambiguity="resolve"
chosen = resolving.parse("fruit flies like bananas")
assert chosen.data != "_ambig"
print(f"with the default resolver, the engine commits to: {chosen.data}")

Interactive parsing

Section titled “Interactive parsing”

Drive an LALR parse programmatically, one token at a time. At any point, ask what could legally come next?, or fork the instance, in order to explore several avenues.

Useful for debugging, error recovery, writing autocomplete, storing “savepoints” for IDEs, and more.

Each program completes partial queries, as an autocomplete would. This needs LALR and the built-in lexer. Python’s loop pauses before each token is fed; the others pause after (driving the built-in lexer). The interactive guide covers every operation per language.

docs/examples/python/interactive_autocomplete.py
"""The interactive parser: drive an LALR parse one token at a time and ask, at
any point, what could come next. That is what an autocomplete needs.


  parse_interactive(text) -> InteractiveParser
      .iter_parse()       yields each token from the built-in lexer before
                          feeding it; a generator, so you can inspect state,
                          retype the yielded token, and `break` to stop
      .accepts()          the terminal names valid in the current state
      .copy()             fork, to try a token without committing
      .exhaust_lexer()    feed every remaining token in one go
      .feed_eof()         finish with a synthesized $END -> parse tree
      .resume_parse()     drive whatever is left + $END -> parse tree


Interactive parsing needs LALR, so parser="lalr".


Run:
    pip install hyperlark
    python interactive_autocomplete.py
"""


import hyperlark
from hyperlark import Token


# A toy query language. The keywords become terminals named after themselves
# (SELECT, FROM, WHERE), so the accept-set reads like a list of suggestions.
GRAMMAR = r"""
start: "SELECT" columns "FROM" NAME where?
columns: "*" | NAME ("," NAME)*
where: "WHERE" NAME COMP value
value: NUMBER | ESCAPED_STRING
COMP: ">" | "<" | "="
NAME: /[a-z_]+/


%import common.NUMBER
%import common.ESCAPED_STRING
%import common.WS
%ignore WS
"""


parser = hyperlark.Lark(GRAMMAR, parser="lalr")


QUERY = "SELECT id, name FROM users WHERE id = 42"


# Labels for the terminal names the parser reports: keywords stand for
# themselves, open-ended terminals get a placeholder, anything else falls
# through to its raw name.
LABELS = {
    "SELECT": "SELECT", "FROM": "FROM", "WHERE": "WHERE",
    "STAR": "*", "COMMA": ",", "COMP": "> < =",
    "NAME": "<column/table>", "NUMBER": "<number>",
    "ESCAPED_STRING": '"<text>"', "$END": "(end of query)",
}




def labelled(names):
    return ", ".join(LABELS.get(n, n) for n in sorted(names))




# --- 1. Walk the parse with iter_parse --------------------------------------
# iter_parse() yields each token before feeding it, so accepts() in the loop
# body answers "what could stand at this token's spot"; the yielded token's
# own type is always among them.
print(f"walking {QUERY!r} token by token:\n")
ip = parser.parse_interactive(QUERY)
for tok in ip.iter_parse():
    print(f"  {tok.type:15} {str(tok.value):8} ← could be: {labelled(ip.accepts())}")


# iter_parse stops at end of input without feeding EOF; finish explicitly.
tree = ip.feed_eof()
print(f"\nfeed_eof() completed the parse: {tree.data}")


# --- 2. Stop early, then take over ------------------------------------------
# `break` leaves the parser suspended mid-input: inspect it, fork it, or drive
# it to the end. The yielded but unfed token (FROM here) stays queued, and the
# next driving call feeds it first.
ip = parser.parse_interactive(QUERY)
for tok in ip.iter_parse():
    if tok.type == "FROM":
        break  # pause the parse at the FROM keyword (yielded, not yet fed)
print(f"\npaused at FROM — the parser expects: {labelled(ip.accepts())}")


# copy() forks an independent parser: try a token on the fork to see whether it
# would be legal here, leaving the real parse untouched.
fork = ip.copy()
try:
    fork.feed_token(Token("COMMA", ","))
    comma_ok = True
except hyperlark.UnexpectedToken:
    comma_ok = False
print(f"would ',' be legal at this point? {comma_ok}")


# resume_parse feeds the rest of the input, starting with the queued FROM,
# then $END, and returns the tree. (In Lark the yielded token is lost and
# this resume fails.)
print("resume_parse() finished the rest:")
print(ip.resume_parse().pretty())




# --- 3. Completion for a half-typed query -----------------------------------
# Feed everything typed so far with exhaust_lexer(), then read the accept-set.
def suggest(prefix):
    ip = parser.parse_interactive(prefix)
    ip.exhaust_lexer()
    return labelled(ip.accepts())




print("completion after each prefix:")
for prefix in ["", "SELECT", "SELECT name", "SELECT name, email FROM users",
               "SELECT * FROM users WHERE age"]:
    print(f"  {prefix or '(empty)':32} → {suggest(prefix)}")

Error reporting

Section titled “Error reporting”

Detect what’s wrong in the input text by matching against example errors: given the parse function and a dictionary mapping each label to some malformed syntax examples, match_examples returns the label for the example that best matches the current error. The program classifies four malformed JSON documents.

match_examples is Python only, and LALR only, because it compares the parser state at the error. Rust errors carry that state too, so you can write the comparison yourself (custom error messages); TypeScript and C errors do not.

docs/examples/python/error_reporting.py
"""Example-driven error reporting: group a few broken inputs under each message
you want to show, and a parse failure gets the message of the example that
fails in the same parser state. Adapted from Lark's error-reporting recipe.


The call is UnexpectedInput.match_examples(): it re-parses each example and
compares its parser state with the real failure.


Run:
    pip install hyperlark
    python error_reporting.py
"""


import hyperlark
from hyperlark import UnexpectedInput


JSON_GRAMMAR = r"""
?start: value
?value: object
      | array
      | ESCAPED_STRING     -> string
      | SIGNED_NUMBER      -> number
      | "true"             -> true
      | "false"            -> false
      | "null"             -> null
array  : "[" [value ("," value)*] "]"
object : "{" [pair ("," pair)*] "}"
pair   : ESCAPED_STRING ":" value


%import common.ESCAPED_STRING
%import common.SIGNED_NUMBER
%import common.WS
%ignore WS
"""


json_parser = hyperlark.Lark(JSON_GRAMMAR, parser="lalr")




# Each subclass is a diagnosis. match_examples() returns one of these classes
# (or None); we raise it with the caret context filled in.
class JsonSyntaxError(SyntaxError):
    def __str__(self):
        context, line, column = self.args
        return f"{self.label} at line {line}, column {column}.\n\n{context}"




class JsonMissingOpening(JsonSyntaxError):
    label = "Missing opening bracket"




class JsonMissingClosing(JsonSyntaxError):
    label = "Missing closing bracket"




class JsonMissingComma(JsonSyntaxError):
    label = "Missing comma"




class JsonTrailingComma(JsonSyntaxError):
    label = "Trailing comma"




def parse(json_text):
    try:
        return json_parser.parse(json_text)
    except UnexpectedInput as u:
        # The keys are the diagnosis classes; the values are broken inputs that
        # fail the same way. match_examples re-parses each and picks the label
        # whose parser state matches the real failure.
        exc_class = u.match_examples(
            json_parser.parse,
            {
                JsonMissingOpening: ['{"foo": ]}', '{"foo": }'],
                JsonMissingClosing: ["{", '{"a": 1', "[1", '{"foo": [}'],
                # Several variants per label, because the parser state depends
                # on more than the mistake: '[1 2]' and '["a" 1]' fail in
                # different states (after SIGNED_NUMBER vs after ESCAPED_STRING).
                JsonMissingComma: ['[1 2]', '["a" 1]', '{"a":1 "b":2}'],
                # '[1,]' and '[1,2,]' fail in different states, so both are
                # needed: the second is what classifies '[1, 2, ]'.
                JsonTrailingComma: ["[,]", "[1,]", "[1,2,]", '{"foo":1,}'],
            },
            use_accepts=True,
        )
        if not exc_class:
            raise  # unrecognized shape: re-raise the raw UnexpectedInput
        raise exc_class(u.get_context(json_text), u.line, u.column) from None




CASES = [
    '{"example1": "value"',   # never closed
    '{"example2": ] ',        # value expected, got a bracket
    '[1 2]',                  # comma missing between elements
    '[1, 2, ]',               # trailing comma
]


for case in CASES:
    try:
        parse(case)
    except JsonSyntaxError as e:
        print(f"input:  {case!r}")
        print(e)
        print()