Skip to content
Hyperlark

Examples

Complete programs, not fragments. Each one is self-contained — grammar and input inline — so you can copy a tab and run it. Where a task exists in more than one target the programs share a grammar and a shape, so you can read the same program across languages and compare the APIs side by side.

Every filename below is a real file in the repository, rendered here from source.

TaskPythonTypeScriptRustC
Calculator — parse and evaluate✅✅✅✅
JSON — transform, query, stream✅✅✅—
Interpreter — top-down, evaluate on demand✅—✅—
Indentation — a postlexer✅———
Ambiguity — Earley, kept explicit✅✅✅—
Interactive — ask what comes next✅✅✅✅
Error reporting — friendly diagnostics✅———

A — almost always means nobody has written that program yet, not that the target can’t do it. The real capability gaps are C’s — no Earley, so no explicit ambiguity; no postlex; and no peer of JS’s subtree streaming. One more spans two targets: the match_examples error recipe needs the LALR state carried on the error, which the TypeScript and C surfaces do not expose (they give accepts/expected only), so that recipe is not portable to either. See the feature matrix for the authoritative answer.

Calculator

Section titled “Calculator”

Parse arithmetic and evaluate it to a number. The classic Lark calculator, running unchanged.

The Python and TypeScript tabs embed the transformer in the parser, so the value comes back during the parse and no tree is ever built. The Rust and C tabs parse first and fold the result with a reusable algebra, which is the shape you want when one parse feeds several folds; each also has a tree-free entry point for the other case — Rust’s parse_fold, C’s lark_parse_fold.

docs/examples/python/calc.py
"""Calculator quickstart: parse arithmetic and evaluate it DURING the parse
with an embedded Transformer. hyperlark is API-compatible with lark — this is
the classic lark calculator, running unchanged.


Run:  python calc.py   (any environment with the hyperlark wheel installed)
"""


import hyperlark
from hyperlark import Transformer, v_args


GRAMMAR = r"""
?start: sum
?sum: product
    | sum "+" product  -> add
    | sum "-" product  -> sub
?product: atom
    | product "*" atom -> mul
    | product "/" atom -> div
?atom: NUMBER          -> number
     | "-" atom        -> neg
     | "(" sum ")"


%import common.NUMBER
%import common.WS_INLINE
%ignore WS_INLINE
"""




@v_args(inline=True)  # children arrive as positional arguments
class Calc(Transformer):
    # The rules map straight onto the operator module's builtins — C-level
    # callables with no Python frame per reduction, so this is also the
    # fastest way to write it (it matters on big inputs, where the
    # transformer runs once per rule application).
    from operator import add, mul, neg, sub
    from operator import truediv as div


    number = float




# A plain parser first, to look at the tree: aliases (`-> add`) label the nodes.
parser = hyperlark.Lark(GRAMMAR, parser="lalr")
print(parser.parse("2 + 3 * 4").pretty())


# transformer= embeds Calc INTO the parse: the native engine folds every rule
# as it reduces, calling straight into those C-level builtins, and no Python
# tree is ever materialized — parse() returns the finished number. (See
# json_to_dict.py for the same transformer run both ways.)
calc = hyperlark.Lark(GRAMMAR, parser="lalr", transformer=Calc())
for expr in ["2 + 3 * 4", "(2 + 3) * 4", "-(10 / 4) + 0.5"]:
    print(f"{expr} = {calc.parse(expr)}")


# Parse errors carry position and context, like lark's.
try:
    calc.parse("1 + * 2")
except hyperlark.exceptions.UnexpectedInput as e:
    print(f"as expected, '1 + * 2' fails at line {e.line}, column {e.column}:")
    print(e.get_context("1 + * 2"))

JSON

Section titled “JSON”

Three things you do with a parsed document, one per target: turn it into native values, query it, and stream it. The Python program runs the transformer both ways — after the parse, and embedded in it — so you can see what the embedded form costs you and what it saves.

docs/examples/python/json_to_dict.py
"""Parse JSON into plain Python objects with a Transformer — the classic lark
tutorial, unchanged on hyperlark.


Shows both ways to run a transformer:
  1. post-parse:  T().transform(parser.parse(text))  — simple, tree in memory
  2. embedded:    Lark(..., transformer=T())         — folds DURING the parse;
     substantially faster on large inputs because the tree is never built


Run:  python json_to_dict.py
"""


import json


import hyperlark
from hyperlark import Transformer, v_args


GRAMMAR = r"""
?start: value
?value: object
      | array
      | string
      | SIGNED_NUMBER    -> number
      | "true"           -> true
      | "false"          -> false
      | "null"           -> null
array  : "[" [value ("," value)*] "]"
object : "{" [pair ("," pair)*] "}"
pair   : string ":" value
string : ESCAPED_STRING


%import common.ESCAPED_STRING
%import common.SIGNED_NUMBER
%import common.WS
%ignore WS
"""


DOC = """
{
  "name": "hyperlark",
  "engines": ["lalr", "earley"],
  "bindings": {"python": true, "js": true, "c": true},
  "empty": [],
  "holes": [null, 1],
  "version": 0.1
}
"""




class ToDict(Transformer):
    """Each method replaces the rule of the same name with a Python value;
    rules whose children are already final map straight onto builtins."""


    @v_args(inline=True)
    def string(self, s):
        # str() first: tokens are FastToken by default on EVERY path (tree and
        # embedded alike) — fast and sliceable, but not a str subclass, so
        # json.loads would reject one. Pass fast_tokens=False for str-subclass
        # tokens. json.loads then unquotes + unescapes.
        return json.loads(str(s))


    @v_args(inline=True)
    def number(self, n):
        return float(n)


    array = list
    pair = tuple
    object = dict


    def true(self, _):
        return True


    def false(self, _):
        return False


    def null(self, _):
        return None




# maybe_placeholders=False, as in lark's own JSON tutorial: without it, an
# empty `[...]` optional inserts a None child — indistinguishable from a real
# JSON null once the transformer has run.
OPTS = dict(parser="lalr", maybe_placeholders=False)


# 1. Post-parse: parse to a tree, then transform it.
tree_parser = hyperlark.Lark(GRAMMAR, **OPTS)
via_tree = ToDict().transform(tree_parser.parse(DOC))


# 2. Embedded: the transformer runs at the reduce seam, during the parse.
folding_parser = hyperlark.Lark(GRAMMAR, transformer=ToDict(), **OPTS)
via_fold = folding_parser.parse(DOC)


assert via_tree == via_fold == json.loads(DOC)
print(json.dumps(via_fold, indent=2))
print("matches json.loads:", via_fold == json.loads(DOC))

Interpreter

Section titled “Interpreter”

A transformer folds bottom-up, so every node is visited exactly once — which is the wrong shape for repeat 3 { … }, where a block must run several times, or for a conditional, where it must run zero. An interpreter walks top-down and lets each method decide what to visit and how often.

docs/examples/python/dsl_interpreter.py
"""A tiny language, run with an Interpreter — the top-down sibling of
Transformer. Adapted from lark's turtle DSL, but headless: instead of driving a
graphics window it computes the pen's path, so it runs anywhere.


Why an Interpreter and not a Transformer here? A Transformer folds bottom-up:
every node is visited exactly once, children before parents. So it cannot
directly produce a VALUE for `repeat 3 { ... }`, where the block must run
several times, or for a conditional, where it must run zero times — by the time
`repeat` is called its block has already been folded, once. (You can still do it
bottom-up by folding each block to a thunk and calling it N times; that is what
a compiler does. This is about which is direct, not which is possible.) An
Interpreter visits top-down and lets each method decide what to visit and how
often, so `repeat` just loops. (Compare rust/interpreter.rs.)


Run:  python dsl_interpreter.py
"""


import math


import hyperlark
from hyperlark.visitors import Interpreter


# f/b = forward/back, l/r = turn left/right (degrees). `repeat N { ... }` takes
# a nested block — that nesting is why we need top-down eval.
GRAMMAR = r"""
start: instruction+


?instruction: MOVE NUMBER              -> move
            | TURN NUMBER              -> turn
            | "repeat" NUMBER block    -> repeat


block: "{" instruction+ "}"


MOVE: "f" | "b"
TURN: "l" | "r"


%import common.NUMBER
%import common.WS
%ignore WS
"""


PROGRAM = """
repeat 4 {
    f 100
    r 90
}
f 50
"""




class Turtle(Interpreter):
    """Walks the tree top-down, keeping pen state as it goes. Each method is
    named after the node it handles — a rule, or the `-> alias` that renamed it.
    `repeat` re-visits its block, which is the whole point of an Interpreter.
    (`start` and `block` here just recurse, which is exactly what the inherited
    __default__ does; they are spelled out to keep the walk readable.)"""


    def __init__(self):
        self.x = self.y = 0.0
        self.heading = 0.0            # degrees; 0 = +x axis
        self.segments = []            # (x0, y0, x1, y1) for each 'f'/'b' move


    def start(self, tree):
        self.visit_children(tree)     # run each top-level instruction in order


    def move(self, tree):
        direction, dist = tree.children          # MOVE token, NUMBER token
        step = float(dist) * (1 if direction == "f" else -1)
        rad = math.radians(self.heading)
        x1 = self.x + step * math.cos(rad)
        y1 = self.y + step * math.sin(rad)
        self.segments.append((self.x, self.y, x1, y1))
        self.x, self.y = x1, y1


    def turn(self, tree):
        direction, degrees = tree.children
        self.heading += float(degrees) * (1 if direction == "l" else -1)


    def repeat(self, tree):
        count, block = tree.children             # NUMBER token, block subtree
        for _ in range(int(count)):
            self.visit(block)                    # re-visit the SAME subtree N times


    def block(self, tree):
        self.visit_children(tree)




parser = hyperlark.Lark(GRAMMAR, parser="lalr")
turtle = Turtle()
turtle.visit(parser.parse(PROGRAM))


# `+ 0.0` normalises the -0.0 that trig at right angles produces, so the closing
# corner reads (0.0, 0.0) rather than (-0.0, -0.0).
def _z(v, places=1):
    return round(v, places) + 0.0




print(f"{len(turtle.segments)} pen strokes:")
for x0, y0, x1, y1 in turtle.segments:
    print(f"  ({_z(x0):6.1f}, {_z(y0):6.1f}) -> ({_z(x1):6.1f}, {_z(y1):6.1f})")


xs = [c for seg in turtle.segments for c in (seg[0], seg[2])]
ys = [c for seg in turtle.segments for c in (seg[1], seg[3])]
print(f"bounding box: x [{_z(min(xs), 0):.0f}, {_z(max(xs), 0):.0f}]  "
      f"y [{_z(min(ys), 0):.0f}, {_z(max(ys), 0):.0f}]")

Indentation

Section titled “Indentation”

Languages like Python and YAML carry structure in their leading whitespace. The grammar declares _INDENT / _DEDENT; the bundled Indenter postlexer turns indentation into those tokens as the stream goes past.

docs/examples/python/indented_language.py
"""Parsing an indentation-based language (like Python or YAML) with the
Indenter postlexer: the grammar declares _INDENT/_DEDENT, the Indenter turns
leading whitespace into those tokens.


Run:  python indented_language.py
"""


import hyperlark
from hyperlark.indenter import Indenter


GRAMMAR = r"""
?start: _NL* tree
tree: NAME _NL [_INDENT tree+ _DEDENT]


NAME: /\w+/
_NL: /(\r?\n[\t ]*)+/


%declare _INDENT _DEDENT


// _NL owns the newline + leading indent; WS_INLINE covers spaces WITHIN a line
%import common.WS_INLINE
%ignore WS_INLINE
"""


DOC = """\
root
    branch
        leaf
        leaf2
    branch2
"""




class TreeIndenter(Indenter):
    NL_type = "_NL"
    OPEN_PAREN_types = []   # brackets that suspend indentation, e.g. ["LPAR"]
    CLOSE_PAREN_types = []
    INDENT_type = "_INDENT"
    DEDENT_type = "_DEDENT"
    tab_len = 8




parser = hyperlark.Lark(GRAMMAR, parser="lalr", postlex=TreeIndenter())
print(parser.parse(DOC).pretty())

Ambiguity

Section titled “Ambiguity”

Some grammars let one input parse more than one way. Earley can keep every reading instead of silently picking one: ambiguity="explicit" leaves an _ambig node wherever the grammar admitted more than one derivation, and CollapseAmbiguities expands that packed tree into the flat list of unambiguous trees it stands for.

docs/examples/python/earley_ambiguity.py
"""Ambiguity, made explicit: some grammars let one input parse more than one
way. The Earley engine can keep every reading instead of silently picking one.
Adapted from lark's classic "fruit flies like bananas" example.


  ambiguity="explicit"  — the parse tree keeps an `_ambig` node wherever the
                          grammar admitted more than one derivation, with each
                          reading as a child.
  CollapseAmbiguities   — expands that packed tree into the flat list of
                          unambiguous trees it stands for.


Only the Earley engine can report ambiguity. LALR still parses this grammar --
it just commits to one reading silently, with no way to ask for the others.


Run:  python earley_ambiguity.py
"""


import hyperlark
from hyperlark.visitors import CollapseAmbiguities


# "fruit flies like bananas" reads two ways: fruit-flies (a kind of fly) that
# like bananas, or fruit that flies the way bananas do. Both terminals overlap
# on purpose — NOUN, VERB and ADJ all admit some of the same words.
GRAMMAR = r"""
sentence: noun verb noun        -> simple
        | noun verb "like" noun -> comparative


noun: adj? NOUN
verb: VERB
adj: ADJ


NOUN: "flies" | "bananas" | "fruit"
VERB: "like" | "flies"
ADJ: "fruit"


%import common.WS
%ignore WS
"""


# ambiguity="explicit" only means something under Earley (the default engine),
# so there's no parser= to set here — it's already Earley.
parser = hyperlark.Lark(GRAMMAR, start="sentence", ambiguity="explicit")


tree = parser.parse("fruit flies like bananas")


# The `_ambig` node at the root holds both readings side by side.
print("packed tree (both readings under one _ambig node):")
print(tree.pretty())


# CollapseAmbiguities flattens the packed tree into the list of plain,
# unambiguous trees it represents — one per reading, ready for a Transformer
# or any ordinary tree code that doesn't expect `_ambig`.
readings = CollapseAmbiguities().transform(tree)
print(f"{len(readings)} distinct readings:")
for i, reading in enumerate(readings, 1):
    # reading.data is the alias of the top alternative — 'simple' vs 'comparative'
    print(f"  {i}. {reading.data}")


# Without ambiguity="explicit", Earley returns a single tree (the default,
# "resolve"). This grammar declares no priorities, so the tie is broken by the
# ORDER the alternatives are written — swap the two `sentence` lines and the
# answer below flips to `comparative`.
resolving = hyperlark.Lark(GRAMMAR, start="sentence")  # ambiguity="resolve"
chosen = resolving.parse("fruit flies like bananas")
assert chosen.data != "_ambig"
print(f"with the default resolver, the engine commits to: {chosen.data}")

Interactive parsing

Section titled “Interactive parsing”

Drive an LALR parse one token at a time and ask, at any point, what could legally come next? That accept-set is exactly what an autocomplete, a linter or a structured editor needs. Fork the parser with copy to try a token speculatively without committing to it.

docs/examples/python/interactive_autocomplete.py
"""The interactive parser: drive an LALR parse one token at a time and ask, at
any point, "what could legally come next?" That accept-set is exactly what an
autocomplete / linter / structured editor needs. This is lark's
`parse_interactive` API, method for method — with the divergences noted at the
bottom of this file.


  parse_interactive(text) -> InteractiveParser
      .iter_parse()       — pull tokens from the built-in lexer, YIELDING each
                            one before feeding it (lark's ordering); it's a
                            generator, so you can inspect state — or retype
                            the yielded token — and `break` to stop
      .accepts()          — the terminal names valid in the current state
      .copy()             — fork, to try a token speculatively without committing
      .exhaust_lexer()    — feed every remaining token in one go
      .feed_eof()         — finish with a synthesized $END -> parse tree
      .resume_parse()     — drive whatever is left + $END -> parse tree


Interactive parsing is an LALR feature, so parser="lalr".


Run:  python interactive_autocomplete.py
"""


import hyperlark
from hyperlark import Token


# A toy query language. The keywords become terminals named after themselves
# (SELECT, FROM, WHERE), so the accept-set reads like a list of suggestions.
GRAMMAR = r"""
start: "SELECT" columns "FROM" NAME where?
columns: "*" | NAME ("," NAME)*
where: "WHERE" NAME COMP value
value: NUMBER | ESCAPED_STRING
COMP: ">" | "<" | "="
NAME: /[a-z_]+/


%import common.NUMBER
%import common.ESCAPED_STRING
%import common.WS
%ignore WS
"""


parser = hyperlark.Lark(GRAMMAR, parser="lalr")


QUERY = "SELECT id, name FROM users WHERE id = 42"


# Friendly labels for the terminal names the parser reports: keywords stand for
# themselves, and the open-ended terminals get a placeholder an editor could
# show. Anything absent falls through to its raw terminal name.
LABELS = {
    "SELECT": "SELECT", "FROM": "FROM", "WHERE": "WHERE",
    "STAR": "*", "COMMA": ",", "COMP": "> < =",
    "NAME": "<column/table>", "NUMBER": "<number>",
    "ESCAPED_STRING": '"<text>"', "$END": "(end of query)",
}




def labelled(names):
    return ", ".join(LABELS.get(n, n) for n in sorted(names))




# --- 1. Walk the parse with iter_parse --------------------------------------
# iter_parse() yields each token BEFORE feeding it (lark's ordering — that's
# what lets you retype a yielded token to steer the parse). So the loop body
# runs with the parser still in the state the token was lexed in: accepts()
# here answers "what could legally stand at this token's spot" — the yielded
# token's own type is always among the suggestions.
print(f"walking {QUERY!r} token by token:\n")
ip = parser.parse_interactive(QUERY)
for tok in ip.iter_parse():
    print(f"  {tok.type:15} {str(tok.value):8} ← could be: {labelled(ip.accepts())}")


# iter_parse stops at end of input without feeding EOF — finish explicitly.
tree = ip.feed_eof()
print(f"\nfeed_eof() completed the parse: {tree.data}")


# --- 2. Stop early, then take over ------------------------------------------
# Because iter_parse is a generator, `break` leaves the parser suspended
# mid-input, fully usable: inspect it, fork it, or drive it to the end. The
# yielded-but-unfed token (FROM here) stays QUEUED, so nothing is lost — the
# next driving call feeds it first.
ip = parser.parse_interactive(QUERY)
for tok in ip.iter_parse():
    if tok.type == "FROM":
        break  # pause the parse at the FROM keyword (yielded, not yet fed)
print(f"\npaused at FROM — the parser expects: {labelled(ip.accepts())}")


# copy() forks an independent parser: try a token on the fork to see whether it
# would be legal here, leaving the real parse untouched.
fork = ip.copy()
try:
    fork.feed_token(Token("COMMA", ","))
    comma_ok = True
except hyperlark.UnexpectedToken:
    comma_ok = False
print(f"would ',' be legal at this point? {comma_ok}")


# The suspended parser still has the rest of the input — the queued FROM
# included; resume_parse drives it plus $END to a finished tree. (In lark the
# abandoned token is stranded and this resume fails; keeping it queued is a
# hyperlark divergence.)
print("resume_parse() finished the rest:")
print(ip.resume_parse().pretty())




# --- 3. Completion for a half-typed query -----------------------------------
# Same idea, packaged: feed everything typed so far, then read the accept-set.
# exhaust_lexer() is iter_parse()'s drive-it-all sibling, for when you don't
# need each token on the way through.
def suggest(prefix):
    ip = parser.parse_interactive(prefix)
    ip.exhaust_lexer()
    return labelled(ip.accepts())




print("completion after each prefix:")
for prefix in ["", "SELECT", "SELECT name", "SELECT name, email FROM users",
               "SELECT * FROM users WHERE age"]:
    print(f"  {prefix or '(empty)':32} → {suggest(prefix)}")

Error reporting

Section titled “Error reporting”

Turn a raw parse failure into a specific, friendly message by matching the parser’s state against a handful of labelled broken inputs. You describe the mistakes by example; match_examples() re-parses each one and compares parser state with the real failure — no hand-written state tables.

docs/examples/python/error_reporting.py
"""Example-driven error reporting: turn a raw parse failure into a friendly,
specific message by matching the parser's state against a handful of labelled
broken inputs — lark's example-driven error-reporting recipe (with the example
sets retuned to this grammar's states; see the note by the examples below).


The trick is UnexpectedInput.match_examples(): you hand it your own malformed
snippets grouped by the message you'd like to show, and it re-parses each one
and compares parser state with the real failure. No hand-written state tables —
you describe the mistakes by example.


Run:  python error_reporting.py
"""


import hyperlark
from hyperlark import UnexpectedInput


JSON_GRAMMAR = r"""
?start: value
?value: object
      | array
      | ESCAPED_STRING     -> string
      | SIGNED_NUMBER      -> number
      | "true"             -> true
      | "false"            -> false
      | "null"             -> null
array  : "[" [value ("," value)*] "]"
object : "{" [pair ("," pair)*] "}"
pair   : ESCAPED_STRING ":" value


%import common.ESCAPED_STRING
%import common.SIGNED_NUMBER
%import common.WS
%ignore WS
"""


json_parser = hyperlark.Lark(JSON_GRAMMAR, parser="lalr")




# Each subclass is a diagnosis. match_examples() returns one of these classes
# (or None); we raise it with the caret context filled in.
class JsonSyntaxError(SyntaxError):
    def __str__(self):
        context, line, column = self.args
        return f"{self.label} at line {line}, column {column}.\n\n{context}"




class JsonMissingOpening(JsonSyntaxError):
    label = "Missing opening bracket"




class JsonMissingClosing(JsonSyntaxError):
    label = "Missing closing bracket"




class JsonMissingComma(JsonSyntaxError):
    label = "Missing comma"




class JsonTrailingComma(JsonSyntaxError):
    label = "Trailing comma"




def parse(json_text):
    try:
        return json_parser.parse(json_text)
    except UnexpectedInput as u:
        # The keys are the diagnosis classes; the values are broken inputs that
        # fail the same way. match_examples re-parses each and picks the label
        # whose parser state matches the real failure.
        exc_class = u.match_examples(
            json_parser.parse,
            {
                JsonMissingOpening: ['{"foo": ]}', '{"foo": }'],
                JsonMissingClosing: ["{", '{"a": 1', "[1", '{"foo": [}'],
                # Several variants per label, because the parser state depends
                # on more than the mistake. Here it is the terminal preceding
                # the junk token: '[1 2]' and '["a" 1]' fail in DIFFERENT states,
                # since the value reduction is deferred and the error lands in
                # the state after SIGNED_NUMBER vs after ESCAPED_STRING.
                JsonMissingComma: ['[1 2]', '["a" 1]', '{"a":1 "b":2}'],
                # And here it is the element count: '[1,]' and '[1,2,]' are
                # distinct states, so the second is what classifies '[1, 2, ]'.
                JsonTrailingComma: ["[,]", "[1,]", "[1,2,]", '{"foo":1,}'],
            },
            use_accepts=True,
        )
        if not exc_class:
            raise  # unrecognized shape — re-raise the raw UnexpectedInput
        raise exc_class(u.get_context(json_text), u.line, u.column) from None




CASES = [
    '{"example1": "value"',   # never closed
    '{"example2": ] ',        # value expected, got a bracket
    '[1 2]',                  # comma missing between elements
    '[1, 2, ]',               # trailing comma
]


for case in CASES:
    try:
        parse(case)
    except JsonSyntaxError as e:
        print(f"input:  {case!r}")
        print(e)
        print()

Where to go next

Section titled “Where to go next”