Skip to content
Hyperlark

How it works

hyperlark is a parsing toolkit: you describe a language as a grammar, and hyperlark turns text in that language into a structured tree you can walk or evaluate. It works the same way everywhere — one Rust engine behind four languages — and needs no build step and no code generation. This page is the mental model; the guides and reference fill in the details.

The pipeline

Section titled “The pipeline”

Every parse follows the same four steps, from the grammar you write to the value you get back:

   ┌───────────────────┐
   │   .lark grammar   │  what you write
   └─────────┬─────────┘
             │
             ▼
   ┌───────────────────┐
   │   Rust compiler   │  builds parse tables at runtime
   └─────────┬─────────┘  (no code generation, no build step)
             │  ParseTable + terminals
             ▼
   ┌───────────────────┐
   │       Parse       │  lex → LALR(1) or Earley
   └─────────┬─────────┘  (lexer chosen automatically)
             │  parse tree
             ▼
   ┌───────────────────┐
   │      Consume      │  walk the tree, or fold during the parse
   └───────────────────┘

  1. Write a grammar. You describe your language in the .lark grammar language — lowercase rules that become tree nodes, UPPERCASE terminals that become tokens, plus directives like %ignore and %import. See the grammar reference.

  2. Compile it — at runtime. hyperlark’s Rust compiler reads the grammar and builds the parse tables in memory the moment you construct a parser: the LALR automaton or Earley item sets, plus the terminal and lexer tables. Nothing is generated to disk and there is no separate build step.

  3. Parse your input. A lexer splits the text into tokens, and the parser — LALR(1) or Earley — drives those tokens through the tables to recognize the structure, producing a parse tree.

  4. Consume the result. Walk the finished tree with a transformer or visitor, or fold a value during the parse with an embedded transformer. See transformers and visitors.

One engine, every language

Section titled “One engine, every language”

There is a single implementation. The core is a pure-Rust crate (hyperlark) that owns the grammar compiler, both parsers, and every lexer. The Python, TypeScript/WASM, and C libraries are thin bindings over that same crate — Python through PyO3, TypeScript/WASM through wasm-bindgen, and C through a small C ABI.

A grammar therefore compiles to the same tables and parses to the same tree in every language. What differs between targets is how results are materialized and which peripheral features are wired up — not the grammar semantics. Start in Python, Rust, TypeScript / WASM, or C.

LALR and Earley

Section titled “LALR and Earley”

hyperlark ships two parsers behind the same surface:

  • LALR(1) is fast and deterministic. It expects an unambiguous grammar and produces exactly one tree, rejecting grammars whose conflicts it cannot resolve. It is the default, and the right choice for most languages.
  • Earley is more general: it parses grammars LALR cannot, including ambiguous ones. When more than one parse is valid, the ambiguity option chooses between resolve (return the single best tree) and explicit (keep the alternatives in the tree as ambiguity nodes).

Start with LALR(1); reach for Earley when your grammar is ambiguous or too general for LALR.

Lexers, chosen for you

Section titled “Lexers, chosen for you”

You rarely pick a lexer by hand: lexer="auto" (the default) resolves to contextual under LALR and dynamic under Earley, with basic available under either. The lexers guide covers overriding that choice, plus indentation handling (postlex) and custom token streams.

Runtime tables, not code generation

Section titled “Runtime tables, not code generation”

Because the tables are built at runtime, there is no code-generation step and no generated parser to check in or keep in sync. A grammar is just data: you can load, edit, swap, or generate one on the fly, and (on LALR) save the compiled form and reload it as a cache. The trade is a constant-factor runtime cost next to parsers that emit specialized code ahead of time — hyperlark keeps that factor small by doing the work in native Rust.

Correctness

Section titled “Correctness”

hyperlark is validated by differential testing: across a large corpus of grammars and inputs, the tokens and trees it produces are compared against the output of Lark, pinned to an exact version. Lark is the answer key — the reference hyperlark is checked against — never a runtime dependency of the toolkit. (The Python suite goes further and runs Lark’s own test suite against hyperlark.)

Where to go next

Section titled “Where to go next”