Grammar reference

A grammar is a list of rules and terminals, that together define a language: terminals define its alphabet, rules its structure. The same .lark grammar works from Python, Rust, TypeScript and C. This page covers the syntax you will use most; Lark’s grammar reference has the rest. All of it works in Hyperlark, with these differences: C parses with LALR only, for now; a few regular-expression constructs are refused (see Terminals); and what %import can reach depends on the target. The feature matrix and Divergences have the remaining limits.

Example

start: NUMBER (OP NUMBER)*
OP: "+" | "-" | "*" | "/"
NUMBER: /[0-9]+/
%ignore " "

Parsing 1 + 2 * 3 with it gives:

start
  1
  +
  2
  *
  3

Here start is a rule, OP and NUMBER terminals, %ignore a directive. Definitions and directives each start on their own line; comments start with // or #. Parsing begins at the rule start, unless the start option says otherwise. Rule names are lowercase, terminal names uppercase.

Loading the grammar

With the example grammar in a string GRAMMAR, each of these builds an LALR parser and prints the tree above. Python, Rust and TypeScript default to Earley; LALR and Earley covers the choice.

import hyperlark as lark
p = lark.Lark(GRAMMAR, parser="lalr")
print(p.parse("1 + 2 * 3").pretty())

use hyperlark::Lark;
let p = Lark::lalr(GRAMMAR)?;
println!("{}", p.parse("1 + 2 * 3")?.pretty());

import { Lark, pretty } from "@hyperlark/hyperlark";
const p = new Lark(GRAMMAR, { parser: "lalr" });
console.log(pretty(p.parse("1 + 2 * 3")));

#include <stdio.h>
#include <string.h>
#include "hyperlark.h"

Lark *p = NULL;
if (lark_from_source(GRAMMAR, strlen(GRAMMAR), LARK_OPT_NONE, &p) != LARK_OK) {
    fprintf(stderr, "%s\n", lark_last_error());
    return 1;
}
LarkParseResult *r = NULL;
if (lark_parse(p, "1 + 2 * 3", 9, NULL, &r) == LARK_OK) {   /* 9 bytes of text; NULL = the start rule */
    printf("%s\n", lark_result_pretty(r));
    lark_result_free(r);
}
lark_free(p);

Terminals

Terminals are used to match text into symbols. They can be defined as a combination of literals and other terminals:

Literal	Matches
`"if"`	the string
`"select"i`	the string, ignoring case
`/[0-9]+/`	a regular expression; flags `i`, `m`, `s`, `u`, `x` go after the slash: `/.+/s`
`"a".."z"`	one character in the range

Terminals also support grammar operators, such as |, +, * and ?. From the built-in common.lark:

DIGIT: "0".."9"
INT: DIGIT+
DECIMAL: INT "." INT? | "." INT

A terminal may not contain itself.

Terminals can be assigned a priority to influence lexing, as in BOOL.2: /true|false/; the lexers guide says when to set one.

Regular expressions use Python’s re syntax and are compiled by Rust regex engines. A construct that means something different in the two is refused when the parser is built. Refused constructs include:

$ outside a character class, without the m flag. Write \z for the end of the text, or add the m flag for the end of a line.
\Z. Write \z.
An octal escape such as \101. Write \x41.
A [ inside a character class. Write \[.
With the x flag, whitespace or a # inside a character class, and non-ASCII whitespace anywhere in the pattern. Escape them with \.

Python and Rust have an experimental translate_python_regex option, off by default, that makes these behave as they do in Python. \N{...} and conditionals such as (?(1)a|b) are refused either way.

Rules

Each matched rule becomes a branch (node) in the tree, with its matches as children, in order. Alternatives are separated by |, and may each start a new line.

Item	Meaning
`rule`, `TERMINAL`	a rule or a terminal
`"string literal"`, `/regexp literal/`	an unnamed terminal
`(item item ..)`	group items
`[item item ..]`	maybe: `(item item ..)?`, but with a placeholder when absent
`item?`	zero or one instances of item
`item*`, `item+`	zero or more, one or more instances of item
`item ~ n`, `item ~ n..m`	exactly n instances; n to m instances
`-> alias`	ends an alternative; names its branch `alias` instead of the rule

All operators, including ~ n, apply directly to the item on their left. Left-recursion is allowed and encouraged: sum: sum "+" product.

Shaping the tree

Terminals are always values in the tree, never branches. Some are filtered out by default, as punctuation: unnamed literals ("keyword", "+") and terminals whose name starts with an underscore (_DIGIT). Unnamed regular expressions (/[0-9]/) and named terminals (DIGIT) stay; keep_all_tokens keeps them all.

A prefix on a rule’s name changes its node:

_name: inlined into its containing rule.
?name: inlined if it has a single child, after filtering; an aliased alternative always keeps its node.
!name: keeps all its terminals; usually a code smell.

This grammar uses ? and aliases to get one node per operation:

start: sum
?sum: product
    | sum "+" product   -> add
?product: NUMBER
        | product "*" NUMBER -> mul
NUMBER: /[0-9]+/
%ignore " "

Parsing 1 + 2 * 3 gives:

start
  add
    1
    mul
      2
      3

Directives

%ignore WS: all occurrences of the terminal are ignored, and won’t be part of the parse. An inline pattern works too: %ignore " ".
%import common.NUMBER brings in a definition from the built-in library, available on every target. common.lark defines INT, NUMBER, ESCAPED_STRING, CNAME, WS and more.
%import .mygrammar (expr, NUMBER) imports from your own .lark file, on every target but C. How the file is found is below. %ignore directives are not imported.
%declare _INDENT _DEDENT declares terminals without defining them, for a postlex or a custom lexer to produce.
%override replaces a rule or terminal everywhere it is referenced, even in imported grammars; %extend adds an alternative to one.

Where an import of your own file is read from:

Python: the grammar file’s directory, with Lark.open or source_path=, and the directories in import_paths.
Rust: the GrammarLoaders in LarkOptions::import_paths; none ships that reads the filesystem, so you write one. Its read returns the resolved path and contents, or None to pass to the next loader.
TypeScript: the import_loader option. The package provides fsLoader(rootDir) (Node only) and mapLoader(sources).