Interactive parsing

The interactive parser gives you advanced control over parsing and error handling when parsing with LALR: feed it tokens one at a time, ask which terminals fit next, and fork it to try a continuation. Use it for autocomplete, REPLs and editors. Python, TypeScript and Rust also have a simpler interface to error handling: on_error, in the errors guide.

LALR and the built-in lexer only

Only works with parser="lalr" ({ parser: "lalr" } in TypeScript, Lark::lalr in Rust); Python, TypeScript and Rust default to Earley, and C is LALR only, for now. A parser with a custom lexer cannot be driven interactively. Per-target table: feature matrix.

Example

The tabs use this grammar as GRAMMAR:

start: NUMBER (OP NUMBER)*
OP: "+" | "-" | "*" | "/"
NUMBER: /[0-9]+/
%ignore " "

Python

import hyperlark as lark
p = lark.Lark(GRAMMAR, parser="lalr")      # interactive parsing needs LALR

ip = p.parse_interactive()                 # no text: tokens are fed by hand
print(ip.accepts())                        # {'NUMBER'}

toks = list(p.lex("1 + 2"))                # anything with .type and .value can be fed
ip.feed_token(toks[0])                     # feed "1"
print(ip.accepts())                        # {'OP', '$END'}: an operator, or the end of the input

branch = ip.copy()                         # feed this to test a token without moving ip
for tok in toks[1:]:
    ip.feed_token(tok)                     # feed "+ 2"
tree = ip.feed_eof()                       # append $END, get the tree
print(tree.pretty())

Rust

use hyperlark::{Lark, OwnedInteractiveParser, Token};

let p = Lark::lalr(GRAMMAR)?;    // interactive parsing needs LALR

// Empty text: tokens are fed by hand. Pass the input instead to drive the lexer.
let mut ip = OwnedInteractiveParser::new(&p, String::new(), None)?;
assert!(ip.accepts().iter().any(|t| t == "NUMBER"));   // only a number fits first

// Look up the terminal id by name and build a Token from it.
let table = ip.parse_table();
ip.feed_token(Token::synthetic(table.token_id("NUMBER").unwrap(), "1"))?;
assert!(ip.accepts().iter().any(|t| t == "$END"));     // a lone number is a complete input

let _branch = ip.copy();                   // feed this to test a token without moving ip
let _tree = ip.feed_eof()?;                // append $END, get a ParsedTree

TypeScript

import { Lark } from "@hyperlark/hyperlark";
const p = new Lark(GRAMMAR, { parser: "lalr" });   // interactive parsing needs LALR

const ip = p.parseInteractive();           // no text: tokens are fed by hand
console.log(ip.accepts());                 // ["NUMBER"]

const toks = p.lex("1 + 2");               // Token objects to feed
ip.feedToken(toks[0]);                     // feed "1"
console.log(ip.accepts());                 // ["$END", "OP"]: the end of the input, or an operator (sorted)

const branch = ip.copy();                  // feed this to test a token without moving ip
for (const t of toks.slice(1)) ip.feedToken(t);
const handle = ip.feedEof();               // append $END, get a ParseHandle
console.log(handle.toJs(handle.rootPos())); // the plain { data, children } tree
handle.free(); branch.free(); ip.free();   // WASM memory; or declare them with `using`

C

// Omitted: #include <stdio.h>, <string.h>, "hyperlark.h", and GRAMMAR as a C string.
// Statuses are not checked here; a real program checks each one.
Lark *p = NULL;
lark_from_source(GRAMMAR, strlen(GRAMMAR), LARK_OPT_NONE, &p);  // the C API is LALR-only, for now

LarkInteractive *ip = NULL;
lark_parse_interactive(p, "", 0, NULL, &ip);   // empty text: tokens are fed by hand

// The accepted terminals, sorted; "$END" appears only when the input may stop here.
for (size_t i = 0, n = lark_interactive_accepts_count(ip); i < n; i++)
    printf("accepts: %s\n", lark_interactive_accepts_get(ip, i));

lark_interactive_feed_token(ip, "NUMBER", "1", 1);   // feed one token by terminal name

LarkInteractive *branch = NULL;
lark_interactive_copy(ip, &branch);            // feed this to test a token without moving ip

LarkParseResult *tree = NULL;
if (lark_interactive_feed_eof(ip, &tree) == LARK_OK)   // append $END; tree stays NULL otherwise
    fputs(lark_result_pretty(tree), stdout);

// A second parser over real text: each feed_next lexes one token and feeds it.
LarkInteractive *sp = NULL;
lark_parse_interactive(p, "1+2", 3, NULL, &sp);
LarkTokenEvent ev;
bool has_tok;
while (lark_interactive_feed_next(sp, &ev, &has_tok) == LARK_OK && has_tok)
    printf("token id %u = %.*s\n", ev.type_id, (int)ev.value_len, ev.value);

lark_result_free(tree);
lark_interactive_free(sp);
lark_interactive_free(branch);
lark_interactive_free(ip);
lark_free(p);

A complete autocomplete program per language: the Examples page.

Operations

parse_interactive starts a session. Given text, the parser lexes it for you (Driving the built-in lexer); given none, you feed each token with feed_token. The start rule defaults to the grammar’s only one.

feed_token advances the parser by one token, as if it came from the lexer. A token that cannot follow what was fed raises UnexpectedToken, as from a failed parse (errors guide).

accepts() returns the set of tokens that will advance the parser into a new valid state. $END, the end-of-input token, is among them when the input may end here. choices() returns the raw parse-table row instead, {name: (kind, id)}. It also lists rule names, and terminals the parser would still reject, so test a token against accepts().

copy creates a new parser with a separate state, so feeding the fork tries a token or a continuation without moving the original.

feed_eof feeds $END and returns the tree; whatever has not been fed is discarded. resume_parse resumes automated parsing, lexing and feeding the rest of the text. Both finish the parser; fork it first to continue from the same state. Tokens you pass to feed_token do not advance the parser’s own lexer, so after feeding tokens by hand resume_parse would lex the text they stood for again: finish with feed_eof instead.

Operation	Python	Rust	TypeScript	C
Create	parse_interactive(text=None, start=None)	OwnedInteractiveParser::new	parseInteractive(text?, start?)	lark_parse_interactive
Feed one token	feed_token	feed_token	feedToken	lark_interactive_feed_token
What fits next	accepts / choices	accepts / choices	accepts / choices	lark_interactive_accepts_count / _get
Lex and feed the next token	feed_next / iter_parse	feed_next	feedNext / iterParse	lark_interactive_feed_next
Lex and feed the rest	exhaust_lexer	exhaust_lexer	exhaustLexer	(loop feed_next)
Peek before feeding	peek_next	peek_next / retype_peeked	(none)	(none)
Finish	feed_eof / resume_parse	feed_eof / resume_parse	feedEof / resumeParse	lark_interactive_feed_eof / lark_interactive_resume_parse
Fork	copy	copy	copy	lark_interactive_copy
Release	(GC)	(drop)	free() or using	lark_interactive_free

Results and cleanup

• Python. feed_eof and resume_parse return a Tree, or with transformer= whatever the start rule’s callback produced.
• Rust. They return a ParsedTree; to walk it with the cursor API (trees guide), pair it with the parse table: ParseResult { tree: Arc::new(tree), table: ip.parse_table() }.
• TypeScript. feedEof() and resumeParse() return a ParseHandle. InteractiveParser and ParseHandle hold WASM memory: free() them when done, or declare them with using (see Large trees in TypeScript).
• C. Check every LarkStatus; on failure feed_eof and resume_parse leave the result NULL. Free each parser handle, forks included, with lark_interactive_free, and the result with lark_result_free.

Driving the built-in lexer

When the parser was given text, feed_next reads one token from the lexer, feeds it and returns it; exhaust_lexer feeds the rest, without $END. Python and TypeScript also have iter_parse / iterParse, which steps through the parse one token per iteration and yields each. Python’s yields each token before feeding it, so accepts() inside the loop reflects the parse before that token; TypeScript’s yields each token after feeding it.

A token feed_next rejects is already consumed, so a loop that ignores the error parses on without it and accepts input the grammar rejects. An unmatched character is not consumed, so the next feed_next reports it again.

Python

ip = p.parse_interactive("1 + 2")   # the parser lexes this text
ip.exhaust_lexer()                  # feed every token
print(ip.accepts())                 # {'OP', '$END'}
tree = ip.feed_eof()

Rust

let mut ip = OwnedInteractiveParser::new(&p, "1 + 2".to_string(), None)?;
ip.exhaust_lexer()?;                                  // feed every token
assert!(ip.accepts().iter().any(|t| t == "$END"));    // ["$END", "OP"]
let tree = ip.feed_eof()?;

TypeScript

const ip = p.parseInteractive("1 + 2");   // the parser lexes this text
ip.exhaustLexer();                        // feed every token
console.log(ip.accepts());                // ["$END", "OP"]
const handle = ip.feedEof();

C

LarkInteractive *sp = NULL;
lark_parse_interactive(p, "1+2", 3, NULL, &sp);   // the parser lexes this text
LarkTokenEvent ev;
bool has_tok;
while (lark_interactive_feed_next(sp, &ev, &has_tok) == LARK_OK && has_tok)
    printf("token id %u = %.*s\n", ev.type_id, (int)ev.value_len, ev.value);

Feeding builds the tree as it goes, even if you only want the tokens; for a value without a tree, use lark_parse_fold (transformers guide).

Retyping a token

When the lexer cannot tell two terminals apart, set the token’s type yourself before it is fed.

Python’s iter_parse yields each token before feeding it, so set tok.type there. With a grammar that accepts NAME or KW:

for tok in ip.iter_parse():
    if tok.type == "NAME" and tok.value == "select":
        tok.type = "KW"             # fed as KW

Outside a loop, Python’s peek_next returns the next token without feeding it; change it and pass it to feed_next(tok). In Rust, retype_peeked(type_id, value) rewrites the peeked token; the next feed_next, exhaust_lexer or resume_parse feeds it, feed_eof discards it, and feed_token inserts a token ahead of it.

TypeScript’s iterParse yields each token after feeding it, too late to retype it. Instead, lex with lex() and feed each token with feedToken({ type, value }), retyping as needed. The parser’s own lexer types some tokens by context (contextual lexer) and lex() cannot, so retype those too. C has no peek call or lex(): lex the text yourself and feed each token by name with lark_interactive_feed_token.

Limits

• TypeScript: parseInteractive ignores the constructor’s transformer; feedEof() and resumeParse() return the untransformed tree.
• Python: copy() does not carry a postlex into the fork, so the fork parses the raw tokens and can return a wrong tree or error, with no warning. An Indenter subclass forks correctly unless it overrides process, _process, handle_NL or always_accept.
• Rust: copy() drops a custom postlex whose stream does not implement clone_stream; the fork continues without it. The built-in Indenter forks correctly.
• Python, TypeScript and C: a token fed with feed_token carries no line or column into the tree or errors, even one from lex(); tokens the parser lexes itself keep theirs. In Rust a fed Token keeps its positions; Token::synthetic has none.
• Python: calling an interactive parser from one of its own callbacks (lexer callback, postlex or transformer= method), or from another thread while a call is running, raises RuntimeError.

Error recovery

In Python and Rust, an on_error handler can insert a token before resuming, through e.interactive_parser.feed_token(..) (except with a custom lexer) and cx.interactive_parser().feed_token(..) (errors guide). TypeScript handlers can only skip or stop, and C has no on_error, so there you drive the parser yourself. Test a token against accepts() or on a copy() first: a rejected feed_token is not undone, so a replacement that would have fit before it can be refused after it.