Skip to content
Hyperlark

Parse errors

A failed parse tells you where it stopped and which terminals were expected there. A mistake in the grammar itself is reported earlier, when the parser is built: GrammarError in Python and TypeScript, LarkBuildError in Rust, LARK_CONSTRUCTION_ERROR in C. In Python a terminal pattern that does not compile can raise LexError instead; catching LarkError covers both.

Error classes

Section titled “Error classes”
  • UnexpectedCharacters: the lexer cannot match the next characters to any terminal, such as a stray @ or an unterminated string. Earley’s default dynamic lexer reports a misplaced token this way as well.
  • UnexpectedToken: the parser received a token that doesn’t match any valid step forward, such as an operator where a number was expected. Under LALR, input that ends too early is reported this way too, with the end-of-input token $END.
  • UnexpectedEOF: the parser expected a token, but the input ended; Earley only. It carries the expected set but no position: line and column are -1 in Python and TypeScript, None in Rust.

By language:

  • Python and TypeScript: all three extend UnexpectedInput, which has helpers for a nicer message: get_context (getContext in TypeScript) and, in Python, match_examples.
  • Rust: they are variants of ParseError, which has others (a callback failure, a configuration mistake), so a match needs a wildcard arm.
  • C: the parse call returns the status code LARK_UNEXPECTED_TOKEN or LARK_UNEXPECTED_CHARACTERS, and lark_last_error* holds the details.
PythonRustTypeScriptC
Tokentokentoken_type, token_valuetokentype, in the message text
Charactercharchcharin the message text
Line and column, 1-basedline, columnline(), column()line, columnlark_last_error_position
Offset, in code pointspos_in_streampos_in_stream()posnone
Expected setexpected (allowed on UnexpectedCharacters)expected (allowed)accepts (allowed)lark_last_error_expected
Parser state, LALR onlystatestatenonenone
Caret viewget_context(text, span=40)get_context(text)getContext(text, span?)lark_last_error_context()

Python’s UnexpectedToken also has accepts, the set of tokens that will advance the parser into a new valid state. It is None under Earley. With the basic lexer, allowed on UnexpectedCharacters is not narrowed to the terminals that fit there.

Catching a parse error

Section titled “Catching a parse error”

The samples parse 1 + + 2 under LALR with this grammar as GRAMMAR; the second + is an UnexpectedToken.

start: NUMBER (OP NUMBER)*
OP: "+" | "-" | "*" | "/"
NUMBER: /[0-9]+/
%ignore " "
import hyperlark as lark


parser = lark.Lark(GRAMMAR, parser="lalr")
text = "1 + + 2"


try:
    parser.parse(text)
except lark.exceptions.UnexpectedInput as e:   # base of all three
    print(e.line, e.column)          # 1 5
    print(e.get_context(text))       # caret view
    if isinstance(e, lark.exceptions.UnexpectedToken):
        print(repr(e.token), e.expected)   # Token('OP', '+') {'NUMBER'}
    elif isinstance(e, lark.exceptions.UnexpectedCharacters):
        print(repr(e.char), e.allowed)

The caret view each tab prints:

1 + + 2
    ^

get_context pinpoints the error in the text, with span characters of context around it, 40 by default and always 40 in Rust. The parser holds no copy of the text, so you pass it again. C renders the view at parse time, so lark_last_error_context takes no arguments.

Recovering from errors

Section titled “Recovering from errors”

To parse past a bad token, pass an on_error function to parse (parse_opts in Rust). It is called on each UnexpectedInput error, with the error as its argument (in Rust, a context holding it); return true to resume parsing, or false to raise it. Resuming skips the rejected token, or one unmatched character, so three bad characters mean three calls. The result is the tree without the skipped parts. A rejected end of input ($END) cannot be skipped; returning true there still fails.

on_error is LALR only; passing it under Earley is an error.

def handler(e):
    print(type(e).__name__, e.line, e.column)   # UnexpectedToken 1 5
    return True          # continue after the token; False raises it


tree = parser.parse("1 + + 2", on_error=handler)
print(tree.pretty())     # the tree for "1 + 2"

When skipping is not enough, feed tokens to e.interactive_parser before returning True: it is the parse, left where it failed, and works like any interactive parser. An error that parse() raises carries it too. With a custom lexer it is None, so the handler can only skip or stop, and an error that the lexer or a lexer callback raises does not reach the handler at all (divergences).

Per-target support: the feature matrix.

Context around an error

Section titled “Context around an error”

An error names the token it stopped on. The token before it, and the statement it is in, are on the interactive parser. Python and Rust keep it when an LALR parse with the built-in lexer fails (e.interactive_parser, failure.interactive_parser; see Recovering from errors); Earley keeps none. Its last_token is the previous token. Its stacks, on parser_state (e.state) in Python and on the parser in Rust, are the state_stack of LALR state ids and the value_stack: the pieces of the tree built so far. In the samples, finished statements are already one tree, so the tokens on the stack are the statement in progress.

In Python and Rust, an LALR error has token_history. When a built-in lexer rejected the input, as the contextual lexer (the LALR default) does in the samples, it holds the previous token. It is None when the parser rejected a token, as on $END. last_token, Hyperlark’s own, is set under both built-in lexers.

In TypeScript and C the error carries no parser and the interactive parser has no last_token or stacks: drive it and record the tokens it feeds.

The samples parse let x = 1; let y 2; under LALR with this grammar as GRAMMAR; it fails at 2, where = was expected.

start: stmt+
stmt: "let" NAME "=" NUMBER ";"   -> let_stmt
    | "print" NAME ";"            -> print_stmt
NAME: /[a-z]+/
NUMBER: /[0-9]+/
%ignore " "
parser = lark.Lark(GRAMMAR, parser="lalr")
text = "let x = 1; let y 2;"


try:
    parser.parse(text)
except lark.exceptions.UnexpectedInput as e:
    e.token_history                     # [Token('NAME', 'y')]
    e.interactive_parser.last_token     # Token('NAME', 'y')


    stack = e.state.value_stack               # each read copies the parse so far
    # [Tree('__start_plus_0', [Tree('let_stmt', [...])]), Token('LET', 'let'), Token('NAME', 'y')]
    next(v for v in stack if isinstance(v, lark.Token))   # Token('LET', 'let')

With a custom lexer the error keeps no parser (divergences). Under transformer=, value_stack raises NotImplementedError (divergences).

Custom error messages

Section titled “Custom error messages”

In Python under LALR, match_examples detects what’s wrong in the input text by matching against example errors. Given the parse function and a dictionary mapping each label to some malformed syntax examples, it returns the label for the example that best matches the current error, or None. Two errors match when they have the same parser state, and preferably the same token.

examples = {
    "two operators in a row": ["1 + + 2", "3 * * 4"],
    "trailing operator":      ["1 +", "2 -"],
}


try:
    parser.parse("1 + + 2")
except lark.exceptions.UnexpectedInput as e:
    label = e.match_examples(parser.parse, examples)
    print(label or "unrecognised syntax error")

A label usually needs several examples, because the state depends on what precedes the bad token. The error reporting example does this on a JSON grammar.

Rust has no helper, but under LALR with the built-in lexer both error variants carry state: parse each example with the same parser and compare; ids are comparable only within one parser instance. TypeScript and C have neither the helper nor the state id, so map the token and the expected set to a message yourself.