Divergences from Lark

Where Hyperlark’s Python package behaves differently from Lark. To switch a project, start at Coming from Lark. For what each target supports, see the feature matrix.

Tokens and trees

Tokens are FastTokens by default, not the str subclass Lark uses. A FastToken passes isinstance(tok, Token), compares equal to its text, has a Token’s attributes, and supports string methods, formatting, concatenation, int(tok) and float(tok). But "".join(children), re.match(pattern, tok), json.dumps(tok) and tok in s raise TypeError, and isinstance(tok, str) is False. Pass tok.value where a str is required, or build with fast_tokens=False for slower str-subclass tokens, as in Lark.

A FastToken’s .type must be a str; assigning any other object raises TypeError. Lark accepts any object.

tree.data is a plain str, not Lark’s Token('RULE', name): tree.data == "start" works in both, tree.data.type only in Lark.

parse() returns a Tree subclass, so type(tree) is Tree is False. Test with isinstance.

Not implemented

hyperlark.reconstruct, hyperlark.tree_templates, hyperlark.tools and TextSlice do not exist (the feature matrix tracks their status), and Lark’s internal modules (lark.parsers, lark.parser_frontends, lark.grammar) have no counterpart.

A truthy value for any of these options raises ConfigurationError when the parser is built. The default, a falsy value, is accepted:

Option	Note
strict=
cache_grammar=	cache= alone works
use_bytes=
edit_terminals=
regex=	the third-party regex module. \p{...} escapes work without it

These values always raise ConfigurationError when the parser is built:

Value	Note
ambiguity='forest'	'resolve' and 'explicit' work
parser='cyk'	'lalr' and 'earley' work
parser=None	Lark’s lexer-only mode. To lex without parsing, build a parser whose rules use the terminals and call .lex()
propagate_positions= a callable	Lark’s node filter. True and False work

ordered_sets= is accepted and ignored: sets are always ordered.

Grammars

Terminal regexes use Python’s re syntax but are compiled by Rust regex engines. A construct the two read differently, such as a trailing $, is refused when the parser is built, and the error names it. Terminals lists the refused constructs and the experimental translate_python_regex=True option that rewrites them instead.

A missing %import raises GrammarError. Lark raises IOError.

A terminal or rule priority outside ±2147483647, or a rule repeat count (~ n..m) outside the signed 64-bit range, is rejected when the parser is built. Lark accepts both.

TBD: a relative %import, such as %import .mygrammar (NAME), in a grammar passed as a string raises GrammarError. Lark looks beside the running script, or in the working directory when there is none. Until then, pass source_path= or import_paths=.

TBD: a terminal that repeats part of its pattern very many times, such as "a" ~ 1..300000 or /[a-z]{1,100000}/, exceeds the regex engine’s size limit and is refused when the parser is built. Lark builds it.

The parser object

Lark has no rules, grammar, lexer_conf, ignore_tokens, serialize(), deserialize() or memo_serialize(). To save and reload a parser, use Lark.save() and Lark.load(). options, terminals and get_terminal() are present, and options holds the same normalized values as Lark’s.

debug=True sets the hyperlark logger to DEBUG. Lark’s debug leaves its logger’s level unchanged.

With transformer=, a method named after a _-prefixed rule raises ConfigurationError when the parser is built. Lark calls it, and so does Hyperlark when the transformer also overrides __default__.

With transformer= and propagate_positions=True, Hyperlark calls __lark_meta__ only on the first and last values of a node. Lark also calls it on the first and last values of each _-rule and of each step of a repetition (item+, item*), so a __lark_meta__ that raises there fails the parse in Lark only. The positions are the same.

Errors and recovery

• An Earley error has state=None, so match_examples raises AssertionError on it; build with parser="lalr" to classify errors.
• err.interactive_parser is None with a custom lexer, so an on_error= handler there can skip the bad input or stop, but not feed tokens. Under LALR, err.state then has only position and equality, and for a lexing error inside the custom lexer it is None.
• With a custom lexer, an UnexpectedInput that the lexer or a lexer callback raises does not reach an on_error= handler: it propagates as raised.
• An UnexpectedInput that the transformer= raises does not reach an on_error= handler: it propagates as raised. Lark hands it over with the parser in the middle of that reduction.
• A transformer= method that raises while an on_error= handler feeds err.interactive_parser ends the parse: if the handler catches the exception and returns True, resuming raises LarkError. Lark resumes.
• Under LALR, an UnexpectedToken that Hyperlark raises in parse() or resume_parse() has expected equal to accepts, the tokens that will advance the parser into a new valid state. Lark’s expected is not narrowed to accepts, so it can name a token the parser would then reject.
• err.token_history names the last token the parser received, where Lark names the last one its lexer produced.
• Under LALR, a non-callable on_error= raises TypeError as soon as parse() is called, even on input that parses. Lark raises only when it calls the handler, at the first error.

Lexer callbacks

lexer_callbacks can be used to interface with the lexer as it generates tokens (Lexer callbacks). It accepts a dictionary of the form {TOKEN_TYPE: callback}, where the callback is of type f(Token) -> Token. The token’s .type and .value are writable, and tok.update(value=...) or type(tok).new_borrow_pos(...) builds a replacement. A callback on a terminal that is not ignored must return a token, or LexError is raised. Under Earley’s dynamic lexer lexer_callbacks= raises GrammarError; use lexer="basic".

It differs from Lark in these ways:

• The returned token’s type must be a terminal of the compiled grammar, or LexError is raised; Lark lets it reach the parser and fail there as UnexpectedToken. To retype a token to a terminal no rule uses, %declare it.
• Position fields the returned token leaves unset are copied from the input token, so a bare Token('X', 'y') keeps the original positions. Lark leaves them None.
• Writing .value changes the emitted token’s text, the string it compares equal to and the tree leaf, with or without fast_tokens. In Lark it changes only the attribute.
• A non-str .value, as in t.update(value=int(t)), is stored as str(value); Lark keeps the object. Convert in a transformer instead.
• Under the contextual lexer, an UnexpectedInput a callback raises reaches you, or an on_error= handler, as raised. Lark’s contextual lexer replaces it with an UnexpectedToken for the next token, or an EOFError at the end of the input.
• Under lex(text, dont_ignore=True), a callback on an %ignored terminal that returns anything but a token leaves the original token in the stream. Lark raises LexError there.

On an %ignored terminal the return value is discarded, so comments.append works as the callback:

import hyperlark

comments = []
parser = hyperlark.Lark(r"""
    start: WORD+
    COMMENT: /#[^\n]*/
    %import common.WORD
    %ignore " "
    %ignore /\n/
    %ignore COMMENT
""", parser="lalr", lexer_callbacks={"COMMENT": comments.append})

parser.parse("hello # a note\nworld")
print(comments)          # [Token('COMMENT', '# a note')]

Postlex

A postlex (postlex=) post-processes the token stream between the lexer and the parser. Its process() generator runs for the whole of one parse or one interactive parser. Lark starts a fresh generator on every parse, resume_parse, exhaust_lexer and iter_parse call, which for an Indenter also resets its paren and indent state. The difference shows when one parse is driven more than once: under on_error, by resume_parse after breaking out of iter_parse, or by a second exhaust_lexer after a rejection.

• State can carry over. A new generator starts only after the lexer has raised an error inside it, which under the default contextual lexer includes a token the parser cannot accept.
• An exception other than an UnexpectedInput that process() itself raises is permanent for that parse: it does not reach an on_error= handler, and every later call that reads from the lexer (exhaust_lexer, iter_parse, resume_parse, peek_next, feed_next) raises it again.
• InteractiveParser.copy(), as_immutable(), and the immutable parser’s peek_next and exhaust_lexer fork without a Python postlex: the fork receives the lexer’s tokens, and nothing warns. Avoid them with a Python postlex.

An Indenter that does not override process, _process, handle_NL or always_accept runs in the built-in implementation. Its paren and indent state also carries across errors, and copy() carries that state into the fork. An exception the Indenter raises is permanent in the same way.

Interactive parsing

parse_interactive is LALR-only, as in Lark, and also requires the built-in lexer: a parser built with a custom lexer raises NotImplementedError. The InteractiveParser adds last_token (the most recent token the parser consumed, or None before the first one and after feed_eof), peek_next() and feed_next(), described in Interactive parsing. It differs from Lark’s in these ways:

• A rejected feed_token raises UnexpectedToken with .accepts filled; Lark’s is None.
• A token fed with feed_token carries no line or column into the tree or into errors, even one that came from lex(); tokens the parser lexes itself keep theirs. The positions of the last_token passed to feed_eof are ignored too.
• choices() values are ('Shift', state_id) or ('Reduce', rule_id) tuples. In Lark the values are its Shift and Reduce action objects, and only a Reduce carries a Rule.
• Breaking out of iter_parse leaves the yielded token queued, so a later call that advances the parse resumes with it. In Lark the lexer has already consumed it, and resume_parse continues without it.
• Inside an iter_parse loop, passing the yielded token to feed_token feeds it once. Lark feeds it a second time when the loop continues.
• The .value rules for lexer callbacks apply to the token iter_parse yields and to a peek_next token committed with feed_next(tok).
• parser_state has only position, state_stack, value_stack and equality. Each value_stack read copies the parse so far. It is empty once the parse has finished, and raises NotImplementedError under an embedded transformer=. lexer_thread and lexer_state are not exposed.
• Under an embedded transformer=, a parser that has returned its tree raises LarkError on any further feed or resume, including the resume after an on_error= handler that finished it. Lark raises UnexpectedToken.
• resume_parse continues the postlex where it stopped, so an indenter keeps its indentation levels. Lark starts the postlex again from no indentation, which breaks recovery inside an indented block or brackets.
• On ImmutableInteractiveParser, feed_next and iter_parse raise NotImplementedError.
• On ImmutableInteractiveParser, the interactive_parser of an error resume_parse raises is a mutable copy. Lark’s shares the immutable parser’s state.
• Calling an interactive parser from one of its own callbacks (a lexer callback, a postlex or a transformer= method), or from a second thread while a call is in progress, raises RuntimeError.

Class identity and compat mode

Hyperlark’s Token, Tree and exception classes are its own. Without the optional compat mode, isinstance(tok, lark.Token) is False, an except lark.exceptions.UnexpectedToken clause does not catch Hyperlark’s exception, and a lark.Transformer applied to a Hyperlark tree returns it unchanged. Using Hyperlark alongside lark has the install and import steps.

With from hyperlark.compat import Lark, parse() returns lark.Tree and lark.Token objects and raises the lark.exceptions classes. Installing the extra alone changes nothing. Compat mode is slower than plain hyperlark.Lark: it converts after the parse and defaults to fast_tokens=False.

Compat mode is incomplete in this beta, and nothing raises at a gap:

• Only parse() converts. lex(), parse_interactive() and parse_eager() return Hyperlark objects.
• An on_error= handler receives the Hyperlark exception.
• A converted error’s interactive_parser and state are Hyperlark’s, so what they return or raise is not converted.
• compat.Lark has no open() or open_from_package(); construct it from a grammar string.

Parsing from inside a callback

Code that runs during a parse (a lexer callback, a custom lexer, a postlex, a transformer method) may itself parse, with the same or another instance, or drive an interactive parser. Nesting deeper than 8 levels on one thread raises RecursionError.

Building a Lark from code the constructor runs (a custom lexer’s __init__, or a postlex, transformer or %import loader it inspects) has no such bound. Unbounded recursion there exhausts the C stack and can kill the process with SIGSEGV, where Lark raises an error. sys.setrecursionlimit() does not help, and a thread with a small stack (under about 4 MiB) crashes far sooner than the 8 MiB main thread. Give such code a base case or a depth counter.

Deep input and deep trees behave as in Lark: the parser is iterative, and repr, pretty and Transformer on a very deep tree raise RecursionError.