Coming from Lark

Hyperlark’s Python package follows Lark 1.3’s API for parser="lalr" and parser="earley": Lark, Tree, Token, Transformer, Visitor, Interpreter, v_args and the exception classes are all there, with Lark’s exception hierarchy. Most code runs after changing the imports. What to check is below; Divergences from Lark lists the differences, and benchmarks has the speed comparison.

Migrating

1. pip install hyperlark (Python 3.10 or later). Drop Lark from your dependencies unless you use compat mode.

2. Replace lark with hyperlark in every import, or alias the package:

   import hyperlark as lark   # was: import lark

   The public submodules keep their names (hyperlark.exceptions, hyperlark.visitors, hyperlark.tree, hyperlark.lexer, hyperlark.indenter, hyperlark.ast_utils, hyperlark.load_grammar). Lark’s internal modules (lark.parsers, lark.grammar, lark.common, lark.utils) have no counterpart.

3. Lark.open(), Lark.open_from_package() and %import common are unchanged.

4. Run your tests.

What to check

Tokens are not str

Tokens are FastTokens by default, including the ones a Transformer receives. A FastToken compares equal to its text, and string methods, formatting, concatenation, int(tok) and float(tok) work. But it is not a str: "".join(children), re.match(pattern, tok), json.dumps(tok) and tok in s raise TypeError, and isinstance(tok, str) is False, so test isinstance(tok, Token) instead.

import hyperlark

parser = hyperlark.Lark(r"""
    start: WORD+
    %import common.WORD
    %ignore " "
""", parser="lalr")
words = parser.parse("hello world").children

# "".join(words) raises TypeError
"".join(w.value for w in words)    # 'helloworld'

Pass .value where a str is required, or build the parser with fast_tokens=False for Lark’s str-subclass tokens everywhere, at a cost in speed.

Terminal regexes

Terminal regexes use Python’s re syntax but are compiled by Rust regex engines. A construct the two read differently, such as a trailing $, is refused when the parser is built, and the error names it. Terminals lists the refused constructs and the translate_python_regex=True option that rewrites them instead.

Not implemented

hyperlark.reconstruct, hyperlark.tree_templates, hyperlark.tools and TextSlice do not exist, and neither do the standalone, nearley and serialize tools. An option Hyperlark recognizes but does not support, such as strict=True, regex=True, use_bytes=True, parser="cyk", parser=None or ambiguity="forest", raises ConfigurationError when the parser is built (full table). \p{...} escapes work without regex=True, so drop it. In place of parser=None, build a parser whose rules use the terminals and call .lex().

Smaller differences

• tree.data is a plain str: tree.data == "start" works, tree.data.type does not exist.
• A missing %import raises GrammarError, not IOError.
• The parser object lacks rules, grammar, lexer_conf, ignore_tokens and serialize() (full list). To save and reload a parser, use save() and load().

Divergences from Lark has the rest: lexer callbacks, postlex, the interactive parser and match_examples on Earley errors.

Using Hyperlark alongside lark

Hyperlark’s Token, Tree and exceptions are separate classes from Lark’s. So when code in the same process still imports lark and relies on its classes, isinstance(x, lark.Token) is False, except lark.exceptions.UnexpectedToken lets Hyperlark’s exception through, and a lark.Transformer applied to a Hyperlark tree returns it unchanged. For that case, install the compat extra and build the parser from hyperlark.compat:

pip install "hyperlark[compat]"

from hyperlark.compat import Lark

parse() then returns real lark.Tree and lark.Token objects and raises the lark.exceptions classes. Compat mode is slower: it defaults to fast_tokens=False, and the tree is converted after the parse.

Compat mode is incomplete in this beta. Only parse() converts; lex() and parse_interactive() return Hyperlark objects without warning. compat.Lark has no open() or open_from_package(), so pass the grammar as a string. The remaining gaps are on the divergences page.

How compatibility is checked

Lark 1.3.1’s own test suite runs against Hyperlark in CI with fast_tokens=False: 849 tests pass and the other 375 are skipped, each with a recorded reason (the CYK parser, the documented divergences, and cases Lark’s suite skips itself). Lark’s test files for the reconstructor, tree templates, the forest transformer, the standalone and nearley tools, and logging are not run.