A custom lexer instance: lex is called lazily once per parse and its
iterator (or iterable — a generator is both) is pulled one token at a time,
interleaved with the parser. Synchronous only (parse is sync).
An exception thrown here escapes parse as the SAME value (===), and is
never routed into on_error — that callback only sees UnexpectedToken /
UnexpectedCharacters, as in Lark. When a parse aborts mid-stream the
abandoned iterator's return() is called, so generator finally blocks
run.
A custom lexer instance:
lexis called lazily once per parse and its iterator (or iterable — a generator is both) is pulled one token at a time, interleaved with the parser. Synchronous only (parseis sync).An exception thrown here escapes
parseas the SAME value (===), and is never routed intoon_error— that callback only seesUnexpectedToken/UnexpectedCharacters, as in Lark. When a parse aborts mid-stream the abandoned iterator'sreturn()is called, so generatorfinallyblocks run.