Struct ContextualLexer 

pub struct ContextualLexer { /* private fields */ }

Per-parser-state lexer set.

Implementations

impl ContextualLexer

pub fn new( table: &ParseTable, conf: &LexerConf, always_accept: &[String], ) -> Result<Self, LexerBuildError>

Build from the table’s accept-sets: derive each state’s effective keep set — its token_actions row restricted to real terminals, unioned with conf.ignore + always_accept (lexer.py:673) — dedup on that, and build one BasicLexer per distinct set; plus the root lexer over all terminals. always_accept arrives as a parameter.

The dedup key is the keep set itself, not the raw accept-set. A built lexer is a pure function of its keep set, and the raw row carries two kinds of id that a keep set drops: $END and %declare’d ids (both >= n_terms, with no TerminalDef row — mirroring Lark’s terminals_by_name filter, lexer.py:675). Keying on the raw row therefore splits states whose lexers are byte-identical, duplicating a scanner build and its DFA. Measured over the conformance corpus: 2095 -> 1706 slots, ~6 % of all per-state terminal compilations. Concentrated in indentation grammars — on python.lark all 4 merges come from the %declare’d _INDENT/_DEDENT, none from $END, so dropping only $END would win the corpus count and none of the cost that matters.

This widens an already-sanctioned divergence: Lark’s key (frozenset(states[state].keys()), lexer.py:669) is coarser still — for LALR it also carries goto-NONTERMINAL names, which never reach terminals_by_name either. Every built lexer is identical either way, so Self::unique_lexer_count stays a documented lower bound on Lark’s cache-entry count; it is simply a tighter one now.

pub fn next_token( &self, state_id: StateId, text: &str, lexer_state: &mut LexerState, ) -> Result<Option<Token>, LexError>

Single-token step for parser state state_id (the interactive+contextual path drives one token at a time; batch-only cannot serve interactive parsing). The same lexer_state threads across calls even as the underlying per-state lexer changes — position/line counting stays continuous because the state carries only lexer-independent cursors.

Root fallback: on this state’s LexError::UnexpectedCharacters, retry the same position with the root lexer on the real state (Lark mutates the shared state — a root match advances the cursor past the token before the error surfaces; the interactive recovery reads that position); a root match raises the wrong-context error (mapped to LexError::WrongContextToken) carrying the original error’s allowed; a root non-match re-raises the original error. Ok(None) = input exhausted.

state_id MUST be a live parser-state id (as produced by the driving parser); an out-of-range id is a driver bug and panics (debug-asserted). The StateId parameter keeps that invariant in the type rather than in a convention — token_actions is TiVec<StateId, _> and the driver already holds a StateId, so no usize hop happens on the way here.

pub fn build_all(&self)

Materialize every lexer a parse can reach WITHOUT a cover rejection — the cover slots, which serve every other slot ([assign_covers]). For measurement (a build number must contain the lexers — see dev/bench/README.md) and for callers that would rather pay at construction than on a first parse. A covered state still builds its own lexer if a cover answer is ever rejected there, so this no longer leaves a parse with nothing at all to build — only with nothing it will build unless a wider scanner actually disagrees.

pub fn unique_lexer_count(&self) -> usize

The number of distinct per-state lexer slots after keep-set dedup. This counts slots, not built lexers — each is materialized lazily on first use, so the number actually built during a parse is typically smaller.

pub fn cover_lexer_count(&self) -> usize

The number of slots that compile a scanner up front: the maximal keep sets. Every other slot reads one of these ([assign_covers]) unless a cover answer is rejected there.

pub fn built_lexer_count(&self) -> usize

Slots whose lexer is materialized right now — cover slots plus every covered slot that has had an answer rejected.

pub fn cover_fallbacks(&self) -> u64

PROTOTYPE instrumentation: cover answers rejected so far (at most one per covered slot per parse-lifetime — the slot latches on its own lexer).