Struct BasicLexer 

pub struct BasicLexer { /* private fields */ }

Lark’s BasicLexer: scanner + ignore/newline sets + composed callbacks + the name↔id tables (ids cover all conf terminals, including scanner-removed absorbed literals).

Everything shareable is behind Arc, so deriving a variant lexer copies pointers rather than tables.

Implementations

impl BasicLexer

pub fn new(conf: LexerConf) -> Result<Self, LexerBuildError>

The build pipeline:

1. every per-terminal derivation ([PreparedTerminals::new]): the sort by (-priority, -max_width, -len(value), name), the newline set, each terminal’s admission and the UNLESS absorb rows;
2. validate ([PreparedTerminals::validate]) unless conf.skip_validation;
3. the refiners and the scanner-removed literals ([PreparedTerminals::create_unless]);
4. compose user callbacks ([PreparedTerminals::compose_callbacks]) via the gated CallChain;
5. assemble the scanner ([PreparedTerminals::compile_scanner]), minus UNLESS-embedded literals.

pub fn names(&self) -> &[String]

Terminal names by id (conf order).

pub fn name_to_id(&self, name: &str) -> Option<TokenId>

pub fn next_token( &self, text: &str, state: &mut LexerState, ) -> Result<Option<Token>, LexError>

Lark’s next_token flow: scan at the byte cursor; on a match, feed the tracker before filling end positions; run the callback after end positions; a token is built when it is not ignored or has a callback, returned only when not ignored; last_token updates only on a returned token. Ok(None) = input exhausted — no synthetic EOF token (boundary inputs).

pub fn next_token_dont_ignore( &self, text: &str, state: &mut LexerState, ) -> Result<Option<Token>, LexError>

Self::next_token re-emitting %ignored terminals — the lex(dont_ignore = true) path. Same terminals, same positions, same callbacks; only the drop of an ignored token is suppressed.

pub fn lex<'l, 't>(&'l self, text: &'t str) -> Lex<'l, 't>

Iterate all tokens of text from a fresh state.

The state is checked out of Self::ls_pool, so the lazy-DFA scan cache is WARM — a repeated lex (the Earley pre-scan and the public Lark.lex both drive this) never re-warms it from cold, the same win the LALR path already took. Observable state is reset per call (LexerState::reset_keep_cache), so a pooled pass is byte-identical to one over a fresh LexerState.

pub fn lex_with<'l, 't>( &'l self, text: &'t str, dont_ignore: bool, ) -> Lex<'l, 't>

Self::lex, re-emitting %ignored terminals when dont_ignore.

One lexer serves both: the flag reaches the loop as a const parameter, so neither path pays for the other, and BOTH share this lexer’s warm state pool. A second lexer with an emptied ignore set would own a second, cold pool and re-warm the lazy DFA on every call.

pub fn is_ignored(&self, id: TokenId) -> bool

Whether id is an %ignore terminal of THIS lexer: the configured set, unaffected by a Self::lex_with pass that re-emits ignored terminals.

pub fn without_ignore(&self) -> Self

👎Deprecated since 0.1.0-beta.2:

use lex_with(text, dont_ignore); this clone owns a cold state pool

The dont_ignore variant: same terminals and callbacks, ignore set emptied. Callbacks survive (they’re Arc).

PREFER Self::lex_with. This clone owns a SEPARATE, COLD state pool, so one built per call re-warms the lazy DFA on every lex — measured 2.9x on a small input. lex_with reaches the same behaviour through a const parameter on this lexer, keeping the warm pool.

Trait Implementations

impl Debug for BasicLexer

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.