Struct Scanner 

pub struct Scanner { /* private fields */ }

The compiled two-partition scanner.

Implementations

impl Scanner

pub fn new( specs: Vec<ScanSpec>, g_regex_flags: u32, ) -> Result<Self, ScannerBuildError>

Compile specs, which MUST already be in Lark’s four-part sort order — rank is the index. Partitioning is by compile capability: try the fast engine, fall back to fancy-regex; failing both is ScannerBuildError::Uncompilable.

g_regex_flags is Python’s re bitmask (I=2 M=8 S=16 X=64), applied at compile time only: the fast partition via the syntax config, the slow partition by wrapping each spec in its own (?FLAGS:…) scope ([slow_source]) — never by editing the spec strings (grammar-wide flags).

pub fn new_with_engine( specs: Vec<ScanSpec>, g_regex_flags: u32, engine: ScannerEngine, ) -> Result<Self, ScannerBuildError>

As Scanner::new, but with an explicit fast-partition ScannerEngine instead of the process default. The slow partition, ranks, and merge semantics are identical across engines — only the fast matcher differs.

pub fn match_at(&self, text: &str, pos: usize) -> Option<ScanMatch>

The highest-ranked match starting exactly at byte offset pos (rank-first cross-engine merge; None = no terminal matches here). pos must be within text and on a UTF-8 char boundary; a pos past the end or mid-character yields None (no token can start there) rather than panicking — the slow anchored path slices text[pos..]. Internal callers always pass a boundary; this keeps the public method total.

pub fn match_at_cached( &self, text: &str, pos: usize, cache: &mut ScanCache, ) -> Option<ScanMatch>

As Scanner::match_at, but reuses cache for the Lazy fast engine’s lazy DFA instead of acquiring a pooled cache per call — the hot lex path threads one ScanCache through the whole token stream (~3-4% of parse). Byte-identical to match_at; the cache is pure memoization.

pub fn token_ids(&self) -> impl Iterator<Item = TokenId> + '_

Token ids present in this scanner (minus UNLESS-embedded literals) — feeds the UnexpectedCharacters.allowed set.