Struct Token
pub struct Token {
pub type_id: TokenId,
pub value: CompactString,
pub start_pos: SentinelOption<u32>,
pub end_pos: SentinelOption<u32>,
pub line: SentinelOption<u32>,
pub end_line: SentinelOption<u32>,
pub column: SentinelOption<u32>,
pub end_column: SentinelOption<u32>,
}Expand description
A lexed token — Lark’s 8 slots. Positions are code
points: 0-based *_pos, 1-based line/column. SentinelOption<u32>
(4 B each; u32::MAX = position-less) rather than Option<u32> (8 B) — a
token can be constructed position-less (callbacks, $END), though the lexer
always sets all six. BasicLexer::next_token guards inputs via
[input_exceeds_u32] (LexError::InputTooLarge), which reserves u32::MAX,
so every position here is provably < u32::MAX and round-trips through the
sentinel losslessly.
Fields§
§type_id: TokenId§value: CompactString§start_pos: SentinelOption<u32>§end_pos: SentinelOption<u32>§line: SentinelOption<u32>§end_line: SentinelOption<u32>§column: SentinelOption<u32>§end_column: SentinelOption<u32>Implementations§
§impl Token
impl Token
pub fn synthetic(type_id: TokenId, value: impl Into<CompactString>) -> Self
pub fn synthetic(type_id: TokenId, value: impl Into<CompactString>) -> Self
A position-less token of a given terminal — the shape a binding builds when
feeding a token by hand to an interactive parser (Lark’s Token(type, value)
with no positions). Lexed tokens always carry positions; a manually-fed token
need not, and the interactive engine accepts position-less feeds.
pub fn line(&self) -> u32
pub fn line(&self) -> u32
1-based line, 0 for a position-less (synthetic) token. The lexer
always sets positions, so lexed tokens never hit the sentinel — this
is the no-unwrap accessor over the Self::line field (0 is
out-of-band precisely because positions are 1-based).
pub fn column(&self) -> u32
pub fn column(&self) -> u32
1-based column, 0 for a position-less token; see Self::line.