Struct InteractiveParser

pub struct InteractiveParser<'p> { /* private fields */ }

Expand description

The mutable interactive parser. Construction: crate::lalr::parse_interactive.

Semantics locked:

feed_eof raises UnexpectedToken('$END') — never UnexpectedEOF (the POC’s rewrite is the parity bug, second site). The synthesized $END from a bare feed_eof() takes Lark’s ELSE-branch — (start_pos=0, line=1, column=1), end_* None — even when the cursor has a last token (Lark’s bare feed_eof() passes last_token=None). resume_parse takes the BORROW-branch from the last fed token (parse_from_state).
feed_token on this mutable type does NOT set result for the token it is given; only iter_parse-style driving (exhaust_lexer) and the immutable wrapper’s feed do. (A proposal parked by peek_next and drained by feed_next is a driven feed, so it does set it.)
accepts() trial-feeds every .isupper() choice including $END (fed is_end=true) over a shallow, callback-free cursor — no reduce callbacks run speculatively, no arena clone. Reuses the engine’s error-time trial-feed machinery ([ParserState::accepts_here]); the terminal predicate is exact because token_actions holds precisely the terminal keys and every Lark terminal name ($END, _NEWLINE, __ANON_n, …) is .isupper()-true while every goto key is a lowercase rule name.
resume_parse mirrors Lark’s parse_from_state loop: exhaust the lexer in place, feed the borrow-branch $END, accept -> tree. The on_error retry loop that keys on the second consecutive $END-typed UnexpectedToken (a self-comparison over shared in-place-mutated state) is deferred with the rich on_error surface — the dropped-UnexpectedEOF decision preserves the token.type == '$END' the guard needs.

Implementations§

§

impl<'p> InteractiveParser<'p>

pub fn feed_token(&mut self, token: Token) -> Result<(), ParseError>

Feed one token. $END-typed tokens feed with is_end = true. This feed does not set result; only iter_parse-style driving (exhaust_lexer) and the immutable wrapper’s feed do. Non-$END feeds yield no value; a $END fed here is accepted but its root is dropped — use Self::feed_eof to retrieve the finished tree. A reject reports the raw .isupper() expected set (direct path, Self::feed_direct).

token is INSERTED into the stream, ahead of anything the cursor has already buffered — a token held by Self::peek_next, or a postlex fan-out awaiting its turn. It does not replace them; they are still fed, by the next driving call. This matches Lark, where a feed_token from inside an iter_parse loop likewise lands before the token the loop has already yielded.

pub fn feed_eof(&mut self) -> Result<ParsedTree, ParseError>

Synthesize and feed a bare $END; on acceptance hand off the finished tree.

The $END takes Lark’s ELSE-branch positions — (start_pos=0, line=1, column=1), end_* None — mirroring Lark’s bare feed_eof() with last_token=None (even after a real lex the bare form does NOT borrow). A rejected $END raises UnexpectedToken('$END') as-is (never UnexpectedEOF).

Arena handoff: acceptance moves the state’s arenas into the ParsedTree ([ParserState::finish_in_place]) — the parser must not be reused after.

Finishing HERE discards whatever the cursor still holds: the rest of the input, and with it any token Self::peek_next left buffered. That is what feed_eof has always meant — it never consults the lexer — and a peeked token is not special-cased out of it. Use Self::resume_parse to finish the remaining input instead.

pub fn copy(&self) -> Self

Deep copy (Lark copy()): an independent ParserState (stacks and arenas) plus a forked cursor. result resets to None (Lark re-inits it in __init__). A non-clonable postlex session is dropped from the copy (clone_box — the fork continues without postlex, documented limit). A token held by Self::peek_next rides along with the forked buffer, so both sides see the same next token — forced, not a nicety: the cursor has already advanced past it (and for a custom source it is gone from a SHARED stream), so a fork that dropped it would be born torn.

pub fn accepts(&self) -> Vec<String>

The terminals (incl. $END) this state can accept — Lark’s exact set via callback-free shallow trial feeds, sorted. Membership is trial-fed, so a terminal that is a reduce action but leads to no valid state (e.g. calc RPAR right after a NUMBER) is excluded even though it appears in Self::choices.

pub fn is_known_token_id(&self, id: usize) -> bool

Whether id names a terminal in this parser’s table. For callers that hold a raw id rather than a name (bindings retyping a queued token): an out-of-range id would otherwise index-panic on the next feed, which across an FFI boundary is an abort rather than a catchable error.

pub fn is_end_token_id(&self, id: usize) -> bool

Whether id is the table’s reserved $END. A queued token retyped to it is sent by an ordinary driving call, which accepts but returns a token and drops the root — bindings reject the retype and route to feed_eof.

pub fn last_token(&self) -> Option<&Token>

The most recent terminal the parser SHIFTED, or None before the first one (and after a finishing feed, which moves the token arena out).

Beyond Lark, and deliberately a different question from Lark’s parser_state.value_stack[-1]: this names the last terminal, where the stack top is whatever the last action left there — a tree, after a reduce. The two coincide exactly where a caller can observe them from a driving loop, because a successful feed_token reduces in a loop and then ENDS by shifting that token, so right after any feed the stack top is that token. Reading the last terminal instead keeps the answer a plain Token, which is why it is exposed and the raw value stack is not: the stack holds splices for every inlined / repetition rule, and a splice carries no rule name to hand back (see NodeValue).

pub fn choices(&self) -> BTreeMap<String, ChoiceAction>

The full per-state action map: token_actions re-merged with goto_actions (rule-name keys included) so the output reproduces Lark’s raw states[position] keys. Sorted map so output is deterministic — differential tests compare keys (target state ids are hyperlark’s canonical renumbering, not Lark’s).

pub fn pretty(&self) -> String

Lark’s pretty() over the merged choices map: a Parser choices: header, one \t- {key} -> {action} line per entry (sorted, not Lark’s insertion order), and a stack size: N footer. Action rendering is hyperlark’s own ((Shift, s) / (Reduce, r)); the shape mirrors Lark but the target ids are the canonical renumbering.

pub fn exhaust_lexer(&mut self) -> Result<Vec<Token>, ParseError>

Feed every remaining lexer token and return the consumed list (Lark exhaust_lexer = list(iter_parse())): drive the cursor keyed on the live parser state, feeding each token in place. The returned Vec is the post-postlex stream in feed order (the indenter’s flushed DEDENTs included). Sets result to each feed’s return (None for a non-$END feed), matching iter_parse. Does NOT feed $END; a reject reports the raw expected set (direct path). On a reject the partial list is dropped (Lark’s list() discards it too).

pub fn peek_next(&mut self) -> Result<Option<&mut Token>, ParseError>

Borrow the next token to be fed WITHOUT feeding it — the lexer’s proposal, open to revision.

This is the negotiation step: peek_next asks “may I send this?”, and the caller answers by mutating the token (retyping it, rewriting its value) or leaving it be. The token stays queued in the cursor exactly where it already was, so the next driving call — Self::feed_next, Self::exhaust_lexer, Self::resume_parse — sends it as edited, with its ORIGINAL positions intact. There is no separate “send the peeked token” call, and nothing to remember to drain: peeking does not take the token out of the stream, it just looks at the front of it.

A replacement is not a substitute for editing in place: Self::feed_token INSERTS ahead of the queued token rather than replacing it (Lark’s ordering), and in a binding a by-name feed builds a Token::synthetic, which has no positions at all. (A Rust caller can position a token by hand — Token’s fields are public — but that means copying six fields to get where mutation already is.)

Idempotent — peeking twice without feeding returns the same token and does not advance the cursor. Self::accepts does not consume it either, so the accepts set still describes the state BEFORE the proposal, which is exactly what you need to judge whether to substitute.

None at end of input: nothing left to propose. Under a postlex the token borrowed here may be a FABRICATED one (an indenter’s DEDENT) rather than a fresh lex — it is whatever the parser would be fed next, which is the useful answer and the one an editing caller wants.

KNOWN DIVERGENCE, low stakes: an edit made here does not reach the lexer’s last_token, which is a positions snapshot taken when the token was PULLED. Self::resume_parse seeds its $END borrow from that snapshot, so if you edit a token, feed it by some other path, and then resume with nothing left to lex, the $END in a rejection carries the token’s ORIGINAL positions. Lark aliases the object rather than snapshotting it, so it reports the edited ones. Matching it would mean tracking which queued tokens are eligible to refresh the snapshot — a postlex FABRICATES tokens that deliberately are not last_token — which is more machinery than the six position fields on one error token are worth.

Retype only to an id the table knows — get one from crate::model::ParseTable::token_id. A TokenId past the table’s terminal count is a programming error, not bad input, and panics on the next feed like any out-of-range index (the same panic a hand-built Token::synthetic with that id has always had). Reporting it as a ParseError instead would mean a new variant on a public enum, which this does not take on.

pub fn peeked(&self) -> Option<&Token>

The token already queued for the next feed, if any — read-only, and it does NOT lex, where Self::peek_next will. None means nothing is queued, which is not the same as end of input.

Not exclusively “what you peeked”: a postlex fan-out queues tokens here too, so this reports the next token to be fed whoever put it there. That is the question worth answering — “is a feed already spoken for?” — and it is the only way to see what an ImmutableInteractiveParser inherited through a Self::copy.

pub fn feed_next(&mut self) -> Result<Option<Token>, ParseError>

One lazy step of iter_parse: pull the next token keyed on the live state, feed it (setting result), and return it — or None at end of input. Unlike Self::exhaust_lexer the caller may stop between tokens; $END is never fed (use Self::feed_eof / Self::resume_parse to finish).

A token held by Self::peek_next is at the front of the cursor buffer, so this is also what sends it — the peek is drained here, not by a separate call.

pub fn resume_parse(&mut self) -> Result<ParsedTree, ParseError>

Resume the normal parse from the current state (Lark resume_parse -> parse_from_state): exhaust the lexer with the in-place state, then feed a $END that borrows the last fed token’s positions (else the (0,1,1) default when nothing was fed). Accept -> finished tree; a rejected $END raises UnexpectedToken('$END') as-is. Same arena handoff as Self::feed_eof.