Fix Typst: support multi-line math blocks ($ ... multi-line ... $)
Build and Deploy Verso / deploy (push) Successful in 14m1s

mathContentTokenizer was stopping at newlines, causing a parse error for
Typst block math that spans multiple lines.  The parser then entered a bad
state that cascaded: the stale error-recovery left the item* parser in a
degraded mode, causing body text below the math block to be highlighted as
t.string (orange).

Fix: switch to contextual: true (only fires inside InlineMath where
MathContent is actually expected) and remove the newline restriction so
the tokenizer reads until the closing '$' regardless of line boundaries.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
claude
2026-06-08 21:03:17 +00:00
parent f976c5ba92
commit 54ab282efc
@@ -238,19 +238,26 @@ export const lineCommentContentTokenizer = new ExternalTokenizer(
)
// ── mathContentTokenizer ────────────────────────────────────────────────
// Emits MathContent — everything between the $...$ delimiters (no newlines).
// External rather than a @tokens rule for the same reason as LineCommentContent:
// ![$\n]+ overlaps with spaces, '<', '@', and other literals in merged states.
// Emits MathContent — everything between the $...$ delimiters.
//
// contextual: true — only fires when MathContent is in the valid set, i.e.
// inside InlineMath → "$" . MathContent? "$". This prevents the tokenizer
// from consuming body text in LALR-merged states.
//
// No newline restriction: Typst block math spans multiple lines
// ($ formula \n continuation $), so we must read until the closing '$'
// regardless of newlines. The grammar's InlineMath rule handles both inline
// ($x^2$) and block ($ ... $) forms with the same production.
export const mathContentTokenizer = new ExternalTokenizer(
(input, _stack) => {
let hasContent = false
while (input.next !== -1 && input.next !== DOLLAR && input.next !== NEWLINE) {
while (input.next !== -1 && input.next !== DOLLAR) {
input.advance()
hasContent = true
}
if (hasContent) input.acceptToken(MathContent)
},
{ contextual: false }
{ contextual: true }
)
// ── codeKeywordTokenizer ─────────────────────────────────────────────────