fix(typst): add newlines to @skip so multi-line code args parse cleanly
Build and Deploy Verso / deploy (push) Successful in 9m31s

'\n' inside CodeArgs was an invalid token, triggering Lezer error recovery
and resetting parser state before codeIdentTokenizer could fire.  Heading
detection is unaffected — headingTokenizer uses raw input.peek(-1) char
reads which see the '\n' byte regardless of what @skip consumes.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
claude
2026-06-09 08:47:47 +00:00
parent 019b4041a8
commit 1b773fdda0
@@ -27,7 +27,6 @@ item {
Label | Label |
Ref | Ref |
Escape | Escape |
Newline |
MarkupContent MarkupContent
} }
@@ -184,10 +183,12 @@ Escape { "\\" EscapeChar }
// ── Regular tokens ──────────────────────────────────────────────────────── // ── Regular tokens ────────────────────────────────────────────────────────
@tokens { @tokens {
// Horizontal whitespace only. Newlines are kept as explicit Newline items // All whitespace including newlines. Heading detection still works because
// so that HeadingMark (which checks start-of-line via input.peek(-1)) can // headingTokenizer uses input.peek(-1) on the raw character stream — it sees
// reliably detect newlines in the raw input stream. // the '\n' byte regardless of what @skip consumes at the token level.
spaces { $[ \t]+ } // Including '\n' here lets multi-line code expressions (e.g. #figure(\n ...\n))
// parse without error instead of triggering Lezer error recovery.
spaces { $[ \t\n\r]+ }
// Boolean / null literals — distinct from keywords for highlighting. // Boolean / null literals — distinct from keywords for highlighting.
CodeBool { "true" | "false" | "none" | "auto" } CodeBool { "true" | "false" | "none" | "auto" }
@@ -213,9 +214,6 @@ Escape { "\\" EscapeChar }
// Escape: any single character after backslash. // Escape: any single character after backslash.
EscapeChar { _ } EscapeChar { _ }
// Newline item — kept out of @skip so heading detection works.
Newline { "\n" }
// Resolve ambiguities in merged states: // Resolve ambiguities in merged states:
// EscapeChar > spaces: after '\', EscapeChar must win over the skip token. // EscapeChar > spaces: after '\', EscapeChar must win over the skip token.
// "(" > "." > "]": callSuffix delimiters must win over MarkupContent after // "(" > "." > "]": callSuffix delimiters must win over MarkupContent after