fix(typst): guard RawBlockBody against LALR-merged body-text states
Build and Deploy Verso / deploy (push) Successful in 10m0s

canShift(RawBlockBody) returns true in states LALR-merged with the
post-RawBlockOpen state, causing the tokenizer to consume all remaining
body text as one giant RawBlockBody. Add a backward character scan:
require newline immediately before input.pos, then walk back past any
lang tag (A-Za-z0-9) and verify three backticks precede it. Body-text
positions never have backtick-backtick-backtick there, so the guard
rejects them.

This was the root cause of everything after the first heading being
black: RawBlockBody swallowed the entire document from the user-name
line onward, making headings, bold, italic and math invisible.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
claude
2026-06-08 19:44:05 +00:00
parent 11d852fe18
commit f5a94c0ced
@@ -105,6 +105,20 @@ export const rawTokenizer = new ExternalTokenizer(
}
if (stack.canShift(RawBlockBody)) {
// Guard: must genuinely follow a RawBlockOpen (which ends with \n).
// Walk backward past any lang tag (A-Za-z0-9) and require ```.
// This blocks spurious LALR-merged states from consuming body text.
if (input.peek(-1) !== NEWLINE) return
let back = -2
while (
(input.peek(back) >= 65 && input.peek(back) <= 90) ||
(input.peek(back) >= 97 && input.peek(back) <= 122) ||
(input.peek(back) >= 48 && input.peek(back) <= 57)
) { back-- }
if (input.peek(back) !== BACKTICK ||
input.peek(back - 1) !== BACKTICK ||
input.peek(back - 2) !== BACKTICK) return
let hasContent = false
while (input.next !== -1) {
if (