fix: revert mathContentTokenizer regression (contextual + no-newline-stop)
Build and Deploy Verso / deploy (push) Successful in 9m43s
Build and Deploy Verso / deploy (push) Successful in 9m43s
The previous change switched mathContentTokenizer to contextual:true with no newline stop, intending to support multi-line Typst block math. However, LALR state merging causes canShift(MathContent) to spuriously return true in body-text positions (e.g. after a RawInline backtick close), so the tokenizer consumed everything until the next '$' — turning a full paragraph orange. Revert to contextual:false with newline stop. This correctly handles both inline ($x^2$) and single-line block ($ integral ... $) math. Multi-line block math ($ formula\n continuation $) remains a separate issue for later. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -238,26 +238,19 @@ export const lineCommentContentTokenizer = new ExternalTokenizer(
|
||||
)
|
||||
|
||||
// ── mathContentTokenizer ────────────────────────────────────────────────
|
||||
// Emits MathContent — everything between the $...$ delimiters.
|
||||
//
|
||||
// contextual: true — only fires when MathContent is in the valid set, i.e.
|
||||
// inside InlineMath → "$" . MathContent? "$". This prevents the tokenizer
|
||||
// from consuming body text in LALR-merged states.
|
||||
//
|
||||
// No newline restriction: Typst block math spans multiple lines
|
||||
// ($ formula \n continuation $), so we must read until the closing '$'
|
||||
// regardless of newlines. The grammar's InlineMath rule handles both inline
|
||||
// ($x^2$) and block ($ ... $) forms with the same production.
|
||||
// Emits MathContent — everything between the $...$ delimiters (no newlines).
|
||||
// External rather than a @tokens rule for the same reason as LineCommentContent:
|
||||
// ![$\n]+ overlaps with spaces, '<', '@', and other literals in merged states.
|
||||
export const mathContentTokenizer = new ExternalTokenizer(
|
||||
(input, _stack) => {
|
||||
let hasContent = false
|
||||
while (input.next !== -1 && input.next !== DOLLAR) {
|
||||
while (input.next !== -1 && input.next !== DOLLAR && input.next !== NEWLINE) {
|
||||
input.advance()
|
||||
hasContent = true
|
||||
}
|
||||
if (hasContent) input.acceptToken(MathContent)
|
||||
},
|
||||
{ contextual: true }
|
||||
{ contextual: false }
|
||||
)
|
||||
|
||||
// ── codeKeywordTokenizer ─────────────────────────────────────────────────
|
||||
|
||||
Reference in New Issue
Block a user