Replace the StreamLanguage tokenizer with a full LR grammar compiled by
@lezer/generator, giving Typst the same parse-tree infrastructure that
LaTeX and BibTeX already use.
Grammar features:
- Headings (=, ==, …) via SOL-detecting external tokenizer
- Code expressions (#keyword, #func(args), #ident.method, #{…}, #[…])
- Named argument highlighting (key: value in function calls)
- Inline and display math ($…$)
- Strong (*…*) and emphasis (_…_) with bold/italic formatting
- Raw blocks (```lang…```) and inline raw (`…`)
- Nested block comments (/* /* */ */) via depth-tracking external tokenizer
- Labels (<name>) and references (@name)
- Backslash escapes
Infrastructure changes:
- lezer-typst/typst.grammar — new Lezer grammar
- lezer-typst/tokens.mjs — external tokenizers for context-sensitive lexing
- scripts/lezer-latex/generate.mjs — Typst added to grammars array so the
existing lezer-latex:generate script (and Dockerfile step) compile it
- .gitignore — generated typst.mjs / typst.terms.mjs excluded from git
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Allow admin access to user PATs
* Tests for new screen in admin panel
* Adding error for invalid token and way to parse error for OAuth 2
* Git bridge handles expired PAT
* Script for alerting on close to expiry and expired git tokens
* Refactoring and simplifying
* Updating email templates to match agreed docs
* tweak to email subject to include Overleaf
* Allowing dry run in scripts and general tidy up
* removing redundant tests and dry running script
* Fixing CI errors
* Adding new tab to admin test expectation
* Address PR feedback on oauth2-server changes
- Replace ad-hoc overleafErrorCode prop with a TokenExpiredError subclass
- Collapse listTokens/listTokensForAdmin into a single hook
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Adding cron definitions for alerting on expiring git pat
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
GitOrigin-RevId: 69b9fd901a201592a580c69abe7bd7d603e85d3a
* Initial working version of library search
draft fetch allowing optional search param
draft debounce search
draft search bar
draft using for search
draft search params
draft data index creation
draft prefix-regex search
draft add fields only on search
draft index setup
draft search tests
draft search tests for extra params
draft using correct display value from bib entry for tokenization
* Library search handles diacritics
* Library styling and refreshing table data without
reloading table
* Updating mongo search query and creating migration
scripts for existing data
* Using Mongo query for sorting results
* Moving copied files into shared directory
* Addressing review comments
* Pulling changes from bibtex-search-token for consistency with migration
* Fixing lint
* Using mongo collation for handling case and diacritics in search queries
* Boosting citation keys with check for tokens
* Removing double foldLatinDigraphs call
* Matching figma designs for Library search component
* Adding cursor for paginated Library search results
* Re-fixing flash after searching library
* Unit test for cursor search
* Using same cursor object for search and get all results
* Data migration moved to manual script
GitOrigin-RevId: b7e6a1f07f775c8450dd97e7269cab3b68ca0eb3
* [docstore] add useSecondary flag to projectHasRanges
The rev-check for unarchiving always consults with the primary.
Two extra changes:
- Add a projection argument to peekDoc in order to skip lines download
from projectHasRanges.
- Add one retry to peekDoc to reduce chances of surfacing a rev-check
violation.
* [web] resync_projects: use the secondaries for all reads
* [web] add default value for useSecondary
* [docstore] add default value for useSecondary
* [k8s] docstore: set MONGO_HAS_SECONDARIES=true
GitOrigin-RevId: f15ec4fdc1cabe74c1eab87bec85f28d6f7a587d
* [history-v1] add endpoint for downloading latest zip
* [web] address review feedback
* [web] tests: do not overwrite db.projects.overleaf, extend it
* [web] set includeReferer flag from downloading zip
GitOrigin-RevId: e63e549f004230086f82eccf03b43fd62bde6071
* [github] code spaces: add extra-split-tests.json
* [github] address review feedback
* [github] add missing bind mount
* [monorepo] mount split test configs with :ro
* [monorepo] only mount backup/split-tests into web
GitOrigin-RevId: 6334a44599a2c8bb79bf1ad698e656c8bee3992b
Migrates the Overleaf monorepo package manager from npm (v11) to Yarn 4 (v4.9.1) using node-modules linker mode.
GitOrigin-RevId: 50d32ab01955c15e29679eff9e9e9cfb897fab2d
* [web] Add `update_group_policy` script
The script allows to update a policy for a subscription, or update
multiple policies via a file containing a list of subscription ids
GitOrigin-RevId: f3b3a6666eb62bb93d13c4c75d60b99f21f23db4
* [github] code spaces: add more standard users with a common password
* [github] code spaces: automatically share dev-env with Overleaf org
* [github] code spaces: merge split tests rather than replacing them
* [github] code spaces: ts-check the setup script in web
GitOrigin-RevId: 1a86ac7e0304d47e68290352f58a0e193eed1d77
* [web] serve site manifest as application/json
* [monorepo] bail out early from dev-env-setup when inside devcontainer
* [monorepo] move utils for waiting for services into shared bash script
* [monorepo] move domain related env-vars into dedicated .env file
The .env file will be overridden in GitHub Code Spaces.
* [monorepo] remove unused initial_mongo_setup Make target
It was previously used for the mongo replica set setup.
That setup is a docker init script now.
* [monorepo] add flag for silencing docker build output
* [monorepo] add integration for GitHub Code Spaces
* [web] make prettier happy
* [web] restore no use-credentials for manifest outside code spaces
* [monorepo] run make install as part of post-start again
GitOrigin-RevId: 8c2cc3b969b7e3e8f9c5dac778192c9412cbf4df
* accept taxInfoPending as tax id comparison point
* convert recurly unit amount to minor units correctly for no cent currencies
* Use original subscription currency in finalization check
* make it easier to spot migrated with tax info pending
---------
Co-authored-by: John Lees-Miller <jdleesmiller@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
GitOrigin-RevId: d3c74032bc06369fcf6bafb4cfaff85c49a3cfef
* add a script to detect drift between recurly and stripe
* only flag when Recurly updated since subscription created in Stripe
* add --comparison-date option instead of stripe subscription created time
* reduce API calls by checking Recurly update time earlier
GitOrigin-RevId: f0ecb5644f7445603da020a46e32622fcd2eebf4
* less strict address validation
* remove redundant name and address coalescing functions
* update missing name handling
GitOrigin-RevId: 4d15b16c840fa3edc50df6592d33f324dd26722c
* [saas-e2e] port history tests to old history
* [web] fix incremental compile from history for old history ids
* [saas-e2e] tweak return type
GitOrigin-RevId: 2c89b570647c292c720cd0d02b6188f66e3e3a69
* add script to query taxes
* Use full address to determine tax
* use all applied taxes
* include status in tax script output
* add warning to the script
* only normalize US and CA
* default to RoW if no countries specified
* use RateLimiter and pqueue
* only normalize US postal codes
---------
Co-authored-by: John Lees-Miller <jdleesmiller@gmail.com>
GitOrigin-RevId: 01ecc1b1461e955b6c6a798a3a568e5b5f6bbcdb
* [clsi] initial implementation of compile from history
* [clsi] copy changes
* [saas-e2e] extend test case with nested folder
* [saas-e2e] add test case for tracked changes
* [web] fix accumulating changes from multiple chunks
* [web] optimize size check for compile request payload
* [clsi] deduplicate globalBlobs
* [clsi] add validation for request body details
* [clsi] add metrics for compile from history
* [clsi] download binary files concurrently
* [clsi] skip download of empty file blob
* [clsi] break down e2e compile time metric by compileFromHistory
GitOrigin-RevId: 0dadef93e89d8a172c35cb130a1042d9d1bec42a
* handle billingInfo / account discrepancies
* fix rate limiting behaviour of fetchRecurlyActiveSubscription
* make force-invalid-tax aware of stripe tax validation errors
* clear taxInfoPending if vatNumber is now valid or not present on second runs
* re-instate pre-validation of tax ids in dry-run mode
* handle tax exemption
* fix tax pre-validation tests
* fix comment typos
* remove redundant hadConflict
* re-use normalizeName function
* throw if more than 1 active recurly subscription when resolving conflict
---------
Co-authored-by: Simon Gardner <simon.gardner@overleaf.com>
GitOrigin-RevId: a88cad87887bd9e043fb457091e999c1d555f546
claude