After a successful compile, web service calls a new CLSI endpoint
(GET /project/:id/user/:uid/build/:bid/thumbnail) which runs pdftocairo
page-1 to a 190px-wide JPEG using the existing thumbnail preset. The
JPEG is stored in Redis (90-day TTL, overwritten on next compile) by
the new ThumbnailManager.
GET /project/:Project_id/thumbnail serves the cached JPEG to authenticated
users, returning 404 when no thumbnail exists. Project cards in the
Lumière grid show the image overlaying the coloured gradient tile; if
the image 404s (project never compiled or cache expired) the onerror
handler hides it and the gradient + initial letter shows through.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* [monorepo] consolidate clsi-lb host/ip env-vars
Target env-var is CLSI_LB_HOST. Keep CLSI_LB_IP populated for a week.
* [clsi] initial version of /convert/pdf-to-jpeg
* [rails] use fake-secrets in CI and Codespaces
* [rails] adapt tests for using clsi to convert PDFs to image
* [rails] add rake task for comparing clsi conversion with transloadit
* [clsi] double check that output.jpg is a regular file
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
* [clsi] fix composing basename
* [monorepo] fix clsi-lb host env-var post merge
* [monorepo] sort dev-environment.env hosts
* [rails] use local pdf file rather than downloading it again
Download from the old renderer code path still. It's dead code.
* [terraform] clsi: enable pdf to jpg conversion
---------
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
GitOrigin-RevId: 5ecaa8559d299486340bb3961f06b29f7c4dfcca
* using CLSI logic for fetching the project contents and skip the .zip export
* Use unique conversion directory for project-to-docx export to avoid corrupting the shared compile
directory when a compile runs concurrently
* Remove X-Accel-Buffering header — not needed as CLSI does not run behind nginx
* moving log before sending the data
* Return CLSI stream directly instead of buffering to disk on web
Previously convertProjectToDocx wrote the CLSI response to a temp file
on disk, then the controller read it back to stream to the client.
Now the stream is returned directly and piped to the response,
avoiding unnecessary disk I/O on the web server.
* Use href redirect for docx export instead of fetching blob into memory
* making functions and files more generic so they can be used in future for other documents exports as well
* adding export-docx split test
* adding unit tests
* adding cypress E2E test
* format:fix
* renaming the route to download from convert
* adding new icon for export docx button
* format:fix
* remove unused showExportDocumentErrorToast export and adding guard against invalid Content-Length header from CLSI
* format:fix
* refactor(clsi): move promisify(parse) into RequestParser
* refactor: generic conversion endpoint with type as route
param
* refactor: use type→extension map for validated conversion types
* refactor(clsi): remove --standalone flag and fix rejection test
* fixing the href in cypress test
* renaming function
* adding type to Metrics.inc
* fix: rename exportProjectDocument, add WithLock wrapper and metrics type label
* format:fix
* fix: hide docx export from anonymous users and add WithLock wrapper
* format fix
* remove redundant Content-Length validation from DocumentConversionManager
* format:fix
* removing trailing icon
GitOrigin-RevId: e9764fefac2c4b625d23be9e942ea4a8b283c70d
* [monorepo] remove PII and variables from error messages
Exclusions:
- scripts
- tests
- fuzzing
- SplitTestManager (messages are sent to admin frontend)
- Group setup (we may want an error per unique tuple)
- sharejs (unused types; text type errors are shadowed already)
- history-v1 error messages that are used by the ErrorRecorder
- errors that flag issues with configuration/call signatures
I've used these search terms for finding unwanted error messages:
- new Error(`
- new Error\(\n\s+` (regex search)
- new OError(`
- new OError\(\n\s+` (regex search)
* [web] throw NotFoundError from ProjectLocator
* [github-sync] fix OError.tag call in script
Co-authored-by: Jessica Lawshe <jessica.lawshe@overleaf.com>
* [templates] revert changes to test client
---------
Co-authored-by: Jessica Lawshe <jessica.lawshe@overleaf.com>
GitOrigin-RevId: 736857a4fc5d9bfb0f8cb03e0f004eda87e5a220
* [clsi] shed load when detecting out-of-disk condition
* [clsi] mark VM as unhealthy when detecting of-of-disk condition
GitOrigin-RevId: 25cda6785c0d973f50ec6206bee389804f35917e
* [clsi-cache] initial revision of the clsi-cache service
* [clsi] send output files to clsi-cache and import from clsi-cache
* [web] pass editorId to clsi
* [web] clear clsi-cache when clearing clsi cache
* [web] add split-tests for controlling clsi-cache rollout
* [web] populate clsi-cache when cloning/creating project from template
* [clsi-cache] produce less noise when populating cache hits 404
* [clsi-cache] push docker image to AR
* [clsi-cache] push docker image to AR
* [clsi-cache] allow compileGroup in job payload
* [clsi-cache] set X-Zone header from latest endpoint
* [clsi-cache] use method POST for /enqueue endpoint
* [web] populate clsi-cache in zone b with template data
* [clsi-cache] limit number of editors per project/user folder to 10
* [web] clone: populate the clsi-cache unless the TeXLive release changed
* [clsi-cache] keep user folder when clearing cache as anonymous user
* [clsi] download old output.tar.gz when synctex finds empty compile dir
* [web] fix lint
* [clsi-cache] multi-zonal lookup of single build output
* [clsi-cache] add more validation and limits
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
* [clsi] do not include clsi-cache tar-ball in output.zip
* [clsi-cache] fix reference after remaining constant
Co-authored-by: Alf Eaton <alf.eaton@overleaf.com>
* [web] consolidate validation of filename into ClsiCacheHandler
* [clsi-cache] extend metrics and event tracking
- break down most of the clsi metrics by label
- compile=initial - new compile dir without previous output files
- compile=recompile - recompile in existing compile dir
- compile=from-cache - compile using previous clsi-cache
- extend segmentation on compile-result-backend event
- isInitialCompile=true - found new compile dir at start of request
- restoredClsiCache=true - restored compile dir from clsi-cache
* [clsi] rename metrics labels for download of clsi-cache
This is in preparation for synctex changes.
* [clsi] use constant for limit of entries in output.tar.gz
Co-authored-by: Eric Mc Sween <eric.mcsween@overleaf.com>
* [clsi-cache] fix cloning of project cache
---------
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
Co-authored-by: Alf Eaton <alf.eaton@overleaf.com>
Co-authored-by: Eric Mc Sween <eric.mcsween@overleaf.com>
GitOrigin-RevId: 4901a65497af13be1549af7f38ceee3188fcf881
* [clsi] make error copying output files a fatal compile error
Co-authored-by: Rebeka <rebeka.dekany@overleaf.com>
* [clsi] remove unused endpoints for downloading files from compileDir
Co-authored-by: Rebeka <rebeka.dekany@overleaf.com>
* [clsi] avoid useless clone of output files
* [clsi] add test for output files when compile did not produce a PDF
---------
Co-authored-by: Rebeka <rebeka.dekany@overleaf.com>
GitOrigin-RevId: cb998b99b4d96cb48ddd70987958f614ad3b40fc
* Add `unicorn/prefer-node-protocol`
* Fix `unicorn/prefer-node-protocol` ESLint errors
* Run `npm run format:fix`
* Add sandboxed-module sourceTransformers in mocha setups
Fix `no such file or directory, open 'node:fs'` in `sandboxed-module`
* Remove `node:` in the SandboxedModule requires
* Fix new linting errors with `node:`
GitOrigin-RevId: 68f6e31e2191fcff4cb8058dd0a6914c14f59926
When the box is over 100% capacity start to shed load.
Use 5 min load average to make the system less likely to start moving
projects off after a temporary cpu spike
* wip generate directory for hash content
* cleanup, remove console logging
* add content caching module
* Return PDF stream ranges with compile response
* Return the PDF file size in the compile response
* PDF range endpoint
* [misc] WIP: pdf caching: preserve the m-time on static content files
* [misc] WIP: pdf caching: improve browser caching, emit caching headers
* [misc] WIP: pdf caching: do not emit very small chunks <1kB
* [misc] keep up with moving output files into a separate directory
* [OutputCacheManager] add global feature flag for enabling pdf caching
* [misc] add contentId into the URL for protecting PDF stream contents
* [misc] support PDF stream caching for anonymous users
* [misc] add per-request feature flag for enabling PDF stream caching
* [misc] enable pdf caching in CI and emit metrics at the end of run
* [misc] expose compile stats and timings to the frontend
* [misc] log an error in case saving output files fails
* [misc] add metrics for pdf bandwidth and pdf caching performance
* [misc] add a dark mode to the pdf caching for computing ranges only
* [misc] move pdf caching metrics into ContentCacheMetrics
* [misc] add a config option for the min chunk size of pdf ranges
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
Co-authored-by: Eric Mc Sween <eric.mcsween@overleaf.com>
