Commit Graph
223 Commits
Author SHA1 Message Date
0200ad7515 [monorepo] format ts files (#28678)
* Add ts files to format package script

* Apply changes to service package.json files

* [monorepo] only support plain .ts files in all the services

---------

Co-authored-by: Jakob Ackermann <jakob.ackermann@overleaf.com>
GitOrigin-RevId: 882cf274da86a0dbe1fff395441f6f2458c1405c
2025-09-25 08:05:30 +00:00
Eric Mc SweenandCopybot 98069966ba Merge pull request #28672 from overleaf/em-reapply-restore-optimization
Reapply file restore optimization

GitOrigin-RevId: c4479141d890e3e0af746f406a0186bf21f3d918
2025-09-25 08:05:25 +00:00
9b2fcbe960 [monorepo] optimize Jenkins build triggers for services (#28477)
Co-authored-by: Andrew Rumble <andrew.rumble@overleaf.com>
Co-authored-by: Mathias Jakobsen <mathias.jakobsen@overleaf.com>
GitOrigin-RevId: ea7d6c435d4a81a0f42ea9b608a0cc79087cdffd
2025-09-24 08:06:43 +00:00
Domagoj KriskovicandCopybot e9b1c63ed7 Revert "File restore optimization - using snapshot timestamp and file paths (#28546)"
This reverts commit 376a53e1f927cb56544e6782b47d80345655038c.

GitOrigin-RevId: 8e4ab1d4042fec8df1b62ab1c5418d873dc6f5eb
2025-09-24 08:06:06 +00:00
005eba7502 File restore optimization - using snapshot timestamp and file paths (#28546)
* Use snapshot timestamp, remove request for paths at version

* Add timestamp to core Snapshot object

* Avoid mutating function argument

* Explain assumption about editable files

* snapshot.toRaw() in getContentAtVersion

* fix project-history acceptance test

* fix history_v1 test

* fix web tests

* Include the snapshot timestamp in stored chunks

---------

Co-authored-by: Eric Mc Sween <5454374+emcsween@users.noreply.github.com>
GitOrigin-RevId: 376a53e1f927cb56544e6782b47d80345655038c
2025-09-24 08:06:02 +00:00
Andrew RumbleandCopybot 423c2a47e5 Merge pull request #28482 from overleaf/ar-move-notifications-changes-into-buildscripts
[monorepo] move notifications changes into buildscripts

GitOrigin-RevId: b5948702627d0bc87bffd3acc3d7c2ab9f4ac23d
2025-09-23 08:05:53 +00:00
Domagoj KriskovicandCopybot 484a01a173 Use history snapshot when doing file/project restore (#28502)
* Add getDocUpdaterCompatibleRanges utility function

* use history snapshot for file/project restore

* move overleaf-editor-core from devDependencies

GitOrigin-RevId: 62481a5304ada9d931e018418be3c0719bccf1f3
2025-09-19 08:07:58 +00:00
Jakob AckermannandCopybot a6e9a5c7e9 [monorepo] run prettier on Jenkinsfiles (#28565)
* [monorepo] consolidate .editorconfig files at the root

* [monorepo] run prettier on Jenkinsfiles

* [saas-e2e] increase timeout for XeTeX compile

GitOrigin-RevId: 48aa82f7c81611899837753ae92c7732998d4ca5
2025-09-19 08:07:49 +00:00
Miguel SerranoandCopybot f1dc6ec155 Upgrade aws-sdk to v3 (re-attempt) (#28410)
* Upgrade `aws-sdk` to v3

Also vendored nodemailer-ses-transport

* Moved default region to CE settings

* Ensure timeout is added to `requestHandler` and `region` is populated

* fix unsigned header in signed URL request

The x-amz-acl header is not needed when using the signed request,
as ACL are already defined when creating the signed URL in
PutObjectCommand constructor.

* Add default AWS region for latexqc service

* remove unnecessary region in compose files

* Use AWS_REGION=us-west-2 for history-v1

* prevent retries uploading streams with PutObjectCommand

* Remove AWS SDK JS message suppression

GitOrigin-RevId: 6fda6f02160023ffed76143397bbd965f86a9509
2025-09-18 08:05:22 +00:00
Miguel SerranoandCopybot 8948e40b22 Revert "Merge pull request #27679 from overleaf/msm-aws-sdk-upgrade" (#28151)
This reverts commit 4989ae920d8b7fd9e79623947b7c40bcc2e56d92.

GitOrigin-RevId: 541d95bfeaa7ce820e8af67f646f013fe4fe5d21
2025-08-28 08:05:52 +00:00
Miguel SerranoandCopybot 39381d808a Merge pull request #27679 from overleaf/msm-aws-sdk-upgrade
Upgrade `aws-sdk` to v3

GitOrigin-RevId: 4989ae920d8b7fd9e79623947b7c40bcc2e56d92
2025-08-28 08:05:36 +00:00
Eric Mc SweenandCopybot 8f0913fafe Merge pull request #27936 from overleaf/em-unit-tests-mongo
Make Mongo available to unit tests in all services

GitOrigin-RevId: b65bbb69883d5bba31d09802b89f35bdc523fe4d
2025-08-25 08:05:19 +00:00
Andrew RumbleandCopybot 6190005f9b [monorepo] Upgrade to node 22.18.0 (#27977)
* Upgrade to node 22.18.0

* Switch projects-data fixture to strip-types compatible form

GitOrigin-RevId: 4a7ec56f4a82395d2be8870526df0c679de38ab5
2025-08-22 08:05:17 +00:00
Jakob AckermannandCopybot 99feb94d4b [monorepo] add workaround for flaky mongo db init (#27833)
* [monorepo] add workaround for flaky mongo db init

Upstream issue: https://github.com/docker-library/mongo/issues/730
Upstream PR: https://github.com/docker-library/mongo/pull/731

* [server-ce] revert entrypoint changes

* [monorepo] fix typo in path

* [server-ce] run E2E tests from internal monorepo in cloudbuild

* [monorepo] avoid repeated changes to mongo entrypoint

GitOrigin-RevId: 617db9caee1600c8a506a7a147723a306e57ded3
2025-08-14 08:05:58 +00:00
Jakob AckermannandCopybot 836e8b5144 [monorepo] extend cleanup in Jenkins VMs (#27793)
* [monorepo] extend docker compose cleanup

* [web] cleanup docker images

Only retain the branch builds for dependencies.

* [server-ce] jenkins: remove build specific image

Retain the branch image as cache.

* [monorepo] jenkins: prune untagged docker images and docker build cache

* [monorepo] jenkins: remove temporary files from monorepo

* [monorepo] jenkins: remove containers from sandboxed compiles

GitOrigin-RevId: c847015986a060c5c84c39cd06236fd73bbf6f3a
2025-08-12 08:06:27 +00:00
Jakob AckermannandCopybot ac9c20e8d2 [monorepo] tweaks for Jenkins/CI (#27761)
* [packer] jenkins-worker: install htop and lsof

* [web] trim tailing space in Jenkinsfile

* [web] cleanup after finishing Jenkins pipeline

* [server-ce] test: cleanup after finishing Jenkins pipeline

* [saas-e2e] add timeout to waiting for rail/web startup

* [monorepo] add 60s timeout for waiting on mongo to startup

* [saas-e2e] cleanup after finishing Jenkins pipeline

GitOrigin-RevId: 9954ab17b4db6a95c444e8e0b55cebb8d5daae59
2025-08-11 08:06:31 +00:00
Alf EatonandCopybot 03a4bbb96b Remove "version" from docker-compose files (#27494)
GitOrigin-RevId: 8d9a0e58e3a9c6c4296e4585115d1b259e26a6cd
2025-07-31 08:05:10 +00:00
Christopher HoskinandCopybot 7ecee2e0aa Merge pull request #27255 from overleaf/revert-27252-revert-26843-csh-issue-26608-mongo8-dev-ci
Revert "Revert "Upgrade the dev environment and CI to mongo 8""

GitOrigin-RevId: 5074b012504e65240017f1fde9b0d8d04c7b8b61
2025-07-22 08:05:25 +00:00
Christopher HoskinandCopybot f4dc8f7ebc Merge pull request #27252 from overleaf/revert-26843-csh-issue-26608-mongo8-dev-ci
Revert "Upgrade the dev environment and CI to mongo 8"

GitOrigin-RevId: f2145812a5c1cf8d3d3ac31c76cc4aed4ea9d46d
2025-07-21 08:05:01 +00:00
Christopher HoskinandCopybot 456f751a18 Merge pull request #26843 from overleaf/csh-issue-26608-mongo8-dev-ci
Upgrade the dev environment and CI to mongo 8

GitOrigin-RevId: 5fc33863094ea2bdb9235372efdc7d0cb492c34a
2025-07-21 08:04:57 +00:00
Andrew RumbleandCopybot 1a1f283245 Update node to 22.17.0
GitOrigin-RevId: 7a15f7a0e95e3331c589955aca1fbc5dfb13a568
2025-07-09 08:06:01 +00:00
Jakob AckermannandCopybot adf399fb95 [project-history] add support for resync of history-ot ranges (#26475)
* [project-history] add support for resync of history-ot ranges

* [project-history] avoid compressing sharejs and history-ot upgrades

* [document-updater] improve error message of some assertions

... by migrating the assertions like this:
```diff
-stub.calledWith().should.equal(true)
+stub.should.have.been.calledWith()
```
```diff
-stub.called.should.equal(false)
+stub.should.not.have.been.called
```

* [document-updater] move content field in resyncDocContent

* [document-updater] add support for resync of history-ot ranges

GitOrigin-RevId: e6104686a26934a5f25a8f095cbe00c163fbbaa7
2025-06-19 08:05:18 +00:00
Jakob AckermannandCopybot 0c2f79b0b8 [misc] make log level configurable when running tests in ESM services (#26465)
GitOrigin-RevId: 153fa99731b2fbff6dfd5edc80ce3e8acf9fc5ce
2025-06-17 08:06:04 +00:00
Jakob AckermannandCopybot 7e9a33841d [project-history] migrate retry_failures script to ESM (#26444)
GitOrigin-RevId: 425e0e6c0a61ef7f8c8c3e07f2d16089f594eb32
2025-06-16 08:07:08 +00:00
Jakob AckermannandCopybot 8b937c91f4 [project-history] fix unit tests after merging multiple PRs (#26442)
GitOrigin-RevId: 9ccbadaa807da2f20010930477724e9370bf103e
2025-06-16 08:06:54 +00:00
Jakob AckermannandCopybot 740b1d3f50 [project-history] add script for retrying failures (#26427)
GitOrigin-RevId: 265cec9719825613de01d7f476b20203bbff8d7f
2025-06-16 08:06:41 +00:00
Jakob AckermannandCopybot 0aa56fbe2c [project-history] fix callback signature when processing no updates (#26420)
* [project-history] fix tests and cover callback for processing updates

The before setup was hiding that some tests were not doing what the
assertions were expecting.

* [project-history] fix callback signature when processing no updates

GitOrigin-RevId: 4fa14d47b9a1afd998316b0c9024d49760785a47
2025-06-16 08:06:36 +00:00
Jakob AckermannandCopybot 6f516b25af [project-history] add metrics for compression of updates (#26307)
* [project-history] add metrics for compression of updates

* [project-history] sample compression metrics

GitOrigin-RevId: 1cd67dc4ec7b44285afb436c62392b464f007f97
2025-06-16 08:06:28 +00:00
Jakob AckermannandCopybot 8b91b3b749 [misc] wait for mongo before running acceptance tests (#26374)
GitOrigin-RevId: 1fde30c2b630a51f5eda9d318ac721a81cc23607
2025-06-13 08:05:08 +00:00
Jakob AckermannandCopybot 4aaf411cd2 [misc] improve logging in history system (#26086)
* [project-history] tag all the errors

* [history-v1] log warnings for unexpected cases

GitOrigin-RevId: 3189fa487eee88985688ff990ec101daad0d13b1
2025-06-04 08:05:50 +00:00
Christopher HoskinandCopybot fe64856be7 Merge pull request #26021 from overleaf/csh-issue-25976-dev-env-ci
Upgrade to Redis 7.4 in dev and CI

GitOrigin-RevId: 068e54899bf50a247fedd0243d66f1545bc7cf01
2025-06-02 08:05:21 +00:00
64984ee86a [history-ot] flush history for projects with short queues ASAP (#25776)
* [document-updater] flush history for projects with short queues ASAP

* [k8s] document-updater: enable short history queue for history-ot demo

* [project-history] flush history for projects with short queues ASAP

* [project-history] wait for mongo before running acceptance tests

* [k8s] project-history: enable short history queue for history-ot demo

* [project-history] change wait-for-mongo step in tests

Co-authored-by: Eric Mc Sween <eric.mcsween@overleaf.com>

---------

Co-authored-by: Eric Mc Sween <eric.mcsween@overleaf.com>
GitOrigin-RevId: 3e989c409e8e9887655b35f2659ce0829e61b357
2025-05-23 08:05:35 +00:00
Christopher HoskinandCopybot 841e32bb64 Merge pull request #25809 from overleaf/csh-issue-22789-ci-mongo-7
Upgrade the dev environment and CI to mongo 7

GitOrigin-RevId: da02881d142d21be47dac7bd2af74520ba8664cd
2025-05-22 08:45:24 +00:00
Antoine ClausseandCopybot 436dcc977f Update Node to 22.15.1 (#25785)
GitOrigin-RevId: 52428d2d7e67c3135a1604fa487dd142aa08bf15
2025-05-22 08:07:42 +00:00
Antoine ClausseandCopybot b667cef262 Revert "Update defaultHighWaterMark to 64KiB (Node 22's default) (#25522)" (#25789)
This reverts commit 19d731abf683066654027de3a4f9ac0b8916f22c.

GitOrigin-RevId: eb7c45ab45e02054601b607a4bfeb432424a1837
2025-05-22 08:07:38 +00:00
Jakob AckermannandCopybot 1e6b13f9d5 [history-ot] rename remaining history-v1-ot references to history-ot (#25428)
* [history-ot] rename remaining history-v1-ot references to history-ot

* [web] rename History-v1 OT -> History OT in admin panel

* [web] rename OT Migration -> History OT Migration in admin panel

GitOrigin-RevId: 103ce816d5320d6379d51009cdc08b8a71aa48e6
2025-05-20 08:06:46 +00:00
Antoine ClausseandCopybot 67ab5a749a Update Node to 22.15.0 (#24699)
* Run `bin/update_node 20.18.2 22.15.0`

* Remove expects on `fetchMock.callHistory.done()` to fix tests: are they necessary?

* Set node version to `22.x` in linked-url-proxy

* Increase test timeout to 30s in `github-sync`, Add waiting steps

* Define `navigator.onLine` in tests setup

GitOrigin-RevId: 75eb556e9f51b665e57497a0879b6915d14069ce
2025-05-16 08:06:26 +00:00
Antoine ClausseandCopybot aa002369cb Update defaultHighWaterMark to 64KiB (Node 22's default) (#25522)
* Set defaultHighWaterMark to 16KiB

This is already the default in Node 20

* Set defaultHighWaterMark to 64KiB

Per https://github.com/overleaf/internal/pull/25522#issuecomment-2872035192

GitOrigin-RevId: 19d731abf683066654027de3a4f9ac0b8916f22c
2025-05-14 08:05:16 +00:00
Brian GoughandCopybot 6eada92966 Merge pull request #25406 from overleaf/em-content-hash-validation-resync
Resync project when content hash validation fails

GitOrigin-RevId: ea9b5a78f89c55276fd67835bc262717bc778e92
2025-05-09 08:05:12 +00:00
e8b5ee2ff9 [history-ot] initial implementation of using doc-level history-ot (#25054)
* [history-v1-ot] initial implementation of using doc-level history-v1-ot

* [web] fix advancing of the otMigrationStage

Use 'nextStage' for the user provided, desired stage when advancing.

Co-authored-by: Brian Gough <brian.gough@overleaf.com>

* [document-updater] document size check in editor-core

* [history-ot] rename history-v1-ot to history-ot and add types

* [history-ot] apply review feedback

- remove extra !!
- merge variable assignment when processing diff-match-match output
- add helper function for getting docstore lines view of StringFileData

Co-authored-by: Alf Eaton <alf.eaton@overleaf.com>

* Revert "[document-updater] add safe rollback point for history-ot (#25283)"

This reverts commit d7230dd14a379a27d2c6ab03a006463a18979d06

Signed-off-by: Jakob Ackermann <jakob.ackermann@overleaf.com>

---------

Signed-off-by: Jakob Ackermann <jakob.ackermann@overleaf.com>
Co-authored-by: Brian Gough <brian.gough@overleaf.com>
Co-authored-by: Alf Eaton <alf.eaton@overleaf.com>
GitOrigin-RevId: 89c497782adb0427635d50d02263d6f535b12481
2025-05-08 08:05:44 +00:00
Jakob AckermannandCopybot 07b2255426 [misc] cleanup .dockerignore and .gitignore files (#25312)
- Remove settings ignore, they are inconsistent and break local prettier
- Remove .dockerignore files, only root ignore file is used
- Move .idea/.run/*.swp/coverage to root
- Remove .npmrc entries, we are no longer writing the rc file
- Remove node_modules/.DS_Store, is contained in root
- Remove cruft

GitOrigin-RevId: 3025fd5acaef343312f55149466c638e759a6e1f
2025-05-07 08:05:24 +00:00
Eric Mc SweenandCopybot fb03fe4d26 Merge pull request #24776 from overleaf/em-project-history-unused-endpoints
Remove unused endpoints in project-history

GitOrigin-RevId: 2940c1c1973177e3200cb78884b307f708fd88c3
2025-04-14 08:05:08 +00:00
Brian GoughandCopybot fe8d6392d5 Merge pull request #24793 from overleaf/bg-update-build-scripts-to-include-history-redis
update build scripts to include HISTORY_REDIS_HOST

GitOrigin-RevId: da0f317c80401067c0f4aa772196cb2f24849b8e
2025-04-11 08:06:03 +00:00
Eric Mc SweenandCopybot 20c7f14b3c Merge pull request #24461 from overleaf/em-non-editable-file-in-history
Graceful fallback for getRangesSnapshot()

GitOrigin-RevId: 2f0e70dbcdd38de87d0b1c7590f7d8a5a7112937
2025-03-26 09:05:21 +00:00
Eric Mc SweenandCopybot 374acf8119 Merge pull request #24462 from overleaf/em-resync-error
Fix error reporting on resyncs in project-history

GitOrigin-RevId: 9e8a454f746c2b4db7c28fafcd52e50935af5957
2025-03-25 09:05:12 +00:00
Brian GoughandCopybot 31b57e2991 Merge pull request #24367 from overleaf/bg-add-new-project-history-flush-script
add new project history flush script

GitOrigin-RevId: 4d6f3be1ada7191334b934e34e1c9eac59a816d0
2025-03-24 10:46:34 +00:00
Eric Mc SweenandCopybot 35dc7faab6 Merge pull request #24224 from overleaf/em-resync-on-flush-failure
Immediately attempt a resync when a history flush fails

GitOrigin-RevId: 098a0a7edf55c0ed47c48e0a3c080e3562cdceaa
2025-03-18 09:05:36 +00:00
f6bd485863 [misc] align initializing of mongodb replica set (#24287)
* [misc] align initializing of mongodb replica set

* [misc] fix volumes for mono container in dev-env

* Remove duplicate volumes key

---------

Co-authored-by: Alf Eaton <alf.eaton@overleaf.com>
GitOrigin-RevId: 0119c42ee8b13a0fca9373e40a27fc5b2a52d73b
2025-03-17 09:04:58 +00:00
Andrew RumbleandCopybot 767591973c Merge pull request #23905 from overleaf/ar-mocha-11
[all] Bump mocha everywhere

GitOrigin-RevId: bf5587047f9ceb88872cbd356c2029717bb8ca9d
2025-02-28 09:05:40 +00:00
Antoine ClausseandCopybot fe03d8d7fd Add shellcheck to web, server-pro, server-ce (#23823)
* Update shellcheck files to include non ".sh" files having the sh shebang

* Add shellcheck to root Makefile

* `make shellcheck_fix`

* Add shellcheck to server-ce and server-pro

* Exclude SC1091 (Not following) in SP/CE

* Fix errors in SP

* Fix errors in CE

* Update root shellcheck to ignore all failing rules. We can later remove rules one by one.

* Add shellcheck to web

* Add shellcheck step for server-ce and server-pro in cloudbuild.yaml

* Revert "Add shellcheck to root Makefile"

This reverts commit e0fa56f2

* Revert "`make shellcheck_fix`"

This reverts commit eb179245c109a9e742a7fdeeb75a4bdd03963587.

* `make shellcheck_fix` in server-ce

* Fix: Use $(...) notation instead of legacy backticked

```
In init_scripts/100_set_docker_host_ipaddress.sh line 5:
echo "`route -n | awk '/UG[ \t]/{print $2}'` dockerhost" >> /etc/hosts
      ^-- SC2006 (style): Use $(...) notation instead of legacy backticks `...`.

Did you mean:
echo "$(route -n | awk '/UG[ \t]/{print $2}') dockerhost" >> /etc/hosts

For more information:
  https://www.shellcheck.net/wiki/SC2006 -- Use $(...) notation instead of le...
```

* `make shellcheck_fix` in web

* Manual corrections on autofix

* Update SHELLCHECK_FILES to Perl regex so it matches at start of file

GitOrigin-RevId: 815d53f319a3792efa22703beb75570de5977450
2025-02-26 09:05:02 +00:00