8b73bb9f13
Merge branch 'master' into bg-filter-track-changes-updates
2019-11-22 10:41:33 +00:00
b7055eecee
add metrics for history flushes
2019-11-22 09:14:32 +00:00
65cf4cf7c7
make flush to track-changes failsafe
2019-11-21 14:58:35 +00:00
dcd7649bad
filter track-changes updates for projects using project-history
2019-11-19 10:02:56 +00:00
d82b180b76
avoid project history queues building up with deferred flush
2019-10-03 04:05:24 +01:00
c1454bc4ac
Merge pull request #92 from overleaf/bg-flush-queue-prod-fixes
...
add continuous background flush
2019-10-02 13:11:00 +01:00
0c14b7d2f8
add comment about background flush limit
2019-10-01 15:06:01 +01:00
2845b23b70
add smoothing of delete spikes
2019-10-01 15:01:53 +01:00
2c22a60052
add random jitter to cutoff time
2019-10-01 15:01:20 +01:00
a32495d2b4
make background flush more adaptive
2019-10-01 14:09:41 +01:00
73b4262186
add continuous background flush
2019-09-30 16:05:53 +01:00
33fadf51c1
fix getDocTimestamps for multiple docs
2019-09-30 13:50:25 +01:00
260923f291
keep flushQueuedProjects in the foreground
2019-09-27 10:46:24 +01:00
7561e05660
check timestamps array length
2019-09-27 10:39:56 +01:00
b7f3b848af
remove unused dryRun option
...
Co-Authored-By: Jakob Ackermann <das7pad@outlook.com >
2019-09-26 15:50:55 +01:00
3bc176259b
fix log line
2019-09-26 15:46:54 +01:00
8cdc8c410a
fix error logging
2019-09-26 15:46:45 +01:00
fc62abfcfa
run flush of queued projects in the background
2019-09-26 15:46:14 +01:00
ba35c73cb6
add comment about ZPOPMIN
2019-09-26 15:18:10 +01:00
a709a0adaa
for simplicity keep the cutoff time the same while flushing the queue
2019-09-26 15:05:38 +01:00
eae4b352ca
remove unnecessary check
2019-09-26 14:59:03 +01:00
b49621b3e9
add comments
2019-09-26 10:14:49 +01:00
83dd43b809
add metric for queue length
2019-09-25 17:04:36 +01:00
f6b2ac7360
queue deletes for deferred processing
2019-09-25 16:42:49 +01:00
912a3a7753
remove redis server-side hashing for performance
...
we still compute the document hash in node, and check it on retrieval
but we don't check the hash at the point of writing it in redis which
was previously done with a redis Lua script.
2019-09-09 15:27:58 +01:00
aa15a76059
added log lines for all project flushing
2019-08-30 07:38:53 +01:00
0ae838dd2d
add logger into project flusher
2019-08-29 20:36:00 +01:00
a76e0dca88
skip history flush when project is cleared by realtime shutdown
...
history is flushed by a background cron job anyway
2019-08-15 09:51:16 +01:00
6721b904a7
Merge pull request #82 from overleaf/bg-mongo-health-check
...
add a combined health check for mongo and redis
2019-08-08 14:16:38 +01:00
06444d2cc4
Improve/fix serializers for update logging ( #80 )
...
* Improve/fix serializers for update logging
2019-08-08 14:10:54 +01:00
40f6494b19
add a combined health check for mongo and redis
2019-08-07 16:25:23 +01:00
df9ca8b272
Add serializer to print only length of large fields in production
2019-07-31 16:42:28 +01:00
618880f99d
remove unnecessary check for doc_id
2019-07-24 16:57:43 +01:00
c9ccf62d71
support per-doc pubsub channels
2019-07-22 12:20:06 +01:00
97487a077e
fix cluster/sentinel connection with real-time
2019-07-10 09:42:05 +01:00
06ad0f7acd
Merge pull request #75 from overleaf/ho-pubsub-connection
...
Remove real time redis connection and consolidate on pubsub
2019-07-08 13:58:41 +01:00
3b3b2da0f5
add pubsub redis connection and remove real time redis connection
2019-07-04 13:34:31 +01:00
16fb297043
Revert "skip hash check when non-BMP characters replaced"
2019-06-27 11:39:45 +01:00
f37860599d
skip hash check when non-BMP characters replaced
2019-06-25 16:36:10 +01:00
fdef197271
Merge branch 'master' into ho-detailed-flush-status
2019-06-13 14:33:22 +01:00
d9a737f97c
return failed and succesfully flushed projects when flushing everything
2019-06-13 14:21:38 +01:00
e8dd1aae9c
Merge pull request #70 from overleaf/bg-metric-for-invalid-hash
...
add metric for invalid hash and other sharejs errors
2019-06-12 13:50:34 +01:00
d50b93df2f
add metric for invalid hash and other sharejs errors
2019-06-11 16:48:06 +01:00
966478cac4
fix hash check to use 'v' field instead of version
2019-06-11 14:11:46 +01:00
e95059f98e
handle non-urgent flushes in background
2019-06-03 10:01:10 +01:00
0bbfa7de27
Merge branch 'master' into bg-downgrade-delete-component-error
2019-05-08 09:07:02 +01:00
27a8248196
convert "Delete component" errors into warnings
2019-05-07 16:55:17 +01:00
daca83a057
add dryRun option to flush all projects
2019-05-02 16:54:22 +01:00
d5d1736a5e
adds /flush_all_projects project
2019-05-02 16:43:35 +01:00
c1c23e4bee
record last author id on document flush
...
This is a multi-steps process:
* get a update's `user_id` from the metadata
* store the `user_id` (`lastUpdatedBy`) and current date (`lastUpdatedAt`) for
the document in Redis on every updates
* fetch `lastUpdatedAt` and `lastUpdatedBy` from Redis on document flush
* send the data to web to be persisted in Mongo
2019-05-02 11:10:02 +01:00
Brian Gough