Gauthier Zirnhelt
9987166b2b
## Summary This PR ensures that the `LifecycleWorker` yields at least once to the Tokio scheduler in between each batch of 100 objects. ## Problem being solved I'm administrating a Garage cluster which has been experiencing timeouts on all endpoints while the lifecycle worker is running at midnight UTC : `Ping timeout` error messages and even requests eventually failing due to `Could not reach quorum ...`. I have found that this happens while the lifecycle worker is working on a big bucket (containing millions of objects) with a lifecycle rule that applies to very few objects. The `process_object()` function does not hit any `await`: - `last_bucket` is always the same, so the `bucket_table` is not read asynchronously - no transaction is made on the `object_table` because my lifecycle rule (almost) never applies to any object The first commit in this PR adds an executable which reproduces the problem that I've been experiencing in a self-contained way : the lifecycle worker starves the Tokio scheduler so much that no other task is able to run (or very rarely). To run it : `cargo run -p garage_model --bin lifecycle-starvation-test`. This commit can be dropped post-review, as it's only useful to demonstrate the starvation. The error messages completely stopped after adding the extra yield to the nodes of my cluster. The duration of the lifecycle worker task does not appear to have changed at all from what I can see (looking at the timestamps produced either by the self-contained binary or by each of my nodes with the `Lifecycle worker finished` message). ## Note An other potential fix would have been to force the `WorkerProcessor` to yield before re-enqueuing a busy task, but this would have affected all Garage workers even though it's only the `LifecycleWorker` being uncooperative. Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/1396 Reviewed-by: Alex <lx@deuxfleurs.fr> Co-authored-by: Gauthier Zirnhelt <gauthier.zirnhelt@insimo.fr> Co-committed-by: Gauthier Zirnhelt <gauthier.zirnhelt@insimo.fr>
Garage 
[ Website and documentation | Binary releases | Git repository | Matrix channel ]
Garage is an S3-compatible distributed object storage service designed for self-hosting at a small-to-medium scale.
Garage is designed for storage clusters composed of nodes running at different physical locations, in order to easily provide a storage service that replicates data at these different locations and stays available even when some servers are unreachable. Garage also focuses on being lightweight, easy to operate, and highly resilient to machine failures.
Garage is built by Deuxfleurs, an experimental small-scale self hosted service provider, which has been using it in production since its first release in 2020.
Learn more on our dedicated documentation pages:
Garage is entirely free software released under the terms of the AGPLv3.