Add a link to the manual install procedure, use a
quote to make the manual install path more visible
and mention the --default-access-key option.
Closes: #1446
Rendered output:
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/1496
PostObject (presigned POST) returns `400 InvalidHeaderValue` when the upload involves a non-ASCII filename. The same key uploads fine vie PUT, as the issue #1489 noted.
The problem was that `handle_post_object` stores the form field values in an `http::HeaderMap`. values go in through `HeaderValue::from_str` but are read back with the strict `HeaderValue::to_str()` (ASCII-only) which fails on any UTF-8 value.
This is the same problem fixed in eab2b81b for `x-amz-meta-*` headers, and the fix is the same:
`std::str::from_utf8(value.as_bytes())` instead of `to_str()`. The `key` field in `post_object.rs` and the standard headers (`content-disposition` etc.) in `extract_metadata_headers`.
Added integration tests for PostObject (there were none): UTF-8 key, `${filename}` substitution, and UTF-8 `Content-Disposition` metadata. The substitution test passes even without the fix, proving that the filename path was never broken, only the form-params round-trip.
Co-authored-by: Mathew Storm <mathew@stormdevelopments.ca>
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/1492
Reviewed-by: trinity-1686a <trinity-1686a@noreply.localhost>
Documenting the structure of a team can be helpful to integrate newcomers, avoid some conflicts and to accompany the evolution of that structure as a FOSS project grows.
This is an attempt to do that for garage, after discussions with multiple maintainers. The aim of this text is to document the status quo, offering a start for further incremental updates.
I propose to put it as `GOVERNANCE.md` file because this naming has emerged as a sort of standard in FOSS projects.
I would be happy to help facilitate such updates if there is interest (while being aware that project members have other things on their plates and that governance work can be taxing).
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/1493
The rebalance op links to the multi-hdd page, but the multi-hdd page
does not refer to the rebalance op description and only states that an
operator can launch a repair procedure without stating which procedure.
Add a link to the rebalance repair procedure in the multi-hdd page to
make it easier to find what procedure to run for rebalance.
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/1491
save_peer_list() ran unconditionally every 60s from the discovery
loop, writing the same data to disk even when the peer set hadn't
changed. Compare the newly encoded peer list against the bytes
already on disk and skip the write when they match.
Fixes#1457
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/1488
Reviewed-by: Alex <lx@deuxfleurs.fr>
This performs synchronous repair-on-read for K2V reads:
- uses the new get_*_monotonic operations in the table module
- implements repair-on-read for the K2V-specific poll operations
`parse_duration` is no longer maintained upstream; `fundu-system` seems like the best option to provide most of the functionality provided by parse_duration and minimize deps.
Fixes#1246
Dep diffstat: +3 -6
## Caveats
- I've done basically no testing of this PR beyond `cargo test`
- See my comment in #1246 for regression risk; this is a breaking change. Is there a document I should update to make note of this? Should I rebase this for `next-v3` instead?
- In theory CVE-2021-29932 is fixed by this PR as `fundu-systemd` doesn't support exponents, but I've done no verification that this is really the case beyond the info in #1246
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/1468
Reviewed-by: Alex <lx@deuxfleurs.fr>
If the files configured in `api_token_file` or `rpc_secret_file` are not found, garage exits with no details (at least not enough) of what went wrong:
2026-06-14T23:40:34.517429Z INFO garage::server: Loading configuration from tmp/config1.toml...
Error: IO error: No such file or directory (os error 2)
This add the kind of file and the file path being read, to the returned error message:
2026-06-14T23:41:41.136213Z INFO garage::server: Loading configuration from tmp/config1.toml...
Error: Failed to read secret file /run/secrets/rpc_secret: No such file or directory (os error 2)
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/1475
Reviewed-by: Alex <lx@deuxfleurs.fr>
The OPTIONS placeholder for buckets without a resolvable global alias returns` Access-Control-Allow-Origin: *` and `Access-Control-Allow-Methods: *` but omits `Access-Control-Allow-Headers`.
Bug verified against Garage v2.2.0 with a local-aliased bucket: OPTIONS placeholder doesn't have `Access-Control-Allow-Headers`, causes the browser to reject signed PUT preflights
The current placeholder fails open for unsigned simple requests but blocks every signed request, undermining the design intent flagged in the FIXME:
```rs
// We take the permissive approach of allowing everything,
// because we don't want to prevent web apps that use
// local bucket names from making API calls.
```
Adds `Access-Control-Allow-Headers: *` so the permissive default is actually permissive for the request shapes that exist in practice.
Refs #258. Does not address the broader FIXME (CORS rule resolution for local-aliased buckets); the placeholder approach is preserved.
All tests are fine locally:
```bash
▲ ~/opensource/garage cargo test -p garage_api_common cors::
running 5 tests
test cors::tests::preflight_with_single_allowed_origin_returns_request_origin ... ok
test cors::tests::preflight_with_multiple_allowed_origins_reflects_request_origin ... ok
test cors::tests::preflight_with_wildcard_allowed_origin_returns_wildcard ... ok
test xml::cors::tests::test_deserialize_norules ... ok
test xml::cors::tests::test_deserialize ... ok
test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 16 filtered out; finished in 0.00s
```
Co-authored-by: smattymatty <smattymatt@gmail.com>
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/1450
Reviewed-by: Alex <lx@deuxfleurs.fr>
## Title
fix(cors): return single matching origin instead of multiple values in `Access-Control-Allow-Origin`
## Summary
This PR fixes bucket CORS responses when a single CORS rule contains multiple `AllowedOrigins`.
Previously, Garage returned the configured origins as a comma-separated list in `Access-Control-Allow-Origin`, for example:
```http
Access-Control-Allow-Origin: https://app.example.test, https://admin.example.test
```
This is not the expected browser-facing behavior.
When a request origin matches a configured rule, the response should reflect **only the matching request origin**, unless the rule contains `*`.
## What changed
- `Access-Control-Allow-Origin` now behaves as follows:
- returns `*` when the matched rule contains a wildcard origin
- otherwise returns the request `Origin` as a **single value**
- added `Vary: Origin` when ACAO reflects the request origin
- added preflight-specific `Vary` handling in the preflight path for:
- `Origin`
- `Access-Control-Request-Method`
- `Access-Control-Request-Headers`
## Scope
This change applies to shared bucket CORS handling paths, including:
- S3 API responses
- K2V API responses
- S3 POST object responses
- web bucket responses
- preflight (`OPTIONS`) bucket CORS responses
This does **not** change admin API fixed CORS behavior.
## Reproduction
A direct repro script is included:
```bash
./script/test-cors-multi-origin.sh
```
It exercises two cases against a direct single-node Garage instance:
1. **single-origin control**
2. **multi-origin repro**
Before this fix, the multi-origin case returned a comma-separated ACAO value.
After this fix, both cases reflect only the request origin.
## Example behavior
### Before
```http
Access-Control-Allow-Origin: https://app.example.test, https://admin.example.test
```
### After
```http
Access-Control-Allow-Origin: https://app.example.test
```
## Tests
Added/updated tests in `src/api/common/cors.rs` for:
- single-origin control
- multiple allowed origins reflecting the request origin
- wildcard origin preserving `*`
- preserving existing `Vary` values while appending `Origin`
## Validation
Used for validation:
```bash
cargo test -p garage_api_common cors::tests -- --nocapture
cargo build -p garage --bin garage
./script/test-cors-multi-origin.sh
```
## Reproducibility
For reviewers who want to validate behavior by commit:
- Before fix: `aa368e4b`
- includes the direct repro script and the regression test setup
- multi-origin ACAO is reproduced as a comma-separated value
- After fix: `f630eb92`
- reflects only the matching request origin
- preserves wildcard behavior
- adds `Vary: Origin` and preflight-specific `Vary` handling
Branch:
- `fix/cors-multiple-allow-origin`
Base used during validation:
- `74ad3bf8` (`main-v2`)
ClosesDeuxfleurs/garage#1149
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/1419
## Summary
Garage's SigV4 canonical-request builder trims leading/trailing whitespace from signed header values but does not collapse sequential internal whitespace, which the SigV4 spec requires:
> Convert sequential spaces to a single space.
— https://docs.aws.amazon.com/IAM/latest/UserGuide/create-signed-request.html
AWS SDKs apply this normalization before computing the signature, but transmit the raw value on the wire. The receiver must therefore apply the same normalization when reconstructing the canonical request, otherwise the recomputed hash differs and the request is rejected as `Invalid signature`.
Same class of canonicalization-drift bug as #1155 / !1382, but on the canonical-headers axis rather than the canonical-URI axis.
## Reproduction
Surfaces in practice with `gitlab-runner`'s S3 cache uploader. I was in the midst of migrating my runner cache from AWS S3 to garage, but I noticed some shared runner caches were no longer uploading.
I was using `sha256sum | sha256sum` to compute my cache keys, which leaves a trailing ` -` on the value. Once GitLab appends `-protected` for protected branches the resulting `x-amz-meta-cachekey` header value contains internal sequential whitespace and triggers the mismatch:
```
x-amz-meta-cachekey:php- --protected
^^
two spaces, preserved by Garage
```
Without the fix the included regression test (`test_presigned_put_with_user_metadata`) fails with HTTP 403; with the fix it returns 200.
`aws-cli` is unaffected because it signs `Content-Type` rather than user metadata, so the specific code path with whitespace-bearing signed header values isn't exercised.
## Fix
In `canonical_request` (`src/api/common/signature/payload.rs`), replace the `.trim()` call on the joined header value with the full SigV4 normalization — `split_whitespace().collect::<Vec<_>>().join(" ")` — which both trims edges and collapses internal runs.
## Tests
* New regression test `test_presigned_put_with_user_metadata` covering a presigned PUT whose `x-amz-meta-*` value contains internal sequential whitespace.
* Full integration suite passes: `40 passed; 0 failed; 2 ignored`.
* `garage_api_common` unit tests pass: `18 passed; 0 failed`.
## Notes
* Backwards-compatible: any signature that validated before still validates, because clients are spec-required to collapse on their side; Garage was only rejecting requests where the client had collapsed correctly but Garage hadn't.
* No config or migration changes.
* Fix applies to both presigned-URL and Authorization-header code paths since they share the canonical-request builder.
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/1424
Reviewed-by: Alex <lx@deuxfleurs.fr>
Merging this first version as a baseline. For future work: write a security.md document to explain how to report security vulnerabilities, and split off the release process in a separate releasing.md document
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/1406
Otherwise the rustls dependency might be built with both aws-lc and ring backends,
leading to the following error in the k2v_client tests when
consul-discovery feature is enabled (including the reqwest dependency):
```
Could not automatically determine the process-level CryptoProvider from Rustls crate features.
Call CryptoProvider::install_default() before this point to select a provider manually, or make sure exactly one of the 'aws-lc-rs' and 'ring' features is enabled.
See the documentation of the CryptoProvider type for more information.
```
Co-authored-by: Yureka <yuka@yuka.dev>
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/1412
Reviewed-by: Alex <lx@deuxfleurs.fr>
known_addrs in PeerInfoInternal is append-only — addresses accumulate
via add_addr() and PeerList gossip but are never removed. In dynamic
environments (k8s pod restarts, DHCP, NAT traversal), this list grows
unboundedly with stale addresses.
Combined with sequential iteration in try_connect() and no TCP connect
timeout in netapp.rs, each unreachable address blocks reconnection for
the kernel's TCP SYN timeout (75-130s on Linux). With 10+ stale
addresses, worst-case reconnection exceeds 750s — a full outage for
replication_factor=3 clusters.
This commit contains the two following changes:
1. Address failure tracking and pruning (peering.rs): Track consecutive
connection failures per address in PeerInfoInternal. After 3 failures,
prune from known_addrs. Reset count when address is re-advertised via
gossip or incoming connection. Prevents unbounded list growth.
2. Shuffle before connecting (peering.rs): Randomize address order in
try_connect() so the valid address (often appended last) gets a fair
chance instead of always trying stale addresses first.
known_addrs in PeerInfoInternal is append-only — addresses accumulate
via add_addr() and PeerList gossip but are never removed. In dynamic
environments (k8s pod restarts, DHCP, NAT traversal), this list grows
unboundedly with stale addresses.
Combined with sequential iteration in try_connect() and no TCP connect
timeout in netapp.rs, each unreachable address blocks reconnection for
the kernel's TCP SYN timeout (75-130s on Linux). With 10+ stale
addresses, worst-case reconnection exceeds 750s — a full outage for
replication_factor=3 clusters.
This patches includes a first change to fix this issue:
1. TCP connect timeout (netapp.rs): Wrap TcpStream::connect() in
tokio::time::timeout(10s). Caps per-address attempt from 75-130s
to 10s, reducing worst-case 10-addr reconnection from ~750s to ~100s.
## Summary
This PR ensures that the `LifecycleWorker` yields at least once to the Tokio scheduler in between each batch of 100 objects.
## Problem being solved
I'm administrating a Garage cluster which has been experiencing timeouts on all endpoints while the lifecycle worker is running at midnight UTC : `Ping timeout` error messages and even requests eventually failing due to `Could not reach quorum ...`.
I have found that this happens while the lifecycle worker is working on a big bucket (containing millions of objects) with a lifecycle rule that applies to very few objects.
The `process_object()` function does not hit any `await`:
- `last_bucket` is always the same, so the `bucket_table` is not read asynchronously
- no transaction is made on the `object_table` because my lifecycle rule (almost) never applies to any object
The first commit in this PR adds an executable which reproduces the problem that I've been experiencing in a self-contained way : the lifecycle worker starves the Tokio scheduler so much that no other task is able to run (or very rarely).
To run it : `cargo run -p garage_model --bin lifecycle-starvation-test`.
This commit can be dropped post-review, as it's only useful to demonstrate the starvation.
The error messages completely stopped after adding the extra yield to the nodes of my cluster.
The duration of the lifecycle worker task does not appear to have changed at all from what I can see (looking at the timestamps produced either by the self-contained binary or by each of my nodes with the `Lifecycle worker finished` message).
## Note
An other potential fix would have been to force the `WorkerProcessor` to yield before re-enqueuing a busy task, but this would have affected all Garage workers even though it's only the `LifecycleWorker` being uncooperative.
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/1396
Reviewed-by: Alex <lx@deuxfleurs.fr>
Co-authored-by: Gauthier Zirnhelt <gauthier.zirnhelt@insimo.fr>
Co-committed-by: Gauthier Zirnhelt <gauthier.zirnhelt@insimo.fr>
This fixes a regression wrt garage-v1, likely caused by the version upgrade of quick_xml.
Currently, garage-v2 will emit empty ErrorDocument/IndexDocument/RedirectAllRequestsTo attributes in the response of GetBucketWebsite if there are no corresponding values.
This is somewhat wrong; at least, the S3 documentation for RedirectAllRequestsTo (https://docs.aws.amazon.com/AmazonS3/latest/API/API_RedirectAllRequestsTo.html) writes that it has a required HostName field. So emitting an empty RedirectAllRequestsTo is invalid.
This PR skips emitting XML attributes for these parameters if they contain no value.
Co-authored-by: Armaël Guéneau <armael.gueneau@ens-lyon.org>
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/1391
Co-authored-by: Armael <armael@noreply.localhost>
Co-committed-by: Armael <armael@noreply.localhost>
## Summary
This PR fixes S3 `DeleteObjects` XML parsing when the request body is pretty-printed (contains indentation/newlines as whitespace text nodes).
Although PR #1324 already tried to address this, parsing could still fail with:
`InvalidRequest: Bad request: Invalid delete XML query`
because non-element nodes were validated but not actually skipped in the parsing loop.
## What changed
- In `src/api/s3/delete.rs`:
- Properly skip non-element whitespace text nodes while iterating over `<Delete>` children.
- Keep rejecting non-whitespace stray text content.
- Parse the root `<Delete>` element more robustly by selecting the first element child.
## Tests added
New unit tests in `src/api/s3/delete.rs`:
- `parse_delete_objects_xml_with_formatting`
- pretty-printed valid XML is accepted.
- `parse_delete_objects_xml_accepts_compact_valid_xml`
- compact valid XML is accepted.
- `parse_delete_objects_xml_rejects_non_whitespace_text_node`
- compact XML with stray text is rejected.
- `parse_delete_objects_xml_rejects_pretty_print_with_stray_text`
- pretty-printed XML with stray text is rejected.
## Validation
Executed:
```bash
cargo test -p garage_api_s3 parse_delete_objects_xml -- --nocapture
```
Result: all parser tests pass.
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/1374
Co-authored-by: milouz1985 <francois.hoyez@gmail.com>
Co-committed-by: milouz1985 <francois.hoyez@gmail.com>
## Problem
`hugo deploy` is broken with Garage on recent hugo versions when using gzip matchers
## Why?
We don't support multi-value headers correctly, in this case this specific headers combination:
```
Content-Encoding: gzip
Content-Encoding: aws-chunked
```
is interpreted as:
```
Content-Encoding: gzip
```
instead of:
```
Content-Encoding: gzip,aws-chunked
```
It fails both 1. the signature check and 2. the streaming check.
## Proposed fix
- Taking into account multi-value headers when building Canonical Request (validated with hugo deploy + AWS SDK v2)
- Taking into account multi-value headers (both comma separated and HeaderEntry separated) when removing `aws-chunked` (validated with hugo deploy + AWS SDK v2)
## Full explanation
Currently, `hugo deploy` on version `hugo v0.152.2` or more recent uses AWS SDK v2 only and supports for sending gzipped content.
That's configured with a matcher like that:
```yaml
deployment:
matchers:
- pattern: "^.+\\.(woff2|woff|svg|ttf|otf|eot|js|css)$"
cacheControl: "max-age=31536000, no-transform, public"
gzip: true # <-------- here
```
Also, with SDK v2, hugo is streaming all of its files.
Thus, it sends that kind of requests:
```python
Request {
method: PUT,
uri: /sebou/pagefind/pagefind.js?x-id=PutObject,
version: HTTP/1.1,
headers: {
"host": "localhost",
"user-agent": "aws-sdk-go-v2/1.39.2 ua/2.1 os/linux lang/go#1.25.6 md/GOOS#linux md/GOARCH#amd64 api/s3#1.84.0 ft/s3-transfer m/E,G,Z,g",
"content-length": "10026",
"accept-encoding": "identity",
"amz-sdk-invocation-id": "aed6df34-a67c-4bab-b63b-2b3777b751a0",
"amz-sdk-request": "attempt=1; max=3",
"authorization": "AWS4-HMAC-SHA256 Credential=GKxxxxx/20260227/garage/s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;cache-control;content-encoding;content-length;content-type;host;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length;x-amz-meta-md5chksum;x-amz-trailer, Signature=76cd9b77f693ca89c2e6dd2a4dc55f83d4a82eca0f563d9d095ff96076f7b057",
"cache-control": "max-age=31536000, no-transform, public",
"content-encoding": "gzip", # <---- see here 1st instance of Content-Encoding
"content-encoding": "aws-chunked", # <---- 2nd instance of Content-Encoding
"content-type": "text/javascript",
"via": "2.0 Caddy",
"x-amz-content-sha256": "STREAMING-UNSIGNED-PAYLOAD-TRAILER",
"x-amz-date": "20260227T132212Z",
"x-amz-decoded-content-length": "9982",
"x-amz-meta-md5chksum": "aad88ac0bf704e91584b8d9ad9796670",
"x-amz-trailer": "x-amz-checksum-crc32",
"x-forwarded-for": "::1",
"x-forwarded-host": "localhost",
"x-forwarded-proto": "https"
},
body: Body(Streaming)
}
```
But our canonical request function only calls `HeaderMap.get()` that returns only the 1st value and not `HeaderMap.get_all()` that returns all the values for a header.
Leading to the following invalid `CanonicalRequest` value:
```python
PUT
/sebou/pagefind/pagefind.js
x-id=PutObject
accept-encoding:identity
amz-sdk-invocation-id:aed6df34-a67c-4bab-b63b-2b3777b751a0
amz-sdk-request:attempt=1; max=3
cache-control:max-age=31536000, no-transform, public
content-encoding:gzip # <----- see here, we kept only gzip and dropped aws-chunked
content-length:10026
content-type:text/javascript
host:localhost
x-amz-content-sha256:STREAMING-UNSIGNED-PAYLOAD-TRAILER
x-amz-date:20260227T132212Z
x-amz-decoded-content-length:9982
x-amz-meta-md5chksum:aad88ac0bf704e91584b8d9ad9796670
x-amz-trailer:x-amz-checksum-crc32
accept-encoding;amz-sdk-invocation-id;amz-sdk-request;cache-control;content-encoding;content-length;content-type;host;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length;x-amz-meta-md5chksum;x-amz-trailer
```
Amazon is crystal clear that, instead of dropping the other values, we should concatenate them with a comma:
https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_sigv-create-signed-request.html#create-canonical-request
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/1369
Reviewed-by: Alex <lx@deuxfleurs.fr>
Co-authored-by: Quentin Dufour <quentin@deuxfleurs.fr>
Co-committed-by: Quentin Dufour <quentin@deuxfleurs.fr>
- use `trim` method of `str` instead of manual implementation with `trim_matches(char::is_whitespace)`
- use result of `trim` for xml parsing instead of use the `str` before trim.
Garage RPC connections have no TCP keepalive enabled. When a connection dies silently (proxy pod restart, NAT timeout, network partition), it's only detected by application-level pings after ~60s (4 failed pings x 15s interval). During this window, the node appears connected but all RPC calls to it fail.
Enable TCP keepalive on both outgoing and incoming RPC connections via socket2:
- Idle time before first probe: 30s (TCP_KEEPALIVE_TIME)
- Probe interval after first: 10s (TCP_KEEPALIVE_INTERVAL)
A helper set_keepalive() function avoids duplicating the socket2 setup. Incoming connection keepalive failures are logged as warnings but don't reject the connection.
Companion to #1345 (stale address pruning + connect timeout). Together they address both halves of the reconnection problem: faster detection (this PR) and faster recovery.
Co-authored-by: Raj Singh <raj@tailscale.com>
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/1348
Reviewed-by: maximilien <git@mricher.fr>
Co-authored-by: rajsinghtech <rajsinghtech@noreply.localhost>
Co-committed-by: rajsinghtech <rajsinghtech@noreply.localhost>
Even when using the catalog an dedicated token for authentication
might be needed.
**Approach**: Support the token header even with client certs was the simplist approach and somebody might need/want to use it.
**Background**: I want to run garage via Nomad but within containers (with host volumes). Nomad generates consul tokens (but at least not at the moment client certs). I need to use the catalog as with the services API garage tries to use the host/node IPs (instead of the actual service IPs).
**Tests**: I deployed this version and it works well.
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/1353
Reviewed-by: Alex <lx@deuxfleurs.fr>
Co-authored-by: Malte Swart <mswart@devtation.de>
Co-committed-by: Malte Swart <mswart@devtation.de>
add add some related tests.
catched from clippy lint `format_collect`
message: use of `format!` to build up a string from an iterator
--> src/api/common/encoding.rs:12:17
|
12 | let value = format!("{}", c)
| _____________________________^
13 | | .bytes()
14 | | .map(|b| format!("%{:02X}", b))
15 | | .collect::<String>();
| |________________________________________^
|
help: call `fold` instead
--> src/api/common/encoding.rs:14:7
|
14 | .map(|b| format!("%{:02X}", b))
| ^^^
help: ... and use the `write!` macro here
--> src/api/common/encoding.rs:14:15
|
14 | .map(|b| format!("%{:02X}", b))
| ^^^^^^^^^^^^^^^^^^^^^
= note: this can be written more efficiently by appending to a `String` directly
= help: for further information visit https://rust-lang.github.io/rust-clippy/rust-1.93.0/index.html#format_collect
and use workspace configuration in each package.
This allow to customize clippy and rust lint configuration for project.
No particular configuration in this commit.
- thin LTO is much much faster to compile, this will help when making
release builds
- strip="debuginfo" allows to still have symbols when unwinding stack
traces, which are usefull to have in user's bug reports. Binary size
is increased from 23M to 27M, so a reasonable increase
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/1342
Co-authored-by: Alex Auvolat <lx@deuxfleurs.fr>
Co-committed-by: Alex Auvolat <lx@deuxfleurs.fr>
this avoid rust-analyzer indicate invalid field error on `Redirect` for
`replace_prefix` and `replace_full` because of a conflict between struct :
`api::s3::website::Redirect` and `model::bucket_table::Redirect`
- migration from `ByteSlice` to `Bytes` heed type.
- not sure about the impact of only one `read_txn` for the entire function `list_trees`, whereas before `open_database` call uses their own access controller.
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/1318
Co-authored-by: Gwen Lg <me@gwenlg.fr>
Co-committed-by: Gwen Lg <me@gwenlg.fr>
- clippy::nonminimal_bool disabled for check_size_filter function
clippy message: this boolean expression can be simplified
help: for further information visit https://rust-lang.github.io/rust-clippy/rust-1.92.0/index.html#nonminimal_bool
- clippy::large_enum_variant for `DecryptStreamState` and `State`
- clippy::too_many_arguments for `put_block_and_meta` and
`test_read_encrypted`
- clippy::deref_addrof for specific unsafe code
- clippy::doc_overindented_list_items and clippy::doc_lazy_continuation
- include the lifetime instead of hide it
lint message: hiding a lifetime that's elided elsewhere is confusing
help: the same lifetime is referred to in inconsistent ways, making the signature confusing
- remove useless lifetime
lint message: the following explicit lifetimes could be elided: 'a
help: for further information visit https://rust-lang.github.io/rust-clippy/rust-1.92.0/index.html#needless_lifetimes
I have successfully setup my Pleroma instance with GarageHQ and would share this in the documentation.
The part with the loop is totally optionnal, but, as I said, in case of old link in Pleroma database I prefer run it.
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/1294
Co-authored-by: Dryusdan <contact@dryusdan.fr>
Co-committed-by: Dryusdan <contact@dryusdan.fr>
Put the features flags into alphabetical order in both the cookbook and
the version information, so that it is easier to document additional
feature flags.
The version control site was migrated from Gitea to Forgejo some time
ago, and Forgejo has declared a hard fork from Gitea[1]. Update the
documentation and links to refer to the site as a Forgejo instance
instead of a Gitea instance.
[1] https://forgejo.org/2024-02-forking-forward/
Buypass Go SSL has stopped providing SSL/TLS certificates, including
from their ACME API endpoints, as of October 2025, and the service is to
be completely phased out by April 2026:
https://community.buypass.com/t/y4y130p
Remove them from the docs as an ACME-capable provider.
When not used. Especially with a lot of small files on the PVC's Garage uses/is configured with,
Pod upstart time can get quite extreme. For more context see a very good description
of the issues and the technical intricacies behind it here: https://github.com/longhorn/longhorn/issues/2131#issuecomment-778897129
I think it's sane to have fsGroupChangePolicy at "OnRootMismatch" as Garage is an S3 system and thereby
is likely to process a lot of files. And somewhat likely that many of these will be small files as well.
---
Using `fsGroupChangePolicy: "OnRootMismatch"` in my env. changed the Garage Pod upstart. E.g. in an
upgrade case from 5-7minutes ( around 50GB of data ). Many small files though. To 1-15sec.
Thank you.
Signed-off-by: larssb <larssb@noreply.localhost>
The systemd-journald is used in most major Linux distros that use systemd.
This enables logging using the systemd-journald native protocol, instead
of just writing to stderr.
i went with a `500`/`InternalError`/`Please try again.` because that is something i've seen AWS S3 report while developing other software, and i'm not convinced all clients would understand a 409 conflict properly (GET don't usually conflict)
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/1053
Co-authored-by: trinity-1686a <trinity@deuxfleurs.fr>
Co-committed-by: trinity-1686a <trinity@deuxfleurs.fr>
- Remove the replication_factor.replication_factor() in favor of
usize::from(replication_factor) to make the conversion more explicit.
- Implement Display on ReplicationFactor so that it can be formatted
without converting to usize
- Use ReplicationFactor in the constructor of LayoutVersion and add a
method to get a ReplicationFactor from a LayoutVersion, despite
LayoutVersion still storing it as usize internally.
See #1006
LMDB files never shrink, so we can end up with a large database that
contains a smaller amount of actual data.
Compacting the snapshots is an easy win: it will write faster to disk,
take less space, and if needed you can reimport an already-compacted
snapshot as the main database.
In particular, it means that "garage meta snapshot --all" will get an exit
code of 1 if any node fails to snapshot.
This makes sure that any external tool trying to snapshot nodes (e.g. from
cron) will be aware of the failure.
Fix#920
Snapshot errors on remote nodes were not reported at all.
We now get proper error output such as:
0fa0f35be69528ab error: Internal error: DB error: LMDB: No space left on device (os error 28)
88d92e2971d14bae ok
Fix#920
admin api: rename EndpointHandler into RequestHandler to avoid confusion with RPC
wip: infrastructure for local api calls
admin api: fix things
admin api: first local endpoint to work with new scheme
admin api: implement SetWorkerVariable
This removes our dependency on cargo2nix, which was causing us some
issues. Whereas cargo2nix creates one Nix derivation per crate, Crane
uses only two derivations:
1. Build dependencies only
2. Build the final binary
This means that during the second step, no caching can be done. For
instance, if we do a change in garage_model, we need to recompile all of
the Garage crates including those that do not depend on garage_model.
On the upside, this allows all of the Garage crates to be built at once
using cargo build logic, which is optimized for high parallelism and
better pipelining between all of the steps of the build. All in all,
this makes most builds faster than cargo2nix.
A few other changes have been made to the build scripts and CI:
- Unit tests are now run within a Nix derivation. In fact, we have
different derivations to run the tests using LMDB and Sqlite as
metadata db engines.
- For debug builds, most CI steps now run in parallel (with the notable
exception of the smoke test that runs after the build, which is
inevitable).
- We no longer pass the GIT_VERSION argument when building debug builds
and running the tests. This means that dev binaries and test
binaries don't know the exact version of Garage they are from. That
shouldn't be an issue in most cases.
- The not-dynamic.sh scripts has been fixed to fail if the file does not
exist.
This change is probably not a proper fix, somebody with more expertise on
this code should look at it.
Here is how to reproduce the crash:
- start with a layout with two zones
- move all nodes of a zone to gateway mode: `garage layout assign fea54bcc081f318 -g`
- `garage layout show` will panic with a backtrace
Fortunately, the crash is only on the RPC client side, not on the Garage
server itself, and `garage layout revert` still works to go back to the
previous state.
As far as I can tell, this bug is present since Garage 0.9.0 which
includes the new layout assignation algorithm:
https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/296
Currently, taking a snapshot of the metadata database with sqlite creates
a sqlite file without extension with the following format:
snapshots/2025-01-26T15:29:17Z
This makes it hard to understand what kind of data this is, and is not
consistent with LMDB:
snapshots/2025-01-26T15:29:17Z/data.mdb
With this change, we now get a directory with a single db.sqlite file:
snapshots/2025-01-26T15:29:17Z/db.sqlite
I followed the documentation and got an error if the layout's version was not specified
```
garage layout apply
Error: Internal error:
Please pass the new layout version number to ensure that you are writing the correct version of the cluster layout.
To know the correct value of the new layout version, invoke `garage layout show` and review the proposed changes.
```
This fixes that
In case `rpc_public_addr` is not set, but autodiscovery is used, this
allows filtering the list of automatically discovered IPs to a specific
subnet.
For example, if nodes should pick *their* IP inside a specific subnet,
but you don't want to explicitly write the IP down (as it's dynamic, or
you want to share configs across nodes), you can use this option.
- We always recalculate the reference count of a block before deleting
it locally, to make sure that it is indeed zero.
- If we had to fetch a remote block but we were not able to get it,
check that refcount is indeed > 0.
- Repair procedure that checks everything
This page of the AWS docs indicate that Content-Type should be part of
the CanonicalHeaders (and therefore SignedHeaders) strings in signature
calculation:
https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-header-based-auth.html
However, testing with Minio Client revealed that it did not sign the
Content-Type header, and therefore we broke CI by expecting it to be
signed. With this commit, we don't mandate Content-Type to be signed
anymore, for better compatibility with the ecosystem. Testing against
the official behavior of S3 on AWS has not been done.
For some users, this might be their first time being interacting with
the `env_logger` crate.
As such, they might not be aware that less verbose log levels exist.
Some might not want to log every incoming request, for example.
This commit also adds syntax hints to the code-fence for bash for better
syntax highlighting of that section, and repeats itself multiple times,
that `info` is, in fact, the default.
No changes to the recommendation of log levels were made.
Having all Engine enum variants conditional causes compilation errors
when *none* of the DB engine features is enabled. This is not an issue
for full garage build, but affects crates that use garage_db as
dependency.
Change all variants to be present at all times. It solves compilation
errors and also allows us to better differentiate between invalid DB
engine name and engine with support not compiled in current binary.
Use optional DB open overrides for both input and output database.
Duplicating the same override flag for input/output would result in too
many, too long flags. It would be too costly for very rare edge-case
where converting between same DB engine, just with different flags.
Because overrides flags for different engines are disjoint and we are
preventing conversion between same input/ouput DB engine, we can have
only one set.
The override flag will be passed either to input or output, based on
engine type it belongs to. It will never be passed to both of them and
cause unwelcome surprise to user.
Garage v0.8.5
This minor release includes the following improvements and fixes:
New features:
- Configuration: make LMDB's `map_size` configurable and make `block_size` and `sled_cache_capacity` expressable as strings (such as `10M`) (#628, #630)
- Add support for binding to Unix sockets for the S3, K2V, Admin and Web API servers (#640)
- Move the `convert_db` command into the main Garage binary (#645)
- Add support for specifying RPC secret and admin tokens as environment variables (#643)
- Add `allow_world_readable_secrets` option to config file (#663, #685)
Bug fixes:
- Use `statvfs` instead of mount list to determine free space in metadata/data directories (#611, #631)
- Add missing casts to fix 32-bit build (#632)
- Fix error when none of the HTTP servers (S3/K2V/Admin/Web) is started and fix shutdown hang (#613, #633)
- Add missing CORS headers to PostObject response (#609, #656)
- Monitoring: finer histogram boundaries in Prometheus exported metrics (#531, #686)
Other:
- Documentation improvements (#641)
Sometimes, the secret files permissions checks gets in the way. It's
by no mean complete, it doesn't take the Posix ACLs into account among
other things. Correctly checking the ACLs would be too involving (see
https://git.deuxfleurs.fr/Deuxfleurs/garage/issues/658#issuecomment-7102)
and would likely still fail in some weird chmod settings.
We're adding a new configuration file key allowing the user to disable
this permission check altogether.
The (already existing) env variable counterpart always take precedence
to this config file option. That's useful in cases where the
configuration file is static and cannot be easily altered.
Fixes https://git.deuxfleurs.fr/Deuxfleurs/garage/issues/658
Co-authored-by: Florian Klink <flokli@flokli.de>
This is still a bit confusing, as normally the flake.defaultNix attrset
gets exposed via a top-level default.nix, but at least it brings us
closer to that.
Compiling garage_db v0.8.2 (garage-0.8.2/src/db)
error: cannot find macro `warn` in this scope
--> src/db/lmdb_adapter.rs:352:2
|
352 | warn!("LMDB is not recommended on 32-bit systems, database size will be limited");
| ^^^^
|
= help: consider importing this macro:
tracing::warn
= note: `warn` is in scope, but it is an attribute: `#[warn]`
error: could not compile `garage_db` due to previous error
* Renamed my_garage_service -> garage-s3-service.
* Defined a web service for port 3902.
* Added a garage-s3 router.
* Pointed website definition at web service.
* Use the /health endpoint for loadBalancer health check.
* Renamed gzip_compress to just compression as traefik v3 will also do
brotli compression.
DaemonSet is a k8s resource that schedules one instance per node,
which is useful for some garage deployment use cases, including
managing garage nodes using k8s node labels
As discussed in the chat yesterday, I want to propose to disable the ingress per default.
The motivation behind this change is, that per default the ingress is "misconfigured"
meaning it can not work with the default values and requires a user of the chart to
add additional configuration. When installing the chart per default, I would not
expect to already expose garage publicly without my explicit configuration to do so
Commenting the ingressClass resource also allows for relying only on
annotations - otherwise the ingressClass would be always set to nginx
or require a user to override it with ingressClass: null
A small change on top, I've added the ability to specify user defined labels per ingress
The default values forces people to create an ingress resources,
where per default an ingress is not necessary to start garage.
If someone wants to utilize an ingress, he would need to define
the values for the ingress either way, so enabling the ingress
explicitly makes more sense, then requiring it to be disabled per default
By default, Nginx does proxy buffering and it may store big replies to a
temporary file up to 1 GB. It also means that Nginx will read data as
fast as possible from Garage, even if the client downloads slowly. Both
behaviours are often not wanted, so disable this temporary file in the example.
Ref: https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffering
Also add an example of upstream with a "backup" server, which may be
useful to only use remote servers as fallback.
- self.node_id_vec was not properly updated when the previous ring was empty
- ClusterLayout::merge was not considering changes in the layout parameters
- have consistent error return types
- store the zone redundancy in a Lww
- print the error and message in the CLI (TODO: for the server Api, should msg be returned in the body response?)
It takes as paramters the replication factor and the zone redundancy, computes the
largest partition size reachable with these constraints, and among the possible
assignation with this partition size, it computes the one that moves the least number
of partitions compared to the previous assignation.
This computation uses graph algorithms defined in graph_algo.rs
Performance improvements included in this PR:
- [x] Use `Bytes` at a few places where appropriate, instead of `Vec<u8>`, to reduce the number of copies
- [x] StreamChunker now accumulates incoming slices in a `Vec<Bytes>` instead of a `VecDeque<u8>`. Replaces calls to `.extend()` and `.drain()` that were quite costly by a simple `concat()` on a vec of slices which is much more optimized
- [x] Hashing (b2, sha256, md5) is now done on a Tokio thread dedicated to cpu-intensive tasks, using `spawn_blocking`
- [x] Block manager now uses 256 independant locks instead of one big lock for writing, reduces contention when writing several/many objects in parallel
- [x] Better LMDB defaults: we now put flags `NoSync` and `NoMetaSync` to avoid `fsync` at each transaction (extremely slow). Also increased number of LMDB readers to accomodate more intensive workloads
Other changes included in this PR:
- [x] Update to hashing and MAC crates: md5 and sha2 from 0.9 to 0.10, hmac from 0.10 to 0.12
- [x] switch to `tracing_subscriber` for logs, which allows to have timing of each event
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/342
This PR includes work from @jirutka :
- [x] Allow linking against system-provided libraries (libsodium, libsqlite, libzstd) #370
- [x] Make OTLP exporter optional and allow building without Prometheus exporter (/metrics) #372
And also:
- [x] Update `.nix` files
- [x] Remove heed default-features
- [x] Bump versions of all Garage crates to 0.8.0
- [x] Make db engines (lmdb, sled, sqlite) optionnal
- [x] Add documentation for available features
- [x] Directly include code of previous versions used for migration in order to reduce dependencies
- [x] Read variable `GIT_VERSION` from garage main instead of in crate garage_util to make builds faster
- [x] Report features used in the build somewhere? (in `garage --version` or something)
- [x] Check we `warn!` correctly if we try to use deactivated feature
- [x] Allow not to launch S3 endpoint if not in config
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/373
Garage currently uses the legacy resolver "1". The new one is used
by default if the root package specifies 'edition = 2021', which
Garage does not (yet).
The problem with the legacy resolver is, among others, that features
enabled by dev-dependencies are propagated to normal dependencies.
This affects e.g. hyper - one of the dev-dependencies enables "http2"
feature that adds many extra dependencies. If we build garage without
opentelemetry-otlp (this is enabled in the following commit), there's
no normal dependency enabling "http2" feature.
See https://doc.rust-lang.org/cargo/reference/resolver.html#feature-resolver-version-2
Included in this PR:
- [x] Small refactor, resync code is moved to a separate `block/resync.rs` file
- [x] Block resync tranquility is no longer in config file, it is set dynamically using `garage worker set resync-tranquility` (this parameter is persisted over Garage restarts)
- [x] Up to 4 block resync workers can be activated to run simultaneously to speed up big resyncs, this parameter is set dynamically using `garage worker set resync-n-workers`
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/369
Unfortunately, rusqlite uses the opposite logic for enabling/disabling
bundled libraries to others (libsodium-sys, zstd-sys). Cargo features
are very limited and doesn't allow to enable feature A in a dependency
iff feature B is disabled.
Note, lmdb-rkv-sys doesn't need any special treatment because it
automatically links against system liblmdb if found via pkgconf.
Linux distros should build garage with
`--no-default-features --features system-libs` to disable bundled-libs
and enable system-libs.
If this feature is enabled, libsodium-sys and zstd-sys will link
dynamically against system-provided libraries instead of building
and linking statically the bundled (possibly outdated and vulnerable)
copies of them. This feature is intended mainly for linux package
maintainers.
By default, structopt reports the value provided by
the env var CARGO_PKG_VERSION, feeded by Cargo when reading
Cargo.toml. However for Garage we use a versioning based on git,
so we often report a version that is behind the real version.
In this commit, we create garage_util::version::garage() that
reports the right version and configure all structopt subcommands
to call this function instead of using the env var.
The function now computes an optimal assignation (with respect to partition size) that minimizes the distance to the former assignation, using flow algorithms.
This commit was written by Mendes Oulamara <mendes.oulamara@pm.me>
2022-05-01 09:54:19 +02:00
737 changed files with 142804 additions and 24929 deletions
To ensure the quality of the codebase and documentation, the use of AI,
including LLMs and coding agents, is strictly restricted in the following way:
- AI **must not** be used to write documentation
- **Do not** use AI to write bug reports, commit descriptions and pull request
messages
- **Do not** use AI agents to make contributions to Garage, all contributions
must be led by a human that know what they are doing at all times
- AI **may** be used for some tedious code generation tasks, limited to very
mechanical translations from API docs or boilerplate writing. The code
generated must be so simple as to make it clear that it cannot be covered by
copyright.
You are free to make use of AI privately to explore the codebase and solve
conceptual problems, but please restrain from copying the output from an LLM
anywhere in your code or on the issue tracker, or from letting an agent edit
the codebase directly.
## Asking questions
Read the documentation before asking questions.
Do not use the issue tracker to ask questions about Garage.
Questions asked on the issue tracker will be closed.
Ask questions on the Matrix channel `#garage:deuxfleurs.fr` so that any
community member can see your question and help you out.
If you need in-depth support from the Garage developers specifically, write to
`garagehq@deuxfleurs.fr`. Even if you do so, we do not commit to giving you an
answer.
## Reporting bugs
When writing a bug report, use this checklist:
- For bugs that can be reproduced:
- confirm that you are using the latest version of Garage and that the bug still exists in this version
- set the log level to debug using the `RUST_LOG=garage=debug` environment variable and reproduce the bug to get more verbose logs
- Check whether there is already an open issue in the bug tracker. If so, your bug report is still valuable but please add it as a comment to the existing issue instead of opening a new one.
- Collect as much information as possible:
- logs of the Garage daemon at the time the issue happened, including logs that show what was happening before the issue occurred
- the output of `garage status`
- the output of `garage stats -a`
- the output of `garage layout history`
- Write a detailed bug report, including:
- a description of your cluster (number of nodes, hardware, operating system, networking, etc)
- a detailed description of what you did that led to the issue, including any code or command line that invoked a Garage API
- what you were expecting
- what actually happened, and how that's different from what you expected
- the information collected previously
- if possible, simple steps to help the developers reproduce the issue locally
Bug reports that are imprecise or otherwise unactionable will be closed.
## Suggesting new features
Garage can be improved in many ways, but just suggesting a new feature does not mean we will implement it.
Feature requests that may lead to an actual implementation are feature requests that:
- are precise and actionable, i.e. include a precise description of the expected behavior and any necessary architectural details required for the implementation
- are motivated by actual need from a variety of users
Moreover, a certain number of features are defined as out-of-scope for Garage, including but not limited to:
- extensions to the S3 API that are not present on AWS
- features that require the implementation of a consensus algorithm
- more generally, features that are incompatible with the architecture of Garage and its goal of staying simple
Only feature requests in one of the following category may stay open in the issue tracker:
- features that the Garage team wants to work on
- features that are being actively worked on by an external contributor which is clearly identified
- features that are easy to implement and could be an easy task for a new contributor that wants to get to know the codebase
All other feature requests will be closed after a few months of inactivity, so as to keep the number of open issues to a manageable level.
Feature requests that are clearly out of scope will be closed directly.
## Improving the documentation
An easy way to contribute to Garage which also adds a lot of value is to
improve the documentation. Make sure to write in clear technical English, and
write unambiguously. Documentation contributions are very appreciated if they
are well-written.
## For developers
We welcome code contributions to Garage that adhere to our standards for quality:
- Changes should be reviewed from a functional perspective to ensure that they work well with the existing codebase and do not introduce bugs or subtle issues.
- You must have tested your contribution to make sure that it does what it says. The amount of testing required is proportional to the complexity of the change introduced.
- Any new feature must be properly documented following existing practices (see below).
- Unit tests should be included when relevant.
- Contributions should pass basic lints for syntactic quality (`cargo fmt`, `cargo clippy`, `typos`).
- Contributions should pass our CI test suite.
- No user-facing breaking changes may be introduced between major releases.
- No internal data model change may be introduced between major releases, to
ensure that Garage daemons with different minor/patch versions numbers can
work together in a cluster. For major releases, a proper migration path
should be implemented and tested thoroughly.
Please follow up on your work when changes are requested, to avoid stale PRs.
Do not take it personally if a Garage developer pushes directly to your branch
to modify your contribution, as this might be necessary to get it merged
faster.
### Properly documenting your contribution
#### Configuration options
New configuration options should be documented in
`doc/book/reference-manual/configuration.md`. The documentation for a
configuration option should be exhaustive. For instance, for choice options all
choices should be listed explicitly with a precise description of their
meaning.
In terms of syntax, all configuration options should appear in three places:
- in the example at the top, with an example value
- in the index of all configuration options which is sorted by alphabetical order
- in its dedicated subsection with full reference text
#### CLI commands and command flags
CLI commands are self-documented using the doc commends in the codebase.
Make sure to write clear and precise comments for all options you are adding.
#### S3 features
If you implement new S3 features, make sure to update the compatibility matrix in `doc/book/reference-manual/s3-compatibility.md`.
#### Admin API
The admin API has an OpenAPI specification that is automatically generated
using Utoipa, from a description of each endpoint that is given in
`src/api/admin/openapi.rs` and a description of data structure schemas in
`src/api/admin/api.rs`. The code in `openapi.rs` is only used to generate the
OpenAPI specification document and not for the actual implementation in Garage,
whereas structures defined in `api.rs` are also used for the implementation of
API calls. Make sure to write good doc comments for all of these items so that
the OpenAPI specification will be precise and accurate.
An up-to-date version of the OpenAPI specification document should be kept in
the repository in `doc/api/garage-admin-v2.json`. When you are making changes
to the admin API, update this document with the following command:
```
cargo run -- admin-api-schema > doc/api/garage-admin-v2.json
```
## Garage team organization
Alex (handle `lx`) is the lead developer and is responsible of ensuring the
correctness of Garage and stability between version upgrades.
The other maintainers are Trinity (handle `trinity-1686a`), Quentin (handle `quentin`) and Maximilien (handle `halfa`).
Maximilien is responsible for coordinating effort on the Kubernetes integration / Helm chart.
## Pull request merging criteria
The following PRs should only be merged after review and approval from Alex:
- PRs that introduce architectural changes, such as changes in the data model
or change in the coordination protocols between nodes
- PRs that introduce changes on the format of data structures used for
persistent disk storage and internal cluster communication (RPC)
- PRs that are suspected of introducing some kind of breakage or unexpected
behavior due to their complexity
PRs that introduce breaking change for users but don't fall in one of the
previous category should be discussed between maintainers to evaluate the
impact on users when upgrading. Alex's approval is not required to merge them
as long as they are clearly identified as breaking in the PR title, and are
properly merged in the branch for the next major version and not in the current
main branch.
All other PRs can be merged by any maintainer on their own, once they are
confident that the quality standards defined in this document are respected
before merging.
## Merging strategy
When merging PRs, maintainers should ensure that a Git commit is created by
Forgejo that records the PR number, its title and its text in the commit
message. If a PR is fixing an issue, make sure that the issue number is
included in the PR title as well. This is to ensure that when releasing a new
version of Garage, the changelog in the release notes can be properly
constructed by reading the Git log since the last release.
We also want to keep the history "almost linear" to facilitate the use of `git
bisect` if it ever were necessary. This leaves the following two merging
strategies:
- For PRs that consist of many commits that should stay independent, the
"rebase and create merge commit" strategy should be used. The merge commit is
created automatically by Forgejo and saves the PR's number, title and text in
the commit message.
- For PRs that consist of only one commit, or a few number of commits that can
be merged, the "create squash commit" strategy should be used. This way a
single commit will be created by Forgejo which also saves the PR's number,
title and text in the commit message.
When cherry-picking commits from one branch to the other, a simple fast-forward
merging strategy can be used if the commit message already references a PR
This documents how the Garage project operates. It reflects the state of the project as of July 2026 and is not optimal. The team is interested to improve it in the future.
## Team organization
* **Contributors**: anyone can contribute by proposing changes in issues and pull requests.
* **Maintainers**: they are responsible for reviewing, merging pull requests, publishing releases and triaging issues.
The current maintainers are:
* Alex (handle `lx`)
* Trinity (handle `trinity-1686a`)
* Quentin (handle `quentin`)
* Maximilien (handle `halfa`), who is in particular responsible for coordinating effort on the Kubernetes integration / Helm chart.
They are added to a white-list of the branch protection rule of the repository to enable them to merge pull requests.
To become a maintainer, you need to be a long-term contributor and earn the personal trust of Alex.
There is no set process for leaving the maintainer role.
* **Lead developer**: Alex (handle `lx`) is the lead developer and is responsible of ensuring the
correctness of Garage and stability between version upgrades. He may transfer this role to someone else as he sees fit.
## Communication channels
The team coordinates in the following channels:
* The issue tracker and pull requests of the official repository.
* The `#garage:deuxfleurs.fr` matrix channel (in English), open to anyone.
On this channel, users may ask for support and discussions about development also happen.
* The `#garage-dev:deuxfleurs.fr` matrix channel (in French), not advertised to contributors but de facto accessible to anyone.
Discussions about development and project coordination happen there.
The moderators for those discussion channels are the Garage maintainers.
## Decision procedures
Decisions are taken by lazy consensus, with the lead developer settling discussions when a consensus cannot be reached.
## Governance changes
There is no set process for changing the governance of garage.
If you wish to report responsibly a security vulnerability about Garage, we ask that you follow the following process.
Please report each security vulnerabilities by filling out the following template:
- PROJECT: A URL to the code repository containing the vulnerable version - be reminded that the source of truth is at https://git.deuxfleurs.fr/deuxfleurs/garage
- PUBLIC: Please let us know if this vulnerability has been made or discussed publicly already, and if so, please let us know where.
- DESCRIPTION: Please provide precise description of the security vulnerability you have found with as much information as you are able and willing to provide.
Please send the above info, along with any other information you feel is pertinent by emailing the core team at: garagehq@deuxfleurs.fr
The Garage Core Team will let you know within a few weeks whether or not your report has been accepted or rejected.
We ask that you please keep the report confidential until we have either responded or made a public announcement.
Garage has many API that you can rely on to build complex applications.
In this section, we reference the existing SDKs and give some code examples.
## ⚠️ DISCLAIMER
**K2V AND ADMIN SDK ARE TECHNICAL PREVIEWS**. The following limitations apply:
- The API is not complete, some actions are possible only through the `garage` binary
- The underlying admin API is not yet stable nor complete, it can breaks at any time
- The generator configuration is currently tweaked, the library might break at any time due to a generator change
- Because the API and the library are not stable, none of them are published in a package manager (npm, pypi, etc.)
- This code has not been extensively tested, some things might not work (please report!)
To have the best experience possible, please consider:
- Make sure that the version of the library you are using is pinned (`go.sum`, `package-lock.json`, `requirements.txt`).
- Before upgrading your Garage cluster, make sure that you can find a version of this SDK that works with your targeted version and that you are able to update your own code to work with this new version of the library.
- Join our Matrix channel at `#garage:deuxfleurs.fr`, say that you are interested by this SDK, and report any friction.
- If stability is critical, mirror this repository on your own infrastructure, regenerate the SDKs and upgrade them at your own pace.
## About the APIs
Code can interact with Garage through 3 different APIs: S3, K2V, and Admin.
Each of them has a specific scope.
### S3
De-facto standard, introduced by Amazon, designed to store blobs of data.
### K2V
A simple database API similar to RiakKV or DynamoDB.
Think a key value store with some additional operations.
Its design is inspired by Distributed Hash Tables (DHT).
More information:
- [In the reference manual](@/documentation/reference-manual/k2v.md)
### Administration
Garage operations can also be automated through a REST API.
We are currently building this SDK for [Python](@/documentation/build/python.md#admin-api), [Javascript](@/documentation/build/javascript.md#administration) and [Golang](@/documentation/build/golang.md#administration).
More information:
- [In the reference manual](@/documentation/reference-manual/admin-api.md)
If you are developing a new application, you may want to use Garage to store your user's media.
The S3 API that Garage uses is a standard REST API, so as long as you can make HTTP requests,
you can query it. You can check the [S3 REST API Reference](https://docs.aws.amazon.com/AmazonS3/latest/API/API_Operations_Amazon_Simple_Storage_Service.html) from Amazon to learn more.
Developing your own wrapper around the REST API is time consuming and complicated.
Instead, there are some libraries already available.
Some of them are maintained by Amazon, some by Minio, others by the community.
Nextcloud will now make Garage encrypt files at rest in the storage bucket.
These files will not be readable by an S3 client that has credentials to the
bucket but doesn't also know the secret encryption key.
### External Storage
**From the GUI.** Activate the "External storage support" app from the "Applications" page (click on your account icon on the top right corner of your screen to display the menu). Go to your parameters page (also located below your account icon). Click on external storage (or the corresponding translation in your language).
@@ -87,7 +136,7 @@ To test your new configuration, just reload your Nextcloud webpage and start sen
*Click on the picture to zoom*
Add a new external storage. Put what you want in "folder name" (eg. "shared"). Select "Amazon S3". Keep "Access Key" for the Authentication field.
In Configuration, put your bucket name (eg. nextcloud), the host (eg. 127.0.0.1), the port (eg. 3900 or 443), the region (garage). Tick the SSL box if you have put an HTTPS proxy in front of garage. You must tick the "Path access" box and you must leave the "Legacy authentication (v2)" box empty. Put your Key ID (eg. GK...) and your Secret Key in the last two input boxes. Finally click on the tick symbol on the right of your screen.
In Configuration, put your bucket name (eg. nextcloud), the host (eg. 127.0.0.1), the port (eg. 3900 or 443), the region ("garage" if you use the default, or the one your configured in your `garage.toml`). Tick the SSL box if you have put an HTTPS proxy in front of garage. You must tick the "Path access" box and you must leave the "Legacy authentication (v2)" box empty. Put your Key ID (eg. GK...) and your Secret Key in the last two input boxes. Finally click on the tick symbol on the right of your screen.
Now go to your "Files" app and a new "linked folder" has appeared with the name you chose earlier (eg. "shared").
@@ -128,21 +177,25 @@ In other words, Peertube is only responsible of the "control plane" and offload
In return, this system is a bit harder to configure.
We show how it is still possible to configure Garage with Peertube, allowing you to spread the load and the bandwidth usage on the Garage cluster.
Starting from version 5.0, Peertube also supports improving the security for private videos by not exposing them directly
but relying on a single control point in the Peertube instance. This is based on S3 per-object and prefix ACL, which are not currently supported
in Garage, so this feature is unsupported. While this technically impedes security for private videos, it is not a blocking issue and could be
a reasonable trade-off for some instances.
### Create resources in Garage
Create a key for Peertube:
```bash
garage key new --name peertube-key
garage key create peertube-key
```
Keep the Key ID and the Secret key in a pad, they will be needed later.
We need two buckets, one for normal videos (named peertube-video) and one for webtorrent videos (named peertube-playlist).
We need two buckets, one for normal videos (named peertube-videos) and one for webtorrent videos (named peertube-playlists).
```bash
garage bucket create peertube-video
garage bucket create peertube-playlist
garage bucket create peertube-videos
garage bucket create peertube-playlists
```
Now we allow our key to read and write on these buckets:
@@ -186,7 +239,7 @@ object_storage:
# Put localhost only if you have a garage instance running on that node
endpoint: 'http://localhost:3900' # or "garage.example.com" if you have TLS on port 443
# Garage supports only one region for now, named garage
# Garage default region is named "garage", edit according to your config
region: 'garage'
credentials:
@@ -195,8 +248,13 @@ object_storage:
max_upload_part: 2GB
proxy:
# You may enable this feature, yet it will not provide any security benefit, so
# you should rather benefit from Garage public endpoint for all videos
proxify_private_files: false
streaming_playlists:
bucket_name:'peertube-playlist'
bucket_name: 'peertube-playlists'
# Keep it empty for our example
prefix: ''
@@ -206,7 +264,7 @@ object_storage:
# Same settings but for webtorrent videos
videos:
bucket_name:'peertube-video'
bucket_name: 'peertube-videos'
prefix: ''
# You must fill this field to make Peertube use our reverse proxy/website logic
# What name gets exposed to users (HTTPS is implicit)
S3_ALIAS_HOST=my-social-media.mydomain.tld
```
For more details, see the [reference Mastodon documentation](https://docs.joinmastodon.org/admin/config/#cdn).
Restart all Mastodon services and everything should now be using Garage!
You can check the URLs of images in the Mastodon web client, they should start
with `https://my-social-media.mydomain.tld`.
### Last migration sync
After Mastodon is successfully using Garage, you can run a last sync from the local filesystem to Garage:
```bash
mc mirror --newer-than "3h" ./public/system/ garage/mastodon-data
```
### References
[cybrespace's guide to migrate to S3](https://github.com/cybrespace/cybrespace-meta/blob/master/s3.md)
(the guide is for Amazon S3, so the configuration is a bit different, but the rest is similar)
## Matrix
Matrix is a chat communication protocol. Its main stable server implementation, [Synapse](https://matrix-org.github.io/synapse/latest/), provides a module to store media on a S3 backend. Additionally, a server independent media store supporting S3 has been developped by the community, it has been made possible thanks to how the matrix API has been designed and will work with implementations like Conduit, Dendrite, etc.
Matrix is a chat communication protocol. Its main stable server implementation, [Synapse](https://matrix-org.github.io/synapse/latest/), provides a module to store media on a S3 backend. Additionally, a server independent media store supporting S3 has been developed by the community, it has been made possible thanks to how the matrix API has been designed and will work with implementations like Conduit, Dendrite, etc.
### synapse-s3-storage-provider (synapse only)
@@ -241,7 +426,7 @@ Supposing you have a working synapse installation, you can add the module with p
Now create a bucket and a key for your matrix instance (note your Key ID and Secret Key somewhere, they will be needed later):
Ente is an alternative for Google Photos and Apple Photos. It [can be selfhosted](https://help.ente.io/self-hosting/) and is working fine with Garage as of May 2024.
As a first step we need to create a bucket and a key for Ente:
```bash
garage bucket create ente
garage key create ente-key
# For the CORS setup to work, the key needs to be --owner as well, at least temporarily.
garage bucket allow ente --read --write --owner --key ente-key
```
We also need to setup some CORS rules to allow the Ente frontend to access the bucket:
aws s3api put-bucket-cors --bucket ente --cors-configuration $CORS
```
Now we need to configure ente-server to use our bucket. This is explained [in the Ente S3 documentation](https://help.ente.io/self-hosting/guides/external-s3).
Prepare a configuration file for ente's backend as `museum.yaml`:
```yaml
credentials-file: /credentials.yaml
apps:
public-albums: https://albums.example.tld # If you want to use the share album feature
internal:
hardcoded-ott:
local-domain-suffix: "@example.com" # Your domain
local-domain-value: 123456 # Custom One-Time Password since we are not sending mail by default
key:
# WARNING -- You MUST CHANGE the values below
# Someone has made an image that can do it for you : https://github.com/EdyTheCow/ente-selfhost/blob/main/images/ente-server-tools/Dockerfile
# Simply build it yourself or run docker run --rm ghcr.io/edythecow/ente-server-tools go run tools/gen-random-keys/main.go
encryption: yvmG/RnzKrbCb9L3mgsmoxXr9H7i2Z4qlbT0mL3ln4w= # CHANGE THIS VALUE
hash: KXYiG07wC7GIgvCSdg+WmyWdXDAn6XKYJtp/wkEU7x573+byBRAYtpTP0wwvi8i/4l37uicX1dVTUzwH3sLZyw== # CHANGE THIS VALUE
jwt:
secret: i2DecQmfGreG6q1vBj5tCokhlN41gcfS2cjOs9Po-u8= # CHANGE THIS VALUE
```
The full configuration file can be found [here](https://github.com/ente-io/ente/blob/main/server/configurations/local.yaml)
Then prepare a credentials file as `credentials.yaml`
```yaml
db:
host: postgres
port: 5432
name: <ente_db_name>
user: <pguser>
password: <pgpass>
s3:
# Override the primary and secondary hot storage. The commented out values
# are the defaults.
#
hot_storage:
primary: b2-eu-cen
# secondary: wasabi-eu-central-2-v3
# If true, enable some workarounds to allow us to use a local minio instance
# for object storage.
#
# 1. Disable SSL.
# 2. Use "path" style S3 URLs (see `use_path_style_urls` below).
# 3. Directly download the file during replication instead of going via the
# Cloudflare worker.
# 4. Do not specify storage classes when uploading objects (since minio does
# not support them, specifically it doesn't support GLACIER).
are_local_buckets: true
# To use "path" style S3 URLs instead of DNS-based bucket access
# default to true if you set "are_local_buckets: true"
# use_path_style_urls: true
b2-eu-cen: # Don't change this key, it is hardcoded
key: <keyID>
secret: <keySecret>
endpoint: garage:3900 # publicly accessible endpoint of your garage instance
region: garage
bucket: <yourbucketName>
use_path_style: true
# you can specify secondary locations, names are hardcoded as well
# wasabi-eu-central-2-v3:
# scw-eu-fr-v3:
# and you can also specify a bucket to be used for embeddings, preview etc..
We also need to expose these buckets publicly to serve their content to users:
```bash
garage bucket website --allow pleroma
```
Note the Key ID and Secret Key.
### Configure Pleroma
Update your Pleroma configuration like that in `/etc/pleroma/config.exs`.
```
config :pleroma, Pleroma.Upload,
uploader: Pleroma.Uploaders.S3,
base_url: "https://pleroma.garage.example.tld"
config :ex_aws, :s3,
access_key_id: "GW...",
secret_access_key: "XXX",
region: "garage",
host: "api.garage.example.tld"
```
And restart Pleroma.
You can found more information in [Pleroma Documentation > Pleroma.Uploaders.S3](https://docs-develop.pleroma.social/backend/configuration/cheatsheet/#pleromauploaderss3)
### Migrating your data
Pleroma have an internal migration tool that can encounter some fatal error
```
** (EXIT from #PID<0.98.0>) an exception was raised:
** (File.Error) could not stream "/var/lib/pleroma/uploads/09/f8": illegal operation on a directory
(elixir 1.17.3) lib/file/stream.ex:100: anonymous fn/3 in Enumerable.File.Stream.reduce/3
(elixir 1.17.3) lib/stream.ex:1675: anonymous fn/5 in Stream.resource/3
On Plakar server, add your Garage as a storage location:
```bash
plakar store add garageS3 s3://my-garage.tld/plakar-backups \
region=garage # Or as you've specified in garage.toml \
access_key=<Key ID from "garage key info my-plakar-key"> \
secret_access_key=<Secret key from "garage key info my-plakar-key">
```
Then create the repository.
```bash
plakar at @garageS3 create -plaintext # Unencrypted
# or
plakar at @garageS3 create #encrypted
```
If you encrypt your backups (Plakar default), you will need to define a strong passphrase. Do not forget to save your password safely. It will be needed to decrypt your backups.
After the repository has been created, check that everything works as expected (that might give an empty result as no file has been added yet, but no error message):
```bash
plakar at @garageS3 check
```
Now that everything is configure, you can use Garage as your backups storage. For instance sync it with a local backup storage:
```bash
$ plakar at ~/backups sync to @garageS3
```
Or list the S3 storage content:
```bash
$ plakar at @garageS3 ls
```
More information in Plakar documentation: https://www.plakar.io/docs/main/quickstart/
## Synology HyperBackup
HyperBackup can be configured to upload backups to garage using a custom S3 destination. However, the HyperBackup client hardcodes the `us-east-1` region that is a critical input to the v4 signature process. If garage is not set to `us-east-1`, HyperBackup will recognize available buckets, but fail during the final setup stage.
@@ -12,6 +12,7 @@ These tools are particularly suitable for debug, backups, website deployments or
| [AWS CLI](#aws-cli) | ✅ | Recommended |
| [rclone](#rclone) | ✅ | |
| [s3cmd](#s3cmd) | ✅ | |
| [s5cmd](#s5cmd) | ✅ | |
| [(Cyber)duck](#cyberduck) | ✅ | |
| [WinSCP (libs3)](#winscp) | ✅ | CLI instructions only |
| [sftpgo](#sftpgo) | ✅ | |
@@ -40,7 +41,7 @@ Some commands:
# list buckets
mc ls garage/
# list objets in a bucket
# list objects in a bucket
mc ls garage/my_files
# copy from your filesystem to garage
@@ -69,16 +70,17 @@ Then a file named `~/.aws/config` and put:
```toml
[default]
region=garage
endpoint_url=http://127.0.0.1:3900
```
Now, supposing Garage is listening on `http://127.0.0.1:3900`, you can list your buckets with:
```bash
aws --endpoint-url http://127.0.0.1:3900 s3 ls
aws s3 ls
```
Passing the `--endpoint-url` parameter to each command is annoying but AWS developers do not provide a corresponding configuration entry.
As a workaround, you can redefine the aws command by editing the file `~/.bashrc`:
If you're using awscli `<1.29.0` or `<2.13.0`, you need to pass `--endpoint-url` to each CLI invocation explicitly.
As a workaround, you can redefine the aws command by editing the file `~/.bashrc` in this case:
```
function aws { command aws --endpoint-url http://127.0.0.1:3900 $@ ; }
@@ -147,6 +149,15 @@ rclone help
This will tremendously accelerate operations such as `rclone sync` or `rclone ncdu` by reducing the number
of ListObjects calls that are made.
**Garage behind Cloudflare proxy:** when running Garage behind Cloudflare proxy, you might see `Response: error 403 Forbidden, Forbidden: Invalid signature` error in your garage logs or `AccessDenied: Forbidden: Invalid signature` error in rclone logs. Try adding `--s3-sign-accept-encoding=false` flag to your rclone command and see if the issue is resolved.
If you are developping a new application, you may want to use Garage to store your user's media.
The S3 API that Garage uses is a standard REST API, so as long as you can make HTTP requests,
you can query it. You can check the [S3 REST API Reference](https://docs.aws.amazon.com/AmazonS3/latest/API/API_Operations_Amazon_Simple_Storage_Service.html) from Amazon to learn more.
Developping your own wrapper around the REST API is time consuming and complicated.
Instead, there are some libraries already avalaible.
Some of them are maintained by Amazon, some by Minio, others by the community.
@@ -17,13 +17,13 @@ Garage can also help you serve this content.
## Gitea
You can use Garage with Gitea to store your [git LFS](https://git-lfs.github.com/) data, your users' avatar, and their attachements.
You can use Garage with Gitea to store your [git LFS](https://git-lfs.github.com/) data, your users' avatar, and their attachments.
You can configure a different target for each data type (check `[lfs]` and `[attachment]` sections of the Gitea documentation) and you can provide a default one through the `[storage]` section.
Let's start by creating a key and a bucket (your key id and secret will be needed later, keep them somewhere):
A cookbook, when you cook, is a collection of recipes.
Similarly, Garage's cookbook contains a collection of recipes that are known to works well!
Similarly, Garage's cookbook contains a collection of recipes that are known to work well!
This chapter could also be referred as "Tutorials" or "Best practices".
- **[Multi-node deployment](@/documentation/cookbook/real-world.md):** This page will walk you through all of the necessary
@@ -16,6 +16,10 @@ This chapter could also be referred as "Tutorials" or "Best practices".
source in case a binary is not provided for your architecture, or if you want to
hack with us!
- **[Binary packages](@/documentation/cookbook/binary-packages.md):** This page
lists the different platforms that provide ready-built software packages for
Garage.
- **[Integration with Systemd](@/documentation/cookbook/systemd.md):** This page explains how to run Garage
as a Systemd service (instead of as a Docker container).
@@ -26,6 +30,10 @@ This chapter could also be referred as "Tutorials" or "Best practices".
- **[Configuring a reverse-proxy](@/documentation/cookbook/reverse-proxy.md):** This page explains how to configure a reverse-proxy to add TLS support to your S3 api endpoint.
- **[Recovering from failures](@/documentation/cookbook/recovering.md):** Garage's first selling point is resilience
to hardware failures. This section explains how to recover from such a failure in the
best possible way.
- **[Deploying on Kubernetes](@/documentation/cookbook/kubernetes.md):** This page explains how to deploy Garage on Kubernetes using our Helm chart.
- **[Deploying with Ansible](@/documentation/cookbook/ansible.md):** This page lists available Ansible roles developed by the community to deploy Garage.
- **[Monitoring Garage](@/documentation/cookbook/monitoring.md)** This page
explains the Prometheus metrics available for monitoring the Garage
There are two methods to expose buckets as website:
There are three methods to expose buckets as website:
1. using the PutBucketWebsite S3 API call, which is allowed for access keys that have the owner permission bit set
2. from the Garage CLI, by an adminstrator of the cluster
2. from the Garage CLI, by an administrator of the cluster
3. using the Garage administration API
The `PutBucketWebsite` API endpoint [is documented](https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutBucketWebsite.html) in the official AWS docs.
This endpoint can also be called [using `aws s3api`](https://docs.aws.amazon.com/cli/latest/reference/s3api/put-bucket-website.html) on the command line.
# This will build ONLY feature1, feature2 and feature3
cargo build --release --no-default-features \
--features feature1,feature2,feature3
```
The following feature flags are available in v0.8.0:
| Feature flag | Enabled | Description |
| ------------ | ------- | ----------- |
| `bundled-libs` | *by default* | Use bundled version of sqlite3, zstd, lmdb and libsodium |
| `consul-discovery` | optional | Enable automatic registration and discovery<br>of cluster nodes through the Consul API |
| `fjall` | experimental | Enable using Fjall to store Garage's metadata |
| `journald` | optional | Enable logging to systemd-journald with<br>`GARAGE_LOG_TO_JOURNALD=true` environment variable set |
| `k2v` | optional | Enable the experimental K2V API (if used, all nodes on your<br>Garage cluster must have it enabled as well) |
| `kubernetes-discovery` | optional | Enable automatic registration and discovery<br>of cluster nodes through the Kubernetes API |
| `lmdb` | *by default* | Enable using LMDB to store Garage's metadata |
| `metrics` | *by default* | Enable collection of metrics in Prometheus format on the admin API |
| `sqlite` | *by default* | Enable using Sqlite3 to store Garage's metadata |
| `syslog` | optional | Enable logging to Syslog with<br>`GARAGE_LOG_TO_SYSLOG=true` environment variable set |
| `system-libs` | optional | Use system version of sqlite3, zstd, lmdb and libsodium<br>if available (exclusive with `bundled-libs`, build using<br>`cargo build --no-default-features --features system-libs`) |
| `telemetry-otlp` | optional | Enable collection of execution traces using OpenTelemetry |
@@ -21,7 +21,7 @@ You can configure Garage as a gateway on all nodes that will consume your S3 API
The instructions are similar to a regular node, the only option that is different is while configuring the node, you must set the `--gateway` parameter:
If you want to manage the CustomResourceDefinition used by garage for its `kubernetes_discovery` outside of the helm chart, add `garage.kubernetesSkipCrd: true` to your custom values and use the kustomization before deploying the helm chart:
After deploying, cluster layout must be configured manually as described in [Creating a cluster layout](@/documentation/quick-start/_index.md#creating-a-cluster-layout). Use the following command to access garage CLI:
```bash
kubectl exec --stdin --tty -n garage garage-0 -- ./garage status
```
## Overriding default values
All possible configuration values can be found with:
```bash
helm show values ./garage
```
This is an example `values.override.yaml` for deploying in a microk8s cluster with a https s3 api ingress route:
```yaml
garage:
# Use only 2 replicas per object
replicationFactor:2
# Start 4 instances (StatefulSets) of garage
deployment:
replicaCount:4
# Override default storage class and size
persistence:
meta:
storageClass:"openebs-hostpath"
size:100Mi
data:
storageClass:"openebs-hostpath"
size:1Gi
ingress:
s3:
api:
enabled:true
className:"public"
annotations:
cert-manager.io/cluster-issuer:"letsencrypt-prod"
nginx.ingress.kubernetes.io/proxy-body-size:500m
hosts:
- host:s3-api.my-domain.com
paths:
- path:/
pathType:Prefix
tls:
- secretName:garage-ingress-cert
hosts:
- s3-api.my-domain.com
```
## Removing
```bash
helm delete --namespace garage garage
```
Note that this will leave behind custom CRD `garagenodes.deuxfleurs.fr`, which must be removed manually if desired.
## Increase PVC size on running Garage instances
Since the Garage Helm chart creates the data and meta PVC based on `StatefulSet` templates, increasing the PVC size can be a bit tricky.
### Confirm the `StorageClass` used for Garage supports volume expansion
Confirm the storage class used for garage.
```bash
kubectl -n garage get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE
If your `StorageClass` does not support volume expansion, double check if you can enable it. Otherwise, your only real option is to spin up a new Garage cluster with increased size and migrate all data over.
If your `StorageClass` supports expansion, you are free to continue.
### Increase the size of the PVCs
Increase the size of all PVCs to your desired size.
```bash
kubectl -n garage edit pvc data-garage-0
kubectl -n garage edit pvc data-garage-1
kubectl -n garage edit pvc data-garage-2
kubectl -n garage edit pvc meta-garage-0
kubectl -n garage edit pvc meta-garage-1
kubectl -n garage edit pvc meta-garage-2
```
### Increase the size of the `StatefulSet` PVC template
This is an optional step, but if not done, future instances of Garage will be created with the original size from the template.
This will remove the Garage `StatefulSet` but leave the pods running. It may seem destructive but needs to be done this way since edits to the size of PVC templates are prohibited.
### Redeploy the `StatefulSet`
Now the size of future PVCs can be increased, and the Garage Helm chart can be upgraded. The new `StatefulSet` should take ownership of the orphaned pods again.
Garage exposes some internal metrics in the Prometheus data format.
This page explains how to exploit these metrics.
## Setting up monitoring
### Enabling the Admin API endpoint
If you have not already enabled the [administration API endpoint](@/documentation/reference-manual/admin-api.md), do so by adding the following lines to your configuration file:
```toml
[admin]
api_bind_addr="0.0.0.0:3903"
```
This will allow anyone to scrape Prometheus metrics by fetching
`http://localhost:3903/metrics`. If you want to restrict access
to the exported metrics, set the `metrics_token` configuration value
to a bearer token to be used when fetching the metrics endpoint.
### Setting up Prometheus and Grafana
Add a scrape config to your Prometheus daemon to scrape metrics from
all of your nodes:
```yaml
scrape_configs:
- job_name:'garage'
static_configs:
- targets:
- 'node1.mycluster:3903'
- 'node2.mycluster:3903'
- 'node3.mycluster:3903'
```
If you have set a metrics token in your Garage configuration file,
add the following lines in your Prometheus scrape config:
```yaml
authorization:
type:Bearer
credentials:'your metrics token'
```
To visualize the scraped data in Grafana,
you can either import our [Grafana dashboard for Garage](https://git.deuxfleurs.fr/Deuxfleurs/garage/raw/branch/main/script/telemetry/grafana-garage-dashboard-prometheus.json)
or make your own.
The list of exported metrics is available on our [dedicated page](@/documentation/reference-manual/monitoring.md) in the Reference manual section.
@@ -11,21 +11,23 @@ We recommend first following the [quick start guide](@/documentation/quick-start
to get familiar with Garage's command line and usage patterns.
## Preparing your environment
## Prerequisites
### Prerequisites
To run a real-world deployment, make sure the following conditions are met:
- You have at least three machines with sufficient storage space available.
- Each machine has a public IP address which is reachable by other machines.
Running behind a NAT is likely to be possible but hasn't been tested for the latest version (TODO).
- Each machine has an IP address which makes it directly reachable by all other machines.
In many cases, nodes will be behind a NAT and will not each have a public
IPv4 addresses. In this case, is recommended that you use IPv6 for this
end-to-end connectivity if it is available. Otherwise, using a mesh VPN such as
[Nebula](https://github.com/slackhq/nebula) or
[Yggdrasil](https://yggdrasil-network.github.io/) are approaches to consider
in addition to building out your own VPN tunneling.
-Ideally, each machine should have a SSD available in addition to the HDD you are dedicating
to Garage. This will allow for faster access to metadata and has the potential
to significantly reduce Garage's response times.
- This guide will assume you are using Docker containers to deploy Garage on each node.
-This guide will assume you are using Docker containers to deploy Garage on each node.
Garage can also be run independently, for instance as a [Systemd service](@/documentation/cookbook/systemd.md).
You can also use an orchestrator such as Nomad or Kubernetes to automatically manage
Docker containers on a fleet of nodes.
@@ -41,7 +43,7 @@ For our example, we will suppose the following infrastructure with IPv6 connecti
| Brussels | Mars | fc00:F::1 | 1.5 TB |
Note that Garage will **always** store the three copies of your data on nodes at different
locations. This means that in the case of this small example, the available capacity
locations. This means that in the case of this small example, the usable capacity
of the cluster is in fact only 1.5 TB, because nodes in Brussels can't store more than that.
This also means that nodes in Paris and London will be under-utilized.
To make better use of the available hardware, you should ensure that the capacity
@@ -49,17 +51,59 @@ available in the different locations of your cluster is roughly the same.
For instance, here, the Mercury node could be moved to Brussels; this would allow the cluster
to store 2 TB of data in total.
### Best practices
- If you have reasonably fast networking between all your nodes, and are planing to store
mostly large files, bump the `block_size` configuration parameter to 10 MB
(`block_size = "10M"`).
- Garage stores its files in two locations: it uses a metadata directory to store frequently-accessed
small metadata items, and a data directory to store data blocks of uploaded objects.
Ideally, the metadata directory would be stored on an SSD (smaller but faster),
and the data directory would be stored on an HDD (larger but slower).
- For the data directory, Garage already does checksumming and integrity verification,
so there is no need to use a filesystem such as BTRFS or ZFS that does it.
We recommend using XFS for the data partition, as it has the best performance.
EXT4 is not recommended as it has more strict limitations on the number of inodes,
which might cause issues with Garage when large numbers of objects are stored.
- Servers with multiple HDDs are supported natively by Garage without resorting
to RAID, see [our dedicated documentation page](@/documentation/operations/multi-hdd.md).
- For the metadata storage, Garage does not do checksumming and integrity
verification on its own, so it is better to use a robust filesystem such as
BTRFS or ZFS. Users have reported that when using the LMDB database engine
(the default), database files have a tendency of becoming corrupted after an
unclean shutdown (e.g. a power outage), so you should take regular snapshots
to be able to recover from such a situation. This can be done using Garage's
built-in automatic snapshotting (since v0.9.4), or by using filesystem level
snapshots. If you cannot do so, you might want to switch to Sqlite which is
more robust.
- LMDB is the fastest and most tested database engine, but it has the following
weaknesses: 1/ data files are not architecture-independent, you cannot simply
move a Garage metadata directory between nodes running different architectures,
and 2/ LMDB is not suited for 32-bit platforms. Sqlite is a viable alternative
if any of these are of concern.
- If you only have an HDD and no SSD, it's fine to put your metadata alongside
the data on the same drive, but then consider your filesystem choice wisely
(see above). Having lots of RAM for your kernel to cache the metadata will
help a lot with performance. The default LMDB database engine is the most
tested and has good performance.
## Get a Docker image
Our docker image is currently named `dxflrs/amd64_garage` and is stored on the [Docker Hub](https://hub.docker.com/r/dxflrs/amd64_garage/tags?page=1&ordering=last_updated).
We encourage you to use a fixed tag (eg. `v0.4.0`) and not the `latest` tag.
For this example, we will use the latest published version at the time of the writing which is `v0.4.0` but it's up to you
to check [the most recent versions on the Docker Hub](https://hub.docker.com/r/dxflrs/amd64_garage/tags?page=1&ordering=last_updated).
Our docker image is currently named `dxflrs/garage` and is stored on the [Docker Hub](https://hub.docker.com/r/dxflrs/garage/tags?page=1&ordering=last_updated).
We encourage you to use a fixed tag (eg. `v2.3.0`) and not the `latest` tag.
For this example, we will use the latest published version at the time of the writing which is `v2.3.0` but it's up to you
to check [the most recent versions on the Docker Hub](https://hub.docker.com/r/dxflrs/garage/tags?page=1&ordering=last_updated).
For example:
```
sudo docker pull dxflrs/amd64_garage:v0.4.0
docker pull dxflrs/garage:v2.3.0
```
## Deploying and configuring Garage
@@ -76,13 +120,15 @@ especially you must consider the following folders/files:
this folder will be your main data storage and must be on a large storage (e.g. large HDD)
A valid `/etc/garage/garage.toml` for our cluster would look as follows:
A valid `/etc/garage.toml` for our cluster would look as follows:
```toml
metadata_dir="/var/lib/garage/meta"
data_dir="/var/lib/garage/data"
db_engine="lmdb"
metadata_auto_snapshot_interval="6h"
replication_mode="3"
replication_factor=3
compression_level=2
@@ -90,8 +136,6 @@ rpc_bind_addr = "[::]:3901"
rpc_public_addr="<this node's public IP>:3901"
rpc_secret="<RPC secret>"
bootstrap_peers=[]
[s3_api]
s3_region="garage"
api_bind_addr="[::]:3900"
@@ -108,6 +152,8 @@ Check the following for your configuration files:
- Make sure `rpc_public_addr` contains the public IP address of the node you are configuring.
This parameter is optional but recommended: if your nodes have trouble communicating with
one another, consider adding it.
Alternatively, you can also set `rpc_public_addr_subnet`, which can filter
the addresses announced to other peers to a specific subnet.
- Make sure `rpc_secret` is the same value on all nodes. It should be a 32-bytes hex-encoded secret key.
You can generate such a key with `openssl rand -hex 32`.
@@ -125,19 +171,37 @@ docker run \
-v /etc/garage.toml:/etc/garage.toml \
-v /var/lib/garage/meta:/var/lib/garage/meta \
-v /var/lib/garage/data:/var/lib/garage/data \
lxpz/garage_amd64:v0.4.0
dxflrs/garage:v2.3.0
```
It should be restarted automatically at each reboot.
Please note that we use host networking as otherwise Docker containers
can not communicate with IPv6.
With this command line, Garage should be started automatically at each boot.
Please note that we use host networking as otherwise the network indirection
added by Docker would prevent Garage nodes from communicating with one another
(especially if using IPv6).
Upgrading between Garage versions should be supported transparently,
but please check the relase notes before doing so!
To upgrade, simply stop and remove this container and
start again the command with a new version of Garage.
If you want to use `docker-compose`, you may use the following `docker-compose.yml` file as a reference:
## Controling the daemon
```yaml
version:"3"
services:
garage:
image:dxflrs/garage:v2.3.0
network_mode:"host"
restart:unless-stopped
volumes:
- /etc/garage.toml:/etc/garage.toml
- /var/lib/garage/meta:/var/lib/garage/meta
- /var/lib/garage/data:/var/lib/garage/data
```
If you wish to upgrade your cluster, make sure to read the corresponding
[documentation page](@/documentation/operations/upgrading.md) first, as well as
the documentation relevant to your version of Garage in the case of major
upgrades. With the containerized setup proposed here, the upgrade process
will require stopping and removing the existing container, and re-creating it
with the upgraded version.
## Controlling the daemon
The `garage` binary has two purposes:
- it acts as a daemon when launched with `garage server`
@@ -146,6 +210,12 @@ The `garage` binary has two purposes:
Ensure an appropriate `garage` binary (the same version as your Docker image) is available in your path.
If your configuration file is at `/etc/garage.toml`, the `garage` binary should work with no further change.
You can also use an alias as follows to use the Garage binary inside your docker container:
@@ -206,10 +302,15 @@ To add a new website, add the following declaration to your Traefik configuratio
```toml
[http.routers]
[http.routers.garage-s3]
rule="Host(`s3.example.org`)"
service="garage-s3-service"
entryPoints=["websecure"]
[http.routers.my_website]
rule="Host(`yoururl.example.org`)"
service="my_garage_service"
entryPoints=["web"]
service="garage-web-service"
entryPoints=["websecure"]
```
Enable HTTPS access to your website with the following configuration section ([documentation](https://doc.traefik.io/traefik/https/overview/)):
@@ -222,7 +323,7 @@ Enable HTTPS access to your website with the following configuration section ([d
...
```
### Adding gzip compression
### Adding compression
Add the following configuration section [to compress response](https://doc.traefik.io/traefik/middlewares/http/compress/) using [gzip](https://developer.mozilla.org/en-US/docs/Glossary/GZip_compression) before sending them to the client:
@@ -230,15 +331,15 @@ Add the following configuration section [to compress response](https://doc.traef
[http.routers]
[http.routers.my_website]
...
middlewares=["gzip_compress"]
middlewares=["compression"]
...
[http.middlewares]
[http.middlewares.gzip_compress.compress]
[http.middlewares.compression.compress]
```
### Add caching response
Traefik's caching middleware is only available on [entreprise version](https://doc.traefik.io/traefik-enterprise/middlewares/http-cache/), however the freely-available [Souin plugin](https://github.com/darkweak/souin#tr%C3%A6fik-container) can also do the job. (section to be completed)
Traefik's caching middleware is only available on [enterprise version](https://doc.traefik.io/traefik-enterprise/middlewares/http-cache/), however the freely-available [Souin plugin](https://github.com/darkweak/souin#tr%C3%A6fik-container) can also do the job. (section to be completed)
### Complete example
@@ -258,25 +359,212 @@ Traefik's caching middleware is only available on [entreprise version](https://d
But at the same time, the `reverse_proxy` is very flexible.
For a production deployment, you should [read its documentation](https://caddyserver.com/docs/caddyfile/directives/reverse_proxy) as it supports features like DNS discovery of upstreams, load balancing with checks, streaming parameters, etc.
### Caching
Caddy can compiled with a
[cache plugin](https://github.com/caddyserver/cache-handler) which can be used
to provide a hot-cache at the webserver-level for static websites hosted by
Garage.
This can be configured as follows:
```caddy
# Caddy global configuration section
{
# Bare minimum configuration to enable cache.
ordercachebeforerewrite
cache
#cache
# allowed_http_verbs GET
# default_cache_control public
# ttl 8h
#}
}
# Site specific section
https://{
cache
#cache {
# timeout {
# backend 30s
# }
#}
reverse_proxy...
}
```
Caching is a complicated subject, and the reader is encouraged to study the
available options provided by the plugin.
### On-demand TLS
Caddy supports a technique called
[on-demand TLS](https://caddyserver.com/docs/automatic-https#on-demand-tls), by
which one can configure the webserver to provision TLS certificates when a
client first connects to it.
In order to prevent an attack vector whereby domains are simply pointed at your
webserver and certificates are requested for them - Caddy can be configured to
ask Garage if a domain is authorized for web hosting, before it then requests
a TLS certificate.
This 'check' endpoint, which is on the admin port (3903 by default), can be
configured in Caddy's global section as follows:
```caddy
{
...
on_demand_tls{
askhttp://localhost:3903/check
interval2m
burst5
}
...
}
```
The host section can then be configured with (note that this uses the web
*A note on hardening: garage will be run as a non privileged user, its user id is dynamically allocated by systemd. It cannot access (read or write) home folders (/home, /root and /run/user), the rest of the filesystem can only be read but not written, only the path seen as /var/lib/garage is writable as seen by the service (mapped to /var/lib/private/garage on your host). Additionnaly, the process can not gain new privileges over time.*
**A note on hardening:** Garage will be run as a non privileged user, its user
id is dynamically allocated by systemd (set with `DynamicUser=true`). It cannot
access (read or write) home folders (`/home`, `/root` and `/run/user`), the
rest of the filesystem can only be read but not written, only the path seen as
`/var/lib/garage` is writable as seen by the service. Additionally, the process
can not gain new privileges over time.
For this to work correctly, your `garage.toml` must be set with
`metadata_dir=/var/lib/garage/meta` and `data_dir=/var/lib/garage/data`. This
is mandatory to use the DynamicUser hardening feature of systemd, which
autocreates these directories as virtual mapping. If the directory
`/var/lib/garage` already exists before starting the server for the first time,
the systemd service might not start correctly. Note that in your host
filesystem, Garage data will be held in `/var/lib/private/garage`.
To start the service then automatically enable it at boot:
Garage is a stateful clustered application, where all nodes are communicating together and share data structures.
It makes upgrade more difficult than stateless applications so you must be more careful when upgrading.
On a new version release, there is 2 possibilities:
- protocols and data structures remained the same ➡️ this is a **straightforward upgrade**
- protocols or data structures changed ➡️ this is an **advanced upgrade**
You can quickly now what type of update you will have to operate by looking at the version identifier.
Following the [SemVer ](https://semver.org/) terminology, if only the *patch* number changed, it will only need a straightforward upgrade.
Example: an upgrade from v0.6.0 from v0.6.1 is a straightforward upgrade.
If the *minor* or *major* number changed however, you will have to do an advanced upgrade. Example: from v0.6.1 to v0.7.0.
Migrations are designed to be run only between contiguous versions (from a *major*.*minor* perspective, *patches* can be skipped).
Example: migrations from v0.6.1 to v0.7.0 and from v0.6.0 to v0.7.0 are supported but migrations from v0.5.0 to v0.7.0 are not supported.
## Straightforward upgrades
Straightforward upgrades do not imply cluster downtime.
Before upgrading, you should still read [the changelog](https://git.deuxfleurs.fr/Deuxfleurs/garage/releases) and ideally test your deployment on a staging cluster before.
When you are ready, start by checking the health of your cluster.
You can force some checks with `garage repair`, we recommend at least running `garage repair --all-nodes --yes` that is very quick to run (less than a minute).
You will see that the command correctly terminated in the logs of your daemon.
Finally, you can simply upgrades nodes one by one.
For each node: stop it, install the new binary, edit the configuration if needed, restart it.
## Advanced upgrades
Advanced upgrades will imply cluster downtime.
Before upgrading, you must read [the changelog](https://git.deuxfleurs.fr/Deuxfleurs/garage/releases) and you must test your deployment on a staging cluster before.
From a high level perspective, an advanced upgrade looks like this:
1. Make sure the health of your cluster is good (see `garage repair`)
2. Disable API access (comment the configuration in your reverse proxy)
3. Check that your cluster is idle
4. Stop the whole cluster
5. Backup the metadata folder of all your nodes, so that you will be able to restore it quickly if the upgrade fails (blocks being immutable, they should not be impacted)
6. Install the new binary, update the configuration
7. Start the whole cluster
8. If needed, run the corresponding migration from `garage migrate`
9. Make sure the health of your cluster is good
10. Enable API access (uncomment the configuration in your reverse proxy)
11. Monitor your cluster while load comes back, check that all your applications are happy with this new version
We write guides for each advanced upgrade, they are stored under the "Working Documents" section of this documentation.
@@ -10,7 +10,7 @@ perspective. It will allow you to understand if Garage is a good fit for
you, how to better use it, how to contribute to it, what can Garage could
and could not do, etc.
- **[Goals and use cases](@/documentation/design/goals.md):** This page explains why Garage was concieved and what practical use cases it targets.
- **[Goals and use cases](@/documentation/design/goals.md):** This page explains why Garage was conceived and what practical use cases it targets.
- **[Related work](@/documentation/design/related-work.md):** This pages presents the theoretical background on which Garage is built, and describes other software storage solutions and why they didn't work for us.
@@ -20,12 +20,14 @@ and could not do, etc.
We love to talk and hear about Garage, that's why we keep a log here:
- [(en, 2023-01-18) Presentation of Garage with some details on CRDTs and data partitioning among nodes](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/commit/4cff37397f626ef063dad29e5b5e97ab1206015d/doc/talks/2023-01-18-tocatta/talk.pdf)
- [(fr, 2022-11-19) De l'auto-hébergement à l'entre-hébergement : Garage, pour conserver ses données ensemble](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/commit/4cff37397f626ef063dad29e5b5e97ab1206015d/doc/talks/2022-11-19-Capitole-du-Libre/pr%C3%A9sentation.pdf)
- [(en, 2022-06-23) General presentation of Garage](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/commit/4cff37397f626ef063dad29e5b5e97ab1206015d/doc/talks/2022-06-23-stack/talk.pdf)
- [(fr, 2021-11-13, video) Garage : Mille et une façons de stocker vos données](https://video.tedomum.net/w/moYKcv198dyMrT8hCS5jz9) and [slides (html)](https://rfid.deuxfleurs.fr/presentations/2021-11-13/garage/) - during [RFID#1](https://rfid.deuxfleurs.fr/programme/2021-11-13/) event
- [(en, 2021-04-28) Distributed object storage is centralised](https://git.deuxfleurs.fr/Deuxfleurs/garage/raw/commit/b1f60579a13d3c5eba7f74b1775c84639ea9b51a/doc/talks/2021-04-28_spirals-team/talk.pdf)
- [(fr, 2020-12-02) Garage : jouer dans la cour des grands quand on est un hébergeur associatif](https://git.deuxfleurs.fr/Deuxfleurs/garage/raw/commit/b1f60579a13d3c5eba7f74b1775c84639ea9b51a/doc/talks/2020-12-02_wide-team/talk.pdf)
*Did you write or talk about Garage? [Open a pull request](https://git.deuxfleurs.fr/Deuxfleurs/garage/) to add a link here!*
- [(en, 2021-04-28) Distributed object storage is centralised](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/commit/b1f60579a13d3c5eba7f74b1775c84639ea9b51a/doc/talks/2021-04-28_spirals-team/talk.pdf)
- [(fr, 2020-12-02) Garage : jouer dans la cour des grands quand on est un hébergeur associatif](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/commit/b1f60579a13d3c5eba7f74b1775c84639ea9b51a/doc/talks/2020-12-02_wide-team/talk.pdf)
With Garage, we wanted to build a software defined storage service that follow the [KISS principle](https://en.wikipedia.org/wiki/KISS_principle),
@@ -15,14 +15,14 @@ The more a user request will require intra-cluster requests to complete, the mor
This is especially true for sequential requests: requests that must wait the result of another request to be sent.
We designed Garage without consensus algorithms (eg. Paxos or Raft) to minimize the number of sequential and parallel requests.
This serie of benchmarks quantifies the impact of this design choice.
This series of benchmarks quantifies the impact of this design choice.
### On a simple simulated network
We start with a controlled environment, all the instances are running on the same (powerful enough) machine.
To control the network latency, we simulate the network with [mknet](https://git.deuxfleurs.fr/trinity-1686a/mknet) (a tool we developped, based on `tc` and the linux network stack).
To mesure S3 endpoints latency, we use our own tool [s3lat](https://git.deuxfleurs.fr/quentin/s3lat/) to observe only the intra-cluster latency and not some contention on the nodes (CPU, RAM, disk I/O, network bandwidth, etc.).
To control the network latency, we simulate the network with [mknet](https://git.deuxfleurs.fr/trinity-1686a/mknet) (a tool we developed, based on `tc` and the linux network stack).
To measure S3 endpoints latency, we use our own tool [s3lat](https://git.deuxfleurs.fr/quentin/s3lat/) to observe only the intra-cluster latency and not some contention on the nodes (CPU, RAM, disk I/O, network bandwidth, etc.).
Compared to other benchmark tools, S3Lat sends only one (small) request at the same time and measures its latency.
We selected 5 standard endpoints that are often in the critical path: ListBuckets, ListObjects, GetObject, PutObject and RemoveObject.
@@ -32,7 +32,7 @@ In this first benchmark, we consider 5 instances that are located in a different
Compared to garage, minio latency drastically increases on 3 endpoints: GetObject, PutObject, RemoveObject.
We suppose that these requests on minio make transactions over Raft, involving 4 sequential requests: 1) sending the message to the leader, 2) having the leader dispatch it to the other nodes, 3) waiting for the confirmation of followers and finally 4) commiting it. With our current configuration, one Raft transaction will take around 400 ms. GetObject seems to correlate to 1 transaction while PutObject and RemoveObject seems to correlate to 2 or 3. Reviewing minio code would be required to confirm this hypothesis.
We suppose that these requests on minio make transactions over Raft, involving 4 sequential requests: 1) sending the message to the leader, 2) having the leader dispatch it to the other nodes, 3) waiting for the confirmation of followers and finally 4) committing it. With our current configuration, one Raft transaction will take around 400 ms. GetObject seems to correlate to 1 transaction while PutObject and RemoveObject seems to correlate to 2 or 3. Reviewing minio code would be required to confirm this hypothesis.
Conversely, garage uses an architecture similar to DynamoDB and never require global cluster coordination to answer a request.
Instead, garage can always contact the right node in charge of the requested data, and can answer in as low as one request in the case of GetObject and PutObject. We also observed that Garage latency, while often lower to minio, is more dispersed: garage is still in beta and has not received any performance optimization yet.
@@ -50,7 +50,7 @@ We plot a similar graph as before:
This new graph is very similar to the one before, neither minio or garage seems to benefit from this new topology, but they also do not suffer from it.
Considering garage, this is expected: nodes in the same DC are put in the same zone, and then data are spread on different zones for data resiliency and availaibility.
Considering garage, this is expected: nodes in the same DC are put in the same zone, and then data are spread on different zones for data resiliency and availability.
Then, in the default mode, requesting data requires to query at least 2 zones to be sure that we have the most up to date information.
These requests will involve at least one inter-DC communication.
In other words, we prioritize data availability and synchronization over raw performances.
object storage protocole. It enables applications to store large blobs such
object storage protocol. It enables applications to store large blobs such
as pictures, video, images, documents, etc., in a redundant multi-node
setting. S3 is versatile enough to also be used to publish a static
website.
Garage is an opinionated object storage solutoin, we focus on the following **desirable properties**:
Garage is an opinionated object storage solution, we focus on the following **desirable properties**:
- **Internet enabled**: made for multi-sites (eg. datacenters, offices, households, etc.) interconnected through regular Internet connections.
- **Self-contained & lightweight**: works everywhere and integrates well in existing environments to target [hyperconverged infrastructures](https://en.wikipedia.org/wiki/Hyper-converged_infrastructure).
- **Highly resilient**: highly resilient to network failures, network latency, disk failures, sysadmin failures.
- **Simple**: simple to understand, simple to operate, simple to debug.
- **Internet enabled**: made for multi-sites (eg. datacenters, offices, households, etc.) interconnected through regular Internet connections.
We also noted that the pursuit of some other goals are detrimental to our initial goals.
The following has been identified as **non-goals** (if these points matter to you, you should not use Garage):
@@ -42,15 +42,30 @@ locations. They use Garage themselves for the following tasks:
- As a [Matrix media backend](https://github.com/matrix-org/synapse-s3-storage-provider)
- To store personal data and shared documents through [Bagage](https://git.deuxfleurs.fr/Deuxfleurs/bagage), a homegrown WebDav-to-S3 proxy
- In the Drone continuous integration platform to store task logs
- As a Nix binary cache
-As a backup target using `rclone`
-To store personal data and shared documents through [Bagage](https://git.deuxfleurs.fr/Deuxfleurs/bagage), a homegrown WebDav-to-S3 and SFTP-to-S3 proxy
- As a backup target using `rclone` and `restic`
The Deuxfleurs Garage cluster is a multi-site cluster currently composed of
4 nodes in 2 physical locations. In the future it will be expanded to at
least 3 physical locations to fully exploit Garage's potential for high
availability.
9 nodes in 3 physical locations.
### Triplebit
[Triplebit](https://www.triplebit.org) is a non-profit hosting provider and
ISP focused on improving access to privacy-related services. They use
Garage themselves for the following tasks:
- Hosting of their homepage, [privacyguides.org](https://www.privacyguides.org/), and various other static sites
- As a PowerDNS authoritative zone backend through [Lightning Stream](https://doc.powerdns.com/lightningstream/latest/index.html) and [LMDB](https://doc.powerdns.com/authoritative/backends/lmdb.html)
- As a Mastodon media storage backend for [mstdn.party](https://mstdn.party/) and [mstdn.plus](https://mstdn.plus/)
- As a PeerTube storage backend for [neat.tube](https://neat.tube/)
- As a [Matrix media backend](https://github.com/matrix-org/synapse-s3-storage-provider)
Triplebit's Garage cluster is a multi-site cluster currently composed of
@@ -37,7 +37,7 @@ However, Amazon S3 source code is not open but alternatives were proposed.
We identified Minio, Pithos, Swift and Ceph.
Minio/Ceph enforces a total order, so properties similar to a (relaxed) filesystem.
Swift and Pithos are probably the most similar to AWS S3 with their consistent hashing ring.
However Pithos is not maintained anymore. More precisely the company that published Pithos version 1 has developped a second version 2 but has not open sourced it.
However Pithos is not maintained anymore. More precisely the company that published Pithos version 1 has developed a second version 2 but has not open sourced it.
Some tests conducted by the [ACIDES project](https://acides.org/) have shown that Openstack Swift consumes way more resources (CPU+RAM) that we can afford. Furthermore, people developing Swift have not designed their software for geo-distribution.
There were many attempts in research too. I am only thinking to [LBFS](https://pdos.csail.mit.edu/papers/lbfs:sosp01/lbfs.pdf) that was used as a basis for Seafile. But none of them have been effectively implemented yet.
@@ -63,17 +63,16 @@ Due to its industry oriented design, Ceph is also far from being *Simple* to ope
In a certain way, Ceph and MinIO are closer together than they are from Garage or OpenStack Swift.
**[Pithos](https://github.com/exoscale/pithos):**
Pithos has been abandonned and should probably not used yet, in the following we explain why we did not pick their design.
Pithos has been abandoned and should probably not used yet, in the following we explain why we did not pick their design.
Pithos was relying as a S3 proxy in front of Cassandra (and was working with Scylla DB too).
From its designers' mouth, storing data in Cassandra has shown its limitations justifying the project abandonment.
They built a closed-source version 2 that does not store blobs in the database (only metadata) but did not communicate further on it.
We considered there v2's design but concluded that it does not fit both our *Self-contained & lightweight* and *Simple* properties. It makes the development, the deployment and the operations more complicated while reducing the flexibility.
We considered their v2's design but concluded that it does not fit both our *Self-contained & lightweight* and *Simple* properties. It makes the development, the deployment and the operations more complicated while reducing the flexibility.
**[IPFS](https://ipfs.io/):** IPFS has design goals radically different from Garage, we have [a blog post](@/blog/2022-ipfs/index.md) talking about it.
If you use submodules in your crate (like `crdt` and `replication` in `garage_table`), you must list them in `default.nix`
The Windows target does not work. it might be solvable through [overrides](https://github.com/cargo2nix/cargo2nix/blob/master/overlay/overrides.nix). Indeed, we pass `x86_64-pc-windows-gnu` but mingw need `x86_64-w64-mingw32`
We have a simple [PR on cargo2nix](https://github.com/cargo2nix/cargo2nix/pull/201) that fixes critical bugs but the project does not seem very active currently. We must use [my patched version of cargo2nix](https://github.com/superboum/cargo2nix) to enable i686 and armv6l compilation. We might need to contribute to cargo2nix in the future.
## Nix
Nix has no armv7 + musl toolchains but armv7l is backward compatible with armv6l.
Garage automatically resyncs all entries stored in the metadata tables every hour,
to ensure that all nodes have the most up-to-date version of all the information
they should be holding.
The resync procedure is based on a Merkle tree that allows to efficiently find
differences between nodes.
In some special cases, e.g. before an upgrade, you might want to run a table
resync manually. This can be done using `garage repair tables`.
## Metadata table reference fixes
In some very rare cases where nodes are unavailable, some references between objects
are broken. For instance, if an object is deleted, the underlying versions or data
blocks may still be held by Garage. If you suspect that such corruption has occurred
in your cluster, you can run one of the following repair procedures:
-`garage repair versions`: checks that all versions belong to a non-deleted object, and purges any orphan version
-`garage repair block-refs`: checks that all block references belong to a non-deleted object version, and purges any orphan block reference (this will then allow the blocks to be garbage-collected)
-`garage repair block-rc`: checks that the reference counters for blocks are in sync with the actual number of non-deleted entries in the block reference table
The cluster layout in Garage is a table that assigns to each node a role in
the cluster. The role of a node in Garage can either be a storage node with
a certain capacity, or a gateway node that does not store data and is only
used as an API entry point for faster cluster access.
An introduction to building cluster layouts can be found in the [production deployment](@/documentation/cookbook/real-world.md) page.
In Garage, all of the data that can be stored in a given cluster is divided
into slices which we call *partitions*. Each partition is stored by
one or several nodes in the cluster
(see [`replication_factor`](@/documentation/reference-manual/configuration.md#replication_factor)).
The layout determines the correspondence between these partitions,
which exist on a logical level, and actual storage nodes.
## How cluster layouts work in Garage
A cluster layout is composed of the following components:
- a table of roles assigned to nodes, defined by the user
- an optimal assignation of partitions to nodes, computed by an algorithm that is ran once when calling `garage layout apply` or the ApplyClusterLayout API endpoint
- a version number
Garage nodes will always use the cluster layout with the highest version number.
Garage nodes also maintain and synchronize between them a set of proposed role
changes that haven't yet been applied. These changes will be applied (or
canceled) in the next version of the layout.
All operations on the layout can be realized using the `garage` CLI or using the
[administration API endpoint](@/documentation/reference-manual/admin-api.md).
We give here a description of CLI commands, the admin API semantics are very similar.
The following commands insert modifications to the set of proposed role changes
for the next layout version (but they do not create the new layout immediately):
```bash
garage layout assign [...]
garage layout remove [...]
```
The following command can be used to inspect the layout that is currently set in the cluster
and the changes proposed for the next layout version, if any:
```bash
garage layout show
```
The following commands create a new layout with the specified version number,
that either takes into account the proposed changes or cancels them:
an overview for which Garage versions are currently in use within a cluster.
## Minor upgrades
Minor upgrades do not imply cluster downtime.
Before upgrading, you should still read [the changelog](https://git.deuxfleurs.fr/Deuxfleurs/garage/releases) and ideally test your deployment on a staging cluster before.
When you are ready, start by checking the health of your cluster.
You can force some checks with `garage repair`, we recommend at least running `garage repair --all-nodes --yes tables` which is very quick to run (less than a minute).
You will see that the command correctly terminated in the logs of your daemon, or using `garage worker list` (the repair workers should be in the `Done` state).
Finally, you can simply upgrade nodes one by one.
For each node: stop it, install the new binary, edit the configuration if needed, restart it.
## Major upgrades
Major upgrades can be done with minimal downtime with a bit of preparation, but the simplest way is usually to put the cluster offline for the duration of the migration.
Before upgrading, you must read [the changelog](https://git.deuxfleurs.fr/Deuxfleurs/garage/releases) and you must test your deployment on a staging cluster before.
We write guides for each major upgrade, they are stored under the "Working Documents" section of this documentation.
### Major upgrades with full downtime
From a high level perspective, a major upgrade looks like this:
1. Disable API access (for instance in your reverse proxy, or by commenting the corresponding section in your Garage configuration file and restarting Garage)
2. Check that your cluster is idle
3. Make sure the health of your cluster is good (see `garage repair`)
4. Stop the whole cluster
5. Back up the metadata folder of all your nodes, so that you will be able to restore it if the upgrade fails (data blocks being immutable, they should not be impacted)
6. Install the new binary, update the configuration
7. Start the whole cluster
8. If needed, run the corresponding migration from `garage migrate`
9. Make sure the health of your cluster is good
10. Enable API access (reverse step 1)
11. Monitor your cluster while load comes back, check that all your applications are happy with this new version
### Major upgrades with minimal downtime
There is only one operation that has to be coordinated cluster-wide: the switch of one version of the internal RPC protocol to the next.
This means that an upgrade with very limited downtime can simply be performed from one major version to the next by restarting all nodes
simultaneously in the new version.
The downtime will simply be the time required for all nodes to stop and start again, which should be less than a minute.
If all nodes fail to stop and restart simultaneously, some nodes might be temporarily shut out from the cluster as nodes using different RPC protocol
versions are prevented to talk to one another.
The entire procedure would look something like this:
1. Make sure the health of your cluster is good (see `garage repair`)
2. Take each node offline individually to back up its metadata folder, bring them back online once the backup is done.
You can do all of the nodes in a single zone at once as that won't impact global cluster availability.
Do not try to manually copy the metadata folder of a running node.
**Since Garage v0.9.4,** you can use the `garage meta snapshot --all` command
to take a simultaneous snapshot of the metadata database files of all your
nodes. This avoids the tedious process of having to take them down one by
one before upgrading. Be careful that if automatic snapshotting is enabled,
Garage only keeps the last two snapshots and deletes older ones, so you might
want to disable automatic snapshotting in your upgraded configuration file
until you have confirmed that the upgrade ran successfully. In addition to
snapshotting the metadata databases of your nodes, you should back-up at
least the `cluster_layout` file of one of your Garage instances (this file
should be the same on all nodes and you can copy it safely while Garage is
running).
3. Prepare your binaries and configuration files for the new Garage version
4. Restart all nodes simultaneously in the new version
5. If any specific migration procedure is required, it is usually in one of the two cases:
- It can be run on online nodes after the new version has started, during regular cluster operation.
- it has to be run offline, in which case you will have to again take all nodes offline one after the other to run the repair
For this last step, please refer to the specific documentation pertaining to the version upgrade you are doing.
Consult the full health check API endpoint at /v2/GetClusterHealth for more details
```
The format of the string for a node to connect to is: `<node ID>@<ip address>:<port>`, same as in the `garage node connect` CLI call.
### On-demand TLS `GET /check`
Example response:
To prevent abuse for on-demand TLS, Caddy developers have specified an endpoint that can be queried by the reverse proxy
to know if a given domain is allowed to get a certificate. Garage implements these endpoints to tell if a given domain is handled by Garage or is garbage.
```json
[
{
"success":true,
"error":null
},
{
"success":false,
"error":"Handshake error"
}
]
Garage responds with the following logic:
- If the domain matches the pattern `<bucket-name>.<s3_api.root_domain>`, returns 200 OK
- If the domain matches the pattern `<bucket-name>.<s3_web.root_domain>` and website is configured for `<bucket>`, returns 200 OK
- If the domain matches the pattern `<bucket-name>` and website is configured for `<bucket>`, returns 200 OK
- Otherwise, returns 404 Not Found, 400 Bad Request or 5xx requests.
*Note 1: because in the path-style URL mode, there is only one domain that is not known by Garage, hence it is not supported by this API endpoint.
You must manually declare the domain in your reverse-proxy. Idem for K2V.*
*Note 2: buckets in a user's namespace are not supported yet by this endpoint. This is a limitation of this endpoint currently.*
**Example:** Suppose a Garage instance is configured with `s3_api.root_domain = .s3.garage.localhost` and `s3_web.root_domain = .web.garage.localhost`.
With a private `media` bucket (name in the global namespace, website is disabled), the endpoint will feature the following behavior:
- [Add option for a backend check to approve use of on-demand TLS](https://github.com/caddyserver/caddy/pull/1939)
- [Serving tens of thousands of domains over HTTPS with Caddy](https://caddy.community/t/serving-tens-of-thousands-of-domains-over-https-with-caddy/11179)
and the hashes of individual blocks are used to dispatch them to storage nodes
and to deduplicate them.
### No RAFT slowing you down
It might seem strange to tout the absence of something as a desirable feature,
but this is in fact a very important point! Garage does not use RAFT or another
consensus algorithm internally to order incoming requests: this means that all requests
directed to a Garage cluster can be handled independently of one another instead
of going through a central bottleneck (the leader node).
As a consequence, requests can be handled much faster, even in cases where latency
between cluster nodes is important (see our [benchmarks](@/documentation/design/benchmarks/index.md) for data on this).
This is particularly useful when nodes are far from one another and talk to one other through standard Internet connections.
### Web server for static websites
A storage bucket can easily be configured to be served directly by Garage as a static web site.
Domain names for multiple websites directly map to bucket names, making it easy to build
a platform for your users to autonomously build and host their websites over Garage.
Surprisingly, none of the other alternative S3 implementations we surveyed (such as Minio
or CEPH) support publishing static websites from S3 buckets, a feature that is however
directly inherited from S3 on AWS.
Read more on our [dedicated documentation page](@/documentation/cookbook/exposing-websites.md).
### Bucket names as aliases
In Garage, a bucket may have several names, known as aliases.
Aliases can easily be added and removed on demand:
this allows to easily rename buckets if needed
without having to copy all of their content, something that cannot be done on AWS.
For buckets served as static websites, having multiple aliases for a bucket can allow
exposing the same content under different domain names.
Garage also supports bucket aliases which are local to a single user:
this allows different users to have different buckets with the same name, thus avoiding naming collisions.
This can be helpful for instance if you want to write an application that creates per-user buckets with always the same name.
This feature is totally invisible to S3 clients and does not break compatibility with AWS.
### Cluster administration API
Garage provides a fully-fledged REST API to administer your cluster programmatically.
Functionality included in the admin API include: setting up and monitoring
cluster nodes, managing access credentials, and managing storage buckets and bucket aliases.
A full reference of the administration API is available [here](@/documentation/reference-manual/admin-api.md).
### Metrics and traces
Garage makes some internal metrics available in the Prometheus data format,
which allows you to build interactive dashboards to visualize the load and internal state of your storage cluster.
For developers and performance-savvy administrators,
Garage also supports exporting traces of what it does internally in OpenTelemetry format.
This allows to monitor the time spent at various steps of the processing of requests,
in order to detect potential performance bottlenecks.
### Kubernetes and Nomad integrations
Garage can automatically discover other nodes in the cluster thanks to integration
with orchestrators such as Kubernetes and Nomad (when used with Consul).
This eases the configuration of your cluster as it removes one step where nodes need
to be manually connected to one another.
### Support for changing IP addresses
As long as all of your nodes don't change their IP address at the same time,
Garage should be able to tolerate nodes with changing/dynamic IP addresses,
as nodes will regularly exchange the IP addresses of their peers and try to
reconnect using newer addresses when existing connections are broken.
### K2V API (experimental)
As part of an ongoing research project, Garage can expose an experimental key/value storage API called K2V.
K2V is made for the storage and retrieval of many small key/value pairs that need to be processed in bulk.
This completes the S3 API with an alternative that can be used to easily store and access metadata
related to objects stored in an S3 bucket.
In the context of our research project, [Aérogramme](https://aerogramme.deuxfleurs.fr),
K2V is used to provide metadata and log storage for operations on encrypted e-mail storage.
Learn more on the specification of K2V [here](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/commit/f8be15c37db857e177d543de7be863692628d567/doc/drafts/k2v-spec.md)
and on how to enable it in Garage [here](@/documentation/reference-manual/k2v.md).
In case of disconnected nodes, when changing the cluster layout to remove these
nodes and add other nodes instead, Garage might not be able to properly evict
the old nodes from the system. This is a built-in security measure to avoid any
inconsistent cluster states.
This manifests by several cluster layout versions staying active even after a
full resync. You can diagnose this situation with `garage layout history`,
which will give you instructions to fix it.
### Tag assignment
In the `garage layout assign` command, the `-t` argument has to be repeated
multiple times to set multiple tags on a node. Writing multiple tags separated
by commas will result in a single string.
## General footguns
Choices made by the developers that users must be aware of if they don't want
to run into potential issues.
### Resync tranquility is conservative by default
By default, the worker parameters `resync-tranquility` and `resync-worker-count` are set to very conservative values, to avoid overloading nodes with I/O when data needs to be resynchronized between nodes.
This can cause issues where the resync queue grows faster than it can be cleared, which in turn causes performance issues in the rest of Garage.
This situation is indicated by a big resync queue with few resync errors (the queue is not caused by a disconnected/malfunctionning node).
To fix it, increase the number of resync workers and reduce the resync tranquility. For instance, if you want to resync as fast as possible:
**WARNING: this documentation is a design draft which was written before Garage's actual implementation.
@@ -42,7 +42,7 @@ The general principle are similar, but details have not been updated.**
A version is defined by the existence of at least one entry in the blocks table for a certain version UUID.
We must keep the following invariant: if a version exists in the blocks table, it has to be referenced in the objects table.
We explicitly manage concurrent versions of an object: the version timestamp and version UUID columns are index columns, thus we may have several concurrent versions of an object.
Important: before deleting an older version from the objects table, we must make sure that we did a successfull delete of the blocks of that version from the blocks table.
Important: before deleting an older version from the objects table, we must make sure that we did a successful delete of the blocks of that version from the blocks table.
Thus, the workflow for reading an object is as follows:
@@ -68,7 +68,7 @@ Workflow for DELETE:
1. Check write permission (LDAP)
2. Get current version (or versions) in object table
3. Do the deletion of those versions NOT IN A BACKGROUND JOB THIS TIME
4. Return succes to the user if we were able to delete blocks from the blocks table and entries from the object table
4. Return success to the user if we were able to delete blocks from the blocks table and entries from the object table
To delete a version:
@@ -92,10 +92,10 @@ Known issue: if someone is reading from a version that we want to delete and the
- file path = /meta/(first 3 hex digits of hash)/(rest of hash)
- map block hash -> set of version UUIDs where it is referenced
Usefull metadata:
Useful metadata:
- list of versions that reference this block in the Casandra table, so that we can do GC by checking in Cassandra that the lines still exist
- list of other nodes that we know have acknowledged a write of this block, usefull in the rebalancing algorithm
- list of other nodes that we know have acknowledged a write of this block, useful in the rebalancing algorithm
Write strategy: have a single thread that does all write IO so that it is serialized (or have several threads that manage independent parts of the hash space). When writing a blob, write it to a temporary file, close, then rename so that a concurrent read gets a consistent result (either not found or found with whole content).
**This guide explains how to migrate to 0.8 if you have an existing 0.7 cluster.
We don't recommend trying to migrate to 0.8 directly from 0.6 or older.**
**We make no guarantee that this migration will work perfectly:
back up all your data before attempting it!**
Garage v0.8 introduces new data tables that allow the counting of objects in buckets in order to implement bucket quotas.
A manual migration step is required to first count objects in Garage buckets and populate these tables with accurate data.
## Simple migration procedure (takes cluster offline for a while)
The migration steps are as follows:
1. Disable API and web access. Garage v0.7 does not support disabling
these endpoints but you can change the port number or stop your reverse proxy for instance.
2. Do `garage repair --all-nodes --yes tables` and `garage repair --all-nodes --yes blocks`,
check the logs and check that all data seems to be synced correctly between
nodes. If you have time, do additional checks (`versions`, `block_refs`, etc.)
3. Check that queues are empty: run `garage stats` to query them or inspect metrics in the Grafana dashboard.
4. Turn off Garage v0.7
5. **Backup the metadata folder of all your nodes!** For instance, use the following command
if your metadata directory is `/var/lib/garage/meta`: `cd /var/lib/garage ; tar -acf meta-v0.7.tar.zst meta/`
6. Install Garage v0.8
7. **Before starting Garage v0.8**, run the offline migration step: `garage offline-repair --yes object_counters`.
This can take a while to run, depending on the number of objects stored in your cluster.
8. Turn on Garage v0.8
9. Do `garage repair --all-nodes --yes tables` and `garage repair --all-nodes --yes blocks`.
Wait for a full table sync to run.
10. Your upgraded cluster should be in a working state. Re-enable API and Web
access and check that everything went well.
11. Monitor your cluster in the next hours to see if it works well under your production load, report any issue.
## Minimal downtime migration procedure
The migration to Garage v0.8 can be done with almost no downtime,
by restarting all nodes at once in the new version. The only limitation with this
method is that bucket sizes and item counts will not be estimated correctly
until all nodes have had a chance to run their offline migration procedure.
The migration steps are as follows:
1. Do `garage repair --all-nodes --yes tables` and `garage repair --all-nodes --yes blocks`,
check the logs and check that all data seems to be synced correctly between
nodes. If you have time, do additional checks (`versions`, `block_refs`, etc.)
2. Turn off each node individually; back up its metadata folder (see above); turn it back on again. This will allow you to take a backup of all nodes without impacting global cluster availability. You can do all nodes of a single zone at once as this does not impact the availability of Garage.
3. Prepare your binaries and configuration files for Garage v0.8
4. Shut down all v0.7 nodes simultaneously, and restart them all simultaneously in v0.8. Use your favorite deployment tool (Ansible, Kubernetes, Nomad) to achieve this as fast as possible.
5. At this point, Garage will indicate invalid values for the size and number of objects in each bucket (most likely, it will indicate zero). To fix this, take each node offline individually to do the offline migration step: `garage offline-repair --yes object_counters`. Again you can do all nodes of a single zone at once.
**This guide explains how to migrate to 0.9 if you have an existing 0.8 cluster.
We don't recommend trying to migrate to 0.9 directly from 0.7 or older.**
This migration procedure has been tested on several clusters without issues.
However, it is still a *critical procedure* that might cause issues.
**Make sure to back up all your data before attempting it!**
You might also want to read our [general documentation on upgrading Garage](@/documentation/operations/upgrading.md).
The following are **breaking changes** in Garage v0.9 that require your attention when migrating:
- LMDB is now the default metadata db engine and Sled is deprecated. If you were using Sled, make sure to specify `db_engine = "sled"` in your configuration file, or take the time to [convert your database](https://garagehq.deuxfleurs.fr/documentation/reference-manual/configuration/#db-engine-since-v0-8-0).
- Capacity values are now in actual byte units. The translation from the old layout will assign 1 capacity = 1Gb by default, which might be wrong for your cluster. This does not cause any data to be moved around, but you might want to re-assign correct capacity values post-migration.
- Multipart uploads that were started in Garage v0.8 will not be visible in Garage v0.9 and will have to be restarted from scratch.
- Changes to the admin API: some `v0/` endpoints have been replaced by `v1/` counterparts with updated/uniformized syntax. All other endpoints have also moved to `v1/` by default, without syntax changes, but are still available under `v0/` for compatibility.
## Simple migration procedure (takes cluster offline for a while)
The migration steps are as follows:
1. Disable API and web access. You may do this by stopping your reverse proxy or by commenting out
the `api_bind_addr` values in your `config.toml` file and restarting Garage.
2. Do `garage repair --all-nodes --yes tables` and `garage repair --all-nodes --yes blocks`,
check the logs and check that all data seems to be synced correctly between
nodes. If you have time, do additional checks (`versions`, `block_refs`, etc.)
3. Check that the block resync queue and Merkle queue are empty:
run `garage stats -a` to query them or inspect metrics in the Grafana dashboard.
4. Turn off Garage v0.8
5. **Backup the metadata folder of all your nodes!** For instance, use the following command
if your metadata directory is `/var/lib/garage/meta`: `cd /var/lib/garage ; tar -acf meta-v0.8.tar.zst meta/`
6. Install Garage v0.9
7. Update your configuration file if necessary.
8. Turn on Garage v0.9
9. Do `garage repair --all-nodes --yes tables` and `garage repair --all-nodes --yes blocks`.
Wait for a full table sync to run.
10. Your upgraded cluster should be in a working state. Re-enable API and Web
access and check that everything went well.
11. Monitor your cluster in the next hours to see if it works well under your production load, report any issue.
12. You might want to assign correct capacity values to all your nodes. Doing so might cause data to be moved
in your cluster, which should also be monitored carefully.
## Minimal downtime migration procedure
The migration to Garage v0.9 can be done with almost no downtime,
by restarting all nodes at once in the new version.
The migration steps are as follows:
1. Do `garage repair --all-nodes --yes tables` and `garage repair --all-nodes --yes blocks`,
check the logs and check that all data seems to be synced correctly between
nodes. If you have time, do additional checks (`versions`, `block_refs`, etc.)
2. Turn off each node individually; back up its metadata folder (see above); turn it back on again.
This will allow you to take a backup of all nodes without impacting global cluster availability.
You can do all nodes of a single zone at once as this does not impact the availability of Garage.
3. Prepare your binaries and configuration files for Garage v0.9
4. Shut down all v0.8 nodes simultaneously, and restart them all simultaneously in v0.9.
Use your favorite deployment tool (Ansible, Kubernetes, Nomad) to achieve this as fast as possible.
Garage v0.9 should be in a working state as soon as it starts.
5. Proceed with repair and monitoring as described in steps 9-12 above.
2. Ensure you have a snapshot of your Garage installation that you can restore
to in case the upgrade goes wrong:
- If you are running Garage v0.9.4 or later, use the `garage meta snapshot
--all` to make a backup snapshot of the metadata directories of your nodes
for backup purposes, and save a copy of the following files in the
metadata directories of your nodes: `cluster_layout`, `data_layout`,
`node_key`, `node_key.pub`.
- If you are running a filesystem such as ZFS or BTRFS that support
snapshotting, you can create a filesystem-level snapshot to be used as a
restoration point if needed.
- In other cases, make a backup using the old procedure: turn off each node
individually; back up its metadata folder (for instance, use the following
command if your metadata directory is `/var/lib/garage/meta`: `cd
/var/lib/garage ; tar -acf meta-v0.9.tar.zst meta/`); turn it back on
again. This will allow you to take a backup of all nodes without
impacting global cluster availability. You can do all nodes of a single
zone at once as this does not impact the availability of Garage.
3. Prepare your updated binaries and configuration files for Garage v1.0
4. Shut down all v0.9 nodes simultaneously, and restart them all simultaneously
in v1.0. Use your favorite deployment tool (Ansible, Kubernetes, Nomad) to
achieve this as fast as possible. Garage v1.0 should be in a working state
as soon as enough nodes have started.
5. Monitor your cluster in the following hours to see if it works well under
your production load.
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Agathe Porte