add final pdf

The OPTIONS placeholder for buckets without a resolvable global alias returns` Access-Control-Allow-Origin: *` and `Access-Control-Allow-Methods: *` but omits `Access-Control-Allow-Headers`.

Bug verified against Garage v2.2.0 with a local-aliased bucket: OPTIONS placeholder doesn't have `Access-Control-Allow-Headers`, causes the browser to reject signed PUT preflights

The current placeholder fails open for unsigned simple requests but blocks every signed request, undermining the design intent flagged in the FIXME:

```rs
// We take the permissive approach of allowing everything,
// because we don't want to prevent web apps that use
// local bucket names from making API calls.
```

Adds `Access-Control-Allow-Headers: *` so the permissive default is actually permissive for the request shapes that exist in practice.

Refs #258. Does not address the broader FIXME (CORS rule resolution for local-aliased buckets); the placeholder approach is preserved.

All tests are fine locally:

```bash
 ▲ ~/opensource/garage cargo test -p garage_api_common cors::

running 5 tests
test cors::tests::preflight_with_single_allowed_origin_returns_request_origin ... ok
test cors::tests::preflight_with_multiple_allowed_origins_reflects_request_origin ... ok
test cors::tests::preflight_with_wildcard_allowed_origin_returns_wildcard ... ok
test xml::cors::tests::test_deserialize_norules ... ok
test xml::cors::tests::test_deserialize ... ok

test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 16 filtered out; finished in 0.00s
```

Co-authored-by: smattymatty <smattymatt@gmail.com>
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/1450
Reviewed-by: Alex <lx@deuxfleurs.fr>

CONTRIBUTING.md

@@ -0,0 +1,231 @@
+# Contributing to Garage
+
+## Policy on AI
+
+To ensure the quality of the codebase and documentation, the use of AI,
+including LLMs and coding agents, is strictly restricted in the following way:
+
+- AI **must not** be used to write documentation
+
+- **Do not** use AI to write bug reports, commit descriptions and pull request
+  messages
+
+- **Do not** use AI agents to make contributions to Garage, all contributions
+  must be led by a human that know what they are doing at all times
+
+- AI **may** be used for some tedious code generation tasks, limited to very
+  mechanical translations from API docs or boilerplate writing. The code
+  generated must be so simple as to make it clear that it cannot be covered by
+  copyright.
+
+You are free to make use of AI privately to explore the codebase and solve
+conceptual problems, but please restrain from copying the output from an LLM
+anywhere in your code or on the issue tracker, or from letting an agent edit
+the codebase directly.
+
+
+## Asking questions
+
+Read the documentation before asking questions.
+Do not use the issue tracker to ask questions about Garage.
+Questions asked on the issue tracker will be closed.
+
+Ask questions on the Matrix channel `#garage:deuxfleurs.fr` so that any
+community member can see your question and help you out.
+
+If you need in-depth support from the Garage developers specifically, write to
+`garagehq@deuxfleurs.fr`. Even if you do so, we do not commit to giving you an
+answer.
+
+
+## Reporting bugs
+
+When writing a bug report, use this checklist:
+
+- For bugs that can be reproduced:
+    - confirm that you are using the latest version of Garage and that the bug still exists in this version
+    - set the log level to debug using the `RUST_LOG=garage=debug` environment variable and reproduce the bug to get more verbose logs
+
+- Check whether there is already an open issue in the bug tracker. If so, your bug report is still valuable but please add it as a comment to the existing issue instead of opening a new one.
+
+- Collect as much information as possible:
+  - logs of the Garage daemon at the time the issue happened, including logs that show what was happening before the issue occurred
+  - the output of `garage status`
+  - the output of `garage stats -a`
+  - the output of `garage layout history`
+
+- Write a detailed bug report, including:
+  - a description of your cluster (number of nodes, hardware, operating system, networking, etc)
+  - a detailed description of what you did that led to the issue, including any code or command line that invoked a Garage API
+  - what you were expecting
+  - what actually happened, and how that's different from what you expected
+  - the information collected previously
+  - if possible, simple steps to help the developers reproduce the issue locally
+
+Bug reports that are imprecise or otherwise unactionable will be closed.
+
+
+## Suggesting new features
+
+Garage can be improved in many ways, but just suggesting a new feature does not mean we will implement it.
+Feature requests that may lead to an actual implementation are feature requests that:
+
+- are precise and actionable, i.e. include a precise description of the expected behavior and any necessary architectural details required for the implementation
+- are motivated by actual need from a variety of users
+
+Moreover, a certain number of features are defined as out-of-scope for Garage, including but not limited to:
+
+- extensions to the S3 API that are not present on AWS
+- features that require the implementation of a consensus algorithm
+- more generally, features that are incompatible with the architecture of Garage and its goal of staying simple
+
+Only feature requests in one of the following category may stay open in the issue tracker:
+
+- features that the Garage team wants to work on
+- features that are being actively worked on by an external contributor which is clearly identified
+- features that are easy to implement and could be an easy task for a new contributor that wants to get to know the codebase
+
+All other feature requests will be closed after a few months of inactivity, so as to keep the number of open issues to a manageable level.
+Feature requests that are clearly out of scope will be closed directly.
+
+
+## Improving the documentation
+
+An easy way to contribute to Garage which also adds a lot of value is to
+improve the documentation.  Make sure to write in clear technical English, and
+write unambiguously.  Documentation contributions are very appreciated if they
+are well-written.
+
+
+## For developers
+
+We welcome code contributions to Garage that adhere to our standards for quality:
+
+- Changes should be reviewed from a functional perspective to ensure that they work well with the existing codebase and do not introduce bugs or subtle issues.
+
+- You must have tested your contribution to make sure that it does what it says. The amount of testing required is proportional to the complexity of the change introduced.
+
+- Any new feature must be properly documented following existing practices (see below).
+
+- Unit tests should be included when relevant.
+
+- Contributions should pass basic lints for syntactic quality (`cargo fmt`, `cargo clippy`, `typos`).
+
+- Contributions should pass our CI test suite.
+
+- No user-facing breaking changes may be introduced between major releases.
+
+- No internal data model change may be introduced between major releases, to
+  ensure that Garage daemons with different minor/patch versions numbers can
+  work together in a cluster. For major releases, a proper migration path
+  should be implemented and tested thoroughly.
+
+Please follow up on your work when changes are requested, to avoid stale PRs.
+Do not take it personally if a Garage developer pushes directly to your branch
+to modify your contribution, as this might be necessary to get it merged
+faster.
+
+### Properly documenting your contribution
+
+#### Configuration options
+
+New configuration options should be documented in
+`doc/book/reference-manual/configuration.md`.  The documentation for a
+configuration option should be exhaustive. For instance, for choice options all
+choices should be listed explicitly with a precise description of their
+meaning.
+
+In terms of syntax, all configuration options should appear in three places:
+
+- in the example at the top, with an example value
+- in the index of all configuration options which is sorted by alphabetical order
+- in its dedicated subsection with full reference text
+
+#### CLI commands and command flags
+
+CLI commands are self-documented using the doc commends in the codebase.
+Make sure to write clear and precise comments for all options you are adding.
+
+#### S3 features
+
+If you implement new S3 features, make sure to update the compatibility matrix in `doc/book/reference-manual/s3-compatibility.md`.
+
+#### Admin API
+
+The admin API has an OpenAPI specification that is automatically generated
+using Utoipa, from a description of each endpoint that is given in
+`src/api/admin/openapi.rs` and a description of data structure schemas in
+`src/api/admin/api.rs`. The code in `openapi.rs` is only used to generate the
+OpenAPI specification document and not for the actual implementation in Garage,
+whereas structures defined in `api.rs` are also used for the implementation of
+API calls. Make sure to write good doc comments for all of these items so that
+the OpenAPI specification will be precise and accurate.
+
+An up-to-date version of the OpenAPI specification document should be kept in
+the repository in `doc/api/garage-admin-v2.json`. When you are making changes
+to the admin API, update this document with the following command:
+
+```
+cargo run -- admin-api-schema > doc/api/garage-admin-v2.json
+```
+
+
+## Garage team organization
+
+Alex (handle `lx`) is the lead developer and is responsible of ensuring the
+correctness of Garage and stability between version upgrades.
+
+The other maintainers are Trinity (handle `trinity-1686a`), Quentin (handle `quentin`) and Maximilien (handle `halfa`).
+
+Maximilien is responsible for coordinating effort on the Kubernetes integration / Helm chart.
+
+## Pull request merging criteria
+
+The following PRs should only be merged after review and approval from Alex:
+
+- PRs that introduce architectural changes, such as changes in the data model
+  or change in the coordination protocols between nodes
+
+- PRs that introduce changes on the format of data structures used for
+  persistent disk storage and internal cluster communication (RPC)
+
+- PRs that are suspected of introducing some kind of breakage or unexpected
+  behavior due to their complexity
+
+PRs that introduce breaking change for users but don't fall in one of the
+previous category should be discussed between maintainers to evaluate the
+impact on users when upgrading.  Alex's approval is not required to merge them
+as long as they are clearly identified as breaking in the PR title, and are
+properly merged in the branch for the next major version and not in the current
+main branch.
+
+All other PRs can be merged by any maintainer on their own, once they are
+confident that the quality standards defined in this document are respected
+before merging.
+
+## Merging strategy
+
+When merging PRs, maintainers should ensure that a Git commit is created by
+Forgejo that records the PR number, its title and its text in the commit
+message.  If a PR is fixing an issue, make sure that the issue number is
+included in the PR title as well.  This is to ensure that when releasing a new
+version of Garage, the changelog in the release notes can be properly
+constructed by reading the Git log since the last release.
+
+We also want to keep the history "almost linear" to facilitate the use of `git
+bisect` if it ever were necessary. This leaves the following two merging
+strategies:
+
+- For PRs that consist of many commits that should stay independent, the
+  "rebase and create merge commit" strategy should be used. The merge commit is
+  created automatically by Forgejo and saves the PR's number, title and text in
+  the commit message.
+
+- For PRs that consist of only one commit, or a few number of commits that can
+  be merged, the "create squash commit" strategy should be used. This way a
+  single commit will be created by Forgejo which also saves the PR's number,
+  title and text in the commit message.
+
+When cherry-picking commits from one branch to the other, a simple fast-forward
+merging strategy can be used if the commit message already references a PR
+number.

Cargo.toml

@@ -16,6 +16,7 @@ members = [
 	"src/garage",
 	"src/k2v-client",
 	"src/format-table",
+    "fuzz",
 ]
 
 default-members = ["src/garage"]
@@ -40,6 +41,7 @@ k2v-client = { version = "0.0.4", path = "src/k2v-client" }
 
 # External crates from crates.io
 arc-swap = "1.8"
+arbitrary = { version = "1.4.2"}
 argon2 = "0.5"
 async-trait = "0.1"
 backtrace = "0.3"
@@ -59,6 +61,7 @@ hmac = "0.12"
 itertools = "0.14"
 ipnet = "2.11"
 lazy_static = "1.5"
+libfuzzer-sys = "0.4"
 md-5 = "0.10"
 mktemp = "0.5"
 nix = { version = "0.31", default-features = false, features = ["fs"] }
@@ -113,7 +116,7 @@ kube = { version = "3.0", default-features = false, features = [
 ] }
 schemars = "1.2"
 reqwest = { version = "0.13", default-features = false, features = [
-    "rustls",
+    "rustls-no-provider",
     "json",
 ] }
 

SECURITY.md

@@ -0,0 +1,14 @@
+# Security Reporting
+
+If you wish to report responsibly a security vulnerability about Garage, we ask that you follow the following process.
+
+Please report each security vulnerabilities by filling out the following template:
+
+- PROJECT: A URL to the code repository containing the vulnerable version - be reminded that the source of truth is at https://git.deuxfleurs.fr/deuxfleurs/garage
+- PUBLIC: Please let us know if this vulnerability has been made or discussed publicly already, and if so, please let us know where.
+- DESCRIPTION: Please provide precise description of the security vulnerability you have found with as much information as you are able and willing to provide.
+
+Please send the above info, along with any other information you feel is pertinent by emailing the core team at: garagehq@deuxfleurs.fr
+
+The Garage Core Team will let you know within a few weeks whether or not your report has been accepted or rejected.
+We ask that you please keep the report confidential until we have either responded or made a public announcement.

doc/api/garage-admin-v2.json

@@ -1797,6 +1797,17 @@
               "type": "string"
             },
             "description": "Plain-text information about the layout computation\n(do not try to parse this)"
+          },
+          "statistics": {
+            "oneOf": [
+              {
+                "type": "null"
+              },
+              {
+                "$ref": "#/components/schemas/ComputationStat",
+                "description": "Structured statistics about the layout computation"
+              }
+            ]
           }
         }
       },
@@ -2119,6 +2130,180 @@
           "Historical"
         ]
       },
+      "ComputationStat": {
+        "type": "object",
+        "required": [
+          "replicationFactor",
+          "effectiveZoneRedundancy",
+          "partitionSize",
+          "lowPartitionSize",
+          "usableCapacity",
+          "totalCapacity",
+          "effectiveCapacity",
+          "lowUsableCapacity",
+          "zones"
+        ],
+        "properties": {
+          "effectiveCapacity": {
+            "type": "integer",
+            "format": "int64",
+            "description": "The final effective capacity of the cluster, accounting for replication",
+            "minimum": 0
+          },
+          "effectiveZoneRedundancy": {
+            "type": "integer",
+            "description": "The zone redundancy factor achieved by this layout",
+            "minimum": 0
+          },
+          "lowPartitionSize": {
+            "type": "boolean",
+            "description": "Warning flag indicating when partitions are very small"
+          },
+          "lowUsableCapacity": {
+            "type": "boolean",
+            "description": "Warning flag indicating that the raw node capacity could not be used\neffectively"
+          },
+          "partitionSize": {
+            "type": "integer",
+            "format": "int64",
+            "description": "The size of a partition, in bytes",
+            "minimum": 0
+          },
+          "previousPartitionSize": {
+            "type": [
+              "integer",
+              "null"
+            ],
+            "format": "int64",
+            "description": "The size of a partition, in bytes, in the previous layout",
+            "minimum": 0
+          },
+          "replicationFactor": {
+            "type": "integer",
+            "description": "The cluster's replication factor",
+            "minimum": 0
+          },
+          "totalCapacity": {
+            "type": "integer",
+            "format": "int64",
+            "description": "The total raw capacity of nodes",
+            "minimum": 0
+          },
+          "totalMovedPartitions": {
+            "type": [
+              "integer",
+              "null"
+            ],
+            "description": "The total number of partitions that will be moved to a new storage node",
+            "minimum": 0
+          },
+          "usableCapacity": {
+            "type": "integer",
+            "format": "int64",
+            "description": "The portion of total raw node capacity that is used by partitions",
+            "minimum": 0
+          },
+          "zones": {
+            "type": "array",
+            "items": {
+              "$ref": "#/components/schemas/ComputationStatZone"
+            },
+            "description": "Per-zone storage statistics"
+          }
+        }
+      },
+      "ComputationStatNode": {
+        "type": "object",
+        "required": [
+          "id",
+          "tags",
+          "storedPartitions",
+          "newPartitions",
+          "totalCapacity",
+          "usableCapacity"
+        ],
+        "properties": {
+          "id": {
+            "type": "string",
+            "description": "The node's ID"
+          },
+          "newPartitions": {
+            "type": "integer",
+            "description": "The number of partitions that are newly replicated on this node",
+            "minimum": 0
+          },
+          "storedPartitions": {
+            "type": "integer",
+            "description": "The number of partitions that are replicated on this node",
+            "minimum": 0
+          },
+          "tags": {
+            "type": "array",
+            "items": {
+              "type": "string"
+            },
+            "description": "The node's tags as defined in the layout"
+          },
+          "totalCapacity": {
+            "type": "integer",
+            "format": "int64",
+            "description": "The node's raw capacity",
+            "minimum": 0
+          },
+          "usableCapacity": {
+            "type": "integer",
+            "format": "int64",
+            "description": "The portion of the node's raw capacity that is used by partitions it stores",
+            "minimum": 0
+          }
+        }
+      },
+      "ComputationStatZone": {
+        "type": "object",
+        "required": [
+          "name",
+          "nodes",
+          "totalReplicatedPartitions",
+          "uniquePartitions",
+          "totalCapacity",
+          "usableCapacity"
+        ],
+        "properties": {
+          "name": {
+            "type": "string",
+            "description": "The name of the zone"
+          },
+          "nodes": {
+            "type": "array",
+            "items": {
+              "$ref": "#/components/schemas/ComputationStatNode"
+            },
+            "description": "Per-node storage statistics for nodes in this zone"
+          },
+          "totalCapacity": {
+            "type": "integer",
+            "format": "int64",
+            "description": "The total raw capacity of nodes in this zone",
+            "minimum": 0
+          },
+          "totalReplicatedPartitions": {
+            "type": "integer",
+            "description": "The total number of partition replicas in this zone",
+            "minimum": 0
+          },
+          "uniquePartitions": {
+            "type": "integer",
+            "description": "The number of unique partitions that have at least one replica in this zone",
+            "minimum": 0
+          },
+          "usableCapacity": {
+            "type": "integer",
+            "format": "int64",
+            "description": "The used portion of the raw capacity of nodes in this zones",
+            "minimum": 0
+          }
+        }
+      },
       "ConnectClusterNodesRequest": {
         "type": "array",
         "items": {
@@ -4221,6 +4406,17 @@
               "newLayout": {
                 "$ref": "#/components/schemas/GetClusterLayoutResponse",
                 "description": "Details about the new cluster layout"
+              },
+              "statistics": {
+                "oneOf": [
+                  {
+                    "type": "null"
+                  },
+                  {
+                    "$ref": "#/components/schemas/ComputationStat",
+                    "description": "Structured statistics about the layout computation"
+                  }
+                ]
               }
             }
           }

doc/talks/2026-06-03-adr/README.md

@@ -0,0 +1,6 @@
+Compile with:
+
+```
+typst compile talk.typ --root ../..
+```
+

doc/talks/2026-06-03-adr/talk.typ

@@ -0,0 +1,378 @@
+#import "@preview/slydst:0.1.5": *
+
+// some display rules
+
+#set par(spacing: 2em)
+#set list(spacing: 1em)
+
+#show link: set text(font: "DejaVu Sans Mono", size: 9pt)
+
+// some functions to customize styles
+
+#let vhcenter(content) = [
+  #v(1fr)
+  #align(center)[#content]
+  #v(1fr)
+]
+
+#let imgcenter(..args) = vhcenter(image(..args))
+
+#let mytable(..args) = {
+  show table.cell: set text(size: 9pt)
+  set table(stroke: 0.5pt + black)
+  grid(
+    columns: (1cm, 1fr, 1cm),
+    [], table(..args), []
+  )
+}
+
+// actual slides
+
+#show: slides.with(
+  //title: "Garage",
+  authors: ("Alex Auvolat",),
+  date: "2026-06-03",
+  layout: "large",
+  //ratio: 16/9,
+  ratio: 4/3,
+  title-color: rgb("#ff9329"),
+)
+
+#title-slide[
+  #align(center)[
+    #image("../../sticker/Garage.png", width: 20%)
+    #v(1em)
+    *An introduction to Garage*\
+    Alex Auvolat, Deuxfleurs
+
+    #v(1em)
+    #link("https://garagehq.deuxfleurs.fr/")\
+    Matrix channel: `#garage:deuxfleurs.fr`
+  ]
+]
+
+== A non-profit initiative
+
+#grid(
+  columns: (2fr, 8fr),
+  [#v(2em)],[],
+  [
+    #image("../assets/logos/deuxfleurs.svg", width: 50%)
+  ],
+  [
+    *Part of a degrowth initiative*\
+    Garage has been created at Deuxfleurs, where we experiment running Internet services without datacenter on commodity and refurbished hardware.
+  ],
+  [#v(2em)],[],
+  [
+    #image("../assets/community.png", width: 50%)
+  ],
+  [
+    *Developed by a community*\
+    #text(size: 0.8em)[Some recent contributors: Arthur C, Charles H, dongdigua, Etienne L, Jonah A, Julien K, Lapineige, MagicRR, Milas B, Niklas M, RockWolf, Schwitzd, trinity-1686a, Xavier S, babykart, Baptiste J, eddster2309, James O'C, Joker9944, Maximilien R, Renjaya RZ, Yureka...]
+  ],
+  [#v(3em)],[],
+  [
+    #image("../assets/logos/AGPLv3_Logo.png", width: 50%)
+  ],
+  [
+    *Owned by nobody*\
+    AGPL + no Contributor License Agreement = Garage ownership spreads among dozens of contributors.
+  ]
+)
+
+== Our initial objective at Deuxfleurs
+
+#v(4em)
+
+#align(center)[
+  #text(weight: "bold")[
+    Promote self-hosting and small-scale hosting\
+    as an alternative to large cloud providers
+  ]
+]
+
+#v(2em)
+
+Why is it hard?
+
+#v(2em)
+
+#align(center)[
+  #underline[Resilience]\
+  #text(size: 0.8em)[we want good uptime/availability with low supervision]
+]
+
+== Our very low-tech infrastructure
+//== Building a resilient system with cheap stuff
+//
+#v(4em)
+
+#[
+  #set list(spacing: 2em)
+  - Commodity hardware (e.g. old desktop PCs)\
+    #text(size: 0.8em)[(can die at any time)]
+  - Regular Internet (e.g. FTTB, FTTH) and power grid connections\
+    #text(size: 0.8em)[(can be unavailable randomly)]
+  - *Geographical redundancy* (multi-site replication)
+]
+
+#pagebreak()
+#imgcenter("../assets/neptune.jpg", width: 100%)
+
+#pagebreak()
+#imgcenter("../assets/atuin.jpg", width: 100%)
+
+#pagebreak()
+#imgcenter("../assets/inframap_jdll2023.pdf", width: 100%)
+
+== Object storage: a crucial component
+
+#vhcenter[
+  #grid(
+    columns: (3fr, 3fr, 3fr),
+    [#image("../assets/logos/Amazon-S3.jpg", height: 6em)],
+    [#image("../assets/logos/minio.png", height: 5em)],
+    [#image("../../logo/garage_hires_crop.png", height: 6em)]
+  )
+]
+
+S3: a de-facto standard, many compatible applications
+
+MinIO: not suited for geo-distributed deployments, becoming closed source
+
+*Garage is a self-hosted drop-in replacement for the Amazon S3 object store*
+
+#v(2em)
+
+== Principle 1: geo-distributed data model
+
+#imgcenter("../assets/map.png", width: 90%)
+
+Garage stores replicas on different zones when possible
+
+== Zone-aware cluster configuration
+
+#imgcenter("../assets/screenshots/garage_status_0.9_prod_zonehl.png", width: 100%)
+
+Trust model: full trust between zones
+
+#v(5em)
+
+== Principle 2: based on CRDTs
+
+#v(1cm)
+
+#underline[Internally, Garage uses only CRDTs] (conflict-free replicated data types)
+
+Why not Raft, Paxos, ...? Issues of consensus algorithms:
+
+- *Software complexity*
+- *Performance issues:*
+  - The leader is a *bottleneck* for all requests
+  - *Sensitive to higher latency* between nodes
+  - *Takes time to reconverge* when disrupted (e.g. node going down)
+
+
+== The data model of object storage
+
+#[
+  #set list(spacing: 1em)
+
+  Object storage is basically a *key-value store*:
+  
+  #mytable(
+    columns: (2fr, 5fr),
+    align: left,
+    [*Key: file path + name*], [*Value: file data + metadata*],
+    [`index.html`], text(size: 8pt)[
+      `Content-Type: text/html; charset=utf-8`\
+      `Content-Length: 24929`\
+      `<binary blob>`
+    ],
+    [`img/logo.svg`], text(size: 8pt)[
+      `Content-Type: text/svg+xml`\
+      `Content-Length: 13429`\
+      `<binary blob>`
+    ],
+    [`download/index.html`], text(size: 8pt)[
+      `Content-Type: text/html; charset=utf-8`\
+      `Content-Length: 26563`\
+      `<binary blob>`
+    ]
+  )
+  
+*Consistency model:*
+
+- Not ACID (not required by S3 spec) / not linearizable
+- *Read-after-write consistency*\
+  #text(size: 0.8em)[(stronger than eventual consistency)]
+]
+
+
+== Performance evaluation
+
+#imgcenter("../assets/perf/endpoint_latency_0.7_0.8_minio.png", width: 100%)
+#pagebreak()
+
+#imgcenter("../assets/perf/ttfb.png", width: 100%)
+#pagebreak()
+
+#imgcenter("../assets/perf/io-0.7-0.8-minio.png", width: 100%)
+
+
+== Garage in the wild
+
+#imgcenter("../assets/cluster_kind.png", width: 100%)
+
+== Size of known deployments
+
+#imgcenter("../assets/cluster_size.png", width: 100%)
+
+_"Petabyte storage setup for a video site. Nginx as CDN in-front using garage-s3-website feature. Each storage node has ~64TB storage with raid10, no replication within garage. 25gbit nic. haproxy to loadbalance across 5 nodes. mostly reads with very few writes."_
+
+_"We currently manage 7 Garage nodes, 28TB total storage, 6M blocks for 3M objects and 4TB of object data. We have been running Garage in production for 2.5 years."_
+
+= Deploying Garage
+
+== Chosing a replication factor
+
+#vhcenter[
+  #mytable(
+    columns: (0.7fr, 1fr, 1.3fr),
+    inset: 0.8em,
+    align: center + horizon,
+    table.header[*Replication factor*][*Pro*][*Cons*],
+    [*1*], [easy single-node setup\ full space efficiency], [no metadata redundancy\ *vunlerable to hardware crash or data corruption*\ no high-availability],
+    [*2*], [redundancy\ limited storage overhead], [limited high-availability\ (read-only when one node is unavailable)],
+    [*3*], [high-availability setup\ best data resilience], [big storage overhead],
+    [*4, 5, ...*], [possible if needed], [...],
+
+  )
+
+  #v(0.5cm)
+  *Important note:* metadata replication == data replication\
+  Choose well, this cannot be changed easily!
+]
+
+== Setting up data and metadata storage
+
+#vhcenter[
+  #mytable(
+    columns: (0.7fr, 1fr, 1fr),
+    inset: 0.8em,
+    align: center + horizon,
+    table.header[][*Metadata storage*][*Data storage*],
+    [*Content*],[access keys, buckets\ index of objects],[raw data blocks],
+    [*Size*],[\< 10\% of data\ rarely over 100GB],[replication × dataset size\ *no erasure-coding*],
+    [*Constraints*],[latency sensitive\ write-intensive under load],[big\ many files],
+    [*Ideal hardware*],[entreprise-grade SSD],[HDD],
+    [*Recommended redundancy*],[RAID1],[none, use disks directly\ *avoid RAID if possible*],
+    [*Recommended filesystem*],[ZFS, Btrfs],[XFS on invidual disks],
+    [*Tunables in Garage*],[database engine\
+                            automatic snapshots],[block size\ compression],
+  )
+]
+
+== Picking a metadata engine
+
+
+#vhcenter[
+  All files-to-block mappings are stored in the metadata engine, including bucket and object metadata. Files below 3KB are stored directly in the metadata engine.
+  #v(0.5cm)
+
+  #mytable(
+    columns: (0.7fr, 1fr, 1.3fr),
+    inset: 0.8em,
+    align: center + horizon,
+    table.header[*Metadata engine*][*Characteristics*][*Use case*],
+    [*SQlite*],[safer],[single node deployment\ small clusters\ clusters with infrequent access],
+    [*LMDB*],[faster\ sometimes has inexplicable corruptions],[larger clusters with metadata redundancy],
+    [*Fjall*],[experimental\ best of both worlds?],[help us test it!],
+  )
+
+  #v(0.5cm)
+  Metadata engine can be set node per-node, and changed later with a migration tool
+]
+
+== Avoiding common issues as soon as possible
+
+#vhcenter[
+  #mytable(
+    columns: (1fr, 1.4fr),
+    inset: 0.8em,
+    align: center + horizon,
+    table.header[*Risk*][*How to avoid*],
+    [*Metadata corruption*\ (esp. with LMDB)],[Configure automatic snapshots with\ `metadata_auto_snapshot_interval`\ Use replication factor 2 or 3],
+    [*Data not well balanced between nodes*],[Avoid clusters with too many nodes\ Target: \#nodes ≤ 10 × replication_factor],
+    [*Performance issues with many objects in one single bucket*],[Spread your data over multiple buckets],
+    [*Performance issues with big objects*],[Increase `block_size` configuration parameter\ Target: object size ≤ 1000 × `block_size`,\ `block_size` ≤ 100MB],
+    [*Performance issues with many small objects*],[Have enough RAM to fit the entire metadata DB],
+  )
+]
+
+== Other things to consider during set-up
+
+#vhcenter[
+  #mytable(
+    columns: (1fr, 1.2fr),
+    inset: 0.8em,
+    align: center + horizon,
+    [*Tools for cluster deployment*],[Ansible + systemd\ NixOS\ Kubernetes or Nomad with Docker],
+    [*Initial cluster setup*],[Manual layout configuration\ Read the documentation!],
+    [*TLS support on public endpoints*],[Add an external reverse-proxy (Nginx, ...)],
+    [*S3 anonymous access*],[Not implemented, use website endpoint],
+    [*Monitoring*],[Prometheus + Grafana for Garage metrics\ External tool to monitor HDD health],
+  )
+]
+
+== Monitoring with Prometheus + Grafana
+
+#imgcenter("../2026-01-31-fosdem/assets/garage-stats.png", width: 83%)
+
+== Common issues and their solutions
+
+#vhcenter[
+  #mytable(
+    columns: (1fr, 1.5fr),
+    inset: 0.8em,
+    align: center + horizon,
+    table.header[*Problem*][*Solution*],
+    table.cell(rowspan: 2)[*S3 access authorization issues*],[Correctly set the `region` parameter in your S3 client\ default = `garage`, not `us-east-1`],[Check your reverse proxy configuration],
+    [*Debugging other API issues*],[Set `RUST_LOG=garage=debug` to investigate],
+    [*Resync queue fills up*],[`garage worker set -a resync-worker-count 8`\ `garage worker set -a resync-tranquility 0`],
+    [*LMDB database too big*],[Stop garage and compact with `mdb_copy -c`],
+    [*Data recovery with dead/unavailable nodes*],[Consistency mode `degraded` allows to read data from an unhealthy cluster. *Do not use it for regular operation.*],
+    [*Other issues*],[Ask us on matrix `#garage:deuxfleurs.fr` or open an issue on `git.deuxfleurs.fr`\
+      Provide the output of `garage status`, `garage stats` and relevant metrics and logs],
+
+  )
+]
+
+== Future developments
+
+#imgcenter("../assets/survey_requested_features.png", width: 80%)
+
+#pagebreak()
+#imgcenter("../2026-01-31-fosdem/assets/Garage Web Admin - Dashboard@2x.png", width: 100%)
+
+#pagebreak()
+#imgcenter("../2026-01-31-fosdem/assets/Garage Web Admin - Bucket details page@2x.png", width: 100%)
+
+== Where to find us
+
+#align(center)[
+  #v(1fr)
+
+  #image("../../logo/garage_hires.png", width: 25%)
+  #link("https://garagehq.deuxfleurs.fr/")\
+  #link("mailto:garagehq@deuxfleurs.fr")\
+  `#garage:deuxfleurs.fr` on Matrix
+
+  #v(1fr)
+  #grid(columns: (6%,3%,13%),
+    image("../assets/logos/rust_logo.png"),
+    [],
+    image("../assets/logos/AGPLv3_Logo.png"),
+  )
+]

flake.lock

@@ -81,17 +81,17 @@
         ]
       },
       "locked": {
-        "lastModified": 1763952169,
-        "narHash": "sha256-+PeDBD8P+NKauH+w7eO/QWCIp8Cx4mCfWnh9sJmy9CM=",
+        "lastModified": 1776914043,
+        "narHash": "sha256-qug5r56yW1qOsjSI99l3Jm15JNT9CvS2otkXNRNtrPI=",
         "owner": "oxalica",
         "repo": "rust-overlay",
-        "rev": "ab726555a9a72e6dc80649809147823a813fa95b",
+        "rev": "2d35c4358d7de3a0e606a6e8b27925d981c01cc3",
         "type": "github"
       },
       "original": {
         "owner": "oxalica",
         "repo": "rust-overlay",
-        "rev": "ab726555a9a72e6dc80649809147823a813fa95b",
+        "rev": "2d35c4358d7de3a0e606a6e8b27925d981c01cc3",
         "type": "github"
       }
     },

flake.nix

@@ -6,9 +6,9 @@
   inputs.nixpkgs.url =
     "github:NixOS/nixpkgs/cfe2c7d5b5d3032862254e68c37a6576b633d632";
 
-  # Rust overlay as of 2025-11-24
+  # Rust overlay as of 2026-04-23
   inputs.rust-overlay.url =
-  "github:oxalica/rust-overlay/ab726555a9a72e6dc80649809147823a813fa95b";
+  "github:oxalica/rust-overlay/2d35c4358d7de3a0e606a6e8b27925d981c01cc3";
   inputs.rust-overlay.inputs.nixpkgs.follows = "nixpkgs";
 
   # Crane as of 2025-01-24
@@ -95,6 +95,14 @@
                 killall
               ];
             };
+
+            # dev shell for fuzzing
+            fuzz = pkgs.mkShell {
+              buildInputs = with pkgs; [
+                targets.toolchainNightly
+                cargo-fuzz
+              ];
+            };
           };
       });
 }

fuzz/.gitignore

@@ -0,0 +1,4 @@
+target
+corpus
+artifacts
+coverage

fuzz/Cargo.toml

@@ -0,0 +1,66 @@
+[package]
+name = "garage-fuzz"
+version = "0.0.0"
+publish = false
+edition = "2018"
+
+[package.metadata]
+cargo-fuzz = true
+
+[dependencies]
+arbitrary = { workspace = true, features = ["derive"]}
+libfuzzer-sys = { workspace = true }
+
+garage_db.workspace = true
+garage_table.workspace = true
+garage_util.workspace = true
+garage_model = { workspace = true, default-features = false, features = ["arbitrary"] }
+
+[[bin]]
+name = "version_crdt"
+path = "fuzz_targets/version_crdt.rs"
+test = false
+doc = false
+bench = false
+
+[[bin]]
+name = "mpu_crdt"
+path = "fuzz_targets/mpu_crdt.rs"
+test = false
+doc = false
+bench = false
+
+[[bin]]
+name = "bucket_crdt"
+path = "fuzz_targets/bucket_crdt.rs"
+test = false
+doc = false
+bench = false
+
+[[bin]]
+name = "block_ref_crdt"
+path = "fuzz_targets/block_ref_crdt.rs"
+test = false
+doc = false
+bench = false
+
+[[bin]]
+name = "admin_api_token_crdt"
+path = "fuzz_targets/admin_api_token_crdt.rs"
+test = false
+doc = false
+bench = false
+
+[[bin]]
+name = "key_crdt"
+path = "fuzz_targets/key_crdt.rs"
+test = false
+doc = false
+bench = false
+
+[[bin]]
+name = "bucket_alias_crdt"
+path = "fuzz_targets/bucket_alias_crdt.rs"
+test = false
+doc = false
+bench = false

fuzz/README.md

@@ -0,0 +1,11 @@
+# Fuzzing
+
+## Setup
+
+Install cargo fuzz: `cargo install cargo-fuzz`
+
+## Launch
+
+Run `cargo fuzz run <fuzz_target>` where `<fuzz_target>` is the name (without extension) of one of the `.rs` files in the `fuzz_targets` directory.
+
+If you launch the command outside of the fuzz directory, you need to force the nightly toolchain with `cargo +nightly`.

fuzz/fuzz_targets/admin_api_token_crdt.rs

@@ -0,0 +1,37 @@
+#![no_main]
+
+use garage_fuzz::check_crdt_laws;
+use garage_model::admin_token_table::{AdminApiToken, AdminApiTokenParams, AdminApiTokenScope};
+use garage_util::crdt;
+use libfuzzer_sys::fuzz_target;
+
+type Input = (
+	bool,
+	crdt::Lww<String>,
+	crdt::Lww<Option<u64>>,
+	crdt::Lww<AdminApiTokenScope>,
+);
+
+fn make(input: Input) -> AdminApiToken {
+	let (deleted, name, expiration, scope) = input;
+	let state = if deleted {
+		crdt::Deletable::Deleted
+	} else {
+		crdt::Deletable::present(AdminApiTokenParams {
+			created: 0,
+			token_hash: String::new(),
+			name,
+			expiration,
+			scope,
+		})
+	};
+	AdminApiToken {
+		prefix: String::new(),
+		state,
+	}
+}
+
+fuzz_target!(|inputs: (Input, Input, Input)| {
+	let (a, b, c) = inputs;
+	check_crdt_laws(make(a), make(b), make(c));
+});

fuzz/fuzz_targets/block_ref_crdt.rs

@@ -0,0 +1,20 @@
+#![no_main]
+
+use garage_fuzz::check_crdt_laws;
+use garage_model::s3::block_ref_table::BlockRef;
+use libfuzzer_sys::fuzz_target;
+
+/// Build a BlockRef with a fixed block hash and version UUID so that CRDT state
+/// can be compared across merge results. Only the deleted flag varies.
+fn make_block_ref(deleted: bool) -> BlockRef {
+	BlockRef {
+		block: [0u8; 32].into(),
+		version: [0u8; 32].into(),
+		deleted: deleted.into(),
+	}
+}
+
+fuzz_target!(|inputs: (bool, bool, bool)| {
+	let (d1, d2, d3) = inputs;
+	check_crdt_laws(make_block_ref(d1), make_block_ref(d2), make_block_ref(d3));
+});

fuzz/fuzz_targets/bucket_alias_crdt.rs

@@ -0,0 +1,25 @@
+#![no_main]
+
+use garage_fuzz::check_crdt_laws;
+use garage_model::bucket_alias_table::BucketAlias;
+use garage_util::data::Uuid;
+use libfuzzer_sys::fuzz_target;
+
+/// Build a BucketAlias with a fixed name so that CRDT state can be compared
+/// across merge results. The timestamp and optional bucket ID are the CRDT state.
+fn make_bucket_alias(ts: u64, bucket_id: Option<[u8; 32]>) -> BucketAlias {
+	BucketAlias::new(String::new(), ts, bucket_id.map(Uuid::from))
+}
+
+fuzz_target!(|inputs: (
+	(u64, Option<[u8; 32]>),
+	(u64, Option<[u8; 32]>),
+	(u64, Option<[u8; 32]>)
+)| {
+	let ((ts1, b1), (ts2, b2), (ts3, b3)) = inputs;
+	check_crdt_laws(
+		make_bucket_alias(ts1, b1),
+		make_bucket_alias(ts2, b2),
+		make_bucket_alias(ts3, b3),
+	);
+});

fuzz/fuzz_targets/bucket_crdt.rs

@@ -0,0 +1,22 @@
+#![no_main]
+
+use garage_fuzz::check_crdt_laws;
+use garage_model::bucket_table::{Bucket, BucketParams};
+use garage_util::crdt::{self, Deletable};
+use libfuzzer_sys::fuzz_target;
+
+fn make(state: Deletable<BucketParams>) -> Bucket {
+	Bucket {
+		id: [0u8; 32].into(),
+		state,
+	}
+}
+
+fuzz_target!(|inputs: (
+	crdt::Deletable<BucketParams>,
+	crdt::Deletable<BucketParams>,
+	crdt::Deletable<BucketParams>
+)| {
+	let (a, b, c) = inputs;
+	check_crdt_laws(make(a), make(b), make(c));
+});

fuzz/fuzz_targets/key_crdt.rs

@@ -0,0 +1,43 @@
+#![no_main]
+
+use garage_fuzz::check_crdt_laws;
+use garage_model::key_table::{Key, KeyParams};
+use garage_model::permission::BucketKeyPerm;
+use garage_util::crdt;
+use garage_util::data::Uuid;
+use libfuzzer_sys::fuzz_target;
+
+type Input = (
+	bool,
+	crdt::Lww<String>,
+	crdt::Lww<Option<u64>>,
+	crdt::Lww<bool>,
+	crdt::Map<Uuid, BucketKeyPerm>,
+	crdt::LwwMap<String, Option<Uuid>>,
+);
+
+fn make(input: Input) -> Key {
+	let (deleted, name, expiration, allow_create_bucket, authorized_buckets, local_aliases) = input;
+	let state = if deleted {
+		crdt::Deletable::Deleted
+	} else {
+		crdt::Deletable::present(KeyParams {
+			created: None,
+			secret_key: String::new(),
+			name,
+			expiration,
+			allow_create_bucket,
+			authorized_buckets,
+			local_aliases,
+		})
+	};
+	Key {
+		key_id: String::new(),
+		state,
+	}
+}
+
+fuzz_target!(|inputs: (Input, Input, Input)| {
+	let (a, b, c) = inputs;
+	check_crdt_laws(make(a), make(b), make(c));
+});

fuzz/fuzz_targets/mpu_crdt.rs

@@ -0,0 +1,37 @@
+#![no_main]
+
+use garage_fuzz::check_crdt_laws;
+use garage_model::s3::mpu_table::{MpuPart, MpuPartKey, MultipartUpload};
+use libfuzzer_sys::fuzz_target;
+
+/// Build a MultipartUpload from an arbitrary deleted flag and parts list, using a fixed
+/// upload_id/bucket_id/key so that CRDT state can be compared across merge results.
+/// `MpuPart.version` is fixed to a constant since it is identity data, not CRDT state:
+/// two replicas of the same part (same MpuPartKey) always share the same version UUID.
+/// If deleted, parts are cleared to ensure a valid initial CRDT state.
+fn make_mpu(deleted: bool, parts: Vec<(MpuPartKey, MpuPart)>) -> MultipartUpload {
+	let mut mpu = MultipartUpload::new(
+		[0u8; 32].into(),
+		0,
+		[0u8; 32].into(),
+		String::new(),
+		deleted,
+	);
+	for (key, mut part) in parts {
+		part.version = [0u8; 32].into();
+		mpu.parts.put(key, part);
+	}
+	if mpu.deleted.get() {
+		mpu.parts.clear();
+	}
+	mpu
+}
+
+fuzz_target!(|inputs: (
+	(bool, Vec<(MpuPartKey, MpuPart)>),
+	(bool, Vec<(MpuPartKey, MpuPart)>),
+	(bool, Vec<(MpuPartKey, MpuPart)>)
+)| {
+	let ((d1, p1), (d2, p2), (d3, p3)) = inputs;
+	check_crdt_laws(make_mpu(d1, p1), make_mpu(d2, p2), make_mpu(d3, p3));
+});

fuzz/fuzz_targets/version_crdt.rs

@@ -0,0 +1,42 @@
+#![no_main]
+
+use garage_fuzz::check_crdt_laws;
+use garage_model::s3::version_table::{Version, VersionBacklink, VersionBlock, VersionBlockKey};
+use libfuzzer_sys::fuzz_target;
+
+/// Build a Version from an arbitrary deleted flag and block list, using a fixed uuid/backlink
+/// so that CRDT state can be compared across merge results.
+/// Duplicate block keys are dropped before construction.
+/// If deleted, blocks are cleared to ensure a valid initial CRDT state.
+fn make_version(deleted: bool, mut blocks: Vec<(VersionBlockKey, VersionBlock)>) -> Version {
+	blocks.sort_by_key(|(k, _)| *k);
+	blocks.dedup_by_key(|(k, _)| *k);
+	let mut v = Version::new(
+		[0u8; 32].into(),
+		VersionBacklink::Object {
+			bucket_id: [0u8; 32].into(),
+			key: String::new(),
+		},
+		deleted,
+	);
+	for (key, block) in blocks {
+		v.blocks.put(key, block);
+	}
+	if v.deleted.get() {
+		v.blocks.clear();
+	}
+	v
+}
+
+fuzz_target!(|inputs: (
+	(bool, Vec<(VersionBlockKey, VersionBlock)>),
+	(bool, Vec<(VersionBlockKey, VersionBlock)>),
+	(bool, Vec<(VersionBlockKey, VersionBlock)>)
+)| {
+	let ((d1, b1), (d2, b2), (d3, b3)) = inputs;
+	check_crdt_laws(
+		make_version(d1, b1),
+		make_version(d2, b2),
+		make_version(d3, b3),
+	);
+});

fuzz/rust-toolchain.toml

@@ -0,0 +1,2 @@
+[toolchain]
+channel = "nightly"

fuzz/src/lib.rs

@@ -0,0 +1,56 @@
+use garage_table::crdt::Crdt;
+use std::fmt::Debug;
+
+pub fn check_crdt_laws<T>(a: T, b: T, c: T)
+where
+	T: Crdt + PartialEq + Clone + Debug,
+{
+	// Idempotency: merge(a, a) == a
+	{
+		let mut a2 = a.clone();
+		a2.merge(&a);
+		assert_eq!(a2, a, "merge is not idempotent: {a2:#?} != {a:#?}");
+	}
+
+	// Commutativity: merge(a, b) == merge(b, a)
+	let ab = {
+		let mut t = a.clone();
+		t.merge(&b);
+		t
+	};
+	let ba = {
+		let mut t = b.clone();
+		t.merge(&a);
+		t
+	};
+	assert_eq!(ab, ba, "merge is not commutative: {ab:#?} != {ba:#?}");
+
+	// LX's corrolary: merge(merge(a,b),b) = merge(a,b)
+	let ab_b = {
+		let mut t = ab.clone();
+		t.merge(&b);
+		t
+	};
+	assert_eq!(ab, ab_b);
+
+	// Associativity: merge(merge(a, b), c) == merge(a, merge(b, c))
+	let ab_c = {
+		let mut t = ab;
+		t.merge(&c);
+		t
+	};
+	let bc = {
+		let mut t = b;
+		t.merge(&c);
+		t
+	};
+	let a_bc = {
+		let mut t = a;
+		t.merge(&bc);
+		t
+	};
+	assert_eq!(
+		ab_c, a_bc,
+		"merge is not associative: {ab_c:#?} != {a_bc:#?}"
+	);
+}

nix/compile.nix

@@ -48,7 +48,7 @@ let
 
   inherit (pkgs) lib stdenv;
 
-  toolchainFn = (p: p.rust-bin.stable."1.91.0".default.override {
+  toolchainFn = (p: p.rust-bin.stable."1.95.0".default.override {
     targets = lib.optionals (target != null) [ rustTarget ];
     extensions = [
       "rust-src"
@@ -148,6 +148,14 @@ let
 in rec {
   toolchain = toolchainFn pkgs;
 
+  toolchainNightly = pkgs.rust-bin.selectLatestNightlyWith (toolchain: toolchain.default.override {
+    targets = lib.optionals (target != null) [ rustTarget ];
+    extensions = [
+      "rust-src"
+      "rustfmt"
+    ];
+  });
+
   devShell = pkgs.mkShell {
     buildInputs = [
       toolchain

script/helm/garage/README.md

@@ -33,11 +33,14 @@ S3-compatible object store for small self-hosted geo-distributed deployments
 | garage.replicationFactor | string | `"3"` | Default to 3 replicas, see the replication_factor section at https://garagehq.deuxfleurs.fr/documentation/reference-manual/configuration/#replication_factor |
 | garage.consistencyMode | string | `"consistent"` | Default to read-after-write consistency, see the consistency_mode section at https://garagehq.deuxfleurs.fr/documentation/reference-manual/configuration/#consistency_mode |
 | garage.metadataAutoSnapshotInterval | string | `""` | If this value is set, Garage will automatically take a snapshot of the metadata DB file at a regular interval and save it in the metadata directory. https://garagehq.deuxfleurs.fr/documentation/reference-manual/configuration/#metadata_auto_snapshot_interval |
+| garage.admin.apiBindAddr | string | `"[::]:3903"` |  |
 | garage.rpcBindAddr | string | `"[::]:3901"` |  |
 | garage.rpcSecret | string | `""` | If not given, a random secret will be generated and stored in a Secret object |
+| garage.s3.api.bindAddr | string | `"[::]:3900"` |  |
 | garage.s3.api.region | string | `"garage"` |  |
 | garage.s3.api.rootDomain | string | `".s3.garage.tld"` |  |
 | garage.s3.web.index | string | `"index.html"` |  |
+| garage.s3.web.bindAddr | string | `"[::]:3902"` |  |
 | garage.s3.web.rootDomain | string | `".web.garage.tld"` |  |
 | image.pullPolicy | string | `"IfNotPresent"` |  |
 | image.repository | string | `"dxflrs/amd64_garage"` | default to amd64 docker image |

script/helm/garage/templates/_helpers.tpl

@@ -71,6 +71,13 @@ Create the name of the service account to use
 {{- end }}
 {{- end }}
 
+{{/*
+Extract the trailing port number from a bind address like [::]:3900 or 0.0.0.0:3900.
+*/}}
+{{- define "garage.portFromBindAddr" -}}
+{{- regexFind "[0-9]+$" . -}}
+{{- end }}
+
 {{/*
     Returns given number of random Hex characters.
     In practice, it generates up to 100 randAlphaNum strings

script/helm/garage/templates/configmap.yaml

@@ -45,16 +45,16 @@ data:
 
     [s3_api]
     s3_region = "{{ .Values.garage.s3.api.region }}"
-    api_bind_addr = "[::]:3900"
+    api_bind_addr = "{{ .Values.garage.s3.api.bindAddr }}"
     root_domain = "{{ .Values.garage.s3.api.rootDomain }}"
 
     [s3_web]
-    bind_addr = "[::]:3902"
+    bind_addr = "{{ .Values.garage.s3.web.bindAddr }}"
     root_domain = "{{ .Values.garage.s3.web.rootDomain }}"
     index = "{{ .Values.garage.s3.web.index }}"
 
     [admin]
-    api_bind_addr = "[::]:3903"
+    api_bind_addr = "{{ .Values.garage.admin.apiBindAddr }}"
     {{- if .Values.monitoring.tracing.sink }}
     trace_sink = "{{ .Values.monitoring.tracing.sink }}"
     {{- end }}

script/helm/garage/templates/service-headless.yaml

@@ -10,11 +10,11 @@ spec:
   clusterIP: None
   ports:
     - port: {{ .Values.service.s3.api.port }}
-      targetPort: 3900
+      targetPort: {{ include "garage.portFromBindAddr" .Values.garage.s3.api.bindAddr | int }}
       protocol: TCP
       name: s3-api
     - port: {{ .Values.service.s3.web.port }}
-      targetPort: 3902
+      targetPort: {{ include "garage.portFromBindAddr" .Values.garage.s3.web.bindAddr | int }}
       protocol: TCP
       name: s3-web
   selector:

script/helm/garage/templates/service.yaml

@@ -12,11 +12,11 @@ spec:
   type: {{ .Values.service.type }}
   ports:
     - port: {{ .Values.service.s3.api.port }}
-      targetPort: 3900
+      targetPort: {{ include "garage.portFromBindAddr" .Values.garage.s3.api.bindAddr | int }}
       protocol: TCP
       name: s3-api
     - port: {{ .Values.service.s3.web.port }}
-      targetPort: 3902
+      targetPort: {{ include "garage.portFromBindAddr" .Values.garage.s3.web.bindAddr | int }}
       protocol: TCP
       name: s3-web
   selector:
@@ -35,8 +35,8 @@ spec:
   type: ClusterIP
   clusterIP: None
   ports:
-    - port: 3903
-      targetPort: 3903
+    - port: {{ include "garage.portFromBindAddr" .Values.garage.admin.apiBindAddr | int }}
+      targetPort: {{ include "garage.portFromBindAddr" .Values.garage.admin.apiBindAddr | int }}
       protocol: TCP
       name: metrics
   selector:

script/helm/garage/templates/workload.yaml

@@ -60,11 +60,11 @@ spec:
           image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
           imagePullPolicy: {{ .Values.image.pullPolicy }}
           ports:
-            - containerPort: 3900
+            - containerPort: {{ include "garage.portFromBindAddr" .Values.garage.s3.api.bindAddr | int }}
               name: s3-api
-            - containerPort: 3902
+            - containerPort: {{ include "garage.portFromBindAddr" .Values.garage.s3.web.bindAddr | int }}
               name: web-api
-            - containerPort: 3903
+            - containerPort: {{ include "garage.portFromBindAddr" .Values.garage.admin.apiBindAddr | int }}
               name: admin
           {{- with .Values.environment }}
           env:

script/helm/garage/values.yaml

@@ -48,11 +48,15 @@ garage:
   kubernetesSkipCrd: false
   s3:
     api:
+      bindAddr: "[::]:3900"
       region: "garage"
       rootDomain: ".s3.garage.tld"
     web:
+      bindAddr: "[::]:3902"
       rootDomain: ".web.garage.tld"
       index: "index.html"
+  admin:
+    apiBindAddr: "[::]:3903"
 
   # -- Additional configuration to append to garage.toml. Use a multi-line string for custom config.
   # Example:
@@ -221,14 +225,14 @@ resources: {}
 livenessProbe: {}
   #httpGet:
   #  path: /health
-  #  port: 3903
+  #  port: 3903 # or the port from garage.admin.apiBindAddr
   #initialDelaySeconds: 5
   #periodSeconds: 30
 # -- Specifies a readinessProbe
 readinessProbe: {}
   #httpGet:
   #  path: /health
-  #  port: 3903
+  #  port: 3903 # or the port from garage.admin.apiBindAddr
   #initialDelaySeconds: 5
   #periodSeconds: 30
 

src/api/admin/api.rs

@@ -618,6 +618,10 @@ pub enum PreviewClusterLayoutChangesResponse {
 		/// Plain-text information about the layout computation
 		/// (do not try to parse this)
 		message: Vec<String>,
+		/// Structured statistics about the layout computation
+		// FIXME for v3: remove default and skip_serializing_if
+		#[serde(default, skip_serializing_if = "Option::is_none")]
+		statistics: Option<Box<garage_rpc::layout::ComputationStat>>,
 		/// Details about the new cluster layout
 		new_layout: GetClusterLayoutResponse,
 	},
@@ -639,6 +643,10 @@ pub struct ApplyClusterLayoutResponse {
 	/// Plain-text information about the layout computation
 	/// (do not try to parse this)
 	pub message: Vec<String>,
+	/// Structured statistics about the layout computation
+	// FIXME for v3: remove default and skip_serializing_if
+	#[serde(default, skip_serializing_if = "Option::is_none")]
+	pub statistics: Option<garage_rpc::layout::ComputationStat>,
 	/// Details about the new cluster layout
 	pub layout: GetClusterLayoutResponse,
 }
@@ -1165,6 +1173,29 @@ pub struct LocalGetNodeInfoResponse {
 	pub rust_version: String,
 	/// database engine used for metadata
 	pub db_engine: String,
+	// FIXME for v3: remove Option<> and serde(default) for field below
+	// FIXME for v3: merge LocalGetNodeInfoResponse and NodeResp
+	/// Socket address used by other nodes to connect to this node for RPC
+	#[serde(default, skip_serializing_if = "Option::is_none")]
+	#[schema(value_type = Option<String>)]
+	pub addr: Option<SocketAddr>,
+	/// Whether this node is connected in the cluster
+	#[serde(default, skip_serializing_if = "Option::is_none")]
+	pub is_up: Option<bool>,
+	/// Role assigned to this node in the current cluster layout
+	#[serde(default, skip_serializing_if = "Option::is_none")]
+	pub role: Option<NodeAssignedRole>,
+	/// Whether this node is part of an older layout version and is draining data.
+	#[serde(default, skip_serializing_if = "Option::is_none")]
+	pub draining: Option<bool>,
+	/// Total and available space on the disk partition(s) containing the data
+	/// directory(ies)
+	#[serde(default, skip_serializing_if = "Option::is_none")]
+	pub data_partition: Option<FreeSpaceResp>,
+	/// Total and available space on the disk partition containing the
+	/// metadata directory
+	#[serde(default, skip_serializing_if = "Option::is_none")]
+	pub metadata_partition: Option<FreeSpaceResp>,
 }
 
 // ---- GetNodeStatistics ----

src/api/admin/layout.rs

@@ -53,7 +53,7 @@ fn format_cluster_layout(layout: &layout::LayoutHistory) -> GetClusterLayoutResp
 		.roles
 		.items()
 		.iter()
-		.filter(|(k, _, v)| current.roles.get(k) != Some(v))
+		.filter(|(k, _, v)| current.roles.get(k).and_then(|vv| vv.0.as_ref()) != v.0.as_ref())
 		.map(|(k, _, v)| match &v.0 {
 			None => NodeRoleChange {
 				id: hex::encode(k),
@@ -255,10 +255,14 @@ impl RequestHandler for PreviewClusterLayoutChangesRequest {
 				Ok(PreviewClusterLayoutChangesResponse::Error { error })
 			}
 			Err(e) => Err(e.into()),
-			Ok((new_layout, msg)) => Ok(PreviewClusterLayoutChangesResponse::Success {
-				message: msg,
-				new_layout: format_cluster_layout(&new_layout),
-			}),
+			Ok((new_layout, stat)) => {
+				let message = stat.to_message();
+				Ok(PreviewClusterLayoutChangesResponse::Success {
+					message,
+					statistics: Some(Box::new(stat)),
+					new_layout: format_cluster_layout(&new_layout),
+				})
+			}
 		}
 	}
 }
@@ -272,7 +276,8 @@ impl RequestHandler for ApplyClusterLayoutRequest {
 		_admin: &Admin,
 	) -> Result<ApplyClusterLayoutResponse, Error> {
 		let layout = garage.system.cluster_layout().inner().clone();
-		let (layout, msg) = layout.apply_staged_changes(self.version)?;
+		let (layout, stat) = layout.apply_staged_changes(self.version)?;
+		let message = stat.to_message();
 
 		garage
 			.system
@@ -281,7 +286,8 @@ impl RequestHandler for ApplyClusterLayoutRequest {
 			.await?;
 
 		Ok(ApplyClusterLayoutResponse {
-			message: msg,
+			message,
+			statistics: Some(stat),
 			layout: format_cluster_layout(&layout),
 		})
 	}

src/api/admin/node.rs

@@ -25,6 +25,9 @@ impl RequestHandler for LocalGetNodeInfoRequest {
 		let sys_status = garage.system.local_status();
 		let hostname = sys_status.hostname.unwrap_or_default().to_string();
 
+		let layout = garage.system.cluster_layout();
+		let current_layout = layout.inner().current();
+
 		Ok(LocalGetNodeInfoResponse {
 			node_id: hex::encode(garage.system.id),
 			hostname: Some(hostname),
@@ -33,6 +36,41 @@ impl RequestHandler for LocalGetNodeInfoRequest {
 				.map(|features| features.iter().map(ToString::to_string).collect()),
 			rust_version: garage_util::version::rust_version().to_string(),
 			db_engine: garage.db.engine(),
+			is_up: Some(true),
+			addr: garage
+				.system
+				.get_known_nodes()
+				.iter()
+				.find(|x| x.id == garage.system.id)
+				.and_then(|x| x.addr),
+			draining: Some(
+				current_layout.node_role(&garage.system.id).is_none()
+					&& layout
+						.inner()
+						.versions
+						.iter()
+						.filter(|x| x.version != current_layout.version)
+						.any(|x| x.node_role(&garage.system.id).is_some()),
+			),
+			role: current_layout
+				.node_role(&garage.system.id)
+				.map(|v| NodeAssignedRole {
+					zone: v.zone.clone(),
+					capacity: v.capacity,
+					tags: v.tags.clone(),
+				}),
+			data_partition: sys_status
+				.data_disk_avail
+				.map(|(avail, total)| FreeSpaceResp {
+					available: avail,
+					total,
+				}),
+			metadata_partition: sys_status
+				.meta_disk_avail
+				.map(|(avail, total)| FreeSpaceResp {
+					available: avail,
+					total,
+				}),
 		})
 	}
 }

src/api/common/cors.rs

@@ -1,8 +1,9 @@
 use std::sync::Arc;
 
 use http::header::{
-	ACCESS_CONTROL_ALLOW_HEADERS, ACCESS_CONTROL_ALLOW_METHODS, ACCESS_CONTROL_ALLOW_ORIGIN,
-	ACCESS_CONTROL_EXPOSE_HEADERS, ACCESS_CONTROL_REQUEST_HEADERS, ACCESS_CONTROL_REQUEST_METHOD,
+	HeaderValue, ACCESS_CONTROL_ALLOW_HEADERS, ACCESS_CONTROL_ALLOW_METHODS,
+	ACCESS_CONTROL_ALLOW_ORIGIN, ACCESS_CONTROL_EXPOSE_HEADERS, ACCESS_CONTROL_REQUEST_HEADERS,
+	ACCESS_CONTROL_REQUEST_METHOD, VARY,
 };
 use hyper::{body::Body, body::Incoming as IncomingBody, Request, Response, StatusCode};
 
@@ -12,10 +13,12 @@ use garage_model::garage::Garage;
 use crate::common_error::{CommonError, OkOrBadRequest, OkOrInternalError};
 use crate::helpers::*;
 
+// Return both the matching rule and the parsed Origin header so callers that
+// apply CORS headers don't have to repeat Origin lookup and validation.
 pub fn find_matching_cors_rule<'a, B>(
 	bucket_params: &'a BucketParams,
-	req: &Request<B>,
-) -> Result<Option<&'a GarageCorsRule>, CommonError> {
+	req: &'a Request<B>,
+) -> Result<Option<(&'a GarageCorsRule, &'a str)>, CommonError> {
 	if let Some(cors_config) = bucket_params.cors_config.get() {
 		if let Some(origin) = req.headers().get("Origin") {
 			let origin = origin.to_str()?;
@@ -23,9 +26,12 @@ pub fn find_matching_cors_rule<'a, B>(
 				Some(h) => h.to_str()?.split(',').map(|h| h.trim()).collect::<Vec<_>>(),
 				None => vec![],
 			};
-			return Ok(cors_config.iter().find(|rule| {
-				cors_rule_matches(rule, origin, req.method().as_ref(), request_headers.iter())
-			}));
+			return Ok(cors_config
+				.iter()
+				.find(|rule| {
+					cors_rule_matches(rule, origin, req.method().as_ref(), request_headers.iter())
+				})
+				.map(|rule| (rule, origin)));
 		}
 	}
 	Ok(None)
@@ -53,12 +59,16 @@ where
 pub fn add_cors_headers(
 	resp: &mut Response<impl Body>,
 	rule: &GarageCorsRule,
+	request_origin: &str,
 ) -> Result<(), http::header::InvalidHeaderValue> {
 	let h = resp.headers_mut();
-	h.insert(
-		ACCESS_CONTROL_ALLOW_ORIGIN,
-		rule.allow_origins.join(", ").parse()?,
-	);
+	let is_wildcard_origin = rule.allow_origins.iter().any(|origin| origin == "*");
+	let allow_origin = if is_wildcard_origin {
+		"*"
+	} else {
+		request_origin
+	};
+	h.insert(ACCESS_CONTROL_ALLOW_ORIGIN, allow_origin.parse()?);
 	h.insert(
 		ACCESS_CONTROL_ALLOW_METHODS,
 		rule.allow_methods.join(", ").parse()?,
@@ -71,6 +81,12 @@ pub fn add_cors_headers(
 		ACCESS_CONTROL_EXPOSE_HEADERS,
 		rule.expose_headers.join(", ").parse()?,
 	);
+	// When ACAO reflects the request origin instead of returning "*",
+	// caches must vary on the Origin request header to avoid reusing
+	// a response generated for one origin when serving another origin.
+	if !is_wildcard_origin {
+		h.insert(VARY, HeaderValue::from_static("Origin"));
+	}
 	Ok(())
 }
 
@@ -107,6 +123,7 @@ pub fn handle_options_api(
 			Ok(Response::builder()
 				.header(ACCESS_CONTROL_ALLOW_ORIGIN, "*")
 				.header(ACCESS_CONTROL_ALLOW_METHODS, "*")
+				.header(ACCESS_CONTROL_ALLOW_HEADERS, "*")
 				.status(StatusCode::OK)
 				.body(EmptyBody::new())?)
 		}
@@ -149,7 +166,17 @@ pub fn handle_options_for_bucket<B>(
 			let mut resp = Response::builder()
 				.status(StatusCode::OK)
 				.body(EmptyBody::new())?;
-			add_cors_headers(&mut resp, rule).ok_or_internal_error("Invalid CORS configuration")?;
+			add_cors_headers(&mut resp, rule, origin)
+				.ok_or_internal_error("Invalid CORS configuration")?;
+			// Preflight responses vary not only on Origin but also on the
+			// requested method and requested headers, so caches must not
+			// reuse one preflight decision for a different preflight input.
+			resp.headers_mut().insert(
+				VARY,
+				"Origin, Access-Control-Request-Method, Access-Control-Request-Headers"
+					.parse()
+					.expect("static vary header"),
+			);
 			return Ok(resp);
 		}
 	}
@@ -158,3 +185,98 @@ pub fn handle_options_for_bucket<B>(
 		"This CORS request is not allowed.".into(),
 	))
 }
+
+#[cfg(test)]
+mod tests {
+	use super::*;
+
+	fn bucket_params_with_rule(allow_origins: Vec<&str>) -> BucketParams {
+		let mut bucket_params = BucketParams::default();
+		bucket_params.cors_config.update(Some(vec![GarageCorsRule {
+			id: Some("cors-test".into()),
+			max_age_seconds: None,
+			allow_origins: allow_origins.into_iter().map(str::to_string).collect(),
+			allow_methods: vec!["GET".into(), "PUT".into()],
+			allow_headers: vec!["*".into()],
+			expose_headers: vec![],
+		}]));
+		bucket_params
+	}
+
+	fn preflight_request(origin: &str) -> Request<()> {
+		Request::builder()
+			.method("OPTIONS")
+			.uri("http://example.test/bucket")
+			.header("Origin", origin)
+			.header(ACCESS_CONTROL_REQUEST_METHOD, "PUT")
+			.body(())
+			.unwrap()
+	}
+
+	#[test]
+	fn preflight_with_single_allowed_origin_returns_request_origin() {
+		let bucket_params = bucket_params_with_rule(vec!["https://app.example.test"]);
+		let req = preflight_request("https://app.example.test");
+
+		let resp = handle_options_for_bucket(&req, &bucket_params).unwrap();
+
+		assert_eq!(
+			resp.headers().get(ACCESS_CONTROL_ALLOW_ORIGIN).unwrap(),
+			"https://app.example.test"
+		);
+		let vary_values: Vec<_> = resp
+			.headers()
+			.get_all(VARY)
+			.iter()
+			.map(|value| value.to_str().unwrap())
+			.collect();
+		assert_eq!(
+			vary_values,
+			vec!["Origin, Access-Control-Request-Method, Access-Control-Request-Headers",]
+		);
+	}
+
+	#[test]
+	fn preflight_with_multiple_allowed_origins_reflects_request_origin() {
+		let bucket_params = bucket_params_with_rule(vec![
+			"https://app.example.test",
+			"https://admin.example.test",
+		]);
+		let req = preflight_request("https://app.example.test");
+
+		let resp = handle_options_for_bucket(&req, &bucket_params).unwrap();
+
+		// This assertion documents the behavior browsers expect:
+		// even if multiple origins are allowed by configuration, the
+		// response should reflect the request origin rather than emit
+		// a comma-separated list. It currently fails and is meant to
+		// turn green once header generation is corrected.
+		assert_eq!(
+			resp.headers().get(ACCESS_CONTROL_ALLOW_ORIGIN).unwrap(),
+			"https://app.example.test"
+		);
+	}
+
+	#[test]
+	fn preflight_with_wildcard_allowed_origin_returns_wildcard() {
+		let bucket_params = bucket_params_with_rule(vec!["*"]);
+		let req = preflight_request("https://app.example.test");
+
+		let resp = handle_options_for_bucket(&req, &bucket_params).unwrap();
+
+		assert_eq!(
+			resp.headers().get(ACCESS_CONTROL_ALLOW_ORIGIN).unwrap(),
+			"*"
+		);
+		let vary_values: Vec<_> = resp
+			.headers()
+			.get_all(VARY)
+			.iter()
+			.map(|value| value.to_str().unwrap())
+			.collect();
+		assert_eq!(
+			vary_values,
+			vec!["Origin, Access-Control-Request-Method, Access-Control-Request-Headers",]
+		);
+	}
+}

src/api/common/signature/payload.rs

@@ -357,7 +357,13 @@ pub fn canonical_request(
 		items.join("&")
 	};
 
-	// Canonical header string calculated from signed headers
+	// Canonical header string calculated from signed headers.
+	//
+	// Per the SigV4 spec, signed header values must have sequential
+	// internal whitespace collapsed to a single space, in addition to
+	// being trimmed. AWS SDKs do this before computing the signature
+	// but transmit the raw value on the wire, so we must match.
+	// -> https://docs.aws.amazon.com/IAM/latest/UserGuide/create-signed-request.html
 	let canonical_header_string = signed_headers
 		.iter()
 		.map(|name| {
@@ -372,7 +378,11 @@ pub fn canonical_request(
 				built_string.push(',');
 				built_string.push_str(extend_string);
 			}
-			Ok(format!("{}:{}", name.as_str(), built_string.trim()))
+			let normalized = built_string
+				.split_whitespace()
+				.collect::<Vec<_>>()
+				.join(" ");
+			Ok(format!("{}:{}", name.as_str(), normalized))
 		})
 		.collect::<Result<Vec<String>, Error>>()?
 		.join("\n");

src/api/k2v/api_server.rs

@@ -111,7 +111,7 @@ impl ApiHandler for K2VApiServer {
 			Method::GET | Method::HEAD | Method::POST => {
 				find_matching_cors_rule(&bucket_params, &req)
 					.ok_or_internal_error("Error looking up CORS rule")?
-					.cloned()
+					.map(|(rule, origin)| (rule.clone(), origin.to_string()))
 			}
 			_ => None,
 		};
@@ -164,8 +164,8 @@ impl ApiHandler for K2VApiServer {
 		// If request was a success and we have a CORS rule that applies to it,
 		// add the corresponding CORS headers to the response
 		let mut resp_ok = resp?;
-		if let Some(rule) = matching_cors_rule {
-			add_cors_headers(&mut resp_ok, &rule)
+		if let Some((rule, origin)) = matching_cors_rule {
+			add_cors_headers(&mut resp_ok, &rule, &origin)
 				.ok_or_internal_error("Invalid bucket CORS configuration")?;
 		}
 

src/api/s3/api_server.rs

@@ -159,7 +159,8 @@ impl ApiHandler for S3ApiServer {
 			return Err(Error::forbidden("Operation is not allowed for this key."));
 		}
 
-		let matching_cors_rule = find_matching_cors_rule(&bucket_params, &req)?.cloned();
+		let matching_cors = find_matching_cors_rule(&bucket_params, &req)?
+			.map(|(rule, origin)| (rule.clone(), origin.to_string()));
 
 		let ctx = ReqCtx {
 			garage,
@@ -334,8 +335,8 @@ impl ApiHandler for S3ApiServer {
 		// If request was a success and we have a CORS rule that applies to it,
 		// add the corresponding CORS headers to the response
 		let mut resp_ok = resp?;
-		if let Some(rule) = matching_cors_rule {
-			add_cors_headers(&mut resp_ok, &rule)
+		if let Some((rule, origin)) = matching_cors {
+			add_cors_headers(&mut resp_ok, &rule, &origin)
 				.ok_or_internal_error("Invalid bucket CORS configuration")?;
 		}
 

src/api/s3/post_object.rs

@@ -121,7 +121,7 @@ pub async fn handle_post_object(
 		&bucket_params,
 		&Request::from_parts(head.clone(), empty_body::<Infallible>()),
 	)?
-	.cloned();
+	.map(|(rule, origin)| (rule.clone(), origin.to_string()));
 
 	let decoded_policy = BASE64_STANDARD
 		.decode(policy)
@@ -351,8 +351,8 @@ pub async fn handle_post_object(
 		}
 	};
 
-	if let Some(rule) = matching_cors_rule {
-		add_cors_headers(&mut resp, &rule)
+	if let Some((rule, origin)) = matching_cors_rule {
+		add_cors_headers(&mut resp, &rule, &origin)
 			.ok_or_internal_error("Invalid bucket CORS configuration")?;
 	}
 
@@ -473,12 +473,10 @@ where
 					))));
 				}
 			}
-			Poll::Ready(None) => {
-				if !self.length.contains(&self.read) {
-					return Poll::Ready(Some(Err(Error::bad_request(
-						"File size does not match policy",
-					))));
-				}
+			Poll::Ready(None) if !self.length.contains(&self.read) => {
+				return Poll::Ready(Some(Err(Error::bad_request(
+					"File size does not match policy",
+				))));
 			}
 			_ => {}
 		}

src/block/manager.rs

@@ -224,7 +224,13 @@ impl BlockManager {
 				|p, tranquility| p.set_with(|x| x.tranquility = tranquility),
 			);
 			vars.register_ro(&self.scrub_persister, "scrub-last-completed", |p| {
-				p.get_with(|x| msec_to_rfc3339(x.time_last_complete_scrub))
+				p.get_with(|x| {
+					if x.time_last_complete_scrub == 0 {
+						"never".to_string()
+					} else {
+						msec_to_rfc3339(x.time_last_complete_scrub)
+					}
+				})
 			});
 			vars.register_ro(&self.scrub_persister, "scrub-next-run", |p| {
 				p.get_with(|x| msec_to_rfc3339(x.time_next_run_scrub))

src/block/repair.rs

@@ -648,7 +648,7 @@ impl BlockStoreIterator {
 
 		let mut cum_cap = 0;
 		let mut todo = vec![];
-		for (dir, cap) in data_layout.data_dirs.iter().zip(dir_cap.into_iter()) {
+		for (dir, cap) in data_layout.data_dirs.iter().zip(dir_cap) {
 			let progress_min = (cum_cap * PROGRESS_FP) / sum_cap;
 			let progress_max = ((cum_cap + cap as u64) * PROGRESS_FP) / sum_cap;
 			cum_cap += cap as u64;

src/db/sqlite_adapter.rs

@@ -426,15 +426,20 @@ impl<'a> ITx for SqliteTx<'a> {
 // complicated, they must hold the Statement and Row objects
 // therefore quite some unsafe code (it is a self-referential struct)
 
-struct DbValueIterator<'a> {
+struct DbValueIterator {
 	db: Connection,
-	stmt: Option<Statement<'a>>,
-	iter: Option<Rows<'a>>,
+	// These two are not really static (they are actually self referential :o)
+	stmt: Option<Statement<'static>>,
+	iter: Option<Rows<'static>>,
 	_pin: PhantomPinned,
 }
 
-impl<'a> DbValueIterator<'a> {
-	fn make<P: rusqlite::Params>(db: Connection, sql: &str, args: P) -> Result<ValueIter<'a>> {
+impl DbValueIterator {
+	fn make<'res, P: rusqlite::Params>(
+		db: Connection,
+		sql: &str,
+		args: P,
+	) -> Result<ValueIter<'res>> {
 		let res = DbValueIterator {
 			db,
 			stmt: None,
@@ -468,7 +473,7 @@ impl<'a> DbValueIterator<'a> {
 	}
 }
 
-impl<'a> Drop for DbValueIterator<'a> {
+impl Drop for DbValueIterator {
 	fn drop(&mut self) {
 		trace!("drop iter");
 		drop(self.iter.take());
@@ -476,9 +481,9 @@ impl<'a> Drop for DbValueIterator<'a> {
 	}
 }
 
-struct DbValueIteratorPin<'a>(Pin<Box<DbValueIterator<'a>>>);
+struct DbValueIteratorPin(Pin<Box<DbValueIterator>>);
 
-impl<'a> Iterator for DbValueIteratorPin<'a> {
+impl Iterator for DbValueIteratorPin {
 	type Item = Result<(Value, Value)>;
 
 	fn next(&mut self) -> Option<Self::Item> {

src/garage/cli/remote/cluster.rs

@@ -9,6 +9,31 @@ use crate::cli::remote::*;
 use crate::cli::structs::*;
 
 impl Cli {
+	pub async fn cmd_health(&self, quiet: bool) -> Result<(), Error> {
+		let health = self.api_request(GetClusterHealthRequest).await?;
+
+		if !quiet {
+			let table = vec![
+				format!("Cluster health:\t{}", health.status.to_uppercase()),
+				format!("Known nodes:\t{}", health.known_nodes),
+				format!("Connected nodes:\t{}", health.connected_nodes),
+				format!("Storage nodes:\t{}", health.storage_nodes),
+				format!("Storage nodes up:\t{}", health.storage_nodes_up),
+				format!("Partitions:\t{}", health.partitions),
+				format!("Partitions with quorum:\t{}", health.partitions_quorum),
+				format!("Fully healthy partitions:\t{}", health.partitions_all_ok),
+			];
+			format_table(table);
+		}
+
+		match health.status.as_str() {
+			"unavailable" => Err(Error::Message(
+				"Cluster is currently unavailable".to_string(),
+			)),
+			_ => Ok(()),
+		}
+	}
+
 	pub async fn cmd_status(&self) -> Result<(), Error> {
 		let status = self.api_request(GetClusterStatusRequest).await?;
 		let layout = self.api_request(GetClusterLayoutRequest).await?;

src/garage/cli/remote/layout.rs

@@ -40,6 +40,7 @@ impl Cli {
 				PreviewClusterLayoutChangesResponse::Success {
 					message,
 					new_layout,
+					..
 				} => {
 					println!();
 					println!("==== NEW CLUSTER LAYOUT AFTER APPLYING CHANGES ====");

src/garage/cli/remote/mod.rs

@@ -33,6 +33,7 @@ impl Cli {
 	pub async fn handle(&self, cmd: Command) -> Result<(), Error> {
 		match cmd {
 			Command::Status => self.cmd_status().await,
+			Command::Health(opt) => self.cmd_health(opt.quiet).await,
 			Command::Node(NodeOperation::Connect(connect_opt)) => {
 				self.cmd_connect(connect_opt).await
 			}

src/garage/cli/structs.rs

@@ -10,6 +10,10 @@ pub enum Command {
 	#[structopt(name = "server", version = garage_version())]
 	Server(ServerOpt),
 
+	/// Check the cluster health and set the exit code to 1 if it is unavailable
+	#[structopt(name = "health", version = garage_version())]
+	Health(HealthOpt),
+
 	/// Get network status
 	#[structopt(name = "status", version = garage_version())]
 	Status,
@@ -103,6 +107,17 @@ pub struct ServerOpt {
 	pub(crate) default_bucket: bool,
 }
 
+// -------------------------
+// ---- garage health ----
+// -------------------------
+
+#[derive(StructOpt, Debug)]
+pub struct HealthOpt {
+	/// Do not print healthyness to stdout
+	#[structopt(short = "q", long = "quiet")]
+	pub(crate) quiet: bool,
+}
+
 // -------------------------
 // ---- garage node ... ----
 // -------------------------

src/garage/secrets.rs

@@ -19,7 +19,12 @@ pub struct Secrets {
 
 	/// RPC secret network key, used to replace `rpc_secret` in config.toml when running the
 	/// daemon or doing admin operations
-	#[structopt(short = "s", long = "rpc-secret", env = "GARAGE_RPC_SECRET")]
+	#[structopt(
+		short = "s",
+		long = "rpc-secret",
+		env = "GARAGE_RPC_SECRET",
+		hide_env_values = true
+	)]
 	pub rpc_secret: Option<String>,
 
 	/// RPC secret network key, used to replace `rpc_secret` in config.toml and rpc-secret
@@ -29,7 +34,11 @@ pub struct Secrets {
 
 	/// Admin API authentication token, replaces `admin.admin_token` in config.toml when
 	/// running the Garage daemon
-	#[structopt(long = "admin-token", env = "GARAGE_ADMIN_TOKEN")]
+	#[structopt(
+		long = "admin-token",
+		env = "GARAGE_ADMIN_TOKEN",
+		hide_env_values = true
+	)]
 	pub admin_token: Option<String>,
 
 	/// Admin API authentication token file path, replaces `admin.admin_token` in config.toml
@@ -39,7 +48,11 @@ pub struct Secrets {
 
 	/// Metrics API authentication token, replaces `admin.metrics_token` in config.toml when
 	/// running the Garage daemon
-	#[structopt(long = "metrics-token", env = "GARAGE_METRICS_TOKEN")]
+	#[structopt(
+		long = "metrics-token",
+		env = "GARAGE_METRICS_TOKEN",
+		hide_env_values = true
+	)]
 	pub metrics_token: Option<String>,
 
 	/// Metrics API authentication token file path, replaces `admin.metrics_token` in config.toml

src/garage/server.rs

@@ -228,10 +228,10 @@ async fn initial_config(garage: &Arc<Garage>, opt: ServerOpt) -> Result<(), Erro
 				})),
 			);
 
-			let (layout, msg) = layout.apply_staged_changes(1)?;
+			let (layout, stat) = layout.apply_staged_changes(1)?;
 			info!(
 				"Created initial layout for single-node configuration:\n{}",
-				msg.join("\n")
+				stat.to_message().join("\n")
 			);
 
 			garage

src/garage/tests/k2v/item.rs

@@ -11,6 +11,7 @@ use http_body_util::BodyExt;
 use hyper::{Method, StatusCode};
 
 #[tokio::test]
+#[ignore = "flaky"]
 async fn test_items_and_indices() {
 	let ctx = common::context();
 	let bucket = ctx.create_bucket("test-k2v-item-and-index");

src/garage/tests/s3/cors.rs

@@ -0,0 +1,121 @@
+use aws_sdk_s3::types::{CorsConfiguration, CorsRule};
+use hyper::{Method, StatusCode};
+
+use crate::common;
+
+const REQUEST_ORIGIN: &str = "https://app.example.test";
+const SECOND_ALLOWED_ORIGIN: &str = "https://admin.example.test";
+const OBJECT_KEY: &str = "probe.txt";
+const BODY: &[u8] = b"hello from integration repro\n";
+
+async fn send_preflight(
+	ctx: &common::Context,
+	bucket: &str,
+	origin: &str,
+) -> hyper::Response<common::custom_requester::Body> {
+	ctx.custom_request
+		.builder(bucket.to_string())
+		.method(Method::OPTIONS)
+		.path(OBJECT_KEY)
+		.unsigned_header("origin", origin)
+		.unsigned_header("access-control-request-method", "PUT")
+		.unsigned_header(
+			"access-control-request-headers",
+			"content-type,x-amz-meta-demo",
+		)
+		.body(vec![])
+		.send()
+		.await
+		.unwrap()
+}
+
+async fn send_put(
+	ctx: &common::Context,
+	bucket: &str,
+	origin: &str,
+) -> hyper::Response<common::custom_requester::Body> {
+	ctx.custom_request
+		.builder(bucket.to_string())
+		.method(Method::PUT)
+		.path(OBJECT_KEY)
+		.signed_header("content-type", "text/plain")
+		.signed_header("x-amz-meta-demo", "1")
+		.unsigned_header("origin", origin)
+		.body(BODY.to_vec())
+		.send()
+		.await
+		.unwrap()
+}
+
+async fn apply_bucket_cors(ctx: &common::Context, bucket: &str, allowed_origins: &[&str]) {
+	let rule = allowed_origins.iter().fold(
+		CorsRule::builder()
+			.allowed_headers("*")
+			.allowed_methods("PUT")
+			.expose_headers("ETag"),
+		|rule, origin| rule.allowed_origins(*origin),
+	);
+
+	let cors = CorsConfiguration::builder()
+		.cors_rules(rule.build().unwrap())
+		.build()
+		.unwrap();
+
+	ctx.client
+		.put_bucket_cors()
+		.bucket(bucket)
+		.cors_configuration(cors)
+		.send()
+		.await
+		.unwrap();
+}
+
+#[tokio::test]
+async fn test_s3_api_cors_reflects_request_origin() {
+	let ctx = common::context();
+	let bucket = ctx.create_bucket("s3-cors-direct");
+
+	apply_bucket_cors(&ctx, &bucket, &[REQUEST_ORIGIN]).await;
+
+	let control_preflight = send_preflight(&ctx, &bucket, REQUEST_ORIGIN).await;
+	assert_eq!(control_preflight.status(), StatusCode::OK);
+	assert_eq!(
+		control_preflight
+			.headers()
+			.get("access-control-allow-origin")
+			.unwrap(),
+		REQUEST_ORIGIN
+	);
+
+	let control_put = send_put(&ctx, &bucket, REQUEST_ORIGIN).await;
+	assert_eq!(control_put.status(), StatusCode::OK);
+	assert_eq!(
+		control_put
+			.headers()
+			.get("access-control-allow-origin")
+			.unwrap(),
+		REQUEST_ORIGIN
+	);
+
+	apply_bucket_cors(&ctx, &bucket, &[REQUEST_ORIGIN, SECOND_ALLOWED_ORIGIN]).await;
+
+	let repro_preflight = send_preflight(&ctx, &bucket, REQUEST_ORIGIN).await;
+	assert_eq!(repro_preflight.status(), StatusCode::OK);
+	assert_eq!(
+		repro_preflight
+			.headers()
+			.get("access-control-allow-origin")
+			.unwrap(),
+		REQUEST_ORIGIN
+	);
+
+	let repro_put = send_put(&ctx, &bucket, REQUEST_ORIGIN).await;
+	assert_eq!(repro_put.status(), StatusCode::OK);
+	assert_eq!(
+		repro_put
+			.headers()
+			.get("access-control-allow-origin")
+			.unwrap(),
+		REQUEST_ORIGIN
+	);
+}

src/garage/tests/s3/mod.rs

@@ -1,3 +1,4 @@
+mod cors;
 mod list;
 mod multipart;
 mod objects;

src/garage/tests/s3/presigned.rs

@@ -70,3 +70,43 @@ async fn test_presigned_url() {
 		assert_eq!(body, body2);
 	}
 }
+
+// Presigned PUT with a user-metadata header whose value contains
+// internal sequential whitespace. SigV4 requires collapsing such
+// whitespace in canonical header values; missing that normalization
+// produces an `Invalid signature` 403 on otherwise-valid requests.
+#[tokio::test]
+async fn test_presigned_put_with_user_metadata() {
+	let ctx = common::context();
+	let bucket = ctx.create_bucket("presigned-metadata");
+
+	let key = "cache-archive";
+	let metadata_value = "cache-key  --protected";
+	let body = Bytes::from_static(b"presigned PUT with user metadata");
+
+	let psc = PresigningConfig::builder()
+		.start_time(SystemTime::now() - Duration::from_secs(60))
+		.expires_in(Duration::from_secs(3600))
+		.build()
+		.unwrap();
+
+	let presigned = ctx
+		.client
+		.put_object()
+		.bucket(&bucket)
+		.key(key)
+		.metadata("cachekey", metadata_value)
+		.presigned(psc)
+		.await
+		.unwrap();
+
+	let req_builder = Request::builder().method("PUT").uri(presigned.uri());
+	let req = presigned
+		.headers()
+		.fold(req_builder, |b, (k, v)| b.header(k, v))
+		.body(Full::new(body))
+		.unwrap();
+
+	let res = ctx.custom_request.client().request(req).await.unwrap();
+	assert_eq!(res.status(), 200);
+}

src/model/Cargo.toml

@@ -21,6 +21,7 @@ garage_block.workspace = true
 garage_util.workspace = true
 garage_net.workspace = true
 
+arbitrary = { optional = true, workspace = true }
 argon2.workspace = true
 async-trait.workspace = true
 blake2.workspace = true
@@ -46,6 +47,7 @@ k2v = ["garage_util/k2v"]
 lmdb = ["garage_db/lmdb"]
 sqlite = ["garage_db/sqlite"]
 fjall = ["garage_db/fjall"]
+arbitrary = ["dep:arbitrary","garage_util/arbitrary"]
 
 [lints]
 workspace = true

src/model/admin_token_table.rs

@@ -51,6 +51,14 @@ mod v2 {
 
 pub use v2::*;
 
+#[cfg(feature = "arbitrary")]
+impl<'a> arbitrary::Arbitrary<'a> for AdminApiTokenScope {
+	fn arbitrary(u: &mut arbitrary::Unstructured<'a>) -> arbitrary::Result<Self> {
+		let set: std::collections::BTreeSet<String> = arbitrary::Arbitrary::arbitrary(u)?;
+		Ok(AdminApiTokenScope(set.into_iter().collect()))
+	}
+}
+
 impl Crdt for AdminApiTokenParams {
 	fn merge(&mut self, o: &Self) {
 		self.name.merge(&o.name);

src/model/bucket_table.rs

@@ -63,6 +63,7 @@ mod v08 {
 	}
 
 	#[derive(PartialEq, Eq, Clone, Debug, Serialize, Deserialize)]
+	#[cfg_attr(feature = "arbitrary", derive(arbitrary::Arbitrary))]
 	pub struct CorsRule {
 		pub id: Option<String>,
 		pub max_age_seconds: Option<u64>,
@@ -74,6 +75,7 @@ mod v08 {
 
 	/// Lifecycle configuration rule
 	#[derive(PartialEq, Eq, Clone, Debug, Serialize, Deserialize)]
+	#[cfg_attr(feature = "arbitrary", derive(arbitrary::Arbitrary))]
 	pub struct LifecycleRule {
 		/// The ID of the rule
 		pub id: Option<String>,
@@ -91,6 +93,7 @@ mod v08 {
 	/// For each condition, if it is None, it is not verified (always true),
 	/// and if it is Some(x), then it is verified for value x
 	#[derive(PartialEq, Eq, Clone, Debug, Serialize, Deserialize, Default)]
+	#[cfg_attr(feature = "arbitrary", derive(arbitrary::Arbitrary))]
 	pub struct LifecycleFilter {
 		/// If Some(x), object key has to start with prefix x
 		pub prefix: Option<String>,
@@ -101,6 +104,7 @@ mod v08 {
 	}
 
 	#[derive(PartialEq, Eq, Clone, Debug, Serialize, Deserialize)]
+	#[cfg_attr(feature = "arbitrary", derive(arbitrary::Arbitrary))]
 	pub enum LifecycleExpiration {
 		/// Objects expire x days after they were created
 		AfterDays(usize),
@@ -109,6 +113,7 @@ mod v08 {
 	}
 
 	#[derive(Default, PartialEq, Eq, PartialOrd, Ord, Clone, Debug, Serialize, Deserialize)]
+	#[cfg_attr(feature = "arbitrary", derive(arbitrary::Arbitrary))]
 	pub struct BucketQuotas {
 		/// Maximum size in bytes (bucket size = sum of sizes of objects in the bucket)
 		pub max_size: Option<u64>,
@@ -139,6 +144,7 @@ mod v2 {
 
 	/// Configuration for a bucket
 	#[derive(PartialEq, Eq, Clone, Debug, Serialize, Deserialize)]
+	#[cfg_attr(feature = "arbitrary", derive(arbitrary::Arbitrary))]
 	pub struct BucketParams {
 		/// Bucket's creation date
 		pub creation_date: u64,
@@ -168,6 +174,7 @@ mod v2 {
 	}
 
 	#[derive(PartialEq, Eq, Clone, Debug, Serialize, Deserialize)]
+	#[cfg_attr(feature = "arbitrary", derive(arbitrary::Arbitrary))]
 	pub struct WebsiteConfig {
 		pub index_document: String,
 		pub error_document: Option<String>,
@@ -178,24 +185,28 @@ mod v2 {
 	}
 
 	#[derive(PartialEq, Eq, Clone, Debug, Serialize, Deserialize)]
+	#[cfg_attr(feature = "arbitrary", derive(arbitrary::Arbitrary))]
 	pub struct RedirectAll {
 		pub hostname: String,
 		pub protocol: String,
 	}
 
 	#[derive(PartialEq, Eq, Clone, Debug, Serialize, Deserialize)]
+	#[cfg_attr(feature = "arbitrary", derive(arbitrary::Arbitrary))]
 	pub struct RoutingRule {
 		pub condition: Option<RedirectCondition>,
 		pub redirect: Redirect,
 	}
 
 	#[derive(PartialEq, Eq, Clone, Debug, Serialize, Deserialize)]
+	#[cfg_attr(feature = "arbitrary", derive(arbitrary::Arbitrary))]
 	pub struct RedirectCondition {
 		pub http_error_code: Option<u16>,
 		pub prefix: Option<String>,
 	}
 
 	#[derive(PartialEq, Eq, Clone, Debug, Serialize, Deserialize)]
+	#[cfg_attr(feature = "arbitrary", derive(arbitrary::Arbitrary))]
 	pub struct Redirect {
 		pub hostname: Option<String>,
 		pub http_redirect_code: u16,

src/model/permission.rs

@@ -6,6 +6,7 @@ use garage_util::crdt::*;
 
 /// Permission given to a key in a bucket
 #[derive(PartialOrd, Ord, PartialEq, Eq, Clone, Copy, Debug, Serialize, Deserialize)]
+#[cfg_attr(feature = "arbitrary", derive(arbitrary::Arbitrary))]
 pub struct BucketKeyPerm {
 	/// Timestamp at which the permission was given
 	pub timestamp: u64,

src/model/s3/mpu_table.rs

@@ -48,6 +48,7 @@ mod v09 {
 	}
 
 	#[derive(PartialEq, Eq, Clone, Copy, Debug, Serialize, Deserialize)]
+	#[cfg_attr(feature = "arbitrary", derive(arbitrary::Arbitrary))]
 	pub struct MpuPartKey {
 		/// Number of the part
 		pub part_number: u64,
@@ -57,6 +58,7 @@ mod v09 {
 
 	/// The version of an uploaded part
 	#[derive(PartialEq, Eq, Clone, Debug, Serialize, Deserialize)]
+	#[cfg_attr(feature = "arbitrary", derive(arbitrary::Arbitrary))]
 	pub struct MpuPart {
 		/// Links to a Version in `VersionTable`
 		pub version: Uuid,

src/model/s3/object_table.rs

@@ -291,6 +291,7 @@ mod v010 {
 
 	/// Checksum value for x-amz-checksum-algorithm
 	#[derive(PartialEq, Eq, PartialOrd, Ord, Clone, Copy, Debug, Serialize, Deserialize)]
+	#[cfg_attr(feature = "arbitrary", derive(arbitrary::Arbitrary))]
 	pub enum ChecksumValue {
 		Crc32(#[serde(with = "serde_bytes")] [u8; 4]),
 		Crc32c(#[serde(with = "serde_bytes")] [u8; 4]),

src/model/s3/version_table.rs

@@ -41,6 +41,7 @@ mod v08 {
 	}
 
 	#[derive(PartialEq, Eq, Clone, Copy, Debug, Serialize, Deserialize)]
+	#[cfg_attr(feature = "arbitrary", derive(arbitrary::Arbitrary))]
 	pub struct VersionBlockKey {
 		/// Number of the part
 		pub part_number: u64,
@@ -51,6 +52,7 @@ mod v08 {
 
 	/// Information about a single block
 	#[derive(PartialEq, Eq, Ord, PartialOrd, Clone, Copy, Debug, Serialize, Deserialize)]
+	#[cfg_attr(feature = "arbitrary", derive(arbitrary::Arbitrary))]
 	pub struct VersionBlock {
 		/// Blake2 sum of the block
 		pub hash: Hash,

src/net/peering.rs

@@ -313,10 +313,8 @@ impl PeeringManager {
 								to_ping.push(*id);
 							}
 						}
-						PeerConnState::Waiting(_, t) => {
-							if Instant::now() >= t {
-								to_retry.push(*id);
-							}
+						PeerConnState::Waiting(_, t) if Instant::now() >= t => {
+							to_retry.push(*id);
 						}
 						_ => (),
 					}

src/net/test.rs

@@ -13,12 +13,14 @@ use crate::peering::*;
 use crate::NodeID;
 
 #[tokio::test(flavor = "current_thread")]
+#[ignore = "flaky"]
 async fn test_with_basic_scheduler() {
 	pretty_env_logger::init();
 	run_test(19980).await;
 }
 
 #[tokio::test(flavor = "multi_thread", worker_threads = 4)]
+#[ignore = "flaky"]
 async fn test_with_threaded_scheduler() {
 	run_test(19990).await;
 }

src/rpc/Cargo.toml

@@ -33,6 +33,7 @@ async-trait.workspace = true
 serde.workspace = true
 serde_bytes.workspace = true
 serde_json.workspace = true
+utoipa.workspace = true
 thiserror = { workspace = true, optional = true }
 
 # newer version requires rust edition 2021

src/rpc/layout/history.rs

@@ -6,6 +6,7 @@ use garage_util::encode::nonversioned_encode;
 use garage_util::error::*;
 
 use super::*;
+use crate::layout::ComputationStat;
 use crate::replication_mode::*;
 
 impl LayoutHistory {
@@ -269,13 +270,13 @@ impl LayoutHistory {
 		changed
 	}
 
-	pub fn apply_staged_changes(mut self, version: u64) -> Result<(Self, Message), Error> {
+	pub fn apply_staged_changes(mut self, version: u64) -> Result<(Self, ComputationStat), Error> {
 		if version != self.current().version + 1 {
 			return Err(Error::Message("Invalid new layout version".into()));
 		}
 
 		// Compute new version and add it to history
-		let (new_version, msg) = self
+		let (new_version, stat) = self
 			.current()
 			.clone()
 			.calculate_next_version(self.staging.get())?;
@@ -289,7 +290,7 @@ impl LayoutHistory {
 			roles: LwwMap::new(),
 		});
 
-		Ok((self, msg))
+		Ok((self, stat))
 	}
 
 	pub fn revert_staged_changes(mut self) -> Result<Self, Error> {

src/rpc/layout/mod.rs

@@ -389,10 +389,6 @@ impl NodeRole {
 			None => "gateway".to_string(),
 		}
 	}
-
-	pub fn tags_string(&self) -> String {
-		self.tags.join(",")
-	}
 }
 
 impl UpdateTracker {

src/rpc/layout/test.rs

@@ -79,8 +79,8 @@ fn check_against_naive(cl: &LayoutVersion) -> bool {
 	false
 }
 
-fn show_msg(msg: &Message) {
-	for s in msg.iter() {
+fn show_stat(stat: &ComputationStat) {
+	for s in stat.to_message().iter() {
 		println!("{}", s);
 	}
 }
@@ -123,8 +123,8 @@ fn test_assignment() {
 	let mut cl = LayoutHistory::new(ReplicationFactor::new(3).unwrap());
 	update_layout(&mut cl, &node_capacity_vec, &node_zone_vec, 3);
 	let v = cl.current().version;
-	let (mut cl, msg) = cl.apply_staged_changes(v + 1).unwrap();
-	show_msg(&msg);
+	let (mut cl, stat) = cl.apply_staged_changes(v + 1).unwrap();
+	show_stat(&stat);
 	assert_eq!(cl.check(), Ok(()));
 	assert!(check_against_naive(cl.current()));
 
@@ -132,16 +132,16 @@ fn test_assignment() {
 	node_zone_vec = vec!["A", "B", "C", "C", "C", "B", "G", "H", "I"];
 	update_layout(&mut cl, &node_capacity_vec, &node_zone_vec, 2);
 	let v = cl.current().version;
-	let (mut cl, msg) = cl.apply_staged_changes(v + 1).unwrap();
-	show_msg(&msg);
+	let (mut cl, stat) = cl.apply_staged_changes(v + 1).unwrap();
+	show_stat(&stat);
 	assert_eq!(cl.check(), Ok(()));
 	assert!(check_against_naive(cl.current()));
 
 	node_capacity_vec = vec![4000, 1000, 2000, 7000, 1000, 1000, 2000, 10000, 2000];
 	update_layout(&mut cl, &node_capacity_vec, &node_zone_vec, 3);
 	let v = cl.current().version;
-	let (mut cl, msg) = cl.apply_staged_changes(v + 1).unwrap();
-	show_msg(&msg);
+	let (mut cl, stat) = cl.apply_staged_changes(v + 1).unwrap();
+	show_stat(&stat);
 	assert_eq!(cl.check(), Ok(()));
 	assert!(check_against_naive(cl.current()));
 
@@ -150,8 +150,8 @@ fn test_assignment() {
 	];
 	update_layout(&mut cl, &node_capacity_vec, &node_zone_vec, 1);
 	let v = cl.current().version;
-	let (cl, msg) = cl.apply_staged_changes(v + 1).unwrap();
-	show_msg(&msg);
+	let (cl, stat) = cl.apply_staged_changes(v + 1).unwrap();
+	show_stat(&stat);
 	assert_eq!(cl.check(), Ok(()));
 	assert!(check_against_naive(cl.current()));
 }

src/rpc/layout/version.rs

@@ -4,6 +4,8 @@ use std::convert::TryInto;
 
 use bytesize::ByteSize;
 use itertools::Itertools;
+use serde::{Deserialize, Serialize};
+use utoipa::ToSchema;
 
 use garage_util::crdt::{Crdt, LwwMap};
 use garage_util::data::*;
@@ -13,9 +15,6 @@ use super::graph_algo::*;
 use super::*;
 use crate::replication_mode::*;
 
-// The Message type will be used to collect information on the algorithm.
-pub type Message = Vec<String>;
-
 impl LayoutVersion {
 	pub fn new(replication_factor: ReplicationFactor) -> Self {
 		// We set the default zone redundancy to be Maximum, meaning that the maximum
@@ -291,16 +290,16 @@ impl LayoutVersion {
 	pub(crate) fn calculate_next_version(
 		mut self,
 		staging: &LayoutStaging,
-	) -> Result<(Self, Message), Error> {
+	) -> Result<(Self, ComputationStat), Error> {
 		self.version += 1;
 
 		self.roles.merge(&staging.roles);
 		self.roles.retain(|(_, _, v)| v.0.is_some());
 		self.parameters = *staging.parameters.get();
 
-		let msg = self.calculate_partition_assignment()?;
+		let stat = self.calculate_partition_assignment()?;
 
-		Ok((self, msg))
+		Ok((self, stat))
 	}
 
 	/// This function calculates a new partition-to-node assignment.
@@ -312,21 +311,15 @@ impl LayoutVersion {
 	/// data to be moved.
 	/// Staged role changes must be merged with nodes roles before calling this function,
 	/// hence it must only be called from `apply_staged_changes()` and hence is not public.
-	fn calculate_partition_assignment(&mut self) -> Result<Message, Error> {
+	fn calculate_partition_assignment(&mut self) -> Result<ComputationStat, Error> {
 		// We update the node ids, since the node role list might have changed with the
 		// changes in the layout. We retrieve the old_assignment reframed with new ids
 		let old_assignment_opt = self.update_node_id_vec()?;
 
 		let zone_redundancy = self.effective_zone_redundancy();
 
-		let mut msg = Message::new();
-		msg.push("==== COMPUTATION OF A NEW PARTITION ASSIGNATION ====".into());
-		msg.push("".into());
-		msg.push(format!(
-			"Partitions are \
-        replicated {} times on at least {} distinct zones.",
-			self.replication_factor, zone_redundancy
-		));
+		let stat_replication_factor = self.replication_factor;
+		let stat_effective_zone_redundancy = zone_redundancy;
 
 		// We generate for once numerical ids for the zones of non gateway nodes,
 		// to use them as indices in the flow graphs.
@@ -355,28 +348,17 @@ impl LayoutVersion {
 		// optimality.
 		let partition_size = self.compute_optimal_partition_size(&zone_to_id, zone_redundancy)?;
 
-		msg.push("".into());
-		if old_assignment_opt.is_some() {
-			msg.push(format!(
-				"Optimal partition size:                     {} ({} in previous layout)",
-				ByteSize::b(partition_size).display().iec(),
-				ByteSize::b(self.partition_size).display().iec()
-			));
+		let stat_partition_size = partition_size;
+		let stat_previous_partition_size = if old_assignment_opt.is_some() {
+			Some(self.partition_size)
 		} else {
-			msg.push(format!(
-				"Optimal partition size:                     {}",
-				ByteSize::b(partition_size).display().iec()
-			));
-		}
+			None
+		};
+
 		// We write the partition size.
 		self.partition_size = partition_size;
 
-		if partition_size < 100 {
-			msg.push(
-				"WARNING: The partition size is low (< 100), make sure the capacities of your nodes are correct and are of at least a few MB"
-					.into(),
-			);
-		}
+		let stat_low_partition_size = partition_size < 100;
 
 		// We compute a first flow/assignment that is heuristically close to the previous
 		// assignment
@@ -388,18 +370,28 @@ impl LayoutVersion {
 		}
 
 		// We display statistics of the computation
-		msg.extend(self.output_stat(&gflow, &old_assignment_opt, &zone_to_id, &id_to_zone)?);
+		let stat = self.output_stat(
+			&gflow,
+			&old_assignment_opt,
+			&zone_to_id,
+			&id_to_zone,
+			stat_replication_factor,
+			stat_effective_zone_redundancy,
+			stat_partition_size,
+			stat_previous_partition_size,
+			stat_low_partition_size,
+		)?;
 
 		// We update the layout structure
 		self.update_ring_from_flow(id_to_zone.len(), &gflow)?;
 
 		if let Err(e) = self.check() {
 			return Err(Error::Message(
-				format!("Layout check returned an error: {}\nOriginal result of computation: <<<<\n{}\n>>>>", e, msg.join("\n"))
+				format!("Layout check returned an error: {}\nOriginal result of computation: <<<<\n{}\n>>>>", e, stat.to_message().join("\n"))
 			));
 		}
 
-		Ok(msg)
+		Ok(stat)
 	}
 
 	/// The `LwwMap` of node roles might have changed. This function updates the `node_id_vec`
@@ -706,47 +698,26 @@ impl LayoutVersion {
 
 	/// This function returns a message summing up the partition repartition of the new
 	/// layout, and other statistics of the partition assignment computation.
+	#[allow(clippy::too_many_arguments)]
 	fn output_stat(
 		&self,
 		gflow: &Graph<FlowEdge>,
 		prev_assign_opt: &Option<Vec<Vec<usize>>>,
 		zone_to_id: &HashMap<String, usize>,
 		id_to_zone: &[String],
-	) -> Result<Message, Error> {
-		let mut msg = Message::new();
+		replication_factor: usize,
+		effective_zone_redundancy: usize,
+		partition_size: u64,
+		previous_partition_size: Option<u64>,
+		low_partition_size: bool,
+	) -> Result<ComputationStat, Error> {
+		let usable_capacity =
+			self.partition_size * NB_PARTITIONS as u64 * self.replication_factor as u64;
+		let total_capacity = self.get_total_capacity();
+		let effective_capacity = usable_capacity / replication_factor as u64;
 
-		let used_cap = self.partition_size * NB_PARTITIONS as u64 * self.replication_factor as u64;
-		let total_cap = self.get_total_capacity();
-		let percent_cap = 100.0 * (used_cap as f32) / (total_cap as f32);
-		msg.push(format!(
-			"Usable capacity / total cluster capacity:   {} / {} ({:.1} %)",
-			ByteSize::b(used_cap).display().iec(),
-			ByteSize::b(total_cap).display().iec(),
-			percent_cap
-		));
-		msg.push(format!(
-			"Effective capacity (replication factor {}):  {}",
-			self.replication_factor,
-			ByteSize::b(used_cap / self.replication_factor as u64)
-				.display()
-				.iec()
-		));
-		if percent_cap < 80. {
-			msg.push("".into());
-			msg.push(
-				"If the percentage is too low, it might be that the \
-            cluster topology and redundancy constraints are forcing the use of nodes/zones with small \
-            storage capacities."
-					.into(),
-			);
-			msg.push(
-				"You might want to move storage capacity between zones or relax the redundancy constraint."
-					.into(),
-			);
-			msg.push(
-				"See the detailed statistics below and look for saturated nodes/zones.".into(),
-			);
-		}
+		let percent_cap = 100.0 * (usable_capacity as f32) / (total_capacity as f32);
+		let low_usable_capacity = percent_cap < 80.;
 
 		// We define and fill in the following tables
 		let storing_nodes = self.nongateway_nodes();
@@ -795,18 +766,13 @@ impl LayoutVersion {
 
 		// We display the statistics
 
-		msg.push("".into());
-		if prev_assign_opt.is_some() {
-			let total_new_partitions: usize = new_partitions.iter().sum();
-			msg.push(format!(
-				"A total of {} new copies of partitions need to be \
-                             transferred.",
-				total_new_partitions
-			));
-			msg.push("".into());
-		}
+		let total_moved_partitions = if prev_assign_opt.is_some() {
+			Some(new_partitions.iter().sum())
+		} else {
+			None
+		};
 
-		let mut table = vec![];
+		let mut zones = vec![];
 		for z in 0..id_to_zone.len() {
 			let mut nodes_of_z = Vec::<usize>::new();
 			for n in 0..storing_nodes.len() {
@@ -814,49 +780,224 @@ impl LayoutVersion {
 					nodes_of_z.push(n);
 				}
 			}
-			let replicated_partitions: usize =
+			let replicated_partitions_z: usize =
 				nodes_of_z.iter().map(|n| stored_partitions[*n]).sum();
-			table.push(format!(
-				"{}\tTags\tPartitions\tCapacity\tUsable capacity",
-				id_to_zone[z]
-			));
 
-			let available_cap_z: u64 = self.partition_size * replicated_partitions as u64;
+			let available_cap_z: u64 = self.partition_size * replicated_partitions_z as u64;
 			let mut total_cap_z = 0;
 			for n in nodes_of_z.iter() {
 				total_cap_z += self.expect_get_node_capacity(&self.node_id_vec[*n]);
 			}
-			let percent_cap_z = 100.0 * (available_cap_z as f32) / (total_cap_z as f32);
 
+			let mut nodes = vec![];
 			for n in nodes_of_z.iter() {
 				let available_cap_n = stored_partitions[*n] as u64 * self.partition_size;
 				let total_cap_n = self.expect_get_node_capacity(&self.node_id_vec[*n]);
-				let tags_n = (self.node_role(&self.node_id_vec[*n]).ok_or("<??>"))?.tags_string();
+				let tags_n = (self.node_role(&self.node_id_vec[*n]).ok_or("<??>"))?
+					.tags
+					.clone();
+				nodes.push(ComputationStatNode {
+					id: hex::encode(self.node_id_vec[*n]),
+					tags: tags_n,
+					stored_partitions: stored_partitions[*n],
+					new_partitions: new_partitions[*n],
+					total_capacity: total_cap_n,
+					usable_capacity: available_cap_n,
+				});
+			}
+
+			zones.push(ComputationStatZone {
+				name: id_to_zone[z].to_string(),
+				nodes,
+				total_replicated_partitions: replicated_partitions_z,
+				unique_partitions: stored_partitions_zone[z],
+				total_capacity: total_cap_z,
+				usable_capacity: available_cap_z,
+			});
+		}
+
+		Ok(ComputationStat {
+			replication_factor,
+			effective_zone_redundancy,
+			partition_size,
+			previous_partition_size,
+			low_partition_size,
+			usable_capacity,
+			total_capacity,
+			effective_capacity,
+			low_usable_capacity,
+			total_moved_partitions,
+			zones,
+		})
+	}
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize, ToSchema)]
+#[serde(rename_all = "camelCase")]
+pub struct ComputationStat {
+	/// The cluster's replication factor
+	pub replication_factor: usize,
+	/// The zone redundancy factor achieved by this layout
+	pub effective_zone_redundancy: usize,
+	/// The size of a partition, in bytes
+	pub partition_size: u64,
+	/// The size of a partition, in bytes, in the previous layout
+	pub previous_partition_size: Option<u64>,
+	/// Warning flag indicating when partitions are very small
+	pub low_partition_size: bool,
+	/// The portion of total raw node capacity that is used by partitions
+	pub usable_capacity: u64,
+	/// The total raw capacity of nodes
+	pub total_capacity: u64,
+	/// The final effective capacity of the cluster, accounting for replication
+	pub effective_capacity: u64,
+	/// Warning flag indicating that the raw node capacity could not be used
+	/// effectively
+	pub low_usable_capacity: bool,
+	/// The total number of partitions that will be moved to a new storage node
+	pub total_moved_partitions: Option<usize>,
+	/// Per-zone storage statistics
+	pub zones: Vec<ComputationStatZone>,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize, ToSchema)]
+#[serde(rename_all = "camelCase")]
+pub struct ComputationStatZone {
+	/// The name of the zone
+	pub name: String,
+	/// Per-node storage statistics for nodes in this zone
+	pub nodes: Vec<ComputationStatNode>,
+	/// The total number of partition replicas in this zone
+	pub total_replicated_partitions: usize,
+	/// The number of unique partitions that have at least one replica in this zone
+	pub unique_partitions: usize,
+	/// The total raw capacity of nodes in this zone
+	pub total_capacity: u64,
+	/// The used portion of the raw capacity of nodes in this zones
+	pub usable_capacity: u64,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize, ToSchema)]
+#[serde(rename_all = "camelCase")]
+pub struct ComputationStatNode {
+	/// The node's ID
+	pub id: String,
+	/// The node's tags as defined in the layout
+	pub tags: Vec<String>,
+	/// The number of partitions that are replicated on this node
+	pub stored_partitions: usize,
+	/// The number of partitions that are newly replicated on this node
+	pub new_partitions: usize,
+	/// The node's raw capacity
+	pub total_capacity: u64,
+	/// The portion of the node's raw capacity that is used by partitions it stores
+	pub usable_capacity: u64,
+}
+
+impl ComputationStat {
+	pub fn to_message(&self) -> Vec<String> {
+		let mut msg = Vec::new();
+		msg.push("==== COMPUTATION OF A NEW PARTITION ASSIGNATION ====".into());
+		msg.push("".into());
+		msg.push(format!(
+			"Partitions are \
+        replicated {} times on at least {} distinct zones.",
+			self.replication_factor, self.effective_zone_redundancy
+		));
+
+		msg.push("".into());
+		if let Some(prev) = self.previous_partition_size {
+			msg.push(format!(
+				"Optimal partition size:                     {} ({} in previous layout)",
+				ByteSize::b(self.partition_size).display().iec(),
+				ByteSize::b(prev).display().iec()
+			));
+		} else {
+			msg.push(format!(
+				"Optimal partition size:                     {}",
+				ByteSize::b(self.partition_size).display().iec()
+			));
+		}
+
+		if self.low_partition_size {
+			msg.push(
+				"WARNING: The partition size is low (< 100), make sure the capacities of your nodes are correct and are of at least a few MB"
+					.into(),
+			);
+		}
+
+		let percent_cap = 100.0 * (self.usable_capacity as f32) / (self.total_capacity as f32);
+		msg.push(format!(
+			"Usable capacity / total cluster capacity:   {} / {} ({:.1} %)",
+			ByteSize::b(self.usable_capacity).display().iec(),
+			ByteSize::b(self.total_capacity).display().iec(),
+			percent_cap
+		));
+		msg.push(format!(
+			"Effective capacity (replication factor {}):  {}",
+			self.replication_factor,
+			ByteSize::b(self.effective_capacity).display().iec()
+		));
+
+		if self.low_usable_capacity {
+			msg.push("".into());
+			msg.push(
+				"If the percentage is too low, it might be that the \
+            cluster topology and redundancy constraints are forcing the use of nodes/zones with small \
+            storage capacities."
+					.into(),
+			);
+			msg.push(
+				"You might want to move storage capacity between zones or relax the redundancy constraint."
+					.into(),
+			);
+			msg.push(
+				"See the detailed statistics below and look for saturated nodes/zones.".into(),
+			);
+		}
+
+		msg.push("".into());
+		if let Some(tmp) = self.total_moved_partitions {
+			msg.push(format!(
+				"A total of {} new copies of partitions need to be \
+                             transferred.",
+				tmp
+			));
+			msg.push("".into());
+		}
+
+		let mut table = vec![];
+		for z in self.zones.iter() {
+			table.push(format!(
+				"{}\tTags\tPartitions\tCapacity\tUsable capacity",
+				z.name
+			));
+
+			for n in z.nodes.iter() {
 				table.push(format!(
-					"  {:?}\t[{}]\t{} ({} new)\t{}\t{} ({:.1}%)",
-					self.node_id_vec[*n],
-					tags_n,
-					stored_partitions[*n],
-					new_partitions[*n],
-					ByteSize::b(total_cap_n).display().iec(),
-					ByteSize::b(available_cap_n).display().iec(),
-					(available_cap_n as f32) / (total_cap_n as f32) * 100.0,
+					"  {:.16}\t[{}]\t{} ({} new)\t{}\t{} ({:.1}%)",
+					n.id,
+					n.tags.join(","),
+					n.stored_partitions,
+					n.new_partitions,
+					ByteSize::b(n.total_capacity).display().iec(),
+					ByteSize::b(n.usable_capacity).display().iec(),
+					(n.usable_capacity as f32) / (n.total_capacity as f32) * 100.0,
 				));
 			}
 
 			table.push(format!(
 				"  TOTAL\t\t{} ({} unique)\t{}\t{} ({:.1}%)",
-				replicated_partitions,
-				stored_partitions_zone[z],
-				//new_partitions_zone[z],
-				ByteSize::b(total_cap_z).display().iec(),
-				ByteSize::b(available_cap_z).display().iec(),
-				percent_cap_z
+				z.total_replicated_partitions,
+				z.unique_partitions,
+				ByteSize::b(z.total_capacity).display().iec(),
+				ByteSize::b(z.usable_capacity).display().iec(),
+				(z.usable_capacity as f32) / (z.total_capacity as f32) * 100.0,
 			));
 			table.push("".into());
 		}
 		msg.push(format_table::format_table_to_string(table));
 
-		Ok(msg)
+		msg
 	}
 }

src/rpc/rpc_helper.rs

@@ -241,11 +241,7 @@ impl RpcHelper {
 		)
 		.with_context(Context::current_with_span(span))
 		.await;
-		Ok(to
-			.iter()
-			.cloned()
-			.zip(resps.into_iter())
-			.collect::<Vec<_>>())
+		Ok(to.iter().cloned().zip(resps).collect::<Vec<_>>())
 	}
 
 	pub async fn broadcast<M, N, H, S>(

src/util/Cargo.toml

@@ -17,6 +17,7 @@ path = "lib.rs"
 garage_db.workspace = true
 garage_net.workspace = true
 
+arbitrary = { optional = true, workspace = true }
 arc-swap.workspace = true
 async-trait.workspace = true
 blake2.workspace = true
@@ -52,6 +53,7 @@ mktemp.workspace = true
 
 [features]
 k2v = []
+arbitrary = ["dep:arbitrary"]
 
 [lints]
 workspace = true

src/util/crdt/deletable.rs

@@ -4,6 +4,7 @@ use crate::crdt::crdt::*;
 
 /// Deletable object (once deleted, cannot go back)
 #[derive(Clone, Copy, Debug, Serialize, Deserialize, PartialEq, Eq)]
+#[cfg_attr(feature = "arbitrary", derive(arbitrary::Arbitrary))]
 pub enum Deletable<T> {
 	Present(T),
 	Deleted,

src/util/crdt/lww.rs

@@ -38,6 +38,7 @@ use crate::crdt::crdt::*;
 /// This scheme is used by AWS S3 or Soundcloud and often without knowing
 /// in enterprise when reconciliating databases with ad-hoc scripts.
 #[derive(Clone, Debug, Serialize, Deserialize, PartialEq, Eq)]
+#[cfg_attr(feature = "arbitrary", derive(arbitrary::Arbitrary))]
 pub struct Lww<T> {
 	ts: u64,
 	v: T,

src/util/crdt/lww_map.rs

@@ -200,3 +200,19 @@ where
 		Self::new()
 	}
 }
+
+#[cfg(feature = "arbitrary")]
+impl<'a, K, V> arbitrary::Arbitrary<'a> for LwwMap<K, V>
+where
+	K: arbitrary::Arbitrary<'a> + Clone + Ord,
+	V: arbitrary::Arbitrary<'a> + Clone + Crdt,
+{
+	fn arbitrary(u: &mut arbitrary::Unstructured<'a>) -> arbitrary::Result<Self> {
+		let map: std::collections::BTreeMap<K, (u64, V)> = arbitrary::Arbitrary::arbitrary(u)?;
+		let mut result = LwwMap::new();
+		for (k, (ts, v)) in map {
+			result.merge_raw(&k, ts, &v);
+		}
+		Ok(result)
+	}
+}

src/util/crdt/map.rs

@@ -123,3 +123,15 @@ where
 		Self { vals }
 	}
 }
+
+#[cfg(feature = "arbitrary")]
+impl<'a, K, V> arbitrary::Arbitrary<'a> for Map<K, V>
+where
+	K: arbitrary::Arbitrary<'a> + Clone + Ord,
+	V: arbitrary::Arbitrary<'a> + Clone + Crdt,
+{
+	fn arbitrary(u: &mut arbitrary::Unstructured<'a>) -> arbitrary::Result<Self> {
+		let map: std::collections::BTreeMap<K, V> = arbitrary::Arbitrary::arbitrary(u)?;
+		Ok(map.into_iter().collect())
+	}
+}

src/util/data.rs

@@ -6,6 +6,7 @@ use std::fmt;
 
 /// An array of 32 bytes
 #[derive(Default, Clone, Hash, PartialEq, Eq, PartialOrd, Ord, Copy)]
+#[cfg_attr(feature = "arbitrary", derive(arbitrary::Arbitrary))]
 pub struct FixedBytes32([u8; 32]);
 
 impl From<[u8; 32]> for FixedBytes32 {

src/web/web_server.rs

@@ -405,8 +405,8 @@ impl WebServer {
 			}
 			Ok(mut resp) => {
 				// Maybe add CORS headers
-				if let Some(rule) = find_matching_cors_rule(&bucket_params, req)? {
-					add_cors_headers(&mut resp, rule)
+				if let Some((rule, origin)) = find_matching_cors_rule(&bucket_params, req)? {
+					add_cors_headers(&mut resp, rule, origin)
 						.ok_or_internal_error("Invalid bucket CORS configuration")?;
 				}
 				Ok(resp)

typos.toml

@@ -3,4 +3,4 @@ PN = "PN"
 substituters = "substituters"
 
 [files]
-extend-exclude = ["CHANGELOG.md", "**.js", "**.svg", "doc/talks/*"]
+extend-exclude = ["CHANGELOG.md", "**.js", "**.svg", "doc/talks/*", "fuzz/*"]