Allow creation of roles instead of cluster roles

Merge pull request 'helm: add unit tests, fix edge cases' (#1519 ) from helm-unit-tests into main-v2
Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/1519
2026-09-06 20:19:15 +00:00 · 2026-09-06 22:08:39 +02:00 · 2026-09-06 19:57:04 +00:00 · 2026-09-06 21:45:40 +02:00 · 2026-09-06 21:45:34 +02:00 · 2026-09-06 21:44:30 +02:00
175 changed files with 6211 additions and 1963 deletions
@@ -53,3 +53,15 @@ steps:
      - nix-build -j4 --attr flakePackages.dev
      - nix-shell --attr ci --run ./script/test-smoke.sh || (cat /tmp/garage.log; false)
    depends_on: [ build ]
+
+  - name: helm chart tests
+    image: helmunittest/helm-unittest:4.2.3-1.1.2
+    commands:
+      - helm lint --strict script/helm/garage
+      - helm lint --strict script/helm/garage -f script/helm/garage/tests/values/daemonset.yaml
+      - helm lint --strict script/helm/garage -f script/helm/garage/tests/values/ingress.yaml
+      - helm lint --strict script/helm/garage -f script/helm/garage/tests/values/existing-secret.yaml
+      - helm lint --strict script/helm/garage -f script/helm/garage/tests/values/monitoring.yaml
+      - helm lint --strict script/helm/garage -f script/helm/garage/tests/values/minimal.yaml
+      - helm lint --strict script/helm/garage -f script/helm/garage/complex-values.yaml
+      - helm unittest --strict script/helm/garage
@@ -0,0 +1,231 @@
+# Contributing to Garage
+
+## Policy on AI
+
+To ensure the quality of the codebase and documentation, the use of AI,
+including LLMs and coding agents, is strictly restricted in the following way:
+
+- AI **must not** be used to write documentation
+
+- **Do not** use AI to write bug reports, commit descriptions and pull request
+  messages
+
+- **Do not** use AI agents to make contributions to Garage, all contributions
+  must be led by a human that know what they are doing at all times
+
+- AI **may** be used for some tedious code generation tasks, limited to very
+  mechanical translations from API docs or boilerplate writing. The code
+  generated must be so simple as to make it clear that it cannot be covered by
+  copyright.
+
+You are free to make use of AI privately to explore the codebase and solve
+conceptual problems, but please restrain from copying the output from an LLM
+anywhere in your code or on the issue tracker, or from letting an agent edit
+the codebase directly.
+
+
+## Asking questions
+
+Read the documentation before asking questions.
+Do not use the issue tracker to ask questions about Garage.
+Questions asked on the issue tracker will be closed.
+
+Ask questions on the Matrix channel `#garage:deuxfleurs.fr` so that any
+community member can see your question and help you out.
+
+If you need in-depth support from the Garage developers specifically, write to
+`garagehq@deuxfleurs.fr`. Even if you do so, we do not commit to giving you an
+answer.
+
+
+## Reporting bugs
+
+When writing a bug report, use this checklist:
+
+- For bugs that can be reproduced:
+    - confirm that you are using the latest version of Garage and that the bug still exists in this version
+    - set the log level to debug using the `RUST_LOG=garage=debug` environment variable and reproduce the bug to get more verbose logs
+
+- Check whether there is already an open issue in the bug tracker. If so, your bug report is still valuable but please add it as a comment to the existing issue instead of opening a new one.
+
+- Collect as much information as possible:
+  - logs of the Garage daemon at the time the issue happened, including logs that show what was happening before the issue occurred
+  - the output of `garage status`
+  - the output of `garage stats -a`
+  - the output of `garage layout history`
+
+- Write a detailed bug report, including:
+  - a description of your cluster (number of nodes, hardware, operating system, networking, etc)
+  - a detailed description of what you did that led to the issue, including any code or command line that invoked a Garage API
+  - what you were expecting
+  - what actually happened, and how that's different from what you expected
+  - the information collected previously
+  - if possible, simple steps to help the developers reproduce the issue locally
+
+Bug reports that are imprecise or otherwise unactionable will be closed.
+
+
+## Suggesting new features
+
+Garage can be improved in many ways, but just suggesting a new feature does not mean we will implement it.
+Feature requests that may lead to an actual implementation are feature requests that:
+
+- are precise and actionable, i.e. include a precise description of the expected behavior and any necessary architectural details required for the implementation
+- are motivated by actual need from a variety of users
+
+Moreover, a certain number of features are defined as out-of-scope for Garage, including but not limited to:
+
+- extensions to the S3 API that are not present on AWS
+- features that require the implementation of a consensus algorithm
+- more generally, features that are incompatible with the architecture of Garage and its goal of staying simple
+
+Only feature requests in one of the following category may stay open in the issue tracker:
+
+- features that the Garage team wants to work on
+- features that are being actively worked on by an external contributor which is clearly identified
+- features that are easy to implement and could be an easy task for a new contributor that wants to get to know the codebase
+
+All other feature requests will be closed after a few months of inactivity, so as to keep the number of open issues to a manageable level.
+Feature requests that are clearly out of scope will be closed directly.
+
+
+## Improving the documentation
+
+An easy way to contribute to Garage which also adds a lot of value is to
+improve the documentation.  Make sure to write in clear technical English, and
+write unambiguously.  Documentation contributions are very appreciated if they
+are well-written.
+
+
+## For developers
+
+We welcome code contributions to Garage that adhere to our standards for quality:
+
+- Changes should be reviewed from a functional perspective to ensure that they work well with the existing codebase and do not introduce bugs or subtle issues.
+
+- You must have tested your contribution to make sure that it does what it says. The amount of testing required is proportional to the complexity of the change introduced.
+
+- Any new feature must be properly documented following existing practices (see below).
+
+- Unit tests should be included when relevant.
+
+- Contributions should pass basic lints for syntactic quality (`cargo fmt`, `cargo clippy`, `typos`).
+
+- Contributions should pass our CI test suite.
+
+- No user-facing breaking changes may be introduced between major releases.
+
+- No internal data model change may be introduced between major releases, to
+  ensure that Garage daemons with different minor/patch versions numbers can
+  work together in a cluster. For major releases, a proper migration path
+  should be implemented and tested thoroughly.
+
+Please follow up on your work when changes are requested, to avoid stale PRs.
+Do not take it personally if a Garage developer pushes directly to your branch
+to modify your contribution, as this might be necessary to get it merged
+faster.
+
+### Properly documenting your contribution
+
+#### Configuration options
+
+New configuration options should be documented in
+`doc/book/reference-manual/configuration.md`.  The documentation for a
+configuration option should be exhaustive. For instance, for choice options all
+choices should be listed explicitly with a precise description of their
+meaning.
+
+In terms of syntax, all configuration options should appear in three places:
+
+- in the example at the top, with an example value
+- in the index of all configuration options which is sorted by alphabetical order
+- in its dedicated subsection with full reference text
+
+#### CLI commands and command flags
+
+CLI commands are self-documented using the doc commends in the codebase.
+Make sure to write clear and precise comments for all options you are adding.
+
+#### S3 features
+
+If you implement new S3 features, make sure to update the compatibility matrix in `doc/book/reference-manual/s3-compatibility.md`.
+
+#### Admin API
+
+The admin API has an OpenAPI specification that is automatically generated
+using Utoipa, from a description of each endpoint that is given in
+`src/api/admin/openapi.rs` and a description of data structure schemas in
+`src/api/admin/api.rs`. The code in `openapi.rs` is only used to generate the
+OpenAPI specification document and not for the actual implementation in Garage,
+whereas structures defined in `api.rs` are also used for the implementation of
+API calls. Make sure to write good doc comments for all of these items so that
+the OpenAPI specification will be precise and accurate.
+
+An up-to-date version of the OpenAPI specification document should be kept in
+the repository in `doc/api/garage-admin-v2.json`. When you are making changes
+to the admin API, update this document with the following command:
+
+```
+cargo run -- admin-api-schema > doc/api/garage-admin-v2.json
+```
+
+
+## Garage team organization
+
+Alex (handle `lx`) is the lead developer and is responsible of ensuring the
+correctness of Garage and stability between version upgrades.
+
+The other maintainers are Trinity (handle `trinity-1686a`), Quentin (handle `quentin`) and Maximilien (handle `halfa`).
+
+Maximilien is responsible for coordinating effort on the Kubernetes integration / Helm chart.
+
+## Pull request merging criteria
+
+The following PRs should only be merged after review and approval from Alex:
+
+- PRs that introduce architectural changes, such as changes in the data model
+  or change in the coordination protocols between nodes
+
+- PRs that introduce changes on the format of data structures used for
+  persistent disk storage and internal cluster communication (RPC)
+
+- PRs that are suspected of introducing some kind of breakage or unexpected
+  behavior due to their complexity
+
+PRs that introduce breaking change for users but don't fall in one of the
+previous category should be discussed between maintainers to evaluate the
+impact on users when upgrading.  Alex's approval is not required to merge them
+as long as they are clearly identified as breaking in the PR title, and are
+properly merged in the branch for the next major version and not in the current
+main branch.
+
+All other PRs can be merged by any maintainer on their own, once they are
+confident that the quality standards defined in this document are respected
+before merging.
+
+## Merging strategy
+
+When merging PRs, maintainers should ensure that a Git commit is created by
+Forgejo that records the PR number, its title and its text in the commit
+message.  If a PR is fixing an issue, make sure that the issue number is
+included in the PR title as well.  This is to ensure that when releasing a new
+version of Garage, the changelog in the release notes can be properly
+constructed by reading the Git log since the last release.
+
+We also want to keep the history "almost linear" to facilitate the use of `git
+bisect` if it ever were necessary. This leaves the following two merging
+strategies:
+
+- For PRs that consist of many commits that should stay independent, the
+  "rebase and create merge commit" strategy should be used. The merge commit is
+  created automatically by Forgejo and saves the PR's number, title and text in
+  the commit message.
+
+- For PRs that consist of only one commit, or a few number of commits that can
+  be merged, the "create squash commit" strategy should be used. This way a
+  single commit will be created by Forgejo which also saves the PR's number,
+  title and text in the commit message.
+
+When cherry-picking commits from one branch to the other, a simple fast-forward
+merging strategy can be used if the commit message already references a PR
+number.
@@ -16,6 +16,7 @@ members = [
 	"src/garage",
 	"src/k2v-client",
 	"src/format-table",
+    "fuzz",
 ]

 default-members = ["src/garage"]
@@ -24,22 +25,23 @@ default-members = ["src/garage"]

 # Internal Garage crates
 format_table = { version = "0.1.1", path = "src/format-table" }
-garage_api_common = { version = "2.3.0", path = "src/api/common" }
-garage_api_admin = { version = "2.3.0", path = "src/api/admin" }
-garage_api_s3 = { version = "2.3.0", path = "src/api/s3" }
-garage_api_k2v = { version = "2.3.0", path = "src/api/k2v" }
-garage_block = { version = "2.3.0", path = "src/block" }
-garage_db = { version = "2.3.0", path = "src/db", default-features = false }
-garage_model = { version = "2.3.0", path = "src/model", default-features = false }
-garage_net = { version = "2.3.0", path = "src/net" }
-garage_rpc = { version = "2.3.0", path = "src/rpc" }
-garage_table = { version = "2.3.0", path = "src/table" }
-garage_util = { version = "2.3.0", path = "src/util" }
-garage_web = { version = "2.3.0", path = "src/web" }
+garage_api_common = { version = "2.4.0", path = "src/api/common" }
+garage_api_admin = { version = "2.4.0", path = "src/api/admin" }
+garage_api_s3 = { version = "2.4.0", path = "src/api/s3" }
+garage_api_k2v = { version = "2.4.0", path = "src/api/k2v" }
+garage_block = { version = "2.4.0", path = "src/block" }
+garage_db = { version = "2.4.0", path = "src/db", default-features = false }
+garage_model = { version = "2.4.0", path = "src/model", default-features = false }
+garage_net = { version = "2.4.0", path = "src/net" }
+garage_rpc = { version = "2.4.0", path = "src/rpc" }
+garage_table = { version = "2.4.0", path = "src/table" }
+garage_util = { version = "2.4.0", path = "src/util" }
+garage_web = { version = "2.4.0", path = "src/web" }
 k2v-client = { version = "0.0.4", path = "src/k2v-client" }

 # External crates from crates.io
 arc-swap = "1.8"
+arbitrary = { version = "1.4.2"}
 argon2 = "0.5"
 async-trait = "0.1"
 backtrace = "0.3"
@@ -51,20 +53,23 @@ cfg-if = "1.0"
 chrono = { version = "0.4", features = ["serde"] }
 crc-fast = "1.9"
 crypto-common = "0.1"
+fundu = "2.0"
+fundu-systemd = "0.3"
 gethostname = "1.1"
 git-version = "0.3"
 hex = "0.4"
 hexdump = "0.1"
+html-escape = "0.2.13"
 hmac = "0.12"
 itertools = "0.14"
 ipnet = "2.11"
 lazy_static = "1.5"
+libfuzzer-sys = "0.4"
 md-5 = "0.10"
 mktemp = "0.5"
 nix = { version = "0.31", default-features = false, features = ["fs"] }
 nom = "8.0"
 parking_lot = "0.12"
-parse_duration = "2.1"
 paste = "1.0"
 pin-project = "1.1"
 pnet_datalink = "0.35"
@@ -113,7 +118,7 @@ kube = { version = "3.0", default-features = false, features = [
 ] }
 schemars = "1.2"
 reqwest = { version = "0.13", default-features = false, features = [
-    "rustls",
+    "rustls-no-provider",
    "json",
 ] }

@@ -0,0 +1,45 @@
+# Governance of Gararge
+
+This documents how the Garage project operates. It reflects the state of the project as of July 2026 and is not optimal. The team is interested to improve it in the future.
+
+## Team organization
+
+* **Contributors**: anyone can contribute by proposing changes in issues and pull requests.
+
+* **Maintainers**: they are responsible for reviewing, merging pull requests, publishing releases and triaging issues.
+  The current maintainers are:
+   * Alex (handle `lx`)
+   * Trinity (handle `trinity-1686a`)
+   * Quentin (handle `quentin`)
+   * Maximilien (handle `halfa`), who is in particular responsible for coordinating effort on the Kubernetes integration / Helm chart.
+ 
+  They are added to a white-list of the branch protection rule of the repository to enable them to merge pull requests.
+  To become a maintainer, you need to be a long-term contributor and earn the personal trust of Alex.
+  There is no set process for leaving the maintainer role.
+
+* **Lead developer**: Alex (handle `lx`) is the lead developer and is responsible of ensuring the
+correctness of Garage and stability between version upgrades. He may transfer this role to someone else as he sees fit.
+
+## Communication channels
+
+The team coordinates in the following channels:
+* The issue tracker and pull requests of the official repository.
+* The `#garage:deuxfleurs.fr` matrix channel (in English), open to anyone.
+  On this channel, users may ask for support and discussions about development also happen.
+* The `#garage-dev:deuxfleurs.fr` matrix channel (in French), not advertised to contributors but de facto accessible to anyone.
+  Discussions about development and project coordination happen there.
+
+The moderators for those discussion channels are the Garage maintainers.
+
+## Decision procedures
+
+Decisions are taken by lazy consensus, with the lead developer settling discussions when a consensus cannot be reached.
+
+## Governance changes
+
+There is no set process for changing the governance of garage.
+
+## See also
+
+* [Project goals](https://garagehq.deuxfleurs.fr/documentation/design/goals/)
+* [Contributing instructions](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/main-v2/CONTRIBUTING.md)
@@ -0,0 +1,14 @@
+# Security Reporting
+
+If you wish to report responsibly a security vulnerability about Garage, we ask that you follow the following process.
+
+Please report each security vulnerabilities by filling out the following template:
+
+- PROJECT: A URL to the code repository containing the vulnerable version - be reminded that the source of truth is at https://git.deuxfleurs.fr/deuxfleurs/garage
+- PUBLIC: Please let us know if this vulnerability has been made or discussed publicly already, and if so, please let us know where.
+- DESCRIPTION: Please provide precise description of the security vulnerability you have found with as much information as you are able and willing to provide.
+
+Please send the above info, along with any other information you feel is pertinent by emailing the core team at: garagehq@deuxfleurs.fr
+
+The Garage Core Team will let you know within a few weeks whether or not your report has been accepted or rejected.
+We ask that you please keep the report confidential until we have either responded or made a public announcement.
@@ -12,7 +12,7 @@
      "name": "AGPL-3.0",
      "identifier": "AGPL-3.0"
    },
-    "version": "v2.3.0"
+    "version": "v2.4.0"
  },
  "servers": [
    {
@@ -1797,6 +1797,17 @@
              "type": "string"
            },
            "description": "Plain-text information about the layout computation\n(do not try to parse this)"
+          },
+          "statistics": {
+            "oneOf": [
+              {
+                "type": "null"
+              },
+              {
+                "$ref": "#/components/schemas/ComputationStat",
+                "description": "Structured statistics about the layout computation"
+              }
+            ]
          }
        }
      },
@@ -2119,6 +2130,180 @@
          "Historical"
        ]
      },
+      "ComputationStat": {
+        "type": "object",
+        "required": [
+          "replicationFactor",
+          "effectiveZoneRedundancy",
+          "partitionSize",
+          "lowPartitionSize",
+          "usableCapacity",
+          "totalCapacity",
+          "effectiveCapacity",
+          "lowUsableCapacity",
+          "zones"
+        ],
+        "properties": {
+          "effectiveCapacity": {
+            "type": "integer",
+            "format": "int64",
+            "description": "The final effective capacity of the cluster, accounting for replication",
+            "minimum": 0
+          },
+          "effectiveZoneRedundancy": {
+            "type": "integer",
+            "description": "The zone redundancy factor achieved by this layout",
+            "minimum": 0
+          },
+          "lowPartitionSize": {
+            "type": "boolean",
+            "description": "Warning flag indicating when partitions are very small"
+          },
+          "lowUsableCapacity": {
+            "type": "boolean",
+            "description": "Warning flag indicating that the raw node capacity could not be used\neffectively"
+          },
+          "partitionSize": {
+            "type": "integer",
+            "format": "int64",
+            "description": "The size of a partition, in bytes",
+            "minimum": 0
+          },
+          "previousPartitionSize": {
+            "type": [
+              "integer",
+              "null"
+            ],
+            "format": "int64",
+            "description": "The size of a partition, in bytes, in the previous layout",
+            "minimum": 0
+          },
+          "replicationFactor": {
+            "type": "integer",
+            "description": "The cluster's replication factor",
+            "minimum": 0
+          },
+          "totalCapacity": {
+            "type": "integer",
+            "format": "int64",
+            "description": "The total raw capacity of nodes",
+            "minimum": 0
+          },
+          "totalMovedPartitions": {
+            "type": [
+              "integer",
+              "null"
+            ],
+            "description": "The total number of partitions that will be moved to a new storage node",
+            "minimum": 0
+          },
+          "usableCapacity": {
+            "type": "integer",
+            "format": "int64",
+            "description": "The portion of total raw node capacity that is used by partitions",
+            "minimum": 0
+          },
+          "zones": {
+            "type": "array",
+            "items": {
+              "$ref": "#/components/schemas/ComputationStatZone"
+            },
+            "description": "Per-zone storage statistics"
+          }
+        }
+      },
+      "ComputationStatNode": {
+        "type": "object",
+        "required": [
+          "id",
+          "tags",
+          "storedPartitions",
+          "newPartitions",
+          "totalCapacity",
+          "usableCapacity"
+        ],
+        "properties": {
+          "id": {
+            "type": "string",
+            "description": "The node's ID"
+          },
+          "newPartitions": {
+            "type": "integer",
+            "description": "The number of partitions that are newly replicated on this node",
+            "minimum": 0
+          },
+          "storedPartitions": {
+            "type": "integer",
+            "description": "The number of partitions that are replicated on this node",
+            "minimum": 0
+          },
+          "tags": {
+            "type": "array",
+            "items": {
+              "type": "string"
+            },
+            "description": "The node's tags as defined in the layout"
+          },
+          "totalCapacity": {
+            "type": "integer",
+            "format": "int64",
+            "description": "The node's raw capacity",
+            "minimum": 0
+          },
+          "usableCapacity": {
+            "type": "integer",
+            "format": "int64",
+            "description": "The portion of the node's raw capacity that is used by partitions it stores",
+            "minimum": 0
+          }
+        }
+      },
+      "ComputationStatZone": {
+        "type": "object",
+        "required": [
+          "name",
+          "nodes",
+          "totalReplicatedPartitions",
+          "uniquePartitions",
+          "totalCapacity",
+          "usableCapacity"
+        ],
+        "properties": {
+          "name": {
+            "type": "string",
+            "description": "The name of the zone"
+          },
+          "nodes": {
+            "type": "array",
+            "items": {
+              "$ref": "#/components/schemas/ComputationStatNode"
+            },
+            "description": "Per-node storage statistics for nodes in this zone"
+          },
+          "totalCapacity": {
+            "type": "integer",
+            "format": "int64",
+            "description": "The total raw capacity of nodes in this zone",
+            "minimum": 0
+          },
+          "totalReplicatedPartitions": {
+            "type": "integer",
+            "description": "The total number of partition replicas in this zone",
+            "minimum": 0
+          },
+          "uniquePartitions": {
+            "type": "integer",
+            "description": "The number of unique partitions that have at least one replica in this zone",
+            "minimum": 0
+          },
+          "usableCapacity": {
+            "type": "integer",
+            "format": "int64",
+            "description": "The used portion of the raw capacity of nodes in this zones",
+            "minimum": 0
+          }
+        }
+      },
      "ConnectClusterNodesRequest": {
        "type": "array",
        "items": {
@@ -3136,10 +3321,35 @@
          "dbEngine"
        ],
        "properties": {
+          "addr": {
+            "type": [
+              "string",
+              "null"
+            ],
+            "description": "Socket address used by other nodes to connect to this node for RPC"
+          },
+          "dataPartition": {
+            "oneOf": [
+              {
+                "type": "null"
+              },
+              {
+                "$ref": "#/components/schemas/FreeSpaceResp",
+                "description": "Total and available space on the disk partition(s) containing the data\ndirectory(ies)"
+              }
+            ]
+          },
          "dbEngine": {
            "type": "string",
            "description": "database engine used for metadata"
          },
+          "draining": {
+            "type": [
+              "boolean",
+              "null"
+            ],
+            "description": "Whether this node is part of an older layout version and is draining data."
+          },
          "garageFeatures": {
            "type": [
              "array",
@@ -3161,9 +3371,38 @@
            ],
            "description": "hostname of this node"
          },
+          "isUp": {
+            "type": [
+              "boolean",
+              "null"
+            ],
+            "description": "Whether this node is connected in the cluster"
+          },
+          "metadataPartition": {
+            "oneOf": [
+              {
+                "type": "null"
+              },
+              {
+                "$ref": "#/components/schemas/FreeSpaceResp",
+                "description": "Total and available space on the disk partition containing the\nmetadata directory"
+              }
+            ]
+          },
          "nodeId": {
            "type": "string"
          },
+          "role": {
+            "oneOf": [
+              {
+                "type": "null"
+              },
+              {
+                "$ref": "#/components/schemas/NodeAssignedRole",
+                "description": "Role assigned to this node in the current cluster layout"
+              }
+            ]
+          },
          "rustVersion": {
            "type": "string",
            "description": "rustc version with which this garage release was compiled"
@@ -3499,10 +3738,35 @@
                "dbEngine"
              ],
              "properties": {
+                "addr": {
+                  "type": [
+                    "string",
+                    "null"
+                  ],
+                  "description": "Socket address used by other nodes to connect to this node for RPC"
+                },
+                "dataPartition": {
+                  "oneOf": [
+                    {
+                      "type": "null"
+                    },
+                    {
+                      "$ref": "#/components/schemas/FreeSpaceResp",
+                      "description": "Total and available space on the disk partition(s) containing the data\ndirectory(ies)"
+                    }
+                  ]
+                },
                "dbEngine": {
                  "type": "string",
                  "description": "database engine used for metadata"
                },
+                "draining": {
+                  "type": [
+                    "boolean",
+                    "null"
+                  ],
+                  "description": "Whether this node is part of an older layout version and is draining data."
+                },
                "garageFeatures": {
                  "type": [
                    "array",
@@ -3524,9 +3788,38 @@
                  ],
                  "description": "hostname of this node"
                },
+                "isUp": {
+                  "type": [
+                    "boolean",
+                    "null"
+                  ],
+                  "description": "Whether this node is connected in the cluster"
+                },
+                "metadataPartition": {
+                  "oneOf": [
+                    {
+                      "type": "null"
+                    },
+                    {
+                      "$ref": "#/components/schemas/FreeSpaceResp",
+                      "description": "Total and available space on the disk partition containing the\nmetadata directory"
+                    }
+                  ]
+                },
                "nodeId": {
                  "type": "string"
                },
+                "role": {
+                  "oneOf": [
+                    {
+                      "type": "null"
+                    },
+                    {
+                      "$ref": "#/components/schemas/NodeAssignedRole",
+                      "description": "Role assigned to this node in the current cluster layout"
+                    }
+                  ]
+                },
                "rustVersion": {
                  "type": "string",
                  "description": "rustc version with which this garage release was compiled"
@@ -4221,6 +4514,17 @@
              "newLayout": {
                "$ref": "#/components/schemas/GetClusterLayoutResponse",
                "description": "Details about the new cluster layout"
+              },
+              "statistics": {
+                "oneOf": [
+                  {
+                    "type": "null"
+                  },
+                  {
+                    "$ref": "#/components/schemas/ComputationStat",
+                    "description": "Structured statistics about the layout computation"
+                  }
+                ]
              }
            }
          }
@@ -547,7 +547,7 @@ ejabberdctl module_install mod_s3_upload
 Create the required key and bucket with:

 ```bash
-garage key new --name ejabberd
+garage key create ejabberd
 garage bucket create objects.xmpp-server.fr
 garage bucket allow objects.xmpp-server.fr --read --write --key ejabberd
 garage bucket website --allow objects.xmpp-server.fr
@@ -678,7 +678,7 @@ For more information on deployment you can check the [ente documentation](https:
 This is the usual Garage setup:

 ```bash
-garage key new --name pleroma-key
+garage key create pleroma-key
 garage bucket create pleroma
 garage bucket allow pleroma --read --write --owner --key pleroma-key
 ```
@@ -759,7 +759,7 @@ This feature requires `pict-rs >= 4.0.0`.
 This is the usual Garage setup:

 ```bash
-garage key new --name pictrs-key
+garage key create pictrs-key
 garage bucket create pictrs-data
 garage bucket allow pictrs-data --read --write --key pictrs-key
 ```
@@ -22,7 +22,7 @@ Note that `git-annex` requires to be compiled with Haskell package version
 `aws-0.24` to work with Garage.

 ```bash
-garage key new --name my-key
+garage key create my-key
 garage bucket create my-git-annex
 garage bucket allow my-git-annex --read --write --key my-key
 ```
@@ -268,7 +268,7 @@ duck --delete garage:/my-files/an-object.txt

 ## WinSCP (libs3) {#winscp}

-*You can find instructions on how to use the GUI in french [in our wiki](https://guide.deuxfleurs.fr/prise_en_main/winscp/).*
+*You can find instructions on how to use the GUI in french [in our wiki](https://guide.deuxfleurs.fr/services/winscp/).*

 How to use `winscp.com`, the CLI interface of WinSCP:

@@ -27,7 +27,7 @@ which support storing metrics in an object store:
 This can be configured with Garage with the following:

 ```bash
-garage key new --name vector-system-logs
+garage key create vector-system-logs
 garage bucket create system-logs
 garage bucket allow system-logs --read --write --key vector-system-logs
 ```
@@ -25,6 +25,13 @@ garage bucket website --allow my-website

 Now it will be **publicly** exposed on the web endpoint (by default listening on port 3902).

+> The bucket needs to have a *global alias* to be exposed as a website. If the
+> bucket was created with `garage bucket create` it will have an alias;
+> if created via the S3 API [you will have to manually add the alias
+> ](https://git.deuxfleurs.fr/Deuxfleurs/garage/issues/742) first.
+> Creating globally aliased buckets from the S3 API is [currently under
+> discussion](https://git.deuxfleurs.fr/Deuxfleurs/garage/issues/649).
+
 ## How exposed websites work

 Our website serving logic is as follow:
@@ -96,14 +96,14 @@ to store 2 TB of data in total.
 ## Get a Docker image

 Our docker image is currently named `dxflrs/garage` and is stored on the [Docker Hub](https://hub.docker.com/r/dxflrs/garage/tags?page=1&ordering=last_updated).
-We encourage you to use a fixed tag (eg. `v2.3.0`) and not the `latest` tag.
-For this example, we will use the latest published version at the time of the writing which is `v2.3.0` but it's up to you
+We encourage you to use a fixed tag (eg. `v2.4.0`) and not the `latest` tag.
+For this example, we will use the latest published version at the time of the writing which is `v2.4.0` but it's up to you
 to check [the most recent versions on the Docker Hub](https://hub.docker.com/r/dxflrs/garage/tags?page=1&ordering=last_updated).

 For example:

 ```
-docker pull dxflrs/garage:v2.3.0
+docker pull dxflrs/garage:v2.4.0
 ```

 ## Deploying and configuring Garage
@@ -171,7 +171,7 @@ docker run \
  -v /etc/garage.toml:/etc/garage.toml \
  -v /var/lib/garage/meta:/var/lib/garage/meta \
  -v /var/lib/garage/data:/var/lib/garage/data \
-  dxflrs/garage:v2.3.0
+  dxflrs/garage:v2.4.0
 ```

 With this command line, Garage should be started automatically at each boot.
@@ -185,7 +185,7 @@ If you want to use `docker-compose`, you may use the following `docker-compose.y
 version: "3"
 services:
  garage:
-    image: dxflrs/garage:v2.3.0
+    image: dxflrs/garage:v2.4.0
    network_mode: "host"
    restart: unless-stopped
    volumes:
@@ -91,7 +91,7 @@ is definitely lost, then there is no other choice than to declare your S3 object
 as unrecoverable, and to delete them properly from the data store. This can be done
 using the `garage block purge` command.

-## Rebalancing data directories
+## Rebalancing data directories {#rebalance}

 In [multi-HDD setups](@/documentation/operations/multi-hdd.md), to ensure that
 data blocks are well balanced between storage locations, you may run a
@@ -68,10 +68,11 @@ To rebalance data, two strategies can be used:
  secondary directory. This might never end up rebalancing everything if there
  are data blocks that are only read and never written.

- Active rebalancing: an operator of a Garage node can explicitly launch a repair
-  procedure that rebalances the data directories, moving all blocks to their
-  primary location. Once done, all secondary locations for all hash slices are
-  removed so that they won't be checked anymore when looking for a data block.
+- Active rebalancing: an operator of a Garage node can [explicitly launch a
+  repair procedure](@/documentation/operations/durability-repairs.md#rebalance)
+  that rebalances the data directories, moving all blocks to their primary
+  location. Once done, all secondary locations for all hash slices are removed
+  so that they won't be checked anymore when looking for a data block.

 ## Read-only storage locations

@@ -111,7 +111,7 @@ your data to be persisted properly.

 ### Configuring initial access credentials

-Since `v2.3.0`, Garage can automatically create a default access key and a default storage bucket,
+Since `v2.n.0`, Garage can automatically create a default access key and a default storage bucket,
 based on values provided in environment variables.

 To use this feature, export the following environment variables:
@@ -133,12 +133,17 @@ Use the following command to launch the Garage server:
 garage server --single-node --default-bucket
 ```

-The `--single-node` flag instructs Garage to automatically configure a single-node cluster without data replication.
-The `--default-bucket` flag instructs Garage to create a default access key and a default bucket using the environment variables we defined above.
-Both flags are optional and can be omitted, in which case you will have to follow manual configuration steps described below.
+- the `--single-node` flag instructs Garage to automatically configure a
+  single-node cluster without data replication;
+- the `--default-bucket` flag instructs Garage to create a default access key
+  and a default bucket using the environment variables we defined above (it
+  implies `--default-access-key`).

-**For older versions of Garage (before v2.3.0):** automatic configuration using `--single-node` and `--default-bucket` is not available,
-you must follow the manual configuration steps.
+> You can refer to the [manual configuration
+> steps](#manual-configuration) if:
+>
+> - you decide to no use these optional flags;
+> - you are running an **older version of Garage (before v2.3.0)**.

 Alternatively, if you cannot or do not wish to run the Garage binary directly,
 you may use Docker to run Garage in a container using the following command:
@@ -152,7 +157,7 @@ docker run \
  -e GARAGE_DEFAULT_ACCESS_KEY \
  -e GARAGE_DEFAULT_SECRET_KEY \
  -e GARAGE_DEFAULT_BUCKET \
-  dxflrs/garage:v2.3.0
+  dxflrs/garage:v2.4.0
  /garage server --single-node --default-bucket
 ```

@@ -292,7 +297,7 @@ An exhaustive list is maintained in the ["Integrations" > "Browsing tools" secti



-## Manual configuration
+## Manual configuration {#manual-configuration}

 This section provides instructions that are equivalent to using the
 `--single-node` and `--default-bucket` flags for automatic configuration.  If
@@ -332,7 +337,7 @@ When you first start a cluster without automatic configuration, the output of `g
 ```
 ==== HEALTHY NODES ====
 ID                Hostname  Address         Tags  Zone  Capacity          DataAvail  Version
-563e1ac825ee3323  linuxbox  127.0.0.1:3901              NO ROLE ASSIGNED             v2.3.0
+563e1ac825ee3323  linuxbox  127.0.0.1:3901              NO ROLE ASSIGNED             v2.4.0
 ```

 Creating a cluster layout for a Garage deployment means informing Garage of the
@@ -175,6 +175,9 @@ they do not exist in the configuration file:
  Garage daemon send its logs to `journald` (using the native protocol of `systemd-journald`)
  instead of printing to stderr.

+- `NO_COLOR` (since `v2.4.0`): set this to `0` or `false` to disable
+  ANSI color codes in Garage's logs.
+
 The following environment variables can be used to override the corresponding
 values in the configuration file:

@@ -448,7 +451,7 @@ If this value is set, Garage will automatically take a snapshot of the metadata
 DB file at a regular interval and save it in the metadata directory,
 or in [`metadata_snapshots_dir`](#metadata_snapshots_dir) if it is set.
 This parameter can take any duration string that can be parsed by
-the [`parse_duration`](https://docs.rs/parse_duration/latest/parse_duration/#syntax) crate.
+the [`fundu_systemd`](https://docs.rs/fundu-systemd) crate.

 Snapshots can allow to recover from situations where the metadata DB file is
 corrupted, for instance after an unclean shutdown.  See [this
@@ -8,12 +8,11 @@ which is an alternative storage API designed to help efficiently store
 many small values in buckets (in opposition to S3 which is more designed
 to store large blobs).

-K2V is currently disabled at compile time in all builds, as the
-specification is still subject to changes. To build a Garage version with
-K2V, the Cargo feature flag `k2v` must be activated.  Special builds with
-the `k2v` feature flag enabled can be obtained from our download page under
-"Extra builds": such builds can be identified easily as their tag name ends
-with `-k2v` (example: `v0.7.2-k2v`).
+K2V is included in release builds since version 0.8.0. Precompiled builds
+of earlier versions including `k2v` can be found in our download page under
+"Extra builds": they can be easily identified as their tag name ends with
+`-k2v` (example: `v0.7.2-k2v`). Otherwise, when compiling Garage, the Cargo
+feature flag `k2v` must be activated.

 The specification of the K2V API can be found
 [here](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/commit/f8be15c37db857e177d543de7be863692628d567/doc/drafts/k2v-spec.md).
@@ -166,6 +166,25 @@ that map to zeroes.  Note that we need to filter out values from nodes that are
 no longer part of the cluster layout, as when nodes are removed they won't
 necessarily have had the time to set their counters to zero.

+### Consistency guarantees
+
+K2V provides the following consistency guarantees:
+
+**Read after Write**. After a write has been acknowledged (the request returned
+successfully), a subsequent read is guaranteed to contain the value that was
+written.
+
+**Monotonic Reads**. Two sequential reads will return values in an order that is
+consistent with the order in which they are written (e.g. by concurrent writes).
+For example, consider a scenario where a value is set initially set to 0 and a
+request writing 1 is performed. Doing two subsequent reads concurrently with the
+write is guaranteed to return either `0`, `0` or `0`,`1` or `1`,`1`, but not
+`1`,`0`.
+
+It is also possible to perform non-monotonic reads (allowing this last
+behavior), which are slightly faster than monotonic reads. This is done by
+passing a dedicated flag to read operations (see the endpoints documentation).
+
 ## Important details

 **THIS SECTION CONTAINS A FEW WARNINGS ON THE K2V API WHICH ARE IMPORTANT
@@ -210,6 +229,12 @@ Query parameters:
 |------------|---------------|----------------------------------|
 | `sort_key` | **mandatory** | The sort key of the item to read |

+Headers:
+
+| name                          | default value | meaning                                  |
+|-------------------------------|---------------|------------------------------------------|
+| `X-Garage-Non-Monotonic-Read` | `false`       | Whether to allow for non-monotonic reads |
+
 Returns the item with specified partition key and sort key. Values can be
 returned in either of two ways:

@@ -325,6 +350,12 @@ Query parameters:

 The timeout can be set to any number of seconds, with a maximum of 600 seconds (10 minutes).

+Headers:
+
+| name                          | default value | meaning                                  |
+|-------------------------------|---------------|------------------------------------------|
+| `X-Garage-Non-Monotonic-Read` | `false`       | Whether to allow for non-monotonic reads |
+

 **InsertItem: `PUT /<bucket>/<partition key>?sort_key=<sort_key>`**

@@ -521,6 +552,14 @@ HTTP/1.1 204 NO CONTENT

 Batch read of triplets in a bucket.

+Headers:
+
+| name                          | default value | meaning                                  |
+|-------------------------------|---------------|------------------------------------------|
+| `X-Garage-Non-Monotonic-Read` | `false`       | Whether to allow for non-monotonic reads |
+
+Body:
+
 The request body is a JSON list of searches, that each specify a range of
 items to get (to get single items, set `singleItem` to `true`). A search is a
 JSON struct with the following fields:
@@ -711,6 +750,14 @@ HTTP/1.1 200 OK

 Polls a range of items for changes.

+Headers:
+
+| name                          | default value | meaning                                  |
+|-------------------------------|---------------|------------------------------------------|
+| `X-Garage-Non-Monotonic-Read` | `false`       | Whether to allow for non-monotonic reads |
+
+Body:
+
 The query body is a JSON object consisting of the following fields:

 | name            | default value | meaning                                                                                |
@@ -81,17 +81,17 @@
        ]
      },
      "locked": {
-        "lastModified": 1763952169,
-        "narHash": "sha256-+PeDBD8P+NKauH+w7eO/QWCIp8Cx4mCfWnh9sJmy9CM=",
+        "lastModified": 1776914043,
+        "narHash": "sha256-qug5r56yW1qOsjSI99l3Jm15JNT9CvS2otkXNRNtrPI=",
        "owner": "oxalica",
        "repo": "rust-overlay",
-        "rev": "ab726555a9a72e6dc80649809147823a813fa95b",
+        "rev": "2d35c4358d7de3a0e606a6e8b27925d981c01cc3",
        "type": "github"
      },
      "original": {
        "owner": "oxalica",
        "repo": "rust-overlay",
-        "rev": "ab726555a9a72e6dc80649809147823a813fa95b",
+        "rev": "2d35c4358d7de3a0e606a6e8b27925d981c01cc3",
        "type": "github"
      }
    },
@@ -6,9 +6,9 @@
  inputs.nixpkgs.url =
    "github:NixOS/nixpkgs/cfe2c7d5b5d3032862254e68c37a6576b633d632";

-  # Rust overlay as of 2025-11-24
+  # Rust overlay as of 2026-04-23
  inputs.rust-overlay.url =
-  "github:oxalica/rust-overlay/ab726555a9a72e6dc80649809147823a813fa95b";
+  "github:oxalica/rust-overlay/2d35c4358d7de3a0e606a6e8b27925d981c01cc3";
  inputs.rust-overlay.inputs.nixpkgs.follows = "nixpkgs";

  # Crane as of 2025-01-24
@@ -95,6 +95,14 @@
                killall
              ];
            };
+
+            # dev shell for fuzzing
+            fuzz = pkgs.mkShell {
+              buildInputs = with pkgs; [
+                targets.toolchainNightly
+                cargo-fuzz
+              ];
+            };
          };
      });
 }
@@ -0,0 +1,4 @@
+target
+corpus
+artifacts
+coverage
@@ -0,0 +1,73 @@
+[package]
+name = "garage-fuzz"
+version = "0.0.0"
+publish = false
+edition = "2018"
+
+[package.metadata]
+cargo-fuzz = true
+
+[dependencies]
+arbitrary = { workspace = true, features = ["derive"]}
+libfuzzer-sys = { workspace = true }
+
+garage_db.workspace = true
+garage_table.workspace = true
+garage_util.workspace = true
+garage_model = { workspace = true, default-features = false, features = ["arbitrary", "k2v"] }
+
+[[bin]]
+name = "version_crdt"
+path = "fuzz_targets/version_crdt.rs"
+test = false
+doc = false
+bench = false
+
+[[bin]]
+name = "mpu_crdt"
+path = "fuzz_targets/mpu_crdt.rs"
+test = false
+doc = false
+bench = false
+
+[[bin]]
+name = "bucket_crdt"
+path = "fuzz_targets/bucket_crdt.rs"
+test = false
+doc = false
+bench = false
+
+[[bin]]
+name = "block_ref_crdt"
+path = "fuzz_targets/block_ref_crdt.rs"
+test = false
+doc = false
+bench = false
+
+[[bin]]
+name = "admin_api_token_crdt"
+path = "fuzz_targets/admin_api_token_crdt.rs"
+test = false
+doc = false
+bench = false
+
+[[bin]]
+name = "key_crdt"
+path = "fuzz_targets/key_crdt.rs"
+test = false
+doc = false
+bench = false
+
+[[bin]]
+name = "bucket_alias_crdt"
+path = "fuzz_targets/bucket_alias_crdt.rs"
+test = false
+doc = false
+bench = false
+
+[[bin]]
+name = "k2v_item_crdt"
+path = "fuzz_targets/k2v_item_crdt.rs"
+test = false
+doc = false
+bench = false
@@ -0,0 +1,11 @@
+# Fuzzing
+
+## Setup
+
+Install cargo fuzz: `cargo install cargo-fuzz`
+
+## Launch
+
+Run `cargo fuzz run <fuzz_target>` where `<fuzz_target>` is the name (without extension) of one of the `.rs` files in the `fuzz_targets` directory.
+
+If you launch the command outside of the fuzz directory, you need to force the nightly toolchain with `cargo +nightly`.
@@ -0,0 +1,38 @@
+#![no_main]
+
+use garage_fuzz::check_crdt_laws;
+use garage_model::admin_token_table::{AdminApiToken, AdminApiTokenParams, AdminApiTokenScope};
+use garage_model::permission::ExpirationTime;
+use garage_util::crdt;
+use libfuzzer_sys::fuzz_target;
+
+type Input = (
+	bool,
+	crdt::Lww<String>,
+	crdt::Lww<crdt::MergingOption<ExpirationTime>>,
+	crdt::Lww<AdminApiTokenScope>,
+);
+
+fn make(input: Input) -> AdminApiToken {
+	let (deleted, name, expiration, scope) = input;
+	let state = if deleted {
+		crdt::Deletable::Deleted
+	} else {
+		crdt::Deletable::present(AdminApiTokenParams {
+			created: 0,
+			token_hash: String::new(),
+			name,
+			expiration,
+			scope,
+		})
+	};
+	AdminApiToken {
+		prefix: String::new(),
+		state,
+	}
+}
+
+fuzz_target!(|inputs: (Input, Input, Input)| {
+	let (a, b, c) = inputs;
+	check_crdt_laws(make(a), make(b), make(c));
+});
@@ -0,0 +1,20 @@
+#![no_main]
+
+use garage_fuzz::check_crdt_laws;
+use garage_model::s3::block_ref_table::BlockRef;
+use libfuzzer_sys::fuzz_target;
+
+/// Build a BlockRef with a fixed block hash and version UUID so that CRDT state
+/// can be compared across merge results. Only the deleted flag varies.
+fn make_block_ref(deleted: bool) -> BlockRef {
+	BlockRef {
+		block: [0u8; 32].into(),
+		version: [0u8; 32].into(),
+		deleted: deleted.into(),
+	}
+}
+
+fuzz_target!(|inputs: (bool, bool, bool)| {
+	let (d1, d2, d3) = inputs;
+	check_crdt_laws(make_block_ref(d1), make_block_ref(d2), make_block_ref(d3));
+});
@@ -0,0 +1,25 @@
+#![no_main]
+
+use garage_fuzz::check_crdt_laws;
+use garage_model::bucket_alias_table::BucketAlias;
+use garage_util::data::Uuid;
+use libfuzzer_sys::fuzz_target;
+
+/// Build a BucketAlias with a fixed name so that CRDT state can be compared
+/// across merge results. The timestamp and optional bucket ID are the CRDT state.
+fn make_bucket_alias(ts: u64, bucket_id: Option<[u8; 32]>) -> BucketAlias {
+	BucketAlias::new(String::new(), ts, bucket_id.map(Uuid::from))
+}
+
+fuzz_target!(|inputs: (
+	(u64, Option<[u8; 32]>),
+	(u64, Option<[u8; 32]>),
+	(u64, Option<[u8; 32]>)
+)| {
+	let ((ts1, b1), (ts2, b2), (ts3, b3)) = inputs;
+	check_crdt_laws(
+		make_bucket_alias(ts1, b1),
+		make_bucket_alias(ts2, b2),
+		make_bucket_alias(ts3, b3),
+	);
+});
@@ -0,0 +1,22 @@
+#![no_main]
+
+use garage_fuzz::check_crdt_laws;
+use garage_model::bucket_table::{Bucket, BucketParams};
+use garage_util::crdt::{self, Deletable};
+use libfuzzer_sys::fuzz_target;
+
+fn make(state: Deletable<BucketParams>) -> Bucket {
+	Bucket {
+		id: [0u8; 32].into(),
+		state,
+	}
+}
+
+fuzz_target!(|inputs: (
+	crdt::Deletable<BucketParams>,
+	crdt::Deletable<BucketParams>,
+	crdt::Deletable<BucketParams>
+)| {
+	let (a, b, c) = inputs;
+	check_crdt_laws(make(a), make(b), make(c));
+});
@@ -0,0 +1,36 @@
+#![no_main]
+
+use std::collections::BTreeMap;
+
+use garage_fuzz::check_crdt_laws;
+use garage_model::k2v::item_table::{DvvsEntry, DvvsValue, K2VItem};
+use libfuzzer_sys::fuzz_target;
+
+// Timestamps are encoded as `(ts << 32) | shift` so that items built with different
+// shifts (0, 1, 2) have disjoint timestamp spaces that still interleave in the sorted merge.
+fn make(raw: BTreeMap<u64, (u32, BTreeMap<u32, DvvsValue>)>, shift: u32) -> K2VItem {
+	let shift = shift as u64;
+	let items = raw
+		.into_iter()
+		.map(|(node, (t_discard, values))| {
+			let entry = DvvsEntry::from_raw(
+				(t_discard as u64) << 32 | shift,
+				values
+					.into_iter()
+					.map(|(ts, v)| ((ts as u64) << 32 | shift, v))
+					.collect(),
+			);
+			(node, entry)
+		})
+		.collect();
+	K2VItem::with_raw_items(items)
+}
+
+fuzz_target!(|inputs: (
+	BTreeMap<u64, (u32, BTreeMap<u32, DvvsValue>)>,
+	BTreeMap<u64, (u32, BTreeMap<u32, DvvsValue>)>,
+	BTreeMap<u64, (u32, BTreeMap<u32, DvvsValue>)>,
+)| {
+	let (a, b, c) = inputs;
+	check_crdt_laws(make(a, 0), make(b, 1), make(c, 2));
+});
@@ -0,0 +1,43 @@
+#![no_main]
+
+use garage_fuzz::check_crdt_laws;
+use garage_model::key_table::{Key, KeyParams};
+use garage_model::permission::{BucketKeyPerm, ExpirationTime};
+use garage_util::crdt;
+use garage_util::data::Uuid;
+use libfuzzer_sys::fuzz_target;
+
+type Input = (
+	bool,
+	crdt::Lww<String>,
+	crdt::Lww<crdt::MergingOption<ExpirationTime>>,
+	crdt::Lww<bool>,
+	crdt::Map<Uuid, BucketKeyPerm>,
+	crdt::LwwMap<String, crdt::CancelingOption<Uuid>>,
+);
+
+fn make(input: Input) -> Key {
+	let (deleted, name, expiration, allow_create_bucket, authorized_buckets, local_aliases) = input;
+	let state = if deleted {
+		crdt::Deletable::Deleted
+	} else {
+		crdt::Deletable::present(KeyParams {
+			created: None,
+			secret_key: String::new(),
+			name,
+			expiration,
+			allow_create_bucket,
+			authorized_buckets,
+			local_aliases,
+		})
+	};
+	Key {
+		key_id: String::new(),
+		state,
+	}
+}
+
+fuzz_target!(|inputs: (Input, Input, Input)| {
+	let (a, b, c) = inputs;
+	check_crdt_laws(make(a), make(b), make(c));
+});
@@ -0,0 +1,37 @@
+#![no_main]
+
+use garage_fuzz::check_crdt_laws;
+use garage_model::s3::mpu_table::{MpuPart, MpuPartKey, MultipartUpload};
+use libfuzzer_sys::fuzz_target;
+
+/// Build a MultipartUpload from an arbitrary deleted flag and parts list, using a fixed
+/// upload_id/bucket_id/key so that CRDT state can be compared across merge results.
+/// `MpuPart.version` is fixed to a constant since it is identity data, not CRDT state:
+/// two replicas of the same part (same MpuPartKey) always share the same version UUID.
+/// If deleted, parts are cleared to ensure a valid initial CRDT state.
+fn make_mpu(deleted: bool, parts: Vec<(MpuPartKey, MpuPart)>) -> MultipartUpload {
+	let mut mpu = MultipartUpload::new(
+		[0u8; 32].into(),
+		0,
+		[0u8; 32].into(),
+		String::new(),
+		deleted,
+	);
+	for (key, mut part) in parts {
+		part.version = [0u8; 32].into();
+		mpu.parts.put(key, part);
+	}
+	if mpu.deleted.get() {
+		mpu.parts.clear();
+	}
+	mpu
+}
+
+fuzz_target!(|inputs: (
+	(bool, Vec<(MpuPartKey, MpuPart)>),
+	(bool, Vec<(MpuPartKey, MpuPart)>),
+	(bool, Vec<(MpuPartKey, MpuPart)>)
+)| {
+	let ((d1, p1), (d2, p2), (d3, p3)) = inputs;
+	check_crdt_laws(make_mpu(d1, p1), make_mpu(d2, p2), make_mpu(d3, p3));
+});
@@ -0,0 +1,42 @@
+#![no_main]
+
+use garage_fuzz::check_crdt_laws;
+use garage_model::s3::version_table::{Version, VersionBacklink, VersionBlock, VersionBlockKey};
+use libfuzzer_sys::fuzz_target;
+
+/// Build a Version from an arbitrary deleted flag and block list, using a fixed uuid/backlink
+/// so that CRDT state can be compared across merge results.
+/// Duplicate block keys are dropped before construction.
+/// If deleted, blocks are cleared to ensure a valid initial CRDT state.
+fn make_version(deleted: bool, mut blocks: Vec<(VersionBlockKey, VersionBlock)>) -> Version {
+	blocks.sort_by_key(|(k, _)| *k);
+	blocks.dedup_by_key(|(k, _)| *k);
+	let mut v = Version::new(
+		[0u8; 32].into(),
+		VersionBacklink::Object {
+			bucket_id: [0u8; 32].into(),
+			key: String::new(),
+		},
+		deleted,
+	);
+	for (key, block) in blocks {
+		v.blocks.put(key, block);
+	}
+	if v.deleted.get() {
+		v.blocks.clear();
+	}
+	v
+}
+
+fuzz_target!(|inputs: (
+	(bool, Vec<(VersionBlockKey, VersionBlock)>),
+	(bool, Vec<(VersionBlockKey, VersionBlock)>),
+	(bool, Vec<(VersionBlockKey, VersionBlock)>)
+)| {
+	let ((d1, b1), (d2, b2), (d3, b3)) = inputs;
+	check_crdt_laws(
+		make_version(d1, b1),
+		make_version(d2, b2),
+		make_version(d3, b3),
+	);
+});
@@ -0,0 +1,2 @@
+[toolchain]
+channel = "nightly"
@@ -0,0 +1,56 @@
+use garage_table::crdt::Crdt;
+use std::fmt::Debug;
+
+pub fn check_crdt_laws<T>(a: T, b: T, c: T)
+where
+	T: Crdt + PartialEq + Clone + Debug,
+{
+	// Idempotency: merge(a, a) == a
+	{
+		let mut a2 = a.clone();
+		a2.merge(&a);
+		assert_eq!(a2, a, "merge is not idempotent: {a2:#?} != {a:#?}");
+	}
+
+	// Commutativity: merge(a, b) == merge(b, a)
+	let ab = {
+		let mut t = a.clone();
+		t.merge(&b);
+		t
+	};
+	let ba = {
+		let mut t = b.clone();
+		t.merge(&a);
+		t
+	};
+	assert_eq!(ab, ba, "merge is not commutative: {ab:#?} != {ba:#?}");
+
+	// LX's corrolary: merge(merge(a,b),b) = merge(a,b)
+	let ab_b = {
+		let mut t = ab.clone();
+		t.merge(&b);
+		t
+	};
+	assert_eq!(ab, ab_b);
+
+	// Associativity: merge(merge(a, b), c) == merge(a, merge(b, c))
+	let ab_c = {
+		let mut t = ab;
+		t.merge(&c);
+		t
+	};
+	let bc = {
+		let mut t = b;
+		t.merge(&c);
+		t
+	};
+	let a_bc = {
+		let mut t = a;
+		t.merge(&bc);
+		t
+	};
+	assert_eq!(
+		ab_c, a_bc,
+		"merge is not associative: {ab_c:#?} != {a_bc:#?}"
+	);
+}
@@ -48,7 +48,7 @@ let

  inherit (pkgs) lib stdenv;

-  toolchainFn = (p: p.rust-bin.stable."1.91.0".default.override {
+  toolchainFn = (p: p.rust-bin.stable."1.95.0".default.override {
    targets = lib.optionals (target != null) [ rustTarget ];
    extensions = [
      "rust-src"
@@ -148,6 +148,14 @@ let
 in rec {
  toolchain = toolchainFn pkgs;

+  toolchainNightly = pkgs.rust-bin.selectLatestNightlyWith (toolchain: toolchain.default.override {
+    targets = lib.optionals (target != null) [ rustTarget ];
+    extensions = [
+      "rust-src"
+      "rustfmt"
+    ];
+  });
+
  devShell = pkgs.mkShell {
    buildInputs = [
      toolchain
@@ -21,3 +21,5 @@
 .idea/
 *.tmproj
 .vscode/
+# helm-unittest test suites
+tests/
@@ -2,8 +2,8 @@ apiVersion: v2
 name: garage
 description: S3-compatible object store for small self-hosted geo-distributed deployments
 type: application
-version: 0.9.3
-appVersion: "v2.3.0"
+version: 0.10.0
+appVersion: "v2.4.0"
 home: https://garagehq.deuxfleurs.fr/
 icon: https://garagehq.deuxfleurs.fr/images/garage-logo.svg

@@ -15,4 +15,6 @@ keywords:
 sources:
 - https://git.deuxfleurs.fr/Deuxfleurs/garage.git

-maintainers: []
+maintainers:
+- name: Garage maintainer team
+  email: garagehq@deuxfleurs.fr
@@ -1,6 +1,6 @@
 # garage

-![Version: 0.9.3](https://img.shields.io/badge/Version-0.9.3-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![AppVersion: v2.3.0](https://img.shields.io/badge/AppVersion-v2.3.0-informational?style=flat-square)
+![Version: 0.9.4](https://img.shields.io/badge/Version-0.9.4-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![AppVersion: v2.4.0](https://img.shields.io/badge/AppVersion-v2.4.0-informational?style=flat-square)

 S3-compatible object store for small self-hosted geo-distributed deployments

@@ -33,11 +33,14 @@ S3-compatible object store for small self-hosted geo-distributed deployments
 | garage.replicationFactor | string | `"3"` | Default to 3 replicas, see the replication_factor section at https://garagehq.deuxfleurs.fr/documentation/reference-manual/configuration/#replication_factor |
 | garage.consistencyMode | string | `"consistent"` | Default to read-after-write consistency, see the consistency_mode section at https://garagehq.deuxfleurs.fr/documentation/reference-manual/configuration/#consistency_mode |
 | garage.metadataAutoSnapshotInterval | string | `""` | If this value is set, Garage will automatically take a snapshot of the metadata DB file at a regular interval and save it in the metadata directory. https://garagehq.deuxfleurs.fr/documentation/reference-manual/configuration/#metadata_auto_snapshot_interval |
+| garage.admin.apiBindAddr | string | `"[::]:3903"` |  |
 | garage.rpcBindAddr | string | `"[::]:3901"` |  |
 | garage.rpcSecret | string | `""` | If not given, a random secret will be generated and stored in a Secret object |
+| garage.s3.api.bindAddr | string | `"[::]:3900"` |  |
 | garage.s3.api.region | string | `"garage"` |  |
 | garage.s3.api.rootDomain | string | `".s3.garage.tld"` |  |
 | garage.s3.web.index | string | `"index.html"` |  |
+| garage.s3.web.bindAddr | string | `"[::]:3902"` |  |
 | garage.s3.web.rootDomain | string | `".web.garage.tld"` |  |
 | image.pullPolicy | string | `"IfNotPresent"` |  |
 | image.repository | string | `"dxflrs/amd64_garage"` | default to amd64 docker image |
@@ -0,0 +1,331 @@
+# An "everything and the kitchen sink" values file for the helm chart: combines many non-default
+# settings at once, including examples for the fields that default to empty in
+# values.yaml and are therefore hard to guess the expected shape of.
+#
+# Aside the documentation value, it doubles as an integration-test fixture:
+# CI renders and lints the chart with this file (see .woodpecker/debug.yaml)
+# to catch feature interactions that per-feature fixtures wouldn't exercise together
+# (e.g. both ingresses enabled at once, monitoring + custom service account,
+# a DaemonSet-incompatible field set alongside a StatefulSet, ...).
+#
+# Try it locally with:
+#   helm template script/helm/garage -f script/helm/garage/complex-values.yaml
+#   helm lint --strict script/helm/garage -f script/helm/garage/complex-values.yaml
+
+# -- Additional labels to add to all resources created by this chart
+commonLabels:
+  app.kubernetes.io/part-of: storage
+  team: platform-infrastructure
+
+# Garage configuration. Values under this are written to garage.toml
+garage:
+  # -- sqlite for durability, lmdb for performance
+  # https://garagehq.deuxfleurs.fr/documentation/reference-manual/configuration/#db_engine
+  dbEngine: "sqlite"
+
+  # -- Here set to 10MiB
+  # An increase can result in better performance in certain scenarios
+  # https://garagehq.deuxfleurs.fr/documentation/reference-manual/configuration/#block_size
+  blockSize: "10485760"
+
+  # https://garagehq.deuxfleurs.fr/documentation/reference-manual/configuration/#replication_factor
+  replicationFactor: "5"
+
+  # https://garagehq.deuxfleurs.fr/documentation/reference-manual/configuration/#consistency_mode
+  consistencyMode: "dangerous"
+
+  # -- zstd compression level of stored blocks
+  # https://garagehq.deuxfleurs.fr/documentation/reference-manual/configuration/#compression_level
+  compressionLevel: "5"
+
+  # -- If this value is set, Garage will automatically take a snapshot of the metadata DB file and save it in the metadata directory.
+  # https://garagehq.deuxfleurs.fr/documentation/reference-manual/configuration/#metadata_auto_snapshot_interval
+  metadataAutoSnapshotInterval: "30 days"
+
+  rpcBindAddr: "[::]:3901"
+  # -- If not given, a random secret will be generated and stored in a Secret object
+  rpcSecret: ""
+  # -- If you want to provide an rpcSecret within an existing k8s secret,
+  # specify the secret name here, and store the value under the secret key `rpcSecret`
+  # ! the default secret will not be created
+  existingRpcSecret: ""
+  # -- This is not required if you use the integrated kubernetes discovery. Each
+  # entry is "<garage_node_id>@<host>:<port>", where <garage_node_id> is the node's public key
+  # (shown by `garage node id` on that node).
+  bootstrapPeers:
+    - "563e1ac825ee3323aa441e72c26d1030d6d4222c43c986812dbf7cd47d18aef@garage-0.garage-headless:3901"
+    - "86f0f26ae4afbd59aaf9cfb302af3fe0464f2f7b5b21f80f7e6f4e9989b5c1f8@garage-1.garage-headless:3901"
+  # -- Set to true if you want to use k8s discovery but install the CRDs manually outside
+  # of the helm chart, for example if you operate at namespace level without cluster resources
+  kubernetesSkipCrd: true
+  s3:
+    api:
+      bindAddr: "[::]:3900"
+      region: "garage"
+      rootDomain: ".s3.garage.tld"
+    web:
+      bindAddr: "[::]:3902"
+      rootDomain: ".web.garage.tld"
+      index: "index.html"
+  admin:
+    apiBindAddr: "[::]:3903"
+
+  # -- Additional configuration to append to garage.toml. Use a multi-line string for custom config.
+  additionalTopLevelConfig: |-
+    data_fsync = true
+
+  # -- if not empty string, allow using an existing ConfigMap for the garage.toml,
+  # if set, ignores garage.toml
+  existingConfigMap: ""
+
+  # -- String Template for the garage configuration.
+  # if set, ignores every other garage.* value above and is rendered with `tpl`,
+  # so it can reference .Values/.Release/.Chart, e.g.:
+  #   garageTomlString: |-
+  #     metadata_dir = "/mnt/meta"
+  #     data_dir = "/mnt/data"
+  #     replication_factor = {{ .Values.garage.replicationFactor }}
+  #     rpc_bind_addr = "{{ .Values.garage.rpcBindAddr }}"
+  #     rpc_secret = "__RPC_SECRET_REPLACE__"
+  #     [kubernetes_discovery]
+  #     namespace = "{{ .Release.Namespace }}"
+  #     service_name = "{{ include "garage.fullname" . }}"
+  # A rendering-verified version of this example lives in tests/configmap_test.yaml.
+  garageTomlString: ""
+
+# Data persistence
+persistence:
+  enabled: true
+  meta:
+    storageClass: "fast-ssd"
+    size: 100Mi
+    # used only for daemon sets
+    hostPath: /var/lib/garage/meta
+  data:
+    storageClass: "standard"
+    size: 100Mi
+    # used only for daemon sets
+    hostPath: /var/lib/garage/data
+
+# Deployment configuration
+deployment:
+  # -- Switchable to DaemonSet
+  kind: StatefulSet
+  # -- Number of StatefulSet replicas/garage nodes to start
+  replicaCount: 3
+  # -- If using statefulset, allow Parallel or OrderedReady (default)
+  podManagementPolicy: OrderedReady
+
+image:
+  # -- default to amd64 docker image
+  repository: dxflrs/amd64_garage
+  # -- set the image tag, please prefer using the chart version and not this
+  # to avoid compatibility issues
+  tag: ""
+  pullPolicy: IfNotPresent
+
+initImage:
+  repository: busybox
+  tag: stable
+  pullPolicy: IfNotPresent
+
+# -- set if you need credentials to pull your custom image. Each entry needs a
+# `name:` key, matching a Secret of type kubernetes.io/dockerconfigjson.
+imagePullSecrets:
+  - name: my-pull-secret
+nameOverride: ""
+fullnameOverride: ""
+
+serviceAccount:
+  # -- Specifies whether a service account should be created
+  create: true
+  # -- Annotations to add to the service account. Example below is for AWS IRSA.
+  annotations:
+    eks.amazonaws.com/role-arn: "arn:aws:iam::123456789012:role/garage-s3"
+  # -- The name of the service account to use.
+  # If not set and create is true, a name is generated using the fullname template
+  name: ""
+
+# -- additional pod annotations
+podAnnotations:
+  example.com/has-an-annotation: "true"
+
+podSecurityContext:
+  runAsUser: 1000
+  runAsGroup: 1000
+  fsGroup: 1000
+  fsGroupChangePolicy: "OnRootMismatch"
+  runAsNonRoot: true
+
+securityContext:
+  capabilities:
+    drop:
+      - ALL
+  readOnlyRootFilesystem: true
+
+service:
+  # -- You can rely on any service to expose your cluster
+  # - ClusterIP (+ Ingress)
+  # - NodePort (+ Ingress)
+  # - LoadBalancer
+  type: ClusterIP
+  # -- Annotations to add to the service. Example below is for an AWS NLB.
+  annotations:
+    service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
+  s3:
+    api:
+      port: 3900
+    web:
+      port: 3902
+    # NOTE: the admin API is excluded for now as it is not consistent across nodes
+
+ingress:
+  s3:
+    api:
+      enabled: true
+      className: "nginx"
+      annotations:
+        cert-manager.io/cluster-issuer: "letsencrypt-prod"
+      labels: {}
+      hosts:
+        # -- garage S3 API endpoint, to be used with awscli for example
+        - host: "s3.garage.tld"
+          paths:
+            - path: /
+              pathType: Prefix
+        # -- garage S3 API endpoint, DNS style bucket access
+        - host: "*.s3.garage.tld"
+          paths:
+            - path: /
+              pathType: Prefix
+      tls:
+        - secretName: garage-s3-api-tls
+          hosts:
+            - s3.garage.tld
+            - "*.s3.garage.tld"
+    web:
+      enabled: true
+      className: "nginx"
+      annotations:
+        cert-manager.io/cluster-issuer: "letsencrypt-prod"
+      labels: {}
+      hosts:
+        # --  wildcard website access with bucket name prefix
+        - host: "*.web.garage.tld"
+          paths:
+            - path: /
+              pathType: Prefix
+        # -- specific bucket access with FQDN bucket
+        - host: "mywebpage.example.com"
+          paths:
+            - path: /
+              pathType: Prefix
+      tls:
+        - secretName: garage-s3-web-tls
+          hosts:
+            - "*.web.garage.tld"
+            - mywebpage.example.com
+
+# The following are indicative for a small-size deployment, for anything serious double them.
+resources:
+  limits:
+    cpu: 200m
+    memory: 2048Mi
+  requests:
+    cpu: 100m
+    memory: 1024Mi
+
+# -- Specifies a livenessProbe
+livenessProbe:
+  httpGet:
+    path: /health
+    port: 3903 # or the port from garage.admin.apiBindAddr
+  initialDelaySeconds: 5
+  periodSeconds: 30
+# -- Specifies a readinessProbe
+readinessProbe:
+  httpGet:
+    path: /health
+    port: 3903 # or the port from garage.admin.apiBindAddr
+  initialDelaySeconds: 5
+  periodSeconds: 30
+
+# -- Example: pin pods to a dedicated storage node pool, paired with the
+# toleration below.
+nodeSelector:
+  node-role.kubernetes.io/storage: "true"
+
+tolerations:
+  - key: "dedicated"
+    operator: "Equal"
+    value: "storage"
+    effect: "NoSchedule"
+
+# -- Example: spread garage replicas across different nodes, since it is a
+# geo-distributed store that only helps availability if replicas don't share
+# a failure domain or availability zone.
+affinity:
+  podAntiAffinity:
+    preferredDuringSchedulingIgnoredDuringExecution:
+      - weight: 100
+        podAffinityTerm:
+          labelSelector:
+            matchLabels:
+              app.kubernetes.io/name: garage
+          topologyKey: kubernetes.io/hostname
+
+# -- Optional priority class name to assign to the pods.
+# See https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption/
+# This is expected to reference a PriorityClass you define yourself.
+priorityClassName: "high-priority-storage"
+
+# -- Extra container env vars. Note this is a [] of {name, value} objects (ie. a pod env stanza)
+# GARAGE_ADMIN_TOKEN_FILE below points garage at the token file mounted by
+# extraVolumes/extraVolumeMounts, see below.
+environment:
+  - name: RUST_LOG
+    value: "garage=debug"
+  - name: GARAGE_ADMIN_TOKEN_FILE
+    value: /mnt/secrets-store/admin-token
+
+# -- Extra volumes/volumeMounts. Both are []. Example here mounts the admin API
+# token from an external secrets manager via the Secrets Store CSI driver
+# (https://secrets-store-csi-driver.sigs.k8s.io/) instead of a Secret volume.
+# This allows, for example, providing the tokens without creating a Kubernetes
+# secret. garage reads the mounted file through GARAGE_ADMIN_TOKEN_FILE above.
+extraVolumes:
+  - name: secrets-store
+    csi:
+      driver: secrets-store.csi.k8s.io
+      readOnly: true
+      volumeAttributes:
+        secretProviderClass: garage-admin-token
+extraVolumeMounts:
+  - name: secrets-store
+    mountPath: /mnt/secrets-store
+    readOnly: true
+
+monitoring:
+  metrics:
+    # -- If true, a service for monitoring is created with a prometheus.io/scrape annotation
+    enabled: true
+    serviceMonitor:
+      # -- If true, a ServiceMonitor CRD is created for a prometheus operator
+      # https://github.com/coreos/prometheus-operator
+      enabled: true
+      path: /metrics
+      # -- Defaults to the namespace the chart is deployed to; this field is
+      # templated, so it can also reference .Release.Namespace itself.
+      namespace: "monitoring"
+      labels:
+        release: prometheus
+      interval: 30s
+      scheme: http
+      tlsConfig: {}
+      scrapeTimeout: 10s
+      relabelings:
+        - sourceLabels: ["__meta_kubernetes_pod_node_name"]
+          targetLabel: node
+  tracing:
+    # -- specify a sink endpoint for OpenTelemetry Traces, eg. `http://localhost:4317`
+    sink: "http://otel-collector.monitoring.svc:4317"
@@ -71,6 +71,13 @@ Create the name of the service account to use
 {{- end }}
 {{- end }}

+{{/*
+Extract the trailing port number from a bind address like [::]:3900 or 0.0.0.0:3900.
+*/}}
+{{- define "garage.portFromBindAddr" -}}
+{{- regexFind "[0-9]+$" . -}}
+{{- end }}
+
 {{/*
    Returns given number of random Hex characters.
    In practice, it generates up to 100 randAlphaNum strings
@@ -1,3 +1,4 @@
+{{- if eq .Values.garage.noClusterRole false }}
 apiVersion: rbac.authorization.k8s.io/v1
 kind: ClusterRole
 metadata:
@@ -28,3 +29,4 @@ roleRef:
  kind: ClusterRole
  name: manage-crds-{{ .Release.Namespace }}-{{ .Release.Name }}
  apiGroup: rbac.authorization.k8s.io
+{{ end }}
@@ -45,16 +45,16 @@ data:

    [s3_api]
    s3_region = "{{ .Values.garage.s3.api.region }}"
-    api_bind_addr = "[::]:3900"
+    api_bind_addr = "{{ .Values.garage.s3.api.bindAddr }}"
    root_domain = "{{ .Values.garage.s3.api.rootDomain }}"

    [s3_web]
-    bind_addr = "[::]:3902"
+    bind_addr = "{{ .Values.garage.s3.web.bindAddr }}"
    root_domain = "{{ .Values.garage.s3.web.rootDomain }}"
    index = "{{ .Values.garage.s3.web.index }}"

    [admin]
-    api_bind_addr = "[::]:3903"
+    api_bind_addr = "{{ .Values.garage.admin.apiBindAddr }}"
    {{- if .Values.monitoring.tracing.sink }}
    trace_sink = "{{ .Values.monitoring.tracing.sink }}"
    {{- end }}
@@ -62,7 +62,9 @@ spec:
          {{- end }}
    {{- end }}
 {{- end }}
+{{- if and .Values.ingress.s3.api.enabled .Values.ingress.s3.web.enabled }}
 ---
+{{ end }}
 {{- if .Values.ingress.s3.web.enabled -}}
 {{- $fullName := include "garage.fullname" . -}}
 {{- $svcPort := .Values.service.s3.web.port -}}
@@ -0,0 +1,27 @@
+{{- if eq .Values.garage.noClusterRole true }}
+apiVersion: rbac.authorization.k8s.io/v1
+kind: Role
+metadata:
+  name: manage-crds-{{ .Release.Namespace }}-{{ .Release.Name }}
+  labels:
+    {{- include "garage.labels" . | nindent 4 }}
+rules:
+- apiGroups: ["deuxfleurs.fr"]
+  resources: ["garagenodes"]
+  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: allow-crds-for-{{ .Release.Namespace }}-{{ .Release.Name }}
+  labels:
+    {{- include "garage.labels" . | nindent 4 }}
+subjects:
+- kind: ServiceAccount
+  name: {{ include "garage.serviceAccountName" . }}
+  namespace: {{ .Release.Namespace }}
+roleRef:
+  kind: Role
+  name: manage-crds-{{ .Release.Namespace }}-{{ .Release.Name }}
+  apiGroup: rbac.authorization.k8s.io
+{{ end }}
@@ -10,11 +10,11 @@ spec:
  clusterIP: None
  ports:
    - port: {{ .Values.service.s3.api.port }}
-      targetPort: 3900
+      targetPort: {{ include "garage.portFromBindAddr" .Values.garage.s3.api.bindAddr | int }}
      protocol: TCP
      name: s3-api
    - port: {{ .Values.service.s3.web.port }}
-      targetPort: 3902
+      targetPort: {{ include "garage.portFromBindAddr" .Values.garage.s3.web.bindAddr | int }}
      protocol: TCP
      name: s3-web
  selector:
@@ -12,11 +12,11 @@ spec:
  type: {{ .Values.service.type }}
  ports:
    - port: {{ .Values.service.s3.api.port }}
-      targetPort: 3900
+      targetPort: {{ include "garage.portFromBindAddr" .Values.garage.s3.api.bindAddr | int }}
      protocol: TCP
      name: s3-api
    - port: {{ .Values.service.s3.web.port }}
-      targetPort: 3902
+      targetPort: {{ include "garage.portFromBindAddr" .Values.garage.s3.web.bindAddr | int }}
      protocol: TCP
      name: s3-web
  selector:
@@ -35,8 +35,8 @@ spec:
  type: ClusterIP
  clusterIP: None
  ports:
-    - port: 3903
-      targetPort: 3903
+    - port: {{ include "garage.portFromBindAddr" .Values.garage.admin.apiBindAddr | int }}
+      targetPort: {{ include "garage.portFromBindAddr" .Values.garage.admin.apiBindAddr | int }}
      protocol: TCP
      name: metrics
  selector:
@@ -28,11 +28,11 @@ spec:
      scheme: {{ .Values.monitoring.metrics.serviceMonitor.scheme }}
    {{- with .Values.monitoring.metrics.serviceMonitor.tlsConfig }}
      tlsConfig:
-      {{- toYaml . | nindent 6 }}
+      {{- toYaml . | nindent 8 }}
    {{- end }}
    {{- with .Values.monitoring.metrics.serviceMonitor.relabelings }}
      relabelings:
-      {{- toYaml . | nindent 6 }}
+      {{- toYaml . | nindent 8 }}
    {{- end }}
  jobLabel: "{{ .Release.Name }}"
  selector:
@@ -41,4 +41,4 @@ spec:
  namespaceSelector:
    matchNames:
      - {{ .Release.Namespace }}
-{{- end }}
+{{- end }}
@@ -60,11 +60,11 @@ spec:
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
          imagePullPolicy: {{ .Values.image.pullPolicy }}
          ports:
-            - containerPort: 3900
+            - containerPort: {{ include "garage.portFromBindAddr" .Values.garage.s3.api.bindAddr | int }}
              name: s3-api
-            - containerPort: 3902
+            - containerPort: {{ include "garage.portFromBindAddr" .Values.garage.s3.web.bindAddr | int }}
              name: web-api
-            - containerPort: 3903
+            - containerPort: {{ include "garage.portFromBindAddr" .Values.garage.admin.apiBindAddr | int }}
              name: admin
          {{- with .Values.environment }}
          env:
@@ -0,0 +1,31 @@
+suite: rbac
+templates:
+  - templates/clusterrole.yaml
+tests:
+  - it: allows managing the garage CRD by default
+    asserts:
+      - hasDocuments:
+          count: 2
+      - documentIndex: 0
+        isKind:
+          of: ClusterRole
+      - documentIndex: 0
+        contains:
+          path: rules[0].resources
+          content: customresourcedefinitions
+      - documentIndex: 1
+        isKind:
+          of: ClusterRoleBinding
+      - documentIndex: 1
+        equal:
+          path: subjects[0].name
+          value: RELEASE-NAME-garage
+
+  - it: skips the CRD management rule when the CRD is installed manually
+    set:
+      garage.kubernetesSkipCrd: true
+    asserts:
+      - documentIndex: 0
+        notContains:
+          path: rules[0].resources
+          content: customresourcedefinitions
@@ -0,0 +1,82 @@
+# Integration-style suite: renders the whole chart with complex-values.yaml
+# (many non-default features combined at once) and checks that they don't
+# clobber each other, rather than testing any single feature in isolation
+# (that's what the other tests/*_test.yaml suites are for).
+suite: complex-values integration
+templates:
+  - templates/workload.yaml
+  - templates/service.yaml
+  - templates/service-headless.yaml
+  - templates/ingress.yaml
+  - templates/servicemonitor.yaml
+  - templates/serviceaccount.yaml
+  - templates/configmap.yaml
+  - templates/clusterrole.yaml
+tests:
+  - it: renders a self-consistent deployment with every optional feature enabled
+    values:
+      - ../complex-values.yaml
+    asserts:
+      - template: templates/workload.yaml
+        isKind:
+          of: StatefulSet
+      - template: templates/workload.yaml
+        equal:
+          path: metadata.labels.team
+          value: platform-infrastructure
+      - template: templates/workload.yaml
+        equal:
+          path: spec.template.spec.containers[0].env[0].name
+          value: RUST_LOG
+      - template: templates/workload.yaml
+        contains:
+          path: spec.template.spec.volumes
+          content:
+            name: secrets-store
+            csi:
+              driver: secrets-store.csi.k8s.io
+              readOnly: true
+              volumeAttributes:
+                secretProviderClass: garage-admin-token
+      - template: templates/workload.yaml
+        equal:
+          path: spec.template.spec.containers[0].env[1].name
+          value: GARAGE_ADMIN_TOKEN_FILE
+      - template: templates/workload.yaml
+        equal:
+          path: spec.volumeClaimTemplates[0].spec.storageClassName
+          value: fast-ssd
+      - template: templates/workload.yaml
+        contains:
+          path: spec.template.spec.imagePullSecrets
+          content:
+            name: my-pull-secret
+      - template: templates/service.yaml
+        hasDocuments:
+          count: 2 # main service + metrics service, since monitoring.metrics.enabled is true here
+      - template: templates/service-headless.yaml
+        hasDocuments:
+          count: 1 # StatefulSet still gets a headless service
+      - template: templates/ingress.yaml
+        hasDocuments:
+          count: 2 # both s3 api and s3 web ingresses enabled together
+      - template: templates/servicemonitor.yaml
+        hasDocuments:
+          count: 1
+      - template: templates/servicemonitor.yaml
+        equal:
+          path: spec.endpoints[0].relabelings[0].targetLabel
+          value: node
+      - template: templates/serviceaccount.yaml
+        equal:
+          path: metadata.annotations["eks.amazonaws.com/role-arn"]
+          value: "arn:aws:iam::123456789012:role/garage-s3"
+      - template: templates/configmap.yaml
+        matchRegex:
+          path: data["garage.toml"]
+          pattern: 'data_fsync = true'
+      - template: templates/clusterrole.yaml
+        documentIndex: 0
+        notContains:
+          path: rules[0].resources
+          content: customresourcedefinitions # garage.kubernetesSkipCrd is true here
@@ -0,0 +1,136 @@
+suite: configmap
+templates:
+  - templates/configmap.yaml
+tests:
+  - it: renders garage.toml with the default configuration
+    asserts:
+      - hasDocuments:
+          count: 1
+      - isKind:
+          of: ConfigMap
+      - equal:
+          path: metadata.name
+          value: RELEASE-NAME-garage-config
+      - matchRegex:
+          path: data["garage.toml"]
+          pattern: 'metadata_dir = "/mnt/meta"'
+      - matchRegex:
+          path: data["garage.toml"]
+          pattern: 'data_dir = "/mnt/data"'
+      - matchRegex:
+          path: data["garage.toml"]
+          pattern: 'db_engine = "lmdb"'
+      - matchRegex:
+          path: data["garage.toml"]
+          pattern: 'block_size = "1048576"'
+      - matchRegex:
+          path: data["garage.toml"]
+          pattern: 'replication_factor = 3'
+      - matchRegex:
+          path: data["garage.toml"]
+          pattern: 'consistency_mode = "consistent"'
+      - matchRegex:
+          path: data["garage.toml"]
+          pattern: 'compression_level = 1'
+      - matchRegex:
+          path: data["garage.toml"]
+          pattern: 'rpc_bind_addr = "\[::\]:3901"'
+      - matchRegex:
+          path: data["garage.toml"]
+          pattern: 'rpc_secret = "__RPC_SECRET_REPLACE__"'
+      - matchRegex:
+          path: data["garage.toml"]
+          pattern: '(?s)\[kubernetes_discovery\]\s*namespace = "NAMESPACE"\s*service_name = "RELEASE-NAME-garage"\s*skip_crd = false'
+      - matchRegex:
+          path: data["garage.toml"]
+          pattern: '(?s)\[s3_api\]\s*s3_region = "garage"\s*api_bind_addr = "\[::\]:3900"\s*root_domain = "\.s3\.garage\.tld"'
+      - matchRegex:
+          path: data["garage.toml"]
+          pattern: '(?s)\[s3_web\]\s*bind_addr = "\[::\]:3902"\s*root_domain = "\.web\.garage\.tld"\s*index = "index.html"'
+      - matchRegex:
+          path: data["garage.toml"]
+          pattern: '(?s)\[admin\]\s*api_bind_addr = "\[::\]:3903"'
+      - notMatchRegex:
+          path: data["garage.toml"]
+          pattern: 'metadata_auto_snapshot_interval'
+      - notMatchRegex:
+          path: data["garage.toml"]
+          pattern: 'trace_sink'
+
+  - it: reflects custom garage settings, bootstrap peers and additional config
+    set:
+      garage.dbEngine: sqlite
+      garage.blockSize: "2097152"
+      garage.replicationFactor: "5"
+      garage.consistencyMode: degraded
+      garage.compressionLevel: "3"
+      garage.metadataAutoSnapshotInterval: 6h
+      garage.bootstrapPeers:
+        - abc@peer1:3901
+        - def@peer2:3901
+      garage.additionalTopLevelConfig: "data_fsync = true"
+      monitoring.tracing.sink: http://otel:4317
+    asserts:
+      - matchRegex:
+          path: data["garage.toml"]
+          pattern: 'db_engine = "sqlite"'
+      - matchRegex:
+          path: data["garage.toml"]
+          pattern: 'block_size = "2097152"'
+      - matchRegex:
+          path: data["garage.toml"]
+          pattern: 'replication_factor = 5'
+      - matchRegex:
+          path: data["garage.toml"]
+          pattern: 'consistency_mode = "degraded"'
+      - matchRegex:
+          path: data["garage.toml"]
+          pattern: 'compression_level = 3'
+      - matchRegex:
+          path: data["garage.toml"]
+          pattern: 'metadata_auto_snapshot_interval = "6h"'
+      - matchRegex:
+          path: data["garage.toml"]
+          pattern: 'bootstrap_peers = \["abc@peer1:3901"\s*, "def@peer2:3901"'
+      - matchRegex:
+          path: data["garage.toml"]
+          pattern: 'data_fsync = true'
+      - matchRegex:
+          path: data["garage.toml"]
+          pattern: 'trace_sink = "http://otel:4317"'
+
+  - it: uses garageTomlString verbatim when set, ignoring the structured values
+    set:
+      garage.garageTomlString: |-
+        metadata_dir = "/custom/meta"
+        replication_factor = 1
+      garage.dbEngine: sqlite
+    asserts:
+      - equal:
+          path: data["garage.toml"]
+          value: |-
+            metadata_dir = "/custom/meta"
+            replication_factor = 1
+      - notMatchRegex:
+          path: data["garage.toml"]
+          pattern: 'db_engine'
+
+  - it: templates garageTomlString against the release and values context
+    set:
+      garage.garageTomlString: |-
+        # namespace: {{ .Release.Namespace }}
+        replication_factor = {{ .Values.garage.replicationFactor }}
+      garage.replicationFactor: "7"
+    asserts:
+      - equal:
+          path: data["garage.toml"]
+          value: |-
+            # namespace: NAMESPACE
+            replication_factor = 7
+
+  - it: does not render a ConfigMap when an existing one is referenced
+    set:
+      garage.existingConfigMap: my-external-cm
+    asserts:
+      - hasDocuments:
+          count: 0
@@ -0,0 +1,95 @@
+suite: ingress
+templates:
+  - templates/ingress.yaml
+tests:
+  - it: renders no ingress by default
+    asserts:
+      - hasDocuments:
+          count: 0
+
+  - it: renders api and web ingresses with tls when enabled
+    values:
+      - ./values/ingress.yaml
+    asserts:
+      - hasDocuments:
+          count: 2
+      - isKind:
+          of: Ingress
+      - documentIndex: 0
+        equal:
+          path: metadata.name
+          value: RELEASE-NAME-garage-s3-api
+      - documentIndex: 0
+        equal:
+          path: spec.ingressClassName
+          value: nginx
+      - documentIndex: 0
+        equal:
+          path: spec.rules[0].host
+          value: s3.example.com
+      - documentIndex: 0
+        equal:
+          path: spec.tls[0].secretName
+          value: garage-s3-api-tls
+      - documentIndex: 1
+        equal:
+          path: metadata.name
+          value: RELEASE-NAME-garage-s3-web
+      - documentIndex: 1
+        equal:
+          path: spec.rules[0].host
+          value: "*.web.example.com"
+      - documentIndex: 1
+        equal:
+          path: spec.tls[0].secretName
+          value: garage-s3-web-tls
+
+  - it: can enable only the s3 api ingress
+    set:
+      ingress.s3.api.enabled: true
+      ingress.s3.api.hosts[0].host: s3.example.com
+      ingress.s3.api.hosts[0].paths[0].path: /
+      ingress.s3.api.hosts[0].paths[0].pathType: Prefix
+    asserts:
+      - hasDocuments:
+          count: 1
+      - equal:
+          path: metadata.name
+          value: RELEASE-NAME-garage-s3-api
+
+  - it: omits ingressClassName and tls when neither is configured
+    set:
+      ingress.s3.api.enabled: true
+      ingress.s3.api.hosts[0].host: s3.example.com
+      ingress.s3.api.hosts[0].paths[0].path: /
+      ingress.s3.api.hosts[0].paths[0].pathType: Prefix
+    asserts:
+      - isNull:
+          path: spec.ingressClassName
+      - isNull:
+          path: spec.tls
+
+  - it: renders multiple hosts on the same ingress
+    set:
+      ingress.s3.api.enabled: true
+      ingress.s3.api.hosts:
+        - host: s3.example.com
+          paths:
+            - path: /
+              pathType: Prefix
+        - host: s3-alt.example.com
+          paths:
+            - path: /
+              pathType: Prefix
+    asserts:
+      - hasDocuments:
+          count: 1
+      - lengthEqual:
+          path: spec.rules
+          count: 2
+      - equal:
+          path: spec.rules[0].host
+          value: s3.example.com
+      - equal:
+          path: spec.rules[1].host
+          value: s3-alt.example.com
@@ -0,0 +1,56 @@
+suite: naming and common labels
+templates:
+  - templates/workload.yaml
+  - templates/configmap.yaml
+tests:
+  - it: applies commonLabels alongside the default chart labels
+    template: templates/workload.yaml
+    set:
+      commonLabels:
+        team: storage
+    asserts:
+      - equal:
+          path: metadata.labels.team
+          value: storage
+      - equal:
+          path: metadata.labels["app.kubernetes.io/managed-by"]
+          value: Helm
+
+  - it: uses fullnameOverride verbatim for resource names
+    template: templates/workload.yaml
+    set:
+      fullnameOverride: my-garage-cluster
+    asserts:
+      - equal:
+          path: metadata.name
+          value: my-garage-cluster
+      - equal:
+          path: spec.serviceName
+          value: my-garage-cluster-headless
+
+  - it: does not double-prefix when the release name already contains the chart name
+    template: templates/workload.yaml
+    release:
+      name: garage
+    asserts:
+      - equal:
+          path: metadata.name
+          value: garage
+
+  - it: prefixes the release name with the chart name otherwise
+    template: templates/workload.yaml
+    release:
+      name: prod
+    asserts:
+      - equal:
+          path: metadata.name
+          value: prod-garage
+
+  - it: truncates an overly long fullname to 63 characters and trims a trailing dash
+    template: templates/workload.yaml
+    set:
+      fullnameOverride: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa-suffix-that-will-be-cut-off
+    asserts:
+      - equal:
+          path: metadata.name
+          value: aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
@@ -0,0 +1,33 @@
+suite: rpc secret
+templates:
+  - templates/secret.yaml
+tests:
+  - it: generates a Secret holding the rpc secret by default
+    asserts:
+      - hasDocuments:
+          count: 1
+      - isKind:
+          of: Secret
+      - equal:
+          path: metadata.name
+          value: RELEASE-NAME-garage-rpc-secret
+      - equal:
+          path: type
+          value: Opaque
+      - isNotNull:
+          path: data.rpcSecret
+
+  - it: does not render a Secret when an existing one is referenced
+    values:
+      - ./values/existing-secret.yaml
+    asserts:
+      - hasDocuments:
+          count: 0
+
+  - it: base64-encodes an explicitly provided rpc secret
+    set:
+      garage.rpcSecret: my-plain-secret
+    asserts:
+      - equal:
+          path: data.rpcSecret
+          value: bXktcGxhaW4tc2VjcmV0
@@ -0,0 +1,26 @@
+suite: headless service
+templates:
+  - templates/service-headless.yaml
+tests:
+  - it: creates a headless service for a StatefulSet by default
+    asserts:
+      - hasDocuments:
+          count: 1
+      - isKind:
+          of: Service
+      - equal:
+          path: metadata.name
+          value: RELEASE-NAME-garage-headless
+      - equal:
+          path: spec.clusterIP
+          value: None
+      - equal:
+          path: spec.type
+          value: ClusterIP
+
+  - it: does not create a headless service for a DaemonSet
+    values:
+      - ./values/daemonset.yaml
+    asserts:
+      - hasDocuments:
+          count: 0
@@ -0,0 +1,61 @@
+suite: service
+templates:
+  - templates/service.yaml
+tests:
+  - it: creates a ClusterIP service with s3-api and s3-web ports by default
+    asserts:
+      - hasDocuments:
+          count: 1
+      - isKind:
+          of: Service
+      - equal:
+          path: spec.type
+          value: ClusterIP
+      - equal:
+          path: spec.ports[0].name
+          value: s3-api
+      - equal:
+          path: spec.ports[0].port
+          value: 3900
+      - equal:
+          path: spec.ports[1].name
+          value: s3-web
+      - equal:
+          path: spec.ports[1].port
+          value: 3902
+
+  - it: honors a custom service type and port
+    set:
+      service.type: LoadBalancer
+      service.s3.api.port: 9000
+    asserts:
+      - equal:
+          path: spec.type
+          value: LoadBalancer
+      - equal:
+          path: spec.ports[0].port
+          value: 9000
+
+  - it: does not create a metrics service by default
+    asserts:
+      - hasDocuments:
+          count: 1
+
+  - it: adds a headless metrics service when monitoring is enabled
+    values:
+      - ./values/monitoring.yaml
+    asserts:
+      - hasDocuments:
+          count: 2
+      - documentIndex: 1
+        equal:
+          path: metadata.name
+          value: RELEASE-NAME-garage-metrics
+      - documentIndex: 1
+        equal:
+          path: spec.clusterIP
+          value: None
+      - documentIndex: 1
+        equal:
+          path: metadata.annotations["prometheus.io/scrape"]
+          value: "true"
@@ -0,0 +1,28 @@
+suite: service account
+templates:
+  - templates/serviceaccount.yaml
+tests:
+  - it: creates a ServiceAccount by default
+    asserts:
+      - hasDocuments:
+          count: 1
+      - isKind:
+          of: ServiceAccount
+      - equal:
+          path: metadata.name
+          value: RELEASE-NAME-garage
+
+  - it: does not create a ServiceAccount when disabled
+    values:
+      - ./values/minimal.yaml
+    asserts:
+      - hasDocuments:
+          count: 0
+
+  - it: honors a custom service account name
+    set:
+      serviceAccount.name: my-garage-sa
+    asserts:
+      - equal:
+          path: metadata.name
+          value: my-garage-sa
@@ -0,0 +1,65 @@
+suite: service monitor
+templates:
+  - templates/servicemonitor.yaml
+tests:
+  - it: renders no ServiceMonitor by default
+    asserts:
+      - hasDocuments:
+          count: 0
+
+  - it: renders no ServiceMonitor when only metrics are enabled
+    set:
+      monitoring.metrics.enabled: true
+    asserts:
+      - hasDocuments:
+          count: 0
+
+  - it: renders a ServiceMonitor when explicitly enabled
+    values:
+      - ./values/monitoring.yaml
+    asserts:
+      - hasDocuments:
+          count: 1
+      - isKind:
+          of: ServiceMonitor
+      - equal:
+          path: metadata.name
+          value: RELEASE-NAME-garage
+      - equal:
+          path: metadata.namespace
+          value: NAMESPACE
+      - equal:
+          path: spec.endpoints[0].interval
+          value: 30s
+
+  - it: templates a custom namespace against the release context
+    values:
+      - ./values/monitoring.yaml
+    set:
+      monitoring.metrics.serviceMonitor.namespace: "{{ .Release.Namespace }}-monitoring"
+    asserts:
+      - equal:
+          path: metadata.namespace
+          value: NAMESPACE-monitoring
+
+  - it: applies custom labels, tlsConfig and relabelings
+    values:
+      - ./values/monitoring.yaml
+    set:
+      monitoring.metrics.serviceMonitor.labels:
+        team: storage
+      monitoring.metrics.serviceMonitor.tlsConfig:
+        insecureSkipVerify: true
+      monitoring.metrics.serviceMonitor.relabelings:
+        - sourceLabels: ["__meta_kubernetes_pod_name"]
+          targetLabel: pod
+    asserts:
+      - equal:
+          path: metadata.labels.team
+          value: storage
+      - equal:
+          path: spec.endpoints[0].tlsConfig.insecureSkipVerify
+          value: true
+      - equal:
+          path: spec.endpoints[0].relabelings[0].targetLabel
+          value: pod
@@ -0,0 +1,10 @@
+# Run garage as a DaemonSet (one pod per node) instead of the default StatefulSet,
+# using hostPath volumes for meta/data persistence.
+deployment:
+  kind: DaemonSet
+persistence:
+  enabled: true
+  meta:
+    hostPath: /var/lib/garage/meta
+  data:
+    hostPath: /var/lib/garage/data
@@ -0,0 +1,5 @@
+# Use a pre-existing Kubernetes Secret for the RPC secret instead of letting
+# the chart generate/manage one.
+garage:
+  rpcSecret: ""
+  existingRpcSecret: "garage-rpc-secret-external"
@@ -0,0 +1,27 @@
+# Expose both the S3 API and website endpoints through Ingress, with TLS.
+ingress:
+  s3:
+    api:
+      enabled: true
+      className: "nginx"
+      hosts:
+        - host: "s3.example.com"
+          paths:
+            - path: /
+              pathType: Prefix
+      tls:
+        - secretName: garage-s3-api-tls
+          hosts:
+            - s3.example.com
+    web:
+      enabled: true
+      className: "nginx"
+      hosts:
+        - host: "*.web.example.com"
+          paths:
+            - path: /
+              pathType: Prefix
+      tls:
+        - secretName: garage-s3-web-tls
+          hosts:
+            - "*.web.example.com"
@@ -0,0 +1,8 @@
+# Minimal single-node deployment without persistent storage or a dedicated
+# service account, e.g. for local testing.
+deployment:
+  replicaCount: 1
+persistence:
+  enabled: false
+serviceAccount:
+  create: false
@@ -0,0 +1,7 @@
+# Enable Prometheus metrics scraping and a ServiceMonitor for the prometheus-operator.
+monitoring:
+  metrics:
+    enabled: true
+    serviceMonitor:
+      enabled: true
+      interval: 30s
@@ -0,0 +1,187 @@
+suite: workload (StatefulSet/DaemonSet)
+templates:
+  - templates/workload.yaml
+  - templates/configmap.yaml
+tests:
+  - it: defaults to a StatefulSet with 3 replicas and 2 volumes
+    template: templates/workload.yaml
+    asserts:
+      - isKind:
+          of: StatefulSet
+      - equal:
+          path: spec.replicas
+          value: 3
+      - equal:
+          path: spec.podManagementPolicy
+          value: OrderedReady
+      - equal:
+          path: spec.template.spec.volumes[1].name
+          value: etc
+      - lengthEqual:
+          path: spec.template.spec.volumes
+          count: 2
+      - isNotNull:
+          path: spec.volumeClaimTemplates
+
+  - it: uses a StatefulSet with PVC-backed volumeClaimTemplates by default
+    template: templates/workload.yaml
+    asserts:
+      - isKind:
+          of: StatefulSet
+      - isNotNull:
+          path: spec.volumeClaimTemplates
+      - equal:
+          path: spec.volumeClaimTemplates[0].spec.resources.requests.storage
+          value: 100Mi
+
+  - it: switches to a DaemonSet with hostPath volumes when requested
+    template: templates/workload.yaml
+    values:
+      - ./values/daemonset.yaml
+    asserts:
+      - isKind:
+          of: DaemonSet
+      - isNull:
+          path: spec.replicas
+      - isNull:
+          path: spec.volumeClaimTemplates
+      - contains:
+          path: spec.template.spec.volumes
+          content:
+            name: meta
+            hostPath:
+              path: /var/lib/garage/meta
+              type: DirectoryOrCreate
+      - contains:
+          path: spec.template.spec.volumes
+          content:
+            name: data
+            hostPath:
+              path: /var/lib/garage/data
+              type: DirectoryOrCreate
+
+  - it: renders emptyDir volumes when persistence is disabled
+    template: templates/workload.yaml
+    values:
+      - ./values/minimal.yaml
+    asserts:
+      - contains:
+          path: spec.template.spec.volumes
+          content:
+            name: meta
+            emptyDir: {}
+      - contains:
+          path: spec.template.spec.volumes
+          content:
+            name: data
+            emptyDir: {}
+      - isNull:
+          path: spec.volumeClaimTemplates
+
+  - it: honors a custom replicaCount
+    template: templates/workload.yaml
+    set:
+      deployment.replicaCount: 5
+    asserts:
+      - equal:
+          path: spec.replicas
+          value: 5
+
+  - it: points the init container at the configured rpc secret
+    template: templates/workload.yaml
+    asserts:
+      - equal:
+          path: spec.template.spec.initContainers[0].env[0].valueFrom.secretKeyRef.name
+          value: RELEASE-NAME-garage-rpc-secret
+
+  - it: points the init container at an existing rpc secret when configured
+    template: templates/workload.yaml
+    values:
+      - ./values/existing-secret.yaml
+    asserts:
+      - equal:
+          path: spec.template.spec.initContainers[0].env[0].valueFrom.secretKeyRef.name
+          value: garage-rpc-secret-external
+
+  - it: sets the container image from repository and tag
+    template: templates/workload.yaml
+    set:
+      image.repository: dxflrs/amd64_garage
+      image.tag: v1.2.3
+    asserts:
+      - equal:
+          path: spec.template.spec.containers[0].image
+          value: dxflrs/amd64_garage:v1.2.3
+
+  - it: falls back to the chart appVersion when no image tag is set
+    template: templates/workload.yaml
+    asserts:
+      - matchRegex:
+          path: spec.template.spec.containers[0].image
+          pattern: ^dxflrs/amd64_garage:v
+
+  - it: omits storageClassName from volumeClaimTemplates by default
+    template: templates/workload.yaml
+    asserts:
+      - isNull:
+          path: spec.volumeClaimTemplates[0].spec.storageClassName
+      - isNull:
+          path: spec.volumeClaimTemplates[1].spec.storageClassName
+
+  - it: sets storageClassName in volumeClaimTemplates when configured
+    template: templates/workload.yaml
+    set:
+      persistence.meta.storageClass: fast-storage
+      persistence.data.storageClass: slow-storage
+    asserts:
+      - equal:
+          path: spec.volumeClaimTemplates[0].spec.storageClassName
+          value: fast-storage
+      - equal:
+          path: spec.volumeClaimTemplates[1].spec.storageClassName
+          value: slow-storage
+
+  - it: renders emptyDir volumes for a DaemonSet when persistence is disabled
+    template: templates/workload.yaml
+    set:
+      deployment.kind: DaemonSet
+      persistence.enabled: false
+    asserts:
+      - contains:
+          path: spec.template.spec.volumes
+          content:
+            name: meta
+            emptyDir: {}
+      - contains:
+          path: spec.template.spec.volumes
+          content:
+            name: data
+            emptyDir: {}
+
+  - it: mounts the existing ConfigMap volume when configured
+    template: templates/workload.yaml
+    set:
+      garage.existingConfigMap: my-external-cm
+    asserts:
+      - equal:
+          path: spec.template.spec.volumes[0].configMap.name
+          value: my-external-cm
+
+  - it: uses a custom service account name without creating one when disabled
+    template: templates/workload.yaml
+    set:
+      serviceAccount.create: false
+      serviceAccount.name: my-external-sa
+    asserts:
+      - equal:
+          path: spec.template.spec.serviceAccountName
+          value: my-external-sa
+
+  - it: falls back to the default service account when disabled without a custom name
+    template: templates/workload.yaml
+    set:
+      serviceAccount.create: false
+    asserts:
+      - equal:
+          path: spec.template.spec.serviceAccountName
+          value: default
@@ -46,13 +46,19 @@ garage:
  # -- Set to true if you want to use k8s discovery but install the CRDs manually outside
  # of the helm chart, for example if you operate at namespace level without cluster resources
  kubernetesSkipCrd: false
+  # -- Set to true if you want to use roles instead of cluster roles
+  noClusterRole: false
  s3:
    api:
+      bindAddr: "[::]:3900"
      region: "garage"
      rootDomain: ".s3.garage.tld"
    web:
+      bindAddr: "[::]:3902"
      rootDomain: ".web.garage.tld"
      index: "index.html"
+  admin:
+    apiBindAddr: "[::]:3903"

  # -- Additional configuration to append to garage.toml. Use a multi-line string for custom config.
  # Example:
@@ -221,14 +227,14 @@ resources: {}
 livenessProbe: {}
  #httpGet:
  #  path: /health
-  #  port: 3903
+  #  port: 3903 # or the port from garage.admin.apiBindAddr
  #initialDelaySeconds: 5
  #periodSeconds: 30
 # -- Specifies a readinessProbe
 readinessProbe: {}
  #httpGet:
  #  path: /health
-  #  port: 3903
+  #  port: 3903 # or the port from garage.admin.apiBindAddr
  #initialDelaySeconds: 5
  #periodSeconds: 30

@@ -242,10 +248,14 @@ affinity: {}
 # See https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption/
 priorityClassName: ""

+# -- Extra container env vars, as a list of {name, value} objects (same shape
+# as a Pod container's env)
 environment: {}

+# -- Extra volumes, as a list of volume objects (same shape as a PodSpec's volumes)
 extraVolumes: {}

+# -- Extra volume mounts, as a list of mount objects (same shape as a container's volumeMounts)
 extraVolumeMounts: {}

 monitoring:
@@ -1,6 +1,6 @@
 [package]
 name = "garage_api_admin"
-version = "2.3.0"
+version = "2.4.0"
 authors = ["Alex Auvolat <alex@adnab.me>"]
 edition = "2018"
 license = "AGPL-3.0"
@@ -7,6 +7,7 @@ use garage_util::time::now_msec;

 use garage_model::admin_token_table::*;
 use garage_model::garage::Garage;
+use garage_model::permission::ExpirationTime;

 use crate::api::*;
 use crate::error::*;
@@ -244,8 +245,8 @@ fn admin_token_info_results(token: &AdminApiToken, now: u64) -> GetAdminTokenInf
 				.expect("invalid timestamp stored in db"),
 		),
 		name: params.name.get().to_string(),
-		expiration: params.expiration.get().map(|x| {
-			DateTime::from_timestamp_millis(x as i64).expect("invalid timestamp stored in db")
+		expiration: params.expiration.get().inner().map(|x| {
+			DateTime::from_timestamp_millis(x.0 as i64).expect("invalid timestamp stored in db")
 		}),
 		expired: params.is_expired(now),
 		scope: params.scope.get().0.clone(),
@@ -279,10 +280,10 @@ fn apply_token_updates(
 	if let Some(expiration) = updates.expiration {
 		params
 			.expiration
-			.update(Some(expiration.timestamp_millis() as u64));
+			.update(Some(ExpirationTime(expiration.timestamp_millis() as u64)).into());
 	}
 	if updates.never_expires {
-		params.expiration.update(None);
+		params.expiration.update(None.into());
 	}
 	if let Some(scope) = updates.scope {
 		params.scope.update(AdminApiTokenScope(scope));
@@ -618,6 +618,10 @@ pub enum PreviewClusterLayoutChangesResponse {
 		/// Plain-text information about the layout computation
 		/// (do not try to parse this)
 		message: Vec<String>,
+		/// Structured statistics about the layout computation
+		// FIXME for v3: remove default and skip_serializing_if
+		#[serde(default, skip_serializing_if = "Option::is_none")]
+		statistics: Option<Box<garage_rpc::layout::ComputationStat>>,
 		/// Details about the new cluster layout
 		new_layout: GetClusterLayoutResponse,
 	},
@@ -639,6 +643,10 @@ pub struct ApplyClusterLayoutResponse {
 	/// Plain-text information about the layout computation
 	/// (do not try to parse this)
 	pub message: Vec<String>,
+	/// Structured statistics about the layout computation
+	// FIXME for v3: remove default and skip_serializing_if
+	#[serde(default, skip_serializing_if = "Option::is_none")]
+	pub statistics: Option<garage_rpc::layout::ComputationStat>,
 	/// Details about the new cluster layout
 	pub layout: GetClusterLayoutResponse,
 }
@@ -1165,6 +1173,29 @@ pub struct LocalGetNodeInfoResponse {
 	pub rust_version: String,
 	/// database engine used for metadata
 	pub db_engine: String,
+	// FIXME for v3: remove Option<> and serde(default) for field below
+	// FIXME for v3: merge LocalGetNodeInfoResponse and NodeResp
+	/// Socket address used by other nodes to connect to this node for RPC
+	#[serde(default, skip_serializing_if = "Option::is_none")]
+	#[schema(value_type = Option<String>)]
+	pub addr: Option<SocketAddr>,
+	/// Whether this node is connected in the cluster
+	#[serde(default, skip_serializing_if = "Option::is_none")]
+	pub is_up: Option<bool>,
+	/// Role assigned to this node in the current cluster layout
+	#[serde(default, skip_serializing_if = "Option::is_none")]
+	pub role: Option<NodeAssignedRole>,
+	/// Whether this node is part of an older layout version and is draining data.
+	#[serde(default, skip_serializing_if = "Option::is_none")]
+	pub draining: Option<bool>,
+	/// Total and available space on the disk partition(s) containing the data
+	/// directory(ies)
+	#[serde(default, skip_serializing_if = "Option::is_none")]
+	pub data_partition: Option<FreeSpaceResp>,
+	/// Total and available space on the disk partition containing the
+	/// metadata directory
+	#[serde(default, skip_serializing_if = "Option::is_none")]
+	pub metadata_partition: Option<FreeSpaceResp>,
 }

 // ---- GetNodeStatistics ----
@@ -90,7 +90,7 @@ impl RequestHandler for GetBucketInfoRequest {
 				.bucket_alias_table
 				.get(&EmptyKey, &ga)
 				.await?
-				.and_then(|x| *x.state.get())
+				.and_then(|x| x.state.get().into_inner())
 				.ok_or_else(|| HelperError::NoSuchBucket(ga.to_string()))?,
 			(None, None, Some(search)) => {
 				let helper = garage.bucket_helper();
@@ -168,7 +168,7 @@ impl RequestHandler for CreateBucketRequest {
 			}

 			if let Some(alias) = garage.bucket_alias_table.get(&EmptyKey, ga).await? {
-				if alias.state.get().is_some() {
+				if alias.state.get().inner().is_some() {
 					return Err(CommonError::BucketAlreadyExists.into());
 				}
 			}
@@ -297,7 +297,7 @@ impl RequestHandler for UpdateBucketRequest {
 				let redirect_all = state
 					.website_config
 					.get()
-					.as_ref()
+					.inner()
 					.and_then(|wc| wc.redirect_all.clone());

 				let routing_rules = if let Some(rr) = wa.routing_rules {
@@ -311,26 +311,29 @@ impl RequestHandler for UpdateBucketRequest {
 					state
 						.website_config
 						.get()
-						.as_ref()
+						.inner()
 						.map(|wc| wc.routing_rules.clone())
 						.unwrap_or_default()
 				};

-				state.website_config.update(Some(WebsiteConfig {
-					index_document: wa.index_document.ok_or_bad_request(
-						"Please specify indexDocument when enabling website access.",
-					)?,
-					error_document: wa.error_document,
-					redirect_all,
-					routing_rules,
-				}));
+				state.website_config.update(
+					Some(WebsiteConfig {
+						index_document: wa.index_document.ok_or_bad_request(
+							"Please specify indexDocument when enabling website access.",
+						)?,
+						error_document: wa.error_document,
+						redirect_all,
+						routing_rules,
+					})
+					.into(),
+				);
 			} else {
 				if wa.index_document.is_some() || wa.error_document.is_some() {
 					return Err(Error::bad_request(
                        "Cannot specify indexDocument or errorDocument when disabling website access.",
                    ));
 				}
-				state.website_config.update(None);
+				state.website_config.update(None.into());
 			}
 		}

@@ -353,7 +356,7 @@ impl RequestHandler for UpdateBucketRequest {
 				Some(cc.into_garage_cors_config()?)
 			};

-			state.cors_config.update(cors_config);
+			state.cors_config.update(cors_config.into());
 		}

 		if let Some(lr) = self.body.lifecycle_rules {
@@ -370,7 +373,7 @@ impl RequestHandler for UpdateBucketRequest {
 				)
 			};

-			state.lifecycle_config.update(lifecycle_config);
+			state.lifecycle_config.update(lifecycle_config.into());
 		}

 		garage.bucket_table.insert(&bucket).await?;
@@ -687,45 +690,36 @@ async fn bucket_info_results(
 		.map(|x| x.filtered_values(&garage.system.cluster_layout()))
 		.unwrap_or_default();

-	let mut relevant_keys = HashMap::new();
-	for (k, _) in bucket
-		.state
-		.as_option()
-		.unwrap()
+	let state = bucket.state.as_option().unwrap();
+
+	let keys1 = state
 		.authorized_keys
 		.items()
 		.iter()
-	{
-		if let Some(key) = garage
-			.key_table
-			.get(&EmptyKey, k)
-			.await?
-			.filter(|k| !k.is_deleted())
-		{
-			if !key.state.is_deleted() {
-				relevant_keys.insert(k.clone(), key);
-			}
-		}
-	}
-	for ((k, _), _, _) in bucket
-		.state
-		.as_option()
-		.unwrap()
+		.filter(|(_, p)| p.is_any())
+		.map(|(k, _)| k);
+	let keys2 = state
 		.local_aliases
 		.items()
 		.iter()
-	{
-		if relevant_keys.contains_key(k) {
+		.filter(|(_, _, p)| *p)
+		.map(|((k, _), _, _)| k);
+
+	let mut relevant_keys = HashMap::new();
+	for key_id in keys1.chain(keys2) {
+		if relevant_keys.contains_key(key_id) {
 			continue;
 		}
-		if let Some(key) = garage.key_table.get(&EmptyKey, k).await? {
-			if !key.state.is_deleted() {
-				relevant_keys.insert(k.clone(), key);
-			}
+		if let Some(key) = garage.key_table.get(&EmptyKey, key_id).await? {
+			relevant_keys.insert(key_id.clone(), key);
+		} else {
+			warn!(
+				"Bucket {:?} references non-existent key {}",
+				bucket.id, key_id
+			);
 		}
 	}
-
-	let state = bucket.state.as_option().unwrap();
+	relevant_keys.retain(|_, k| !k.is_deleted());

 	let quotas = state.quotas.get();
 	let res = GetBucketInfoResponse {
@@ -739,8 +733,8 @@ async fn bucket_info_results(
 			.filter(|(_, _, a)| *a)
 			.map(|(n, _, _)| n.to_string())
 			.collect::<Vec<_>>(),
-		website_access: state.website_config.get().is_some(),
-		website_config: state.website_config.get().clone().map(|wsc| {
+		website_access: state.website_config.get().inner().is_some(),
+		website_config: state.website_config.get().inner().cloned().map(|wsc| {
 			GetBucketInfoWebsiteResponse {
 				index_document: wsc.index_document,
 				error_document: wsc.error_document,
@@ -752,42 +746,43 @@ async fn bucket_info_results(
 				),
 			}
 		}),
-		cors_rules: state.cors_config.get().as_ref().map(|rules| {
+		cors_rules: state.cors_config.get().inner().map(|rules| {
 			rules
 				.iter()
 				.map(xml::cors::CorsRule::from_garage_cors_rule)
 				.collect::<Vec<_>>()
 		}),
-		lifecycle_rules: state.lifecycle_config.get().as_ref().map(|lc| {
+		lifecycle_rules: state.lifecycle_config.get().inner().map(|lc| {
 			lc.iter()
 				.map(xml::lifecycle::LifecycleRule::from_garage_lifecycle_rule)
 				.collect::<Vec<_>>()
 		}),
 		keys: relevant_keys
 			.into_values()
-			.filter_map(|key| {
-				let p = key.state.as_option().unwrap();
-				let permissions = p
+			.map(|key| {
+				let st = key.state.as_option().unwrap();
+				let permissions = st
 					.authorized_buckets
 					.get(&bucket.id)
-					.filter(|p| p.is_any())
 					.map(|p| ApiBucketKeyPerm {
 						read: p.allow_read,
 						write: p.allow_write,
 						owner: p.allow_owner,
-					})?;
-				Some(GetBucketInfoKey {
+					})
+					.unwrap_or_default();
+				let bucket_local_aliases = st
+					.local_aliases
+					.items()
+					.iter()
+					.filter(|(_, _, b)| b.into_inner() == Some(bucket.id))
+					.map(|(n, _, _)| n.to_string())
+					.collect::<Vec<_>>();
+				GetBucketInfoKey {
 					access_key_id: key.key_id,
-					name: p.name.get().to_string(),
+					name: st.name.get().to_string(),
 					permissions,
-					bucket_local_aliases: p
-						.local_aliases
-						.items()
-						.iter()
-						.filter(|(_, _, b)| *b == Some(bucket.id))
-						.map(|(n, _, _)| n.to_string())
-						.collect::<Vec<_>>(),
-				})
+					bucket_local_aliases,
+				}
 			})
 			.collect::<Vec<_>>(),
 		objects: *counters.get(OBJECTS).unwrap_or(&0),
@@ -8,6 +8,7 @@ use garage_util::time::now_msec;

 use garage_model::garage::Garage;
 use garage_model::key_table::*;
+use garage_model::permission::ExpirationTime;

 use crate::api::*;
 use crate::error::*;
@@ -40,8 +41,8 @@ impl RequestHandler for ListKeysRequest {
 						DateTime::from_timestamp_millis(x as i64)
 							.expect("invalid timestamp stored in db")
 					}),
-					expiration: p.expiration.get().map(|x| {
-						DateTime::from_timestamp_millis(x as i64)
+					expiration: p.expiration.get().inner().map(|x| {
+						DateTime::from_timestamp_millis(x.0 as i64)
 							.expect("invalid timestamp stored in db")
 					}),
 					expired: p.is_expired(now),
@@ -187,38 +188,42 @@ async fn key_info_results(
 	key: Key,
 	show_secret: bool,
 ) -> Result<GetKeyInfoResponse, Error> {
-	let mut relevant_buckets = HashMap::new();
-
 	let key_state = key.state.as_option().unwrap();

-	for id in key_state
+	let buckets1 = key_state
 		.authorized_buckets
 		.items()
 		.iter()
-		.map(|(id, _)| id)
-		.chain(
-			key_state
-				.local_aliases
-				.items()
-				.iter()
-				.filter_map(|(_, _, v)| v.as_ref()),
-		) {
-		if !relevant_buckets.contains_key(id) {
-			if let Some(b) = garage.bucket_table.get(&EmptyKey, id).await? {
-				if b.state.as_option().is_some() {
-					relevant_buckets.insert(*id, b);
-				}
+		.filter(|(_, p)| p.is_any())
+		.map(|(id, _)| id);
+	let buckets2 = key_state
+		.local_aliases
+		.items()
+		.iter()
+		.filter_map(|(_, _, v)| v.inner());
+
+	let mut relevant_buckets = HashMap::new();
+	for bucket_id in buckets1.chain(buckets2) {
+		if !relevant_buckets.contains_key(bucket_id) {
+			if let Some(b) = garage.bucket_table.get(&EmptyKey, bucket_id).await? {
+				relevant_buckets.insert(*bucket_id, b);
+			} else {
+				warn!(
+					"Key {} references non-existent bucket {:?}",
+					key.key_id, bucket_id
+				);
 			}
 		}
 	}
+	relevant_buckets.retain(|_, b| !b.is_deleted());

 	let res = GetKeyInfoResponse {
 		name: key_state.name.get().clone(),
 		created: key_state.created.map(|x| {
 			DateTime::from_timestamp_millis(x as i64).expect("invalid timestamp stored in db")
 		}),
-		expiration: key_state.expiration.get().map(|x| {
-			DateTime::from_timestamp_millis(x as i64).expect("invalid timestamp stored in db")
+		expiration: key_state.expiration.get().inner().map(|x| {
+			DateTime::from_timestamp_millis(x.0 as i64).expect("invalid timestamp stored in db")
 		}),
 		expired: key_state.is_expired(now_msec()),
 		access_key_id: key.key_id.clone(),
@@ -232,7 +237,7 @@ async fn key_info_results(
 		},
 		buckets: relevant_buckets
 			.into_values()
-			.filter_map(|bucket| {
+			.map(|bucket| {
 				let state = bucket.state.as_option().unwrap();
 				let permissions = key_state
 					.authorized_buckets
@@ -242,8 +247,9 @@ async fn key_info_results(
 						read: p.allow_read,
 						write: p.allow_write,
 						owner: p.allow_owner,
-					})?;
-				Some(KeyInfoBucketResponse {
+					})
+					.unwrap_or_default();
+				KeyInfoBucketResponse {
 					id: hex::encode(bucket.id),
 					global_aliases: state
 						.aliases
@@ -260,7 +266,7 @@ async fn key_info_results(
 						.map(|((_, n), _, _)| n.to_string())
 						.collect::<Vec<_>>(),
 					permissions,
-				})
+				}
 			})
 			.collect::<Vec<_>>(),
 	};
@@ -283,10 +289,10 @@ fn apply_key_updates(key: &mut Key, updates: UpdateKeyRequestBody) -> Result<(),
 	if let Some(expiration) = updates.expiration {
 		key_state
 			.expiration
-			.update(Some(expiration.timestamp_millis() as u64));
+			.update(Some(ExpirationTime(expiration.timestamp_millis() as u64)).into());
 	}
 	if updates.never_expires {
-		key_state.expiration.update(None);
+		key_state.expiration.update(None.into());
 	}
 	if let Some(allow) = updates.allow {
 		if allow.create_bucket {
@@ -53,7 +53,7 @@ fn format_cluster_layout(layout: &layout::LayoutHistory) -> GetClusterLayoutResp
 		.roles
 		.items()
 		.iter()
-		.filter(|(k, _, v)| current.roles.get(k) != Some(v))
+		.filter(|(k, _, v)| current.roles.get(k).and_then(|vv| vv.0.as_ref()) != v.0.as_ref())
 		.map(|(k, _, v)| match &v.0 {
 			None => NodeRoleChange {
 				id: hex::encode(k),
@@ -255,10 +255,14 @@ impl RequestHandler for PreviewClusterLayoutChangesRequest {
 				Ok(PreviewClusterLayoutChangesResponse::Error { error })
 			}
 			Err(e) => Err(e.into()),
-			Ok((new_layout, msg)) => Ok(PreviewClusterLayoutChangesResponse::Success {
-				message: msg,
-				new_layout: format_cluster_layout(&new_layout),
-			}),
+			Ok((new_layout, stat)) => {
+				let message = stat.to_message();
+				Ok(PreviewClusterLayoutChangesResponse::Success {
+					message,
+					statistics: Some(Box::new(stat)),
+					new_layout: format_cluster_layout(&new_layout),
+				})
+			}
 		}
 	}
 }
@@ -272,7 +276,8 @@ impl RequestHandler for ApplyClusterLayoutRequest {
 		_admin: &Admin,
 	) -> Result<ApplyClusterLayoutResponse, Error> {
 		let layout = garage.system.cluster_layout().inner().clone();
-		let (layout, msg) = layout.apply_staged_changes(self.version)?;
+		let (layout, stat) = layout.apply_staged_changes(self.version)?;
+		let message = stat.to_message();

 		garage
 			.system
@@ -281,7 +286,8 @@ impl RequestHandler for ApplyClusterLayoutRequest {
 			.await?;

 		Ok(ApplyClusterLayoutResponse {
-			message: msg,
+			message,
+			statistics: Some(stat),
 			layout: format_cluster_layout(&layout),
 		})
 	}
@@ -25,6 +25,9 @@ impl RequestHandler for LocalGetNodeInfoRequest {
 		let sys_status = garage.system.local_status();
 		let hostname = sys_status.hostname.unwrap_or_default().to_string();

+		let layout = garage.system.cluster_layout();
+		let current_layout = layout.inner().current();
+
 		Ok(LocalGetNodeInfoResponse {
 			node_id: hex::encode(garage.system.id),
 			hostname: Some(hostname),
@@ -33,6 +36,41 @@ impl RequestHandler for LocalGetNodeInfoRequest {
 				.map(|features| features.iter().map(ToString::to_string).collect()),
 			rust_version: garage_util::version::rust_version().to_string(),
 			db_engine: garage.db.engine(),
+			is_up: Some(true),
+			addr: garage
+				.system
+				.get_known_nodes()
+				.iter()
+				.find(|x| x.id == garage.system.id)
+				.and_then(|x| x.addr),
+			draining: Some(
+				current_layout.node_role(&garage.system.id).is_none()
+					&& layout
+						.inner()
+						.versions
+						.iter()
+						.filter(|x| x.version != current_layout.version)
+						.any(|x| x.node_role(&garage.system.id).is_some()),
+			),
+			role: current_layout
+				.node_role(&garage.system.id)
+				.map(|v| NodeAssignedRole {
+					zone: v.zone.clone(),
+					capacity: v.capacity,
+					tags: v.tags.clone(),
+				}),
+			data_partition: sys_status
+				.data_disk_avail
+				.map(|(avail, total)| FreeSpaceResp {
+					available: avail,
+					total,
+				}),
+			metadata_partition: sys_status
+				.meta_disk_avail
+				.map(|(avail, total)| FreeSpaceResp {
+					available: avail,
+					total,
+				}),
 		})
 	}
 }
@@ -876,7 +876,7 @@ impl Modify for SecurityAddon {
 #[derive(OpenApi)]
 #[openapi(
    info(
-        version = "v2.3.0",
+        version = "v2.4.0",
        title = "Garage administration API",
        description = "Administrate your Garage cluster programmatically, including status, layout, keys, buckets, and maintenance tasks.

@@ -360,14 +360,7 @@ impl Worker for BlockRcRepair {
 		_must_exit: &mut watch::Receiver<bool>,
 	) -> Result<WorkerState, GarageError> {
 		for _i in 0..RC_REPAIR_ITER_COUNT {
-			let next1 = self
-				.block_manager
-				.rc
-				.rc_table
-				.range(self.cursor.as_slice()..)?
-				.next()
-				.transpose()?
-				.map(|(k, _)| Hash::try_from(k.as_slice()).unwrap());
+			let next1 = self.block_manager.rc.get_first_hash_from(self.cursor)?;
 			let next2 = self
 				.block_ref_table
 				.data
@@ -164,7 +164,7 @@ async fn check_domain(garage: &Arc<Garage>, domain: &str) -> Result<bool, Error>
 	}

 	let bucket_state = bucket.state.as_option().unwrap();
-	let bucket_website_config = bucket_state.website_config.get();
+	let bucket_website_config = bucket_state.website_config.get().inner();

 	match bucket_website_config {
 		Some(_v) => Ok(true),
@@ -1,6 +1,6 @@
 [package]
 name = "garage_api_common"
-version = "2.3.0"
+version = "2.4.0"
 authors = ["Alex Auvolat <alex@adnab.me>"]
 edition = "2018"
 license = "AGPL-3.0"
@@ -1,8 +1,9 @@
 use std::sync::Arc;

 use http::header::{
-	ACCESS_CONTROL_ALLOW_HEADERS, ACCESS_CONTROL_ALLOW_METHODS, ACCESS_CONTROL_ALLOW_ORIGIN,
-	ACCESS_CONTROL_EXPOSE_HEADERS, ACCESS_CONTROL_REQUEST_HEADERS, ACCESS_CONTROL_REQUEST_METHOD,
+	HeaderValue, ACCESS_CONTROL_ALLOW_HEADERS, ACCESS_CONTROL_ALLOW_METHODS,
+	ACCESS_CONTROL_ALLOW_ORIGIN, ACCESS_CONTROL_EXPOSE_HEADERS, ACCESS_CONTROL_REQUEST_HEADERS,
+	ACCESS_CONTROL_REQUEST_METHOD, VARY,
 };
 use hyper::{body::Body, body::Incoming as IncomingBody, Request, Response, StatusCode};

@@ -12,20 +13,25 @@ use garage_model::garage::Garage;
 use crate::common_error::{CommonError, OkOrBadRequest, OkOrInternalError};
 use crate::helpers::*;

+// Return both the matching rule and the parsed Origin header so callers that
+// apply CORS headers don't have to repeat Origin lookup and validation.
 pub fn find_matching_cors_rule<'a, B>(
 	bucket_params: &'a BucketParams,
-	req: &Request<B>,
-) -> Result<Option<&'a GarageCorsRule>, CommonError> {
-	if let Some(cors_config) = bucket_params.cors_config.get() {
+	req: &'a Request<B>,
+) -> Result<Option<(&'a GarageCorsRule, &'a str)>, CommonError> {
+	if let Some(cors_config) = bucket_params.cors_config.get().inner() {
 		if let Some(origin) = req.headers().get("Origin") {
 			let origin = origin.to_str()?;
 			let request_headers = match req.headers().get(ACCESS_CONTROL_REQUEST_HEADERS) {
 				Some(h) => h.to_str()?.split(',').map(|h| h.trim()).collect::<Vec<_>>(),
 				None => vec![],
 			};
-			return Ok(cors_config.iter().find(|rule| {
-				cors_rule_matches(rule, origin, req.method().as_ref(), request_headers.iter())
-			}));
+			return Ok(cors_config
+				.iter()
+				.find(|rule| {
+					cors_rule_matches(rule, origin, req.method().as_ref(), request_headers.iter())
+				})
+				.map(|rule| (rule, origin)));
 		}
 	}
 	Ok(None)
@@ -41,24 +47,39 @@ where
 	HI: Iterator<Item = S>,
 	S: AsRef<str>,
 {
-	rule.allow_origins.iter().any(|x| x == "*" || x == origin)
+	rule.allow_origins.iter().any(|x| wildcard_match(x, origin))
 		&& rule.allow_methods.iter().any(|x| x == "*" || x == method)
 		&& request_headers.all(|h| {
 			rule.allow_headers
 				.iter()
-				.any(|x| x == "*" || x == h.as_ref())
+				.any(|x| wildcard_match(x, h.as_ref()))
 		})
 }

+/// Checks whether `candidate` matches the pattern `allowed_wildcard`.
+#[inline]
+fn wildcard_match(allowed_wildcard: &String, candidate: &str) -> bool {
+	if allowed_wildcard.contains("*") {
+		let parts = allowed_wildcard.split("*").collect::<Vec<&str>>();
+		parts.len() == 2 && candidate.starts_with(parts[0]) && candidate.ends_with(parts[1])
+	} else {
+		candidate == allowed_wildcard
+	}
+}
+
 pub fn add_cors_headers(
 	resp: &mut Response<impl Body>,
 	rule: &GarageCorsRule,
+	request_origin: &str,
 ) -> Result<(), http::header::InvalidHeaderValue> {
 	let h = resp.headers_mut();
-	h.insert(
-		ACCESS_CONTROL_ALLOW_ORIGIN,
-		rule.allow_origins.join(", ").parse()?,
-	);
+	let is_wildcard_origin = rule.allow_origins.iter().any(|origin| origin == "*");
+	let allow_origin = if is_wildcard_origin {
+		"*"
+	} else {
+		request_origin
+	};
+	h.insert(ACCESS_CONTROL_ALLOW_ORIGIN, allow_origin.parse()?);
 	h.insert(
 		ACCESS_CONTROL_ALLOW_METHODS,
 		rule.allow_methods.join(", ").parse()?,
@@ -71,6 +92,12 @@ pub fn add_cors_headers(
 		ACCESS_CONTROL_EXPOSE_HEADERS,
 		rule.expose_headers.join(", ").parse()?,
 	);
+	// When ACAO reflects the request origin instead of returning "*",
+	// caches must vary on the Origin request header to avoid reusing
+	// a response generated for one origin when serving another origin.
+	if !is_wildcard_origin {
+		h.insert(VARY, HeaderValue::from_static("Origin"));
+	}
 	Ok(())
 }

@@ -107,6 +134,7 @@ pub fn handle_options_api(
 			Ok(Response::builder()
 				.header(ACCESS_CONTROL_ALLOW_ORIGIN, "*")
 				.header(ACCESS_CONTROL_ALLOW_METHODS, "*")
+				.header(ACCESS_CONTROL_ALLOW_HEADERS, "*")
 				.status(StatusCode::OK)
 				.body(EmptyBody::new())?)
 		}
@@ -141,7 +169,7 @@ pub fn handle_options_for_bucket<B>(
 		None => vec![],
 	};

-	if let Some(cors_config) = bucket_params.cors_config.get() {
+	if let Some(cors_config) = bucket_params.cors_config.get().inner() {
 		let matching_rule = cors_config
 			.iter()
 			.find(|rule| cors_rule_matches(rule, origin, request_method, request_headers.iter()));
@@ -149,7 +177,17 @@ pub fn handle_options_for_bucket<B>(
 			let mut resp = Response::builder()
 				.status(StatusCode::OK)
 				.body(EmptyBody::new())?;
-			add_cors_headers(&mut resp, rule).ok_or_internal_error("Invalid CORS configuration")?;
+			add_cors_headers(&mut resp, rule, origin)
+				.ok_or_internal_error("Invalid CORS configuration")?;
+			// Preflight responses vary not only on Origin but also on the
+			// requested method and requested headers, so caches must not
+			// reuse one preflight decision for a different preflight input.
+			resp.headers_mut().insert(
+				VARY,
+				"Origin, Access-Control-Request-Method, Access-Control-Request-Headers"
+					.parse()
+					.expect("static vary header"),
+			);
 			return Ok(resp);
 		}
 	}
@@ -158,3 +196,316 @@ pub fn handle_options_for_bucket<B>(
 		"This CORS request is not allowed.".into(),
 	))
 }
+
+#[cfg(test)]
+mod tests {
+	use super::*;
+
+	fn cors_rule(
+		allow_origins: &[&str],
+		allow_methods: &[&str],
+		allow_headers: &[&str],
+	) -> GarageCorsRule {
+		GarageCorsRule {
+			id: None,
+			max_age_seconds: None,
+			allow_origins: allow_origins.iter().map(|s| s.to_string()).collect(),
+			allow_methods: allow_methods.iter().map(|s| s.to_string()).collect(),
+			allow_headers: allow_headers.iter().map(|s| s.to_string()).collect(),
+			expose_headers: vec![],
+		}
+	}
+
+	#[test]
+	fn matches_when_origin_method_and_headers_are_explicitly_allowed() {
+		let rule = cors_rule(
+			&["https://app.example.test"],
+			&["GET", "PUT"],
+			&["content-type", "x-custom"],
+		);
+		let headers = vec!["content-type", "x-custom"];
+
+		assert!(cors_rule_matches(
+			&rule,
+			"https://app.example.test",
+			"PUT",
+			headers.iter(),
+		));
+	}
+
+	#[test]
+	fn does_not_match_when_origin_is_not_allowed() {
+		let rule = cors_rule(&["https://app.example.test"], &["GET"], &["*"]);
+
+		assert!(!cors_rule_matches(
+			&rule,
+			"https://evil.example.test",
+			"GET",
+			std::iter::empty::<&str>(),
+		));
+	}
+
+	#[test]
+	fn does_not_match_when_method_is_not_allowed() {
+		let rule = cors_rule(&["*"], &["GET"], &["*"]);
+
+		assert!(!cors_rule_matches(
+			&rule,
+			"https://app.example.test",
+			"DELETE",
+			std::iter::empty::<&str>(),
+		));
+	}
+
+	#[test]
+	fn does_not_match_when_a_requested_header_is_not_allowed() {
+		let rule = cors_rule(&["*"], &["GET"], &["content-type"]);
+		let headers = vec!["content-type", "x-not-allowed"];
+
+		assert!(!cors_rule_matches(
+			&rule,
+			"https://app.example.test",
+			"GET",
+			headers.iter(),
+		));
+	}
+
+	#[test]
+	fn wildcard_origin_method_and_headers_match_anything() {
+		let rule = cors_rule(&["*"], &["*"], &["*"]);
+		let headers = vec!["x-anything"];
+
+		assert!(cors_rule_matches(
+			&rule,
+			"https://app.example.test",
+			"DELETE",
+			headers.iter(),
+		));
+	}
+
+	#[test]
+	fn wildcard_origin_regex() {
+		let rule = cors_rule(&["https://*.localhost.com"], &["*"], &["*"]);
+		let headers = vec!["x-anything"];
+
+		assert!(cors_rule_matches(
+			&rule,
+			"https://s3.localhost.com",
+			"DELETE",
+			headers.iter(),
+		));
+	}
+
+	#[test]
+	fn origin_matching_cases() {
+		// (allow_origins, origin, expect_match)
+		let cases: &[(&[&str], &str, bool)] = &[
+			// exact match
+			(
+				&["https://app.example.test"],
+				"https://app.example.test",
+				true,
+			),
+			(
+				&["https://app.example.test"],
+				"https://other.example.test",
+				false,
+			),
+			// full wildcard
+			(&["*"], "https://anything.example.test", true),
+			// subdomain glob
+			(
+				&["https://*.example.test"],
+				"https://foo.example.test",
+				true,
+			),
+			(&["https://*.example.test"], "https://example.test", false),
+			(
+				&["https://*.example.test"],
+				"http://foo.example.test",
+				false,
+			),
+			// multiple allowed origins, at least one should match
+			(
+				&["https://a.example.test", "https://b.example.test"],
+				"https://b.example.test",
+				true,
+			),
+			// match multiple origins
+			(
+				&["https://a*.example.test", "https://ab*.example.test"],
+				"https://abc.example.test",
+				true,
+			),
+			(
+				&["https://a.example.test", "https://b.example.test"],
+				"https://c.example.test",
+				false,
+			),
+			// at most one '*' in a pattern is allowed
+			(&["https://*.example.*"], "https://a.example.test", false),
+			// domain changed with wildcard
+			(
+				&["https://*example.test"],
+				"https://garageexample.test",
+				true,
+			),
+			// trailing '*' matches any suffix, including the empty string,
+			// so this also matches origins with anything (or nothing) after
+			// "example."
+			(&["https://example.*"], "https://example.test", true),
+			(&["https://*example.test"], "https://example.test", true),
+			(&["https://example.*"], "https://example.", true),
+		];
+
+		for (allow_origins, origin, expect_match) in cases {
+			let rule = cors_rule(allow_origins, &["GET"], &["*"]);
+			let got = cors_rule_matches(&rule, origin, "GET", std::iter::empty::<&str>());
+			assert_eq!(
+				got, *expect_match,
+				"allow_origins={allow_origins:?}, origin={origin:?}: expected match={expect_match}, got {got}"
+			);
+		}
+	}
+
+	#[test]
+	fn header_matching_cases() {
+		// (allow_headers, requested_headers, expect_match)
+		let cases: &[(&[&str], &[&str], bool)] = &[
+			// exact match
+			(&["content-type"], &["content-type"], true),
+			(&["content-type"], &["x-custom"], false),
+			// full wildcard
+			(&["*"], &["x-anything"], true),
+			// no headers requested always matches, regardless of allow_headers
+			(&["content-type"], &[], true),
+			(&[], &[], true),
+			// prefix glob
+			(&["x-amz-*"], &["x-amz-meta-foo"], true),
+			(&["x-amz-*"], &["x-amz-"], true),
+			(&["x-amz-*"], &["x-other"], false),
+			// suffix glob
+			(&["*-meta"], &["foo-meta"], true),
+			(&["*-meta"], &["-meta"], true),
+			(&["*-meta"], &["foo-meta-bar"], false),
+			// multiple allowed headers, at least one should match per requested header
+			(
+				&["content-type", "x-amz-*"],
+				&["content-type", "x-amz-meta-foo"],
+				true,
+			),
+			(&["content-type", "x-amz-*"], &["x-other"], false),
+			// all requested headers must be covered
+			(&["content-type"], &["content-type", "x-custom"], false),
+			// at most one '*' in a pattern is allowed
+			(&["x-*-*"], &["x-a-b"], false),
+		];
+
+		for (allow_headers, requested_headers, expect_match) in cases {
+			let rule = cors_rule(&["*"], &["GET"], allow_headers);
+			let got = cors_rule_matches(
+				&rule,
+				"https://app.example.test",
+				"GET",
+				requested_headers.iter(),
+			);
+			assert_eq!(
+				got, *expect_match,
+				"allow_headers={allow_headers:?}, requested_headers={requested_headers:?}: expected match={expect_match}, got {got}"
+			);
+		}
+	}
+
+	fn bucket_params_with_rule(allow_origins: Vec<&str>) -> BucketParams {
+		let mut bucket_params = BucketParams::default();
+		bucket_params.cors_config.update(
+			Some(vec![GarageCorsRule {
+				id: Some("cors-test".into()),
+				max_age_seconds: None,
+				allow_origins: allow_origins.into_iter().map(str::to_string).collect(),
+				allow_methods: vec!["GET".into(), "PUT".into()],
+				allow_headers: vec!["*".into()],
+				expose_headers: vec![],
+			}])
+			.into(),
+		);
+		bucket_params
+	}
+
+	fn preflight_request(origin: &str) -> Request<()> {
+		Request::builder()
+			.method("OPTIONS")
+			.uri("http://example.test/bucket")
+			.header("Origin", origin)
+			.header(ACCESS_CONTROL_REQUEST_METHOD, "PUT")
+			.body(())
+			.unwrap()
+	}
+
+	#[test]
+	fn preflight_with_single_allowed_origin_returns_request_origin() {
+		let bucket_params = bucket_params_with_rule(vec!["https://app.example.test"]);
+		let req = preflight_request("https://app.example.test");
+
+		let resp = handle_options_for_bucket(&req, &bucket_params).unwrap();
+
+		assert_eq!(
+			resp.headers().get(ACCESS_CONTROL_ALLOW_ORIGIN).unwrap(),
+			"https://app.example.test"
+		);
+		let vary_values: Vec<_> = resp
+			.headers()
+			.get_all(VARY)
+			.iter()
+			.map(|value| value.to_str().unwrap())
+			.collect();
+		assert_eq!(
+			vary_values,
+			vec!["Origin, Access-Control-Request-Method, Access-Control-Request-Headers",]
+		);
+	}
+
+	#[test]
+	fn preflight_with_multiple_allowed_origins_reflects_request_origin() {
+		let bucket_params = bucket_params_with_rule(vec![
+			"https://app.example.test",
+			"https://admin.example.test",
+		]);
+		let req = preflight_request("https://app.example.test");
+
+		let resp = handle_options_for_bucket(&req, &bucket_params).unwrap();
+
+		// This assertion documents the behavior browsers expect:
+		// even if multiple origins are allowed by configuration, the
+		// response should reflect the request origin rather than emit
+		// a comma-separated list. It currently fails and is meant to
+		// turn green once header generation is corrected.
+		assert_eq!(
+			resp.headers().get(ACCESS_CONTROL_ALLOW_ORIGIN).unwrap(),
+			"https://app.example.test"
+		);
+	}
+
+	#[test]
+	fn preflight_with_wildcard_allowed_origin_returns_wildcard() {
+		let bucket_params = bucket_params_with_rule(vec!["*"]);
+		let req = preflight_request("https://app.example.test");
+
+		let resp = handle_options_for_bucket(&req, &bucket_params).unwrap();
+
+		assert_eq!(
+			resp.headers().get(ACCESS_CONTROL_ALLOW_ORIGIN).unwrap(),
+			"*"
+		);
+		let vary_values: Vec<_> = resp
+			.headers()
+			.get_all(VARY)
+			.iter()
+			.map(|value| value.to_str().unwrap())
+			.collect();
+		assert_eq!(
+			vary_values,
+			vec!["Origin, Access-Control-Request-Method, Access-Control-Request-Headers",]
+		);
+	}
+}
@@ -357,7 +357,13 @@ pub fn canonical_request(
 		items.join("&")
 	};

-	// Canonical header string calculated from signed headers
+	// Canonical header string calculated from signed headers.
+	//
+	// Per the SigV4 spec, signed header values must have sequential
+	// internal whitespace collapsed to a single space, in addition to
+	// being trimmed. AWS SDKs do this before computing the signature
+	// but transmit the raw value on the wire, so we must match.
+	// -> https://docs.aws.amazon.com/IAM/latest/UserGuide/create-signed-request.html
 	let canonical_header_string = signed_headers
 		.iter()
 		.map(|name| {
@@ -372,7 +378,11 @@ pub fn canonical_request(
 				built_string.push(',');
 				built_string.push_str(extend_string);
 			}
-			Ok(format!("{}:{}", name.as_str(), built_string.trim()))
+			let normalized = built_string
+				.split_whitespace()
+				.collect::<Vec<_>>()
+				.join(" ");
+			Ok(format!("{}:{}", name.as_str(), normalized))
 		})
 		.collect::<Result<Vec<String>, Error>>()?
 		.join("\n");
@@ -1,6 +1,6 @@
 [package]
 name = "garage_api_k2v"
-version = "2.3.0"
+version = "2.4.0"
 authors = ["Alex Auvolat <alex@adnab.me>"]
 edition = "2018"
 license = "AGPL-3.0"
@@ -111,7 +111,7 @@ impl ApiHandler for K2VApiServer {
 			Method::GET | Method::HEAD | Method::POST => {
 				find_matching_cors_rule(&bucket_params, &req)
 					.ok_or_internal_error("Error looking up CORS rule")?
-					.cloned()
+					.map(|(rule, origin)| (rule.clone(), origin.to_string()))
 			}
 			_ => None,
 		};
@@ -164,8 +164,8 @@ impl ApiHandler for K2VApiServer {
 		// If request was a success and we have a CORS rule that applies to it,
 		// add the corresponding CORS headers to the response
 		let mut resp_ok = resp?;
-		if let Some(rule) = matching_cors_rule {
-			add_cors_headers(&mut resp_ok, &rule)
+		if let Some((rule, origin)) = matching_cors_rule {
+			add_cors_headers(&mut resp_ok, &rule, &origin)
 				.ok_or_internal_error("Invalid bucket CORS configuration")?;
 		}

@@ -5,12 +5,13 @@ use serde::{Deserialize, Serialize};
 use garage_table::{EnumerationOrder, TableSchema};

 use garage_model::k2v::item_table::*;
+use garage_model::k2v::rpc::K2VMonotonicRead;

 use garage_api_common::helpers::*;

 use crate::api_server::{ReqBody, ResBody};
 use crate::error::*;
-use crate::item::parse_causality_token;
+use crate::item::{is_monotonic_read, parse_causality_token};
 use crate::range::read_range;

 pub async fn handle_insert_batch(
@@ -47,12 +48,13 @@ pub async fn handle_read_batch(
 	ctx: ReqCtx,
 	req: Request<ReqBody>,
 ) -> Result<Response<ResBody>, Error> {
+	let monotonic_read = is_monotonic_read(&req)?;
 	let queries = req.into_body().json::<Vec<ReadBatchQuery>>().await?;

 	let resp_results = futures::future::join_all(
 		queries
 			.into_iter()
-			.map(|q| handle_read_batch_query(&ctx, q)),
+			.map(|q| handle_read_batch_query(&ctx, q, monotonic_read)),
 	)
 	.await;

@@ -67,6 +69,7 @@ pub async fn handle_read_batch(
 async fn handle_read_batch_query(
 	ctx: &ReqCtx,
 	query: ReadBatchQuery,
+	monotonic_read: K2VMonotonicRead,
 ) -> Result<ReadBatchResponse, Error> {
 	let ReqCtx {
 		garage, bucket_id, ..
@@ -90,12 +93,13 @@ async fn handle_read_batch_query(
 			.start
 			.as_ref()
 			.ok_or_bad_request("start should be specified if single_item is set")?;
-		let item = garage
-			.k2v
-			.item_table
-			.get(&partition, sk)
-			.await?
-			.filter(|e| K2VItemTable::matches_filter(e, &filter));
+		let item = match monotonic_read {
+			K2VMonotonicRead::Monotonic => {
+				garage.k2v.item_table.get_monotonic(&partition, sk).await?
+			}
+			K2VMonotonicRead::NonMonotonic => garage.k2v.item_table.get(&partition, sk).await?,
+		}
+		.filter(|e| K2VItemTable::matches_filter(e, &filter));
 		match item {
 			Some(i) => (vec![ReadBatchResponseItem::from(i)], false, None),
 			None => (vec![], false, None),
@@ -110,6 +114,7 @@ async fn handle_read_batch_query(
 			query.limit,
 			Some(filter),
 			EnumerationOrder::from_reverse(query.reverse),
+			monotonic_read,
 		)
 		.await?;

@@ -218,6 +223,7 @@ async fn handle_delete_batch_query(
 			None,
 			Some(filter),
 			EnumerationOrder::Forward,
+			K2VMonotonicRead::NonMonotonic,
 		)
 		.await?;
 		assert!(!more);
@@ -260,6 +266,7 @@ pub(crate) async fn handle_poll_range(
 	let ReqCtx {
 		garage, bucket_id, ..
 	} = ctx;
+	let monotonic_read = is_monotonic_read(&req)?;
 	use garage_model::k2v::sub::PollRange;

 	let query = req.into_body().json::<PollRangeQuery>().await?;
@@ -281,6 +288,7 @@ pub(crate) async fn handle_poll_range(
 			},
 			query.seen_marker,
 			timeout_msec,
+			monotonic_read,
 		)
 		.await
 		.map_err(pass_helper_error)?;
@@ -44,6 +44,10 @@ pub enum Error {
 	#[error("Invalid causality token")]
 	InvalidCausalityToken,

+	/// Invalid parameter for x-garage-non-monotonic-read
+	#[error("Invalid X-Garage-Non-Monotonic-Read value: {0}")]
+	InvalidNonMonotonicRead(String),
+
 	/// The client asked for an invalid return format (invalid Accept header)
 	#[error("Not acceptable: {0}")]
 	NotAcceptable(String),
@@ -85,6 +89,7 @@ impl Error {
 			Error::InvalidBase64(_) => "InvalidBase64",
 			Error::InvalidUtf8Str(_) => "InvalidUtf8String",
 			Error::InvalidCausalityToken => "CausalityToken",
+			Error::InvalidNonMonotonicRead(_) => "InvalidNonMonotonicRead",
 			Error::InvalidDigest(_) => "InvalidDigest",
 		}
 	}
@@ -101,7 +106,8 @@ impl ApiError for Error {
 			| Error::InvalidBase64(_)
 			| Error::InvalidUtf8Str(_)
 			| Error::InvalidDigest(_)
-			| Error::InvalidCausalityToken => StatusCode::BAD_REQUEST,
+			| Error::InvalidCausalityToken
+			| Error::InvalidNonMonotonicRead(_) => StatusCode::BAD_REQUEST,
 		}
 	}

@@ -4,6 +4,7 @@ use serde::Serialize;
 use garage_table::util::*;

 use garage_model::k2v::item_table::{BYTES, CONFLICTS, ENTRIES, VALUES};
+use garage_model::k2v::rpc::K2VMonotonicRead;

 use garage_api_common::helpers::*;

@@ -40,6 +41,7 @@ pub async fn handle_read_index(
 		limit,
 		Some((DeletedFilter::NotDeleted, node_id_vec)),
 		EnumerationOrder::from_reverse(reverse),
+		K2VMonotonicRead::NonMonotonic,
 	)
 	.await?;

@@ -5,6 +5,7 @@ use hyper::{Request, Response, StatusCode};

 use garage_model::k2v::causality::*;
 use garage_model::k2v::item_table::*;
+use garage_model::k2v::rpc::K2VMonotonicRead;

 use garage_api_common::helpers::*;

@@ -12,6 +13,7 @@ use crate::api_server::{ReqBody, ResBody};
 use crate::error::*;

 pub const X_GARAGE_CAUSALITY_TOKEN: &str = "X-Garage-Causality-Token";
+pub const X_GARAGE_NON_MONOTONIC_READ: &str = "X-Garage-Non-Monotonic-Read";

 pub enum ReturnFormat {
 	Json,
@@ -23,6 +25,21 @@ pub(crate) fn parse_causality_token(s: &str) -> Result<CausalContext, Error> {
 	CausalContext::parse(s).ok_or(Error::InvalidCausalityToken)
 }

+pub(crate) fn is_monotonic_read(req: &Request<ReqBody>) -> Result<K2VMonotonicRead, Error> {
+	let v_opt = req
+		.headers()
+		.get(X_GARAGE_NON_MONOTONIC_READ)
+		.map(|s| s.to_str())
+		.transpose()?;
+
+	match v_opt {
+		Some("true") => Ok(K2VMonotonicRead::NonMonotonic),
+		// Reads are monotonic by default
+		Some("false") | None => Ok(K2VMonotonicRead::Monotonic),
+		Some(s) => Err(Error::InvalidNonMonotonicRead(s.to_string())),
+	}
+}
+
 impl ReturnFormat {
 	pub fn from(req: &Request<ReqBody>) -> Result<Self, Error> {
 		let accept = match req.headers().get(header::ACCEPT) {
@@ -108,21 +125,26 @@ pub async fn handle_read_item(
 	let ReqCtx {
 		garage, bucket_id, ..
 	} = &ctx;
-
+	let monotonic_read = is_monotonic_read(req)?;
 	let format = ReturnFormat::from(req)?;
+	let partition_key = K2VItemPartition {
+		bucket_id: *bucket_id,
+		partition_key: partition_key.to_string(),
+	};

-	let item = garage
-		.k2v
-		.item_table
-		.get(
-			&K2VItemPartition {
-				bucket_id: *bucket_id,
-				partition_key: partition_key.to_string(),
-			},
-			sort_key,
-		)
-		.await?
-		.ok_or(Error::NoSuchKey)?;
+	let item = match monotonic_read {
+		K2VMonotonicRead::Monotonic => {
+			garage
+				.k2v
+				.item_table
+				.get_monotonic(&partition_key, sort_key)
+				.await?
+		}
+		K2VMonotonicRead::NonMonotonic => {
+			garage.k2v.item_table.get(&partition_key, sort_key).await?
+		}
+	}
+	.ok_or(Error::NoSuchKey)?;

 	format.make_response(&item)
 }
@@ -214,6 +236,7 @@ pub async fn handle_poll_item(
 	let ReqCtx {
 		garage, bucket_id, ..
 	} = &ctx;
+	let monotonic_read = is_monotonic_read(req)?;
 	let format = ReturnFormat::from(req)?;

 	let causal_context =
@@ -230,6 +253,7 @@ pub async fn handle_poll_item(
 			sort_key,
 			causal_context,
 			timeout_msec,
+			monotonic_read,
 		)
 		.await?;

@@ -4,6 +4,8 @@

 use std::sync::Arc;

+use garage_model::k2v::rpc::K2VMonotonicRead;
+
 use garage_table::replication::TableShardedReplication;
 use garage_table::*;

@@ -23,6 +25,7 @@ pub(crate) async fn read_range<F>(
 	limit: Option<u64>,
 	filter: Option<F::Filter>,
 	enumeration_order: EnumerationOrder,
+	monotonic_read: K2VMonotonicRead,
 ) -> Result<(Vec<F::E>, bool, Option<String>), Error>
 where
 	F: TableSchema<S = String> + 'static,
@@ -53,15 +56,30 @@ where
 			1000,
 			limit.map(|x| x as usize).unwrap_or(usize::MAX - 10) - entries.len() + 2,
 		);
-		let get_ret = table
-			.get_range(
-				partition_key,
-				start.clone(),
-				filter.clone(),
-				n_get,
-				enumeration_order,
-			)
-			.await?;
+		let get_ret = match monotonic_read {
+			K2VMonotonicRead::Monotonic => {
+				table
+					.get_range_monotonic(
+						partition_key,
+						start.clone(),
+						filter.clone(),
+						n_get,
+						enumeration_order,
+					)
+					.await?
+			}
+			K2VMonotonicRead::NonMonotonic => {
+				table
+					.get_range(
+						partition_key,
+						start.clone(),
+						filter.clone(),
+						n_get,
+						enumeration_order,
+					)
+					.await?
+			}
+		};

 		let get_ret_len = get_ret.len();

@@ -1,6 +1,6 @@
 [package]
 name = "garage_api_s3"
-version = "2.3.0"
+version = "2.4.0"
 authors = ["Alex Auvolat <alex@adnab.me>"]
 edition = "2018"
 license = "AGPL-3.0"
@@ -159,7 +159,8 @@ impl ApiHandler for S3ApiServer {
 			return Err(Error::forbidden("Operation is not allowed for this key."));
 		}

-		let matching_cors_rule = find_matching_cors_rule(&bucket_params, &req)?.cloned();
+		let matching_cors = find_matching_cors_rule(&bucket_params, &req)?
+			.map(|(rule, origin)| (rule.clone(), origin.to_string()));

 		let ctx = ReqCtx {
 			garage,
@@ -334,8 +335,8 @@ impl ApiHandler for S3ApiServer {
 		// If request was a success and we have a CORS rule that applies to it,
 		// add the corresponding CORS headers to the response
 		let mut resp_ok = resp?;
-		if let Some(rule) = matching_cors_rule {
-			add_cors_headers(&mut resp_ok, &rule)
+		if let Some((rule, origin)) = matching_cors {
+			add_cors_headers(&mut resp_ok, &rule, &origin)
 				.ok_or_internal_error("Invalid bucket CORS configuration")?;
 		}

@@ -122,7 +122,7 @@ pub async fn handle_list_buckets(
 			for (alias, _, _active) in bucket.aliases().iter().filter(|(_, _, active)| *active) {
 				let alias_opt = garage.bucket_alias_table.get(&EmptyKey, alias).await?;
 				if let Some(alias_ent) = alias_opt {
-					if *alias_ent.state.get() == Some(*bucket_id) {
+					if alias_ent.state.get().inner() == Some(bucket_id) {
 						aliases.insert(alias_ent.name().to_string(), *bucket_id);
 					}
 				}
@@ -134,7 +134,7 @@ pub async fn handle_list_buckets(
 	}

 	for (alias, _, id_opt) in key_p.local_aliases.items() {
-		if let Some(id) = id_opt {
+		if let Some(id) = id_opt.inner() {
 			aliases.insert(alias.clone(), *id);
 		}
 	}
@@ -256,7 +256,10 @@ pub async fn handle_delete_bucket(ctx: ReqCtx) -> Result<Response<ResBody>, Erro

 	let key_params = api_key.params().unwrap();

-	let is_local_alias = matches!(key_params.local_aliases.get(bucket_name), Some(Some(_)));
+	let is_local_alias = matches!(
+		key_params.local_aliases.get(bucket_name).map(|x| x.inner()),
+		Some(Some(_))
+	);

 	// If the bucket has no other aliases, this is a true deletion.
 	// Otherwise, it is just an alias removal.
@@ -13,7 +13,7 @@ use crate::xml::to_xml_with_header;

 pub async fn handle_get_cors(ctx: ReqCtx) -> Result<Response<ResBody>, Error> {
 	let ReqCtx { bucket_params, .. } = ctx;
-	if let Some(cors) = bucket_params.cors_config.get() {
+	if let Some(cors) = bucket_params.cors_config.get().inner() {
 		let wc = CorsConfiguration {
 			xmlns: (),
 			cors_rules: cors
@@ -38,7 +38,7 @@ pub async fn handle_delete_cors(ctx: ReqCtx) -> Result<Response<ResBody>, Error>
 		mut bucket_params,
 		..
 	} = ctx;
-	bucket_params.cors_config.update(None);
+	bucket_params.cors_config.update(None.into());
 	garage
 		.bucket_table
 		.insert(&Bucket::present(bucket_id, bucket_params))
@@ -67,7 +67,7 @@ pub async fn handle_put_cors(

 	bucket_params
 		.cors_config
-		.update(Some(conf.into_garage_cors_config()?));
+		.update(Some(conf.into_garage_cors_config()?).into());
 	garage
 		.bucket_table
 		.insert(&Bucket::present(bucket_id, bucket_params))
@@ -83,8 +83,21 @@ pub async fn handle_delete_objects(
 				}
 				ret_deleted.push(s3_xml::Deleted {
 					key: s3_xml::Value(obj.key.clone()),
-					version_id: s3_xml::Value(hex::encode(deleted_version)),
-					delete_marker_version_id: s3_xml::Value(hex::encode(delete_marker_version)),
+					version_id: Some(s3_xml::Value(hex::encode(deleted_version))),
+					delete_marker_version_id: Some(s3_xml::Value(hex::encode(
+						delete_marker_version,
+					))),
+				});
+			}
+			Err(Error::NoSuchKey) => {
+				if cmd.quiet {
+					continue;
+				}
+				// Deleting a non-existent key is a success in S3
+				ret_deleted.push(s3_xml::Deleted {
+					key: s3_xml::Value(obj.key.clone()),
+					version_id: None,
+					delete_marker_version_id: None,
 				});
 			}
 			Err(e) => {
@@ -698,9 +698,14 @@ fn body_from_blocks_range(
 	// range, as well as their "true offset", which is their actual offset in the complete
 	// file (whereas block.offset designates the offset of the block WITHIN THE PART
 	// block.part_number, which is not the same in the case of a multipart upload)
+	// A version with no blocks yields no data, so the capacity hint must not index
+	// into an empty slice.
+	let capacity_block_size = all_blocks
+		.first()
+		.map_or(1024, |(_, b)| std::cmp::max(b.size, 1024));
 	let mut blocks: Vec<(VersionBlock, u64)> = Vec::with_capacity(std::cmp::min(
 		all_blocks.len(),
-		4 + ((end - begin) / std::cmp::max(all_blocks[0].1.size, 1024)) as usize,
+		4 + ((end - begin) / capacity_block_size) as usize,
 	));
 	let mut block_offset: u64 = 0;
 	for (_, b) in all_blocks.iter() {
@@ -14,7 +14,7 @@ use garage_model::bucket_table::Bucket;
 pub async fn handle_get_lifecycle(ctx: ReqCtx) -> Result<Response<ResBody>, Error> {
 	let ReqCtx { bucket_params, .. } = ctx;

-	if let Some(lifecycle) = bucket_params.lifecycle_config.get() {
+	if let Some(lifecycle) = bucket_params.lifecycle_config.get().inner() {
 		let wc = LifecycleConfiguration::from_garage_lifecycle_config(lifecycle);
 		let xml = to_xml_with_header(&wc)?;
 		Ok(Response::builder()
@@ -33,7 +33,7 @@ pub async fn handle_delete_lifecycle(ctx: ReqCtx) -> Result<Response<ResBody>, E
 		mut bucket_params,
 		..
 	} = ctx;
-	bucket_params.lifecycle_config.update(None);
+	bucket_params.lifecycle_config.update(None.into());
 	garage
 		.bucket_table
 		.insert(&Bucket::present(bucket_id, bucket_params))
@@ -62,7 +62,7 @@ pub async fn handle_put_lifecycle(
 		.validate_into_garage_lifecycle_config()
 		.ok_or_bad_request("Invalid lifecycle configuration")?;

-	bucket_params.lifecycle_config.update(Some(config));
+	bucket_params.lifecycle_config.update(Some(config).into());
 	garage
 		.bucket_table
 		.insert(&Bucket::present(bucket_id, bucket_params))
@@ -476,7 +476,13 @@ pub async fn handle_complete_multipart_upload(
 			size: total_size,
 			etag: etag.clone(),
 		},
-		final_version.blocks.items()[0].1.hash,
+		final_version
+			.blocks
+			.items()
+			.first()
+			.ok_or_internal_error("Multipart completion produced a final version with no blocks")?
+			.1
+			.hash,
 	));

 	let final_object = Object::new(*bucket_id, key.clone(), vec![object_version]);
@@ -83,10 +83,12 @@ pub async fn handle_post_object(
 	};

 	// Current part is file. Do some checks before handling to PutObject code
-	let key = params
-		.get("key")
-		.ok_or_bad_request("No key was provided")?
-		.to_str()?;
+	let key = std::str::from_utf8(
+		params
+			.get("key")
+			.ok_or_bad_request("No key was provided")?
+			.as_bytes(),
+	)?;
 	let policy = params
 		.get("policy")
 		.ok_or_bad_request("No policy was provided")?
@@ -121,7 +123,7 @@ pub async fn handle_post_object(
 		&bucket_params,
 		&Request::from_parts(head.clone(), empty_body::<Infallible>()),
 	)?
-	.cloned();
+	.map(|(rule, origin)| (rule.clone(), origin.to_string()));

 	let decoded_policy = BASE64_STANDARD
 		.decode(policy)
@@ -351,8 +353,8 @@ pub async fn handle_post_object(
 		}
 	};

-	if let Some(rule) = matching_cors_rule {
-		add_cors_headers(&mut resp, &rule)
+	if let Some((rule, origin)) = matching_cors_rule {
+		add_cors_headers(&mut resp, &rule, &origin)
 			.ok_or_internal_error("Invalid bucket CORS configuration")?;
 	}

@@ -473,12 +475,10 @@ where
 					))));
 				}
 			}
-			Poll::Ready(None) => {
-				if !self.length.contains(&self.read) {
-					return Poll::Ready(Some(Err(Error::bad_request(
-						"File size does not match policy",
-					))));
-				}
+			Poll::Ready(None) if !self.length.contains(&self.read) => {
+				return Poll::Ready(Some(Err(Error::bad_request(
+					"File size does not match policy",
+				))));
 			}
 			_ => {}
 		}
@@ -679,7 +679,10 @@ pub(crate) fn extract_metadata_headers(
 	];
 	for name in standard_header.iter() {
 		if let Some(value) = headers.get(name) {
-			ret.push((name.to_string(), value.to_str()?.to_string()));
+			ret.push((
+				name.to_string(),
+				std::str::from_utf8(value.as_bytes())?.to_string(),
+			));
 		}
 	}

@@ -16,7 +16,7 @@ pub const X_AMZ_WEBSITE_REDIRECT_LOCATION: HeaderName =

 pub async fn handle_get_website(ctx: ReqCtx) -> Result<Response<ResBody>, Error> {
 	let ReqCtx { bucket_params, .. } = ctx;
-	if let Some(website) = bucket_params.website_config.get() {
+	if let Some(website) = bucket_params.website_config.get().inner() {
 		let wc = WebsiteConfiguration {
 			xmlns: (),
 			error_document: website.error_document.as_ref().map(|v| Key {
@@ -54,7 +54,7 @@ pub async fn handle_delete_website(ctx: ReqCtx) -> Result<Response<ResBody>, Err
 		mut bucket_params,
 		..
 	} = ctx;
-	bucket_params.website_config.update(None);
+	bucket_params.website_config.update(None.into());
 	garage
 		.bucket_table
 		.insert(&Bucket::present(bucket_id, bucket_params))
@@ -83,7 +83,7 @@ pub async fn handle_put_website(

 	bucket_params
 		.website_config
-		.update(Some(conf.into_garage_website_config()?));
+		.update(Some(conf.into_garage_website_config()?).into());
 	garage
 		.bucket_table
 		.insert(&Bucket::present(bucket_id, bucket_params))
@@ -44,10 +44,13 @@ pub struct LocationConstraint {
 pub struct Deleted {
 	#[serde(rename = "Key")]
 	pub key: Value,
-	#[serde(rename = "VersionId")]
-	pub version_id: Value,
-	#[serde(rename = "DeleteMarkerVersionId")]
-	pub delete_marker_version_id: Value,
+	#[serde(rename = "VersionId", skip_serializing_if = "Option::is_none")]
+	pub version_id: Option<Value>,
+	#[serde(
+		rename = "DeleteMarkerVersionId",
+		skip_serializing_if = "Option::is_none"
+	)]
+	pub delete_marker_version_id: Option<Value>,
 }

 #[derive(Debug, Serialize, PartialEq, Eq)]
@@ -497,13 +500,13 @@ mod tests {
 			deleted: vec![
 				Deleted {
 					key: Value("a/plop".to_string()),
-					version_id: Value("qsdfjklm".to_string()),
-					delete_marker_version_id: Value("wxcvbn".to_string()),
+					version_id: Some(Value("qsdfjklm".to_string())),
+					delete_marker_version_id: Some(Value("wxcvbn".to_string())),
 				},
 				Deleted {
 					key: Value("b/plip".to_string()),
-					version_id: Value("1234".to_string()),
-					delete_marker_version_id: Value("4321".to_string()),
+					version_id: Some(Value("1234".to_string())),
+					delete_marker_version_id: Some(Value("4321".to_string())),
 				},
 			],
 			errors: vec![
@@ -1,6 +1,6 @@
 [package]
 name = "garage_block"
-version = "2.3.0"
+version = "2.4.0"
 authors = ["Alex Auvolat <alex@adnab.me>"]
 edition = "2018"
 license = "AGPL-3.0"
@@ -144,7 +144,7 @@ impl BlockManager {

 		// Open metadata tables
 		let rc = db
-			.open_tree("block_local_rc")
+			.open_typed_tree("block_local_rc")
 			.expect("Unable to open block_local_rc tree");
 		let rc = BlockRc::new(rc);

@@ -158,9 +158,9 @@ impl BlockManager {

 		let metrics = BlockManagerMetrics::new(
 			config.compression_level,
-			rc.rc_table.clone(),
-			resync.queue.clone(),
-			resync.errors.clone(),
+			rc.rc_table.untyped().clone(),
+			resync.queue.untyped().clone(),
+			resync.errors.untyped().clone(),
 			buffer_kb_semaphore.clone(),
 		);

@@ -224,7 +224,13 @@ impl BlockManager {
 				|p, tranquility| p.set_with(|x| x.tranquility = tranquility),
 			);
 			vars.register_ro(&self.scrub_persister, "scrub-last-completed", |p| {
-				p.get_with(|x| msec_to_rfc3339(x.time_last_complete_scrub))
+				p.get_with(|x| {
+					if x.time_last_complete_scrub == 0 {
+						"never".to_string()
+					} else {
+						msec_to_rfc3339(x.time_last_complete_scrub)
+					}
+				})
 			});
 			vars.register_ro(&self.scrub_persister, "scrub-next-run", |p| {
 				p.get_with(|x| msec_to_rfc3339(x.time_next_run_scrub))
@@ -443,9 +449,8 @@ impl BlockManager {
 		let mut blocks = Vec::with_capacity(self.resync.errors.approximate_len()?);
 		for ent in self.resync.errors.iter()? {
 			let (hash, cnt) = ent?;
-			let cnt = ErrorCounter::decode(&cnt);
 			blocks.push(BlockResyncErrorInfo {
-				hash: Hash::try_from(&hash).unwrap(),
+				hash,
 				refcount: 0,
 				error_count: cnt.errors,
 				last_try: cnt.last_try,
--- a/Show More
+++ b/Show More