fix: improve routing of keys starting with "/" (fix #1178) (#1465)

Path-style URLs of the form /bucket//key address an object whose key begins with "/". Two greedy uses of `trim_start_matches('/')` were collapsing these leading slashes away:

- `uri.path().trim_start_matches('/')` stripped all leading slashes from the raw path before any further parsing.
- `p.trim_start_matches('/')` stripped leading slashes from the remainder after `split_once('/')` had already consumed the bucket/key separator

The combined effect wath that `HEAD /bucket//` and `GET /bucket//` produced an empty key, which the router treated as bucket-level operations (HeadBucket -> 200 OK, and ListObjectsV2) instead of an object-level op (HeadObject/GetObject -> 404 NoSuchKey).

The fix is simple: Replace the first `trim_start_matches` with `strip_prefix` (to strip exactly one separator slash) and remove the second one entirely. Path-style and vhost-style requests are now consistent: a double slash in the URL correctly addresses a key whose name begins with "/".

Regression tests added for `HEAD //` and `GET //` requests in both request styles.

Fixes: #1464

---

Disclaimer: I'm not fluent in Rust and I did use an LLM to explain the code to me. All code was written by me.

I'm not sure whether the large `test_cases!` block in the `test_aws_doc_examples` function is the right place for my tests (it certainly was a convenient one).

Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/1465

CONTRIBUTING.md

@@ -0,0 +1,231 @@
+# Contributing to Garage
+
+## Policy on AI
+
+To ensure the quality of the codebase and documentation, the use of AI,
+including LLMs and coding agents, is strictly restricted in the following way:
+
+- AI **must not** be used to write documentation
+
+- **Do not** use AI to write bug reports, commit descriptions and pull request
+  messages
+
+- **Do not** use AI agents to make contributions to Garage, all contributions
+  must be led by a human that know what they are doing at all times
+
+- AI **may** be used for some tedious code generation tasks, limited to very
+  mechanical translations from API docs or boilerplate writing. The code
+  generated must be so simple as to make it clear that it cannot be covered by
+  copyright.
+
+You are free to make use of AI privately to explore the codebase and solve
+conceptual problems, but please restrain from copying the output from an LLM
+anywhere in your code or on the issue tracker, or from letting an agent edit
+the codebase directly.
+
+
+## Asking questions
+
+Read the documentation before asking questions.
+Do not use the issue tracker to ask questions about Garage.
+Questions asked on the issue tracker will be closed.
+
+Ask questions on the Matrix channel `#garage:deuxfleurs.fr` so that any
+community member can see your question and help you out.
+
+If you need in-depth support from the Garage developers specifically, write to
+`garagehq@deuxfleurs.fr`. Even if you do so, we do not commit to giving you an
+answer.
+
+
+## Reporting bugs
+
+When writing a bug report, use this checklist:
+
+- For bugs that can be reproduced:
+    - confirm that you are using the latest version of Garage and that the bug still exists in this version
+    - set the log level to debug using the `RUST_LOG=garage=debug` environment variable and reproduce the bug to get more verbose logs
+
+- Check whether there is already an open issue in the bug tracker. If so, your bug report is still valuable but please add it as a comment to the existing issue instead of opening a new one.
+
+- Collect as much information as possible:
+  - logs of the Garage daemon at the time the issue happened, including logs that show what was happening before the issue occurred
+  - the output of `garage status`
+  - the output of `garage stats -a`
+  - the output of `garage layout history`
+
+- Write a detailed bug report, including:
+  - a description of your cluster (number of nodes, hardware, operating system, networking, etc)
+  - a detailed description of what you did that led to the issue, including any code or command line that invoked a Garage API
+  - what you were expecting
+  - what actually happened, and how that's different from what you expected
+  - the information collected previously
+  - if possible, simple steps to help the developers reproduce the issue locally
+
+Bug reports that are imprecise or otherwise unactionable will be closed.
+
+
+## Suggesting new features
+
+Garage can be improved in many ways, but just suggesting a new feature does not mean we will implement it.
+Feature requests that may lead to an actual implementation are feature requests that:
+
+- are precise and actionable, i.e. include a precise description of the expected behavior and any necessary architectural details required for the implementation
+- are motivated by actual need from a variety of users
+
+Moreover, a certain number of features are defined as out-of-scope for Garage, including but not limited to:
+
+- extensions to the S3 API that are not present on AWS
+- features that require the implementation of a consensus algorithm
+- more generally, features that are incompatible with the architecture of Garage and its goal of staying simple
+
+Only feature requests in one of the following category may stay open in the issue tracker:
+
+- features that the Garage team wants to work on
+- features that are being actively worked on by an external contributor which is clearly identified
+- features that are easy to implement and could be an easy task for a new contributor that wants to get to know the codebase
+
+All other feature requests will be closed after a few months of inactivity, so as to keep the number of open issues to a manageable level.
+Feature requests that are clearly out of scope will be closed directly.
+
+
+## Improving the documentation
+
+An easy way to contribute to Garage which also adds a lot of value is to
+improve the documentation.  Make sure to write in clear technical English, and
+write unambiguously.  Documentation contributions are very appreciated if they
+are well-written.
+
+
+## For developers
+
+We welcome code contributions to Garage that adhere to our standards for quality:
+
+- Changes should be reviewed from a functional perspective to ensure that they work well with the existing codebase and do not introduce bugs or subtle issues.
+
+- You must have tested your contribution to make sure that it does what it says. The amount of testing required is proportional to the complexity of the change introduced.
+
+- Any new feature must be properly documented following existing practices (see below).
+
+- Unit tests should be included when relevant.
+
+- Contributions should pass basic lints for syntactic quality (`cargo fmt`, `cargo clippy`, `typos`).
+
+- Contributions should pass our CI test suite.
+
+- No user-facing breaking changes may be introduced between major releases.
+
+- No internal data model change may be introduced between major releases, to
+  ensure that Garage daemons with different minor/patch versions numbers can
+  work together in a cluster. For major releases, a proper migration path
+  should be implemented and tested thoroughly.
+
+Please follow up on your work when changes are requested, to avoid stale PRs.
+Do not take it personally if a Garage developer pushes directly to your branch
+to modify your contribution, as this might be necessary to get it merged
+faster.
+
+### Properly documenting your contribution
+
+#### Configuration options
+
+New configuration options should be documented in
+`doc/book/reference-manual/configuration.md`.  The documentation for a
+configuration option should be exhaustive. For instance, for choice options all
+choices should be listed explicitly with a precise description of their
+meaning.
+
+In terms of syntax, all configuration options should appear in three places:
+
+- in the example at the top, with an example value
+- in the index of all configuration options which is sorted by alphabetical order
+- in its dedicated subsection with full reference text
+
+#### CLI commands and command flags
+
+CLI commands are self-documented using the doc commends in the codebase.
+Make sure to write clear and precise comments for all options you are adding.
+
+#### S3 features
+
+If you implement new S3 features, make sure to update the compatibility matrix in `doc/book/reference-manual/s3-compatibility.md`.
+
+#### Admin API
+
+The admin API has an OpenAPI specification that is automatically generated
+using Utoipa, from a description of each endpoint that is given in
+`src/api/admin/openapi.rs` and a description of data structure schemas in
+`src/api/admin/api.rs`. The code in `openapi.rs` is only used to generate the
+OpenAPI specification document and not for the actual implementation in Garage,
+whereas structures defined in `api.rs` are also used for the implementation of
+API calls. Make sure to write good doc comments for all of these items so that
+the OpenAPI specification will be precise and accurate.
+
+An up-to-date version of the OpenAPI specification document should be kept in
+the repository in `doc/api/garage-admin-v2.json`. When you are making changes
+to the admin API, update this document with the following command:
+
+```
+cargo run -- admin-api-schema > doc/api/garage-admin-v2.json
+```
+
+
+## Garage team organization
+
+Alex (handle `lx`) is the lead developer and is responsible of ensuring the
+correctness of Garage and stability between version upgrades.
+
+The other maintainers are Trinity (handle `trinity-1686a`), Quentin (handle `quentin`) and Maximilien (handle `halfa`).
+
+Maximilien is responsible for coordinating effort on the Kubernetes integration / Helm chart.
+
+## Pull request merging criteria
+
+The following PRs should only be merged after review and approval from Alex:
+
+- PRs that introduce architectural changes, such as changes in the data model
+  or change in the coordination protocols between nodes
+
+- PRs that introduce changes on the format of data structures used for
+  persistent disk storage and internal cluster communication (RPC)
+
+- PRs that are suspected of introducing some kind of breakage or unexpected
+  behavior due to their complexity
+
+PRs that introduce breaking change for users but don't fall in one of the
+previous category should be discussed between maintainers to evaluate the
+impact on users when upgrading.  Alex's approval is not required to merge them
+as long as they are clearly identified as breaking in the PR title, and are
+properly merged in the branch for the next major version and not in the current
+main branch.
+
+All other PRs can be merged by any maintainer on their own, once they are
+confident that the quality standards defined in this document are respected
+before merging.
+
+## Merging strategy
+
+When merging PRs, maintainers should ensure that a Git commit is created by
+Forgejo that records the PR number, its title and its text in the commit
+message.  If a PR is fixing an issue, make sure that the issue number is
+included in the PR title as well.  This is to ensure that when releasing a new
+version of Garage, the changelog in the release notes can be properly
+constructed by reading the Git log since the last release.
+
+We also want to keep the history "almost linear" to facilitate the use of `git
+bisect` if it ever were necessary. This leaves the following two merging
+strategies:
+
+- For PRs that consist of many commits that should stay independent, the
+  "rebase and create merge commit" strategy should be used. The merge commit is
+  created automatically by Forgejo and saves the PR's number, title and text in
+  the commit message.
+
+- For PRs that consist of only one commit, or a few number of commits that can
+  be merged, the "create squash commit" strategy should be used. This way a
+  single commit will be created by Forgejo which also saves the PR's number,
+  title and text in the commit message.
+
+When cherry-picking commits from one branch to the other, a simple fast-forward
+merging strategy can be used if the commit message already references a PR
+number.

Cargo.toml

@@ -16,6 +16,7 @@ members = [
 	"src/garage",
 	"src/k2v-client",
 	"src/format-table",
+    "fuzz",
 ]
 
 default-members = ["src/garage"]
@@ -40,6 +41,7 @@ k2v-client = { version = "0.0.4", path = "src/k2v-client" }
 
 # External crates from crates.io
 arc-swap = "1.8"
+arbitrary = { version = "1.4.2"}
 argon2 = "0.5"
 async-trait = "0.1"
 backtrace = "0.3"
@@ -59,6 +61,7 @@ hmac = "0.12"
 itertools = "0.14"
 ipnet = "2.11"
 lazy_static = "1.5"
+libfuzzer-sys = "0.4"
 md-5 = "0.10"
 mktemp = "0.5"
 nix = { version = "0.31", default-features = false, features = ["fs"] }
@@ -113,7 +116,7 @@ kube = { version = "3.0", default-features = false, features = [
 ] }
 schemars = "1.2"
 reqwest = { version = "0.13", default-features = false, features = [
-    "rustls",
+    "rustls-no-provider",
     "json",
 ] }
 

Dockerfile

@@ -4,4 +4,6 @@ ENV RUST_BACKTRACE=1
 ENV RUST_LOG=garage=info
 
 COPY result/bin/garage /
-CMD [ "/garage", "server"]
+
+ENTRYPOINT ["/garage"]
+CMD ["server"]

SECURITY.md

@@ -0,0 +1,14 @@
+# Security Reporting
+
+If you wish to report responsibly a security vulnerability about Garage, we ask that you follow the following process.
+
+Please report each security vulnerabilities by filling out the following template:
+
+- PROJECT: A URL to the code repository containing the vulnerable version - be reminded that the source of truth is at https://git.deuxfleurs.fr/deuxfleurs/garage
+- PUBLIC: Please let us know if this vulnerability has been made or discussed publicly already, and if so, please let us know where.
+- DESCRIPTION: Please provide precise description of the security vulnerability you have found with as much information as you are able and willing to provide.
+
+Please send the above info, along with any other information you feel is pertinent by emailing the core team at: garagehq@deuxfleurs.fr
+
+The Garage Core Team will let you know within a few weeks whether or not your report has been accepted or rejected.
+We ask that you please keep the report confidential until we have either responded or made a public announcement.

doc/api/garage-admin-v2.json

@@ -1797,6 +1797,17 @@
               "type": "string"
             },
             "description": "Plain-text information about the layout computation\n(do not try to parse this)"
+          },
+          "statistics": {
+            "oneOf": [
+              {
+                "type": "null"
+              },
+              {
+                "$ref": "#/components/schemas/ComputationStat",
+                "description": "Structured statistics about the layout computation"
+              }
+            ]
           }
         }
       },
@@ -2119,6 +2130,180 @@
           "Historical"
         ]
       },
+      "ComputationStat": {
+        "type": "object",
+        "required": [
+          "replicationFactor",
+          "effectiveZoneRedundancy",
+          "partitionSize",
+          "lowPartitionSize",
+          "usableCapacity",
+          "totalCapacity",
+          "effectiveCapacity",
+          "lowUsableCapacity",
+          "zones"
+        ],
+        "properties": {
+          "effectiveCapacity": {
+            "type": "integer",
+            "format": "int64",
+            "description": "The final effective capacity of the cluster, accounting for replication",
+            "minimum": 0
+          },
+          "effectiveZoneRedundancy": {
+            "type": "integer",
+            "description": "The zone redundancy factor achieved by this layout",
+            "minimum": 0
+          },
+          "lowPartitionSize": {
+            "type": "boolean",
+            "description": "Warning flag indicating when partitions are very small"
+          },
+          "lowUsableCapacity": {
+            "type": "boolean",
+            "description": "Warning flag indicating that the raw node capacity could not be used\neffectively"
+          },
+          "partitionSize": {
+            "type": "integer",
+            "format": "int64",
+            "description": "The size of a partition, in bytes",
+            "minimum": 0
+          },
+          "previousPartitionSize": {
+            "type": [
+              "integer",
+              "null"
+            ],
+            "format": "int64",
+            "description": "The size of a partition, in bytes, in the previous layout",
+            "minimum": 0
+          },
+          "replicationFactor": {
+            "type": "integer",
+            "description": "The cluster's replication factor",
+            "minimum": 0
+          },
+          "totalCapacity": {
+            "type": "integer",
+            "format": "int64",
+            "description": "The total raw capacity of nodes",
+            "minimum": 0
+          },
+          "totalMovedPartitions": {
+            "type": [
+              "integer",
+              "null"
+            ],
+            "description": "The total number of partitions that will be moved to a new storage node",
+            "minimum": 0
+          },
+          "usableCapacity": {
+            "type": "integer",
+            "format": "int64",
+            "description": "The portion of total raw node capacity that is used by partitions",
+            "minimum": 0
+          },
+          "zones": {
+            "type": "array",
+            "items": {
+              "$ref": "#/components/schemas/ComputationStatZone"
+            },
+            "description": "Per-zone storage statistics"
+          }
+        }
+      },
+      "ComputationStatNode": {
+        "type": "object",
+        "required": [
+          "id",
+          "tags",
+          "storedPartitions",
+          "newPartitions",
+          "totalCapacity",
+          "usableCapacity"
+        ],
+        "properties": {
+          "id": {
+            "type": "string",
+            "description": "The node's ID"
+          },
+          "newPartitions": {
+            "type": "integer",
+            "description": "The number of partitions that are newly replicated on this node",
+            "minimum": 0
+          },
+          "storedPartitions": {
+            "type": "integer",
+            "description": "The number of partitions that are replicated on this node",
+            "minimum": 0
+          },
+          "tags": {
+            "type": "array",
+            "items": {
+              "type": "string"
+            },
+            "description": "The node's tags as defined in the layout"
+          },
+          "totalCapacity": {
+            "type": "integer",
+            "format": "int64",
+            "description": "The node's raw capacity",
+            "minimum": 0
+          },
+          "usableCapacity": {
+            "type": "integer",
+            "format": "int64",
+            "description": "The portion of the node's raw capacity that is used by partitions it stores",
+            "minimum": 0
+          }
+        }
+      },
+      "ComputationStatZone": {
+        "type": "object",
+        "required": [
+          "name",
+          "nodes",
+          "totalReplicatedPartitions",
+          "uniquePartitions",
+          "totalCapacity",
+          "usableCapacity"
+        ],
+        "properties": {
+          "name": {
+            "type": "string",
+            "description": "The name of the zone"
+          },
+          "nodes": {
+            "type": "array",
+            "items": {
+              "$ref": "#/components/schemas/ComputationStatNode"
+            },
+            "description": "Per-node storage statistics for nodes in this zone"
+          },
+          "totalCapacity": {
+            "type": "integer",
+            "format": "int64",
+            "description": "The total raw capacity of nodes in this zone",
+            "minimum": 0
+          },
+          "totalReplicatedPartitions": {
+            "type": "integer",
+            "description": "The total number of partition replicas in this zone",
+            "minimum": 0
+          },
+          "uniquePartitions": {
+            "type": "integer",
+            "description": "The number of unique partitions that have at least one replica in this zone",
+            "minimum": 0
+          },
+          "usableCapacity": {
+            "type": "integer",
+            "format": "int64",
+            "description": "The used portion of the raw capacity of nodes in this zones",
+            "minimum": 0
+          }
+        }
+      },
       "ConnectClusterNodesRequest": {
         "type": "array",
         "items": {
@@ -4221,6 +4406,17 @@
               "newLayout": {
                 "$ref": "#/components/schemas/GetClusterLayoutResponse",
                 "description": "Details about the new cluster layout"
+              },
+              "statistics": {
+                "oneOf": [
+                  {
+                    "type": "null"
+                  },
+                  {
+                    "$ref": "#/components/schemas/ComputationStat",
+                    "description": "Structured statistics about the layout computation"
+                  }
+                ]
               }
             }
           }

doc/book/cookbook/real-world.md

@@ -213,7 +213,12 @@ If your configuration file is at `/etc/garage.toml`, the `garage` binary should
 You can also use an alias as follows to use the Garage binary inside your docker container:
 
 ```bash
+# garage 3.x, we have an entrypoint and you can use
+alias garage="docker exec -ti <container name>"
+
+# For garage 2.x, you need to specify the absolute path to binary
 alias garage="docker exec -ti <container name> /garage"
+
 ```
 
 You can test your `garage` CLI utility by running a simple command such as:

doc/book/quick-start/_index.md

@@ -178,8 +178,11 @@ garage status
 
 If you are running Garage in a Docker container, you can use the following command instead:
 
+NOTE: Garage 3.x uses docker `ENTRYPOINT` and it's easier to use,
+while garage 2.x does not and you need to specify path `/garage`
+
 ```bash
-docker exec garage-container /garage status
+docker exec garage-container status
 ```
 
 This should show something like this:
@@ -320,7 +323,7 @@ If you are running Garage in a Docker container, you can set the following alias
 to provide a fake `garage`command that uses the Garage binary inside your container:
 
 ```bash
-alias garage="docker exec -ti <container name> /garage"
+alias garage="docker exec -ti <container name>"
 ```
 
 You can test that your `garage` CLI is configured correctly by running a basic command such as `garage status`.

doc/book/reference-manual/admin-api.md

@@ -182,15 +182,15 @@ content-type: text/plain; version=0.0.4
 content-length: 12145
 date: Tue, 08 Aug 2023 07:25:05 GMT
 
-# HELP api_admin_error_counter Number of API calls to the various Admin API endpoints that resulted in errors
+# HELP garage_api_admin_error_count Number of API calls to the various Admin API endpoints that resulted in errors
-# TYPE api_admin_error_counter counter
+# TYPE garage_api_admin_error_count counter
-api_admin_error_counter{api_endpoint="CheckWebsiteEnabled",status_code="400"} 1
+garage_api_admin_error_count{api_endpoint="CheckWebsiteEnabled",status_code="400"} 1
-api_admin_error_counter{api_endpoint="CheckWebsiteEnabled",status_code="404"} 3
+garage_api_admin_error_count{api_endpoint="CheckWebsiteEnabled",status_code="404"} 3
-# HELP api_admin_request_counter Number of API calls to the various Admin API endpoints
+# HELP garage_api_admin_request_count Number of API calls to the various Admin API endpoints
-# TYPE api_admin_request_counter counter
+# TYPE garage_api_admin_request_count counter
-api_admin_request_counter{api_endpoint="CheckWebsiteEnabled"} 7
+garage_api_admin_request_count{api_endpoint="CheckWebsiteEnabled"} 7
-api_admin_request_counter{api_endpoint="Health"} 3
+garage_api_admin_request_count{api_endpoint="Health"} 3
-# HELP api_admin_request_duration Duration of API calls to the various Admin API endpoints
+# HELP garage_api_admin_request_duration Duration of API calls to the various Admin API endpoints
 ...
 ```
 

doc/book/reference-manual/monitoring.md

@@ -40,146 +40,146 @@ garage_local_disk_total{volume="metadata"} 763063566336
 
 ### Cluster health status metrics
 
-#### `cluster_healthy` (gauge)
+#### `garage_cluster_healthy` (gauge)
 
 Whether all storage nodes are connected (0 or 1)
 
 ```
-cluster_healthy 0
+garage_cluster_healthy 0
 ```
 
-#### `cluster_available` (gauge)
+#### `garage_cluster_available` (gauge)
 
 Whether all requests can be served, even if some storage nodes are disconnected
 
 ```
-cluster_available 1
+garage_cluster_available 1
 ```
 
-#### `cluster_connected_nodes` (gauge)
+#### `garage_cluster_connected_nodes` (gauge)
 
 Number of nodes currently connected
 
 ```
-cluster_connected_nodes 3
+garage_cluster_connected_nodes 3
 ```
 
-#### `cluster_known_nodes` (gauge)
+#### `garage_cluster_known_nodes` (gauge)
 
 Number of nodes already seen once in the cluster
 
 ```
-cluster_known_nodes 3
+garage_cluster_known_nodes 3
 ```
 
-#### `cluster_layout_node_connected` (gauge)
+#### `garage_cluster_layout_node_connected` (gauge)
 
 Connection status for individual nodes of the cluster layout
 
 ```
-cluster_layout_node_connected{id="62b218d848e86a64",role_capacity="1000000000",role_gateway="0",role_zone="dc1"} 1
+garage_cluster_layout_node_connected{id="62b218d848e86a64",role_capacity="1000000000",role_gateway="0",role_zone="dc1"} 1
-cluster_layout_node_connected{id="a11c7cf18af29737",role_capacity="1000000000",role_gateway="0",role_zone="dc1"} 0
+garage_cluster_layout_node_connected{id="a11c7cf18af29737",role_capacity="1000000000",role_gateway="0",role_zone="dc1"} 0
-cluster_layout_node_connected{id="a235ac7695e0c54d",role_capacity="1000000000",role_gateway="0",role_zone="dc1"} 1
+garage_cluster_layout_node_connected{id="a235ac7695e0c54d",role_capacity="1000000000",role_gateway="0",role_zone="dc1"} 1
-cluster_layout_node_connected{id="b10c110e4e854e5a",role_capacity="1000000000",role_gateway="0",role_zone="dc1"} 1
+garage_cluster_layout_node_connected{id="b10c110e4e854e5a",role_capacity="1000000000",role_gateway="0",role_zone="dc1"} 1
 ```
 
-#### `cluster_layout_node_disconnected_time` (gauge)
+#### `garage_cluster_layout_node_disconnected_time` (gauge)
 
 Time (in seconds) since last connection to individual nodes of the cluster layout
 
 ```
-cluster_layout_node_disconnected_time{id="62b218d848e86a64",role_capacity="1000000000",role_gateway="0",role_zone="dc1"} 0
+garage_cluster_layout_node_disconnected_time{id="62b218d848e86a64",role_capacity="1000000000",role_gateway="0",role_zone="dc1"} 0
-cluster_layout_node_disconnected_time{id="a235ac7695e0c54d",role_capacity="1000000000",role_gateway="0",role_zone="dc1"} 0
+garage_cluster_layout_node_disconnected_time{id="a235ac7695e0c54d",role_capacity="1000000000",role_gateway="0",role_zone="dc1"} 0
-cluster_layout_node_disconnected_time{id="b10c110e4e854e5a",role_capacity="1000000000",role_gateway="0",role_zone="dc1"} 0
+garage_cluster_layout_node_disconnected_time{id="b10c110e4e854e5a",role_capacity="1000000000",role_gateway="0",role_zone="dc1"} 0
 ```
 
-#### `cluster_storage_nodes` (gauge)
+#### `garage_cluster_storage_nodes` (gauge)
 
 Number of storage nodes declared in the current layout
 
 ```
-cluster_storage_nodes 4
+garage_cluster_storage_nodes 4
 ```
 
-#### `cluster_storage_nodes_ok` (gauge)
+#### `garage_cluster_storage_nodes_ok` (gauge)
 
 Number of storage nodes currently connected
 
 ```
-cluster_storage_nodes_ok 3
+garage_cluster_storage_nodes_ok 3
 ```
 
-#### `cluster_partitions` (gauge)
+#### `garage_cluster_partitions` (gauge)
 
 Number of partitions in the layout (this is always 256)
 
 ```
-cluster_partitions 256
+garage_cluster_partitions 256
 ```
 
-#### `cluster_partitions_all_ok` (gauge)
+#### `garage_cluster_partitions_all_ok` (gauge)
 
 Number of partitions for which all storage nodes are connected
 
 ```
-cluster_partitions_all_ok 64
+garage_cluster_partitions_all_ok 64
 ```
 
-#### `cluster_partitions_quorum` (gauge)
+#### `garage_cluster_partitions_quorum` (gauge)
 
 Number of partitions for which we have a quorum of connected nodes and all requests can be served
 
 ```
-cluster_partitions_quorum 256
+garage_cluster_partitions_quorum 256
 ```
 
 ### Metrics of the API endpoints
 
-#### `api_admin_request_counter` (counter)
+#### `garage_api_admin_request_count` (counter)
 
 Counts the number of requests to a given endpoint of the administration API. Example:
 
 ```
-api_admin_request_counter{api_endpoint="Metrics"} 127041
+garage_api_admin_request_count{api_endpoint="Metrics"} 127041
 ```
 
-#### `api_admin_request_duration` (histogram)
+#### `garage_api_admin_request_duration` (histogram)
 
 Evaluates the duration of API calls to the various administration API endpoint. Example:
 
 ```
-api_admin_request_duration_bucket{api_endpoint="Metrics",le="0.5"} 127041
+garage_api_admin_request_duration_bucket{api_endpoint="Metrics",le="0.5"} 127041
-api_admin_request_duration_sum{api_endpoint="Metrics"} 605.250344830999
+garage_api_admin_request_duration_sum{api_endpoint="Metrics"} 605.250344830999
-api_admin_request_duration_count{api_endpoint="Metrics"} 127041
+garage_api_admin_request_duration_count{api_endpoint="Metrics"} 127041
 ```
 
-#### `api_s3_request_counter` (counter)
+#### `garage_api_s3_request_count` (counter)
 
 Counts the number of requests to a given endpoint of the S3 API. Example:
 
 ```
-api_s3_request_counter{api_endpoint="CreateMultipartUpload"} 1
+garage_api_s3_request_count{api_endpoint="CreateMultipartUpload"} 1
 ```
 
-#### `api_s3_error_counter` (counter)
+#### `garage_api_s3_error_count` (counter)
 
 Counts the number of requests to a given endpoint of the S3 API that returned an error. Example:
 
 ```
-api_s3_error_counter{api_endpoint="GetObject",status_code="404"} 39
+garage_api_s3_error_count{api_endpoint="GetObject",status_code="404"} 39
 ```
 
-#### `api_s3_request_duration` (histogram)
+#### `garage_api_s3_request_duration` (histogram)
 
 Evaluates the duration of API calls to the various S3 API endpoints. Example:
 
 ```
-api_s3_request_duration_bucket{api_endpoint="CreateMultipartUpload",le="0.5"} 1
+garage_api_s3_request_duration_bucket{api_endpoint="CreateMultipartUpload",le="0.5"} 1
-api_s3_request_duration_sum{api_endpoint="CreateMultipartUpload"} 0.046340762
+garage_api_s3_request_duration_sum{api_endpoint="CreateMultipartUpload"} 0.046340762
-api_s3_request_duration_count{api_endpoint="CreateMultipartUpload"} 1
+garage_api_s3_request_duration_count{api_endpoint="CreateMultipartUpload"} 1
 ```
 
-#### `api_k2v_request_counter` (counter), `api_k2v_error_counter` (counter), `api_k2v_error_duration` (histogram)
+#### `garage_api_k2v_request_count` (counter), `garage_api_k2v_error_count` (counter), `garage_api_k2v_error_duration` (histogram)
 
 Same as for S3, for the K2V API.
 
@@ -187,45 +187,45 @@ Same as for S3, for the K2V API.
 ### Metrics of the Web endpoint
 
 
-#### `web_request_counter` (counter)
+#### `garage_web_request_count` (counter)
 
 Number of requests to the web endpoint
 
 ```
-web_request_counter{method="GET"} 80
+garage_web_request_count{method="GET"} 80
 ```
 
-#### `web_request_duration` (histogram)
+#### `garage_web_request_duration` (histogram)
 
 Duration of requests to the web endpoint
 
 ```
-web_request_duration_bucket{method="GET",le="0.5"} 80
+garage_web_request_duration_bucket{method="GET",le="0.5"} 80
-web_request_duration_sum{method="GET"} 1.0528433229999998
+garage_web_request_duration_sum{method="GET"} 1.0528433229999998
-web_request_duration_count{method="GET"} 80
+garage_web_request_duration_count{method="GET"} 80
 ```
 
-#### `web_error_counter` (counter)
+#### `garage_web_error_count` (counter)
 
 Number of requests to the web endpoint resulting in errors
 
 ```
-web_error_counter{method="GET",status_code="404 Not Found"} 64
+garage_web_error_count{method="GET",status_code="404 Not Found"} 64
 ```
 
 
 ### Metrics of the data block manager
 
-#### `block_bytes_read`, `block_bytes_written` (counter)
+#### `garage_block_bytes_read`, `garage_block_bytes_written` (counter)
 
 Number of bytes read/written to/from disk in the data storage directory.
 
 ```
-block_bytes_read 120586322022
+garage_block_bytes_read 120586322022
-block_bytes_written 3386618077
+garage_block_bytes_written 3386618077
 ```
 
-#### `block_ram_buffer_free_kb` (gauge)
+#### `garage_block_ram_buffer_free_kb` (gauge)
 
 Kibibytes available for buffering blocks that have to be sent to remote nodes.
 When clients send too much data to this node and a storage node is not receiving
@@ -233,170 +233,168 @@ data fast enough due to slower network conditions, this will decrease down to
 zero and backpressure will be applied.
 
 ```
-block_ram_buffer_free_kb 219829
+garage_block_ram_buffer_free_kb 219829
 ```
 
-#### `block_compression_level` (counter)
+#### `garage_block_compression_level` (counter)
 
 Exposes the block compression level configured for the Garage node.
 
 ```
-block_compression_level 3
+garage_block_compression_level 3
 ```
 
-#### `block_read_duration`, `block_write_duration` (histograms)
+#### `garage_block_read_duration`, `garage_block_write_duration` (histograms)
 
 Evaluates the duration of the reading/writing of individual data blocks in the data storage directory.
 
 ```
-block_read_duration_bucket{le="0.5"} 169229
+garage_block_read_duration_bucket{le="0.5"} 169229
-block_read_duration_sum 2761.6902550310056
+garage_block_read_duration_sum 2761.6902550310056
-block_read_duration_count 169240
+garage_block_read_duration_count 169240
-block_write_duration_bucket{le="0.5"} 3559
+garage_block_write_duration_bucket{le="0.5"} 3559
-block_write_duration_sum 195.59170078500006
+garage_block_write_duration_sum 195.59170078500006
-block_write_duration_count 3571
+garage_block_write_duration_count 3571
 ```
 
-#### `block_delete_counter` (counter)
+#### `garage_block_delete_count` (counter)
 
 Counts the number of data blocks that have been deleted from storage.
 
 ```
-block_delete_counter 122
+garage_block_delete_count 122
 ```
 
-#### `block_resync_counter` (counter), `block_resync_duration` (histogram)
+#### `garage_block_resync_count` (counter), `garage_block_resync_duration` (histogram)
 
 Counts the number of resync operations the node has executed, and evaluates their duration.
 
 ```
-block_resync_counter 308897
+garage_block_resync_count 308897
-block_resync_duration_bucket{le="0.5"} 308892
+garage_block_resync_duration_bucket{le="0.5"} 308892
-block_resync_duration_sum 139.64204196100016
+garage_block_resync_duration_sum 139.64204196100016
-block_resync_duration_count 308897
+garage_block_resync_duration_count 308897
 ```
 
-#### `block_resync_queue_length` (gauge)
+#### `garage_block_resync_queue_length` (gauge)
 
 The number of block hashes currently queued for a resync.
 This is normal to be nonzero for long periods of time.
 
 ```
-block_resync_queue_length 0
+garage_block_resync_queue_length 0
 ```
 
-#### `block_resync_errored_blocks` (gauge)
+#### `garage_block_resync_errored_blocks` (gauge)
 
 The number of block hashes that we were unable to resync last time we tried.
 **THIS SHOULD BE ZERO, OR FALL BACK TO ZERO RAPIDLY, IN A HEALTHY CLUSTER.**
 Persistent nonzero values indicate that some data is likely to be lost.
 
 ```
-block_resync_errored_blocks 0
+garage_block_resync_errored_blocks 0
 ```
 
 
 ### Metrics related to RPCs (remote procedure calls) between nodes
 
-#### `rpc_netapp_request_counter` (counter)
+#### `garage_rpc_netapp_request_count` (counter)
 
 Number of RPC requests emitted
 
 ```
-rpc_request_counter{from="<this node>",rpc_endpoint="garage_block/manager.rs/Rpc",to="<remote node>"} 176
+garage_rpc_request_count{from="<this node>",rpc_endpoint="garage_block/manager.rs/Rpc",to="<remote node>"} 176
 ```
 
-#### `rpc_netapp_error_counter` (counter)
+#### `garage_rpc_netapp_error_count` (counter)
 
 Number of communication errors (errors in the Netapp library, generally due to disconnected nodes)
 
 ```
-rpc_netapp_error_counter{from="<this node>",rpc_endpoint="garage_block/manager.rs/Rpc",to="<remote node>"} 354
+garage_rpc_netapp_error_count{from="<this node>",rpc_endpoint="garage_block/manager.rs/Rpc",to="<remote node>"} 354
 ```
 
-#### `rpc_timeout_counter` (counter)
+#### `garage_rpc_timeout_count` (counter)
 
 Number of RPC timeouts, should be close to zero in a healthy cluster.
 
 ```
-rpc_timeout_counter{from="<this node>",rpc_endpoint="garage_rpc/membership.rs/SystemRpc",to="<remote node>"} 1
+garage_rpc_timeout_count{from="<this node>",rpc_endpoint="garage_rpc/membership.rs/SystemRpc",to="<remote node>"} 1
 ```
 
-#### `rpc_duration` (histogram)
+#### `garage_rpc_duration` (histogram)
 
 The duration of internal RPC calls between Garage nodes.
 
 ```
-rpc_duration_bucket{from="<this node>",rpc_endpoint="garage_block/manager.rs/Rpc",to="<remote node>",le="0.5"} 166
+garage_rpc_duration_bucket{from="<this node>",rpc_endpoint="garage_block/manager.rs/Rpc",to="<remote node>",le="0.5"} 166
-rpc_duration_sum{from="<this node>",rpc_endpoint="garage_block/manager.rs/Rpc",to="<remote node>"} 35.172253716
+garage_rpc_duration_sum{from="<this node>",rpc_endpoint="garage_block/manager.rs/Rpc",to="<remote node>"} 35.172253716
-rpc_duration_count{from="<this node>",rpc_endpoint="garage_block/manager.rs/Rpc",to="<remote node>"} 174
+garage_rpc_duration_count{from="<this node>",rpc_endpoint="garage_block/manager.rs/Rpc",to="<remote node>"} 174
 ```
 
 
 ### Metrics of the metadata table manager
 
-#### `table_gc_todo_queue_length` (gauge)
+#### `garage_table_gc_todo_queue_length` (gauge)
 
 Table garbage collector TODO queue length
 
 ```
-table_gc_todo_queue_length{table_name="block_ref"} 0
+garage_table_gc_todo_queue_length{table_name="block_ref"} 0
 ```
 
-#### `table_get_request_counter` (counter), `table_get_request_duration` (histogram)
+#### `garage_table_get_request_count` (counter), `garage_table_get_request_duration` (histogram)
 
 Number of get/get_range requests internally made on each table, and their duration.
 
 ```
-table_get_request_counter{table_name="bucket_alias"} 315
+garage_table_get_request_count{table_name="bucket_alias"} 315
-table_get_request_duration_bucket{table_name="bucket_alias",le="0.5"} 315
+garage_table_get_request_duration_bucket{table_name="bucket_alias",le="0.5"} 315
-table_get_request_duration_sum{table_name="bucket_alias"} 0.048509778000000024
+garage_table_get_request_duration_sum{table_name="bucket_alias"} 0.048509778000000024
-table_get_request_duration_count{table_name="bucket_alias"} 315
+garage_table_get_request_duration_count{table_name="bucket_alias"} 315
 ```
 
 
-#### `table_put_request_counter` (counter), `table_put_request_duration` (histogram)
+#### `garage_table_put_request_count` (counter), `garage_table_put_request_duration` (histogram)
 
 Number of insert/insert_many requests internally made on this table, and their duration
 
 ```
-table_put_request_counter{table_name="block_ref"} 677
+garage_table_put_request_count{table_name="block_ref"} 677
-table_put_request_duration_bucket{table_name="block_ref",le="0.5"} 677
+garage_table_put_request_duration_bucket{table_name="block_ref",le="0.5"} 677
-table_put_request_duration_sum{table_name="block_ref"} 61.617528636
+garage_table_put_request_duration_sum{table_name="block_ref"} 61.617528636
-table_put_request_duration_count{table_name="block_ref"} 677
+garage_table_put_request_duration_count{table_name="block_ref"} 677
 ```
 
-#### `table_internal_delete_counter` (counter)
+#### `garage_table_internal_delete_count` (counter)
 
 Number of value deletions in the tree (due to GC or repartitioning)
 
 ```
-table_internal_delete_counter{table_name="block_ref"} 2296
+garage_table_internal_delete_count{table_name="block_ref"} 2296
 ```
 
-#### `table_internal_update_counter` (counter)
+#### `garage_table_internal_update_count` (counter)
 
 Number of value updates where the value actually changes (includes creation of new key and update of existing key)
 
 ```
-table_internal_update_counter{table_name="block_ref"} 5996
+garage_table_internal_update_count{table_name="block_ref"} 5996
 ```
 
-#### `table_merkle_updater_todo_queue_length` (gauge)
+#### `garage_table_merkle_updater_todo_queue_length` (gauge)
 
 Merkle tree updater TODO queue length (should fall to zero rapidly)
 
 ```
-table_merkle_updater_todo_queue_length{table_name="block_ref"} 0
+garage_table_merkle_updater_todo_queue_length{table_name="block_ref"} 0
 ```
 
-#### `table_sync_items_received`, `table_sync_items_sent` (counters)
+#### `garage_table_sync_items_received`, `garage_table_sync_items_sent` (counters)
 
 Number of data items sent to/received from other nodes during resync procedures
 
 ```
-table_sync_items_received{from="<remote node>",table_name="bucket_v2"} 3
+garage_table_sync_items_received{from="<remote node>",table_name="bucket_v2"} 3
-table_sync_items_sent{table_name="block_ref",to="<remote node>"} 2
+garage_table_sync_items_sent{table_name="block_ref",to="<remote node>"} 2
 ```
-
-

flake.lock

@@ -81,17 +81,17 @@
         ]
       },
       "locked": {
-        "lastModified": 1763952169,
+        "lastModified": 1776914043,
-        "narHash": "sha256-+PeDBD8P+NKauH+w7eO/QWCIp8Cx4mCfWnh9sJmy9CM=",
+        "narHash": "sha256-qug5r56yW1qOsjSI99l3Jm15JNT9CvS2otkXNRNtrPI=",
         "owner": "oxalica",
         "repo": "rust-overlay",
-        "rev": "ab726555a9a72e6dc80649809147823a813fa95b",
+        "rev": "2d35c4358d7de3a0e606a6e8b27925d981c01cc3",
         "type": "github"
       },
       "original": {
         "owner": "oxalica",
         "repo": "rust-overlay",
-        "rev": "ab726555a9a72e6dc80649809147823a813fa95b",
+        "rev": "2d35c4358d7de3a0e606a6e8b27925d981c01cc3",
         "type": "github"
       }
     },

flake.nix

@@ -6,9 +6,9 @@
   inputs.nixpkgs.url =
     "github:NixOS/nixpkgs/cfe2c7d5b5d3032862254e68c37a6576b633d632";
 
-  # Rust overlay as of 2025-11-24
+  # Rust overlay as of 2026-04-23
   inputs.rust-overlay.url =
-  "github:oxalica/rust-overlay/ab726555a9a72e6dc80649809147823a813fa95b";
+  "github:oxalica/rust-overlay/2d35c4358d7de3a0e606a6e8b27925d981c01cc3";
   inputs.rust-overlay.inputs.nixpkgs.follows = "nixpkgs";
 
   # Crane as of 2025-01-24
@@ -95,6 +95,14 @@
                 killall
               ];
             };
+
+            # dev shell for fuzzing
+            fuzz = pkgs.mkShell {
+              buildInputs = with pkgs; [
+                targets.toolchainNightly
+                cargo-fuzz
+              ];
+            };
           };
       });
 }

fuzz/.gitignore

@@ -0,0 +1,4 @@
+target
+corpus
+artifacts
+coverage

fuzz/Cargo.toml

@@ -0,0 +1,73 @@
+[package]
+name = "garage-fuzz"
+version = "0.0.0"
+publish = false
+edition = "2018"
+
+[package.metadata]
+cargo-fuzz = true
+
+[dependencies]
+arbitrary = { workspace = true, features = ["derive"]}
+libfuzzer-sys = { workspace = true }
+
+garage_db.workspace = true
+garage_table.workspace = true
+garage_util.workspace = true
+garage_model = { workspace = true, default-features = false, features = ["arbitrary", "k2v"] }
+
+[[bin]]
+name = "version_crdt"
+path = "fuzz_targets/version_crdt.rs"
+test = false
+doc = false
+bench = false
+
+[[bin]]
+name = "mpu_crdt"
+path = "fuzz_targets/mpu_crdt.rs"
+test = false
+doc = false
+bench = false
+
+[[bin]]
+name = "bucket_crdt"
+path = "fuzz_targets/bucket_crdt.rs"
+test = false
+doc = false
+bench = false
+
+[[bin]]
+name = "block_ref_crdt"
+path = "fuzz_targets/block_ref_crdt.rs"
+test = false
+doc = false
+bench = false
+
+[[bin]]
+name = "admin_api_token_crdt"
+path = "fuzz_targets/admin_api_token_crdt.rs"
+test = false
+doc = false
+bench = false
+
+[[bin]]
+name = "key_crdt"
+path = "fuzz_targets/key_crdt.rs"
+test = false
+doc = false
+bench = false
+
+[[bin]]
+name = "bucket_alias_crdt"
+path = "fuzz_targets/bucket_alias_crdt.rs"
+test = false
+doc = false
+bench = false
+
+[[bin]]
+name = "k2v_item_crdt"
+path = "fuzz_targets/k2v_item_crdt.rs"
+test = false
+doc = false
+bench = false

fuzz/README.md

@@ -0,0 +1,11 @@
+# Fuzzing
+
+## Setup
+
+Install cargo fuzz: `cargo install cargo-fuzz`
+
+## Launch
+
+Run `cargo fuzz run <fuzz_target>` where `<fuzz_target>` is the name (without extension) of one of the `.rs` files in the `fuzz_targets` directory.
+
+If you launch the command outside of the fuzz directory, you need to force the nightly toolchain with `cargo +nightly`.

fuzz/fuzz_targets/admin_api_token_crdt.rs

@@ -0,0 +1,38 @@
+#![no_main]
+
+use garage_fuzz::check_crdt_laws;
+use garage_model::admin_token_table::{AdminApiToken, AdminApiTokenParams, AdminApiTokenScope};
+use garage_model::permission::ExpirationTime;
+use garage_util::crdt;
+use libfuzzer_sys::fuzz_target;
+
+type Input = (
+	bool,
+	crdt::Lww<String>,
+	crdt::Lww<crdt::MergingOption<ExpirationTime>>,
+	crdt::Lww<AdminApiTokenScope>,
+);
+
+fn make(input: Input) -> AdminApiToken {
+	let (deleted, name, expiration, scope) = input;
+	let state = if deleted {
+		crdt::Deletable::Deleted
+	} else {
+		crdt::Deletable::present(AdminApiTokenParams {
+			created: 0,
+			token_hash: String::new(),
+			name,
+			expiration,
+			scope,
+		})
+	};
+	AdminApiToken {
+		prefix: String::new(),
+		state,
+	}
+}
+
+fuzz_target!(|inputs: (Input, Input, Input)| {
+	let (a, b, c) = inputs;
+	check_crdt_laws(make(a), make(b), make(c));
+});

fuzz/fuzz_targets/block_ref_crdt.rs

@@ -0,0 +1,20 @@
+#![no_main]
+
+use garage_fuzz::check_crdt_laws;
+use garage_model::s3::block_ref_table::BlockRef;
+use libfuzzer_sys::fuzz_target;
+
+/// Build a BlockRef with a fixed block hash and version UUID so that CRDT state
+/// can be compared across merge results. Only the deleted flag varies.
+fn make_block_ref(deleted: bool) -> BlockRef {
+	BlockRef {
+		block: [0u8; 32].into(),
+		version: [0u8; 32].into(),
+		deleted: deleted.into(),
+	}
+}
+
+fuzz_target!(|inputs: (bool, bool, bool)| {
+	let (d1, d2, d3) = inputs;
+	check_crdt_laws(make_block_ref(d1), make_block_ref(d2), make_block_ref(d3));
+});

fuzz/fuzz_targets/bucket_alias_crdt.rs

@@ -0,0 +1,25 @@
+#![no_main]
+
+use garage_fuzz::check_crdt_laws;
+use garage_model::bucket_alias_table::BucketAlias;
+use garage_util::data::Uuid;
+use libfuzzer_sys::fuzz_target;
+
+/// Build a BucketAlias with a fixed name so that CRDT state can be compared
+/// across merge results. The timestamp and optional bucket ID are the CRDT state.
+fn make_bucket_alias(ts: u64, bucket_id: Option<[u8; 32]>) -> BucketAlias {
+	BucketAlias::new(String::new(), ts, bucket_id.map(Uuid::from))
+}
+
+fuzz_target!(|inputs: (
+	(u64, Option<[u8; 32]>),
+	(u64, Option<[u8; 32]>),
+	(u64, Option<[u8; 32]>)
+)| {
+	let ((ts1, b1), (ts2, b2), (ts3, b3)) = inputs;
+	check_crdt_laws(
+		make_bucket_alias(ts1, b1),
+		make_bucket_alias(ts2, b2),
+		make_bucket_alias(ts3, b3),
+	);
+});

fuzz/fuzz_targets/bucket_crdt.rs

@@ -0,0 +1,22 @@
+#![no_main]
+
+use garage_fuzz::check_crdt_laws;
+use garage_model::bucket_table::{Bucket, BucketParams};
+use garage_util::crdt::{self, Deletable};
+use libfuzzer_sys::fuzz_target;
+
+fn make(state: Deletable<BucketParams>) -> Bucket {
+	Bucket {
+		id: [0u8; 32].into(),
+		state,
+	}
+}
+
+fuzz_target!(|inputs: (
+	crdt::Deletable<BucketParams>,
+	crdt::Deletable<BucketParams>,
+	crdt::Deletable<BucketParams>
+)| {
+	let (a, b, c) = inputs;
+	check_crdt_laws(make(a), make(b), make(c));
+});

fuzz/fuzz_targets/k2v_item_crdt.rs

@@ -0,0 +1,36 @@
+#![no_main]
+
+use std::collections::BTreeMap;
+
+use garage_fuzz::check_crdt_laws;
+use garage_model::k2v::item_table::{DvvsEntry, DvvsValue, K2VItem};
+use libfuzzer_sys::fuzz_target;
+
+// Timestamps are encoded as `(ts << 32) | shift` so that items built with different
+// shifts (0, 1, 2) have disjoint timestamp spaces that still interleave in the sorted merge.
+fn make(raw: BTreeMap<u64, (u32, BTreeMap<u32, DvvsValue>)>, shift: u32) -> K2VItem {
+	let shift = shift as u64;
+	let items = raw
+		.into_iter()
+		.map(|(node, (t_discard, values))| {
+			let entry = DvvsEntry::from_raw(
+				(t_discard as u64) << 32 | shift,
+				values
+					.into_iter()
+					.map(|(ts, v)| ((ts as u64) << 32 | shift, v))
+					.collect(),
+			);
+			(node, entry)
+		})
+		.collect();
+	K2VItem::with_raw_items(items)
+}
+
+fuzz_target!(|inputs: (
+	BTreeMap<u64, (u32, BTreeMap<u32, DvvsValue>)>,
+	BTreeMap<u64, (u32, BTreeMap<u32, DvvsValue>)>,
+	BTreeMap<u64, (u32, BTreeMap<u32, DvvsValue>)>,
+)| {
+	let (a, b, c) = inputs;
+	check_crdt_laws(make(a, 0), make(b, 1), make(c, 2));
+});

fuzz/fuzz_targets/key_crdt.rs

@@ -0,0 +1,43 @@
+#![no_main]
+
+use garage_fuzz::check_crdt_laws;
+use garage_model::key_table::{Key, KeyParams};
+use garage_model::permission::{BucketKeyPerm, ExpirationTime};
+use garage_util::crdt;
+use garage_util::data::Uuid;
+use libfuzzer_sys::fuzz_target;
+
+type Input = (
+	bool,
+	crdt::Lww<String>,
+	crdt::Lww<crdt::MergingOption<ExpirationTime>>,
+	crdt::Lww<bool>,
+	crdt::Map<Uuid, BucketKeyPerm>,
+	crdt::LwwMap<String, crdt::CancelingOption<Uuid>>,
+);
+
+fn make(input: Input) -> Key {
+	let (deleted, name, expiration, allow_create_bucket, authorized_buckets, local_aliases) = input;
+	let state = if deleted {
+		crdt::Deletable::Deleted
+	} else {
+		crdt::Deletable::present(KeyParams {
+			created: None,
+			secret_key: String::new(),
+			name,
+			expiration,
+			allow_create_bucket,
+			authorized_buckets,
+			local_aliases,
+		})
+	};
+	Key {
+		key_id: String::new(),
+		state,
+	}
+}
+
+fuzz_target!(|inputs: (Input, Input, Input)| {
+	let (a, b, c) = inputs;
+	check_crdt_laws(make(a), make(b), make(c));
+});

fuzz/fuzz_targets/mpu_crdt.rs

@@ -0,0 +1,37 @@
+#![no_main]
+
+use garage_fuzz::check_crdt_laws;
+use garage_model::s3::mpu_table::{MpuPart, MpuPartKey, MultipartUpload};
+use libfuzzer_sys::fuzz_target;
+
+/// Build a MultipartUpload from an arbitrary deleted flag and parts list, using a fixed
+/// upload_id/bucket_id/key so that CRDT state can be compared across merge results.
+/// `MpuPart.version` is fixed to a constant since it is identity data, not CRDT state:
+/// two replicas of the same part (same MpuPartKey) always share the same version UUID.
+/// If deleted, parts are cleared to ensure a valid initial CRDT state.
+fn make_mpu(deleted: bool, parts: Vec<(MpuPartKey, MpuPart)>) -> MultipartUpload {
+	let mut mpu = MultipartUpload::new(
+		[0u8; 32].into(),
+		0,
+		[0u8; 32].into(),
+		String::new(),
+		deleted,
+	);
+	for (key, mut part) in parts {
+		part.version = [0u8; 32].into();
+		mpu.parts.put(key, part);
+	}
+	if mpu.deleted.get() {
+		mpu.parts.clear();
+	}
+	mpu
+}
+
+fuzz_target!(|inputs: (
+	(bool, Vec<(MpuPartKey, MpuPart)>),
+	(bool, Vec<(MpuPartKey, MpuPart)>),
+	(bool, Vec<(MpuPartKey, MpuPart)>)
+)| {
+	let ((d1, p1), (d2, p2), (d3, p3)) = inputs;
+	check_crdt_laws(make_mpu(d1, p1), make_mpu(d2, p2), make_mpu(d3, p3));
+});

fuzz/fuzz_targets/version_crdt.rs

@@ -0,0 +1,42 @@
+#![no_main]
+
+use garage_fuzz::check_crdt_laws;
+use garage_model::s3::version_table::{Version, VersionBacklink, VersionBlock, VersionBlockKey};
+use libfuzzer_sys::fuzz_target;
+
+/// Build a Version from an arbitrary deleted flag and block list, using a fixed uuid/backlink
+/// so that CRDT state can be compared across merge results.
+/// Duplicate block keys are dropped before construction.
+/// If deleted, blocks are cleared to ensure a valid initial CRDT state.
+fn make_version(deleted: bool, mut blocks: Vec<(VersionBlockKey, VersionBlock)>) -> Version {
+	blocks.sort_by_key(|(k, _)| *k);
+	blocks.dedup_by_key(|(k, _)| *k);
+	let mut v = Version::new(
+		[0u8; 32].into(),
+		VersionBacklink::Object {
+			bucket_id: [0u8; 32].into(),
+			key: String::new(),
+		},
+		deleted,
+	);
+	for (key, block) in blocks {
+		v.blocks.put(key, block);
+	}
+	if v.deleted.get() {
+		v.blocks.clear();
+	}
+	v
+}
+
+fuzz_target!(|inputs: (
+	(bool, Vec<(VersionBlockKey, VersionBlock)>),
+	(bool, Vec<(VersionBlockKey, VersionBlock)>),
+	(bool, Vec<(VersionBlockKey, VersionBlock)>)
+)| {
+	let ((d1, b1), (d2, b2), (d3, b3)) = inputs;
+	check_crdt_laws(
+		make_version(d1, b1),
+		make_version(d2, b2),
+		make_version(d3, b3),
+	);
+});

fuzz/rust-toolchain.toml

@@ -0,0 +1,2 @@
+[toolchain]
+channel = "nightly"

fuzz/src/lib.rs

@@ -0,0 +1,56 @@
+use garage_table::crdt::Crdt;
+use std::fmt::Debug;
+
+pub fn check_crdt_laws<T>(a: T, b: T, c: T)
+where
+	T: Crdt + PartialEq + Clone + Debug,
+{
+	// Idempotency: merge(a, a) == a
+	{
+		let mut a2 = a.clone();
+		a2.merge(&a);
+		assert_eq!(a2, a, "merge is not idempotent: {a2:#?} != {a:#?}");
+	}
+
+	// Commutativity: merge(a, b) == merge(b, a)
+	let ab = {
+		let mut t = a.clone();
+		t.merge(&b);
+		t
+	};
+	let ba = {
+		let mut t = b.clone();
+		t.merge(&a);
+		t
+	};
+	assert_eq!(ab, ba, "merge is not commutative: {ab:#?} != {ba:#?}");
+
+	// LX's corrolary: merge(merge(a,b),b) = merge(a,b)
+	let ab_b = {
+		let mut t = ab.clone();
+		t.merge(&b);
+		t
+	};
+	assert_eq!(ab, ab_b);
+
+	// Associativity: merge(merge(a, b), c) == merge(a, merge(b, c))
+	let ab_c = {
+		let mut t = ab;
+		t.merge(&c);
+		t
+	};
+	let bc = {
+		let mut t = b;
+		t.merge(&c);
+		t
+	};
+	let a_bc = {
+		let mut t = a;
+		t.merge(&bc);
+		t
+	};
+	assert_eq!(
+		ab_c, a_bc,
+		"merge is not associative: {ab_c:#?} != {a_bc:#?}"
+	);
+}

nix/compile.nix

@@ -48,7 +48,7 @@ let
 
   inherit (pkgs) lib stdenv;
 
-  toolchainFn = (p: p.rust-bin.stable."1.91.0".default.override {
+  toolchainFn = (p: p.rust-bin.stable."1.95.0".default.override {
     targets = lib.optionals (target != null) [ rustTarget ];
     extensions = [
       "rust-src"
@@ -148,6 +148,14 @@ let
 in rec {
   toolchain = toolchainFn pkgs;
 
+  toolchainNightly = pkgs.rust-bin.selectLatestNightlyWith (toolchain: toolchain.default.override {
+    targets = lib.optionals (target != null) [ rustTarget ];
+    extensions = [
+      "rust-src"
+      "rustfmt"
+    ];
+  });
+
   devShell = pkgs.mkShell {
     buildInputs = [
       toolchain

script/helm/garage/README.md

@@ -33,11 +33,14 @@ S3-compatible object store for small self-hosted geo-distributed deployments
 | garage.replicationFactor | string | `"3"` | Default to 3 replicas, see the replication_factor section at https://garagehq.deuxfleurs.fr/documentation/reference-manual/configuration/#replication_factor |
 | garage.consistencyMode | string | `"consistent"` | Default to read-after-write consistency, see the consistency_mode section at https://garagehq.deuxfleurs.fr/documentation/reference-manual/configuration/#consistency_mode |
 | garage.metadataAutoSnapshotInterval | string | `""` | If this value is set, Garage will automatically take a snapshot of the metadata DB file at a regular interval and save it in the metadata directory. https://garagehq.deuxfleurs.fr/documentation/reference-manual/configuration/#metadata_auto_snapshot_interval |
+| garage.admin.apiBindAddr | string | `"[::]:3903"` |  |
 | garage.rpcBindAddr | string | `"[::]:3901"` |  |
 | garage.rpcSecret | string | `""` | If not given, a random secret will be generated and stored in a Secret object |
+| garage.s3.api.bindAddr | string | `"[::]:3900"` |  |
 | garage.s3.api.region | string | `"garage"` |  |
 | garage.s3.api.rootDomain | string | `".s3.garage.tld"` |  |
 | garage.s3.web.index | string | `"index.html"` |  |
+| garage.s3.web.bindAddr | string | `"[::]:3902"` |  |
 | garage.s3.web.rootDomain | string | `".web.garage.tld"` |  |
 | image.pullPolicy | string | `"IfNotPresent"` |  |
 | image.repository | string | `"dxflrs/amd64_garage"` | default to amd64 docker image |

script/helm/garage/templates/_helpers.tpl

@@ -71,6 +71,13 @@ Create the name of the service account to use
 {{- end }}
 {{- end }}
 
+{{/*
+Extract the trailing port number from a bind address like [::]:3900 or 0.0.0.0:3900.
+*/}}
+{{- define "garage.portFromBindAddr" -}}
+{{- regexFind "[0-9]+$" . -}}
+{{- end }}
+
 {{/*
     Returns given number of random Hex characters.
     In practice, it generates up to 100 randAlphaNum strings

script/helm/garage/templates/configmap.yaml

@@ -45,16 +45,16 @@ data:
 
     [s3_api]
     s3_region = "{{ .Values.garage.s3.api.region }}"
-    api_bind_addr = "[::]:3900"
+    api_bind_addr = "{{ .Values.garage.s3.api.bindAddr }}"
     root_domain = "{{ .Values.garage.s3.api.rootDomain }}"
 
     [s3_web]
-    bind_addr = "[::]:3902"
+    bind_addr = "{{ .Values.garage.s3.web.bindAddr }}"
     root_domain = "{{ .Values.garage.s3.web.rootDomain }}"
     index = "{{ .Values.garage.s3.web.index }}"
 
     [admin]
-    api_bind_addr = "[::]:3903"
+    api_bind_addr = "{{ .Values.garage.admin.apiBindAddr }}"
     {{- if .Values.monitoring.tracing.sink }}
     trace_sink = "{{ .Values.monitoring.tracing.sink }}"
     {{- end }}

script/helm/garage/templates/service-headless.yaml

@@ -10,11 +10,11 @@ spec:
   clusterIP: None
   ports:
     - port: {{ .Values.service.s3.api.port }}
-      targetPort: 3900
+      targetPort: {{ include "garage.portFromBindAddr" .Values.garage.s3.api.bindAddr | int }}
       protocol: TCP
       name: s3-api
     - port: {{ .Values.service.s3.web.port }}
-      targetPort: 3902
+      targetPort: {{ include "garage.portFromBindAddr" .Values.garage.s3.web.bindAddr | int }}
       protocol: TCP
       name: s3-web
   selector:

script/helm/garage/templates/service.yaml

@@ -12,11 +12,11 @@ spec:
   type: {{ .Values.service.type }}
   ports:
     - port: {{ .Values.service.s3.api.port }}
-      targetPort: 3900
+      targetPort: {{ include "garage.portFromBindAddr" .Values.garage.s3.api.bindAddr | int }}
       protocol: TCP
       name: s3-api
     - port: {{ .Values.service.s3.web.port }}
-      targetPort: 3902
+      targetPort: {{ include "garage.portFromBindAddr" .Values.garage.s3.web.bindAddr | int }}
       protocol: TCP
       name: s3-web
   selector:
@@ -35,8 +35,8 @@ spec:
   type: ClusterIP
   clusterIP: None
   ports:
-    - port: 3903
+    - port: {{ include "garage.portFromBindAddr" .Values.garage.admin.apiBindAddr | int }}
-      targetPort: 3903
+      targetPort: {{ include "garage.portFromBindAddr" .Values.garage.admin.apiBindAddr | int }}
       protocol: TCP
       name: metrics
   selector:

script/helm/garage/templates/workload.yaml

@@ -60,11 +60,11 @@ spec:
           image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
           imagePullPolicy: {{ .Values.image.pullPolicy }}
           ports:
-            - containerPort: 3900
+            - containerPort: {{ include "garage.portFromBindAddr" .Values.garage.s3.api.bindAddr | int }}
               name: s3-api
-            - containerPort: 3902
+            - containerPort: {{ include "garage.portFromBindAddr" .Values.garage.s3.web.bindAddr | int }}
               name: web-api
-            - containerPort: 3903
+            - containerPort: {{ include "garage.portFromBindAddr" .Values.garage.admin.apiBindAddr | int }}
               name: admin
           {{- with .Values.environment }}
           env:

script/helm/garage/values.yaml

@@ -48,11 +48,15 @@ garage:
   kubernetesSkipCrd: false
   s3:
     api:
+      bindAddr: "[::]:3900"
       region: "garage"
       rootDomain: ".s3.garage.tld"
     web:
+      bindAddr: "[::]:3902"
       rootDomain: ".web.garage.tld"
       index: "index.html"
+  admin:
+    apiBindAddr: "[::]:3903"
 
   # -- Additional configuration to append to garage.toml. Use a multi-line string for custom config.
   # Example:
@@ -221,14 +225,14 @@ resources: {}
 livenessProbe: {}
   #httpGet:
   #  path: /health
-  #  port: 3903
+  #  port: 3903 # or the port from garage.admin.apiBindAddr
   #initialDelaySeconds: 5
   #periodSeconds: 30
 # -- Specifies a readinessProbe
 readinessProbe: {}
   #httpGet:
   #  path: /health
-  #  port: 3903
+  #  port: 3903 # or the port from garage.admin.apiBindAddr
   #initialDelaySeconds: 5
   #periodSeconds: 30
 

script/telemetry/grafana-garage-dashboard-elasticsearch.json

@@ -161,7 +161,7 @@
           },
           "metrics": [
             {
-              "field": "api_request_counter",
+              "field": "garage_api_request_count",
               "hide": true,
               "id": "1",
               "type": "sum"
@@ -284,7 +284,7 @@
           "hide": false,
           "metrics": [
             {
-              "field": "api_request_duration",
+              "field": "garage_api_request_duration",
               "id": "1",
               "type": "avg"
             }
@@ -412,7 +412,7 @@
           },
           "metrics": [
             {
-              "field": "api_error_counter",
+              "field": "garage_api_error_count",
               "hide": true,
               "id": "1",
               "type": "sum"
@@ -540,7 +540,7 @@
           },
           "metrics": [
             {
-              "field": "web_request_counter",
+              "field": "garage_web_request_count",
               "hide": true,
               "id": "1",
               "type": "sum"
@@ -666,7 +666,7 @@
           "hide": false,
           "metrics": [
             {
-              "field": "web_request_duration",
+              "field": "garage_web_request_duration",
               "id": "1",
               "type": "avg"
             }
@@ -794,7 +794,7 @@
           },
           "metrics": [
             {
-              "field": "web_error_counter",
+              "field": "garage_web_error_count",
               "hide": true,
               "id": "1",
               "type": "sum"
@@ -918,7 +918,7 @@
           "hide": false,
           "metrics": [
             {
-              "field": "table_get_request_counter",
+              "field": "garage_table_get_request_count",
               "hide": true,
               "id": "1",
               "type": "sum"
@@ -1042,7 +1042,7 @@
           "hide": false,
           "metrics": [
             {
-              "field": "table_put_request_counter",
+              "field": "garage_table_put_request_count",
               "hide": true,
               "id": "1",
               "type": "sum"
@@ -1154,7 +1154,7 @@
           "hide": false,
           "metrics": [
             {
-              "field": "block_bytes_read",
+              "field": "garage_block_bytes_read",
               "hide": true,
               "id": "1",
               "type": "sum"
@@ -1270,7 +1270,7 @@
           },
           "metrics": [
             {
-              "field": "block_bytes_written",
+              "field": "garage_block_bytes_written",
               "hide": true,
               "id": "1",
               "type": "sum"
@@ -1386,7 +1386,7 @@
           },
           "metrics": [
             {
-              "field": "block_resync_counter",
+              "field": "garage_block_resync_count",
               "hide": true,
               "id": "1",
               "type": "sum"
@@ -1500,7 +1500,7 @@
           "hide": false,
           "metrics": [
             {
-              "field": "block_resync_queue_length",
+              "field": "garage_block_resync_queue_length",
               "id": "1",
               "type": "avg"
             }
@@ -1610,7 +1610,7 @@
           },
           "metrics": [
             {
-              "field": "table_merkle_updater_todo_queue_length",
+              "field": "garage_table_merkle_updater_todo_queue_length",
               "id": "1",
               "type": "avg"
             }
@@ -1724,7 +1724,7 @@
           },
           "metrics": [
             {
-              "field": "table_gc_todo_queue_length",
+              "field": "garage_table_gc_todo_queue_length",
               "id": "1",
               "type": "avg"
             }
@@ -1824,7 +1824,7 @@
           },
           "metrics": [
             {
-              "field": "block_resync_error_counter",
+              "field": "garage_block_resync_error_count",
               "hide": true,
               "id": "1",
               "settings": {},
@@ -1938,7 +1938,7 @@
           },
           "metrics": [
             {
-              "field": "block_resync_errored_blocks",
+              "field": "garage_block_resync_errored_blocks",
               "hide": false,
               "id": "1",
               "type": "sum"
@@ -2041,7 +2041,7 @@
           },
           "metrics": [
             {
-              "field": "block_corruption_counter",
+              "field": "garage_block_corruption_count",
               "hide": true,
               "id": "1",
               "type": "sum"
@@ -2165,7 +2165,7 @@
           },
           "metrics": [
             {
-              "field": "rpc_netapp_error_counter",
+              "field": "garage_rpc_netapp_error_count",
               "hide": true,
               "id": "1",
               "type": "sum"
@@ -2292,7 +2292,7 @@
           },
           "metrics": [
             {
-              "field": "rpc_request_counter",
+              "field": "garage_rpc_request_count",
               "hide": true,
               "id": "1",
               "type": "sum"
@@ -2418,7 +2418,7 @@
           },
           "metrics": [
             {
-              "field": "rpc_duration",
+              "field": "garage_rpc_duration",
               "id": "1",
               "type": "avg"
             }
@@ -2521,7 +2521,7 @@
           },
           "metrics": [
             {
-              "field": "admin_http_requests_total",
+              "field": "garage_admin_http_requests_total",
               "hide": true,
               "id": "1",
               "type": "sum"
@@ -2654,7 +2654,7 @@
           },
           "metrics": [
             {
-              "field": "rpc_garage_error_counter",
+              "field": "garage_rpc_garage_error_count",
               "hide": true,
               "id": "1",
               "type": "sum"
@@ -2765,7 +2765,7 @@
           },
           "metrics": [
             {
-              "field": "rpc_duration",
+              "field": "garage_rpc_duration",
               "id": "1",
               "type": "avg"
             }
@@ -2995,4 +2995,4 @@
   "uid": "ODT8K4B7e",
   "version": 7,
   "weekStart": ""
-}
+}

script/telemetry/grafana-garage-dashboard-prometheus.json

@@ -143,7 +143,7 @@
             "uid": "${DS_DS_PROMETHEUS}"
           },
           "exemplar": true,
-          "expr": "sum(rate(block_bytes_read{job=\"garage\"}[$__rate_interval])    )",
+          "expr": "sum(rate(garage_block_bytes_read{job=\"garage\"}[$__rate_interval])    )",
           "hide": false,
           "interval": "",
           "legendFormat": "Disk bytes read",
@@ -155,7 +155,7 @@
             "uid": "${DS_DS_PROMETHEUS}"
           },
           "exemplar": true,
-          "expr": "-sum(rate(block_bytes_written{job=\"garage\"}[$__rate_interval])    )",
+          "expr": "-sum(rate(garage_block_bytes_written{job=\"garage\"}[$__rate_interval])    )",
           "hide": false,
           "interval": "",
           "legendFormat": "Disk bytes written",
@@ -250,7 +250,7 @@
           },
           "editorMode": "code",
           "exemplar": true,
-          "expr": "sum by (api_endpoint) (rate(api_s3_request_counter {job=\"garage\"}[$__rate_interval]))",
+          "expr": "sum by (api_endpoint) (rate(garage_api_s3_request_count {job=\"garage\"}[$__rate_interval]))",
           "hide": false,
           "interval": "",
           "legendFormat": "{{api_endpoint}}",
@@ -345,7 +345,7 @@
             "uid": "${DS_DS_PROMETHEUS}"
           },
           "exemplar": true,
-          "expr": "sum(rate(web_request_counter {job=\"garage\"}[$__rate_interval]))",
+          "expr": "sum(rate(garage_web_request_count {job=\"garage\"}[$__rate_interval]))",
           "hide": false,
           "interval": "",
           "legendFormat": "Web request rate",
@@ -439,7 +439,7 @@
             "uid": "${DS_DS_PROMETHEUS}"
           },
           "exemplar": true,
-          "expr": "sum by (rpc_endpoint) (rate(rpc_request_counter {job=\"garage\"}[$__rate_interval]))",
+          "expr": "sum by (rpc_endpoint) (rate(garage_rpc_request_count {job=\"garage\"}[$__rate_interval]))",
           "hide": false,
           "interval": "",
           "legendFormat": "{{rpc_endpoint}}",
@@ -534,7 +534,7 @@
           },
           "editorMode": "code",
           "exemplar": true,
-          "expr": "sum by (api_endpoint, status_code) (rate(api_s3_error_counter {job=\"garage\"}[$__rate_interval]))",
+          "expr": "sum by (api_endpoint, status_code) (rate(garage_api_s3_error_count {job=\"garage\"}[$__rate_interval]))",
           "hide": false,
           "interval": "",
           "legendFormat": "{{api_endpoint}} {{status_code}}",
@@ -629,7 +629,7 @@
             "uid": "${DS_DS_PROMETHEUS}"
           },
           "exemplar": true,
-          "expr": "sum by(status_code) (rate(web_error_counter {job=\"garage\"}[$__rate_interval]))",
+          "expr": "sum by(status_code) (rate(garage_web_error_count {job=\"garage\"}[$__rate_interval]))",
           "hide": false,
           "interval": "",
           "legendFormat": "{{status_code}}",
@@ -722,7 +722,7 @@
             "uid": "${DS_DS_PROMETHEUS}"
           },
           "exemplar": true,
-          "expr": "block_resync_queue_length{job=\"garage\"}",
+          "expr": "garage_block_resync_queue_length{job=\"garage\"}",
           "interval": "",
           "legendFormat": "{{instance}}",
           "refId": "A"
@@ -814,7 +814,7 @@
             "uid": "${DS_DS_PROMETHEUS}"
           },
           "exemplar": true,
-          "expr": "sum by(table_name) (table_gc_todo_queue_length{job=\"garage\"})",
+          "expr": "sum by(table_name) (garage_table_gc_todo_queue_length{job=\"garage\"})",
           "interval": "",
           "legendFormat": "{{ table_name}}",
           "refId": "A"
@@ -906,7 +906,7 @@
             "uid": "${DS_DS_PROMETHEUS}"
           },
           "exemplar": true,
-          "expr": "sum by(table_name) (table_merkle_updater_todo_queue_length{job=\"garage\"})",
+          "expr": "sum by(table_name) (garage_table_merkle_updater_todo_queue_length{job=\"garage\"})",
           "interval": "",
           "legendFormat": "{{ table_name}}",
           "refId": "A"
@@ -998,7 +998,7 @@
             "uid": "${DS_DS_PROMETHEUS}"
           },
           "exemplar": true,
-          "expr": "block_resync_errored_blocks{job=\"garage\"}",
+          "expr": "garage_block_resync_errored_blocks{job=\"garage\"}",
           "interval": "",
           "legendFormat": "{{instance}}",
           "refId": "A"
@@ -1025,4 +1025,4 @@
   "uid": "ys3pnpZ4k",
   "version": 26,
   "weekStart": ""
-}
+}

src/api/admin/admin_token.rs

@@ -7,6 +7,7 @@ use garage_util::time::now_msec;
 
 use garage_model::admin_token_table::*;
 use garage_model::garage::Garage;
+use garage_model::permission::ExpirationTime;
 
 use crate::api::*;
 use crate::error::*;
@@ -244,8 +245,8 @@ fn admin_token_info_results(token: &AdminApiToken, now: u64) -> GetAdminTokenInf
 				.expect("invalid timestamp stored in db"),
 		),
 		name: params.name.get().to_string(),
-		expiration: params.expiration.get().map(|x| {
+		expiration: params.expiration.get().inner().map(|x| {
-			DateTime::from_timestamp_millis(x as i64).expect("invalid timestamp stored in db")
+			DateTime::from_timestamp_millis(x.0 as i64).expect("invalid timestamp stored in db")
 		}),
 		expired: params.is_expired(now),
 		scope: params.scope.get().0.clone(),
@@ -279,10 +280,10 @@ fn apply_token_updates(
 	if let Some(expiration) = updates.expiration {
 		params
 			.expiration
-			.update(Some(expiration.timestamp_millis() as u64));
+			.update(Some(ExpirationTime(expiration.timestamp_millis() as u64)).into());
 	}
 	if updates.never_expires {
-		params.expiration.update(None);
+		params.expiration.update(None.into());
 	}
 	if let Some(scope) = updates.scope {
 		params.scope.update(AdminApiTokenScope(scope));

src/api/admin/api.rs

@@ -618,6 +618,10 @@ pub enum PreviewClusterLayoutChangesResponse {
 		/// Plain-text information about the layout computation
 		/// (do not try to parse this)
 		message: Vec<String>,
+		/// Structured statistics about the layout computation
+		// FIXME for v3: remove default and skip_serializing_if
+		#[serde(default, skip_serializing_if = "Option::is_none")]
+		statistics: Option<Box<garage_rpc::layout::ComputationStat>>,
 		/// Details about the new cluster layout
 		new_layout: GetClusterLayoutResponse,
 	},
@@ -639,6 +643,10 @@ pub struct ApplyClusterLayoutResponse {
 	/// Plain-text information about the layout computation
 	/// (do not try to parse this)
 	pub message: Vec<String>,
+	/// Structured statistics about the layout computation
+	// FIXME for v3: remove default and skip_serializing_if
+	#[serde(default, skip_serializing_if = "Option::is_none")]
+	pub statistics: Option<garage_rpc::layout::ComputationStat>,
 	/// Details about the new cluster layout
 	pub layout: GetClusterLayoutResponse,
 }
@@ -1165,6 +1173,29 @@ pub struct LocalGetNodeInfoResponse {
 	pub rust_version: String,
 	/// database engine used for metadata
 	pub db_engine: String,
+	// FIXME for v3: remove Option<> and serde(default) for field below
+	// FIXME for v3: merge LocalGetNodeInfoResponse and NodeResp
+	/// Socket address used by other nodes to connect to this node for RPC
+	#[serde(default, skip_serializing_if = "Option::is_none")]
+	#[schema(value_type = Option<String>)]
+	pub addr: Option<SocketAddr>,
+	/// Whether this node is connected in the cluster
+	#[serde(default, skip_serializing_if = "Option::is_none")]
+	pub is_up: Option<bool>,
+	/// Role assigned to this node in the current cluster layout
+	#[serde(default, skip_serializing_if = "Option::is_none")]
+	pub role: Option<NodeAssignedRole>,
+	/// Whether this node is part of an older layout version and is draining data.
+	#[serde(default, skip_serializing_if = "Option::is_none")]
+	pub draining: Option<bool>,
+	/// Total and available space on the disk partition(s) containing the data
+	/// directory(ies)
+	#[serde(default, skip_serializing_if = "Option::is_none")]
+	pub data_partition: Option<FreeSpaceResp>,
+	/// Total and available space on the disk partition containing the
+	/// metadata directory
+	#[serde(default, skip_serializing_if = "Option::is_none")]
+	pub metadata_partition: Option<FreeSpaceResp>,
 }
 
 // ---- GetNodeStatistics ----

src/api/admin/bucket.rs

@@ -90,7 +90,7 @@ impl RequestHandler for GetBucketInfoRequest {
 				.bucket_alias_table
 				.get(&EmptyKey, &ga)
 				.await?
-				.and_then(|x| *x.state.get())
+				.and_then(|x| x.state.get().into_inner())
 				.ok_or_else(|| HelperError::NoSuchBucket(ga.to_string()))?,
 			(None, None, Some(search)) => {
 				let helper = garage.bucket_helper();
@@ -168,7 +168,7 @@ impl RequestHandler for CreateBucketRequest {
 			}
 
 			if let Some(alias) = garage.bucket_alias_table.get(&EmptyKey, ga).await? {
-				if alias.state.get().is_some() {
+				if alias.state.get().inner().is_some() {
 					return Err(CommonError::BucketAlreadyExists.into());
 				}
 			}
@@ -297,7 +297,7 @@ impl RequestHandler for UpdateBucketRequest {
 				let redirect_all = state
 					.website_config
 					.get()
-					.as_ref()
+					.inner()
 					.and_then(|wc| wc.redirect_all.clone());
 
 				let routing_rules = if let Some(rr) = wa.routing_rules {
@@ -311,26 +311,29 @@ impl RequestHandler for UpdateBucketRequest {
 					state
 						.website_config
 						.get()
-						.as_ref()
+						.inner()
 						.map(|wc| wc.routing_rules.clone())
 						.unwrap_or_default()
 				};
 
-				state.website_config.update(Some(WebsiteConfig {
+				state.website_config.update(
-					index_document: wa.index_document.ok_or_bad_request(
+					Some(WebsiteConfig {
-						"Please specify indexDocument when enabling website access.",
+						index_document: wa.index_document.ok_or_bad_request(
-					)?,
+							"Please specify indexDocument when enabling website access.",
-					error_document: wa.error_document,
+						)?,
-					redirect_all,
+						error_document: wa.error_document,
-					routing_rules,
+						redirect_all,
-				}));
+						routing_rules,
+					})
+					.into(),
+				);
 			} else {
 				if wa.index_document.is_some() || wa.error_document.is_some() {
 					return Err(Error::bad_request(
                         "Cannot specify indexDocument or errorDocument when disabling website access.",
                     ));
 				}
-				state.website_config.update(None);
+				state.website_config.update(None.into());
 			}
 		}
 
@@ -353,7 +356,7 @@ impl RequestHandler for UpdateBucketRequest {
 				Some(cc.into_garage_cors_config()?)
 			};
 
-			state.cors_config.update(cors_config);
+			state.cors_config.update(cors_config.into());
 		}
 
 		if let Some(lr) = self.body.lifecycle_rules {
@@ -370,7 +373,7 @@ impl RequestHandler for UpdateBucketRequest {
 				)
 			};
 
-			state.lifecycle_config.update(lifecycle_config);
+			state.lifecycle_config.update(lifecycle_config.into());
 		}
 
 		garage.bucket_table.insert(&bucket).await?;
@@ -739,8 +742,8 @@ async fn bucket_info_results(
 			.filter(|(_, _, a)| *a)
 			.map(|(n, _, _)| n.to_string())
 			.collect::<Vec<_>>(),
-		website_access: state.website_config.get().is_some(),
+		website_access: state.website_config.get().inner().is_some(),
-		website_config: state.website_config.get().clone().map(|wsc| {
+		website_config: state.website_config.get().inner().cloned().map(|wsc| {
 			GetBucketInfoWebsiteResponse {
 				index_document: wsc.index_document,
 				error_document: wsc.error_document,
@@ -752,13 +755,13 @@ async fn bucket_info_results(
 				),
 			}
 		}),
-		cors_rules: state.cors_config.get().as_ref().map(|rules| {
+		cors_rules: state.cors_config.get().inner().map(|rules| {
 			rules
 				.iter()
 				.map(xml::cors::CorsRule::from_garage_cors_rule)
 				.collect::<Vec<_>>()
 		}),
-		lifecycle_rules: state.lifecycle_config.get().as_ref().map(|lc| {
+		lifecycle_rules: state.lifecycle_config.get().inner().map(|lc| {
 			lc.iter()
 				.map(xml::lifecycle::LifecycleRule::from_garage_lifecycle_rule)
 				.collect::<Vec<_>>()
@@ -784,7 +787,7 @@ async fn bucket_info_results(
 						.local_aliases
 						.items()
 						.iter()
-						.filter(|(_, _, b)| *b == Some(bucket.id))
+						.filter(|(_, _, b)| b.into_inner() == Some(bucket.id))
 						.map(|(n, _, _)| n.to_string())
 						.collect::<Vec<_>>(),
 				})

src/api/admin/key.rs

@@ -8,6 +8,7 @@ use garage_util::time::now_msec;
 
 use garage_model::garage::Garage;
 use garage_model::key_table::*;
+use garage_model::permission::ExpirationTime;
 
 use crate::api::*;
 use crate::error::*;
@@ -40,8 +41,8 @@ impl RequestHandler for ListKeysRequest {
 						DateTime::from_timestamp_millis(x as i64)
 							.expect("invalid timestamp stored in db")
 					}),
-					expiration: p.expiration.get().map(|x| {
+					expiration: p.expiration.get().inner().map(|x| {
-						DateTime::from_timestamp_millis(x as i64)
+						DateTime::from_timestamp_millis(x.0 as i64)
 							.expect("invalid timestamp stored in db")
 					}),
 					expired: p.is_expired(now),
@@ -201,7 +202,7 @@ async fn key_info_results(
 				.local_aliases
 				.items()
 				.iter()
-				.filter_map(|(_, _, v)| v.as_ref()),
+				.filter_map(|(_, _, v)| v.inner()),
 		) {
 		if !relevant_buckets.contains_key(id) {
 			if let Some(b) = garage.bucket_table.get(&EmptyKey, id).await? {
@@ -217,8 +218,8 @@ async fn key_info_results(
 		created: key_state.created.map(|x| {
 			DateTime::from_timestamp_millis(x as i64).expect("invalid timestamp stored in db")
 		}),
-		expiration: key_state.expiration.get().map(|x| {
+		expiration: key_state.expiration.get().inner().map(|x| {
-			DateTime::from_timestamp_millis(x as i64).expect("invalid timestamp stored in db")
+			DateTime::from_timestamp_millis(x.0 as i64).expect("invalid timestamp stored in db")
 		}),
 		expired: key_state.is_expired(now_msec()),
 		access_key_id: key.key_id.clone(),
@@ -283,10 +284,10 @@ fn apply_key_updates(key: &mut Key, updates: UpdateKeyRequestBody) -> Result<(),
 	if let Some(expiration) = updates.expiration {
 		key_state
 			.expiration
-			.update(Some(expiration.timestamp_millis() as u64));
+			.update(Some(ExpirationTime(expiration.timestamp_millis() as u64)).into());
 	}
 	if updates.never_expires {
-		key_state.expiration.update(None);
+		key_state.expiration.update(None.into());
 	}
 	if let Some(allow) = updates.allow {
 		if allow.create_bucket {

src/api/admin/layout.rs

@@ -53,7 +53,7 @@ fn format_cluster_layout(layout: &layout::LayoutHistory) -> GetClusterLayoutResp
 		.roles
 		.items()
 		.iter()
-		.filter(|(k, _, v)| current.roles.get(k) != Some(v))
+		.filter(|(k, _, v)| current.roles.get(k).and_then(|vv| vv.0.as_ref()) != v.0.as_ref())
 		.map(|(k, _, v)| match &v.0 {
 			None => NodeRoleChange {
 				id: hex::encode(k),
@@ -255,10 +255,14 @@ impl RequestHandler for PreviewClusterLayoutChangesRequest {
 				Ok(PreviewClusterLayoutChangesResponse::Error { error })
 			}
 			Err(e) => Err(e.into()),
-			Ok((new_layout, msg)) => Ok(PreviewClusterLayoutChangesResponse::Success {
+			Ok((new_layout, stat)) => {
-				message: msg,
+				let message = stat.to_message();
-				new_layout: format_cluster_layout(&new_layout),
+				Ok(PreviewClusterLayoutChangesResponse::Success {
-			}),
+					message,
+					statistics: Some(Box::new(stat)),
+					new_layout: format_cluster_layout(&new_layout),
+				})
+			}
 		}
 	}
 }
@@ -272,7 +276,8 @@ impl RequestHandler for ApplyClusterLayoutRequest {
 		_admin: &Admin,
 	) -> Result<ApplyClusterLayoutResponse, Error> {
 		let layout = garage.system.cluster_layout().inner().clone();
-		let (layout, msg) = layout.apply_staged_changes(self.version)?;
+		let (layout, stat) = layout.apply_staged_changes(self.version)?;
+		let message = stat.to_message();
 
 		garage
 			.system
@@ -281,7 +286,8 @@ impl RequestHandler for ApplyClusterLayoutRequest {
 			.await?;
 
 		Ok(ApplyClusterLayoutResponse {
-			message: msg,
+			message,
+			statistics: Some(stat),
 			layout: format_cluster_layout(&layout),
 		})
 	}

src/api/admin/node.rs

@@ -25,6 +25,9 @@ impl RequestHandler for LocalGetNodeInfoRequest {
 		let sys_status = garage.system.local_status();
 		let hostname = sys_status.hostname.unwrap_or_default().to_string();
 
+		let layout = garage.system.cluster_layout();
+		let current_layout = layout.inner().current();
+
 		Ok(LocalGetNodeInfoResponse {
 			node_id: hex::encode(garage.system.id),
 			hostname: Some(hostname),
@@ -33,6 +36,41 @@ impl RequestHandler for LocalGetNodeInfoRequest {
 				.map(|features| features.iter().map(ToString::to_string).collect()),
 			rust_version: garage_util::version::rust_version().to_string(),
 			db_engine: garage.db.engine(),
+			is_up: Some(true),
+			addr: garage
+				.system
+				.get_known_nodes()
+				.iter()
+				.find(|x| x.id == garage.system.id)
+				.and_then(|x| x.addr),
+			draining: Some(
+				current_layout.node_role(&garage.system.id).is_none()
+					&& layout
+						.inner()
+						.versions
+						.iter()
+						.filter(|x| x.version != current_layout.version)
+						.any(|x| x.node_role(&garage.system.id).is_some()),
+			),
+			role: current_layout
+				.node_role(&garage.system.id)
+				.map(|v| NodeAssignedRole {
+					zone: v.zone.clone(),
+					capacity: v.capacity,
+					tags: v.tags.clone(),
+				}),
+			data_partition: sys_status
+				.data_disk_avail
+				.map(|(avail, total)| FreeSpaceResp {
+					available: avail,
+					total,
+				}),
+			metadata_partition: sys_status
+				.meta_disk_avail
+				.map(|(avail, total)| FreeSpaceResp {
+					available: avail,
+					total,
+				}),
 		})
 	}
 }

src/api/admin/special.rs

@@ -164,7 +164,7 @@ async fn check_domain(garage: &Arc<Garage>, domain: &str) -> Result<bool, Error>
 	}
 
 	let bucket_state = bucket.state.as_option().unwrap();
-	let bucket_website_config = bucket_state.website_config.get();
+	let bucket_website_config = bucket_state.website_config.get().inner();
 
 	match bucket_website_config {
 		Some(_v) => Ok(true),

src/api/common/cors.rs

@@ -1,8 +1,9 @@
 use std::sync::Arc;
 
 use http::header::{
-	ACCESS_CONTROL_ALLOW_HEADERS, ACCESS_CONTROL_ALLOW_METHODS, ACCESS_CONTROL_ALLOW_ORIGIN,
+	HeaderValue, ACCESS_CONTROL_ALLOW_HEADERS, ACCESS_CONTROL_ALLOW_METHODS,
-	ACCESS_CONTROL_EXPOSE_HEADERS, ACCESS_CONTROL_REQUEST_HEADERS, ACCESS_CONTROL_REQUEST_METHOD,
+	ACCESS_CONTROL_ALLOW_ORIGIN, ACCESS_CONTROL_EXPOSE_HEADERS, ACCESS_CONTROL_REQUEST_HEADERS,
+	ACCESS_CONTROL_REQUEST_METHOD, VARY,
 };
 use hyper::{body::Body, body::Incoming as IncomingBody, Request, Response, StatusCode};
 
@@ -12,20 +13,25 @@ use garage_model::garage::Garage;
 use crate::common_error::{CommonError, OkOrBadRequest, OkOrInternalError};
 use crate::helpers::*;
 
+// Return both the matching rule and the parsed Origin header so callers that
+// apply CORS headers don't have to repeat Origin lookup and validation.
 pub fn find_matching_cors_rule<'a, B>(
 	bucket_params: &'a BucketParams,
-	req: &Request<B>,
+	req: &'a Request<B>,
-) -> Result<Option<&'a GarageCorsRule>, CommonError> {
+) -> Result<Option<(&'a GarageCorsRule, &'a str)>, CommonError> {
-	if let Some(cors_config) = bucket_params.cors_config.get() {
+	if let Some(cors_config) = bucket_params.cors_config.get().inner() {
 		if let Some(origin) = req.headers().get("Origin") {
 			let origin = origin.to_str()?;
 			let request_headers = match req.headers().get(ACCESS_CONTROL_REQUEST_HEADERS) {
 				Some(h) => h.to_str()?.split(',').map(|h| h.trim()).collect::<Vec<_>>(),
 				None => vec![],
 			};
-			return Ok(cors_config.iter().find(|rule| {
+			return Ok(cors_config
-				cors_rule_matches(rule, origin, req.method().as_ref(), request_headers.iter())
+				.iter()
-			}));
+				.find(|rule| {
+					cors_rule_matches(rule, origin, req.method().as_ref(), request_headers.iter())
+				})
+				.map(|rule| (rule, origin)));
 		}
 	}
 	Ok(None)
@@ -53,12 +59,16 @@ where
 pub fn add_cors_headers(
 	resp: &mut Response<impl Body>,
 	rule: &GarageCorsRule,
+	request_origin: &str,
 ) -> Result<(), http::header::InvalidHeaderValue> {
 	let h = resp.headers_mut();
-	h.insert(
+	let is_wildcard_origin = rule.allow_origins.iter().any(|origin| origin == "*");
-		ACCESS_CONTROL_ALLOW_ORIGIN,
+	let allow_origin = if is_wildcard_origin {
-		rule.allow_origins.join(", ").parse()?,
+		"*"
-	);
+	} else {
+		request_origin
+	};
+	h.insert(ACCESS_CONTROL_ALLOW_ORIGIN, allow_origin.parse()?);
 	h.insert(
 		ACCESS_CONTROL_ALLOW_METHODS,
 		rule.allow_methods.join(", ").parse()?,
@@ -71,6 +81,12 @@ pub fn add_cors_headers(
 		ACCESS_CONTROL_EXPOSE_HEADERS,
 		rule.expose_headers.join(", ").parse()?,
 	);
+	// When ACAO reflects the request origin instead of returning "*",
+	// caches must vary on the Origin request header to avoid reusing
+	// a response generated for one origin when serving another origin.
+	if !is_wildcard_origin {
+		h.insert(VARY, HeaderValue::from_static("Origin"));
+	}
 	Ok(())
 }
 
@@ -107,6 +123,7 @@ pub fn handle_options_api(
 			Ok(Response::builder()
 				.header(ACCESS_CONTROL_ALLOW_ORIGIN, "*")
 				.header(ACCESS_CONTROL_ALLOW_METHODS, "*")
+				.header(ACCESS_CONTROL_ALLOW_HEADERS, "*")
 				.status(StatusCode::OK)
 				.body(EmptyBody::new())?)
 		}
@@ -141,7 +158,7 @@ pub fn handle_options_for_bucket<B>(
 		None => vec![],
 	};
 
-	if let Some(cors_config) = bucket_params.cors_config.get() {
+	if let Some(cors_config) = bucket_params.cors_config.get().inner() {
 		let matching_rule = cors_config
 			.iter()
 			.find(|rule| cors_rule_matches(rule, origin, request_method, request_headers.iter()));
@@ -149,7 +166,17 @@ pub fn handle_options_for_bucket<B>(
 			let mut resp = Response::builder()
 				.status(StatusCode::OK)
 				.body(EmptyBody::new())?;
-			add_cors_headers(&mut resp, rule).ok_or_internal_error("Invalid CORS configuration")?;
+			add_cors_headers(&mut resp, rule, origin)
+				.ok_or_internal_error("Invalid CORS configuration")?;
+			// Preflight responses vary not only on Origin but also on the
+			// requested method and requested headers, so caches must not
+			// reuse one preflight decision for a different preflight input.
+			resp.headers_mut().insert(
+				VARY,
+				"Origin, Access-Control-Request-Method, Access-Control-Request-Headers"
+					.parse()
+					.expect("static vary header"),
+			);
 			return Ok(resp);
 		}
 	}
@@ -158,3 +185,101 @@ pub fn handle_options_for_bucket<B>(
 		"This CORS request is not allowed.".into(),
 	))
 }
+
+#[cfg(test)]
+mod tests {
+	use super::*;
+
+	fn bucket_params_with_rule(allow_origins: Vec<&str>) -> BucketParams {
+		let mut bucket_params = BucketParams::default();
+		bucket_params.cors_config.update(
+			Some(vec![GarageCorsRule {
+				id: Some("cors-test".into()),
+				max_age_seconds: None,
+				allow_origins: allow_origins.into_iter().map(str::to_string).collect(),
+				allow_methods: vec!["GET".into(), "PUT".into()],
+				allow_headers: vec!["*".into()],
+				expose_headers: vec![],
+			}])
+			.into(),
+		);
+		bucket_params
+	}
+
+	fn preflight_request(origin: &str) -> Request<()> {
+		Request::builder()
+			.method("OPTIONS")
+			.uri("http://example.test/bucket")
+			.header("Origin", origin)
+			.header(ACCESS_CONTROL_REQUEST_METHOD, "PUT")
+			.body(())
+			.unwrap()
+	}
+
+	#[test]
+	fn preflight_with_single_allowed_origin_returns_request_origin() {
+		let bucket_params = bucket_params_with_rule(vec!["https://app.example.test"]);
+		let req = preflight_request("https://app.example.test");
+
+		let resp = handle_options_for_bucket(&req, &bucket_params).unwrap();
+
+		assert_eq!(
+			resp.headers().get(ACCESS_CONTROL_ALLOW_ORIGIN).unwrap(),
+			"https://app.example.test"
+		);
+		let vary_values: Vec<_> = resp
+			.headers()
+			.get_all(VARY)
+			.iter()
+			.map(|value| value.to_str().unwrap())
+			.collect();
+		assert_eq!(
+			vary_values,
+			vec!["Origin, Access-Control-Request-Method, Access-Control-Request-Headers",]
+		);
+	}
+
+	#[test]
+	fn preflight_with_multiple_allowed_origins_reflects_request_origin() {
+		let bucket_params = bucket_params_with_rule(vec![
+			"https://app.example.test",
+			"https://admin.example.test",
+		]);
+		let req = preflight_request("https://app.example.test");
+
+		let resp = handle_options_for_bucket(&req, &bucket_params).unwrap();
+
+		// This assertion documents the behavior browsers expect:
+		// even if multiple origins are allowed by configuration, the
+		// response should reflect the request origin rather than emit
+		// a comma-separated list. It currently fails and is meant to
+		// turn green once header generation is corrected.
+		assert_eq!(
+			resp.headers().get(ACCESS_CONTROL_ALLOW_ORIGIN).unwrap(),
+			"https://app.example.test"
+		);
+	}
+
+	#[test]
+	fn preflight_with_wildcard_allowed_origin_returns_wildcard() {
+		let bucket_params = bucket_params_with_rule(vec!["*"]);
+		let req = preflight_request("https://app.example.test");
+
+		let resp = handle_options_for_bucket(&req, &bucket_params).unwrap();
+
+		assert_eq!(
+			resp.headers().get(ACCESS_CONTROL_ALLOW_ORIGIN).unwrap(),
+			"*"
+		);
+		let vary_values: Vec<_> = resp
+			.headers()
+			.get_all(VARY)
+			.iter()
+			.map(|value| value.to_str().unwrap())
+			.collect();
+		assert_eq!(
+			vary_values,
+			vec!["Origin, Access-Control-Request-Method, Access-Control-Request-Headers",]
+		);
+	}
+}

src/api/common/generic_server.rs

@@ -84,21 +84,21 @@ impl<A: ApiHandler> ApiServer<A> {
 			region,
 			api_handler,
 			request_counter: meter
-				.u64_counter(format!("api.{}.request_counter", A::API_NAME))
+				.u64_counter(format!("garage_api.{}.request_count", A::API_NAME))
 				.with_description(format!(
 					"Number of API calls to the various {} API endpoints",
 					A::API_NAME_DISPLAY
 				))
 				.init(),
 			error_counter: meter
-				.u64_counter(format!("api.{}.error_counter", A::API_NAME))
+				.u64_counter(format!("garage_api.{}.error_count", A::API_NAME))
 				.with_description(format!(
 					"Number of API calls to the various {} API endpoints that resulted in errors",
 					A::API_NAME_DISPLAY
 				))
 				.init(),
 			request_duration: meter
-				.f64_value_recorder(format!("api.{}.request_duration", A::API_NAME))
+				.f64_value_recorder(format!("garage_api.{}.request_duration", A::API_NAME))
 				.with_description(format!(
 					"Duration of API calls to the various {} API endpoints",
 					A::API_NAME_DISPLAY

src/api/common/signature/payload.rs

@@ -357,7 +357,13 @@ pub fn canonical_request(
 		items.join("&")
 	};
 
-	// Canonical header string calculated from signed headers
+	// Canonical header string calculated from signed headers.
+	//
+	// Per the SigV4 spec, signed header values must have sequential
+	// internal whitespace collapsed to a single space, in addition to
+	// being trimmed. AWS SDKs do this before computing the signature
+	// but transmit the raw value on the wire, so we must match.
+	// -> https://docs.aws.amazon.com/IAM/latest/UserGuide/create-signed-request.html
 	let canonical_header_string = signed_headers
 		.iter()
 		.map(|name| {
@@ -372,7 +378,11 @@ pub fn canonical_request(
 				built_string.push(',');
 				built_string.push_str(extend_string);
 			}
-			Ok(format!("{}:{}", name.as_str(), built_string.trim()))
+			let normalized = built_string
+				.split_whitespace()
+				.collect::<Vec<_>>()
+				.join(" ");
+			Ok(format!("{}:{}", name.as_str(), normalized))
 		})
 		.collect::<Result<Vec<String>, Error>>()?
 		.join("\n");

src/api/k2v/api_server.rs

@@ -111,7 +111,7 @@ impl ApiHandler for K2VApiServer {
 			Method::GET | Method::HEAD | Method::POST => {
 				find_matching_cors_rule(&bucket_params, &req)
 					.ok_or_internal_error("Error looking up CORS rule")?
-					.cloned()
+					.map(|(rule, origin)| (rule.clone(), origin.to_string()))
 			}
 			_ => None,
 		};
@@ -164,8 +164,8 @@ impl ApiHandler for K2VApiServer {
 		// If request was a success and we have a CORS rule that applies to it,
 		// add the corresponding CORS headers to the response
 		let mut resp_ok = resp?;
-		if let Some(rule) = matching_cors_rule {
+		if let Some((rule, origin)) = matching_cors_rule {
-			add_cors_headers(&mut resp_ok, &rule)
+			add_cors_headers(&mut resp_ok, &rule, &origin)
 				.ok_or_internal_error("Invalid bucket CORS configuration")?;
 		}
 

src/api/s3/api_server.rs

@@ -159,7 +159,8 @@ impl ApiHandler for S3ApiServer {
 			return Err(Error::forbidden("Operation is not allowed for this key."));
 		}
 
-		let matching_cors_rule = find_matching_cors_rule(&bucket_params, &req)?.cloned();
+		let matching_cors = find_matching_cors_rule(&bucket_params, &req)?
+			.map(|(rule, origin)| (rule.clone(), origin.to_string()));
 
 		let ctx = ReqCtx {
 			garage,
@@ -334,8 +335,8 @@ impl ApiHandler for S3ApiServer {
 		// If request was a success and we have a CORS rule that applies to it,
 		// add the corresponding CORS headers to the response
 		let mut resp_ok = resp?;
-		if let Some(rule) = matching_cors_rule {
+		if let Some((rule, origin)) = matching_cors {
-			add_cors_headers(&mut resp_ok, &rule)
+			add_cors_headers(&mut resp_ok, &rule, &origin)
 				.ok_or_internal_error("Invalid bucket CORS configuration")?;
 		}
 

src/api/s3/bucket.rs

@@ -122,7 +122,7 @@ pub async fn handle_list_buckets(
 			for (alias, _, _active) in bucket.aliases().iter().filter(|(_, _, active)| *active) {
 				let alias_opt = garage.bucket_alias_table.get(&EmptyKey, alias).await?;
 				if let Some(alias_ent) = alias_opt {
-					if *alias_ent.state.get() == Some(*bucket_id) {
+					if alias_ent.state.get().inner() == Some(bucket_id) {
 						aliases.insert(alias_ent.name().to_string(), *bucket_id);
 					}
 				}
@@ -134,7 +134,7 @@ pub async fn handle_list_buckets(
 	}
 
 	for (alias, _, id_opt) in key_p.local_aliases.items() {
-		if let Some(id) = id_opt {
+		if let Some(id) = id_opt.inner() {
 			aliases.insert(alias.clone(), *id);
 		}
 	}
@@ -256,7 +256,10 @@ pub async fn handle_delete_bucket(ctx: ReqCtx) -> Result<Response<ResBody>, Erro
 
 	let key_params = api_key.params().unwrap();
 
-	let is_local_alias = matches!(key_params.local_aliases.get(bucket_name), Some(Some(_)));
+	let is_local_alias = matches!(
+		key_params.local_aliases.get(bucket_name).map(|x| x.inner()),
+		Some(Some(_))
+	);
 
 	// If the bucket has no other aliases, this is a true deletion.
 	// Otherwise, it is just an alias removal.

src/api/s3/cors.rs

@@ -13,7 +13,7 @@ use crate::xml::to_xml_with_header;
 
 pub async fn handle_get_cors(ctx: ReqCtx) -> Result<Response<ResBody>, Error> {
 	let ReqCtx { bucket_params, .. } = ctx;
-	if let Some(cors) = bucket_params.cors_config.get() {
+	if let Some(cors) = bucket_params.cors_config.get().inner() {
 		let wc = CorsConfiguration {
 			xmlns: (),
 			cors_rules: cors
@@ -38,7 +38,7 @@ pub async fn handle_delete_cors(ctx: ReqCtx) -> Result<Response<ResBody>, Error>
 		mut bucket_params,
 		..
 	} = ctx;
-	bucket_params.cors_config.update(None);
+	bucket_params.cors_config.update(None.into());
 	garage
 		.bucket_table
 		.insert(&Bucket::present(bucket_id, bucket_params))
@@ -67,7 +67,7 @@ pub async fn handle_put_cors(
 
 	bucket_params
 		.cors_config
-		.update(Some(conf.into_garage_cors_config()?));
+		.update(Some(conf.into_garage_cors_config()?).into());
 	garage
 		.bucket_table
 		.insert(&Bucket::present(bucket_id, bucket_params))

src/api/s3/lifecycle.rs

@@ -14,7 +14,7 @@ use garage_model::bucket_table::Bucket;
 pub async fn handle_get_lifecycle(ctx: ReqCtx) -> Result<Response<ResBody>, Error> {
 	let ReqCtx { bucket_params, .. } = ctx;
 
-	if let Some(lifecycle) = bucket_params.lifecycle_config.get() {
+	if let Some(lifecycle) = bucket_params.lifecycle_config.get().inner() {
 		let wc = LifecycleConfiguration::from_garage_lifecycle_config(lifecycle);
 		let xml = to_xml_with_header(&wc)?;
 		Ok(Response::builder()
@@ -33,7 +33,7 @@ pub async fn handle_delete_lifecycle(ctx: ReqCtx) -> Result<Response<ResBody>, E
 		mut bucket_params,
 		..
 	} = ctx;
-	bucket_params.lifecycle_config.update(None);
+	bucket_params.lifecycle_config.update(None.into());
 	garage
 		.bucket_table
 		.insert(&Bucket::present(bucket_id, bucket_params))
@@ -62,7 +62,7 @@ pub async fn handle_put_lifecycle(
 		.validate_into_garage_lifecycle_config()
 		.ok_or_bad_request("Invalid lifecycle configuration")?;
 
-	bucket_params.lifecycle_config.update(Some(config));
+	bucket_params.lifecycle_config.update(Some(config).into());
 	garage
 		.bucket_table
 		.insert(&Bucket::present(bucket_id, bucket_params))

src/api/s3/post_object.rs

@@ -121,7 +121,7 @@ pub async fn handle_post_object(
 		&bucket_params,
 		&Request::from_parts(head.clone(), empty_body::<Infallible>()),
 	)?
-	.cloned();
+	.map(|(rule, origin)| (rule.clone(), origin.to_string()));
 
 	let decoded_policy = BASE64_STANDARD
 		.decode(policy)
@@ -351,8 +351,8 @@ pub async fn handle_post_object(
 		}
 	};
 
-	if let Some(rule) = matching_cors_rule {
+	if let Some((rule, origin)) = matching_cors_rule {
-		add_cors_headers(&mut resp, &rule)
+		add_cors_headers(&mut resp, &rule, &origin)
 			.ok_or_internal_error("Invalid bucket CORS configuration")?;
 	}
 
@@ -473,12 +473,10 @@ where
 					))));
 				}
 			}
-			Poll::Ready(None) => {
+			Poll::Ready(None) if !self.length.contains(&self.read) => {
-				if !self.length.contains(&self.read) {
+				return Poll::Ready(Some(Err(Error::bad_request(
-					return Poll::Ready(Some(Err(Error::bad_request(
+					"File size does not match policy",
-						"File size does not match policy",
+				))));
-					))));
-				}
 			}
 			_ => {}
 		}

src/api/s3/router.rs

@@ -315,7 +315,11 @@ impl Endpoint {
 		bucket: Option<String>,
 	) -> Result<(Self, Option<String>), Error> {
 		let uri = req.uri();
-		let path = uri.path().trim_start_matches('/');
+		let path = uri.path().strip_prefix('/');
+		if path.is_none() {
+			return Err(Error::bad_request("URI path must start with a '/'"));
+		}
+		let path = path.unwrap();
 		let query = uri.query();
 		if bucket.is_none() && path.is_empty() {
 			if *req.method() == Method::OPTIONS {
@@ -329,7 +333,7 @@ impl Endpoint {
 			(bucket, path)
 		} else {
 			path.split_once('/')
-				.map(|(b, p)| (b.to_owned(), p.trim_start_matches('/')))
+				.map(|(b, p)| (b.to_owned(), p))
 				.unwrap_or_else(|| (path.to_owned(), ""))
 		};
 
@@ -843,6 +847,40 @@ mod tests {
 			"&+?%é/something"
 		);
 
+		// A double-slash in the URL means the key begins with '/'.
+		// path-style: HEAD /bucket// → key "/"
+		assert_eq!(
+			parse("HEAD", "/my_bucket//", None, None)
+				.0
+				.get_key()
+				.unwrap(),
+			"/"
+		);
+		// virtual-hosted-style: HEAD // → key "/"
+		assert_eq!(
+			parse("HEAD", "//", Some("my_bucket".to_owned()), None)
+				.0
+				.get_key()
+				.unwrap(),
+			"/"
+		);
+		// same for GET: path-style GET /bucket// → key "/"
+		assert_eq!(
+			parse("GET", "/my_bucket//", None, None)
+				.0
+				.get_key()
+				.unwrap(),
+			"/"
+		);
+		// virtual-hosted-style: GET // → key "/"
+		assert_eq!(
+			parse("GET", "//", Some("my_bucket".to_owned()), None)
+				.0
+				.get_key()
+				.unwrap(),
+			"/"
+		);
+
 		/*
 		 * this case is failing. We should verify how clients encode space in url
 		assert_eq!(
@@ -933,6 +971,7 @@ mod tests {
 			GET "/{Key+}?torrent" => GetObjectTorrent
 			GET "/?publicAccessBlock" => GetPublicAccessBlock
 			HEAD "/" => HeadBucket
+			HEAD "//" => HeadObject
 			HEAD "/my-image.jpg" => HeadObject
 			HEAD "/my-image.jpg?versionId=3HL4kqCxf3vjVBH40Nrjfkd" => HeadObject
 			HEAD "/Key+?partNumber=3&versionId=VersionId" => HeadObject
@@ -949,6 +988,7 @@ mod tests {
 			GET "/?uploads&delimiter=/&prefix=photos/2006/" => ListMultipartUploads
 			GET "/?uploads&delimiter=D&encoding-type=EncodingType&key-marker=KeyMarker&max-uploads=1&prefix=Prefix&upload-id-marker=UploadIdMarker" => ListMultipartUploads
 			GET "/" => ListObjects
+			GET "//" => GetObject
 			GET "/?prefix=N&marker=Need&max-keys=40" => ListObjects
 			GET "/?delimiter=/" => ListObjects
 			GET "/?prefix=photos/2006/&delimiter=/" => ListObjects

src/api/s3/website.rs

@@ -16,7 +16,7 @@ pub const X_AMZ_WEBSITE_REDIRECT_LOCATION: HeaderName =
 
 pub async fn handle_get_website(ctx: ReqCtx) -> Result<Response<ResBody>, Error> {
 	let ReqCtx { bucket_params, .. } = ctx;
-	if let Some(website) = bucket_params.website_config.get() {
+	if let Some(website) = bucket_params.website_config.get().inner() {
 		let wc = WebsiteConfiguration {
 			xmlns: (),
 			error_document: website.error_document.as_ref().map(|v| Key {
@@ -54,7 +54,7 @@ pub async fn handle_delete_website(ctx: ReqCtx) -> Result<Response<ResBody>, Err
 		mut bucket_params,
 		..
 	} = ctx;
-	bucket_params.website_config.update(None);
+	bucket_params.website_config.update(None.into());
 	garage
 		.bucket_table
 		.insert(&Bucket::present(bucket_id, bucket_params))
@@ -83,7 +83,7 @@ pub async fn handle_put_website(
 
 	bucket_params
 		.website_config
-		.update(Some(conf.into_garage_website_config()?));
+		.update(Some(conf.into_garage_website_config()?).into());
 	garage
 		.bucket_table
 		.insert(&Bucket::present(bucket_id, bucket_params))

src/block/manager.rs

@@ -224,7 +224,13 @@ impl BlockManager {
 				|p, tranquility| p.set_with(|x| x.tranquility = tranquility),
 			);
 			vars.register_ro(&self.scrub_persister, "scrub-last-completed", |p| {
-				p.get_with(|x| msec_to_rfc3339(x.time_last_complete_scrub))
+				p.get_with(|x| {
+					if x.time_last_complete_scrub == 0 {
+						"never".to_string()
+					} else {
+						msec_to_rfc3339(x.time_last_complete_scrub)
+					}
+				})
 			});
 			vars.register_ro(&self.scrub_persister, "scrub-next-run", |p| {
 				p.get_with(|x| msec_to_rfc3339(x.time_next_run_scrub))

src/block/metrics.rs

@@ -41,7 +41,7 @@ impl BlockManagerMetrics {
 		let meter = global::meter("garage_model/block");
 		Self {
 			_compression_level: meter
-				.u64_value_observer("block.compression_level", move |observer| {
+				.u64_value_observer("garage_block.compression_level", move |observer| {
 					match compression_level {
 						Some(v) => observer.observe(v as u64, &[]),
 						None => observer.observe(0_u64, &[]),
@@ -50,7 +50,7 @@ impl BlockManagerMetrics {
 				.with_description("Garage compression level for node")
 				.init(),
 			_rc_size: meter
-				.u64_value_observer("block.rc_size", move |observer| {
+				.u64_value_observer("garage_block.rc_size", move |observer| {
 					if let Ok(value) = rc_tree.approximate_len() {
 						observer.observe(value as u64, &[]);
 					}
@@ -58,7 +58,7 @@ impl BlockManagerMetrics {
 				.with_description("Number of blocks known to the reference counter")
 				.init(),
 			_resync_queue_len: meter
-				.u64_value_observer("block.resync_queue_length", move |observer| {
+				.u64_value_observer("garage_block.resync_queue_length", move |observer| {
 					if let Ok(value) = resync_queue.approximate_len() {
 						observer.observe(value as u64, &[]);
 					}
@@ -68,7 +68,7 @@ impl BlockManagerMetrics {
 				)
 				.init(),
 			_resync_errored_blocks: meter
-				.u64_value_observer("block.resync_errored_blocks", move |observer| {
+				.u64_value_observer("garage_block.resync_errored_blocks", move |observer| {
 					if let Ok(value) = resync_errors.approximate_len() {
 						observer.observe(value as u64, &[]);
 					}
@@ -77,7 +77,7 @@ impl BlockManagerMetrics {
 				.init(),
 
 			_buffer_free_kb: meter
-				.u64_value_observer("block.ram_buffer_free_kb", move |observer| {
+				.u64_value_observer("garage_block.ram_buffer_free_kb", move |observer| {
 					observer.observe(buffer_semaphore.available_permits() as u64, &[]);
 				})
 				.with_description(
@@ -86,63 +86,63 @@ impl BlockManagerMetrics {
 				.init(),
 
 			resync_counter: meter
-				.u64_counter("block.resync_counter")
+				.u64_counter("garage_block.resync_count")
 				.with_description("Number of calls to resync_block")
 				.init()
 				.bind(&[]),
 			resync_error_counter: meter
-				.u64_counter("block.resync_error_counter")
+				.u64_counter("garage_block.resync_error_count")
 				.with_description("Number of calls to resync_block that returned an error")
 				.init()
 				.bind(&[]),
 			resync_duration: meter
-				.f64_value_recorder("block.resync_duration")
+				.f64_value_recorder("garage_block.resync_duration")
 				.with_description("Duration of resync_block operations")
 				.init()
 				.bind(&[]),
 			resync_send_counter: meter
-				.u64_counter("block.resync_send_counter")
+				.u64_counter("garage_block.resync_send_count")
 				.with_description("Number of blocks sent to another node in resync operations")
 				.init(),
 			resync_recv_counter: meter
-				.u64_counter("block.resync_recv_counter")
+				.u64_counter("garage_block.resync_recv_count")
 				.with_description("Number of blocks received from other nodes in resync operations")
 				.init()
 				.bind(&[]),
 
 			bytes_read: meter
-				.u64_counter("block.bytes_read")
+				.u64_counter("garage_block.bytes_read")
 				.with_description("Number of bytes read from disk")
 				.init()
 				.bind(&[]),
 			block_read_duration: meter
-				.f64_value_recorder("block.read_duration")
+				.f64_value_recorder("garage_block.read_duration")
 				.with_description("Duration of block read operations")
 				.init()
 				.bind(&[]),
 			block_read_semaphore_timeouts: meter
-				.u64_counter("block.read_semaphore_timeouts")
+				.u64_counter("garage_block.read_semaphore_timeouts")
 				.with_description("Number of block reads that failed due to semaphore acquire timeout")
 				.init()
 				.bind(&[]),
 			bytes_written: meter
-				.u64_counter("block.bytes_written")
+				.u64_counter("garage_block.bytes_written")
 				.with_description("Number of bytes written to disk")
 				.init()
 				.bind(&[]),
 			block_write_duration: meter
-				.f64_value_recorder("block.write_duration")
+				.f64_value_recorder("garage_block.write_duration")
 				.with_description("Duration of block write operations")
 				.init()
 				.bind(&[]),
 			delete_counter: meter
-				.u64_counter("block.delete_counter")
+				.u64_counter("garage_block.delete_count")
 				.with_description("Number of blocks deleted")
 				.init()
 				.bind(&[]),
 
 			corruption_counter: meter
-				.u64_counter("block.corruption_counter")
+				.u64_counter("garage_block.corruption_count")
 				.with_description("Data corruptions detected on block reads")
 				.init()
 				.bind(&[]),

src/block/repair.rs

@@ -648,7 +648,7 @@ impl BlockStoreIterator {
 
 		let mut cum_cap = 0;
 		let mut todo = vec![];
-		for (dir, cap) in data_layout.data_dirs.iter().zip(dir_cap.into_iter()) {
+		for (dir, cap) in data_layout.data_dirs.iter().zip(dir_cap) {
 			let progress_min = (cum_cap * PROGRESS_FP) / sum_cap;
 			let progress_max = ((cum_cap + cap as u64) * PROGRESS_FP) / sum_cap;
 			cum_cap += cap as u64;

src/db/sqlite_adapter.rs

@@ -426,15 +426,20 @@ impl<'a> ITx for SqliteTx<'a> {
 // complicated, they must hold the Statement and Row objects
 // therefore quite some unsafe code (it is a self-referential struct)
 
-struct DbValueIterator<'a> {
+struct DbValueIterator {
 	db: Connection,
-	stmt: Option<Statement<'a>>,
+	// These two are not really static (they are actually self referential :o)
-	iter: Option<Rows<'a>>,
+	stmt: Option<Statement<'static>>,
+	iter: Option<Rows<'static>>,
 	_pin: PhantomPinned,
 }
 
-impl<'a> DbValueIterator<'a> {
+impl DbValueIterator {
-	fn make<P: rusqlite::Params>(db: Connection, sql: &str, args: P) -> Result<ValueIter<'a>> {
+	fn make<'res, P: rusqlite::Params>(
+		db: Connection,
+		sql: &str,
+		args: P,
+	) -> Result<ValueIter<'res>> {
 		let res = DbValueIterator {
 			db,
 			stmt: None,
@@ -468,7 +473,7 @@ impl<'a> DbValueIterator<'a> {
 	}
 }
 
-impl<'a> Drop for DbValueIterator<'a> {
+impl Drop for DbValueIterator {
 	fn drop(&mut self) {
 		trace!("drop iter");
 		drop(self.iter.take());
@@ -476,9 +481,9 @@ impl<'a> Drop for DbValueIterator<'a> {
 	}
 }
 
-struct DbValueIteratorPin<'a>(Pin<Box<DbValueIterator<'a>>>);
+struct DbValueIteratorPin(Pin<Box<DbValueIterator>>);
 
-impl<'a> Iterator for DbValueIteratorPin<'a> {
+impl Iterator for DbValueIteratorPin {
 	type Item = Result<(Value, Value)>;
 
 	fn next(&mut self) -> Option<Self::Item> {

src/garage/cli/remote/cluster.rs

@@ -9,6 +9,31 @@ use crate::cli::remote::*;
 use crate::cli::structs::*;
 
 impl Cli {
+	pub async fn cmd_health(&self, quiet: bool) -> Result<(), Error> {
+		let health = self.api_request(GetClusterHealthRequest).await?;
+
+		if !quiet {
+			let table = vec![
+				format!("Cluster health:\t{}", health.status.to_uppercase()),
+				format!("Known nodes:\t{}", health.known_nodes),
+				format!("Connected nodes:\t{}", health.connected_nodes),
+				format!("Storage nodes:\t{}", health.storage_nodes),
+				format!("Storage nodes up:\t{}", health.storage_nodes_up),
+				format!("Partitions:\t{}", health.partitions),
+				format!("Partitions with quorum:\t{}", health.partitions_quorum),
+				format!("Fully healthy partitions:\t{}", health.partitions_all_ok),
+			];
+			format_table(table);
+		}
+
+		match health.status.as_str() {
+			"unavailable" => Err(Error::Message(
+				"Cluster is currently unavailable".to_string(),
+			)),
+			_ => Ok(()),
+		}
+	}
+
 	pub async fn cmd_status(&self) -> Result<(), Error> {
 		let status = self.api_request(GetClusterStatusRequest).await?;
 		let layout = self.api_request(GetClusterLayoutRequest).await?;

src/garage/cli/remote/layout.rs

@@ -40,6 +40,7 @@ impl Cli {
 				PreviewClusterLayoutChangesResponse::Success {
 					message,
 					new_layout,
+					..
 				} => {
 					println!();
 					println!("==== NEW CLUSTER LAYOUT AFTER APPLYING CHANGES ====");

src/garage/cli/remote/mod.rs

@@ -33,6 +33,7 @@ impl Cli {
 	pub async fn handle(&self, cmd: Command) -> Result<(), Error> {
 		match cmd {
 			Command::Status => self.cmd_status().await,
+			Command::Health(opt) => self.cmd_health(opt.quiet).await,
 			Command::Node(NodeOperation::Connect(connect_opt)) => {
 				self.cmd_connect(connect_opt).await
 			}

src/garage/cli/structs.rs

@@ -10,6 +10,10 @@ pub enum Command {
 	#[structopt(name = "server", version = garage_version())]
 	Server(ServerOpt),
 
+	/// Check the cluster health and set the exit code to 1 if it is unavailable
+	#[structopt(name = "health", version = garage_version())]
+	Health(HealthOpt),
+
 	/// Get network status
 	#[structopt(name = "status", version = garage_version())]
 	Status,
@@ -103,6 +107,17 @@ pub struct ServerOpt {
 	pub(crate) default_bucket: bool,
 }
 
+// -------------------------
+// ---- garage health ----
+// -------------------------
+
+#[derive(StructOpt, Debug)]
+pub struct HealthOpt {
+	/// Do not print healthyness to stdout
+	#[structopt(short = "q", long = "quiet")]
+	pub(crate) quiet: bool,
+}
+
 // -------------------------
 // ---- garage node ... ----
 // -------------------------

src/garage/secrets.rs

@@ -19,7 +19,12 @@ pub struct Secrets {
 
 	/// RPC secret network key, used to replace `rpc_secret` in config.toml when running the
 	/// daemon or doing admin operations
-	#[structopt(short = "s", long = "rpc-secret", env = "GARAGE_RPC_SECRET")]
+	#[structopt(
+		short = "s",
+		long = "rpc-secret",
+		env = "GARAGE_RPC_SECRET",
+		hide_env_values = true
+	)]
 	pub rpc_secret: Option<String>,
 
 	/// RPC secret network key, used to replace `rpc_secret` in config.toml and rpc-secret
@@ -29,7 +34,11 @@ pub struct Secrets {
 
 	/// Admin API authentication token, replaces `admin.admin_token` in config.toml when
 	/// running the Garage daemon
-	#[structopt(long = "admin-token", env = "GARAGE_ADMIN_TOKEN")]
+	#[structopt(
+		long = "admin-token",
+		env = "GARAGE_ADMIN_TOKEN",
+		hide_env_values = true
+	)]
 	pub admin_token: Option<String>,
 
 	/// Admin API authentication token file path, replaces `admin.admin_token` in config.toml
@@ -39,7 +48,11 @@ pub struct Secrets {
 
 	/// Metrics API authentication token, replaces `admin.metrics_token` in config.toml when
 	/// running the Garage daemon
-	#[structopt(long = "metrics-token", env = "GARAGE_METRICS_TOKEN")]
+	#[structopt(
+		long = "metrics-token",
+		env = "GARAGE_METRICS_TOKEN",
+		hide_env_values = true
+	)]
 	pub metrics_token: Option<String>,
 
 	/// Metrics API authentication token file path, replaces `admin.metrics_token` in config.toml

src/garage/server.rs

@@ -228,10 +228,10 @@ async fn initial_config(garage: &Arc<Garage>, opt: ServerOpt) -> Result<(), Erro
 				})),
 			);
 
-			let (layout, msg) = layout.apply_staged_changes(1)?;
+			let (layout, stat) = layout.apply_staged_changes(1)?;
 			info!(
 				"Created initial layout for single-node configuration:\n{}",
-				msg.join("\n")
+				stat.to_message().join("\n")
 			);
 
 			garage

src/garage/tests/common/garage.rs

@@ -194,6 +194,15 @@ api_bind_addr = "127.0.0.1:{admin_port}"
 			.expect("Could not build garage endpoint URI")
 	}
 
+	pub fn admin_uri(&self, path: &str) -> http::Uri {
+		format!(
+			"http://127.0.0.1:{admin_port}/{path}",
+			admin_port = self.admin_port,
+		)
+		.parse()
+		.expect("Could not build garage endpoint URI")
+	}
+
 	pub fn key(&self, maybe_name: Option<&str>) -> Key {
 		let mut key = Key::default();
 

src/garage/tests/k2v/item.rs

@@ -11,6 +11,7 @@ use http_body_util::BodyExt;
 use hyper::{Method, StatusCode};
 
 #[tokio::test]
+#[ignore = "flaky"]
 async fn test_items_and_indices() {
 	let ctx = common::context();
 	let bucket = ctx.create_bucket("test-k2v-item-and-index");

src/garage/tests/lib.rs

@@ -4,6 +4,9 @@ mod common;
 mod admin;
 mod bucket;
 
+#[cfg(feature = "metrics")]
+mod metrics;
+
 mod s3;
 
 #[cfg(feature = "k2v")]

src/garage/tests/metrics.rs

@@ -0,0 +1,49 @@
+use bytes::Bytes;
+use http::{Request, StatusCode};
+use http_body_util::{BodyExt, Full};
+
+use crate::common;
+
+#[tokio::test]
+async fn check_metrics_name() {
+	let ctx = common::context();
+
+	let req_url = ctx.garage.admin_uri("metrics");
+	let client = ctx.custom_request.client();
+	let get_metrics_req = Request::builder()
+		.method("GET")
+		.uri(req_url)
+		.body(Full::new(Bytes::new()))
+		.unwrap();
+
+	let response = client
+		.request(get_metrics_req)
+		.await
+		.expect("failed to build 'get metrics' request");
+
+	assert_eq!(response.status(), StatusCode::OK);
+	let body = BodyExt::collect(response.into_body())
+		.await
+		.expect("failed to collect bytes from body stream")
+		.to_bytes();
+	let body = String::from_utf8_lossy(&body);
+
+	//dbg!(&body);
+	let invalid_metrics_name = body
+		.lines()
+		.filter(isnot_comment_line) // skip the comment lines
+		.filter(hasnt_prefix_garage)
+		.collect::<Vec<_>>();
+
+	if !invalid_metrics_name.is_empty() {
+		panic!("metrics name should all start with 'garage_' prefix.\nDoc: https://prometheus.io/docs/practices/naming/#metric-names\n\nInvalid:\n{:#?}", invalid_metrics_name);
+	}
+}
+
+fn isnot_comment_line(line: &&str) -> bool {
+	!line.starts_with("#")
+}
+
+fn hasnt_prefix_garage(line: &&str) -> bool {
+	!line.starts_with("garage_")
+}

src/garage/tests/s3/cors.rs

@@ -0,0 +1,121 @@
+use aws_sdk_s3::types::{CorsConfiguration, CorsRule};
+use hyper::{Method, StatusCode};
+
+use crate::common;
+
+const REQUEST_ORIGIN: &str = "https://app.example.test";
+const SECOND_ALLOWED_ORIGIN: &str = "https://admin.example.test";
+const OBJECT_KEY: &str = "probe.txt";
+const BODY: &[u8] = b"hello from integration repro\n";
+
+async fn send_preflight(
+	ctx: &common::Context,
+	bucket: &str,
+	origin: &str,
+) -> hyper::Response<common::custom_requester::Body> {
+	ctx.custom_request
+		.builder(bucket.to_string())
+		.method(Method::OPTIONS)
+		.path(OBJECT_KEY)
+		.unsigned_header("origin", origin)
+		.unsigned_header("access-control-request-method", "PUT")
+		.unsigned_header(
+			"access-control-request-headers",
+			"content-type,x-amz-meta-demo",
+		)
+		.body(vec![])
+		.send()
+		.await
+		.unwrap()
+}
+
+async fn send_put(
+	ctx: &common::Context,
+	bucket: &str,
+	origin: &str,
+) -> hyper::Response<common::custom_requester::Body> {
+	ctx.custom_request
+		.builder(bucket.to_string())
+		.method(Method::PUT)
+		.path(OBJECT_KEY)
+		.signed_header("content-type", "text/plain")
+		.signed_header("x-amz-meta-demo", "1")
+		.unsigned_header("origin", origin)
+		.body(BODY.to_vec())
+		.send()
+		.await
+		.unwrap()
+}
+
+async fn apply_bucket_cors(ctx: &common::Context, bucket: &str, allowed_origins: &[&str]) {
+	let rule = allowed_origins.iter().fold(
+		CorsRule::builder()
+			.allowed_headers("*")
+			.allowed_methods("PUT")
+			.expose_headers("ETag"),
+		|rule, origin| rule.allowed_origins(*origin),
+	);
+
+	let cors = CorsConfiguration::builder()
+		.cors_rules(rule.build().unwrap())
+		.build()
+		.unwrap();
+
+	ctx.client
+		.put_bucket_cors()
+		.bucket(bucket)
+		.cors_configuration(cors)
+		.send()
+		.await
+		.unwrap();
+}
+
+#[tokio::test]
+async fn test_s3_api_cors_reflects_request_origin() {
+	let ctx = common::context();
+	let bucket = ctx.create_bucket("s3-cors-direct");
+
+	apply_bucket_cors(&ctx, &bucket, &[REQUEST_ORIGIN]).await;
+
+	let control_preflight = send_preflight(&ctx, &bucket, REQUEST_ORIGIN).await;
+	assert_eq!(control_preflight.status(), StatusCode::OK);
+	assert_eq!(
+		control_preflight
+			.headers()
+			.get("access-control-allow-origin")
+			.unwrap(),
+		REQUEST_ORIGIN
+	);
+
+	let control_put = send_put(&ctx, &bucket, REQUEST_ORIGIN).await;
+	assert_eq!(control_put.status(), StatusCode::OK);
+	assert_eq!(
+		control_put
+			.headers()
+			.get("access-control-allow-origin")
+			.unwrap(),
+		REQUEST_ORIGIN
+	);
+
+	apply_bucket_cors(&ctx, &bucket, &[REQUEST_ORIGIN, SECOND_ALLOWED_ORIGIN]).await;
+
+	let repro_preflight = send_preflight(&ctx, &bucket, REQUEST_ORIGIN).await;
+	assert_eq!(repro_preflight.status(), StatusCode::OK);
+	assert_eq!(
+		repro_preflight
+			.headers()
+			.get("access-control-allow-origin")
+			.unwrap(),
+		REQUEST_ORIGIN
+	);
+
+	let repro_put = send_put(&ctx, &bucket, REQUEST_ORIGIN).await;
+	assert_eq!(repro_put.status(), StatusCode::OK);
+	assert_eq!(
+		repro_put
+			.headers()
+			.get("access-control-allow-origin")
+			.unwrap(),
+		REQUEST_ORIGIN
+	);
+}

src/garage/tests/s3/mod.rs

@@ -1,3 +1,4 @@
+mod cors;
 mod list;
 mod multipart;
 mod objects;

src/garage/tests/s3/presigned.rs

@@ -70,3 +70,43 @@ async fn test_presigned_url() {
 		assert_eq!(body, body2);
 	}
 }
+
+// Presigned PUT with a user-metadata header whose value contains
+// internal sequential whitespace. SigV4 requires collapsing such
+// whitespace in canonical header values; missing that normalization
+// produces an `Invalid signature` 403 on otherwise-valid requests.
+#[tokio::test]
+async fn test_presigned_put_with_user_metadata() {
+	let ctx = common::context();
+	let bucket = ctx.create_bucket("presigned-metadata");
+
+	let key = "cache-archive";
+	let metadata_value = "cache-key  --protected";
+	let body = Bytes::from_static(b"presigned PUT with user metadata");
+
+	let psc = PresigningConfig::builder()
+		.start_time(SystemTime::now() - Duration::from_secs(60))
+		.expires_in(Duration::from_secs(3600))
+		.build()
+		.unwrap();
+
+	let presigned = ctx
+		.client
+		.put_object()
+		.bucket(&bucket)
+		.key(key)
+		.metadata("cachekey", metadata_value)
+		.presigned(psc)
+		.await
+		.unwrap();
+
+	let req_builder = Request::builder().method("PUT").uri(presigned.uri());
+	let req = presigned
+		.headers()
+		.fold(req_builder, |b, (k, v)| b.header(k, v))
+		.body(Full::new(body))
+		.unwrap();
+
+	let res = ctx.custom_request.client().request(req).await.unwrap();
+	assert_eq!(res.status(), 200);
+}

src/model/Cargo.toml

@@ -21,6 +21,7 @@ garage_block.workspace = true
 garage_util.workspace = true
 garage_net.workspace = true
 
+arbitrary = { optional = true, workspace = true }
 argon2.workspace = true
 async-trait.workspace = true
 blake2.workspace = true
@@ -46,6 +47,7 @@ k2v = ["garage_util/k2v"]
 lmdb = ["garage_db/lmdb"]
 sqlite = ["garage_db/sqlite"]
 fjall = ["garage_db/fjall"]
+arbitrary = ["dep:arbitrary","garage_util/arbitrary"]
 
 [lints]
 workspace = true

src/model/admin_token_table.rs

@@ -8,6 +8,7 @@ use garage_table::{EmptyKey, Entry, TableSchema};
 pub use crate::key_table::KeyFilter;
 
 mod v2 {
+	use crate::permission::ExpirationTime;
 	use garage_util::crdt;
 	use serde::{Deserialize, Serialize};
 
@@ -35,7 +36,7 @@ mod v2 {
 		pub name: crdt::Lww<String>,
 
 		/// The optional time of expiration of the token
-		pub expiration: crdt::Lww<Option<u64>>,
+		pub expiration: crdt::Lww<crdt::MergingOption<ExpirationTime>>,
 
 		/// The scope of the token, i.e. list of authorized admin API calls
 		pub scope: crdt::Lww<AdminApiTokenScope>,
@@ -51,6 +52,14 @@ mod v2 {
 
 pub use v2::*;
 
+#[cfg(feature = "arbitrary")]
+impl<'a> arbitrary::Arbitrary<'a> for AdminApiTokenScope {
+	fn arbitrary(u: &mut arbitrary::Unstructured<'a>) -> arbitrary::Result<Self> {
+		let set: std::collections::BTreeSet<String> = arbitrary::Arbitrary::arbitrary(u)?;
+		Ok(AdminApiTokenScope(set.into_iter().collect()))
+	}
+}
+
 impl Crdt for AdminApiTokenParams {
 	fn merge(&mut self, o: &Self) {
 		self.name.merge(&o.name);
@@ -98,7 +107,7 @@ impl AdminApiToken {
 				created: now_msec(),
 				token_hash: hashed_token,
 				name: crdt::Lww::new(name.to_string()),
-				expiration: crdt::Lww::new(None),
+				expiration: crdt::Lww::new(None.into()),
 				scope: crdt::Lww::new(AdminApiTokenScope(vec!["*".to_string()])),
 			}),
 		};
@@ -139,9 +148,9 @@ impl AdminApiToken {
 
 impl AdminApiTokenParams {
 	pub fn is_expired(&self, ts_now: u64) -> bool {
-		match *self.expiration.get() {
+		match self.expiration.get().inner() {
 			None => false,
-			Some(exp) => ts_now >= exp,
+			Some(exp) => ts_now >= exp.0,
 		}
 	}
 

src/model/bucket_alias_table.rs

@@ -13,7 +13,7 @@ mod v08 {
 	#[derive(PartialEq, Eq, Clone, Debug, Serialize, Deserialize)]
 	pub struct BucketAlias {
 		pub(super) name: String,
-		pub state: crdt::Lww<Option<Uuid>>,
+		pub state: crdt::Lww<crdt::CancelingOption<Uuid>>,
 	}
 
 	impl garage_util::migrate::InitialFormat for BucketAlias {}
@@ -25,12 +25,12 @@ impl BucketAlias {
 	pub fn new(name: String, ts: u64, bucket_id: Option<Uuid>) -> Self {
 		BucketAlias {
 			name,
-			state: crdt::Lww::raw(ts, bucket_id),
+			state: crdt::Lww::raw(ts, CancelingOption(bucket_id)),
 		}
 	}
 
 	pub fn is_deleted(&self) -> bool {
-		self.state.get().is_none()
+		self.state.get().inner().is_none()
 	}
 	pub fn name(&self) -> &str {
 		&self.name

src/model/bucket_table.rs

@@ -45,12 +45,12 @@ mod v08 {
 		/// Whether this bucket is allowed for website access
 		/// (under all of its global alias names),
 		/// and if so, the website configuration XML document
-		pub website_config: crdt::Lww<Option<WebsiteConfig>>,
+		pub website_config: crdt::Lww<crdt::CancelingOption<WebsiteConfig>>,
 		/// CORS rules
-		pub cors_config: crdt::Lww<Option<Vec<CorsRule>>>,
+		pub cors_config: crdt::Lww<crdt::CancelingOption<Vec<CorsRule>>>,
 		/// Lifecycle configuration
 		#[serde(default)]
-		pub lifecycle_config: crdt::Lww<Option<Vec<LifecycleRule>>>,
+		pub lifecycle_config: crdt::Lww<crdt::CancelingOption<Vec<LifecycleRule>>>,
 		/// Bucket quotas
 		#[serde(default)]
 		pub quotas: crdt::Lww<BucketQuotas>,
@@ -63,6 +63,7 @@ mod v08 {
 	}
 
 	#[derive(PartialEq, Eq, Clone, Debug, Serialize, Deserialize)]
+	#[cfg_attr(feature = "arbitrary", derive(arbitrary::Arbitrary))]
 	pub struct CorsRule {
 		pub id: Option<String>,
 		pub max_age_seconds: Option<u64>,
@@ -74,6 +75,7 @@ mod v08 {
 
 	/// Lifecycle configuration rule
 	#[derive(PartialEq, Eq, Clone, Debug, Serialize, Deserialize)]
+	#[cfg_attr(feature = "arbitrary", derive(arbitrary::Arbitrary))]
 	pub struct LifecycleRule {
 		/// The ID of the rule
 		pub id: Option<String>,
@@ -91,6 +93,7 @@ mod v08 {
 	/// For each condition, if it is None, it is not verified (always true),
 	/// and if it is Some(x), then it is verified for value x
 	#[derive(PartialEq, Eq, Clone, Debug, Serialize, Deserialize, Default)]
+	#[cfg_attr(feature = "arbitrary", derive(arbitrary::Arbitrary))]
 	pub struct LifecycleFilter {
 		/// If Some(x), object key has to start with prefix x
 		pub prefix: Option<String>,
@@ -101,6 +104,7 @@ mod v08 {
 	}
 
 	#[derive(PartialEq, Eq, Clone, Debug, Serialize, Deserialize)]
+	#[cfg_attr(feature = "arbitrary", derive(arbitrary::Arbitrary))]
 	pub enum LifecycleExpiration {
 		/// Objects expire x days after they were created
 		AfterDays(usize),
@@ -109,6 +113,7 @@ mod v08 {
 	}
 
 	#[derive(Default, PartialEq, Eq, PartialOrd, Ord, Clone, Debug, Serialize, Deserialize)]
+	#[cfg_attr(feature = "arbitrary", derive(arbitrary::Arbitrary))]
 	pub struct BucketQuotas {
 		/// Maximum size in bytes (bucket size = sum of sizes of objects in the bucket)
 		pub max_size: Option<u64>,
@@ -139,6 +144,7 @@ mod v2 {
 
 	/// Configuration for a bucket
 	#[derive(PartialEq, Eq, Clone, Debug, Serialize, Deserialize)]
+	#[cfg_attr(feature = "arbitrary", derive(arbitrary::Arbitrary))]
 	pub struct BucketParams {
 		/// Bucket's creation date
 		pub creation_date: u64,
@@ -158,16 +164,17 @@ mod v2 {
 		/// Whether this bucket is allowed for website access
 		/// (under all of its global alias names),
 		/// and if so, the website configuration XML document
-		pub website_config: crdt::Lww<Option<WebsiteConfig>>,
+		pub website_config: crdt::Lww<crdt::CancelingOption<WebsiteConfig>>,
 		/// CORS rules
-		pub cors_config: crdt::Lww<Option<Vec<CorsRule>>>,
+		pub cors_config: crdt::Lww<crdt::CancelingOption<Vec<CorsRule>>>,
 		/// Lifecycle configuration
-		pub lifecycle_config: crdt::Lww<Option<Vec<LifecycleRule>>>,
+		pub lifecycle_config: crdt::Lww<crdt::CancelingOption<Vec<LifecycleRule>>>,
 		/// Bucket quotas
 		pub quotas: crdt::Lww<BucketQuotas>,
 	}
 
 	#[derive(PartialEq, Eq, Clone, Debug, Serialize, Deserialize)]
+	#[cfg_attr(feature = "arbitrary", derive(arbitrary::Arbitrary))]
 	pub struct WebsiteConfig {
 		pub index_document: String,
 		pub error_document: Option<String>,
@@ -178,24 +185,28 @@ mod v2 {
 	}
 
 	#[derive(PartialEq, Eq, Clone, Debug, Serialize, Deserialize)]
+	#[cfg_attr(feature = "arbitrary", derive(arbitrary::Arbitrary))]
 	pub struct RedirectAll {
 		pub hostname: String,
 		pub protocol: String,
 	}
 
 	#[derive(PartialEq, Eq, Clone, Debug, Serialize, Deserialize)]
+	#[cfg_attr(feature = "arbitrary", derive(arbitrary::Arbitrary))]
 	pub struct RoutingRule {
 		pub condition: Option<RedirectCondition>,
 		pub redirect: Redirect,
 	}
 
 	#[derive(PartialEq, Eq, Clone, Debug, Serialize, Deserialize)]
+	#[cfg_attr(feature = "arbitrary", derive(arbitrary::Arbitrary))]
 	pub struct RedirectCondition {
 		pub http_error_code: Option<u16>,
 		pub prefix: Option<String>,
 	}
 
 	#[derive(PartialEq, Eq, Clone, Debug, Serialize, Deserialize)]
+	#[cfg_attr(feature = "arbitrary", derive(arbitrary::Arbitrary))]
 	pub struct Redirect {
 		pub hostname: Option<String>,
 		pub http_redirect_code: u16,
@@ -248,9 +259,9 @@ impl BucketParams {
 			authorized_keys: crdt::Map::new(),
 			aliases: crdt::LwwMap::new(),
 			local_aliases: crdt::LwwMap::new(),
-			website_config: crdt::Lww::new(None),
+			website_config: crdt::Lww::new(None.into()),
-			cors_config: crdt::Lww::new(None),
+			cors_config: crdt::Lww::new(None.into()),
-			lifecycle_config: crdt::Lww::new(None),
+			lifecycle_config: crdt::Lww::new(None.into()),
 			quotas: crdt::Lww::new(BucketQuotas::default()),
 		}
 	}

src/model/helper/bucket.rs

@@ -52,7 +52,7 @@ impl<'a> BucketHelper<'a> {
 					.0
 					.bucket_alias_table
 					.get_local(&EmptyKey, bucket_name)?
-					.and_then(|x| *x.state.get());
+					.and_then(|x| x.state.get().into_inner());
 				match alias {
 					Some(id) => id,
 					None => return Ok(None),
@@ -91,15 +91,18 @@ impl<'a> BucketHelper<'a> {
 			.as_option()
 			.ok_or_message("Key should not be deleted at this point")?;
 
-		let bucket_opt =
+		let bucket_opt = if let Some(bucket_id) = api_key_params
-			if let Some(Some(bucket_id)) = api_key_params.local_aliases.get(bucket_name) {
+			.local_aliases
-				self.0
+			.get(bucket_name)
-					.bucket_table
+			.and_then(|x| x.inner())
-					.get_local(&EmptyKey, bucket_id)?
+		{
-					.filter(|x| !x.state.is_deleted())
+			self.0
-			} else {
+				.bucket_table
-				self.resolve_global_bucket_fast(bucket_name)?
+				.get_local(&EmptyKey, bucket_id)?
-			};
+				.filter(|x| !x.state.is_deleted())
+		} else {
+			self.resolve_global_bucket_fast(bucket_name)?
+		};
 		bucket_opt.ok_or_else(|| Error::NoSuchBucket(bucket_name.to_string()))
 	}
 
@@ -125,7 +128,7 @@ impl<'a> BucketHelper<'a> {
 					.bucket_alias_table
 					.get(&EmptyKey, bucket_name)
 					.await?
-					.and_then(|x| *x.state.get());
+					.and_then(|x| x.state.get().into_inner());
 				match alias {
 					Some(id) => id,
 					None => return Ok(None),
@@ -163,8 +166,7 @@ impl<'a> BucketHelper<'a> {
 			.ok_or_else(|| GarageError::Message(format!("access key {} has been deleted", key_id)))?
 			.local_aliases
 			.get(bucket_name)
-			.copied()
+			.and_then(|x| x.inner().copied());
-			.flatten();
 
 		if let Some(bucket_id) = local_alias {
 			Ok(self

src/model/helper/locked.rs

@@ -74,8 +74,8 @@ impl<'a> LockedHelper<'a> {
 		let alias = self.0.bucket_alias_table.get(&EmptyKey, alias_name).await?;
 
 		if let Some(existing_alias) = alias.as_ref() {
-			if let Some(p_bucket) = existing_alias.state.get() {
+			if let Some(p_bucket) = existing_alias.state.get().into_inner() {
-				if *p_bucket != bucket_id {
+				if p_bucket != bucket_id {
 					return Err(Error::BadRequest(format!(
 						"Alias {} already exists and points to different bucket: {:?}",
 						alias_name, p_bucket
@@ -98,7 +98,7 @@ impl<'a> LockedHelper<'a> {
 		let alias = match alias {
 			None => BucketAlias::new(alias_name.clone(), alias_ts, Some(bucket_id)),
 			Some(mut a) => {
-				a.state = Lww::raw(alias_ts, Some(bucket_id));
+				a.state = Lww::raw(alias_ts, Some(bucket_id).into());
 				a
 			}
 		};
@@ -128,7 +128,13 @@ impl<'a> LockedHelper<'a> {
 			.bucket_alias_table
 			.get(&EmptyKey, alias_name)
 			.await?
-			.filter(|a| a.state.get().map(|x| x == bucket_id).unwrap_or(false))
+			.filter(|a| {
+				a.state
+					.get()
+					.into_inner()
+					.map(|x| x == bucket_id)
+					.unwrap_or(false)
+			})
 			.ok_or_message(format!(
 				"Internal error: alias not found or does not point to bucket {:?}",
 				bucket_id
@@ -157,7 +163,7 @@ impl<'a> LockedHelper<'a> {
 		// ---- timestamp-ensured causality barrier ----
 		// writes are now done and all writes use timestamp alias_ts
 
-		alias.state = Lww::raw(alias_ts, None);
+		alias.state = Lww::raw(alias_ts, None.into());
 		self.0.bucket_alias_table.insert(&alias).await?;
 
 		bucket_state.aliases = LwwMap::raw_item(alias_name.clone(), alias_ts, false);
@@ -199,8 +205,8 @@ impl<'a> LockedHelper<'a> {
 		// ---- timestamp-ensured causality barrier ----
 		// writes are now done and all writes use timestamp alias_ts
 
-		if alias.state.get() == &Some(bucket_id) {
+		if alias.state.get().inner() == Some(&bucket_id) {
-			alias.state = Lww::raw(alias_ts, None);
+			alias.state = Lww::raw(alias_ts, None.into());
 			self.0.bucket_alias_table.insert(&alias).await?;
 		}
 
@@ -237,7 +243,11 @@ impl<'a> LockedHelper<'a> {
 
 		let key_param = key.state.as_option_mut().unwrap();
 
-		if let Some(Some(existing_alias)) = key_param.local_aliases.get(alias_name) {
+		if let Some(Some(existing_alias)) = key_param
+			.local_aliases
+			.get(alias_name)
+			.map(CancelingOption::inner)
+		{
 			if *existing_alias != bucket_id {
 				return Err(Error::BadRequest(format!("Alias {} already exists in namespace of key {} and points to different bucket: {:?}", alias_name,  key.key_id, existing_alias)));
 			}
@@ -261,7 +271,8 @@ impl<'a> LockedHelper<'a> {
 		// ---- timestamp-ensured causality barrier ----
 		// writes are now done and all writes use timestamp alias_ts
 
-		key_param.local_aliases = LwwMap::raw_item(alias_name.clone(), alias_ts, Some(bucket_id));
+		key_param.local_aliases =
+			LwwMap::raw_item(alias_name.clone(), alias_ts, Some(bucket_id).into());
 		self.0.key_table.insert(&key).await?;
 
 		bucket_p.local_aliases = LwwMap::raw_item(bucket_p_local_alias_key, alias_ts, true);
@@ -288,7 +299,12 @@ impl<'a> LockedHelper<'a> {
 		let key_p = key.state.as_option().unwrap();
 		let bucket_p = bucket.state.as_option_mut().unwrap();
 
-		if key_p.local_aliases.get(alias_name).cloned().flatten() != Some(bucket_id) {
+		if key_p
+			.local_aliases
+			.get(alias_name)
+			.and_then(CancelingOption::inner)
+			!= Some(&bucket_id)
+		{
 			return Err(GarageError::Message(format!(
 				"Bucket {:?} does not have alias {} in namespace of key {}",
 				bucket_id, alias_name, key_id
@@ -325,7 +341,7 @@ impl<'a> LockedHelper<'a> {
 		// writes are now done and all writes use timestamp alias_ts
 
 		key.state.as_option_mut().unwrap().local_aliases =
-			LwwMap::raw_item(alias_name.clone(), alias_ts, None);
+			LwwMap::raw_item(alias_name.clone(), alias_ts, None.into());
 		self.0.key_table.insert(&key).await?;
 
 		bucket_p.local_aliases = LwwMap::raw_item(bucket_p_local_alias_key, alias_ts, false);
@@ -367,7 +383,7 @@ impl<'a> LockedHelper<'a> {
 		// writes are now done and all writes use timestamp alias_ts
 
 		if let Some(kp) = key.state.as_option_mut() {
-			kp.local_aliases = LwwMap::raw_item(alias_name.clone(), alias_ts, None);
+			kp.local_aliases = LwwMap::raw_item(alias_name.clone(), alias_ts, None.into());
 			self.0.key_table.insert(&key).await?;
 		}
 
@@ -444,8 +460,8 @@ impl<'a> LockedHelper<'a> {
 
 		// 1. Delete local aliases
 		for (alias, _, to) in state.local_aliases.items().iter() {
-			if let Some(bucket_id) = to {
+			if let Some(bucket_id) = to.into_inner() {
-				self.purge_local_bucket_alias(*bucket_id, &key.key_id, alias)
+				self.purge_local_bucket_alias(bucket_id, &key.key_id, alias)
 					.await?;
 			}
 		}
@@ -501,7 +517,7 @@ impl<'a> LockedHelper<'a> {
 						.data
 						.decode_entry(&(item?.1))
 						.map_err(db::TxError::Abort)?;
-					if let Some(id) = alias.state.get() {
+					if let Some(id) = alias.state.get().inner() {
 						if all_buckets.contains(id) {
 							// keep aliases
 							global_aliases.insert(alias.name().to_string(), *id);
@@ -512,7 +528,7 @@ impl<'a> LockedHelper<'a> {
 								alias.name(),
 								id
 							);
-							alias.state.update(None);
+							alias.state.update(None.into());
 							delete_global.push(alias);
 						}
 					}
@@ -544,7 +560,7 @@ impl<'a> LockedHelper<'a> {
 					};
 					let mut has_changes = false;
 					for (name, _, to) in p.local_aliases.items().to_vec() {
-						if let Some(id) = to {
+						if let Some(id) = to.into_inner() {
 							if all_buckets.contains(&id) {
 								local_aliases.insert((key.key_id.clone(), name), id);
 							} else {
@@ -552,7 +568,7 @@ impl<'a> LockedHelper<'a> {
 									"local alias: remove ({}, {}) -> {:?} (bucket is deleted)",
 									key.key_id, name, id
 								);
-								p.local_aliases.update_in_place(name, None);
+								p.local_aliases.update_in_place(name, None.into());
 								has_changes = true;
 							}
 						}

src/model/k2v/item_table.rs

@@ -45,6 +45,7 @@ mod v08 {
 	}
 
 	#[derive(PartialEq, Eq, Clone, Debug, Serialize, Deserialize)]
+	#[cfg_attr(feature = "arbitrary", derive(arbitrary::Arbitrary))]
 	pub enum DvvsValue {
 		Value(#[serde(with = "serde_bytes")] Vec<u8>),
 		Deleted,
@@ -131,9 +132,26 @@ impl K2VItem {
 			ent.discard();
 		}
 	}
+
+	pub fn with_raw_items(items: BTreeMap<K2VNodeId, DvvsEntry>) -> Self {
+		let mut item = K2VItem {
+			partition: K2VItemPartition {
+				bucket_id: [0u8; 32].into(),
+				partition_key: String::new(),
+			},
+			sort_key: String::new(),
+			items,
+		};
+		item.discard();
+		item
+	}
 }
 
 impl DvvsEntry {
+	pub fn from_raw(t_discard: u64, values: Vec<(u64, DvvsValue)>) -> Self {
+		DvvsEntry { t_discard, values }
+	}
+
 	fn max_time(&self) -> u64 {
 		self.values
 			.iter()
@@ -162,15 +180,26 @@ impl Crdt for K2VItem {
 
 impl Crdt for DvvsEntry {
 	fn merge(&mut self, other: &Self) {
-		self.t_discard = std::cmp::max(self.t_discard, other.t_discard);
+		let mut slf = std::mem::take(&mut self.values).into_iter().peekable();
-		self.discard();
+		let mut otr = other.values.iter().peekable();
-
+		while let (Some((slf_t, _)), Some((otr_t, _))) = (slf.peek(), otr.peek()) {
-		let t_max = self.max_time();
+			match slf_t.cmp(otr_t) {
-		for (vt, vv) in other.values.iter() {
+				std::cmp::Ordering::Less => {
-			if *vt > t_max {
+					self.values.push(slf.next().unwrap());
-				self.values.push((*vt, vv.clone()));
+				}
+				std::cmp::Ordering::Equal => {
+					self.values.push(slf.next().unwrap());
+					otr.next();
+				}
+				std::cmp::Ordering::Greater => {
+					self.values.push(otr.next().unwrap().clone());
+				}
 			}
 		}
+		self.values.extend(slf);
+		self.values.extend(otr.cloned());
+		self.t_discard = std::cmp::max(self.t_discard, other.t_discard);
+		self.discard();
 	}
 }
 

src/model/key_table.rs

@@ -43,7 +43,7 @@ mod v08 {
 
 		/// A key can have a local view of buckets names it is
 		/// the only one to see, this is the namespace for these aliases
-		pub local_aliases: crdt::LwwMap<String, Option<Uuid>>,
+		pub local_aliases: crdt::LwwMap<String, crdt::CancelingOption<Uuid>>,
 	}
 
 	impl garage_util::migrate::InitialFormat for Key {}
@@ -51,6 +51,7 @@ mod v08 {
 
 mod v2 {
 	use crate::permission::BucketKeyPerm;
+	use crate::permission::ExpirationTime;
 	use garage_util::crdt;
 	use garage_util::data::Uuid;
 	use serde::{Deserialize, Serialize};
@@ -79,7 +80,7 @@ mod v2 {
 		/// Name for the key
 		pub name: crdt::Lww<String>,
 		/// The optional time of expiration of the key
-		pub expiration: crdt::Lww<Option<u64>>,
+		pub expiration: crdt::Lww<crdt::MergingOption<ExpirationTime>>,
 
 		/// Flag to allow users having this key to create buckets
 		pub allow_create_bucket: crdt::Lww<bool>,
@@ -91,7 +92,7 @@ mod v2 {
 
 		/// A key can have a local view of buckets names it is
 		/// the only one to see, this is the namespace for these aliases
-		pub local_aliases: crdt::LwwMap<String, Option<Uuid>>,
+		pub local_aliases: crdt::LwwMap<String, crdt::CancelingOption<Uuid>>,
 	}
 
 	impl garage_util::migrate::Migrate for Key {
@@ -106,7 +107,7 @@ mod v2 {
 					created: None,
 					secret_key: x.secret_key,
 					name: x.name,
-					expiration: crdt::Lww::raw(0, None),
+					expiration: crdt::Lww::raw(0, None.into()),
 					allow_create_bucket: x.allow_create_bucket,
 					authorized_buckets: x.authorized_buckets,
 					local_aliases: x.local_aliases,
@@ -124,7 +125,7 @@ impl KeyParams {
 			created: Some(now_msec()),
 			secret_key: secret_key.to_string(),
 			name: crdt::Lww::new(name.to_string()),
-			expiration: crdt::Lww::new(None),
+			expiration: crdt::Lww::new(None.into()),
 			allow_create_bucket: crdt::Lww::new(false),
 			authorized_buckets: crdt::Map::new(),
 			local_aliases: crdt::LwwMap::new(),
@@ -229,9 +230,9 @@ impl Key {
 
 impl KeyParams {
 	pub fn is_expired(&self, ts_now: u64) -> bool {
-		match *self.expiration.get() {
+		match self.expiration.get().inner() {
 			None => false,
-			Some(exp) => ts_now >= exp,
+			Some(exp) => ts_now >= exp.0,
 		}
 	}
 }

src/model/permission.rs

@@ -6,6 +6,7 @@ use garage_util::crdt::*;
 
 /// Permission given to a key in a bucket
 #[derive(PartialOrd, Ord, PartialEq, Eq, Clone, Copy, Debug, Serialize, Deserialize)]
+#[cfg_attr(feature = "arbitrary", derive(arbitrary::Arbitrary))]
 pub struct BucketKeyPerm {
 	/// Timestamp at which the permission was given
 	pub timestamp: u64,
@@ -62,3 +63,15 @@ impl Crdt for BucketKeyPerm {
 		}
 	}
 }
+
+/// Expiration date for a key or token
+#[derive(PartialOrd, Ord, PartialEq, Eq, Clone, Copy, Debug, Serialize, Deserialize)]
+#[cfg_attr(feature = "arbitrary", derive(arbitrary::Arbitrary))]
+#[serde(transparent)]
+pub struct ExpirationTime(pub u64);
+
+impl Crdt for ExpirationTime {
+	fn merge(&mut self, other: &Self) {
+		self.0 = std::cmp::min(self.0, other.0);
+	}
+}

src/model/s3/lifecycle_worker.rs

@@ -271,7 +271,7 @@ async fn process_object(
 	let lifecycle_policy: &[LifecycleRule] = bucket
 		.state
 		.as_option()
-		.and_then(|s| s.lifecycle_config.get().as_deref())
+		.and_then(|s| s.lifecycle_config.get().inner().map(|x| &x[..]))
 		.unwrap_or_default();
 
 	if lifecycle_policy.iter().all(|x| !x.enabled) {

src/model/s3/mpu_table.rs

@@ -48,6 +48,7 @@ mod v09 {
 	}
 
 	#[derive(PartialEq, Eq, Clone, Copy, Debug, Serialize, Deserialize)]
+	#[cfg_attr(feature = "arbitrary", derive(arbitrary::Arbitrary))]
 	pub struct MpuPartKey {
 		/// Number of the part
 		pub part_number: u64,
@@ -57,6 +58,7 @@ mod v09 {
 
 	/// The version of an uploaded part
 	#[derive(PartialEq, Eq, Clone, Debug, Serialize, Deserialize)]
+	#[cfg_attr(feature = "arbitrary", derive(arbitrary::Arbitrary))]
 	pub struct MpuPart {
 		/// Links to a Version in `VersionTable`
 		pub version: Uuid,

src/model/s3/object_table.rs

@@ -291,6 +291,7 @@ mod v010 {
 
 	/// Checksum value for x-amz-checksum-algorithm
 	#[derive(PartialEq, Eq, PartialOrd, Ord, Clone, Copy, Debug, Serialize, Deserialize)]
+	#[cfg_attr(feature = "arbitrary", derive(arbitrary::Arbitrary))]
 	pub enum ChecksumValue {
 		Crc32(#[serde(with = "serde_bytes")] [u8; 4]),
 		Crc32c(#[serde(with = "serde_bytes")] [u8; 4]),

src/model/s3/version_table.rs

@@ -41,6 +41,7 @@ mod v08 {
 	}
 
 	#[derive(PartialEq, Eq, Clone, Copy, Debug, Serialize, Deserialize)]
+	#[cfg_attr(feature = "arbitrary", derive(arbitrary::Arbitrary))]
 	pub struct VersionBlockKey {
 		/// Number of the part
 		pub part_number: u64,
@@ -51,6 +52,7 @@ mod v08 {
 
 	/// Information about a single block
 	#[derive(PartialEq, Eq, Ord, PartialOrd, Clone, Copy, Debug, Serialize, Deserialize)]
+	#[cfg_attr(feature = "arbitrary", derive(arbitrary::Arbitrary))]
 	pub struct VersionBlock {
 		/// Blake2 sum of the block
 		pub hash: Hash,

src/net/netapp.rs

@@ -43,6 +43,7 @@ pub(crate) const NETAPP_VERSION_TAG: u64 = 0x6772676e65740010; // grgnet 0x0010
 /// Time a connection must be idle before the first keepalive probe is sent.
 const TCP_KEEPALIVE_TIME: Duration = Duration::from_secs(30);
 /// Interval between keepalive probes after the first.
+#[cfg(not(target_os = "openbsd"))]
 const TCP_KEEPALIVE_INTERVAL: Duration = Duration::from_secs(10);
 
 /// Timeout for outgoing TCP connection attempts.
@@ -52,9 +53,15 @@ const CONNECT_TIMEOUT: Duration = Duration::from_secs(10);
 
 fn set_keepalive(stream: &TcpStream) -> Result<(), std::io::Error> {
 	let sock_ref = socket2::SockRef::from(stream);
+	// OpenBSD does not support with_interval method
+	#[cfg(not(target_os = "openbsd"))]
 	let keepalive = socket2::TcpKeepalive::new()
 		.with_time(TCP_KEEPALIVE_TIME)
 		.with_interval(TCP_KEEPALIVE_INTERVAL);
+
+	#[cfg(target_os = "openbsd")]
+	let keepalive = socket2::TcpKeepalive::new().with_time(TCP_KEEPALIVE_TIME);
+
 	sock_ref.set_tcp_keepalive(&keepalive)
 }
 

src/net/peering.rs

@@ -313,10 +313,8 @@ impl PeeringManager {
 								to_ping.push(*id);
 							}
 						}
-						PeerConnState::Waiting(_, t) => {
+						PeerConnState::Waiting(_, t) if Instant::now() >= t => {
-							if Instant::now() >= t {
+							to_retry.push(*id);
-								to_retry.push(*id);
-							}
 						}
 						_ => (),
 					}

src/net/test.rs

@@ -13,12 +13,14 @@ use crate::peering::*;
 use crate::NodeID;
 
 #[tokio::test(flavor = "current_thread")]
+#[ignore = "flaky"]
 async fn test_with_basic_scheduler() {
 	pretty_env_logger::init();
 	run_test(19980).await;
 }
 
 #[tokio::test(flavor = "multi_thread", worker_threads = 4)]
+#[ignore = "flaky"]
 async fn test_with_threaded_scheduler() {
 	run_test(19990).await;
 }

src/rpc/Cargo.toml

@@ -33,6 +33,7 @@ async-trait.workspace = true
 serde.workspace = true
 serde_bytes.workspace = true
 serde_json.workspace = true
+utoipa.workspace = true
 thiserror = { workspace = true, optional = true }
 
 # newer version requires rust edition 2021

src/rpc/layout/history.rs

@@ -6,6 +6,7 @@ use garage_util::encode::nonversioned_encode;
 use garage_util::error::*;
 
 use super::*;
+use crate::layout::ComputationStat;
 use crate::replication_mode::*;
 
 impl LayoutHistory {
@@ -269,13 +270,13 @@ impl LayoutHistory {
 		changed
 	}
 
-	pub fn apply_staged_changes(mut self, version: u64) -> Result<(Self, Message), Error> {
+	pub fn apply_staged_changes(mut self, version: u64) -> Result<(Self, ComputationStat), Error> {
 		if version != self.current().version + 1 {
 			return Err(Error::Message("Invalid new layout version".into()));
 		}
 
 		// Compute new version and add it to history
-		let (new_version, msg) = self
+		let (new_version, stat) = self
 			.current()
 			.clone()
 			.calculate_next_version(self.staging.get())?;
@@ -289,7 +290,7 @@ impl LayoutHistory {
 			roles: LwwMap::new(),
 		});
 
-		Ok((self, msg))
+		Ok((self, stat))
 	}
 
 	pub fn revert_staged_changes(mut self) -> Result<Self, Error> {

src/rpc/layout/mod.rs

@@ -389,10 +389,6 @@ impl NodeRole {
 			None => "gateway".to_string(),
 		}
 	}
-
-	pub fn tags_string(&self) -> String {
-		self.tags.join(",")
-	}
 }
 
 impl UpdateTracker {

src/rpc/layout/test.rs

@@ -79,8 +79,8 @@ fn check_against_naive(cl: &LayoutVersion) -> bool {
 	false
 }
 
-fn show_msg(msg: &Message) {
+fn show_stat(stat: &ComputationStat) {
-	for s in msg.iter() {
+	for s in stat.to_message().iter() {
 		println!("{}", s);
 	}
 }
@@ -123,8 +123,8 @@ fn test_assignment() {
 	let mut cl = LayoutHistory::new(ReplicationFactor::new(3).unwrap());
 	update_layout(&mut cl, &node_capacity_vec, &node_zone_vec, 3);
 	let v = cl.current().version;
-	let (mut cl, msg) = cl.apply_staged_changes(v + 1).unwrap();
+	let (mut cl, stat) = cl.apply_staged_changes(v + 1).unwrap();
-	show_msg(&msg);
+	show_stat(&stat);
 	assert_eq!(cl.check(), Ok(()));
 	assert!(check_against_naive(cl.current()));
 
@@ -132,16 +132,16 @@ fn test_assignment() {
 	node_zone_vec = vec!["A", "B", "C", "C", "C", "B", "G", "H", "I"];
 	update_layout(&mut cl, &node_capacity_vec, &node_zone_vec, 2);
 	let v = cl.current().version;
-	let (mut cl, msg) = cl.apply_staged_changes(v + 1).unwrap();
+	let (mut cl, stat) = cl.apply_staged_changes(v + 1).unwrap();
-	show_msg(&msg);
+	show_stat(&stat);
 	assert_eq!(cl.check(), Ok(()));
 	assert!(check_against_naive(cl.current()));
 
 	node_capacity_vec = vec![4000, 1000, 2000, 7000, 1000, 1000, 2000, 10000, 2000];
 	update_layout(&mut cl, &node_capacity_vec, &node_zone_vec, 3);
 	let v = cl.current().version;
-	let (mut cl, msg) = cl.apply_staged_changes(v + 1).unwrap();
+	let (mut cl, stat) = cl.apply_staged_changes(v + 1).unwrap();
-	show_msg(&msg);
+	show_stat(&stat);
 	assert_eq!(cl.check(), Ok(()));
 	assert!(check_against_naive(cl.current()));
 
@@ -150,8 +150,8 @@ fn test_assignment() {
 	];
 	update_layout(&mut cl, &node_capacity_vec, &node_zone_vec, 1);
 	let v = cl.current().version;
-	let (cl, msg) = cl.apply_staged_changes(v + 1).unwrap();
+	let (cl, stat) = cl.apply_staged_changes(v + 1).unwrap();
-	show_msg(&msg);
+	show_stat(&stat);
 	assert_eq!(cl.check(), Ok(()));
 	assert!(check_against_naive(cl.current()));
 }

src/rpc/layout/version.rs

@@ -4,6 +4,8 @@ use std::convert::TryInto;
 
 use bytesize::ByteSize;
 use itertools::Itertools;
+use serde::{Deserialize, Serialize};
+use utoipa::ToSchema;
 
 use garage_util::crdt::{Crdt, LwwMap};
 use garage_util::data::*;
@@ -13,9 +15,6 @@ use super::graph_algo::*;
 use super::*;
 use crate::replication_mode::*;
 
-// The Message type will be used to collect information on the algorithm.
-pub type Message = Vec<String>;
-
 impl LayoutVersion {
 	pub fn new(replication_factor: ReplicationFactor) -> Self {
 		// We set the default zone redundancy to be Maximum, meaning that the maximum
@@ -291,16 +290,16 @@ impl LayoutVersion {
 	pub(crate) fn calculate_next_version(
 		mut self,
 		staging: &LayoutStaging,
-	) -> Result<(Self, Message), Error> {
+	) -> Result<(Self, ComputationStat), Error> {
 		self.version += 1;
 
 		self.roles.merge(&staging.roles);
 		self.roles.retain(|(_, _, v)| v.0.is_some());
 		self.parameters = *staging.parameters.get();
 
-		let msg = self.calculate_partition_assignment()?;
+		let stat = self.calculate_partition_assignment()?;
 
-		Ok((self, msg))
+		Ok((self, stat))
 	}
 
 	/// This function calculates a new partition-to-node assignment.
@@ -312,21 +311,15 @@ impl LayoutVersion {
 	/// data to be moved.
 	/// Staged role changes must be merged with nodes roles before calling this function,
 	/// hence it must only be called from `apply_staged_changes()` and hence is not public.
-	fn calculate_partition_assignment(&mut self) -> Result<Message, Error> {
+	fn calculate_partition_assignment(&mut self) -> Result<ComputationStat, Error> {
 		// We update the node ids, since the node role list might have changed with the
 		// changes in the layout. We retrieve the old_assignment reframed with new ids
 		let old_assignment_opt = self.update_node_id_vec()?;
 
 		let zone_redundancy = self.effective_zone_redundancy();
 
-		let mut msg = Message::new();
+		let stat_replication_factor = self.replication_factor;
-		msg.push("==== COMPUTATION OF A NEW PARTITION ASSIGNATION ====".into());
+		let stat_effective_zone_redundancy = zone_redundancy;
-		msg.push("".into());
-		msg.push(format!(
-			"Partitions are \
-        replicated {} times on at least {} distinct zones.",
-			self.replication_factor, zone_redundancy
-		));
 
 		// We generate for once numerical ids for the zones of non gateway nodes,
 		// to use them as indices in the flow graphs.
@@ -355,28 +348,17 @@ impl LayoutVersion {
 		// optimality.
 		let partition_size = self.compute_optimal_partition_size(&zone_to_id, zone_redundancy)?;
 
-		msg.push("".into());
+		let stat_partition_size = partition_size;
-		if old_assignment_opt.is_some() {
+		let stat_previous_partition_size = if old_assignment_opt.is_some() {
-			msg.push(format!(
+			Some(self.partition_size)
-				"Optimal partition size:                     {} ({} in previous layout)",
-				ByteSize::b(partition_size).display().iec(),
-				ByteSize::b(self.partition_size).display().iec()
-			));
 		} else {
-			msg.push(format!(
+			None
-				"Optimal partition size:                     {}",
+		};
-				ByteSize::b(partition_size).display().iec()
+
-			));
-		}
 		// We write the partition size.
 		self.partition_size = partition_size;
 
-		if partition_size < 100 {
+		let stat_low_partition_size = partition_size < 100;
-			msg.push(
-				"WARNING: The partition size is low (< 100), make sure the capacities of your nodes are correct and are of at least a few MB"
-					.into(),
-			);
-		}
 
 		// We compute a first flow/assignment that is heuristically close to the previous
 		// assignment
@@ -388,18 +370,28 @@ impl LayoutVersion {
 		}
 
 		// We display statistics of the computation
-		msg.extend(self.output_stat(&gflow, &old_assignment_opt, &zone_to_id, &id_to_zone)?);
+		let stat = self.output_stat(
+			&gflow,
+			&old_assignment_opt,
+			&zone_to_id,
+			&id_to_zone,
+			stat_replication_factor,
+			stat_effective_zone_redundancy,
+			stat_partition_size,
+			stat_previous_partition_size,
+			stat_low_partition_size,
+		)?;
 
 		// We update the layout structure
 		self.update_ring_from_flow(id_to_zone.len(), &gflow)?;
 
 		if let Err(e) = self.check() {
 			return Err(Error::Message(
-				format!("Layout check returned an error: {}\nOriginal result of computation: <<<<\n{}\n>>>>", e, msg.join("\n"))
+				format!("Layout check returned an error: {}\nOriginal result of computation: <<<<\n{}\n>>>>", e, stat.to_message().join("\n"))
 			));
 		}
 
-		Ok(msg)
+		Ok(stat)
 	}
 
 	/// The `LwwMap` of node roles might have changed. This function updates the `node_id_vec`
@@ -706,47 +698,26 @@ impl LayoutVersion {
 
 	/// This function returns a message summing up the partition repartition of the new
 	/// layout, and other statistics of the partition assignment computation.
+	#[allow(clippy::too_many_arguments)]
 	fn output_stat(
 		&self,
 		gflow: &Graph<FlowEdge>,
 		prev_assign_opt: &Option<Vec<Vec<usize>>>,
 		zone_to_id: &HashMap<String, usize>,
 		id_to_zone: &[String],
-	) -> Result<Message, Error> {
+		replication_factor: usize,
-		let mut msg = Message::new();
+		effective_zone_redundancy: usize,
+		partition_size: u64,
+		previous_partition_size: Option<u64>,
+		low_partition_size: bool,
+	) -> Result<ComputationStat, Error> {
+		let usable_capacity =
+			self.partition_size * NB_PARTITIONS as u64 * self.replication_factor as u64;
+		let total_capacity = self.get_total_capacity();
+		let effective_capacity = usable_capacity / replication_factor as u64;
 
-		let used_cap = self.partition_size * NB_PARTITIONS as u64 * self.replication_factor as u64;
+		let percent_cap = 100.0 * (usable_capacity as f32) / (total_capacity as f32);
-		let total_cap = self.get_total_capacity();
+		let low_usable_capacity = percent_cap < 80.;
-		let percent_cap = 100.0 * (used_cap as f32) / (total_cap as f32);
-		msg.push(format!(
-			"Usable capacity / total cluster capacity:   {} / {} ({:.1} %)",
-			ByteSize::b(used_cap).display().iec(),
-			ByteSize::b(total_cap).display().iec(),
-			percent_cap
-		));
-		msg.push(format!(
-			"Effective capacity (replication factor {}):  {}",
-			self.replication_factor,
-			ByteSize::b(used_cap / self.replication_factor as u64)
-				.display()
-				.iec()
-		));
-		if percent_cap < 80. {
-			msg.push("".into());
-			msg.push(
-				"If the percentage is too low, it might be that the \
-            cluster topology and redundancy constraints are forcing the use of nodes/zones with small \
-            storage capacities."
-					.into(),
-			);
-			msg.push(
-				"You might want to move storage capacity between zones or relax the redundancy constraint."
-					.into(),
-			);
-			msg.push(
-				"See the detailed statistics below and look for saturated nodes/zones.".into(),
-			);
-		}
 
 		// We define and fill in the following tables
 		let storing_nodes = self.nongateway_nodes();
@@ -795,18 +766,13 @@ impl LayoutVersion {
 
 		// We display the statistics
 
-		msg.push("".into());
+		let total_moved_partitions = if prev_assign_opt.is_some() {
-		if prev_assign_opt.is_some() {
+			Some(new_partitions.iter().sum())
-			let total_new_partitions: usize = new_partitions.iter().sum();
+		} else {
-			msg.push(format!(
+			None
-				"A total of {} new copies of partitions need to be \
+		};
-                             transferred.",
-				total_new_partitions
-			));
-			msg.push("".into());
-		}
 
-		let mut table = vec![];
+		let mut zones = vec![];
 		for z in 0..id_to_zone.len() {
 			let mut nodes_of_z = Vec::<usize>::new();
 			for n in 0..storing_nodes.len() {
@@ -814,49 +780,224 @@ impl LayoutVersion {
 					nodes_of_z.push(n);
 				}
 			}
-			let replicated_partitions: usize =
+			let replicated_partitions_z: usize =
 				nodes_of_z.iter().map(|n| stored_partitions[*n]).sum();
-			table.push(format!(
-				"{}\tTags\tPartitions\tCapacity\tUsable capacity",
-				id_to_zone[z]
-			));
 
-			let available_cap_z: u64 = self.partition_size * replicated_partitions as u64;
+			let available_cap_z: u64 = self.partition_size * replicated_partitions_z as u64;
 			let mut total_cap_z = 0;
 			for n in nodes_of_z.iter() {
 				total_cap_z += self.expect_get_node_capacity(&self.node_id_vec[*n]);
 			}
-			let percent_cap_z = 100.0 * (available_cap_z as f32) / (total_cap_z as f32);
 
+			let mut nodes = vec![];
 			for n in nodes_of_z.iter() {
 				let available_cap_n = stored_partitions[*n] as u64 * self.partition_size;
 				let total_cap_n = self.expect_get_node_capacity(&self.node_id_vec[*n]);
-				let tags_n = (self.node_role(&self.node_id_vec[*n]).ok_or("<??>"))?.tags_string();
+				let tags_n = (self.node_role(&self.node_id_vec[*n]).ok_or("<??>"))?
+					.tags
+					.clone();
+				nodes.push(ComputationStatNode {
+					id: hex::encode(self.node_id_vec[*n]),
+					tags: tags_n,
+					stored_partitions: stored_partitions[*n],
+					new_partitions: new_partitions[*n],
+					total_capacity: total_cap_n,
+					usable_capacity: available_cap_n,
+				});
+			}
+
+			zones.push(ComputationStatZone {
+				name: id_to_zone[z].to_string(),
+				nodes,
+				total_replicated_partitions: replicated_partitions_z,
+				unique_partitions: stored_partitions_zone[z],
+				total_capacity: total_cap_z,
+				usable_capacity: available_cap_z,
+			});
+		}
+
+		Ok(ComputationStat {
+			replication_factor,
+			effective_zone_redundancy,
+			partition_size,
+			previous_partition_size,
+			low_partition_size,
+			usable_capacity,
+			total_capacity,
+			effective_capacity,
+			low_usable_capacity,
+			total_moved_partitions,
+			zones,
+		})
+	}
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize, ToSchema)]
+#[serde(rename_all = "camelCase")]
+pub struct ComputationStat {
+	/// The cluster's replication factor
+	pub replication_factor: usize,
+	/// The zone redundancy factor achieved by this layout
+	pub effective_zone_redundancy: usize,
+	/// The size of a partition, in bytes
+	pub partition_size: u64,
+	/// The size of a partition, in bytes, in the previous layout
+	pub previous_partition_size: Option<u64>,
+	/// Warning flag indicating when partitions are very small
+	pub low_partition_size: bool,
+	/// The portion of total raw node capacity that is used by partitions
+	pub usable_capacity: u64,
+	/// The total raw capacity of nodes
+	pub total_capacity: u64,
+	/// The final effective capacity of the cluster, accounting for replication
+	pub effective_capacity: u64,
+	/// Warning flag indicating that the raw node capacity could not be used
+	/// effectively
+	pub low_usable_capacity: bool,
+	/// The total number of partitions that will be moved to a new storage node
+	pub total_moved_partitions: Option<usize>,
+	/// Per-zone storage statistics
+	pub zones: Vec<ComputationStatZone>,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize, ToSchema)]
+#[serde(rename_all = "camelCase")]
+pub struct ComputationStatZone {
+	/// The name of the zone
+	pub name: String,
+	/// Per-node storage statistics for nodes in this zone
+	pub nodes: Vec<ComputationStatNode>,
+	/// The total number of partition replicas in this zone
+	pub total_replicated_partitions: usize,
+	/// The number of unique partitions that have at least one replica in this zone
+	pub unique_partitions: usize,
+	/// The total raw capacity of nodes in this zone
+	pub total_capacity: u64,
+	/// The used portion of the raw capacity of nodes in this zones
+	pub usable_capacity: u64,
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize, ToSchema)]
+#[serde(rename_all = "camelCase")]
+pub struct ComputationStatNode {
+	/// The node's ID
+	pub id: String,
+	/// The node's tags as defined in the layout
+	pub tags: Vec<String>,
+	/// The number of partitions that are replicated on this node
+	pub stored_partitions: usize,
+	/// The number of partitions that are newly replicated on this node
+	pub new_partitions: usize,
+	/// The node's raw capacity
+	pub total_capacity: u64,
+	/// The portion of the node's raw capacity that is used by partitions it stores
+	pub usable_capacity: u64,
+}
+
+impl ComputationStat {
+	pub fn to_message(&self) -> Vec<String> {
+		let mut msg = Vec::new();
+		msg.push("==== COMPUTATION OF A NEW PARTITION ASSIGNATION ====".into());
+		msg.push("".into());
+		msg.push(format!(
+			"Partitions are \
+        replicated {} times on at least {} distinct zones.",
+			self.replication_factor, self.effective_zone_redundancy
+		));
+
+		msg.push("".into());
+		if let Some(prev) = self.previous_partition_size {
+			msg.push(format!(
+				"Optimal partition size:                     {} ({} in previous layout)",
+				ByteSize::b(self.partition_size).display().iec(),
+				ByteSize::b(prev).display().iec()
+			));
+		} else {
+			msg.push(format!(
+				"Optimal partition size:                     {}",
+				ByteSize::b(self.partition_size).display().iec()
+			));
+		}
+
+		if self.low_partition_size {
+			msg.push(
+				"WARNING: The partition size is low (< 100), make sure the capacities of your nodes are correct and are of at least a few MB"
+					.into(),
+			);
+		}
+
+		let percent_cap = 100.0 * (self.usable_capacity as f32) / (self.total_capacity as f32);
+		msg.push(format!(
+			"Usable capacity / total cluster capacity:   {} / {} ({:.1} %)",
+			ByteSize::b(self.usable_capacity).display().iec(),
+			ByteSize::b(self.total_capacity).display().iec(),
+			percent_cap
+		));
+		msg.push(format!(
+			"Effective capacity (replication factor {}):  {}",
+			self.replication_factor,
+			ByteSize::b(self.effective_capacity).display().iec()
+		));
+
+		if self.low_usable_capacity {
+			msg.push("".into());
+			msg.push(
+				"If the percentage is too low, it might be that the \
+            cluster topology and redundancy constraints are forcing the use of nodes/zones with small \
+            storage capacities."
+					.into(),
+			);
+			msg.push(
+				"You might want to move storage capacity between zones or relax the redundancy constraint."
+					.into(),
+			);
+			msg.push(
+				"See the detailed statistics below and look for saturated nodes/zones.".into(),
+			);
+		}
+
+		msg.push("".into());
+		if let Some(tmp) = self.total_moved_partitions {
+			msg.push(format!(
+				"A total of {} new copies of partitions need to be \
+                             transferred.",
+				tmp
+			));
+			msg.push("".into());
+		}
+
+		let mut table = vec![];
+		for z in self.zones.iter() {
+			table.push(format!(
+				"{}\tTags\tPartitions\tCapacity\tUsable capacity",
+				z.name
+			));
+
+			for n in z.nodes.iter() {
 				table.push(format!(
-					"  {:?}\t[{}]\t{} ({} new)\t{}\t{} ({:.1}%)",
+					"  {:.16}\t[{}]\t{} ({} new)\t{}\t{} ({:.1}%)",
-					self.node_id_vec[*n],
+					n.id,
-					tags_n,
+					n.tags.join(","),
-					stored_partitions[*n],
+					n.stored_partitions,
-					new_partitions[*n],
+					n.new_partitions,
-					ByteSize::b(total_cap_n).display().iec(),
+					ByteSize::b(n.total_capacity).display().iec(),
-					ByteSize::b(available_cap_n).display().iec(),
+					ByteSize::b(n.usable_capacity).display().iec(),
-					(available_cap_n as f32) / (total_cap_n as f32) * 100.0,
+					(n.usable_capacity as f32) / (n.total_capacity as f32) * 100.0,
 				));
 			}
 
 			table.push(format!(
 				"  TOTAL\t\t{} ({} unique)\t{}\t{} ({:.1}%)",
-				replicated_partitions,
+				z.total_replicated_partitions,
-				stored_partitions_zone[z],
+				z.unique_partitions,
-				//new_partitions_zone[z],
+				ByteSize::b(z.total_capacity).display().iec(),
-				ByteSize::b(total_cap_z).display().iec(),
+				ByteSize::b(z.usable_capacity).display().iec(),
-				ByteSize::b(available_cap_z).display().iec(),
+				(z.usable_capacity as f32) / (z.total_capacity as f32) * 100.0,
-				percent_cap_z
 			));
 			table.push("".into());
 		}
 		msg.push(format_table::format_table_to_string(table));
 
-		Ok(msg)
+		msg
 	}
 }

src/rpc/metrics.rs

@@ -14,23 +14,23 @@ impl RpcMetrics {
 		let meter = global::meter("garage_rpc");
 		RpcMetrics {
 			rpc_counter: meter
-				.u64_counter("rpc.request_counter")
+				.u64_counter("garage_rpc.request_count")
 				.with_description("Number of RPC requests emitted")
 				.init(),
 			rpc_timeout_counter: meter
-				.u64_counter("rpc.timeout_counter")
+				.u64_counter("garage_rpc.timeout_count")
 				.with_description("Number of RPC timeouts")
 				.init(),
 			rpc_netapp_error_counter: meter
-				.u64_counter("rpc.netapp_error_counter")
+				.u64_counter("garage_rpc.netapp_error_count")
 				.with_description("Number of communication errors (errors in the Netapp library)")
 				.init(),
 			rpc_garage_error_counter: meter
-				.u64_counter("rpc.garage_error_counter")
+				.u64_counter("garage_rpc.garage_error_count")
 				.with_description("Number of RPC errors (errors happening when handling the RPC)")
 				.init(),
 			rpc_duration: meter
-				.f64_value_recorder("rpc.duration")
+				.f64_value_recorder("garage_rpc.duration")
 				.with_description("Duration of RPCs")
 				.init(),
 		}

src/rpc/rpc_helper.rs

@@ -241,11 +241,7 @@ impl RpcHelper {
 		)
 		.with_context(Context::current_with_span(span))
 		.await;
-		Ok(to
+		Ok(to.iter().cloned().zip(resps).collect::<Vec<_>>())
-			.iter()
-			.cloned()
-			.zip(resps.into_iter())
-			.collect::<Vec<_>>())
 	}
 
 	pub async fn broadcast<M, N, H, S>(

src/rpc/system_metrics.rs

@@ -110,7 +110,7 @@ impl SystemMetrics {
 			_cluster_healthy: {
 				let get_health = get_health.clone();
 				meter
-					.u64_value_observer("cluster_healthy", move |observer| {
+					.u64_value_observer("garage_cluster_healthy", move |observer| {
 						let h = get_health();
 						if h.status == ClusterHealthStatus::Healthy {
 							observer.observe(1, &[]);
@@ -123,7 +123,7 @@ impl SystemMetrics {
 			},
 			_cluster_available: {
 				let get_health = get_health.clone();
-				meter.u64_value_observer("cluster_available", move |observer| {
+				meter.u64_value_observer("garage_cluster_available", move |observer| {
 					let h = get_health();
 					if h.status != ClusterHealthStatus::Unavailable {
 						observer.observe(1, &[]);
@@ -137,7 +137,7 @@ impl SystemMetrics {
 			_known_nodes: {
 				let get_health = get_health.clone();
 				meter
-					.u64_value_observer("cluster_known_nodes", move |observer| {
+					.u64_value_observer("garage_cluster_known_nodes", move |observer| {
 						let h = get_health();
 						observer.observe(h.known_nodes as u64, &[]);
 					})
@@ -147,7 +147,7 @@ impl SystemMetrics {
 			_connected_nodes: {
 				let get_health = get_health.clone();
 				meter
-					.u64_value_observer("cluster_connected_nodes", move |observer| {
+					.u64_value_observer("garage_cluster_connected_nodes", move |observer| {
 						let h = get_health();
 						observer.observe(h.connected_nodes as u64, &[]);
 					})
@@ -157,7 +157,7 @@ impl SystemMetrics {
 			_storage_nodes: {
 				let get_health = get_health.clone();
 				meter
-					.u64_value_observer("cluster_storage_nodes", move |observer| {
+					.u64_value_observer("garage_cluster_storage_nodes", move |observer| {
 						let h = get_health();
 						observer.observe(h.storage_nodes as u64, &[]);
 					})
@@ -167,7 +167,7 @@ impl SystemMetrics {
 			_storage_nodes_ok: {
 				let get_health = get_health.clone();
 				meter
-					.u64_value_observer("cluster_storage_nodes_ok", move |observer| {
+					.u64_value_observer("garage_cluster_storage_nodes_ok", move |observer| {
 						let h = get_health();
 						observer.observe(h.storage_nodes_ok as u64, &[]);
 					})
@@ -177,7 +177,7 @@ impl SystemMetrics {
 			_partitions: {
 				let get_health = get_health.clone();
 				meter
-					.u64_value_observer("cluster_partitions", move |observer| {
+					.u64_value_observer("garage_cluster_partitions", move |observer| {
 						let h = get_health();
 						observer.observe(h.partitions as u64, &[]);
 					})
@@ -187,7 +187,7 @@ impl SystemMetrics {
 			_partitions_quorum: {
 				let get_health = get_health.clone();
 				meter
-					.u64_value_observer("cluster_partitions_quorum", move |observer| {
+					.u64_value_observer("garage_cluster_partitions_quorum", move |observer| {
 						let h = get_health();
 						observer.observe(h.partitions_quorum as u64, &[]);
 					})
@@ -199,7 +199,7 @@ impl SystemMetrics {
 			_partitions_all_ok: {
 				let get_health = get_health.clone();
 				meter
-					.u64_value_observer("cluster_partitions_all_ok", move |observer| {
+					.u64_value_observer("garage_cluster_partitions_all_ok", move |observer| {
 						let h = get_health();
 						observer.observe(h.partitions_all_ok as u64, &[]);
 					})
@@ -213,7 +213,7 @@ impl SystemMetrics {
 			_layout_node_connected: {
 				let system = system.clone();
 				meter
-					.u64_value_observer("cluster_layout_node_connected", move |observer| {
+					.u64_value_observer("garage_cluster_layout_node_connected", move |observer| {
 						let layout = system.cluster_layout();
 						let nodes = system.get_known_nodes();
 						for id in layout.all_nodes().unwrap_or_default().iter() {
@@ -260,44 +260,47 @@ impl SystemMetrics {
 			_layout_node_disconnected_time: {
 				let system = system.clone();
 				meter
-					.u64_value_observer("cluster_layout_node_disconnected_time", move |observer| {
+					.u64_value_observer(
-						let layout = system.cluster_layout();
+						"garage_cluster_layout_node_disconnected_time",
-						let nodes = system.get_known_nodes();
+						move |observer| {
-						for id in layout.all_nodes().unwrap_or_default().iter() {
+							let layout = system.cluster_layout();
-							let mut kv = vec![KeyValue::new("id", format!("{:?}", id))];
+							let nodes = system.get_known_nodes();
-							if let Some(role) = layout
+							for id in layout.all_nodes().unwrap_or_default().iter() {
-								.current()
+								let mut kv = vec![KeyValue::new("id", format!("{:?}", id))];
-								.ok()
+								if let Some(role) = layout
-								.and_then(|l| l.roles.get(id))
+									.current()
-								.and_then(|r| r.0.as_ref())
+									.ok()
-							{
+									.and_then(|l| l.roles.get(id))
-								kv.push(KeyValue::new("role_zone", role.zone.clone()));
+									.and_then(|r| r.0.as_ref())
-								match role.capacity {
+								{
-									Some(cap) => {
+									kv.push(KeyValue::new("role_zone", role.zone.clone()));
-										kv.push(KeyValue::new("role_capacity", cap as i64));
+									match role.capacity {
-										kv.push(KeyValue::new("role_gateway", 0));
+										Some(cap) => {
-									}
+											kv.push(KeyValue::new("role_capacity", cap as i64));
-									None => {
+											kv.push(KeyValue::new("role_gateway", 0));
-										kv.push(KeyValue::new("role_gateway", 1));
+										}
+										None => {
+											kv.push(KeyValue::new("role_gateway", 1));
+										}
 									}
 								}
-							}
 
-							if let Some(node) = nodes.iter().find(|n| n.id == *id) {
+								if let Some(node) = nodes.iter().find(|n| n.id == *id) {
-								// TODO: see comment above
+									// TODO: see comment above
-								// kv.push(KeyValue::new("address", node.addr.to_string()));
+									// kv.push(KeyValue::new("address", node.addr.to_string()));
-								// kv.push(KeyValue::new(
+									// kv.push(KeyValue::new(
-								//	"hostname",
+									//	"hostname",
-								//	node.status.hostname.clone(),
+									//	node.status.hostname.clone(),
-								// ));
+									// ));
-								if node.is_up {
+									if node.is_up {
-									observer.observe(0, &kv);
+										observer.observe(0, &kv);
-								} else if let Some(secs) = node.last_seen_secs_ago {
+									} else if let Some(secs) = node.last_seen_secs_ago {
-									observer.observe(secs, &kv);
+										observer.observe(secs, &kv);
+									}
 								}
 							}
-						}
+						},
-					})
+					)
 					.with_description(
 						"Time (in seconds) since last connection to nodes in the cluster layout",
 					)

src/table/metrics.rs

@@ -34,7 +34,7 @@ impl TableMetrics {
 		TableMetrics {
 			_table_size: meter
 				.u64_value_observer(
-					"table.size",
+					"garage_table.size",
 					move |observer| {
 						if let Ok(value) = store.approximate_len() {
 							observer.observe(
@@ -48,7 +48,7 @@ impl TableMetrics {
 				.init(),
 			_merkle_tree_size: meter
 				.u64_value_observer(
-					"table.merkle_tree_size",
+					"garage_table.merkle_tree_size",
 					move |observer| {
 						if let Ok(value) = merkle_tree.approximate_len() {
 							observer.observe(
@@ -62,7 +62,7 @@ impl TableMetrics {
 				.init(),
 			_merkle_todo_len: meter
 				.u64_value_observer(
-					"table.merkle_updater_todo_queue_length",
+					"garage_table.merkle_updater_todo_queue_length",
 					move |observer| {
 						if let Ok(v) = merkle_todo.approximate_len() {
 							observer.observe(
@@ -76,7 +76,7 @@ impl TableMetrics {
 				.init(),
 			_insert_queue_len: meter
 				.u64_value_observer(
-					"table.insert_queue_length",
+					"garage_table.insert_queue_length",
 					move |observer| {
 						if let Ok(v) = insert_queue.approximate_len() {
 							observer.observe(
@@ -90,7 +90,7 @@ impl TableMetrics {
 				.init(),
 			_gc_todo_len: meter
 				.u64_value_observer(
-					"table.gc_todo_queue_length",
+					"garage_table.gc_todo_queue_length",
 					move |observer| {
                         if let Ok(value) = gc_todo.approximate_len() {
                             observer.observe(
@@ -104,43 +104,43 @@ impl TableMetrics {
 				.init(),
 
 			get_request_counter: meter
-				.u64_counter("table.get_request_counter")
+				.u64_counter("garage_table.get_request_count")
 				.with_description("Number of get/get_range requests internally made on this table")
 				.init()
 				.bind(&[KeyValue::new("table_name", table_name)]),
 			get_request_duration: meter
-				.f64_value_recorder("table.get_request_duration")
+				.f64_value_recorder("garage_table.get_request_duration")
 				.with_description("Duration of get/get_range requests internally made on this table, in seconds")
 				.init()
 				.bind(&[KeyValue::new("table_name", table_name)]),
 			put_request_counter: meter
-				.u64_counter("table.put_request_counter")
+				.u64_counter("garage_table.put_request_count")
 				.with_description("Number of insert/insert_many requests internally made on this table")
 				.init()
 				.bind(&[KeyValue::new("table_name", table_name)]),
 			put_request_duration: meter
-				.f64_value_recorder("table.put_request_duration")
+				.f64_value_recorder("garage_table.put_request_duration")
 				.with_description("Duration of insert/insert_many requests internally made on this table, in seconds")
 				.init()
 				.bind(&[KeyValue::new("table_name", table_name)]),
 
 			internal_update_counter: meter
-				.u64_counter("table.internal_update_counter")
+				.u64_counter("garage_table.internal_update_count")
 				.with_description("Number of value updates where the value actually changes (includes creation of new key and update of existing key)")
 				.init()
 				.bind(&[KeyValue::new("table_name", table_name)]),
 			internal_delete_counter: meter
-				.u64_counter("table.internal_delete_counter")
+				.u64_counter("garage_table.internal_delete_count")
 				.with_description("Number of value deletions in the tree (due to GC or repartitioning)")
 				.init()
 				.bind(&[KeyValue::new("table_name", table_name)]),
 
 			sync_items_sent: meter
-				.u64_counter("table.sync_items_sent")
+				.u64_counter("garage_table.sync_items_sent")
 				.with_description("Number of data items sent to other nodes during resync procedures")
 				.init(),
 			sync_items_received: meter
-				.u64_counter("table.sync_items_received")
+				.u64_counter("garage_table.sync_items_received")
 				.with_description("Number of data items received from other nodes during resync procedures")
 				.init(),
 		}

src/util/Cargo.toml

@@ -17,6 +17,7 @@ path = "lib.rs"
 garage_db.workspace = true
 garage_net.workspace = true
 
+arbitrary = { optional = true, workspace = true }
 arc-swap.workspace = true
 async-trait.workspace = true
 blake2.workspace = true
@@ -52,6 +53,7 @@ mktemp.workspace = true
 
 [features]
 k2v = []
+arbitrary = ["dep:arbitrary"]
 
 [lints]
 workspace = true

src/util/crdt/crdt.rs

@@ -26,28 +26,6 @@ pub trait Crdt {
 	fn merge(&mut self, other: &Self);
 }
 
-/// `Option<T>` implements Crdt for any type T, even if T doesn't implement CRDT itself: when
-/// different values are detected, they are always merged to None.  This can be used for value
-/// types which shoulnd't be merged, instead of trying to merge things when we know we don't want
-/// to merge them (which is what the `AutoCrdt` trait is used for most of the time). This cases
-/// arises very often, for example with a Lww or a `LwwMap`: the value type has to be a CRDT so that
-/// we have a rule for what to do when timestamps aren't enough to disambiguate (in a distributed
-/// system, anything can happen!), and with `AutoCrdt` the rule is to make an arbitrary (but
-/// deterministic) choice between the two.  When using an `Option<T>` instead with this impl, ambiguity
-/// cases are explicitly stored as None, which allows us to detect the ambiguity and handle it in
-/// the way we want. (this can only work if we are happy with losing the value when an ambiguity
-/// arises)
-impl<T> Crdt for Option<T>
-where
-	T: Eq,
-{
-	fn merge(&mut self, other: &Self) {
-		if self != other {
-			*self = None;
-		}
-	}
-}
-
 /// All types that implement `Ord` (a total order) can also implement a trivial CRDT
 /// defined by the merge rule: `a ⊔ b = max(a, b)`. Implement this trait for your type
 /// to enable this behavior.

src/util/crdt/deletable.rs

@@ -4,6 +4,7 @@ use crate::crdt::crdt::*;
 
 /// Deletable object (once deleted, cannot go back)
 #[derive(Clone, Copy, Debug, Serialize, Deserialize, PartialEq, Eq)]
+#[cfg_attr(feature = "arbitrary", derive(arbitrary::Arbitrary))]
 pub enum Deletable<T> {
 	Present(T),
 	Deleted,

src/util/crdt/lww.rs

@@ -38,6 +38,7 @@ use crate::crdt::crdt::*;
 /// This scheme is used by AWS S3 or Soundcloud and often without knowing
 /// in enterprise when reconciliating databases with ad-hoc scripts.
 #[derive(Clone, Debug, Serialize, Deserialize, PartialEq, Eq)]
+#[cfg_attr(feature = "arbitrary", derive(arbitrary::Arbitrary))]
 pub struct Lww<T> {
 	ts: u64,
 	v: T,