update to v0.2.1

Support old CPUs
Update Cargo.toml files with AGPL license info
2026-08-13 15:46:53 +00:00 · 2021-03-19 13:39:18 +01:00 · 2021-03-19 12:19:40 +01:00 · 2021-03-18 21:59:17 +01:00 · 2021-03-18 21:49:12 +01:00 · 2021-03-18 21:26:34 +01:00
101 changed files with 7265 additions and 4159 deletions
@@ -0,0 +1,81 @@
+kind: pipeline
+name: default
+
+workspace:
+  base: /drone/garage
+
+steps:
+  - name: restore-cache
+    image: meltwater/drone-cache:dev
+    environment:
+      AWS_ACCESS_KEY_ID:
+        from_secret: cache_aws_access_key_id
+      AWS_SECRET_ACCESS_KEY:
+        from_secret: cache_aws_secret_access_key
+    pull: true
+    settings:
+      restore: true
+      archive_format: "gzip"
+      bucket: drone-cache
+      cache_key: '{{ .Repo.Name }}_{{ checksum "garage/Cargo.lock" }}_{{ arch }}_{{ os }}_gzip'
+      region: garage
+      mount:
+        - 'target'
+        - '/drone/cargo/registry/index'
+        - '/drone/cargo/registry/cache'
+        - '/drone/cargo/bin'
+        - '/drone/cargo/git/db'
+      path_style: true
+      endpoint: https://garage.deuxfleurs.fr
+
+  - name: build
+    image: rust:buster
+    environment:
+      CARGO_HOME: /drone/cargo
+    commands:
+      - apt-get update
+      - apt-get install --yes libsodium-dev
+      - pwd
+      - cargo build
+
+  - name: cargo-test
+    image: rust:buster
+    environment:
+      CARGO_HOME: /drone/cargo
+    commands:
+      - apt-get update
+      - apt-get install --yes libsodium-dev
+      - cargo test
+
+  - name: rebuild-cache
+    image: meltwater/drone-cache:dev
+    environment:
+      AWS_ACCESS_KEY_ID:
+        from_secret: cache_aws_access_key_id
+      AWS_SECRET_ACCESS_KEY:
+        from_secret: cache_aws_secret_access_key
+    pull: true
+    settings:
+      rebuild: true
+      archive_format: "gzip"
+      bucket: drone-cache
+      cache_key: '{{ .Repo.Name }}_{{ checksum "garage/Cargo.lock" }}_{{ arch }}_{{ os }}_gzip'
+      region: garage
+      mount:
+        - 'target'
+        - '/drone/cargo/registry/index'
+        - '/drone/cargo/registry/cache'
+        - '/drone/cargo/git/db'
+        - '/drone/cargo/bin'
+      path_style: true
+      endpoint: https://garage.deuxfleurs.fr
+
+  - name: smoke-test
+    image: rust:buster
+    environment:
+      CARGO_HOME: /drone/cargo
+    commands:
+      - apt-get update
+      - apt-get install --yes libsodium-dev awscli python-pip
+      - pip install s3cmd
+      - ./script/test-smoke.sh || (cat /tmp/garage.log; false)
@@ -3,7 +3,7 @@ FROM archlinux:latest
 RUN mkdir -p /garage/meta
 RUN mkdir -p /garage/data
 ENV RUST_BACKTRACE=1
-ENV RUST_LOG=garage=debug
+ENV RUST_LOG=garage=info

 COPY target/release/garage.stripped /garage/garage

@@ -1,5 +1,5 @@
-                    GNU GENERAL PUBLIC LICENSE
-                       Version 3, 29 June 2007
+                    GNU AFFERO GENERAL PUBLIC LICENSE
+                       Version 3, 19 November 2007

 Copyright (C) 2007 Free Software Foundation, Inc. <https://fsf.org/>
 Everyone is permitted to copy and distribute verbatim copies
@@ -7,17 +7,15 @@

                            Preamble

-  The GNU General Public License is a free, copyleft license for
-software and other kinds of works.
+  The GNU Affero General Public License is a free, copyleft license for
+software and other kinds of works, specifically designed to ensure
+cooperation with the community in the case of network server software.

  The licenses for most software and other practical works are designed
 to take away your freedom to share and change the works.  By contrast,
-the GNU General Public License is intended to guarantee your freedom to
+our General Public Licenses are intended to guarantee your freedom to
 share and change all versions of a program--to make sure it remains free
-software for all its users.  We, the Free Software Foundation, use the
-GNU General Public License for most of our software; it applies also to
-any other work released this way by its authors.  You can apply it to
-your programs, too.
+software for all its users.

  When we speak of free software, we are referring to freedom, not
 price.  Our General Public Licenses are designed to make sure that you
@@ -26,44 +24,34 @@ them if you wish), that you receive source code or can get it if you
 want it, that you can change the software or use pieces of it in new
 free programs, and that you know you can do these things.

-  To protect your rights, we need to prevent others from denying you
-these rights or asking you to surrender the rights.  Therefore, you have
-certain responsibilities if you distribute copies of the software, or if
-you modify it: responsibilities to respect the freedom of others.
+  Developers that use our General Public Licenses protect your rights
+with two steps: (1) assert copyright on the software, and (2) offer
+you this License which gives you legal permission to copy, distribute
+and/or modify the software.

-  For example, if you distribute copies of such a program, whether
-gratis or for a fee, you must pass on to the recipients the same
-freedoms that you received.  You must make sure that they, too, receive
-or can get the source code.  And you must show them these terms so they
-know their rights.
+  A secondary benefit of defending all users' freedom is that
+improvements made in alternate versions of the program, if they
+receive widespread use, become available for other developers to
+incorporate.  Many developers of free software are heartened and
+encouraged by the resulting cooperation.  However, in the case of
+software used on network servers, this result may fail to come about.
+The GNU General Public License permits making a modified version and
+letting the public access it on a server without ever releasing its
+source code to the public.

-  Developers that use the GNU GPL protect your rights with two steps:
-(1) assert copyright on the software, and (2) offer you this License
-giving you legal permission to copy, distribute and/or modify it.
+  The GNU Affero General Public License is designed specifically to
+ensure that, in such cases, the modified source code becomes available
+to the community.  It requires the operator of a network server to
+provide the source code of the modified version running there to the
+users of that server.  Therefore, public use of a modified version, on
+a publicly accessible server, gives the public access to the source
+code of the modified version.

-  For the developers' and authors' protection, the GPL clearly explains
-that there is no warranty for this free software.  For both users' and
-authors' sake, the GPL requires that modified versions be marked as
-changed, so that their problems will not be attributed erroneously to
-authors of previous versions.
-
-  Some devices are designed to deny users access to install or run
-modified versions of the software inside them, although the manufacturer
-can do so.  This is fundamentally incompatible with the aim of
-protecting users' freedom to change the software.  The systematic
-pattern of such abuse occurs in the area of products for individuals to
-use, which is precisely where it is most unacceptable.  Therefore, we
-have designed this version of the GPL to prohibit the practice for those
-products.  If such problems arise substantially in other domains, we
-stand ready to extend this provision to those domains in future versions
-of the GPL, as needed to protect the freedom of users.
-
-  Finally, every program is threatened constantly by software patents.
-States should not allow patents to restrict development and use of
-software on general-purpose computers, but in those that do, we wish to
-avoid the special danger that patents applied to a free program could
-make it effectively proprietary.  To prevent this, the GPL assures that
-patents cannot be used to render the program non-free.
+  An older license, called the Affero General Public License and
+published by Affero, was designed to accomplish similar goals.  This is
+a different license, not a version of the Affero GPL, but Affero has
+released a new version of the Affero GPL which permits relicensing under
+this license.

  The precise terms and conditions for copying, distribution and
 modification follow.
@@ -72,7 +60,7 @@ modification follow.

  0. Definitions.

-  "This License" refers to version 3 of the GNU General Public License.
+  "This License" refers to version 3 of the GNU Affero General Public License.

  "Copyright" also means copyright-like laws that apply to other kinds of
 works, such as semiconductor masks.
@@ -549,35 +537,45 @@ to collect a royalty for further conveying from those to whom you convey
 the Program, the only way you could satisfy both those terms and this
 License would be to refrain entirely from conveying the Program.

-  13. Use with the GNU Affero General Public License.
+  13. Remote Network Interaction; Use with the GNU General Public License.
+
+  Notwithstanding any other provision of this License, if you modify the
+Program, your modified version must prominently offer all users
+interacting with it remotely through a computer network (if your version
+supports such interaction) an opportunity to receive the Corresponding
+Source of your version by providing access to the Corresponding Source
+from a network server at no charge, through some standard or customary
+means of facilitating copying of software.  This Corresponding Source
+shall include the Corresponding Source for any work covered by version 3
+of the GNU General Public License that is incorporated pursuant to the
+following paragraph.

  Notwithstanding any other provision of this License, you have
 permission to link or combine any covered work with a work licensed
-under version 3 of the GNU Affero General Public License into a single
+under version 3 of the GNU General Public License into a single
 combined work, and to convey the resulting work.  The terms of this
 License will continue to apply to the part which is the covered work,
-but the special requirements of the GNU Affero General Public License,
-section 13, concerning interaction through a network will apply to the
-combination as such.
+but the work with which it is combined will remain governed by version
+3 of the GNU General Public License.

  14. Revised Versions of this License.

  The Free Software Foundation may publish revised and/or new versions of
-the GNU General Public License from time to time.  Such new versions will
-be similar in spirit to the present version, but may differ in detail to
+the GNU Affero General Public License from time to time.  Such new versions
+will be similar in spirit to the present version, but may differ in detail to
 address new problems or concerns.

  Each version is given a distinguishing version number.  If the
-Program specifies that a certain numbered version of the GNU General
+Program specifies that a certain numbered version of the GNU Affero General
 Public License "or any later version" applies to it, you have the
 option of following the terms and conditions either of that numbered
 version or of any later version published by the Free Software
 Foundation.  If the Program does not specify a version number of the
-GNU General Public License, you may choose any version ever published
+GNU Affero General Public License, you may choose any version ever published
 by the Free Software Foundation.

  If the Program specifies that a proxy can decide which future
-versions of the GNU General Public License can be used, that proxy's
+versions of the GNU Affero General Public License can be used, that proxy's
 public statement of acceptance of a version permanently authorizes you
 to choose that version for the Program.

@@ -635,41 +633,29 @@ the "copyright" line and a pointer to where the full notice is found.
    Copyright (C) <year>  <name of author>

    This program is free software: you can redistribute it and/or modify
-    it under the terms of the GNU General Public License as published by
+    it under the terms of the GNU Affero General Public License as published by
    the Free Software Foundation, either version 3 of the License, or
    (at your option) any later version.

    This program is distributed in the hope that it will be useful,
    but WITHOUT ANY WARRANTY; without even the implied warranty of
    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-    GNU General Public License for more details.
+    GNU Affero General Public License for more details.

-    You should have received a copy of the GNU General Public License
+    You should have received a copy of the GNU Affero General Public License
    along with this program.  If not, see <https://www.gnu.org/licenses/>.

 Also add information on how to contact you by electronic and paper mail.

-  If the program does terminal interaction, make it output a short
-notice like this when it starts in an interactive mode:
-
-    <program>  Copyright (C) <year>  <name of author>
-    This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
-    This is free software, and you are welcome to redistribute it
-    under certain conditions; type `show c' for details.
-
-The hypothetical commands `show w' and `show c' should show the appropriate
-parts of the General Public License.  Of course, your program's commands
-might be different; for a GUI interface, you would use an "about box".
+  If your software can interact with users remotely through a computer
+network, you should also make sure that it provides a way for users to
+get its source.  For example, if your program is a web application, its
+interface could display a "Source" link that leads users to an archive
+of the code.  There are many ways you could offer source, and different
+solutions will be better for different programs; see section 13 for the
+specific requirements.

  You should also get your employer (if you work as a programmer) or school,
 if any, to sign a "copyright disclaimer" for the program, if necessary.
-For more information on this, and how to apply and follow the GNU GPL, see
+For more information on this, and how to apply and follow the GNU AGPL, see
 <https://www.gnu.org/licenses/>.
-
-  The GNU General Public License does not permit incorporating your program
-into proprietary programs.  If your program is a subroutine library, you
-may consider it more useful to permit linking proprietary applications with
-the library.  If this is what you want to do, use the GNU Lesser General
-Public License instead of this License.  But first, please read
-<https://www.gnu.org/licenses/why-not-lgpl.html>.
-
@@ -2,13 +2,10 @@ BIN=target/release/garage
 DOCKER=lxpz/garage_amd64

 all:
-	#cargo fmt || true
-	#RUSTFLAGS="-C link-arg=-fuse-ld=lld" cargo build
-	cargo build
+	clear; RUSTFLAGS="-C link-arg=-fuse-ld=lld -C target-cpu=x86-64 -C target-feature=+sse2" cargo build --no-default-features

 $(BIN):
-	#RUSTFLAGS="-C link-arg=-fuse-ld=lld" cargo build --release
-	cargo build --release
+	RUSTFLAGS="-C link-arg=-fuse-ld=lld -C target-cpu=x86-64 -C target-feature=+sse2" cargo build --release --no-default-features

 $(BIN).stripped: $(BIN)
 	cp $^ $@
@@ -1,4 +1,11 @@
-# Garage
+Garage [![Build Status](https://drone.deuxfleurs.fr/api/badges/Deuxfleurs/garage/status.svg)](https://drone.deuxfleurs.fr/Deuxfleurs/garage)
+===
+
+<p align="center" style="text-align:center;">
+	<a href="https://git.deuxfleurs.fr/Deuxfleurs/garage">
+	<img alt="Garage logo" src="doc/logo/garage.png" height="200" />
+	</a>
+</p>

 Garage is a lightweight S3-compatible distributed object store, with the following goals:

@@ -12,121 +19,8 @@ Non-goals include:

 - Extremely high performance
 - Complete implementation of the S3 API
- Erasure coding (our replication model is simply to copy the data as is on several nodes)
+- Erasure coding (our replication model is simply to copy the data as is on several nodes, in different datacenters if possible)

 Our main use case is to provide a distributed storage layer for small-scale self hosted services such as [Deuxfleurs](https://deuxfleurs.fr).

-## Development
-
-We propose the following quickstart to setup a full dev. environment as quickly as possible:
-
-  1. Setup a rust/cargo environment. eg. `dnf install rust cargo`
-  2. Install awscli v2 by following the guide [here](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html).
-  3. Run `cargo build` to build the project
-  4. Run `./script/dev-cluster.sh` to launch a test cluster (feel free to read the script)
-  5. Run `./script/dev-configure.sh` to configure your test cluster with default values (same datacenter, 100 tokens)
-  6. Run `./script/dev-bucket.sh` to create a bucket named `eprouvette` and an API key that will be stored in `/tmp/garage.s3`
-  7. Run `source ./script/dev-env-aws.sh` to configure your CLI environment
-  8. You can use `garage` to manage the cluster. Try `garage --help`.
-  9. You can use the `awsgrg` alias to add, remove, and delete files. Try `awsgrg help`, `awsgrg cp /proc/cpuinfo s3://eprouvette/cpuinfo.txt`, or `awsgrg ls s3://eprouvette`. `awsgrg` is a wrapper on the `aws s3` command pre-configured with the previously generated API key (the one in `/tmp/garage.s3`) and localhost as the endpoint.
-
-Now you should be ready to start hacking on garage!
-
-## S3 compatibility
-
-Only a subset of S3 is supported: adding, listing, getting and deleting files in a bucket.
-Bucket management, ACL and other advanced features are not (yet?) handled through the S3 API but through the `garage` CLI.
-We primarily test `garage` against the `awscli` tool and `nextcloud`.
-
-## Setting up Garage
-
-Use the `genkeys.sh` script to generate TLS keys for encrypting communications between Garage nodes.
-The script takes no arguments and will generate keys in `pki/`.
-This script creates a certificate authority `garage-ca` which signs certificates for individual Garage nodes.
-Garage nodes from a same cluster authenticate themselves by verifying that they have certificates signed by the same certificate authority.
-
-Garage requires two locations to store its data: a metadata directory, and a data directory.
-The metadata directory is used to store metadata such as object lists, and should ideally be located on an SSD drive.
-The data directory is used to store the chunks of data of the objects stored in Garage.
-In a typical deployment the data directory is stored on a standard HDD.
-
-Garage does not handle TLS for its S3 API endpoint. This should be handled by adding a reverse proxy.
-
-Create a configuration file with the following structure:
-
-```
-block_size = 1048576			# objects are split in blocks of maximum this number of bytes
-
-metadata_dir = "/path/to/ssd/metadata/directory"
-data_dir = "/path/to/hdd/data/directory"
-
-rpc_bind_addr = "[::]:3901"		# the port other Garage nodes will use to talk to this node
-
-bootstrap_peers = [
-	# Ideally this list should contain the IP addresses of all other Garage nodes of the cluster.
-	# Use Ansible or any kind of configuration templating to generate this automatically.
-	"10.0.0.1:3901",
-	"10.0.0.2:3901",
-	"10.0.0.3:3901",
-]
-
-# optionnal: garage can find cluster nodes automatically using a Consul server
-# garage only does lookup but does not register itself, registration should be handled externally by e.g. Nomad
-consul_host = "localhost:8500"	# optionnal: host name of a Consul server for automatic peer discovery
-consul_service_name = "garage"  # optionnal: service name to look up on Consul
-
-max_concurrent_rpc_requests = 12
-data_replication_factor = 3
-meta_replication_factor = 3
-meta_epidemic_fanout = 3
-
-[rpc_tls]
-# NOT RECOMMENDED: you can skip this section if you don't want to encrypt intra-cluster traffic
-# Thanks to genkeys.sh, generating the keys and certificates is easy, so there is NO REASON NOT TO DO IT.
-ca_cert = "/path/to/garage/pki/garage-ca.crt"
-node_cert = "/path/to/garage/pki/garage.crt"
-node_key = "/path/to/garage/pki/garage.key"
-
-[s3_api]
-api_bind_addr = "[::1]:3900"	# the S3 API port, HTTP without TLS. Add a reverse proxy for the TLS part.
-s3_region = "garage"				# set this to anything. S3 API calls will fail if they are not made against the region set here.
-
-[s3_web]
-bind_addr = "[::1]:3902"
-root_domain = ".garage.tld"
-index = "index.html"
-```
-
-Build Garage using `cargo build --release`.
-Then, run it using either `./target/release/garage server -c path/to/config_file.toml` or `cargo run --release -- server -c path/to/config_file.toml`.
-
-Set the `RUST_LOG` environment to `garage=debug` to dump some debug information.
-Set it to `garage=trace` to dump even more debug information.
-Set it to `garage=warn` to show nothing except warnings and errors.
-
-## Setting up cluster nodes
-
-Once all your `garage` nodes are running, you will need to:
-
-1. check that they are correctly talking to one another;
-2. configure them with their physical location (in the case of a multi-dc deployment) and a number of "ring tokens" proportionnal to the storage space available on each node;
-3. create some S3 API keys and buckets;
-4. ???;
-5. profit!
-
-To run these administrative tasks, you will need to use the `garage` command line tool and it to connect to any of the cluster's nodes on the RPC port.
-The `garage` CLI also needs TLS keys and certificates of its own to authenticate and be authenticated in the cluster.
-A typicall invocation will be as follows:
-
-```
-./target/release/garage --ca-cert=pki/garage-ca.crt --client-cert=pki/garage-client.crt --client-key=pki/garage-client.key <...>
-```
-
-
-## Notes to self
-
-### What to repair
-
- `tables`: to do a full sync of metadata, should not be necessary because it is done every hour by the system
- `versions` and `block_refs`: very time consuming, usefull if deletions have not been propagated, improves garbage collection
- `blocks`: very usefull to resync/rebalance blocks betweeen nodes
+**[Go to the documentation](https://garagehq.deuxfleurs.fr)**
@@ -0,0 +1 @@
+book
@@ -0,0 +1,6 @@
+[book]
+authors = ["Quentin Dufour"]
+language = "en"
+multilingual = false
+src = "src"
+title = "Garage Documentation"
@@ -0,0 +1,31 @@
+# Summary
+
+[The Garage Data Store](./intro.md)
+
+- [Getting Started](./getting_started/index.md)
+  - [Get a binary](./getting_started/binary.md)
+  - [Configure the daemon](./getting_started/daemon.md)
+  - [Control the daemon](./getting_started/control.md)
+  - [Configure a cluster](./getting_started/cluster.md)
+  - [Create buckets and keys](./getting_started/bucket.md)
+  - [Handle files](./getting_started/files.md)
+
+- [Cookbook](./cookbook/index.md)
+  - [Host a website](./cookbook/website.md)
+  - [Integrate as a media backend]()
+  - [Operate a cluster]()
+
+- [Reference Manual](./reference_manual/index.md)
+  - [Garage CLI]()
+  - [S3 API](./reference_manual/s3_compatibility.md)
+
+- [Design](./design/index.md)
+  - [Related Work](./design/related_work.md)
+  - [Internals](./design/internals.md)
+
+- [Development](./development/index.md)
+  - [Setup your environment](./development/devenv.md)
+  - [Your first contribution]()
+
+- [Working Documents](./working_documents/index.md)
+  - [Load Balancing Data](./working_documents/load_balancing.md)
@@ -0,0 +1,5 @@
+# Cookbook
+
+A cookbook, when you cook, is a collection of recipes.
+Similarly, Garage's cookbook contains a collection of recipes that are known to works well!
+This chapter could also be referred as "Tutorials" or "Best practises".
@@ -0,0 +1 @@
+# Host a website
@@ -0,0 +1,5 @@
+# Design
+
+The design section helps you to see Garage from a "big picture" perspective.
+It will allow you to understand if Garage is a good fit for you,
+how to better use it, how to contribute to it, what can Garage could and could not do, etc.
@@ -0,0 +1,158 @@
+**WARNING: this documentation is more a "design draft", which was written before Garage's actual implementation. The general principle is similar but details have not yet been updated.**
+
+#### Modules
+
+- `membership/`: configuration, membership management (gossip of node's presence and status), ring generation --> what about Serf (used by Consul/Nomad) : https://www.serf.io/? Seems a huge library with many features so maybe overkill/hard to integrate
+- `metadata/`: metadata management
+- `blocks/`: block management, writing, GC and rebalancing
+- `internal/`: server to server communication (HTTP server and client that reuses connections, TLS if we want, etc)
+- `api/`: S3 API
+- `web/`: web management interface
+
+#### Metadata tables
+
+**Objects:**
+
+- *Hash key:* Bucket name (string)
+- *Sort key:* Object key (string)
+- *Sort key:* Version timestamp (int)
+- *Sort key:* Version UUID (string)
+- Complete: bool
+- Inline: bool, true for objects < threshold (say 1024)
+- Object size (int)
+- Mime type (string)
+- Data for inlined objects (blob)
+- Hash of first block otherwise (string)
+
+*Having only a hash key on the bucket name will lead to storing all file entries of this table for a specific bucket on a single node. At the same time, it is the only way I see to rapidly being able to list all bucket entries...*
+
+**Blocks:**
+
+- *Hash key:* Version UUID (string)
+- *Sort key:* Offset of block in total file (int)
+- Hash of data block (string)
+
+A version is defined by the existence of at least one entry in the blocks table for a certain version UUID.
+We must keep the following invariant: if a version exists in the blocks table, it has to be referenced in the objects table.
+We explicitly manage concurrent versions of an object: the version timestamp and version UUID columns are index columns, thus we may have several concurrent versions of an object.
+Important: before deleting an older version from the objects table, we must make sure that we did a successfull delete of the blocks of that version from the blocks table.
+
+Thus, the workflow for reading an object is as follows:
+
+1. Check permissions (LDAP)
+2. Read entry in object table. If data is inline, we have its data, stop here.
+   -> if several versions, take newest one and launch deletion of old ones in background
+3. Read first block from cluster. If size <= 1 block, stop here.
+4. Simultaneously with previous step, if size > 1 block: query the Blocks table for the IDs of the next blocks
+5. Read subsequent blocks from cluster
+
+Workflow for PUT:
+
+1. Check write permission (LDAP)
+2. Select a new version UUID
+3. Write a preliminary entry for the new version in the objects table with complete = false
+4. Send blocks to cluster and write entries in the blocks table
+5. Update the version with complete = true and all of the accurate information (size, etc)
+6. Return success to the user
+7. Launch a background job to check and delete older versions
+
+Workflow for DELETE:
+
+1. Check write permission (LDAP)
+2. Get current version (or versions) in object table
+3. Do the deletion of those versions NOT IN A BACKGROUND JOB THIS TIME
+4. Return succes to the user if we were able to delete blocks from the blocks table and entries from the object table
+
+To delete a version:
+
+1. List the blocks from Cassandra
+2. For each block, delete it from cluster. Don't care if some deletions fail, we can do GC.
+3. Delete all of the blocks from the blocks table
+4. Finally, delete the version from the objects table
+
+Known issue: if someone is reading from a version that we want to delete and the object is big, the read might be interrupted. I think it is ok to leave it like this, we just cut the connection if data disappears during a read.
+
+("Soit P un problème, on s'en fout est une solution à ce problème")
+
+#### Block storage on disk
+
+**Blocks themselves:**
+
+- file path = /blobs/(first 3 hex digits of hash)/(rest of hash)
+
+**Reverse index for GC & other block-level metadata:**
+
+- file path = /meta/(first 3 hex digits of hash)/(rest of hash)
+- map block hash -> set of version UUIDs where it is referenced
+
+Usefull metadata:
+
+- list of versions that reference this block in the Casandra table, so that we can do GC by checking in Cassandra that the lines still exist
+- list of other nodes that we know have acknowledged a write of this block, usefull in the rebalancing algorithm
+
+Write strategy: have a single thread that does all write IO so that it is serialized (or have several threads that manage independent parts of the hash space). When writing a blob, write it to a temporary file, close, then rename so that a concurrent read gets a consistent result (either not found or found with whole content).
+
+Read strategy: the only read operation is get(hash) that returns either the data or not found (can do a corruption check as well and return corrupted state if it is the case). Can be done concurrently with writes.
+
+**Internal API:**
+
+- get(block hash) -> ok+data/not found/corrupted
+- put(block hash & data, version uuid + offset) -> ok/error
+- put with no data(block hash, version uuid + offset) -> ok/not found plz send data/error
+- delete(block hash, version uuid + offset) -> ok/error
+
+GC: when last ref is deleted, delete block.
+Long GC procedure: check in Cassandra that version UUIDs still exist and references this block.
+
+Rebalancing: takes as argument the list of newly added nodes.
+
+- List all blocks that we have. For each block:
+- If it hits a newly introduced node, send it to them.
+  Use put with no data first to check if it has to be sent to them already or not.
+  Use a random listing order to avoid race conditions (they do no harm but we might have two nodes sending the same thing at the same time thus wasting time).
+- If it doesn't hit us anymore, delete it and its reference list.
+
+Only one balancing can be running at a same time. It can be restarted at the beginning with new parameters.
+
+#### Membership management
+
+Two sets of nodes:
+
+- set of nodes from which a ping was recently received, with status: number of stored blocks, request counters, error counters, GC%, rebalancing%
+  (eviction from this set after say 30 seconds without ping)
+- set of nodes that are part of the system, explicitly modified by the operator using the web UI (persisted to disk),
+  is a CRDT using a version number for the value of the whole set
+
+Thus, three states for nodes:
+
+- healthy: in both sets
+- missing: not pingable but part of desired cluster
+- unused/draining: currently present but not part of the desired cluster, empty = if contains nothing, draining = if still contains some blocks
+
+Membership messages between nodes:
+
+- ping with current state + hash of current membership info -> reply with same info
+- send&get back membership info (the ids of nodes that are in the two sets): used when no local membership change in a long time and membership info hash discrepancy detected with first message (passive membership fixing with full CRDT gossip)
+- inform of newly pingable node(s) -> no result, when receive new info repeat to all (reliable broadcast)
+- inform of operator membership change -> no result, when receive new info repeat to all (reliable broadcast)
+
+Ring: generated from the desired set of nodes, however when doing read/writes on the ring, skip nodes that are known to be not pingable.
+The tokens are generated in a deterministic fashion from node IDs (hash of node id + token number from 1 to K).
+Number K of tokens per node: decided by the operator & stored in the operator's list of nodes CRDT. Default value proposal: with node status information also broadcast disk total size and free space, and propose a default number of tokens equal to 80%Free space / 10Gb. (this is all user interface)
+
+
+#### Constants
+
+- Block size: around 1MB ? --> Exoscale use 16MB chunks
+- Number of tokens in the hash ring: one every 10Gb of allocated storage
+- Threshold for storing data directly in Cassandra objects table: 1kb bytes (maybe up to 4kb?)
+- Ping timeout (time after which a node is registered as unresponsive/missing): 30 seconds
+- Ping interval: 10 seconds
+- ??
+
+#### Links
+
+- CDC: <https://www.usenix.org/system/files/conference/atc16/atc16-paper-xia.pdf>
+- Erasure coding: <http://web.eecs.utk.edu/~jplank/plank/papers/CS-08-627.html>
+- [Openstack Storage Concepts](https://docs.openstack.org/arch-design/design-storage/design-storage-concepts.html)
+- [RADOS](https://ceph.com/wp-content/uploads/2016/08/weil-rados-pdsw07.pdf)
@@ -0,0 +1,56 @@
+# Related Work
+
+## Context
+
+Data storage is critical: it can lead to data loss if done badly and/or on hardware failure.
+Filesystems + RAID can help on a single machine but a machine failure can put the whole storage offline.
+Moreover, it put a hard limit on scalability. Often this limit can be pushed back far away by buying expensive machines.
+But here we consider non specialized off the shelf machines that can be as low powered and subject to failures as a raspberry pi.
+
+Distributed storage may help to solve both availability and scalability problems on these machines.
+Many solutions were proposed, they can be categorized as block storage, file storage and object storage depending on the abstraction they provide.
+
+## Overview
+
+Block storage is the most low level one, it's like exposing your raw hard drive over the network.
+It requires very low latencies and stable network, that are often dedicated.
+However it provides disk devices that can be manipulated by the operating system with the less constraints: it can be partitioned with any filesystem, meaning that it supports even the most exotic features.
+We can cite [iSCSI](https://en.wikipedia.org/wiki/ISCSI) or [Fibre Channel](https://en.wikipedia.org/wiki/Fibre_Channel).
+Openstack Cinder proxy previous solution to provide an uniform API.
+
+File storage provides a higher abstraction, they are one filesystem among others, which means they don't necessarily have all the exotic features of every filesystem.
+Often, they relax some POSIX constraints while many applications will still be compatible without any modification.
+As an example, we are able to run MariaDB (very slowly) over GlusterFS...
+We can also mention CephFS (read [RADOS](https://ceph.com/wp-content/uploads/2016/08/weil-rados-pdsw07.pdf) whitepaper), Lustre, LizardFS, MooseFS, etc.
+OpenStack Manila proxy previous solutions to provide an uniform API.
+
+Finally object storages provide the highest level abstraction.
+They are the testimony that the POSIX filesystem API is not adapted to distributed filesystems.
+Especially, the strong concistency has been dropped in favor of eventual consistency which is way more convenient and powerful in presence of high latencies and unreliability.
+We often read about S3 that pioneered the concept that it's a filesystem for the WAN.
+Applications must be adapted to work for the desired object storage service.
+Today, the S3 HTTP REST API acts as a standard in the industry.
+However, Amazon S3 source code is not open but alternatives were proposed.
+We identified Minio, Pithos, Swift and Ceph.
+Minio/Ceph enforces a total order, so properties similar to a (relaxed) filesystem.
+Swift and Pithos are probably the most similar to AWS S3 with their consistent hashing ring.
+However Pithos is not maintained anymore. More precisely the company that published Pithos version 1 has developped a second version 2 but has not open sourced it.
+Some tests conducted by the [ACIDES project](https://acides.org/) have shown that Openstack Swift consumes way more resources (CPU+RAM) that we can afford. Furthermore, people developing Swift have not designed their software for geo-distribution.
+
+There were many attempts in research too. I am only thinking to [LBFS](https://pdos.csail.mit.edu/papers/lbfs:sosp01/lbfs.pdf) that was used as a basis for Seafile. But none of them have been effectively implemented yet.
+
+## Existing software
+
+**[Pithos](https://github.com/exoscale/pithos) :** 
+Pithos has been abandonned and should probably not used yet, in the following we explain why we did not pick their design.
+Pithos was relying as a S3 proxy in front of Cassandra (and was working with Scylla DB too).
+From its designers' mouth, storing data in Cassandra has shown its limitations justifying the project abandonment.
+They built a closed-source version 2 that does not store blobs in the database (only metadata) but did not communicate further on it.
+We considered there v2's design but concluded that it does not fit both our *Self-contained & lightweight* and *Simple* properties. It makes the development, the deployment and the operations more complicated while reducing the flexibility.
+
+**[IPFS](https://ipfs.io/) :**
+*Not written yet*
+
+## Specific research papers
+
+*Not yet written*
@@ -0,0 +1,17 @@
+# Setup your development environment
+
+We propose the following quickstart to setup a full dev. environment as quickly as possible:
+
+  1. Setup a rust/cargo environment. eg. `dnf install rust cargo`
+  2. Install awscli v2 by following the guide [here](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html).
+  3. Run `cargo build` to build the project
+  4. Run `./script/dev-cluster.sh` to launch a test cluster (feel free to read the script)
+  5. Run `./script/dev-configure.sh` to configure your test cluster with default values (same datacenter, 100 tokens)
+  6. Run `./script/dev-bucket.sh` to create a bucket named `eprouvette` and an API key that will be stored in `/tmp/garage.s3`
+  7. Run `source ./script/dev-env-aws.sh` to configure your CLI environment
+  8. You can use `garage` to manage the cluster. Try `garage --help`.
+  9. You can use the `awsgrg` alias to add, remove, and delete files. Try `awsgrg help`, `awsgrg cp /proc/cpuinfo s3://eprouvette/cpuinfo.txt`, or `awsgrg ls s3://eprouvette`. `awsgrg` is a wrapper on the `aws s3` command pre-configured with the previously generated API key (the one in `/tmp/garage.s3`) and localhost as the endpoint.
+
+Now you should be ready to start hacking on garage!
+
+
@@ -0,0 +1,4 @@
+# Development
+
+Now that you are a Garage expert, you want to enhance it, you are in the right place!
+We discuss here how to hack on Garage, how we manage its development, etc.
@@ -0,0 +1,44 @@
+# Get a binary
+
+Currently, only two installations procedures are supported for Garage: from Docker (x86\_64 for Linux) and from source.
+In the future, we plan to add a third one, by publishing a compiled binary (x86\_64 for Linux).
+We did not test other architecture/operating system but, as long as your architecture/operating system is supported by Rust, you should be able to run Garage (feel free to report your tests!).
+
+## From Docker
+
+Our docker image is currently named `lxpz/garage_amd64` and is stored on the [Docker Hub](https://hub.docker.com/r/lxpz/garage_amd64/tags?page=1&ordering=last_updated).
+We encourage you to use a fixed tag (eg. `v0.1.1d`) and not the `latest` tag.
+For this example, we will use the latest published version at the time of the writing which is `v0.1.1d` but it's up to you
+to check [the most recent versions on the Docker Hub](https://hub.docker.com/r/lxpz/garage_amd64/tags?page=1&ordering=last_updated).
+
+For example:
+
+```
+sudo docker pull lxpz/garage_amd64:v0.1.1d
+```
+
+## From source
+
+Garage is a standard Rust project.
+First, you need `rust` and `cargo`.
+On Debian:
+
+```bash
+sudo apt-get update
+sudo apt-get install -y rustc cargo
+```
+
+Then, you can ask cargo to install the binary for you:
+
+```bash
+cargo install garage
+```
+
+That's all, `garage` should be in `$HOME/.cargo/bin`.
+You can add this folder to your `$PATH` or copy the binary somewhere else on your system.
+For the following, we will assume you copied it in `/usr/local/bin/garage`:
+
+```bash
+sudo cp $HOME/.cargo/bin/garage /usr/local/bin/garage
+```
+
@@ -0,0 +1,78 @@
+# Create buckets and keys
+
+*We use a command named `garagectl` which is in fact an alias you must define as explained in the [Control the daemon](./daemon.md) section.*
+
+In this section, we will suppose that we want to create a bucket named `nextcloud-bucket`
+that will be accessed through a key named `nextcloud-app-key`.
+
+Don't forget that `help` command and `--help` subcommands can help you anywhere, the CLI tool is self-documented! Two examples:
+
+```
+garagectl help
+garagectl bucket allow --help
+```
+
+## Create a bucket
+
+Fine, now let's create a bucket (we imagine that you want to deploy nextcloud):
+
+```
+garagectl bucket create nextcloud-bucket
+```
+
+Check that everything went well:
+
+```
+garagectl bucket list
+garagectl bucket info nextcloud-bucket
+```
+
+## Create an API key
+
+Now we will generate an API key to access this bucket.
+Note that API keys are independent of buckets: one key can access multiple buckets, multiple keys can access one bucket.
+
+Now, let's start by creating a key only for our PHP application:
+
+```
+garagectl key new --name nextcloud-app-key
+```
+
+You will have the following output (this one is fake, `key_id` and `secret_key` were generated with the openssl CLI tool):
+
+```javascript
+Key { 
+  key_id: "GK3515373e4c851ebaad366558", 
+  secret_key: "7d37d093435a41f2aab8f13c19ba067d9776c90215f56614adad6ece597dbb34", 
+  name: "nextcloud-app-key", 
+  name_timestamp: 1603280506694, 
+  deleted: false, 
+  authorized_buckets: [] 
+}
+```
+
+Check that everything works as intended (be careful, info works only with your key identifier and not with its friendly name!):
+
+```
+garagectl key list
+garagectl key info GK3515373e4c851ebaad366558
+```
+
+## Allow a key to access a bucket
+
+Now that we have a bucket and a key, we need to give permissions to the key on the bucket!
+
+```
+garagectl bucket allow \
+  --read \
+  --write 
+  nextcloud-bucket \
+  --key GK3515373e4c851ebaad366558
+```
+
+You can check at any times allowed keys on your bucket with:
+
+```
+garagectl bucket info nextcloud-bucket
+```
+
@@ -0,0 +1,72 @@
+# Configure a cluster
+
+*We use a command named `garagectl` which is in fact an alias you must define as explained in the [Control the daemon](./daemon.md) section.*
+
+In this section, we will inform garage of the disk space available on each node of the cluster
+as well as the site (think datacenter) of each machine.
+
+## Test cluster
+
+As this part is not relevant for a test cluster, you can use this one-liner to create a basic topology:
+
+```bash
+garagectl status | grep UNCONFIGURED | grep -Po '^[0-9a-f]+' | while read id; do 
+    garagectl node configure -d dc1 -n 10 $id
+done
+```
+
+## Real-world cluster
+
+For our example, we will suppose we have the following infrastructure (Tokens, Identifier and Datacenter are specific values to garage described in the following):
+
+| Location | Name    | Disk Space | `Tokens` | `Identifier` | `Datacenter` |
+|----------|---------|------------|----------|--------------|--------------|
+| Paris    | Mercury | 1 To       | `100`    | `8781c5`     | `par1`       |
+| Paris    | Venus   | 2 To       | `200`    | `2a638e`     | `par1`       |
+| London   | Earth   | 2 To       | `200`    | `68143d`     | `lon1`       |
+| Brussels | Mars    | 1.5 To     | `150`    | `212f75`     | `bru1`       |
+
+### Identifier
+
+After its first launch, garage generates a random and unique identifier for each nodes, such as:
+
+```
+8781c50c410a41b363167e9d49cc468b6b9e4449b6577b64f15a249a149bdcbc
+```
+
+Often a shorter form can be used, containing only the beginning of the identifier, like `8781c5`,
+which identifies the server "Mercury" located in "Paris" according to our previous table.
+
+The most simple way to match an identifier to a node is to run:
+
+```
+garagectl status
+```
+
+It will display the IP address associated with each node; from the IP address you will be able to recognize the node.
+
+### Tokens
+
+Garage reasons on an arbitrary metric about disk storage that is named "tokens".
+The number of tokens must be proportional to the disk space dedicated to the node.
+Additionaly, ideally the number of tokens must be in the order of magnitude of 100
+to provide a good trade-off between data load balancing and performances (*this sentence must be verified, it may be wrong*).
+
+Here we chose 1 token = 10 Go but you are free to select the value that best fit your needs.
+
+### Datacenter
+
+Datacenter are simply a user-chosen identifier that identify a group of server that are located in the same place.
+It is up to the system administrator deploying garage to identify what does "the same place" means.
+Behind the scene, garage will try to store the same data on different sites to provide high availability despite a data center failure.
+
+### Inject the topology
+
+Given the information above, we will configure our cluster as follow:
+
+```
+garagectl node configure --datacenter par1 -n 100 -t mercury 8781c5
+garagectl node configure --datacenter par1 -n 200 -t venus 2a638e
+garagectl node configure --datacenter lon1 -n 200 -t earth 68143d
+garagectl node configure --datacenter bru1 -n 150 -t mars 212f75
+```
@@ -0,0 +1,77 @@
+# Control the daemon
+
+The `garage` binary has two purposes:
+  - it acts as a daemon when launched with `garage server ...`
+  - it acts as a control tool for the daemon when launched with any other command
+
+In this section, we will see how to use the `garage` binary as a control tool for the daemon we just started.
+You first need to get a shell having access to this binary, which depends of your configuration:
+  - with `docker-compose`, run `sudo docker-compose exec g1 bash` then `/garage/garage`
+  - with `docker`, run `sudo docker exec -ti garaged bash` then `/garage/garage`
+  - with `systemd`, simply run `/usr/local/bin/garage` if you followed previous instructions
+
+*You can also install the binary on your machine to remotely control the cluster.*
+
+## Talk to the daemon and create an alias
+
+`garage` requires 4 options to talk with the daemon:
+
+```
+--ca-cert <ca-cert>            
+--client-cert <client-cert>    
+--client-key <client-key>      
+-h, --rpc-host <rpc-host>
+```
+
+The 3 first ones are certificates and keys needed by TLS, the last one is simply the address of garage's RPC endpoint.
+Because we configure garage directly from the server, we do not need to set `--rpc-host`.
+To avoid typing the 3 first options each time we want to run a command, we will create an alias.
+
+### `docker-compose` alias
+
+```bash
+alias garagectl='/garage/garage \
+  --ca-cert /pki/garage-ca.crt \
+  --client-cert /pki/garage.crt \
+  --client-key /pki/garage.key'
+```
+
+### `docker` alias
+
+```bash
+alias garagectl='/garage/garage \
+  --ca-cert /etc/garage/pki/garage-ca.crt \
+  --client-cert /etc/garage/pki/garage.crt \
+  --client-key /etc/garage/pki/garage.key'
+```
+
+
+### raw binary alias
+
+```bash
+alias garagectl='/usr/local/bin/garage \
+  --ca-cert /etc/garage/pki/garage-ca.crt \
+  --client-cert /etc/garage/pki/garage.crt \
+  --client-key /etc/garage/pki/garage.key'
+```
+
+Of course, if your deployment does not match exactly one of this alias, feel free to adapt it to your needs!
+
+## Test the alias
+
+You can test your alias by running a simple command such as:
+
+```
+garagectl status
+```
+
+You should get something like that as result:
+
+```
+Healthy nodes:
+2a638ed6c775b69a…	37f0ba978d27	[::ffff:172.20.0.101]:3901	UNCONFIGURED/REMOVED
+68143d720f20c89d…	9795a2f7abb5	[::ffff:172.20.0.103]:3901	UNCONFIGURED/REMOVED
+8781c50c410a41b3…	758338dde686	[::ffff:172.20.0.102]:3901	UNCONFIGURED/REMOVED
+```
+
+...which means that you are ready to configure your cluster!
@@ -0,0 +1,222 @@
+# Configure the daemon
+
+Garage is a software that can be run only in a cluster and requires at least 3 instances.
+In our getting started guide, we document two deployment types:
+  - [Test deployment](#test-deployment) though `docker-compose`
+  - [Real-world deployment](#real-world-deployment) through `docker` or `systemd`
+
+In any case, you first need to generate TLS certificates, as traffic is encrypted between Garage's nodes.
+
+## Generating a TLS Certificate
+
+To generate your TLS certificates, run on your machine:
+
+```
+wget https://git.deuxfleurs.fr/Deuxfleurs/garage/raw/branch/master/genkeys.sh
+chmod +x genkeys.sh
+./genkeys.sh
+```
+
+It will creates a folder named `pki` containing the keys that you will used for the cluster.
+
+## Test deployment
+
+Single machine deployment is only described through `docker-compose`.
+
+Before starting, we recommend you create a folder for our deployment:
+
+```bash
+mkdir garage-single
+cd garage-single
+```
+
+We start by creating a file named `docker-compose.yml` describing our network and our containers:
+
+```yml
+version: '3.4'
+
+networks: { virtnet: { ipam: { config: [ subnet: 172.20.0.0/24 ]}}}
+
+services:
+  g1:
+    image: lxpz/garage_amd64:v0.1.1d
+    networks: { virtnet: { ipv4_address: 172.20.0.101 }}
+    volumes:
+      - "./pki:/pki"
+      - "./config.toml:/garage/config.toml"
+
+  g2:
+    image: lxpz/garage_amd64:v0.1.1d
+    networks: { virtnet: { ipv4_address: 172.20.0.102 }}
+    volumes:
+      - "./pki:/pki"
+      - "./config.toml:/garage/config.toml"
+
+  g3:
+    image: lxpz/garage_amd64:v0.1.1d
+    networks: { virtnet: { ipv4_address: 172.20.0.103 }}
+    volumes:
+      - "./pki:/pki"
+      - "./config.toml:/garage/config.toml"
+```
+
+*We define a static network here which is not considered as a best practise on Docker.
+The rational is that Garage only supports IP address and not domain names in its configuration, so we need to know the IP address in advance.*
+
+and then create the `config.toml` file next to it as follow:
+
+```toml
+metadata_dir = "/garage/meta"
+data_dir = "/garage/data"
+rpc_bind_addr = "[::]:3901"
+bootstrap_peers = [
+  "172.20.0.101:3901",
+  "172.20.0.102:3901",
+  "172.20.0.103:3901",
+]
+
+[rpc_tls]
+ca_cert = "/pki/garage-ca.crt"
+node_cert = "/pki/garage.crt"
+node_key = "/pki/garage.key"
+
+[s3_api]
+s3_region = "garage"
+api_bind_addr = "[::]:3900"
+
+[s3_web]
+bind_addr = "[::]:3902"
+root_domain = ".web.garage"
+index = "index.html"
+```
+
+*Please note that we have not mounted `/garage/meta` or `/garage/data` on the host: data will be lost when the container will be destroyed.*
+
+And that's all, you are ready to launch your cluster!
+
+```
+sudo docker-compose up
+```
+
+While your daemons are up, your cluster is still not configured yet.
+However, you can check that your services are still listening as expected by querying them from your host:
+
+```bash
+curl http://172.20.0.{101,102,103}:3902
+```
+
+which should give you:
+
+```
+Not found
+Not found
+Not found
+```
+
+That's all, you are ready to [configure your cluster!](./cluster.md).
+
+## Real-world deployment
+
+Before deploying garage on your infrastructure, you must inventory your machines.
+For our example, we will suppose the following infrastructure:
+
+| Location | Name    | IP Address | Disk Space |
+|----------|---------|------------|------------|
+| Paris    | Mercury | fc00:1::1  | 1 To       |
+| Paris    | Venus   | fc00:1::2  | 2 To       |
+| London   | Earth   | fc00:1::2  | 2 To       |
+| Brussels | Mars    | fc00:B::1  | 1.5 To     |
+
+On each machine, we will have a similar setup, especially you must consider the following folders/files:
+  - `/etc/garage/pki`: Garage certificates, must be generated on your computer and copied on the servers
+  - `/etc/garage/config.toml`: Garage daemon's configuration (defined below)
+  - `/etc/systemd/system/garage.service`: Service file to start garage at boot automatically (defined below, not required if you use docker)
+  - `/var/lib/garage/meta`: Contains Garage's metadata, put this folder on a SSD if possible
+  - `/var/lib/garage/data`: Contains Garage's data, this folder will grows and must be on a large storage, possibly big HDDs.
+
+A valid `/etc/garage/config.toml` for our cluster would be:
+
+```toml
+metadata_dir = "/var/lib/garage/meta"
+data_dir = "/var/lib/garage/data"
+rpc_bind_addr = "[::]:3901"
+bootstrap_peers = [
+  "[fc00:1::1]:3901",
+  "[fc00:1::2]:3901",
+  "[fc00:B::1]:3901",
+  "[fc00:F::1]:3901",
+]
+
+[rpc_tls]
+ca_cert = "/etc/garage/pki/garage-ca.crt"
+node_cert = "/etc/garage/pki/garage.crt"
+node_key = "/etc/garage/pki/garage.key"
+
+[s3_api]
+s3_region = "garage"
+api_bind_addr = "[::]:3900"
+
+[s3_web]
+bind_addr = "[::]:3902"
+root_domain = ".web.garage"
+index = "index.html"
+```
+
+Please make sure to change `bootstrap_peers` to **your** IP addresses!
+
+### For docker users
+
+On each machine, you can run the daemon with:
+
+```bash
+docker run \
+  -d \
+  --name garaged \
+  --restart always \
+  --network host \
+  -v /etc/garage/pki:/etc/garage/pki \
+  -v /etc/garage/config.toml:/garage/config.toml \
+  -v /var/lib/garage/meta:/var/lib/garage/meta \
+  -v /var/lib/garage/data:/var/lib/garage/data \
+  lxpz/garage_amd64:v0.1.1d
+```
+
+It should be restart automatically at each reboot.
+Please note that we use host networking as otherwise Docker containers can no communicate with IPv6.
+
+To upgrade, simply stop and remove this container and start again the command with a new version of garage.
+
+### For systemd/raw binary users
+
+Create a file named `/etc/systemd/system/garage.service`:
+
+```toml
+[Unit]
+Description=Garage Data Store
+After=network-online.target
+Wants=network-online.target
+
+[Service]
+Environment='RUST_LOG=garage=info' 'RUST_BACKTRACE=1'
+ExecStart=/usr/local/bin/garage server -c /etc/garage/config.toml
+
+[Install]
+WantedBy=multi-user.target
+```
+
+To start the service then automatically enable it at boot:
+
+```bash
+sudo systemctl start garage
+sudo systemctl enable garage
+```
+
+To see if the service is running and to browse its logs:
+
+```bash
+sudo systemctl status garage
+sudo journalctl -u garage
+```
+
+If you want to modify the service file, do not forget to run `systemctl daemon-reload`
+to inform `systemd` of your modifications.
@@ -0,0 +1,42 @@
+# Handle files
+
+We recommend the use of MinIO Client to interact with Garage files (`mc`).
+Instructions to install it and use it are provided on the [MinIO website](https://docs.min.io/docs/minio-client-quickstart-guide.html).
+Before reading the following, you need a working `mc` command on your path.
+
+## Configure `mc`
+
+You need your access key and secret key created in the [previous section](bucket.md).
+You also need to set the endpoint: it must match the IP address of one of the node of the cluster and the API port (3900 by default).
+For this whole configuration, you must set an alias name: we chose `my-garage`, that you will used for all commands.
+
+Adapt the following command accordingly and run it:
+
+```bash
+mc alias set \
+  my-garage \
+  http://172.20.0.101:3900 \
+  <access key> \
+  <secret key> \
+  --api S3v4
+```
+
+You must also add an environment variable to your configuration to inform MinIO of our region (`garage` by default).
+The best way is to add the following snippet to your `$HOME/.bash_profile` or `$HOME/.bashrc` file:
+
+```bash
+export MC_REGION=garage
+```
+
+## Use `mc`
+
+You can not list buckets from `mc` currently.
+
+But the following commands and many more should work:
+
+```bash
+mc cp image.png my-garage/nextcloud-bucket
+mc cp my-garage/nextcloud-bucket/image.png .
+mc ls my-garage/nextcloud-bucket
+mc mirror localdir/ my-garage/another-bucket
+```
@@ -0,0 +1,5 @@
+# Getting Started
+
+Let's start your Garage journey!
+In this chapter, we explain how to deploy a simple garage cluster and start interacting with it.
+Our goal is to introduce you to Garage's workflows.
@@ -0,0 +1,44 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<svg width="128" height="128" version="1.1" viewBox="0 0 33.867 33.867" xmlns="http://www.w3.org/2000/svg" xmlns:cc="http://creativecommons.org/ns#" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
+ <metadata>
+  <rdf:RDF>
+   <cc:Work rdf:about="">
+    <dc:format>image/svg+xml</dc:format>
+    <dc:type rdf:resource="http://purl.org/dc/dcmitype/StillImage"/>
+    <dc:title/>
+   </cc:Work>
+  </rdf:RDF>
+ </metadata>
+ <g stroke-width=".14689">
+  <path d="m20.613 10.981a2.2034 2.2034 0 0 1-0.73445-0.07638l-9.2042-2.4839a2.2342 2.2342 0 0 1-0.69332-0.32757z"/>
+  <g fill="#4e4e4e">
+   <path class="cls-1" d="m6.6028 26.612 1.3661-0.0088h0.01763q0.75796 0 0.75796 0.71389v2.3003a6.5748 6.5748 0 0 1-2.2886 0.37898q-1.2515 0-1.8861-0.8505t-0.63457-2.3179q0-1.4689 0.7888-2.2827a2.5823 2.5823 0 0 1 1.9301-0.81524 3.5371 3.5371 0 0 1 2.0667 0.64338 1.0385 1.0385 0 0 1-0.18068 0.46711 1.2603 1.2603 0 0 1-0.33932 0.35254 2.5926 2.5926 0 0 0-1.5027-0.51999 1.4175 1.4175 0 0 0-1.1854 0.54203q-0.42304 0.53909-0.42304 1.6966 0 2.1769 1.604 2.1769a4.4743 4.4743 0 0 0 0.97829-0.11457v-0.83728q0-0.3966 0.01763-0.58756h-0.64633a0.60519 0.60519 0 0 1-0.40101-0.11018 0.44067 0.44067 0 0 1-0.12779-0.35254 1.51 1.51 0 0 1 0.088134-0.47446z"/>
+   <path class="cls-1" d="m13.401 29.379a1.1413 1.1413 0 0 1-0.14689 0.31288 1.0664 1.0664 0 0 1-0.22474 0.25118 0.99592 0.99592 0 0 1-0.80937-0.51705 1.7847 1.7847 0 0 1-1.2603 0.56406q-0.67863 0-1.0282-0.3966a1.3573 1.3573 0 0 1-0.34372-0.9166q0-0.73445 0.48033-1.1149a1.9404 1.9404 0 0 1 1.2354-0.3687q0.40542 0 0.76677 0.03525v-0.2644q0-0.69626-0.66982-0.69626-0.47592 0-1.3485 0.31728a1.2368 1.2368 0 0 1-0.29378-0.78439 4.9164 4.9164 0 0 1 1.9096-0.3966 1.5526 1.5526 0 0 1 1.0752 0.37016q0.41423 0.37016 0.41423 1.1193v1.7979q-0.0029 0.48474 0.24384 0.68745zm-2.2122-0.22034a1.2471 1.2471 0 0 0 0.88134-0.42304v-0.77852a5.9182 5.9182 0 0 0-0.66982-0.03525 0.73445 0.73445 0 0 0-0.54643 0.18214 0.6331 0.6331 0 0 0-0.18508 0.46711 0.62282 0.62282 0 0 0 0.14689 0.44067 0.48768 0.48768 0 0 0 0.3731 0.14689z"/>
+   <path class="cls-1" d="m14.115 26.012a1.0547 1.0547 0 0 1 0.14689-0.32169 0.88134 0.88134 0 0 1 0.22474-0.25118 1.1017 1.1017 0 0 1 0.92982 0.78439q0.35254-0.78439 1.1369-0.78439a2.7028 2.7028 0 0 1 0.51118 0.06169 1.9786 1.9786 0 0 1-0.2644 1.0282 2.2357 2.2357 0 0 0-0.3966-0.05288q-0.53762 0-0.86372 0.57287v2.8174a3.0627 3.0627 0 0 1-0.53762 0.04407 3.3785 3.3785 0 0 1-0.55525-0.04407v-2.9525q-0.0059-0.6375-0.33197-0.90191z"/>
+   <path class="cls-1" d="m21.157 29.379a1.1413 1.1413 0 0 1-0.15423 0.31288 1.0664 1.0664 0 0 1-0.22474 0.25118 0.99592 0.99592 0 0 1-0.8079-0.51705 1.7847 1.7847 0 0 1-1.2603 0.56406q-0.67864 0-1.0282-0.3966a1.3573 1.3573 0 0 1-0.34372-0.9166q0-0.73445 0.48033-1.1149a1.9404 1.9404 0 0 1 1.2295-0.37457q0.40542 0 0.76677 0.03525v-0.2644q0-0.69626-0.66982-0.69626-0.47592 0-1.3485 0.31728a1.2368 1.2368 0 0 1-0.29378-0.7844 4.9164 4.9164 0 0 1 1.9096-0.3966 1.5526 1.5526 0 0 1 1.0752 0.37016q0.41423 0.37016 0.41423 1.1193v1.8038q0.0088 0.48474 0.25559 0.68745zm-2.2151-0.22034a1.2471 1.2471 0 0 0 0.88134-0.42304v-0.77852a5.9182 5.9182 0 0 0-0.66982-0.03525 0.73445 0.73445 0 0 0-0.54643 0.18508 0.6331 0.6331 0 0 0-0.18508 0.46711 0.62282 0.62282 0 0 0 0.14689 0.44067 0.48768 0.48768 0 0 0 0.3731 0.14395z"/>
+   <path class="cls-1" d="m22.241 29.344q-0.3966-0.60813-0.3966-1.679t0.50236-1.679a1.5188 1.5188 0 0 1 1.2074-0.60813 1.7039 1.7039 0 0 1 1.1898 0.44067 0.99739 0.99739 0 0 1 0.69626-0.37898 0.82552 0.82552 0 0 1 0.23356 0.24677 1.0282 1.0282 0 0 1 0.14689 0.30847q-0.24678 0.21152-0.24678 0.75796v2.4971q0 1.4013-0.4583 1.983-0.4583 0.58169-1.5071 0.58756a4.2598 4.2598 0 0 1-1.5776-0.29378 1.1854 1.1854 0 0 1 0.27322-0.80202 2.882 2.882 0 0 0 1.1854 0.27322q0.57728 0 0.79761-0.29378a1.322 1.322 0 0 0 0.22034-0.81084v-0.35254a1.6936 1.6936 0 0 1-1.1017 0.41423 1.3014 1.3014 0 0 1-1.1648-0.61106zm2.2651-0.71389v-2.0447a1.1355 1.1355 0 0 0-0.75796-0.36135 0.63604 0.63604 0 0 0-0.57728 0.37898 2.2988 2.2988 0 0 0-0.20712 1.0841q0 0.70508 0.18949 1.04a0.56406 0.56406 0 0 0 0.49796 0.33491 1.1193 1.1193 0 0 0 0.8549-0.43186z"/>
+   <path class="cls-1" d="m30.105 28.039h-2.4678a1.4924 1.4924 0 0 0 0.23356 0.80643q0.20712 0.28644 0.72711 0.28644a2.6778 2.6778 0 0 0 1.1546-0.30847 1.159 1.159 0 0 1 0.31728 0.66982 2.8467 2.8467 0 0 1-1.6966 0.50236q-0.99151 0-1.4234-0.64338-0.43186-0.64338-0.43186-1.6657 0-1.0282 0.47592-1.6657a1.5923 1.5923 0 0 1 1.3617-0.64338q0.88134 0 1.3617 0.53321a1.9434 1.9434 0 0 1 0.47593 1.344 3.4519 3.4519 0 0 1-0.08813 0.7844zm-1.701-1.8684q-0.7227 0-0.77558 1.0929h1.5335v-0.10576a1.25 1.25 0 0 0-0.18508-0.71389 0.64338 0.64338 0 0 0-0.567-0.27321z"/>
+  </g>
+  <path d="m17.034 3.0341a2.9114 2.9114 0 0 0-1.1462 0.24753l-11.697 5.1749a0.42304 0.42304 0 0 0-0.22169 0.56586 0.20418 0.20418 0 0 0 0.01757 0.04702l1.8769 3.7099h1.6288l-0.23151-1.2935c-0.0191-0.10429-0.18819-0.84337-0.3483-1.3751l5.4746 1.71c0.07196 0.34089 0.16746 0.65935 0.28112 0.9586h8.8765c0.0978-0.29932 0.17499-0.61834 0.22738-0.9586l5.4627-1.7053c-0.16011 0.53174-0.32713 1.2662-0.34623 1.3705l-0.23151 1.2935h1.6283l1.8593-3.6763 0.01757-0.03359 0.0181-0.04547a0.027909 0.027909 0 0 0 0-0.01188 0.39367 0.39367 0 0 0 0.01757-0.13643 0.41864 0.41864 0 0 0-0.26303-0.4191l-11.697-5.1749a2.9114 2.9114 0 0 0-1.2041-0.24753z" fill="#ffd952"/>
+  <path d="m17.034 5.4825a2.9114 2.9114 0 0 0-1.1462 0.24753l-11.697 5.1749a0.42304 0.42304 0 0 0-0.22169 0.56534 0.20418 0.20418 0 0 0 0.01757 0.04703l1.018 2.0118h2.1632c-0.068234-0.28802-0.15662-0.64282-0.25528-0.97049l3.1073 0.97048h14.121l3.0939-0.96583c-0.09841 0.32682-0.18541 0.67924-0.25321 0.96583h2.1627l1.0005-1.9782 0.01757-0.03359 0.0181-0.04547a0.027909 0.027909 0 0 0 0-0.01188 0.39367 0.39367 0 0 0 0.01757-0.13643 0.41864 0.41864 0 0 0-0.26303-0.41858l-11.697-5.1749a2.9114 2.9114 0 0 0-1.2041-0.24753z" fill="#49c8fa"/>
+  <path class="cls-2" d="m30.198 13.82a0.39367 0.39367 0 0 1-0.01762 0.13661 0.027909 0.027909 0 0 1 0 0.01175l-0.01762 0.04554-0.01762 0.03379-2.8306 5.5965c-0.39367 0.77705-1.1178 0.75355-0.99592-0.03232l0.56993-3.1817c0.0191-0.10429 0.18655-0.83874 0.34666-1.3705l-5.4629 1.7054c-0.85784 5.5716-8.1891 5.6641-9.3848 0l-5.4746-1.7098c0.16011 0.53174 0.32904 1.2706 0.34813 1.3749l0.56994 3.1816c0.12192 0.78586-0.60225 0.80937-0.99592 0.03232l-2.8482-5.6303a0.20418 0.20418 0 0 1-0.01763-0.04701 0.42304 0.42304 0 0 1 0.2218-0.56553l11.697-5.175a2.9114 2.9114 0 0 1 2.3502 0l11.697 5.175a0.41864 0.41864 0 0 1 0.26294 0.41864z" fill="#ffd952"/>
+  <path class="cls-3" d="m20.801 14.796 5.0574-2.0359a0.21446 0.21446 0 0 0 0-0.39807c-0.58756-0.24531-1.3132-0.52734-2.0242-0.82259-0.13073-0.05435-1.369 0.83434-1.4821 0.92541l-2.1799 1.7421c-0.52734 0.44214-0.07051 0.86959 0.62869 0.58903z" fill="#45c8ff"/>
+  <circle class="cls-3" cx="17.135" cy="16.785" r="2.6367" fill="#45c8ff"/>
+  <path d="m20.613 10.981a2.2034 2.2034 0 0 1-0.73445-0.07638l-9.2042-2.4839a2.2342 2.2342 0 0 1-0.69332-0.32757z"/>
+  <g fill="#4e4e4e">
+   <path class="cls-1" d="m6.6028 26.612 1.3661-0.0088h0.01763q0.75796 0 0.75796 0.71389v2.3003a6.5748 6.5748 0 0 1-2.2886 0.37898q-1.2515 0-1.8861-0.8505t-0.63457-2.3179q0-1.4689 0.7888-2.2827a2.5823 2.5823 0 0 1 1.9301-0.81524 3.5371 3.5371 0 0 1 2.0667 0.64338 1.0385 1.0385 0 0 1-0.18068 0.46711 1.2603 1.2603 0 0 1-0.33932 0.35254 2.5926 2.5926 0 0 0-1.5027-0.51999 1.4175 1.4175 0 0 0-1.1854 0.54203q-0.42304 0.53909-0.42304 1.6966 0 2.1769 1.604 2.1769a4.4743 4.4743 0 0 0 0.97829-0.11457v-0.83728q0-0.3966 0.01763-0.58756h-0.64633a0.60519 0.60519 0 0 1-0.40101-0.11018 0.44067 0.44067 0 0 1-0.12779-0.35254 1.51 1.51 0 0 1 0.088134-0.47446z"/>
+   <path class="cls-1" d="m13.401 29.379a1.1413 1.1413 0 0 1-0.14689 0.31288 1.0664 1.0664 0 0 1-0.22474 0.25118 0.99592 0.99592 0 0 1-0.80937-0.51705 1.7847 1.7847 0 0 1-1.2603 0.56406q-0.67863 0-1.0282-0.3966a1.3573 1.3573 0 0 1-0.34372-0.9166q0-0.73445 0.48033-1.1149a1.9404 1.9404 0 0 1 1.2354-0.3687q0.40542 0 0.76677 0.03525v-0.2644q0-0.69626-0.66982-0.69626-0.47592 0-1.3485 0.31728a1.2368 1.2368 0 0 1-0.29378-0.78439 4.9164 4.9164 0 0 1 1.9096-0.3966 1.5526 1.5526 0 0 1 1.0752 0.37016q0.41423 0.37016 0.41423 1.1193v1.7979q-0.0029 0.48474 0.24384 0.68745zm-2.2122-0.22034a1.2471 1.2471 0 0 0 0.88134-0.42304v-0.77852a5.9182 5.9182 0 0 0-0.66982-0.03525 0.73445 0.73445 0 0 0-0.54643 0.18214 0.6331 0.6331 0 0 0-0.18508 0.46711 0.62282 0.62282 0 0 0 0.14689 0.44067 0.48768 0.48768 0 0 0 0.3731 0.14689z"/>
+   <path class="cls-1" d="m14.115 26.012a1.0547 1.0547 0 0 1 0.14689-0.32169 0.88134 0.88134 0 0 1 0.22474-0.25118 1.1017 1.1017 0 0 1 0.92982 0.78439q0.35254-0.78439 1.1369-0.78439a2.7028 2.7028 0 0 1 0.51118 0.06169 1.9786 1.9786 0 0 1-0.2644 1.0282 2.2357 2.2357 0 0 0-0.3966-0.05288q-0.53762 0-0.86372 0.57287v2.8174a3.0627 3.0627 0 0 1-0.53762 0.04407 3.3785 3.3785 0 0 1-0.55525-0.04407v-2.9525q-0.0059-0.6375-0.33197-0.90191z"/>
+   <path class="cls-1" d="m21.157 29.379a1.1413 1.1413 0 0 1-0.15423 0.31288 1.0664 1.0664 0 0 1-0.22474 0.25118 0.99592 0.99592 0 0 1-0.8079-0.51705 1.7847 1.7847 0 0 1-1.2603 0.56406q-0.67864 0-1.0282-0.3966a1.3573 1.3573 0 0 1-0.34372-0.9166q0-0.73445 0.48033-1.1149a1.9404 1.9404 0 0 1 1.2295-0.37457q0.40542 0 0.76677 0.03525v-0.2644q0-0.69626-0.66982-0.69626-0.47592 0-1.3485 0.31728a1.2368 1.2368 0 0 1-0.29378-0.7844 4.9164 4.9164 0 0 1 1.9096-0.3966 1.5526 1.5526 0 0 1 1.0752 0.37016q0.41423 0.37016 0.41423 1.1193v1.8038q0.0088 0.48474 0.25559 0.68745zm-2.2151-0.22034a1.2471 1.2471 0 0 0 0.88134-0.42304v-0.77852a5.9182 5.9182 0 0 0-0.66982-0.03525 0.73445 0.73445 0 0 0-0.54643 0.18508 0.6331 0.6331 0 0 0-0.18508 0.46711 0.62282 0.62282 0 0 0 0.14689 0.44067 0.48768 0.48768 0 0 0 0.3731 0.14395z"/>
+   <path class="cls-1" d="m22.241 29.344q-0.3966-0.60813-0.3966-1.679t0.50236-1.679a1.5188 1.5188 0 0 1 1.2074-0.60813 1.7039 1.7039 0 0 1 1.1898 0.44067 0.99739 0.99739 0 0 1 0.69626-0.37898 0.82552 0.82552 0 0 1 0.23356 0.24677 1.0282 1.0282 0 0 1 0.14689 0.30847q-0.24678 0.21152-0.24678 0.75796v2.4971q0 1.4013-0.4583 1.983-0.4583 0.58169-1.5071 0.58756a4.2598 4.2598 0 0 1-1.5776-0.29378 1.1854 1.1854 0 0 1 0.27322-0.80202 2.882 2.882 0 0 0 1.1854 0.27322q0.57728 0 0.79761-0.29378a1.322 1.322 0 0 0 0.22034-0.81084v-0.35254a1.6936 1.6936 0 0 1-1.1017 0.41423 1.3014 1.3014 0 0 1-1.1648-0.61106zm2.2651-0.71389v-2.0447a1.1355 1.1355 0 0 0-0.75796-0.36135 0.63604 0.63604 0 0 0-0.57728 0.37898 2.2988 2.2988 0 0 0-0.20712 1.0841q0 0.70508 0.18949 1.04a0.56406 0.56406 0 0 0 0.49796 0.33491 1.1193 1.1193 0 0 0 0.8549-0.43186z"/>
+   <path class="cls-1" d="m30.105 28.039h-2.4678a1.4924 1.4924 0 0 0 0.23356 0.80643q0.20712 0.28644 0.72711 0.28644a2.6778 2.6778 0 0 0 1.1546-0.30847 1.159 1.159 0 0 1 0.31728 0.66982 2.8467 2.8467 0 0 1-1.6966 0.50236q-0.99151 0-1.4234-0.64338-0.43186-0.64338-0.43186-1.6657 0-1.0282 0.47592-1.6657a1.5923 1.5923 0 0 1 1.3617-0.64338q0.88134 0 1.3617 0.53321a1.9434 1.9434 0 0 1 0.47593 1.344 3.4519 3.4519 0 0 1-0.08813 0.7844zm-1.701-1.8684q-0.7227 0-0.77558 1.0929h1.5335v-0.10576a1.25 1.25 0 0 0-0.18508-0.71389 0.64338 0.64338 0 0 0-0.567-0.27321z"/>
+  </g>
+  <g>
+   <path d="m17.034 3.0341a2.9114 2.9114 0 0 0-1.1462 0.24753l-11.697 5.1749a0.42304 0.42304 0 0 0-0.22169 0.56586 0.20418 0.20418 0 0 0 0.01757 0.04702l1.8769 3.7099h1.6288l-0.23151-1.2935c-0.0191-0.10429-0.18819-0.84337-0.3483-1.3751l5.4746 1.71c0.07196 0.34089 0.16746 0.65935 0.28112 0.9586h8.8765c0.0978-0.29932 0.17499-0.61834 0.22738-0.9586l5.4627-1.7053c-0.16011 0.53174-0.32713 1.2662-0.34623 1.3705l-0.23151 1.2935h1.6283l1.8593-3.6763 0.01757-0.03359 0.0181-0.04547a0.027909 0.027909 0 0 0 0-0.01188 0.39367 0.39367 0 0 0 0.01757-0.13643 0.41864 0.41864 0 0 0-0.26303-0.4191l-11.697-5.1749a2.9114 2.9114 0 0 0-1.2041-0.24753z" fill="#ff9329"/>
+   <path d="m17.034 5.4825a2.9114 2.9114 0 0 0-1.1462 0.24753l-11.697 5.1749a0.42304 0.42304 0 0 0-0.22169 0.56534 0.20418 0.20418 0 0 0 0.01757 0.04703l1.018 2.0118h2.1632c-0.068234-0.28802-0.15662-0.64282-0.25528-0.97049l3.1073 0.97048h14.121l3.0939-0.96583c-0.09841 0.32682-0.18541 0.67924-0.25321 0.96583h2.1627l1.0005-1.9782 0.01757-0.03359 0.0181-0.04547a0.027909 0.027909 0 0 0 0-0.01188 0.39367 0.39367 0 0 0 0.01757-0.13643 0.41864 0.41864 0 0 0-0.26303-0.41858l-11.697-5.1749a2.9114 2.9114 0 0 0-1.2041-0.24753z" fill="#4e4e4e"/>
+   <path class="cls-2" d="m30.198 13.82a0.39367 0.39367 0 0 1-0.01762 0.13661 0.027909 0.027909 0 0 1 0 0.01175l-0.01762 0.04554-0.01762 0.03379-2.8306 5.5965c-0.39367 0.77705-1.1178 0.75355-0.99592-0.03232l0.56993-3.1817c0.0191-0.10429 0.18655-0.83874 0.34666-1.3705l-5.4629 1.7054c-0.85784 5.5716-8.1891 5.6641-9.3848 0l-5.4746-1.7098c0.16011 0.53174 0.32904 1.2706 0.34813 1.3749l0.56994 3.1816c0.12192 0.78586-0.60225 0.80937-0.99592 0.03232l-2.8482-5.6303a0.20418 0.20418 0 0 1-0.01763-0.04701 0.42304 0.42304 0 0 1 0.2218-0.56553l11.697-5.175a2.9114 2.9114 0 0 1 2.3502 0l11.697 5.175a0.41864 0.41864 0 0 1 0.26294 0.41864z" fill="#ff9329"/>
+   <path class="cls-3" d="m20.801 14.796 5.0574-2.0359a0.21446 0.21446 0 0 0 0-0.39807c-0.58756-0.24531-1.3132-0.52734-2.0242-0.82259-0.13073-0.05435-1.369 0.83434-1.4821 0.92541l-2.1799 1.7421c-0.52734 0.44214-0.07051 0.86959 0.62869 0.58903z" fill="#4e4e4e"/>
+   <circle class="cls-3" cx="17.135" cy="16.785" r="2.6367" fill="#4e4e4e"/>
+  </g>
+ </g>
+</svg>
@@ -0,0 +1,95 @@
+![Garage's Logo](img/logo.svg)
+
+# The Garage Geo-Distributed Data Store
+
+Garage is a lightweight geo-distributed data store.
+It comes from the observation that despite numerous object stores
+many people have broken data management policies (backup/replication on a single site or none at all).
+To promote better data management policies, with focused on the following desirable properties:
+
+  - **Self-contained & lightweight**: works everywhere and integrates well in existing environments to target hyperconverged infrastructures
+  - **Highly resilient**: highly resilient to network failures, network latency, disk failures, sysadmin failures
+  - **Simple**: simple to understand, simple to operate, simple to debug
+  - **Internet enabled**: made for multi-sites (eg. datacenter, offices, etc.) interconnected through a regular internet connection.
+
+We also noted that the pursuit of some other goals are detrimental to our initial goals.
+The following have been identified has non-goals, if it matters to you, you should not use Garage:
+
+  - **Extreme performances**: high performances constrain a lot the design and the infrastructure; we seek performances through minimalism only.
+  - **Feature extensiveness**: complete implementation of the S3 API or any other API to make garage a drop-in replacement is not targeted as it could lead to decisions impacting our desirable properties.
+  - **Storage optimizations**: erasure coding or any other coding technique both increase the difficulty of placing data and synchronizing; we limit ourselves to duplication.
+  - **POSIX/Filesystem compatibility**: we do not aim at being POSIX compatible or to emulate any kind of filesystem. Indeed, in a distributed environment, such syncronizations are translated in network messages that impose severe constraints on the deployment.
+
+## Supported and planned protocols
+
+Garage speaks (or will speak) the following protocols:
+
+  - [S3](https://docs.aws.amazon.com/AmazonS3/latest/API/Welcome.html) - *SUPPORTED* - Enable applications to store large blobs such as pictures, video, images, documents, etc. S3 is versatile enough to also be used to publish a static website.
+  - [IMAP](https://github.com/go-pluto/pluto) - *PLANNED* - email storage is quite complex to get good oerformances.
+To keep performances optimals, most imap servers only support on-disk storage.
+We plan to add logic to Garage to make it a viable solution for email storage.
+  - *More to come*
+
+## Use Cases
+
+**[Deuxfleurs](https://deuxfleurs.fr) :** Garage is used by Deuxfleurs which is a non-profit hosting organization.
+Especially, it is used to host their main website, this documentation and some of its members's blogs. Additionally,
+Garage is used as a [backend for Nextcloud](https://docs.nextcloud.com/server/20/admin_manual/configuration_files/primary_storage.html). Deuxfleurs also plans to use Garage as their [Matrix's media backend](https://github.com/matrix-org/synapse-s3-storage-provider) and has the backend of [OCIS](https://github.com/owncloud/ocis).
+
+*Are you using Garage? [Open a pull request](https://git.deuxfleurs.fr/Deuxfleurs/garage/) to add your organization here!*
+
+## Comparison to existing software
+
+**[Minio](https://min.io/) :** Minio shares our *self-contained & lightweight* goal but selected two of our non-goals: *storage optimizations* through erasure coding and *POSIX/Filesystem compatibility* through strong consistency.
+However, by pursuing these two non-goals, minio do not reach our desirable properties.
+First, it fails on the *simple* property: due to the erasure coding, minio has severe limitations on how drives can be added or deleted from a cluster.
+Second, it fails on the *interned enabled* property: due to its strong consistency, minio is latency sensitive.
+Furthermore, minio has no knowledge of "sites" and thus can not distribute data to minimize the failure of a given site.
+
+**[Openstack Swift](https://docs.openstack.org/swift/latest/) :**
+OpenStack Swift at least fails on the *self-contained & lightweight* goal.
+Starting it requires around 8Gb of RAM, which is too much especially in an hyperconverged infrastructure.
+It seems also to be far from *Simple*.
+
+**[Ceph](https://ceph.io/ceph-storage/object-storage/) :**
+This review holds for the whole Ceph stack, including the RADOS paper, Ceph Object Storage module, the RADOS Gateway, etc.
+At is core, Ceph has been designed to provide *POSIX/Filesystem compatibility* which requires strong consistency, which in turn
+makes Ceph latency sensitive and fails our *Internet enabled* goal. 
+Due to its industry oriented design, Ceph is also far from being *Simple* to operate and from being *self-contained & lightweight* which makes it hard to integrate it in an hyperconverged infrastructure.
+In a certain way, Ceph and Minio are closer togethers than they are from Garage or OpenStack Swift.
+
+*More comparisons are available in our [Related Work](design/related_work.md) chapter.*
+
+## Other Resources
+
+This website is not the only source of information about Garage!
+We reference here other places on the Internet where you can learn more about Garage.
+
+### Rust API (docs.rs)
+
+If you encounter a specific bug in Garage or plan to patch it, you may jump directly to the source code documentation!
+
+  - [garage\_api](https://docs.rs/garage_api/latest/garage_api/) - contains the S3 standard API endpoint
+  - [garage\_model](https://docs.rs/garage_model/latest/garage_model/) - contains Garage's model built on the table abstraction
+  - [garage\_rpc](https://docs.rs/garage_rpc/latest/garage_rpc/) - contains Garage's federation protocol
+  - [garage\_table](https://docs.rs/garage_table/latest/garage_table/) - contains core Garage's CRDT datatypes
+  - [garage\_util](https://docs.rs/garage_util/latest/garage_util/) - contains garage entrypoints (daemon, cli)
+  - [garage\_web](https://docs.rs/garage_web/latest/garage_web/) - contains the S3 website endpoint
+
+### Talks
+
+We love to talk and hear about Garage, that's why we keep a log here:
+
+  - [(fr, 2020-12-02) Garage : jouer dans la cour des grands quand on est un hébergeur associatif](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/master/doc/20201202_talk/talk.pdf)
+
+*Did you write or talk about Garage? [Open a pull request](https://git.deuxfleurs.fr/Deuxfleurs/garage/) to add a link here!*
+
+## Community
+
+If you want to discuss with us, you can join our Matrix channel at [#garage:deuxfleurs.fr](https://matrix.to/#/#garage:deuxfleurs.fr).
+Our code and our issue tracker, which is the place where you should report bugs, are managed on [Deuxfleurs' Gitea](https://git.deuxfleurs.fr/Deuxfleurs/garage).
+
+## License
+
+Garage, all the source code, is released under the [AGPL v3 License](https://www.gnu.org/licenses/agpl-3.0.en.html).
+Please note that if you patch Garage and then use it to provide any service over a network, you must share your code!
@@ -0,0 +1,5 @@
+# Reference Manual
+
+A reference manual contains some extensive descriptions about the features and the behaviour of the software.
+Reading of this chapter is recommended once you have a good knowledge/understanding of Garage.
+It will be useful if you want to tune it or to use it in some exotic conditions.
@@ -0,0 +1,84 @@
+## S3 Compatibility status
+
+### Global S3 features
+
+Implemented:
+
+- path-style URLs (`garage.tld/bucket/key`)
+- putting and getting objects in buckets
+- multipart uploads
+- listing objects
+- access control on a per-key-per-bucket basis
+
+Not implemented:
+
+- vhost-style URLs (`bucket.garage.tld/key`)
+- object-level ACL
+- encryption
+- most `x-amz-` headers
+
+
+### Endpoint implementation
+
+All APIs that are not mentionned are not implemented and will return a 400 bad request.
+
+#### AbortMultipartUpload
+
+Implemented.
+
+#### CompleteMultipartUpload
+
+Implemented badly. Garage will not check that all the parts stored correspond to the list given by the client in the request body. This means that the multipart upload might be completed with an invalid size. This is a bug and will be fixed.
+
+#### CopyObject
+
+Implemented.
+
+#### CreateBucket
+
+Garage does not accept creating buckets or giving access using API calls, it has to be done using the CLI tools. CreateBucket will return a 200 if the bucket exists and user has write access, and a 403 Forbidden in all other cases.
+
+#### CreateMultipartUpload
+
+Implemented.
+
+#### DeleteBucket
+
+Garage does not accept deleting buckets using API calls, it has to be done using the CLI tools. This request will return a 403 Forbidden.
+
+#### DeleteObject
+
+Implemented.
+
+#### DeleteObjects
+
+Implemented.
+
+#### GetObject
+
+Implemented.
+
+#### HeadBucket
+
+Implemented.
+
+#### HeadObject
+
+Implemented.
+
+#### ListObjects
+
+Implemented, but there isn't a very good specification of what `encoding-type=url` covers so there might be some encoding bugs. In our implementation the url-encoded fields are in the same in ListObjects as they are in ListObjectsV2.
+
+#### ListObjectsV2
+
+Implemented.
+
+#### PutObject
+
+Implemented.
+
+#### UploadPart
+
+Implemented.
+
@@ -0,0 +1,8 @@
+# Working Documents
+
+Working documents are documents that reflect the fact that Garage is a software that evolves quickly.
+They are a way to communicate our ideas, our changes, and so on before or while we are implementing them in Garage.
+If you like to live on the edge, it could also serve as a documentation of our next features to be released.
+
+Ideally, once the feature/patch has been merged, the working document should serve as a source to 
+update the rest of the documentation and then be removed.
@@ -0,0 +1,197 @@
+## Load Balancing Data (planned for version 0.2)
+
+I have conducted a quick study of different methods to load-balance data over different Garage nodes using consistent hashing.
+
+### Requirements
+
+- *good balancing*: two nodes that have the same announced capacity should receive close to the same number of items
+
+- *multi-datacenter*: the replicas of a partition should be distributed over as many datacenters as possible
+
+- *minimal disruption*: when adding or removing a node, as few partitions as possible should have to move around
+
+- *order-agnostic*: the same set of nodes (each associated with a datacenter name
+  and a capacity) should always return the same distribution of partition
+  replicas, independently of the order in which nodes were added/removed (this
+  is to keep the implementation simple)
+
+### Methods
+
+#### Naive multi-DC ring walking strategy
+
+This strategy can be used with any ring-like algorithm to make it aware of the *multi-datacenter* requirement:
+
+In this method, the ring is a list of positions, each associated with a single node in the cluster.
+Partitions contain all the keys between two consecutive items of the ring.
+To find the nodes that store replicas of a given partition:
+
+- select the node for the position of the partition's lower bound
+- go clockwise on the ring, skipping nodes that:
+  - we halve already selected
+  - are in a datacenter of a node we have selected, except if we already have nodes from all possible datacenters
+
+In this way the selected nodes will always be distributed over
+`min(n_datacenters, n_replicas)` different datacenters, which is the best we
+can do.
+
+This method was implemented in the first version of Garage, with the basic
+ring construction from Dynamo DB that consists in associating `n_token` random positions to
+each node (I know it's not optimal, the Dynamo paper already studies this).
+
+#### Better rings
+
+The ring construction that selects `n_token` random positions for each nodes gives a ring of positions that
+is not well-balanced: the space between the tokens varies a lot, and some partitions are thus bigger than others.
+This problem was demonstrated in the original Dynamo DB paper.
+
+To solve this, we want to apply a better second method for partitionning our dataset:
+
+1. fix an initially large number of partitions (say 1024) with evenly-spaced delimiters, 
+
+2. attribute each partition randomly to a node, with a probability
+   proportionnal to its capacity (which `n_tokens` represented in the first
+   method)
+
+For now we continue using the multi-DC ring walking described above.
+
+I have studied two ways to do the attribution of partitions to nodes, in a way that is deterministic:
+
+- Min-hash: for each partition, select node that minimizes `hash(node, partition_number)`
+- MagLev: see [here](https://blog.acolyer.org/2016/03/21/maglev-a-fast-and-reliable-software-network-load-balancer/)
+
+MagLev provided significantly better balancing, as it guarantees that the exact
+same number of partitions is attributed to all nodes that have the same
+capacity (and that this number is proportionnal to the node's capacity, except
+for large values), however in both cases:
+
+- the distribution is still bad, because we use the naive multi-DC ring walking
+  that behaves strangely due to interactions between consecutive positions on
+  the ring
+
+- the disruption in case of adding/removing a node is not as low as it can be,
+  as we show with the following method.
+
+A quick description of MagLev (backend = node, lookup table = ring):
+
+> The basic idea of Maglev hashing is to assign a preference list of all the
+> lookup table positions to each backend. Then all the backends take turns
+> filling their most-preferred table positions that are still empty, until the
+> lookup table is completely filled in. Hence, Maglev hashing gives an almost
+> equal share of the lookup table to each of the backends. Heterogeneous
+> backend weights can be achieved by altering the relative frequency of the
+> backends’ turns…
+
+Here are some stats (run `scripts/simulate_ring.py` to reproduce):
+
+```
+##### Custom-ring (min-hash) #####
+
+#partitions per node (capacity in parenthesis):
+- datura (8) :  227
+- digitale (8) :  351
+- drosera (8) :  259
+- geant (16) :  476
+- gipsie (16) :  410
+- io (16) :  495
+- isou (8) :  231
+- mini (4) :  149
+- mixi (4) :  188
+- modi (4) :  127
+- moxi (4) :  159
+
+Variance of load distribution for load normalized to intra-class mean
+(a class being the set of nodes with the same announced capacity): 2.18%     <-- REALLY BAD
+
+Disruption when removing nodes (partitions moved on 0/1/2/3 nodes):
+removing atuin digitale : 63.09% 30.18% 6.64% 0.10%
+removing atuin drosera : 72.36% 23.44% 4.10% 0.10%
+removing atuin datura : 73.24% 21.48% 5.18% 0.10%
+removing jupiter io : 48.34% 38.48% 12.30% 0.88%
+removing jupiter isou : 74.12% 19.73% 6.05% 0.10%
+removing grog mini : 84.47% 12.40% 2.93% 0.20%
+removing grog mixi : 80.76% 16.60% 2.64% 0.00%
+removing grog moxi : 83.59% 14.06% 2.34% 0.00%
+removing grog modi : 87.01% 11.43% 1.46% 0.10%
+removing grisou geant : 48.24% 37.40% 13.67% 0.68%
+removing grisou gipsie : 53.03% 33.59% 13.09% 0.29%
+on average:  69.84% 23.53% 6.40% 0.23%                  <-- COULD BE BETTER
+
+--------
+
+##### MagLev #####
+
+#partitions per node:
+- datura (8) :  273
+- digitale (8) :  256
+- drosera (8) :  267
+- geant (16) :  452
+- gipsie (16) :  427
+- io (16) :  483
+- isou (8) :  272
+- mini (4) :  184
+- mixi (4) :  160
+- modi (4) :  144
+- moxi (4) :  154
+
+Variance of load distribution: 0.37%                <-- Already much better, but not optimal
+
+Disruption when removing nodes (partitions moved on 0/1/2/3 nodes):
+removing atuin digitale : 62.60% 29.20% 7.91% 0.29%
+removing atuin drosera : 65.92% 26.56% 7.23% 0.29%
+removing atuin datura : 63.96% 27.83% 7.71% 0.49%
+removing jupiter io : 44.63% 40.33% 14.06% 0.98%
+removing jupiter isou : 63.38% 27.25% 8.98% 0.39%
+removing grog mini : 72.46% 21.00% 6.35% 0.20%
+removing grog mixi : 72.95% 22.46% 4.39% 0.20%
+removing grog moxi : 74.22% 20.61% 4.98% 0.20%
+removing grog modi : 75.98% 18.36% 5.27% 0.39%
+removing grisou geant : 46.97% 36.62% 15.04% 1.37%
+removing grisou gipsie : 49.22% 36.52% 12.79% 1.46%
+on average:  62.94% 27.89% 8.61% 0.57%                  <-- WORSE THAN PREVIOUSLY
+```
+
+#### The magical solution: multi-DC aware MagLev
+
+Suppose we want to select three replicas for each partition (this is what we do in our simulation and in most Garage deployments).
+We apply MagLev three times consecutively, one for each replica selection.
+The first time is pretty much the same as normal MagLev, but for the following times, when a node runs through its preference
+list to select a partition to replicate, we skip partitions for which adding this node would not bring datacenter-diversity.
+More precisely, we skip a partition in the preference list if:
+
+- the node already replicates the partition (from one of the previous rounds of MagLev)
+- the node is in a datacenter where a node already replicates the partition and there are other datacenters available
+
+Refer to `method4` in the simulation script for a formal definition.
+
+```
+##### Multi-DC aware MagLev #####
+
+#partitions per node:
+- datura (8) :  268                 <-- NODES WITH THE SAME CAPACITY
+- digitale (8) :  267                   HAVE THE SAME NUM OF PARTITIONS
+- drosera (8) :  267                    (+- 1)
+- geant (16) :  470
+- gipsie (16) :  472
+- io (16) :  516
+- isou (8) :  268
+- mini (4) :  136
+- mixi (4) :  136
+- modi (4) :  136
+- moxi (4) :  136
+
+Variance of load distribution: 0.06%                <-- CAN'T DO BETTER THAN THIS
+
+Disruption when removing nodes (partitions moved on 0/1/2/3 nodes):
+removing atuin digitale : 65.72% 33.01% 1.27% 0.00%
+removing atuin drosera : 64.65% 33.89% 1.37% 0.10%
+removing atuin datura : 66.11% 32.62% 1.27% 0.00%
+removing jupiter io : 42.97% 53.42% 3.61% 0.00%
+removing jupiter isou : 66.11% 32.32% 1.56% 0.00%
+removing grog mini : 80.47% 18.85% 0.68% 0.00%
+removing grog mixi : 80.27% 18.85% 0.88% 0.00%
+removing grog moxi : 80.18% 19.04% 0.78% 0.00%
+removing grog modi : 79.69% 19.92% 0.39% 0.00%
+removing grisou geant : 44.63% 52.15% 3.22% 0.00%
+removing grisou gipsie : 43.55% 52.54% 3.91% 0.00%
+on average:  64.94% 33.33% 1.72% 0.01%                  <-- VERY GOOD (VERY LOW VALUES FOR 2 AND 3 NODES)
+```
@@ -0,0 +1,113 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<svg
+   xmlns:dc="http://purl.org/dc/elements/1.1/"
+   xmlns:cc="http://creativecommons.org/ns#"
+   xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
+   xmlns:svg="http://www.w3.org/2000/svg"
+   xmlns="http://www.w3.org/2000/svg"
+   xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
+   xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
+   width="250"
+   height="250"
+   viewBox="0 0 66.145832 66.145831"
+   version="1.1"
+   id="svg916"
+   inkscape:version="1.0.2 (e86c870879, 2021-01-15)"
+   sodipodi:docname="garage-dark-notext.svg"
+   inkscape:export-filename="/home/lx/Deuxfleurs/garage/garage-dark-notext.png"
+   inkscape:export-xdpi="96"
+   inkscape:export-ydpi="96">
+  <defs
+     id="defs910" />
+  <sodipodi:namedview
+     id="base"
+     pagecolor="#ffffff"
+     bordercolor="#666666"
+     borderopacity="1.0"
+     inkscape:pageopacity="0.0"
+     inkscape:pageshadow="2"
+     inkscape:zoom="2.3640695"
+     inkscape:cx="127.28732"
+     inkscape:cy="150.37984"
+     inkscape:document-units="mm"
+     inkscape:current-layer="layer1"
+     inkscape:document-rotation="0"
+     showgrid="false"
+     fit-margin-top="0"
+     fit-margin-left="0"
+     fit-margin-right="0"
+     fit-margin-bottom="0"
+     units="px"
+     inkscape:window-width="1920"
+     inkscape:window-height="1039"
+     inkscape:window-x="0"
+     inkscape:window-y="20"
+     inkscape:window-maximized="0" />
+  <metadata
+     id="metadata913">
+    <rdf:RDF>
+      <cc:Work
+         rdf:about="">
+        <dc:format>image/svg+xml</dc:format>
+        <dc:type
+           rdf:resource="http://purl.org/dc/dcmitype/StillImage" />
+        <dc:title></dc:title>
+      </cc:Work>
+    </rdf:RDF>
+  </metadata>
+  <g
+     inkscape:label="Layer 1"
+     inkscape:groupmode="layer"
+     id="layer1"
+     transform="translate(-141.5009,-98.254059)">
+    <rect
+       style="fill:#4e4e4e;fill-opacity:1;stroke-width:1.01574"
+       id="rect858"
+       width="66.592186"
+       height="66.832306"
+       x="141.5009"
+       y="98.056282" />
+    <g
+       id="g1775"
+       transform="matrix(1.9019239,0,0,1.9019239,-157.45231,-108.13709)">
+      <path
+         class="cls-2"
+         d="m 187.70646,127.72029 a 0.39366647,0.39366647 0 0 1 -0.0176,0.1366 0.02790919,0.02790919 0 0 1 0,0.0117 l -0.0176,0.0455 v 0 l -0.0176,0.0338 -2.83058,5.59653 c -0.39367,0.77705 -1.11784,0.75355 -0.99592,-0.0323 l 0.56994,-3.18164 c 0.0191,-0.1043 0.18655,-0.83875 0.34666,-1.37049 l -5.46286,1.7054 c -0.85784,5.57155 -8.18914,5.66409 -9.38483,0 l -5.47461,-1.70981 c 0.16011,0.53174 0.32904,1.2706 0.34813,1.3749 l 0.56994,3.18164 c 0.12192,0.78587 -0.60225,0.80937 -0.99592,0.0323 l -2.84822,-5.63031 a 0.20417776,0.20417776 0 0 1 -0.0176,-0.047 0.42304456,0.42304456 0 0 1 0.22181,-0.56552 l 11.69689,-5.17495 a 2.9113691,2.9113691 0 0 1 2.35024,0 l 11.69689,5.17495 a 0.41863785,0.41863785 0 0 1 0.26293,0.41864 z"
+         id="path24-31"
+         style="stroke-width:0.146891" />
+      <path
+         class="cls-3"
+         d="m 178.30988,128.69564 5.05744,-2.03591 a 0.21446009,0.21446009 0 0 0 0,-0.39807 c -0.58756,-0.2453 -1.3132,-0.52733 -2.02415,-0.82259 -0.13073,-0.0543 -1.36902,0.83434 -1.48213,0.92542 l -2.17985,1.74212 c -0.52734,0.44214 -0.0705,0.86959 0.62869,0.58903 z"
+         id="path26-9"
+         style="stroke-width:0.146891" />
+      <circle
+         class="cls-3"
+         cx="174.64349"
+         cy="130.68452"
+         r="2.6366842"
+         id="circle28-4"
+         style="stroke-width:0.146891" />
+      <path
+         id="path24-3-6-9-0"
+         style="fill:#ff9329;fill-opacity:1;stroke-width:0.146891"
+         d="m 174.54269,116.93385 a 2.9113691,2.9113691 0 0 0 -1.14618,0.24753 l -11.69696,5.17488 a 0.42304456,0.42304456 0 0 0 -0.22169,0.56586 0.20417776,0.20417776 0 0 0 0.0176,0.047 l 0.79634,1.57355 11.10475,-4.91288 a 2.9113691,2.9113691 0 0 1 1.14618,-0.24753 2.9113691,2.9113691 0 0 1 1.20406,0.24753 l 11.12387,4.92115 0.7829,-1.54823 0.0176,-0.0336 0.0181,-0.0455 a 0.02790919,0.02790919 0 0 0 0,-0.0119 0.39366647,0.39366647 0 0 0 0.0176,-0.13642 0.41863785,0.41863785 0 0 0 -0.26303,-0.4191 l -11.69697,-5.17488 a 2.9113691,2.9113691 0 0 0 -1.20406,-0.24753 z m -10.12134,9.52449 c 0.0218,0.0723 0.0408,0.14674 0.0615,0.22066 h 0.51831 l -0.008,-0.0419 z m 20.32227,0.005 -0.57103,0.17828 -0.007,0.0377 h 0.5178 c 0.0202,-0.0723 0.0386,-0.14514 0.0599,-0.216 z" />
+      <path
+         class="cls-2"
+         d="m 187.70647,127.72029 a 0.39366647,0.39366647 0 0 1 -0.0176,0.13661 0.02790919,0.02790919 0 0 1 0,0.0117 l -0.0176,0.0455 v 0 l -0.0176,0.0338 -2.83058,5.59652 c -0.39366,0.77705 -1.11783,0.75355 -0.99591,-0.0323 l 0.56993,-3.18165 c 0.0191,-0.10429 0.18655,-0.83874 0.34666,-1.37049 l -5.46285,1.7054 c -0.85784,5.57156 -8.18915,5.6641 -9.38484,0 l -5.4746,-1.70981 c 0.16011,0.53175 0.32903,1.27061 0.34813,1.3749 l 0.56993,3.18165 c 0.12192,0.78586 -0.60225,0.80936 -0.99592,0.0323 l -2.84822,-5.63031 a 0.20417776,0.20417776 0 0 1 -0.0176,-0.047 0.42304456,0.42304456 0 0 1 0.22181,-0.56553 l 11.69688,-5.17495 a 2.9113691,2.9113691 0 0 1 2.35025,0 l 11.69689,5.17495 a 0.41863785,0.41863785 0 0 1 0.26293,0.41864 z"
+         id="path24-0-3"
+         style="fill:#ff9329;fill-opacity:1;stroke-width:0.146891" />
+      <path
+         class="cls-3"
+         d="m 178.30988,128.69564 5.05744,-2.0359 a 0.21446009,0.21446009 0 0 0 0,-0.39807 c -0.58756,-0.24531 -1.3132,-0.52734 -2.02415,-0.82259 -0.13073,-0.0543 -1.36902,0.83434 -1.48212,0.92541 l -2.17986,1.74212 c -0.52734,0.44214 -0.0705,0.86959 0.62869,0.58903 z"
+         id="path26-2-2"
+         style="fill:#4e4e4e;fill-opacity:1;stroke-width:0.146891" />
+      <circle
+         class="cls-3"
+         cx="174.64349"
+         cy="130.68452"
+         r="2.6366842"
+         id="circle28-3-0"
+         style="fill:#4e4e4e;fill-opacity:1;stroke-width:0.146891" />
+    </g>
+  </g>
+</svg>
@@ -0,0 +1,174 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<svg
+   xmlns:dc="http://purl.org/dc/elements/1.1/"
+   xmlns:cc="http://creativecommons.org/ns#"
+   xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
+   xmlns:svg="http://www.w3.org/2000/svg"
+   xmlns="http://www.w3.org/2000/svg"
+   xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
+   xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
+   width="250"
+   height="250"
+   viewBox="0 0 66.145832 66.145831"
+   version="1.1"
+   id="svg916"
+   inkscape:version="1.0.2 (e86c870879, 2021-01-15)"
+   sodipodi:docname="garage-dark.svg">
+  <defs
+     id="defs910" />
+  <sodipodi:namedview
+     id="base"
+     pagecolor="#ffffff"
+     bordercolor="#666666"
+     borderopacity="1.0"
+     inkscape:pageopacity="0.0"
+     inkscape:pageshadow="2"
+     inkscape:zoom="2.3640695"
+     inkscape:cx="132.7426"
+     inkscape:cy="151.74366"
+     inkscape:document-units="mm"
+     inkscape:current-layer="layer1"
+     inkscape:document-rotation="0"
+     showgrid="false"
+     fit-margin-top="0"
+     fit-margin-left="0"
+     fit-margin-right="0"
+     fit-margin-bottom="0"
+     units="px"
+     inkscape:window-width="1920"
+     inkscape:window-height="1039"
+     inkscape:window-x="0"
+     inkscape:window-y="20"
+     inkscape:window-maximized="0" />
+  <metadata
+     id="metadata913">
+    <rdf:RDF>
+      <cc:Work
+         rdf:about="">
+        <dc:format>image/svg+xml</dc:format>
+        <dc:type
+           rdf:resource="http://purl.org/dc/dcmitype/StillImage" />
+        <dc:title></dc:title>
+      </cc:Work>
+    </rdf:RDF>
+  </metadata>
+  <g
+     inkscape:label="Layer 1"
+     inkscape:groupmode="layer"
+     id="layer1"
+     transform="translate(-141.5009,-98.254059)">
+    <rect
+       style="fill:#4e4e4e;fill-opacity:1;stroke-width:1.01574"
+       id="rect858"
+       width="66.592186"
+       height="66.832306"
+       x="141.5009"
+       y="98.056282" />
+    <g
+       id="g1637"
+       transform="translate(1.5164686,-0.22143797)">
+      <g
+         id="g1034-5"
+         transform="matrix(0.26458333,0,0,0.26458333,140.0054,98.562655)">
+        <path
+           class="cls-1"
+           d="m 85.377935,159.38378 5.163143,-0.0333 h 0.06662 q 2.864711,0 2.864711,2.69816 v 8.69407 a 24.849705,24.849705 0 0 1 -8.649651,1.43235 q -4.730105,0 -7.128468,-3.21447 -2.398363,-3.21447 -2.398363,-8.76068 0,-5.55177 2.981299,-8.62745 a 9.7600046,9.7600046 0 0 1 7.29502,-3.08123 13.368653,13.368653 0 0 1 7.811335,2.43167 3.9250986,3.9250986 0 0 1 -0.682867,1.76547 4.7634152,4.7634152 0 0 1 -1.282458,1.33242 9.798867,9.798867 0 0 0 -5.679457,-1.96533 5.3574542,5.3574542 0 0 0 -4.480275,2.04861 q -1.598909,2.03749 -1.598909,6.41229 0,8.22771 6.062529,8.22771 a 16.910679,16.910679 0 0 0 3.697476,-0.43303 v -3.16451 q 0,-1.49898 0.06662,-2.22071 h -2.442777 a 2.2873276,2.2873276 0 0 1 -1.515632,-0.41638 1.6655298,1.6655298 0 0 1 -0.483004,-1.33242 5.7072154,5.7072154 0 0 1 0.333106,-1.79322 z"
+           id="path8-2"
+           style="stroke-width:0.555177" />
+        <path
+           class="cls-1"
+           d="m 111.07151,169.8433 a 4.3137222,4.3137222 0 0 1 -0.55518,1.18253 4.0305821,4.0305821 0 0 1 -0.84942,0.94935 3.7640973,3.7640973 0 0 1 -3.05902,-1.95422 6.7453957,6.7453957 0 0 1 -4.76342,2.13188 q -2.564913,0 -3.886233,-1.49898 a 5.1298318,5.1298318 0 0 1 -1.299113,-3.4643 q 0,-2.77588 1.815427,-4.21379 a 7.3338829,7.3338829 0 0 1 4.669039,-1.3935 q 1.53228,0 2.89802,0.13325 v -0.99932 q 0,-2.63154 -2.53161,-2.63154 -1.79877,0 -5.096518,1.19918 a 4.674587,4.674587 0 0 1 -1.110353,-2.96464 18.581761,18.581761 0 0 1 7.217291,-1.49898 5.8682167,5.8682167 0 0 1 4.0639,1.39905 q 1.56559,1.39904 1.56559,4.23044 v 6.79537 q -0.0111,1.83208 0.9216,2.59822 z m -8.36096,-0.83276 a 4.7134493,4.7134493 0 0 0 3.33106,-1.59891 v -2.94244 a 22.368065,22.368065 0 0 0 -2.53161,-0.13324 2.775883,2.775883 0 0 0 -2.06525,0.68842 2.3928111,2.3928111 0 0 0 -0.69953,1.76546 2.3539488,2.3539488 0 0 0 0.55518,1.66553 1.8431863,1.8431863 0 0 0 1.41015,0.55518 z"
+           id="path10-28"
+           style="stroke-width:0.555177" />
+        <path
+           class="cls-1"
+           d="m 113.76966,157.11865 a 3.986168,3.986168 0 0 1 0.55518,-1.21583 3.3310596,3.3310596 0 0 1 0.84942,-0.94935 4.1638245,4.1638245 0 0 1 3.51427,2.96464 q 1.33242,-2.96464 4.29707,-2.96464 a 10.215249,10.215249 0 0 1 1.93201,0.23317 7.4782288,7.4782288 0 0 1 -0.99932,3.88624 8.4497879,8.4497879 0 0 0 -1.49897,-0.19987 q -2.03195,0 -3.26444,2.16519 v 10.64829 a 11.575432,11.575432 0 0 1 -2.03195,0.16655 12.769062,12.769062 0 0 1 -2.09857,-0.16655 v -11.15905 q -0.0222,-2.40947 -1.2547,-3.40879 z"
+           id="path12-9"
+           style="stroke-width:0.555177" />
+        <path
+           class="cls-1"
+           d="m 140.38483,169.8433 a 4.3137222,4.3137222 0 0 1 -0.58293,1.18253 4.0305821,4.0305821 0 0 1 -0.84942,0.94935 3.7640973,3.7640973 0 0 1 -3.05348,-1.95422 6.7453957,6.7453957 0 0 1 -4.76341,2.13188 q -2.56492,0 -3.88624,-1.49898 a 5.1298318,5.1298318 0 0 1 -1.29911,-3.4643 q 0,-2.77588 1.81543,-4.21379 a 7.3338829,7.3338829 0 0 1 4.64682,-1.4157 q 1.53229,0 2.89803,0.13324 v -0.99932 q 0,-2.63153 -2.53161,-2.63153 -1.79877,0 -5.09652,1.19918 a 4.674587,4.674587 0 0 1 -1.11035,-2.96465 18.581761,18.581761 0 0 1 7.21729,-1.49897 5.8682167,5.8682167 0 0 1 4.0639,1.39904 q 1.56559,1.39905 1.56559,4.23045 v 6.81757 q 0.0333,1.83208 0.96601,2.59822 z m -8.37206,-0.83276 a 4.7134493,4.7134493 0 0 0 3.33106,-1.59891 v -2.94244 a 22.368065,22.368065 0 0 0 -2.53161,-0.13324 2.775883,2.775883 0 0 0 -2.06526,0.69952 2.3928111,2.3928111 0 0 0 -0.69952,1.76546 2.3539488,2.3539488 0 0 0 0.55518,1.66553 1.8431863,1.8431863 0 0 0 1.41015,0.54408 z"
+           id="path14-7"
+           style="stroke-width:0.555177" />
+        <path
+           class="cls-1"
+           d="m 144.48203,169.71006 q -1.49897,-2.29843 -1.49897,-6.34567 0,-4.04724 1.8987,-6.34567 a 5.740526,5.740526 0 0 1 4.56355,-2.29843 6.4400486,6.4400486 0 0 1 4.49693,1.66553 3.7696491,3.7696491 0 0 1 2.63154,-1.43235 3.1200925,3.1200925 0 0 1 0.88273,0.93269 3.8862362,3.8862362 0 0 1 0.55518,1.16587 q -0.9327,0.79946 -0.9327,2.86472 v 9.438 q 0,5.29638 -1.73215,7.49488 -1.73215,2.1985 -5.69611,2.22071 a 16.100121,16.100121 0 0 1 -5.9626,-1.11036 4.4802752,4.4802752 0 0 1 1.03263,-3.03126 10.892565,10.892565 0 0 0 4.48028,1.03263 q 2.18184,0 3.0146,-1.11035 a 4.9965894,4.9965894 0 0 0 0.83277,-3.06458 V 170.454 a 6.4011862,6.4011862 0 0 1 -4.16383,1.56559 4.9188647,4.9188647 0 0 1 -4.40255,-2.30953 z m 8.56083,-2.69816 v -7.72806 a 4.2915151,4.2915151 0 0 0 -2.86471,-1.36573 2.4039147,2.4039147 0 0 0 -2.18185,1.43235 8.6885138,8.6885138 0 0 0 -0.7828,4.09721 q 0,2.66485 0.71618,3.93065 a 2.1318781,2.1318781 0 0 0 1.88205,1.2658 4.2304457,4.2304457 0 0 0 3.23113,-1.63222 z"
+           id="path16-3"
+           style="stroke-width:0.555177" />
+        <path
+           class="cls-1"
+           d="m 174.20619,164.78009 h -9.32697 a 5.6405943,5.6405943 0 0 0 0.88273,3.04792 q 0.7828,1.0826 2.74813,1.0826 a 10.120869,10.120869 0 0 0 4.36369,-1.16587 4.3803434,4.3803434 0 0 1 1.19918,2.5316 10.759323,10.759323 0 0 1 -6.41229,1.8987 q -3.74744,0 -5.37966,-2.43167 -1.63222,-2.43167 -1.63222,-6.2957 0,-3.88624 1.79877,-6.2957 a 6.0181143,6.0181143 0 0 1 5.14649,-2.43168 q 3.33106,0 5.14648,2.01529 a 7.3449864,7.3449864 0 0 1 1.79878,5.07987 13.04665,13.04665 0 0 1 -0.33311,2.96464 z m -6.42895,-7.06184 q -2.73146,0 -2.93133,4.13051 h 5.79605 v -0.39973 a 4.7245529,4.7245529 0 0 0 -0.69953,-2.69816 2.4316735,2.4316735 0 0 0 -2.14298,-1.03262 z"
+           id="path18-6"
+           style="stroke-width:0.555177" />
+        <path
+           class="cls-2"
+           d="m 174.55595,111.039 a 1.4878733,1.4878733 0 0 1 -0.0666,0.51631 0.10548355,0.10548355 0 0 1 0,0.0444 l -0.0666,0.17211 v 0 l -0.0666,0.12769 -10.69826,21.15223 c -1.48787,2.93688 -4.22489,2.84806 -3.76409,-0.12214 l 2.15408,-12.02512 c 0.0722,-0.39418 0.70508,-3.17006 1.31022,-5.1798 l -20.64702,6.4456 c -3.24223,21.05785 -30.95109,21.40761 -35.47023,0 l -20.691432,-6.46226 c 0.605143,2.00974 1.243596,4.80228 1.315769,5.19646 l 2.154085,12.02512 c 0.460796,2.9702 -2.276224,3.05902 -3.764098,0.12214 L 75.49024,111.77183 a 0.77169547,0.77169547 0 0 1 -0.06662,-0.17766 1.5989086,1.5989086 0 0 1 0.838317,-2.13743 L 120.47065,89.897871 a 11.0036,11.0036 0 0 1 8.88282,0 l 44.20871,19.558869 a 1.5822533,1.5822533 0 0 1 0.99377,1.58226 z"
+           id="path24-31"
+           style="stroke-width:0.555177" />
+        <path
+           class="cls-3"
+           d="m 139.0413,114.72537 19.11473,-7.69475 a 0.81055784,0.81055784 0 0 0 0,-1.50453 c -2.2207,-0.92714 -4.96328,-1.99308 -7.65033,-3.10899 -0.49411,-0.20541 -5.17425,3.15341 -5.60173,3.49762 l -8.23882,6.58439 c -1.99309,1.67108 -0.26649,3.28665 2.37615,2.22626 z"
+           id="path26-9"
+           style="stroke-width:0.555177" />
+        <circle
+           class="cls-3"
+           cx="125.18409"
+           cy="122.24245"
+           r="9.9654207"
+           id="circle28-4"
+           style="stroke-width:0.555177" />
+      </g>
+      <path
+         class="cls-1"
+         d="m 162.59498,140.73295 1.36608,-0.009 h 0.0176 q 0.75796,0 0.75796,0.71389 v 2.30031 a 6.5748177,6.5748177 0 0 1 -2.28855,0.37897 q -1.25151,0 -1.88608,-0.85049 -0.63456,-0.8505 -0.63456,-2.31793 0,-1.46891 0.7888,-2.28268 a 2.5823345,2.5823345 0 0 1 1.93014,-0.81524 3.5371227,3.5371227 0 0 1 2.06675,0.64338 1.0385157,1.0385157 0 0 1 -0.18068,0.46711 1.2603203,1.2603203 0 0 1 -0.33931,0.35254 2.5926169,2.5926169 0 0 0 -1.5027,-0.52 1.4174931,1.4174931 0 0 0 -1.1854,0.54203 q -0.42305,0.53909 -0.42305,1.69658 0,2.17692 1.60405,2.17692 a 4.4742838,4.4742838 0 0 0 0.97829,-0.11457 v -0.83728 q 0,-0.3966 0.0176,-0.58756 h -0.64632 a 0.60518875,0.60518875 0 0 1 -0.40101,-0.11017 0.44067142,0.44067142 0 0 1 -0.12779,-0.35254 1.5100341,1.5100341 0 0 1 0.0881,-0.47445 z"
+         id="path8-6-4"
+         style="fill:#c3c3c3;fill-opacity:1;stroke-width:0.146891" />
+      <path
+         class="cls-1"
+         d="m 169.39307,143.50037 a 1.141339,1.141339 0 0 1 -0.14689,0.31288 1.0664248,1.0664248 0 0 1 -0.22474,0.25118 0.9959174,0.9959174 0 0 1 -0.80937,-0.51706 1.7847193,1.7847193 0 0 1 -1.26032,0.56406 q -0.67863,0 -1.02823,-0.3966 a 1.357268,1.357268 0 0 1 -0.34373,-0.9166 q 0,-0.73445 0.48034,-1.1149 a 1.9404232,1.9404232 0 0 1 1.23535,-0.36869 q 0.40541,0 0.76676,0.0352 v -0.2644 q 0,-0.69626 -0.66982,-0.69626 -0.47592,0 -1.34845,0.31728 a 1.2368178,1.2368178 0 0 1 -0.29378,-0.78439 4.9164242,4.9164242 0 0 1 1.90957,-0.39661 1.5526323,1.5526323 0 0 1 1.07524,0.37017 q 0.41423,0.37016 0.41423,1.1193 v 1.79794 q -0.003,0.48474 0.24384,0.68745 z m -2.21217,-0.22034 a 1.2471001,1.2471001 0 0 0 0.88134,-0.42304 v -0.77852 a 5.9182171,5.9182171 0 0 0 -0.66982,-0.0353 0.73445237,0.73445237 0 0 0 -0.54643,0.18215 0.63309793,0.63309793 0 0 0 -0.18508,0.46711 0.62281561,0.62281561 0 0 0 0.14689,0.44067 0.48767637,0.48767637 0 0 0 0.3731,0.14689 z"
+         id="path10-2-5"
+         style="fill:#c3c3c3;fill-opacity:1;stroke-width:0.146891" />
+      <path
+         class="cls-1"
+         d="m 170.10696,140.13364 a 1.0546736,1.0546736 0 0 1 0.14689,-0.32169 0.88134284,0.88134284 0 0 1 0.22474,-0.25118 1.1016786,1.1016786 0 0 1 0.92982,0.78439 q 0.35254,-0.78439 1.13693,-0.78439 a 2.7027846,2.7027846 0 0 1 0.51118,0.0617 1.9786147,1.9786147 0 0 1 -0.2644,1.02823 2.235673,2.235673 0 0 0 -0.39661,-0.0529 q -0.53762,0 -0.86371,0.57287 v 2.81736 a 3.0626663,3.0626663 0 0 1 -0.53762,0.0441 3.3784809,3.3784809 0 0 1 -0.55525,-0.0441 v -2.95249 q -0.006,-0.63751 -0.33197,-0.90191 z"
+         id="path12-6-0"
+         style="fill:#c3c3c3;fill-opacity:1;stroke-width:0.146891" />
+      <path
+         class="cls-1"
+         d="m 177.14889,143.50037 a 1.141339,1.141339 0 0 1 -0.15424,0.31288 1.0664248,1.0664248 0 0 1 -0.22474,0.25118 0.9959174,0.9959174 0 0 1 -0.8079,-0.51706 1.7847193,1.7847193 0 0 1 -1.26032,0.56406 q -0.67863,0 -1.02823,-0.3966 a 1.357268,1.357268 0 0 1 -0.34372,-0.9166 q 0,-0.73445 0.48033,-1.1149 a 1.9404232,1.9404232 0 0 1 1.22947,-0.37457 q 0.40542,0 0.76677,0.0353 v -0.26441 q 0,-0.69626 -0.66982,-0.69626 -0.47593,0 -1.34846,0.31729 a 1.2368178,1.2368178 0 0 1 -0.29378,-0.7844 4.9164242,4.9164242 0 0 1 1.90958,-0.3966 1.5526323,1.5526323 0 0 1 1.07524,0.37016 q 0.41423,0.37017 0.41423,1.11931 v 1.80381 q 0.009,0.48474 0.25559,0.68745 z m -2.21511,-0.22034 a 1.2471001,1.2471001 0 0 0 0.88134,-0.42304 v -0.77852 a 5.9182171,5.9182171 0 0 0 -0.66982,-0.0353 0.73445237,0.73445237 0 0 0 -0.54643,0.18509 0.63309793,0.63309793 0 0 0 -0.18508,0.46711 0.62281561,0.62281561 0 0 0 0.14689,0.44067 0.48767637,0.48767637 0 0 0 0.3731,0.14395 z"
+         id="path14-1-3"
+         style="fill:#c3c3c3;fill-opacity:1;stroke-width:0.146891" />
+      <path
+         class="cls-1"
+         d="m 178.23294,143.46511 q -0.3966,-0.60812 -0.3966,-1.67895 0,-1.07084 0.50236,-1.67896 a 1.5188475,1.5188475 0 0 1 1.20744,-0.60813 1.7039295,1.7039295 0 0 1 1.18981,0.44067 0.99738631,0.99738631 0 0 1 0.69626,-0.37897 0.82552446,0.82552446 0 0 1 0.23356,0.24677 1.0282333,1.0282333 0 0 1 0.14689,0.30847 q -0.24678,0.21152 -0.24678,0.75796 v 2.49714 q 0,1.40133 -0.45829,1.98302 -0.4583,0.58168 -1.5071,0.58756 a 4.2598236,4.2598236 0 0 1 -1.5776,-0.29378 1.1854061,1.1854061 0 0 1 0.27321,-0.80203 2.8819911,2.8819911 0 0 0 1.18541,0.27322 q 0.57728,0 0.79761,-0.29378 a 1.3220143,1.3220143 0 0 0 0.22034,-0.81084 v -0.35253 a 1.6936472,1.6936472 0 0 1 -1.10168,0.41423 1.3014496,1.3014496 0 0 1 -1.16484,-0.61107 z m 2.26505,-0.71388 v -2.04472 a 1.1354634,1.1354634 0 0 0 -0.75795,-0.36135 0.63603576,0.63603576 0 0 0 -0.57728,0.37898 2.2988359,2.2988359 0 0 0 -0.20712,1.08405 q 0,0.70508 0.18949,1.03998 a 0.56405941,0.56405941 0 0 0 0.49796,0.33491 1.1193054,1.1193054 0 0 0 0.8549,-0.43185 z"
+         id="path16-8-6"
+         style="fill:#c3c3c3;fill-opacity:1;stroke-width:0.146891" />
+      <path
+         class="cls-1"
+         d="m 186.09746,142.16073 h -2.46776 a 1.4924072,1.4924072 0 0 0 0.23355,0.80643 q 0.20712,0.28643 0.72711,0.28643 a 2.6778132,2.6778132 0 0 0 1.15456,-0.30847 1.1589658,1.1589658 0 0 1 0.31728,0.66982 2.8467375,2.8467375 0 0 1 -1.69658,0.50237 q -0.99151,0 -1.42337,-0.64338 -0.43186,-0.64338 -0.43186,-1.66574 0,-1.02823 0.47593,-1.66574 a 1.5922927,1.5922927 0 0 1 1.36167,-0.64338 q 0.88134,0 1.36167,0.53321 a 1.943361,1.943361 0 0 1 0.47593,1.34405 3.4519261,3.4519261 0 0 1 -0.0881,0.7844 z m -1.701,-1.86845 q -0.7227,0 -0.77558,1.09287 h 1.53354 v -0.10577 a 1.2500379,1.2500379 0 0 0 -0.18508,-0.71388 0.64338027,0.64338027 0 0 0 -0.567,-0.27322 z"
+         id="path18-7-1"
+         style="fill:#c3c3c3;fill-opacity:1;stroke-width:0.146891" />
+      <path
+         id="path24-3-6-9-0"
+         style="fill:#ff9329;fill-opacity:1;stroke-width:0.146891"
+         d="m 173.02622,117.15529 a 2.9113691,2.9113691 0 0 0 -1.14618,0.24753 l -11.69696,5.17488 a 0.42304456,0.42304456 0 0 0 -0.22169,0.56586 0.20417776,0.20417776 0 0 0 0.0176,0.047 l 0.79634,1.57355 11.10475,-4.91288 a 2.9113691,2.9113691 0 0 1 1.14618,-0.24753 2.9113691,2.9113691 0 0 1 1.20406,0.24753 l 11.12387,4.92115 0.7829,-1.54823 0.0176,-0.0336 0.0181,-0.0455 a 0.02790919,0.02790919 0 0 0 0,-0.0119 0.39366647,0.39366647 0 0 0 0.0176,-0.13642 0.41863785,0.41863785 0 0 0 -0.26303,-0.4191 l -11.69697,-5.17488 a 2.9113691,2.9113691 0 0 0 -1.20406,-0.24753 z m -10.12134,9.52449 c 0.0218,0.0723 0.0408,0.14674 0.0615,0.22066 h 0.51831 l -0.008,-0.0419 z m 20.32227,0.005 -0.57103,0.17828 -0.007,0.0377 h 0.5178 c 0.0202,-0.0723 0.0386,-0.14514 0.0599,-0.216 z" />
+      <path
+         class="cls-2"
+         d="m 186.19,127.94173 a 0.39366647,0.39366647 0 0 1 -0.0176,0.13661 0.02790919,0.02790919 0 0 1 0,0.0117 l -0.0176,0.0455 v 0 l -0.0176,0.0338 -2.83058,5.59652 c -0.39366,0.77705 -1.11783,0.75355 -0.99591,-0.0323 l 0.56993,-3.18165 c 0.0191,-0.10429 0.18655,-0.83874 0.34666,-1.37049 l -5.46285,1.7054 c -0.85784,5.57156 -8.18915,5.6641 -9.38484,0 l -5.4746,-1.70981 c 0.16011,0.53175 0.32903,1.27061 0.34813,1.3749 l 0.56993,3.18165 c 0.12192,0.78586 -0.60225,0.80936 -0.99592,0.0323 l -2.84822,-5.63031 a 0.20417776,0.20417776 0 0 1 -0.0176,-0.047 0.42304456,0.42304456 0 0 1 0.22181,-0.56553 l 11.69688,-5.17495 a 2.9113691,2.9113691 0 0 1 2.35025,0 l 11.69689,5.17495 a 0.41863785,0.41863785 0 0 1 0.26293,0.41864 z"
+         id="path24-0-3"
+         style="fill:#ff9329;fill-opacity:1;stroke-width:0.146891" />
+      <path
+         class="cls-3"
+         d="m 176.79341,128.91708 5.05744,-2.0359 a 0.21446009,0.21446009 0 0 0 0,-0.39807 c -0.58756,-0.24531 -1.3132,-0.52734 -2.02415,-0.82259 -0.13073,-0.0543 -1.36902,0.83434 -1.48212,0.92541 l -2.17986,1.74212 c -0.52734,0.44214 -0.0705,0.86959 0.62869,0.58903 z"
+         id="path26-2-2"
+         style="fill:#4e4e4e;fill-opacity:1;stroke-width:0.146891" />
+      <circle
+         class="cls-3"
+         cx="173.12703"
+         cy="130.90596"
+         r="2.6366842"
+         id="circle28-3-0"
+         style="fill:#4e4e4e;fill-opacity:1;stroke-width:0.146891" />
+    </g>
+  </g>
+</svg>
@@ -0,0 +1,146 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<svg
+   xmlns:dc="http://purl.org/dc/elements/1.1/"
+   xmlns:cc="http://creativecommons.org/ns#"
+   xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
+   xmlns:svg="http://www.w3.org/2000/svg"
+   xmlns="http://www.w3.org/2000/svg"
+   xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
+   xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
+   id="Calque_1"
+   data-name="Calque 1"
+   width="250"
+   height="250"
+   viewBox="0 0 249.99999 250"
+   version="1.1"
+   sodipodi:docname="garage-notext.svg"
+   inkscape:version="1.0.2 (e86c870879, 2021-01-15)"
+   inkscape:export-filename="/home/lx/Deuxfleurs/garage/garage-notext.png"
+   inkscape:export-xdpi="96"
+   inkscape:export-ydpi="96">
+  <metadata
+     id="metadata33">
+    <rdf:RDF>
+      <cc:Work
+         rdf:about="">
+        <dc:format>image/svg+xml</dc:format>
+        <dc:type
+           rdf:resource="http://purl.org/dc/dcmitype/StillImage" />
+        <dc:title></dc:title>
+      </cc:Work>
+    </rdf:RDF>
+  </metadata>
+  <sodipodi:namedview
+     pagecolor="#ffffff"
+     bordercolor="#666666"
+     borderopacity="1"
+     objecttolerance="10"
+     gridtolerance="10"
+     guidetolerance="10"
+     inkscape:pageopacity="1"
+     inkscape:pageshadow="2"
+     inkscape:window-width="1920"
+     inkscape:window-height="1039"
+     id="namedview31"
+     showgrid="false"
+     inkscape:zoom="2.1842656"
+     inkscape:cx="143.86571"
+     inkscape:cy="118.5836"
+     inkscape:window-x="0"
+     inkscape:window-y="20"
+     inkscape:window-maximized="0"
+     inkscape:current-layer="Calque_1"
+     inkscape:document-rotation="0"
+     units="px"
+     showguides="false"
+     inkscape:guide-bbox="true"
+     inkscape:snap-global="false"
+     width="250mm">
+    <sodipodi:guide
+       position="102.90662,161.07694"
+       orientation="0,-1"
+       id="guide1016" />
+    <sodipodi:guide
+       position="122.45269,170.65683"
+       orientation="0,-1"
+       id="guide1018" />
+    <sodipodi:guide
+       position="128.86504,180.08221"
+       orientation="0,-1"
+       id="guide1020" />
+  </sodipodi:namedview>
+  <defs
+     id="defs4">
+    <style
+       id="style2">.cls-1{fill:#3b2100;}.cls-2{fill:#ffd952;}.cls-3{fill:#45c8ff;}</style>
+  </defs>
+  <rect
+     style="fill:#ffffff;stroke-width:3.60793"
+     id="rect3824"
+     width="251.68179"
+     height="250.98253"
+     x="-0.59092933"
+     y="-0.31321606" />
+  <g
+     id="g1719"
+     transform="matrix(1.9099251,0,0,1.9099251,-113.74064,-74.610597)">
+    <path
+       d="m 138.41049,100.63656 a 8.327649,8.327649 0 0 1 -2.77589,-0.28869 l -34.78736,-9.388039 a 8.4442361,8.4442361 0 0 1 -2.620438,-1.238044 z"
+       id="path6"
+       style="stroke-width:0.555177" />
+    <path
+       id="path24-3-6"
+       style="fill:#ffd952;fill-opacity:1;stroke-width:0.555177"
+       d="m 124.88254,70.600847 a 11.0036,11.0036 0 0 0 -4.33203,0.935547 L 76.341524,91.094987 a 1.5989086,1.5989086 0 0 0 -0.837891,2.138672 0.77169547,0.77169547 0 0 0 0.06641,0.177735 l 7.09375,14.021486 h 6.15625 l -0.875,-4.88867 c -0.07217,-0.39418 -0.711263,-3.187537 -1.316406,-5.197269 l 20.691403,6.462899 c 0.27198,1.28839 0.63292,2.49204 1.0625,3.62304 h 33.54883 c 0.36964,-1.13128 0.66138,-2.33705 0.85938,-3.62304 l 20.64648,-6.445321 c -0.60514,2.009734 -1.23639,4.785511 -1.30859,5.179691 l -0.875,4.88867 h 6.15429 l 7.02735,-13.894533 0.0664,-0.126953 0.0684,-0.171875 a 0.10548355,0.10548355 0 0 0 0,-0.04492 1.4878733,1.4878733 0 0 0 0.0664,-0.515625 1.5822533,1.5822533 0 0 0 -0.99414,-1.583985 L 129.43333,71.536394 a 11.0036,11.0036 0 0 0 -4.55079,-0.935547 z" />
+    <path
+       id="path24-3"
+       style="fill:#49c8fa;fill-opacity:1;stroke-width:0.555177"
+       d="m 124.88254,79.854518 a 11.0036,11.0036 0 0 0 -4.33203,0.935547 L 76.341524,100.34866 a 1.5989086,1.5989086 0 0 0 -0.837891,2.13672 0.77169547,0.77169547 0 0 0 0.06641,0.17773 l 3.847657,7.60352 h 8.175781 c -0.257897,-1.08856 -0.591943,-2.42953 -0.964844,-3.66797 l 11.744141,3.66797 h 53.371092 l 11.69336,-3.65039 c -0.37193,1.23522 -0.70076,2.56719 -0.95703,3.65039 h 8.17383 l 3.78125,-7.47656 0.0664,-0.12696 0.0684,-0.17187 a 0.10548355,0.10548355 0 0 0 0,-0.0449 1.4878733,1.4878733 0 0 0 0.0664,-0.51563 1.5822533,1.5822533 0 0 0 -0.99414,-1.58203 L 129.43333,80.790065 a 11.0036,11.0036 0 0 0 -4.55079,-0.935547 z" />
+    <path
+       class="cls-2"
+       d="m 174.63576,111.36813 a 1.4878733,1.4878733 0 0 1 -0.0666,0.51631 0.10548355,0.10548355 0 0 1 0,0.0444 l -0.0666,0.17211 v 0 l -0.0666,0.12769 -10.69826,21.15223 c -1.48787,2.93688 -4.22489,2.84806 -3.76409,-0.12214 l 2.15408,-12.02512 c 0.0722,-0.39418 0.70508,-3.17006 1.31022,-5.1798 l -20.64702,6.4456 c -3.24223,21.05785 -30.95109,21.40761 -35.47023,0 l -20.691437,-6.46226 c 0.605143,2.00974 1.243596,4.80228 1.315769,5.19646 l 2.154085,12.02512 c 0.460796,2.9702 -2.276224,3.05902 -3.764098,0.12214 L 75.570045,112.10096 a 0.77169547,0.77169547 0 0 1 -0.06662,-0.17766 1.5989086,1.5989086 0 0 1 0.838317,-2.13743 L 120.55046,90.226998 a 11.0036,11.0036 0 0 1 8.88282,0 l 44.20871,19.558872 a 1.5822533,1.5822533 0 0 1 0.99377,1.58226 z"
+       id="path24"
+       style="stroke-width:0.555177" />
+    <path
+       class="cls-3"
+       d="m 139.12111,115.0545 19.11473,-7.69475 a 0.81055784,0.81055784 0 0 0 0,-1.50453 c -2.2207,-0.92714 -4.96328,-1.99308 -7.65033,-3.10899 -0.49411,-0.20541 -5.17425,3.15341 -5.60173,3.49762 l -8.23882,6.58439 c -1.99309,1.67108 -0.26649,3.28665 2.37615,2.22626 z"
+       id="path26"
+       style="stroke-width:0.555177" />
+    <circle
+       class="cls-3"
+       cx="125.26389"
+       cy="122.57157"
+       r="9.9654207"
+       id="circle28"
+       style="stroke-width:0.555177" />
+    <path
+       d="m 138.41049,100.63656 a 8.327649,8.327649 0 0 1 -2.77589,-0.28869 l -34.78736,-9.388039 a 8.4442361,8.4442361 0 0 1 -2.620438,-1.238044 z"
+       id="path6-0"
+       style="stroke-width:0.555177" />
+    <path
+       id="path24-3-6-9"
+       style="fill:#ff9329;fill-opacity:1;stroke-width:0.555177"
+       d="m 124.88254,70.600847 a 11.0036,11.0036 0 0 0 -4.33203,0.935547 L 76.341524,91.094987 a 1.5989086,1.5989086 0 0 0 -0.837891,2.138672 0.77169547,0.77169547 0 0 0 0.06641,0.177735 l 7.09375,14.021486 h 6.15625 l -0.875,-4.88867 c -0.07217,-0.39418 -0.711263,-3.187537 -1.316406,-5.197269 l 20.691403,6.462899 c 0.27198,1.28839 0.63292,2.49204 1.0625,3.62304 h 33.54883 c 0.36964,-1.13128 0.66138,-2.33705 0.85938,-3.62304 l 20.64648,-6.445321 c -0.60514,2.009734 -1.23639,4.785511 -1.30859,5.179691 l -0.875,4.88867 h 6.15429 l 7.02735,-13.894533 0.0664,-0.126953 0.0684,-0.171875 a 0.10548355,0.10548355 0 0 0 0,-0.04492 1.4878733,1.4878733 0 0 0 0.0664,-0.515625 1.5822533,1.5822533 0 0 0 -0.99414,-1.583985 L 129.43333,71.536394 a 11.0036,11.0036 0 0 0 -4.55079,-0.935547 z" />
+    <path
+       id="path24-3-2"
+       style="fill:#4e4e4e;fill-opacity:1;stroke-width:0.555177"
+       d="m 124.88254,79.854518 a 11.0036,11.0036 0 0 0 -4.33203,0.935547 L 76.341524,100.34866 a 1.5989086,1.5989086 0 0 0 -0.837891,2.13672 0.77169547,0.77169547 0 0 0 0.06641,0.17773 l 3.847657,7.60352 h 8.175781 c -0.257897,-1.08856 -0.591943,-2.42953 -0.964844,-3.66797 l 11.744141,3.66797 h 53.371092 l 11.69336,-3.65039 c -0.37193,1.23522 -0.70076,2.56719 -0.95703,3.65039 h 8.17383 l 3.78125,-7.47656 0.0664,-0.12696 0.0684,-0.17187 a 0.10548355,0.10548355 0 0 0 0,-0.0449 1.4878733,1.4878733 0 0 0 0.0664,-0.51563 1.5822533,1.5822533 0 0 0 -0.99414,-1.58203 L 129.43333,80.790065 a 11.0036,11.0036 0 0 0 -4.55079,-0.935547 z" />
+    <path
+       class="cls-2"
+       d="m 174.63576,111.36813 a 1.4878733,1.4878733 0 0 1 -0.0666,0.51631 0.10548355,0.10548355 0 0 1 0,0.0444 l -0.0666,0.17211 v 0 l -0.0666,0.12769 -10.69826,21.15223 c -1.48787,2.93688 -4.22489,2.84806 -3.76409,-0.12214 l 2.15408,-12.02512 c 0.0722,-0.39418 0.70508,-3.17006 1.31022,-5.1798 l -20.64702,6.4456 c -3.24223,21.05785 -30.95109,21.40761 -35.47023,0 l -20.691437,-6.46226 c 0.605143,2.00974 1.243596,4.80228 1.315769,5.19646 l 2.154085,12.02512 c 0.460796,2.9702 -2.276224,3.05902 -3.764098,0.12214 L 75.570045,112.10096 a 0.77169547,0.77169547 0 0 1 -0.06662,-0.17766 1.5989086,1.5989086 0 0 1 0.838317,-2.13743 L 120.55046,90.226998 a 11.0036,11.0036 0 0 1 8.88282,0 l 44.20871,19.558872 a 1.5822533,1.5822533 0 0 1 0.99377,1.58226 z"
+       id="path24-0"
+       style="fill:#ff9329;fill-opacity:1;stroke-width:0.555177" />
+    <path
+       class="cls-3"
+       d="m 139.12111,115.0545 19.11473,-7.69475 a 0.81055784,0.81055784 0 0 0 0,-1.50453 c -2.2207,-0.92714 -4.96328,-1.99308 -7.65033,-3.10899 -0.49411,-0.20541 -5.17425,3.15341 -5.60173,3.49762 l -8.23882,6.58439 c -1.99309,1.67108 -0.26649,3.28665 2.37615,2.22626 z"
+       id="path26-2"
+       style="fill:#4e4e4e;fill-opacity:1;stroke-width:0.555177" />
+    <circle
+       class="cls-3"
+       cx="125.26389"
+       cy="122.57157"
+       r="9.9654207"
+       id="circle28-3"
+       style="fill:#4e4e4e;fill-opacity:1;stroke-width:0.555177" />
+  </g>
+</svg>
@@ -0,0 +1,206 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<svg
+   xmlns:dc="http://purl.org/dc/elements/1.1/"
+   xmlns:cc="http://creativecommons.org/ns#"
+   xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
+   xmlns:svg="http://www.w3.org/2000/svg"
+   xmlns="http://www.w3.org/2000/svg"
+   xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
+   xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
+   id="Calque_1"
+   data-name="Calque 1"
+   width="250"
+   height="250"
+   viewBox="0 0 249.99999 250"
+   version="1.1"
+   sodipodi:docname="garage.svg"
+   inkscape:version="1.0.2 (e86c870879, 2021-01-15)"
+   inkscape:export-filename="/home/lx/Deuxfleurs/garage/doc/logo/garage.png"
+   inkscape:export-xdpi="96"
+   inkscape:export-ydpi="96">
+  <metadata
+     id="metadata33">
+    <rdf:RDF>
+      <cc:Work
+         rdf:about="">
+        <dc:format>image/svg+xml</dc:format>
+        <dc:type
+           rdf:resource="http://purl.org/dc/dcmitype/StillImage" />
+        <dc:title />
+      </cc:Work>
+    </rdf:RDF>
+  </metadata>
+  <sodipodi:namedview
+     pagecolor="#ffffff"
+     bordercolor="#666666"
+     borderopacity="1"
+     objecttolerance="10"
+     gridtolerance="10"
+     guidetolerance="10"
+     inkscape:pageopacity="1"
+     inkscape:pageshadow="2"
+     inkscape:window-width="1920"
+     inkscape:window-height="1080"
+     id="namedview31"
+     showgrid="false"
+     inkscape:zoom="2.1842656"
+     inkscape:cx="90.853672"
+     inkscape:cy="123.63257"
+     inkscape:window-x="0"
+     inkscape:window-y="0"
+     inkscape:window-maximized="0"
+     inkscape:current-layer="Calque_1"
+     inkscape:document-rotation="0"
+     units="px"
+     showguides="false"
+     inkscape:guide-bbox="true"
+     inkscape:snap-global="false"
+     width="250mm">
+    <sodipodi:guide
+       position="102.90662,161.07694"
+       orientation="0,-1"
+       id="guide1016" />
+    <sodipodi:guide
+       position="122.45269,170.65683"
+       orientation="0,-1"
+       id="guide1018" />
+    <sodipodi:guide
+       position="128.86504,180.08221"
+       orientation="0,-1"
+       id="guide1020" />
+  </sodipodi:namedview>
+  <defs
+     id="defs4">
+    <style
+       id="style2">.cls-1{fill:#3b2100;}.cls-2{fill:#ffd952;}.cls-3{fill:#45c8ff;}</style>
+  </defs>
+  <rect
+     style="fill:#ffffff;stroke-width:3.60793"
+     id="rect3824"
+     width="251.68179"
+     height="250.98253"
+     x="-0.59092933"
+     y="-0.31321606" />
+  <g
+     id="g1663"
+     transform="matrix(1.7099534,0,0,1.7099534,-88.607712,-87.994557)">
+    <path
+       d="m 138.33068,100.19817 a 8.327649,8.327649 0 0 1 -2.77589,-0.288688 l -34.78736,-9.388036 a 8.4442361,8.4442361 0 0 1 -2.620433,-1.238044 z"
+       id="path6"
+       style="stroke-width:0.555177" />
+    <path
+       class="cls-1"
+       d="m 85.377935,159.27452 5.163143,-0.0333 h 0.06662 q 2.864711,0 2.864711,2.69816 v 8.69407 a 24.849705,24.849705 0 0 1 -8.649651,1.43235 q -4.730105,0 -7.128468,-3.21447 -2.398363,-3.21447 -2.398363,-8.76068 0,-5.55177 2.981299,-8.62745 a 9.7600046,9.7600046 0 0 1 7.29502,-3.08123 13.368653,13.368653 0 0 1 7.811335,2.43167 3.9250986,3.9250986 0 0 1 -0.682867,1.76547 4.7634152,4.7634152 0 0 1 -1.282458,1.33242 9.798867,9.798867 0 0 0 -5.679457,-1.96533 5.3574542,5.3574542 0 0 0 -4.480275,2.04861 q -1.598909,2.03749 -1.598909,6.41229 0,8.22771 6.062529,8.22771 a 16.910679,16.910679 0 0 0 3.697476,-0.43303 v -3.16451 q 0,-1.49898 0.06662,-2.22071 h -2.442777 a 2.2873276,2.2873276 0 0 1 -1.515632,-0.41638 1.6655298,1.6655298 0 0 1 -0.483004,-1.33242 5.7072154,5.7072154 0 0 1 0.333106,-1.79322 z"
+       id="path8"
+       style="fill:#4e4e4e;fill-opacity:1;stroke-width:0.555177" />
+    <path
+       class="cls-1"
+       d="m 111.07151,169.73404 a 4.3137222,4.3137222 0 0 1 -0.55518,1.18253 4.0305821,4.0305821 0 0 1 -0.84942,0.94935 3.7640973,3.7640973 0 0 1 -3.05902,-1.95422 6.7453957,6.7453957 0 0 1 -4.76342,2.13188 q -2.564913,0 -3.886233,-1.49898 a 5.1298318,5.1298318 0 0 1 -1.299113,-3.4643 q 0,-2.77588 1.815427,-4.21379 a 7.3338829,7.3338829 0 0 1 4.669039,-1.3935 q 1.53228,0 2.89802,0.13325 v -0.99932 q 0,-2.63154 -2.53161,-2.63154 -1.79877,0 -5.096518,1.19918 a 4.674587,4.674587 0 0 1 -1.110353,-2.96464 18.581761,18.581761 0 0 1 7.217291,-1.49898 5.8682167,5.8682167 0 0 1 4.0639,1.39905 q 1.56559,1.39904 1.56559,4.23044 v 6.79537 q -0.0111,1.83208 0.9216,2.59822 z m -8.36096,-0.83276 a 4.7134493,4.7134493 0 0 0 3.33106,-1.59891 v -2.94244 a 22.368065,22.368065 0 0 0 -2.53161,-0.13324 2.775883,2.775883 0 0 0 -2.06525,0.68842 2.3928111,2.3928111 0 0 0 -0.69953,1.76546 2.3539488,2.3539488 0 0 0 0.55518,1.66553 1.8431863,1.8431863 0 0 0 1.41015,0.55518 z"
+       id="path10"
+       style="fill:#4e4e4e;fill-opacity:1;stroke-width:0.555177" />
+    <path
+       class="cls-1"
+       d="m 113.76966,157.00939 a 3.986168,3.986168 0 0 1 0.55518,-1.21583 3.3310596,3.3310596 0 0 1 0.84942,-0.94935 4.1638245,4.1638245 0 0 1 3.51427,2.96464 q 1.33242,-2.96464 4.29707,-2.96464 a 10.215249,10.215249 0 0 1 1.93201,0.23317 7.4782288,7.4782288 0 0 1 -0.99932,3.88624 8.4497879,8.4497879 0 0 0 -1.49897,-0.19987 q -2.03195,0 -3.26444,2.16519 v 10.64829 a 11.575432,11.575432 0 0 1 -2.03195,0.16655 12.769062,12.769062 0 0 1 -2.09857,-0.16655 v -11.15905 q -0.0222,-2.40947 -1.2547,-3.40879 z"
+       id="path12"
+       style="fill:#4e4e4e;fill-opacity:1;stroke-width:0.555177" />
+    <path
+       class="cls-1"
+       d="m 140.38483,169.73404 a 4.3137222,4.3137222 0 0 1 -0.58293,1.18253 4.0305821,4.0305821 0 0 1 -0.84942,0.94935 3.7640973,3.7640973 0 0 1 -3.05348,-1.95422 6.7453957,6.7453957 0 0 1 -4.76341,2.13188 q -2.56492,0 -3.88624,-1.49898 a 5.1298318,5.1298318 0 0 1 -1.29911,-3.4643 q 0,-2.77588 1.81543,-4.21379 a 7.3338829,7.3338829 0 0 1 4.64682,-1.4157 q 1.53229,0 2.89803,0.13324 v -0.99932 q 0,-2.63153 -2.53161,-2.63153 -1.79877,0 -5.09652,1.19918 a 4.674587,4.674587 0 0 1 -1.11035,-2.96465 18.581761,18.581761 0 0 1 7.21729,-1.49897 5.8682167,5.8682167 0 0 1 4.0639,1.39904 q 1.56559,1.39905 1.56559,4.23045 v 6.81757 q 0.0333,1.83208 0.96601,2.59822 z m -8.37206,-0.83276 a 4.7134493,4.7134493 0 0 0 3.33106,-1.59891 v -2.94244 a 22.368065,22.368065 0 0 0 -2.53161,-0.13324 2.775883,2.775883 0 0 0 -2.06526,0.69952 2.3928111,2.3928111 0 0 0 -0.69952,1.76546 2.3539488,2.3539488 0 0 0 0.55518,1.66553 1.8431863,1.8431863 0 0 0 1.41015,0.54408 z"
+       id="path14"
+       style="fill:#4e4e4e;fill-opacity:1;stroke-width:0.555177" />
+    <path
+       class="cls-1"
+       d="m 144.48203,169.6008 q -1.49897,-2.29843 -1.49897,-6.34567 0,-4.04724 1.8987,-6.34567 a 5.740526,5.740526 0 0 1 4.56355,-2.29843 6.4400486,6.4400486 0 0 1 4.49693,1.66553 3.7696491,3.7696491 0 0 1 2.63154,-1.43235 3.1200925,3.1200925 0 0 1 0.88273,0.93269 3.8862362,3.8862362 0 0 1 0.55518,1.16587 q -0.9327,0.79946 -0.9327,2.86472 v 9.438 q 0,5.29638 -1.73215,7.49488 -1.73215,2.1985 -5.69611,2.22071 a 16.100121,16.100121 0 0 1 -5.9626,-1.11036 4.4802752,4.4802752 0 0 1 1.03263,-3.03126 10.892565,10.892565 0 0 0 4.48028,1.03263 q 2.18184,0 3.0146,-1.11035 a 4.9965894,4.9965894 0 0 0 0.83277,-3.06458 v -1.33242 a 6.4011862,6.4011862 0 0 1 -4.16383,1.56559 4.9188647,4.9188647 0 0 1 -4.40255,-2.30953 z m 8.56083,-2.69816 v -7.72806 a 4.2915151,4.2915151 0 0 0 -2.86471,-1.36573 2.4039147,2.4039147 0 0 0 -2.18185,1.43235 8.6885138,8.6885138 0 0 0 -0.7828,4.09721 q 0,2.66485 0.71618,3.93065 a 2.1318781,2.1318781 0 0 0 1.88205,1.2658 4.2304457,4.2304457 0 0 0 3.23113,-1.63222 z"
+       id="path16"
+       style="fill:#4e4e4e;fill-opacity:1;stroke-width:0.555177" />
+    <path
+       class="cls-1"
+       d="m 174.20619,164.67083 h -9.32697 a 5.6405943,5.6405943 0 0 0 0.88273,3.04792 q 0.7828,1.0826 2.74813,1.0826 a 10.120869,10.120869 0 0 0 4.36369,-1.16587 4.3803434,4.3803434 0 0 1 1.19918,2.5316 10.759323,10.759323 0 0 1 -6.41229,1.8987 q -3.74744,0 -5.37966,-2.43167 -1.63222,-2.43167 -1.63222,-6.2957 0,-3.88624 1.79877,-6.2957 a 6.0181143,6.0181143 0 0 1 5.14649,-2.43168 q 3.33106,0 5.14648,2.01529 a 7.3449864,7.3449864 0 0 1 1.79878,5.07987 13.04665,13.04665 0 0 1 -0.33311,2.96464 z m -6.42895,-7.06184 q -2.73146,0 -2.93133,4.13051 h 5.79605 v -0.39973 a 4.7245529,4.7245529 0 0 0 -0.69953,-2.69816 2.4316735,2.4316735 0 0 0 -2.14298,-1.03262 z"
+       id="path18"
+       style="fill:#4e4e4e;fill-opacity:1;stroke-width:0.555177" />
+    <path
+       id="path24-3-6"
+       style="fill:#ffd952;fill-opacity:1;stroke-width:0.555177"
+       d="m 124.80273,70.162462 a 11.0036,11.0036 0 0 0 -4.33203,0.935547 L 76.261719,90.656602 a 1.5989086,1.5989086 0 0 0 -0.837891,2.138672 0.77169547,0.77169547 0 0 0 0.06641,0.177735 l 7.09375,14.021481 h 6.15625 l -0.875,-4.88867 c -0.07217,-0.39418 -0.711263,-3.187532 -1.316406,-5.197264 l 20.691398,6.462894 c 0.27198,1.28839 0.63292,2.49204 1.0625,3.62304 h 33.54883 c 0.36964,-1.13128 0.66138,-2.33705 0.85938,-3.62304 l 20.64648,-6.445316 c -0.60514,2.009734 -1.23639,4.785506 -1.30859,5.179686 l -0.875,4.88867 h 6.15429 l 7.02735,-13.894528 0.0664,-0.126953 0.0684,-0.171875 a 0.10548355,0.10548355 0 0 0 0,-0.04492 1.4878733,1.4878733 0 0 0 0.0664,-0.515625 1.5822533,1.5822533 0 0 0 -0.99414,-1.583985 L 129.35352,71.098009 a 11.0036,11.0036 0 0 0 -4.55079,-0.935547 z" />
+    <path
+       id="path24-3"
+       style="fill:#49c8fa;fill-opacity:1;stroke-width:0.555177"
+       d="M 124.80273,79.416133 A 11.0036,11.0036 0 0 0 120.4707,80.35168 L 76.261719,99.910272 a 1.5989086,1.5989086 0 0 0 -0.837891,2.136718 0.77169547,0.77169547 0 0 0 0.06641,0.17773 l 3.847657,7.60352 h 8.175781 c -0.257897,-1.08856 -0.591943,-2.42953 -0.964844,-3.66797 l 11.744141,3.66797 h 53.371087 l 11.69336,-3.65039 c -0.37193,1.23522 -0.70076,2.56719 -0.95703,3.65039 h 8.17383 l 3.78125,-7.47656 0.0664,-0.12696 0.0684,-0.17187 a 0.10548355,0.10548355 0 0 0 0,-0.0449 1.4878733,1.4878733 0 0 0 0.0664,-0.51563 1.5822533,1.5822533 0 0 0 -0.99414,-1.582028 L 129.35352,80.35168 a 11.0036,11.0036 0 0 0 -4.55079,-0.935547 z" />
+    <path
+       class="cls-2"
+       d="m 174.55595,110.92974 a 1.4878733,1.4878733 0 0 1 -0.0666,0.51631 0.10548355,0.10548355 0 0 1 0,0.0444 l -0.0666,0.17211 v 0 l -0.0666,0.12769 -10.69826,21.15223 c -1.48787,2.93688 -4.22489,2.84806 -3.76409,-0.12214 l 2.15408,-12.02512 c 0.0722,-0.39418 0.70508,-3.17006 1.31022,-5.1798 l -20.64702,6.4456 c -3.24223,21.05785 -30.95109,21.40761 -35.47023,0 l -20.691432,-6.46226 c 0.605143,2.00974 1.243596,4.80228 1.315769,5.19646 l 2.154085,12.02512 c 0.460796,2.9702 -2.276224,3.05902 -3.764098,0.12214 L 75.49024,111.66257 a 0.77169547,0.77169547 0 0 1 -0.06662,-0.17766 1.5989086,1.5989086 0 0 1 0.838317,-2.13743 L 120.47065,89.788613 a 11.0036,11.0036 0 0 1 8.88282,0 l 44.20871,19.558867 a 1.5822533,1.5822533 0 0 1 0.99377,1.58226 z"
+       id="path24"
+       style="stroke-width:0.555177" />
+    <path
+       class="cls-3"
+       d="m 139.0413,114.61611 19.11473,-7.69475 a 0.81055784,0.81055784 0 0 0 0,-1.50453 c -2.2207,-0.92714 -4.96328,-1.99308 -7.65033,-3.10899 -0.49411,-0.20541 -5.17425,3.15341 -5.60173,3.49762 l -8.23882,6.58439 c -1.99309,1.67108 -0.26649,3.28665 2.37615,2.22626 z"
+       id="path26"
+       style="stroke-width:0.555177" />
+    <circle
+       class="cls-3"
+       cx="125.18409"
+       cy="122.13319"
+       r="9.9654207"
+       id="circle28"
+       style="stroke-width:0.555177" />
+    <path
+       d="m 138.33068,100.19817 a 8.327649,8.327649 0 0 1 -2.77589,-0.288688 l -34.78736,-9.388036 a 8.4442361,8.4442361 0 0 1 -2.620433,-1.238044 z"
+       id="path6-0"
+       style="stroke-width:0.555177" />
+    <path
+       class="cls-1"
+       d="m 85.377935,159.27452 5.163143,-0.0333 h 0.06662 q 2.864711,0 2.864711,2.69816 v 8.69407 a 24.849705,24.849705 0 0 1 -8.649651,1.43235 q -4.730105,0 -7.128468,-3.21447 -2.398363,-3.21447 -2.398363,-8.76068 0,-5.55177 2.981299,-8.62745 a 9.7600046,9.7600046 0 0 1 7.29502,-3.08123 13.368653,13.368653 0 0 1 7.811335,2.43167 3.9250986,3.9250986 0 0 1 -0.682867,1.76547 4.7634152,4.7634152 0 0 1 -1.282458,1.33242 9.798867,9.798867 0 0 0 -5.679457,-1.96533 5.3574542,5.3574542 0 0 0 -4.480275,2.04861 q -1.598909,2.03749 -1.598909,6.41229 0,8.22771 6.062529,8.22771 a 16.910679,16.910679 0 0 0 3.697476,-0.43303 v -3.16451 q 0,-1.49898 0.06662,-2.22071 h -2.442777 a 2.2873276,2.2873276 0 0 1 -1.515632,-0.41638 1.6655298,1.6655298 0 0 1 -0.483004,-1.33242 5.7072154,5.7072154 0 0 1 0.333106,-1.79322 z"
+       id="path8-6"
+       style="fill:#4e4e4e;fill-opacity:1;stroke-width:0.555177" />
+    <path
+       class="cls-1"
+       d="m 111.07151,169.73404 a 4.3137222,4.3137222 0 0 1 -0.55518,1.18253 4.0305821,4.0305821 0 0 1 -0.84942,0.94935 3.7640973,3.7640973 0 0 1 -3.05902,-1.95422 6.7453957,6.7453957 0 0 1 -4.76342,2.13188 q -2.564913,0 -3.886233,-1.49898 a 5.1298318,5.1298318 0 0 1 -1.299113,-3.4643 q 0,-2.77588 1.815427,-4.21379 a 7.3338829,7.3338829 0 0 1 4.669039,-1.3935 q 1.53228,0 2.89802,0.13325 v -0.99932 q 0,-2.63154 -2.53161,-2.63154 -1.79877,0 -5.096518,1.19918 a 4.674587,4.674587 0 0 1 -1.110353,-2.96464 18.581761,18.581761 0 0 1 7.217291,-1.49898 5.8682167,5.8682167 0 0 1 4.0639,1.39905 q 1.56559,1.39904 1.56559,4.23044 v 6.79537 q -0.0111,1.83208 0.9216,2.59822 z m -8.36096,-0.83276 a 4.7134493,4.7134493 0 0 0 3.33106,-1.59891 v -2.94244 a 22.368065,22.368065 0 0 0 -2.53161,-0.13324 2.775883,2.775883 0 0 0 -2.06525,0.68842 2.3928111,2.3928111 0 0 0 -0.69953,1.76546 2.3539488,2.3539488 0 0 0 0.55518,1.66553 1.8431863,1.8431863 0 0 0 1.41015,0.55518 z"
+       id="path10-2"
+       style="fill:#4e4e4e;fill-opacity:1;stroke-width:0.555177" />
+    <path
+       class="cls-1"
+       d="m 113.76966,157.00939 a 3.986168,3.986168 0 0 1 0.55518,-1.21583 3.3310596,3.3310596 0 0 1 0.84942,-0.94935 4.1638245,4.1638245 0 0 1 3.51427,2.96464 q 1.33242,-2.96464 4.29707,-2.96464 a 10.215249,10.215249 0 0 1 1.93201,0.23317 7.4782288,7.4782288 0 0 1 -0.99932,3.88624 8.4497879,8.4497879 0 0 0 -1.49897,-0.19987 q -2.03195,0 -3.26444,2.16519 v 10.64829 a 11.575432,11.575432 0 0 1 -2.03195,0.16655 12.769062,12.769062 0 0 1 -2.09857,-0.16655 v -11.15905 q -0.0222,-2.40947 -1.2547,-3.40879 z"
+       id="path12-6"
+       style="fill:#4e4e4e;fill-opacity:1;stroke-width:0.555177" />
+    <path
+       class="cls-1"
+       d="m 140.38483,169.73404 a 4.3137222,4.3137222 0 0 1 -0.58293,1.18253 4.0305821,4.0305821 0 0 1 -0.84942,0.94935 3.7640973,3.7640973 0 0 1 -3.05348,-1.95422 6.7453957,6.7453957 0 0 1 -4.76341,2.13188 q -2.56492,0 -3.88624,-1.49898 a 5.1298318,5.1298318 0 0 1 -1.29911,-3.4643 q 0,-2.77588 1.81543,-4.21379 a 7.3338829,7.3338829 0 0 1 4.64682,-1.4157 q 1.53229,0 2.89803,0.13324 v -0.99932 q 0,-2.63153 -2.53161,-2.63153 -1.79877,0 -5.09652,1.19918 a 4.674587,4.674587 0 0 1 -1.11035,-2.96465 18.581761,18.581761 0 0 1 7.21729,-1.49897 5.8682167,5.8682167 0 0 1 4.0639,1.39904 q 1.56559,1.39905 1.56559,4.23045 v 6.81757 q 0.0333,1.83208 0.96601,2.59822 z m -8.37206,-0.83276 a 4.7134493,4.7134493 0 0 0 3.33106,-1.59891 v -2.94244 a 22.368065,22.368065 0 0 0 -2.53161,-0.13324 2.775883,2.775883 0 0 0 -2.06526,0.69952 2.3928111,2.3928111 0 0 0 -0.69952,1.76546 2.3539488,2.3539488 0 0 0 0.55518,1.66553 1.8431863,1.8431863 0 0 0 1.41015,0.54408 z"
+       id="path14-1"
+       style="fill:#4e4e4e;fill-opacity:1;stroke-width:0.555177" />
+    <path
+       class="cls-1"
+       d="m 144.48203,169.6008 q -1.49897,-2.29843 -1.49897,-6.34567 0,-4.04724 1.8987,-6.34567 a 5.740526,5.740526 0 0 1 4.56355,-2.29843 6.4400486,6.4400486 0 0 1 4.49693,1.66553 3.7696491,3.7696491 0 0 1 2.63154,-1.43235 3.1200925,3.1200925 0 0 1 0.88273,0.93269 3.8862362,3.8862362 0 0 1 0.55518,1.16587 q -0.9327,0.79946 -0.9327,2.86472 v 9.438 q 0,5.29638 -1.73215,7.49488 -1.73215,2.1985 -5.69611,2.22071 a 16.100121,16.100121 0 0 1 -5.9626,-1.11036 4.4802752,4.4802752 0 0 1 1.03263,-3.03126 10.892565,10.892565 0 0 0 4.48028,1.03263 q 2.18184,0 3.0146,-1.11035 a 4.9965894,4.9965894 0 0 0 0.83277,-3.06458 v -1.33242 a 6.4011862,6.4011862 0 0 1 -4.16383,1.56559 4.9188647,4.9188647 0 0 1 -4.40255,-2.30953 z m 8.56083,-2.69816 v -7.72806 a 4.2915151,4.2915151 0 0 0 -2.86471,-1.36573 2.4039147,2.4039147 0 0 0 -2.18185,1.43235 8.6885138,8.6885138 0 0 0 -0.7828,4.09721 q 0,2.66485 0.71618,3.93065 a 2.1318781,2.1318781 0 0 0 1.88205,1.2658 4.2304457,4.2304457 0 0 0 3.23113,-1.63222 z"
+       id="path16-8"
+       style="fill:#4e4e4e;fill-opacity:1;stroke-width:0.555177" />
+    <path
+       class="cls-1"
+       d="m 174.20619,164.67083 h -9.32697 a 5.6405943,5.6405943 0 0 0 0.88273,3.04792 q 0.7828,1.0826 2.74813,1.0826 a 10.120869,10.120869 0 0 0 4.36369,-1.16587 4.3803434,4.3803434 0 0 1 1.19918,2.5316 10.759323,10.759323 0 0 1 -6.41229,1.8987 q -3.74744,0 -5.37966,-2.43167 -1.63222,-2.43167 -1.63222,-6.2957 0,-3.88624 1.79877,-6.2957 a 6.0181143,6.0181143 0 0 1 5.14649,-2.43168 q 3.33106,0 5.14648,2.01529 a 7.3449864,7.3449864 0 0 1 1.79878,5.07987 13.04665,13.04665 0 0 1 -0.33311,2.96464 z m -6.42895,-7.06184 q -2.73146,0 -2.93133,4.13051 h 5.79605 v -0.39973 a 4.7245529,4.7245529 0 0 0 -0.69953,-2.69816 2.4316735,2.4316735 0 0 0 -2.14298,-1.03262 z"
+       id="path18-7"
+       style="fill:#4e4e4e;fill-opacity:1;stroke-width:0.555177" />
+    <path
+       id="path24-3-6-9"
+       style="fill:#ff9329;fill-opacity:1;stroke-width:0.555177"
+       d="m 124.80273,70.162462 a 11.0036,11.0036 0 0 0 -4.33203,0.935547 L 76.261719,90.656602 a 1.5989086,1.5989086 0 0 0 -0.837891,2.138672 0.77169547,0.77169547 0 0 0 0.06641,0.177735 l 7.09375,14.021481 h 6.15625 l -0.875,-4.88867 c -0.07217,-0.39418 -0.711263,-3.187532 -1.316406,-5.197264 l 20.691398,6.462894 c 0.27198,1.28839 0.63292,2.49204 1.0625,3.62304 h 33.54883 c 0.36964,-1.13128 0.66138,-2.33705 0.85938,-3.62304 l 20.64648,-6.445316 c -0.60514,2.009734 -1.23639,4.785506 -1.30859,5.179686 l -0.875,4.88867 h 6.15429 l 7.02735,-13.894528 0.0664,-0.126953 0.0684,-0.171875 a 0.10548355,0.10548355 0 0 0 0,-0.04492 1.4878733,1.4878733 0 0 0 0.0664,-0.515625 1.5822533,1.5822533 0 0 0 -0.99414,-1.583985 L 129.35352,71.098009 a 11.0036,11.0036 0 0 0 -4.55079,-0.935547 z" />
+    <path
+       id="path24-3-2"
+       style="fill:#4e4e4e;fill-opacity:1;stroke-width:0.555177"
+       d="M 124.80273,79.416133 A 11.0036,11.0036 0 0 0 120.4707,80.35168 L 76.261719,99.910272 a 1.5989086,1.5989086 0 0 0 -0.837891,2.136718 0.77169547,0.77169547 0 0 0 0.06641,0.17773 l 3.847657,7.60352 h 8.175781 c -0.257897,-1.08856 -0.591943,-2.42953 -0.964844,-3.66797 l 11.744141,3.66797 h 53.371087 l 11.69336,-3.65039 c -0.37193,1.23522 -0.70076,2.56719 -0.95703,3.65039 h 8.17383 l 3.78125,-7.47656 0.0664,-0.12696 0.0684,-0.17187 a 0.10548355,0.10548355 0 0 0 0,-0.0449 1.4878733,1.4878733 0 0 0 0.0664,-0.51563 1.5822533,1.5822533 0 0 0 -0.99414,-1.582028 L 129.35352,80.35168 a 11.0036,11.0036 0 0 0 -4.55079,-0.935547 z" />
+    <path
+       class="cls-2"
+       d="m 174.55595,110.92974 a 1.4878733,1.4878733 0 0 1 -0.0666,0.51631 0.10548355,0.10548355 0 0 1 0,0.0444 l -0.0666,0.17211 v 0 l -0.0666,0.12769 -10.69826,21.15223 c -1.48787,2.93688 -4.22489,2.84806 -3.76409,-0.12214 l 2.15408,-12.02512 c 0.0722,-0.39418 0.70508,-3.17006 1.31022,-5.1798 l -20.64702,6.4456 c -3.24223,21.05785 -30.95109,21.40761 -35.47023,0 l -20.691432,-6.46226 c 0.605143,2.00974 1.243596,4.80228 1.315769,5.19646 l 2.154085,12.02512 c 0.460796,2.9702 -2.276224,3.05902 -3.764098,0.12214 L 75.49024,111.66257 a 0.77169547,0.77169547 0 0 1 -0.06662,-0.17766 1.5989086,1.5989086 0 0 1 0.838317,-2.13743 L 120.47065,89.788613 a 11.0036,11.0036 0 0 1 8.88282,0 l 44.20871,19.558867 a 1.5822533,1.5822533 0 0 1 0.99377,1.58226 z"
+       id="path24-0"
+       style="fill:#ff9329;fill-opacity:1;stroke-width:0.555177" />
+    <path
+       class="cls-3"
+       d="m 139.0413,114.61611 19.11473,-7.69475 a 0.81055784,0.81055784 0 0 0 0,-1.50453 c -2.2207,-0.92714 -4.96328,-1.99308 -7.65033,-3.10899 -0.49411,-0.20541 -5.17425,3.15341 -5.60173,3.49762 l -8.23882,6.58439 c -1.99309,1.67108 -0.26649,3.28665 2.37615,2.22626 z"
+       id="path26-2"
+       style="fill:#4e4e4e;fill-opacity:1;stroke-width:0.555177" />
+    <circle
+       class="cls-3"
+       cx="125.18409"
+       cy="122.13319"
+       r="9.9654207"
+       id="circle28-3"
+       style="fill:#4e4e4e;fill-opacity:1;stroke-width:0.555177" />
+  </g>
+</svg>
@@ -1,119 +0,0 @@
-<?xml version="1.0" encoding="UTF-8" standalone="no"?>
-<svg
-   xmlns:dc="http://purl.org/dc/elements/1.1/"
-   xmlns:cc="http://creativecommons.org/ns#"
-   xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
-   xmlns:svg="http://www.w3.org/2000/svg"
-   xmlns="http://www.w3.org/2000/svg"
-   xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
-   xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
-   width="108.2099mm"
-   height="108.00987mm"
-   viewBox="0 0 108.2099 108.00987"
-   version="1.1"
-   id="svg8"
-   inkscape:version="1.0.1 (3bc2e813f5, 2020-09-07)"
-   sodipodi:docname="garage.svg"
-   inkscape:export-filename="/home/lx/garage.png"
-   inkscape:export-xdpi="96"
-   inkscape:export-ydpi="96">
-  <defs
-     id="defs2" />
-  <sodipodi:namedview
-     id="base"
-     pagecolor="#ffffff"
-     bordercolor="#666666"
-     borderopacity="1.0"
-     inkscape:pageopacity="1"
-     inkscape:pageshadow="2"
-     inkscape:zoom="0.5"
-     inkscape:cx="-212.52783"
-     inkscape:cy="204.9547"
-     inkscape:document-units="mm"
-     inkscape:current-layer="layer1"
-     inkscape:document-rotation="0"
-     showgrid="false"
-     fit-margin-top="20"
-     fit-margin-left="20"
-     fit-margin-right="20"
-     fit-margin-bottom="20"
-     inkscape:window-width="1404"
-     inkscape:window-height="1016"
-     inkscape:window-x="103"
-     inkscape:window-y="27"
-     inkscape:window-maximized="0" />
-  <metadata
-     id="metadata5">
-    <rdf:RDF>
-      <cc:Work
-         rdf:about="">
-        <dc:format>image/svg+xml</dc:format>
-        <dc:type
-           rdf:resource="http://purl.org/dc/dcmitype/StillImage" />
-        <dc:title></dc:title>
-      </cc:Work>
-    </rdf:RDF>
-  </metadata>
-  <g
-     inkscape:label="Layer 1"
-     inkscape:groupmode="layer"
-     id="layer1"
-     transform="translate(-45.667412,-33.028536)">
-    <path
-       style="fill:none;stroke:#000000;stroke-width:2.065;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
-       d="M 66.78016,73.340623 99.921832,54.219898 132.84481,73.130965 V 120.00591 H 66.701651 Z"
-       id="path124"
-       sodipodi:nodetypes="cccccc" />
-    <g
-       id="g1106-5"
-       transform="matrix(0,0.95201267,-0.95201267,0,194.01664,-65.058377)"
-       style="stroke-width:2.17959;stroke-miterlimit:4;stroke-dasharray:none">
-      <g
-         id="g1061-3"
-         style="stroke-width:2.17959;stroke-miterlimit:4;stroke-dasharray:none">
-        <circle
-           style="fill:none;stroke:#000000;stroke-width:2.17959;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
-           id="path956-5"
-           cx="168.8569"
-           cy="92.889587"
-           r="13.125794" />
-        <circle
-           style="fill:none;stroke:#000000;stroke-width:2.17959;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
-           id="path958-6"
-           cx="168.77444"
-           cy="92.702293"
-           r="3.0778286" />
-        <path
-           id="path960-2"
-           style="fill:none;stroke:#000000;stroke-width:2.17959;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
-           d="m 169.46072,82.84435 c 4.95795,0.336608 8.87296,4.341959 9.09638,9.306301"
-           sodipodi:nodetypes="cc" />
-      </g>
-      <path
-         style="fill:none;stroke:#000000;stroke-width:2.17959;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
-         d="m 154.67824,112.84018 11.89881,-13.038071 c 1.46407,-1.552664 3.79541,0.878511 2.81832,2.089181 l -10.57965,14.481 c -1.8851,2.02632 -6.10786,-1.06119 -4.13748,-3.53211 z"
-         id="path964-9"
-         sodipodi:nodetypes="ccccc" />
-      <g
-         id="g1071-1"
-         style="stroke-width:2.17959;stroke-miterlimit:4;stroke-dasharray:none" />
-      <g
-         id="g1065-3"
-         style="stroke-width:2.17959;stroke-miterlimit:4;stroke-dasharray:none">
-        <rect
-           style="fill:none;stroke:#000000;stroke-width:2.17959;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
-           id="rect949-6"
-           width="35.576611"
-           height="48.507355"
-           x="150.9623"
-           y="74.698929"
-           ry="2.7302756" />
-        <path
-           style="fill:none;stroke:#000000;stroke-width:2.17959;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
-           d="m 150.76919,106.16944 6.36181,-0.0223 c 2.53845,3.46232 6.29787,4.20243 10.1055,4.40362 l 0.0176,13.09251"
-           id="path1033-0"
-           sodipodi:nodetypes="cccc" />
-      </g>
-    </g>
-  </g>
-</svg>
@@ -1,5 +1,7 @@
 #!/bin/bash

+set -ex
+
 SCRIPT_FOLDER="`dirname \"$0\"`"
 REPO_FOLDER="${SCRIPT_FOLDER}/../"
 GARAGE_DEBUG="${REPO_FOLDER}/target/debug/"
@@ -9,7 +11,7 @@ PATH="${GARAGE_DEBUG}:${GARAGE_RELEASE}:$PATH"
 garage bucket create eprouvette
 KEY_INFO=`garage key new --name opérateur`
 ACCESS_KEY=`echo $KEY_INFO|grep -Po 'GK[a-f0-9]+'`
-SECRET_KEY=`echo $KEY_INFO|grep -Po 'secret_key: "[a-f0-9]+'|grep -Po '[a-f0-9]+$'`
+SECRET_KEY=`echo $KEY_INFO|grep -Po 'Secret key: [a-f0-9]+'|grep -Po '[a-f0-9]+$'`
 garage bucket allow eprouvette --read --write --key $ACCESS_KEY
 echo "$ACCESS_KEY $SECRET_KEY" > /tmp/garage.s3

@@ -24,11 +24,11 @@ cat > $CONF_PATH <<EOF
 block_size = 1048576			# objects are split in blocks of maximum this number of bytes
 metadata_dir = "/tmp/garage-meta-$count"
 data_dir = "/tmp/garage-data-$count"
-rpc_bind_addr = "[::]:$((3900+$count))"		# the port other Garage nodes will use to talk to this node
+rpc_bind_addr = "0.0.0.0:$((3900+$count))"		# the port other Garage nodes will use to talk to this node
 bootstrap_peers = [
-  "[::1]:3901",
-  "[::1]:3902",
-  "[::1]:3903"
+  "127.0.0.1:3901",
+  "127.0.0.1:3902",
+  "127.0.0.1:3903"
 ]
 max_concurrent_rpc_requests = 12
 data_replication_factor = 3
@@ -36,11 +36,11 @@ meta_replication_factor = 3
 meta_epidemic_fanout = 3

 [s3_api]
-api_bind_addr = "[::]:$((3910+$count))"	# the S3 API port, HTTP without TLS. Add a reverse proxy for the TLS part.
+api_bind_addr = "0.0.0.0:$((3910+$count))"	# the S3 API port, HTTP without TLS. Add a reverse proxy for the TLS part.
 s3_region = "garage"				# set this to anything. S3 API calls will fail if they are not made against the region set here.

 [s3_web]
-bind_addr = "[::]:$((3920+$count))"
+bind_addr = "0.0.0.0:$((3920+$count))"
 root_domain = ".garage.tld"
 index = "index.html"
 EOF
@@ -1,11 +1,14 @@
 #!/bin/bash

+set -ex
+
 SCRIPT_FOLDER="`dirname \"$0\"`"
 REPO_FOLDER="${SCRIPT_FOLDER}/../"
 GARAGE_DEBUG="${REPO_FOLDER}/target/debug/"
 GARAGE_RELEASE="${REPO_FOLDER}/target/release/"
 PATH="${GARAGE_DEBUG}:${GARAGE_RELEASE}:$PATH"

+sleep 5
 until garage status 2>&1|grep -q Healthy ; do 
 	echo "cluster starting..."
 	sleep 1
@@ -15,6 +18,6 @@ garage status \
 	| grep UNCONFIGURED \
 	| grep -Po '^[0-9a-f]+' \
 	| while read id; do 
-	  garage node configure -d dc1 -n 100 $id
+	  garage node configure -d dc1 -c 1 $id
 	done

@@ -0,0 +1,290 @@
+#!/usr/bin/env python3
+
+import hashlib
+import bisect
+import xxhash
+import numpy as np
+
+REPLICATION_FACTOR = 3
+
+def hash_str(s):
+    xxh = xxhash.xxh64()
+    xxh.update(s.encode('ascii'))
+    return xxh.hexdigest()
+
+def sha256_str(s):
+    return hashlib.sha256(s.encode('ascii')).hexdigest()
+
+def walk_ring_from_pos(tokens, dcs, start):
+    ret = []
+    ret_dcs = set()
+    delta = 0
+    while len(ret) < REPLICATION_FACTOR:
+        i = (start + delta) % len(tokens)
+        delta = delta + 1
+
+        (token_k, token_dc, token_node) = tokens[i]
+        if token_dc not in ret_dcs:
+            ret_dcs |= set([token_dc])
+            ret.append(token_node)
+        elif len(ret_dcs) == len(dcs) and token_node not in ret:
+            ret.append(token_node)
+    return ret
+
+"""
+def count_tokens_per_node(tokens):
+    tokens_of_node = {}
+    for _, _, token_node in tokens:
+        if token_node not in tokens_of_node:
+            tokens_of_node[token_node] = 0
+        tokens_of_node[token_node] += 1
+    print("#tokens per node:")
+    for node, ntok in sorted(list(tokens_of_node.items())):
+        print("-", node, ": ", ntok)
+"""
+
+def count_partitions_per_node(ring_node_list):
+    tokens_of_node = {}
+    for nodelist in ring_node_list:
+        for node_id in nodelist:
+            if node_id not in tokens_of_node:
+                tokens_of_node[node_id] = 0
+            tokens_of_node[node_id] += 1
+    print("#partitions per node:")
+    for node, ntok in sorted(list(tokens_of_node.items())):
+        print("-", node, ": ", ntok)
+
+
+def method1(nodes):
+    tokens = []
+    dcs = set()
+    for (node_id, dc, n_tokens) in nodes:
+        dcs |= set([dc])
+        for i in range(n_tokens):
+            token = hash_str(f"{node_id} {i}")
+            tokens.append((token, dc, node_id))
+    tokens.sort(key=lambda tok: tok[0])
+
+    space_of_node = {}
+
+    def walk_ring(v):
+        i = bisect.bisect_left([tok for tok, _, _ in tokens], hash_str(v))
+        return walk_ring_from_pos(tokens, dcs, i)
+
+    ring_node_list = [walk_ring_from_pos(tokens, dcs, i) for i in range(len(tokens))]
+
+    return walk_ring, ring_node_list
+
+
+def method2(nodes):
+    partition_bits = 10
+    partitions = list(range(2**partition_bits))
+    def partition_node(i):
+        h, hn, hndc = None, None, None
+        for (node_id, node_dc, n_tokens) in nodes:
+            for tok in range(n_tokens):
+                hnode = hash_str(f"partition {i} node {node_id} token {tok}")
+                if h is None or hnode < h:
+                    h = hnode
+                    hn = node_id
+                    hndc = node_dc
+        return (i, hndc, hn)
+
+    partition_nodes = [partition_node(i) for i in partitions]
+
+    dcs = list(set(node_dc for _, node_dc, _ in nodes))
+
+        
+    def walk_ring(v):
+        # xxh = xxhash.xxh32()
+        # xxh.update(v.encode('ascii'))
+        # vh = xxh.intdigest()
+        # i = vh % (2**partition_bits)
+        vh = hashlib.sha256(v.encode('ascii')).digest()
+        i = (vh[0]<<8 | vh[1]) % (2**partition_bits)
+        return walk_ring_from_pos(partition_nodes, dcs, i)
+
+    ring_node_list = [walk_ring_from_pos(partition_nodes, dcs, i) for i in range(len(partition_nodes))]
+
+    return walk_ring, ring_node_list
+
+
+def method3(nodes):
+    partition_bits = 10
+
+    queues = []
+    for (node_id, node_dc, n_tokens) in nodes:
+        que = [(i, hash_str(f"{node_id} {i}")) for i in range(2**partition_bits)]
+        que.sort(key=lambda x: x[1])
+        que = [x[0] for x in que]
+        queues.append((node_id, node_dc, n_tokens, que))
+
+    partitions = [None for _ in range(2**partition_bits)]
+    queues.sort(key=lambda x: hash_str(x[0]))
+
+    # Maglev
+    remaining = 2**partition_bits
+    while remaining > 0:
+        for toktok in range(100):
+            for iq in range(len(queues)):
+                node_id, node_dc, n_tokens, node_queue = queues[iq]
+                if toktok >= n_tokens:
+                    continue
+                for qi, qv in enumerate(node_queue):
+                    if partitions[qv] == None:
+                        partitions[qv] = (qv, node_dc, node_id)
+                        remaining -= 1
+                        queues[iq] = (node_id, node_dc, n_tokens, node_queue[qi+1:])
+                        break
+
+    dcs = list(set(node_dc for _, node_dc, _ in nodes))
+
+    def walk_ring(v):
+        vh = hashlib.sha256(v.encode('ascii')).digest()
+        i = (vh[0]<<8 | vh[1]) % (2**partition_bits)
+        return walk_ring_from_pos(partitions, dcs, i)
+
+    ring_node_list = [walk_ring_from_pos(partitions, dcs, i) for i in range(len(partitions))]
+
+    return walk_ring, ring_node_list
+
+
+
+def method4(nodes):
+    partition_bits = 10
+
+    partitions = [[] for _ in range(2**partition_bits)]
+    dcs = list(set(node_dc for _, node_dc, _ in nodes))
+
+    # Maglev, improved for several replicas on several DCs
+    for ri in range(REPLICATION_FACTOR):
+        queues = []
+        for (node_id, node_dc, n_tokens) in nodes:
+            que = [(i, hash_str(f"{node_id} {i}")) for i in range(2**partition_bits)]
+            que.sort(key=lambda x: x[1])
+            que = [x[0] for x in que]
+            queues.append((node_id, node_dc, n_tokens, que))
+
+        queues.sort(key=lambda x: hash_str("{} {}".format(ri, x[0])))
+
+        remaining = 2**partition_bits
+        while remaining > 0:
+            for toktok in range(100):
+                for iq in range(len(queues)):
+                    node_id, node_dc, n_tokens, node_queue = queues[iq]
+                    if toktok >= n_tokens:
+                        continue
+                    for qi, qv in enumerate(node_queue):
+                        if len(partitions[qv]) != ri:
+                            continue
+                        p_dcs = set([x[0] for x in partitions[qv]])
+                        p_nodes = [x[1] for x in partitions[qv]]
+                        if node_dc not in p_dcs or (len(p_dcs) == len(dcs) and node_id not in p_nodes):
+                            partitions[qv].append((node_dc, node_id))
+                            remaining -= 1
+                            queues[iq] = (node_id, node_dc, n_tokens, node_queue[qi+1:])
+                            break
+
+    def walk_ring(v):
+        vh = hashlib.sha256(v.encode('ascii')).digest()
+        i = (vh[0]<<8 | vh[1]) % (2**partition_bits)
+        assert len(set([node_dc for node_dc, _ in partitions[i]])) == min(REPLICATION_FACTOR, len(dcs))
+        return [node_id for _, node_id in partitions[i]]
+
+    ring_node_list = [[node_id for _, node_id in p] for p in partitions]
+
+    return walk_ring, ring_node_list
+
+def evaluate_method(method, nodes):
+    walk_ring, ring_node_list = method(nodes)
+    print("Ring length:", len(ring_node_list))
+    count_partitions_per_node(ring_node_list)
+
+    print("Number of data items per node (100000 simulation):")
+    node_data_counts = {}
+    for i in range(100000):
+        inodes = walk_ring(f"{i}")
+        for n in inodes:
+            if n not in node_data_counts:
+                node_data_counts[n] = 0
+            node_data_counts[n] += 1
+    for n, v in sorted(list(node_data_counts.items())):
+        print("-", n, ": ", v)
+
+    dclist_per_ntok = {}
+    for node_id, _, ntok in nodes:
+        if ntok not in dclist_per_ntok:
+            dclist_per_ntok[ntok] = []
+        dclist_per_ntok[ntok].append(node_data_counts[node_id])
+    list_normalized = []
+    for ntok, dclist in dclist_per_ntok.items():
+        avg = sum(dclist)/len(dclist)
+        for v in dclist:
+            list_normalized.append(v / avg)
+    print("variance wrt. same-ntok mean:", "%.2f%%"%(np.var(list_normalized)*100))
+
+    num_changes_sum = [0, 0, 0, 0]
+    for n in nodes:
+        nodes2 = [nn for nn in nodes if nn != n]
+        _, ring_node_list_2 = method(nodes2)
+        if len(ring_node_list_2) != len(ring_node_list):
+            continue
+        num_changes = [0, 0, 0, 0] 
+        for (ns1, ns2) in zip(ring_node_list, ring_node_list_2):
+            changes = sum(1 for x in ns1 if x not in ns2)
+            num_changes[changes] += 1
+        for i, v in enumerate(num_changes):
+            num_changes_sum[i] += v / len(ring_node_list)
+        print("removing", n[1], n[0], ":", " ".join(["%.2f%%"%(x*100/len(ring_node_list)) for x in num_changes]))
+    print("1-node removal: partitions moved on 0/1/2/3 nodes: ", " ".join(["%.2f%%"%(x*100/len(nodes)) for x in num_changes_sum]))
+
+
+if __name__ == "__main__":
+    print("------")
+    print("method 1 (standard ring)")
+    nodes = [('digitale', 'atuin', 64),
+             ('drosera', 'atuin', 64),
+             ('datura', 'atuin', 64),
+             ('io', 'jupiter', 128)]
+    nodes2 = [('digitale', 'atuin', 64),
+             ('drosera', 'atuin', 64),
+             ('datura', 'atuin', 64),
+             ('io', 'jupiter', 128),
+             ('isou', 'jupiter', 64),
+             ('mini', 'grog', 32),
+             ('mixi', 'grog', 32),
+             ('moxi', 'grog', 32),
+             ('modi', 'grog', 32),
+             ('geant', 'grisou', 128),
+             ('gipsie', 'grisou', 128),
+             ]
+    evaluate_method(method1, nodes2)
+
+    print("------")
+    print("method 2 (custom ring)")
+    nodes = [('digitale', 'atuin', 1),
+             ('drosera', 'atuin', 1),
+             ('datura', 'atuin', 1),
+             ('io', 'jupiter', 2)]
+    nodes2 = [('digitale', 'atuin', 2),
+             ('drosera', 'atuin', 2),
+             ('datura', 'atuin', 2),
+             ('io', 'jupiter', 4),
+             ('isou', 'jupiter', 2),
+             ('mini', 'grog', 1),
+             ('mixi', 'grog', 1),
+             ('moxi', 'grog', 1),
+             ('modi', 'grog', 1),
+             ('geant', 'grisou', 4),
+             ('gipsie', 'grisou', 4),
+             ]
+    evaluate_method(method2, nodes2)
+
+    print("------")
+    print("method 3 (maglev)")
+    evaluate_method(method3, nodes2)
+
+
+    print("------")
+    print("method 4 (maglev, multi-dc twist)")
+    evaluate_method(method4, nodes2)
@@ -1,9 +1,9 @@
 [package]
 name = "garage_api"
-version = "0.1.1"
+version = "0.2.1"
 authors = ["Alex Auvolat <alex@adnab.me>"]
 edition = "2018"
-license = "GPL-3.0"
+license = "AGPL-3.0"
 description = "S3 API server crate for the Garage object store"
 repository = "https://git.deuxfleurs.fr/Deuxfleurs/garage"

@@ -13,30 +13,29 @@ path = "lib.rs"
 # See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

 [dependencies]
-garage_util = { version = "0.1.1", path = "../util" }
-garage_table = { version = "0.1.1", path = "../table" }
-garage_model = { version = "0.1.1", path = "../model" }
+garage_model = { version = "0.2.1", path = "../model" }
+garage_table = { version = "0.2.1", path = "../table" }
+garage_util = { version = "0.2.1", path = "../util" }

-err-derive = "0.2.3"
-bytes = "0.4"
-hex = "0.3"
 base64 = "0.13"
-log = "0.4"
+bytes = "1.0"
 chrono = "0.4"
-md-5 = "0.9.1"
-sha2 = "0.8"
-hmac = "0.7"
-crypto-mac = "0.7"
-rand = "0.7"
+crypto-mac = "0.10"
+err-derive = "0.3"
+hex = "0.4"
+hmac = "0.10"
+log = "0.4"
+md-5 = "0.9"
+sha2 = "0.9"

 futures = "0.3"
 futures-util = "0.3"
-tokio = { version = "0.2", default-features = false, features = ["rt-core", "rt-threaded", "io-driver", "net", "tcp", "time", "macros", "sync", "signal", "fs"] }
+tokio = { version = "1.0", default-features = false, features = ["rt", "rt-multi-thread", "io-util", "net", "time", "macros", "sync", "signal", "fs"] }

 http = "0.2"
-hyper = "^0.13.6"
-url = "2.1"
 httpdate = "0.3"
-percent-encoding = "2.1.0"
-roxmltree = "0.11"
 http-range = "0.1"
+hyper = "0.14"
+percent-encoding = "2.1.0"
+roxmltree = "0.14"
+url = "2.1"
@@ -62,7 +62,11 @@ async fn handler(
 			let body: Body = Body::from(format!("{}\n", e));
 			let mut http_error = Response::new(body);
 			*http_error.status_mut() = e.http_status_code();
-			warn!("Response: error {}, {}", e.http_status_code(), e);
+			if e.http_status_code().is_server_error() {
+				warn!("Response: error {}, {}", e.http_status_code(), e);
+			} else {
+				info!("Response: error {}, {}", e.http_status_code(), e);
+			}
 			Ok(http_error)
 		}
 	}
@@ -97,7 +101,7 @@ async fn handler_inner(garage: Arc<Garage>, req: Request<Body>) -> Result<Respon
 		match req.method() {
 			&Method::HEAD => {
 				// HeadObject query
-				Ok(handle_head(garage, &bucket, &key).await?)
+				Ok(handle_head(garage, &req, &bucket, &key).await?)
 			}
 			&Method::GET => {
 				// GetObject query
@@ -133,7 +137,10 @@ async fn handler_inner(garage: Arc<Garage>, req: Request<Body>) -> Result<Respon
 						)));
 					}
 					let source_key = source_key.ok_or_bad_request("No source key specified")?;
-					Ok(handle_copy(garage, &bucket, &key, &source_bucket, &source_key).await?)
+					Ok(
+						handle_copy(garage, &req, &bucket, &key, &source_bucket, &source_key)
+							.await?,
+					)
 				} else {
 					// PutObject query
 					Ok(handle_put(garage, req, &bucket, &key, content_sha256).await?)
@@ -156,10 +163,15 @@ async fn handler_inner(garage: Arc<Garage>, req: Request<Body>) -> Result<Respon
 				} else if params.contains_key(&"uploadid".to_string()) {
 					// CompleteMultipartUpload call
 					let upload_id = params.get("uploadid").unwrap();
-					Ok(
-						handle_complete_multipart_upload(garage, req, &bucket, &key, upload_id)
-							.await?,
+					Ok(handle_complete_multipart_upload(
+						garage,
+						req,
+						&bucket,
+						&key,
+						upload_id,
+						content_sha256,
 					)
+					.await?)
 				} else {
 					Err(Error::BadRequest(format!(
 						"Not a CreateMultipartUpload call, what is it?"
@@ -198,36 +210,14 @@ async fn handler_inner(garage: Arc<Garage>, req: Request<Body>) -> Result<Respon
 				))
 			}
 			&Method::GET => {
-				// ListObjects query
-				let delimiter = params.get("delimiter").map(|x| x.as_str()).unwrap_or(&"");
-				let max_keys = params
-					.get("max-keys")
-					.map(|x| {
-						x.parse::<usize>()
-							.ok_or_bad_request("Invalid value for max-keys")
-					})
-					.unwrap_or(Ok(1000))?;
-				let prefix = params.get("prefix").map(|x| x.as_str()).unwrap_or(&"");
-				let urlencode_resp = params
-					.get("encoding-type")
-					.map(|x| x == "url")
-					.unwrap_or(false);
-				let marker = params.get("marker").map(String::as_str);
-				Ok(handle_list(
-					garage,
-					bucket,
-					delimiter,
-					max_keys,
-					prefix,
-					marker,
-					urlencode_resp,
-				)
-				.await?)
+				// ListObjects or ListObjectsV2 query
+				let q = parse_list_objects_query(bucket, &params)?;
+				Ok(handle_list(garage, &q).await?)
 			}
 			&Method::POST => {
 				if params.contains_key(&"delete".to_string()) {
 					// DeleteObjects
-					Ok(handle_delete_objects(garage, bucket, req).await?)
+					Ok(handle_delete_objects(garage, bucket, req, content_sha256).await?)
 				} else {
 					debug!(
 						"Body: {}",
@@ -24,10 +24,16 @@ pub enum Error {

 	// Category: bad request
 	#[error(display = "Invalid UTF-8: {}", _0)]
-	InvalidUTF8(#[error(source)] std::str::Utf8Error),
+	InvalidUTF8Str(#[error(source)] std::str::Utf8Error),
+
+	#[error(display = "Invalid UTF-8: {}", _0)]
+	InvalidUTF8String(#[error(source)] std::string::FromUtf8Error),
+
+	#[error(display = "Invalid base64: {}", _0)]
+	InvalidBase64(#[error(source)] base64::DecodeError),

 	#[error(display = "Invalid XML: {}", _0)]
-	InvalidXML(#[error(source)] roxmltree::Error),
+	InvalidXML(String),

 	#[error(display = "Invalid header value: {}", _0)]
 	InvalidHeader(#[error(source)] hyper::header::ToStrError),
@@ -39,6 +45,12 @@ pub enum Error {
 	BadRequest(String),
 }

+impl From<roxmltree::Error> for Error {
+	fn from(err: roxmltree::Error) -> Self {
+		Self::InvalidXML(format!("{}", err))
+	}
+}
+
 impl Error {
 	pub fn http_status_code(&self) -> StatusCode {
 		match self {
@@ -1,11 +1,11 @@
 use std::fmt::Write;
 use std::sync::Arc;

-use chrono::{SecondsFormat, Utc};
-use hyper::{Body, Response};
+use hyper::{Body, Request, Response};

 use garage_table::*;
 use garage_util::data::*;
+use garage_util::time::*;

 use garage_model::block_ref_table::*;
 use garage_model::garage::Garage;
@@ -13,9 +13,11 @@ use garage_model::object_table::*;
 use garage_model::version_table::*;

 use crate::error::*;
+use crate::s3_put::get_headers;

 pub async fn handle_copy(
 	garage: Arc<Garage>,
+	req: &Request<Body>,
 	dest_bucket: &str,
 	dest_key: &str,
 	source_bucket: &str,
@@ -41,17 +43,37 @@ pub async fn handle_copy(
 	};

 	let new_uuid = gen_uuid();
-	let dest_object_version = ObjectVersion {
-		uuid: new_uuid,
-		timestamp: now_msec(),
-		state: ObjectVersionState::Complete(source_last_state.clone()),
-	};
+	let new_timestamp = now_msec();

-	match &source_last_state {
+	// Implement x-amz-metadata-directive: REPLACE
+	let old_meta = match source_last_state {
 		ObjectVersionData::DeleteMarker => {
 			return Err(Error::NotFound);
 		}
-		ObjectVersionData::Inline(_meta, _bytes) => {
+		ObjectVersionData::Inline(meta, _bytes) => meta,
+		ObjectVersionData::FirstBlock(meta, _fbh) => meta,
+	};
+	let new_meta = match req.headers().get("x-amz-metadata-directive") {
+		Some(v) if v == hyper::header::HeaderValue::from_static("REPLACE") => ObjectVersionMeta {
+			headers: get_headers(req)?,
+			size: old_meta.size,
+			etag: old_meta.etag.clone(),
+		},
+		_ => old_meta.clone(),
+	};
+
+	// Save object copy
+	match source_last_state {
+		ObjectVersionData::DeleteMarker => unreachable!(),
+		ObjectVersionData::Inline(_meta, bytes) => {
+			let dest_object_version = ObjectVersion {
+				uuid: new_uuid,
+				timestamp: new_timestamp,
+				state: ObjectVersionState::Complete(ObjectVersionData::Inline(
+					new_meta,
+					bytes.clone(),
+				)),
+			};
 			let dest_object = Object::new(
 				dest_bucket.to_string(),
 				dest_key.to_string(),
@@ -59,49 +81,91 @@ pub async fn handle_copy(
 			);
 			garage.object_table.insert(&dest_object).await?;
 		}
-		ObjectVersionData::FirstBlock(_meta, _first_block_hash) => {
+		ObjectVersionData::FirstBlock(_meta, first_block_hash) => {
+			// Get block list from source version
 			let source_version = garage
 				.version_table
 				.get(&source_last_v.uuid, &EmptyKey)
 				.await?;
 			let source_version = source_version.ok_or(Error::NotFound)?;

-			let dest_version = Version::new(
+			// Write an "uploading" marker in Object table
+			// This holds a reference to the object in the Version table
+			// so that it won't be deleted, e.g. by repair_versions.
+			let tmp_dest_object_version = ObjectVersion {
+				uuid: new_uuid,
+				timestamp: new_timestamp,
+				state: ObjectVersionState::Uploading(new_meta.headers.clone()),
+			};
+			let tmp_dest_object = Object::new(
+				dest_bucket.to_string(),
+				dest_key.to_string(),
+				vec![tmp_dest_object_version],
+			);
+			garage.object_table.insert(&tmp_dest_object).await?;
+
+			// Write version in the version table. Even with empty block list,
+			// this means that the BlockRef entries linked to this version cannot be
+			// marked as deleted (they are marked as deleted only if the Version
+			// doesn't exist or is marked as deleted).
+			let mut dest_version = Version::new(
 				new_uuid,
 				dest_bucket.to_string(),
 				dest_key.to_string(),
 				false,
-				source_version.blocks().to_vec(),
 			);
+			garage.version_table.insert(&dest_version).await?;
+
+			// Fill in block list for version and insert block refs
+			for (bk, bv) in source_version.blocks.items().iter() {
+				dest_version.blocks.put(*bk, *bv);
+			}
+			let dest_block_refs = dest_version
+				.blocks
+				.items()
+				.iter()
+				.map(|b| BlockRef {
+					block: b.1.hash,
+					version: new_uuid,
+					deleted: false.into(),
+				})
+				.collect::<Vec<_>>();
+			futures::try_join!(
+				garage.version_table.insert(&dest_version),
+				garage.block_ref_table.insert_many(&dest_block_refs[..]),
+			)?;
+
+			// Insert final object
+			// We do this last because otherwise there is a race condition in the case where
+			// the copy call has the same source and destination (this happens, rclone does
+			// it to update the modification timestamp for instance). If we did this concurrently
+			// with the stuff before, the block's reference counts could be decremented before
+			// they are incremented again for the new version, leading to data being deleted.
+			let dest_object_version = ObjectVersion {
+				uuid: new_uuid,
+				timestamp: new_timestamp,
+				state: ObjectVersionState::Complete(ObjectVersionData::FirstBlock(
+					new_meta,
+					*first_block_hash,
+				)),
+			};
 			let dest_object = Object::new(
 				dest_bucket.to_string(),
 				dest_key.to_string(),
 				vec![dest_object_version],
 			);
-			let dest_block_refs = dest_version
-				.blocks()
-				.iter()
-				.map(|b| BlockRef {
-					block: b.hash,
-					version: new_uuid,
-					deleted: false,
-				})
-				.collect::<Vec<_>>();
-			futures::try_join!(
-				garage.object_table.insert(&dest_object),
-				garage.version_table.insert(&dest_version),
-				garage.block_ref_table.insert_many(&dest_block_refs[..]),
-			)?;
+			garage.object_table.insert(&dest_object).await?;
 		}
 	}

-	let now = Utc::now();
-	let last_modified = now.to_rfc3339_opts(SecondsFormat::Secs, true);
+	let last_modified = msec_to_rfc3339(new_timestamp);
 	let mut xml = String::new();
 	writeln!(&mut xml, r#"<?xml version="1.0" encoding="UTF-8"?>"#).unwrap();
 	writeln!(&mut xml, r#"<CopyObjectResult>"#).unwrap();
 	writeln!(&mut xml, "\t<LastModified>{}</LastModified>", last_modified).unwrap();
 	writeln!(&mut xml, "</CopyObjectResult>").unwrap();

-	Ok(Response::new(Body::from(xml.into_bytes())))
+	Ok(Response::builder()
+		.header("Content-Type", "application/xml")
+		.body(Body::from(xml.into_bytes()))?)
 }
@@ -4,12 +4,14 @@ use std::sync::Arc;
 use hyper::{Body, Request, Response};

 use garage_util::data::*;
+use garage_util::time::*;

 use garage_model::garage::Garage;
 use garage_model::object_table::*;

 use crate::encoding::*;
 use crate::error::*;
+use crate::signature::verify_signed_content;

 async fn handle_delete_internal(
 	garage: &Garage,
@@ -28,16 +30,16 @@ async fn handle_delete_internal(
 		_ => true,
 	});

-	let mut must_delete = None;
+	let mut version_to_delete = None;
 	let mut timestamp = now_msec();
 	for v in interesting_versions {
-		if v.timestamp + 1 > timestamp || must_delete.is_none() {
-			must_delete = Some(v.uuid);
+		if v.timestamp + 1 > timestamp || version_to_delete.is_none() {
+			version_to_delete = Some(v.uuid);
 		}
 		timestamp = std::cmp::max(timestamp, v.timestamp + 1);
 	}

-	let deleted_version = must_delete.ok_or(Error::NotFound)?;
+	let deleted_version = version_to_delete.ok_or(Error::NotFound)?;

 	let version_uuid = gen_uuid();

@@ -46,7 +48,7 @@ async fn handle_delete_internal(
 		key.into(),
 		vec![ObjectVersion {
 			uuid: version_uuid,
-			timestamp: now_msec(),
+			timestamp,
 			state: ObjectVersionState::Complete(ObjectVersionData::DeleteMarker),
 		}],
 	);
@@ -73,8 +75,11 @@ pub async fn handle_delete_objects(
 	garage: Arc<Garage>,
 	bucket: &str,
 	req: Request<Body>,
+	content_sha256: Option<Hash>,
 ) -> Result<Response<Body>, Error> {
 	let body = hyper::body::to_bytes(req.into_body()).await?;
+	verify_signed_content(content_sha256, &body[..])?;
+
 	let cmd_xml = roxmltree::Document::parse(&std::str::from_utf8(&body)?)?;
 	let cmd = parse_delete_objects_xml(&cmd_xml).ok_or_bad_request("Invalid delete XML query")?;

@@ -86,7 +91,7 @@ pub async fn handle_delete_objects(
 		match handle_delete_internal(&garage, bucket, &obj.key).await {
 			Ok((deleted_version, delete_marker_version)) => {
 				writeln!(&mut retxml, "\t<Deleted>").unwrap();
-				writeln!(&mut retxml, "\t\t<Key>{}</Key>", obj.key).unwrap();
+				writeln!(&mut retxml, "\t\t<Key>{}</Key>", xml_escape(&obj.key)).unwrap();
 				writeln!(
 					&mut retxml,
 					"\t\t<VersionId>{}</VersionId>",
@@ -104,7 +109,7 @@ pub async fn handle_delete_objects(
 			Err(e) => {
 				writeln!(&mut retxml, "\t<Error>").unwrap();
 				writeln!(&mut retxml, "\t\t<Code>{}</Code>", e.http_status_code()).unwrap();
-				writeln!(&mut retxml, "\t\t<Key>{}</Key>", obj.key).unwrap();
+				writeln!(&mut retxml, "\t\t<Key>{}</Key>", xml_escape(&obj.key)).unwrap();
 				writeln!(
 					&mut retxml,
 					"\t\t<Message>{}</Message>",
@@ -118,7 +123,9 @@ pub async fn handle_delete_objects(

 	writeln!(&mut retxml, "</DeleteObjectsOutput>").unwrap();

-	Ok(Response::new(Body::from(retxml.into_bytes())))
+	Ok(Response::builder()
+		.header("Content-Type", "application/xml")
+		.body(Body::from(retxml.into_bytes()))?)
 }

 struct DeleteRequest {
@@ -129,33 +136,27 @@ struct DeleteObject {
 	key: String,
 }

-fn parse_delete_objects_xml(xml: &roxmltree::Document) -> Result<DeleteRequest, String> {
+fn parse_delete_objects_xml(xml: &roxmltree::Document) -> Option<DeleteRequest> {
 	let mut ret = DeleteRequest { objects: vec![] };

 	let root = xml.root();
-	let delete = root.first_child().ok_or(format!("Delete tag not found"))?;
+	let delete = root.first_child()?;

 	if !delete.has_tag_name("Delete") {
-		return Err(format!("Invalid root tag: {:?}", root));
+		return None;
 	}

 	for item in delete.children() {
 		if item.has_tag_name("Object") {
-			if let Some(key) = item.children().find(|e| e.has_tag_name("Key")) {
-				if let Some(key_str) = key.text() {
-					ret.objects.push(DeleteObject {
-						key: key_str.to_string(),
-					});
-				} else {
-					return Err(format!("No text for key: {:?}", key));
-				}
-			} else {
-				return Err(format!("No delete key for item: {:?}", item));
-			}
+			let key = item.children().find(|e| e.has_tag_name("Key"))?;
+			let key_str = key.text()?;
+			ret.objects.push(DeleteObject {
+				key: key_str.to_string(),
+			});
 		} else {
-			return Err(format!("Invalid delete item: {:?}", item));
+			return None;
 		}
 	}

-	Ok(ret)
+	Some(ret)
 }
@@ -16,6 +16,8 @@ fn object_headers(
 	version: &ObjectVersion,
 	version_meta: &ObjectVersionMeta,
 ) -> http::response::Builder {
+	debug!("Version meta: {:?}", version_meta);
+
 	let date = UNIX_EPOCH + Duration::from_millis(version.timestamp);
 	let date_str = httpdate::fmt_http_date(date);

@@ -38,8 +40,48 @@ fn object_headers(
 	resp
 }

+fn try_answer_cached(
+	version: &ObjectVersion,
+	version_meta: &ObjectVersionMeta,
+	req: &Request<Body>,
+) -> Option<Response<Body>> {
+	// <trinity> It is possible, and is even usually the case, [that both If-None-Match and
+	// If-Modified-Since] are present in a request. In this situation If-None-Match takes
+	// precedence and If-Modified-Since is ignored (as per 6.Precedence from rfc7232). The rational
+	// being that etag based matching is more accurate, it has no issue with sub-second precision
+	// for instance (in case of very fast updates)
+	let cached = if let Some(none_match) = req.headers().get(http::header::IF_NONE_MATCH) {
+		let none_match = none_match.to_str().ok()?;
+		let expected = format!("\"{}\"", version_meta.etag);
+		let found = none_match
+			.split(',')
+			.map(str::trim)
+			.any(|etag| etag == expected || etag == "\"*\"");
+		found
+	} else if let Some(modified_since) = req.headers().get(http::header::IF_MODIFIED_SINCE) {
+		let modified_since = modified_since.to_str().ok()?;
+		let client_date = httpdate::parse_http_date(modified_since).ok()?;
+		let server_date = UNIX_EPOCH + Duration::from_millis(version.timestamp);
+		client_date >= server_date
+	} else {
+		false
+	};
+
+	if cached {
+		Some(
+			Response::builder()
+				.status(StatusCode::NOT_MODIFIED)
+				.body(Body::empty())
+				.unwrap(),
+		)
+	} else {
+		None
+	}
+}
+
 pub async fn handle_head(
 	garage: Arc<Garage>,
+	req: &Request<Body>,
 	bucket: &str,
 	key: &str,
 ) -> Result<Response<Body>, Error> {
@@ -63,7 +105,11 @@ pub async fn handle_head(
 		_ => unreachable!(),
 	};

-	let body: Body = Body::from(vec![]);
+	if let Some(cached) = try_answer_cached(&version, version_meta, req) {
+		return Ok(cached);
+	}
+
+	let body: Body = Body::empty();
 	let response = object_headers(&version, version_meta)
 		.header("Content-Length", format!("{}", version_meta.size))
 		.status(StatusCode::OK)
@@ -102,6 +148,10 @@ pub async fn handle_get(
 		ObjectVersionData::FirstBlock(meta, _) => meta,
 	};

+	if let Some(cached) = try_answer_cached(&last_v, last_v_meta, req) {
+		return Ok(cached);
+	}
+
 	let range = match req.headers().get("range") {
 		Some(range) => {
 			let range_str = range.to_str()?;
@@ -144,9 +194,10 @@ pub async fn handle_get(
 			let version = version.ok_or(Error::NotFound)?;

 			let mut blocks = version
-				.blocks()
+				.blocks
+				.items()
 				.iter()
-				.map(|vb| (vb.hash, None))
+				.map(|(_, vb)| (vb.hash, None))
 				.collect::<Vec<_>>();
 			blocks[0].1 = Some(first_block);

@@ -217,11 +268,12 @@ pub async fn handle_get_range(
 			// file (whereas block.offset designates the offset of the block WITHIN THE PART
 			// block.part_number, which is not the same in the case of a multipart upload)
 			let mut blocks = Vec::with_capacity(std::cmp::min(
-				version.blocks().len(),
-				4 + ((end - begin) / std::cmp::max(version.blocks()[0].size as u64, 1024)) as usize,
+				version.blocks.len(),
+				4 + ((end - begin) / std::cmp::max(version.blocks.items()[0].1.size as u64, 1024))
+					as usize,
 			));
 			let mut true_offset = 0;
-			for b in version.blocks().iter() {
+			for (_, b) in version.blocks.items().iter() {
 				if true_offset >= end {
 					break;
 				}
@@ -1,11 +1,11 @@
-use std::collections::{BTreeMap, BTreeSet};
+use std::collections::{BTreeMap, BTreeSet, HashMap};
 use std::fmt::Write;
 use std::sync::Arc;

-use chrono::{DateTime, NaiveDateTime, SecondsFormat, Utc};
 use hyper::{Body, Response};

-use garage_util::error::Error;
+use garage_util::error::Error as GarageError;
+use garage_util::time::*;

 use garage_model::garage::Garage;
 use garage_model::object_table::*;
@@ -13,6 +13,20 @@ use garage_model::object_table::*;
 use garage_table::DeletedFilter;

 use crate::encoding::*;
+use crate::error::*;
+
+#[derive(Debug)]
+pub struct ListObjectsQuery {
+	pub is_v2: bool,
+	pub bucket: String,
+	pub delimiter: Option<String>,
+	pub max_keys: usize,
+	pub prefix: String,
+	pub marker: Option<String>,
+	pub continuation_token: Option<String>,
+	pub start_after: Option<String>,
+	pub urlencode_resp: bool,
+}

 #[derive(Debug)]
 struct ListResultInfo {
@@ -21,55 +35,92 @@ struct ListResultInfo {
 	etag: String,
 }

+pub fn parse_list_objects_query(
+	bucket: &str,
+	params: &HashMap<String, String>,
+) -> Result<ListObjectsQuery, Error> {
+	Ok(ListObjectsQuery {
+		is_v2: params.get("list-type").map(|x| x == "2").unwrap_or(false),
+		bucket: bucket.to_string(),
+		delimiter: params.get("delimiter").filter(|x| !x.is_empty()).cloned(),
+		max_keys: params
+			.get("max-keys")
+			.map(|x| {
+				x.parse::<usize>()
+					.ok_or_bad_request("Invalid value for max-keys")
+			})
+			.unwrap_or(Ok(1000))?,
+		prefix: params.get("prefix").cloned().unwrap_or(String::new()),
+		marker: params.get("marker").cloned(),
+		continuation_token: params.get("continuation-token").cloned(),
+		start_after: params.get("start-after").cloned(),
+		urlencode_resp: params
+			.get("encoding-type")
+			.map(|x| x == "url")
+			.unwrap_or(false),
+	})
+}
+
 pub async fn handle_list(
 	garage: Arc<Garage>,
-	bucket: &str,
-	delimiter: &str,
-	max_keys: usize,
-	prefix: &str,
-	marker: Option<&str>,
-	urlencode_resp: bool,
+	query: &ListObjectsQuery,
 ) -> Result<Response<Body>, Error> {
 	let mut result_keys = BTreeMap::<String, ListResultInfo>::new();
 	let mut result_common_prefixes = BTreeSet::<String>::new();

-	let mut next_chunk_start = marker.unwrap_or(prefix).to_string();
+	let mut next_chunk_start = if query.is_v2 {
+		if let Some(ct) = &query.continuation_token {
+			String::from_utf8(base64::decode(ct.as_bytes())?)?
+		} else {
+			query.start_after.clone().unwrap_or(query.prefix.clone())
+		}
+	} else {
+		query.marker.clone().unwrap_or(query.prefix.clone())
+	};

-	debug!("List request: `{}` {} `{}`", delimiter, max_keys, prefix);
+	debug!(
+		"List request: `{:?}` {} `{}`",
+		query.delimiter, query.max_keys, query.prefix
+	);

 	let truncated;
 	'query_loop: loop {
 		let objects = garage
 			.object_table
 			.get_range(
-				&bucket.to_string(),
+				&query.bucket,
 				Some(next_chunk_start.clone()),
 				Some(DeletedFilter::NotDeleted),
-				max_keys + 1,
+				query.max_keys + 1,
 			)
 			.await?;
 		debug!(
 			"List: get range {} (max {}), results: {}",
 			next_chunk_start,
-			max_keys + 1,
+			query.max_keys + 1,
 			objects.len()
 		);

 		for object in objects.iter() {
-			if !object.key.starts_with(prefix) {
+			if !object.key.starts_with(&query.prefix) {
 				truncated = None;
 				break 'query_loop;
 			}
+
+			if query.is_v2 && query.start_after.as_ref() == Some(&object.key) {
+				continue;
+			}
+
 			if let Some(version) = object.versions().iter().find(|x| x.is_data()) {
-				if result_keys.len() + result_common_prefixes.len() >= max_keys {
+				if result_keys.len() + result_common_prefixes.len() >= query.max_keys {
 					truncated = Some(object.key.to_string());
 					break 'query_loop;
 				}
-				let common_prefix = if delimiter.len() > 0 {
-					let relative_key = &object.key[prefix.len()..];
+				let common_prefix = if let Some(delimiter) = &query.delimiter {
+					let relative_key = &object.key[query.prefix.len()..];
 					relative_key
 						.find(delimiter)
-						.map(|i| &object.key[..prefix.len() + i + delimiter.len()])
+						.map(|i| &object.key[..query.prefix.len() + i + delimiter.len()])
 				} else {
 					None
 				};
@@ -90,14 +141,17 @@ pub async fn handle_list(
 							etag: meta.etag.to_string(),
 						},
 						Some(_lri) => {
-							return Err(Error::Message(format!("Duplicate key?? {}", object.key)))
+							return Err(Error::InternalError(GarageError::Message(format!(
+								"Duplicate key?? {}",
+								object.key
+							))))
 						}
 					};
 					result_keys.insert(object.key.clone(), info);
 				};
 			}
 		}
-		if objects.len() < max_keys + 1 {
+		if objects.len() < query.max_keys + 1 {
 			truncated = None;
 			break 'query_loop;
 		}
@@ -113,32 +167,92 @@ pub async fn handle_list(
 		r#"<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">"#
 	)
 	.unwrap();
-	writeln!(&mut xml, "\t<Name>{}</Name>", bucket).unwrap();
-	writeln!(&mut xml, "\t<Prefix>{}</Prefix>", prefix).unwrap();
-	if let Some(mkr) = marker {
-		writeln!(&mut xml, "\t<Marker>{}</Marker>", mkr).unwrap();
+
+	writeln!(&mut xml, "\t<Name>{}</Name>", query.bucket).unwrap();
+
+	// TODO: in V1, is this supposed to be urlencoded when encoding-type is URL??
+	writeln!(
+		&mut xml,
+		"\t<Prefix>{}</Prefix>",
+		xml_encode_key(&query.prefix, query.urlencode_resp),
+	)
+	.unwrap();
+
+	if let Some(delim) = &query.delimiter {
+		// TODO: in V1, is this supposed to be urlencoded when encoding-type is URL??
+		writeln!(
+			&mut xml,
+			"\t<Delimiter>{}</Delimiter>",
+			xml_encode_key(delim, query.urlencode_resp),
+		)
+		.unwrap();
 	}
-	writeln!(&mut xml, "\t<KeyCount>{}</KeyCount>", result_keys.len()).unwrap();
-	writeln!(&mut xml, "\t<MaxKeys>{}</MaxKeys>", max_keys).unwrap();
+
+	writeln!(&mut xml, "\t<MaxKeys>{}</MaxKeys>", query.max_keys).unwrap();
+	if query.urlencode_resp {
+		writeln!(&mut xml, "\t<EncodingType>url</EncodingType>").unwrap();
+	}
+
+	writeln!(
+		&mut xml,
+		"\t<KeyCount>{}</KeyCount>",
+		result_keys.len() + result_common_prefixes.len()
+	)
+	.unwrap();
 	writeln!(
 		&mut xml,
 		"\t<IsTruncated>{}</IsTruncated>",
 		truncated.is_some()
 	)
 	.unwrap();
-	if let Some(next_marker) = truncated {
-		writeln!(&mut xml, "\t<NextMarker>{}</NextMarker>", next_marker).unwrap();
+
+	if query.is_v2 {
+		if let Some(ct) = &query.continuation_token {
+			writeln!(&mut xml, "\t<ContinuationToken>{}</ContinuationToken>", ct).unwrap();
+		}
+		if let Some(sa) = &query.start_after {
+			writeln!(
+				&mut xml,
+				"\t<StartAfter>{}</StartAfter>",
+				xml_encode_key(sa, query.urlencode_resp)
+			)
+			.unwrap();
+		}
+		if let Some(nct) = truncated {
+			writeln!(
+				&mut xml,
+				"\t<NextContinuationToken>{}</NextContinuationToken>",
+				base64::encode(nct.as_bytes())
+			)
+			.unwrap();
+		}
+	} else {
+		// TODO: are these supposed to be urlencoded when encoding-type is URL??
+		if let Some(mkr) = &query.marker {
+			writeln!(
+				&mut xml,
+				"\t<Marker>{}</Marker>",
+				xml_encode_key(mkr, query.urlencode_resp)
+			)
+			.unwrap();
+		}
+		if let Some(next_marker) = truncated {
+			writeln!(
+				&mut xml,
+				"\t<NextMarker>{}</NextMarker>",
+				xml_encode_key(&next_marker, query.urlencode_resp)
+			)
+			.unwrap();
+		}
 	}
+
 	for (key, info) in result_keys.iter() {
-		let last_modif = NaiveDateTime::from_timestamp(info.last_modified as i64 / 1000, 0);
-		let last_modif = DateTime::<Utc>::from_utc(last_modif, Utc);
-		let last_modif = last_modif.to_rfc3339_opts(SecondsFormat::Millis, true);
+		let last_modif = msec_to_rfc3339(info.last_modified);
 		writeln!(&mut xml, "\t<Contents>").unwrap();
 		writeln!(
 			&mut xml,
 			"\t\t<Key>{}</Key>",
-			xml_escape(key),
-			//xml_encode_key(key, urlencode_resp)       // doesn't work with nextcloud, wtf
+			xml_encode_key(key, query.urlencode_resp),
 		)
 		.unwrap();
 		writeln!(&mut xml, "\t\t<LastModified>{}</LastModified>", last_modif).unwrap();
@@ -149,21 +263,23 @@ pub async fn handle_list(
 		writeln!(&mut xml, "\t\t<StorageClass>STANDARD</StorageClass>").unwrap();
 		writeln!(&mut xml, "\t</Contents>").unwrap();
 	}
-	if result_common_prefixes.len() > 0 {
-		for pfx in result_common_prefixes.iter() {
-			writeln!(&mut xml, "\t<CommonPrefixes>").unwrap();
-			writeln!(
-				&mut xml,
-				"\t\t<Prefix>{}</Prefix>",
-				xml_escape(pfx),
-				//xml_encode_key(pfx, urlencode_resp)
-			)
-			.unwrap();
-			writeln!(&mut xml, "\t</CommonPrefixes>").unwrap();
-		}
+
+	for pfx in result_common_prefixes.iter() {
+		writeln!(&mut xml, "\t<CommonPrefixes>").unwrap();
+		//TODO: in V1, are these urlencoded when urlencode_resp is true ?? (proably)
+		writeln!(
+			&mut xml,
+			"\t\t<Prefix>{}</Prefix>",
+			xml_encode_key(pfx, query.urlencode_resp),
+		)
+		.unwrap();
+		writeln!(&mut xml, "\t</CommonPrefixes>").unwrap();
 	}
+
 	writeln!(&mut xml, "</ListBucketResult>").unwrap();
 	debug!("{}", xml);

-	Ok(Response::new(Body::from(xml.into_bytes())))
+	Ok(Response::builder()
+		.header("Content-Type", "application/xml")
+		.body(Body::from(xml.into_bytes()))?)
 }
@@ -5,13 +5,13 @@ use std::sync::Arc;
 use futures::stream::*;
 use hyper::{Body, Request, Response};
 use md5::{digest::generic_array::*, Digest as Md5Digest, Md5};
-use sha2::{Digest as Sha256Digest, Sha256};
+use sha2::Sha256;

 use garage_table::*;
 use garage_util::data::*;
 use garage_util::error::Error as GarageError;
+use garage_util::time::*;

-use crate::error::*;
 use garage_model::block::INLINE_THRESHOLD;
 use garage_model::block_ref_table::*;
 use garage_model::garage::Garage;
@@ -19,6 +19,8 @@ use garage_model::object_table::*;
 use garage_model::version_table::*;

 use crate::encoding::*;
+use crate::error::*;
+use crate::signature::verify_signed_content;

 pub async fn handle_put(
 	garage: Arc<Garage>,
@@ -27,80 +29,113 @@ pub async fn handle_put(
 	key: &str,
 	content_sha256: Option<Hash>,
 ) -> Result<Response<Body>, Error> {
+	// Generate identity of new version
 	let version_uuid = gen_uuid();
+	let version_timestamp = now_msec();
+
+	// Retrieve interesting headers from request
 	let headers = get_headers(&req)?;
+	debug!("Object headers: {:?}", headers);
+
 	let content_md5 = match req.headers().get("content-md5") {
 		Some(x) => Some(x.to_str()?.to_string()),
 		None => None,
 	};

+	// Parse body of uploaded file
 	let body = req.into_body();

 	let mut chunker = BodyChunker::new(body, garage.config.block_size);
 	let first_block = chunker.next().await?.unwrap_or(vec![]);

-	let mut object_version = ObjectVersion {
-		uuid: version_uuid,
-		timestamp: now_msec(),
-		state: ObjectVersionState::Uploading(headers.clone()),
-	};
-
+	// If body is small enough, store it directly in the object table
+	// as "inline data". We can then return immediately.
 	if first_block.len() < INLINE_THRESHOLD {
 		let mut md5sum = Md5::new();
 		md5sum.update(&first_block[..]);
-		let md5sum_arr = md5sum.finalize();
-		let md5sum_hex = hex::encode(md5sum_arr);
+		let data_md5sum = md5sum.finalize();
+		let data_md5sum_hex = hex::encode(data_md5sum);

-		let sha256sum_hash = hash(&first_block[..]);
+		let data_sha256sum = sha256sum(&first_block[..]);

 		ensure_checksum_matches(
-			md5sum_arr.as_slice(),
-			sha256sum_hash,
+			data_md5sum.as_slice(),
+			data_sha256sum,
 			content_md5.as_deref(),
 			content_sha256,
 		)?;

-		object_version.state = ObjectVersionState::Complete(ObjectVersionData::Inline(
-			ObjectVersionMeta {
-				headers,
-				size: first_block.len() as u64,
-				etag: md5sum_hex.clone(),
-			},
-			first_block,
-		));
+		let object_version = ObjectVersion {
+			uuid: version_uuid,
+			timestamp: version_timestamp,
+			state: ObjectVersionState::Complete(ObjectVersionData::Inline(
+				ObjectVersionMeta {
+					headers,
+					size: first_block.len() as u64,
+					etag: data_md5sum_hex.clone(),
+				},
+				first_block,
+			)),
+		};

 		let object = Object::new(bucket.into(), key.into(), vec![object_version]);
 		garage.object_table.insert(&object).await?;
-		return Ok(put_response(version_uuid, md5sum_hex));
+
+		return Ok(put_response(version_uuid, data_md5sum_hex));
 	}

-	let version = Version::new(version_uuid, bucket.into(), key.into(), false, vec![]);
-
-	let first_block_hash = hash(&first_block[..]);
+	// Write version identifier in object table so that we have a trace
+	// that we are uploading something
+	let mut object_version = ObjectVersion {
+		uuid: version_uuid,
+		timestamp: version_timestamp,
+		state: ObjectVersionState::Uploading(headers.clone()),
+	};
 	let object = Object::new(bucket.into(), key.into(), vec![object_version.clone()]);
 	garage.object_table.insert(&object).await?;

-	let (total_size, md5sum_arr, sha256sum) = read_and_put_blocks(
+	// Initialize corresponding entry in version table
+	// Write this entry now, even with empty block list,
+	// to prevent block_ref entries from being deleted (they can be deleted
+	// if the reference a version that isn't found in the version table)
+	let version = Version::new(version_uuid, bucket.into(), key.into(), false);
+	garage.version_table.insert(&version).await?;
+
+	// Transfer data and verify checksum
+	let first_block_hash = blake2sum(&first_block[..]);
+	let tx_result = read_and_put_blocks(
 		&garage,
-		version,
+		&version,
 		1,
 		first_block,
 		first_block_hash,
 		&mut chunker,
 	)
-	.await?;
+	.await
+	.and_then(|(total_size, data_md5sum, data_sha256sum)| {
+		ensure_checksum_matches(
+			data_md5sum.as_slice(),
+			data_sha256sum,
+			content_md5.as_deref(),
+			content_sha256,
+		)
+		.map(|()| (total_size, data_md5sum))
+	});

-	ensure_checksum_matches(
-		md5sum_arr.as_slice(),
-		sha256sum,
-		content_md5.as_deref(),
-		content_sha256,
-	)?;
-
-	// TODO: if at any step we have an error, we should undo everything we did
+	// If something went wrong, clean up
+	let (total_size, md5sum_arr) = match tx_result {
+		Ok(rv) => rv,
+		Err(e) => {
+			// Mark object as aborted, this will free the blocks further down
+			object_version.state = ObjectVersionState::Aborted;
+			let object = Object::new(bucket.into(), key.into(), vec![object_version.clone()]);
+			garage.object_table.insert(&object).await?;
+			return Err(e);
+		}
+	};

+	// Save final object state, marked as Complete
 	let md5sum_hex = hex::encode(md5sum_arr);
-
 	object_version.state = ObjectVersionState::Complete(ObjectVersionData::FirstBlock(
 		ObjectVersionMeta {
 			headers,
@@ -109,7 +144,6 @@ pub async fn handle_put(
 		},
 		first_block_hash,
 	));
-
 	let object = Object::new(bucket.into(), key.into(), vec![object_version]);
 	garage.object_table.insert(&object).await?;

@@ -119,13 +153,13 @@ pub async fn handle_put(
 /// Validate MD5 sum against content-md5 header
 /// and sha256sum against signed content-sha256
 fn ensure_checksum_matches(
-	md5sum: &[u8],
-	sha256sum: garage_util::data::FixedBytes32,
+	data_md5sum: &[u8],
+	data_sha256sum: garage_util::data::FixedBytes32,
 	content_md5: Option<&str>,
 	content_sha256: Option<garage_util::data::FixedBytes32>,
 ) -> Result<(), Error> {
 	if let Some(expected_sha256) = content_sha256 {
-		if expected_sha256 != sha256sum {
+		if expected_sha256 != data_sha256sum {
 			return Err(Error::BadRequest(format!(
 				"Unable to validate x-amz-content-sha256"
 			)));
@@ -134,7 +168,7 @@ fn ensure_checksum_matches(
 		}
 	}
 	if let Some(expected_md5) = content_md5 {
-		if expected_md5.trim_matches('"') != base64::encode(md5sum) {
+		if expected_md5.trim_matches('"') != base64::encode(data_md5sum) {
 			return Err(Error::BadRequest(format!("Unable to validate content-md5")));
 		} else {
 			trace!("Successfully validated content-md5");
@@ -144,21 +178,21 @@ fn ensure_checksum_matches(
 }

 async fn read_and_put_blocks(
-	garage: &Arc<Garage>,
-	version: Version,
+	garage: &Garage,
+	version: &Version,
 	part_number: u64,
 	first_block: Vec<u8>,
 	first_block_hash: Hash,
 	chunker: &mut BodyChunker,
 ) -> Result<(u64, GenericArray<u8, typenum::U16>, Hash), Error> {
-	let mut md5sum = Md5::new();
-	let mut sha256sum = Sha256::new();
-	md5sum.update(&first_block[..]);
-	sha256sum.input(&first_block[..]);
+	let mut md5hasher = Md5::new();
+	let mut sha256hasher = Sha256::new();
+	md5hasher.update(&first_block[..]);
+	sha256hasher.update(&first_block[..]);

 	let mut next_offset = first_block.len();
 	let mut put_curr_version_block = put_block_meta(
-		garage.clone(),
+		&garage,
 		&version,
 		part_number,
 		0,
@@ -173,12 +207,12 @@ async fn read_and_put_blocks(
 		let (_, _, next_block) =
 			futures::try_join!(put_curr_block, put_curr_version_block, chunker.next())?;
 		if let Some(block) = next_block {
-			md5sum.update(&block[..]);
-			sha256sum.input(&block[..]);
-			let block_hash = hash(&block[..]);
+			md5hasher.update(&block[..]);
+			sha256hasher.update(&block[..]);
+			let block_hash = blake2sum(&block[..]);
 			let block_len = block.len();
 			put_curr_version_block = put_block_meta(
-				garage.clone(),
+				&garage,
 				&version,
 				part_number,
 				next_offset as u64,
@@ -193,39 +227,35 @@ async fn read_and_put_blocks(
 	}

 	let total_size = next_offset as u64;
-	let md5sum_arr = md5sum.finalize();
+	let data_md5sum = md5hasher.finalize();

-	let sha256sum = sha256sum.result();
-	let mut hash = [0u8; 32];
-	hash.copy_from_slice(&sha256sum[..]);
-	let sha256sum = Hash::from(hash);
+	let data_sha256sum = sha256hasher.finalize();
+	let data_sha256sum = Hash::try_from(&data_sha256sum[..]).unwrap();

-	Ok((total_size, md5sum_arr, sha256sum))
+	Ok((total_size, data_md5sum, data_sha256sum))
 }

 async fn put_block_meta(
-	garage: Arc<Garage>,
+	garage: &Garage,
 	version: &Version,
 	part_number: u64,
 	offset: u64,
 	hash: Hash,
 	size: u64,
 ) -> Result<(), GarageError> {
-	// TODO: don't clone, restart from empty block list ??
 	let mut version = version.clone();
-	version
-		.add_block(VersionBlock {
+	version.blocks.put(
+		VersionBlockKey {
 			part_number,
 			offset,
-			hash,
-			size,
-		})
-		.unwrap();
+		},
+		VersionBlock { hash, size },
+	);

 	let block_ref = BlockRef {
 		block: hash,
 		version: version.uuid,
-		deleted: false,
+		deleted: false.into(),
 	};

 	futures::try_join!(
@@ -290,6 +320,7 @@ pub async fn handle_create_multipart_upload(
 	let version_uuid = gen_uuid();
 	let headers = get_headers(req)?;

+	// Create object in object table
 	let object_version = ObjectVersion {
 		uuid: version_uuid,
 		timestamp: now_msec(),
@@ -298,6 +329,14 @@ pub async fn handle_create_multipart_upload(
 	let object = Object::new(bucket.to_string(), key.to_string(), vec![object_version]);
 	garage.object_table.insert(&object).await?;

+	// Insert empty version so that block_ref entries refer to something
+	// (they are inserted concurrently with blocks in the version table, so
+	// there is the possibility that they are inserted before the version table
+	// is created, in which case it is allowed to delete them, e.g. in repair_*)
+	let version = Version::new(version_uuid, bucket.into(), key.into(), false);
+	garage.version_table.insert(&version).await?;
+
+	// Send success response
 	let mut xml = String::new();
 	writeln!(&mut xml, r#"<?xml version="1.0" encoding="UTF-8"?>"#).unwrap();
 	writeln!(
@@ -340,13 +379,12 @@ pub async fn handle_put_part(
 	};

 	// Read first chuck, and at the same time try to get object to see if it exists
-	let mut chunker = BodyChunker::new(req.into_body(), garage.config.block_size);
-
 	let bucket = bucket.to_string();
 	let key = key.to_string();
-	let get_object_fut = garage.object_table.get(&bucket, &key);
-	let get_first_block_fut = chunker.next();
-	let (object, first_block) = futures::try_join!(get_object_fut, get_first_block_fut)?;
+	let mut chunker = BodyChunker::new(req.into_body(), garage.config.block_size);
+
+	let (object, first_block) =
+		futures::try_join!(garage.object_table.get(&bucket, &key), chunker.next(),)?;

 	// Check object is valid and multipart block can be accepted
 	let first_block = first_block.ok_or(Error::BadRequest(format!("Empty body")))?;
@@ -357,17 +395,15 @@ pub async fn handle_put_part(
 		.iter()
 		.any(|v| v.uuid == version_uuid && v.is_uploading())
 	{
-		return Err(Error::BadRequest(format!(
-			"Multipart upload does not exist or is otherwise invalid"
-		)));
+		return Err(Error::NotFound);
 	}

 	// Copy block to store
-	let version = Version::new(version_uuid, bucket, key, false, vec![]);
-	let first_block_hash = hash(&first_block[..]);
-	let (_, md5sum_arr, sha256sum) = read_and_put_blocks(
+	let version = Version::new(version_uuid, bucket, key, false);
+	let first_block_hash = blake2sum(&first_block[..]);
+	let (_, data_md5sum, data_sha256sum) = read_and_put_blocks(
 		&garage,
-		version,
+		&version,
 		part_number,
 		first_block,
 		first_block_hash,
@@ -375,15 +411,24 @@ pub async fn handle_put_part(
 	)
 	.await?;

+	// Verify that checksums map
 	ensure_checksum_matches(
-		md5sum_arr.as_slice(),
-		sha256sum,
+		data_md5sum.as_slice(),
+		data_sha256sum,
 		content_md5.as_deref(),
 		content_sha256,
 	)?;

+	// Store part etag in version
+	let data_md5sum_hex = hex::encode(data_md5sum);
+	let mut version = version;
+	version
+		.parts_etags
+		.put(part_number, data_md5sum_hex.clone());
+	garage.version_table.insert(&version).await?;
+
 	let response = Response::builder()
-		.header("ETag", format!("\"{}\"", hex::encode(md5sum_arr)))
+		.header("ETag", format!("\"{}\"", data_md5sum_hex))
 		.body(Body::from(vec![]))
 		.unwrap();
 	Ok(response)
@@ -391,11 +436,23 @@ pub async fn handle_put_part(

 pub async fn handle_complete_multipart_upload(
 	garage: Arc<Garage>,
-	_req: Request<Body>,
+	req: Request<Body>,
 	bucket: &str,
 	key: &str,
 	upload_id: &str,
+	content_sha256: Option<Hash>,
 ) -> Result<Response<Body>, Error> {
+	let body = hyper::body::to_bytes(req.into_body()).await?;
+	verify_signed_content(content_sha256, &body[..])?;
+
+	let body_xml = roxmltree::Document::parse(&std::str::from_utf8(&body)?)?;
+	let body_list_of_parts = parse_complete_multpart_upload_body(&body_xml)
+		.ok_or_bad_request("Invalid CompleteMultipartUpload XML")?;
+	debug!(
+		"CompleteMultipartUpload list of parts: {:?}",
+		body_list_of_parts
+	);
+
 	let version_uuid = decode_upload_id(upload_id)?;

 	let bucket = bucket.to_string();
@@ -404,67 +461,70 @@ pub async fn handle_complete_multipart_upload(
 		garage.object_table.get(&bucket, &key),
 		garage.version_table.get(&version_uuid, &EmptyKey),
 	)?;
-	let object = object.ok_or(Error::BadRequest(format!("Object not found")))?;

-	let object_version = object
+	let object = object.ok_or(Error::BadRequest(format!("Object not found")))?;
+	let mut object_version = object
 		.versions()
 		.iter()
-		.find(|v| v.uuid == version_uuid && v.is_uploading());
-	let mut object_version = match object_version {
-		None => {
-			return Err(Error::BadRequest(format!(
-				"Multipart upload does not exist or has already been completed"
-			)))
-		}
-		Some(x) => x.clone(),
-	};
-	let version = version.ok_or(Error::BadRequest(format!("Version not found")))?;
+		.find(|v| v.uuid == version_uuid && v.is_uploading())
+		.cloned()
+		.ok_or(Error::BadRequest(format!("Version not found")))?;

-	if version.blocks().len() == 0 {
+	let version = version.ok_or(Error::BadRequest(format!("Version not found")))?;
+	if version.blocks.len() == 0 {
 		return Err(Error::BadRequest(format!("No data was uploaded")));
 	}
+
 	let headers = match object_version.state {
 		ObjectVersionState::Uploading(headers) => headers.clone(),
 		_ => unreachable!(),
 	};

-	// ETag calculation: we produce ETags that have the same form as
-	// those of S3 multipart uploads, but we don't use their actual
-	// calculation for the first part (we use random bytes). This
-	// shouldn't impact compatibility as the S3 docs specify that
-	// the ETag is an opaque value in case of a multipart upload.
-	// See also: https://teppen.io/2018/06/23/aws_s3_etags/
-	let num_parts = version.blocks().last().unwrap().part_number
-		- version.blocks().first().unwrap().part_number
-		+ 1;
-	let etag = format!(
-		"{}-{}",
-		hex::encode(&rand::random::<[u8; 16]>()[..]),
-		num_parts
-	);
-
-	// TODO: check that all the parts that they pretend they gave us are indeed there
-	// TODO: when we read the XML from _req, remember to check the sha256 sum of the payload
-	//       against the signed x-amz-content-sha256
-	// TODO: check MD5 sum of all uploaded parts? but that would mean we have to store them somewhere...
-
-	let total_size = version
-		.blocks()
+	// Check that the list of parts they gave us corresponds to the parts we have here
+	debug!("Expected parts from request: {:?}", body_list_of_parts);
+	debug!("Parts stored in version: {:?}", version.parts_etags.items());
+	let parts = version
+		.parts_etags
+		.items()
 		.iter()
-		.map(|x| x.size)
-		.fold(0, |x, y| x + y);
+		.map(|pair| (&pair.0, &pair.1));
+	let same_parts = body_list_of_parts
+		.iter()
+		.map(|x| (&x.part_number, &x.etag))
+		.eq(parts);
+	if !same_parts {
+		return Err(Error::BadRequest(format!("We don't have the same parts")));
+	}
+
+	// Calculate etag of final object
+	// To understand how etags are calculated, read more here:
+	// https://teppen.io/2018/06/23/aws_s3_etags/
+	let num_parts = version.blocks.items().last().unwrap().0.part_number
+		- version.blocks.items().first().unwrap().0.part_number
+		+ 1;
+	let mut etag_md5_hasher = Md5::new();
+	for (_, etag) in version.parts_etags.items().iter() {
+		etag_md5_hasher.update(etag.as_bytes());
+	}
+	let etag = format!("{}-{}", hex::encode(etag_md5_hasher.finalize()), num_parts);
+
+	// Calculate total size of final object
+	let total_size = version.blocks.items().iter().map(|x| x.1.size).sum();
+
+	// Write final object version
 	object_version.state = ObjectVersionState::Complete(ObjectVersionData::FirstBlock(
 		ObjectVersionMeta {
 			headers,
 			size: total_size,
-			etag: etag,
+			etag,
 		},
-		version.blocks()[0].hash,
+		version.blocks.items()[0].1.hash,
 	));

 	let final_object = Object::new(bucket.clone(), key.clone(), vec![object_version]);
 	garage.object_table.insert(&final_object).await?;

+	// Send response saying ok we're done
 	let mut xml = String::new();
 	writeln!(&mut xml, r#"<?xml version="1.0" encoding="UTF-8"?>"#).unwrap();
 	writeln!(
@@ -504,11 +564,7 @@ pub async fn handle_abort_multipart_upload(
 		.iter()
 		.find(|v| v.uuid == version_uuid && v.is_uploading());
 	let mut object_version = match object_version {
-		None => {
-			return Err(Error::BadRequest(format!(
-				"Multipart upload does not exist or has already been completed"
-			)))
-		}
+		None => return Err(Error::NotFound),
 		Some(x) => x.clone(),
 	};

@@ -528,26 +584,48 @@ fn get_mime_type(req: &Request<Body>) -> Result<String, Error> {
 		.to_string())
 }

-fn get_headers(req: &Request<Body>) -> Result<ObjectVersionHeaders, Error> {
+pub(crate) fn get_headers(req: &Request<Body>) -> Result<ObjectVersionHeaders, Error> {
 	let content_type = get_mime_type(req)?;
-	let other_headers = vec![
+	let mut other = BTreeMap::new();
+
+	// Preserve standard headers
+	let standard_header = vec![
 		hyper::header::CACHE_CONTROL,
 		hyper::header::CONTENT_DISPOSITION,
 		hyper::header::CONTENT_ENCODING,
 		hyper::header::CONTENT_LANGUAGE,
 		hyper::header::EXPIRES,
 	];
-	let mut other = BTreeMap::new();
-	for h in other_headers.iter() {
+	for h in standard_header.iter() {
 		if let Some(v) = req.headers().get(h) {
-			if let Ok(v_str) = v.to_str() {
-				other.insert(h.to_string(), v_str.to_string());
+			match v.to_str() {
+				Ok(v_str) => {
+					other.insert(h.to_string(), v_str.to_string());
+				}
+				Err(e) => {
+					warn!("Discarding header {}, error in .to_str(): {}", h, e);
+				}
 			}
 		}
 	}
+
+	// Preserve x-amz-meta- headers
+	for (k, v) in req.headers().iter() {
+		if k.as_str().starts_with("x-amz-meta-") {
+			match v.to_str() {
+				Ok(v_str) => {
+					other.insert(k.to_string(), v_str.to_string());
+				}
+				Err(e) => {
+					warn!("Discarding header {}, error in .to_str(): {}", k, e);
+				}
+			}
+		}
+	}
+
 	Ok(ObjectVersionHeaders {
 		content_type,
-		other: BTreeMap::new(),
+		other,
 	})
 }

@@ -560,3 +638,39 @@ fn decode_upload_id(id: &str) -> Result<UUID, Error> {
 	uuid.copy_from_slice(&id_bin[..]);
 	Ok(UUID::from(uuid))
 }
+
+#[derive(Debug)]
+struct CompleteMultipartUploadPart {
+	etag: String,
+	part_number: u64,
+}
+
+fn parse_complete_multpart_upload_body(
+	xml: &roxmltree::Document,
+) -> Option<Vec<CompleteMultipartUploadPart>> {
+	let mut parts = vec![];
+
+	let root = xml.root();
+	let cmu = root.first_child()?;
+	if !cmu.has_tag_name("CompleteMultipartUpload") {
+		return None;
+	}
+
+	for item in cmu.children() {
+		if item.has_tag_name("Part") {
+			let etag = item.children().find(|e| e.has_tag_name("ETag"))?.text()?;
+			let part_number = item
+				.children()
+				.find(|e| e.has_tag_name("PartNumber"))?
+				.text()?;
+			parts.push(CompleteMultipartUploadPart {
+				etag: etag.trim_matches('"').to_string(),
+				part_number: part_number.parse().ok()?,
+			});
+		} else {
+			return None;
+		}
+	}
+
+	Some(parts)
+}
@@ -1,12 +1,12 @@
 use std::collections::HashMap;

 use chrono::{DateTime, Duration, NaiveDateTime, Utc};
-use hmac::{Hmac, Mac};
+use hmac::{Hmac, Mac, NewMac};
 use hyper::{Body, Method, Request};
 use sha2::{Digest, Sha256};

 use garage_table::*;
-use garage_util::data::Hash;
+use garage_util::data::{sha256sum, Hash};

 use garage_model::garage::Garage;
 use garage_model::key_table::*;
@@ -91,8 +91,8 @@ pub async fn check_signature(
 		"s3",
 	)
 	.ok_or_internal_error("Unable to build signing HMAC")?;
-	hmac.input(string_to_sign.as_bytes());
-	let signature = hex::encode(hmac.result().code());
+	hmac.update(string_to_sign.as_bytes());
+	let signature = hex::encode(hmac.finalize().into_bytes());

 	if authorization.signature != signature {
 		trace!("Canonical request: ``{}``", canonical_request);
@@ -106,12 +106,10 @@ pub async fn check_signature(
 	} else {
 		let bytes = hex::decode(authorization.content_sha256)
 			.ok_or_bad_request("Invalid content sha256 hash")?;
-		let mut hash = [0u8; 32];
-		if bytes.len() != 32 {
-			return Err(Error::BadRequest(format!("Invalid content sha256 hash")));
-		}
-		hash.copy_from_slice(&bytes[..]);
-		Some(Hash::from(hash))
+		Some(
+			Hash::try_from(&bytes[..])
+				.ok_or(Error::BadRequest(format!("Invalid content sha256 hash")))?,
+		)
 	};

 	Ok((key, content_sha256))
@@ -220,12 +218,12 @@ fn parse_credential(cred: &str) -> Result<(String, String), Error> {

 fn string_to_sign(datetime: &DateTime<Utc>, scope_string: &str, canonical_req: &str) -> String {
 	let mut hasher = Sha256::default();
-	hasher.input(canonical_req.as_bytes());
+	hasher.update(canonical_req.as_bytes());
 	[
 		"AWS4-HMAC-SHA256",
 		&datetime.format(LONG_DATETIME).to_string(),
 		scope_string,
-		&hex::encode(hasher.result().as_slice()),
+		&hex::encode(hasher.finalize().as_slice()),
 	]
 	.join("\n")
 }
@@ -238,14 +236,14 @@ fn signing_hmac(
 ) -> Result<HmacSha256, crypto_mac::InvalidKeyLength> {
 	let secret = String::from("AWS4") + secret_key;
 	let mut date_hmac = HmacSha256::new_varkey(secret.as_bytes())?;
-	date_hmac.input(datetime.format(SHORT_DATE).to_string().as_bytes());
-	let mut region_hmac = HmacSha256::new_varkey(&date_hmac.result().code())?;
-	region_hmac.input(region.as_bytes());
-	let mut service_hmac = HmacSha256::new_varkey(&region_hmac.result().code())?;
-	service_hmac.input(service.as_bytes());
-	let mut signing_hmac = HmacSha256::new_varkey(&service_hmac.result().code())?;
-	signing_hmac.input(b"aws4_request");
-	let hmac = HmacSha256::new_varkey(&signing_hmac.result().code())?;
+	date_hmac.update(datetime.format(SHORT_DATE).to_string().as_bytes());
+	let mut region_hmac = HmacSha256::new_varkey(&date_hmac.finalize().into_bytes())?;
+	region_hmac.update(region.as_bytes());
+	let mut service_hmac = HmacSha256::new_varkey(&region_hmac.finalize().into_bytes())?;
+	service_hmac.update(service.as_bytes());
+	let mut signing_hmac = HmacSha256::new_varkey(&service_hmac.finalize().into_bytes())?;
+	signing_hmac.update(b"aws4_request");
+	let hmac = HmacSha256::new_varkey(&signing_hmac.finalize().into_bytes())?;
 	Ok(hmac)
 }

@@ -293,3 +291,14 @@ fn canonical_query_string(uri: &hyper::Uri) -> String {
 		"".to_string()
 	}
 }
+
+pub fn verify_signed_content(content_sha256: Option<Hash>, body: &[u8]) -> Result<(), Error> {
+	let expected_sha256 =
+		content_sha256.ok_or_bad_request("Request content hash not signed, aborting.")?;
+	if expected_sha256 != sha256sum(body) {
+		return Err(Error::BadRequest(format!(
+			"Request content hash does not match signed hash"
+		)));
+	}
+	Ok(())
+}
@@ -1,9 +1,9 @@
 [package]
 name = "garage"
-version = "0.1.1"
+version = "0.2.1"
 authors = ["Alex Auvolat <alex@adnab.me>"]
 edition = "2018"
-license = "GPL-3.0"
+license = "AGPL-3.0"
 description = "Garage, an S3-compatible distributed object store for self-hosted deployments"
 repository = "https://git.deuxfleurs.fr/Deuxfleurs/garage"

@@ -14,27 +14,27 @@ path = "main.rs"
 # See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

 [dependencies]
-garage_util = { version = "0.1.1", path = "../util" }
-garage_rpc = { version = "0.1.1", path = "../rpc" }
-garage_table = { version = "0.1.1", path = "../table" }
-garage_model = { version = "0.1.1", path = "../model" }
-garage_api = { version = "0.1.1", path = "../api" }
-garage_web = { version = "0.1.1", path = "../web" }
+garage_api = { version = "0.2.1", path = "../api" }
+garage_model = { version = "0.2.1", path = "../model" }
+garage_rpc = { version = "0.2.1", path = "../rpc" }
+garage_table = { version = "0.2.1", path = "../table" }
+garage_util = { version = "0.2.1", path = "../util" }
+garage_web = { version = "0.2.1", path = "../web" }

-bytes = "0.4"
-rand = "0.7"
-hex = "0.3"
-sha2 = "0.8"
+bytes = "1.0"
+git-version = "0.3.4"
+hex = "0.4"
 log = "0.4"
 pretty_env_logger = "0.4"
+rand = "0.8"

-sled = "0.31"
+sled = "0.34"

+rmp-serde = "0.15"
+serde = { version = "1.0", default-features = false, features = ["derive", "rc"] }
 structopt = { version = "0.3", default-features = false }
 toml = "0.5"
-rmp-serde = "0.14.3"
-serde = { version = "1.0", default-features = false, features = ["derive", "rc"] }

 futures = "0.3"
 futures-util = "0.3"
-tokio = { version = "0.2", default-features = false, features = ["rt-core", "rt-threaded", "io-driver", "net", "tcp", "time", "macros", "sync", "signal", "fs"] }
+tokio = { version = "1.0", default-features = false, features = ["rt", "rt-multi-thread", "io-util", "net", "time", "macros", "sync", "signal", "fs"] }
@@ -1,3 +1,5 @@
+use std::collections::HashMap;
+use std::fmt::Write;
 use std::sync::Arc;

 use serde::{Deserialize, Serialize};
@@ -5,6 +7,7 @@ use serde::{Deserialize, Serialize};
 use garage_util::error::Error;

 use garage_table::crdt::CRDT;
+use garage_table::replication::*;
 use garage_table::*;

 use garage_rpc::rpc_client::*;
@@ -14,6 +17,7 @@ use garage_model::bucket_table::*;
 use garage_model::garage::Garage;
 use garage_model::key_table::*;

+use crate::cli::*;
 use crate::repair::Repair;
 use crate::*;

@@ -25,6 +29,7 @@ pub enum AdminRPC {
 	BucketOperation(BucketOperation),
 	KeyOperation(KeyOperation),
 	LaunchRepair(RepairOpt),
+	Stats(StatsOpt),

 	// Replies
 	Ok(String),
@@ -55,6 +60,7 @@ impl AdminRpcHandler {
 					AdminRPC::BucketOperation(bo) => self2.handle_bucket_cmd(bo).await,
 					AdminRPC::KeyOperation(ko) => self2.handle_key_cmd(ko).await,
 					AdminRPC::LaunchRepair(opt) => self2.handle_launch_repair(opt).await,
+					AdminRPC::Stats(opt) => self2.handle_stats(opt).await,
 					_ => Err(Error::BadRPC(format!("Invalid RPC"))),
 				}
 			}
@@ -116,7 +122,7 @@ impl AdminRpcHandler {
 				for (key_id, _, _) in bucket.authorized_keys() {
 					if let Some(key) = self.garage.key_table.get(&EmptyKey, key_id).await? {
 						if !key.deleted.get() {
-							self.update_key_bucket(key, &bucket.name, false, false)
+							self.update_key_bucket(&key, &bucket.name, false, false)
 								.await?;
 						}
 					} else {
@@ -128,31 +134,31 @@ impl AdminRpcHandler {
 				Ok(AdminRPC::Ok(format!("Bucket {} was deleted.", query.name)))
 			}
 			BucketOperation::Allow(query) => {
-				let key = self.get_existing_key(&query.key_id).await?;
+				let key = self.get_existing_key(&query.key_pattern).await?;
 				let bucket = self.get_existing_bucket(&query.bucket).await?;
 				let allow_read = query.read || key.allow_read(&query.bucket);
 				let allow_write = query.write || key.allow_write(&query.bucket);
-				self.update_key_bucket(key, &query.bucket, allow_read, allow_write)
+				self.update_key_bucket(&key, &query.bucket, allow_read, allow_write)
 					.await?;
-				self.update_bucket_key(bucket, &query.key_id, allow_read, allow_write)
+				self.update_bucket_key(bucket, &key.key_id, allow_read, allow_write)
 					.await?;
 				Ok(AdminRPC::Ok(format!(
 					"New permissions for {} on {}: read {}, write {}.",
-					&query.key_id, &query.bucket, allow_read, allow_write
+					&key.key_id, &query.bucket, allow_read, allow_write
 				)))
 			}
 			BucketOperation::Deny(query) => {
-				let key = self.get_existing_key(&query.key_id).await?;
+				let key = self.get_existing_key(&query.key_pattern).await?;
 				let bucket = self.get_existing_bucket(&query.bucket).await?;
 				let allow_read = !query.read && key.allow_read(&query.bucket);
 				let allow_write = !query.write && key.allow_write(&query.bucket);
-				self.update_key_bucket(key, &query.bucket, allow_read, allow_write)
+				self.update_key_bucket(&key, &query.bucket, allow_read, allow_write)
 					.await?;
-				self.update_bucket_key(bucket, &query.key_id, allow_read, allow_write)
+				self.update_bucket_key(bucket, &key.key_id, allow_read, allow_write)
 					.await?;
 				Ok(AdminRPC::Ok(format!(
 					"New permissions for {} on {}: read {}, write {}.",
-					&query.key_id, &query.bucket, allow_read, allow_write
+					&key.key_id, &query.bucket, allow_read, allow_write
 				)))
 			}
 			BucketOperation::Website(query) => {
@@ -187,7 +193,12 @@ impl AdminRpcHandler {
 				let key_ids = self
 					.garage
 					.key_table
-					.get_range(&EmptyKey, None, Some(DeletedFilter::NotDeleted), 10000)
+					.get_range(
+						&EmptyKey,
+						None,
+						Some(KeyFilter::Deleted(DeletedFilter::NotDeleted)),
+						10000,
+					)
 					.await?
 					.iter()
 					.map(|k| (k.key_id.to_string(), k.name.get().clone()))
@@ -195,7 +206,7 @@ impl AdminRpcHandler {
 				Ok(AdminRPC::KeyList(key_ids))
 			}
 			KeyOperation::Info(query) => {
-				let key = self.get_existing_key(&query.key_id).await?;
+				let key = self.get_existing_key(&query.key_pattern).await?;
 				Ok(AdminRPC::KeyInfo(key))
 			}
 			KeyOperation::New(query) => {
@@ -204,13 +215,13 @@ impl AdminRpcHandler {
 				Ok(AdminRPC::KeyInfo(key))
 			}
 			KeyOperation::Rename(query) => {
-				let mut key = self.get_existing_key(&query.key_id).await?;
+				let mut key = self.get_existing_key(&query.key_pattern).await?;
 				key.name.update(query.new_name);
 				self.garage.key_table.insert(&key).await?;
 				Ok(AdminRPC::KeyInfo(key))
 			}
 			KeyOperation::Delete(query) => {
-				let key = self.get_existing_key(&query.key_id).await?;
+				let key = self.get_existing_key(&query.key_pattern).await?;
 				if !query.yes {
 					return Err(Error::BadRPC(format!(
 						"Add --yes flag to really perform this operation"
@@ -227,13 +238,24 @@ impl AdminRpcHandler {
 						return Err(Error::Message(format!("Bucket not found: {}", ab_name)));
 					}
 				}
-				let del_key = Key::delete(key.key_id);
+				let del_key = Key::delete(key.key_id.to_string());
 				self.garage.key_table.insert(&del_key).await?;
 				Ok(AdminRPC::Ok(format!(
 					"Key {} was deleted successfully.",
-					query.key_id
+					key.key_id
 				)))
 			}
+			KeyOperation::Import(query) => {
+				let prev_key = self.garage.key_table.get(&EmptyKey, &query.key_id)
+					.await?;
+				if prev_key.is_some() {
+					return Err(Error::Message(format!("Key {} already exists in data store. Even if it is deleted, we can't let you create a new key with the same ID. Sorry.", query.key_id)));
+				}
+				let imported_key = Key::import(&query.key_id, &query.secret_key, &query.name);
+				self.garage.key_table.insert(&imported_key).await?;
+				Ok(AdminRPC::KeyInfo(imported_key))
+
+			}
 		}
 	}

@@ -250,14 +272,28 @@ impl AdminRpcHandler {
 			))))
 	}

-	async fn get_existing_key(&self, id: &String) -> Result<Key, Error> {
-		self.garage
+	async fn get_existing_key(&self, pattern: &str) -> Result<Key, Error> {
+		let candidates = self
+			.garage
 			.key_table
-			.get(&EmptyKey, id)
+			.get_range(
+				&EmptyKey,
+				None,
+				Some(KeyFilter::Matches(pattern.to_string())),
+				10,
+			)
 			.await?
+			.into_iter()
 			.filter(|k| !k.deleted.get())
-			.map(Ok)
-			.unwrap_or(Err(Error::BadRPC(format!("Key {} does not exist", id))))
+			.collect::<Vec<_>>();
+		if candidates.len() != 1 {
+			Err(Error::Message(format!(
+				"{} matching keys",
+				candidates.len()
+			)))
+		} else {
+			Ok(candidates.into_iter().next().unwrap())
+		}
 	}

 	/// Update **bucket table** to inform of the new linked key
@@ -290,11 +326,12 @@ impl AdminRpcHandler {
 	/// Update **key table** to inform of the new linked bucket
 	async fn update_key_bucket(
 		&self,
-		mut key: Key,
+		key: &Key,
 		bucket: &String,
 		allow_read: bool,
 		allow_write: bool,
 	) -> Result<(), Error> {
+		let mut key = key.clone();
 		let old_map = key.authorized_buckets.take_and_clear();
 		key.authorized_buckets.merge(&old_map.update_mutator(
 			bucket.clone(),
@@ -350,12 +387,118 @@ impl AdminRpcHandler {
 				.background
 				.spawn_worker("Repair worker".into(), move |must_exit| async move {
 					repair.repair_worker(opt, must_exit).await
-				})
-				.await;
+				});
 			Ok(AdminRPC::Ok(format!(
 				"Repair launched on {:?}",
 				self.garage.system.id
 			)))
 		}
 	}
+
+	async fn handle_stats(&self, opt: StatsOpt) -> Result<AdminRPC, Error> {
+		if opt.all_nodes {
+			let mut ret = String::new();
+			let ring = self.garage.system.ring.borrow().clone();
+
+			for node in ring.config.members.keys() {
+				let mut opt = opt.clone();
+				opt.all_nodes = false;
+
+				writeln!(&mut ret, "\n======================").unwrap();
+				writeln!(&mut ret, "Stats for node {:?}:", node).unwrap();
+				match self
+					.rpc_client
+					.call(*node, AdminRPC::Stats(opt), ADMIN_RPC_TIMEOUT)
+					.await
+				{
+					Ok(AdminRPC::Ok(s)) => writeln!(&mut ret, "{}", s).unwrap(),
+					Ok(x) => writeln!(&mut ret, "Bad answer: {:?}", x).unwrap(),
+					Err(e) => writeln!(&mut ret, "Error: {}", e).unwrap(),
+				}
+			}
+			Ok(AdminRPC::Ok(ret))
+		} else {
+			Ok(AdminRPC::Ok(self.gather_stats_local(opt)?))
+		}
+	}
+
+	fn gather_stats_local(&self, opt: StatsOpt) -> Result<String, Error> {
+		let mut ret = String::new();
+		writeln!(
+			&mut ret,
+			"\nGarage version: {}",
+			git_version::git_version!()
+		)
+		.unwrap();
+
+		// Gather ring statistics
+		let ring = self.garage.system.ring.borrow().clone();
+		let mut ring_nodes = HashMap::new();
+		for r in ring.ring.iter() {
+			for n in r.nodes.iter() {
+				if !ring_nodes.contains_key(n) {
+					ring_nodes.insert(*n, 0usize);
+				}
+				*ring_nodes.get_mut(n).unwrap() += 1;
+			}
+		}
+		writeln!(&mut ret, "\nRing nodes & partition count:").unwrap();
+		for (n, c) in ring_nodes.iter() {
+			writeln!(&mut ret, "  {:?} {}", n, c).unwrap();
+		}
+
+		self.gather_table_stats(&mut ret, &self.garage.bucket_table, &opt)?;
+		self.gather_table_stats(&mut ret, &self.garage.key_table, &opt)?;
+		self.gather_table_stats(&mut ret, &self.garage.object_table, &opt)?;
+		self.gather_table_stats(&mut ret, &self.garage.version_table, &opt)?;
+		self.gather_table_stats(&mut ret, &self.garage.block_ref_table, &opt)?;
+
+		writeln!(&mut ret, "\nBlock manager stats:").unwrap();
+		if opt.detailed {
+			writeln!(
+				&mut ret,
+				"  number of blocks: {}",
+				self.garage.block_manager.rc_len()
+			)
+			.unwrap();
+		}
+		writeln!(
+			&mut ret,
+			"  resync queue length: {}",
+			self.garage.block_manager.resync_queue_len()
+		)
+		.unwrap();
+
+		Ok(ret)
+	}
+
+	fn gather_table_stats<F, R>(
+		&self,
+		to: &mut String,
+		t: &Arc<Table<F, R>>,
+		opt: &StatsOpt,
+	) -> Result<(), Error>
+	where
+		F: TableSchema + 'static,
+		R: TableReplication + 'static,
+	{
+		writeln!(to, "\nTable stats for {}", t.data.name).unwrap();
+		if opt.detailed {
+			writeln!(to, "  number of items: {}", t.data.store.len()).unwrap();
+			writeln!(
+				to,
+				"  Merkle tree size: {}",
+				t.merkle_updater.merkle_tree_len()
+			)
+			.unwrap();
+		}
+		writeln!(
+			to,
+			"  Merkle updater todo queue length: {}",
+			t.merkle_updater.todo_len()
+		)
+		.unwrap();
+		writeln!(to, "  GC todo queue length: {}", t.data.gc_todo_len()).unwrap();
+		Ok(())
+	}
 }
@@ -0,0 +1,560 @@
+use std::collections::HashSet;
+use std::net::SocketAddr;
+use std::path::PathBuf;
+
+use serde::{Deserialize, Serialize};
+use structopt::StructOpt;
+
+use garage_util::error::Error;
+use garage_util::data::UUID;
+use garage_util::time::*;
+
+use garage_rpc::membership::*;
+use garage_rpc::ring::*;
+use garage_rpc::rpc_client::*;
+
+use garage_model::bucket_table::*;
+use garage_model::key_table::*;
+
+use crate::admin_rpc::*;
+
+#[derive(StructOpt, Debug)]
+pub enum Command {
+	/// Run Garage server
+	#[structopt(name = "server")]
+	Server(ServerOpt),
+
+	/// Get network status
+	#[structopt(name = "status")]
+	Status,
+
+	/// Garage node operations
+	#[structopt(name = "node")]
+	Node(NodeOperation),
+
+	/// Bucket operations
+	#[structopt(name = "bucket")]
+	Bucket(BucketOperation),
+
+	/// Key operations
+	#[structopt(name = "key")]
+	Key(KeyOperation),
+
+	/// Start repair of node data
+	#[structopt(name = "repair")]
+	Repair(RepairOpt),
+
+	/// Gather node statistics
+	#[structopt(name = "stats")]
+	Stats(StatsOpt),
+}
+
+#[derive(StructOpt, Debug)]
+pub struct ServerOpt {
+	/// Configuration file
+	#[structopt(short = "c", long = "config", default_value = "./config.toml")]
+	pub config_file: PathBuf,
+}
+
+#[derive(StructOpt, Debug)]
+pub enum NodeOperation {
+	/// Configure Garage node
+	#[structopt(name = "configure")]
+	Configure(ConfigureNodeOpt),
+
+	/// Remove Garage node from cluster
+	#[structopt(name = "remove")]
+	Remove(RemoveNodeOpt),
+}
+
+#[derive(StructOpt, Debug)]
+pub struct ConfigureNodeOpt {
+	/// Node to configure (prefix of hexadecimal node id)
+	node_id: String,
+
+	/// Location (datacenter) of the node
+	#[structopt(short = "d", long = "datacenter")]
+	datacenter: Option<String>,
+
+	/// Capacity (in relative terms, use 1 to represent your smallest server)
+	#[structopt(short = "c", long = "capacity")]
+	capacity: Option<u32>,
+
+	/// Optionnal node tag
+	#[structopt(short = "t", long = "tag")]
+	tag: Option<String>,
+
+	/// Replaced node(s): list of node IDs that will be removed from the current cluster
+	#[structopt(long = "replace")]
+	replace: Vec<String>,
+}
+
+#[derive(StructOpt, Debug)]
+pub struct RemoveNodeOpt {
+	/// Node to configure (prefix of hexadecimal node id)
+	node_id: String,
+
+	/// If this flag is not given, the node won't be removed
+	#[structopt(long = "yes")]
+	yes: bool,
+}
+
+#[derive(Serialize, Deserialize, StructOpt, Debug)]
+pub enum BucketOperation {
+	/// List buckets
+	#[structopt(name = "list")]
+	List,
+
+	/// Get bucket info
+	#[structopt(name = "info")]
+	Info(BucketOpt),
+
+	/// Create bucket
+	#[structopt(name = "create")]
+	Create(BucketOpt),
+
+	/// Delete bucket
+	#[structopt(name = "delete")]
+	Delete(DeleteBucketOpt),
+
+	/// Allow key to read or write to bucket
+	#[structopt(name = "allow")]
+	Allow(PermBucketOpt),
+
+	/// Allow key to read or write to bucket
+	#[structopt(name = "deny")]
+	Deny(PermBucketOpt),
+
+	/// Expose as website or not
+	#[structopt(name = "website")]
+	Website(WebsiteOpt),
+}
+
+#[derive(Serialize, Deserialize, StructOpt, Debug)]
+pub struct WebsiteOpt {
+	/// Create
+	#[structopt(long = "allow")]
+	pub allow: bool,
+
+	/// Delete
+	#[structopt(long = "deny")]
+	pub deny: bool,
+
+	/// Bucket name
+	pub bucket: String,
+}
+
+#[derive(Serialize, Deserialize, StructOpt, Debug)]
+pub struct BucketOpt {
+	/// Bucket name
+	pub name: String,
+}
+
+#[derive(Serialize, Deserialize, StructOpt, Debug)]
+pub struct DeleteBucketOpt {
+	/// Bucket name
+	pub name: String,
+
+	/// If this flag is not given, the bucket won't be deleted
+	#[structopt(long = "yes")]
+	pub yes: bool,
+}
+
+#[derive(Serialize, Deserialize, StructOpt, Debug)]
+pub struct PermBucketOpt {
+	/// Access key name or ID
+	#[structopt(long = "key")]
+	pub key_pattern: String,
+
+	/// Allow/deny read operations
+	#[structopt(long = "read")]
+	pub read: bool,
+
+	/// Allow/deny write operations
+	#[structopt(long = "write")]
+	pub write: bool,
+
+	/// Bucket name
+	pub bucket: String,
+}
+
+#[derive(Serialize, Deserialize, StructOpt, Debug)]
+pub enum KeyOperation {
+	/// List keys
+	#[structopt(name = "list")]
+	List,
+
+	/// Get key info
+	#[structopt(name = "info")]
+	Info(KeyOpt),
+
+	/// Create new key
+	#[structopt(name = "new")]
+	New(KeyNewOpt),
+
+	/// Rename key
+	#[structopt(name = "rename")]
+	Rename(KeyRenameOpt),
+
+	/// Delete key
+	#[structopt(name = "delete")]
+	Delete(KeyDeleteOpt),
+
+	/// Import key
+	#[structopt(name = "import")]
+	Import(KeyImportOpt),
+}
+
+#[derive(Serialize, Deserialize, StructOpt, Debug)]
+pub struct KeyOpt {
+	/// ID or name of the key
+	pub key_pattern: String,
+}
+
+#[derive(Serialize, Deserialize, StructOpt, Debug)]
+pub struct KeyNewOpt {
+	/// Name of the key
+	#[structopt(long = "name", default_value = "Unnamed key")]
+	pub name: String,
+}
+
+#[derive(Serialize, Deserialize, StructOpt, Debug)]
+pub struct KeyRenameOpt {
+	/// ID or name of the key
+	pub key_pattern: String,
+
+	/// New name of the key
+	pub new_name: String,
+}
+
+#[derive(Serialize, Deserialize, StructOpt, Debug)]
+pub struct KeyDeleteOpt {
+	/// ID or name of the key
+	pub key_pattern: String,
+
+	/// Confirm deletion
+	#[structopt(long = "yes")]
+	pub yes: bool,
+}
+
+#[derive(Serialize, Deserialize, StructOpt, Debug)]
+pub struct KeyImportOpt {
+	/// Access key ID
+	pub key_id: String,
+
+	/// Secret access key
+	pub secret_key: String,
+
+	/// Key name
+	#[structopt(short = "n", default_value = "Imported key")]
+	pub name: String,
+}
+
+#[derive(Serialize, Deserialize, StructOpt, Debug, Clone)]
+pub struct RepairOpt {
+	/// Launch repair operation on all nodes
+	#[structopt(short = "a", long = "all-nodes")]
+	pub all_nodes: bool,
+
+	/// Confirm the launch of the repair operation
+	#[structopt(long = "yes")]
+	pub yes: bool,
+
+	#[structopt(subcommand)]
+	pub what: Option<RepairWhat>,
+}
+
+#[derive(Serialize, Deserialize, StructOpt, Debug, Eq, PartialEq, Clone)]
+pub enum RepairWhat {
+	/// Only do a full sync of metadata tables
+	#[structopt(name = "tables")]
+	Tables,
+	/// Only repair (resync/rebalance) the set of stored blocks
+	#[structopt(name = "blocks")]
+	Blocks,
+	/// Only redo the propagation of object deletions to the version table (slow)
+	#[structopt(name = "versions")]
+	Versions,
+	/// Only redo the propagation of version deletions to the block ref table (extremely slow)
+	#[structopt(name = "block_refs")]
+	BlockRefs,
+}
+
+#[derive(Serialize, Deserialize, StructOpt, Debug, Clone)]
+pub struct StatsOpt {
+	/// Gather statistics from all nodes
+	#[structopt(short = "a", long = "all-nodes")]
+	pub all_nodes: bool,
+
+	/// Gather detailed statistics (this can be long)
+	#[structopt(short = "d", long = "detailed")]
+	pub detailed: bool,
+}
+
+pub async fn cli_cmd(
+	cmd: Command,
+	membership_rpc_cli: RpcAddrClient<Message>,
+	admin_rpc_cli: RpcAddrClient<AdminRPC>,
+	rpc_host: SocketAddr,
+) -> Result<(), Error> {
+	match cmd {
+		Command::Status => cmd_status(membership_rpc_cli, rpc_host).await,
+		Command::Node(NodeOperation::Configure(configure_opt)) => {
+			cmd_configure(membership_rpc_cli, rpc_host, configure_opt).await
+		}
+		Command::Node(NodeOperation::Remove(remove_opt)) => {
+			cmd_remove(membership_rpc_cli, rpc_host, remove_opt).await
+		}
+		Command::Bucket(bo) => {
+			cmd_admin(admin_rpc_cli, rpc_host, AdminRPC::BucketOperation(bo)).await
+		}
+		Command::Key(ko) => cmd_admin(admin_rpc_cli, rpc_host, AdminRPC::KeyOperation(ko)).await,
+		Command::Repair(ro) => cmd_admin(admin_rpc_cli, rpc_host, AdminRPC::LaunchRepair(ro)).await,
+		Command::Stats(so) => cmd_admin(admin_rpc_cli, rpc_host, AdminRPC::Stats(so)).await,
+		_ => unreachable!(),
+	}
+}
+
+pub async fn cmd_status(
+	rpc_cli: RpcAddrClient<Message>,
+	rpc_host: SocketAddr,
+) -> Result<(), Error> {
+	let status = match rpc_cli
+		.call(&rpc_host, &Message::PullStatus, ADMIN_RPC_TIMEOUT)
+		.await??
+	{
+		Message::AdvertiseNodesUp(nodes) => nodes,
+		resp => return Err(Error::Message(format!("Invalid RPC response: {:?}", resp))),
+	};
+	let config = match rpc_cli
+		.call(&rpc_host, &Message::PullConfig, ADMIN_RPC_TIMEOUT)
+		.await??
+	{
+		Message::AdvertiseConfig(cfg) => cfg,
+		resp => return Err(Error::Message(format!("Invalid RPC response: {:?}", resp))),
+	};
+
+	println!("Healthy nodes:");
+	for adv in status.iter().filter(|x| x.is_up) {
+		if let Some(cfg) = config.members.get(&adv.id) {
+			println!(
+				"{:?}\t{}\t{}\t[{}]\t{}\t{}",
+				adv.id, adv.state_info.hostname, adv.addr, cfg.tag, cfg.datacenter, cfg.capacity
+			);
+		} else {
+			println!(
+				"{:?}\t{}\t{}\tUNCONFIGURED/REMOVED",
+				adv.id, adv.state_info.hostname, adv.addr
+			);
+		}
+	}
+
+	let status_keys = status.iter().map(|x| x.id).collect::<HashSet<_>>();
+	let failure_case_1 = status.iter().any(|x| !x.is_up);
+	let failure_case_2 = config
+		.members
+		.iter()
+		.any(|(id, _)| !status_keys.contains(id));
+	if failure_case_1 || failure_case_2 {
+		println!("\nFailed nodes:");
+		for adv in status.iter().filter(|x| !x.is_up) {
+			if let Some(cfg) = config.members.get(&adv.id) {
+				println!(
+					"{:?}\t{}\t{}\t[{}]\t{}\t{}\tlast seen: {}s ago",
+					adv.id,
+					adv.state_info.hostname,
+					adv.addr,
+					cfg.tag,
+					cfg.datacenter,
+					cfg.capacity,
+					(now_msec() - adv.last_seen) / 1000,
+				);
+			}
+		}
+		for (id, cfg) in config.members.iter() {
+			if !status.iter().any(|x| x.id == *id) {
+				println!(
+					"{:?}\t{}\t{}\t{}\tnever seen",
+					id, cfg.tag, cfg.datacenter, cfg.capacity
+				);
+			}
+		}
+	}
+
+	Ok(())
+}
+
+pub fn find_matching_node(cand: impl std::iter::Iterator<Item=UUID>, pattern: &str) -> Result<UUID, Error> {
+	let mut candidates = vec![];
+	for c in cand {
+		if hex::encode(&c).starts_with(&pattern) {
+			candidates.push(c);
+		}
+	}
+	if candidates.len() != 1 {
+		Err(Error::Message(format!(
+			"{} nodes match '{}'",
+			candidates.len(),
+			pattern,
+		)))
+	} else {
+		Ok(candidates[0])
+	}
+}
+
+pub async fn cmd_configure(
+	rpc_cli: RpcAddrClient<Message>,
+	rpc_host: SocketAddr,
+	args: ConfigureNodeOpt,
+) -> Result<(), Error> {
+	let status = match rpc_cli
+		.call(&rpc_host, &Message::PullStatus, ADMIN_RPC_TIMEOUT)
+		.await??
+	{
+		Message::AdvertiseNodesUp(nodes) => nodes,
+		resp => return Err(Error::Message(format!("Invalid RPC response: {:?}", resp))),
+	};
+
+	let added_node = find_matching_node(status.iter().map(|x| x.id), &args.node_id)?;
+
+	let mut config = match rpc_cli
+		.call(&rpc_host, &Message::PullConfig, ADMIN_RPC_TIMEOUT)
+		.await??
+	{
+		Message::AdvertiseConfig(cfg) => cfg,
+		resp => return Err(Error::Message(format!("Invalid RPC response: {:?}", resp))),
+	};
+
+	for replaced in args.replace.iter() {
+		let replaced_node = find_matching_node(config.members.keys().cloned(), replaced)?;
+		if config.members.remove(&replaced_node).is_none() {
+			return Err(Error::Message(format!("Cannot replace node {:?} as it is not in current configuration", replaced_node)));
+		}
+	}
+
+	let new_entry = match config.members.get(&added_node) {
+		None => NetworkConfigEntry {
+			datacenter: args
+				.datacenter
+				.expect("Please specifiy a datacenter with the -d flag"),
+			capacity: args
+				.capacity
+				.expect("Please specifiy a capacity with the -c flag"),
+			tag: args.tag.unwrap_or("".to_string()),
+		},
+		Some(old) => NetworkConfigEntry {
+			datacenter: args.datacenter.unwrap_or(old.datacenter.to_string()),
+			capacity: args.capacity.unwrap_or(old.capacity),
+			tag: args.tag.unwrap_or(old.tag.to_string()),
+		},
+	};
+
+	config.members.insert(added_node, new_entry);
+	config.version += 1;
+
+	rpc_cli
+		.call(
+			&rpc_host,
+			&Message::AdvertiseConfig(config),
+			ADMIN_RPC_TIMEOUT,
+		)
+		.await??;
+	Ok(())
+}
+
+pub async fn cmd_remove(
+	rpc_cli: RpcAddrClient<Message>,
+	rpc_host: SocketAddr,
+	args: RemoveNodeOpt,
+) -> Result<(), Error> {
+	let mut config = match rpc_cli
+		.call(&rpc_host, &Message::PullConfig, ADMIN_RPC_TIMEOUT)
+		.await??
+	{
+		Message::AdvertiseConfig(cfg) => cfg,
+		resp => return Err(Error::Message(format!("Invalid RPC response: {:?}", resp))),
+	};
+
+	let deleted_node = find_matching_node(config.members.keys().cloned(), &args.node_id)?;
+
+	if !args.yes {
+		return Err(Error::Message(format!(
+			"Add the flag --yes to really remove {:?} from the cluster",
+			deleted_node
+		)));
+	}
+
+	config.members.remove(&deleted_node);
+	config.version += 1;
+
+	rpc_cli
+		.call(
+			&rpc_host,
+			&Message::AdvertiseConfig(config),
+			ADMIN_RPC_TIMEOUT,
+		)
+		.await??;
+	Ok(())
+}
+
+pub async fn cmd_admin(
+	rpc_cli: RpcAddrClient<AdminRPC>,
+	rpc_host: SocketAddr,
+	args: AdminRPC,
+) -> Result<(), Error> {
+	match rpc_cli.call(&rpc_host, args, ADMIN_RPC_TIMEOUT).await?? {
+		AdminRPC::Ok(msg) => {
+			println!("{}", msg);
+		}
+		AdminRPC::BucketList(bl) => {
+			println!("List of buckets:");
+			for bucket in bl {
+				println!("{}", bucket);
+			}
+		}
+		AdminRPC::BucketInfo(bucket) => {
+			print_bucket_info(&bucket);
+		}
+		AdminRPC::KeyList(kl) => {
+			println!("List of keys:");
+			for key in kl {
+				println!("{}\t{}", key.0, key.1);
+			}
+		}
+		AdminRPC::KeyInfo(key) => {
+			print_key_info(&key);
+		}
+		r => {
+			error!("Unexpected response: {:?}", r);
+		}
+	}
+	Ok(())
+}
+
+fn print_key_info(key: &Key) {
+	println!("Key name: {}", key.name.get());
+	println!("Key ID: {}", key.key_id);
+	println!("Secret key: {}", key.secret_key);
+	if key.deleted.get() {
+		println!("Key is deleted.");
+	} else {
+		println!("Authorized buckets:");
+		for (b, _, perm) in key.authorized_buckets.items().iter() {
+			println!("- {} R:{} W:{}", b, perm.allow_read, perm.allow_write);
+		}
+	}
+}
+
+fn print_bucket_info(bucket: &Bucket) {
+	println!("Bucket name: {}", bucket.name);
+	match bucket.state.get() {
+		BucketState::Deleted => println!("Bucket is deleted."),
+		BucketState::Present(p) => {
+			println!("Authorized keys:");
+			for (k, _, perm) in p.authorized_keys.items().iter() {
+				println!("- {} R:{} W:{}", k, perm.allow_read, perm.allow_write);
+			}
+			println!("Website access: {}", p.website.get());
+		}
+	};
+}
@@ -4,288 +4,67 @@
 extern crate log;

 mod admin_rpc;
+mod cli;
 mod repair;
 mod server;

-use std::collections::HashSet;
 use std::net::SocketAddr;
-use std::path::PathBuf;
 use std::sync::Arc;
 use std::time::Duration;

-use serde::{Deserialize, Serialize};
 use structopt::StructOpt;

 use garage_util::config::TlsConfig;
-use garage_util::data::*;
 use garage_util::error::Error;

 use garage_rpc::membership::*;
 use garage_rpc::rpc_client::*;

 use admin_rpc::*;
+use cli::*;

 #[derive(StructOpt, Debug)]
 #[structopt(name = "garage")]
 pub struct Opt {
 	/// RPC connect to this host to execute client operations
 	#[structopt(short = "h", long = "rpc-host", default_value = "127.0.0.1:3901")]
-	rpc_host: SocketAddr,
+	pub rpc_host: SocketAddr,

 	#[structopt(long = "ca-cert")]
-	ca_cert: Option<String>,
+	pub ca_cert: Option<String>,
 	#[structopt(long = "client-cert")]
-	client_cert: Option<String>,
+	pub client_cert: Option<String>,
 	#[structopt(long = "client-key")]
-	client_key: Option<String>,
+	pub client_key: Option<String>,

 	#[structopt(subcommand)]
 	cmd: Command,
 }

-#[derive(StructOpt, Debug)]
-pub enum Command {
-	/// Run Garage server
-	#[structopt(name = "server")]
-	Server(ServerOpt),
-
-	/// Get network status
-	#[structopt(name = "status")]
-	Status,
-
-	/// Garage node operations
-	#[structopt(name = "node")]
-	Node(NodeOperation),
-
-	/// Bucket operations
-	#[structopt(name = "bucket")]
-	Bucket(BucketOperation),
-
-	/// Key operations
-	#[structopt(name = "key")]
-	Key(KeyOperation),
-
-	/// Start repair of node data
-	#[structopt(name = "repair")]
-	Repair(RepairOpt),
-}
-
-#[derive(StructOpt, Debug)]
-pub struct ServerOpt {
-	/// Configuration file
-	#[structopt(short = "c", long = "config", default_value = "./config.toml")]
-	config_file: PathBuf,
-}
-
-#[derive(StructOpt, Debug)]
-pub enum NodeOperation {
-	/// Configure Garage node
-	#[structopt(name = "configure")]
-	Configure(ConfigureNodeOpt),
-
-	/// Remove Garage node from cluster
-	#[structopt(name = "remove")]
-	Remove(RemoveNodeOpt),
-}
-
-#[derive(StructOpt, Debug)]
-pub struct ConfigureNodeOpt {
-	/// Node to configure (prefix of hexadecimal node id)
-	node_id: String,
-
-	/// Location (datacenter) of the node
-	#[structopt(short = "d", long = "datacenter")]
-	datacenter: Option<String>,
-
-	/// Number of tokens
-	#[structopt(short = "n", long = "n-tokens")]
-	n_tokens: Option<u32>,
-
-	/// Optionnal node tag
-	#[structopt(short = "t", long = "tag")]
-	tag: Option<String>,
-}
-
-#[derive(StructOpt, Debug)]
-pub struct RemoveNodeOpt {
-	/// Node to configure (prefix of hexadecimal node id)
-	node_id: String,
-
-	/// If this flag is not given, the node won't be removed
-	#[structopt(long = "yes")]
-	yes: bool,
-}
-
-#[derive(Serialize, Deserialize, StructOpt, Debug)]
-pub enum BucketOperation {
-	/// List buckets
-	#[structopt(name = "list")]
-	List,
-
-	/// Get bucket info
-	#[structopt(name = "info")]
-	Info(BucketOpt),
-
-	/// Create bucket
-	#[structopt(name = "create")]
-	Create(BucketOpt),
-
-	/// Delete bucket
-	#[structopt(name = "delete")]
-	Delete(DeleteBucketOpt),
-
-	/// Allow key to read or write to bucket
-	#[structopt(name = "allow")]
-	Allow(PermBucketOpt),
-
-	/// Allow key to read or write to bucket
-	#[structopt(name = "deny")]
-	Deny(PermBucketOpt),
-
-	/// Expose as website or not
-	#[structopt(name = "website")]
-	Website(WebsiteOpt),
-}
-
-#[derive(Serialize, Deserialize, StructOpt, Debug)]
-pub struct WebsiteOpt {
-	/// Create
-	#[structopt(long = "allow")]
-	pub allow: bool,
-
-	/// Delete
-	#[structopt(long = "deny")]
-	pub deny: bool,
-
-	/// Bucket name
-	pub bucket: String,
-}
-
-#[derive(Serialize, Deserialize, StructOpt, Debug)]
-pub struct BucketOpt {
-	/// Bucket name
-	pub name: String,
-}
-
-#[derive(Serialize, Deserialize, StructOpt, Debug)]
-pub struct DeleteBucketOpt {
-	/// Bucket name
-	pub name: String,
-
-	/// If this flag is not given, the bucket won't be deleted
-	#[structopt(long = "yes")]
-	pub yes: bool,
-}
-
-#[derive(Serialize, Deserialize, StructOpt, Debug)]
-pub struct PermBucketOpt {
-	/// Access key ID
-	#[structopt(long = "key")]
-	pub key_id: String,
-
-	/// Allow/deny read operations
-	#[structopt(long = "read")]
-	pub read: bool,
-
-	/// Allow/deny write operations
-	#[structopt(long = "write")]
-	pub write: bool,
-
-	/// Bucket name
-	pub bucket: String,
-}
-
-#[derive(Serialize, Deserialize, StructOpt, Debug)]
-pub enum KeyOperation {
-	/// List keys
-	#[structopt(name = "list")]
-	List,
-
-	/// Get key info
-	#[structopt(name = "info")]
-	Info(KeyOpt),
-
-	/// Create new key
-	#[structopt(name = "new")]
-	New(KeyNewOpt),
-
-	/// Rename key
-	#[structopt(name = "rename")]
-	Rename(KeyRenameOpt),
-
-	/// Delete key
-	#[structopt(name = "delete")]
-	Delete(KeyDeleteOpt),
-}
-
-#[derive(Serialize, Deserialize, StructOpt, Debug)]
-pub struct KeyOpt {
-	/// ID of the key
-	key_id: String,
-}
-
-#[derive(Serialize, Deserialize, StructOpt, Debug)]
-pub struct KeyNewOpt {
-	/// Name of the key
-	#[structopt(long = "name", default_value = "Unnamed key")]
-	name: String,
-}
-
-#[derive(Serialize, Deserialize, StructOpt, Debug)]
-pub struct KeyRenameOpt {
-	/// ID of the key
-	key_id: String,
-
-	/// New name of the key
-	new_name: String,
-}
-
-#[derive(Serialize, Deserialize, StructOpt, Debug)]
-pub struct KeyDeleteOpt {
-	/// ID of the key
-	key_id: String,
-
-	/// Confirm deletion
-	#[structopt(long = "yes")]
-	yes: bool,
-}
-
-#[derive(Serialize, Deserialize, StructOpt, Debug, Clone)]
-pub struct RepairOpt {
-	/// Launch repair operation on all nodes
-	#[structopt(short = "a", long = "all-nodes")]
-	pub all_nodes: bool,
-
-	/// Confirm the launch of the repair operation
-	#[structopt(long = "yes")]
-	pub yes: bool,
-
-	#[structopt(subcommand)]
-	pub what: Option<RepairWhat>,
-}
-
-#[derive(Serialize, Deserialize, StructOpt, Debug, Eq, PartialEq, Clone)]
-pub enum RepairWhat {
-	/// Only do a full sync of metadata tables
-	#[structopt(name = "tables")]
-	Tables,
-	/// Only repair (resync/rebalance) the set of stored blocks
-	#[structopt(name = "blocks")]
-	Blocks,
-	/// Only redo the propagation of object deletions to the version table (slow)
-	#[structopt(name = "versions")]
-	Versions,
-	/// Only redo the propagation of version deletions to the block ref table (extremely slow)
-	#[structopt(name = "block_refs")]
-	BlockRefs,
-}
-
 #[tokio::main]
 async fn main() {
 	pretty_env_logger::init();

 	let opt = Opt::from_args();

+	let res = if let Command::Server(server_opt) = opt.cmd {
+		// Abort on panic (same behavior as in Go)
+		std::panic::set_hook(Box::new(|panic_info| {
+			error!("{}", panic_info.to_string());
+			std::process::abort();
+		}));
+
+		server::run_server(server_opt.config_file).await
+	} else {
+		cli_command(opt).await
+	};
+
+	if let Err(e) = res {
+		error!("{}", e);
+	}
+}
+
+async fn cli_command(opt: Opt) -> Result<(), Error> {
 	let tls_config = match (opt.ca_cert, opt.client_cert, opt.client_key) {
 		(Some(ca_cert), Some(client_cert), Some(client_key)) => Some(TlsConfig {
 			ca_cert,
@@ -305,245 +84,5 @@ async fn main() {
 		RpcAddrClient::new(rpc_http_cli.clone(), MEMBERSHIP_RPC_PATH.to_string());
 	let admin_rpc_cli = RpcAddrClient::new(rpc_http_cli.clone(), ADMIN_RPC_PATH.to_string());

-	let resp = match opt.cmd {
-		Command::Server(server_opt) => {
-			// Abort on panic (same behavior as in Go)
-			std::panic::set_hook(Box::new(|panic_info| {
-				error!("{}", panic_info.to_string());
-				std::process::abort();
-			}));
-
-			server::run_server(server_opt.config_file).await
-		}
-		Command::Status => cmd_status(membership_rpc_cli, opt.rpc_host).await,
-		Command::Node(NodeOperation::Configure(configure_opt)) => {
-			cmd_configure(membership_rpc_cli, opt.rpc_host, configure_opt).await
-		}
-		Command::Node(NodeOperation::Remove(remove_opt)) => {
-			cmd_remove(membership_rpc_cli, opt.rpc_host, remove_opt).await
-		}
-		Command::Bucket(bo) => {
-			cmd_admin(admin_rpc_cli, opt.rpc_host, AdminRPC::BucketOperation(bo)).await
-		}
-		Command::Key(bo) => {
-			cmd_admin(admin_rpc_cli, opt.rpc_host, AdminRPC::KeyOperation(bo)).await
-		}
-		Command::Repair(ro) => {
-			cmd_admin(admin_rpc_cli, opt.rpc_host, AdminRPC::LaunchRepair(ro)).await
-		}
-	};
-
-	if let Err(e) = resp {
-		error!("Error: {}", e);
-	}
-}
-
-async fn cmd_status(rpc_cli: RpcAddrClient<Message>, rpc_host: SocketAddr) -> Result<(), Error> {
-	let status = match rpc_cli
-		.call(&rpc_host, &Message::PullStatus, ADMIN_RPC_TIMEOUT)
-		.await??
-	{
-		Message::AdvertiseNodesUp(nodes) => nodes,
-		resp => return Err(Error::Message(format!("Invalid RPC response: {:?}", resp))),
-	};
-	let config = match rpc_cli
-		.call(&rpc_host, &Message::PullConfig, ADMIN_RPC_TIMEOUT)
-		.await??
-	{
-		Message::AdvertiseConfig(cfg) => cfg,
-		resp => return Err(Error::Message(format!("Invalid RPC response: {:?}", resp))),
-	};
-
-	println!("Healthy nodes:");
-	for adv in status.iter().filter(|x| x.is_up) {
-		if let Some(cfg) = config.members.get(&adv.id) {
-			println!(
-				"{:?}\t{}\t{}\t[{}]\t{}\t{}",
-				adv.id, adv.state_info.hostname, adv.addr, cfg.tag, cfg.datacenter, cfg.n_tokens
-			);
-		} else {
-			println!(
-				"{:?}\t{}\t{}\tUNCONFIGURED/REMOVED",
-				adv.id, adv.state_info.hostname, adv.addr
-			);
-		}
-	}
-
-	let status_keys = status.iter().map(|x| x.id).collect::<HashSet<_>>();
-	let failure_case_1 = status.iter().any(|x| !x.is_up);
-	let failure_case_2 = config
-		.members
-		.iter()
-		.any(|(id, _)| !status_keys.contains(id));
-	if failure_case_1 || failure_case_2 {
-		println!("\nFailed nodes:");
-		for adv in status.iter().filter(|x| !x.is_up) {
-			if let Some(cfg) = config.members.get(&adv.id) {
-				println!(
-					"{:?}\t{}\t{}\t[{}]\t{}\t{}\tlast seen: {}s ago",
-					adv.id,
-					adv.state_info.hostname,
-					adv.addr,
-					cfg.tag,
-					cfg.datacenter,
-					cfg.n_tokens,
-					(now_msec() - adv.last_seen) / 1000,
-				);
-			}
-		}
-		for (id, cfg) in config.members.iter() {
-			if !status.iter().any(|x| x.id == *id) {
-				println!(
-					"{:?}\t{}\t{}\t{}\tnever seen",
-					id, cfg.tag, cfg.datacenter, cfg.n_tokens
-				);
-			}
-		}
-	}
-
-	Ok(())
-}
-
-async fn cmd_configure(
-	rpc_cli: RpcAddrClient<Message>,
-	rpc_host: SocketAddr,
-	args: ConfigureNodeOpt,
-) -> Result<(), Error> {
-	let status = match rpc_cli
-		.call(&rpc_host, &Message::PullStatus, ADMIN_RPC_TIMEOUT)
-		.await??
-	{
-		Message::AdvertiseNodesUp(nodes) => nodes,
-		resp => return Err(Error::Message(format!("Invalid RPC response: {:?}", resp))),
-	};
-
-	let mut candidates = vec![];
-	for adv in status.iter() {
-		if hex::encode(&adv.id).starts_with(&args.node_id) {
-			candidates.push(adv.id);
-		}
-	}
-	if candidates.len() != 1 {
-		return Err(Error::Message(format!(
-			"{} matching nodes",
-			candidates.len()
-		)));
-	}
-
-	let mut config = match rpc_cli
-		.call(&rpc_host, &Message::PullConfig, ADMIN_RPC_TIMEOUT)
-		.await??
-	{
-		Message::AdvertiseConfig(cfg) => cfg,
-		resp => return Err(Error::Message(format!("Invalid RPC response: {:?}", resp))),
-	};
-
-	let new_entry = match config.members.get(&candidates[0]) {
-		None => NetworkConfigEntry {
-			datacenter: args
-				.datacenter
-				.expect("Please specifiy a datacenter with the -d flag"),
-			n_tokens: args
-				.n_tokens
-				.expect("Please specifiy a number of tokens with the -n flag"),
-			tag: args.tag.unwrap_or("".to_string()),
-		},
-		Some(old) => NetworkConfigEntry {
-			datacenter: args.datacenter.unwrap_or(old.datacenter.to_string()),
-			n_tokens: args.n_tokens.unwrap_or(old.n_tokens),
-			tag: args.tag.unwrap_or(old.tag.to_string()),
-		},
-	};
-
-	config.members.insert(candidates[0].clone(), new_entry);
-	config.version += 1;
-
-	rpc_cli
-		.call(
-			&rpc_host,
-			&Message::AdvertiseConfig(config),
-			ADMIN_RPC_TIMEOUT,
-		)
-		.await??;
-	Ok(())
-}
-
-async fn cmd_remove(
-	rpc_cli: RpcAddrClient<Message>,
-	rpc_host: SocketAddr,
-	args: RemoveNodeOpt,
-) -> Result<(), Error> {
-	let mut config = match rpc_cli
-		.call(&rpc_host, &Message::PullConfig, ADMIN_RPC_TIMEOUT)
-		.await??
-	{
-		Message::AdvertiseConfig(cfg) => cfg,
-		resp => return Err(Error::Message(format!("Invalid RPC response: {:?}", resp))),
-	};
-
-	let mut candidates = vec![];
-	for (key, _) in config.members.iter() {
-		if hex::encode(key).starts_with(&args.node_id) {
-			candidates.push(*key);
-		}
-	}
-	if candidates.len() != 1 {
-		return Err(Error::Message(format!(
-			"{} matching nodes",
-			candidates.len()
-		)));
-	}
-
-	if !args.yes {
-		return Err(Error::Message(format!(
-			"Add the flag --yes to really remove {:?} from the cluster",
-			candidates[0]
-		)));
-	}
-
-	config.members.remove(&candidates[0]);
-	config.version += 1;
-
-	rpc_cli
-		.call(
-			&rpc_host,
-			&Message::AdvertiseConfig(config),
-			ADMIN_RPC_TIMEOUT,
-		)
-		.await??;
-	Ok(())
-}
-
-async fn cmd_admin(
-	rpc_cli: RpcAddrClient<AdminRPC>,
-	rpc_host: SocketAddr,
-	args: AdminRPC,
-) -> Result<(), Error> {
-	match rpc_cli.call(&rpc_host, args, ADMIN_RPC_TIMEOUT).await?? {
-		AdminRPC::Ok(msg) => {
-			println!("{}", msg);
-		}
-		AdminRPC::BucketList(bl) => {
-			println!("List of buckets:");
-			for bucket in bl {
-				println!("{}", bucket);
-			}
-		}
-		AdminRPC::BucketInfo(bucket) => {
-			println!("{:?}", bucket);
-		}
-		AdminRPC::KeyList(kl) => {
-			println!("List of keys:");
-			for key in kl {
-				println!("{}\t{}", key.0, key.1);
-			}
-		}
-		AdminRPC::KeyInfo(key) => {
-			println!("{:?}", key);
-		}
-		r => {
-			error!("Unexpected response: {:?}", r);
-		}
-	}
-	Ok(())
+	cli_cmd(opt.cmd, membership_rpc_cli, admin_rpc_cli, opt.rpc_host).await
 }
@@ -16,7 +16,13 @@ pub struct Repair {
 }

 impl Repair {
-	pub async fn repair_worker(
+	pub async fn repair_worker(&self, opt: RepairOpt, must_exit: watch::Receiver<bool>) {
+		if let Err(e) = self.repair_worker_aux(opt, must_exit).await {
+			warn!("Repair worker failed with error: {}", e);
+		}
+	}
+
+	async fn repair_worker_aux(
 		&self,
 		opt: RepairOpt,
 		must_exit: watch::Receiver<bool>,
@@ -25,41 +31,11 @@ impl Repair {

 		if todo(RepairWhat::Tables) {
 			info!("Launching a full sync of tables");
-			self.garage
-				.bucket_table
-				.syncer
-				.load_full()
-				.unwrap()
-				.add_full_scan()
-				.await;
-			self.garage
-				.object_table
-				.syncer
-				.load_full()
-				.unwrap()
-				.add_full_scan()
-				.await;
-			self.garage
-				.version_table
-				.syncer
-				.load_full()
-				.unwrap()
-				.add_full_scan()
-				.await;
-			self.garage
-				.block_ref_table
-				.syncer
-				.load_full()
-				.unwrap()
-				.add_full_scan()
-				.await;
-			self.garage
-				.key_table
-				.syncer
-				.load_full()
-				.unwrap()
-				.add_full_scan()
-				.await;
+			self.garage.bucket_table.syncer.add_full_sync();
+			self.garage.object_table.syncer.add_full_sync();
+			self.garage.version_table.syncer.add_full_sync();
+			self.garage.block_ref_table.syncer.add_full_sync();
+			self.garage.key_table.syncer.add_full_sync();
 		}

 		// TODO: wait for full sync to finish before proceeding to the rest?
@@ -93,11 +69,13 @@ impl Repair {
 	async fn repair_versions(&self, must_exit: &watch::Receiver<bool>) -> Result<(), Error> {
 		let mut pos = vec![];

-		while let Some((item_key, item_bytes)) = self.garage.version_table.store.get_gt(&pos)? {
+		while let Some((item_key, item_bytes)) =
+			self.garage.version_table.data.store.get_gt(&pos)?
+		{
 			pos = item_key.to_vec();

 			let version = rmp_serde::decode::from_read_ref::<_, Version>(item_bytes.as_ref())?;
-			if version.deleted {
+			if version.deleted.get() {
 				continue;
 			}
 			let object = self
@@ -110,13 +88,7 @@ impl Repair {
 					.versions()
 					.iter()
 					.any(|x| x.uuid == version.uuid && x.state != ObjectVersionState::Aborted),
-				None => {
-					warn!(
-						"Repair versions: object for version {:?} not found, skipping.",
-						version
-					);
-					continue;
-				}
+				None => false,
 			};
 			if !version_exists {
 				info!("Repair versions: marking version as deleted: {:?}", version);
@@ -127,7 +99,6 @@ impl Repair {
 						version.bucket,
 						version.key,
 						true,
-						vec![],
 					))
 					.await?;
 			}
@@ -142,11 +113,13 @@ impl Repair {
 	async fn repair_block_ref(&self, must_exit: &watch::Receiver<bool>) -> Result<(), Error> {
 		let mut pos = vec![];

-		while let Some((item_key, item_bytes)) = self.garage.block_ref_table.store.get_gt(&pos)? {
+		while let Some((item_key, item_bytes)) =
+			self.garage.block_ref_table.data.store.get_gt(&pos)?
+		{
 			pos = item_key.to_vec();

 			let block_ref = rmp_serde::decode::from_read_ref::<_, BlockRef>(item_bytes.as_ref())?;
-			if block_ref.deleted {
+			if block_ref.deleted.get() {
 				continue;
 			}
 			let version = self
@@ -154,16 +127,8 @@ impl Repair {
 				.version_table
 				.get(&block_ref.version, &EmptyKey)
 				.await?;
-			let ref_exists = match version {
-				Some(v) => !v.deleted,
-				None => {
-					warn!(
-						"Block ref repair: version for block ref {:?} not found, skipping.",
-						block_ref
-					);
-					continue;
-				}
-			};
+			// The version might not exist if it has been GC'ed
+			let ref_exists = version.map(|v| !v.deleted.get()).unwrap_or(false);
 			if !ref_exists {
 				info!(
 					"Repair block ref: marking block_ref as deleted: {:?}",
@@ -174,7 +139,7 @@ impl Repair {
 					.insert(&BlockRef {
 						block: block_ref.block,
 						version: block_ref.version,
-						deleted: true,
+						deleted: true.into(),
 					})
 					.await?;
 			}
@@ -21,13 +21,13 @@ async fn shutdown_signal(send_cancel: watch::Sender<bool>) -> Result<(), Error>
 		.await
 		.expect("failed to install CTRL+C signal handler");
 	info!("Received CTRL+C, shutting down.");
-	send_cancel.broadcast(true)?;
+	send_cancel.send(true)?;
 	Ok(())
 }

 async fn wait_from(mut chan: watch::Receiver<bool>) -> () {
-	while let Some(exit_now) = chan.recv().await {
-		if exit_now {
+	while !*chan.borrow() {
+		if chan.changed().await.is_err() {
 			return;
 		}
 	}
@@ -40,16 +40,22 @@ pub async fn run_server(config_file: PathBuf) -> Result<(), Error> {
 	info!("Opening database...");
 	let mut db_path = config.metadata_dir.clone();
 	db_path.push("db");
-	let db = sled::open(db_path).expect("Unable to open DB");
+	let db = sled::open(&db_path).expect("Unable to open sled DB");

 	info!("Initialize RPC server...");
 	let mut rpc_server = RpcServer::new(config.rpc_bind_addr.clone(), config.rpc_tls.clone());

 	info!("Initializing background runner...");
 	let (send_cancel, watch_cancel) = watch::channel(false);
-	let background = BackgroundRunner::new(16, watch_cancel.clone());
+	let (background, await_background_done) = BackgroundRunner::new(16, watch_cancel.clone());

-	let garage = Garage::new(config, db, background.clone(), &mut rpc_server).await;
+	info!("Initializing Garage main data store...");
+	let garage = Garage::new(config.clone(), db, background, &mut rpc_server);
+	let bootstrap = garage.system.clone().bootstrap(
+		&config.bootstrap_peers[..],
+		config.consul_host,
+		config.consul_service_name,
+	);

 	info!("Crate admin RPC handler...");
 	AdminRpcHandler::new(garage.clone()).register_handler(&mut rpc_server);
@@ -57,21 +63,13 @@ pub async fn run_server(config_file: PathBuf) -> Result<(), Error> {
 	info!("Initializing RPC and API servers...");
 	let run_rpc_server = Arc::new(rpc_server).run(wait_from(watch_cancel.clone()));
 	let api_server = api_server::run_api_server(garage.clone(), wait_from(watch_cancel.clone()));
-	let web_server = web_server::run_web_server(garage.clone(), wait_from(watch_cancel.clone()));
+	let web_server = web_server::run_web_server(garage, wait_from(watch_cancel.clone()));

 	futures::try_join!(
-		garage
-			.system
-			.clone()
-			.bootstrap(
-				&garage.config.bootstrap_peers[..],
-				garage.config.consul_host.clone(),
-				garage.config.consul_service_name.clone()
-			)
-			.map(|rv| {
-				info!("Bootstrap done");
-				Ok(rv)
-			}),
+		bootstrap.map(|rv| {
+			info!("Bootstrap done");
+			Ok(rv)
+		}),
 		run_rpc_server.map(|rv| {
 			info!("RPC server exited");
 			rv
@@ -84,9 +82,9 @@ pub async fn run_server(config_file: PathBuf) -> Result<(), Error> {
 			info!("Web server exited");
 			rv
 		}),
-		background.run().map(|rv| {
-			info!("Background runner exited");
-			Ok(rv)
+		await_background_done.map(|rv| {
+			info!("Background runner exited: {:?}", rv);
+			Ok(())
 		}),
 		shutdown_signal(send_cancel),
 	)?;
@@ -1,9 +1,9 @@
 [package]
 name = "garage_model"
-version = "0.1.1"
+version = "0.2.1"
 authors = ["Alex Auvolat <alex@adnab.me>"]
 edition = "2018"
-license = "GPL-3.0"
+license = "AGPL-3.0"
 description = "Core data model for the Garage object store"
 repository = "https://git.deuxfleurs.fr/Deuxfleurs/garage"

@@ -13,26 +13,21 @@ path = "lib.rs"
 # See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

 [dependencies]
-garage_util = { version = "0.1.1", path = "../util" }
-garage_rpc = { version = "0.1.1", path = "../rpc" }
-garage_table = { version = "0.1.1", path = "../table" }
-model010 = { package = "garage_model_010b", version = "0.0.1" }
+garage_rpc = { version = "0.2.1", path = "../rpc" }
+garage_table = { version = "0.2.1", path = "../table" }
+garage_util = { version = "0.2.1", path = "../util" }

-bytes = "0.4"
-rand = "0.7"
-hex = "0.3"
-sha2 = "0.8"
-arc-swap = "0.4"
+arc-swap = "1.0"
+hex = "0.4"
 log = "0.4"
+rand = "0.8"

-sled = "0.31"
+sled = "0.34"

-rmp-serde = "0.14.3"
+rmp-serde = "0.15"
 serde = { version = "1.0", default-features = false, features = ["derive", "rc"] }
 serde_bytes = "0.11"

-async-trait = "0.1.30"
 futures = "0.3"
 futures-util = "0.3"
-tokio = { version = "0.2", default-features = false, features = ["rt-core", "rt-threaded", "io-driver", "net", "tcp", "time", "macros", "sync", "signal", "fs"] }
-
+tokio = { version = "1.0", default-features = false, features = ["rt", "rt-multi-thread", "io-util", "net", "time", "macros", "sync", "signal", "fs"] }
@@ -5,22 +5,20 @@ use std::time::Duration;
 use arc_swap::ArcSwapOption;
 use futures::future::*;
 use futures::select;
-use futures::stream::*;
 use serde::{Deserialize, Serialize};
 use tokio::fs;
-use tokio::prelude::*;
+use tokio::io::{AsyncReadExt, AsyncWriteExt};
 use tokio::sync::{watch, Mutex, Notify};

-use garage_util::data;
 use garage_util::data::*;
 use garage_util::error::Error;
+use garage_util::time::*;

 use garage_rpc::membership::System;
 use garage_rpc::rpc_client::*;
 use garage_rpc::rpc_server::*;

-use garage_table::table_sharded::TableShardedReplication;
-use garage_table::{DeletedFilter, TableReplication};
+use garage_table::replication::{sharded::TableShardedReplication, TableReplication};

 use crate::block_ref_table::*;

@@ -28,7 +26,10 @@ use crate::garage::Garage;

 pub const INLINE_THRESHOLD: usize = 3072;

+pub const BACKGROUND_WORKERS: u64 = 1;
+
 const BLOCK_RW_TIMEOUT: Duration = Duration::from_secs(42);
+const BLOCK_GC_TIMEOUT: Duration = Duration::from_secs(60);
 const NEED_BLOCK_QUERY_TIMEOUT: Duration = Duration::from_secs(5);
 const RESYNC_RETRY_TIMEOUT: Duration = Duration::from_secs(10);

@@ -56,14 +57,14 @@ pub struct BlockManager {
 	pub data_dir: PathBuf,
 	pub data_dir_lock: Mutex<()>,

-	pub rc: sled::Tree,
+	rc: sled::Tree,

-	pub resync_queue: sled::Tree,
-	pub resync_notify: Notify,
+	resync_queue: sled::Tree,
+	resync_notify: Notify,

-	pub system: Arc<System>,
+	system: Arc<System>,
 	rpc_client: Arc<RpcClient<Message>>,
-	pub garage: ArcSwapOption<Garage>,
+	pub(crate) garage: ArcSwapOption<Garage>,
 }

 impl BlockManager {
@@ -77,7 +78,6 @@ impl BlockManager {
 		let rc = db
 			.open_tree("block_local_rc")
 			.expect("Unable to open block_local_rc tree");
-		rc.set_merge_operator(rc_merge);

 		let resync_queue = db
 			.open_tree("block_local_resync_queue")
@@ -127,18 +127,16 @@ impl BlockManager {
 		}
 	}

-	pub async fn spawn_background_worker(self: Arc<Self>) {
+	pub fn spawn_background_worker(self: Arc<Self>) {
 		// Launch 2 simultaneous workers for background resync loop preprocessing
-		for i in 0..2usize {
+		for i in 0..BACKGROUND_WORKERS {
 			let bm2 = self.clone();
 			let background = self.system.background.clone();
 			tokio::spawn(async move {
-				tokio::time::delay_for(Duration::from_secs(10)).await;
-				background
-					.spawn_worker(format!("block resync worker {}", i), move |must_exit| {
-						bm2.resync_loop(must_exit)
-					})
-					.await;
+				tokio::time::sleep(Duration::from_secs(10 * (i + 1))).await;
+				background.spawn_worker(format!("block resync worker {}", i), move |must_exit| {
+					bm2.resync_loop(must_exit)
+				});
 			});
 		}
 	}
@@ -168,7 +166,7 @@ impl BlockManager {
 			Ok(f) => f,
 			Err(e) => {
 				// Not found but maybe we should have had it ??
-				self.put_to_resync(hash, 0)?;
+				self.put_to_resync(hash, Duration::from_millis(0))?;
 				return Err(Into::into(e));
 			}
 		};
@@ -176,11 +174,16 @@ impl BlockManager {
 		f.read_to_end(&mut data).await?;
 		drop(f);

-		if data::hash(&data[..]) != *hash {
+		if blake2sum(&data[..]) != *hash {
 			let _lock = self.data_dir_lock.lock().await;
-			warn!("Block {:?} is corrupted. Deleting and resyncing.", hash);
-			fs::remove_file(path).await?;
-			self.put_to_resync(&hash, 0)?;
+			warn!(
+				"Block {:?} is corrupted. Renaming to .corrupted and resyncing.",
+				hash
+			);
+			let mut path2 = path.clone();
+			path2.set_extension(".corrupted");
+			fs::rename(path, path2).await?;
+			self.put_to_resync(&hash, Duration::from_millis(0))?;
 			return Err(Error::CorruptData(*hash));
 		}

@@ -191,7 +194,7 @@ impl BlockManager {
 		let needed = self
 			.rc
 			.get(hash.as_ref())?
-			.map(|x| u64_from_bytes(x.as_ref()) > 0)
+			.map(|x| u64_from_be_bytes(x) > 0)
 			.unwrap_or(false);
 		if needed {
 			let path = self.block_path(hash);
@@ -215,84 +218,95 @@ impl BlockManager {
 	}

 	pub fn block_incref(&self, hash: &Hash) -> Result<(), Error> {
-		let old_rc = self.rc.get(&hash)?;
-		self.rc.merge(&hash, vec![1])?;
-		if old_rc.map(|x| u64_from_bytes(&x[..]) == 0).unwrap_or(true) {
-			self.put_to_resync(&hash, BLOCK_RW_TIMEOUT.as_millis() as u64)?;
+		let old_rc = self.rc.fetch_and_update(&hash, |old| {
+			let old_v = old.map(u64_from_be_bytes).unwrap_or(0);
+			Some(u64::to_be_bytes(old_v + 1).to_vec())
+		})?;
+		let old_rc = old_rc.map(u64_from_be_bytes).unwrap_or(0);
+		if old_rc == 0 {
+			self.put_to_resync(&hash, BLOCK_RW_TIMEOUT)?;
 		}
 		Ok(())
 	}

 	pub fn block_decref(&self, hash: &Hash) -> Result<(), Error> {
-		let new_rc = self.rc.merge(&hash, vec![0])?;
-		if new_rc.map(|x| u64_from_bytes(&x[..]) == 0).unwrap_or(true) {
-			self.put_to_resync(&hash, 0)?;
+		let new_rc = self.rc.update_and_fetch(&hash, |old| {
+			let old_v = old.map(u64_from_be_bytes).unwrap_or(0);
+			if old_v > 1 {
+				Some(u64::to_be_bytes(old_v - 1).to_vec())
+			} else {
+				None
+			}
+		})?;
+		if new_rc.is_none() {
+			self.put_to_resync(&hash, BLOCK_GC_TIMEOUT)?;
 		}
 		Ok(())
 	}

-	fn put_to_resync(&self, hash: &Hash, delay_millis: u64) -> Result<(), Error> {
-		let when = now_msec() + delay_millis;
+	fn put_to_resync(&self, hash: &Hash, delay: Duration) -> Result<(), Error> {
+		let when = now_msec() + delay.as_millis() as u64;
 		trace!("Put resync_queue: {} {:?}", when, hash);
 		let mut key = u64::to_be_bytes(when).to_vec();
 		key.extend(hash.as_ref());
 		self.resync_queue.insert(key, hash.as_ref())?;
-		self.resync_notify.notify();
+		self.resync_notify.notify_waiters();
 		Ok(())
 	}

-	async fn resync_loop(
-		self: Arc<Self>,
-		mut must_exit: watch::Receiver<bool>,
-	) -> Result<(), Error> {
-		let mut n_failures = 0usize;
+	async fn resync_loop(self: Arc<Self>, mut must_exit: watch::Receiver<bool>) {
 		while !*must_exit.borrow() {
-			if let Some((time_bytes, hash_bytes)) = self.resync_queue.pop_min()? {
-				let time_msec = u64_from_bytes(&time_bytes[0..8]);
-				let now = now_msec();
-				if now >= time_msec {
-					let mut hash = [0u8; 32];
-					hash.copy_from_slice(hash_bytes.as_ref());
-					let hash = Hash::from(hash);
-
-					if let Err(e) = self.resync_iter(&hash).await {
-						warn!("Failed to resync block {:?}, retrying later: {}", hash, e);
-						self.put_to_resync(&hash, RESYNC_RETRY_TIMEOUT.as_millis() as u64)?;
-						n_failures += 1;
-						if n_failures >= 10 {
-							warn!("Too many resync failures, throttling.");
-							tokio::time::delay_for(Duration::from_secs(1)).await;
-						}
-					} else {
-						n_failures = 0;
-					}
-				} else {
-					self.resync_queue.insert(time_bytes, hash_bytes)?;
-					let delay = tokio::time::delay_for(Duration::from_millis(time_msec - now));
-					select! {
-						_ = delay.fuse() => (),
-						_ = self.resync_notify.notified().fuse() => (),
-						_ = must_exit.recv().fuse() => (),
-					}
-				}
-			} else {
+			if let Err(e) = self.resync_iter(&mut must_exit).await {
+				warn!("Error in block resync loop: {}", e);
 				select! {
-					_ = self.resync_notify.notified().fuse() => (),
-					_ = must_exit.recv().fuse() => (),
+					_ = tokio::time::sleep(Duration::from_secs(1)).fuse() => (),
+					_ = must_exit.changed().fuse() => (),
 				}
 			}
 		}
+	}
+
+	async fn resync_iter(&self, must_exit: &mut watch::Receiver<bool>) -> Result<(), Error> {
+		if let Some(first_item) = self.resync_queue.iter().next() {
+			let (time_bytes, hash_bytes) = first_item?;
+			let time_msec = u64_from_be_bytes(&time_bytes[0..8]);
+			let now = now_msec();
+			if now >= time_msec {
+				let hash = Hash::try_from(&hash_bytes[..]).unwrap();
+				let res = self.resync_block(&hash).await;
+				if let Err(e) = &res {
+					warn!("Error when resyncing {:?}: {}", hash, e);
+					self.put_to_resync(&hash, RESYNC_RETRY_TIMEOUT)?;
+				}
+				self.resync_queue.remove(&time_bytes)?;
+				res?; // propagate error to delay main loop
+			} else {
+				let delay = tokio::time::sleep(Duration::from_millis(time_msec - now));
+				select! {
+					_ = delay.fuse() => (),
+					_ = self.resync_notify.notified().fuse() => (),
+					_ = must_exit.changed().fuse() => (),
+				}
+			}
+		} else {
+			select! {
+				_ = self.resync_notify.notified().fuse() => (),
+				_ = must_exit.changed().fuse() => (),
+			}
+		}
 		Ok(())
 	}

-	async fn resync_iter(&self, hash: &Hash) -> Result<(), Error> {
+	async fn resync_block(&self, hash: &Hash) -> Result<(), Error> {
+		let lock = self.data_dir_lock.lock().await;
+
 		let path = self.block_path(hash);

 		let exists = fs::metadata(&path).await.is_ok();
 		let needed = self
 			.rc
 			.get(hash.as_ref())?
-			.map(|x| u64_from_bytes(x.as_ref()) > 0)
+			.map(|x| u64_from_be_bytes(x) > 0)
 			.unwrap_or(false);

 		if exists != needed {
@@ -303,61 +317,67 @@ impl BlockManager {
 		}

 		if exists && !needed {
-			let garage = self.garage.load_full().unwrap();
-			let active_refs = garage
-				.block_ref_table
-				.get_range(&hash, None, Some(DeletedFilter::NotDeleted), 1)
-				.await?;
-			let needed_by_others = !active_refs.is_empty();
-			if needed_by_others {
-				let ring = self.system.ring.borrow().clone();
-				let who = self.replication.replication_nodes(&hash, &ring);
-				let msg = Arc::new(Message::NeedBlockQuery(*hash));
-				let who_needs_fut = who.iter().map(|to| {
-					self.rpc_client
-						.call_arc(*to, msg.clone(), NEED_BLOCK_QUERY_TIMEOUT)
-				});
-				let who_needs = join_all(who_needs_fut).await;
+			trace!("Offloading block {:?}", hash);

-				let mut need_nodes = vec![];
-				for (node, needed) in who.into_iter().zip(who_needs.iter()) {
-					match needed {
-						Ok(Message::NeedBlockReply(needed)) => {
-							if *needed {
-								need_nodes.push(node);
-							}
-						}
-						Err(e) => {
-							return Err(Error::Message(format!(
-								"Should delete block, but unable to confirm that all other nodes that need it have it: {}",
-								e
-							)));
-						}
-						Ok(_) => {
-							return Err(Error::Message(format!(
-								"Unexpected response to NeedBlockQuery RPC"
-							)));
+			let mut who = self.replication.write_nodes(&hash);
+			if who.len() < self.replication.write_quorum() {
+				return Err(Error::Message(format!("Not trying to offload block because we don't have a quorum of nodes to write to")));
+			}
+			who.retain(|id| *id != self.system.id);
+
+			let msg = Arc::new(Message::NeedBlockQuery(*hash));
+			let who_needs_fut = who.iter().map(|to| {
+				self.rpc_client
+					.call_arc(*to, msg.clone(), NEED_BLOCK_QUERY_TIMEOUT)
+			});
+			let who_needs_resps = join_all(who_needs_fut).await;
+
+			let mut need_nodes = vec![];
+			for (node, needed) in who.iter().zip(who_needs_resps.into_iter()) {
+				match needed? {
+					Message::NeedBlockReply(needed) => {
+						if needed {
+							need_nodes.push(*node);
 						}
 					}
-				}
-
-				if need_nodes.len() > 0 {
-					let put_block_message = self.read_block(hash).await?;
-					self.rpc_client
-						.try_call_many(
-							&need_nodes[..],
-							put_block_message,
-							RequestStrategy::with_quorum(need_nodes.len())
-								.with_timeout(BLOCK_RW_TIMEOUT),
-						)
-						.await?;
+					_ => {
+						return Err(Error::Message(format!(
+							"Unexpected response to NeedBlockQuery RPC"
+						)));
+					}
 				}
 			}
+
+			if need_nodes.len() > 0 {
+				trace!(
+					"Block {:?} needed by {} nodes, sending",
+					hash,
+					need_nodes.len()
+				);
+
+				let put_block_message = self.read_block(hash).await?;
+				self.rpc_client
+					.try_call_many(
+						&need_nodes[..],
+						put_block_message,
+						RequestStrategy::with_quorum(need_nodes.len())
+							.with_timeout(BLOCK_RW_TIMEOUT),
+					)
+					.await?;
+			}
+			info!(
+				"Deleting block {:?}, offload finished ({} / {})",
+				hash,
+				need_nodes.len(),
+				who.len()
+			);
+
 			fs::remove_file(path).await?;
-			self.resync_queue.remove(&hash)?;
 		}

 		if needed && !exists {
+			drop(lock);
+
 			// TODO find a way to not do this if they are sending it to us
 			// Let's suppose this isn't an issue for now with the BLOCK_RW_TIMEOUT delay
 			// between the RC being incremented and this part being called.
@@ -369,7 +389,7 @@ impl BlockManager {
 	}

 	pub async fn rpc_get_block(&self, hash: &Hash) -> Result<Vec<u8>, Error> {
-		let who = self.replication.read_nodes(&hash, &self.system);
+		let who = self.replication.read_nodes(&hash);
 		let resps = self
 			.rpc_client
 			.try_call_many(
@@ -393,7 +413,7 @@ impl BlockManager {
 	}

 	pub async fn rpc_put_block(&self, hash: Hash, data: Vec<u8>) -> Result<(), Error> {
-		let who = self.replication.write_nodes(&hash, &self.system);
+		let who = self.replication.write_nodes(&hash);
 		self.rpc_client
 			.try_call_many(
 				&who[..],
@@ -410,15 +430,15 @@ impl BlockManager {
 		let garage = self.garage.load_full().unwrap();
 		let mut last_hash = None;
 		let mut i = 0usize;
-		for entry in garage.block_ref_table.store.iter() {
+		for entry in garage.block_ref_table.data.store.iter() {
 			let (_k, v_bytes) = entry?;
 			let block_ref = rmp_serde::decode::from_read_ref::<_, BlockRef>(v_bytes.as_ref())?;
 			if Some(&block_ref.block) == last_hash.as_ref() {
 				continue;
 			}
-			if !block_ref.deleted {
+			if !block_ref.deleted.get() {
 				last_hash = Some(block_ref.block);
-				self.put_to_resync(&block_ref.block, 0)?;
+				self.put_to_resync(&block_ref.block, Duration::from_secs(0))?;
 			}
 			i += 1;
 			if i & 0xFF == 0 && *must_exit.borrow() {
@@ -427,80 +447,69 @@ impl BlockManager {
 		}

 		// 2. Repair blocks actually on disk
-		let mut ls_data_dir = fs::read_dir(&self.data_dir).await?;
-		while let Some(data_dir_ent) = ls_data_dir.next().await {
-			let data_dir_ent = data_dir_ent?;
-			let dir_name = data_dir_ent.file_name();
-			let dir_name = match dir_name.into_string() {
-				Ok(x) => x,
-				Err(_) => continue,
-			};
-			if dir_name.len() != 2 || hex::decode(&dir_name).is_err() {
-				continue;
-			}
+		self.repair_aux_read_dir_rec(&self.data_dir, must_exit)
+			.await?;

-			let mut ls_data_dir_2 = match fs::read_dir(data_dir_ent.path()).await {
-				Err(e) => {
-					warn!(
-						"Warning: could not list dir {:?}: {}",
-						data_dir_ent.path().to_str(),
-						e
-					);
-					continue;
-				}
-				Ok(x) => x,
-			};
-			while let Some(file) = ls_data_dir_2.next().await {
-				let file = file?;
-				let file_name = file.file_name();
-				let file_name = match file_name.into_string() {
+		Ok(())
+	}
+
+	fn repair_aux_read_dir_rec<'a>(
+		&'a self,
+		path: &'a PathBuf,
+		must_exit: &'a watch::Receiver<bool>,
+	) -> BoxFuture<'a, Result<(), Error>> {
+		// Lists all blocks on disk and adds them to the resync queue.
+		// This allows us to find blocks we are storing but don't actually need,
+		// so that we can offload them if necessary and then delete them locally.
+		async move {
+			let mut ls_data_dir = fs::read_dir(path).await?;
+			loop {
+				let data_dir_ent = ls_data_dir.next_entry().await?;
+				let data_dir_ent = match data_dir_ent {
+					Some(x) => x,
+					None => break,
+				};
+				let name = data_dir_ent.file_name();
+				let name = match name.into_string() {
 					Ok(x) => x,
 					Err(_) => continue,
 				};
-				if file_name.len() != 64 {
-					continue;
+				let ent_type = data_dir_ent.file_type().await?;
+
+				if name.len() == 2 && hex::decode(&name).is_ok() && ent_type.is_dir() {
+					self.repair_aux_read_dir_rec(&data_dir_ent.path(), must_exit)
+						.await?;
+				} else if name.len() == 64 {
+					let hash_bytes = match hex::decode(&name) {
+						Ok(h) => h,
+						Err(_) => continue,
+					};
+					let mut hash = [0u8; 32];
+					hash.copy_from_slice(&hash_bytes[..]);
+					self.put_to_resync(&hash.into(), Duration::from_secs(0))?;
 				}
-				let hash_bytes = match hex::decode(&file_name) {
-					Ok(h) => h,
-					Err(_) => continue,
-				};
-				let mut hash = [0u8; 32];
-				hash.copy_from_slice(&hash_bytes[..]);
-				self.put_to_resync(&hash.into(), 0)?;

 				if *must_exit.borrow() {
-					return Ok(());
+					break;
 				}
 			}
+			Ok(())
 		}
-		Ok(())
+		.boxed()
+	}
+
+	pub fn resync_queue_len(&self) -> usize {
+		self.resync_queue.len()
+	}
+
+	pub fn rc_len(&self) -> usize {
+		self.rc.len()
 	}
 }

-fn u64_from_bytes(bytes: &[u8]) -> u64 {
-	assert!(bytes.len() == 8);
+fn u64_from_be_bytes<T: AsRef<[u8]>>(bytes: T) -> u64 {
+	assert!(bytes.as_ref().len() == 8);
 	let mut x8 = [0u8; 8];
-	x8.copy_from_slice(bytes);
+	x8.copy_from_slice(bytes.as_ref());
 	u64::from_be_bytes(x8)
 }
-
-fn rc_merge(_key: &[u8], old: Option<&[u8]>, new: &[u8]) -> Option<Vec<u8>> {
-	let old = old.map(u64_from_bytes).unwrap_or(0);
-	assert!(new.len() == 1);
-	let new = match new[0] {
-		0 => {
-			if old > 0 {
-				old - 1
-			} else {
-				0
-			}
-		}
-		1 => old + 1,
-		_ => unreachable!(),
-	};
-	if new == 0 {
-		None
-	} else {
-		Some(u64::to_be_bytes(new).to_vec())
-	}
-}
@@ -1,11 +1,9 @@
-use async_trait::async_trait;
 use serde::{Deserialize, Serialize};
 use std::sync::Arc;

-use garage_util::background::*;
 use garage_util::data::*;
-use garage_util::error::Error;

+use garage_table::crdt::CRDT;
 use garage_table::*;

 use crate::block::*;
@@ -19,7 +17,7 @@ pub struct BlockRef {
 	pub version: UUID,

 	// Keep track of deleted status
-	pub deleted: bool,
+	pub deleted: crdt::Bool,
 }

 impl Entry<Hash, UUID> for BlockRef {
@@ -29,40 +27,44 @@ impl Entry<Hash, UUID> for BlockRef {
 	fn sort_key(&self) -> &UUID {
 		&self.version
 	}
+	fn is_tombstone(&self) -> bool {
+		self.deleted.get()
+	}
+}

+impl CRDT for BlockRef {
 	fn merge(&mut self, other: &Self) {
-		if other.deleted {
-			self.deleted = true;
-		}
+		self.deleted.merge(&other.deleted);
 	}
 }

 pub struct BlockRefTable {
-	pub background: Arc<BackgroundRunner>,
 	pub block_manager: Arc<BlockManager>,
 }

-#[async_trait]
 impl TableSchema for BlockRefTable {
 	type P = Hash;
 	type S = UUID;
 	type E = BlockRef;
 	type Filter = DeletedFilter;

-	async fn updated(&self, old: Option<Self::E>, new: Option<Self::E>) -> Result<(), Error> {
+	fn updated(&self, old: Option<Self::E>, new: Option<Self::E>) {
 		let block = &old.as_ref().or(new.as_ref()).unwrap().block;
-		let was_before = old.as_ref().map(|x| !x.deleted).unwrap_or(false);
-		let is_after = new.as_ref().map(|x| !x.deleted).unwrap_or(false);
+		let was_before = old.as_ref().map(|x| !x.deleted.get()).unwrap_or(false);
+		let is_after = new.as_ref().map(|x| !x.deleted.get()).unwrap_or(false);
 		if is_after && !was_before {
-			self.block_manager.block_incref(block)?;
+			if let Err(e) = self.block_manager.block_incref(block) {
+				warn!("block_incref failed for block {:?}: {}", block, e);
+			}
 		}
 		if was_before && !is_after {
-			self.block_manager.block_decref(block)?;
+			if let Err(e) = self.block_manager.block_decref(block) {
+				warn!("block_decref failed for block {:?}: {}", block, e);
+			}
 		}
-		Ok(())
 	}

 	fn matches_filter(entry: &Self::E, filter: &Self::Filter) -> bool {
-		filter.apply(entry.deleted)
+		filter.apply(entry.deleted.get())
 	}
 }
@@ -1,18 +1,10 @@
-use async_trait::async_trait;
 use serde::{Deserialize, Serialize};

 use garage_table::crdt::CRDT;
 use garage_table::*;

-use garage_util::error::Error;
-
 use crate::key_table::PermissionSet;

-// We import the same file but in its version 0.1.0.
-// We can then access v0.1.0 data structures.
-// We use them to perform migrations.
-use model010::bucket_table as prev;
-
 /// A bucket is a collection of objects
 ///
 /// Its parameters are not directly accessible as:
@@ -92,7 +84,9 @@ impl Entry<EmptyKey, String> for Bucket {
 	fn sort_key(&self) -> &String {
 		&self.name
 	}
+}

+impl CRDT for Bucket {
 	fn merge(&mut self, other: &Self) {
 		self.state.merge(&other.state);
 	}
@@ -100,53 +94,13 @@ impl Entry<EmptyKey, String> for Bucket {

 pub struct BucketTable;

-#[async_trait]
 impl TableSchema for BucketTable {
 	type P = EmptyKey;
 	type S = String;
 	type E = Bucket;
 	type Filter = DeletedFilter;

-	async fn updated(&self, _old: Option<Self::E>, _new: Option<Self::E>) -> Result<(), Error> {
-		Ok(())
-	}
-
 	fn matches_filter(entry: &Self::E, filter: &Self::Filter) -> bool {
 		filter.apply(entry.is_deleted())
 	}
-
-	fn try_migrate(bytes: &[u8]) -> Option<Self::E> {
-		let old = match rmp_serde::decode::from_read_ref::<_, prev::Bucket>(bytes) {
-			Ok(x) => x,
-			Err(_) => return None,
-		};
-		if old.deleted {
-			Some(Bucket {
-				name: old.name,
-				state: crdt::LWW::migrate_from_raw(old.timestamp, BucketState::Deleted),
-			})
-		} else {
-			let mut keys = crdt::LWWMap::new();
-			for ak in old.authorized_keys() {
-				keys.merge(&crdt::LWWMap::migrate_from_raw_item(
-					ak.key_id.clone(),
-					ak.timestamp,
-					PermissionSet {
-						allow_read: ak.allow_read,
-						allow_write: ak.allow_write,
-					},
-				));
-			}
-
-			let params = BucketParams {
-				authorized_keys: keys,
-				website: crdt::LWW::new(false),
-			};
-
-			Some(Bucket {
-				name: old.name,
-				state: crdt::LWW::migrate_from_raw(old.timestamp, BucketState::Present(params)),
-			})
-		}
-	}
 }
@@ -7,8 +7,8 @@ use garage_rpc::membership::System;
 use garage_rpc::rpc_client::RpcHttpClient;
 use garage_rpc::rpc_server::RpcServer;

-use garage_table::table_fullcopy::*;
-use garage_table::table_sharded::*;
+use garage_table::replication::fullcopy::*;
+use garage_table::replication::sharded::*;
 use garage_table::*;

 use crate::block::*;
@@ -35,7 +35,7 @@ pub struct Garage {
 }

 impl Garage {
-	pub async fn new(
+	pub fn new(
 		config: Config,
 		db: sled::Db,
 		background: Arc<BackgroundRunner>,
@@ -54,21 +54,23 @@ impl Garage {
 		);

 		let data_rep_param = TableShardedReplication {
+			system: system.clone(),
 			replication_factor: config.data_replication_factor,
 			write_quorum: (config.data_replication_factor + 1) / 2,
 			read_quorum: 1,
 		};

 		let meta_rep_param = TableShardedReplication {
+			system: system.clone(),
 			replication_factor: config.meta_replication_factor,
 			write_quorum: (config.meta_replication_factor + 1) / 2,
 			read_quorum: (config.meta_replication_factor + 1) / 2,
 		};

-		let control_rep_param = TableFullReplication::new(
-			config.meta_epidemic_fanout,
-			(config.meta_epidemic_fanout + 1) / 2,
-		);
+		let control_rep_param = TableFullReplication {
+			system: system.clone(),
+			max_faults: config.control_write_max_faults,
+		};

 		info!("Initialize block manager...");
 		let block_manager = BlockManager::new(
@@ -82,7 +84,6 @@ impl Garage {
 		info!("Initialize block_ref_table...");
 		let block_ref_table = Table::new(
 			BlockRefTable {
-				background: background.clone(),
 				block_manager: block_manager.clone(),
 			},
 			data_rep_param.clone(),
@@ -90,8 +91,7 @@ impl Garage {
 			&db,
 			"block_ref".to_string(),
 			rpc_server,
-		)
-		.await;
+		);

 		info!("Initialize version_table...");
 		let version_table = Table::new(
@@ -104,8 +104,7 @@ impl Garage {
 			&db,
 			"version".to_string(),
 			rpc_server,
-		)
-		.await;
+		);

 		info!("Initialize object_table...");
 		let object_table = Table::new(
@@ -118,8 +117,7 @@ impl Garage {
 			&db,
 			"object".to_string(),
 			rpc_server,
-		)
-		.await;
+		);

 		info!("Initialize bucket_table...");
 		let bucket_table = Table::new(
@@ -129,8 +127,7 @@ impl Garage {
 			&db,
 			"bucket".to_string(),
 			rpc_server,
-		)
-		.await;
+		);

 		info!("Initialize key_table_table...");
 		let key_table = Table::new(
@@ -140,8 +137,7 @@ impl Garage {
 			&db,
 			"key".to_string(),
 			rpc_server,
-		)
-		.await;
+		);

 		info!("Initialize Garage...");
 		let garage = Arc::new(Self {
@@ -159,7 +155,7 @@ impl Garage {

 		info!("Start block manager background thread...");
 		garage.block_manager.garage.swap(Some(garage.clone()));
-		garage.block_manager.clone().spawn_background_worker().await;
+		garage.block_manager.clone().spawn_background_worker();

 		garage
 	}
@@ -1,13 +1,8 @@
-use async_trait::async_trait;
 use serde::{Deserialize, Serialize};

-use garage_table::crdt::CRDT;
+use garage_table::crdt::*;
 use garage_table::*;

-use garage_util::error::Error;
-
-use model010::key_table as prev;
-
 #[derive(PartialEq, Clone, Debug, Serialize, Deserialize)]
 pub struct Key {
 	// Primary key
@@ -39,6 +34,15 @@ impl Key {
 			authorized_buckets: crdt::LWWMap::new(),
 		}
 	}
+	pub fn import(key_id: &str, secret_key: &str, name: &str) -> Self {
+		Self {
+			key_id: key_id.to_string(),
+			secret_key: secret_key.to_string(),
+			name: crdt::LWW::new(name.to_string()),
+			deleted: crdt::Bool::new(false),
+			authorized_buckets: crdt::LWWMap::new(),
+		}
+	}
 	pub fn delete(key_id: String) -> Self {
 		Self {
 			key_id,
@@ -69,6 +73,10 @@ pub struct PermissionSet {
 	pub allow_write: bool,
 }

+impl AutoCRDT for PermissionSet {
+	const WARN_IF_DIFFERENT: bool = true;
+}
+
 impl Entry<EmptyKey, String> for Key {
 	fn partition_key(&self) -> &EmptyKey {
 		&EmptyKey
@@ -76,60 +84,43 @@ impl Entry<EmptyKey, String> for Key {
 	fn sort_key(&self) -> &String {
 		&self.key_id
 	}
+}

+impl CRDT for Key {
 	fn merge(&mut self, other: &Self) {
 		self.name.merge(&other.name);
 		self.deleted.merge(&other.deleted);

 		if self.deleted.get() {
 			self.authorized_buckets.clear();
-			return;
+		} else {
+			self.authorized_buckets.merge(&other.authorized_buckets);
 		}
-
-		self.authorized_buckets.merge(&other.authorized_buckets);
 	}
 }

 pub struct KeyTable;

-#[async_trait]
+#[derive(Clone, Debug, Serialize, Deserialize)]
+pub enum KeyFilter {
+	Deleted(DeletedFilter),
+	Matches(String),
+}
+
 impl TableSchema for KeyTable {
 	type P = EmptyKey;
 	type S = String;
 	type E = Key;
-	type Filter = DeletedFilter;
-
-	async fn updated(&self, _old: Option<Self::E>, _new: Option<Self::E>) -> Result<(), Error> {
-		Ok(())
-	}
+	type Filter = KeyFilter;

 	fn matches_filter(entry: &Self::E, filter: &Self::Filter) -> bool {
-		filter.apply(entry.deleted.get())
-	}
-
-	fn try_migrate(bytes: &[u8]) -> Option<Self::E> {
-		let old = match rmp_serde::decode::from_read_ref::<_, prev::Key>(bytes) {
-			Ok(x) => x,
-			Err(_) => return None,
-		};
-		let mut new = Self::E {
-			key_id: old.key_id.clone(),
-			secret_key: old.secret_key.clone(),
-			name: crdt::LWW::migrate_from_raw(old.name_timestamp, old.name.clone()),
-			deleted: crdt::Bool::new(old.deleted),
-			authorized_buckets: crdt::LWWMap::new(),
-		};
-		for ab in old.authorized_buckets() {
-			let it = crdt::LWWMap::migrate_from_raw_item(
-				ab.bucket.clone(),
-				ab.timestamp,
-				PermissionSet {
-					allow_read: ab.allow_read,
-					allow_write: ab.allow_write,
-				},
-			);
-			new.authorized_buckets.merge(&it);
+		match filter {
+			KeyFilter::Deleted(df) => df.apply(entry.deleted.get()),
+			KeyFilter::Matches(pat) => {
+				let pat = pat.to_lowercase();
+				entry.key_id.to_lowercase().starts_with(&pat)
+					|| entry.name.get().to_lowercase() == pat
+			}
 		}
-		Some(new)
 	}
 }
@@ -1,19 +1,16 @@
-use async_trait::async_trait;
 use serde::{Deserialize, Serialize};
 use std::collections::BTreeMap;
 use std::sync::Arc;

 use garage_util::background::BackgroundRunner;
 use garage_util::data::*;
-use garage_util::error::Error;

-use garage_table::table_sharded::*;
+use garage_table::crdt::*;
+use garage_table::replication::sharded::*;
 use garage_table::*;

 use crate::version_table::*;

-use model010::object_table as prev;
-
 #[derive(PartialEq, Clone, Debug, Serialize, Deserialize)]
 pub struct Object {
 	// Primary key
@@ -72,7 +69,7 @@ pub enum ObjectVersionState {
 	Aborted,
 }

-impl ObjectVersionState {
+impl CRDT for ObjectVersionState {
 	fn merge(&mut self, other: &Self) {
 		use ObjectVersionState::*;
 		match other {
@@ -93,37 +90,30 @@ impl ObjectVersionState {
 	}
 }

-#[derive(PartialEq, Clone, Debug, Serialize, Deserialize)]
+#[derive(PartialEq, Eq, PartialOrd, Ord, Clone, Debug, Serialize, Deserialize)]
 pub enum ObjectVersionData {
 	DeleteMarker,
 	Inline(ObjectVersionMeta, #[serde(with = "serde_bytes")] Vec<u8>),
 	FirstBlock(ObjectVersionMeta, Hash),
 }

-#[derive(PartialEq, Clone, Debug, Serialize, Deserialize)]
+impl AutoCRDT for ObjectVersionData {
+	const WARN_IF_DIFFERENT: bool = true;
+}
+
+#[derive(PartialEq, Eq, PartialOrd, Ord, Clone, Debug, Serialize, Deserialize)]
 pub struct ObjectVersionMeta {
 	pub headers: ObjectVersionHeaders,
 	pub size: u64,
 	pub etag: String,
 }

-#[derive(PartialEq, Clone, Debug, Serialize, Deserialize)]
+#[derive(PartialEq, Eq, PartialOrd, Ord, Clone, Debug, Serialize, Deserialize)]
 pub struct ObjectVersionHeaders {
 	pub content_type: String,
 	pub other: BTreeMap<String, String>,
 }

-impl ObjectVersionData {
-	fn merge(&mut self, b: &Self) {
-		if *self != *b {
-			warn!(
-				"Inconsistent object version data: {:?} (local) vs {:?} (remote)",
-				self, b
-			);
-		}
-	}
-}
-
 impl ObjectVersion {
 	fn cmp_key(&self) -> (u64, UUID) {
 		(self.timestamp, self.uuid)
@@ -156,8 +146,14 @@ impl Entry<String, String> for Object {
 	fn sort_key(&self) -> &String {
 		&self.key
 	}
+	fn is_tombstone(&self) -> bool {
+		self.versions.len() == 1 && self.versions[0].state == ObjectVersionState::Complete(ObjectVersionData::DeleteMarker)
+	}
+}

+impl CRDT for Object {
 	fn merge(&mut self, other: &Self) {
+		// Merge versions from other into here
 		for other_v in other.versions.iter() {
 			match self
 				.versions
@@ -171,6 +167,9 @@ impl Entry<String, String> for Object {
 				}
 			}
 		}
+
+		// Remove versions which are obsolete, i.e. those that come
+		// before the last version which .is_complete().
 		let last_complete = self
 			.versions
 			.iter()
@@ -191,96 +190,41 @@ pub struct ObjectTable {
 	pub version_table: Arc<Table<VersionTable, TableShardedReplication>>,
 }

-#[async_trait]
 impl TableSchema for ObjectTable {
 	type P = String;
 	type S = String;
 	type E = Object;
 	type Filter = DeletedFilter;

-	async fn updated(&self, old: Option<Self::E>, new: Option<Self::E>) -> Result<(), Error> {
+	fn updated(&self, old: Option<Self::E>, new: Option<Self::E>) {
 		let version_table = self.version_table.clone();
-		if let (Some(old_v), Some(new_v)) = (old, new) {
-			// Propagate deletion of old versions
-			for v in old_v.versions.iter() {
-				let newly_deleted = match new_v
-					.versions
-					.binary_search_by(|nv| nv.cmp_key().cmp(&v.cmp_key()))
-				{
-					Err(_) => true,
-					Ok(i) => {
-						new_v.versions[i].state == ObjectVersionState::Aborted
-							&& v.state != ObjectVersionState::Aborted
+		self.background.spawn(async move {
+			if let (Some(old_v), Some(new_v)) = (old, new) {
+				// Propagate deletion of old versions
+				for v in old_v.versions.iter() {
+					let newly_deleted = match new_v
+						.versions
+						.binary_search_by(|nv| nv.cmp_key().cmp(&v.cmp_key()))
+					{
+						Err(_) => true,
+						Ok(i) => {
+							new_v.versions[i].state == ObjectVersionState::Aborted
+								&& v.state != ObjectVersionState::Aborted
+						}
+					};
+					if newly_deleted {
+						let deleted_version =
+							Version::new(v.uuid, old_v.bucket.clone(), old_v.key.clone(), true);
+						version_table.insert(&deleted_version).await?;
 					}
-				};
-				if newly_deleted {
-					let deleted_version = Version::new(
-						v.uuid,
-						old_v.bucket.clone(),
-						old_v.key.clone(),
-						true,
-						vec![],
-					);
-					version_table.insert(&deleted_version).await?;
 				}
 			}
-		}
-		Ok(())
+			Ok(())
+		})
 	}

 	fn matches_filter(entry: &Self::E, filter: &Self::Filter) -> bool {
 		let deleted = !entry.versions.iter().any(|v| v.is_data());
 		filter.apply(deleted)
 	}
-
-	fn try_migrate(bytes: &[u8]) -> Option<Self::E> {
-		let old = match rmp_serde::decode::from_read_ref::<_, prev::Object>(bytes) {
-			Ok(x) => x,
-			Err(_) => return None,
-		};
-		let new_v = old
-			.versions()
-			.iter()
-			.map(migrate_version)
-			.collect::<Vec<_>>();
-		let new = Object::new(old.bucket.clone(), old.key.clone(), new_v);
-		Some(new)
-	}
-}
-
-fn migrate_version(old: &prev::ObjectVersion) -> ObjectVersion {
-	let headers = ObjectVersionHeaders {
-		content_type: old.mime_type.clone(),
-		other: BTreeMap::new(),
-	};
-	let meta = ObjectVersionMeta {
-		headers: headers.clone(),
-		size: old.size,
-		etag: "".to_string(),
-	};
-	let state = match old.state {
-		prev::ObjectVersionState::Uploading => ObjectVersionState::Uploading(headers),
-		prev::ObjectVersionState::Aborted => ObjectVersionState::Aborted,
-		prev::ObjectVersionState::Complete => match &old.data {
-			prev::ObjectVersionData::Uploading => ObjectVersionState::Uploading(headers),
-			prev::ObjectVersionData::DeleteMarker => {
-				ObjectVersionState::Complete(ObjectVersionData::DeleteMarker)
-			}
-			prev::ObjectVersionData::Inline(x) => {
-				ObjectVersionState::Complete(ObjectVersionData::Inline(meta, x.clone()))
-			}
-			prev::ObjectVersionData::FirstBlock(h) => {
-				let mut hash = [0u8; 32];
-				hash.copy_from_slice(h.as_ref());
-				ObjectVersionState::Complete(ObjectVersionData::FirstBlock(meta, Hash::from(hash)))
-			}
-		},
-	};
-	let mut uuid = [0u8; 32];
-	uuid.copy_from_slice(old.uuid.as_ref());
-	ObjectVersion {
-		uuid: UUID::from(uuid),
-		timestamp: old.timestamp,
-		state,
-	}
 }
@@ -1,12 +1,11 @@
-use async_trait::async_trait;
 use serde::{Deserialize, Serialize};
 use std::sync::Arc;

 use garage_util::background::BackgroundRunner;
 use garage_util::data::*;
-use garage_util::error::Error;

-use garage_table::table_sharded::*;
+use garage_table::crdt::*;
+use garage_table::replication::sharded::*;
 use garage_table::*;

 use crate::block_ref_table::*;
@@ -17,8 +16,11 @@ pub struct Version {
 	pub uuid: UUID,

 	// Actual data: the blocks for this version
-	pub deleted: bool,
-	blocks: Vec<VersionBlock>,
+	// In the case of a multipart upload, also store the etags
+	// of individual parts and check them when doing CompleteMultipartUpload
+	pub deleted: crdt::Bool,
+	pub blocks: crdt::Map<VersionBlockKey, VersionBlock>,
+	pub parts_etags: crdt::Map<u64, String>,

 	// Back link to bucket+key so that we can figure if
 	// this was deleted later on
@@ -27,56 +29,46 @@ pub struct Version {
 }

 impl Version {
-	pub fn new(
-		uuid: UUID,
-		bucket: String,
-		key: String,
-		deleted: bool,
-		blocks: Vec<VersionBlock>,
-	) -> Self {
-		let mut ret = Self {
+	pub fn new(uuid: UUID, bucket: String, key: String, deleted: bool) -> Self {
+		Self {
 			uuid,
-			deleted,
-			blocks: vec![],
+			deleted: deleted.into(),
+			blocks: crdt::Map::new(),
+			parts_etags: crdt::Map::new(),
 			bucket,
 			key,
-		};
-		for b in blocks {
-			ret.add_block(b)
-				.expect("Twice the same VersionBlock in Version constructor");
 		}
-		ret
-	}
-	/// Adds a block if it wasn't already present
-	pub fn add_block(&mut self, new: VersionBlock) -> Result<(), ()> {
-		match self
-			.blocks
-			.binary_search_by(|b| b.cmp_key().cmp(&new.cmp_key()))
-		{
-			Err(i) => {
-				self.blocks.insert(i, new);
-				Ok(())
-			}
-			Ok(_) => Err(()),
-		}
-	}
-	pub fn blocks(&self) -> &[VersionBlock] {
-		&self.blocks[..]
 	}
 }

-#[derive(PartialEq, Clone, Debug, Serialize, Deserialize)]
-pub struct VersionBlock {
+#[derive(PartialEq, Eq, Clone, Copy, Debug, Serialize, Deserialize)]
+pub struct VersionBlockKey {
 	pub part_number: u64,
 	pub offset: u64,
+}
+
+impl Ord for VersionBlockKey {
+	fn cmp(&self, other: &Self) -> std::cmp::Ordering {
+		self.part_number
+			.cmp(&other.part_number)
+			.then(self.offset.cmp(&other.offset))
+	}
+}
+
+impl PartialOrd for VersionBlockKey {
+	fn partial_cmp(&self, other: &Self) -> Option<std::cmp::Ordering> {
+		Some(self.cmp(other))
+	}
+}
+
+#[derive(PartialEq, Eq, Ord, PartialOrd, Clone, Copy, Debug, Serialize, Deserialize)]
+pub struct VersionBlock {
 	pub hash: Hash,
 	pub size: u64,
 }

-impl VersionBlock {
-	fn cmp_key(&self) -> (u64, u64) {
-		(self.part_number, self.offset)
-	}
+impl AutoCRDT for VersionBlock {
+	const WARN_IF_DIFFERENT: bool = true;
 }

 impl Entry<Hash, EmptyKey> for Version {
@@ -86,23 +78,21 @@ impl Entry<Hash, EmptyKey> for Version {
 	fn sort_key(&self) -> &EmptyKey {
 		&EmptyKey
 	}
+	fn is_tombstone(&self) -> bool {
+		self.deleted.get()
+	}
+}

+impl CRDT for Version {
 	fn merge(&mut self, other: &Self) {
-		if other.deleted {
-			self.deleted = true;
+		self.deleted.merge(&other.deleted);
+
+		if self.deleted.get() {
 			self.blocks.clear();
-		} else if !self.deleted {
-			for bi in other.blocks.iter() {
-				match self
-					.blocks
-					.binary_search_by(|x| x.cmp_key().cmp(&bi.cmp_key()))
-				{
-					Ok(_) => (),
-					Err(pos) => {
-						self.blocks.insert(pos, bi.clone());
-					}
-				}
-			}
+			self.parts_etags.clear();
+		} else {
+			self.blocks.merge(&other.blocks);
+			self.parts_etags.merge(&other.parts_etags);
 		}
 	}
 }
@@ -112,34 +102,36 @@ pub struct VersionTable {
 	pub block_ref_table: Arc<Table<BlockRefTable, TableShardedReplication>>,
 }

-#[async_trait]
 impl TableSchema for VersionTable {
 	type P = Hash;
 	type S = EmptyKey;
 	type E = Version;
 	type Filter = DeletedFilter;

-	async fn updated(&self, old: Option<Self::E>, new: Option<Self::E>) -> Result<(), Error> {
+	fn updated(&self, old: Option<Self::E>, new: Option<Self::E>) {
 		let block_ref_table = self.block_ref_table.clone();
-		if let (Some(old_v), Some(new_v)) = (old, new) {
-			// Propagate deletion of version blocks
-			if new_v.deleted && !old_v.deleted {
-				let deleted_block_refs = old_v
-					.blocks
-					.iter()
-					.map(|vb| BlockRef {
-						block: vb.hash,
-						version: old_v.uuid,
-						deleted: true,
-					})
-					.collect::<Vec<_>>();
-				block_ref_table.insert_many(&deleted_block_refs[..]).await?;
+		self.background.spawn(async move {
+			if let (Some(old_v), Some(new_v)) = (old, new) {
+				// Propagate deletion of version blocks
+				if new_v.deleted.get() && !old_v.deleted.get() {
+					let deleted_block_refs = old_v
+						.blocks
+						.items()
+						.iter()
+						.map(|(_k, vb)| BlockRef {
+							block: vb.hash,
+							version: old_v.uuid,
+							deleted: true.into(),
+						})
+						.collect::<Vec<_>>();
+					block_ref_table.insert_many(&deleted_block_refs[..]).await?;
+				}
 			}
-		}
-		Ok(())
+			Ok(())
+		})
 	}

 	fn matches_filter(entry: &Self::E, filter: &Self::Filter) -> bool {
-		filter.apply(entry.deleted)
+		filter.apply(entry.deleted.get())
 	}
 }
@@ -1,9 +1,9 @@
 [package]
 name = "garage_rpc"
-version = "0.1.1"
+version = "0.2.1"
 authors = ["Alex Auvolat <alex@adnab.me>"]
 edition = "2018"
-license = "GPL-3.0"
+license = "AGPL-3.0"
 description = "Cluster membership management and RPC protocol for the Garage object store"
 repository = "https://git.deuxfleurs.fr/Deuxfleurs/garage"

@@ -13,29 +13,28 @@ path = "lib.rs"
 # See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

 [dependencies]
-garage_util = { version = "0.1.1", path = "../util" }
+garage_util = { version = "0.2.1", path = "../util" }

-bytes = "0.4"
-rand = "0.7"
-hex = "0.3"
-sha2 = "0.8"
-arc-swap = "0.4"
+arc-swap = "1.0"
+bytes = "1.0"
 gethostname = "0.2"
+hex = "0.4"
 log = "0.4"

-rmp-serde = "0.14.3"
+rmp-serde = "0.15"
 serde = { version = "1.0", default-features = false, features = ["derive", "rc"] }
 serde_json = "1.0"

 futures = "0.3"
 futures-util = "0.3"
-tokio = { version = "0.2", default-features = false, features = ["rt-core", "rt-threaded", "io-driver", "net", "tcp", "time", "macros", "sync", "signal", "fs"] }
+tokio = { version = "1.0", default-features = false, features = ["rt", "rt-multi-thread", "io-util", "net", "time", "macros", "sync", "signal", "fs"] }
+tokio-stream = { version = "0.1", features = ["net"] }

 http = "0.2"
-hyper = "0.13"
-rustls = "0.17"
-tokio-rustls = "0.13"
-hyper-rustls = { version = "0.20", default-features = false }
+hyper = { version = "0.14", features = ["full"] }
+hyper-rustls = { version = "0.22", default-features = false }
+rustls = "0.19"
+tokio-rustls = "0.22"
 webpki = "0.21"


@@ -2,7 +2,10 @@
 extern crate log;

 pub mod consul;
+pub(crate) mod tls_util;
+
 pub mod membership;
+pub mod ring;
+
 pub mod rpc_client;
 pub mod rpc_server;
-pub(crate) mod tls_util;
@@ -1,6 +1,5 @@
 use std::collections::HashMap;
-use std::hash::Hash as StdHash;
-use std::hash::Hasher;
+use std::fmt::Write as FmtWrite;
 use std::io::{Read, Write};
 use std::net::{IpAddr, SocketAddr};
 use std::path::PathBuf;
@@ -12,16 +11,17 @@ use futures::future::join_all;
 use futures::select;
 use futures_util::future::*;
 use serde::{Deserialize, Serialize};
-use sha2::{Digest, Sha256};
-use tokio::prelude::*;
+use tokio::io::AsyncWriteExt;
 use tokio::sync::watch;
 use tokio::sync::Mutex;

 use garage_util::background::BackgroundRunner;
 use garage_util::data::*;
 use garage_util::error::Error;
+use garage_util::time::*;

 use crate::consul::get_consul_nodes;
+use crate::ring::*;
 use crate::rpc_client::*;
 use crate::rpc_server::*;

@@ -66,19 +66,6 @@ pub struct AdvertisedNode {
 	pub state_info: StateInfo,
 }

-#[derive(Clone, Debug, Serialize, Deserialize)]
-pub struct NetworkConfig {
-	pub members: HashMap<UUID, NetworkConfigEntry>,
-	pub version: u64,
-}
-
-#[derive(Clone, Debug, Serialize, Deserialize)]
-pub struct NetworkConfigEntry {
-	pub datacenter: String,
-	pub n_tokens: u32,
-	pub tag: String,
-}
-
 pub struct System {
 	pub id: UUID,

@@ -90,7 +77,7 @@ pub struct System {
 	rpc_http_client: Arc<RpcHttpClient>,
 	rpc_client: Arc<RpcClient<Message>>,

-	pub status: watch::Receiver<Arc<Status>>,
+	pub(crate) status: watch::Receiver<Arc<Status>>,
 	pub ring: watch::Receiver<Arc<Ring>>,

 	update_lock: Mutex<(watch::Sender<Arc<Status>>, watch::Sender<Arc<Ring>>)>,
@@ -123,20 +110,6 @@ pub struct StateInfo {
 	pub hostname: String,
 }

-#[derive(Clone)]
-pub struct Ring {
-	pub config: NetworkConfig,
-	pub ring: Vec<RingEntry>,
-	pub n_datacenters: usize,
-}
-
-#[derive(Clone, Debug)]
-pub struct RingEntry {
-	pub location: Hash,
-	pub node: UUID,
-	pub datacenter: u64,
-}
-
 impl Status {
 	fn handle_ping(&mut self, ip: IpAddr, info: &PingMessage) -> bool {
 		let addr = SocketAddr::new(ip, info.rpc_port);
@@ -162,96 +135,14 @@ impl Status {
 		let mut nodes = self.nodes.iter().collect::<Vec<_>>();
 		nodes.sort_unstable_by_key(|(id, _status)| *id);

-		let mut hasher = Sha256::new();
+		let mut nodes_txt = String::new();
 		debug!("Current set of pingable nodes: --");
 		for (id, status) in nodes {
 			debug!("{} {}", hex::encode(&id), status.addr);
-			hasher.input(format!("{} {}\n", hex::encode(&id), status.addr));
+			writeln!(&mut nodes_txt, "{} {}", hex::encode(&id), status.addr).unwrap();
 		}
 		debug!("END --");
-		self.hash
-			.as_slice_mut()
-			.copy_from_slice(&hasher.result()[..]);
-	}
-}
-
-impl Ring {
-	fn rebuild_ring(&mut self) {
-		let mut new_ring = vec![];
-		let mut datacenters = vec![];
-
-		for (id, config) in self.config.members.iter() {
-			let mut dc_hasher = std::collections::hash_map::DefaultHasher::new();
-			config.datacenter.hash(&mut dc_hasher);
-			let datacenter = dc_hasher.finish();
-
-			if !datacenters.contains(&datacenter) {
-				datacenters.push(datacenter);
-			}
-
-			for i in 0..config.n_tokens {
-				let location = hash(format!("{} {}", hex::encode(&id), i).as_bytes());
-
-				new_ring.push(RingEntry {
-					location: location.into(),
-					node: *id,
-					datacenter,
-				})
-			}
-		}
-
-		new_ring.sort_unstable_by(|x, y| x.location.cmp(&y.location));
-		self.ring = new_ring;
-		self.n_datacenters = datacenters.len();
-
-		// eprintln!("RING: --");
-		// for e in self.ring.iter() {
-		// 	eprintln!("{:?}", e);
-		// }
-		// eprintln!("END --");
-	}
-
-	pub fn walk_ring(&self, from: &Hash, n: usize) -> Vec<UUID> {
-		if n >= self.config.members.len() {
-			return self.config.members.keys().cloned().collect::<Vec<_>>();
-		}
-
-		let start = match self.ring.binary_search_by(|x| x.location.cmp(from)) {
-			Ok(i) => i,
-			Err(i) => {
-				if i == 0 {
-					self.ring.len() - 1
-				} else {
-					i - 1
-				}
-			}
-		};
-
-		self.walk_ring_from_pos(start, n)
-	}
-
-	fn walk_ring_from_pos(&self, start: usize, n: usize) -> Vec<UUID> {
-		if n >= self.config.members.len() {
-			return self.config.members.keys().cloned().collect::<Vec<_>>();
-		}
-
-		let mut ret = vec![];
-		let mut datacenters = vec![];
-
-		let mut delta = 0;
-		while ret.len() < n {
-			let i = (start + delta) % self.ring.len();
-			delta += 1;
-
-			if !datacenters.contains(&self.ring[i].datacenter) {
-				ret.push(self.ring[i].node);
-				datacenters.push(self.ring[i].datacenter);
-			} else if datacenters.len() == self.n_datacenters && !ret.contains(&self.ring[i].node) {
-				ret.push(self.ring[i].node);
-			}
-		}
-
-		ret
+		self.hash = blake2sum(nodes_txt.as_bytes());
 	}
 }

@@ -312,10 +203,7 @@ impl System {
 					"No valid previous network configuration stored ({}), starting fresh.",
 					e
 				);
-				NetworkConfig {
-					members: HashMap::new(),
-					version: 0,
-				}
+				NetworkConfig::new()
 			}
 		};
 		let mut status = Status {
@@ -331,12 +219,7 @@ impl System {
 				.unwrap_or("<invalid utf-8>".to_string()),
 		};

-		let mut ring = Ring {
-			config: net_config,
-			ring: Vec::new(),
-			n_datacenters: 0,
-		};
-		ring.rebuild_ring();
+		let ring = Ring::new(net_config);
 		let (update_ring, ring) = watch::channel(Arc::new(ring));

 		let rpc_path = MEMBERSHIP_RPC_PATH.to_string();
@@ -433,23 +316,17 @@ impl System {
 		self.clone().ping_nodes(bootstrap_peers).await;

 		let self2 = self.clone();
-		self.clone()
-			.background
+		self.background
 			.spawn_worker(format!("ping loop"), |stop_signal| {
-				self2.ping_loop(stop_signal).map(Ok)
-			})
-			.await;
+				self2.ping_loop(stop_signal)
+			});

 		if let (Some(consul_host), Some(consul_service_name)) = (consul_host, consul_service_name) {
 			let self2 = self.clone();
-			self.clone()
-				.background
+			self.background
 				.spawn_worker(format!("Consul loop"), |stop_signal| {
-					self2
-						.consul_loop(stop_signal, consul_host, consul_service_name)
-						.map(Ok)
-				})
-				.await;
+					self2.consul_loop(stop_signal, consul_host, consul_service_name)
+				});
 		}
 	}

@@ -517,7 +394,7 @@ impl System {
 		if has_changes {
 			status.recalculate_hash();
 		}
-		if let Err(e) = update_locked.0.broadcast(Arc::new(status)) {
+		if let Err(e) = update_locked.0.send(Arc::new(status)) {
 			error!("In ping_nodes: could not save status update ({})", e);
 		}
 		drop(update_locked);
@@ -543,7 +420,7 @@ impl System {
 		let status_hash = status.hash;
 		let config_version = self.ring.borrow().config.version;

-		update_locked.0.broadcast(Arc::new(status))?;
+		update_locked.0.send(Arc::new(status))?;
 		drop(update_locked);

 		if is_new || status_hash != ping.status_hash {
@@ -625,7 +502,7 @@ impl System {
 		if has_changed {
 			status.recalculate_hash();
 		}
-		update_lock.0.broadcast(Arc::new(status))?;
+		update_lock.0.send(Arc::new(status))?;
 		drop(update_lock);

 		if to_ping.len() > 0 {
@@ -641,12 +518,11 @@ impl System {
 		adv: &NetworkConfig,
 	) -> Result<Message, Error> {
 		let update_lock = self.update_lock.lock().await;
-		let mut ring: Ring = self.ring.borrow().as_ref().clone();
+		let ring: Arc<Ring> = self.ring.borrow().clone();

 		if adv.version > ring.config.version {
-			ring.config = adv.clone();
-			ring.rebuild_ring();
-			update_lock.1.broadcast(Arc::new(ring))?;
+			let ring = Ring::new(adv.clone());
+			update_lock.1.send(Arc::new(ring))?;
 			drop(update_lock);

 			self.background.spawn_cancellable(
@@ -662,7 +538,7 @@ impl System {

 	async fn ping_loop(self: Arc<Self>, mut stop_signal: watch::Receiver<bool>) {
 		loop {
-			let restart_at = tokio::time::delay_for(PING_INTERVAL);
+			let restart_at = tokio::time::sleep(PING_INTERVAL);

 			let status = self.status.borrow().clone();
 			let ping_addrs = status
@@ -676,10 +552,9 @@ impl System {

 			select! {
 				_ = restart_at.fuse() => (),
-				must_exit = stop_signal.recv().fuse() => {
-					match must_exit {
-						None | Some(true) => return,
-						_ => (),
+				_ = stop_signal.changed().fuse() => {
+					if *stop_signal.borrow() {
+						return;
 					}
 				}
 			}
@@ -692,8 +567,8 @@ impl System {
 		consul_host: String,
 		consul_service_name: String,
 	) {
-		loop {
-			let restart_at = tokio::time::delay_for(CONSUL_INTERVAL);
+		while !*stop_signal.borrow() {
+			let restart_at = tokio::time::sleep(CONSUL_INTERVAL);

 			match get_consul_nodes(&consul_host, &consul_service_name).await {
 				Ok(mut node_list) => {
@@ -707,12 +582,7 @@ impl System {

 			select! {
 				_ = restart_at.fuse() => (),
-				must_exit = stop_signal.recv().fuse() => {
-					match must_exit {
-						None | Some(true) => return,
-						_ => (),
-					}
-				}
+				_ = stop_signal.changed().fuse() => (),
 			}
 		}
 	}
@@ -0,0 +1,215 @@
+use std::collections::{HashMap, HashSet};
+use std::convert::TryInto;
+
+use serde::{Deserialize, Serialize};
+
+use garage_util::data::*;
+
+// A partition number is encoded on 16 bits,
+// i.e. we have up to 2**16 partitions.
+// (in practice we have exactly 2**PARTITION_BITS partitions)
+pub type Partition = u16;
+
+// TODO: make this constant parametrizable in the config file
+// For deployments with many nodes it might make sense to bump
+// it up to 10.
+// Maximum value : 16
+pub const PARTITION_BITS: usize = 8;
+
+const PARTITION_MASK_U16: u16 = ((1 << PARTITION_BITS) - 1) << (16 - PARTITION_BITS);
+
+// TODO: make this constant paraetrizable in the config file
+// (most deployments use a replication factor of 3, so...)
+pub const MAX_REPLICATION: usize = 3;
+
+#[derive(Clone, Debug, Serialize, Deserialize)]
+pub struct NetworkConfig {
+	pub members: HashMap<UUID, NetworkConfigEntry>,
+	pub version: u64,
+}
+
+impl NetworkConfig {
+	pub(crate) fn new() -> Self {
+		Self {
+			members: HashMap::new(),
+			version: 0,
+		}
+	}
+}
+
+#[derive(Clone, Debug, Serialize, Deserialize)]
+pub struct NetworkConfigEntry {
+	pub datacenter: String,
+	pub capacity: u32,
+	pub tag: String,
+}
+
+#[derive(Clone)]
+pub struct Ring {
+	pub config: NetworkConfig,
+	pub ring: Vec<RingEntry>,
+}
+
+#[derive(Clone, Debug)]
+pub struct RingEntry {
+	pub location: Hash,
+	pub nodes: [UUID; MAX_REPLICATION],
+}
+
+impl Ring {
+	pub(crate) fn new(config: NetworkConfig) -> Self {
+		// Create a vector of partition indices (0 to 2**PARTITION_BITS-1)
+		let partitions_idx = (0usize..(1usize << PARTITION_BITS)).collect::<Vec<_>>();
+
+		let datacenters = config
+			.members
+			.iter()
+			.map(|(_id, info)| info.datacenter.as_str())
+			.collect::<HashSet<&str>>();
+		let n_datacenters = datacenters.len();
+
+		// Prepare ring
+		let mut partitions: Vec<Vec<(&UUID, &NetworkConfigEntry)>> = partitions_idx
+			.iter()
+			.map(|_i| Vec::new())
+			.collect::<Vec<_>>();
+
+		// Create MagLev priority queues for each node
+		let mut queues = config
+			.members
+			.iter()
+			.map(|(node_id, node_info)| {
+				let mut parts = partitions_idx
+					.iter()
+					.map(|i| {
+						let part_data =
+							[&u16::to_be_bytes(*i as u16)[..], node_id.as_slice()].concat();
+						(*i, fasthash(&part_data[..]))
+					})
+					.collect::<Vec<_>>();
+				parts.sort_by_key(|(_i, h)| *h);
+				let parts_i = parts.iter().map(|(i, _h)| *i).collect::<Vec<_>>();
+				(node_id, node_info, parts_i, 0)
+			})
+			.collect::<Vec<_>>();
+
+		let max_capacity = config
+			.members
+			.iter()
+			.map(|(_, node_info)| node_info.capacity)
+			.fold(0, std::cmp::max);
+
+		// Fill up ring
+		for rep in 0..MAX_REPLICATION {
+			queues.sort_by_key(|(ni, _np, _q, _p)| {
+				let queue_data = [&u16::to_be_bytes(rep as u16)[..], ni.as_slice()].concat();
+				fasthash(&queue_data[..])
+			});
+
+			for (_, _, _, pos) in queues.iter_mut() {
+				*pos = 0;
+			}
+
+			let mut remaining = partitions_idx.len();
+			while remaining > 0 {
+				let remaining0 = remaining;
+				for i_round in 0..max_capacity {
+					for (node_id, node_info, q, pos) in queues.iter_mut() {
+						if i_round >= node_info.capacity {
+							continue;
+						}
+						for pos2 in *pos..q.len() {
+							let qv = q[pos2];
+							if partitions[qv].len() != rep {
+								continue;
+							}
+							let p_dcs = partitions[qv]
+								.iter()
+								.map(|(_id, info)| info.datacenter.as_str())
+								.collect::<HashSet<&str>>();
+							if (p_dcs.len() < n_datacenters
+								&& !p_dcs.contains(&node_info.datacenter.as_str()))
+								|| (p_dcs.len() == n_datacenters
+									&& !partitions[qv].iter().any(|(id, _i)| id == node_id))
+							{
+								partitions[qv].push((node_id, node_info));
+								remaining -= 1;
+								*pos = pos2 + 1;
+								break;
+							}
+						}
+					}
+				}
+				if remaining == remaining0 {
+					// No progress made, exit
+					warn!("Could not build ring, not enough nodes configured.");
+					return Self {
+						config,
+						ring: vec![],
+					};
+				}
+			}
+		}
+
+		let ring = partitions
+			.iter()
+			.enumerate()
+			.map(|(i, nodes)| {
+				let top = (i as u16) << (16 - PARTITION_BITS);
+				let mut hash = [0u8; 32];
+				hash[0..2].copy_from_slice(&u16::to_be_bytes(top)[..]);
+				let nodes = nodes.iter().map(|(id, _info)| **id).collect::<Vec<UUID>>();
+				RingEntry {
+					location: hash.into(),
+					nodes: nodes.try_into().unwrap(),
+				}
+			})
+			.collect::<Vec<_>>();
+
+		// eprintln!("RING: --");
+		// for e in ring.iter() {
+		// 	eprintln!("{:?}", e);
+		// }
+		// eprintln!("END --");
+
+		Self { config, ring }
+	}
+
+	pub fn partition_of(&self, from: &Hash) -> Partition {
+		let top = u16::from_be_bytes(from.as_slice()[0..2].try_into().unwrap());
+		top >> (16 - PARTITION_BITS)
+	}
+
+	pub fn partitions(&self) -> Vec<(Partition, Hash)> {
+		let mut ret = vec![];
+
+		for (i, entry) in self.ring.iter().enumerate() {
+			ret.push((i as u16, entry.location));
+		}
+		if ret.len() > 0 {
+			assert_eq!(ret[0].1, [0u8; 32].into());
+		}
+
+		ret
+	}
+
+	pub fn walk_ring(&self, from: &Hash, n: usize) -> Vec<UUID> {
+		if self.ring.len() != 1 << PARTITION_BITS {
+			warn!("Ring not yet ready, read/writes will be lost!");
+			return vec![];
+		}
+
+		let top = u16::from_be_bytes(from.as_slice()[0..2].try_into().unwrap());
+		let partition_idx = (top >> (16 - PARTITION_BITS)) as usize;
+		assert_eq!(partition_idx, self.partition_of(from) as usize);
+
+		let partition = &self.ring[partition_idx];
+
+		let partition_top =
+			u16::from_be_bytes(partition.location.as_slice()[0..2].try_into().unwrap());
+		assert_eq!(partition_top & PARTITION_MASK_U16, top & PARTITION_MASK_U16);
+
+		assert!(n <= partition.nodes.len());
+		partition.nodes[..n].iter().cloned().collect::<Vec<_>>()
+	}
+}
@@ -7,7 +7,6 @@ use std::sync::Arc;
 use std::time::Duration;

 use arc_swap::ArcSwapOption;
-use bytes::IntoBuf;
 use futures::future::Future;
 use futures::stream::futures_unordered::FuturesUnordered;
 use futures::stream::StreamExt;
@@ -197,11 +196,8 @@ impl<M: RpcMessage + 'static> RpcClient<M> {
 			if !strategy.rs_interrupt_after_quorum {
 				let wait_finished_fut = tokio::spawn(async move {
 					resp_stream.collect::<Vec<_>>().await;
-					Ok(())
 				});
-				self.background.spawn(wait_finished_fut.map(|x| {
-					x.unwrap_or_else(|e| Err(Error::Message(format!("Await failed: {}", e))))
-				}));
+				self.background.spawn(wait_finished_fut.map(|_| Ok(())));
 			}

 			Ok(results)
@@ -310,7 +306,9 @@ impl RpcHttpClient {
 			ClientMethod::HTTPS(client) => client.request(req).fuse(),
 		};

+		trace!("({}) Acquiring request_limiter slot...", path);
 		let slot = self.request_limiter.acquire().await;
+		trace!("({}) Got slot, doing request to {}...", path, to_addr);
 		let resp = tokio::time::timeout(timeout, resp_fut)
 			.await
 			.map_err(|e| {
@@ -330,10 +328,11 @@ impl RpcHttpClient {
 			})?;

 		let status = resp.status();
+		trace!("({}) Request returned, got status {}", path, status);
 		let body = hyper::body::to_bytes(resp.into_body()).await?;
 		drop(slot);

-		match rmp_serde::decode::from_read::<_, Result<M, String>>(body.into_buf())? {
+		match rmp_serde::decode::from_read::<_, Result<M, String>>(&body[..])? {
 			Err(e) => Ok(Err(Error::RemoteError(e, status))),
 			Ok(x) => Ok(Ok(x)),
 		}
@@ -4,7 +4,6 @@ use std::pin::Pin;
 use std::sync::Arc;
 use std::time::Instant;

-use bytes::IntoBuf;
 use futures::future::Future;
 use futures_util::future::*;
 use futures_util::stream::*;
@@ -15,6 +14,7 @@ use serde::{Deserialize, Serialize};
 use tokio::net::{TcpListener, TcpStream};
 use tokio_rustls::server::TlsStream;
 use tokio_rustls::TlsAcceptor;
+use tokio_stream::wrappers::TcpListenerStream;

 use garage_util::config::TlsConfig;
 use garage_util::data::*;
@@ -47,7 +47,17 @@ where
 {
 	let begin_time = Instant::now();
 	let whole_body = hyper::body::to_bytes(req.into_body()).await?;
-	let msg = rmp_serde::decode::from_read::<_, M>(whole_body.into_buf())?;
+	let msg = rmp_serde::decode::from_read::<_, M>(&whole_body[..])?;
+
+	trace!(
+		"Request message: {}",
+		serde_json::to_string(&msg)
+			.unwrap_or("<json error>".into())
+			.chars()
+			.take(100)
+			.collect::<String>()
+	);
+
 	match handler(msg, sockaddr).await {
 		Ok(resp) => {
 			let resp_bytes = rmp_to_vec_all_named::<Result<M, String>>(&Ok(resp))?;
@@ -112,7 +122,8 @@ impl RpcServer {
 			return Ok(bad_request);
 		}

-		let path = &req.uri().path()[1..];
+		let path = &req.uri().path()[1..].to_string();
+
 		let handler = match self.handlers.get(path) {
 			Some(h) => h,
 			None => {
@@ -122,6 +133,8 @@ impl RpcServer {
 			}
 		};

+		trace!("({}) Handling request", path);
+
 		let resp_waiter = tokio::spawn(handler(req, addr));
 		match resp_waiter.await {
 			Err(err) => {
@@ -131,11 +144,15 @@ impl RpcServer {
 				Ok(ise)
 			}
 			Ok(Err(err)) => {
+				trace!("({}) Request handler failed: {}", path, err);
 				let mut bad_request = Response::new(Body::from(format!("{}", err)));
 				*bad_request.status_mut() = StatusCode::BAD_REQUEST;
 				Ok(bad_request)
 			}
-			Ok(Ok(resp)) => Ok(resp),
+			Ok(Ok(resp)) => {
+				trace!("({}) Request handler succeeded", path);
+				Ok(resp)
+			}
 		}
 	}

@@ -158,8 +175,8 @@ impl RpcServer {
 			config.set_single_cert([&node_certs[..], &ca_certs[..]].concat(), node_key)?;
 			let tls_acceptor = Arc::new(TlsAcceptor::from(Arc::new(config)));

-			let mut listener = TcpListener::bind(&self.bind_addr).await?;
-			let incoming = listener.incoming().filter_map(|socket| async {
+			let listener = TcpListener::bind(&self.bind_addr).await?;
+			let incoming = TcpListenerStream::new(listener).filter_map(|socket| async {
 				match socket {
 					Ok(stream) => match tls_acceptor.clone().accept(stream).await {
 						Ok(x) => Some(Ok::<_, hyper::Error>(x)),
@@ -1,9 +1,9 @@
 [package]
 name = "garage_table"
-version = "0.1.1"
+version = "0.2.1"
 authors = ["Alex Auvolat <alex@adnab.me>"]
 edition = "2018"
-license = "GPL-3.0"
+license = "AGPL-3.0"
 description = "Table sharding and replication engine (DynamoDB-like) for the Garage object store"
 repository = "https://git.deuxfleurs.fr/Deuxfleurs/garage"

@@ -13,24 +13,21 @@ path = "lib.rs"
 # See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

 [dependencies]
-garage_util = { version = "0.1.1", path = "../util" }
-garage_rpc = { version = "0.1.1", path = "../rpc" }
+garage_rpc = { version = "0.2.1", path = "../rpc" }
+garage_util = { version = "0.2.1", path = "../util" }

-bytes = "0.4"
-rand = "0.7"
-hex = "0.3"
-arc-swap = "0.4"
-log = "0.4"
+bytes = "1.0"
 hexdump = "0.1"
+log = "0.4"
+rand = "0.8"

-sled = "0.31"
+sled = "0.34"

-rmp-serde = "0.14.3"
+rmp-serde = "0.15"
 serde = { version = "1.0", default-features = false, features = ["derive", "rc"] }
 serde_bytes = "0.11"

-async-trait = "0.1.30"
 futures = "0.3"
 futures-util = "0.3"
-tokio = { version = "0.2", default-features = false, features = ["rt-core", "rt-threaded", "io-driver", "net", "tcp", "time", "macros", "sync", "signal", "fs"] }
+tokio = { version = "1.0", default-features = false, features = ["rt", "rt-multi-thread", "io-util", "net", "time", "macros", "sync", "signal", "fs"] }

@@ -1,327 +0,0 @@
-//! This package provides a simple implementation of conflict-free replicated data types (CRDTs)
-//!
-//! CRDTs are a type of data structures that do not require coordination.  In other words, we can
-//! edit them in parallel, we will always find a way to merge it.
-//!
-//! A general example is a counter. Its initial value is 0.  Alice and Bob get a copy of the
-//! counter.  Alice does +1 on her copy, she reads 1.  Bob does +3 on his copy, he reads 3.  Now,
-//! it is easy to merge their counters, order does not count: we always get 4.
-//!
-//! Learn more about CRDT [on Wikipedia](https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type)
-
-use serde::{Deserialize, Serialize};
-
-use garage_util::data::*;
-
-/// Definition of a CRDT - all CRDT Rust types implement this.
-///
-/// A CRDT is defined as a merge operator that respects a certain set of axioms.
-///
-/// In particular, the merge operator must be commutative, associative,
-/// idempotent, and monotonic.
-/// In other words, if `a`, `b` and `c` are CRDTs, and `⊔` denotes the merge operator,
-/// the following axioms must apply:
-///
-/// ```text
-/// a ⊔ b = b ⊔ a                   (commutativity)
-/// (a ⊔ b) ⊔ c = a ⊔ (b ⊔ c)       (associativity)
-/// (a ⊔ b) ⊔ b = a ⊔ b             (idempotence)
-/// ```
-///
-/// Moreover, the relationship `≥` defined by `a ≥ b ⇔ ∃c. a = b ⊔ c` must be a partial order.
-/// This implies a few properties such as: if `a ⊔ b ≠ a`, then there is no `c` such that `(a ⊔ b) ⊔ c = a`,
-/// as this would imply a cycle in the partial order.
-pub trait CRDT {
-	/// Merge the two datastructures according to the CRDT rules.
-	/// `self` is modified to contain the merged CRDT value. `other` is not modified.
-	///
-	/// # Arguments
-	///
-	/// * `other` - the other CRDT we wish to merge with
-	fn merge(&mut self, other: &Self);
-}
-
-/// All types that implement `Ord` (a total order) also implement a trivial CRDT
-/// defined by the merge rule: `a ⊔ b = max(a, b)`.
-impl<T> CRDT for T
-where
-	T: Ord + Clone,
-{
-	fn merge(&mut self, other: &Self) {
-		if other > self {
-			*self = other.clone();
-		}
-	}
-}
-
-// ---- LWW Register ----
-
-/// Last Write Win (LWW)
-///
-/// An LWW CRDT associates a timestamp with a value, in order to implement a
-/// time-based reconciliation rule: the most recent write wins.
-/// For completeness, the LWW reconciliation rule must also be defined for two LWW CRDTs
-/// with the same timestamp but different values.
-///
-/// In our case, we add the constraint that the value that is wrapped inside the LWW CRDT must
-/// itself be a CRDT: in the case when the timestamp does not allow us to decide on which value to
-/// keep, the merge rule of the inner CRDT is applied on the wrapped values.  (Note that all types
-/// that implement the `Ord` trait get a default CRDT implemetnation that keeps the maximum value.
-/// This enables us to use LWW directly with primitive data types such as numbers or strings. It is
-/// generally desirable in this case to never explicitly produce LWW values with the same timestamp
-/// but different inner values, as the rule to keep the maximum value isn't generally the desired
-/// semantics.)
-///
-/// As multiple computers clocks are always desynchronized,
-/// when operations are close enough, it is equivalent to
-/// take one copy and drop the other one.
-///
-/// Given that clocks are not too desynchronized, this assumption
-/// is enough for most cases, as there is few chance that two humans
-/// coordonate themself faster than the time difference between two NTP servers.
-///
-/// As a more concret example, let's suppose you want to upload a file
-/// with the same key (path) in the same bucket at the very same time.
-/// For each request, the file will be timestamped by the receiving server
-/// and may differ from what you observed with your atomic clock!
-///
-/// This scheme is used by AWS S3 or Soundcloud and often without knowing
-/// in entreprise when reconciliating databases with ad-hoc scripts.
-#[derive(Clone, Debug, Serialize, Deserialize, PartialEq)]
-pub struct LWW<T> {
-	ts: u64,
-	v: T,
-}
-
-impl<T> LWW<T>
-where
-	T: CRDT,
-{
-	/// Creates a new CRDT
-	///
-	/// CRDT's internal timestamp is set with current node's clock.
-	pub fn new(value: T) -> Self {
-		Self {
-			ts: now_msec(),
-			v: value,
-		}
-	}
-
-	/// Build a new CRDT from a previous non-compatible one
-	///
-	/// Compared to new, the CRDT's timestamp is not set to now
-	/// but must be set to the previous, non-compatible, CRDT's timestamp.
-	pub fn migrate_from_raw(ts: u64, value: T) -> Self {
-		Self { ts, v: value }
-	}
-
-	/// Update the LWW CRDT while keeping some causal ordering.
-	///
-	/// The timestamp of the LWW CRDT is updated to be the current node's clock
-	/// at time of update, or the previous timestamp + 1 if that's bigger,
-	/// so that the new timestamp is always strictly larger than the previous one.
-	/// This ensures that merging the update with the old value will result in keeping
-	/// the updated value.
-	pub fn update(&mut self, new_value: T) {
-		self.ts = std::cmp::max(self.ts + 1, now_msec());
-		self.v = new_value;
-	}
-
-	/// Get the CRDT value
-	pub fn get(&self) -> &T {
-		&self.v
-	}
-
-	/// Get a mutable reference to the CRDT's value
-	///
-	/// This is usefull to mutate the inside value without changing the LWW timestamp.
-	/// When such mutation is done, the merge between two LWW values is done using the inner
-	/// CRDT's merge operation. This is usefull in the case where the inner CRDT is a large
-	/// data type, such as a map, and we only want to change a single item in the map.
-	/// To do this, we can produce a "CRDT delta", i.e. a LWW that contains only the modification.
-	/// This delta consists in a LWW with the same timestamp, and the map
-	/// inside only contains the updated value.
-	/// The advantage of such a delta is that it is much smaller than the whole map.
-	///
-	/// Avoid using this if the inner data type is a primitive type such as a number or a string,
-	/// as you will then rely on the merge function defined on `Ord` types by keeping the maximum
-	/// of both values.
-	pub fn get_mut(&mut self) -> &mut T {
-		&mut self.v
-	}
-}
-
-impl<T> CRDT for LWW<T>
-where
-	T: Clone + CRDT,
-{
-	fn merge(&mut self, other: &Self) {
-		if other.ts > self.ts {
-			self.ts = other.ts;
-			self.v = other.v.clone();
-		} else if other.ts == self.ts {
-			self.v.merge(&other.v);
-		}
-	}
-}
-
-/// Boolean, where `true` is an absorbing state
-#[derive(Clone, Copy, Debug, Serialize, Deserialize, PartialEq)]
-pub struct Bool(bool);
-
-impl Bool {
-	/// Create a new boolean with the specified value
-	pub fn new(b: bool) -> Self {
-		Self(b)
-	}
-	/// Set the boolean to true
-	pub fn set(&mut self) {
-		self.0 = true;
-	}
-	/// Get the boolean value
-	pub fn get(&self) -> bool {
-		self.0
-	}
-}
-
-impl CRDT for Bool {
-	fn merge(&mut self, other: &Self) {
-		self.0 = self.0 || other.0;
-	}
-}
-
-/// Last Write Win Map
-///
-/// This types defines a CRDT for a map from keys to values.
-/// The values have an associated timestamp, such that the last written value
-/// takes precedence over previous ones. As for the simpler `LWW` type, the value
-/// type `V` is also required to implement the CRDT trait.
-/// We do not encourage mutating the values associated with a given key
-/// without updating the timestamp, in fact at the moment we do not provide a `.get_mut()`
-/// method that would allow that.
-///
-/// Internally, the map is stored as a vector of keys and values, sorted by ascending key order.
-/// This is why the key type `K` must implement `Ord` (and also to ensure a unique serialization,
-/// such that two values can be compared for equality based on their hashes). As a consequence,
-/// insertions take `O(n)` time. This means that LWWMap should be used for reasonably small maps.
-/// However, note that even if we were using a more efficient data structure such as a `BTreeMap`,
-/// the serialization cost `O(n)` would still have to be paid at each modification, so we are
-/// actually not losing anything here.
-#[derive(Clone, Debug, Serialize, Deserialize, PartialEq)]
-pub struct LWWMap<K, V> {
-	vals: Vec<(K, u64, V)>,
-}
-
-impl<K, V> LWWMap<K, V>
-where
-	K: Ord,
-	V: CRDT,
-{
-	/// Create a new empty map CRDT
-	pub fn new() -> Self {
-		Self { vals: vec![] }
-	}
-	/// Used to migrate from a map defined in an incompatible format. This produces
-	/// a map that contains a single item with the specified timestamp (copied from
-	/// the incompatible format). Do this as many times as you have items to migrate,
-	/// and put them all together using the CRDT merge operator.
-	pub fn migrate_from_raw_item(k: K, ts: u64, v: V) -> Self {
-		Self {
-			vals: vec![(k, ts, v)],
-		}
-	}
-	/// Returns a map that contains a single mapping from the specified key to the specified value.
-	/// This map is a mutator, or a delta-CRDT, such that when it is merged with the original map,
-	/// the previous value will be replaced with the one specified here.
-	/// The timestamp in the provided mutator is set to the maximum of the current system's clock
-	/// and 1 + the previous value's timestamp (if there is one), so that the new value will always
-	/// take precedence (LWW rule).
-	///
-	/// Typically, to update the value associated to a key in the map, you would do the following:
-	///
-	/// ```ignore
-	/// let my_update = my_crdt.update_mutator(key_to_modify, new_value);
-	/// my_crdt.merge(&my_update);
-	/// ```
-	///
-	/// However extracting the mutator on its own and only sending that on the network is very
-	/// interesting as it is much smaller than the whole map.
-	pub fn update_mutator(&self, k: K, new_v: V) -> Self {
-		let new_vals = match self.vals.binary_search_by(|(k2, _, _)| k2.cmp(&k)) {
-			Ok(i) => {
-				let (_, old_ts, _) = self.vals[i];
-				let new_ts = std::cmp::max(old_ts + 1, now_msec());
-				vec![(k, new_ts, new_v)]
-			}
-			Err(_) => vec![(k, now_msec(), new_v)],
-		};
-		Self { vals: new_vals }
-	}
-	/// Takes all of the values of the map and returns them. The current map is reset to the
-	/// empty map. This is very usefull to produce in-place a new map that contains only a delta
-	/// that modifies a certain value:
-	///
-	/// ```ignore
-	/// let mut a = get_my_crdt_value();
-	/// let old_a = a.take_and_clear();
-	/// a.merge(&old_a.update_mutator(key_to_modify, new_value));
-	/// put_my_crdt_value(a);
-	/// ```
-	///
-	/// Of course in this simple example we could have written simply
-	/// `pyt_my_crdt_value(a.update_mutator(key_to_modify, new_value))`,
-	/// but in the case where the map is a field in a struct for instance (as is always the case),
-	/// this becomes very handy:
-	///
-	/// ```ignore
-	/// let mut a = get_my_crdt_value();
-	/// let old_a_map = a.map_field.take_and_clear();
-	/// a.map_field.merge(&old_a_map.update_mutator(key_to_modify, new_value));
-	/// put_my_crdt_value(a);
-	/// ```
-	pub fn take_and_clear(&mut self) -> Self {
-		let vals = std::mem::replace(&mut self.vals, vec![]);
-		Self { vals }
-	}
-	/// Removes all values from the map
-	pub fn clear(&mut self) {
-		self.vals.clear();
-	}
-	/// Get a reference to the value assigned to a key
-	pub fn get(&self, k: &K) -> Option<&V> {
-		match self.vals.binary_search_by(|(k2, _, _)| k2.cmp(&k)) {
-			Ok(i) => Some(&self.vals[i].2),
-			Err(_) => None,
-		}
-	}
-	/// Gets a reference to all of the items, as a slice. Usefull to iterate on all map values.
-	/// In most case you will want to ignore the timestamp (second item of the tuple).
-	pub fn items(&self) -> &[(K, u64, V)] {
-		&self.vals[..]
-	}
-}
-
-impl<K, V> CRDT for LWWMap<K, V>
-where
-	K: Clone + Ord,
-	V: Clone + CRDT,
-{
-	fn merge(&mut self, other: &Self) {
-		for (k, ts2, v2) in other.vals.iter() {
-			match self.vals.binary_search_by(|(k2, _, _)| k2.cmp(&k)) {
-				Ok(i) => {
-					let (_, ts1, _v1) = &self.vals[i];
-					if ts2 > ts1 {
-						self.vals[i].1 = *ts2;
-						self.vals[i].2 = v2.clone();
-					} else if ts1 == ts2 {
-						self.vals[i].2.merge(&v2);
-					}
-				}
-				Err(i) => {
-					self.vals.insert(i, (k.clone(), *ts2, v2.clone()));
-				}
-			}
-		}
-	}
-}
@@ -0,0 +1,34 @@
+use serde::{Deserialize, Serialize};
+
+use crate::crdt::crdt::*;
+
+/// Boolean, where `true` is an absorbing state
+#[derive(Clone, Copy, Debug, Serialize, Deserialize, PartialEq)]
+pub struct Bool(bool);
+
+impl Bool {
+	/// Create a new boolean with the specified value
+	pub fn new(b: bool) -> Self {
+		Self(b)
+	}
+	/// Set the boolean to true
+	pub fn set(&mut self) {
+		self.0 = true;
+	}
+	/// Get the boolean value
+	pub fn get(&self) -> bool {
+		self.0
+	}
+}
+
+impl From<bool> for Bool {
+	fn from(b: bool) -> Bool {
+		Bool::new(b)
+	}
+}
+
+impl CRDT for Bool {
+	fn merge(&mut self, other: &Self) {
+		self.0 = self.0 || other.0;
+	}
+}
@@ -0,0 +1,73 @@
+use garage_util::data::*;
+
+/// Definition of a CRDT - all CRDT Rust types implement this.
+///
+/// A CRDT is defined as a merge operator that respects a certain set of axioms.
+///
+/// In particular, the merge operator must be commutative, associative,
+/// idempotent, and monotonic.
+/// In other words, if `a`, `b` and `c` are CRDTs, and `⊔` denotes the merge operator,
+/// the following axioms must apply:
+///
+/// ```text
+/// a ⊔ b = b ⊔ a                   (commutativity)
+/// (a ⊔ b) ⊔ c = a ⊔ (b ⊔ c)       (associativity)
+/// (a ⊔ b) ⊔ b = a ⊔ b             (idempotence)
+/// ```
+///
+/// Moreover, the relationship `≥` defined by `a ≥ b ⇔ ∃c. a = b ⊔ c` must be a partial order.
+/// This implies a few properties such as: if `a ⊔ b ≠ a`, then there is no `c` such that `(a ⊔ b) ⊔ c = a`,
+/// as this would imply a cycle in the partial order.
+pub trait CRDT {
+	/// Merge the two datastructures according to the CRDT rules.
+	/// `self` is modified to contain the merged CRDT value. `other` is not modified.
+	///
+	/// # Arguments
+	///
+	/// * `other` - the other CRDT we wish to merge with
+	fn merge(&mut self, other: &Self);
+}
+
+/// All types that implement `Ord` (a total order) can also implement a trivial CRDT
+/// defined by the merge rule: `a ⊔ b = max(a, b)`. Implement this trait for your type
+/// to enable this behavior.
+pub trait AutoCRDT: Ord + Clone + std::fmt::Debug {
+	/// WARN_IF_DIFFERENT: emit a warning when values differ. Set this to true if
+	/// different values in your application should never happen. Set this to false
+	/// if you are actually relying on the semantics of `a ⊔ b = max(a, b)`.
+	const WARN_IF_DIFFERENT: bool;
+}
+
+impl<T> CRDT for T
+where
+	T: AutoCRDT,
+{
+	fn merge(&mut self, other: &Self) {
+		if Self::WARN_IF_DIFFERENT && self != other {
+			warn!(
+				"Different CRDT values should be the same (logic error!): {:?} vs {:?}",
+				self, other
+			);
+			if other > self {
+				*self = other.clone();
+			}
+			warn!("Making an arbitrary choice: {:?}", self);
+		} else {
+			if other > self {
+				*self = other.clone();
+			}
+		}
+	}
+}
+
+impl AutoCRDT for String {
+	const WARN_IF_DIFFERENT: bool = true;
+}
+
+impl AutoCRDT for bool {
+	const WARN_IF_DIFFERENT: bool = true;
+}
+
+impl AutoCRDT for FixedBytes32 {
+	const WARN_IF_DIFFERENT: bool = true;
+}
@@ -0,0 +1,114 @@
+use serde::{Deserialize, Serialize};
+
+use garage_util::time::now_msec;
+
+use crate::crdt::crdt::*;
+
+/// Last Write Win (LWW)
+///
+/// An LWW CRDT associates a timestamp with a value, in order to implement a
+/// time-based reconciliation rule: the most recent write wins.
+/// For completeness, the LWW reconciliation rule must also be defined for two LWW CRDTs
+/// with the same timestamp but different values.
+///
+/// In our case, we add the constraint that the value that is wrapped inside the LWW CRDT must
+/// itself be a CRDT: in the case when the timestamp does not allow us to decide on which value to
+/// keep, the merge rule of the inner CRDT is applied on the wrapped values.  (Note that all types
+/// that implement the `Ord` trait get a default CRDT implemetnation that keeps the maximum value.
+/// This enables us to use LWW directly with primitive data types such as numbers or strings. It is
+/// generally desirable in this case to never explicitly produce LWW values with the same timestamp
+/// but different inner values, as the rule to keep the maximum value isn't generally the desired
+/// semantics.)
+///
+/// As multiple computers clocks are always desynchronized,
+/// when operations are close enough, it is equivalent to
+/// take one copy and drop the other one.
+///
+/// Given that clocks are not too desynchronized, this assumption
+/// is enough for most cases, as there is few chance that two humans
+/// coordonate themself faster than the time difference between two NTP servers.
+///
+/// As a more concret example, let's suppose you want to upload a file
+/// with the same key (path) in the same bucket at the very same time.
+/// For each request, the file will be timestamped by the receiving server
+/// and may differ from what you observed with your atomic clock!
+///
+/// This scheme is used by AWS S3 or Soundcloud and often without knowing
+/// in entreprise when reconciliating databases with ad-hoc scripts.
+#[derive(Clone, Debug, Serialize, Deserialize, PartialEq)]
+pub struct LWW<T> {
+	ts: u64,
+	v: T,
+}
+
+impl<T> LWW<T>
+where
+	T: CRDT,
+{
+	/// Creates a new CRDT
+	///
+	/// CRDT's internal timestamp is set with current node's clock.
+	pub fn new(value: T) -> Self {
+		Self {
+			ts: now_msec(),
+			v: value,
+		}
+	}
+
+	/// Build a new CRDT from a previous non-compatible one
+	///
+	/// Compared to new, the CRDT's timestamp is not set to now
+	/// but must be set to the previous, non-compatible, CRDT's timestamp.
+	pub fn migrate_from_raw(ts: u64, value: T) -> Self {
+		Self { ts, v: value }
+	}
+
+	/// Update the LWW CRDT while keeping some causal ordering.
+	///
+	/// The timestamp of the LWW CRDT is updated to be the current node's clock
+	/// at time of update, or the previous timestamp + 1 if that's bigger,
+	/// so that the new timestamp is always strictly larger than the previous one.
+	/// This ensures that merging the update with the old value will result in keeping
+	/// the updated value.
+	pub fn update(&mut self, new_value: T) {
+		self.ts = std::cmp::max(self.ts + 1, now_msec());
+		self.v = new_value;
+	}
+
+	/// Get the CRDT value
+	pub fn get(&self) -> &T {
+		&self.v
+	}
+
+	/// Get a mutable reference to the CRDT's value
+	///
+	/// This is usefull to mutate the inside value without changing the LWW timestamp.
+	/// When such mutation is done, the merge between two LWW values is done using the inner
+	/// CRDT's merge operation. This is usefull in the case where the inner CRDT is a large
+	/// data type, such as a map, and we only want to change a single item in the map.
+	/// To do this, we can produce a "CRDT delta", i.e. a LWW that contains only the modification.
+	/// This delta consists in a LWW with the same timestamp, and the map
+	/// inside only contains the updated value.
+	/// The advantage of such a delta is that it is much smaller than the whole map.
+	///
+	/// Avoid using this if the inner data type is a primitive type such as a number or a string,
+	/// as you will then rely on the merge function defined on `Ord` types by keeping the maximum
+	/// of both values.
+	pub fn get_mut(&mut self) -> &mut T {
+		&mut self.v
+	}
+}
+
+impl<T> CRDT for LWW<T>
+where
+	T: Clone + CRDT,
+{
+	fn merge(&mut self, other: &Self) {
+		if other.ts > self.ts {
+			self.ts = other.ts;
+			self.v = other.v.clone();
+		} else if other.ts == self.ts {
+			self.v.merge(&other.v);
+		}
+	}
+}
@@ -0,0 +1,145 @@
+use serde::{Deserialize, Serialize};
+
+use garage_util::time::now_msec;
+
+use crate::crdt::crdt::*;
+
+/// Last Write Win Map
+///
+/// This types defines a CRDT for a map from keys to values.
+/// The values have an associated timestamp, such that the last written value
+/// takes precedence over previous ones. As for the simpler `LWW` type, the value
+/// type `V` is also required to implement the CRDT trait.
+/// We do not encourage mutating the values associated with a given key
+/// without updating the timestamp, in fact at the moment we do not provide a `.get_mut()`
+/// method that would allow that.
+///
+/// Internally, the map is stored as a vector of keys and values, sorted by ascending key order.
+/// This is why the key type `K` must implement `Ord` (and also to ensure a unique serialization,
+/// such that two values can be compared for equality based on their hashes). As a consequence,
+/// insertions take `O(n)` time. This means that LWWMap should be used for reasonably small maps.
+/// However, note that even if we were using a more efficient data structure such as a `BTreeMap`,
+/// the serialization cost `O(n)` would still have to be paid at each modification, so we are
+/// actually not losing anything here.
+#[derive(Clone, Debug, Serialize, Deserialize, PartialEq)]
+pub struct LWWMap<K, V> {
+	vals: Vec<(K, u64, V)>,
+}
+
+impl<K, V> LWWMap<K, V>
+where
+	K: Ord,
+	V: CRDT,
+{
+	/// Create a new empty map CRDT
+	pub fn new() -> Self {
+		Self { vals: vec![] }
+	}
+	/// Used to migrate from a map defined in an incompatible format. This produces
+	/// a map that contains a single item with the specified timestamp (copied from
+	/// the incompatible format). Do this as many times as you have items to migrate,
+	/// and put them all together using the CRDT merge operator.
+	pub fn migrate_from_raw_item(k: K, ts: u64, v: V) -> Self {
+		Self {
+			vals: vec![(k, ts, v)],
+		}
+	}
+	/// Returns a map that contains a single mapping from the specified key to the specified value.
+	/// This map is a mutator, or a delta-CRDT, such that when it is merged with the original map,
+	/// the previous value will be replaced with the one specified here.
+	/// The timestamp in the provided mutator is set to the maximum of the current system's clock
+	/// and 1 + the previous value's timestamp (if there is one), so that the new value will always
+	/// take precedence (LWW rule).
+	///
+	/// Typically, to update the value associated to a key in the map, you would do the following:
+	///
+	/// ```ignore
+	/// let my_update = my_crdt.update_mutator(key_to_modify, new_value);
+	/// my_crdt.merge(&my_update);
+	/// ```
+	///
+	/// However extracting the mutator on its own and only sending that on the network is very
+	/// interesting as it is much smaller than the whole map.
+	pub fn update_mutator(&self, k: K, new_v: V) -> Self {
+		let new_vals = match self.vals.binary_search_by(|(k2, _, _)| k2.cmp(&k)) {
+			Ok(i) => {
+				let (_, old_ts, _) = self.vals[i];
+				let new_ts = std::cmp::max(old_ts + 1, now_msec());
+				vec![(k, new_ts, new_v)]
+			}
+			Err(_) => vec![(k, now_msec(), new_v)],
+		};
+		Self { vals: new_vals }
+	}
+	/// Takes all of the values of the map and returns them. The current map is reset to the
+	/// empty map. This is very usefull to produce in-place a new map that contains only a delta
+	/// that modifies a certain value:
+	///
+	/// ```ignore
+	/// let mut a = get_my_crdt_value();
+	/// let old_a = a.take_and_clear();
+	/// a.merge(&old_a.update_mutator(key_to_modify, new_value));
+	/// put_my_crdt_value(a);
+	/// ```
+	///
+	/// Of course in this simple example we could have written simply
+	/// `pyt_my_crdt_value(a.update_mutator(key_to_modify, new_value))`,
+	/// but in the case where the map is a field in a struct for instance (as is always the case),
+	/// this becomes very handy:
+	///
+	/// ```ignore
+	/// let mut a = get_my_crdt_value();
+	/// let old_a_map = a.map_field.take_and_clear();
+	/// a.map_field.merge(&old_a_map.update_mutator(key_to_modify, new_value));
+	/// put_my_crdt_value(a);
+	/// ```
+	pub fn take_and_clear(&mut self) -> Self {
+		let vals = std::mem::replace(&mut self.vals, vec![]);
+		Self { vals }
+	}
+	/// Removes all values from the map
+	pub fn clear(&mut self) {
+		self.vals.clear();
+	}
+	/// Get a reference to the value assigned to a key
+	pub fn get(&self, k: &K) -> Option<&V> {
+		match self.vals.binary_search_by(|(k2, _, _)| k2.cmp(&k)) {
+			Ok(i) => Some(&self.vals[i].2),
+			Err(_) => None,
+		}
+	}
+	/// Gets a reference to all of the items, as a slice. Usefull to iterate on all map values.
+	/// In most case you will want to ignore the timestamp (second item of the tuple).
+	pub fn items(&self) -> &[(K, u64, V)] {
+		&self.vals[..]
+	}
+	/// Returns the number of items in the map
+	pub fn len(&self) -> usize {
+		self.vals.len()
+	}
+}
+
+impl<K, V> CRDT for LWWMap<K, V>
+where
+	K: Clone + Ord,
+	V: Clone + CRDT,
+{
+	fn merge(&mut self, other: &Self) {
+		for (k, ts2, v2) in other.vals.iter() {
+			match self.vals.binary_search_by(|(k2, _, _)| k2.cmp(&k)) {
+				Ok(i) => {
+					let (_, ts1, _v1) = &self.vals[i];
+					if ts2 > ts1 {
+						self.vals[i].1 = *ts2;
+						self.vals[i].2 = v2.clone();
+					} else if ts1 == ts2 {
+						self.vals[i].2.merge(&v2);
+					}
+				}
+				Err(i) => {
+					self.vals.insert(i, (k.clone(), *ts2, v2.clone()));
+				}
+			}
+		}
+	}
+}
@@ -0,0 +1,83 @@
+use serde::{Deserialize, Serialize};
+
+use crate::crdt::crdt::*;
+
+/// Simple CRDT Map
+///
+/// This types defines a CRDT for a map from keys to values. Values are CRDT types which
+/// can have their own updating logic.
+///
+/// Internally, the map is stored as a vector of keys and values, sorted by ascending key order.
+/// This is why the key type `K` must implement `Ord` (and also to ensure a unique serialization,
+/// such that two values can be compared for equality based on their hashes). As a consequence,
+/// insertions take `O(n)` time. This means that Map should be used for reasonably small maps.
+/// However, note that even if we were using a more efficient data structure such as a `BTreeMap`,
+/// the serialization cost `O(n)` would still have to be paid at each modification, so we are
+/// actually not losing anything here.
+#[derive(Clone, Debug, Serialize, Deserialize, PartialEq)]
+pub struct Map<K, V> {
+	vals: Vec<(K, V)>,
+}
+
+impl<K, V> Map<K, V>
+where
+	K: Clone + Ord,
+	V: Clone + CRDT,
+{
+	/// Create a new empty map CRDT
+	pub fn new() -> Self {
+		Self { vals: vec![] }
+	}
+
+	/// Returns a map that contains a single mapping from the specified key to the specified value.
+	/// This can be used to build a delta-mutator:
+	/// when merged with another map, the value will be added or CRDT-merged if a previous
+	/// value already exists.
+	pub fn put_mutator(k: K, v: V) -> Self {
+		Self { vals: vec![(k, v)] }
+	}
+
+	pub fn put(&mut self, k: K, v: V) {
+		self.merge(&Self::put_mutator(k, v));
+	}
+
+	/// Removes all values from the map
+	pub fn clear(&mut self) {
+		self.vals.clear();
+	}
+
+	/// Get a reference to the value assigned to a key
+	pub fn get(&self, k: &K) -> Option<&V> {
+		match self.vals.binary_search_by(|(k2, _)| k2.cmp(&k)) {
+			Ok(i) => Some(&self.vals[i].1),
+			Err(_) => None,
+		}
+	}
+	/// Gets a reference to all of the items, as a slice. Usefull to iterate on all map values.
+	pub fn items(&self) -> &[(K, V)] {
+		&self.vals[..]
+	}
+	/// Returns the number of items in the map
+	pub fn len(&self) -> usize {
+		self.vals.len()
+	}
+}
+
+impl<K, V> CRDT for Map<K, V>
+where
+	K: Clone + Ord,
+	V: Clone + CRDT,
+{
+	fn merge(&mut self, other: &Self) {
+		for (k, v2) in other.vals.iter() {
+			match self.vals.binary_search_by(|(k2, _)| k2.cmp(&k)) {
+				Ok(i) => {
+					self.vals[i].1.merge(&v2);
+				}
+				Err(i) => {
+					self.vals.insert(i, (k.clone(), v2.clone()));
+				}
+			}
+		}
+	}
+}
@@ -0,0 +1,22 @@
+//! This package provides a simple implementation of conflict-free replicated data types (CRDTs)
+//!
+//! CRDTs are a type of data structures that do not require coordination.  In other words, we can
+//! edit them in parallel, we will always find a way to merge it.
+//!
+//! A general example is a counter. Its initial value is 0.  Alice and Bob get a copy of the
+//! counter.  Alice does +1 on her copy, she reads 1.  Bob does +3 on his copy, he reads 3.  Now,
+//! it is easy to merge their counters, order does not count: we always get 4.
+//!
+//! Learn more about CRDT [on Wikipedia](https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type)
+
+mod bool;
+mod crdt;
+mod lww;
+mod lww_map;
+mod map;
+
+pub use self::bool::*;
+pub use crdt::*;
+pub use lww::*;
+pub use lww_map::*;
+pub use map::*;
@@ -0,0 +1,254 @@
+use core::borrow::Borrow;
+use std::sync::Arc;
+
+use log::warn;
+use serde_bytes::ByteBuf;
+use sled::Transactional;
+use tokio::sync::Notify;
+
+use garage_util::data::*;
+use garage_util::error::*;
+
+use garage_rpc::membership::System;
+
+use crate::crdt::CRDT;
+use crate::replication::*;
+use crate::schema::*;
+
+pub struct TableData<F: TableSchema, R: TableReplication> {
+	system: Arc<System>,
+
+	pub name: String,
+	pub(crate) instance: F,
+	pub(crate) replication: R,
+
+	pub store: sled::Tree,
+
+	pub(crate) merkle_tree: sled::Tree,
+	pub(crate) merkle_todo: sled::Tree,
+	pub(crate) merkle_todo_notify: Notify,
+	pub(crate) gc_todo: sled::Tree,
+}
+
+impl<F, R> TableData<F, R>
+where
+	F: TableSchema,
+	R: TableReplication,
+{
+	pub fn new(system: Arc<System>, name: String, instance: F, replication: R, db: &sled::Db) -> Arc<Self> {
+		let store = db
+			.open_tree(&format!("{}:table", name))
+			.expect("Unable to open DB tree");
+
+		let merkle_tree = db
+			.open_tree(&format!("{}:merkle_tree", name))
+			.expect("Unable to open DB Merkle tree tree");
+		let merkle_todo = db
+			.open_tree(&format!("{}:merkle_todo", name))
+			.expect("Unable to open DB Merkle TODO tree");
+
+		let gc_todo = db
+			.open_tree(&format!("{}:gc_todo", name))
+			.expect("Unable to open DB tree");
+
+		Arc::new(Self {
+			system,
+			name,
+			instance,
+			replication,
+			store,
+			merkle_tree,
+			merkle_todo,
+			merkle_todo_notify: Notify::new(),
+			gc_todo,
+		})
+	}
+
+	// Read functions
+
+	pub fn read_entry(&self, p: &F::P, s: &F::S) -> Result<Option<ByteBuf>, Error> {
+		let tree_key = self.tree_key(p, s);
+		if let Some(bytes) = self.store.get(&tree_key)? {
+			Ok(Some(ByteBuf::from(bytes.to_vec())))
+		} else {
+			Ok(None)
+		}
+	}
+
+	pub fn read_range(
+		&self,
+		p: &F::P,
+		s: &Option<F::S>,
+		filter: &Option<F::Filter>,
+		limit: usize,
+	) -> Result<Vec<Arc<ByteBuf>>, Error> {
+		let partition_hash = p.hash();
+		let first_key = match s {
+			None => partition_hash.to_vec(),
+			Some(sk) => self.tree_key(p, sk),
+		};
+		let mut ret = vec![];
+		for item in self.store.range(first_key..) {
+			let (key, value) = item?;
+			if &key[..32] != partition_hash.as_slice() {
+				break;
+			}
+			let keep = match filter {
+				None => true,
+				Some(f) => {
+					let entry = self.decode_entry(value.as_ref())?;
+					F::matches_filter(&entry, f)
+				}
+			};
+			if keep {
+				ret.push(Arc::new(ByteBuf::from(value.as_ref())));
+			}
+			if ret.len() >= limit {
+				break;
+			}
+		}
+		Ok(ret)
+	}
+
+	// Mutation functions
+	// When changing this code, take care of propagating modifications correctly:
+	// - When an entry is modified or deleted, call the updated() function
+	//   on the table instance
+	// - When an entry is modified or deleted, add it to the merkle updater's todo list.
+	//   This has to be done atomically with the modification for the merkle updater
+	//   to maintain consistency. The merkle updater must then be notified with todo_notify.
+	// - When an entry is updated to be a tombstone, add it to the gc_todo tree
+
+	pub(crate) fn update_many<T: Borrow<ByteBuf>>(&self, entries: &[T]) -> Result<(), Error> {
+		for update_bytes in entries.iter() {
+			self.update_entry(update_bytes.borrow().as_slice())?;
+		}
+		Ok(())
+	}
+
+	pub(crate) fn update_entry(&self, update_bytes: &[u8]) -> Result<(), Error> {
+		let update = self.decode_entry(update_bytes)?;
+		let tree_key = self.tree_key(update.partition_key(), update.sort_key());
+
+		let changed = (&self.store, &self.merkle_todo).transaction(|(store, mkl_todo)| {
+			let (old_entry, new_entry) = match store.get(&tree_key)? {
+				Some(prev_bytes) => {
+					let old_entry = self
+						.decode_entry(&prev_bytes)
+						.map_err(sled::transaction::ConflictableTransactionError::Abort)?;
+					let mut new_entry = old_entry.clone();
+					new_entry.merge(&update);
+					(Some(old_entry), new_entry)
+				}
+				None => (None, update.clone()),
+			};
+
+			if Some(&new_entry) != old_entry.as_ref() {
+				let new_bytes = rmp_to_vec_all_named(&new_entry)
+					.map_err(Error::RMPEncode)
+					.map_err(sled::transaction::ConflictableTransactionError::Abort)?;
+				let new_bytes_hash = blake2sum(&new_bytes[..]);
+				mkl_todo.insert(tree_key.clone(), new_bytes_hash.as_slice())?;
+				store.insert(tree_key.clone(), new_bytes)?;
+				Ok(Some((old_entry, new_entry, new_bytes_hash)))
+			} else {
+				Ok(None)
+			}
+		})?;
+
+		if let Some((old_entry, new_entry, new_bytes_hash)) = changed {
+			let is_tombstone = new_entry.is_tombstone();
+			self.instance.updated(old_entry, Some(new_entry));
+			self.merkle_todo_notify.notify_one();
+			if is_tombstone {
+				// We are only responsible for GC'ing this item if we are the
+				// "leader" of the partition, i.e. the first node in the
+				// set of nodes that replicates this partition.
+				// This avoids GC loops and does not change the termination properties
+				// of the GC algorithm, as in all cases GC is suspended if
+				// any node of the partition is unavailable.
+				let pk_hash = Hash::try_from(&tree_key[..32]).unwrap();
+				let nodes = self.replication.write_nodes(&pk_hash);
+				if nodes.first() == Some(&self.system.id) {
+					self.gc_todo.insert(&tree_key, new_bytes_hash.as_slice())?;
+				}
+			}
+		}
+
+		Ok(())
+	}
+
+	pub(crate) fn delete_if_equal(self: &Arc<Self>, k: &[u8], v: &[u8]) -> Result<bool, Error> {
+		let removed = (&self.store, &self.merkle_todo).transaction(|(store, mkl_todo)| {
+			if let Some(cur_v) = store.get(k)? {
+				if cur_v == v {
+					store.remove(k)?;
+					mkl_todo.insert(k, vec![])?;
+					return Ok(true);
+				}
+			}
+			Ok(false)
+		})?;
+
+		if removed {
+			let old_entry = self.decode_entry(v)?;
+			self.instance.updated(Some(old_entry), None);
+			self.merkle_todo_notify.notify_one();
+		}
+		Ok(removed)
+	}
+
+	pub(crate) fn delete_if_equal_hash(
+		self: &Arc<Self>,
+		k: &[u8],
+		vhash: Hash,
+	) -> Result<bool, Error> {
+		let removed = (&self.store, &self.merkle_todo).transaction(|(store, mkl_todo)| {
+			if let Some(cur_v) = store.get(k)? {
+				if blake2sum(&cur_v[..]) == vhash {
+					store.remove(k)?;
+					mkl_todo.insert(k, vec![])?;
+					return Ok(Some(cur_v));
+				}
+			}
+			Ok(None)
+		})?;
+
+		if let Some(old_v) = removed {
+			let old_entry = self.decode_entry(&old_v[..])?;
+			self.instance.updated(Some(old_entry), None);
+			self.merkle_todo_notify.notify_one();
+			Ok(true)
+		} else {
+			Ok(false)
+		}
+	}
+
+	// ---- Utility functions ----
+
+	pub(crate) fn tree_key(&self, p: &F::P, s: &F::S) -> Vec<u8> {
+		let mut ret = p.hash().to_vec();
+		ret.extend(s.sort_key());
+		ret
+	}
+
+	pub(crate) fn decode_entry(&self, bytes: &[u8]) -> Result<F::E, Error> {
+		match rmp_serde::decode::from_read_ref::<_, F::E>(bytes) {
+			Ok(x) => Ok(x),
+			Err(e) => match F::try_migrate(bytes) {
+				Some(x) => Ok(x),
+				None => {
+					warn!("Unable to decode entry of {}: {}", self.name, e);
+					for line in hexdump::hexdump_iter(bytes) {
+						debug!("{}", line);
+					}
+					Err(e.into())
+				}
+			},
+		}
+	}
+
+	pub fn gc_todo_len(&self) -> usize {
+		self.gc_todo.len()
+	}
+}
@@ -0,0 +1,248 @@
+use std::collections::HashMap;
+use std::sync::Arc;
+use std::time::Duration;
+
+use serde::{Deserialize, Serialize};
+use serde_bytes::ByteBuf;
+
+use futures::future::join_all;
+use futures::select;
+use futures_util::future::*;
+use tokio::sync::watch;
+
+use garage_util::data::*;
+use garage_util::error::Error;
+
+use garage_rpc::membership::System;
+use garage_rpc::rpc_client::*;
+use garage_rpc::rpc_server::*;
+
+use crate::data::*;
+use crate::replication::*;
+use crate::schema::*;
+
+const TABLE_GC_BATCH_SIZE: usize = 1024;
+const TABLE_GC_RPC_TIMEOUT: Duration = Duration::from_secs(30);
+
+pub struct TableGC<F: TableSchema, R: TableReplication> {
+	system: Arc<System>,
+	data: Arc<TableData<F, R>>,
+
+	rpc_client: Arc<RpcClient<GcRPC>>,
+}
+
+#[derive(Serialize, Deserialize)]
+enum GcRPC {
+	Update(Vec<ByteBuf>),
+	DeleteIfEqualHash(Vec<(ByteBuf, Hash)>),
+	Ok,
+}
+
+impl RpcMessage for GcRPC {}
+
+impl<F, R> TableGC<F, R>
+where
+	F: TableSchema + 'static,
+	R: TableReplication + 'static,
+{
+	pub(crate) fn launch(
+		system: Arc<System>,
+		data: Arc<TableData<F, R>>,
+		rpc_server: &mut RpcServer,
+	) -> Arc<Self> {
+		let rpc_path = format!("table_{}/gc", data.name);
+		let rpc_client = system.rpc_client::<GcRPC>(&rpc_path);
+
+		let gc = Arc::new(Self {
+			system: system.clone(),
+			data: data.clone(),
+			rpc_client,
+		});
+
+		gc.register_handler(rpc_server, rpc_path);
+
+		let gc1 = gc.clone();
+		system.background.spawn_worker(
+			format!("GC loop for {}", data.name),
+			move |must_exit: watch::Receiver<bool>| gc1.gc_loop(must_exit),
+		);
+
+		gc
+	}
+
+	async fn gc_loop(self: Arc<Self>, mut must_exit: watch::Receiver<bool>) {
+		while !*must_exit.borrow() {
+			match self.gc_loop_iter().await {
+				Ok(true) => {
+					// Stuff was done, loop imediately
+					continue;
+				}
+				Ok(false) => {
+					// Nothing was done, sleep for some time (below)
+				}
+				Err(e) => {
+					warn!("({}) Error doing GC: {}", self.data.name, e);
+				}
+			}
+			select! {
+				_ = tokio::time::sleep(Duration::from_secs(10)).fuse() => (),
+				_ = must_exit.changed().fuse() => (),
+			}
+		}
+	}
+
+	async fn gc_loop_iter(&self) -> Result<bool, Error> {
+		let mut entries = vec![];
+		let mut excluded = vec![];
+
+		for item in self.data.gc_todo.iter() {
+			let (k, vhash) = item?;
+
+			let vhash = Hash::try_from(&vhash[..]).unwrap();
+
+			let v_opt = self
+				.data
+				.store
+				.get(&k[..])?
+				.filter(|v| blake2sum(&v[..]) == vhash);
+
+			if let Some(v) = v_opt {
+				entries.push((ByteBuf::from(k.to_vec()), vhash, ByteBuf::from(v.to_vec())));
+				if entries.len() >= TABLE_GC_BATCH_SIZE {
+					break;
+				}
+			} else {
+				excluded.push((k, vhash));
+			}
+		}
+
+		for (k, vhash) in excluded {
+			self.todo_remove_if_equal(&k[..], vhash)?;
+		}
+
+		if entries.len() == 0 {
+			// Nothing to do in this iteration
+			return Ok(false);
+		}
+
+		debug!("({}) GC: doing {} items", self.data.name, entries.len());
+
+		let mut partitions = HashMap::new();
+		for (k, vhash, v) in entries {
+			let pkh = Hash::try_from(&k[..32]).unwrap();
+			let mut nodes = self.data.replication.write_nodes(&pkh);
+			nodes.retain(|x| *x != self.system.id);
+			nodes.sort();
+
+			if !partitions.contains_key(&nodes) {
+				partitions.insert(nodes.clone(), vec![]);
+			}
+			partitions.get_mut(&nodes).unwrap().push((k, vhash, v));
+		}
+
+		let resps = join_all(
+			partitions
+				.into_iter()
+				.map(|(nodes, items)| self.try_send_and_delete(nodes, items)),
+		)
+		.await;
+
+		let mut errs = vec![];
+		for resp in resps {
+			if let Err(e) = resp {
+				errs.push(e);
+			}
+		}
+
+		if errs.is_empty() {
+			Ok(true)
+		} else {
+			Err(Error::Message(errs.into_iter().map(|x| format!("{}", x)).collect::<Vec<_>>().join(", ")))
+		}
+	}
+
+	async fn try_send_and_delete(
+		&self,
+		nodes: Vec<UUID>,
+		items: Vec<(ByteBuf, Hash, ByteBuf)>,
+	) -> Result<(), Error> {
+		let n_items = items.len();
+
+		let mut updates = vec![];
+		let mut deletes = vec![];
+		for (k, vhash, v) in items {
+			updates.push(v);
+			deletes.push((k, vhash));
+		}
+
+		self.rpc_client
+			.try_call_many(
+				&nodes[..],
+				GcRPC::Update(updates),
+				RequestStrategy::with_quorum(nodes.len()).with_timeout(TABLE_GC_RPC_TIMEOUT),
+			)
+			.await?;
+
+		info!(
+			"({}) GC: {} items successfully pushed, will try to delete.",
+			self.data.name, n_items
+		);
+
+		self.rpc_client
+			.try_call_many(
+				&nodes[..],
+				GcRPC::DeleteIfEqualHash(deletes.clone()),
+				RequestStrategy::with_quorum(nodes.len()).with_timeout(TABLE_GC_RPC_TIMEOUT),
+			)
+			.await?;
+
+		for (k, vhash) in deletes {
+			self.data.delete_if_equal_hash(&k[..], vhash)?;
+			self.todo_remove_if_equal(&k[..], vhash)?;
+		}
+
+		Ok(())
+	}
+
+	fn todo_remove_if_equal(&self, key: &[u8], vhash: Hash) -> Result<(), Error> {
+		let _ = self
+			.data
+			.gc_todo
+			.compare_and_swap::<_, _, Vec<u8>>(key, Some(vhash), None)?;
+		Ok(())
+	}
+
+	// ---- RPC HANDLER ----
+
+	fn register_handler(self: &Arc<Self>, rpc_server: &mut RpcServer, path: String) {
+		let self2 = self.clone();
+		rpc_server.add_handler::<GcRPC, _, _>(path, move |msg, _addr| {
+			let self2 = self2.clone();
+			async move { self2.handle_rpc(&msg).await }
+		});
+
+		let self2 = self.clone();
+		self.rpc_client
+			.set_local_handler(self.system.id, move |msg| {
+				let self2 = self2.clone();
+				async move { self2.handle_rpc(&msg).await }
+			});
+	}
+
+	async fn handle_rpc(self: &Arc<Self>, message: &GcRPC) -> Result<GcRPC, Error> {
+		match message {
+			GcRPC::Update(items) => {
+				self.data.update_many(items)?;
+				Ok(GcRPC::Ok)
+			}
+			GcRPC::DeleteIfEqualHash(items) => {
+				for (key, vhash) in items.iter() {
+					self.data.delete_if_equal_hash(&key[..], *vhash)?;
+					self.todo_remove_if_equal(&key[..], *vhash)?;
+				}
+				Ok(GcRPC::Ok)
+			}
+			_ => Err(Error::Message(format!("Unexpected GC RPC"))),
+		}
+	}
+}
@@ -7,10 +7,12 @@ pub mod crdt;
 pub mod schema;
 pub mod util;

+pub mod data;
+pub mod gc;
+pub mod merkle;
+pub mod replication;
+pub mod sync;
 pub mod table;
-pub mod table_fullcopy;
-pub mod table_sharded;
-pub mod table_sync;

 pub use schema::*;
 pub use table::*;
@@ -0,0 +1,454 @@
+use std::sync::Arc;
+use std::time::Duration;
+
+use futures::select;
+use futures_util::future::*;
+use log::{debug, warn};
+use serde::{Deserialize, Serialize};
+use sled::transaction::{
+	ConflictableTransactionError, ConflictableTransactionResult, TransactionalTree,
+};
+use tokio::sync::watch;
+
+use garage_util::background::BackgroundRunner;
+use garage_util::data::*;
+use garage_util::error::Error;
+
+use garage_rpc::ring::*;
+
+use crate::data::*;
+use crate::replication::*;
+use crate::schema::*;
+
+// This modules partitions the data in 2**16 partitions, based on the top
+// 16 bits (two bytes) of item's partition keys' hashes.
+// It builds one Merkle tree for each of these 2**16 partitions.
+
+pub struct MerkleUpdater<F: TableSchema, R: TableReplication> {
+	data: Arc<TableData<F, R>>,
+
+	// Content of the todo tree: items where
+	// - key = the key of an item in the main table, ie hash(partition_key)+sort_key
+	// - value = the hash of the full serialized item, if present,
+	//			 or an empty vec if item is absent (deleted)
+	// Fields in data:
+	//		pub(crate) merkle_todo: sled::Tree,
+	//		pub(crate) merkle_todo_notify: Notify,
+
+	// Content of the merkle tree: items where
+	// - key = .bytes() for MerkleNodeKey
+	// - value = serialization of a MerkleNode, assumed to be MerkleNode::empty if not found
+	// Field in data:
+	//		pub(crate) merkle_tree: sled::Tree,
+	empty_node_hash: Hash,
+}
+
+#[derive(Clone, Debug, Serialize, Deserialize)]
+pub struct MerkleNodeKey {
+	// partition number
+	pub partition: Partition,
+
+	// prefix: a prefix for the hash of full keys, i.e. hash(hash(partition_key)+sort_key)
+	#[serde(with = "serde_bytes")]
+	pub prefix: Vec<u8>,
+}
+
+#[derive(PartialEq, Eq, Debug, Serialize, Deserialize)]
+pub enum MerkleNode {
+	// The empty Merkle node
+	Empty,
+
+	// An intermediate Merkle tree node for a prefix
+	// Contains the hashes of the 256 possible next prefixes
+	Intermediate(Vec<(u8, Hash)>),
+
+	// A final node for an item
+	// Contains the full key of the item and the hash of the value
+	Leaf(Vec<u8>, Hash),
+}
+
+impl<F, R> MerkleUpdater<F, R>
+where
+	F: TableSchema + 'static,
+	R: TableReplication + 'static,
+{
+	pub(crate) fn launch(background: &BackgroundRunner, data: Arc<TableData<F, R>>) -> Arc<Self> {
+		let empty_node_hash = blake2sum(&rmp_to_vec_all_named(&MerkleNode::Empty).unwrap()[..]);
+
+		let ret = Arc::new(Self {
+			data,
+			empty_node_hash,
+		});
+
+		let ret2 = ret.clone();
+		background.spawn_worker(
+			format!("Merkle tree updater for {}", ret.data.name),
+			|must_exit: watch::Receiver<bool>| ret2.updater_loop(must_exit),
+		);
+
+		ret
+	}
+
+	async fn updater_loop(self: Arc<Self>, mut must_exit: watch::Receiver<bool>) {
+		while !*must_exit.borrow() {
+			if let Some(x) = self.data.merkle_todo.iter().next() {
+				match x {
+					Ok((key, valhash)) => {
+						if let Err(e) = self.update_item(&key[..], &valhash[..]) {
+							warn!(
+								"({}) Error while updating Merkle tree item: {}",
+								self.data.name, e
+							);
+						}
+					}
+					Err(e) => {
+						warn!(
+							"({}) Error while iterating on Merkle todo tree: {}",
+							self.data.name, e
+						);
+						tokio::time::sleep(Duration::from_secs(10)).await;
+					}
+				}
+			} else {
+				select! {
+					_ = self.data.merkle_todo_notify.notified().fuse() => (),
+					_ = must_exit.changed().fuse() => (),
+				}
+			}
+		}
+	}
+
+	fn update_item(&self, k: &[u8], vhash_by: &[u8]) -> Result<(), Error> {
+		let khash = blake2sum(k);
+
+		let new_vhash = if vhash_by.len() == 0 {
+			None
+		} else {
+			Some(Hash::try_from(&vhash_by[..]).unwrap())
+		};
+
+		let key = MerkleNodeKey {
+			partition: self
+				.data
+				.replication
+				.partition_of(&Hash::try_from(&k[0..32]).unwrap()),
+			prefix: vec![],
+		};
+		self.data
+			.merkle_tree
+			.transaction(|tx| self.update_item_rec(tx, k, &khash, &key, new_vhash))?;
+
+		let deleted = self
+			.data
+			.merkle_todo
+			.compare_and_swap::<_, _, Vec<u8>>(k, Some(vhash_by), None)?
+			.is_ok();
+
+		if !deleted {
+			debug!(
+				"({}) Item not deleted from Merkle todo because it changed: {:?}",
+				self.data.name, k
+			);
+		}
+		Ok(())
+	}
+
+	fn update_item_rec(
+		&self,
+		tx: &TransactionalTree,
+		k: &[u8],
+		khash: &Hash,
+		key: &MerkleNodeKey,
+		new_vhash: Option<Hash>,
+	) -> ConflictableTransactionResult<Option<Hash>, Error> {
+		let i = key.prefix.len();
+
+		// Read node at current position (defined by the prefix stored in key)
+		// Calculate an update to apply to this node
+		// This update is an Option<_>, so that it is None if the update is a no-op
+		// and we can thus skip recalculating and re-storing everything
+		let mutate = match self.read_node_txn(tx, &key)? {
+			MerkleNode::Empty => {
+				if let Some(vhv) = new_vhash {
+					Some(MerkleNode::Leaf(k.to_vec(), vhv))
+				} else {
+					// Nothing to do, keep empty node
+					None
+				}
+			}
+			MerkleNode::Intermediate(mut children) => {
+				let key2 = key.next_key(khash);
+				if let Some(subhash) = self.update_item_rec(tx, k, khash, &key2, new_vhash)? {
+					// Subtree changed, update this node as well
+					if subhash == self.empty_node_hash {
+						intermediate_rm_child(&mut children, key2.prefix[i]);
+					} else {
+						intermediate_set_child(&mut children, key2.prefix[i], subhash);
+					}
+
+					if children.len() == 0 {
+						// should not happen
+						warn!(
+							"({}) Replacing intermediate node with empty node, should not happen.",
+							self.data.name
+						);
+						Some(MerkleNode::Empty)
+					} else if children.len() == 1 {
+						// We now have a single node (case when the update deleted one of only two
+						// children). If that node is a leaf, move it to this level.
+						let key_sub = key.add_byte(children[0].0);
+						let subnode = self.read_node_txn(tx, &key_sub)?;
+						match subnode {
+							MerkleNode::Empty => {
+								warn!("({}) Single subnode in tree is empty Merkle node", self.data.name);
+								Some(MerkleNode::Empty)
+							}
+							MerkleNode::Intermediate(_) => {
+								Some(MerkleNode::Intermediate(children))
+							}
+							x @ MerkleNode::Leaf(_, _) => {
+								tx.remove(key_sub.encode())?;
+								Some(x)
+							}
+						}
+					} else {
+						Some(MerkleNode::Intermediate(children))
+					}
+				} else {
+					// Subtree not changed, nothing to do
+					None
+				}
+			}
+			MerkleNode::Leaf(exlf_k, exlf_vhash) => {
+				if exlf_k == k {
+					// This leaf is for the same key that the one we are updating
+					match new_vhash {
+						Some(vhv) if vhv == exlf_vhash => None,
+						Some(vhv) => Some(MerkleNode::Leaf(k.to_vec(), vhv)),
+						None => Some(MerkleNode::Empty),
+					}
+				} else {
+					// This is an only leaf for another key
+					if new_vhash.is_some() {
+						// Move that other key to a subnode, create another subnode for our
+						// insertion and replace current node by an intermediary node
+						let mut int = vec![];
+
+						let exlf_khash = blake2sum(&exlf_k[..]);
+						assert_eq!(khash.as_slice()[..i], exlf_khash.as_slice()[..i]);
+
+						{
+							let exlf_subkey = key.next_key(&exlf_khash);
+							let exlf_sub_hash = self.update_item_rec(tx, &exlf_k[..], &exlf_khash, &exlf_subkey, Some(exlf_vhash))?.unwrap();
+							intermediate_set_child(&mut int, exlf_subkey.prefix[i], exlf_sub_hash);
+							assert_eq!(int.len(), 1);
+						}
+
+						{
+							let key2 = key.next_key(khash);
+							let subhash = self.update_item_rec(tx, k, khash, &key2, new_vhash)?.unwrap();
+							intermediate_set_child(&mut int, key2.prefix[i], subhash);
+							if exlf_khash.as_slice()[i] == khash.as_slice()[i] {
+								assert_eq!(int.len(), 1);
+							} else {
+								assert_eq!(int.len(), 2);
+							}
+						}
+						Some(MerkleNode::Intermediate(int))
+					} else {
+						// Nothing to do, we don't want to insert this value because it is None,
+						// and we don't want to change the other value because it's for something
+						// else
+						None
+					}
+				}
+			}
+		};
+
+		if let Some(new_node) = mutate {
+			let hash = self.put_node_txn(tx, &key, &new_node)?;
+			Ok(Some(hash))
+		} else {
+			Ok(None)
+		}
+	}
+
+	// Merkle tree node manipulation
+
+	fn read_node_txn(
+		&self,
+		tx: &TransactionalTree,
+		k: &MerkleNodeKey,
+	) -> ConflictableTransactionResult<MerkleNode, Error> {
+		let ent = tx.get(k.encode())?;
+		MerkleNode::decode_opt(ent).map_err(ConflictableTransactionError::Abort)
+	}
+
+	fn put_node_txn(
+		&self,
+		tx: &TransactionalTree,
+		k: &MerkleNodeKey,
+		v: &MerkleNode,
+	) -> ConflictableTransactionResult<Hash, Error> {
+		trace!("Put Merkle node: {:?} => {:?}", k, v);
+		if *v == MerkleNode::Empty {
+			tx.remove(k.encode())?;
+			Ok(self.empty_node_hash)
+		} else {
+			let vby = rmp_to_vec_all_named(v)
+				.map_err(|e| ConflictableTransactionError::Abort(e.into()))?;
+			let rethash = blake2sum(&vby[..]);
+			tx.insert(k.encode(), vby)?;
+			Ok(rethash)
+		}
+	}
+
+	// Access a node in the Merkle tree, used by the sync protocol
+	pub(crate) fn read_node(&self, k: &MerkleNodeKey) -> Result<MerkleNode, Error> {
+		let ent = self.data.merkle_tree.get(k.encode())?;
+		MerkleNode::decode_opt(ent)
+	}
+
+	pub fn merkle_tree_len(&self) -> usize {
+		self.data.merkle_tree.len()
+	}
+
+	pub fn todo_len(&self) -> usize {
+		self.data.merkle_todo.len()
+	}
+}
+
+impl MerkleNodeKey {
+	fn encode(&self) -> Vec<u8> {
+		let mut ret = Vec::with_capacity(2 + self.prefix.len());
+		ret.extend(&u16::to_be_bytes(self.partition)[..]);
+		ret.extend(&self.prefix[..]);
+		ret
+	}
+
+	pub fn next_key(&self, h: &Hash) -> Self {
+		assert_eq!(h.as_slice()[0..self.prefix.len()], self.prefix[..]);
+		let mut s2 = self.clone();
+		s2.prefix.push(h.as_slice()[self.prefix.len()]);
+		s2
+	}
+
+	pub fn add_byte(&self, b: u8) -> Self {
+		let mut s2 = self.clone();
+		s2.prefix.push(b);
+		s2
+	}
+}
+
+impl MerkleNode {
+	fn decode_opt(ent: Option<sled::IVec>) -> Result<Self, Error> {
+		match ent {
+			None => Ok(MerkleNode::Empty),
+			Some(v) => Ok(rmp_serde::decode::from_read_ref::<_, MerkleNode>(&v[..])?),
+		}
+	}
+
+	pub fn is_empty(&self) -> bool {
+		*self == MerkleNode::Empty
+	}
+}
+
+fn intermediate_set_child(ch: &mut Vec<(u8, Hash)>, pos: u8, v: Hash) {
+	for i in 0..ch.len() {
+		if ch[i].0 == pos {
+			ch[i].1 = v;
+			return;
+		} else if ch[i].0 > pos {
+			ch.insert(i, (pos, v));
+			return;
+		}
+	}
+	ch.push((pos, v));
+}
+
+fn intermediate_rm_child(ch: &mut Vec<(u8, Hash)>, pos: u8) {
+	for i in 0..ch.len() {
+		if ch[i].0 == pos {
+			ch.remove(i);
+			return;
+		}
+	}
+}
+
+#[test]
+fn test_intermediate_aux() {
+	let mut v = vec![];
+
+	intermediate_set_child(&mut v, 12u8, [12u8; 32].into());
+	assert_eq!(v, vec![(12u8, [12u8; 32].into())]);
+
+	intermediate_set_child(&mut v, 42u8, [42u8; 32].into());
+	assert_eq!(
+		v,
+		vec![(12u8, [12u8; 32].into()), (42u8, [42u8; 32].into())]
+	);
+
+	intermediate_set_child(&mut v, 4u8, [4u8; 32].into());
+	assert_eq!(
+		v,
+		vec![
+			(4u8, [4u8; 32].into()),
+			(12u8, [12u8; 32].into()),
+			(42u8, [42u8; 32].into())
+		]
+	);
+
+	intermediate_set_child(&mut v, 12u8, [8u8; 32].into());
+	assert_eq!(
+		v,
+		vec![
+			(4u8, [4u8; 32].into()),
+			(12u8, [8u8; 32].into()),
+			(42u8, [42u8; 32].into())
+		]
+	);
+
+	intermediate_set_child(&mut v, 6u8, [6u8; 32].into());
+	assert_eq!(
+		v,
+		vec![
+			(4u8, [4u8; 32].into()),
+			(6u8, [6u8; 32].into()),
+			(12u8, [8u8; 32].into()),
+			(42u8, [42u8; 32].into())
+		]
+	);
+
+	intermediate_rm_child(&mut v, 42u8);
+	assert_eq!(
+		v,
+		vec![
+			(4u8, [4u8; 32].into()),
+			(6u8, [6u8; 32].into()),
+			(12u8, [8u8; 32].into())
+		]
+	);
+
+	intermediate_rm_child(&mut v, 11u8);
+	assert_eq!(
+		v,
+		vec![
+			(4u8, [4u8; 32].into()),
+			(6u8, [6u8; 32].into()),
+			(12u8, [8u8; 32].into())
+		]
+	);
+
+	intermediate_rm_child(&mut v, 6u8);
+	assert_eq!(v, vec![(4u8, [4u8; 32].into()), (12u8, [8u8; 32].into())]);
+
+	intermediate_set_child(&mut v, 6u8, [7u8; 32].into());
+	assert_eq!(
+		v,
+		vec![
+			(4u8, [4u8; 32].into()),
+			(6u8, [7u8; 32].into()),
+			(12u8, [8u8; 32].into())
+		]
+	);
+}
@@ -0,0 +1,51 @@
+use std::sync::Arc;
+
+use garage_rpc::membership::System;
+use garage_rpc::ring::*;
+use garage_util::data::*;
+
+use crate::replication::*;
+
+#[derive(Clone)]
+pub struct TableFullReplication {
+	pub system: Arc<System>,
+	pub max_faults: usize,
+}
+
+impl TableReplication for TableFullReplication {
+	// Full replication schema: all nodes store everything
+	// Writes are disseminated in an epidemic manner in the network
+
+	// Advantage: do all reads locally, extremely fast
+	// Inconvenient: only suitable to reasonably small tables
+
+	fn read_nodes(&self, _hash: &Hash) -> Vec<UUID> {
+		vec![self.system.id]
+	}
+	fn read_quorum(&self) -> usize {
+		1
+	}
+
+	fn write_nodes(&self, _hash: &Hash) -> Vec<UUID> {
+		let ring = self.system.ring.borrow();
+		ring.config.members.keys().cloned().collect::<Vec<_>>()
+	}
+	fn write_quorum(&self) -> usize {
+		let nmembers = self.system.ring.borrow().config.members.len();
+		if nmembers > self.max_faults {
+			nmembers - self.max_faults
+		} else {
+			1
+		}
+	}
+	fn max_write_errors(&self) -> usize {
+		self.max_faults
+	}
+
+	fn partition_of(&self, _hash: &Hash) -> Partition {
+		0u16
+	}
+	fn partitions(&self) -> Vec<(Partition, Hash)> {
+		vec![(0u16, [0u8; 32].into())]
+	}
+}
@@ -0,0 +1,6 @@
+mod parameters;
+
+pub mod fullcopy;
+pub mod sharded;
+
+pub use parameters::*;
@@ -0,0 +1,21 @@
+use garage_rpc::ring::*;
+
+use garage_util::data::*;
+
+pub trait TableReplication: Send + Sync {
+	// See examples in table_sharded.rs and table_fullcopy.rs
+	// To understand various replication methods
+
+	// Which nodes to send reads from
+	fn read_nodes(&self, hash: &Hash) -> Vec<UUID>;
+	fn read_quorum(&self) -> usize;
+
+	// Which nodes to send writes to
+	fn write_nodes(&self, hash: &Hash) -> Vec<UUID>;
+	fn write_quorum(&self) -> usize;
+	fn max_write_errors(&self) -> usize;
+
+	// Accessing partitions, for Merkle tree & sync
+	fn partition_of(&self, hash: &Hash) -> Partition;
+	fn partitions(&self) -> Vec<(Partition, Hash)>;
+}
@@ -1,10 +1,14 @@
-use garage_rpc::membership::{Ring, System};
+use std::sync::Arc;
+
+use garage_rpc::membership::System;
+use garage_rpc::ring::*;
 use garage_util::data::*;

-use crate::*;
+use crate::replication::*;

 #[derive(Clone)]
 pub struct TableShardedReplication {
+	pub system: Arc<System>,
 	pub replication_factor: usize,
 	pub read_quorum: usize,
 	pub write_quorum: usize,
@@ -18,16 +22,16 @@ impl TableReplication for TableShardedReplication {
 	// - reads are done on all of the nodes that replicate the data
 	// - writes as well

-	fn read_nodes(&self, hash: &Hash, system: &System) -> Vec<UUID> {
-		let ring = system.ring.borrow().clone();
+	fn read_nodes(&self, hash: &Hash) -> Vec<UUID> {
+		let ring = self.system.ring.borrow().clone();
 		ring.walk_ring(&hash, self.replication_factor)
 	}
 	fn read_quorum(&self) -> usize {
 		self.read_quorum
 	}

-	fn write_nodes(&self, hash: &Hash, system: &System) -> Vec<UUID> {
-		let ring = system.ring.borrow().clone();
+	fn write_nodes(&self, hash: &Hash) -> Vec<UUID> {
+		let ring = self.system.ring.borrow();
 		ring.walk_ring(&hash, self.replication_factor)
 	}
 	fn write_quorum(&self) -> usize {
@@ -36,21 +40,11 @@ impl TableReplication for TableShardedReplication {
 	fn max_write_errors(&self) -> usize {
 		self.replication_factor - self.write_quorum
 	}
-	fn epidemic_writes(&self) -> bool {
-		false
-	}

-	fn replication_nodes(&self, hash: &Hash, ring: &Ring) -> Vec<UUID> {
-		ring.walk_ring(&hash, self.replication_factor)
+	fn partition_of(&self, hash: &Hash) -> Partition {
+		self.system.ring.borrow().partition_of(hash)
 	}
-	fn split_points(&self, ring: &Ring) -> Vec<Hash> {
-		let mut ret = vec![];
-
-		ret.push([0u8; 32].into());
-		for entry in ring.ring.iter() {
-			ret.push(entry.location);
-		}
-		ret.push([0xFFu8; 32].into());
-		ret
+	fn partitions(&self) -> Vec<(Partition, Hash)> {
+		self.system.ring.borrow().partitions()
 	}
 }
@@ -1,8 +1,8 @@
-use async_trait::async_trait;
 use serde::{Deserialize, Serialize};

 use garage_util::data::*;
-use garage_util::error::Error;
+
+use crate::crdt::CRDT;

 pub trait PartitionKey {
 	fn hash(&self) -> Hash;
@@ -10,7 +10,7 @@ pub trait PartitionKey {

 impl PartitionKey for String {
 	fn hash(&self) -> Hash {
-		hash(self.as_bytes())
+		blake2sum(self.as_bytes())
 	}
 }

@@ -37,15 +37,16 @@ impl SortKey for Hash {
 }

 pub trait Entry<P: PartitionKey, S: SortKey>:
-	PartialEq + Clone + Serialize + for<'de> Deserialize<'de> + Send + Sync
+	CRDT + PartialEq + Clone + Serialize + for<'de> Deserialize<'de> + Send + Sync
 {
 	fn partition_key(&self) -> &P;
 	fn sort_key(&self) -> &S;

-	fn merge(&mut self, other: &Self);
+	fn is_tombstone(&self) -> bool {
+		false
+	}
 }

-#[async_trait]
 pub trait TableSchema: Send + Sync {
 	type P: PartitionKey + Clone + PartialEq + Serialize + for<'de> Deserialize<'de> + Send + Sync;
 	type S: SortKey + Clone + Serialize + for<'de> Deserialize<'de> + Send + Sync;
@@ -58,7 +59,12 @@ pub trait TableSchema: Send + Sync {
 		None
 	}

-	async fn updated(&self, old: Option<Self::E>, new: Option<Self::E>) -> Result<(), Error>;
+	// Updated triggers some stuff downstream, but it is not supposed to block or fail,
+	// as the update itself is an unchangeable fact that will never go back
+	// due to CRDT logic. Typically errors in propagation of info should be logged
+	// to stderr.
+	fn updated(&self, _old: Option<Self::E>, _new: Option<Self::E>) {}
+
 	fn matches_filter(_entry: &Self::E, _filter: &Self::Filter) -> bool {
 		true
 	}
@@ -0,0 +1,614 @@
+use std::collections::VecDeque;
+use std::sync::{Arc, Mutex};
+use std::time::{Duration, Instant};
+
+use futures::select;
+use futures_util::future::*;
+use futures_util::stream::*;
+use rand::Rng;
+use serde::{Deserialize, Serialize};
+use serde_bytes::ByteBuf;
+use tokio::sync::{mpsc, watch};
+
+use garage_util::data::*;
+use garage_util::error::Error;
+
+use garage_rpc::membership::System;
+use garage_rpc::ring::*;
+use garage_rpc::rpc_client::*;
+use garage_rpc::rpc_server::*;
+
+use crate::data::*;
+use crate::merkle::*;
+use crate::replication::*;
+use crate::*;
+
+const TABLE_SYNC_RPC_TIMEOUT: Duration = Duration::from_secs(30);
+
+// Do anti-entropy every 10 minutes
+const ANTI_ENTROPY_INTERVAL: Duration = Duration::from_secs(10 * 60);
+
+pub struct TableSyncer<F: TableSchema, R: TableReplication> {
+	system: Arc<System>,
+	data: Arc<TableData<F, R>>,
+	merkle: Arc<MerkleUpdater<F, R>>,
+
+	todo: Mutex<SyncTodo>,
+	rpc_client: Arc<RpcClient<SyncRPC>>,
+}
+
+#[derive(Serialize, Deserialize)]
+pub(crate) enum SyncRPC {
+	RootCkHash(Partition, Hash),
+	RootCkDifferent(bool),
+	GetNode(MerkleNodeKey),
+	Node(MerkleNodeKey, MerkleNode),
+	Items(Vec<Arc<ByteBuf>>),
+	Ok,
+}
+
+impl RpcMessage for SyncRPC {}
+
+struct SyncTodo {
+	todo: Vec<TodoPartition>,
+}
+
+#[derive(Debug, Clone)]
+struct TodoPartition {
+	partition: Partition,
+	begin: Hash,
+	end: Hash,
+
+	// Are we a node that stores this partition or not?
+	retain: bool,
+}
+
+impl<F, R> TableSyncer<F, R>
+where
+	F: TableSchema + 'static,
+	R: TableReplication + 'static,
+{
+	pub(crate) fn launch(
+		system: Arc<System>,
+		data: Arc<TableData<F, R>>,
+		merkle: Arc<MerkleUpdater<F, R>>,
+		rpc_server: &mut RpcServer,
+	) -> Arc<Self> {
+		let rpc_path = format!("table_{}/sync", data.name);
+		let rpc_client = system.rpc_client::<SyncRPC>(&rpc_path);
+
+		let todo = SyncTodo { todo: vec![] };
+
+		let syncer = Arc::new(Self {
+			system: system.clone(),
+			data: data.clone(),
+			merkle,
+			todo: Mutex::new(todo),
+			rpc_client,
+		});
+
+		syncer.register_handler(rpc_server, rpc_path);
+
+		let (busy_tx, busy_rx) = mpsc::unbounded_channel();
+
+		let s1 = syncer.clone();
+		system.background.spawn_worker(
+			format!("table sync watcher for {}", data.name),
+			move |must_exit: watch::Receiver<bool>| s1.watcher_task(must_exit, busy_rx),
+		);
+
+		let s2 = syncer.clone();
+		system.background.spawn_worker(
+			format!("table syncer for {}", data.name),
+			move |must_exit: watch::Receiver<bool>| s2.syncer_task(must_exit, busy_tx),
+		);
+
+		let s3 = syncer.clone();
+		tokio::spawn(async move {
+			tokio::time::sleep(Duration::from_secs(20)).await;
+			s3.add_full_sync();
+		});
+
+		syncer
+	}
+
+	fn register_handler(self: &Arc<Self>, rpc_server: &mut RpcServer, path: String) {
+		let self2 = self.clone();
+		rpc_server.add_handler::<SyncRPC, _, _>(path, move |msg, _addr| {
+			let self2 = self2.clone();
+			async move { self2.handle_rpc(&msg).await }
+		});
+
+		let self2 = self.clone();
+		self.rpc_client
+			.set_local_handler(self.system.id, move |msg| {
+				let self2 = self2.clone();
+				async move { self2.handle_rpc(&msg).await }
+			});
+	}
+
+	async fn watcher_task(
+		self: Arc<Self>,
+		mut must_exit: watch::Receiver<bool>,
+		mut busy_rx: mpsc::UnboundedReceiver<bool>,
+	) {
+		let mut prev_ring: Arc<Ring> = self.system.ring.borrow().clone();
+		let mut ring_recv: watch::Receiver<Arc<Ring>> = self.system.ring.clone();
+		let mut nothing_to_do_since = Some(Instant::now());
+
+		while !*must_exit.borrow() {
+			select! {
+				_ = ring_recv.changed().fuse() => {
+					let new_ring = ring_recv.borrow();
+					if !Arc::ptr_eq(&new_ring, &prev_ring) {
+						debug!("({}) Ring changed, adding full sync to syncer todo list", self.data.name);
+						self.add_full_sync();
+						prev_ring = new_ring.clone();
+					}
+				}
+				busy_opt = busy_rx.recv().fuse() => {
+					if let Some(busy) = busy_opt {
+						if busy {
+							nothing_to_do_since = None;
+						} else {
+							if nothing_to_do_since.is_none() {
+								nothing_to_do_since = Some(Instant::now());
+							}
+						}
+					}
+				}
+				_ = must_exit.changed().fuse() => (),
+				_ = tokio::time::sleep(Duration::from_secs(1)).fuse() => {
+					if nothing_to_do_since.map(|t| Instant::now() - t >= ANTI_ENTROPY_INTERVAL).unwrap_or(false) {
+						nothing_to_do_since = None;
+						debug!("({}) Interval passed, adding full sync to syncer todo list", self.data.name);
+						self.add_full_sync();
+					}
+				}
+			}
+		}
+	}
+
+	pub fn add_full_sync(&self) {
+		self.todo
+			.lock()
+			.unwrap()
+			.add_full_sync(&self.data, &self.system);
+	}
+
+	async fn syncer_task(
+		self: Arc<Self>,
+		mut must_exit: watch::Receiver<bool>,
+		busy_tx: mpsc::UnboundedSender<bool>,
+	) {
+		while !*must_exit.borrow() {
+			let task = self.todo.lock().unwrap().pop_task();
+			if let Some(partition) = task {
+				busy_tx.send(true).unwrap();
+				let res = self
+					.clone()
+					.sync_partition(&partition, &mut must_exit)
+					.await;
+				if let Err(e) = res {
+					warn!(
+						"({}) Error while syncing {:?}: {}",
+						self.data.name, partition, e
+					);
+				}
+			} else {
+				busy_tx.send(false).unwrap();
+				tokio::time::sleep(Duration::from_secs(1)).await;
+			}
+		}
+	}
+
+	async fn sync_partition(
+		self: Arc<Self>,
+		partition: &TodoPartition,
+		must_exit: &mut watch::Receiver<bool>,
+	) -> Result<(), Error> {
+		if partition.retain {
+			let my_id = self.system.id;
+
+			let nodes = self
+				.data
+				.replication
+				.write_nodes(&partition.begin)
+				.into_iter()
+				.filter(|node| *node != my_id)
+				.collect::<Vec<_>>();
+
+			debug!(
+				"({}) Syncing {:?} with {:?}...",
+				self.data.name, partition, nodes
+			);
+			let mut sync_futures = nodes
+				.iter()
+				.map(|node| {
+					self.clone()
+						.do_sync_with(partition.clone(), *node, must_exit.clone())
+				})
+				.collect::<FuturesUnordered<_>>();
+
+			let mut n_errors = 0;
+			while let Some(r) = sync_futures.next().await {
+				if let Err(e) = r {
+					n_errors += 1;
+					warn!("({}) Sync error: {}", self.data.name, e);
+				}
+			}
+			if n_errors > self.data.replication.max_write_errors() {
+				return Err(Error::Message(format!(
+					"Sync failed with too many nodes (should have been: {:?}).",
+					nodes
+				)));
+			}
+		} else {
+			self.offload_partition(&partition.begin, &partition.end, must_exit)
+				.await?;
+		}
+
+		Ok(())
+	}
+
+	// Offload partition: this partition is not something we are storing,
+	// so send it out to all other nodes that store it and delete items locally.
+	// We don't bother checking if the remote nodes already have the items,
+	// we just batch-send everything. Offloading isn't supposed to happen very often.
+	// If any of the nodes that are supposed to store the items is unable to
+	// save them, we interrupt the process.
+	async fn offload_partition(
+		self: &Arc<Self>,
+		begin: &Hash,
+		end: &Hash,
+		must_exit: &mut watch::Receiver<bool>,
+	) -> Result<(), Error> {
+		let mut counter: usize = 0;
+
+		while !*must_exit.borrow() {
+			let mut items = Vec::new();
+
+			for item in self.data.store.range(begin.to_vec()..end.to_vec()) {
+				let (key, value) = item?;
+				items.push((key.to_vec(), Arc::new(ByteBuf::from(value.as_ref()))));
+
+				if items.len() >= 1024 {
+					break;
+				}
+			}
+
+			if items.len() > 0 {
+				let nodes = self
+					.data
+					.replication
+					.write_nodes(&begin)
+					.into_iter()
+					.collect::<Vec<_>>();
+				if nodes.contains(&self.system.id) {
+					warn!(
+						"({}) Interrupting offload as partitions seem to have changed",
+						self.data.name
+					);
+					break;
+				}
+				if nodes.len() < self.data.replication.write_quorum() {
+					return Err(Error::Message(format!(
+						"Not offloading as we don't have a quorum of nodes to write to."
+					)));
+				}
+
+				counter += 1;
+				info!(
+					"({}) Offloading {} items from {:?}..{:?} ({})",
+					self.data.name,
+					items.len(),
+					begin,
+					end,
+					counter
+				);
+				self.offload_items(&items, &nodes[..]).await?;
+			} else {
+				break;
+			}
+		}
+
+		Ok(())
+	}
+
+	async fn offload_items(
+		self: &Arc<Self>,
+		items: &Vec<(Vec<u8>, Arc<ByteBuf>)>,
+		nodes: &[UUID],
+	) -> Result<(), Error> {
+		let values = items.iter().map(|(_k, v)| v.clone()).collect::<Vec<_>>();
+
+		self.rpc_client
+			.try_call_many(
+				&nodes[..],
+				SyncRPC::Items(values),
+				RequestStrategy::with_quorum(nodes.len()).with_timeout(TABLE_SYNC_RPC_TIMEOUT),
+			)
+			.await?;
+
+		// All remote nodes have written those items, now we can delete them locally
+		let mut not_removed = 0;
+		for (k, v) in items.iter() {
+			if !self.data.delete_if_equal(&k[..], &v[..])? {
+				not_removed += 1;
+			}
+		}
+
+		if not_removed > 0 {
+			debug!("({}) {} items not removed during offload because they changed in between (trying again...)", self.data.name, not_removed);
+		}
+
+		Ok(())
+	}
+
+	// ======= SYNCHRONIZATION PROCEDURE -- DRIVER SIDE ======
+	// The driver side is only concerned with sending out the item it has
+	// and the other side might not have. Receiving items that differ from one
+	// side to the other will happen when the other side syncs with us,
+	// which they also do regularly.
+
+	fn get_root_ck(&self, partition: Partition) -> Result<(MerkleNodeKey, MerkleNode), Error> {
+		let key = MerkleNodeKey {
+			partition,
+			prefix: vec![],
+		};
+		let node = self.merkle.read_node(&key)?;
+		Ok((key, node))
+	}
+
+	async fn do_sync_with(
+		self: Arc<Self>,
+		partition: TodoPartition,
+		who: UUID,
+		must_exit: watch::Receiver<bool>,
+	) -> Result<(), Error> {
+		let (root_ck_key, root_ck) = self.get_root_ck(partition.partition)?;
+		if root_ck.is_empty() {
+			debug!(
+				"({}) Sync {:?} with {:?}: partition is empty.",
+				self.data.name, partition, who
+			);
+			return Ok(());
+		}
+		let root_ck_hash = hash_of::<MerkleNode>(&root_ck)?;
+
+		// Check if they have the same root checksum
+		// If so, do nothing.
+		let root_resp = self
+			.rpc_client
+			.call(
+				who,
+				SyncRPC::RootCkHash(partition.partition, root_ck_hash),
+				TABLE_SYNC_RPC_TIMEOUT,
+			)
+			.await?;
+
+		let mut todo = match root_resp {
+			SyncRPC::RootCkDifferent(false) => {
+				debug!(
+					"({}) Sync {:?} with {:?}: no difference",
+					self.data.name, partition, who
+				);
+				return Ok(());
+			}
+			SyncRPC::RootCkDifferent(true) => VecDeque::from(vec![root_ck_key]),
+			x => {
+				return Err(Error::Message(format!(
+					"Invalid respone to RootCkHash RPC: {}",
+					debug_serialize(x)
+				)));
+			}
+		};
+
+		let mut todo_items = vec![];
+
+		while !todo.is_empty() && !*must_exit.borrow() {
+			let key = todo.pop_front().unwrap();
+			let node = self.merkle.read_node(&key)?;
+
+			match node {
+				MerkleNode::Empty => {
+					// They have items we don't have.
+					// We don't request those items from them, they will send them.
+					// We only bother with pushing items that differ
+				}
+				MerkleNode::Leaf(ik, ivhash) => {
+					// Just send that item directly
+					if let Some(val) = self.data.store.get(&ik[..])? {
+						if blake2sum(&val[..]) != ivhash {
+							warn!("({}) Hashes differ between stored value and Merkle tree, key: {:?} (if your server is very busy, don't worry, this happens when the Merkle tree can't be updated fast enough)", self.data.name, ik);
+						}
+						todo_items.push(val.to_vec());
+					} else {
+						warn!("({}) Item from Merkle tree not found in store: {:?} (if your server is very busy, don't worry, this happens when the Merkle tree can't be updated fast enough)", self.data.name, ik);
+					}
+				}
+				MerkleNode::Intermediate(l) => {
+					// Get Merkle node for this tree position at remote node
+					// and compare it with local node
+					let remote_node = match self
+						.rpc_client
+						.call(who, SyncRPC::GetNode(key.clone()), TABLE_SYNC_RPC_TIMEOUT)
+						.await?
+					{
+						SyncRPC::Node(_, node) => node,
+						x => {
+							return Err(Error::Message(format!(
+								"Invalid respone to GetNode RPC: {}",
+								debug_serialize(x)
+							)));
+						}
+					};
+					let int_l2 = match remote_node {
+						// If they have an intermediate node at this tree position,
+						// we can compare them to find differences
+						MerkleNode::Intermediate(l2) => l2,
+						// Otherwise, treat it as if they have nothing for this subtree,
+						// which will have the consequence of sending them everything
+						_ => vec![],
+					};
+
+					let join = join_ordered(&l[..], &int_l2[..]);
+					for (p, v1, v2) in join.into_iter() {
+						let diff = match (v1, v2) {
+							(Some(_), None) | (None, Some(_)) => true,
+							(Some(a), Some(b)) => a != b,
+							_ => false,
+						};
+						if diff {
+							todo.push_back(key.add_byte(*p));
+						}
+					}
+				}
+			}
+
+			if todo_items.len() >= 256 {
+				self.send_items(who, std::mem::replace(&mut todo_items, vec![]))
+					.await?;
+			}
+		}
+
+		if !todo_items.is_empty() {
+			self.send_items(who, todo_items).await?;
+		}
+
+		Ok(())
+	}
+
+	async fn send_items(&self, who: UUID, item_value_list: Vec<Vec<u8>>) -> Result<(), Error> {
+		info!(
+			"({}) Sending {} items to {:?}",
+			self.data.name,
+			item_value_list.len(),
+			who
+		);
+
+		let values = item_value_list
+			.into_iter()
+			.map(|x| Arc::new(ByteBuf::from(x)))
+			.collect::<Vec<_>>();
+
+		let rpc_resp = self
+			.rpc_client
+			.call(who, SyncRPC::Items(values), TABLE_SYNC_RPC_TIMEOUT)
+			.await?;
+		if let SyncRPC::Ok = rpc_resp {
+			Ok(())
+		} else {
+			Err(Error::Message(format!(
+				"Unexpected response to RPC Update: {}",
+				debug_serialize(&rpc_resp)
+			)))
+		}
+	}
+
+	// ======= SYNCHRONIZATION PROCEDURE -- RECEIVER SIDE ======
+
+	async fn handle_rpc(self: &Arc<Self>, message: &SyncRPC) -> Result<SyncRPC, Error> {
+		match message {
+			SyncRPC::RootCkHash(range, h) => {
+				let (_root_ck_key, root_ck) = self.get_root_ck(*range)?;
+				let hash = hash_of::<MerkleNode>(&root_ck)?;
+				Ok(SyncRPC::RootCkDifferent(hash != *h))
+			}
+			SyncRPC::GetNode(k) => {
+				let node = self.merkle.read_node(&k)?;
+				Ok(SyncRPC::Node(k.clone(), node))
+			}
+			SyncRPC::Items(items) => {
+				self.data.update_many(items)?;
+				Ok(SyncRPC::Ok)
+			}
+			_ => Err(Error::Message(format!("Unexpected sync RPC"))),
+		}
+	}
+}
+
+impl SyncTodo {
+	fn add_full_sync<F: TableSchema, R: TableReplication>(
+		&mut self,
+		data: &TableData<F, R>,
+		system: &System,
+	) {
+		let my_id = system.id;
+
+		self.todo.clear();
+
+		let partitions = data.replication.partitions();
+
+		for i in 0..partitions.len() {
+			let begin = partitions[i].1;
+
+			let end = if i + 1 < partitions.len() {
+				partitions[i + 1].1
+			} else {
+				[0xFFu8; 32].into()
+			};
+
+			let nodes = data.replication.write_nodes(&begin);
+
+			let retain = nodes.contains(&my_id);
+			if !retain {
+				// Check if we have some data to send, otherwise skip
+				if data.store.range(begin..end).next().is_none() {
+					continue;
+				}
+			}
+
+			self.todo.push(TodoPartition {
+				partition: partitions[i].0,
+				begin,
+				end,
+				retain,
+			});
+		}
+	}
+
+	fn pop_task(&mut self) -> Option<TodoPartition> {
+		if self.todo.is_empty() {
+			return None;
+		}
+
+		let i = rand::thread_rng().gen_range(0..self.todo.len());
+		if i == self.todo.len() - 1 {
+			self.todo.pop()
+		} else {
+			let replacement = self.todo.pop().unwrap();
+			let ret = std::mem::replace(&mut self.todo[i], replacement);
+			Some(ret)
+		}
+	}
+}
+
+fn hash_of<T: Serialize>(x: &T) -> Result<Hash, Error> {
+	Ok(blake2sum(&rmp_to_vec_all_named(x)?[..]))
+}
+
+fn join_ordered<'a, K: Ord + Eq, V1, V2>(
+	x: &'a [(K, V1)],
+	y: &'a [(K, V2)],
+) -> Vec<(&'a K, Option<&'a V1>, Option<&'a V2>)> {
+	let mut ret = vec![];
+	let mut i = 0;
+	let mut j = 0;
+	while i < x.len() || j < y.len() {
+		if i < x.len() && j < y.len() && x[i].0 == y[j].0 {
+			ret.push((&x[i].0, Some(&x[i].1), Some(&y[j].1)));
+			i += 1;
+			j += 1;
+		} else if i < x.len() && (j == y.len() || x[i].0 < y[j].0) {
+			ret.push((&x[i].0, Some(&x[i].1), None));
+			i += 1;
+		} else if j < y.len() && (i == x.len() || x[i].0 > y[j].0) {
+			ret.push((&y[j].0, None, Some(&y[j].1)));
+			j += 1;
+		} else {
+			unreachable!();
+		}
+	}
+	ret
+}
@@ -2,9 +2,6 @@ use std::collections::{BTreeMap, HashMap};
 use std::sync::Arc;
 use std::time::Duration;

-use log::warn;
-
-use arc_swap::ArcSwapOption;
 use futures::stream::*;
 use serde::{Deserialize, Serialize};
 use serde_bytes::ByteBuf;
@@ -12,29 +9,30 @@ use serde_bytes::ByteBuf;
 use garage_util::data::*;
 use garage_util::error::Error;

-use garage_rpc::membership::{Ring, System};
+use garage_rpc::membership::System;
 use garage_rpc::rpc_client::*;
 use garage_rpc::rpc_server::*;

+use crate::crdt::CRDT;
+use crate::data::*;
+use crate::gc::*;
+use crate::merkle::*;
+use crate::replication::*;
 use crate::schema::*;
-use crate::table_sync::*;
+use crate::sync::*;

 const TABLE_RPC_TIMEOUT: Duration = Duration::from_secs(10);

 pub struct Table<F: TableSchema, R: TableReplication> {
-	pub instance: F,
-	pub replication: R,
-
-	pub name: String,
-	pub rpc_client: Arc<RpcClient<TableRPC<F>>>,
-
 	pub system: Arc<System>,
-	pub store: sled::Tree,
-	pub syncer: ArcSwapOption<TableSyncer<F, R>>,
+	pub data: Arc<TableData<F, R>>,
+	pub merkle_updater: Arc<MerkleUpdater<F, R>>,
+	pub syncer: Arc<TableSyncer<F, R>>,
+	rpc_client: Arc<RpcClient<TableRPC<F>>>,
 }

 #[derive(Serialize, Deserialize)]
-pub enum TableRPC<F: TableSchema> {
+pub(crate) enum TableRPC<F: TableSchema> {
 	Ok,

 	ReadEntry(F::P, F::S),
@@ -44,31 +42,10 @@ pub enum TableRPC<F: TableSchema> {
 	ReadRange(F::P, Option<F::S>, Option<F::Filter>, usize),

 	Update(Vec<Arc<ByteBuf>>),
-
-	SyncRPC(SyncRPC),
 }

 impl<F: TableSchema> RpcMessage for TableRPC<F> {}

-pub trait TableReplication: Send + Sync {
-	// See examples in table_sharded.rs and table_fullcopy.rs
-	// To understand various replication methods
-
-	// Which nodes to send reads from
-	fn read_nodes(&self, hash: &Hash, system: &System) -> Vec<UUID>;
-	fn read_quorum(&self) -> usize;
-
-	// Which nodes to send writes to
-	fn write_nodes(&self, hash: &Hash, system: &System) -> Vec<UUID>;
-	fn write_quorum(&self) -> usize;
-	fn max_write_errors(&self) -> usize;
-	fn epidemic_writes(&self) -> bool;
-
-	// Which are the nodes that do actually replicate the data
-	fn replication_nodes(&self, hash: &Hash, ring: &Ring) -> Vec<UUID>;
-	fn split_points(&self, ring: &Ring) -> Vec<Hash>;
-}
-
 impl<F, R> Table<F, R>
 where
 	F: TableSchema + 'static,
@@ -76,7 +53,7 @@ where
 {
 	// =============== PUBLIC INTERFACE FUNCTIONS (new, insert, get, etc) ===============

-	pub async fn new(
+	pub fn new(
 		instance: F,
 		replication: R,
 		system: Arc<System>,
@@ -84,31 +61,37 @@ where
 		name: String,
 		rpc_server: &mut RpcServer,
 	) -> Arc<Self> {
-		let store = db.open_tree(&name).expect("Unable to open DB tree");
-
 		let rpc_path = format!("table_{}", name);
 		let rpc_client = system.rpc_client::<TableRPC<F>>(&rpc_path);

-		let table = Arc::new(Self {
-			instance,
-			replication,
-			name,
-			rpc_client,
-			system,
-			store,
-			syncer: ArcSwapOption::from(None),
-		});
-		table.clone().register_handler(rpc_server, rpc_path);
+		let data = TableData::new(system.clone(), name, instance, replication, db);

-		let syncer = TableSyncer::launch(table.clone()).await;
-		table.syncer.swap(Some(syncer));
+		let merkle_updater = MerkleUpdater::launch(&system.background, data.clone());
+
+		let syncer = TableSyncer::launch(
+			system.clone(),
+			data.clone(),
+			merkle_updater.clone(),
+			rpc_server,
+		);
+		TableGC::launch(system.clone(), data.clone(), rpc_server);
+
+		let table = Arc::new(Self {
+			system,
+			data,
+			merkle_updater,
+			syncer,
+			rpc_client,
+		});
+
+		table.clone().register_handler(rpc_server, rpc_path);

 		table
 	}

 	pub async fn insert(&self, e: &F::E) -> Result<(), Error> {
 		let hash = e.partition_key().hash();
-		let who = self.replication.write_nodes(&hash, &self.system);
+		let who = self.data.replication.write_nodes(&hash);
 		//eprintln!("insert who: {:?}", who);

 		let e_enc = Arc::new(ByteBuf::from(rmp_to_vec_all_named(e)?));
@@ -118,7 +101,7 @@ where
 			.try_call_many(
 				&who[..],
 				rpc,
-				RequestStrategy::with_quorum(self.replication.write_quorum())
+				RequestStrategy::with_quorum(self.data.replication.write_quorum())
 					.with_timeout(TABLE_RPC_TIMEOUT),
 			)
 			.await?;
@@ -130,7 +113,7 @@ where

 		for entry in entries.iter() {
 			let hash = entry.partition_key().hash();
-			let who = self.replication.write_nodes(&hash, &self.system);
+			let who = self.data.replication.write_nodes(&hash);
 			let e_enc = Arc::new(ByteBuf::from(rmp_to_vec_all_named(entry)?));
 			for node in who {
 				if !call_list.contains_key(&node) {
@@ -154,7 +137,7 @@ where
 				errors.push(e);
 			}
 		}
-		if errors.len() > self.replication.max_write_errors() {
+		if errors.len() > self.data.replication.max_write_errors() {
 			Err(Error::Message("Too many errors".into()))
 		} else {
 			Ok(())
@@ -167,7 +150,7 @@ where
 		sort_key: &F::S,
 	) -> Result<Option<F::E>, Error> {
 		let hash = partition_key.hash();
-		let who = self.replication.read_nodes(&hash, &self.system);
+		let who = self.data.replication.read_nodes(&hash);
 		//eprintln!("get who: {:?}", who);

 		let rpc = TableRPC::<F>::ReadEntry(partition_key.clone(), sort_key.clone());
@@ -176,7 +159,7 @@ where
 			.try_call_many(
 				&who[..],
 				rpc,
-				RequestStrategy::with_quorum(self.replication.read_quorum())
+				RequestStrategy::with_quorum(self.data.replication.read_quorum())
 					.with_timeout(TABLE_RPC_TIMEOUT)
 					.interrupt_after_quorum(true),
 			)
@@ -187,7 +170,7 @@ where
 		for resp in resps {
 			if let TableRPC::ReadEntryResponse(value) = resp {
 				if let Some(v_bytes) = value {
-					let v = self.decode_entry(v_bytes.as_slice())?;
+					let v = self.data.decode_entry(v_bytes.as_slice())?;
 					ret = match ret {
 						None => Some(v),
 						Some(mut x) => {
@@ -223,7 +206,7 @@ where
 		limit: usize,
 	) -> Result<Vec<F::E>, Error> {
 		let hash = partition_key.hash();
-		let who = self.replication.read_nodes(&hash, &self.system);
+		let who = self.data.replication.read_nodes(&hash);

 		let rpc = TableRPC::<F>::ReadRange(partition_key.clone(), begin_sort_key, filter, limit);

@@ -232,7 +215,7 @@ where
 			.try_call_many(
 				&who[..],
 				rpc,
-				RequestStrategy::with_quorum(self.replication.read_quorum())
+				RequestStrategy::with_quorum(self.data.replication.read_quorum())
 					.with_timeout(TABLE_RPC_TIMEOUT)
 					.interrupt_after_quorum(true),
 			)
@@ -243,8 +226,8 @@ where
 		for resp in resps {
 			if let TableRPC::Update(entries) = resp {
 				for entry_bytes in entries.iter() {
-					let entry = self.decode_entry(entry_bytes.as_slice())?;
-					let entry_key = self.tree_key(entry.partition_key(), entry.sort_key());
+					let entry = self.data.decode_entry(entry_bytes.as_slice())?;
+					let entry_key = self.data.tree_key(entry.partition_key(), entry.sort_key());
 					match ret.remove(&entry_key) {
 						None => {
 							ret.insert(entry_key, Some(entry));
@@ -313,164 +296,18 @@ where
 	async fn handle(self: &Arc<Self>, msg: &TableRPC<F>) -> Result<TableRPC<F>, Error> {
 		match msg {
 			TableRPC::ReadEntry(key, sort_key) => {
-				let value = self.handle_read_entry(key, sort_key)?;
+				let value = self.data.read_entry(key, sort_key)?;
 				Ok(TableRPC::ReadEntryResponse(value))
 			}
 			TableRPC::ReadRange(key, begin_sort_key, filter, limit) => {
-				let values = self.handle_read_range(key, begin_sort_key, filter, *limit)?;
+				let values = self.data.read_range(key, begin_sort_key, filter, *limit)?;
 				Ok(TableRPC::Update(values))
 			}
 			TableRPC::Update(pairs) => {
-				self.handle_update(pairs).await?;
+				self.data.update_many(pairs)?;
 				Ok(TableRPC::Ok)
 			}
-			TableRPC::SyncRPC(rpc) => {
-				let syncer = self.syncer.load_full().unwrap();
-				let response = syncer
-					.handle_rpc(rpc, self.system.background.stop_signal.clone())
-					.await?;
-				Ok(TableRPC::SyncRPC(response))
-			}
 			_ => Err(Error::BadRPC(format!("Unexpected table RPC"))),
 		}
 	}
-
-	fn handle_read_entry(&self, p: &F::P, s: &F::S) -> Result<Option<ByteBuf>, Error> {
-		let tree_key = self.tree_key(p, s);
-		if let Some(bytes) = self.store.get(&tree_key)? {
-			Ok(Some(ByteBuf::from(bytes.to_vec())))
-		} else {
-			Ok(None)
-		}
-	}
-
-	fn handle_read_range(
-		&self,
-		p: &F::P,
-		s: &Option<F::S>,
-		filter: &Option<F::Filter>,
-		limit: usize,
-	) -> Result<Vec<Arc<ByteBuf>>, Error> {
-		let partition_hash = p.hash();
-		let first_key = match s {
-			None => partition_hash.to_vec(),
-			Some(sk) => self.tree_key(p, sk),
-		};
-		let mut ret = vec![];
-		for item in self.store.range(first_key..) {
-			let (key, value) = item?;
-			if &key[..32] != partition_hash.as_slice() {
-				break;
-			}
-			let keep = match filter {
-				None => true,
-				Some(f) => {
-					let entry = self.decode_entry(value.as_ref())?;
-					F::matches_filter(&entry, f)
-				}
-			};
-			if keep {
-				ret.push(Arc::new(ByteBuf::from(value.as_ref())));
-			}
-			if ret.len() >= limit {
-				break;
-			}
-		}
-		Ok(ret)
-	}
-
-	pub async fn handle_update(self: &Arc<Self>, entries: &[Arc<ByteBuf>]) -> Result<(), Error> {
-		let syncer = self.syncer.load_full().unwrap();
-		let mut epidemic_propagate = vec![];
-
-		for update_bytes in entries.iter() {
-			let update = self.decode_entry(update_bytes.as_slice())?;
-
-			let tree_key = self.tree_key(update.partition_key(), update.sort_key());
-
-			let (old_entry, new_entry) = self.store.transaction(|db| {
-				let (old_entry, new_entry) = match db.get(&tree_key)? {
-					Some(prev_bytes) => {
-						let old_entry = self
-							.decode_entry(&prev_bytes)
-							.map_err(sled::ConflictableTransactionError::Abort)?;
-						let mut new_entry = old_entry.clone();
-						new_entry.merge(&update);
-						(Some(old_entry), new_entry)
-					}
-					None => (None, update.clone()),
-				};
-
-				let new_bytes = rmp_to_vec_all_named(&new_entry)
-					.map_err(Error::RMPEncode)
-					.map_err(sled::ConflictableTransactionError::Abort)?;
-				db.insert(tree_key.clone(), new_bytes)?;
-				Ok((old_entry, new_entry))
-			})?;
-
-			if old_entry.as_ref() != Some(&new_entry) {
-				if self.replication.epidemic_writes() {
-					epidemic_propagate.push(new_entry.clone());
-				}
-
-				self.instance.updated(old_entry, Some(new_entry)).await?;
-				self.system
-					.background
-					.spawn_cancellable(syncer.clone().invalidate(tree_key));
-			}
-		}
-
-		if epidemic_propagate.len() > 0 {
-			let self2 = self.clone();
-			self.system
-				.background
-				.spawn_cancellable(async move { self2.insert_many(&epidemic_propagate[..]).await });
-		}
-
-		Ok(())
-	}
-
-	pub async fn delete_range(&self, begin: &Hash, end: &Hash) -> Result<(), Error> {
-		let syncer = self.syncer.load_full().unwrap();
-
-		debug!("({}) Deleting range {:?} - {:?}", self.name, begin, end);
-		let mut count = 0;
-		while let Some((key, _value)) = self.store.get_lt(end.as_slice())? {
-			if key.as_ref() < begin.as_slice() {
-				break;
-			}
-			if let Some(old_val) = self.store.remove(&key)? {
-				let old_entry = self.decode_entry(&old_val)?;
-				self.instance.updated(Some(old_entry), None).await?;
-				self.system
-					.background
-					.spawn_cancellable(syncer.clone().invalidate(key.to_vec()));
-				count += 1;
-			}
-		}
-		debug!("({}) {} entries deleted", self.name, count);
-		Ok(())
-	}
-
-	fn tree_key(&self, p: &F::P, s: &F::S) -> Vec<u8> {
-		let mut ret = p.hash().to_vec();
-		ret.extend(s.sort_key());
-		ret
-	}
-
-	fn decode_entry(&self, bytes: &[u8]) -> Result<F::E, Error> {
-		match rmp_serde::decode::from_read_ref::<_, F::E>(bytes) {
-			Ok(x) => Ok(x),
-			Err(e) => match F::try_migrate(bytes) {
-				Some(x) => Ok(x),
-				None => {
-					warn!("Unable to decode entry of {}: {}", self.name, e);
-					for line in hexdump::hexdump_iter(bytes) {
-						debug!("{}", line);
-					}
-					Err(e.into())
-				}
-			},
-		}
-	}
 }
@@ -1,101 +0,0 @@
-use arc_swap::ArcSwapOption;
-use std::sync::Arc;
-
-use garage_rpc::membership::{Ring, System};
-use garage_util::data::*;
-
-use crate::*;
-
-#[derive(Clone)]
-pub struct TableFullReplication {
-	pub write_factor: usize,
-	pub write_quorum: usize,
-
-	neighbors: ArcSwapOption<Neighbors>,
-}
-
-#[derive(Clone)]
-struct Neighbors {
-	ring: Arc<Ring>,
-	neighbors: Vec<UUID>,
-}
-
-impl TableFullReplication {
-	pub fn new(write_factor: usize, write_quorum: usize) -> Self {
-		TableFullReplication {
-			write_factor,
-			write_quorum,
-			neighbors: ArcSwapOption::from(None),
-		}
-	}
-
-	fn get_neighbors(&self, system: &System) -> Vec<UUID> {
-		let neighbors = self.neighbors.load_full();
-		if let Some(n) = neighbors {
-			if Arc::ptr_eq(&n.ring, &system.ring.borrow()) {
-				return n.neighbors.clone();
-			}
-		}
-
-		// Recalculate neighbors
-		let ring = system.ring.borrow().clone();
-		let my_id = system.id;
-
-		let mut nodes = vec![];
-		for (node, _) in ring.config.members.iter() {
-			let node_ranking = hash(&[node.as_slice(), my_id.as_slice()].concat());
-			nodes.push((*node, node_ranking));
-		}
-		nodes.sort_by(|(_, rank1), (_, rank2)| rank1.cmp(rank2));
-		let mut neighbors = nodes
-			.drain(..)
-			.map(|(node, _)| node)
-			.filter(|node| *node != my_id)
-			.take(self.write_factor)
-			.collect::<Vec<_>>();
-		neighbors.push(my_id);
-		self.neighbors.swap(Some(Arc::new(Neighbors {
-			ring,
-			neighbors: neighbors.clone(),
-		})));
-		neighbors
-	}
-}
-
-impl TableReplication for TableFullReplication {
-	// Full replication schema: all nodes store everything
-	// Writes are disseminated in an epidemic manner in the network
-
-	// Advantage: do all reads locally, extremely fast
-	// Inconvenient: only suitable to reasonably small tables
-
-	fn read_nodes(&self, _hash: &Hash, system: &System) -> Vec<UUID> {
-		vec![system.id]
-	}
-	fn read_quorum(&self) -> usize {
-		1
-	}
-
-	fn write_nodes(&self, _hash: &Hash, system: &System) -> Vec<UUID> {
-		self.get_neighbors(system)
-	}
-	fn write_quorum(&self) -> usize {
-		self.write_quorum
-	}
-	fn max_write_errors(&self) -> usize {
-		self.write_factor - self.write_quorum
-	}
-	fn epidemic_writes(&self) -> bool {
-		true
-	}
-
-	fn replication_nodes(&self, _hash: &Hash, ring: &Ring) -> Vec<UUID> {
-		ring.config.members.keys().cloned().collect::<Vec<_>>()
-	}
-	fn split_points(&self, _ring: &Ring) -> Vec<Hash> {
-		let mut ret = vec![];
-		ret.push([0u8; 32].into());
-		ret.push([0xFFu8; 32].into());
-		ret
-	}
-}
@@ -1,792 +0,0 @@
-use rand::Rng;
-use std::collections::{BTreeMap, VecDeque};
-use std::sync::Arc;
-use std::time::{Duration, Instant};
-
-use futures::future::BoxFuture;
-use futures::{pin_mut, select};
-use futures_util::future::*;
-use futures_util::stream::*;
-use serde::{Deserialize, Serialize};
-use serde_bytes::ByteBuf;
-use tokio::sync::Mutex;
-use tokio::sync::{mpsc, watch};
-
-use garage_rpc::membership::Ring;
-use garage_util::data::*;
-use garage_util::error::Error;
-
-use crate::*;
-
-const MAX_DEPTH: usize = 16;
-const SCAN_INTERVAL: Duration = Duration::from_secs(3600);
-const CHECKSUM_CACHE_TIMEOUT: Duration = Duration::from_secs(1800);
-const TABLE_SYNC_RPC_TIMEOUT: Duration = Duration::from_secs(30);
-
-pub struct TableSyncer<F: TableSchema, R: TableReplication> {
-	table: Arc<Table<F, R>>,
-	todo: Mutex<SyncTodo>,
-	cache: Vec<Mutex<BTreeMap<SyncRange, RangeChecksumCache>>>,
-}
-
-#[derive(Serialize, Deserialize)]
-pub enum SyncRPC {
-	GetRootChecksumRange(Hash, Hash),
-	RootChecksumRange(SyncRange),
-	Checksums(Vec<RangeChecksum>, bool),
-	Difference(Vec<SyncRange>, Vec<Arc<ByteBuf>>),
-}
-
-pub struct SyncTodo {
-	todo: Vec<TodoPartition>,
-}
-
-#[derive(Debug, Clone)]
-struct TodoPartition {
-	begin: Hash,
-	end: Hash,
-	retain: bool,
-}
-
-// A SyncRange defines a query on the dataset stored by a node, in the following way:
-// - all items whose key are >= `begin`
-// - stopping at the first item whose key hash has at least `level` leading zero bytes (excluded)
-// - except if the first item of the range has such many leading zero bytes
-// - and stopping at `end` (excluded) if such an item is not found
-// The checksum itself does not store all of the items in the database, only the hashes of the "sub-ranges"
-// i.e. of ranges of level `level-1` that cover the same range
-// (ranges of level 0 do not exist and their hash is simply the hash of the first item >= begin)
-// See RangeChecksum for the struct that stores this information.
-#[derive(Hash, PartialEq, Eq, Debug, Clone, Serialize, Deserialize)]
-pub struct SyncRange {
-	begin: Vec<u8>,
-	end: Vec<u8>,
-	level: usize,
-}
-
-impl std::cmp::PartialOrd for SyncRange {
-	fn partial_cmp(&self, other: &Self) -> Option<std::cmp::Ordering> {
-		Some(self.cmp(other))
-	}
-}
-impl std::cmp::Ord for SyncRange {
-	fn cmp(&self, other: &Self) -> std::cmp::Ordering {
-		self.begin
-			.cmp(&other.begin)
-			.then(self.level.cmp(&other.level))
-			.then(self.end.cmp(&other.end))
-	}
-}
-
-#[derive(Debug, Clone, Serialize, Deserialize)]
-pub struct RangeChecksum {
-	bounds: SyncRange,
-	children: Vec<(SyncRange, Hash)>,
-	found_limit: Option<Vec<u8>>,
-
-	#[serde(skip, default = "std::time::Instant::now")]
-	time: Instant,
-}
-
-#[derive(Debug, Clone)]
-pub struct RangeChecksumCache {
-	hash: Option<Hash>, // None if no children
-	found_limit: Option<Vec<u8>>,
-	time: Instant,
-}
-
-impl<F, R> TableSyncer<F, R>
-where
-	F: TableSchema + 'static,
-	R: TableReplication + 'static,
-{
-	pub async fn launch(table: Arc<Table<F, R>>) -> Arc<Self> {
-		let todo = SyncTodo { todo: Vec::new() };
-		let syncer = Arc::new(TableSyncer {
-			table: table.clone(),
-			todo: Mutex::new(todo),
-			cache: (0..MAX_DEPTH)
-				.map(|_| Mutex::new(BTreeMap::new()))
-				.collect::<Vec<_>>(),
-		});
-
-		let (busy_tx, busy_rx) = mpsc::unbounded_channel();
-
-		let s1 = syncer.clone();
-		table
-			.system
-			.background
-			.spawn_worker(
-				format!("table sync watcher for {}", table.name),
-				move |must_exit: watch::Receiver<bool>| s1.watcher_task(must_exit, busy_rx),
-			)
-			.await;
-
-		let s2 = syncer.clone();
-		table
-			.system
-			.background
-			.spawn_worker(
-				format!("table syncer for {}", table.name),
-				move |must_exit: watch::Receiver<bool>| s2.syncer_task(must_exit, busy_tx),
-			)
-			.await;
-
-		let s3 = syncer.clone();
-		tokio::spawn(async move {
-			tokio::time::delay_for(Duration::from_secs(20)).await;
-			s3.add_full_scan().await;
-		});
-
-		syncer
-	}
-
-	async fn watcher_task(
-		self: Arc<Self>,
-		mut must_exit: watch::Receiver<bool>,
-		mut busy_rx: mpsc::UnboundedReceiver<bool>,
-	) -> Result<(), Error> {
-		let mut prev_ring: Arc<Ring> = self.table.system.ring.borrow().clone();
-		let mut ring_recv: watch::Receiver<Arc<Ring>> = self.table.system.ring.clone();
-		let mut nothing_to_do_since = Some(Instant::now());
-
-		while !*must_exit.borrow() {
-			let s_ring_recv = ring_recv.recv().fuse();
-			let s_busy = busy_rx.recv().fuse();
-			let s_must_exit = must_exit.recv().fuse();
-			let s_timeout = tokio::time::delay_for(Duration::from_secs(1)).fuse();
-			pin_mut!(s_ring_recv, s_busy, s_must_exit, s_timeout);
-
-			select! {
-				new_ring_r = s_ring_recv => {
-					if let Some(new_ring) = new_ring_r {
-						debug!("({}) Adding ring difference to syncer todo list", self.table.name);
-						self.todo.lock().await.add_ring_difference(&self.table, &prev_ring, &new_ring);
-						prev_ring = new_ring;
-					}
-				}
-				busy_opt = s_busy => {
-					if let Some(busy) = busy_opt {
-						if busy {
-							nothing_to_do_since = None;
-						} else {
-							if nothing_to_do_since.is_none() {
-								nothing_to_do_since = Some(Instant::now());
-							}
-						}
-					}
-				}
-				must_exit_v = s_must_exit => {
-					if must_exit_v.unwrap_or(false) {
-						break;
-					}
-				}
-				_ = s_timeout => {
-					if nothing_to_do_since.map(|t| Instant::now() - t >= SCAN_INTERVAL).unwrap_or(false) {
-						nothing_to_do_since = None;
-						debug!("({}) Adding full scan to syncer todo list", self.table.name);
-						self.add_full_scan().await;
-					}
-				}
-			}
-		}
-		Ok(())
-	}
-
-	pub async fn add_full_scan(&self) {
-		self.todo.lock().await.add_full_scan(&self.table);
-	}
-
-	async fn syncer_task(
-		self: Arc<Self>,
-		mut must_exit: watch::Receiver<bool>,
-		busy_tx: mpsc::UnboundedSender<bool>,
-	) -> Result<(), Error> {
-		while !*must_exit.borrow() {
-			if let Some(partition) = self.todo.lock().await.pop_task() {
-				busy_tx.send(true)?;
-				let res = self
-					.clone()
-					.sync_partition(&partition, &mut must_exit)
-					.await;
-				if let Err(e) = res {
-					warn!(
-						"({}) Error while syncing {:?}: {}",
-						self.table.name, partition, e
-					);
-				}
-			} else {
-				busy_tx.send(false)?;
-				tokio::time::delay_for(Duration::from_secs(1)).await;
-			}
-		}
-		Ok(())
-	}
-
-	async fn sync_partition(
-		self: Arc<Self>,
-		partition: &TodoPartition,
-		must_exit: &mut watch::Receiver<bool>,
-	) -> Result<(), Error> {
-		let my_id = self.table.system.id;
-		let nodes = self
-			.table
-			.replication
-			.write_nodes(&partition.begin, &self.table.system)
-			.into_iter()
-			.filter(|node| *node != my_id)
-			.collect::<Vec<_>>();
-
-		debug!(
-			"({}) Preparing to sync {:?} with {:?}...",
-			self.table.name, partition, nodes
-		);
-		let root_cks = self
-			.root_checksum(&partition.begin, &partition.end, must_exit)
-			.await?;
-
-		let mut sync_futures = nodes
-			.iter()
-			.map(|node| {
-				self.clone().do_sync_with(
-					partition.clone(),
-					root_cks.clone(),
-					*node,
-					partition.retain,
-					must_exit.clone(),
-				)
-			})
-			.collect::<FuturesUnordered<_>>();
-
-		let mut n_errors = 0;
-		while let Some(r) = sync_futures.next().await {
-			if let Err(e) = r {
-				n_errors += 1;
-				warn!("({}) Sync error: {}", self.table.name, e);
-			}
-		}
-		if n_errors > self.table.replication.max_write_errors() {
-			return Err(Error::Message(format!(
-				"Sync failed with too many nodes (should have been: {:?}).",
-				nodes
-			)));
-		}
-
-		if !partition.retain {
-			self.table
-				.delete_range(&partition.begin, &partition.end)
-				.await?;
-		}
-
-		Ok(())
-	}
-
-	async fn root_checksum(
-		self: &Arc<Self>,
-		begin: &Hash,
-		end: &Hash,
-		must_exit: &mut watch::Receiver<bool>,
-	) -> Result<RangeChecksum, Error> {
-		for i in 1..MAX_DEPTH {
-			let rc = self
-				.range_checksum(
-					&SyncRange {
-						begin: begin.to_vec(),
-						end: end.to_vec(),
-						level: i,
-					},
-					must_exit,
-				)
-				.await?;
-			if rc.found_limit.is_none() {
-				return Ok(rc);
-			}
-		}
-		Err(Error::Message(format!(
-			"Unable to compute root checksum (this should never happen)"
-		)))
-	}
-
-	async fn range_checksum(
-		self: &Arc<Self>,
-		range: &SyncRange,
-		must_exit: &mut watch::Receiver<bool>,
-	) -> Result<RangeChecksum, Error> {
-		assert!(range.level != 0);
-
-		if range.level == 1 {
-			let mut children = vec![];
-			for item in self
-				.table
-				.store
-				.range(range.begin.clone()..range.end.clone())
-			{
-				let (key, value) = item?;
-				let key_hash = hash(&key[..]);
-				if children.len() > 0
-					&& key_hash.as_slice()[0..range.level]
-						.iter()
-						.all(|x| *x == 0u8)
-				{
-					return Ok(RangeChecksum {
-						bounds: range.clone(),
-						children,
-						found_limit: Some(key.to_vec()),
-						time: Instant::now(),
-					});
-				}
-				let item_range = SyncRange {
-					begin: key.to_vec(),
-					end: vec![],
-					level: 0,
-				};
-				children.push((item_range, hash(&value[..])));
-			}
-			Ok(RangeChecksum {
-				bounds: range.clone(),
-				children,
-				found_limit: None,
-				time: Instant::now(),
-			})
-		} else {
-			let mut children = vec![];
-			let mut sub_range = SyncRange {
-				begin: range.begin.clone(),
-				end: range.end.clone(),
-				level: range.level - 1,
-			};
-			let mut time = Instant::now();
-			while !*must_exit.borrow() {
-				let sub_ck = self
-					.range_checksum_cached_hash(&sub_range, must_exit)
-					.await?;
-
-				if let Some(hash) = sub_ck.hash {
-					children.push((sub_range.clone(), hash));
-					if sub_ck.time < time {
-						time = sub_ck.time;
-					}
-				}
-
-				if sub_ck.found_limit.is_none() || sub_ck.hash.is_none() {
-					return Ok(RangeChecksum {
-						bounds: range.clone(),
-						children,
-						found_limit: None,
-						time,
-					});
-				}
-				let found_limit = sub_ck.found_limit.unwrap();
-
-				let actual_limit_hash = hash(&found_limit[..]);
-				if actual_limit_hash.as_slice()[0..range.level]
-					.iter()
-					.all(|x| *x == 0u8)
-				{
-					return Ok(RangeChecksum {
-						bounds: range.clone(),
-						children,
-						found_limit: Some(found_limit.clone()),
-						time,
-					});
-				}
-
-				sub_range.begin = found_limit;
-			}
-			Err(Error::Message(format!("Exiting.")))
-		}
-	}
-
-	fn range_checksum_cached_hash<'a>(
-		self: &'a Arc<Self>,
-		range: &'a SyncRange,
-		must_exit: &'a mut watch::Receiver<bool>,
-	) -> BoxFuture<'a, Result<RangeChecksumCache, Error>> {
-		async move {
-			let mut cache = self.cache[range.level].lock().await;
-			if let Some(v) = cache.get(&range) {
-				if Instant::now() - v.time < CHECKSUM_CACHE_TIMEOUT {
-					return Ok(v.clone());
-				}
-			}
-			cache.remove(&range);
-			drop(cache);
-
-			let v = self.range_checksum(&range, must_exit).await?;
-			trace!(
-				"({}) New checksum calculated for {}-{}/{}, {} children",
-				self.table.name,
-				hex::encode(&range.begin)
-					.chars()
-					.take(16)
-					.collect::<String>(),
-				hex::encode(&range.end).chars().take(16).collect::<String>(),
-				range.level,
-				v.children.len()
-			);
-
-			let hash = if v.children.len() > 0 {
-				Some(hash(&rmp_to_vec_all_named(&v)?[..]))
-			} else {
-				None
-			};
-			let cache_entry = RangeChecksumCache {
-				hash,
-				found_limit: v.found_limit,
-				time: v.time,
-			};
-
-			let mut cache = self.cache[range.level].lock().await;
-			cache.insert(range.clone(), cache_entry.clone());
-			Ok(cache_entry)
-		}
-		.boxed()
-	}
-
-	async fn do_sync_with(
-		self: Arc<Self>,
-		partition: TodoPartition,
-		root_ck: RangeChecksum,
-		who: UUID,
-		retain: bool,
-		mut must_exit: watch::Receiver<bool>,
-	) -> Result<(), Error> {
-		let mut todo = VecDeque::new();
-
-		// If their root checksum has level > than us, use that as a reference
-		let root_cks_resp = self
-			.table
-			.rpc_client
-			.call(
-				who,
-				TableRPC::<F>::SyncRPC(SyncRPC::GetRootChecksumRange(
-					partition.begin.clone(),
-					partition.end.clone(),
-				)),
-				TABLE_SYNC_RPC_TIMEOUT,
-			)
-			.await?;
-		if let TableRPC::<F>::SyncRPC(SyncRPC::RootChecksumRange(range)) = root_cks_resp {
-			if range.level > root_ck.bounds.level {
-				let their_root_range_ck = self.range_checksum(&range, &mut must_exit).await?;
-				todo.push_back(their_root_range_ck);
-			} else {
-				todo.push_back(root_ck);
-			}
-		} else {
-			return Err(Error::Message(format!(
-				"Invalid respone to GetRootChecksumRange RPC: {}",
-				debug_serialize(root_cks_resp)
-			)));
-		}
-
-		while !todo.is_empty() && !*must_exit.borrow() {
-			let total_children = todo.iter().map(|x| x.children.len()).fold(0, |x, y| x + y);
-			trace!(
-				"({}) Sync with {:?}: {} ({}) remaining",
-				self.table.name,
-				who,
-				todo.len(),
-				total_children
-			);
-
-			let step_size = std::cmp::min(16, todo.len());
-			let step = todo.drain(..step_size).collect::<Vec<_>>();
-
-			let rpc_resp = self
-				.table
-				.rpc_client
-				.call(
-					who,
-					TableRPC::<F>::SyncRPC(SyncRPC::Checksums(step, retain)),
-					TABLE_SYNC_RPC_TIMEOUT,
-				)
-				.await?;
-			if let TableRPC::<F>::SyncRPC(SyncRPC::Difference(mut diff_ranges, diff_items)) =
-				rpc_resp
-			{
-				if diff_ranges.len() > 0 || diff_items.len() > 0 {
-					info!(
-						"({}) Sync with {:?}: difference {} ranges, {} items",
-						self.table.name,
-						who,
-						diff_ranges.len(),
-						diff_items.len()
-					);
-				}
-				let mut items_to_send = vec![];
-				for differing in diff_ranges.drain(..) {
-					if differing.level == 0 {
-						items_to_send.push(differing.begin);
-					} else {
-						let checksum = self.range_checksum(&differing, &mut must_exit).await?;
-						todo.push_back(checksum);
-					}
-				}
-				if retain && diff_items.len() > 0 {
-					self.table.handle_update(&diff_items[..]).await?;
-				}
-				if items_to_send.len() > 0 {
-					self.send_items(who, items_to_send).await?;
-				}
-			} else {
-				return Err(Error::Message(format!(
-					"Unexpected response to sync RPC checksums: {}",
-					debug_serialize(&rpc_resp)
-				)));
-			}
-		}
-		Ok(())
-	}
-
-	async fn send_items(&self, who: UUID, item_list: Vec<Vec<u8>>) -> Result<(), Error> {
-		info!(
-			"({}) Sending {} items to {:?}",
-			self.table.name,
-			item_list.len(),
-			who
-		);
-
-		let mut values = vec![];
-		for item in item_list.iter() {
-			if let Some(v) = self.table.store.get(&item[..])? {
-				values.push(Arc::new(ByteBuf::from(v.as_ref())));
-			}
-		}
-		let rpc_resp = self
-			.table
-			.rpc_client
-			.call(who, TableRPC::<F>::Update(values), TABLE_SYNC_RPC_TIMEOUT)
-			.await?;
-		if let TableRPC::<F>::Ok = rpc_resp {
-			Ok(())
-		} else {
-			Err(Error::Message(format!(
-				"Unexpected response to RPC Update: {}",
-				debug_serialize(&rpc_resp)
-			)))
-		}
-	}
-
-	pub async fn handle_rpc(
-		self: &Arc<Self>,
-		message: &SyncRPC,
-		mut must_exit: watch::Receiver<bool>,
-	) -> Result<SyncRPC, Error> {
-		match message {
-			SyncRPC::GetRootChecksumRange(begin, end) => {
-				let root_cks = self.root_checksum(&begin, &end, &mut must_exit).await?;
-				Ok(SyncRPC::RootChecksumRange(root_cks.bounds))
-			}
-			SyncRPC::Checksums(checksums, retain) => {
-				self.handle_checksums_rpc(&checksums[..], *retain, &mut must_exit)
-					.await
-			}
-			_ => Err(Error::Message(format!("Unexpected sync RPC"))),
-		}
-	}
-
-	async fn handle_checksums_rpc(
-		self: &Arc<Self>,
-		checksums: &[RangeChecksum],
-		retain: bool,
-		must_exit: &mut watch::Receiver<bool>,
-	) -> Result<SyncRPC, Error> {
-		let mut ret_ranges = vec![];
-		let mut ret_items = vec![];
-
-		for their_ckr in checksums.iter() {
-			let our_ckr = self.range_checksum(&their_ckr.bounds, must_exit).await?;
-			for (their_range, their_hash) in their_ckr.children.iter() {
-				let differs = match our_ckr
-					.children
-					.binary_search_by(|(our_range, _)| our_range.cmp(&their_range))
-				{
-					Err(_) => {
-						if their_range.level >= 1 {
-							let cached_hash = self
-								.range_checksum_cached_hash(&their_range, must_exit)
-								.await?;
-							cached_hash.hash.map(|h| h != *their_hash).unwrap_or(true)
-						} else {
-							true
-						}
-					}
-					Ok(i) => our_ckr.children[i].1 != *their_hash,
-				};
-				if differs {
-					ret_ranges.push(their_range.clone());
-					if retain && their_range.level == 0 {
-						if let Some(item_bytes) =
-							self.table.store.get(their_range.begin.as_slice())?
-						{
-							ret_items.push(Arc::new(ByteBuf::from(item_bytes.to_vec())));
-						}
-					}
-				}
-			}
-			for (our_range, _hash) in our_ckr.children.iter() {
-				if let Some(their_found_limit) = &their_ckr.found_limit {
-					if our_range.begin.as_slice() > their_found_limit.as_slice() {
-						break;
-					}
-				}
-
-				let not_present = our_ckr
-					.children
-					.binary_search_by(|(their_range, _)| their_range.cmp(&our_range))
-					.is_err();
-				if not_present {
-					if our_range.level > 0 {
-						ret_ranges.push(our_range.clone());
-					}
-					if retain && our_range.level == 0 {
-						if let Some(item_bytes) =
-							self.table.store.get(our_range.begin.as_slice())?
-						{
-							ret_items.push(Arc::new(ByteBuf::from(item_bytes.to_vec())));
-						}
-					}
-				}
-			}
-		}
-		let n_checksums = checksums
-			.iter()
-			.map(|x| x.children.len())
-			.fold(0, |x, y| x + y);
-		if ret_ranges.len() > 0 || ret_items.len() > 0 {
-			trace!(
-				"({}) Checksum comparison RPC: {} different + {} items for {} received",
-				self.table.name,
-				ret_ranges.len(),
-				ret_items.len(),
-				n_checksums
-			);
-		}
-		Ok(SyncRPC::Difference(ret_ranges, ret_items))
-	}
-
-	pub async fn invalidate(self: Arc<Self>, item_key: Vec<u8>) -> Result<(), Error> {
-		for i in 1..MAX_DEPTH {
-			let needle = SyncRange {
-				begin: item_key.to_vec(),
-				end: vec![],
-				level: i,
-			};
-			let mut cache = self.cache[i].lock().await;
-			if let Some(cache_entry) = cache.range(..=needle).rev().next() {
-				if cache_entry.0.begin <= item_key && cache_entry.0.end > item_key {
-					let index = cache_entry.0.clone();
-					drop(cache_entry);
-					cache.remove(&index);
-				}
-			}
-		}
-		Ok(())
-	}
-}
-
-impl SyncTodo {
-	fn add_full_scan<F: TableSchema, R: TableReplication>(&mut self, table: &Table<F, R>) {
-		let my_id = table.system.id;
-
-		self.todo.clear();
-
-		let ring = table.system.ring.borrow().clone();
-		let split_points = table.replication.split_points(&ring);
-
-		for i in 0..split_points.len() - 1 {
-			let begin = split_points[i];
-			let end = split_points[i + 1];
-			let nodes = table.replication.replication_nodes(&begin, &ring);
-
-			let retain = nodes.contains(&my_id);
-			if !retain {
-				// Check if we have some data to send, otherwise skip
-				if table.store.range(begin..end).next().is_none() {
-					continue;
-				}
-			}
-
-			self.todo.push(TodoPartition { begin, end, retain });
-		}
-	}
-
-	fn add_ring_difference<F: TableSchema, R: TableReplication>(
-		&mut self,
-		table: &Table<F, R>,
-		old_ring: &Ring,
-		new_ring: &Ring,
-	) {
-		let my_id = table.system.id;
-
-		// If it is us who are entering or leaving the system,
-		// initiate a full sync instead of incremental sync
-		if old_ring.config.members.contains_key(&my_id)
-			!= new_ring.config.members.contains_key(&my_id)
-		{
-			self.add_full_scan(table);
-			return;
-		}
-
-		let mut all_points = None
-			.into_iter()
-			.chain(table.replication.split_points(old_ring).drain(..))
-			.chain(table.replication.split_points(new_ring).drain(..))
-			.chain(self.todo.iter().map(|x| x.begin))
-			.chain(self.todo.iter().map(|x| x.end))
-			.collect::<Vec<_>>();
-		all_points.sort();
-		all_points.dedup();
-
-		let mut old_todo = std::mem::replace(&mut self.todo, vec![]);
-		old_todo.sort_by(|x, y| x.begin.cmp(&y.begin));
-		let mut new_todo = vec![];
-
-		for i in 0..all_points.len() - 1 {
-			let begin = all_points[i];
-			let end = all_points[i + 1];
-			let was_ours = table
-				.replication
-				.replication_nodes(&begin, &old_ring)
-				.contains(&my_id);
-			let is_ours = table
-				.replication
-				.replication_nodes(&begin, &new_ring)
-				.contains(&my_id);
-
-			let was_todo = match old_todo.binary_search_by(|x| x.begin.cmp(&begin)) {
-				Ok(_) => true,
-				Err(j) => {
-					(j > 0 && old_todo[j - 1].begin < end && begin < old_todo[j - 1].end)
-						|| (j < old_todo.len()
-							&& old_todo[j].begin < end && begin < old_todo[j].end)
-				}
-			};
-			if was_todo || (is_ours && !was_ours) || (was_ours && !is_ours) {
-				new_todo.push(TodoPartition {
-					begin,
-					end,
-					retain: is_ours,
-				});
-			}
-		}
-
-		self.todo = new_todo;
-	}
-
-	fn pop_task(&mut self) -> Option<TodoPartition> {
-		if self.todo.is_empty() {
-			return None;
-		}
-
-		let i = rand::thread_rng().gen_range::<usize, _, _>(0, self.todo.len());
-		if i == self.todo.len() - 1 {
-			self.todo.pop()
-		} else {
-			let replacement = self.todo.pop().unwrap();
-			let ret = std::mem::replace(&mut self.todo[i], replacement);
-			Some(ret)
-		}
-	}
-}
@@ -1,9 +1,9 @@
 [package]
 name = "garage_util"
-version = "0.1.1"
+version = "0.2.1"
 authors = ["Alex Auvolat <alex@adnab.me>"]
 edition = "2018"
-license = "GPL-3.0"
+license = "AGPL-3.0"
 description = "Utility crate for the Garage object store"
 repository = "https://git.deuxfleurs.fr/Deuxfleurs/garage"

@@ -13,27 +13,26 @@ path = "lib.rs"
 # See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

 [dependencies]
-rand = "0.7"
-hex = "0.3"
-sha2 = "0.8"
-err-derive = "0.2.3"
+blake2 = "0.9"
+err-derive = "0.3"
+xxhash-rust = { version = "0.8", default-features = false, features = ["xxh3"] }
+hex = "0.4"
 log = "0.4"
+rand = "0.8"
+sha2 = "0.9"

-sled = "0.31"
+sled = "0.34"

-toml = "0.5"
-rmp-serde = "0.14.3"
+chrono = "0.4"
+rmp-serde = "0.15"
 serde = { version = "1.0", default-features = false, features = ["derive", "rc"] }
 serde_json = "1.0"
+toml = "0.5"

 futures = "0.3"
-futures-util = "0.3"
-tokio = { version = "0.2", default-features = false, features = ["rt-core", "rt-threaded", "io-driver", "net", "tcp", "time", "macros", "sync", "signal", "fs"] }
+tokio = { version = "1.0", default-features = false, features = ["rt", "rt-multi-thread", "io-util", "net", "time", "macros", "sync", "signal", "fs"] }

 http = "0.2"
-hyper = "0.13"
-rustls = "0.17"
+hyper = "0.14"
+rustls = "0.19"
 webpki = "0.21"
-
-roxmltree = "0.11"
-
@@ -1,12 +1,11 @@
 use core::future::Future;
 use std::pin::Pin;
-
-use futures::future::join_all;
-use futures::select;
-use futures_util::future::*;
 use std::sync::Arc;
-use tokio::sync::Mutex;
-use tokio::sync::{mpsc, watch, Notify};
+use std::time::Duration;
+
+use futures::future::*;
+use futures::select;
+use tokio::sync::{mpsc, watch, Mutex};

 use crate::error::Error;

@@ -14,54 +13,106 @@ type JobOutput = Result<(), Error>;
 type Job = Pin<Box<dyn Future<Output = JobOutput> + Send>>;

 pub struct BackgroundRunner {
-	n_runners: usize,
 	pub stop_signal: watch::Receiver<bool>,

 	queue_in: mpsc::UnboundedSender<(Job, bool)>,
-	queue_out: Mutex<mpsc::UnboundedReceiver<(Job, bool)>>,
-	job_notify: Notify,
-
-	workers: Mutex<Vec<tokio::task::JoinHandle<()>>>,
+	worker_in: mpsc::UnboundedSender<tokio::task::JoinHandle<()>>,
 }

 impl BackgroundRunner {
-	pub fn new(n_runners: usize, stop_signal: watch::Receiver<bool>) -> Arc<Self> {
+	pub fn new(
+		n_runners: usize,
+		stop_signal: watch::Receiver<bool>,
+	) -> (Arc<Self>, tokio::task::JoinHandle<()>) {
+		let (worker_in, mut worker_out) = mpsc::unbounded_channel();
+
+		let stop_signal_2 = stop_signal.clone();
+		let await_all_done = tokio::spawn(async move {
+			loop {
+				let wkr = {
+					select! {
+						item = worker_out.recv().fuse() => {
+							match item {
+								Some(x) => x,
+								None => break,
+							}
+						}
+						_ = tokio::time::sleep(Duration::from_secs(5)).fuse() => {
+							if *stop_signal_2.borrow() {
+								break;
+							} else {
+								continue;
+							}
+						}
+					}
+				};
+				if let Err(e) = wkr.await {
+					error!("Error while awaiting for worker: {}", e);
+				}
+			}
+		});
+
 		let (queue_in, queue_out) = mpsc::unbounded_channel();
-		Arc::new(Self {
-			n_runners,
+		let queue_out = Arc::new(Mutex::new(queue_out));
+
+		for i in 0..n_runners {
+			let queue_out = queue_out.clone();
+			let stop_signal = stop_signal.clone();
+
+			worker_in
+				.send(tokio::spawn(async move {
+					loop {
+						let (job, cancellable) = {
+							select! {
+								item = wait_job(&queue_out).fuse() => match item {
+									// We received a task, process it
+									Some(x) => x,
+									// We received a signal that no more tasks will ever be sent
+									// because the sending side was dropped. Exit now.
+									None => break,
+								},
+								_ = tokio::time::sleep(Duration::from_secs(5)).fuse() => {
+									if *stop_signal.borrow() {
+										// Nothing has been going on for 5 secs, and we are shutting
+										// down. Exit now.
+										break;
+									} else {
+										// Nothing is going on but we don't want to exit.
+										continue;
+									}
+								}
+							}
+						};
+						if cancellable && *stop_signal.borrow() {
+							continue;
+						}
+						if let Err(e) = job.await {
+							error!("Job failed: {}", e)
+						}
+					}
+					info!("Background worker {} exiting", i);
+				}))
+				.unwrap();
+		}
+
+		let bgrunner = Arc::new(Self {
 			stop_signal,
 			queue_in,
-			queue_out: Mutex::new(queue_out),
-			job_notify: Notify::new(),
-			workers: Mutex::new(Vec::new()),
-		})
-	}
-
-	pub async fn run(self: Arc<Self>) {
-		let mut workers = self.workers.lock().await;
-		for i in 0..self.n_runners {
-			workers.push(tokio::spawn(self.clone().runner(i)));
-		}
-		drop(workers);
-
-		let mut stop_signal = self.stop_signal.clone();
-		while let Some(exit_now) = stop_signal.recv().await {
-			if exit_now {
-				let mut workers = self.workers.lock().await;
-				let workers_vec = workers.drain(..).collect::<Vec<_>>();
-				join_all(workers_vec).await;
-				return;
-			}
-		}
+			worker_in,
+		});
+		(bgrunner, await_all_done)
 	}

+	// Spawn a task to be run in background
 	pub fn spawn<T>(&self, job: T)
 	where
 		T: Future<Output = JobOutput> + Send + 'static,
 	{
 		let boxed: Job = Box::pin(job);
-		let _: Result<_, _> = self.queue_in.clone().send((boxed, false));
-		self.job_notify.notify();
+		self.queue_in
+			.send((boxed, false))
+			.map_err(|_| "could not put job in queue")
+			.unwrap();
 	}

 	pub fn spawn_cancellable<T>(&self, job: T)
@@ -69,56 +120,30 @@ impl BackgroundRunner {
 		T: Future<Output = JobOutput> + Send + 'static,
 	{
 		let boxed: Job = Box::pin(job);
-		let _: Result<_, _> = self.queue_in.clone().send((boxed, true));
-		self.job_notify.notify();
+		self.queue_in
+			.send((boxed, true))
+			.map_err(|_| "could not put job in queue")
+			.unwrap();
 	}

-	pub async fn spawn_worker<F, T>(&self, name: String, worker: F)
+	pub fn spawn_worker<F, T>(&self, name: String, worker: F)
 	where
 		F: FnOnce(watch::Receiver<bool>) -> T + Send + 'static,
-		T: Future<Output = JobOutput> + Send + 'static,
+		T: Future<Output = ()> + Send + 'static,
 	{
-		let mut workers = self.workers.lock().await;
 		let stop_signal = self.stop_signal.clone();
-		workers.push(tokio::spawn(async move {
-			if let Err(e) = worker(stop_signal).await {
-				error!("Worker stopped with error: {}, error: {}", name, e);
-			} else {
-				info!("Worker exited successfully: {}", name);
-			}
-		}));
-	}
-
-	async fn runner(self: Arc<Self>, i: usize) {
-		let mut stop_signal = self.stop_signal.clone();
-		loop {
-			let must_exit: bool = *stop_signal.borrow();
-			if let Some(job) = self.dequeue_job(must_exit).await {
-				if let Err(e) = job.await {
-					error!("Job failed: {}", e)
-				}
-			} else {
-				if must_exit {
-					info!("Background runner {} exiting", i);
-					return;
-				}
-				select! {
-					_ = self.job_notify.notified().fuse() => (),
-					_ = stop_signal.recv().fuse() => (),
-				}
-			}
-		}
-	}
-
-	async fn dequeue_job(&self, must_exit: bool) -> Option<Job> {
-		let mut queue = self.queue_out.lock().await;
-		while let Ok((job, cancellable)) = queue.try_recv() {
-			if cancellable && must_exit {
-				continue;
-			} else {
-				return Some(job);
-			}
-		}
-		None
+		let task = tokio::spawn(async move {
+			info!("Worker started: {}", name);
+			worker(stop_signal).await;
+			info!("Worker exited: {}", name);
+		});
+		self.worker_in
+			.send(task)
+			.map_err(|_| "could not put job in queue")
+			.unwrap();
 	}
 }
+
+async fn wait_job(q: &Mutex<mpsc::UnboundedReceiver<(Job, bool)>>) -> Option<(Job, bool)> {
+	q.lock().await.recv().await
+}
@@ -2,7 +2,7 @@ use std::io::Read;
 use std::net::SocketAddr;
 use std::path::PathBuf;

-use serde::Deserialize;
+use serde::{de, Deserialize};

 use crate::error::Error;

@@ -13,6 +13,7 @@ pub struct Config {

 	pub rpc_bind_addr: SocketAddr,

+	#[serde(deserialize_with = "deserialize_vec_addr")]
 	pub bootstrap_peers: Vec<SocketAddr>,
 	pub consul_host: Option<String>,
 	pub consul_service_name: Option<String>,
@@ -23,12 +24,12 @@ pub struct Config {
 	#[serde(default = "default_block_size")]
 	pub block_size: usize,

+	#[serde(default = "default_control_write_max_faults")]
+	pub control_write_max_faults: usize,
+
 	#[serde(default = "default_replication_factor")]
 	pub meta_replication_factor: usize,

-	#[serde(default = "default_epidemic_fanout")]
-	pub meta_epidemic_fanout: usize,
-
 	#[serde(default = "default_replication_factor")]
 	pub data_replication_factor: usize,

@@ -68,8 +69,8 @@ fn default_block_size() -> usize {
 fn default_replication_factor() -> usize {
 	3
 }
-fn default_epidemic_fanout() -> usize {
-	3
+fn default_control_write_max_faults() -> usize {
+	1
 }

 pub fn read_config(config_file: PathBuf) -> Result<Config, Error> {
@@ -82,3 +83,28 @@ pub fn read_config(config_file: PathBuf) -> Result<Config, Error> {

 	Ok(toml::from_str(&config)?)
 }
+
+fn deserialize_vec_addr<'de, D>(deserializer: D) -> Result<Vec<SocketAddr>, D::Error>
+where
+	D: de::Deserializer<'de>,
+{
+	use std::net::ToSocketAddrs;
+
+	Ok(<Vec<&str>>::deserialize(deserializer)?
+		.iter()
+		.filter_map(|&name| {
+			name.to_socket_addrs()
+				.map(|iter| (name, iter))
+				.map_err(|_| warn!("Error resolving \"{}\"", name))
+				.ok()
+		})
+		.map(|(name, iter)| {
+			let v = iter.collect::<Vec<_>>();
+			if v.is_empty() {
+				warn!("Error resolving \"{}\"", name)
+			}
+			v
+		})
+		.flatten()
+		.collect())
+}
@@ -1,9 +1,7 @@
 use rand::Rng;
 use serde::de::{self, Visitor};
 use serde::{Deserialize, Deserializer, Serialize, Serializer};
-use sha2::{Digest, Sha256};
 use std::fmt;
-use std::time::{SystemTime, UNIX_EPOCH};

 #[derive(Default, PartialOrd, Ord, Clone, Hash, PartialEq, Copy)]
 pub struct FixedBytes32([u8; 32]);
@@ -72,30 +70,53 @@ impl FixedBytes32 {
 	pub fn to_vec(&self) -> Vec<u8> {
 		self.0.to_vec()
 	}
+	pub fn try_from(by: &[u8]) -> Option<Self> {
+		if by.len() != 32 {
+			return None;
+		}
+		let mut ret = [0u8; 32];
+		ret.copy_from_slice(by);
+		Some(Self(ret))
+	}
 }

 pub type UUID = FixedBytes32;
 pub type Hash = FixedBytes32;

-pub fn hash(data: &[u8]) -> Hash {
+pub fn sha256sum(data: &[u8]) -> Hash {
+	use sha2::{Digest, Sha256};
+
 	let mut hasher = Sha256::new();
-	hasher.input(data);
+	hasher.update(data);
 	let mut hash = [0u8; 32];
-	hash.copy_from_slice(&hasher.result()[..]);
+	hash.copy_from_slice(&hasher.finalize()[..]);
 	hash.into()
 }

+pub fn blake2sum(data: &[u8]) -> Hash {
+	use blake2::{Blake2b, Digest};
+
+	let mut hasher = Blake2b::new();
+	hasher.update(data);
+	let mut hash = [0u8; 32];
+	hash.copy_from_slice(&hasher.finalize()[..32]);
+	hash.into()
+}
+
+pub type FastHash = u64;
+
+pub fn fasthash(data: &[u8]) -> FastHash {
+	use xxhash_rust::xxh3::Xxh3;
+
+	let mut h = Xxh3::new();
+	h.update(data);
+	h.digest()
+}
+
 pub fn gen_uuid() -> UUID {
 	rand::thread_rng().gen::<[u8; 32]>().into()
 }

-pub fn now_msec() -> u64 {
-	SystemTime::now()
-		.duration_since(UNIX_EPOCH)
-		.expect("Fix your clock :o")
-		.as_millis() as u64
-}
-
 // RMP serialization with names of fields and variants

 pub fn rmp_to_vec_all_named<T>(val: &T) -> Result<Vec<u8>, rmp_serde::encode::Error>
@@ -8,16 +8,22 @@ use crate::data::*;
 pub enum RPCError {
 	#[error(display = "Node is down: {:?}.", _0)]
 	NodeDown(UUID),
+
 	#[error(display = "Timeout: {}", _0)]
-	Timeout(#[error(source)] tokio::time::Elapsed),
+	Timeout(#[error(source)] tokio::time::error::Elapsed),
+
 	#[error(display = "HTTP error: {}", _0)]
 	HTTP(#[error(source)] http::Error),
+
 	#[error(display = "Hyper error: {}", _0)]
 	Hyper(#[error(source)] hyper::Error),
+
 	#[error(display = "Messagepack encode error: {}", _0)]
 	RMPEncode(#[error(source)] rmp_serde::encode::Error),
+
 	#[error(display = "Messagepack decode error: {}", _0)]
 	RMPDecode(#[error(source)] rmp_serde::decode::Error),
+
 	#[error(display = "Too many errors: {:?}", _0)]
 	TooManyErrors(Vec<String>),
 }
@@ -73,11 +79,11 @@ pub enum Error {
 	Message(String),
 }

-impl From<sled::TransactionError<Error>> for Error {
-	fn from(e: sled::TransactionError<Error>) -> Error {
+impl From<sled::transaction::TransactionError<Error>> for Error {
+	fn from(e: sled::transaction::TransactionError<Error>) -> Error {
 		match e {
-			sled::TransactionError::Abort(x) => x,
-			sled::TransactionError::Storage(x) => Error::Sled(x),
+			sled::transaction::TransactionError::Abort(x) => x,
+			sled::transaction::TransactionError::Storage(x) => Error::Sled(x),
 		}
 	}
 }
@@ -5,3 +5,4 @@ pub mod background;
 pub mod config;
 pub mod data;
 pub mod error;
+pub mod time;
@@ -0,0 +1,16 @@
+use chrono::{SecondsFormat, TimeZone, Utc};
+use std::time::{SystemTime, UNIX_EPOCH};
+
+pub fn now_msec() -> u64 {
+	SystemTime::now()
+		.duration_since(UNIX_EPOCH)
+		.expect("Fix your clock :o")
+		.as_millis() as u64
+}
+
+pub fn msec_to_rfc3339(msecs: u64) -> String {
+	let secs = msecs as i64 / 1000;
+	let nanos = (msecs as i64 % 1000) as u32 * 1_000_000;
+	let timestamp = Utc.timestamp(secs, nanos);
+	timestamp.to_rfc3339_opts(SecondsFormat::Secs, true)
+}
@@ -1,9 +1,9 @@
 [package]
 name = "garage_web"
-version = "0.1.1"
+version = "0.2.1"
 authors = ["Alex Auvolat <alex@adnab.me>", "Quentin Dufour <quentin@dufour.io>"]
 edition = "2018"
-license = "GPL-3.0"
+license = "AGPL-3.0"
 description = "S3-like website endpoint crate for the Garage object store"
 repository = "https://git.deuxfleurs.fr/Deuxfleurs/garage"

@@ -13,16 +13,17 @@ path = "lib.rs"
 # See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

 [dependencies]
-garage_util = { version = "0.1.1", path = "../util" }
-garage_table = { version = "0.1.1", path = "../table" }
-garage_model = { version = "0.1.1", path = "../model" }
-garage_api = { version = "0.1.1", path = "../api" }
+garage_api = { version = "0.2.1", path = "../api" }
+garage_model = { version = "0.2.1", path = "../model" }
+garage_util = { version = "0.2.1", path = "../util" }
+garage_table = { version = "0.2.1", path = "../table" }

-err-derive = "0.2.3"
-log = "0.4"
-futures = "0.3"
-http = "0.2"
-hyper = "0.13"
-percent-encoding = "2.1.0"
-roxmltree = "0.11"
+err-derive = "0.3"
 idna = "0.2"
+log = "0.4"
+percent-encoding = "2.1.0"
+
+futures = "0.3"
+
+http = "0.2"
+hyper = "0.14"
--- a/Show More
+++ b/Show More
Author	SHA1	Message	Date
Alex Auvolat	f859d15062	update to v0.2.1	2021-03-19 13:39:18 +01:00
Alex Auvolat	fd8f4caa81	Support old CPUs	2021-03-19 12:19:40 +01:00
Alex Auvolat	4c26a0b9c1	Update Cargo.toml files with AGPL license info	2021-03-18 21:59:17 +01:00
Alex Auvolat	a1014224d3	garage node configure --replace <old_node_id> <new_node_id>	2021-03-18 21:49:12 +01:00
Alex	e19e267b7e	Merge pull request 'add support for using domain name in configuration' (#46 ) from trinity-1686a/garage:bind-domain into master Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/46	2021-03-18 21:26:34 +01:00
Trinity Pointard	f17cb6c969	resolve domain to multiple addresses And warn instead of failling when a domain can't be resolved	2021-03-18 21:04:30 +01:00
Trinity Pointard	c8a7ce5cdf	remove domain resolution for *_bind_addr	2021-03-18 19:47:51 +01:00
Trinity Pointard	81e9db783f	simplify addresse deserialialiser and limit allocations	2021-03-18 19:47:51 +01:00
Trinity Pointard	ae3b7029a9	add support for using domain name in configuration	2021-03-18 19:47:51 +01:00
Alex Auvolat	6edbc65847	Add trinity's comment in the code	2021-03-18 19:46:43 +01:00
Alex	bfa0ff8f82	Merge pull request 'add support for caching headers' (#49 ) from trinity-1686a/garage:cache-headers into master Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/49	2021-03-18 19:45:02 +01:00
Alex Auvolat	dead945c8f	Prepare for release 0.2	2021-03-18 19:33:15 +01:00
Alex Auvolat	4348bde180	Merge branch 'dev-0.2'	2021-03-18 19:27:02 +01:00
Alex Auvolat	4eb16e8863	Allow to import keys from previous Garage instance	2021-03-18 19:24:59 +01:00
Alex	8e317e2783	Merge pull request 'try fixing CI' (#50 ) from trinity-1686a/garage:try-fix-ci into dev-0.2 Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/50	2021-03-18 19:07:56 +01:00
Trinity Pointard	ec26acb161	try fixing CI maybe absolute path break caching, who knows	2021-03-18 17:45:26 +01:00
Trinity Pointard	b4c903371c	add support for caching headers	2021-03-18 15:46:33 +01:00
Alex	5b659b28ce	Merge pull request 'Add a `mdbook` documentation to present garage and help user on-boarding' (#45 ) from feature/mdbook into master Reviewed-on: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/45	2021-03-18 10:39:55 +01:00
Quentin Dufour	ea21c54434	Add handle files section to the doc	2021-03-17 22:44:35 +01:00
Quentin Dufour	1a5af9d1fc	WIP getting started	2021-03-17 22:06:37 +01:00
Quentin Dufour	b82a61fba2	Simplify our README	2021-03-17 20:58:30 +01:00
Quentin Dufour	44d0815ff9	Wrote daemon	2021-03-17 20:04:27 +01:00
Quentin Dufour	468e45ed7f	WIP doc	2021-03-17 18:01:06 +01:00
Quentin Dufour	60f994a118	Working on the getting started guide	2021-03-17 17:24:11 +01:00
Quentin Dufour	002538f92c	Refactor file organization	2021-03-17 16:15:18 +01:00
Quentin Dufour	c50113acf3	Work on structure + Getting started is reworked	2021-03-17 15:44:29 +01:00
Quentin Dufour	0afc701a69	Doc skeleton + intro	2021-03-17 14:44:14 +01:00
Alex Auvolat	797cda1c33	Add logo in repo	2021-03-17 10:26:50 +01:00
Alex Auvolat	46c7226fe4	New logo!!	2021-03-17 08:41:07 +01:00
Alex Auvolat	390ab02f41	Todo make a test for the Merkle updater	2021-03-16 20:13:07 +01:00
Alex Auvolat	7b10245dfb	Leader-based GC	2021-03-16 18:42:33 +01:00
Alex Auvolat	08bcd51956	GC object table in a specific case	2021-03-16 16:51:15 +01:00
Alex Auvolat	1bfe9c129e	Switch to AGPL	2021-03-16 16:35:46 +01:00
Alex Auvolat	3fadc5cbbd	Small changes	2021-03-16 16:35:10 +01:00
Alex Auvolat	f4346cc5f4	Update dependencies	2021-03-16 15:58:40 +01:00
Alex Auvolat	2a41b82384	Simpler Merkle & sync	2021-03-16 12:18:03 +01:00
Alex Auvolat	0aad2f2e06	some reordering	2021-03-16 11:47:39 +01:00
Alex Auvolat	515029d026	Refactor code	2021-03-16 11:43:58 +01:00
Alex Auvolat	1d9961e411	Simplify replication logic	2021-03-16 11:14:27 +01:00
Alex Auvolat	6a8439fd13	Some improvements in background worker but we terminate late	2021-03-15 23:14:12 +01:00
Alex Auvolat	0cd5b2ae19	WIP migrate to tokio 1	2021-03-15 22:36:41 +01:00
Alex Auvolat	4d4117f2b4	Refactor block resync loop; make workers infaillible	2021-03-15 20:09:44 +01:00
Alex Auvolat	667e4e72a8	Small fixes	2021-03-15 19:51:16 +01:00
Alex Auvolat	642bed601f	Make it case-insensitive	2021-03-15 19:16:42 +01:00
Alex Auvolat	5ee1d956b6	Allow manipulation of keys by their shorthand in the CLI	2021-03-15 19:14:26 +01:00
Alex Auvolat	537f652fec	Tiny things	2021-03-15 18:40:27 +01:00
Alex Auvolat	0290afe1f8	Make block rc code more understandable	2021-03-15 18:27:26 +01:00
Alex Auvolat	3bf2df622a	Time and metadata improvements	2021-03-15 16:21:41 +01:00
Alex Auvolat	097c339d98	Fix race condition	2021-03-15 15:26:29 +01:00
Alex Auvolat	bdcbdd1cd8	Fix list API bug	2021-03-15 14:46:37 +01:00
Alex Auvolat	9b118160a8	Optim & refactor	2021-03-12 22:06:56 +01:00
Alex Auvolat	831eb35763	cargo fmt	2021-03-12 21:52:19 +01:00
Alex Auvolat	c475471e7a	Implement table gc, currently for block_ref and version only	2021-03-12 19:57:37 +01:00
Alex Auvolat	f4aad8fe6e	cargo fmt	2021-03-12 18:16:03 +01:00
Alex Auvolat	5ab33fddac	Refactor CLI and prettify CLI outpu	2021-03-12 18:12:31 +01:00
Alex Auvolat	a1442f072a	Implement garage stats to get info on node contents	2021-03-12 15:40:54 +01:00
Alex Auvolat	cbe7e1a66a	Move table rpc client out of tableaux	2021-03-12 15:07:23 +01:00
Alex Auvolat	8860aa19b8	Make syncer have its own rpc client/server	2021-03-12 15:05:26 +01:00
Alex Auvolat	1fea257291	Don't sync at beginning	2021-03-12 14:51:17 +01:00
Alex Auvolat	7fdaf7aef0	Fix merkle updater not being notified; improved logging	2021-03-12 14:37:46 +01:00
Alex Auvolat	1ec49980ec	whoops	2021-03-11 19:30:24 +01:00
Alex Auvolat	3f7a496355	More security: don't delete stuff too easily	2021-03-11 19:06:27 +01:00
Alex Auvolat	f7c2cd1cd7	Add comment, and also whoops, this wasn't doing what we expected	2021-03-11 18:56:18 +01:00
Alex Auvolat	fae5104a2c	Add a nice warning	2021-03-11 18:50:32 +01:00
Alex Auvolat	db7a9d4948	Tiny changes	2021-03-11 18:45:26 +01:00
Alex Auvolat	046b649bcc	(not well tested) use merkle tree for sync	2021-03-11 18:28:27 +01:00
Alex Auvolat	94f3d28774	WIP big refactoring	2021-03-11 16:54:15 +01:00
Alex Auvolat	8d63738cb0	Checkpoint: add merkle tree in data table	2021-03-11 13:47:21 +01:00
Alex Auvolat	3214dd52dd	Very minor changes	2021-03-10 21:50:09 +01:00
Alex Auvolat	af7600f989	Correctly implement CompleteMultipartUpload with etag check of parts	2021-03-10 17:01:05 +01:00
Alex Auvolat	445912dc6a	Remove migration paths from 0.1 branch	2021-03-10 16:38:31 +01:00
Alex Auvolat	0fd7df8fa0	Switch to blake2 sum for identifying blocks by their data	2021-03-10 16:33:31 +01:00
Alex Auvolat	2afd2c81ba	Change hash function to blake2 for partition keys based on strings	2021-03-10 16:23:57 +01:00
Alex Auvolat	f319a7d374	Refactor model stuff, including cleaner CRDTs	2021-03-10 16:21:56 +01:00
Alex Auvolat	6a3dcf3974	Rename n_tokens into capacity	2021-03-10 14:52:03 +01:00
Alex Auvolat	7cda917b6b	update condition	2021-03-05 17:08:03 +01:00
Alex Auvolat	5e33c3cfc9	Use smaller capacities for nodes	2021-03-05 16:58:34 +01:00
Alex Auvolat	d7e005251d	Not fully tested: new multi-dc MagLev	2021-03-05 16:22:29 +01:00
Alex Auvolat	3882d5ba36	Remove epidemic propagation for fully replicated stuff: write directly to all nodes	2021-03-05 15:09:18 +01:00
Alex Auvolat	d7e148d302	Description of MultiDC MagLev	2021-02-25 11:37:42 +01:00
Alex Auvolat	0522983aec	precisions	2021-02-25 11:27:13 +01:00
Alex Auvolat	fdf908e845	Add precisions	2021-02-25 11:23:22 +01:00
Alex Auvolat	2b4b69938f	Add write-up about load-balancing	2021-02-25 11:18:44 +01:00
Alex Auvolat	49c25a1509	Simulate stuff moving around	2021-02-25 10:53:33 +01:00
Alex Auvolat	5fe95ebae7	fix tracing	2021-02-24 12:18:01 +01:00
Alex Auvolat	13e2eda0c2	Arrange block manager	2021-02-24 11:58:03 +01:00
Alex Auvolat	09fd6ea7f0	I was tired yesterday	2021-02-24 11:05:59 +01:00
Alex Auvolat	a52ab69640	fix misuse of sled transactions	2021-02-23 22:45:36 +01:00
Alex Auvolat	20e6e9fa20	Update sled & try to debug deadlock (but its in sled...)	2021-02-23 21:27:28 +01:00
Alex Auvolat	bf25c95fe2	Make updated() be a sync function that doesn't fail	2021-02-23 20:25:15 +01:00
Alex Auvolat	28bc967c83	Handle correctly deletion dues to offloading	2021-02-23 19:59:43 +01:00
Alex Auvolat	55156cca9d	Several changes in table_sync: - separate path for case of offloading a partition we don't store - use sync::Mutex instead of tokio::Mutex, make less fn's async	2021-02-23 19:11:02 +01:00
Alex Auvolat	40763fd749	Cargo fmt	2021-02-23 18:46:25 +01:00
Alex Auvolat	6e6f7e8555	Replace some checksums where it makes sense	2021-02-23 18:14:37 +01:00
Alex Auvolat	e8e4418ca7	Add blake2 and xxhash hash functions	2021-02-23 17:52:28 +01:00
Alex Auvolat	1abbca37c4	Add adapted version of maglev for multi-dc	2021-02-21 19:14:28 +01:00
Alex Auvolat	24f924afdb	Maglev simulation	2021-02-21 18:32:13 +01:00
Alex Auvolat	b1b640ae8b	rename hash() to sha256sum(), we might want to change it at some places	2021-02-21 15:24:30 +01:00
Alex Auvolat	e59322041a	Evaluate hash functions	2021-02-21 15:11:15 +01:00
Alex Auvolat	217b0dfd68	Add script to simulate different kinds of rings	2021-02-21 14:30:26 +01:00
Alex Auvolat	80892df8cc	Some refactoring	2021-02-21 13:11:10 +01:00
Alex Auvolat	3bcbbe1e31	More precise logging (warn only when returning a 500)	2021-02-20 00:30:39 +01:00
Alex Auvolat	10b983b8e7	Add verification of part numbers in CompleteMultipartUpload (WIP #30 )	2021-02-20 00:13:07 +01:00
Alex Auvolat	1de96248e0	add application/xml header and missing xml escapes	2021-02-19 23:40:18 +01:00
Alex Auvolat	0ddfee92c5	add precision	2021-02-19 19:11:55 +01:00
Alex Auvolat	5d1fa591d9	Add compatibility list	2021-02-19 19:10:23 +01:00
Alex Auvolat	e64ecbdccd	S3 compatibility: return 404 instead of 400 on some multipart commands	2021-02-19 18:51:05 +01:00
Alex Auvolat	55a2a636ca	Implement ListObjectsV2	2021-02-19 16:44:06 +01:00
Alex Auvolat	02d512f3fd	Fix #28 , extra headers being ignored (because of profound stupidity)	2021-02-19 12:38:22 +01:00
Alex Auvolat	76390085ef	Small improvements in the S3 put workflow	2021-02-19 12:11:02 +01:00
Alex Auvolat	3b023c0c3b	try to fix smoke test dev cluster: don't ipv6 (fixes smoke test in container?)	2021-02-17 22:13:11 +01:00
Alex Auvolat	b5dc4d33a9	CI improvements	2021-02-17 22:13:08 +01:00
Alex Auvolat	a3d9d0faac	yea more gzip	2021-02-10 13:11:13 +01:00
Alex Auvolat	bcf727f9c1	oops, missed a step	2021-02-10 12:54:11 +01:00
Alex Auvolat	24766a0eb9	change dockerfile; add gzip caching	2021-02-10 12:53:00 +01:00
Alex Auvolat	75d96bdc58	Add drone caching	2021-02-08 16:32:29 +01:00
Alex Auvolat	b3a27b9ebb	Add drone badge	2021-02-08 15:04:22 +01:00
Alex Auvolat	9a8b387bb0	fix drone	2021-02-08 15:02:03 +01:00
Alex Auvolat	536bbb2277	add drone.yml	2021-02-08 14:58:24 +01:00
Alex Auvolat	f26c795c99	Add warning	2021-01-23 19:15:57 +01:00
Alex Auvolat	6f6bf23bec	Move doc files here from Deuxfleur's site repo	2021-01-23 19:12:10 +01:00