diff --git a/doc/book/operations/recovering.md b/doc/book/operations/recovering.md index 05322b67..fb20656d 100644 --- a/doc/book/operations/recovering.md +++ b/doc/book/operations/recovering.md @@ -161,4 +161,7 @@ your recovery options are as follows: - **Option 3: restoring a filesystem-level snapshot.** If you are using ZFS or BTRFS to snapshot your metadata partition, refer to their specific - documentation on rolling back or copying files from an old snapshot. + documentation on rolling back or copying files from an old snapshot. + Note that, depending on the properties of the filesystem and of the DB engine, + if these snapshots were taken during a write operation to the database, they may + also be corrupted and thus unfit for recovery. diff --git a/doc/book/reference-manual/configuration.md b/doc/book/reference-manual/configuration.md index 9aad04e5..d26d9253 100644 --- a/doc/book/reference-manual/configuration.md +++ b/doc/book/reference-manual/configuration.md @@ -98,7 +98,7 @@ The following gives details about each available configuration option. Top-level configuration options, in alphabetical order: [`allow_punycode`](#allow_punycode), [`allow_world_readable_secrets`](#allow_world_readable_secrets), -[`block_max_concurrent_reads`](`block_max_concurrent_reads), +[`block_max_concurrent_reads`](#block_max_concurrent_reads), [`block_ram_buffer_max`](#block_ram_buffer_max), [`block_size`](#block_size), [`bootstrap_peers`](#bootstrap_peers), @@ -337,7 +337,7 @@ Since `v0.8.0`, Garage can use alternative storage backends as follows: | --------- | ----------------- | ------------- | | [LMDB](https://www.symas.com/lmdb) (since `v0.8.0`, default since `v0.9.0`) | `"lmdb"` | `/db.lmdb/` | | [Sqlite](https://sqlite.org) (since `v0.8.0`) | `"sqlite"` | `/db.sqlite` | -| [Fjall](https://github.com/fjall-rs/fjall) (**experimental support** since `v1.3.0`) | `"fjall"` | `/db.fjall/` | +| [Fjall](https://github.com/fjall-rs/fjall) (**experimental support** since `v1.3.0`/`v2.1.0`) | `"fjall"` | `/db.fjall/` | | [Sled](https://sled.rs) (old default, removed since `v1.0`) | `"sled"` | `/db/` | Sled was supported until Garage v0.9.x, and was removed in Garage v1.0. @@ -346,8 +346,16 @@ old Sled metadata databases to another engine. Performance characteristics of the different DB engines are as follows: -- LMDB: the recommended database engine for high-performance distributed clusters. -LMDB works very well, but is known to have the following limitations: +- **LMDB:** the recommended database engine for high-performance distributed clusters + with `replication_factor` ≥ 2. + LMDB works well, but is known to have the following limitations: + + - LMDB is prone to database corruption after an unclean shutdown (e.g. a process kill + or a power outage). It is recommended to configure + [`metadata_auto_snapshot_interval`](#metadata_auto_snapshot_interval) to be + able to easily recover from this situation. With `replication_factor` ≥ 2, + metadata can also be reconstructed from remote nodes upon corruption + (see [Recovering from failures](@/documentation/operations/recovering.md#corrupted_meta)). - The data format of LMDB is not portable between architectures, so for instance the Garage database of an x86-64 node cannot be moved to an ARM64 @@ -357,30 +365,21 @@ LMDB works very well, but is known to have the following limitations: node to very small database sizes due to how LMDB works; it is therefore not recommended. - - Several users have reported corrupted LMDB database files after an unclean - shutdown (e.g. a power outage). This situation can generally be recovered - from if your cluster is geo-replicated (by rebuilding your metadata db from - other nodes), or if you have saved regular snapshots at the filesystem - level. - - Keys in LMDB are limited to 511 bytes. This limit translates to limits on object keys in S3 and sort keys in K2V that are limted to 479 bytes. -- Sqlite: Garage supports Sqlite as an alternative storage backend for - metadata, which does not have the issues listed above for LMDB. - On versions 0.8.x and earlier, Sqlite should be avoided due to abysmal - performance, which was fixed with the addition of `metadata_fsync`. - Sqlite is still probably slower than LMDB due to the way we use it, - so it is not the best choice for high-performance storage clusters, - but it should work fine in many cases. +- **Sqlite:** Garage supports Sqlite as an alternative storage backend for + metadata, which does not have the issues listed above for LMDB. Sqlite is + slower than LMDB, so it is not the best choice for high-performance storage + clusters. -- Fjall: a storage engine based on LSM trees, which theoretically allow for +- **Fjall:** a storage engine based on LSM trees, which theoretically allow for higher write throughput than other storage engines that are based on B-trees. Using Fjall could potentially improve Garage's performance significantly in write-heavy workloads. **Support for Fjall is experimental at this point**, - we have added it to Garage for evaluation purposes only. **Do not use it for - production-critical workloads.** - + we have added it to Garage for evaluation purposes only. **Use it only with + test data, and report any issues to our bug tracker. Do not use it for + production workloads.** It is possible to convert Garage's metadata directory from one format to another using the `garage convert-db` command, which should be used as follows: @@ -439,7 +438,8 @@ if geographical replication is used. #### `metadata_auto_snapshot_interval` (since `v0.9.4`) {#metadata_auto_snapshot_interval} If this value is set, Garage will automatically take a snapshot of the metadata -DB file at a regular interval and save it in the metadata directory. +DB file at a regular interval and save it in the metadata directory, +or in [`metadata_snapshots_dir`](#metadata_snapshots_dir) if it is set. This parameter can take any duration string that can be parsed by the [`parse_duration`](https://docs.rs/parse_duration/latest/parse_duration/#syntax) crate. @@ -448,14 +448,19 @@ corrupted, for instance after an unclean shutdown. See [this page](@/documentation/operations/recovering.md#corrupted_meta) for details. Garage keeps only the two most recent snapshots of the metadata DB and deletes older ones automatically. +You can also create metadata snapshots manually at any point using the +`garage meta snapshot` command. + +Using snapshots created by Garage is the best option to make snapshots of your +node's metadata for potential recovery, as they are guaranteed to be clean and +consistent, contrarily to filesystem-level snapshots that may be taken while +some writes are in-flight and thus might be corrupted. Note that taking a metadata snapshot is a relatively intensive operation as the entire data file is copied. A snapshot being taken might have performance impacts on the Garage node while it is running. If the cluster is under heavy write load when a snapshot operation is running, this might also cause the database file to grow in size significantly as pages cannot be recycled easily. -For this reason, it might be better to use filesystem-level snapshots instead -if possible. #### `disable_scrub` {#disable_scrub}