# Durability modes (drive sync tiers) RustFS lets operators choose how much fsync work runs on the object write path. The default (`strict`) preserves the fully synced behavior RustFS has always shipped; the relaxed tiers are **opt-in** trades of power-loss durability for latency/IOPS. ## Configuration ```bash # New tiered switch (wins when set to a valid value) RUSTFS_DURABILITY_MODE=strict|relaxed|none # default: strict # Legacy binary switch (kept for compatibility, superseded by the above) RUSTFS_DRIVE_SYNC_ENABLE=true|false # default: true ``` Resolution rules: | `RUSTFS_DURABILITY_MODE` | `RUSTFS_DRIVE_SYNC_ENABLE` | Effective mode | | --- | --- | --- | | unset | unset | `strict` (default) | | unset | `true` | `strict` | | unset | `false` | `legacy-off` (the historical "everything off" semantics) | | `strict` / `relaxed` / `none` | anything | the named mode | | invalid value | any | logged warning, falls back to the legacy switch, then the default | Values are case-insensitive and whitespace-tolerant. The mode is resolved **once per process** and cached (it also removes the per-call `getenv` the old switch performed a dozen times per PUT); changing the environment requires a restart. The resolved mode is logged at startup under the `disk_local_durability_mode` event. `legacy-off` is not a value of `RUSTFS_DURABILITY_MODE`; it is only reachable through `RUSTFS_DRIVE_SYNC_ENABLE=false` so that existing deployments keep their exact current behavior. It is deprecated and will be retained for at least one major version. ## What each mode fsyncs Write points on the object path and how each mode treats them: | Write point | `strict` | `relaxed` | `none` | `legacy-off` | | --- | --- | --- | --- | --- | | Erasure shard files (fdatasync before the commit rename) | yes | **yes** | no | no | | Multipart part payload (fdatasync before `rename_part` commit) | yes | **yes** | no | no | | xl.meta contents (tmp write before the commit rename) | yes | no | no | no | | Inline objects (data embedded in xl.meta) | yes | no | no | no | | Old-metadata rollback backups | yes | no | no | no | | Directory entries of commit renames (fsync of the parent dir) | yes | no | no | no | | System-critical writes (see pinning below) | yes | yes (pinned) | yes (pinned) | **no** | ## Power-loss guarantees, honestly stated **`strict` (default).** Every acknowledged write (PUT, UploadPart, CompleteMultipartUpload, delete markers, metadata updates) is durable on the individual drive before the 200 OK: payload bytes, xl.meta, rollback backups, and the directory entries of the commit renames are all fsynced. A whole-node (or whole-cluster) power failure does not lose acknowledged data. This is the current mainline behavior, unchanged. **`relaxed`.** Payload bytes of non-inline objects and multipart parts are fdatasynced to the device before the acknowledgement, but the metadata commits — xl.meta contents, rollback backups, and the directory entries of the commit renames — are left to the page cache. Consequences on a power failure: - On the affected drive, a **recently acknowledged version can be lost entirely** — not merely "the directory entry rolls back". When neither the xl.meta bytes nor the rename's directory entry are synced, the commit itself can vanish; surviving shard bytes become unreferenced orphans on that drive. - **Inline (small) objects receive no per-object fsync at all** in this mode: their data lives inside xl.meta, and xl.meta is not synced. This matches MinIO's default posture (no per-object fsync) but means small objects have the widest loss window. - Durability of acknowledged writes therefore rests on **erasure-coded redundancy across other nodes** plus the unclean-shutdown heal introduced in PR #4221 converging the affected drive afterwards. Deployment rule for `relaxed`: only multi-node clusters whose nodes sit in **independent power domains** (separate feeds/UPS). If all nodes can lose power simultaneously — the exact incident class that motivated PR #4221 — `relaxed` can lose recently acknowledged objects cluster-wide. Single-node deployments must stay on `strict`. **`none`.** No fsync on the object data path at all; acknowledged objects can vanish wholesale on power loss, payload included. System-critical writes are still pinned (below). This is the tier equivalent of the old escape hatch, useful for throwaway/benchmark data only. **`legacy-off`.** The historical semantics of `RUSTFS_DRIVE_SYNC_ENABLE=false`, preserved bit for bit for existing deployments: nothing is fsynced anywhere, **including system-critical metadata** such as `format.json`. Prefer `RUSTFS_DURABILITY_MODE=none`, which keeps the system-critical writes safe. ## System-critical pinning Writes that commit into system namespaces are pinned to `strict` regardless of the configured tier (except under `legacy-off`, see above): - `.rustfs.sys` — `format.json`, IAM and cluster configuration, bucket metadata, and everything else outside the scratch namespaces; - `.minio.sys` — the same namespace during MinIO migration. The scratch namespaces `.rustfs.sys/tmp` and `.rustfs.sys/multipart` stage in-flight **user object data** and follow the configured tier — they are exactly the writes the relaxed tiers exist for. Their durability is decided by the destination volume at commit time, so an IAM or bucket-metadata object staged in tmp still commits with full `strict` durability. The durability mode is server-side configuration only; it cannot be raised or lowered by any request header. ## Per-bucket durability (phase 2) A bucket can override the process-wide mode with its own tier. The override is stored in the bucket's metadata (a `durability.json` entry in `.rustfs.sys/buckets//.metadata.bin`, written as a RustFS extension field that MinIO's decoder skips) and resolved per write at the commit points, so no restart is needed. ### Configuration ```bash # Set an override (admin credentials, ConfigUpdateAdminAction) curl -X PUT "http:///rustfs/admin/v3/bucket-durability/" \ -d '{"mode":"relaxed"}' # strict | relaxed | none # Read it back ("mode": null means the bucket inherits the global mode) curl "http:///rustfs/admin/v3/bucket-durability/" # Clear it (the bucket inherits the global mode again) curl -X DELETE "http:///rustfs/admin/v3/bucket-durability/" ``` `mc` integration is a follow-up; for now the admin API above is the configuration plane. ### Resolution order For a write committing into volume `V`: 1. system-critical namespaces (`.rustfs.sys`, `.minio.sys` outside the scratch dirs) are always `strict` — a bucket override can never be attached to them, and any attempt is rejected and logged; 2. otherwise, if the destination bucket has an override, the override wins — in **both** directions (a bucket can be `relaxed` under a `strict` global default, or pinned `strict` under a `relaxed` global default); 3. otherwise the process-wide mode applies. Under `legacy-off` (`RUSTFS_DRIVE_SYNC_ENABLE=false`) per-bucket overrides do not apply at all: the legacy switch keeps its historical semantics bit for bit. `legacy-off` is also not a valid per-bucket tier. Because scratch-staged data commits under the destination volume, an object staged in `.rustfs.sys/tmp` follows the override of the bucket it commits into, exactly like the global tier. ### Propagation and effect latency The override rides the existing bucket-metadata cache, with no invalidation channel of its own: - on the node that applies the config change, the new tier is effective for writes that resolve their durability after the update completes; - other nodes are told to reload the bucket's metadata right away (the same peer notification used for every bucket config change); if a peer misses the notification, the periodic bucket-metadata refresh loop (15 minutes) converges it; - an in-flight operation keeps the tier it resolved at its start — a single commit is never half-old-tier, half-new-tier; - deleting the bucket drops the override with the rest of its metadata. Until a peer has reloaded the metadata, its writes use the previous tier. This is the same eventual-consistency window every other bucket config (policy, quota, versioning) already has. ### Power-loss guarantees Identical to the corresponding global tier, scoped to the bucket. In particular the `relaxed` deployment rule (multi-node clusters in independent power domains only) applies per bucket: overriding a bucket to `relaxed` on a single-node deployment can lose recently acknowledged objects in that bucket on power failure. ## Performance expectations The often-quoted 26x PUT throughput delta was measured on macOS with the old binary switch fully **off** (equivalent to `none`/`legacy-off`), where `F_FULLFSYNC` heavily amplifies sync cost. `relaxed` keeps the per-shard fdatasync, so its gain is necessarily smaller and must be measured on the target platform (Linux ext4/xfs) before being relied on. Do not use `none` numbers to size `relaxed`. ## Scope Phase 1 (rustfs/backlog#926) shipped the global, per-process tier configured by environment variable. Phase 2 (rustfs/backlog#938) adds the per-bucket override described above, configured through the admin API and stored in bucket metadata. `mc admin` integration for the per-bucket tier is a follow-up. ## Related Recovery Guides - [No-parity bitrot recovery](no-parity-bitrot-recovery.md) explains how to diagnose historical no-parity objects whose raw `part.N` files are present but fail bitrot validation.