Chris ea596b2e2d fix(decommission): adopt target intents left by a killed leader (#8710)
* fix(decommission): adopt target intents left by a killed leader

A decommission leader killed after persisting a target capacity intent and
before finalizing it left that intent in the exclusive pending slot of the
still-active reservation. The resumed routine derives the same reservation
identity, but every other mutation on that target was rejected as "another
mutation" until the interrupted object happened to be retried on the same
target, so the resumed decommission exhausted its retries and stalled.

At routine start, before any worker runs, the resumed routine now adopts the
reservation's pending intents: under the target fence (or the legacy pool
metadata fence) it moves each one into a mutation-scoped inflight marker that
keeps its identity and bytes. The interrupted mutation's retry settles the
marker through the existing commit, temporary-release and equivalent-target
paths, and an equivalent committed target now settles a mutation's own marker
when no pending intent remains. Adoption is pinned to the persisted
decommission start time and reservation identity, never runs mid-incarnation,
and a failure pauses the routine instead of marking the decommission failed.

* fix(decommission): keep adopted intents within budget and settle them once

Address the adversarial review of the leftover intent adoption.

- Merge an adopted intent into its mutation's staged marker by summing only
  while the target's temporary-copy budget allows, never below the larger of
  the two. A multipart completion moves its staged parts into the object, so
  summing them could exceed the budget and pause every later routine start;
  the adopted marker now exceeds the budget only where the completion's own
  finalize would have. Reuse resolve and record_inflight for the move.
- Distinguish a temporary release that proved the exact committed target from
  one that could not prove the target identity. When the committed mutation no
  longer has a pending intent, release only the bytes the cleanup freed and
  keep the adopted completion for the equivalent-target reconciliation.
- Let an equivalent target settle a mutation's own marker while another
  mutation holds the pending slot, instead of failing on that unrelated intent.
- Cover real leader kills before and after the target commit at routine level
  for both lock models, an injected adoption failure that pauses and then
  completes, an adopted multipart completion surviving a background abort, and
  a live legacy intent. Routine tests reuse the large-stack test helper.

* fix(decommission): pause instead of failing on target slot contention

The S00 leader-kill run with the null-version fix merged ended the resumed
decommission as failed. A re-run of a versioned key reconciled its delete
marker against an already charged target copy while another worker of the
same incarnation held the target's pending slot between a failed write and
its retry. The reconciliation rejected that unrelated intent, the entry
context wrapper flattened the error to a string the intent-conflict retry
could not recognize, and the bucket runner reported a sibling bucket's
OperationCanceled instead of the capacity block, so the routine marked the
decommission failed.

- An equivalent-target reconciliation, including the temporary checkpoint
  variant, no longer depends on another mutation's pending intent: a target
  holds one pending intent at a time, so this mutation has none to settle.
- The entry context keeps the typed cause, and the capacity-block and
  intent-conflict classifiers also recognize the rendered message that an
  error clone keeps.
- Concurrent bucket and listing failures keep the root cause over sibling
  cancellations, so a capacity block pauses the routine.

* test(decommission): delete marker copy waits for a busy target gate

The Linux run of the S00 matrix showed the delete marker copy meeting a busy
target capacity gate without any leader kill. The copy already waits on that
gate like an ordinary object copy; pin it with an entry-level test that holds
the gate, checks the entry keeps waiting, and expects the key to migrate once
the gate is released. Share the versioned delete marker fixture with the
foreign pending slot test.

* fix(decommission): keep an adopted completion unless the target is absent

Close the remaining path where an adopted completion could lose its record
before being charged. A multipart cleanup that found a target it could not
prove as the exact owned copy still dropped the whole marker of a mutation
without a pending intent, after which the equivalent-target reconciliation
would treat the commit as already charged.

Any cleanup that did not prove the exact target absent now releases only the
bytes it freed when the mutation has no pending intent, and leaves the rest
for the reconciliation. This makes the three-state release unnecessary, so
the release proof goes back to the existing clear-pending flag.

* chore(ci): shrink the error format ratchet baseline for pools.rs

9eae47ce8 removed one Error::other(format!) call site from crates/ecstore/src/core/pools.rs, so the ratchet requires the baseline to drop from 178 to 177.

* fix(decommission): keep adopted completions in their own capacity marker

An adopted completion shared the mutation's staged marker, so whether
it survived a cleanup depended on the pool-wide free-space delta, and a
cleanup that freed little space left staged bytes behind and repeated
forever.

Record each adopted intent under a mutation id derived from the
interrupted mutation (decommission_adopted_intent_mutation_id) as a
separate temporaryMutations entry. A commit or an equivalent-target
reconciliation settles both markers, a temporary cleanup releases the
staged marker as before, and only a cleanup that proved the exact
target absent drops the adopted one. While the target holds only that
adopted completion, a cleanup keeps the delayed-release exemption of
the published pending intent it replaced. Older nodes see an ordinary
temporary mutation they never release, which only over-reserves.

Also log capacity_intents_adopted as soon as the adopted ledger is
durable, merge the root-cause helpers into decommission_root_cause,
drop the decommission_capacity_blocked: prefix fallback, and log the
already-charged equivalent-target branch at debug level.

Refs rustfs/backlog#2873, rustfs/backlog#2872

* fix(decommission): return uncommitted retries to their adopted target

A retry that had not committed yet went to the reserved target with the
most remaining allocation. When that was not the target holding the
mutation's adopted completion, the commit settled only its own target
and the adopted marker stayed inflight until the reservation was
released.

Prefer a reserved target that holds the mutation's adopted marker while
it still fits the object (decommission_reserved_target_choice).

Also cover the remaining settlement paths of the derived marker
(non-growing replacement, equivalent temporary reconciliation and its
discovery gate, scanner backlog retirement), repeated adoption of one
mutation and the zero-byte staging record in both lock models, drop two
conditions that always equal another one, and correct the adopted-id
doc comment about nodes without adoption.

Refs rustfs/backlog#2873, rustfs/backlog#2872

* fix(decommission): count a completion adopted twice only once

A mutation killed twice leaves an adopted completion and then a new
intent for the same object version. Adoption added the two, so an
11-byte object counted 22 inflight bytes and could trip the capacity
check once later commits shrank the budget. Both reserve the same
completion, so merge them by their maximum before adding staged parts;
a single adoption is unchanged.

Also assert in the three-pool sticky test that no entry failed, the
retry is charged on its adopted target and the source is cleaned, run
it for the legacy lock model too, stop saying operators rely on the
adoption log before rollback, and document that the selection fallback
cannot help an adopted completion.

Refs rustfs/backlog#2873, rustfs/backlog#2872

* fix(decommission): log where an adopted retry is placed

An e2e run could not show whether a resumed retry went back to the
target holding its adopted completion. Target selection now logs
adopted_target_reused when it does, and adopted_target_bypassed with
the adopted target pools when the object no longer fits there and the
adopted marker stays behind. Both carry the mutation id and the source
and target pools.

Refs rustfs/backlog#2873, rustfs/backlog#2872
2026-10-20 11:28:22 +09:00
2025-12-18 20:13:24 +08:00
2025-07-08 09:04:37 +08:00
2025-08-07 22:37:05 +08:00

RustFS

RustFS is a high-performance, distributed object storage system built in Rust.

CI Build and Push Docker Images GitHub commit activity Github Last Commit Discord Featured|HelloGitHub

rustfs%2Frustfs | Trendshift

Getting Started · Docs · Bug reports · Discussions

English | 简体中文 | Deutsch | Español | français | 日本語 | 한국어 | Portuguese | Русский

RustFS is a high-performance, distributed object storage system built in Rust—one of the most loved programming languages worldwide. RustFS combines the simplicity of MinIO with the memory safety and raw performance of Rust. It offers broad S3 API compatibility for supported features, is completely open-source, and is optimized for data lakes, AI, and big data workloads.

Unlike other storage systems, RustFS is released under the permissible Apache 2.0 license, avoiding the restrictions of AGPL. With Rust as its foundation, RustFS delivers superior speed and secure distributed features for next-generation object storage.

Feature & Status

  • High Performance: Built with Rust to ensure maximum speed and resource efficiency.
  • Distributed Architecture: Scalable and fault-tolerant design suitable for large-scale deployments.
  • S3 Compatibility: Seamless integration with common S3-compatible applications and tools; current coverage is tracked in the S3 compatibility matrix.
  • OpenStack Swift API: Native support for Swift protocol with Keystone authentication.
  • OpenStack Keystone Integration: Native support for OpenStack Keystone authentication with X-Auth-Token headers.
  • Data Lake Support: Optimized for high-throughput big data and AI workloads.
  • Open Source: Licensed under Apache 2.0, encouraging unrestricted community contributions and commercial usage.
  • User-Friendly: Designed with simplicity in mind for easy deployment and management.

Status legend: ✅ Available — shipped and covered by CI gates; 🧪 Preview — shipped behind an opt-in flag or with a bounded compatibility claim.

Feature Status Feature Status
S3 Core Features ✅ Available Distributed Mode ✅ Available
Upload / Download ✅ Available Single Node Mode ✅ Available
Versioning ✅ Available Bitrot Protection ✅ Available
Object Lock (WORM) ✅ Available Healing & Scanner ✅ Available
Server-Side Encryption ✅ Available Pool Expansion / Decommission ✅ Available
RustFS KMS ✅ Available Bucket Replication ✅ Available
Lifecycle Management (ILM) ✅ Available Site Replication ✅ Available
ILM Tiering (Remote S3) ✅ Available Bucket Quota ✅ Available
S3 Select ✅ Available Event Notifications ✅ Available
S3 Tables (Iceberg REST) 🧪 Preview Audit Logging ✅ Available
IAM / Policies ✅ Available Logging & Observability ✅ Available
OIDC / SSO ✅ Available Web Console ✅ Available
Keystone Auth ✅ Available K8s Helm Charts ✅ Available
Swift API ✅ Available FTPS / WebDAV ✅ Available
Multi-Tenancy ✅ Available SFTP ✅ Available
MinIO On-Disk Compatibility 🧪 Preview

Notes:

  • RustFS KMS: Vault (KV2 / Transit) and AWS KMS backends are supported for production. The Local and Static backends are for development and testing only. See KMS backend security properties.
  • Swift API / SFTP: opt-in cargo features (--features swift, --features sftp, or full). FTPS and WebDAV are enabled in the default build.
  • S3 Tables: ships as an Iceberg REST Catalog with automated PyIceberg and DuckDB coverage; other engines and vendor profiles carry bounded claims listed in the S3 Tables support matrix.
  • MinIO On-Disk Compatibility: gated behind the rio-v2 feature and not part of the default build. Objects MinIO encrypted are not readable by RustFS. See MinIO file-format interoperability.

RustFS vs MinIO Performance

Stress Test Environment:

Type Parameter Remark
CPU 2 Core Intel Xeon (Sapphire Rapids) Platinum 8475B, 2.7/3.2 GHz
Memory 4GB
Network 15Gbps
Drive 40GB x 4 IOPS 3800 / Drive

https://github.com/user-attachments/assets/2e4979b5-260c-4f2c-ac12-c87fd558072a

RustFS vs Other Object Storage

Feature RustFS Other Object Storage
Console Experience Powerful Console
Comprehensive management interface.
Basic / Limited Console
Often overly simple or lacking critical features.
Language & Safety Rust-based
Memory safety by design.
Go or C-based
Potential for memory GC pauses or leaks.
Data Sovereignty No Telemetry / Full Compliance
Guards against unauthorized cross-border data egress. Compliant with GDPR (EU/UK), CCPA (US), and APPI (Japan).
Potential Risk
Possible legal exposure and unwanted data telemetry.
Licensing Permissive Apache 2.0
Business-friendly, no "poison pill" clauses.
Restrictive AGPL v3
Risk of license traps and intellectual property pollution.
Compatibility S3-Compatible Core
Works with common S3-compatible clients, with coverage tracked in the compatibility matrix.
Variable Compatibility
May lack support for local cloud vendors or specific APIs.
Edge & IoT Strong Edge Support
Ideal for secure, innovative edge devices.
Weak Edge Support
Often too heavy for edge gateways.
Risk Profile Enterprise Risk Mitigation
Clear IP rights and safe for commercial use.
Legal Risks
Intellectual property ambiguity and usage restrictions.

Staying ahead

Star RustFS on GitHub and be instantly notified of new releases.

Quickstart

Important

Pool expansion notice:

  • A single-node single-drive (SNSD) deployment is supported only as a standalone local path. It cannot expand in place or be added as a Pool. To move to a multi-drive topology, create a new deployment and migrate data through S3.
  • Keep an existing multi-drive Pool's endpoints and Erasure Set width unchanged; expand by appending a new Pool. With ellipsis-based expansion, every Pool argument must contain an ellipsis expression and expand to at least two drive endpoints.
  • Single-node multi-drive Pools and multi-node Pools with one drive per node are allowed, subject to valid Erasure Set geometry and EC settings; acceptance does not guarantee host-failure tolerance.

These topology rules follow MinIO, but automatic parity selection differs between the projects. See the Pool layout compatibility and regression tests before expanding a deployment.

To get started with RustFS, follow these steps:

1. One-click Installation (Option 1)

curl -O https://rustfs.com/install_rustfs.sh && bash install_rustfs.sh

2. Docker Quick Start (Option 2)

The RustFS container runs as a non-root user rustfs (UID/GID 10001:10001). If you bind-mount host directories with Docker or Compose, every mounted path must be writable by that user, otherwise startup may fail with permission denied errors. This applies to data directories, log directories, and TLS certificate directories when RUSTFS_TLS_PATH is enabled.

# Create data and logs directories
mkdir -p data logs

# Change the owner of these directories
chown -R 10001:10001 data logs

# Using latest version
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest

# Using specific version
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.3

If you use podman instead of docker, you can install the RustFS with the below command

# Create data and logs directories
mkdir -p data logs

# Run the container (podman will automatically set the folders ownership)
podman run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data:Z,U -v $(pwd)/logs:/logs:Z,U rustfs/rustfs:latest

If you enable TLS with a bind-mounted certificate directory, prepare that mount the same way:

mkdir -p certs
chown -R 10001:10001 certs

You can also use Docker Compose. Using the docker-compose-simple.yml file in the root directory:

docker compose -f docker-compose-simple.yml up -d

Before running Compose with host bind mounts:

  • Ensure every mounted host path is writable by 10001:10001.
  • If you enable TLS, ensure the certificate mount for /opt/tls is also readable by 10001:10001.
  • If matching host ownership is not practical, run the rustfs service with user: "<host-uid>:<host-gid>" instead.
  • docker-compose-simple.yml includes a volume-permission-helper service for named volumes. docker-compose-simple.yml relies on you to prepare bind-mounted host paths in advance.

Similarly, you can run the command with podman

podman compose -f docker-compose-simple.yml up -d

Webhook notification quick start (Docker):

docker run -d --name rustfs -p 9000:9000 \
  -e RUSTFS_NOTIFY_ENABLE=true \
  -e RUSTFS_NOTIFY_WEBHOOK_ENABLE_PRIMARY=on \
  -e RUSTFS_NOTIFY_WEBHOOK_ENDPOINT_PRIMARY=http://<host-ip>:3020/webhook \
  -e RUSTFS_NOTIFY_WEBHOOK_QUEUE_DIR_PRIMARY=/tmp/rustfs-events \
  -e RUSTFS_OUTBOUND_ALLOW_ORIGINS=http://<host-ip>:3020 \
  rustfs/rustfs:latest

Notes:

  • RUSTFS_NOTIFY_ENABLE=true enables the global notify module switch.
  • For ARN arn:rustfs:sqs::primary:webhook, use instance-scoped env vars with _PRIMARY.
  • If queue dir is omitted, the official image maps /opt/rustfs/events into the persistent /data volume so it is writable by the runtime user and pending events survive container recreation. Override queue_dir when using another deployment layout.
  • RUSTFS_NOTIFY_WEBHOOK_SKIP_TLS_VERIFY_PRIMARY defaults to false; enabling it skips webhook TLS certificate verification, allows MITM attacks, and emits a startup warning. Prefer RUSTFS_NOTIFY_WEBHOOK_CLIENT_CA_PRIMARY for private CAs.
  • Since 1.0.0-beta.11, webhook endpoints on private or container networks (Docker Compose service names, host.docker.internal, RFC 1918 addresses) are blocked unless their exact scheme://host:port origin is listed in RUSTFS_OUTBOUND_ALLOW_ORIGINS (the origin only, without the path). See Outbound Connection Policy.

NOTE: We recommend reviewing the docker-compose.yml file before running. It defines several services including Grafana, Prometheus, and Jaeger, which are helpful for RustFS observability. If you wish to start Redis or Nginx containers, you can specify the corresponding profiles.

3. Build from Source (Option 3) - Advanced Users

For developers who want to build RustFS Docker images from source with multi-architecture support:

# Build multi-architecture images locally
./docker-buildx.sh

# Build a single-platform image locally
./docker-buildx.sh -p linux/amd64

# Build and push to registry
./docker-buildx.sh --push

# Build specific version
./docker-buildx.sh --release 1.0.3 --push

# Build for custom registry
./docker-buildx.sh --registry your-registry.com --namespace yourname --push

The docker-buildx.sh script supports:

  • Multi-architecture builds: linux/amd64, linux/arm64
  • Automatic version detection: Uses git tags or commit hashes
  • Registry flexibility: Supports Docker Hub, GitHub Container Registry, etc.
  • Build optimization: Includes caching and parallel builds

You can also use Make targets for convenience:

make docker-buildx                    # Build locally
make docker-buildx-push               # Build and push
make docker-buildx-version VERSION=1.0.3  # Build specific version
make help-docker                      # Show all Docker-related commands

Heads-up (macOS cross-compilation): macOS keeps the default ulimit -n at 256, so cargo zigbuild or ./build-rustfs.sh --platform ... may fail with ProcessFdQuotaExceeded when targeting Linux. The build script attempts to raise the limit automatically, but if you still see the warning, run ulimit -n 4096 (or higher) in your shell before building.

4. Build with Helm Chart (Option 4) - Cloud Native

Follow the instructions in the Helm Chart README to install RustFS on a Kubernetes cluster.

For scanner pacing, cycle budgets, bitrot cadence, lifecycle transition status, and single-node single-disk idle CPU tuning, see Scanner Runtime Controls. For repeatable scanner-pressure validation, see Scanner Benchmark Runbook. For drive timeout knobs on slow storage — including the walk stall budget that governs ListObjects on large prefixes — see Drive Timeout Tuning.

5. Nix Flake (Option 5)

If you have Nix with flakes enabled:

# Run directly without installing
nix run github:rustfs/rustfs

# Build the binary
nix build github:rustfs/rustfs
./result/bin/rustfs --help

# Or from a local checkout
nix build
nix run

The flake also exports a NixOS module and the RustFS rc client. Add the module to your system and provide credentials through runtime files (for example, sops-nix or agenix) so secrets are never stored in the Nix store:

imports = [ inputs.rustfs.nixosModules.rustfs ];

services.rustfs = {
  enable = true;
  accessKeyFile = "/run/secrets/rustfs-access-key";
  secretKeyFile = "/run/secrets/rustfs-secret-key";
  volumes = [ "/var/lib/rustfs" ];
};

Install the S3-compatible client with nix profile install github:rustfs/rustfs#rustfs-client (the executable is named rc), or use inputs.rustfs.packages.${pkgs.system}.rustfs-client in a system configuration.

6. X-CMD (Option 6)

If you are an x-cmd user:

# Run directly without installing
x rustfs

# Download the binary and install it to the global environment
x env use rustfs
rustfs --help

Accessing RustFS

  1. Access the Console: Open your web browser and navigate to http://localhost:9001 to access the RustFS console.
    • Default credentials: rustfsadmin / rustfsadmin
  2. Create a Bucket: Use the console to create a new bucket for your objects.
  3. Upload Objects: You can upload files directly through the console or use S3-compatible APIs/clients to interact with your RustFS instance.

NOTE: To access the RustFS instance via https, please refer to the TLS Configuration Docs.

OIDC Roles Claim (Microsoft Entra ID)

RustFS supports mapping an OIDC claim containing role values into the existing authorization pipeline. The roles_claim setting is optional: when unset or empty, only the groups claim contributes to authorization (same as older RustFS releases). For Microsoft Entra ID app roles, set roles_claim=roles so both console admin checks and bucket IAM policies can evaluate those roles.

Example environment configuration (opt-in roles claim):

RUSTFS_IDENTITY_OPENID_ENABLE=on
RUSTFS_IDENTITY_OPENID_CONFIG_URL="https://login.microsoftonline.com/<tenant-id>/v2.0/.well-known/openid-configuration"
RUSTFS_IDENTITY_OPENID_CLIENT_ID="<client-id>"
RUSTFS_IDENTITY_OPENID_CLIENT_SECRET="<client-secret>"
RUSTFS_IDENTITY_OPENID_SCOPES="openid,profile,email"
RUSTFS_IDENTITY_OPENID_GROUPS_CLAIM="groups"
RUSTFS_IDENTITY_OPENID_ROLES_CLAIM="roles"

Policy condition example (evaluate app roles directly with jwt:roles; when roles_claim is configured, RustFS also merges those values into jwt:groups for backward compatibility with older policies):

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["admin:*"],
      "Resource": ["arn:aws:s3:::*"],
      "Condition": {
        "ForAnyValue:StringEquals": {
          "jwt:roles": ["RustFS.ConsoleAdmin"]
        }
      }
    }
  ]
}

Documentation

For detailed documentation, including configuration options, API references, and advanced usage, please visit our Documentation.

Getting Help

If you have any questions or need assistance:

  • Check the FAQ for common issues and solutions.
  • Join our GitHub Discussions to ask questions and share your experiences.
  • Open an issue on our GitHub Issues page for bug reports or feature requests.

Contact

Contributors

RustFS is a community-driven project, and we appreciate all contributions. Check out the Contributors page to see the amazing people who have helped make RustFS better.

RustFS contributors

Star History

RustFS star history chart

License

Apache 2.0

RustFS is a trademark of RustFS, Inc. All other trademarks are the property of their respective owners.

S
Description
2.3x faster than MinIO for 4KB object payloads. RustFS is an open-source, S3-compatible high-performance object storage system supporting migration and coexistence with other S3-compatible platforms such as MinIO and Ceph.
Readme Apache-2.0 311 MiB
Languages
Rust 94.4%
Shell 2.8%
Python 2.7%