houseme 15b8b13698 feat(ecstore): cache part-file descriptors for io_uring reads (#4658)
* feat(ecstore): run one io_uring ring per shard on each disk (backlog#1145)

A buffered read that hits the page cache completes inline inside
`io_uring_enter`, so the thread driving a ring performs that read's
memcpy. One ring per disk therefore capped cache-hit reads at a single
core's memory bandwidth: measured on a 16-core host, one driver thread sat
pinned at 100% CPU while throughput stayed flat at ~5 GB/s regardless of
read size, against 50 GB/s for the blocking-pool baseline.

rustfs/uring#6 taught the driver to hold N independent rings, each with its
own thread, pending table, backpressure semaphore, and eventfd. Wire it up:
`UringBackend::try_new` now calls `probe_and_start_sharded`, and
`RUSTFS_IO_URING_SHARDS` selects the count per disk.

The default is a quarter of the available parallelism clamped to `1..=4`,
because the cost is `disks × shards` driver threads (each normally blocked
in `poll(2)`). Any override is clamped to `1..=16`, so a mistyped value can
neither disable the driver (0) nor spawn threads without bound; an
unparseable value falls back to the default.

Effect (warm page cache, 16-core, rustfs/uring's concurrent_pread_bench):

  1 MiB, conc 8:    1 shard  4911 MB/s -> 8 shards 47361 MB/s (9.6x);
                    the blocking-pool baseline is 50662 MB/s
  64 KiB, conc 32:  StdBackend 153678 IOPS, p999 3030 us
                    8 shards   345402 IOPS, p999  897 us
  64 KiB, conc 128: StdBackend 135155 IOPS, p999 10716 us
                    8 shards    389047 IOPS, p999  4092 us

Sharding removes the throughput deficit *and* keeps io_uring's tail-latency
advantage, rather than trading one for the other.

Unchanged: io_uring read stays gray-off by default
(`RUSTFS_IO_URING_READ_ENABLE`), reads are byte-for-byte identical to
StdBackend, the per-disk degradation latches and probe cache (backlog#1101)
and the O_DIRECT tiered fallback (backlog#1102) all still apply. Rings stay
per-disk, so a stalled disk cannot starve another disk's rings
(backlog#1055). Bumps the rustfs-uring pin to the merged #6 commit.

Verified on a real Linux host (16-core, real io_uring): cargo clippy
--tests -D warnings clean; disk::local tests 132 passed, 0 failed —
including the existing io_uring and O_DIRECT cases now running on the
sharded driver, plus a new test covering the shard-count default, override,
and clamping.

Co-Authored-By: heihutu <heihutu@gmail.com>

* feat(ecstore): cache part-file descriptors for io_uring reads (backlog#1145)

`pread_uring` opened the file on the blocking pool for every read, so each
read paid a `spawn_blocking` round trip — the very thread hop io_uring
exists to avoid. Sharding the driver (backlog#1145) removed the previous
ceiling and left this as the binding cost. Measured on a 16-core host with
a 4-shard driver, warm page cache:

  64 KiB, conc 8:   143942 -> 263054 IOPS (+83%),  p999 240 -> 65 us
  64 KiB, conc 32:  150128 -> 204876 IOPS (+36%),  p999 2508 -> 871 us
  64 KiB, conc 128: 129172 -> 361287 IOPS (+180%), p999 15329 -> 3046 us
  1 MiB,  conc 32:   33875 ->  42301 IOPS (+25%)

At 64 KiB / conc 128 the open is what masked io_uring entirely: with it,
io_uring beat StdBackend by 3.5%; without it, by 189%.

Add a bounded per-disk descriptor cache used by the io_uring read path.
A hit takes no `open` and no `spawn_blocking`, so the read never leaves the
runtime worker.

Why caching a part-file descriptor is safe:
  * only `<object>/<data_dir>/part.N` reaches this backend's `pread_bytes`;
    `xl.meta` — the one path replaced in place — is read through `read_all`
    / `read_metadata` and never gets here;
  * part files are never rewritten in place. A replacement is always
    write-new-tmp then `rename`, which swaps the inode, so a cached
    descriptor can never observe a torn shard.

Why invalidation is nevertheless REQUIRED: heal reuses the existing
version's `data_dir` and renames a rebuilt shard onto the SAME part path.
A cached descriptor would keep serving the pre-heal (corrupt) inode,
defeating the heal and eroding read quorum. `delete` likewise unlinks a
part that a cached descriptor would keep readable. So `rename_data`,
`rename_file`, and `delete` all call the new
`LocalIoBackend::invalidate_cached_fds` after they mutate, and a 5s TTL
bounds the blast radius should a future mutation path forget to.

Two preamble checks the miss path runs are not silently lost on a hit:
  * bounds — the driver only short-reads at EOF (it resubmits otherwise),
    so `bytes.len() != length` is exactly the old `meta.len() < end_offset`
    check, and now yields the same `FileCorrupt`;
  * volume access — skipped while an entry is live. An unreachable disk
    keeps serving already-open descriptors for at most the TTL, after which
    the re-open re-runs the check. Disk health is tracked independently of
    this per-read probe.

Scope: buffered io_uring reads only. The O_DIRECT path keeps opening per
read (its reads are >= 4 MiB, so the open is a small fraction), and
StdBackend is untouched — it must take the blocking hop for the pread
regardless, so caching would buy it only 2-6% while carrying the same
invalidation risk. `RUSTFS_IO_URING_FD_CACHE=false` restores open-per-read.

Verified on a real Linux host (16-core, real io_uring): clippy --tests
-D warnings clean; disk::local tests 135 passed, 0 failed. The new
heal-staleness test first asserts a read still returns the PRE-heal bytes —
proving the cache is live and the hazard real — then that invalidation makes
the healed shard visible. A second test drives `rename_file` and `delete`
through `LocalDisk` to prove those paths actually invalidate, and a unit
test pins prefix invalidation to component boundaries (`a/b` must not drop
`a/bc`).

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-10 08:33:50 +00:00
2025-12-18 20:13:24 +08:00
2025-07-08 09:04:37 +08:00
2025-08-07 22:37:05 +08:00
2025-06-30 21:27:45 +08:00

RustFS

RustFS is a high-performance, distributed object storage system built in Rust.

CI Build and Push Docker Images GitHub commit activity Github Last Commit Discord Featured|HelloGitHub

rustfs%2Frustfs | Trendshift ROSS Index - Fastest Growing Open-Source Startups in Q4 2025 | Runa Capital

Getting Started · Docs · Bug reports · Discussions

English | 简体中文 | Deutsch | Español | français | 日本語 | 한국어 | Portuguese | Русский

RustFS is a high-performance, distributed object storage system built in Rust—one of the most loved programming languages worldwide. RustFS combines the simplicity of MinIO with the memory safety and raw performance of Rust. It offers broad S3 API compatibility for supported features, is completely open-source, and is optimized for data lakes, AI, and big data workloads.

Unlike other storage systems, RustFS is released under the permissible Apache 2.0 license, avoiding the restrictions of AGPL. With Rust as its foundation, RustFS delivers superior speed and secure distributed features for next-generation object storage.

Feature & Status

  • High Performance: Built with Rust to ensure maximum speed and resource efficiency.
  • Distributed Architecture: Scalable and fault-tolerant design suitable for large-scale deployments.
  • S3 Compatibility: Seamless integration with common S3-compatible applications and tools; current coverage is tracked in the S3 compatibility matrix.
  • OpenStack Swift API: Native support for Swift protocol with Keystone authentication.
  • OpenStack Keystone Integration: Native support for OpenStack Keystone authentication with X-Auth-Token headers.
  • Data Lake Support: Optimized for high-throughput big data and AI workloads.
  • Open Source: Licensed under Apache 2.0, encouraging unrestricted community contributions and commercial usage.
  • User-Friendly: Designed with simplicity in mind for easy deployment and management.
Feature Status Feature Status
S3 Core Features Available Bitrot Protection Available
Upload / Download Available Single Node Mode Available
Versioning Available Bucket Replication Available
Logging Available Lifecycle Management 🚧 Under Testing
Event Notifications Available Distributed Mode 🚧 Under Testing
K8s Helm Charts Available RustFS KMS 🚧 Under Testing
Keystone Auth Available Multi-Tenancy Available
Swift API Available Swift Metadata Ops 🚧 Partial

RustFS vs MinIO Performance

Stress Test Environment:

Type Parameter Remark
CPU 2 Core Intel Xeon (Sapphire Rapids) Platinum 8475B, 2.7/3.2 GHz
Memory 4GB
Network 15Gbps
Drive 40GB x 4 IOPS 3800 / Drive

https://github.com/user-attachments/assets/2e4979b5-260c-4f2c-ac12-c87fd558072a

RustFS vs Other Object Storage

Feature RustFS Other Object Storage
Console Experience Powerful Console
Comprehensive management interface.
Basic / Limited Console
Often overly simple or lacking critical features.
Language & Safety Rust-based
Memory safety by design.
Go or C-based
Potential for memory GC pauses or leaks.
Data Sovereignty No Telemetry / Full Compliance
Guards against unauthorized cross-border data egress. Compliant with GDPR (EU/UK), CCPA (US), and APPI (Japan).
Potential Risk
Possible legal exposure and unwanted data telemetry.
Licensing Permissive Apache 2.0
Business-friendly, no "poison pill" clauses.
Restrictive AGPL v3
Risk of license traps and intellectual property pollution.
Compatibility S3-Compatible Core
Works with common S3-compatible clients, with coverage tracked in the compatibility matrix.
Variable Compatibility
May lack support for local cloud vendors or specific APIs.
Edge & IoT Strong Edge Support
Ideal for secure, innovative edge devices.
Weak Edge Support
Often too heavy for edge gateways.
Risk Profile Enterprise Risk Mitigation
Clear IP rights and safe for commercial use.
Legal Risks
Intellectual property ambiguity and usage restrictions.

Staying ahead

Star RustFS on GitHub and be instantly notified of new releases.

Quickstart

To get started with RustFS, follow these steps:

1. One-click Installation (Option 1)

curl -O https://rustfs.com/install_rustfs.sh && bash install_rustfs.sh

2. Docker Quick Start (Option 2)

The RustFS container runs as a non-root user rustfs (UID/GID 10001:10001). If you bind-mount host directories with Docker or Compose, every mounted path must be writable by that user, otherwise startup may fail with permission denied errors. This applies to data directories, log directories, and TLS certificate directories when RUSTFS_TLS_PATH is enabled.

# Create data and logs directories
mkdir -p data logs

# Change the owner of these directories
chown -R 10001:10001 data logs

# Using latest version
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest

# Using specific version
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0-beta.8

If you use podman instead of docker, you can install the RustFS with the below command

# Create data and logs directories
mkdir -p data logs

# Run the container (podman will automatically set the folders ownership)
podman run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data:Z,U -v $(pwd)/logs:/logs:Z,U rustfs/rustfs:latest

If you enable TLS with a bind-mounted certificate directory, prepare that mount the same way:

mkdir -p certs
chown -R 10001:10001 certs

You can also use Docker Compose. Using the docker-compose-simple.yml file in the root directory:

docker compose -f docker-compose-simple.yml up -d

Before running Compose with host bind mounts:

  • Ensure every mounted host path is writable by 10001:10001.
  • If you enable TLS, ensure the certificate mount for /opt/tls is also readable by 10001:10001.
  • If matching host ownership is not practical, run the rustfs service with user: "<host-uid>:<host-gid>" instead.
  • docker-compose-simple.yml includes a volume-permission-helper service for named volumes. docker-compose-simple.yml relies on you to prepare bind-mounted host paths in advance.

Similarly, you can run the command with podman

podman compose -f docker-compose-simple.yml up -d

Webhook notification quick start (Docker):

docker run -d --name rustfs -p 9000:9000 \
  -e RUSTFS_NOTIFY_ENABLE=true \
  -e RUSTFS_NOTIFY_WEBHOOK_ENABLE_PRIMARY=on \
  -e RUSTFS_NOTIFY_WEBHOOK_ENDPOINT_PRIMARY=http://<host-ip>:3020/webhook \
  -e RUSTFS_NOTIFY_WEBHOOK_QUEUE_DIR_PRIMARY=/tmp/rustfs-events \
  rustfs/rustfs:latest

Notes:

  • RUSTFS_NOTIFY_ENABLE=true enables the global notify module switch.
  • For ARN arn:rustfs:sqs::primary:webhook, use instance-scoped env vars with _PRIMARY.
  • If queue dir is omitted, default is /opt/rustfs/events; ensure it is writable by the container runtime user.
  • RUSTFS_NOTIFY_WEBHOOK_SKIP_TLS_VERIFY_PRIMARY defaults to false; enabling it skips webhook TLS certificate verification, allows MITM attacks, and emits a startup warning. Prefer RUSTFS_NOTIFY_WEBHOOK_CLIENT_CA_PRIMARY for private CAs.

NOTE: We recommend reviewing the docker-compose.yml file before running. It defines several services including Grafana, Prometheus, and Jaeger, which are helpful for RustFS observability. If you wish to start Redis or Nginx containers, you can specify the corresponding profiles.

3. Build from Source (Option 3) - Advanced Users

For developers who want to build RustFS Docker images from source with multi-architecture support:

# Build multi-architecture images locally
./docker-buildx.sh --build-arg RELEASE=latest

# Build and push to registry
./docker-buildx.sh --push

# Build specific version
./docker-buildx.sh --release v1.0.0 --push

# Build for custom registry
./docker-buildx.sh --registry your-registry.com --namespace yourname --push

The docker-buildx.sh script supports:

  • Multi-architecture builds: linux/amd64, linux/arm64
  • Automatic version detection: Uses git tags or commit hashes
  • Registry flexibility: Supports Docker Hub, GitHub Container Registry, etc.
  • Build optimization: Includes caching and parallel builds

You can also use Make targets for convenience:

make docker-buildx                    # Build locally
make docker-buildx-push               # Build and push
make docker-buildx-version VERSION=v1.0.0  # Build specific version
make help-docker                      # Show all Docker-related commands

Heads-up (macOS cross-compilation): macOS keeps the default ulimit -n at 256, so cargo zigbuild or ./build-rustfs.sh --platform ... may fail with ProcessFdQuotaExceeded when targeting Linux. The build script attempts to raise the limit automatically, but if you still see the warning, run ulimit -n 4096 (or higher) in your shell before building.

4. Build with Helm Chart (Option 4) - Cloud Native

Follow the instructions in the Helm Chart README to install RustFS on a Kubernetes cluster.

For scanner pacing, cycle budgets, bitrot cadence, lifecycle transition status, and single-node single-disk idle CPU tuning, see Scanner Runtime Controls. For repeatable scanner-pressure validation, see Scanner Benchmark Runbook.

5. Nix Flake (Option 5)

If you have Nix with flakes enabled:

# Run directly without installing
nix run github:rustfs/rustfs

# Build the binary
nix build github:rustfs/rustfs
./result/bin/rustfs --help

# Or from a local checkout
nix build
nix run

6. X-CMD (Option 6)

If you are an x-cmd user:

# Run directly without installing
x rustfs

# Download the binary and install it to the global environment
x env use rustfs
rustfs --help

Accessing RustFS

  1. Access the Console: Open your web browser and navigate to http://localhost:9001 to access the RustFS console.
    • Default credentials: rustfsadmin / rustfsadmin
  2. Create a Bucket: Use the console to create a new bucket for your objects.
  3. Upload Objects: You can upload files directly through the console or use S3-compatible APIs/clients to interact with your RustFS instance.

NOTE: To access the RustFS instance via https, please refer to the TLS Configuration Docs.

OIDC Roles Claim (Microsoft Entra ID)

RustFS supports mapping an OIDC claim containing role values into the existing authorization pipeline. The roles_claim setting is optional: when unset or empty, only the groups claim contributes to authorization (same as older RustFS releases). For Microsoft Entra ID app roles, set roles_claim=roles so both console admin checks and bucket IAM policies can evaluate those roles.

Example environment configuration (opt-in roles claim):

RUSTFS_IDENTITY_OPENID_ENABLE=on
RUSTFS_IDENTITY_OPENID_CONFIG_URL="https://login.microsoftonline.com/<tenant-id>/v2.0/.well-known/openid-configuration"
RUSTFS_IDENTITY_OPENID_CLIENT_ID="<client-id>"
RUSTFS_IDENTITY_OPENID_CLIENT_SECRET="<client-secret>"
RUSTFS_IDENTITY_OPENID_SCOPES="openid,profile,email"
RUSTFS_IDENTITY_OPENID_GROUPS_CLAIM="groups"
RUSTFS_IDENTITY_OPENID_ROLES_CLAIM="roles"

Policy condition example (evaluate app roles directly with jwt:roles; when roles_claim is configured, RustFS also merges those values into jwt:groups for backward compatibility with older policies):

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["admin:*"],
      "Resource": ["arn:aws:s3:::*"],
      "Condition": {
        "ForAnyValue:StringEquals": {
          "jwt:roles": ["RustFS.ConsoleAdmin"]
        }
      }
    }
  ]
}

Documentation

For detailed documentation, including configuration options, API references, and advanced usage, please visit our Documentation.

Getting Help

If you have any questions or need assistance:

  • Check the FAQ for common issues and solutions.
  • Join our GitHub Discussions to ask questions and share your experiences.
  • Open an issue on our GitHub Issues page for bug reports or feature requests.

Contact

Contributors

RustFS is a community-driven project, and we appreciate all contributions. Check out the Contributors page to see the amazing people who have helped make RustFS better.

Contributors

Star History

Star History Chart

License

Apache 2.0

RustFS is a trademark of RustFS, Inc. All other trademarks are the property of their respective owners.

S
Description
2.3x faster than MinIO for 4KB object payloads. RustFS is an open-source, S3-compatible high-performance object storage system supporting migration and coexistence with other S3-compatible platforms such as MinIO and Ceph.
Readme Apache-2.0 108 MiB
Languages
Rust 94.9%
Shell 3.9%
Python 1%