houseme 3b139e5267 fix(obs/cleaner): harden log cleaner durability, symlink safety, and retention (audit OLC-01..14) (#4776)
* fix(obs/cleaner): fsync archive and log dir before deleting source logs

OLC-01: the compression path flushed the BufWriter but never synced the
archive data or the parent directory before renaming, and the source was
then unlinked with no durability barrier. A crash after rename but before
the page cache reached disk could leave a truncated/zero-length archive
while the source was already gone — permanent log/audit data loss. Because
the archive is always renamed to a brand-new name (guaranteed by the
existing exists() guard), ext4 auto_da_alloc does not mask this.

Hand the underlying File back from the writer closure, sync_all() it before
rename, fsync the parent directory, and fsync the log directory after the
unlinks so a delete cannot be reordered ahead of the archive it justified.
Guard the temp file with an RAII cleanup so an early return or panic cannot
leak a *.tmp orphan.

Ref: rustfs/backlog#1194 (audit rustfs/backlog#1193)

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(obs/cleaner): create compression temp file with O_EXCL and O_NOFOLLOW

OLC-03: the temp archive was opened with File::create at a predictable
`<source>.gz.tmp` path with no O_EXCL/O_NOFOLLOW, so an actor with write
access to the log directory could pre-plant that path as a symlink and have
the compressor follow it — truncating and overwriting an arbitrary external
file, then chmod-ing it to the source log's mode. This mirrors the symlink
refusal already enforced on the deletion path (secure_delete).

Route temp creation through create_tmp_archive(), which uses create_new
(O_CREAT|O_EXCL) to refuse a pre-existing entry and, on Unix, O_NOFOLLOW to
refuse a symlink at the final path component.

Ref: rustfs/backlog#1196 (audit rustfs/backlog#1193)

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(obs/cleaner): create compression temp file with restrictive mode

OLC-06: File::create left the temp archive world-readable (0644 & ~umask)
for the entire duration of compressing a large log, exposing the full
plaintext of a possibly-0600 audit log on shared hosts until the mode was
copied only after the write completed. Pass the source mode into
create_tmp_archive and open the temp file with it (default 0600) so it is
restrictive from creation; the post-write chmod still tightens/matches the
source mode exactly.

Ref: rustfs/backlog#1199 (audit rustfs/backlog#1193)

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(obs/cleaner): validate existing archive before skipping recompression

OLC-02: the idempotency guard used Path::exists() (which follows symlinks)
and trusted whatever it found, then the caller deleted the source. A planted
`<archive>.gz` symlink, or a zero-length/truncated archive left by a crashed
run (OLC-01), would green-light deleting the source with no valid backup —
data loss / log destruction.

Replace exists() with symlink_metadata (no follow) and only treat the entry
as a completed prior result when it is a regular, non-empty file whose header
matches the codec magic (gzip 1f 8b / zstd 28 b5 2f fd). Anything else falls
through to recompression, whose atomic create_new+rename replaces the bad
entry (a symlink is replaced, never followed or deleted through).

Ref: rustfs/backlog#1195 (audit rustfs/backlog#1193)

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(obs/cleaner): stop dry-run overstating reclaimed bytes for compression

OLC-08: in dry-run, compress_with_writer returned output_bytes = 0, so
projected_freed_bytes = input and delete_files reported the full input as
freed. A real run keeps the archive on disk (freed = input - archive), so
dry-run overstated reclaim by the whole archive footprint. Estimate the
archive with a deliberately conservative ratio so the projection never
exceeds what a real run reclaims.

Ref: rustfs/backlog#1201 (audit rustfs/backlog#1193)

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(obs/cleaner): make freed-byte accounting resilient; document steal metric

OLC-12: input/output byte sizes were read via metadata().unwrap_or(0), which
silently reports 0 on failure and skews freed-byte metrics (input - 0 = full
input, overstating reclaim). Use the copy() byte count as the authoritative
input size and, when the archive metadata read fails, conservatively assume
no savings instead of 0. Also document that the steal_success_rate counts
only victim steals (batch = one success), so it reads as a relative
rebalancing signal, not absolute task acquisition.

Ref: rustfs/backlog#1205 (audit rustfs/backlog#1193)

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(obs/cleaner): preserve level-0 semantics, allow zstd 22, log effective levels

OLC-09: build() and the codec calls clamped gzip/zstd levels to [1,9]/[1,21],
silently rewriting gzip level 0 (store) and zstd level 0 (codec default) to 1
and blocking the legal zstd maximum of 22. Clamp to [0,9]/[0,22] so those
meanings survive, and echo the effective (post-clamp) levels in the startup
log via new effective_gzip_level()/effective_zstd_level() getters so the log
matches what actually runs.

Ref: rustfs/backlog#1202 (audit rustfs/backlog#1193)

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(obs/cleaner): bound compressed archives by byte cap; warn on retention=0

OLC-04: archive expiry was gated on compressed_file_retention_days > 0, so
retention=0 disabled it entirely while compression kept producing archives,
and max_total_size_bytes only ever bounded uncompressed logs — unbounded disk
growth. Replace select_expired_compressed with select_archives_to_delete,
which applies age expiry (when retention is on) and, regardless of retention,
trims the oldest archives until the set fits under max_total_size_bytes. Also
warn at startup when compression is on with retention=0 so the "keep forever"
semantics are not mistaken for "delete immediately".

Ref: rustfs/backlog#1197 (audit rustfs/backlog#1193)

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(obs/cleaner): warn on invalid exclude glob instead of dropping silently

OLC-05: build() dropped unparseable exclude globs via filter_map(...ok()), so
a typo (or a literal comma splitting a char-class in the config string) turned
"protect this file" into "delete this file" with no signal. Log a warning per
rejected pattern with the raw string and parse error so the misconfiguration
is visible.

Ref: rustfs/backlog#1198 (audit rustfs/backlog#1193)

Co-Authored-By: heihutu <heihutu@gmail.com>

* perf(obs/cleaner): backoff idle workers, cap worker count, lower small-host floor

OLC-11: the work-stealing loop re-spun on Steal::Retry with no yield and used
yield_now on the empty path, burning CPU during redistribution windows;
worker_count had no upper bound so a mis-set parallel_workers over a directory
of thousands of logs could spawn thousands of threads; and
default_parallel_workers forced >=4 workers even on 1-2 vCPU hosts. Use
crossbeam_utils::Backoff (spin->yield, reset on work) on the idle paths, clamp
worker_count to MAX_PARALLEL_COMPRESS_WORKERS, and lower the default floor to 1.

Ref: rustfs/backlog#1204 (audit rustfs/backlog#1193)

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(obs/cleaner): warn when active-file guard is disabled by empty filename

OLC-13 (defense-in-depth): the scanner protects the live log purely by exact
filename equality against active_filename. An empty active_filename silently
disables that protection, so a non-empty file_pattern could make the live log
a deletion candidate via the public builder. Warn in build() when that unsafe
combination is configured. The audit's "never delete the newest match"
structural guard is intentionally not implemented: it would conflict with the
legitimate keep_files=0 semantics (purge all rotated logs). The naming
contract is instead locked by regression tests (OLC-14).

Ref: rustfs/backlog#1206 (audit rustfs/backlog#1193)

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(obs/cleaner): warn on unknown algorithm/match_mode, echo match_mode

OLC-07: from_config_str silently fell back to defaults for unrecognized
compression algorithm and match mode (any non-"prefix" value became Suffix),
hiding operator typos like "prefixx" that could make the cleaner match no
rotated logs. Warn on a non-empty unrecognized value in both parsers, and
echo the resolved match_mode in the startup log alongside the algorithm.

Ref: rustfs/backlog#1200 (audit rustfs/backlog#1193)

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(obs/cleaner): derive orphan .tmp suffixes and exempt them from min age

OLC-10: orphan `*.gz.tmp`/`*.zst.tmp` cleanup was gated by min_file_age_seconds
(default 3600), so crash-left orphans lingered up to an hour, and the tmp
suffix list was hardcoded rather than derived from compressed_suffixes() — a
new codec would leave `*.<ext>.tmp` orphans the scanner never recognizes.
Derive the temp suffix from CompressionAlgorithm::compressed_suffixes(), and
gate orphan removal on a small fixed grace window instead of min_file_age
(orphans are never live-written after the rename that would promote them).

Ref: rustfs/backlog#1203 (audit rustfs/backlog#1193)

Co-Authored-By: heihutu <heihutu@gmail.com>

* test(obs/cleaner): cover symlink, archive expiry, idempotency, and edge cases

OLC-14: add regression tests for the previously-untested safety/correctness
branches — symlink rejection (external target never deleted), archive age
expiry vs fresh retention, archive byte-cap trim with retention disabled,
gz/zst classification, max_single_file_size selection, min_age protecting a
fresh non-empty log, active-file exclusion when the active name also matches
the pattern, invalid exclude glob not aborting build, dry-run + compression
creating no archive, gzip round-trip validity, and the idempotent-archive
branch trusting a valid prior archive.

Ref: rustfs/backlog#1207 (audit rustfs/backlog#1193)

Co-Authored-By: heihutu <heihutu@gmail.com>

* style(obs/cleaner): apply rustfmt and collapse nested if (clippy)

Formatting-only cleanup over the audit fix series: rustfmt normalization of the
multi-line expressions introduced in compress.rs/core.rs, plus collapsing the
delete_files directory-fsync into a single let-chain to satisfy
clippy::collapsible_if. No behavior change.

Co-Authored-By: heihutu <heihutu@gmail.com>

* refactor(obs/cleaner): collapse redundant source stats in compression path

The per-issue fixes to compress_with_writer accumulated three metadata()
syscalls on the source in the real compression path: an input_bytes read that
was immediately shadowed by the copied byte count (a dead read), a source_mode
read (OLC-06), and the pre-OLC-06 post-write chmod re-reading the same mode.
Collapse to a single fd-based read — move the dry-run input_bytes read into
the dry-run branch, read source_mode from the already-open fd (no path stat,
no TOCTOU), and reuse it for the post-write chmod. Behavior is unchanged
(same inode's mode, written for input size); 3 source stats -> 1.

Co-Authored-By: heihutu <heihutu@gmail.com>

* chore(obs/cleaner): fix typo flagged by CI (mis-set -> misconfigured)

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-12 12:01:52 +00:00
2025-12-18 20:13:24 +08:00
2025-07-08 09:04:37 +08:00
2025-08-07 22:37:05 +08:00
2025-06-30 21:27:45 +08:00

RustFS

RustFS is a high-performance, distributed object storage system built in Rust.

CI Build and Push Docker Images GitHub commit activity Github Last Commit Discord Featured|HelloGitHub

rustfs%2Frustfs | Trendshift ROSS Index - Fastest Growing Open-Source Startups in Q4 2025 | Runa Capital

Getting Started · Docs · Bug reports · Discussions

English | 简体中文 | Deutsch | Español | français | 日本語 | 한국어 | Portuguese | Русский

RustFS is a high-performance, distributed object storage system built in Rust—one of the most loved programming languages worldwide. RustFS combines the simplicity of MinIO with the memory safety and raw performance of Rust. It offers broad S3 API compatibility for supported features, is completely open-source, and is optimized for data lakes, AI, and big data workloads.

Unlike other storage systems, RustFS is released under the permissible Apache 2.0 license, avoiding the restrictions of AGPL. With Rust as its foundation, RustFS delivers superior speed and secure distributed features for next-generation object storage.

Feature & Status

  • High Performance: Built with Rust to ensure maximum speed and resource efficiency.
  • Distributed Architecture: Scalable and fault-tolerant design suitable for large-scale deployments.
  • S3 Compatibility: Seamless integration with common S3-compatible applications and tools; current coverage is tracked in the S3 compatibility matrix.
  • OpenStack Swift API: Native support for Swift protocol with Keystone authentication.
  • OpenStack Keystone Integration: Native support for OpenStack Keystone authentication with X-Auth-Token headers.
  • Data Lake Support: Optimized for high-throughput big data and AI workloads.
  • Open Source: Licensed under Apache 2.0, encouraging unrestricted community contributions and commercial usage.
  • User-Friendly: Designed with simplicity in mind for easy deployment and management.
Feature Status Feature Status
S3 Core Features Available Bitrot Protection Available
Upload / Download Available Single Node Mode Available
Versioning Available Bucket Replication Available
Logging Available Lifecycle Management 🚧 Under Testing
Event Notifications Available Distributed Mode 🚧 Under Testing
K8s Helm Charts Available RustFS KMS 🚧 Under Testing
Keystone Auth Available Multi-Tenancy Available
Swift API Available Swift Metadata Ops 🚧 Partial

RustFS vs MinIO Performance

Stress Test Environment:

Type Parameter Remark
CPU 2 Core Intel Xeon (Sapphire Rapids) Platinum 8475B, 2.7/3.2 GHz
Memory 4GB
Network 15Gbps
Drive 40GB x 4 IOPS 3800 / Drive

https://github.com/user-attachments/assets/2e4979b5-260c-4f2c-ac12-c87fd558072a

RustFS vs Other Object Storage

Feature RustFS Other Object Storage
Console Experience Powerful Console
Comprehensive management interface.
Basic / Limited Console
Often overly simple or lacking critical features.
Language & Safety Rust-based
Memory safety by design.
Go or C-based
Potential for memory GC pauses or leaks.
Data Sovereignty No Telemetry / Full Compliance
Guards against unauthorized cross-border data egress. Compliant with GDPR (EU/UK), CCPA (US), and APPI (Japan).
Potential Risk
Possible legal exposure and unwanted data telemetry.
Licensing Permissive Apache 2.0
Business-friendly, no "poison pill" clauses.
Restrictive AGPL v3
Risk of license traps and intellectual property pollution.
Compatibility S3-Compatible Core
Works with common S3-compatible clients, with coverage tracked in the compatibility matrix.
Variable Compatibility
May lack support for local cloud vendors or specific APIs.
Edge & IoT Strong Edge Support
Ideal for secure, innovative edge devices.
Weak Edge Support
Often too heavy for edge gateways.
Risk Profile Enterprise Risk Mitigation
Clear IP rights and safe for commercial use.
Legal Risks
Intellectual property ambiguity and usage restrictions.

Staying ahead

Star RustFS on GitHub and be instantly notified of new releases.

Quickstart

To get started with RustFS, follow these steps:

1. One-click Installation (Option 1)

curl -O https://rustfs.com/install_rustfs.sh && bash install_rustfs.sh

2. Docker Quick Start (Option 2)

The RustFS container runs as a non-root user rustfs (UID/GID 10001:10001). If you bind-mount host directories with Docker or Compose, every mounted path must be writable by that user, otherwise startup may fail with permission denied errors. This applies to data directories, log directories, and TLS certificate directories when RUSTFS_TLS_PATH is enabled.

# Create data and logs directories
mkdir -p data logs

# Change the owner of these directories
chown -R 10001:10001 data logs

# Using latest version
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest

# Using specific version
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0-beta.8

If you use podman instead of docker, you can install the RustFS with the below command

# Create data and logs directories
mkdir -p data logs

# Run the container (podman will automatically set the folders ownership)
podman run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data:Z,U -v $(pwd)/logs:/logs:Z,U rustfs/rustfs:latest

If you enable TLS with a bind-mounted certificate directory, prepare that mount the same way:

mkdir -p certs
chown -R 10001:10001 certs

You can also use Docker Compose. Using the docker-compose-simple.yml file in the root directory:

docker compose -f docker-compose-simple.yml up -d

Before running Compose with host bind mounts:

  • Ensure every mounted host path is writable by 10001:10001.
  • If you enable TLS, ensure the certificate mount for /opt/tls is also readable by 10001:10001.
  • If matching host ownership is not practical, run the rustfs service with user: "<host-uid>:<host-gid>" instead.
  • docker-compose-simple.yml includes a volume-permission-helper service for named volumes. docker-compose-simple.yml relies on you to prepare bind-mounted host paths in advance.

Similarly, you can run the command with podman

podman compose -f docker-compose-simple.yml up -d

Webhook notification quick start (Docker):

docker run -d --name rustfs -p 9000:9000 \
  -e RUSTFS_NOTIFY_ENABLE=true \
  -e RUSTFS_NOTIFY_WEBHOOK_ENABLE_PRIMARY=on \
  -e RUSTFS_NOTIFY_WEBHOOK_ENDPOINT_PRIMARY=http://<host-ip>:3020/webhook \
  -e RUSTFS_NOTIFY_WEBHOOK_QUEUE_DIR_PRIMARY=/tmp/rustfs-events \
  rustfs/rustfs:latest

Notes:

  • RUSTFS_NOTIFY_ENABLE=true enables the global notify module switch.
  • For ARN arn:rustfs:sqs::primary:webhook, use instance-scoped env vars with _PRIMARY.
  • If queue dir is omitted, default is /opt/rustfs/events; ensure it is writable by the container runtime user.
  • RUSTFS_NOTIFY_WEBHOOK_SKIP_TLS_VERIFY_PRIMARY defaults to false; enabling it skips webhook TLS certificate verification, allows MITM attacks, and emits a startup warning. Prefer RUSTFS_NOTIFY_WEBHOOK_CLIENT_CA_PRIMARY for private CAs.

NOTE: We recommend reviewing the docker-compose.yml file before running. It defines several services including Grafana, Prometheus, and Jaeger, which are helpful for RustFS observability. If you wish to start Redis or Nginx containers, you can specify the corresponding profiles.

3. Build from Source (Option 3) - Advanced Users

For developers who want to build RustFS Docker images from source with multi-architecture support:

# Build multi-architecture images locally
./docker-buildx.sh --build-arg RELEASE=latest

# Build and push to registry
./docker-buildx.sh --push

# Build specific version
./docker-buildx.sh --release v1.0.0 --push

# Build for custom registry
./docker-buildx.sh --registry your-registry.com --namespace yourname --push

The docker-buildx.sh script supports:

  • Multi-architecture builds: linux/amd64, linux/arm64
  • Automatic version detection: Uses git tags or commit hashes
  • Registry flexibility: Supports Docker Hub, GitHub Container Registry, etc.
  • Build optimization: Includes caching and parallel builds

You can also use Make targets for convenience:

make docker-buildx                    # Build locally
make docker-buildx-push               # Build and push
make docker-buildx-version VERSION=v1.0.0  # Build specific version
make help-docker                      # Show all Docker-related commands

Heads-up (macOS cross-compilation): macOS keeps the default ulimit -n at 256, so cargo zigbuild or ./build-rustfs.sh --platform ... may fail with ProcessFdQuotaExceeded when targeting Linux. The build script attempts to raise the limit automatically, but if you still see the warning, run ulimit -n 4096 (or higher) in your shell before building.

4. Build with Helm Chart (Option 4) - Cloud Native

Follow the instructions in the Helm Chart README to install RustFS on a Kubernetes cluster.

For scanner pacing, cycle budgets, bitrot cadence, lifecycle transition status, and single-node single-disk idle CPU tuning, see Scanner Runtime Controls. For repeatable scanner-pressure validation, see Scanner Benchmark Runbook.

5. Nix Flake (Option 5)

If you have Nix with flakes enabled:

# Run directly without installing
nix run github:rustfs/rustfs

# Build the binary
nix build github:rustfs/rustfs
./result/bin/rustfs --help

# Or from a local checkout
nix build
nix run

6. X-CMD (Option 6)

If you are an x-cmd user:

# Run directly without installing
x rustfs

# Download the binary and install it to the global environment
x env use rustfs
rustfs --help

Accessing RustFS

  1. Access the Console: Open your web browser and navigate to http://localhost:9001 to access the RustFS console.
    • Default credentials: rustfsadmin / rustfsadmin
  2. Create a Bucket: Use the console to create a new bucket for your objects.
  3. Upload Objects: You can upload files directly through the console or use S3-compatible APIs/clients to interact with your RustFS instance.

NOTE: To access the RustFS instance via https, please refer to the TLS Configuration Docs.

OIDC Roles Claim (Microsoft Entra ID)

RustFS supports mapping an OIDC claim containing role values into the existing authorization pipeline. The roles_claim setting is optional: when unset or empty, only the groups claim contributes to authorization (same as older RustFS releases). For Microsoft Entra ID app roles, set roles_claim=roles so both console admin checks and bucket IAM policies can evaluate those roles.

Example environment configuration (opt-in roles claim):

RUSTFS_IDENTITY_OPENID_ENABLE=on
RUSTFS_IDENTITY_OPENID_CONFIG_URL="https://login.microsoftonline.com/<tenant-id>/v2.0/.well-known/openid-configuration"
RUSTFS_IDENTITY_OPENID_CLIENT_ID="<client-id>"
RUSTFS_IDENTITY_OPENID_CLIENT_SECRET="<client-secret>"
RUSTFS_IDENTITY_OPENID_SCOPES="openid,profile,email"
RUSTFS_IDENTITY_OPENID_GROUPS_CLAIM="groups"
RUSTFS_IDENTITY_OPENID_ROLES_CLAIM="roles"

Policy condition example (evaluate app roles directly with jwt:roles; when roles_claim is configured, RustFS also merges those values into jwt:groups for backward compatibility with older policies):

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["admin:*"],
      "Resource": ["arn:aws:s3:::*"],
      "Condition": {
        "ForAnyValue:StringEquals": {
          "jwt:roles": ["RustFS.ConsoleAdmin"]
        }
      }
    }
  ]
}

Documentation

For detailed documentation, including configuration options, API references, and advanced usage, please visit our Documentation.

Getting Help

If you have any questions or need assistance:

  • Check the FAQ for common issues and solutions.
  • Join our GitHub Discussions to ask questions and share your experiences.
  • Open an issue on our GitHub Issues page for bug reports or feature requests.

Contact

Contributors

RustFS is a community-driven project, and we appreciate all contributions. Check out the Contributors page to see the amazing people who have helped make RustFS better.

Contributors

Star History

Star History Chart

License

Apache 2.0

RustFS is a trademark of RustFS, Inc. All other trademarks are the property of their respective owners.

S
Description
2.3x faster than MinIO for 4KB object payloads. RustFS is an open-source, S3-compatible high-performance object storage system supporting migration and coexistence with other S3-compatible platforms such as MinIO and Ceph.
Readme Apache-2.0 108 MiB
Languages
Rust 94.9%
Shell 3.9%
Python 1%