The individual policy layers were tested in isolation, but the cross-layer
resolution — which layer wins when IAM and bucket policy disagree — had no
coverage. A priority error there is a data-exposure or data-lockout bug.
Add crates/policy/tests/bucket_iam_authz_matrix.rs, a pure table-driven test
(no globals, no IAM store, no server) that drives the real Policy::is_allowed
and BucketPolicy::is_allowed through a helper modeling the request-layer
orchestration in rustfs::storage::access::authorize_request (bucket explicit-Deny
gate -> IAM allow -> bucket allow fallback; anonymous = bucket policy alone). It
pins all four Allow/Deny quadrants plus the anonymous case, and adds intra-policy
Deny-beats-Allow checks for both layers.
The matrix surfaces the resolution invariant explicitly: a BUCKET explicit Deny
is a hard gate and always wins, but an IAM explicit Deny is SOFT — a bucket Allow
fallback overrides it, which diverges from AWS "an explicit Deny in any policy
always wins". The test characterizes the current (MinIO-lineage) behavior; if IAM
Deny is later hardened into a gate, iam_deny_x_bucket_allow flips to false and
must be updated deliberately.
Refs: backlog#1151 (sec-8)
Co-authored-by: houseme <housemecn@gmail.com>
test(security): pin GHSA-m77q STS root-secret token signing (sec-7)
GHSA-m77q-r63m-pj89 (intentionally UNFIXED) is that STS session tokens are
signed with the shared root secret: crates/iam/src/root_credentials.rs
token_signing_key() returns the root secret_key, so anyone holding the root
secret can forge STS session tokens. No test named the advisory, and the
existing test_created_sts_credentials_authorize_with_session_token_claims uses
token_signing_key() for both signing and verifying, so it pins "same key signs
and verifies" but not the m77q-specific "signing key IS the root secret" — a
future fix that decouples the STS key from the root secret would pass it
silently.
Add a flow-level pin, test_ghsa_m77q_sts_session_token_signed_with_root_secret,
that captures the advisory's exact signature: (1) token_signing_key() == the
root secret; (2) an AssumeRole-style session token issued with that key decodes
with the root secret and NOT with any other secret; (3) it authorizes through
the STS path. All three assert CURRENT (by-design-vulnerable) behavior, so a
real m77q fix (a dedicated STS signing key) turns them red and forces a
red -> green regression update.
Also GHSA-name the existing characterization test and token_signing_key() with
doc comments and the advisory URL, and update the m77q row in
docs/testing/security-regressions.md. No production behavior changes.
Refs: backlog#1151 (sec-7)
Co-authored-by: houseme <housemecn@gmail.com>
chore(scripts): index scripts/ and archive 29 one-shot scripts
backlog#1153 infra-13. scripts/ had 80+ unlabelled top-level entries
mixing CI gates with finished one-shot issue-validation scripts.
- scripts/README.md — one index row per entry with status
(ci-gate / dev-tool / archived), purpose, and wiring; subdirectories
get one row each. run_scanner_benchmarks.sh is annotated
"disposition owned by backlog perf-10" and deliberately untouched.
- git mv 29 confirmed-stale one-shot entries to scripts/archive/:
11 issue-scoped validation/perf-capture scripts, the 5-script
backlog#706 large-PUT breakdown family, the 4-file GET-optimization
stress suite, 2 gt1g one-shots, and 7 other orphaned one-shots.
Evidence: a whole-tree boundary-aware reference census showed zero
references from CI/Makefiles/docs/code for every moved entry (or
references only from other scripts inside the same archived set);
re-run after the move shows zero dangling references.
- docs/testing/README.md links the index.
Moves only — no script content changed.
bench(rustfs): remove the fake s3_operations benchmark
rustfs/benches/s3_operations.rs never measured S3: all three groups
(put_object/get_object/list_objects) only black_box(data.len())/black_box(count)
with a 'In a real benchmark, this would call the actual S3 client' comment. It
gave a false 'S3 operations have benchmark coverage' signal and was dead weight
on rustfs/Cargo.toml.
Delete the file and its [[bench]] entry. S3-face performance is covered solely
by the warp A/B gate (performance-ab.yml); the runbook scope note now says so
explicitly. Micro-benchmarks stay at the function level (perf-8).
Refs rustfs/backlog#1152 (perf-9).
The script cannot run on any machine: WORKSPACE_ROOT is hardcoded to a personal
path (/home/dandan/code/rust/rustfs) and it targets the rustfs-ahm crate, which
no longer exists in the workspace (scanner is crates/scanner, heal is
crates/heal, neither ships benches/). It gave a false impression that scanner
performance automation existed.
There is no current scanner-bench requirement, so retire it (option a). Nothing
references the script anywhere else in the repo.
Refs rustfs/backlog#1152 (perf-10).
ci(coverage): weekly cargo-llvm-cov workspace baseline (non-blocking)
Add the weekly line-coverage report (backlog#1153 infra-5):
- .github/workflows/coverage.yml — Sunday 07:00 UTC + workflow_dispatch;
runs `cargo llvm-cov nextest --workspace --exclude e2e_test` under
NEXTEST_PROFILE=ci (same scope and profile as the ci.yml test gate),
writes a per-crate line-coverage table to the job summary, uploads
lcov + JSON as a 90-day artifact, and routes scheduled failures
through the shared schedule-failure-issue action (ci-8). Runs only on
schedule/dispatch, so it can never become a required PR check.
- scripts/coverage_per_crate.py — stdlib-only aggregation of the
llvm-cov JSON export into the per-crate markdown table (worst-first,
TOTAL row), shared by the workflow and the local target.
- make coverage (.config/make/coverage.mak) — local equivalent with the
same command sequence; fails with install hints when cargo-llvm-cov
or cargo-nextest are missing. Listed in make help.
- docs/testing/README.md — Coverage section: cadence, where the table
and artifacts live, the trend-comparison method, and what is not
measured (doctests, e2e_test).
Every push to main now builds the release binary once and stores it in the
actions cache as rustfs-baseline-<sha> (build-baseline-cache job). The A/B job
restores it for origin/main and passes --baseline-bin; on the nightly, where
the candidate commit equals the baseline, one cached binary serves both phases
with --skip-build and the run does zero source builds. A cache miss falls back
to the source double-build via --baseline-ref origin/main.
This removes the ~32min-per-side double build that pushed the 24-cell nightly
past its 120min ceiling (2026-07-11..07-14 all cancelled on timeout).
Also:
- alert-on-failure now fires on cancelled/timed-out, not just failure, so a
timed-out nightly is no longer silent (the composite action already reports
cancelled jobs); removed the stale perf-2 TODO now that the alert job exists.
- run_hotpath_warp_ab.sh appends a Provenance section to gate.md (baseline and
candidate SHAs + binary source, runner, warp version, matrix params) via a
repeatable --provenance-note; the gate exit code is preserved.
Refs rustfs/backlog#1152 (perf-3).
The header-SigV4 (sec-1), presigned-URL (sec-2), and admin-gate (sec-4) negative
auth-rejection e2e suites merged earlier but only presigned_negative was actually
selected by any CI profile; negative_sigv4_test and admin_auth_test compiled and
sat unrun. Add both to the e2e-smoke default-filter so all three attacker-facing
S3 auth-rejection suites execute on every PR. They already meet the smoke
admission criteria (RustFSTestEnvironment, random ports, parallel-safe, no
#[ignore], no feature gates), so this is a pure filterset change — the single
e2e-in-CI wiring mechanism (backlog#1149 ci-4), not a new job.
Because the filter selects by module name, a rename or deletion could silently
drop a suite out of the security gate with no CI signal. Add
scripts/check_security_smoke_count.sh (infra-12 count-floor mechanism): it lists
what the e2e-smoke profile selects and fails if the count of security
auth-rejection tests drops below the committed floor in
.config/security-smoke-floor.txt (16). Invoked from the e2e-tests job, before the
smoke run, so the nextest list compiles the binaries the run reuses.
The GHSA-3p3x FTPS/WebDAV constant-time e2e (protocols::test_protocol_core_suite)
stays out by topology: it binds fixed ports, needs the ftps,webdav features, and
is #[serial], so it cannot join the random-port default-feature smoke profile as
a filterset change (global ruling G5). Its GHSA-r5qv sibling is a unit test that
already runs in the default CI pass. docs/testing/security-regressions.md now
carries the full CI-execution map and flags the GHSA-3p3x e2e CI-lane gap as a
ci-domain follow-up.
Refs: backlog#1151 (sec-5)
ci(mint): bypass local physical-disk-independence guard so mint boots
The mint container maps four RUSTFS_VOLUMES data dirs onto a single
runner device, so the startup physical-disk-independence guard aborts
with a FATAL before mint can run. The scheduled run 29183544431
(2026-07-12) crashed at 'Wait for RustFS ready' with 'local erasure
endpoints must use distinct physical disks'.
Set RUSTFS_UNSAFE_BYPASS_DISK_CHECK=true on the container, the CI use
the guard explicitly sanctions -- mirroring the e2e-s3tests harness
fixed in #4768. Completes the ci-2 acceptance (a mint run that actually
produces log.json and the per-suite summary).
docs(testing): populate testing pyramid, naming, serial/nextest rules
Fill the docs/testing/README.md skeleton (backlog#1153 infra-11):
- Test taxonomy table for all eight layers (unit / ecstore black-box /
e2e / s3s-e2e / S3 compatibility / chaos / fuzz / bench) with a
verified entry command and a qualitative "when it runs" per layer; the
event x timeout x required-status matrix stays owned by
docs/testing/ci-gates.md (ci-15), linked not duplicated.
- Naming conventions section with the migration-gate reserved substrings
(data_movement / rebalance / decommission / source_cleanup /
delete_marker) linking the infra-12 count-floor guard, closing that
task's docs cross-link.
- Serial execution & nextest rules: nextest as a hard dependency with the
RUSTFS_ALLOW_CARGO_TEST_FALLBACK escape hatch and the runner-semantics
difference (folds the infra-14 README half), why #[serial] is a no-op
under nextest, and the default/ci/e2e-smoke/e2e-repl-nightly profiles.
- A time-control placeholder for infra-4 to fill.
The pre-existing flake-policy section (ci-10) is preserved verbatim.
Add pointers from CLAUDE.md and CONTRIBUTING.md.
* test(ecstore): assert decrypted_size for MinIO SSE interop round-trip
The ignored MinIO interop round-trip tests asserted `ObjectInfo.size`
against the plaintext length. For SSE objects `size` is the on-disk
DARE-encrypted size (plaintext + 32 bytes per 64 KiB block), so the
assertion can never hold once real fixtures are present — the two
`#[ignore]` tests failed the moment a real MinIO-written fixture was fed
in, even though the decoded data was byte-identical.
The client-visible object size comes from `decrypted_size()` /
`get_actual_size()`, which correctly reads MinIO's
`x-*-internal-actual-size` metadata (verified: both SSE-S3 and SSE-KMS
8 MiB multipart fixtures now report 8388608). Assert against that
instead and keep the plaintext length and SHA-256 data checks.
With real 4-drive MinIO fixtures (RELEASE.2025-09-07) all four tests
pass, confirming RustFS reads MinIO erasure-coded SSE objects with
byte-identical data and correct logical size.
Co-Authored-By: heihutu <heihutu@gmail.com>
* ci(ecstore): nightly MinIO interop check + dockerized fixture capture
Wire the ignored MinIO on-disk interop reader tests into a nightly,
non-required CI job, and make their fixtures reproducible without a host
MinIO install.
- Dockerfile + capture_via_docker.sh: build a throwaway image carrying
the official MinIO server binary (pinned RELEASE.2025-09-07) plus the
fixture lab on a small Python base, then run `lab.py capture-matrix` to
write the SSE-S3 / SSE-KMS multipart fixtures the tests consume. lab.py
drives MinIO's S3 API directly, so no `mc` is needed.
- .github/workflows/minio-interop.yml: nightly + manual workflow on
GitHub-hosted ubuntu-latest (reliable Docker + Python, unlike the
self-hosted fleet — see e2e-s3tests.yml infra note). Regenerates the
gitignored fixtures each run and executes the #[ignore] reader tests.
Not a PR gate.
- README: document the Docker capture path.
Validated end to end: the script builds the image, captures the two
multipart cases, and `cargo nextest run --run-ignored ignored-only`
passes all four interop tests.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
test(replication): lock outbound checksum consistency for new algorithms (T3)
Adds a consistency test at the replication put-options boundary confirming that
the AWS 2026-04 additional checksum algorithms (XXHash3/64/128, SHA-512, MD5) are
forwarded into replication user_metadata identically to the classic five. The
outbound replication path routes a stored object checksum through the
algorithm-agnostic decrypt_checksums -> user_metadata flow, so the new algorithms
(already covered by rustfs-rio read_checksums) need no new-algorithm-specific
handling. This locks that behavior against regressions.
Investigation summary (no code change needed on the outbound side): the
per-algorithm ChecksumMode selection path is dormant (opts.checksum is never set
to a specific algorithm; tiering uses Content-MD5; the add_crc bool is dead
code), so extending ChecksumMode was unnecessary.
Co-authored-by: heihutu <heihutu@gmail.com>
* feat(rio): wire XXHash3/64/128 and SHA-512 into ChecksumType (S2)
Add the AWS 2026-04 additional checksum algorithms as base types in
rustfs-rio's ChecksumType, covering every dispatch site (key, raw_byte_len,
hasher, Display, from_string_with_obj_type, BASE_CHECKSUM_TYPES) so no path
silently strips them. Derive BASE_TYPE_MASK from BASE_CHECKSUM_TYPES as the
single source of truth, allocate the new base-type bits append-only above
bit 9 to preserve the on-disk varint format, and add streaming hashers whose
digest uses the S3 canonical big-endian encoding (seed 0).
The new algorithms are COMPOSITE-only: an explicit FULL_OBJECT request is
rejected and they are never routed through add_part()/can_merge(). A
round-trip guardrail test asserts every base type survives all dispatch
sites, failing loudly if a future algorithm is added but a match arm or the
mask is forgotten.
Refs rustfs/backlog#1254 rustfs/backlog#1252
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(rio): pin XXHash/SHA-512 digests to official vectors, big-endian (S3)
Lock the byte order and seed of the new algorithms against the OFFICIAL
upstream xxHash / SHA-512 empty-input test vectors (XXH3-64, XXH64, XXH3-128,
SHA-512), in big-endian, so the stored and echoed checksum is byte-for-byte
identical to what AWS SDKs (awscrt) compute — the interop correctness this
feature hinges on. Add a non-empty regression lock (official "fox" vectors)
that also asserts the encoded field is the standard-base64 of the raw digest.
Refs rustfs/backlog#1255rustfs/backlog#1252
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(rio): lock on-disk checksum round-trip and forward-compat degrade (S8)
Cover the xl.meta varint (de)serialization for the new algorithms:
to_bytes() -> read_checksums() must recover the value under the Display key
for XXHASH3/64/128 and SHA512. Pin the rolling-upgrade contract that a node
reading a future, unknown base-type bit degrades safely — skips the entry and
returns without panicking or mis-decoding a length. Combined with the
append-only bit allocation from S2, this protects mixed-version clusters.
Refs rustfs/backlog#1260rustfs/backlog#1252
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(head): echo XXHash/SHA-512 additional checksums on HeadObject (S5)
HeadObject with x-amz-checksum-mode: ENABLED now returns the XXHash3/64/128
and SHA-512 checksums that S3 stored, closing the head_object gap in #4800.
s3s HeadObjectOutput has no typed field for these, so they are emitted as raw
response headers via response.headers (the same mechanism RustFS already uses
for tagging-count), keyed by ChecksumType::key(). The existing five typed
algorithms are unchanged. Also carries the Cargo.lock update for the
xxhash-rust dependency introduced in S2.
Refs rustfs/backlog#1257rustfs/backlog#1252
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(checksums): fail-closed on unknown checksum algorithm (S7)
A. Harden unknown/unsupported checksum algorithms to fail closed instead of
panicking. ChecksumMode::base() in the outbound S3 client
(crates/ecstore/src/client/checksum.rs) previously did
`panic!("enum err.")` for any mode without a concrete base algorithm (e.g.
a bare ChecksumFullObject flag); it now falls back to ChecksumNone. Added
unit tests proving base() never panics and hasher() returns Err for
unsupported modes. rustfs-checksums FromStr already returns Err on unknown
names; added a regression test asserting garbage/unknown names fail closed.
B. Extend rustfs-checksums ChecksumAlgorithm with the AWS 2026-04 additional
algorithms Sha512/Xxhash3/Xxhash64/Xxhash128. Updated FromStr, as_str,
into_impl, name constants, the x-amz-checksum-* header constants and the
HttpChecksum impls. Byte order/seed matches the server-side rustfs-rio
spec: xxh3/xxh64 as u64 big-endian (8 bytes, seed 0), xxh128 as u128
big-endian (16 bytes), sha512 via sha2::Sha512. Added tests validating each
digest against a direct library computation. MD5 stays intentionally
rejected (PR #4513) and is left untouched.
C. crates/ecstore/src/client/checksum.rs ChecksumMode is enumset repr="u8"
with 7 variants already consuming 7 bits; adding the 4 new algorithms would
overflow u8 and require a breaking repr change, so ChecksumMode is left
unchanged. The new algorithms are available through the rustfs-checksums
ChecksumAlgorithm path.
Refs rustfs/backlog#1259rustfs/backlog#1252
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(get,put): echo XXHash/SHA-512 checksums on GetObject and PutObject (S5-GET, S4)
Complete the additional-checksum round-trip so AWS SDKs can verify integrity on
download and confirm it on upload:
- GetObject with x-amz-checksum-mode: ENABLED now returns XXHash3/64/128 and
SHA-512 checksums (the download-side path SDKs auto-verify). The values flow
from build_get_object_checksums through GetObjectOutputContext into
finalize_get_object_response and are emitted after wrap_response_with_cors.
- PutObject echoes the server-computed additional checksum on its response,
captured at the want_checksum set points before opts is moved.
Both reuse a single centralized helper, inject_additional_checksum_headers,
which HeadObject now also uses. This is the ONLY place that emits these headers,
so when s3s gains typed fields for these algorithms the migration is one spot
(fill the typed field, drop the insert) with no risk of duplicate headers.
The five s3s-typed algorithms are unchanged. Trailing-checksum PUT echo (value
lands after the body) is left for e2e coverage in S10.
Refs rustfs/backlog#1257rustfs/backlog#1256rustfs/backlog#1252
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(multipart): support XXHash/SHA-512 composite multipart checksums (S9)
Make multipart uploads work end-to-end for the composite-only algorithms
(XXHash3/64/128, SHA-512):
- complete_part_checksum previously returned the outer None for any algorithm
outside the five typed ones, which failed CompleteMultipartUpload with
InvalidPart. It now accepts any valid base type with no double-check value
(Some(None)) — mirroring the missing-value path of the typed algorithms —
since s3s CompletePart has no field to carry a client-supplied per-part
value and the part was already verified server-side at UploadPart. Genuinely
unset/invalid types are still rejected.
- The existing COMPOSITE assembly (Checksum::new_from_data over the
concatenated per-part raw digests; full_object_requested() is false so
add_part() is correctly bypassed) already works for these algorithms via the
S2 wiring. A rio test locks the assembly and that add_part refuses them.
- UploadPart and CompleteMultipartUpload echo the new-algorithm checksum on
their responses via the shared inject_additional_checksum_headers helper
(now pub(crate)), since s3s has no typed output field.
Refs rustfs/backlog#1261rustfs/backlog#1252
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(rio): add MD5 as an additional checksum (x-amz-checksum-md5) (S6)
Wire MD5 into ChecksumType as an additional (flexible) checksum, distinct from
the legacy Content-MD5 / ETag path: header x-amz-checksum-md5, 16-byte digest,
COMPOSITE-only, md-5 hasher. Pinned to the official empty-input MD5 vector.
Thanks to the single-source-of-truth wiring from S2, every dispatch site
(GetObject/HeadObject/PutObject echo, multipart complete_part_checksum and the
COMPOSITE assembly) picks MD5 up automatically via base()/key()/the catch-all
arm — no handler changes needed. Tests are extended to cover MD5 across them.
Coordination with #4513: that PR made the OUTBOUND rustfs-checksums client
reject "md5" so it could never silently fall back to CRC32. This change is on
the server-side rio path and never falls back — it implements MD5 correctly
rather than substituting another algorithm — so the #4513 intent is preserved,
and the outbound client keeps rejecting md5 (S7).
Refs rustfs/backlog#1258rustfs/backlog#1252
Co-Authored-By: heihutu <heihutu@gmail.com>
* perf(rio): drop per-request to_uppercase alloc in checksum parsing (S11)
from_string_with_obj_type ran alg.to_uppercase() on every checksummed request,
allocating a String just to compare against a fixed set of algorithm names.
Replace it with eq_ignore_ascii_case, which is allocation-free and, for the
ASCII algorithm names involved, exactly equivalent. A test locks that
case-insensitivity, the CRC64NVME full-object assumption, composite-only
FULL_OBJECT rejection, and unknown/empty handling are all unchanged.
The other S11 notes are intentionally not acted on: the Phase-0 header scan is
N/A (we chose full support over rejection, so there is no reject guard), and
parallelizing the serialized hash passes is deferred pending a measured need.
Refs rustfs/backlog#1263rustfs/backlog#1252
Co-Authored-By: heihutu <heihutu@gmail.com>
* refactor(checksums): collapse 5 duplicated response-checksum loops into one
Review of the accumulated commits found the same "iterate decrypted checksums,
match five typed algorithms, drop the rest" loop copy-pasted across five
response paths (GetObject, HeadObject, GetObjectAttributes object-level and
part-level, CompleteMultipartUpload). That was patch-on-patch duplication.
Collapse it into a single source of truth:
- rustfs-rio gains ChecksumType::is_s3s_typed() — the one place that defines the
five-typed vs additional-algorithm split.
- object_usecase gains ResponseChecksums + classify_response_checksums(), which
performs the typed/extra split once. All five call sites now destructure its
result; additional_checksum_echo_pairs() also uses is_s3s_typed() instead of a
hand-rolled five-way comparison.
Behaviour is unchanged (GetObjectAttributes still cannot surface the additional
algorithms — an s3s XML-body limitation, now documented in one spot). One pass
over the map; extra pairs pushed only when a new-algorithm checksum is present.
Refs rustfs/backlog#1252
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(checksums): unit tests for classifier/echo helpers + fix unused import
Add direct unit tests for the refactored single-source-of-truth helpers:
- rio ChecksumType::is_s3s_typed() — exhaustive typed-vs-additional split, and
that flags (FULL_OBJECT/MULTIPART) on a base type don't change classification.
- object_usecase classify_response_checksums() — typed fields vs `extra` headers,
the checksum-type marker, and empty input.
- additional_checksum_echo_pairs() — echo pair only for additional algorithms,
none for the five typed ones, none for None.
- inject_additional_checksum_headers() — writes all pairs; empty is a no-op.
Also drop the now-unused AMZ_CHECKSUM_TYPE import in multipart_usecase.rs left
by the classifier refactor (would fail the -D warnings gate).
Refs rustfs/backlog#1252
Co-Authored-By: heihutu <heihutu@gmail.com>
* style(rio): fix typo flagged by CI (mis-decoding -> decoding a wrong length)
The Typos CI check flagged "mis-decoding" (it reads "mis" as a word). Reword
the S8 forward-compat comment; no code change.
Refs rustfs/backlog#1260
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(e2e): integration test for XXHash/SHA-512/MD5 additional checksums (S10)
Permanent verify-on-write integration test in the e2e suite for the AWS 2026-04
additional algorithms. aws_sdk_s3 has no typed builder for these, so the
x-amz-checksum-<algo> header is injected via mutate_request (value from
rustfs-rio, byte-for-byte identical to awscrt). Uses a client with automatic
checksum calculation disabled (request_checksum_calculation=WhenRequired) so the
injected header is the only checksum on the wire. For each of XXHash3/64/128,
SHA-512 and MD5: a correct value is accepted and the object stored intact; a
mismatched value is rejected with BadDigest and nothing is stored.
Verified passing locally (1 passed) alongside a boto3+awscrt round-trip that
additionally confirms the HEAD/GET header echo (14/14).
Refs rustfs/backlog#1262rustfs/backlog#1252
Co-Authored-By: heihutu <heihutu@gmail.com>
* style(get): allow too_many_arguments on finalize_get_object_response
The classifier refactor added an extra_checksum_headers parameter, pushing
finalize_get_object_response to 8 args and tripping clippy::too_many_arguments
under CI's `-D warnings`. Add the same #[allow] the sibling GET helpers already
carry; no behavior change.
Refs rustfs/backlog#1252
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
feat(targets): add an opt-in NATS JetStream publish path for the notify and audit targets
The NATS notify and audit targets publish through NATS Core, which returns
before the server has durably accepted the message. A broker restart or a
connection drop between the publish and the flush loses the event, even though
the send queue has already cleared it, and no acknowledgement gates that clear.
An opt-in JetStream publish path clears a queued event only after the server
returns a durable PublishAck, so delivery is at-least-once across a broker
restart or a reconnect. It applies to both the notify and audit NATS targets, is
off by default, and is byte-identical to the NATS Core path when disabled.
The path includes durable store-and-forward, a stable dedup id sent as the
Nats-Msg-Id header so a replayed event is collapsed by the stream duplicate
window, pre-flight stream validation, and a bounded failed-events store for
terminally-failed and retry-exhausted events. Three configuration keys per
target select it: JETSTREAM_ENABLE, JETSTREAM_STREAM_NAME, and
JETSTREAM_ACK_TIMEOUT_SECS, under the RUSTFS_NOTIFY_NATS_ and RUSTFS_AUDIT_NATS_
prefixes.
The on-disk batch filename separator changes from colon to underscore so
batch names are valid on Windows filesystems, with transparent read-back
of files written under the previous separator. The migration affects the
shared queue store for every target type and lands with this feature
because the store gains its first Windows-exercised paths here.
Co-authored-by: houseme <housemecn@gmail.com>
The pulsar config validation rejected any non-`pulsar+ssl` broker whenever
`tls_allow_insecure` was set or `tls_hostname_verification` was disabled.
Both toggles are inert on a plaintext `pulsar://` broker — the Pulsar client
only applies them for `pulsar+ssl` — so treating a non-default value as fatal
is wrong.
This surfaced as issue #4796: the console persists `tls_hostname_verification`
as `false` (the checkbox defaults to unchecked while the server default is
`on`). At creation time the value is not present in the unmerged request and
falls back to the safe default, so the connectivity check passes and the
target comes online. On restart the persisted config is merged and validated,
the `false` is read back, and the target is rejected — permanently offline
even though Pulsar is reachable.
Relax both the loader-side (`validate_pulsar_broker_config`) and runtime-side
(`PulsarArgs::validate`) checks so only a `tls_ca` bundle — real TLS trust
material that is never defaulted to a non-empty value — requires a
`pulsar+ssl` scheme. The inert toggles no longer fail the target. This also
heals configs already persisted with the bad value. Add regression coverage
for both paths.
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(deps): pin hyper to flush-before-shutdown fix for large-GET unexpected EOF
hyper <= 1.10.1 can call poll_shutdown() on an HTTP/1 socket while response
bytes are still buffered (a prior poll_flush() returned Poll::Pending and the
result was discarded), so a backpressured/slow-reading peer receives a graceful
FIN before the full Content-Length body is flushed. Standard S3 clients
(minio-go/warp) report this as sporadic `unexpected EOF` on large-object GET
under load (rustfs/backlog#1232). This is the transport-layer bug from
Cloudflare's "hyper-bug" writeup — distinct from the EC-reconstruct-desync EOF
already fixed via lockstep decode; here the app layer delivers the full body and
truncation is purely hyper->socket.
Fixed upstream in hyperium/hyper#4018 (commit 72046cc7, "fix(http1): flush
buffered data before shutdown"), which is not in any crates.io release yet as of
hyper 1.10.1. Pin hyper via [patch.crates-io] to git rev ccc1e850 (a descendant
of the fix that still declares version 1.10.1).
Use [patch.crates-io], not a git+rev on the workspace `hyper` dependency: the
server connection is driven by the transitive hyper-util (conn::auto /
GracefulShutdown), and a git source would not unify with the crates.io hyper
hyper-util resolves, leaving two hyper copies with the server path still on the
buggy one. The patch rewrites the crates.io source globally, so the lockfile
holds a single git-sourced hyper.
Add rustfs/tests/hyper_h1_shutdown_flush_regression.rs, a deterministic guard
(mirrors hyper's own h1_shutdown_while_buffered) that fails if the pin is dropped
or hyper is downgraded below the fix. Its load-bearing assertion is a
synchronously-set flag, so a slow runner can only under-detect, never false-red.
Closesrustfs/backlog#1232.
* build(deny): allow the hyperium/hyper git source for the flush-before-shutdown pin
cargo-deny's [sources] check denies unknown git sources. The hyper
[patch.crates-io] pin added for rustfs/backlog#1232 uses a github.com/hyperium
git source, so add it to allow-git with an owner/review note and a removal
condition (drop once a released hyper > 1.10.1 carries commit 72046cc7).
* fix(obs): open cleaner compression source with O_NOFOLLOW
The compressor opened the source log via File::open, which follows a
symlink at the final path component. Between the scanner selecting a
regular file and this open, an attacker with write access to the log
directory could swap the entry for a symlink (TOCTOU) pointing at, say,
/etc/shadow, whose contents would then be copied into an archive. Open
the source with O_NOFOLLOW on Unix so such a swap fails with ELOOP; the
temp/archive path already refused symlinks, this closes the source side.
Refs rustfs/backlog#1210
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(obs): recompress instead of trusting leftover cleaner archives
archive_header_ok only checked the first 2-4 magic bytes before treating
an existing .gz/.zst as a completed prior result and letting the caller
delete the source log. A file with valid magic but a truncated or forged
body passes that check, so an attacker with write access to the log
directory (or a crashed prior run) could plant such a stub and make the
cleaner delete the real log without ever producing a usable archive —
silent audit-data loss.
Chosen fix: stop trusting cross-process leftovers entirely and always
recompress the source in this pass, rather than fully decoding every
leftover to validate it. Full-decode validation would add real CPU cost
and decode-bug surface for a rare crash-recovery case; the existing
atomic create_new+rename already overwrites whatever sits at the archive
path (a planted symlink is replaced, never followed) with a freshly
written, fsync'd archive, so a partial/forged leftover can never gate
source deletion. This is the lowest-regression option.
Refs rustfs/backlog#1211
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(object-data-cache): cap memory-gate reservation at cache growth headroom
The memory gate subtracts `admitted_since_refresh` from the snapshot's
available bytes so a burst arriving faster than the 5 s refresh cannot
over-allocate. That counter is GROSS: it only rolls over on the refresh and
never rolls back when a fill is later evicted, cancelled, or loses the
invalidation race. Under sustained high-throughput churn (net footprint flat
and far below `max_capacity`) the raw counter balloons past the memory the
cache actually holds, so `effective_available` collapses and the gate reports
false memory pressure — skipping the hottest fills with SkippedMemoryPressure
until the next 5 s refresh. This only lowers hit rate; it never returns wrong
data and self-heals each refresh.
Fix direction 1 (minimal regression): cap the reservation deduction at the
cache's own growth headroom (`max_capacity - weighted_size()`) instead of
letting the unbounded gross counter shrink the system-available budget. The
cache can never hold more than `max_capacity`, so a burst adds at most that
headroom of real memory before moka evicts to stay bounded (net-zero churn
beyond that point) — capping the deduction there keeps the reservation honest
without treating gross churn as growth. Chosen over net-accounting (direction
2, releasing bytes on every failure/cancel/eviction path) because that only
plugs the leak on failed fills and would not address the core defect: churn of
*successful* insert/evict fills over the 5 s window. It also touches only the
gate plus one call site rather than every failure path in moka_backend.
The cap only ever raises `effective_available`, so real memory pressure (a low
snapshot at refresh) still suppresses fills; when the cache is at capacity the
headroom is 0 and the deduction vanishes, correctly reflecting net-zero churn.
`MokaBackend` now stores `max_capacity` and passes the live headroom into
`allows_fill`. Adds targeted gate tests: gross churn far above headroom no
longer falsely suppresses, yet the reservation still bounds a burst while the
cache can genuinely grow.
Refs rustfs/backlog#1212
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(ecstore): assert native O_DIRECT path runs in uring read test
uring_preserves_o_direct_for_eligible_reads only compared bytes through
LocalDisk::read_file_mmap_copy. On a filesystem that rejects O_DIRECT the
read silently degrades to the buffered StdBackend fallback and the byte
check still passes, so the test could go green without the native
read_at_direct path ever executing -- a vacuous pass.
Add a per-disk native_direct_reads counter on UringBackend, incremented
only when pread_uring_direct completes, and rebuild the test to drive a
real UringBackend's pread_bytes and assert the counter is non-zero (every
eligible read went through the native tier). When io_uring or O_DIRECT is
unavailable on the host filesystem (restricted CI runners, tmpfs), the
test skips loudly via eprintln instead of asserting a tautology, while
still checking byte-correctness on whatever tier served the read.
The counter also gives a gray release a positive signal that the O_DIRECT
tier is serving reads, not just a fallback count.
Refs rustfs/backlog#1213
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ecstore): warn + count read-time EINVAL on native O_DIRECT reads
classify_direct_read_error is only reached from the read side: the
O_DIRECT open in pread_uring_direct already succeeded (an open-time
refusal is handled earlier as DirectOpenError::ODirectRefused). So an
EINVAL/EOPNOTSUPP arriving here is a read-time error on an fd the kernel
accepted for O_DIRECT -- far more likely an alignment bug in the aligned
read path than an unsupported filesystem. The old code latched the disk's
native path off with only a once-per-disk debug trace, hiding a potential
correctness regression behind a silent buffered-read downgrade.
Diagnostics only: the fallback behaviour is unchanged (the native path is
still latched off and the caller still reads via StdBackend). This adds a
rustfs_io_uring_direct_read_einval_total counter and promotes the
once-per-disk trace from debug to warn so an operator can see an alignment
regression instead of an unexplained latency/CPU shift.
Refs rustfs/backlog#1214
Co-Authored-By: heihutu <heihutu@gmail.com>
* docs(ecstore): document data-blocks-first default and its tail-latency cost
DEFAULT_RUSTFS_GET_DATA_BLOCKS_FIRST_READER_SETUP is true and must stay
true: deferred-parity is the deliberate, already-rolled-out full-object
GET default from backlog#1159/#923. Flipping it back to false in code
would silently revert that rollout for every deployment that has not set
the env var, so this commit only documents -- no behaviour change.
The added notes explain what data-blocks-first does (schedule data shards
up front, engage parity lazily on a missing/corrupt data shard), the known
trade-off (parity is engaged late, so a slow-but-not-dead data drive
raises GET p99 because the faster parity shards are not raced against it
until a data shard is declared missing), and the operational rollback
switch (RUSTFS_GET_DATA_BLOCKS_FIRST_READER_SETUP=false), which is
intentionally an env override rather than a code default change.
No metric was added: the low-risk observability hook for "slow data drive
engaged deferred parity" would live at the deferred-stripe engage point,
which is out of this file's scope; this change stays documentation-only to
avoid touching the hot GET path.
Refs rustfs/backlog#1215
Co-Authored-By: heihutu <heihutu@gmail.com>
* docs(ecstore): document wide-directory walk stall hazard and tuning
list_dir enumerates a whole directory in one os::read_dir call (count =
-1), and the walk caller bounds that entire enumeration with the per-read
stall budget (default 5s) as if it were a single read. For a wide, flat
prefix -- one directory holding millions of immediate children -- a single
readdir can exceed the budget on a healthy disk, trip DiskError::Timeout,
and surface as a ListObjects 500 quorum failure though the drive is fine
(a #2999 sub-class).
This is documented, not rewritten: turning the one-shot readdir into a
streaming/batched enumeration that refreshes the stall deadline between
chunks is an architecture-level change with high regression surface
(ordering, the count contract, quorum merge) and belongs in a separate
follow-up. The supported mitigation today is operational, so the comments
point wide-directory deployments at RUSTFS_DRIVE_WALKDIR_STALL_TIMEOUT_SECS
and the high-latency drive-timeout profile, which widen the budget with no
code change. Notes were added at list_dir, the scan_dir call site, and
get_drive_walkdir_stall_timeout. No behaviour change.
Refs rustfs/backlog#1216
Co-Authored-By: heihutu <heihutu@gmail.com>
* docs(ecstore): document consumer-peek vs producer-stall coupling
In list_path_raw the consumer's peek_timeout is drawn from the same source
and same value (walkdir_stall_timeout, default 5s) as the producer-side
walk stall budget, but the two measure different things: the producer
stall bounds a single drive read, while the consumer peek bounds the gap
between two ADJACENT entries arriving from a reader. Because they share a
value, the consumer cannot wait meaningfully longer for the next entry
than the producer is allowed to spend producing one. Walking a region
dense with non-listable internal items can make a HEALTHY drive miss the
budget between visible entries; the consumer then declares it stalled and
detaches it, dropping a good drive from the merge and capping the "large
prefix succeeds" guarantee.
Documented, not decoupled: giving the consumer peek an independent,
strictly-larger budget would cut these false detaches but equally delays
detaching a genuinely dead drive and shifts listing tail-latency
semantics, so it wants soak data before changing the default. The comment
records the invariant any such follow-up must keep -- consumer peek >=
producer stall, never stricter -- so it can never fail a drive before the
producer would. No behaviour change.
Refs rustfs/backlog#1217
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(io-metrics): add time-based trigger for low-IOPS latency percentiles
Percentiles were recomputed only every 128 IOs and seeded to 0, so a
low-traffic deployment exported p95/p99 = 0/stale for a long time after
startup. Add a 10s wall-clock trigger alongside the count throttle so the
first recompute can fire before 128 samples accrue. Hot-path per-op mean
update is unchanged.
Refs rustfs/backlog#1218
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(e2e): cover codec-streaming parity under fault injection and NoSuchKey
The codec-streaming compat A/B previously ran only against a healthy
4-disk EC set with successful full GETs: the DiskFaultHarness was
constructed but never faulted, the error path was untested, and the
range assertion silently compared legacy-vs-legacy (ranges always fall
back to the duplex path), overstating what it proved.
Add two genuinely-failable scenarios reusing the existing harness and
fixtures:
- Parity reconstruction A/B: take one data disk offline and re-run the
full object matrix on both phases while the EC 2+2 set rebuilds each
large object from the surviving shards. Assert codec == legacy
byte-for-byte (sha256) and header-for-header, and assert the codec
phase served the reconstructed objects with zero duplex-pipe fallback
(the reader gate is drive-health-independent, so the codec fast path
is really exercised through reconstruction).
- NoSuchKey negative path: compare the HTTP status + S3 error code of a
missing-key GET across the legacy and codec phases and require them to
be identical (404/NoSuchKey), guarding against the codec env
perturbing the error path.
Also clarify the range-phase comment so it is not misread as
codec-range correctness coverage: both sides are served by the same
legacy range path, so the assertion only proves ranges keep working and
keep falling back to legacy with the gates open.
Verified: cargo check/--no-run pass and the test passes locally
(1 passed; dup_codec=0 confirms the codec path ran).
Refs rustfs/backlog#1219
Co-Authored-By: heihutu <heihutu@gmail.com>
* ci(ecstore): exercise native O_DIRECT read path on an ext4 loopback
The uring-integration leg ran on the runner's default TMPDIR, which may sit
on tmpfs/overlayfs where open(O_DIRECT) fails and the native read_at_direct
path silently latches off to the aligned StdBackend fallback. Mount a
dedicated ext4 loopback and point TMPDIR at it so the real io_uring dep
(bumped git->0.1.0->0.2.0->0.2.1) and the native O_DIRECT read path are
actually covered rather than validated only by signature diffing.
Refs rustfs/backlog#1220
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
PR #4356 wired `reclaim_orphan_data_dirs` only into `heal_object`'s
post-heal tail, which runs after the `disks_to_heal_count == 0` early
return. That early return is exactly the state of the objects the sweep
targets: a valid `xl.meta` with all shards present plus a leaked
pre-#3510 data dir needs no shard healing, so a healthy heal returned
before reclaim and swept nothing. On a healthy deployment (single node,
no degraded disks) the reclaim was therefore dead code — an admin heal
walked the objects, "healed" them, and reclaimed no leaked space.
Run the best-effort reclaim on the `disks_to_heal_count == 0` path as
well, gated on `!opts.dry_run`. The shared match+log block is factored
into `reclaim_orphan_data_dirs_best_effort` so both exits behave
identically. A reclaim failure still never fails the heal.
Adds an end-to-end regression: put a healthy non-inline object, plant an
unreferenced UUID data dir under it on every disk that holds the object,
then drive `heal_object`. A dry-run heal must leave the stray in place; a
real heal must reclaim it while preserving the live data dirs, `xl.meta`,
and object contents. The test fails against the pre-fix control flow.
Refs #3231, #3191, #4356.
Co-authored-by: heihutu <heihutu@gmail.com>
Follow-up to #4772. After forcing every-cycle ILM evaluation the Days=1
plateau is reliable, but test_lifecycle_expiration / test_lifecyclev2_expiration
still flaked intermittently on the *second* assertion (`assert 4 == 2`):
the Days=5 (expire3) objects were not expired within their poll window.
Cause: _wait_for_lifecycle_count for expire3 starts its N*lc_interval
deadline only after the Days=1 poll returns (~1 debug-day in). With N=5
and lc_interval == debug_day, the 5*debug_day terms cancel and the slack
past the Days=5 due time is only ~1 debug-day (~9s) -- which a single slow
scanner cycle (observed ~13s spacing under CI load) can exceed, leaving the
count stalled at 4. Bumping the two expire3 windows to 8*lc_interval raises
the slack to ~39s, comfortably above the observed cycle jitter.
Test-only, lane-scoped: does not touch RUSTFS_ILM_DEBUG_DAY_SECS or the
4*lc_interval < 5*debug_day plateau invariant. The expire3 target count (2)
is terminal (nothing expires after it), so a wider window can never
over-expire. Also documents the #4772 RUSTFS_DATA_USAGE_UPDATE_DIR_CYCLES=1
knob in lifecycle_behavior_tests.txt.
chore(ci): guard against committed planning docs; remove re-added ones
PR #4771 removed the docs/superpowers planning-doc archive and added a rule,
but #4765 force-added two more (git add -f bypasses .gitignore):
docs/superpowers/plans/2026-07-12-observability-single-writer.md and
docs/superpowers/specs/2026-07-12-observability-single-writer-design.md — an
agentic implementation plan and its design spec. Delete both.
Close the enforcement gap so this cannot recur:
- New scripts/check_no_planning_docs.sh fails if anything is tracked under
docs/superpowers/, regardless of how it was added.
- Wire it into make pre-commit/pre-pr/dev-check.
- Run it in CI: ci.yml for code/mixed PRs, and ci-docs-only.yml (which was a
green stub) so docs-only PRs can no longer slip a planning doc past the
required 'Test and Lint' check.
- Document the guard in AGENTS.md and the arch-checks skill.
* fix(obs/cleaner): fsync archive and log dir before deleting source logs
OLC-01: the compression path flushed the BufWriter but never synced the
archive data or the parent directory before renaming, and the source was
then unlinked with no durability barrier. A crash after rename but before
the page cache reached disk could leave a truncated/zero-length archive
while the source was already gone — permanent log/audit data loss. Because
the archive is always renamed to a brand-new name (guaranteed by the
existing exists() guard), ext4 auto_da_alloc does not mask this.
Hand the underlying File back from the writer closure, sync_all() it before
rename, fsync the parent directory, and fsync the log directory after the
unlinks so a delete cannot be reordered ahead of the archive it justified.
Guard the temp file with an RAII cleanup so an early return or panic cannot
leak a *.tmp orphan.
Ref: rustfs/backlog#1194 (audit rustfs/backlog#1193)
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(obs/cleaner): create compression temp file with O_EXCL and O_NOFOLLOW
OLC-03: the temp archive was opened with File::create at a predictable
`<source>.gz.tmp` path with no O_EXCL/O_NOFOLLOW, so an actor with write
access to the log directory could pre-plant that path as a symlink and have
the compressor follow it — truncating and overwriting an arbitrary external
file, then chmod-ing it to the source log's mode. This mirrors the symlink
refusal already enforced on the deletion path (secure_delete).
Route temp creation through create_tmp_archive(), which uses create_new
(O_CREAT|O_EXCL) to refuse a pre-existing entry and, on Unix, O_NOFOLLOW to
refuse a symlink at the final path component.
Ref: rustfs/backlog#1196 (audit rustfs/backlog#1193)
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(obs/cleaner): create compression temp file with restrictive mode
OLC-06: File::create left the temp archive world-readable (0644 & ~umask)
for the entire duration of compressing a large log, exposing the full
plaintext of a possibly-0600 audit log on shared hosts until the mode was
copied only after the write completed. Pass the source mode into
create_tmp_archive and open the temp file with it (default 0600) so it is
restrictive from creation; the post-write chmod still tightens/matches the
source mode exactly.
Ref: rustfs/backlog#1199 (audit rustfs/backlog#1193)
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(obs/cleaner): validate existing archive before skipping recompression
OLC-02: the idempotency guard used Path::exists() (which follows symlinks)
and trusted whatever it found, then the caller deleted the source. A planted
`<archive>.gz` symlink, or a zero-length/truncated archive left by a crashed
run (OLC-01), would green-light deleting the source with no valid backup —
data loss / log destruction.
Replace exists() with symlink_metadata (no follow) and only treat the entry
as a completed prior result when it is a regular, non-empty file whose header
matches the codec magic (gzip 1f 8b / zstd 28 b5 2f fd). Anything else falls
through to recompression, whose atomic create_new+rename replaces the bad
entry (a symlink is replaced, never followed or deleted through).
Ref: rustfs/backlog#1195 (audit rustfs/backlog#1193)
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(obs/cleaner): stop dry-run overstating reclaimed bytes for compression
OLC-08: in dry-run, compress_with_writer returned output_bytes = 0, so
projected_freed_bytes = input and delete_files reported the full input as
freed. A real run keeps the archive on disk (freed = input - archive), so
dry-run overstated reclaim by the whole archive footprint. Estimate the
archive with a deliberately conservative ratio so the projection never
exceeds what a real run reclaims.
Ref: rustfs/backlog#1201 (audit rustfs/backlog#1193)
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(obs/cleaner): make freed-byte accounting resilient; document steal metric
OLC-12: input/output byte sizes were read via metadata().unwrap_or(0), which
silently reports 0 on failure and skews freed-byte metrics (input - 0 = full
input, overstating reclaim). Use the copy() byte count as the authoritative
input size and, when the archive metadata read fails, conservatively assume
no savings instead of 0. Also document that the steal_success_rate counts
only victim steals (batch = one success), so it reads as a relative
rebalancing signal, not absolute task acquisition.
Ref: rustfs/backlog#1205 (audit rustfs/backlog#1193)
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(obs/cleaner): preserve level-0 semantics, allow zstd 22, log effective levels
OLC-09: build() and the codec calls clamped gzip/zstd levels to [1,9]/[1,21],
silently rewriting gzip level 0 (store) and zstd level 0 (codec default) to 1
and blocking the legal zstd maximum of 22. Clamp to [0,9]/[0,22] so those
meanings survive, and echo the effective (post-clamp) levels in the startup
log via new effective_gzip_level()/effective_zstd_level() getters so the log
matches what actually runs.
Ref: rustfs/backlog#1202 (audit rustfs/backlog#1193)
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(obs/cleaner): bound compressed archives by byte cap; warn on retention=0
OLC-04: archive expiry was gated on compressed_file_retention_days > 0, so
retention=0 disabled it entirely while compression kept producing archives,
and max_total_size_bytes only ever bounded uncompressed logs — unbounded disk
growth. Replace select_expired_compressed with select_archives_to_delete,
which applies age expiry (when retention is on) and, regardless of retention,
trims the oldest archives until the set fits under max_total_size_bytes. Also
warn at startup when compression is on with retention=0 so the "keep forever"
semantics are not mistaken for "delete immediately".
Ref: rustfs/backlog#1197 (audit rustfs/backlog#1193)
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(obs/cleaner): warn on invalid exclude glob instead of dropping silently
OLC-05: build() dropped unparseable exclude globs via filter_map(...ok()), so
a typo (or a literal comma splitting a char-class in the config string) turned
"protect this file" into "delete this file" with no signal. Log a warning per
rejected pattern with the raw string and parse error so the misconfiguration
is visible.
Ref: rustfs/backlog#1198 (audit rustfs/backlog#1193)
Co-Authored-By: heihutu <heihutu@gmail.com>
* perf(obs/cleaner): backoff idle workers, cap worker count, lower small-host floor
OLC-11: the work-stealing loop re-spun on Steal::Retry with no yield and used
yield_now on the empty path, burning CPU during redistribution windows;
worker_count had no upper bound so a mis-set parallel_workers over a directory
of thousands of logs could spawn thousands of threads; and
default_parallel_workers forced >=4 workers even on 1-2 vCPU hosts. Use
crossbeam_utils::Backoff (spin->yield, reset on work) on the idle paths, clamp
worker_count to MAX_PARALLEL_COMPRESS_WORKERS, and lower the default floor to 1.
Ref: rustfs/backlog#1204 (audit rustfs/backlog#1193)
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(obs/cleaner): warn when active-file guard is disabled by empty filename
OLC-13 (defense-in-depth): the scanner protects the live log purely by exact
filename equality against active_filename. An empty active_filename silently
disables that protection, so a non-empty file_pattern could make the live log
a deletion candidate via the public builder. Warn in build() when that unsafe
combination is configured. The audit's "never delete the newest match"
structural guard is intentionally not implemented: it would conflict with the
legitimate keep_files=0 semantics (purge all rotated logs). The naming
contract is instead locked by regression tests (OLC-14).
Ref: rustfs/backlog#1206 (audit rustfs/backlog#1193)
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(obs/cleaner): warn on unknown algorithm/match_mode, echo match_mode
OLC-07: from_config_str silently fell back to defaults for unrecognized
compression algorithm and match mode (any non-"prefix" value became Suffix),
hiding operator typos like "prefixx" that could make the cleaner match no
rotated logs. Warn on a non-empty unrecognized value in both parsers, and
echo the resolved match_mode in the startup log alongside the algorithm.
Ref: rustfs/backlog#1200 (audit rustfs/backlog#1193)
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(obs/cleaner): derive orphan .tmp suffixes and exempt them from min age
OLC-10: orphan `*.gz.tmp`/`*.zst.tmp` cleanup was gated by min_file_age_seconds
(default 3600), so crash-left orphans lingered up to an hour, and the tmp
suffix list was hardcoded rather than derived from compressed_suffixes() — a
new codec would leave `*.<ext>.tmp` orphans the scanner never recognizes.
Derive the temp suffix from CompressionAlgorithm::compressed_suffixes(), and
gate orphan removal on a small fixed grace window instead of min_file_age
(orphans are never live-written after the rename that would promote them).
Ref: rustfs/backlog#1203 (audit rustfs/backlog#1193)
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(obs/cleaner): cover symlink, archive expiry, idempotency, and edge cases
OLC-14: add regression tests for the previously-untested safety/correctness
branches — symlink rejection (external target never deleted), archive age
expiry vs fresh retention, archive byte-cap trim with retention disabled,
gz/zst classification, max_single_file_size selection, min_age protecting a
fresh non-empty log, active-file exclusion when the active name also matches
the pattern, invalid exclude glob not aborting build, dry-run + compression
creating no archive, gzip round-trip validity, and the idempotent-archive
branch trusting a valid prior archive.
Ref: rustfs/backlog#1207 (audit rustfs/backlog#1193)
Co-Authored-By: heihutu <heihutu@gmail.com>
* style(obs/cleaner): apply rustfmt and collapse nested if (clippy)
Formatting-only cleanup over the audit fix series: rustfmt normalization of the
multi-line expressions introduced in compress.rs/core.rs, plus collapsing the
delete_files directory-fsync into a single let-chain to satisfy
clippy::collapsible_if. No behavior change.
Co-Authored-By: heihutu <heihutu@gmail.com>
* refactor(obs/cleaner): collapse redundant source stats in compression path
The per-issue fixes to compress_with_writer accumulated three metadata()
syscalls on the source in the real compression path: an input_bytes read that
was immediately shadowed by the copied byte count (a dead read), a source_mode
read (OLC-06), and the pre-OLC-06 post-write chmod re-reading the same mode.
Collapse to a single fd-based read — move the dry-run input_bytes read into
the dry-run branch, read source_mode from the already-open fd (no path stat,
no TOCTOU), and reuse it for the post-write chmod. Behavior is unchanged
(same inode's mode, written for input size); 3 source stats -> 1.
Co-Authored-By: heihutu <heihutu@gmail.com>
* chore(obs/cleaner): fix typo flagged by CI (mis-set -> misconfigured)
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
* Change Rust toolchain channel to stable
Signed-off-by: houseme <housemecn@gmail.com>
* style: apply clippy --fix and cargo fix lint suggestions
Run `cargo clippy --fix --all-targets --all-features` and
`cargo fix --lib --all-targets` across the workspace, then resolve the
remaining warnings by hand:
- collapse needless borrows in `format!` args, prefer `?` over explicit
early returns, and use `.values()` / `.flatten()` iterator adapters
- rewrite the `Md5` scan loop via `manual_flatten` and re-indent the
`select!` macro body (rustfmt skips macro interiors)
- annotate the intentional dead-code `Md5` inherent methods (constructed
only by the test factory) with `#[allow(dead_code)]`
Behavior is unchanged.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(obs): prevent rolling log stdout aliasing
* fix(ecstore): bound hot-path tracing payloads
* test(logging): guard service and disk log invariants
* fix(obs): silence useless_conversion on st_dev for Linux clippy
rustix's Stat.st_dev is u64 on Linux/glibc, making u64::try_from a no-op
that trips clippy::useless_conversion under -D warnings. The conversion is
still needed on macOS/BSD where st_dev is a signed dev_t, so suppress the
lint on that line rather than dropping the portable fallible conversion.
* fix(logging): keep stdout sink validation portable (#4769)
* fix(obs): keep stdout device conversion portable
* docs(logging): update single-writer plan status
---------
Co-authored-by: overtrue <anzhengchao@gmail.com>
Co-authored-by: houseme <housemecn@gmail.com>
The s3-tests lifecycle expiration cases (test_lifecycle_expiration,
test_lifecyclev2_expiration, test_lifecycle_deletemarker_expiration)
flaked with counts stalling one plateau behind (e.g. `assert 6 == 4`).
Root cause: a compacted directory is only re-descended -- and its
objects re-evaluated against ILM rules -- once every
DATA_USAGE_UPDATE_DIR_CYCLES scanner cycles (default 16). At the lane's
accelerated RUSTFS_SCANNER_CYCLE=2 that is ~32s between evaluations, so
a Days=1 object due at debug_day (10s) is not actually expired until the
next ~32s boundary (~42s) -- just past the test's 4*lc_interval (40s)
poll window, so the list still returns the pre-expiry count.
Set RUSTFS_DATA_USAGE_UPDATE_DIR_CYCLES=1 for this debug-only lane so
compacted directories are re-descended every cycle and ILM fires within
~2s of the due time, comfortably inside the poll window. This does not
touch the 4*lc_interval < 5*debug_day plateau invariant.
docs: remove agent-generated planning docs, forbid committing them
Delete one-shot planning/progress artifacts that were checked into the tree:
the 14 superpowers plan/tracker docs under docs/superpowers/plans/, plus
issue-scoped implementation plans, optimization conclusions, and dated
benchmark-result snapshots under docs/ (issue-4003 ListObjectsV2 plans,
get-small-file conclusion, issue824/issue829 benchmark results, issue-713
>1GiB GET baseline summary and ops guide).
Codify the rule so they do not come back:
- .gitignore drops the docs/superpowers whitelist, so anything new under
docs/ stays ignored unless force-added.
- AGENTS.md gains an explicit 'do not commit planning-type documents' rule
scoping version control to the durable architecture/operations/testing sets.
- docs/architecture/README.md, overview.md, arch-checks SKILL.md, and
check_doc_paths.sh drop their references to the removed archive.
The scheduled e2e-s3tests sweep failed at "Wait for RustFS ready" in both
topologies because the server never started (issue #4762).
Two independent startup faults, both surfaced now that the local
physical-disk-independence guard is enforced:
- single: RUSTFS_VOLUMES=/data/rustfs{0...3} all live on one physical
device on the runner, so the guard aborts startup. Set the
CI-sanctioned RUSTFS_UNSAFE_BYPASS_DISK_CHECK=true (what the guard's own
error message and the e2e tests already use).
- multi: the entrypoint's process_data_volumes skipped every non-absolute
entry, so the distributed URL form
"http://rustfs{1...4}:9000/data/rustfs{0...3}" never created
/data/rustfs0..3. LocalDisk init then aborts with VolumeNotFound because
resolve_local_disk_root no longer auto-creates the disk root. This also
broke the shipped .docker/compose/docker-compose.cluster.yaml for real
distributed docker deployments.
entrypoint.sh now:
1. Expands multiple {N...M} ranges per token (the URL form carries two).
The previous single-pass expander collapsed it to "http://rustfs1"
and dropped the disk path; it now re-scans until no ranges remain,
operating on only the first brace to keep multi-range tokens intact.
2. Creates the local filesystem path for URL-form endpoints (stripping
scheme://host:port) without appending them as CLI args — rustfs reads
the distributed form from RUSTFS_VOLUMES directly.
Absolute-path and default (/data) inputs expand byte-identically to before.
The multi compose also gets the CI disk-check bypass, since the four disks
share one device inside each node's container.
fix(cache): drain pending removals until entry_count reaches zero in clear()
The previous clear() implementation used a single run_pending_tasks()
call after invalidate_all(), which was insufficient under concurrent
fill pressure — moka processes invalidations lazily in batches, so
entries can linger after a single maintenance pass.
Replace the fixed single call with a drain loop (up to 256 rounds) that
calls run_pending_tasks() and yields between iterations until
entry_count() reaches zero. This ensures the concurrency storm test
(moka_backend_concurrency_storm_leaves_no_leaked_state) passes reliably.
The earlier fix (#4759) added a pre-invalidate_all drain and an 8-pass
loop but reordered operations in a way that introduced a new race. This
commit keeps the original invalidate_all-first ordering and only adds
the drain loop after the initial run_pending_tasks() call.
Zhengchao An