Groundwork for encrypted-object replication (backlog#1783, PR-A of 3):
- classify_replication_source_encryption: accept the AES256 marker that
every stored SSE-C object carries; the SseC arm was unreachable.
- Fail closed on sealed material without an SSE marker (MinIO-written
objects) instead of replicating ciphertext as plaintext.
- Replace the dead VALID_SSE_REPLICATION_HEADERS table with a transport
map keyed by the metadata keys the SSE writer actually persists, shared
via the new rustfs_utils::http::object_encryption_keys module.
- Structurally strip all encryption metadata from outbound replication
(x-rustfs-encryption-* envelopes previously passed the filters).
- Skip decrypt_checksums for encrypted objects at the boundary so its
is_multipart=false (a response-path contract) cannot misroute
encrypted multipart objects once managed replication opens.
- Redact X-Rustfs-Replication-* SSE transport values in FileInfo Debug.
A reconciliation test pins that every key encryption_material_to_metadata
produces is either transport-mapped or stripped. All four SSE replication
e2e contracts still assert FAILED unchanged.
* fix(build): support non-Linux Unix targets (illumos/Solaris/*BSD)
Two independent build-infrastructure blockers kept RustFS from building on
non-Linux Unix platforms. Neither touches runtime logic.
1. pulsar regenerates its protobuf bindings in build.rs on every build, which
needs `protoc`. Platforms without a packaged protoc (illumos/Solaris/*BSD)
now enable pulsar's `protobuf-src` feature via a cfg-gated dependency, which
builds a vendored protoc from C++ sources. Mainstream targets keep the lean
dependency and their existing system/CI protoc.
2. clocksource 0.8.3 (pulled in transitively by ratelimit 0.10) used the
Linux-only `CLOCK_MONOTONIC_COARSE`. ratelimit 2.0 dropped the clocksource
dependency entirely, so upgrading removes the portability problem at the
root rather than patching clocksource. The bandwidth throttle's bulk
`consume()` is rewritten onto ratelimit 2.0's `try_wait_n`, preserving the
best-effort partial-consumption semantics.
Verified: cargo check + bandwidth monitor unit tests pass; cargo tree confirms
protobuf-src is enabled only for illumos/Solaris/*BSD and clocksource is gone
from the graph. The final illumos build must be confirmed on-platform.
Closes#3195
* fix(ecstore): guard ratelimit v2 capacity overflow
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(ecstore): avoid slow bandwidth reader timeout
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(targets): drop vendored pulsar protobuf build
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(filemeta): classify xl.meta CRC mismatch as FileCorrupt so heal repairs it
A failed CRC means the metadata bytes on disk are not the bytes that were
written — bitrot. Raising it as Error::other() surfaces a generic Io error,
which should_heal_object_on_disk does not recognise as heal-worthy: the drive
is skipped, disks_to_heal_count stays 0, heal_object returns ok, and the
corrupted xl.meta is never rewritten — while the scanner re-submits the same
no-op heal every deep-scan cycle. An explicit admin deep heal fails the same
way, so no heal path repairs metadata bitrot, and every one of them reports
success.
check_xl2_v1 already classifies a short or wrong-magic header as FileCorrupt
for exactly this reason (#5716); this completes the pattern for the two CRC
sites. The existing From<rustfs_filemeta::Error> for DiskError conversion maps
the variant to DiskError::FileCorrupt, which the heal path already handles.
The previously silent is_indexed_meta site now logs the mismatch (structured
event shape) like unmarshal_msg does.
Regression test: corrupt one byte of a marshalled FileMeta and assert
unmarshal_msg reports FileCorrupt; fails on the previous code, which returned
Io(Other).
Verified end-to-end on a 3-node / 12-drive EC:4 cluster: xl.meta corrupted on
2 of 12 drives via dd, admin deep heal — before this change the heal returns
ok with the corruption intact and the scanner loops forever; with it, both
copies are rewritten (decode-identical to the healthy quorum), the object
reads back byte-correct, and a follow-up heal reports all twelve drives
clean.
* test(filemeta): cover crc heal classification
Add regression coverage for the indexed xl.meta CRC path and the metadata-heal decision that consumes FileCorrupt.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: terem42 <9478806+terem42@users.noreply.github.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
A failed CRC means the metadata bytes on disk are not the bytes that were
written — bitrot. Raising it as Error::other() surfaces a generic Io error,
which should_heal_object_on_disk does not recognise as heal-worthy: the drive
is skipped, disks_to_heal_count stays 0, heal_object returns ok, and the
corrupted xl.meta is never rewritten — while the scanner re-submits the same
no-op heal every deep-scan cycle. An explicit admin deep heal fails the same
way, so no heal path repairs metadata bitrot, and every one of them reports
success.
check_xl2_v1 already classifies a short or wrong-magic header as FileCorrupt
for exactly this reason (#5716); this completes the pattern for the two CRC
sites. The existing From<rustfs_filemeta::Error> for DiskError conversion maps
the variant to DiskError::FileCorrupt, which the heal path already handles.
The previously silent is_indexed_meta site now logs the mismatch (structured
event shape) like unmarshal_msg does.
Regression test: corrupt one byte of a marshalled FileMeta and assert
unmarshal_msg reports FileCorrupt; fails on the previous code, which returned
Io(Other).
Verified end-to-end on a 3-node / 12-drive EC:4 cluster: xl.meta corrupted on
2 of 12 drives via dd, admin deep heal — before this change the heal returns
ok with the corruption intact and the scanner loops forever; with it, both
copies are rewritten (decode-identical to the healthy quorum), the object
reads back byte-correct, and a follow-up heal reports all twelve drives
clean.
* fix(ecstore): purge the stale destination data dir on healing rename_data commits
Heal commits reuse the version's existing data_dir, so when repairing
in-place corruption (bitrot) the destination directory still exists and
holds the corrupt shard files. rename(2) cannot replace a non-empty
directory (EEXIST on XFS, ENOTEMPTY on ext4), so the commit failed on
every attempt — including all scheduler retries — and in-place bitrot was
detected and reconstructed but never repaired.
Purge the stale destination data dir (move_to_trash) before the commit
rename, for healing commits only: fresh PUTs mint a new data_dir and can
never collide, and a non-healing collision keeps failing loudly. Adds the
FileInfo::is_healing() reader for the marker set_healing() already writes.
* style(ecstore): emit the heal purge failure as a structured event
The new warning was the only sentence-style log in `rename_data`'s commit
path — it sat ten lines above `info!(event = EVENT_DISK_LOCAL_RENAME_REJECTED,
component = ..., subsystem = ...)` and interpolated its values into the
message instead of carrying them as fields, so it is invisible to any operator
query keyed on `event`.
Give it the shape the rest of the file uses: a named
`EVENT_DISK_LOCAL_HEAL_PURGE_FAILED`, `component`/`subsystem`, `dst_path` and
`error` as fields, and a short label as the message. Level stays `warn` — the
purge is best effort and the rename below fails closed — and the condition,
the branch, and the control flow are unchanged.
---------
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
The RustFS event shape (`event`/`component`/`subsystem`/`result` + context,
message last) is specified only in
`.agents/skills/rustfs-logging-governance/SKILL.md`, and nothing routes a
change to it:
- `AGENTS.md`, which is what an agent actually loads by default, never
mentions logging. Its only related line is "log unknown fields at `warn`"
under Serde Safety, which is about level, not shape.
- The skill's `description` says "use when editing or reviewing RustFS logs",
so a bugfix that adds one log line in passing — how most new log sites enter
this repo — never matches it.
- `scripts/check_logging_guardrails.sh` is a blocklist: 500+ `rg -F` literals
that retire log lines which already shipped. It cannot see a newly written
one. For `crates/ecstore/src/disk/local.rs` the only check is that
`#[tracing::instrument]` is TRACE-only; `warn!`/`info!` shape is unchecked.
PR #5822 landed `warn!("heal rename_data: purging ... {:?} failed: {}", ...)`
in `disk/local.rs` — sentence-style, no fields, directly beside `info!(event =
EVENT_DISK_LOCAL_RENAME_REJECTED, component = ..., subsystem = ...)` — with
every check green. That is the gap, not an authoring mistake.
Close all three:
- `AGENTS.md`: a Logging section stating the field shape, the level policy,
the reuse-the-file's-constants rule, and that it applies to any `tracing`
macro added in passing, not only to log-focused changes.
- Skill `description`: trigger on adding or editing any `tracing` macro,
naming the single-line-added-in-passing case explicitly.
- Guardrail: assert the event shape positively on the already-governed disk
files — `error!`/`warn!`/`info!` must open with fields or a `target:`, never
a bare string. Commented-out macros are excluded; `debug!`/`trace!` stay out
of scope as targeted diagnostics. Self-test fixtures cover both directions.
`crates/ecstore/src/disk/mod.rs` carried the one live violation in that file
set (`conv_part_err_to_int`), so it is converted here; the guardrail would
otherwise fail on an untouched file.
Verification:
- `./scripts/check_logging_guardrails.sh` — passes
- Negative control: re-inserting PR #5822's exact `warn!` line into
`disk/local.rs` makes it exit 1 pointing at that line
- `cargo fmt -p rustfs-ecstore -- --check`, `cargo check -p rustfs-ecstore`
* refactor(kms): share the DEK spec mapping and stop re-parsing opened envelopes
- generate_key_material is now the single spec->length mapping for every
backend that mints DEKs itself; the inline copies in the Static and Local
backends are gone, and ChaCha20 (32 bytes, same as AES_256) is accepted
uniformly instead of only by Static.
- The pub(crate) client decrypt of the Local, Vault KV2 and Vault Transit
backends returns (plaintext, master_key_id), so KmsBackend::decrypt no
longer re-parses the envelope it just opened (one JSON parse per SSE GET
instead of two, and unknown-field observability is no longer double-counted).
- Malformed-envelope parse failures now report CryptographicError("parse")
on all backends; Local was the last one mapping them to SerializationError.
- The four KmsBackend::generate_data_key adapters take fields out of
DataKeyInfo instead of cloning, dropping a redundant un-zeroized plaintext
DEK copy and a full ciphertext clone per call; a missing plaintext now
fails closed everywhere instead of returning an empty key on three of four
backends.
* test(kms): pin legacy header fallback, stored-AAD, and decrypt key-id contracts
- a_legacy_aws_kms_object_without_the_cipher_header_still_opens rebuilds the
true pre-internal-header shape (aws:kms mode + S3 key-id header, no
x-rustfs-* headers) and asserts the fallback normalizes the cipher and
re-projects it.
- a_rewritten_sse_c_context_header_fails_authentication is the SSE-C flank of
the stored-AAD tamper check; metadata_without_stored_context_bytes_still_opens
covers the derived-AAD path for both flavours and pins the seal side to the
canonical bytes (mutation-verified).
- data_key_spec_controls_the_length_of_the_generated_key requires every
backend in the matrix to honour all three specs, asserts the envelope
records the requested spec, and round-trips each blob.
- corrupt_ciphertext_fails_cleanly pins unparseable ciphertext to
CryptographicError instead of merely not-InternalError.
- Deleted the never-called assert_validation_error / assert_cryptographic_error
helpers.
The write-side filter is_reserved_user_metadata_key only namespaced
x-amz-, x-rustfs-internal- and x-minio-internal- keys, while the
read-side should_skip_object_metadata_key also strips
x-rustfs-encryption-* / x-minio-encryption-* as internal. A client PUT
of x-amz-meta-x-rustfs-encryption-algorithm therefore landed on disk as
the bare internal key x-rustfs-encryption-algorithm, which the KMS
headers_to_metadata path treats as the preferred cipher selector. Not
exploitable today (the production decrypt path discards the parsed
algorithm and FromStr rejects invalid values), but any future wiring of
headers_to_metadata into decryption would hand cipher choice to the
client.
Reserve both encryption prefixes on the write side so client-supplied
keys are namespaced under x-amz-meta- like other reserved keys, hoist
the prefix constants to module scope shared with the read-side skip
logic, and pin the attack form (header injection and CopyObject REPLACE
metadata), the bare-header form, and the legitimate server-written SSE
metadata flow with regression tests.
#5804 added the on-disk segment budget to check_bucket_and_object_names, but PUT validates through check_put_object_args, which has its own checks and never calls it. An over-NAME_MAX key therefore still reached the disk layer and came back to the client as ENAMETOOLONG → InternalError 500, exactly the behavior #5785 reported.
Caught by re-running the acceptance suite against the locked build 4b2d79f5d, which contains #5804: S3-003 still failed with a 512-byte key.
Multipart is unaffected — check_new_multipart_args and check_multipart_object_args both route through check_object_args → check_bucket_and_object_names, which already carries the budget.
Verification: new test pins the same boundaries on check_put_object_args (255 ok / 256 rejected, byte-based via CJK, multi-segment long keys ok, __XLDIR__ budget for directory keys); cargo test -p rustfs-ecstore --lib -- bucket::utils 17 passed; cargo clippy -p rustfs-ecstore --all-targets clean; make pre-commit green.
* fix(admin): serve the usage a scan measured instead of blanking the whole snapshot
query_data_usage_info_with_store replaced the entire DataUsageInfo with the default empty shape whenever the persisted snapshot did not cover every currently listed bucket. A freshly created bucket is by definition absent from the last completed scan, so every bucket creation zeroed out usage reporting for the whole deployment until a cycle covered it (rustfs#5806).
Instrumented on a single node (fresh data dir, 10 PUTs into one new bucket, polling the admin API every second while watching the on-disk documents): the scanner persisted correct usage within ~6s — the authoritative document held scanner_cycle=2, objects_total_count=10 and the bucket's entry — while the API kept answering with a default-constructed DataUsageInfo (scanner_cycle: None) for roughly another minute.
Narrow the snapshot to what it measured instead of discarding it. Buckets the scan never reached stay absent from buckets_usage, which already means unknown on the wire and stays distinct from a present zero, and the response is marked usage_snapshot_converged = Some(false) so clients can tell it is not the whole namespace. The protections that motivated the blanking are kept: a structurally incomplete snapshot is still dropped, and so is one that measured nothing the namespace still contains. Buckets deleted since the scan are now dropped from the response rather than lingering.
The old data_usage_snapshot_covers_namespace predicate has no callers left and is removed along with the test that pinned its all-or-nothing behavior; the new test covers the partial, full, stale-bucket, incomplete and empty-namespace cases.
Verification: cargo test -p rustfs --lib -- admin_usecase (24 passed), cargo clippy -p rustfs --lib clean, make pre-commit green.
* fix(admin): drop the orphaned test attribute left by the removed coverage test
Removing data_usage_snapshot_covers_namespace's test left its #[test] behind, which then attached to the following test as a duplicate attribute. Local 'cargo clippy -p rustfs --lib' does not build the test target, so it only surfaced in CI's --all-targets lane.
A superseded cycle is the expected outcome of the dirty-usage fast path, not a signal of pathological load: a write burst marks buckets dirty, the scanner wakes within milliseconds, and the still-landing writes then supersede the snapshot it just took. Charging that first race SUPERSEDED_RETRY_BASE_INTERVAL = 60s meant the burst surfaced in usage and quota accounting roughly two cycles late.
Measured on an idle single-node instance (fresh data dir, 10 PUTs, polling /rustfs/admin/v3/datausageinfo every 5s): the dirty-usage wake fires 0.3s after the PUTs, its cycle is superseded 0.2s later, and the retry was then scheduled 55.6s out; usage first became visible at t+120s. With the base at 5s the retry is scheduled 4.7s out and usage becomes visible at t+70s.
The exponential growth in retry_interval is what protects against a persistently hot bucket driving an unbroken full-scan loop, so the base does not need to be a whole cycle: 5s, 10s, 20s, 40s ... still reaches minute-scale backoff within a handful of consecutive supersedes and keeps the SUPERSEDED_RETRY_MAX_INTERVAL cap. A configured cycle shorter than the base still wins, since retrying faster than the operator's own cadence buys nothing.
Verification: cargo test -p rustfs-scanner --lib (444 passed) with the three superseded-backoff tests updated to the new schedule; make pre-commit green; end-to-end probe above.
Implement a local AsyncFileReader over DataFusion's object store re-export so Parquet metadata loading no longer uses the deprecated ParquetObjectReader adapter.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
terem42