mirror of
https://github.com/rustfs/rustfs.git
synced 2026-07-26 08:18:18 +00:00
1fac7a5871c89f959c2b926d581eed8b4110173e
677 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 houseme
|
1fac7a5871 |
chore(deps): refresh workspace dependencies (#5067)
* chore(deps): refresh workspace dependencies Refresh compatible workspace dependencies while preserving the requested version pins. Update async-nats and Hyper, and replace the temporary Hyper Git patch with the released crate. Co-Authored-By: heihutu <heihutu@gmail.com> * chore(deps): bump hotpath to 0.21.5 Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
cxymds
|
eeafc355d4 |
feat(heal): define fenced control wire envelopes (#4997)
* fix(rpc): bind internode auth to exact targets * fix(heal): initialize the runtime atomically * fix(heal): aggregate status across cluster nodes * fix(heal): return canonical tokens for duplicate starts * feat(heal): add authenticated control RPC contract * feat(heal): gate control capability by cluster topology * feat(heal): define fenced control wire envelopes * fix(heal): return canonical tokens for duplicate starts (#4992) --------- Co-authored-by: Zhengchao An <anzhengchao@gmail.com> |
||
|
houseme
|
c92e99ba95 |
chore(deps): refresh workspace dependencies (#5044)
Update compatible workspace dependencies and refresh the lockfile while keeping ratelimit pinned. Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
GatewayJ
|
8ebedddfa1 |
fix(s3select): reject truncated object streams (#5027)
* fix(s3select): reject truncated object streams * fix(s3select): validate raw stream before conversion |
||
|
Zhengchao An
|
7f5873dac8 |
fix(ecstore): resolve erasure parity per pool (#4801) (#5015)
* fix(ecstore): add fallible erasure construction (cherry picked from commit |
||
|
Zhengchao An
|
3ed682be42 |
feat: offline log fault-analysis system (rustfs diagnose) (#4876)
* feat(log-analyzer): add crate skeleton and unified event model
Implements LA-1 (rustfs/backlog#1282) of the log fault-analysis system
(rustfs/backlog#1281): new synchronous rustfs-log-analyzer crate with the
LogEvent/LogLevel/SourceRef/EventKind/ParseStats model shared by all
later stages. No tokio, no rustfs-* internal deps by design.
Note: thiserror listed in the issue is deferred until a stage actually
defines error types (LA-3/LA-4) to avoid an unused dependency.
* feat(log-analyzer): add line parsing layer
Implements LA-2 (rustfs/backlog#1283): four parse channels tried in order
per line — native tracing JSON, container-prefix stripping (K8s CRI /
docker compose / journald) with JSON retry, multi-line Rust panic block
folding (both pre- and post-1.65 formats, stderr has no JSON logger), and
a plain-text fallback that never fails. Parse accounting feeds the report
parse-ratio disclosure.
* feat(log-analyzer): add ingest layer for directories and archives
Implements LA-3 (rustfs/backlog#1284): expands customer inputs (files,
directories, zip/tar/tar.gz/.zst/.gz, stdin-like readers) into parsed
events. Magic-byte detection with extension fallback, recursive archive
walking with depth/entry/byte/memory caps (every capped input disclosed
in IngestReport.skipped), first-level directory names become node labels,
and nothing is ever extracted to disk so hostile entry paths are inert.
Adds tar 0.4 to workspace deps (sync; the async astral-tokio-tar used by
rustfs-zip does not fit this crate's no-tokio contract).
* feat(log-analyzer): add rule model, matching engine, and finding aggregation
Implements LA-4 (rustfs/backlog#1285): owned serde-round-trippable Rule/
Matcher/Severity types (external JSON rules deserialize into the same
types later), fail-fast RuleSet validation that reports every problem at
once, a linear-scan engine with regexes compiled once, and an
order-independent FindingsCollector (commutative aggregates only; the
test asserts byte-identical output across shuffled input orders).
* feat(log-analyzer): add built-in seed rule library (68 rules, 12 categories)
Implements LA-5 (rustfs/backlog#1286): the 2026-07 repository-wide
failure-log survey distilled into rules across disk health, erasure/
bitrot, quorum, network/RPC, distributed locks, heal, scanner, IAM,
startup/config/TLS, capacity, decommission/rebalance, and process panics.
Every anchor was verified verbatim against the source tree (94/94 hits,
zero corrections needed). Quorum rules pre-fill implies_root_cause for
the Phase-2 folding (rustfs/backlog#1290); client-side rules carry burst
thresholds (min_count) so isolated client mistakes don't clutter reports.
Tests: one realistic positive sample per rule (table-driven), exact-set
smoke samples including the intentional internode/client signature
double-hit, and negative cases.
* feat(log-analyzer): add analysis orchestration, report rendering, and redaction
Implements LA-6 (rustfs/backlog#1287): a single-pass Analyzer that does
rule matching, minute-bucket timelines (gap-filled, merged to <=60
buckets), unmatched WARN/ERROR template clustering (placeholders for
numbers/uuids/paths/addresses/quotes, 5000-template cap disclosed as
<overflow>), mixed-UTC-offset detection, and below-min_count demotion to
a low-confidence section. Renderers: pipe-friendly terminal text, stable
JSON (schema_version=1), and ticket-pasteable Markdown. --redact hashes
customer identifiers (stable h:sha256[..8]) in samples/evidence/messages
while keeping rule ids, targets, and panic locations intact.
* feat(rustfs): add 'rustfs diagnose' subcommand for offline log fault analysis
Implements LA-7 (rustfs/backlog#1288): wires rustfs-log-analyzer into the
main binary as a diagnose subcommand that short-circuits before
observability/storage init (same pattern as 'info' / 'tls inspect') so
the report on stdout is never wrapped by the JSON logger.
rustfs diagnose <paths>... [--format text|json|md] [--since 24h]
[--until ...] [--min-level warn] [--redact] [--top N] [--samples N]
Accepts files, directories, archives (.zip/.tar/.tar.gz/.zst/.gz) and '-'
for stdin. Exit codes: 0 = diagnosis completed (findings never fail the
process), 2 = bad arguments / no readable input.
diagnose_e2e covers the six MVP acceptance scenarios from
rustfs/backlog#1281 (directory+zst archive, multi-node zip attribution,
CRI-prefixed kubectl logs, panic folding, stable JSON schema, CLI parsing
incl. the legacy 'rustfs <volume>' preprocessor regression); the
full-binary smoke test is #[ignore]d (run with -- --ignored).
Usage doc: docs/operations/log-diagnose.md.
* ci(log-analyzer): guard rule anchors against log-message drift
Implements LA-8 (rustfs/backlog#1289): every seed-rule anchor must exist
verbatim in the rustfs source tree, so changing a log message without
updating its rule fails the gate instead of silently killing the rule.
- la-dump-anchors bin emits 'rule_id<TAB>anchor' TSV;
- scripts/check_log_analyzer_rules.sh greps each anchor (fixed-string,
*.rs only, excluding crates/log-analyzer itself to avoid self-matches);
- RuleSet::new now rejects anchors that are blank, contain tab/newline,
or are shorter than 8 bytes (no discriminating power); the '[FATAL]'
anchor gained its trailing space to meet the floor while still matching
the emit_fatal_stderr format string;
- wired as log-analyzer-rules-check into the pre-pr gate (it compiles the
crate, so it stays out of the fast pre-commit set).
Negative self-test: breaking an anchor makes the script exit 1 naming the
rule ('MISSING anchor for rule inconsistent-drive: zzz-not-exist-anchor').
* refactor(log-analyzer): use root-relative provenance for directory inputs
Binary smoke run showed report samples citing full absolute paths, which
drowns the useful part. Directory inputs now label sources as
"<root-name>/<relative-path>" (e.g. "smoke-logs/node1/rustfs.log");
archives and single files keep their existing provenance.
* chore(log-analyzer): reword comment to satisfy the typos gate
* fix(log-analyzer): declare chrono serde feature locally after workspace feature localization
* fix(log-analyzer): bound line reads so a newline-less input cannot bypass the byte cap
read_until grew the line buffer with the entire remaining stream before the
max_total_bytes check ran, so a single multi-GB line (decompression bomb or
corrupt file) could allocate unboundedly. Replace it with a capped reader that
enforces the remaining global budget chunk-by-chunk and adds a per-line cap
(IngestOptions::max_line_bytes, default 1 MiB); over-cap tails are discarded
but still charged, and truncation is disclosed once per file as the new
line_too_long skip reason. Flagged by Codex review on #4876.
* fix(log-analyzer): redact field-shaped identifiers inside message text and widen the hash to 64 bits
--redact only hashed IPv4 literals in unstructured message text, so
bucket/object/access-key values embedded in messages (access_key=AK123,
'bucket: media, object: private/a.bin') leaked into reports documented as
safe to forward. Apply the SENSITIVE_FIELDS list to key=value / key: value
shapes in message text with the same hash as structured fields, and extend
the hash from 8 to 16 hex chars so cross-identifier collisions stay
negligible. Flagged by Codex and Copilot review on #4876.
* fix(log-analyzer): strip collector prefixes before panic-block absorption
An open panic block tested continuation lines against absorbs() before their
CRI/compose/journald prefix was stripped, so containerized panics stored the
prefix in the payload and split into a truncated panic plus text noise as soon
as the note/backtrace lines arrived. Stripping now happens once at the top of
feed() and every channel judges the payload. Flagged by Codex review on #4876.
* fix(log-analyzer): remove two input-order dependencies in representative selection
The unmatched-cluster target stayed pinned to the first-seen event while the
representative sample could be replaced, so sample and target could come from
different events and vary with input order; the pair now updates together by
lexicographic (sample, target) min. Sample selection tie-broke on line number
alone, which is only unique within one file; the key now includes the source
file. Flagged by Copilot review on #4876.
* fix(rustfs): reject negative relative times in diagnose --since/--until
parse_time_arg accepted "-24h" and produced a future timestamp, contradicting
the documented 'counted back from now' semantics. The amount now parses as
unsigned, so a leading '-' fails with the usual invalid-time error. Flagged by
Copilot review on #4876.
* feat(log-analyzer): Phase 2 — causal folding, timeline anomalies, external rules (LA-9) (#4942)
* feat(log-analyzer): collapse cascade symptoms under their root-cause finding
Phase-2 sub-item A (rustfs/backlog#1290): a finding whose rule declares
implies_root_cause edges folds under a qualifying root — root.first_seen <=
symptom.first_seen + 5min and root.last_seen >= symptom.first_seen - 30min,
existence-based when either side has no timestamps (pure stderr panics).
Findings gain collapsed_into/caused; text/markdown render the root block with
an indented cascade line and stop listing collapsed symptoms flat, while JSON
keeps every finding. Roots are promoted to the most severe position among
their block so the report top still answers 'the most likely cause'.
* feat(log-analyzer): detect timeline/clock anomalies (schema v2)
Phase-2 sub-item B (rustfs/backlog#1290): three deterministic hints rendered
between the summary and findings — mixed UTC offsets (with a clock-skew note
when signature-mismatch findings coexist), per-node time ranges that do not
overlap at all (both nodes >100 timestamped events), and log gaps of at least
max(15min, 3x bucket width) after >=3 consecutive active minutes, upgraded to
restart evidence when a startup-class finding begins within 5min after the
gap. JSON gains timeline_anomalies and schema_version bumps to 2.
* feat(diagnose): load external rules with --rules <file.json>
Phase-2 sub-item C (rustfs/backlog#1290): an external JSON rule file
({schema_version: 1, rules: [Rule...]}, the exact serde shape of the built-in
rules) merges over the seed library, with same-id rules replacing built-ins so
the support team can hotfix a misfiring rule without a release. The merged set
validates as a whole and any problem (bad regex, duplicate id, empty matcher
group, wrong schema version) prints every error and exits 2 — analysis never
runs on a half-broken set. External anchors are exempt from the CI anchor
guard, documented as author-owned quality. Adds the custom-rules section to
docs/operations/log-diagnose.md.
* fix(log-analyzer): address PR #4876 review (redaction coverage, order-independence, guards)
Redaction (--redact) now honours its "forwardable" intent across every report surface instead of a 15-name field whitelist applied over a subset:
- redact_event scrubs the full fields map (sensitive names hashed whole, every other value run through redact_text) plus provenance, so the JSON/Markdown full-sample dump no longer leaks non-whitelisted fields (client_ip, url, user, ...).
- node labels are hashed once at ingestion, so summary.nodes, per-node timeline ranges, samples and timeline anomalies stay consistent and correlatable under one stable hash.
- evidence values, unmatched-cluster templates and skipped-input paths are now redacted; peer/disk/drive/volume/node/user added to the sensitive set; IPv6 literals are hashed (without touching `rust::paths` or HH:MM:SS clocks); provenance keeps the leaf filename and hashes the customer directory/archive prefix.
- redact.rs and docs/operations/log-diagnose.md reworded to "best-effort identifier scrubbing", not an anonymization guarantee.
Order-independence (the crate's headline contract):
- the evidence value cap keeps the lexicographically smallest N distinct values instead of the first-N-by-arrival (previously order-dependent).
- first_seen/last_seen break equal-instant ties on the offset, so the serialized RFC3339 offset no longer depends on input order.
Parsing:
- a new-format panic header no longer swallows the line immediately after it when that line is itself a JSON event or a second panic header (previously dropped an interleaved ERROR in merged stdout/stderr, or merged a panic-during-panic); trailing "note: ..." backtrace lines now fold into the block.
CLI:
- diagnose --since/--until reject absurd relative amounts via checked_sub_signed instead of panicking; the exit-code doc now matches actual behaviour.
CI:
- check_log_analyzer_rules.sh is wired into the ci.yml test-and-lint job (it was only in make pre-pr, so anchor drift from other PRs could merge green).
Markdown table cells escape '|' so customer log text cannot break the table structure.
---------
Co-authored-by: houseme <housemecn@gmail.com>
|
||
|
houseme
|
15f4e75870 |
fix(cache): harden object data cache coordination (#5004)
* fix(cache): enforce projected entry capacity Refs: rustfs/backlog#1335 Co-Authored-By: heihutu <heihutu@gmail.com> * fix(cache): fence identity budget eviction by generation Refs rustfs/backlog#1334. Co-Authored-By: heihutu <heihutu@gmail.com> * fix(cache): fence clear against concurrent fills Refs rustfs/backlog#1333 Co-Authored-By: heihutu <heihutu@gmail.com> * fix(cache): linearize memory reservation claims Co-Authored-By: heihutu <heihutu@gmail.com> * fix(cache): retain allocation memory claims Co-Authored-By: heihutu <heihutu@gmail.com> * fix(cache): publish memory snapshots by epoch Co-Authored-By: heihutu <heihutu@gmail.com> * fix(cache): coordinate cold object fills Co-Authored-By: heihutu <heihutu@gmail.com> * fix(ecstore): fence metadata cache transition races Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
houseme
|
79509aad2d |
chore(deps): refresh workspace dependencies (#5002)
Update direct dependency constraints and refresh the lockfile while keeping ratelimit pinned. Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
Henry Guo
|
361334ab08 |
chore(deps): sync merged s3s SigV4 fix (#4987)
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com> |
||
|
Zhengchao An
|
87128682b0 | fix(ecstore): recover data usage snapshots safely (#4979) | ||
|
Zhengchao An
|
f67e8a6cdc | chore(release): prepare 1.0.0-beta.10 (#4946) | ||
|
Zhengchao An
|
caf42018b1 |
fix: pin tokio to 1.52.3 and fix stale doc path reference (#4931)
fix(deps): pin tokio to 1.52.3 to fix dial9-tokio-telemetry build tokio 1.52.4 made private, breaking dial9-tokio-telemetry 0.3.14 which references it as public. Pin tokio back to 1.52.3 until the upstream dependency fixes compatibility. Also fix a stale doc path reference in unified-object-generation.md where is a planned file that does not exist yet. |
||
|
houseme
|
75c3403dcc |
chore(release): prepare 1.0.0-beta.10-preview.4 (#4908)
Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
houseme
|
07e643cac2 |
chore(release): prepare 1.0.0-beta.10-preview.1 (#4898)
Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
houseme
|
48b328d0d2 |
chore(deps): tighten crate dependency features (#4896)
* chore(deps): tighten crate dependency features Narrow Tokio and dependency feature declarations for protocols, TLS runtime, utils, targets, and replication based on direct crate usage. Co-Authored-By: heihutu <heihutu@gmail.com> * chore(deps): trim hyper-rustls features Keep direct hyper-rustls features aligned with the actual RustFS call sites. rustfs-targets only needs native root loading and the rustls provider/TLS policy features for MQTT TLS config construction, while rustfs-ecstore needs the HTTP connector protocol features but not webpki roots. Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
houseme
|
56179210ab |
chore(deps): simplify dependency features (#4890)
* chore(deps): remove redundant dependency features Remove manifest feature entries that are implied by other requested features in the same dependency declaration. Verified that the resolved Cargo feature graph is unchanged after the cleanup. Co-Authored-By: heihutu <heihutu@gmail.com> * chore(deps): narrow tokio and reqwest features Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
houseme
|
f3a7a4b0da |
chore(deps): localize workspace dependency features (#4888)
Move workspace-level dependency feature lists into the member crates that consume each dependency while keeping required default-features flags at the workspace root. Also refresh starshard to 2.2.2 via cargo update and cargo upgrade --exclude ratelimit. Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
houseme
|
0fa6dc5946 |
ci: switch Linux builds to custom runners (#4884)
Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
Zhengchao An
|
299b739f9f | fix(replication): stop active-active replay loops (#4878) | ||
|
houseme
|
5ef2731a6b |
ci: tune build workflow runners (#4871)
* chore: extend build timeout and trim dev deps * ci: switch linux builds to ubicloud runners * ci: use larger linux build runners |
||
|
houseme
|
081c10f073 | chore(deps): trim s3select datafusion features (#4869) | ||
|
Zhengchao An
|
a7827ed91b |
test(policy): add property tests for the policy evaluation algebra (#4840)
crates/policy had zero property tests: the wildcard Action/Resource matching and Deny-first evaluation in Policy::is_allowed — the algebra authorization safety rests on — were guarded only by example-based tests. Add crates/policy/tests/policy_eval_proptest.rs with three generated-input invariants (pure evaluation, no IO or global state, parallel-safe): - explicit_deny_anywhere_denies: a Deny matching the request denies it no matter how many broad Allow statements co-exist or at which index the Deny sits; a built-in sanity assertion first proves the Allows alone would permit, so the Deny is what flips the decision. - wildcard_superset_implies_concrete_match: any request allowed by an exact-action/exact-resource policy is also allowed after widening to s3:* on bucket/* and to * on arn:aws:s3:::*, checked both on the built grant and on random probe requests (implication form). - empty_policy_denies_everything: a statement-less policy and Policy::default() deny every non-owner request. Statements are built from JSON exactly as production policies arrive via PutPolicy. Generated names avoid wildcard metacharacters so the only wildcards in play are the ones the properties introduce deliberately. proptest joins crates/policy dev-deps following the filemeta/ecstore precedent. Refs: backlog#1151 (sec-9) |
||
|
houseme
|
6d528414ee |
chore(deps): update workspace dependencies (#4865)
Run `cargo update` followed by `cargo upgrade --exclude ratelimit` on the latest main to pull the newest compatible versions. Manifest bumps (Cargo.toml): - redis 1.3.0 -> 1.4.0 - xxhash-rust 0.8.16 -> 0.8.17 Lockfile-only bumps: simd-adler32 0.3.10, syn 2.0.119, and toml_edit 0.25.13. `ratelimit` is held at 0.10.1 as requested. Verification: cargo check --workspace --all-targets, cargo clippy --all-targets -D warnings (plus the rio-v2 feature variant), cargo nextest run --all --exclude e2e_test (8964 passed), and cargo deny check advisories/sources/bans/licenses all pass. Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
Zhengchao An
|
f05a69d51b |
test(utils): add rustfs-test-utils crate and shared ECStore bootstrap (#4850)
* test(utils): add rustfs-test-utils crate, absorb heal/iam ECStore bootstrap
backlog#1153 infra-1. The ~50-line "build a real temp-disk ECStore"
bootstrap was copy-pasted (and drifting) across the heal and iam
integration tests. This adds crates/test-utils (rustfs-test-utils, a
dev-dependency-only crate) owning that bootstrap and converts the four
copies into thin wrappers:
- TestECStoreEnvBuilder: disk_count (default 4), prefix (uuid-suffixed
/tmp dir), base_dir (caller-owned dir, e.g. tempfile::TempDir),
init_bucket_metadata (default true; the iam bootstrap test opts out
to preserve its historical semantics). TestECStoreEnv exposes
temp_root/disk_paths/ecstore plus a versioned-bucket helper, and
init_tracing() replaces the per-file Once blocks.
- All rustfs_ecstore imports stay behind src/ecstore_test_compat.rs,
the sanctioned test-compat boundary pattern (mirrors
crates/iam/tests/ecstore_test_compat).
- heal: heal_integration_test / heal_b5_versioned_regression_test /
heal_b920_subquorum_union_test drop their setup_test_env{,_n} copies
for heal_env{,_n} wrappers; the tests/storage_api.rs integration
surface shrinks to what test bodies still touch.
- iam: iam_bootstrap_no_lock_test drops build_local_ecstore; its
ecstore_test_compat fixture shrinks to SetupType +
update_erasure_type.
rg 'async fn setup_test_env' crates/heal crates/iam now returns 0.
Scanner's lifecycle tests are deliberately NOT absorbed (gated on
ilm-1; 14 of 15 are #[ignore]d today). Net -230 lines.
* fix(heal): drop tokio::fs import orphaned by the b920 bootstrap move
* fix(heal): drop tokio::fs import orphaned by the b5 bootstrap move
|
||
|
Zhengchao An
|
6ea6832aef |
test(lifecycle): add property-based coverage for rule evaluation (#4824)
Co-authored-by: houseme <housemecn@gmail.com> |
||
|
Henry Guo
|
a6a0e29282 |
fix(table-catalog): support Spark REST commits (#4788)
* fix(table-catalog): support Spark REST commits * chore(deps): update s3s SigV4 revision --------- Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com> |
||
|
houseme
|
83fe12d6aa |
chore(release): prepare 1.0.0-beta.9 (#4807)
* chore(release): prepare 1.0.0-beta.9 Co-Authored-By: heihutu <heihutu@gmail.com> * chore(release): align release assets for 1.0.0-beta.9 Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
houseme
|
750e5d15eb |
feat(checksums): add native S3 additional checksum support (#4805)
* feat(rio): wire XXHash3/64/128 and SHA-512 into ChecksumType (S2) Add the AWS 2026-04 additional checksum algorithms as base types in rustfs-rio's ChecksumType, covering every dispatch site (key, raw_byte_len, hasher, Display, from_string_with_obj_type, BASE_CHECKSUM_TYPES) so no path silently strips them. Derive BASE_TYPE_MASK from BASE_CHECKSUM_TYPES as the single source of truth, allocate the new base-type bits append-only above bit 9 to preserve the on-disk varint format, and add streaming hashers whose digest uses the S3 canonical big-endian encoding (seed 0). The new algorithms are COMPOSITE-only: an explicit FULL_OBJECT request is rejected and they are never routed through add_part()/can_merge(). A round-trip guardrail test asserts every base type survives all dispatch sites, failing loudly if a future algorithm is added but a match arm or the mask is forgotten. Refs rustfs/backlog#1254 rustfs/backlog#1252 Co-Authored-By: heihutu <heihutu@gmail.com> * test(rio): pin XXHash/SHA-512 digests to official vectors, big-endian (S3) Lock the byte order and seed of the new algorithms against the OFFICIAL upstream xxHash / SHA-512 empty-input test vectors (XXH3-64, XXH64, XXH3-128, SHA-512), in big-endian, so the stored and echoed checksum is byte-for-byte identical to what AWS SDKs (awscrt) compute — the interop correctness this feature hinges on. Add a non-empty regression lock (official "fox" vectors) that also asserts the encoded field is the standard-base64 of the raw digest. Refs rustfs/backlog#1255 rustfs/backlog#1252 Co-Authored-By: heihutu <heihutu@gmail.com> * test(rio): lock on-disk checksum round-trip and forward-compat degrade (S8) Cover the xl.meta varint (de)serialization for the new algorithms: to_bytes() -> read_checksums() must recover the value under the Display key for XXHASH3/64/128 and SHA512. Pin the rolling-upgrade contract that a node reading a future, unknown base-type bit degrades safely — skips the entry and returns without panicking or mis-decoding a length. Combined with the append-only bit allocation from S2, this protects mixed-version clusters. Refs rustfs/backlog#1260 rustfs/backlog#1252 Co-Authored-By: heihutu <heihutu@gmail.com> * feat(head): echo XXHash/SHA-512 additional checksums on HeadObject (S5) HeadObject with x-amz-checksum-mode: ENABLED now returns the XXHash3/64/128 and SHA-512 checksums that S3 stored, closing the head_object gap in #4800. s3s HeadObjectOutput has no typed field for these, so they are emitted as raw response headers via response.headers (the same mechanism RustFS already uses for tagging-count), keyed by ChecksumType::key(). The existing five typed algorithms are unchanged. Also carries the Cargo.lock update for the xxhash-rust dependency introduced in S2. Refs rustfs/backlog#1257 rustfs/backlog#1252 Co-Authored-By: heihutu <heihutu@gmail.com> * fix(checksums): fail-closed on unknown checksum algorithm (S7) A. Harden unknown/unsupported checksum algorithms to fail closed instead of panicking. ChecksumMode::base() in the outbound S3 client (crates/ecstore/src/client/checksum.rs) previously did `panic!("enum err.")` for any mode without a concrete base algorithm (e.g. a bare ChecksumFullObject flag); it now falls back to ChecksumNone. Added unit tests proving base() never panics and hasher() returns Err for unsupported modes. rustfs-checksums FromStr already returns Err on unknown names; added a regression test asserting garbage/unknown names fail closed. B. Extend rustfs-checksums ChecksumAlgorithm with the AWS 2026-04 additional algorithms Sha512/Xxhash3/Xxhash64/Xxhash128. Updated FromStr, as_str, into_impl, name constants, the x-amz-checksum-* header constants and the HttpChecksum impls. Byte order/seed matches the server-side rustfs-rio spec: xxh3/xxh64 as u64 big-endian (8 bytes, seed 0), xxh128 as u128 big-endian (16 bytes), sha512 via sha2::Sha512. Added tests validating each digest against a direct library computation. MD5 stays intentionally rejected (PR #4513) and is left untouched. C. crates/ecstore/src/client/checksum.rs ChecksumMode is enumset repr="u8" with 7 variants already consuming 7 bits; adding the 4 new algorithms would overflow u8 and require a breaking repr change, so ChecksumMode is left unchanged. The new algorithms are available through the rustfs-checksums ChecksumAlgorithm path. Refs rustfs/backlog#1259 rustfs/backlog#1252 Co-Authored-By: heihutu <heihutu@gmail.com> * feat(get,put): echo XXHash/SHA-512 checksums on GetObject and PutObject (S5-GET, S4) Complete the additional-checksum round-trip so AWS SDKs can verify integrity on download and confirm it on upload: - GetObject with x-amz-checksum-mode: ENABLED now returns XXHash3/64/128 and SHA-512 checksums (the download-side path SDKs auto-verify). The values flow from build_get_object_checksums through GetObjectOutputContext into finalize_get_object_response and are emitted after wrap_response_with_cors. - PutObject echoes the server-computed additional checksum on its response, captured at the want_checksum set points before opts is moved. Both reuse a single centralized helper, inject_additional_checksum_headers, which HeadObject now also uses. This is the ONLY place that emits these headers, so when s3s gains typed fields for these algorithms the migration is one spot (fill the typed field, drop the insert) with no risk of duplicate headers. The five s3s-typed algorithms are unchanged. Trailing-checksum PUT echo (value lands after the body) is left for e2e coverage in S10. Refs rustfs/backlog#1257 rustfs/backlog#1256 rustfs/backlog#1252 Co-Authored-By: heihutu <heihutu@gmail.com> * feat(multipart): support XXHash/SHA-512 composite multipart checksums (S9) Make multipart uploads work end-to-end for the composite-only algorithms (XXHash3/64/128, SHA-512): - complete_part_checksum previously returned the outer None for any algorithm outside the five typed ones, which failed CompleteMultipartUpload with InvalidPart. It now accepts any valid base type with no double-check value (Some(None)) — mirroring the missing-value path of the typed algorithms — since s3s CompletePart has no field to carry a client-supplied per-part value and the part was already verified server-side at UploadPart. Genuinely unset/invalid types are still rejected. - The existing COMPOSITE assembly (Checksum::new_from_data over the concatenated per-part raw digests; full_object_requested() is false so add_part() is correctly bypassed) already works for these algorithms via the S2 wiring. A rio test locks the assembly and that add_part refuses them. - UploadPart and CompleteMultipartUpload echo the new-algorithm checksum on their responses via the shared inject_additional_checksum_headers helper (now pub(crate)), since s3s has no typed output field. Refs rustfs/backlog#1261 rustfs/backlog#1252 Co-Authored-By: heihutu <heihutu@gmail.com> * feat(rio): add MD5 as an additional checksum (x-amz-checksum-md5) (S6) Wire MD5 into ChecksumType as an additional (flexible) checksum, distinct from the legacy Content-MD5 / ETag path: header x-amz-checksum-md5, 16-byte digest, COMPOSITE-only, md-5 hasher. Pinned to the official empty-input MD5 vector. Thanks to the single-source-of-truth wiring from S2, every dispatch site (GetObject/HeadObject/PutObject echo, multipart complete_part_checksum and the COMPOSITE assembly) picks MD5 up automatically via base()/key()/the catch-all arm — no handler changes needed. Tests are extended to cover MD5 across them. Coordination with #4513: that PR made the OUTBOUND rustfs-checksums client reject "md5" so it could never silently fall back to CRC32. This change is on the server-side rio path and never falls back — it implements MD5 correctly rather than substituting another algorithm — so the #4513 intent is preserved, and the outbound client keeps rejecting md5 (S7). Refs rustfs/backlog#1258 rustfs/backlog#1252 Co-Authored-By: heihutu <heihutu@gmail.com> * perf(rio): drop per-request to_uppercase alloc in checksum parsing (S11) from_string_with_obj_type ran alg.to_uppercase() on every checksummed request, allocating a String just to compare against a fixed set of algorithm names. Replace it with eq_ignore_ascii_case, which is allocation-free and, for the ASCII algorithm names involved, exactly equivalent. A test locks that case-insensitivity, the CRC64NVME full-object assumption, composite-only FULL_OBJECT rejection, and unknown/empty handling are all unchanged. The other S11 notes are intentionally not acted on: the Phase-0 header scan is N/A (we chose full support over rejection, so there is no reject guard), and parallelizing the serialized hash passes is deferred pending a measured need. Refs rustfs/backlog#1263 rustfs/backlog#1252 Co-Authored-By: heihutu <heihutu@gmail.com> * refactor(checksums): collapse 5 duplicated response-checksum loops into one Review of the accumulated commits found the same "iterate decrypted checksums, match five typed algorithms, drop the rest" loop copy-pasted across five response paths (GetObject, HeadObject, GetObjectAttributes object-level and part-level, CompleteMultipartUpload). That was patch-on-patch duplication. Collapse it into a single source of truth: - rustfs-rio gains ChecksumType::is_s3s_typed() — the one place that defines the five-typed vs additional-algorithm split. - object_usecase gains ResponseChecksums + classify_response_checksums(), which performs the typed/extra split once. All five call sites now destructure its result; additional_checksum_echo_pairs() also uses is_s3s_typed() instead of a hand-rolled five-way comparison. Behaviour is unchanged (GetObjectAttributes still cannot surface the additional algorithms — an s3s XML-body limitation, now documented in one spot). One pass over the map; extra pairs pushed only when a new-algorithm checksum is present. Refs rustfs/backlog#1252 Co-Authored-By: heihutu <heihutu@gmail.com> * test(checksums): unit tests for classifier/echo helpers + fix unused import Add direct unit tests for the refactored single-source-of-truth helpers: - rio ChecksumType::is_s3s_typed() — exhaustive typed-vs-additional split, and that flags (FULL_OBJECT/MULTIPART) on a base type don't change classification. - object_usecase classify_response_checksums() — typed fields vs `extra` headers, the checksum-type marker, and empty input. - additional_checksum_echo_pairs() — echo pair only for additional algorithms, none for the five typed ones, none for None. - inject_additional_checksum_headers() — writes all pairs; empty is a no-op. Also drop the now-unused AMZ_CHECKSUM_TYPE import in multipart_usecase.rs left by the classifier refactor (would fail the -D warnings gate). Refs rustfs/backlog#1252 Co-Authored-By: heihutu <heihutu@gmail.com> * style(rio): fix typo flagged by CI (mis-decoding -> decoding a wrong length) The Typos CI check flagged "mis-decoding" (it reads "mis" as a word). Reword the S8 forward-compat comment; no code change. Refs rustfs/backlog#1260 Co-Authored-By: heihutu <heihutu@gmail.com> * test(e2e): integration test for XXHash/SHA-512/MD5 additional checksums (S10) Permanent verify-on-write integration test in the e2e suite for the AWS 2026-04 additional algorithms. aws_sdk_s3 has no typed builder for these, so the x-amz-checksum-<algo> header is injected via mutate_request (value from rustfs-rio, byte-for-byte identical to awscrt). Uses a client with automatic checksum calculation disabled (request_checksum_calculation=WhenRequired) so the injected header is the only checksum on the wire. For each of XXHash3/64/128, SHA-512 and MD5: a correct value is accepted and the object stored intact; a mismatched value is rejected with BadDigest and nothing is stored. Verified passing locally (1 passed) alongside a boto3+awscrt round-trip that additionally confirms the HEAD/GET header echo (14/14). Refs rustfs/backlog#1262 rustfs/backlog#1252 Co-Authored-By: heihutu <heihutu@gmail.com> * style(get): allow too_many_arguments on finalize_get_object_response The classifier refactor added an extra_checksum_headers parameter, pushing finalize_get_object_response to 8 args and tripping clippy::too_many_arguments under CI's `-D warnings`. Add the same #[allow] the sibling GET helpers already carry; no behavior change. Refs rustfs/backlog#1252 Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
houseme
|
24e7f8f19c | chore(deps): refresh workspace dependencies (#4804) | ||
|
escapecode
|
a80699b6dd |
feat: add an opt-in NATS JetStream publish path for the notify and audit targets (#4634)
feat(targets): add an opt-in NATS JetStream publish path for the notify and audit targets The NATS notify and audit targets publish through NATS Core, which returns before the server has durably accepted the message. A broker restart or a connection drop between the publish and the flush loses the event, even though the send queue has already cleared it, and no acknowledgement gates that clear. An opt-in JetStream publish path clears a queued event only after the server returns a durable PublishAck, so delivery is at-least-once across a broker restart or a reconnect. It applies to both the notify and audit NATS targets, is off by default, and is byte-identical to the NATS Core path when disabled. The path includes durable store-and-forward, a stable dedup id sent as the Nats-Msg-Id header so a replayed event is collapsed by the stream duplicate window, pre-flight stream validation, and a bounded failed-events store for terminally-failed and retry-exhausted events. Three configuration keys per target select it: JETSTREAM_ENABLE, JETSTREAM_STREAM_NAME, and JETSTREAM_ACK_TIMEOUT_SECS, under the RUSTFS_NOTIFY_NATS_ and RUSTFS_AUDIT_NATS_ prefixes. The on-disk batch filename separator changes from colon to underscore so batch names are valid on Windows filesystems, with transparent read-back of files written under the previous separator. The migration affects the shared queue store for every target type and lands with this feature because the store gains its first Windows-exercised paths here. Co-authored-by: houseme <housemecn@gmail.com> |
||
|
Zhengchao An
|
0f83a27f6a |
fix(deps): pin hyper to flush-before-shutdown fix for large-GET unexpected EOF (#4797)
* fix(deps): pin hyper to flush-before-shutdown fix for large-GET unexpected EOF hyper <= 1.10.1 can call poll_shutdown() on an HTTP/1 socket while response bytes are still buffered (a prior poll_flush() returned Poll::Pending and the result was discarded), so a backpressured/slow-reading peer receives a graceful FIN before the full Content-Length body is flushed. Standard S3 clients (minio-go/warp) report this as sporadic `unexpected EOF` on large-object GET under load (rustfs/backlog#1232). This is the transport-layer bug from Cloudflare's "hyper-bug" writeup — distinct from the EC-reconstruct-desync EOF already fixed via lockstep decode; here the app layer delivers the full body and truncation is purely hyper->socket. Fixed upstream in hyperium/hyper#4018 (commit 72046cc7, "fix(http1): flush buffered data before shutdown"), which is not in any crates.io release yet as of hyper 1.10.1. Pin hyper via [patch.crates-io] to git rev ccc1e850 (a descendant of the fix that still declares version 1.10.1). Use [patch.crates-io], not a git+rev on the workspace `hyper` dependency: the server connection is driven by the transitive hyper-util (conn::auto / GracefulShutdown), and a git source would not unify with the crates.io hyper hyper-util resolves, leaving two hyper copies with the server path still on the buggy one. The patch rewrites the crates.io source globally, so the lockfile holds a single git-sourced hyper. Add rustfs/tests/hyper_h1_shutdown_flush_regression.rs, a deterministic guard (mirrors hyper's own h1_shutdown_while_buffered) that fails if the pin is dropped or hyper is downgraded below the fix. Its load-bearing assertion is a synchronously-set flag, so a slow runner can only under-detect, never false-red. Closes rustfs/backlog#1232. * build(deny): allow the hyperium/hyper git source for the flush-before-shutdown pin cargo-deny's [sources] check denies unknown git sources. The hyper [patch.crates-io] pin added for rustfs/backlog#1232 uses a github.com/hyperium git source, so add it to allow-git with an owner/review note and a removal condition (drop once a released hyper > 1.10.1 carries commit 72046cc7). |
||
|
houseme
|
0ac7f0d0cf | chore: refresh erasure codec and rust toolchains (#4795) | ||
|
houseme
|
2e85709634 |
chore: refresh workspace lockfile (#4785)
* chore: refresh workspace lockfile * chore: bump uuid to 1.23.5 * chore: bump pollster and path-absolutize |
||
|
houseme
|
3b139e5267 |
fix(obs/cleaner): harden log cleaner durability, symlink safety, and retention (audit OLC-01..14) (#4776)
* fix(obs/cleaner): fsync archive and log dir before deleting source logs OLC-01: the compression path flushed the BufWriter but never synced the archive data or the parent directory before renaming, and the source was then unlinked with no durability barrier. A crash after rename but before the page cache reached disk could leave a truncated/zero-length archive while the source was already gone — permanent log/audit data loss. Because the archive is always renamed to a brand-new name (guaranteed by the existing exists() guard), ext4 auto_da_alloc does not mask this. Hand the underlying File back from the writer closure, sync_all() it before rename, fsync the parent directory, and fsync the log directory after the unlinks so a delete cannot be reordered ahead of the archive it justified. Guard the temp file with an RAII cleanup so an early return or panic cannot leak a *.tmp orphan. Ref: rustfs/backlog#1194 (audit rustfs/backlog#1193) Co-Authored-By: heihutu <heihutu@gmail.com> * fix(obs/cleaner): create compression temp file with O_EXCL and O_NOFOLLOW OLC-03: the temp archive was opened with File::create at a predictable `<source>.gz.tmp` path with no O_EXCL/O_NOFOLLOW, so an actor with write access to the log directory could pre-plant that path as a symlink and have the compressor follow it — truncating and overwriting an arbitrary external file, then chmod-ing it to the source log's mode. This mirrors the symlink refusal already enforced on the deletion path (secure_delete). Route temp creation through create_tmp_archive(), which uses create_new (O_CREAT|O_EXCL) to refuse a pre-existing entry and, on Unix, O_NOFOLLOW to refuse a symlink at the final path component. Ref: rustfs/backlog#1196 (audit rustfs/backlog#1193) Co-Authored-By: heihutu <heihutu@gmail.com> * fix(obs/cleaner): create compression temp file with restrictive mode OLC-06: File::create left the temp archive world-readable (0644 & ~umask) for the entire duration of compressing a large log, exposing the full plaintext of a possibly-0600 audit log on shared hosts until the mode was copied only after the write completed. Pass the source mode into create_tmp_archive and open the temp file with it (default 0600) so it is restrictive from creation; the post-write chmod still tightens/matches the source mode exactly. Ref: rustfs/backlog#1199 (audit rustfs/backlog#1193) Co-Authored-By: heihutu <heihutu@gmail.com> * fix(obs/cleaner): validate existing archive before skipping recompression OLC-02: the idempotency guard used Path::exists() (which follows symlinks) and trusted whatever it found, then the caller deleted the source. A planted `<archive>.gz` symlink, or a zero-length/truncated archive left by a crashed run (OLC-01), would green-light deleting the source with no valid backup — data loss / log destruction. Replace exists() with symlink_metadata (no follow) and only treat the entry as a completed prior result when it is a regular, non-empty file whose header matches the codec magic (gzip 1f 8b / zstd 28 b5 2f fd). Anything else falls through to recompression, whose atomic create_new+rename replaces the bad entry (a symlink is replaced, never followed or deleted through). Ref: rustfs/backlog#1195 (audit rustfs/backlog#1193) Co-Authored-By: heihutu <heihutu@gmail.com> * fix(obs/cleaner): stop dry-run overstating reclaimed bytes for compression OLC-08: in dry-run, compress_with_writer returned output_bytes = 0, so projected_freed_bytes = input and delete_files reported the full input as freed. A real run keeps the archive on disk (freed = input - archive), so dry-run overstated reclaim by the whole archive footprint. Estimate the archive with a deliberately conservative ratio so the projection never exceeds what a real run reclaims. Ref: rustfs/backlog#1201 (audit rustfs/backlog#1193) Co-Authored-By: heihutu <heihutu@gmail.com> * fix(obs/cleaner): make freed-byte accounting resilient; document steal metric OLC-12: input/output byte sizes were read via metadata().unwrap_or(0), which silently reports 0 on failure and skews freed-byte metrics (input - 0 = full input, overstating reclaim). Use the copy() byte count as the authoritative input size and, when the archive metadata read fails, conservatively assume no savings instead of 0. Also document that the steal_success_rate counts only victim steals (batch = one success), so it reads as a relative rebalancing signal, not absolute task acquisition. Ref: rustfs/backlog#1205 (audit rustfs/backlog#1193) Co-Authored-By: heihutu <heihutu@gmail.com> * fix(obs/cleaner): preserve level-0 semantics, allow zstd 22, log effective levels OLC-09: build() and the codec calls clamped gzip/zstd levels to [1,9]/[1,21], silently rewriting gzip level 0 (store) and zstd level 0 (codec default) to 1 and blocking the legal zstd maximum of 22. Clamp to [0,9]/[0,22] so those meanings survive, and echo the effective (post-clamp) levels in the startup log via new effective_gzip_level()/effective_zstd_level() getters so the log matches what actually runs. Ref: rustfs/backlog#1202 (audit rustfs/backlog#1193) Co-Authored-By: heihutu <heihutu@gmail.com> * fix(obs/cleaner): bound compressed archives by byte cap; warn on retention=0 OLC-04: archive expiry was gated on compressed_file_retention_days > 0, so retention=0 disabled it entirely while compression kept producing archives, and max_total_size_bytes only ever bounded uncompressed logs — unbounded disk growth. Replace select_expired_compressed with select_archives_to_delete, which applies age expiry (when retention is on) and, regardless of retention, trims the oldest archives until the set fits under max_total_size_bytes. Also warn at startup when compression is on with retention=0 so the "keep forever" semantics are not mistaken for "delete immediately". Ref: rustfs/backlog#1197 (audit rustfs/backlog#1193) Co-Authored-By: heihutu <heihutu@gmail.com> * fix(obs/cleaner): warn on invalid exclude glob instead of dropping silently OLC-05: build() dropped unparseable exclude globs via filter_map(...ok()), so a typo (or a literal comma splitting a char-class in the config string) turned "protect this file" into "delete this file" with no signal. Log a warning per rejected pattern with the raw string and parse error so the misconfiguration is visible. Ref: rustfs/backlog#1198 (audit rustfs/backlog#1193) Co-Authored-By: heihutu <heihutu@gmail.com> * perf(obs/cleaner): backoff idle workers, cap worker count, lower small-host floor OLC-11: the work-stealing loop re-spun on Steal::Retry with no yield and used yield_now on the empty path, burning CPU during redistribution windows; worker_count had no upper bound so a mis-set parallel_workers over a directory of thousands of logs could spawn thousands of threads; and default_parallel_workers forced >=4 workers even on 1-2 vCPU hosts. Use crossbeam_utils::Backoff (spin->yield, reset on work) on the idle paths, clamp worker_count to MAX_PARALLEL_COMPRESS_WORKERS, and lower the default floor to 1. Ref: rustfs/backlog#1204 (audit rustfs/backlog#1193) Co-Authored-By: heihutu <heihutu@gmail.com> * fix(obs/cleaner): warn when active-file guard is disabled by empty filename OLC-13 (defense-in-depth): the scanner protects the live log purely by exact filename equality against active_filename. An empty active_filename silently disables that protection, so a non-empty file_pattern could make the live log a deletion candidate via the public builder. Warn in build() when that unsafe combination is configured. The audit's "never delete the newest match" structural guard is intentionally not implemented: it would conflict with the legitimate keep_files=0 semantics (purge all rotated logs). The naming contract is instead locked by regression tests (OLC-14). Ref: rustfs/backlog#1206 (audit rustfs/backlog#1193) Co-Authored-By: heihutu <heihutu@gmail.com> * fix(obs/cleaner): warn on unknown algorithm/match_mode, echo match_mode OLC-07: from_config_str silently fell back to defaults for unrecognized compression algorithm and match mode (any non-"prefix" value became Suffix), hiding operator typos like "prefixx" that could make the cleaner match no rotated logs. Warn on a non-empty unrecognized value in both parsers, and echo the resolved match_mode in the startup log alongside the algorithm. Ref: rustfs/backlog#1200 (audit rustfs/backlog#1193) Co-Authored-By: heihutu <heihutu@gmail.com> * fix(obs/cleaner): derive orphan .tmp suffixes and exempt them from min age OLC-10: orphan `*.gz.tmp`/`*.zst.tmp` cleanup was gated by min_file_age_seconds (default 3600), so crash-left orphans lingered up to an hour, and the tmp suffix list was hardcoded rather than derived from compressed_suffixes() — a new codec would leave `*.<ext>.tmp` orphans the scanner never recognizes. Derive the temp suffix from CompressionAlgorithm::compressed_suffixes(), and gate orphan removal on a small fixed grace window instead of min_file_age (orphans are never live-written after the rename that would promote them). Ref: rustfs/backlog#1203 (audit rustfs/backlog#1193) Co-Authored-By: heihutu <heihutu@gmail.com> * test(obs/cleaner): cover symlink, archive expiry, idempotency, and edge cases OLC-14: add regression tests for the previously-untested safety/correctness branches — symlink rejection (external target never deleted), archive age expiry vs fresh retention, archive byte-cap trim with retention disabled, gz/zst classification, max_single_file_size selection, min_age protecting a fresh non-empty log, active-file exclusion when the active name also matches the pattern, invalid exclude glob not aborting build, dry-run + compression creating no archive, gzip round-trip validity, and the idempotent-archive branch trusting a valid prior archive. Ref: rustfs/backlog#1207 (audit rustfs/backlog#1193) Co-Authored-By: heihutu <heihutu@gmail.com> * style(obs/cleaner): apply rustfmt and collapse nested if (clippy) Formatting-only cleanup over the audit fix series: rustfmt normalization of the multi-line expressions introduced in compress.rs/core.rs, plus collapsing the delete_files directory-fsync into a single let-chain to satisfy clippy::collapsible_if. No behavior change. Co-Authored-By: heihutu <heihutu@gmail.com> * refactor(obs/cleaner): collapse redundant source stats in compression path The per-issue fixes to compress_with_writer accumulated three metadata() syscalls on the source in the real compression path: an input_bytes read that was immediately shadowed by the copied byte count (a dead read), a source_mode read (OLC-06), and the pre-OLC-06 post-write chmod re-reading the same mode. Collapse to a single fd-based read — move the dry-run input_bytes read into the dry-run branch, read source_mode from the already-open fd (no path stat, no TOCTOU), and reuse it for the post-write chmod. Behavior is unchanged (same inode's mode, written for input size); 3 source stats -> 1. Co-Authored-By: heihutu <heihutu@gmail.com> * chore(obs/cleaner): fix typo flagged by CI (mis-set -> misconfigured) Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
cxymds
|
d8e69a3adf |
fix(logging): enforce single-writer sinks and bound tracing (#4765)
* fix(obs): prevent rolling log stdout aliasing * fix(ecstore): bound hot-path tracing payloads * test(logging): guard service and disk log invariants * fix(obs): silence useless_conversion on st_dev for Linux clippy rustix's Stat.st_dev is u64 on Linux/glibc, making u64::try_from a no-op that trips clippy::useless_conversion under -D warnings. The conversion is still needed on macOS/BSD where st_dev is a signed dev_t, so suppress the lint on that line rather than dropping the portable fallible conversion. * fix(logging): keep stdout sink validation portable (#4769) * fix(obs): keep stdout device conversion portable * docs(logging): update single-writer plan status --------- Co-authored-by: overtrue <anzhengchao@gmail.com> Co-authored-by: houseme <housemecn@gmail.com> |
||
|
houseme
|
4f29dcdcec |
chore(deps): bump rustfs-uring to 0.2.1 (#4731)
chore(deps): bump rustfs-uring to 0.2.1 from crates.io rustfs-uring 0.2.1 routes the driver's runtime diagnostics through `tracing` with structured fields instead of `eprintln!` (rustfs/uring#13). No public API change — UringDriver::probe_and_start_sharded, read_at, and read_at_direct keep their signatures — and the read path / cancel-safety ownership model are untouched, so this is a drop-in patch bump of the version requirement plus the lockfile entry. 0.2.1 adds a `tracing` dependency; it is recorded in the rustfs-uring lock entry. tracing is already in the workspace graph, so nothing new is pulled in. The Cargo.lock change is scoped to the rustfs-uring package only; unrelated lockfile drift is left for its own change. Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
houseme
|
ab60244d7d |
chore(deps): bump rustfs-uring to 0.2.0 (#4730)
chore(deps): bump rustfs-uring to 0.2.0 from crates.io rustfs-uring 0.2.0 is published on crates.io. It carries the read-driver hardening from the backlog#1160 adversarial audit (cancel-safety and graceful-degradation fixes on the Linux io_uring read path). The public API ecstore consumes is unchanged — UringDriver::probe_and_start_sharded, read_at, read_at_direct all keep their 0.1.0 signatures — so this is a drop-in bump of the version requirement plus the lockfile entry. The Cargo.lock change is intentionally scoped to the rustfs-uring package only; unrelated lockfile drift is left for its own change. Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
Zhengchao An
|
bec5227018 |
test(e2e): add negative presigned URL SigV4 coverage (#4714)
test(e2e): negative presigned-URL SigV4 suite (backlog#1151 sec-2) Add crates/e2e_test/src/presigned_negative_test.rs, the query-string SigV4 sibling of the header-SigV4 suite (sec-1, #4708). Presigned URLs were only exercised on the happy path, so nothing pinned our end-to-end enforcement of expiry or query-signature verification. Against a live single-node RustFSTestEnvironment (random port, isolated temp dir), the suite asserts HTTP status + S3 error <Code> for: - expired presigned GET -> 403 AccessDenied (Request has expired) - tampered X-Amz-Signature -> 403 SignatureDoesNotMatch - wrong secret key -> 403 SignatureDoesNotMatch - tampered target key (swapped after signing) -> 403 SignatureDoesNotMatch - tampered presigned PUT -> 403 SignatureDoesNotMatch + object not stored - positive controls: valid presigned GET (body match) and PUT (HEAD verify) Expiry is controlled without real waiting via the AWS SDK presigner's start_time (sign as of one hour ago with a 60s window). s3s checks expiry before the signature, so the expired case surfaces AccessDenied. Wired into the e2e-smoke nextest profile (fast, single-node, no external deps) so it runs on every PR per the crates/e2e_test/README.md admission criteria. Refs rustfs/backlog#1151 (sec-2), rustfs/backlog#1155. |
||
|
houseme
|
f9874b591a |
[DO NOT MERGE until published] chore(ecstore): switch rustfs-uring to crates.io 0.1.0 (#4701)
* chore(ecstore): switch rustfs-uring from the git rev to crates.io 0.1.0
Prepared ahead of the rustfs-uring 0.1.0 crates.io release (rustfs/uring):
flip ecstore's dependency from the pinned git revision to the published
`0.1.0`, and drop the now-unneeded git-source allowance in deny.toml.
DO NOT MERGE until rustfs-uring 0.1.0 is on crates.io. Until then cargo
cannot resolve `rustfs-uring = "0.1.0"`, so this branch will not build and
CI will be red — that is expected, not a defect in the change.
After the crate is published:
1. `cargo update -p rustfs-uring --precise 0.1.0` to regenerate Cargo.lock
(deliberately not touched here — the registry entry, with its checksum,
cannot be written before the crate exists);
2. push the Cargo.lock;
3. CI goes green; merge.
No code change. The guard `scripts/check_no_tokio_io_uring.sh` is unaffected
(it bans only tokio's io-uring runtime feature, not an explicit rustfs-uring
dependency). `audit.yml`'s `allow-dependencies-licenses` for rustfs-uring is
left in place — it is a first-party Apache-2.0 crate either way.
Co-Authored-By: heihutu <heihutu@gmail.com>
* chore: refresh rustfs-uring lockfile
---------
Co-authored-by: heihutu <heihutu@gmail.com>
|
||
|
houseme
|
60ad15a7f9 |
fix(obs): remove the dial9 task-dump switch that could never work; correct measured claims (#4688)
* fix(obs): remove the dial9 task-dump switch that could never do anything Measured on a bench host (Linux x86_64) against the code merged in #4663: with `RUSTFS_RUNTIME_DIAL9_TASK_DUMP_ENABLED=true`, a `dial9-taskdump` build, and `--cfg tokio_taskdump`, dial9 recorded **zero** TaskDump events. dial9 captures a task dump only for futures it wrapped itself — those spawned through `dial9_tokio_telemetry::spawn`, which is where `TaskDumped<F>` gets applied. `tokio::spawn` gets no wrapper, and RustFS spawns with `tokio::spawn` throughout. Same workload, same binary, only the spawner changed: tokio::spawn -> 0 dumps dial9::spawn -> 14709 dumps, all with callchains Upstream documents this (README line 151) and tracks the doc gap at dial9-rs/dial9#477. I did not read it before wiring `with_task_dumps` in #4663, and so shipped exactly the kind of lying configuration knob that PR set out to delete. Remove it: the two environment variables, the config fields, the `with_task_dumps` call, and the `dial9-taskdump` feature — whose only effect was to constrain the build to Linux while recording nothing. Re-adding it only makes sense together with migrating the paths under investigation to dial9's spawner. Tracked as D9-16 in rustfs/backlog#1157. Also drop the `--cfg tokio_taskdump` requirement from the Makefile. Measured: dumps are captured with and without it (14709 vs 14674, within noise), and upstream never asked for it. That requirement was mine, invented and untested. Cargo.lock loses tokio's `backtrace` dependency, which `tokio/taskdump` pulled in. Co-Authored-By: heihutu <heihutu@gmail.com> * docs(obs): replace guessed dial9 retention numbers with measured ones Three corrections, all to claims I wrote in #4663 without measuring them. "Under a high poll rate that budget can wrap in minutes" was a guess. Measured on a single-node 4-drive cluster under warp mixed (66 MiB/s, 110 obj/s, 32 concurrent): 13023 events/s, 0.16 MiB/s, so the default 1 GiB budget wraps after roughly 108 minutes. Even at ten times the throughput that is ~11 minutes. State the measured rate and how to scale it instead. dial9 was described as the tool for drive stalls. It is not. RustFS does disk I/O on the blocking pool and through io_uring, never on an async worker, so a slow drive never lengthens a poll. Injecting 200 ms of latency on one of four drives cut throughput by 64% and left the poll distribution unchanged (polls >= 5 ms: 49 -> 56; p999: 2.67 ms -> 2.75 ms). Enabling dial9's CPU and sched profilers does not help: sched events are per-worker only, and the CPU profiler samples on-CPU while a stalled drive is an off-CPU wait. Say so plainly, and point at the `rustfs_io_*` metrics instead. What dial9 *is* good for, on the same traces: single polls of 418-625 ms with no fault injected at all — real worker stalls nothing else in the obs stack surfaces. Lead with that. Also link the two upstream issues filed for the gaps we documented: dial9-rs/dial9#658 (writer death unobservable) and #659 (worker-s3 CVEs). Measurements: rustfs/backlog#1157 (D9-11, D9-13, D9-18). Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
houseme
|
b604074217 |
perf(object-data-cache): take fill off the GET path and fix the metrics (#4678)
* fix(object-data-cache): make cache metrics one-increment-per-GET
Rework the object data cache observability so each counter carries one
clear meaning, and drop dead per-entry state.
ODC-17 (backlog#1122): split requests_total, which was incremented by
both plan_get and lookup_body, into plan_total{decision,reason,size_class}
(emitted only by plan_get) and lookup_total{result,size_class} (emitted
only by lookup_body). Each is now incremented exactly once per GET per
layer; help text states this.
ODC-18 (backlog#1123): add a JoinedInflightFill result (label
joined_inflight) so a singleflight waiter is no longer counted as an
insert. record_fill_result now always counts the outcome but records
fill bytes only when non-zero and the duration histogram only when
present, so joined_inflight / skipped_by_mode / skipped_size_mismatch
no longer inflate fill_bytes_total or add non-fill duration samples.
The waiter->JoinedInflightFill mapping lives in MokaBackend (owned by a
concurrent branch); cache.rs handles the variant already.
ODC-29 (backlog#1134): stop refreshing the cache-state gauge on every
lookup, and debounce it on fill/invalidate to at most once per second
via an AtomicU64 millis timestamp, since moka's entry_count is a
settling approximation.
ODC-36 (backlog#1141): give invalidations_total an outcome label
(removed|noop) and skip the gauge refresh on the no-op path. Extend
ObjectDataCacheInvalidationResult with Removed{keys}/NoOp (Success kept
as a transitional variant until MokaBackend reports the removal count);
NoopBackend now reports NoOp.
ODC-30 (backlog#1141): drop the dead content_length/etag/inserted_at
fields (and getters) from ObjectDataCacheEntry, which are redundant with
the moka key identity; the constructor keeps its arity so the
out-of-scope MokaBackend caller still compiles. Gate is_null_version
behind cfg(test).
Tests use a thread-local metrics recorder to avoid the global-singleton
flakiness. Fill-enabled test configs set min_free_memory_percent=0 so
the cgroup gate does not refuse fills in CI pods.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(object-data-cache): move fill off GET path, bound & harden the fill
Implements five object-data-cache audit findings.
ODC-15 (backlog#1120): the GET response no longer blocks on cache fill.
The buffered/materialized body is already in hand, so the fill now runs
in a detached task and the response is built immediately. Singleflight is
made non-blocking: `try_acquire` returns Leader or Busy, and a non-leader
skips its own fill (SkippedSingleflightBusy) instead of waiting on another
request's leader. The inner fill spawn is KEPT: once the index key is
registered, the recheck/undo must complete even if the enclosing fill
future is aborted, so removing it would weaken the cancellation-safety
guarantee (ODC-13 test) and the invalidation-race undo.
ODC-11 (backlog#1116): enforce the fill-concurrency knobs. MokaBackend
now holds a Semaphore sized min(per_cpu * parallelism, max), acquired
after winning leadership and before the memory gate. On saturation it
rejects (try_acquire_owned) with SkippedFillConcurrency rather than
queueing, so the fill path never reintroduces GET-latency coupling.
ODC-14 (backlog#1119): the memory gate no longer does a blocking sysinfo
refresh under a mutex on the async fill path. A dedicated periodic
refresher (tokio interval + spawn_blocking) updates an atomic snapshot
every 5s; allows_fill is now lock-free. The min_free_memory_percent == 0
short-circuit still runs before any snapshot read. No runtime at
construction (sync unit tests) simply keeps the seed snapshot.
ODC-32 (backlog#1137): singleflight no longer emits metrics while holding
the map mutex. try_acquire/remove_entry capture the map length under the
guard, drop it, then emit the gauge/counter.
ODC-07 (backlog#1112): the materialize read is bounded via
take(capacity + 1) so an over-long stream cannot grow the buffer past the
in-memory GET threshold, and any length mismatch is now a hard error
matching the direct-memory GET path, instead of warn-and-serve.
cache.rs is owned by a concurrent branch; this commit only appends the
SkippedFillConcurrency and SkippedSingleflightBusy variants + label arms.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(object-data-cache): report JoinedInflightFill and invalidation outcome
Reconciles the moka_backend half of two metrics-semantics findings after
merging fix/odc-metrics-semantics (which landed the cache.rs half).
ODC-18 (backlog#1123): the singleflight non-leader path now returns
JoinedInflightFill instead of counting as an insert, so N concurrent GETs
of one cold key record one insert, not N. This composes with the ODC-15
redesign: the non-leader does NOT wait for the leader (it already owns the
body and never re-serves from the fill result), so no blocking wait is
reintroduced. Drops the redundant local SkippedSingleflightBusy variant in
favor of the canonical JoinedInflightFill (mapped to no bytes / no duration
by the facade). SkippedFillConcurrency (ODC-11) is retained.
ODC-36 (backlog#1141): MokaBackend::invalidate_object now returns
Removed { keys } / NoOp instead of the transitional Success, so the facade
labels invalidations_total{outcome=removed|noop} and skips the cache-state
gauge refresh on the no-op path.
Tests: duplicate-fill test asserts JoinedInflightFill (bites when reverted
to Inserted); new no-op invalidation test; matching-identity test asserts
Removed { keys: 1 }.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(object-data-cache): drop unused mut on materialize stream binding
The ODC-07 change moves `final_stream` into `AsyncReadExt::take`, so the
`build_get_object_body_with_cache` parameter is no longer mutated in place.
Co-Authored-By: heihutu <heihutu@gmail.com>
* refactor(object-data-cache): close the cross-branch coordination seams
Merging the metrics and hot-path branches left four transitional shims that
only existed because each side could not edit the other's files. With both
sides present they are dead weight, and two of them actively misreport.
- ObjectDataCacheEntry::new took content_length and etag only to keep the
caller in moka_backend.rs compiling, then discarded both. Drop them; the
entry is a Bytes wrapper and the caller now passes only the body.
- SkippedSingleflightClosed lost its only producer when the waiter model was
replaced by Leader/Busy election. Remove the variant.
- The fill-accounting match ended in a wildcard that charged fill_bytes and a
duration to every non-Inserted outcome, so a fill rejected by the memory
gate or the concurrency semaphore — which never touched the backend and
wrote nothing — still inflated fill_bytes_total. Enumerate every variant
explicitly: only outcomes that wrote the body report bytes and duration.
Exhaustiveness also forces a future variant to state its accounting rather
than inherit a wrong default.
- InvalidationResult::Success mapped a no-op to outcome=removed, the exact lie
backlog#1141 set out to fix, and its doc comment claimed MokaBackend still
returned it after that backend had been widened. It has no producer left.
Remove it, and tighten the app-layer test from "any successful variant" to
the real contract: the pre-mutation call reports Removed{keys:1}, the
post-delete call NoOp.
Refs: backlog#1123, backlog#1141, backlog#1135
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
|
||
|
houseme
|
00536da80c |
refactor(obs): make dial9 telemetry opt-in and actually record events (#4663)
* refactor(obs): make dial9 telemetry opt-in and actually record events
The dial9 Tokio-runtime profiler was disabled by default, yet every build
paid for it, and enabling it produced trace files with no events in them.
Recorded empty traces
---------------------
`build_traced_runtime` called `TracedRuntime::builder()...build(..)`, but dial9
only starts recording in `build_and_start*`. `build` still returns a live guard
whose `is_enabled()` reports true, and still creates and seals segment files —
they just contain a header and no events. It also skipped `with_trace_path`, so
the background worker driving the segment pipeline was never spawned.
Measured on the new smoke example: 310 bytes of bare segment header, against
5640 bytes for the same workload once recording actually starts.
Switch to `with_trace_path(..).build_and_start(..)`.
Cost was unconditional
----------------------
`--cfg tokio_unstable` was a global `[build] rustflags` entry and `rustfs-obs`
depended on `dial9-tokio-telemetry` unconditionally, so all builds depended on
Tokio's non-semver API. Worse, an environment `RUSTFLAGS` replaces (never
appends to) the config-file value, so any caller exporting their own RUSTFLAGS
silently dropped the flag — the long comment in build.yml was a scar from that.
dial9 is now an opt-in feature (`dial9`, plus `dial9-s3` and `dial9-taskdump`),
the global rustflag is gone, and `crates/obs/build.rs` fails the compile if the
feature is on without the flag. Telemetry builds go through `make build-profiling`.
Metrics that could not lie
--------------------------
`rustfs_dial9_{events_total,bytes_written_total,rotations_total,cpu_overhead_percent}`
were hard-coded to zero — a Counter pinned at 0 reads as "nothing happened".
Removed. `rustfs_dial9_enabled` was sourced from the environment, so it read 1
even when the traced runtime failed and the process fell back to a standard
runtime; it is replaced by `rustfs_dial9_supported` (compile-time),
`rustfs_dial9_configured` (intent) and `rustfs_dial9_active_sessions` (reality).
No `writer_healthy` gauge is exported: dial9's `RotatingWriter` can enter its
`Finished` state and stop writing, but exposes no way to observe that, so the
gauge could only ever be hard-coded to 1. Documented as a known gap instead.
Final events were lost
----------------------
The `TelemetryGuard` lived in a `static OnceLock`, which is never dropped, so
buffered events were never flushed at exit. `build_tokio_runtime` now returns
the guard and `run_process` drops it before any exit path.
Also
----
- `disk_usage_bytes` was a `read_dir` + per-file `stat` on the metrics
collection path. It is now sampled by a background task into an atomic.
- `SAMPLING_RATE`/`S3_BUCKET`/`S3_PREFIX` were parsed, warned about, and
discarded. S3 upload is now wired to dial9's `with_s3_uploader` behind
`dial9-s3`; `SAMPLING_RATE` has no upstream equivalent and is removed.
- Wire `with_task_dumps` (async backtraces of stalled tasks), configurable via
`RUSTFS_RUNTIME_DIAL9_TASK_DUMP_{ENABLED,IDLE_THRESHOLD_MS}`.
- Split `telemetry/dial9.rs` into `config`/`state`/`enabled`/`disabled`; the
stub keeps the public API identical so callers need no `#[cfg]`.
- Drop four print-only examples and the manual test bin that exercised the
removed `init_session` scaffolding.
Verified: cargo check/clippy/test across default, `dial9`, and `dial9-s3`;
build.rs correctly rejects `dial9` without `--cfg tokio_unstable`;
`make pre-commit` passes.
Co-Authored-By: heihutu <heihutu@gmail.com>
* docs(obs): document dial9 as an on-demand profiler
scripts/run.sh advertised a `SAMPLING_RATE` knob that was never passed to dial9,
and claimed "CPU overhead < 5% (with sampling rate 1.0)" and "lower values reduce
CPU overhead" on the strength of it. The knob is gone; the guidance built on it
had to go too.
Replace it with what is actually true: dial9 needs a `make build-profiling`
binary, its disk budget evicts oldest-first (so a high poll rate can overwrite
the incident you are chasing), and it cannot be toggled without a restart.
Add docs/operations/dial9-runtime-profiling.md covering the build variants, an
investigation walkthrough, the configuration table, how to read the three
supported/configured/active_sessions gauges against each other, and the upstream
gap that makes writer death only indirectly observable.
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(obs): add a dial9 smoke example that proves events are recorded
The bug this guards against is invisible to every existing signal: with
`build` instead of `build_and_start`, dial9 creates the trace file, seals
segments, and reports `TelemetryGuard::is_enabled() == true` — it simply
records no events. Only the segment's byte count tells the two apart.
Measured on this workload: 5640 bytes when recording, 310 bytes (a bare
segment header) when not. The example asserts >= 2048 bytes, and was verified
to fail with the `build` call restored.
Also correct the comment on the `is_enabled` check in `finish_traced_runtime`.
It claimed to catch "recording silently off"; it does not. It only rejects the
inert guard a lenient config yields after a build failure. Recording is
guaranteed by `build_and_start`, not by that check.
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(rustfs): accept Unsupported runtime telemetry capability
A binary built without the `dial9` feature now reports the runtime-telemetry
capability as `Unsupported` rather than `Disabled`. The distinction matters to
operators: `Disabled` implies the capability can be switched on by setting an
environment variable, which is not true here — telemetry needs a rebuild.
Widen the assertion and pin the new semantics: when `dial9::is_supported()` is
false, the state must be exactly `Unsupported`.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(obs): drop the dial9-s3 feature, its TLS stack is vulnerable
CI's Dependency Review and `cargo deny` both reject the branch: dial9's
`worker-s3` feature depends on aws-sdk-s3-transfer-manager 0.1.3, which pins
aws-smithy-http-client onto hyper-rustls 0.24 and rustls-webpki 0.101.7. That
webpki carries RUSTSEC-2026-0098, -0099 and -0104.
0.1.3 is the latest release of the transfer manager, and 1.2.0 the latest of the
smithy client, so there is nothing to upgrade to. Cargo's feature unification can
add features but cannot drop a transitive dependency, so it cannot be worked
around from here either — the rest of the workspace already resolves to the safe
rustls-webpki 0.103 / hyper-rustls 0.27.
Remove the `dial9-s3` feature and the `with_s3_uploader` wiring. The two S3
environment variables stay parsed and warned about, now naming the real reason
rather than a missing build feature. Trace segments are collected from the output
directory instead. Tracked as D9-14 in rustfs/backlog#1157.
With this, Cargo.lock is byte-identical to main: the PR no longer touches the
dependency graph at all.
Also correct the `dial9-taskdump` documentation. It claimed the feature "compiles
to a no-op elsewhere"; in fact `tokio/taskdump` raises a `compile_error!` on any
target other than linux/{aarch64,x86,x86_64}. Verified by trying to build it on
macOS, which is how the claim was found to be wrong.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
|
||
|
houseme
|
904554b417 |
fix(object-data-cache): make memory container-aware and bound cache config (#4655)
* fix(object-data-cache): bound cache config and make memory container-aware Harden the object data cache configuration path so a single bad input degrades to the disabled adapter instead of OOM-killing pods, panicking at boot, or silently mis-sizing the cache (backlog#1110/1113/1114/1115/ 1127/1140/1130). - ODC-05: resolve capacity and the fill gate from the effective (container-aware) memory. Prefer sysinfo cgroup_limits() and use min(host, cgroup) as the total plus cgroup-derived availability, falling back to host values when no cgroup limit constrains. Log the resolved capacity and its basis once at startup. - ODC-08: cap ttl/time_to_idle at 30 days in validate() so an unbounded Duration can no longer trip moka's ~1000-year builder assertion. - ODC-09: drop the max_entry_bytes floor from the derived-capacity clamp so MAX_ENTRY_BYTES can no longer inflate total capacity above the safety clamp; reject an entry larger than the resolved capacity instead. - ODC-10: require an explicit max_bytes to clear max_entry_bytes plus the weigher overhead so a fillable-but-unretainable cache is rejected. - ODC-22: reject max_entry_bytes at/above the u32 weigher boundary. - ODC-35: warn (not reject) when time_to_idle exceeds ttl and document the min(ttl, time_to_idle) expiry interaction. - ODC-25: give numeric env overrides two-valued semantics via a new rustfs_utils::get_env_parse_outcome; a malformed value now disables the whole cache with one aggregated warning instead of silently keeping defaults. Co-Authored-By: heihutu <heihutu@gmail.com> * fix(object-data-cache): require identity_keys_max of at least 2 The identity index admits a new key by evicting the oldest one, so a budget of 1 evicts the previous key on every fill and can never hold two live keys of one object at once — a versioned bucket alternating two versions then hits a permanent 0% hit rate. Reject the degenerate value at config validation time, where the adapter turns it into a disabled cache with a warning, since the bounded-eviction policy cannot rescue it at runtime. Handed off from the identity-index batch (backlog#1128), which changed the overflow policy but could not touch validate(). Refs: backlog#1115, backlog#1128 Co-Authored-By: heihutu <heihutu@gmail.com> * fix(object-data-cache): let a zero free-memory floor opt out of the gate Making the memory gate container-aware turned it live in CI: the runners are Kubernetes pods, so the gate now reads the pod's cgroup free memory, finds it below the 20% floor, and refuses the fill. Tests that assert a fill succeeds then failed on CI while passing on a developer host, which has no cgroup and falls back to host memory. The gate is behaving correctly — the tests were the ones depending on a live memory reading. Treat min_free_memory_percent = 0 as a deliberate opt-out rather than an invalid value: allows_fill returns early before it touches any snapshot, so admission becomes independent of where the suite runs. Operators gain the same escape hatch. Every test that requires a fill to succeed now sets the floor to 0. The tests that exercise the gate keep it enabled via memory_gated_config, so the ODC-05 coverage they provide is preserved rather than short-circuited. Refs: backlog#1110 Co-Authored-By: heihutu <heihutu@gmail.com> * fix(object-data-cache): opt the usecase fill tests out of the memory gate The previous commit exempted the fill-dependent tests it could find, but missed the six adapters built inside object_usecase.rs. Fixing the first batch moved nextest's fail-fast point forward and CI surfaced them: build_get_object_body_with_cache_materializes_once_and_hits_later and ..._uses_cached_body_without_reader_preread both assert a fill lands, and both ran with the default 20% free-memory floor. Set the floor to 0 on all six, matching the sibling test modules. Verified by pinning the gate's snapshot to 0% available — harsher than any CI pod — and confirming these tests still pass, which shows the exemption path never reads memory at all. An exhaustive sweep over every fill-enabled ObjectDataCacheConfig in the tree now shows no remaining site: the only unexempted ones are fill_enabled()'s matches! arm and two adapter tests that assert the adapter is disabled and therefore never fill. Refs: backlog#1110 Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
houseme
|
8ffff47529 |
chore(deps): update workspace dependencies (#4643)
Run `cargo update` followed by `cargo upgrade --exclude ratelimit` on the latest main to pull the newest compatible versions. Manifest bumps (Cargo.toml): - md5 0.8.0 -> 0.8.1 - regex 1.12.4 -> 1.13.0 - sysinfo 0.39.5 -> 0.39.6 Lockfile-only bumps: der 0.8.1, dial9-tokio-telemetry 0.3.14, lru 0.18.1, regex-automata 0.4.15, symbolic-common/symbolic-demangle 13.9.0, and zlib-rs 0.6.6. Resolver re-selection moves crypto-common 0.1.7 -> 0.1.6 and generic-array 0.14.7 -> 0.14.9 (both semver-compatible). `ratelimit` is held at 0.10.1 as requested; `pollster` 1.0.0 is an incompatible major bump and is intentionally not taken. Verification: cargo check --workspace --all-targets, cargo clippy --all-features -D warnings, cargo nextest run --all --exclude e2e_test (8467 passed), cargo test --all --doc, and cargo deny check advisories/sources/bans/licenses all pass. Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
houseme
|
da755eb66b |
feat(ecstore): runtime-probed io_uring read backend, gray-off by default (backlog#1104) (#4632)
feat(ecstore): add runtime-probed io_uring read backend, gray-off by default (rustfs/backlog#1104) Wire the standalone rustfs-uring crate (https://github.com/rustfs/uring) into LocalIoBackend as an opt-in read backend. When RUSTFS_IO_URING_READ_ENABLE is set AND the per-disk io_uring probe succeeds, positioned reads go through the cancel-safe UringDriver; otherwise — default, or on a restricted host, or on any per-read driver error — reads fall back to StdBackend byte-for-byte, so the default build is unchanged. - Cargo.toml: rustfs-uring as a Linux-only git dependency (pinned rev). The guard scripts/check_no_tokio_io_uring.sh allows an explicit io-uring integration; only the tokio "io-uring" runtime feature is banned. - UringBackend mirrors StdBackend::pread_bytes's resolution/access/bounds preamble and only swaps the raw byte read; the other three trait methods delegate to the inner StdBackend. - Backend selection at LocalDisk construction via build_local_io_backend. - Differential test uring_backend_reads_match_std: with the flag set, every positioned read returns byte-identical data (driver-served when io_uring is available, StdBackend fallback otherwise). Verified: cargo check -p rustfs-ecstore on Linux; the differential test passes under real io_uring (seccomp=unconfined). The runtime errno degradation latch, per-disk probe cache, and productionization are tracked in rustfs/backlog#1101/#1102. Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
houseme
|
c0c7b9b074 |
chore(deps): update workspace dependencies (#4608)
Run `cargo update` followed by `cargo upgrade --exclude ratelimit` to pull the latest compatible versions across the workspace. Manifest bumps (Cargo.toml): - aws-config 1.8.18 -> 1.9.0, aws-credential-types 1.2.14 -> 1.3.0 - aws-sdk-s3 1.137.0 -> 1.138.0, aws-smithy-http-client 1.1.13 -> 1.2.0 - aws-smithy-runtime-api 1.12.3 -> 1.13.0, aws-smithy-types 1.5.0 -> 1.6.1 - bytes 1.12.0 -> 1.12.1, jiff 0.2.31 -> 0.2.32 - rust-embed 8.11.0 -> 8.12.0, pyroscope 2.0.6 -> 2.1.0 Lockfile-only bumps include the arrow 59.1.0 and aws-smithy families, metrique, memchr, zerocopy, and parquet. `crypto-common` 0.1.6 -> 0.1.7 re-pins the transitive `generic-array` 0.14.x line to 0.14.7 (patch, semver-compatible). `ratelimit` is held at 0.10.1 (2.0.0 available) as requested; its 2.0 API is a breaking change tracked separately. Verification: cargo check --workspace --all-targets, cargo clippy --all-features -D warnings, cargo nextest run --all --exclude e2e_test (8428 passed), cargo test --all --doc, and cargo deny check advisories/sources/bans/licenses all pass. Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
Zhengchao An
|
19a0a73fed |
fix(rustfs): sanitize user-metadata in metadata=true object listing (#4592)
fix(rustfs): sanitize user-metadata in metadata=true object listing (#2743) The console listing path (`list-type=2&metadata=true`) serializes each object's user-metadata key as a raw XML *element name* and its value as XML text without validation. User-metadata keys derived from HTTP headers can legally contain characters that are illegal in an XML `Name` (space, `$`, `%`, `#`, a leading digit, control bytes), and values can contain C0 control characters that are illegal in XML 1.0 text. A single object carrying such a key or value produced a malformed `ListBucketResult` document, which the console's XML parser rejected wholesale — so every object under that prefix vanished from the Web UI, while plain `ListObjectsV2` (which never serializes user metadata) kept working. The breakage appeared the moment any one object in a prefix had XML-unsafe metadata and cleared once that object was removed, matching the report. Guard the serialization in `ObjectMetadataExtension::serialize_content` (shared by the list-objects and list-versions metadata outputs): skip entries whose key is not a valid XML element name, and strip XML-1.0-illegal control characters from values. A single poison object can no longer corrupt the whole listing document. |
||
|
Zhengchao An
|
ed36d1ed97 |
fix(io-metrics): drop client-controlled bucket label from s3 ops counter (backlog#806) (#4580)
fix(io-metrics): drop client-controlled bucket label from rustfs_s3_operations_total (backlog#806-22)
The bucket dimension on the rustfs_s3_operations_total counter is
client-controlled and unbounded, letting any client explode metric
cardinality (a Prometheus/OTEL cardinality DoS that grows the in-process
OTEL aggregation store even without a scrape). Drop the bucket label so the
series is keyed by the bounded op dimension only (<=122 variants), matching
MinIO which never labels its default operation counters with bucket.
BREAKING: rustfs_s3_operations_total no longer carries a "bucket" label.
record_s3_op(op, bucket) -> record_s3_op(op); all call sites in
rustfs/src/storage/{ecfs.rs,helper.rs} updated (bucket bindings retained
where still used by audit/notify paths). Add metrics-util debugging recorder
dev-dep and tests asserting the series carries only the op label and that
series count equals distinct-op count.
|
||
|
Zhengchao An
|
05833063c7 |
refactor(concurrency): remove zero-caller facade modules, fix feature build (#4530)
refactor(concurrency): remove zero-caller facade modules and fix no-default-features build (backlog#1025) The audit in rustfs/backlog#1010 (consistent with #805) established that most of crates/concurrency was a decorative facade with zero production callers; the real runtime concurrency control lives in rustfs/src/storage/*. This deletes the dead facades and keeps only what the workspace actually consumes. Deleted (zero callers verified by workspace-wide grep): - manager.rs: ConcurrencyManager, lifecycle start/stop, misleading 'started' lifecycle logs - config.rs: ConcurrencyConfig, ConcurrencyFeatures, from_env - timeout.rs: TimeoutManager, TimeoutGuard, TimeoutManagerPolicy - lock.rs: LockManager, LockScopeGuard, OptimizedLockGuard - scheduler.rs: SchedulerManager, SchedulerPolicy, IoStrategy - deadlock.rs facade: DeadlockManager, RequestTracker - backpressure.rs facade: BackpressureManager, BackpressurePipe - the prelude module, unused io-core re-exports, and all feature flags Kept (real callers in ecstore/heal/rustfs): - workload.rs admission contract types (unchanged) - workers.rs Workers pool (unchanged, retained per #4498) - GetObjectQueueSnapshot (moved from manager.rs to new queue.rs) - PipeBackpressurePolicy (used by rustfs/src/storage/backpressure.rs) - DeadlockMonitorPolicy (used by rustfs/src/storage/deadlock_detector.rs) - OperationProgress re-export (used by rustfs/src/storage/timeout_wrapper.rs) Removing the feature flags fixes the previously broken cargo check -p rustfs-concurrency --no-default-features (E0432). Docs and the logging guardrail file list are updated to match. Ref: rustfs/backlog#1025 |
||
|
houseme
|
cd327c81f5 |
fix(targets): harden control-plane, TLS reload, and runtime extension points (#4504)
Addresses security/correctness audit findings in the target-plugin control plane, TLS reload coordinator, and runtime extension registries. control_plane (backlog#977): - Install now preserves the currently installed revision as previous_revision so Rollback actually restores the prior version instead of a no-op. - Split the gate: circuit-breaker and runtime-activation checks apply only to Install/Enable; Disable and Rollback stay available as break-glass remediation while the breaker is open. - Enforce the sidecar runtime protocol version for every external transport (previously skippable by declaring a non-gRPC transport) and validate the plugin api_compatibility_version at planning time. - Download/signature/provenance host allowlisting now matches the full host authority, so an allowlisted host never implicitly authorizes a different host:port. TLS reload coordinator/validate (backlog#981, #970-coordinator): - Compute the fingerprint before building material in register() to remove the TOCTOU that could permanently pin an old certificate; rotation self-heals. - Always start a detection loop when reload is enabled; Watch mode no longer returns success without any loop. - validate_cert_key_pairing now verifies the private key matches the certificate's public key instead of only parsing both files. - Normalize a zero poll interval to a positive minimum to avoid a panic that silently killed the poll loop. - Serialize reload cycles with a per-target mutex; a first-step fingerprint read failure now records last_error and a failure metric. - Duplicate label registration stops and joins the previous loop before publishing the replacement (stop-before-start), preventing orphaned loops. runtime extension points (backlog#983 runtime subset): - ops_profiler/ops_diagnostics authorize before probing the registry so an unauthorized caller cannot learn whether a backend/surface exists. - s3_hooks dispatch_post_auth actually traverses the registered hooks for the point instead of unconditionally returning Continue. - sidecar send_with_timeout redacts errors and uses the configurable failure threshold via policy, and a successful send resets the breaker accounting. Relates to rustfs/backlog#977 Relates to rustfs/backlog#981 Relates to rustfs/backlog#970 Relates to rustfs/backlog#983 Co-authored-by: heihutu <heihutu@gmail.com> |