Compare commits

..

1 Commits

Author SHA1 Message Date
overtrue 94f9c5b718 fix(test): make 4 tests resilient to shared global state in full suite
The scanner_activity, background_heal_status, and two capacity dirty
scope tests assumed no global AppContext or object store was registered.
When run as part of the full test binary, shared_gating_ecstore()
initialises global state (ECStore + notification system) that other tests
pick up via runtime_sources::current_app_context().

- scanner_activity: accept either Unavailable (no storage) or a valid
  response (storage available from shared env)
- background_heal_status: accept either success=false (no storage) or
  success=true with non-empty state
- capacity dirty scope: use filter_map on canonicalize to skip dirty
  disks whose paths are not resolvable on the local filesystem
2026-07-20 21:10:39 +08:00
1042 changed files with 59471 additions and 357123 deletions
+16 -19
View File
@@ -1,6 +1,6 @@
---
name: adversarial-validation
description: Execute the Adversarial Validation policy from the root AGENTS.md — run the applicable reviewer roles with RustFS-specific attack probes. Use on every behavior-affecting code change, bug fix, design proposal, or agent-instruction change that alters execution before declaring it done.
description: Execute the Adversarial Validation policy from the root AGENTS.md — run the seven reviewer roles (correctness, simplicity, security, concurrency/durability, compatibility, performance, test coverage) with RustFS-specific attack probes. Use on every behavior-affecting code change, bug fix, or design proposal before declaring it done.
---
# Adversarial Validation Playbooks
@@ -61,15 +61,14 @@ Null report example: "Attacked quorum-1 error reduction, exact max-keys listing
### Simplicity adversary
- Smaller-diff attack: inspect production growth separately from tests, fixtures, generated code, and documentation; test additions have no growth budget. Rewrite the production diff mentally (or in scratch) as the minimal equivalent edit. Report a finding only with a concrete smaller design that preserves correctness, compatibility, readability, and real boundaries; fewer lines alone are not evidence.
- Smaller-diff attack: rewrite the diff's change mentally (or actually, in scratch) as the minimal in-place edit and compare. Flag as findings: a helper function with exactly one caller introduced by this diff; a file rewrite where a 3-line edit inside the existing control flow suffices; reshaped control flow in init/locking/metadata/quorum paths beyond what the fix requires; new string literals duplicating existing constants (grep the token first); #[path] module inclusion. If the smaller diff achieves identical behavior, report it with the concrete replacement.
- Where: Any diff; extra scrutiny for crates/ecstore, crates/lock, rustfs/src/storage where 'preserve the existing control-flow shape' is an explicit rule
- Evidence: AGENTS.md 'Change Style for Existing Logic' (conditional extraction rule, preserve sensitive control flow, canonical modules) and 'Reuse Before You Write'; the Adversarial Validation roles list charters this attack.
- Reuse-and-necessity attack: for each new helper, search `crates/utils`, `crates/common`, the touched crate, the likely domain owner, and relevant direct dependencies. A reimplementation is a finding, but forced reuse with mismatched normalization, error, backoff, or durability semantics is also a finding. Demand a nameable trigger for new defensive branches. Tests remain subject to validity and near-duplicate coverage review, never a size limit.
- Where: Any diff adding helpers, branches on decoded/peer data, or tests
- Evidence: AGENTS.md 'Change Style for Existing Logic' (one-off helper ban, preserve control-flow shape in distributed/locking/metadata paths, no #[path]) and 'Reuse Before You Write' (constants clause); the Adversarial Validation roles list charters the simplicity adversary with exactly this attack.
- Reuse-and-necessity attack: for each new helper the diff introduces, run `ls crates/utils/src crates/common/src` and `rg -i 'fn \w*<term>'` over those dirs plus the touched crate (snake_case signatures — a full-text single-word grep drowns, a multi-word phrase returns nothing). A reimplementation of an existing workspace utility, or of plain std/tokio behavior no wrapper refines, is a finding but so is forced reuse with mismatched semantics (normalization such as `clean` resolving `.`/`..` against raw S3 keys, error type, backoff, durability gating). For each new defensive branch, demand the nameable trigger and flag re-validation of what a validated upstream layer on the SAME path already guarantees — excluding the Cross-Cutting Domain Invariant patterns (nil/empty/absent UUID, dual metadata keys, unversioned-tier versionId) and re-checks before destructive actions, which are load-bearing even when redundant on the happy path. For each new test, flag near-duplicates pinning the same code path AND poison-value class as an existing test — boundary companions (n==max vs max+1, absent vs empty vs nil UUID, MetaObject vs MetaDeleteMarker) are never near-duplicates; the test-coverage skeptic playbook below mandates them.
- Where: Any diff adding helpers, branches on decoded/peer data, or tests; helper checks against crates/utils, crates/common, and the touched crate
- Evidence: AGENTS.md 'Reuse Before You Write' and 'Necessary Code Only'; GHSA-f4vq-9ffr-m8m3 (normalization-asymmetry traversal — why forced reuse of normalizing helpers on raw keys is itself an attack); docs/operations/tier-ilm-debugging.md nil-versionId incident (why boundary re-checks are load-bearing).
- Replacement-and-comment attack: when the diff introduces a replacement path or representation, trace all callers and flag a superseded in-scope path left behind without a compatibility requirement. Keep one canonical core behind compatibility adapters. Comments must state non-obvious invariants completely without narration or change history. Never demand unrelated deletion or trade away correctness, compatibility, or readability to reduce the diff.
Null report example: "Separated production growth from tests/docs, tested a smaller equivalent, checked helper reuse and superseded paths, and found no break."
Null report example: "Rewrote the diff as an in-place edit (no smaller equivalent exists), grepped both new helpers against crates/utils, crates/common, and the touched crate (no existing equivalent; call-site semantics checked), verified the two new defensive branches name concrete corrupt-input triggers, and checked the added tests against the existing suite (each pins a distinct poison-value class) — no break found."
### Security reviewer
@@ -85,9 +84,6 @@ Null report example: "Separated production growth from tests/docs, tested a smal
- For any secret/token/signature/password comparison in the diff, check it uses a constant-time compare (e.g. subtle/constant_time_eq), not == or early-return byte loops. Then check the failure-response paths: construct an invalid-user request and an invalid-secret request and confirm they are indistinguishable (same error, no early length short-circuit) so an attacker cannot enumerate valid users or time-side-channel the secret.
- Where: crates/protocols/ (FTPS/WebDAV/FormPost auth), crates/credentials/, rustfs/src/auth.rs, RPC signature verification
- Evidence: GHSA-3p3x-734c-h5vx (FTPS/WebDAV early-return string equality + distinguishable invalid-user vs invalid-password). Fix commits 3c3113619 (constant-time FTPS/WebDAV) and c41062f27 (constant-time FormPost signature). 3p3x was fixed by PR #4403.
- If the diff parses or transports secret-bearing config (env vars, key files, connection strings), grep every error-construction and format site on that value's path (`format!` feeding `Error::other`/`configuration_error`/`panic!`/`expect`) for interpolation of the raw value or of variables named like secret material. Construct the likeliest misconfiguration: the operator supplies the bare secret without the expected `<name>:` prefix (or with a stray newline) — if the parse-failure hint echoes the input, the secret lands in startup logs. Error strings are log content; the hint may name the env var and expected format, never the value. If the diff re-implements an existing parse helper, diff the two error paths — the duplicate is where the leak hides.
- Where: rustfs/src/init.rs (env plumbing), crates/kms/src/config.rs, crates/credentials/, any from_env/parse on secret values; mechanical backstop in scripts/check_logging_guardrails.sh (secret-interpolation check)
- Evidence: PR #5222 introduced `got: {secret_str}` in build_static_kms_config's format-hint error — a bare base64 key (the secret itself) would have been echoed into startup logs; fixed by PR #5243. The parallel parse in KmsConfig::from_env already omitted the value: the leak lived only in the duplicated copy (AGENTS.md 'Reuse Before You Write').
- If the diff touches internode/RPC auth secret handling, trace whether the RPC HMAC secret can fall back to a public default (e.g. 'rustfsadmin', 'rustfs rpc') or be derived deterministically from the S3 root credentials. Construct the case where RUSTFS_RPC_SECRET is unset and confirm the code fails closed rather than silently using a default or a root-derived key. Verify RPC signing keys are independent random secrets, not reused across S3-root/RPC-HMAC/STS-JWT roles.
- Where: crates/credentials/, crates/ecstore/src/rpc/, internode auth setup
- Evidence: GHSA-r5qv-rc46-hv8q (fell back to 'rustfsadmin'), GHSA-75fx/68cw (RPC secret derivable from root creds → forgeable signatures), GHSA-h956 (hard-coded 'rustfs rpc'), GHSA-m77q (STS JWT reused root secret). Fix commit 7b2055405 (fail closed when deriving RPC secret from default credentials, PR#4402).
@@ -196,9 +192,9 @@ Null report example: "Attacked dual-key metadata writes/removals against MinIO-o
### Performance reviewer
- For each `.clone()` or allocation added to a per-request/per-object path, identify the copied data and execution frequency. Report a finding only for a concrete repeated cost or benchmark regression. Recommend borrowing, moving, `Bytes`/`Arc`, `Cow`, or capacity reservation only when it reduces that cost without obscuring ownership or APIs.
- For every `.clone()` the diff adds or moves onto a per-request/per-object path, open the cloned type and count heap fields (String, Vec, HashMap, Bytes). If >5 heap fields or it contains an EC block buffer, construct the cost: N concurrent PUTs x M objects -> N*M deep copies per second. Demand Arc-wrapping of heavy fields or pass-by-reference; also flag new `String` allocations in header/path/signature parsing where `&str`/`Cow<str>` suffices.
- Where: crates/ecstore/src/set_disk/**, crates/ecstore/src/store*.rs, rustfs/src/storage/, crates/filemeta/, request handlers in rustfs/src/
- Evidence: crates/ecstore/AGENTS.md 'Allocation Discipline in Hot Paths'; .agents/skills/rust-code-quality/SKILL.md requires a concrete hot-path cost rather than a proxy metric
- Evidence: crates/ecstore/AGENTS.md 'Allocation Discipline in Hot Paths' (no Clone on >5-heap-field structs, Arc for large buffers, &str/Cow for temporary computations); .agents/skills/rust-code-quality/SKILL.md ranks 'unnecessary clone in hot path' as P1 must-fix
- For every new sync_all/sync_data/fdatasync/flush/File::sync call in the diff, trace the call chain to DurabilityMode / RUSTFS_DRIVE_SYNC_ENABLE resolution (crates/ecstore/src/disk/local.rs:291 DurabilityMode, :347 resolve_durability_mode) and to per-bucket durability overrides. Construct the run where the operator sets mode=none (or legacy RUSTFS_DRIVE_SYNC_ENABLE=false) and the new fsync still fires — that is an ungated durability cost and a regression on 4KiB writes.
- Where: crates/ecstore/src/disk/local.rs, crates/ecstore/src/bucket/durability.rs, crates/ecstore/src/set_disk/** (rename_data/commit paths), any crate doing tokio::fs or std::fs writes
- Evidence: #4221 fsync work caused a measured -10% 4KiB write regression (#814 investigation), later gated; durability modes added in eaff17cad (#4397), per-bucket tier overrides in 13e48d93a (#4407); 2df315baf (#4493) shows even ancestor-dir fsyncs are routed through the gate
@@ -231,12 +227,12 @@ Null report example: "Attacked the new rename_data commit-section work, durabili
### Test-coverage skeptic
- For every testable behavior claim in the PR description, revert that hunk and name the focused test or executable check that detects the revert. If no reasonable check exists, require the reason and residual risk from the validation floor. Especially verify the check exercises the real production path, not a lookalike helper.
- For every behavior claim in the PR description, revert that hunk (git stash / manual undo of the changed lines) and name the exact test (`cargo test -p <crate> <test_name>`) that fails. If no test fails on revert, the behavior is untested — file a finding, not a note. Especially verify the test exercises the REAL production call path, not a lookalike helper.
- Where: All crates; highest value in crates/ecstore, rustfs/src/storage, crates/heal
- Evidence: AGENTS.md testable-behavior exit criterion. Real bug: PR #4220 (ghost-directory cleanup) merged with green tests but its fix never executed on the real delete path — required follow-up rustfs#4307, backlog#798 stayed OPEN. The tests exercised a path the production flow never took.
- Evidence: AGENTS.md exit criterion 'Every behavior change has a test that fails without it'. Real bug: PR #4220 (ghost-directory cleanup) merged with green tests but its fix never executed on the real delete path — required follow-up rustfs#4307, backlog#798 stayed OPEN. The tests exercised a path the production flow never took.
- Read each added/modified test and confirm it asserts the real outcome (returned value, stored bytes, error variant), not merely 'call succeeded' or 'no panic'. Flag any test whose only observable is that the function returned, and any `assert!(result.is_err())` that never checks WHICH error. Then check: does the test prove the exploit/failure form is denied, or only that the intended form still works?
- Where: crates/e2e_test (security_boundary_test.rs pattern), and every #[cfg(test)] module in the diff
- Evidence: Commit dee8e4e63 (#4466) had to rewrite 277 lines of crates/e2e_test/src/security_boundary_test.rs because 'security boundary tests' passed without asserting real outcomes. .agents/skills/rust-code-quality/SKILL.md requires an observable failure criterion; .agents/skills/security-advisory-lessons/SKILL.md asks whether the exploit form is denied.
- Evidence: Commit dee8e4e63 (#4466) had to rewrite 277 lines of crates/e2e_test/src/security_boundary_test.rs because 'security boundary tests' passed without asserting real outcomes. .agents/skills/rust-code-quality/SKILL.md checklist: 'Every test function has at least one assert!'; .agents/skills/security-advisory-lessons/SKILL.md: 'Does the test prove the exploit form is denied, or only that the intended form still works?'
- When the diff adds a boolean/mode parameter or config flag, find the test that fails if the flag's effect is INVERTED inside the changed function. Tests that were mechanically updated to pass `false`/default at every call site assert nothing about the new behavior. Execute the check: flip the flag's branch in the source and confirm at least one test goes red for each branch.
- Where: crates/ecstore/src/set_disk/ (e.g. build_codec_streaming_part_reader), any function gaining a parameter
- Evidence: Commit 05890d6e2 (#4573): PR #4560 added a 15th param allow_inplace_legacy_fallback; the arity tests were fixed by passing `false` everywhere — they assert Err outcomes independent of the flag, so the fallback behavior itself has no revert-detecting test at those sites.
@@ -261,7 +257,7 @@ Null report example: "Attacked the new rename_data commit-section work, durabili
- For any pagination/limit/truncation change, construct the exact-boundary test: result count == max (page exactly full), max+1, and a delimiter re-fold that lands precisely on the page boundary — assert both the item count AND the is_truncated/continuation marker. Off-by-one at the page boundary is a recurring shipped bug here.
- Where: crates/ecstore listing paths (list_objects, ListMultipartUploads, metacache), S3 handlers in rustfs/src/storage
- Evidence: Two shipped boundary bugs: fefa70b31 (#4447) ListMultipartUploads returned one upload past max-uploads; d91f4d455 (#4538) delimiter re-fold of a full page lost the truncation flag. Both survived existing tests because no test pinned n == max exactly.
- A green focused test is evidence only for the targets it builds. Follow the `AGENTS.md` validation tier: add package-scoped Clippy or broader test-target compilation only when changed targets, features, or dependents remain uncovered; do not require a workspace-wide build by default.
- Green `cargo test -p <crate>` on the touched crate is not a coverage verdict for the diff's test code itself: run `cargo clippy --all-targets -p <crate>` and a workspace-wide test BUILD (`cargo check --workspace --all-targets` at minimum) before accepting the tests as evidence. Test-only code that doesn't compile workspace-wide or fails clippy has repeatedly broken main and masked whether tests ran at all.
- Where: All crates; especially concurrent-branch merges into crates/ecstore
- Evidence: #4322 broke main because only cargo test ran (field_reassign_with_default is clippy-only). b06f3df6b (#4441) and 05890d6e2 (#4573): test code broke the workspace test build (E0061) on main after textually-clean merges, failing CI for every open PR.
@@ -272,6 +268,7 @@ Null report example: "Attacked revert-detection for all 3 claimed behaviors (eac
Probes are distilled from shipped bugs in git history (commit/PR references
above), GitHub security advisories (see the security-advisory-lessons
skill), scoped `AGENTS.md` rules, and invariants under `docs/architecture/`
and `docs/operations/`. Line numbers drift; re-locate the invariant. Merge
new incidents into an existing probe when they share a failure class; add a
new probe only for a distinct attack, rather than growing the root policy.
and `docs/operations/`. Line numbers drift; when a cited location no longer
matches, trust the invariant and re-locate the code. When a new bug class
ships, add a probe with its evidence here rather than growing the policy
section in `AGENTS.md`.
+2 -8
View File
@@ -10,16 +10,10 @@ never weaken a check to get green.
## `check_layer_dependencies.sh` — layer DAG in `rustfs/src`
Enforces `composition (server, startup/init) → interface (admin,
storage/ecfs, storage/s3_api) → app → infra`; no upward imports. Server source
files are composition roots, while imports of their exported HTTP contracts
are classified as interface dependencies. Known legacy violations live in
Enforces `interface (admin, storage/ecfs, storage/s3_api) → app → infra`; no
upward imports. Known legacy violations live in
`scripts/layer-dependency-baseline.txt`.
Dedicated `*_test.rs` and `tests/` modules are outside this production guard.
Inline `#[cfg(test)]` imports remain checked under their source file's layer;
move architecture-crossing test scaffolding into a dedicated test module.
- **New violation**: restructure your change so the dependency points
downward (move the shared type/function to the lower layer).
- **You legitimately removed a baseline entry**: run
+4 -7
View File
@@ -24,17 +24,15 @@ Use this skill before `gh pr create`, before `gh pr edit`, or when reviewing whe
2. Inspect change scope
- Review the diff and summarize what changed.
- Inspect `git diff --stat` and `git diff --numstat`; assess production-code growth separately. Tests, fixtures, generated code, and documentation have no growth budget. Treat line counts as signals, not quotas.
- Call out unrelated edits, generated artifacts, logs, or secrets as blockers.
- Mark risky areas explicitly: auth, storage, config, network, migrations, breaking changes.
- Use the simplicity-adversary verdict instead of producing a per-symbol inventory. Block growth only when the review identifies duplication or gives a concrete smaller design that preserves correctness, compatibility, readability, and real boundaries.
- Confirm replacement implementations remove the superseded in-scope path or adapt compatibility at the boundary to one canonical core.
- Scan the diff for newly added string literals and confirm whether they duplicate values already defined as constants/enums/typed wrappers in the same module or shared modules.
- Treat introducing a new hardcoded literal where a project constant already exists as a likely regression risk; require either a refactor to reuse the constant or an explicit exception explanation in the PR body.
3. Verify readiness requirements
- Select checks from `AGENTS.md` "Verification Before PR" based on the final diff's risk tier. Do not replace a focused behavioral test with `make pre-commit`, or a required high-risk `make pre-pr` with a narrower gate.
- For focused verification, state why the selected tier is sufficient and list the scope-specific commands in the PR body.
- Require `make pre-commit` before marking PRs ready when the diff changes Rust code, product behavior, CI behavior, runtime configuration, security-sensitive logic, migrations, storage, auth, networking, or other high-risk paths.
- For documentation-only, agent-instruction-only, or local developer-tooling-only changes, allow focused verification instead of `make pre-commit` when it directly validates the changed surface.
- For focused verification, explain why the full gate was not run and list the scope-specific commands in the PR body.
- If `make` is unavailable, use the equivalent commands from `.config/make/`.
- Add scope-specific verification commands when the changed area needs more than the baseline.
- If required checks fail, stop and return `BLOCKED`.
@@ -83,14 +81,13 @@ Use this skill before `gh pr create`, before `gh pr edit`, or when reviewing whe
## Blocker rules
- Return `BLOCKED` if the checks required by the `AGENTS.md` validation tier have not passed.
- Return `BLOCKED` if a code, behavior, CI, runtime configuration, security-sensitive, migration, storage, auth, networking, or other high-risk change has not passed `make pre-commit`.
- Return `BLOCKED` if a documentation-only, agent-instruction-only, or local developer-tooling-only change lacks focused verification for the changed surface.
- Return `BLOCKED` if the diff contains unrelated changes that are not acknowledged.
- Return `BLOCKED` if required template sections are missing.
- Return `BLOCKED` if the title/body is not in English.
- Return `BLOCKED` if the title does not follow the repository's Conventional Commit rule.
- Return `BLOCKED` if the diff introduces string literals that should use existing constants but did not.
- Return `BLOCKED` for production-code growth only when the review identifies a duplicated or superseded implementation, or supplies a concrete smaller design with equivalent semantics. Fewer lines alone are not evidence.
## Reference
@@ -3,8 +3,8 @@
- Confirm the branch is based on current `main`.
- Confirm the diff matches the stated scope.
- Confirm no secrets, logs, temp files, or unrelated refactors are included.
- Confirm the checks required by the `AGENTS.md` validation tier passed.
- For focused verification, confirm it covered the changed surface and the PR body explains why the selected tier is sufficient.
- Confirm `make pre-commit` passed for code, behavior, CI, runtime configuration, security-sensitive, migration, storage, auth, networking, or other high-risk changes.
- For documentation-only, agent-instruction-only, or local developer-tooling-only changes, confirm focused verification covered the changed surface and the PR body explains why the full gate was not run.
- Confirm extra verification commands are listed for risky changes.
- Confirm the PR title uses Conventional Commits and stays within 72 characters.
- Confirm the PR title does not use tool-specific prefixes such as `[codex]`.
+32 -32
View File
@@ -1,6 +1,6 @@
---
name: rust-code-quality
description: Enforce Rust-specific code quality rules on every Rust change. Use before merge to catch unwrap abuse, silent truncation, unnecessary cloning, lock ordering violations, recursion risks, and error type anti-patterns.
description: Enforce Rust-specific code quality rules on every code change. Use before merge to catch unwrap abuse, silent truncation, unnecessary cloning, lock ordering violations, recursion risks, and error type anti-patterns.
---
# Rust Code Quality Gate
@@ -12,29 +12,27 @@ Use this skill on every Rust code change to enforce quality rules that `cargo cl
1. Identify changed `.rs` files.
2. Run automated checks on changed files.
3. Run manual review checklist on the diff.
4. Resolve or rebut every finding with evidence; P0/P1 findings cannot be deferred.
4. Report findings; block merge if P0/P1 issues exist.
## Automated Checks
Use these searches to find candidates in changed `.rs` files. Inspect syntax,
`#[cfg(test)]` scope, and the changed hunk before reporting a finding; text
filters do not reliably distinguish production code from tests.
Run these on every changed `.rs` file (excluding test modules):
```bash
# 1. unwrap/expect candidates
rg -n '\.unwrap\(\)|\.expect\(' <changed-files>
# 1. unwrap/expect in production code
rg -n '\.unwrap\(\)|\.expect\(' <changed-files> | grep -v '#\[cfg(test)\]' | grep -v 'test' | grep -v 'bench'
# 2. Silent type truncation via `as` cast
rg -n ' as (u8|u16|u32|u64|usize|i8|i16|i32|i64|isize)\b' <changed-files>
# 3. String as error type
rg -n 'Result<.*String>' <changed-files>
rg -n 'Result<.*String>' <changed-files> | grep -v test
# 4. Box<dyn Error> in public APIs
rg -n 'Box<dyn.*Error' <changed-files>
rg -n 'Box<dyn.*Error' <changed-files> | grep -v test
# 5. println/eprintln in production
rg -n 'println!\|eprintln!' <changed-files>
rg -n 'println!\|eprintln!' <changed-files> | grep -v test
# 6. Ordering::Relaxed usage (verify each is intentional)
rg -n 'Ordering::Relaxed' <changed-files>
@@ -48,35 +46,37 @@ rg -n 'unwrap_or_default\(\)|unwrap_or\(' <changed-files>
For every Rust code change, verify:
### Error Handling
- [ ] Every production `unwrap()` or `expect()` is infallible by type or a checked invariant; explain only non-obvious invariants, using an existing type, a useful `expect` message, or a concise comment
- [ ] No `unwrap()` or `expect()` in production code without justification comment
- [ ] No `Result<_, String>` in public API signatures
- [ ] Public library APIs use domain errors unless deliberate error erasure at a boundary is part of the contract
- [ ] No `Box<dyn Error>` in public trait/struct methods
- [ ] `Error::source()` is overridden when inner error is stored
- [ ] Error messages are actionable without exposing secret input
- [ ] Error messages are actionable (what failed, with what input)
### Type Safety
- [ ] No silent `as` truncation (negative→unsigned, large→small)
- [ ] Fallible numeric conversions use `TryFrom`/`try_into()` and return a typed error; clamp or saturate only when the domain explicitly requires it
- [ ] Floating-point to integer conversion validates finiteness, sign, and range before conversion
- [ ] `try_into()` or explicit clamping used for numeric conversions
- [ ] No `f64 as usize` without prior clamping
### Concurrency
- [ ] Lock acquisition order is documented when multiple locks are used, and matches every other call site taking any overlapping subset (ABBA check)
- [ ] No `tokio::sync` lock guard (read or write) held across `.await` without bounded hold time — long-lived read guards wedge writers (#4195)
- [ ] Atomic read-modify-write uses the direct `fetch_*` operation when possible; use `compare_exchange` only for conditional updates
- [ ] Concurrent counters use `compare_exchange` loops, not load-then-store
- [ ] `std::sync::Mutex` in async context is held only briefly, never across `.await`
### Memory and Performance
- [ ] On an identified hot path, report cloning or allocation only with a concrete per-request/per-object cost or benchmark signal
- [ ] Prefer borrowing, moving, `Bytes`/`Arc`, or capacity reservation only when it reduces that cost without obscuring ownership or APIs
- [ ] No `.clone()` on structs with >5 heap-allocated fields in hot paths
- [ ] `HashMap::with_capacity()` / `Vec::with_capacity()` used when size is known
- [ ] Large buffers wrapped in `Arc` rather than cloned
- [ ] Temporary string computations use `&str` or `Cow<str>` instead of `String`
### Recursion Safety
- [ ] Recursion over untrusted, persisted, or otherwise unbounded input has a depth limit or uses iterative traversal
- [ ] Recursive functions have a depth limit or use iterative traversal
- [ ] Tree/cache traversals handle corrupted/cyclic input safely
### Testing
- [ ] Tests have an observable failure criterion; delegated assertions, `#[should_panic]`, snapshot/property checks, and meaningful `Result` failures do not need a redundant `assert!`
- [ ] Use `expect` only when its message improves failure diagnosis; do not add boilerplate to self-evident test setup
- [ ] Test volume and line count are never treated as production-code growth
- [ ] Every test function has at least one `assert!`
- [ ] Tests use `.expect("context")` not bare `.unwrap()`
- [ ] No `println!`/`eprintln!` in production code (use `tracing`)
### Serde
- [ ] Structs from untrusted input have `#[serde(deny_unknown_fields)]`
@@ -88,18 +88,18 @@ For every Rust code change, verify:
- [ ] New string literals don't duplicate existing constants
### Reuse and Necessity
- [ ] No new helper duplicates `crates/utils`, `crates/common`, the touched crate, the likely domain-owning crate, a relevant direct dependency, or plain std/tokio behavior; reused helpers match the call site's semantics
- [ ] No new helper duplicating an existing workspace utility (`crates/utils`, `crates/common`, the touched crate) or plain std/tokio behavior no wrapper refines; reused helpers match the call site's semantics (normalization, error type, backoff, durability gating)
- [ ] No branch without a nameable concrete trigger; no re-validation of what a validated upstream layer on the same path already guarantees (Cross-Cutting Domain Invariant patterns and pre-destructive-action re-checks are load-bearing — keep them)
- [ ] Error context attached once where actionable, not re-wrapped at every hop; no typed→generic error conversion below aggregation/quorum layers
- [ ] Comments avoid narration and change history while completely stating non-obvious lock, `SAFETY`, durability, compatibility, and unwrap invariants
- [ ] No comments narrating the next line, restating a signature, or describing the change itself (invariant comments — lock ordering, `SAFETY`, unwrap justification — are not narration)
- [ ] No near-duplicate test pinning the same code path and poison-value class as an existing test (boundary companions — n==max vs max+1, absent/empty/nil UUID — are never near-duplicates)
## Severity Classification
- **P0 (Block merge)**: demonstrated data loss, security breach, remote crash, or deadlock
- **P1 (Must fix)**: concrete correctness, compatibility, or material hot-path regression
- **P2 (Should fix)**: avoidable duplication or maintainability issue with a concrete simpler replacement
- **P3 (Nice to fix)**: local style or clarity issue with no behavioral risk
- **P0 (Block merge)**: `unwrap()` in request hot path, silent truncation on user input, lock ordering violation, recursion without depth limit
- **P1 (Must fix)**: `Result<_, String>` in public API, unnecessary clone in hot path, `Box<dyn Error>` in trait method, `unwrap_or_default()` on a domain-required value (metadata, quorum, version id)
- **P2 (Should fix)**: Missing `assert!` in test, `println!` in production, missing `with_capacity`, new helper duplicating an existing workspace utility, defensive branch with no nameable trigger (corrupt or stale persisted/peer data is always a nameable trigger for boundary-crossing values), near-duplicate test, redundant error re-wrapping
- **P3 (Nice to fix)**: Naming convention violation, missing doc comment, `as_ptr()` vs `Arc::ptr_eq`, narrating comment
## Output Template
@@ -107,10 +107,10 @@ For every Rust code change, verify:
## Rust Code Quality Report
### Automated Scan
- unwrap/expect candidates inspected: N
- numeric-cast candidates inspected: N
- error-type candidates inspected: N
- output-macro candidates inspected: N
- unwrap/expect in production: N found
- as casts: N found
- String errors: N found
- println/eprintln: N found
### Findings
- [P1] `path:line` — description
@@ -0,0 +1,52 @@
# Rust Code Quality Checklist
Use this as a quick pre-merge checklist for every Rust code change.
## Critical (P0 — block merge)
| Check | Command |
|-------|---------|
| No `unwrap()` in request/storage hot path | `rg '\.unwrap\(\)' <files> \| grep -v test` |
| No `as` truncation on user input | `rg ' as (u32\|usize\|i32)' <files>` |
| Lock order consistent across call sites | Manual: trace all lock acquisitions |
| Recursive functions have depth limit | Manual: check for `max_depth` or iterative pattern |
| No `panic!`/`unwrap_or_else(panic!)` in production | `rg 'panic!\|unwrap_or_else.*panic' <files> \| grep -v test` |
## High (P1 — must fix)
| Check | Command |
|-------|---------|
| No `Result<_, String>` in public API | `rg 'Result<.*String>' <files> \| grep -v test` |
| No `Box<dyn Error>` in public trait | `rg 'Box<dyn.*Error' <files> \| grep -v test` |
| No unnecessary `.clone()` in hot path | Manual: check loops and per-request paths |
| `Error::source()` implemented when inner error stored | Manual: check `impl Error` |
| No `eprintln!`/`println!` in production | `rg 'println!\|eprintln!' <files> \| grep -v test` |
## Medium (P2 — should fix)
| Check | Command |
|-------|---------|
| Tests have assertions | Manual: check for `assert` in test functions |
| `HashMap`/`Vec` use `with_capacity` when size known | Manual: check `::new()` in loops |
| No `#![allow(dead_code)]` at crate root | `rg 'allow.dead_code' <files> \| grep 'lib.rs'` |
| Serde structs from untrusted input have `deny_unknown_fields` | Manual: check `#[derive(Deserialize)]` |
## Low (P3 — nice to fix)
| Check | Command |
|-------|---------|
| No camelCase statics | `rg 'static ref [a-z]' <files>` |
| `Arc::ptr_eq` instead of `as_ptr + ptr::eq` | `rg 'as_ptr\|ptr::eq' <files>` |
| Public functions have doc comments | `rg 'pub fn' <files> \| grep -v '///'` |
## Quick One-Liner
```bash
# Run all automated checks on changed files
CHANGED=$(git diff --name-only HEAD~1 -- '*.rs' | grep -v test | grep -v bench)
echo "=== unwrap/expect ===" && rg -c '\.unwrap\(\)|\.expect\(' $CHANGED 2>/dev/null
echo "=== as casts ===" && rg -c ' as (u8|u16|u32|u64|usize|i8|i16|i32|i64|isize)\b' $CHANGED 2>/dev/null
echo "=== String errors ===" && rg -c 'Result<.*String>' $CHANGED 2>/dev/null
echo "=== println ===" && rg -c 'println!|eprintln!' $CHANGED 2>/dev/null
echo "=== Ordering::Relaxed ===" && rg -c 'Ordering::Relaxed' $CHANGED 2>/dev/null
```
@@ -1,6 +1,6 @@
---
name: rustfs-logging-governance
description: Standardize and review RustFS logging with structured `tracing` events, lower noise on hot paths, preserve security-sensitive diagnostics, and extend guardrails to prevent legacy logging patterns from returning. Use whenever a change adds or edits any `tracing` macro call (`error!`/`warn!`/`info!`/`debug!`/`trace!`/`#[instrument]`) — including a single log line added in passing while fixing unrelated logic, which is how most new log sites enter the repo — and when reviewing RustFS logs, startup/config diagnostics, cloud metadata logs, request validation logs, or `scripts/check_logging_guardrails.sh`.
description: Standardize and review RustFS logging with structured `tracing` events, lower noise on hot paths, preserve security-sensitive diagnostics, and extend guardrails to prevent legacy logging patterns from returning. Use when editing or reviewing RustFS logs, startup/config diagnostics, cloud metadata logs, request validation logs, or `scripts/check_logging_guardrails.sh`.
---
# RustFS Logging Governance
+16 -80
View File
@@ -1,21 +1,19 @@
---
name: rustfs-release-publish
description: "End-to-end RustFS release pipeline: first publish any merged-but-unreleased rustfs/console changes and wait for its latest Release asset, then bump RustFS version files on main directly to the final target, publish a visible GitHub prerelease from a preview tag without updating latest channels, validate it, and publish the final tag on the SAME commit. Use whenever the user wants to release/publish a RustFS version (发版/发布)."
description: "End-to-end RustFS release pipeline: bump version files on main directly to the final target version, cut a preview tag on that commit, verify the CI build and release artifacts, run the downloaded binary locally and exercise the console, validate the server with the latest rc client, then publish the final tag on the SAME validated commit — never a new bump commit, never latest main. Use whenever the user wants to release/publish a RustFS version (发版/发布)."
---
# RustFS Release Publish (preview-validated pipeline)
This skill orchestrates a full release. It wraps `rustfs-release-version-bump` (which only edits version files and opens the PR) with a mandatory preview-tag validation loop before the final tag is published.
Core design: **version files never carry a `-preview.N` suffix**. The preview suffix exists only in tag names. A preview tag creates a visible GitHub Release marked Prerelease and uploads versioned assets, but it never becomes GitHub Latest and never updates `*-latest`, `latest.json`, R2, Docker, or Helm channels. This works because the binary self-reports the git tag it was built from (`build::TAG` via shadow_rs, see `rustfs/src/config/cli.rs` `SHORT_VERSION`), and `build.yml` derives artifact names and preview classification from the tag name — Cargo.toml's version is only a no-tag fallback. Therefore the preview tag and the final tag can (and MUST) point at the exact same commit: what you validated is byte-for-byte the source that ships.
Core design: **version files never carry a `-preview.N` suffix**. The preview suffix exists only in tag names. This works because the binary self-reports the git tag it was built from (`build::TAG` via shadow_rs, see `rustfs/src/config/cli.rs` `SHORT_VERSION`), and `build.yml` derives artifact names and prerelease classification from the tag name — Cargo.toml's version is only a no-tag fallback. Therefore the preview tag and the final tag can (and MUST) point at the exact same commit: what you validated is byte-for-byte the source that ships.
Pipeline shape:
```
check console main against its latest Release
-> if ahead: publish console -> wait for Release asset + latest API
-> bump RustFS version files to <target> (final version, ONE commit) -> merge
bump version files to <target> (final version, ONE commit) -> merge
-> tag <preview-tag> at that commit -> CI green
-> verify preview Release assets -> run binary locally + console checks
-> verify release artifacts -> run binary locally + console checks
-> validate with latest rc client
-> tag <target> at the SAME commit (zero delta) -> re-verify CI/release
```
@@ -25,7 +23,7 @@ On validation failure: fix lands on main via normal PR (version files are alread
## Required inputs
- Final target version, for example `1.0.0-beta.10`.
- Preview iteration `N` (default: next unused preview tag for that target; check with `git tag -l '<target>-preview.*'` after `git fetch --tags`).
- Preview iteration `N` (default: next unused preview tag for that target; check with `git tag -l '<target>-preview.*'` — and for stable targets `git tag -l '<target>-rc.*'` after `git fetch --tags`).
If the target version is missing or ambiguous, stop and ask before doing anything (see the semver gate below).
@@ -47,18 +45,14 @@ Rules:
## Preview tag naming
- Use `<target>-preview.N` for every target, e.g. `1.0.0-beta.10-preview.3` or `1.1.0-preview.1`.
- The canonical suffix is exactly `-preview.<digits>`. `build.yml` recognizes it before alpha/beta/rc classification and routes it to the preview-only path; any other tag containing `-preview` fails closed instead of being treated as a release.
- A preview Release MUST be published with `isPrerelease=true` and `isLatest=false`. Any `*-latest` preview asset or preview-triggered `latest.json`, R2, Docker, or Helm publication is a pipeline failure.
- Prerelease target (contains `alpha`/`beta`/`rc`): preview tag is `<target>-preview.N`, e.g. `1.0.0-beta.10-preview.3`. It contains `beta`, so `build.yml`'s substring-based classification marks it prerelease — safe.
- **Stable** target (e.g. `1.1.0`): NEVER tag `1.1.0-preview.N``build.yml` marks a tag prerelease only if its name contains `alpha`, `beta`, or `rc`, so `1.1.0-preview.N` would be treated as a stable release and overwrite `latest.json` as stable. Use `1.1.0-rc.N` as the preview tag instead.
## Hard rules
- Version files (Cargo.toml, Cargo.lock, README, flake.nix, Chart.yaml, rustfs.spec) are bumped ONCE, directly to `<target>`. Never write a `-preview.N` suffix into any version file. If `rustfs-release-version-bump` is ever asked for a `-preview` version, that is a pipeline bug — stop.
- Preview Release assets are versioned and intentionally visible on the Releases page. Do not label them Latest or use them to update any latest distribution channel.
- Tags have no `v` prefix. Always annotated: `git tag -a <tag> -m "Release <tag>"`.
- The final tag MUST point at exactly `PREVIEW_HASH` — the commit the validated preview tag points at. Never tag current `main` HEAD (commits merged after validation are unvalidated), and never create an extra version-bump commit between preview and final.
- When a previous deliverable exists, GitHub Release notes for the preview and final tags MUST use it as their shared comparison baseline: the most recently published non-preview Release before the target. Internal `-preview.N` Releases are explicitly excluded from that selection, even when they point at the same commit as the final tag. If no previous deliverable exists, omit `previous_tag_name` and record that GitHub's default baseline fallback was used.
- Generated Release notes carry a workflow-management marker so retries can repair them. Before manually curating a generated body, remove that marker; unmarked non-placeholder notes are preserved by later workflow runs.
- Phases run in order; a failure in any phase blocks everything after it. After the fix lands on main, restart from Phase 2 with the next preview iteration against the new `origin/main` hash — do not resume mid-pipeline against a stale hash.
- If the release is abandoned after Phase 1 merged, main's version files claim a version that was never tagged. Either revert the bump PR or leave it to be overwritten by the next release — but tell the user explicitly and record the decision.
- User-facing status updates in Chinese; commits, PR titles/bodies, and tag messages in English. No hard-wrapping in commit messages, PR bodies, or documentation prose — one logical line per sentence/paragraph, let soft wrap handle display.
@@ -69,61 +63,6 @@ Rules:
- `gh auth status` works; confirm you can view `gh release list -L 3`.
- Confirm the exact final target version with the user if not explicit.
### Console release gate
Complete this gate before changing any RustFS version file or creating any RustFS tag. RustFS `build.yml` downloads the asset returned by `repos/rustfs/console/releases/latest`, so a successful Console build alone is insufficient.
1. Read the latest published Console tag and compare it with Console `main`:
```bash
CONSOLE_REPO="rustfs/console"
CONSOLE_LATEST=$(gh api "repos/${CONSOLE_REPO}/releases/latest" --jq .tag_name)
gh api "repos/${CONSOLE_REPO}/compare/${CONSOLE_LATEST}...main" \
--jq '{status, ahead_by, behind_by, commits: [.commits[] | {sha, message: .commit.message}]}'
```
- `ahead_by == 0`: no merged Console change is waiting for release. Still verify the current latest asset using step 4, then continue to Phase 1.
- `ahead_by > 0` and `behind_by == 0`: publish Console before continuing. Report the merged commits and select the next unused `vX.Y.Z` tag. Default to the next patch version when the changes are fixes or backward-compatible UI work; stop for confirmation if a minor/major bump is plausible.
- Any diverged history or `behind_by > 0`: stop and resolve the Console release baseline explicitly. Do not guess a range or publish RustFS.
2. Clone/fetch `rustfs/console` into a scratch directory and record its exact `main` commit. Before creating a tag, check for a `v*` tag or Release workflow already associated with that hash. If one is in progress, wait for it instead of creating another version:
```bash
CONSOLE_SCRATCH=$(mktemp -d)
gh repo clone "$CONSOLE_REPO" "$CONSOLE_SCRATCH/console"
git -C "$CONSOLE_SCRATCH/console" fetch origin main --tags
CONSOLE_HASH=$(git -C "$CONSOLE_SCRATCH/console" rev-parse origin/main)
git -C "$CONSOLE_SCRATCH/console" tag --points-at "$CONSOLE_HASH" 'v*'
gh run list -R "$CONSOLE_REPO" --workflow release.yml --commit "$CONSOLE_HASH" --limit 5
```
If no release exists or is running for `CONSOLE_HASH`, create the selected annotated tag at that exact hash and push it:
```bash
git -C "$CONSOLE_SCRATCH/console" tag -a "<console-tag>" -m "Release <console-tag>" "$CONSOLE_HASH"
git -C "$CONSOLE_SCRATCH/console" push origin "<console-tag>"
```
Console tags include the `v` prefix. Pushing the tag triggers `.github/workflows/release.yml` (`🚀 Release`). Remove `CONSOLE_SCRATCH` after the gate completes.
3. Find the exact tag run and wait for completion:
```bash
gh run list -R "$CONSOLE_REPO" --workflow release.yml --branch "<console-tag>" --limit 1
gh run watch -R "$CONSOLE_REPO" "<console-run-id>" --exit-status
```
4. Block until the published Release is non-draft, the latest endpoint returns the expected tag, and `rustfs-console-<console-tag>.zip` is uploaded, non-empty, and carries a `sha256:` digest:
```bash
gh release view -R "$CONSOLE_REPO" "<console-tag>" --json isDraft,isPrerelease,assets,url
test "$(gh api "repos/${CONSOLE_REPO}/releases/latest" --jq .tag_name)" = "<console-tag>"
test "$(gh api "repos/${CONSOLE_REPO}/releases/tags/<console-tag>" \
--jq '[.assets[] | select(.name == "rustfs-console-<console-tag>.zip" and .state == "uploaded" and .size > 0 and (.digest | startswith("sha256:")))] | length')" -eq 1
```
Treat a missing/mismatched asset, digest, latest tag, or failed/cancelled workflow as BLOCKED. Do not start Phase 1 until the Console gate passes. Record `CONSOLE_TAG`, `CONSOLE_HASH`, Console run URL, and Release URL for the final report.
## Phase 1 — Version bump to the final target (once)
- If main's version files already read `<target>` (e.g. this is a restart after a failed preview), verify with `rg -n "<target>" Cargo.toml rustfs.spec helm/rustfs/Chart.yaml` and skip to Phase 2.
@@ -146,17 +85,16 @@ git push origin "<preview-tag>"
Pushing the tag triggers `.github/workflows/build.yml` ("Build and Release"); `docker.yml` chains off it via `workflow_run`.
The preview run builds versioned artifacts and publishes them in a GitHub prerelease. Its latest-channel, R2, Docker, and Helm jobs must be skipped. Those publication paths run only after the final tag is pushed.
On a restart (N+1), refresh `PREVIEW_HASH=$(git rev-parse origin/main)` first — it must contain the fix — and re-report it.
## Phase 3 — CI and preview Release verification
## Phase 3 — CI and artifact verification
- Find and watch the tag build: `gh run list --workflow build.yml --branch "<preview-tag>" --limit 1` then `gh run watch <run-id>`. Every build matrix target must succeed (linux x86_64/aarch64 × musl/gnu, macos-aarch64, windows-x86_64).
- Confirm the Release publication jobs (`create-release`, `upload-release-assets`, and `publish-release`) succeed while `update-latest-version` is skipped.
- Verify `gh release view "<preview-tag>" --json isPrerelease,assets,url`: `isPrerelease` must be `true`, and the Release must contain all 6 versioned platform zips, checksums, SBOM, and provenance with no `-latest` assets. Confirm `gh api repos/{owner}/{repo}/releases/latest --jq .tag_name` does not return `<preview-tag>`.
- Record `PREVIOUS_DELIVERABLE`, selected from published Releases by `publishedAt` after excluding the current tag and every `-preview.N` tag. Verify `gh release view "<preview-tag>" --json body --jq .body` contains `## What's Changed` and, when `PREVIOUS_DELIVERABLE` exists, `**Full Changelog**: https://github.com/rustfs/rustfs/compare/<PREVIOUS_DELIVERABLE>...<preview-tag>`. For a repository with no previous deliverable, verify a Full Changelog link exists and record the GitHub baseline fallback.
- Confirm preview-triggered Docker and Helm jobs are skipped. Preview validation covers the built RustFS binaries, embedded console, and rc compatibility; Docker image construction and Helm publication are deferred to the final tag because the Dockerfiles consume GitHub Release assets.
- Watch the tag build: `gh run list --workflow build.yml --limit 5` then `gh run watch <run-id>`. Every matrix target must succeed (linux x86_64/aarch64 × musl/gnu, macos-aarch64, windows-x86_64) plus the release and latest.json jobs.
- Verify the GitHub release: `gh release view "<preview-tag>" --json isPrerelease,assets`
- `isPrerelease` must be `true`.
- Assets must include all 6 platform zips in both versioned (`rustfs-<platform>-v<tag>.zip`) and `-latest` forms, plus `SHA256SUMS`, `SHA512SUMS`, `rustfs-<tag>.sbom.cdx.json`, `rustfs-<tag>.provenance.json`.
- Verify the chained Docker run succeeded: `gh run list --workflow docker.yml --limit 3`.
- Checksum spot-check for the platform you will run locally: download the zip and `SHA256SUMS`, verify with `shasum -a 256 -c` (grep to one line).
## Phase 4 — Run the artifact locally, verify the console
@@ -219,15 +157,13 @@ git push origin "<target>"
```
- CI rebuilds from the same source; the only changed input is the tag name, so the binary now self-reports `<target>`.
- Verify the final tag's complete publication path: all matrix and release jobs green; `gh release view "<target>"` shows the full versioned and `-latest` asset set plus checksums, SBOM, and provenance; Docker and Helm workflows succeed; `latest.json` points to `<target>`. A stable target must have `isPrerelease=false` and `isLatest=true`. An alpha/beta/rc target must have `isPrerelease=true`; GitHub does not permit prereleases to be Latest, but the project `latest.json` still advances to the final non-preview target.
- Verify the final Release body contains `## What's Changed` and a Full Changelog link. When `PREVIOUS_DELIVERABLE` exists, the link MUST be `https://github.com/rustfs/rustfs/compare/<PREVIOUS_DELIVERABLE>...<target>` and the baseline MUST equal the preview Release baseline; for example, both `1.0.0-beta.12-preview.1` and `1.0.0-beta.12` compare from `1.0.0-beta.11`.
- Re-run the Phase 3 verification against the final tag: all matrix jobs green; `gh release view "<target>"` shows the full asset set; for a prerelease target `isPrerelease` is `true`, for a stable target it must be `false` and `latest.json` must be updated.
- Optionally spot-check `./rustfs --version` from a final-tag artifact — it must report `<target>`.
## Output contract
Always report:
- Console gate result: previous/latest Console tags, whether merged changes required a release, `CONSOLE_HASH`, and Console run/Release URLs when a release was published.
- Target version, preview tag(s) used, `PREVIEW_HASH` (which both tags point at).
- Per-phase result (PASS/FAIL/BLOCKED) with key evidence: preview and final Release URLs, preview `isPrerelease`/`isLatest` state, final latest-channel state, console check results, and the rc command matrix.
- Per-phase result (PASS/FAIL/BLOCKED) with key evidence: CI run URLs, release URLs, console check results, the rc command matrix.
- Any deviation from this pipeline and why the user approved it.
@@ -18,7 +18,7 @@ Validated baseline: release pattern used in PR `#2957`.
If target version is missing or ambiguous, stop and ask before editing.
Reject any target version containing `-preview`: preview identifiers are tag-only (see `rustfs-release-publish`) and must never be written into version files. If asked for one, stop and point to the release pipeline instead of editing.
Reject any target version containing a `-preview.` suffix: preview identifiers are tag-only (see `rustfs-release-publish`) and must never be written into version files. If asked for one, stop and point to the release pipeline instead of editing.
## Read before editing
@@ -66,23 +66,14 @@ For the full pattern map, read [advisory-patterns.md](references/advisory-patter
### STS, OIDC, and federation flows
- Every STS endpoint must have an explicit authentication story: SigV4 where required, OIDC token verification for web identity, and role/session policy validation before issuing credentials.
- For web identity, the JWT is the credential; exemption from SigV4 is not itself an authentication bypass. Treat pre-verification claims only as untrusted routing hints, bound token size, normalize public failures, rate-limit discovery, and issue credentials only after signature, issuer, audience, and expiration checks.
- JWT session tokens must be signed and verified by a trusted issuer/key path, not by service-account-controlled material or a reused root secret.
- JWT verification must enforce required claims and expiration for every bearer token path; "allow missing exp" is never acceptable for user-presented credentials.
- Public OIDC bootstrap and callback routes must treat `Host`, `X-Forwarded-Proto`, redirect targets, `state`, and callback parameters as untrusted; credential-bearing redirects require a configured, allowlisted origin.
- OIDC discovery and validation URLs are SSRF sinks. Resolve and classify hostnames at connection time, reject rebinding to loopback/private/link-local ranges, and do not rely on literal string checks.
### IAM policy conditions and plugins
- Treat request headers as attacker-controlled even after SigV4; callers sign their own spoofed headers. Do not merge them into server-derived condition keys such as identity, groups, version ID, signature version, JWT, or LDAP claims.
- Keep the condition-key namespace explicit. Reserved server-derived keys must reject or ignore colliding headers, while intentional request-header keys such as `s3:x-amz-*` remain available.
- Quantified IAM condition tests need partially overlapping multi-value sets. Fully contained and fully disjoint sets cannot distinguish `ForAllValues` from `ForAnyValue` bugs.
- External policy plugins must receive the same security context as built-in policy evaluation. If OPA or another plugin depends on existing object tags, load and pass `ExistingObjectTag/*` before the plugin decision.
### S3 object actions, copy, multipart, and presigned POST
- Version-aware object requests need version-aware actions. Explicit `versionId` reads and copy sources must authorize `s3:GetObjectVersion`, not only `s3:GetObject`.
### S3 copy, multipart, and presigned POST
- Multipart copy must enforce source `GetObject` and destination `PutObject` semantics equivalent to `CopyObject`, including copy-source and policy conditions.
- Do not let `CreateMultipartUpload`, `UploadPartCopy`, `CompleteMultipartUpload`, or `AbortMultipartUpload` return success without authorization.
- Fallbacks from version actions to non-version actions must still pass the same public-access-block, anonymous-deny, and post-authorization gates as a direct allow.
- Presigned POST policies are server-side contracts. Enforce `content-length-range`, key prefix, exact metadata/content-type, and all signed policy conditions.
### Protocol frontends and IAM parity
@@ -108,8 +99,6 @@ For the full pattern map, read [advisory-patterns.md](references/advisory-patter
### Logging and debug output
- Logs must never include access keys beyond safe identifiers, secret keys, session tokens, JWT claims, HMAC secrets, expected signatures, license secrets, or raw response bodies containing credentials.
- Treat `Debug` implementations, `?value` tracing, merged config dumps, and dependency-level HTTP body logging as leak surfaces.
- Error and panic messages are log content: they propagate through `?` and get printed by `error!`/startup logging far from where they were constructed. Never interpolate a raw config or credential value into an error string.
- A value that fails secret-format parsing is usually the secret itself (e.g. a bare base64 key missing its `<name>:` prefix), so a parse-failure hint must name the env var or file and the expected format, never echo the input. Redacting `Debug` impls does not cover this channel.
- Add log-capture tests or targeted unit tests for redaction wrappers when changing credential structs or response bodies.
### RPC, parsing, and panic safety
@@ -141,11 +130,6 @@ For the full pattern map, read [advisory-patterns.md](references/advisory-patter
- When touching reader/writer wrappers such as hashing, encryption, compression, or warp readers, verify wrapper order and inspect stored bytes in regression tests.
- Avoid helper shortcuts that unwrap nested readers and accidentally bypass encryption or integrity layers.
### Object Lock and retention invariants
- Object Lock state must fail closed when bucket metadata is unreadable, fabricated, or unparsable. Only a confirmed absence of Object Lock configuration may permit unprotected deletes or writes.
- Do not collapse metadata read faults, missing persisted metadata, parse failures, and genuinely absent Object Lock config into one "not configured" result.
- Retention enforcement must cover foreground deletes, batch deletes, force-delete helpers, default-retention materialization on PUT, lifecycle expiry, scanner sweeps, and all-versions expiry.
## Review Prompts
Use these prompts while reviewing a diff:
@@ -155,16 +139,11 @@ Use these prompts while reviewing a diff:
- Does a public/default/empty config change security behavior from fail-closed to fail-open?
- Is any attacker-controlled value later used as a path, policy condition, credential identity, log field, URL, Origin, or response body?
- Does this response contain stored replication, remote target, or service credentials that need redaction or stricter authorization?
- Does any error constructor or `format!` interpolate a variable that can hold secret material, including a config parse error that echoes the raw input?
- Does an IAM export/import path expose or trust plaintext credential secrets beyond the caller's intended authority?
- Can this STS/OIDC path issue credentials without SigV4, trusted issuer validation, allowlisted redirects, or trusted-proxy host/scheme handling?
- Can a service-account or STS token omit `exp`, forge `sessionPolicy`, or use a principal-controlled key as signing authority?
- Does this outbound validation path resolve attacker-supplied hostnames and reject private, loopback, link-local, and rebound addresses at the actual connection boundary?
- Is an archive entry, object key, or policy resource normalized differently between authorization and storage?
- Is the same operation implemented in multiple paths, such as `CopyObject` vs `UploadPartCopy`, and do all paths enforce the same security contract?
- Does an explicit object version, fallback action, or plugin authorization path pass through the same action and post-authorization gates as the direct S3 path?
- Can a caller-controlled header populate a condition key that should be derived only by the server?
- Do condition tests include partially overlapping multi-value inputs for quantified operators?
- Does unreadable bucket metadata make Object Lock or retention enforcement fail closed rather than disappear?
- Does a preview or browser-surface fix preserve the original security invariant when adding alternate viewers or file-type detection?
- Does the test prove the exploit form is denied, or only that the intended form still works?
@@ -35,21 +35,12 @@ Update this file only when an advisory adds or changes a reusable lesson, affect
### STS, OIDC, and federation flows
- `GHSA-5qfg-mf7r-jp3w` and `GHSA-3473-5353-xhwh`: `AssumeRoleWithWebIdentity` was reachable through unauthenticated `POST /` routing and could issue temporary credentials from crafted web identity input. Lesson: every STS route needs explicit SigV4 or trusted identity-provider validation before role assumption, and unauthenticated exemptions must be narrowed to the exact action with uniform failure responses.
- `GHSA-jxrr-r6pv-h958`: unsigned JWT issuer data was decoded before verification to select an OIDC provider, and distinguishable failures could expose provider configuration. Lesson: web-identity routing may be unauthenticated, but pre-verification claims are untrusted routing hints; bound and rate-limit the request, normalize public errors, and verify signature, issuer, audience, and expiration before issuing credentials.
- `GHSA-ccrv-v8v9-ch9q`, `GHSA-48rf-7j3q-3hfv`, and `GHSA-xvfh-7c9g-hpw2`: service-account-controlled material could self-sign JWT session tokens with forged policy claims, and missing `exp` was accepted for service-account tokens. Lesson: session tokens must be signed by a trusted issuer/key path, enforce required claims and expiration, and reject self-signed or principal-controlled tokens.
- `GHSA-ccrv-v8v9-ch9q` and `GHSA-48rf-7j3q-3hfv`: service-account-controlled material could self-sign JWT session tokens with forged policy claims, and missing `exp` was accepted for service-account tokens. Lesson: session tokens must be signed by a trusted issuer/key path, enforce required claims and expiration, and reject self-signed or principal-controlled tokens.
- `GHSA-9pjf-w3c2-m32r`, `GHSA-4x2q-cpx9-9h26`, and `GHSA-xvpm-p3f7-34c3`: public OIDC authorize/callback flows trusted request `Host` or forwarded scheme when building credential-bearing redirects. Lesson: OIDC redirects must use configured allowlisted origins and trusted-proxy handling; never derive the post-login credential destination from direct client headers.
- `GHSA-m479-9x88-94w6`, `GHSA-frwq-mfqx-83p8`, `GHSA-q9q8-rf9r-fg9f`, and `GHSA-j5c2-hhf7-6gf5`: OIDC validation accepted attacker-controlled discovery URLs because hostname checks rejected only literal forbidden IPs, allowing DNS rebinding SSRF. Lesson: outbound federation URL validation must resolve and classify hostnames at the connection boundary and reject loopback, private, link-local, and rebound addresses.
### IAM policy conditions and external policy plugins
### S3 copy, multipart, and upload policy validation
- `GHSA-6r96-hmgc-726c`: request headers collided with lowercase server-derived condition keys such as `userid`, `groups`, `versionid`, and JWT/LDAP claims. Lesson: never let caller-controlled headers append to or replace server-derived policy context; reserve trusted condition keys and keep intentional request-header keys separate.
- `GHSA-v9cp-qfw9-9pfp`: quantified negated string conditions applied negation after aggregation, transposing `ForAllValues` and `ForAnyValue` semantics. Lesson: push negation into the per-value predicate for quantified operators and test partially overlapping multi-value sets.
- `GHSA-5w8r-p896-6vq2`: OPA policy mode skipped `ExistingObjectTag/*` loading, so tagged objects looked untagged to external policies. Lesson: external authorization plugins need the same object-tag and request context as built-in policy evaluation before they decide.
### S3 object actions, copy, multipart, and upload policy validation
- `GHSA-3ppv-fx5m-m749`: explicit `versionId` reads and copy sources authorized `s3:GetObject` instead of `s3:GetObjectVersion`. Lesson: version-specific object access must select version-specific actions for direct reads, `CopyObject`, and `UploadPartCopy`, with tests proving the backend is not reached on denial.
- `GHSA-x298-9x87-fvjq`: anonymous `ListObjectVersions` fell back to `ListBucket` and returned before public-access-block gates. Lesson: compatibility fallbacks must converge on the same post-authorization checks as direct grants, especially `RestrictPublicBuckets` and anonymous data-plane denies.
- `GHSA-mx42-j6wv-px98`: `UploadPartCopy` missed source authorization and allowed cross-bucket object exfiltration. Lesson: multipart copy must enforce the same source and destination contract as `CopyObject`.
- `GHSA-wfxj-ph3v-7mjf`: `UploadPartCopy` checked source and destination independently but missed destination copy-source policy constraints. Lesson: source read and destination write checks are not sufficient when policy constrains allowed copy sources.
- `GHSA-w5fh-f8xh-5x3p`: presigned POST accepted uploads without enforcing signed policy conditions. Lesson: parse and enforce all POST policy constraints server-side, including size, key prefix, and content type.
@@ -68,7 +59,7 @@ Update this file only when an advisory adds or changes a reusable lesson, affect
### Secrets, defaults, and cryptographic misuse
- `GHSA-j59h-h7q5-q348`, `GHSA-3wm5-wpm5-hmfm`, `GHSA-6wc8-xm48-qhmx`, `GHSA-9gf3-jx4p-4xxf`, `GHSA-63xc-c3w3-m2cf`, and `GHSA-ch63-6q4v-hwp5`: RustFS shipped known default root credentials that could authenticate to S3, admin APIs, IAM, KMS, console, and token-signing surfaces. Lesson: root credentials must be operator-provided or generated per install; known defaults and warnings are not acceptable for network-reachable deployments.
- `GHSA-j59h-h7q5-q348`, `GHSA-3wm5-wpm5-hmfm`, `GHSA-6wc8-xm48-qhmx`, `GHSA-9gf3-jx4p-4xxf`, and `GHSA-63xc-c3w3-m2cf`: RustFS shipped known default root credentials that could authenticate to S3, admin APIs, IAM, KMS, console, and token-signing surfaces. Lesson: root credentials must be operator-provided or generated per install; known defaults and warnings are not acceptable for network-reachable deployments.
- `GHSA-h956-rh7x-ppgj`: gRPC used the hard-coded token `rustfs rpc` on both client and server. Lesson: source-visible shared tokens are authentication bypasses.
- `GHSA-r5qv-rc46-hv8q`: internode RPC HMAC secret fell back to the public default `rustfsadmin`. Lesson: RPC/internode auth must fail closed instead of silently using public defaults.
- `GHSA-75fx-qg6f-8rm7` and `GHSA-68cw-96m3-h2cf`: internode RPC secrets were derivable from known root credentials, making raw storage RPC signatures forgeable when explicit RPC secrets were unset. Lesson: RPC auth keys must be independent random secrets, never derived from S3 root credentials, and raw storage RPC should not share the public S3 listener without an internode-only boundary.
@@ -101,10 +92,6 @@ Update this file only when an advisory adds or changes a reusable lesson, affect
- `GHSA-xrrf-67jm-3c2r`: SSE metadata reported encryption while reader composition bypassed `EncryptReader` and stored plaintext. Lesson: test actual bytes on disk and wrapper order, not only API metadata.
### Object Lock and retention invariants
- `GHSA-j548-9grx-fh4f`: Object Lock enforcement treated unreadable, fabricated, or unparsable bucket metadata as absent configuration and allowed retained objects to be deleted or expired. Lesson: retention must fail closed unless Object Lock absence is authoritative, and every delete, lifecycle, scanner, force-delete, and default-retention path needs the same state distinction.
### Serde deserialization and input validation
- No `#[serde(deny_unknown_fields)]` found across the entire codebase. Lesson: all structs deserialized from untrusted input (S3 API XML/JSON, lifecycle rules, bucket policies, replication configs) should have `#[serde(deny_unknown_fields)]` to reject malformed or adversarial payloads.
@@ -120,13 +107,11 @@ Use these targeted searches when a diff touches security-sensitive code:
rg -n "validate_admin_request|check_permissions|AdminAction::|deny_only|is_allowed" rustfs crates
rg -n "authorize_operation|FtpsDriver|SftpDriver|RETR|MKD|SIZE|MDTM|CreateBucket|GetObject|HeadObject" crates/protocols rustfs
rg -n "UploadPartCopy|upload_part_copy|CompleteMultipart|PostObject|content-length-range|starts-with" rustfs crates
rg -n "ListBucketVersions|GetObjectVersion|versionId|VersionId|ExistingObjectTag|ForAllValues|ForAnyValue|POLICY_PLUGIN|opa" rustfs crates
rg -n "normalize_extract_entry_key|Snowball|auto-extract|PathBuf::join|canonicalize|\\.\\.|x-forwarded-for|x-real-ip|SourceIp" rustfs crates
rg -n "DEFAULT_SECRET|DEFAULT_ACCESS|TEST_PRIVATE_KEY|rustfs rpc|RUSTFS_RPC_SECRET" rustfs crates
rg -n "TONIC_RPC_PREFIX|verify_rpc_signature|check_auth|NodeServiceServer|x-rustfs-signature" rustfs crates
rg -n "debug!|trace!|info!|error!|\\?resp|\\?merged_config|session_token|secret_key" rustfs crates
rg -n "HashReader|EncryptReader|SSE|server-side encryption|Access-Control-Allow-Credentials|Origin" rustfs crates
rg -n "ObjectLock|object_lock|retention|COMPLIANCE|GOVERNANCE|delete_prefix|lifecycle|scanner" rustfs crates
rg -n "deny_unknown_fields|serde.default|as u32|as usize|as i32" rustfs crates
```
@@ -136,12 +121,9 @@ rg -n "deny_unknown_fields|serde.default|as u32|as usize|as i32" rustfs crates
- Protocol frontend authz fixes: include denied `RETR`, `SIZE`/`MDTM`, `MKD`, bucket probe, and sibling allowed-operation cases, and assert denied paths do not reach the storage backend.
- IAM fixes: include import/update/list service-account cases with attacker-controlled parent, claims, access key, secret key, and policy.
- Copy/upload fixes: include cross-bucket, cross-user, source-denied, destination-denied, copy-source-condition, and multipart completion cases.
- Version-action fixes: include historical UUID, explicit current version, `null`, range, partNumber, presigned, STS/session, service-account, anonymous bucket-policy, copy source, and multipart-copy source cases.
- Policy-condition fixes: include reserved-key header collisions, missing keys, partially overlapping multi-value sets, plugin mode, and built-in policy mode.
- Path fixes: include encoded traversal, absolute path, nested traversal, archive entries with `..`, valid object keys that resemble traversal text but should be rejected, and canonical bucket/prefix boundary checks.
- Logging fixes: assert redacted output for structs and response bodies that may contain credentials.
- IAM export fixes: assert exported archives omit plaintext user and service-account secrets unless the format deliberately encrypts or seals them.
- RPC auth fixes: include captured metadata replay across two concrete methods, stale timestamps, wrong path, wrong method surrogate, wrong secret, and valid same-method calls.
- Browser/CORS fixes: assert no credentials on reflected/default origins, correct behavior for explicit allowlists, and no same-origin script execution for previewed object content.
- SSE fixes: inspect stored bytes and verify API metadata, read-back behavior, and on-disk ciphertext together.
- Object Lock fixes: include unreadable metadata, fabricated metadata defaults, unparsable config, confirmed absent config, COMPLIANCE/GOVERNANCE retention, lifecycle expiry, scanner sweeps, and force-delete paths.
-10
View File
@@ -60,16 +60,6 @@ body-cache-whitelist-check: ## Check the body-cache eligibility gate stays a fai
@echo "🧱 Checking body-cache whitelist guard..."
./scripts/check_body_cache_whitelist.sh
.PHONY: s3s-footprint-check
s3s-footprint-check: ## Check the s3s dependency footprint ratchet stays frozen
@echo "📦 Checking s3s footprint ratchet..."
./scripts/check_s3s_footprint.sh
.PHONY: fips-wording-check
fips-wording-check: ## Check outward docs do not make unsupported FIPS claims
@echo "📣 Checking FIPS wording guard..."
./scripts/check_fips_wording.sh
.PHONY: log-analyzer-rules-check
log-analyzer-rules-check: core-deps ## Check log-analyzer rule anchors still exist verbatim in source
@echo "🩺 Checking log-analyzer rule anchors..."
+3 -3
View File
@@ -19,13 +19,13 @@ planning-docs-check: ## Check that no planning-type documents are committed
./scripts/check_no_planning_docs.sh
.PHONY: pre-commit
pre-commit: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check s3s-footprint-check fips-wording-check doc-paths-check planning-docs-check quick-check ## Run fast pre-commit checks without clippy/full tests
pre-commit: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check doc-paths-check planning-docs-check quick-check ## Run fast pre-commit checks without clippy/full tests
@echo "✅ All pre-commit checks passed!"
.PHONY: pre-pr
pre-pr: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check s3s-footprint-check fips-wording-check doc-paths-check planning-docs-check log-analyzer-rules-check clippy-check test ## Run full pre-PR checks with clippy and tests
pre-pr: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check doc-paths-check planning-docs-check log-analyzer-rules-check clippy-check test ## Run full pre-PR checks with clippy and tests
@echo "✅ All pre-PR checks passed!"
.PHONY: dev-check
dev-check: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check s3s-footprint-check fips-wording-check doc-paths-check planning-docs-check quick-check ## Run fast local development checks
dev-check: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check doc-paths-check planning-docs-check quick-check ## Run fast local development checks
@echo "✅ Fast development checks passed!"
-8
View File
@@ -25,15 +25,7 @@ TEST_THREADS ?= 1
script-tests: ## Run shell script tests
@echo "Running script tests..."
./scripts/test_build_rustfs_options.sh
./scripts/test_docker_runtime_timezone.sh
./scripts/test_entrypoint_credentials.sh
./scripts/test_internode_grpc_ab_bench.sh
./scripts/test_object_batch_bench_enhanced.sh
./scripts/test_hotpath_warp_ab_gate.sh
./scripts/test_hotpath_warp_abba.sh
./scripts/test_exact_1mib_handoff_abba.sh
./scripts/test_pinned_paired_abba_bench.sh
./scripts/test_manual_transition_runbooks.sh
bash -n ./scripts/validate_object_data_cache_cold_stampede.sh
python3 ./scripts/check_object_data_cache_follower_samples.py --self-test
./scripts/validate_object_data_cache_cold_stampede.sh --self-test
+34 -114
View File
@@ -1,16 +1,17 @@
# nextest configuration for RustFS.
#
# Serialize the ecstore tests that share the process-wide disk registry or
# exercise a multi-disk commit handoff across nextest process boundaries.
# Serialize two known load-sensitive / global-state-sharing ecstore test groups
# so the full parallel nextest suite stops producing spurious failures
# (backlog #937). These tests pass in isolation but flake under the loaded
# parallel run for two distinct reasons:
#
# * store::bucket::tests::bucket_delete_* share process/global state (disk
# registry, lock client) and race make_bucket into InsufficientWriteQuorum
# when run concurrently with other ecstore tests.
# * bucket_lifecycle_ops::tests::concurrent_resend_same_part_commits_one_generation
# uses the shared multipart fixture and a deterministic uploadId-lock
# handoff, so it must not overlap another process mutating that fixture.
# * bucket::metadata_sys::tests::concurrent_config_writes_from_separate_nodes_do_not_lose_writes
# uses the shared transaction lock and must not overlap other ecstore tests.
# asserts a lock-acquire correctness property whose serialized cross-disk
# commits exceed the (already max'd, 60s) acquire deadline only when the
# suite saturates disk I/O.
#
# serial_test's #[serial] attribute does NOT serialize these across runs:
# nextest executes each test in its own process, where the in-process
@@ -29,23 +30,19 @@
[test-groups]
ecstore-serial-flaky = { max-threads = 1 }
embedded-test-ports = { max-threads = 1 }
e2e-vault = { max-threads = 1 }
# Reliability / fault-injection e2e tests each spawn a single-node 4-disk RustFS
# server and manipulate its disk directories at runtime (crates/e2e_test:
# reliability_disk_fault_test, degraded_read_eof_regression_test / dist-13, and
# replacement_privileged_e2e_test when explicitly run as root on Linux). They
# reliability_disk_fault_test, degraded_read_eof_regression_test / dist-13). They
# are correct in isolation but resource-heavy; serialize them under nextest's
# process boundary (serial_test's #[serial] does not cross it) so several 4-disk
# servers never run at once. ci-7's nightly picks these up via the e2e suite;
# they are deliberately NOT in the fast PR `e2e-smoke` filter.
e2e-reliability = { max-threads = 1 }
e2e-inline-boundaries = { max-threads = 1 }
# --- default profile (local): serialize the flaky groups, never retry --------
[[profile.default.overrides]]
filter = 'package(rustfs-ecstore) & (test(concurrent_resend_same_part_commits_one_generation) | test(concurrent_config_writes_from_separate_nodes_do_not_lose_writes) | test(/^store::bucket::tests::bucket_delete_(mark_delete|purge_removes|default_s3_delete)/))'
filter = 'package(rustfs-ecstore) & (test(concurrent_resend_same_part_commits_one_generation) | test(/^store::bucket::tests::bucket_delete_(mark_delete_marks|purge_removes|default_s3_delete)/))'
test-group = 'ecstore-serial-flaky'
# Serialize the multipart crash-consistency scenarios (dist-2, backlog#1150):
@@ -57,53 +54,13 @@ test-group = 'ecstore-serial-flaky'
filter = 'package(rustfs-ecstore) & test(/^set_disk::ops::multipart::tests::crash_consistency::/)'
test-group = 'ecstore-serial-flaky'
# The production-handler relocation regression builds an isolated 8-disk,
# 2-pool store and commits a 72 MiB multipart object. Keep that cross-disk IO
# from overlapping the ecstore commit fixtures above.
[[profile.default.overrides]]
filter = 'package(rustfs) & test(execute_get_object_resumes_from_relocated_pool_without_splicing_body)'
test-group = 'ecstore-serial-flaky'
# Embedded integration-test binaries discover an ephemeral port and release
# the probe listener before RustFS binds it. Serialize that cross-process
# TOCTOU window; retries would only hide real startup failures.
[[profile.default.overrides]]
filter = 'package(rustfs) & (binary(/^embedded.*_test$/) | binary(admin_diagnostic_capability_e2e))'
test-group = 'embedded-test-ports'
# Serialize the durable manual-transition checkpoint test across nextest's
# process boundary; it mutates bucket lifecycle metadata and is not quarantined.
[[profile.default.overrides]]
filter = 'package(rustfs-ecstore) & test(manual_transition_page_checkpoint_persists_durable_job_progress)'
test-group = 'ecstore-serial-flaky'
# Serialize the bucket-incarnation / lifecycle-fence tests. They drive
# init_bucket_metadata_sys and bucket_metadata_sys_of, i.e. process-global
# OnceLock state that serial_test's #[serial] cannot protect across nextest's
# process boundary, and they delete+recreate buckets — the same shape that
# raced into InsufficientWriteQuorum in backlog#937. Preventive only, no
# retries. The matching ci-profile override is after [profile.ci].
[[profile.default.overrides]]
filter = 'package(rustfs-ecstore) & (test(bucket_delete_waits_for_config_mutation_fence) | test(stale_config_request_cannot_mutate_a_recreated_bucket) | test(disk_incarnation_read_detects_stale_cache_until_peer_reload) | test(lifecycle_expiry_fails_closed_on_corrupt_object_lock_metadata) | test(expiry_configs_are_resolved_from_the_owning_store))'
test-group = 'ecstore-serial-flaky'
# Serialize the 4-disk reliability / degraded-read e2e tests (see the
# e2e-reliability test-group note above). The matching ci-profile override is at
# the end of the file, after [profile.ci] is declared.
[[profile.default.overrides]]
filter = 'package(e2e_test) & test(/^(reliability_disk_fault|degraded_read_eof_regression|replacement_privileged_e2e)_test::/)'
filter = 'package(e2e_test) & test(/^(reliability_disk_fault|degraded_read_eof_regression)_test::/)'
test-group = 'e2e-reliability'
[[profile.default.overrides]]
filter = 'package(e2e_test) & test(/^inline_fast_path_cluster_test::/)'
test-group = 'e2e-inline-boundaries'
# Vault KMS tests share the fixed dev-server port 8200. serial_test's #[serial]
# does not cross nextest process boundaries, so keep these tests in one group.
[[profile.default.overrides]]
filter = 'package(e2e_test) & test(/^kms::kms_vault_test::/)'
test-group = 'e2e-vault'
# ---------------------------------------------------------------------------
# ci profile — the strict CI gate (ci.yml `cargo nextest run --profile ci`)
# ---------------------------------------------------------------------------
@@ -132,17 +89,19 @@ path = "junit.xml"
# profile's own overrides list, not the default profile's).
# ===========================================================================
# QUARANTINE: OPEN backlog#937 — store::bucket::tests::bucket_delete_* race
# make_bucket into InsufficientWriteQuorum via shared global state under load.
# QUARANTINE: OPEN backlog#937 — concurrent_resend lock-acquire deadline flakes
# under saturated disk I/O in the full parallel suite.
[[profile.ci.overrides]]
filter = 'package(rustfs-ecstore) & test(/^store::bucket::tests::bucket_delete_(mark_delete|purge_removes|default_s3_delete)/)'
filter = 'package(rustfs-ecstore) & test(concurrent_resend_same_part_commits_one_generation)'
test-group = 'ecstore-serial-flaky'
retries = 2
# Keep deterministic ECStore write handoffs isolated across nextest processes.
# QUARANTINE: OPEN backlog#937 — store::bucket::tests::bucket_delete_* race
# make_bucket into InsufficientWriteQuorum via shared global state under load.
[[profile.ci.overrides]]
filter = 'package(rustfs-ecstore) & (test(concurrent_resend_same_part_commits_one_generation) | test(concurrent_config_writes_from_separate_nodes_do_not_lose_writes))'
filter = 'package(rustfs-ecstore) & test(/^store::bucket::tests::bucket_delete_(mark_delete_marks|purge_removes|default_s3_delete)/)'
test-group = 'ecstore-serial-flaky'
retries = 2
# QUARANTINE: OPEN rustfs#4690 — walk_dir stall-budget accounting test depends
# on producer/consumer timing windows that stretch past the budget on loaded
@@ -156,7 +115,7 @@ retries = 2
# quarantine: no retries, just single-threaded so several 4-disk servers never
# run concurrently when ci-7's nightly runs the full e2e suite.
[[profile.ci.overrides]]
filter = 'package(e2e_test) & test(/^(reliability_disk_fault|degraded_read_eof_regression|replacement_privileged_e2e)_test::/)'
filter = 'package(e2e_test) & test(/^(reliability_disk_fault|degraded_read_eof_regression)_test::/)'
test-group = 'e2e-reliability'
# Serialize the multipart crash-consistency scenarios under the ci profile too
@@ -166,28 +125,6 @@ test-group = 'e2e-reliability'
filter = 'package(rustfs-ecstore) & test(/^set_disk::ops::multipart::tests::crash_consistency::/)'
test-group = 'ecstore-serial-flaky'
[[profile.ci.overrides]]
filter = 'package(rustfs) & test(execute_get_object_resumes_from_relocated_pool_without_splicing_body)'
test-group = 'ecstore-serial-flaky'
# Match the default-profile embedded test isolation without quarantining or
# retrying failures in CI.
[[profile.ci.overrides]]
filter = 'package(rustfs) & (binary(/^embedded.*_test$/) | binary(admin_diagnostic_capability_e2e))'
test-group = 'embedded-test-ports'
# Serialize the durable manual-transition checkpoint test under the ci profile
# too. No retries: failures stay visible.
[[profile.ci.overrides]]
filter = 'package(rustfs-ecstore) & test(manual_transition_page_checkpoint_persists_durable_job_progress)'
test-group = 'ecstore-serial-flaky'
# Serialize the bucket-incarnation / lifecycle-fence tests under the ci profile
# too (see the matching default-profile override near the top). No retries.
[[profile.ci.overrides]]
filter = 'package(rustfs-ecstore) & (test(bucket_delete_waits_for_config_mutation_fence) | test(stale_config_request_cannot_mutate_a_recreated_bucket) | test(disk_incarnation_read_detects_stale_cache_until_peer_reload) | test(lifecycle_expiry_fails_closed_on_corrupt_object_lock_metadata) | test(expiry_configs_are_resolved_from_the_owning_store))'
test-group = 'ecstore-serial-flaky'
# ---------------------------------------------------------------------------
# e2e-smoke profile — PR smoke subset of the e2e_test crate (backlog#1149 ci-4)
# ---------------------------------------------------------------------------
@@ -219,7 +156,7 @@ test-group = 'ecstore-serial-flaky'
# the nightly profile derives its set as "the replication module MINUS this
# allowlist", so any new replication test lands in nightly by default (never
# silently unrun) until it is explicitly blessed as fast here. Keep the two
# regexes byte-identical. Count invariant: 20 here + 49 nightly = 69 total
# regexes byte-identical. Count invariant: 20 here + 27 nightly = 47 total
# (authority: `cargo nextest list`; docs/testing/e2e-suite-inventory.md).
# HISTORY (2026-07-11): the 20 fast tests were briefly pulled out of this lane
# (#4724) because they set a loopback (127.0.0.1) replication target that the
@@ -255,7 +192,7 @@ test-group = 'ecstore-serial-flaky'
[profile.e2e-smoke]
default-filter = """
package(e2e_test) & (
test(/^(delete_marker_migration_semantics|version_id_regression|list_objects_v2_pagination|list_object_versions_regression|list_objects_duplicates|list_buckets_double_slash|list_buckets_auth|list_buckets_iam_filter|leading_slash_key|special_chars|create_bucket_region|delete_objects_versioning|head_object_consistency|head_object_range|copy_object_metadata|copy_object_tagging|copy_source_invalid_date|content_encoding|multipart_storage_class|storage_class_capability|ssec_copy|anonymous_access|bucket_policy_check|presigned_negative|negative_sigv4|admin_auth|notification_webhook|tls_hot_reload|console_smoke|admin_iam_crud|admin_pools|sts_query_compat)_test::|^fake_s3_target::/)
test(/^(delete_marker_migration_semantics|version_id_regression|list_objects_v2_pagination|list_object_versions_regression|list_objects_duplicates|list_buckets_double_slash|leading_slash_key|special_chars|create_bucket_region|delete_objects_versioning|head_object_consistency|head_object_range|copy_object_metadata|copy_source_invalid_date|content_encoding|anonymous_access|bucket_policy_check|presigned_negative|negative_sigv4|admin_auth|notification_webhook|tls_hot_reload|console_smoke|admin_iam_crud)_test::|^fake_s3_target::/)
| test(/^replication_extension_test::(test_replication_check_succeeds_with_remote_target|test_replication_check_rejects_target_without_object_lock|test_set_remote_target_rejects_unversioned_source_bucket|test_replication_check_rejects_unversioned_source_bucket|test_replication_check_rejects_missing_replication_config|test_replication_check_rejects_invalid_bucket|test_set_remote_target_rejects_same_bucket_on_same_deployment|test_set_remote_target_rejects_unversioned_target_bucket|test_set_remote_target_update_requires_arn|test_set_remote_target_update_rejects_missing_target|test_set_remote_target_rejects_invalid_target_url|test_set_remote_target_rejects_self_signed_https_target_without_skip_tls_verify|test_set_remote_target_rejects_private_ca_https_target_without_ca_cert_pem|test_list_remote_targets_rejects_empty_bucket|test_list_remote_targets_rejects_invalid_bucket|test_remove_remote_target_rejects_missing_target|test_remove_remote_target_rejects_missing_arn|test_remove_remote_target_rejects_invalid_bucket|test_remove_remote_target_rejects_target_used_by_replication|test_delete_bucket_replication_removes_remote_target)$/)
| test(/^reliant::lifecycle::/)
| test(/^reliant::tiering::/)
@@ -263,17 +200,6 @@ default-filter = """
"""
fail-fast = false
[profile.e2e-smoke.junit]
path = "junit.xml"
# The pagination boundary cases can stall when a server/listing regression
# prevents the continuation request from completing. Keep the timeout scoped
# to those known failure modes so legitimate lifecycle/tiering waits retain
# their test-level timing budget.
[[profile.e2e-smoke.overrides]]
filter = 'package(e2e_test) & test(/^list_objects_v2_pagination_test::tests::(test_list_objects_v2_delimiter_small_page_traverses_all|test_list_objects_v2_max_keys_above_limit_returns_token|test_list_objects_v2_maxkeys_above_limit_with_delimiter)$/)'
slow-timeout = { period = "60s", terminate-after = 2, grace-period = "10s" }
# ---------------------------------------------------------------------------
# e2e-repl-nightly profile — scheduled full replication e2e lane (repl-1)
# ---------------------------------------------------------------------------
@@ -281,13 +207,11 @@ slow-timeout = { period = "60s", terminate-after = 2, grace-period = "10s" }
# tests that are unfit for the per-PR e2e-smoke gate:
#
# * 2 remote-target TLS validation tests.
# * 15 bucket-replication data-plane/helper tests — they PUT/delete objects
# and poll until source and target converge; two replicate over HTTPS,
# six pin SSE replication contracts (managed SSE-S3/SSE-KMS re-encrypt on
# the target incl. multipart and the resync path, SSE-C and
# target-without-KMS stay fail-closed), and one guards event/history
# observers.
# * 12 `_real_dual_node` site-replication tests — each spawns TWO full rustfs
# * 12 bucket-replication data-plane/helper tests — they PUT/delete objects
# and poll until source and target converge; two replicate over HTTPS, two
# pin active SSE failure contracts, and one guards event/history observers.
# The SSE-S3 contract remains ignored under backlog#1291.
# * 11 `_real_dual_node` site-replication tests — each spawns TWO full rustfs
# servers and drives the cross-process site-replication control plane.
# * 1 `_real_three_node` site-replication test.
# * 1 `_real_single_node` service-account round-trip test.
@@ -345,16 +269,16 @@ path = "junit.xml"
# object_lambda) — too heavy for the merge budget; they run in ci-7's
# nightly 4-node lane.
# * replication_extension_test — repl-1 already splits it into the PR
# `e2e-smoke` (20 fast) and `e2e-repl-nightly` (49 slow) lanes and reserves
# `e2e-smoke` (20 fast) and `e2e-repl-nightly` (27 slow) lanes and reserves
# it for those, so e2e-full does not double-run it.
# * #[ignore]d tests — nextest skips them by default (no --run-ignored); the
# manual-localhost:9000 reliant/policy tests are ci-13's migration.
#
# Each e2e test spawns its own single-node rustfs server on a random port with
# an isolated temp dir (crates/e2e_test/src/common.rs), so the set is
# parallel-safe — the same property e2e-smoke relies on. The exceptions are the
# 4-disk reliability / degraded-read fault-injection tests and the fixed-port
# Vault tests, both serialized below.
# parallel-safe — the same property e2e-smoke relies on. The exception is the
# 4-disk reliability / degraded-read fault-injection tests, serialized below
# (identical to the ci profile) so several 4-disk servers never run at once.
# KNOWN-FAILURE EXCLUSIONS (characterization run 29381309848, 2026-07-15:
# 341 ran / 32 failed on the suites' first automated run ever). Deterministic
# product failures cannot be quarantined away with retries, so each family is
@@ -364,6 +288,9 @@ path = "junit.xml"
# negative-path siblings of each family stay in as regression guards.
# * rustfs#4843 — over-limit archive entry paths hard-reject the whole
# archive even under ignore-errors semantics.
# * rustfs#4846 — distributed-lock quorum tests misclassify as timeout
# under parallel load (multi-node in-process clusters; natural home is
# ci-7's nightly cluster lane).
[profile.e2e-full]
default-filter = """
package(e2e_test)
@@ -372,6 +299,7 @@ default-filter = """
& !test(/^replication_extension_test::/)
& !test(/^multipart_auth_test::test_signed_put_object_extract_skips_invalid_entry_when_ignore_errors_enabled$/)
& !test(/^snowball_auto_extract_test::tests::snowball_auto_extract_(ignores_invalid_entries_when_requested|supports_standard_headers_with_combined_extract_options)$/)
& !test(/^reliant::lock::test_distributed_lock_(2_nodes_grpc_read_survives_failed_node|4_nodes_grpc_read_write_quorum_split_with_two_failed_nodes)$/)
"""
fail-fast = false
@@ -384,13 +312,5 @@ path = "junit.xml"
# quarantine: no retries, just single-threaded so several 4-disk servers never
# run concurrently.
[[profile.e2e-full.overrides]]
filter = 'package(e2e_test) & test(/^(reliability_disk_fault|degraded_read_eof_regression|replacement_privileged_e2e)_test::/)'
filter = 'package(e2e_test) & test(/^(reliability_disk_fault|degraded_read_eof_regression)_test::/)'
test-group = 'e2e-reliability'
[[profile.e2e-full.overrides]]
filter = 'package(e2e_test) & test(/^inline_fast_path_cluster_test::/)'
test-group = 'e2e-inline-boundaries'
[[profile.e2e-full.overrides]]
filter = 'package(e2e_test) & test(/^kms::kms_vault_test::/)'
test-group = 'e2e-vault'
-25
View File
@@ -60,16 +60,6 @@ The file `prometheus-rules/rustfs-get-optimization-alerts.yaml` contains pre-con
| `CodecStreamingFallbackSpike` | Warning | Codec streaming fallback > 10x baseline for 10m |
| `IoQueueSaturation` | Warning | IO queue utilization > 90% for 5m |
The file `prometheus-rules/rustfs-kms-alerts.yml` contains alerting rules for the KMS backend operation metrics. Thresholds are conservative defaults pending staging baseline calibration; response procedures live in `docs/operations/kms-observability-runbook.md`, and the matching dashboard is `deploy/observability/grafana/rustfs-kms-observability.json`.
| Alert | Severity | Condition |
|-------|----------|-----------|
| `KmsBackendFatalErrors` | Critical | Fatal (non-retryable) attempt failures > 0 for 5m |
| `KmsBackendHighErrorRate` | Critical | Non-success operation ratio > 5% for 10m (with traffic guard) |
| `KmsBackendP99LatencyHigh` | Warning | Operation p99 duration (incl. retries) > 2s for 10m |
| `KmsBackendAttemptFailureSpike` | Warning | Attempt failure rate > 0.5/s for 10m |
| `KmsBackendRetryBudgetExhausted` | Warning | budget_exhausted / deadline_exceeded outcomes > 0.05/s for 10m |
### Enabling Alert Rules
Add the alert rules file to your Prometheus configuration:
@@ -170,10 +160,6 @@ Important behavior notes:
- Logs and metrics usually appear during startup, so seeing those two signals
first is expected.
- The OpenTelemetry bridge sends `tracing` fields as log attributes. Loki stores
those attributes as structured metadata, and the Collector also mirrors the
common troubleshooting fields into the log line so simple line filters can
find them.
- Visible trace data usually requires real HTTP/S3/gRPC request traffic after
startup, because request-path spans are created on demand.
- `RUSTFS_OBS_LOGGER_LEVEL=info` keeps the top-level request span but filters
@@ -199,17 +185,6 @@ curl -I http://127.0.0.1:9000/health/ready
# Jaeger: http://localhost:16686
```
For a structured RustFS log such as an inter-node RPC authentication failure,
the Loki line now includes fields such as `event`, `component`, `subsystem`,
`failure_reason`, `rpc_service`, `rpc_method`, and `expected_audience`. Useful
LogQL checks:
```logql
{service_name="RustFS"} |= "RPC signature verification failed"
{service_name="RustFS"} |= "failure_reason="
{service_name="RustFS"} | failure_reason != ""
```
If logs and metrics are present but traces are sparse, the most common cause is
"no real request traffic yet" or "`info` level filtered nested spans", not an
OTLP routing failure.
-19
View File
@@ -60,16 +60,6 @@
| `CodecStreamingFallbackSpike` | 警告 | Codec streaming 回退 > 10x 基线,持续 10 分钟 |
| `IoQueueSaturation` | 警告 | IO 队列利用率 > 90%,持续 5 分钟 |
文件 `prometheus-rules/rustfs-kms-alerts.yml` 包含 KMS 后端操作指标的告警规则。阈值为保守默认值,待 staging 基线校准;响应流程见 `docs/operations/kms-observability-runbook.md`,配套仪表盘为 `deploy/observability/grafana/rustfs-kms-observability.json`
| 告警 | 级别 | 条件 |
|------|------|------|
| `KmsBackendFatalErrors` | 严重 | fatal(不可重试)尝试失败 > 0,持续 5 分钟 |
| `KmsBackendHighErrorRate` | 严重 | 非 success 操作占比 > 5%,持续 10 分钟(含流量下限保护) |
| `KmsBackendP99LatencyHigh` | 警告 | 操作 p99 耗时(含重试)> 2s,持续 10 分钟 |
| `KmsBackendAttemptFailureSpike` | 警告 | 尝试失败率 > 0.5/s,持续 10 分钟 |
| `KmsBackendRetryBudgetExhausted` | 警告 | budget_exhausted / deadline_exceeded 结果 > 0.05/s,持续 10 分钟 |
### 启用告警规则
在 Prometheus 配置中添加告警规则文件:
@@ -169,7 +159,6 @@ RustFS 会自动在该基础 URL 后补全:
需要注意:
- 启动阶段通常会先看到日志和指标,因此“先有日志/指标、后有 trace”是正常现象。
- OpenTelemetry bridge 会把 `tracing` 字段作为日志 attributes 发送。Loki 会将这些 attributes 存为 structured metadata,同时 Collector 会把常用排障字段镜像进日志行,方便用简单的行内容过滤直接查到。
- 可见的 trace 数据通常依赖启动后的真实 HTTP/S3/gRPC 请求流量,因为请求路径上的 span 是按需创建的。
- `RUSTFS_OBS_LOGGER_LEVEL=info` 会保留顶层请求 span,但会过滤掉很多 `debug` 级别的嵌套 span。
如果 Tempo 或 Jaeger 中的 trace 看起来很稀疏,建议先改成 `RUSTFS_OBS_LOGGER_LEVEL=debug`,再判断是否是 collector 或 Tempo 问题。
@@ -193,14 +182,6 @@ curl -I http://127.0.0.1:9000/health/ready
# Jaeger: http://localhost:16686
```
对于 RustFS 结构化日志,例如节点间 RPC 鉴权失败,Loki 日志行现在会包含 `event``component``subsystem``failure_reason``rpc_service``rpc_method``expected_audience` 等字段。常用 LogQL 检查:
```logql
{service_name="RustFS"} |= "RPC signature verification failed"
{service_name="RustFS"} |= "failure_reason="
{service_name="RustFS"} | failure_reason != ""
```
如果日志和指标已经正常,但 trace 仍然稀疏,最常见的原因通常是
“还没有真实请求流量”或“`info` 级别过滤了嵌套 span”,而不是 OTLP 路由失败。
File diff suppressed because it is too large Load Diff
@@ -29,27 +29,11 @@ processors:
limit_mib: 1024
spike_limit_mib: 256
transform/logs:
error_mode: ignore
log_statements:
- context: log
statements:
- set(attributes["message"], body.string) where IsString(body)
- set(attributes["log.body"], body.string) where IsString(body)
- set(body, Concat([body, " event=", attributes["event"]], "")) where IsString(body) and attributes["event"] != nil
- set(body, Concat([body, " component=", attributes["component"]], "")) where IsString(body) and attributes["component"] != nil
- set(body, Concat([body, " subsystem=", attributes["subsystem"]], "")) where IsString(body) and attributes["subsystem"] != nil
- set(body, Concat([body, " state=", attributes["state"]], "")) where IsString(body) and attributes["state"] != nil
- set(body, Concat([body, " result=", attributes["result"]], "")) where IsString(body) and attributes["result"] != nil
- set(body, Concat([body, " reason=", attributes["reason"]], "")) where IsString(body) and attributes["reason"] != nil
- set(body, Concat([body, " failure_reason=", attributes["failure_reason"]], "")) where IsString(body) and attributes["failure_reason"] != nil
- set(body, Concat([body, " rpc_path=", attributes["rpc_path"]], "")) where IsString(body) and attributes["rpc_path"] != nil
- set(body, Concat([body, " rpc_service=", attributes["rpc_service"]], "")) where IsString(body) and attributes["rpc_service"] != nil
- set(body, Concat([body, " rpc_method=", attributes["rpc_method"]], "")) where IsString(body) and attributes["rpc_method"] != nil
- set(body, Concat([body, " expected_audience=", attributes["expected_audience"]], "")) where IsString(body) and attributes["expected_audience"] != nil
- set(body, Concat([body, " peer_addr=", attributes["peer_addr"]], "")) where IsString(body) and attributes["peer_addr"] != nil
- set(body, Concat([body, " replay_scope_bootstrap_allowed=", attributes["replay_scope_bootstrap_allowed"]], "")) where IsString(body) and attributes["replay_scope_bootstrap_allowed"] != nil
- set(body, Concat([body, " error=", attributes["error"]], "")) where IsString(body) and attributes["error"] != nil
- set(body, Concat([body, " exception_message=", attributes["exception.message"]], "")) where IsString(body) and attributes["exception.message"] != nil
- set(attributes["message"], body.string)
- set(attributes["log.body"], body.string)
exporters:
otlp/tempo:
@@ -1,251 +0,0 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# =============================================================================
# RustFS KMS backend — Prometheus alerting rules
# =============================================================================
#
# Metric source: the KMS operation-policy choke point in
# crates/kms/src/policy.rs, except KmsKeyRotationOverdue, which reads the
# label-less key-lifecycle gauge published by the deletion worker's sweep
# (crates/kms/src/deletion_worker.rs). All label values are bounded static
# strings (operation, op_class, outcome, error_class, backend, scope); key
# identifiers, key material, and tokens never appear in labels.
#
# Response procedures: docs/operations/kms-observability-runbook.md
#
# IMPORTANT — threshold status: every numeric threshold below is a
# conservative default chosen without a production baseline. Calibrate against
# a staging baseline before relying on these alerts for paging, and prefer
# loosening over tightening until the baseline exists. Formal SLO targets are
# deliberately not encoded here (see rustfs/backlog#1584).
#
# NOTE: prometheus.yml loads /etc/prometheus/rules/*.yml — keep the .yml
# extension or the file is silently ignored by the docker-compose stack.
#
# Validate: promtool check rules rustfs-kms-alerts.yml
# =============================================================================
groups:
# ==========================================================================
# Critical alerts — immediate action required
# ==========================================================================
- name: rustfs-kms-critical
interval: 30s
rules:
# ------------------------------------------------------------------
# 1. KmsBackendFatalErrors
# Any attempt failure classified as fatal (non-retryable): auth
# or permission errors, malformed requests, missing keys. The
# policy never retries these, so even a low rate means real
# operations are failing right now.
# ------------------------------------------------------------------
- alert: KmsBackendFatalErrors
expr: |
sum by (operation) (rate(rustfs_kms_backend_attempt_failures_total{error_class="fatal"}[5m])) > 0
for: 5m
labels:
severity: critical
component: kms
annotations:
summary: "KMS backend fatal errors on operation {{ $labels.operation }}"
description: >-
Attempt failures classified as fatal are occurring at
{{ $value | printf "%.3f" }}/s on operation
{{ $labels.operation }}. Fatal failures are not retried:
each one is a KMS backend call that failed permanently
(authentication, permissions, malformed request, or a
missing key/version).
runbook_url: "https://github.com/rustfs/rustfs/blob/main/docs/operations/kms-observability-runbook.md#kmsbackendfatalerrors"
# ------------------------------------------------------------------
# 2. KmsBackendHighErrorRate
# Sustained share of operations terminating without success
# (fatal, budget/deadline exhaustion, admission backpressure,
# or an open circuit). The cancelled outcome is excluded because
# shutdowns legitimately produce it.
# The traffic guard keeps a single failure on a near-idle
# cluster from firing the alert.
# Threshold: 5% for 10m — conservative default, calibrate
# against a staging baseline.
# ------------------------------------------------------------------
- alert: KmsBackendHighErrorRate
expr: |
(
sum(rate(rustfs_kms_backend_operations_total{outcome!~"success|cancelled"}[5m]))
/
clamp_min(sum(rate(rustfs_kms_backend_operations_total[5m])), 1e-9)
) > 0.05
and
sum(rate(rustfs_kms_backend_operations_total[5m])) > 0.02
for: 10m
labels:
severity: critical
component: kms
annotations:
summary: "KMS backend non-success ratio above 5% for 10m"
description: >-
{{ $value | humanizePercentage }} of KMS backend operations
are terminating in fatal, budget_exhausted,
deadline_exceeded, backpressure_timeout,
backpressure_rejected, or circuit_open. Object encryption
and decryption paths depending on the KMS are degraded or
failing.
runbook_url: "https://github.com/rustfs/rustfs/blob/main/docs/operations/kms-observability-runbook.md#kmsbackendhigherrorrate"
# ==========================================================================
# Warning alerts — investigation needed
# ==========================================================================
- name: rustfs-kms-warning
interval: 30s
rules:
# ------------------------------------------------------------------
# 3. KmsBackendP99LatencyHigh
# p99 wall-clock duration of whole operations (attempts plus
# backoff) is sustained above 2 seconds. Because the histogram
# includes retries, a high p99 usually means the retry policy
# is absorbing backend failures, not that every call is slow.
# Threshold: 2s for 10m — conservative default, calibrate
# against a staging baseline.
# ------------------------------------------------------------------
- alert: KmsBackendP99LatencyHigh
expr: |
histogram_quantile(0.99,
sum by (le) (rate(rustfs_kms_backend_operation_duration_seconds_bucket[5m]))
) > 2
for: 10m
labels:
severity: warning
component: kms
annotations:
summary: "KMS backend operation p99 latency above 2s for 10m"
description: >-
The 99th-percentile KMS backend operation duration is
{{ $value | humanizeDuration }}, including retries and
backoff. Encryption and decryption latency is leaking into
S3 request latency.
runbook_url: "https://github.com/rustfs/rustfs/blob/main/docs/operations/kms-observability-runbook.md#kmsbackendp99latencyhigh"
# ------------------------------------------------------------------
# 4. KmsBackendAttemptFailureSpike
# Aggregate attempt-failure rate (all error classes) sustained
# above an absolute floor. An absolute threshold is used instead
# of an offset-1d baseline ratio because fresh deployments have
# no baseline and an empty offset vector would keep a ratio
# alert from ever firing; switch to a baseline-relative form
# (see rustfs-get-optimization-alerts.yaml for the pattern)
# once a stable staging baseline exists.
# Threshold: 0.5/s for 10m — conservative default, calibrate
# against a staging baseline.
# ------------------------------------------------------------------
- alert: KmsBackendAttemptFailureSpike
expr: |
sum(rate(rustfs_kms_backend_attempt_failures_total[5m])) > 0.5
for: 10m
labels:
severity: warning
component: kms
annotations:
summary: "KMS backend attempt failures above 0.5/s for 10m"
description: >-
KMS backend attempts are failing at
{{ $value | printf "%.2f" }}/s across all error classes.
The retry policy may still be masking these from callers —
check the error-class breakdown before it stops absorbing
them.
runbook_url: "https://github.com/rustfs/rustfs/blob/main/docs/operations/kms-observability-runbook.md#kmsbackendattemptfailurespike"
# ------------------------------------------------------------------
# 5. KmsBackendRetryBudgetExhausted
# Operations are running out of retry budget (budget_exhausted)
# or operation deadline (deadline_exceeded). These surface to
# callers as failed KMS operations even though every individual
# failure was retryable — the backend is unhealthy for longer
# than the policy can bridge.
# Threshold: 0.05/s for 10m — conservative default, calibrate
# against a staging baseline.
# ------------------------------------------------------------------
- alert: KmsBackendRetryBudgetExhausted
expr: |
sum by (outcome) (rate(rustfs_kms_backend_operations_total{outcome=~"budget_exhausted|deadline_exceeded"}[5m])) > 0.05
for: 10m
labels:
severity: warning
component: kms
annotations:
summary: "KMS backend operations exhausting retry budget ({{ $labels.outcome }})"
description: >-
KMS backend operations are terminating as
{{ $labels.outcome }} at {{ $value | printf "%.3f" }}/s.
Retryable failures are outlasting the retry budget, so
callers are seeing hard failures.
runbook_url: "https://github.com/rustfs/rustfs/blob/main/docs/operations/kms-observability-runbook.md#kmsbackendretrybudgetexhausted"
# ------------------------------------------------------------------
# 6. KmsBackendCircuitOpen
# Direct circuit-state signal, independent of operation traffic.
# A transient open can recover on its first half-open probe; alert
# only when the circuit remains open or half-open for one minute.
# ------------------------------------------------------------------
- alert: KmsBackendCircuitOpen
expr: |
rustfs_kms_backend_circuit_open > 0
for: 1m
labels:
severity: warning
component: kms
annotations:
summary: "KMS backend circuit open ({{ $labels.backend }}/{{ $labels.scope }})"
description: >-
The KMS backend circuit for {{ $labels.backend }} scope
{{ $labels.scope }} has remained open or half-open for one
minute. Operations in this scope can terminate as
circuit_open until the half-open probe succeeds or returns
a non-retryable failure.
runbook_url: "https://github.com/rustfs/rustfs/blob/main/docs/operations/kms-observability-runbook.md#kmsbackendcircuitopen"
# ------------------------------------------------------------------
# 7. KmsKeyRotationOverdue
# The least recently rotated usable key has gone more than 400
# days without a rotation (measured from creation for keys with
# no recorded rotation). Direct gauge state published by the
# deletion worker's sweep, so no traffic guard applies; the
# one-hour hold only bridges scrape gaps. The worker runs only
# on backends with the schedule_deletion capability, so on the
# Static backend the series never exists and this alert cannot
# fire — that backend cannot rotate either; see the rotation
# driver matrix in docs/operations/kms-backend-security.md.
# Threshold: 400 days — conservative default sitting above a
# one-year rotation policy. Align it with the rotation period
# your compliance policy requires, and with
# RUSTFS_KMS_ROTATION_MAX_AGE_SECS so the per-key rotation_due
# verdict and this aggregate alert agree.
# ------------------------------------------------------------------
- alert: KmsKeyRotationOverdue
expr: |
rustfs_kms_oldest_key_rotation_age_seconds > (400 * 86400)
for: 1h
labels:
severity: warning
component: kms
annotations:
summary: "Oldest KMS key unrotated for more than 400 days"
description: >-
The least recently rotated usable KMS key was last rotated
{{ $value | humanizeDuration }} ago (measured from creation
for keys with no recorded rotation). List keys through the
admin API and read rotation_due / rotation_due_reason for
the per-key verdict; an "unsupported" reason means the
backend cannot rotate at all.
runbook_url: "https://github.com/rustfs/rustfs/blob/main/docs/operations/kms-observability-runbook.md#kmskeyrotationoverdue"
@@ -18,7 +18,6 @@ set -eu
ACCESS_KEY="${RUSTFS_SITE_REPL_ACCESS_KEY:-rustfsadmin}"
SECRET_KEY="${RUSTFS_SITE_REPL_SECRET_KEY:-rustfsadmin}"
BUCKET="${RUSTFS_SITE_REPL_FLOW_BUCKET:-site-repl-flow-check}"
DELETE_BUCKET="${RUSTFS_SITE_REPL_DELETE_BUCKET:-site-repl-delete-$(date +%Y%m%d-%H%M%S)-$$}"
PREFIX="${RUSTFS_SITE_REPL_FLOW_PREFIX:-flow-$(date +%Y%m%d-%H%M%S)}"
WAIT_ATTEMPTS="${RUSTFS_SITE_REPL_WAIT_ATTEMPTS:-90}"
WAIT_SLEEP_SECONDS="${RUSTFS_SITE_REPL_WAIT_SLEEP_SECONDS:-2}"
@@ -86,39 +85,17 @@ wait_for_object() {
wait_for_bucket() {
site="$1"
bucket="${2:-$BUCKET}"
attempt=1
while [ "$attempt" -le "$WAIT_ATTEMPTS" ]; do
if mc stat "$site/$bucket" >/dev/null 2>&1; then
if mc stat "$site/$BUCKET" >/dev/null 2>&1; then
return 0
fi
sleep "$WAIT_SLEEP_SECONDS"
attempt=$((attempt + 1))
done
echo "bucket was not replicated in time: $site/$bucket" >&2
return 1
}
wait_for_bucket_delete() {
site="$1"
bucket="$2"
attempt=1
while [ "$attempt" -le "$WAIT_ATTEMPTS" ]; do
if result="$(mc stat --json "$site/$bucket" 2>&1)"; then
:
else
case "$result" in
*NoSuchBucket*) return 0 ;;
esac
fi
sleep "$WAIT_SLEEP_SECONDS"
attempt=$((attempt + 1))
done
echo "bucket deletion was not replicated in time: $site/$bucket" >&2
echo "bucket was not replicated in time: $site/$BUCKET" >&2
return 1
}
@@ -209,20 +186,6 @@ EOF
echo "verified replicated downloads for $object_name"
done
echo "creating empty bucket for replicated delete check: $DELETE_BUCKET"
mc mb "site1/$DELETE_BUCKET" >/dev/null
for site in site1 site2 site3; do
wait_for_bucket "$site" "$DELETE_BUCKET"
done
echo "deleting empty bucket on site1: $DELETE_BUCKET"
mc rb "site1/$DELETE_BUCKET" >/dev/null
for site in site1 site2 site3; do
wait_for_bucket_delete "$site" "$DELETE_BUCKET"
done
echo "site replication object flow check passed"
echo "bucket: $BUCKET"
echo "prefix: $PREFIX"
+13 -44
View File
@@ -25,13 +25,9 @@ inputs:
required: false
default: "rustfs-deps"
cache-save-if:
description: >-
Whether to save the cache. The fail-safe default is 'false': a caller that
wants to populate a cache must opt in explicitly, so a forgotten input
costs a cold cache (minutes) rather than silently consuming the
repository-wide 10GB Actions cache quota and evicting other lanes.
description: "Condition for saving cache"
required: false
default: "false"
default: "true"
install-cross-tools:
description: "Install cross-compilation tools"
required: false
@@ -40,48 +36,33 @@ inputs:
description: "Target architecture to add"
required: false
default: ""
install-build-packaging-tools:
description: >-
Install musl-tools/zip/unzip, needed for musl linking and release
packaging. Off for CI test lanes, which use none of them.
github-token:
description: "GitHub token for API access"
required: false
default: "true"
install-test-tools:
description: >-
Install cargo-nextest and the rustfmt/clippy components. Off for release
and audit lanes, which run no tests and no lints.
required: false
default: "true"
default: ""
runs:
using: "composite"
steps:
# protobuf-compiler is deliberately absent: the setup-protoc step below
# installs 35.1 into the tool cache and prepends it to PATH, so the apt
# build (older, and never version-matched) was shadowed on every run and
# simply never used.
- name: Install system dependencies (Ubuntu)
if: runner.os == 'Linux'
shell: bash
run: |
sudo apt-get update
sudo apt-get install -y \
musl-tools \
build-essential \
pkg-config \
libssl-dev \
ripgrep
# musl-gcc is needed by the native musl release leg, and zip/unzip by the
# release packaging steps. No CI test lane touches any of them.
- name: Install packaging and cross-linking dependencies (Ubuntu)
if: runner.os == 'Linux' && inputs.install-build-packaging-tools == 'true'
shell: bash
run: sudo apt-get install -y musl-tools zip unzip
ripgrep \
unzip \
zip \
protobuf-compiler
- name: Install protoc
uses: rustfs/setup-protoc@a3705324d8f9bf5b6c3573fb6cf8ae421db55dd6 # v3.0.1
with:
version: "35.1"
version: "34.1"
repo-token: ${{ github.token }}
- name: Install flatc
@@ -94,7 +75,7 @@ runs:
with:
toolchain: ${{ inputs.rust-version }}
targets: ${{ inputs.target }}
components: ${{ inputs.install-test-tools == 'true' && 'rustfmt, clippy' || '' }}
components: rustfmt, clippy
- name: Install Zig
if: inputs.install-cross-tools == 'true'
@@ -105,24 +86,12 @@ runs:
uses: taiki-e/install-action@a21ae4029b089b9ddc45704028756f51ab8abe48 # cargo-zigbuild
- name: Install cargo-nextest
if: inputs.install-test-tools == 'true'
uses: taiki-e/install-action@96c7780c1d8a2b8723e12031def873a434d39d8d # nextest
- name: Setup Rust cache
uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2
with:
# false is rust-cache's own default. With true, cleanup.ts returns
# *before* pruning ~/.cargo/registry/src, and config.ts archives the
# whole registry — so every cache carried the unpacked source tree of
# every dependency, not just "a few extra crates".
#
# No coverage is lost: getPackages runs `cargo metadata --all-features`,
# a strict superset of any single lane's feature closure, and -sys crates
# are explicitly exempted from pruning (their src timestamps would
# otherwise trigger rebuilds). Anything pruned is re-unpacked from the
# .crate files still in registry/cache, whose mtimes crates.io
# normalises, so cargo fingerprints stay valid.
cache-all-crates: false
cache-all-crates: true
cache-on-failure: true
shared-key: ${{ inputs.cache-shared-key }}
save-if: ${{ inputs.cache-save-if }}
Binary file not shown.

Before

Width:  |  Height:  |  Size: 105 KiB

@@ -37,7 +37,6 @@ jobs:
name: Cancel Closed PR Runs
if: github.event_name == 'pull_request' && github.event.action == 'closed'
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- name: Explain cancellation run
run: echo "PR closed; this run only cancels older runs in the same concurrency group."
@@ -46,11 +45,8 @@ jobs:
name: Architecture Migration Rules
if: github.event_name != 'pull_request' || github.event.action != 'closed'
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Install ripgrep
run: |
+4 -82
View File
@@ -23,9 +23,6 @@ on:
- 'deny.toml'
- '.github/actions/**'
- '.github/workflows/**'
- 'scripts/release/create_or_update_release.sh'
- 'scripts/security/check_performance_ab_workflow.sh'
- 'scripts/security/check_preview_release_workflow.sh'
- 'scripts/security/check_workflow_pins.sh'
pull_request:
types: [ opened, synchronize, reopened, closed ]
@@ -36,17 +33,9 @@ on:
- 'deny.toml'
- '.github/actions/**'
- '.github/workflows/**'
- 'scripts/release/create_or_update_release.sh'
- 'scripts/security/check_performance_ab_workflow.sh'
- 'scripts/security/check_preview_release_workflow.sh'
- 'scripts/security/check_workflow_pins.sh'
schedule:
# Daily, not weekly. This schedule exists to catch RustSec advisories
# published against an unchanged dependency tree; at weekly cadence a new
# advisory could sit unnoticed for seven days. The check list is unchanged —
# splitting it into a light daily advisories-only run and a weekly full run
# would create runs where sources/bans/licenses go unverified.
- cron: '0 3 * * *' # Daily 03:00 UTC (staggered after the midnight ci/build crons)
- cron: '0 3 * * 0' # Weekly on Sunday 03:00 UTC (staggered after the midnight ci/build crons)
workflow_dispatch:
permissions:
@@ -66,7 +55,6 @@ jobs:
name: Cancel Closed PR Runs
if: github.event_name == 'pull_request' && github.event.action == 'closed'
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- name: Explain cancellation run
run: echo "PR closed; this run only cancels older runs in the same concurrency group."
@@ -82,32 +70,11 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
# cargo-deny compiles nothing, so the full setup composite (apt packages,
# protoc, flatc, nextest, rustfmt/clippy) was pure overhead here. It does
# still need a real cargo: `cargo deny check` runs `cargo metadata`, and
# Cargo.toml pins datafusion and s3s as git dependencies, which must be
# materialised into ~/.cargo/git — a cold clone is hundreds of MB, so the
# cache stays.
- name: Install Rust toolchain
uses: dtolnay/rust-toolchain@29eef336d9b2848a0b548edc03f92a220660cdb8 # stable
# Was relying on the composite's default, which used to be "true": every
# PR touching Cargo.toml/Cargo.lock saved a second, PR-scoped copy of this
# cache and pushed the main-scoped lanes out of the 10GB quota. The
# default is now "false", but state it explicitly — see
# scripts/security/check_cache_save_if.sh.
- name: Setup Rust cache
uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
# Same reasoning as the setup composite: true archives every
# dependency's unpacked source tree.
cache-all-crates: false
cache-on-failure: true
shared-key: rustfs-cargo-deny
save-if: ${{ github.ref == 'refs/heads/main' }}
cache-shared-key: rustfs-cargo-deny
- name: Install cargo-deny
uses: taiki-e/install-action@bffeee26d4db9be238a4ea78d8826604ebcb594d # v2
@@ -125,31 +92,13 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Report unpinned GitHub Actions
run: ./scripts/security/check_workflow_pins.sh --enforce
- name: Check setup cache-save-if is explicit
run: ./scripts/security/check_cache_save_if.sh
- name: Check every job declares a timeout
run: ./scripts/security/check_job_timeouts.sh
- name: Check checkouts clear their credentials
run: ./scripts/security/check_persist_credentials.sh
- name: Check preview release workflow policy
run: ./scripts/security/check_preview_release_workflow.sh
- name: Check performance A/B workflow trust boundary
run: ./scripts/security/check_performance_ab_workflow.sh
dependency-review:
name: Dependency Review
runs-on: ubuntu-latest
timeout-minutes: 30
if: github.event_name == 'pull_request' && github.event.action != 'closed'
permissions:
contents: read
@@ -157,8 +106,6 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Dependency Review
uses: actions/dependency-review-action@a1d282b36b6f3519aa1f3fc636f609c47dddb294 # v5
@@ -171,28 +118,3 @@ jobs:
# conscious re-review of the license/provenance claim (backlog#1181).
allow-dependencies-licenses: pkg:cargo/rustfs-uring@0.1.0
comment-summary-in-pr: always
alert-on-failure:
name: Alert on scheduled failure
# dependency-review is deliberately excluded: it only runs on pull_request,
# so it can never contribute a failure to a scheduled run.
needs: [cargo-deny, workflow-pin-report]
# A scheduled cargo-deny failure usually means the dependency tree just
# matched a newly published advisory — the single most important signal this
# workflow produces, and until now it was only visible to whoever happened to
# open the Actions tab. Same ci-8 mechanism coverage.yml and
# e2e-replication-nightly.yml already use.
if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
runs-on: ubuntu-latest
timeout-minutes: 10
permissions:
contents: read
issues: write
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Open or update failure-tracking issue
uses: ./.github/actions/schedule-failure-issue
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
+85 -89
View File
@@ -50,18 +50,12 @@ on:
- "**/*.svg"
- ".gitignore"
- ".dockerignore"
- "flake.lock"
schedule:
- cron: "0 1 * * 0" # Weekly on Sunday 01:00 UTC (staggered after the ci.yml midnight cron)
workflow_dispatch:
inputs:
build_docker:
# Advisory only. docker.yml triggers on workflow_run and its job-level
# condition requires the triggering event to be a tag push, so a manual
# dispatch of this workflow never produces images regardless of this
# value. Kept because the summary step reports it; wiring it up would
# mean teaching docker.yml's version parser a second event shape.
description: "Build and push Docker images after binary build (ignored: dispatch runs never reach docker.yml)"
description: "Build and push Docker images after binary build"
required: false
default: true
type: boolean
@@ -89,7 +83,6 @@ jobs:
build-check:
name: Build Strategy Check
runs-on: ubuntu-latest
timeout-minutes: 30
outputs:
should_build: ${{ steps.check.outputs.should_build }}
build_type: ${{ steps.check.outputs.build_type }}
@@ -99,8 +92,6 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Determine build strategy
id: check
@@ -116,21 +107,13 @@ jobs:
# Determine build type based on trigger
if [[ "${{ startsWith(github.ref, 'refs/tags/') }}" == "true" ]]; then
# Tag push - preview, release, or prerelease
# Tag push - release or prerelease
should_build=true
tag_name="${GITHUB_REF#refs/tags/}"
version="${tag_name}"
# Preview tags publish a GitHub prerelease for validation, but
# must not update any latest channel.
if [[ "$tag_name" =~ -preview\.[0-9]+$ ]]; then
build_type="preview"
is_prerelease=true
echo "🔍 Preview build detected: $tag_name"
elif [[ "$tag_name" == *"-preview"* ]]; then
echo "❌ Invalid preview tag: $tag_name (expected suffix: -preview.<number>)" >&2
exit 1
elif [[ "$tag_name" == *"alpha"* ]] || [[ "$tag_name" == *"beta"* ]] || [[ "$tag_name" == *"rc"* ]]; then
# Check if this is a prerelease
if [[ "$tag_name" == *"alpha"* ]] || [[ "$tag_name" == *"beta"* ]] || [[ "$tag_name" == *"rc"* ]]; then
build_type="prerelease"
is_prerelease=true
echo "🚀 Prerelease build detected: $tag_name"
@@ -173,7 +156,6 @@ jobs:
name: Prepare Platform Matrix
needs: build-check
runs-on: ubuntu-latest
timeout-minutes: 30
outputs:
matrix: ${{ steps.select.outputs.matrix }}
selected: ${{ steps.select.outputs.selected }}
@@ -181,14 +163,10 @@ jobs:
- name: Select target platforms
id: select
shell: bash
env:
# via env, not interpolation: a dispatch input is free-form text and
# would otherwise be pasted into the script for bash to evaluate.
RAW_PLATFORMS: ${{ github.event_name == 'workflow_dispatch' && github.event.inputs.platforms || 'all' }}
run: |
set -euo pipefail
selected="$RAW_PLATFORMS"
selected="${{ github.event_name == 'workflow_dispatch' && github.event.inputs.platforms || 'all' }}"
selected="$(echo "${selected}" | tr -d '[:space:]')"
if [[ -z "${selected}" ]]; then
selected="all"
@@ -259,7 +237,6 @@ jobs:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
fetch-depth: 0
- name: Setup Rust environment
@@ -268,17 +245,9 @@ jobs:
rust-version: stable
target: ${{ matrix.target }}
cache-shared-key: build-${{ matrix.target }}
# main only. A cache saved on refs/tags/X is scoped to that tag: no
# other tag, no main run and no PR can restore it, so every release
# cycle wrote up to 12 entries of 1-2GB (preview tag plus final tag,
# six legs each) that nobody could read, evicting the hot lanes from
# the repo-wide 10GB quota. Tag builds still restore the main-scoped
# cache, since default-branch caches are readable from every ref.
# The one real cost: re-running a failed leg of the same tag no longer
# finds that tag's own warm cache and falls back to main's.
cache-save-if: ${{ github.ref == 'refs/heads/main' }}
github-token: ${{ secrets.GITHUB_TOKEN }}
cache-save-if: ${{ github.ref == 'refs/heads/main' || startsWith(github.ref, 'refs/tags/') }}
install-cross-tools: ${{ matrix.cross }}
install-test-tools: 'false'
- name: Download static console assets
shell: bash
@@ -725,14 +694,9 @@ jobs:
needs: [ build-check, build-rustfs ]
if: always() && needs.build-check.outputs.should_build == 'true'
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- name: Build completion summary
shell: bash
env:
# dispatch input via env: free-form text must not be pasted into the
# script for bash to evaluate.
INPUT_BUILD_DOCKER: ${{ github.event.inputs.build_docker }}
run: |
BUILD_TYPE="${{ needs.build-check.outputs.build_type }}"
VERSION="${{ needs.build-check.outputs.version }}"
@@ -750,10 +714,6 @@ jobs:
echo ""
case "$BUILD_TYPE" in
"preview")
echo "🔍 Preview artifacts are published in a GitHub prerelease"
echo "⏭️ Preview releases do not update latest channels"
;;
"development")
echo "🛠️ Development build artifacts have been uploaded to OSS dev directory"
echo "⚠️ This is a development build - not suitable for production use"
@@ -772,9 +732,7 @@ jobs:
echo ""
echo "🐳 Docker Images:"
if [[ "$BUILD_TYPE" == "preview" ]]; then
echo "⏭️ Preview tags do not publish Docker images"
elif [[ "$INPUT_BUILD_DOCKER" == "false" ]]; then
if [[ "${{ github.event.inputs.build_docker }}" == "false" ]]; then
echo "⏭️ Docker image build was skipped (binary only build)"
elif [[ "$BUILD_STATUS" == "success" ]]; then
echo "🔄 Docker images will be built and pushed automatically via workflow_run event"
@@ -782,13 +740,12 @@ jobs:
echo "❌ Docker image build will be skipped due to build failure"
fi
# Create GitHub Release for every valid release tag, including previews
# Create GitHub Release (only for tag pushes)
create-release:
name: Create GitHub Release
needs: [ build-check, build-rustfs ]
if: startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'preview' || needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
if: startsWith(github.ref, 'refs/tags/') && needs.build-check.outputs.build_type != 'development'
runs-on: ubuntu-latest
timeout-minutes: 30
permissions:
contents: write
outputs:
@@ -798,7 +755,6 @@ jobs:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
fetch-depth: 0
- name: Create GitHub Release
@@ -811,12 +767,9 @@ jobs:
VERSION="${{ needs.build-check.outputs.version }}"
IS_PRERELEASE="${{ needs.build-check.outputs.is_prerelease }}"
BUILD_TYPE="${{ needs.build-check.outputs.build_type }}"
TARGET_COMMITISH=$(git rev-parse --verify "refs/tags/${TAG}^{commit}")
# Determine release type for title
if [[ "$BUILD_TYPE" == "preview" ]]; then
RELEASE_TYPE="preview"
elif [[ "$BUILD_TYPE" == "prerelease" ]]; then
if [[ "$BUILD_TYPE" == "prerelease" ]]; then
if [[ "$TAG" == *"alpha"* ]]; then
RELEASE_TYPE="alpha"
elif [[ "$TAG" == *"beta"* ]]; then
@@ -830,34 +783,61 @@ jobs:
RELEASE_TYPE="release"
fi
# Create release title
if [[ "$IS_PRERELEASE" == "true" ]]; then
TITLE="RustFS $VERSION (${RELEASE_TYPE})"
# Check if release already exists
if gh release view "$TAG" >/dev/null 2>&1; then
echo "Release $TAG already exists"
RELEASE_ID=$(gh release view "$TAG" --json databaseId --jq '.databaseId')
RELEASE_URL=$(gh release view "$TAG" --json url --jq '.url')
else
TITLE="RustFS $VERSION"
# Get release notes from tag message
RELEASE_NOTES=$(git tag -l --format='%(contents)' "${TAG}")
if [[ -z "$RELEASE_NOTES" || "$RELEASE_NOTES" =~ ^[[:space:]]*$ ]]; then
if [[ "$IS_PRERELEASE" == "true" ]]; then
RELEASE_NOTES="Pre-release ${VERSION} (${RELEASE_TYPE})"
else
RELEASE_NOTES="Release ${VERSION}"
fi
fi
# Create release title
if [[ "$IS_PRERELEASE" == "true" ]]; then
TITLE="RustFS $VERSION (${RELEASE_TYPE})"
else
TITLE="RustFS $VERSION"
fi
# Create the release
PRERELEASE_FLAG=""
if [[ "$IS_PRERELEASE" == "true" ]]; then
PRERELEASE_FLAG="--prerelease"
fi
gh release create "$TAG" \
--title "$TITLE" \
--notes "$RELEASE_NOTES" \
$PRERELEASE_FLAG \
--draft
RELEASE_ID=$(gh release view "$TAG" --json databaseId --jq '.databaseId')
RELEASE_URL=$(gh release view "$TAG" --json url --jq '.url')
fi
./scripts/release/create_or_update_release.sh \
"$TAG" \
"$TARGET_COMMITISH" \
"$TITLE" \
"$IS_PRERELEASE"
echo "release_id=$RELEASE_ID" >> "$GITHUB_OUTPUT"
echo "release_url=$RELEASE_URL" >> "$GITHUB_OUTPUT"
echo "Created release: $RELEASE_URL"
# Prepare and upload release assets
upload-release-assets:
name: Upload Release Assets
needs: [ build-check, build-rustfs, create-release ]
if: startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'preview' || needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
if: startsWith(github.ref, 'refs/tags/') && needs.build-check.outputs.build_type != 'development'
runs-on: ubuntu-latest
timeout-minutes: 30
permissions:
contents: write
actions: read
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Download all build artifacts
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
@@ -940,10 +920,9 @@ jobs:
# the pointed-to version is a prerelease.
update-latest-version:
name: Update Latest Version
needs: [ build-check, publish-release ]
if: startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
needs: [ build-check, upload-release-assets ]
if: startsWith(github.ref, 'refs/tags/')
runs-on: ubuntu-latest
timeout-minutes: 30
steps:
- name: Update latest.json
env:
@@ -1001,34 +980,51 @@ jobs:
publish-release:
name: Publish Release
needs: [ build-check, create-release, upload-release-assets ]
if: startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'preview' || needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
if: startsWith(github.ref, 'refs/tags/') && needs.build-check.outputs.build_type != 'development'
runs-on: ubuntu-latest
timeout-minutes: 30
permissions:
contents: write
steps:
- name: Publish release
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- name: Update release notes and publish
env:
GH_TOKEN: ${{ github.token }}
shell: bash
run: |
TAG="${{ needs.build-check.outputs.version }}"
VERSION="${{ needs.build-check.outputs.version }}"
IS_PRERELEASE="${{ needs.build-check.outputs.is_prerelease }}"
BUILD_TYPE="${{ needs.build-check.outputs.build_type }}"
RELEASE_ID="${{ needs.create-release.outputs.release_id }}"
# Publish the release and correct its channel state on retries.
# Only a stable final release may become GitHub Latest.
if [[ "$BUILD_TYPE" == "release" ]]; then
gh api --method PATCH "repos/${GITHUB_REPOSITORY}/releases/${RELEASE_ID}" \
-F draft=false \
-F prerelease=false \
-f make_latest=true >/dev/null
# Determine release type
if [[ "$BUILD_TYPE" == "prerelease" ]]; then
if [[ "$TAG" == *"alpha"* ]]; then
RELEASE_TYPE="alpha"
elif [[ "$TAG" == *"beta"* ]]; then
RELEASE_TYPE="beta"
elif [[ "$TAG" == *"rc"* ]]; then
RELEASE_TYPE="rc"
else
RELEASE_TYPE="prerelease"
fi
else
gh api --method PATCH "repos/${GITHUB_REPOSITORY}/releases/${RELEASE_ID}" \
-F draft=false \
-F prerelease=true \
-f make_latest=false >/dev/null
RELEASE_TYPE="release"
fi
# Get original release notes from tag
ORIGINAL_NOTES=$(git tag -l --format='%(contents)' "${TAG}")
if [[ -z "$ORIGINAL_NOTES" || "$ORIGINAL_NOTES" =~ ^[[:space:]]*$ ]]; then
if [[ "$IS_PRERELEASE" == "true" ]]; then
ORIGINAL_NOTES="Pre-release ${VERSION} (${RELEASE_TYPE})"
else
ORIGINAL_NOTES="Release ${VERSION}"
fi
fi
# Publish the release (remove draft status)
gh release edit "$TAG" --draft=false
echo "🎉 Released $TAG successfully!"
echo "📄 Release URL: ${{ needs.create-release.outputs.release_url }}"
-265
View File
@@ -1,265 +0,0 @@
# Copyright 2026 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Sole writer of the Rust dependency caches that ci.yml restores.
#
# Why this is a separate workflow rather than steps inside ci.yml: ci.yml's
# concurrency group cancels in-progress runs on main pushes, and merges land far
# faster than its 70-minute pipeline. Measured over 15 consecutive main pushes:
# 12 cancelled, 2 failed, 0 succeeded. A cancelled run never reaches
# Swatinem/rust-cache's post step (cache-on-failure does not cover cancellation),
# so the writer lanes were saving nothing and every PR paid a cold restore —
# 11.8-20.9 minutes of "Setup Rust environment" against 0.7-3.4 warm.
#
# Splitting cache writing out of the test pipeline lets ci.yml keep cancelling
# superseded runs (which is correct — nobody needs test results for a commit
# that is already three merges behind) while the caches still get written.
#
# The group below deliberately does NOT cancel in progress; see the comment on
# it for how that bounds concurrency and why it is scoped by event.
#
# Each job below owns exactly one shared-key and is the only place that sets
# cache-save-if to anything but 'false' for it; every lane in ci.yml reads.
# scripts/security/check_cache_save_if.sh keeps the declarations explicit.
#
# The builds are supersets of what the reading lanes compile, because a reader
# restores only what the writer saved. Feature resolution matters here: a lane
# built with e2e-test-hooks resolves dependency features differently, which
# changes -Cmetadata, so the plain build does not cover it. See
# rustfs/backlog#1600.
name: Cache Warm
on:
push:
branches: [ main ]
# Mirrors ci.yml's push paths-ignore: if a commit cannot change what ci.yml
# compiles, it cannot change what ci.yml needs restored either.
paths-ignore:
- "**.md"
- "docs/**"
- "deploy/**"
- "scripts/dev_*.sh"
- "scripts/probe.sh"
- "LICENSE*"
- ".gitignore"
- ".dockerignore"
- "README*"
- "**/*.png"
- "**/*.jpg"
- "**/*.svg"
- ".github/workflows/build.yml"
- ".github/workflows/docker.yml"
- ".github/workflows/audit.yml"
workflow_dispatch:
inputs:
emit_timings:
description: >-
Also emit cargo --timings for the ci-dev build and upload it. Used to
decide whether sccache is worth adopting (rustfs/backlog#1601 gate).
required: false
default: false
type: boolean
permissions:
contents: read
# Scoped by event. A push run and a dispatch run do not compete: GitHub keeps
# one running plus one pending per group, so with a single shared group a
# manually dispatched run was displaced as pending by the next merge and
# cancelled — observed three times in a row, which made the --timings gate in
# rustfs/backlog#1601 effectively impossible to trigger while main was busy.
#
# Still no cancel-in-progress: a burst of merges collapses into "current run
# finishes, newest queued run follows" rather than a pile-up, which is what
# bounds this workflow to one self-hosted runner per event type.
#
# The two paths can now overlap and race to save the same key. That is benign:
# the loser finds the key already present and skips, and both builds produce the
# same artifacts from the same commit.
concurrency:
group: cache-warm-${{ github.event_name }}
cancel-in-progress: false
env:
CARGO_TERM_COLOR: always
jobs:
# Readers: test-and-lint, test-ilm-integration-serial, build-rustfs-debug-binary,
# e2e-tests, e2e-full.
warm-ci-dev:
name: Warm ci-dev
runs-on: sm-standard-4
timeout-minutes: 90
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: stable
cache-shared-key: ci-dev
cache-save-if: 'true'
install-build-packaging-tools: 'false'
# rustfs/backlog#1601 gate. sccache can only cache compilation units whose
# --emit includes link, so it covers workspace rlibs and nothing else:
# clippy is metadata-only, and the ~100 test binaries, the rustfs bin and
# every build script invoke the system linker. Before spending a bucket,
# credentials and a supply-chain boundary on it, measure how much of the
# build is actually rlib codegen.
#
# Read from the report: workspace lib codegen as a share of the build, and
# s3select-query's own rlib as a share. The plan adopts sccache only above
# 50% and 25% respectively; if linking dominates instead, the answer is
# mold/lld plus split-debuginfo, which is exactly the part sccache cannot
# touch. Off by default — this doubles the ci-dev build.
- name: Build ci-dev superset (with --timings)
if: inputs.emit_timings
env:
CARGO_BUILD_JOBS: "2"
run: cargo build --workspace --all-targets --timings
- name: Upload cargo timings report
if: inputs.emit_timings
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
with:
name: cargo-timings-ci-dev
path: target/cargo-timings/
retention-days: 30
if-no-files-found: error
# --all-targets covers the test binaries nextest builds, including
# e2e_test, which test-and-lint's own run excludes. The second build adds
# the e2e-test-hooks feature resolution that build-rustfs-debug-binary uses
# and that no lint lane enables.
- name: Build ci-dev superset
env:
# Same limit ci.yml puts on its nextest step: this builds the same
# ~100 workspace test binaries, and three concurrent links saturate the
# self-hosted runner's overlay I/O and can wedge Cargo (#5394).
CARGO_BUILD_JOBS: "2"
run: |
cargo build --workspace --all-targets
cargo build -p rustfs --bins --features e2e-test-hooks
# Runs before rust-cache's post step, so these are the sizes it is about
# to archive. Reported so the cache-all-crates decision stays evidence-led:
# registry/src is what that flag prunes, registry/cache is what the pruned
# sources are re-unpacked from. See rustfs/backlog#1600.
- name: Report cache input sizes
if: always()
run: |
# tee, not a plain redirect: sent only to $GITHUB_STEP_SUMMARY these
# numbers are readable in the UI but absent from the job log, and the
# REST API exposes the log, not the summary — which made the figures
# unreachable for exactly the scripted comparison they exist for.
sizes="$(du -sh ~/.cargo/registry/src ~/.cargo/registry/cache \
~/.cargo/registry/index ~/.cargo/git target 2>/dev/null || true)"
echo "cache-input-sizes-begin"
printf '%s\n' "$sizes"
echo "cache-input-sizes-end"
{
echo "### Cache input sizes (ci-dev)"
echo '```'
printf '%s\n' "$sizes"
echo '```'
} >> "$GITHUB_STEP_SUMMARY"
# Readers: test-and-lint-rio-v2, build-rustfs-debug-binary-rio-v2.
warm-ci-feat-rio:
name: Warm ci-feat-rio
runs-on: sm-standard-4
timeout-minutes: 90
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: stable
cache-shared-key: ci-feat-rio
cache-save-if: 'true'
install-build-packaging-tools: 'false'
- name: Build ci-feat-rio superset
run: |
cargo build -p rustfs -p rustfs-ecstore --all-targets --features rio-v2
cargo build -p rustfs --bins --features rio-v2,e2e-test-hooks
# Readers: the swift and sftp legs of test-and-lint-protocols. Built in
# sequence rather than as `--features swift,sftp`, which is a combination no
# lane actually compiles; running both leaves the union in target/.
warm-ci-feat-proto:
name: Warm ci-feat-proto
runs-on: sm-standard-4
timeout-minutes: 90
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: stable
cache-shared-key: ci-feat-proto
cache-save-if: 'true'
install-build-packaging-tools: 'false'
- name: Build ci-feat-proto superset
run: |
cargo build -p rustfs -p rustfs-protocols --all-targets --features swift
cargo build -p rustfs -p rustfs-protocols --all-targets --features sftp
# Reader: uring-integration. Runs on ubuntu-latest to match it: rust-cache's
# key covers runner.os and arch but not the runner label or image, so a cache
# written on sm-standard-4 would be restored by the hosted runner as if it
# belonged to it.
warm-ci-uring:
name: Warm ci-uring
runs-on: ubuntu-latest
timeout-minutes: 60
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: stable
cache-shared-key: ci-uring
cache-save-if: 'true'
install-build-packaging-tools: 'false'
- name: Install build dependencies
run: sudo apt-get update && sudo apt-get install -y protobuf-compiler
- name: Build ci-uring superset
run: cargo build -p rustfs-ecstore --all-targets
+8 -85
View File
@@ -12,24 +12,18 @@
# See the License for the specific language governing permissions and
# limitations under the License.
# Companion to ci.yml for required status checks.
# Companion to ci.yml for the required "Test and Lint" status check.
#
# ci.yml skips docs-only pull requests via paths-ignore, but the branch ruleset
# requires a check named "Test and Lint" — without this workflow a docs-only PR
# would wait on it forever. This workflow triggers on exactly the paths ci.yml
# ignores and reports success under the same job name. Mixed PRs trigger both
# workflows and the real check still gates: a required check with any failing
# run blocks the merge.
# ci.yml skips docs-only pull requests via paths-ignore, but the branch
# ruleset requires a check named "Test and Lint" — without this workflow a
# docs-only PR would wait on that check forever. This workflow triggers on
# exactly the paths ci.yml ignores and reports an instant success under the
# same job name. Mixed PRs trigger both workflows and the real check still
# gates: a required check with any failing run blocks the merge.
# https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/defining-the-mergeability-of-pull-requests/troubleshooting-required-status-checks#handling-skipped-but-required-checks
#
# "Quick Checks" is mirrored here ahead of the ruleset change that will make it
# required too (rustfs/backlog#1599). Until that change lands this job is
# inert; mirroring it first is what lets the ruleset change happen without
# stranding docs-only PRs on a check nobody reports.
#
# Keep the paths list below in sync with the pull_request paths-ignore list
# in ci.yml, and keep the quick-checks steps below byte-identical to the
# quick-checks job in ci.yml.
# in ci.yml.
name: Continuous Integration (docs only)
@@ -53,88 +47,17 @@ on:
- ".github/workflows/build.yml"
- ".github/workflows/docker.yml"
- ".github/workflows/audit.yml"
- "flake.lock"
permissions:
contents: read
jobs:
# Deliberately NOT a bare `echo`. Once "Quick Checks" becomes a required
# check, ci.yml gates every expensive job behind it, so a mixed PR reports
# two check runs with this name: the real one (45-51s) and this companion.
# GitHub has no written contract for how it picks between same-named
# required check runs ("latest wins" vs "any failure blocks"), so instead of
# relying on ordering we make both runs execute the same commands against
# the same merge ref — their conclusions are then necessarily identical and
# the choice does not matter. Keep these steps byte-identical to the
# quick-checks job in ci.yml (a guard script that asserts this, and the paths
# sync below, is tracked in rustfs/backlog#1603).
#
# For a genuinely docs-only PR this adds no strictness (no code changed, so
# fmt and the guards always pass) and costs ~50s of ubuntu-latest.
quick-checks:
name: Quick Checks
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Install ripgrep
run: sudo apt-get update && sudo apt-get install -y ripgrep
- name: Install Rust toolchain
uses: dtolnay/rust-toolchain@29eef336d9b2848a0b548edc03f92a220660cdb8 # stable
with:
components: rustfmt
- name: Check code formatting
run: cargo fmt --all --check
- name: Check unsafe code allowances
run: ./scripts/check_unsafe_code_allowances.sh
- name: Check layered dependencies
run: ./scripts/check_layer_dependencies.sh
- name: Check architecture migration rules
run: ./scripts/check_architecture_migration_rules.sh
- name: Check logging guardrails
run: ./scripts/check_logging_guardrails.sh
- name: Check tokio io-uring feature guard
run: ./scripts/check_no_tokio_io_uring.sh
- name: Check extension schema boundaries
run: ./scripts/check_extension_schema_boundaries.sh
- name: Check body-cache whitelist guard
run: ./scripts/check_body_cache_whitelist.sh
- name: Check s3s footprint ratchet
run: ./scripts/check_s3s_footprint.sh
- name: Check no planning docs committed
run: ./scripts/check_no_planning_docs.sh
- name: Check CI paths stay in sync
run: ./scripts/check_ci_paths_sync.sh
- name: Check io_uring lane --lib precondition
run: ./scripts/check_uring_lane_lib_only.sh
test-and-lint:
name: Test and Lint
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
# Docs-only PRs skip the full code CI, but they are exactly where a
# planning-type document could be slipped in (git add -f bypasses
+58 -346
View File
@@ -33,7 +33,6 @@ on:
- ".github/workflows/build.yml"
- ".github/workflows/docker.yml"
- ".github/workflows/audit.yml"
- "flake.lock"
pull_request:
types: [ opened, synchronize, reopened, closed ]
branches: [ main ]
@@ -55,7 +54,6 @@ on:
- ".github/workflows/build.yml"
- ".github/workflows/docker.yml"
- ".github/workflows/audit.yml"
- "flake.lock"
merge_group:
types: [ checks_requested ]
schedule:
@@ -83,7 +81,6 @@ jobs:
name: Cancel Closed PR Runs
if: github.event_name == 'pull_request' && github.event.action == 'closed'
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- name: Explain cancellation run
run: echo "PR closed; this run only cancels older runs in the same concurrency group."
@@ -92,20 +89,13 @@ jobs:
name: Typos
if: github.event_name != 'pull_request' || github.event.action != 'closed'
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Typos check with custom config file
uses: crate-ci/typos@37bb98842b0d8c4ffebdb75301a13db0267cef89 # master
# Fast, compile-free checks that fail early so contributors get feedback in
# ~1 minute instead of waiting for the full test job.
#
# These steps are mirrored byte-for-byte in ci-docs-only.yml so that a mixed
# PR, which reports two check runs named "Quick Checks", cannot get one red
# and one green. Edit both jobs together.
quick-checks:
name: Quick Checks
if: github.event_name != 'pull_request' || github.event.action != 'closed'
@@ -114,8 +104,6 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Install ripgrep
run: sudo apt-get update && sudo apt-get install -y ripgrep
@@ -137,9 +125,6 @@ jobs:
- name: Check architecture migration rules
run: ./scripts/check_architecture_migration_rules.sh
- name: Check logging guardrails
run: ./scripts/check_logging_guardrails.sh
- name: Check tokio io-uring feature guard
run: ./scripts/check_no_tokio_io_uring.sh
@@ -149,163 +134,38 @@ jobs:
- name: Check body-cache whitelist guard
run: ./scripts/check_body_cache_whitelist.sh
- name: Check s3s footprint ratchet
run: ./scripts/check_s3s_footprint.sh
- name: Check no planning docs committed
run: ./scripts/check_no_planning_docs.sh
- name: Check CI paths stay in sync
run: ./scripts/check_ci_paths_sync.sh
- name: Check io_uring lane --lib precondition
run: ./scripts/check_uring_lane_lib_only.sh
test-and-lint:
name: Test and Lint
if: github.event_name != 'pull_request' || github.event.action != 'closed'
needs: [ quick-checks ]
runs-on: sm-standard-4
timeout-minutes: 90
# Both lines are required. Job-level `permissions` replaces the workflow
# block rather than merging with it, so declaring only `actions: write`
# would drop `contents: read` and break this job's checkout and the
# repo-token the setup action hands to setup-protoc.
permissions:
contents: read
actions: write
timeout-minutes: 60
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
# This job's token can cancel runs and delete Actions caches. Checkout
# otherwise writes it into .git/config, where a PR's own build.rs or
# proc-macro could read it back out.
persist-credentials: false
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: stable
# Every lane in this workflow reads its cache and none writes it.
# cache-warm.yml is the sole writer for all four keys: this workflow
# cancels superseded runs on main, and a cancelled run never reaches
# rust-cache's post step, so writing from here saved nothing (12 of 15
# consecutive main-push runs were cancelled). See rustfs/backlog#1600.
cache-shared-key: ci-dev
cache-save-if: 'false'
install-build-packaging-tools: 'false'
- name: Prepare test evidence
run: |
mkdir -p artifacts/test-and-lint
{
echo "run_id=${GITHUB_RUN_ID}"
echo "job=${GITHUB_JOB}"
echo "runner=${RUNNER_NAME}"
echo "started_at=$(date --utc --iso-8601=seconds)"
} > artifacts/test-and-lint/run-metadata.txt
cache-shared-key: ci-test
github-token: ${{ secrets.GITHUB_TOKEN }}
cache-save-if: ${{ github.ref == 'refs/heads/main' }}
# Clippy runs before the test pass: lint failures are the most common
# CI-only breakage and should surface in minutes, not after 20+ minutes
# of tests.
# Sampled too: clippy is the natural control arm for any CARGO_BUILD_JOBS
# experiment, since --all-targets is check-only for workspace members and
# never links the ~100 test binaries the limit exists to throttle.
- name: Run clippy lints
run: |
./scripts/ci/resource_sampler.sh start clippy
trap './scripts/ci/resource_sampler.sh stop' EXIT
cargo clippy --all-targets -- -D warnings
run: cargo clippy --all-targets -- -D warnings
- name: Run nextest tests
env:
# #5394 mitigation, now under a measured experiment (backlog#1601).
#
# 2 was chosen when three concurrent workspace test links were believed
# to saturate the runner's overlay I/O and wedge Cargo until the 75m
# timeout. cgroup v2 readings from the sampler show the pod actually
# has 14 CPUs and 28GB (peak use 2.1GB), so 2 throttles compilation to
# a seventh of what is available and memory was never the constraint —
# the label name "sm-standard-4" had led everyone, including the
# original mitigation, to assume 4 cores.
#
# Raised to 3 on main pushes and manual dispatches; PRs keep 2 so the
# merge path is untouched while the experiment runs.
#
# Dispatch is included because push alone cannot supply the samples:
# this workflow cancels superseded runs on main, and only 4 of the last
# 20 push-triggered Test and Lint jobs reached a terminal state — at
# that rate ten samples would take roughly fifty merges. The
# concurrency group is scoped by event_name, so a dispatched run has
# its own group and is not cancelled by merge traffic, which makes the
# sample collectable on demand rather than by waiting.
#
# Baseline over 17 samples at 2:
# median nextest/clippy step ratio 1.95, spread 1.85-2.06. The gate-2
# criterion is that ratio dropping at least 10% (below ~1.76) with no
# 75m timeout and no run showing three consecutive samples of
# rustc/collect2/rust-lld in D state. If it does not, the conclusion is
# "this limit is not the bottleneck" — fix it back at 2 and record the
# experiment, which is a result, not a failure.
#
# Must stay step-level: rust-cache hashes CARGO/CC/CFLAGS/CXX/CMAKE/RUST
# prefixed variables from process.env into the cache key, so promoting
# this to job level would rotate every key on this lane.
CARGO_BUILD_JOBS: ${{ (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && '3' || '2' }}
- name: Run tests
run: |
mkdir -p artifacts/test-and-lint
./scripts/ci/resource_sampler.sh start nextest
trap './scripts/ci/resource_sampler.sh stop' EXIT
set +e
NEXTEST_HIDE_PROGRESS_BAR=1 timeout --verbose --signal=TERM --kill-after=30s 75m \
cargo nextest run --profile ci --all --exclude e2e_test \
--status-level all --final-status-level all \
2>&1 | tee artifacts/test-and-lint/nextest.log
status=${PIPESTATUS[0]}
{
echo "command=cargo nextest run --profile ci --all --exclude e2e_test"
echo "exit_status=${status}"
echo "finished_at=$(date --utc --iso-8601=seconds)"
echo
echo "Remaining test-related processes:"
pgrep -af 'cargo|nextest|target/.*/deps/' || true
echo
echo "Kernel OOM / kill events:"
dmesg -T 2>/dev/null | grep -iE 'oom|out of memory|killed process' | tail -20 || true
} > artifacts/test-and-lint/nextest-diagnostics.txt
exit "${status}"
- name: Run documentation tests
run: |
mkdir -p artifacts/test-and-lint
set +e
timeout --verbose --signal=TERM --kill-after=30s 15m \
cargo test --all --doc \
2>&1 | tee artifacts/test-and-lint/doctest.log
status=${PIPESTATUS[0]}
{
echo "command=cargo test --all --doc"
echo "exit_status=${status}"
echo "finished_at=$(date --utc --iso-8601=seconds)"
echo
echo "Remaining test-related processes:"
pgrep -af 'cargo|rustdoc|target/.*/deps/' || true
} > artifacts/test-and-lint/doctest-diagnostics.txt
exit "${status}"
- name: Upload test reports and diagnostics
if: always()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
with:
name: junit-test-and-lint-${{ github.run_number }}
path: |
target/nextest/ci/junit.xml
artifacts/test-and-lint
retention-days: 3
if-no-files-found: error
cargo nextest run --profile ci --all --exclude e2e_test
cargo test --all --doc
# rustfs/backlog#1289: fail if a seed rule's log anchor no longer exists
# verbatim in the source tree (log message drifted without updating the
@@ -314,6 +174,15 @@ jobs:
- name: Check log-analyzer rule anchors
run: ./scripts/check_log_analyzer_rules.sh
- name: Upload test junit report
if: always()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
with:
name: junit-test-and-lint-${{ github.run_number }}
path: target/nextest/ci/junit.xml
retention-days: 3
if-no-files-found: ignore
# Explicit gate for migration-critical suites. These tests already ran in
# the full nextest pass above; a single filtered nextest invocation keeps
# the named gate without rebuilding or re-running them one package at a time.
@@ -330,50 +199,6 @@ jobs:
- name: Run rebalance/decommission migration proofs
run: ./scripts/check_migration_gate_count.sh
# Early stop. Once this job has failed the PR cannot merge, so the sibling
# lanes are burning runners on a result nobody can act on: on run
# 30674613104 three lanes had already failed while Test and Lint and the
# rio-v2 variant kept going past 70 minutes.
#
# Only this job may cancel. The lanes that are NOT required checks
# (protocols, ILM, e2e, s3-tests) must never hold that power: a flake in
# one of them would turn the required "Test and Lint" into `cancelled`,
# which blocks the merge. Today a maintainer can merge with sftp red, and
# that has to stay true.
#
# These steps run last so the `if: always()` artifact upload above still
# captures logs and diagnostics before the run goes away.
- name: Annotate early-stop reason
if: failure() && github.event_name == 'pull_request'
run: |
{
echo "## CI early-stop"
echo "Job \`${GITHUB_JOB}\` (Test and Lint) failed; cancelling run ${GITHUB_RUN_ID} to free runners."
echo "Sibling jobs showing **cancelled** were stopped by this job, not by their own failure."
} >> "$GITHUB_STEP_SUMMARY"
# curl rather than `gh`: every existing `gh` call in this repo runs on
# ubuntu-latest, and the sm-standard-* images are custom and trimmed (they
# ship no C toolchain, see the e2e job below), so `gh` is not known to
# exist here.
#
# Fork PRs are excluded explicitly instead of relying on the error path:
# their GITHUB_TOKEN is forced read-only and job-level permissions cannot
# raise it, so the call would always 403. Skipping keeps their logs clean.
- name: Cancel run on failure (same-repo PR only)
if: >-
failure() && github.event_name == 'pull_request'
&& github.event.pull_request.head.repo.full_name == github.repository
continue-on-error: true
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
curl -fsS -X POST \
-H "Authorization: Bearer ${GH_TOKEN}" \
-H "Accept: application/vnd.github+json" \
-H "X-GitHub-Api-Version: 2022-11-28" \
"${GITHUB_API_URL}/repos/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}/cancel" || true
# Dedicated serial lane for the ILM / lifecycle integration tests. These tests
# drive the object layer through process-global singletons (the GLOBAL_ENV
# ECStore, the global tier-config manager, background-expiry workers) and bind
@@ -387,7 +212,6 @@ jobs:
test-ilm-integration-serial:
name: ILM Integration (serial)
if: github.event_name != 'pull_request' || github.event.action != 'closed'
needs: [ quick-checks ]
runs-on: sm-standard-4
timeout-minutes: 45
env:
@@ -395,16 +219,14 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: stable
cache-shared-key: ci-dev
cache-save-if: 'false'
install-build-packaging-tools: 'false'
cache-shared-key: ci-ilm-serial
github-token: ${{ secrets.GITHUB_TOKEN }}
cache-save-if: ${{ github.ref == 'refs/heads/main' }}
# test_transition_and_restore_flows was re-enabled by rustfs/backlog#1303:
# its "missing xl.meta on disk2" was a test-util bug (open_disk hardcoded
@@ -427,7 +249,6 @@ jobs:
test-and-lint-rio-v2:
name: Test and Lint (rio-v2)
if: github.event_name != 'pull_request' || github.event.action != 'closed'
needs: [ quick-checks ]
runs-on: sm-standard-4
timeout-minutes: 60
env:
@@ -435,16 +256,14 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: stable
cache-shared-key: ci-feat-rio
cache-save-if: 'false'
install-build-packaging-tools: 'false'
cache-shared-key: ci-test-rio-v2
github-token: ${{ secrets.GITHUB_TOKEN }}
cache-save-if: ${{ github.ref == 'refs/heads/main' }}
- name: Run rio-v2 clippy lints
run: cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings
@@ -457,17 +276,10 @@ jobs:
test-and-lint-protocols:
name: "Test and Lint (${{ matrix.features.name }})"
if: github.event_name != 'pull_request' || github.event.action != 'closed'
needs: [ quick-checks ]
runs-on: sm-standard-4
timeout-minutes: 60
strategy:
# On a PR, one failing protocol leg is enough to know the PR is not ready,
# so stop the sibling leg instead of paying another ~40 minutes for it.
# Everywhere else (main pushes, the merge queue, the weekly schedule) keep
# the full signal: there we want to know whether swift AND sftp are broken,
# not just whichever failed first. This is the only part of the early-stop
# work that also covers fork PRs, since it needs no token.
fail-fast: ${{ github.event_name == 'pull_request' }}
fail-fast: false
matrix:
features:
- name: swift
@@ -479,16 +291,14 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: stable
cache-shared-key: ci-feat-proto
cache-save-if: 'false'
install-build-packaging-tools: 'false'
cache-shared-key: ci-test-${{ matrix.features.name }}
github-token: ${{ secrets.GITHUB_TOKEN }}
cache-save-if: ${{ github.ref == 'refs/heads/main' }}
- name: Run clippy with ${{ matrix.features.name }}
run: |
@@ -501,7 +311,6 @@ jobs:
build-rustfs-debug-binary:
name: Build RustFS Debug Binary
if: github.event_name != 'pull_request' || github.event.action != 'closed'
needs: [ quick-checks ]
runs-on: sm-standard-4
timeout-minutes: 30
env:
@@ -509,19 +318,17 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: stable
cache-shared-key: ci-dev
cache-save-if: 'false'
install-build-packaging-tools: 'false'
cache-shared-key: ci-rustfs-debug-binary
cache-save-if: ${{ github.ref == 'refs/heads/main' }}
github-token: ${{ secrets.GITHUB_TOKEN }}
- name: Build debug binary
run: cargo build -p rustfs --bins --features e2e-test-hooks
run: cargo build -p rustfs --bins
- name: Upload debug binary
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
@@ -534,7 +341,6 @@ jobs:
build-rustfs-debug-binary-rio-v2:
name: Build RustFS Debug Binary (rio-v2)
if: github.event_name != 'pull_request' || github.event.action != 'closed'
needs: [ quick-checks ]
runs-on: sm-standard-4
timeout-minutes: 30
env:
@@ -542,19 +348,17 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: stable
cache-shared-key: ci-feat-rio
cache-save-if: 'false'
install-build-packaging-tools: 'false'
cache-shared-key: ci-rustfs-debug-binary-rio-v2
cache-save-if: ${{ github.ref == 'refs/heads/main' }}
github-token: ${{ secrets.GITHUB_TOKEN }}
- name: Build debug binary with rio-v2
run: cargo build -p rustfs --bins --features rio-v2,e2e-test-hooks
run: cargo build -p rustfs --bins --features rio-v2
- name: Upload debug binary
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
@@ -566,14 +370,6 @@ jobs:
uring-integration:
name: io_uring Integration (real)
# The pull_request trigger includes `closed` purely so the concurrency
# group cancels in-flight runs of a closed PR; every other job opts out of
# that run with this guard (or is skipped through its `needs` chain). This
# job had neither, so each closed/merged PR really ran the whole io_uring
# suite (measured 4m17s / 7m19s / 7m31s on runs 30678272341 / 30678117601 /
# 30662728539) and kept the cancellation run in progress for minutes.
if: github.event_name != 'pull_request' || github.event.action != 'closed'
needs: [ quick-checks ]
# GitHub-hosted ubuntu-latest runs a recent kernel with io_uring and, unlike
# a container, applies no seccomp filter that would block io_uring_setup — so
# the probe succeeds and the tests exercise the real UringBackend/FdCache/
@@ -584,24 +380,17 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: stable
# Keeps its own key rather than joining ci-dev. rust-cache's key is
# built from runner.os/arch plus rustc and lockfile fingerprints — it
# does NOT include the runner label or image. ubuntu-latest and
# sm-standard-4 are therefore indistinguishable to it, so sharing a key
# would let two different system images overwrite each other's
# artifacts, and would make a 2-core hosted runner unpack ci-dev's ~3GB
# instead of this lane's ~1.3GB. cache-warm.yml warms this key on
# ubuntu-latest for the same reason.
cache-shared-key: ci-uring
cache-save-if: 'false'
install-build-packaging-tools: 'false'
github-token: ${{ secrets.GITHUB_TOKEN }}
cache-save-if: ${{ github.ref == 'refs/heads/main' }}
- name: Install build dependencies
run: sudo apt-get update && sudo apt-get install -y protobuf-compiler
# ext4 supports O_DIRECT; the runner's default TMPDIR may sit on tmpfs or
# overlayfs, where open(O_DIRECT) returns EINVAL/EOPNOTSUPP and the native
@@ -628,17 +417,7 @@ jobs:
RUSTFS_IO_URING_READ_ENABLE: "true"
RUSTFS_URING_TESTS_MUST_RUN: "1"
TMPDIR: /mnt/rustfs-odirect
# --lib narrows what gets compiled, not what gets run: every selected
# test lives in the lib target. The 7 integration binaries under
# crates/ecstore/tests/ each reported "running 0 tests" here, so they
# were compiled and linked for nothing.
#
# The `uring_` filter must stay exactly as it is. libtest matches on
# substring, so it also selects names containing `during_` — 6 of the 18
# selected tests are such incidental matches. Narrowing the filter to
# `io_uring` would silently drop them, which is a coverage change.
# scripts/check_uring_lane_lib_only.sh guards the --lib precondition.
run: cargo test -p rustfs-ecstore --lib uring_ -- --test-threads=1 --nocapture
run: cargo test -p rustfs-ecstore uring_ -- --test-threads=1 --nocapture
e2e-tests:
name: End-to-End Tests
@@ -648,8 +427,6 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
# Full setup with dependency caching: the smoke-suite step below
# compiles the e2e_test crate, which pulls in most of the workspace.
@@ -658,9 +435,9 @@ jobs:
uses: ./.github/actions/setup
with:
rust-version: stable
cache-shared-key: ci-dev
cache-save-if: 'false'
install-build-packaging-tools: 'false'
cache-shared-key: ci-e2e
github-token: ${{ secrets.GITHUB_TOKEN }}
cache-save-if: ${{ github.ref == 'refs/heads/main' }}
# Download after the cache restore so the freshly built binary from the
# build job always wins over anything restored into target/debug.
@@ -673,17 +450,15 @@ jobs:
- name: Make binary executable
run: chmod +x ./target/debug/rustfs
# Build the e2e test graph once. The archive is reused by the security
# count-floor check and the smoke run below, avoiding a second compile of
# the same e2e_test target on cold runners (backlog#1645).
- name: Archive e2e smoke test binaries
env:
NEXTEST_ARCHIVE: ${{ runner.temp }}/rustfs-e2e-smoke.tar.zst
NEXTEST_LISTING: ${{ runner.temp }}/rustfs-e2e-smoke-list.json
run: |
cargo nextest archive --profile e2e-smoke -p e2e_test --archive-file "${NEXTEST_ARCHIVE}"
cargo nextest list --profile e2e-smoke --archive-file "${NEXTEST_ARCHIVE}" --message-format json > "${NEXTEST_LISTING}"
./scripts/check_security_smoke_count.sh check "${NEXTEST_LISTING}"
# Guard the security negative-auth smoke subset (backlog#1151 sec-5)
# against a rename or deletion silently dropping it out of the e2e-smoke
# filter. The script lists what the profile selects and fails if the count
# of security auth-rejection tests falls below the committed floor in
# .config/security-smoke-floor.txt (infra-12 count-floor mechanism). Run
# before the smoke suite so a thinned gate fails fast; the `nextest list`
# here compiles the e2e_test binaries the run below reuses.
- name: Check security smoke subset count floor
run: ./scripts/check_security_smoke_count.sh check
# PR smoke subset of the in-repo e2e suite (backlog#1149 ci-4). The
# profile.e2e-smoke default-filter in .config/nextest.toml is the single
@@ -691,30 +466,7 @@ jobs:
# adding new e2e jobs here. Each test spawns its own rustfs server on a
# random port and reuses the downloaded debug binary above.
- name: Run e2e smoke suite
env:
NEXTEST_ARCHIVE: ${{ runner.temp }}/rustfs-e2e-smoke.tar.zst
RUSTFS_E2E_LOG_DIR: ${{ runner.temp }}/rustfs-e2e-smoke-logs
run: |
cargo nextest run --profile e2e-smoke --archive-file "${NEXTEST_ARCHIVE}" \
--status-level all --final-status-level all --failure-output final
- name: Upload e2e smoke diagnostics
if: failure()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
with:
name: e2e-smoke-diagnostics-${{ github.run_number }}
path: |
${{ runner.temp }}/rustfs-e2e-smoke-logs/
${{ runner.temp }}/rustfs-e2e-smoke-list.json
if-no-files-found: warn
- name: Upload e2e smoke JUnit report
if: always()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
with:
name: e2e-smoke-junit-${{ github.run_number }}
path: target/nextest/e2e-smoke/junit.xml
if-no-files-found: warn
run: cargo nextest run --profile e2e-smoke -p e2e_test
- name: Install s3s-e2e test tool
uses: taiki-e/cache-cargo-install-action@7447f04c51f2ba27ca35e7f1e28fab848c5b3ba7 # v2
@@ -759,42 +511,14 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: stable
cache-shared-key: ci-dev
cache-save-if: 'false'
install-build-packaging-tools: 'false'
- name: Set up Python
uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
with:
python-version: "3.12"
- name: Install awscurl
run: |
python3 -m pip install --user --upgrade pip "awscurl==0.44"
echo "AWSCURL_PATH=$HOME/.local/bin/awscurl" >> "$GITHUB_ENV"
- name: Verify awscurl
run: test -x "$AWSCURL_PATH"
- name: Install Vault
run: |
VAULT_VERSION="1.17.6"
VAULT_ARCHIVE="vault_${VAULT_VERSION}_linux_amd64.zip"
curl -fsSLo "$RUNNER_TEMP/$VAULT_ARCHIVE" "https://releases.hashicorp.com/vault/${VAULT_VERSION}/${VAULT_ARCHIVE}"
echo "0cddc1fbbb88583b5ba5b845f9f8fae47c6fb39a6d48cd543c6ba6fd3ac1a669 $RUNNER_TEMP/$VAULT_ARCHIVE" | sha256sum --check --status
unzip -q "$RUNNER_TEMP/$VAULT_ARCHIVE" -d "$RUNNER_TEMP/vault-bin"
echo "RUSTFS_TEST_VAULT_BIN=$RUNNER_TEMP/vault-bin/vault" >> "$GITHUB_ENV"
- name: Verify Vault
run: |
"$RUSTFS_TEST_VAULT_BIN" version
cache-shared-key: ci-e2e
github-token: ${{ secrets.GITHUB_TOKEN }}
cache-save-if: ${{ github.ref == 'refs/heads/main' }}
# Download after the cache restore so the freshly built binary from the
# build job always wins over anything restored into target/debug.
@@ -830,8 +554,6 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Clean up previous test run
run: |
@@ -886,8 +608,6 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Download debug binary
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
@@ -948,20 +668,12 @@ jobs:
# evaluates ILM within ~2s of the due time, well inside the poll window.
s3-lifecycle-behavior-tests:
name: S3 Lifecycle Behavior Tests
# Also gated on e2e-tests, matching s3-implemented-tests: when the e2e smoke
# suite is already red this lane cannot tell us anything new, and it holds a
# sm-standard-4 for up to 30 minutes doing so. Both lanes only download the
# prebuilt debug binary (no cargo build), and s3-implemented-tests — which
# already waits on e2e-tests — finishes later anyway, so a green PR's total
# wall clock is unchanged.
needs: [ build-rustfs-debug-binary, e2e-tests ]
needs: [ build-rustfs-debug-binary ]
runs-on: sm-standard-4
timeout-minutes: 30
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Download debug binary
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
+4 -23
View File
@@ -22,18 +22,11 @@ on:
issue_comment:
types: [created, edited]
# Least privilege at the top, widened per job below. This workflow runs on
# pull_request_target and issue_comment, so it holds full secrets on every fork
# PR and on any comment anyone writes — the one place in this repository where a
# compromised action would be handed a repo-write token. It does not check out
# or execute PR code, so there is no pwn-request path today, but the blast
# radius should not depend on that staying true.
#
# contents: write in particular was never used: the signature records are
# written to rustfs/cla through the scoped app token created below, and nothing
# here writes to this repository's contents.
permissions:
contents: read
contents: write
pull-requests: write
issues: write
checks: write
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.event.issue.number || github.ref }}
@@ -43,26 +36,14 @@ jobs:
cancel-closed-pr-runs:
name: Cancel Closed PR Runs
if: github.event_name == 'pull_request_target' && github.event.action == 'closed'
# Echoes one line; the run exists only so the concurrency group cancels the
# in-flight run of a closed PR.
permissions: {}
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- name: Explain cancellation run
run: echo "PR closed; this run only cancels older runs in the same concurrency group."
cla:
if: ${{ (github.event_name != 'issue_comment' || github.event.issue.pull_request) && (github.event_name != 'pull_request_target' || github.event.action != 'closed') }}
# checks: write reports the merge-queue check run; pull-requests and issues
# let cla-bot comment and label. contents stays read — see the note above.
permissions:
contents: read
checks: write
issues: write
pull-requests: write
runs-on: ubuntu-latest
timeout-minutes: 30
steps:
- name: Report CLA result for merge queue
if: github.event_name == 'merge_group'
+1 -5
View File
@@ -62,16 +62,14 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: stable
cache-shared-key: ci-coverage
github-token: ${{ secrets.GITHUB_TOKEN }}
cache-save-if: ${{ github.ref == 'refs/heads/main' }}
install-build-packaging-tools: 'false'
- name: Install cargo-llvm-cov
uses: taiki-e/install-action@bffeee26d4db9be238a4ea78d8826604ebcb594d # v2
@@ -118,8 +116,6 @@ jobs:
issues: write
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Open or update failure-tracking issue
uses: ./.github/actions/schedule-failure-issue
with:
+22 -44
View File
@@ -66,7 +66,7 @@ env:
CARGO_TERM_COLOR: always
REGISTRY_DOCKERHUB: rustfs/rustfs
REGISTRY_GHCR: ghcr.io/${{ github.repository }}
REGISTRY_QUAY: quay.io/rustfs/rustfs
REGISTRY_QUAY: quay.io/${{ secrets.QUAY_USERNAME }}/rustfs
DOCKER_PLATFORMS: linux/amd64,linux/arm64
jobs:
@@ -82,10 +82,8 @@ jobs:
github.event_name == 'workflow_dispatch' ||
(github.event.workflow_run.conclusion == 'success' &&
github.event.workflow_run.event == 'push' &&
github.event.workflow_run.head_branch != 'main' &&
!contains(github.event.workflow_run.head_branch, '-preview'))
github.event.workflow_run.head_branch != 'main')
runs-on: ubuntu-latest
timeout-minutes: 30
outputs:
should_build: ${{ steps.check.outputs.should_build }}
should_push: ${{ steps.check.outputs.should_push }}
@@ -98,18 +96,11 @@ jobs:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
# For workflow_run events, checkout the specific commit that triggered the workflow
ref: ${{ github.event.workflow_run.head_sha || github.sha }}
- name: Check build conditions
id: check
env:
# dispatch inputs via env, not `${{ }}` interpolation: they are
# free-form strings and would otherwise be evaluated by bash.
INPUT_VERSION: ${{ github.event.inputs.version }}
INPUT_PUSH_IMAGES: ${{ github.event.inputs.push_images }}
INPUT_FORCE_REBUILD: ${{ github.event.inputs.force_rebuild }}
run: |
should_build=false
should_push=false
@@ -210,9 +201,9 @@ jobs:
elif [[ "${{ github.event_name }}" == "workflow_dispatch" ]]; then
# Manual trigger
input_version="$INPUT_VERSION"
input_version="${{ github.event.inputs.version }}"
version="${input_version}"
should_push="$INPUT_PUSH_IMAGES"
should_push="${{ github.event.inputs.push_images }}"
should_build=true
# Get short SHA
@@ -220,7 +211,7 @@ jobs:
echo "🎯 Manual Docker build triggered:"
echo " 📋 Requested version: $input_version"
echo " 🔧 Force rebuild: $INPUT_FORCE_REBUILD"
echo " 🔧 Force rebuild: ${{ github.event.inputs.force_rebuild }}"
echo " 🚀 Push images: $should_push"
case "$input_version" in
@@ -229,13 +220,6 @@ jobs:
create_latest=true
echo "🚀 Building with latest stable release version"
;;
*-preview*)
build_type="preview"
is_prerelease=true
should_build=false
should_push=false
echo "⏭️ Preview tags do not publish Docker images"
;;
# Prerelease versions (must match first, more specific)
v*alpha*|v*beta*|v*rc*|*alpha*|*beta*|*rc*)
build_type="prerelease"
@@ -306,8 +290,6 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Login to Docker Hub
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
@@ -343,28 +325,32 @@ jobs:
CREATE_LATEST="${{ needs.build-check.outputs.create_latest }}"
VARIANT_SUFFIX="${{ matrix.suffix }}"
# Convert version format for Dockerfile compatibility. The former
# DOCKER_CHANNEL was "release" down every branch and was passed as a
# build-arg no Dockerfile declares, so it is gone.
# Convert version format for Dockerfile compatibility
case "$VERSION" in
"latest")
# For stable latest, use RELEASE=latest + release CHANNEL
DOCKER_RELEASE="latest"
DOCKER_CHANNEL="release"
;;
v*)
# For versioned releases (v1.0.0), remove 'v' prefix for Dockerfile
DOCKER_RELEASE="${VERSION#v}"
DOCKER_CHANNEL="release"
;;
*)
# For other versions, pass as-is
DOCKER_RELEASE="${VERSION}"
DOCKER_CHANNEL="release"
;;
esac
echo "docker_release=$DOCKER_RELEASE" >> "$GITHUB_OUTPUT"
echo "docker_channel=$DOCKER_CHANNEL" >> "$GITHUB_OUTPUT"
echo "🐳 Docker build parameters:"
echo " - Original version: $VERSION"
echo " - Docker RELEASE: $DOCKER_RELEASE"
echo " - Docker CHANNEL: $DOCKER_CHANNEL"
# Generate tags based on build type
# Only support release and prerelease builds (no development builds)
@@ -418,24 +404,18 @@ jobs:
push: ${{ needs.build-check.outputs.should_push == 'true' }}
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
# No layer cache. This build compiles nothing — it downloads a
# release zip and runs apk/apt — so the cache could only save the
# minute or two those take, while creating a correctness problem: with
# RELEASE=latest the binary URL is resolved by curl *inside* a RUN
# layer, and the layer key does not include what that resolved to. A
# rebuild at the same RELEASE value (dispatch with version=latest, or
# a re-run of the same version) would hit the old layer and ship the
# previous release's binary. mode=max also consumed the same 10GB
# Actions cache quota the Rust lanes are fighting over.
#
# Only RELEASE is passed: it is the sole build-arg the Dockerfiles
# declare besides TARGETARCH. BUILDTIME, VERSION, BUILD_TYPE, REVISION
# and CHANNEL were never read by any stage (and BUILDTIME's $(date ...)
# was a literal here, not a shell substitution). BUILD_DATE and VCS_REF
# are declared by the Dockerfiles but deliberately left unset —
# supplying them would change the published image labels.
cache-from: |
type=gha,scope=docker-${{ matrix.variant }}
cache-to: |
type=gha,mode=max,scope=docker-${{ matrix.variant }}
build-args: |
BUILDTIME=$(date -u +'%Y-%m-%dT%H:%M:%SZ')
VERSION=${{ needs.build-check.outputs.version }}
BUILD_TYPE=${{ needs.build-check.outputs.build_type }}
REVISION=${{ github.sha }}
RELEASE=${{ steps.meta.outputs.docker_release }}
CHANNEL=${{ steps.meta.outputs.docker_channel }}
BUILDKIT_INLINE_CACHE=1
provenance: true
sbom: true
# Add retry mechanism by splitting the build process
@@ -451,7 +431,6 @@ jobs:
needs: [ build-check, build-docker ]
if: needs.build-check.outputs.should_build == 'true' && needs.build-check.outputs.should_push == 'true'
runs-on: ubuntu-latest
timeout-minutes: 30
permissions:
contents: read
security-events: write
@@ -506,7 +485,6 @@ jobs:
needs: [ build-check, build-docker ]
if: always() && needs.build-check.outputs.should_build == 'true'
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- name: Docker build completion summary
run: |
+16 -19
View File
@@ -14,24 +14,25 @@
# Nightly full replication e2e lane (backlog#1147 repl-1, deps: ci-4).
#
# The per-PR gate (ci.yml `e2e-tests` job, `--profile e2e-smoke`) runs the
# FAST replication tests. This scheduled lane runs the remaining heavier
# replication e2e tests that are unfit for a per-PR gate: remote-target TLS
# validation, bucket-replication data-plane/helper tests (PUT/delete + poll
# for convergence, HTTPS targets, active SSE failure contracts, event/history
# observers), and the `_real_dual_node` / `_real_three_node` /
# `_real_single_node` site-replication tests that each spawn full rustfs
# server processes.
# The per-PR gate (ci.yml `e2e-tests` job, `--profile e2e-smoke`) runs the 20
# FAST replication tests. This scheduled lane runs the remaining 27
# heavier replication e2e tests that are unfit for a per-PR gate:
#
# * 2 remote-target TLS validation tests.
# * 12 bucket-replication data-plane/helper tests (PUT/delete + poll for
# convergence; two replicate over HTTPS, two pin active SSE failure
# contracts, and one guards event/history observers). The SSE-S3 contract
# remains ignored under backlog#1291.
# * 11 `_real_dual_node` site-replication tests (each spawns TWO rustfs
# servers and drives the cross-process site-replication control plane).
# * 1 `_real_three_node` site-replication test.
# * 1 `_real_single_node` service-account round-trip test.
#
# The selection is the [profile.e2e-repl-nightly] default-filter in
# .config/nextest.toml — the single wiring mechanism (repl-1 / ci-4). Do NOT
# add ad-hoc cargo-test steps here; change the filterset instead. The
# authoritative membership and count come from
# `cargo nextest list -p e2e_test --profile e2e-repl-nightly`; the PR/nightly
# count invariant is maintained next to the filtersets in .config/nextest.toml
# (deliberately not duplicated here).
# add ad-hoc cargo-test steps here; change the filterset instead.
#
# Explicit division of labor: the nightly subset runs ONLY here, never double-run
# Explicit division of labor: these 27 tests run ONLY here, never double-run
# in ci-5's future e2e-full merge gate. TODO(ci-7): once the ci domain's
# consolidated scheduled e2e workflow exists, fold this interim repl-owned lane
# into it rather than growing a second scheduled entrypoint.
@@ -63,16 +64,14 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: stable
cache-shared-key: ci-e2e-repl
github-token: ${{ secrets.GITHUB_TOKEN }}
cache-save-if: ${{ github.ref == 'refs/heads/main' }}
install-build-packaging-tools: 'false'
# awscurl lets the STS dual-node test actually exercise its path. Without
# it the test skips gracefully with a visible log line
@@ -125,8 +124,6 @@ jobs:
issues: write
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Open or update failure-tracking issue
uses: ./.github/actions/schedule-failure-issue
with:
-11
View File
@@ -45,13 +45,6 @@
# The PR gate (ci.yml s3-implemented-tests) is unaffected: it avoids Docker
# via DEPLOY_MODE=binary and defers all pip setup to run.sh's self-bootstrap.
# DISABLED. This workflow is switched off in the repository's Actions settings
# (state: disabled_manually) and does not run on any trigger, including its cron
# and workflow_dispatch. That state lives in GitHub's UI and is invisible when
# reading this file, which has already misled at least one audit — hence this
# banner. Re-enabling is a UI action; anyone doing so should first check that the
# workflow still matches the current CI layout. See rustfs/backlog#1603.
#
name: e2e-s3tests
on:
@@ -142,8 +135,6 @@ jobs:
TEST_MODE: ${{ matrix.test-mode }}
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
# Provision Python explicitly rather than trusting the runner image to
# ship a working pip (ci-1: a bare python3 without pip is what broke the
@@ -363,8 +354,6 @@ jobs:
issues: write
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Open or update failure-tracking issue
uses: ./.github/actions/schedule-failure-issue
with:
+1 -16
View File
@@ -12,13 +12,6 @@
# See the License for the specific language governing permissions and
# limitations under the License.
# DISABLED. This workflow is switched off in the repository's Actions settings
# (state: disabled_manually) and does not run on any trigger, including its cron
# and workflow_dispatch. That state lives in GitHub's UI and is invisible when
# reading this file, which has already misled at least one audit — hence this
# banner. Re-enabling is a UI action; anyone doing so should first check that the
# workflow still matches the current CI layout. See rustfs/backlog#1603.
#
name: Fuzz
on:
@@ -66,7 +59,6 @@ jobs:
name: Cancel Closed PR Runs
if: github.event_name == 'pull_request' && github.event.action == 'closed'
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- name: Explain cancellation run
run: echo "PR closed; this run only cancels older runs in the same concurrency group."
@@ -87,14 +79,13 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: nightly
cache-shared-key: fuzz-${{ hashFiles('fuzz/Cargo.lock') }}
github-token: ${{ secrets.GITHUB_TOKEN }}
cache-save-if: ${{ github.ref == 'refs/heads/main' || github.event_name == 'schedule' }}
- name: Install cargo-fuzz
@@ -154,8 +145,6 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Download prebuilt fuzz binaries
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
@@ -211,8 +200,6 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Download prebuilt fuzz binaries
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
@@ -260,8 +247,6 @@ jobs:
issues: write
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Open or update failure-tracking issue
uses: ./.github/actions/schedule-failure-issue
with:
+9 -35
View File
@@ -32,14 +32,12 @@ permissions:
jobs:
build-helm-package:
runs-on: ubuntu-latest
timeout-minutes: 30
if: |
(github.event_name == 'workflow_dispatch' && !contains(github.event.inputs.version, '-preview')) ||
github.event_name == 'workflow_dispatch' ||
(
github.event.workflow_run.conclusion == 'success' &&
github.event.workflow_run.event == 'push' &&
contains(github.event.workflow_run.head_branch, '.') &&
!contains(github.event.workflow_run.head_branch, '-preview')
contains(github.event.workflow_run.head_branch, '.')
)
outputs:
@@ -50,26 +48,16 @@ jobs:
steps:
- name: Checkout helm chart repo
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
# Both inputs reach the shell through env rather than `${{ }}`
# interpolation. A git ref name may contain `$(...)` — anything without a
# space is a legal tag — and interpolation pastes it into the script
# verbatim, where bash would run it. Reading "$RAW_INPUT" instead makes it
# data.
- name: Normalize release version
id: version
env:
RAW_INPUT: ${{ github.event.inputs.version }}
RAW_BRANCH: ${{ github.event.workflow_run.head_branch }}
run: |
set -eux
if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then
RAW="$RAW_INPUT"
RAW="${{ github.event.inputs.version }}"
else
RAW="$RAW_BRANCH"
RAW="${{ github.event.workflow_run.head_branch }}"
fi
case "$RAW" in
@@ -84,13 +72,10 @@ jobs:
./scripts/helm_chart_version.sh "$RAW_TAG"
- name: Replace chart version and app version
env:
CHART_VERSION: ${{ steps.version.outputs.chart_version }}
APP_VERSION: ${{ steps.version.outputs.app_version }}
run: |
set -eux
sed -i -E "s/^version:.*/version: \"${CHART_VERSION}\"/" helm/rustfs/Chart.yaml
sed -i -E "s/^appVersion:.*/appVersion: \"${APP_VERSION}\"/" helm/rustfs/Chart.yaml
sed -i -E 's/^version:.*/version: "${{ steps.version.outputs.chart_version }}"/' helm/rustfs/Chart.yaml
sed -i -E 's/^appVersion:.*/appVersion: "${{ steps.version.outputs.app_version }}"/' helm/rustfs/Chart.yaml
- name: Set up Helm
uses: azure/setup-helm@b9e51907a09c216f16ebe8536097933489208112 # v4.3.0
@@ -115,7 +100,6 @@ jobs:
publish-helm-package:
runs-on: ubuntu-latest
timeout-minutes: 30
needs: [ build-helm-package ]
if: needs.build-helm-package.result == 'success'
@@ -123,8 +107,6 @@ jobs:
- name: Checkout helm package repo
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
# persist-credentials-exempt: this checkout's token IS the push credential —
# the job git-pushes to rustfs/helm below. Clearing it breaks chart publishing.
repository: rustfs/helm
token: ${{ secrets.RUSTFS_HELM_PACKAGE }}
@@ -140,19 +122,11 @@ jobs:
- name: Generate index
run: helm repo index . --url https://charts.rustfs.com
# app_version is derived from the triggering tag name, and this job holds
# the cross-repository push token with rustfs/helm already checked out —
# the worst place in the repo to paste an attacker-influenced string into
# a shell line. Passed through env so bash treats it as data.
- name: Push helm package and index file
env:
GIT_USERNAME: ${{ secrets.USERNAME }}
GIT_EMAIL: ${{ secrets.EMAIL_ADDRESS }}
APP_VERSION: ${{ needs.build-helm-package.outputs.app_version }}
run: |
set -eux
git config --global user.name "${GIT_USERNAME}"
git config --global user.email "${GIT_EMAIL}"
git config --global user.name "${{ secrets.USERNAME }}"
git config --global user.email "${{ secrets.EMAIL_ADDRESS }}"
git add .
git commit -m "Update rustfs helm package with ${APP_VERSION}." || echo "No changes to commit"
git commit -m "Update rustfs helm package with ${{ needs.build-helm-package.outputs.app_version }}." || echo "No changes to commit"
git push origin main
-8
View File
@@ -12,13 +12,6 @@
# See the License for the specific language governing permissions and
# limitations under the License.
# DISABLED. This workflow is switched off in the repository's Actions settings
# (state: disabled_manually) and does not run on any trigger, including its cron
# and workflow_dispatch. That state lives in GitHub's UI and is invisible when
# reading this file, which has already misled at least one audit — hence this
# banner. Re-enabling is a UI action; anyone doing so should first check that the
# workflow still matches the current CI layout. See rustfs/backlog#1603.
#
name: "issue-translator"
on:
issue_comment:
@@ -33,7 +26,6 @@ permissions:
jobs:
build:
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: usthe/issues-translate-action@b41f55ddc81d7d54bd542a4f289fe28ec081898e # v2.7
with:
+5 -58
View File
@@ -18,25 +18,11 @@
# This is NOT a PR gate. The fixtures are real MinIO backend trees generated on
# the fly (they are gitignored, never committed), so the job regenerates them
# each run with Docker and then runs the `#[ignore]` reader tests in
# rustfs/src/storage/minio_generated_read_test.rs.
#
# Scope: end-to-end MinIO-to-RustFS SSE interop is NOT implemented yet. Both
# envelope parsers reject MinIO's own wrapped-DEK shape — see
# `is_data_key_envelope` in crates/kms/src/encryption/dek.rs and the
# `deny_unknown_fields` `LocalSseDekEnvelope` in rustfs/src/storage/sse.rs — and
# closing that gap is tracked in rustfs/backlog#1638. Treat this job as the
# harness for #1638, not as standing evidence that a MinIO migration reads back.
# crates/ecstore/tests/minio_generated_read_test.rs.
#
# Runner: GitHub-hosted `ubuntu-latest`. It reliably ships Docker + Python,
# unlike the self-hosted fleet, whose pods drift in Docker/pip availability
# (see the infra note in e2e-s3tests.yml). Nightly + manual only.
# DISABLED. This workflow is switched off in the repository's Actions settings
# (state: disabled_manually) and does not run on any trigger, including its cron
# and workflow_dispatch. That state lives in GitHub's UI and is invisible when
# reading this file, which has already misled at least one audit — hence this
# banner. Re-enabling is a UI action; anyone doing so should first check that the
# workflow still matches the current CI layout. See rustfs/backlog#1603.
#
name: minio-interop
on:
@@ -62,62 +48,23 @@ jobs:
env:
# Fixed 32-byte test KMS key baked into the fixture lab; not a secret.
RUSTFS_MINIO_STATIC_KMS_KEY_B64: IyqsU3kMFloCNup4BsZtf/rmfHVcTgznO2F25CkEH1g=
# Single definition of "the interop tests", shared by the guard step and
# the run step so the two cannot drift apart.
#
# These used to live in crates/ecstore/tests/minio_generated_read_test.rs
# and were selected with `-p rustfs-ecstore -E
# 'binary(minio_generated_read_test)'`. #5435 moved them into the `rustfs`
# crate as a `#[cfg(test)] mod`, which deleted that test binary; the
# selector was never updated and has selected zero interop tests ever
# since (cargo-nextest 0.9.140 now rejects it outright: "operator didn't
# match any binary names", exit 94).
INTEROP_PACKAGE: rustfs
INTEROP_FEATURES: rio-v2
INTEROP_FILTER: "test(minio_generated_read_test::)"
INTEROP_REQUIRED_TESTS: '["reads_minio_generated_sse_s3_multipart_fixture", "reads_minio_generated_sse_kms_multipart_fixture", "rejects_minio_generated_sse_s3_fixture_with_wrong_kms_key", "rejects_minio_generated_sse_s3_fixture_with_truncated_ciphertext"]'
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: stable
cache-shared-key: ci-minio-interop
github-token: ${{ secrets.GITHUB_TOKEN }}
cache-save-if: ${{ github.ref == 'refs/heads/main' }}
- name: Generate real MinIO fixtures via Docker
run: bash crates/rio-v2/tests/minio_fixture_lab/capture_via_docker.sh
# `binary(...)` at least dies loudly when nothing matches, but `test(...)`
# is a perfectly valid filterset that matches zero tests, so the next
# rename or module move would leave this job selecting nothing and
# reporting success without executing a single interop assertion. Count
# the selection and require every core reader test, while allowing new
# reader cases to be added without changing this guard.
#
# Count only `filter-match.status == "matches"`: the top-level
# `test-count` in the JSON is the package total and ignores `-E` entirely.
- name: Assert the interop selector still matches tests
run: |
set -euo pipefail
selection="$(cargo nextest list --run-ignored ignored-only \
-p "$INTEROP_PACKAGE" --features "$INTEROP_FEATURES" \
-E "$INTEROP_FILTER" --message-format json \
| python3 -c 'import json,os,sys; d=json.load(sys.stdin); required=json.loads(os.environ["INTEROP_REQUIRED_TESTS"]); matched=[name for suite in d.get("rust-suites", {}).values() for name,test in suite.get("testcases", {}).items() if test.get("filter-match", {}).get("status") == "matches"]; missing=[test for test in required if not any(name.endswith("minio_generated_read_test::" + test) for name in matched)]; print(len(matched)); print(",".join(missing))')"
count="$(printf '%s\n' "$selection" | sed -n '1p')"
missing="$(printf '%s\n' "$selection" | sed -n '2p')"
echo "interop tests selected: ${count}"
if [ -n "${missing}" ]; then
echo "::error::Selector '${INTEROP_FILTER}' in package '${INTEROP_PACKAGE}' is missing required tests: ${missing}. The MinIO interop reader tests have moved or been renamed; fix the selector instead of running an incomplete matrix. Context: rustfs/backlog#1638."
exit 1
fi
- name: Run MinIO interop reader tests
run: |
cargo nextest run --run-ignored ignored-only --no-tests=fail \
-p "$INTEROP_PACKAGE" --features "$INTEROP_FEATURES" \
-E "$INTEROP_FILTER"
cargo nextest run --run-ignored ignored-only \
-p rustfs-ecstore --features rio-v2 \
-E 'binary(minio_generated_read_test)'
-11
View File
@@ -45,13 +45,6 @@
# docker-capable self-hosted `dind-sm-standard-2` label was the alternative but
# has fewer cores and reintroduces fleet-state risk for no reliability gain.
# DISABLED. This workflow is switched off in the repository's Actions settings
# (state: disabled_manually) and does not run on any trigger, including its cron
# and workflow_dispatch. That state lives in GitHub's UI and is invisible when
# reading this file, which has already misled at least one audit — hence this
# banner. Re-enabling is a UI action; anyone doing so should first check that the
# workflow still matches the current CI layout. See rustfs/backlog#1603.
#
name: mint
on:
@@ -125,8 +118,6 @@ jobs:
timeout-minutes: 120
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Enable buildx
uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # v3
@@ -272,8 +263,6 @@ jobs:
issues: write
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Open or update failure-tracking issue
uses: ./.github/actions/schedule-failure-issue
with:
-196
View File
@@ -1,196 +0,0 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
name: Nightly GNU Build
on:
schedule:
- cron: "0 0 * * *"
timezone: "Asia/Shanghai"
workflow_dispatch:
permissions:
contents: read
concurrency:
group: nightly-gnu-build-main-${{ github.event_name }}
cancel-in-progress: ${{ github.event_name == 'workflow_dispatch' }}
env:
CARGO_TERM_COLOR: always
RUST_BACKTRACE: 1
jobs:
build:
name: Build x86_64 GNU
runs-on: sm-standard-2
timeout-minutes: 150
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout main branch
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
ref: main
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
cache-shared-key: build-x86_64-unknown-linux-gnu
cache-save-if: 'false'
install-build-packaging-tools: 'false'
install-test-tools: 'false'
- name: Build RustFS
run: cargo build --release --locked --target x86_64-unknown-linux-gnu -p rustfs --bins
# Live-Vault lane for the rustfs-kms suite (rustfs/backlog#1774).
#
# RUSTFS_KMS_VAULT_TOKEN is the single switch that adds the Vault KV2 and
# Vault Transit backends to every for_each_backend spec in
# crates/kms/tests/behavior_*.rs (see crates/kms/AGENTS.md). rotate and
# versioning are advertised only by the Vault backends, so without this lane
# no CI run ever asserts the working half of behavior_rotation.rs — a
# rotation that silently dropped historical key versions would stay green.
# The same lane runs the dev-Vault #[ignore] tests and the two self-hosting
# live scripts (AppRole login, three-node Raft leader failover).
#
# GitHub-hosted ubuntu-latest, deliberately not the self-hosted sm-standard
# fleet: the HA failover script needs a working Docker daemon, and the
# self-hosted fleet is heterogeneous — a docker-dependent workflow has been
# burned by it before (see the banner in e2e-s3tests.yml, rustfs/backlog#1149).
kms-vault-lane:
name: KMS live Vault lane
runs-on: ubuntu-latest
timeout-minutes: 90
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
# Root token of the ephemeral loopback dev server. Not a secret: the
# server lives only for this job, listens on 127.0.0.1, and holds only
# keys the tests create. The literal value matters — the dev-Vault
# #[ignore] fixtures in crates/kms/src/backends/vault.rs hardcode it.
VAULT_LANE_TOKEN: dev-only-token
VAULT_LANE_ADDR: http://127.0.0.1:8200
# Keeps a runner-level proxy from swallowing the loopback dev-server
# traffic (see crates/kms/AGENTS.md). Actions env keys are
# case-insensitive, so only the uppercase form is set; reqwest reads
# either casing.
NO_PROXY: 127.0.0.1,localhost
steps:
- name: Checkout main branch
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
ref: main
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
# Dedicated key: rust-cache cannot tell runner images apart, so
# sharing a key with an sm-standard lane would let two different
# system images overwrite each other's artifacts (same reasoning as
# ci.yml's ci-uring lane). Saved from this nightly job itself so the
# next night starts warm.
cache-shared-key: kms-vault-lane
cache-save-if: 'true'
install-build-packaging-tools: 'false'
install-test-tools: 'false'
- name: Install Vault CLI
run: |
set -euo pipefail
wget -qO- https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list >/dev/null
sudo apt-get update -qq
sudo apt-get install -y -qq vault
vault version
- name: Start Vault dev server with KV2 and Transit engines
run: |
set -euo pipefail
nohup vault server -dev \
-dev-root-token-id="${VAULT_LANE_TOKEN}" \
-dev-listen-address=127.0.0.1:8200 >/tmp/vault-dev.log 2>&1 &
for _ in $(seq 1 60); do
if curl -fsS "${VAULT_LANE_ADDR}/v1/sys/health" >/dev/null 2>&1; then
break
fi
sleep 1
done
curl -fsS "${VAULT_LANE_ADDR}/v1/sys/health"
export VAULT_ADDR="${VAULT_LANE_ADDR}" VAULT_TOKEN="${VAULT_LANE_TOKEN}"
# Dev mode mounts KV v2 at secret/ by default; Transit is explicit.
# Prove both engines actually work rather than assuming the defaults.
vault secrets enable transit
vault kv put secret/rustfs-ci-lane-probe value=ok >/dev/null
vault kv get secret/rustfs-ci-lane-probe >/dev/null
vault write -f transit/keys/rustfs-ci-lane-probe >/dev/null
- name: Run rustfs-kms suite with the Vault lane on
env:
RUSTFS_KMS_VAULT_TOKEN: ${{ env.VAULT_LANE_TOKEN }}
RUSTFS_KMS_VAULT_ADDR: ${{ env.VAULT_LANE_ADDR }}
run: cargo test -p rustfs-kms --locked
- name: Run dev-Vault ignored tests
env:
RUSTFS_KMS_VAULT_TOKEN: ${{ env.VAULT_LANE_TOKEN }}
RUSTFS_KMS_VAULT_ADDR: ${{ env.VAULT_LANE_ADDR }}
# Filters select the dev-Vault-only #[ignore] tests. The AWS #[ignore]
# tests (backends::aws, service_manager) stay excluded — they need real
# AWS credentials and create billable keys. The AppRole and HA #[ignore]
# tests are excluded here because their own scripts below provision the
# Vault topology they need.
run: |
set -euo pipefail
cargo test -p rustfs-kms --locked --lib backends::contract_tests -- --ignored
cargo test -p rustfs-kms --locked --lib backends::vault -- --ignored
cargo test -p rustfs-kms --locked --test vault_fault_injection -- --ignored
- name: Run AppRole live checks (self-hosting ephemeral Vault)
run: bash scripts/test/vault_approle_kms_live.sh
- name: Show Vault dev server log on failure
if: failure()
run: tail -n 200 /tmp/vault-dev.log || true
# Three-node Raft leader failover (crates/kms/tests/vault_ha_failover_live.rs,
# first validated by rustfs/rustfs#5653). Its own job so an election-timing
# flake cannot mask the main lane's verdict, and vice versa. The script
# provisions and tears down its own Docker cluster.
kms-vault-ha-failover:
name: KMS Vault HA failover lane
runs-on: ubuntu-latest
timeout-minutes: 60
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
NO_PROXY: 127.0.0.1,localhost
steps:
- name: Checkout main branch
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
ref: main
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
cache-shared-key: kms-vault-lane
cache-save-if: 'false'
install-build-packaging-tools: 'false'
install-test-tools: 'false'
- name: Run HA leader failover live checks (three-node Raft cluster in Docker)
run: bash scripts/test/vault_ha_kms_live.sh
+2 -9
View File
@@ -19,12 +19,9 @@ on:
schedule:
- cron: '0 5 * * 0' # Weekly on Sunday 05:00 UTC (staggered after the midnight ci/build crons)
# GITHUB_TOKEN only needs to read the repository here: the branch push and the
# pull request are both created by update-flake-lock using the
# FLAKE_UPDATE_TOKEN PAT below, not by this token. Leaving write on it hands a
# repo-write credential to an unattended weekly job that does not use it.
permissions:
contents: read
contents: write
pull-requests: write
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
@@ -40,10 +37,6 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
# persist-credentials-exempt: update-flake-lock pushes the branch and opens
# the PR. It passes FLAKE_UPDATE_TOKEN to create-pull-request itself rather
# than reusing .git/config, but that is unverified — exempt until a
# workflow_dispatch run confirms it (rustfs/backlog#1602).
- name: Install Nix
uses: DeterminateSystems/determinate-nix-action@629b284231c2a82554b724e357e47fc6020833c8 # v3
-10
View File
@@ -12,13 +12,6 @@
# See the License for the specific language governing permissions and
# limitations under the License.
# DISABLED. This workflow is switched off in the repository's Actions settings
# (state: disabled_manually) and does not run on any trigger, including its cron
# and workflow_dispatch. That state lives in GitHub's UI and is invisible when
# reading this file, which has already misled at least one audit — hence this
# banner. Re-enabling is a UI action; anyone doing so should first check that the
# workflow still matches the current CI layout. See rustfs/backlog#1603.
#
name: Nix CI
on:
@@ -53,7 +46,6 @@ jobs:
name: Cancel Closed PR Runs
if: github.event_name == 'pull_request' && github.event.action == 'closed'
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- name: Explain cancellation run
run: echo "PR closed; this run only cancels older runs in the same concurrency group."
@@ -71,8 +63,6 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Install Nix
uses: DeterminateSystems/determinate-nix-action@4eea0b33e3d1f02ecfe37cf16e7204c424009606 # v3.21.0
-477
View File
@@ -1,477 +0,0 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Package Workflow - Build DEB/RPM packages
#
# This workflow builds DEB and RPM packages from pre-built Linux binaries
# and uploads them to Cloudflare R2.
#
# Trigger:
# - release published: automatically package when a GitHub release is published
# - workflow_dispatch: manual trigger with optional tag/run_id
#
# Flow:
# 1. Find the Build workflow run for the release tag
# 2. Download Linux binaries (x86_64-gnu, aarch64-gnu) from build artifacts
# 3. Build DEB packages for amd64 and arm64
# 4. Build RPM packages for x86_64 and aarch64
# 5. Upload all packages to Cloudflare R2
name: Package DEB/RPM
permissions:
contents: read
actions: read
on:
release:
types: [ published ]
workflow_dispatch:
inputs:
tag:
description: "Release tag to package (e.g. 1.0.0-beta.12). Leave empty for latest main build."
required: false
type: string
build_run_id:
description: "Build workflow run ID (overrides tag lookup)"
required: false
type: string
concurrency:
group: ${{ github.workflow }}-${{ github.event.release.tag_name || github.event.inputs.tag || github.run_id }}
cancel-in-progress: true
jobs:
# Resolve which build run to use and extract version info
resolve:
name: Resolve Build
runs-on: ubuntu-latest
timeout-minutes: 10
outputs:
version: ${{ steps.resolve.outputs.version }}
build_type: ${{ steps.resolve.outputs.build_type }}
build_run_id: ${{ steps.resolve.outputs.build_run_id }}
tag: ${{ steps.resolve.outputs.tag }}
steps:
- name: Resolve build run
id: resolve
shell: bash
env:
GH_TOKEN: ${{ github.token }}
INPUT_TAG: ${{ github.event.inputs.tag }}
INPUT_RUN_ID: ${{ github.event.inputs.build_run_id }}
run: |
set -euo pipefail
# Determine tag
if [[ "${{ github.event_name }}" == "release" ]]; then
TAG="${{ github.event.release.tag_name }}"
elif [[ -n "$INPUT_TAG" ]]; then
TAG="$INPUT_TAG"
else
TAG=""
fi
echo "Tag: ${TAG:-<none>}"
# Determine build run ID
BUILD_RUN_ID=""
if [[ -n "$INPUT_RUN_ID" ]]; then
# Explicit run ID takes priority
BUILD_RUN_ID="$INPUT_RUN_ID"
echo "Using explicit build run ID: $BUILD_RUN_ID"
elif [[ -n "$TAG" ]]; then
# Find the build run that produced this tag
echo "Looking for build run for tag: $TAG"
BUILD_RUN_ID=$(gh api \
"repos/${{ github.repository }}/actions/workflows/build.yml/runs?branch=${TAG}&status=success&per_page=1" \
--jq '.workflow_runs[0].id' 2>/dev/null || echo "")
if [[ -z "$BUILD_RUN_ID" || "$BUILD_RUN_ID" == "null" ]]; then
# Tag might not be a branch; try event=push with head_branch matching
BUILD_RUN_ID=$(gh api \
"repos/${{ github.repository }}/actions/workflows/build.yml/runs?event=push&status=success&per_page=100" \
--jq ".workflow_runs[] | select(.head_branch == \"$TAG\") | .id" 2>/dev/null | head -1 || echo "")
fi
if [[ -z "$BUILD_RUN_ID" || "$BUILD_RUN_ID" == "null" ]]; then
echo "❌ No successful build run found for tag: $TAG"
exit 1
fi
echo "Found build run: $BUILD_RUN_ID"
else
# No tag — latest successful main build
echo "No tag specified, looking for latest main build"
BUILD_RUN_ID=$(gh api \
"repos/${{ github.repository }}/actions/workflows/build.yml/runs?branch=main&status=success&per_page=1" \
--jq '.workflow_runs[0].id' 2>/dev/null || echo "")
if [[ -z "$BUILD_RUN_ID" || "$BUILD_RUN_ID" == "null" ]]; then
echo "❌ No successful main build found"
exit 1
fi
echo "Latest main build: $BUILD_RUN_ID"
fi
# Determine version and build type
if [[ -n "$TAG" ]]; then
VERSION="$TAG"
if [[ "$TAG" == *"-preview"* ]]; then
BUILD_TYPE="preview"
elif [[ "$TAG" == *"alpha"* || "$TAG" == *"beta"* || "$TAG" == *"rc"* ]]; then
BUILD_TYPE="prerelease"
else
BUILD_TYPE="release"
fi
else
SHORT_SHA=$(gh api "repos/${{ github.repository }}/actions/runs/${BUILD_RUN_ID}" \
--jq '.head_sha' 2>/dev/null | head -c 7)
VERSION="dev-${SHORT_SHA}"
BUILD_TYPE="development"
fi
{
echo "version=$VERSION"
echo "build_type=$BUILD_TYPE"
echo "build_run_id=$BUILD_RUN_ID"
echo "tag=${TAG}"
} >> "$GITHUB_OUTPUT"
echo "📊 Resolved:"
echo " Version: $VERSION"
echo " Build type: $BUILD_TYPE"
echo " Build run ID: $BUILD_RUN_ID"
# Build DEB and RPM packages for each architecture
package:
name: Package (${{ matrix.arch }})
needs: resolve
runs-on: ubuntu-latest
timeout-minutes: 30
strategy:
fail-fast: false
matrix:
include:
- arch: x86_64
deb_arch: amd64
rpm_arch: x86_64
artifact_name: "rustfs-linux-x86_64-gnu"
- arch: aarch64
deb_arch: arm64
rpm_arch: aarch64
artifact_name: "rustfs-linux-aarch64-gnu"
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Download binary artifact from build run
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
with:
pattern: ${{ matrix.artifact_name }}*
path: ./binary-artifact
run-id: ${{ needs.resolve.outputs.build_run_id }}
github-token: ${{ github.token }}
merge-multiple: true
- name: Extract binary
id: binary
shell: bash
run: |
set -euo pipefail
ZIP_FILE=$(find ./binary-artifact -name "*.zip" -type f | head -1)
if [[ -z "$ZIP_FILE" ]]; then
echo "❌ No binary artifact found"
ls -la ./binary-artifact/ || true
exit 1
fi
echo "Found artifact: $ZIP_FILE"
mkdir -p ./bin
unzip -o "$ZIP_FILE" -d ./bin
if [[ ! -f ./bin/rustfs ]]; then
echo "❌ rustfs binary not found in archive"
exit 1
fi
chmod +x ./bin/rustfs
ls -lh ./bin/rustfs
echo "✅ Binary extracted"
- name: Build DEB package
id: deb
shell: bash
run: |
set -euo pipefail
VERSION="${{ needs.resolve.outputs.version }}"
DEB_ARCH="${{ matrix.deb_arch }}"
# DEB version: replace - with ~ (1.0.0-beta.12 -> 1.0.0~beta.12)
# Use a variable for ~ to prevent tilde expansion by bash
TILDE='~'
DEB_VERSION="${VERSION/-/$TILDE}"
PKG_DIR="rustfs_${DEB_VERSION}_${DEB_ARCH}"
echo "Building DEB: ${PKG_DIR}.deb"
mkdir -p "${PKG_DIR}/DEBIAN"
mkdir -p "${PKG_DIR}/usr/bin"
mkdir -p "${PKG_DIR}/etc/default"
mkdir -p "${PKG_DIR}/lib/systemd/system"
mkdir -p "${PKG_DIR}/usr/share/doc/rustfs"
cp ./bin/rustfs "${PKG_DIR}/usr/bin/"
chmod 755 "${PKG_DIR}/usr/bin/rustfs"
cp deploy/build/rustfs.service "${PKG_DIR}/lib/systemd/system/"
cat > "${PKG_DIR}/etc/default/rustfs" << 'ENVEOF'
# RustFS Environment Configuration
# See https://rustfs.com/docs/ for more information
# RUSTFS_VOLUMES=""
# RUSTFS_ROOT_USER=""
# RUSTFS_ROOT_PASSWORD=""
ENVEOF
cat > "${PKG_DIR}/DEBIAN/control" << EOF
Package: rustfs
Version: ${DEB_VERSION}
Section: utils
Priority: optional
Architecture: ${DEB_ARCH}
Depends: libc6 (>= 2.31)
Maintainer: RustFS Team <support@rustfs.com>
Description: High-performance distributed object storage
RustFS is a high-performance distributed object storage software
built using Rust. It is compatible with MinIO and S3 API.
Homepage: https://rustfs.com
EOF
cat > "${PKG_DIR}/DEBIAN/postinst" << 'POSTINST'
#!/bin/bash
set -e
if ! getent passwd rustfs > /dev/null 2>&1; then
useradd -r -s /bin/false -d /opt/rustfs rustfs
fi
mkdir -p /opt/rustfs /data/rustfs /var/log/rustfs
chown rustfs:rustfs /opt/rustfs /data/rustfs /var/log/rustfs
if [ -d /run/systemd/system ]; then
systemctl daemon-reload
fi
echo "RustFS installed. Configure /etc/default/rustfs then: systemctl start rustfs"
POSTINST
chmod 755 "${PKG_DIR}/DEBIAN/postinst"
cat > "${PKG_DIR}/DEBIAN/prerm" << 'PRERM'
#!/bin/bash
set -e
if [ -d /run/systemd/system ] && systemctl is-active --quiet rustfs; then
systemctl stop rustfs
fi
PRERM
chmod 755 "${PKG_DIR}/DEBIAN/prerm"
cat > "${PKG_DIR}/DEBIAN/postrm" << 'POSTRM'
#!/bin/bash
set -e
if [ -d /run/systemd/system ]; then
systemctl daemon-reload
fi
POSTRM
chmod 755 "${PKG_DIR}/DEBIAN/postrm"
cp LICENSE "${PKG_DIR}/usr/share/doc/rustfs/"
cp README.md "${PKG_DIR}/usr/share/doc/rustfs/"
fakeroot dpkg-deb --build "${PKG_DIR}"
DEB_FILE="${PKG_DIR}.deb"
ls -lh "$DEB_FILE"
echo "deb_file=$DEB_FILE" >> "$GITHUB_OUTPUT"
echo "✅ DEB built: $DEB_FILE"
- name: Build RPM package
id: rpm
shell: bash
run: |
set -euo pipefail
VERSION="${{ needs.resolve.outputs.version }}"
RPM_ARCH="${{ matrix.rpm_arch }}"
echo "Building RPM for ${RPM_ARCH}"
sudo apt-get update && sudo apt-get install -y ruby ruby-dev build-essential
sudo gem install fpm
# Create config file for fpm (DEB build creates it in its package dir structure,
# but fpm needs the file to exist before packaging)
mkdir -p ./tmp-pkg/etc/default
cat > ./tmp-pkg/etc/default/rustfs << 'ENVEOF'
# RustFS Environment Configuration
# See https://rustfs.com/docs/ for more information
# RUSTFS_VOLUMES=""
# RUSTFS_ROOT_USER=""
# RUSTFS_ROOT_PASSWORD=""
ENVEOF
fpm -s dir -t rpm \
--name rustfs \
--version "$VERSION" \
--architecture "$RPM_ARCH" \
--depends "glibc >= 2.31" \
--maintainer "RustFS Team <support@rustfs.com>" \
--description "High-performance distributed object storage" \
--url "https://rustfs.com" \
--license "Apache-2.0" \
--after-install <(cat <<'POSTINST'
#!/bin/bash
set -e
if ! getent passwd rustfs > /dev/null 2>&1; then
useradd -r -s /bin/false -d /opt/rustfs rustfs
fi
mkdir -p /opt/rustfs /data/rustfs /var/log/rustfs
chown rustfs:rustfs /opt/rustfs /data/rustfs /var/log/rustfs
if [ -d /run/systemd/system ]; then
systemctl daemon-reload
fi
POSTINST
) \
--before-remove <(cat <<'PRERM'
#!/bin/bash
set -e
if [ -d /run/systemd/system ] && systemctl is-active --quiet rustfs; then
systemctl stop rustfs
fi
PRERM
) \
--after-remove <(cat <<'POSTRM'
#!/bin/bash
set -e
if [ -d /run/systemd/system ]; then
systemctl daemon-reload
fi
POSTRM
) \
--config-files /etc/default/rustfs \
./bin/rustfs=/usr/bin/rustfs \
./tmp-pkg/etc/default/rustfs=/etc/default/rustfs \
deploy/build/rustfs.service=/lib/systemd/system/rustfs.service \
LICENSE=/usr/share/doc/rustfs/LICENSE \
README.md=/usr/share/doc/rustfs/README.md
RPM_FILE=$(ls -1 rustfs-*.rpm 2>/dev/null | head -1)
if [[ -z "$RPM_FILE" ]]; then
echo "❌ RPM build failed"
exit 1
fi
ls -lh "$RPM_FILE"
echo "rpm_file=$RPM_FILE" >> "$GITHUB_OUTPUT"
echo "✅ RPM built: $RPM_FILE"
- name: Upload packages to artifacts
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
with:
name: packages-${{ matrix.arch }}
path: |
*.deb
*.rpm
retention-days: 30
- name: Upload packages to Cloudflare R2
if: env.R2_ACCESS_KEY_ID != ''
env:
R2_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
R2_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
R2_ENDPOINT: ${{ secrets.R2_ENDPOINT }}
R2_BUCKET: ${{ secrets.R2_BUCKET }}
AWS_EC2_METADATA_DISABLED: true
shell: bash
run: |
set -euo pipefail
if [[ -z "$R2_ACCESS_KEY_ID" || -z "$R2_SECRET_ACCESS_KEY" || -z "$R2_ENDPOINT" || -z "$R2_BUCKET" ]]; then
echo "⚠️ R2 credentials missing, skipping upload"
exit 0
fi
if ! command -v aws >/dev/null 2>&1; then
sudo apt-get update && sudo apt-get install -y awscli
fi
export AWS_ACCESS_KEY_ID="$R2_ACCESS_KEY_ID"
export AWS_SECRET_ACCESS_KEY="$R2_SECRET_ACCESS_KEY"
export AWS_DEFAULT_REGION="auto"
BUILD_TYPE="${{ needs.resolve.outputs.build_type }}"
if [[ "$BUILD_TYPE" == "development" ]]; then
R2_PREFIX="artifacts/rustfs/packages/dev"
else
R2_PREFIX="artifacts/rustfs/packages/release"
fi
R2_PATH="s3://${R2_BUCKET}/${R2_PREFIX}/"
echo "📤 Uploading to $R2_PATH"
DEB_FILE="${{ steps.deb.outputs.deb_file }}"
RPM_FILE="${{ steps.rpm.outputs.rpm_file }}"
for f in "$DEB_FILE" "$RPM_FILE"; do
if [[ -n "$f" && -f "$f" ]]; then
echo "Uploading: $f"
aws s3 cp "$f" "$R2_PATH" --endpoint-url "$R2_ENDPOINT" --only-show-errors
fi
done
echo "✅ Upload complete"
# Also upload as latest for release/prerelease
if [[ "$BUILD_TYPE" == "release" || "$BUILD_TYPE" == "prerelease" ]]; then
LATEST_PATH="s3://${R2_BUCKET}/artifacts/rustfs/packages/latest/"
for f in "$DEB_FILE" "$RPM_FILE"; do
if [[ -n "$f" && -f "$f" ]]; then
echo "Uploading latest: $(basename "$f")"
aws s3 cp "$f" "$LATEST_PATH" --endpoint-url "$R2_ENDPOINT" --only-show-errors
fi
done
echo "✅ Latest packages updated"
fi
# Summary
summary:
name: Summary
needs: [ resolve, package ]
if: always()
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
- name: Print summary
shell: bash
run: |
echo "## 📦 Package Summary" >> "$GITHUB_STEP_SUMMARY"
echo "" >> "$GITHUB_STEP_SUMMARY"
echo "| Item | Value |" >> "$GITHUB_STEP_SUMMARY"
echo "|------|-------|" >> "$GITHUB_STEP_SUMMARY"
echo "| Version | \`${{ needs.resolve.outputs.version }}\` |" >> "$GITHUB_STEP_SUMMARY"
echo "| Build Type | ${{ needs.resolve.outputs.build_type }} |" >> "$GITHUB_STEP_SUMMARY"
echo "| Build Run | #${{ needs.resolve.outputs.build_run_id }} |" >> "$GITHUB_STEP_SUMMARY"
echo "| Package Status | ${{ needs.package.result }} |" >> "$GITHUB_STEP_SUMMARY"
+72 -85
View File
@@ -17,18 +17,11 @@
# Two entry points, honestly scoped:
# * schedule (nightly, on main): post-merge detection — catches a regression
# within 24h of landing, not before merge.
# * workflow_dispatch: an explicitly selected trusted ref.
# The dispatch input can run the gate with --allow-regression so a deliberate
# correctness cost (e.g. the #4221 fsync durability fix) is recorded, not
# blocked (rustfs/backlog#935 correction 1).
# * pull_request labeled `perf-ab`: opt-in pre-merge gate for a specific PR.
# The `perf-deliberate-tradeoff` label runs the gate with --allow-regression so
# a deliberate correctness cost (e.g. the #4221 fsync durability fix) is
# recorded but does not block (rustfs/backlog#935 correction 1).
# DISABLED. This workflow is switched off in the repository's Actions settings
# (state: disabled_manually) and does not run on any trigger, including its cron
# and workflow_dispatch. That state lives in GitHub's UI and is invisible when
# reading this file, which has already misled at least one audit — hence this
# banner. Re-enabling is a UI action; anyone doing so should first check that the
# workflow still matches the current CI layout. See rustfs/backlog#1603.
#
name: Performance A/B
on:
@@ -46,6 +39,8 @@ on:
required: false
default: false
type: boolean
pull_request:
types: [labeled, synchronize, reopened]
push:
# Every main commit pre-builds and caches its release binary (perf-3) so the
# nightly A/B restores a ready baseline instead of paying the double build.
@@ -53,6 +48,14 @@ on:
permissions:
contents: read
pull-requests: write
# Per-PR: a new push cancels the previous (up to 90-minute) A/B run instead of
# stacking them. Nightly schedule and manual dispatch get a unique group and
# always run to completion.
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
env:
CARGO_TERM_COLOR: always
@@ -60,8 +63,8 @@ env:
jobs:
# perf-3: on every push to main, build the release binary once and cache it
# keyed by commit SHA (rustfs-baseline-<sha>). The warp-ab measurements
# restore this instead of paying the ~32min-per-side source
# keyed by commit SHA (rustfs-baseline-<sha>). The nightly A/B (and, later, the
# perf-7 PR gate) restore this instead of paying the ~32min-per-side source
# build. That double build is what pushed the expanded 24-cell nightly past its
# ceiling — 2026-07-11..07-14 all cancelled on the 120min timeout. Incremental
# builds off the shared cargo cache keep each push cheap, and building on the
@@ -89,8 +92,6 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Setup Rust environment
uses: ./.github/actions/setup
@@ -98,6 +99,7 @@ jobs:
rust-version: stable
cache-shared-key: warp-ab-${{ hashFiles('**/Cargo.lock') }}
cache-save-if: ${{ github.ref == 'refs/heads/main' }}
github-token: ${{ secrets.GITHUB_TOKEN }}
- name: Build release rustfs
run: cargo build --release --bin rustfs
@@ -116,11 +118,17 @@ jobs:
warp-ab:
name: Warp A/B budget gate
# Always run on schedule / manual dispatch. Never on push — that event only
# feeds build-baseline-cache above.
# Always run on schedule / manual dispatch. Opt-in on PRs: only when the
# `perf-ab` label is present, and for `labeled` events only when the label
# being added is `perf-ab` itself (adding an unrelated label to an opted-in
# PR must not re-run the gate). Never on push — that event only feeds
# build-baseline-cache above.
if: >-
github.event_name == 'schedule' ||
github.event_name == 'workflow_dispatch'
github.event_name == 'workflow_dispatch' ||
(github.event_name == 'pull_request' &&
contains(github.event.pull_request.labels.*.name, 'perf-ab') &&
(github.event.action != 'labeled' || github.event.label.name == 'perf-ab'))
runs-on: sm-standard-2
# With perf-3's cached baseline binary the common (cache-hit) nightly is
# measurement-only and finishes well under 50min. This ceiling stays
@@ -134,7 +142,6 @@ jobs:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
fetch-depth: 0 # baseline is built from origin/main
- name: Setup Rust environment
@@ -143,6 +150,7 @@ jobs:
rust-version: stable
cache-shared-key: warp-ab-${{ hashFiles('**/Cargo.lock') }}
cache-save-if: ${{ github.ref == 'refs/heads/main' }}
github-token: ${{ secrets.GITHUB_TOKEN }}
- name: Install warp
run: |
@@ -154,11 +162,13 @@ jobs:
- name: Decide exemption
id: exempt
env:
INPUT_ALLOW_REGRESSION: ${{ github.event.inputs.allow_regression }}
run: |
allow="false"
if [[ "$INPUT_ALLOW_REGRESSION" == "true" ]]; then
if [[ "${{ github.event_name }}" == "pull_request" ]] \
&& ${{ contains(github.event.pull_request.labels.*.name, 'perf-deliberate-tradeoff') }}; then
allow="true"
fi
if [[ "${{ github.event.inputs.allow_regression }}" == "true" ]]; then
allow="true"
fi
echo "allow_regression=$allow" >> "$GITHUB_OUTPUT"
@@ -205,34 +215,8 @@ jobs:
cp target/release/rustfs baseline-bin/rustfs
echo "built=true" >> "$GITHUB_OUTPUT"
- name: Build baseline on cache miss (different candidate)
id: baseline_build
if: >-
steps.baseline_cache.outputs.cache-hit != 'true' &&
steps.commits.outputs.baseline_sha != steps.commits.outputs.candidate_sha
run: |
set -euo pipefail
baseline_root="$RUNNER_TEMP/rustfs-baseline-${{ github.run_id }}"
baseline_target="$RUNNER_TEMP/rustfs-baseline-target-${{ github.run_id }}"
git worktree add --detach "$baseline_root" "${{ steps.commits.outputs.baseline_sha }}"
cargo build --release --manifest-path "$baseline_root/Cargo.toml" --bin rustfs --target-dir "$baseline_target"
mkdir -p baseline-bin
cp "$baseline_target/release/rustfs" baseline-bin/rustfs
git worktree remove --force "$baseline_root"
echo "built=true" >> "$GITHUB_OUTPUT"
- name: Build candidate binary
id: candidate_build
if: steps.commits.outputs.baseline_sha != steps.commits.outputs.candidate_sha
run: |
set -euo pipefail
cargo build --release --bin rustfs
mkdir -p candidate-bin
cp target/release/rustfs candidate-bin/rustfs
echo "built=true" >> "$GITHUB_OUTPUT"
- name: Save self-healed baseline to cache
if: steps.selfheal.outputs.built == 'true' || steps.baseline_build.outputs.built == 'true'
if: steps.selfheal.outputs.built == 'true'
uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6
with:
path: baseline-bin/rustfs
@@ -240,66 +224,62 @@ jobs:
- name: Run warp A/B and gate
id: ab
env:
INPUT_DURATION: ${{ github.event.inputs.duration }}
run: |
set -euo pipefail
# The formal runner executes A1 baseline -> B1 candidate -> B2 candidate
# -> A2 baseline for each workload and drive-sync cell. It requires three
# rounds per leg to emit tail latency and error-rate evidence.
# Budget note: with perf-3's cached baseline the nightly does no source
# build on a cache hit, so the wall-clock is dominated by the short warp
# matrix — duration/rounds/cooldown are kept small to fit all 24 cells
# (6 workloads x 2 phases x 2 drive-sync) rather than dropping cells.
# --health-timeout 180 outlasts the server's own 120s startup-readiness
# budget, which the rig's previous 60s health poll undershot (the first
# two nightly failures). perf-6 recalibrates these once the noise study
# lands.
duration="${INPUT_DURATION:-12s}"
duration="${{ github.event.inputs.duration || '12s' }}"
baseline_sha="${{ steps.commits.outputs.baseline_sha }}"
candidate_sha="${{ steps.commits.outputs.candidate_sha }}"
baseline_hit="${{ steps.baseline_cache.outputs.cache-hit }}"
selfheal_built="${{ steps.selfheal.outputs.built }}"
baseline_built="${{ steps.baseline_build.outputs.built }}"
candidate_built="${{ steps.candidate_build.outputs.built }}"
args=(--duration "$duration" --rounds 3 --cooldown 5 --health-timeout 180 --baseline-revision "$baseline_sha" --candidate-revision "$candidate_sha")
args=(--duration "$duration" --rounds 2 --cooldown 5 --health-timeout 180)
if [[ "$baseline_hit" == "true" || "$selfheal_built" == "true" || "$baseline_built" == "true" ]]; then
if [[ "$baseline_hit" == "true" || "$selfheal_built" == "true" ]]; then
chmod +x baseline-bin/rustfs
base_bin="$PWD/baseline-bin/rustfs"
args+=(--baseline-bin "$base_bin")
if [[ "$baseline_hit" == "true" ]]; then
base_src="actions-cache (rustfs-baseline-$baseline_sha)"
elif [[ "$selfheal_built" == "true" ]]; then
base_src="source build (cache self-heal, saved as rustfs-baseline-$baseline_sha)"
else
base_src="isolated origin/main source build (saved as rustfs-baseline-$baseline_sha)"
base_src="source build (cache self-heal, saved as rustfs-baseline-$baseline_sha)"
fi
if [[ "$candidate_sha" == "$baseline_sha" ]]; then
# Nightly on main: the candidate is the same commit as the baseline,
# so reuse the one binary for both phases and skip all builds.
args+=(--candidate-bin "$base_bin")
args+=(--candidate-bin "$base_bin" --skip-build)
cand_src="same binary as baseline (same commit)"
elif [[ "$candidate_built" == "true" ]]; then
chmod +x candidate-bin/rustfs
args+=(--candidate-bin "$PWD/candidate-bin/rustfs")
cand_src="source build of the checked-out ref"
else
echo "::error::candidate binary was not built" >&2
exit 2
cand_src="source build of the checked-out ref"
fi
else
echo "::error::baseline binary was not restored or built" >&2
exit 2
# Cache miss with candidate != baseline (opt-in PR gate only): fall
# back to the source double-build. With the post-#4806 LTO profile
# this will overrun the job budget and alert; rerun once the push
# cache build for origin/main has completed, or wait for perf-7's
# merge-base caching.
args+=(--baseline-ref origin/main)
base_src="source build of origin/main (cache miss)"
cand_src="source build of the checked-out ref"
fi
echo "baseline binary: $base_src"
echo "candidate binary: $cand_src"
args+=(--provenance-note "baseline commit: $baseline_sha - $base_src")
args+=(--provenance-note "candidate commit: $candidate_sha - $cand_src")
if [[ "${{ steps.exempt.outputs.allow_regression }}" == "true" ]]; then
args+=(--allow-regression --exemption-reason "workflow dispatch override")
args+=(--allow-regression --exemption-reason "labeled perf-deliberate-tradeoff / dispatch override")
fi
# Do not let a gate FAIL abort the job here; capture status and surface
# it after the step summary is written.
# it after the PR comment is posted.
set +e
bash scripts/run_hotpath_warp_abba.sh "${args[@]}"
bash scripts/run_hotpath_warp_ab.sh "${args[@]}"
echo "status=$?" >> "$GITHUB_OUTPUT"
set -e
# Locate the newest run dir + gate.md for the summary/comment/artifact
@@ -307,10 +287,10 @@ jobs:
# holds server-logs/ for diagnosis.
# Run dirs are UTC-timestamp names (no special chars); ls is safe here.
# shellcheck disable=SC2012
run_dir="$(ls -td target/hotpath-abba/*/ 2>/dev/null | head -n1 || true)"
run_dir="$(ls -td target/hotpath-ab/*/ 2>/dev/null | head -n1 || true)"
echo "run_dir=${run_dir%/}" >> "$GITHUB_OUTPUT"
# shellcheck disable=SC2012
gate_md="$(ls -t target/hotpath-abba/*/candidate_gate.md 2>/dev/null | head -n1 || true)"
gate_md="$(ls -t target/hotpath-ab/*/gate.md 2>/dev/null | head -n1 || true)"
echo "gate_md=$gate_md" >> "$GITHUB_OUTPUT"
- name: Upload A/B results
@@ -318,10 +298,10 @@ jobs:
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
with:
name: hotpath-warp-ab-${{ github.run_number }}
# Includes per-cell median_summary.csv / baseline_compare.csv, both gates,
# Includes per-cell median_summary.csv / baseline_compare.csv, gate.md,
# and server-logs/ (rustfs.log + startup env per phase) so a failed run
# is diagnosable. Short retention: this is churny nightly debug data.
path: target/hotpath-abba/
path: target/hotpath-ab/
if-no-files-found: warn
retention-days: 14
@@ -362,6 +342,13 @@ jobs:
fi
} >> "$GITHUB_STEP_SUMMARY"
- name: Comment gate result on PR
if: always() && github.event_name == 'pull_request' && steps.ab.outputs.gate_md != ''
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
gh pr comment "${{ github.event.pull_request.number }}" --body-file "${{ steps.ab.outputs.gate_md }}"
# Scheduled failure alerting is handled by the alert-on-failure job below
# (perf-2 consuming ci-8's schedule-failure-issue composite action).
@@ -370,7 +357,7 @@ jobs:
run: |
status="${{ steps.ab.outputs.status }}"
if [[ "$status" != "0" ]]; then
echo "::error::warp A/B budget gate failed (exit $status). See the step summary / gate.md artifact." >&2
echo "::error::warp A/B budget gate failed (exit $status). See the step summary / PR comment / gate.md artifact." >&2
exit "$status"
fi
echo "warp A/B budget gate passed."
@@ -380,12 +367,14 @@ jobs:
needs: [warp-ab]
# `always()` is required: without it this job is skipped when a needed
# job fails. Alerts only for scheduled (nightly) runs (backlog#1149
# ci-8); manual dispatch failures are already watched by a human.
# ci-8); PR and manual dispatch failures are already watched by a human.
# `cancelled` is included alongside `failure` on purpose: a job that hits
# timeout-minutes ends as `cancelled`, and the 2026-07-11..07-14 nightly
# timeouts went silent precisely because the guard was failure-only. The
# composite action already reports cancelled/timed-out jobs in the issue
# body.
# body. (Scheduled runs get a unique concurrency group with
# cancel-in-progress off, so a cancellation here means a timeout/manual
# abort, never a superseding run.)
if: >-
always() && github.event_name == 'schedule' &&
(contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled'))
@@ -396,8 +385,6 @@ jobs:
issues: write
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Open or update failure-tracking issue
uses: ./.github/actions/schedule-failure-issue
with:
-81
View File
@@ -1,81 +0,0 @@
# Copyright 2026 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Asserts that the self-hosted runners are still ephemeral — one job per pod.
#
# This repository is public and its pull_request jobs run on those runners,
# executing the PR's own build.rs, proc-macros and tests. The only thing keeping
# that code from reaching a later job is that each ARC pod handles exactly one
# job and is then destroyed. That guarantee lives in the ARC scale-set
# configuration, outside this repository, where it can be changed without any PR
# — so it is asserted here from the outside, against real run data, instead of
# being assumed.
#
# Monthly rather than per-PR: the property changes only when someone
# reconfigures the scale set, and the check costs a few dozen API calls.
# See docs/ci/runners.md and rustfs/backlog#1602.
name: Runner Hygiene
on:
schedule:
- cron: "0 6 1 * *" # Monthly, 1st at 06:00 UTC (after the daily audit cron)
workflow_dispatch:
permissions:
contents: read
concurrency:
group: runner-hygiene
cancel-in-progress: false
jobs:
check-ephemerality:
name: Check runner ephemerality
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
# Exit 2 (inconclusive / broken) is deliberately not a pass: a window
# where every sm-* job was still queued would otherwise look identical to
# a clean bill of health.
- name: Assert one job per self-hosted runner
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: ./scripts/ci/check_runner_ephemerality.sh 40
alert-on-failure:
name: Alert on scheduled failure
needs: [check-ephemerality]
# Same ci-8 mechanism as coverage.yml, audit.yml and the nightly lanes:
# scheduled runs file a tracking issue, manual dispatch stays quiet so
# debugging never produces a spurious alert.
if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
runs-on: ubuntu-latest
timeout-minutes: 10
permissions:
contents: read
issues: write
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Open or update failure-tracking issue
uses: ./.github/actions/schedule-failure-issue
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
@@ -24,13 +24,6 @@
# The run itself is expected to end red (the forced failure); only the
# alert-on-failure job result matters.
# DISABLED. This workflow is switched off in the repository's Actions settings
# (state: disabled_manually) and does not run on any trigger, including its cron
# and workflow_dispatch. That state lives in GitHub's UI and is invisible when
# reading this file, which has already misled at least one audit — hence this
# banner. Re-enabling is a UI action; anyone doing so should first check that the
# workflow still matches the current CI layout. See rustfs/backlog#1603.
#
name: Schedule Failure Alert Drill
on:
@@ -63,8 +56,6 @@ jobs:
issues: write
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Open or update failure-tracking issue
uses: ./.github/actions/schedule-failure-issue
with:
-8
View File
@@ -12,13 +12,6 @@
# See the License for the specific language governing permissions and
# limitations under the License.
# DISABLED. This workflow is switched off in the repository's Actions settings
# (state: disabled_manually) and does not run on any trigger, including its cron
# and workflow_dispatch. That state lives in GitHub's UI and is invisible when
# reading this file, which has already misled at least one audit — hence this
# banner. Re-enabling is a UI action; anyone doing so should first check that the
# workflow still matches the current CI layout. See rustfs/backlog#1603.
#
name: "Mark stale issues"
on:
schedule:
@@ -27,7 +20,6 @@ on:
jobs:
stale:
runs-on: ubuntu-latest
timeout-minutes: 30
steps:
- uses: actions/stale@5bef64f19d7facfb25b37b414482c7164d639639 # v9
with:
-1
View File
@@ -15,7 +15,6 @@ concurrency:
jobs:
update:
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: overtrue/repo-visuals-action@72f34d24769ff5d341956da2f23952594ef2f1e2 # v1.3.0
with:
-90
View File
@@ -1,90 +0,0 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
name: Windows Filesystem Tests
on:
push:
branches: [ main ]
paths:
- "crates/ecstore/src/disk/**"
- "crates/ecstore/src/store/init_format.rs"
- "crates/ecstore/Cargo.toml"
- "Cargo.toml"
- "Cargo.lock"
- ".github/actions/setup/**"
- ".github/workflows/windows-filesystem.yml"
pull_request:
branches: [ main ]
paths:
- "crates/ecstore/src/disk/**"
- "crates/ecstore/src/store/init_format.rs"
- "crates/ecstore/Cargo.toml"
- "Cargo.toml"
- "Cargo.lock"
- ".github/actions/setup/**"
- ".github/workflows/windows-filesystem.yml"
workflow_dispatch:
permissions:
contents: read
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
env:
CARGO_TERM_COLOR: always
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
RUST_BACKTRACE: 1
jobs:
rename-safety:
name: Rename Safety
runs-on: windows-latest
timeout-minutes: 60
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: stable
cache-shared-key: build-x86_64-pc-windows-msvc
cache-save-if: 'false'
install-build-packaging-tools: 'false'
install-test-tools: 'false'
- name: Check production Windows dependencies
shell: pwsh
run: cargo check -p rustfs-ecstore --lib
- name: Test guarded rename publication
shell: pwsh
run: cargo test -p rustfs-ecstore --lib rename_all_ -- --nocapture
- name: Test Windows handle guards
shell: pwsh
run: cargo test -p rustfs-ecstore --lib windows_ -- --nocapture
- name: Test startup temporary-directory cleanup
shell: pwsh
run: cargo test -p rustfs-ecstore --lib cleanup_tmp_on_startup_ -- --nocapture
- name: Test fresh format publication
shell: pwsh
run: cargo test -p rustfs-ecstore --lib fresh_format_load_initializes_all_disks -- --nocapture
-4
View File
@@ -83,7 +83,3 @@ worktrees/*
# Local AI-agent review artifacts (omo evidence dumps)
.omo/
# insta scratch files; the accepted .snap files ARE the assertions and are committed
*.snap.new
*.pending-snap
-116
View File
@@ -1,116 +0,0 @@
---
name: issue-triage
description: Triage a GitHub issue — determine if it is already fixed, needs implementation, or should be closed. Searches related commits and PRs, verifies implementation status, and posts a triage comment or closes the issue. Use when the user provides an issue URL and asks whether it can be closed or needs work.
---
# Issue Triage
Use this skill when the user provides a GitHub issue URL and asks "can this be closed?", "is this already implemented?", "check completion status", or similar triage questions.
## Workflow
### 1. Fetch issue context
```bash
gh issue view <N> --repo <owner/repo> --json title,body,state,comments,labels,updatedAt
```
Read the issue body to understand what was requested. Extract:
- The specific feature/fix/behavior described.
- Any linked PRs or commits mentioned in the body or comments.
- Any checklist items or sub-issues.
### 2. Search for related work
Search git history for commits referencing the issue:
```bash
git log --oneline --all --grep="<N>" | head -30
```
Search for related PRs:
```bash
gh pr list --search "fixes #<N> OR closes #<N> OR #<N>" --state all --json number,title,state,mergedAt
```
If the issue mentions specific PRs, check their status:
```bash
gh pr view <PR_N> --json state,mergedAt,title
```
### 3. Verify implementation
For each linked or related PR that is merged, verify the fix is actually present on the current main branch:
```bash
git log --oneline main | grep -i "<keyword>"
# or
git log --oneline main --grep="<PR_N>"
```
If the issue describes a specific defect, check the relevant code to confirm the fix is in place:
```bash
grep -n "<pattern>" crates/<relevant>/src/<file>.rs
```
For issues with checklists, verify each item individually. If sub-items are tracked as separate issues, check those too:
```bash
gh issue view <SUB_N> --repo <owner/repo> --json state
```
### 4. Determine verdict
- **All items fixed and merged**: Close with a summary comment listing what was fixed and which PRs.
- **Some items fixed, some remaining**: Comment with status of each item. Do not close.
- **Not yet implemented**: Comment with a summary of what remains. Do not close.
- **Superseded or no longer relevant**: Close with explanation.
### 5. Take action
Close with comment:
```bash
gh issue close <N> --repo <owner/repo> --comment "<body>"
```
Comment without closing:
```bash
gh issue comment <N> --repo <owner/repo> --body-file /tmp/triage.md
```
Update issue labels if needed:
```bash
gh issue edit <N> --repo <owner/repo> --add-label "completed" --remove-label "needs-triage"
```
Always use `--body-file` for multiline content, never inline `--body`.
### 6. Handle multi-issue batches
When the user asks to check multiple issues (e.g., "check all issues by user X" or "scan backlog for closable issues"):
1. List the issues: `gh issue list --repo <repo> --author <user> --state open --json number,title,updatedAt`
2. For each issue, run steps 1-5 above.
3. Report a summary table of all triaged issues with verdicts.
## Output format
### Issue Triage: #<N> — <title>
**State**: OPEN / CLOSED
**Linked PRs**: <list with merge status>
#### Assessment
<what was requested vs what is implemented>
#### Verdict
- Close — all items resolved by <PR list>
- Keep open — <remaining items>
- Not started — <what needs to be done>
#### Action taken
- Closed with comment / Commented / No action
## Notes
- The user may ask in Chinese ("是否可以关闭", "检查完成情况"); respond in the same language.
- When closing, always include a summary of what was fixed and which PRs resolved it — this creates a useful audit trail.
- For issues in `rustfs/backlog`, use `--repo rustfs/backlog`.
- For issues in `rustfs/rustfs`, use `--repo rustfs/rustfs`.
- If the issue has sub-issues (GitHub sub-issues API), check each one's state before declaring the parent complete.
-147
View File
@@ -1,147 +0,0 @@
---
name: pr-review
description: Review a GitHub PR end-to-end from a URL or number — fetch metadata, inspect the diff, run multi-role adversarial review, check CI status, and post the review comment. Use when the user provides a PR link and asks to review it.
---
# PR Review
Use this skill when the user provides a GitHub PR URL or number and asks to review it. This covers the full review lifecycle: data gathering, code review, CI verification, and posting the result.
## Prerequisites
- Read `AGENTS.md` for the repository's adversarial validation policy and change-style rules.
- The `adversarial-validation` skill handles the review role playbooks; this skill orchestrates the workflow around it.
## Workflow
### 1. Gather PR context
```bash
gh pr view <N> --json title,author,state,body,additions,deletions,changedFiles,commits,baseRefName,headRefName
gh pr diff <N> --name-only
```
Read the PR body and linked issues to understand the change's purpose. If the PR references an issue, fetch that too:
```bash
gh issue view <ISSUE> --json title,body,state
```
### 2. Fetch the diff and classify the change
```bash
git fetch origin pull/<N>/head:pr-<N>
git diff main...pr-<N> --stat
```
Classify the change by risk tier (per AGENTS.md):
- **Exempt**: docs/comments/instruction-only, formatting, typos.
- **Mechanical**: renames, file moves, test-only or tooling changes.
- **Standard** (default): any behavior change.
- **High risk**: locking, erasure coding, quorum/heal, replication, multipart, RPC, lifecycle/tiering, metadata formats, persistence/fsync, IAM/KMS/auth, on-disk/on-wire formats, S3 API-visible behavior.
### 3. Cluster changed files and delegate review
Group the changed files into logical clusters (by crate or functional area). For each cluster, spawn a subagent with a focused review prompt that includes:
- The cluster's changed files and their diffs.
- The applicable adversarial role probes (from the `adversarial-validation` skill).
- The repository's AGENTS.md rules relevant to that domain.
For standard-tier changes: correctness adversary + simplicity adversary + test-coverage skeptic, plus every role whose domain the diff touches.
For high-risk changes: run all seven roles.
Each subagent must produce findings (concrete failure scenario with file:line) or a null report ("attacked X, Y, Z — no break found").
### 4. Check CI status
```bash
gh pr checks <N>
```
If any checks fail, investigate:
```bash
gh run view --log-failed --job=<JOB_ID>
```
Determine whether failures are pre-existing (on main), flaky, or caused by the PR.
### 5. Synthesize findings
Combine all subagent findings into a structured review:
- **Summary**: one-paragraph overview of the change and overall assessment.
- **Findings**: each finding with severity (critical/major/minor/nit), file:line, concrete failure scenario, and suggested fix.
- **CI status**: pass/fail with notes on any failures.
- **Verdict**: APPROVE, REQUEST_CHANGES, or COMMENT.
### 6. Post the review
Write the review body to a temp file and post via CLI:
```bash
# Request changes
gh pr review <N> --request-changes --body-file /tmp/pr_review.md
# Approve
gh pr review <N> --approve --body-file /tmp/pr_review.md
# Comment only (no verdict)
gh pr review <N> --comment --body-file /tmp/pr_review.md
```
For inline comments on specific lines, use the GitHub API:
```bash
cat > /tmp/pr_review.json <<'EOF'
{
"body": "review body",
"event": "REQUEST_CHANGES",
"comments": [
{
"path": "crates/foo/src/bar.rs",
"line": 42,
"body": "finding description"
}
]
}
EOF
gh api --method POST /repos/{owner}/{repo}/pulls/<N>/reviews --input /tmp/pr_review.json
```
Always use `--body-file` or `--input`, never inline multiline `--body`.
### 7. Handle follow-up
If the review requests changes:
- Monitor for new commits: `gh pr view <N> --json commits`
- Re-review changed files only: `git diff pr-<N>..origin/pull/<N>/head`
- Update the review when findings are addressed.
If CI was failing due to pre-existing main breakage:
- Comment on the PR noting the failure is pre-existing.
- Suggest updating the branch: `gh pr update-branch <N>`
## Output format
### PR Review: #<N> — <title>
**Author**: <author>
**Risk tier**: exempt | mechanical | standard | high-risk
**Changed files**: <count> across <cluster count> clusters
#### Summary
<one-paragraph overview>
#### Findings
| Severity | Location | Finding |
|----------|----------|---------|
| critical | file:line | concrete failure scenario |
#### CI Status
- All checks pass / Failing: <details>
#### Verdict
APPROVE / REQUEST_CHANGES / COMMENT
## Notes
- The user may ask for review in Chinese; respond in the same language but keep the review body in English per AGENTS.md rules.
- When the user asks for "多角色对抗 review", run the full adversarial validation protocol — this skill's step 3 covers that.
- If the PR is from a fork, check `maintainerCanModify` before attempting to push fixes.
- For very large PRs (>50 files), cluster aggressively and delegate in parallel to keep review time reasonable.
+8 -35
View File
@@ -172,7 +172,7 @@
],
},
{
"name": "Debug executable target/debug/rustfs with sse kms",
"name": "Debug executable target/debug/rustfs with sse",
"type": "lldb",
"request": "launch",
"program": "${workspaceFolder}/target/debug/rustfs",
@@ -200,7 +200,7 @@
// 2. kms local backend test key
// "RUSTFS_KMS_ENABLE": "true",
// "RUSTFS_KMS_BACKEND": "local",
// "RUSTFS_KMS_KEY_DIR": "/tmp/kms-key-dir",
// "RUSTFS_KMS_KEY_DIR": "./target/kms-key-dir",
// "RUSTFS_KMS_LOCAL_MASTER_KEY": "my-secret-key", // Some Password
// "RUSTFS_KMS_DEFAULT_KEY_ID": "rustfs-master-key",
@@ -212,40 +212,13 @@
// "RUSTFS_KMS_DEFAULT_KEY_ID": "rustfs-master-key",
// 4. kms vault transit backend test key
// "RUSTFS_KMS_ENABLE": "true",
// "RUSTFS_KMS_BACKEND": "vault-transit",
// "RUSTFS_KMS_VAULT_ADDRESS": "http://127.0.0.1:8200",
// "RUSTFS_KMS_VAULT_TOKEN": "Dev Token",
// "RUSTFS_KMS_VAULT_MOUNT_PATH": "transit",
// "RUSTFS_KMS_DEFAULT_KEY_ID": "rustfs-master-key",
// 5、kms static backend test key
"RUSTFS_KMS_ENABLE": "true",
"RUSTFS_KMS_BACKEND": "static",
"RUSTFS_KMS_STATIC_SECRET_KEY": "rustfs-master-key:2dfNXGHlsEflGVCxb+5DIdGEl1sIvtwX+QfmYasi5QM="
},
"sourceLanguages": [
"rust"
],
},
{
"name": "Debug executable target/debug/rustfs with local sse",
"type": "lldb",
"request": "launch",
"program": "${workspaceFolder}/target/debug/rustfs",
"args": [],
"cwd": "${workspaceFolder}",
"env": {
"RUSTFS_ACCESS_KEY": "rustfsadmin",
"RUSTFS_SECRET_KEY": "rustfsadmin",
"RUSTFS_VOLUMES": "./target/volumes/test{1...4}",
"RUSTFS_ADDRESS": ":9000",
"RUSTFS_CONSOLE_ENABLE": "true",
"RUSTFS_CONSOLE_ADDRESS": "127.0.0.1:9001",
"RUSTFS_OBS_LOG_DIRECTORY": "./target/logs",
"RUSTFS_UNSAFE_BYPASS_DISK_CHECK": "true",
"RUSTFS_SSE_S3_MASTER_KEY": "xGb3aYSp825j2tPpg8JrUzghiXsIkfdOtmrsJ/iafiM=",
"RUST_LOG": "rustfs=debug,ecstore=debug,s3s=debug,iam=debug",
"RUSTFS_KMS_BACKEND": "vault-transit",
"RUSTFS_KMS_VAULT_ADDRESS": "http://127.0.0.1:8200",
"RUSTFS_KMS_VAULT_TOKEN": "Dev Token",
"RUSTFS_KMS_VAULT_MOUNT_PATH": "transit",
"RUSTFS_KMS_DEFAULT_KEY_ID": "rustfs-master-key",
},
"sourceLanguages": [
"rust"
+34 -132
View File
@@ -21,28 +21,6 @@ If repo-level instructions conflict, follow the nearest file and keep behavior a
- Avoid redundant file reads, repeated commands, and unnecessary exploratory work once enough context is available.
- A good result is a minimal diff with clear assumptions, no over-engineering, and independent verification that survives Adversarial Validation (below).
## Worktree and Disk Hygiene
- Unless the requester explicitly says otherwise, treat every new implementation task as isolated work: fetch the latest `origin/main`, confirm the requested change is not already present there, and create a dedicated feature branch and worktree from that exact upstream commit before editing. Do not implement new work directly in the primary checkout or reuse a worktree from another task.
- Check available disk space before creating the worktree or starting dependency downloads, builds, tests, coverage, or other artifact-heavy commands. For long-running or artifact-heavy work, re-check disk usage at natural phase boundaries and before broad validation; if remaining space may not safely accommodate the next command, stop and reclaim task-owned artifacts before continuing.
- Keep cleanup scoped and safe: remove generated build/test/coverage artifacts and temporary files created by the task when they are no longer needed, and never delete another task's worktree or uncommitted files. Prefer shared dependency caches where supported instead of duplicating large artifacts across worktrees.
- At handoff, report the disk-space checks, cleanup performed, and any retained worktree or artifacts with the reason they are still needed.
## PR Lifecycle Monitoring
- Creating or updating a PR is not the terminal state. Unless the requester explicitly limits the task to PR creation, monitor the PR through its terminal state: merged, closed, or explicitly handed off because progress requires user or maintainer action.
- While the task is active, monitor CI/check runs, review decisions and unresolved threads, mergeability and conflicts, and unexpected head/base changes. Prefer event-driven or bounded waits provided by the current environment over frequent polling; report only state changes, actionable failures, or meaningful prolonged delays.
- Investigate every failing check and review comment before changing code. Fix failures attributable to the task, run the verification required for the new diff, push the update, respond to or resolve the corresponding review threads, and resume monitoring. Do not weaken checks, dismiss valid feedback, or retry flaky failures merely to obtain a green result.
- Treat opening, green CI, approval, and mergeability as intermediate states. Never merge without the required reviewer approval or explicit authority. If progress depends on credentials, infrastructure, a maintainer decision, or another external action, report the exact blocker and the evidence already collected.
- If the current execution environment cannot remain active until the next PR event, use a supported automation, monitor, or thread wakeup when available and within scope. Otherwise leave an explicit handoff containing the PR, current state, next event to observe, and pending cleanup; do not imply that background monitoring exists when none is scheduled.
- After observing a merge, verify the commits are preserved on the upstream base, ensure the worktree is clean, remove the dedicated worktree, prune stale worktree metadata, and delete the local task branch when it is no longer in use. For a closed or abandoned PR, preserve any unmerged work unless deletion was explicitly authorized. Do not delete remote branches unless explicitly requested or repository automation owns that cleanup.
## Autonomy and Approval Boundaries
- Inquiry tasks (answer, explain, review, diagnose, plan): report findings; do not change files unless a fix is explicitly requested.
- Action tasks (change, build, fix): make in-scope local changes without asking for approval.
- Ask for confirmation before destructive or hard-to-reverse operations (force-pushes, history rewrites, deleting data or branches), merging a PR (reviewer approval required), or any material expansion of the requested scope.
## Communication and Language
- Respond in the same language used by the requester.
@@ -51,25 +29,26 @@ If repo-level instructions conflict, follow the nearest file and keep behavior a
## Change Style for Existing Logic
- Start with the smallest direct, local edit. Add production files, types, traits, helpers, wrappers, or abstraction layers only when current behavior requires them. Extraction must remove present duplication, enforce a real boundary, or materially clarify a non-trivial flow; anticipated reuse is not enough.
- Prefer direct, local code over extracting one-off helpers.
- Extract a helper only when logic is reused or the extraction materially clarifies a non-trivial flow.
- Use Rust's default module file layout (`mod foo;` with `foo.rs` or `foo/mod.rs`/`foo/*.rs`).
Avoid `#[path = "..."]` for module inclusion; move files into the canonical module tree instead.
If an unavoidable generated-code, FFI, or test-fixture exception remains, keep it local and document why the canonical layout cannot work.
- Solve only the requested problem; do not add speculative features, configurability, or adjacent improvements.
- Prefer editing existing code over rewriting files or reshaping unrelated logic.
- Modify only what is required. Remove any in-scope path or representation superseded by the change. If compatibility or rollback requires retention, adapt at the boundary to one canonical core and follow the repository's `RUSTFS_COMPAT_TODO` removal policy; never delete unrelated code merely to improve addition/deletion statistics.
- Modify only what is required and remove only artifacts introduced by your own changes.
- Preserve the existing control-flow and logic shape when fixing bugs or addressing review comments, especially in init, distributed coordination, locking, metadata, and concurrency paths.
- Do not refactor existing code only to make it easier to unit test.
- Keep fixes narrowly aligned with the requested behavior; avoid semantic-adjacent rewrites while touching sensitive paths.
- Keep code elegant, concise, and direct. Prefer the smallest readable design and existing abstractions over parallel managers, factories, adapters, or wrappers added only to make the design look extensible.
- Comments state non-obvious reasons, assumptions, and invariants in the shortest complete form. Their length follows the invariant's complexity: `SAFETY`, lock ordering, durability, and compatibility contracts may need a short list of conditions. Never narrate the next line, restate a signature, or record change history; move durable design rationale to architecture or operations documentation.
- Keep code elegant, concise, and direct. Prefer minimal, readable implementations over over-engineering and excessive abstraction. Use comments to clarify non-obvious intent and invariants, not to compensate for unclear code.
- Do not write comments that narrate what the next line does, restate a signature, or describe the change you just made — that commentary belongs in the PR description, not the code. Required invariant comments — lock ordering, `SAFETY`, unwrap justification, `#[allow(dead_code)]` rationale, `RUSTFS_COMPAT_TODO` — are never narration.
- Mention unrelated issues when useful, but do not fix them as part of a narrow task.
## Reuse Before You Write
Search for an existing implementation before writing a new one; extend what exists instead of duplicating it:
- **Helpers and utilities** (path/string handling, hashing, retry, env parsing, IO wrappers): check `crates/utils`, `crates/common`, the touched crate, the likely domain-owning crate, and relevant direct workspace dependencies from `Cargo.toml`. Search snake_case signatures with a focused term. Reimplementing an existing workspace helper — or hand-rolling what `std`, `tokio`, or an existing dependency already provides — is a review finding, not a style preference.
- **Helpers and utilities** (path/string handling, hashing, retry, env parsing, IO wrappers): check `ls crates/utils/src` first — file names map to operations (`retry.rs`, `envs.rs`, `hash.rs`, `path.rs`, `string.rs`, `io.rs`) — plus `crates/common` (shared structures/globals), then `rg -i 'fn \w*<term>' crates/utils/src crates/common/src <touched-crate>/src` for signatures. Helpers are snake_case: a full-text single-word grep over a large crate drowns you and a multi-word phrase returns nothing. Reimplementing an existing workspace helper — or hand-rolling what `std`, `tokio`, or an existing workspace dependency already provides — is a review finding, not a style preference.
- **Reuse requires matching semantics, not a matching name**: before adopting a helper, check its normalization (`clean` resolves `.`/`..` — never apply it to raw S3 object keys), error type, backoff/deadline behavior, and durability gating against the call site. When semantics differ, a new narrowly-named helper with a comment naming the rejected lookalike is the correct outcome. The inverse also holds: workspace wrappers exist because raw `std`/`tokio` semantics were insufficient (durability gates, retries) — prefer the wrapper over the raw call.
- **Constants and fixed tokens** (protocol labels, error identifiers, header keys, event names, metric names, command tags): search for existing constants/enums that already represent the same semantic value and reuse them. If a value is truly new, define one local constant near related logic; never scatter the literal across sites. When changing existing behavior, align naming and format with the established constants.
- **Test scaffolding**: reuse existing test utilities and fixtures (the touched crate's own `test_util` module and `tests/fixtures`, or `crates/test-utils`) instead of writing new setup code — run `rg -l '<fn-under-test>' <crate>/src <crate>/tests` before writing a test. A new test must pin a failure mode no existing test covers. Near-duplicate means same code path AND same poison-value class: this repo's boundary companions (n==max vs max+1, absent vs empty vs nil UUID bytes, MetaObject vs MetaDeleteMarker) are distinct by definition and must all be written.
@@ -78,7 +57,6 @@ Search for an existing implementation before writing a new one; extend what exis
Net-new code — files, types, branches, comments — is cost to justify, not progress:
- Inspect production-code additions separately. Tests, fixtures, generated code, and documentation do not count as production-code growth. Line counts are signals, not quotas: new production structures must map to a current requirement, and a blocker requires a concrete smaller design that preserves correctness, compatibility, readability, and real boundaries.
- Validate at the trust boundary — untrusted client input, bytes read from disk, RPC payloads, config (see Serde Safety and Cross-Cutting Domain Invariants) — then trust the type: do not re-check what the type system or a validated upstream layer already guarantees, and cite the establishing check (`file:line`) when the guarantee is not obvious.
- The exception is load-bearing: a value that crossed a persistence, RPC, or version boundary is never guaranteed by the code on the other side — a peer may be older or buggy, disk bytes may be corrupt — so the Cross-Cutting Domain Invariant patterns apply at every consumer, and re-checks immediately before a destructive action (delete, overwrite, quorum decision) stay. Deleting an existing guard is a behavior change requiring adversarial review, not cleanup.
- Every new branch needs a nameable trigger: a concrete input, state, or failure that reaches it — for boundary-crossing values, corrupt or stale persisted/peer data is always nameable. If you cannot name one, do not write the branch. If the case is truly unreachable, encode the invariant in the type; where that is impossible, return a typed internal error (fail closed). `debug_assert!` is acceptable only for pure internal arithmetic on values that never crossed a disk/RPC/config boundary — never as the sole guard on decoded or peer-supplied data.
@@ -121,78 +99,33 @@ CI) fails the build if anything is committed under `docs/superpowers/`, even via
## Verification Before PR
Convert changes into independently verifiable outcomes. This section controls
agent-run local validation; preparing a commit or PR does not by itself require
the broadest gate. Inspect only the final task-owned diff, classify it by
behavioral impact rather than line count or path alone, and run the smallest
set of checks that provides meaningful coverage. Do not let unrelated
worktree changes or a generic contributor checklist expand the scope.
Non-exempt changes must also pass Adversarial Validation (next section) before
the checks below count as completion.
Convert changes into independently verifiable outcomes. Prefer focused tests for behavior changes and run the relevant checks before declaring completion.
Non-exempt changes must also pass Adversarial Validation (next section) before the checks below count as completion.
### Validation floor
For code changes, run and pass the following before opening a PR:
- Every change that is not documentation-only must finish with
`cargo fmt --all --check` passing. An umbrella gate that runs this exact
check satisfies the requirement; do not run it twice. Use `cargo fmt --all`
only when formatting needs to be fixed. Run the configured formatter or
validator for other changed languages when one exists.
- Documentation-only or instruction-only means all task-owned changes are
prose or documentation assets and cannot affect runtime, builds, CI,
dependencies, generated code, or tests. Run `git diff --check` and any
relevant documentation guard, but skip Cargo formatting, compilation,
Clippy, tests, `make pre-commit`, and `make pre-pr`.
- Behavior changes require relevant existing or new tests. Prefer the most
focused test or affected package. A passing targeted test can also provide
sufficient compilation coverage when it builds every changed target and
feature involved; do not add a redundant `cargo check` in that case.
- `cargo check` supplements compilation coverage; it never substitutes for a
behavioral test. If a relevant test cannot reasonably be added or run, use
the narrowest compilation check and report the reason and remaining risk.
```bash
make pre-pr
```
### Validation tiers
Before committing code changes, prefer focused verification for the touched
surface and use the faster local gate when a broad smoke check is needed:
1. **Documentation/instruction-only:** Apply the exemption above. Run a guard
such as `make doc-paths-check` only when it is relevant to the edited text.
2. **Non-behavioral source change:** For comments, formatting, or another
demonstrably non-executable change, run the formatting floor. Compilation,
Clippy, and tests may be skipped only when the edit cannot affect
compilation or runtime behavior; run targeted doctests if executable
documentation examples changed.
3. **Localized or bounded behavior change:** Run the formatting floor and the
narrowest relevant tests. Add package-scoped `cargo check` or Clippy only
for changed targets, features, APIs, error handling, async behavior, or
control flow not already covered. When several crates are affected but the
dependency set is identifiable, validate those packages and known
dependents instead of the whole workspace. Use `make pre-commit` only when
a repository-wide fast gate adds useful confidence beyond those checks.
4. **Broad or high-risk change:** Run `make pre-pr` only when targeted coverage
cannot bound the impact, including:
- dependency, feature, build-script, procedural-macro, code-generation,
toolchain, or CI changes that alter compilation or the test matrix;
- cross-crate public APIs, shared foundational code, or broad refactors with
an unbounded dependent set;
- locking, storage durability or formats, erasure coding, replication,
RPC/protocol compatibility, IAM/KMS/auth, cryptography, or other
security-sensitive behavior;
- a targeted check that reveals wider impact, an explicit user request, or
a release policy that requires the full gate.
```bash
make pre-commit
```
Documentation-only and non-behavioral classifications take precedence over
path-based triggers. A small diff can still be high-risk, while a CI comment,
manifest comment, or release-note edit does not require full validation.
For migration batches, do not run the full `make pre-pr` gate before every
intermediate commit. Use focused tests and `make pre-commit` during
development, then reserve `make pre-pr` for the final PR-ready branch.
`make pre-pr` includes `make pre-commit` coverage. Never run both for the same
unchanged diff, and do not repeat equivalent checks during PR preparation or
because a local hook already ran them. Rerun only checks whose scope is affected
by later edits. Full workspace checks do not replace a relevant integration or
E2E test for changed behavior; run that focused test when required and
available, or report why it was not run and the remaining risk.
Before pushing code changes, make sure formatting is clean:
If `make` is unavailable, run the equivalent checks defined under
`.config/make/`. At handoff, list the checks actually run, checks intentionally
skipped, and the reason for the selected tier.
- Run `cargo fmt --all`.
- Run `cargo fmt --all --check` and ensure no files are modified unexpectedly.
If `make` is unavailable, run the equivalent checks defined under `.config/make/`.
Documentation-only or instruction-only changes are exempt from the verification commands above (including the `.config/make/` equivalents), though any locally installed git pre-commit hooks may still run on commit unless explicitly skipped.
After build-based verification completes, clean generated build artifacts before wrapping up to avoid unnecessary disk usage.
Do not open a PR with code changes when the required checks fail.
Make a failing check pass by fixing the cause, never by weakening the gate:
@@ -218,10 +151,9 @@ not to bless it.
Pick the tier from the riskiest file touched; when in doubt, pick the higher.
- **Exempt:** docs/comments, formatting, and typos that cannot affect runtime,
builds, tests, or agent execution. Skip this section.
- **Mechanical:** pure renames, file moves, test-only or tooling changes, and
agent-instruction changes that alter execution —
- **Exempt:** docs/comments/instruction-only changes, formatting, typos with
no runtime surface. Skip this section.
- **Mechanical:** pure renames, file moves, test-only or tooling changes
correctness and simplicity adversaries only.
- **Standard (the default):** any change that affects behavior.
- **High risk:** touches locking, erasure coding, quorum/heal, replication,
@@ -243,7 +175,7 @@ encode this repo's shipped bugs.
- **Correctness adversary** — construct a concrete input/state/interleaving
that yields wrong output, data loss, or a crash. Probe error paths and edge
values (empty, nil UUID, zero-length, quorum1, missing version).
- **Simplicity adversary** — same behavior, less code. Hunt reimplemented helpers, rewrites where an in-place edit suffices, speculative abstractions, defensive branches with no nameable trigger, redundant error wrapping, near-duplicate tests, and narration comments. A one-caller helper is a finding only when it merely forwards or splits a short linear flow without adding domain naming, boundary isolation, an invariant, or useful error context. Report a concrete smaller replacement; fewer lines alone are not evidence.
- **Simplicity adversary** — same behavior, less code. Hunt the materially smaller or more idiomatic diff (see Change Style for Existing Logic, Reuse Before You Write, and Necessary Code Only): reimplemented workspace helpers, one-caller extractions, rewrites where an in-place edit suffices, defensive branches with no nameable trigger, redundant error wrapping, near-duplicate tests, narration comments. A smaller diff achieving identical behavior is a finding, reported with the concrete replacement; forced reuse of a helper with mismatched semantics is equally a finding.
- **Security reviewer** — authn/authz bypass, injection, secret leakage,
untrusted deserialization (see Serde Safety), path traversal, timing leaks.
- **Concurrency/durability reviewer** — lock ordering, races, cancellation,
@@ -254,11 +186,10 @@ encode this repo's shipped bugs.
time across IO, sync or CPU-heavy work on async runtime threads, added
fsync/flush outside the durability gate, hot-path logging noise. A
measurable regression on a per-request or per-object path is a finding.
- **Test-coverage skeptic** — for each testable behavior claim, name the test
or executable check that detects a revert; then name a changed line that
could be wrong while all checks stay green. If a focused check is not
reasonable, require the reason and residual risk from the validation floor.
Test additions have no line-count or growth budget.
- **Test-coverage skeptic** — for each claimed behavior, name the test that
fails if the change is reverted; then name a changed line that could be
wrong while all tests stay green — if one exists, coverage is insufficient.
A missing test is a finding, not a note.
Standard tier: correctness adversary + simplicity adversary + test-coverage
skeptic, plus every role whose domain the diff touches (async or
@@ -284,9 +215,7 @@ High risk: all seven roles.
- Every applicable role has run; every finding is fixed or rebutted with
evidence.
- Every testable behavior change has a focused regression check. Exceptions
follow the validation floor and state why a check is impractical and what
risk remains.
- Every behavior change has a test that fails without it.
- The Verification Before PR gates pass — adversarial review supplements
those gates, never replaces them.
- High risk only: record a one-line verdict per role in the PR description.
@@ -326,28 +255,6 @@ High risk: all seven roles.
- Use environment variables or vault tooling for sensitive configuration.
- For localhost-sensitive tests, verify proxy settings to avoid traffic leakage.
## Logging
Applies to **every** `tracing` macro you add or edit, including a single line
added in passing while fixing something else — not only to log-focused changes.
- Fields first, message second: `event`, `component`, `subsystem`,
`result`/`state`, then key context. The message is a short label, not a
sentence with values interpolated into it.
- Reuse the existing `EVENT_*` / `LOG_COMPONENT_*` / `LOG_SUBSYSTEM_*`
constants of the module you are editing; match the shape of the log sites
already in that file rather than introducing a second style next to them.
- Level policy: `error` for behavior/security-affecting failures, `warn` for
degraded or fallback paths, `info` for low-frequency lifecycle, `debug` for
targeted diagnostics, `trace` for hot paths. Per-object and per-request
success paths are `trace`.
- Never log secrets, tokens, credential payloads, or merged config dumps.
- `scripts/check_logging_guardrails.sh` enforces a subset of this on the files
it lists; passing it is a floor, not evidence the log matches the house style.
See `.agents/skills/rustfs-logging-governance/SKILL.md` for the full event
model, level policy, and guardrail-update checklist.
## Tools
### xl.meta decode tool Quick Use
@@ -373,11 +280,6 @@ cargo run -p rustfs-filemeta --example dump_fileinfo -- "/path/to/file/xl.meta"
absent, empty, and nil all mean "no value", never `Uuid::nil()`.
- A remote-tier version of `None`/`""` means the tier bucket is unversioned:
send **no** `versionId` on tier GET/DELETE.
- Structs persisted in the scanner data-usage cache (`DataUsageCacheInfo`,
`DataUsageEntry`) carry a hand-written map-encoded `Serialize`. MessagePack
encodes derived structs as arrays, where an appended field makes the whole
cache a decode error for older readers — keep new fields `#[serde(default)]`
and keep the map encoding rather than reverting to `derive(Serialize)`.
## Naming Conventions
+20 -64
View File
@@ -1,6 +1,6 @@
# ARCHITECTURE.md
> Last updated: 2026-08-12 · Revision: 3
> Last updated: 2026-07-02 · Revision: 2
>
> This document describes the high-level architecture of RustFS.
> If you want to familiarize yourself with the code base, you are in the right place!
@@ -119,44 +119,19 @@ module split is tracked under `docs/architecture/`.
3. **Each type has exactly one definition.** Types shared across crates must be defined
in one crate and re-exported or imported by others.
- ⚠️ VIOLATED: `ReplicationStats` names three unrelated types
(`crates/data-usage/src/data_usage.rs`,
`crates/obs/src/metrics/collectors/replication.rs`,
`crates/ecstore/src/bucket/replication/replication_state.rs`) — a naming
collision, not copies; renaming is tracked in rustfs/backlog#1847.
- `LastMinuteLatency` has two deliberately different implementations: the
per-second bucketed accumulator in `crates/common/src/last_minute.rs` and
the in-memory endpoint-health sample tracker in
`crates/ecstore/src/bucket/bucket_target_sys.rs` (its doc comment explains
why it stays local).
- ✅ RESOLVED: `BackpressureConfig` and `DataUsageInfo` each have exactly one
definition (`crates/io-core/src/backpressure.rs`,
`crates/data-usage/src/data_usage.rs`). A zero-consumer
`BackpressureSettings` copy lingers in `crates/io-metrics/src/config.rs`;
its removal is tracked in rustfs/backlog#1833.
- ⚠️ VIOLATED: `ReplicationStats` (4 copies), `LastMinuteLatency` (3 copies),
`BackpressureConfig` (3 copies), `DataUsageInfo` (2 copies).
4. **ecstore does not know about HTTP or S3 protocol details.** It operates on
storage-level abstractions (objects, buckets, disks, pools).
- ⚠️ VIOLATED: 58 files under `crates/ecstore/src` reference `s3s`
(`rg -l 's3s' crates/ecstore/src | wc -l`), `crates/ecstore/src/client/`
is a ~9.4K-line embedded S3 HTTP client, and `crates/ecstore/Cargo.toml`
depends on `s3s`, `http`, `hyper`/`hyper-util`/`hyper-rustls`, and
`reqwest`. Target state: the engine's need to act as an S3 client
(tiering, replication targets) is served by an extracted client crate,
and ecstore holds no wire or DTO types.
5. **The `rustfs` binary crate is the only place that wires everything together.**
Individual crates should be testable in isolation.
6. **Error types use `thiserror` with descriptive names** (e.g., `StorageError`,
not bare `Error`).
- ✅ RESOLVED (strategy): `snafu` is gone from source
(`rg -l snafu crates/ rustfs/` is empty) and library code no longer uses
`anyhow` (remaining hits are test code and the `e2e_test` crate; `heal`
uses `thiserror`).
- ⚠️ VIOLATED (naming): 6 crates still export a bare `pub enum Error`:
`crypto`, `filemeta`, `heal`, `iam`, `policy`, and `replication`
(`src/resync.rs`) — all `thiserror`-derived.
- ⚠️ VIOLATED: 6 crates use `pub enum Error`; 2 crates use `snafu`;
`heal` use `anyhow` in library code.
## Known Structural Issues
@@ -165,25 +140,13 @@ module split is tracked under `docs/architecture/`.
### Critical
- **scanner/data-usage duplicate `.usage-cache.bin` serialization types.** The
original finding ("common/scanner code duplication, ~3K lines") is resolved:
`scanner` imports the shared data-usage types from `rustfs-data-usage` (see
the `pub use rustfs_data_usage::…` re-exports at the top of
`crates/scanner/src/data_usage_define.rs`). What remains: `scanner` and
`data-usage` each hold their own serialization types for the scanner cache
file (`DataUsageCacheInfo`/`DataUsageEntryInfo` in
`crates/scanner/src/data_usage_define.rs` vs
`DataUsageCacheInfo`/`DataUsageEntry` in
`crates/data-usage/src/data_usage.rs`); convergence is tracked in
rustfs/backlog#1828.
- **common/scanner code duplication (~3K lines).** `scanner` depends on `common`
but maintains its own copies of `DataUsageInfo`, `LastMinuteLatency`, and related
types instead of importing them.
- **ecstore is a monolith (265 files, ~288K lines — roughly half is inline
`#[cfg(test)]` code).** Measured with
`find crates/ecstore/src -name '*.rs' | xargs wc -l`. It contains disk
management, bucket management, erasure coding, replication, lifecycle, RPC,
and configuration — all in one crate. It should be decomposed along its
existing subdirectories; the split plan lives in
[docs/architecture/ecstore-module-split-plan.md](docs/architecture/ecstore-module-split-plan.md).
- **ecstore is a monolith (87K lines, 163 files).** It contains disk management,
bucket management, erasure coding, replication, lifecycle, RPC, and configuration
— all in one crate. It should be decomposed along its existing subdirectories.
### High
@@ -191,26 +154,19 @@ module split is tracked under `docs/architecture/`.
`common → filemeta/madmin` edges must stay removed so leaf/helper crates do
not regain upward dependencies.
- **Three-layer backpressure/deadlock policy bridging** across io-core,
concurrency, and `rustfs/src/storage`. The config types are no longer
duplicated (`BackpressureConfig` and `DeadlockDetectorConfig` are each
defined once, in io-core). Storage policies expose and consume explicit
projections into the concurrency/io-core policy shapes, and workload
- **Three-layer BackpressureConfig/DeadlockConfig duplication** across io-core,
concurrency, and `rustfs/src/storage`. Storage policies now expose and consume
explicit projections into the concurrency/io-core policy shapes, and workload
admission snapshots are composed through provider registries; later work
should use those bridges before deleting compatibility wrappers.
### Medium
- **Bare `Error` naming.** Error-handling strategy has converged on `thiserror`
(no `snafu`, no `anyhow` in library code); the remaining inconsistency is the
bare `pub enum Error` naming in the 6 crates listed under Invariant 6.
- **Inconsistent error handling.** Three strategies (thiserror/snafu/anyhow) and
mixed naming (bare `Error` vs descriptive names).
- **`common` is mostly parked domain code, not shared utilities.** Of its
6,724 lines, ~83% is scanner/heal domain code stranded there to break
dependency cycles (`metrics.rs`, ~4,810 lines of scanner-domain metrics;
`heal_channel.rs`, ~776 lines of heal-domain channel types). The
"common vs utils" naming ambiguity is secondary to moving that code to its
domain owners.
- **Ambiguous common vs utils boundary.** Both described as "utilities and data
structures." Need clear ownership rules.
## Cross-Cutting Concerns
@@ -276,7 +232,7 @@ The binary (`main.rs`) boots in this order:
```
┌─────────┐
│ rustfs │ (binary + lib)
│ rustfs │ (binary + lib, 75K lines)
│ main │
└────┬────┘
@@ -299,7 +255,7 @@ The binary (`main.rs`) boots in this order:
│ │ │
┌─────▼──────┐ ┌──────▼──────┐ ┌──────▼──────┐
│ ecstore │ │ rio │ │ io-core │
(core) │ │ (readers) │ │ (zero-copy) │
(87K,core) │ │ (readers) │ │ (zero-copy) │
└─────┬──────┘ └─────────────┘ └─────────────┘
┌─────┬──┼──┬─────┬──────┐
Generated
+707 -1075
View File
File diff suppressed because it is too large Load Diff
+116 -106
View File
@@ -68,8 +68,8 @@ resolver = "3"
edition = "2024"
license = "Apache-2.0"
repository = "https://github.com/rustfs/rustfs"
rust-version = "1.97.1"
version = "1.0.0-rc.1"
rust-version = "1.96.0"
version = "1.0.0-beta.10"
homepage = "https://rustfs.com"
description = "RustFS is a high-performance distributed object storage software built using Rust, one of the most popular languages worldwide. "
keywords = ["RustFS", "Minio", "object-storage", "filesystem", "s3"]
@@ -86,84 +86,84 @@ redundant_clone = "warn"
[workspace.dependencies]
# RustFS Internal Crates
rustfs = { path = "./rustfs", version = "1.0.0-rc.1" }
rustfs-heal = { path = "crates/heal", version = "1.0.0-rc.1" }
rustfs-audit = { path = "crates/audit", version = "1.0.0-rc.1" }
rustfs-checksums = { path = "crates/checksums", version = "1.0.0-rc.1" }
rustfs-common = { path = "crates/common", version = "1.0.0-rc.1" }
rustfs-data-usage = { path = "crates/data-usage", version = "1.0.0-rc.1" }
rustfs-config = { path = "./crates/config", version = "1.0.0-rc.1" }
rustfs-concurrency = { path = "./crates/concurrency", version = "1.0.0-rc.1" }
rustfs-credentials = { path = "crates/credentials", version = "1.0.0-rc.1" }
rustfs-crypto = { path = "crates/crypto", version = "1.0.0-rc.1" }
rustfs-ecstore = { path = "crates/ecstore", version = "1.0.0-rc.1" }
rustfs-filemeta = { path = "crates/filemeta", version = "1.0.0-rc.1" }
rustfs-iam = { path = "crates/iam", version = "1.0.0-rc.1" }
rustfs-keystone = { path = "crates/keystone", version = "1.0.0-rc.1" }
rustfs-lifecycle = { path = "crates/lifecycle", version = "1.0.0-rc.1" }
rustfs-kms = { path = "crates/kms", version = "1.0.0-rc.1" }
rustfs-lock = { path = "crates/lock", version = "1.0.0-rc.1" }
rustfs-madmin = { path = "crates/madmin", version = "1.0.0-rc.1" }
rustfs-notify = { path = "crates/notify", version = "1.0.0-rc.1" }
rustfs-io-metrics = { path = "crates/io-metrics", version = "1.0.0-rc.1" }
rustfs-io-core = { path = "crates/io-core", version = "1.0.0-rc.1" }
rustfs-object-capacity = { path = "crates/object-capacity", version = "1.0.0-rc.1" }
rustfs-object-data-cache = { path = "crates/object-data-cache", version = "1.0.0-rc.1", default-features = false }
rustfs-log-analyzer = { path = "crates/log-analyzer", version = "1.0.0-rc.1" }
rustfs-obs = { path = "crates/obs", version = "1.0.0-rc.1" }
rustfs-policy = { path = "crates/policy", version = "1.0.0-rc.1" }
rustfs-protos = { path = "crates/protos", version = "1.0.0-rc.1" }
rustfs-protocols = { path = "crates/protocols", version = "1.0.0-rc.1" }
rustfs-replication = { path = "crates/replication", version = "1.0.0-rc.1" }
rustfs-rio = { path = "crates/rio", version = "1.0.0-rc.1" }
rustfs-rio-v2 = { path = "crates/rio-v2", version = "1.0.0-rc.1" }
rustfs-s3-types = { path = "crates/s3-types", version = "1.0.0-rc.1" }
rustfs-s3-ops = { path = "crates/s3-ops", version = "1.0.0-rc.1" }
rustfs-s3select-api = { path = "crates/s3select-api", version = "1.0.0-rc.1" }
rustfs-s3select-query = { path = "crates/s3select-query", version = "1.0.0-rc.1" }
rustfs-scanner = { path = "crates/scanner", version = "1.0.0-rc.1" }
rustfs-security-governance = { path = "crates/security-governance", version = "1.0.0-rc.1" }
rustfs-extension-schema = { path = "crates/extension-schema", version = "1.0.0-rc.1" }
rustfs-signer = { path = "crates/signer", version = "1.0.0-rc.1" }
rustfs-storage-api = { path = "crates/storage-api", version = "1.0.0-rc.1" }
rustfs-trusted-proxies = { path = "crates/trusted-proxies", version = "1.0.0-rc.1" }
rustfs-targets = { path = "crates/targets", version = "1.0.0-rc.1" }
rustfs-test-utils = { path = "crates/test-utils", version = "1.0.0-rc.1" }
rustfs-tls-runtime = { path = "crates/tls-runtime", version = "1.0.0-rc.1" }
rustfs-utils = { path = "crates/utils", version = "1.0.0-rc.1" }
rustfs-zip = { path = "./crates/zip", version = "1.0.0-rc.1" }
rustfs = { path = "./rustfs", version = "1.0.0-beta.10" }
rustfs-heal = { path = "crates/heal", version = "1.0.0-beta.10" }
rustfs-audit = { path = "crates/audit", version = "1.0.0-beta.10" }
rustfs-checksums = { path = "crates/checksums", version = "1.0.0-beta.10" }
rustfs-common = { path = "crates/common", version = "1.0.0-beta.10" }
rustfs-data-usage = { path = "crates/data-usage", version = "1.0.0-beta.10" }
rustfs-config = { path = "./crates/config", version = "1.0.0-beta.10" }
rustfs-concurrency = { path = "./crates/concurrency", version = "1.0.0-beta.10" }
rustfs-credentials = { path = "crates/credentials", version = "1.0.0-beta.10" }
rustfs-crypto = { path = "crates/crypto", version = "1.0.0-beta.10" }
rustfs-ecstore = { path = "crates/ecstore", version = "1.0.0-beta.10" }
rustfs-filemeta = { path = "crates/filemeta", version = "1.0.0-beta.10" }
rustfs-iam = { path = "crates/iam", version = "1.0.0-beta.10" }
rustfs-keystone = { path = "crates/keystone", version = "1.0.0-beta.10" }
rustfs-lifecycle = { path = "crates/lifecycle", version = "1.0.0-beta.10" }
rustfs-kms = { path = "crates/kms", version = "1.0.0-beta.10" }
rustfs-lock = { path = "crates/lock", version = "1.0.0-beta.10" }
rustfs-madmin = { path = "crates/madmin", version = "1.0.0-beta.10" }
rustfs-notify = { path = "crates/notify", version = "1.0.0-beta.10" }
rustfs-io-metrics = { path = "crates/io-metrics", version = "1.0.0-beta.10" }
rustfs-io-core = { path = "crates/io-core", version = "1.0.0-beta.10" }
rustfs-object-capacity = { path = "crates/object-capacity", version = "1.0.0-beta.10" }
rustfs-object-data-cache = { path = "crates/object-data-cache", version = "1.0.0-beta.10" }
rustfs-log-analyzer = { path = "crates/log-analyzer", version = "1.0.0-beta.10" }
rustfs-obs = { path = "crates/obs", version = "1.0.0-beta.10" }
rustfs-policy = { path = "crates/policy", version = "1.0.0-beta.10" }
rustfs-protos = { path = "crates/protos", version = "1.0.0-beta.10" }
rustfs-protocols = { path = "crates/protocols", version = "1.0.0-beta.10" }
rustfs-replication = { path = "crates/replication", version = "1.0.0-beta.10" }
rustfs-rio = { path = "crates/rio", version = "1.0.0-beta.10" }
rustfs-rio-v2 = { path = "crates/rio-v2", version = "1.0.0-beta.10" }
rustfs-s3-types = { path = "crates/s3-types", version = "1.0.0-beta.10" }
rustfs-s3-ops = { path = "crates/s3-ops", version = "1.0.0-beta.10" }
rustfs-s3select-api = { path = "crates/s3select-api", version = "1.0.0-beta.10" }
rustfs-s3select-query = { path = "crates/s3select-query", version = "1.0.0-beta.10" }
rustfs-scanner = { path = "crates/scanner", version = "1.0.0-beta.10" }
rustfs-security-governance = { path = "crates/security-governance", version = "1.0.0-beta.10" }
rustfs-extension-schema = { path = "crates/extension-schema", version = "1.0.0-beta.10" }
rustfs-signer = { path = "crates/signer", version = "1.0.0-beta.10" }
rustfs-storage-api = { path = "crates/storage-api", version = "1.0.0-beta.10" }
rustfs-trusted-proxies = { path = "crates/trusted-proxies", version = "1.0.0-beta.10" }
rustfs-targets = { path = "crates/targets", version = "1.0.0-beta.10" }
rustfs-test-utils = { path = "crates/test-utils", version = "1.0.0-beta.10" }
rustfs-tls-runtime = { path = "crates/tls-runtime", version = "1.0.0-beta.10" }
rustfs-utils = { path = "crates/utils", version = "1.0.0-beta.10" }
rustfs-zip = { path = "./crates/zip", version = "1.0.0-beta.10" }
# Async Runtime and Networking
async-channel = "2.5.0"
async_zip = { default-features = false, version = "0.0.18" }
mysql_async = { default-features = false, version = "0.37" }
async-compression = { version = "0.4.43" }
async-compression = { version = "0.4.42" }
async-recursion = "1.1.1"
async-trait = "0.1.92"
async-nats = { version = "0.50.0", default-features = false }
async-trait = "0.1.91"
async-nats = "0.49.1"
axum = "0.8.9"
futures = "0.3.34"
futures-core = "0.3.34"
futures = "0.3.33"
futures-core = "0.3.33"
futures-lite = "2.6.1"
futures-util = "0.3.34"
futures-util = "0.3.33"
pollster = "1.0.1"
pulsar = { default-features = false, version = "6.8.0" }
lapin = { default-features = false, version = "4.10.0" }
hyper = { version = "1.11.0" }
hyper = { version = "1.10.1" }
hyper-rustls = { default-features = false, version = "0.27.9" }
hyper-util = { version = "0.1.20" }
http = "1.5.0"
http = "1.4.2"
http-body = "1.1.0"
http-body-util = "0.1.4"
minlz = "1.2.3"
reqwest = "0.13.4"
reqwest = { default-features = false, version = "0.13.4" }
rustfs-kafka-async = { version = "1.2.0" }
socket2 = { version = "0.6.5" }
tokio = { version = "1.53.1" }
tokio = { version = "1.53.0" }
tokio-rustls = { default-features = false, version = "0.26.4" }
tokio-stream = { version = "0.1.19" }
tokio-stream = { version = "0.1.18" }
tokio-test = "0.4.5"
tokio-util = { version = "0.7.19" }
tokio-util = { version = "0.7.18" }
tonic = { version = "0.14.6" }
tonic-prost = { version = "0.14.6" }
tonic-prost-build = { version = "0.14.6" }
@@ -171,9 +171,9 @@ tower = { version = "0.5.3" }
tower-http = { version = "0.7.0" }
# Serialization and Data Formats
apache-avro = "0.22.0"
apache-avro = "0.21.0"
bytes = { version = "1.12.1" }
bytesize = "2.7.0"
bytesize = "2.4.2"
byteorder = "1.5.0"
flatbuffers = "25.12.19"
form_urlencoded = "1.2.2"
@@ -182,7 +182,7 @@ quick-xml = "0.41.0"
rmp = { version = "0.8.15" }
rmp-serde = { version = "1.3.1" }
serde = { version = "1.0.229" }
serde_json = { version = "1.0.151" }
serde_json = { version = "1.0.150" }
serde_urlencoded = "0.7.1"
# Cryptography and Security
@@ -196,13 +196,13 @@ blake2 = "=0.11.0-rc.6"
chacha20poly1305 = { version = "=0.11.0" }
crc-fast = "1.10.0"
hmac = { version = "0.13.0" }
jsonwebtoken = { version = "11.0.0" }
jsonwebtoken = { version = "10.4.0" }
openidconnect = { default-features = false, version = "4.0" }
pbkdf2 = "0.13.0"
rsa = { version = "=0.10.0-rc.18" }
rustls = { default-features = false, version = "0.23.43" }
rustls = { default-features = false, version = "0.23.42" }
rustls-native-certs = "0.8"
rustls-pki-types = "1.15.1"
rustls-pki-types = "1.15.0"
sha1 = "0.11.0"
sha2 = "0.11.0"
subtle = "2.6"
@@ -211,8 +211,8 @@ zeroize = { version = "1.9.0" }
# Time and Date
chrono = { version = "0.4.45" }
humantime = "2.4.0"
jiff = { version = "0.2.35" }
time = { version = "0.3.55" }
jiff = { version = "0.2.34" }
time = { version = "0.3.53" }
# Database
deadpool-postgres = { version = "0.14" }
@@ -225,18 +225,16 @@ arc-swap = "1.9.2"
astral-tokio-tar = "0.6.4"
atoi = "3.1.0"
atomic_enum = "0.3.0"
aws-config = { version = "1.10.1" }
aws-config = { version = "1.9.0" }
aws-credential-types = { version = "1.3.0" }
aws-sdk-kms = { default-features = false, version = "1.114.0" }
aws-sdk-s3 = { default-features = false, version = "1.141.0" }
aws-sdk-sts = { default-features = false, version = "1.110.0" }
aws-sdk-s3 = { default-features = false, version = "1.138.1" }
aws-smithy-http-client = { default-features = false, version = "1.2.0" }
aws-smithy-runtime-api = { version = "1.14.0" }
aws-smithy-runtime-api = { version = "1.13.0" }
aws-smithy-types = { version = "1.6.1" }
base64 = "0.23.1"
base64 = "0.22.1"
base64-simd = "0.8.0"
brotli = "8.0.4"
clap = { version = "4.6.6" }
clap = { version = "4.6.2" }
const-str = { version = "1.1.0" }
convert_case = "0.11.0"
criterion = { version = "0.8" }
@@ -244,57 +242,56 @@ crossbeam-queue = "0.3.13"
crossbeam-channel = "0.5.16"
crossbeam-deque = "0.8.7"
crossbeam-utils = "0.8.22"
datafusion = { default-features = false, git = "https://github.com/apache/datafusion.git", rev = "e08aed1e5de41dcf81d529140dae07723b942a5e" }
#datafusion = { default-features = false, version = "54.1.0" }
datafusion = { default-features = false, git = "https://github.com/apache/datafusion.git", rev = "dae03ee062b2abf986de8df12ea82fb1578a2d99" }
derive_builder = "0.20.2"
enumset = "1.1.14"
enumset = "1.1.13"
faster-hex = "0.10.0"
flate2 = "1.1.9"
glob = "0.3.4"
google-cloud-storage = "1.17.0"
google-cloud-auth = "1.15.0"
glob = "0.3.3"
google-cloud-storage = "1.16.0"
google-cloud-auth = "1.14.0"
hashbrown = { version = "0.17.1" }
hex = "0.4.3"
hex-simd = "0.8.0"
highway = { version = "1.3.0" }
hostname = "0.4.2"
ipnetwork = { version = "0.21.1" }
lazy_static = "1.5.0"
libc = "0.2.189"
libc = "0.2.186"
libsystemd = "0.7.2"
local-ip-address = "0.6.13"
memmap2 = "0.9.11"
lz4 = "1.28.1"
matchit = "0.9.2"
md-5 = "0.11.0"
md5 = "0.8.1"
mime_guess = "2.0.5"
moka = { version = "0.12.16" }
moka = { version = "0.12.15" }
netif = "0.1.6"
num_cpus = { version = "1.17.0" }
nvml-wrapper = "0.12.1"
parking_lot = "0.12.5"
path-absolutize = "4.0.1"
path-clean = "1.0.1"
percent-encoding = "2.3.2"
pin-project-lite = "0.2.17"
pretty_assertions = "1.4.1"
rand = { version = "0.10.2" }
ratelimit = "2.0.0"
ratelimit = "0.10.1"
rayon = "1.12.0"
reed-solomon-erasure = { package = "rustfs-erasure-codec", version = "8.0.2" }
reed-solomon-erasure = { package = "rustfs-erasure-codec", version = "8.0.0" }
reed-solomon-simd = "3.1.0"
regex = { version = "1.13.1" }
rumqttc = { package = "rumqttc-next", version = "0.33.3" }
redis = { version = "1.5.0" }
rustify = { version = "0.7", default-features = false }
rumqttc = { package = "rumqttc-next", version = "0.33.2" }
redis = { version = "1.4.1" }
rustix = { version = "1.1.4" }
rust-embed = { version = "8.12.0" }
rustc-hash = { version = "2.1.3" }
s3s = { git = "https://github.com/rustfs/s3s.git", rev = "d7028511a53f69d41ed3c69f36899f9b1aede647" }
serial_test = "4.0.1"
s3s = { git = "https://github.com/s3s-project/s3s.git", rev = "a5471625975f5014f7b28eee7e4d801f1b32f529" }
serial_test = "3.5.0"
shadow-rs = { default-features = false, version = "2.0.0" }
siphasher = "1.0.3"
smallvec = { version = "1.15.2" }
compact_str = "0.10.0"
smartstring = "1.0.1"
snap = "1.1.2"
starshard = { version = "2.2.2" }
strum = { version = "0.28.0" }
@@ -302,10 +299,9 @@ sysinfo = "0.39.6"
temp-env = "0.3.6"
tempfile = "3.27.0"
test-case = "3.3.1"
thiserror = "2.0.20"
thiserror = "2.0.19"
tracing = { version = "0.1.44" }
tracing-appender = "0.2.5"
tracing-core = "0.1.36"
tracing-error = "0.2.1"
tracing-opentelemetry = { version = "0.33" }
tracing-subscriber = { version = "0.3.23" }
@@ -316,10 +312,8 @@ uuid = { version = "1.24.0" }
vaultrs = { version = "0.8.0" }
tar = "0.4.46"
walkdir = "2.5.0"
winapi-util = "0.1.11"
windows = { version = "0.62.2" }
windows-sys = "0.61.2"
xxhash-rust = { version = "0.8.18" }
xxhash-rust = { version = "0.8.17" }
zip = "8.6.0"
zstd = "0.13.3"
@@ -329,32 +323,30 @@ dial9-tokio-telemetry = "0.3"
opentelemetry = { version = "0.32.0" }
opentelemetry-appender-tracing = { version = "0.32.0" }
opentelemetry-otlp = { version = "0.32.0" }
opentelemetry-proto = { version = "0.32.0", default-features = false, features = ["metrics", "gen-tonic-messages"] }
opentelemetry_sdk = { version = "0.32.1" }
opentelemetry-semantic-conventions = { version = "0.32.1" }
opentelemetry-stdout = { version = "0.32.0" }
pyroscope = { version = "2.1.1" }
pyroscope = { version = "2.1.0" }
# FTP and SFTP
libunftp = { version = "0.23.0" }
unftp-core = "0.1.0"
suppaftp = { version = "10.0.1" }
rcgen = { version = "0.14.9", default-features = false, features = ["aws_lc_rs", "crypto", "pem"] }
russh = { version = "0.62.6" }
russh-sftp = "2.4.0"
rcgen = "0.14.8"
russh = { version = "0.62.2" }
russh-sftp = "2.3.0"
# WebDAV
dav-server = "0.11.0"
# Performance Analysis and Memory Profiling
mimalloc = { version = "0.1.52", git = "https://github.com/xonatius/mimalloc_rust.git", rev = "ce6338661179c8be22e516b00af7483f151485a7" }
libmimalloc-sys = { version = "0.1.49", git = "https://github.com/xonatius/mimalloc_rust.git", rev = "ce6338661179c8be22e516b00af7483f151485a7", features = ["extended"] }
hotpath = { version = "0.23.2", default-features = false }
mimalloc = "0.1.52"
hotpath = "0.21.5"
# Snapshot testing for output format regression detection
insta = { version = "1.48" }
[workspace.metadata.cargo-shear]
ignored = ["hotpath", "rustfs"]
ignored = ["rustfs"]
[profile.dev]
# Full debuginfo roughly doubles compile+link time and produces multi-GB
@@ -378,3 +370,21 @@ inherits = "release"
inherits = "release"
debug = true
strip = "none"
# Pin hyper to a revision that carries the HTTP/1 "flush buffered data before
# shutdown" fix (hyperium/hyper#4018, commit 72046cc7). This lands as a
# `[patch.crates-io]` entry — not on the `hyper` workspace dependency — so that
# every consumer in the tree, including the transitive `hyper-util` server path
# (`conn::auto` / `GracefulShutdown`) that actually drives our connections,
# resolves to the fixed hyper rather than the buggy crates.io copy.
#
# hyper <= 1.10.1 can call `poll_shutdown()` on the socket while response bytes
# are still buffered (a prior `poll_flush()` returned `Poll::Pending` and the
# result was discarded). A backpressured / slow-reading peer then receives a
# graceful FIN before the full Content-Length body is flushed, which standard S3
# clients (minio-go / warp) report as `unexpected EOF` on large-object GET under
# load. The fix is not in any crates.io release yet as of hyper 1.10.1; drop
# this patch once a released version (> 1.10.1) contains commit 72046cc7.
# See rustfs/backlog#1232.
[patch.crates-io]
hyper = { git = "https://github.com/hyperium/hyper.git", rev = "ccc1e850dc0cda3e71b0acd11f60ca3d48d09034" }
+1 -6
View File
@@ -91,12 +91,7 @@ LABEL name="RustFS" \
# Upgrade base-image packages so published images pick up security fixes
# (e.g. openssl/libssl3 CVEs) without waiting for a new Alpine point release.
RUN apk upgrade --no-cache && \
apk add --no-cache \
ca-certificates \
coreutils \
curl \
tzdata \
&& test "$(TZ=Asia/Kolkata date +%z)" = "+0530"
apk add --no-cache ca-certificates coreutils curl
COPY --from=build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
COPY --from=build /build/rustfs /usr/bin/rustfs
+1 -1
View File
@@ -12,7 +12,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.
FROM rust:1.97.1-trixie
FROM rust:1.97-trixie
RUN set -eux; \
export DEBIAN_FRONTEND=noninteractive; \
+1 -3
View File
@@ -96,11 +96,9 @@ LABEL name="RustFS" \
# Upgrade base-image packages so published images pick up security fixes
# (e.g. tar/gzip/perl CVEs) without waiting for a new Ubuntu point release.
RUN apt-get update && apt-get upgrade -y \
&& DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
&& apt-get install -y --no-install-recommends \
ca-certificates \
curl \
tzdata \
&& test "$(TZ=Asia/Kolkata date +%z)" = "+0530" \
&& rm -rf /var/lib/apt/lists/*
COPY --from=build /build/rustfs /usr/bin/rustfs
+1 -1
View File
@@ -32,7 +32,7 @@ ARG RUSTFS_BUILD_FEATURES=""
# -----------------------------
# Build stage
# -----------------------------
FROM rust:1.97.1-trixie AS builder
FROM rust:1.97-trixie AS builder
# Re-declare args after FROM
ARG TARGETPLATFORM
+2 -8
View File
@@ -116,7 +116,7 @@ chown -R 10001:10001 data logs
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest
# Using specific version
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0-rc.1
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0-beta.10
```
If you use [podman](https://github.com/containers/podman) instead of docker, you can install the RustFS with the below command
@@ -163,7 +163,6 @@ docker run -d --name rustfs -p 9000:9000 \
-e RUSTFS_NOTIFY_WEBHOOK_ENABLE_PRIMARY=on \
-e RUSTFS_NOTIFY_WEBHOOK_ENDPOINT_PRIMARY=http://<host-ip>:3020/webhook \
-e RUSTFS_NOTIFY_WEBHOOK_QUEUE_DIR_PRIMARY=/tmp/rustfs-events \
-e RUSTFS_OUTBOUND_ALLOW_ORIGINS=http://<host-ip>:3020 \
rustfs/rustfs:latest
```
@@ -172,11 +171,6 @@ Notes:
- For ARN `arn:rustfs:sqs::primary:webhook`, use instance-scoped env vars with `_PRIMARY`.
- If queue dir is omitted, default is `/opt/rustfs/events`; ensure it is writable by the container runtime user.
- `RUSTFS_NOTIFY_WEBHOOK_SKIP_TLS_VERIFY_PRIMARY` defaults to `false`; enabling it skips webhook TLS certificate verification, allows MITM attacks, and emits a startup warning. Prefer `RUSTFS_NOTIFY_WEBHOOK_CLIENT_CA_PRIMARY` for private CAs.
- Since `1.0.0-beta.11`, webhook endpoints on private or container networks
(`Docker Compose service names`, `host.docker.internal`, RFC 1918 addresses) are
blocked unless their exact `scheme://host:port` origin is listed in
`RUSTFS_OUTBOUND_ALLOW_ORIGINS` (the origin only, without the path). See
[Outbound Connection Policy](docs/operations/outbound-connection-policy.md).
**NOTE**: We recommend reviewing the `docker-compose.yml` file before running. It defines several services including Grafana, Prometheus, and Jaeger, which are helpful for RustFS observability. If you wish to start Redis or Nginx containers, you can specify the corresponding profiles.
@@ -268,7 +262,7 @@ rustfs --help
2. **Create a Bucket**: Use the console to create a new bucket for your objects.
3. **Upload Objects**: You can upload files directly through the console or use S3-compatible APIs/clients to interact with your RustFS instance.
**NOTE**: To access the RustFS instance via `https`, please refer to the [TLS Configuration Docs](https://docs.rustfs.com/integration/tls-configured).
**NOTE**: To access the RustFS instance via `https`, please refer to the [TLS Configuration Docs](https://docs.rustfs.com/integration/tls-configured.html).
### OIDC Roles Claim (Microsoft Entra ID)
+2 -2
View File
@@ -113,7 +113,7 @@ chown -R 10001:10001 data logs
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest
# 使用指定版本运行
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0-rc.1
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0-beta.10
```
如果您通过绑定挂载启用 TLS 证书目录,也请用同样方式准备该目录:
@@ -214,7 +214,7 @@ rustfs --help
2. **创建存储桶**: 使用控制台为您​​的对象创建一个新的存储桶 (Bucket)。
3. **上传对象**: 您可以直接通过控制台上传文件,或使用 S3 兼容的 API/客户端与您的 RustFS 实例进行交互。
**注意**: 如果您希望通过 `https` 访问 RustFS 实例,请参考 [TLS 配置文档](https://docs.rustfs.com/integration/tls-configured)。
**注意**: 如果您希望通过 `https` 访问 RustFS 实例,请参考 [TLS 配置文档](https://docs.rustfs.com/integration/tls-configured.html)。
## 文档
-3
View File
@@ -47,9 +47,6 @@ consts = "consts"
Hashi = "Hashi" # HashiCorp
# Accept alternate spelling used in parser/XML comments.
unparseable = "unparseable"
# Disaster-recovery objectives: recovery time and recovery point.
RTO = "RTO"
rto = "rto"
[files]
extend-exclude = []
+1 -27
View File
@@ -25,40 +25,14 @@ documentation = "https://docs.rs/rustfs-audit/latest/rustfs_audit/"
keywords = ["audit", "target", "management", "fan-out", "RustFS"]
categories = ["web-programming", "development-tools", "asynchronous", "api-bindings"]
[features]
default = []
hotpath = [
"hotpath/hotpath",
"hotpath/tokio",
"hotpath/futures",
"rustfs-config/hotpath",
"rustfs-s3-types/hotpath",
"rustfs-targets/hotpath",
]
hotpath-alloc = [
"hotpath",
"hotpath/hotpath-alloc",
"rustfs-config/hotpath-alloc",
"rustfs-s3-types/hotpath-alloc",
"rustfs-targets/hotpath-alloc",
]
hotpath-cpu = [
"hotpath",
"hotpath/hotpath-cpu",
"rustfs-config/hotpath-cpu",
"rustfs-s3-types/hotpath-cpu",
"rustfs-targets/hotpath-cpu",
]
[dependencies]
hotpath.workspace = true
rustfs-targets = { workspace = true }
rustfs-config = { workspace = true, features = ["audit", "server-config-model"] }
rustfs-s3-types = { workspace = true }
chrono = { workspace = true, features = ["serde"] }
const-str = { workspace = true, features = ["std", "proc"] }
futures = { workspace = true }
hashbrown = { workspace = true, features = ["serde", "rayon"] }
jiff = { workspace = true, features = ["serde"] }
metrics = { workspace = true }
serde = { workspace = true, features = ["derive"] }
serde_json = { workspace = true, features = ["raw_value"] }
+5 -24
View File
@@ -12,8 +12,8 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use chrono::{DateTime, Utc};
use hashbrown::HashMap;
use jiff::Timestamp;
use rustfs_s3_types::EventName;
use serde::{Deserialize, Serialize};
use serde_json::Value;
@@ -151,8 +151,8 @@ pub struct AuditEntry {
pub deployment_id: Option<String>,
#[serde(rename = "siteName", skip_serializing_if = "Option::is_none")]
pub site_name: Option<String>,
#[serde(with = "jiff::fmt::serde::timestamp::millisecond::required")]
pub time: Timestamp,
#[serde(with = "chrono::serde::ts_milliseconds")]
pub time: DateTime<Utc>,
pub event: EventName,
#[serde(rename = "type", skip_serializing_if = "Option::is_none")]
pub entry_type: Option<String>,
@@ -198,7 +198,7 @@ impl AuditEntryBuilder {
pub fn new(version: impl Into<String>, event: EventName, trigger: impl Into<String>, api: ApiDetails) -> Self {
Self(AuditEntry {
version: version.into(),
time: Timestamp::now(),
time: Utc::now(),
event,
trigger: trigger.into(),
api,
@@ -232,7 +232,7 @@ impl AuditEntryBuilder {
self
}
pub fn time(mut self, time: Timestamp) -> Self {
pub fn time(mut self, time: DateTime<Utc>) -> Self {
self.0.time = time;
self
}
@@ -342,23 +342,4 @@ mod tests {
assert_eq!(value["requestID"], Value::String("req-audit-123".to_string()));
assert!(value.get("request_id").is_none(), "historical audit contract must not expose request_id");
}
#[test]
fn audit_entry_time_serializes_as_epoch_milliseconds() {
let entry = AuditEntryBuilder::new(
"1",
EventName::ObjectCreatedPut,
"s3",
ApiDetailsBuilder::new()
.name("PutObject")
.status("OK")
.status_code(200)
.build(),
)
.time(Timestamp::from_millisecond(1_711_423_698_870).expect("timestamp should be valid"))
.build();
let value = serde_json::to_value(entry).expect("audit entry should serialize");
assert_eq!(value["time"], Value::Number(1_711_423_698_870_i64.into()));
}
}
+3 -32
View File
@@ -292,8 +292,8 @@ impl AuditPipeline {
}
pub async fn snapshot_target_health(&self) -> Vec<rustfs_targets::RuntimeTargetHealthSnapshot> {
let targets = self.registry.lock().await.list_target_values();
rustfs_targets::health_snapshots_for_targets(targets).await
let registry = self.registry.lock().await;
registry.runtime_manager().health_snapshots().await
}
}
@@ -570,7 +570,7 @@ mod tests {
use rustfs_targets::target::{EntityTarget, QueuedPayload, QueuedPayloadMeta};
use rustfs_targets::{StoreError, Target, TargetError};
use std::sync::Arc;
use tokio::sync::{Mutex, Notify};
use tokio::sync::Mutex;
/// Mock target whose `save()` outcome is fixed at construction so tests can
/// force full-success / full-failure / partial-failure fan-outs.
@@ -578,7 +578,6 @@ mod tests {
struct MockTarget {
id: TargetID,
fail: bool,
health_gate: Option<(Arc<Notify>, Arc<Notify>)>,
}
impl MockTarget {
@@ -586,14 +585,8 @@ mod tests {
Self {
id: TargetID::new(id.to_string(), "webhook".to_string()),
fail,
health_gate: None,
}
}
fn with_health_gate(mut self, started: Arc<Notify>, release: Arc<Notify>) -> Self {
self.health_gate = Some((started, release));
self
}
}
#[async_trait]
@@ -606,10 +599,6 @@ mod tests {
}
async fn is_active(&self) -> Result<bool, TargetError> {
if let Some((started, release)) = &self.health_gate {
started.notify_one();
release.notified().await;
}
Ok(true)
}
@@ -684,24 +673,6 @@ mod tests {
pipeline.dispatch(entry()).await.expect("no targets should return Ok");
}
#[tokio::test]
async fn health_probe_does_not_hold_the_registry_lock() {
let started = Arc::new(Notify::new());
let release = Arc::new(Notify::new());
let pipeline = pipeline_with(vec![MockTarget::new("blocked", false).with_health_gate(started.clone(), release.clone())]);
let registry = Arc::clone(&pipeline.registry);
let snapshot_task = tokio::spawn(async move { pipeline.snapshot_target_health().await });
started.notified().await;
let guard = tokio::time::timeout(std::time::Duration::from_secs(1), registry.lock())
.await
.expect("network health probe must not retain the audit registry lock");
drop(guard);
release.notify_one();
assert_eq!(snapshot_task.await.expect("snapshot task should finish").len(), 1);
}
// backlog#962: dispatch_batch must mirror dispatch and propagate a
// whole-batch loss instead of returning Ok.
#[tokio::test]
+3 -3
View File
@@ -97,7 +97,7 @@ async fn test_audit_log_dispatch_performance() {
return; // Alternatively: assert!(false, "AuditSystem failed to start");
}
use jiff::Timestamp;
use chrono::Utc;
use rustfs_targets::EventName;
use serde_json::json;
use std::collections::HashMap;
@@ -136,7 +136,7 @@ async fn test_audit_log_dispatch_performance() {
version: "1".to_string(),
deployment_id: Some(format!("test-deployment-{id}")),
site_name: Some("test-site".to_string()),
time: Timestamp::now(),
time: Utc::now(),
event: EventName::ObjectCreatedPut,
entry_type: Some("object".to_string()),
trigger: "api".to_string(),
@@ -298,7 +298,7 @@ fn test_performance_requirements() {
for i in 0..3000 {
// Simulate event name parsing and processing
let _event_id = format!("s3:ObjectCreated:Put_{i}");
let _timestamp = jiff::Timestamp::now().to_string();
let _timestamp = chrono::Utc::now().to_rfc3339();
// Simulate basic audit entry creation overhead
let _entry_size = 512; // bytes
@@ -264,7 +264,7 @@ fn create_sample_audit_entry() -> AuditEntry {
}
fn create_sample_audit_entry_with_id(id: u32) -> AuditEntry {
use jiff::Timestamp;
use chrono::Utc;
use rustfs_targets::EventName;
use serde_json::json;
@@ -301,7 +301,7 @@ fn create_sample_audit_entry_with_id(id: u32) -> AuditEntry {
version: "1".to_string(),
deployment_id: Some(format!("test-deployment-{id}")),
site_name: Some("test-site".to_string()),
time: Timestamp::now(),
time: Utc::now(),
event: EventName::ObjectCreatedPut,
entry_type: Some("object".to_string()),
trigger: "api".to_string(),
-10
View File
@@ -25,17 +25,7 @@ keywords = ["checksum-calculation", "verification", "integrity", "authenticity",
categories = ["web-programming", "development-tools", "network-programming"]
documentation = "https://docs.rs/rustfs-checksums/latest/rustfs_checksum/"
[lints]
workspace = true
[features]
default = []
hotpath = ["hotpath/hotpath"]
hotpath-alloc = ["hotpath", "hotpath/hotpath-alloc"]
hotpath-cpu = ["hotpath", "hotpath/hotpath-cpu"]
[dependencies]
hotpath.workspace = true
bytes = { workspace = true, features = ["serde"] }
crc-fast = { workspace = true }
http = { workspace = true }
-11
View File
@@ -27,27 +27,16 @@ categories = ["web-programming", "development-tools", "data-structures"]
[lints]
workspace = true
[features]
default = []
hotpath = ["hotpath/hotpath", "hotpath/tokio"]
hotpath-alloc = ["hotpath", "hotpath/hotpath-alloc"]
hotpath-cpu = ["hotpath", "hotpath/hotpath-cpu"]
[dependencies]
hotpath.workspace = true
tokio = { workspace = true, features = ["fs", "rt-multi-thread"] }
tonic = { workspace = true, features = ["gzip", "deflate"] }
uuid = { workspace = true, features = ["v4", "fast-rng", "macro-diagnostics"] }
chrono = { workspace = true, features = ["serde"] }
jiff = { workspace = true, features = ["serde"] }
metrics = { workspace = true }
serde = { workspace = true, features = ["derive"] }
rmp-serde = { workspace = true }
s3s = { workspace = true, features = ["minio"] }
tracing = { workspace = true }
[dev-dependencies]
serde_json = { workspace = true }
[lib]
doctest = false
-4
View File
@@ -356,8 +356,6 @@ pub struct HealChannelRequest {
pub recursive: Option<bool>,
/// Whether to dry run
pub dry_run: Option<bool>,
/// Whether to skip namespace locking
pub no_lock: Option<bool>,
/// Timeout in seconds (optional)
pub timeout_seconds: Option<u64>,
/// Origin of the request for operational status and queue accounting
@@ -562,7 +560,6 @@ pub fn create_heal_request(
update_parity: None,
recursive: None,
dry_run: None,
no_lock: None,
timeout_seconds: None,
source: HealRequestSource::Internal,
disk: None,
@@ -721,7 +718,6 @@ pub async fn send_heal_disk(set_disk_id: String, priority: Option<HealChannelPri
update_parity: None,
recursive: None,
dry_run: None,
no_lock: None,
timeout_seconds: None,
source: HealRequestSource::AutoHeal,
};
-1
View File
@@ -19,7 +19,6 @@ pub mod heal_channel;
pub mod last_minute;
pub mod metrics;
mod readiness;
pub mod table_catalog;
pub use globals::*;
pub use readiness::{GlobalReadiness, SystemStage};
File diff suppressed because it is too large Load Diff
-17
View File
@@ -1,17 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
/// Cross-crate lock identity used to fence table-bucket publication against
/// object mutations that bypass the S3 request authorization layer.
pub const TABLE_BUCKET_PUBLICATION_LOCK_PATH: &str = ".rustfs-table/warehouses/default/publication.lock";
-10
View File
@@ -10,17 +10,7 @@ description = "Shared concurrency contract types for RustFS - workload admission
keywords = ["rustfs", "concurrency", "admission", "backpressure", "workers"]
categories = ["concurrency", "filesystem"]
[lints]
workspace = true
[features]
default = []
hotpath = ["hotpath/hotpath", "hotpath/tokio", "rustfs-io-core/hotpath"]
hotpath-alloc = ["hotpath", "hotpath/hotpath-alloc", "rustfs-io-core/hotpath-alloc"]
hotpath-cpu = ["hotpath", "hotpath/hotpath-cpu", "rustfs-io-core/hotpath-cpu"]
[dependencies]
hotpath.workspace = true
# Internal crates
rustfs-io-core = { workspace = true }
serde = { workspace = true, features = ["derive"] }
-4
View File
@@ -25,7 +25,6 @@ keywords = ["configuration", "settings", "management", "rustfs", "Minio"]
categories = ["web-programming", "development-tools", "config"]
[dependencies]
hotpath.workspace = true
const-str = { workspace = true, optional = true, features = ["std", "proc"] }
serde = { workspace = true, optional = true, features = ["derive"] }
serde_json = { workspace = true, optional = true, features = ["raw_value"] }
@@ -35,9 +34,6 @@ workspace = true
[features]
default = ["constants"]
hotpath = ["hotpath/hotpath"]
hotpath-alloc = ["hotpath", "hotpath/hotpath-alloc"]
hotpath-cpu = ["hotpath", "hotpath/hotpath-cpu"]
audit = ["dep:const-str", "constants"]
constants = ["dep:const-str"]
notify = ["dep:const-str", "constants"]
-12
View File
@@ -66,10 +66,6 @@ Current guidance:
- `RUSTFS_BROWSER_REDIRECT_URL` sets the externally reachable browser origin used for OIDC callback, console success redirect, and logout fallback URLs. Configure it to the public scheme and authority without a path, for example `https://console.example.com`. In load-balancer deployments, keep OIDC authorize and callback requests on the same backend node because the in-flight OIDC `state` is local to the RustFS node.
## Distributed endpoint locality
- `RUSTFS_LOCAL_ENDPOINT_HOST` identifies this server's host in a distributed `RUSTFS_VOLUMES` topology without resolving every peer during startup. Set it to exactly one host, without a scheme, port, or path. It is accepted only for orchestrated URL topologies and must match at least one endpoint on the RustFS server port; invalid or unmatched values fail startup. Leave it unset to retain DNS-based locality discovery.
## Scanner environment aliases
- `RUSTFS_SCANNER_SPEED` (canonical, also accepts `MINIO_SCANNER_SPEED`)
@@ -97,14 +93,6 @@ Current guidance:
- enables minimal payload mode for GET health responses (`status`, `ready` only).
- `RUSTFS_HEALTH_READINESS_CACHE_TTL_MS`
- TTL for readiness cache evaluation.
- `RUSTFS_HEALTH_OBJECT_PROGRESS_ENABLE`
- withdraws readiness when bounded object read/write stages stop completing while requests remain active.
- default is `true`.
- `RUSTFS_HEALTH_OBJECT_PROGRESS_TIMEOUT_MS`
- maximum time without completion in a bounded object stage before readiness is withdrawn.
- default is `30000`; `0` uses the default.
- the effective value is at least 5 seconds longer than `RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT`.
- this readiness SLO is independent of disk read/write failure deadlines and may withdraw traffic before those deadlines expire.
- `RUSTFS_HEALTH_COMPAT_BUSY_CHECK_ENABLE`
- enables busy protection behavior for health probes.
- default is `false`.
-17
View File
@@ -131,10 +131,6 @@ pub const ENV_RUSTFS_ADDRESS: &str = "RUSTFS_ADDRESS";
/// Environment variable for server volumes.
pub const ENV_RUSTFS_VOLUMES: &str = "RUSTFS_VOLUMES";
/// Environment variable identifying this server's host in distributed endpoint
/// lists without relying on DNS locality discovery.
pub const ENV_LOCAL_ENDPOINT_HOST: &str = "RUSTFS_LOCAL_ENDPOINT_HOST";
/// Environment variable to explicitly bypass local physical disk independence checks.
pub const ENV_UNSAFE_BYPASS_DISK_CHECK: &str = "RUSTFS_UNSAFE_BYPASS_DISK_CHECK";
@@ -230,19 +226,6 @@ pub const ENV_RUSTFS_KMS_ENABLE: &str = "RUSTFS_KMS_ENABLE";
/// Default value: false
pub const DEFAULT_KMS_ENABLE: bool = false;
/// Environment variable enabling per-key KMS authorization on the SSE-KMS data path.
///
/// When enabled, an SSE-KMS write additionally requires `kms:GenerateDataKey` and an
/// SSE-KMS read additionally requires `kms:Decrypt` on the resolved key, evaluated as
/// the requesting identity. SSE-S3 and SSE-C are unaffected.
pub const ENV_RUSTFS_KMS_ENFORCE_SSE_KEY_POLICY: &str = "RUSTFS_KMS_ENFORCE_SSE_KEY_POLICY";
/// Default per-key KMS authorization mode for the SSE-KMS data path.
///
/// Off for now so deployments whose identity policies only grant s3 actions keep
/// working; the default flips to on in a later release.
pub const DEFAULT_KMS_ENFORCE_SSE_KEY_POLICY: bool = false;
/// Environment variable for server KMS backend.
pub const ENV_RUSTFS_KMS_BACKEND: &str = "RUSTFS_KMS_BACKEND";
@@ -28,15 +28,6 @@ pub const MAX_ADMIN_REQUEST_BODY_SIZE: usize = 1024 * 1024; // 1 MB
/// Rationale: ZIP archives with hundreds of IAM entities. 10MB allows ~10,000 small configs.
pub const MAX_IAM_IMPORT_SIZE: usize = 10 * 1024 * 1024; // 10 MB
/// Maximum total size the members of an IAM import ZIP may expand to (100 MB).
/// Used for: bounding decompression of `ImportIam` archive members.
/// Rationale: `MAX_IAM_IMPORT_SIZE` caps the *compressed* upload only. Deflate
/// reaches ratios far above 100:1, so without a separate budget a 10 MB archive
/// can expand without bound. 100 MB keeps a 10x headroom over the compressed cap
/// — ample for legitimate IAM exports, which are small JSON documents — while
/// keeping the worst case bounded.
pub const MAX_IAM_IMPORT_EXPANDED_SIZE: u64 = 100 * 1024 * 1024; // 100 MB
/// Maximum size for bucket metadata import operations (100 MB)
/// Used for: Bucket metadata import containing configurations for many buckets
/// Rationale: Large deployments may have thousands of buckets with various configs.
@@ -63,12 +54,3 @@ pub const MAX_HEAL_REQUEST_SIZE: usize = 1024 * 1024; // 1 MB
/// 10MB provides generous headroom for legitimate responses while preventing
/// memory exhaustion from malicious or misconfigured remote services.
pub const MAX_S3_CLIENT_RESPONSE_SIZE: usize = 10 * 1024 * 1024; // 10 MB
/// Maximum size for OIDC provider response bodies (1 MB)
/// Used for: discovery documents, JWKS documents and token endpoint responses
/// Rationale: a hostile or compromised identity provider must not be able to exhaust
/// memory through an arbitrarily large or endless response body.
/// - Discovery documents: typically < 10KB
/// - JWKS documents: typically < 50KB
/// - Token responses: typically < 10KB
pub const MAX_OIDC_RESPONSE_SIZE: usize = 1024 * 1024; // 1 MB
-5
View File
@@ -39,11 +39,6 @@ pub const DEFAULT_DRIVE_WALKDIR_TIMEOUT_SECS: u64 = 5;
pub const ENV_DRIVE_WALKDIR_STALL_TIMEOUT_SECS: &str = "RUSTFS_DRIVE_WALKDIR_STALL_TIMEOUT_SECS";
pub const DEFAULT_DRIVE_WALKDIR_STALL_TIMEOUT_SECS: u64 = 5;
/// Maximum time the metacache merge consumer waits for the next visible
/// `walk_dir()` entry from a reader before detaching it from the merge.
pub const ENV_DRIVE_WALKDIR_PEEK_TIMEOUT_SECS: &str = "RUSTFS_DRIVE_WALKDIR_PEEK_TIMEOUT_SECS";
pub const DEFAULT_DRIVE_WALKDIR_PEEK_TIMEOUT_SECS: u64 = 10;
/// Interval in seconds between active health probes for local and remote drives.
pub const ENV_DRIVE_ACTIVE_CHECK_INTERVAL_SECS: &str = "RUSTFS_DRIVE_ACTIVE_CHECK_INTERVAL_SECS";
pub const DEFAULT_DRIVE_ACTIVE_CHECK_INTERVAL_SECS: u64 = 15;
-13
View File
@@ -22,19 +22,6 @@ pub const DEFAULT_HEALTH_ENDPOINT_ENABLE: bool = true;
pub const ENV_HEALTH_READINESS_CACHE_TTL_MS: &str = "RUSTFS_HEALTH_READINESS_CACHE_TTL_MS";
pub const DEFAULT_HEALTH_READINESS_CACHE_TTL_MS: u64 = 1000;
/// Enable readiness withdrawal when bounded object read/write stages stop
/// completing while requests remain active.
pub const ENV_HEALTH_OBJECT_PROGRESS_ENABLE: &str = "RUSTFS_HEALTH_OBJECT_PROGRESS_ENABLE";
pub const DEFAULT_HEALTH_OBJECT_PROGRESS_ENABLE: bool = true;
/// Requested time without completion in a bounded object stage before local
/// readiness is withdrawn (milliseconds). A value of `0` uses the default;
/// runtime adds a safety floor based on the object-lock acquisition timeout.
pub const ENV_HEALTH_OBJECT_PROGRESS_TIMEOUT_MS: &str = "RUSTFS_HEALTH_OBJECT_PROGRESS_TIMEOUT_MS";
pub const DEFAULT_HEALTH_OBJECT_PROGRESS_TIMEOUT_MS: u64 = 30_000;
/// Additional time beyond the configured object-lock acquisition deadline.
pub const HEALTH_OBJECT_PROGRESS_LOCK_MARGIN_MS: u64 = 5_000;
/// Timeout for cluster health readiness collectors (milliseconds).
/// This bounds expensive storage and lock quorum checks used by cluster probes.
pub const ENV_HEALTH_CLUSTER_TIMEOUT_MS: &str = "RUSTFS_HEALTH_CLUSTER_TIMEOUT_MS";
+7 -111
View File
@@ -97,95 +97,19 @@ pub const ENV_INTERNODE_RPC_MAX_MESSAGE_SIZE: &str = "RUSTFS_INTERNODE_RPC_MAX_M
pub const ENV_INTERNODE_RPC_LARGE_PAYLOAD_WARN_BYTES: &str = "RUSTFS_INTERNODE_RPC_LARGE_PAYLOAD_WARN_BYTES";
pub const DEFAULT_INTERNODE_RPC_LARGE_PAYLOAD_WARN_BYTES: usize = 8 * 1024 * 1024;
/// Request stopping the JSON compatibility strings on internode metadata RPCs and sending only the
/// Stop dual-writing the JSON compatibility strings on internode metadata RPCs and send only the
/// msgpack `_bin` payloads (grpc-optimization P2-1).
///
/// Defaults to `false` (dual-write, byte-for-byte legacy behavior). This is only a request; RustFS
/// keeps JSON compatibility fields unless [`ENV_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED`] is also
/// true after the release-window convergence and rollback gates pass. See
/// Defaults to `false` (dual-write, byte-for-byte legacy behavior). This is a rollout lever, not a
/// wire-format change: it may only be enabled **after** the JSON-fallback counter
/// (`rustfs_system_network_internode_msgpack_json_fallback_total`) has read zero across a release
/// window fleet-wide, confirming every peer decodes `_bin` first. Single-env rollback. See
/// `docs/operations/internode-msgpack-json-convergence-runbook.md`.
pub const ENV_INTERNODE_RPC_MSGPACK_ONLY: &str = "RUSTFS_INTERNODE_RPC_MSGPACK_ONLY";
pub const DEFAULT_INTERNODE_RPC_MSGPACK_ONLY: bool = false;
/// Explicit fleet-wide confirmation gate for [`ENV_INTERNODE_RPC_MSGPACK_ONLY`].
///
/// This separate default-off guard prevents a single legacy flag from accidentally emptying JSON
/// fields in a mixed-version fleet where an older peer still reads the JSON field.
pub const ENV_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED: &str = "RUSTFS_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED";
pub const DEFAULT_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED: bool = false;
// Compile-time invariants: dual-write by default so the base build is byte-for-byte legacy behavior.
// Compile-time invariant: dual-write by default so the base build is byte-for-byte legacy behavior.
const _: () = assert!(!DEFAULT_INTERNODE_RPC_MSGPACK_ONLY);
const _: () = assert!(!DEFAULT_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED);
/// Require target-bound v2 signatures on every internode gRPC request, rejecting the legacy
/// constant-target fallback instead of accepting it (<https://github.com/rustfs/backlog/issues/1327>).
///
/// Defaults to `false` (fail-open): a request without any v2 auth headers keeps authenticating
/// through the legacy signature, so legacy-only peers survive rolling upgrades with byte-for-byte
/// the pre-gate acceptance behavior. This is a rollout lever, not a wire-format change: it may only
/// be enabled **after** the v1-fallback counter
/// (`rustfs_system_network_internode_signature_v1_fallback_total`) has read zero across a release
/// window fleet-wide, confirming every peer already sends v2 authentication on every internode gRPC
/// request. Single-env rollback. Requests that do carry v2 headers are unaffected by this switch:
/// they are always verified as v2 with no downgrade, strict or not.
pub const ENV_INTERNODE_RPC_SIGNATURE_STRICT: &str = "RUSTFS_INTERNODE_RPC_SIGNATURE_STRICT";
pub const DEFAULT_INTERNODE_RPC_SIGNATURE_STRICT: bool = false;
// Compile-time invariant: fail-open by default so legacy-only peers keep authenticating during
// rolling upgrades until the fleet-wide v1-fallback counter reads zero.
const _: () = assert!(!DEFAULT_INTERNODE_RPC_SIGNATURE_STRICT);
/// Require a signature-bound canonical body digest on every mutating internode disk RPC
/// (RenameData, DeleteVersion, DeleteVersions, WriteMetadata, UpdateMetadata, WriteAll, Delete,
/// DeletePaths, RenameFile, RenamePart, DeleteVolume, MakeVolume, MakeVolumes), rejecting requests
/// that authenticate without one (<https://github.com/rustfs/backlog/issues/1327>).
///
/// Defaults to `false` (fail-open): a mutating request without a body digest keeps authenticating
/// through the method-bound v2 (or legacy) signature, so peers from releases that predate
/// body-digest signing survive rolling upgrades unchanged. Requests that do carry a digest are
/// always verified with no downgrade, strict or not — the digest value is part of the signed v2
/// scope, so an on-path attacker cannot strip it without invalidating the signature. This is a
/// rollout lever gated on the body-digest fallback counter
/// (`rustfs_system_network_internode_body_digest_fallback_total`) reading zero across a release
/// window fleet-wide. Single-env rollback. It is deliberately separate from
/// [`ENV_INTERNODE_RPC_SIGNATURE_STRICT`]: the two enforcement flips converge on different
/// counters and must not gate each other.
pub const ENV_INTERNODE_RPC_BODY_DIGEST_STRICT: &str = "RUSTFS_INTERNODE_RPC_BODY_DIGEST_STRICT";
pub const DEFAULT_INTERNODE_RPC_BODY_DIGEST_STRICT: bool = false;
// Compile-time invariant: fail-open by default so digestless peers keep authenticating during
// rolling upgrades until the fleet-wide body-digest fallback counter reads zero.
const _: () = assert!(!DEFAULT_INTERNODE_RPC_BODY_DIGEST_STRICT);
/// Require the replay-scoped internode RPC signature after the fleet has converged on it.
///
/// The default keeps v1/v2 peers available during a rolling upgrade. Operators may set this only
/// after `rustfs_system_network_internode_replay_scope_fallback_total` remains zero for a full
/// release window. The node still accepts a v2-authenticated `Ping` carrying an epoch challenge:
/// that narrowly scoped bootstrap lets an upgraded client learn the receiving process epoch and
/// immediately retry with the replay-scoped signature after a peer restart.
pub const ENV_INTERNODE_RPC_REPLAY_SCOPE_STRICT: &str = "RUSTFS_INTERNODE_RPC_REPLAY_SCOPE_STRICT";
pub const DEFAULT_INTERNODE_RPC_REPLAY_SCOPE_STRICT: bool = false;
// Compile-time invariant: mixed-version clusters must remain available until operators make the
// observed fallback counter an explicit strictness decision.
const _: () = assert!(!DEFAULT_INTERNODE_RPC_REPLAY_SCOPE_STRICT);
/// Capacity (distinct nonces) of the process-local internode RPC replay cache that enforces
/// one-time consumption of authenticated RPC signatures.
///
/// The cache retains each nonce for the ~10-minute signature freshness envelope. Once peers use
/// replay-scoped v3 authentication, every authenticated RPC consumes one entry, so the steady
/// state holds roughly `authenticated RPC RPS x 601s` entries. This default is the minimum floor:
/// explicit operator values and resource-aware auto sizing both clamp upward to at least this
/// value. Overflow fails closed — legitimate signed traffic is the only thing that can fill the
/// cache (replays are rejected before insertion, and an attacker cannot mint valid nonces without
/// the shared secret) — and increments
/// `rustfs_system_network_internode_replay_cache_overflow_total`, so a sustained non-zero overflow
/// counter means this capacity is undersized for the node's peak authenticated RPC rate.
pub const ENV_INTERNODE_RPC_REPLAY_CACHE_CAPACITY: &str = "RUSTFS_INTERNODE_RPC_REPLAY_CACHE_CAPACITY";
pub const DEFAULT_INTERNODE_RPC_REPLAY_CACHE_CAPACITY: usize = 1_048_576;
/// Consecutive-failure threshold after which an internode peer is marked offline (grpc-optimization
/// P3 observability).
@@ -349,36 +273,8 @@ mod tests {
#[test]
fn internode_msgpack_only_env_name_is_stable() {
// The dual-write-by-default invariants are asserted at compile time next to the definitions.
// The dual-write-by-default invariant is asserted at compile time next to the definition.
assert_eq!(ENV_INTERNODE_RPC_MSGPACK_ONLY, "RUSTFS_INTERNODE_RPC_MSGPACK_ONLY");
assert_eq!(
ENV_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED,
"RUSTFS_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED"
);
}
#[test]
fn internode_signature_strict_env_name_is_stable() {
// The fail-open default invariant is asserted at compile time next to the definition.
assert_eq!(ENV_INTERNODE_RPC_SIGNATURE_STRICT, "RUSTFS_INTERNODE_RPC_SIGNATURE_STRICT");
}
#[test]
fn internode_body_digest_strict_env_name_is_stable() {
// The fail-open default invariant is asserted at compile time next to the definition.
assert_eq!(ENV_INTERNODE_RPC_BODY_DIGEST_STRICT, "RUSTFS_INTERNODE_RPC_BODY_DIGEST_STRICT");
}
#[test]
fn internode_replay_scope_strict_env_name_is_stable() {
// The fail-open default invariant is asserted at compile time next to the definition.
assert_eq!(ENV_INTERNODE_RPC_REPLAY_SCOPE_STRICT, "RUSTFS_INTERNODE_RPC_REPLAY_SCOPE_STRICT");
}
#[test]
fn internode_replay_cache_capacity_defaults_and_env_name() {
assert_eq!(ENV_INTERNODE_RPC_REPLAY_CACHE_CAPACITY, "RUSTFS_INTERNODE_RPC_REPLAY_CACHE_CAPACITY");
assert_eq!(DEFAULT_INTERNODE_RPC_REPLAY_CACHE_CAPACITY, 1_048_576);
}
#[test]
-33
View File
@@ -116,27 +116,6 @@ pub const ENV_OBJECT_GET_SKIP_BITROT_VERIFY: &str = "RUSTFS_OBJECT_GET_SKIP_BITR
/// Default: bitrot verification is enabled on GetObject reads (do not skip).
pub const DEFAULT_OBJECT_GET_SKIP_BITROT_VERIFY: bool = false;
/// Request writing the complete remote-tier version state into object metadata.
///
/// This remains ineffective until
/// [`ENV_TIER_REMOTE_VERSION_STATE_FLEET_CONFIRMED`] is also enabled.
pub const ENV_TIER_REMOTE_VERSION_STATE_WRITE: &str = "RUSTFS_TIER_REMOTE_VERSION_STATE_WRITE";
pub const DEFAULT_TIER_REMOTE_VERSION_STATE_WRITE: bool = false;
/// Operator-attested fleet-wide confirmation for
/// [`ENV_TIER_REMOTE_VERSION_STATE_WRITE`].
///
/// This flag is an operational contract, not automatic capability discovery.
/// Operators may enable it only after every node that can write or read
/// transitioned object metadata supports the remote version-state schema and
/// semantics. Keeping the confirmation separate makes a single-node request or
/// a writer whose local opt-in is removed fail closed.
pub const ENV_TIER_REMOTE_VERSION_STATE_FLEET_CONFIRMED: &str = "RUSTFS_TIER_REMOTE_VERSION_STATE_FLEET_CONFIRMED";
pub const DEFAULT_TIER_REMOTE_VERSION_STATE_FLEET_CONFIRMED: bool = false;
const _: () = assert!(!DEFAULT_TIER_REMOTE_VERSION_STATE_WRITE);
const _: () = assert!(!DEFAULT_TIER_REMOTE_VERSION_STATE_FLEET_CONFIRMED);
// =============================================================================
// Concurrent Request Fix - Timeout and Backpressure Configuration
// =============================================================================
@@ -638,15 +617,3 @@ pub const ENV_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY: &str = "RUSTFS_OBJ
/// Default read-ahead disable concurrency threshold: 4.
pub const DEFAULT_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY: usize = 4;
#[cfg(test)]
mod remote_version_state_tests {
#[test]
fn remote_version_state_gate_uses_stable_environment_names() {
assert_eq!(super::ENV_TIER_REMOTE_VERSION_STATE_WRITE, "RUSTFS_TIER_REMOTE_VERSION_STATE_WRITE");
assert_eq!(
super::ENV_TIER_REMOTE_VERSION_STATE_FLEET_CONFIRMED,
"RUSTFS_TIER_REMOTE_VERSION_STATE_FLEET_CONFIRMED"
);
}
}
+1 -5
View File
@@ -14,7 +14,6 @@
// OIDC configuration field keys (used in KVS)
pub const OIDC_CONFIG_URL: &str = "config_url";
pub const OIDC_ISSUER: &str = "issuer";
pub const OIDC_CLIENT_ID: &str = "client_id";
pub const OIDC_CLIENT_SECRET: &str = "client_secret";
pub const OIDC_SCOPES: &str = "scopes";
@@ -34,7 +33,6 @@ pub const OIDC_HIDE_FROM_UI: &str = "hide_from_ui";
// Environment variable names for OIDC
pub const ENV_IDENTITY_OPENID_ENABLE: &str = "RUSTFS_IDENTITY_OPENID_ENABLE";
pub const ENV_IDENTITY_OPENID_CONFIG_URL: &str = "RUSTFS_IDENTITY_OPENID_CONFIG_URL";
pub const ENV_IDENTITY_OPENID_ISSUER: &str = "RUSTFS_IDENTITY_OPENID_ISSUER";
pub const ENV_IDENTITY_OPENID_CLIENT_ID: &str = "RUSTFS_IDENTITY_OPENID_CLIENT_ID";
pub const ENV_IDENTITY_OPENID_CLIENT_SECRET: &str = "RUSTFS_IDENTITY_OPENID_CLIENT_SECRET";
pub const ENV_IDENTITY_OPENID_SCOPES: &str = "RUSTFS_IDENTITY_OPENID_SCOPES";
@@ -52,10 +50,9 @@ pub const ENV_IDENTITY_OPENID_USERNAME_CLAIM: &str = "RUSTFS_IDENTITY_OPENID_USE
pub const ENV_IDENTITY_OPENID_HIDE_FROM_UI: &str = "RUSTFS_IDENTITY_OPENID_HIDE_FROM_UI";
/// List of all environment variable keys for an OIDC provider.
pub const ENV_IDENTITY_OPENID_KEYS: &[&str; 18] = &[
pub const ENV_IDENTITY_OPENID_KEYS: &[&str; 17] = &[
ENV_IDENTITY_OPENID_ENABLE,
ENV_IDENTITY_OPENID_CONFIG_URL,
ENV_IDENTITY_OPENID_ISSUER,
ENV_IDENTITY_OPENID_CLIENT_ID,
ENV_IDENTITY_OPENID_CLIENT_SECRET,
ENV_IDENTITY_OPENID_SCOPES,
@@ -77,7 +74,6 @@ pub const ENV_IDENTITY_OPENID_KEYS: &[&str; 18] = &[
pub const IDENTITY_OPENID_KEYS: &[&str] = &[
crate::ENABLE_KEY,
OIDC_CONFIG_URL,
OIDC_ISSUER,
OIDC_CLIENT_ID,
OIDC_CLIENT_SECRET,
OIDC_SCOPES,
-1
View File
@@ -57,7 +57,6 @@ pub const ENV_WEBDAV_CERTS_DIR: &str = "RUSTFS_WEBDAV_CERTS_DIR";
pub const ENV_WEBDAV_CA_FILE: &str = "RUSTFS_WEBDAV_CA_FILE";
pub const ENV_WEBDAV_MAX_BODY_SIZE: &str = "RUSTFS_WEBDAV_MAX_BODY_SIZE";
pub const ENV_WEBDAV_REQUEST_TIMEOUT: &str = "RUSTFS_WEBDAV_REQUEST_TIMEOUT";
pub const ENV_WEBDAV_MAX_CONNECTIONS: &str = "RUSTFS_WEBDAV_MAX_CONNECTIONS";
/// Default SFTP server bind address.
pub const DEFAULT_SFTP_ADDRESS: &str = "0.0.0.0:2222";
-2
View File
@@ -81,8 +81,6 @@ pub const ENV_TEST_IAM_FAIL_INIT_ATTEMPTS: &str = "RUSTFS_TEST_IAM_FAIL_INIT_ATT
pub const ENV_TEST_IAM_RETRY_INTERVAL_MS: &str = "RUSTFS_TEST_IAM_RETRY_INTERVAL_MS";
/// Runtime env var controlling the transition worker count.
pub const ENV_TRANSITION_WORKERS: &str = "RUSTFS_MAX_TRANSITION_WORKERS";
/// Runtime env var controlling the expiry worker count.
pub const ENV_MAX_EXPIRY_WORKERS: &str = "RUSTFS_MAX_EXPIRY_WORKERS";
/// Runtime env var controlling the absolute maximum transition workers.
pub const ENV_TRANSITION_WORKERS_ABSOLUTE_MAX: &str = "RUSTFS_ABSOLUTE_MAX_WORKERS";
/// Runtime env var controlling the transition queue capacity.
+4 -2
View File
@@ -220,10 +220,12 @@ pub const ENV_SCANNER_YIELD_EVERY_N_OBJECTS: &str = "RUSTFS_SCANNER_YIELD_EVERY_
pub const DEFAULT_SCANNER_IDLE_MODE: bool = true;
/// Default set scan concurrency budget.
pub const DEFAULT_SCANNER_MAX_CONCURRENT_SET_SCANS: usize = 4;
/// `0` means no additional limit beyond deployment topology.
pub const DEFAULT_SCANNER_MAX_CONCURRENT_SET_SCANS: usize = 0;
/// Default disk scan concurrency budget.
pub const DEFAULT_SCANNER_MAX_CONCURRENT_DISK_SCANS: usize = 4;
/// `0` means no additional limit beyond available disks in the set.
pub const DEFAULT_SCANNER_MAX_CONCURRENT_DISK_SCANS: usize = 0;
/// Default object interval for cooperative scanner yields.
pub const DEFAULT_SCANNER_YIELD_EVERY_N_OBJECTS: u64 = 128;
-9
View File
@@ -36,11 +36,6 @@ pub const ENV_TRUST_SYSTEM_CA: &str = "RUSTFS_TRUST_SYSTEM_CA";
/// To change this behavior, set the environment variable RUSTFS_TRUST_SYSTEM_CA=1
pub const DEFAULT_TRUST_SYSTEM_CA: bool = false;
/// Environment variable for an extra outbound root CA certificate bundle.
/// Use this to trust an internal CA for outbound HTTPS clients without replacing
/// the default operating-system/web PKI roots via SSL_CERT_FILE.
pub const ENV_RUSTFS_EXTRA_CA_CERT: &str = "RUSTFS_EXTRA_CA_CERT";
/// Environment variable to trust leaf certificates as CA
/// When set to "1", RustFS will treat leaf certificates as CA certificates for trust validation.
/// By default, this is disabled.
@@ -147,10 +142,6 @@ pub const DEFAULT_H2_KEEP_ALIVE_TIMEOUT: u64 = 10;
/// proxy's upstream idle-keepalive, or lower the proxy's keepalive below this
/// value. Environments that expose RustFS directly to untrusted slow clients and
/// want tighter slowloris protection can lower it via the env var below.
///
/// The same budget bounds the TLS handshake on the listener, so an unauthenticated
/// peer cannot park an accept task and its socket indefinitely by opening a
/// connection and then stalling the handshake.
pub const ENV_HTTP1_HEADER_READ_TIMEOUT: &str = "RUSTFS_HTTP1_HEADER_READ_TIMEOUT";
pub const DEFAULT_HTTP1_HEADER_READ_TIMEOUT: u64 = 75;
-30
View File
@@ -56,33 +56,3 @@ pub const DEFAULT_OBJECT_MMAP_READ_ENABLE: bool = true;
///
/// Prefer [`DEFAULT_OBJECT_MMAP_READ_ENABLE`].
pub const DEFAULT_OBJECT_ZERO_COPY_ENABLE: bool = DEFAULT_OBJECT_MMAP_READ_ENABLE;
/// Environment variable capping the byte length a single mmap-copy read may
/// materialize in memory.
///
/// The mmap-copy read path returns the whole requested range as one owned
/// allocation before the first byte is served. GET/heal shard reads request
/// the entire part span in one call, so for a large single-part object
/// (e.g. a multi-gigabyte non-multipart upload) an uncapped mmap-copy read
/// allocates the whole shard in memory — stalling first-byte latency past the
/// disk-read timeout and OOM-killing memory-limited deployments
/// (<https://github.com/rustfs/rustfs/issues/5123>). Reads longer than this
/// cap fall back to the bounded streaming reader instead.
///
/// - Purpose: Bound per-shard-read memory for mmap-based reads
/// - Acceptable values: byte count as an unsigned integer; `0` disables
/// mmap-copy for all non-empty reads (every read streams)
/// - Example: `export RUSTFS_OBJECT_MMAP_READ_MAX_LENGTH=8388608`
pub const ENV_OBJECT_MMAP_READ_MAX_LENGTH: &str = "RUSTFS_OBJECT_MMAP_READ_MAX_LENGTH";
/// Default mmap-copy read length cap: 32 MiB per shard read.
///
/// Large enough that typical multipart part shards (parts up to a few hundred
/// megabytes across the erasure set) keep the mmap fast path, small enough
/// that whole-part reads of huge single-part objects stream instead of
/// materializing gigabytes per shard.
///
/// The cap bounds memory per shard reader, so a single part read can still
/// materialize up to `data_shards x cap` bytes; raising the cap raises that
/// per-request bound proportionally.
pub const DEFAULT_OBJECT_MMAP_READ_MAX_LENGTH: usize = 32 * 1024 * 1024;
-7
View File
@@ -24,14 +24,7 @@ description = "Credentials management utilities for RustFS, enabling secure hand
keywords = ["rustfs", "Minio", "credentials", "authentication", "authorization"]
categories = ["web-programming", "development-tools", "data-structures", "security"]
[features]
default = []
hotpath = ["hotpath/hotpath"]
hotpath-alloc = ["hotpath", "hotpath/hotpath-alloc"]
hotpath-cpu = ["hotpath", "hotpath/hotpath-cpu"]
[dependencies]
hotpath.workspace = true
base64-simd = { workspace = true }
hmac = { workspace = true }
rand = { workspace = true, features = ["serde"] }
-4
View File
@@ -29,7 +29,6 @@ documentation = "https://docs.rs/rustfs-crypto/latest/rustfs_crypto/"
workspace = true
[dependencies]
hotpath.workspace = true
aes-gcm = { workspace = true, optional = true, features = ["rand_core"] }
argon2 = { workspace = true, optional = true }
chacha20poly1305 = { workspace = true, optional = true }
@@ -50,9 +49,6 @@ time = { workspace = true, features = ["parsing", "formatting", "macros", "serde
[features]
default = ["crypto", "fips"]
hotpath = ["hotpath/hotpath"]
hotpath-alloc = ["hotpath", "hotpath/hotpath-alloc"]
hotpath-cpu = ["hotpath", "hotpath/hotpath-cpu"]
fips = []
crypto = [
"dep:aes-gcm",
+2 -7
View File
@@ -27,16 +27,11 @@ categories = ["data-structures", "filesystem"]
[lints]
workspace = true
[features]
default = []
hotpath = ["hotpath/hotpath", "rustfs-filemeta/hotpath"]
hotpath-alloc = ["hotpath", "hotpath/hotpath-alloc", "rustfs-filemeta/hotpath-alloc"]
hotpath-cpu = ["hotpath", "hotpath/hotpath-cpu", "rustfs-filemeta/hotpath-cpu"]
[dependencies]
hotpath.workspace = true
serde = { workspace = true, features = ["derive"] }
path-clean = { workspace = true }
rmp-serde = { workspace = true }
async-trait = { workspace = true }
rustfs-filemeta = { workspace = true }
[lib]
File diff suppressed because it is too large Load Diff
+2 -56
View File
@@ -25,64 +25,14 @@ workspace = true
[features]
default = []
hotpath = [
"hotpath/hotpath",
"hotpath/tokio",
"hotpath/futures",
"hotpath/reqwest-0-13",
"rustfs-config/hotpath",
"rustfs-credentials/hotpath",
"rustfs-data-usage/hotpath",
"rustfs-ecstore/hotpath",
"rustfs-filemeta/hotpath",
"rustfs-lock/hotpath",
"rustfs-madmin/hotpath",
"rustfs-protos/hotpath",
"rustfs-rio/hotpath",
"rustfs-signer/hotpath",
"rustfs-utils/hotpath",
]
hotpath-alloc = [
"hotpath",
"hotpath/hotpath-alloc",
"rustfs-config/hotpath-alloc",
"rustfs-credentials/hotpath-alloc",
"rustfs-data-usage/hotpath-alloc",
"rustfs-ecstore/hotpath-alloc",
"rustfs-filemeta/hotpath-alloc",
"rustfs-lock/hotpath-alloc",
"rustfs-madmin/hotpath-alloc",
"rustfs-protos/hotpath-alloc",
"rustfs-rio/hotpath-alloc",
"rustfs-signer/hotpath-alloc",
"rustfs-utils/hotpath-alloc",
]
hotpath-cpu = [
"hotpath",
"hotpath/hotpath-cpu",
"rustfs-config/hotpath-cpu",
"rustfs-credentials/hotpath-cpu",
"rustfs-data-usage/hotpath-cpu",
"rustfs-ecstore/hotpath-cpu",
"rustfs-filemeta/hotpath-cpu",
"rustfs-lock/hotpath-cpu",
"rustfs-madmin/hotpath-cpu",
"rustfs-protos/hotpath-cpu",
"rustfs-rio/hotpath-cpu",
"rustfs-signer/hotpath-cpu",
"rustfs-utils/hotpath-cpu",
]
ftps = []
sftp = []
[dependencies]
hotpath.workspace = true
rustfs-config = { workspace = true, features = ["constants"] }
rustfs-credentials.workspace = true
rustfs-ecstore.workspace = true
rustfs-data-usage.workspace = true
rustfs-rio.workspace = true
rustfs-utils = { workspace = true, features = ["egress"] }
flatbuffers.workspace = true
futures.workspace = true
rustfs-lock.workspace = true
@@ -98,7 +48,6 @@ rustfs-filemeta.workspace = true
bytes = { workspace = true, features = ["serde"] }
serial_test = { workspace = true }
aws-sdk-s3 = { workspace = true, default-features = false, features = ["sigv4a", "default-https-client", "rt-tokio"] }
aws-sdk-sts = { workspace = true, default-features = false, features = ["default-https-client", "rt-tokio"] }
aws-config = { workspace = true }
aws-smithy-http-client = { workspace = true, default-features = false, features = ["rustls-aws-lc"] }
async-compression = { workspace = true, features = ["tokio", "bzip2", "xz"] }
@@ -108,7 +57,7 @@ http.workspace = true
http-body-util.workspace = true
hyper = { workspace = true, features = ["http2", "http1", "server"] }
hyper-util = { workspace = true, features = ["tokio", "server-auto", "server-graceful", "tracing"] }
reqwest = { workspace = true, features = ["json", "multipart", "stream"] }
reqwest = { workspace = true, default-features = false, features = ["rustls", "charset", "http2", "system-proxy", "stream", "json", "multipart"] }
rustfs-signer.workspace = true
tracing = { workspace = true }
tracing-subscriber = { workspace = true, features = ["env-filter", "time"] }
@@ -118,10 +67,7 @@ walkdir.workspace = true
base64 = { workspace = true }
rand = { workspace = true, features = ["serde"] }
chrono = { workspace = true, features = ["serde"] }
hex = { workspace = true }
md-5 = { workspace = true }
opentelemetry-proto = { workspace = true }
prost.workspace = true
md5 = { workspace = true }
sha2 = { workspace = true }
astral-tokio-tar = { workspace = true }
s3s = { workspace = true, features = ["minio"] }
-163
View File
@@ -46,45 +46,6 @@ mod tests {
use std::time::{Duration, Instant};
const ADMIN_INFO_PATH: &str = "/rustfs/admin/v3/info";
const ADMIN_MANUAL_TRANSITION_BUCKET: &str = "auth-deny-manual-transition";
const ADMIN_MANUAL_TRANSITION_PATH: &str =
"/rustfs/admin/v3/ilm/transition/run?bucket=auth-deny-manual-transition&maxObjects=1&mode=async";
fn assert_no_raw_manual_transition_markers(body: &str, context: &str) {
assert!(
!body.contains("\"marker\"") && !body.contains("\"versionMarker\"") && !body.contains("\"version_marker\""),
"{context} must not expose raw manual transition resume markers, body: {body}"
);
}
async fn wait_for_terminal_manual_transition_job(
env: &RustFSTestEnvironment,
status_endpoint: &str,
) -> Result<String, Box<dyn Error + Send + Sync>> {
let deadline = Instant::now() + Duration::from_secs(30);
loop {
let (status, body) =
signed_request(&env.url, http::Method::GET, status_endpoint, None, &env.access_key, &env.secret_key).await?;
assert_eq!(
status,
reqwest::StatusCode::OK,
"root credential must query manual transition job status, body: {body}"
);
assert_no_raw_manual_transition_markers(&body, "manual transition status response");
let value: serde_json::Value = serde_json::from_str(&body)?;
let job_status = value
.get("status")
.and_then(serde_json::Value::as_str)
.ok_or("manual transition job status response must include status")?;
if matches!(job_status, "completed" | "partial" | "cancelled" | "failed" | "unknown") {
return Ok(body);
}
if Instant::now() >= deadline {
return Err(format!("manual transition job did not reach terminal status within 30s; last={body}").into());
}
tokio::time::sleep(Duration::from_millis(250)).await;
}
}
/// Send a SigV4-signed request to `path` (optionally with a JSON `body`) and
/// return `(status, body)`. Uses the `UNSIGNED_PAYLOAD` content hash so a
@@ -197,130 +158,6 @@ mod tests {
Ok(())
}
#[tokio::test(flavor = "multi_thread")]
#[serial]
async fn non_admin_credential_denied_on_manual_transition_run() -> Result<(), Box<dyn Error + Send + Sync>> {
init_logging();
let mut env = RustFSTestEnvironment::new().await?;
env.start_rustfs_server(vec![]).await?;
let user_ak = "ilmtransitionlimited";
let user_sk = "ilmtransitionlimitedsecret";
create_limited_user(&env, user_ak, user_sk).await?;
env.create_s3_client()
.create_bucket()
.bucket(ADMIN_MANUAL_TRANSITION_BUCKET)
.send()
.await?;
let (root_status, root_body) = signed_request(
&env.url,
http::Method::POST,
ADMIN_MANUAL_TRANSITION_PATH,
None,
&env.access_key,
&env.secret_key,
)
.await?;
assert_eq!(
root_status,
reqwest::StatusCode::ACCEPTED,
"root credential must reach the manual transition handler, body: {root_body}"
);
assert!(
root_body.contains("\"mode\":\"durable_job\""),
"root response should be the durable manual transition JSON contract, body: {root_body}"
);
assert_no_raw_manual_transition_markers(&root_body, "manual transition run response");
let root_value: serde_json::Value = serde_json::from_str(&root_body)?;
let job_id = root_value
.get("job_id")
.and_then(serde_json::Value::as_str)
.ok_or("manual transition async response must include job_id")?;
let status_endpoint = root_value
.get("status_endpoint")
.and_then(serde_json::Value::as_str)
.ok_or("manual transition async response must include status_endpoint")?;
let cancel_endpoint = root_value
.get("cancel_endpoint")
.and_then(serde_json::Value::as_str)
.ok_or("manual transition async response must include cancel_endpoint")?;
assert_eq!(
cancel_endpoint, status_endpoint,
"manual transition durable jobs currently use the same status/cancel endpoint"
);
assert!(
status_endpoint.ends_with(job_id),
"status endpoint must address the returned job id, job_id={job_id}, status_endpoint={status_endpoint}"
);
let terminal_body = wait_for_terminal_manual_transition_job(&env, status_endpoint).await?;
let terminal: serde_json::Value = serde_json::from_str(&terminal_body)?;
assert_eq!(terminal.get("job_id").and_then(serde_json::Value::as_str), Some(job_id));
assert_eq!(
terminal
.get("report")
.and_then(|report| report.get("bucket"))
.and_then(serde_json::Value::as_str),
Some(ADMIN_MANUAL_TRANSITION_BUCKET)
);
let (root_status, root_body) =
signed_request(&env.url, http::Method::DELETE, status_endpoint, None, &env.access_key, &env.secret_key).await?;
assert_eq!(
root_status,
reqwest::StatusCode::OK,
"root credential must cancel/query a terminal manual transition job idempotently, body: {root_body}"
);
assert_no_raw_manual_transition_markers(&root_body, "manual transition cancel response");
let root_cancel: serde_json::Value = serde_json::from_str(&root_body)?;
assert_eq!(root_cancel.get("job_id").and_then(serde_json::Value::as_str), Some(job_id));
assert!(
matches!(
root_cancel.get("status").and_then(serde_json::Value::as_str),
Some("completed" | "partial" | "failed" | "unknown")
),
"terminal cancel must not rewrite the job into cancelled state, body: {root_body}"
);
let (status, body) =
signed_request(&env.url, http::Method::POST, ADMIN_MANUAL_TRANSITION_PATH, None, user_ak, user_sk).await?;
assert_eq!(
status,
reqwest::StatusCode::FORBIDDEN,
"non-admin credential must get 403 on manual transition run, body: {body}"
);
assert!(
body.contains("AccessDenied"),
"manual transition rejection must carry the AccessDenied S3 error code, body: {body}"
);
let (status, body) = signed_request(&env.url, http::Method::GET, status_endpoint, None, user_ak, user_sk).await?;
assert_eq!(
status,
reqwest::StatusCode::FORBIDDEN,
"non-admin credential must get 403 on manual transition status, body: {body}"
);
assert_no_raw_manual_transition_markers(&body, "manual transition status rejection");
assert!(
body.contains("AccessDenied"),
"manual transition status rejection must carry the AccessDenied S3 error code, body: {body}"
);
let (status, body) = signed_request(&env.url, http::Method::DELETE, status_endpoint, None, user_ak, user_sk).await?;
assert_eq!(
status,
reqwest::StatusCode::FORBIDDEN,
"non-admin credential must get 403 on manual transition cancel, body: {body}"
);
assert_no_raw_manual_transition_markers(&body, "manual transition cancel rejection");
assert!(
body.contains("AccessDenied"),
"manual transition cancel rejection must carry the AccessDenied S3 error code, body: {body}"
);
env.stop_server();
Ok(())
}
/// Rotating the root credentials (restart with new `--access-key` /
/// `--secret-key` on the same data directory) takes effect: the new
/// credential is accepted and the old one is rejected, on both the S3 data
+60 -259
View File
@@ -26,18 +26,75 @@
//! Later batches tracked on backlog#1154: config get/set, info, pools status,
//! group lifecycle, import/export IAM.
use crate::common::{
RustFSTestEnvironment, admin_ok, admin_request, admin_request_with_session_token, build_test_sts_client, init_logging,
};
use crate::common::{RustFSTestEnvironment, init_logging, local_http_client};
use aws_sdk_s3::config::{Credentials, Region};
use aws_sdk_s3::primitives::ByteStream;
use aws_sdk_s3::{Client, Config};
use http::header::{CONTENT_TYPE, HOST};
use reqwest::StatusCode;
use rustfs_signer::constants::UNSIGNED_PAYLOAD;
use rustfs_signer::sign_v4;
use s3s::Body;
use serial_test::serial;
use std::error::Error;
use tokio::time::{Duration, sleep};
type TestResult = Result<(), Box<dyn Error + Send + Sync>>;
type BoxError = Box<dyn Error + Send + Sync>;
/// Signs and sends an admin HTTP request with the given credential, returning
/// status and body. Native `/rustfs/admin/v3` requests and responses are plain
/// JSON (the MinIO-compat encryption applies only to `/minio/admin/v3` paths).
async fn admin_request(
base_url: &str,
method: http::Method,
path_and_query: &str,
body: Option<String>,
access_key: &str,
secret_key: &str,
) -> Result<(StatusCode, String), BoxError> {
let url = format!("{base_url}{path_and_query}");
let uri = url.parse::<http::Uri>()?;
let authority = uri.authority().ok_or("admin URL missing authority")?.to_string();
let mut builder = http::Request::builder()
.method(method.clone())
.uri(uri)
.header(HOST, authority)
.header("x-amz-content-sha256", UNSIGNED_PAYLOAD);
if body.is_some() {
builder = builder.header(CONTENT_TYPE, "application/json");
}
let content_len = body.as_ref().map(|b| b.len() as i64).unwrap_or_default();
let signed = sign_v4(builder.body(Body::empty())?, content_len, access_key, secret_key, "", "us-east-1");
let reqwest_method = reqwest::Method::from_bytes(method.as_str().as_bytes())?;
let mut request = local_http_client().request(reqwest_method, &url);
for (name, value) in signed.headers() {
request = request.header(name, value);
}
if let Some(body) = body {
request = request.body(body);
}
let response = request.send().await?;
let status = response.status();
let text = response.text().await.unwrap_or_default();
Ok((status, text))
}
/// Root-credential admin request that must succeed; returns the response body.
async fn admin_ok(
env: &RustFSTestEnvironment,
method: http::Method,
path_and_query: &str,
body: Option<String>,
) -> Result<String, BoxError> {
let (status, text) = admin_request(&env.url, method.clone(), path_and_query, body, &env.access_key, &env.secret_key).await?;
if !status.is_success() {
return Err(format!("{method} {path_and_query} failed: {status} {text}").into());
}
Ok(text)
}
fn build_s3_client(url: &str, access_key: &str, secret_key: &str) -> Client {
let config = Config::builder()
@@ -89,262 +146,6 @@ fn bucket_rw_policy(bucket: &str) -> String {
.to_string()
}
async fn create_user_with_service_account_update_policy(
env: &RustFSTestEnvironment,
user: &str,
secret: &str,
policy: &str,
) -> TestResult {
admin_ok(
env,
http::Method::PUT,
&format!("/rustfs/admin/v3/add-user?accessKey={user}"),
Some(serde_json::json!({ "secretKey": secret, "status": "enabled" }).to_string()),
)
.await?;
admin_ok(
env,
http::Method::PUT,
&format!("/rustfs/admin/v3/add-canned-policy?name={policy}"),
Some(
serde_json::json!({
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": ["admin:UpdateServiceAccount"]
},
{
"Effect": "Allow",
"Action": ["sts:AssumeRole"],
"Resource": ["arn:aws:s3:::*"]
}
]
})
.to_string(),
),
)
.await?;
admin_ok(
env,
http::Method::POST,
"/rustfs/admin/v3/idp/builtin/policy/attach",
Some(serde_json::json!({ "policies": [policy], "user": user }).to_string()),
)
.await?;
Ok(())
}
async fn create_service_account_for(
env: &RustFSTestEnvironment,
parent: &str,
) -> Result<(String, String), Box<dyn Error + Send + Sync>> {
let response = admin_ok(
env,
http::Method::PUT,
"/rustfs/admin/v3/add-service-accounts",
Some(serde_json::json!({ "targetUser": parent }).to_string()),
)
.await?;
let response: serde_json::Value = serde_json::from_str(&response)?;
let access_key = response["credentials"]["accessKey"]
.as_str()
.ok_or("service account response should contain credentials.accessKey")?
.to_owned();
let secret_key = response["credentials"]["secretKey"]
.as_str()
.ok_or("service account response should contain credentials.secretKey")?
.to_owned();
Ok((access_key, secret_key))
}
async fn assert_admin_status(
env: &RustFSTestEnvironment,
credentials: (&str, &str, Option<&str>),
path: &str,
body: String,
expected: StatusCode,
context: &str,
) -> TestResult {
let (access_key, secret_key, session_token) = credentials;
let (status, response) =
admin_request_with_session_token(&env.url, http::Method::POST, path, Some(body), access_key, secret_key, session_token)
.await?;
assert_eq!(status, expected, "{context}: got {status}: {response}");
if expected == StatusCode::FORBIDDEN {
assert!(response.contains("AccessDenied"), "{context}: expected AccessDenied body, got {response}");
}
Ok(())
}
#[tokio::test]
#[serial]
async fn test_update_service_account_enforces_owner_and_parent_scope() -> TestResult {
init_logging();
let mut env = RustFSTestEnvironment::new().await?;
env.start_rustfs_server(vec![]).await?;
let parent = "updateparent";
let parent_secret = "updateparentsecret";
let outsider = "updateoutsider";
let outsider_secret = "updateoutsidersecret";
let ordinary = "updateordinary";
let ordinary_secret = "updateordinarysecret";
create_user_with_service_account_update_policy(&env, parent, parent_secret, "update-parent-policy").await?;
create_user_with_service_account_update_policy(&env, outsider, outsider_secret, "update-outsider-policy").await?;
admin_ok(
&env,
http::Method::POST,
"/rustfs/admin/v3/idp/builtin/policy/attach",
Some(serde_json::json!({ "policies": ["consoleAdmin"], "user": outsider }).to_string()),
)
.await?;
admin_ok(
&env,
http::Method::PUT,
&format!("/rustfs/admin/v3/add-user?accessKey={ordinary}"),
Some(serde_json::json!({ "secretKey": ordinary_secret, "status": "enabled" }).to_string()),
)
.await?;
let (target_access_key, _) = create_service_account_for(&env, parent).await?;
let target_path = format!("/rustfs/admin/v3/update-service-account?accessKey={target_access_key}");
assert_admin_status(
&env,
(&env.access_key, &env.secret_key, None),
&target_path,
serde_json::json!({}).to_string(),
StatusCode::NO_CONTENT,
"root no-op update across parents must succeed",
)
.await?;
let custom_policy = serde_json::json!({
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Action": ["s3:GetObject"],
"Resource": ["arn:aws:s3:::update-scope/*"]
}]
});
assert_admin_status(
&env,
(&env.access_key, &env.secret_key, None),
&target_path,
serde_json::json!({ "newPolicy": custom_policy }).to_string(),
StatusCode::NO_CONTENT,
"root implied-to-custom update across parents must succeed",
)
.await?;
assert_admin_status(
&env,
(parent, parent_secret, None),
&target_path,
serde_json::json!({ "newDescription": "updated by parent" }).to_string(),
StatusCode::NO_CONTENT,
"parent with UpdateServiceAccount may update its own service account",
)
.await?;
let takeover = serde_json::json!({
"newSecretKey": "cross-parent-takeover-secret",
"newDescription": "cross-parent takeover"
})
.to_string();
assert_admin_status(
&env,
(ordinary, ordinary_secret, None),
&target_path,
takeover.clone(),
StatusCode::FORBIDDEN,
"ordinary user must not update another parent's service account",
)
.await?;
assert_admin_status(
&env,
(outsider, outsider_secret, None),
&target_path,
takeover.clone(),
StatusCode::FORBIDDEN,
"non-owner consoleAdmin must not update across parents",
)
.await?;
let (derived_access_key, derived_secret_key) = create_service_account_for(&env, outsider).await?;
assert_admin_status(
&env,
(&derived_access_key, &derived_secret_key, None),
&target_path,
takeover.clone(),
StatusCode::FORBIDDEN,
"service-account credential must not update across parents",
)
.await?;
let assumed = build_test_sts_client(&env.url, outsider, outsider_secret, None, "e2e-admin-update-service-account")
.assume_role()
.role_arn("arn:aws:iam::123456789012:role/update-service-account")
.role_session_name("update-service-account-scope")
.send()
.await?;
let temporary = assumed
.credentials()
.ok_or("AssumeRole response should contain credentials")?;
assert_admin_status(
&env,
(temporary.access_key_id(), temporary.secret_access_key(), Some(temporary.session_token())),
&target_path,
takeover,
StatusCode::FORBIDDEN,
"temporary credential must not update across parents",
)
.await?;
let info = admin_ok(
&env,
http::Method::GET,
&format!("/rustfs/admin/v3/info-service-account?accessKey={target_access_key}"),
None,
)
.await?;
let info: serde_json::Value = serde_json::from_str(&info)?;
assert_eq!(
info["impliedPolicy"].as_bool(),
Some(false),
"root update must replace the implied policy with a custom policy"
);
assert!(
info["policy"].as_str().is_some_and(|policy| policy.contains("s3:GetObject")),
"custom policy must round-trip through the handler: {info}"
);
assert_eq!(
info["description"].as_str(),
Some("updated by parent"),
"denied takeover attempts must not mutate target"
);
let (missing_status, missing_body) = admin_request(
&env.url,
http::Method::POST,
"/rustfs/admin/v3/update-service-account?accessKey=missing-service-account",
Some(serde_json::json!({}).to_string()),
&env.access_key,
&env.secret_key,
)
.await?;
assert_eq!(missing_status, StatusCode::NOT_FOUND, "missing target must fail closed: {missing_body}");
assert!(
missing_body.contains("NoSuchResource"),
"missing target must preserve the lookup error: {missing_body}"
);
env.stop_server();
Ok(())
}
/// Full user -> policy -> service-account lifecycle, proving each management
/// call takes effect on the data plane, not just that the endpoint answers 200.
#[tokio::test]
-79
View File
@@ -1,79 +0,0 @@
// Copyright 2026 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use crate::common::{RustFSTestEnvironment, init_logging, local_http_client};
use http::header::HOST;
use reqwest::StatusCode;
use rustfs_signer::constants::UNSIGNED_PAYLOAD;
use rustfs_signer::sign_v4;
use s3s::Body;
use serde::Deserialize;
use std::error::Error;
#[derive(Debug, Deserialize)]
struct PoolListItem {
id: usize,
cmdline: String,
status: String,
}
async fn signed_admin_get(env: &RustFSTestEnvironment, path: &str) -> Result<reqwest::Response, Box<dyn Error + Send + Sync>> {
let url = format!("{}{path}", env.url);
let uri = url.parse::<http::Uri>()?;
let authority = uri.authority().ok_or("admin URL missing authority")?.to_string();
let request = http::Request::builder()
.method(http::Method::GET)
.uri(uri)
.header(HOST, authority)
.header("x-amz-content-sha256", UNSIGNED_PAYLOAD)
.body(Body::empty())?;
let signed = sign_v4(request, 0, &env.access_key, &env.secret_key, "", "us-east-1");
let mut request = local_http_client().get(&url);
for (name, value) in signed.headers() {
request = request.header(name, value);
}
Ok(request.send().await?)
}
#[tokio::test]
async fn single_drive_pools_list_succeeds_without_enabling_decommission_status() -> Result<(), Box<dyn Error + Send + Sync>> {
init_logging();
let mut env = RustFSTestEnvironment::new().await?;
env.start_rustfs_server(vec![]).await?;
let response = signed_admin_get(&env, "/rustfs/admin/v3/pools/list").await?;
let status = response.status();
let body = response.bytes().await?;
assert_eq!(status, StatusCode::OK, "pools list failed: {}", String::from_utf8_lossy(&body));
let pools: Vec<PoolListItem> = serde_json::from_slice(&body)?;
assert_eq!(pools.len(), 1);
assert_eq!(pools[0].id, 0);
assert_eq!(pools[0].cmdline, env.temp_dir);
assert_eq!(pools[0].status, "active");
let response = signed_admin_get(&env, "/rustfs/admin/v3/decommission/status").await?;
let status = response.status();
let body = response.text().await?;
assert_eq!(
status,
StatusCode::NOT_IMPLEMENTED,
"decommission status changed for a single pool: {body}"
);
assert!(body.contains("NotImplemented"), "unexpected decommission error body: {body}");
Ok(())
}
@@ -170,81 +170,3 @@ async fn test_anonymous_access_allowed_when_restrict_public_buckets_disabled()
info!("Test passed: anonymous access allowed with RestrictPublicBuckets=false");
Ok(())
}
/// A policy granting anonymous `s3:ListBucket` also permits ListObjectVersions.
/// That grant must still be subject to RestrictPublicBuckets: the versions listing
/// reaches authorization through a fallback branch, and that branch has to apply the
/// same public-access gate as a direct grant.
#[tokio::test]
#[serial]
async fn ghsa_x298_anonymous_list_object_versions_denied_when_restrict_public_buckets_enabled()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
info!("Starting test: anonymous ListObjectVersions denied with RestrictPublicBuckets=true...");
let mut env = RustFSTestEnvironment::new().await?;
env.start_rustfs_server(vec![]).await?;
let bucket_name = "anon-test-restrict-versions";
let admin_client = env.create_s3_client();
admin_client.create_bucket().bucket(bucket_name).send().await?;
let policy_json = serde_json::json!({
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AllowAnonymousListBucket",
"Effect": "Allow",
"Principal": "*",
"Action": ["s3:ListBucket"],
"Resource": [format!("arn:aws:s3:::{}", bucket_name)]
}
]
})
.to_string();
admin_client
.put_bucket_policy()
.bucket(bucket_name)
.policy(&policy_json)
.send()
.await?;
admin_client
.put_object()
.bucket(bucket_name)
.key("test.txt")
.body(aws_sdk_s3::primitives::ByteStream::from_static(b"hello anonymous"))
.send()
.await?;
// Without the public-access block the fallback grant is expected to work.
let versions_url = format!("{}/{}?versions=", env.url, bucket_name);
let resp = local_http_client().get(&versions_url).send().await?;
assert_eq!(
resp.status().as_u16(),
200,
"Anonymous ListObjectVersions should succeed via the s3:ListBucket grant"
);
admin_client
.put_public_access_block()
.bucket(bucket_name)
.public_access_block_configuration(
PublicAccessBlockConfiguration::builder()
.restrict_public_buckets(true)
.build(),
)
.send()
.await?;
let resp = local_http_client().get(&versions_url).send().await?;
assert_eq!(
resp.status().as_u16(),
403,
"Anonymous ListObjectVersions must be denied when RestrictPublicBuckets is true"
);
info!("Test passed: anonymous ListObjectVersions denied with RestrictPublicBuckets=true");
Ok(())
}
+105 -83
View File
@@ -16,17 +16,91 @@
#[cfg(test)]
mod tests {
use crate::common::{RustFSTestEnvironment, init_logging, signed_s3_request};
use crate::common::{RustFSTestEnvironment, init_logging};
use aws_sdk_s3::error::ProvideErrorMetadata;
use aws_sdk_s3::types::{
AccelerateConfiguration, BucketAccelerateStatus, BucketLoggingStatus, IndexDocument, LoggingEnabled, Payer,
RequestPaymentConfiguration, WebsiteConfiguration,
};
use http::Method;
use http::header::CONTENT_TYPE;
use serial_test::serial;
use std::path::PathBuf;
use std::process::Command;
use tracing::info;
fn awscurl_binary_path() -> PathBuf {
std::env::var_os("AWSCURL_PATH")
.map(PathBuf::from)
.unwrap_or_else(|| PathBuf::from("awscurl"))
}
fn awscurl_available() -> bool {
Command::new(awscurl_binary_path()).arg("--version").output().is_ok()
}
fn execute_s3_awscurl(
method: &str,
url: &str,
access_key: &str,
secret_key: &str,
) -> Result<String, Box<dyn std::error::Error + Send + Sync>> {
let output = Command::new(awscurl_binary_path())
.args([
"--service",
"s3",
"--region",
"us-east-1",
"--access_key",
access_key,
"--secret_key",
secret_key,
"-i",
"-X",
method,
url,
])
.output()?;
if !output.status.success() {
let stderr = String::from_utf8_lossy(&output.stderr);
let stdout = String::from_utf8_lossy(&output.stdout);
return Err(format!("awscurl failed: stderr='{stderr}', stdout='{stdout}'").into());
}
Ok(String::from_utf8_lossy(&output.stdout).to_string())
}
fn parse_status(raw: &str) -> Option<u16> {
raw.lines()
.filter_map(|line| {
if line.starts_with("HTTP/") {
line.split_whitespace().nth(1)?.parse::<u16>().ok()
} else {
None
}
})
.next_back()
}
fn parse_body(raw: &str) -> String {
if let Some(pos) = raw.rfind("\r\n\r\n") {
return raw[pos + 4..].to_string();
}
if let Some(pos) = raw.rfind("\n\n") {
return raw[pos + 2..].to_string();
}
String::new()
}
fn parse_headers(raw: &str) -> String {
let start = raw.rfind("HTTP/").unwrap_or(0);
let tail = &raw[start..];
if let Some(pos) = tail.find("\r\n\r\n") {
return tail[..pos].to_string();
}
if let Some(pos) = tail.find("\n\n") {
return tail[..pos].to_string();
}
tail.to_string()
}
#[tokio::test]
#[serial]
async fn test_dummy_bucket_compatibility_endpoints() {
@@ -396,6 +470,10 @@ mod tests {
async fn test_dummy_bucket_endpoints_http_contracts() {
init_logging();
info!("Starting test: dummy-compat bucket API HTTP contracts");
if !awscurl_available() {
info!("Skipping test_dummy_bucket_endpoints_http_contracts: awscurl binary not found");
return;
}
let mut env = RustFSTestEnvironment::new().await.expect("Failed to create test environment");
env.start_rustfs_server(vec![]).await.expect("Failed to start RustFS");
@@ -410,112 +488,56 @@ mod tests {
.await
.expect("Failed to create bucket");
let logging_response = signed_s3_request(
Method::GET,
&format!("{}/{bucket}?logging=", env.url),
None,
None,
&env.access_key,
&env.secret_key,
)
.await
.expect("GetBucketLogging HTTP request failed");
assert_eq!(logging_response.status(), 200, "GetBucketLogging should return 200");
let logging_body = logging_response
.text()
.await
.expect("Failed to read GetBucketLogging response body");
let logging_raw = execute_s3_awscurl("GET", &format!("{}/{bucket}?logging=", env.url), &env.access_key, &env.secret_key)
.expect("GetBucketLogging HTTP request failed");
assert_eq!(parse_status(&logging_raw), Some(200), "GetBucketLogging should return 200");
let logging_body = parse_body(&logging_raw);
assert!(
logging_body.contains("<BucketLoggingStatus"),
"GetBucketLogging response should contain BucketLoggingStatus XML, got: {logging_body}"
);
let accel_response = signed_s3_request(
Method::GET,
&format!("{}/{bucket}?accelerate=", env.url),
None,
None,
&env.access_key,
&env.secret_key,
)
.await
.expect("GetBucketAccelerateConfiguration HTTP request failed");
assert_eq!(accel_response.status(), 200, "GetBucketAccelerateConfiguration should return 200");
let accel_body = accel_response
.text()
.await
.expect("Failed to read GetBucketAccelerateConfiguration response body");
let accel_raw = execute_s3_awscurl("GET", &format!("{}/{bucket}?accelerate=", env.url), &env.access_key, &env.secret_key)
.expect("GetBucketAccelerateConfiguration HTTP request failed");
assert_eq!(parse_status(&accel_raw), Some(200), "GetBucketAccelerateConfiguration should return 200");
let accel_body = parse_body(&accel_raw);
assert!(
accel_body.contains("<AccelerateConfiguration"),
"GetBucketAccelerateConfiguration response should contain AccelerateConfiguration XML, got: {accel_body}"
);
let payment_response = signed_s3_request(
Method::GET,
&format!("{}/{bucket}?requestPayment=", env.url),
None,
None,
&env.access_key,
&env.secret_key,
)
.await
.expect("GetBucketRequestPayment HTTP request failed");
assert_eq!(payment_response.status(), 200, "GetBucketRequestPayment should return 200");
let payment_body = payment_response
.text()
.await
.expect("Failed to read GetBucketRequestPayment response body");
let payment_raw =
execute_s3_awscurl("GET", &format!("{}/{bucket}?requestPayment=", env.url), &env.access_key, &env.secret_key)
.expect("GetBucketRequestPayment HTTP request failed");
assert_eq!(parse_status(&payment_raw), Some(200), "GetBucketRequestPayment should return 200");
let payment_body = parse_body(&payment_raw);
assert!(
payment_body.contains("<Payer>BucketOwner</Payer>"),
"GetBucketRequestPayment should return BucketOwner payer, got: {payment_body}"
);
let website_response = signed_s3_request(
Method::GET,
&format!("{}/{bucket}?website=", env.url),
None,
None,
&env.access_key,
&env.secret_key,
)
.await
.expect("GetBucketWebsite HTTP request failed");
let website_raw = execute_s3_awscurl("GET", &format!("{}/{bucket}?website=", env.url), &env.access_key, &env.secret_key)
.expect("GetBucketWebsite HTTP request failed");
assert_eq!(
website_response.status(),
404,
parse_status(&website_raw),
Some(404),
"GetBucketWebsite should return 404 when website config is absent"
);
let website_content_type = website_response
.headers()
.get(CONTENT_TYPE)
.expect("GetBucketWebsite response should include Content-Type")
.to_str()
.expect("GetBucketWebsite Content-Type should be valid ASCII")
.to_ascii_lowercase();
let website_content_type = parse_headers(&website_raw).to_ascii_lowercase();
assert!(
website_content_type.contains("xml"),
website_content_type.contains("content-type:") && website_content_type.contains("xml"),
"GetBucketWebsite error response should be XML, got content-type: {website_content_type}"
);
let website_body = website_response
.text()
.await
.expect("Failed to read GetBucketWebsite response body");
let website_body = parse_body(&website_raw);
assert!(
website_body.contains("<Code>NoSuchWebsiteConfiguration</Code>"),
"GetBucketWebsite should return NoSuchWebsiteConfiguration code, got: {website_body}"
);
let delete_response = signed_s3_request(
Method::DELETE,
&format!("{}/{bucket}?website=", env.url),
None,
None,
&env.access_key,
&env.secret_key,
)
.await
.expect("DeleteBucketWebsite HTTP request failed");
assert_eq!(delete_response.status(), 204, "DeleteBucketWebsite should return 204");
let delete_raw =
execute_s3_awscurl("DELETE", &format!("{}/{bucket}?website=", env.url), &env.access_key, &env.secret_key)
.expect("DeleteBucketWebsite HTTP request failed");
assert_eq!(parse_status(&delete_raw), Some(204), "DeleteBucketWebsite should return 204");
env.stop_server();
}

Some files were not shown because too many files have changed in this diff Show More