mirror of
https://github.com/rustfs/rustfs.git
synced 2026-07-26 16:28:15 +00:00
copilot/research-github-issue-5076
384 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Zhengchao An
|
3ed682be42 |
feat: offline log fault-analysis system (rustfs diagnose) (#4876)
* feat(log-analyzer): add crate skeleton and unified event model
Implements LA-1 (rustfs/backlog#1282) of the log fault-analysis system
(rustfs/backlog#1281): new synchronous rustfs-log-analyzer crate with the
LogEvent/LogLevel/SourceRef/EventKind/ParseStats model shared by all
later stages. No tokio, no rustfs-* internal deps by design.
Note: thiserror listed in the issue is deferred until a stage actually
defines error types (LA-3/LA-4) to avoid an unused dependency.
* feat(log-analyzer): add line parsing layer
Implements LA-2 (rustfs/backlog#1283): four parse channels tried in order
per line — native tracing JSON, container-prefix stripping (K8s CRI /
docker compose / journald) with JSON retry, multi-line Rust panic block
folding (both pre- and post-1.65 formats, stderr has no JSON logger), and
a plain-text fallback that never fails. Parse accounting feeds the report
parse-ratio disclosure.
* feat(log-analyzer): add ingest layer for directories and archives
Implements LA-3 (rustfs/backlog#1284): expands customer inputs (files,
directories, zip/tar/tar.gz/.zst/.gz, stdin-like readers) into parsed
events. Magic-byte detection with extension fallback, recursive archive
walking with depth/entry/byte/memory caps (every capped input disclosed
in IngestReport.skipped), first-level directory names become node labels,
and nothing is ever extracted to disk so hostile entry paths are inert.
Adds tar 0.4 to workspace deps (sync; the async astral-tokio-tar used by
rustfs-zip does not fit this crate's no-tokio contract).
* feat(log-analyzer): add rule model, matching engine, and finding aggregation
Implements LA-4 (rustfs/backlog#1285): owned serde-round-trippable Rule/
Matcher/Severity types (external JSON rules deserialize into the same
types later), fail-fast RuleSet validation that reports every problem at
once, a linear-scan engine with regexes compiled once, and an
order-independent FindingsCollector (commutative aggregates only; the
test asserts byte-identical output across shuffled input orders).
* feat(log-analyzer): add built-in seed rule library (68 rules, 12 categories)
Implements LA-5 (rustfs/backlog#1286): the 2026-07 repository-wide
failure-log survey distilled into rules across disk health, erasure/
bitrot, quorum, network/RPC, distributed locks, heal, scanner, IAM,
startup/config/TLS, capacity, decommission/rebalance, and process panics.
Every anchor was verified verbatim against the source tree (94/94 hits,
zero corrections needed). Quorum rules pre-fill implies_root_cause for
the Phase-2 folding (rustfs/backlog#1290); client-side rules carry burst
thresholds (min_count) so isolated client mistakes don't clutter reports.
Tests: one realistic positive sample per rule (table-driven), exact-set
smoke samples including the intentional internode/client signature
double-hit, and negative cases.
* feat(log-analyzer): add analysis orchestration, report rendering, and redaction
Implements LA-6 (rustfs/backlog#1287): a single-pass Analyzer that does
rule matching, minute-bucket timelines (gap-filled, merged to <=60
buckets), unmatched WARN/ERROR template clustering (placeholders for
numbers/uuids/paths/addresses/quotes, 5000-template cap disclosed as
<overflow>), mixed-UTC-offset detection, and below-min_count demotion to
a low-confidence section. Renderers: pipe-friendly terminal text, stable
JSON (schema_version=1), and ticket-pasteable Markdown. --redact hashes
customer identifiers (stable h:sha256[..8]) in samples/evidence/messages
while keeping rule ids, targets, and panic locations intact.
* feat(rustfs): add 'rustfs diagnose' subcommand for offline log fault analysis
Implements LA-7 (rustfs/backlog#1288): wires rustfs-log-analyzer into the
main binary as a diagnose subcommand that short-circuits before
observability/storage init (same pattern as 'info' / 'tls inspect') so
the report on stdout is never wrapped by the JSON logger.
rustfs diagnose <paths>... [--format text|json|md] [--since 24h]
[--until ...] [--min-level warn] [--redact] [--top N] [--samples N]
Accepts files, directories, archives (.zip/.tar/.tar.gz/.zst/.gz) and '-'
for stdin. Exit codes: 0 = diagnosis completed (findings never fail the
process), 2 = bad arguments / no readable input.
diagnose_e2e covers the six MVP acceptance scenarios from
rustfs/backlog#1281 (directory+zst archive, multi-node zip attribution,
CRI-prefixed kubectl logs, panic folding, stable JSON schema, CLI parsing
incl. the legacy 'rustfs <volume>' preprocessor regression); the
full-binary smoke test is #[ignore]d (run with -- --ignored).
Usage doc: docs/operations/log-diagnose.md.
* ci(log-analyzer): guard rule anchors against log-message drift
Implements LA-8 (rustfs/backlog#1289): every seed-rule anchor must exist
verbatim in the rustfs source tree, so changing a log message without
updating its rule fails the gate instead of silently killing the rule.
- la-dump-anchors bin emits 'rule_id<TAB>anchor' TSV;
- scripts/check_log_analyzer_rules.sh greps each anchor (fixed-string,
*.rs only, excluding crates/log-analyzer itself to avoid self-matches);
- RuleSet::new now rejects anchors that are blank, contain tab/newline,
or are shorter than 8 bytes (no discriminating power); the '[FATAL]'
anchor gained its trailing space to meet the floor while still matching
the emit_fatal_stderr format string;
- wired as log-analyzer-rules-check into the pre-pr gate (it compiles the
crate, so it stays out of the fast pre-commit set).
Negative self-test: breaking an anchor makes the script exit 1 naming the
rule ('MISSING anchor for rule inconsistent-drive: zzz-not-exist-anchor').
* refactor(log-analyzer): use root-relative provenance for directory inputs
Binary smoke run showed report samples citing full absolute paths, which
drowns the useful part. Directory inputs now label sources as
"<root-name>/<relative-path>" (e.g. "smoke-logs/node1/rustfs.log");
archives and single files keep their existing provenance.
* chore(log-analyzer): reword comment to satisfy the typos gate
* fix(log-analyzer): declare chrono serde feature locally after workspace feature localization
* fix(log-analyzer): bound line reads so a newline-less input cannot bypass the byte cap
read_until grew the line buffer with the entire remaining stream before the
max_total_bytes check ran, so a single multi-GB line (decompression bomb or
corrupt file) could allocate unboundedly. Replace it with a capped reader that
enforces the remaining global budget chunk-by-chunk and adds a per-line cap
(IngestOptions::max_line_bytes, default 1 MiB); over-cap tails are discarded
but still charged, and truncation is disclosed once per file as the new
line_too_long skip reason. Flagged by Codex review on #4876.
* fix(log-analyzer): redact field-shaped identifiers inside message text and widen the hash to 64 bits
--redact only hashed IPv4 literals in unstructured message text, so
bucket/object/access-key values embedded in messages (access_key=AK123,
'bucket: media, object: private/a.bin') leaked into reports documented as
safe to forward. Apply the SENSITIVE_FIELDS list to key=value / key: value
shapes in message text with the same hash as structured fields, and extend
the hash from 8 to 16 hex chars so cross-identifier collisions stay
negligible. Flagged by Codex and Copilot review on #4876.
* fix(log-analyzer): strip collector prefixes before panic-block absorption
An open panic block tested continuation lines against absorbs() before their
CRI/compose/journald prefix was stripped, so containerized panics stored the
prefix in the payload and split into a truncated panic plus text noise as soon
as the note/backtrace lines arrived. Stripping now happens once at the top of
feed() and every channel judges the payload. Flagged by Codex review on #4876.
* fix(log-analyzer): remove two input-order dependencies in representative selection
The unmatched-cluster target stayed pinned to the first-seen event while the
representative sample could be replaced, so sample and target could come from
different events and vary with input order; the pair now updates together by
lexicographic (sample, target) min. Sample selection tie-broke on line number
alone, which is only unique within one file; the key now includes the source
file. Flagged by Copilot review on #4876.
* fix(rustfs): reject negative relative times in diagnose --since/--until
parse_time_arg accepted "-24h" and produced a future timestamp, contradicting
the documented 'counted back from now' semantics. The amount now parses as
unsigned, so a leading '-' fails with the usual invalid-time error. Flagged by
Copilot review on #4876.
* feat(log-analyzer): Phase 2 — causal folding, timeline anomalies, external rules (LA-9) (#4942)
* feat(log-analyzer): collapse cascade symptoms under their root-cause finding
Phase-2 sub-item A (rustfs/backlog#1290): a finding whose rule declares
implies_root_cause edges folds under a qualifying root — root.first_seen <=
symptom.first_seen + 5min and root.last_seen >= symptom.first_seen - 30min,
existence-based when either side has no timestamps (pure stderr panics).
Findings gain collapsed_into/caused; text/markdown render the root block with
an indented cascade line and stop listing collapsed symptoms flat, while JSON
keeps every finding. Roots are promoted to the most severe position among
their block so the report top still answers 'the most likely cause'.
* feat(log-analyzer): detect timeline/clock anomalies (schema v2)
Phase-2 sub-item B (rustfs/backlog#1290): three deterministic hints rendered
between the summary and findings — mixed UTC offsets (with a clock-skew note
when signature-mismatch findings coexist), per-node time ranges that do not
overlap at all (both nodes >100 timestamped events), and log gaps of at least
max(15min, 3x bucket width) after >=3 consecutive active minutes, upgraded to
restart evidence when a startup-class finding begins within 5min after the
gap. JSON gains timeline_anomalies and schema_version bumps to 2.
* feat(diagnose): load external rules with --rules <file.json>
Phase-2 sub-item C (rustfs/backlog#1290): an external JSON rule file
({schema_version: 1, rules: [Rule...]}, the exact serde shape of the built-in
rules) merges over the seed library, with same-id rules replacing built-ins so
the support team can hotfix a misfiring rule without a release. The merged set
validates as a whole and any problem (bad regex, duplicate id, empty matcher
group, wrong schema version) prints every error and exits 2 — analysis never
runs on a half-broken set. External anchors are exempt from the CI anchor
guard, documented as author-owned quality. Adds the custom-rules section to
docs/operations/log-diagnose.md.
* fix(log-analyzer): address PR #4876 review (redaction coverage, order-independence, guards)
Redaction (--redact) now honours its "forwardable" intent across every report surface instead of a 15-name field whitelist applied over a subset:
- redact_event scrubs the full fields map (sensitive names hashed whole, every other value run through redact_text) plus provenance, so the JSON/Markdown full-sample dump no longer leaks non-whitelisted fields (client_ip, url, user, ...).
- node labels are hashed once at ingestion, so summary.nodes, per-node timeline ranges, samples and timeline anomalies stay consistent and correlatable under one stable hash.
- evidence values, unmatched-cluster templates and skipped-input paths are now redacted; peer/disk/drive/volume/node/user added to the sensitive set; IPv6 literals are hashed (without touching `rust::paths` or HH:MM:SS clocks); provenance keeps the leaf filename and hashes the customer directory/archive prefix.
- redact.rs and docs/operations/log-diagnose.md reworded to "best-effort identifier scrubbing", not an anonymization guarantee.
Order-independence (the crate's headline contract):
- the evidence value cap keeps the lexicographically smallest N distinct values instead of the first-N-by-arrival (previously order-dependent).
- first_seen/last_seen break equal-instant ties on the offset, so the serialized RFC3339 offset no longer depends on input order.
Parsing:
- a new-format panic header no longer swallows the line immediately after it when that line is itself a JSON event or a second panic header (previously dropped an interleaved ERROR in merged stdout/stderr, or merged a panic-during-panic); trailing "note: ..." backtrace lines now fold into the block.
CLI:
- diagnose --since/--until reject absurd relative amounts via checked_sub_signed instead of panicking; the exit-code doc now matches actual behaviour.
CI:
- check_log_analyzer_rules.sh is wired into the ci.yml test-and-lint job (it was only in make pre-pr, so anchor drift from other PRs could merge green).
Markdown table cells escape '|' so customer log text cannot break the table structure.
---------
Co-authored-by: houseme <housemecn@gmail.com>
|
||
|
Zhengchao An
|
4faea7fcbc |
ci: bump repo-visuals-action to v1.3.0 (#5014)
Backward-compatible feature release; existing chart-style/animate/contributors inputs are unchanged and continue to work. |
||
|
Zhengchao An
|
53728a03d3 | chore: self-host contributor wall (#5006) | ||
|
Zhengchao An
|
825bf0e2d8 |
ci: self-host star history chart (#5001)
* ci: self-host star history chart * ci: use animated gradient star chart |
||
|
Zhengchao An
|
cf9e9c6fd5 |
fix(ilm): implement expire_restored delete semantics for restore expiry (#4950)
DeleteRestoredAction is supposed to demote a restored object back to its pure transitioned state: remove only the local restored copy, strip the x-amz-restore headers, and leave the version (and the remote tier data) untouched. expire_transitioned_object set opts.transition.expire_restored accordingly, but no delete path ever read the flag, so delete_object ran an ordinary delete: on unversioned buckets the whole object vanished and the free-version record scheduled remote tier cleanup (tier data loss); on versioned buckets the latest version got a spurious delete marker that replication propagated. Route expire_restored explicitly in SetDisks::delete_object before delete-marker resolution and replication dispatch: target the found version with FileInfo.expire_restored=true and return early. The FileMeta::delete_version layer already implements the semantics (strip restore headers, keep the version, hand back the local data dir); this wires it up. Also fix the action matching in expire_transitioned_object (extracted into transitioned_object_delete_opts): DeleteRestoredVersionAction previously fell through to the full transitioned-object delete, which removed the remote tier data of a noncurrent restored version. It now routes through the same restored-copy cleanup with the exact version id, matching MinIO's Action.DeleteVersioned()/DeleteRestored() dispatch. Re-enable test_restore_chain_local_read_expiry_keeps_remote_and_allows_ re_restore in the ILM Integration (serial) lane; add unit tests pinning the event->options routing and the filemeta expire_restored branch. Closes rustfs/backlog#1302 |
||
|
Zhengchao An
|
c818177b54 |
test(ilm): re-enable test_transition_and_restore_flows; fix test-util disk-open and restore error-path lock (#4945)
test(ilm): re-enable test_transition_and_restore_flows; fix test-util disk-open and restore error-path lock (rustfs/backlog#1303) The excluded test's 'missing xl.meta ... on disk2' was NOT an EC metadata-distribution issue: after a transition all four shard disks hold a fully consistent xl.meta (verified by decoding each shard). The panic came from the tier test util's open_disk, which hardcoded disk_index 0 for every disk path; LocalDisk::new validates the endpoint's (set_idx, disk_idx) against the disk's own format.json and rejected every non-slot-0 disk with InconsistentDisk, which read_transition_meta collapsed into 'missing xl.meta'. Derive the real indices from format.json instead. This also un-breaks free_version_count / wait_for_free_version_absence for non-first disks (silently 0 before). With that fixed, the test advanced to the #4877 restore self-deadlock, whose main paths #4886 already fixed. Complete that fix on the one path it missed: update_restore_metadata (the restore-failure metadata rewrite) still rebuilt copy_object options with no_lock=false and would re-acquire the object write lock the restore handler already holds. Propagate the caller's no_lock there too. Remove the test from the serial-lane exclusion list; the four remaining exclusions are unrelated known issues and stay. |
||
|
Zhengchao An
|
701c3eee5b |
fix(ilm): replace whole-copy-back restore lock with accept-path CAS (#4956)
fix(ilm): serialize RestoreObject accepts with a CAS guard, not the whole copy-back Implements the backlog#1304 decision: replace #4877's object write lock held across the entire tier copy-back with an atomic compare-and-set on the restore ongoing flag. - ecstore: add ECStore::acquire_restore_accept_guard + RestoreAcceptGuard (opaque, purpose-scoped object write lock with an is_lock_lost fence); handle_restore_transitioned_object no longer locks the copy-back, so HEAD/get_object_info stay non-blocking during a restore and the inner put_object/complete_multipart_upload commit locks no longer self-deadlock. - API: execute_restore_object holds the guard across the restore-status read-check-write (no_lock inside the scope), drops it before spawning the copy-back; SELECT-type restores keep the plain read-locked path; the copy-back pins the resolved version so a concurrent PUT cannot strand the flagged version at ongoing=true; concurrent restores are rejected with 409 RestoreAlreadyInProgress (was a retryable 500) and guard contention maps to 503 SlowDown. - tests: drop the #4877 entry-blocking unit test (semantics deliberately reversed), pin accept-guard mutual exclusion, update the ilm-8 restore integration test to the final semantics, and add a concurrent double-POST test asserting exactly one acceptance and one tier GET. |
||
|
Zhengchao An
|
dc4b85eb2d | ci: log console version and completion after asset download (#4910) | ||
|
Zhengchao An
|
381639fbbc | fix(release): require embedded console assets (#4907) | ||
|
Zhengchao An
|
d5baaa67b8 |
ci: restore latest.json updates for prerelease tags (#4903)
PR #4582 gated the update-latest-version job to stable release tags so prereleases could not overwrite the stable version pointer. But the project currently ships prerelease tags only (beta previews), so the job never runs and latest.json has been stale since. Run the job for every release tag again, and keep the honest half of the #4582 fix by writing release_type from the actual build type instead of hardcoding "stable". |
||
|
houseme
|
8a126bb176 |
ci: drop macOS x86_64 release target (#4899)
Remove the Intel macOS entry from the Build and Release platform matrix so official release builds only publish the Apple Silicon macOS binary. Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
Zhengchao An
|
ae15f5804d |
test(ilm): fix restore integration test object key to match transition filter (#4886)
* test(ilm): fix restore test object key to match transition filter restore_object_usecase_reports_ongoing_conflict_and_completion used the object key "restore/api-object.bin", but the shared set_bucket_lifecycle_transition_with_tier helper only transitions objects under the "test/" prefix. enqueue_transition_for_existing_objects therefore matched nothing and wait_for_transition timed out at 15s, failing the test deterministically. The test was added in #4860 but its ILM Integration (serial) lane is skipped on regular PRs, so it merged red and has failed on every main run since. Move the object under the test/ prefix like every passing sibling test in this file. * ci(ilm): exclude broken RestoreObject API test from serial lane restore_object_usecase_reports_ongoing_conflict_and_completion exposes a real regression, not a test bug: the RestoreObject copy-back (handle_restore_transitioned_object) now holds the object write lock added in #4877 across the entire tier read-back, so it never releases in time and the test's concurrent get_object_info times out with Lock(Timeout, 5s). The failure is deterministic and independent of the mock tier's injected latency. This is the same class of known-broken restore/transition failure already tracked under backlog#1148 (three sibling scanner tests are excluded here by name for the same reason), so exclude this one the same way until the restore copy-back path is fixed or the #4877 lock scope is revisited. The prior commit keeps its correct fix (the object key must live under the test/ transition prefix); that was masking this deeper issue by never letting the object transition in the first place. Restore copy-back deadlock/hang under the #4877 lock is escalated separately for a product-level decision (fix the copy-back vs. narrow/revert #4877). * test(ilm): fix scanner restore test object keys to match transition filter test_restore_chain_local_read_expiry_keeps_remote_and_allows_re_restore and test_multipart_restore_preserves_parts_and_etag (both added in #4860) keyed their objects under restore/ instead of the test/ prefix that set_bucket_lifecycle_transition_with_tier filters on, so the objects never transitioned and wait_for_transition timed out at 15s. These surfaced only after the prior commit excluded the rustfs-side restore API test: nextest runs -j1 fail-fast, so that earlier failure stopped the run before these scanner tests executed. Unlike the excluded API test, both call restore_transitioned_object().await sequentially and only read afterwards, so they don't hit the concurrent-read-vs-#4877-write-lock timeout; the key prefix was their only problem. * ci(ilm): exclude the two remaining #4877-broken restore tests test_multipart_restore_preserves_parts_and_etag and test_restore_chain_local_read_expiry_keeps_remote_and_allows_re_restore both call restore_transitioned_object().await, which since #4877 acquires the object write lock and deterministically times out (Lock Timeout, 5s) against an already-held lock, so restore never completes. They surfaced one at a time because nextest runs -j1 fail-fast. The earlier prefix fix was necessary but only advanced them from the transition wait to this restore-lock timeout. Exclude both by name alongside their already-excluded sibling test_transition_and_restore_flows (same root cause, tracked under backlog#1148) so the ILM Integration (serial) lane goes green. The #4877 lock scope still needs a product fix before any of these re-enable. * docs(ilm): describe the excluded restore tests' symptom as a lock timeout, not a deadlock The #4877 write lock is held across the tier read-back and outlives the 5s lock timeout; nothing proves a true deadlock. Wording flagged by Copilot review. * fix(ecstore): stop restore copy-back self-deadlocking on the #4877 write lock #4877 made handle_restore_transitioned_object hold the object write lock for the whole restore and forward no_lock=true so the set layer would not reacquire it. But the set-level copy-back rebuilds its own options (put_restore_opts -> ropts, and the complete_multipart_upload opts) that default no_lock=false, so the inner put_object / new_multipart_upload / complete_multipart_upload each re-acquire this object's write lock in their commit phase and block on the lock the restore already holds -> Lock(Timeout, 5s), and restore never completes. Confirmed via RUSTFS_OBJECT_LOCK_DIAG_ENABLE: restore_transitioned_object acquires the write lock, then holds it ~10.5s across two nested 5s acquire timeouts before failing. This is a real product deadlock: a RestoreObject on any transitioned object (multipart especially) hangs, not just the tests. Propagate no_lock into the copy-back options so the inner writes inherit the already-held lock. Use opts.no_lock (not a hardcoded true) so a caller that restores without the outer lock still locks correctly. put_object_part is left as-is: it locks the multipart upload-id resource, not the object key, so it does not conflict. Verified test_multipart_restore_preserves_parts_and_etag now passes (3.6s, was a 15s+ hang). * ci(ilm): re-enable multipart restore test; scope remaining exclusions The prior commit fixes the #4877 restore self-deadlock, so test_multipart_restore_preserves_parts_and_etag passes again - drop it from the serial-lane exclusion list and remove its 'currently excluded' note. The other restore/transition tests still fail, but each on a DIFFERENT, independent issue unrelated to the (now-fixed) lock, verified locally: - test_restore_chain_...: DeleteRestoredAction sets expire_restored but no delete path reads it, so cleanup deletes the whole object (unimplemented semantics), not the local restored copy only. - test_transition_and_restore_flows: transition xl.meta missing on one drive (EC metadata distribution), not restore. - restore_object_usecase_reports_ongoing_conflict_and_completion: asserts a concurrent mid-restore ongoing=true read that #4877's read-vs-restore serialization rules out (backlog#1148 ilm-8 criterion 1, an API-semantics decision). Comments and #[ignore] reasons updated to reflect each real cause. All remain tracked under backlog#1148. |
||
|
Zhengchao An
|
61dbaf60d3 | ci: install zip and aws CLI on self-hosted runners (#4897) | ||
|
Zhengchao An
|
7b3d9e0c04 | ci: install C build tools before compiling s3s-e2e on custom runners (#4894) | ||
|
houseme
|
0fa6dc5946 |
ci: switch Linux builds to custom runners (#4884)
Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
Zhengchao An
|
b3a781bce0 | test(replication): pin SSE replication contracts (#4883) | ||
|
Zhengchao An
|
49e04bf343 | test(replication): fix nightly site peer setup (#4882) | ||
|
Zhengchao An
|
299b739f9f | fix(replication): stop active-active replay loops (#4878) | ||
|
houseme
|
35c5693486 | ci: switch linux builds to ubicloud runners (#4873) | ||
|
houseme
|
8f15dd2cef |
ci: add zip packaging fallback (#4872)
ci: add python zip packaging fallback |
||
|
houseme
|
5ef2731a6b |
ci: tune build workflow runners (#4871)
* chore: extend build timeout and trim dev deps * ci: switch linux builds to ubicloud runners * ci: use larger linux build runners |
||
|
Zhengchao An
|
242424b0fc |
ci(perf): fit the baseline cache build to the post-LTO profile and self-heal nightly misses (#4841)
* ci(perf): fit the baseline cache build to the post-LTO profile and self-heal nightly misses
Every build-baseline-cache run on 2026-07-15 died on its 60min ceiling
('exceeded the maximum execution time of 1h0m0s'): #4806 put thin LTO +
codegen-units=1 on [profile.release], pushing a single release build past
60min on sm-standard-2, so the baseline cache never populated. The measured
binary must keep the production profile, so raise the job budget to 100min
instead of weakening the profile.
Also make the A/B job self-heal a same-commit cache miss: when the candidate
commit equals the baseline commit (nightly/dispatch on main) and the restore
misses, build the binary once, reuse it for both phases, and save it back to
the cache under rustfs-baseline-<sha> — previously that miss made the rig
build the identical commit twice, which no job budget fits post-#4806. The
PR-context miss (candidate != baseline, opt-in gate) keeps the double-build
fallback and now documents that it will overrun and alert.
Refs rustfs/backlog#1152 (perf-3 follow-up).
* ci(perf): latest-wins concurrency for the baseline cache build
Consumers only restore the binary for the current origin/main tip, so a
superseded push build's output is dead weight; cancel it instead of stacking
~65min jobs on the shared sm-standard-2 pool when main merges quickly. A
skipped intermediate SHA at most costs one same-commit self-heal in the A/B
job.
|
||
|
Zhengchao An
|
f29455b32b |
ci: add e2e-full merge-gate job running the full single-node e2e suite (#4847)
* ci: add e2e-full merge-gate job (single-node full e2e via nextest) Adds the [profile.e2e-full] nextest profile and a matching e2e-full job in ci.yml (push main + merge_group + workflow_dispatch) that runs the never-automated user-visible e2e suites (KMS, object_lock, multipart_auth, quota, checksum, encryption, security-boundary, ...) the fast PR e2e-smoke subset skips. The filter is the whole e2e_test crate minus the sets other lanes own: protocols:: (ci-6/ci-7), the 6 RustFSTestClusterEnvironment cluster suites (ci-7 nightly), replication_extension_test (repl-1's smoke + repl-nightly lanes), and #[ignore]d tests (ci-13). The 4-disk reliability tests are serialized, mirroring the ci profile. Reuses the downloaded debug binary and uploads junit. backlog#1149 ci-5. * ci: exclude characterized known-failures from e2e-full with tracked issues First-ever automated run of the full suite (dispatch 29381309848: 341 ran / 32 failed / 460s) surfaced five real product-bug families, each now tracked: rustfs#4842 (extract 500s, mtime=0 OffsetDateTime), rustfs#4843 (path-limit vs ignore-errors), rustfs#4844 (anonymous POST SSE-S3 500), rustfs#4845 (object-lock POST + list metadata=true 403), rustfs#4846 (lock quorum misclassified as timeout under load). Deterministic failures cannot be retried away, so each family is excluded with its OPEN issue cited and the fixing PR contract to delete the exclusion — passing negative-path siblings stay in as regression guards. |
||
|
Zhengchao An
|
5fd2e8b6a1 |
ci(coverage): add weekly cargo-llvm-cov baseline workflow (#4820)
ci(coverage): weekly cargo-llvm-cov workspace baseline (non-blocking) Add the weekly line-coverage report (backlog#1153 infra-5): - .github/workflows/coverage.yml — Sunday 07:00 UTC + workflow_dispatch; runs `cargo llvm-cov nextest --workspace --exclude e2e_test` under NEXTEST_PROFILE=ci (same scope and profile as the ci.yml test gate), writes a per-crate line-coverage table to the job summary, uploads lcov + JSON as a 90-day artifact, and routes scheduled failures through the shared schedule-failure-issue action (ci-8). Runs only on schedule/dispatch, so it can never become a required PR check. - scripts/coverage_per_crate.py — stdlib-only aggregation of the llvm-cov JSON export into the per-crate markdown table (worst-first, TOTAL row), shared by the workflow and the local target. - make coverage (.config/make/coverage.mak) — local equivalent with the same command sequence; fails with install hints when cargo-llvm-cov or cargo-nextest are missing. Listed in make help. - docs/testing/README.md — Coverage section: cadence, where the table and artifacts live, the trend-comparison method, and what is not measured (doctests, e2e_test). |
||
|
houseme
|
04616e32c8 | ci: route build jobs through matrix runner labels (#4830) | ||
|
Zhengchao An
|
d73c8a783a |
ci(perf): cache the main baseline binary to cut the nightly double build (#4816)
Every push to main now builds the release binary once and stores it in the actions cache as rustfs-baseline-<sha> (build-baseline-cache job). The A/B job restores it for origin/main and passes --baseline-bin; on the nightly, where the candidate commit equals the baseline, one cached binary serves both phases with --skip-build and the run does zero source builds. A cache miss falls back to the source double-build via --baseline-ref origin/main. This removes the ~32min-per-side double build that pushed the 24-cell nightly past its 120min ceiling (2026-07-11..07-14 all cancelled on timeout). Also: - alert-on-failure now fires on cancelled/timed-out, not just failure, so a timed-out nightly is no longer silent (the composite action already reports cancelled jobs); removed the stale perf-2 TODO now that the alert job exists. - run_hotpath_warp_ab.sh appends a Provenance section to gate.md (baseline and candidate SHAs + binary source, runner, warp version, matrix params) via a repeatable --provenance-note; the gate exit code is preserved. Refs rustfs/backlog#1152 (perf-3). |
||
|
Zhengchao An
|
0e7b8ea16b |
test(security): wire negative-auth suites into e2e-smoke with a count-floor guard (#4815)
The header-SigV4 (sec-1), presigned-URL (sec-2), and admin-gate (sec-4) negative auth-rejection e2e suites merged earlier but only presigned_negative was actually selected by any CI profile; negative_sigv4_test and admin_auth_test compiled and sat unrun. Add both to the e2e-smoke default-filter so all three attacker-facing S3 auth-rejection suites execute on every PR. They already meet the smoke admission criteria (RustFSTestEnvironment, random ports, parallel-safe, no #[ignore], no feature gates), so this is a pure filterset change — the single e2e-in-CI wiring mechanism (backlog#1149 ci-4), not a new job. Because the filter selects by module name, a rename or deletion could silently drop a suite out of the security gate with no CI signal. Add scripts/check_security_smoke_count.sh (infra-12 count-floor mechanism): it lists what the e2e-smoke profile selects and fails if the count of security auth-rejection tests drops below the committed floor in .config/security-smoke-floor.txt (16). Invoked from the e2e-tests job, before the smoke run, so the nextest list compiles the binaries the run reuses. The GHSA-3p3x FTPS/WebDAV constant-time e2e (protocols::test_protocol_core_suite) stays out by topology: it binds fixed ports, needs the ftps,webdav features, and is #[serial], so it cannot join the random-port default-feature smoke profile as a filterset change (global ruling G5). Its GHSA-r5qv sibling is a unit test that already runs in the default CI pass. docs/testing/security-regressions.md now carries the full CI-execution map and flags the GHSA-3p3x e2e CI-lane gap as a ci-domain follow-up. Refs: backlog#1151 (sec-5) |
||
|
Zhengchao An
|
c111d25a6c |
ci(mint): set RUSTFS_UNSAFE_BYPASS_DISK_CHECK so the mint container boots (#4814)
ci(mint): bypass local physical-disk-independence guard so mint boots The mint container maps four RUSTFS_VOLUMES data dirs onto a single runner device, so the startup physical-disk-independence guard aborts with a FATAL before mint can run. The scheduled run 29183544431 (2026-07-12) crashed at 'Wait for RustFS ready' with 'local erasure endpoints must use distinct physical disks'. Set RUSTFS_UNSAFE_BYPASS_DISK_CHECK=true on the container, the CI use the guard explicitly sanctions -- mirroring the e2e-s3tests harness fixed in #4768. Completes the ci-2 acceptance (a mint run that actually produces log.json and the per-suite summary). |
||
|
houseme
|
ea2e24ac13 |
test/ci(ecstore): fix MinIO SSE interop size assertion + nightly dockerized interop check (#4809)
* test(ecstore): assert decrypted_size for MinIO SSE interop round-trip The ignored MinIO interop round-trip tests asserted `ObjectInfo.size` against the plaintext length. For SSE objects `size` is the on-disk DARE-encrypted size (plaintext + 32 bytes per 64 KiB block), so the assertion can never hold once real fixtures are present — the two `#[ignore]` tests failed the moment a real MinIO-written fixture was fed in, even though the decoded data was byte-identical. The client-visible object size comes from `decrypted_size()` / `get_actual_size()`, which correctly reads MinIO's `x-*-internal-actual-size` metadata (verified: both SSE-S3 and SSE-KMS 8 MiB multipart fixtures now report 8388608). Assert against that instead and keep the plaintext length and SHA-256 data checks. With real 4-drive MinIO fixtures (RELEASE.2025-09-07) all four tests pass, confirming RustFS reads MinIO erasure-coded SSE objects with byte-identical data and correct logical size. Co-Authored-By: heihutu <heihutu@gmail.com> * ci(ecstore): nightly MinIO interop check + dockerized fixture capture Wire the ignored MinIO on-disk interop reader tests into a nightly, non-required CI job, and make their fixtures reproducible without a host MinIO install. - Dockerfile + capture_via_docker.sh: build a throwaway image carrying the official MinIO server binary (pinned RELEASE.2025-09-07) plus the fixture lab on a small Python base, then run `lab.py capture-matrix` to write the SSE-S3 / SSE-KMS multipart fixtures the tests consume. lab.py drives MinIO's S3 API directly, so no `mc` is needed. - .github/workflows/minio-interop.yml: nightly + manual workflow on GitHub-hosted ubuntu-latest (reliable Docker + Python, unlike the self-hosted fleet — see e2e-s3tests.yml infra note). Regenerates the gitignored fixtures each run and executes the #[ignore] reader tests. Not a PR gate. - README: document the Docker capture path. Validated end to end: the script builds the image, captures the two multipart cases, and `cargo nextest run --run-ignored ignored-only` passes all four interop tests. Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
houseme
|
1553dc3f62 |
Address P2 follow-ups from the 2026-07-10..12 merged-PR review (backlog#1210-1220) (#4783)
* fix(obs): open cleaner compression source with O_NOFOLLOW The compressor opened the source log via File::open, which follows a symlink at the final path component. Between the scanner selecting a regular file and this open, an attacker with write access to the log directory could swap the entry for a symlink (TOCTOU) pointing at, say, /etc/shadow, whose contents would then be copied into an archive. Open the source with O_NOFOLLOW on Unix so such a swap fails with ELOOP; the temp/archive path already refused symlinks, this closes the source side. Refs rustfs/backlog#1210 Co-Authored-By: heihutu <heihutu@gmail.com> * fix(obs): recompress instead of trusting leftover cleaner archives archive_header_ok only checked the first 2-4 magic bytes before treating an existing .gz/.zst as a completed prior result and letting the caller delete the source log. A file with valid magic but a truncated or forged body passes that check, so an attacker with write access to the log directory (or a crashed prior run) could plant such a stub and make the cleaner delete the real log without ever producing a usable archive — silent audit-data loss. Chosen fix: stop trusting cross-process leftovers entirely and always recompress the source in this pass, rather than fully decoding every leftover to validate it. Full-decode validation would add real CPU cost and decode-bug surface for a rare crash-recovery case; the existing atomic create_new+rename already overwrites whatever sits at the archive path (a planted symlink is replaced, never followed) with a freshly written, fsync'd archive, so a partial/forged leftover can never gate source deletion. This is the lowest-regression option. Refs rustfs/backlog#1211 Co-Authored-By: heihutu <heihutu@gmail.com> * fix(object-data-cache): cap memory-gate reservation at cache growth headroom The memory gate subtracts `admitted_since_refresh` from the snapshot's available bytes so a burst arriving faster than the 5 s refresh cannot over-allocate. That counter is GROSS: it only rolls over on the refresh and never rolls back when a fill is later evicted, cancelled, or loses the invalidation race. Under sustained high-throughput churn (net footprint flat and far below `max_capacity`) the raw counter balloons past the memory the cache actually holds, so `effective_available` collapses and the gate reports false memory pressure — skipping the hottest fills with SkippedMemoryPressure until the next 5 s refresh. This only lowers hit rate; it never returns wrong data and self-heals each refresh. Fix direction 1 (minimal regression): cap the reservation deduction at the cache's own growth headroom (`max_capacity - weighted_size()`) instead of letting the unbounded gross counter shrink the system-available budget. The cache can never hold more than `max_capacity`, so a burst adds at most that headroom of real memory before moka evicts to stay bounded (net-zero churn beyond that point) — capping the deduction there keeps the reservation honest without treating gross churn as growth. Chosen over net-accounting (direction 2, releasing bytes on every failure/cancel/eviction path) because that only plugs the leak on failed fills and would not address the core defect: churn of *successful* insert/evict fills over the 5 s window. It also touches only the gate plus one call site rather than every failure path in moka_backend. The cap only ever raises `effective_available`, so real memory pressure (a low snapshot at refresh) still suppresses fills; when the cache is at capacity the headroom is 0 and the deduction vanishes, correctly reflecting net-zero churn. `MokaBackend` now stores `max_capacity` and passes the live headroom into `allows_fill`. Adds targeted gate tests: gross churn far above headroom no longer falsely suppresses, yet the reservation still bounds a burst while the cache can genuinely grow. Refs rustfs/backlog#1212 Co-Authored-By: heihutu <heihutu@gmail.com> * test(ecstore): assert native O_DIRECT path runs in uring read test uring_preserves_o_direct_for_eligible_reads only compared bytes through LocalDisk::read_file_mmap_copy. On a filesystem that rejects O_DIRECT the read silently degrades to the buffered StdBackend fallback and the byte check still passes, so the test could go green without the native read_at_direct path ever executing -- a vacuous pass. Add a per-disk native_direct_reads counter on UringBackend, incremented only when pread_uring_direct completes, and rebuild the test to drive a real UringBackend's pread_bytes and assert the counter is non-zero (every eligible read went through the native tier). When io_uring or O_DIRECT is unavailable on the host filesystem (restricted CI runners, tmpfs), the test skips loudly via eprintln instead of asserting a tautology, while still checking byte-correctness on whatever tier served the read. The counter also gives a gray release a positive signal that the O_DIRECT tier is serving reads, not just a fallback count. Refs rustfs/backlog#1213 Co-Authored-By: heihutu <heihutu@gmail.com> * fix(ecstore): warn + count read-time EINVAL on native O_DIRECT reads classify_direct_read_error is only reached from the read side: the O_DIRECT open in pread_uring_direct already succeeded (an open-time refusal is handled earlier as DirectOpenError::ODirectRefused). So an EINVAL/EOPNOTSUPP arriving here is a read-time error on an fd the kernel accepted for O_DIRECT -- far more likely an alignment bug in the aligned read path than an unsupported filesystem. The old code latched the disk's native path off with only a once-per-disk debug trace, hiding a potential correctness regression behind a silent buffered-read downgrade. Diagnostics only: the fallback behaviour is unchanged (the native path is still latched off and the caller still reads via StdBackend). This adds a rustfs_io_uring_direct_read_einval_total counter and promotes the once-per-disk trace from debug to warn so an operator can see an alignment regression instead of an unexplained latency/CPU shift. Refs rustfs/backlog#1214 Co-Authored-By: heihutu <heihutu@gmail.com> * docs(ecstore): document data-blocks-first default and its tail-latency cost DEFAULT_RUSTFS_GET_DATA_BLOCKS_FIRST_READER_SETUP is true and must stay true: deferred-parity is the deliberate, already-rolled-out full-object GET default from backlog#1159/#923. Flipping it back to false in code would silently revert that rollout for every deployment that has not set the env var, so this commit only documents -- no behaviour change. The added notes explain what data-blocks-first does (schedule data shards up front, engage parity lazily on a missing/corrupt data shard), the known trade-off (parity is engaged late, so a slow-but-not-dead data drive raises GET p99 because the faster parity shards are not raced against it until a data shard is declared missing), and the operational rollback switch (RUSTFS_GET_DATA_BLOCKS_FIRST_READER_SETUP=false), which is intentionally an env override rather than a code default change. No metric was added: the low-risk observability hook for "slow data drive engaged deferred parity" would live at the deferred-stripe engage point, which is out of this file's scope; this change stays documentation-only to avoid touching the hot GET path. Refs rustfs/backlog#1215 Co-Authored-By: heihutu <heihutu@gmail.com> * docs(ecstore): document wide-directory walk stall hazard and tuning list_dir enumerates a whole directory in one os::read_dir call (count = -1), and the walk caller bounds that entire enumeration with the per-read stall budget (default 5s) as if it were a single read. For a wide, flat prefix -- one directory holding millions of immediate children -- a single readdir can exceed the budget on a healthy disk, trip DiskError::Timeout, and surface as a ListObjects 500 quorum failure though the drive is fine (a #2999 sub-class). This is documented, not rewritten: turning the one-shot readdir into a streaming/batched enumeration that refreshes the stall deadline between chunks is an architecture-level change with high regression surface (ordering, the count contract, quorum merge) and belongs in a separate follow-up. The supported mitigation today is operational, so the comments point wide-directory deployments at RUSTFS_DRIVE_WALKDIR_STALL_TIMEOUT_SECS and the high-latency drive-timeout profile, which widen the budget with no code change. Notes were added at list_dir, the scan_dir call site, and get_drive_walkdir_stall_timeout. No behaviour change. Refs rustfs/backlog#1216 Co-Authored-By: heihutu <heihutu@gmail.com> * docs(ecstore): document consumer-peek vs producer-stall coupling In list_path_raw the consumer's peek_timeout is drawn from the same source and same value (walkdir_stall_timeout, default 5s) as the producer-side walk stall budget, but the two measure different things: the producer stall bounds a single drive read, while the consumer peek bounds the gap between two ADJACENT entries arriving from a reader. Because they share a value, the consumer cannot wait meaningfully longer for the next entry than the producer is allowed to spend producing one. Walking a region dense with non-listable internal items can make a HEALTHY drive miss the budget between visible entries; the consumer then declares it stalled and detaches it, dropping a good drive from the merge and capping the "large prefix succeeds" guarantee. Documented, not decoupled: giving the consumer peek an independent, strictly-larger budget would cut these false detaches but equally delays detaching a genuinely dead drive and shifts listing tail-latency semantics, so it wants soak data before changing the default. The comment records the invariant any such follow-up must keep -- consumer peek >= producer stall, never stricter -- so it can never fail a drive before the producer would. No behaviour change. Refs rustfs/backlog#1217 Co-Authored-By: heihutu <heihutu@gmail.com> * fix(io-metrics): add time-based trigger for low-IOPS latency percentiles Percentiles were recomputed only every 128 IOs and seeded to 0, so a low-traffic deployment exported p95/p99 = 0/stale for a long time after startup. Add a 10s wall-clock trigger alongside the count throttle so the first recompute can fire before 128 samples accrue. Hot-path per-op mean update is unchanged. Refs rustfs/backlog#1218 Co-Authored-By: heihutu <heihutu@gmail.com> * test(e2e): cover codec-streaming parity under fault injection and NoSuchKey The codec-streaming compat A/B previously ran only against a healthy 4-disk EC set with successful full GETs: the DiskFaultHarness was constructed but never faulted, the error path was untested, and the range assertion silently compared legacy-vs-legacy (ranges always fall back to the duplex path), overstating what it proved. Add two genuinely-failable scenarios reusing the existing harness and fixtures: - Parity reconstruction A/B: take one data disk offline and re-run the full object matrix on both phases while the EC 2+2 set rebuilds each large object from the surviving shards. Assert codec == legacy byte-for-byte (sha256) and header-for-header, and assert the codec phase served the reconstructed objects with zero duplex-pipe fallback (the reader gate is drive-health-independent, so the codec fast path is really exercised through reconstruction). - NoSuchKey negative path: compare the HTTP status + S3 error code of a missing-key GET across the legacy and codec phases and require them to be identical (404/NoSuchKey), guarding against the codec env perturbing the error path. Also clarify the range-phase comment so it is not misread as codec-range correctness coverage: both sides are served by the same legacy range path, so the assertion only proves ranges keep working and keep falling back to legacy with the gates open. Verified: cargo check/--no-run pass and the test passes locally (1 passed; dup_codec=0 confirms the codec path ran). Refs rustfs/backlog#1219 Co-Authored-By: heihutu <heihutu@gmail.com> * ci(ecstore): exercise native O_DIRECT read path on an ext4 loopback The uring-integration leg ran on the runner's default TMPDIR, which may sit on tmpfs/overlayfs where open(O_DIRECT) fails and the native read_at_direct path silently latches off to the aligned StdBackend fallback. Mount a dedicated ext4 loopback and point TMPDIR at it so the real io_uring dep (bumped git->0.1.0->0.2.0->0.2.1) and the native O_DIRECT read path are actually covered rather than validated only by signature diffing. Refs rustfs/backlog#1220 Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
Zhengchao An
|
f84ba243a1 |
chore(ci): guard against committed planning docs and remove re-added ones (#4777)
chore(ci): guard against committed planning docs; remove re-added ones PR #4771 removed the docs/superpowers planning-doc archive and added a rule, but #4765 force-added two more (git add -f bypasses .gitignore): docs/superpowers/plans/2026-07-12-observability-single-writer.md and docs/superpowers/specs/2026-07-12-observability-single-writer-design.md — an agentic implementation plan and its design spec. Delete both. Close the enforcement gap so this cannot recur: - New scripts/check_no_planning_docs.sh fails if anything is tracked under docs/superpowers/, regardless of how it was added. - Wire it into make pre-commit/pre-pr/dev-check. - Run it in CI: ci.yml for code/mixed PRs, and ci-docs-only.yml (which was a green stub) so docs-only PRs can no longer slip a planning doc past the required 'Test and Lint' check. - Document the guard in AGENTS.md and the arch-checks skill. |
||
|
Zhengchao An
|
71497ba39b |
fix(ci): evaluate ILM every scanner cycle in lifecycle behavior lane (#4772)
The s3-tests lifecycle expiration cases (test_lifecycle_expiration, test_lifecyclev2_expiration, test_lifecycle_deletemarker_expiration) flaked with counts stalling one plateau behind (e.g. `assert 6 == 4`). Root cause: a compacted directory is only re-descended -- and its objects re-evaluated against ILM rules -- once every DATA_USAGE_UPDATE_DIR_CYCLES scanner cycles (default 16). At the lane's accelerated RUSTFS_SCANNER_CYCLE=2 that is ~32s between evaluations, so a Days=1 object due at debug_day (10s) is not actually expired until the next ~32s boundary (~42s) -- just past the test's 4*lc_interval (40s) poll window, so the list still returns the pre-expiry count. Set RUSTFS_DATA_USAGE_UPDATE_DIR_CYCLES=1 for this debug-only lane so compacted directories are re-descended every cycle and ILM fires within ~2s of the due time, comfortably inside the poll window. This does not touch the 4*lc_interval < 5*debug_day plateau invariant. |
||
|
Zhengchao An
|
b235762fdb |
fix(ci): unblock e2e-s3tests startup and create disk dirs for distributed volumes (#4768)
The scheduled e2e-s3tests sweep failed at "Wait for RustFS ready" in both topologies because the server never started (issue #4762). Two independent startup faults, both surfaced now that the local physical-disk-independence guard is enforced: - single: RUSTFS_VOLUMES=/data/rustfs{0...3} all live on one physical device on the runner, so the guard aborts startup. Set the CI-sanctioned RUSTFS_UNSAFE_BYPASS_DISK_CHECK=true (what the guard's own error message and the e2e tests already use). - multi: the entrypoint's process_data_volumes skipped every non-absolute entry, so the distributed URL form "http://rustfs{1...4}:9000/data/rustfs{0...3}" never created /data/rustfs0..3. LocalDisk init then aborts with VolumeNotFound because resolve_local_disk_root no longer auto-creates the disk root. This also broke the shipped .docker/compose/docker-compose.cluster.yaml for real distributed docker deployments. entrypoint.sh now: 1. Expands multiple {N...M} ranges per token (the URL form carries two). The previous single-pass expander collapsed it to "http://rustfs1" and dropped the disk path; it now re-scans until no ranges remain, operating on only the first brace to keep multi-range tokens intact. 2. Creates the local filesystem path for URL-form endpoints (stripping scheme://host:port) without appending them as CLI args — rustfs reads the distributed form from RUSTFS_VOLUMES directly. Absolute-path and default (/data) inputs expand byte-identically to before. The multi compose also gets the CI disk-check bypass, since the four disks share one device inside each node's container. |
||
|
Zhengchao An
|
e3533a4611 |
test(replication): cover version deletion convergence (#4764)
test(replication): cover version delete convergence |
||
|
Zhengchao An
|
0ed0760fa2 | fix(ci): restore lifecycle debug day to 10s to fix flaky expiration test (#4766) | ||
|
Zhengchao An
|
b449af160c | test(ci): stabilize lifecycle behavior checks (#4755) | ||
|
houseme
|
e742a540a4 |
test(cache): guard the body-cache eligibility gate against deny-list regressions (#1146) (#4742)
`full_object_plaintext_len` decides whether a body-cache hook hit may serve bytes in place of the erasure read. It is a fail-closed allow-list: it excludes every read whose `ReadPlan::build` applies some other transform (ranged/part, raw/data-movement, restore, encrypted, remote) with an early `return None`, then returns a `Some(..)` length only for the whole-plaintext cases. A newly added `ReadPlan` branch that nobody teaches this gate about falls through to `None` and safely bypasses the cache. Flip it to a deny-list and the same new branch silently serves bytes in the wrong representation — the backlog#1108 / #1109 / #1146 class of bug. The existing unit and e2e tests only cover the branches that exist today. This adds `scripts/check_body_cache_whitelist.sh`, a structural guard wired into pre-commit / pre-pr / dev-check and CI, that asserts every exclusion predicate and a `return None` still precede the first `Some(..)`. Reordering a predicate, dropping one, moving the positive return ahead of the gate, or renaming/removing the function all fail; wording, formatting, and adding a new exclusion in the same gate do not. Mutation-tested against all four regression shapes. This machine-enforces the structural invariant that backlog#1146 was kept open to guard by hand. Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
houseme
|
ffca98cdbf |
fix(ecstore): harden io_uring integration (#4726)
* fix(ecstore): close the fd-cache open-then-insert race with a generation guard (rustfs/backlog#1176) pread_uring's miss path opened a descriptor on the blocking pool and only then inserted it into the moka cache. moka's invalidations cover only entries present at call time, so a heal/delete commit that invalidated between the open and the insert could not stop the just-opened stale inode from being cached afterwards — serving the pre-heal/pre-delete inode for up to the TTL and defeating the heal. Add an invalidation generation to FdCache, bumped by invalidate_exact and invalidate_under before they touch moka. The read path snapshots the generation before opening and inserts via insert_if_fresh, which refuses the insert if the generation moved during the open and, with a post-insert re-check, removes the entry if an invalidation raced the insert itself. Reads that never miss are unaffected. Co-Authored-By: heihutu <heihutu@gmail.com> * fix(ecstore): invalidate the fd cache on the primary object-delete paths (rustfs/backlog#1175) The fd-cache invalidation contract was only wired into DiskAPI::delete, rename_file and rename_data, but object deletion almost never goes through LocalDisk::delete — DeleteObject(s) reach delete_version, delete_versions -> delete_versions_internal, and delete_paths, all of which remove a version's data dir (move_to_trash / rename_all staging) with no invalidation. A cached io_uring descriptor kept the deleted part.N inode readable for up to the TTL, so a GET in that window could still return deleted data. Invalidate every cached fd under the removed data dir at each site: in delete_version and delete_versions_internal the data_dir uuid and object path are in hand (invalidate_cached_fds_under(volume, "{path}/{uuid}")); delete_paths invalidates under each removed path. A later rollback that restores a data dir just causes the next read to re-open it. Co-Authored-By: heihutu <heihutu@gmail.com> * fix(ecstore): close remaining fd-cache invalidation gaps (rustfs/backlog#1177) Three residual paths could keep serving a stale descriptor: - delete_volume removed the whole bucket tree (remove_dir_all/remove_dir) with no invalidation, and the cache-hit read path skips the volume-access check, so a cached fd kept a removed object readable. Add invalidate_cached_fds_for_volume (a per-volume moka predicate) and call it after the bucket is removed. - A retired LocalDisk instance (renew_disk on reconnect builds a fresh one) kept its populated cache alive while still referenced by in-flight ops, so invalidations through the new instance never reached it. close() now clears the backend's cache via clear_cached_fds. - rename_data's post-commit rollback (a commit-metadata fsync failure under strict durability) restored the old data dir without dropping fds cached during the committed window; the streaming branch now invalidates the dst part fds on those rollback paths. The inline branch's rollback runs inside spawn_blocking and is left to the TTL backstop. Co-Authored-By: heihutu <heihutu@gmail.com> * fix(ecstore): narrow the io_uring latch classes to match StdBackend (rustfs/backlog#1171) The runtime degradation classification reused the probe-time restriction errnos, which the driver's C7 contract explicitly warns against, so a single per-file error could latch a whole disk off io_uring: - is_io_uring_unsupported no longer includes EACCES: at read time on an already-open fd it is per-file (an LSM hooks security_file_permission on every read) and StdBackend hits the same denial, so falling back masks nothing and a full-disk latch would be wrong. ENOSYS and EPERM (seccomp/LSM applied after startup) remain. EOPNOTSUPP is now classified per-path by the caller. - pread_uring_direct's read-error arm now mirrors StdBackend: an O_DIRECT-shape error (EINVAL/EOPNOTSUPP) latches only direct_uring.supported so eligible reads take StdBackend's aligned path, instead of over-latching the whole io_uring backend or never latching a read-side EINVAL at all. - try_new only negative-caches genuine restriction-class probe failures in URING_UNSUPPORTED_DISKS; an unexpected (possibly transient) probe failure now falls back without latching, so the next reconnect re-probes. Co-Authored-By: heihutu <heihutu@gmail.com> * feat(ecstore): log when a disk latches io_uring off at runtime (rustfs/backlog#1172) A probe-gated gray release was flying blind: the permanent per-disk `active` latch flipped with no log and no metric, so the only message operators ever saw was the startup "io_uring read backend enabled" line — which stayed true on dashboards even after the very first read latched the disk back to StdBackend forever. Add latch_active_off, which flips the latch with `swap` and logs the true->false transition exactly once at warn with a dedicated event constant, disk root, and errno. Both the buffered and O_DIRECT read paths use it. A fallback/latch metric counter and periodic export of the driver StatsSnapshot (cq_overflow, cancel_already) remain as follow-ups that need rustfs_io_metrics plumbing. Co-Authored-By: heihutu <heihutu@gmail.com> * chore(audit): correct the stale rustfs-uring license-allow rationale (rustfs/backlog#1181) The dependency-review allow said rustfs-uring is "pulled as a git dependency", but ecstore now pins it from crates.io. Update the rationale and scope the allow to the exact pinned version (pkg:cargo/rustfs-uring@0.1.0) so a future version bump forces a conscious re-review of the license/provenance claim instead of being waved through on an outdated justification. Co-Authored-By: heihutu <heihutu@gmail.com> * fix(ecstore): offload io_uring driver teardown off the tokio worker (rustfs/backlog#1170) UringBackend held Arc<UringDriver> and had no Drop, so when the last LocalDisk reference dropped in async context (disk reconnect via renew_disk, or shutdown), UringDriver's own Drop ran on that thread — sending Shutdown and joining each shard thread, which can block up to the bounded-drain timeout (5s) on a hung / D-state disk, stalling a tokio worker. Wrap the driver in ManuallyDrop (deref is transparent, so read call sites are unchanged) and add a Drop that takes the Arc and, when a runtime is present, drops it on a blocking thread so the potentially-blocking join never runs on a runtime worker. Off-runtime it drops inline. Co-Authored-By: heihutu <heihutu@gmail.com> * fix(ecstore): restore StdBackend read parity on the uring paths (rustfs/backlog#1173) Two byte-for-byte parity breaks against StdBackend on the io_uring read paths: - A zero-length read on an fd-cache hit returned Ok(empty) without any bounds check, while StdBackend and the uring miss path return FileCorrupt for an offset past EOF. Fstat the cached descriptor on the length==0 path and match. - reclaim_read_range fadvise(DONTNEED)'d the raw unaligned [offset, offset+len) range, but fadvise only drops fully-covered pages, so the head partial page stayed resident — whereas StdBackend's mmap path reclaims the page-aligned superset. Bitrot shards' 32-byte block headers keep offsets off page boundaries, so this diverged on the common case. Page-align the reclaim window to match the mmap path exactly. (The third parity item from the audit — a failed reclaim fadvise failing the read — is already parity: StdBackend's mmap path propagates the same fadvise error with `?`, so no change is needed.) Co-Authored-By: heihutu <heihutu@gmail.com> * fix(ecstore): bound worst-case in-flight memory by chunking huge uring reads (rustfs/backlog#1174) The driver's backpressure permits count operations, not bytes, and it zero-fills a full-size buffer per op, so a single unbounded read could pin ~length bytes per permit (128 permits x shards x up to ~2 GiB). ecstore passes a whole part's shard range as one pread_bytes with no upstream chunking. On the buffered path, split reads larger than URING_MAX_OP_LEN (128 MiB) into sequential chunks, awaited one at a time, so worst-case in-flight memory is bounded by permits x URING_MAX_OP_LEN per shard. The threshold is high enough that ordinary shard reads keep the single-op, zero-copy fast path unchanged. The O_DIRECT path (opt-in, alignment-constrained) is left for a follow-up. Co-Authored-By: heihutu <heihutu@gmail.com> * fix(ecstore): gate the io_uring fd cache on RLIMIT_NOFILE headroom (rustfs/backlog#1178) The fd cache holds up to FD_CACHE_CAPACITY (512) descriptors per disk, but try_new cannot know the disk count and nothing checked the process fd budget. On a bare-metal / non-systemd run with the common 1024 soft RLIMIT_NOFILE, two disks would already exhaust fds with EMFILE surfacing on reads and probes. Check the soft limit at try_new: enable the cache only with ample headroom (>= 16384), otherwise log a warning once and fall back to open-per-read. The packaged systemd unit sets 1,048,576, so tuned deployments are unaffected. Co-Authored-By: heihutu <heihutu@gmail.com> * test(ecstore): make io_uring test skips visible and gate non-vacuity (rustfs/backlog#1179) The ecstore io_uring tests degrade to a silent pass when io_uring is unavailable (bare `return`s or plain eprintlns), so a CI leg on a restricted runner never exercises the real UringBackend/FdCache/latch paths yet still goes green — an integration regression could merge unseen. Add uring_test_skip: it emits a grep-able `SKIP <name>` line and, when RUSTFS_URING_TESTS_MUST_RUN is set (a CI leg that guarantees io_uring, e.g. a seccomp=unconfined container), panics instead of skipping. Route the silent-skip sites through it. Wiring a dedicated CI leg that sets that env on a capable runner is tracked in the issue; this provides the enforcement mechanism. Co-Authored-By: heihutu <heihutu@gmail.com> * test(ecstore): cover delete_paths fd-cache invalidation (rustfs/backlog#1180) Add an end-to-end test that seeds the descriptor cache with a read, removes the part via disk.delete_paths (one of the primary object-delete entry points that does not go through LocalDisk::delete), and asserts the next read no longer returns the removed inode — pinning the invalidation added in #1175. The sharded cancel-routing half of #1180 is covered in the rustfs-uring PR. Co-Authored-By: heihutu <heihutu@gmail.com> * io_uring audit follow-ups: O_DIRECT chunking, inline invalidation, metrics, CI leg (backlog#1160) (#4729) * fix(ecstore): chunk large O_DIRECT reads too, bounding in-flight memory (rustfs/backlog#1174) The buffered read path already splits reads above URING_MAX_OP_LEN into sequential chunks; do the same for the O_DIRECT path, which was left for a follow-up. read_at_direct aligns each chunk's sub-range internally, and chunk sizes are a multiple of URING_MAX_OP_LEN so a boundary re-read is at most one block. Extract classify_direct_read_error so the single-op and chunked paths share one copy of the EINVAL/EOPNOTSUPP-vs-subsystem latch classification rather than duplicating it. Co-Authored-By: heihutu <heihutu@gmail.com> * fix(ecstore): invalidate cached fds on the inline rename_data rollback (rustfs/backlog#1177) The streaming rename_data branch invalidates cached part fds on its post-commit rollback paths, but the inline branch runs its commit and rollback inside a single spawn_blocking closure where the async invalidate cannot be called, so it was left to the TTL backstop. Capture the closure's result instead of `??`-propagating it: on error (a commit-metadata fsync failure under strict durability rolls the committed rename back), invalidate the dst part paths at the async level before returning. Inline objects keep their data in xl.meta rather than separate part inodes, so this is largely defensive, but it removes the caveat and keeps the two branches consistent. Co-Authored-By: heihutu <heihutu@gmail.com> * feat(ecstore): export io_uring latch/fallback and driver stats metrics (rustfs/backlog#1172) Complete the gray-release observability. Beyond the warn log added earlier, emit metrics so a dashboard can answer "how much traffic is on io_uring vs falling back, and is any disk degrading": - rustfs_io_uring_latch_off_total — a disk latching io_uring off at runtime. - rustfs_io_uring_read_fallback_total — each io_uring -> StdBackend read fallback (latched-off short-circuit, O_DIRECT error, buffered error). - a low-frequency per-disk exporter of the driver StatsSnapshot as gauges (in_flight, cq_overflow, cancel_already), spawned in try_new. It holds only a Weak reference so it never keeps the driver alive, and drops any temporary strong reference on the blocking pool so a last-reference UringDriver::Drop join never runs on an async worker (rustfs/backlog#1170). submit_errors is deliberately not exported yet: it is a field added in the unreleased rustfs-uring 0.2.0, and ecstore still pins 0.1.0. It lands once the dependency is bumped (rustfs/backlog#1181). Co-Authored-By: heihutu <heihutu@gmail.com> * ci: add a real-io_uring integration leg on ubuntu-latest (rustfs/backlog#1179) The existing self-hosted sm-standard runners cannot guarantee io_uring is available (a container seccomp filter can block io_uring_setup), so the ecstore uring tests degrade to a silent skip and never exercise the real UringBackend/FdCache/latch paths in CI. Add a job on GitHub-hosted ubuntu-latest, which runs a recent kernel with no container seccomp filter, running the uring-named ecstore tests with RUSTFS_IO_URING_READ_ENABLE=true and RUSTFS_URING_TESTS_MUST_RUN=1 — the non-vacuity gate makes the leg fail rather than skip if io_uring is unavailable, so an integration regression can no longer merge green behind a vacuous pass. Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
Zhengchao An
|
3f1acfe8a7 |
fix(docker): warn instead of rejecting default or missing credentials (#4728)
* fix(docker): warn instead of rejecting default or missing credentials The entrypoint hard-reject from #4278 broke the container-first UX and the repo's own scheduled e2e lanes (e2e-s3tests, mint boot rustfs-ci images with default credentials and died at startup). Maintainer decision: ship no baked-in credentials, warn instead of block. - missing credentials: warn and start; wording accounts for the RUSTFS_ROOT_*/MINIO_* alias sources the entrypoint does not inspect - default rustfsadmin via env/CLI/file: warn and start; the warning notes all-default pairs cannot derive an internode RPC secret - malformed config stays fatal: source conflicts, unreadable files, empty or whitespace-only values, flags missing their argument - present-but-empty env vars now hit the empty-value hard failure instead of running the binary with an empty root credential - empty/default checks trim CR and blanks like the binary; files without a trailing newline are no longer falsely rejected as empty - the no-baked-credentials guard covers all four Dockerfiles, and the test harness refuses hosts where /usr/bin/rustfs exists - e2e-s3tests/mint move to non-default credentials (rustfsadmin-ci), which also restores RPC-secret derivation for the multi-node lane GHSA-68cw fail-closed RPC derivation (#4402) is untouched; helm stays fail-closed by design. * chore(docker): reword entrypoint comment flagged by typos check |
||
|
Zhengchao An
|
f63af3df63 |
chore: retire completed-migration scaffolding, wire orphaned boundary check (#4719)
The ecstore/global-state migrations are done (backlog#815, #939, #1052 all closed). Review of every migration-era test/gate measure found three things actually retirable or broken — everything else is a live anti-regression guard and stays. Remove: - scripts/check_metrics_migration_refs.sh — guards a migration that finished: rustfs_metrics:: has zero hits, the metrics crate no longer exists, and the script was never wired into CI or make (only reference was one line in config-model-boundary-adr.md, also removed). - crates/obs init_metrics_collectors — the "backward-compatible alias kept during migration" the removed script was guarding. Zero callers; pure delegate to init_metrics_runtime. Archive (docs/superpowers/plans/, continuing the 2026-07 convention, with the standard archived banner): - startup-timeline.md, scheduler-baseline.md, profiling-numa-capability-inventory.md, kms-development-defaults-inventory.md — one-shot snapshots whose only consumer is the already-archived migration-progress ledger (their same-dir links there start resolving again after the move); zero script pins; fed the closed backlog#660/#665 architecture-review ledger. Fixed the one outbound link (startup-timeline -> readiness-matrix) that the move would have broken — check_doc_paths.sh deliberately does not scan plans/, so nothing else would have caught it. Wire (found orphaned by the same review): - scripts/check_extension_schema_boundaries.sh guards a live contract crate but was never invoked anywhere. Add lint-fmt.mak target, include in pre-commit/pre-pr/dev-check, add ci.yml Quick Checks step (job already installs ripgrep), sync the CONTRIBUTING.md enumerated list, and harden the script against a silently-passing rg probe when src/ is missing. Keep (verified live, documented so the next cleanup pass does not repeat this analysis): - scripts/check_architecture_migration_rules.sh — added a header stating it is a permanent boundary guard, not retirable migration scaffolding; 'migration' in the name is historical. - check_migration_gate_count.sh + floor, delete-marker e2e proof, all pinned docs, compat-cleanup-register sync, remaining inventories (referenced by live docs). Verification: all 7 guard scripts pass, actionlint clean, cargo check --workspace (excl e2e) clean, cargo fmt --check clean. Adversarially reviewed by two independent skeptic passes; their 7 findings (alias left behind, broken outbound link, missing banners, wrong backlog attribution, CONTRIBUTING drift, rg exit-2 hole, missing header rationale) are all folded in. |
||
|
Zhengchao An
|
4b83efaf36 |
ci(ilm): add gated s3-tests lane for lifecycle expiration cases (#4715)
* ci(ilm): move first batch of 5 s3-tests lifecycle expiration cases into a gated behavior lane (backlog#1148 ilm-10)
The 20 lifecycle cases in excluded_tests.txt were labeled "vendor-specific"
but the real blocker was the absence of a Ceph lc_debug_interval equivalent.
RUSTFS_ILM_DEBUG_DAY_SECS (ilm-5) now provides that, so Days>=1 expiration
behavior is testable in seconds.
These cases assert that objects/versions/uploads are actually removed by the
background scanner and the stale-multipart cleanup loop. They cannot join the
default single-server s3-implemented-tests gate: it disables the scanner, and a
global RUSTFS_ILM_DEBUG_DAY_SECS also shrinks the x-amz-expiration header that
the already-passing test_lifecycle_expiration_header_* cases assert on. So this
adds an isolated lane instead of putting them in implemented_tests.txt.
First batch (5), each verified against the pinned upstream source and the
RustFS evaluator/scanner/stale-multipart execution paths:
- test_lifecycle_expiration (prefix Days=1/Days=5, scanner delete)
- test_lifecyclev2_expiration (same via ListObjectsV2)
- test_lifecycle_noncur_expiration (NoncurrentVersionExpiration)
- test_lifecycle_deletemarker_expiration (ExpiredObjectDeleteMarker cascade)
- test_lifecycle_multipart_expiration (AbortIncompleteMultipartUpload)
Dropped from the batch, kept excluded with reason:
- test_lifecycle_expiration_days0: RustFS accepts Expiration{Days:0} (returns
200; only Days<0 is rejected in crates/lifecycle/src/core.rs validate()),
while AWS/this test expect InvalidArgument. Real validation gap.
Changes:
- scripts/s3-tests/lifecycle_behavior_tests.txt: new exact-node-id run set.
- scripts/s3-tests/run.sh: honor an IMPLEMENTED_TESTS_FILE override so a lane
can point at an alternate whitelist.
- .github/workflows/ci.yml: new s3-lifecycle-behavior-tests PR-gate job that
starts rustfs with RUSTFS_ILM_DEBUG_DAY_SECS=10, scanner enabled (cycle 2s,
no start delay) and a 2s stale-multipart cleanup interval, running the new
list serially.
- scripts/s3-tests/report_compat.py: recognize the behavior list so the weekly
scope=all sweep (plain server) does not misclassify expected failures.
- scripts/s3-tests/excluded_tests.txt: remove the 5 enabled cases; re-annotate
the remaining 15 lifecycle exclusions with real reasons + batch-2 plan.
- docs/architecture/s3-compatibility-matrix.md: sync counts (implemented 451,
excluded 274, behavior 5) and document the lane.
Refs backlog#1148 (ilm-10), master plan backlog#1155.
* fix(lifecycle): reject zero-day expiration/noncurrent/abort rules (backlog#1148 ilm-10) (#4722)
|
||
|
Zhengchao An
|
5c7c757a30 |
ci(repl): wire replication e2e suite into nextest profiles (backlog#1147 repl-1) (#4712)
Activate the 36 dormant replication e2e tests in crates/e2e_test/src/replication_extension_test.rs (zero ran anywhere before). Split via the ci-4 nextest profile mechanism, no hand-rolled cargo-test lane: - PR smoke (profile.e2e-smoke, existing e2e-tests job): the 20 fast bucket-replication tests (target-registration / replication-check / list / remove / delete admin paths) that validate config synchronously and never wait for async convergence. Each spawns its own single-node rustfs server(s) on random ports with isolated temp dirs, so parallel-safe by construction (serial_test's #[serial] is a no-op under nextest's process-per-test model; no test-group needed). - Nightly (profile.e2e-repl-nightly + .github/workflows/e2e-replication-nightly.yml): the remaining 16 = 6 slow data-plane tests + 9 _real_dual_node + 1 _real_single_node. Defined as 'replication module MINUS the PR allowlist' so new replication tests default to nightly and are never silently unrun. The nightly workflow builds the binary once, installs awscurl so the STS dual-node test runs (skips gracefully with a visible log line otherwise), and routes scheduled failures through .github/actions/schedule-failure-issue (ci-8). Explicit division of labor with ci-5 e2e-full: these run only here. Counts (cargo nextest list): e2e-smoke 83 (63 + 20), e2e-repl-nightly 16. Docs updated: e2e-suite-inventory.md, e2e_test/README.md. Refs backlog#1147 repl-1, backlog#1155. |
||
|
Zhengchao An
|
ca1463a6c5 |
ci: run e2e smoke subset via nextest e2e-smoke profile (#4674)
ci: run e2e smoke subset via nextest e2e-smoke profile (backlog#1149 ci-4) The e2e-tests job previously ran a single test (delete_marker_migration _semantics) out of ~400 in the e2e_test crate. Define a nextest profile.e2e-smoke whose default-filter selects 17 fast single-node dependency-free modules (63 tests) and run it in the job, reusing the downloaded debug binary. The profile is the single wiring mechanism for e2e tests in CI; admission criteria live in crates/e2e_test/README.md, and docs/testing/e2e-suite-inventory.md records authoritative per-module counts from cargo nextest list. |
||
|
Zhengchao An
|
515e14cd42 |
test(ilm): activate ignored ILM integration tests via serial CI lane (#4682)
Implements ilm-1 from the ILM test-strategy plan: the 33 #[ignore]d ILM integration tests (14 in crates/scanner/tests/lifecycle_integration_test.rs, 19 in rustfs/src/app/lifecycle_transition_api_test.rs) never ran anywhere. This adds a dedicated serialized CI job and repairs the app-layer suite, activating 29 of them. - ci.yml: new test-ilm-integration-serial job running the two ILM test surfaces with 'cargo nextest run -j1 --run-ignored ignored-only'. The tests share process-global singletons and fixed ports (9002/9003); serial_test's #[serial] does not serialize across nextest's process-per-test boundary, so -j1 is the serialization mechanism. - app suite repair: the 19 app tests rotted after the InstanceContext migration (usecases resolve the store via AppContext; the test env never installed one, so every store-touching call failed with InternalError 'Not init'). Add a test-only install_test_app_context helper behind the sanctioned context/runtime_sources boundaries and switch the tests from without_context() to from_global(). All 19 now pass. - delete test_lifecycle_transition_basic (+3 orphaned helpers): required an external MinIO at localhost:9000 and slept 1200s; superseded by the mock-tier tests in the same file. - fix a timing flake: poll free-version metadata removal instead of asserting a single read right after remote-object absence. - 3 scanner tests fail on main for product reasons (multipart restore UnexpectedEof; noncurrent transition/expiry after immediate compensation transition on versioned buckets); they keep #[ignore] with a backlog reference and are excluded from the lane by name. Lane: 29 tests, ~43s test time, green twice locally. Refs rustfs/backlog#1148 (ilm-1), rustfs/backlog#1155. |
||
|
Zhengchao An
|
12c44abce0 |
ci: add count-floor guard to migration-critical test gate (#4673)
The migration-proof step in ci.yml selects tests by name substring (data_movement / rebalance / decommission / source_cleanup / delete_marker), so a rename can silently thin the gate to zero with no CI signal. Move the filter expression into scripts/check_migration_gate_count.sh as its single source of truth: the script counts the selected tests with cargo nextest list, fails if the count drops below the committed floor in .config/migration-gate-floor.txt, and then runs the gate with the same filter so the check and the run cannot drift. The floor equals the exact selected-test count at landing (571), so any reduction fails CI and intentional shrinks must edit the floor file in the same PR, keeping gate changes visible in diffs. Refs rustfs/backlog#1153 (infra-12), rustfs/backlog#1155. Signed-off-by: Zhengchao An <anzhengchao@gmail.com> |
||
|
Zhengchao An
|
4310b55238 |
ci: add schedule-failure-issue composite action and wire nightly alerts (#4671)
Scheduled workflow failures previously went unnoticed (15 consecutive red weekly s3-tests sweeps, silent perf nightly failures). Add a reusable composite action that opens a tracking issue titled "[scheduled-failure] <workflow name>" — or appends a comment to the existing open one (dedupe key = workflow name) — with the run URL, run attempt, and failed job names, labeled "infrastructure". Wire it into the scheduled paths of e2e-s3tests, mint, fuzz (nightly) and performance-ab via a final alert-on-failure job gated by "always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')", with issues:write scoped to that job only. e2e-s3tests and mint previously had no permissions key; they now get workflow-level contents:read, reducing every other job's token. Add a dispatch-only drill workflow that forces a job failure and runs the action through the exact consumer wiring, for end-to-end verification. The ci-7 e2e nightly does not exist yet; it is recorded as a pending consumer in the action README. Refs: rustfs/backlog#1149 (ci-8), rustfs/backlog#1155 |
||
|
Zhengchao An
|
fd18b1df49 |
ci(perf): fix nightly warp A/B startup failure, make it diagnosable, add small-object + 10MiB cells (#4669)
Both nightly runs of the warp A/B rig failed at server bring-up: `endpoint http://127.0.0.1:9000/health did not become healthy` — exactly 60s after start. The server binds its public listener only after startup converges (erasure-format load + IAM); its own budget (RUSTFS_STARTUP_READINESS_MAX_WAIT_SECS) defaults to 120s, so the rig's 60s health poll gave up before a slow cold start on the shared sm-standard-2 runner finished. The rustfs.log lived under /tmp (not the uploaded target/hotpath-ab/), so the root cause was invisible in the artifact. Root-cause + diagnosability (scripts/run_hotpath_warp_ab.sh): - Health poll is configurable via --health-timeout, default 180s (> the 120s server startup budget). Fails fast if the local server process exits before becoming healthy instead of polling out the full window. - Server logs + a startup-env file now write under OUT_DIR/server-logs/ so they ride along in the CI artifact. On a health failure the rig dumps the last 50 log lines to the job log. Workflow (.github/workflows/performance-ab.yml): - On every run, write gate.md (or, on a pre-gate failure, the failing phase's server-log tails) into $GITHUB_STEP_SUMMARY; short artifact retention. - Short measurement matrix (--duration/--rounds/--cooldown) so the expanded matrix fits; timeout-minutes 90 -> 120 as a Phase-0 stopgap until perf-3 caches the baseline binary (the ~65min double build dominates). - TODO(ci-8/perf-2) hook comment for the failure-alert composite action. Workload cells (absorbed perf-4): add put-4kib/get-4kib (the #4221 fsync regression size, ~-10% @4KiB, previously invisible) and get-10mib (historical large-GET EOF size) to both the PR-labeled and nightly matrices — 6 workloads x 2 phases x 2 drive-sync = 24 cells. A 1KiB cell is deferred to protect the budget until perf-3. Refs rustfs/backlog#1152 (perf-1, absorbed perf-4), rustfs/backlog#1155. |
||
|
Zhengchao An
|
7cfd08d4f5 |
ci(mint): restore multi-SDK pipeline on ubuntu-latest + pin mint digest (#4668)
The mint multi-SDK compatibility pipeline had exactly one recorded run (28730530597, 2026-07-05), which died at "Enable buildx": the self-hosted sm-standard-4 pod it landed on had no /var/run/docker.sock, so `docker buildx create` failed to connect to the Docker API before any test ran. Same heterogeneous-fleet root cause ci-1 fixed for the weekly s3-tests sweep. - Pin the job to GitHub-hosted ubuntu-latest, which reliably ships a running Docker daemon + buildx + python3. Header note records the diagnosis and the dind-sm-standard-2 tradeoff. - Pin MINT_IMAGE default to the linux/amd64 digest for minio/mint:edge resolved 2026-07-10 (was the rolling :edge tag); workflow_dispatch can still override. Header comment records the refresh command. - Quote shellcheck-flagged vars and drop an unused loop var so actionlint passes with zero findings. - Left report-only (ci-3 adds baseline gating later, per adjudication G4) and a TODO(ci-8) alerting hook on the scheduled job. Refs: rustfs/backlog#1149 (ci-2), rustfs/backlog#1155 |
||
|
Zhengchao An
|
89ea931ee1 |
test(ci): strict nextest ci profile with quarantine + flake policy (#4666)
test(ci): add strict nextest ci profile with quarantine + flake policy Formalize the existing ecstore-serial-flaky mechanism into a strict CI gate (ci-10, absorbs infra-15; backlog#1149). - .config/nextest.toml: add [profile.ci] with global retries=0 (never mask a new race's first occurrence), fail-fast=false, and JUnit output at target/nextest/ci/junit.xml. Add a quarantine section where flaky tests get retries=2 under the ci profile only; each entry links one OPEN issue. First members are the two backlog#937 ecstore groups (concurrent_resend_same_part_commits_one_generation and store::bucket::tests::bucket_delete_*), which keep their existing ecstore-serial-flaky test-group serialization. Local default profile still never retries. - .github/workflows/ci.yml: run the main test step with --profile ci and upload the JUnit report (if: always(), 3-day retention, run-number in name). The migration-proof step stays on the default profile to avoid clobbering the ci JUnit artifact (its tests are not quarantined). - docs/testing/README.md: new skeleton (owned by backlog#1153 infra-11) holding the flake policy: discover -> open issue within 24h -> quarantine with issue link -> fix or delete within 30 days. AGENTS.md points to it. Refs: rustfs/backlog#1149, rustfs/backlog#937, rustfs/backlog#1155 |
||
|
Zhengchao An
|
f0b1202b65 |
ci(s3tests): fix weekly sweep runner infra (pin ubuntu-latest + setup-python) (#4665)
The weekly full ceph/s3-tests sweep (e2e-s3tests.yml, schedule path) has failed every recorded run. The 2026-07-05 run (rustfs/rustfs #28727691591) died at "Install Python tools" with `No module named pip`: the self-hosted `sm-standard-4` label is served by heterogeneous runners and the scheduled job landed on a minimal pod with neither `pip` (bare python3) nor `docker.sock`, so no test ever executed. Make the infrastructure assumptions explicit instead of trusting drifting self-hosted runner state: - Pin the job to GitHub-hosted `ubuntu-latest`, which reliably ships Docker, docker compose, python3 and pip. - Add an explicit actions/setup-python step before any pip usage so the workflow stays correct across runner-image drift. - Document the fix and rationale in the workflow header comment. Also quote the shell variables flagged by shellcheck (SC2086) in the Docker build/run steps and drop the unused loop variable (SC2034) so actionlint passes with zero findings on the changed file. The PR gate (ci.yml s3-implemented-tests) is unaffected: it avoids Docker via DEPLOY_MODE=binary and defers pip setup to run.sh's self-bootstrap. Refs: rustfs/backlog#1149 (ci-1), rustfs/backlog#1155 |