test(e2e): prove bucket config survives rc.5 upgrade and rollback
Add two upgrade-compatibility scenarios pinned to the on-demand-migration
series' on-disk surfaces: BucketMetadata's 44 -> 46 msgpack keys, the
fail-closed bucket-config reads of rustfs#7172, the encryption-gated PUT
path of rustfs#7183, and the default-on migration module of rustfs#7089.
The upgrade case writes versioning, SSE-S3 default encryption, a validated
replication target plus rule, lifecycle, tags, quota, a public access block,
a bucket policy and an object lock configuration with the pinned previous
release, then asserts each one reads back unchanged on the current build,
that list-remote-targets still reports the target, that writes to the
encrypted and plain buckets keep their encryption posture, that every
pre-upgrade object including a multipart one is byte-identical, and that an
unconfigured bucket reports no migration and still answers NoSuchKey.
The rollback case is the reverse: the current build writes the 46-key blob
and the previous release must decode it by skipping the two unknown keys.
* fix(ci): preserve reported functional suite failures
* fix(ci): isolate functional evidence and preserve every result
* fix(ci): exclude sensitive scratch files from suite artifacts
Two fixes for the functional test chain:
1. Report upload fails with 'jq: Argument list too long' when the base64
report is passed through '--arg content' (pool reports exceed the OS
argv limit; last night's pool run lost its Step Results report this
way). Write the base64 payload to a temp file and load it in jq via
--rawfile instead. Applied uniformly to all nine suite workflows
that share this upload step.
2. The security workflow cloned rustfs/auto-testing into the workspace
and then ran actions/checkout at the workspace root for the OIDC
live gate script, which wiped the auto-testing clone and killed the
suite with 'chmod: cannot access auto-testing/rustfs-security-test.sh'.
Check out the repository into the rustfs-repo/ subdirectory instead
and point RUSTFS_SECURITY_OIDC_LIVE_SCRIPT there.
Co-authored-by: rustfs-ci <ci@rustfs.com>
The workflow_dispatch inputs already accept arbitrary release tags, but
the run failed late and unclearly when a tag had no .deb asset, and the
from_version default pointed at 1.0.0-rc.4-preview.1, whose release
ships no .deb at all - so scheduled runs died on a 404 while installing
the old package.
- Add a fail-fast preflight that resolves each requested tag via the
GitHub release API and verifies the rustfs_<tag>_amd64.deb asset
exists before the suite starts, with an actionable error message
otherwise (e.g. 1.0.0-rc.4 ships only zip/sbom assets).
- Change the from_version default to 1.0.0-rc.3, the newest release
that actually ships a .deb asset.
- Reword the from_version/to_version descriptions so manual triggers
state the .deb-asset requirement and the nightly fallback.
- Pass PF_TESTING_GH_TOKEN as GH_TOKEN to the suite step for the gh api
release lookups, matching the other functional workflows.
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
* test(ecstore): require core invariant tests in existing CI lane
* test(ci): require a fresh core JUnit report
* test(ecstore): match sealed context fixture map type
* ci(upgrade): render an upgrade matrix in the report; fix from default
The upgrade report only ever showed the requested deb URLs and a case
table. The nightly chain runs died installing the OLD package (default
from_version 1.0.0-rc.4-preview.1 has no .deb asset on its release, and
release 1.0.0-rc.4 ships none either), leaving an empty Total: 0 report
with no indication of what was upgraded.
- Default from_version is now 1.0.0-rc.3 (ships rustfs_1.0.0.rc.3_amd64.deb).
Matches the auto-testing default from PR #32.
- The report generator also parses the [UPG-TOPO] lines the suite now
emits and renders an 'Upgrade Matrix' section: per topology and KMS
backend, the versions actually in place before/after (captured via
'rustfs --version' on the node) and the aggregated result. When the
suite dies before any topology completes, the matrix says so instead
of silently showing nothing.
* fix(ci): use English headers in the upgrade matrix table
* ci(heal,pool): render step results and version in the reports
The heal and pool-expansion reports were only a raw log tail: no
structured indication of which steps passed, no overall verdict, and no
version information for the cluster under test.
The suites now emit machine-readable lines (auto-testing PR):
[HEAL-STEP] <n> <desc> PASS|FAIL [POOL-STEP] <n> <desc> PASS|FAIL
[HEAL-VERSION] <ver> (node <n>) [POOL-VERSION] <ver> (node <n>)
[HEAL-RESULT] PASS|FAIL <detail> [POOL-RESULT] PASS|FAIL <detail>
Both report generators parse them and emit a '## Step Results' section
before the log tail: the version captured in place via 'rustfs --version'
on a node, the overall verdict, and a per-step table. When a run dies
before any step reports (old script or early crash), the table shows a
NOT RUN placeholder row instead of silently showing nothing.
* test(odm): drive the migration cases from an env-named source
The ODM e2e suite only ever migrates from the in-process fake source, so
path-style addressing, region handling, ETag shape and list pagination on
real implementations stay untested. OdmInteropEnv resolves the source from
RUSTFS_ODM_INTEROP_*, seeding into a per-run source_prefix so a shared real
bucket can host concurrent runs and every seeded key is removed afterwards.
A named provider with a missing variable is an error, never a silent
fallback to the fake source.
interop_test holds the four cases that run against either source, and the
e2e-odm-interop profile is the lane that selects them; e2e-full excludes
them, so its committed selection is unchanged. wait_until_odm_engaged
replaces the fake source's journal probe for the readiness wait, since a
real source keeps no journal.
* ci(odm): add the scheduled provider interop lane
on-demand-migration-interop.yml runs the interop cases against a pinned
MinIO container with a 5,000-object backfill - past the fake source's 4,096
version and journal caps - and the three-case minimum against AWS, R2 and
GCS when their ODM_INTEROP_* secrets exist, skipping with a summary note
when they do not. Each provider gets one JSON report merging the per-case
entries with the nextest JUnit, which stays authoritative for what ran.
Report-only and never required: it depends on third-party endpoints and on
secrets a fork does not have.
PR #7023 rewrote the handoff retry scripts with shell parameter
expansions collapsed into Actions expression syntax: ${GH_TOKEN:-},
${{attempt}}, ${{DISPATCHED:-0}}, ${{TITLE}} etc. GitHub parses
${{...}} as workflow expressions, and bare identifiers are invalid
there, so all seven shared-VM suite workflows (upgrade, s3-compat, kms,
tier, storage, heal, pool-expand) were rejected as invalid workflow
files on main.
Symptoms since 2026-09-01 23:11 +0800 (bba9347):
- every push to any branch produced 'failure' runs with no jobs
('This run likely failed because of a workflow file issue')
- the nightly functional chain dispatched rustfs-chain-upgrade at
17:08Z but the event was silently dropped: zero repository_dispatch
runs for all eight shared-VM suites overnight (only performance,
whose file was untouched, ran)
- the workflows API listed them by path instead of name
Fix: restore the shell expansions (${VAR}, ${VAR:-default}); quote the
expected-event name without legacy backticks; render the markdown fence
via printf so shellcheck can parse the block. actionlint and YAML
validation now pass clean on all eleven rustfs-*.yml workflows.
- New RustFS Replication Test workflow (rustfs-replication-test.yml):
standalone workflow_dispatch (suite selector bucket/site/all) and
repository_dispatch rustfs-chain-replication; runs on the shared
smoke-testing runner under the shared functional concurrency group.
- Suite never fails the workflow (continue-on-error): failures are filed
as redacted issues in rustfs/backlog (deduped per run) and the report is
uploaded to rustfs/dashboard functional-reports/replication/<date>.md.
- Security now hands off to Replication, making it the tenth and final
link: upgrade -> s3 -> kms -> tier -> storage -> heal -> pool ->
security -> replication (performance stays parallel on pf-testing).
- Depends on rustfs/auto-testing#27 (rustfs-replication-test.sh).
Co-authored-by: houseme <housemecn@gmail.com>
The repository_dispatch handoff step was continue-on-error with a single
attempt: if the call failed (token lacking contents:write, transient API
error), the chain stalled silently while every job stayed green.
Each handoff now retries 3x and, if all attempts fail, files an alert
issue in rustfs/backlog with the exact recovery command before exiting 1
(still continue-on-error, so suite workflows themselves never fail).
Problem: the nightly functional chain has not completed end-to-end.
Evidence from recent runs:
- workflow_run events are fire-and-forget: after KMS finished at 17:09Z
on 8/31 no tier run was created; rustfs-storage-test.yml has never run.
- 'if: conclusion == success' gates skip downstream suites on any
failure (security was skipped after pool failed on 9/1 01:48Z).
- rustfs-pool-expand-test.yml embedded a heal pass without
continue-on-error, so a heal failure failed the whole workflow.
Fixes:
- Add rustfs-functional-chain.yml: entry point that dispatches the first
suite via repository_dispatch; each suite hands off to the next with an
explicit, re-drivable API call instead of workflow_run triggers.
- Split heal out of the pool workflow (renamed to RustFS Pool Expansion
Test): heal now runs exactly once per chain, in rustfs-heal-test.yml
(storage -> heal -> pool).
- Every suite job gets continue-on-error so a failing test never fails
the workflow; failures are filed as issues in rustfs/backlog (report
+ redacted log tail) and the chain moves on.
- Clone rustfs/auto-testing with the PF token via 'gh repo clone' plus a
5-attempt retry loop (transient clone failures aborted whole suites).
- Stop rewriting functional/index.html from every suite (divergent
copies raced each other with stale SHAs); the canonical index now
lives in the dashboard repo.
- Standalone workflow_dispatch runs are unchanged and never forward the
chain; performance runs on its own runner, dispatched in parallel.
* fix(release): normalize development package versions
* ci: build only the rustfs release binary
---------
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
New RustFS Upgrade Test workflow (SUITE: upgrade) runs first in the
nightly functional chain:
- Nightly GNU Build -> Upgrade -> S3 -> KMS -> Tier -> Pool/Heal -> Security
- S3 compatibility now triggers on "RustFS Upgrade Test" completion, so an
upgrade regression gates the rest of the chain.
- Security suite moves to the end, after pool/heal, on the shared VMs.
- The upgrade suite drives auto-testing's rustfs-upgrade-test.sh
(UPG-101..402): seed golden data/identity/config on the OLD deb, upgrade
in place to the NEW deb, verify byte-identical preservation, and publish
functional-reports/upgrade/<date>.md.
- Add the Upgrade tab to every dashboard index writer so the shared
functional/index.html stays consistent.
The check_preview_release_workflow.sh script uses exact line matching
(grep -Fxq) to verify the cleanup-preview-releases job contains:
gh release delete "$preview_tag" --yes
The extra --repo flag is unnecessary in GitHub Actions context since
gh auto-detects the repository from GITHUB_REPOSITORY, and it causes
the Workflow Pin Report check to fail on all PRs.
The test workflows checkout the private rustfs/auto-testing repository, but
the default GITHUB_TOKEN only has access to rustfs/rustfs, so every checkout
failed with 'repository ... not found' (nightly runs on 2026-08-28).
Pass secrets.PF_TESTING_GH_TOKEN (the existing cross-repo PAT already used
by the performance workflow) to the auto-testing checkout steps in all three
workflows.
* test(heal): relative disk target and fail fast on terminal-but-short
The absolute 40 GiB heal target was calibrated to the background scanner
(auto-heal), which is now disabled for determinism; with only the explicit
heal the recovered node lands at ~36 GiB for 40 GiB survivors. Make the
success criterion relative: the outage node must reach at least 90% of the
least-used surviving node (absolute HEAL_TARGET_GB floor optional, default
0 = relative only).
Also fail fast when the heal task reaches a terminal success but the disk
target is not met (previously the monitor kept polling until timeout), and
drop the misleading 'progress absent' warning on the final (cleaned) task
response — mid-run progress is reported correctly.
Validated live: heal summary=finished, 0 failed, vm000/vm001=40GB,
vm002=40GB (target 36GB), test PASSED.
* test(heal): gate success on server verdict + data read-back, drop disk GB gate
The per-node disk-usage target (40 GiB / 90% of survivors) is not a
code-level invariant: EC distributes different shards per node, so the
final GB per node depends on the layout, not on heal correctness. Gate the
test on what the server actually verifies:
- Heal task terminal success (finished/completed) with objectsFailed == 0
(the server's per-object scan/repair verdict).
- S3 read-back verification: list the test bucket and GET a sample of
objects, requiring HTTP 200 for every read (end-to-end proof the data is
still reconstructable after repair). The GET uses a discard mode so
binary bodies are not captured (no null-byte warnings / SIGPIPE).
Per-node disk usage stays in the output as observability (with a warning if
the outage node gained no usage), not as the pass/fail gate. Removes the
heal_target_gb input and the relative-target logic.
Validated live: heal summary=finished, 0 failed, 20/20 objects read back,
vm002_used=40GB, PASS.
A fixed /tmp path (log file, final heal status, warp log) can be owned by
another user on the shared runner (e.g. a previous root run), which made the
github-runner user fail: tee could not append the test log, the final heal
status write killed step 6 with EACCES, and upload-artifact could not read
stale root-owned warp logs. The heal scenario itself had passed
(summary=finished, vm002 reached the target) before the status-save died.
- Log files fall back to a unique mktemp path when the configured path is not
writable (heal + pool scripts).
- The final heal status is written to a mktemp file (best effort).
- Workflow artifact uploads use globs for the fallback names.
* test(heal): add node-outage heal E2E script and workflow
RustFS heal test on the 3x4 cluster (3 nodes x 4 disks, same
RUSTFS_VOLUMES expression on every node): write data with warp, stop the
outage node mid-write, restart it, start cluster heal via the admin API,
and pass only when the heal task finishes with 0 failures AND the outage
node's disk usage reaches the target.
Includes the GitHub Actions workflow (smoke-testing runner, nightly deb by
default) and a README. Validated end-to-end on the test environment:
40/40/16 GiB before heal -> 40/40/40 GiB after heal, summary=finished.
The script also writes RUSTFS_HEAL_TASK_TIMEOUT_SECS (default 6h) into the
node config because the server default (5 min) is far too short for
healing tens of GiB.
* ci(pool-test): chain heal regression after the pool test
The pool-expansion workflow is now triggered by the Nightly GNU Build
(workflow_run, replacing the schedule) and runs two sequential jobs on the
shared test environment:
1. pool-expansion-test (existing) — skipped if the nightly build failed.
2. heal-test — runs after the pool test regardless of its outcome
(if: always()): a pool failure makes the run red but does not block the
heal regression. Runs the heal script (reset -> install/start 3x4 ->
write/outage -> heal -> verify -> reset).
* test(heal): address review — camelCase progress, fail-closed, workflow hygiene
- Heal progress fields are camelCase in the API (objectsScanned/objectsHealed/
objectsFailed/progressPercentage); read them with a snake_case fallback and
distinguish null (absent) progress from zero, logging null as evidence
(rustfs/backlog#2035) instead of silently coercing.
- Fail closed in step 3: the outage node must actually be inactive after stop,
the write target must be reached, and an unobserved outage or incomplete
write fails the test instead of warning.
- Step 4 waits (bounded) for the cluster to report an active pool after the
outage-node restart instead of swallowing the verification error.
- Heal start fails fast on 400/403 (deterministic request/auth problems) and
only retries transient server errors.
- Disable the background scanner (RUSTFS_HEAL_AUTO_HEAL_ENABLE=false) so the
explicit heal is the only repair mechanism and the outage is observable.
- Workflows: heal and pool share one concurrency group; workflow_run requires
an exact successful nightly conclusion; checkout is pinned to the triggering
SHA; comma-separated step args are quoted (actionlint SC2054).
Zhengchao An