The nightly channel was hardwired to the repository's default branch:
scheduled runs built whatever GITHUB_SHA pointed at, and the concurrency
group even had 'main' in its literal name. For the GA cycle the channel
needs to track the release branch instead, and back to main afterwards —
ideally without editing this file twice.
- Scheduled builds now follow the NIGHTLY_BRANCH repository variable,
falling back to main when the variable is unset or empty. Switching
the channel is a variable change, not a code change.
- workflow_dispatch gains a input for ad-hoc builds of any ref;
empty input falls back to the branch the run was dispatched from.
- All three jobs (build, publish, kms-vault-lane) check out
NIGHTLY_BUILD_REF explicitly so every lane builds the same tree.
- The publish step's candidate manifest advertised source_sha with a
hard equality check against GITHUB_SHA. Under a ref override that is
wrong by construction (schedule pins GITHUB_SHA to the default branch
at trigger time), so the manifest now always records the actual
checked-out HEAD.
- Concurrency group is branch-aware so a release-channel build and a
manual main build do not cancel each other.
Pin the star history workflow to the freshly released v1.4.0 of
overtrue/repo-visuals-action, which adds the chart-layout input and a
refreshed editorial chart/contributor-wall style. The workflow keeps its
existing inputs, so chart-layout stays at the editorial default.
Since rc.1 the latest tag has been frozen at 1.0.0-beta.12 because the
docker workflow only allowed alpha/beta prereleases to update latest
(#2732 dropped the rc case). Before 1.0.0 GA, latest is expected to
track the newest test build.
- Prereleases (alpha/beta/rc) now update latest as long as no stable
vX.Y.Z tag exists on origin, so the rule retires itself at GA.
- Channel tags (alpha/beta/rc) are now always added for prereleases;
the previous if/elif skipped the channel tag whenever latest was
created, which is why no :beta tag was ever published.
fix(heal): fall back to set-wide format for directory-backed replacement targets
Since #7018 `renew_disk` routes an unformatted local endpoint through the automatic replacement heal, which requires the target to be an independently mounted disk. Directory-backed deployments (the operator set `RUSTFS_UNSAFE_BYPASS_DISK_CHECK`, which the startup disk-independence check already requires for endpoints sharing a device) can never pass that admission, so a runtime-wiped or replaced directory disk stayed unformatted forever: the heal task failed with "replacement target is not a stable mounted disk" and the auto-scan kept deferring the endpoint. This broke the Issue #1533 contract and the `heal_erasure_disk_rebuild_test` lane on main since 2026-09-02.
When the disk-check bypass is set and the replacement target fails admission, the erasure-set heal now logs a warning and downgrades to the ordinary set-wide `heal_format` path that predated replacement admission, and the auto-scan no longer defers such endpoints. The mount admission itself is unchanged and still cannot be bypassed by any environment variable; deployments without the bypass keep failing closed.
The endpoint-blackhole heal scenario now probes whether `iptables` can read the OUTPUT chain and logs an explicit skip when the host lacks `CAP_NET_ADMIN` (containerised runners report "Permission denied" from the nf_tables backend even under sudo); `RUSTFS_E2E_REQUIRE_NET_FAULT_INJECTION=1` turns that into a failure for lanes that provision the capability. The CI full-gate job surfaces the missing capability as a workflow warning, and the runtime-wipe fixture retries `remove_dir_all` on the listing race macOS surfaces as `DirectoryNotEmpty`.
Refs rustfs/backlog#2357.
* test(e2e): add distributed 4x4 validation
* ci(e2e): use tmpfs for distributed pool isolation
sm-standard-4 is an ARC pod without usable loop devices, so
mount -o loop fails with ENOENT before any pool filesystem is
attached. Sized tmpfs still gives each pool a distinct st_dev
and independent 1G statfs capacity.
Co-authored-by: RustFS <hello@rustfs.com>
* test(e2e): prove operations overlap data movement
---------
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
test(e2e): prove bucket config survives rc.5 upgrade and rollback
Add two upgrade-compatibility scenarios pinned to the on-demand-migration
series' on-disk surfaces: BucketMetadata's 44 -> 46 msgpack keys, the
fail-closed bucket-config reads of rustfs#7172, the encryption-gated PUT
path of rustfs#7183, and the default-on migration module of rustfs#7089.
The upgrade case writes versioning, SSE-S3 default encryption, a validated
replication target plus rule, lifecycle, tags, quota, a public access block,
a bucket policy and an object lock configuration with the pinned previous
release, then asserts each one reads back unchanged on the current build,
that list-remote-targets still reports the target, that writes to the
encrypted and plain buckets keep their encryption posture, that every
pre-upgrade object including a multipart one is byte-identical, and that an
unconfigured bucket reports no migration and still answers NoSuchKey.
The rollback case is the reverse: the current build writes the 46-key blob
and the previous release must decode it by skipping the two unknown keys.
* fix(ci): preserve reported functional suite failures
* fix(ci): isolate functional evidence and preserve every result
* fix(ci): exclude sensitive scratch files from suite artifacts
Two fixes for the functional test chain:
1. Report upload fails with 'jq: Argument list too long' when the base64
report is passed through '--arg content' (pool reports exceed the OS
argv limit; last night's pool run lost its Step Results report this
way). Write the base64 payload to a temp file and load it in jq via
--rawfile instead. Applied uniformly to all nine suite workflows
that share this upload step.
2. The security workflow cloned rustfs/auto-testing into the workspace
and then ran actions/checkout at the workspace root for the OIDC
live gate script, which wiped the auto-testing clone and killed the
suite with 'chmod: cannot access auto-testing/rustfs-security-test.sh'.
Check out the repository into the rustfs-repo/ subdirectory instead
and point RUSTFS_SECURITY_OIDC_LIVE_SCRIPT there.
Co-authored-by: rustfs-ci <ci@rustfs.com>
The workflow_dispatch inputs already accept arbitrary release tags, but
the run failed late and unclearly when a tag had no .deb asset, and the
from_version default pointed at 1.0.0-rc.4-preview.1, whose release
ships no .deb at all - so scheduled runs died on a 404 while installing
the old package.
- Add a fail-fast preflight that resolves each requested tag via the
GitHub release API and verifies the rustfs_<tag>_amd64.deb asset
exists before the suite starts, with an actionable error message
otherwise (e.g. 1.0.0-rc.4 ships only zip/sbom assets).
- Change the from_version default to 1.0.0-rc.3, the newest release
that actually ships a .deb asset.
- Reword the from_version/to_version descriptions so manual triggers
state the .deb-asset requirement and the nightly fallback.
- Pass PF_TESTING_GH_TOKEN as GH_TOKEN to the suite step for the gh api
release lookups, matching the other functional workflows.
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
* test(ecstore): require core invariant tests in existing CI lane
* test(ci): require a fresh core JUnit report
* test(ecstore): match sealed context fixture map type
* ci(upgrade): render an upgrade matrix in the report; fix from default
The upgrade report only ever showed the requested deb URLs and a case
table. The nightly chain runs died installing the OLD package (default
from_version 1.0.0-rc.4-preview.1 has no .deb asset on its release, and
release 1.0.0-rc.4 ships none either), leaving an empty Total: 0 report
with no indication of what was upgraded.
- Default from_version is now 1.0.0-rc.3 (ships rustfs_1.0.0.rc.3_amd64.deb).
Matches the auto-testing default from PR #32.
- The report generator also parses the [UPG-TOPO] lines the suite now
emits and renders an 'Upgrade Matrix' section: per topology and KMS
backend, the versions actually in place before/after (captured via
'rustfs --version' on the node) and the aggregated result. When the
suite dies before any topology completes, the matrix says so instead
of silently showing nothing.
* fix(ci): use English headers in the upgrade matrix table
* ci(heal,pool): render step results and version in the reports
The heal and pool-expansion reports were only a raw log tail: no
structured indication of which steps passed, no overall verdict, and no
version information for the cluster under test.
The suites now emit machine-readable lines (auto-testing PR):
[HEAL-STEP] <n> <desc> PASS|FAIL [POOL-STEP] <n> <desc> PASS|FAIL
[HEAL-VERSION] <ver> (node <n>) [POOL-VERSION] <ver> (node <n>)
[HEAL-RESULT] PASS|FAIL <detail> [POOL-RESULT] PASS|FAIL <detail>
Both report generators parse them and emit a '## Step Results' section
before the log tail: the version captured in place via 'rustfs --version'
on a node, the overall verdict, and a per-step table. When a run dies
before any step reports (old script or early crash), the table shows a
NOT RUN placeholder row instead of silently showing nothing.
* test(odm): drive the migration cases from an env-named source
The ODM e2e suite only ever migrates from the in-process fake source, so
path-style addressing, region handling, ETag shape and list pagination on
real implementations stay untested. OdmInteropEnv resolves the source from
RUSTFS_ODM_INTEROP_*, seeding into a per-run source_prefix so a shared real
bucket can host concurrent runs and every seeded key is removed afterwards.
A named provider with a missing variable is an error, never a silent
fallback to the fake source.
interop_test holds the four cases that run against either source, and the
e2e-odm-interop profile is the lane that selects them; e2e-full excludes
them, so its committed selection is unchanged. wait_until_odm_engaged
replaces the fake source's journal probe for the readiness wait, since a
real source keeps no journal.
* ci(odm): add the scheduled provider interop lane
on-demand-migration-interop.yml runs the interop cases against a pinned
MinIO container with a 5,000-object backfill - past the fake source's 4,096
version and journal caps - and the three-case minimum against AWS, R2 and
GCS when their ODM_INTEROP_* secrets exist, skipping with a summary note
when they do not. Each provider gets one JSON report merging the per-case
entries with the nextest JUnit, which stays authoritative for what ran.
Report-only and never required: it depends on third-party endpoints and on
secrets a fork does not have.
PR #7023 rewrote the handoff retry scripts with shell parameter
expansions collapsed into Actions expression syntax: ${GH_TOKEN:-},
${{attempt}}, ${{DISPATCHED:-0}}, ${{TITLE}} etc. GitHub parses
${{...}} as workflow expressions, and bare identifiers are invalid
there, so all seven shared-VM suite workflows (upgrade, s3-compat, kms,
tier, storage, heal, pool-expand) were rejected as invalid workflow
files on main.
Symptoms since 2026-09-01 23:11 +0800 (bba9347):
- every push to any branch produced 'failure' runs with no jobs
('This run likely failed because of a workflow file issue')
- the nightly functional chain dispatched rustfs-chain-upgrade at
17:08Z but the event was silently dropped: zero repository_dispatch
runs for all eight shared-VM suites overnight (only performance,
whose file was untouched, ran)
- the workflows API listed them by path instead of name
Fix: restore the shell expansions (${VAR}, ${VAR:-default}); quote the
expected-event name without legacy backticks; render the markdown fence
via printf so shellcheck can parse the block. actionlint and YAML
validation now pass clean on all eleven rustfs-*.yml workflows.
- New RustFS Replication Test workflow (rustfs-replication-test.yml):
standalone workflow_dispatch (suite selector bucket/site/all) and
repository_dispatch rustfs-chain-replication; runs on the shared
smoke-testing runner under the shared functional concurrency group.
- Suite never fails the workflow (continue-on-error): failures are filed
as redacted issues in rustfs/backlog (deduped per run) and the report is
uploaded to rustfs/dashboard functional-reports/replication/<date>.md.
- Security now hands off to Replication, making it the tenth and final
link: upgrade -> s3 -> kms -> tier -> storage -> heal -> pool ->
security -> replication (performance stays parallel on pf-testing).
- Depends on rustfs/auto-testing#27 (rustfs-replication-test.sh).
Co-authored-by: houseme <housemecn@gmail.com>
The repository_dispatch handoff step was continue-on-error with a single
attempt: if the call failed (token lacking contents:write, transient API
error), the chain stalled silently while every job stayed green.
Each handoff now retries 3x and, if all attempts fail, files an alert
issue in rustfs/backlog with the exact recovery command before exiting 1
(still continue-on-error, so suite workflows themselves never fail).
Problem: the nightly functional chain has not completed end-to-end.
Evidence from recent runs:
- workflow_run events are fire-and-forget: after KMS finished at 17:09Z
on 8/31 no tier run was created; rustfs-storage-test.yml has never run.
- 'if: conclusion == success' gates skip downstream suites on any
failure (security was skipped after pool failed on 9/1 01:48Z).
- rustfs-pool-expand-test.yml embedded a heal pass without
continue-on-error, so a heal failure failed the whole workflow.
Fixes:
- Add rustfs-functional-chain.yml: entry point that dispatches the first
suite via repository_dispatch; each suite hands off to the next with an
explicit, re-drivable API call instead of workflow_run triggers.
- Split heal out of the pool workflow (renamed to RustFS Pool Expansion
Test): heal now runs exactly once per chain, in rustfs-heal-test.yml
(storage -> heal -> pool).
- Every suite job gets continue-on-error so a failing test never fails
the workflow; failures are filed as issues in rustfs/backlog (report
+ redacted log tail) and the chain moves on.
- Clone rustfs/auto-testing with the PF token via 'gh repo clone' plus a
5-attempt retry loop (transient clone failures aborted whole suites).
- Stop rewriting functional/index.html from every suite (divergent
copies raced each other with stale SHAs); the canonical index now
lives in the dashboard repo.
- Standalone workflow_dispatch runs are unchanged and never forward the
chain; performance runs on its own runner, dispatched in parallel.
* fix(release): normalize development package versions
* ci: build only the rustfs release binary
---------
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
New RustFS Upgrade Test workflow (SUITE: upgrade) runs first in the
nightly functional chain:
- Nightly GNU Build -> Upgrade -> S3 -> KMS -> Tier -> Pool/Heal -> Security
- S3 compatibility now triggers on "RustFS Upgrade Test" completion, so an
upgrade regression gates the rest of the chain.
- Security suite moves to the end, after pool/heal, on the shared VMs.
- The upgrade suite drives auto-testing's rustfs-upgrade-test.sh
(UPG-101..402): seed golden data/identity/config on the OLD deb, upgrade
in place to the NEW deb, verify byte-identical preservation, and publish
functional-reports/upgrade/<date>.md.
- Add the Upgrade tab to every dashboard index writer so the shared
functional/index.html stays consistent.
xiaomage