The scheduled e2e-s3tests sweep failed at "Wait for RustFS ready" in both
topologies because the server never started (issue #4762).
Two independent startup faults, both surfaced now that the local
physical-disk-independence guard is enforced:
- single: RUSTFS_VOLUMES=/data/rustfs{0...3} all live on one physical
device on the runner, so the guard aborts startup. Set the
CI-sanctioned RUSTFS_UNSAFE_BYPASS_DISK_CHECK=true (what the guard's own
error message and the e2e tests already use).
- multi: the entrypoint's process_data_volumes skipped every non-absolute
entry, so the distributed URL form
"http://rustfs{1...4}:9000/data/rustfs{0...3}" never created
/data/rustfs0..3. LocalDisk init then aborts with VolumeNotFound because
resolve_local_disk_root no longer auto-creates the disk root. This also
broke the shipped .docker/compose/docker-compose.cluster.yaml for real
distributed docker deployments.
entrypoint.sh now:
1. Expands multiple {N...M} ranges per token (the URL form carries two).
The previous single-pass expander collapsed it to "http://rustfs1"
and dropped the disk path; it now re-scans until no ranges remain,
operating on only the first brace to keep multi-range tokens intact.
2. Creates the local filesystem path for URL-form endpoints (stripping
scheme://host:port) without appending them as CLI args — rustfs reads
the distributed form from RUSTFS_VOLUMES directly.
Absolute-path and default (/data) inputs expand byte-identically to before.
The multi compose also gets the CI disk-check bypass, since the four disks
share one device inside each node's container.
* fix(docker): warn instead of rejecting default or missing credentials
The entrypoint hard-reject from #4278 broke the container-first UX and
the repo's own scheduled e2e lanes (e2e-s3tests, mint boot rustfs-ci
images with default credentials and died at startup). Maintainer
decision: ship no baked-in credentials, warn instead of block.
- missing credentials: warn and start; wording accounts for the
RUSTFS_ROOT_*/MINIO_* alias sources the entrypoint does not inspect
- default rustfsadmin via env/CLI/file: warn and start; the warning
notes all-default pairs cannot derive an internode RPC secret
- malformed config stays fatal: source conflicts, unreadable files,
empty or whitespace-only values, flags missing their argument
- present-but-empty env vars now hit the empty-value hard failure
instead of running the binary with an empty root credential
- empty/default checks trim CR and blanks like the binary; files
without a trailing newline are no longer falsely rejected as empty
- the no-baked-credentials guard covers all four Dockerfiles, and the
test harness refuses hosts where /usr/bin/rustfs exists
- e2e-s3tests/mint move to non-default credentials (rustfsadmin-ci),
which also restores RPC-secret derivation for the multi-node lane
GHSA-68cw fail-closed RPC derivation (#4402) is untouched; helm stays
fail-closed by design.
* chore(docker): reword entrypoint comment flagged by typos check
Scheduled workflow failures previously went unnoticed (15 consecutive red
weekly s3-tests sweeps, silent perf nightly failures). Add a reusable
composite action that opens a tracking issue titled
"[scheduled-failure] <workflow name>" — or appends a comment to the
existing open one (dedupe key = workflow name) — with the run URL, run
attempt, and failed job names, labeled "infrastructure".
Wire it into the scheduled paths of e2e-s3tests, mint, fuzz (nightly) and
performance-ab via a final alert-on-failure job gated by
"always() && github.event_name == 'schedule' &&
contains(needs.*.result, 'failure')", with issues:write scoped to that
job only. e2e-s3tests and mint previously had no permissions key; they now
get workflow-level contents:read, reducing every other job's token.
Add a dispatch-only drill workflow that forces a job failure and runs the
action through the exact consumer wiring, for end-to-end verification.
The ci-7 e2e nightly does not exist yet; it is recorded as a pending
consumer in the action README.
Refs: rustfs/backlog#1149 (ci-8), rustfs/backlog#1155
The weekly full ceph/s3-tests sweep (e2e-s3tests.yml, schedule path) has
failed every recorded run. The 2026-07-05 run (rustfs/rustfs #28727691591)
died at "Install Python tools" with `No module named pip`: the self-hosted
`sm-standard-4` label is served by heterogeneous runners and the scheduled
job landed on a minimal pod with neither `pip` (bare python3) nor
`docker.sock`, so no test ever executed.
Make the infrastructure assumptions explicit instead of trusting drifting
self-hosted runner state:
- Pin the job to GitHub-hosted `ubuntu-latest`, which reliably ships Docker,
docker compose, python3 and pip.
- Add an explicit actions/setup-python step before any pip usage so the
workflow stays correct across runner-image drift.
- Document the fix and rationale in the workflow header comment.
Also quote the shell variables flagged by shellcheck (SC2086) in the
Docker build/run steps and drop the unused loop variable (SC2034) so
actionlint passes with zero findings on the changed file.
The PR gate (ci.yml s3-implemented-tests) is unaffected: it avoids Docker
via DEPLOY_MODE=binary and defers pip setup to run.sh's self-bootstrap.
Refs: rustfs/backlog#1149 (ci-1), rustfs/backlog#1155
Follow-ups to the compatibility harness rework:
- Weekly full sweep now runs as a matrix over both topologies: single
node and the 4-node distributed cluster. Manual dispatch keeps the
test-mode input.
- New mint workflow (weekly + dispatch) runs MinIO Mint against RustFS:
functional suites of real client SDKs and tools (awscli, mc,
aws-sdk-*, minio-*, s3cmd, ...), catching client-specific signing and
streaming edge cases that boto3-only ceph/s3-tests cannot. Report-only
for test failures with a per-suite summary table; fails only when mint
produces no results.
- New scripts/s3-tests/api_coverage.py quantifies API surface coverage
by diffing the s3s S3 trait (at the Cargo.toml pinned revision)
against the methods overridden in impl S3 for FS, distinguishing
NotImplemented defaults from delegating defaults (e.g. post_object).
Current state: 76/101 operations covered (75 overridden, 1 delegated).
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Reworks the S3 compatibility test harness for reproducibility and faster
feedback:
- Pin ceph/s3-tests to a fixed commit (S3TESTS_REV, fetch-by-SHA) so the
449-test PR gate is reproducible; previously every run cloned upstream
master, letting test renames or assertion changes break CI silently.
- PR gate (ci.yml s3-implemented-tests): MAXFAIL=0 + XDIST=4 so a single
CI round reports every failure in parallel instead of stopping at the
first one serially.
- Add TEST_SCOPE=all to run.sh to run the entire upstream suite, and
report_compat.py to diff junit results against the classification
lists (regressions, promotion candidates, unclassified tests).
- Rewrite e2e-s3tests.yml: delegate execution to run.sh (single source
of truth; also fixes the broken config generation that left S3_PORT
empty), add a weekly scheduled full sweep that fails only on whitelist
regressions, and fix the multi-node topology to a real distributed
cluster (endpoint-style RUSTFS_VOLUMES) instead of four independent
single-node stores behind a load balancer.
- Docs: rewrite stale .github/s3tests/README.md (marker-era strategy),
update scripts/s3-tests/README.md, fix dead build_testexpr.sh
reference in S3_COMPAT_WORKFLOW.md, drop legacy non_standard_tests.txt.
All 747 classified test names verified present at the pinned revision;
13 upstream tests are currently unclassified and will surface in the
first full-sweep report.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Zhengchao An