Compare commits

..

1 Commits

Author SHA1 Message Date
overtrue 495e1d3b87 feat(scanner): observe cache read origins 2026-10-28 11:01:05 +08:00
1857 changed files with 81156 additions and 403174 deletions
+24 -7
View File
@@ -1,13 +1,30 @@
# Adversarial Review Shape
Use when root `AGENTS.md` triggers adversarial validation or for a substantial PR review. Paths below are repository-relative. The root finding standard and completion rule apply; selecting a lens does not require finding a defect.
Use when root `AGENTS.md` triggers adversarial validation or for a substantial PR
review. Paths below are repository-relative. The root finding standard and
completion rule apply; selecting a lens does not require finding a defect.
Risk and review shape:
- **Exempt:** documentation, comments, formatting, or typos with no runtime, build, test, or agent-execution effect.
- **Mechanical:** changes shown to preserve behavior, such as a symbol rename with unchanged callers or a file move with equivalent wiring. Run correctness and simplicity lenses. Classify tests, tooling, CI, and agent instructions by the behavior and authority they change, not their file type; deleting regression protection or changing failure propagation, publishing, or destructive actions can be high risk.
- **Standard:** localized behavior changes. Run one integrated final-diff pass covering correctness, simplicity, and test coverage; add only domain lenses matched by the diff.
- **High risk / substantial PR review:** high risk includes locking, erasure/quorum/heal, replication, multipart, RPC, lifecycle/tiering, persistence/fsync, IAM/KMS/auth, cryptography, on-disk/on-wire formats, and S3-visible semantics. Cover the applicable lenses from two perspectives: first trace the intended contract and implementation, then independently challenge its assumptions, failure paths, and evidence. Use separate reviewers when delegation is authorized and useful, or fresh sequential passes otherwise. Choose staffing and split lenses to fit the risk; a fixed reviewer count or duplicated review is not the goal.
- **Outbound client defaults:** what `TargetClient`, `PutObjectOptions`, or the remote SDK configuration sends to every replication or migration target is high risk for every target class even when the change fixes one. Follow the SOP in `docs/postmortems/2026-09-03-replication-checksum-default-regression.md`: run the outbound target matrix, document each new env knob in the same PR, and list verified and unverified target classes in the PR Impact section.
- **Exempt:** documentation, comments, formatting, or typos with no runtime,
build, test, or agent-execution effect.
- **Mechanical:** renames, moves, test/tooling-only changes, and agent-rule
changes. Run correctness and simplicity lenses.
- **Standard:** localized behavior changes. Run one integrated final-diff pass
covering correctness, simplicity, and test coverage; add only domain lenses
matched by the diff.
- **High risk / substantial PR review:** high risk includes locking,
erasure/quorum/heal, replication, multipart, RPC, lifecycle/tiering,
persistence/fsync, IAM/KMS/auth, cryptography, on-disk/on-wire formats, and
S3-visible semantics. Cover all applicable lenses using exactly two
independent reviewers when delegation is explicitly authorized. Split the
lenses between them. Otherwise perform two fresh sequential passes.
- **Outbound client defaults:** what `TargetClient`, `PutObjectOptions`, or
the remote SDK configuration sends to every replication or migration target
is high risk for every target class even when the change fixes one. Follow
the SOP in `docs/postmortems/2026-09-03-replication-checksum-default-regression.md`:
run the outbound target matrix, document each new env knob in the same PR,
and list verified and unverified target classes in the PR Impact section.
Available domain lenses are security, concurrency/durability, compatibility, and performance; select only those matched by the change.
Available domain lenses are security, concurrency/durability, compatibility,
and performance; select only those matched by the change.
+50 -57
View File
@@ -1,79 +1,72 @@
# Implementation Rules
These implementation, style, and quality rules apply to all contributions, regardless of author or editor. Paths below are repository-relative. Read only the relevant sections during read-only review; select verification from the root `AGENTS.md`.
## Shared Style Contract
- Use the toolchain declared in `rust-toolchain.toml`, respect the minimum supported Rust version in `Cargo.toml`, and let `rustfmt.toml` and the configured Rust/Clippy lints define mechanical style. Do not override repository settings with personal formatter or lint preferences.
- Follow the owning module's organization, imports, and domain terminology where repository rules leave a choice. Existing violations do not justify new ones; avoid unrelated cleanup in a focused change.
- Rules using "must" or "do not" are requirements. Rules using "prefer" are defaults; depart when correctness, ownership, performance, or readability benefits justify it. Explain consequential tradeoffs, not every idiomatic choice. Where several approaches meet the contract equally well, the implementer chooses.
- Follow the declared toolchain locally and the affected CI lane's toolchain for reproduction. The repository currently tracks `stable`; for a formatter/lint mismatch, compare `rustc -Vv`, `cargo fmt --version`, and `cargo clippy --version` with the failing run before changing code. Toolchain upgrades must keep local and CI settings aligned and pass their affected gates; do not silently raise `rust-version` or change lint policy to make a check pass.
Applies when changing code or running artifact-heavy work. Paths below
are repository-relative. Read only the relevant sections during read-only review.
## Worktree and Disk Hygiene
- For a new task, start from the latest `origin/main` unless a different base is requested, and confirm the requested change is not already present. When continuing existing work or a PR, preserve its branch, base, and task-owned changes; do not restart it from `main`.
- An existing clean, isolated task worktree is sufficient. Create another worktree only when the current checkout is shared, dirty with unrelated work, or belongs to another task.
- Start implementation from the latest `origin/main` and confirm the requested
change is not already present.
- An existing clean, isolated task worktree is sufficient. Create another
worktree only when the current checkout is shared, dirty with unrelated work,
or belongs to another task.
- Never commit from a shared checkout.
- Use a task-specific branch named `<type>/<topic>`, such as `fix/...`, `feat/...`, `test/...`, or `docs/...`, unless the user specifies a name.
- Do not include agent, tool, contributor, account, or organization names in branch names.
- Push to the user-requested remote or the repository's configured push remote. Do not hard-code or infer a remote from an account name.
- Check free space before artifact-heavy builds, tests, coverage, or downloads. Re-check before a broad gate when space is tight.
- Before handoff, remove the Cargo target directories and temporary worktrees the task created unless they stay intentionally or hold unmerged work. Identify build output by Cargo's markers (`.rustc_info.json`, `CACHEDIR.TAG`), not by directory name: `target` is also a source directory here.
- Remove only task-owned temporary/build artifacts. Never delete another task's worktree or uncommitted data.
- At handoff, mention disk or cleanup details only when they affected execution or artifacts/worktrees remain intentionally.
- For Docker verification, ensure the tested image contains the intended binary. `buildx build` without `--load` leaves the image in the build cache; use the Makefile's loaded image and rebuild without cache only when stale artifacts are evidenced.
- Use a task-specific branch named `<type>/<topic>`, such as `fix/...`,
`feat/...`, `test/...`, or `docs/...`, unless the user specifies a name.
- Do not include agent, tool, contributor, account, or organization names in
branch names.
- Push to the user-requested remote or the repository's configured push remote.
Do not hard-code or infer a remote from an account name.
- Check free space before artifact-heavy builds, tests, coverage, or downloads.
Re-check before a broad gate when space is tight.
- Remove only task-owned temporary/build artifacts. Never delete another task's
worktree or uncommitted data.
- At handoff, mention disk or cleanup details only when they affected execution
or artifacts/worktrees remain intentionally.
## Change Style
- Preserve unrelated control flow. Restructure touched code only when needed for the requested behavior or an explicitly requested refactor; style preferences alone do not justify changing execution order or cleanup behavior.
- Prefer a direct local edit over new files, wrappers, managers, or speculative abstractions.
- Add a helper only when it removes current duplication, names a real domain boundary, or isolates a non-trivial invariant. Keep helpers near the code they support when the module's organization permits it.
- Remove an in-scope path superseded by the change. If compatibility requires it, adapt at the boundary to one canonical core and use the repository's `RUSTFS_COMPAT_TODO` policy.
- Comments explain non-obvious invariants or reasons. Do not narrate code or record change history.
- Preserve existing control flow unless changing it is required for correctness.
- Prefer a direct local edit over new files, wrappers, managers, or speculative
abstractions.
- Add a helper only when it removes current duplication, names a real domain
boundary, or isolates a non-trivial invariant.
- Remove an in-scope path superseded by the change. If compatibility requires it,
adapt at the boundary to one canonical core and use the repository's
`RUSTFS_COMPAT_TODO` policy.
- Comments explain non-obvious invariants or reasons. Do not narrate code or
record change history.
- Mention unrelated problems when useful; do not fix them in a narrow task.
## Rust Style
These repository-owned rules incorporate selected ideas from the [OpenCode style guide](https://github.com/anomalyco/opencode/blob/dev/AGENTS.md#style-guide), adapted to RustFS. That link is background, not an additional rule source. Apply these defaults to new or changed code; correctness and domain invariants take priority.
- Default to immutable `let` bindings; use `mut` for actual state changes. Prefer `if` or `match` expressions when branches compute a value instead of initializing a placeholder and reassigning it.
- Use type inference for obvious local values. Add annotations when inference needs help or the numeric, collection, or error type matters to understanding the code. Keep explicit types in function signatures and public contracts.
- Keep the main path readable with guard clauses, `let ... else`, and `?` when immediate propagation is correct. Preserve explicit handling for recovery, cleanup, aggregation, and quorum decisions; `if`/`else` expressions and exhaustive `match` remain idiomatic.
- Prefer readable iterator chains for straightforward mapping and filtering. Keep loops when they make state changes, early exits, or async operations clearer. Do not add intermediate collections or clones just to use iterators.
- Inline trivial single-use expressions only when a binding adds no meaning. Keep names that explain domain concepts, aid debugging, or make ownership and guard lifetimes clear. Do not change lock release or drop timing for style.
- Keep pure parsing, validation, and option-building helpers synchronous unless an async interface contract requires otherwise. This does not make blocking I/O or CPU-heavy work safe to run on async executor threads; offload it when needed to keep those threads responsive.
## Reuse and Boundary Rules
- Before adding helpers, constants, fixtures, or wrappers, search the touched code and domain owner; check `crates/utils`, `crates/common`, and direct dependencies when they could own the behavior. Stop once the relevant reuse boundary is clear.
- Reuse requires matching semantics: normalization, error types, deadlines, durability, and compatibility must fit the call site. A narrowly named local helper is better than forced reuse with different semantics.
- Validate untrusted input at the boundary that owns it, then pass validated types inward without repeating checks whose guarantees still hold. Deserializing into a Rust type alone does not establish domain validity. Values newly read from disk, RPC, persistence, or another version must be validated at that consuming boundary.
- For destructive actions such as delete or overwrite and for quorum decisions, check the relevant identity, generation, and authority under the operation's existing synchronization or precondition contract. A separate unguarded recheck does not prevent a check/use race.
- Every new control-flow branch needs a concrete triggering input/state. For decoded or peer data, corruption and mixed-version input are valid triggers.
- Required values must return a typed error when absent or corrupt; do not use a default that converts corruption into a plausible result.
- Attach error context once where it is actionable. Do not erase typed errors below aggregation or quorum layers.
- Before adding helpers, constants, fixtures, or wrappers, search the touched
crate, the domain-owning crate, `crates/utils`, `crates/common`, and relevant
direct dependencies.
- Reuse requires matching semantics: normalization, error types, deadlines,
durability, and compatibility must fit the call site. A narrowly named local
helper is better than forced reuse with different semantics.
- Validate untrusted input at its trust boundary, then trust the validated type.
Values crossing disk, RPC, persistence, or version boundaries remain
untrusted at every consumer.
- Re-check boundary values immediately before destructive actions such as
delete, overwrite, or quorum decisions.
- Every new branch needs a concrete triggering input/state. For decoded or peer
data, corruption and mixed-version input are valid triggers.
- Required values must return a typed error when absent or corrupt; do not use a
default that converts corruption into a plausible result.
- Attach error context once where it is actionable. Do not erase typed errors
below aggregation or quorum layers.
## Rust Boundaries
- Production `unwrap`/`expect` must follow a type guarantee or checked invariant; explain only non-obvious guarantees. Stricter scoped prohibitions, including `crates/AGENTS.md`, still apply. Invalid client, disk, or peer data must return errors rather than trigger panics.
- Production `unwrap`/`expect` must follow a type guarantee or checked invariant; explain only non-obvious guarantees.
- Public APIs do not return `Result<_, String>`. Library APIs use domain errors unless intentional erasure is part of the boundary contract; expose a stored inner error through `Error::source()`.
- Numeric conversions must not silently truncate. Use fallible conversions for untrusted values; validate finiteness, sign, and range before float-to-integer conversion. Clamp/saturate only when required by the domain.
- Do not add crate-root `#![allow(dead_code)]`.
## Naming
Use Rust API naming: `SCREAMING_SNAKE_CASE` constants/statics, `snake_case` functions/variables/modules, and `PascalCase` types/traits. Name domain concepts and include units where a value would otherwise be ambiguous; do not optimize for short names. Do not rename public, serialized, or persisted fields merely to match a local style preference, or rename unrelated existing violations.
## API and Operational Contracts
- For new or changed public APIs, document the caller-visible behavior and relevant `# Errors`, `# Panics`, and `# Safety` conditions. Include a useful example or link to one when usage is non-obvious; avoid boilerplate sections that say nothing. Follow the [Rust API documentation guidelines](https://rust-lang.github.io/api-guidelines/documentation.html) within the scope of the change.
- For async or storage APIs, state cancellation, retry/idempotency, partial-completion, ownership, and durability guarantees when callers rely on them. Make failure cleanup and commit points explicit in code and tests; do not imply atomicity that the implementation does not provide.
- For protocol, persisted-format, configuration, dependency, or operational changes, check affected consumers and supported versions. Update relevant docs/examples and describe compatibility, upgrade/rollback limits, and new configuration defaults when they change. New dependencies must solve a current need and fit the workspace's license, toolchain, and feature policy.
- For hot-path or resource-management changes, check allocation/copying, blocking, unbounded work, and lock duration where affected. Support performance claims with an appropriate measurement or clearly label them as expectations; a benchmark is not required for unrelated changes.
## Behavioral Tests
- For behavior changes, add or update the narrowest test that calls the real implementation and asserts an observable result. Existing tests can provide coverage when they directly exercise the changed behavior; do not duplicate production logic as the test oracle.
- Cover the changed contract's relevant failure and boundary cases, not only success. A bug regression should fail without the fix; do not broaden assertions or rely only on a successful call when the result or side effect is what matters.
- Follow `docs/testing/README.md` for test layers, runners, isolation, synchronization, and flaky tests. Use controlled clocks or explicit synchronization for time/concurrency tests instead of fixed sleeps, and keep waits bounded.
- Report the checks actually run and their outcomes, including skipped or blocked checks and the behavior they leave unverified. Do not describe focused local checks as full CI coverage.
Use Rust API naming: `SCREAMING_SNAKE_CASE` constants/statics, `snake_case`
functions/variables, and `PascalCase` types. Do not rename unrelated existing
violations.
+35 -15
View File
@@ -1,31 +1,51 @@
# Git and Pull Request Rules
Applies to commits, pushes, PRs, and issue/discussion actions. Paths below are repository-relative. User authorization and root `AGENTS.md` still govern scope.
Applies to commits, pushes, PRs, and issue/discussion actions. Paths below are
repository-relative. User authorization and root `AGENTS.md` still govern scope.
## Final PR Preflight
Before creating or updating a PR, reuse completed review and verification:
- Verify the actual base (normally `origin/main`) and the complete task diff, including file names and whitespace. Exclude secrets, logs, generated artifacts, and unrelated edits. Retain an existing PR's base unless requested.
- Confirm the final diff passed the root verification tier; fix task-owned failures and run missing scoped checks. Report unresolved required checks or authority without expanding the task. Do not start another general review.
- Keep the English Conventional Commit title at most 72 characters. Use the template headings, actual checks, material risks, and rollback notes.
- Immediately before writing to GitHub, confirm the head and task diff are unchanged. Rerun only checks invalidated by edits or relevant state changes.
- Verify the actual base (normally `origin/main`) and the complete task diff,
including file names and whitespace. Exclude secrets, logs, generated
artifacts, and unrelated edits. Retain an existing PR's base unless requested.
- Confirm the final diff passed the root verification tier; fix task-owned
failures and run missing scoped checks. Report unresolved required checks or
authority without expanding the task. Do not start another general review.
- Keep the English Conventional Commit title at most 72 characters. Use the
template headings, actual checks, material risks, and rollback notes.
- Immediately before writing to GitHub, confirm the head and task diff are
unchanged. Rerun only checks invalidated by edits or relevant state changes.
## Pull Request Lifecycle
- Creating or updating a PR includes one immediate snapshot of checks, mergeability, reviews, and unresolved threads.
- Unless the user explicitly requests monitoring, a release workflow requires it, or an automation already owns it, hand off after the PR is open with the current state and next event to watch. Do not delay ordinary handoff with fixed quiet-period sleeps.
- For requested monitoring, use event-driven or bounded waits. Report only state changes, actionable failures, or a meaningful prolonged delay.
- Investigate failures/comments before changing code. Monitoring alone authorizes observation and diagnosis; fixes, pushes, replies, and thread resolution require their own existing task authority. When authorized, fix task-attributable issues, rerun affected verification, and resume the requested monitor. Otherwise report the concrete failure and next action without expanding scope.
- Merge only with existing user authorization and satisfied applicable reviewer, check, and branch-protection requirements. Reviewer approval does not grant merge authority; user authorization does not bypass repository protections.
- After an observed merge, verify the commit reached the base, then clean the task worktree/branch when safe. Preserve unmerged work for closed PRs unless deletion was explicitly authorized.
- Creating or updating a PR includes one immediate snapshot of checks,
mergeability, reviews, and unresolved threads.
- Unless the user explicitly requests monitoring, a release workflow requires
it, or an automation already owns it, hand off after the PR is open with the
current state and next event to watch. Do not delay ordinary handoff with
fixed quiet-period sleeps.
- For requested monitoring, use event-driven or bounded waits. Report only state
changes, actionable failures, or a meaningful prolonged delay.
- Investigate failures/comments before changing code. Fix task-attributable
issues, rerun affected verification, push, reply or resolve the thread, then
resume the requested monitor.
- Never merge without required reviewer approval or explicit authority.
- After an observed merge, verify the commit reached the base, then clean the
task worktree/branch when safe. Preserve unmerged work for closed PRs unless
deletion was explicitly authorized.
## Git and PR Baseline
- Follow Conventional Commits; keep the subject at most 72 characters.
- Source comments, commits, PR titles, and PR bodies are in English.
- Keep every heading from `.github/pull_request_template.md`; use `N/A` where needed and include commands actually run.
- Keep every heading from `.github/pull_request_template.md`; use `N/A` where
needed and include commands actually run.
- Use `--body-file` for multiline `gh pr create`/`gh pr edit` content.
- PR/issue/discussion content must not contain the literal sequence `\n` or hard-wrapped prose paragraphs.
- Do not include local absolute paths or tool-specific labels/prefixes in GitHub content.
- Resolve review threads after the underlying issue is fixed. If declining a suggestion, reply with a short evidence-based reason.
- PR/issue/discussion content must not contain the literal sequence `\n` or
hard-wrapped prose paragraphs.
- Do not include local absolute paths or tool-specific labels/prefixes in GitHub
content.
- Resolve review threads after the underlying issue is fixed. If declining a
suggestion, reply with a short evidence-based reason.
+52 -12
View File
@@ -16,28 +16,68 @@ The "plugin system" spans four surfaces that must stay consistent:
## Hard invariants (verify before merging)
1. **Secrets have one source of truth.** Secret config keys are declared only in the plugin manifest (`TargetPluginManifest.secret_fields`, `crates/targets/src/manifest.rs`) and flow to admin via `AdminTargetSpec.secret_fields`. Never add a hand-maintained per-service secret table in a handler; if redaction misses a field, fix the manifest.
1. **Secrets have one source of truth.** Secret config keys are declared only
in the plugin manifest (`TargetPluginManifest.secret_fields`,
`crates/targets/src/manifest.rs`) and flow to admin via
`AdminTargetSpec.secret_fields`. Never add a hand-maintained per-service
secret table in a handler; if redaction misses a field, fix the manifest.
2. **Redaction must round-trip.** Instance GET responses replace secret values with `***redacted***` (`REDACTED_SECRET_VALUE` in `plugins_instances.rs`). Instance PUT restores the stored secret when it receives that placeholder back (`restore_redacted_secret_values`). Any new read or write path for target config must keep both halves: redact on the way out, restore the placeholder on the way in. The placeholder literal must never be persisted.
2. **Redaction must round-trip.** Instance GET responses replace secret values
with `***redacted***` (`REDACTED_SECRET_VALUE` in `plugins_instances.rs`).
Instance PUT restores the stored secret when it receives that placeholder
back (`restore_redacted_secret_values`). Any new read or write path for
target config must keep both halves: redact on the way out, restore the
placeholder on the way in. The placeholder literal must never be persisted.
3. **Fixtures never reach production responses.** `example_external_webhook_plugin()` (`crates/targets/src/catalog/mod.rs`) is a test/demo fixture for control-plane planning tests. Production catalog/extension handlers must not include it; regression tests (`plugin_catalog_never_exposes_example_or_external_fixtures`, `extension_catalog_never_exposes_example_or_external_fixtures`) enforce it.
3. **Fixtures never reach production responses.**
`example_external_webhook_plugin()` (`crates/targets/src/catalog/mod.rs`)
is a test/demo fixture for control-plane planning tests. Production
catalog/extension handlers must not include it; regression tests
(`plugin_catalog_never_exposes_example_or_external_fixtures`,
`extension_catalog_never_exposes_example_or_external_fixtures`) enforce it.
4. **External plugin flow is planning-only and deny-by-default.** `plan_external_target_plugin_action` returns decisions, it executes nothing. `TargetPluginExternalFlowGate::default()` is fully closed and `TargetPluginInstallPolicy::default().allowed_download_hosts` is empty — keep it that way; tests opt in via explicit policies. Install validation requires https, an allowlisted host, a full 64-hex-char sha256 digest, signature and provenance URIs, and an artifact matching the host `target_triple`.
4. **External plugin flow is planning-only and deny-by-default.**
`plan_external_target_plugin_action` returns decisions, it executes
nothing. `TargetPluginExternalFlowGate::default()` is fully closed and
`TargetPluginInstallPolicy::default().allowed_download_hosts` is empty —
keep it that way; tests opt in via explicit policies. Install validation
requires https, an allowlisted host, a full 64-hex-char sha256 digest,
signature and provenance URIs, and an artifact matching the host
`target_triple`.
5. **Custom target types must not collide.** Unknown target types get an interned unique `custom:<type>` plugin id (`custom_plugin_id` in `manifest.rs`). Custom plugins with secrets must register via `TargetPluginDescriptor::with_manifest` and declare `secret_fields`; `::new` derives a manifest with no secrets.
5. **Custom target types must not collide.** Unknown target types get an
interned unique `custom:<type>` plugin id (`custom_plugin_id` in
`manifest.rs`). Custom plugins with secrets must register via
`TargetPluginDescriptor::with_manifest` and declare `secret_fields`;
`::new` derives a manifest with no secrets.
## Changing the admin JSON contract
- Shapes are locked twice in `plugin_contract.rs` tests: insta snapshots (`rustfs/src/admin/snapshots/`) plus literal `json!` assertions. Update both deliberately; a shape change is a console-facing API change.
- Field naming is `snake_case`, except discovery blocks (`runtimeCapabilities`, `clusterSnapshot`, `extensionsCatalog`) which are camelCase **by cross-endpoint convention** (same shape in `system.rs`, `console.rs`, `pools.rs`). Do not "fix" that inconsistency locally.
- Contract types deliberately duplicate `rustfs_targets` types (anti-corruption layer). Add a `From` impl; do not serialize internal types directly.
- Shapes are locked twice in `plugin_contract.rs` tests: insta snapshots
(`rustfs/src/admin/snapshots/`) plus literal `json!` assertions. Update
both deliberately; a shape change is a console-facing API change.
- Field naming is `snake_case`, except discovery blocks
(`runtimeCapabilities`, `clusterSnapshot`, `extensionsCatalog`) which are
camelCase **by cross-endpoint convention** (same shape in `system.rs`,
`console.rs`, `pools.rs`). Do not "fix" that inconsistency locally.
- Contract types deliberately duplicate `rustfs_targets` types
(anti-corruption layer). Add a `From` impl; do not serialize internal
types directly.
## Handler conventions
- Every new admin plugin/extension route needs authorization at the top of `call` and an `include_str!` guard test asserting it (repo-wide pattern — see `plugin_instance_handlers_require_admin_authorization_contract`). The source guard checks wiring, not runtime authorization; use focused behavior evidence for changed allow/deny semantics.
- Reads use `GetBucketTargetAction` (instances) or `ServerInfoAdminAction` (catalogs); writes use `SetBucketTargetAction`.
- Refresh persisted module switches once per request (`refresh_persisted_module_switches`), then evaluate the sync `module_disabled_block_reason` per domain — do not re-read the store per domain or per instance.
- Every new admin plugin/extension route needs authorization at the top of
`call` and an `include_str!` guard test asserting it (repo-wide pattern —
see `plugin_instance_handlers_require_admin_authorization_contract`).
- Reads use `GetBucketTargetAction` (instances) or `ServerInfoAdminAction`
(catalogs); writes use `SetBucketTargetAction`.
- Refresh persisted module switches once per request
(`refresh_persisted_module_switches`), then evaluate the sync
`module_disabled_block_reason` per domain — do not re-read the store per
domain or per instance.
## Generic bounds
Event-payload generics use the `PluginEvent` blanket trait (`crates/targets/src/plugin.rs`). Do not respell `Send + Sync + 'static + Clone + Serialize + DeserializeOwned`.
Event-payload generics use the `PluginEvent` blanket trait
(`crates/targets/src/plugin.rs`). Do not respell
`Send + Sync + 'static + Clone + Serialize + DeserializeOwned`.
+3 -3
View File
@@ -13,8 +13,8 @@ description: Review a GitHub PR using its exact base/head and repository risk po
```
Record both SHAs; refresh if either moved during fetching. Follow relevant [implementation boundaries](../../references/implementation.md).
2. Apply the [risk policy](../../references/adversarial-validation.md), including its two perspectives for high-risk work. Ordinary reviews use the root finding standard directly. File count does not determine risk or authorize delegation; keep any delegated review within the authorized scope.
2. Apply the [risk policy](../../references/adversarial-validation.md). Ordinary reviews use the root finding standard directly. File count does not authorize delegation; use exactly two independent reviewers only when authorized, otherwise two fresh sequential passes for that risk tier. Reviewers do not spawn further agents.
3. Check `gh pr checks <N> --repo <owner/repo>`. Investigate failures when relevant to a finding or a requested CI/readiness diagnosis. Distinguish pending, unavailable, pre-existing, flaky, and PR-caused states using evidence; a code review does not authorize unrelated repairs.
4. Report the reviewed SHAs, risk tier, supported findings or `No findings`, material verification limits, check state, and verdict. Set severity from evidenced impact, triggering conditions, reach, and urgency: P0 is a critical issue requiring immediate action with broad or unavoidable impact; P1 is a high-impact defect requiring prompt correction; P2 is an ordinary concrete correctness, compatibility, performance, or maintainability defect; P3 is an optional improvement only when requested. A remote trigger, crash, or deadlock does not by itself establish P0. P0/P1 findings block approval; evaluate other findings and verification gaps against the actual acceptance requirements rather than approving automatically below that threshold.
5. When posting is authorized, follow [Git and PR rules](../../references/pull-requests.md), recheck the remote base and head, and review any changed diff or integration assumptions before submitting. Reuse unaffected evidence. Use `gh pr review` with `--approve`, `--request-changes`, or `--comment` and `--body-file`; use [the API example](references/posting.md) for authorized inline comments. Reuse existing authorization.
4. Report the reviewed SHAs, risk tier, supported findings or `No findings`, material verification limits, check state, and verdict. P0/P1 findings block approval. Severity: P0 = demonstrated data loss, security breach, remote crash, or deadlock; P1 = correctness, compatibility, or material hot-path regression; P2 = a concrete maintainability defect; P3 = optional style, only when requested.
5. When posting is authorized, follow [Git and PR rules](../../references/pull-requests.md), recheck the remote head, and review any delta before submitting. Use `gh pr review` with `--approve`, `--request-changes`, or `--comment` and `--body-file`; use [the API example](references/posting.md) for authorized inline comments. Reuse existing authorization.
6. Follow the repository PR lifecycle and explicit monitoring scope. Fetch new heads before comparing to the reviewed SHA; never rely on an unfetched `origin/pull/<N>/head`. Push fixes or resolve threads only when authorized; check `maintainerCanModify` before a fork push.
@@ -4,20 +4,19 @@ Read only when an inline review is authorized. Recheck the PR head before postin
For inline comments on specific lines, use the GitHub API:
```bash
REVIEW_PAYLOAD=$(mktemp "${TMPDIR:-/tmp}/pr-review.XXXXXX")
cat > "$REVIEW_PAYLOAD" <<'EOF'
cat > /tmp/pr_review.json <<'EOF'
{
"commit_id": "<reviewed-head-sha>",
"body": "review body",
"event": "REQUEST_CHANGES",
"comments": [
{
"path": "crates/<crate>/src/<file>.rs",
"path": "crates/foo/src/bar.rs",
"line": 42,
"body": "finding description"
}
]
}
EOF
gh api --method POST /repos/{owner}/{repo}/pulls/<N>/reviews --input "$REVIEW_PAYLOAD" && rm -f "$REVIEW_PAYLOAD"
gh api --method POST /repos/{owner}/{repo}/pulls/<N>/reviews --input /tmp/pr_review.json
```
+25 -21
View File
@@ -1,38 +1,42 @@
---
name: rustfs-release-publish
description: Publish a RustFS version through frozen Linux candidate acceptance, desktop completion, manifest approval, byte-preserving publication and post-release follow-up. Use only for an explicit release request.
description: Publish a RustFS version through Console, release-branch CI, preview acceptance, human confirmation, and final publication. Use only for an explicit release request.
---
# RustFS Release
Require the exact final SemVer target before version edits or publication. If unspecified, inspect releases/tags and ask with concrete candidates; do not guess the channel or bump. Reuse the user's confirmed target and authorized delivery scope. Independent read-only preflight may continue.
Require the exact final SemVer target before version edits or publication. If unspecified, inspect current releases/tags and ask with concrete candidates; do not guess the channel or bump. A named bump type resolves to an exact version that still needs confirmation. Report the confirmed target verbatim; resolve any later mismatch before proceeding. Independent read-only preflight may continue.
## Invariants
- Cargo and every candidate executable contain the final product version. `<target>-preview.<run_number>` is an immutable candidate identifier, not a public tag or displayed binary version.
- Freeze the exact source, reviewed workflow recipe, toolchain, Console, CLI, dependency lock, previous stable version and validation plan. Newer unrelated main or Console releases do not change those inputs.
- Retain `release/<target>` history when a release branch is used; backport only reviewed fixes. Pin `CANDIDATE_SHA` to its exact CI-validated commit. Never force-push a release branch or move a public tag.
- Build Linux first. Complete field acceptance before macOS/Windows and packaging. Full certification and final acceptance are required before publication; a partial Linux PASS grants no release authority.
- Promote the accepted bytes. Final publication never recompiles binaries or rebuilds packages/images. Infrastructure retries reuse successful artifacts; changed source or inputs require a new candidate and affected acceptance.
- Phase failures block dependents. No latest/default-channel update before versioned delivery. Use the [pipeline runbook](../../../docs/operations/release-pipeline.md) for executable entry points, locks and recovery.
- Cargo stores `<target>`, never `-preview.N`. Before publication, installation defaults retain the previous available deliverable. Final and preview binaries embed their own tag names, so version/asset names differ despite identical source.
- Use annotated tags without `v`. Preview names are exactly `<target>-preview.<digits>`; choose the next unused iteration after fetching tags. Preview Releases are prereleases, never Latest; publish only versioned assets/exact Docker tags, never latest assets, R2, Helm, moving Docker channels, or installation/banner updates.
- `release/<target>` owns the candidate. Retain its history, backport only reviewed fixes, and never merge main wholesale or force-push it. Pin `PREVIEW_HASH` to its exact commit with complete passing CI. Preview and final tags both point there; never tag a later main/branch HEAD or insert another bump.
- Phase failures block dependents. Keep preview Releases until final publication and notes generation finish; CI removes Releases while retaining tags. Never move an existing preview or final tag.
## Read by phase
## Read by Phase
Read only the current phase's procedure; retain completed evidence across turns. Follow root verification, Git/PR, and authorization rules.
| Phase | Procedure |
|---|---|
| 0 | Clean tree, fetch main/tags, verify access, then complete the mandatory [Console gate](references/console-gate.md). |
| 1–2 | [Prepare the exact source and run branch CI](references/release-candidate.md). Version edits use `rustfs-release-version-bump`. |
| 3 | [Build and verify the Linux candidate](references/preview-release.md). |
| 4–5 | [Run artifact, field, Console and rc acceptance](references/preview-acceptance.md), then complete desktop/package/image/chart validation. |
| Confirmation | Report the complete manifest and obtain the confirmation below. |
| 6 | [Publish the accepted manifest](references/final-release.md). |
| 7 | [Reconcile advisories, milestones, installation references, website and release branch](references/post-release-updates.md). |
| 0 | Clean tree, fetch main/tags, verify GitHub access, then complete the mandatory [Console release gate](references/console-gate.md). A green Console build alone is insufficient. |
| 1–2 | [Prepare/reuse the release candidate and run exact-SHA branch CI](references/release-candidate.md). Source version changes use `rustfs-release-version-bump` with authorized delivery scope. |
| 3 | [Verify the preview build, assets, channels, and notes](references/preview-release.md). |
| 4–5 | [Run the downloaded binary, Console CRUD, and latest-rc matrix](references/preview-acceptance.md). Every check must pass. |
| Confirmation | Complete the gate below and end the turn. |
| 6 | After fresh confirmation, [publish and verify the final tag](references/final-release.md) at `PREVIEW_HASH`. |
| 7 | [Reconcile Security Advisories, milestones, installation references, and the website; reconcile main and remove the completed release branch](references/post-release-updates.md) only after complete publication. |
## Human confirmation
## Human Confirmation
After all acceptance passes, report the target, source SHA, source CI run, Linux and completion run URLs, final manifest SHA-256, accepted Console/CLI identities, compatibility/performance results and backlog report. Explicitly ask whether to publish this exact manifest and end the turn without dispatching publication.
After Phases 3–5 pass, recheck that the release branch still equals `PREVIEW_HASH`. Report the target, branch, preview tag/hash, branch CI run, preview Release URL, Console result, and rc matrix. Explicitly ask whether to publish the final tag and end the turn without creating/pushing it.
Only a new affirmative reply to that exact report authorizes Phase 6. An earlier general release request, silence or automatic continuation does not count. Reuse confirmation for retries of that same manifest; do not ask again merely because a destination failed. A changed source, input, manifest or acceptance result invalidates it and requires renewed acceptance/confirmation.
Only a new affirmative user reply to that exact report authorizes Phase 6. The original release request, earlier confirmation, silence, or automated follow-up does not count. Confirmation binds the target, preview tag, and hash; a changed value or repeated/failed acceptance cycle invalidates it. Revalidate and obtain fresh confirmation before continuing.
## Recovery and reporting
## Recovery
Resume failed infrastructure jobs without rebuilding successful units. Fix product defects on main, backport when needed, and start a new candidate. After a final tag exists, retry the same bytes or use a new patch version; never move the tag. Do not conflate implemented, CI-passed, field-accepted, approved, published and post-release-complete states. Report incomplete work and deviations with evidence; a tag or green build alone is not a completed release.
- Before the final tag exists, land fixes on main, backport to the candidate, verify target versions, and restart branch CI/acceptance. Use a new preview iteration once a preview was tagged; main may advance independently.
- Once the final tag exists, retry publication jobs for that exact tag. A source fix requires a newly confirmed target, not retagging or a newer preview. Phase 7 retries recheck artifact availability and superseding releases.
- If abandoned after the bump merged, report the unpublished version on main and record whether to revert or let the next release overwrite it. Deleting unpublished candidate branches or preview tags requires an explicit decision. Completed release branches follow Phase 7 cleanup; keep release tags.
Report completed phase results and incomplete work with evidence: Console previous/latest tags, release need, hash and run/Release URLs; target, branch CI/SHA, previews and hash, confirmation state; preview/final assets, latest-channel state, Console/rc results and cleanup; advisory reconciliation results and pending disclosure decisions; both milestone states, installation PR checks/merge state, and website text/link/PR/deployment state. Keep artifact publication separate from follow-up delivery; do not call an open PR a live update. Report deviations and their authorization.
@@ -4,7 +4,7 @@ Read during Phase 0, before changing RustFS version files or tags. Follow the pa
### Console release gate
Complete this gate before changing any RustFS version file or creating any RustFS tag. Candidate creation resolves `repos/rustfs/console/releases/latest` once and freezes the exact asset for every build, so a successful Console build alone is insufficient. After freezing, later Console releases do not silently change or invalidate the accepted inputs.
Complete this gate before changing any RustFS version file or creating any RustFS tag. RustFS `build.yml` downloads the asset returned by `repos/rustfs/console/releases/latest`, so a successful Console build alone is insufficient.
1. Read the latest published Console tag and compare it with Console `main`:
@@ -37,7 +37,7 @@ git -C "$CONSOLE_SCRATCH/console" tag -a "<console-tag>" -m "Release <console-ta
git -C "$CONSOLE_SCRATCH/console" push origin "<console-tag>"
```
Console tags include the `v` prefix. Pushing the tag triggers the Console repository's [release workflow](https://github.com/rustfs/console/blob/main/.github/workflows/release.yml) (`🚀 Release`). Remove the task-owned `CONSOLE_SCRATCH` after the gate completes.
Console tags include the `v` prefix. Pushing the tag triggers `.github/workflows/release.yml` (`🚀 Release`). Remove `CONSOLE_SCRATCH` after the gate completes.
3. Find the exact tag run and wait for completion:
@@ -55,4 +55,4 @@ test "$(gh api "repos/${CONSOLE_REPO}/releases/tags/<console-tag>" \
--jq '[.assets[] | select(.name == "rustfs-console-<console-tag>.zip" and .state == "uploaded" and .size > 0 and (.digest | startswith("sha256:")))] | length')" -eq 1
```
Treat a missing/mismatched asset, digest, latest tag, or failed/cancelled workflow as BLOCKED. Do not start Phase 1 until the Console gate passes. Record `CONSOLE_TAG`, `CONSOLE_HASH`, the asset URL/SHA256, Console run URL, and Release URL for comparison with the inputs actually consumed by the RustFS builds and the final report.
Treat a missing/mismatched asset, digest, latest tag, or failed/cancelled workflow as BLOCKED. Do not start Phase 1 until the Console gate passes. Record `CONSOLE_TAG`, `CONSOLE_HASH`, Console run URL, and Release URL for the final report.
@@ -1,13 +1,22 @@
# Final release publication
# Final Release Publication
Read only after fresh confirmation of the complete candidate report. The [parent confirmation gate](../SKILL.md#human-confirmation) applies.
Read only after fresh user confirmation of the preview acceptance report. The [parent confirmation gate](../SKILL.md#human-confirmation) applies.
Dispatch `release-publish.yml` on main with the successful completion run ID and approved `release-manifest.json` SHA-256. Do not create the tag manually. The workflow revalidates source, bytes and trusted certification, then creates the annotated final tag at `CANDIDATE_SHA`. A recipe change fails closed and requires a new candidate; unrelated main changes do not require a rebuild.
## Phase 6 — Publish the final tag on the validated commit
Require both versioned publication and channel publication jobs to succeed. Verify the final tag's commit and manifest identity; all six platform ZIPs and compatibility aliases; DEB/RPM, checksums, SBOM, provenance and evidence bundle; both multi-architecture image variants in Docker Hub, GHCR and Quay; and the accepted Helm chart in the chart repository. Download a host-compatible final artifact and confirm its digest is exactly the accepted candidate digest and its version/source output agrees.
No second source version bump. Before creating the final tag, recheck the remote `release/<target>` HEAD equals `PREVIEW_HASH`. If it moved after confirmation, the approval no longer applies; validate and obtain confirmation for a new preview. The final tag goes on the exact commit the preview validated:
For stable releases verify GitHub Latest, R2 download aliases and `latest.json` unless a newer stable release already owns those channels. Prereleases must remain GitHub prereleases and leave stable pointers unchanged. Verify generated release notes compare against the frozen previous stable release, excluding internal previews.
```bash
git fetch origin --tags
git rev-parse "<preview-tag>^{commit}" # must equal PREVIEW_HASH — abort if not
git tag -a "<target>" -m "Release <target>" "$PREVIEW_HASH"
git push origin "<target>"
```
On infrastructure failure retry the same workflow/seal; matching destinations are reused. Never clobber an asset, force a tag, rebuild the image or silently accept a changed digest. A source fix after the final tag requires a new patch version. Cross-registry writes are not atomic: report the unfinished destination rather than claiming publication complete.
- CI rebuilds from the same source; the only changed input is the tag name, so the binary now self-reports `<target>`.
- Verify the final tag's complete publication path: all matrix and release jobs green; `gh release view "<target>"` shows the full versioned and `-latest` asset set plus checksums, SBOM, and provenance; Docker and Helm workflows succeed; `latest.json` points to `<target>`. A stable target must have `isPrerelease=false` and `isLatest=true`. An alpha/beta/rc target must have `isPrerelease=true`; GitHub does not permit prereleases to be Latest, but the project `latest.json` still advances to the final non-preview target.
- Verify the final Release body contains `## What's Changed` and a Full Changelog link. When `PREVIOUS_DELIVERABLE` exists, the link MUST be `https://github.com/rustfs/rustfs/compare/<PREVIOUS_DELIVERABLE>...<target>` and the baseline MUST equal the preview Release baseline; for example, both `1.0.0-beta.12-preview.1` and `1.0.0-beta.12` compare from `1.0.0-beta.11`.
- Verify the preview cleanup: `cleanup-preview-releases` must succeed, `gh release view "<preview-tag>"` must then report `release not found` for every preview iteration of this target, and `git rev-parse "<preview-tag>^{commit}"` must still resolve to `PREVIEW_HASH` (the tag is kept). If the job failed, delete the leftover Releases manually with `gh release delete "<preview-tag>" --yes` and report it.
- Optionally spot-check `./rustfs --version` from a final-tag artifact — it must report `<target>`.
New candidates create no preview Releases to clean up. For a legacy preview release, preserve its tag and remove its Release only after final delivery and notes are verified. Do not pass `--cleanup-tag`. Complete the [post-release follow-up](post-release-updates.md) separately.
Never pass `--cleanup-tag` when removing preview Releases. Remove them manually only if `cleanup-preview-releases` failed, after final publication and notes generation. Record deleted Releases and surviving tags.
@@ -24,7 +24,7 @@ Reconcile advisories after every completed final release, including Console-only
1. Re-read the published deliverable state before edits or a retry. If a newer release has superseded this target, do not downgrade installation references or the website announcement. Report the superseding release and leave those defaults intact.
2. Invoke `rustfs-release-version-bump` with stage `post-release`, the exact target, and the authorized delivery scope. Prepare the follow-up on current main; do not edit or move either validated release tag. Cargo versions may already be preparing the next target and must remain untouched.
3. Verify the exact Docker image manifest, Release download links, RPM source archive, and Helm repository entry before publishing the installation PR. The chart is packaged and tested before sealing; publication uploads those same bytes only after versioned Docker images are available.
3. Verify the exact Docker image manifest, Release download links, RPM source archive, and Helm repository entry before publishing the installation PR. The chart workflow overrides chart/app versions from the final tag, pins the source checkout to that release, and waits for its default Docker image before packaging.
4. Report the installation PR and its merge state separately from the completed release. Reuse existing merge authorization and repository checks; do not infer new merge authority from this reference.
## rustfs.com top banner
@@ -54,4 +54,4 @@ Use the existing announcement configuration from [website PR #112](https://githu
1. Compare the exact published tag with current main. Audit every release-side commit since their merge base, including inherited backports. An ahead count is a history difference, not proof of missing changes. Use ancestry and patch equivalence, then inspect current code and regression evidence for squashed, adapted, superseded, or reverted changes. Record each commit as present, superseded, release-only, or missing; a positive `git cherry` result alone does not prove a missing fix.
2. Bring missing shared changes to main through a reviewed PR and the required checks before closing the follow-up. Preserve main's later changes; decide version bumps and pinned Console configuration separately. If a history merge is needed, use a merge commit and verify the release commit is an ancestor of main; squash does not establish that ancestry. Do not merge main into the published release branch or move its tags. Report unresolved changes instead of claiming reconciliation is complete.
3. After publication and authorized follow-ups are complete, delete the remote `release/<target>` branch. Immediately before deletion, verify its HEAD still equals the published tag's commit and no open PR targets or originates from it. Preserve a branch with newer work or open PRs and report the blocker. Use a deletion with an expected-old-SHA check to avoid removing a concurrently advanced branch.
4. Verify the remote branch is absent and the final tag and any legacy preview tags still resolve to their recorded commits. Remove a local release branch only if it is task-owned, unchecked out, and has no unpreserved work. Record cleanup separately from artifact publication; an explicitly authorized deletion before reconciliation does not make unresolved main changes complete.
4. Verify the remote branch is absent and the final and preview tags still resolve to their recorded commits. Remove a local release branch only if it is task-owned, unchecked out, and has no unpreserved work. Record cleanup separately from artifact publication; an explicitly authorized deletion before reconciliation does not make unresolved main changes complete.
@@ -1,68 +1,52 @@
# Candidate Artifact Acceptance
# Preview Artifact Acceptance
Read after Phase 3 succeeds. Complete every check below before the parent skill's manual confirmation gate. These checks cover the downloaded artifact, embedded Console, and the rc release frozen in the candidate inputs.
Read after Phase 3 succeeds. Complete every check below before the parent skill's manual confirmation gate. These checks cover the downloaded artifact, embedded Console, and latest rc client.
## Phase 4 — Run the artifact locally, verify the console
Use a unique task-owned scratch directory. Select the versioned archive matching the acceptance host, and verify it against the sealed manifest before extraction. Before desktop completion, use the Linux acceptance environment rather than rebuilding for the local desktop. Do not substitute a different build when a matching artifact is unavailable; report that acceptance gap. The shell example below assumes the verified executable is at `RUSTFS_BINARY` (`rustfs.exe` on Windows), `SCRATCH` is the task directory, and `S3_PORT` and `CONSOLE_PORT` are distinct available ports. Use equivalent native commands when the shell differs.
Bind both listeners to loopback. Choose task-only credentials, set loopback proxy exclusions for the server/client processes, and record the server PID. A port probe does not reserve a port; retry startup with fresh ports on a bind conflict rather than connecting to an existing service.
Work inside the session scratchpad directory; never leave stray data dirs.
```bash
"$RUSTFS_BINARY" --version
mkdir -p "$SCRATCH/data"
S3_ENDPOINT="http://127.0.0.1:${S3_PORT}"
CONSOLE_ENDPOINT="http://127.0.0.1:${CONSOLE_PORT}"
RUSTFS_ACCESS_KEY="$PREVIEW_ACCESS_KEY" RUSTFS_SECRET_KEY="$PREVIEW_SECRET_KEY" \
"$RUSTFS_BINARY" server \
--address "127.0.0.1:${S3_PORT}" \
--console-enable \
--console-address "127.0.0.1:${CONSOLE_PORT}" \
"$SCRATCH/data" >"$SCRATCH/server.log" 2>&1 &
PREVIEW_SERVER_PID=$!
gh release download "<preview-tag>" -p "rustfs-macos-aarch64-v<preview-tag>.zip" -D "$SCRATCH"
cd "$SCRATCH" && unzip -o rustfs-*.zip
./rustfs --version # must report the PREVIEW TAG (build::TAG), not the Cargo.toml version, plus expected short SHA
mkdir -p data
RUSTFS_ACCESS_KEY=rustfsadmin RUSTFS_SECRET_KEY=rustfsadmin ./rustfs ./data
```
Defaults: S3 endpoint `:9000`, embedded console `:9001`.
Checks (all must pass):
- `"$RUSTFS_BINARY" --version` reports the exact final `<target>` and full `CANDIDATE_SHA`. A preview suffix, another version or another source is a failure.
- Wait with a bounded readiness deadline while checking the recorded server process remains alive; `curl --noproxy '*' -fsS "$S3_ENDPOINT/health/ready"` must return ready from this process's listener.
- `./rustfs --version` reports the preview tag name and the short SHA of `PREVIEW_HASH`. Reporting `<target>` without the `-preview.N` suffix means the build did not embed the tag — treat as FAIL and investigate before proceeding.
- `curl -fsS http://localhost:9000/health/ready` returns ready.
- Startup log shows the embedded console being served (this was the regression that `fix(release): require embedded console assets` guards).
- Open `CONSOLE_ENDPOINT` in the browser and log in with the task credentials. The dashboard must render without JS console errors; create a bucket, upload a file, download it back (byte-identical), and delete the object and bucket. Keep the server running for Phase 5.
- Open `http://localhost:9001` in the browser: login with `rustfsadmin`/`rustfsadmin`; dashboard renders without JS console errors; create a bucket, upload a file, download it back (byte-identical), delete the object and bucket. Keep the server running for Phase 5.
## Phase 5 — Validate with the pinned rc client
## Phase 5 — Validate with the latest rc client
`rc` is the RustFS CLI client from <https://github.com/rustfs/cli>.
- Read the CLI release and asset digests from `inputs.json`; verify the local client's version or download a host-compatible asset from that exact release into the task directory. If the host is a desktop, resolve its asset within the same pinned CLI release and record the digest; do not independently follow `latest`. Set `RC_BIN` to that executable and record its version/digest. Avoid changing the user's installed client for this check.
- Inspect the client's current help for an isolated configuration option and use it when supported. Otherwise choose a unique `PREVIEW_ALIAS` and confirm it is absent before creating it; never overwrite an existing alias. Use task-owned bucket/user names and input/output files. The names below are safe only inside the newly started isolated server.
- Point the client at `S3_ENDPOINT` with loopback proxy exclusions and run the command matrix, recording PASS/FAIL per command:
- Ensure the latest release is installed: compare `rc --version` against `gh api repos/rustfs/cli/releases/latest --jq .tag_name`; update via `brew upgrade rustfs/tap/rc` (or download the release binary).
- Point it at the preview server and run the command matrix, recording PASS/FAIL per command:
```bash
"$RC_BIN" alias set "$PREVIEW_ALIAS" "$S3_ENDPOINT" "$PREVIEW_ACCESS_KEY" "$PREVIEW_SECRET_KEY"
"$RC_BIN" ls "$PREVIEW_ALIAS/"
"$RC_BIN" mb "$PREVIEW_ALIAS/rel-check"
"$RC_BIN" cp <local-file> "$PREVIEW_ALIAS/rel-check/"
"$RC_BIN" stat "$PREVIEW_ALIAS/rel-check/<file>"
"$RC_BIN" cat "$PREVIEW_ALIAS/rel-check/<file>" # matches source
"$RC_BIN" cp "$PREVIEW_ALIAS/rel-check/<file>" "$SCRATCH/out"
cmp <local-file> "$SCRATCH/out"
"$RC_BIN" cp -r <local-dir>/ "$PREVIEW_ALIAS/rel-check/dir/"
"$RC_BIN" find "$PREVIEW_ALIAS/rel-check" --name "*"
"$RC_BIN" share download "$PREVIEW_ALIAS/rel-check/<file>" --expire 1h
# Fetch the presigned URL with curl --noproxy '*' and compare its body.
"$RC_BIN" rm "$PREVIEW_ALIAS/rel-check/<file>"
"$RC_BIN" rm -r --force "$PREVIEW_ALIAS/rel-check/dir"
"$RC_BIN" rb "$PREVIEW_ALIAS/rel-check"
"$RC_BIN" admin user list "$PREVIEW_ALIAS/"
"$RC_BIN" admin user add "$PREVIEW_ALIAS/" relcheckuser relchecksecret12
"$RC_BIN" admin user remove "$PREVIEW_ALIAS/" relcheckuser
"$RC_BIN" alias remove "$PREVIEW_ALIAS"
rc alias set preview http://localhost:9000 rustfsadmin rustfsadmin
rc ls preview/
rc mb preview/rel-check
rc cp <local-file> preview/rel-check/
rc stat preview/rel-check/<file>
rc cat preview/rel-check/<file> # matches source
rc cp preview/rel-check/<file> ./out && cmp <local-file> ./out
rc cp -r <local-dir>/ preview/rel-check/dir/
rc find preview/rel-check --name "*"
rc share download preview/rel-check/<file> --expire 1h # presigned URL fetchable via curl
rc rm preview/rel-check/<file> && rc rm -r --force preview/rel-check/dir
rc rb preview/rel-check
rc admin user list preview/
rc admin user add preview/ relcheckuser relchecksecret12
rc admin user remove preview/ relcheckuser
rc alias remove preview
```
- Any FAIL blocks acceptance. On success or failure, stop only the recorded task process and remove only task-created aliases/data after retaining the needed failure evidence. Do not erase an existing client configuration or unrelated server data.
## Field acceptance and desktop completion
Before accepting the Linux manifest, record Scanner/Heal, multi-pool, memory, major features, high-concurrency 503, performance and previous-version integrity/upgrade results in backlog. Preserve the release-specific agreed limits and baseline; a PASS label without artifact identities or test results does not satisfy acceptance.
Dispatch `release-complete.yml` only after that report passes, using the exact Linux manifest digest and backlog URL. Require native macOS/Windows checks, package extraction, image startup/scans, chart tests and complete certification to pass. Complete any desktop Console/client checks applicable to the release against these same archives. Record the completion run and final manifest SHA-256 before entering the parent confirmation gate. Partial Linux acceptance does not authorize final publication.
- Any FAIL blocks the release. Afterwards stop the server and delete the scratch data directory.
@@ -1,9 +1,24 @@
# Linux candidate verification
# Preview Release Verification
Read for Phase 3 after the source passed exact-SHA CI. The [parent invariants](../SKILL.md) apply; commands and recovery live in the [pipeline runbook](../../../../docs/operations/release-pipeline.md).
Read for Phase 3 after the candidate passed exact-SHA CI. The [parent release invariants](../SKILL.md) apply.
Dispatch `release-candidate.yml` on main with the confirmed final version and `CANDIDATE_SHA`. Record the run ID, frozen `inputs.json`, every Linux archive digest and `linux-manifest.json` SHA-256. Require the four Linux builds, native startup/S3 checks, previous-release upgrade matrix and Linux seal to succeed. Missing, skipped, cancelled or failed required work is incomplete acceptance.
## Preview channels
Verify the frozen Console identity equals the one accepted in Phase 0. Every matrix job consumes that same digest and exact toolchain. The binary must report the final product version and source commit; its candidate ID is separate. No public tag, Release, registry tag, R2 release path, Helm repository or latest pointer should change during this stage.
- Use `<target>-preview.N` for every target, e.g. `1.0.0-beta.10-preview.3` or `1.1.0-preview.1`.
- The canonical suffix is exactly `-preview.<digits>`. `build.yml` recognizes it before alpha/beta/rc classification and routes it to the preview-only path; any other tag containing `-preview` fails closed instead of being treated as a release.
- A preview Release MUST be published with `isPrerelease=true` and `isLatest=false`. Any `*-latest` preview asset or preview-triggered `latest.json`, R2, Helm, or moving Docker channel tag is a pipeline failure. Docker images may use the exact preview version tag, including its existing variant suffixes.
- Preview Releases are cleaned up by the `cleanup-preview-releases` job after `publish-release` succeeds for the deliverable tag. It deletes every Release whose tag is exactly `<target>-preview.<digits>` and never passes `--cleanup-tag`, so the tags survive.
A green Linux run is ready for field acceptance, not publication. The partial certification must have `publish_allowed: false`. Retain the exact Linux manifest for the next stage; a new source/build-input candidate must not inherit the old approval. A newer unrelated main commit does not invalidate this evidence.
## Phase 3 — CI and preview Release verification
- Find and watch the tag build: `gh run list --workflow build.yml --branch "<preview-tag>" --limit 1` then `gh run watch <run-id>`. Every build matrix target must succeed (linux x86_64/aarch64 × musl/gnu, macos-aarch64, windows-x86_64).
- Confirm the preview tag resolves to the Phase 2 CI-validated `PREVIEW_HASH`. If the release branch advances during acceptance, repeat Phases 2–5 with a new preview tag.
- Confirm the Release publication jobs (`create-release`, `upload-release-assets`, and `publish-release`) succeed while `update-latest-version` is skipped.
- Verify `gh release view "<preview-tag>" --json isPrerelease,assets,url`: `isPrerelease` must be `true`, and the Release must contain all 6 versioned platform zips, checksums, SBOM, and provenance with no `-latest` assets. Confirm `gh api repos/{owner}/{repo}/releases/latest --jq .tag_name` does not return `<preview-tag>`.
- Record `PREVIOUS_DELIVERABLE`, selected from published Releases by `publishedAt` after excluding the current tag and every `-preview.N` tag. Verify `gh release view "<preview-tag>" --json body --jq .body` contains `## What's Changed` and, when `PREVIOUS_DELIVERABLE` exists, `**Full Changelog**: https://github.com/rustfs/rustfs/compare/<PREVIOUS_DELIVERABLE>...<preview-tag>`. For a repository with no previous deliverable, verify a Full Changelog link exists and record the GitHub baseline fallback.
- Confirm Helm is skipped. If preview Docker images are published, confirm they use only exact preview tags (and variant suffixes), with no `latest`, `alpha`, `beta`, or `rc` channel updates. Preview validation covers the built RustFS binaries, embedded console, and rc compatibility; it does not authorize advancing any installation default.
## Release notes
- When a previous deliverable exists, GitHub Release notes for the preview and final tags MUST use it as their shared comparison baseline: the most recently published non-preview Release before the target. Internal `-preview.N` Releases are explicitly excluded from that selection, even when they point at the same commit as the final tag — cleanup runs after the notes are generated, so the preview Release is still present and would otherwise be picked as the baseline. If no previous deliverable exists, omit `previous_tag_name` and record that GitHub's default baseline fallback was used.
- Generated Release notes carry a workflow-management marker so retries can repair them. Before manually curating a generated body, remove that marker; unmarked non-placeholder notes are preserved by later workflow runs.
@@ -16,13 +16,22 @@ Read for Phases 1–2. The [parent release invariants](../SKILL.md) and Console
- Only when neither the release branch, an unfinished preview tag, nor current main already has `<target>`: if main is at a later version, select and verify a historical commit containing `<target>` rather than downgrading main; block if none exists. Otherwise invoke the `rustfs-release-version-bump` skill with stage `prepare`, the final `<target>` (NOT a preview version), and full GitHub flow (commit/push/PR).
- If a bump PR was needed, get it merged into main and record its merge commit. For a new release branch when main already has `<target>`, identify the main commit that introduced those versions. Phase 2 chooses a release candidate containing that commit; an existing release branch keeps its own history and a newer main HEAD is not required.
Do not wait for the moving latest main HEAD to become green. The release-branch CI in Phase 2 validates the chosen commit before a candidate build is dispatched.
Do not wait for the moving latest main HEAD to become green. The release-branch CI in Phase 2 validates the chosen commit before a preview tag is created.
## Phase 2 — Pin and validate the release source
## Phase 2 — Pin and validate the release candidate, then publish the preview tag
- Use `release/<target>`. On the first attempt, create it from a selected main commit containing the version bump and intended changes; it need not be the latest main HEAD. On a restart, retain the existing branch and backport only reviewed fixes from main. Verify the branch's Cargo versions still equal `<target>`. A legacy preview can anchor source selection, but its tag and old artifacts do not constitute a candidate in the new protocol; freeze and verify a new candidate after the release recipe is aligned.
- The current `ci.yml` runs automatically on main pushes, not release-branch pushes. Dispatch it explicitly with `gh workflow run ci.yml --ref "release/<target>"`. Locate that dispatch run, verify its `head_sha` equals the release branch's remote HEAD, and require the full expected CI job set to pass, including `Quick Checks`, `Test and Lint`, and `End-to-End Tests (full merge gate)`. Treat a missing or skipped required lane as incomplete. A successful PR check or CI run for a different SHA does not qualify. If the branch advances or a run is cancelled, validate the new SHA again. Do not dispatch a candidate with failing or incomplete CI.
- Use `release/<target>`. On the first attempt, create it from a selected main commit containing the version bump and intended changes; it need not be the latest main HEAD. On a restart, retain the existing branch and backport only reviewed fixes from main. Verify the branch's Cargo versions still equal `<target>`. If adopting an existing preview, run this branch CI before continuing its acceptance; reuse the tag only when it points to the validated SHA.
- The current `ci.yml` runs automatically on main pushes, not release-branch pushes. Dispatch it explicitly with `gh workflow run ci.yml --ref "release/<target>"`. Locate that dispatch run, verify its `head_sha` equals the release branch's remote HEAD, and require the full expected CI job set to pass, including `Quick Checks`, `Test and Lint`, and `End-to-End Tests (full merge gate)`. Treat a missing or skipped required lane as incomplete. A successful PR check or CI run for a different SHA does not qualify. If the branch advances or a run is cancelled, validate the new SHA again. Do not tag a candidate with failing or incomplete CI.
- Prefer protecting `release/*` against force pushes and deletion and requiring reviewed backport PRs. Check live rulesets rather than assuming the main ruleset covers release branches; report a missing release-branch rule as a gap.
- Set `CANDIDATE_SHA` to the CI-validated release-branch commit. Record the branch, exact commit and CI run URL. The new candidate protocol starts from a commit whose release recipe matches the reviewed main dispatcher; backport the recipe before freezing an older release branch if needed.
- Dispatch the Linux candidate as described in [candidate verification](preview-release.md). Do not create a preview or final tag here. The final executable already contains `<target>`; publication later promotes those same bytes.
- After a source or build-input change, run the affected source checks and create a new candidate. Reuse the frozen candidate after an infrastructure failure. Do not restart acceptance for unrelated main advances.
- Set `PREVIEW_HASH` to the CI-validated release-branch commit and recheck the remote branch HEAD immediately before tagging. Report the branch, commit, and CI run URL; do not derive `PREVIEW_HASH` from a later `origin/main` fetch. When adopting an existing preview tag at this SHA, skip tag creation and continue Phase 3.
```bash
git tag -a "<preview-tag>" -m "Release <preview-tag>" "$PREVIEW_HASH"
git push origin "<preview-tag>"
```
Pushing the tag triggers `.github/workflows/build.yml` ("Build and Release"); `docker.yml` chains off it via `workflow_run`.
The preview run builds versioned artifacts and publishes them in a GitHub prerelease. Docker may publish exact preview image tags. Latest-channel, R2, and Helm publication must be skipped; installation references and the website announcement stay unchanged.
On a restart after a backport, re-run release-branch CI before setting the new `PREVIEW_HASH`. Use the next unused preview iteration when the validated commit differs from an existing preview tag; never move an existing tag.
@@ -4,18 +4,18 @@ description: Prepare an exact RustFS Cargo version or align installation referen
---
# Release Version Files
Require an exact target; ask if missing or ambiguous. Reject targets containing `-preview`: those identify candidates, not product versions or new public tags. Infer `post-release` only from an explicit request or parent workflow; otherwise use `prepare`. Delivery defaults to local edit/verify; commit, push, PR, and merge follow existing authorization and [Git rules](../../references/pull-requests.md).
Require an exact target; ask if missing or ambiguous. Reject targets containing `-preview`: those are tag-only. Infer `post-release` only from an explicit request or parent workflow; otherwise use `prepare`. Delivery defaults to local edit/verify; commit, push, PR, and merge follow existing authorization and [Git rules](../../references/pull-requests.md).
| Stage | Files and constraints |
|---|---|
| `prepare` | Only `Cargo.toml` and `Cargo.lock`: workspace package, internal dependency, and workspace member versions become `<target>`. External dependencies stay unchanged. |
| `post-release` | Only `README.md`, `README_ZH.md`, `flake.nix`, `helm/rustfs/Chart.yaml`, and `rustfs.spec`. Cargo may already target the next release; leave it untouched. |
Before installation edits, verify the exact non-preview Release, downloadable assets/source archive, pullable `rustfs/rustfs:<target>` image manifest, and published Helm chart. Block on missing artifacts. Do not downgrade defaults if a newer deliverable superseded the target. Candidate preparation retains the previous published installation references; follow-up commits never replace validated tags or sealed artifacts.
Before installation edits, verify the exact non-preview Release, downloadable assets/source archive, pullable `rustfs/rustfs:<target>` image manifest, and published Helm chart. Block on missing artifacts. Do not downgrade defaults if a newer deliverable superseded the target. Preview preparation retains the previous published installation references; follow-up commits never replace either validated tag.
Packaging policy:
- Docker image tags have no `v` prefix. Derive chart/app versions with `scripts/helm_chart_version.sh <target>`; both equal the final product SemVer, including prerelease suffixes. Helm CI uses the frozen candidate version before sealing, so no early installation chart bump is required. Previously published chart versions remain unchanged.
- Docker image tags have no `v` prefix. Derive chart/app versions with `scripts/helm_chart_version.sh <target>` (`beta.N -> 0.N.0`; other versions unchanged). Helm CI derives versions from the final tag, so no early chart bump is required.
- RPM `Version` is numeric; `Release` is the prerelease suffix or `1` for stable. Expanded `Source0` and the unpack directory must resolve to the exact target including its suffix. Add the changelog entry using current git identity, date, and target; never copy an example identity/date.
- Changing these mappings requires explicit confirmation.
-11
View File
@@ -31,14 +31,3 @@
# the flag, so the two can no longer drift apart unnoticed.
#
# For CPU profiling, add `-C force-frame-pointers=yes` to that RUSTFLAGS value.
# Exact internode HTTP/2 PING ACK timeout accounting is also deliberately
# default-off. Hyper exposes the exact timeout only through its unstable tracing
# event, so diagnostic builds must opt in to both the feature and cfg:
#
# RUSTFLAGS="--cfg hyper_unstable_tracing" \
# cargo build -p rustfs --features diagnostic-internode-http2-ping
#
# `crates/protos/build.rs` and `crates/obs/build.rs` fail fast if the feature and
# cfg drift apart. Do not move this cfg into global rustflags: doing so would
# recompile every dependency under an unstable, non-semver build contract.
-16
View File
@@ -1,16 +0,0 @@
{
"schema_version": 1,
"rollout": "enforced",
"core_packages": ["rustfs", "rustfs-ecstore", "rustfs-filemeta", "rustfs-iam", "rustfs-policy"],
"connect_packages": [
"rustfs-credentials", "rustfs-config", "rustfs-common", "rustfs-crypto",
"rustfs-iam", "rustfs-policy", "rustfs-kms", "rustfs-license", "rustfs-security-governance",
"rustfs-utils", "rustfs-tls-runtime", "rustfs-trusted-proxies", "rustfs-filemeta",
"rustfs-lock", "rustfs-obs", "rustfs-signer", "rustfs-io-metrics", "rustfs-s3-ops"
],
"full_paths": [
"Cargo.toml", "Cargo.lock", "*/Cargo.toml", "rust-toolchain*", "rustfmt.toml",
"clippy.toml", ".cargo/*", ".config/*", ".github/*", "Makefile",
"scripts/*", "build.rs", "*.proto"
]
}
-10
View File
@@ -1,10 +0,0 @@
# Rulings for the gateway conformance lane (.github/workflows/gateway-acceptance.yml).
#
# One row per case that fails on the GATEWAY stack for a reviewed reason: `case`; `ledger`, the
# rustfs/backlog#2684 entry that accepts the difference (R-0xx or K-n); a one-line `reason`; and
# `failure`, a prefix of the failure the ruling accepts, as the runner's report.json renders it
# (`[expect/status] at /expect/status: exchange #1, attempt 1: expected status 400, observed 403`).
# scripts/check_conformance_rulings.py fails the lane on a failed case without a row, on a ruled
# case whose failures do not start with its `failure`, and on a row whose case no longer fails or
# no longer exists. The legacy stack is a reference run and takes no rulings: the `rustfs` profile
# describes the gateway edge, not the legacy one.
+2 -2
View File
@@ -1,2 +1,2 @@
sha256-linux=348fa5eed9ae58962c06b33283ff4f15122cff97bce9b4eb1e3ffcb227264b0b
sha256-darwin=348fa5eed9ae58962c06b33283ff4f15122cff97bce9b4eb1e3ffcb227264b0b
sha256-linux=eb33067d90ee66c0a97e8f3658e46c152e47f897d2fbfdc6cb5ad93cfa55414b
sha256-darwin=eb33067d90ee66c0a97e8f3658e46c152e47f897d2fbfdc6cb5ad93cfa55414b
+2 -2
View File
@@ -1,2 +1,2 @@
sha256-darwin=24a8cfaa1b9125bd69d7d4a7ff752a8ab8ed350c761d90ae8472ef52393e2686
sha256-linux=014257197fbbfbb48cab2864da9e10d96ee877d478fa29d119f8186532ea293d
sha256-darwin=a67a347f26be186c598b5beaf68e6bcbe1855350a35e1423dd5b8a81bcba06a5
sha256-linux=ad4fabe811c6cca1aba89fad34ccd404b5832a6a955875c2826f6e0fa1ea4f1f
+2 -2
View File
@@ -1,2 +1,2 @@
sha256-darwin=e84e04208f19dc0596b10d4d1b799ce43d6cc431415902f9c5458aa8232bc2d8
sha256-linux=1f9c7add72c44d1c12ef4ac050bb7917431be657c0c90b29f70c3df56f8bbae3
sha256-darwin=3814cdde8cacf4187e18188d342ed70931cc32e268e42e5bb438d565bb4d0f81
sha256-linux=663025a80273579f95ae3f49af02c0afcbfce94df09701c360a13e607b7637db
+2 -2
View File
@@ -1,2 +1,2 @@
sha256-darwin=dcc076f8a36320d2fe36b4d2f26e2d54c3b9bc2ff7e14c6e90241f4fd8cf4a33
sha256-linux=229ebc0cbc9b35863600ae60c6fbb548516efe2a2f41f4b5db503b3283c2f64f
sha256-darwin=a2cb67d91eb0d093928371d6e84a84b2dd41b2ba975a28094e81879bd65838ad
sha256-linux=82db0561cf9b111f11782d647d0859c9d2a44c5c68481b5e39c48505828bd392
+1 -1
View File
@@ -1 +1 @@
sha256=4eac269f9a59b2f9be4e7a87ea6aa38454f5ebe10ede35c9b8d4218137d2cd35
sha256=e4ee3898b840a8374cb02c64342cc15f8d7ecd1becaf4b4a2a4657290f9e7ff1
+1 -1
View File
@@ -1 +1 @@
sha256=fd5683273bba1eae3b4402a4e854a5d168ace845bd0e5fc801f8e336485a4394
sha256=f065e7071aac9d48a7c1b301f328fa7cc14f11c4e7dc46da3e55057c4ea29bd5
-21
View File
@@ -70,26 +70,6 @@ s3s-footprint-check: ## Check the s3s dependency footprint ratchet stays frozen
@echo "📦 Checking s3s footprint ratchet..."
./scripts/check_s3s_footprint.sh
.PHONY: release-feature-boundary-check
release-feature-boundary-check: ## Check no release-reachable feature enables a debug-only one
@echo "🚧 Checking release feature boundaries..."
$(RUSTFS_PYTHON_BIN) ./scripts/check_connect_release_boundary.py --features-only
.PHONY: test-count-ratchet-check
test-count-ratchet-check: ## Check no workspace crate fell below its test floor or rose above its #[ignore] ceiling in the committed baseline
@echo "🧪 Checking per-crate test-count ratchet..."
./scripts/check_test_count_ratchet.sh
.PHONY: test-seams-check
test-seams-check: ## Check tests outside an operation's owner call its use-case only through rustfs/src/app/test_seams
@echo "🧪 Checking cross-owner test seams..."
$(RUSTFS_PYTHON_BIN) ./scripts/check_test_seams.py
.PHONY: test-retirements-check
test-retirements-check: ## Check every test retired by a migration slice names a replacement that exists
@echo "🧪 Checking test retirement tables..."
$(RUSTFS_PYTHON_BIN) ./scripts/check_test_retirements.py
.PHONY: fips-wording-check
fips-wording-check: ## Check docs and crates/kms do not over-claim crypto capabilities
@echo "📣 Checking cryptographic capability wording guard..."
@@ -110,7 +90,6 @@ test-wiring-check: ## Check tests stay registered and selected by their intended
@echo "🧪 Checking test wiring..."
$(RUSTFS_PYTHON_BIN) ./scripts/check_test_wiring.py
$(RUSTFS_PYTHON_BIN) ./scripts/ci_gate.py --check-workflow
$(RUSTFS_PYTHON_BIN) ./scripts/validation_registry.py check
.PHONY: log-analyzer-rules-check
log-analyzer-rules-check: core-deps ## Check log-analyzer rule anchors still exist verbatim in source
+2 -2
View File
@@ -20,11 +20,11 @@ planning-docs-check: ## Check that no planning-type documents are committed
./scripts/check_no_planning_docs.sh
.PHONY: pre-commit
pre-commit: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check error-other-ratchet-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check s3s-footprint-check release-feature-boundary-check test-count-ratchet-check test-seams-check test-retirements-check fips-wording-check embedded-secrets-check test-wiring-check doc-paths-check planning-docs-check quick-check ## Run fast pre-commit checks without clippy/full tests
pre-commit: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check error-other-ratchet-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check s3s-footprint-check fips-wording-check embedded-secrets-check test-wiring-check doc-paths-check planning-docs-check quick-check ## Run fast pre-commit checks without clippy/full tests
@echo "✅ All pre-commit checks passed!"
.PHONY: pre-pr
pre-pr: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check error-other-ratchet-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check s3s-footprint-check release-feature-boundary-check test-count-ratchet-check test-seams-check test-retirements-check fips-wording-check embedded-secrets-check test-wiring-check doc-paths-check planning-docs-check log-analyzer-rules-check offline-enrollment-e2e-check clippy-check test ## Run full pre-PR checks with clippy and tests
pre-pr: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check error-other-ratchet-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check s3s-footprint-check fips-wording-check embedded-secrets-check test-wiring-check doc-paths-check planning-docs-check log-analyzer-rules-check offline-enrollment-e2e-check clippy-check test ## Run full pre-PR checks with clippy and tests
@echo "✅ All pre-PR checks passed!"
.PHONY: dev-check
-44
View File
@@ -22,28 +22,14 @@ TEST_THREADS ?= 1
# differ from CI and .config/nextest.toml test-groups will NOT apply.
.PHONY: script-tests
.PHONY: release-package-tests
release-package-tests: ## Run release package integration on Linux with fpm, dpkg and rpm installed
$(RUSTFS_PYTHON_BIN) ./scripts/test_release_packages.py
.PHONY: release-image-tests
release-image-tests: ## Run OCI integration with Docker, Go and skopeo; set RELEASE_IMAGE_VARIANT and RELEASE_IMAGE_OUTPUT
$(RUSTFS_PYTHON_BIN) ./scripts/test_release_images.py
script-tests: ## Run shell script tests
@echo "Running script tests..."
trap 'status=$$?; printf "ERROR: script-tests failed (exit %s): %s\n" "$$status" "$$BASH_COMMAND" >&2; exit "$$status"' ERR
./scripts/test_build_rustfs_options.sh
./scripts/test_docker_runtime_timezone.sh
./scripts/test_entrypoint_credentials.sh
./scripts/test_entrypoint_directories.sh
./scripts/test_internode_grpc_ab_bench.sh
$(RUSTFS_PYTHON_BIN) ./scripts/test_internode_grpc_window.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_internode_rpc_timeout_boundary.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_internode_msgpack_peer_canary.py
./scripts/test_internode_msgpack_peer_canary_runner.sh
./scripts/test_object_batch_bench_enhanced.sh
$(RUSTFS_PYTHON_BIN) ./scripts/test_inline_docker_lab.py
./scripts/test_package_service_scripts.sh
./scripts/test_hotpath_warp_ab_gate.sh
./scripts/test_hotpath_warp_abba.sh
@@ -63,54 +49,24 @@ script-tests: ## Run shell script tests
./scripts/test_manual_transition_runbooks.sh
./scripts/test_fuzz_runner.sh
./scripts/test_python_bin.sh
$(RUSTFS_PYTHON_BIN) ./scripts/test_check_doc_paths.py
./scripts/check_embedded_secrets.sh --self-test
./scripts/check_s3s_footprint.sh --self-test
./scripts/test_check_test_count_ratchet.sh
./scripts/test_guard_source_set.sh
./scripts/test_split_guard_rules.sh
$(RUSTFS_PYTHON_BIN) ./scripts/check_test_wiring.py --self-test
$(RUSTFS_PYTHON_BIN) ./scripts/codemods/s3_error_to_s3_types.py --self-test
$(RUSTFS_PYTHON_BIN) ./scripts/codemods/split_bucket_usecase.py --self-test
$(RUSTFS_PYTHON_BIN) ./scripts/codemods/split_ecfs_legacy_edge.py --self-test
$(RUSTFS_PYTHON_BIN) ./scripts/codemods/split_lifecycle_transition_api_test.py --self-test
$(RUSTFS_PYTHON_BIN) ./scripts/check_test_seams.py --self-test
$(RUSTFS_PYTHON_BIN) ./scripts/check_test_retirements.py --self-test
$(RUSTFS_PYTHON_BIN) ./scripts/test_e2e_binary.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_download_release_asset.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_reference_binary.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_inspect_btf_prototypes.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_release_build_workflow.py
$(RUSTFS_PYTHON_BIN) -m unittest discover -s scripts/release -p "test_*.py"
$(RUSTFS_PYTHON_BIN) ./scripts/test_release_glibc.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_upgrade_workflow.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_migration_gate_evidence.py
$(RUSTFS_PYTHON_BIN) ./scripts/ci_gate.py --self-test
$(RUSTFS_PYTHON_BIN) ./scripts/check_connect_release_boundary.py --self-test
$(RUSTFS_PYTHON_BIN) ./scripts/test_ci_test_artifact.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_ci_merge_group_proof.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_ci_merge_queue.py
$(RUSTFS_PYTHON_BIN) -m unittest discover -s scripts/ci -p 'test_*.py'
$(RUSTFS_PYTHON_BIN) ./scripts/check_security_coverage.py --self-test
$(RUSTFS_PYTHON_BIN) ./scripts/check_scheduled_validation_freshness.py --self-test
$(RUSTFS_PYTHON_BIN) ./scripts/test_security_workflow.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_nightly_candidate.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_functional_chain.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_functional_chain_health.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_repro_rejoin_readiness_docker.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_validation_registry.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_inspect_validation_runner.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_retain_validation_evidence.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_ci_timing_report.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_junit_diff.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_dual_stack.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_mint_stack_diff.py
$(RUSTFS_PYTHON_BIN) ./scripts/check_conformance_rulings.py --self-test
$(RUSTFS_PYTHON_BIN) ./scripts/compat_matrix_report.py --self-test
$(RUSTFS_PYTHON_BIN) ./scripts/s3-tests/test_report_compat.py
$(RUSTFS_PYTHON_BIN) ./scripts/s3-tests/test_runner_tools.py
$(RUSTFS_PYTHON_BIN) ./scripts/s3-tests/test_api_coverage.py
$(RUSTFS_PYTHON_BIN) ./scripts/s3-tests/test_reference_parity.py
bash -n ./scripts/validate_object_data_cache_cold_stampede.sh
$(RUSTFS_PYTHON_BIN) ./scripts/check_object_data_cache_follower_samples.py --self-test
./scripts/validate_object_data_cache_cold_stampede.sh --self-test
+18 -107
View File
@@ -57,11 +57,6 @@ command = ['sh', '-c', 'echo RUST_MIN_STACK=4194304 >> "$NEXTEST_ENV"']
[scripts.setup.ecstore-large-stack]
command = ['sh', '-c', 'echo RUST_MIN_STACK=33554432 >> "$NEXTEST_ENV"']
# The receipt corruption oracle does not simulate a crash. Prefer tmpfs only
# when its full fixture fits; retain the ordinary temporary directory otherwise.
[scripts.setup.ecstore-receipt-tmpdir]
command = ['sh', 'scripts/ci/receipt-fixture-tmpdir.sh']
# The serial ILM selection builds the same deep storage futures in both the
# lifecycle transition module and scanner integration binary. Different tests
# in each have overflowed first across otherwise unrelated CI runs.
@@ -96,12 +91,7 @@ threads-required = "num-test-threads"
# acquisition budget. Reserve capacity from sibling test processes while
# preserving the probes' internal concurrency, deadlines, and assertions.
[[profile.default.overrides]]
filter = '(package(rustfs-ecstore) & (test(/^core::pools_test::decommission_lock_order_tests::(active_decommission_migration_does_not_invert_capacity_and_delete_object(_tag)?_locks|decommission_moves_a_transitioned_version_expired_by_a_delete_all_rule_without_stalling)$/) | test(=services::integrity::tests::audit_migration_and_ack_recovery_preserve_source_and_reject_conflicts))) | (package(rustfs-heal) & test(=heal::mrf_queue::partial_write::tests::durable_retry_submission_error_keeps_checkpoint_and_uses_backoff))'
threads-required = "num-test-threads"
# These eight-disk namespace-owner probes overlap durable cleanup with a lock contender. Reserve capacity so sibling disk fixtures cannot consume their lock and completion budgets.
[[profile.default.overrides]]
filter = 'package(rustfs-ecstore) & test(/^store::object::tests::(canceled_create_retains_usage_cleanup_namespace_owner|batch_delete_excludes_recovery_on_nonzero_pool|delete_excludes_recovery_with_(disjoint|identical|overlapping)_pool_lock_clients)$/)'
filter = '(package(rustfs-ecstore) & test(=core::pools_test::decommission_lock_order_tests::active_decommission_migration_does_not_invert_capacity_and_delete_object_tag_locks)) | (package(rustfs-heal) & test(=heal::mrf_queue::partial_write::tests::durable_retry_submission_error_keeps_checkpoint_and_uses_backoff))'
threads-required = "num-test-threads"
# These eight-disk decommission fixtures exercise concurrent retries,
@@ -115,7 +105,7 @@ threads-required = "num-test-threads"
# Reserve capacity so sibling disk-heavy tests cannot consume its 2s
# registration budget; retain the queued backup and deadlock assertions.
[[profile.default.overrides]]
filter = 'package(rustfs) & test(=admin::handlers::table_catalog::tests::cold_object_registration_completes_while_other_bucket_backup_waits)'
filter = 'package(rustfs) & test(=admin::handlers::table_catalog::tests::cold_strong_registration_completes_while_other_bucket_backup_waits)'
threads-required = "num-test-threads"
[[profile.default.overrides]]
@@ -138,12 +128,11 @@ setup = 'ecstore-large-stack'
# path on a real store; measured with RUST_MIN_STACK on 2026-10-15 they need
# 2016-2040 KiB of the 2 MiB libtest default and overflowed on Linux CI.
[[profile.default.scripts]]
filter = '(package(rustfs) & test(/^(admin::handlers::replication::target_repair_tests::[a-z_]+|admin::handlers::plugin_attempt::v2_tests::delegation_v2_persists_and_vends_with_real_parent_intersection|admin::handlers::site_replication::tests::durable_iam_state_apply_iam_item_rejects_a_stale_recreate_after_a_replicated_delete|admin::service::federated_identity::tests::(binding_keeps_explicit_iam_mapping_strict|markerless_legacy_oidc_session_crosses_auth_and_iam_deletion_boundaries)|storage::scanner_dirty_journal::tests::(clear_callback_keeps_unknown_coverage_until_all_pending_work_is_confirmed|owned_journals_keep_peer_records_after_interleaved_writes_and_clear))$/)) | (package(rustfs-heal) & ((binary(heal_b5_versioned_regression_test) & test(/^serial_tests::(test_heal_unversioned_bucket_e2e|test_historical_null_recursive_heal_restores_physical_versions|test_recursive_deep_heal_reports_stale_delete_marker_repair)$/)) | (binary(usage_observation_read_repair_test) & test(=usage_observation_read_repair_survives_snapshot_cleanup)))) | (package(rustfs-test-utils) & test(=data_usage_snapshot_tests::data_usage_snapshot_recovery_wires_primary_backup_and_negative_cache))'
filter = '(package(rustfs) & test(/^(admin::handlers::replication::target_repair_tests::[a-z_]+|admin::handlers::site_replication::tests::durable_iam_state_apply_iam_item_rejects_a_stale_recreate_after_a_replicated_delete|admin::service::federated_identity::tests::(binding_keeps_explicit_iam_mapping_strict|markerless_legacy_oidc_session_crosses_auth_and_iam_deletion_boundaries)|storage::scanner_dirty_journal::tests::(clear_callback_keeps_unknown_coverage_until_all_pending_work_is_confirmed|owned_journals_keep_peer_records_after_interleaved_writes_and_clear))$/)) | (package(rustfs-heal) & ((binary(heal_b5_versioned_regression_test) & test(/^serial_tests::(test_heal_unversioned_bucket_e2e|test_historical_null_recursive_heal_restores_physical_versions|test_recursive_deep_heal_reports_stale_delete_marker_repair)$/)) | (binary(usage_observation_read_repair_test) & test(=usage_observation_read_repair_survives_snapshot_cleanup)))) | (package(rustfs-test-utils) & test(=data_usage_snapshot_tests::data_usage_snapshot_recovery_wires_primary_backup_and_negative_cache))'
setup = 'ecstore-base-stack'
# The multipart refusal fixture composes real-store setup, abort, quota and legal-hold futures.
[[profile.default.scripts]]
filter = 'package(rustfs-ecstore) | package(rustfs-s3select-api) | package(rustfs-scanner) | (package(rustfs) & test(/^(app::multipart_usecase::complete::tests::completion_absent_binding_refusal_order_baseline|app::multipart_usecase::complete::tests::concurrent_completions_share_durable_bucket_quota_reservations|app::object::delete::tests::compressed_delete_requests_update_observed_usage_without_releasing_quota_floor|app::object::internal_put::tests::internal_multipart_roundtrip_completes_and_abort_leaves_nothing|app::object::restore::tests::execute_restore_object_maps_failures_to_typed_s3_errors|storage::access::tests::(delete_object_access_captures_authorized_bucket_incarnation|copy_operations_reject_recreated_source_bucket_after_authorization|request_slot_keeps_bucket_policy_bound_to_its_store))$/))'
filter = 'package(rustfs-ecstore) | package(rustfs-s3select-api) | package(rustfs-scanner) | (package(rustfs) & test(/^(app::multipart_usecase::tests::concurrent_completions_share_durable_bucket_quota_reservations|app::object::delete::tests::compressed_delete_requests_update_observed_usage_without_releasing_quota_floor|app::object::internal_put::tests::internal_multipart_roundtrip_completes_and_abort_leaves_nothing|app::object::restore::tests::execute_restore_object_maps_failures_to_typed_s3_errors|storage::access::tests::(delete_object_access_captures_authorized_bucket_incarnation|copy_operations_reject_recreated_source_bucket_after_authorization|request_slot_keeps_bucket_policy_bound_to_its_store))$/))'
setup = 'ecstore-base-stack'
# These handler/store fixtures and the relocated-pool GET compose deep storage
@@ -156,16 +145,10 @@ setup = 'ecstore-base-stack'
filter = 'binary(lifecycle_integration_test) | (package(rustfs) & test(/^app::lifecycle_transition_api_test::/))'
setup = 'lifecycle-large-stack'
# This conditional GET fixture composes real-store futures that overflowed
# the default Linux test-thread stack.
# The pool metadata placement regression composes the full erasure-heal
# startup future and overflows the default Linux test-thread stack.
[[profile.default.scripts]]
filter = 'package(rustfs) & test(=app::object::get::tests::conditional_get_304_uses_metadata_without_decoding_damaged_data)'
setup = 'ecstore-large-stack'
# These heal fixtures compose deep real-store recovery futures that can
# overflow the default Linux test-thread stack.
[[profile.default.scripts]]
filter = 'package(rustfs-heal) & (test(=replacement_pool_metadata_follows_real_two_set_placement) | (binary(heal_integration_test) & test(=bucket_removal_recovers_delete_proofs_before_removing_empty_volumes)))'
filter = 'package(rustfs-heal) & test(=replacement_pool_metadata_follows_real_two_set_placement)'
setup = 'ecstore-large-stack'
# These real-disk scanner fixtures assert quiet-workload publication and
@@ -240,7 +223,7 @@ test-group = 'embedded-test-ports'
# processes cannot exhaust those budgets. Keep deadlines and assertions intact.
# The short HTTP fixtures are tracked in rustfs/backlog#2714.
[[profile.default.overrides]]
filter = 'package(rustfs) & (binary(/^connect_registration$/) | binary(connect_inventory) | binary(connect_perf_drive) | (binary(connect_perf_client) & test(=configured_proxy_and_direct_endpoint_failures_are_distinct)) | (binary(connect_perf_object) & test(/^(real_rustfs_endpoint_and_production_cli_support_bounded_get_and_put|cleanup_uses_a_budget_independent_of_the_measurement_window|cleanup_accepts_missing_object_and_still_deletes_the_bucket|cleanup_diagnostics_identify_each_disconnected_transport_stage|cleanup_diagnostics_identify_each_failed_http_stage_without_private_context|cleanup_diagnostics_identify_timeouts_without_resetting_the_shared_deadline|oversized_response_is_rejected_without_buffering_it)$/)))'
filter = 'package(rustfs) & (binary(/^connect_registration$/) | binary(connect_inventory) | binary(connect_perf_drive) | (binary(connect_perf_client) & test(=configured_proxy_and_direct_endpoint_failures_are_distinct)) | (binary(connect_perf_object) & test(/^(real_rustfs_endpoint_and_production_cli_support_bounded_get_and_put|cleanup_uses_a_budget_independent_of_the_measurement_window|oversized_response_is_rejected_without_buffering_it)$/)))'
threads-required = "num-test-threads"
# This cancellation probe hashes the serving executable before acquiring its
@@ -362,12 +345,6 @@ threads-required = "num-test-threads"
filter = 'package(rustfs-ecstore) & test(=store::init::tests::dispatch_manifest_rollback_bounded_concurrency_reaches_tail_behind_slow_member)'
threads-required = "num-test-threads"
# These short-deadline real-disk progress probes need the test run's capacity
# for scanner and object-lock handoffs; keep their assertions and deadlines.
[[profile.default.overrides]]
filter = '(package(rustfs-scanner) & (test(=scanner::tests::cycle_persist_failure::later_cycle_post_commit_failure_reloads_durable_state) | test(=scanner::tests::running_main_loop_catches_up_pause_cleared_after_startup_observe))) | (package(rustfs-ecstore) & test(=store::init::tests::restored_transitioned_delete_causally_enqueues_free_version))'
threads-required = "num-test-threads"
# ---------------------------------------------------------------------------
# ci profile — the strict CI gate (ci.yml `cargo nextest run --profile ci`)
# ---------------------------------------------------------------------------
@@ -408,12 +385,7 @@ threads-required = "num-test-threads"
# acquisition budget. Reserve capacity from sibling test processes while
# preserving the probes' internal concurrency, deadlines, and assertions.
[[profile.ci.overrides]]
filter = '(package(rustfs-ecstore) & (test(/^core::pools_test::decommission_lock_order_tests::(active_decommission_migration_does_not_invert_capacity_and_delete_object(_tag)?_locks|decommission_moves_a_transitioned_version_expired_by_a_delete_all_rule_without_stalling)$/) | test(=services::integrity::tests::audit_migration_and_ack_recovery_preserve_source_and_reject_conflicts))) | (package(rustfs-heal) & test(=heal::mrf_queue::partial_write::tests::durable_retry_submission_error_keeps_checkpoint_and_uses_backoff))'
threads-required = "num-test-threads"
# These eight-disk namespace-owner probes overlap durable cleanup with a lock contender. Reserve capacity so sibling disk fixtures cannot consume their lock and completion budgets.
[[profile.ci.overrides]]
filter = 'package(rustfs-ecstore) & test(/^store::object::tests::(canceled_create_retains_usage_cleanup_namespace_owner|batch_delete_excludes_recovery_on_nonzero_pool|delete_excludes_recovery_with_(disjoint|identical|overlapping)_pool_lock_clients)$/)'
filter = '(package(rustfs-ecstore) & test(=core::pools_test::decommission_lock_order_tests::active_decommission_migration_does_not_invert_capacity_and_delete_object_tag_locks)) | (package(rustfs-heal) & test(=heal::mrf_queue::partial_write::tests::durable_retry_submission_error_keeps_checkpoint_and_uses_backoff))'
threads-required = "num-test-threads"
# These eight-disk decommission fixtures exercise concurrent retries,
@@ -427,7 +399,7 @@ threads-required = "num-test-threads"
# Reserve capacity so sibling disk-heavy tests cannot consume its 2s
# registration budget; retain the queued backup and deadlock assertions.
[[profile.ci.overrides]]
filter = 'package(rustfs) & test(=admin::handlers::table_catalog::tests::cold_object_registration_completes_while_other_bucket_backup_waits)'
filter = 'package(rustfs) & test(=admin::handlers::table_catalog::tests::cold_strong_registration_completes_while_other_bucket_backup_waits)'
threads-required = "num-test-threads"
[[profile.ci.overrides]]
@@ -441,32 +413,17 @@ threads-required = "num-test-threads"
filter = 'package(rustfs-scanner) & (test(/^scanner::backlog::tests::native_/) | test(=scanner_folder::tests::mrf_ownership::mrf_ownership_manager_completion_preserves_scanner_pending) | test(/^scanner_io::tests::service_cohort::/))'
threads-required = "num-test-threads"
# Keep short-deadline real-disk progress probes from competing with unrelated
# tests for scanner and object-lock capacity. Assertions and deadlines stay on.
[[profile.ci.overrides]]
filter = '(package(rustfs-scanner) & (test(=scanner::tests::cycle_persist_failure::later_cycle_post_commit_failure_reloads_durable_state) | test(=scanner::tests::running_main_loop_catches_up_pause_cleared_after_startup_observe))) | (package(rustfs-ecstore) & test(=store::init::tests::restored_transitioned_delete_causally_enqueues_free_version))'
threads-required = "num-test-threads"
# Emit this expensive fixture's stage timings in CI and use tmpfs when safe.
[[profile.ci.overrides]]
filter = 'package(rustfs-ecstore) & test(=store::init::tests::decommission_durable_ilm_receipt_pagination_fails_closed_on_second_page)'
success-output = 'immediate'
[[profile.ci.scripts]]
filter = 'package(rustfs-ecstore) & test(=store::init::tests::decommission_durable_ilm_receipt_pagination_fails_closed_on_second_page)'
setup = 'ecstore-receipt-tmpdir'
[[profile.ci.scripts]]
filter = 'package(rustfs-ecstore) & test(/^(bucket::lifecycle::bucket_lifecycle_ops::tests::manual_transition_worker_result_recovery_marks_unknown_for_corrupt_marker|services::rebalance::entry::tests::real_rebalance_run_fence_loss_blocks_multipart_publication|store::init::tests::(batch_transitioned_delete_uses_free_version_per_item|decommission_entry_(allows_free_version_consumed_before_source_lock|rejects_subquorum_free_version_conflict_and_retains_source|skips_cleanup_only_marker_when_free_version_is_present)|dispatched_tier_delete_recovery_(checks_later_pool_then_commits_after_source_removal|finds_directory_source_on_encoded_set|retains_journal_on_source_metadata_error)|force_tier_remove_blocks_on_physical_free_version_hidden_by_other_pool|legacy_unknown_transition_delete_falls_back_for_single_batch_and_blocks_prefix|multi_pool_(recursive_prefix_rejects_legacy_or_hidden_merge_loser_before_delete|same_remote_tuple_(batch|single)_delete_waits_for_all_sources|same_tuple_recursive_prefix_uses_one_journal_owner|transitioned_delete_persists_one_free_version_per_remote_tuple)|recursive_prefix_partial_(pool|set)_failure_keeps_prepared_cleanup_owners|restored_transitioned_delete_uses_free_version_as_cleanup_owner|stable_transitioned_recursive_prefix_delete_uses_journal_owners|suspended_null_transition_delete_uses_free_version_as_sole_owner|tier_mutation_peer_handler_applies_prepare_commit_and_abort_idempotently|transition_response_loss_persists_unknown_outcome_for_provider_recovery|transition_transaction_recovery_(drops_record_after_confirmed_local_commit|keeps_cleanup_pending_local_commit)|transitioned_delete_(free_version_replays_after_store_restart|local_quorum_failure_rolls_back_without_cleanup_owner|uses_free_version_as_cleanup_owner)|versioned_delete_marker_keeps_transitioned_source_and_remote_object|versioned_explicit_transition_delete_preserves_other_version_then_allows_bucket_delete))$/)'
setup = 'ecstore-large-stack'
# These cross-crate real-storage fixtures need the same bounded stack as ECStore tests.
[[profile.ci.scripts]]
filter = '(package(rustfs) & test(/^(admin::handlers::replication::target_repair_tests::[a-z_]+|admin::handlers::plugin_attempt::v2_tests::delegation_v2_persists_and_vends_with_real_parent_intersection|admin::handlers::site_replication::tests::durable_iam_state_apply_iam_item_rejects_a_stale_recreate_after_a_replicated_delete|admin::service::federated_identity::tests::(binding_keeps_explicit_iam_mapping_strict|markerless_legacy_oidc_session_crosses_auth_and_iam_deletion_boundaries)|storage::scanner_dirty_journal::tests::(clear_callback_keeps_unknown_coverage_until_all_pending_work_is_confirmed|owned_journals_keep_peer_records_after_interleaved_writes_and_clear))$/)) | (package(rustfs-heal) & ((binary(heal_b5_versioned_regression_test) & test(/^serial_tests::(test_heal_unversioned_bucket_e2e|test_historical_null_recursive_heal_restores_physical_versions|test_recursive_deep_heal_reports_stale_delete_marker_repair)$/)) | (binary(usage_observation_read_repair_test) & test(=usage_observation_read_repair_survives_snapshot_cleanup)))) | (package(rustfs-test-utils) & test(=data_usage_snapshot_tests::data_usage_snapshot_recovery_wires_primary_backup_and_negative_cache))'
filter = '(package(rustfs) & test(/^(admin::handlers::replication::target_repair_tests::[a-z_]+|admin::handlers::site_replication::tests::durable_iam_state_apply_iam_item_rejects_a_stale_recreate_after_a_replicated_delete|admin::service::federated_identity::tests::(binding_keeps_explicit_iam_mapping_strict|markerless_legacy_oidc_session_crosses_auth_and_iam_deletion_boundaries)|storage::scanner_dirty_journal::tests::(clear_callback_keeps_unknown_coverage_until_all_pending_work_is_confirmed|owned_journals_keep_peer_records_after_interleaved_writes_and_clear))$/)) | (package(rustfs-heal) & ((binary(heal_b5_versioned_regression_test) & test(/^serial_tests::(test_heal_unversioned_bucket_e2e|test_historical_null_recursive_heal_restores_physical_versions|test_recursive_deep_heal_reports_stale_delete_marker_repair)$/)) | (binary(usage_observation_read_repair_test) & test(=usage_observation_read_repair_survives_snapshot_cleanup)))) | (package(rustfs-test-utils) & test(=data_usage_snapshot_tests::data_usage_snapshot_recovery_wires_primary_backup_and_negative_cache))'
setup = 'ecstore-base-stack'
[[profile.ci.scripts]]
filter = 'package(rustfs-ecstore) | package(rustfs-s3select-api) | package(rustfs-scanner) | (package(rustfs) & test(/^(app::multipart_usecase::complete::tests::completion_absent_binding_refusal_order_baseline|app::multipart_usecase::complete::tests::concurrent_completions_share_durable_bucket_quota_reservations|app::object::delete::tests::compressed_delete_requests_update_observed_usage_without_releasing_quota_floor|app::object::internal_put::tests::internal_multipart_roundtrip_completes_and_abort_leaves_nothing|app::object::restore::tests::execute_restore_object_maps_failures_to_typed_s3_errors|storage::access::tests::(delete_object_access_captures_authorized_bucket_incarnation|copy_operations_reject_recreated_source_bucket_after_authorization|request_slot_keeps_bucket_policy_bound_to_its_store))$/))'
filter = 'package(rustfs-ecstore) | package(rustfs-s3select-api) | package(rustfs-scanner) | (package(rustfs) & test(/^(app::multipart_usecase::tests::concurrent_completions_share_durable_bucket_quota_reservations|app::object::delete::tests::compressed_delete_requests_update_observed_usage_without_releasing_quota_floor|app::object::internal_put::tests::internal_multipart_roundtrip_completes_and_abort_leaves_nothing|app::object::restore::tests::execute_restore_object_maps_failures_to_typed_s3_errors|storage::access::tests::(delete_object_access_captures_authorized_bucket_incarnation|copy_operations_reject_recreated_source_bucket_after_authorization|request_slot_keeps_bucket_policy_bound_to_its_store))$/))'
setup = 'ecstore-base-stack'
[[profile.ci.scripts]]
@@ -477,16 +434,10 @@ setup = 'ecstore-base-stack'
filter = 'binary(lifecycle_integration_test) | (package(rustfs) & test(/^app::lifecycle_transition_api_test::/))'
setup = 'lifecycle-large-stack'
# This conditional GET fixture composes real-store futures that overflowed
# the default Linux test-thread stack.
# The pool metadata placement regression composes the full erasure-heal
# startup future and overflows the default Linux test-thread stack.
[[profile.ci.scripts]]
filter = 'package(rustfs) & test(=app::object::get::tests::conditional_get_304_uses_metadata_without_decoding_damaged_data)'
setup = 'ecstore-large-stack'
# These heal fixtures compose deep real-store recovery futures that can
# overflow the default Linux test-thread stack.
[[profile.ci.scripts]]
filter = 'package(rustfs-heal) & (test(=replacement_pool_metadata_follows_real_two_set_placement) | (binary(heal_integration_test) & test(=bucket_removal_recovers_delete_proofs_before_removing_empty_volumes)))'
filter = 'package(rustfs-heal) & test(=replacement_pool_metadata_follows_real_two_set_placement)'
setup = 'ecstore-large-stack'
# These real-disk scanner fixtures assert quiet-workload publication and
@@ -571,7 +522,7 @@ filter = 'package(rustfs) & (binary(/^embedded.*_test$/) | binary(admin_diagnost
test-group = 'embedded-test-ports'
[[profile.ci.overrides]]
filter = 'package(rustfs) & (binary(/^connect_registration$/) | binary(connect_inventory) | binary(connect_perf_drive) | (binary(connect_perf_client) & test(=configured_proxy_and_direct_endpoint_failures_are_distinct)) | (binary(connect_perf_object) & test(/^(real_rustfs_endpoint_and_production_cli_support_bounded_get_and_put|cleanup_uses_a_budget_independent_of_the_measurement_window|cleanup_accepts_missing_object_and_still_deletes_the_bucket|cleanup_diagnostics_identify_each_disconnected_transport_stage|cleanup_diagnostics_identify_each_failed_http_stage_without_private_context|cleanup_diagnostics_identify_timeouts_without_resetting_the_shared_deadline|oversized_response_is_rejected_without_buffering_it)$/)))'
filter = 'package(rustfs) & (binary(/^connect_registration$/) | binary(connect_inventory) | binary(connect_perf_drive) | (binary(connect_perf_client) & test(=configured_proxy_and_direct_endpoint_failures_are_distinct)) | (binary(connect_perf_object) & test(/^(real_rustfs_endpoint_and_production_cli_support_bounded_get_and_put|cleanup_uses_a_budget_independent_of_the_measurement_window|oversized_response_is_rejected_without_buffering_it)$/)))'
threads-required = "num-test-threads"
# Preserve the cancellation probe's startup budget and assertions in CI too.
@@ -721,23 +672,14 @@ test-group = 'ecstore-serial-flaky'
# the PR-lane e2e coverage for scanner usage snapshots consumed by quota and
# admin surfaces. It uses the same single-node, random-port, isolated-temp-dir
# fixture as the existing smoke modules.
#
# Dual-stack differential (rustfs/backlog#2740): `dual_stack_differential::`
# starts two single-node servers from the same binary, one per
# RUSTFS_S3_STACK value, on random ports with isolated temp dirs, and runs one
# SDK script covering every operation of `impl S3 for FS` against both. Its
# comparator unit tests need no server. It meets the admission criteria
# unchanged; the module clause admits a new differential test without another
# filter edit.
[profile.e2e-smoke]
default-filter = """
package(e2e_test) & (
test(/^(delete_marker_migration_semantics|version_id_regression|list_objects_v2_pagination|list_object_versions_regression|list_objects_duplicates|list_buckets_double_slash|list_buckets_auth|list_buckets_iam_filter|leading_slash_key|special_chars|create_bucket_region|delete_objects_versioning|head_object_consistency|head_object_range|copy_object_metadata|copy_object_tagging|copy_source_invalid_date|content_encoding|compression|multipart_storage_class|multipart_upload_id|storage_class_capability|ssec_copy|anonymous_access|bucket_policy_check|presigned_negative|negative_sigv4|admin_auth|notification_webhook|tls_hot_reload|console_smoke|admin_iam_crud|admin_pools|sts_query_compat|data_usage)_test::|^fake_s3_target::/)
test(/^(delete_marker_migration_semantics|version_id_regression|list_objects_v2_pagination|list_object_versions_regression|list_objects_duplicates|list_buckets_double_slash|list_buckets_auth|list_buckets_iam_filter|leading_slash_key|special_chars|create_bucket_region|delete_objects_versioning|head_object_consistency|head_object_range|copy_object_metadata|copy_object_tagging|copy_source_invalid_date|content_encoding|compression|multipart_storage_class|storage_class_capability|ssec_copy|anonymous_access|bucket_policy_check|presigned_negative|negative_sigv4|admin_auth|notification_webhook|tls_hot_reload|console_smoke|admin_iam_crud|admin_pools|sts_query_compat|data_usage)_test::|^fake_s3_target::/)
| test(/^replication_extension_test::(test_replication_check_succeeds_with_remote_target|test_replication_check_rejects_target_without_object_lock|test_set_remote_target_rejects_unversioned_source_bucket|test_replication_check_rejects_unversioned_source_bucket|test_replication_check_rejects_missing_replication_config|test_replication_check_rejects_invalid_bucket|test_set_remote_target_rejects_same_bucket_on_same_deployment|test_set_remote_target_rejects_unversioned_target_bucket|test_set_remote_target_update_requires_arn|test_set_remote_target_update_rejects_missing_target|test_set_remote_target_rejects_invalid_target_url|test_set_remote_target_rejects_self_signed_https_target_without_skip_tls_verify|test_set_remote_target_rejects_private_ca_https_target_without_ca_cert_pem|test_list_remote_targets_rejects_empty_bucket|test_list_remote_targets_rejects_invalid_bucket|test_remove_remote_target_rejects_missing_target|test_remove_remote_target_rejects_missing_arn|test_remove_remote_target_rejects_invalid_bucket|test_remove_remote_target_rejects_target_used_by_replication|test_delete_bucket_replication_removes_remote_target)$/)
| test(/^reliant::lifecycle::/)
| test(/^conditional_delete_objects_test::/)
| test(/^reliant::tiering::/)
| test(/^dual_stack_differential::/)
| test(/^upgrade_compatibility_test::upgrade_write_readiness_tests::/)
| test(/^on_demand_migration::(get_basic_test::(get_miss_pulls_inline_and_serves_locally_afterwards|head_miss_answers_from_the_source_without_persisting)|interaction_test::test_odm_admin_config_is_redacted_and_status_counts_match_the_source)$/)
)
@@ -859,7 +801,7 @@ slow-timeout = { period = "120s", terminate-after = 12, grace-period = "10s" }
# ---------------------------------------------------------------------------
# e2e-distributed profile — 4-node 4-disk Actions suite
# ---------------------------------------------------------------------------
# Nightly and manual-dispatch lane owned by
# Storage-sensitive PR / nightly / dispatch lane owned by
# .github/workflows/e2e-distributed.yml.
# Each case starts four rustfs processes (and for site replication, two
# clusters). Upgrade cases also require RUSTFS_UPGRADE_SOURCE_BINARY.
@@ -957,10 +899,6 @@ path = "junit.xml"
# parallel-safe — the same property e2e-smoke relies on. The exceptions are the
# 4-disk reliability / degraded-read fault-injection tests, multi-node heal
# clusters, and the fixed-port Vault tests, all serialized below.
# `dual_stack_differential::reference::` holds hermetic unit tests of how the
# differential picks side A (shell-script stand-ins, no server, no reference
# binary). They already run on every pull request in e2e-smoke, so e2e-full
# excludes them to keep its selection digest unchanged.
[profile.e2e-full]
default-filter = """
package(e2e_test)
@@ -971,7 +909,6 @@ default-filter = """
& !test(/^replication_target_matrix_test::/)
& !test(/^on_demand_migration::(concurrency_test|fault_test|interop_test|real_source_test)::/)
& !test(/^upgrade_compatibility_test::upgrade_write_readiness_tests::/)
& !test(/^dual_stack_differential::reference::/)
"""
fail-fast = false
@@ -997,12 +934,6 @@ test-group = 'e2e-inline-boundaries'
filter = 'package(e2e_test) & test(/^heal_erasure_disk_rebuild_test::/)'
test-group = 'e2e-cluster-nightly'
# The multi-pool crash fixture seeds three nodes and 36 drives. Reserve this
# nextest run's workers so other multi-node tests do not starve its deadline.
[[profile.e2e-full.overrides]]
filter = 'package(e2e_test) & test(=heal_erasure_disk_rebuild_test::tests::test_cluster_root_heal_recovers_ec84_shards_across_multi_pool_after_background_target_crash)'
threads-required = "num-test-threads"
[[profile.e2e-full.overrides]]
filter = 'package(e2e_test) & (test(/^kms::kms_vault_test::/) | test(/^kms::kms_rekey_sweep_test::/) | test(/^kms::configured_roundtrip_test::test_configured_vault_kms_admin_and_versioned_cleanup$/))'
test-group = 'e2e-vault'
@@ -1012,23 +943,3 @@ test-group = 'e2e-vault'
[[profile.e2e-full.overrides]]
filter = 'package(e2e_test) & (test(/^list_objects_v2_pagination_test::tests::(test_list_objects_v2_delimiter_small_page_traverses_all|test_list_objects_v2_max_keys_above_limit_returns_token|test_list_objects_v2_maxkeys_above_limit_with_delimiter)$/) | test(=reliant::tiering::test_hermetic_transition_restore_failure_expiry_and_retry))'
threads-required = "num-test-threads"
# The mc fixture uploads hundreds of objects concurrently before measuring listing.
# Reserve the runner so other server fixtures do not starve its disk operations.
[[profile.e2e-full.overrides]]
filter = 'package(e2e_test) & test(=mc_mirror_small_bucket_test::test_mc_mirror_small_bucket_completes_without_list_timeout)'
threads-required = "num-test-threads"
# Fixed, small contract for every executable server change. These existing
# scenarios exercise write/read, anonymous denial and retained bytes on restart.
[profile.core-contract]
default-filter = '''
package(e2e_test) & test(/^(anonymous_access_test::test_anonymous_access_denied_when_restrict_public_buckets_enabled|head_object_consistency_test::head_object_consistency_after_write_and_multipart_and_presigned_head|storage_class_capability_test::tests::historical_label_only_metadata_is_standard_without_hiding_a_real_transition_tier)$/)
'''
retries = 0
fail-fast = false
test-threads = 1
slow-timeout = { period = "60s", terminate-after = 3 }
[profile.core-contract.junit]
path = "junit.xml"
-20
View File
@@ -1,20 +0,0 @@
# Pinned RustFS release used as the "reference" target of parity lanes
# (.github/workflows/reference-parity.yml). This is the single place that names
# the reference release; .github/actions/reference-binary resolves, downloads and
# verifies it, and fails on a missing pin, a missing asset or a digest mismatch.
#
# Bumping the reference is a reviewed event: every per-test difference between
# the candidate and the reference is measured against this exact build.
#
# Each sha256 is the value published in the release's SHA256SUMS asset, which
# matches the digest GitHub reports for the same asset. The action checks both.
schema = 1
repository = "rustfs/rustfs"
version = "1.0.5"
tag = "1.0.5"
[platforms.linux-x86_64-gnu]
asset = "rustfs-linux-x86_64-gnu-v1.0.5.zip"
sha256 = "5addae2f74756bf80104d0975e6347cda843bc2daffdfcd1f0cb80118b95cbd0"
executable = "rustfs"
-24
View File
@@ -1,24 +0,0 @@
# Targets of the s3-tests reference parity lane (.github/workflows/reference-parity.yml).
#
# Exactly one target uses `binary = "reference"` (the release pinned in
# .config/reference-binary.toml). Every other target uses `binary = "candidate"`
# (the binary built from the branch under test) and is compared per node id
# against the reference. All targets run the same harness, test list and server
# configuration; `env` adds only the target's own RUSTFS_* settings.
#
# Adding a target is one more [[targets]] table, for example:
#
# [[targets]]
# name = "candidate-gateway"
# binary = "candidate"
# env = { RUSTFS_S3_STACK = "gateway" }
schema = 1
[[targets]]
name = "reference"
binary = "reference"
[[targets]]
name = "candidate-legacy"
binary = "candidate"
-12
View File
@@ -1,12 +0,0 @@
# Tolerated per-test differences of the s3-tests reference parity lane.
#
# One entry per line, four fields separated by " | ":
#
# <target> | <pytest node id> | <tracking issue URL> | <reason>
#
# <target> is a candidate name from .config/reference-parity-targets.toml, the
# node id is exactly as the lane prints it (s3tests/functional/test_s3.py::...),
# and the issue URL is https://github.com/rustfs/<repo>/issues/<number>.
#
# Same semantics as scripts/junit_diff.py --allow: an entry whose difference no
# longer occurs is stale and fails the lane, so remove it when it is fixed.
-135
View File
@@ -1,135 +0,0 @@
{
"schema_version": 1,
"plan_id": "rustfs-release-validation",
"description": "Certification for immutable candidates. All shipped platforms require native identity and startup; stable and prerelease require the pinned upgrade matrix. Linux-only acceptance never authorizes publication.",
"release_types": [
"preview",
"prerelease",
"stable"
],
"targets": [
{
"os": "linux",
"arch": "x86_64",
"abi": "musl"
},
{
"os": "linux",
"arch": "aarch64",
"abi": "musl"
},
{
"os": "linux",
"arch": "x86_64",
"abi": "gnu"
},
{
"os": "linux",
"arch": "aarch64",
"abi": "gnu"
},
{
"os": "macos",
"arch": "aarch64",
"abi": "none"
},
{
"os": "windows",
"arch": "x86_64",
"abi": "msvc"
}
],
"targets_basis": "Platform matrix of .github/workflows/build.yml (prepare-platform-matrix). gnu and macOS targets build with the pyroscope feature; musl and Windows targets build with no extra features.",
"features": [
"pyroscope"
],
"trusted_workflows": [
{
"repository": "rustfs/rustfs",
"workflow_id": "release-candidate.yml"
},
{
"repository": "rustfs/rustfs",
"workflow_id": "release-complete.yml"
}
],
"trusted_workflows_basis": "Receipts are produced by the reviewed main workflows and fetched by exact run ID with GitHub artifact digest verification. Candidate inputs freeze the harness commit separately from product source.",
"lanes": [
{
"lane": "identity",
"harness_repository": "rustfs/rustfs",
"harness_sha": "candidate.workflow_sha",
"harness_basis": "The exact workflow commit is frozen in candidate inputs; no mutable branch is accepted as a harness identity.",
"timeout_seconds": 1800,
"environment_capabilities": [
"process-exec"
],
"cases": [
"artifact-identity",
"startup"
],
"applies_to": {
"release_types": [
"preview",
"prerelease",
"stable"
],
"platforms": "all"
},
"applies_basis": "Artifact identity and startup checks are mandatory for every release type, preview included, on every shipped target (rustfs/backlog#2662 frozen design item 5)."
},
{
"lane": "upgrade",
"harness_repository": "rustfs/rustfs",
"harness_sha": "candidate.workflow_sha",
"harness_basis": "The exact workflow commit is frozen in candidate inputs; no mutable branch is accepted as a harness identity.",
"timeout_seconds": 5400,
"environment_capabilities": [
"process-exec",
"previous-release-artifact"
],
"cases": [
"direct-upgrade",
"mixed-version-upgrade",
"bucket-config-upgrade",
"bucket-config-rollback",
"multipart-layout-upgrade",
"multipart-layout-baseline"
],
"cases_basis": "Case IDs of the pinned previous-release matrix in .github/workflows/e2e-upgrade.yml (source.version/asset/sha256 and the case list mapped there).",
"applies_to": {
"release_types": [
"prerelease",
"stable"
],
"platforms": [
{
"os": "linux",
"arch": "x86_64",
"abi": "gnu"
}
]
},
"applies_basis": "preview releases use the pre-declared smaller set (identity lane only). The upgrade lane is only exercised on linux-x86_64-gnu, the platform of the pinned source assets."
}
],
"upgrade_sources": {
"basis": "Exactly the two previous-release sources pinned in .github/workflows/e2e-upgrade.yml are exercised today (source.version / source.asset / source.sha256). No other N-1/N-2 compatibility is claimed, and these digests are not inferred from tags.",
"sources": [
{
"version": "1.0.1",
"role": "N-1",
"artifact_name": "rustfs-linux-x86_64-gnu-v1.0.1.zip",
"artifact_sha256": "9742e183cb0dfb0d48fa909797af572fe47b7f827569036baa4156e0d3958ee5"
},
{
"version": "1.0.0-rc.5",
"role": "N-2",
"artifact_name": "rustfs-linux-x86_64-gnu-v1.0.0-rc.5.zip",
"artifact_sha256": "3ee8df71e8edcfada533be452c4135868f697bc515460ae97b027313eade7a3d"
}
],
"rollback_supported": false,
"rollback_basis": "Only the bucket-config-rollback case (rollback_to_previous_release_reads_current_bucket_metadata) is exercised. Full data-plane rollback is not certified and is never a default claim."
}
}
-37
View File
@@ -1,37 +0,0 @@
# Files that may still reference s3s (`s3s::` or `s3s_sigv4`) when Phase 1 of
# the gateway migration ends (rustfs/backlog#2734 task T0.6, rustfs/backlog#2741).
# Consumed only by `scripts/check_s3s_footprint.sh --mode allowlist`.
#
# One entry per line: a repository-relative file path or a git `:(glob)`
# pathspec. `**` matches across directories, `*` stays within one path
# component, and a directory entry matches everything under it. Blank lines
# and lines starting with `#` are ignored; an entry may not contain whitespace,
# so a note belongs on its own comment line.
#
# Every Rust file listed by `git grep -l '\bs3s\(::\|_sigv4\)' -- '*.rs'` that
# no entry matches fails the allowlist mode. Task T1.9 switches CI to that
# mode; until then `--dry-run` prints the files still outside this list, which
# is the Phase 1 tracking list.
#
# Never add crates/** other than crates/e2e_test: the engine crates leave s3s
# entirely, and the script rejects such an entry. Only the s3s-facing edge of
# the rustfs binary and the e2e oracle keep it.
# rustfs binary: the s3s-facing edge that survives Phase 1
rustfs/src/storage/ecfs.rs
rustfs/src/storage/s3_compat/**
rustfs/src/storage/access.rs
rustfs/src/auth.rs
rustfs/src/error.rs
rustfs/src/storage_api.rs
rustfs/src/app/storage_api.rs
rustfs/src/admin/router.rs
rustfs/src/app/metadata_route.rs
rustfs/src/server/http.rs
rustfs/src/server/layer.rs
rustfs/src/server/hybrid.rs
rustfs/src/protocols/client.rs
# e2e oracle: the test target that verifies S3 behaviour against s3s
crates/e2e_test/src/fake_s3_target/**
crates/e2e_test/src/**
-18
View File
@@ -1,18 +0,0 @@
# Baseline for the direct s3s footprint, consumed only by
# scripts/check_s3s_footprint.sh (rustfs/backlog#1677; rustfs/backlog#2734 T0.4;
# comparison rules on rustfs/backlog#2765).
#
# Each line is a ceiling everywhere: a counter above it fails, because new code
# widened the s3s surface; route it through the gateway abstractions instead and
# never raise a value here. Pull requests and merge groups are additionally held
# to their base revision's measurement, so a slice that shrinks the surface need
# not edit this file. A counter below its line passes and the check prints the
# lowered line; the migration coordinator lowers the lines in batches.
#
# files Rust files referencing s3s paths, repo-wide minus crates/e2e_test
# s3_error_lines lines holding s3_error!, same scope, minus those the script
# header's rule clears (rustfs_s3_types's macro); fails closed
# ecstore_files Rust files referencing s3s paths under crates/ecstore/src
files=63
s3_error_lines=209
ecstore_files=0
-67
View File
@@ -1,67 +0,0 @@
# Per-crate test-count baseline consumed by scripts/check_test_count_ratchet.sh
# (rustfs/backlog#2734 section 1 rule 3; T0.3 in rustfs/backlog#2738).
#
# One row per workspace member: <package>, tests, ignored, tab-separated.
# tests counts #[test], #[tokio::test], #[tokio::test(...)] and #[test_case(...)]
# attribute lines under the package directory; ignored counts #[ignore] and
# #[ignore = "..."]. The tests column is a floor and the ignored column is a
# ceiling: CI fails when a crate's tests fall below its row or its ignored count
# rises above it. Extra tests and fewer ignores pass; --update tightens the rows.
# Lowering a tests row or raising an ignored row is a reviewed statement that
# tests were intentionally removed or ignored; justify it in the PR description.
# Regenerate with:
# scripts/check_test_count_ratchet.sh --update
e2e_test 1026 20
rustfs 6449 64
rustfs-audit 70 0
rustfs-checksums 28 1
rustfs-common 36 0
rustfs-concurrency 87 0
rustfs-config 43 0
rustfs-connect-symbol-catalog 6 0
rustfs-credentials 24 0
rustfs-crypto 67 0
rustfs-data-usage 58 0
rustfs-ecstore 7858 3
rustfs-extension-schema 23 0
rustfs-filemeta 361 0
rustfs-gateway-adapter 74 0
rustfs-heal 1128 3
rustfs-heal-contracts 8 0
rustfs-iam 415 0
rustfs-io-core 52 0
rustfs-io-metrics 157 0
rustfs-keystone 31 0
rustfs-kms 716 25
rustfs-license 2 0
rustfs-lifecycle 156 0
rustfs-lock 182 0
rustfs-log-analyzer 91 0
rustfs-madmin 210 0
rustfs-notify 140 0
rustfs-object-capacity 92 0
rustfs-object-data-cache 141 0
rustfs-obs 387 0
rustfs-policy 758 0
rustfs-protocols 770 1
rustfs-protos 75 0
rustfs-replication 209 0
rustfs-rio 225 1
rustfs-rio-v2 34 6
rustfs-s3-client 71 0
rustfs-s3-ops 8 0
rustfs-s3-types 105 0
rustfs-s3select-api 232 0
rustfs-s3select-query 235 2
rustfs-scanner 1392 20
rustfs-scanner-metrics 90 0
rustfs-security-governance 20 0
rustfs-signer 34 0
rustfs-storage-api 34 0
rustfs-targets 612 26
rustfs-test-utils 1 0
rustfs-tls-runtime 44 0
rustfs-trusted-proxies 72 0
rustfs-utils 253 1
rustfs-zip 30 0
zstd 11 0
@@ -1,3 +0,0 @@
# Tests the bucket_access slice retires (rustfs/backlog#2765); format and checks: scripts/check_test_retirements.py.
# <crate> | <module path>::<test fn> | <kind: test, differential or e2e> | <replacement> | <reason>
# The retired test must still exist when its entry is added; the change that deletes it deletes the entry.
@@ -1,3 +0,0 @@
# Tests the bucket_configs slice retires (rustfs/backlog#2765); format and checks: scripts/check_test_retirements.py.
# <crate> | <module path>::<test fn> | <kind: test, differential or e2e> | <replacement> | <reason>
# The retired test must still exist when its entry is added; the change that deletes it deletes the entry.
-3
View File
@@ -1,3 +0,0 @@
# Tests the bucket_core slice retires (rustfs/backlog#2765); format and checks: scripts/check_test_retirements.py.
# <crate> | <module path>::<test fn> | <kind: test, differential or e2e> | <replacement> | <reason>
# The retired test must still exist when its entry is added; the change that deletes it deletes the entry.
@@ -1,3 +0,0 @@
# Tests the bucket_replication_lifecycle slice retires (rustfs/backlog#2765); format and checks: scripts/check_test_retirements.py.
# <crate> | <module path>::<test fn> | <kind: test, differential or e2e> | <replacement> | <reason>
# The retired test must still exist when its entry is added; the change that deletes it deletes the entry.
-3
View File
@@ -1,3 +0,0 @@
# Tests the cutover slice retires (rustfs/backlog#2765); format and checks: scripts/check_test_retirements.py.
# <crate> | <module path>::<test fn> | <kind: test, differential or e2e> | <replacement> | <reason>
# The retired test must still exist when its entry is added; the change that deletes it deletes the entry.
@@ -1,3 +0,0 @@
# Tests the listing_multipart slice retires (rustfs/backlog#2765); format and checks: scripts/check_test_retirements.py.
# <crate> | <module path>::<test fn> | <kind: test, differential or e2e> | <replacement> | <reason>
# The retired test must still exist when its entry is added; the change that deletes it deletes the entry.
@@ -1,3 +0,0 @@
# Tests the object_commit slice retires (rustfs/backlog#2765); format and checks: scripts/check_test_retirements.py.
# <crate> | <module path>::<test fn> | <kind: test, differential or e2e> | <replacement> | <reason>
# The retired test must still exist when its entry is added; the change that deletes it deletes the entry.
-3
View File
@@ -1,3 +0,0 @@
# Tests the object_meta slice retires (rustfs/backlog#2765); format and checks: scripts/check_test_retirements.py.
# <crate> | <module path>::<test fn> | <kind: test, differential or e2e> | <replacement> | <reason>
# The retired test must still exist when its entry is added; the change that deletes it deletes the entry.
-3
View File
@@ -1,3 +0,0 @@
# Tests the object_misc slice retires (rustfs/backlog#2765); format and checks: scripts/check_test_retirements.py.
# <crate> | <module path>::<test fn> | <kind: test, differential or e2e> | <replacement> | <reason>
# The retired test must still exist when its entry is added; the change that deletes it deletes the entry.
-3
View File
@@ -1,3 +0,0 @@
# Tests the object_read slice retires (rustfs/backlog#2765); format and checks: scripts/check_test_retirements.py.
# <crate> | <module path>::<test fn> | <kind: test, differential or e2e> | <replacement> | <reason>
# The retired test must still exist when its entry is added; the change that deletes it deletes the entry.
@@ -1,3 +0,0 @@
# Tests the object_write slice retires (rustfs/backlog#2765); format and checks: scripts/check_test_retirements.py.
# <crate> | <module path>::<test fn> | <kind: test, differential or e2e> | <replacement> | <reason>
# The retired test must still exist when its entry is added; the change that deletes it deletes the entry.
-3
View File
@@ -1,3 +0,0 @@
# Tests the t211 slice retires (rustfs/backlog#2765); format and checks: scripts/check_test_retirements.py.
# <crate> | <module path>::<test fn> | <kind: test, differential or e2e> | <replacement> | <reason>
# The retired test must still exist when its entry is added; the change that deletes it deletes the entry.
-3
View File
@@ -1,3 +0,0 @@
# Tests the t212 slice retires (rustfs/backlog#2765); format and checks: scripts/check_test_retirements.py.
# <crate> | <module path>::<test fn> | <kind: test, differential or e2e> | <replacement> | <reason>
# The retired test must still exist when its entry is added; the change that deletes it deletes the entry.
+3 -39
View File
@@ -22,9 +22,9 @@ The following files in the project root are essential for Docker operations:
| File | Description | Usage |
| :--- | :--- | :--- |
| **`docker-buildx.sh`** | **Multi-Arch Build Script**<br>Automates building and pushing Docker images for `amd64` and `arm64`. Supports release and dev channels. | `./docker-buildx.sh --push` |
| **`Dockerfile`** | **Production Image (Alpine)**<br>Lightweight image using musl libc. Downloads verified RustFS and `rc` binaries from GitHub Releases and bundles `/etc/rustfs/bootstrap`. | `docker build -t rustfs:latest .` |
| **`Dockerfile.glibc`** | **Production Image (Ubuntu)**<br>Standard image using glibc. Includes the bundled bootstrap configuration. | `docker build -f Dockerfile.glibc .` |
| **`Dockerfile.source`** | **Development Image**<br>Builds RustFS from source code and bundles `rc` and the default bootstrap configuration. | `docker build -f Dockerfile.source .` |
| **`Dockerfile`** | **Production Image (Alpine)**<br>Lightweight image using musl libc. Downloads pre-built binaries from GitHub Releases. | `docker build -t rustfs:latest .` |
| **`Dockerfile.glibc`** | **Production Image (Ubuntu)**<br>Standard image using glibc. Useful if you need specific dynamic libraries. | `docker build -f Dockerfile.glibc .` |
| **`Dockerfile.source`** | **Development Image**<br>Builds RustFS from source code. Includes build tools. Ideal for local development and CI. | `docker build -f Dockerfile.source .` |
### Docker Compose Configurations
@@ -110,42 +110,6 @@ docker compose up -d
---
## Docker Compose environment values
Use map syntax when quoting environment values in YAML:
```yaml
services:
rustfs:
image: rustfs/rustfs:latest
ports:
- "9000:9000"
- "9001:9001"
volumes:
- ./data:/data
- ./logs:/var/log/rustfs
environment:
RUSTFS_ADDRESS: ":9000"
RUSTFS_CONSOLE_ADDRESS: ":9001"
RUSTFS_CONSOLE_ENABLE: "true"
RUSTFS_OBS_LOGGER_LEVEL: error
RUSTFS_OBS_LOG_DIRECTORY: "/var/log/rustfs/"
RUSTFS_ACCESS_KEY: "${RUSTFS_ACCESS_KEY:?Set RUSTFS_ACCESS_KEY}"
RUSTFS_SECRET_KEY: "${RUSTFS_SECRET_KEY:?Set RUSTFS_SECRET_KEY}"
```
Create the host directories and grant the container user write access before starting this example; official images run as UID/GID `10001:10001`. Set both credential variables to your own values, then run `docker compose config` to inspect the resolved configuration locally before `docker compose up -d`. The resolved output includes credentials, so do not share it publicly.
With list syntax, use `- RUSTFS_OBS_LOG_DIRECTORY=/var/log/rustfs/` or quote the whole item as `- "RUSTFS_OBS_LOG_DIRECTORY=/var/log/rustfs/"`. Writing `- RUSTFS_OBS_LOG_DIRECTORY="/var/log/rustfs/"` passes the inner quotes literally, making this a relative path beginning with `"`, which can fail with `mkdir: cannot create directory '"': Permission denied`. This also applies to addresses and credentials: fix every affected list item, rather than only the log directory. The entrypoint preserves configured values, including literal quotes in valid paths and secrets.
---
## Local storage pool expansion
To append a second four-drive pool, set `RUSTFS_VOLUMES: "/data/rustfs0/{0...3} /data/rustfs1/{0...3}"` and keep the original pool's drives, order, and erasure set width. The container prepares each drive directory and passes one argument per pool to RustFS.
If an older image already formatted both ranges together as a single eight-drive pool, list the eight individual paths in their original order to keep that layout when upgrading.
## 🔧 Common Operations
### Cleaning Up
@@ -34,10 +34,6 @@ services:
- RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT:-5}
- RUSTFS_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_LOCK_ACQUIRE_TIMEOUT:-5}
- RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
- RUSTFS_DURABILITY_MODE=${RUSTFS_DURABILITY_MODE:-strict}
- RUSTFS_OBJECT_DIRECT_IO_WRITE_ENABLE=${RUSTFS_OBJECT_DIRECT_IO_WRITE_ENABLE:-false}
- RUSTFS_EXPERIMENTAL_FILE_FDATASYNC_GROUP_COMMIT_ENABLE=${RUSTFS_EXPERIMENTAL_FILE_FDATASYNC_GROUP_COMMIT_ENABLE:-false}
- RUSTFS_EXPERIMENTAL_FILE_FDATASYNC_GROUP_COMMIT_WAIT_MICROS=${RUSTFS_EXPERIMENTAL_FILE_FDATASYNC_GROUP_COMMIT_WAIT_MICROS:-0}
# Internode gRPC optimization knobs (grpc-optimization P0-P3). Defaults match the binary
# defaults, so leaving these unset is a no-op; the A/B driver exports them to toggle a stage.
- RUSTFS_INTERNODE_RPC_TCP_NODELAY=${RUSTFS_INTERNODE_RPC_TCP_NODELAY:-true}
@@ -47,8 +43,6 @@ services:
- RUSTFS_INTERNODE_CHANNEL_ISOLATION=${RUSTFS_INTERNODE_CHANNEL_ISOLATION:-false}
- RUSTFS_INTERNODE_BULK_CHANNELS=${RUSTFS_INTERNODE_BULK_CHANNELS:-2}
- RUSTFS_INTERNODE_RPC_MSGPACK_ONLY=${RUSTFS_INTERNODE_RPC_MSGPACK_ONLY:-false}
- RUSTFS_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED=${RUSTFS_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED:-false}
- RUSTFS_OBS_PUT_STAGE_METRICS_ENABLED=${RUSTFS_OBS_PUT_STAGE_METRICS_ENABLED:-false}
- RUSTFS_INTERNODE_PREWARM=${RUSTFS_INTERNODE_PREWARM:-false}
- RUSTFS_INTERNODE_OFFLINE_BYPASS=${RUSTFS_INTERNODE_OFFLINE_BYPASS:-false}
- RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS=${RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS:-5}
@@ -62,7 +56,7 @@ services:
- node1_data_2:/data/rustfs2
- node1_data_3:/data/rustfs3
ports:
- "${RUSTFS_NODE1_PORT:-9000}:9000"
- "9000:9000"
networks:
- rustfs-cluster-net
@@ -87,10 +81,6 @@ services:
- RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT:-5}
- RUSTFS_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_LOCK_ACQUIRE_TIMEOUT:-5}
- RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
- RUSTFS_DURABILITY_MODE=${RUSTFS_DURABILITY_MODE:-strict}
- RUSTFS_OBJECT_DIRECT_IO_WRITE_ENABLE=${RUSTFS_OBJECT_DIRECT_IO_WRITE_ENABLE:-false}
- RUSTFS_EXPERIMENTAL_FILE_FDATASYNC_GROUP_COMMIT_ENABLE=${RUSTFS_EXPERIMENTAL_FILE_FDATASYNC_GROUP_COMMIT_ENABLE:-false}
- RUSTFS_EXPERIMENTAL_FILE_FDATASYNC_GROUP_COMMIT_WAIT_MICROS=${RUSTFS_EXPERIMENTAL_FILE_FDATASYNC_GROUP_COMMIT_WAIT_MICROS:-0}
# Internode gRPC optimization knobs (grpc-optimization P0-P3). Defaults match the binary
# defaults, so leaving these unset is a no-op; the A/B driver exports them to toggle a stage.
- RUSTFS_INTERNODE_RPC_TCP_NODELAY=${RUSTFS_INTERNODE_RPC_TCP_NODELAY:-true}
@@ -100,8 +90,6 @@ services:
- RUSTFS_INTERNODE_CHANNEL_ISOLATION=${RUSTFS_INTERNODE_CHANNEL_ISOLATION:-false}
- RUSTFS_INTERNODE_BULK_CHANNELS=${RUSTFS_INTERNODE_BULK_CHANNELS:-2}
- RUSTFS_INTERNODE_RPC_MSGPACK_ONLY=${RUSTFS_INTERNODE_RPC_MSGPACK_ONLY:-false}
- RUSTFS_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED=${RUSTFS_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED:-false}
- RUSTFS_OBS_PUT_STAGE_METRICS_ENABLED=${RUSTFS_OBS_PUT_STAGE_METRICS_ENABLED:-false}
- RUSTFS_INTERNODE_PREWARM=${RUSTFS_INTERNODE_PREWARM:-false}
- RUSTFS_INTERNODE_OFFLINE_BYPASS=${RUSTFS_INTERNODE_OFFLINE_BYPASS:-false}
- RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS=${RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS:-5}
@@ -115,7 +103,7 @@ services:
- node2_data_2:/data/rustfs2
- node2_data_3:/data/rustfs3
ports:
- "${RUSTFS_NODE2_PORT:-9001}:9000"
- "9001:9000"
networks:
- rustfs-cluster-net
@@ -140,10 +128,6 @@ services:
- RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT:-5}
- RUSTFS_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_LOCK_ACQUIRE_TIMEOUT:-5}
- RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
- RUSTFS_DURABILITY_MODE=${RUSTFS_DURABILITY_MODE:-strict}
- RUSTFS_OBJECT_DIRECT_IO_WRITE_ENABLE=${RUSTFS_OBJECT_DIRECT_IO_WRITE_ENABLE:-false}
- RUSTFS_EXPERIMENTAL_FILE_FDATASYNC_GROUP_COMMIT_ENABLE=${RUSTFS_EXPERIMENTAL_FILE_FDATASYNC_GROUP_COMMIT_ENABLE:-false}
- RUSTFS_EXPERIMENTAL_FILE_FDATASYNC_GROUP_COMMIT_WAIT_MICROS=${RUSTFS_EXPERIMENTAL_FILE_FDATASYNC_GROUP_COMMIT_WAIT_MICROS:-0}
# Internode gRPC optimization knobs (grpc-optimization P0-P3). Defaults match the binary
# defaults, so leaving these unset is a no-op; the A/B driver exports them to toggle a stage.
- RUSTFS_INTERNODE_RPC_TCP_NODELAY=${RUSTFS_INTERNODE_RPC_TCP_NODELAY:-true}
@@ -153,8 +137,6 @@ services:
- RUSTFS_INTERNODE_CHANNEL_ISOLATION=${RUSTFS_INTERNODE_CHANNEL_ISOLATION:-false}
- RUSTFS_INTERNODE_BULK_CHANNELS=${RUSTFS_INTERNODE_BULK_CHANNELS:-2}
- RUSTFS_INTERNODE_RPC_MSGPACK_ONLY=${RUSTFS_INTERNODE_RPC_MSGPACK_ONLY:-false}
- RUSTFS_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED=${RUSTFS_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED:-false}
- RUSTFS_OBS_PUT_STAGE_METRICS_ENABLED=${RUSTFS_OBS_PUT_STAGE_METRICS_ENABLED:-false}
- RUSTFS_INTERNODE_PREWARM=${RUSTFS_INTERNODE_PREWARM:-false}
- RUSTFS_INTERNODE_OFFLINE_BYPASS=${RUSTFS_INTERNODE_OFFLINE_BYPASS:-false}
- RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS=${RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS:-5}
@@ -168,7 +150,7 @@ services:
- node3_data_2:/data/rustfs2
- node3_data_3:/data/rustfs3
ports:
- "${RUSTFS_NODE3_PORT:-9002}:9000"
- "9002:9000"
networks:
- rustfs-cluster-net
@@ -193,10 +175,6 @@ services:
- RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT:-5}
- RUSTFS_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_LOCK_ACQUIRE_TIMEOUT:-5}
- RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
- RUSTFS_DURABILITY_MODE=${RUSTFS_DURABILITY_MODE:-strict}
- RUSTFS_OBJECT_DIRECT_IO_WRITE_ENABLE=${RUSTFS_OBJECT_DIRECT_IO_WRITE_ENABLE:-false}
- RUSTFS_EXPERIMENTAL_FILE_FDATASYNC_GROUP_COMMIT_ENABLE=${RUSTFS_EXPERIMENTAL_FILE_FDATASYNC_GROUP_COMMIT_ENABLE:-false}
- RUSTFS_EXPERIMENTAL_FILE_FDATASYNC_GROUP_COMMIT_WAIT_MICROS=${RUSTFS_EXPERIMENTAL_FILE_FDATASYNC_GROUP_COMMIT_WAIT_MICROS:-0}
# Internode gRPC optimization knobs (grpc-optimization P0-P3). Defaults match the binary
# defaults, so leaving these unset is a no-op; the A/B driver exports them to toggle a stage.
- RUSTFS_INTERNODE_RPC_TCP_NODELAY=${RUSTFS_INTERNODE_RPC_TCP_NODELAY:-true}
@@ -206,8 +184,6 @@ services:
- RUSTFS_INTERNODE_CHANNEL_ISOLATION=${RUSTFS_INTERNODE_CHANNEL_ISOLATION:-false}
- RUSTFS_INTERNODE_BULK_CHANNELS=${RUSTFS_INTERNODE_BULK_CHANNELS:-2}
- RUSTFS_INTERNODE_RPC_MSGPACK_ONLY=${RUSTFS_INTERNODE_RPC_MSGPACK_ONLY:-false}
- RUSTFS_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED=${RUSTFS_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED:-false}
- RUSTFS_OBS_PUT_STAGE_METRICS_ENABLED=${RUSTFS_OBS_PUT_STAGE_METRICS_ENABLED:-false}
- RUSTFS_INTERNODE_PREWARM=${RUSTFS_INTERNODE_PREWARM:-false}
- RUSTFS_INTERNODE_OFFLINE_BYPASS=${RUSTFS_INTERNODE_OFFLINE_BYPASS:-false}
- RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS=${RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS:-5}
@@ -221,7 +197,7 @@ services:
- node4_data_2:/data/rustfs2
- node4_data_3:/data/rustfs3
ports:
- "${RUSTFS_NODE4_PORT:-9003}:9000"
- "9003:9000"
networks:
- rustfs-cluster-net
-58
View File
@@ -1,58 +0,0 @@
# syntax=docker/dockerfile:1.6
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Server-only Linux validation image. The build context supplies a Git bundle.
FROM rust:1.100.0-trixie AS build-tools
RUN apt-get update && apt-get install -y --no-install-recommends \
ca-certificates clang lld libssl-dev pkg-config protobuf-compiler python3 python3-boto3 \
&& rm -rf /var/lib/apt/lists/*
ENV RUSTUP_TOOLCHAIN=1.100.0 \
CARGO_BUILD_JOBS=4 \
CARGO_NET_GIT_FETCH_WITH_CLI=true \
CARGO_TARGET_DIR=/work/rustfs/target \
RUSTFLAGS="-C link-arg=-fuse-ld=lld"
FROM build-tools AS build
ARG SOURCE_REVISION
COPY repository.bundle /tmp/repository.bundle
COPY inspect_geometry.rs /tmp/inspect_geometry.rs
RUN git clone /tmp/repository.bundle /work/rustfs \
&& git -C /work/rustfs checkout --detach "$SOURCE_REVISION" \
&& test "$(git -C /work/rustfs rev-parse HEAD)" = "$SOURCE_REVISION"
WORKDIR /work/rustfs
RUN --mount=type=cache,id=rustfs-inline-registry,target=/usr/local/cargo/registry \
--mount=type=cache,id=rustfs-inline-git,target=/usr/local/cargo/git \
--mount=type=cache,id=rustfs-inline-linux-target,target=/work/rustfs/target \
cargo build --locked --release --bin rustfs \
&& install -Dm755 target/release/rustfs /out/rustfs
# Build the diagnostic tool after preserving the clean-source server binary.
RUN cp /tmp/inspect_geometry.rs crates/filemeta/examples/inline_lab_inspect.rs
RUN --mount=type=cache,id=rustfs-inline-registry,target=/usr/local/cargo/registry \
--mount=type=cache,id=rustfs-inline-git,target=/usr/local/cargo/git \
--mount=type=cache,id=rustfs-inline-linux-target,target=/work/rustfs/target \
cargo build --locked --release -p rustfs-filemeta --example inline_lab_inspect \
&& install -Dm755 target/release/examples/inline_lab_inspect /out/inline_lab_inspect
FROM debian:trixie-slim AS runtime
RUN apt-get update && apt-get install -y --no-install-recommends \
ca-certificates curl libssl3t64 \
&& rm -rf /var/lib/apt/lists/*
ARG SOURCE_REVISION
LABEL org.opencontainers.image.revision="$SOURCE_REVISION" \
rustfs.validation="inline"
COPY --from=build /out/rustfs /usr/local/bin/rustfs
COPY --from=build /out/inline_lab_inspect /usr/local/bin/inline_lab_inspect
ENTRYPOINT ["/usr/local/bin/rustfs"]
-5
View File
@@ -1,5 +0,0 @@
FROM debian:trixie-slim
COPY mc /usr/bin/mc
RUN chmod 0755 /usr/bin/mc && /usr/bin/mc --version
LABEL rustfs.validation="inline"
ENTRYPOINT ["/usr/bin/mc"]
-133
View File
@@ -1,133 +0,0 @@
# Inline geometry Linux lab
**Use this when** validating inline placement, block geometry, Range reads and reader rollback in a task-owned local Docker cluster.
The adapter reuses the repository's four-node compose topology with 16 logical Docker volumes and EC12+4. It binds one S3 port to loopback, keeps strict durability and checks running binary hashes and persisted object geometry. Docker volumes share the host storage substrate. Performance observations describe this local virtualized lab.
## Build a frozen server image
Use the current stable toolchain pinned in the lab Dockerfile. Supply a Git bundle and the metadata inspector source as the build context. The server is compiled from a clean checkout before the diagnostic example is added.
```bash
set -euo pipefail
LAB_BUILD_DIR="$(mktemp -d)"
LAB_REVISION="$(git rev-parse HEAD)"
git bundle create "$LAB_BUILD_DIR/repository.bundle" HEAD
cp crates/filemeta/examples/inline_lab_inspect.rs "$LAB_BUILD_DIR/inspect_geometry.rs"
docker --context "$LAB_CONTEXT" build --target build-tools \
-t rustfs-inline-build-tools:1.100 \
-f .docker/inline-lab/Dockerfile .docker/inline-lab
docker --context "$LAB_CONTEXT" build \
--build-arg SOURCE_REVISION="$LAB_REVISION" \
-t rustfs-inline-validation:local \
-f .docker/inline-lab/Dockerfile "$LAB_BUILD_DIR"
LAB_EXPORT_ID="$(docker --context "$LAB_CONTEXT" create \
--label rustfs.validation=inline rustfs-inline-validation:local)"
docker --context "$LAB_CONTEXT" cp \
"$LAB_EXPORT_ID:/usr/local/bin/rustfs" "$LAB_BUILD_DIR/rustfs-linux"
docker --context "$LAB_CONTEXT" rm "$LAB_EXPORT_ID"
```
## Configure and probe
Select a new compose project and a private state directory. The adapter generates test credentials there. Existing resources require matching ownership labels. Configure checks the image's source revision and pins immutable image IDs. Each operation takes an exclusive state lock. Deployment waits for `/health/ready` on every node, covering storage, IAM, lock quorum and peer dependencies.
```bash
scripts/python_bin.sh scripts/inline_docker_lab.py --state-dir "$LAB_STATE" configure \
--context "$LAB_CONTEXT" --project "$LAB_PROJECT" \
--image rustfs-inline-validation:local --revision "$LAB_REVISION" \
--binary "$LAB_BUILD_DIR/rustfs-linux" --port 19080
scripts/python_bin.sh scripts/inline_docker_lab.py --state-dir "$LAB_STATE" deploy \
--bucket inline-geometry-probe --shard-bytes 0
scripts/python_bin.sh scripts/inline_docker_lab.py --state-dir "$LAB_STATE" probe \
--bucket inline-geometry-probe
```
The probe verifies complete bytes, HEAD sizes, Range contents, EC12+4 and the actual persisted block. It records inline markers for 4 KiB, 1 MiB, 1 MiB + 1 byte, 1.25 MiB, 1,572,856 bytes, 1,572,857 bytes and 1.5 MiB objects in `probe.json`. The server-only image includes the existing filemeta decoder through a small geometry inspector. Console assets and the administration CLI are outside this lab image.
## ABBA deployment hook
Use `inline_docker_lab.py ... deploy` in the existing `run_hotpath_warp_abba.sh` external deploy hook. Export `RUSTFS_ERASURE_SHARD_BYTES` according to `HOTPATH_ABBA_PHASE`. The adapter uses the supplied leg, phase, bucket, binary and evidence-file context, resets only owned lab volumes, checks running configuration and binary hashes, creates the fresh bucket and writes the runner's deployment evidence.
Use the same frozen binary for both sides of a geometry experiment. Fix the inline budget and encoding mode, then compare zero with one candidate shard value. Include the CPU, memory and image identities from every node in each result. The state `config.json` supplies the generated test credentials for the load driver. Set `DOCKER_CONTEXT` to the selected context when the driver captures Docker resource snapshots.
For content compatibility across an image or geometry change, configure the next selected image and artifact using the same state, then deploy with `--keep-data` and an existing bucket. This retains owned volumes. Compare existing objects and version IDs with the manifest prepared before the change.
## Cleanup
```bash
scripts/python_bin.sh scripts/inline_docker_lab.py --state-dir "$LAB_STATE" down --purge
```
Cleanup checks ownership for containers, volumes and networks. A configuration cannot adopt another project's resources. Raw measurements, state, Git bundles and build outputs stay in task artifact directories.
Unit checks run with `scripts/python_bin.sh scripts/test_inline_docker_lab.py` and are included in `make script-tests`.
## Linux content-probe client
The PUT after-probe requires a working `mc`. For a macOS client affected by cgo initialization failures, compile the same frozen public mc module for Linux with CGO disabled, build it with `Dockerfile.mc`, and use `scripts/inline_docker_mc.sh` as the process-local `mc` command. Record the module version, checksum and resulting image ID with the run.
The adapter rewrites only the lab loopback alias endpoint to the private `node1` endpoint. Alias configuration stays in the probe's own `MC_CONFIG_DIR`, and `cat` streams raw bytes. The existing HEAD, GET and hash checks continue to execute.
```bash
export RUSTFS_INLINE_LAB_STATE="$LAB_STATE"
export RUSTFS_INLINE_MC_IMAGE="$LAB_MC_IMAGE"
mkdir -p "$LAB_TOOLS"
ln -s "$PWD/scripts/inline_docker_mc.sh" "$LAB_TOOLS/mc"
export PATH="$LAB_TOOLS:$PATH"
```
## Compare transport and placement independently
`deploy --rpc-mode json-compat` explicitly clears both MessagePack-only flags on every node. `deploy --rpc-mode msgpack-only` enables both flags for the selected homogeneous lab fleet. Deployment evidence records the mode and checks both running flags. The default is `json-compat`. The ABBA hook may select the same mode through `RUSTFS_INLINE_LAB_RPC_MODE`.
First compare the base and bounded-multi-block binaries at `--shard-bytes 0` in JSON compatibility mode. Then repeat with MessagePack-only on both sides. A same-binary transport ABBA isolates the encoding effect. Keep the 128 KiB budget, CPU sets, strict durability, load and object set fixed. Check actual markers and block sizes at the boundary sizes before timing. Retain baseline-drift and candidate gates, error logs and raw request distributions.
For a rolling downgrade, redeploy the current binary with `--keep-data --rpc-mode json-compat` before introducing an older node. Check old and new reads, version IDs and batch deletes against the persisted manifest. A running MessagePack-only flag pair establishes the lab configuration. Compatibility with a particular old reader requires the explicit reader test.
## Preserve a compatibility manifest
Run the seed once in a fresh lab bucket. It records full byte oracles and the observed version inventory, including any additional version matching a known body, and leaves a 5 MiB multipart part ready for the next reader. Each bucket has its own private `compat-<bucket>.json` manifest. Keep the same state and volumes across image changes. A fresh deployment or cleanup that resets volumes moves existing compatibility manifests into `compat-history`. Retained-data deployments preserve the active manifests.
```bash
scripts/python_bin.sh scripts/inline_docker_lab.py --state-dir "$LAB_STATE" compat-seed --bucket inline-compat
scripts/python_bin.sh scripts/inline_docker_lab.py --state-dir "$LAB_STATE" compat-check --bucket inline-compat
# Configure the next frozen image and binary, then retain the data.
scripts/python_bin.sh scripts/inline_docker_lab.py --state-dir "$LAB_STATE" deploy \
--bucket inline-compat --shard-bytes 0 --rpc-mode json-compat --keep-data
scripts/python_bin.sh scripts/inline_docker_lab.py --state-dir "$LAB_STATE" compat-check --bucket inline-compat
scripts/python_bin.sh scripts/inline_docker_lab.py --state-dir "$LAB_STATE" compat-finish-mpu --bucket inline-compat
```
The check compares every persisted version ID, HEAD length, complete SHA256, cross-block Range and per-version geometry from `xl.meta`. The seed includes zero-length objects, null versions, the 192 KiB versioned admission boundary and the accumulated 128 KiB shard boundary. Seed a second bucket on the new writer before the downgrade, then check both manifests using the selected old reader. Complete an old upload on the new binary and a new upload on the old binary. After returning to the current fleet, `compat-delete --bucket <bucket>` deletes the recorded versions in one S3 batch and checks the exact remaining inventory and bytes.
Successful multipart-completion and batch-delete receipts are saved before subsequent content and geometry checks. If either check fails, rerun the same command with the same state directory to resume verification from the saved completion state. After a seed check fails with a saved manifest, use `compat-check` to resume verification.
For mixed-version testing, retain JSON compatibility mode throughout the rolling replacement. Verify each node's immutable image and binary, then perform manifest checks and new writes through both coordinator versions using `--coordinator node1` and `--coordinator node4`. The manifest records the actual writer coordinator revision from the running image. Stop one owned node to check read availability at EC12+4, restore it and check the manifests again. Record the exact oldest tested source revision, toolchain and build profile with the evidence. The version boundary is established by these executed cells.
An owned lab can replace just its fourth node through a compose image override. Select the frozen old image first and verify its revision label. Keep `compose.json` in JSON compatibility mode for this cell.
```bash
set -euo pipefail
OLD_IMAGE_ID="$(docker --context "$LAB_CONTEXT" image inspect "$OLD_IMAGE" --format '{{.Id}}')"
test "$(docker --context "$LAB_CONTEXT" image inspect "$OLD_IMAGE_ID" --format '{{index .Config.Labels "org.opencontainers.image.revision"}}')" = "$OLD_REVISION"
INLINE_OLD_IMAGE_ID="$OLD_IMAGE_ID" scripts/python_bin.sh - <<'PYTHON' > "$LAB_STATE/mixed-image.json"
import json
import os
import sys
override = {"services": {"node4": {"image": os.environ["INLINE_OLD_IMAGE_ID"]}}}
json.dump(override, sys.stdout)
PYTHON
docker --context "$LAB_CONTEXT" compose --project-name "$LAB_PROJECT" \
--file "$LAB_STATE/compose.json" --file "$LAB_STATE/mixed-image.json" \
up --detach --no-deps --no-build --pull never node4
scripts/python_bin.sh scripts/inline_docker_lab.py --state-dir "$LAB_STATE" compat-check \
--bucket inline-compat --coordinator node4
scripts/python_bin.sh scripts/inline_docker_lab.py --state-dir "$LAB_STATE" compat-seed \
--bucket inline-mixed-writer --coordinator node4
scripts/python_bin.sh scripts/inline_docker_lab.py --state-dir "$LAB_STATE" compat-check \
--bucket inline-mixed-writer --coordinator node1
```
Inspect all four running image IDs and binary hashes before recording the mixed-fleet result. Redeploy the configured current image with `--keep-data --rpc-mode json-compat` to restore a homogeneous fleet. The same manifest checks apply after stopping and restarting one owned node through the base compose file.
@@ -28,15 +28,6 @@ processors:
check_interval: 1s
limit_mib: 1024
spike_limit_mib: 256
transform/metrics:
error_mode: ignore
metric_statements:
# RustFS process metrics already expose process_pid as a metric label.
# Remove only the duplicate datapoint label and keep the shared resource
# process.pid available to sibling metrics and lock histograms.
- context: datapoint
statements:
- delete_key(attributes, "process_pid") where IsMatch(metric.name, "^rustfs_system_process_")
transform/logs:
error_mode: ignore
log_statements:
@@ -121,7 +112,7 @@ service:
exporters: [ otlp/tempo, otlp/jaeger ]
metrics:
receivers: [ otlp ]
processors: [ transform/metrics, memory_limiter, batch ]
processors: [ batch ]
exporters: [ prometheus ]
logs:
receivers: [ otlp ]
-40
View File
@@ -1,40 +0,0 @@
# syntax=docker/dockerfile:1.6
# Copyright 2026 RustFS Team
# Licensed under the Apache License, Version 2.0.
# A compatible native runtime with Python 3.11+, getent and stat is required by
# the isolated fixture's staging/provenance checks. This is a test image.
ARG RUNTIME_BASE=python:3.13-slim-trixie
FROM ${RUNTIME_BASE}
ARG SOURCE_REVISION
ARG BINARY_SHA256
ARG BUILD_PROFILE=dev
COPY rustfs /usr/bin/rustfs
RUN python3 - "$SOURCE_REVISION" "$BINARY_SHA256" "$BUILD_PROFILE" <<'PY'
import hashlib
import os
import re
import shutil
import sys
source, digest, profile = sys.argv[1:]
if not re.fullmatch(r"[0-9a-f]{40}", source):
raise SystemExit("invalid source revision")
if not re.fullmatch(r"[0-9a-f]{64}", digest):
raise SystemExit("invalid binary digest")
if profile not in ("dev", "release"):
raise SystemExit("invalid build profile")
with open("/usr/bin/rustfs", "rb") as binary:
if hashlib.file_digest(binary, "sha256").hexdigest() != digest:
raise SystemExit("binary digest mismatch")
for tool in ("getent", "stat"):
if shutil.which(tool) is None:
raise SystemExit(f"missing runtime tool: {tool}")
os.chmod("/usr/bin/rustfs", 0o755)
PY
LABEL io.rustfs.repro.source_revision="${SOURCE_REVISION}" \
io.rustfs.repro.binary_sha256="${BINARY_SHA256}" \
io.rustfs.repro.build_profile="${BUILD_PROFILE}"
USER 10001:10001
ENTRYPOINT ["/usr/bin/rustfs"]
-5
View File
@@ -1,5 +0,0 @@
{
"context": {
"fileName": "AGENTS.md"
}
}
+25 -6
View File
@@ -4,15 +4,34 @@ Applies to `.github/`.
## Pull Requests
PR conventions (English title/body, template usage, `--body-file` with the heredoc pattern, review-thread etiquette) live in [Git and PR rules](../.agents/references/pull-requests.md); do not duplicate them here.
PR conventions (English title/body, template usage, `--body-file` with the
heredoc pattern, review-thread etiquette) live in the root `AGENTS.md` under
"Git and PR Baseline" — that section is the single normative home; do not
duplicate its rules here.
## Workflow Changes
- Changes to workflow YAML or `.github/actionlint.yaml` must pass `actionlint` (run from the repo root) with zero findings before commit/PR. Install via `brew install actionlint`, or the download script in the actionlint repo (rhysd/actionlint); `make pre-pr` does not cover workflow files, so this is the required check for them.
- Custom self-hosted runner labels (`sm-standard-*`, `dind-*`) are declared in `.github/actionlint.yaml`. When introducing a new `runs-on:` label, declare it there in the same change. Never use the config or `-ignore` to silence real findings — fix the workflow (root `AGENTS.md`: make checks pass by fixing the cause, not by weakening the gate).
- If `shellcheck` is installed, actionlint also lints `run:` scripts; treat those findings as part of the gate.
- Runner placement and billing rules live in `.github/workflows/README.md`. A new job defaults to `ubuntu-latest` (free for this public repository); a self-hosted `runs-on` needs one of the reasons listed there, stated in a comment next to it, and a `rust-cache` key is never shared between the hosted and self-hosted images.
- Any change touching `.github/workflows/` must pass `actionlint` (run from
the repo root) with zero findings before commit/PR. Install via
`brew install actionlint`, or the download script in the actionlint repo
(rhysd/actionlint); `make pre-pr` does not cover workflow files, so this is
the required check for them.
- Custom self-hosted runner labels (`sm-standard-*`, `dind-*`) are declared
in `.github/actionlint.yaml`. When introducing a new `runs-on:` label,
declare it there in the same change. Never use the config or `-ignore` to
silence real findings — fix the workflow (root `AGENTS.md`: make checks
pass by fixing the cause, not by weakening the gate).
- If `shellcheck` is installed, actionlint also lints `run:` scripts; treat
those findings as part of the gate.
- Runner placement and billing rules live in `.github/workflows/README.md`.
A new job defaults to `ubuntu-latest` (free for this public repository);
a self-hosted `runs-on` needs one of the reasons listed there, stated in a
comment next to it, and a `rust-cache` key is never shared between the
hosted and self-hosted images.
## CI Alignment
When changing CI-sensitive behavior, keep local validation aligned with `.github/workflows/ci.yml`. Read the workflow file directly for the current gate steps — do not rely on (or add) a copied command list here; copies go stale. `make pre-pr` covers the shared local gates, not the full CI feature, platform, or end-to-end matrix. Select checks using the root verification tiers.
When changing CI-sensitive behavior, keep local validation aligned with
`.github/workflows/ci.yml`. Read the workflow file directly for the current
gate steps — do not rely on (or add) a copied command list here; copies go
stale. `make pre-pr` is the local equivalent of the main gate.
-54
View File
@@ -1,54 +0,0 @@
# Copyright 2026 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
name: Both-stack leg
description: Prove which S3 stack one e2e leg's servers selected and upload the leg for the summary (call with if always)
inputs:
profile:
description: The nextest profile the leg ran
required: true
stack:
description: The stack the leg requested, legacy or gateway
required: true
logs:
description: Directory or file holding the servers' output
required: true
runs:
using: composite
steps:
- name: Prove which stack the servers selected
shell: bash
env:
PROFILE: ${{ inputs.profile }}
STACK: ${{ inputs.stack }}
LOGS: ${{ inputs.logs }}
run: |
# Outside the workspace: e2e_binary.py refuses a run whose source tree changed.
mkdir -p "$RUNNER_TEMP/leg"
junit="target/nextest/${PROFILE}/junit.xml"
if [ -s "$junit" ]; then cp "$junit" "$RUNNER_TEMP/leg/junit.xml"; fi
# A gateway leg stopped by its preflight has no output; the preflight's stack.json stays.
if [ -e "$LOGS" ]; then python3 scripts/dual_stack.py stack --expect "$STACK" --out "$RUNNER_TEMP/leg/stack.json" "$LOGS"; fi
- name: Upload the leg
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: dual-stack-leg-${{ inputs.profile }}-${{ inputs.stack }}
overwrite: true
path: ${{ runner.temp }}/leg
if-no-files-found: warn
retention-days: 7
@@ -1,56 +0,0 @@
# Copyright 2026 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
name: Both-stack summary
description: Diff the legacy and gateway legs of this run (scripts/junit_diff.py) and upload one dated summary
inputs:
profiles:
description: Comma-separated nextest profiles whose two legs this run uploaded
required: true
runs:
using: composite
steps:
- name: Download the legs
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
pattern: dual-stack-leg-*
path: legs
- name: Diff every profile and write the summary
id: summary
shell: bash
env:
PROFILES: ${{ inputs.profiles }}
run: |
# The shell runs with -e; the exit status is reported after the summary is written.
set +e -u -o pipefail
date="$(date -u +%F)"
echo "date=${date}" >> "$GITHUB_OUTPUT"
mkdir -p legs
python3 scripts/dual_stack.py summary --profiles "$PROFILES" --legs legs --date "$date" --out-dir dual-stack
status=$?
if [ -f dual-stack/summary.md ]; then cat dual-stack/summary.md >> "$GITHUB_STEP_SUMMARY"; fi
exit "$status"
- name: Upload the dated summary
if: always() && steps.summary.outputs.date != ''
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: dual-stack-summary-${{ steps.summary.outputs.date }}
overwrite: true
path: dual-stack
if-no-files-found: error
retention-days: 30
+6 -46
View File
@@ -13,7 +13,7 @@
# limitations under the License.
name: Quick Checks
description: Run the shared RustFS quality checks.
description: Run the shared compile-free RustFS quality checks.
runs:
using: composite
@@ -24,30 +24,19 @@ runs:
tool: |
ripgrep@15.2.0
shellcheck@0.11.0
cargo-nextest@0.9.138
- name: Install actionlint
shell: bash
env:
GH_TOKEN: ${{ github.token }}
ASSET_DOWNLOADER: ${{ github.action_path }}/../setup/download_release_asset.py
run: |
actionlint_dir="$(mktemp -d "${RUNNER_TEMP}/actionlint.XXXXXX")"
python3 "$ASSET_DOWNLOADER" --repository rhysd/actionlint --tag v1.7.12 --asset actionlint_1.7.12_linux_amd64.tar.gz --output "$actionlint_dir/actionlint.tar.gz" --sha256 8aca8db96f1b94770f1b0d72b6dddcb1ebb8123cb3712530b08cc387b349a3d8
curl --fail --location --silent --show-error \
--output "$actionlint_dir/actionlint.tar.gz" \
https://github.com/rhysd/actionlint/releases/download/v1.7.12/actionlint_1.7.12_linux_amd64.tar.gz
echo "8aca8db96f1b94770f1b0d72b6dddcb1ebb8123cb3712530b08cc387b349a3d8 $actionlint_dir/actionlint.tar.gz" | sha256sum --check --status
tar -xzf "$actionlint_dir/actionlint.tar.gz" -C "$actionlint_dir" actionlint
rm "$actionlint_dir/actionlint.tar.gz"
echo "$actionlint_dir" >> "$GITHUB_PATH"
- name: Install Python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.12"
- name: Install workflow parser
shell: bash
run: python3 -m pip install --disable-pip-version-check PyYAML==6.0.3
- name: Install Rust toolchain
uses: dtolnay/rust-toolchain@29eef336d9b2848a0b548edc03f92a220660cdb8 # stable
with:
@@ -95,33 +84,7 @@ runs:
- name: Check s3s footprint ratchet
shell: bash
env:
# Pull requests and merge groups compare with their base revision; a
# push or scheduled run holds the tree to the baseline file exactly.
FOOTPRINT_BASE: ${{ github.event.pull_request.base.sha || github.event.merge_group.base_sha }}
run: |
if [[ -z "$FOOTPRINT_BASE" ]]; then
./scripts/check_s3s_footprint.sh
exit
fi
git cat-file -e "$FOOTPRINT_BASE^{commit}" 2>/dev/null || git fetch --no-tags --depth=1 origin "$FOOTPRINT_BASE"
./scripts/check_s3s_footprint.sh --base "$FOOTPRINT_BASE"
- name: Check release feature boundaries
shell: bash
run: python3 scripts/check_connect_release_boundary.py --features-only
- name: Check per-crate test-count ratchet
shell: bash
run: ./scripts/check_test_count_ratchet.sh
- name: Check cross-owner tests use test seams
shell: bash
run: python3 ./scripts/check_test_seams.py
- name: Check retired tests name existing replacements
shell: bash
run: python3 ./scripts/check_test_retirements.py
run: ./scripts/check_s3s_footprint.sh
- name: Check cryptographic capability wording
shell: bash
@@ -141,10 +104,7 @@ runs:
- name: Check test wiring
shell: bash
run: |
python3 ./scripts/check_test_wiring.py
python3 ./scripts/ci_gate.py --check-workflow
python3 ./scripts/validation_registry.py check
run: python3 ./scripts/check_test_wiring.py
- name: Check no planning docs committed
shell: bash
@@ -1,96 +0,0 @@
# Copyright 2026 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
name: "Reference RustFS binary"
description: "Download the pinned reference RustFS release, verify its sha256 and expose the executable path. A missing pin, missing asset or digest mismatch fails the step."
inputs:
pin-file:
description: "Pin file naming the reference release (repository-relative)"
required: false
default: ".config/reference-binary.toml"
platform:
description: "Platform table in the pin file"
required: false
default: "linux-x86_64-gnu"
destination:
description: "Directory that receives the executable; defaults to $RUNNER_TEMP/reference-binary"
required: false
default: ""
outputs:
path:
description: "Absolute path of the verified executable"
value: ${{ steps.unpack.outputs.path }}
version:
description: "Pinned reference version"
value: ${{ steps.pin.outputs.version }}
sha256:
description: "Pinned and verified archive sha256"
value: ${{ steps.pin.outputs.sha256 }}
runs:
using: "composite"
steps:
- name: Resolve reference pin
id: pin
shell: bash
env:
PIN_TOOL: ${{ github.action_path }}/reference_binary.py
PIN_FILE: ${{ inputs.pin-file }}
PIN_PLATFORM: ${{ inputs.platform }}
run: |
set -euo pipefail
resolved="$(python3 "$PIN_TOOL" resolve --pin "$PIN_FILE" --platform "$PIN_PLATFORM")"
printf '%s\n' "$resolved"
printf '%s\n' "$resolved" >> "$GITHUB_OUTPUT"
- name: Download and verify reference archive
id: download
shell: bash
env:
GH_TOKEN: ${{ github.token }}
ASSET_DOWNLOADER: ${{ github.action_path }}/../setup/download_release_asset.py
REF_REPOSITORY: ${{ steps.pin.outputs.repository }}
REF_TAG: ${{ steps.pin.outputs.tag }}
REF_ASSET: ${{ steps.pin.outputs.asset }}
REF_SHA256: ${{ steps.pin.outputs.sha256 }}
run: |
set -euo pipefail
archive="$RUNNER_TEMP/reference-archive/$REF_ASSET"
# Fails unless the release lists exactly one asset of this name whose
# published digest equals the pin and whose downloaded bytes match it.
python3 "$ASSET_DOWNLOADER" --repository "$REF_REPOSITORY" --tag "$REF_TAG" \
--asset "$REF_ASSET" --output "$archive" --sha256 "$REF_SHA256"
echo "archive=$archive" >> "$GITHUB_OUTPUT"
- name: Unpack reference executable
id: unpack
shell: bash
env:
PIN_TOOL: ${{ github.action_path }}/reference_binary.py
REF_ARCHIVE: ${{ steps.download.outputs.archive }}
REF_SHA256: ${{ steps.pin.outputs.sha256 }}
REF_EXECUTABLE: ${{ steps.pin.outputs.executable }}
REF_DESTINATION: ${{ inputs.destination }}
run: |
set -euo pipefail
destination="${REF_DESTINATION:-$RUNNER_TEMP/reference-binary}"
unpacked="$(python3 "$PIN_TOOL" unpack --archive "$REF_ARCHIVE" --sha256 "$REF_SHA256" \
--executable "$REF_EXECUTABLE" --destination "$destination")"
path="${unpacked#path=}"
test -x "$path"
echo "Reference executable: $path"
"$path" --version
echo "path=$path" >> "$GITHUB_OUTPUT"
@@ -1,144 +0,0 @@
#!/usr/bin/env python3
# Copyright 2026 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Resolve and unpack the pinned reference RustFS release binary.
`resolve` reads the pin file and prints the selected platform's identity as
`key=value` lines (GitHub step-output format). `unpack` re-verifies the
downloaded archive digest against the pin and extracts exactly one executable.
Every missing or malformed input is a failure; nothing is skipped.
"""
import argparse
import hashlib
from pathlib import Path
import re
import sys
import tomllib
import zipfile
REPOSITORY = re.compile(r"[A-Za-z0-9][A-Za-z0-9_.-]*/[A-Za-z0-9][A-Za-z0-9_.-]*")
TAG = re.compile(r"[A-Za-z0-9][A-Za-z0-9_.+-]*")
PLATFORM = re.compile(r"[a-z0-9][a-z0-9_-]*")
ASSET = re.compile(r"[A-Za-z0-9][A-Za-z0-9_.+-]*\.zip")
SHA256 = re.compile(r"[0-9a-f]{64}")
EXECUTABLE = re.compile(r"[A-Za-z0-9][A-Za-z0-9_.-]*")
MAX_EXECUTABLE_BYTES = 1024 * 1024 * 1024
class PinError(Exception):
"""A pin, archive or argument that cannot be trusted."""
def field(table, key, pattern, where):
value = table.get(key)
if not isinstance(value, str) or not pattern.fullmatch(value):
raise PinError(f"{where}: missing or invalid '{key}'")
return value
def resolve(pin_path, platform):
if not PLATFORM.fullmatch(platform or ""):
raise PinError("invalid platform name")
try:
data = tomllib.loads(Path(pin_path).read_text(encoding="utf-8"))
except FileNotFoundError as exc:
raise PinError(f"{pin_path}: reference pin file is missing") from exc
except (OSError, UnicodeDecodeError, tomllib.TOMLDecodeError) as exc:
raise PinError(f"{pin_path}: cannot read reference pin: {exc}") from exc
if data.get("schema") != 1 or type(data.get("schema")) is not int:
raise PinError(f"{pin_path}: unsupported pin schema")
identity = {
"repository": field(data, "repository", REPOSITORY, pin_path),
"version": field(data, "version", TAG, pin_path),
"tag": field(data, "tag", TAG, pin_path),
}
platforms = data.get("platforms")
if not isinstance(platforms, dict) or not platforms:
raise PinError(f"{pin_path}: no pinned platforms")
entry = platforms.get(platform)
if not isinstance(entry, dict):
raise PinError(f"{pin_path}: no pin for platform '{platform}'")
where = f"{pin_path} [platforms.{platform}]"
identity["asset"] = field(entry, "asset", ASSET, where)
identity["sha256"] = field(entry, "sha256", SHA256, where)
identity["executable"] = field(entry, "executable", EXECUTABLE, where)
if identity["version"] not in identity["asset"]:
raise PinError(f"{where}: asset name does not carry version {identity['version']}")
unknown = set(entry) - {"asset", "sha256", "executable"}
if unknown:
raise PinError(f"{where}: unknown keys {sorted(unknown)}")
return identity
def file_sha256(path):
with open(path, "rb") as stream:
return hashlib.file_digest(stream, "sha256").hexdigest()
def unpack(archive, sha256, executable, destination):
if not SHA256.fullmatch(sha256 or "") or not EXECUTABLE.fullmatch(executable or ""):
raise PinError("invalid expected digest or executable name")
try:
actual = file_sha256(archive)
except OSError as exc:
raise PinError(f"{archive}: reference archive is missing or unreadable") from exc
if actual != sha256:
raise PinError(f"{archive}: sha256 {actual} does not match the pinned digest")
try:
with zipfile.ZipFile(archive) as bundle:
matches = [info for info in bundle.infolist() if info.filename == executable]
if len(matches) != 1:
raise PinError(f"{archive}: must contain exactly one top-level '{executable}'")
info = matches[0]
if info.is_dir() or not 0 < info.file_size <= MAX_EXECUTABLE_BYTES:
raise PinError(f"{archive}: '{executable}' is not a plausible executable")
destination = Path(destination)
destination.mkdir(parents=True, exist_ok=True)
target = destination / executable
with bundle.open(info) as source, open(target, "wb") as sink:
while chunk := source.read(1024 * 1024):
sink.write(chunk)
except zipfile.BadZipFile as exc:
raise PinError(f"{archive}: not a valid zip archive") from exc
target.chmod(0o755)
return target
def main(argv=None):
parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
sub = parser.add_subparsers(dest="mode", required=True)
res = sub.add_parser("resolve")
res.add_argument("--pin", required=True)
res.add_argument("--platform", required=True)
unp = sub.add_parser("unpack")
unp.add_argument("--archive", required=True)
unp.add_argument("--sha256", required=True)
unp.add_argument("--executable", required=True)
unp.add_argument("--destination", required=True)
args = parser.parse_args(argv)
try:
if args.mode == "resolve":
for key, value in resolve(args.pin, args.platform).items():
print(f"{key}={value}")
else:
print(f"path={unpack(args.archive, args.sha256, args.executable, args.destination)}")
except PinError as exc:
print(f"reference binary: {exc}", file=sys.stderr)
return 1
return 0
if __name__ == "__main__":
sys.exit(main())
-107
View File
@@ -1,107 +0,0 @@
name: Runner capability and build pilot
description: Inspect an existing runner and measure isolated fixed-source cold/warm builds
inputs:
workload:
description: probe, server or workspace-tests
required: true
source-sha:
description: Exact merged main commit for builds
default: ""
toolchain:
description: Exact Rust version for builds
default: ""
scratch-root:
description: Existing absolute scratch root
default: ""
build-jobs:
description: Cargo concurrency
default: "2"
local-ssd:
description: Require operator storage attestation
default: "false"
runs:
using: composite
steps:
- name: Validate pilot input and inspect existing capabilities
shell: bash
env:
PILOT_WORKLOAD: ${{ inputs.workload }}
PILOT_SHA: ${{ inputs.source-sha }}
PILOT_TOOLCHAIN: ${{ inputs.toolchain }}
PILOT_SCRATCH: ${{ inputs.scratch-root }}
PILOT_JOBS: ${{ inputs.build-jobs }}
PILOT_SSD: ${{ inputs.local-ssd }}
run: |
[[ "$PILOT_WORKLOAD" =~ ^(probe|server|workspace-tests)$ ]]
[[ "$PILOT_JOBS" =~ ^([1-9]|1[0-6])$ ]]
if [[ "$PILOT_WORKLOAD" != probe ]]; then
[[ "$PILOT_SHA" =~ ^[a-f0-9]{40}$ ]]
[[ "$PILOT_TOOLCHAIN" =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]]
fi
scratch="${PILOT_SCRATCH:-$RUNNER_TEMP}"
[[ "$scratch" == /* && -d "$scratch" ]]
minimum_disk=1
minimum_memory=0
if [[ "$PILOT_WORKLOAD" == server ]]; then minimum_disk=24; minimum_memory=8; fi
if [[ "$PILOT_WORKLOAD" == workspace-tests ]]; then minimum_disk=90; minimum_memory=16; fi
extra=()
if [[ "$PILOT_SSD" == true ]]; then extra+=(--local-ssd-attestation /etc/rustfs-runner/storage.json); fi
python3 scripts/ci/runner_pilot.py --root "$scratch" \
--output "$RUNNER_TEMP/runner-pilot-evidence" \
--min-free-gib "$minimum_disk" --min-memory-gib "$minimum_memory" --architecture x86_64 "${extra[@]}"
- name: Checkout the trusted baseline history
if: inputs.workload != 'probe'
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
ref: main
path: .runner-pilot-source
fetch-depth: 0
persist-credentials: false
- name: Bind the build to a merged main commit
if: inputs.workload != 'probe'
shell: bash
env:
PILOT_SHA: ${{ inputs.source-sha }}
run: |
git -C .runner-pilot-source merge-base --is-ancestor "$PILOT_SHA" HEAD
git -C .runner-pilot-source checkout --detach "$PILOT_SHA"
- name: Install build dependencies
if: inputs.workload != 'probe'
shell: bash
run: |
sudo apt-get update
sudo apt-get install -y build-essential pkg-config libssl-dev cmake clang time
- name: Install the fixed protobuf compiler
if: inputs.workload != 'probe'
uses: rustfs/setup-protoc@a3705324d8f9bf5b6c3573fb6cf8ae421db55dd6 # v3.0.1
with:
version: "35.1"
repo-token: ${{ github.token }}
- name: Install the fixed flatbuffer compiler
if: inputs.workload != 'probe'
uses: Nugine/setup-flatc@698800de72a96bfb22cf60431dc21a2ff9a7e07b # v1
with:
version: "25.12.19"
- name: Install the requested exact Rust toolchain
if: inputs.workload != 'probe'
uses: dtolnay/rust-toolchain@29eef336d9b2848a0b548edc03f92a220660cdb8 # stable
with:
toolchain: ${{ inputs.toolchain }}
- name: Measure cold and warm builds without shared target caches
if: inputs.workload != 'probe'
shell: bash
env:
PILOT_WORKLOAD: ${{ inputs.workload }}
PILOT_SHA: ${{ inputs.source-sha }}
PILOT_TOOLCHAIN: ${{ inputs.toolchain }}
PILOT_SCRATCH: ${{ inputs.scratch-root }}
PILOT_JOBS: ${{ inputs.build-jobs }}
PILOT_SSD: ${{ inputs.local-ssd }}
run: |
extra=()
if [[ "$PILOT_SSD" == true ]]; then extra+=(--local-ssd-attestation /etc/rustfs-runner/storage.json); fi
python3 scripts/ci/runner_pilot.py --root "${PILOT_SCRATCH:-$RUNNER_TEMP}" \
--output "$RUNNER_TEMP/runner-pilot-evidence" \
--source .runner-pilot-source --source-sha "$PILOT_SHA" \
--toolchain "$PILOT_TOOLCHAIN" --benchmark "$PILOT_WORKLOAD" \
--jobs "$PILOT_JOBS" --timeout-seconds 3600 "${extra[@]}"
+5 -29
View File
@@ -63,15 +63,13 @@ inputs:
required: false
default: "true"
outputs:
cache-hit:
description: "'true' when the Rust cache was restored from an exact key match"
value: ${{ steps.rust-cache.outputs.cache-hit }}
runs:
using: "composite"
steps:
# Use the pinned code generators rather than Ubuntu's older packages.
# protobuf-compiler is deliberately absent: the setup-protoc step below
# installs 35.1 into the tool cache and prepends it to PATH, so the apt
# build (older, and never version-matched) was shadowed on every run and
# simply never used.
- name: Install system dependencies (Ubuntu)
if: runner.os == 'Linux'
shell: bash
@@ -81,9 +79,7 @@ runs:
build-essential \
pkg-config \
libssl-dev \
ripgrep \
python3 \
gh
ripgrep
# musl-gcc is needed by the native musl release leg, and zip/unzip by the
# release packaging steps. No CI test lane touches any of them.
@@ -92,32 +88,13 @@ runs:
shell: bash
run: sudo apt-get install -y musl-tools zip unzip
- name: Install pinned code generators through GitHub API (Linux x64)
if: runner.os == 'Linux' && runner.arch == 'X64'
shell: bash
env:
GH_TOKEN: ${{ github.token }}
ASSET_DOWNLOADER: ${{ github.action_path }}/download_release_asset.py
run: |
python3 "$ASSET_DOWNLOADER" --repository protocolbuffers/protobuf --tag v35.1 --asset protoc-35.1-linux-x86_64.zip --output "$RUNNER_TEMP/protoc.zip"
python3 "$ASSET_DOWNLOADER" --repository google/flatbuffers --tag v25.12.19 --asset 'Linux.flatc.binary.g++-13.zip' --output "$RUNNER_TEMP/flatc.zip"
python3 -m zipfile -e "$RUNNER_TEMP/protoc.zip" "$RUNNER_TEMP/protoc"
python3 -m zipfile -e "$RUNNER_TEMP/flatc.zip" "$RUNNER_TEMP/flatc"
chmod +x "$RUNNER_TEMP/protoc/bin/protoc" "$RUNNER_TEMP/flatc/flatc"
"$RUNNER_TEMP/protoc/bin/protoc" --version
"$RUNNER_TEMP/flatc/flatc" --version
echo "$RUNNER_TEMP/protoc/bin" >> "$GITHUB_PATH"
echo "$RUNNER_TEMP/flatc" >> "$GITHUB_PATH"
- name: Install protoc
if: runner.os != 'Linux' || runner.arch != 'X64'
uses: rustfs/setup-protoc@a3705324d8f9bf5b6c3573fb6cf8ae421db55dd6 # v3.0.1
with:
version: "35.1"
repo-token: ${{ github.token }}
- name: Install flatc
if: runner.os != 'Linux' || runner.arch != 'X64'
uses: Nugine/setup-flatc@698800de72a96bfb22cf60431dc21a2ff9a7e07b # v1
with:
version: "25.12.19"
@@ -148,7 +125,6 @@ runs:
tool: cargo-nextest@0.9.138
- name: Setup Rust cache
id: rust-cache
uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2
with:
# false is rust-cache's own default. With true, cleanup.ts returns
@@ -1,84 +0,0 @@
#!/usr/bin/env python3
"""Download a named GitHub release asset through its API and verify its digest."""
import argparse
import hashlib
import json
from pathlib import Path
import re
import subprocess
import sys
import tempfile
from urllib.parse import quote
def download(repository, tag, name, output, expected_sha256=None):
if not re.fullmatch(r"[A-Za-z0-9][A-Za-z0-9_.-]*/[A-Za-z0-9][A-Za-z0-9_.-]*", repository):
raise ValueError("invalid release repository")
if not tag or not name or "/" in name or "\\" in name:
raise ValueError("invalid release tag or asset name")
if expected_sha256 is not None and not re.fullmatch(r"[0-9a-f]{64}", expected_sha256):
raise ValueError("invalid pinned asset digest")
result = subprocess.run(
["gh", "api", "--hostname", "github.com", f"repos/{repository}/releases/tags/{quote(tag, safe='')}"],
check=True, capture_output=True, text=True, timeout=60,
)
release = json.loads(result.stdout)
if not isinstance(release, dict) or release.get("tag_name") != tag:
raise ValueError("release metadata does not match the requested tag")
assets = release.get("assets")
if not isinstance(assets, list) or any(not isinstance(asset, dict) for asset in assets):
raise ValueError("invalid release asset inventory")
matches = [asset for asset in assets if asset.get("name") == name]
if len(matches) != 1:
raise ValueError("release must contain exactly one named asset")
asset = matches[0]
asset_id, size, digest = asset.get("id"), asset.get("size"), asset.get("digest")
if type(asset_id) is not int or asset_id <= 0 or type(size) is not int or not 0 < size <= 512 * 1024 * 1024:
raise ValueError("invalid release asset identity or size")
if not isinstance(digest, str) or not re.fullmatch(r"sha256:[0-9a-f]{64}", digest):
raise ValueError("release asset has no valid SHA-256 digest")
sha256 = digest.removeprefix("sha256:")
if expected_sha256 is not None and expected_sha256 != sha256:
raise ValueError("release digest differs from the pinned digest")
output = Path(output)
output.parent.mkdir(parents=True, exist_ok=True)
temporary = None
try:
with tempfile.NamedTemporaryFile(dir=output.parent, prefix=".release-asset-", delete=False) as stream:
temporary = Path(stream.name)
# gh authenticates to the API; its HTTP client strips credentials on
# cross-host redirects. No browser URL or token is supplied in argv.
subprocess.run(
["gh", "api", "--hostname", "github.com", "--header", "Accept: application/octet-stream", f"repos/{repository}/releases/assets/{asset_id}"],
check=True, stdout=stream, timeout=300,
)
with temporary.open("rb") as stream:
actual = hashlib.file_digest(stream, "sha256").hexdigest()
if temporary.stat().st_size != size or actual != sha256:
raise ValueError("downloaded release asset failed size or SHA-256 verification")
temporary.replace(output)
print(f"Verified release asset: {repository} {tag} {name} sha256={actual}")
finally:
if temporary is not None:
temporary.unlink(missing_ok=True)
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--repository", required=True)
parser.add_argument("--tag", required=True)
parser.add_argument("--asset", required=True)
parser.add_argument("--output", required=True, type=Path)
parser.add_argument("--sha256")
args = parser.parse_args()
try:
download(args.repository, args.tag, args.asset, args.output, args.sha256)
except (ValueError, KeyError, TypeError, json.JSONDecodeError, subprocess.SubprocessError, OSError) as error:
print(f"Release asset download failed: {error}", file=sys.stderr)
return 1
return 0
if __name__ == "__main__":
raise SystemExit(main())
-16
View File
@@ -1,16 +0,0 @@
name: Verification runtime
description: Install the runtime for prebuilt test archives without a target cache
runs:
using: composite
steps:
- name: Install runtime tools
shell: bash
run: |
sudo apt-get update
sudo apt-get install -y libssl-dev ripgrep
- name: Install Rust toolchain
uses: dtolnay/rust-toolchain@29eef336d9b2848a0b548edc03f92a220660cdb8 # stable
- name: Install cargo-nextest
uses: taiki-e/install-action@96c7780c1d8a2b8723e12031def873a434d39d8d # nextest
with:
tool: cargo-nextest@0.9.138
-2
View File
@@ -32,8 +32,6 @@ updates:
- "overtrue"
- "majinghe"
ignore:
- dependency-name: "pulsar"
versions: [ "6.9.x" ]
- dependency-name: "object_store"
versions: [ "0.13.x" ]
- dependency-name: "libunftp"
+2 -1
View File
@@ -24,7 +24,8 @@ List relevant checks not run and the remaining risk. Use the validation tier in
## Impact
<!--
Describe user-facing, compatibility, API, deployment, configuration, or documentation impact. Use N/A when there is no expected impact.
Describe user-facing, compatibility, API, deployment, configuration, or
documentation impact. Use N/A when there is no expected impact.
-->
## Additional Notes
File diff suppressed because it is too large Load Diff
+5 -17
View File
@@ -5,7 +5,7 @@
## Billing facts
This repository is public. For a public repository, GitHub's standard hosted runners (`ubuntu-latest`, `windows-latest`, `macos-latest`) cost nothing and have no minute cap, whatever plan the organisation is on. GitHub bills only two things: larger hosted runners, including the `macos-latest-xlarge` release lane used here, and standard runners in private repositories.
This repository is public. For a public repository, GitHub's standard hosted runners (`ubuntu-latest`, `windows-latest`, `macos-latest`) cost nothing and have no minute cap, whatever plan the organisation is on. GitHub bills only two things: larger hosted runners, which have custom labels and are not used here, and standard runners in private repositories.
Every other label in this repository is capacity RustFS pays for. ARC pods and lab machines accrue cost while the machine exists. A Ubicloud label is metered for the minutes a job holds its VM:
@@ -13,11 +13,10 @@ Every other label in this repository is capacity RustFS pays for. ARC pods and l
|---|---|---|---|
| `ubuntu-latest` | nobody | GitHub-hosted, Ubuntu 24.04, 4 vCPU, 16 GB RAM, about 14 GB of free disk. Ships a C toolchain, `docker`, `gh`, `curl`, `wget`, `jq`, Python. Can `mount` tmpfs and loop devices; io_uring works | the default for every job |
| `windows-latest`, `macos-latest` | nobody | GitHub-hosted standard runners | Windows and macOS targets |
| `macos-latest-xlarge` | RustFS | GitHub-hosted larger Apple Silicon runner | macOS release builds, deferred until Linux acceptance for candidates |
| `sm-standard-2` | RustFS | ARC pod, Ubuntu 24.04, minimal image: no `gh`, no C toolchain; Docker only on the `dind-` label | only a job that a hosted runner cannot run (see the placement rule) |
| `sm-standard-4` | RustFS | ARC pod, Ubuntu 24.04, 14 CPUs and 28 GB (the label is not the core count), large disk | workspace-scale Cargo builds |
| `ubicloud-standard-16-arm` | RustFS | Ubicloud VM, alias of `ubicloud-standard-16-arm-ubuntu-2404`: Ubuntu 24.04 arm64, 16 vCPU, 48 GB, 300 GB disk. Reached through the `build.yml` matrix, so `actionlint.yaml` does not list it | native aarch64 release builds (`linux-aarch64-musl`, `linux-aarch64-gnu`) |
| `dind-sm-standard-2` | RustFS | ARC pod with Docker-in-Docker | Docker workloads with a measured resource or environment requirement beyond the hosted runner |
| `dind-sm-standard-2` | RustFS | ARC pod with Docker-in-Docker | image builds that need `buildx` and a registry push |
| `smoke-testing`, `pf-testing` | RustFS | long-lived lab machines | the packaged functional chain and performance suites |
The self-hosted pods and the hosted image are both Ubuntu 24.04, so a binary built on one runs on the other; `ci.yml` builds the debug binary on `sm-standard-4` and runs it on both pools.
@@ -28,15 +27,11 @@ A job runs on `ubuntu-latest` unless it has one of these reasons, and the reason
1. **Disk.** It restores or produces the workspace `target/`: `cargo build` or `cargo test` with `--all-targets`, `cargo nextest run --all`, or any reader of the `ci-dev` / `ci-feat-*` caches. The workspace test build leaves about 64 GB in `target/`; a hosted runner has about 14 GB.
2. **Critical-path CPU.** It is a build that several downstream lanes wait on and would blow its timeout at 4 vCPU (`build-rustfs-debug-binary` gates the e2e and s3-tests lanes).
3. **Hardware or environment.** Native aarch64 or access to the lab machines. Docker alone is not a reason: hosted Ubuntu ships Docker and supports the QEMU and Buildx setup actions.
3. **Hardware.** Native aarch64, Docker-in-Docker, or the lab machines.
4. **Cache ownership.** It reads a `rust-cache` key that a self-hosted job writes (see the cache rule below).
Jobs that do only the following have no reason to be self-hosted: shell or Python scripts, `cargo fmt`, `cargo deny`, typos, actionlint, API calls (`gh`, dependency review, issue and stale bots, CLA checks), downloading an artifact and running a prebuilt binary, or a `cargo build -p rustfs` with its own cache key. `e2e-distributed.yml` and `e2e-upgrade.yml` build the server and the `e2e_test` harness on `ubuntu-latest` inside their budgets, so "it compiles something" alone is not a reason.
The functional chain's `prepare` and `complete-chain` jobs, the performance runner availability check, and runner hygiene use `ubuntu-latest`: they inspect APIs or artifacts, not the lab hosts. The actual functional and performance suites remain on their lab runners. Functional-chain health collection uses `ubuntu-latest` so a lab outage cannot also stop its independent monitor. Docker image packaging also uses `ubuntu-latest` for both musl and glibc variants; it wraps verified candidate binaries rather than compiling the workspace, and still sets up QEMU and Buildx explicitly.
Moving these jobs off AKS reduces capacity demand, not necessarily the Azure bill immediately. VM costs fall only when nodes scale down or smaller capacity replaces them.
A hosted runner is also the only choice for a few things the pods cannot do: `mount -o loop` and `mount -t tmpfs` fail inside the ARC pods, and a container runtime cannot guarantee that `io_uring_setup` is allowed. `uring-integration` and `e2e-distributed` are on `ubuntu-latest` for that reason, not for cost.
## Cache rule
@@ -44,7 +39,6 @@ A hosted runner is also the only choice for a few things the pods cannot do: `mo
`Swatinem/rust-cache` builds its key from the OS, architecture, toolchain and lockfile. It does not include the runner label or image, so a cache written on an `sm-standard-4` pod and one written on `ubuntu-latest` would overwrite each other under the same `shared-key`.
- Every `shared-key` has exactly one writer. `cache-warm.yml` owns `ci-dev`, `ci-feat-rio`, `ci-feat-proto` and `ci-uring`; every other `./.github/actions/setup` call states `cache-save-if` explicitly, and `scripts/security/check_cache_save_if.sh` fails the build when one does not.
- The `ci-dev` key ends in a digest of `warm-ci-dev`'s build commands. Changing what that job warms therefore needs a new key, or readers keep restoring the old artifacts: `scripts/ci_gate.py --check-workflow` prints the key the commands require and fails until the writer and every reader in `ci.yml` and `reference-parity.yml` name it.
- All readers of a key run on the same image as its writer. Keys that live on hosted runners say so in their name (`ci-uring` is warmed on `ubuntu-latest`; `ci-e2e-distributed-hosted`, `e2e-upgrade-server-hosted`, `e2e-upgrade-tests-hosted`).
- Moving a job between pools therefore means renaming its key. The new key is empty until a run from `main` (a scheduled run, or a manual dispatch of that workflow from `main`) saves it; until then that lane does a cold build. The `build.yml` aarch64 release legs append `-ubicloud-arm` for this reason. `cache-warm.yml` does not write those keys.
- `CARGO_BUILD_JOBS` and every other `CARGO*`, `CC*`, `CFLAGS*`, `CXX*`, `CMAKE*` and `RUST*` variable is hashed into the key. `ci.yml` sets `CARGO_BUILD_JOBS` at step level for that reason; promoting it to job level rotates the key for every lane that shares it.
@@ -54,7 +48,7 @@ A hosted runner is also the only choice for a few things the pods cannot do: `mo
These are the patterns that have cost real money here without validating anything more:
- An `if: always()` summary or alert job on a self-hosted label inside a `workflow_run`-triggered workflow. The upstream workflow completes many times a day, the real jobs skip, and the summary still starts a pod each time. Gate it on the upstream job having run, and keep it on `ubuntu-latest`.
- Per-push workflows on `main`. A merge triggers cache warming and a `ci.yml` run, which validates the merged tree in full unless a merge-queue run already validated that tree; `build.yml` development builds are scheduled or manual; with tens of merges a day that is as much self-hosted time as all pull requests together. Anything that does not verify the merged tree itself belongs on a schedule or behind a `paths` filter.
- Per-push workflows on `main`. A merge triggers the full `ci.yml` matrix, `cache-warm.yml` and the `build.yml` development build; with tens of merges a day that is as much self-hosted time as all pull requests together. Anything that does not verify the merged tree itself belongs on a schedule or behind a `paths` filter.
- `apt-get install` of tools the hosted image already ships. It is only needed on the minimal pod images.
- A throttled `CARGO_BUILD_JOBS` on a 14-CPU pod. The limit exists for link-storm reasons documented in `ci.yml`; measure before raising it, and measure before lowering it further.
- Tiny jobs on ARC pods. Each one pays roughly a minute of pod start-up before its first step; a compile-free job that takes ten seconds costs more to schedule than to run.
@@ -68,7 +62,7 @@ gh api "repos/rustfs/rustfs/actions/runs/<run_id>/jobs?per_page=100" \
--jq '.jobs[] | "\(.conclusion)\t\(.labels | join(","))\t\(.started_at)\t\(.completed_at)\t\(.name)"'
```
Rows using `macos-latest-xlarge` or a label starting with `sm-`, `dind-`, `ubicloud-`, `smoke-` or `pf-` use paid capacity. At the October 2026 review, a full pull-request run of `ci.yml` spent about 150 self-hosted minutes, a third of it in jobs that never compiled; treat that as the scale of what a placement mistake costs, not as a current figure.
Only rows whose label starts with `sm-`, `dind-`, `ubicloud-`, `smoke-` or `pf-` cost anything. At the October 2026 review, a full pull-request run of `ci.yml` spent about 150 self-hosted minutes, a third of it in jobs that never compiled; treat that as the scale of what a placement mistake costs, not as a current figure.
## Checklist for a new or moved job
@@ -78,9 +72,3 @@ Rows using `macos-latest-xlarge` or a label starting with `sm-`, `dind-`, `ubicl
- A self-hosted `runs-on` carries a comment naming its reason from the placement rule.
- Every `./.github/actions/setup` call states `cache-save-if`, and a key read from one pool is not written from the other.
- Moving a job does not change its check name, so the `main` ruleset and `docs/testing/ci-gates.md` stay as they are; renaming it does, and then that document's change checklist applies.
## Release stages
The [candidate release pipeline](../../docs/operations/release-pipeline.md) builds Linux first and defers paid desktop capacity until acceptance. The daily `build.yml` schedule selects Linux; the Sunday 16:17 UTC schedule selects macOS and Windows. Manual dispatch can still select any platform list. All six platforms remain required before publication. The pipeline records and promotes the exact tested bytes. Builder workflows have no public-release credentials. Keep periodic development builds, PR verification and field-test chains separate when their contracts differ; do not re-enable disabled workflows as part of a source-file migration.
Larger runners remain billed for public repositories; see [GitHub runner billing](https://docs.github.com/en/actions/concepts/runners/larger-runners#billing). Actual spending comes from organization billing, not runner labels or an estimated build duration.
+8 -149
View File
@@ -23,16 +23,9 @@ on:
- 'deny.toml'
- '.github/actions/**'
- '.github/workflows/**'
- 'scripts/release/**'
- 'scripts/security/generate_release_supply_chain_assets.py'
- '.config/release-validation-plan.json'
- '.github/scheduled-validations.json'
- 'scripts/test_helm_release_workflow.py'
- 'scripts/test_release_packages.py'
- 'scripts/test_release_images.py'
- 'Dockerfile*'
- 'entrypoint.sh'
- 'bootstrap/**'
- 'scripts/release/create_or_update_release.sh'
- 'scripts/release/package_service_scripts.sh'
- 'scripts/release/package_versions.sh'
- 'scripts/test_package_service_scripts.sh'
- 'scripts/test_package_versions.sh'
- 'scripts/test_docker_workflow.py'
@@ -48,16 +41,9 @@ on:
- 'deny.toml'
- '.github/actions/**'
- '.github/workflows/**'
- 'scripts/release/**'
- 'scripts/security/generate_release_supply_chain_assets.py'
- '.config/release-validation-plan.json'
- '.github/scheduled-validations.json'
- 'scripts/test_helm_release_workflow.py'
- 'scripts/test_release_packages.py'
- 'scripts/test_release_images.py'
- 'Dockerfile*'
- 'entrypoint.sh'
- 'bootstrap/**'
- 'scripts/release/create_or_update_release.sh'
- 'scripts/release/package_service_scripts.sh'
- 'scripts/release/package_versions.sh'
- 'scripts/test_package_service_scripts.sh'
- 'scripts/test_package_versions.sh'
- 'scripts/test_docker_workflow.py'
@@ -146,38 +132,12 @@ jobs:
if: github.repository == 'rustfs/rustfs' && (github.event_name != 'pull_request' || github.event.action != 'closed')
runs-on: ubuntu-latest
timeout-minutes: 5
permissions:
contents: read
actions: read
outputs:
release_changed: ${{ steps.scope.outputs.release_changed }}
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
fetch-depth: 0
persist-credentials: false
- name: Scope native packaging checks to release changes
id: scope
env:
BASE_SHA: ${{ github.event.pull_request.base.sha || github.event.before }}
run: |
python3 - <<'PY'
import os, re, subprocess
event, base = os.environ['GITHUB_EVENT_NAME'], os.environ.get('BASE_SHA', '')
changed = event == 'workflow_dispatch'
if event in ('push', 'pull_request'):
if not re.fullmatch(r'[0-9a-f]{40}', base) or base == '0' * 40:
changed = True
else:
paths = subprocess.check_output(['git', 'diff', '--name-only', base, os.environ['GITHUB_SHA']], text=True).splitlines()
pattern = r'^(scripts/release/|scripts/test_release_|Dockerfile|entrypoint.sh$|bootstrap/|\.dockerignore$|\.config/release-validation-plan.json$|\.github/workflows/(audit|build|docker|package|helm-package|release-[^.]+)\.yml$)'
changed = any(re.match(pattern, path) for path in paths)
with open(os.environ['GITHUB_OUTPUT'], 'a') as output:
output.write(f'release_changed={str(changed).lower()}\n')
PY
- name: Report unpinned GitHub Actions
run: ./scripts/security/check_workflow_pins.sh --enforce
@@ -190,35 +150,8 @@ jobs:
- name: Check checkouts clear their credentials
run: ./scripts/security/check_persist_credentials.sh
- name: Check release candidate and promotion contracts
run: python3 -m unittest discover -s scripts/release -p 'test_*.py'
- name: Prepare artifact transport rehearsal
run: |
mkdir -p "$RUNNER_TEMP/release-transport"
printf '{"purpose":"transport contract, not release evidence"}\n' > "$RUNNER_TEMP/release-transport/transport.json"
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: release-transport-contract-${{ github.run_attempt }}
path: ${{ runner.temp }}/release-transport/transport.json
if-no-files-found: error
retention-days: 1
- name: Verify actual GitHub artifact metadata and downloaded bytes
env:
GH_TOKEN: ${{ github.token }}
run: |
python3 - <<'PY'
import os, sys
from pathlib import Path
sys.path.insert(0, 'scripts/release')
import candidate as c
run = c.github(f"repos/{c.REPOSITORY}/actions/runs/{os.environ['GITHUB_RUN_ID']}")
destination = Path(os.environ['RUNNER_TEMP']) / 'release-transport-downloaded'
c.download_artifact(run, f"release-transport-contract-{os.environ['GITHUB_RUN_ATTEMPT']}", destination)
c.require(c.read_json(destination / 'transport.json') == {'purpose': 'transport contract, not release evidence'}, 'artifact round trip changed')
PY
- name: Check preview release workflow policy
run: ./scripts/security/check_preview_release_workflow.sh
- name: Check tier evidence workflow isolation
run: ./scripts/security/check_tier_artifact_workflow.sh
@@ -229,80 +162,6 @@ jobs:
- name: Check package service lifecycle contract
run: ./scripts/test_package_service_scripts.sh
release-package-contract:
name: Release package contract
needs: workflow-pin-report
if: needs.workflow-pin-report.outputs.release_changed == 'true'
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Install real package managers
run: |
sudo apt-get update
sudo apt-get install -y ruby ruby-dev build-essential rpm fakeroot cpio
sudo gem install fpm --version 1.17.0 --no-document
- name: Exercise accepted and substituted payloads through both recipes
run: python3 scripts/test_release_packages.py
release-image-contract:
name: Release image contract (${{ matrix.variant }})
needs: workflow-pin-report
if: needs.workflow-pin-report.outputs.release_changed == 'true'
runs-on: ubuntu-latest
timeout-minutes: 35
strategy:
fail-fast: false
matrix:
variant: [musl, gnu]
env:
RELEASE_IMAGE_VARIANT: ${{ matrix.variant }}
NO_PROXY: 127.0.0.1,localhost,::1
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Install OCI tools
run: |
echo "RELEASE_IMAGE_OUTPUT=$RUNNER_TEMP/image-contract" >> "$GITHUB_ENV"
echo 'no_proxy=127.0.0.1,localhost,::1' >> "$GITHUB_ENV"
sudo apt-get update
sudo apt-get install -y skopeo
go version
- uses: docker/setup-qemu-action@c7c53464625b32c7a7e944ae62b3e17d2b600130 # v3
- uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # v3
- name: Exercise real OCI packaging, startup, registry retry and conflicts
run: python3 scripts/test_release_images.py
- name: Exercise amd64 OCI archive scanning
uses: aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25
env:
TRIVY_PLATFORM: linux/amd64
with:
input: ${{ runner.temp }}/image-contract-scan/image-${{ matrix.variant }}-amd64.tar
format: sarif
output: ${{ runner.temp }}/image-contract/scan-${{ matrix.variant }}-amd64.sarif
exit-code: "0"
- name: Exercise arm64 OCI archive scanning
uses: aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25
env:
TRIVY_PLATFORM: linux/arm64
with:
input: ${{ runner.temp }}/image-contract-scan/image-${{ matrix.variant }}-arm64.tar
format: sarif
output: ${{ runner.temp }}/image-contract/scan-${{ matrix.variant }}-arm64.sarif
exit-code: "0"
- name: Verify scan receipts before recording the archive
run: |
python3 - <<'PY'
import os, sys
from pathlib import Path
sys.path.insert(0, 'scripts/release')
import deliver, test_candidate
deliver.image_record(test_candidate.inputs(), os.environ['RELEASE_IMAGE_VARIANT'], Path(os.environ['RELEASE_IMAGE_OUTPUT']))
PY
dependency-review:
name: Dependency Review
runs-on: ubuntu-latest
+472 -216
View File
@@ -12,34 +12,51 @@
# See the License for the specific language governing permissions and
# limitations under the License.
# Scheduled development artifacts and reusable immutable-candidate builds.
# The daily cron builds Linux. Sunday 16:17 UTC builds macOS and Windows.
# Public release writes live exclusively in release-publish.yml. Candidate callers
# select Linux or desktop and pin release inputs before any matrix starts.
# Build and Release Workflow
#
# This workflow builds RustFS binaries and automatically triggers Docker image builds.
#
# Flow:
# 1. Build binaries for multiple platforms
# 2. Upload binaries to OSS storage
# 3. Trigger docker.yml to build and push images using the uploaded binaries
#
# Platform scope:
# - Development builds come from the nightly schedule and manual dispatch and
# cover the full platform matrix; they feed the dev download channel
# (artifacts/rustfs/dev on R2, the *-main-latest packages). Tags build the
# release matrix. Main pushes do not build: at the usual merge rate most
# per-merge builds were cancelled by the next merge before publishing
# anything, and the dev channel does not need per-merge granularity.
#
# Manual Parameters:
# - publish: Publish binaries/releases for manual runs (default: false)
# - build_docker: Advisory only; manual runs do not trigger Docker publishing
# - platforms: Comma-separated platform IDs or 'all' (default: all)
name: Build and Release
on:
workflow_call:
inputs:
candidate_manifest:
description: Frozen candidate inputs; candidates never publish from this workflow
required: true
type: string
platforms:
description: linux, desktop, all, or an explicit platform list
required: true
type: string
push:
tags: [ "*.*.*" ]
schedule:
- cron: "13 1 * * *" # Daily Linux, 01:13 UTC
- cron: "17 16 * * 0" # Weekly macOS and Windows, Sunday 16:17 UTC
- cron: "13 1 * * *" # Nightly 01:13 UTC
workflow_dispatch:
inputs:
publish:
description: "Publish the development download channel (never a release)"
description: "Publish to download channels and GitHub Releases (false keeps only workflow artifacts)"
required: false
default: false
type: boolean
build_docker:
# Advisory only. docker.yml triggers on workflow_run and its job-level
# condition requires the triggering event to be a tag push, so a manual
# dispatch of this workflow never produces images regardless of this
# value. Publishing remains controlled by the explicit publish input.
description: "Build and push Docker images after binary build (ignored: dispatch runs never reach docker.yml)"
required: false
default: true
type: boolean
platforms:
description: "Comma-separated targets or 'all' (e.g. linux-x86_64-musl,macos-aarch64)"
required: false
@@ -48,11 +65,10 @@ on:
permissions:
contents: read
actions: read
concurrency:
group: ${{ github.workflow }}-${{ github.event_name == 'push' && github.ref || github.run_id }}
cancel-in-progress: false
cancel-in-progress: ${{ github.event_name == 'push' }}
env:
CARGO_TERM_COLOR: always
@@ -68,39 +84,19 @@ jobs:
timeout-minutes: 30
outputs:
should_build: ${{ steps.check.outputs.should_build }}
should_publish: ${{ inputs.candidate_manifest == '' && github.ref == 'refs/heads/main' && (github.event_name == 'schedule' || inputs.publish == true) }}
should_publish: ${{ github.event_name != 'workflow_dispatch' || inputs.publish == true }}
build_type: ${{ steps.check.outputs.build_type }}
version: ${{ steps.check.outputs.version }}
short_sha: ${{ steps.check.outputs.short_sha }}
is_prerelease: ${{ steps.check.outputs.is_prerelease }}
candidate_id: ${{ steps.candidate.outputs.candidate_id }}
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
ref: ${{ inputs.candidate_manifest != '' && fromJson(inputs.candidate_manifest).source_sha || github.sha }}
- name: Validate frozen candidate
if: inputs.candidate_manifest != ''
id: candidate
env:
CANDIDATE_MANIFEST: ${{ inputs.candidate_manifest }}
run: |
python3 - <<'PYINPUT'
import json, os, sys
sys.path.insert(0, 'scripts/release')
import candidate
value = candidate.validate_inputs(json.loads(os.environ['CANDIDATE_MANIFEST']))
candidate.write_json('candidate-inputs.json', value)
candidate.emit_output('candidate_id', value['candidate_id'])
PYINPUT
python3 scripts/release/candidate.py validate-inputs --inputs candidate-inputs.json --check-checkout
- name: Determine build strategy
id: check
env:
CANDIDATE_MANIFEST: ${{ inputs.candidate_manifest }}
run: |
should_build=false
build_type="none"
@@ -111,14 +107,8 @@ jobs:
# Get short SHA for all builds
short_sha=$(git rev-parse --short HEAD)
# Candidates contain the final product version but have no release
# authority. Their only publication is immutable workflow artifacts.
if [[ -n "$CANDIDATE_MANIFEST" ]]; then
should_build=true
build_type="candidate"
version=$(python3 -c 'import json; print(json.load(open("candidate-inputs.json"))["version"])')
# Determine build type based on trigger
elif [[ "${{ startsWith(github.ref, 'refs/tags/') }}" == "true" ]]; then
if [[ "${{ startsWith(github.ref, 'refs/tags/') }}" == "true" ]]; then
# Tag push - preview, release, or prerelease
should_build=true
tag_name="${GITHUB_REF#refs/tags/}"
@@ -165,46 +155,6 @@ jobs:
echo " - Short SHA: $short_sha"
echo " - Is prerelease: $is_prerelease"
prepare-build-inputs:
needs: build-check
if: needs.build-check.outputs.should_build == 'true'
runs-on: ubuntu-latest
timeout-minutes: 15
outputs:
console: ${{ steps.inputs.outputs.console }}
toolchain: ${{ steps.inputs.outputs.toolchain }}
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
ref: ${{ inputs.candidate_manifest != '' && fromJson(inputs.candidate_manifest).source_sha || github.sha }}
- name: Resolve one Console and toolchain identity
id: inputs
env:
GH_TOKEN: ${{ github.token }}
CANDIDATE_MANIFEST: ${{ inputs.candidate_manifest }}
shell: bash
run: |
set -euo pipefail
if [[ -z "$CANDIDATE_MANIFEST" ]]; then
gh api repos/rustfs/console/releases/latest > console-release.json
rustup update stable --no-self-update
fi
python3 - <<'PYINPUT'
import json, os, subprocess, sys
sys.path.insert(0, 'scripts/release')
import candidate
raw = os.environ['CANDIDATE_MANIFEST']
if raw:
frozen = candidate.validate_inputs(json.loads(raw))
console, toolchain = frozen['console'], frozen['toolchain']
else:
console = candidate.console_asset(candidate.read_json('console-release.json'))
toolchain = subprocess.check_output(['rustc', '+stable', '--version'], text=True).split()[1]
candidate.emit_output('console', json.dumps(console, separators=(',', ':')))
candidate.emit_output('toolchain', toolchain)
PYINPUT
# Build RustFS binaries
prepare-platform-matrix:
name: Prepare Platform Matrix
@@ -221,33 +171,14 @@ jobs:
env:
# via env, not interpolation: a dispatch input is free-form text and
# would otherwise be pasted into the script for bash to evaluate.
# A direct schedule has no platforms input. Choose its subset from
# github.event.schedule, which is fixed when that cron fires, so a
# queued job cannot switch subsets because the runner clock moved.
# Callers that pass platforms keep that list, including a scheduled
# caller: inputs.platforms is set even though event_name stays schedule.
RAW_PLATFORMS: ${{ inputs.platforms || 'linux' }}
HAS_PLATFORM_INPUT: ${{ inputs.platforms && 'true' || 'false' }}
EVENT_NAME: ${{ github.event_name }}
EVENT_SCHEDULE: ${{ github.event.schedule }}
RAW_PLATFORMS: ${{ github.event_name == 'workflow_dispatch' && github.event.inputs.platforms || 'all' }}
run: |
set -euo pipefail
if [[ "${EVENT_NAME}" == "schedule" && "${HAS_PLATFORM_INPUT}" != "true" ]]; then
if [[ "${EVENT_SCHEDULE}" == "13 1 * * *" ]]; then
selected="linux"
elif [[ "${EVENT_SCHEDULE}" == "17 16 * * 0" ]]; then
selected="desktop"
else
echo "Unrecognized schedule cron: ${EVENT_SCHEDULE}" >&2
exit 1
fi
else
selected="${RAW_PLATFORMS}"
selected="$(echo "${selected}" | tr -d '[:space:]')"
if [[ -z "${selected}" ]]; then
selected="all"
fi
selected="$RAW_PLATFORMS"
selected="$(echo "${selected}" | tr -d '[:space:]')"
if [[ -z "${selected}" ]]; then
selected="all"
fi
# linux-aarch64-* need a real aarch64 host (placement rule: hardware)
@@ -262,14 +193,10 @@ jobs:
{"target_id":"linux-aarch64-musl","os":"ubicloud-standard-16-arm","target":"aarch64-unknown-linux-musl","cross":false,"platform":"linux","rustflags":"","features":"","glibc":""},
{"target_id":"linux-x86_64-gnu","os":"sm-standard-4","target":"x86_64-unknown-linux-gnu","cross":true,"platform":"linux","rustflags":"","features":"pyroscope","glibc":"2.31"},
{"target_id":"linux-aarch64-gnu","os":"ubicloud-standard-16-arm","target":"aarch64-unknown-linux-gnu","cross":true,"platform":"linux","rustflags":"","features":"pyroscope","glibc":"2.31"},
{"target_id":"macos-aarch64","os":"macos-latest-xlarge","target":"aarch64-apple-darwin","cross":false,"platform":"macos","rustflags":"","features":"pyroscope","glibc":""},
{"target_id":"macos-aarch64","os":"macos-latest","target":"aarch64-apple-darwin","cross":false,"platform":"macos","rustflags":"","features":"pyroscope","glibc":""},
{"target_id":"windows-x86_64","os":"windows-latest","target":"x86_64-pc-windows-msvc","cross":false,"platform":"windows","rustflags":"","features":"","glibc":""}
]}'
if [[ "$selected" == "linux" || "$selected" == "desktop" ]]; then
selected=$(jq -r --arg group "$selected" '.include[] | select(if $group == "linux" then .platform == "linux" else .platform != "linux" end) | .target_id' <<<"$all" | paste -sd ',' -)
fi
if [[ "${selected}" == "all" ]]; then
matrix="$(jq -c . <<<"${all}")"
else
@@ -297,13 +224,12 @@ jobs:
build-rustfs:
# Acceptance workflows select artifacts by this stable platform name.
name: Build RustFS (${{ matrix.target_id }})
needs: [ build-check, prepare-platform-matrix, prepare-build-inputs ]
needs: [ build-check, prepare-platform-matrix ]
if: needs.build-check.outputs.should_build == 'true' && needs.prepare-platform-matrix.result == 'success'
runs-on: ${{ matrix.os }}
timeout-minutes: 180
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
RUSTUP_TOOLCHAIN: ${{ needs.prepare-build-inputs.outputs.toolchain }}
# Release binaries ship without dial9 telemetry and therefore do not need
# --cfg tokio_unstable. Telemetry builds opt in explicitly (see
# `make build-profiling`); crates/obs/build.rs fails the compile if the
@@ -318,33 +244,11 @@ jobs:
with:
persist-credentials: false
fetch-depth: 0
ref: ${{ inputs.candidate_manifest != '' && fromJson(inputs.candidate_manifest).source_sha || github.sha }}
- name: Ensure Python for artifact verification
if: runner.os == 'Linux'
shell: bash
run: |
if ! command -v python3 >/dev/null 2>&1; then
sudo apt-get update
sudo apt-get install -y python3
fi
- name: Reuse completed candidate platform
id: reuse
if: needs.build-check.outputs.build_type == 'candidate'
env:
GH_TOKEN: ${{ github.token }}
CANDIDATE_MANIFEST: ${{ inputs.candidate_manifest }}
TARGET: ${{ matrix.target_id }}
shell: bash
run: |
python3 scripts/release/pipeline.py reuse-platform --target "$TARGET"
- name: Setup Rust environment
if: steps.reuse.outputs.reused != 'true'
uses: ./.github/actions/setup
with:
rust-version: ${{ needs.prepare-build-inputs.outputs.toolchain }}
rust-version: stable
target: ${{ matrix.target }}
# rust-cache keys include OS and architecture but not the runner image.
# The Ubicloud aarch64 legs therefore use their own suffix so they
@@ -364,35 +268,128 @@ jobs:
install-test-tools: 'false'
- name: Download static console assets
if: steps.reuse.outputs.reused != 'true'
shell: bash
env:
GH_TOKEN: ${{ github.token }}
CONSOLE_IDENTITY: ${{ needs.prepare-build-inputs.outputs.console }}
GITHUB_TOKEN: ${{ github.token }}
run: |
python3 - <<'PYCONSOLE'
import json, os, pathlib, subprocess, zipfile
value = json.loads(os.environ['CONSOLE_IDENTITY'])
subprocess.run(['python3', '.github/actions/setup/download_release_asset.py',
'--repository', 'rustfs/console', '--tag', value['tag'], '--asset', value['name'],
'--sha256', value['sha256'], '--output', 'console.zip'], check=True)
destination = pathlib.Path('rustfs/static')
destination.mkdir(parents=True, exist_ok=True)
with zipfile.ZipFile('console.zip') as archive:
for member in archive.infolist():
path = pathlib.PurePosixPath(member.filename)
if path.is_absolute() or '..' in path.parts or '\\' in member.filename:
raise SystemExit('unsafe Console archive member')
archive.extractall(destination)
if not (destination / 'index.html').is_file():
raise SystemExit('Console archive has no index.html')
PYCONSOLE
mkdir -p ./rustfs/static
verify_sha256() {
local expected="$1"
local file="$2"
local actual=""
local actual_line=""
if command -v sha256sum >/dev/null 2>&1; then
actual_line=$(sha256sum -- "$file") || return 1
elif command -v shasum >/dev/null 2>&1; then
actual_line=$(shasum -a 256 -- "$file") || return 1
else
echo "No SHA-256 verification tool found" >&2
return 1
fi
actual="${actual_line%%[[:space:]]*}"
if [ "$actual" = "$expected" ]; then
echo "SHA256 verified OK: $actual"
return 0
else
echo "SHA256 mismatch: expected=$expected actual=$actual" >&2
return 1
fi
}
download_console_assets() {
local curl_bin="curl"
local python_bin="python3"
local console_api="https://api.github.com/repos/rustfs/console/releases/latest"
local console_json="console-release.json"
local console_url
local console_sha256
local curl_auth_args=()
console_tag=""
if [[ "${{ matrix.platform }}" == "windows" ]]; then
curl_bin="curl.exe"
else
chmod +w ./rustfs/static/LICENSE || true
fi
if ! command -v "$python_bin" >/dev/null 2>&1; then
python_bin="python"
fi
if ! command -v "$python_bin" >/dev/null 2>&1; then
echo "No Python interpreter found for release metadata parsing" >&2
return 1
fi
if [[ -n "${GITHUB_TOKEN:-}" ]]; then
curl_auth_args=(-H "Authorization: Bearer ${GITHUB_TOKEN}" -H "X-GitHub-Api-Version: 2022-11-28")
fi
"$curl_bin" "${curl_auth_args[@]}" --fail -L "$console_api" \
-o "$console_json" --retry 3 --retry-delay 5 --max-time 300 || return 1
read -r console_tag console_url console_sha256 < <("$python_bin" - "$console_json" <<'PY'
import json
import re
import sys
with open(sys.argv[1], encoding="utf-8") as handle:
release = json.load(handle)
tag = release.get("tag_name", "") or "unknown"
for asset in release.get("assets", []):
name = asset.get("name", "")
digest = asset.get("digest", "")
url = asset.get("browser_download_url", "")
if name.startswith("rustfs-console-") and name.endswith(".zip") and digest.startswith("sha256:") and url:
sha256 = digest.split(":", 1)[1]
if not re.fullmatch(r"[0-9a-fA-F]{64}", sha256):
raise SystemExit(f"console zip asset has invalid sha256 digest: {sha256}")
sys.stdout.buffer.write(f"{tag} {url} {sha256}\n".encode("utf-8"))
break
else:
raise SystemExit("no console zip asset with sha256 digest found")
PY
) || return 1
if [[ -z "$console_url" || -z "$console_sha256" ]]; then
echo "Console release metadata is missing URL or SHA-256 digest" >&2
return 1
fi
echo "Console release: ${console_tag}"
echo "Downloading console asset: ${console_url}"
"$curl_bin" --fail -L "$console_url" -o console.zip --retry 3 --retry-delay 5 --max-time 300 || return 1
verify_sha256 "$console_sha256" console.zip || return 2
unzip -o console.zip -d ./rustfs/static || return 2
}
if download_console_assets; then
rm -f console.zip console-release.json
else
status=$?
rm -f console.zip console-release.json
if [[ "$status" -eq 2 ]]; then
echo "Console asset integrity verification failed" >&2
fi
echo "Failed to download verified console assets" >&2
exit 1
fi
if [[ ! -s ./rustfs/static/index.html ]]; then
echo "Console asset archive is missing static/index.html" >&2
exit 1
fi
asset_count=$(find ./rustfs/static -type f | wc -l | tr -d '[:space:]')
echo "Console assets ready: version=${console_tag:-unknown}, ${asset_count} files extracted to ./rustfs/static"
- name: Build RustFS
if: steps.reuse.outputs.reused != 'true'
shell: bash
env:
RUSTFS_BUILD_VERSION: ${{ needs.build-check.outputs.build_type == 'candidate' && needs.build-check.outputs.version || '' }}
BUILD_TARGET: ${{ matrix.target }}
BUILD_OS: ${{ matrix.platform }}-${{ startsWith(matrix.target, 'aarch64-') && 'aarch64' || 'x86_64' }}
run: |
@@ -443,7 +440,7 @@ jobs:
fi
- name: Verify macOS binary has no dynamic liblzma dependency
if: steps.reuse.outputs.reused != 'true' && (matrix.platform == 'macos')
if: matrix.platform == 'macos'
shell: bash
run: |
binary="target/${{ matrix.target }}/release/rustfs"
@@ -454,7 +451,7 @@ jobs:
"$binary" --version
- name: Generate the CPU symbol catalog
if: steps.reuse.outputs.reused != 'true' && (endsWith(matrix.target, '-linux-gnu') && matrix.features == 'pyroscope')
if: endsWith(matrix.target, '-linux-gnu') && matrix.features == 'pyroscope'
shell: bash
run: |
cargo metadata --locked --format-version 1 --filter-platform '${{ matrix.target }}' \
@@ -471,7 +468,6 @@ jobs:
"$release_dir/rustfs.cpu-symbol-catalog.json" > "$release_dir/rustfs.cpu-symbol-catalog.sha256"
- name: Create release package
if: steps.reuse.outputs.reused != 'true'
id: package
shell: bash
run: |
@@ -672,7 +668,7 @@ jobs:
echo "📊 Version: ${VERSION}"
- name: Verify packaged Linux binary
if: steps.reuse.outputs.reused != 'true' && (matrix.platform == 'linux')
if: matrix.platform == 'linux'
shell: bash
run: |
set -euo pipefail
@@ -701,7 +697,7 @@ jobs:
"$binary_dir/rustfs" --version
- name: Verify packaged console
if: steps.reuse.outputs.reused != 'true' && (matrix.target == 'x86_64-unknown-linux-gnu' || matrix.target == 'aarch64-unknown-linux-gnu')
if: matrix.target == 'x86_64-unknown-linux-gnu' || matrix.target == 'aarch64-unknown-linux-gnu'
shell: bash
run: |
set -euo pipefail
@@ -754,49 +750,15 @@ jobs:
cat "$log_file"
exit 1
- name: Verify native candidate and retain receipts
if: needs.build-check.outputs.build_type == 'candidate' && steps.reuse.outputs.reused != 'true'
env:
CANDIDATE_MANIFEST: ${{ inputs.candidate_manifest }}
TARGET: ${{ matrix.target_id }}
ARCHIVE: ${{ steps.package.outputs.package_file }}
shell: bash
run: |
python3 -c 'import os; from pathlib import Path; Path("inputs.json").write_text(os.environ["CANDIDATE_MANIFEST"])'
python3 scripts/release/verify.py native --inputs inputs.json --archive "$ARCHIVE" --target "$TARGET" --output .
- name: Record candidate artifact identity
if: steps.reuse.outputs.reused != 'true' && (needs.build-check.outputs.build_type == 'candidate')
env:
CANDIDATE_MANIFEST: ${{ inputs.candidate_manifest }}
TARGET_ID: ${{ matrix.target_id }}
PACKAGE_FILE: ${{ steps.package.outputs.package_file }}
shell: bash
run: |
python3 - <<'PYRECORD'
import json, os, sys
sys.path.insert(0, 'scripts/release')
import candidate
value = candidate.validate_inputs(json.loads(os.environ['CANDIDATE_MANIFEST']))
record = candidate.archive_record(os.environ['PACKAGE_FILE'], os.environ['TARGET_ID'], value)
candidate.write_json('record-' + os.environ['TARGET_ID'] + '.json', record)
PYRECORD
- name: Upload to GitHub artifacts
if: steps.reuse.outputs.reused != 'true'
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: ${{ needs.build-check.outputs.build_type == 'candidate' && format('candidate-{0}-{1}', needs.build-check.outputs.candidate_id, matrix.target_id) || steps.package.outputs.package_name }}
path: |
rustfs-*.zip
record-${{ matrix.target_id }}.json
identity-${{ matrix.target_id }}*
startup-${{ matrix.target_id }}.log
if-no-files-found: error
retention-days: ${{ (startsWith(github.ref, 'refs/tags/') || needs.build-check.outputs.build_type == 'candidate') && 30 || 7 }}
name: ${{ steps.package.outputs.package_name }}
path: "rustfs-*.zip"
retention-days: ${{ startsWith(github.ref, 'refs/tags/') && 30 || 7 }}
- name: Upload to Cloudflare R2
if: steps.reuse.outputs.reused != 'true' && (needs.build-check.outputs.should_publish == 'true' && env.R2_ACCESS_KEY_ID != '' && (needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease' || needs.build-check.outputs.build_type == 'development'))
if: needs.build-check.outputs.should_publish == 'true' && env.R2_ACCESS_KEY_ID != '' && (needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease' || needs.build-check.outputs.build_type == 'development')
env:
R2_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
R2_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
@@ -852,18 +814,13 @@ jobs:
export AWS_SECRET_ACCESS_KEY="$R2_SECRET_ACCESS_KEY"
export AWS_DEFAULT_REGION="auto"
# Determine upload path based on build type. Release and prerelease
# builds only stage their bytes here: the official
# artifacts/rustfs/release/ path is written solely by the
# certification-gated promote-release-artifacts job, from these same
# staged objects. Development builds keep the uncertified dev channel
# and never touch official or latest channels.
# Determine upload path based on build type
if [[ "$BUILD_TYPE" == "development" ]]; then
R2_PATH="s3://${R2_BUCKET}/artifacts/rustfs/dev/"
echo "📤 Uploading development build to R2 dev directory"
else
R2_PATH="s3://${R2_BUCKET}/artifacts/rustfs/staging/${GITHUB_RUN_ID}/${GITHUB_RUN_ATTEMPT}/"
echo "📤 Staging release build for certification-gated promotion"
R2_PATH="s3://${R2_BUCKET}/artifacts/rustfs/release/"
echo "📤 Uploading release build to R2 release directory"
fi
# Upload all rustfs zip files to R2 using S3-compatible API
@@ -878,7 +835,6 @@ jobs:
echo "✅ Upload completed successfully"
# Build summary
build-summary:
name: Build Summary
@@ -913,9 +869,309 @@ jobs:
echo "This run does not trigger Docker publishing."
fi
# Create GitHub Release for every valid release tag, including previews
create-release:
name: Create GitHub Release
needs: [ build-check, build-rustfs ]
if: needs.build-check.outputs.should_publish == 'true' && startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'preview' || needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
timeout-minutes: 30
permissions:
contents: write
outputs:
release_id: ${{ steps.create.outputs.release_id }}
release_url: ${{ steps.create.outputs.release_url }}
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
fetch-depth: 0
- name: Create GitHub Release
id: create
env:
GH_TOKEN: ${{ github.token }}
shell: bash
run: |
TAG="${{ needs.build-check.outputs.version }}"
VERSION="${{ needs.build-check.outputs.version }}"
IS_PRERELEASE="${{ needs.build-check.outputs.is_prerelease }}"
BUILD_TYPE="${{ needs.build-check.outputs.build_type }}"
TARGET_COMMITISH=$(git rev-parse --verify "refs/tags/${TAG}^{commit}")
# Determine release type for title
if [[ "$BUILD_TYPE" == "preview" ]]; then
RELEASE_TYPE="preview"
elif [[ "$BUILD_TYPE" == "prerelease" ]]; then
if [[ "$TAG" == *"alpha"* ]]; then
RELEASE_TYPE="alpha"
elif [[ "$TAG" == *"beta"* ]]; then
RELEASE_TYPE="beta"
elif [[ "$TAG" == *"rc"* ]]; then
RELEASE_TYPE="rc"
else
RELEASE_TYPE="prerelease"
fi
else
RELEASE_TYPE="release"
fi
# Create release title
if [[ "$IS_PRERELEASE" == "true" ]]; then
TITLE="RustFS $VERSION (${RELEASE_TYPE})"
else
TITLE="RustFS $VERSION"
fi
./scripts/release/create_or_update_release.sh \
"$TAG" \
"$TARGET_COMMITISH" \
"$TITLE" \
"$IS_PRERELEASE"
# Prepare and upload release assets
upload-release-assets:
name: Upload Release Assets
needs: [ build-check, build-rustfs, create-release ]
if: needs.build-check.outputs.should_publish == 'true' && startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'preview' || needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
timeout-minutes: 30
permissions:
contents: write
actions: read
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Download all build artifacts
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
path: ./artifacts
pattern: rustfs-*
merge-multiple: true
- name: Prepare release assets
id: prepare
shell: bash
run: |
VERSION="${{ needs.build-check.outputs.version }}"
TAG="${{ needs.build-check.outputs.version }}"
mkdir -p ./release-assets
# Copy and verify artifacts (including latest files created during build)
ASSETS_COUNT=0
for file in ./artifacts/*.zip; do
if [[ -f "$file" ]]; then
cp "$file" ./release-assets/
ASSETS_COUNT=$((ASSETS_COUNT + 1))
fi
done
if [[ $ASSETS_COUNT -eq 0 ]]; then
echo "❌ No artifacts found!"
exit 1
fi
cd ./release-assets
# Generate checksums for all files (including latest versions)
if compgen -G "*.zip" >/dev/null; then
sha256sum -- *.zip > SHA256SUMS
sha512sum -- *.zip > SHA512SUMS
fi
cd ..
python3 scripts/security/generate_release_supply_chain_assets.py \
--asset-dir ./release-assets \
--version "$VERSION" \
--tag "$TAG" \
--build-type "${{ needs.build-check.outputs.build_type }}" \
--repository "${{ github.repository }}" \
--ref "${{ github.ref }}" \
--sha "${{ github.sha }}" \
--run-id "${{ github.run_id }}" \
--run-attempt "${{ github.run_attempt }}"
cd ./release-assets
echo "📦 Prepared assets:"
ls -la
echo "🔢 Total asset count: $ASSETS_COUNT"
- name: Upload to GitHub Release
env:
GH_TOKEN: ${{ github.token }}
shell: bash
run: |
TAG="${{ needs.build-check.outputs.version }}"
cd ./release-assets
# Upload all files
for file in *; do
if [[ -f "$file" ]]; then
echo "📤 Uploading $file..."
gh release upload "$TAG" "$file" --clobber
fi
done
echo "✅ All assets uploaded successfully"
# Update latest.json for every release tag (stable and prerelease): the
# project currently ships prerelease tags only, so gating this to stable
# left the version pointer permanently stale. release_type records whether
# the pointed-to version is a prerelease.
update-latest-version:
name: Update Latest Version
needs: [ build-check, publish-release ]
if: needs.build-check.outputs.should_publish == 'true' && startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
runs-on: ubuntu-latest
timeout-minutes: 30
steps:
- name: Update latest.json
env:
OSS_ACCESS_KEY_ID: ${{ secrets.ALICLOUDOSS_KEY_ID }}
OSS_ACCESS_KEY_SECRET: ${{ secrets.ALICLOUDOSS_KEY_SECRET }}
OSS_REGION: cn-beijing
OSS_ENDPOINT: https://oss-cn-beijing.aliyuncs.com
shell: bash
run: |
if [[ -z "$OSS_ACCESS_KEY_ID" ]]; then
echo "⚠️ OSS credentials not available, skipping latest.json update"
exit 0
fi
VERSION="${{ needs.build-check.outputs.version }}"
TAG="${{ needs.build-check.outputs.version }}"
if [[ "${{ needs.build-check.outputs.build_type }}" == "prerelease" ]]; then
RELEASE_TYPE="prerelease"
else
RELEASE_TYPE="stable"
fi
# Install ossutil
OSSUTIL_VERSION="2.1.1"
OSSUTIL_ZIP="ossutil-${OSSUTIL_VERSION}-linux-amd64.zip"
OSSUTIL_DIR="ossutil-${OSSUTIL_VERSION}-linux-amd64"
OSSUTIL_SHA256="60f3a808cbbdfd4b5269b8f967c6a66f564c9dacff52d93a5d21a588976ab5a2"
curl --fail -L -o "$OSSUTIL_ZIP" "https://gosspublic.alicdn.com/ossutil/v2/${OSSUTIL_VERSION}/${OSSUTIL_ZIP}"
printf '%s %s\n' "$OSSUTIL_SHA256" "$OSSUTIL_ZIP" | sha256sum -c -
unzip "$OSSUTIL_ZIP"
OSSUTIL_BIN="${RUNNER_TEMP}/ossutil"
mv "${OSSUTIL_DIR}/ossutil" "$OSSUTIL_BIN"
rm -rf "$OSSUTIL_DIR" "$OSSUTIL_ZIP"
chmod +x "$OSSUTIL_BIN"
# Create latest.json
cat > latest.json << EOF
{
"version": "${VERSION}",
"tag": "${TAG}",
"release_date": "$(date -u +%Y-%m-%dT%H:%M:%SZ)",
"release_type": "${RELEASE_TYPE}",
"download_url": "https://github.com/${{ github.repository }}/releases/tag/${TAG}"
}
EOF
# Upload to OSS
"$OSSUTIL_BIN" cp latest.json oss://rustfs-version/latest.json --force
echo "✅ Updated latest.json for ${RELEASE_TYPE} release $VERSION"
# Publish release (remove draft status)
publish-release:
name: Publish Release
needs: [ build-check, create-release, upload-release-assets ]
if: needs.build-check.outputs.should_publish == 'true' && startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'preview' || needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
timeout-minutes: 30
permissions:
contents: write
steps:
- name: Publish release
env:
GH_TOKEN: ${{ github.token }}
shell: bash
run: |
TAG="${{ needs.build-check.outputs.version }}"
BUILD_TYPE="${{ needs.build-check.outputs.build_type }}"
RELEASE_ID="${{ needs.create-release.outputs.release_id }}"
# Publish the release and correct its channel state on retries.
# Only a stable final release may become GitHub Latest.
if [[ "$BUILD_TYPE" == "release" ]]; then
gh api --method PATCH "repos/${GITHUB_REPOSITORY}/releases/${RELEASE_ID}" \
-F draft=false \
-F prerelease=false \
-f make_latest=true >/dev/null
else
gh api --method PATCH "repos/${GITHUB_REPOSITORY}/releases/${RELEASE_ID}" \
-F draft=false \
-F prerelease=true \
-f make_latest=false >/dev/null
fi
echo "🎉 Released $TAG successfully!"
echo "📄 Release URL: ${{ needs.create-release.outputs.release_url }}"
# Remove the internal preview releases once the deliverable release is live.
# Only the Releases are deleted; the -preview.N tags stay so the validated
# commit remains traceable.
cleanup-preview-releases:
name: Cleanup Preview Releases
needs: [ build-check, publish-release ]
if: needs.build-check.outputs.should_publish == 'true' && startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
timeout-minutes: 30
permissions:
contents: write
steps:
- name: Delete preview releases for this target
env:
GH_TOKEN: ${{ github.token }}
shell: bash
run: |
set -euo pipefail
TAG="${{ needs.build-check.outputs.version }}"
RELEASES_JSON="${RUNNER_TEMP}/releases.json"
# Fetch before filtering: a failed listing must abort here instead of
# looking like "nothing to clean up".
gh api --paginate "repos/${GITHUB_REPOSITORY}/releases?per_page=100" > "$RELEASES_JSON"
# Match only <target>-preview.<digits>. String operations, not a
# regex over the tag, so dots in the version cannot widen the match.
DELETED=0
while IFS= read -r preview_tag; do
[[ -n "$preview_tag" ]] || continue
echo "🧹 Deleting preview release $preview_tag (tag kept)"
gh release delete "$preview_tag" --repo "${GITHUB_REPOSITORY}" --yes
DELETED=$((DELETED + 1))
done < <(
jq -r --arg tag "$TAG" '
.[]
| select(.tag_name | startswith($tag + "-preview."))
| select(.tag_name | ltrimstr($tag + "-preview.") | test("^[0-9]+$"))
| .tag_name
' "$RELEASES_JSON"
)
if [[ "$DELETED" -eq 0 ]]; then
echo "ℹ️ No preview releases to clean up for $TAG"
else
echo "✅ Removed $DELETED preview release(s) for $TAG"
fi
alert-on-failure:
name: Alert on scheduled failure
needs: [build-check, prepare-build-inputs, prepare-platform-matrix, build-rustfs, build-summary]
needs: [build-check, prepare-platform-matrix, build-rustfs, build-summary]
if: >-
always() && github.event_name == 'schedule' &&
(contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled'))
+25 -36
View File
@@ -33,22 +33,22 @@
# cache-save-if to anything but 'false' for it; every lane in ci.yml reads.
# scripts/security/check_cache_save_if.sh keeps the declarations explicit.
#
# A reader restores only what the writer saved, and Cargo reuses a dependency
# only when it resolves the same features for it. Every package selection and
# feature set resolves differently and changes -Cmetadata, so each job builds
# its readers' own invocations, not a superset (rustfs/backlog#1600).
#
# Rust-cache never replaces an entry restored from an exact key match: the key
# hashes the lockfile, toolchain and compiler environment, not the commit. A
# job whose restore was an exact hit therefore skips its builds; they could
# only produce what is already stored.
# The builds are supersets of what the reading lanes compile, because a reader
# restores only what the writer saved. Feature resolution matters here: a lane
# built with e2e-test-hooks resolves dependency features differently, which
# changes -Cmetadata, so the plain build does not cover it. See
# rustfs/backlog#1600.
name: Cache Warm
on:
schedule:
# Writes a key that no push created, such as one for a new stable
# toolchain; with every key present the jobs stop after the restore.
# Every six hours; the 22:41 UTC run lands before ci.yml's nightly 00:11
# UTC matrix. A reader restores the newest entry under its key and
# rebuilds whatever its own diff changed, so a cache up to six hours old
# costs a PR the handful of crates merged since. Warming after every merge
# (fourteen completed runs a day, about two sm-standard-4 hours each)
# spent far more than that to avoid it.
- cron: "41 4,10,16,22 * * *"
push:
branches: [ main ]
@@ -96,8 +96,8 @@ env:
RUST_BACKTRACE: 1
jobs:
# Readers: workspace-clippy, workspace-lint-and-build, build-rustfs-debug-binary,
# connect-feature-boundaries (nightly) and reference-parity.yml.
# Readers: test-and-lint (workspace and ILM), build-rustfs-debug-binary,
# e2e-tests, e2e-full.
warm-ci-dev:
name: Warm ci-dev
runs-on: sm-standard-4
@@ -111,13 +111,10 @@ jobs:
persist-credentials: false
- name: Setup Rust environment
id: setup
uses: ./.github/actions/setup
with:
rust-version: stable
# Digest of the build steps below; scripts/ci_gate.py --check-workflow
# names the new key when they change, and every reader must follow.
cache-shared-key: ci-dev-ea7eec74fe14
cache-shared-key: ci-dev
cache-save-if: 'true'
install-build-packaging-tools: 'false'
@@ -133,7 +130,7 @@ jobs:
# 50% and 25% respectively; if linking dominates instead, the answer is
# mold/lld plus split-debuginfo, which is exactly the part sccache cannot
# touch. Off by default — this doubles the ci-dev build.
- name: Build the workspace with --timings
- name: Build ci-dev superset (with --timings)
if: inputs.emit_timings
env:
CARGO_BUILD_JOBS: "2"
@@ -148,18 +145,18 @@ jobs:
retention-days: 30
if-no-files-found: error
# The workspace and E2E test archives of workspace-lint-and-build and
# build-rustfs-debug-binary, then that job's hooks server.
- name: Build ci-dev reader selections
if: steps.setup.outputs.cache-hit != 'true'
# --all-targets covers the test binaries nextest builds, including
# e2e_test, which test-and-lint's own run excludes. The second build adds
# the e2e-test-hooks feature resolution that build-rustfs-debug-binary uses
# and that no lint lane enables.
- name: Build ci-dev superset
env:
# Same limit ci.yml puts on its archive steps: three concurrent test
# binary links saturate the self-hosted runner's overlay I/O and can
# wedge Cargo (#5394).
# Same limit ci.yml puts on its nextest step: this builds the same
# ~100 workspace test binaries, and three concurrent links saturate the
# self-hosted runner's overlay I/O and can wedge Cargo (#5394).
CARGO_BUILD_JOBS: "2"
run: |
python3 scripts/ci_test_artifact.py cargo warm --configuration workspace
python3 scripts/ci_test_artifact.py cargo warm --configuration e2e
cargo build --workspace --all-targets
cargo build -p rustfs --bins --features e2e-test-hooks
# ci.yml runs Clippy on every pull request. Clippy's check-mode artifacts
@@ -168,9 +165,8 @@ jobs:
# touches. A lint failure on main must not stop the warm-up: the cache is
# still saved (cache-on-failure), and the failure belongs to ci.yml.
- name: Warm Clippy check artifacts
if: steps.setup.outputs.cache-hit != 'true'
run: |
if ! python3 scripts/ci_test_artifact.py cargo lint --configuration workspace; then
if ! cargo clippy --all-targets -- -D warnings; then
echo "::warning::cargo clippy reports findings on this commit; check artifacts are still cached"
fi
@@ -217,7 +213,6 @@ jobs:
persist-credentials: false
- name: Setup Rust environment
id: setup
uses: ./.github/actions/setup
with:
rust-version: stable
@@ -226,7 +221,6 @@ jobs:
install-build-packaging-tools: 'false'
- name: Build ci-feat-rio superset
if: steps.setup.outputs.cache-hit != 'true'
env:
# --all-targets links the same test binaries as the reader lane.
CARGO_BUILD_JOBS: "2"
@@ -250,7 +244,6 @@ jobs:
persist-credentials: false
- name: Setup Rust environment
id: setup
uses: ./.github/actions/setup
with:
rust-version: stable
@@ -259,7 +252,6 @@ jobs:
install-build-packaging-tools: 'false'
- name: Build ci-feat-proto superset
if: steps.setup.outputs.cache-hit != 'true'
env:
# Avoid an unbounded burst of concurrent test-binary links (#5394).
CARGO_BUILD_JOBS: "2"
@@ -282,7 +274,6 @@ jobs:
persist-credentials: false
- name: Setup Rust environment
id: setup
uses: ./.github/actions/setup
with:
rust-version: stable
@@ -291,9 +282,7 @@ jobs:
install-build-packaging-tools: 'false'
- name: Install build dependencies
if: steps.setup.outputs.cache-hit != 'true'
run: sudo apt-get update && sudo apt-get install -y protobuf-compiler
- name: Build ci-uring superset
if: steps.setup.outputs.cache-hit != 'true'
run: cargo build -p rustfs-ecstore --all-targets
@@ -1,60 +0,0 @@
name: CI Resource Accounting
on:
workflow_dispatch:
inputs:
since:
description: "Inclusive run creation timestamp, for example 2026-10-01T00:00:00Z"
required: true
type: string
until:
description: "Inclusive end timestamp; keep the window below 1000 runs"
required: true
type: string
permissions:
contents: read
actions: read
pull-requests: read
concurrency:
group: ci-resource-accounting
cancel-in-progress: false
jobs:
collect:
if: github.repository == 'rustfs/rustfs'
runs-on: ubuntu-latest
timeout-minutes: 30
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Collect all workflow attempts in the window
env:
GH_TOKEN: ${{ github.token }}
ACCOUNTING_SINCE: ${{ inputs.since }}
ACCOUNTING_UNTIL: ${{ inputs.until }}
ACCOUNTING_REPOSITORY: ${{ github.repository }}
run: |
python3 scripts/ci/actions_cost_report.py \
--repository "$ACCOUNTING_REPOSITORY" \
--since "$ACCOUNTING_SINCE" --until "$ACCOUNTING_UNTIL" \
--output "$RUNNER_TEMP/ci-accounting.json" \
--summary "$RUNNER_TEMP/ci-accounting.md"
- name: Show available accounting summary
if: always()
run: |
if [[ -f "$RUNNER_TEMP/ci-accounting.md" ]]; then
cat "$RUNNER_TEMP/ci-accounting.md" >> "$GITHUB_STEP_SUMMARY"
fi
- name: Retain recomputable accounting evidence
if: always()
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
with:
name: ci-accounting-${{ github.run_id }}-${{ github.run_attempt }}
path: |
${{ runner.temp }}/ci-accounting.json
${{ runner.temp }}/ci-accounting.md
if-no-files-found: error
retention-days: 30
-92
View File
@@ -1,92 +0,0 @@
name: CI Runner Pilot
on:
workflow_dispatch:
inputs:
pool:
description: "Existing runner pool; no infrastructure is created"
required: true
type: choice
options: [hosted, arc]
default: hosted
workload:
description: "Probe only, or perform cold/warm fixed-source builds"
required: true
type: choice
options: [probe, server, workspace-tests]
default: probe
source_sha:
description: "Exact 40-character merged main SHA; required for builds"
type: string
toolchain:
description: "Exact Rust version such as 1.92.0; required for builds"
type: string
scratch_root:
description: "Existing absolute scratch directory; empty uses runner.temp"
type: string
build_jobs:
description: "Fixed Cargo concurrency for both comparisons (1-16)"
type: string
default: "2"
local_ssd:
description: "Require operator /etc/rustfs-runner/storage.json attestation"
type: boolean
default: false
permissions:
contents: read
concurrency:
group: ci-runner-pilot
cancel-in-progress: false
jobs:
hosted:
if: github.repository == 'rustfs/rustfs' && github.ref == 'refs/heads/main' && inputs.pool == 'hosted'
runs-on: ubuntu-latest
timeout-minutes: 150
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- uses: ./.github/actions/runner-pilot
with:
workload: ${{ inputs.workload }}
source-sha: ${{ inputs.source_sha }}
toolchain: ${{ inputs.toolchain }}
scratch-root: ${{ inputs.scratch_root }}
build-jobs: ${{ inputs.build_jobs }}
local-ssd: ${{ inputs.local_ssd }}
- name: Retain hosted pilot evidence
if: always()
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
with:
name: runner-pilot-hosted-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/runner-pilot-evidence
if-no-files-found: error
retention-days: 30
arc:
if: github.repository == 'rustfs/rustfs' && github.ref == 'refs/heads/main' && inputs.pool == 'arc'
# Disk and hardware: measure the existing workspace build pool's storage.
runs-on: sm-standard-4
timeout-minutes: 150
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- uses: ./.github/actions/runner-pilot
with:
workload: ${{ inputs.workload }}
source-sha: ${{ inputs.source_sha }}
toolchain: ${{ inputs.toolchain }}
scratch-root: ${{ inputs.scratch_root }}
build-jobs: ${{ inputs.build_jobs }}
local-ssd: ${{ inputs.local_ssd }}
- name: Retain ARC pilot evidence
if: always()
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
with:
name: runner-pilot-arc-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/runner-pilot-evidence
if-no-files-found: error
retention-days: 30
+212 -607
View File
File diff suppressed because it is too large Load Diff
+1 -1
View File
@@ -95,7 +95,7 @@ jobs:
- name: Run CLA Bot
if: github.event_name != 'merge_group'
uses: overtrue/cla-bot@4bd1387c19691168c42fcf5edfa17b15819b2ab3 # v0.1.1
uses: overtrue/cla-bot@5e5929846a60cfb42b16d0d38e135ef67d94fc88 # v0.0.9
with:
github-token: ${{ github.token }}
registry-token: ${{ steps.registry-token.outputs.token }}
+571 -71
View File
@@ -1,99 +1,599 @@
# Copyright 2026 RustFS Team
# SPDX-License-Identifier: Apache-2.0
name: Build candidate images
on:
workflow_call:
inputs:
candidate_manifest:
type: string
required: true
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Docker Images Workflow
#
# This workflow builds Docker images using pre-built binaries from the build workflow.
#
# Trigger Types:
# 1. workflow_run: Automatically triggered when "Build and Release" workflow completes
# 2. workflow_dispatch: Manual trigger for standalone Docker builds
#
# Key Features:
# - Only triggers when Linux builds (x86_64 + aarch64) are successful
# - Independent of macOS/Windows build status
# - Uses workflow_run event for precise control
# - Only builds Docker images for releases and prereleases (development builds are skipped)
name: Docker Images
# Permissions needed for workflow_run event and Docker registry access
permissions:
contents: read
actions: read
packages: write
on:
# Automatically triggered when build workflow completes
workflow_run:
workflows: [ "Build and Release" ]
types: [ completed ]
# Manual trigger with same parameters for consistency
workflow_dispatch:
inputs:
push_images:
description: "Push images to registries"
required: false
default: true
type: boolean
version:
description: "Version to build (latest, v1.0.0, v1.0.0-alpha1, or v1.0.1-preview.1)"
required: false
default: "latest"
type: string
force_rebuild:
description: "Force rebuild even if binary exists (useful for testing)"
required: false
default: false
type: boolean
env:
CONCLUSION: ${{ github.event.workflow_run.conclusion }}
HEAD_BRANCH: ${{ github.event.workflow_run.head_branch }}
HEAD_SHA: ${{ github.event.workflow_run.head_sha }}
TRIGGERING_EVENT: ${{ github.event.workflow_run.event }}
DOCKERHUB_USERNAME: rustfs
CARGO_TERM_COLOR: always
REGISTRY_DOCKERHUB: rustfs/rustfs
REGISTRY_GHCR: ghcr.io/${{ github.repository }}
REGISTRY_QUAY: quay.io/rustfs/rustfs
DOCKER_PLATFORMS: linux/amd64,linux/arm64
jobs:
build-docker:
# Check if we should build Docker images.
# Short-circuit at job level so per-merge development builds (workflow_run
# from a push to main) skip the whole run without spawning a checkout job:
# images are only published for tag builds — build.yml push triggers cover
# only main and tags, so a non-main head_branch is a tag name — and for
# manual dispatch.
build-check:
name: Docker Build Check
if: >-
github.event_name == 'workflow_dispatch' ||
(github.event.workflow_run.conclusion == 'success' &&
github.event.workflow_run.event == 'push' &&
github.event.workflow_run.head_branch != 'main')
runs-on: ubuntu-latest
timeout-minutes: 90
timeout-minutes: 30
outputs:
should_build: ${{ steps.check.outputs.should_build }}
should_push: ${{ steps.check.outputs.should_push }}
build_type: ${{ steps.check.outputs.build_type }}
version: ${{ steps.check.outputs.version }}
short_sha: ${{ steps.check.outputs.short_sha }}
is_prerelease: ${{ steps.check.outputs.is_prerelease }}
create_latest: ${{ steps.check.outputs.create_latest }}
source_ref: ${{ steps.check.outputs.source_ref }}
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
# For workflow_run events, checkout the specific commit that triggered the workflow
ref: ${{ github.event.workflow_run.head_sha || github.sha }}
- name: Check build conditions
id: check
env:
# dispatch inputs via env, not `${{ }}` interpolation: they are
# free-form strings and would otherwise be evaluated by bash.
INPUT_VERSION: ${{ github.event.inputs.version }}
INPUT_PUSH_IMAGES: ${{ github.event.inputs.push_images }}
INPUT_FORCE_REBUILD: ${{ github.event.inputs.force_rebuild }}
run: |
should_build=false
should_push=false
build_type="none"
version=""
short_sha=""
is_prerelease=false
create_latest=false
source_ref="$GITHUB_SHA"
# Pre-GA policy: until the first stable (vX.Y.Z) tag exists, every
# prerelease (alpha/beta/rc) also moves `latest`, so users pulling
# `latest` get the newest test build. Once a stable tag is published
# this returns false and `latest` follows stable releases only.
prerelease_moves_latest() {
local stable_tags
stable_tags=$(git ls-remote --tags --refs origin 2>/dev/null \
| awk '{print $2}' \
| grep -E '^refs/tags/v?[0-9]+\.[0-9]+\.[0-9]+$' || true)
if [[ -z "$stable_tags" ]]; then
return 0
fi
echo "ℹ️ Stable release tag(s) already exist; prereleases no longer update latest"
return 1
}
if [[ "${{ github.event_name }}" == "workflow_run" ]]; then
# Triggered by build workflow completion
echo "🔗 Triggered by build workflow completion"
# Check if the triggering workflow was successful
# If the workflow succeeded, it means ALL builds (including Linux x86_64 and aarch64) succeeded
if [[ "$CONCLUSION" == "success" ]]; then
echo "✅ Build workflow succeeded, all builds including Linux are successful"
should_build=true
should_push=true
else
echo "❌ Build workflow failed (conclusion: $CONCLUSION), skipping Docker build"
should_build=false
fi
# Extract version info from commit message or use commit SHA
# Use Git to generate consistent short SHA (ensures uniqueness like build.yml)
short_sha=$(git rev-parse --short "$HEAD_SHA")
source_ref="$HEAD_SHA"
# Determine build type based on triggering workflow event and ref
triggering_event="$TRIGGERING_EVENT"
head_branch="$HEAD_BRANCH"
echo "🔍 Analyzing triggering workflow:"
echo " 📋 Event: $triggering_event"
echo " 🌿 Head branch: $head_branch"
echo " 📎 Head SHA: $HEAD_SHA"
# Check if this was triggered by a tag push
if [[ "$triggering_event" == "push" ]]; then
# For tag pushes, head_branch will be like "refs/tags/v1.0.0" or just "v1.0.0"
if [[ "$head_branch" == refs/tags/* ]]; then
# Extract tag name from refs/tags/TAG_NAME
tag_name="${head_branch#refs/tags/}"
version="$tag_name"
elif [[ "$head_branch" =~ ^v?[0-9]+\.[0-9]+\.[0-9]+ ]]; then
# Direct tag name like "v1.0.0" or "1.0.0-alpha.1"
version="$head_branch"
elif [[ "$head_branch" == "main" ]]; then
# Regular branch push to main
build_type="development"
version="dev-${short_sha}"
should_build=false
echo "⏭️ Skipping Docker build for development version (main branch push)"
else
# Other branch push
build_type="development"
version="dev-${short_sha}"
should_build=false
echo "⏭️ Skipping Docker build for development version (branch: $head_branch)"
fi
# If we extracted a version (tag), determine release type
if [[ -n "$version" ]] && [[ "$version" != "dev-${short_sha}" ]]; then
# Remove 'v' prefix if present for consistent version format
if [[ "$version" == v* ]]; then
version="${version#v}"
fi
if [[ "$version" =~ -preview\.[0-9]+$ ]]; then
build_type="preview"
is_prerelease=true
echo "🔍 Building Docker image for preview: $version (version tags only)"
elif [[ "$version" == *"-preview"* ]]; then
echo "❌ Invalid preview tag: $version (expected suffix: -preview.<number>)" >&2
exit 1
elif [[ "$version" == *"alpha"* ]] || [[ "$version" == *"beta"* ]] || [[ "$version" == *"rc"* ]]; then
build_type="prerelease"
is_prerelease=true
# Pre-GA policy: prereleases update latest until the first stable tag exists.
if prerelease_moves_latest; then
create_latest=true
echo "🧪 Building Docker image for prerelease: $version (creating latest tag)"
else
echo "🧪 Building Docker image for prerelease: $version"
fi
else
build_type="release"
create_latest=true
echo "🚀 Building Docker image for release: $version"
fi
fi
else
# Non-push events
build_type="development"
version="dev-${short_sha}"
should_build=false
echo "⏭️ Skipping Docker build for development version (event: $triggering_event)"
fi
echo "🔄 Build triggered by workflow_run:"
echo " 📋 Conclusion: $CONCLUSION"
echo " 🌿 Branch: $HEAD_BRANCH"
echo " 📎 SHA: $HEAD_SHA"
echo " 🎯 Event: $TRIGGERING_EVENT"
elif [[ "${{ github.event_name }}" == "workflow_dispatch" ]]; then
# Manual trigger
input_version="$INPUT_VERSION"
version="${input_version}"
should_push="$INPUT_PUSH_IMAGES"
should_build=true
# Get short SHA
short_sha=$(git rev-parse --short HEAD)
echo "🎯 Manual Docker build triggered:"
echo " 📋 Requested version: $input_version"
echo " 🔧 Force rebuild: $INPUT_FORCE_REBUILD"
echo " 🚀 Push images: $should_push"
case "$input_version" in
"latest")
build_type="release"
create_latest=true
echo "🚀 Building with latest stable release version"
;;
*-preview*)
if [[ ! "$input_version" =~ ^v?[0-9]+\.[0-9]+\.[0-9]+(-[0-9A-Za-z.-]+)?-preview\.[0-9]+$ ]]; then
echo "❌ Invalid preview version: $input_version" >&2
exit 1
fi
build_type="preview"
is_prerelease=true
version="${input_version#v}"
echo "🔍 Building with preview version: $version (version tags only)"
;;
# Prerelease versions (must match first, more specific)
v*alpha*|v*beta*|v*rc*|*alpha*|*beta*|*rc*)
build_type="prerelease"
is_prerelease=true
# Pre-GA policy: prereleases update latest until the first stable tag exists.
if prerelease_moves_latest; then
create_latest=true
echo "🧪 Building with prerelease version: $input_version (creating latest tag)"
else
echo "🧪 Building with prerelease version: $input_version"
fi
;;
# Release versions (match after prereleases, more general)
v[0-9]*|[0-9]*.*.*)
build_type="release"
create_latest=true
echo "📦 Building with specific release version: $input_version"
;;
*)
# Invalid version for Docker build
should_build=false
echo "❌ Invalid version for Docker build: $input_version"
echo "⚠️ Only release versions (latest, v1.0.0, 1.0.0) and prereleases (v1.0.0-alpha1, 1.0.0-beta2) are supported"
;;
esac
if [[ "$should_build" == true && "$input_version" != "latest" ]]; then
tag_ref="refs/tags/$input_version"
if ! git ls-remote --exit-code origin "$tag_ref" >/dev/null 2>&1; then
if [[ "$input_version" == v* ]]; then
tag_ref="refs/tags/${input_version#v}"
else
tag_ref="refs/tags/v$input_version"
fi
fi
if ! git ls-remote --exit-code origin "$tag_ref" >/dev/null 2>&1; then
echo "❌ Release tag not found for Docker build: $input_version"
exit 1
fi
source_ref="$tag_ref"
fi
fi
{
echo "should_build=$should_build"
echo "should_push=$should_push"
echo "build_type=$build_type"
echo "version=$version"
echo "short_sha=$short_sha"
echo "is_prerelease=$is_prerelease"
echo "create_latest=$create_latest"
echo "source_ref=$source_ref"
} >> "$GITHUB_OUTPUT"
echo "🐳 Docker Build Summary:"
echo " - Should build: $should_build"
echo " - Should push: $should_push"
echo " - Build type: $build_type"
echo " - Version: $version"
echo " - Short SHA: $short_sha"
echo " - Is prerelease: $is_prerelease"
echo " - Create latest: $create_latest"
echo " - Source ref: $source_ref"
# Build multi-arch Docker images
# Strategy: Build images using pre-built binaries from dl.rustfs.com
# Supports both release and dev channel binaries based on build context
# Only runs when should_build is true (which includes workflow success check)
build-docker:
name: Build Docker Images
needs: build-check
if: needs.build-check.outputs.should_build == 'true'
runs-on: dind-sm-standard-2
timeout-minutes: 60
strategy:
fail-fast: false
matrix:
include:
- variant: musl
file: Dockerfile
- variant: gnu
suffix: ""
- variant: glibc
file: Dockerfile.glibc
env:
CANDIDATE_MANIFEST: ${{ inputs.candidate_manifest }}
VARIANT: ${{ matrix.variant }}
suffix: "-glibc"
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
ref: ${{ fromJson(inputs.candidate_manifest).source_sha }}
persist-credentials: false
- name: Reuse completed image
id: reuse
env:
GH_TOKEN: ${{ github.token }}
run: python3 scripts/release/deliver.py reuse --kind image --target "$VARIANT"
- name: Install OCI tools
if: steps.reuse.outputs.reused != 'true'
ref: ${{ needs.build-check.outputs.source_ref }}
- name: Login to Docker Hub
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
with:
username: ${{ env.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Login to GitHub Container Registry
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
with:
registry: ghcr.io
username: ${{ secrets.GHCR_USERNAME }}
password: ${{ secrets.GHCR_PASSWORD }}
- name: Login to Quay.io
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
with:
registry: quay.io
username: ${{ secrets.QUAY_USERNAME }}
password: ${{ secrets.QUAY_PASSWORD }}
- name: Set up QEMU
uses: docker/setup-qemu-action@c7c53464625b32c7a7e944ae62b3e17d2b600130 # v3
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # v3
- name: Extract metadata and generate tags
id: meta
run: |
sudo apt-get update
sudo apt-get install -y skopeo
- uses: docker/setup-qemu-action@c7c53464625b32c7a7e944ae62b3e17d2b600130 # v3
if: steps.reuse.outputs.reused != 'true'
- uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # v3
if: steps.reuse.outputs.reused != 'true'
- name: Supply accepted binaries and pinned CLI
if: steps.reuse.outputs.reused != 'true'
env:
GH_TOKEN: ${{ github.token }}
run: python3 scripts/release/deliver.py image-prepare --target "$VARIANT"
- name: Build OCI archive without registry writes
if: steps.reuse.outputs.reused != 'true'
BUILD_TYPE="${{ needs.build-check.outputs.build_type }}"
VERSION="${{ needs.build-check.outputs.version }}"
CREATE_LATEST="${{ needs.build-check.outputs.create_latest }}"
VARIANT_SUFFIX="${{ matrix.suffix }}"
# Convert version format for Dockerfile compatibility. The former
# DOCKER_CHANNEL was "release" down every branch and was passed as a
# build-arg no Dockerfile declares, so it is gone.
case "$VERSION" in
"latest")
DOCKER_RELEASE="latest"
;;
v*)
# For versioned releases (v1.0.0), remove 'v' prefix for Dockerfile
DOCKER_RELEASE="${VERSION#v}"
;;
*)
# For other versions, pass as-is
DOCKER_RELEASE="${VERSION}"
;;
esac
echo "docker_release=$DOCKER_RELEASE" >> "$GITHUB_OUTPUT"
echo "🐳 Docker build parameters:"
echo " - Original version: $VERSION"
echo " - Docker RELEASE: $DOCKER_RELEASE"
# Generate tags based on build type
# Only support release and prerelease builds (no development builds)
TAG_BASE="${VERSION}${VARIANT_SUFFIX}"
TAGS="${{ env.REGISTRY_DOCKERHUB }}:$TAG_BASE,${{ env.REGISTRY_GHCR }}:$TAG_BASE,${{ env.REGISTRY_QUAY }}:$TAG_BASE"
# Add latest when requested (stable releases, and prereleases before GA)
if [[ "$CREATE_LATEST" == "true" ]]; then
TAGS="$TAGS,${{ env.REGISTRY_DOCKERHUB }}:latest${VARIANT_SUFFIX},${{ env.REGISTRY_GHCR }}:latest${VARIANT_SUFFIX},${{ env.REGISTRY_QUAY }}:latest${VARIANT_SUFFIX}"
fi
# Always add the channel tag for prereleases, independent of latest
if [[ "$BUILD_TYPE" == "prerelease" ]]; then
# Prerelease channel tags (alpha, beta, rc)
if [[ "$VERSION" == *"alpha"* ]]; then
CHANNEL="alpha"
elif [[ "$VERSION" == *"beta"* ]]; then
CHANNEL="beta"
elif [[ "$VERSION" == *"rc"* ]]; then
CHANNEL="rc"
fi
if [[ -n "$CHANNEL" ]]; then
TAGS="$TAGS,${{ env.REGISTRY_DOCKERHUB }}:${CHANNEL}${VARIANT_SUFFIX},${{ env.REGISTRY_GHCR }}:${CHANNEL}${VARIANT_SUFFIX},${{ env.REGISTRY_QUAY }}:${CHANNEL}${VARIANT_SUFFIX}"
fi
fi
# Output tags
echo "tags=$TAGS" >> "$GITHUB_OUTPUT"
# Generate labels
LABELS="org.opencontainers.image.title=RustFS"
LABELS="$LABELS,org.opencontainers.image.description=RustFS distributed object storage system"
LABELS="$LABELS,org.opencontainers.image.version=$VERSION"
SOURCE_REVISION="$(git rev-parse HEAD)"
LABELS="$LABELS,org.opencontainers.image.revision=$SOURCE_REVISION"
LABELS="$LABELS,org.opencontainers.image.source=${{ github.server_url }}/${{ github.repository }}"
LABELS="$LABELS,org.opencontainers.image.created=$(date -u +'%Y-%m-%dT%H:%M:%SZ')"
LABELS="$LABELS,org.opencontainers.image.build-type=$BUILD_TYPE"
echo "labels=$LABELS" >> "$GITHUB_OUTPUT"
echo "🐳 Generated Docker tags:"
echo "$TAGS" | tr ',' '\n' | sed 's/^/ - /'
echo "📋 Build type: $BUILD_TYPE"
echo "🔖 Version: $VERSION"
- name: Build and push Docker image
uses: docker/build-push-action@10e90e3645eae34f1e60eeb005ba3a3d33f178e8 # v6
with:
context: .
file: ${{ matrix.file }}
platforms: linux/amd64,linux/arm64
push: false
provenance: false
outputs: type=oci,dest=delivery/image-${{ matrix.variant }}.tar
platforms: ${{ env.DOCKER_PLATFORMS }}
push: ${{ needs.build-check.outputs.should_push == 'true' }}
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
# No layer cache. This build compiles nothing — it downloads a
# release zip and runs apk/apt — so the cache could only save the
# minute or two those take, while creating a correctness problem: with
# RELEASE=latest the binary URL is resolved by curl *inside* a RUN
# layer, and the layer key does not include what that resolved to. A
# rebuild at the same RELEASE value (dispatch with version=latest, or
# a re-run of the same version) would hit the old layer and ship the
# previous release's binary. mode=max also consumed the same 10GB
# Actions cache quota the Rust lanes are fighting over.
#
# Only RELEASE is passed: it is the sole build-arg the Dockerfiles
# declare besides TARGETARCH. BUILDTIME, VERSION, BUILD_TYPE, REVISION
# and CHANNEL were never read by any stage (and BUILDTIME's $(date ...)
# was a literal here, not a shell substitution). BUILD_DATE and VCS_REF
# are declared by the Dockerfiles but deliberately left unset —
# supplying them would change the published image labels.
build-args: |
BINARY_SOURCE=supplied
RELEASE=${{ fromJson(inputs.candidate_manifest).version }}
VCS_REF=${{ fromJson(inputs.candidate_manifest).source_sha }}
- name: Verify wrapped binaries and startup on both architectures
if: steps.reuse.outputs.reused != 'true'
run: python3 scripts/release/deliver.py image-smoke --target "$VARIANT"
- name: Scan amd64 image before sealing
if: steps.reuse.outputs.reused != 'true'
uses: aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25
env:
TRIVY_PLATFORM: linux/amd64
RELEASE=${{ steps.meta.outputs.docker_release }}
provenance: true
sbom: true
# Add retry mechanism by splitting the build process
no-cache: false
pull: true
# Note: Manifest creation is no longer needed as we only build one variant
# Multi-arch manifests are automatically created by docker/build-push-action
# Report-only container image vulnerability scan
scan-docker-image:
name: Scan Docker Images
needs: [ build-check, build-docker ]
if: needs.build-check.outputs.should_build == 'true' && needs.build-check.outputs.should_push == 'true'
runs-on: ubuntu-latest
timeout-minutes: 30
permissions:
contents: read
security-events: write
strategy:
fail-fast: false
matrix:
include:
- variant: musl
suffix: ""
- variant: glibc
suffix: "-glibc"
steps:
- name: Login to GitHub Container Registry
# docker/login-action v3
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9
with:
input: delivery-scan/image-${{ matrix.variant }}-amd64.tar
format: sarif
output: delivery/scan-${{ matrix.variant }}-amd64.sarif
exit-code: "0" # Keep the repository's report-only vulnerability policy.
- name: Scan arm64 image before sealing
if: steps.reuse.outputs.reused != 'true'
registry: ghcr.io
username: ${{ secrets.GHCR_USERNAME }}
password: ${{ secrets.GHCR_PASSWORD }}
- name: Scan image with Trivy
# aquasecurity/trivy-action v0.36.0
uses: aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25
env:
TRIVY_PLATFORM: linux/arm64
with:
input: delivery-scan/image-${{ matrix.variant }}-arm64.tar
image-ref: ${{ env.REGISTRY_GHCR }}:${{ needs.build-check.outputs.version }}${{ matrix.suffix }}
format: sarif
output: delivery/scan-${{ matrix.variant }}-arm64.sarif
output: trivy-${{ matrix.variant }}.sarif
ignore-unfixed: true
vuln-type: os,library
severity: CRITICAL,HIGH
exit-code: "0"
- name: Record verified OCI identity
if: steps.reuse.outputs.reused != 'true'
run: python3 scripts/release/deliver.py image-record --target "$VARIANT"
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
if: steps.reuse.outputs.reused != 'true'
# Surface findings in the Security tab; an artifact alone is write-only
# reporting nobody reviews.
- name: Upload scan results to code scanning
uses: github/codeql-action/upload-sarif@99df26d4f13ea111d4ec1a7dddef6063f76b97e9 # v4.37.0
with:
name: image-${{ matrix.variant }}
path: delivery/*
sarif_file: trivy-${{ matrix.variant }}.sarif
category: container-image-${{ matrix.variant }}
- name: Upload container scan report
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: container-image-scan-${{ matrix.variant }}
path: trivy-${{ matrix.variant }}.sarif
if-no-files-found: error
retention-days: 30
compression-level: 0
# Docker build summary
docker-summary:
name: Docker Build Summary
needs: [ build-check, build-docker ]
if: always() && needs.build-check.outputs.should_build == 'true'
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- name: Docker build completion summary
run: |
BUILD_TYPE="${{ needs.build-check.outputs.build_type }}"
VERSION="${{ needs.build-check.outputs.version }}"
CREATE_LATEST="${{ needs.build-check.outputs.create_latest }}"
echo "🐳 Docker build completed successfully!"
echo "📦 Build type: $BUILD_TYPE"
echo "🔢 Version: $VERSION"
echo "🚀 Strategy: Images using pre-built binaries (release channel only)"
echo ""
case "$BUILD_TYPE" in
"preview")
echo "🔍 Preview Docker image has been built with ${VERSION} tags"
echo "⏭️ Preview images do not update latest or prerelease channels"
;;
"release")
echo "🚀 Release Docker image has been built with ${VERSION} tags"
echo "✅ This image is ready for production use"
if [[ "$CREATE_LATEST" == "true" ]]; then
echo "🏷️ Latest tag has been created for stable release"
fi
;;
"prerelease")
echo "🧪 Prerelease Docker image has been built with ${VERSION} tags"
echo "⚠️ This is a prerelease image - use with caution"
# Prereleases move latest until the first stable tag exists (pre-GA policy).
if [[ "$CREATE_LATEST" == "true" ]]; then
echo "🏷️ Latest tag has been created for prerelease: $VERSION"
else
echo "🚫 Latest tag NOT created for prerelease"
fi
;;
*)
echo "❌ Unexpected build type: $BUILD_TYPE"
;;
esac
-242
View File
@@ -1,242 +0,0 @@
# Copyright 2026 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Nightly both-stack diff of the two e2e profiles only ci.yml runs, e2e-smoke
# and e2e-full (rustfs/backlog#2772, T3.1). The other profiles carry a `stack`
# dimension in their own workflows, where the existing lane is the legacy leg.
#
# One build serves every leg. One preflight starts that binary with
# RUSTFS_S3_STACK=gateway (scripts/dual_stack.py preflight); without a proven
# gateway start the gateway legs do not run and the summary says `skipped:
# gateway stack refuses startup: <why>`. Which stack a leg ran is read from the
# servers' own start-up line. The gateway legs are report-only until T2.10.
#
# Left out of both stacks: `dual_stack_differential::` (servers with an explicit
# stack), `fake_s3_target::`, the upgrade write-readiness cases and one deadline
# case (no server), and `distributed_startup_regression_test::` (needs ci.yml's
# hooks-binary manifest). Both e2e-full stacks use ci.yml's pinned mc and
# Vault binaries and network fault-injection tools.
name: dual-stack-nightly
on:
workflow_dispatch:
schedule:
# 03:13 UTC nightly, clear of e2e-nightly (04:29), the ODM lane (05:23) and
# e2e-distributed (05:53).
- cron: "13 3 * * *"
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: ${{ github.event_name != 'schedule' }}
permissions:
contents: read
defaults:
run:
shell: bash
jobs:
build:
name: Build once and probe the gateway stack
if: github.repository == 'rustfs/rustfs'
# The runner and cache of ci.yml's build-rustfs-debug-binary.
runs-on: sm-standard-4
timeout-minutes: 60
outputs:
stacks: ${{ steps.preflight.outputs.stacks }}
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: stable
cache-shared-key: ci-dev
cache-save-if: 'false'
install-build-packaging-tools: 'false'
- name: Build the rustfs binary
run: python3 scripts/e2e_binary.py build --bins --features e2e-test-hooks
- name: Upload the binary
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: dual-stack-binary
path: |
target/debug/rustfs
target/debug/rustfs.e2e.json
if-no-files-found: error
retention-days: 1
# Without a proven gateway start there is no gateway leg to run.
- name: Gateway preflight
id: preflight
run: |
if python3 scripts/dual_stack.py preflight --binary target/debug/rustfs --out "$RUNNER_TEMP/preflight/stack.json"; then
echo 'stacks=["legacy","gateway"]' >> "$GITHUB_OUTPUT"
else
echo 'stacks=["legacy"]' >> "$GITHUB_OUTPUT"
fi
# The evidence stands in for a gateway leg that is skipped; a leg that runs overwrites it.
- name: Upload the preflight evidence for e2e-smoke
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: dual-stack-leg-e2e-smoke-gateway
path: ${{ runner.temp }}/preflight
if-no-files-found: warn
retention-days: 7
- name: Upload the preflight evidence for e2e-full
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: dual-stack-leg-e2e-full-gateway
path: ${{ runner.temp }}/preflight
if-no-files-found: warn
retention-days: 7
leg:
name: ${{ matrix.profile }} (${{ matrix.stack }}${{ matrix.stack == 'gateway' && ', report-only until T2.10' || '' }})
needs: [ build ]
runs-on: sm-standard-4
timeout-minutes: 120
# Report-only until T2.10: the gateway stack cannot serve the suites yet.
continue-on-error: ${{ matrix.stack == 'gateway' }}
strategy:
fail-fast: false
matrix:
profile: [ e2e-smoke, e2e-full ]
stack: ${{ fromJSON(needs.build.outputs.stacks) }}
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: stable
cache-shared-key: ci-dev
cache-save-if: 'false'
install-build-packaging-tools: 'false'
- name: Download the binary
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: dual-stack-binary
path: target/debug
- name: Set up Python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.12"
- name: Install awscurl
run: |
python3 -m pip install --user --upgrade pip "awscurl==0.44"
echo "AWSCURL_PATH=$HOME/.local/bin/awscurl" >> "$GITHUB_ENV"
test -x "$HOME/.local/bin/awscurl"
- name: Install network fault-injection tools
if: matrix.profile == 'e2e-full'
run: |
sudo apt-get install -y iptables
sudo -n iptables --version
# The endpoint-blackhole heal scenario needs CAP_NET_ADMIN. Containerised
# runners can run iptables but not touch the rule set; the test then logs
# a skip instead of failing, so surface that here where it is visible.
if ! sudo -n iptables -w 5 -S OUTPUT >/dev/null 2>&1; then
echo "::warning::iptables cannot read the OUTPUT chain on this runner (no CAP_NET_ADMIN); the endpoint-blackhole heal scenario will be skipped"
fi
- name: Install mc
if: matrix.profile == 'e2e-full'
env:
GH_TOKEN: ${{ github.token }}
MC_VERSION: RELEASE.2025-08-13T08-35-41Z
MC_SHA256: 01f866e9c5f9b87c2b09116fa5d7c06695b106242d829a8bb32990c00312e891
run: |
if ! command -v gh >/dev/null 2>&1; then
sudo apt-get update
sudo apt-get install -y gh
fi
MC_BINARY="mc.linux-amd64.${MC_VERSION}"
python3 .github/actions/setup/download_release_asset.py --repository minio/mc --tag "$MC_VERSION" --asset "$MC_BINARY" --output "$RUNNER_TEMP/mc" --sha256 "$MC_SHA256"
echo "${MC_SHA256} $RUNNER_TEMP/mc" | sha256sum --check --status
chmod +x "$RUNNER_TEMP/mc"
echo "$RUNNER_TEMP" >> "$GITHUB_PATH"
- name: Verify mc
if: matrix.profile == 'e2e-full'
run: mc --version
- name: Install Vault
if: matrix.profile == 'e2e-full'
run: |
VAULT_VERSION="1.17.6"
VAULT_ARCHIVE="vault_${VAULT_VERSION}_linux_amd64.zip"
curl -fsSLo "$RUNNER_TEMP/$VAULT_ARCHIVE" "https://releases.hashicorp.com/vault/${VAULT_VERSION}/${VAULT_ARCHIVE}"
echo "0cddc1fbbb88583b5ba5b845f9f8fae47c6fb39a6d48cd543c6ba6fd3ac1a669 $RUNNER_TEMP/$VAULT_ARCHIVE" | sha256sum --check --status
unzip -q "$RUNNER_TEMP/$VAULT_ARCHIVE" -d "$RUNNER_TEMP/vault-bin"
echo "RUSTFS_TEST_VAULT_BIN=$RUNNER_TEMP/vault-bin/vault" >> "$GITHUB_ENV"
- name: Verify Vault
if: matrix.profile == 'e2e-full'
run: |
"$RUSTFS_TEST_VAULT_BIN" version
- name: Run the profile on the ${{ matrix.stack }} stack
env:
PROFILE: ${{ matrix.profile }}
RUSTFS_S3_STACK: ${{ matrix.stack }}
RUSTFS_E2E_LOG_DIR: ${{ runner.temp }}/server-logs
run: |
chmod +x target/debug/rustfs
filter='not test(/^(dual_stack_differential|fake_s3_target|distributed_startup_regression_test)::/)'
filter="$filter & not test(/^upgrade_compatibility_test::upgrade_write_readiness_tests::/)"
filter="$filter & not test(=list_objects_v2_pagination_test::tests::test_list_objects_v2_deadline_rejects_unresponsive_peer)"
python3 scripts/e2e_binary.py run --features e2e-test-hooks -- cargo nextest run --profile "$PROFILE" -p e2e_test -E "$filter"
- name: Prove the stack and upload the leg
if: always()
uses: ./.github/actions/dual-stack-leg
with: { profile: "${{ matrix.profile }}", stack: "${{ matrix.stack }}", logs: "${{ runner.temp }}/server-logs" }
summary:
name: Both-stack diff summary
needs: [ build, leg ]
if: github.repository == 'rustfs/rustfs' && always()
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- uses: ./.github/actions/dual-stack-summary
with:
profiles: e2e-smoke,e2e-full
+30 -35
View File
@@ -17,8 +17,8 @@
# Each selected test starts a real localhost cluster via
# `RustFSTestClusterEnvironment` (4 processes; 4 drives per node unless the
# case is a two-site 4-node 1-drive pair or a 4-node upgrade). Membership is
# `[profile.e2e-distributed]` in `.config/nextest.toml`. The nightly schedule
# and manual dispatches execute the same fail-closed suite.
# `[profile.e2e-distributed]` in `.config/nextest.toml`. Storage-sensitive PRs,
# nightly runs, and manual dispatches all execute the same fail-closed suite.
# Upgrade cases download the same pinned previous release as e2e-upgrade.yml.
#
# Isolated pool filesystems: expand/decommission/rebalance cases require
@@ -33,6 +33,32 @@
name: e2e-distributed
on:
pull_request:
paths:
- "Cargo.lock"
- "Cargo.toml"
- ".config/nextest.toml"
- ".config/e2e-distributed-selection.txt"
- ".github/workflows/e2e-distributed.yml"
- "crates/audit/**"
- "crates/common/**"
- "crates/config/**"
- "crates/e2e_test/**"
- "crates/ecstore/**"
- "crates/filemeta/**"
- "crates/heal/**"
- "crates/iam/**"
- "crates/lock/**"
- "crates/madmin/**"
- "crates/notify/**"
- "crates/replication/**"
- "crates/s3-client/**"
- "crates/s3-ops/**"
- "crates/s3-types/**"
- "crates/scanner/**"
- "crates/storage-api/**"
- "crates/utils/**"
- "rustfs/**"
workflow_dispatch:
inputs:
filter:
@@ -53,14 +79,11 @@ concurrency:
jobs:
distributed:
if: github.repository == 'rustfs/rustfs'
name: Distributed 4-node 4-disk e2e (${{ matrix.stack }}${{ matrix.stack == 'gateway' && ', report-only until T2.10' || '' }})
name: Distributed 4-node 4-disk e2e
# GitHub-hosted VM: loop and tmpfs mounts work here. sm-standard-4 is an
# ARC pod and rejects both (`mount -o loop` ENOENT, tmpfs "read-only").
runs-on: ubuntu-latest
timeout-minutes: 180
# The gateway leg is report-only until T2.10.
continue-on-error: ${{ matrix.stack == 'gateway' }}
strategy: { fail-fast: false, matrix: { stack: [legacy, gateway] } }
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
NO_PROXY: 127.0.0.1,localhost
@@ -132,11 +155,6 @@ jobs:
run: |
python3 scripts/e2e_binary.py build --bins --features e2e-test-hooks
# A gateway binary that refuses to start skips the suite; the summary says why.
- name: Gateway preflight
if: matrix.stack == 'gateway'
run: python3 scripts/dual_stack.py preflight --binary target/debug/rustfs --out "$RUNNER_TEMP/leg/stack.json"
- name: Verify distributed e2e membership
env:
NEXTEST_LISTING: ${{ runner.temp }}/rustfs-e2e-distributed-list.json
@@ -148,7 +166,6 @@ jobs:
env:
RUSTFS_E2E_LOG_DIR: ${{ runner.temp }}/rustfs-e2e-distributed-logs
FILTER: ${{ inputs.filter }}
RUSTFS_S3_STACK: ${{ matrix.stack }}
run: |
set -euo pipefail
rm -f target/nextest/e2e-distributed/junit.xml
@@ -159,19 +176,11 @@ jobs:
fi
test -s target/nextest/e2e-distributed/junit.xml
- name: Prove the stack and upload the leg
if: always()
uses: ./.github/actions/dual-stack-leg
with:
profile: e2e-distributed
stack: ${{ matrix.stack }}
logs: ${{ runner.temp }}/rustfs-e2e-distributed-logs
- name: Upload distributed e2e diagnostics
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: e2e-distributed-${{ github.run_number }}-${{ matrix.stack }}
name: e2e-distributed-${{ github.run_number }}
path: |
target/nextest/e2e-distributed/junit.xml
${{ runner.temp }}/rustfs-e2e-distributed-list.json
@@ -191,20 +200,6 @@ jobs:
fi
done
dual-stack-summary:
name: Both-stack diff summary
needs: [distributed]
if: github.repository == 'rustfs/rustfs' && always()
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- uses: ./.github/actions/dual-stack-summary
with:
profiles: e2e-distributed
alert-on-failure:
name: Alert on scheduled failure
needs: [distributed]
+11 -82
View File
@@ -54,18 +54,14 @@ concurrency:
jobs:
repl-nightly:
if: github.repository == 'rustfs/rustfs'
name: Replication e2e (nightly, ${{ matrix.stack }}${{ matrix.stack == 'gateway' && ', report-only until T2.10' || '' }})
name: Replication e2e (nightly)
# dual-node tests spawn two full rustfs servers each; the extra headroom
# over the per-PR sm-standard-2 e2e job keeps the parallel process fan-out
# from starving the runner. Mirrors the ILM serial lane's runner class.
runs-on: sm-standard-4
timeout-minutes: 45
# The gateway leg is report-only until T2.10.
continue-on-error: ${{ matrix.stack == 'gateway' }}
strategy: { fail-fast: false, matrix: { stack: [legacy, gateway] } }
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
RUSTFS_BUILD_FEATURES: ${{ matrix.stack == 'gateway' && 'gateway-stack' || '' }}
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
@@ -97,12 +93,7 @@ jobs:
# Build once and carry its source/binary identity into the test invocation.
- name: Build rustfs binary
run: |
python3 scripts/e2e_binary.py build --bins --features "$RUSTFS_BUILD_FEATURES"
# A gateway binary that refuses to start skips the suite; the summary says why.
- name: Gateway preflight
if: matrix.stack == 'gateway'
run: python3 scripts/dual_stack.py preflight --binary target/debug/rustfs --out "$RUNNER_TEMP/leg/stack.json"
python3 scripts/e2e_binary.py build --bins
- name: Verify replication e2e membership
env:
@@ -114,22 +105,13 @@ jobs:
- name: Run replication e2e nightly suite
env:
RUSTFS_E2E_LOG_DIR: ${{ runner.temp }}/rustfs-e2e-repl-nightly-logs
RUSTFS_S3_STACK: ${{ matrix.stack }}
run: python3 scripts/e2e_binary.py run --features "$RUSTFS_BUILD_FEATURES" -- cargo nextest run --profile e2e-repl-nightly -p e2e_test
- name: Prove the stack and upload the leg
if: always()
uses: ./.github/actions/dual-stack-leg
with:
profile: e2e-repl-nightly
stack: ${{ matrix.stack }}
logs: ${{ runner.temp }}/rustfs-e2e-repl-nightly-logs
run: python3 scripts/e2e_binary.py run -- cargo nextest run --profile e2e-repl-nightly -p e2e_test
- name: Upload nextest junit report
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: e2e-replication-nightly-junit-${{ github.run_number }}-${{ matrix.stack }}
name: e2e-replication-nightly-junit-${{ github.run_number }}
path: |
target/nextest/e2e-repl-nightly/junit.xml
${{ runner.temp }}/rustfs-e2e-repl-nightly-list.json
@@ -139,12 +121,9 @@ jobs:
cluster-nightly:
if: github.repository == 'rustfs/rustfs'
name: Cluster fault e2e (nightly, ${{ matrix.stack }}${{ matrix.stack == 'gateway' && ', report-only until T2.10' || '' }})
name: Cluster fault e2e (nightly)
runs-on: sm-standard-4
timeout-minutes: 90
# The gateway leg is report-only until T2.10.
continue-on-error: ${{ matrix.stack == 'gateway' }}
strategy: { fail-fast: false, matrix: { stack: [legacy, gateway] } }
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
@@ -165,11 +144,6 @@ jobs:
run: |
python3 scripts/e2e_binary.py build --bins --features e2e-test-hooks
# A gateway binary that refuses to start skips the suite; the summary says why.
- name: Gateway preflight
if: matrix.stack == 'gateway'
run: python3 scripts/dual_stack.py preflight --binary target/debug/rustfs --out "$RUNNER_TEMP/leg/stack.json"
- name: Verify cluster fault e2e membership
env:
NEXTEST_LISTING: ${{ runner.temp }}/rustfs-e2e-nightly-list.json
@@ -181,22 +155,13 @@ jobs:
env:
CARGO_BIN_EXE_rustfs: ${{ github.workspace }}/target/debug/rustfs
RUSTFS_E2E_LOG_DIR: ${{ runner.temp }}/rustfs-e2e-nightly-logs
RUSTFS_S3_STACK: ${{ matrix.stack }}
run: python3 scripts/e2e_binary.py run --features e2e-test-hooks -- cargo nextest run --profile e2e-nightly -p e2e_test
- name: Prove the stack and upload the leg
if: always()
uses: ./.github/actions/dual-stack-leg
with:
profile: e2e-nightly
stack: ${{ matrix.stack }}
logs: ${{ runner.temp }}/rustfs-e2e-nightly-logs
- name: Upload cluster fault diagnostics
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: e2e-cluster-nightly-${{ github.run_number }}-${{ matrix.stack }}
name: e2e-cluster-nightly-${{ github.run_number }}
path: |
target/nextest/e2e-nightly/junit.xml
${{ runner.temp }}/rustfs-e2e-nightly-list.json
@@ -206,15 +171,12 @@ jobs:
protocols-nightly:
if: github.repository == 'rustfs/rustfs'
name: Protocol e2e (nightly, ${{ matrix.stack }}${{ matrix.stack == 'gateway' && ', report-only until T2.10' || '' }})
name: Protocol e2e (nightly)
runs-on: sm-standard-4
timeout-minutes: 90
# The gateway leg is report-only until T2.10.
continue-on-error: ${{ matrix.stack == 'gateway' }}
strategy: { fail-fast: false, matrix: { stack: [legacy, gateway] } }
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
RUSTFS_BUILD_FEATURES: ${{ matrix.stack == 'gateway' && 'ftps,webdav,sftp,gateway-stack' || 'ftps,webdav,sftp' }}
RUSTFS_BUILD_FEATURES: ftps,webdav,sftp
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
@@ -238,11 +200,6 @@ jobs:
- name: Build protocol server
run: python3 scripts/e2e_binary.py build --features "$RUSTFS_BUILD_FEATURES"
# A gateway binary that refuses to start skips the suite; the summary says why.
- name: Gateway preflight
if: matrix.stack == 'gateway'
run: python3 scripts/dual_stack.py preflight --binary target/debug/rustfs --out "$RUNNER_TEMP/leg/stack.json"
# The suite owns fixed protocol ports and serializes its internal cases.
- name: Verify protocol e2e membership
env:
@@ -254,28 +211,14 @@ jobs:
- name: Run protocol e2e nightly suite
env:
RUSTFS_E2E_LOG_DIR: ${{ runner.temp }}/rustfs-protocol-e2e-logs
RUSTFS_S3_STACK: ${{ matrix.stack }}
# These servers set no log level of their own; the stack proof reads their start-up line.
RUST_LOG: rustfs=info
# The servers inherit nextest's stdout (--no-capture): keep it where the stack proof reads it.
run: |
set -o pipefail
mkdir -p "$RUSTFS_E2E_LOG_DIR"
python3 scripts/e2e_binary.py run --features "$RUSTFS_BUILD_FEATURES" -- cargo nextest run -j 1 --profile e2e-protocols -p e2e_test --no-capture 2>&1 | tee "$RUSTFS_E2E_LOG_DIR/nextest.log"
- name: Prove the stack and upload the leg
if: always()
uses: ./.github/actions/dual-stack-leg
with:
profile: e2e-protocols
stack: ${{ matrix.stack }}
logs: ${{ runner.temp }}/rustfs-protocol-e2e-logs
run: >-
python3 scripts/e2e_binary.py run --features "$RUSTFS_BUILD_FEATURES" -- cargo nextest run -j 1 --profile e2e-protocols -p e2e_test --no-capture
- name: Upload protocol diagnostics
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: e2e-protocol-nightly-${{ github.run_number }}-${{ matrix.stack }}
name: e2e-protocol-nightly-${{ github.run_number }}
path: |
target/nextest/e2e-protocols/junit.xml
${{ runner.temp }}/rustfs-e2e-protocols-list.json
@@ -283,20 +226,6 @@ jobs:
retention-days: 7
if-no-files-found: warn
dual-stack-summary:
name: Both-stack diff summary
needs: [repl-nightly, cluster-nightly, protocols-nightly]
if: github.repository == 'rustfs/rustfs' && always()
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- uses: ./.github/actions/dual-stack-summary
with:
profiles: e2e-repl-nightly,e2e-nightly,e2e-protocols
alert-on-failure:
name: Alert on scheduled failure
needs: [repl-nightly, cluster-nightly, protocols-nightly]
+18 -7
View File
@@ -12,19 +12,30 @@
# See the License for the specific language governing permissions and
# limitations under the License.
# Single-node upgrade and rollback lane.
#
# The weekly Monday schedule, version tags, and workflow_dispatch run this
# suite. Distributed upgrade cases stay on the nightly e2e-distributed lane.
name: Upgrade Compatibility
on:
pull_request:
paths:
- ".github/workflows/e2e-upgrade.yml"
- "crates/e2e_test/src/common.rs"
- "crates/e2e_test/src/distributed/**"
- "crates/e2e_test/src/fake_s3_target/**"
- "crates/e2e_test/src/lib.rs"
- "crates/e2e_test/src/replication_extension_test.rs"
- "crates/e2e_test/src/rpc_fault_proxy.rs"
- "crates/e2e_test/src/upgrade_compatibility_test.rs"
- "crates/ecstore/**"
- "crates/filemeta/**"
- "crates/heal/**"
- "crates/kms/**"
- "crates/storage-api/**"
- "rustfs/**"
- "Cargo.lock"
push:
tags:
- "[0-9]*.[0-9]*.[0-9]*"
schedule:
# 03:17 UTC every Monday.
- cron: "17 3 * * 1"
workflow_dispatch:
@@ -33,7 +44,7 @@ permissions:
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: false
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
env:
CARGO_TERM_COLOR: always
+33 -2
View File
@@ -12,13 +12,44 @@ concurrency:
jobs:
collect:
if: github.repository == 'rustfs/rustfs'
# Health remains available when either laboratory fleet is offline.
runs-on: ubuntu-latest
runs-on: smoke-testing
timeout-minutes: 15
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Ensure GitHub CLI
shell: bash
run: |
if command -v gh >/dev/null 2>&1; then
gh --version
exit 0
fi
gh_version="2.101.0"
case "$(uname -m)" in
x86_64) gh_arch="amd64" ;;
aarch64|arm64) gh_arch="arm64" ;;
*)
echo "Unsupported Linux architecture for GitHub CLI: $(uname -m)" >&2
exit 1
;;
esac
gh_install_root="${RUNNER_TEMP:-$PWD}/gh-cli"
rm -rf "$gh_install_root"
mkdir -p "$gh_install_root/bin"
archive="$gh_install_root/gh.tar.gz"
curl --fail --location --retry 3 \
"https://github.com/cli/cli/releases/download/v${gh_version}/gh_${gh_version}_linux_${gh_arch}.tar.gz" \
--output "$archive"
tar -xzf "$archive" -C "$gh_install_root"
install -m 0755 \
"$gh_install_root/gh_${gh_version}_linux_${gh_arch}/bin/gh" \
"$gh_install_root/bin/gh"
echo "$gh_install_root/bin" >> "$GITHUB_PATH"
export PATH="$gh_install_root/bin:$PATH"
gh --version
- name: Collect and publish verified chain health
env:
GH_TOKEN: ${{ secrets.PF_TESTING_GH_TOKEN }}
-395
View File
@@ -1,395 +0,0 @@
# Copyright 2026 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Gateway black-box acceptance lanes (rustfs/backlog#2775 T3.4, #2776 T3.5).
#
# Runs two suites of the rustfs/gateway repository against the candidate binary,
# once per S3 stack (RUSTFS_S3_STACK=legacy and =gateway), one server per stack:
#
# - the conformance corpus, `--profile rustfs`. That profile describes the
# gateway edge, so scripts/check_conformance_rulings.py judges the gateway
# stack against .config/conformance-rulings.toml (red: an unruled failure, a
# ruled case failing differently, a stale ruling, a skip on the server's own
# answer) and publishes the legacy run as a reference that is never judged:
# it only has to serve, stay up and complete;
# - the client matrix, ci/compat/run_matrix.sh --external, judged per stack by
# scripts/compat_matrix_report.py (no `fail`, no `sut-unregistered`, every
# client ran; a client that could not be installed is reported as skipped).
# The legacy matrix is the baseline: every cell that passes on legacy must
# pass on gateway (`compare`).
#
# The suites are checked out at the rustfs/gateway rev Cargo.toml pins, so the
# suite and the framework under test never drift apart. The verdict of each
# stack is computed from what the run left on disk (startup record, the stack
# the server's own log names, runner exit code, liveness after the corpus,
# report), never from the exit status of an earlier step. Needs
# scripts/dual_stack.py and the `s3_stack` startup field of rustfs/rustfs#9367.
#
# The gateway stack is report-only until T2.10: its job may fail without failing
# the workflow, and its summary says what it observed (did not start, or how many
# cases ran, passed, failed and were answered 501). The stack comparison is
# report-only for the same reason; its conformance half is information only.
#
# Not a pull-request gate. Scheduled failures are owned by the scheduled-validation
# watchdog (.github/scheduled-validations.json).
name: Gateway Acceptance
on:
schedule:
- cron: "29 3 * * *"
workflow_dispatch:
# Matches ci.yml: these variables are hashed into the rust-cache key, so the
# candidate build reads the same ci-dev cache as build-rustfs-debug-binary.
env:
CARGO_TERM_COLOR: always
RUST_BACKTRACE: 1
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
permissions:
contents: read
defaults:
run:
shell: bash
jobs:
plan:
name: Read the gateway pin
runs-on: ubuntu-latest
timeout-minutes: 5
outputs:
gateway-rev: ${{ steps.pin.outputs.rev }}
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Read the rustfs/gateway rev from Cargo.toml
id: pin
run: |
set -euo pipefail
python3 - <<'PY' >> "$GITHUB_OUTPUT"
import re, sys, tomllib
dependencies = tomllib.load(open("Cargo.toml", "rb"))["workspace"]["dependencies"]
revs = {spec.get("rev") for spec in dependencies.values()
if isinstance(spec, dict) and spec.get("git") == "https://github.com/rustfs/gateway.git"}
if len(revs) != 1 or not re.fullmatch(r"[0-9a-f]{40}", str(next(iter(revs)))):
sys.exit(f"Cargo.toml must pin rustfs/gateway at exactly one full rev, found {sorted(map(str, revs))}")
print(f"rev={revs.pop()}")
PY
cat "$GITHUB_OUTPUT"
build-candidate:
name: Build candidate binary
# The same debug build as ci.yml's build-rustfs-debug-binary, reading the
# ci-dev cache written on sm-standard-4.
runs-on: sm-standard-4
timeout-minutes: 60
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: stable
cache-shared-key: ci-dev
cache-save-if: 'false'
install-build-packaging-tools: 'false'
- name: Build debug binary
run: python3 scripts/e2e_binary.py build --bins --features e2e-test-hooks
- name: Upload candidate binary
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: gateway-acceptance-candidate-binary
overwrite: true
path: |
target/debug/rustfs
target/debug/rustfs.e2e.json
if-no-files-found: error
retention-days: 1
build-gateway-tools:
name: Build gateway conformance runner and compat-sut
needs: [ plan ]
# The gateway workspace builds with its own rust-toolchain.toml, which rustup
# installs on first use; nothing here names a toolchain.
runs-on: ubuntu-latest
timeout-minutes: 60
steps:
- name: Checkout rustfs/gateway at the pinned rev
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
repository: rustfs/gateway
ref: ${{ needs.plan.outputs.gateway-rev }}
path: gateway
persist-credentials: false
- name: Build
working-directory: gateway
run: |
cargo build --locked --release -p rustfs-gateway-conformance -p rustfs-gateway-compat-sut \
--bin rustfs-gateway-conformance --bin compat-sut
- name: Upload tools
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: gateway-acceptance-tools
overwrite: true
path: |
gateway/target/release/rustfs-gateway-conformance
gateway/target/release/compat-sut
if-no-files-found: error
retention-days: 1
acceptance:
name: ${{ matrix.stack == 'gateway' && 'Acceptance (gateway, report-only until T2.10)' || 'Acceptance (legacy)' }}
needs: [ plan, build-candidate, build-gateway-tools ]
# GitHub-hosted like the s3-tests targets of reference-parity.yml: the job
# only runs prebuilt binaries and the pinned clients, none of which needs Docker.
runs-on: ubuntu-latest
# About 3x the 15 minutes the legacy job took in run 43325101234 (client install about
# 8, matrix about 3, corpus about 3), so a hung server costs 45 runner minutes, not more.
timeout-minutes: 45
continue-on-error: ${{ matrix.stack == 'gateway' }}
strategy:
fail-fast: false
matrix:
stack: [ legacy, gateway ]
env:
STACK: ${{ matrix.stack }}
ENDPOINT: http://127.0.0.1:9000
# The conformance suite signs with these fixed fixture credentials (the AWS
# documentation example pair), so the candidate is started with them and the
# client matrix is handed the same pair.
ACCESS_KEY: AKIAIOSFODNN7EXAMPLE
SECRET_KEY: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Checkout rustfs/gateway at the pinned rev
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
repository: rustfs/gateway
ref: ${{ needs.plan.outputs.gateway-rev }}
path: gateway
persist-credentials: false
- name: Download candidate binary
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: gateway-acceptance-candidate-binary
path: target/debug
- name: Download gateway tools
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: gateway-acceptance-tools
path: tools
# Records `ready`, or what the process said when it exited, in lane/startup.txt. `ready`
# also needs the server's own `server_starting` log event to name the requested stack
# (its `s3_stack` field comes from rustfs/rustfs#9367; scripts/dual_stack.py reads it).
- name: Start candidate (${{ matrix.stack }} stack)
run: |
set -uo pipefail
chmod +x target/debug/rustfs tools/*
mkdir -p lane "$RUNNER_TEMP"/data/rustfs{0..3}
target/debug/rustfs --version | tee lane/candidate-version.txt
# The four directories share the runner's one disk; the bypass is the CI case the
# physical-disk guard sanctions (as in mint.yml).
RUSTFS_S3_STACK="$STACK" RUSTFS_UNSAFE_BYPASS_DISK_CHECK=true RUST_LOG=info target/debug/rustfs --address 127.0.0.1:9000 \
--access-key "$ACCESS_KEY" --secret-key "$SECRET_KEY" "$RUNNER_TEMP"/data/rustfs{0..3} \
> lane/rustfs.log 2>&1 &
pid=$!
echo "$pid" > lane/rustfs.pid
# Liveness first: a ready answer counts only while this process is still running.
for _ in $(seq 120); do
if ! kill -0 "$pid" 2>/dev/null; then
status=0
wait "$pid" || status=$?
echo "did not start: exit ${status}: $(grep -v '^[[:space:]]*$' lane/rustfs.log | tail -n 1)" > lane/startup.txt
cat lane/startup.txt
exit 1
fi
if curl -sf "$ENDPOINT/health/ready" >/dev/null; then
if python3 scripts/dual_stack.py stack --expect "$STACK" --out lane/stack.json lane/rustfs.log > /dev/null; then
echo ready > lane/startup.txt
exit 0
fi
echo "did not start: stack not observed" > lane/startup.txt
cat lane/startup.txt
exit 1
fi
sleep 1
done
echo "did not start: /health/ready did not answer within 120 seconds" > lane/startup.txt
exit 1
# The exit code is data for the judge step, not this step's result.
- name: Run gateway conformance (rustfs profile)
run: |
set +e
tools/rustfs-gateway-conformance run --root gateway/conformance --profile rustfs \
--endpoint "$ENDPOINT" --allow-external-fixtures \
--json lane/report.json > lane/conformance.log 2>&1
echo "$?" > lane/runner-exit.txt
if kill -0 "$(cat lane/rustfs.pid)" 2>/dev/null; then echo alive; else echo exited; fi > lane/alive-after-conformance.txt
tail -n 20 lane/conformance.log
- name: Set up Python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.12"
# Toolchains for the matrix's SDK driver programs, as the gateway's own
# client-matrix.yml sets them up; the client versions come from
# gateway/compat/versions.toml.
- uses: actions/setup-go@41dfa10bad2bb2ae585af6ee5bb4d7d973ad74ed # v5.1.0
with:
go-version: "1.26"
cache: false
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
node-version: "22"
- uses: actions/setup-dotnet@67a3573c9a986a3f9c594539f4ab511d57bb3ce9 # v4.3.1
with:
dotnet-version: "8.0.x"
- uses: actions/setup-java@cf277c60eb25467037889841efdb72551f06f6c3 # v4.9.1
with:
distribution: temurin
java-version: "21"
- name: Install Maven
env:
MAVEN_VERSION: "3.9.16"
MAVEN_SHA512: "831a8591fe20c8243b1dbe7d71e3244f31d1665b0804b2e825e38cbbe5ce0cafb8338851f90780735568773e0a6cd07bbec107cda0b896b008b861075358b6f6"
run: |
set -euo pipefail
archive="$RUNNER_TEMP/apache-maven.tar.gz"
curl -sSfL -o "$archive" "https://archive.apache.org/dist/maven/maven-3/${MAVEN_VERSION}/binaries/apache-maven-${MAVEN_VERSION}-bin.tar.gz"
echo "${MAVEN_SHA512} ${archive}" | sha512sum -c -
tar -xzf "$archive" -C "$RUNNER_TEMP"
echo "$RUNNER_TEMP/apache-maven-${MAVEN_VERSION}/bin" >> "$GITHUB_PATH"
# One client at a time, so a client that cannot be installed is reported
# as skipped with its reason instead of stopping the others.
- name: Install pinned clients
working-directory: gateway
run: |
set -uo pipefail
python3 -m pip install --disable-pip-version-check "pyyaml==6.0.2"
installed=()
for client in $(python3 -c 'import tomllib; print(" ".join(sorted(tomllib.load(open("compat/versions.toml", "rb"))["clients"])))'); do
if bash ci/compat/install_clients.sh --clients "$client" > "../lane/install-${client}.log" 2>&1; then
installed+=("$client")
else
echo "${client}: skipped: $(tail -n 1 "../lane/install-${client}.log")" >> ../lane/skipped-clients.txt
fi
done
echo "COMPAT_CLIENTS=$(IFS=,; echo "${installed[*]}")" >> "$GITHUB_ENV"
cat ../lane/skipped-clients.txt 2>/dev/null || echo "every pinned client installed"
# The exit code is data for the judge step, not this step's result.
- name: Run client matrix (external)
working-directory: gateway
env:
GATEWAY_COMPAT_SUT_BIN: ${{ github.workspace }}/tools/compat-sut
GATEWAY_COMPAT_RUN_DIR: ${{ github.workspace }}/lane/compat
GATEWAY_COMPAT_MATRIX_OUT: ${{ github.workspace }}/lane/compat/matrix.json
run: |
set +e
export GATEWAY_COMPAT_ACCESS_KEY="$ACCESS_KEY" GATEWAY_COMPAT_SECRET_KEY="$SECRET_KEY"
PATH="$(go env GOPATH)/bin:$PATH"
if [[ -f ../lane/skipped-clients.txt ]]; then
bash ci/compat/run_matrix.sh --external "$ENDPOINT" --product rustfs --clients "$COMPAT_CLIENTS"
else
bash ci/compat/run_matrix.sh --external "$ENDPOINT" --product rustfs
fi
echo "$?" > ../lane/matrix-exit.txt
- name: Judge and summarize
if: always()
run: |
set -uo pipefail
status=0
python3 scripts/check_conformance_rulings.py --stack "$STACK" --run-dir lane | tee -a "$GITHUB_STEP_SUMMARY" || status=1
python3 scripts/compat_matrix_report.py check --stack "$STACK" --run-dir lane | tee -a "$GITHUB_STEP_SUMMARY" || status=1
exit "$status"
- name: Upload lane results
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: gateway-acceptance-lane-${{ matrix.stack }}
overwrite: true
path: |
lane/*.txt
lane/*.log
lane/report.json
lane/stack.json
lane/compat/matrix.json
lane/compat/probe.jsonl
lane/compat/sut.log
lane/compat/results/**
if-no-files-found: error
retention-days: 14
compare:
name: Compare stacks (report-only until T2.10)
needs: [ acceptance ]
if: always()
runs-on: ubuntu-latest
timeout-minutes: 10
continue-on-error: true
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Download lane results
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
pattern: gateway-acceptance-lane-*
path: lanes
- name: Diff legacy and gateway
run: |
set -uo pipefail
status=0
{
python3 scripts/check_conformance_rulings.py \
--diff lanes/gateway-acceptance-lane-legacy lanes/gateway-acceptance-lane-gateway || status=1
echo
python3 scripts/compat_matrix_report.py diff \
--legacy lanes/gateway-acceptance-lane-legacy --gateway lanes/gateway-acceptance-lane-gateway || status=1
} > diff.md
cat diff.md
cat diff.md >> "$GITHUB_STEP_SUMMARY"
exit "$status"
+183 -50
View File
@@ -1,26 +1,136 @@
# Copyright 2026 RustFS Team
# SPDX-License-Identifier: Apache-2.0
name: Build candidate Helm chart
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
name: Publish helm chart to artifacthub
on:
workflow_call:
workflow_run:
workflows: [ "Build and Release" ]
types: [ completed ]
workflow_dispatch:
inputs:
candidate_manifest:
type: string
version:
description: "Release version to publish, e.g. 1.0.0-beta.1 or v1.0.0-beta.1"
required: true
default: "1.0.0-beta.1"
type: string
permissions:
contents: read
actions: read
jobs:
package:
build-helm-package:
runs-on: ubuntu-latest
timeout-minutes: 15
timeout-minutes: 30
if: |
(github.event_name == 'workflow_dispatch' && !contains(github.event.inputs.version, '-preview')) ||
(
github.event.workflow_run.conclusion == 'success' &&
github.event.workflow_run.event == 'push' &&
contains(github.event.workflow_run.head_branch, '.') &&
!contains(github.event.workflow_run.head_branch, '-preview')
)
outputs:
raw_tag: ${{ steps.version.outputs.raw_tag }}
app_version: ${{ steps.version.outputs.app_version }}
chart_version: ${{ steps.version.outputs.chart_version }}
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
- name: Checkout release source
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
ref: ${{ fromJson(inputs.candidate_manifest).source_sha }}
ref: ${{ github.event.workflow_run.head_sha || (startsWith(inputs.version, 'refs/tags/') && inputs.version || format('refs/tags/{0}', inputs.version)) }}
persist-credentials: false
- uses: azure/setup-helm@b9e51907a09c216f16ebe8536097933489208112 # v4.3.0
- name: Install pinned yq
# Both inputs reach the shell through env rather than `${{ }}`
# interpolation. A git ref name may contain `$(...)` — anything without a
# space is a legal tag — and interpolation pastes it into the script
# verbatim, where bash would run it. Reading "$RAW_INPUT" instead makes it
# data.
- name: Normalize release version
id: version
env:
RAW_INPUT: ${{ github.event.inputs.version }}
RAW_BRANCH: ${{ github.event.workflow_run.head_branch }}
run: |
set -eux
if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then
RAW="$RAW_INPUT"
else
RAW="$RAW_BRANCH"
fi
case "$RAW" in
refs/tags/*)
RAW_TAG="${RAW#refs/tags/}"
;;
*)
RAW_TAG="$RAW"
;;
esac
./scripts/helm_chart_version.sh "$RAW_TAG"
# Docker and Helm share the binary-build trigger. Do not publish a chart
# until the image selected by its default appVersion is pullable.
- name: Wait for release image
timeout-minutes: 20
env:
APP_VERSION: ${{ steps.version.outputs.app_version }}
run: |
set -euo pipefail
if [[ ! "$APP_VERSION" =~ ^[0-9]+\.[0-9]+\.[0-9]+(-[0-9A-Za-z.-]+)?$ ]] ||
[[ "$APP_VERSION" == *-preview* ]]; then
echo "Invalid deliverable image version: $APP_VERSION" >&2
exit 1
fi
for attempt in {1..60}; do
if token=$(curl --fail --silent --show-error --connect-timeout 10 --max-time 20 \
'https://auth.docker.io/token?service=registry.docker.io&scope=repository:rustfs/rustfs:pull' \
| jq -er '.token // empty') &&
curl --fail --silent --show-error --head --connect-timeout 10 --max-time 20 \
--header "Authorization: Bearer $token" \
--header 'Accept: application/vnd.oci.image.index.v1+json, application/vnd.docker.distribution.manifest.list.v2+json' \
"https://registry-1.docker.io/v2/rustfs/rustfs/manifests/$APP_VERSION" > /dev/null; then
echo "Release image is available: rustfs/rustfs:$APP_VERSION"
exit 0
fi
echo "Waiting for rustfs/rustfs:$APP_VERSION ($attempt/60)"
if (( attempt < 60 )); then
sleep 15
fi
done
echo "Release image is unavailable; refusing to publish Helm chart" >&2
exit 1
- name: Replace chart version and app version
env:
CHART_VERSION: ${{ steps.version.outputs.chart_version }}
APP_VERSION: ${{ steps.version.outputs.app_version }}
run: |
set -eux
sed -i -E "s/^version:.*/version: \"${CHART_VERSION}\"/" helm/rustfs/Chart.yaml
sed -i -E "s/^appVersion:.*/appVersion: \"${APP_VERSION}\"/" helm/rustfs/Chart.yaml
- name: Set up Helm
uses: azure/setup-helm@b9e51907a09c216f16ebe8536097933489208112 # v4.3.0
- name: Install yq
run: |
set -euo pipefail
yq_version="4.54.1"
@@ -35,43 +145,66 @@ jobs:
echo "$tools_dir" >> "$GITHUB_PATH"
"$tools_dir/yq" --version
- name: Test and package the exact application version
id: chart
env:
GH_TOKEN: ${{ github.token }}
CANDIDATE_MANIFEST: ${{ inputs.candidate_manifest }}
- name: Test Helm Chart Templates
run: ./scripts/test_helm_templates.sh
- name: Package Helm Chart
run: |
python3 - <<'PYCHART'
import sys, subprocess, shutil
from pathlib import Path
sys.path.insert(0, 'scripts/release')
import candidate as c, pipeline
inputs = pipeline.manifest_env()
current = pipeline.current_run('release-complete.yml')
if any(a['name'] == 'helm-package' for a in c.artifact_list(current['id'])):
c.download_artifact(current, 'helm-package', 'chart')
record = c.read_json('chart/chart.json')
c.require(record['inputs_sha256'] == c.digest(inputs), 'chart belongs to another candidate')
c.emit_output('reused', 'true')
else:
chart = Path('helm/rustfs/Chart.yaml')
import re
value = chart.read_text()
for key in ('version', 'appVersion'):
value, count = re.subn(rf'(?m)^{key}:.*$', f'{key}: "{inputs["version"]}"', value)
c.require(count == 1, 'ambiguous chart metadata')
chart.write_text(value)
subprocess.run(['bash', 'scripts/test_helm_templates.sh'], check=True)
shutil.copyfile('helm/README.md', 'helm/rustfs/README.md')
subprocess.run(['helm', 'package', 'helm/rustfs', '--destination', 'chart'], check=True)
path = Path('chart') / f"rustfs-{inputs['version']}.tgz"
c.write_json('chart/chart.json', {'version': inputs['version'], 'inputs_sha256': c.digest(inputs),
'name': path.name, 'sha256': c.file_digest(path), 'size': path.stat().st_size})
PYCHART
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
if: steps.chart.outputs.reused != 'true'
set -eux
cp helm/README.md helm/rustfs/
helm package ./helm/rustfs \
--destination helm/rustfs/ \
--version "${{ steps.version.outputs.chart_version }}"
- name: Upload helm package as artifact
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: helm-package
path: chart/*
if-no-files-found: error
retention-days: 30
path: |
helm/rustfs/*.tgz
helm/rustfs/README.md
retention-days: 1
publish-helm-package:
runs-on: ubuntu-latest
timeout-minutes: 30
needs: [ build-helm-package ]
if: needs.build-helm-package.result == 'success'
steps:
- name: Checkout helm package repo
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
# persist-credentials-exempt: this checkout's token IS the push credential —
# the job git-pushes to rustfs/helm below. Clearing it breaks chart publishing.
repository: rustfs/helm
token: ${{ secrets.RUSTFS_HELM_PACKAGE }}
- name: Download helm package
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: helm-package
path: ./
- name: Set up helm
uses: azure/setup-helm@b9e51907a09c216f16ebe8536097933489208112 # v4.3.0
- name: Generate index
run: helm repo index . --url https://charts.rustfs.com
# app_version is derived from the triggering tag name, and this job holds
# the cross-repository push token with rustfs/helm already checked out —
# the worst place in the repo to paste an attacker-influenced string into
# a shell line. Passed through env so bash treats it as data.
- name: Push helm package and index file
env:
GIT_USERNAME: ${{ secrets.USERNAME }}
GIT_EMAIL: ${{ secrets.EMAIL_ADDRESS }}
APP_VERSION: ${{ needs.build-helm-package.outputs.app_version }}
run: |
set -eux
git config --global user.name "${GIT_USERNAME}"
git config --global user.email "${GIT_EMAIL}"
git add .
git commit -m "Update rustfs helm package with ${APP_VERSION}." || echo "No changes to commit"
git push origin main
-463
View File
@@ -1,463 +0,0 @@
# Copyright 2026 RustFS Team
# SPDX-License-Identifier: Apache-2.0
# Manual Linux zip/deb/rpm packaging. Workflow artifacts only.
# Dispatch uses this file from the selected branch. An empty ref packages
# the tip of refs/heads/main. A non-empty ref must already exist; the
# resolver fetches that tag, branch, or commit so checkout is not limited
# to the default branch. Nothing is published.
name: Manual Linux packages
on:
workflow_dispatch:
inputs:
architecture:
description: "Linux targets: x86 (x86_64 musl+gnu), arm (aarch64 musl+gnu), or all"
required: true
type: choice
options:
- x86
- arm
- all
default: all
ref:
description: "Git tag to package. Leave empty to package the tip of origin/main. A branch name or full commit SHA is also accepted."
required: false
type: string
default: ""
version:
description: "Package version: MAJOR.MINOR.PATCH, or MAJOR.MINOR.PATCH-(alpha|beta|rc).N. No leading v. Required for origin/main. For a tag, leave empty to derive it from the tag; an explicit value is used as entered."
required: false
type: string
default: ""
permissions:
contents: read
env:
CARGO_TERM_COLOR: always
RUST_BACKTRACE: 1
CARGO_INCREMENTAL: 0
jobs:
prepare:
name: Resolve source and version
if: github.repository == 'rustfs/rustfs'
runs-on: ubuntu-latest
timeout-minutes: 15
outputs:
source_sha: ${{ steps.source.outputs.source_sha }}
source_kind: ${{ steps.source.outputs.source_kind }}
source_name: ${{ steps.source.outputs.source_name }}
version: ${{ steps.source.outputs.version }}
packaging: ${{ steps.source.outputs.packaging }}
tool_sha: ${{ steps.source.outputs.tool_sha }}
confirm_from: ${{ steps.source.outputs.confirm_from }}
toolchain: ${{ steps.inputs.outputs.toolchain }}
console: ${{ steps.inputs.outputs.console }}
matrix: ${{ steps.matrix.outputs.matrix }}
steps:
# Tooling checkout. An empty ref then detaches this tree to the tip of
# origin/main. A tag or other ref is fetched and pinned without
# replacing this tree, because console resolution needs candidate.py.
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
ref: main
fetch-depth: 0
persist-credentials: false
- name: Resolve source and version
id: source
shell: bash
env:
SOURCE_REF: ${{ inputs.ref }}
PACKAGE_VERSION: ${{ inputs.version }}
run: |
set -euo pipefail
bash scripts/release/package_manual_artifacts.sh resolve-source \
--source "$SOURCE_REF" \
--version "$PACKAGE_VERSION" | tee -a "$GITHUB_OUTPUT"
- name: Install stable Rust
uses: dtolnay/rust-toolchain@29eef336d9b2848a0b548edc03f92a220660cdb8 # stable
with:
toolchain: stable
- name: Resolve console assets and toolchain
id: inputs
shell: bash
env:
GH_TOKEN: ${{ github.token }}
run: |
set -euo pipefail
toolchain=$(rustc --version | awk '{print $2}')
[[ "$toolchain" =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]]
gh api repos/rustfs/console/releases/latest > console-release.json
TOOLCHAIN="$toolchain" python3 - <<'PY'
import json, os, sys
sys.path.insert(0, "scripts/release")
import candidate
console = candidate.console_asset(candidate.read_json("console-release.json"))
candidate.emit_output("console", json.dumps(console, separators=(",", ":")))
candidate.emit_output("toolchain", os.environ["TOOLCHAIN"])
print(f"toolchain={os.environ['TOOLCHAIN']}")
print(f"console={console['tag']} {console['name']}")
PY
- name: Select Linux targets
id: matrix
shell: bash
env:
ARCHITECTURE: ${{ inputs.architecture }}
run: |
set -euo pipefail
matrix=$(bash scripts/release/package_manual_artifacts.sh matrix "$ARCHITECTURE")
count=$(python3 -c 'import json,sys; print(len(json.loads(sys.argv[1])["include"]))' "$matrix")
[[ "$count" -gt 0 ]]
printf 'matrix=%s\n' "$matrix" >> "$GITHUB_OUTPUT"
printf 'Selected %s target(s) for %s\n' "$count" "$ARCHITECTURE"
package:
name: Package RustFS (${{ matrix.target_id }})
needs: prepare
if: github.repository == 'rustfs/rustfs'
# x86_64 uses sm-standard-4, the official release compile pool: the workspace
# release build needs that disk (placement rule: disk) and reads build.yml's
# cache without writing it. aarch64 uses ubicloud-standard-16-arm so the
# packaged binary and console smoke execute natively (placement rule: hardware).
runs-on: ${{ matrix.os }}
timeout-minutes: 210
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
RUSTUP_TOOLCHAIN: ${{ needs.prepare.outputs.toolchain }}
RUSTFLAGS: "${{ matrix.rustflags }}"
strategy:
fail-fast: false
matrix: ${{ fromJson(needs.prepare.outputs.matrix) }}
steps:
- name: Checkout the pinned source commit
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
ref: ${{ needs.prepare.outputs.source_sha }}
fetch-depth: 0
persist-credentials: false
# Tags cut before the manual packaging script still compile from their
# own tree. The official zip/deb/rpm recipe, and confirm-source when the
# tagged script does not implement it, come from the prepare checkout.
- name: Checkout packaging recipe
if: ${{ needs.prepare.outputs.packaging == 'tool' || needs.prepare.outputs.confirm_from == 'tool' }}
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
ref: ${{ needs.prepare.outputs.tool_sha }}
path: packaging-tools
fetch-depth: 1
persist-credentials: false
- name: Confirm the pinned source
shell: bash
env:
SOURCE_KIND: ${{ needs.prepare.outputs.source_kind }}
SOURCE_NAME: ${{ needs.prepare.outputs.source_name }}
SOURCE_SHA: ${{ needs.prepare.outputs.source_sha }}
run: |
set -euo pipefail
helper=scripts/release/package_manual_artifacts.sh
if [[ ! -f "$helper" ]] || ! grep -q 'confirm-source)' "$helper"; then
helper=packaging-tools/scripts/release/package_manual_artifacts.sh
fi
if [[ ! -f "$helper" ]] || ! grep -q 'confirm-source)' "$helper"; then
printf 'no packaging script implements confirm-source\n' >&2
exit 1
fi
bash "$helper" confirm-source \
--kind "$SOURCE_KIND" \
--name "$SOURCE_NAME" \
--sha "$SOURCE_SHA"
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: ${{ needs.prepare.outputs.toolchain }}
target: ${{ matrix.target }}
cache-shared-key: build-${{ matrix.target }}${{ matrix.glibc && format('-glibc-{0}', matrix.glibc) || '' }}${{ matrix.os == 'ubicloud-standard-16-arm' && '-ubicloud-arm' || '' }}
# build.yml owns this key. This job only restores it.
cache-save-if: 'false'
install-cross-tools: ${{ matrix.cross }}
install-build-packaging-tools: 'true'
install-test-tools: 'false'
- name: Download static console assets
shell: bash
env:
GH_TOKEN: ${{ github.token }}
CONSOLE_IDENTITY: ${{ needs.prepare.outputs.console }}
run: |
set -euo pipefail
python3 - <<'PYCONSOLE'
import json, os, pathlib, subprocess, zipfile
value = json.loads(os.environ["CONSOLE_IDENTITY"])
subprocess.run(["python3", ".github/actions/setup/download_release_asset.py",
"--repository", "rustfs/console", "--tag", value["tag"], "--asset", value["name"],
"--sha256", value["sha256"], "--output", "console.zip"], check=True)
destination = pathlib.Path("rustfs/static")
destination.mkdir(parents=True, exist_ok=True)
with zipfile.ZipFile("console.zip") as archive:
for member in archive.infolist():
path = pathlib.PurePosixPath(member.filename)
if path.is_absolute() or ".." in path.parts or "\\" in member.filename:
raise SystemExit("unsafe Console archive member")
archive.extractall(destination)
if not (destination / "index.html").is_file():
raise SystemExit("Console archive has no index.html")
PYCONSOLE
rm -f console.zip
- name: Build RustFS
shell: bash
env:
RUSTFS_BUILD_VERSION: ${{ needs.prepare.outputs.version }}
BUILD_TARGET: ${{ matrix.target }}
BUILD_OS: ${{ matrix.build_os }}
BUILD_FEATURES: ${{ matrix.features }}
BUILD_GLIBC: ${{ matrix.glibc }}
BUILD_CROSS: ${{ matrix.cross }}
run: |
set -euo pipefail
touch rustfs/build.rs
feature_args=()
if [[ -n "$BUILD_FEATURES" ]]; then
feature_args=(--features "$BUILD_FEATURES")
fi
# gnu targets pin glibc 2.31 through cargo-zigbuild (<target>.<glibc>).
target_arg="$BUILD_TARGET"
if [[ -n "$BUILD_GLIBC" ]]; then
target_arg="${target_arg}.${BUILD_GLIBC}"
fi
profile_args=()
if [[ "$BUILD_TARGET" == *-linux-gnu && "$BUILD_FEATURES" == pyroscope ]]; then
profile_args=(--config 'profile.release.strip="none"')
fi
case "$BUILD_CROSS" in
true)
cargo zigbuild --release --locked --target "$target_arg" "${feature_args[@]}" "${profile_args[@]}" -p rustfs --bin rustfs
;;
false)
cargo build --release --locked --target "$target_arg" "${feature_args[@]}" "${profile_args[@]}" -p rustfs --bin rustfs
;;
*)
printf 'unexpected cross flag: %q\n' "$BUILD_CROSS" >&2
exit 1
;;
esac
bin="target/${BUILD_TARGET}/release/rustfs"
version_output=$("$bin" --version)
first=${version_output%%$'\n'*}
if [[ "$first" != "rustfs ${RUSTFS_BUILD_VERSION}" ]]; then
printf 'binary version line is %q, expected %q\n' "$first" "rustfs ${RUSTFS_BUILD_VERSION}" >&2
printf '%s\n' "$version_output" >&2
exit 1
fi
if ! grep -Fxq "build target : ${BUILD_TARGET}" <<< "$version_output" || \
! grep -Fxq "build os : ${BUILD_OS}" <<< "$version_output" || \
! grep -Fxq "rust channel : stable-${BUILD_TARGET}" <<< "$version_output"; then
printf 'release binary reports an unexpected compile target\n' >&2
printf '%s\n' "$version_output" >&2
exit 1
fi
if [[ -n "$BUILD_GLIBC" ]]; then
glibc_check=scripts/release/check_glibc.py
if [[ ! -f "$glibc_check" ]]; then
glibc_check=packaging-tools/scripts/release/check_glibc.py
fi
if [[ ! -f "$glibc_check" ]]; then
printf 'glibc check script not found\n' >&2
exit 1
fi
python3 "$glibc_check" "$BUILD_GLIBC" "$bin"
fi
- name: Generate the CPU symbol catalog
if: endsWith(matrix.target, '-linux-gnu') && matrix.features == 'pyroscope'
shell: bash
env:
BUILD_TARGET: ${{ matrix.target }}
run: |
set -euo pipefail
cargo metadata --locked --format-version 1 --filter-platform "$BUILD_TARGET" \
--features rustfs/pyroscope > "$RUNNER_TEMP/cpu-catalog-metadata.json"
features=$(jq -ce '
(.packages[] | select(.name == "rustfs") | .id) as $id |
.resolve.nodes[] | select(.id == $id) | .features | map(select(. != "default")) | sort
' "$RUNNER_TEMP/cpu-catalog-metadata.json")
release_dir="target/${BUILD_TARGET}/release"
rm -f "$release_dir/rustfs.cpu-symbol-catalog.json" "$release_dir/rustfs.cpu-symbol-catalog.sha256"
cargo run --locked -p rustfs-connect-symbol-catalog -- \
"$release_dir/rustfs" "$(git rev-parse HEAD)" "$BUILD_TARGET" "$features" \
"$release_dir/rustfs.cpu-symbol-catalog.json" > "$release_dir/rustfs.cpu-symbol-catalog.sha256"
- name: Install DEB and RPM tools
shell: bash
run: |
set -euo pipefail
sudo apt-get update
sudo apt-get install -y ruby ruby-dev build-essential rpm fakeroot cpio
sudo gem install fpm --version 1.17.0 --no-document
- name: Package zip, deb, and rpm
shell: bash
env:
PACKAGE_VERSION: ${{ needs.prepare.outputs.version }}
ARCH: ${{ matrix.arch }}
LIBC: ${{ matrix.libc }}
BUILD_TARGET: ${{ matrix.target }}
BUILD_FEATURES: ${{ matrix.features }}
SOURCE_KIND: ${{ needs.prepare.outputs.source_kind }}
SOURCE_SHA: ${{ needs.prepare.outputs.source_sha }}
PACKAGING: ${{ needs.prepare.outputs.packaging }}
run: |
set -euo pipefail
case "$PACKAGING" in
source) script=scripts/release/package_manual_artifacts.sh ;;
tool) script=packaging-tools/scripts/release/package_manual_artifacts.sh ;;
*)
printf 'unexpected packaging recipe: %q\n' "$PACKAGING" >&2
exit 1
;;
esac
if [[ ! -f "$script" ]]; then
printf 'packaging script not found: %s\n' "$script" >&2
exit 1
fi
binary="target/${BUILD_TARGET}/release/rustfs"
extra=()
if [[ "$BUILD_TARGET" == *-linux-gnu && "$BUILD_FEATURES" == pyroscope ]]; then
extra=(
--zip-extra "target/${BUILD_TARGET}/release/rustfs.cpu-symbol-catalog.json"
--zip-extra "target/${BUILD_TARGET}/release/rustfs.cpu-symbol-catalog.sha256"
)
fi
bash "$script" package \
--version "$PACKAGE_VERSION" \
--arch "$ARCH" \
--libc "$LIBC" \
--binary "$binary" \
--output dist \
"${extra[@]}"
printf 'Packaged %s from %s %s\n' "$PACKAGE_VERSION" "$SOURCE_KIND" "$SOURCE_SHA"
- name: Verify the packaged binary
shell: bash
env:
PACKAGE_VERSION: ${{ needs.prepare.outputs.version }}
ARCH: ${{ matrix.arch }}
LIBC: ${{ matrix.libc }}
BUILD_TARGET: ${{ matrix.target }}
BUILD_OS: ${{ matrix.build_os }}
BUILD_FEATURES: ${{ matrix.features }}
run: |
set -euo pipefail
stem="rustfs-linux-${ARCH}-${LIBC}-v${PACKAGE_VERSION}"
package_file="dist/${stem}.zip"
[[ -f "$package_file" && -f "dist/${stem}.deb" && -f "dist/${stem}.rpm" ]]
[[ "$(find dist -maxdepth 1 -type f | wc -l)" -eq 3 ]]
archive_members=$(unzip -Z1 "$package_file" | sort)
if [[ "$BUILD_TARGET" == *-linux-gnu && "$BUILD_FEATURES" == pyroscope ]]; then
expected_members=$(printf '%s\n' rustfs rustfs.cpu-symbol-catalog.json rustfs.cpu-symbol-catalog.sha256 | sort)
else
expected_members=$(printf '%s\n' rustfs)
fi
if [[ "$archive_members" != "$expected_members" ]]; then
printf 'Linux package does not contain the expected files\n' >&2
printf '%s\n' "$archive_members" >&2
exit 1
fi
binary_dir=$(mktemp -d)
trap 'rm -rf "$binary_dir"' EXIT
unzip -q "$package_file" rustfs -d "$binary_dir"
version_output=$("$binary_dir/rustfs" --version)
first=${version_output%%$'\n'*}
if [[ "$first" != "rustfs ${PACKAGE_VERSION}" ]]; then
printf 'packaged binary version line is %q, expected %q\n' "$first" "rustfs ${PACKAGE_VERSION}" >&2
exit 1
fi
if ! grep -Fxq "build target : ${BUILD_TARGET}" <<< "$version_output" || \
! grep -Fxq "build os : ${BUILD_OS}" <<< "$version_output"; then
printf 'packaged binary reports an unexpected compile target\n' >&2
exit 1
fi
- name: Verify packaged console
if: matrix.target == 'x86_64-unknown-linux-gnu' || matrix.target == 'aarch64-unknown-linux-gnu'
shell: bash
env:
PACKAGE_VERSION: ${{ needs.prepare.outputs.version }}
ARCH: ${{ matrix.arch }}
LIBC: ${{ matrix.libc }}
run: |
set -euo pipefail
package_file="dist/rustfs-linux-${ARCH}-${LIBC}-v${PACKAGE_VERSION}.zip"
find_free_port() {
python3 - <<'PY'
import socket
with socket.socket() as sock:
sock.bind(("127.0.0.1", 0))
print(sock.getsockname()[1])
PY
}
package_dir=$(mktemp -d)
data_dir=$(mktemp -d)
log_file="${RUNNER_TEMP}/rustfs-console-smoke.log"
server_pid=""
trap '
if [[ -n "${server_pid:-}" ]]; then
kill "$server_pid" 2>/dev/null || true
wait "$server_pid" 2>/dev/null || true
fi
rm -rf "$package_dir" "$data_dir"
' EXIT
unzip -q "$package_file" -d "$package_dir"
api_port=$(find_free_port)
console_port=$(find_free_port)
console_url="http://127.0.0.1:${console_port}/rustfs/console/"
"$package_dir/rustfs" server \
--address "127.0.0.1:${api_port}" \
--console-enable \
--console-address "127.0.0.1:${console_port}" \
--access-key console-smoke \
--secret-key console-smoke-secret \
"$data_dir" >"$log_file" 2>&1 &
server_pid=$!
for _ in {1..40}; do
response=$(curl --silent --output /dev/null --write-out '%{http_code} %{content_type}' "$console_url" || true)
if [[ "$response" == 200\ text/html* ]]; then
exit 0
fi
sleep 0.25
done
printf 'Console endpoint did not return 200 text/html: %s\n' "$response" >&2
cat "$log_file"
exit 1
- name: Upload packages
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: packages-${{ matrix.target_id }}-v${{ needs.prepare.outputs.version }}
path: dist/*
if-no-files-found: error
compression-level: 0
retention-days: 30
+9 -90
View File
@@ -44,13 +44,6 @@
# well inside the 120-minute budget and buys deterministic infra. The
# docker-capable self-hosted `dind-sm-standard-2` label was the alternative but
# has fewer cores and reintroduces fleet-state risk for no reliability gain.
#
# Stack dimension (rustfs/backlog#2774, T3.3): one image, run with RUSTFS_S3_STACK=legacy and
# =gateway. `compare` (scripts/mint_stack_diff.py) holds the gate that every case legacy passes
# passes on gateway with no new failure. The gateway leg and `compare` are report-only until
# T2.10. A gateway server that refuses to start stops its leg at "Wait for RustFS ready", before
# mint runs; the summary says `skipped: gateway stack refuses startup: <why>`. The stack is read
# from the server's own start-up line (scripts/dual_stack.py stack).
name: mint
@@ -115,9 +108,13 @@ defaults:
shell: bash
jobs:
image:
name: Build the RustFS image once
runs-on: ubuntu-latest
mint:
# GitHub-hosted: reliably provides a running Docker daemon + buildx +
# python3. See the header note (ci-2) for why the self-hosted sm-standard-4
# label was abandoned (no docker.sock on the pod its only run landed on).
# TODO(ci-8): scheduled-failure alerting (auto-open issue on a red weekly
# run) is added by the ci-8 composite action; do not implement it here.
runs-on: sm-standard-2
timeout-minutes: 120
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
@@ -135,48 +132,6 @@ jobs:
--cache-to "type=gha,mode=max,scope=${BUILDX_CACHE_SCOPE}" \
-t rustfs-ci \
-f Dockerfile.source .
docker save rustfs-ci --output "${RUNNER_TEMP}/rustfs-ci.tar"
- name: Upload the image
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: mint-image
path: ${{ runner.temp }}/rustfs-ci.tar
retention-days: 1
mint:
name: mint (${{ matrix.stack }}${{ matrix.stack == 'gateway' && ', report-only until T2.10' || '' }})
needs: [image]
# Report-only until T2.10: the gateway stack cannot serve the suites yet.
continue-on-error: ${{ matrix.stack == 'gateway' }}
strategy:
fail-fast: false
matrix:
stack: [legacy, gateway]
env:
STACK: ${{ matrix.stack }}
# GitHub-hosted: reliably provides a running Docker daemon + buildx +
# python3. See the header note (ci-2) for why the self-hosted sm-standard-4
# label was abandoned (no docker.sock on the pod its only run landed on).
# sm-standard-2 has no docker.sock either (weekly runs failed at "Enable
# buildx" from 2026-09-27 to 2026-11-08), so keep this job on ubuntu-latest.
# TODO(ci-8): scheduled-failure alerting (auto-open issue on a red weekly
# run) is added by the ci-8 composite action; do not implement it here.
runs-on: ubuntu-latest
timeout-minutes: 120
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Download the image
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: mint-image
path: ${{ runner.temp }}
- name: Load the image
run: docker load --input "${RUNNER_TEMP}/rustfs-ci.tar"
- name: Start RustFS
run: |
@@ -194,8 +149,6 @@ jobs:
-e RUSTFS_SECRET_KEY="${S3_SECRET_KEY}" \
-e RUSTFS_VOLUMES="/data/rustfs{0...3}" \
-e RUSTFS_UNSAFE_BYPASS_DISK_CHECK=true \
-e RUSTFS_S3_STACK="${STACK}" \
-e RUST_LOG=rustfs=info \
-v /tmp/rustfs-mint:/data \
rustfs-ci
@@ -291,50 +244,16 @@ jobs:
mkdir -p artifacts/rustfs-mint
docker logs rustfs-mint > artifacts/rustfs-mint/rustfs.log 2>&1 || true
- name: Prove which stack the server selected
if: always()
run: python3 scripts/dual_stack.py stack --expect "${STACK}" --out artifacts/stack.json artifacts/rustfs-mint
- name: Upload artifacts
if: always() && env.ACT != 'true'
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: mint-${{ matrix.stack }}
name: mint
path: artifacts/**
compare:
name: Compare stacks (report-only until T2.10)
needs: [mint]
if: always()
continue-on-error: true
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Download both legs
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
pattern: mint-{legacy,gateway}
path: legs
# A leg that never uploaded leaves no directory: create both so that the
# comparison reports "no result" for it instead of failing to find it.
- name: Compare the stacks
run: |
# The shell runs with -e; the exit status is reported after the summary is written.
set +e -u -o pipefail
mkdir -p legs/mint-legacy legs/mint-gateway
python3 scripts/mint_stack_diff.py legs/mint-legacy legs/mint-gateway | tee report.txt
status="${PIPESTATUS[0]}"
{ echo '## Mint: gateway versus legacy'; echo; echo '```'; cat report.txt; echo '```'; } >> "$GITHUB_STEP_SUMMARY"
exit "$status"
alert-on-failure:
name: Alert on scheduled failure
needs: [image, mint]
needs: [mint]
# `always()` is required: without it this job is skipped when a needed
# job fails. Alerts only for scheduled runs (backlog#1149 ci-8) — manual
# dispatch failures are already watched by a human.
+7 -279
View File
@@ -87,206 +87,18 @@ jobs:
install-build-packaging-tools: 'false'
install-test-tools: 'false'
- name: Download static console assets
shell: bash
env:
GITHUB_TOKEN: ${{ github.token }}
run: |
# Packaging tools are off on this lane, so setup does not install unzip.
if ! command -v unzip >/dev/null 2>&1; then
sudo apt-get update -qq
sudo apt-get install -y -qq unzip
fi
if ! command -v unzip >/dev/null 2>&1; then
echo "unzip is required to extract console assets" >&2
exit 1
fi
mkdir -p ./rustfs/static
verify_sha256() {
local expected="$1"
local file="$2"
local actual=""
local actual_line=""
if command -v sha256sum >/dev/null 2>&1; then
actual_line=$(sha256sum -- "$file") || return 1
elif command -v shasum >/dev/null 2>&1; then
actual_line=$(shasum -a 256 -- "$file") || return 1
else
echo "No SHA-256 verification tool found" >&2
return 1
fi
actual="${actual_line%%[[:space:]]*}"
if [ "$actual" = "$expected" ]; then
echo "SHA256 verified OK: $actual"
return 0
else
echo "SHA256 mismatch: expected=$expected actual=$actual" >&2
return 1
fi
}
download_console_assets() {
local curl_bin="curl"
local python_bin="python3"
local console_api="https://api.github.com/repos/rustfs/console/releases/latest"
local console_json="console-release.json"
local console_url
local console_sha256
local curl_auth_args=()
console_tag=""
chmod +w ./rustfs/static/LICENSE || true
if ! command -v "$python_bin" >/dev/null 2>&1; then
python_bin="python"
fi
if ! command -v "$python_bin" >/dev/null 2>&1; then
echo "No Python interpreter found for release metadata parsing" >&2
return 1
fi
if [[ -n "${GITHUB_TOKEN:-}" ]]; then
curl_auth_args=(-H "Authorization: Bearer ${GITHUB_TOKEN}" -H "X-GitHub-Api-Version: 2022-11-28")
fi
"$curl_bin" "${curl_auth_args[@]}" --fail -L "$console_api" \
-o "$console_json" --retry 3 --retry-delay 5 --max-time 300 || return 1
read -r console_tag console_url console_sha256 < <("$python_bin" - "$console_json" <<'PY'
import json
import re
import sys
with open(sys.argv[1], encoding="utf-8") as handle:
release = json.load(handle)
tag = release.get("tag_name", "") or "unknown"
for asset in release.get("assets", []):
name = asset.get("name", "")
digest = asset.get("digest", "")
url = asset.get("browser_download_url", "")
if name.startswith("rustfs-console-") and name.endswith(".zip") and digest.startswith("sha256:") and url:
sha256 = digest.split(":", 1)[1]
if not re.fullmatch(r"[0-9a-fA-F]{64}", sha256):
raise SystemExit(f"console zip asset has invalid sha256 digest: {sha256}")
sys.stdout.buffer.write(f"{tag} {url} {sha256}\n".encode("utf-8"))
break
else:
raise SystemExit("no console zip asset with sha256 digest found")
PY
) || return 1
if [[ -z "$console_url" || -z "$console_sha256" ]]; then
echo "Console release metadata is missing URL or SHA-256 digest" >&2
return 1
fi
echo "Console release: ${console_tag}"
echo "Downloading console asset: ${console_url}"
"$curl_bin" --fail -L "$console_url" -o console.zip --retry 3 --retry-delay 5 --max-time 300 || return 1
verify_sha256 "$console_sha256" console.zip || return 2
unzip -o console.zip -d ./rustfs/static || return 2
}
if download_console_assets; then
rm -f console.zip console-release.json
else
status=$?
rm -f console.zip console-release.json
if [[ "$status" -eq 2 ]]; then
echo "Console asset integrity verification failed" >&2
fi
echo "Failed to download verified console assets" >&2
exit 1
fi
if [[ ! -s ./rustfs/static/index.html ]]; then
echo "Console asset archive is missing static/index.html" >&2
exit 1
fi
asset_count=$(find ./rustfs/static -type f | wc -l | tr -d '[:space:]')
echo "Console assets ready: version=${console_tag:-unknown}, ${asset_count} files extracted to ./rustfs/static"
- name: Build RustFS
# Pin the glibc baseline (`<target>.<glibc>` for cargo-zigbuild) so
# nightlies run on older distros such as RHEL/Rocky 9 (glibc 2.34); #8062.
shell: bash
run: |
# Force rebuild by touching build.rs so a restored cache cannot
# reuse a rustfs artifact compiled before these console assets existed.
touch rustfs/build.rs
cargo zigbuild --release --locked --target x86_64-unknown-linux-gnu.2.31 -p rustfs --bin rustfs
run: cargo zigbuild --release --locked --target x86_64-unknown-linux-gnu.2.31 -p rustfs --bins
- name: Verify glibc symbol baseline
run: |
python3 scripts/release/check_glibc.py 2.31 \
target/x86_64-unknown-linux-gnu/release/rustfs
target/x86_64-unknown-linux-gnu/release/rustfs \
target/x86_64-unknown-linux-gnu/release/rustfs-cli
target/x86_64-unknown-linux-gnu/release/rustfs --version | grep -Fx 'build target : x86_64-unknown-linux-gnu'
- name: Verify console
shell: bash
run: |
set -euo pipefail
find_free_port() {
python3 - <<'PY'
import socket
with socket.socket() as sock:
sock.bind(("127.0.0.1", 0))
print(sock.getsockname()[1])
PY
}
binary="target/x86_64-unknown-linux-gnu/release/rustfs"
data_dir="$(mktemp -d)"
log_file="${RUNNER_TEMP}/rustfs-console-smoke.log"
server_pid=""
trap '
if [[ -n "${server_pid:-}" ]]; then
kill "$server_pid" 2>/dev/null || true
wait "$server_pid" 2>/dev/null || true
fi
rm -rf "$data_dir"
' EXIT
if [[ ! -x "$binary" ]]; then
echo "rustfs binary not found: ${binary}" >&2
exit 1
fi
api_port="$(find_free_port)"
console_port="$(find_free_port)"
console_url="http://127.0.0.1:${console_port}/rustfs/console/"
"$binary" server \
--address "127.0.0.1:${api_port}" \
--console-enable \
--console-address "127.0.0.1:${console_port}" \
--access-key console-smoke \
--secret-key console-smoke-secret \
"$data_dir" >"$log_file" 2>&1 &
server_pid=$!
response=""
for _ in {1..40}; do
response="$(curl --silent --output /dev/null --write-out '%{http_code} %{content_type}' "$console_url" || true)"
if [[ "$response" == 200\ text/html* ]]; then
exit 0
fi
sleep 0.25
done
echo "Console endpoint did not return 200 text/html: $response" >&2
cat "$log_file"
exit 1
- name: Build DEB package
id: deb
shell: bash
@@ -534,12 +346,12 @@ jobs:
path: ${{ steps.publish.outputs.candidate_file }}
if-no-files-found: error
# Publish the deb/rpm pair to the auto-testing repo's `assets` branch so
# engineers can download and install the nightly directly. The branch is
# a single-commit orphan rewritten on every build, which keeps the repo
# small while the latest files stay reachable at stable raw URLs.
# Publish the deb/rpm pair as assets of the rolling `nightly` release on
# rustfs/auto-testing (see scripts/release/publish_nightly_assets.sh).
# The dispatch below runs only after this upload and the immutable R2
# candidate both succeed. It does not replace rustfs-functional-chain.yml,
# which still starts from this workflow's workflow_run completion.
# kms-vault-lane and kms-vault-ha-failover stay parallel with this job.
- name: Publish packages to auto-testing release assets
env:
ASSETS_TOKEN: ${{ secrets.PF_TESTING_GH_TOKEN }}
@@ -548,90 +360,6 @@ jobs:
DEB_DATE: ${{ steps.deb.outputs.deb_date }}
BUILD_REF: ${{ env.NIGHTLY_BUILD_REF }}
run: bash scripts/release/publish_nightly_assets.sh
# Cross-repo repository_dispatch cannot use GITHUB_TOKEN. The secret is a
# PAT or GitHub App token for rustfs/auto-testing: classic `repo` scope,
# or fine-grained Contents write (the permission GitHub checks for
# POST /repos/{owner}/{repo}/dispatches). A dispatch failure fails this
# step, and therefore this job, so the scheduled failure alert fires.
# GitHub allows at most 10 top-level client_payload properties, so the
# schema 2 candidate object is nested under `candidate` and the release
# fields sit beside it.
- name: Dispatch nightly candidate to auto-testing
if: ${{ steps.publish.outputs.candidate_file != '' }}
env:
DISPATCH_TOKEN: ${{ secrets.AUTO_TESTING_DISPATCH_TOKEN }}
CANDIDATE_FILE: ${{ steps.publish.outputs.candidate_file }}
DEB_DATE: ${{ steps.deb.outputs.deb_date }}
SOURCE_REPO: ${{ github.repository }}
shell: bash
run: |
set -euo pipefail
: "${DISPATCH_TOKEN:?AUTO_TESTING_DISPATCH_TOKEN is required to dispatch rustfs/auto-testing}"
: "${CANDIDATE_FILE:?nightly candidate manifest is required}"
: "${DEB_DATE:?DEB_DATE is required}"
[[ "${DEB_DATE}" =~ ^[0-9]{4}-[0-9]{2}-[0-9]{2}$ ]]
[[ "${SOURCE_REPO}" == "rustfs/rustfs" ]]
payload="${RUNNER_TEMP}/nightly-candidate-dispatch-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}.json"
jq -n --slurpfile candidate "${CANDIDATE_FILE}" \
--arg event_type "nightly-candidate-ready" \
--arg source_repo "${SOURCE_REPO}" \
--arg latest_alias_url "https://dl.rustfs.com/artifacts/rustfs/packages/nightly/rustfs-nightly-latest.deb" \
--arg auto_testing_release "https://github.com/rustfs/auto-testing/releases/tag/nightly" \
--arg channel "nightly" \
--arg deb_date "${DEB_DATE}" \
--arg nightly_workflow ".github/workflows/nightly-gnu.yml" \
'{
event_type: $event_type,
source_repo: $source_repo,
latest_alias_url: $latest_alias_url,
auto_testing_release: $auto_testing_release,
channel: $channel,
deb_date: $deb_date,
nightly_workflow: $nightly_workflow,
candidate: $candidate[0]
}' > "${payload}"
if ! jq -e '
.event_type == "nightly-candidate-ready"
and .source_repo == "rustfs/rustfs"
and .channel == "nightly"
and .deb_date == env.DEB_DATE
and .nightly_workflow == ".github/workflows/nightly-gnu.yml"
and .latest_alias_url == "https://dl.rustfs.com/artifacts/rustfs/packages/nightly/rustfs-nightly-latest.deb"
and .auto_testing_release == "https://github.com/rustfs/auto-testing/releases/tag/nightly"
and .candidate.schema == 2
and ((.candidate | keys | sort) == ["build_run_attempt","build_run_id","package_sha256","package_url","schema","source_ref","source_sha","workflow_sha"])
and ((keys | sort) == ["auto_testing_release","candidate","channel","deb_date","event_type","latest_alias_url","nightly_workflow","source_repo"])
' "${payload}" >/dev/null; then
echo "nightly dispatch payload does not match the nightly-candidate-ready contract" >&2
exit 1
fi
body="${RUNNER_TEMP}/nightly-candidate-dispatch-body-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}.json"
jq -n --slurpfile client_payload "${payload}" \
'{event_type: "nightly-candidate-ready", client_payload: $client_payload[0]}' > "${body}"
response="${RUNNER_TEMP}/nightly-candidate-dispatch-response-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}.txt"
status="$(curl -sS --retry 3 --connect-timeout 15 --max-time 60 \
-o "${response}" -w '%{http_code}' \
-X POST \
-H "Authorization: Bearer ${DISPATCH_TOKEN}" \
-H "Accept: application/vnd.github+json" \
-H "X-GitHub-Api-Version: 2022-11-28" \
-H "Content-Type: application/json" \
--data-binary @"${body}" \
https://api.github.com/repos/rustfs/auto-testing/dispatches)" || {
echo "repository_dispatch request to rustfs/auto-testing failed before an HTTP status" >&2
exit 1
}
if [[ "${status}" != "204" ]]; then
echo "repository_dispatch to rustfs/auto-testing failed: HTTP ${status}" >&2
if [[ -s "${response}" ]]; then
cat "${response}" >&2
fi
exit 1
fi
echo "dispatched nightly-candidate-ready to rustfs/auto-testing (deb_date=${DEB_DATE}, run=${GITHUB_RUN_ID}, attempt=${GITHUB_RUN_ATTEMPT})"
# Live-Vault lane for the rustfs-kms suite (rustfs/backlog#1774).
#
# RUSTFS_KMS_VAULT_TOKEN is the single switch that adds the Vault KV2 and
+2 -2
View File
@@ -47,8 +47,8 @@ permissions:
contents: read
concurrency:
group: oidc-keycloak-live-${{ github.event_name }}-${{ github.ref }}
cancel-in-progress: ${{ github.event_name != 'schedule' }}
group: oidc-keycloak-live-${{ github.ref }}
cancel-in-progress: true
jobs:
oidc-keycloak-live:
@@ -68,14 +68,11 @@ env:
jobs:
minio-source:
name: MinIO source (read-through, list-through, backfill, ${{ matrix.stack }}${{ matrix.stack == 'gateway' && ', report-only until T2.10' || '' }})
name: MinIO source (read-through, list-through, backfill)
# Keep this scheduled interoperability lane scoped to the upstream repo.
if: github.repository == 'rustfs/rustfs'
runs-on: ubuntu-latest
timeout-minutes: 90
# The gateway leg is report-only until T2.10.
continue-on-error: ${{ matrix.stack == 'gateway' }}
strategy: { fail-fast: false, matrix: { stack: [legacy, gateway] } }
env:
NO_PROXY: 127.0.0.1,localhost
# Fixed credentials of the container this job starts and throws away;
@@ -140,11 +137,6 @@ jobs:
- name: Build the RustFS binary under test
run: python3 scripts/e2e_binary.py build --bins
# A gateway binary that refuses to start skips the suite; the summary says why.
- name: Gateway preflight
if: matrix.stack == 'gateway'
run: python3 scripts/dual_stack.py preflight --binary target/debug/rustfs --out "$RUNNER_TEMP/leg/stack.json"
# The lane selects tests by module, so a rename would quietly shrink it.
# The committed digest in .config/e2e-odm-interop-selection.txt fails
# closed on that.
@@ -154,19 +146,8 @@ jobs:
python3 ./scripts/check_test_wiring.py --check-profile e2e-odm-interop "${NEXTEST_LISTING}"
- name: Run the interop cases against MinIO
env:
RUSTFS_S3_STACK: ${{ matrix.stack }}
RUSTFS_E2E_LOG_DIR: ${{ runner.temp }}/rustfs-e2e-logs
run: python3 scripts/e2e_binary.py run -- cargo nextest run --profile e2e-odm-interop -p e2e_test --no-tests=fail
- name: Prove the stack and upload the leg
if: always()
uses: ./.github/actions/dual-stack-leg
with:
profile: e2e-odm-interop
stack: ${{ matrix.stack }}
logs: ${{ runner.temp }}/rustfs-e2e-logs
- name: Build the MinIO interop report
if: always()
uses: ./.github/actions/odm-interop-report
@@ -190,7 +171,7 @@ jobs:
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: odm-interop-minio-${{ github.run_number }}-${{ github.run_attempt }}-${{ matrix.stack }}
name: odm-interop-minio-${{ github.run_number }}-${{ github.run_attempt }}
path: |
artifacts/odm-interop/minio
target/nextest/e2e-odm-interop/junit.xml
@@ -316,20 +297,6 @@ jobs:
target/nextest/e2e-odm-interop/junit.xml
retention-days: 14
dual-stack-summary:
name: Both-stack diff summary
needs: [minio-source]
if: github.repository == 'rustfs/rustfs' && always()
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- uses: ./.github/actions/dual-stack-summary
with:
profiles: e2e-odm-interop
alert-on-failure:
name: Alert on scheduled failure
needs: [minio-source, cloud-source]
+630 -39
View File
@@ -1,53 +1,644 @@
# Copyright 2026 RustFS Team
# SPDX-License-Identifier: Apache-2.0
name: Build candidate packages
on:
workflow_call:
inputs:
candidate_manifest:
type: string
required: true
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Package Workflow - Build DEB/RPM packages
#
# This workflow builds DEB and RPM packages from pre-built Linux binaries
# and uploads them to Cloudflare R2 and the GitHub release.
#
# Trigger:
# - workflow_run: automatically package after "Build and Release" completes
# for a release tag (the mac/windows/linux binaries are already uploaded
# to the GitHub release before packaging starts)
# - workflow_dispatch: manual fallback with a release tag and/or exact build run ID
#
# Flow:
# 1. Resolve and validate the selected Build workflow run and source identity
# 2. Download Linux binaries (x86_64-gnu, aarch64-gnu, x86_64-musl,
# aarch64-musl) from build artifacts
# 3. Build DEB packages for amd64 and arm64, gnu and musl variants
# 4. Build RPM packages for x86_64 and aarch64, gnu and musl variants
# 5. Upload all packages to Cloudflare R2 and the GitHub release
name: Package DEB/RPM
permissions:
contents: read
# contents: write is required to upload packages to the GitHub release
contents: write
actions: read
on:
# Follows the same pattern as docker.yml: run after the release build
# workflow completes, so packaging is triggered only by release tags
# (e.g. 1.0.0-rc.2, 1.0.0-rc.3), never by development builds.
workflow_run:
workflows: [ "Build and Release" ]
types: [ completed ]
workflow_dispatch:
inputs:
tag:
description: "Release tag to package (e.g. 1.0.0-beta.12). Leave empty for latest main build."
required: false
type: string
build_run_id:
description: "Build workflow run ID (when combined with tag, both must identify the same release commit)"
required: false
type: string
concurrency:
group: ${{ github.workflow }}-${{ github.event.workflow_run.head_branch || github.event.inputs.tag || github.run_id }}
cancel-in-progress: true
env:
HEAD_BRANCH: ${{ github.event.workflow_run.head_branch }}
WORKFLOW_RUN_ID: ${{ github.event.workflow_run.id }}
jobs:
# Resolve which build run to use and extract version info
resolve:
name: Resolve Build
# Auto-trigger only from successful tag builds of "Build and Release".
# Tag pushes arrive as event == push with head_branch != main (a
# non-main push head_branch is the release tag name). Manual dispatch
# stays available as a fallback for backfills and re-runs.
if: >-
github.event_name == 'workflow_dispatch' ||
(github.event.workflow_run.conclusion == 'success' &&
github.event.workflow_run.event == 'push' &&
github.event.workflow_run.head_branch != 'main')
runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
timeout-minutes: 10
outputs:
version: ${{ steps.resolve.outputs.version }}
build_type: ${{ steps.resolve.outputs.build_type }}
build_run_id: ${{ steps.resolve.outputs.build_run_id }}
build_run_number: ${{ steps.resolve.outputs.build_run_number }}
head_sha: ${{ steps.resolve.outputs.head_sha }}
dev_sequence: ${{ steps.resolve.outputs.dev_sequence }}
tag: ${{ steps.resolve.outputs.tag }}
steps:
- name: Resolve build run
id: resolve
shell: bash
env:
GH_TOKEN: ${{ github.token }}
EVENT_NAME: ${{ github.event_name }}
REPOSITORY: ${{ github.repository }}
INPUT_TAG: ${{ github.event.inputs.tag }}
INPUT_RUN_ID: ${{ github.event.inputs.build_run_id }}
run: |
set -euo pipefail
fail() {
echo "❌ $1" >&2
exit 1
}
TAG=""
BUILD_RUN_ID=""
case "$EVENT_NAME" in
workflow_run)
TAG="$HEAD_BRANCH"
BUILD_RUN_ID="$WORKFLOW_RUN_ID"
;;
workflow_dispatch)
TAG="$INPUT_TAG"
BUILD_RUN_ID="$INPUT_RUN_ID"
;;
*) fail "unsupported event: $EVENT_NAME" ;;
esac
# Validate and classify tags before using them in API paths or logs.
semver_core='(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)'
prerelease_id='(alpha|beta|rc)\.(0|[1-9][0-9]*)'
if [[ -n "$TAG" ]]; then
if [[ "$TAG" =~ ^${semver_core}(-${prerelease_id})?-preview\.(0|[1-9][0-9]*)$ ]]; then
BUILD_TYPE=preview
elif [[ "$TAG" =~ ^${semver_core}-${prerelease_id}$ ]]; then
BUILD_TYPE=prerelease
elif [[ "$TAG" =~ ^${semver_core}$ ]]; then
BUILD_TYPE=release
else
fail "tag is not a supported strict package version"
fi
else
BUILD_TYPE=development
fi
if [[ -n "$BUILD_RUN_ID" ]]; then
[[ "$BUILD_RUN_ID" =~ ^[1-9][0-9]*$ ]] || fail "build run ID must be a positive decimal integer"
echo "Using selected build run: $BUILD_RUN_ID"
elif [[ -n "$TAG" ]]; then
echo "Looking for build run for tag: $TAG"
BUILD_RUN_ID=$(gh api --method GET \
"repos/${REPOSITORY}/actions/workflows/build.yml/runs" \
-f branch="$TAG" -f status=success -F per_page=1 \
--jq '.workflow_runs[0].id // empty' 2>/dev/null || true)
if [[ -z "$BUILD_RUN_ID" ]]; then
BUILD_RUN_ID=$(gh api --method GET \
"repos/${REPOSITORY}/actions/workflows/build.yml/runs" \
-f event=push -f status=success -F per_page=100 2>/dev/null |
jq -r --arg tag "$TAG" \
'[.workflow_runs[] | select(.head_branch == $tag)][0].id // empty' || true)
fi
[[ "$BUILD_RUN_ID" =~ ^[1-9][0-9]*$ ]] || fail "no successful build run found for tag"
echo "Found build run: $BUILD_RUN_ID"
else
echo "No tag specified, looking for latest main build"
BUILD_RUN_ID=$(gh api --method GET \
"repos/${REPOSITORY}/actions/workflows/build.yml/runs" \
-f branch=main -f status=success -F per_page=1 \
--jq '.workflow_runs[0].id // empty' 2>/dev/null || true)
[[ "$BUILD_RUN_ID" =~ ^[1-9][0-9]*$ ]] || fail "no successful main build found"
echo "Latest main build: $BUILD_RUN_ID"
fi
# Fetch once and use the same immutable run metadata for identity,
# ordering, workflow provenance, and release-channel validation.
RUN_JSON=$(gh api "repos/${REPOSITORY}/actions/runs/${BUILD_RUN_ID}") ||
fail "cannot read selected build run"
RUN_ID=$(jq -r '.id // empty' <<<"$RUN_JSON")
RUN_NUMBER=$(jq -r '.run_number // empty' <<<"$RUN_JSON")
RUN_STATUS=$(jq -r '.status // empty' <<<"$RUN_JSON")
RUN_CONCLUSION=$(jq -r '.conclusion // empty' <<<"$RUN_JSON")
RUN_PATH=$(jq -r '.path // empty' <<<"$RUN_JSON")
HEAD_SHA=$(jq -r '.head_sha // empty' <<<"$RUN_JSON")
RUN_HEAD_BRANCH=$(jq -r '.head_branch // empty' <<<"$RUN_JSON")
[[ "$RUN_ID" == "$BUILD_RUN_ID" ]] || fail "run metadata ID mismatch"
[[ "$RUN_NUMBER" =~ ^[1-9][0-9]*$ ]] || fail "build run number must be a positive decimal integer"
[[ "$RUN_STATUS" == completed && "$RUN_CONCLUSION" == success ]] || fail "selected build run is not successful"
[[ "$RUN_PATH" == .github/workflows/build.yml ]] || fail "selected run is not Build and Release"
[[ "$HEAD_SHA" =~ ^[0-9a-f]{40}$ ]] || fail "selected build run has an invalid head SHA"
[[ "$RUN_HEAD_BRANCH" != *$'\n'* && -n "$RUN_HEAD_BRANCH" ]] || fail "selected build run has an invalid head branch"
if [[ -n "$TAG" ]]; then
[[ "$RUN_HEAD_BRANCH" == "$TAG" ]] || fail "tag and build run head branch do not match"
TAG_REF_JSON=$(gh api "repos/${REPOSITORY}/git/ref/tags/${TAG}") ||
fail "cannot resolve release tag ref"
TAG_OBJECT_TYPE=$(jq -r '.object.type // empty' <<<"$TAG_REF_JSON")
TAG_OBJECT_SHA=$(jq -r '.object.sha // empty' <<<"$TAG_REF_JSON")
depth=0
while [[ "$TAG_OBJECT_TYPE" == tag && $depth -lt 5 ]]; do
TAG_OBJECT_JSON=$(gh api "repos/${REPOSITORY}/git/tags/${TAG_OBJECT_SHA}") ||
fail "cannot peel annotated release tag"
TAG_OBJECT_TYPE=$(jq -r '.object.type // empty' <<<"$TAG_OBJECT_JSON")
TAG_OBJECT_SHA=$(jq -r '.object.sha // empty' <<<"$TAG_OBJECT_JSON")
depth=$((depth + 1))
done
[[ "$TAG_OBJECT_TYPE" == commit && "$TAG_OBJECT_SHA" =~ ^[0-9a-f]{40}$ ]] ||
fail "release tag does not resolve to a commit"
[[ "$TAG_OBJECT_SHA" == "$HEAD_SHA" ]] || fail "release tag commit and build run head SHA do not match"
VERSION="$TAG"
DEV_SEQUENCE=""
else
VERSION="dev-${HEAD_SHA}"
DEV_SEQUENCE="$RUN_NUMBER"
fi
{
echo "version=$VERSION"
echo "build_type=$BUILD_TYPE"
echo "build_run_id=$BUILD_RUN_ID"
echo "build_run_number=$RUN_NUMBER"
echo "head_sha=$HEAD_SHA"
echo "dev_sequence=$DEV_SEQUENCE"
echo "tag=${TAG}"
} >> "$GITHUB_OUTPUT"
echo "📊 Resolved:"
echo " Version: $VERSION"
echo " Build type: $BUILD_TYPE"
echo " Build run ID: $BUILD_RUN_ID"
echo " Build run number: $RUN_NUMBER"
# Build DEB and RPM packages for each architecture and libc variant
package:
runs-on: ubuntu-latest
timeout-minutes: 45
name: Package (${{ matrix.arch }}-${{ matrix.libc }})
needs: resolve
runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
timeout-minutes: 30
strategy:
fail-fast: false
max-parallel: 1
matrix:
target: [linux-x86_64-musl, linux-aarch64-musl, linux-x86_64-gnu, linux-aarch64-gnu]
include:
- arch: x86_64
libc: gnu
deb_arch: amd64
rpm_arch: x86_64
artifact_name: "rustfs-linux-x86_64-gnu"
# gnu binaries dynamically link glibc; musl binaries are static
# and run without a libc dependency.
deb_depends: "libc6 (>= 2.31)"
rpm_depends: "glibc >= 2.31"
- arch: aarch64
libc: gnu
deb_arch: arm64
rpm_arch: aarch64
artifact_name: "rustfs-linux-aarch64-gnu"
deb_depends: "libc6 (>= 2.31)"
rpm_depends: "glibc >= 2.31"
- arch: x86_64
libc: musl
deb_arch: amd64
rpm_arch: x86_64
artifact_name: "rustfs-linux-x86_64-musl"
deb_depends: ""
rpm_depends: ""
- arch: aarch64
libc: musl
deb_arch: arm64
rpm_arch: aarch64
artifact_name: "rustfs-linux-aarch64-musl"
deb_depends: ""
rpm_depends: ""
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
ref: ${{ fromJson(inputs.candidate_manifest).source_sha }}
persist-credentials: false
- name: Reuse completed package set
id: reuse
- name: Normalize package metadata
id: versions
shell: bash
env:
GH_TOKEN: ${{ github.token }}
CANDIDATE_MANIFEST: ${{ inputs.candidate_manifest }}
TARGET: ${{ matrix.target }}
run: python3 scripts/release/deliver.py reuse --kind packages --target "$TARGET"
- name: Install packaging tools
if: steps.reuse.outputs.reused != 'true'
BUILD_TYPE: ${{ needs.resolve.outputs.build_type }}
SOURCE_VERSION: ${{ needs.resolve.outputs.version }}
DEV_SEQUENCE: ${{ needs.resolve.outputs.dev_sequence }}
ARCH: ${{ matrix.rpm_arch }}
LIBC: ${{ matrix.libc }}
run: |
sudo apt-get update
sudo apt-get install -y ruby ruby-dev build-essential rpm fakeroot cpio
sudo gem install fpm --version 1.17.0 --no-document
- name: Package accepted executable and verify contents
if: steps.reuse.outputs.reused != 'true'
set -euo pipefail
normalized=$(./scripts/release/package_versions.sh \
"$BUILD_TYPE" "$SOURCE_VERSION" "$DEV_SEQUENCE" "$ARCH" "$LIBC")
printf '%s\n' "$normalized" >> "$GITHUB_OUTPUT"
- name: Download binary artifact from build run
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
pattern: ${{ matrix.artifact_name }}*
path: ./binary-artifact
run-id: ${{ needs.resolve.outputs.build_run_id }}
github-token: ${{ github.token }}
merge-multiple: true
- name: Extract binary
id: binary
shell: bash
run: |
set -euo pipefail
ZIP_FILE=$(find ./binary-artifact -name "*.zip" -type f | head -1)
if [[ -z "$ZIP_FILE" ]]; then
echo "❌ No binary artifact found"
find ./binary-artifact -mindepth 1 -maxdepth 1 -print 2>/dev/null || true
exit 1
fi
echo "Found artifact: $ZIP_FILE"
mkdir -p ./bin
unzip -o "$ZIP_FILE" -d ./bin
if [[ ! -f ./bin/rustfs ]]; then
echo "❌ rustfs binary not found in archive"
exit 1
fi
chmod +x ./bin/rustfs
stat --printf='%n %s bytes\n' ./bin/rustfs
echo "✅ Binary extracted"
- name: Build DEB package
id: deb
shell: bash
env:
DEB_VERSION: ${{ steps.versions.outputs.deb_version }}
DEB_ARCH: ${{ matrix.deb_arch }}
DEB_FILE: ${{ steps.versions.outputs.deb_file }}
DEB_DEPENDS: ${{ matrix.deb_depends }}
LIBC: ${{ matrix.libc }}
run: |
set -euo pipefail
PKG_DIR="${DEB_FILE%.deb}"
echo "Building DEB: ${DEB_FILE}"
mkdir -p "${PKG_DIR}/DEBIAN"
mkdir -p "${PKG_DIR}/usr/bin"
mkdir -p "${PKG_DIR}/etc/default"
mkdir -p "${PKG_DIR}/lib/systemd/system"
mkdir -p "${PKG_DIR}/usr/share/doc/rustfs"
cp ./bin/rustfs "${PKG_DIR}/usr/bin/"
chmod 755 "${PKG_DIR}/usr/bin/rustfs"
cp deploy/build/rustfs.service "${PKG_DIR}/lib/systemd/system/"
cat > "${PKG_DIR}/etc/default/rustfs" << 'ENVEOF'
# RustFS Environment Configuration
# See https://docs.rustfs.com for more information
# RUSTFS_ACCESS_KEY=<changeme>
# RUSTFS_SECRET_KEY=<changeme>
# RUSTFS_VOLUMES=""
# RUSTFS_ADDRESS=":9000"
# RUSTFS_CONSOLE_ADDRESS=":9001"
# RUSTFS_CONSOLE_ENABLE=true
# RUSTFS_OBS_LOGGER_LEVEL=error
# RUSTFS_OBS_LOG_DIRECTORY="/var/log/rustfs/"
ENVEOF
cat > "${PKG_DIR}/DEBIAN/control" << EOF
Package: rustfs
Version: ${DEB_VERSION}
Section: utils
Priority: optional
Architecture: ${DEB_ARCH}
EOF
# gnu binaries dynamically link glibc and must declare it; musl
# binaries are statically linked and need no libc dependency.
if [[ -n "$DEB_DEPENDS" ]]; then
echo "Depends: ${DEB_DEPENDS}" >> "${PKG_DIR}/DEBIAN/control"
fi
cat >> "${PKG_DIR}/DEBIAN/control" << EOF
Maintainer: RustFS Team <support@rustfs.com>
Description: High-performance distributed object storage (${LIBC} build)
RustFS is a high-performance distributed object storage software
built using Rust. It is compatible with MinIO and S3 API.
Homepage: https://rustfs.com
EOF
# Declare /etc/default/rustfs as a conffile so dpkg preserves user
# modifications on upgrade instead of silently overwriting them.
cat > "${PKG_DIR}/DEBIAN/conffiles" << 'CONFFILES'
/etc/default/rustfs
CONFFILES
./scripts/release/package_service_scripts.sh after-install > "${PKG_DIR}/DEBIAN/postinst"
chmod 755 "${PKG_DIR}/DEBIAN/postinst"
./scripts/release/package_service_scripts.sh before-remove > "${PKG_DIR}/DEBIAN/prerm"
chmod 755 "${PKG_DIR}/DEBIAN/prerm"
./scripts/release/package_service_scripts.sh after-remove > "${PKG_DIR}/DEBIAN/postrm"
chmod 755 "${PKG_DIR}/DEBIAN/postrm"
cp LICENSE "${PKG_DIR}/usr/share/doc/rustfs/"
cp README.md "${PKG_DIR}/usr/share/doc/rustfs/"
fakeroot dpkg-deb --build "${PKG_DIR}" "$DEB_FILE"
[[ $(dpkg-deb -f "$DEB_FILE" Package) == rustfs ]]
[[ $(dpkg-deb -f "$DEB_FILE" Version) == "$DEB_VERSION" ]]
[[ $(dpkg-deb -f "$DEB_FILE" Architecture) == "$DEB_ARCH" ]]
dpkg-deb --fsys-tarfile "$DEB_FILE" | tar -tf - | grep -Fx './usr/bin/rustfs' >/dev/null
stat --printf='%n %s bytes\n' "$DEB_FILE"
echo "deb_file=$DEB_FILE" >> "$GITHUB_OUTPUT"
echo "✅ DEB built: $DEB_FILE"
- name: Build RPM package
id: rpm
shell: bash
env:
RPM_VERSION: ${{ steps.versions.outputs.rpm_version }}
RPM_RELEASE: ${{ steps.versions.outputs.rpm_release }}
RPM_ARCH: ${{ matrix.rpm_arch }}
RPM_FILE: ${{ steps.versions.outputs.rpm_file }}
RPM_DEPENDS: ${{ matrix.rpm_depends }}
LIBC: ${{ matrix.libc }}
run: |
set -euo pipefail
echo "Building RPM for ${RPM_ARCH} (${LIBC})"
sudo apt-get update && sudo apt-get install -y ruby ruby-dev build-essential rpm
sudo gem install fpm
./scripts/test_package_versions.sh --require-package-managers
# gnu binaries dynamically link glibc and must declare it; musl
# binaries are statically linked and need no libc dependency.
depends_args=()
if [[ -n "$RPM_DEPENDS" ]]; then
depends_args+=(--depends "$RPM_DEPENDS")
fi
# Create config file for fpm (DEB build creates it in its package dir structure,
# but fpm needs the file to exist before packaging)
mkdir -p ./tmp-pkg/etc/default
cat > ./tmp-pkg/etc/default/rustfs << 'ENVEOF'
# RustFS Environment Configuration
# See https://docs.rustfs.com for more information
# RUSTFS_ACCESS_KEY=<changeme>
# RUSTFS_SECRET_KEY=<changeme>
# RUSTFS_VOLUMES=""
# RUSTFS_ADDRESS=":9000"
# RUSTFS_CONSOLE_ADDRESS=":9001"
# RUSTFS_CONSOLE_ENABLE=true
# RUSTFS_OBS_LOGGER_LEVEL=error
# RUSTFS_OBS_LOG_DIRECTORY="/var/log/rustfs/"
ENVEOF
fpm -s dir -t rpm \
--name rustfs \
--version "$RPM_VERSION" \
--iteration "$RPM_RELEASE" \
--architecture "$RPM_ARCH" \
--package "$RPM_FILE" \
"${depends_args[@]}" \
--description "High-performance distributed object storage (${LIBC} build)" \
--maintainer "RustFS Team <support@rustfs.com>" \
--url "https://rustfs.com" \
--license "Apache-2.0" \
--before-install <(./scripts/release/package_service_scripts.sh rpm-before-install) \
--before-upgrade <(./scripts/release/package_service_scripts.sh rpm-before-upgrade) \
--after-install <(./scripts/release/package_service_scripts.sh after-install) \
--after-upgrade <(./scripts/release/package_service_scripts.sh after-install) \
--before-remove <(./scripts/release/package_service_scripts.sh before-remove) \
--after-remove <(./scripts/release/package_service_scripts.sh after-remove) \
--rpm-posttrans <(./scripts/release/package_service_scripts.sh rpm-posttrans) \
--config-files /etc/default/rustfs \
./bin/rustfs=/usr/bin/rustfs \
./tmp-pkg/etc/default/rustfs=/etc/default/rustfs \
deploy/build/rustfs.service=/lib/systemd/system/rustfs.service \
LICENSE=/usr/share/doc/rustfs/LICENSE \
README.md=/usr/share/doc/rustfs/README.md
if [[ ! -f "$RPM_FILE" ]]; then
echo "❌ RPM build failed"
exit 1
fi
RPM_METADATA=$(rpm -qp --qf '%{NAME}\n%{VERSION}\n%{RELEASE}\n%{ARCH}\n' "$RPM_FILE")
EXPECTED_METADATA=$(printf 'rustfs\n%s\n%s\n%s' "$RPM_VERSION" "$RPM_RELEASE" "$RPM_ARCH")
[[ "$RPM_METADATA" == "$EXPECTED_METADATA" ]]
rpm -qpl "$RPM_FILE" | grep -Fx '/usr/bin/rustfs' >/dev/null
stat --printf='%n %s bytes\n' "$RPM_FILE"
echo "rpm_file=$RPM_FILE" >> "$GITHUB_OUTPUT"
echo "✅ RPM built: $RPM_FILE"
- name: Upload packages to artifacts
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: packages-${{ matrix.arch }}-${{ matrix.libc }}
path: |
*.deb
*.rpm
retention-days: 30
- name: Upload packages to Cloudflare R2
if: needs.resolve.outputs.build_type != 'preview' && env.R2_ACCESS_KEY_ID != ''
env:
R2_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
R2_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
R2_ENDPOINT: ${{ secrets.R2_ENDPOINT }}
R2_BUCKET: ${{ secrets.R2_BUCKET }}
AWS_EC2_METADATA_DISABLED: true
BUILD_TYPE: ${{ needs.resolve.outputs.build_type }}
DEB_FILE: ${{ steps.deb.outputs.deb_file }}
RPM_FILE: ${{ steps.rpm.outputs.rpm_file }}
shell: bash
run: |
set -euo pipefail
if [[ -z "$R2_ACCESS_KEY_ID" || -z "$R2_SECRET_ACCESS_KEY" || -z "$R2_ENDPOINT" || -z "$R2_BUCKET" ]]; then
echo "⚠️ R2 credentials missing, skipping upload"
exit 0
fi
if ! command -v aws >/dev/null 2>&1; then
sudo apt-get update && sudo apt-get install -y awscli
fi
export AWS_ACCESS_KEY_ID="$R2_ACCESS_KEY_ID"
export AWS_SECRET_ACCESS_KEY="$R2_SECRET_ACCESS_KEY"
export AWS_DEFAULT_REGION="auto"
if [[ "$BUILD_TYPE" == "development" ]]; then
R2_PREFIX="artifacts/rustfs/packages/dev"
else
R2_PREFIX="artifacts/rustfs/packages/release"
fi
R2_PATH="s3://${R2_BUCKET}/${R2_PREFIX}/"
echo "📤 Uploading to $R2_PATH"
for f in "$DEB_FILE" "$RPM_FILE"; do
if [[ -n "$f" && -f "$f" ]]; then
echo "Uploading: $f"
aws s3 cp "$f" "$R2_PATH" --endpoint-url "$R2_ENDPOINT" --only-show-errors
fi
done
echo "✅ Upload complete"
# Also upload as latest for release/prerelease
if [[ "$BUILD_TYPE" == "release" || "$BUILD_TYPE" == "prerelease" ]]; then
LATEST_PATH="s3://${R2_BUCKET}/artifacts/rustfs/packages/latest/"
for f in "$DEB_FILE" "$RPM_FILE"; do
if [[ -n "$f" && -f "$f" ]]; then
echo "Uploading latest: $(basename "$f")"
aws s3 cp "$f" "$LATEST_PATH" --endpoint-url "$R2_ENDPOINT" --only-show-errors
fi
done
echo "✅ Latest packages updated"
fi
- name: Upload packages to GitHub Release
if: needs.resolve.outputs.tag != ''
env:
GH_TOKEN: ${{ github.token }}
CANDIDATE_MANIFEST: ${{ inputs.candidate_manifest }}
TARGET: ${{ matrix.target }}
run: python3 scripts/release/deliver.py package --target "$TARGET" --output delivery
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
if: steps.reuse.outputs.reused != 'true'
with:
name: packages-${{ matrix.target }}
path: delivery/*
if-no-files-found: error
compression-level: 0
retention-days: 30
TAG: ${{ needs.resolve.outputs.tag }}
DEB_FILE: ${{ steps.deb.outputs.deb_file }}
RPM_FILE: ${{ steps.rpm.outputs.rpm_file }}
shell: bash
run: |
set -euo pipefail
# Upload the packages, then refresh the release checksums so the new
# assets are covered, matching the binary release flow.
for f in "$DEB_FILE" "$RPM_FILE"; do
if [[ -n "$f" && -f "$f" ]]; then
echo "📤 Uploading $(basename "$f") to GitHub release ${TAG}..."
gh release upload "$TAG" "$f" --clobber
fi
done
CHECKSUM_DIR="$(mktemp -d)"
gh release download "$TAG" -p 'SHA256SUMS' -p 'SHA512SUMS' \
-D "$CHECKSUM_DIR" --clobber
for spec in "SHA256SUMS:sha256sum" "SHA512SUMS:sha512sum"; do
asset="${spec%%:*}"
checksum_cmd="${spec##*:}"
checksum_file="${CHECKSUM_DIR}/${asset}"
for f in "$DEB_FILE" "$RPM_FILE"; do
if [[ -n "$f" && -f "$f" ]]; then
base="$(basename "$f")"
# GitHub stores release asset names with '~' normalized to '.'.
# Current package file names embed the raw tag and contain no
# '~', but older releases may still ship such assets, so the
# checksum entries must reference the name as stored on the
# release.
github_base="${base//\~/.}"
# Remove any stale entry (both naming variants), then append
grep -Fv -- "$base" "$checksum_file" > "${checksum_file}.tmp" || true
grep -Fv -- "$github_base" "${checksum_file}.tmp" > "${checksum_file}.tmp2" || true
mv "${checksum_file}.tmp2" "$checksum_file"
digest=$("$checksum_cmd" -- "$f" | awk '{print $1}')
printf '%s %s\n' "$digest" "$github_base" >> "$checksum_file"
fi
done
echo "📤 Updating ${asset} for release ${TAG}..."
gh release upload "$TAG" "$checksum_file" --clobber
done
echo "✅ GitHub release assets updated"
# Summary
summary:
name: Summary
needs: [ resolve, package ]
# Every completed Build and Release run triggers this workflow; when
# resolve declined it (not a release tag), there is nothing to summarise.
if: always() && needs.resolve.result != 'skipped'
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
- name: Print summary
shell: bash
env:
SUMMARY_VERSION: ${{ needs.resolve.outputs.version }}
SUMMARY_BUILD_TYPE: ${{ needs.resolve.outputs.build_type }}
SUMMARY_BUILD_RUN_ID: ${{ needs.resolve.outputs.build_run_id }}
SUMMARY_PACKAGE_STATUS: ${{ needs.package.result }}
run: |
{
echo "## 📦 Package Summary"
echo ""
echo "| Item | Value |"
echo "|------|-------|"
echo "| Version | \`${SUMMARY_VERSION}\` |"
echo "| Build Type | ${SUMMARY_BUILD_TYPE} |"
echo "| Build Run | #${SUMMARY_BUILD_RUN_ID} |"
echo "| Package Status | ${SUMMARY_PACKAGE_STATUS} |"
} >> "$GITHUB_STEP_SUMMARY"
-91
View File
@@ -1,91 +0,0 @@
name: S3-over-RDMA feature check
# The `nixl` and `cuobject` features are off in every other build, so the code
# behind them is compiled nowhere else. This job compiles the `nixl` half on
# Linux and runs rustfs-nixl's unit tests; nixl-sys builds through its stub
# fallback, so no NIXL SDK is needed.
# The cuObject half needs the NVIDIA cuObject SDK and is not compiled here.
# Not a required check and not part of the `Test and Lint` aggregate.
on:
pull_request:
paths:
- "crates/nixl/**"
- "crates/cuobject/**"
- "rustfs/src/storage/rpc/nixl*"
- "rustfs/src/app/object/cuobject.rs"
- "rustfs/src/app/object/get.rs"
- "rustfs/src/app/object/put.rs"
- "rustfs/src/app/multipart_usecase/upload_part.rs"
- "Cargo.toml"
- "rustfs/Cargo.toml"
- ".github/workflows/rdma-feature-check.yml"
schedule:
- cron: "37 5 * * 1"
workflow_dispatch:
permissions:
contents: read
concurrency:
group: rdma-feature-check-${{ github.ref }}
cancel-in-progress: true
jobs:
nixl-check:
name: cargo check and unit tests (nixl)
runs-on: sm-standard-4
timeout-minutes: 60
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: stable
cache-shared-key: ci-feat-nixl
cache-save-if: 'false'
install-build-packaging-tools: 'false'
- name: Install only missing native build tools
run: |
packages=()
command -v g++ >/dev/null || packages+=(g++)
ldconfig -p | grep -q 'libclang' || packages+=(libclang-dev)
if (( ${#packages[@]} )); then
sudo apt-get update
sudo apt-get install --no-install-recommends -y "${packages[@]}"
fi
- name: Check the nixl feature build
run: cargo check --locked -p rustfs --features nixl
# These unit tests use a fake sysfs and plain host memory; none needs an
# RDMA device or the NIXL libraries.
- name: Run the rustfs-nixl unit tests
run: cargo test --locked -p rustfs-nixl --features sys
alert-on-failure:
name: Alert on scheduled failure
needs: [nixl-check]
# Only scheduled runs open or update the tracking issue; pull-request and
# manual runs stay quiet.
if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
runs-on: ubuntu-latest
timeout-minutes: 10
permissions:
contents: read
issues: write
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Open or update failure-tracking issue
uses: ./.github/actions/schedule-failure-issue
with:
github-token: ${{ secrets.BACKLOG_ISSUE_TOKEN }}
-444
View File
@@ -1,444 +0,0 @@
# Copyright 2026 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# s3-tests reference parity lane (rustfs/backlog#2773).
#
# Runs the ceph/s3-tests implemented list (scripts/s3-tests/implemented_tests.txt)
# against every target in .config/reference-parity-targets.toml and fails on
# any per-node-id difference between a candidate and the reference:
#
# - reference: the release pinned in .config/reference-binary.toml, downloaded
# and sha256-verified by .github/actions/reference-binary;
# - candidate targets: the debug binary built from this checkout exactly as
# ci.yml's build-rustfs-debug-binary job builds it for S3 Implemented Tests.
#
# Every target runs the same harness (scripts/s3-tests/run.sh from this
# checkout), test list and server configuration on its own hosted runner. A
# target's failures get one rerun against a fresh server; a case that passes it
# is flaky, not a difference. scripts/s3-tests/reference_parity.py judges the
# result and honours .config/reference-parity-tolerated.txt, whose entries must
# each match a real difference.
#
# The `differential` job adds the 73-operation dual-stack differential
# (crates/e2e_test/src/dual_stack_differential, rustfs/backlog#2740) against the
# same pinned reference: side A is the released binary (RUSTFS_E2E_REFERENCE_BINARY,
# started without RUSTFS_S3_STACK), side B is the candidate built above on its
# legacy stack. It reuses the candidate build artifact, never builds the candidate
# twice, and prints reference_side, version_a and version_b from the report the test
# wrote; it fails when side A was not the reference or both sides report the same
# version. Every (op, field, a, b) difference goes to the job summary. Report-only
# until the first result has been reviewed.
#
# Report-only: not part of the Test and Lint aggregate. Scheduled failures are
# owned by the scheduled-validation watchdog (.github/scheduled-validations.json).
name: Reference Parity
on:
pull_request:
paths:
- ".github/workflows/reference-parity.yml"
- ".github/actions/reference-binary/**"
- ".config/reference-binary.toml"
- ".config/reference-parity-targets.toml"
- ".config/reference-parity-tolerated.txt"
- "scripts/s3-tests/reference_parity.py"
- "scripts/junit_diff.py"
- "crates/e2e_test/src/dual_stack_differential/**"
schedule:
- cron: "37 1 * * *"
workflow_dispatch:
# Matches ci.yml: these variables are hashed into the rust-cache key, so the
# candidate build reads the same ci-dev cache as build-rustfs-debug-binary.
env:
CARGO_TERM_COLOR: always
RUST_BACKTRACE: 1
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
permissions:
contents: read
defaults:
run:
shell: bash
jobs:
plan:
name: Plan parity targets
runs-on: ubuntu-latest
timeout-minutes: 5
outputs:
targets: ${{ steps.targets.outputs.targets }}
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Validate target list
id: targets
run: |
set -euo pipefail
targets="$(python3 scripts/s3-tests/reference_parity.py targets --config .config/reference-parity-targets.toml)"
echo "targets=${targets}" >> "$GITHUB_OUTPUT"
echo "Parity targets: ${targets}"
build-candidate:
name: Build candidate binary
# Critical-path CPU and cache ownership: the same debug build as ci.yml's
# build-rustfs-debug-binary, reading the ci-dev cache written on sm-standard-4.
runs-on: sm-standard-4
timeout-minutes: 60
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: stable
cache-shared-key: ci-dev-ea7eec74fe14
cache-save-if: 'false'
install-build-packaging-tools: 'false'
- name: Build debug binary
run: python3 scripts/e2e_binary.py build --bins --features e2e-test-hooks
- name: Upload candidate binary
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: reference-parity-candidate-binary
overwrite: true
path: |
target/debug/rustfs
target/debug/rustfs.e2e.json
if-no-files-found: error
retention-days: 1
s3tests:
name: s3-tests (${{ matrix.target.name }})
needs: [ plan, build-candidate ]
runs-on: ubuntu-latest
timeout-minutes: 75
strategy:
fail-fast: false
matrix:
target: ${{ fromJSON(needs.plan.outputs.targets) }}
env:
TARGET_NAME: ${{ matrix.target.name }}
TARGET_KIND: ${{ matrix.target.binary }}
# Above this many first-run failures the target is systemically broken;
# a rerun would only repeat it, so the first run is judged as is.
RERUN_CAP: "50"
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Fetch pinned reference binary
id: reference
if: matrix.target.binary == 'reference'
uses: ./.github/actions/reference-binary
- name: Download candidate binary
if: matrix.target.binary == 'candidate'
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: reference-parity-candidate-binary
path: target/debug
- name: Select target binary
env:
REFERENCE_PATH: ${{ steps.reference.outputs.path }}
run: |
set -euo pipefail
case "$TARGET_KIND" in
reference) binary="$REFERENCE_PATH" ;;
candidate) chmod +x target/debug/rustfs; binary="$PWD/target/debug/rustfs" ;;
*) echo "unknown target binary kind: $TARGET_KIND" >&2; exit 1 ;;
esac
test -x "$binary"
echo "TARGET_BINARY=$binary" >> "$GITHUB_ENV"
mkdir -p "parity/$TARGET_NAME"
{
echo "target=$TARGET_NAME"
echo "binary=$TARGET_KIND"
echo "sha256=$(sha256sum "$binary" | cut -d' ' -f1)"
echo "version=$("$binary" --version | tr '\n' ' ')"
} | tee "parity/$TARGET_NAME/target.txt"
- name: Apply target environment
env:
TARGET_JSON: ${{ toJSON(matrix.target) }}
run: |
set -euo pipefail
python3 scripts/s3-tests/reference_parity.py export-env --target-json "$TARGET_JSON" \
| tee -a "$GITHUB_ENV" "parity/$TARGET_NAME/target-env.txt"
- name: Run implemented s3-tests
run: |
set -uo pipefail
report="parity/$TARGET_NAME"
harness="artifacts/s3tests-single"
# Same invocation as ci.yml's S3 Implemented Tests, with a fresh data
# root and port per run. $1 optionally replaces the test list.
run_suite() {
local list="$1" label="$2" root port
root="${RUNNER_TEMP}/rustfs-s3tests-${label}"
rm -rf "$harness" "$root"
mkdir -p "$root"
port="$(python3 -c 'import socket; s=socket.socket(); s.bind(("127.0.0.1", 0)); print(s.getsockname()[1]); s.close()')"
env DEPLOY_MODE=binary \
RUSTFS_BINARY="$TARGET_BINARY" \
TEST_MODE=single \
MAXFAIL=0 \
XDIST=4 \
S3_PORT="$port" \
DATA_ROOT="$root" \
S3TESTS_CONF="$harness/s3tests.conf" \
${list:+IMPLEMENTED_TESTS_FILE="$list"} \
./scripts/s3-tests/run.sh
}
# A run is usable only if it produced a JUnit report; test failures
# are data for the comparison, a missing report is a lane failure.
keep() {
local prefix="$1"
for file in pytest.log compat-report.md; do
if [[ -f "$harness/$file" ]]; then cp "$harness/$file" "$report/${prefix}${file}"; fi
done
if [[ -f "$harness/rustfs-single/rustfs.log" ]]; then cp "$harness/rustfs-single/rustfs.log" "$report/${prefix}rustfs.log"; fi
test -s "$harness/junit.xml" || { echo "::error::$TARGET_NAME produced no JUnit report (${prefix:-first} run)"; return 1; }
cp "$harness/junit.xml" "$report/${prefix}junit.xml"
}
run_suite "" first
echo "first_run_exit=$?" >> "$report/target.txt"
keep "" || exit 1
cp "$harness/selected-nodeids.txt" "$report/selected-nodeids.txt" || exit 1
failures="$(python3 scripts/s3-tests/reference_parity.py failed --report "$report")" || exit 1
count="$(printf '%s' "$failures" | grep -c . || true)"
echo "first_run_failures=$count" >> "$report/target.txt"
if (( count == 0 )); then
exit 0
fi
if (( count > RERUN_CAP )); then
echo "::warning::$TARGET_NAME has $count failures (cap $RERUN_CAP); judging the first run without a rerun"
echo "rerun=skipped-over-cap" >> "$report/target.txt"
exit 0
fi
printf '%s\n' "$failures" > "$RUNNER_TEMP/rerun-tests.txt"
run_suite "$RUNNER_TEMP/rerun-tests.txt" rerun
echo "rerun_exit=$?" >> "$report/target.txt"
keep rerun- || exit 1
- name: Upload target report
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: reference-parity-report-${{ matrix.target.name }}
overwrite: true
path: parity/${{ matrix.target.name }}/
if-no-files-found: error
retention-days: 14
differential:
name: Dual-stack differential (reference vs candidate)
needs: [ build-candidate ]
# Compiles the e2e_test harness against the ci-dev cache like the other
# sm-standard-4 jobs; the candidate server itself is the downloaded artifact.
runs-on: sm-standard-4
timeout-minutes: 90
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
NO_PROXY: 127.0.0.1,localhost,::1
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: stable
cache-shared-key: ci-dev-ea7eec74fe14
cache-save-if: 'false'
install-build-packaging-tools: 'false'
- name: Fetch pinned reference binary
id: reference
uses: ./.github/actions/reference-binary
- name: Download candidate binary
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: reference-parity-candidate-binary
path: target/debug
- name: Run dual-stack differential
id: differential
continue-on-error: true
env:
REFERENCE_PATH: ${{ steps.reference.outputs.path }}
RUSTFS_E2E_LOG_DIR: ${{ runner.temp }}/rustfs-differential-logs
run: |
set -euo pipefail
export no_proxy="$NO_PROXY"
chmod +x target/debug/rustfs
test -x "$REFERENCE_PATH"
mkdir -p "$RUSTFS_E2E_LOG_DIR"
# Side A: the released binary. Side B: the candidate on its legacy stack.
export RUSTFS_E2E_REFERENCE_BINARY="$REFERENCE_PATH"
test_name='dual_stack_differential::tests::both_stacks_answer_every_operation_identically'
python3 scripts/e2e_binary.py run --features e2e-test-hooks -- \
cargo test --locked -p e2e_test --lib "$test_name" -- --exact --nocapture \
2>&1 | tee "$RUNNER_TEMP/differential-test.log"
- name: Summarize report and verify the reference side
if: always()
run: |
set -euo pipefail
python3 - <<'PY'
import glob, json, os, sys
summary = os.environ["GITHUB_STEP_SUMMARY"]
reports = sorted(glob.glob("target/dual-stack/self-comparison-*.json"))
lines = ["## Dual-stack differential: reference vs candidate", ""]
failures = []
if len(reports) != 1:
failures.append(f"expected exactly one self-comparison report, found {len(reports)}: {reports}")
else:
report = json.load(open(reports[0]))
side = report.get("reference_side")
version_a = report.get("version_a")
version_b = report.get("version_b")
lines += [
f"- report: `{reports[0]}`",
f"- reference_side: `{side}`",
f"- version_a: `{(version_a or '').splitlines()[0] if version_a else version_a}`",
f"- version_b: `{(version_b or '').splitlines()[0] if version_b else version_b}`",
f"- stack_a: `{report.get('stack_a')}`, stack_b: `{report.get('stack_b')}`",
f"- compared_ops: {report.get('compared_ops')}, compared_cases: {report.get('compared_cases')}",
"",
]
print(f"reference_side={side}\nversion_a={version_a}\nversion_b={version_b}")
lines += ["```", "version_a:", version_a or "", "", "version_b:", version_b or "", "```", ""]
if side != "a":
failures.append(f"reference_side is {side!r}, expected 'a'")
if not version_a or not version_b:
failures.append("version_a or version_b is missing")
elif version_a == version_b:
failures.append("version_a equals version_b: the reference was replaced by the candidate")
diffs = report.get("differences") or []
lines.append(f"### Differences: {len(diffs)}")
by_op = {}
for d in diffs:
by_op.setdefault(d["op"], []).append(d)
for op in sorted(by_op):
lines += ["", f"#### {op}", "", "| case | field | a (reference) | b (candidate) |", "| --- | --- | --- | --- |"]
for d in by_op[op]:
cell = lambda v: "`" + str(v).replace("|", "\\|").replace("`", "'") + "`"
lines.append(f"| {d['case']} | {d['field']} | {cell(d['a'])} | {cell(d['b'])} |")
if failures:
lines += ["", "### Observation check failed", ""] + [f"- {f}" for f in failures]
open(summary, "a").write("\n".join(lines) + "\n")
print("\n".join(lines))
for f in failures:
print(f"::error::{f}")
sys.exit(1 if failures else 0)
PY
- name: Upload differential reports
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: reference-differential-report
overwrite: true
path: |
target/dual-stack/*.json
${{ runner.temp }}/differential-test.log
if-no-files-found: warn
retention-days: 14
- name: Fail when the differential found differences
if: steps.differential.outcome == 'failure'
run: |
echo "::error::the dual-stack differential failed; see the job summary for every (op, field, a, b)"
exit 1
compare:
name: Compare targets
needs: [ plan, s3tests ]
if: always() && needs.plan.result == 'success'
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Download target reports
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
pattern: reference-parity-report-*
path: parity-download
- name: Compare per node id
env:
TARGETS: ${{ needs.plan.outputs.targets }}
S3TESTS_RESULT: ${{ needs.s3tests.result }}
run: |
set -uo pipefail
mkdir -p parity
for dir in parity-download/reference-parity-report-*; do
if [[ -d "$dir" ]]; then mv "$dir" "parity/${dir#parity-download/reference-parity-report-}"; fi
done
python3 scripts/s3-tests/reference_parity.py compare \
--targets-json "$TARGETS" \
--reports parity \
--tolerated .config/reference-parity-tolerated.txt \
--markdown "$GITHUB_STEP_SUMMARY" \
--json parity/diff.json > parity/diff.md
status=$?
cat parity/diff.md
if [[ "$S3TESTS_RESULT" != "success" ]]; then
echo "::error::s3-tests target jobs finished with '$S3TESTS_RESULT'"
status=1
fi
exit "$status"
- name: Upload parity diff
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: reference-parity-diff
overwrite: true
path: |
parity/diff.md
parity/diff.json
if-no-files-found: warn
retention-days: 14
-133
View File
@@ -1,133 +0,0 @@
# Copyright 2026 RustFS Team
# SPDX-License-Identifier: Apache-2.0
name: Release candidate (Linux)
on:
workflow_dispatch:
inputs:
version:
description: Final product version (no preview suffix)
type: string
required: true
source_sha:
description: Full reviewed source commit
type: string
required: true
permissions:
contents: read
actions: read
concurrency:
group: release-candidate-${{ inputs.version }}
cancel-in-progress: false
jobs:
freeze:
if: github.repository == 'rustfs/rustfs' && github.ref == 'refs/heads/main'
runs-on: ubuntu-latest
timeout-minutes: 15
outputs:
manifest: ${{ steps.freeze.outputs.manifest }}
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
fetch-depth: 0
persist-credentials: false
- name: Freeze or recover original inputs
id: freeze
env:
GH_TOKEN: ${{ github.token }}
VERSION: ${{ inputs.version }}
SOURCE_SHA: ${{ inputs.source_sha }}
run: python3 scripts/release/pipeline.py prepare --version "$VERSION" --source "$SOURCE_SHA" --directory frozen
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
if: steps.freeze.outputs.reused != 'true'
with:
name: release-inputs
path: frozen/inputs.json
if-no-files-found: error
retention-days: 30
linux:
needs: freeze
uses: ./.github/workflows/build.yml
with:
candidate_manifest: ${{ needs.freeze.outputs.manifest }}
platforms: linux
upgrade:
needs: [freeze, linux]
runs-on: ubuntu-latest
timeout-minutes: 150
env:
RUSTUP_TOOLCHAIN: ${{ fromJson(needs.freeze.outputs.manifest).toolchain }}
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
ref: ${{ fromJson(needs.freeze.outputs.manifest).workflow_sha }}
persist-credentials: false
- uses: ./.github/actions/setup
with:
rust-version: ${{ fromJson(needs.freeze.outputs.manifest).toolchain }}
cache-shared-key: release-upgrade-harness-hosted
cache-save-if: ${{ github.ref == 'refs/heads/main' }}
install-build-packaging-tools: "false"
install-test-tools: "false"
- name: Verify archived server against previous releases
env:
GH_TOKEN: ${{ github.token }}
CANDIDATE_MANIFEST: ${{ needs.freeze.outputs.manifest }}
run: |
python3 - <<'PYVERIFY'
import sys
from pathlib import Path
sys.path.insert(0, 'scripts/release')
import candidate as c, pipeline, verify
inputs = pipeline.manifest_env()
run = pipeline.current_run('release-candidate.yml')
if any(a['name'] == 'release-upgrade' for a in c.artifact_list(run['id'])):
c.download_artifact(run, 'release-upgrade', 'receipts')
pipeline.verify_evidence(inputs, Path('receipts'), {run['id']: run})
c.emit_output('reused', 'true')
else:
target = 'linux-x86_64-gnu'
c.download_artifact(run, f"candidate-{inputs['candidate_id']}-{target}", 'binary')
verify.upgrade(inputs, Path('binary') / f"rustfs-{target}-v{inputs['version']}.zip", Path('receipts'))
PYVERIFY
id: verify
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
if: steps.verify.outputs.reused != 'true'
with:
name: release-upgrade
path: receipts/*
if-no-files-found: error
retention-days: 30
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
if: failure()
with:
name: release-upgrade-failure-${{ github.run_attempt }}
path: receipts/*
retention-days: 7
seal-linux:
needs: [freeze, linux, upgrade]
runs-on: ubuntu-latest
timeout-minutes: 30
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Collect trusted receipts and seal Linux bytes
id: seal
env:
GH_TOKEN: ${{ github.token }}
run: |
python3 scripts/release/pipeline.py linux-seal --directory sealed
{
echo '## Linux acceptance'
echo 'Use this run ID and manifest SHA-256 when starting Release candidate (complete).'
sha256sum sealed/linux-manifest.json
echo 'Record cluster, performance, integrity and upgrade acceptance in backlog before continuing.'
} >> "$GITHUB_STEP_SUMMARY"
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
if: steps.seal.outputs.reused != 'true'
with:
name: release-linux
path: sealed/*
if-no-files-found: error
compression-level: 0
retention-days: 30
-97
View File
@@ -1,97 +0,0 @@
# Copyright 2026 RustFS Team
# SPDX-License-Identifier: Apache-2.0
name: Release candidate (complete)
on:
workflow_dispatch:
inputs:
candidate_run_id:
description: Successful Linux candidate run ID
type: string
required: true
linux_manifest_sha256:
description: Exact accepted Linux manifest SHA-256
type: string
required: true
acceptance_report:
description: Backlog report accepting cluster, performance, integrity and upgrade results
type: string
required: true
permissions:
contents: read
actions: read
concurrency:
group: release-complete-${{ inputs.candidate_run_id }}
cancel-in-progress: false
jobs:
acceptance:
if: github.repository == 'rustfs/rustfs' && github.ref == 'refs/heads/main'
runs-on: ubuntu-latest
timeout-minutes: 30
environment: release-linux-acceptance
outputs:
manifest: ${{ steps.accept.outputs.manifest }}
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
fetch-depth: 0
persist-credentials: false
- name: Bind acceptance to the exact Linux manifest
id: accept
env:
GH_TOKEN: ${{ github.token }}
CANDIDATE_RUN: ${{ inputs.candidate_run_id }}
MANIFEST_SHA: ${{ inputs.linux_manifest_sha256 }}
REPORT: ${{ inputs.acceptance_report }}
run: python3 scripts/release/pipeline.py complete-prepare --run-id "$CANDIDATE_RUN" --digest "$MANIFEST_SHA" --report "$REPORT" --directory linux
desktop:
needs: acceptance
uses: ./.github/workflows/build.yml
with:
candidate_manifest: ${{ needs.acceptance.outputs.manifest }}
platforms: desktop
packages:
needs: acceptance
uses: ./.github/workflows/package.yml
with:
candidate_manifest: ${{ needs.acceptance.outputs.manifest }}
images:
needs: acceptance
uses: ./.github/workflows/docker.yml
with:
candidate_manifest: ${{ needs.acceptance.outputs.manifest }}
chart:
needs: acceptance
uses: ./.github/workflows/helm-package.yml
with:
candidate_manifest: ${{ needs.acceptance.outputs.manifest }}
seal:
needs: [acceptance, desktop, packages, images, chart]
runs-on: ubuntu-latest
timeout-minutes: 45
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
fetch-depth: 0
persist-credentials: false
- name: Verify every deliverable and seal the candidate
id: seal
env:
GH_TOKEN: ${{ github.token }}
CANDIDATE_RUN: ${{ inputs.candidate_run_id }}
MANIFEST_SHA: ${{ inputs.linux_manifest_sha256 }}
REPORT: ${{ inputs.acceptance_report }}
run: |
python3 scripts/release/pipeline.py complete-seal --run-id "$CANDIDATE_RUN" --digest "$MANIFEST_SHA" --report "$REPORT" --directory sealed
{
echo '## Final publication approval'
echo 'Review this manifest, then explicitly dispatch Publish accepted release with its SHA-256.'
sha256sum sealed/release-manifest.json
} >> "$GITHUB_STEP_SUMMARY"
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
if: steps.seal.outputs.reused != 'true'
with:
name: release-seal
path: sealed/*
if-no-files-found: error
compression-level: 0
retention-days: 30
-193
View File
@@ -1,193 +0,0 @@
# Copyright 2026 RustFS Team
# SPDX-License-Identifier: Apache-2.0
name: Publish accepted release
on:
workflow_dispatch:
inputs:
complete_run_id:
description: Successful completed candidate run ID
type: string
required: true
manifest_sha256:
description: Exact approved final manifest SHA-256 (authorizes publication)
type: string
required: true
permissions:
contents: read
actions: read
concurrency:
group: release-publish-${{ inputs.complete_run_id }}
cancel-in-progress: false
jobs:
verify:
if: github.repository == 'rustfs/rustfs' && github.ref == 'refs/heads/main'
runs-on: ubuntu-latest
timeout-minutes: 45
outputs:
version: ${{ steps.verify.outputs.version }}
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
fetch-depth: 0
persist-credentials: false
- name: Recheck approved bytes and trusted certification
id: verify
env:
GH_TOKEN: ${{ github.token }}
COMPLETE_RUN: ${{ inputs.complete_run_id }}
MANIFEST_SHA: ${{ inputs.manifest_sha256 }}
run: python3 scripts/release/publish.py verify --run-id "$COMPLETE_RUN" --digest "$MANIFEST_SHA" --directory sealed
version:
needs: verify
runs-on: ubuntu-latest
timeout-minutes: 90
concurrency:
group: release-version-${{ needs.verify.outputs.version }}
cancel-in-progress: false
environment: release-publication
permissions:
contents: write
actions: read
packages: write
security-events: write
env:
GH_TOKEN: ${{ github.token }}
COMPLETE_RUN: ${{ inputs.complete_run_id }}
MANIFEST_SHA: ${{ inputs.manifest_sha256 }}
AWS_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
AWS_DEFAULT_REGION: auto
AWS_REQUEST_CHECKSUM_CALCULATION: when_required
AWS_RESPONSE_CHECKSUM_VALIDATION: when_required
AWS_EC2_METADATA_DISABLED: "true"
R2_ENDPOINT: ${{ secrets.R2_ENDPOINT }}
R2_BUCKET: ${{ secrets.R2_BUCKET }}
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
fetch-depth: 0
persist-credentials: false
- name: Install promotion tools
run: |
sudo apt-get update
sudo apt-get install -y skopeo
aws --version
- uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
with:
username: rustfs
password: ${{ secrets.DOCKERHUB_TOKEN }}
- uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
with:
registry: ghcr.io
username: ${{ secrets.GHCR_USERNAME }}
password: ${{ secrets.GHCR_PASSWORD }}
- uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
with:
registry: quay.io
username: ${{ secrets.QUAY_USERNAME }}
password: ${{ secrets.QUAY_PASSWORD }}
- name: Share registry credentials with the OCI copier
run: |
install -m 600 "$HOME/.docker/config.json" "$RUNNER_TEMP/registry-auth.json"
echo "REGISTRY_AUTH_FILE=$RUNNER_TEMP/registry-auth.json" >> "$GITHUB_ENV"
- name: Promote verified version
run: python3 scripts/release/publish.py version --run-id "$COMPLETE_RUN" --digest "$MANIFEST_SHA" --directory sealed
- name: Publish accepted musl amd64 scan
uses: github/codeql-action/upload-sarif@99df26d4f13ea111d4ec1a7dddef6063f76b97e9 # v4
with:
sarif_file: sealed/scan-musl-amd64.sarif
category: container-image-musl-amd64
- name: Publish accepted musl arm64 scan
uses: github/codeql-action/upload-sarif@99df26d4f13ea111d4ec1a7dddef6063f76b97e9 # v4
with:
sarif_file: sealed/scan-musl-arm64.sarif
category: container-image-musl-arm64
- name: Publish accepted GNU amd64 scan
uses: github/codeql-action/upload-sarif@99df26d4f13ea111d4ec1a7dddef6063f76b97e9 # v4
with:
sarif_file: sealed/scan-gnu-amd64.sarif
category: container-image-gnu-amd64
- name: Publish accepted GNU arm64 scan
uses: github/codeql-action/upload-sarif@99df26d4f13ea111d4ec1a7dddef6063f76b97e9 # v4
with:
sarif_file: sealed/scan-gnu-arm64.sarif
category: container-image-gnu-arm64
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: release-version-receipt-${{ github.run_attempt }}
path: publication.json
if-no-files-found: error
retention-days: 90
channels:
needs: [verify, version]
runs-on: ubuntu-latest
timeout-minutes: 90
concurrency:
group: rustfs-release-channels
cancel-in-progress: false
permissions:
contents: write
actions: read
packages: write
env:
GH_TOKEN: ${{ github.token }}
COMPLETE_RUN: ${{ inputs.complete_run_id }}
MANIFEST_SHA: ${{ inputs.manifest_sha256 }}
AWS_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
AWS_DEFAULT_REGION: auto
AWS_REQUEST_CHECKSUM_CALCULATION: when_required
AWS_RESPONSE_CHECKSUM_VALIDATION: when_required
AWS_EC2_METADATA_DISABLED: "true"
R2_ENDPOINT: ${{ secrets.R2_ENDPOINT }}
R2_BUCKET: ${{ secrets.R2_BUCKET }}
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
fetch-depth: 0
persist-credentials: false
- name: Install promotion tools
run: |
sudo apt-get update
sudo apt-get install -y skopeo
aws --version
- uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
with:
username: rustfs
password: ${{ secrets.DOCKERHUB_TOKEN }}
- uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
with:
registry: ghcr.io
username: ${{ secrets.GHCR_USERNAME }}
password: ${{ secrets.GHCR_PASSWORD }}
- uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
with:
registry: quay.io
username: ${{ secrets.QUAY_USERNAME }}
password: ${{ secrets.QUAY_PASSWORD }}
- name: Share registry credentials with the OCI copier
run: |
install -m 600 "$HOME/.docker/config.json" "$RUNNER_TEMP/registry-auth.json"
echo "REGISTRY_AUTH_FILE=$RUNNER_TEMP/registry-auth.json" >> "$GITHUB_ENV"
- uses: azure/setup-helm@b9e51907a09c216f16ebe8536097933489208112 # v4.3.0
- name: Install pinned OSS client
run: |
curl --fail --location --retry 3 --output "$RUNNER_TEMP/ossutil.zip" https://gosspublic.alicdn.com/ossutil/v2/2.1.1/ossutil-2.1.1-linux-amd64.zip
printf '%s %s\n' 60f3a808cbbdfd4b5269b8f967c6a66f564c9dacff52d93a5d21a588976ab5a2 "$RUNNER_TEMP/ossutil.zip" | sha256sum --check -
unzip -q "$RUNNER_TEMP/ossutil.zip" -d "$RUNNER_TEMP/ossutil-extract"
install -m 755 "$RUNNER_TEMP/ossutil-extract/ossutil-2.1.1-linux-amd64/ossutil" "$RUNNER_TEMP/ossutil"
echo "OSSUTIL_BIN=$RUNNER_TEMP/ossutil" >> "$GITHUB_ENV"
- name: Promote verified channels
env:
HELM_TOKEN: ${{ secrets.RUSTFS_HELM_PACKAGE }}
OSS_ACCESS_KEY_ID: ${{ secrets.ALICLOUDOSS_KEY_ID }}
OSS_ACCESS_KEY_SECRET: ${{ secrets.ALICLOUDOSS_KEY_SECRET }}
OSS_REGION: cn-beijing
OSS_ENDPOINT: https://oss-cn-beijing.aliyuncs.com
run: python3 scripts/release/publish.py channels --run-id "$COMPLETE_RUN" --digest "$MANIFEST_SHA" --directory sealed
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: release-channels-receipt-${{ github.run_attempt }}
path: publication.json
if-no-files-found: error
retention-days: 90
+1 -1
View File
@@ -44,7 +44,7 @@ jobs:
check-ephemerality:
if: github.repository == 'rustfs/rustfs'
name: Check runner ephemerality
runs-on: ubuntu-latest
runs-on: sm-standard-2
timeout-minutes: 15
steps:
- name: Checkout repository
@@ -93,12 +93,6 @@ jobs:
printf 'TMPDIR=%s-scratch\n' "${FUNCTIONAL_ARTIFACTS_DIR}"
} >> "${GITHUB_ENV}"
- name: Checkout environment guard
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
path: rustfs-tools
- name: Checkout auto-testing scripts
timeout-minutes: 5
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
@@ -124,13 +118,6 @@ jobs:
echo "auto-testing: ${AUTO_TESTING_SHA}"
df -h /data | tail -1
- name: Admit clean shared environment
id: environment
timeout-minutes: 5
env:
FUNCTIONAL_ENVIRONMENT_RECEIPT: ${{ runner.temp }}/environment-matrix.json
run: python3 rustfs-tools/scripts/functional_environment.py enter --suite fault-tolerance
- name: Cleanup environment (before)
timeout-minutes: 5
if: ${{ inputs.cleanup_before != 'false' }}
@@ -228,24 +215,7 @@ jobs:
if-no-files-found: warn
- name: Cleanup environment (after)
id: environment_cleanup
timeout-minutes: 5
if: ${{ always() && inputs.cleanup_after != 'false' && steps.environment.outcome == 'success' }}
if: ${{ always() && inputs.cleanup_after != 'false' }}
run: |
./auto-testing/rustfs-fault-tolerance-test.sh --cleanup -y --log-file "${LOG_FILE}"
- name: Release environment only after successful cleanup
if: ${{ always() && steps.environment.outcome == 'success' }}
timeout-minutes: 5
env:
FUNCTIONAL_ENVIRONMENT_RECEIPT: ${{ runner.temp }}/environment-matrix.json
CLEANUP_OUTCOME: ${{ steps.environment_cleanup.outcome }}
run: python3 rustfs-tools/scripts/functional_environment.py finish --suite fault-tolerance --cleanup-outcome "$CLEANUP_OUTCOME"
- name: Retain environment ownership and cleanup receipt
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: functional-environment-matrix-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/environment-matrix.json
if-no-files-found: error
./auto-testing/rustfs-fault-tolerance-test.sh --cleanup -y --log-file "${LOG_FILE}" || true
@@ -50,6 +50,10 @@ on:
description: 'Reset the nodes after the test (DESTROYS test data/config)'
type: boolean
default: true
repository_dispatch:
# Chain handoff: dispatched when the replication suite finishes, ahead of
# the performance suite.
types: [rustfs-chain-fault-tolerance]
permissions:
contents: read
@@ -76,7 +80,7 @@ jobs:
fault-tolerance-test:
runs-on: smoke-testing
timeout-minutes: 60
if: ${{ inputs.chain_manifest != '' || github.event_name == 'workflow_dispatch' }}
if: ${{ inputs.chain_manifest != '' || github.event_name == 'workflow_dispatch' || github.event_name == 'repository_dispatch' }}
steps:
- name: Checkout repository (for report parser)
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
@@ -132,13 +136,6 @@ jobs:
echo "auto-testing: ${AUTO_TESTING_SHA}"
df -h /data | tail -1
- name: Admit clean shared environment
id: environment
timeout-minutes: 5
env:
FUNCTIONAL_ENVIRONMENT_RECEIPT: ${{ runner.temp }}/environment-fault-tolerance.json
run: python3 scripts/functional_environment.py enter --suite fault-tolerance
- name: Cleanup environment (before)
timeout-minutes: 5
if: ${{ inputs.cleanup_before != 'false' }}
@@ -197,21 +194,21 @@ jobs:
jq -e --arg testing_sha "${AUTO_TESTING_SHA}" '
.schema_version == 1 and
.auto_testing_sha == $testing_sha and
.selected_scenarios == ["A", "B", "C", "C2", "D", "E", "F"] and
.selected_scenarios == ["A", "B", "C", "C2", "D", "E"] and
.complete == true and
.counts.expected == 54 and
.counts.seen == 54 and
.counts.expected == 44 and
.counts.seen == 44 and
.counts.harness_errors == 0 and
(.expected_cases | length) == 54 and
(.expected_cases | unique | length) == 54 and
(.seen_cases | length) == 54 and
(.seen_cases | unique | length) == 54 and
(.expected_cases | length) == 44 and
(.expected_cases | unique | length) == 44 and
(.seen_cases | length) == 44 and
(.seen_cases | unique | length) == 44 and
(.seen_cases | sort) == (.expected_cases | sort) and
(.missing_cases | length) == 0 and
(.harness_errors | length) == 0 and
(.cases | length) == 54 and
(.cases | length) == 44 and
([.cases[].case_id] | sort) == (.expected_cases | sort) and
([.cases[].case_id] | unique | length) == 54 and
([.cases[].case_id] | unique | length) == 44 and
all(.cases[]; .verdict == "pass" or .verdict == "known-divergence" or .verdict == "UNEXPECTED") and
.counts.unexpected == ([.cases[] | select(.verdict == "UNEXPECTED")] | length) and
.counts.known_divergence == ([.cases[] | select(.verdict == "known-divergence")] | length)
@@ -308,27 +305,58 @@ jobs:
if-no-files-found: warn
- name: Cleanup environment (after)
id: environment_cleanup
timeout-minutes: 5
if: ${{ always() && inputs.cleanup_after != 'false' && steps.environment.outcome == 'success' }}
if: ${{ always() && inputs.cleanup_after != 'false' }}
run: |
./auto-testing/rustfs-fault-tolerance-test.sh --cleanup -y --log-file "${LOG_FILE}"
./auto-testing/rustfs-fault-tolerance-test.sh --cleanup -y --log-file "${LOG_FILE}" || true
- name: Release environment only after successful cleanup
if: ${{ always() && steps.environment.outcome == 'success' }}
timeout-minutes: 5
- name: "Continue functional chain (next: Performance)"
# Only chain-triggered runs forward to the next suite; standalone
# workflow_dispatch runs stop after their own cleanup. A failed
# handoff retries, then files an alert issue in rustfs/backlog.
if: ${{ always() && github.event_name == 'repository_dispatch' }}
continue-on-error: true
env:
FUNCTIONAL_ENVIRONMENT_RECEIPT: ${{ runner.temp }}/environment-fault-tolerance.json
CLEANUP_OUTCOME: ${{ steps.environment_cleanup.outcome }}
run: python3 scripts/functional_environment.py finish --suite fault-tolerance --cleanup-outcome "$CLEANUP_OUTCOME"
- name: Retain environment ownership and cleanup receipt
if: ${{ always() }}
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: functional-environment-fault-tolerance-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/environment-fault-tolerance.json
if-no-files-found: error
GH_TOKEN: ${{ secrets.PF_TESTING_GH_TOKEN }}
run: |
set -uo pipefail
if [ -z "${GH_TOKEN:-}" ]; then
echo "PF_TESTING_GH_TOKEN is not configured; cannot dispatch the next suite" >&2
exit 1
fi
DISPATCHED=0
for attempt in 1 2 3; do
if gh api --method POST repos/rustfs/rustfs/dispatches \
-f event_type='rustfs-chain-performance' \
-F 'client_payload[from_suite]=fault-tolerance'; then
echo "dispatched next suite Performance (attempt ${attempt})"
DISPATCHED=1
break
fi
echo "dispatch attempt ${attempt} failed; retrying in ${attempt}0s" >&2
sleep "${attempt}0"
done
if [ "${DISPATCHED:-0}" -ne 1 ]; then
echo "ERROR: functional chain stalled: could not dispatch Performance after 3 attempts" >&2
TITLE="[functional][chain] stalled after fault-tolerance (run ${GITHUB_RUN_ID})"
BODY_FILE="$(mktemp)"
{
echo "The functional chain could not hand off from **fault-tolerance** to **Performance** after 3 attempts."
echo ""
echo "- Failed suite job: ${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}"
echo "- Expected next event: 'rustfs-chain-performance'"
echo "- Likely cause: PF_TESTING_GH_TOKEN lacks contents:write on rustfs/rustfs, or the GitHub API was unavailable."
echo "- Recovery: re-dispatch manually with"
FENCE="$(printf "\x60\x60\x60")"; echo " ${FENCE}"
echo " gh api --method POST repos/rustfs/rustfs/dispatches -f event_type='rustfs-chain-performance'"
FENCE="$(printf "\x60\x60\x60")"; echo " ${FENCE}"
} > "${BODY_FILE}"
gh issue create -R rustfs/backlog --title "${TITLE}" \
--body-file "${BODY_FILE}" --label functional-test \
|| gh issue create -R rustfs/backlog --title "${TITLE}" --body-file "${BODY_FILE}" \
|| echo "could not file the stall alert issue either; check the token" >&2
exit 1
fi
- name: Notify on failure
if: failure()
@@ -363,13 +391,3 @@ jobs:
name: functional-chain-fault-tolerance-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/chain-fault-tolerance-${{ github.run_id }}-${{ github.run_attempt }}/fault-tolerance.json
if-no-files-found: error
- name: Enforce suite and report result
if: ${{ always() }}
env:
SUITE_OUTCOME: ${{ steps.test.outcome }}
REPORT_OUTCOME: ${{ steps.chain_report.outcome }}
run: |
test "$SUITE_OUTCOME" = success
test "$REPORT_OUTCOME" = success
python3 scripts/functional_chain_evidence.py verify-report --suite fault-tolerance --report "${FUNCTIONAL_ARTIFACTS_DIR}/suite.log"
+25 -644
View File
@@ -13,109 +13,20 @@
# limitations under the License.
# Reusable workflows run from this driver's commit in one Actions run. Each
# selected suite still runs after an earlier failure; the final job checks its plan.
# suite still runs after an earlier suite fails; the final job requires all twelve.
name: RustFS Functional Chain
on:
workflow_dispatch:
inputs:
validation_plan:
description: Registered cadence plan (existing suite frequencies are preserved)
type: choice
options: [nightly, weekly, monthly]
required: false
default: nightly
suites:
description: Comma-separated suite IDs, or empty for the complete registered plan
type: string
required: false
default: ''
build_run_id:
description: Successful nightly or validation package run; empty for package-only build
description: Successful nightly build run on main
type: string
required: false
default: ''
required: true
build_run_attempt:
description: Exact successful build attempt
type: string
required: false
default: ''
validation_package_sha:
description: Exact source SHA for an isolated package-only build
type: string
required: false
default: ''
validation_case:
description: Canonical case ID, environment.inspect for read-only resources, or empty for functional chain
type: string
required: false
default: ''
i05_diagnostic_branch:
description: I05 source-plan runs both H1/H2 controllers with exact native evidence; subsets remain diagnostic
type: choice
options: [source-plan, timeout, queued-successor]
required: false
default: source-plan
auto_testing_sha:
description: Exact auto-testing commit for a validation case
type: string
required: false
default: ''
validation_retain_evidence_run_id:
description: Upload existing persistent evidence from one interrupted validation run without fleet access
type: string
required: false
default: ''
validation_retain_evidence_run_attempt:
description: Exact interrupted run attempt to retain
type: string
required: false
default: '1'
validation_runner_inspect_only:
description: Inspect GitHub runner registration and queued jobs on a hosted runner without fleet access
type: boolean
required: false
default: false
validation_tools_only:
description: Build candidate-bound validation tools without running a fleet case
type: boolean
required: false
default: false
validation_native_tools:
description: Include exact-source native regressions in a tools-only build
type: boolean
required: false
default: false
validation_shared_tools:
description: Include the controlled C10 shared-reference encoder in a tools-only build
type: boolean
required: false
default: false
validation_manager_tools:
description: Include exact checkpoint/MRF/incarnation regressions in a tools-only build
type: boolean
required: false
default: false
validation_cleanup_run_id:
description: Failed validation run whose supported retained environment needs evidence-based finalization
type: string
required: false
default: ''
validation_cleanup_retained_evidence:
description: Recover from verified persistent runner-local evidence when the failed run uploaded no artifact
type: boolean
required: false
default: false
validation_tools_run_id:
description: Successful tool build run; required by Heal physical metadata checks
type: string
required: false
default: ''
validation_tools_run_attempt:
description: Exact tool build attempt
type: string
required: false
default: '1'
required: true
workflow_run:
workflows: ["Nightly GNU Build"]
types: [completed]
@@ -125,127 +36,21 @@ permissions:
actions: read
concurrency:
group: ${{ github.event_name == 'workflow_dispatch' && inputs.validation_runner_inspect_only && 'rustfs-functional-validation-runner-inspection' || github.event_name == 'workflow_dispatch' && inputs.validation_package_sha != '' && format('rustfs-functional-validation-package-{0}', inputs.validation_package_sha) || github.event_name == 'workflow_dispatch' && inputs.validation_tools_only && 'rustfs-functional-validation-tools' || 'rustfs-functional-chain-runs' }}
group: rustfs-functional-chain-runs
cancel-in-progress: false
jobs:
validation-runner-inspection:
if: ${{ github.event_name == 'workflow_dispatch' && inputs.validation_runner_inspect_only }}
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Inspect shared runner registration and queue without fleet access
env:
GH_TOKEN: ${{ secrets.PF_TESTING_GH_TOKEN }}
RUNNER_INSPECTION_INPUTS: ${{ toJSON(inputs) }}
run: >-
python3 -B scripts/inspect_validation_runner.py
--output "${RUNNER_TEMP}/validation-runner-inspection/report.json"
- name: Upload read-only runner inspection including incomplete reports
if: ${{ always() }}
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: validation-runner-inspection-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/validation-runner-inspection/report.json
if-no-files-found: error
validation-evidence-retention:
if: ${{ github.event_name == 'workflow_dispatch' && !inputs.validation_runner_inspect_only && inputs.validation_retain_evidence_run_id != '' }}
# The original interrupted evidence exists only on this long-lived runner.
runs-on: smoke-testing
timeout-minutes: 15
concurrency:
group: rustfs-shared-functional-tests-v2
cancel-in-progress: false
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Verify existing interrupted evidence without changing it
env:
GH_TOKEN: ${{ github.token }}
EVIDENCE_RETAIN_INPUTS: ${{ toJSON(inputs) }}
run: >-
python3 -B scripts/retain_validation_evidence.py
--report "${RUNNER_TEMP}/validation-evidence-retention/report.json"
- name: Upload admitted original evidence under collection provenance
if: ${{ env.VALIDATION_RETAIN_ROOT != '' }}
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: rustfs-validation-retained-${{ env.VALIDATION_RETAIN_SOURCE_RUN }}-${{ env.VALIDATION_RETAIN_SOURCE_ATTEMPT }}-by-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ env.VALIDATION_RETAIN_ROOT }}
if-no-files-found: error
include-hidden-files: true
- name: Upload retention receipt including incomplete reports
if: ${{ always() }}
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: validation-evidence-retention-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/validation-evidence-retention/report.json
if-no-files-found: error
validation-package-request:
if: ${{ github.event_name == 'workflow_dispatch' && !inputs.validation_runner_inspect_only && inputs.validation_retain_evidence_run_id == '' && inputs.validation_package_sha != '' }}
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
- name: Validate exclusive package-only request
env:
PACKAGE_INPUTS: ${{ toJSON(inputs) }}
run: |
python3 - <<'PY'
import json
import os
import re
value = json.loads(os.environ["PACKAGE_INPUTS"])
if re.fullmatch(r"[0-9a-f]{40}", value["validation_package_sha"]) is None:
raise ValueError("package build requires an exact source SHA")
exclusive = ("validation_retain_evidence_run_id", "validation_runner_inspect_only", "build_run_id", "build_run_attempt", "validation_case", "auto_testing_sha",
"validation_tools_only", "validation_native_tools", "validation_shared_tools",
"validation_manager_tools", "validation_cleanup_run_id", "validation_cleanup_retained_evidence", "validation_tools_run_id")
if any(value.get(name) for name in exclusive):
raise ValueError("package-only build cannot select existing candidate, tools, fleet or recovery execution")
PY
validation-package:
needs: validation-package-request
if: ${{ github.event_name == 'workflow_dispatch' && !inputs.validation_runner_inspect_only && inputs.validation_retain_evidence_run_id == '' && inputs.validation_package_sha != '' }}
uses: ./.github/workflows/rustfs-validation-package.yml
with:
source_sha: ${{ inputs.validation_package_sha }}
secrets:
R2_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
R2_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
R2_ENDPOINT: ${{ secrets.R2_ENDPOINT }}
R2_BUCKET: ${{ secrets.R2_BUCKET }}
prepare:
if: ${{ (github.event_name == 'workflow_dispatch' && !inputs.validation_runner_inspect_only && inputs.validation_retain_evidence_run_id == '' && inputs.validation_package_sha == '' && inputs.validation_case == '' && !inputs.validation_tools_only && !inputs.validation_native_tools && !inputs.validation_shared_tools && !inputs.validation_manager_tools && !inputs.validation_cleanup_retained_evidence && inputs.validation_cleanup_run_id == '') || github.event.workflow_run.event == 'schedule' }}
runs-on: ubuntu-latest
if: ${{ github.event_name == 'workflow_dispatch' || github.event.workflow_run.event == 'schedule' }}
runs-on: smoke-testing
timeout-minutes: 10
outputs:
manifest: ${{ steps.candidate.outputs.manifest }}
performance_ready: ${{ steps.perf_probe.outputs.performance_ready }}
plan: ${{ steps.selection.outputs.plan }}
suites: ${{ steps.selection.outputs.suites }}
optional_suites: ${{ steps.selection.outputs.optional_suites }}
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Select registered suites
id: selection
env:
VALIDATION_PLAN: ${{ inputs.validation_plan || 'nightly' }}
VALIDATION_SUITES: ${{ inputs.suites }}
run: python3 scripts/validation_registry.py select --output "${RUNNER_TEMP}/suite-plan.json"
- name: Retain selected responsibilities
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: functional-plan-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/suite-plan.json
if-no-files-found: error
- name: Resolve published candidate
id: candidate
env:
@@ -265,7 +70,6 @@ jobs:
if-no-files-found: error
- name: Check shared functional fleet runner before scheduling suites
if: ${{ contains(fromJSON(steps.selection.outputs.environments), 'functional-fleet') }}
# The eleven functional suites run on the smoke-testing fleet; a hard
# requirement. The performance lane runs on pf-testing, which is a
# separate fleet - probe it without failing the whole chain so an
@@ -275,439 +79,19 @@ jobs:
run: python3 scripts/check_functional_runners.py smoke-testing
- name: Probe performance fleet runner
if: ${{ contains(fromJSON(steps.selection.outputs.suites), 'performance') }}
id: perf_probe
env:
GH_TOKEN: ${{ secrets.PF_TESTING_GH_TOKEN }}
PERFORMANCE_OPTIONAL: ${{ contains(fromJSON(steps.selection.outputs.optional_suites), 'performance') }}
run: |
# TEMPORARY: the pf-testing environment is broken, so the
# performance lane is disabled until it is repaired. Forcing
# offline keeps the chain on its designed skip path: the lane job
# is skipped, aggregate receives --allow-skipped performance
# (required=false in scheduled-validations.json), and the report
# records the optional-lane skip with its reason.
# Restore the probe below once the environment is healthy again.
# if python3 scripts/check_functional_runners.py pf-testing; then
# echo "performance_ready=online" >> "$GITHUB_OUTPUT"
# else
# echo "performance_ready=offline" >> "$GITHUB_OUTPUT"
# test "$PERFORMANCE_OPTIONAL" = true
# fi
echo "performance_ready=offline" >> "$GITHUB_OUTPUT"
validation:
if: ${{ github.event_name == 'workflow_dispatch' && !inputs.validation_runner_inspect_only && inputs.validation_retain_evidence_run_id == '' && inputs.validation_package_sha == '' && (inputs.validation_case != '' || inputs.validation_tools_only || inputs.validation_native_tools || inputs.validation_shared_tools || inputs.validation_manager_tools || inputs.validation_cleanup_run_id != '' || inputs.validation_cleanup_retained_evidence) }}
runs-on: ${{ inputs.validation_tools_only && 'ubuntu-latest' || 'smoke-testing' }}
timeout-minutes: 90
outputs:
source_sha: ${{ steps.candidate.outputs.source_sha }}
concurrency:
group: ${{ inputs.validation_tools_only && 'rustfs-validation-tools-resolver' || 'rustfs-shared-functional-tests-v2' }}
cancel-in-progress: false
env:
RUSTFS_ACCESS_KEY: ${{ !inputs.validation_tools_only && secrets.RUSTFS_ACCESS_KEY || '' }}
RUSTFS_SECRET_KEY: ${{ !inputs.validation_tools_only && secrets.RUSTFS_SECRET_KEY || '' }}
RUSTFS_API_ENDPOINT: ${{ !inputs.validation_tools_only && (secrets.RUSTFS_API_ENDPOINT || vars.RUSTFS_API_ENDPOINT || vars.RUSTFS_RC_ENDPOINT) || '' }}
RUSTFS_NODES: ${{ !inputs.validation_tools_only && (secrets.RUSTFS_NODES || vars.RUSTFS_NODES) || '' }}
RUSTFS_SSH_USER: ${{ !inputs.validation_tools_only && (secrets.RUSTFS_SSH_USER || vars.RUSTFS_SSH_USER) || '' }}
steps:
- name: Checkout candidate resolver
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Initialize validation evidence
env:
VALIDATION_TOOLS_ONLY: ${{ inputs.validation_tools_only }}
run: |
set -euo pipefail
umask 077
python3 -B scripts/initialize_validation_evidence.py
- name: Resolve immutable candidate
id: candidate
env:
GH_TOKEN: ${{ secrets.PF_TESTING_GH_TOKEN }}
BUILD_RUN_ID: ${{ inputs.build_run_id }}
BUILD_RUN_ATTEMPT: ${{ inputs.build_run_attempt }}
AUTO_TESTING_SHA: ${{ inputs.auto_testing_sha }}
CLEANUP_RUN_ID: ${{ inputs.validation_cleanup_run_id }}
CLEANUP_RETAINED_EVIDENCE: ${{ inputs.validation_cleanup_retained_evidence }}
SELECTED_CASE: ${{ inputs.validation_case }}
TOOLS_ONLY: ${{ inputs.validation_tools_only }}
NATIVE_TOOLS: ${{ inputs.validation_native_tools }}
SHARED_TOOLS: ${{ inputs.validation_shared_tools }}
MANAGER_TOOLS: ${{ inputs.validation_manager_tools }}
I05_DIAGNOSTIC_BRANCH: ${{ inputs.i05_diagnostic_branch }}
run: |
set -euo pipefail
python3 - <<'PY'
import json
import os
from pathlib import Path
import re
from scripts.resolve_functional_candidate import resolve
revision = os.environ["AUTO_TESTING_SHA"]
cleanup = os.environ["CLEANUP_RUN_ID"]
if os.environ["CLEANUP_RETAINED_EVIDENCE"] == "true" and not cleanup:
raise ValueError("persistent recovery input requires one exact cleanup run")
if os.environ["TOOLS_ONLY"] == "true" and (cleanup or os.environ["SELECTED_CASE"]):
raise ValueError("tools-only execution cannot select a fleet case or cleanup")
if os.environ["NATIVE_TOOLS"] == "true" and (os.environ["TOOLS_ONLY"] != "true" or cleanup or os.environ["SELECTED_CASE"]):
raise ValueError("native regressions require tools-only execution, without a fleet case or cleanup")
if os.environ["SHARED_TOOLS"] == "true" and (os.environ["TOOLS_ONLY"] != "true" or cleanup or os.environ["SELECTED_CASE"]):
raise ValueError("shared fixture builds require tools-only execution, without a fleet case or cleanup")
if os.environ["MANAGER_TOOLS"] == "true" and (os.environ["TOOLS_ONLY"] != "true" or cleanup or os.environ["SELECTED_CASE"]):
raise ValueError("manager regressions require tools-only execution, without a fleet case or cleanup")
if cleanup and (not cleanup.isdigit() or int(cleanup) <= 0 or os.environ["SELECTED_CASE"] or os.environ["TOOLS_ONLY"] == "true"):
raise ValueError("cleanup requires one exact failed run and is exclusive with case/tool execution")
diagnostic_branch = os.environ["I05_DIAGNOSTIC_BRANCH"] or "source-plan"
if diagnostic_branch not in {"source-plan", "timeout", "queued-successor"}:
raise ValueError("I05 diagnostic branch is not registered")
if diagnostic_branch == "queued-successor" and os.environ["SELECTED_CASE"] != "heal.lock-timeout-and-caller-abort":
raise ValueError("queued-successor is available only for the I05 diagnostic case")
if re.fullmatch(r"[0-9a-f]{40}", revision) is None:
raise ValueError("validation requires an exact auto-testing commit SHA")
candidate = resolve(int(os.environ["BUILD_RUN_ID"]), int(os.environ["BUILD_RUN_ATTEMPT"]), allow_validation=True)
manifest = candidate["manifest"]
root = Path(os.environ["VALIDATION_ROOT"])
(root / "candidate-source.json").write_text(json.dumps(candidate, sort_keys=True) + "\n")
(root / "candidate.json").write_text(json.dumps({
"package_url": manifest["package_url"],
"package_sha256": manifest["package_sha256"],
"source_sha": manifest["source_sha"],
}, sort_keys=True) + "\n")
with open(os.environ["GITHUB_OUTPUT"], "a") as output:
output.write("source_sha=" + manifest["source_sha"] + "\n")
PY
- name: Checkout exact auto-testing revision
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
repository: rustfs/auto-testing
ref: ${{ inputs.auto_testing_sha }}
token: ${{ secrets.PF_TESTING_GH_TOKEN }}
path: auto-testing
persist-credentials: false
- name: Prepare validation Python runtime
if: ${{ !inputs.validation_tools_only && (startsWith(inputs.validation_case, 'ilm.') || startsWith(inputs.validation_case, 'pool.') || startsWith(inputs.validation_case, 'bucket-replication.') || startsWith(inputs.validation_case, 'site-replication.') || startsWith(inputs.validation_case, 'heal.')) }}
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.12"
- name: Install validation Python client
if: ${{ !inputs.validation_tools_only && (startsWith(inputs.validation_case, 'ilm.') || startsWith(inputs.validation_case, 'pool.') || startsWith(inputs.validation_case, 'bucket-replication.') || startsWith(inputs.validation_case, 'site-replication.') || startsWith(inputs.validation_case, 'heal.')) }}
run: |
set -euo pipefail
validation_python="${RUNNER_TEMP}/rustfs-validation-python-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}"
test ! -e "${validation_python}"
python3 -m venv "${validation_python}"
"${validation_python}/bin/python" -m pip install \
-r auto-testing/scripts/validation_ilm/requirements.txt
printf '%s\n' "${validation_python}/bin" >> "${GITHUB_PATH}"
- name: Validate Heal tool build identity
if: ${{ !inputs.validation_tools_only && startsWith(inputs.validation_case, 'heal.') }}
env:
GH_TOKEN: ${{ github.token }}
TOOL_RUN_ID: ${{ inputs.validation_tools_run_id }}
TOOL_RUN_ATTEMPT: ${{ inputs.validation_tools_run_attempt }}
run: |
set -euo pipefail
[[ "${TOOL_RUN_ID}" =~ ^[1-9][0-9]*$ ]]
[[ "${TOOL_RUN_ATTEMPT}" =~ ^[1-9][0-9]*$ ]]
gh api "repos/${GITHUB_REPOSITORY}/actions/runs/${TOOL_RUN_ID}/attempts/${TOOL_RUN_ATTEMPT}" \
> "${VALIDATION_ROOT}/tool-run.json"
python3 - <<'PY'
import json
import os
from pathlib import Path
value = json.loads((Path(os.environ["VALIDATION_ROOT"]) / "tool-run.json").read_text())
if (value.get("conclusion") != "success" or value.get("status") != "completed"
or value.get("run_attempt") != int(os.environ["TOOL_RUN_ATTEMPT"])
or value.get("path") not in {".github/workflows/rustfs-functional-chain.yml", ".github/workflows/rustfs-validation-tools.yml"}):
raise ValueError("Heal tool run is not an exact successful validation build attempt")
PY
- name: Download pinned Heal metadata inspector
if: ${{ !inputs.validation_tools_only && startsWith(inputs.validation_case, 'heal.') }}
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
github-token: ${{ github.token }}
run-id: ${{ inputs.validation_tools_run_id }}
name: rustfs-validation-tools-${{ inputs.validation_tools_run_id }}-${{ inputs.validation_tools_run_attempt }}
path: ${{ runner.temp }}/rustfs-validation-tools-${{ github.run_id }}-${{ github.run_attempt }}
- name: Verify Heal metadata inspector
if: ${{ !inputs.validation_tools_only && startsWith(inputs.validation_case, 'heal.') }}
run: |
set -euo pipefail
TOOL_DIRECTORY="${RUNNER_TEMP}/rustfs-validation-tools-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}"
python3 -B auto-testing/scripts/validation_heal/inspector.py \
--directory "${TOOL_DIRECTORY}" --candidate "${VALIDATION_ROOT}/candidate.json" \
> "${VALIDATION_ROOT}/tool-manifest.json"
printf 'VALIDATION_HEAL_TOOL_DIRECTORY=%s\n' "${TOOL_DIRECTORY}" >> "${GITHUB_ENV}"
- name: Select exact-source race native evidence
id: race_native
if: ${{ inputs.validation_case == 'heal.race-unversioned-overwrite' || inputs.validation_case == 'heal.race-versioned-put' || inputs.validation_case == 'heal.race-version-delete' || inputs.validation_case == 'heal.race-markers-and-batch-delete' || inputs.validation_case == 'heal.race-multipart-complete-abort' || inputs.validation_case == 'heal.admin-nolock-safety' || inputs.validation_case == 'heal.race-copy-and-metadata-update' }}
env:
CANDIDATE_SHA: ${{ steps.candidate.outputs.source_sha }}
VALIDATION_CASE_ID: ${{ inputs.validation_case }}
run: |
set -euo pipefail
required=false
if [ -f auto-testing/scripts/validation_heal/race_native.py ]; then
status=0
python3 -B auto-testing/scripts/validation_heal/race_native.py supports \
--commit "${CANDIDATE_SHA}" --case-id "${VALIDATION_CASE_ID}" || status=$?
if [ "${status}" -eq 0 ]; then
required=true
elif [ "${status}" -ne 2 ]; then
exit "${status}"
fi
fi
reviewed=false
case "${CANDIDATE_SHA}:${VALIDATION_CASE_ID}" in
6afdec9fe1b0b88007f2e5b445c19bf73d51d3ca:heal.race-unversioned-overwrite | \
bd0671ba781d8ebf509f4f03533228dfb44496f1:heal.race-unversioned-overwrite | \
bd0671ba781d8ebf509f4f03533228dfb44496f1:heal.race-versioned-put | \
bd0671ba781d8ebf509f4f03533228dfb44496f1:heal.race-version-delete | \
8380466f3a27c3c3ef4115084178a0ab99a78f32:heal.race-unversioned-overwrite | \
8380466f3a27c3c3ef4115084178a0ab99a78f32:heal.race-versioned-put | \
8380466f3a27c3c3ef4115084178a0ab99a78f32:heal.race-version-delete | \
8380466f3a27c3c3ef4115084178a0ab99a78f32:heal.race-markers-and-batch-delete | \
757b751482bb970bc321e92059690a7698554358:heal.admin-nolock-safety | \
757b751482bb970bc321e92059690a7698554358:heal.race-multipart-complete-abort | \
757b751482bb970bc321e92059690a7698554358:heal.race-copy-and-metadata-update) reviewed=true ;;
esac
if [ "${reviewed}" = true ] && [ "${required}" != true ]; then
printf 'Race candidate requires its reviewed native collector before fleet execution\n' >&2
exit 1
fi
printf 'required=%s\n' "${required}" >> "${GITHUB_OUTPUT}"
- name: Download pinned native storage evidence
if: ${{ inputs.validation_case == 'heal.conflicting-subquorum-authority' || inputs.validation_case == 'heal.bucket-incarnation-recreation' || steps.race_native.outputs.required == 'true' || (inputs.validation_case == 'heal.lock-timeout-and-caller-abort' && inputs.i05_diagnostic_branch != 'timeout') || (inputs.validation_case == 'heal.checkpoint-and-mrf-corruption' && steps.candidate.outputs.source_sha == '1aa9be87d18cf941e5020f19e8622c26ba4b10ac') }}
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
github-token: ${{ github.token }}
run-id: ${{ inputs.validation_tools_run_id }}
name: rustfs-validation-native-${{ inputs.validation_tools_run_id }}-${{ inputs.validation_tools_run_attempt }}
path: ${{ runner.temp }}/rustfs-validation-native-${{ github.run_id }}-${{ github.run_attempt }}
- name: Verify C07 native source and exact test evidence
if: ${{ inputs.validation_case == 'heal.conflicting-subquorum-authority' }}
run: |
set -euo pipefail
NATIVE_DIRECTORY="${RUNNER_TEMP}/rustfs-validation-native-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}"
cp "${VALIDATION_ROOT}/tool-run.json" "${NATIVE_DIRECTORY}/tool-run.json"
python3 -B auto-testing/scripts/validation_heal/authority.py \
--directory "${NATIVE_DIRECTORY}" --candidate "${VALIDATION_ROOT}/candidate.json" \
> "${VALIDATION_ROOT}/native-authority-manifest.json"
printf 'VALIDATION_HEAL_NATIVE_DIRECTORY=%s\n' "${NATIVE_DIRECTORY}" >> "${GITHUB_ENV}"
- name: Verify C11 physical storage and exact final-rename barrier
if: ${{ inputs.validation_case == 'heal.bucket-incarnation-recreation' }}
run: |
set -euo pipefail
NATIVE_DIRECTORY="${RUNNER_TEMP}/rustfs-validation-native-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}"
cp "${VALIDATION_ROOT}/tool-run.json" "${NATIVE_DIRECTORY}/tool-run.json"
python3 -B auto-testing/scripts/validation_heal/incarnation_native.py \
--directory "${NATIVE_DIRECTORY}" --candidate "${VALIDATION_ROOT}/candidate.json" \
--testing-commit "${{ inputs.auto_testing_sha }}" \
> "${VALIDATION_ROOT}/native-incarnation-manifest.json"
printf 'VALIDATION_HEAL_NATIVE_DIRECTORY=%s\n' "${NATIVE_DIRECTORY}" >> "${GITHUB_ENV}"
- name: Verify exact-source race native and runtime evidence
if: ${{ steps.race_native.outputs.required == 'true' }}
env:
AUTO_TESTING_SHA: ${{ inputs.auto_testing_sha }}
VALIDATION_CASE_ID: ${{ inputs.validation_case }}
run: |
set -euo pipefail
NATIVE_DIRECTORY="${RUNNER_TEMP}/rustfs-validation-native-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}"
cp "${VALIDATION_ROOT}/tool-run.json" "${NATIVE_DIRECTORY}/tool-run.json"
python3 -B auto-testing/scripts/validation_heal/race_native_evidence.py \
--directory "${NATIVE_DIRECTORY}" --candidate "${VALIDATION_ROOT}/candidate.json" \
--testing-commit "${AUTO_TESTING_SHA}" --case-id "${VALIDATION_CASE_ID}" \
> "${VALIDATION_ROOT}/native-race-manifest.json"
runtime_driver=race.py
if [ "${VALIDATION_CASE_ID}" = 'heal.admin-nolock-safety' ]; then
runtime_driver=nolock.py
fi
VALIDATION_HEAL_NATIVE_DIRECTORY="${NATIVE_DIRECTORY}" \
python3 -B "auto-testing/scripts/validation_heal/${runtime_driver}" preflight \
--candidate "${VALIDATION_ROOT}/candidate.json" --testing-commit "${AUTO_TESTING_SHA}" \
--case-id "${VALIDATION_CASE_ID}" \
> "${VALIDATION_ROOT}/native-race-runtime-preflight.json"
printf 'VALIDATION_HEAL_NATIVE_DIRECTORY=%s\n' "${NATIVE_DIRECTORY}" >> "${GITHUB_ENV}"
- name: Verify I05 source-plan or queued-successor native prerequisites
if: ${{ inputs.validation_case == 'heal.lock-timeout-and-caller-abort' && inputs.i05_diagnostic_branch != 'timeout' }}
env:
AUTO_TESTING_SHA: ${{ inputs.auto_testing_sha }}
I05_DIAGNOSTIC_BRANCH: ${{ inputs.i05_diagnostic_branch }}
run: |
set -euo pipefail
NATIVE_DIRECTORY="${RUNNER_TEMP}/rustfs-validation-native-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}"
cp "${VALIDATION_ROOT}/tool-run.json" "${NATIVE_DIRECTORY}/tool-run.json"
if [ "${I05_DIAGNOSTIC_BRANCH}" = 'queued-successor' ]; then
python3 -B auto-testing/scripts/validation_heal/race_native_evidence.py \
--directory "${NATIVE_DIRECTORY}" --candidate "${VALIDATION_ROOT}/candidate.json" \
--testing-commit "${AUTO_TESTING_SHA}" --case-id heal.lock-timeout-and-caller-abort \
> "${VALIDATION_ROOT}/native-i05-prerequisite.json"
if python3 scripts/check_functional_runners.py pf-testing; then
echo "performance_ready=online" >> "$GITHUB_OUTPUT"
else
python3 -B auto-testing/scripts/validation_heal/lock_source.py \
--directory "${NATIVE_DIRECTORY}" --candidate "${VALIDATION_ROOT}/candidate.json" \
--testing-commit "${AUTO_TESTING_SHA}" \
> "${VALIDATION_ROOT}/native-i05-source-prerequisites.json"
echo "performance_ready=offline" >> "$GITHUB_OUTPUT"
fi
printf 'VALIDATION_HEAL_NATIVE_DIRECTORY=%s\n' "${NATIVE_DIRECTORY}" >> "${GITHUB_ENV}"
- name: Bind F07 exact-candidate directory fsync evidence
if: ${{ inputs.validation_case == 'heal.checkpoint-and-mrf-corruption' && steps.candidate.outputs.source_sha == '1aa9be87d18cf941e5020f19e8622c26ba4b10ac' }}
run: |
set -euo pipefail
NATIVE_DIRECTORY="${RUNNER_TEMP}/rustfs-validation-native-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}"
cp "${VALIDATION_ROOT}/tool-run.json" "${NATIVE_DIRECTORY}/tool-run.json"
printf 'VALIDATION_HEAL_NATIVE_DIRECTORY=%s\n' "${NATIVE_DIRECTORY}" >> "${GITHUB_ENV}"
- name: Download pinned manager persistence and incarnation evidence
if: ${{ inputs.validation_case == 'heal.checkpoint-and-mrf-corruption' || inputs.validation_case == 'heal.bucket-incarnation-recreation' || inputs.validation_case == 'heal.outcome-counters-and-error-truth' }}
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
github-token: ${{ github.token }}
run-id: ${{ inputs.validation_tools_run_id }}
name: rustfs-validation-manager-native-${{ inputs.validation_tools_run_id }}-${{ inputs.validation_tools_run_attempt }}
path: ${{ runner.temp }}/rustfs-validation-manager-native-${{ github.run_id }}-${{ github.run_attempt }}
- name: Verify exact manager native prerequisites before fleet mutation
if: ${{ inputs.validation_case == 'heal.checkpoint-and-mrf-corruption' || inputs.validation_case == 'heal.bucket-incarnation-recreation' || inputs.validation_case == 'heal.outcome-counters-and-error-truth' }}
env:
VALIDATION_CASE: ${{ inputs.validation_case }}
run: |
set -euo pipefail
MANAGER_DIRECTORY="${RUNNER_TEMP}/rustfs-validation-manager-native-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}"
cp "${VALIDATION_ROOT}/tool-run.json" "${MANAGER_DIRECTORY}/tool-run.json"
python3 -B auto-testing/scripts/validation_heal/manager_fixture.py \
--directory "${MANAGER_DIRECTORY}" --candidate "${VALIDATION_ROOT}/candidate.json" \
--case-id "${VALIDATION_CASE}" > "${VALIDATION_ROOT}/manager-prerequisite-manifest.json"
printf 'VALIDATION_HEAL_MANAGER_DIRECTORY=%s\n' "${MANAGER_DIRECTORY}" >> "${GITHUB_ENV}"
- name: Download pinned C10 shared-reference encoder
if: ${{ inputs.validation_case == 'heal.dangling-and-orphan-safety' }}
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
github-token: ${{ github.token }}
run-id: ${{ inputs.validation_tools_run_id }}
name: rustfs-validation-shared-${{ inputs.validation_tools_run_id }}-${{ inputs.validation_tools_run_attempt }}
path: ${{ runner.temp }}/rustfs-validation-shared-${{ github.run_id }}-${{ github.run_attempt }}
- name: Verify C10 exact-source codec and original tests
if: ${{ inputs.validation_case == 'heal.dangling-and-orphan-safety' }}
run: |
set -euo pipefail
SHARED_DIRECTORY="${RUNNER_TEMP}/rustfs-validation-shared-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}"
cp "${VALIDATION_ROOT}/tool-run.json" "${SHARED_DIRECTORY}/tool-run.json"
python3 -B auto-testing/scripts/validation_heal/shared_fixture.py \
--directory "${SHARED_DIRECTORY}" --candidate "${VALIDATION_ROOT}/candidate.json" \
> "${VALIDATION_ROOT}/shared-fixture-manifest.json"
printf 'VALIDATION_HEAL_SHARED_DIRECTORY=%s\n' "${SHARED_DIRECTORY}" >> "${GITHUB_ENV}"
- name: Reject quarantined suite environment before canonical validation
if: ${{ !inputs.validation_tools_only && inputs.validation_cleanup_run_id == '' && inputs.validation_case != 'environment.inspect' }}
run: python3 scripts/functional_environment.py check --suite heal
- name: Run selected canonical validation case
if: ${{ !inputs.validation_tools_only && inputs.validation_cleanup_run_id == '' && inputs.validation_case != 'environment.inspect' }}
env:
VALIDATION_CASE: ${{ inputs.validation_case }}
AUTO_TESTING_SHA: ${{ inputs.auto_testing_sha }}
I05_DIAGNOSTIC_BRANCH: ${{ inputs.i05_diagnostic_branch }}
run: |
set -euo pipefail
test "$(git -C auto-testing rev-parse HEAD)" = "${AUTO_TESTING_SHA}"
./auto-testing/scripts/run_validation_case.sh \
"${VALIDATION_CASE}" "${VALIDATION_ROOT}/candidate.json" \
"${AUTO_TESTING_SHA}" "${VALIDATION_ROOT}/run"
- name: Inspect current Heal fleet resources without mutation
if: ${{ !inputs.validation_tools_only && inputs.validation_case == 'environment.inspect' }}
env:
AUTO_TESTING_SHA: ${{ inputs.auto_testing_sha }}
run: |
set -euo pipefail
test "$(git -C auto-testing rev-parse HEAD)" = "${AUTO_TESTING_SHA}"
python3 -B auto-testing/scripts/validation_heal/environment.py \
--candidate "${VALIDATION_ROOT}/candidate.json" \
--testing-commit "${AUTO_TESTING_SHA}" --evidence "${VALIDATION_ROOT}/environment"
- name: Audit and finalize retained validation environment
if: ${{ inputs.validation_cleanup_run_id != '' }}
env:
GH_TOKEN: ${{ github.token }}
FAILED_RUN_ID: ${{ inputs.validation_cleanup_run_id }}
AUTO_TESTING_SHA: ${{ inputs.auto_testing_sha }}
CLEANUP_RETAINED_EVIDENCE: ${{ inputs.validation_cleanup_retained_evidence }}
run: |
set -euo pipefail
test "$(git -C auto-testing rev-parse HEAD)" = "${AUTO_TESTING_SHA}"
gh run view "${FAILED_RUN_ID}" --repo "${GITHUB_REPOSITORY}" \
--json status,conclusion,headBranch,workflowName > "${VALIDATION_ROOT}/failed-run.json"
python3 - <<'PY'
import json
import os
from pathlib import Path
run = json.loads((Path(os.environ["VALIDATION_ROOT"]) / "failed-run.json").read_text())
if (run["status"] != "completed" or run["conclusion"] not in {"failure", "cancelled"}
or run["workflowName"] != "RustFS Functional Chain" or run["headBranch"] != os.environ["GITHUB_REF_NAME"]):
raise ValueError("cleanup source must be a completed failed or cancelled Functional Chain run")
PY
if [ "${CLEANUP_RETAINED_EVIDENCE}" = true ]; then
python3 -B scripts/prepare_retained_validation_recovery.py \
--report "${VALIDATION_ROOT}/retained-recovery-input.json"
else
gh run download "${FAILED_RUN_ID}" --repo "${GITHUB_REPOSITORY}" \
--pattern "rustfs-validation-${FAILED_RUN_ID}-*" --dir "${VALIDATION_ROOT}/failed-artifact"
fi
python3 -B auto-testing/scripts/validation_heal/recover_environment.py \
--failed-run-id "${FAILED_RUN_ID}" --artifact "${VALIDATION_ROOT}/failed-artifact" \
--candidate "${VALIDATION_ROOT}/candidate.json" --evidence "${VALIDATION_ROOT}/recovery"
- name: Show validation report
if: ${{ always() }}
run: |
if [ -f "${VALIDATION_ROOT}/run/report.md" ]; then
cat "${VALIDATION_ROOT}/run/report.md" >> "${GITHUB_STEP_SUMMARY}"
fi
- name: Upload validation evidence
if: ${{ always() && env.VALIDATION_ROOT != '' }}
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: rustfs-validation-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ env.VALIDATION_ROOT }}
if-no-files-found: error
validation-tools:
needs: [validation]
if: ${{ github.event_name == 'workflow_dispatch' && inputs.validation_package_sha == '' && inputs.validation_tools_only }}
uses: ./.github/workflows/rustfs-validation-tools.yml
with:
candidate_sha: ${{ needs.validation.outputs.source_sha }}
auto_testing_sha: ${{ inputs.auto_testing_sha }}
build_native_heal: ${{ inputs.validation_native_tools }}
build_shared_heal: ${{ inputs.validation_shared_tools }}
build_manager_native: ${{ inputs.validation_manager_tools }}
secrets: inherit
upgrade:
needs: [prepare]
if: ${{ always() && needs.prepare.result == 'success' && contains(fromJSON(needs.prepare.outputs.suites || '[]'), 'upgrade') }}
if: ${{ always() && needs.prepare.result == 'success' }}
uses: ./.github/workflows/rustfs-upgrade-test.yml
with:
chain_manifest: ${{ needs.prepare.outputs.manifest }}
@@ -715,7 +99,7 @@ jobs:
s3:
needs: [prepare, upgrade]
if: ${{ always() && needs.prepare.result == 'success' && contains(fromJSON(needs.prepare.outputs.suites || '[]'), 's3') }}
if: ${{ always() && needs.prepare.result == 'success' }}
uses: ./.github/workflows/rustfs-s3-compat-test.yml
with:
chain_manifest: ${{ needs.prepare.outputs.manifest }}
@@ -723,7 +107,7 @@ jobs:
kms:
needs: [prepare, s3]
if: ${{ always() && needs.prepare.result == 'success' && contains(fromJSON(needs.prepare.outputs.suites || '[]'), 'kms') }}
if: ${{ always() && needs.prepare.result == 'success' }}
uses: ./.github/workflows/rustfs-kms-test.yml
with:
chain_manifest: ${{ needs.prepare.outputs.manifest }}
@@ -731,7 +115,7 @@ jobs:
tier:
needs: [prepare, kms]
if: ${{ always() && needs.prepare.result == 'success' && contains(fromJSON(needs.prepare.outputs.suites || '[]'), 'tier') }}
if: ${{ always() && needs.prepare.result == 'success' }}
uses: ./.github/workflows/rustfs-tier-test.yml
with:
chain_manifest: ${{ needs.prepare.outputs.manifest }}
@@ -739,7 +123,7 @@ jobs:
storage:
needs: [prepare, tier]
if: ${{ always() && needs.prepare.result == 'success' && contains(fromJSON(needs.prepare.outputs.suites || '[]'), 'storage') }}
if: ${{ always() && needs.prepare.result == 'success' }}
uses: ./.github/workflows/rustfs-storage-test.yml
with:
chain_manifest: ${{ needs.prepare.outputs.manifest }}
@@ -747,7 +131,7 @@ jobs:
heal:
needs: [prepare, storage]
if: ${{ always() && needs.prepare.result == 'success' && contains(fromJSON(needs.prepare.outputs.suites || '[]'), 'heal') }}
if: ${{ always() && needs.prepare.result == 'success' }}
uses: ./.github/workflows/rustfs-heal-test.yml
with:
chain_manifest: ${{ needs.prepare.outputs.manifest }}
@@ -755,7 +139,7 @@ jobs:
pool:
needs: [prepare, heal]
if: ${{ always() && needs.prepare.result == 'success' && contains(fromJSON(needs.prepare.outputs.suites || '[]'), 'pool') }}
if: ${{ always() && needs.prepare.result == 'success' }}
uses: ./.github/workflows/rustfs-pool-expand-test.yml
with:
chain_manifest: ${{ needs.prepare.outputs.manifest }}
@@ -763,7 +147,7 @@ jobs:
security:
needs: [prepare, pool]
if: ${{ always() && needs.prepare.result == 'success' && contains(fromJSON(needs.prepare.outputs.suites || '[]'), 'security') }}
if: ${{ always() && needs.prepare.result == 'success' }}
uses: ./.github/workflows/rustfs-security-test.yml
with:
chain_manifest: ${{ needs.prepare.outputs.manifest }}
@@ -771,7 +155,7 @@ jobs:
replication:
needs: [prepare, security]
if: ${{ always() && needs.prepare.result == 'success' && contains(fromJSON(needs.prepare.outputs.suites || '[]'), 'replication') }}
if: ${{ always() && needs.prepare.result == 'success' }}
uses: ./.github/workflows/rustfs-replication-test.yml
with:
chain_manifest: ${{ needs.prepare.outputs.manifest }}
@@ -779,7 +163,7 @@ jobs:
fault-tolerance:
needs: [prepare, replication]
if: ${{ always() && needs.prepare.result == 'success' && contains(fromJSON(needs.prepare.outputs.suites || '[]'), 'fault-tolerance') }}
if: ${{ always() && needs.prepare.result == 'success' }}
uses: ./.github/workflows/rustfs-fault-tolerance-test.yml
with:
chain_manifest: ${{ needs.prepare.outputs.manifest }}
@@ -787,15 +171,15 @@ jobs:
table:
needs: [prepare, fault-tolerance]
if: ${{ always() && needs.prepare.result == 'success' && contains(fromJSON(needs.prepare.outputs.suites || '[]'), 'table') }}
if: ${{ always() && needs.prepare.result == 'success' }}
uses: ./.github/workflows/rustfs-table-test.yml
with:
chain_manifest: ${{ needs.prepare.outputs.manifest }}
secrets: inherit
performance:
needs: [prepare]
if: ${{ always() && needs.prepare.result == 'success' && contains(fromJSON(needs.prepare.outputs.suites || '[]'), 'performance') && needs.prepare.outputs.performance_ready == 'online' }}
needs: [prepare, table]
if: ${{ always() && needs.prepare.result == 'success' && needs.prepare.outputs.performance_ready == 'online' }}
uses: ./.github/workflows/rustfs-performance-test.yml
with:
chain_manifest: ${{ needs.prepare.outputs.manifest }}
@@ -804,7 +188,7 @@ jobs:
complete-chain:
needs: [prepare, upgrade, s3, kms, tier, storage, heal, pool, security, replication, fault-tolerance, table, performance]
if: ${{ always() && needs.prepare.result != 'skipped' }}
runs-on: ubuntu-latest
runs-on: smoke-testing
timeout-minutes: 10
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
@@ -823,10 +207,8 @@ jobs:
env:
CHAIN_MANIFEST: ${{ needs.prepare.outputs.manifest }}
CHAIN_NEEDS: ${{ toJSON(needs) }}
CHAIN_PLAN: ${{ needs.prepare.outputs.plan }}
run: >-
python3 scripts/functional_chain_evidence.py summarize
--allow-skipped "${{ contains(fromJSON(needs.prepare.outputs.optional_suites || '[]'), 'performance') && needs.prepare.outputs.performance_ready != 'online' && 'performance' || '' }}"
--directory "${RUNNER_TEMP}/chain-evidence"
--output "${RUNNER_TEMP}/chain-report.json"
- name: Retain chain report regardless of test verdict
@@ -843,10 +225,9 @@ jobs:
env:
CHAIN_MANIFEST: ${{ needs.prepare.outputs.manifest }}
CHAIN_NEEDS: ${{ toJSON(needs) }}
CHAIN_PLAN: ${{ needs.prepare.outputs.plan }}
run: >-
python3 scripts/functional_chain_evidence.py aggregate
--allow-skipped "${{ contains(fromJSON(needs.prepare.outputs.optional_suites || '[]'), 'performance') && needs.prepare.outputs.performance_ready != 'online' && 'performance' || '' }}"
--allow-skipped "${{ needs.prepare.outputs.performance_ready == 'online' && '' || 'performance' }}"
--directory "${RUNNER_TEMP}/chain-evidence"
--output "${RUNNER_TEMP}/chain-complete.json"
- name: Upload complete-chain evidence
+62 -41
View File
@@ -29,6 +29,11 @@ on:
description: 'Reset the nodes after the test (DESTROYS test data/config)'
type: boolean
default: true
repository_dispatch:
# Chain handoff: dispatched when the storage suite finishes. Heal runs
# exactly once per chain; the pool expansion workflow no longer embeds
# its own heal pass.
types: [rustfs-chain-heal]
permissions:
contents: read
@@ -55,11 +60,10 @@ env:
jobs:
heal-test:
runs-on: smoke-testing
# 90-minute heal run (see the step below) plus setup, report and cleanup.
timeout-minutes: 120
timeout-minutes: 60
# Standalone manual run, or one link of the nightly functional chain
# (storage -> heal -> pool). Pool expansion no longer re-runs heal.
if: ${{ inputs.chain_manifest != '' || github.event_name == 'workflow_dispatch' }}
if: ${{ inputs.chain_manifest != '' || github.event_name == 'workflow_dispatch' || github.event_name == 'repository_dispatch' }}
steps:
- name: Checkout chain tooling
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
@@ -105,13 +109,6 @@ jobs:
warp --version || true
df -h /data | tail -1
- name: Admit clean shared environment
id: environment
timeout-minutes: 5
env:
FUNCTIONAL_ENVIRONMENT_RECEIPT: ${{ runner.temp }}/environment-heal.json
run: python3 scripts/functional_environment.py enter --suite heal
- name: Cleanup environment (before)
timeout-minutes: 5
if: ${{ inputs.cleanup_before != 'false' }}
@@ -155,9 +152,10 @@ jobs:
./auto-testing/rustfs_heal_test.sh "${ARGS[@]}" --log-file "${LOG_FILE}"
- name: Run heal test (write -> outage -> heal -> verify)
timeout-minutes: 90
timeout-minutes: 45
id: test
# Collect reports and backlog evidence before the final suite verdict.
# Case failures keep the run green: the report and the backlog
# issue manager carry the product signal.
continue-on-error: true
run: |
./auto-testing/rustfs_heal_test.sh \
@@ -243,8 +241,9 @@ jobs:
RESULT=success
fi
# Preserve harness completion separately; the final verdict checks
# every case after retaining reports and cleanup evidence.
# Product gate: failing steps keep the run green — they are reported
# below and tracked in rustfs/backlog. Only harness/environment
# breakdowns turn the workflow red.
STEPS_TOTAL="$(grep -cE '^\[HEAL-STEP\]' "${LOG_FILE}" 2>/dev/null || true)"
STEPS_FAIL="$(grep -cE '^\[HEAL-STEP\].* FAIL$' "${LOG_FILE}" 2>/dev/null || true)"
STEPS_TOTAL=$(( ${STEPS_TOTAL:-0} + 0 )); STEPS_FAIL=$(( ${STEPS_FAIL:-0} + 0 ))
@@ -282,7 +281,7 @@ jobs:
fi
} | tee "${REPORT_FILE}"
cat "${REPORT_FILE}" >> "${GITHUB_STEP_SUMMARY}"
# Preserve the harness verdict separately from the final case verdict.
# Red only for harness/environment breakdowns; step failures stay green.
[ "${HARNESS_OK}" = "1" ]
- name: Upload functional report to dashboard
@@ -378,9 +377,8 @@ jobs:
if-no-files-found: error
- name: Cleanup environment (after)
id: environment_cleanup
timeout-minutes: 5
if: ${{ always() && inputs.cleanup_after != 'false' && steps.environment.outcome == 'success' }}
if: ${{ always() && inputs.cleanup_after != 'false' }}
run: |
set -euo pipefail
read -r -a NODES <<< "${RUSTFS_NODES:-vm000 vm001 vm002}"
@@ -398,21 +396,54 @@ jobs:
'
done
- name: Release environment only after successful cleanup
if: ${{ always() && steps.environment.outcome == 'success' }}
timeout-minutes: 5
- name: "Continue functional chain (next: Pool expansion)"
# Only chain-triggered runs forward to the next suite; standalone
# workflow_dispatch runs stop after their own cleanup. A failed
# handoff must never pass silently: it retries, then files an alert
# issue in rustfs/backlog so a stalled chain is visible.
if: ${{ always() && github.event_name == 'repository_dispatch' }}
continue-on-error: true
env:
FUNCTIONAL_ENVIRONMENT_RECEIPT: ${{ runner.temp }}/environment-heal.json
CLEANUP_OUTCOME: ${{ steps.environment_cleanup.outcome }}
run: python3 scripts/functional_environment.py finish --suite heal --cleanup-outcome "$CLEANUP_OUTCOME"
- name: Retain environment ownership and cleanup receipt
if: ${{ always() }}
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: functional-environment-heal-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/environment-heal.json
if-no-files-found: error
GH_TOKEN: ${{ secrets.PF_TESTING_GH_TOKEN }}
run: |
set -uo pipefail
if [ -z "${GH_TOKEN:-}" ]; then
echo "PF_TESTING_GH_TOKEN is not configured; cannot dispatch the next suite" >&2
exit 1
fi
DISPATCHED=0
for attempt in 1 2 3; do
if gh api --method POST repos/rustfs/rustfs/dispatches \
-f event_type='rustfs-chain-pool' \
-F 'client_payload[from_suite]=heal'; then
echo "dispatched next suite Pool expansion (attempt ${attempt})"
DISPATCHED=1
break
fi
echo "dispatch attempt ${attempt} failed; retrying in ${attempt}0s" >&2
sleep "${attempt}0"
done
if [ "${DISPATCHED:-0}" -ne 1 ]; then
echo "ERROR: functional chain stalled: could not dispatch Pool expansion after 3 attempts" >&2
TITLE="[functional][chain] stalled after heal (run ${GITHUB_RUN_ID})"
BODY_FILE="$(mktemp)"
{
echo "The functional chain could not hand off from **heal** to **Pool expansion** after 3 attempts."
echo ""
echo "- Failed suite job: ${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}"
echo "- Expected next event: 'rustfs-chain-pool'"
echo "- Likely cause: PF_TESTING_GH_TOKEN lacks contents:write on rustfs/rustfs, or the GitHub API was unavailable."
echo "- Recovery: re-dispatch manually with"
FENCE="$(printf "\x60\x60\x60")"; echo " ${FENCE}"
echo " gh api --method POST repos/rustfs/rustfs/dispatches -f event_type='rustfs-chain-pool'"
FENCE="$(printf "\x60\x60\x60")"; echo " ${FENCE}"
} > "${BODY_FILE}"
gh issue create -R rustfs/backlog --title "${TITLE}" \
--body-file "${BODY_FILE}" --label functional-test \
|| gh issue create -R rustfs/backlog --title "${TITLE}" --body-file "${BODY_FILE}" \
|| echo "could not file the stall alert issue either; check the token" >&2
exit 1
fi
- name: Notify on failure
if: failure()
@@ -441,13 +472,3 @@ jobs:
name: functional-chain-heal-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/chain-heal-${{ github.run_id }}-${{ github.run_attempt }}/heal.json
if-no-files-found: error
- name: Enforce suite and report result
if: ${{ always() }}
env:
SUITE_OUTCOME: ${{ steps.test.outcome }}
REPORT_OUTCOME: ${{ steps.chain_report.outcome }}
run: |
test "$SUITE_OUTCOME" = success
test "$REPORT_OUTCOME" = success
python3 scripts/functional_chain_evidence.py verify-report --suite heal --report "${FUNCTIONAL_ARTIFACTS_DIR}/steps.md"

Some files were not shown because too many files have changed in this diff Show More