mirror of
https://github.com/rustfs/rustfs.git
synced 2026-11-10 20:28:47 +00:00
Compare commits
2 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 Chris An
|
8b7bbdb17d | ||
|
Chris An
|
2780bba67b |
@@ -1,43 +1,38 @@
|
||||
---
|
||||
name: rustfs-release-publish
|
||||
description: Publish a RustFS version through Console, release-branch CI, preview acceptance, human confirmation, and final publication. Use only for an explicit release request.
|
||||
description: Publish a RustFS version through frozen Linux candidate acceptance, desktop completion, manifest approval, byte-preserving publication and post-release follow-up. Use only for an explicit release request.
|
||||
---
|
||||
# RustFS Release
|
||||
|
||||
Require the exact final SemVer target before version edits or publication. If unspecified, inspect current releases/tags and ask with concrete candidates; do not guess the channel or bump. A named bump type resolves to an exact version that still needs confirmation. Report the confirmed target verbatim; resolve any later mismatch before proceeding. Independent read-only preflight may continue.
|
||||
Require the exact final SemVer target before version edits or publication. If unspecified, inspect releases/tags and ask with concrete candidates; do not guess the channel or bump. Reuse the user's confirmed target and authorized delivery scope. Independent read-only preflight may continue.
|
||||
|
||||
## Invariants
|
||||
|
||||
- Cargo stores `<target>`, never `-preview.N`. Before publication, installation defaults retain the previous available deliverable. Final and preview binaries embed their own tag names, so version/asset names differ despite identical source.
|
||||
- Use annotated tags without `v`. Preview names are exactly `<target>-preview.<digits>`; choose the next unused iteration after fetching tags. Preview Releases are prereleases, never Latest; publish only versioned assets/exact Docker tags, never latest assets, R2, Helm, moving Docker channels, or installation/banner updates.
|
||||
- `release/<target>` owns the candidate. Retain its history, backport only reviewed fixes, and never merge main wholesale or force-push it. Pin `PREVIEW_HASH` to its exact commit with complete passing CI. Preview and final tags both point there; never tag a later main/branch HEAD or insert another bump.
|
||||
- Phase failures block dependents. Keep preview Releases until final publication and notes generation finish; CI removes Releases while retaining tags. Never move an existing preview or final tag.
|
||||
- Cargo and every candidate executable contain the final product version. `<target>-preview.<run_number>` is an immutable candidate identifier, not a public tag or displayed binary version.
|
||||
- Freeze the exact source, reviewed workflow recipe, toolchain, Console, CLI, dependency lock, previous stable version and validation plan. Newer unrelated main or Console releases do not change those inputs.
|
||||
- Retain `release/<target>` history when a release branch is used; backport only reviewed fixes. Pin `CANDIDATE_SHA` to its exact CI-validated commit. Never force-push a release branch or move a public tag.
|
||||
- Build Linux first. Complete field acceptance before macOS/Windows and packaging. Full certification and final acceptance are required before publication; a partial Linux PASS grants no release authority.
|
||||
- Promote the accepted bytes. Final publication never recompiles binaries or rebuilds packages/images. Infrastructure retries reuse successful artifacts; changed source or inputs require a new candidate and affected acceptance.
|
||||
- Phase failures block dependents. No latest/default-channel update before versioned delivery. Use the [pipeline runbook](../../../docs/operations/release-pipeline.md) for executable entry points, locks and recovery.
|
||||
|
||||
## Read by Phase
|
||||
|
||||
Read only the current phase's procedure; retain completed evidence across turns. Follow root verification, Git/PR, and authorization rules.
|
||||
## Read by phase
|
||||
|
||||
| Phase | Procedure |
|
||||
|---|---|
|
||||
| 0 | Clean tree, fetch main/tags, verify GitHub access, then complete the mandatory [Console release gate](references/console-gate.md). A green Console build alone is insufficient. |
|
||||
| 1–2 | [Prepare/reuse the release candidate and run exact-SHA branch CI](references/release-candidate.md). Source version changes use `rustfs-release-version-bump` with authorized delivery scope. |
|
||||
| 3 | [Verify the preview build, assets, channels, and notes](references/preview-release.md). |
|
||||
| 4–5 | [Run the downloaded binary, Console CRUD, and latest-rc matrix](references/preview-acceptance.md). Every check must pass. |
|
||||
| Confirmation | Complete the gate below and end the turn. |
|
||||
| 6 | After fresh confirmation, [publish and verify the final tag](references/final-release.md) at `PREVIEW_HASH`. |
|
||||
| 7 | [Reconcile Security Advisories, milestones, installation references, and the website; reconcile main and remove the completed release branch](references/post-release-updates.md) only after complete publication. |
|
||||
| 0 | Clean tree, fetch main/tags, verify access, then complete the mandatory [Console gate](references/console-gate.md). |
|
||||
| 1–2 | [Prepare the exact source and run branch CI](references/release-candidate.md). Version edits use `rustfs-release-version-bump`. |
|
||||
| 3 | [Build and verify the Linux candidate](references/preview-release.md). |
|
||||
| 4–5 | [Run artifact, field, Console and rc acceptance](references/preview-acceptance.md), then complete desktop/package/image/chart validation. |
|
||||
| Confirmation | Report the complete manifest and obtain the confirmation below. |
|
||||
| 6 | [Publish the accepted manifest](references/final-release.md). |
|
||||
| 7 | [Reconcile advisories, milestones, installation references, website and release branch](references/post-release-updates.md). |
|
||||
|
||||
## Human Confirmation
|
||||
## Human confirmation
|
||||
|
||||
After Phases 3–5 pass, recheck that the release branch still equals `PREVIEW_HASH` and that the current Console asset tag/digest match the preview's recorded build inputs. Report the target, branch, preview tag/hash, branch CI run, preview Release URL, accepted Console asset identity/result, and rc matrix. Explicitly ask whether to publish the final tag and end the turn without creating/pushing it.
|
||||
After all acceptance passes, report the target, source SHA, source CI run, Linux and completion run URLs, final manifest SHA-256, accepted Console/CLI identities, compatibility/performance results and backlog report. Explicitly ask whether to publish this exact manifest and end the turn without dispatching publication.
|
||||
|
||||
Only a new affirmative user reply to that exact report authorizes Phase 6. The original release request, earlier confirmation, silence, or automated follow-up does not count. Confirmation binds the target, preview tag, hash, and accepted build inputs; a changed value or repeated/failed acceptance cycle invalidates it. Revalidate and obtain fresh confirmation before continuing.
|
||||
Only a new affirmative reply to that exact report authorizes Phase 6. An earlier general release request, silence or automatic continuation does not count. Reuse confirmation for retries of that same manifest; do not ask again merely because a destination failed. A changed source, input, manifest or acceptance result invalidates it and requires renewed acceptance/confirmation.
|
||||
|
||||
## Recovery
|
||||
## Recovery and reporting
|
||||
|
||||
- Before the final tag exists, land fixes on main, backport to the candidate, verify target versions, and restart branch CI/acceptance. Use a new preview iteration once a preview was tagged; main may advance independently.
|
||||
- If only an external build input changed, record it, create a new preview iteration even when the source SHA is unchanged, and repeat the affected build and artifact acceptance before fresh confirmation. Reuse source checks whose inputs remain valid.
|
||||
- Once the final tag exists, retry publication jobs for that exact tag. A source fix requires a newly confirmed target, not retagging or a newer preview. Phase 7 retries recheck artifact availability and superseding releases.
|
||||
- If abandoned after the bump merged, report the unpublished version on main and record whether to revert or let the next release overwrite it. Deleting unpublished candidate branches or preview tags requires an explicit decision. Completed release branches follow Phase 7 cleanup; keep release tags.
|
||||
|
||||
Report completed phase results and incomplete work with evidence: Console previous/latest tags, release need, hash and run/Release URLs; target, branch CI/SHA, previews and hash, confirmation state; preview/final assets, latest-channel state, Console/rc results and cleanup; advisory reconciliation results and pending disclosure decisions; both milestone states, installation PR checks/merge state, and website text/link/PR/deployment state. Keep artifact publication separate from follow-up delivery; do not call an open PR a live update. Report deviations and their authorization.
|
||||
Resume failed infrastructure jobs without rebuilding successful units. Fix product defects on main, backport when needed, and start a new candidate. After a final tag exists, retry the same bytes or use a new patch version; never move the tag. Do not conflate implemented, CI-passed, field-accepted, approved, published and post-release-complete states. Report incomplete work and deviations with evidence; a tag or green build alone is not a completed release.
|
||||
|
||||
@@ -4,7 +4,7 @@ Read during Phase 0, before changing RustFS version files or tags. Follow the pa
|
||||
|
||||
### Console release gate
|
||||
|
||||
Complete this gate before changing any RustFS version file or creating any RustFS tag. RustFS `build.yml` downloads the asset returned by `repos/rustfs/console/releases/latest`, so a successful Console build alone is insufficient.
|
||||
Complete this gate before changing any RustFS version file or creating any RustFS tag. Candidate creation resolves `repos/rustfs/console/releases/latest` once and freezes the exact asset for every build, so a successful Console build alone is insufficient. After freezing, later Console releases do not silently change or invalidate the accepted inputs.
|
||||
|
||||
1. Read the latest published Console tag and compare it with Console `main`:
|
||||
|
||||
|
||||
@@ -1,22 +1,13 @@
|
||||
# Final Release Publication
|
||||
# Final release publication
|
||||
|
||||
Read only after fresh user confirmation of the preview acceptance report. The [parent confirmation gate](../SKILL.md#human-confirmation) applies.
|
||||
Read only after fresh confirmation of the complete candidate report. The [parent confirmation gate](../SKILL.md#human-confirmation) applies.
|
||||
|
||||
## Phase 6 — Publish the final tag on the validated commit
|
||||
Dispatch `release-publish.yml` on main with the successful completion run ID and approved `release-manifest.json` SHA-256. Do not create the tag manually. The workflow revalidates source, bytes and trusted certification, then creates the annotated final tag at `CANDIDATE_SHA`. A recipe change fails closed and requires a new candidate; unrelated main changes do not require a rebuild.
|
||||
|
||||
No second source version bump. Before creating the final tag, recheck the remote `release/<target>` HEAD equals `PREVIEW_HASH` and the current Console release tag/asset digest still match the accepted preview inputs. If either changed after confirmation, the approval no longer applies; validate the changed inputs with a new preview and obtain fresh confirmation. The final tag goes on the exact commit the preview validated:
|
||||
Require both versioned publication and channel publication jobs to succeed. Verify the final tag's commit and manifest identity; all six platform ZIPs and compatibility aliases; DEB/RPM, checksums, SBOM, provenance and evidence bundle; both multi-architecture image variants in Docker Hub, GHCR and Quay; and the accepted Helm chart in the chart repository. Download a host-compatible final artifact and confirm its digest is exactly the accepted candidate digest and its version/source output agrees.
|
||||
|
||||
```bash
|
||||
git fetch origin --tags
|
||||
git rev-parse "<preview-tag>^{commit}" # must equal PREVIEW_HASH — abort if not
|
||||
git tag -a "<target>" -m "Release <target>" "$PREVIEW_HASH"
|
||||
git push origin "<target>"
|
||||
```
|
||||
For stable releases verify GitHub Latest, R2 download aliases and `latest.json` unless a newer stable release already owns those channels. Prereleases must remain GitHub prereleases and leave stable pointers unchanged. Verify generated release notes compare against the frozen previous stable release, excluding internal previews.
|
||||
|
||||
- CI rebuilds from the same source with a different tag name, so the binary self-reports `<target>`. Other inputs can change: `build.yml` reads Console `releases/latest` during each build. Compare the actual Console tag, URL, and verified digest in every final build job with the accepted preview record, and inspect available build provenance for other relevant changes. A pre-tag check cannot prevent a later input change. On a mismatch or missing evidence, repeat affected acceptance against the final artifacts and report the limitation; failed acceptance leaves release validation incomplete even if publication already succeeded. Never move the final tag to conceal a failure.
|
||||
- Verify the final tag's complete publication path: all matrix and release jobs green; `gh release view "<target>"` shows the full versioned and `-latest` asset set plus checksums, SBOM, and provenance; Docker and Helm workflows succeed; `latest.json` points to `<target>`. A stable target must have `isPrerelease=false` and `isLatest=true`. An alpha/beta/rc target must have `isPrerelease=true`; GitHub does not permit prereleases to be Latest, but the project `latest.json` still advances to the final non-preview target.
|
||||
- Verify the final Release body contains `## What's Changed` and a Full Changelog link. When `PREVIOUS_DELIVERABLE` exists, the link MUST be `https://github.com/rustfs/rustfs/compare/<PREVIOUS_DELIVERABLE>...<target>` and the baseline MUST equal the preview Release baseline; for example, both `1.0.0-beta.12-preview.1` and `1.0.0-beta.12` compare from `1.0.0-beta.11`.
|
||||
- Verify the preview cleanup: `cleanup-preview-releases` must succeed and `gh release view "<preview-tag>"` must then report `release not found` for every preview iteration of this target. Each surviving preview tag must still resolve to its own recorded commit; the accepted preview resolves to `PREVIEW_HASH`. If the job failed, delete the leftover Releases manually with `gh release delete "<preview-tag>" --yes` and report it.
|
||||
- Verify a host-compatible final artifact's checksum and `./rustfs --version`; it must report `<target>` and the expected source SHA. This identity check does not replace affected acceptance when build inputs changed.
|
||||
On infrastructure failure retry the same workflow/seal; matching destinations are reused. Never clobber an asset, force a tag, rebuild the image or silently accept a changed digest. A source fix after the final tag requires a new patch version. Cross-registry writes are not atomic: report the unfinished destination rather than claiming publication complete.
|
||||
|
||||
Never pass `--cleanup-tag` when removing preview Releases. Remove them manually only if `cleanup-preview-releases` failed, after final publication and notes generation. Record deleted Releases and surviving tags.
|
||||
New candidates create no preview Releases to clean up. For a legacy preview release, preserve its tag and remove its Release only after final delivery and notes are verified. Do not pass `--cleanup-tag`. Complete the [post-release follow-up](post-release-updates.md) separately.
|
||||
|
||||
@@ -24,7 +24,7 @@ Reconcile advisories after every completed final release, including Console-only
|
||||
|
||||
1. Re-read the published deliverable state before edits or a retry. If a newer release has superseded this target, do not downgrade installation references or the website announcement. Report the superseding release and leave those defaults intact.
|
||||
2. Invoke `rustfs-release-version-bump` with stage `post-release`, the exact target, and the authorized delivery scope. Prepare the follow-up on current main; do not edit or move either validated release tag. Cargo versions may already be preparing the next target and must remain untouched.
|
||||
3. Verify the exact Docker image manifest, Release download links, RPM source archive, and Helm repository entry before publishing the installation PR. The chart workflow overrides chart/app versions from the final tag, pins the source checkout to that release, and waits for its default Docker image before packaging.
|
||||
3. Verify the exact Docker image manifest, Release download links, RPM source archive, and Helm repository entry before publishing the installation PR. The chart is packaged and tested before sealing; publication uploads those same bytes only after versioned Docker images are available.
|
||||
4. Report the installation PR and its merge state separately from the completed release. Reuse existing merge authorization and repository checks; do not infer new merge authority from this reference.
|
||||
|
||||
## rustfs.com top banner
|
||||
@@ -54,4 +54,4 @@ Use the existing announcement configuration from [website PR #112](https://githu
|
||||
1. Compare the exact published tag with current main. Audit every release-side commit since their merge base, including inherited backports. An ahead count is a history difference, not proof of missing changes. Use ancestry and patch equivalence, then inspect current code and regression evidence for squashed, adapted, superseded, or reverted changes. Record each commit as present, superseded, release-only, or missing; a positive `git cherry` result alone does not prove a missing fix.
|
||||
2. Bring missing shared changes to main through a reviewed PR and the required checks before closing the follow-up. Preserve main's later changes; decide version bumps and pinned Console configuration separately. If a history merge is needed, use a merge commit and verify the release commit is an ancestor of main; squash does not establish that ancestry. Do not merge main into the published release branch or move its tags. Report unresolved changes instead of claiming reconciliation is complete.
|
||||
3. After publication and authorized follow-ups are complete, delete the remote `release/<target>` branch. Immediately before deletion, verify its HEAD still equals the published tag's commit and no open PR targets or originates from it. Preserve a branch with newer work or open PRs and report the blocker. Use a deletion with an expected-old-SHA check to avoid removing a concurrently advanced branch.
|
||||
4. Verify the remote branch is absent and the final and preview tags still resolve to their recorded commits. Remove a local release branch only if it is task-owned, unchecked out, and has no unpreserved work. Record cleanup separately from artifact publication; an explicitly authorized deletion before reconciliation does not make unresolved main changes complete.
|
||||
4. Verify the remote branch is absent and the final tag and any legacy preview tags still resolve to their recorded commits. Remove a local release branch only if it is task-owned, unchecked out, and has no unpreserved work. Record cleanup separately from artifact publication; an explicitly authorized deletion before reconciliation does not make unresolved main changes complete.
|
||||
|
||||
@@ -1,10 +1,10 @@
|
||||
# Preview Artifact Acceptance
|
||||
# Candidate Artifact Acceptance
|
||||
|
||||
Read after Phase 3 succeeds. Complete every check below before the parent skill's manual confirmation gate. These checks cover the downloaded artifact, embedded Console, and latest rc client.
|
||||
Read after Phase 3 succeeds. Complete every check below before the parent skill's manual confirmation gate. These checks cover the downloaded artifact, embedded Console, and the rc release frozen in the candidate inputs.
|
||||
|
||||
## Phase 4 — Run the artifact locally, verify the console
|
||||
|
||||
Use a unique task-owned scratch directory. Select the versioned archive matching the host OS, architecture, and runtime, download its checksum file, and verify the archive before extraction. Do not substitute a different build when a matching artifact is unavailable; report that acceptance gap. The shell example below assumes the verified executable is at `RUSTFS_BINARY` (`rustfs.exe` on Windows), `SCRATCH` is the task directory, and `S3_PORT` and `CONSOLE_PORT` are distinct available ports. Use equivalent native commands when the shell differs.
|
||||
Use a unique task-owned scratch directory. Select the versioned archive matching the acceptance host, and verify it against the sealed manifest before extraction. Before desktop completion, use the Linux acceptance environment rather than rebuilding for the local desktop. Do not substitute a different build when a matching artifact is unavailable; report that acceptance gap. The shell example below assumes the verified executable is at `RUSTFS_BINARY` (`rustfs.exe` on Windows), `SCRATCH` is the task directory, and `S3_PORT` and `CONSOLE_PORT` are distinct available ports. Use equivalent native commands when the shell differs.
|
||||
|
||||
Bind both listeners to loopback. Choose task-only credentials, set loopback proxy exclusions for the server/client processes, and record the server PID. A port probe does not reserve a port; retry startup with fresh ports on a bind conflict rather than connecting to an existing service.
|
||||
|
||||
@@ -24,16 +24,16 @@ PREVIEW_SERVER_PID=$!
|
||||
|
||||
Checks (all must pass):
|
||||
|
||||
- `"$RUSTFS_BINARY" --version` reports the preview tag name and the short SHA of `PREVIEW_HASH`. Reporting `<target>` without the `-preview.N` suffix means the build did not embed the tag — treat as FAIL and investigate before proceeding.
|
||||
- `"$RUSTFS_BINARY" --version` reports the exact final `<target>` and full `CANDIDATE_SHA`. A preview suffix, another version or another source is a failure.
|
||||
- Wait with a bounded readiness deadline while checking the recorded server process remains alive; `curl --noproxy '*' -fsS "$S3_ENDPOINT/health/ready"` must return ready from this process's listener.
|
||||
- Startup log shows the embedded console being served (this was the regression that `fix(release): require embedded console assets` guards).
|
||||
- Open `CONSOLE_ENDPOINT` in the browser and log in with the task credentials. The dashboard must render without JS console errors; create a bucket, upload a file, download it back (byte-identical), and delete the object and bucket. Keep the server running for Phase 5.
|
||||
|
||||
## Phase 5 — Validate with the latest rc client
|
||||
## Phase 5 — Validate with the pinned rc client
|
||||
|
||||
`rc` is the RustFS CLI client from <https://github.com/rustfs/cli>.
|
||||
|
||||
- Resolve the latest release with `gh api repos/rustfs/cli/releases/latest --jq .tag_name`; verify the local client's version or download and verify a host-compatible client into the task directory. Set `RC_BIN` to that executable and record its version/digest. Avoid changing the user's installed client for this check.
|
||||
- Read the CLI release and asset digests from `inputs.json`; verify the local client's version or download a host-compatible asset from that exact release into the task directory. If the host is a desktop, resolve its asset within the same pinned CLI release and record the digest; do not independently follow `latest`. Set `RC_BIN` to that executable and record its version/digest. Avoid changing the user's installed client for this check.
|
||||
- Inspect the client's current help for an isolated configuration option and use it when supported. Otherwise choose a unique `PREVIEW_ALIAS` and confirm it is absent before creating it; never overwrite an existing alias. Use task-owned bucket/user names and input/output files. The names below are safe only inside the newly started isolated server.
|
||||
- Point the client at `S3_ENDPOINT` with loopback proxy exclusions and run the command matrix, recording PASS/FAIL per command:
|
||||
|
||||
@@ -60,3 +60,9 @@ cmp <local-file> "$SCRATCH/out"
|
||||
```
|
||||
|
||||
- Any FAIL blocks acceptance. On success or failure, stop only the recorded task process and remove only task-created aliases/data after retaining the needed failure evidence. Do not erase an existing client configuration or unrelated server data.
|
||||
|
||||
## Field acceptance and desktop completion
|
||||
|
||||
Before accepting the Linux manifest, record Scanner/Heal, multi-pool, memory, major features, high-concurrency 503, performance and previous-version integrity/upgrade results in backlog. Preserve the release-specific agreed limits and baseline; a PASS label without artifact identities or test results does not satisfy acceptance.
|
||||
|
||||
Dispatch `release-complete.yml` only after that report passes, using the exact Linux manifest digest and backlog URL. Require native macOS/Windows checks, package extraction, image startup/scans, chart tests and complete certification to pass. Complete any desktop Console/client checks applicable to the release against these same archives. Record the completion run and final manifest SHA-256 before entering the parent confirmation gate. Partial Linux acceptance does not authorize final publication.
|
||||
|
||||
@@ -1,25 +1,9 @@
|
||||
# Preview Release Verification
|
||||
# Linux candidate verification
|
||||
|
||||
Read for Phase 3 after the candidate passed exact-SHA CI. The [parent release invariants](../SKILL.md) apply.
|
||||
Read for Phase 3 after the source passed exact-SHA CI. The [parent invariants](../SKILL.md) apply; commands and recovery live in the [pipeline runbook](../../../../docs/operations/release-pipeline.md).
|
||||
|
||||
## Preview channels
|
||||
Dispatch `release-candidate.yml` on main with the confirmed final version and `CANDIDATE_SHA`. Record the run ID, frozen `inputs.json`, every Linux archive digest and `linux-manifest.json` SHA-256. Require the four Linux builds, native startup/S3 checks, previous-release upgrade matrix and Linux seal to succeed. Missing, skipped, cancelled or failed required work is incomplete acceptance.
|
||||
|
||||
- Use `<target>-preview.N` for every target, e.g. `1.0.0-beta.10-preview.3` or `1.1.0-preview.1`.
|
||||
- The canonical suffix is exactly `-preview.<digits>`. `build.yml` recognizes it before alpha/beta/rc classification and routes it to the preview-only path; any other tag containing `-preview` fails closed instead of being treated as a release.
|
||||
- A preview Release MUST be published with `isPrerelease=true` and `isLatest=false`. Any `*-latest` preview asset or preview-triggered `latest.json`, R2, Helm, or moving Docker channel tag is a pipeline failure. Docker images may use the exact preview version tag, including its existing variant suffixes.
|
||||
- Preview Releases are cleaned up by the `cleanup-preview-releases` job after `publish-release` succeeds for the deliverable tag. It deletes every Release whose tag is exactly `<target>-preview.<digits>` and never passes `--cleanup-tag`, so the tags survive.
|
||||
Verify the frozen Console identity equals the one accepted in Phase 0. Every matrix job consumes that same digest and exact toolchain. The binary must report the final product version and source commit; its candidate ID is separate. No public tag, Release, registry tag, R2 release path, Helm repository or latest pointer should change during this stage.
|
||||
|
||||
## Phase 3 — CI and preview Release verification
|
||||
|
||||
- Find and watch the tag build: `gh run list --workflow build.yml --branch "<preview-tag>" --limit 1` then `gh run watch <run-id>`. Every build matrix target must succeed (linux x86_64/aarch64 × musl/gnu, macos-aarch64, windows-x86_64).
|
||||
- Confirm the preview tag resolves to the Phase 2 CI-validated `PREVIEW_HASH`. If the release branch advances during acceptance, repeat Phases 2–5 with a new preview tag.
|
||||
- Confirm the Release publication jobs (`create-release`, `upload-release-assets`, and `publish-release`) succeed while `update-latest-version` is skipped.
|
||||
- Record the inputs actually consumed by each build job: source SHA and the embedded Console release tag, asset URL, and verified SHA256 from the download logs. The workflow reads Console `releases/latest` separately for each build; confirm the matrix used the Console asset accepted in Phase 0. A missing identity or differing asset blocks acceptance until resolved. Retain available toolchain and build provenance with the artifact checksums; the same source SHA alone does not establish equivalent builds.
|
||||
- Verify `gh release view "<preview-tag>" --json isPrerelease,assets,url`: `isPrerelease` must be `true`, and the Release must contain all 6 versioned platform zips, checksums, SBOM, and provenance with no `-latest` assets. Confirm `gh api repos/{owner}/{repo}/releases/latest --jq .tag_name` does not return `<preview-tag>`.
|
||||
- Record `PREVIOUS_DELIVERABLE`, selected from published Releases by `publishedAt` after excluding the current tag and every `-preview.N` tag. Verify `gh release view "<preview-tag>" --json body --jq .body` contains `## What's Changed` and, when `PREVIOUS_DELIVERABLE` exists, `**Full Changelog**: https://github.com/rustfs/rustfs/compare/<PREVIOUS_DELIVERABLE>...<preview-tag>`. For a repository with no previous deliverable, verify a Full Changelog link exists and record the GitHub baseline fallback.
|
||||
- Confirm Helm is skipped. If preview Docker images are published, confirm they use only exact preview tags (and variant suffixes), with no `latest`, `alpha`, `beta`, or `rc` channel updates. Preview validation covers the built RustFS binaries, embedded console, and rc compatibility; it does not authorize advancing any installation default.
|
||||
|
||||
## Release notes
|
||||
|
||||
- When a previous deliverable exists, GitHub Release notes for the preview and final tags MUST use it as their shared comparison baseline: the most recently published non-preview Release before the target. Internal `-preview.N` Releases are explicitly excluded from that selection, even when they point at the same commit as the final tag — cleanup runs after the notes are generated, so the preview Release is still present and would otherwise be picked as the baseline. If no previous deliverable exists, omit `previous_tag_name` and record that GitHub's default baseline fallback was used.
|
||||
- Generated Release notes carry a workflow-management marker so retries can repair them. Before manually curating a generated body, remove that marker; unmarked non-placeholder notes are preserved by later workflow runs.
|
||||
A green Linux run is ready for field acceptance, not publication. The partial certification must have `publish_allowed: false`. Retain the exact Linux manifest for the next stage; a new source/build-input candidate must not inherit the old approval. A newer unrelated main commit does not invalidate this evidence.
|
||||
|
||||
@@ -16,22 +16,13 @@ Read for Phases 1–2. The [parent release invariants](../SKILL.md) and Console
|
||||
- Only when neither the release branch, an unfinished preview tag, nor current main already has `<target>`: if main is at a later version, select and verify a historical commit containing `<target>` rather than downgrading main; block if none exists. Otherwise invoke the `rustfs-release-version-bump` skill with stage `prepare`, the final `<target>` (NOT a preview version), and full GitHub flow (commit/push/PR).
|
||||
- If a bump PR was needed, get it merged into main and record its merge commit. For a new release branch when main already has `<target>`, identify the main commit that introduced those versions. Phase 2 chooses a release candidate containing that commit; an existing release branch keeps its own history and a newer main HEAD is not required.
|
||||
|
||||
Do not wait for the moving latest main HEAD to become green. The release-branch CI in Phase 2 validates the chosen commit before a preview tag is created.
|
||||
Do not wait for the moving latest main HEAD to become green. The release-branch CI in Phase 2 validates the chosen commit before a candidate build is dispatched.
|
||||
|
||||
## Phase 2 — Pin and validate the release candidate, then publish the preview tag
|
||||
## Phase 2 — Pin and validate the release source
|
||||
|
||||
- Use `release/<target>`. On the first attempt, create it from a selected main commit containing the version bump and intended changes; it need not be the latest main HEAD. On a restart, retain the existing branch and backport only reviewed fixes from main. Verify the branch's Cargo versions still equal `<target>`. If adopting an existing preview, run this branch CI before continuing its acceptance; reuse the tag only when it points to the validated SHA.
|
||||
- The current `ci.yml` runs automatically on main pushes, not release-branch pushes. Dispatch it explicitly with `gh workflow run ci.yml --ref "release/<target>"`. Locate that dispatch run, verify its `head_sha` equals the release branch's remote HEAD, and require the full expected CI job set to pass, including `Quick Checks`, `Test and Lint`, and `End-to-End Tests (full merge gate)`. Treat a missing or skipped required lane as incomplete. A successful PR check or CI run for a different SHA does not qualify. If the branch advances or a run is cancelled, validate the new SHA again. Do not tag a candidate with failing or incomplete CI.
|
||||
- Use `release/<target>`. On the first attempt, create it from a selected main commit containing the version bump and intended changes; it need not be the latest main HEAD. On a restart, retain the existing branch and backport only reviewed fixes from main. Verify the branch's Cargo versions still equal `<target>`. A legacy preview can anchor source selection, but its tag and old artifacts do not constitute a candidate in the new protocol; freeze and verify a new candidate after the release recipe is aligned.
|
||||
- The current `ci.yml` runs automatically on main pushes, not release-branch pushes. Dispatch it explicitly with `gh workflow run ci.yml --ref "release/<target>"`. Locate that dispatch run, verify its `head_sha` equals the release branch's remote HEAD, and require the full expected CI job set to pass, including `Quick Checks`, `Test and Lint`, and `End-to-End Tests (full merge gate)`. Treat a missing or skipped required lane as incomplete. A successful PR check or CI run for a different SHA does not qualify. If the branch advances or a run is cancelled, validate the new SHA again. Do not dispatch a candidate with failing or incomplete CI.
|
||||
- Prefer protecting `release/*` against force pushes and deletion and requiring reviewed backport PRs. Check live rulesets rather than assuming the main ruleset covers release branches; report a missing release-branch rule as a gap.
|
||||
- Set `PREVIEW_HASH` to the CI-validated release-branch commit and recheck the remote branch HEAD immediately before tagging. Report the branch, commit, and CI run URL; do not derive `PREVIEW_HASH` from a later `origin/main` fetch. When adopting an existing preview tag at this SHA, skip tag creation and continue Phase 3.
|
||||
|
||||
```bash
|
||||
git tag -a "<preview-tag>" -m "Release <preview-tag>" "$PREVIEW_HASH"
|
||||
git push origin "<preview-tag>"
|
||||
```
|
||||
|
||||
Pushing the tag triggers `.github/workflows/build.yml` ("Build and Release"); `docker.yml` chains off it via `workflow_run`.
|
||||
|
||||
The preview run builds versioned artifacts and publishes them in a GitHub prerelease. Docker may publish exact preview image tags. Latest-channel, R2, and Helm publication must be skipped; installation references and the website announcement stay unchanged.
|
||||
|
||||
On a restart after a backport, re-run release-branch CI before setting the new `PREVIEW_HASH`. Use the next unused preview iteration when the validated commit differs from an existing preview tag; never move an existing tag.
|
||||
- Set `CANDIDATE_SHA` to the CI-validated release-branch commit. Record the branch, exact commit and CI run URL. The new candidate protocol starts from a commit whose release recipe matches the reviewed main dispatcher; backport the recipe before freezing an older release branch if needed.
|
||||
- Dispatch the Linux candidate as described in [candidate verification](preview-release.md). Do not create a preview or final tag here. The final executable already contains `<target>`; publication later promotes those same bytes.
|
||||
- After a source or build-input change, run the affected source checks and create a new candidate. Reuse the frozen candidate after an infrastructure failure. Do not restart acceptance for unrelated main advances.
|
||||
|
||||
@@ -4,18 +4,18 @@ description: Prepare an exact RustFS Cargo version or align installation referen
|
||||
---
|
||||
# Release Version Files
|
||||
|
||||
Require an exact target; ask if missing or ambiguous. Reject targets containing `-preview`: those are tag-only. Infer `post-release` only from an explicit request or parent workflow; otherwise use `prepare`. Delivery defaults to local edit/verify; commit, push, PR, and merge follow existing authorization and [Git rules](../../references/pull-requests.md).
|
||||
Require an exact target; ask if missing or ambiguous. Reject targets containing `-preview`: those identify candidates, not product versions or new public tags. Infer `post-release` only from an explicit request or parent workflow; otherwise use `prepare`. Delivery defaults to local edit/verify; commit, push, PR, and merge follow existing authorization and [Git rules](../../references/pull-requests.md).
|
||||
|
||||
| Stage | Files and constraints |
|
||||
|---|---|
|
||||
| `prepare` | Only `Cargo.toml` and `Cargo.lock`: workspace package, internal dependency, and workspace member versions become `<target>`. External dependencies stay unchanged. |
|
||||
| `post-release` | Only `README.md`, `README_ZH.md`, `flake.nix`, `helm/rustfs/Chart.yaml`, and `rustfs.spec`. Cargo may already target the next release; leave it untouched. |
|
||||
|
||||
Before installation edits, verify the exact non-preview Release, downloadable assets/source archive, pullable `rustfs/rustfs:<target>` image manifest, and published Helm chart. Block on missing artifacts. Do not downgrade defaults if a newer deliverable superseded the target. Preview preparation retains the previous published installation references; follow-up commits never replace either validated tag.
|
||||
Before installation edits, verify the exact non-preview Release, downloadable assets/source archive, pullable `rustfs/rustfs:<target>` image manifest, and published Helm chart. Block on missing artifacts. Do not downgrade defaults if a newer deliverable superseded the target. Candidate preparation retains the previous published installation references; follow-up commits never replace validated tags or sealed artifacts.
|
||||
|
||||
Packaging policy:
|
||||
|
||||
- Docker image tags have no `v` prefix. Derive chart/app versions with `scripts/helm_chart_version.sh <target>` (`beta.N -> 0.N.0`; other versions unchanged). Helm CI derives versions from the final tag, so no early chart bump is required.
|
||||
- Docker image tags have no `v` prefix. Derive chart/app versions with `scripts/helm_chart_version.sh <target>`; both equal the final product SemVer, including prerelease suffixes. Helm CI uses the frozen candidate version before sealing, so no early installation chart bump is required. Previously published chart versions remain unchanged.
|
||||
- RPM `Version` is numeric; `Release` is the prerelease suffix or `1` for stable. Expanded `Source0` and the unpack directory must resolve to the exact target including its suffix. Add the changelog entry using current git identity, date, and target; never copy an example identity/date.
|
||||
- Changing these mappings requires explicit confirmation.
|
||||
|
||||
|
||||
@@ -22,6 +22,14 @@ TEST_THREADS ?= 1
|
||||
# differ from CI and .config/nextest.toml test-groups will NOT apply.
|
||||
|
||||
.PHONY: script-tests
|
||||
.PHONY: release-package-tests
|
||||
release-package-tests: ## Run release package integration on Linux with fpm, dpkg and rpm installed
|
||||
$(RUSTFS_PYTHON_BIN) ./scripts/test_release_packages.py
|
||||
|
||||
.PHONY: release-image-tests
|
||||
release-image-tests: ## Run OCI integration with Docker, Go and skopeo; set RELEASE_IMAGE_VARIANT and RELEASE_IMAGE_OUTPUT
|
||||
$(RUSTFS_PYTHON_BIN) ./scripts/test_release_images.py
|
||||
|
||||
script-tests: ## Run shell script tests
|
||||
@echo "Running script tests..."
|
||||
trap 'status=$$?; printf "ERROR: script-tests failed (exit %s): %s\n" "$$status" "$$BASH_COMMAND" >&2; exit "$$status"' ERR
|
||||
@@ -57,6 +65,7 @@ script-tests: ## Run shell script tests
|
||||
$(RUSTFS_PYTHON_BIN) ./scripts/test_download_release_asset.py
|
||||
$(RUSTFS_PYTHON_BIN) ./scripts/test_inspect_btf_prototypes.py
|
||||
$(RUSTFS_PYTHON_BIN) ./scripts/test_release_build_workflow.py
|
||||
$(RUSTFS_PYTHON_BIN) -m unittest discover -s scripts/release -p "test_*.py"
|
||||
$(RUSTFS_PYTHON_BIN) ./scripts/test_release_glibc.py
|
||||
$(RUSTFS_PYTHON_BIN) ./scripts/test_upgrade_workflow.py
|
||||
$(RUSTFS_PYTHON_BIN) ./scripts/test_migration_gate_evidence.py
|
||||
|
||||
@@ -1,44 +1,93 @@
|
||||
{
|
||||
"schema_version": 1,
|
||||
"plan_id": "rustfs-release-validation",
|
||||
"description": "Versioned certification plan for every public RustFS release exit (rustfs/backlog#2662, R01). preview, prerelease and stable all require this explicit plan. Scripts/release/release_gate.py fail-closes publication on any missing, duplicate, stale, substituted or non-PASS required item.",
|
||||
"release_types": ["preview", "prerelease", "stable"],
|
||||
"description": "Certification for immutable candidates. All shipped platforms require native identity and startup; stable and prerelease require the pinned upgrade matrix. Linux-only acceptance never authorizes publication.",
|
||||
"release_types": [
|
||||
"preview",
|
||||
"prerelease",
|
||||
"stable"
|
||||
],
|
||||
"targets": [
|
||||
{"os": "linux", "arch": "x86_64", "abi": "musl"},
|
||||
{"os": "linux", "arch": "aarch64", "abi": "musl"},
|
||||
{"os": "linux", "arch": "x86_64", "abi": "gnu"},
|
||||
{"os": "linux", "arch": "aarch64", "abi": "gnu"},
|
||||
{"os": "macos", "arch": "aarch64", "abi": "none"},
|
||||
{"os": "windows", "arch": "x86_64", "abi": "msvc"}
|
||||
{
|
||||
"os": "linux",
|
||||
"arch": "x86_64",
|
||||
"abi": "musl"
|
||||
},
|
||||
{
|
||||
"os": "linux",
|
||||
"arch": "aarch64",
|
||||
"abi": "musl"
|
||||
},
|
||||
{
|
||||
"os": "linux",
|
||||
"arch": "x86_64",
|
||||
"abi": "gnu"
|
||||
},
|
||||
{
|
||||
"os": "linux",
|
||||
"arch": "aarch64",
|
||||
"abi": "gnu"
|
||||
},
|
||||
{
|
||||
"os": "macos",
|
||||
"arch": "aarch64",
|
||||
"abi": "none"
|
||||
},
|
||||
{
|
||||
"os": "windows",
|
||||
"arch": "x86_64",
|
||||
"abi": "msvc"
|
||||
}
|
||||
],
|
||||
"targets_basis": "Platform matrix of .github/workflows/build.yml (prepare-platform-matrix). gnu and macOS targets build with the pyroscope feature; musl and Windows targets build with no extra features.",
|
||||
"features": ["pyroscope"],
|
||||
"trusted_workflows": [
|
||||
{"repository": "rustfs/s3chaos", "workflow_id": "release-gate.yml"}
|
||||
"features": [
|
||||
"pyroscope"
|
||||
],
|
||||
"trusted_workflows_basis": "The shared evidence envelope producer lives in rustfs/s3chaos (G01, fault/release_gate.rs) and its plan/live split workflow is .github/workflows/release-gate.yml there. Only results from these identities may enter a release certification; PR-uploaded JSON never gains release authority. R02 must extend this list when additional verifier workflows land, never widen it to untrusted sources.",
|
||||
"trusted_workflows": [
|
||||
{
|
||||
"repository": "rustfs/rustfs",
|
||||
"workflow_id": "release-candidate.yml"
|
||||
},
|
||||
{
|
||||
"repository": "rustfs/rustfs",
|
||||
"workflow_id": "release-complete.yml"
|
||||
}
|
||||
],
|
||||
"trusted_workflows_basis": "Receipts are produced by the reviewed main workflows and fetched by exact run ID with GitHub artifact digest verification. Candidate inputs freeze the harness commit separately from product source.",
|
||||
"lanes": [
|
||||
{
|
||||
"lane": "identity",
|
||||
"harness_repository": "rustfs/s3chaos",
|
||||
"harness_sha": "9a48f55c1d6d58ce63a8fb1ea7317e024f32f11d",
|
||||
"harness_basis": "Fixed G01 evidence-envelope producer revision (rustfs/s3chaos PR #102 head recorded in rustfs/backlog#2658). R02 re-pins this when the live verifier ships.",
|
||||
"harness_repository": "rustfs/rustfs",
|
||||
"harness_sha": "candidate.workflow_sha",
|
||||
"harness_basis": "The exact workflow commit is frozen in candidate inputs; no mutable branch is accepted as a harness identity.",
|
||||
"timeout_seconds": 1800,
|
||||
"environment_capabilities": ["process-exec"],
|
||||
"cases": ["artifact-identity", "startup"],
|
||||
"environment_capabilities": [
|
||||
"process-exec"
|
||||
],
|
||||
"cases": [
|
||||
"artifact-identity",
|
||||
"startup"
|
||||
],
|
||||
"applies_to": {
|
||||
"release_types": ["preview", "prerelease", "stable"],
|
||||
"release_types": [
|
||||
"preview",
|
||||
"prerelease",
|
||||
"stable"
|
||||
],
|
||||
"platforms": "all"
|
||||
},
|
||||
"applies_basis": "Artifact identity and startup checks are mandatory for every release type, preview included, on every shipped target (rustfs/backlog#2662 frozen design item 5)."
|
||||
},
|
||||
{
|
||||
"lane": "upgrade",
|
||||
"harness_repository": "rustfs/s3chaos",
|
||||
"harness_sha": "9a48f55c1d6d58ce63a8fb1ea7317e024f32f11d",
|
||||
"harness_basis": "Same fixed envelope producer as the identity lane; the workflow SHA and the product build SHA are bound separately and need not be equal.",
|
||||
"harness_repository": "rustfs/rustfs",
|
||||
"harness_sha": "candidate.workflow_sha",
|
||||
"harness_basis": "The exact workflow commit is frozen in candidate inputs; no mutable branch is accepted as a harness identity.",
|
||||
"timeout_seconds": 5400,
|
||||
"environment_capabilities": ["process-exec", "previous-release-artifact"],
|
||||
"environment_capabilities": [
|
||||
"process-exec",
|
||||
"previous-release-artifact"
|
||||
],
|
||||
"cases": [
|
||||
"direct-upgrade",
|
||||
"mixed-version-upgrade",
|
||||
@@ -49,8 +98,17 @@
|
||||
],
|
||||
"cases_basis": "Case IDs of the pinned previous-release matrix in .github/workflows/e2e-upgrade.yml (source.version/asset/sha256 and the case list mapped there).",
|
||||
"applies_to": {
|
||||
"release_types": ["prerelease", "stable"],
|
||||
"platforms": [{"os": "linux", "arch": "x86_64", "abi": "gnu"}]
|
||||
"release_types": [
|
||||
"prerelease",
|
||||
"stable"
|
||||
],
|
||||
"platforms": [
|
||||
{
|
||||
"os": "linux",
|
||||
"arch": "x86_64",
|
||||
"abi": "gnu"
|
||||
}
|
||||
]
|
||||
},
|
||||
"applies_basis": "preview releases use the pre-declared smaller set (identity lane only). The upgrade lane is only exercised on linux-x86_64-gnu, the platform of the pinned source assets."
|
||||
}
|
||||
|
||||
@@ -350,10 +350,10 @@
|
||||
"roles": [
|
||||
"periodic",
|
||||
"manual",
|
||||
"release"
|
||||
"reusable"
|
||||
],
|
||||
"events": [
|
||||
"push",
|
||||
"workflow_call",
|
||||
"schedule",
|
||||
"workflow_dispatch"
|
||||
],
|
||||
@@ -367,7 +367,7 @@
|
||||
"monitor": "workflow",
|
||||
"max_age_hours": 36,
|
||||
"result_owner": "build.yml",
|
||||
"disposition": "retained independent entry",
|
||||
"disposition": "Builds artifacts only; public release writes belong to release-publish.yml.",
|
||||
"failure_owner": "workflow",
|
||||
"origin": "baseline",
|
||||
"alert_contract": "scheduled-validation"
|
||||
@@ -745,14 +745,13 @@
|
||||
},
|
||||
{
|
||||
"workflow": ".github/workflows/docker.yml",
|
||||
"name": "Docker Images",
|
||||
"name": "Build candidate images",
|
||||
"roles": [
|
||||
"manual",
|
||||
"release"
|
||||
"release",
|
||||
"reusable"
|
||||
],
|
||||
"events": [
|
||||
"workflow_run",
|
||||
"workflow_dispatch"
|
||||
"workflow_call"
|
||||
],
|
||||
"cadence": "event",
|
||||
"timezone": "UTC",
|
||||
@@ -761,7 +760,7 @@
|
||||
"policy": "active",
|
||||
"monitor": "none",
|
||||
"result_owner": "docker.yml",
|
||||
"disposition": "retained independent entry",
|
||||
"disposition": "Builds artifacts only; public release writes belong to release-publish.yml.",
|
||||
"monitor_reason": "Event-driven or operator-requested work has no periodic freshness obligation.",
|
||||
"origin": "baseline"
|
||||
},
|
||||
@@ -954,14 +953,13 @@
|
||||
},
|
||||
{
|
||||
"workflow": ".github/workflows/helm-package.yml",
|
||||
"name": "Publish helm chart to artifacthub",
|
||||
"name": "Build candidate Helm chart",
|
||||
"roles": [
|
||||
"manual",
|
||||
"release"
|
||||
"release",
|
||||
"reusable"
|
||||
],
|
||||
"events": [
|
||||
"workflow_run",
|
||||
"workflow_dispatch"
|
||||
"workflow_call"
|
||||
],
|
||||
"cadence": "event",
|
||||
"timezone": "UTC",
|
||||
@@ -970,7 +968,7 @@
|
||||
"policy": "active",
|
||||
"monitor": "none",
|
||||
"result_owner": "helm-package.yml",
|
||||
"disposition": "retained independent entry",
|
||||
"disposition": "Builds artifacts only; public release writes belong to release-publish.yml.",
|
||||
"monitor_reason": "Event-driven or operator-requested work has no periodic freshness obligation.",
|
||||
"origin": "baseline"
|
||||
},
|
||||
@@ -1182,14 +1180,13 @@
|
||||
},
|
||||
{
|
||||
"workflow": ".github/workflows/package.yml",
|
||||
"name": "Package DEB/RPM",
|
||||
"name": "Build candidate packages",
|
||||
"roles": [
|
||||
"manual",
|
||||
"release"
|
||||
"release",
|
||||
"reusable"
|
||||
],
|
||||
"events": [
|
||||
"workflow_run",
|
||||
"workflow_dispatch"
|
||||
"workflow_call"
|
||||
],
|
||||
"cadence": "event",
|
||||
"timezone": "UTC",
|
||||
@@ -1198,7 +1195,7 @@
|
||||
"policy": "active",
|
||||
"monitor": "none",
|
||||
"result_owner": "package.yml",
|
||||
"disposition": "retained independent entry",
|
||||
"disposition": "Builds artifacts only; public release writes belong to release-publish.yml.",
|
||||
"monitor_reason": "Event-driven or operator-requested work has no periodic freshness obligation.",
|
||||
"origin": "baseline"
|
||||
},
|
||||
@@ -1727,6 +1724,69 @@
|
||||
"disposition": "retained independent entry",
|
||||
"monitor_reason": "Event-driven or operator-requested work has no periodic freshness obligation.",
|
||||
"origin": "baseline"
|
||||
},
|
||||
{
|
||||
"workflow": ".github/workflows/release-candidate.yml",
|
||||
"name": "Release candidate (Linux)",
|
||||
"roles": [
|
||||
"manual",
|
||||
"release"
|
||||
],
|
||||
"events": [
|
||||
"workflow_dispatch"
|
||||
],
|
||||
"cadence": "event",
|
||||
"timezone": "UTC",
|
||||
"schedule": [],
|
||||
"owner": "@overtrue",
|
||||
"policy": "active",
|
||||
"monitor": "none",
|
||||
"result_owner": "release-candidate.yml",
|
||||
"disposition": "Explicit immutable candidate stage",
|
||||
"monitor_reason": "Operator-requested release stage has no periodic freshness obligation.",
|
||||
"origin": "release-candidate-migration"
|
||||
},
|
||||
{
|
||||
"workflow": ".github/workflows/release-complete.yml",
|
||||
"name": "Release candidate (complete)",
|
||||
"roles": [
|
||||
"manual",
|
||||
"release"
|
||||
],
|
||||
"events": [
|
||||
"workflow_dispatch"
|
||||
],
|
||||
"cadence": "event",
|
||||
"timezone": "UTC",
|
||||
"schedule": [],
|
||||
"owner": "@overtrue",
|
||||
"policy": "active",
|
||||
"monitor": "none",
|
||||
"result_owner": "release-complete.yml",
|
||||
"disposition": "Explicit immutable candidate stage",
|
||||
"monitor_reason": "Operator-requested release stage has no periodic freshness obligation.",
|
||||
"origin": "release-candidate-migration"
|
||||
},
|
||||
{
|
||||
"workflow": ".github/workflows/release-publish.yml",
|
||||
"name": "Publish accepted release",
|
||||
"roles": [
|
||||
"manual",
|
||||
"release"
|
||||
],
|
||||
"events": [
|
||||
"workflow_dispatch"
|
||||
],
|
||||
"cadence": "event",
|
||||
"timezone": "UTC",
|
||||
"schedule": [],
|
||||
"owner": "@overtrue",
|
||||
"policy": "active",
|
||||
"monitor": "none",
|
||||
"result_owner": "release-publish.yml",
|
||||
"disposition": "Explicit immutable candidate stage",
|
||||
"monitor_reason": "Operator-requested release stage has no periodic freshness obligation.",
|
||||
"origin": "release-candidate-migration"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
@@ -5,7 +5,7 @@
|
||||
|
||||
## Billing facts
|
||||
|
||||
This repository is public. For a public repository, GitHub's standard hosted runners (`ubuntu-latest`, `windows-latest`, `macos-latest`) cost nothing and have no minute cap, whatever plan the organisation is on. GitHub bills only two things: larger hosted runners, which have custom labels and are not used here, and standard runners in private repositories.
|
||||
This repository is public. For a public repository, GitHub's standard hosted runners (`ubuntu-latest`, `windows-latest`, `macos-latest`) cost nothing and have no minute cap, whatever plan the organisation is on. GitHub bills only two things: larger hosted runners, including the `macos-latest-xlarge` release lane used here, and standard runners in private repositories.
|
||||
|
||||
Every other label in this repository is capacity RustFS pays for. ARC pods and lab machines accrue cost while the machine exists. A Ubicloud label is metered for the minutes a job holds its VM:
|
||||
|
||||
@@ -13,6 +13,7 @@ Every other label in this repository is capacity RustFS pays for. ARC pods and l
|
||||
|---|---|---|---|
|
||||
| `ubuntu-latest` | nobody | GitHub-hosted, Ubuntu 24.04, 4 vCPU, 16 GB RAM, about 14 GB of free disk. Ships a C toolchain, `docker`, `gh`, `curl`, `wget`, `jq`, Python. Can `mount` tmpfs and loop devices; io_uring works | the default for every job |
|
||||
| `windows-latest`, `macos-latest` | nobody | GitHub-hosted standard runners | Windows and macOS targets |
|
||||
| `macos-latest-xlarge` | RustFS | GitHub-hosted larger Apple Silicon runner | macOS release builds, deferred until Linux acceptance for candidates |
|
||||
| `sm-standard-2` | RustFS | ARC pod, Ubuntu 24.04, minimal image: no `gh`, no C toolchain; Docker only on the `dind-` label | only a job that a hosted runner cannot run (see the placement rule) |
|
||||
| `sm-standard-4` | RustFS | ARC pod, Ubuntu 24.04, 14 CPUs and 28 GB (the label is not the core count), large disk | workspace-scale Cargo builds |
|
||||
| `ubicloud-standard-16-arm` | RustFS | Ubicloud VM, alias of `ubicloud-standard-16-arm-ubuntu-2404`: Ubuntu 24.04 arm64, 16 vCPU, 48 GB, 300 GB disk. Reached through the `build.yml` matrix, so `actionlint.yaml` does not list it | native aarch64 release builds (`linux-aarch64-musl`, `linux-aarch64-gnu`) |
|
||||
@@ -32,7 +33,7 @@ A job runs on `ubuntu-latest` unless it has one of these reasons, and the reason
|
||||
|
||||
Jobs that do only the following have no reason to be self-hosted: shell or Python scripts, `cargo fmt`, `cargo deny`, typos, actionlint, API calls (`gh`, dependency review, issue and stale bots, CLA checks), downloading an artifact and running a prebuilt binary, or a `cargo build -p rustfs` with its own cache key. `e2e-distributed.yml` and `e2e-upgrade.yml` build the server and the `e2e_test` harness on `ubuntu-latest` inside their budgets, so "it compiles something" alone is not a reason.
|
||||
|
||||
The functional chain's `prepare` and `complete-chain` jobs, the performance runner availability check, and runner hygiene use `ubuntu-latest`: they inspect APIs or artifacts, not the lab hosts. The actual functional and performance suites remain on their lab runners. Functional-chain health collection uses `ubuntu-latest` so a lab outage cannot also stop its independent monitor. Docker image packaging also uses `ubuntu-latest` for both musl and glibc variants; it downloads release binaries rather than compiling the workspace, and still sets up QEMU and Buildx explicitly.
|
||||
The functional chain's `prepare` and `complete-chain` jobs, the performance runner availability check, and runner hygiene use `ubuntu-latest`: they inspect APIs or artifacts, not the lab hosts. The actual functional and performance suites remain on their lab runners. Functional-chain health collection uses `ubuntu-latest` so a lab outage cannot also stop its independent monitor. Docker image packaging also uses `ubuntu-latest` for both musl and glibc variants; it wraps verified candidate binaries rather than compiling the workspace, and still sets up QEMU and Buildx explicitly.
|
||||
|
||||
Moving these jobs off AKS reduces capacity demand, not necessarily the Azure bill immediately. VM costs fall only when nodes scale down or smaller capacity replaces them.
|
||||
|
||||
@@ -52,7 +53,7 @@ A hosted runner is also the only choice for a few things the pods cannot do: `mo
|
||||
These are the patterns that have cost real money here without validating anything more:
|
||||
|
||||
- An `if: always()` summary or alert job on a self-hosted label inside a `workflow_run`-triggered workflow. The upstream workflow completes many times a day, the real jobs skip, and the summary still starts a pod each time. Gate it on the upstream job having run, and keep it on `ubuntu-latest`.
|
||||
- Per-push workflows on `main`. A merge triggers the full `ci.yml` matrix, `cache-warm.yml` and the `build.yml` development build; with tens of merges a day that is as much self-hosted time as all pull requests together. Anything that does not verify the merged tree itself belongs on a schedule or behind a `paths` filter.
|
||||
- Per-push workflows on `main`. A merge triggers the applicable `ci.yml` matrix and cache warming; `build.yml` development builds are scheduled or manual; with tens of merges a day that is as much self-hosted time as all pull requests together. Anything that does not verify the merged tree itself belongs on a schedule or behind a `paths` filter.
|
||||
- `apt-get install` of tools the hosted image already ships. It is only needed on the minimal pod images.
|
||||
- A throttled `CARGO_BUILD_JOBS` on a 14-CPU pod. The limit exists for link-storm reasons documented in `ci.yml`; measure before raising it, and measure before lowering it further.
|
||||
- Tiny jobs on ARC pods. Each one pays roughly a minute of pod start-up before its first step; a compile-free job that takes ten seconds costs more to schedule than to run.
|
||||
@@ -66,7 +67,7 @@ gh api "repos/rustfs/rustfs/actions/runs/<run_id>/jobs?per_page=100" \
|
||||
--jq '.jobs[] | "\(.conclusion)\t\(.labels | join(","))\t\(.started_at)\t\(.completed_at)\t\(.name)"'
|
||||
```
|
||||
|
||||
Only rows whose label starts with `sm-`, `dind-`, `ubicloud-`, `smoke-` or `pf-` cost anything. At the October 2026 review, a full pull-request run of `ci.yml` spent about 150 self-hosted minutes, a third of it in jobs that never compiled; treat that as the scale of what a placement mistake costs, not as a current figure.
|
||||
Rows using `macos-latest-xlarge` or a label starting with `sm-`, `dind-`, `ubicloud-`, `smoke-` or `pf-` use paid capacity. At the October 2026 review, a full pull-request run of `ci.yml` spent about 150 self-hosted minutes, a third of it in jobs that never compiled; treat that as the scale of what a placement mistake costs, not as a current figure.
|
||||
|
||||
## Checklist for a new or moved job
|
||||
|
||||
@@ -76,3 +77,9 @@ Only rows whose label starts with `sm-`, `dind-`, `ubicloud-`, `smoke-` or `pf-`
|
||||
- A self-hosted `runs-on` carries a comment naming its reason from the placement rule.
|
||||
- Every `./.github/actions/setup` call states `cache-save-if`, and a key read from one pool is not written from the other.
|
||||
- Moving a job does not change its check name, so the `main` ruleset and `docs/testing/ci-gates.md` stay as they are; renaming it does, and then that document's change checklist applies.
|
||||
|
||||
## Release stages
|
||||
|
||||
The [candidate release pipeline](../../docs/operations/release-pipeline.md) builds Linux first and defers paid desktop capacity until acceptance. Daily development builds also select Linux; use manual platform selection for desktop development artifacts. All six platforms remain required before publication. The pipeline records and promotes the exact tested bytes. Builder workflows have no public-release credentials. Keep periodic development builds, PR verification and field-test chains separate when their contracts differ; do not re-enable disabled workflows as part of a source-file migration.
|
||||
|
||||
Larger runners remain billed for public repositories; see [GitHub runner billing](https://docs.github.com/en/actions/concepts/runners/larger-runners#billing). Actual spending comes from organization billing, not runner labels or an estimated build duration.
|
||||
|
||||
+149
-8
@@ -23,9 +23,16 @@ on:
|
||||
- 'deny.toml'
|
||||
- '.github/actions/**'
|
||||
- '.github/workflows/**'
|
||||
- 'scripts/release/create_or_update_release.sh'
|
||||
- 'scripts/release/package_service_scripts.sh'
|
||||
- 'scripts/release/package_versions.sh'
|
||||
- 'scripts/release/**'
|
||||
- 'scripts/security/generate_release_supply_chain_assets.py'
|
||||
- '.config/release-validation-plan.json'
|
||||
- '.github/scheduled-validations.json'
|
||||
- 'scripts/test_helm_release_workflow.py'
|
||||
- 'scripts/test_release_packages.py'
|
||||
- 'scripts/test_release_images.py'
|
||||
- 'Dockerfile*'
|
||||
- 'entrypoint.sh'
|
||||
- 'bootstrap/**'
|
||||
- 'scripts/test_package_service_scripts.sh'
|
||||
- 'scripts/test_package_versions.sh'
|
||||
- 'scripts/test_docker_workflow.py'
|
||||
@@ -41,9 +48,16 @@ on:
|
||||
- 'deny.toml'
|
||||
- '.github/actions/**'
|
||||
- '.github/workflows/**'
|
||||
- 'scripts/release/create_or_update_release.sh'
|
||||
- 'scripts/release/package_service_scripts.sh'
|
||||
- 'scripts/release/package_versions.sh'
|
||||
- 'scripts/release/**'
|
||||
- 'scripts/security/generate_release_supply_chain_assets.py'
|
||||
- '.config/release-validation-plan.json'
|
||||
- '.github/scheduled-validations.json'
|
||||
- 'scripts/test_helm_release_workflow.py'
|
||||
- 'scripts/test_release_packages.py'
|
||||
- 'scripts/test_release_images.py'
|
||||
- 'Dockerfile*'
|
||||
- 'entrypoint.sh'
|
||||
- 'bootstrap/**'
|
||||
- 'scripts/test_package_service_scripts.sh'
|
||||
- 'scripts/test_package_versions.sh'
|
||||
- 'scripts/test_docker_workflow.py'
|
||||
@@ -132,12 +146,38 @@ jobs:
|
||||
if: github.repository == 'rustfs/rustfs' && (github.event_name != 'pull_request' || github.event.action != 'closed')
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 5
|
||||
permissions:
|
||||
contents: read
|
||||
actions: read
|
||||
outputs:
|
||||
release_changed: ${{ steps.scope.outputs.release_changed }}
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
|
||||
with:
|
||||
fetch-depth: 0
|
||||
persist-credentials: false
|
||||
|
||||
- name: Scope native packaging checks to release changes
|
||||
id: scope
|
||||
env:
|
||||
BASE_SHA: ${{ github.event.pull_request.base.sha || github.event.before }}
|
||||
run: |
|
||||
python3 - <<'PY'
|
||||
import os, re, subprocess
|
||||
event, base = os.environ['GITHUB_EVENT_NAME'], os.environ.get('BASE_SHA', '')
|
||||
changed = event == 'workflow_dispatch'
|
||||
if event in ('push', 'pull_request'):
|
||||
if not re.fullmatch(r'[0-9a-f]{40}', base) or base == '0' * 40:
|
||||
changed = True
|
||||
else:
|
||||
paths = subprocess.check_output(['git', 'diff', '--name-only', base, os.environ['GITHUB_SHA']], text=True).splitlines()
|
||||
pattern = r'^(scripts/release/|scripts/test_release_|Dockerfile|entrypoint.sh$|bootstrap/|\.dockerignore$|\.config/release-validation-plan.json$|\.github/workflows/(audit|build|docker|package|helm-package|release-[^.]+)\.yml$)'
|
||||
changed = any(re.match(pattern, path) for path in paths)
|
||||
with open(os.environ['GITHUB_OUTPUT'], 'a') as output:
|
||||
output.write(f'release_changed={str(changed).lower()}\n')
|
||||
PY
|
||||
|
||||
- name: Report unpinned GitHub Actions
|
||||
run: ./scripts/security/check_workflow_pins.sh --enforce
|
||||
|
||||
@@ -150,8 +190,35 @@ jobs:
|
||||
- name: Check checkouts clear their credentials
|
||||
run: ./scripts/security/check_persist_credentials.sh
|
||||
|
||||
- name: Check preview release workflow policy
|
||||
run: ./scripts/security/check_preview_release_workflow.sh
|
||||
- name: Check release candidate and promotion contracts
|
||||
run: python3 -m unittest discover -s scripts/release -p 'test_*.py'
|
||||
|
||||
- name: Prepare artifact transport rehearsal
|
||||
run: |
|
||||
mkdir -p "$RUNNER_TEMP/release-transport"
|
||||
printf '{"purpose":"transport contract, not release evidence"}\n' > "$RUNNER_TEMP/release-transport/transport.json"
|
||||
|
||||
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
|
||||
with:
|
||||
name: release-transport-contract-${{ github.run_attempt }}
|
||||
path: ${{ runner.temp }}/release-transport/transport.json
|
||||
if-no-files-found: error
|
||||
retention-days: 1
|
||||
|
||||
- name: Verify actual GitHub artifact metadata and downloaded bytes
|
||||
env:
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
run: |
|
||||
python3 - <<'PY'
|
||||
import os, sys
|
||||
from pathlib import Path
|
||||
sys.path.insert(0, 'scripts/release')
|
||||
import candidate as c
|
||||
run = c.github(f"repos/{c.REPOSITORY}/actions/runs/{os.environ['GITHUB_RUN_ID']}")
|
||||
destination = Path(os.environ['RUNNER_TEMP']) / 'release-transport-downloaded'
|
||||
c.download_artifact(run, f"release-transport-contract-{os.environ['GITHUB_RUN_ATTEMPT']}", destination)
|
||||
c.require(c.read_json(destination / 'transport.json') == {'purpose': 'transport contract, not release evidence'}, 'artifact round trip changed')
|
||||
PY
|
||||
|
||||
- name: Check tier evidence workflow isolation
|
||||
run: ./scripts/security/check_tier_artifact_workflow.sh
|
||||
@@ -162,6 +229,80 @@ jobs:
|
||||
- name: Check package service lifecycle contract
|
||||
run: ./scripts/test_package_service_scripts.sh
|
||||
|
||||
release-package-contract:
|
||||
name: Release package contract
|
||||
needs: workflow-pin-report
|
||||
if: needs.workflow-pin-report.outputs.release_changed == 'true'
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 15
|
||||
steps:
|
||||
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
|
||||
with:
|
||||
persist-credentials: false
|
||||
- name: Install real package managers
|
||||
run: |
|
||||
sudo apt-get update
|
||||
sudo apt-get install -y ruby ruby-dev build-essential rpm fakeroot cpio
|
||||
sudo gem install fpm --version 1.17.0 --no-document
|
||||
- name: Exercise accepted and substituted payloads through both recipes
|
||||
run: python3 scripts/test_release_packages.py
|
||||
|
||||
release-image-contract:
|
||||
name: Release image contract (${{ matrix.variant }})
|
||||
needs: workflow-pin-report
|
||||
if: needs.workflow-pin-report.outputs.release_changed == 'true'
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 35
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
variant: [musl, gnu]
|
||||
env:
|
||||
RELEASE_IMAGE_VARIANT: ${{ matrix.variant }}
|
||||
NO_PROXY: 127.0.0.1,localhost,::1
|
||||
steps:
|
||||
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
|
||||
with:
|
||||
persist-credentials: false
|
||||
- name: Install OCI tools
|
||||
run: |
|
||||
echo "RELEASE_IMAGE_OUTPUT=$RUNNER_TEMP/image-contract" >> "$GITHUB_ENV"
|
||||
echo 'no_proxy=127.0.0.1,localhost,::1' >> "$GITHUB_ENV"
|
||||
sudo apt-get update
|
||||
sudo apt-get install -y skopeo
|
||||
go version
|
||||
- uses: docker/setup-qemu-action@c7c53464625b32c7a7e944ae62b3e17d2b600130 # v3
|
||||
- uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # v3
|
||||
- name: Exercise real OCI packaging, startup, registry retry and conflicts
|
||||
run: python3 scripts/test_release_images.py
|
||||
- name: Exercise amd64 OCI archive scanning
|
||||
uses: aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25
|
||||
env:
|
||||
TRIVY_PLATFORM: linux/amd64
|
||||
with:
|
||||
input: ${{ runner.temp }}/image-contract-scan/image-${{ matrix.variant }}-amd64.tar
|
||||
format: sarif
|
||||
output: ${{ runner.temp }}/image-contract/scan-${{ matrix.variant }}-amd64.sarif
|
||||
exit-code: "0"
|
||||
- name: Exercise arm64 OCI archive scanning
|
||||
uses: aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25
|
||||
env:
|
||||
TRIVY_PLATFORM: linux/arm64
|
||||
with:
|
||||
input: ${{ runner.temp }}/image-contract-scan/image-${{ matrix.variant }}-arm64.tar
|
||||
format: sarif
|
||||
output: ${{ runner.temp }}/image-contract/scan-${{ matrix.variant }}-arm64.sarif
|
||||
exit-code: "0"
|
||||
- name: Verify scan receipts before recording the archive
|
||||
run: |
|
||||
python3 - <<'PY'
|
||||
import os, sys
|
||||
from pathlib import Path
|
||||
sys.path.insert(0, 'scripts/release')
|
||||
import deliver, test_candidate
|
||||
deliver.image_record(test_candidate.inputs(), os.environ['RELEASE_IMAGE_VARIANT'], Path(os.environ['RELEASE_IMAGE_OUTPUT']))
|
||||
PY
|
||||
|
||||
dependency-review:
|
||||
name: Dependency Review
|
||||
runs-on: ubuntu-latest
|
||||
|
||||
+183
-764
File diff suppressed because it is too large
Load Diff
+71
-751
@@ -1,779 +1,99 @@
|
||||
# Copyright 2024 RustFS Team
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
# Docker Images Workflow
|
||||
#
|
||||
# This workflow builds Docker images using pre-built binaries from the build workflow.
|
||||
#
|
||||
# Trigger Types:
|
||||
# 1. workflow_run: Automatically triggered when "Build and Release" workflow completes
|
||||
# 2. workflow_dispatch: Manual trigger for standalone Docker builds
|
||||
#
|
||||
# Key Features:
|
||||
# - Only triggers when Linux builds (x86_64 + aarch64) are successful
|
||||
# - Independent of macOS/Windows build status
|
||||
# - Uses workflow_run event for precise control
|
||||
# - Only builds Docker images for releases and prereleases (development builds are skipped)
|
||||
|
||||
name: Docker Images
|
||||
|
||||
# Permissions needed for workflow_run event and Docker registry access
|
||||
# Copyright 2026 RustFS Team
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
name: Build candidate images
|
||||
on:
|
||||
workflow_call:
|
||||
inputs:
|
||||
candidate_manifest:
|
||||
type: string
|
||||
required: true
|
||||
permissions:
|
||||
contents: read
|
||||
packages: write
|
||||
|
||||
on:
|
||||
# Automatically triggered when build workflow completes
|
||||
workflow_run:
|
||||
workflows: [ "Build and Release" ]
|
||||
types: [ completed ]
|
||||
# Manual trigger with same parameters for consistency
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
push_images:
|
||||
description: "Push images to registries"
|
||||
required: false
|
||||
default: true
|
||||
type: boolean
|
||||
version:
|
||||
description: "Version to build (latest, v1.0.0, v1.0.0-alpha1, or v1.0.1-preview.1)"
|
||||
required: false
|
||||
default: "latest"
|
||||
type: string
|
||||
force_rebuild:
|
||||
description: "Force rebuild even if binary exists (useful for testing)"
|
||||
required: false
|
||||
default: false
|
||||
type: boolean
|
||||
|
||||
env:
|
||||
CONCLUSION: ${{ github.event.workflow_run.conclusion }}
|
||||
HEAD_BRANCH: ${{ github.event.workflow_run.head_branch }}
|
||||
HEAD_SHA: ${{ github.event.workflow_run.head_sha }}
|
||||
TRIGGERING_EVENT: ${{ github.event.workflow_run.event }}
|
||||
DOCKERHUB_USERNAME: rustfs
|
||||
CARGO_TERM_COLOR: always
|
||||
REGISTRY_DOCKERHUB: rustfs/rustfs
|
||||
REGISTRY_GHCR: ghcr.io/${{ github.repository }}
|
||||
REGISTRY_QUAY: quay.io/rustfs/rustfs
|
||||
DOCKER_PLATFORMS: linux/amd64,linux/arm64
|
||||
|
||||
actions: read
|
||||
jobs:
|
||||
# Check if we should build Docker images.
|
||||
# Short-circuit at job level so per-merge development builds (workflow_run
|
||||
# from a push to main) skip the whole run without spawning a checkout job:
|
||||
# images are only published for tag builds — build.yml push triggers cover
|
||||
# only main and tags, so a non-main head_branch is a tag name — and for
|
||||
# manual dispatch.
|
||||
build-check:
|
||||
name: Docker Build Check
|
||||
if: >-
|
||||
github.event_name == 'workflow_dispatch' ||
|
||||
(github.event.workflow_run.conclusion == 'success' &&
|
||||
github.event.workflow_run.event == 'push' &&
|
||||
github.event.workflow_run.head_branch != 'main')
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 30
|
||||
outputs:
|
||||
should_build: ${{ steps.check.outputs.should_build }}
|
||||
should_push: ${{ steps.check.outputs.should_push }}
|
||||
build_type: ${{ steps.check.outputs.build_type }}
|
||||
version: ${{ steps.check.outputs.version }}
|
||||
short_sha: ${{ steps.check.outputs.short_sha }}
|
||||
is_prerelease: ${{ steps.check.outputs.is_prerelease }}
|
||||
create_latest: ${{ steps.check.outputs.create_latest }}
|
||||
source_ref: ${{ steps.check.outputs.source_ref }}
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
|
||||
with:
|
||||
persist-credentials: false
|
||||
# For workflow_run events, checkout the specific commit that triggered the workflow
|
||||
ref: ${{ github.event.workflow_run.head_sha || github.sha }}
|
||||
|
||||
- name: Check build conditions
|
||||
id: check
|
||||
env:
|
||||
# dispatch inputs via env, not `${{ }}` interpolation: they are
|
||||
# free-form strings and would otherwise be evaluated by bash.
|
||||
INPUT_VERSION: ${{ github.event.inputs.version }}
|
||||
INPUT_PUSH_IMAGES: ${{ github.event.inputs.push_images }}
|
||||
INPUT_FORCE_REBUILD: ${{ github.event.inputs.force_rebuild }}
|
||||
run: |
|
||||
should_build=false
|
||||
should_push=false
|
||||
build_type="none"
|
||||
version=""
|
||||
short_sha=""
|
||||
is_prerelease=false
|
||||
create_latest=false
|
||||
source_ref="$GITHUB_SHA"
|
||||
|
||||
# Pre-GA policy: until the first stable (vX.Y.Z) tag exists, every
|
||||
# prerelease (alpha/beta/rc) also moves `latest`, so users pulling
|
||||
# `latest` get the newest test build. Once a stable tag is published
|
||||
# this returns false and `latest` follows stable releases only.
|
||||
prerelease_moves_latest() {
|
||||
local stable_tags
|
||||
stable_tags=$(git ls-remote --tags --refs origin 2>/dev/null \
|
||||
| awk '{print $2}' \
|
||||
| grep -E '^refs/tags/v?[0-9]+\.[0-9]+\.[0-9]+$' || true)
|
||||
if [[ -z "$stable_tags" ]]; then
|
||||
return 0
|
||||
fi
|
||||
echo "ℹ️ Stable release tag(s) already exist; prereleases no longer update latest"
|
||||
return 1
|
||||
}
|
||||
|
||||
if [[ "${{ github.event_name }}" == "workflow_run" ]]; then
|
||||
# Triggered by build workflow completion
|
||||
echo "🔗 Triggered by build workflow completion"
|
||||
|
||||
# Check if the triggering workflow was successful
|
||||
# If the workflow succeeded, it means ALL builds (including Linux x86_64 and aarch64) succeeded
|
||||
if [[ "$CONCLUSION" == "success" ]]; then
|
||||
echo "✅ Build workflow succeeded, all builds including Linux are successful"
|
||||
should_build=true
|
||||
should_push=true
|
||||
else
|
||||
echo "❌ Build workflow failed (conclusion: $CONCLUSION), skipping Docker build"
|
||||
should_build=false
|
||||
fi
|
||||
|
||||
# Extract version info from commit message or use commit SHA
|
||||
# Use Git to generate consistent short SHA (ensures uniqueness like build.yml)
|
||||
short_sha=$(git rev-parse --short "$HEAD_SHA")
|
||||
source_ref="$HEAD_SHA"
|
||||
|
||||
# Determine build type based on triggering workflow event and ref
|
||||
triggering_event="$TRIGGERING_EVENT"
|
||||
head_branch="$HEAD_BRANCH"
|
||||
|
||||
echo "🔍 Analyzing triggering workflow:"
|
||||
echo " 📋 Event: $triggering_event"
|
||||
echo " 🌿 Head branch: $head_branch"
|
||||
echo " 📎 Head SHA: $HEAD_SHA"
|
||||
|
||||
# Check if this was triggered by a tag push
|
||||
if [[ "$triggering_event" == "push" ]]; then
|
||||
# For tag pushes, head_branch will be like "refs/tags/v1.0.0" or just "v1.0.0"
|
||||
if [[ "$head_branch" == refs/tags/* ]]; then
|
||||
# Extract tag name from refs/tags/TAG_NAME
|
||||
tag_name="${head_branch#refs/tags/}"
|
||||
version="$tag_name"
|
||||
elif [[ "$head_branch" =~ ^v?[0-9]+\.[0-9]+\.[0-9]+ ]]; then
|
||||
# Direct tag name like "v1.0.0" or "1.0.0-alpha.1"
|
||||
version="$head_branch"
|
||||
elif [[ "$head_branch" == "main" ]]; then
|
||||
# Regular branch push to main
|
||||
build_type="development"
|
||||
version="dev-${short_sha}"
|
||||
should_build=false
|
||||
echo "⏭️ Skipping Docker build for development version (main branch push)"
|
||||
else
|
||||
# Other branch push
|
||||
build_type="development"
|
||||
version="dev-${short_sha}"
|
||||
should_build=false
|
||||
echo "⏭️ Skipping Docker build for development version (branch: $head_branch)"
|
||||
fi
|
||||
|
||||
# If we extracted a version (tag), determine release type
|
||||
if [[ -n "$version" ]] && [[ "$version" != "dev-${short_sha}" ]]; then
|
||||
# Remove 'v' prefix if present for consistent version format
|
||||
if [[ "$version" == v* ]]; then
|
||||
version="${version#v}"
|
||||
fi
|
||||
|
||||
if [[ "$version" =~ -preview\.[0-9]+$ ]]; then
|
||||
build_type="preview"
|
||||
is_prerelease=true
|
||||
echo "🔍 Building Docker image for preview: $version (version tags only)"
|
||||
elif [[ "$version" == *"-preview"* ]]; then
|
||||
echo "❌ Invalid preview tag: $version (expected suffix: -preview.<number>)" >&2
|
||||
exit 1
|
||||
elif [[ "$version" == *"alpha"* ]] || [[ "$version" == *"beta"* ]] || [[ "$version" == *"rc"* ]]; then
|
||||
build_type="prerelease"
|
||||
is_prerelease=true
|
||||
# Pre-GA policy: prereleases update latest until the first stable tag exists.
|
||||
if prerelease_moves_latest; then
|
||||
create_latest=true
|
||||
echo "🧪 Building Docker image for prerelease: $version (creating latest tag)"
|
||||
else
|
||||
echo "🧪 Building Docker image for prerelease: $version"
|
||||
fi
|
||||
else
|
||||
build_type="release"
|
||||
create_latest=true
|
||||
echo "🚀 Building Docker image for release: $version"
|
||||
fi
|
||||
fi
|
||||
else
|
||||
# Non-push events
|
||||
build_type="development"
|
||||
version="dev-${short_sha}"
|
||||
should_build=false
|
||||
echo "⏭️ Skipping Docker build for development version (event: $triggering_event)"
|
||||
fi
|
||||
|
||||
echo "🔄 Build triggered by workflow_run:"
|
||||
echo " 📋 Conclusion: $CONCLUSION"
|
||||
echo " 🌿 Branch: $HEAD_BRANCH"
|
||||
echo " 📎 SHA: $HEAD_SHA"
|
||||
echo " 🎯 Event: $TRIGGERING_EVENT"
|
||||
|
||||
elif [[ "${{ github.event_name }}" == "workflow_dispatch" ]]; then
|
||||
# Manual trigger
|
||||
input_version="$INPUT_VERSION"
|
||||
version="${input_version}"
|
||||
should_push="$INPUT_PUSH_IMAGES"
|
||||
should_build=true
|
||||
|
||||
# Get short SHA
|
||||
short_sha=$(git rev-parse --short HEAD)
|
||||
|
||||
echo "🎯 Manual Docker build triggered:"
|
||||
echo " 📋 Requested version: $input_version"
|
||||
echo " 🔧 Force rebuild: $INPUT_FORCE_REBUILD"
|
||||
echo " 🚀 Push images: $should_push"
|
||||
|
||||
case "$input_version" in
|
||||
"latest")
|
||||
build_type="release"
|
||||
create_latest=true
|
||||
echo "🚀 Building with latest stable release version"
|
||||
;;
|
||||
*-preview*)
|
||||
if [[ ! "$input_version" =~ ^v?[0-9]+\.[0-9]+\.[0-9]+(-[0-9A-Za-z.-]+)?-preview\.[0-9]+$ ]]; then
|
||||
echo "❌ Invalid preview version: $input_version" >&2
|
||||
exit 1
|
||||
fi
|
||||
build_type="preview"
|
||||
is_prerelease=true
|
||||
version="${input_version#v}"
|
||||
echo "🔍 Building with preview version: $version (version tags only)"
|
||||
;;
|
||||
# Prerelease versions (must match first, more specific)
|
||||
v*alpha*|v*beta*|v*rc*|*alpha*|*beta*|*rc*)
|
||||
build_type="prerelease"
|
||||
is_prerelease=true
|
||||
# Pre-GA policy: prereleases update latest until the first stable tag exists.
|
||||
if prerelease_moves_latest; then
|
||||
create_latest=true
|
||||
echo "🧪 Building with prerelease version: $input_version (creating latest tag)"
|
||||
else
|
||||
echo "🧪 Building with prerelease version: $input_version"
|
||||
fi
|
||||
;;
|
||||
# Release versions (match after prereleases, more general)
|
||||
v[0-9]*|[0-9]*.*.*)
|
||||
build_type="release"
|
||||
create_latest=true
|
||||
echo "📦 Building with specific release version: $input_version"
|
||||
;;
|
||||
*)
|
||||
# Invalid version for Docker build
|
||||
should_build=false
|
||||
echo "❌ Invalid version for Docker build: $input_version"
|
||||
echo "⚠️ Only release versions (latest, v1.0.0, 1.0.0) and prereleases (v1.0.0-alpha1, 1.0.0-beta2) are supported"
|
||||
;;
|
||||
esac
|
||||
|
||||
if [[ "$should_build" == true && "$input_version" != "latest" ]]; then
|
||||
tag_ref="refs/tags/$input_version"
|
||||
if ! git ls-remote --exit-code origin "$tag_ref" >/dev/null 2>&1; then
|
||||
if [[ "$input_version" == v* ]]; then
|
||||
tag_ref="refs/tags/${input_version#v}"
|
||||
else
|
||||
tag_ref="refs/tags/v$input_version"
|
||||
fi
|
||||
fi
|
||||
|
||||
if ! git ls-remote --exit-code origin "$tag_ref" >/dev/null 2>&1; then
|
||||
echo "❌ Release tag not found for Docker build: $input_version"
|
||||
exit 1
|
||||
fi
|
||||
source_ref="$tag_ref"
|
||||
fi
|
||||
fi
|
||||
|
||||
{
|
||||
echo "should_build=$should_build"
|
||||
echo "should_push=$should_push"
|
||||
echo "build_type=$build_type"
|
||||
echo "version=$version"
|
||||
echo "short_sha=$short_sha"
|
||||
echo "is_prerelease=$is_prerelease"
|
||||
echo "create_latest=$create_latest"
|
||||
echo "source_ref=$source_ref"
|
||||
} >> "$GITHUB_OUTPUT"
|
||||
|
||||
echo "🐳 Docker Build Summary:"
|
||||
echo " - Should build: $should_build"
|
||||
echo " - Should push: $should_push"
|
||||
echo " - Build type: $build_type"
|
||||
echo " - Version: $version"
|
||||
echo " - Short SHA: $short_sha"
|
||||
echo " - Is prerelease: $is_prerelease"
|
||||
echo " - Create latest: $create_latest"
|
||||
echo " - Source ref: $source_ref"
|
||||
|
||||
# Consume the trusted release certification for the exact version this
|
||||
# workflow wraps and verify the certified input digests. A manual dispatch
|
||||
# without that certification fails here and pushes no official tag; only
|
||||
# results from the trusted verification sources carry release authority.
|
||||
release-certification:
|
||||
name: Release Certification
|
||||
needs: build-check
|
||||
if: needs.build-check.outputs.should_build == 'true'
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 30
|
||||
permissions:
|
||||
contents: read
|
||||
actions: read
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
|
||||
with:
|
||||
persist-credentials: false
|
||||
ref: ${{ needs.build-check.outputs.source_ref }}
|
||||
|
||||
- name: Download and verify release certification
|
||||
env:
|
||||
GH_TOKEN: ${{ secrets.RELEASE_VALIDATION_TOKEN || github.token }}
|
||||
TRIGGERING_RUN_ID: ${{ github.event.workflow_run.id }}
|
||||
VERSION: ${{ needs.build-check.outputs.version }}
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
mkdir -p certification
|
||||
run_id="$TRIGGERING_RUN_ID"
|
||||
if [[ -z "$run_id" ]]; then
|
||||
run_id="$(gh api --method GET \
|
||||
"repos/${GITHUB_REPOSITORY}/actions/workflows/build.yml/runs" \
|
||||
-f branch="$VERSION" -f status=success -F per_page=1 \
|
||||
--jq '.workflow_runs[0].id // empty' || true)"
|
||||
fi
|
||||
if [[ -z "$run_id" ]]; then
|
||||
echo "No certified Build and Release run found for ${VERSION}; failing closed" >&2
|
||||
exit 1
|
||||
fi
|
||||
if ! gh run download "$run_id" -n release-certification -D certification; then
|
||||
echo "Build run ${run_id} published no release-certification artifact; failing closed" >&2
|
||||
exit 1
|
||||
fi
|
||||
python3 scripts/release/release_gate.py verify-promotion \
|
||||
--plan .config/release-validation-plan.json \
|
||||
--candidate certification/candidate.json \
|
||||
--summary certification/release-gate-output \
|
||||
--artifact-sha256 "$(jq -r '.deliverables[0].sha256' certification/candidate.json)" \
|
||||
--artifact-name "$(jq -r '.deliverables[0].name' certification/candidate.json)"
|
||||
|
||||
# Input digest binding: every zip this image wraps must be the
|
||||
# certified byte-for-byte content, never a mutable latest pointer.
|
||||
while IFS=$'\t' read -r name location expected; do
|
||||
echo "Verifying certified input ${name}"
|
||||
actual="$(curl -fsSL "$location" | sha256sum | awk '{print $1}')"
|
||||
if [[ "$actual" != "$expected" ]]; then
|
||||
echo "Certified input ${name} does not match its recorded digest" >&2
|
||||
exit 1
|
||||
fi
|
||||
done < <(jq -r '.deliverables[] | select(.name | contains("linux")) | [.name, .location, .sha256] | @tsv' certification/candidate.json)
|
||||
|
||||
# Build multi-arch Docker images
|
||||
# Strategy: Build images using pre-built binaries from dl.rustfs.com
|
||||
# Supports both release and dev channel binaries based on build context
|
||||
# Only runs when should_build is true (which includes workflow success check)
|
||||
build-docker:
|
||||
name: Build Docker Images
|
||||
needs: [ build-check, release-certification ]
|
||||
if: needs.build-check.outputs.should_build == 'true'
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 60
|
||||
timeout-minutes: 90
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
include:
|
||||
- variant: musl
|
||||
file: Dockerfile
|
||||
suffix: ""
|
||||
- variant: glibc
|
||||
- variant: gnu
|
||||
file: Dockerfile.glibc
|
||||
suffix: "-glibc"
|
||||
|
||||
env:
|
||||
CANDIDATE_MANIFEST: ${{ inputs.candidate_manifest }}
|
||||
VARIANT: ${{ matrix.variant }}
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
|
||||
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
|
||||
with:
|
||||
ref: ${{ fromJson(inputs.candidate_manifest).source_sha }}
|
||||
persist-credentials: false
|
||||
ref: ${{ needs.build-check.outputs.source_ref }}
|
||||
|
||||
- name: Login to Docker Hub
|
||||
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
|
||||
with:
|
||||
username: ${{ env.DOCKERHUB_USERNAME }}
|
||||
password: ${{ secrets.DOCKERHUB_TOKEN }}
|
||||
|
||||
- name: Login to GitHub Container Registry
|
||||
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
|
||||
with:
|
||||
registry: ghcr.io
|
||||
username: ${{ secrets.GHCR_USERNAME }}
|
||||
password: ${{ secrets.GHCR_PASSWORD }}
|
||||
|
||||
- name: Login to Quay.io
|
||||
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
|
||||
with:
|
||||
registry: quay.io
|
||||
username: ${{ secrets.QUAY_USERNAME }}
|
||||
password: ${{ secrets.QUAY_PASSWORD }}
|
||||
|
||||
- name: Set up QEMU
|
||||
uses: docker/setup-qemu-action@c7c53464625b32c7a7e944ae62b3e17d2b600130 # v3
|
||||
|
||||
- name: Set up Docker Buildx
|
||||
uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # v3
|
||||
|
||||
- name: Extract metadata and generate tags
|
||||
id: meta
|
||||
- name: Reuse completed image
|
||||
id: reuse
|
||||
env:
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
run: python3 scripts/release/deliver.py reuse --kind image --target "$VARIANT"
|
||||
- name: Install OCI tools
|
||||
if: steps.reuse.outputs.reused != 'true'
|
||||
run: |
|
||||
BUILD_TYPE="${{ needs.build-check.outputs.build_type }}"
|
||||
VERSION="${{ needs.build-check.outputs.version }}"
|
||||
CREATE_LATEST="${{ needs.build-check.outputs.create_latest }}"
|
||||
VARIANT_SUFFIX="${{ matrix.suffix }}"
|
||||
|
||||
# Convert version format for Dockerfile compatibility. The former
|
||||
# DOCKER_CHANNEL was "release" down every branch and was passed as a
|
||||
# build-arg no Dockerfile declares, so it is gone.
|
||||
case "$VERSION" in
|
||||
"latest")
|
||||
DOCKER_RELEASE="latest"
|
||||
;;
|
||||
v*)
|
||||
# For versioned releases (v1.0.0), remove 'v' prefix for Dockerfile
|
||||
DOCKER_RELEASE="${VERSION#v}"
|
||||
;;
|
||||
*)
|
||||
# For other versions, pass as-is
|
||||
DOCKER_RELEASE="${VERSION}"
|
||||
;;
|
||||
esac
|
||||
|
||||
echo "docker_release=$DOCKER_RELEASE" >> "$GITHUB_OUTPUT"
|
||||
|
||||
echo "🐳 Docker build parameters:"
|
||||
echo " - Original version: $VERSION"
|
||||
echo " - Docker RELEASE: $DOCKER_RELEASE"
|
||||
|
||||
# Generate tags based on build type
|
||||
# Only support release and prerelease builds (no development builds)
|
||||
TAG_BASE="${VERSION}${VARIANT_SUFFIX}"
|
||||
TAGS="${{ env.REGISTRY_DOCKERHUB }}:$TAG_BASE,${{ env.REGISTRY_GHCR }}:$TAG_BASE,${{ env.REGISTRY_QUAY }}:$TAG_BASE"
|
||||
|
||||
# Add latest when requested (stable releases, and prereleases before GA)
|
||||
if [[ "$CREATE_LATEST" == "true" ]]; then
|
||||
TAGS="$TAGS,${{ env.REGISTRY_DOCKERHUB }}:latest${VARIANT_SUFFIX},${{ env.REGISTRY_GHCR }}:latest${VARIANT_SUFFIX},${{ env.REGISTRY_QUAY }}:latest${VARIANT_SUFFIX}"
|
||||
fi
|
||||
|
||||
# Always add the channel tag for prereleases, independent of latest
|
||||
if [[ "$BUILD_TYPE" == "prerelease" ]]; then
|
||||
# Prerelease channel tags (alpha, beta, rc)
|
||||
if [[ "$VERSION" == *"alpha"* ]]; then
|
||||
CHANNEL="alpha"
|
||||
elif [[ "$VERSION" == *"beta"* ]]; then
|
||||
CHANNEL="beta"
|
||||
elif [[ "$VERSION" == *"rc"* ]]; then
|
||||
CHANNEL="rc"
|
||||
fi
|
||||
|
||||
if [[ -n "$CHANNEL" ]]; then
|
||||
TAGS="$TAGS,${{ env.REGISTRY_DOCKERHUB }}:${CHANNEL}${VARIANT_SUFFIX},${{ env.REGISTRY_GHCR }}:${CHANNEL}${VARIANT_SUFFIX},${{ env.REGISTRY_QUAY }}:${CHANNEL}${VARIANT_SUFFIX}"
|
||||
fi
|
||||
fi
|
||||
|
||||
# Output tags
|
||||
echo "tags=$TAGS" >> "$GITHUB_OUTPUT"
|
||||
|
||||
# Staging identity for this run's promotion: official tags are only
|
||||
# created after the staged image verifies against the certification,
|
||||
# and they point at this same digest (no rebuild-after-verify).
|
||||
STAGING_TAG="staging-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}${VARIANT_SUFFIX}"
|
||||
STAGING_TAGS="${{ env.REGISTRY_DOCKERHUB }}:${STAGING_TAG},${{ env.REGISTRY_GHCR }}:${STAGING_TAG},${{ env.REGISTRY_QUAY }}:${STAGING_TAG}"
|
||||
echo "staging_tags=$STAGING_TAGS" >> "$GITHUB_OUTPUT"
|
||||
echo "staging_tag=$STAGING_TAG" >> "$GITHUB_OUTPUT"
|
||||
|
||||
# Generate labels
|
||||
LABELS="org.opencontainers.image.title=RustFS"
|
||||
LABELS="$LABELS,org.opencontainers.image.description=RustFS distributed object storage system"
|
||||
LABELS="$LABELS,org.opencontainers.image.version=$VERSION"
|
||||
SOURCE_REVISION="$(git rev-parse HEAD)"
|
||||
LABELS="$LABELS,org.opencontainers.image.revision=$SOURCE_REVISION"
|
||||
LABELS="$LABELS,org.opencontainers.image.source=${{ github.server_url }}/${{ github.repository }}"
|
||||
LABELS="$LABELS,org.opencontainers.image.created=$(date -u +'%Y-%m-%dT%H:%M:%SZ')"
|
||||
LABELS="$LABELS,org.opencontainers.image.build-type=$BUILD_TYPE"
|
||||
|
||||
echo "labels=$LABELS" >> "$GITHUB_OUTPUT"
|
||||
|
||||
echo "🐳 Generated Docker tags:"
|
||||
echo "$TAGS" | tr ',' '\n' | sed 's/^/ - /'
|
||||
echo "📋 Build type: $BUILD_TYPE"
|
||||
echo "🔖 Version: $VERSION"
|
||||
|
||||
- name: Build and push Docker image
|
||||
sudo apt-get update
|
||||
sudo apt-get install -y skopeo
|
||||
- uses: docker/setup-qemu-action@c7c53464625b32c7a7e944ae62b3e17d2b600130 # v3
|
||||
if: steps.reuse.outputs.reused != 'true'
|
||||
- uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # v3
|
||||
if: steps.reuse.outputs.reused != 'true'
|
||||
- name: Supply accepted binaries and pinned CLI
|
||||
if: steps.reuse.outputs.reused != 'true'
|
||||
env:
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
run: python3 scripts/release/deliver.py image-prepare --target "$VARIANT"
|
||||
- name: Build OCI archive without registry writes
|
||||
if: steps.reuse.outputs.reused != 'true'
|
||||
uses: docker/build-push-action@10e90e3645eae34f1e60eeb005ba3a3d33f178e8 # v6
|
||||
with:
|
||||
context: .
|
||||
file: ${{ matrix.file }}
|
||||
platforms: ${{ env.DOCKER_PLATFORMS }}
|
||||
push: ${{ needs.build-check.outputs.should_push == 'true' }}
|
||||
# Push to staging tags only: official tags are created by the
|
||||
# certification-gated promotion step below from this same digest.
|
||||
tags: ${{ steps.meta.outputs.staging_tags }}
|
||||
labels: ${{ steps.meta.outputs.labels }}
|
||||
# No layer cache. This build compiles nothing — it downloads a
|
||||
# release zip and runs apk/apt — so the cache could only save the
|
||||
# minute or two those take, while creating a correctness problem: with
|
||||
# RELEASE=latest the binary URL is resolved by curl *inside* a RUN
|
||||
# layer, and the layer key does not include what that resolved to. A
|
||||
# rebuild at the same RELEASE value (dispatch with version=latest, or
|
||||
# a re-run of the same version) would hit the old layer and ship the
|
||||
# previous release's binary. mode=max also consumed the same 10GB
|
||||
# Actions cache quota the Rust lanes are fighting over.
|
||||
#
|
||||
# Only RELEASE is passed: it is the sole build-arg the Dockerfiles
|
||||
# declare besides TARGETARCH. BUILDTIME, VERSION, BUILD_TYPE, REVISION
|
||||
# and CHANNEL were never read by any stage (and BUILDTIME's $(date ...)
|
||||
# was a literal here, not a shell substitution). BUILD_DATE and VCS_REF
|
||||
# are declared by the Dockerfiles but deliberately left unset —
|
||||
# supplying them would change the published image labels.
|
||||
platforms: linux/amd64,linux/arm64
|
||||
push: false
|
||||
provenance: false
|
||||
outputs: type=oci,dest=delivery/image-${{ matrix.variant }}.tar
|
||||
build-args: |
|
||||
RELEASE=${{ steps.meta.outputs.docker_release }}
|
||||
provenance: true
|
||||
sbom: true
|
||||
# Add retry mechanism by splitting the build process
|
||||
no-cache: false
|
||||
pull: true
|
||||
|
||||
# A later-built wrapper: the staged image must complete this platform's
|
||||
# own binary identity verification before any official tag is written.
|
||||
# It may not inherit an unverified package's conclusion.
|
||||
- name: Verify staged image against certification
|
||||
if: needs.build-check.outputs.should_push == 'true'
|
||||
env:
|
||||
GH_TOKEN: ${{ secrets.RELEASE_VALIDATION_TOKEN || github.token }}
|
||||
TRIGGERING_RUN_ID: ${{ github.event.workflow_run.id }}
|
||||
VERSION: ${{ needs.build-check.outputs.version }}
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
mkdir -p certification
|
||||
run_id="$TRIGGERING_RUN_ID"
|
||||
if [[ -z "$run_id" ]]; then
|
||||
run_id="$(gh api --method GET \
|
||||
"repos/${GITHUB_REPOSITORY}/actions/workflows/build.yml/runs" \
|
||||
-f branch="$VERSION" -f status=success -F per_page=1 \
|
||||
--jq '.workflow_runs[0].id // empty' || true)"
|
||||
fi
|
||||
gh run download "$run_id" -n release-certification -D certification
|
||||
|
||||
STAGING_REF="${{ env.REGISTRY_DOCKERHUB }}:staging-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}${{ matrix.suffix }}"
|
||||
case "${{ matrix.variant }}" in
|
||||
musl) libc="musl" ;;
|
||||
glibc) libc="gnu" ;;
|
||||
esac
|
||||
for platform in linux/amd64 linux/arm64; do
|
||||
case "$platform" in
|
||||
linux/amd64) arch_sub="x86_64" ;;
|
||||
linux/arm64) arch_sub="aarch64" ;;
|
||||
esac
|
||||
docker pull --platform "$platform" "$STAGING_REF"
|
||||
container="$(docker create --platform "$platform" "$STAGING_REF")"
|
||||
docker cp "$container:/usr/bin/rustfs" "$RUNNER_TEMP/staged-rustfs"
|
||||
docker rm "$container" >/dev/null
|
||||
actual="$(sha256sum "$RUNNER_TEMP/staged-rustfs" | awk '{print $1}')"
|
||||
expected="$(jq -r --arg needle "linux-${arch_sub}-${libc}" \
|
||||
'[.deliverables[] | select(.name | contains($needle)) | .binary_sha256][0] // empty' \
|
||||
certification/candidate.json)"
|
||||
if [[ -z "$expected" || "$actual" != "$expected" ]]; then
|
||||
echo "Staged image binary for ${platform} does not match the certified binary digest" >&2
|
||||
exit 1
|
||||
fi
|
||||
python3 scripts/release/release_gate.py verify-promotion \
|
||||
--plan .config/release-validation-plan.json \
|
||||
--candidate certification/candidate.json \
|
||||
--summary certification/release-gate-output \
|
||||
--artifact "$RUNNER_TEMP/staged-rustfs" \
|
||||
--artifact-role binary
|
||||
done
|
||||
|
||||
- name: Promote staged image to official tags
|
||||
if: needs.build-check.outputs.should_push == 'true'
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
STAGING_REF="${{ env.REGISTRY_DOCKERHUB }}:staging-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}${{ matrix.suffix }}"
|
||||
IFS=',' read -r -a official_tags <<< "${{ steps.meta.outputs.tags }}"
|
||||
for tag in "${official_tags[@]}"; do
|
||||
docker buildx imagetools create -t "$tag" "$STAGING_REF"
|
||||
echo "Promoted $tag -> $STAGING_REF"
|
||||
done
|
||||
|
||||
# Downstream consumers (Helm) verify this record instead of inheriting
|
||||
# an unverified wrapper's conclusion.
|
||||
- name: Record image promotion
|
||||
if: needs.build-check.outputs.should_push == 'true'
|
||||
shell: bash
|
||||
env:
|
||||
GH_TOKEN: ${{ secrets.RELEASE_VALIDATION_TOKEN || github.token }}
|
||||
TRIGGERING_RUN_ID: ${{ github.event.workflow_run.id }}
|
||||
VERSION: ${{ needs.build-check.outputs.version }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
mkdir -p certification
|
||||
run_id="$TRIGGERING_RUN_ID"
|
||||
if [[ -z "$run_id" ]]; then
|
||||
run_id="$(gh api --method GET \
|
||||
"repos/${GITHUB_REPOSITORY}/actions/workflows/build.yml/runs" \
|
||||
-f branch="$VERSION" -f status=success -F per_page=1 \
|
||||
--jq '.workflow_runs[0].id // empty' || true)"
|
||||
fi
|
||||
gh run download "$run_id" -n release-certification -D certification
|
||||
STAGING_REF="${{ env.REGISTRY_DOCKERHUB }}:staging-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}${{ matrix.suffix }}"
|
||||
digest="$(docker buildx imagetools inspect "$STAGING_REF" --format '{{json .Manifest}}' | jq -r '.digest')"
|
||||
candidate_sha="$(python3 -c "import json,hashlib,sys; sys.path.insert(0,'scripts/release'); import release_gate; print(release_gate.canonical_sha256(json.load(open('certification/candidate.json'))))")"
|
||||
jq -n \
|
||||
--arg version "$VERSION" \
|
||||
--arg digest "$digest" \
|
||||
--arg staging_ref "$STAGING_REF" \
|
||||
--arg candidate_sha256 "$candidate_sha" \
|
||||
--arg tags "${{ steps.meta.outputs.tags }}" \
|
||||
--slurpfile candidate certification/candidate.json \
|
||||
'{schema_version: 1, version: $version, image_digest: $digest, staging_ref: $staging_ref,
|
||||
promoted_tags: ($tags | split(",")), candidate_sha256: $candidate_sha256,
|
||||
binary_sha256s: ($candidate[0].deliverables | map(.binary_sha256) | unique)}' \
|
||||
> image-promotion-record.json
|
||||
|
||||
- name: Upload image promotion record
|
||||
if: needs.build-check.outputs.should_push == 'true'
|
||||
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
|
||||
with:
|
||||
name: image-promotion-record${{ matrix.suffix }}
|
||||
path: image-promotion-record.json
|
||||
if-no-files-found: error
|
||||
retention-days: 30
|
||||
|
||||
# Note: Manifest creation is no longer needed as we only build one variant
|
||||
# Multi-arch manifests are automatically created by docker/build-push-action
|
||||
|
||||
# Report-only container image vulnerability scan
|
||||
scan-docker-image:
|
||||
name: Scan Docker Images
|
||||
needs: [ build-check, build-docker ]
|
||||
if: needs.build-check.outputs.should_build == 'true' && needs.build-check.outputs.should_push == 'true'
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 30
|
||||
permissions:
|
||||
contents: read
|
||||
security-events: write
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
include:
|
||||
- variant: musl
|
||||
suffix: ""
|
||||
- variant: glibc
|
||||
suffix: "-glibc"
|
||||
steps:
|
||||
- name: Login to GitHub Container Registry
|
||||
# docker/login-action v3
|
||||
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9
|
||||
with:
|
||||
registry: ghcr.io
|
||||
username: ${{ secrets.GHCR_USERNAME }}
|
||||
password: ${{ secrets.GHCR_PASSWORD }}
|
||||
|
||||
- name: Scan image with Trivy
|
||||
# aquasecurity/trivy-action v0.36.0
|
||||
BINARY_SOURCE=supplied
|
||||
RELEASE=${{ fromJson(inputs.candidate_manifest).version }}
|
||||
VCS_REF=${{ fromJson(inputs.candidate_manifest).source_sha }}
|
||||
- name: Verify wrapped binaries and startup on both architectures
|
||||
if: steps.reuse.outputs.reused != 'true'
|
||||
run: python3 scripts/release/deliver.py image-smoke --target "$VARIANT"
|
||||
- name: Scan amd64 image before sealing
|
||||
if: steps.reuse.outputs.reused != 'true'
|
||||
uses: aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25
|
||||
env:
|
||||
TRIVY_PLATFORM: linux/amd64
|
||||
with:
|
||||
image-ref: ${{ env.REGISTRY_GHCR }}:${{ needs.build-check.outputs.version }}${{ matrix.suffix }}
|
||||
input: delivery-scan/image-${{ matrix.variant }}-amd64.tar
|
||||
format: sarif
|
||||
output: trivy-${{ matrix.variant }}.sarif
|
||||
ignore-unfixed: true
|
||||
vuln-type: os,library
|
||||
severity: CRITICAL,HIGH
|
||||
output: delivery/scan-${{ matrix.variant }}-amd64.sarif
|
||||
exit-code: "0" # Keep the repository's report-only vulnerability policy.
|
||||
- name: Scan arm64 image before sealing
|
||||
if: steps.reuse.outputs.reused != 'true'
|
||||
uses: aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25
|
||||
env:
|
||||
TRIVY_PLATFORM: linux/arm64
|
||||
with:
|
||||
input: delivery-scan/image-${{ matrix.variant }}-arm64.tar
|
||||
format: sarif
|
||||
output: delivery/scan-${{ matrix.variant }}-arm64.sarif
|
||||
exit-code: "0"
|
||||
|
||||
# Surface findings in the Security tab; an artifact alone is write-only
|
||||
# reporting nobody reviews.
|
||||
- name: Upload scan results to code scanning
|
||||
uses: github/codeql-action/upload-sarif@99df26d4f13ea111d4ec1a7dddef6063f76b97e9 # v4.37.0
|
||||
- name: Record verified OCI identity
|
||||
if: steps.reuse.outputs.reused != 'true'
|
||||
run: python3 scripts/release/deliver.py image-record --target "$VARIANT"
|
||||
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
|
||||
if: steps.reuse.outputs.reused != 'true'
|
||||
with:
|
||||
sarif_file: trivy-${{ matrix.variant }}.sarif
|
||||
category: container-image-${{ matrix.variant }}
|
||||
|
||||
- name: Upload container scan report
|
||||
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
|
||||
with:
|
||||
name: container-image-scan-${{ matrix.variant }}
|
||||
path: trivy-${{ matrix.variant }}.sarif
|
||||
name: image-${{ matrix.variant }}
|
||||
path: delivery/*
|
||||
if-no-files-found: error
|
||||
retention-days: 30
|
||||
|
||||
# Docker build summary
|
||||
docker-summary:
|
||||
name: Docker Build Summary
|
||||
needs: [ build-check, build-docker ]
|
||||
if: always() && needs.build-check.outputs.should_build == 'true'
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 10
|
||||
steps:
|
||||
- name: Docker build completion summary
|
||||
run: |
|
||||
BUILD_TYPE="${{ needs.build-check.outputs.build_type }}"
|
||||
VERSION="${{ needs.build-check.outputs.version }}"
|
||||
CREATE_LATEST="${{ needs.build-check.outputs.create_latest }}"
|
||||
|
||||
echo "🐳 Docker build completed successfully!"
|
||||
echo "📦 Build type: $BUILD_TYPE"
|
||||
echo "🔢 Version: $VERSION"
|
||||
echo "🚀 Strategy: Images using pre-built binaries (release channel only)"
|
||||
echo ""
|
||||
|
||||
case "$BUILD_TYPE" in
|
||||
"preview")
|
||||
echo "🔍 Preview Docker image has been built with ${VERSION} tags"
|
||||
echo "⏭️ Preview images do not update latest or prerelease channels"
|
||||
;;
|
||||
"release")
|
||||
echo "🚀 Release Docker image has been built with ${VERSION} tags"
|
||||
echo "✅ This image is ready for production use"
|
||||
if [[ "$CREATE_LATEST" == "true" ]]; then
|
||||
echo "🏷️ Latest tag has been created for stable release"
|
||||
fi
|
||||
;;
|
||||
"prerelease")
|
||||
echo "🧪 Prerelease Docker image has been built with ${VERSION} tags"
|
||||
echo "⚠️ This is a prerelease image - use with caution"
|
||||
# Prereleases move latest until the first stable tag exists (pre-GA policy).
|
||||
if [[ "$CREATE_LATEST" == "true" ]]; then
|
||||
echo "🏷️ Latest tag has been created for prerelease: $VERSION"
|
||||
else
|
||||
echo "🚫 Latest tag NOT created for prerelease"
|
||||
fi
|
||||
;;
|
||||
*)
|
||||
echo "❌ Unexpected build type: $BUILD_TYPE"
|
||||
;;
|
||||
esac
|
||||
compression-level: 0
|
||||
|
||||
@@ -1,229 +1,26 @@
|
||||
# Copyright 2024 RustFS Team
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
name: Publish helm chart to artifacthub
|
||||
|
||||
# Copyright 2026 RustFS Team
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
name: Build candidate Helm chart
|
||||
on:
|
||||
workflow_run:
|
||||
workflows: [ "Build and Release" ]
|
||||
types: [ completed ]
|
||||
workflow_dispatch:
|
||||
workflow_call:
|
||||
inputs:
|
||||
version:
|
||||
description: "Release version to publish, e.g. 1.0.0-beta.1 or v1.0.0-beta.1"
|
||||
required: true
|
||||
default: "1.0.0-beta.1"
|
||||
candidate_manifest:
|
||||
type: string
|
||||
|
||||
required: true
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
actions: read
|
||||
jobs:
|
||||
# Consume the same trusted release certification as every other publication
|
||||
# exit and verify the image promotion record of the Docker workflow. The
|
||||
# chart wraps no binary of its own, so it may only reference an image whose
|
||||
# platform verification already passed; it must not inherit an unverified
|
||||
# package's conclusion. A run without that evidence fails closed here and
|
||||
# never reaches the chart repository.
|
||||
release-certification:
|
||||
package:
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 30
|
||||
permissions:
|
||||
contents: read
|
||||
actions: read
|
||||
outputs:
|
||||
version: ${{ steps.version.outputs.raw_tag }}
|
||||
timeout-minutes: 15
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
|
||||
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
|
||||
with:
|
||||
ref: ${{ fromJson(inputs.candidate_manifest).source_sha }}
|
||||
persist-credentials: false
|
||||
|
||||
- name: Normalize release version
|
||||
id: version
|
||||
env:
|
||||
RAW_INPUT: ${{ github.event.inputs.version }}
|
||||
RAW_BRANCH: ${{ github.event.workflow_run.head_branch }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then
|
||||
RAW="$RAW_INPUT"
|
||||
else
|
||||
RAW="$RAW_BRANCH"
|
||||
fi
|
||||
case "$RAW" in
|
||||
refs/tags/*) RAW_TAG="${RAW#refs/tags/}" ;;
|
||||
*) RAW_TAG="$RAW" ;;
|
||||
esac
|
||||
echo "raw_tag=$RAW_TAG" >> "$GITHUB_OUTPUT"
|
||||
|
||||
- name: Download and verify release certification and image record
|
||||
env:
|
||||
GH_TOKEN: ${{ secrets.RELEASE_VALIDATION_TOKEN || github.token }}
|
||||
TRIGGERING_RUN_ID: ${{ github.event.workflow_run.id }}
|
||||
RAW_TAG: ${{ steps.version.outputs.raw_tag }}
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
mkdir -p certification image-record
|
||||
|
||||
build_run="$TRIGGERING_RUN_ID"
|
||||
if [[ -z "$build_run" ]]; then
|
||||
build_run="$(gh api --method GET \
|
||||
"repos/${GITHUB_REPOSITORY}/actions/workflows/build.yml/runs" \
|
||||
-f branch="$RAW_TAG" -f status=success -F per_page=1 \
|
||||
--jq '.workflow_runs[0].id // empty' || true)"
|
||||
fi
|
||||
if [[ -z "$build_run" ]]; then
|
||||
echo "No certified Build and Release run found for ${RAW_TAG}; failing closed" >&2
|
||||
exit 1
|
||||
fi
|
||||
if ! gh run download "$build_run" -n release-certification -D certification; then
|
||||
echo "Build run ${build_run} published no release-certification artifact; failing closed" >&2
|
||||
exit 1
|
||||
fi
|
||||
python3 scripts/release/release_gate.py verify-promotion \
|
||||
--plan .config/release-validation-plan.json \
|
||||
--candidate certification/candidate.json \
|
||||
--summary certification/release-gate-output \
|
||||
--artifact-sha256 "$(jq -r '.deliverables[0].sha256' certification/candidate.json)" \
|
||||
--artifact-name "$(jq -r '.deliverables[0].name' certification/candidate.json)"
|
||||
|
||||
docker_run="$(gh api --method GET \
|
||||
"repos/${GITHUB_REPOSITORY}/actions/workflows/docker.yml/runs" \
|
||||
-f branch="$RAW_TAG" -f status=success -F per_page=1 \
|
||||
--jq '.workflow_runs[0].id // empty' || true)"
|
||||
if [[ -z "$docker_run" ]] || ! gh run download "$docker_run" -n image-promotion-record -D image-record; then
|
||||
echo "No image promotion record for ${RAW_TAG}; a chart may not wrap an unverified image" >&2
|
||||
exit 1
|
||||
fi
|
||||
candidate_sha="$(python3 -c "import json,hashlib,sys; sys.path.insert(0,'scripts/release'); import release_gate; print(release_gate.canonical_sha256(json.load(open('certification/candidate.json'))))")"
|
||||
record_sha="$(jq -r '.candidate_sha256 // empty' image-record/image-promotion-record.json)"
|
||||
if [[ "$record_sha" != "$candidate_sha" ]]; then
|
||||
echo "Image promotion record does not belong to this certification" >&2
|
||||
exit 1
|
||||
fi
|
||||
overlap="$(jq -r --slurpfile candidate certification/candidate.json \
|
||||
'[.binary_sha256s[]] - ($candidate[0].deliverables | map(.binary_sha256)) | length' \
|
||||
image-record/image-promotion-record.json)"
|
||||
if [[ "$overlap" -ne 0 ]]; then
|
||||
echo "Image promotion record carries binary digests outside the certification" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
build-helm-package:
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 30
|
||||
needs: [ release-certification ]
|
||||
if: |
|
||||
(github.event_name == 'workflow_dispatch' && !contains(github.event.inputs.version, '-preview')) ||
|
||||
(
|
||||
github.event.workflow_run.conclusion == 'success' &&
|
||||
github.event.workflow_run.event == 'push' &&
|
||||
contains(github.event.workflow_run.head_branch, '.') &&
|
||||
!contains(github.event.workflow_run.head_branch, '-preview')
|
||||
)
|
||||
|
||||
outputs:
|
||||
raw_tag: ${{ steps.version.outputs.raw_tag }}
|
||||
app_version: ${{ steps.version.outputs.app_version }}
|
||||
chart_version: ${{ steps.version.outputs.chart_version }}
|
||||
|
||||
steps:
|
||||
- name: Checkout release source
|
||||
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
|
||||
with:
|
||||
ref: ${{ github.event.workflow_run.head_sha || (startsWith(inputs.version, 'refs/tags/') && inputs.version || format('refs/tags/{0}', inputs.version)) }}
|
||||
persist-credentials: false
|
||||
|
||||
# Both inputs reach the shell through env rather than `${{ }}`
|
||||
# interpolation. A git ref name may contain `$(...)` — anything without a
|
||||
# space is a legal tag — and interpolation pastes it into the script
|
||||
# verbatim, where bash would run it. Reading "$RAW_INPUT" instead makes it
|
||||
# data.
|
||||
- name: Normalize release version
|
||||
id: version
|
||||
env:
|
||||
RAW_INPUT: ${{ github.event.inputs.version }}
|
||||
RAW_BRANCH: ${{ github.event.workflow_run.head_branch }}
|
||||
run: |
|
||||
set -eux
|
||||
|
||||
if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then
|
||||
RAW="$RAW_INPUT"
|
||||
else
|
||||
RAW="$RAW_BRANCH"
|
||||
fi
|
||||
|
||||
case "$RAW" in
|
||||
refs/tags/*)
|
||||
RAW_TAG="${RAW#refs/tags/}"
|
||||
;;
|
||||
*)
|
||||
RAW_TAG="$RAW"
|
||||
;;
|
||||
esac
|
||||
|
||||
./scripts/helm_chart_version.sh "$RAW_TAG"
|
||||
|
||||
# Docker and Helm share the binary-build trigger. Do not publish a chart
|
||||
# until the image selected by its default appVersion is pullable.
|
||||
- name: Wait for release image
|
||||
timeout-minutes: 20
|
||||
env:
|
||||
APP_VERSION: ${{ steps.version.outputs.app_version }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
if [[ ! "$APP_VERSION" =~ ^[0-9]+\.[0-9]+\.[0-9]+(-[0-9A-Za-z.-]+)?$ ]] ||
|
||||
[[ "$APP_VERSION" == *-preview* ]]; then
|
||||
echo "Invalid deliverable image version: $APP_VERSION" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
for attempt in {1..60}; do
|
||||
if token=$(curl --fail --silent --show-error --connect-timeout 10 --max-time 20 \
|
||||
'https://auth.docker.io/token?service=registry.docker.io&scope=repository:rustfs/rustfs:pull' \
|
||||
| jq -er '.token // empty') &&
|
||||
curl --fail --silent --show-error --head --connect-timeout 10 --max-time 20 \
|
||||
--header "Authorization: Bearer $token" \
|
||||
--header 'Accept: application/vnd.oci.image.index.v1+json, application/vnd.docker.distribution.manifest.list.v2+json' \
|
||||
"https://registry-1.docker.io/v2/rustfs/rustfs/manifests/$APP_VERSION" > /dev/null; then
|
||||
echo "Release image is available: rustfs/rustfs:$APP_VERSION"
|
||||
exit 0
|
||||
fi
|
||||
echo "Waiting for rustfs/rustfs:$APP_VERSION ($attempt/60)"
|
||||
if (( attempt < 60 )); then
|
||||
sleep 15
|
||||
fi
|
||||
done
|
||||
|
||||
echo "Release image is unavailable; refusing to publish Helm chart" >&2
|
||||
exit 1
|
||||
|
||||
- name: Replace chart version and app version
|
||||
env:
|
||||
CHART_VERSION: ${{ steps.version.outputs.chart_version }}
|
||||
APP_VERSION: ${{ steps.version.outputs.app_version }}
|
||||
run: |
|
||||
set -eux
|
||||
sed -i -E "s/^version:.*/version: \"${CHART_VERSION}\"/" helm/rustfs/Chart.yaml
|
||||
sed -i -E "s/^appVersion:.*/appVersion: \"${APP_VERSION}\"/" helm/rustfs/Chart.yaml
|
||||
|
||||
- name: Set up Helm
|
||||
uses: azure/setup-helm@b9e51907a09c216f16ebe8536097933489208112 # v4.3.0
|
||||
|
||||
- name: Install yq
|
||||
- uses: azure/setup-helm@b9e51907a09c216f16ebe8536097933489208112 # v4.3.0
|
||||
- name: Install pinned yq
|
||||
run: |
|
||||
set -euo pipefail
|
||||
yq_version="4.54.1"
|
||||
@@ -238,66 +35,43 @@ jobs:
|
||||
echo "$tools_dir" >> "$GITHUB_PATH"
|
||||
"$tools_dir/yq" --version
|
||||
|
||||
- name: Test Helm Chart Templates
|
||||
run: ./scripts/test_helm_templates.sh
|
||||
|
||||
- name: Package Helm Chart
|
||||
run: |
|
||||
set -eux
|
||||
cp helm/README.md helm/rustfs/
|
||||
helm package ./helm/rustfs \
|
||||
--destination helm/rustfs/ \
|
||||
--version "${{ steps.version.outputs.chart_version }}"
|
||||
|
||||
- name: Upload helm package as artifact
|
||||
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
|
||||
with:
|
||||
name: helm-package
|
||||
path: |
|
||||
helm/rustfs/*.tgz
|
||||
helm/rustfs/README.md
|
||||
retention-days: 1
|
||||
|
||||
publish-helm-package:
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 30
|
||||
needs: [ build-helm-package ]
|
||||
if: needs.build-helm-package.result == 'success'
|
||||
|
||||
steps:
|
||||
- name: Checkout helm package repo
|
||||
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
|
||||
with:
|
||||
# persist-credentials-exempt: this checkout's token IS the push credential —
|
||||
# the job git-pushes to rustfs/helm below. Clearing it breaks chart publishing.
|
||||
repository: rustfs/helm
|
||||
token: ${{ secrets.RUSTFS_HELM_PACKAGE }}
|
||||
|
||||
- name: Download helm package
|
||||
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
|
||||
with:
|
||||
name: helm-package
|
||||
path: ./
|
||||
|
||||
- name: Set up helm
|
||||
uses: azure/setup-helm@b9e51907a09c216f16ebe8536097933489208112 # v4.3.0
|
||||
|
||||
- name: Generate index
|
||||
run: helm repo index . --url https://charts.rustfs.com
|
||||
|
||||
# app_version is derived from the triggering tag name, and this job holds
|
||||
# the cross-repository push token with rustfs/helm already checked out —
|
||||
# the worst place in the repo to paste an attacker-influenced string into
|
||||
# a shell line. Passed through env so bash treats it as data.
|
||||
- name: Push helm package and index file
|
||||
- name: Test and package the exact application version
|
||||
id: chart
|
||||
env:
|
||||
GIT_USERNAME: ${{ secrets.USERNAME }}
|
||||
GIT_EMAIL: ${{ secrets.EMAIL_ADDRESS }}
|
||||
APP_VERSION: ${{ needs.build-helm-package.outputs.app_version }}
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
CANDIDATE_MANIFEST: ${{ inputs.candidate_manifest }}
|
||||
run: |
|
||||
set -eux
|
||||
git config --global user.name "${GIT_USERNAME}"
|
||||
git config --global user.email "${GIT_EMAIL}"
|
||||
git add .
|
||||
git commit -m "Update rustfs helm package with ${APP_VERSION}." || echo "No changes to commit"
|
||||
git push origin main
|
||||
python3 - <<'PYCHART'
|
||||
import sys, subprocess, shutil
|
||||
from pathlib import Path
|
||||
sys.path.insert(0, 'scripts/release')
|
||||
import candidate as c, pipeline
|
||||
inputs = pipeline.manifest_env()
|
||||
current = pipeline.current_run('release-complete.yml')
|
||||
if any(a['name'] == 'helm-package' for a in c.artifact_list(current['id'])):
|
||||
c.download_artifact(current, 'helm-package', 'chart')
|
||||
record = c.read_json('chart/chart.json')
|
||||
c.require(record['inputs_sha256'] == c.digest(inputs), 'chart belongs to another candidate')
|
||||
c.emit_output('reused', 'true')
|
||||
else:
|
||||
chart = Path('helm/rustfs/Chart.yaml')
|
||||
import re
|
||||
value = chart.read_text()
|
||||
for key in ('version', 'appVersion'):
|
||||
value, count = re.subn(rf'(?m)^{key}:.*$', f'{key}: "{inputs["version"]}"', value)
|
||||
c.require(count == 1, 'ambiguous chart metadata')
|
||||
chart.write_text(value)
|
||||
subprocess.run(['bash', 'scripts/test_helm_templates.sh'], check=True)
|
||||
shutil.copyfile('helm/README.md', 'helm/rustfs/README.md')
|
||||
subprocess.run(['helm', 'package', 'helm/rustfs', '--destination', 'chart'], check=True)
|
||||
path = Path('chart') / f"rustfs-{inputs['version']}.tgz"
|
||||
c.write_json('chart/chart.json', {'version': inputs['version'], 'inputs_sha256': c.digest(inputs),
|
||||
'name': path.name, 'sha256': c.file_digest(path), 'size': path.stat().st_size})
|
||||
PYCHART
|
||||
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
|
||||
if: steps.chart.outputs.reused != 'true'
|
||||
with:
|
||||
name: helm-package
|
||||
path: chart/*
|
||||
if-no-files-found: error
|
||||
retention-days: 30
|
||||
|
||||
@@ -47,8 +47,8 @@ permissions:
|
||||
contents: read
|
||||
|
||||
concurrency:
|
||||
group: oidc-keycloak-live-${{ github.ref }}
|
||||
cancel-in-progress: true
|
||||
group: oidc-keycloak-live-${{ github.event_name }}-${{ github.ref }}
|
||||
cancel-in-progress: ${{ github.event_name != 'schedule' }}
|
||||
|
||||
jobs:
|
||||
oidc-keycloak-live:
|
||||
|
||||
+38
-825
@@ -1,840 +1,53 @@
|
||||
# Copyright 2024 RustFS Team
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
# Package Workflow - Build DEB/RPM packages
|
||||
#
|
||||
# This workflow builds DEB and RPM packages from pre-built Linux binaries
|
||||
# and uploads them to Cloudflare R2 and the GitHub release.
|
||||
#
|
||||
# Trigger:
|
||||
# - workflow_run: automatically package after "Build and Release" completes
|
||||
# for a release tag (the mac/windows/linux binaries are already uploaded
|
||||
# to the GitHub release before packaging starts)
|
||||
# - workflow_dispatch: manual fallback with a release tag and/or exact build run ID
|
||||
#
|
||||
# Flow:
|
||||
# 1. Resolve and validate the selected Build workflow run and source identity
|
||||
# 2. Download Linux binaries (x86_64-gnu, aarch64-gnu, x86_64-musl,
|
||||
# aarch64-musl) from build artifacts
|
||||
# 3. Build DEB packages for amd64 and arm64, gnu and musl variants
|
||||
# 4. Build RPM packages for x86_64 and aarch64, gnu and musl variants
|
||||
# 5. Upload all packages to Cloudflare R2 and the GitHub release
|
||||
|
||||
name: Package DEB/RPM
|
||||
|
||||
permissions:
|
||||
# contents: write is required to upload packages to the GitHub release
|
||||
contents: write
|
||||
actions: read
|
||||
|
||||
# Copyright 2026 RustFS Team
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
name: Build candidate packages
|
||||
on:
|
||||
# Follows the same pattern as docker.yml: run after the release build
|
||||
# workflow completes, so packaging is triggered only by release tags
|
||||
# (e.g. 1.0.0-rc.2, 1.0.0-rc.3), never by development builds.
|
||||
workflow_run:
|
||||
workflows: [ "Build and Release" ]
|
||||
types: [ completed ]
|
||||
workflow_dispatch:
|
||||
workflow_call:
|
||||
inputs:
|
||||
tag:
|
||||
description: "Release tag to package (e.g. 1.0.0-beta.12). Leave empty for latest main build."
|
||||
required: false
|
||||
candidate_manifest:
|
||||
type: string
|
||||
build_run_id:
|
||||
description: "Build workflow run ID (when combined with tag, both must identify the same release commit)"
|
||||
required: false
|
||||
type: string
|
||||
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.event.workflow_run.head_branch || github.event.inputs.tag || github.run_id }}
|
||||
cancel-in-progress: true
|
||||
|
||||
env:
|
||||
HEAD_BRANCH: ${{ github.event.workflow_run.head_branch }}
|
||||
WORKFLOW_RUN_ID: ${{ github.event.workflow_run.id }}
|
||||
|
||||
required: true
|
||||
permissions:
|
||||
contents: read
|
||||
actions: read
|
||||
jobs:
|
||||
# Resolve which build run to use and extract version info
|
||||
resolve:
|
||||
name: Resolve Build
|
||||
# Auto-trigger only from successful tag builds of "Build and Release".
|
||||
# Tag pushes arrive as event == push with head_branch != main (a
|
||||
# non-main push head_branch is the release tag name). Manual dispatch
|
||||
# stays available as a fallback for backfills and re-runs.
|
||||
if: >-
|
||||
github.event_name == 'workflow_dispatch' ||
|
||||
(github.event.workflow_run.conclusion == 'success' &&
|
||||
github.event.workflow_run.event == 'push' &&
|
||||
github.event.workflow_run.head_branch != 'main')
|
||||
runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
|
||||
timeout-minutes: 10
|
||||
outputs:
|
||||
version: ${{ steps.resolve.outputs.version }}
|
||||
build_type: ${{ steps.resolve.outputs.build_type }}
|
||||
build_run_id: ${{ steps.resolve.outputs.build_run_id }}
|
||||
build_run_number: ${{ steps.resolve.outputs.build_run_number }}
|
||||
head_sha: ${{ steps.resolve.outputs.head_sha }}
|
||||
dev_sequence: ${{ steps.resolve.outputs.dev_sequence }}
|
||||
tag: ${{ steps.resolve.outputs.tag }}
|
||||
steps:
|
||||
- name: Resolve build run
|
||||
id: resolve
|
||||
shell: bash
|
||||
env:
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
EVENT_NAME: ${{ github.event_name }}
|
||||
REPOSITORY: ${{ github.repository }}
|
||||
INPUT_TAG: ${{ github.event.inputs.tag }}
|
||||
INPUT_RUN_ID: ${{ github.event.inputs.build_run_id }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
|
||||
fail() {
|
||||
echo "❌ $1" >&2
|
||||
exit 1
|
||||
}
|
||||
|
||||
TAG=""
|
||||
BUILD_RUN_ID=""
|
||||
case "$EVENT_NAME" in
|
||||
workflow_run)
|
||||
TAG="$HEAD_BRANCH"
|
||||
BUILD_RUN_ID="$WORKFLOW_RUN_ID"
|
||||
;;
|
||||
workflow_dispatch)
|
||||
TAG="$INPUT_TAG"
|
||||
BUILD_RUN_ID="$INPUT_RUN_ID"
|
||||
;;
|
||||
*) fail "unsupported event: $EVENT_NAME" ;;
|
||||
esac
|
||||
|
||||
# Validate and classify tags before using them in API paths or logs.
|
||||
semver_core='(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)'
|
||||
prerelease_id='(alpha|beta|rc)\.(0|[1-9][0-9]*)'
|
||||
if [[ -n "$TAG" ]]; then
|
||||
if [[ "$TAG" =~ ^${semver_core}(-${prerelease_id})?-preview\.(0|[1-9][0-9]*)$ ]]; then
|
||||
BUILD_TYPE=preview
|
||||
elif [[ "$TAG" =~ ^${semver_core}-${prerelease_id}$ ]]; then
|
||||
BUILD_TYPE=prerelease
|
||||
elif [[ "$TAG" =~ ^${semver_core}$ ]]; then
|
||||
BUILD_TYPE=release
|
||||
else
|
||||
fail "tag is not a supported strict package version"
|
||||
fi
|
||||
else
|
||||
BUILD_TYPE=development
|
||||
fi
|
||||
|
||||
if [[ -n "$BUILD_RUN_ID" ]]; then
|
||||
[[ "$BUILD_RUN_ID" =~ ^[1-9][0-9]*$ ]] || fail "build run ID must be a positive decimal integer"
|
||||
echo "Using selected build run: $BUILD_RUN_ID"
|
||||
elif [[ -n "$TAG" ]]; then
|
||||
echo "Looking for build run for tag: $TAG"
|
||||
BUILD_RUN_ID=$(gh api --method GET \
|
||||
"repos/${REPOSITORY}/actions/workflows/build.yml/runs" \
|
||||
-f branch="$TAG" -f status=success -F per_page=1 \
|
||||
--jq '.workflow_runs[0].id // empty' 2>/dev/null || true)
|
||||
|
||||
if [[ -z "$BUILD_RUN_ID" ]]; then
|
||||
BUILD_RUN_ID=$(gh api --method GET \
|
||||
"repos/${REPOSITORY}/actions/workflows/build.yml/runs" \
|
||||
-f event=push -f status=success -F per_page=100 2>/dev/null |
|
||||
jq -r --arg tag "$TAG" \
|
||||
'[.workflow_runs[] | select(.head_branch == $tag)][0].id // empty' || true)
|
||||
fi
|
||||
[[ "$BUILD_RUN_ID" =~ ^[1-9][0-9]*$ ]] || fail "no successful build run found for tag"
|
||||
echo "Found build run: $BUILD_RUN_ID"
|
||||
else
|
||||
echo "No tag specified, looking for latest main build"
|
||||
BUILD_RUN_ID=$(gh api --method GET \
|
||||
"repos/${REPOSITORY}/actions/workflows/build.yml/runs" \
|
||||
-f branch=main -f status=success -F per_page=1 \
|
||||
--jq '.workflow_runs[0].id // empty' 2>/dev/null || true)
|
||||
[[ "$BUILD_RUN_ID" =~ ^[1-9][0-9]*$ ]] || fail "no successful main build found"
|
||||
echo "Latest main build: $BUILD_RUN_ID"
|
||||
fi
|
||||
|
||||
# Fetch once and use the same immutable run metadata for identity,
|
||||
# ordering, workflow provenance, and release-channel validation.
|
||||
RUN_JSON=$(gh api "repos/${REPOSITORY}/actions/runs/${BUILD_RUN_ID}") ||
|
||||
fail "cannot read selected build run"
|
||||
RUN_ID=$(jq -r '.id // empty' <<<"$RUN_JSON")
|
||||
RUN_NUMBER=$(jq -r '.run_number // empty' <<<"$RUN_JSON")
|
||||
RUN_STATUS=$(jq -r '.status // empty' <<<"$RUN_JSON")
|
||||
RUN_CONCLUSION=$(jq -r '.conclusion // empty' <<<"$RUN_JSON")
|
||||
RUN_PATH=$(jq -r '.path // empty' <<<"$RUN_JSON")
|
||||
HEAD_SHA=$(jq -r '.head_sha // empty' <<<"$RUN_JSON")
|
||||
RUN_HEAD_BRANCH=$(jq -r '.head_branch // empty' <<<"$RUN_JSON")
|
||||
|
||||
[[ "$RUN_ID" == "$BUILD_RUN_ID" ]] || fail "run metadata ID mismatch"
|
||||
[[ "$RUN_NUMBER" =~ ^[1-9][0-9]*$ ]] || fail "build run number must be a positive decimal integer"
|
||||
[[ "$RUN_STATUS" == completed && "$RUN_CONCLUSION" == success ]] || fail "selected build run is not successful"
|
||||
[[ "$RUN_PATH" == .github/workflows/build.yml ]] || fail "selected run is not Build and Release"
|
||||
[[ "$HEAD_SHA" =~ ^[0-9a-f]{40}$ ]] || fail "selected build run has an invalid head SHA"
|
||||
[[ "$RUN_HEAD_BRANCH" != *$'\n'* && -n "$RUN_HEAD_BRANCH" ]] || fail "selected build run has an invalid head branch"
|
||||
|
||||
if [[ -n "$TAG" ]]; then
|
||||
[[ "$RUN_HEAD_BRANCH" == "$TAG" ]] || fail "tag and build run head branch do not match"
|
||||
|
||||
TAG_REF_JSON=$(gh api "repos/${REPOSITORY}/git/ref/tags/${TAG}") ||
|
||||
fail "cannot resolve release tag ref"
|
||||
TAG_OBJECT_TYPE=$(jq -r '.object.type // empty' <<<"$TAG_REF_JSON")
|
||||
TAG_OBJECT_SHA=$(jq -r '.object.sha // empty' <<<"$TAG_REF_JSON")
|
||||
depth=0
|
||||
while [[ "$TAG_OBJECT_TYPE" == tag && $depth -lt 5 ]]; do
|
||||
TAG_OBJECT_JSON=$(gh api "repos/${REPOSITORY}/git/tags/${TAG_OBJECT_SHA}") ||
|
||||
fail "cannot peel annotated release tag"
|
||||
TAG_OBJECT_TYPE=$(jq -r '.object.type // empty' <<<"$TAG_OBJECT_JSON")
|
||||
TAG_OBJECT_SHA=$(jq -r '.object.sha // empty' <<<"$TAG_OBJECT_JSON")
|
||||
depth=$((depth + 1))
|
||||
done
|
||||
[[ "$TAG_OBJECT_TYPE" == commit && "$TAG_OBJECT_SHA" =~ ^[0-9a-f]{40}$ ]] ||
|
||||
fail "release tag does not resolve to a commit"
|
||||
[[ "$TAG_OBJECT_SHA" == "$HEAD_SHA" ]] || fail "release tag commit and build run head SHA do not match"
|
||||
VERSION="$TAG"
|
||||
DEV_SEQUENCE=""
|
||||
else
|
||||
VERSION="dev-${HEAD_SHA}"
|
||||
DEV_SEQUENCE="$RUN_NUMBER"
|
||||
fi
|
||||
|
||||
{
|
||||
echo "version=$VERSION"
|
||||
echo "build_type=$BUILD_TYPE"
|
||||
echo "build_run_id=$BUILD_RUN_ID"
|
||||
echo "build_run_number=$RUN_NUMBER"
|
||||
echo "head_sha=$HEAD_SHA"
|
||||
echo "dev_sequence=$DEV_SEQUENCE"
|
||||
echo "tag=${TAG}"
|
||||
} >> "$GITHUB_OUTPUT"
|
||||
|
||||
echo "📊 Resolved:"
|
||||
echo " Version: $VERSION"
|
||||
echo " Build type: $BUILD_TYPE"
|
||||
echo " Build run ID: $BUILD_RUN_ID"
|
||||
echo " Build run number: $RUN_NUMBER"
|
||||
|
||||
# Consume the trusted release certification of the resolved build run and
|
||||
# verify the certified input digests before any package is wrapped. A run
|
||||
# without that certification fails closed here; packages built later must
|
||||
# complete their own verification and cannot inherit an unverified
|
||||
# conclusion.
|
||||
release-certification:
|
||||
name: Release Certification
|
||||
needs: resolve
|
||||
if: needs.resolve.result == 'success'
|
||||
runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
|
||||
timeout-minutes: 30
|
||||
permissions:
|
||||
contents: read
|
||||
actions: read
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
|
||||
with:
|
||||
persist-credentials: false
|
||||
|
||||
- name: Download and verify release certification
|
||||
env:
|
||||
GH_TOKEN: ${{ secrets.RELEASE_VALIDATION_TOKEN || github.token }}
|
||||
BUILD_RUN_ID: ${{ needs.resolve.outputs.build_run_id }}
|
||||
BUILD_TYPE: ${{ needs.resolve.outputs.build_type }}
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
mkdir -p certification
|
||||
if [[ "$BUILD_TYPE" == "development" ]]; then
|
||||
# Development artifacts are explicitly uncertified and never touch
|
||||
# official or latest channels; only the staging/dev paths remain.
|
||||
jq -n '{schema_version: 1, certified: false, reason: "development artifacts are explicitly uncertified"}' \
|
||||
> certification/development-uncertified.json
|
||||
exit 0
|
||||
fi
|
||||
if ! gh run download "$BUILD_RUN_ID" -n release-certification -D certification; then
|
||||
echo "Build run ${BUILD_RUN_ID} published no release-certification artifact; failing closed" >&2
|
||||
exit 1
|
||||
fi
|
||||
python3 scripts/release/release_gate.py verify-promotion \
|
||||
--plan .config/release-validation-plan.json \
|
||||
--candidate certification/candidate.json \
|
||||
--summary certification/release-gate-output \
|
||||
--artifact-sha256 "$(jq -r '.deliverables[0].sha256' certification/candidate.json)" \
|
||||
--artifact-name "$(jq -r '.deliverables[0].name' certification/candidate.json)"
|
||||
|
||||
# Build DEB and RPM packages for each architecture and libc variant
|
||||
package:
|
||||
name: Package (${{ matrix.arch }}-${{ matrix.libc }})
|
||||
needs: [ resolve, release-certification ]
|
||||
runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
|
||||
timeout-minutes: 30
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 45
|
||||
strategy:
|
||||
fail-fast: false
|
||||
max-parallel: 1
|
||||
matrix:
|
||||
include:
|
||||
- arch: x86_64
|
||||
libc: gnu
|
||||
deb_arch: amd64
|
||||
rpm_arch: x86_64
|
||||
artifact_name: "rustfs-linux-x86_64-gnu"
|
||||
# gnu binaries dynamically link glibc; musl binaries are static
|
||||
# and run without a libc dependency.
|
||||
deb_depends: "libc6 (>= 2.31)"
|
||||
rpm_depends: "glibc >= 2.31"
|
||||
- arch: aarch64
|
||||
libc: gnu
|
||||
deb_arch: arm64
|
||||
rpm_arch: aarch64
|
||||
artifact_name: "rustfs-linux-aarch64-gnu"
|
||||
deb_depends: "libc6 (>= 2.31)"
|
||||
rpm_depends: "glibc >= 2.31"
|
||||
- arch: x86_64
|
||||
libc: musl
|
||||
deb_arch: amd64
|
||||
rpm_arch: x86_64
|
||||
artifact_name: "rustfs-linux-x86_64-musl"
|
||||
deb_depends: ""
|
||||
rpm_depends: ""
|
||||
- arch: aarch64
|
||||
libc: musl
|
||||
deb_arch: arm64
|
||||
rpm_arch: aarch64
|
||||
artifact_name: "rustfs-linux-aarch64-musl"
|
||||
deb_depends: ""
|
||||
rpm_depends: ""
|
||||
target: [linux-x86_64-musl, linux-aarch64-musl, linux-x86_64-gnu, linux-aarch64-gnu]
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
|
||||
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
|
||||
with:
|
||||
ref: ${{ fromJson(inputs.candidate_manifest).source_sha }}
|
||||
persist-credentials: false
|
||||
|
||||
- name: Normalize package metadata
|
||||
id: versions
|
||||
shell: bash
|
||||
env:
|
||||
BUILD_TYPE: ${{ needs.resolve.outputs.build_type }}
|
||||
SOURCE_VERSION: ${{ needs.resolve.outputs.version }}
|
||||
DEV_SEQUENCE: ${{ needs.resolve.outputs.dev_sequence }}
|
||||
ARCH: ${{ matrix.rpm_arch }}
|
||||
LIBC: ${{ matrix.libc }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
|
||||
normalized=$(./scripts/release/package_versions.sh \
|
||||
"$BUILD_TYPE" "$SOURCE_VERSION" "$DEV_SEQUENCE" "$ARCH" "$LIBC")
|
||||
printf '%s\n' "$normalized" >> "$GITHUB_OUTPUT"
|
||||
|
||||
- name: Download binary artifact from build run
|
||||
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
|
||||
with:
|
||||
pattern: ${{ matrix.artifact_name }}*
|
||||
path: ./binary-artifact
|
||||
run-id: ${{ needs.resolve.outputs.build_run_id }}
|
||||
github-token: ${{ github.token }}
|
||||
merge-multiple: true
|
||||
|
||||
- name: Extract binary
|
||||
id: binary
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
|
||||
ZIP_FILE=$(find ./binary-artifact -name "*.zip" -type f | head -1)
|
||||
if [[ -z "$ZIP_FILE" ]]; then
|
||||
echo "❌ No binary artifact found"
|
||||
find ./binary-artifact -mindepth 1 -maxdepth 1 -print 2>/dev/null || true
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "Found artifact: $ZIP_FILE"
|
||||
|
||||
mkdir -p ./bin
|
||||
unzip -o "$ZIP_FILE" -d ./bin
|
||||
|
||||
if [[ ! -f ./bin/rustfs ]]; then
|
||||
echo "❌ rustfs binary not found in archive"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
chmod +x ./bin/rustfs
|
||||
stat --printf='%n %s bytes\n' ./bin/rustfs
|
||||
echo "✅ Binary extracted"
|
||||
|
||||
- name: Build DEB package
|
||||
id: deb
|
||||
shell: bash
|
||||
env:
|
||||
DEB_VERSION: ${{ steps.versions.outputs.deb_version }}
|
||||
DEB_ARCH: ${{ matrix.deb_arch }}
|
||||
DEB_FILE: ${{ steps.versions.outputs.deb_file }}
|
||||
DEB_DEPENDS: ${{ matrix.deb_depends }}
|
||||
LIBC: ${{ matrix.libc }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
|
||||
PKG_DIR="${DEB_FILE%.deb}"
|
||||
|
||||
echo "Building DEB: ${DEB_FILE}"
|
||||
|
||||
mkdir -p "${PKG_DIR}/DEBIAN"
|
||||
mkdir -p "${PKG_DIR}/usr/bin"
|
||||
mkdir -p "${PKG_DIR}/etc/default"
|
||||
mkdir -p "${PKG_DIR}/lib/systemd/system"
|
||||
mkdir -p "${PKG_DIR}/usr/share/doc/rustfs"
|
||||
|
||||
cp ./bin/rustfs "${PKG_DIR}/usr/bin/"
|
||||
chmod 755 "${PKG_DIR}/usr/bin/rustfs"
|
||||
|
||||
cp deploy/build/rustfs.service "${PKG_DIR}/lib/systemd/system/"
|
||||
|
||||
cat > "${PKG_DIR}/etc/default/rustfs" << 'ENVEOF'
|
||||
# RustFS Environment Configuration
|
||||
# See https://docs.rustfs.com for more information
|
||||
# RUSTFS_ACCESS_KEY=<changeme>
|
||||
# RUSTFS_SECRET_KEY=<changeme>
|
||||
# RUSTFS_VOLUMES=""
|
||||
# RUSTFS_ADDRESS=":9000"
|
||||
# RUSTFS_CONSOLE_ADDRESS=":9001"
|
||||
# RUSTFS_CONSOLE_ENABLE=true
|
||||
# RUSTFS_OBS_LOGGER_LEVEL=error
|
||||
# RUSTFS_OBS_LOG_DIRECTORY="/var/log/rustfs/"
|
||||
ENVEOF
|
||||
|
||||
cat > "${PKG_DIR}/DEBIAN/control" << EOF
|
||||
Package: rustfs
|
||||
Version: ${DEB_VERSION}
|
||||
Section: utils
|
||||
Priority: optional
|
||||
Architecture: ${DEB_ARCH}
|
||||
EOF
|
||||
|
||||
# gnu binaries dynamically link glibc and must declare it; musl
|
||||
# binaries are statically linked and need no libc dependency.
|
||||
if [[ -n "$DEB_DEPENDS" ]]; then
|
||||
echo "Depends: ${DEB_DEPENDS}" >> "${PKG_DIR}/DEBIAN/control"
|
||||
fi
|
||||
|
||||
cat >> "${PKG_DIR}/DEBIAN/control" << EOF
|
||||
Maintainer: RustFS Team <support@rustfs.com>
|
||||
Description: High-performance distributed object storage (${LIBC} build)
|
||||
RustFS is a high-performance distributed object storage software
|
||||
built using Rust. It is compatible with MinIO and S3 API.
|
||||
Homepage: https://rustfs.com
|
||||
EOF
|
||||
|
||||
# Declare /etc/default/rustfs as a conffile so dpkg preserves user
|
||||
# modifications on upgrade instead of silently overwriting them.
|
||||
cat > "${PKG_DIR}/DEBIAN/conffiles" << 'CONFFILES'
|
||||
/etc/default/rustfs
|
||||
CONFFILES
|
||||
|
||||
./scripts/release/package_service_scripts.sh after-install > "${PKG_DIR}/DEBIAN/postinst"
|
||||
chmod 755 "${PKG_DIR}/DEBIAN/postinst"
|
||||
|
||||
./scripts/release/package_service_scripts.sh before-remove > "${PKG_DIR}/DEBIAN/prerm"
|
||||
chmod 755 "${PKG_DIR}/DEBIAN/prerm"
|
||||
|
||||
./scripts/release/package_service_scripts.sh after-remove > "${PKG_DIR}/DEBIAN/postrm"
|
||||
chmod 755 "${PKG_DIR}/DEBIAN/postrm"
|
||||
|
||||
cp LICENSE "${PKG_DIR}/usr/share/doc/rustfs/"
|
||||
cp README.md "${PKG_DIR}/usr/share/doc/rustfs/"
|
||||
|
||||
fakeroot dpkg-deb --build "${PKG_DIR}" "$DEB_FILE"
|
||||
|
||||
[[ $(dpkg-deb -f "$DEB_FILE" Package) == rustfs ]]
|
||||
[[ $(dpkg-deb -f "$DEB_FILE" Version) == "$DEB_VERSION" ]]
|
||||
[[ $(dpkg-deb -f "$DEB_FILE" Architecture) == "$DEB_ARCH" ]]
|
||||
dpkg-deb --fsys-tarfile "$DEB_FILE" | tar -tf - | grep -Fx './usr/bin/rustfs' >/dev/null
|
||||
stat --printf='%n %s bytes\n' "$DEB_FILE"
|
||||
echo "deb_file=$DEB_FILE" >> "$GITHUB_OUTPUT"
|
||||
echo "✅ DEB built: $DEB_FILE"
|
||||
|
||||
- name: Build RPM package
|
||||
id: rpm
|
||||
shell: bash
|
||||
env:
|
||||
RPM_VERSION: ${{ steps.versions.outputs.rpm_version }}
|
||||
RPM_RELEASE: ${{ steps.versions.outputs.rpm_release }}
|
||||
RPM_ARCH: ${{ matrix.rpm_arch }}
|
||||
RPM_FILE: ${{ steps.versions.outputs.rpm_file }}
|
||||
RPM_DEPENDS: ${{ matrix.rpm_depends }}
|
||||
LIBC: ${{ matrix.libc }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
|
||||
echo "Building RPM for ${RPM_ARCH} (${LIBC})"
|
||||
|
||||
sudo apt-get update && sudo apt-get install -y ruby ruby-dev build-essential rpm
|
||||
sudo gem install fpm
|
||||
./scripts/test_package_versions.sh --require-package-managers
|
||||
|
||||
# gnu binaries dynamically link glibc and must declare it; musl
|
||||
# binaries are statically linked and need no libc dependency.
|
||||
depends_args=()
|
||||
if [[ -n "$RPM_DEPENDS" ]]; then
|
||||
depends_args+=(--depends "$RPM_DEPENDS")
|
||||
fi
|
||||
|
||||
# Create config file for fpm (DEB build creates it in its package dir structure,
|
||||
# but fpm needs the file to exist before packaging)
|
||||
mkdir -p ./tmp-pkg/etc/default
|
||||
cat > ./tmp-pkg/etc/default/rustfs << 'ENVEOF'
|
||||
# RustFS Environment Configuration
|
||||
# See https://docs.rustfs.com for more information
|
||||
# RUSTFS_ACCESS_KEY=<changeme>
|
||||
# RUSTFS_SECRET_KEY=<changeme>
|
||||
# RUSTFS_VOLUMES=""
|
||||
# RUSTFS_ADDRESS=":9000"
|
||||
# RUSTFS_CONSOLE_ADDRESS=":9001"
|
||||
# RUSTFS_CONSOLE_ENABLE=true
|
||||
# RUSTFS_OBS_LOGGER_LEVEL=error
|
||||
# RUSTFS_OBS_LOG_DIRECTORY="/var/log/rustfs/"
|
||||
ENVEOF
|
||||
|
||||
fpm -s dir -t rpm \
|
||||
--name rustfs \
|
||||
--version "$RPM_VERSION" \
|
||||
--iteration "$RPM_RELEASE" \
|
||||
--architecture "$RPM_ARCH" \
|
||||
--package "$RPM_FILE" \
|
||||
"${depends_args[@]}" \
|
||||
--description "High-performance distributed object storage (${LIBC} build)" \
|
||||
--maintainer "RustFS Team <support@rustfs.com>" \
|
||||
--url "https://rustfs.com" \
|
||||
--license "Apache-2.0" \
|
||||
--before-install <(./scripts/release/package_service_scripts.sh rpm-before-install) \
|
||||
--before-upgrade <(./scripts/release/package_service_scripts.sh rpm-before-upgrade) \
|
||||
--after-install <(./scripts/release/package_service_scripts.sh after-install) \
|
||||
--after-upgrade <(./scripts/release/package_service_scripts.sh after-install) \
|
||||
--before-remove <(./scripts/release/package_service_scripts.sh before-remove) \
|
||||
--after-remove <(./scripts/release/package_service_scripts.sh after-remove) \
|
||||
--rpm-posttrans <(./scripts/release/package_service_scripts.sh rpm-posttrans) \
|
||||
--config-files /etc/default/rustfs \
|
||||
./bin/rustfs=/usr/bin/rustfs \
|
||||
./tmp-pkg/etc/default/rustfs=/etc/default/rustfs \
|
||||
deploy/build/rustfs.service=/lib/systemd/system/rustfs.service \
|
||||
LICENSE=/usr/share/doc/rustfs/LICENSE \
|
||||
README.md=/usr/share/doc/rustfs/README.md
|
||||
|
||||
if [[ ! -f "$RPM_FILE" ]]; then
|
||||
echo "❌ RPM build failed"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
RPM_METADATA=$(rpm -qp --qf '%{NAME}\n%{VERSION}\n%{RELEASE}\n%{ARCH}\n' "$RPM_FILE")
|
||||
EXPECTED_METADATA=$(printf 'rustfs\n%s\n%s\n%s' "$RPM_VERSION" "$RPM_RELEASE" "$RPM_ARCH")
|
||||
[[ "$RPM_METADATA" == "$EXPECTED_METADATA" ]]
|
||||
rpm -qpl "$RPM_FILE" | grep -Fx '/usr/bin/rustfs' >/dev/null
|
||||
stat --printf='%n %s bytes\n' "$RPM_FILE"
|
||||
echo "rpm_file=$RPM_FILE" >> "$GITHUB_OUTPUT"
|
||||
echo "✅ RPM built: $RPM_FILE"
|
||||
|
||||
- name: Download certification evidence
|
||||
if: needs.resolve.outputs.build_type != 'development'
|
||||
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
|
||||
with:
|
||||
name: release-certification
|
||||
path: ./certification
|
||||
|
||||
# Later-built wrapper verification: the binary inside each package must
|
||||
# match the certified binary digest for this platform before anything is
|
||||
# uploaded, and the package itself stays on staging until promotion.
|
||||
- name: Verify wrapped binary against certification
|
||||
if: needs.resolve.outputs.build_type != 'development'
|
||||
shell: bash
|
||||
env:
|
||||
DEB_FILE: ${{ steps.deb.outputs.deb_file }}
|
||||
RPM_FILE: ${{ steps.rpm.outputs.rpm_file }}
|
||||
ARTIFACT_NAME: ${{ matrix.artifact_name }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
expected="$(jq -r --arg needle "$ARTIFACT_NAME" \
|
||||
'[.deliverables[] | select(.name | contains($needle)) | .binary_sha256][0] // empty' \
|
||||
certification/candidate.json)"
|
||||
if [[ -z "$expected" ]]; then
|
||||
echo "No certified binary digest recorded for ${ARTIFACT_NAME}" >&2
|
||||
exit 1
|
||||
fi
|
||||
if ! command -v rpm2cpio >/dev/null 2>&1; then
|
||||
sudo apt-get update && sudo apt-get install -y rpm2cpio
|
||||
fi
|
||||
if ! command -v cpio >/dev/null 2>&1; then
|
||||
sudo apt-get update && sudo apt-get install -y cpio
|
||||
fi
|
||||
for package_file in "$DEB_FILE" "$RPM_FILE"; do
|
||||
case "$package_file" in
|
||||
*.deb)
|
||||
dpkg-deb --fsys-tarfile "$package_file" | tar -xOf - ./usr/bin/rustfs > "$RUNNER_TEMP/wrapped-rustfs"
|
||||
;;
|
||||
*.rpm)
|
||||
rpm2cpio "$package_file" | cpio -i --to-stdout ./usr/bin/rustfs > "$RUNNER_TEMP/wrapped-rustfs"
|
||||
;;
|
||||
esac
|
||||
python3 scripts/release/release_gate.py verify-promotion \
|
||||
--plan .config/release-validation-plan.json \
|
||||
--candidate certification/candidate.json \
|
||||
--summary certification/release-gate-output \
|
||||
--artifact "$RUNNER_TEMP/wrapped-rustfs" \
|
||||
--artifact-role binary
|
||||
done
|
||||
|
||||
- name: Upload packages to artifacts
|
||||
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
|
||||
with:
|
||||
name: packages-${{ matrix.arch }}-${{ matrix.libc }}
|
||||
path: |
|
||||
*.deb
|
||||
*.rpm
|
||||
retention-days: 30
|
||||
|
||||
- name: Upload packages to Cloudflare R2
|
||||
if: needs.resolve.outputs.build_type != 'preview' && env.R2_ACCESS_KEY_ID != ''
|
||||
env:
|
||||
R2_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
|
||||
R2_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
|
||||
R2_ENDPOINT: ${{ secrets.R2_ENDPOINT }}
|
||||
R2_BUCKET: ${{ secrets.R2_BUCKET }}
|
||||
AWS_EC2_METADATA_DISABLED: true
|
||||
BUILD_TYPE: ${{ needs.resolve.outputs.build_type }}
|
||||
DEB_FILE: ${{ steps.deb.outputs.deb_file }}
|
||||
RPM_FILE: ${{ steps.rpm.outputs.rpm_file }}
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
|
||||
if [[ -z "$R2_ACCESS_KEY_ID" || -z "$R2_SECRET_ACCESS_KEY" || -z "$R2_ENDPOINT" || -z "$R2_BUCKET" ]]; then
|
||||
echo "⚠️ R2 credentials missing, skipping upload"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
if ! command -v aws >/dev/null 2>&1; then
|
||||
sudo apt-get update && sudo apt-get install -y awscli
|
||||
fi
|
||||
|
||||
export AWS_ACCESS_KEY_ID="$R2_ACCESS_KEY_ID"
|
||||
export AWS_SECRET_ACCESS_KEY="$R2_SECRET_ACCESS_KEY"
|
||||
export AWS_DEFAULT_REGION="auto"
|
||||
|
||||
# Build-phase writes stay on staging for release types: the official
|
||||
# packages/release and packages/latest paths are written only by the
|
||||
# certification-gated promote-packages job, from these same staged
|
||||
# objects. Development packages keep the uncertified dev channel.
|
||||
if [[ "$BUILD_TYPE" == "development" ]]; then
|
||||
R2_PREFIX="artifacts/rustfs/packages/dev"
|
||||
else
|
||||
R2_PREFIX="artifacts/rustfs/packages/staging/${GITHUB_RUN_ID}/${GITHUB_RUN_ATTEMPT}"
|
||||
fi
|
||||
R2_PATH="s3://${R2_BUCKET}/${R2_PREFIX}/"
|
||||
|
||||
echo "📤 Uploading to $R2_PATH"
|
||||
|
||||
for f in "$DEB_FILE" "$RPM_FILE"; do
|
||||
if [[ -n "$f" && -f "$f" ]]; then
|
||||
echo "Uploading: $f"
|
||||
aws s3 cp "$f" "$R2_PATH" --endpoint-url "$R2_ENDPOINT" --only-show-errors
|
||||
fi
|
||||
done
|
||||
|
||||
echo "✅ Upload complete"
|
||||
|
||||
- name: Upload packages to GitHub Release
|
||||
if: needs.resolve.outputs.tag != ''
|
||||
- name: Reuse completed package set
|
||||
id: reuse
|
||||
env:
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
TAG: ${{ needs.resolve.outputs.tag }}
|
||||
DEB_FILE: ${{ steps.deb.outputs.deb_file }}
|
||||
RPM_FILE: ${{ steps.rpm.outputs.rpm_file }}
|
||||
shell: bash
|
||||
CANDIDATE_MANIFEST: ${{ inputs.candidate_manifest }}
|
||||
TARGET: ${{ matrix.target }}
|
||||
run: python3 scripts/release/deliver.py reuse --kind packages --target "$TARGET"
|
||||
- name: Install packaging tools
|
||||
if: steps.reuse.outputs.reused != 'true'
|
||||
run: |
|
||||
set -euo pipefail
|
||||
|
||||
# Upload the packages, then refresh the release checksums so the new
|
||||
# assets are covered, matching the binary release flow.
|
||||
for f in "$DEB_FILE" "$RPM_FILE"; do
|
||||
if [[ -n "$f" && -f "$f" ]]; then
|
||||
echo "📤 Uploading $(basename "$f") to GitHub release ${TAG}..."
|
||||
gh release upload "$TAG" "$f" --clobber
|
||||
fi
|
||||
done
|
||||
|
||||
CHECKSUM_DIR="$(mktemp -d)"
|
||||
gh release download "$TAG" -p 'SHA256SUMS' -p 'SHA512SUMS' \
|
||||
-D "$CHECKSUM_DIR" --clobber
|
||||
|
||||
for spec in "SHA256SUMS:sha256sum" "SHA512SUMS:sha512sum"; do
|
||||
asset="${spec%%:*}"
|
||||
checksum_cmd="${spec##*:}"
|
||||
checksum_file="${CHECKSUM_DIR}/${asset}"
|
||||
|
||||
for f in "$DEB_FILE" "$RPM_FILE"; do
|
||||
if [[ -n "$f" && -f "$f" ]]; then
|
||||
base="$(basename "$f")"
|
||||
# GitHub stores release asset names with '~' normalized to '.'.
|
||||
# Current package file names embed the raw tag and contain no
|
||||
# '~', but older releases may still ship such assets, so the
|
||||
# checksum entries must reference the name as stored on the
|
||||
# release.
|
||||
github_base="${base//\~/.}"
|
||||
# Remove any stale entry (both naming variants), then append
|
||||
grep -Fv -- "$base" "$checksum_file" > "${checksum_file}.tmp" || true
|
||||
grep -Fv -- "$github_base" "${checksum_file}.tmp" > "${checksum_file}.tmp2" || true
|
||||
mv "${checksum_file}.tmp2" "$checksum_file"
|
||||
digest=$("$checksum_cmd" -- "$f" | awk '{print $1}')
|
||||
printf '%s %s\n' "$digest" "$github_base" >> "$checksum_file"
|
||||
fi
|
||||
done
|
||||
|
||||
echo "📤 Updating ${asset} for release ${TAG}..."
|
||||
gh release upload "$TAG" "$checksum_file" --clobber
|
||||
done
|
||||
|
||||
echo "✅ GitHub release assets updated"
|
||||
|
||||
# The only writer of the official packages/release and packages/latest R2
|
||||
# paths. Promotion re-checks each package's wrapped binary against the
|
||||
# passed certification list and copies the staged objects byte-for-byte:
|
||||
# no rebuild-after-verify, no mutable latest resolution downstream.
|
||||
promote-packages:
|
||||
name: Promote Packages
|
||||
needs: [ resolve, package, release-certification ]
|
||||
# Target-channel lock for the official package paths.
|
||||
concurrency:
|
||||
group: release-channel-r2-packages
|
||||
cancel-in-progress: false
|
||||
if: needs.resolve.outputs.tag != '' && (needs.resolve.outputs.build_type == 'release' || needs.resolve.outputs.build_type == 'prerelease')
|
||||
runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
|
||||
timeout-minutes: 30
|
||||
permissions:
|
||||
contents: read
|
||||
actions: read
|
||||
env:
|
||||
R2_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
|
||||
R2_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
|
||||
R2_ENDPOINT: ${{ secrets.R2_ENDPOINT }}
|
||||
R2_BUCKET: ${{ secrets.R2_BUCKET }}
|
||||
AWS_EC2_METADATA_DISABLED: true
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
|
||||
with:
|
||||
persist-credentials: false
|
||||
|
||||
- name: Download packages and certification
|
||||
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
|
||||
with:
|
||||
path: ./packages
|
||||
pattern: packages-*
|
||||
merge-multiple: true
|
||||
|
||||
- name: Download certification evidence
|
||||
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
|
||||
with:
|
||||
name: release-certification
|
||||
path: ./certification
|
||||
|
||||
- name: Verify and promote staged packages
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
|
||||
if [[ -z "$R2_ACCESS_KEY_ID" || -z "$R2_SECRET_ACCESS_KEY" || -z "$R2_ENDPOINT" || -z "$R2_BUCKET" ]]; then
|
||||
echo "❌ R2 credentials missing; refusing to promote to the official channel" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
STAGING_PREFIX="s3://${R2_BUCKET}/artifacts/rustfs/packages/staging/${GITHUB_RUN_ID}/${GITHUB_RUN_ATTEMPT}/"
|
||||
RELEASE_PATH="s3://${R2_BUCKET}/artifacts/rustfs/packages/release/"
|
||||
LATEST_PATH="s3://${R2_BUCKET}/artifacts/rustfs/packages/latest/"
|
||||
|
||||
if ! command -v aws >/dev/null 2>&1; then
|
||||
sudo apt-get update && sudo apt-get install -y awscli
|
||||
fi
|
||||
export AWS_ACCESS_KEY_ID="$R2_ACCESS_KEY_ID"
|
||||
export AWS_SECRET_ACCESS_KEY="$R2_SECRET_ACCESS_KEY"
|
||||
export AWS_DEFAULT_REGION="auto"
|
||||
|
||||
if ! command -v rpm2cpio >/dev/null 2>&1; then
|
||||
sudo apt-get install -y rpm2cpio
|
||||
fi
|
||||
if ! command -v cpio >/dev/null 2>&1; then
|
||||
sudo apt-get install -y cpio
|
||||
fi
|
||||
|
||||
promoted=0
|
||||
for package_file in packages/*.deb packages/*.rpm; do
|
||||
[[ -f "$package_file" ]] || continue
|
||||
name="$(basename "$package_file")"
|
||||
case "$package_file" in
|
||||
*.deb)
|
||||
dpkg-deb --fsys-tarfile "$package_file" | tar -xOf - ./usr/bin/rustfs > "$RUNNER_TEMP/wrapped-rustfs"
|
||||
;;
|
||||
*.rpm)
|
||||
rpm2cpio "$package_file" | cpio -i --to-stdout ./usr/bin/rustfs > "$RUNNER_TEMP/wrapped-rustfs"
|
||||
;;
|
||||
esac
|
||||
python3 scripts/release/release_gate.py verify-promotion \
|
||||
--plan .config/release-validation-plan.json \
|
||||
--candidate certification/candidate.json \
|
||||
--summary certification/release-gate-output \
|
||||
--artifact "$RUNNER_TEMP/wrapped-rustfs" \
|
||||
--artifact-role binary
|
||||
|
||||
staged="$(mktemp)"
|
||||
aws s3 cp "${STAGING_PREFIX}${name}" "$staged" --endpoint-url "$R2_ENDPOINT" --only-show-errors
|
||||
staged_sha="$(sha256sum "$staged" | awk '{print $1}')"
|
||||
built_sha="$(sha256sum "$package_file" | awk '{print $1}')"
|
||||
if [[ "$staged_sha" != "$built_sha" ]]; then
|
||||
echo "❌ Staged package ${name} does not match the verified bytes" >&2
|
||||
exit 1
|
||||
fi
|
||||
aws s3 cp "$staged" "${RELEASE_PATH}${name}" --endpoint-url "$R2_ENDPOINT" --only-show-errors
|
||||
aws s3 cp "$staged" "${LATEST_PATH}${name}" --endpoint-url "$R2_ENDPOINT" --only-show-errors
|
||||
rm -f "$staged"
|
||||
promoted=$((promoted + 1))
|
||||
echo "✅ Promoted ${name} (${built_sha})"
|
||||
done
|
||||
if [[ "$promoted" -eq 0 ]]; then
|
||||
echo "No staged packages found to promote" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Summary
|
||||
summary:
|
||||
name: Summary
|
||||
needs: [ resolve, package ]
|
||||
# Every completed Build and Release run triggers this workflow; when
|
||||
# resolve declined it (not a release tag), there is nothing to summarise.
|
||||
if: always() && needs.resolve.result != 'skipped'
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 5
|
||||
steps:
|
||||
- name: Print summary
|
||||
shell: bash
|
||||
sudo apt-get update
|
||||
sudo apt-get install -y ruby ruby-dev build-essential rpm fakeroot cpio
|
||||
sudo gem install fpm --version 1.17.0 --no-document
|
||||
- name: Package accepted executable and verify contents
|
||||
if: steps.reuse.outputs.reused != 'true'
|
||||
env:
|
||||
SUMMARY_VERSION: ${{ needs.resolve.outputs.version }}
|
||||
SUMMARY_BUILD_TYPE: ${{ needs.resolve.outputs.build_type }}
|
||||
SUMMARY_BUILD_RUN_ID: ${{ needs.resolve.outputs.build_run_id }}
|
||||
SUMMARY_PACKAGE_STATUS: ${{ needs.package.result }}
|
||||
run: |
|
||||
{
|
||||
echo "## 📦 Package Summary"
|
||||
echo ""
|
||||
echo "| Item | Value |"
|
||||
echo "|------|-------|"
|
||||
echo "| Version | \`${SUMMARY_VERSION}\` |"
|
||||
echo "| Build Type | ${SUMMARY_BUILD_TYPE} |"
|
||||
echo "| Build Run | #${SUMMARY_BUILD_RUN_ID} |"
|
||||
echo "| Package Status | ${SUMMARY_PACKAGE_STATUS} |"
|
||||
} >> "$GITHUB_STEP_SUMMARY"
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
CANDIDATE_MANIFEST: ${{ inputs.candidate_manifest }}
|
||||
TARGET: ${{ matrix.target }}
|
||||
run: python3 scripts/release/deliver.py package --target "$TARGET" --output delivery
|
||||
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
|
||||
if: steps.reuse.outputs.reused != 'true'
|
||||
with:
|
||||
name: packages-${{ matrix.target }}
|
||||
path: delivery/*
|
||||
if-no-files-found: error
|
||||
compression-level: 0
|
||||
retention-days: 30
|
||||
|
||||
@@ -0,0 +1,133 @@
|
||||
# Copyright 2026 RustFS Team
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
name: Release candidate (Linux)
|
||||
on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
version:
|
||||
description: Final product version (no preview suffix)
|
||||
type: string
|
||||
required: true
|
||||
source_sha:
|
||||
description: Full reviewed source commit
|
||||
type: string
|
||||
required: true
|
||||
permissions:
|
||||
contents: read
|
||||
actions: read
|
||||
concurrency:
|
||||
group: release-candidate-${{ inputs.version }}
|
||||
cancel-in-progress: false
|
||||
jobs:
|
||||
freeze:
|
||||
if: github.repository == 'rustfs/rustfs' && github.ref == 'refs/heads/main'
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 15
|
||||
outputs:
|
||||
manifest: ${{ steps.freeze.outputs.manifest }}
|
||||
steps:
|
||||
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
|
||||
with:
|
||||
fetch-depth: 0
|
||||
persist-credentials: false
|
||||
- name: Freeze or recover original inputs
|
||||
id: freeze
|
||||
env:
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
VERSION: ${{ inputs.version }}
|
||||
SOURCE_SHA: ${{ inputs.source_sha }}
|
||||
run: python3 scripts/release/pipeline.py prepare --version "$VERSION" --source "$SOURCE_SHA" --directory frozen
|
||||
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
|
||||
if: steps.freeze.outputs.reused != 'true'
|
||||
with:
|
||||
name: release-inputs
|
||||
path: frozen/inputs.json
|
||||
if-no-files-found: error
|
||||
retention-days: 30
|
||||
linux:
|
||||
needs: freeze
|
||||
uses: ./.github/workflows/build.yml
|
||||
with:
|
||||
candidate_manifest: ${{ needs.freeze.outputs.manifest }}
|
||||
platforms: linux
|
||||
upgrade:
|
||||
needs: [freeze, linux]
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 150
|
||||
env:
|
||||
RUSTUP_TOOLCHAIN: ${{ fromJson(needs.freeze.outputs.manifest).toolchain }}
|
||||
steps:
|
||||
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
|
||||
with:
|
||||
ref: ${{ fromJson(needs.freeze.outputs.manifest).workflow_sha }}
|
||||
persist-credentials: false
|
||||
- uses: ./.github/actions/setup
|
||||
with:
|
||||
rust-version: ${{ fromJson(needs.freeze.outputs.manifest).toolchain }}
|
||||
cache-shared-key: release-upgrade-harness-hosted
|
||||
cache-save-if: ${{ github.ref == 'refs/heads/main' }}
|
||||
install-build-packaging-tools: "false"
|
||||
install-test-tools: "false"
|
||||
- name: Verify archived server against previous releases
|
||||
env:
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
CANDIDATE_MANIFEST: ${{ needs.freeze.outputs.manifest }}
|
||||
run: |
|
||||
python3 - <<'PYVERIFY'
|
||||
import sys
|
||||
from pathlib import Path
|
||||
sys.path.insert(0, 'scripts/release')
|
||||
import candidate as c, pipeline, verify
|
||||
inputs = pipeline.manifest_env()
|
||||
run = pipeline.current_run('release-candidate.yml')
|
||||
if any(a['name'] == 'release-upgrade' for a in c.artifact_list(run['id'])):
|
||||
c.download_artifact(run, 'release-upgrade', 'receipts')
|
||||
pipeline.verify_evidence(inputs, Path('receipts'), {run['id']: run})
|
||||
c.emit_output('reused', 'true')
|
||||
else:
|
||||
target = 'linux-x86_64-gnu'
|
||||
c.download_artifact(run, f"candidate-{inputs['candidate_id']}-{target}", 'binary')
|
||||
verify.upgrade(inputs, Path('binary') / f"rustfs-{target}-v{inputs['version']}.zip", Path('receipts'))
|
||||
PYVERIFY
|
||||
id: verify
|
||||
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
|
||||
if: steps.verify.outputs.reused != 'true'
|
||||
with:
|
||||
name: release-upgrade
|
||||
path: receipts/*
|
||||
if-no-files-found: error
|
||||
retention-days: 30
|
||||
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
|
||||
if: failure()
|
||||
with:
|
||||
name: release-upgrade-failure-${{ github.run_attempt }}
|
||||
path: receipts/*
|
||||
retention-days: 7
|
||||
seal-linux:
|
||||
needs: [freeze, linux, upgrade]
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 30
|
||||
steps:
|
||||
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
|
||||
with:
|
||||
persist-credentials: false
|
||||
- name: Collect trusted receipts and seal Linux bytes
|
||||
id: seal
|
||||
env:
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
run: |
|
||||
python3 scripts/release/pipeline.py linux-seal --directory sealed
|
||||
{
|
||||
echo '## Linux acceptance'
|
||||
echo 'Use this run ID and manifest SHA-256 when starting Release candidate (complete).'
|
||||
sha256sum sealed/linux-manifest.json
|
||||
echo 'Record cluster, performance, integrity and upgrade acceptance in backlog before continuing.'
|
||||
} >> "$GITHUB_STEP_SUMMARY"
|
||||
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
|
||||
if: steps.seal.outputs.reused != 'true'
|
||||
with:
|
||||
name: release-linux
|
||||
path: sealed/*
|
||||
if-no-files-found: error
|
||||
compression-level: 0
|
||||
retention-days: 30
|
||||
@@ -0,0 +1,97 @@
|
||||
# Copyright 2026 RustFS Team
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
name: Release candidate (complete)
|
||||
on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
candidate_run_id:
|
||||
description: Successful Linux candidate run ID
|
||||
type: string
|
||||
required: true
|
||||
linux_manifest_sha256:
|
||||
description: Exact accepted Linux manifest SHA-256
|
||||
type: string
|
||||
required: true
|
||||
acceptance_report:
|
||||
description: Backlog report accepting cluster, performance, integrity and upgrade results
|
||||
type: string
|
||||
required: true
|
||||
permissions:
|
||||
contents: read
|
||||
actions: read
|
||||
concurrency:
|
||||
group: release-complete-${{ inputs.candidate_run_id }}
|
||||
cancel-in-progress: false
|
||||
jobs:
|
||||
acceptance:
|
||||
if: github.repository == 'rustfs/rustfs' && github.ref == 'refs/heads/main'
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 30
|
||||
environment: release-linux-acceptance
|
||||
outputs:
|
||||
manifest: ${{ steps.accept.outputs.manifest }}
|
||||
steps:
|
||||
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
|
||||
with:
|
||||
fetch-depth: 0
|
||||
persist-credentials: false
|
||||
- name: Bind acceptance to the exact Linux manifest
|
||||
id: accept
|
||||
env:
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
CANDIDATE_RUN: ${{ inputs.candidate_run_id }}
|
||||
MANIFEST_SHA: ${{ inputs.linux_manifest_sha256 }}
|
||||
REPORT: ${{ inputs.acceptance_report }}
|
||||
run: python3 scripts/release/pipeline.py complete-prepare --run-id "$CANDIDATE_RUN" --digest "$MANIFEST_SHA" --report "$REPORT" --directory linux
|
||||
desktop:
|
||||
needs: acceptance
|
||||
uses: ./.github/workflows/build.yml
|
||||
with:
|
||||
candidate_manifest: ${{ needs.acceptance.outputs.manifest }}
|
||||
platforms: desktop
|
||||
packages:
|
||||
needs: acceptance
|
||||
uses: ./.github/workflows/package.yml
|
||||
with:
|
||||
candidate_manifest: ${{ needs.acceptance.outputs.manifest }}
|
||||
images:
|
||||
needs: acceptance
|
||||
uses: ./.github/workflows/docker.yml
|
||||
with:
|
||||
candidate_manifest: ${{ needs.acceptance.outputs.manifest }}
|
||||
chart:
|
||||
needs: acceptance
|
||||
uses: ./.github/workflows/helm-package.yml
|
||||
with:
|
||||
candidate_manifest: ${{ needs.acceptance.outputs.manifest }}
|
||||
seal:
|
||||
needs: [acceptance, desktop, packages, images, chart]
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 45
|
||||
steps:
|
||||
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
|
||||
with:
|
||||
fetch-depth: 0
|
||||
persist-credentials: false
|
||||
- name: Verify every deliverable and seal the candidate
|
||||
id: seal
|
||||
env:
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
CANDIDATE_RUN: ${{ inputs.candidate_run_id }}
|
||||
MANIFEST_SHA: ${{ inputs.linux_manifest_sha256 }}
|
||||
REPORT: ${{ inputs.acceptance_report }}
|
||||
run: |
|
||||
python3 scripts/release/pipeline.py complete-seal --run-id "$CANDIDATE_RUN" --digest "$MANIFEST_SHA" --report "$REPORT" --directory sealed
|
||||
{
|
||||
echo '## Final publication approval'
|
||||
echo 'Review this manifest, then explicitly dispatch Publish accepted release with its SHA-256.'
|
||||
sha256sum sealed/release-manifest.json
|
||||
} >> "$GITHUB_STEP_SUMMARY"
|
||||
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
|
||||
if: steps.seal.outputs.reused != 'true'
|
||||
with:
|
||||
name: release-seal
|
||||
path: sealed/*
|
||||
if-no-files-found: error
|
||||
compression-level: 0
|
||||
retention-days: 30
|
||||
@@ -0,0 +1,193 @@
|
||||
# Copyright 2026 RustFS Team
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
name: Publish accepted release
|
||||
on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
complete_run_id:
|
||||
description: Successful completed candidate run ID
|
||||
type: string
|
||||
required: true
|
||||
manifest_sha256:
|
||||
description: Exact approved final manifest SHA-256 (authorizes publication)
|
||||
type: string
|
||||
required: true
|
||||
permissions:
|
||||
contents: read
|
||||
actions: read
|
||||
concurrency:
|
||||
group: release-publish-${{ inputs.complete_run_id }}
|
||||
cancel-in-progress: false
|
||||
jobs:
|
||||
verify:
|
||||
if: github.repository == 'rustfs/rustfs' && github.ref == 'refs/heads/main'
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 45
|
||||
outputs:
|
||||
version: ${{ steps.verify.outputs.version }}
|
||||
steps:
|
||||
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
|
||||
with:
|
||||
fetch-depth: 0
|
||||
persist-credentials: false
|
||||
- name: Recheck approved bytes and trusted certification
|
||||
id: verify
|
||||
env:
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
COMPLETE_RUN: ${{ inputs.complete_run_id }}
|
||||
MANIFEST_SHA: ${{ inputs.manifest_sha256 }}
|
||||
run: python3 scripts/release/publish.py verify --run-id "$COMPLETE_RUN" --digest "$MANIFEST_SHA" --directory sealed
|
||||
version:
|
||||
needs: verify
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 90
|
||||
concurrency:
|
||||
group: release-version-${{ needs.verify.outputs.version }}
|
||||
cancel-in-progress: false
|
||||
environment: release-publication
|
||||
permissions:
|
||||
contents: write
|
||||
actions: read
|
||||
packages: write
|
||||
security-events: write
|
||||
env:
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
COMPLETE_RUN: ${{ inputs.complete_run_id }}
|
||||
MANIFEST_SHA: ${{ inputs.manifest_sha256 }}
|
||||
AWS_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
|
||||
AWS_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
|
||||
AWS_DEFAULT_REGION: auto
|
||||
AWS_REQUEST_CHECKSUM_CALCULATION: when_required
|
||||
AWS_RESPONSE_CHECKSUM_VALIDATION: when_required
|
||||
AWS_EC2_METADATA_DISABLED: "true"
|
||||
R2_ENDPOINT: ${{ secrets.R2_ENDPOINT }}
|
||||
R2_BUCKET: ${{ secrets.R2_BUCKET }}
|
||||
steps:
|
||||
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
|
||||
with:
|
||||
fetch-depth: 0
|
||||
persist-credentials: false
|
||||
- name: Install promotion tools
|
||||
run: |
|
||||
sudo apt-get update
|
||||
sudo apt-get install -y skopeo
|
||||
aws --version
|
||||
- uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
|
||||
with:
|
||||
username: rustfs
|
||||
password: ${{ secrets.DOCKERHUB_TOKEN }}
|
||||
- uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
|
||||
with:
|
||||
registry: ghcr.io
|
||||
username: ${{ secrets.GHCR_USERNAME }}
|
||||
password: ${{ secrets.GHCR_PASSWORD }}
|
||||
- uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
|
||||
with:
|
||||
registry: quay.io
|
||||
username: ${{ secrets.QUAY_USERNAME }}
|
||||
password: ${{ secrets.QUAY_PASSWORD }}
|
||||
- name: Share registry credentials with the OCI copier
|
||||
run: |
|
||||
install -m 600 "$HOME/.docker/config.json" "$RUNNER_TEMP/registry-auth.json"
|
||||
echo "REGISTRY_AUTH_FILE=$RUNNER_TEMP/registry-auth.json" >> "$GITHUB_ENV"
|
||||
- name: Promote verified version
|
||||
run: python3 scripts/release/publish.py version --run-id "$COMPLETE_RUN" --digest "$MANIFEST_SHA" --directory sealed
|
||||
- name: Publish accepted musl amd64 scan
|
||||
uses: github/codeql-action/upload-sarif@99df26d4f13ea111d4ec1a7dddef6063f76b97e9 # v4
|
||||
with:
|
||||
sarif_file: sealed/scan-musl-amd64.sarif
|
||||
category: container-image-musl-amd64
|
||||
- name: Publish accepted musl arm64 scan
|
||||
uses: github/codeql-action/upload-sarif@99df26d4f13ea111d4ec1a7dddef6063f76b97e9 # v4
|
||||
with:
|
||||
sarif_file: sealed/scan-musl-arm64.sarif
|
||||
category: container-image-musl-arm64
|
||||
- name: Publish accepted GNU amd64 scan
|
||||
uses: github/codeql-action/upload-sarif@99df26d4f13ea111d4ec1a7dddef6063f76b97e9 # v4
|
||||
with:
|
||||
sarif_file: sealed/scan-gnu-amd64.sarif
|
||||
category: container-image-gnu-amd64
|
||||
- name: Publish accepted GNU arm64 scan
|
||||
uses: github/codeql-action/upload-sarif@99df26d4f13ea111d4ec1a7dddef6063f76b97e9 # v4
|
||||
with:
|
||||
sarif_file: sealed/scan-gnu-arm64.sarif
|
||||
category: container-image-gnu-arm64
|
||||
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
|
||||
with:
|
||||
name: release-version-receipt-${{ github.run_attempt }}
|
||||
path: publication.json
|
||||
if-no-files-found: error
|
||||
retention-days: 90
|
||||
channels:
|
||||
needs: [verify, version]
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 90
|
||||
concurrency:
|
||||
group: rustfs-release-channels
|
||||
cancel-in-progress: false
|
||||
permissions:
|
||||
contents: write
|
||||
actions: read
|
||||
packages: write
|
||||
env:
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
COMPLETE_RUN: ${{ inputs.complete_run_id }}
|
||||
MANIFEST_SHA: ${{ inputs.manifest_sha256 }}
|
||||
AWS_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
|
||||
AWS_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
|
||||
AWS_DEFAULT_REGION: auto
|
||||
AWS_REQUEST_CHECKSUM_CALCULATION: when_required
|
||||
AWS_RESPONSE_CHECKSUM_VALIDATION: when_required
|
||||
AWS_EC2_METADATA_DISABLED: "true"
|
||||
R2_ENDPOINT: ${{ secrets.R2_ENDPOINT }}
|
||||
R2_BUCKET: ${{ secrets.R2_BUCKET }}
|
||||
steps:
|
||||
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
|
||||
with:
|
||||
fetch-depth: 0
|
||||
persist-credentials: false
|
||||
- name: Install promotion tools
|
||||
run: |
|
||||
sudo apt-get update
|
||||
sudo apt-get install -y skopeo
|
||||
aws --version
|
||||
- uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
|
||||
with:
|
||||
username: rustfs
|
||||
password: ${{ secrets.DOCKERHUB_TOKEN }}
|
||||
- uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
|
||||
with:
|
||||
registry: ghcr.io
|
||||
username: ${{ secrets.GHCR_USERNAME }}
|
||||
password: ${{ secrets.GHCR_PASSWORD }}
|
||||
- uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
|
||||
with:
|
||||
registry: quay.io
|
||||
username: ${{ secrets.QUAY_USERNAME }}
|
||||
password: ${{ secrets.QUAY_PASSWORD }}
|
||||
- name: Share registry credentials with the OCI copier
|
||||
run: |
|
||||
install -m 600 "$HOME/.docker/config.json" "$RUNNER_TEMP/registry-auth.json"
|
||||
echo "REGISTRY_AUTH_FILE=$RUNNER_TEMP/registry-auth.json" >> "$GITHUB_ENV"
|
||||
- uses: azure/setup-helm@b9e51907a09c216f16ebe8536097933489208112 # v4.3.0
|
||||
- name: Install pinned OSS client
|
||||
run: |
|
||||
curl --fail --location --retry 3 --output "$RUNNER_TEMP/ossutil.zip" https://gosspublic.alicdn.com/ossutil/v2/2.1.1/ossutil-2.1.1-linux-amd64.zip
|
||||
printf '%s %s\n' 60f3a808cbbdfd4b5269b8f967c6a66f564c9dacff52d93a5d21a588976ab5a2 "$RUNNER_TEMP/ossutil.zip" | sha256sum --check -
|
||||
unzip -q "$RUNNER_TEMP/ossutil.zip" -d "$RUNNER_TEMP/ossutil-extract"
|
||||
install -m 755 "$RUNNER_TEMP/ossutil-extract/ossutil-2.1.1-linux-amd64/ossutil" "$RUNNER_TEMP/ossutil"
|
||||
echo "OSSUTIL_BIN=$RUNNER_TEMP/ossutil" >> "$GITHUB_ENV"
|
||||
- name: Promote verified channels
|
||||
env:
|
||||
HELM_TOKEN: ${{ secrets.RUSTFS_HELM_PACKAGE }}
|
||||
OSS_ACCESS_KEY_ID: ${{ secrets.ALICLOUDOSS_KEY_ID }}
|
||||
OSS_ACCESS_KEY_SECRET: ${{ secrets.ALICLOUDOSS_KEY_SECRET }}
|
||||
OSS_REGION: cn-beijing
|
||||
OSS_ENDPOINT: https://oss-cn-beijing.aliyuncs.com
|
||||
run: python3 scripts/release/publish.py channels --run-id "$COMPLETE_RUN" --digest "$MANIFEST_SHA" --directory sealed
|
||||
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
|
||||
with:
|
||||
name: release-channels-receipt-${{ github.run_attempt }}
|
||||
path: publication.json
|
||||
if-no-files-found: error
|
||||
retention-days: 90
|
||||
@@ -26,7 +26,13 @@ permissions:
|
||||
jobs:
|
||||
alert-on-incomplete-run:
|
||||
name: Alert on incomplete scheduled run
|
||||
if: github.repository == 'rustfs/rustfs'
|
||||
# The registry still owns responsibility and cadence selection. Successful
|
||||
# and PR runs can never produce an incomplete scheduled-run alert.
|
||||
if: >-
|
||||
github.repository == 'rustfs/rustfs' &&
|
||||
github.event.workflow_run.conclusion != 'success' &&
|
||||
(github.event.workflow_run.event == 'schedule' ||
|
||||
github.event.workflow_run.event == 'workflow_run')
|
||||
# Stay on GitHub-hosted runners: this monitor must keep running when the
|
||||
# self-hosted sm-standard-* pool is down, which is exactly when the
|
||||
# scheduled validations it watches stall.
|
||||
|
||||
+11
-2
@@ -12,7 +12,9 @@
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
FROM alpine:3.24.2 AS build
|
||||
ARG BINARY_SOURCE=download
|
||||
|
||||
FROM alpine:3.24.2 AS download
|
||||
|
||||
ARG TARGETARCH
|
||||
ARG RELEASE=latest
|
||||
@@ -74,6 +76,14 @@ COPY scripts/download_rc.sh /usr/local/bin/download_rc.sh
|
||||
RUN sh /usr/local/bin/download_rc.sh "$TARGETARCH" musl "$RC_RELEASE" /build/rc
|
||||
|
||||
|
||||
# Candidate packaging supplies the accepted executables from verified archives.
|
||||
# The default download stage remains available for local image builds.
|
||||
FROM scratch AS supplied
|
||||
ARG TARGETARCH
|
||||
COPY .release-inputs/${TARGETARCH}/ /build/
|
||||
|
||||
FROM ${BINARY_SOURCE} AS build
|
||||
|
||||
FROM alpine:3.24.2
|
||||
|
||||
ARG RELEASE=latest
|
||||
@@ -103,7 +113,6 @@ RUN apk upgrade --no-cache && \
|
||||
tzdata \
|
||||
&& test "$(TZ=Asia/Kolkata date +%z)" = "+0530"
|
||||
|
||||
COPY --from=build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
|
||||
COPY --from=build /build/rustfs /usr/bin/rustfs
|
||||
COPY --from=build /build/rc /usr/bin/rc
|
||||
COPY entrypoint.sh /etc/rustfs/entrypoint.sh
|
||||
|
||||
+11
-1
@@ -12,7 +12,9 @@
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
FROM ubuntu:26.04 AS build
|
||||
ARG BINARY_SOURCE=download
|
||||
|
||||
FROM ubuntu:26.04 AS download
|
||||
|
||||
ARG TARGETARCH
|
||||
ARG RELEASE=latest
|
||||
@@ -80,6 +82,14 @@ RUN set -eux; \
|
||||
COPY scripts/download_rc.sh /usr/local/bin/download_rc.sh
|
||||
RUN sh /usr/local/bin/download_rc.sh "$TARGETARCH" gnu "$RC_RELEASE" /build/rc
|
||||
|
||||
# Candidate packaging supplies the accepted executables from verified archives.
|
||||
# The default download stage remains available for local image builds.
|
||||
FROM scratch AS supplied
|
||||
ARG TARGETARCH
|
||||
COPY .release-inputs/${TARGETARCH}/ /build/
|
||||
|
||||
FROM ${BINARY_SOURCE} AS build
|
||||
|
||||
FROM ubuntu:26.04
|
||||
|
||||
ARG RELEASE=latest
|
||||
|
||||
@@ -145,7 +145,7 @@ chown -R 10001:10001 data logs
|
||||
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest
|
||||
|
||||
# Using specific version
|
||||
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.4
|
||||
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.5
|
||||
```
|
||||
|
||||
If you use [podman](https://github.com/containers/podman) instead of docker, you can install the RustFS with the below command
|
||||
@@ -224,7 +224,7 @@ For developers who want to build RustFS Docker images from source with multi-arc
|
||||
./docker-buildx.sh --push
|
||||
|
||||
# Build specific version
|
||||
./docker-buildx.sh --release 1.0.4 --push
|
||||
./docker-buildx.sh --release 1.0.5 --push
|
||||
|
||||
# Build for custom registry
|
||||
./docker-buildx.sh --registry your-registry.com --namespace yourname --push
|
||||
@@ -242,7 +242,7 @@ You can also use Make targets for convenience:
|
||||
```bash
|
||||
make docker-buildx # Build locally
|
||||
make docker-buildx-push # Build and push
|
||||
make docker-buildx-version VERSION=1.0.4 # Build specific version
|
||||
make docker-buildx-version VERSION=1.0.5 # Build specific version
|
||||
make help-docker # Show all Docker-related commands
|
||||
```
|
||||
|
||||
|
||||
+3
-3
@@ -138,7 +138,7 @@ chown -R 10001:10001 data logs
|
||||
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest
|
||||
|
||||
# 使用指定版本运行
|
||||
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.4
|
||||
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.5
|
||||
```
|
||||
|
||||
如果您通过绑定挂载启用 TLS 证书目录，也请用同样方式准备该目录：
|
||||
@@ -176,7 +176,7 @@ docker compose -f docker-compose-simple.yml up -d
|
||||
./docker-buildx.sh --push
|
||||
|
||||
# 构建指定版本
|
||||
./docker-buildx.sh --release 1.0.4 --push
|
||||
./docker-buildx.sh --release 1.0.5 --push
|
||||
|
||||
# 构建并推送到自定义仓库
|
||||
./docker-buildx.sh --registry your-registry.com --namespace yourname --push
|
||||
@@ -193,7 +193,7 @@ docker compose -f docker-compose-simple.yml up -d
|
||||
```bash
|
||||
make docker-buildx # 本地构建
|
||||
make docker-buildx-push # 构建并推送
|
||||
make docker-buildx-version VERSION=1.0.4 # 构建指定版本
|
||||
make docker-buildx-version VERSION=1.0.5 # 构建指定版本
|
||||
make help-docker # 显示所有 Docker 相关命令
|
||||
```
|
||||
|
||||
|
||||
@@ -41,12 +41,8 @@ use std::{
|
||||
future::Future,
|
||||
io::ErrorKind,
|
||||
pin::Pin,
|
||||
sync::{
|
||||
Arc, LazyLock, Mutex,
|
||||
atomic::{AtomicU8, Ordering},
|
||||
},
|
||||
sync::{LazyLock, Mutex},
|
||||
task::{Context, Poll},
|
||||
time::Duration,
|
||||
};
|
||||
use tonic::{service::interceptor::InterceptedService, transport::Channel};
|
||||
use tower::Service;
|
||||
@@ -74,7 +70,7 @@ pub(crate) async fn scanner_control_time_out_client(
|
||||
let channel = get_channel_for_class(addr, ChannelClass::Control)
|
||||
.await
|
||||
.map_err(|err| crate::error::Error::other(err.to_string()))?;
|
||||
let channel = ReplayScopeChannel::new(BatchLockDeadlineChannel::new(channel), interceptor.replay_scope_audience());
|
||||
let channel = ReplayScopeChannel::new(channel, interceptor.replay_scope_audience());
|
||||
let limit = rustfs_protos::scoped_dirty_usage::SCOPED_DIRTY_USAGE_MAX_REQUEST_BYTES as usize;
|
||||
Ok(ScannerControlServiceClient::with_interceptor(channel, interceptor)
|
||||
.max_decoding_message_size(limit)
|
||||
@@ -88,7 +84,7 @@ pub async fn heal_control_time_out_client(
|
||||
let interceptor = interceptor.with_rpc_audience(addr)?;
|
||||
let channel = get_channel_for_class(addr, ChannelClass::Control).await?;
|
||||
let max_message_size = rustfs_protos::HEAL_CONTROL_RPC_MAX_MESSAGE_SIZE;
|
||||
let channel = ReplayScopeChannel::new(BatchLockDeadlineChannel::new(channel), interceptor.replay_scope_audience());
|
||||
let channel = ReplayScopeChannel::new(channel, interceptor.replay_scope_audience());
|
||||
Ok(HealControlServiceClient::with_interceptor(channel, interceptor)
|
||||
.max_decoding_message_size(max_message_size)
|
||||
.max_encoding_message_size(max_message_size))
|
||||
@@ -101,7 +97,7 @@ pub async fn tier_mutation_control_time_out_client(
|
||||
let interceptor = interceptor.with_rpc_audience(addr)?;
|
||||
let channel = get_channel_for_class(addr, ChannelClass::Control).await?;
|
||||
let max_message_size = rustfs_protos::TIER_MUTATION_RPC_MAX_MESSAGE_SIZE;
|
||||
let channel = ReplayScopeChannel::new(BatchLockDeadlineChannel::new(channel), interceptor.replay_scope_audience());
|
||||
let channel = ReplayScopeChannel::new(channel, interceptor.replay_scope_audience());
|
||||
Ok(TierMutationControlServiceClient::with_interceptor(channel, interceptor)
|
||||
.max_decoding_message_size(max_message_size)
|
||||
.max_encoding_message_size(max_message_size))
|
||||
@@ -122,7 +118,7 @@ pub async fn node_service_time_out_client_for_class(
|
||||
let channel = get_channel_for_class(addr, class).await?;
|
||||
|
||||
let max_message_size = rustfs_protos::internode_rpc_max_message_size();
|
||||
let channel = ReplayScopeChannel::new(BatchLockDeadlineChannel::new(channel), interceptor.replay_scope_audience());
|
||||
let channel = ReplayScopeChannel::new(channel, interceptor.replay_scope_audience());
|
||||
Ok(NodeServiceClient::with_interceptor(channel, interceptor)
|
||||
.max_decoding_message_size(max_message_size)
|
||||
.max_encoding_message_size(max_message_size))
|
||||
@@ -137,7 +133,7 @@ pub(crate) async fn node_service_delete_recovery_client(
|
||||
let channel = rustfs_protos::create_channel_with_timeout(addr, timeout)
|
||||
.await
|
||||
.map_err(|error| DiskError::RemoteClientUnavailable(error.to_string()))?;
|
||||
let channel = ReplayScopeChannel::new(BatchLockDeadlineChannel::new(channel), interceptor.replay_scope_audience());
|
||||
let channel = ReplayScopeChannel::new(channel, interceptor.replay_scope_audience());
|
||||
let max_message_size = rustfs_protos::internode_rpc_max_message_size();
|
||||
Ok(NodeServiceClient::with_interceptor(channel, interceptor)
|
||||
.max_decoding_message_size(max_message_size)
|
||||
@@ -253,95 +249,7 @@ pub struct ReplayScopeChannel<S> {
|
||||
}
|
||||
|
||||
/// The channel type used by internode clients after v2 authentication and replay-scope handling.
|
||||
pub type AuthenticatedChannel = ReplayScopeChannel<BatchLockDeadlineChannel<Channel>>;
|
||||
|
||||
/// Gate lock-batch payloads after the transport's internal readiness queues.
|
||||
/// Other RPC bodies pass through without an additional wrapper.
|
||||
#[derive(Clone, Debug)]
|
||||
pub struct BatchLockDeadlineChannel<S> {
|
||||
inner: S,
|
||||
}
|
||||
|
||||
impl<S> BatchLockDeadlineChannel<S> {
|
||||
fn new(inner: S) -> Self {
|
||||
Self { inner }
|
||||
}
|
||||
}
|
||||
|
||||
impl<S> Service<HttpRequest<tonic::body::Body>> for BatchLockDeadlineChannel<S>
|
||||
where
|
||||
S: Service<HttpRequest<tonic::body::Body>>,
|
||||
{
|
||||
type Response = S::Response;
|
||||
type Error = S::Error;
|
||||
type Future = S::Future;
|
||||
|
||||
fn poll_ready(&mut self, cx: &mut Context<'_>) -> Poll<Result<(), Self::Error>> {
|
||||
self.inner.poll_ready(cx)
|
||||
}
|
||||
|
||||
fn call(&mut self, mut request: HttpRequest<tonic::body::Body>) -> Self::Future {
|
||||
if let Some(deadline) = request.extensions().get::<BatchLockRpcDeadline>().cloned() {
|
||||
let body = std::mem::replace(request.body_mut(), tonic::body::Body::empty());
|
||||
*request.body_mut() = tonic::body::Body::new(BatchLockDeadlineBody {
|
||||
inner: body,
|
||||
deadline,
|
||||
dispatched: false,
|
||||
failed: false,
|
||||
});
|
||||
}
|
||||
self.inner.call(request)
|
||||
}
|
||||
}
|
||||
|
||||
struct BatchLockDeadlineBody {
|
||||
inner: tonic::body::Body,
|
||||
deadline: BatchLockRpcDeadline,
|
||||
dispatched: bool,
|
||||
failed: bool,
|
||||
}
|
||||
|
||||
impl http_body::Body for BatchLockDeadlineBody {
|
||||
type Data = bytes::Bytes;
|
||||
type Error = tonic::Status;
|
||||
|
||||
fn poll_frame(
|
||||
mut self: Pin<&mut Self>,
|
||||
cx: &mut Context<'_>,
|
||||
) -> Poll<Option<Result<http_body::Frame<Self::Data>, Self::Error>>> {
|
||||
if self.failed {
|
||||
return Poll::Ready(None);
|
||||
}
|
||||
if !self.dispatched
|
||||
&& let Err(error) = self.deadline.remaining_before_dispatch()
|
||||
{
|
||||
self.failed = true;
|
||||
return Poll::Ready(Some(Err(error)));
|
||||
}
|
||||
let frame = std::task::ready!(Pin::new(&mut self.inner).poll_frame(cx));
|
||||
if let Some(Ok(frame)) = &frame
|
||||
&& frame.is_data()
|
||||
&& !self.dispatched
|
||||
{
|
||||
// Codec work can consume the deadline in this poll. Arbitrate only
|
||||
// after encoding, immediately before any payload leaves the body.
|
||||
if let Err(error) = self.deadline.mark_dispatched() {
|
||||
self.failed = true;
|
||||
return Poll::Ready(Some(Err(error)));
|
||||
}
|
||||
self.dispatched = true;
|
||||
}
|
||||
Poll::Ready(frame)
|
||||
}
|
||||
|
||||
fn is_end_stream(&self) -> bool {
|
||||
self.failed || self.inner.is_end_stream()
|
||||
}
|
||||
|
||||
fn size_hint(&self) -> http_body::SizeHint {
|
||||
self.inner.size_hint()
|
||||
}
|
||||
}
|
||||
pub type AuthenticatedChannel = ReplayScopeChannel<Channel>;
|
||||
|
||||
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
|
||||
enum PeerReplayCapability {
|
||||
@@ -518,9 +426,7 @@ where
|
||||
)
|
||||
}) {
|
||||
Ok(headers) => request.headers_mut().extend(headers),
|
||||
Err(error) => {
|
||||
debug!(error = %error, "could not attach replay-scoped RPC signature")
|
||||
}
|
||||
Err(error) => debug!(error = %error, "could not attach replay-scoped RPC signature"),
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -582,82 +488,6 @@ where
|
||||
}
|
||||
}
|
||||
|
||||
/// A batch may leave local readiness queues only while its original budget is live.
|
||||
/// Cancellation and submission arbitrate once so a cancelled caller cannot send later.
|
||||
#[derive(Clone, Debug)]
|
||||
pub(crate) struct BatchLockRpcDeadline {
|
||||
deadline: tokio::time::Instant,
|
||||
max_acquire_wait: Duration,
|
||||
dispatch: Arc<AtomicU8>,
|
||||
}
|
||||
|
||||
impl BatchLockRpcDeadline {
|
||||
const PENDING: u8 = 0;
|
||||
const DISPATCHED: u8 = 1;
|
||||
const CANCELLED: u8 = 2;
|
||||
|
||||
pub(crate) fn new(deadline: tokio::time::Instant, max_acquire_wait: Duration) -> Self {
|
||||
Self {
|
||||
deadline,
|
||||
max_acquire_wait,
|
||||
dispatch: Arc::new(AtomicU8::new(Self::PENDING)),
|
||||
}
|
||||
}
|
||||
|
||||
/// True proves that this batch cannot send payload or enter the server lock handler.
|
||||
/// Transport headers may already have left an internal channel queue.
|
||||
pub(crate) fn cancel_before_dispatch(&self) -> bool {
|
||||
match self
|
||||
.dispatch
|
||||
.compare_exchange(Self::PENDING, Self::CANCELLED, Ordering::AcqRel, Ordering::Acquire)
|
||||
{
|
||||
Ok(_) => true,
|
||||
Err(state) => state == Self::CANCELLED,
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) fn was_dispatched(&self) -> bool {
|
||||
self.dispatch.load(Ordering::Acquire) == Self::DISPATCHED
|
||||
}
|
||||
|
||||
fn remaining_before_dispatch(&self) -> Result<Duration, tonic::Status> {
|
||||
let remaining = self.deadline.saturating_duration_since(tokio::time::Instant::now());
|
||||
if remaining <= self.max_acquire_wait || self.dispatch.load(Ordering::Acquire) == Self::CANCELLED {
|
||||
return Err(tonic::Status::deadline_exceeded("Lock batch budget exhausted before payload dispatch"));
|
||||
}
|
||||
Ok(remaining)
|
||||
}
|
||||
|
||||
fn update_timeout(&self, request: &mut tonic::Request<()>) -> Result<(), tonic::Status> {
|
||||
request.set_timeout(self.remaining_before_dispatch()?);
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn mark_dispatched(&self) -> Result<(), tonic::Status> {
|
||||
self.remaining_before_dispatch()?;
|
||||
self.dispatch
|
||||
.compare_exchange(Self::PENDING, Self::DISPATCHED, Ordering::AcqRel, Ordering::Acquire)
|
||||
.map(|_| ())
|
||||
.map_err(|_| tonic::Status::deadline_exceeded("Lock batch cancelled before dispatch"))
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
pub(super) async fn dispatch_payload_for_test(&self) -> Result<(), tonic::Status> {
|
||||
use http_body_util::BodyExt as _;
|
||||
let mut body = BatchLockDeadlineBody {
|
||||
inner: tonic::body::Body::new(http_body_util::Full::new(bytes::Bytes::from_static(b"payload"))),
|
||||
deadline: self.clone(),
|
||||
dispatched: false,
|
||||
failed: false,
|
||||
};
|
||||
match body.frame().await {
|
||||
Some(Ok(frame)) if frame.is_data() => Ok(()),
|
||||
Some(Err(error)) => Err(error),
|
||||
_ => Err(tonic::Status::internal("test lock payload did not produce a data frame")),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
pub struct TonicSignatureInterceptor {
|
||||
audience: Option<String>,
|
||||
body_digest_strict: bool,
|
||||
@@ -752,21 +582,11 @@ impl TonicInterceptor {
|
||||
}
|
||||
|
||||
impl tonic::service::Interceptor for TonicInterceptor {
|
||||
fn call(&mut self, mut req: tonic::Request<()>) -> Result<tonic::Request<()>, tonic::Status> {
|
||||
let batch_deadline = req.extensions().get::<BatchLockRpcDeadline>().cloned();
|
||||
if let Some(deadline) = &batch_deadline {
|
||||
deadline.update_timeout(&mut req)?;
|
||||
}
|
||||
let mut req = match self {
|
||||
fn call(&mut self, req: tonic::Request<()>) -> Result<tonic::Request<()>, tonic::Status> {
|
||||
match self {
|
||||
TonicInterceptor::Signature(interceptor) => interceptor.call(req),
|
||||
TonicInterceptor::NoOp(interceptor) => interceptor.call(req),
|
||||
}?;
|
||||
if let Some(deadline) = batch_deadline {
|
||||
// Signing consumes budget, but Channel readiness only reserves a
|
||||
// queue slot. The payload body arbitrates actual dispatch later.
|
||||
deadline.update_timeout(&mut req)?;
|
||||
}
|
||||
Ok(req)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -962,136 +782,6 @@ mod tests {
|
||||
test_request_for("Ping")
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn batch_dispatch_and_cancellation_have_one_winner() {
|
||||
for _ in 0..100 {
|
||||
let deadline = BatchLockRpcDeadline::new(tokio::time::Instant::now() + Duration::from_secs(1), Duration::ZERO);
|
||||
let other = deadline.clone();
|
||||
let barrier = Arc::new(std::sync::Barrier::new(2));
|
||||
let other_barrier = barrier.clone();
|
||||
let submit = std::thread::spawn(move || {
|
||||
other_barrier.wait();
|
||||
other.mark_dispatched().is_ok()
|
||||
});
|
||||
barrier.wait();
|
||||
let cancelled = deadline.cancel_before_dispatch();
|
||||
let submitted = submit.join().expect("submission thread");
|
||||
assert_ne!(cancelled, submitted);
|
||||
assert_eq!(deadline.was_dispatched(), submitted);
|
||||
assert!(deadline.mark_dispatched().is_err());
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn batch_interceptor_rejects_exhausted_wait_reserve_before_submission() {
|
||||
let deadline =
|
||||
BatchLockRpcDeadline::new(tokio::time::Instant::now() + Duration::from_millis(20), Duration::from_millis(30));
|
||||
let mut request = tonic::Request::new(());
|
||||
request.extensions_mut().insert(deadline.clone());
|
||||
let status = TonicInterceptor::NoOp(NoOpInterceptor)
|
||||
.call(request)
|
||||
.expect_err("insufficient wait reserve");
|
||||
assert_eq!(status.code(), tonic::Code::DeadlineExceeded);
|
||||
assert!(!deadline.was_dispatched());
|
||||
assert!(deadline.cancel_before_dispatch());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn batch_interceptor_rejects_cancelled_ready_request_and_accepts_live_budget() {
|
||||
let deadline = BatchLockRpcDeadline::new(tokio::time::Instant::now() + Duration::from_secs(2), Duration::from_millis(1));
|
||||
let mut request = tonic::Request::new(());
|
||||
request.extensions_mut().insert(deadline.clone());
|
||||
assert!(deadline.cancel_before_dispatch());
|
||||
assert!(TonicInterceptor::NoOp(NoOpInterceptor).call(request).is_err());
|
||||
assert!(!deadline.was_dispatched());
|
||||
|
||||
let deadline = BatchLockRpcDeadline::new(tokio::time::Instant::now() + Duration::from_secs(2), Duration::from_millis(1));
|
||||
let mut request = tonic::Request::new(());
|
||||
request.extensions_mut().insert(deadline.clone());
|
||||
let request = TonicInterceptor::NoOp(NoOpInterceptor).call(request).expect("live budget");
|
||||
assert!(!deadline.was_dispatched(), "interceptor completion does not bypass the transport queue");
|
||||
assert!(request.metadata().contains_key("grpc-timeout"));
|
||||
assert!(deadline.cancel_before_dispatch());
|
||||
}
|
||||
|
||||
#[tokio::test(start_paused = true)]
|
||||
async fn batch_body_rejects_payload_after_channel_queue_deadline() {
|
||||
let deadline = BatchLockRpcDeadline::new(tokio::time::Instant::now() + Duration::from_millis(20), Duration::ZERO);
|
||||
tokio::time::advance(Duration::from_millis(21)).await;
|
||||
let error = deadline
|
||||
.dispatch_payload_for_test()
|
||||
.await
|
||||
.expect_err("queued payload expired");
|
||||
assert_eq!(error.code(), tonic::Code::DeadlineExceeded);
|
||||
assert!(!deadline.was_dispatched());
|
||||
assert!(deadline.cancel_before_dispatch());
|
||||
}
|
||||
|
||||
#[tokio::test(start_paused = true)]
|
||||
async fn batch_body_rejects_payload_when_encoding_crosses_deadline_in_one_poll() {
|
||||
use http_body_util::BodyExt as _;
|
||||
struct SlowEncodingBody {
|
||||
deadline: tokio::time::Instant,
|
||||
}
|
||||
impl http_body::Body for SlowEncodingBody {
|
||||
type Data = bytes::Bytes;
|
||||
type Error = tonic::Status;
|
||||
|
||||
fn poll_frame(
|
||||
self: Pin<&mut Self>,
|
||||
_cx: &mut Context<'_>,
|
||||
) -> Poll<Option<Result<http_body::Frame<Self::Data>, Self::Error>>> {
|
||||
// Model codec work that begins before the deadline but blocks
|
||||
// this single poll until the original budget has expired.
|
||||
tokio::time::resume();
|
||||
std::thread::sleep(
|
||||
self.deadline.saturating_duration_since(tokio::time::Instant::now()) + Duration::from_millis(1),
|
||||
);
|
||||
Poll::Ready(Some(Ok(http_body::Frame::data(bytes::Bytes::from_static(b"encoded payload")))))
|
||||
}
|
||||
}
|
||||
let expires = tokio::time::Instant::now() + Duration::from_millis(20);
|
||||
let deadline = BatchLockRpcDeadline::new(expires, Duration::ZERO);
|
||||
let mut body = BatchLockDeadlineBody {
|
||||
inner: tonic::body::Body::new(SlowEncodingBody { deadline: expires }),
|
||||
deadline: deadline.clone(),
|
||||
dispatched: false,
|
||||
failed: false,
|
||||
};
|
||||
let error = body
|
||||
.frame()
|
||||
.await
|
||||
.expect("first frame result")
|
||||
.expect_err("late encoded payload");
|
||||
assert_eq!(error.code(), tonic::Code::DeadlineExceeded);
|
||||
assert!(!deadline.was_dispatched());
|
||||
assert!(deadline.cancel_before_dispatch());
|
||||
assert!(body.frame().await.is_none(), "failed body must not expose a later frame");
|
||||
}
|
||||
|
||||
#[tokio::test(start_paused = true)]
|
||||
async fn batch_body_keeps_cleanup_responsibility_after_first_payload() {
|
||||
use http_body_util::BodyExt as _;
|
||||
let deadline = BatchLockRpcDeadline::new(tokio::time::Instant::now() + Duration::from_millis(20), Duration::ZERO);
|
||||
let frames = futures::stream::iter([
|
||||
Ok::<_, tonic::Status>(http_body::Frame::data(bytes::Bytes::from_static(b"first"))),
|
||||
Ok(http_body::Frame::data(bytes::Bytes::from_static(b"last"))),
|
||||
]);
|
||||
let mut body = BatchLockDeadlineBody {
|
||||
inner: tonic::body::Body::new(http_body_util::StreamBody::new(frames)),
|
||||
deadline: deadline.clone(),
|
||||
dispatched: false,
|
||||
failed: false,
|
||||
};
|
||||
assert!(body.frame().await.expect("first frame").expect("live payload").is_data());
|
||||
assert!(deadline.was_dispatched());
|
||||
assert!(!deadline.cancel_before_dispatch());
|
||||
tokio::time::advance(Duration::from_millis(21)).await;
|
||||
assert!(body.frame().await.expect("last frame").expect("late drain").is_data());
|
||||
assert!(body.frame().await.is_none());
|
||||
assert!(!deadline.cancel_before_dispatch(), "a sent payload requires late cleanup ownership");
|
||||
}
|
||||
|
||||
fn test_request_for(method: &'static str) -> tonic::Request<()> {
|
||||
let mut request = tonic::Request::new(());
|
||||
request
|
||||
|
||||
@@ -169,72 +169,6 @@ fn decode_delete_recovery_batch_errors(codes: Vec<Option<u32>>, expected_len: us
|
||||
.collect()
|
||||
}
|
||||
|
||||
fn log_delete_recovery_batch_result(
|
||||
endpoint: &Endpoint,
|
||||
expected_len: usize,
|
||||
started: Instant,
|
||||
result: &Result<Vec<Option<DiskError>>>,
|
||||
) {
|
||||
let elapsed_ms = started.elapsed().as_millis();
|
||||
let mut disk_error_counts = std::collections::BTreeMap::<u32, usize>::new();
|
||||
let (failure_scope, rpc_status) = match result {
|
||||
Ok(errors) => {
|
||||
for error in errors.iter().flatten() {
|
||||
*disk_error_counts.entry(error.to_u32()).or_default() += 1;
|
||||
}
|
||||
("object_group", None)
|
||||
}
|
||||
Err(error) => {
|
||||
disk_error_counts.insert(error.to_u32(), expected_len);
|
||||
let rpc_status = match error {
|
||||
DiskError::Io(source) => crate::cluster::rpc::client::embedded_tonic_status(source).map(tonic::Status::code),
|
||||
_ => None,
|
||||
};
|
||||
("rpc_or_reply", rpc_status)
|
||||
}
|
||||
};
|
||||
let failed_group_count = disk_error_counts.values().sum::<usize>();
|
||||
let result = if result.is_err() || failed_group_count > 0 {
|
||||
"failed"
|
||||
} else {
|
||||
"ok"
|
||||
};
|
||||
if result == "failed" || elapsed_ms >= 1_000 {
|
||||
warn!(
|
||||
event = EVENT_REMOTE_DISK_RPC,
|
||||
component = LOG_COMPONENT_ECSTORE,
|
||||
subsystem = LOG_SUBSYSTEM_REMOTE_DISK,
|
||||
result,
|
||||
operation = "apply_versions",
|
||||
failure_scope,
|
||||
pool_index = endpoint.pool_idx,
|
||||
set_index = endpoint.set_idx,
|
||||
disk_index = endpoint.disk_idx,
|
||||
attempted_group_count = expected_len,
|
||||
failed_group_count,
|
||||
disk_error_counts = ?disk_error_counts,
|
||||
rpc_status = ?rpc_status,
|
||||
elapsed_ms,
|
||||
"Recoverable batch delete RPC completed"
|
||||
);
|
||||
} else {
|
||||
debug!(
|
||||
event = EVENT_REMOTE_DISK_RPC,
|
||||
component = LOG_COMPONENT_ECSTORE,
|
||||
subsystem = LOG_SUBSYSTEM_REMOTE_DISK,
|
||||
result,
|
||||
operation = "apply_versions",
|
||||
pool_index = endpoint.pool_idx,
|
||||
set_index = endpoint.set_idx,
|
||||
disk_index = endpoint.disk_idx,
|
||||
attempted_group_count = expected_len,
|
||||
failed_group_count,
|
||||
elapsed_ms,
|
||||
"Recoverable batch delete RPC completed"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
fn validate_delete_recovery_inspection(
|
||||
volume: &str,
|
||||
object: &str,
|
||||
@@ -4439,7 +4373,6 @@ impl DiskAPI for RemoteDisk {
|
||||
opts: DeleteOptions,
|
||||
) -> Vec<Option<DiskError>> {
|
||||
let expected_len = versions.len();
|
||||
let started = Instant::now();
|
||||
let result = match self
|
||||
.recover_delete_transaction(volume, DeleteRecoveryRpcOperation::ApplyVersions { versions, opts })
|
||||
.await
|
||||
@@ -4448,7 +4381,6 @@ impl DiskAPI for RemoteDisk {
|
||||
Ok(_) => Err(Error::other("unexpected recoverable batch delete reply")),
|
||||
Err(error) => Err(error),
|
||||
};
|
||||
log_delete_recovery_batch_result(&self.endpoint, expected_len, started, &result);
|
||||
match result {
|
||||
Ok(errors) => errors,
|
||||
Err(error) => vec![Some(error); expected_len],
|
||||
@@ -5741,62 +5673,6 @@ mod tests {
|
||||
assert!(decode_delete_recovery_batch_errors(vec![Some(u32::MAX)], 1).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn recoverable_batch_delete_diagnostics_preserve_context_and_redact_error_details() {
|
||||
let logs = CapturedLogs::default();
|
||||
let subscriber = Registry::default().with(
|
||||
tracing_subscriber::fmt::layer()
|
||||
.with_writer(logs.clone())
|
||||
.with_ansi(false)
|
||||
.without_time()
|
||||
.json()
|
||||
.flatten_event(true)
|
||||
.with_span_list(true),
|
||||
);
|
||||
let _guard = tracing::subscriber::set_default(subscriber);
|
||||
let _callsite_pin = crate::test_tracing::pin_callsite_interest_for_test();
|
||||
let endpoint = Endpoint {
|
||||
url: url::Url::parse("http://127.0.0.1:9000/private-disk-path").expect("valid diagnostic endpoint"),
|
||||
is_local: false,
|
||||
pool_idx: 1,
|
||||
set_idx: 2,
|
||||
disk_idx: 3,
|
||||
};
|
||||
let rpc_failure = Err(DiskError::from(Status::deadline_exceeded("private-error-detail")));
|
||||
let group_failures = Ok(vec![None, Some(DiskError::FileNotFound), Some(DiskError::FileNotFound)]);
|
||||
tracing::info_span!("request-span", request_id = "req-batch-delete-diagnostics").in_scope(|| {
|
||||
log_delete_recovery_batch_result(&endpoint, 1000, Instant::now(), &rpc_failure);
|
||||
log_delete_recovery_batch_result(&endpoint, 3, Instant::now(), &group_failures);
|
||||
});
|
||||
let entries = logs.lines();
|
||||
assert_eq!(entries.len(), 2, "one summary per disk batch, never per object");
|
||||
assert_eq!(entries[0]["failure_scope"], "rpc_or_reply");
|
||||
assert_eq!(entries[0]["failed_group_count"], 1000);
|
||||
assert_eq!(entries[0]["rpc_status"], "Some(DeadlineExceeded)");
|
||||
assert_eq!(entries[1]["failure_scope"], "object_group");
|
||||
assert_eq!(entries[1]["failed_group_count"], 2);
|
||||
for entry in entries {
|
||||
assert_eq!(entry["pool_index"], 1);
|
||||
assert_eq!(entry["set_index"], 2);
|
||||
assert_eq!(entry["disk_index"], 3);
|
||||
assert!(
|
||||
entry["spans"]
|
||||
.as_array()
|
||||
.expect("request spans recorded")
|
||||
.iter()
|
||||
.any(|span| { span["request_id"] == "req-batch-delete-diagnostics" })
|
||||
);
|
||||
let text = entry.to_string();
|
||||
assert!(!text.contains("private-error-detail"));
|
||||
assert!(!text.contains("private-disk-path"));
|
||||
}
|
||||
assert!(rpc_failure.is_err(), "diagnostics borrow and preserve the original failure");
|
||||
assert_eq!(
|
||||
group_failures,
|
||||
Ok(vec![None, Some(DiskError::FileNotFound), Some(DiskError::FileNotFound)])
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn delete_recovery_inspection_rejects_wrong_transaction_object_and_protocol() {
|
||||
let transaction = Uuid::new_v4();
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -68,92 +68,6 @@ use super::super::{
|
||||
use super::bitrot_self_verify::{BitrotSelfVerifyTarget, drop_failed_writer_disks, verify_written_bitrot_shards};
|
||||
use super::delete_recovery::DeleteRecoveryContext;
|
||||
|
||||
const EVENT_SET_DISK_DELETE_OBJECTS_STAGE: &str = "set_disk_delete_objects_stage";
|
||||
const DELETE_OBJECTS_STAGE_SLOW_MS: u128 = 1_000;
|
||||
|
||||
/// Counts use the stage's actual inputs, not failures inherited from an earlier stage.
|
||||
struct DeleteObjectsStageSummary {
|
||||
stage: &'static str,
|
||||
started: Instant,
|
||||
attempted_object_count: usize,
|
||||
attempted_group_count: usize,
|
||||
fence_lost: bool,
|
||||
failed_object_count: usize,
|
||||
failed_group_count: usize,
|
||||
disk_error_counts: std::collections::BTreeMap<u32, usize>,
|
||||
}
|
||||
|
||||
impl DeleteObjectsStageSummary {
|
||||
fn new(stage: &'static str, attempted_object_count: usize, attempted_group_count: usize) -> Self {
|
||||
Self {
|
||||
stage,
|
||||
started: Instant::now(),
|
||||
attempted_object_count,
|
||||
attempted_group_count,
|
||||
fence_lost: false,
|
||||
failed_object_count: 0,
|
||||
failed_group_count: 0,
|
||||
disk_error_counts: Default::default(),
|
||||
}
|
||||
}
|
||||
|
||||
fn record_disk_failure(&mut self, error: &DiskError, object_count: usize) {
|
||||
self.failed_object_count += object_count;
|
||||
self.failed_group_count += 1;
|
||||
// Wire codes preserve typed disk causes without exposing error messages or paths.
|
||||
*self.disk_error_counts.entry(error.to_u32()).or_default() += 1;
|
||||
}
|
||||
|
||||
fn log(self, pool_index: usize, set_index: usize, disk_index: Option<usize>) {
|
||||
let elapsed_ms = self.started.elapsed().as_millis();
|
||||
let state = if self.fence_lost { "fence_lost" } else { "completed" };
|
||||
let result = if self.fence_lost || self.failed_object_count > 0 {
|
||||
"failed"
|
||||
} else {
|
||||
"ok"
|
||||
};
|
||||
if self.fence_lost || self.failed_object_count > 0 || elapsed_ms >= DELETE_OBJECTS_STAGE_SLOW_MS {
|
||||
warn!(
|
||||
event = EVENT_SET_DISK_DELETE_OBJECTS_STAGE,
|
||||
component = LOG_COMPONENT_ECSTORE,
|
||||
subsystem = LOG_SUBSYSTEM_SET_DISK,
|
||||
state,
|
||||
result,
|
||||
stage = self.stage,
|
||||
pool_index,
|
||||
set_index,
|
||||
disk_index,
|
||||
attempted_object_count = self.attempted_object_count,
|
||||
attempted_group_count = self.attempted_group_count,
|
||||
failed_object_count = self.failed_object_count,
|
||||
failed_group_count = self.failed_group_count,
|
||||
disk_error_counts = ?self.disk_error_counts,
|
||||
elapsed_ms,
|
||||
"Batch delete stage completed"
|
||||
);
|
||||
} else {
|
||||
debug!(
|
||||
event = EVENT_SET_DISK_DELETE_OBJECTS_STAGE,
|
||||
component = LOG_COMPONENT_ECSTORE,
|
||||
subsystem = LOG_SUBSYSTEM_SET_DISK,
|
||||
state,
|
||||
result,
|
||||
stage = self.stage,
|
||||
pool_index,
|
||||
set_index,
|
||||
disk_index,
|
||||
attempted_object_count = self.attempted_object_count,
|
||||
attempted_group_count = self.attempted_group_count,
|
||||
failed_object_count = self.failed_object_count,
|
||||
failed_group_count = self.failed_group_count,
|
||||
disk_error_counts = ?self.disk_error_counts,
|
||||
elapsed_ms,
|
||||
"Batch delete stage completed"
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Default)]
|
||||
struct DeleteCommitContext<'a> {
|
||||
incarnation: Option<Uuid>,
|
||||
@@ -10042,7 +9956,6 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
|
||||
let dist_erasure = self.ctx.is_dist_erasure().await;
|
||||
let mut dist_batch_lock_ids = vec![Vec::new(); self.lockers.len()];
|
||||
|
||||
let mut raw_lock_summary = DeleteObjectsStageSummary::new("raw_acquire", objects.len(), unique_lock_count);
|
||||
if opts.no_lock {
|
||||
locked_objects = unique_objects;
|
||||
} else if dist_erasure {
|
||||
@@ -10082,13 +9995,8 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
|
||||
for (i, lock_name) in lock_names.iter().enumerate() {
|
||||
if let Some(err) = failed_map.get(&(bucket.to_string(), lock_name.to_string())) {
|
||||
del_errs[i] = Some(Error::other(err.to_string()));
|
||||
raw_lock_summary.failed_object_count += 1;
|
||||
}
|
||||
}
|
||||
if !opts.no_lock {
|
||||
raw_lock_summary.failed_group_count = failed_map.len();
|
||||
raw_lock_summary.log(self.pool_index, self.set_index, None);
|
||||
}
|
||||
|
||||
let outer_namespace_lock_fence = opts.namespace_lock_fence.clone();
|
||||
let mut transaction_guards = std::collections::BTreeMap::new();
|
||||
@@ -10096,18 +10004,12 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
|
||||
// Every raw batch lock is acquired first. Canonical sorting also
|
||||
// gives aliases of one disk key exactly one transaction owner.
|
||||
let keys: std::collections::BTreeSet<_> = locked_objects.iter().cloned().collect();
|
||||
// Batch unresolved keys per peer under one total acquisition budget.
|
||||
// Confirmed cleanup precedes retries; each key keeps its own quorum
|
||||
// and failure result at the set layer.
|
||||
// Bulk delete is the only multi-key transaction-lock holder; one
|
||||
// batched lock round replaces one quorum round-trip per key and is
|
||||
// deadlock-free by construction (a single `lock_batch` RPC per lock
|
||||
// client, global (shard, key) order at each server). Per-key
|
||||
// failures keep the set layer's per-key error reporting.
|
||||
let key_names: Vec<&str> = keys.iter().map(|key| key.as_str()).collect();
|
||||
let mut transaction_summary = DeleteObjectsStageSummary::new(
|
||||
"transaction_acquire",
|
||||
lock_names
|
||||
.iter()
|
||||
.filter(|name| locked_objects.contains(name.as_ref()))
|
||||
.count(),
|
||||
key_names.len(),
|
||||
);
|
||||
let acquired = self
|
||||
.acquire_delete_transaction_locks_batch(bucket, &key_names, super::super::get_lock_acquire_timeout())
|
||||
.await;
|
||||
@@ -10118,17 +10020,14 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
|
||||
transaction_guards.insert(key, guard);
|
||||
}
|
||||
Err(error) => {
|
||||
transaction_summary.failed_group_count += 1;
|
||||
for (index, lock_name) in lock_names.iter().enumerate() {
|
||||
if lock_name.as_ref() == key {
|
||||
del_errs[index] = Some(error.clone());
|
||||
transaction_summary.failed_object_count += 1;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
transaction_summary.log(self.pool_index, self.set_index, None);
|
||||
}
|
||||
let transaction_guards = Arc::new(transaction_guards);
|
||||
let current_generations = opts.conditional_delete_generations.as_deref();
|
||||
@@ -10586,18 +10485,11 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
|
||||
.as_ref()
|
||||
.is_some_and(NamespaceLockFence::is_lock_lost)
|
||||
{
|
||||
let mut fence_summary = DeleteObjectsStageSummary::new(
|
||||
"fence_before_prepare",
|
||||
del_errs.iter().filter(|error| error.is_none()).count(),
|
||||
0,
|
||||
);
|
||||
fence_summary.fence_lost = true;
|
||||
if dist_erasure {
|
||||
self.release_dist_delete_object_locks_batch(dist_batch_lock_ids).await;
|
||||
}
|
||||
for (index, object) in objects.iter().enumerate() {
|
||||
if del_errs[index].is_none() {
|
||||
fence_summary.failed_object_count += 1;
|
||||
del_errs[index] = Some(Error::NamespaceLockQuorumUnavailable {
|
||||
mode: "delete_objects_commit",
|
||||
bucket: bucket.to_string(),
|
||||
@@ -10607,7 +10499,6 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
|
||||
});
|
||||
}
|
||||
}
|
||||
fence_summary.log(self.pool_index, self.set_index, None);
|
||||
return (del_objects, del_errs, accounting);
|
||||
}
|
||||
|
||||
@@ -10674,9 +10565,8 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
|
||||
|
||||
// let mut errors = Vec::with_capacity(disks.len());
|
||||
|
||||
for (disk_index, disk) in disks.iter().enumerate() {
|
||||
for disk in disks.iter() {
|
||||
let disk_namespace_owner = namespace_owner.clone();
|
||||
let group_inputs = &vers;
|
||||
let vers = vers.clone();
|
||||
let namespace_lock_fence = opts.namespace_lock_fence.clone();
|
||||
let lock_lost_during_commit = Arc::clone(&lock_lost_during_commit);
|
||||
@@ -10685,12 +10575,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
|
||||
lock_lost_during_commit.store(true, std::sync::atomic::Ordering::Release);
|
||||
return (0..vers.len()).map(|_| Some(DiskError::DiskOngoingReq)).collect();
|
||||
}
|
||||
let mut prepare_summary = DeleteObjectsStageSummary::new(
|
||||
"prepare",
|
||||
group_inputs.iter().map(|group| group.versions.len()).sum(),
|
||||
group_inputs.len(),
|
||||
);
|
||||
let errors = if let Some(disk) = disk {
|
||||
if let Some(disk) = disk {
|
||||
let options = DeleteOptions {
|
||||
old_data_dir: Some(rollback_dir),
|
||||
delete_recovery_bucket_incarnation: opts.expected_bucket_incarnation_id,
|
||||
@@ -10758,28 +10643,14 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
|
||||
errs.push(Some(DiskError::DiskNotFound));
|
||||
}
|
||||
errs
|
||||
};
|
||||
for (error, group) in errors.iter().zip(group_inputs) {
|
||||
if let Some(error) = error {
|
||||
prepare_summary.record_disk_failure(error, group.versions.len());
|
||||
}
|
||||
}
|
||||
prepare_summary.log(self.pool_index, self.set_index, Some(disk_index));
|
||||
errors
|
||||
});
|
||||
}
|
||||
|
||||
let results = join_all(futures).await;
|
||||
if transaction_guards.values().any(|guard| guard.is_lock_lost()) {
|
||||
let mut fence_summary = DeleteObjectsStageSummary::new(
|
||||
"transaction_fence_after_prepare",
|
||||
del_errs.iter().filter(|error| error.is_none()).count(),
|
||||
0,
|
||||
);
|
||||
fence_summary.fence_lost = true;
|
||||
for (index, object) in objects.iter().enumerate() {
|
||||
if del_errs[index].is_none() {
|
||||
fence_summary.failed_object_count += 1;
|
||||
accounting[index] = None;
|
||||
del_errs[index] = Some(StorageError::NamespaceLockQuorumUnavailable {
|
||||
mode: "delete_transaction",
|
||||
@@ -10790,7 +10661,6 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
|
||||
});
|
||||
}
|
||||
}
|
||||
fence_summary.log(self.pool_index, self.set_index, None);
|
||||
if dist_erasure {
|
||||
self.release_dist_delete_object_locks_batch(dist_batch_lock_ids).await;
|
||||
}
|
||||
@@ -10857,15 +10727,8 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
|
||||
}
|
||||
|
||||
if lock_lost_during_commit.load(std::sync::atomic::Ordering::Acquire) {
|
||||
let mut fence_summary = DeleteObjectsStageSummary::new(
|
||||
"fence_during_prepare",
|
||||
del_errs.iter().filter(|error| error.is_none()).count(),
|
||||
0,
|
||||
);
|
||||
fence_summary.fence_lost = true;
|
||||
for (index, object) in objects.iter().enumerate() {
|
||||
if del_errs[index].is_none() {
|
||||
fence_summary.failed_object_count += 1;
|
||||
del_errs[index] = Some(Error::NamespaceLockQuorumUnavailable {
|
||||
mode: "delete_objects_commit",
|
||||
bucket: bucket.to_string(),
|
||||
@@ -10875,7 +10738,6 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
|
||||
});
|
||||
}
|
||||
}
|
||||
fence_summary.log(self.pool_index, self.set_index, None);
|
||||
}
|
||||
|
||||
self.record_capacity_scope_if_needed(opts.capacity_scope_token, &disks);
|
||||
@@ -10894,11 +10756,6 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
|
||||
// Mark all attempted groups before starting publication. A lost
|
||||
// response can hide a persisted decision and must never roll back.
|
||||
let groups = groups.collect::<Vec<_>>();
|
||||
let mut receipt_summary = DeleteObjectsStageSummary::new(
|
||||
"receipt",
|
||||
groups.iter().map(|index| vers[*index].versions.len()).sum(),
|
||||
groups.len(),
|
||||
);
|
||||
published_groups.extend(groups.iter().map(|index| vers[*index].name.clone()));
|
||||
// Journal publish fans out per object per drive; keep enough
|
||||
// publications in flight to feed the drives' durable-mutation
|
||||
@@ -10946,13 +10803,11 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
|
||||
for (index, result) in publications {
|
||||
let group = &vers[index];
|
||||
if let Err(error) = result {
|
||||
receipt_summary.record_disk_failure(&error, group.versions.len());
|
||||
for version in &group.versions {
|
||||
del_errs[version.idx] = Some(to_object_err(error.clone().into(), vec![bucket, &group.name]));
|
||||
}
|
||||
}
|
||||
}
|
||||
receipt_summary.log(self.pool_index, self.set_index, None);
|
||||
}
|
||||
let mut rollback_futures = Vec::new();
|
||||
let committed_receipt_indices =
|
||||
@@ -28953,24 +28808,3 @@ mod delete_recovery_inventory_tests {
|
||||
|
||||
#[cfg(all(test, feature = "test-util"))]
|
||||
mod data_movement_prefix_cleanup_tests;
|
||||
|
||||
#[cfg(test)]
|
||||
mod delete_objects_stage_summary_tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn batch_delete_diagnostics_count_groups_and_versions_separately() {
|
||||
let mut summary = DeleteObjectsStageSummary::new("receipt", 6, 3);
|
||||
summary.record_disk_failure(&DiskError::ErasureWriteQuorum, 3);
|
||||
summary.record_disk_failure(&DiskError::ErasureWriteQuorum, 2);
|
||||
|
||||
assert_eq!(summary.attempted_object_count, 6);
|
||||
assert_eq!(summary.attempted_group_count, 3);
|
||||
assert_eq!(summary.failed_object_count, 5);
|
||||
assert_eq!(summary.failed_group_count, 2);
|
||||
assert_eq!(
|
||||
summary.disk_error_counts,
|
||||
std::collections::BTreeMap::from([(DiskError::ErasureWriteQuorum.to_u32(), 2)])
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -2533,7 +2533,6 @@ fn return_batch_delete_lock_error_with_accounting(
|
||||
}
|
||||
|
||||
const EVENT_DETACHED_DELETE_FAILED: &str = "detached_delete_failed";
|
||||
const EVENT_BULK_DELETE_LOCK_ACQUIRE: &str = "bulk_delete_lock_acquire";
|
||||
const LOG_SUBSYSTEM_STORE_DELETE: &str = "store_delete";
|
||||
|
||||
/// A detached delete only fails to join when it panicked or the runtime is
|
||||
@@ -3564,10 +3563,12 @@ impl ECStore {
|
||||
}
|
||||
|
||||
/// Acquire the per-object write locks a bulk delete needs before any
|
||||
/// mutation. Each peer receives one batch of unresolved keys per round;
|
||||
/// retries share the original deadline and require confirmed cleanup of
|
||||
/// partial grants. Outcomes remain independent per key so other keys can
|
||||
/// proceed when a contended key exhausts its budget.
|
||||
/// mutation, one batched lock round for the whole key set. Bulk delete is
|
||||
/// the only multi-key write-lock holder, and the batched round is
|
||||
/// deadlock-free by construction (one `lock_batch` RPC per lock client,
|
||||
/// global key order at each server), so no optimistic wave or serial
|
||||
/// fallback is needed. Outcomes are per key: a locked-out key fails alone
|
||||
/// and the rest of the batch proceeds.
|
||||
async fn acquire_delete_objects_write_locks(
|
||||
&self,
|
||||
bucket: &str,
|
||||
@@ -3579,7 +3580,6 @@ impl ECStore {
|
||||
}
|
||||
|
||||
let object_names = sorted_unique_delete_object_names(objects);
|
||||
let acquire_started = Instant::now();
|
||||
let acquired = self.pools[0].disk_set[0]
|
||||
.acquire_object_write_locks_batch(bucket, &object_names, get_lock_acquire_timeout())
|
||||
.await;
|
||||
@@ -3610,35 +3610,6 @@ impl ECStore {
|
||||
}
|
||||
}
|
||||
}
|
||||
if failures.is_empty() {
|
||||
debug!(
|
||||
event = EVENT_BULK_DELETE_LOCK_ACQUIRE,
|
||||
component = LOG_COMPONENT_ECSTORE,
|
||||
subsystem = LOG_SUBSYSTEM_STORE_DELETE,
|
||||
result = "success",
|
||||
stage = "raw_acquire",
|
||||
requested_objects = objects.len(),
|
||||
unique_keys = object_names.len(),
|
||||
successful_keys = guards.len(),
|
||||
failed_keys = failures.len(),
|
||||
elapsed_ms = acquire_started.elapsed().as_millis(),
|
||||
"Bulk delete namespace lock acquisition completed"
|
||||
);
|
||||
} else {
|
||||
warn!(
|
||||
event = EVENT_BULK_DELETE_LOCK_ACQUIRE,
|
||||
component = LOG_COMPONENT_ECSTORE,
|
||||
subsystem = LOG_SUBSYSTEM_STORE_DELETE,
|
||||
result = "partial_failure",
|
||||
stage = "raw_acquire",
|
||||
requested_objects = objects.len(),
|
||||
unique_keys = object_names.len(),
|
||||
successful_keys = guards.len(),
|
||||
failed_keys = failures.len(),
|
||||
elapsed_ms = acquire_started.elapsed().as_millis(),
|
||||
"Bulk delete namespace lock acquisition failed for some keys"
|
||||
);
|
||||
}
|
||||
opts.no_lock = true;
|
||||
for signal in guards.iter().filter_map(ObjectLockDiagGuard::lock_lost_signal) {
|
||||
opts.add_namespace_lock_lost_signal(signal);
|
||||
@@ -14430,7 +14401,8 @@ mod batch_delete_lock_tests {
|
||||
let second = run(second);
|
||||
|
||||
// Whatever mix of per-key lock outcomes the overlap produces, both
|
||||
// batches must finish within their acquisition and mutation budgets.
|
||||
// batches must finish: the batched round is deadlock-free by
|
||||
// construction (global key order at each lock server).
|
||||
for batch in [first, second] {
|
||||
let (_, errors, _) = tokio::time::timeout(Duration::from_secs(30), batch)
|
||||
.await
|
||||
|
||||
+140
-376
@@ -143,49 +143,6 @@ struct LocalGuardEntry {
|
||||
ttl: Duration,
|
||||
}
|
||||
|
||||
/// The shared acquisition algorithm reports facts before either API formats them.
|
||||
enum LocalAcquireFailure {
|
||||
Timeout,
|
||||
Conflict {
|
||||
current_owner: Arc<str>,
|
||||
current_mode: crate::LockMode,
|
||||
},
|
||||
InvalidResource,
|
||||
}
|
||||
|
||||
impl LocalAcquireFailure {
|
||||
fn into_response(self, request: &LockRequest) -> LockResponse {
|
||||
match self {
|
||||
Self::Timeout => LockResponse::failure("Lock acquisition timeout", request.acquire_timeout),
|
||||
Self::Conflict {
|
||||
current_owner,
|
||||
current_mode,
|
||||
} => LockResponse::failure(
|
||||
format!("Lock conflict: resource held by {current_owner} in {current_mode:?} mode"),
|
||||
Duration::ZERO,
|
||||
),
|
||||
Self::InvalidResource => {
|
||||
LockResponse::failure("Lock id resource does not match the requested resource", Duration::ZERO)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn into_batch_outcome(self, request: &LockRequest) -> super::BatchAcquireOutcome {
|
||||
use super::BatchAcquireOutcome;
|
||||
match self {
|
||||
Self::Timeout => BatchAcquireOutcome::RetryableNoGrant {
|
||||
error: crate::LockError::timeout(request.resource.to_string(), request.acquire_timeout),
|
||||
},
|
||||
Self::Conflict { current_owner, .. } => BatchAcquireOutcome::RetryableNoGrant {
|
||||
error: crate::LockError::already_locked(request.resource.to_string(), current_owner.to_string()),
|
||||
},
|
||||
Self::InvalidResource => BatchAcquireOutcome::TerminalNoGrant {
|
||||
error: crate::LockError::invalid_handle(request.lock_id.to_string()),
|
||||
},
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl LocalGuardEntry {
|
||||
fn new(guard: FastLockGuard, ttl: Duration) -> Self {
|
||||
let acquired_at = SystemTime::now();
|
||||
@@ -214,158 +171,6 @@ impl LocalGuardEntry {
|
||||
}
|
||||
|
||||
impl LocalClient {
|
||||
async fn acquire_lock_until(
|
||||
&self,
|
||||
request: &LockRequest,
|
||||
acquire_deadline: Instant,
|
||||
) -> std::result::Result<LockInfo, LocalAcquireFailure> {
|
||||
let started = Instant::now();
|
||||
self.ensure_reaper();
|
||||
if request.lock_id.resource != request.resource {
|
||||
return Err(LocalAcquireFailure::InvalidResource);
|
||||
}
|
||||
let lock_manager = self.get_lock_manager();
|
||||
let initial_remaining = acquire_deadline.saturating_duration_since(Instant::now());
|
||||
let reclaimed_before_acquire = if initial_remaining.is_zero() {
|
||||
0
|
||||
} else {
|
||||
timeout(initial_remaining, self.reclaim_expired_guards_for_resource(&request.resource))
|
||||
.await
|
||||
.unwrap_or(0)
|
||||
};
|
||||
let reclaim_elapsed = started.elapsed();
|
||||
|
||||
let build_lock_request = |acquire_timeout| match request.lock_type {
|
||||
LockType::Exclusive => crate::ObjectLockRequest::new_write(request.resource.clone(), request.owner.clone())
|
||||
.with_acquire_timeout(acquire_timeout),
|
||||
LockType::Shared => crate::ObjectLockRequest::new_read(request.resource.clone(), request.owner.clone())
|
||||
.with_acquire_timeout(acquire_timeout),
|
||||
};
|
||||
|
||||
let mut retried_after_reclaim = reclaimed_before_acquire > 0;
|
||||
loop {
|
||||
let remaining = acquire_deadline.saturating_duration_since(Instant::now());
|
||||
if remaining.is_zero() {
|
||||
return Err(LocalAcquireFailure::Timeout);
|
||||
}
|
||||
let manager_started = Instant::now();
|
||||
// The backend uses its own relative wait; enforce the caller's
|
||||
// absolute budget across that entire future as well. Dropping an
|
||||
// uncompleted manager acquisition releases its publication ticket.
|
||||
let acquired = match timeout_at(acquire_deadline, lock_manager.acquire_lock(build_lock_request(remaining))).await {
|
||||
Ok(acquired) => acquired,
|
||||
Err(_) => return Err(LocalAcquireFailure::Timeout),
|
||||
};
|
||||
match acquired {
|
||||
Ok(guard) => {
|
||||
let manager_elapsed = manager_started.elapsed();
|
||||
let registration_started = Instant::now();
|
||||
let physical_guard_id = guard.guard_id();
|
||||
let lock_id = request.lock_id.clone();
|
||||
let entry = LocalGuardEntry::new(guard, request.ttl);
|
||||
let acquired_at = entry.acquired_at;
|
||||
let expires_at = entry.expires_at;
|
||||
|
||||
let (replaced_entry, registration_wait, registration_mutation, registry_entries, registry_capacity) = {
|
||||
let shard = self.get_shard(&lock_id);
|
||||
let Ok(mut guards) = timeout_at(acquire_deadline, shard.write()).await else {
|
||||
return Err(LocalAcquireFailure::Timeout);
|
||||
};
|
||||
// A ready table cannot publish a grant after the peer
|
||||
// budget expired while its acquisition task was idle.
|
||||
if Instant::now() >= acquire_deadline {
|
||||
drop(guards);
|
||||
return Err(LocalAcquireFailure::Timeout);
|
||||
}
|
||||
let registration_wait = registration_started.elapsed();
|
||||
let mutation_started = Instant::now();
|
||||
let replaced_entry = guards.insert(lock_id.clone(), entry);
|
||||
(
|
||||
replaced_entry,
|
||||
registration_wait,
|
||||
mutation_started.elapsed(),
|
||||
guards.len(),
|
||||
guards.capacity(),
|
||||
)
|
||||
};
|
||||
// Repeated shared requests can replace an entry. Releasing
|
||||
// its physical guard must not reenter locks under the table.
|
||||
drop(replaced_entry);
|
||||
|
||||
let registration_elapsed = registration_started.elapsed();
|
||||
if started.elapsed() >= Duration::from_millis(100) {
|
||||
tracing::trace!(
|
||||
target: "rustfs_lock::namespace_owner",
|
||||
event = EVENT_NAMESPACE_LOCK_ACQUIRE,
|
||||
component = LOG_COMPONENT_LOCK,
|
||||
subsystem = LOG_SUBSYSTEM_NAMESPACE,
|
||||
state = "local_completed",
|
||||
resource = %request.resource,
|
||||
lock_id = %lock_id,
|
||||
mode = ?request.lock_type,
|
||||
physical_guard_id,
|
||||
reclaim_ms = reclaim_elapsed.as_millis(),
|
||||
manager_ms = manager_elapsed.as_millis(),
|
||||
registration_ms = registration_elapsed.as_millis(),
|
||||
registration_wait_ms = registration_wait.as_millis(),
|
||||
registration_mutation_ms = registration_mutation.as_millis(),
|
||||
registry_entries,
|
||||
registry_capacity,
|
||||
acquire_ms = started.elapsed().as_millis(),
|
||||
"Namespace local lock acquisition completed slowly"
|
||||
);
|
||||
}
|
||||
|
||||
let lock_info = LockInfo {
|
||||
id: lock_id,
|
||||
resource: request.resource.clone(),
|
||||
lock_type: request.lock_type,
|
||||
status: crate::types::LockStatus::Acquired,
|
||||
owner: request.owner.clone(),
|
||||
acquired_at,
|
||||
expires_at,
|
||||
last_refreshed: acquired_at,
|
||||
metadata: request.metadata.clone(),
|
||||
priority: request.priority,
|
||||
wait_start_time: None,
|
||||
};
|
||||
return Ok(lock_info);
|
||||
}
|
||||
Err(crate::fast_lock::LockResult::Timeout) => {
|
||||
if !retried_after_reclaim {
|
||||
let remaining = acquire_deadline.saturating_duration_since(Instant::now());
|
||||
if !remaining.is_zero() {
|
||||
let _ = timeout(remaining, self.reclaim_expired_guards_for_resource(&request.resource)).await;
|
||||
}
|
||||
retried_after_reclaim = true;
|
||||
continue;
|
||||
}
|
||||
return Err(LocalAcquireFailure::Timeout);
|
||||
}
|
||||
Err(crate::fast_lock::LockResult::Conflict {
|
||||
current_owner,
|
||||
current_mode,
|
||||
}) => {
|
||||
if !retried_after_reclaim {
|
||||
let remaining = acquire_deadline.saturating_duration_since(Instant::now());
|
||||
if !remaining.is_zero() {
|
||||
let _ = timeout(remaining, self.reclaim_expired_guards_for_resource(&request.resource)).await;
|
||||
}
|
||||
retried_after_reclaim = true;
|
||||
continue;
|
||||
}
|
||||
return Err(LocalAcquireFailure::Conflict {
|
||||
current_owner,
|
||||
current_mode,
|
||||
});
|
||||
}
|
||||
Err(crate::fast_lock::LockResult::Acquired) => {
|
||||
unreachable!("Acquired should not be an error")
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Create new local client with default shard count
|
||||
pub fn new() -> Self {
|
||||
Self::with_shard_count(DEFAULT_GUARD_SHARD_COUNT)
|
||||
@@ -515,29 +320,149 @@ impl Default for LocalClient {
|
||||
impl LockClient for LocalClient {
|
||||
async fn acquire_lock(&self, request: &LockRequest) -> Result<LockResponse> {
|
||||
let started = Instant::now();
|
||||
let deadline = started.checked_add(request.acquire_timeout).unwrap_or(started);
|
||||
Ok(match self.acquire_lock_until(request, deadline).await {
|
||||
Ok(info) => LockResponse::success(info, Duration::ZERO),
|
||||
Err(failure) => failure.into_response(request),
|
||||
})
|
||||
}
|
||||
self.ensure_reaper();
|
||||
if request.lock_id.resource != request.resource {
|
||||
return Ok(LockResponse::failure(
|
||||
"Lock id resource does not match the requested resource",
|
||||
Duration::ZERO,
|
||||
));
|
||||
}
|
||||
let acquire_deadline = Instant::now()
|
||||
.checked_add(request.acquire_timeout)
|
||||
.unwrap_or_else(Instant::now);
|
||||
let lock_manager = self.get_lock_manager();
|
||||
let initial_remaining = acquire_deadline.saturating_duration_since(Instant::now());
|
||||
let reclaimed_before_acquire = if initial_remaining.is_zero() {
|
||||
0
|
||||
} else {
|
||||
timeout(initial_remaining, self.reclaim_expired_guards_for_resource(&request.resource))
|
||||
.await
|
||||
.unwrap_or(0)
|
||||
};
|
||||
let reclaim_elapsed = started.elapsed();
|
||||
|
||||
async fn acquire_locks_batch_until(&self, requests: &[LockRequest], deadline: Instant) -> Vec<super::BatchAcquireOutcome> {
|
||||
// Every item shares the entry time, including items first polled later.
|
||||
let started = Instant::now();
|
||||
futures::future::join_all(requests.iter().map(|request| {
|
||||
let item_deadline = deadline.min(started.checked_add(request.acquire_timeout).unwrap_or(started));
|
||||
async move {
|
||||
match self.acquire_lock_until(request, item_deadline).await {
|
||||
Ok(info) => super::BatchAcquireOutcome::Granted {
|
||||
lock_info: Some(Box::new(info)),
|
||||
wait_time: Duration::ZERO,
|
||||
},
|
||||
Err(failure) => failure.into_batch_outcome(request),
|
||||
let build_lock_request = |acquire_timeout| match request.lock_type {
|
||||
LockType::Exclusive => crate::ObjectLockRequest::new_write(request.resource.clone(), request.owner.clone())
|
||||
.with_acquire_timeout(acquire_timeout),
|
||||
LockType::Shared => crate::ObjectLockRequest::new_read(request.resource.clone(), request.owner.clone())
|
||||
.with_acquire_timeout(acquire_timeout),
|
||||
};
|
||||
|
||||
let mut retried_after_reclaim = reclaimed_before_acquire > 0;
|
||||
loop {
|
||||
let remaining = acquire_deadline.saturating_duration_since(Instant::now());
|
||||
if remaining.is_zero() {
|
||||
return Ok(LockResponse::failure("Lock acquisition timeout", request.acquire_timeout));
|
||||
}
|
||||
let manager_started = Instant::now();
|
||||
match lock_manager.acquire_lock(build_lock_request(remaining)).await {
|
||||
Ok(guard) => {
|
||||
let manager_elapsed = manager_started.elapsed();
|
||||
let registration_started = Instant::now();
|
||||
let physical_guard_id = guard.guard_id();
|
||||
let lock_id = request.lock_id.clone();
|
||||
let entry = LocalGuardEntry::new(guard, request.ttl);
|
||||
let acquired_at = entry.acquired_at;
|
||||
let expires_at = entry.expires_at;
|
||||
|
||||
let (replaced_entry, registration_wait, registration_mutation, registry_entries, registry_capacity) = {
|
||||
let shard = self.get_shard(&lock_id);
|
||||
let Ok(mut guards) = timeout_at(acquire_deadline, shard.write()).await else {
|
||||
return Ok(LockResponse::failure("Lock acquisition timeout", request.acquire_timeout));
|
||||
};
|
||||
// A ready table cannot publish a grant after the peer
|
||||
// budget expired while its acquisition task was idle.
|
||||
if Instant::now() >= acquire_deadline {
|
||||
drop(guards);
|
||||
return Ok(LockResponse::failure("Lock acquisition timeout", request.acquire_timeout));
|
||||
}
|
||||
let registration_wait = registration_started.elapsed();
|
||||
let mutation_started = Instant::now();
|
||||
let replaced_entry = guards.insert(lock_id.clone(), entry);
|
||||
(
|
||||
replaced_entry,
|
||||
registration_wait,
|
||||
mutation_started.elapsed(),
|
||||
guards.len(),
|
||||
guards.capacity(),
|
||||
)
|
||||
};
|
||||
// Repeated shared requests can replace an entry. Releasing
|
||||
// its physical guard must not reenter locks under the table.
|
||||
drop(replaced_entry);
|
||||
|
||||
let registration_elapsed = registration_started.elapsed();
|
||||
if started.elapsed() >= Duration::from_millis(100) {
|
||||
tracing::trace!(
|
||||
target: "rustfs_lock::namespace_owner",
|
||||
event = EVENT_NAMESPACE_LOCK_ACQUIRE,
|
||||
component = LOG_COMPONENT_LOCK,
|
||||
subsystem = LOG_SUBSYSTEM_NAMESPACE,
|
||||
state = "local_completed",
|
||||
resource = %request.resource,
|
||||
lock_id = %lock_id,
|
||||
mode = ?request.lock_type,
|
||||
physical_guard_id,
|
||||
reclaim_ms = reclaim_elapsed.as_millis(),
|
||||
manager_ms = manager_elapsed.as_millis(),
|
||||
registration_ms = registration_elapsed.as_millis(),
|
||||
registration_wait_ms = registration_wait.as_millis(),
|
||||
registration_mutation_ms = registration_mutation.as_millis(),
|
||||
registry_entries,
|
||||
registry_capacity,
|
||||
acquire_ms = started.elapsed().as_millis(),
|
||||
"Namespace local lock acquisition completed slowly"
|
||||
);
|
||||
}
|
||||
|
||||
let lock_info = LockInfo {
|
||||
id: lock_id,
|
||||
resource: request.resource.clone(),
|
||||
lock_type: request.lock_type,
|
||||
status: crate::types::LockStatus::Acquired,
|
||||
owner: request.owner.clone(),
|
||||
acquired_at,
|
||||
expires_at,
|
||||
last_refreshed: acquired_at,
|
||||
metadata: request.metadata.clone(),
|
||||
priority: request.priority,
|
||||
wait_start_time: None,
|
||||
};
|
||||
return Ok(LockResponse::success(lock_info, Duration::ZERO));
|
||||
}
|
||||
Err(crate::fast_lock::LockResult::Timeout) => {
|
||||
if !retried_after_reclaim {
|
||||
let remaining = acquire_deadline.saturating_duration_since(Instant::now());
|
||||
if !remaining.is_zero() {
|
||||
let _ = timeout(remaining, self.reclaim_expired_guards_for_resource(&request.resource)).await;
|
||||
}
|
||||
retried_after_reclaim = true;
|
||||
continue;
|
||||
}
|
||||
return Ok(LockResponse::failure("Lock acquisition timeout", request.acquire_timeout));
|
||||
}
|
||||
Err(crate::fast_lock::LockResult::Conflict {
|
||||
current_owner,
|
||||
current_mode,
|
||||
}) => {
|
||||
if !retried_after_reclaim {
|
||||
let remaining = acquire_deadline.saturating_duration_since(Instant::now());
|
||||
if !remaining.is_zero() {
|
||||
let _ = timeout(remaining, self.reclaim_expired_guards_for_resource(&request.resource)).await;
|
||||
}
|
||||
retried_after_reclaim = true;
|
||||
continue;
|
||||
}
|
||||
return Ok(LockResponse::failure(
|
||||
format!("Lock conflict: resource held by {current_owner} in {current_mode:?} mode"),
|
||||
Duration::ZERO,
|
||||
));
|
||||
}
|
||||
Err(crate::fast_lock::LockResult::Acquired) => {
|
||||
unreachable!("Acquired should not be an error")
|
||||
}
|
||||
}
|
||||
}))
|
||||
.await
|
||||
}
|
||||
}
|
||||
|
||||
async fn release(&self, lock_id: &LockId) -> Result<bool> {
|
||||
@@ -684,167 +609,6 @@ mod tests {
|
||||
use futures::FutureExt;
|
||||
use std::task::Poll;
|
||||
|
||||
#[tokio::test(start_paused = true)]
|
||||
async fn typed_batch_preserves_free_key_grants_beside_a_contended_key() {
|
||||
let client = LocalClient::with_manager(Arc::new(GlobalLockManager::new()));
|
||||
let hot = crate::ObjectKey::new("bucket", "hot");
|
||||
let holder = request(hot.clone(), "holder", Duration::from_secs(30));
|
||||
assert!(client.acquire_lock(&holder).await.expect("hold contended key").success);
|
||||
let requests = [
|
||||
request(crate::ObjectKey::new("bucket", "free"), "batch", Duration::from_secs(30)),
|
||||
request(hot, "batch", Duration::from_secs(30)),
|
||||
];
|
||||
let outcomes = client
|
||||
.acquire_locks_batch_until(&requests, Instant::now() + Duration::from_secs(1))
|
||||
.await;
|
||||
assert_eq!(outcomes.len(), 2);
|
||||
assert!(matches!(outcomes[0], super::super::BatchAcquireOutcome::Granted { .. }));
|
||||
assert!(matches!(outcomes[1], super::super::BatchAcquireOutcome::RetryableNoGrant { .. }));
|
||||
assert!(
|
||||
client
|
||||
.check_status(&requests[0].lock_id)
|
||||
.await
|
||||
.expect("check free grant")
|
||||
.is_some()
|
||||
);
|
||||
assert!(
|
||||
client
|
||||
.check_status(&requests[1].lock_id)
|
||||
.await
|
||||
.expect("check rejected attempt")
|
||||
.is_none()
|
||||
);
|
||||
assert!(client.release(&holder.lock_id).await.expect("release holder"));
|
||||
assert!(client.release(&requests[0].lock_id).await.expect("release free key"));
|
||||
assert!(client.list_lock_leases().await.is_empty());
|
||||
}
|
||||
|
||||
#[tokio::test(start_paused = true)]
|
||||
async fn typed_batch_manager_wait_obeys_absolute_deadline_without_releasing_holder() {
|
||||
let manager = Arc::new(GlobalLockManager::new());
|
||||
let client = LocalClient::with_manager(manager.clone());
|
||||
let resource = crate::ObjectKey::new("bucket", "manager-deadline");
|
||||
let holder = manager
|
||||
.acquire_lock(crate::ObjectLockRequest::new_write(resource.clone(), "holder"))
|
||||
.await
|
||||
.expect("hold the physical lock independently of a backend lease");
|
||||
let requests = [request(resource.clone(), "batch", Duration::from_secs(30)).with_acquire_timeout(Duration::from_secs(5))];
|
||||
let started = Instant::now();
|
||||
let deadline = started + Duration::from_millis(100);
|
||||
let outcomes = timeout_at(deadline + Duration::from_millis(1), client.acquire_locks_batch_until(&requests, deadline))
|
||||
.await
|
||||
.expect("the manager wait must finish at the absolute caller deadline");
|
||||
assert!(started.elapsed() <= Duration::from_millis(100));
|
||||
assert!(matches!(
|
||||
outcomes.as_slice(),
|
||||
[super::super::BatchAcquireOutcome::RetryableNoGrant { .. }]
|
||||
));
|
||||
assert!(client.list_lock_leases().await.is_empty(), "no failed attempt is registered");
|
||||
assert_eq!(
|
||||
manager
|
||||
.as_fast_lock_manager()
|
||||
.expect("enabled manager")
|
||||
.get_shard(&resource)
|
||||
.active_guard_count(),
|
||||
1,
|
||||
"the original holder is never released by a timed-out attempt"
|
||||
);
|
||||
drop(holder);
|
||||
let successor = manager
|
||||
.acquire_lock(crate::ObjectLockRequest::new_write(resource.clone(), "successor"))
|
||||
.await
|
||||
.expect("no cancelled manager waiter may take the released lock later");
|
||||
drop(successor);
|
||||
assert_eq!(
|
||||
manager
|
||||
.as_fast_lock_manager()
|
||||
.expect("enabled manager")
|
||||
.get_shard(&resource)
|
||||
.active_guard_count(),
|
||||
0
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test(start_paused = true)]
|
||||
async fn typed_batch_rejects_late_poll_without_resetting_the_caller_deadline() {
|
||||
let client = LocalClient::with_manager(Arc::new(GlobalLockManager::new()));
|
||||
let requests = [
|
||||
request(crate::ObjectKey::new("bucket", "late"), "batch", Duration::from_secs(30))
|
||||
.with_acquire_timeout(Duration::from_secs(5)),
|
||||
];
|
||||
let deadline = Instant::now() + Duration::from_millis(20);
|
||||
let acquire = client.acquire_locks_batch_until(&requests, deadline);
|
||||
tokio::time::advance(Duration::from_millis(21)).await;
|
||||
assert!(matches!(
|
||||
acquire.await.as_slice(),
|
||||
[super::super::BatchAcquireOutcome::RetryableNoGrant {
|
||||
error: crate::LockError::Timeout { .. }
|
||||
}]
|
||||
));
|
||||
assert!(client.list_lock_leases().await.is_empty(), "an expired item must never register a grant");
|
||||
}
|
||||
|
||||
#[tokio::test(start_paused = true)]
|
||||
async fn typed_batch_registration_timeout_drops_the_unpublished_physical_guard() {
|
||||
let manager = Arc::new(GlobalLockManager::new());
|
||||
let client = LocalClient::with_manager(manager.clone());
|
||||
client.reaper_started.store(true, Ordering::Release);
|
||||
let resource = crate::ObjectKey::new("bucket", "typed-registration-timeout");
|
||||
let blocker = manager
|
||||
.acquire_lock(crate::ObjectLockRequest::new_write(resource.clone(), "holder"))
|
||||
.await
|
||||
.expect("hold physical lock until preflight completes");
|
||||
let requests = [request(resource.clone(), "batch", Duration::from_secs(30)).with_acquire_timeout(Duration::from_secs(5))];
|
||||
let deadline = Instant::now() + Duration::from_millis(20);
|
||||
let mut acquire = Box::pin(tokio::task::unconstrained(client.acquire_locks_batch_until(&requests, deadline)));
|
||||
assert!(matches!(futures::poll!(&mut acquire), Poll::Pending));
|
||||
let (release_table, table_owner) = hold_guard_shard(client.get_shard(&requests[0].lock_id).clone());
|
||||
drop(blocker);
|
||||
assert!(matches!(futures::poll!(&mut acquire), Poll::Pending));
|
||||
assert_eq!(
|
||||
manager
|
||||
.as_fast_lock_manager()
|
||||
.expect("enabled manager")
|
||||
.get_shard(&resource)
|
||||
.active_guard_count(),
|
||||
1
|
||||
);
|
||||
tokio::time::advance(Duration::from_millis(21)).await;
|
||||
let result = acquire.await;
|
||||
assert!(matches!(result.as_slice(), [super::super::BatchAcquireOutcome::RetryableNoGrant { .. }]));
|
||||
assert_eq!(
|
||||
manager
|
||||
.as_fast_lock_manager()
|
||||
.expect("enabled manager")
|
||||
.get_shard(&resource)
|
||||
.active_guard_count(),
|
||||
0,
|
||||
"NoGrant must only be returned after the unpublished physical guard is dropped"
|
||||
);
|
||||
release_table.send(()).expect("release table");
|
||||
table_owner.join().expect("table owner finishes");
|
||||
assert!(client.list_lock_leases().await.is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn legacy_acquisition_messages_remain_compatible() {
|
||||
let request = request(crate::ObjectKey::new("bucket", "legacy"), "owner", Duration::from_secs(30));
|
||||
assert_eq!(
|
||||
LocalAcquireFailure::Timeout.into_response(&request).error.as_deref(),
|
||||
Some("Lock acquisition timeout")
|
||||
);
|
||||
assert_eq!(
|
||||
LocalAcquireFailure::Conflict {
|
||||
current_owner: Arc::from("holder"),
|
||||
current_mode: crate::LockMode::Exclusive
|
||||
}
|
||||
.into_response(&request)
|
||||
.error
|
||||
.as_deref(),
|
||||
Some("Lock conflict: resource held by holder in Exclusive mode")
|
||||
);
|
||||
}
|
||||
|
||||
fn request(resource: crate::ObjectKey, owner: &str, ttl: Duration) -> LockRequest {
|
||||
LockRequest::new(resource, LockType::Exclusive, owner)
|
||||
.with_ttl(ttl)
|
||||
|
||||
@@ -15,49 +15,10 @@
|
||||
pub mod local;
|
||||
// pub mod remote;
|
||||
|
||||
use crate::{LockError, LockId, LockInfo, LockLeaseInfo, LockRequest, LockResponse, LockStats, Result};
|
||||
use crate::{LockId, LockInfo, LockLeaseInfo, LockRequest, LockResponse, LockStats, Result};
|
||||
use async_trait::async_trait;
|
||||
use futures::future::join_all;
|
||||
use std::sync::Arc;
|
||||
use std::time::Duration;
|
||||
use tokio::time::Instant;
|
||||
|
||||
/// A batch peer's result for one request, in the original request order.
|
||||
#[derive(Clone, Debug)]
|
||||
pub enum BatchAcquireOutcome {
|
||||
/// The peer confirmed a grant. Missing legacy metadata uses the requested ID.
|
||||
Granted {
|
||||
lock_info: Option<Box<LockInfo>>,
|
||||
wait_time: Duration,
|
||||
},
|
||||
/// This attempt cannot grant later, and another attempt may succeed.
|
||||
RetryableNoGrant { error: LockError },
|
||||
/// This attempt cannot grant later, and this peer must not be retried.
|
||||
TerminalNoGrant { error: LockError },
|
||||
/// A grant may exist or arrive later. Stop using this peer for this key.
|
||||
Ambiguous {
|
||||
error: LockError,
|
||||
/// The client already retains exact-ID cleanup until the attempt settles.
|
||||
/// Otherwise the coordinator must retain uncertain cleanup ownership.
|
||||
cleanup_owned_by_client: bool,
|
||||
},
|
||||
}
|
||||
|
||||
impl BatchAcquireOutcome {
|
||||
fn from_legacy_response(response: LockResponse) -> Self {
|
||||
if response.success {
|
||||
Self::Granted {
|
||||
lock_info: response.lock_info.map(Box::new),
|
||||
wait_time: response.wait_time,
|
||||
}
|
||||
} else {
|
||||
Self::Ambiguous {
|
||||
error: LockError::internal(response.error.unwrap_or_else(|| "Legacy batch acquisition failed".to_owned())),
|
||||
cleanup_owned_by_client: false,
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Outcome of cleaning up a lock whose acquisition has already completed.
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
|
||||
@@ -84,45 +45,6 @@ pub trait LockClient: Send + Sync + std::fmt::Debug {
|
||||
.collect::<Result<Vec<_>>>()?)
|
||||
}
|
||||
|
||||
/// Acquire a batch within the caller's absolute deadline, preserving per-key outcomes.
|
||||
///
|
||||
/// Legacy clients cannot prove that failed or missing responses did not grant.
|
||||
/// Their calls retain completion ownership in the coordinator if the caller
|
||||
/// stops waiting. Implementations may narrow each request's backend wait;
|
||||
/// they must never reset the deadline after connection or queue delays.
|
||||
async fn acquire_locks_batch_until(&self, requests: &[LockRequest], deadline: Instant) -> Vec<BatchAcquireOutcome> {
|
||||
if Instant::now() >= deadline {
|
||||
return requests
|
||||
.iter()
|
||||
.map(|request| BatchAcquireOutcome::RetryableNoGrant {
|
||||
error: LockError::timeout(request.resource.to_string(), request.acquire_timeout),
|
||||
})
|
||||
.collect();
|
||||
}
|
||||
match self.acquire_locks_batch(requests).await {
|
||||
Ok(responses) => {
|
||||
let mut responses = responses.into_iter();
|
||||
requests
|
||||
.iter()
|
||||
.map(|_| match responses.next() {
|
||||
Some(response) => BatchAcquireOutcome::from_legacy_response(response),
|
||||
None => BatchAcquireOutcome::Ambiguous {
|
||||
error: LockError::internal("Legacy peer returned an incomplete batch response"),
|
||||
cleanup_owned_by_client: false,
|
||||
},
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
Err(error) => requests
|
||||
.iter()
|
||||
.map(|_| BatchAcquireOutcome::Ambiguous {
|
||||
error: error.clone(),
|
||||
cleanup_owned_by_client: false,
|
||||
})
|
||||
.collect(),
|
||||
}
|
||||
}
|
||||
|
||||
/// Release lock
|
||||
async fn release(&self, lock_id: &LockId) -> Result<bool>;
|
||||
|
||||
@@ -200,39 +122,3 @@ impl ClientFactory {
|
||||
// Arc::new(remote::RemoteClient::new(endpoint))
|
||||
// }
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn legacy_failure_text_never_proves_no_grant() {
|
||||
for message in [
|
||||
"Lock acquisition timeout",
|
||||
"Lock conflict: resource held by holder",
|
||||
"Remote lock RPC timed out",
|
||||
] {
|
||||
assert!(matches!(
|
||||
BatchAcquireOutcome::from_legacy_response(LockResponse::failure(message, Duration::ZERO)),
|
||||
BatchAcquireOutcome::Ambiguous {
|
||||
cleanup_owned_by_client: false,
|
||||
..
|
||||
}
|
||||
));
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn legacy_success_without_metadata_remains_a_grant() {
|
||||
assert!(matches!(
|
||||
BatchAcquireOutcome::from_legacy_response(LockResponse {
|
||||
success: true,
|
||||
lock_info: None,
|
||||
error: None,
|
||||
wait_time: Duration::ZERO,
|
||||
position_in_queue: None,
|
||||
}),
|
||||
BatchAcquireOutcome::Granted { lock_info: None, .. }
|
||||
));
|
||||
}
|
||||
}
|
||||
|
||||
+105
-710
@@ -14,7 +14,7 @@
|
||||
|
||||
use crate::{
|
||||
ObjectKey,
|
||||
client::{BatchAcquireOutcome, LockCleanupOutcome, LockClient},
|
||||
client::{LockCleanupOutcome, LockClient},
|
||||
error::{LockError, Result},
|
||||
types::{LockId, LockInfo, LockRequest, LockResponse, LockStatus, LockType},
|
||||
};
|
||||
@@ -33,7 +33,7 @@ use std::time::Duration;
|
||||
use tokio::sync::Notify;
|
||||
use tokio::task::{JoinHandle, JoinSet};
|
||||
use tokio::time::Instant;
|
||||
use tracing::{Instrument as _, debug, warn};
|
||||
use tracing::{debug, warn};
|
||||
use uuid::Uuid;
|
||||
|
||||
const EVENT_DISTRIBUTED_LOCK_CLEANUP: &str = "distributed_lock_cleanup";
|
||||
@@ -68,121 +68,6 @@ enum LockAcquireFailureKind {
|
||||
UnrecoverableQuorum,
|
||||
}
|
||||
|
||||
#[derive(PartialEq, Eq)]
|
||||
struct BatchLockPeerDiagnostics {
|
||||
peer_index: usize,
|
||||
granted_keys: usize,
|
||||
retryable_no_grant_keys: usize,
|
||||
terminal_no_grant_keys: usize,
|
||||
ambiguous_keys: usize,
|
||||
failed_response_keys: usize,
|
||||
missing_response_keys: usize,
|
||||
call_failed_keys: usize,
|
||||
extra_responses: usize,
|
||||
call_error_kind: Option<&'static str>,
|
||||
elapsed_ms: u128,
|
||||
}
|
||||
|
||||
impl std::fmt::Debug for BatchLockPeerDiagnostics {
|
||||
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||
formatter
|
||||
.debug_struct("BatchLockPeerDiagnostics")
|
||||
.field("peer_index", &self.peer_index)
|
||||
.field("granted_keys", &self.granted_keys)
|
||||
.field("retryable_no_grant_keys", &self.retryable_no_grant_keys)
|
||||
.field("terminal_no_grant_keys", &self.terminal_no_grant_keys)
|
||||
.field("ambiguous_keys", &self.ambiguous_keys)
|
||||
.field("failed_response_keys", &self.failed_response_keys)
|
||||
.field("missing_response_keys", &self.missing_response_keys)
|
||||
.field("call_failed_keys", &self.call_failed_keys)
|
||||
.field("extra_responses", &self.extra_responses)
|
||||
.field("call_error_kind", &self.call_error_kind)
|
||||
.field("elapsed_ms", &self.elapsed_ms)
|
||||
.finish()
|
||||
}
|
||||
}
|
||||
|
||||
impl BatchLockPeerDiagnostics {
|
||||
fn from_outcomes(peer_index: usize, key_count: usize, outcomes: &[BatchAcquireOutcome], elapsed: Duration) -> Self {
|
||||
let mut diagnostics = Self {
|
||||
peer_index,
|
||||
granted_keys: 0,
|
||||
retryable_no_grant_keys: 0,
|
||||
terminal_no_grant_keys: 0,
|
||||
ambiguous_keys: 0,
|
||||
failed_response_keys: 0,
|
||||
missing_response_keys: key_count.saturating_sub(outcomes.len()),
|
||||
call_failed_keys: 0,
|
||||
extra_responses: outcomes.len().saturating_sub(key_count),
|
||||
call_error_kind: None,
|
||||
elapsed_ms: elapsed.as_millis(),
|
||||
};
|
||||
for outcome in outcomes.iter().take(key_count) {
|
||||
match outcome {
|
||||
BatchAcquireOutcome::Granted { .. } => diagnostics.granted_keys += 1,
|
||||
BatchAcquireOutcome::RetryableNoGrant { .. } => diagnostics.retryable_no_grant_keys += 1,
|
||||
BatchAcquireOutcome::TerminalNoGrant { .. } => diagnostics.terminal_no_grant_keys += 1,
|
||||
BatchAcquireOutcome::Ambiguous { .. } => diagnostics.ambiguous_keys += 1,
|
||||
}
|
||||
}
|
||||
diagnostics.failed_response_keys =
|
||||
diagnostics.retryable_no_grant_keys + diagnostics.terminal_no_grant_keys + diagnostics.ambiguous_keys;
|
||||
diagnostics
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
fn from_response(peer_index: usize, key_count: usize, response: &Result<Vec<LockResponse>>, elapsed: Duration) -> Self {
|
||||
let mut diagnostics = Self {
|
||||
peer_index,
|
||||
granted_keys: 0,
|
||||
retryable_no_grant_keys: 0,
|
||||
terminal_no_grant_keys: 0,
|
||||
ambiguous_keys: 0,
|
||||
failed_response_keys: 0,
|
||||
missing_response_keys: 0,
|
||||
call_failed_keys: 0,
|
||||
extra_responses: 0,
|
||||
call_error_kind: None,
|
||||
elapsed_ms: elapsed.as_millis(),
|
||||
};
|
||||
match response {
|
||||
Ok(responses) => {
|
||||
for response in responses.iter().take(key_count) {
|
||||
if response.success {
|
||||
diagnostics.granted_keys += 1;
|
||||
} else {
|
||||
// Legacy response errors have already lost their source;
|
||||
// a failed response is not proof of contention or an RPC timeout.
|
||||
diagnostics.failed_response_keys += 1;
|
||||
}
|
||||
}
|
||||
diagnostics.missing_response_keys = key_count.saturating_sub(responses.len());
|
||||
diagnostics.extra_responses = responses.len().saturating_sub(key_count);
|
||||
}
|
||||
Err(error) => {
|
||||
diagnostics.call_failed_keys = key_count;
|
||||
diagnostics.call_error_kind = Some(match error {
|
||||
LockError::Timeout { .. } => "timeout",
|
||||
LockError::ResourceNotFound { .. } => "resource_not_found",
|
||||
LockError::PermissionDenied { .. } => "permission_denied",
|
||||
LockError::Network { .. } => "network",
|
||||
LockError::Internal { .. } => "internal",
|
||||
LockError::AlreadyLocked { .. } => "already_locked",
|
||||
LockError::InvalidHandle { .. } => "invalid_handle",
|
||||
LockError::Configuration { .. } => "configuration",
|
||||
LockError::Serialization { .. } => "serialization",
|
||||
LockError::Deserialization { .. } => "deserialization",
|
||||
LockError::InsufficientNodes { .. } => "insufficient_nodes",
|
||||
LockError::QuorumNotReached { .. } => "quorum_not_reached",
|
||||
LockError::QueueFull { .. } => "queue_full",
|
||||
LockError::NotOwner { .. } => "not_owner",
|
||||
});
|
||||
}
|
||||
}
|
||||
diagnostics
|
||||
}
|
||||
}
|
||||
|
||||
/// Generate a new aggregate lock ID for multiple client locks
|
||||
fn generate_aggregate_lock_id(resource: &ObjectKey) -> LockId {
|
||||
LockId {
|
||||
@@ -392,29 +277,6 @@ impl Drop for LockAcquireRound {
|
||||
}
|
||||
}
|
||||
|
||||
/// Keep completed grants and every unconsumed batch reply owned across cancellation.
|
||||
struct LockBatchAcquireRound {
|
||||
pending: JoinSet<LockBatchAcquireTaskResult>,
|
||||
grants: Vec<Vec<HeldLockEntry>>,
|
||||
clients: Arc<[Arc<dyn LockClient>]>,
|
||||
span: tracing::Span,
|
||||
}
|
||||
|
||||
impl Drop for LockBatchAcquireRound {
|
||||
fn drop(&mut self) {
|
||||
let _span = self.span.enter();
|
||||
let entries = self.grants.iter().flatten().map(HeldLockEntry::release_entry).collect();
|
||||
DistributedLock::spawn_release_cleanup(entries, "distributed_lock_cancelled_batch_acquisition");
|
||||
if !self.pending.is_empty() {
|
||||
DistributedLock::spawn_pending_batch_cleanup(
|
||||
std::mem::take(&mut self.pending),
|
||||
self.clients.clone(),
|
||||
"distributed_lock_cancelled_batch_acquisition_cleanup",
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug)]
|
||||
struct LockHeartbeatConfig {
|
||||
ttl: Duration,
|
||||
@@ -683,33 +545,6 @@ pub struct DistributedLock {
|
||||
}
|
||||
|
||||
type LockAcquireTaskResult = (usize, Result<LockResponse>);
|
||||
struct LockBatchAcquireTaskResult {
|
||||
peer_index: usize,
|
||||
key_indices: Vec<usize>,
|
||||
requests: Arc<[LockRequest]>,
|
||||
outcomes: Vec<BatchAcquireOutcome>,
|
||||
started: Instant,
|
||||
elapsed: Duration,
|
||||
}
|
||||
|
||||
fn normalized_batch_outcome(request: &LockRequest, outcome: Option<BatchAcquireOutcome>) -> BatchAcquireOutcome {
|
||||
match outcome {
|
||||
Some(BatchAcquireOutcome::Granted {
|
||||
lock_info: Some(ref info),
|
||||
..
|
||||
}) if info.id != request.lock_id || info.resource != request.resource || info.lock_type != request.lock_type => {
|
||||
BatchAcquireOutcome::Ambiguous {
|
||||
error: LockError::internal("Batch peer returned mismatched grant metadata"),
|
||||
cleanup_owned_by_client: false,
|
||||
}
|
||||
}
|
||||
Some(outcome) => outcome,
|
||||
None => BatchAcquireOutcome::Ambiguous {
|
||||
error: LockError::internal("Batch peer omitted an acquisition outcome"),
|
||||
cleanup_owned_by_client: false,
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
struct LockAcquireQuorumResult {
|
||||
response: LockResponse,
|
||||
@@ -904,14 +739,21 @@ impl DistributedLock {
|
||||
self.acquire_guard(&req).await
|
||||
}
|
||||
|
||||
/// Acquire independent exclusive locks with one sub-batch per peer and round.
|
||||
/// A successful key starts its heartbeat immediately. Only completed, explicitly
|
||||
/// retryable attempts whose grants have been cleaned may enter another round.
|
||||
/// The acquisition, rollback wait and backoff share one absolute deadline.
|
||||
/// Acquire exclusive locks for many independent resources in one batched
|
||||
/// round: one [`LockClient::acquire_locks_batch`] RPC per client instead of
|
||||
/// one quorum round-trip per key. Returns one guard or error per resource.
|
||||
///
|
||||
/// Cancellation or expiry transfers pending replies and exact-ID cleanup to
|
||||
/// independent owners. Exhausted cleanup retries still rely on backend leases;
|
||||
/// uncertain replies never establish that another attempt is safe on that peer.
|
||||
/// Bulk operations (S3 `DeleteObjects`) are the only multi-key write-lock
|
||||
/// holders, and their keys are independent. Per-key outcomes are
|
||||
/// deadlock-free without any serial fallback: the wire call is a single
|
||||
/// round per client and every lock server orders concurrent batches by the
|
||||
/// same global (shard, key) order, so no acquirer can hold one key while
|
||||
/// waiting for another. A key that fails to reach quorum releases its
|
||||
/// partial grants (awaited) before its error is returned.
|
||||
///
|
||||
/// Guards keep the same TTL heartbeat and lock-lost semantics as
|
||||
/// [`Self::acquire_guard`]. Contention is waited out server-side up to
|
||||
/// `timeout` (each request carries it), matching the single-key path.
|
||||
pub async fn acquire_guard_batch(
|
||||
&self,
|
||||
resources: &[ObjectKey],
|
||||
@@ -919,8 +761,6 @@ impl DistributedLock {
|
||||
timeout: Duration,
|
||||
ttl: Duration,
|
||||
) -> Vec<Result<DistributedLockGuard>> {
|
||||
let entered = Instant::now();
|
||||
let deadline = entered.checked_add(timeout).unwrap_or(entered);
|
||||
if resources.is_empty() {
|
||||
return Vec::new();
|
||||
}
|
||||
@@ -932,317 +772,97 @@ impl DistributedLock {
|
||||
}
|
||||
|
||||
let required_quorum = self.required_quorum(LockType::Exclusive);
|
||||
let mut results: Vec<Option<Result<DistributedLockGuard>>> = (0..resources.len()).map(|_| None).collect();
|
||||
let mut eligible = vec![vec![true; self.clients.len()]; resources.len()];
|
||||
let mut achieved = vec![0; resources.len()];
|
||||
let mut attempt = 0usize;
|
||||
while results.iter().any(Option::is_none) && Instant::now() < deadline {
|
||||
attempt = attempt.saturating_add(1);
|
||||
let round_started = Instant::now();
|
||||
let wait = self.peer_acquire_timeout(deadline.saturating_duration_since(round_started));
|
||||
let round_deadline = round_started.checked_add(wait).unwrap_or(round_started).min(deadline);
|
||||
let requests: Vec<_> = resources
|
||||
.iter()
|
||||
.enumerate()
|
||||
.map(|(key, resource)| {
|
||||
results[key].is_none().then(|| {
|
||||
LockRequest::new(resource.clone(), LockType::Exclusive, owner)
|
||||
.with_acquire_timeout(wait)
|
||||
.with_ttl(ttl)
|
||||
})
|
||||
})
|
||||
.collect();
|
||||
let mut round = LockBatchAcquireRound {
|
||||
pending: JoinSet::new(),
|
||||
grants: vec![Vec::new(); resources.len()],
|
||||
clients: self.clients.clone(),
|
||||
span: tracing::Span::current(),
|
||||
};
|
||||
for (peer_index, client) in self.clients.iter().enumerate() {
|
||||
let key_indices: Vec<_> = (0..resources.len())
|
||||
.filter(|&key| results[key].is_none() && eligible[key][peer_index])
|
||||
.collect();
|
||||
if key_indices.is_empty() {
|
||||
continue;
|
||||
}
|
||||
let requests: Arc<[LockRequest]> = key_indices.iter().filter_map(|&key| requests[key].clone()).collect();
|
||||
let client = client.clone();
|
||||
round.pending.spawn(
|
||||
async move {
|
||||
// The local deadline is fixed before scheduling or is_local().
|
||||
// Remote clients need the total deadline for queue/RPC headroom.
|
||||
let local = client.is_local().await;
|
||||
let peer_deadline = if local { round_deadline } else { deadline };
|
||||
let outcomes = if Instant::now() >= peer_deadline {
|
||||
requests
|
||||
.iter()
|
||||
.map(|request| BatchAcquireOutcome::RetryableNoGrant {
|
||||
error: LockError::timeout(request.resource.to_string(), request.acquire_timeout),
|
||||
})
|
||||
.collect()
|
||||
} else {
|
||||
client.acquire_locks_batch_until(&requests, peer_deadline).await
|
||||
};
|
||||
LockBatchAcquireTaskResult {
|
||||
peer_index,
|
||||
key_indices,
|
||||
requests,
|
||||
outcomes,
|
||||
started: round_started,
|
||||
elapsed: round_started.elapsed(),
|
||||
}
|
||||
}
|
||||
.in_current_span(),
|
||||
);
|
||||
}
|
||||
let attempt_started = Instant::now();
|
||||
let requests: Vec<LockRequest> = resources
|
||||
.iter()
|
||||
.map(|resource| {
|
||||
LockRequest::new(resource.clone(), LockType::Exclusive, owner)
|
||||
.with_acquire_timeout(timeout)
|
||||
.with_ttl(ttl)
|
||||
})
|
||||
.collect();
|
||||
|
||||
let mut pending_by_key: Vec<Vec<bool>> = eligible
|
||||
.iter()
|
||||
.enumerate()
|
||||
.map(|(key, peers)| peers.iter().map(|&enabled| enabled && results[key].is_none()).collect())
|
||||
.collect();
|
||||
let mut excluded_pending = false;
|
||||
let mut peer_outcomes = Vec::with_capacity(self.clients.len());
|
||||
let mut rollback_grants = 0usize;
|
||||
// A peer batch can contain both successful and unresolved keys. Keep
|
||||
// consuming its entire output until every key is settled or D_total.
|
||||
while !round.pending.is_empty() && results.iter().any(Option::is_none) {
|
||||
let joined = tokio::select! {
|
||||
biased;
|
||||
_ = tokio::time::sleep_until(deadline) => break,
|
||||
joined = round.pending.join_next() => joined,
|
||||
};
|
||||
let Some(joined) = joined else { break };
|
||||
let reply = match joined {
|
||||
Ok(reply) => reply,
|
||||
Err(error) => {
|
||||
warn!(
|
||||
event = EVENT_NAMESPACE_LOCK_ACQUIRE,
|
||||
component = LOG_COMPONENT_LOCK,
|
||||
subsystem = LOG_SUBSYSTEM_NAMESPACE,
|
||||
state = "batch_task_failed",
|
||||
failure = if error.is_cancelled() { "cancelled" } else { "panicked" },
|
||||
"Namespace batch lock peer task failed"
|
||||
);
|
||||
// A panicked implementation cannot prove no-grant. Do not
|
||||
// retry this round's requests with an unknown outcome.
|
||||
for (key, peers) in eligible.iter_mut().enumerate() {
|
||||
if results[key].is_none() {
|
||||
peers.fill(false);
|
||||
}
|
||||
}
|
||||
continue;
|
||||
}
|
||||
};
|
||||
peer_outcomes.push(BatchLockPeerDiagnostics::from_outcomes(
|
||||
reply.peer_index,
|
||||
reply.requests.len(),
|
||||
&reply.outcomes,
|
||||
reply.elapsed,
|
||||
));
|
||||
let client = self.clients[reply.peer_index].clone();
|
||||
let valid_until = reply.started.checked_add(ttl).unwrap_or(reply.started);
|
||||
let mut outcomes = reply.outcomes.into_iter();
|
||||
let mut redundant = Vec::new();
|
||||
let mut uncertain = Vec::new();
|
||||
for (&key, request) in reply.key_indices.iter().zip(reply.requests.iter()) {
|
||||
pending_by_key[key][reply.peer_index] = false;
|
||||
let outcome = normalized_batch_outcome(request, outcomes.next());
|
||||
match outcome {
|
||||
BatchAcquireOutcome::Granted { .. } => {
|
||||
let entry = HeldLockEntry {
|
||||
lock_id: request.lock_id.clone(),
|
||||
// One batched round per client; a client-level failure is a per-key
|
||||
// failure for every key that still needed that client's vote.
|
||||
let rounds = join_all(self.clients.iter().map(|client| {
|
||||
let requests = requests.clone();
|
||||
async move { client.acquire_locks_batch(&requests).await }
|
||||
}))
|
||||
.await;
|
||||
|
||||
let mut results: Vec<Result<DistributedLockGuard>> = Vec::with_capacity(resources.len());
|
||||
let mut orphaned_entries: Vec<(LockId, Arc<dyn LockClient>)> = Vec::new();
|
||||
for (key_idx, resource) in resources.iter().enumerate() {
|
||||
let mut entries: Vec<HeldLockEntry> = Vec::new();
|
||||
let mut failure: Option<String> = None;
|
||||
for (client_idx, client) in self.clients.iter().enumerate() {
|
||||
match &rounds[client_idx] {
|
||||
Ok(responses) => match responses.get(key_idx) {
|
||||
Some(response) if response.success => {
|
||||
let lock_id = response
|
||||
.lock_info
|
||||
.as_ref()
|
||||
.map(|info| info.id.clone())
|
||||
.unwrap_or_else(|| LockId::new_unique(resource));
|
||||
entries.push(HeldLockEntry {
|
||||
lock_id,
|
||||
client: client.clone(),
|
||||
valid_until,
|
||||
};
|
||||
if results[key].is_some() {
|
||||
redundant.push(entry.release_entry());
|
||||
} else {
|
||||
round.grants[key].push(entry);
|
||||
}
|
||||
valid_until: attempt_started.checked_add(ttl).unwrap_or(attempt_started),
|
||||
});
|
||||
}
|
||||
BatchAcquireOutcome::RetryableNoGrant { .. } => {}
|
||||
BatchAcquireOutcome::TerminalNoGrant { .. } => eligible[key][reply.peer_index] = false,
|
||||
BatchAcquireOutcome::Ambiguous {
|
||||
cleanup_owned_by_client, ..
|
||||
} => {
|
||||
eligible[key][reply.peer_index] = false;
|
||||
if !cleanup_owned_by_client {
|
||||
uncertain.push((request.lock_id.clone(), client.clone()));
|
||||
}
|
||||
Some(response) => {
|
||||
failure.get_or_insert_with(|| {
|
||||
response
|
||||
.error
|
||||
.clone()
|
||||
.unwrap_or_else(|| "distributed lock acquisition failed".to_string())
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
// Transfer cleanup synchronously before the next cancellation point.
|
||||
rollback_grants += redundant.len();
|
||||
Self::spawn_release_cleanup(redundant, "distributed_lock_batch_redundant_grants");
|
||||
Self::spawn_uncertain_batch_cleanup(uncertain, "distributed_lock_batch_ambiguous");
|
||||
|
||||
let now = Instant::now();
|
||||
for (key, resource) in resources.iter().enumerate() {
|
||||
if results[key].is_some() {
|
||||
continue;
|
||||
}
|
||||
achieved[key] = round.grants[key].iter().filter(|entry| now < entry.valid_until).count();
|
||||
if now < deadline && achieved[key] >= required_quorum {
|
||||
let (entries, expired): (Vec<_>, Vec<_>) = std::mem::take(&mut round.grants[key])
|
||||
.into_iter()
|
||||
.partition(|entry| now < entry.valid_until);
|
||||
rollback_grants += expired.len();
|
||||
Self::spawn_release_cleanup(
|
||||
expired.iter().map(HeldLockEntry::release_entry).collect(),
|
||||
"distributed_lock_batch_expired_grants",
|
||||
);
|
||||
let heartbeat = LockHeartbeatConfig {
|
||||
ttl,
|
||||
interval: derive_refresh_interval(entries.len(), ttl, None),
|
||||
refresh_quorum: required_quorum,
|
||||
owner: owner.to_owned(),
|
||||
resource: resource.clone(),
|
||||
};
|
||||
results[key] = Some(Ok(DistributedLockGuard::new(
|
||||
LockId::new_unique(resource),
|
||||
entries,
|
||||
LockType::Exclusive,
|
||||
heartbeat,
|
||||
)));
|
||||
}
|
||||
}
|
||||
// Stop waiting only when no unfinished key can gain a current
|
||||
// quorum and each retryable key already has enough completed
|
||||
// eligible peers. Pending requests are permanently excluded.
|
||||
let can_retry_without_pending = results.iter().enumerate().all(|(key, result)| {
|
||||
if result.is_some() {
|
||||
return true;
|
||||
}
|
||||
let pending_votes = pending_by_key[key].iter().filter(|&&value| value).count();
|
||||
pending_votes == 0
|
||||
|| (achieved[key] + pending_votes < required_quorum
|
||||
&& eligible[key]
|
||||
.iter()
|
||||
.zip(&pending_by_key[key])
|
||||
.filter(|(eligible, pending)| **eligible && !**pending)
|
||||
.count()
|
||||
>= required_quorum)
|
||||
});
|
||||
if can_retry_without_pending && !round.pending.is_empty() {
|
||||
for (key, peers) in eligible.iter_mut().enumerate() {
|
||||
if results[key].is_none() {
|
||||
for (peer, enabled) in peers.iter_mut().enumerate() {
|
||||
if pending_by_key[key][peer] {
|
||||
*enabled = false;
|
||||
}
|
||||
}
|
||||
None => {
|
||||
failure.get_or_insert_with(|| "client returned incomplete batch lock response".to_string());
|
||||
}
|
||||
},
|
||||
Err(err) => {
|
||||
failure.get_or_insert_with(|| err.to_string());
|
||||
}
|
||||
excluded_pending = true;
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
let mut cleanup_entries = Vec::new();
|
||||
for (key, grants) in round.grants.iter_mut().enumerate() {
|
||||
if results[key].is_none() {
|
||||
achieved[key] = grants.iter().filter(|entry| Instant::now() < entry.valid_until).count();
|
||||
}
|
||||
cleanup_entries.extend(std::mem::take(grants).into_iter().map(|entry| (key, entry)));
|
||||
}
|
||||
rollback_grants += cleanup_entries.len();
|
||||
// Dropping the round detaches all late replies before rollback awaits.
|
||||
// A selected peer is retried only when the old wave fully completed.
|
||||
let round_completed = round.pending.is_empty() || excluded_pending;
|
||||
drop(round);
|
||||
let mut blocked_cleanup = Vec::new();
|
||||
if !cleanup_entries.is_empty() {
|
||||
let cleanup = tokio::spawn(Self::release_batch_for_retry(cleanup_entries).in_current_span());
|
||||
match tokio::time::timeout_at(deadline, cleanup).await {
|
||||
Ok(Ok(blocked)) => blocked_cleanup = blocked,
|
||||
Ok(Err(_)) | Err(_) => {
|
||||
for (key, result) in results.iter().enumerate() {
|
||||
if result.is_none() {
|
||||
blocked_cleanup.push(key);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
for (key, result) in results.iter_mut().enumerate() {
|
||||
if result.is_none()
|
||||
&& (!round_completed
|
||||
|| blocked_cleanup.contains(&key)
|
||||
|| eligible[key].iter().filter(|&&value| value).count() < required_quorum
|
||||
|| Instant::now() >= deadline)
|
||||
{
|
||||
*result = Some(Err(LockError::QuorumNotReached {
|
||||
required: required_quorum,
|
||||
achieved: achieved[key],
|
||||
}));
|
||||
}
|
||||
}
|
||||
let mut keys_by_achieved_votes = vec![0; self.clients.len() + 1];
|
||||
for &votes in &achieved {
|
||||
keys_by_achieved_votes[votes] += 1;
|
||||
}
|
||||
let failed_keys = results.iter().filter(|result| matches!(result, Some(Err(_)))).count();
|
||||
let successful_keys = results.iter().filter(|result| matches!(result, Some(Ok(_)))).count();
|
||||
peer_outcomes.sort_unstable_by_key(|outcome| outcome.peer_index);
|
||||
if failed_keys > 0 {
|
||||
warn!(
|
||||
event = EVENT_NAMESPACE_LOCK_ACQUIRE,
|
||||
component = LOG_COMPONENT_LOCK,
|
||||
subsystem = LOG_SUBSYSTEM_NAMESPACE,
|
||||
state = "batch_quorum_failed",
|
||||
round = attempt,
|
||||
key_count = resources.len(),
|
||||
peer_count = self.clients.len(),
|
||||
required_quorum,
|
||||
successful_keys,
|
||||
failed_keys,
|
||||
?keys_by_achieved_votes,
|
||||
?peer_outcomes,
|
||||
rollback_grants,
|
||||
acquire_ms = round_started.elapsed().as_millis(),
|
||||
"Namespace batch lock acquisition did not reach quorum for all keys"
|
||||
);
|
||||
if entries.len() >= required_quorum {
|
||||
let heartbeat = LockHeartbeatConfig {
|
||||
ttl,
|
||||
interval: derive_refresh_interval(entries.len(), ttl, None),
|
||||
refresh_quorum: required_quorum,
|
||||
owner: owner.to_string(),
|
||||
resource: resource.clone(),
|
||||
};
|
||||
results.push(Ok(DistributedLockGuard::new(
|
||||
LockId::new_unique(resource),
|
||||
entries,
|
||||
LockType::Exclusive,
|
||||
heartbeat,
|
||||
)));
|
||||
} else {
|
||||
debug!(
|
||||
event = EVENT_NAMESPACE_LOCK_ACQUIRE,
|
||||
component = LOG_COMPONENT_LOCK,
|
||||
subsystem = LOG_SUBSYSTEM_NAMESPACE,
|
||||
state = "batch_round_completed",
|
||||
round = attempt,
|
||||
key_count = resources.len(),
|
||||
peer_count = self.clients.len(),
|
||||
required_quorum,
|
||||
successful_keys,
|
||||
retrying_keys = resources.len() - successful_keys,
|
||||
?keys_by_achieved_votes,
|
||||
?peer_outcomes,
|
||||
rollback_grants,
|
||||
acquire_ms = round_started.elapsed().as_millis(),
|
||||
"Namespace batch lock acquisition round completed"
|
||||
);
|
||||
}
|
||||
if results.iter().any(Option::is_none) && Instant::now() < deadline {
|
||||
let remaining = deadline.saturating_duration_since(Instant::now());
|
||||
let backoff = Self::lock_acquire_retry_backoff(attempt, &mut rand::rng()).min(remaining / 2);
|
||||
if backoff.is_zero() {
|
||||
break;
|
||||
let achieved = entries.len();
|
||||
orphaned_entries.extend(entries.iter().map(HeldLockEntry::release_entry));
|
||||
if let Some(error) = failure {
|
||||
tracing::debug!(resource = %resource, owner, required_quorum, achieved, "batch lock acquire failed: {error}");
|
||||
}
|
||||
tokio::time::sleep_until(Instant::now().checked_add(backoff).unwrap_or(deadline).min(deadline)).await;
|
||||
results.push(Err(LockError::QuorumNotReached {
|
||||
required: required_quorum,
|
||||
achieved,
|
||||
}));
|
||||
}
|
||||
}
|
||||
|
||||
// One bounded synchronous pass so the caller cannot race these
|
||||
// unlocks on an immediate retry; anything a slow peer still holds
|
||||
// keeps retrying in the background instead of parking the request
|
||||
// on the deferred unlock schedule.
|
||||
if !orphaned_entries.is_empty() {
|
||||
Self::release_entries_bounded(orphaned_entries, "distributed_lock_batch_quorum_rollback").await;
|
||||
}
|
||||
results
|
||||
.into_iter()
|
||||
.map(|result| match result {
|
||||
Some(Ok(guard)) if !guard.is_lock_lost() => Ok(guard),
|
||||
Some(Ok(_)) | None => Err(LockError::QuorumNotReached {
|
||||
required: required_quorum,
|
||||
achieved: 0,
|
||||
}),
|
||||
Some(Err(error)) => Err(error),
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
|
||||
fn spawn_lock_requests_until(
|
||||
@@ -1346,79 +966,16 @@ impl DistributedLock {
|
||||
Self::release_entries_deferred(pending, context).await;
|
||||
}
|
||||
|
||||
/// Retain rollback ownership independently of the foreground deadline. Keys
|
||||
/// with an unconfirmed grant are not retried; their remaining cleanup continues.
|
||||
async fn release_batch_for_retry(mut entries: Vec<(usize, HeldLockEntry)>) -> Vec<usize> {
|
||||
for attempt in 1..=UNLOCK_RETRY_ATTEMPTS {
|
||||
entries = join_all(entries.into_iter().map(|(key, entry)| async move {
|
||||
match entry.client.cleanup_grant(&entry.lock_id).await {
|
||||
Ok(LockCleanupOutcome::Released | LockCleanupOutcome::AlreadyAbsent) => None,
|
||||
Ok(LockCleanupOutcome::Unconfirmed) | Err(_) => Some((key, entry)),
|
||||
}
|
||||
}))
|
||||
.await
|
||||
.into_iter()
|
||||
.flatten()
|
||||
.collect();
|
||||
if entries.is_empty() {
|
||||
return Vec::new();
|
||||
}
|
||||
if attempt < UNLOCK_RETRY_ATTEMPTS {
|
||||
let multiplier = u32::try_from(attempt).unwrap_or(u32::MAX);
|
||||
tokio::time::sleep(UNLOCK_RETRY_BACKOFF.saturating_mul(multiplier)).await;
|
||||
}
|
||||
}
|
||||
let mut blocked: Vec<_> = entries.iter().map(|(key, _)| *key).collect();
|
||||
blocked.sort_unstable();
|
||||
blocked.dedup();
|
||||
Self::spawn_cleanup(
|
||||
Self::release_entries_deferred(
|
||||
entries.into_iter().map(|(_, entry)| entry.release_entry()).collect(),
|
||||
"distributed_lock_batch_quorum_rollback",
|
||||
),
|
||||
"distributed_lock_batch_quorum_rollback",
|
||||
);
|
||||
blocked
|
||||
}
|
||||
|
||||
fn spawn_uncertain_batch_cleanup(entries: Vec<(LockId, Arc<dyn LockClient>)>, context: &'static str) {
|
||||
if !entries.is_empty() {
|
||||
Self::spawn_cleanup(Self::release_uncertain_batch(entries, context), context);
|
||||
}
|
||||
}
|
||||
|
||||
/// Ambiguous acquisitions can still grant after an absence reply. Only an
|
||||
/// exact-ID release acknowledgement ends this conservative cleanup schedule.
|
||||
async fn release_uncertain_batch(mut pending: Vec<(LockId, Arc<dyn LockClient>)>, context: &'static str) {
|
||||
let delays = [Duration::ZERO, UNLOCK_RETRY_BACKOFF, UNLOCK_RETRY_BACKOFF * 2]
|
||||
.into_iter()
|
||||
.chain(DEFERRED_UNLOCK_BACKOFF);
|
||||
for delay in delays {
|
||||
if pending.is_empty() {
|
||||
return;
|
||||
}
|
||||
tokio::time::sleep(delay).await;
|
||||
pending = join_all(pending.into_iter().map(|(lock_id, client)| async move {
|
||||
match client.release(&lock_id).await {
|
||||
Ok(true) => None,
|
||||
Ok(false) | Err(_) => Some((lock_id, client)),
|
||||
}
|
||||
}))
|
||||
.await
|
||||
.into_iter()
|
||||
.flatten()
|
||||
.collect();
|
||||
}
|
||||
/// Release entries with one synchronous pass, then continue the retry
|
||||
/// schedule in the background.
|
||||
///
|
||||
/// The batch-acquire rollback path runs on the request: it must not park
|
||||
/// the caller on the deferred unlock schedule when a peer is unreachable,
|
||||
/// while still confirming the healthy quorum is free before returning.
|
||||
async fn release_entries_bounded(entries: Vec<(LockId, Arc<dyn LockClient>)>, context: &'static str) {
|
||||
let pending = Self::release_pending_once(entries, 1, context).await;
|
||||
if !pending.is_empty() {
|
||||
warn!(
|
||||
event = EVENT_DISTRIBUTED_LOCK_CLEANUP,
|
||||
component = LOG_COMPONENT_LOCK,
|
||||
subsystem = LOG_SUBSYSTEM_NAMESPACE,
|
||||
state = "uncertain",
|
||||
context,
|
||||
remaining = pending.len(),
|
||||
"Ambiguous batch grant cleanup remains unconfirmed after bounded retries"
|
||||
);
|
||||
Self::spawn_release_cleanup(pending, context);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1509,7 +1066,6 @@ impl DistributedLock {
|
||||
}
|
||||
|
||||
fn spawn_cleanup(cleanup: impl std::future::Future<Output = ()> + Send + 'static, context: &'static str) {
|
||||
let cleanup = cleanup.in_current_span();
|
||||
if let Ok(handle) = tokio::runtime::Handle::try_current() {
|
||||
let join_handle = handle.spawn(cleanup);
|
||||
drop(join_handle);
|
||||
@@ -1531,96 +1087,6 @@ impl DistributedLock {
|
||||
drop(join_handle);
|
||||
}
|
||||
|
||||
fn spawn_pending_batch_cleanup(
|
||||
mut pending: JoinSet<LockBatchAcquireTaskResult>,
|
||||
clients: Arc<[Arc<dyn LockClient>]>,
|
||||
context: &'static str,
|
||||
) {
|
||||
Self::spawn_cleanup(
|
||||
async move {
|
||||
let mut releases = JoinSet::new();
|
||||
while let Some(joined) = pending.join_next().await {
|
||||
match joined {
|
||||
Ok(reply) => {
|
||||
let peer_outcome = BatchLockPeerDiagnostics::from_outcomes(
|
||||
reply.peer_index,
|
||||
reply.requests.len(),
|
||||
&reply.outcomes,
|
||||
reply.elapsed,
|
||||
);
|
||||
debug!(
|
||||
event = EVENT_DISTRIBUTED_LOCK_CLEANUP,
|
||||
component = LOG_COMPONENT_LOCK,
|
||||
subsystem = LOG_SUBSYSTEM_NAMESPACE,
|
||||
state = "batch_reply_completed",
|
||||
context,
|
||||
?peer_outcome,
|
||||
"Unconsumed batch lock reply retained for cleanup"
|
||||
);
|
||||
let Some(client) = clients.get(reply.peer_index) else { continue };
|
||||
let mut outcomes = reply.outcomes.into_iter();
|
||||
let mut confirmed = Vec::new();
|
||||
let mut uncertain = Vec::new();
|
||||
for request in reply.requests.iter() {
|
||||
match normalized_batch_outcome(request, outcomes.next()) {
|
||||
BatchAcquireOutcome::Granted { .. } => {
|
||||
confirmed.push((request.lock_id.clone(), client.clone()))
|
||||
}
|
||||
BatchAcquireOutcome::Ambiguous {
|
||||
cleanup_owned_by_client: false,
|
||||
..
|
||||
} => {
|
||||
uncertain.push((request.lock_id.clone(), client.clone()));
|
||||
}
|
||||
BatchAcquireOutcome::RetryableNoGrant { .. }
|
||||
| BatchAcquireOutcome::TerminalNoGrant { .. }
|
||||
| BatchAcquireOutcome::Ambiguous {
|
||||
cleanup_owned_by_client: true,
|
||||
..
|
||||
} => {}
|
||||
}
|
||||
}
|
||||
// Keep cleanup concurrent across peers without blocking the
|
||||
// drain on one release or one deferred uncertainty schedule.
|
||||
releases.spawn(
|
||||
async move {
|
||||
tokio::join!(
|
||||
Self::release_entries(confirmed, context),
|
||||
Self::release_uncertain_batch(uncertain, context)
|
||||
);
|
||||
}
|
||||
.in_current_span(),
|
||||
);
|
||||
}
|
||||
Err(error) => warn!(
|
||||
event = EVENT_DISTRIBUTED_LOCK_CLEANUP,
|
||||
component = LOG_COMPONENT_LOCK,
|
||||
subsystem = LOG_SUBSYSTEM_NAMESPACE,
|
||||
state = "task_failed",
|
||||
context,
|
||||
failure = if error.is_cancelled() { "cancelled" } else { "panicked" },
|
||||
"Pending batch lock acquisition task failed"
|
||||
),
|
||||
}
|
||||
}
|
||||
while let Some(result) = releases.join_next().await {
|
||||
if let Err(error) = result {
|
||||
warn!(
|
||||
event = EVENT_DISTRIBUTED_LOCK_CLEANUP,
|
||||
component = LOG_COMPONENT_LOCK,
|
||||
subsystem = LOG_SUBSYSTEM_NAMESPACE,
|
||||
state = "task_failed",
|
||||
context,
|
||||
failure = if error.is_cancelled() { "cancelled" } else { "panicked" },
|
||||
"Pending batch grant release cleanup task failed"
|
||||
);
|
||||
}
|
||||
}
|
||||
},
|
||||
context,
|
||||
);
|
||||
}
|
||||
|
||||
fn spawn_pending_cleanup(
|
||||
mut pending: JoinSet<LockAcquireTaskResult>,
|
||||
clients: Arc<[Arc<dyn LockClient>]>,
|
||||
@@ -2185,18 +1651,12 @@ fn record_lock_held_release(lock_type: LockType) {
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod batch_acquire_tests;
|
||||
|
||||
#[cfg(test)]
|
||||
mod batch_retry_tests;
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::{
|
||||
BatchLockPeerDiagnostics, DEFERRED_UNLOCK_BACKOFF, DistributedLock, LOCK_ACQUIRE_RETRY_INITIAL_BACKOFF,
|
||||
LOCK_ACQUIRE_SPARE_HEDGES, LOCK_PEER_ACQUIRE_TIMEOUT, LockAcquireFailureKind, LockLostSignal, UNLOCK_RETRY_ATTEMPTS,
|
||||
UNLOCK_RETRY_BACKOFF, is_definitive_lock_contention, is_remote_lock_rpc_failure, should_warn_lock_failure,
|
||||
DEFERRED_UNLOCK_BACKOFF, DistributedLock, LOCK_ACQUIRE_RETRY_INITIAL_BACKOFF, LOCK_ACQUIRE_SPARE_HEDGES,
|
||||
LOCK_PEER_ACQUIRE_TIMEOUT, LockAcquireFailureKind, LockLostSignal, UNLOCK_RETRY_ATTEMPTS, UNLOCK_RETRY_BACKOFF,
|
||||
is_definitive_lock_contention, is_remote_lock_rpc_failure, should_warn_lock_failure,
|
||||
};
|
||||
use crate::{
|
||||
GlobalLockManager, LocalClient, LockCleanupOutcome, LockError, LockId, LockInfo, LockRequest, LockResponse, LockStats,
|
||||
@@ -4567,71 +4027,6 @@ mod tests {
|
||||
drop(guard);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn batch_peer_diagnostics_preserves_response_failure_and_missing_counts() {
|
||||
let success = LockResponse {
|
||||
success: true,
|
||||
lock_info: None,
|
||||
error: None,
|
||||
wait_time: Duration::ZERO,
|
||||
position_in_queue: None,
|
||||
};
|
||||
let responses = Ok(vec![
|
||||
success.clone(),
|
||||
LockResponse::failure("Remote lock RPC timed out", Duration::ZERO),
|
||||
success,
|
||||
]);
|
||||
assert_eq!(
|
||||
BatchLockPeerDiagnostics::from_response(2, 4, &responses, Duration::from_millis(17)),
|
||||
BatchLockPeerDiagnostics {
|
||||
peer_index: 2,
|
||||
granted_keys: 2,
|
||||
retryable_no_grant_keys: 0,
|
||||
terminal_no_grant_keys: 0,
|
||||
ambiguous_keys: 0,
|
||||
failed_response_keys: 1,
|
||||
missing_response_keys: 1,
|
||||
call_failed_keys: 0,
|
||||
extra_responses: 0,
|
||||
call_error_kind: None,
|
||||
elapsed_ms: 17,
|
||||
},
|
||||
"failed legacy responses and missing entries must remain distinct without guessing their cause"
|
||||
);
|
||||
let extra = BatchLockPeerDiagnostics::from_response(0, 1, &responses, Duration::ZERO);
|
||||
assert_eq!(extra.granted_keys, 1, "out-of-range responses cannot add valid grant votes");
|
||||
assert_eq!(extra.failed_response_keys, 0);
|
||||
assert_eq!(extra.missing_response_keys, 0);
|
||||
assert_eq!(extra.extra_responses, 2, "extra peer entries must remain visible");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn batch_peer_diagnostics_counts_all_keys_for_a_typed_call_failure() {
|
||||
for (error, expected_kind) in [
|
||||
(LockError::timeout("peer", Duration::from_secs(3)), "timeout"),
|
||||
(LockError::internal("Remote lock RPC timed out"), "internal"),
|
||||
(LockError::permission_denied("request rejected"), "permission_denied"),
|
||||
] {
|
||||
assert_eq!(
|
||||
BatchLockPeerDiagnostics::from_response(1, 1000, &Err(error), Duration::from_secs(3)),
|
||||
BatchLockPeerDiagnostics {
|
||||
peer_index: 1,
|
||||
granted_keys: 0,
|
||||
retryable_no_grant_keys: 0,
|
||||
terminal_no_grant_keys: 0,
|
||||
ambiguous_keys: 0,
|
||||
failed_response_keys: 0,
|
||||
missing_response_keys: 0,
|
||||
call_failed_keys: 1000,
|
||||
extra_responses: 0,
|
||||
call_error_kind: Some(expected_kind),
|
||||
elapsed_ms: 3000,
|
||||
},
|
||||
"one failed peer call affects every requested key, and only its typed variant identifies the cause"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/// Per-object scripted client: grants only the named objects and records
|
||||
/// every released lock's resource.
|
||||
#[derive(Debug)]
|
||||
|
||||
@@ -1,451 +0,0 @@
|
||||
// Copyright 2024 RustFS Team
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
use super::{DistributedLock, LockBatchAcquireRound};
|
||||
use crate::{
|
||||
GlobalLockManager, LocalClient, LockCleanupOutcome, LockError, LockId, LockInfo, LockRequest, LockResponse, LockStats,
|
||||
LockType, ObjectKey, client::LockClient,
|
||||
};
|
||||
use std::sync::{
|
||||
Arc, Mutex,
|
||||
atomic::{AtomicBool, AtomicUsize, Ordering},
|
||||
};
|
||||
use std::time::Duration;
|
||||
use tokio::sync::{Notify, Semaphore};
|
||||
use tokio::task::JoinSet;
|
||||
|
||||
#[derive(Debug)]
|
||||
struct ObservedBatchClient {
|
||||
backend: LocalClient,
|
||||
deny_acquire: bool,
|
||||
acquire_gate: Option<Arc<Semaphore>>,
|
||||
cleanup_gate: Option<Arc<Semaphore>>,
|
||||
granted: AtomicUsize,
|
||||
grant_changed: Notify,
|
||||
released: AtomicUsize,
|
||||
release_changed: Notify,
|
||||
cleanup_entered: Notify,
|
||||
cleanup_calls: AtomicUsize,
|
||||
cleanup_changed: Notify,
|
||||
cleanup_ids: Mutex<Vec<LockId>>,
|
||||
omit_lock_info: AtomicBool,
|
||||
lose_release_ack: AtomicBool,
|
||||
release_ack_gate: Semaphore,
|
||||
}
|
||||
|
||||
impl ObservedBatchClient {
|
||||
fn new(deny_acquire: bool, acquire_gate: Option<Arc<Semaphore>>, cleanup_gate: Option<Arc<Semaphore>>) -> Arc<Self> {
|
||||
Arc::new(Self {
|
||||
backend: LocalClient::with_manager(Arc::new(GlobalLockManager::new())),
|
||||
deny_acquire,
|
||||
acquire_gate,
|
||||
cleanup_gate,
|
||||
granted: AtomicUsize::new(0),
|
||||
grant_changed: Notify::new(),
|
||||
released: AtomicUsize::new(0),
|
||||
release_changed: Notify::new(),
|
||||
cleanup_entered: Notify::new(),
|
||||
cleanup_calls: AtomicUsize::new(0),
|
||||
cleanup_changed: Notify::new(),
|
||||
cleanup_ids: Mutex::new(Vec::new()),
|
||||
omit_lock_info: AtomicBool::new(false),
|
||||
lose_release_ack: AtomicBool::new(false),
|
||||
release_ack_gate: Semaphore::new(0),
|
||||
})
|
||||
}
|
||||
|
||||
async fn wait_for_count(counter: &AtomicUsize, changed: &Notify, expected: usize) {
|
||||
tokio::time::timeout(Duration::from_secs(1), async {
|
||||
loop {
|
||||
let notified = changed.notified();
|
||||
if counter.load(Ordering::SeqCst) >= expected {
|
||||
return;
|
||||
}
|
||||
notified.await;
|
||||
}
|
||||
})
|
||||
.await
|
||||
.expect("expected real backend grant/release event before the test deadline");
|
||||
}
|
||||
|
||||
async fn wait_for_grants(&self, expected: usize) {
|
||||
Self::wait_for_count(&self.granted, &self.grant_changed, expected).await;
|
||||
}
|
||||
|
||||
async fn wait_for_releases(&self, expected: usize) {
|
||||
Self::wait_for_count(&self.released, &self.release_changed, expected).await;
|
||||
}
|
||||
}
|
||||
|
||||
#[async_trait::async_trait]
|
||||
impl LockClient for ObservedBatchClient {
|
||||
async fn acquire_lock(&self, request: &LockRequest) -> crate::Result<LockResponse> {
|
||||
if self.deny_acquire {
|
||||
return Ok(LockResponse::failure("injected peer refusal", Duration::ZERO));
|
||||
}
|
||||
if let Some(gate) = &self.acquire_gate {
|
||||
gate.acquire().await.expect("test acquisition gate remains open").forget();
|
||||
}
|
||||
let mut response = self.backend.acquire_lock(request).await?;
|
||||
if response.success {
|
||||
self.granted.fetch_add(1, Ordering::SeqCst);
|
||||
self.grant_changed.notify_one();
|
||||
if self.omit_lock_info.load(Ordering::SeqCst) {
|
||||
response.lock_info = None;
|
||||
}
|
||||
}
|
||||
Ok(response)
|
||||
}
|
||||
|
||||
async fn cleanup_grant(&self, lock_id: &LockId) -> crate::Result<LockCleanupOutcome> {
|
||||
self.cleanup_ids
|
||||
.lock()
|
||||
.expect("test cleanup ID log is available")
|
||||
.push(lock_id.clone());
|
||||
self.cleanup_calls.fetch_add(1, Ordering::SeqCst);
|
||||
self.cleanup_changed.notify_one();
|
||||
self.cleanup_entered.notify_one();
|
||||
if let Some(gate) = &self.cleanup_gate {
|
||||
gate.acquire().await.expect("test cleanup gate remains open").forget();
|
||||
}
|
||||
let outcome = self.backend.cleanup_grant(lock_id).await?;
|
||||
if outcome == LockCleanupOutcome::Released {
|
||||
self.released.fetch_add(1, Ordering::SeqCst);
|
||||
self.release_changed.notify_one();
|
||||
if self.lose_release_ack.swap(false, Ordering::SeqCst) {
|
||||
self.release_ack_gate
|
||||
.acquire()
|
||||
.await
|
||||
.expect("test ACK gate remains open")
|
||||
.forget();
|
||||
return Err(LockError::internal("injected lost release ACK"));
|
||||
}
|
||||
}
|
||||
Ok(outcome)
|
||||
}
|
||||
|
||||
async fn release(&self, lock_id: &LockId) -> crate::Result<bool> {
|
||||
self.cleanup_grant(lock_id)
|
||||
.await
|
||||
.map(|outcome| outcome == LockCleanupOutcome::Released)
|
||||
}
|
||||
|
||||
async fn refresh(&self, lock_id: &LockId) -> crate::Result<bool> {
|
||||
self.backend.refresh(lock_id).await
|
||||
}
|
||||
|
||||
async fn force_release(&self, lock_id: &LockId) -> crate::Result<bool> {
|
||||
self.backend.force_release(lock_id).await
|
||||
}
|
||||
|
||||
async fn check_status(&self, lock_id: &LockId) -> crate::Result<Option<LockInfo>> {
|
||||
self.backend.check_status(lock_id).await
|
||||
}
|
||||
|
||||
async fn get_stats(&self) -> crate::Result<LockStats> {
|
||||
self.backend.get_stats().await
|
||||
}
|
||||
|
||||
async fn close(&self) -> crate::Result<()> {
|
||||
self.backend.close().await
|
||||
}
|
||||
|
||||
async fn is_online(&self) -> bool {
|
||||
true
|
||||
}
|
||||
|
||||
async fn is_local(&self) -> bool {
|
||||
true
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test(start_paused = true)]
|
||||
async fn cancelled_batch_acquisition_cleans_completed_and_late_grants() {
|
||||
let late_gate = Arc::new(Semaphore::new(0));
|
||||
let completed = ObservedBatchClient::new(false, None, None);
|
||||
let late = ObservedBatchClient::new(false, Some(late_gate.clone()), None);
|
||||
let lock = DistributedLock::new("cancelled-batch".to_string(), vec![completed.clone(), late.clone()], 2);
|
||||
let resources = (0..1000)
|
||||
.map(|key| ObjectKey::new("cancelled-batch", format!("key-{key}")))
|
||||
.collect::<Vec<_>>();
|
||||
let operation = tokio::spawn(async move {
|
||||
lock.acquire_guard_batch(&resources, "owner", Duration::from_secs(5), Duration::from_secs(30))
|
||||
.await
|
||||
});
|
||||
|
||||
completed.wait_for_grants(1000).await;
|
||||
assert_eq!(completed.backend.list_lock_leases().await.len(), 1000);
|
||||
assert!(!operation.is_finished(), "the other peer still cannot complete its batch");
|
||||
operation.abort();
|
||||
assert!(operation.await.expect_err("caller was cancelled").is_cancelled());
|
||||
late_gate.add_permits(1000);
|
||||
|
||||
completed.wait_for_releases(1000).await;
|
||||
late.wait_for_releases(1000).await;
|
||||
assert!(completed.backend.list_lock_leases().await.is_empty());
|
||||
assert!(late.backend.list_lock_leases().await.is_empty());
|
||||
}
|
||||
|
||||
#[tokio::test(start_paused = true)]
|
||||
async fn cancelled_batch_rollback_keeps_its_cleanup_owner() {
|
||||
let cleanup_gate = Arc::new(Semaphore::new(0));
|
||||
let granted = ObservedBatchClient::new(false, None, Some(cleanup_gate.clone()));
|
||||
let refused = ObservedBatchClient::new(true, None, None);
|
||||
let lock = DistributedLock::new("cancelled-rollback".to_string(), vec![granted.clone(), refused], 2);
|
||||
let operation = tokio::spawn(async move {
|
||||
lock.acquire_guard_batch(
|
||||
&[ObjectKey::new("cancelled-rollback", "object")],
|
||||
"owner",
|
||||
Duration::from_secs(5),
|
||||
Duration::from_secs(30),
|
||||
)
|
||||
.await
|
||||
});
|
||||
|
||||
tokio::time::timeout(Duration::from_secs(1), granted.cleanup_entered.notified())
|
||||
.await
|
||||
.expect("below-quorum rollback entered the real grant cleanup");
|
||||
assert_eq!(granted.backend.list_lock_leases().await.len(), 1);
|
||||
operation.abort();
|
||||
assert!(
|
||||
operation
|
||||
.await
|
||||
.expect_err("caller was cancelled during rollback")
|
||||
.is_cancelled()
|
||||
);
|
||||
cleanup_gate.add_permits(1);
|
||||
|
||||
granted.wait_for_releases(1).await;
|
||||
assert!(granted.backend.list_lock_leases().await.is_empty());
|
||||
}
|
||||
|
||||
#[tokio::test(start_paused = true)]
|
||||
async fn cancelled_batch_round_drains_ready_unjoined_replies_with_original_ids() {
|
||||
for omit_lock_info in [false, true] {
|
||||
let first = ObservedBatchClient::new(false, None, None);
|
||||
let second = ObservedBatchClient::new(false, None, None);
|
||||
first.omit_lock_info.store(omit_lock_info, Ordering::SeqCst);
|
||||
second.omit_lock_info.store(omit_lock_info, Ordering::SeqCst);
|
||||
let clients: Arc<[Arc<dyn LockClient>]> = vec![first.clone() as Arc<dyn LockClient>, second.clone()].into();
|
||||
let requests: Arc<[LockRequest]> = vec![
|
||||
LockRequest::new(ObjectKey::new("ready-batch", "object"), LockType::Exclusive, "owner")
|
||||
.with_acquire_timeout(Duration::from_secs(5))
|
||||
.with_ttl(Duration::from_secs(30)),
|
||||
]
|
||||
.into();
|
||||
let mut round = LockBatchAcquireRound {
|
||||
pending: JoinSet::new(),
|
||||
grants: vec![Vec::new()],
|
||||
clients,
|
||||
span: tracing::Span::current(),
|
||||
};
|
||||
let mut tasks = Vec::new();
|
||||
for (peer_index, client) in round.clients.iter().enumerate() {
|
||||
let client = client.clone();
|
||||
let requests = requests.clone();
|
||||
tasks.push(round.pending.spawn(async move {
|
||||
let started = tokio::time::Instant::now();
|
||||
let outcomes = client
|
||||
.acquire_locks_batch_until(&requests, started + Duration::from_secs(5))
|
||||
.await;
|
||||
super::LockBatchAcquireTaskResult {
|
||||
peer_index,
|
||||
key_indices: vec![0],
|
||||
requests,
|
||||
outcomes,
|
||||
started,
|
||||
elapsed: Duration::ZERO,
|
||||
}
|
||||
}));
|
||||
}
|
||||
first.wait_for_grants(1).await;
|
||||
second.wait_for_grants(1).await;
|
||||
assert!(tasks.iter().all(tokio::task::AbortHandle::is_finished));
|
||||
assert_eq!(round.pending.len(), 2, "both ready replies remain unjoined");
|
||||
assert!(round.grants.iter().all(Vec::is_empty), "no reply has been transferred yet");
|
||||
drop(round);
|
||||
|
||||
first.wait_for_releases(1).await;
|
||||
second.wait_for_releases(1).await;
|
||||
assert!(first.backend.list_lock_leases().await.is_empty());
|
||||
assert!(second.backend.list_lock_leases().await.is_empty());
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test(start_paused = true)]
|
||||
async fn cancelled_batch_preserves_a_locally_partially_completed_batch() {
|
||||
let client = ObservedBatchClient::new(false, None, None);
|
||||
let blocked = ObjectKey::new("local-partial-batch", "blocked");
|
||||
let blocker = LockRequest::new(blocked.clone(), LockType::Exclusive, "blocker")
|
||||
.with_acquire_timeout(Duration::from_secs(5))
|
||||
.with_ttl(Duration::from_secs(30));
|
||||
assert!(
|
||||
client
|
||||
.backend
|
||||
.acquire_lock(&blocker)
|
||||
.await
|
||||
.expect("hold one local key")
|
||||
.success
|
||||
);
|
||||
let lock = DistributedLock::new("local-partial-batch".to_string(), vec![client.clone()], 1);
|
||||
let operation = tokio::spawn(async move {
|
||||
lock.acquire_guard_batch(
|
||||
&[
|
||||
blocked,
|
||||
ObjectKey::new("local-partial-batch", "free-one"),
|
||||
ObjectKey::new("local-partial-batch", "free-two"),
|
||||
],
|
||||
"batch-owner",
|
||||
Duration::from_secs(5),
|
||||
Duration::from_secs(30),
|
||||
)
|
||||
.await
|
||||
});
|
||||
client.wait_for_grants(2).await;
|
||||
assert_eq!(client.backend.list_lock_leases().await.len(), 3, "blocker plus two partial grants");
|
||||
assert!(!operation.is_finished());
|
||||
operation.abort();
|
||||
assert!(
|
||||
operation
|
||||
.await
|
||||
.expect_err("cancel the partially completed batch")
|
||||
.is_cancelled()
|
||||
);
|
||||
assert_eq!(
|
||||
client
|
||||
.backend
|
||||
.cleanup_grant(&blocker.lock_id)
|
||||
.await
|
||||
.expect("release the original holder"),
|
||||
LockCleanupOutcome::Released
|
||||
);
|
||||
|
||||
client.wait_for_grants(3).await;
|
||||
client.wait_for_releases(3).await;
|
||||
assert!(client.backend.list_lock_leases().await.is_empty());
|
||||
}
|
||||
|
||||
#[tokio::test(start_paused = true)]
|
||||
async fn successful_batch_guards_retain_their_real_grants_until_drop() {
|
||||
let client = ObservedBatchClient::new(false, None, None);
|
||||
let lock = DistributedLock::new("successful-batch".to_string(), vec![client.clone()], 1);
|
||||
let results = lock
|
||||
.acquire_guard_batch(
|
||||
&[
|
||||
ObjectKey::new("successful-batch", "one"),
|
||||
ObjectKey::new("successful-batch", "two"),
|
||||
],
|
||||
"owner",
|
||||
Duration::from_secs(5),
|
||||
Duration::from_secs(30),
|
||||
)
|
||||
.await;
|
||||
assert!(results.iter().all(Result::is_ok));
|
||||
assert_eq!(client.backend.list_lock_leases().await.len(), 2);
|
||||
assert_eq!(client.cleanup_calls.load(Ordering::SeqCst), 0, "round drop cannot clean live guards");
|
||||
drop(results);
|
||||
|
||||
client.wait_for_releases(2).await;
|
||||
assert!(client.backend.list_lock_leases().await.is_empty());
|
||||
}
|
||||
|
||||
#[tokio::test(start_paused = true)]
|
||||
async fn batch_guard_and_rollback_use_the_request_id_when_lock_info_is_missing() {
|
||||
for deny_other_peer in [false, true] {
|
||||
let granted = ObservedBatchClient::new(false, None, None);
|
||||
granted.omit_lock_info.store(true, Ordering::SeqCst);
|
||||
let other = ObservedBatchClient::new(deny_other_peer, None, None);
|
||||
let lock = DistributedLock::new("missing-batch-info".to_string(), vec![granted.clone(), other.clone()], 2);
|
||||
let results = lock
|
||||
.acquire_guard_batch(
|
||||
&[ObjectKey::new("missing-batch-info", "object")],
|
||||
"owner",
|
||||
Duration::from_secs(5),
|
||||
Duration::from_secs(30),
|
||||
)
|
||||
.await;
|
||||
assert_eq!(results[0].is_err(), deny_other_peer);
|
||||
if !deny_other_peer {
|
||||
assert_eq!(granted.backend.list_lock_leases().await.len(), 1);
|
||||
assert_eq!(granted.cleanup_calls.load(Ordering::SeqCst), 0);
|
||||
}
|
||||
drop(results);
|
||||
granted.wait_for_releases(1).await;
|
||||
if !deny_other_peer {
|
||||
other.wait_for_releases(1).await;
|
||||
}
|
||||
assert!(granted.backend.list_lock_leases().await.is_empty());
|
||||
assert!(other.backend.list_lock_leases().await.is_empty());
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test(start_paused = true)]
|
||||
async fn cancelled_batch_rollback_retries_lost_ack_without_releasing_a_successor() {
|
||||
let granted = ObservedBatchClient::new(false, None, None);
|
||||
granted.lose_release_ack.store(true, Ordering::SeqCst);
|
||||
let refused = ObservedBatchClient::new(true, None, None);
|
||||
let resource = ObjectKey::new("batch-lost-ack", "object");
|
||||
let lock = DistributedLock::new("batch-lost-ack".to_string(), vec![granted.clone(), refused], 2);
|
||||
let acquire_resource = resource.clone();
|
||||
let operation = tokio::spawn(async move {
|
||||
lock.acquire_guard_batch(&[acquire_resource], "predecessor", Duration::from_secs(5), Duration::from_secs(30))
|
||||
.await
|
||||
});
|
||||
granted.wait_for_releases(1).await;
|
||||
let successor = LockRequest::new(resource, LockType::Exclusive, "successor")
|
||||
.with_acquire_timeout(Duration::from_secs(5))
|
||||
.with_ttl(Duration::from_secs(30));
|
||||
assert!(
|
||||
granted
|
||||
.backend
|
||||
.acquire_lock(&successor)
|
||||
.await
|
||||
.expect("successor acquires after physical release")
|
||||
.success
|
||||
);
|
||||
operation.abort();
|
||||
assert!(
|
||||
operation
|
||||
.await
|
||||
.expect_err("cancel caller while release ACK is blocked")
|
||||
.is_cancelled()
|
||||
);
|
||||
granted.release_ack_gate.add_permits(1);
|
||||
|
||||
ObservedBatchClient::wait_for_count(&granted.cleanup_calls, &granted.cleanup_changed, 2).await;
|
||||
let cleanup_ids = granted.cleanup_ids.lock().expect("cleanup ID log").clone();
|
||||
assert!(cleanup_ids.len() >= 2, "the uncertain ACK must be retried");
|
||||
assert!(
|
||||
cleanup_ids.iter().all(|id| id == &cleanup_ids[0]),
|
||||
"cleanup retains the exact predecessor ID"
|
||||
);
|
||||
assert_ne!(cleanup_ids[0], successor.lock_id);
|
||||
assert!(
|
||||
granted
|
||||
.backend
|
||||
.check_status(&successor.lock_id)
|
||||
.await
|
||||
.expect("check successor")
|
||||
.is_some()
|
||||
);
|
||||
assert_eq!(granted.backend.list_lock_leases().await.len(), 1);
|
||||
assert_eq!(
|
||||
granted
|
||||
.backend
|
||||
.cleanup_grant(&successor.lock_id)
|
||||
.await
|
||||
.expect("clean successor test grant"),
|
||||
LockCleanupOutcome::Released
|
||||
);
|
||||
}
|
||||
@@ -1,687 +0,0 @@
|
||||
// Copyright 2024 RustFS Team
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
use super::{BatchLockPeerDiagnostics, DistributedLock};
|
||||
use crate::{
|
||||
BatchAcquireOutcome, GlobalLockManager, LocalClient, LockCleanupOutcome, LockError, LockId, LockInfo, LockRequest,
|
||||
LockResponse, LockStats, ObjectKey, client::LockClient,
|
||||
};
|
||||
use std::{
|
||||
collections::VecDeque,
|
||||
sync::{
|
||||
Arc, Mutex,
|
||||
atomic::{AtomicBool, AtomicUsize, Ordering},
|
||||
},
|
||||
time::Duration,
|
||||
};
|
||||
use tokio::{
|
||||
sync::{Notify, Semaphore},
|
||||
time::Instant,
|
||||
};
|
||||
|
||||
#[derive(Clone, Debug)]
|
||||
enum Plan {
|
||||
Acquire,
|
||||
Retryable,
|
||||
Terminal,
|
||||
Ambiguous,
|
||||
ReplyGate(Arc<Semaphore>),
|
||||
}
|
||||
|
||||
#[derive(Debug)]
|
||||
struct TypedClient {
|
||||
backend: LocalClient,
|
||||
plans: Mutex<VecDeque<Plan>>,
|
||||
calls: Mutex<Vec<Vec<LockId>>>,
|
||||
deadlines: Mutex<Vec<Instant>>,
|
||||
changed: Notify,
|
||||
grants: AtomicUsize,
|
||||
cleanup_calls: AtomicUsize,
|
||||
released: AtomicUsize,
|
||||
refreshes: AtomicUsize,
|
||||
refresh_allowed: AtomicBool,
|
||||
cleanup_failures: AtomicUsize,
|
||||
cleanup_gate: Option<Arc<Semaphore>>,
|
||||
local: bool,
|
||||
local_gate: Mutex<Option<Arc<Semaphore>>>,
|
||||
}
|
||||
|
||||
impl TypedClient {
|
||||
fn new(plans: Vec<Plan>, local: bool, cleanup_gate: Option<Arc<Semaphore>>) -> Arc<Self> {
|
||||
Arc::new(Self {
|
||||
backend: LocalClient::with_manager(Arc::new(GlobalLockManager::new())),
|
||||
plans: Mutex::new(plans.into()),
|
||||
calls: Mutex::new(Vec::new()),
|
||||
deadlines: Mutex::new(Vec::new()),
|
||||
changed: Notify::new(),
|
||||
grants: AtomicUsize::new(0),
|
||||
cleanup_calls: AtomicUsize::new(0),
|
||||
released: AtomicUsize::new(0),
|
||||
refreshes: AtomicUsize::new(0),
|
||||
refresh_allowed: AtomicBool::new(true),
|
||||
cleanup_failures: AtomicUsize::new(0),
|
||||
cleanup_gate,
|
||||
local,
|
||||
local_gate: Mutex::new(None),
|
||||
})
|
||||
}
|
||||
|
||||
async fn wait_until(&self, predicate: impl Fn() -> bool) {
|
||||
tokio::time::timeout(Duration::from_secs(1), async {
|
||||
loop {
|
||||
let notified = self.changed.notified();
|
||||
if predicate() {
|
||||
return;
|
||||
}
|
||||
notified.await;
|
||||
}
|
||||
})
|
||||
.await
|
||||
.expect("expected a real acquisition or cleanup event");
|
||||
}
|
||||
|
||||
async fn wait_calls(&self, count: usize) {
|
||||
self.wait_until(|| self.calls.lock().expect("call log").len() >= count).await;
|
||||
}
|
||||
|
||||
async fn wait_grants(&self, count: usize) {
|
||||
self.wait_until(|| self.grants.load(Ordering::SeqCst) >= count).await;
|
||||
}
|
||||
|
||||
async fn wait_releases(&self, count: usize) {
|
||||
self.wait_until(|| self.released.load(Ordering::SeqCst) >= count).await;
|
||||
}
|
||||
}
|
||||
|
||||
#[async_trait::async_trait]
|
||||
impl LockClient for TypedClient {
|
||||
async fn acquire_lock(&self, request: &LockRequest) -> crate::Result<LockResponse> {
|
||||
self.backend.acquire_lock(request).await
|
||||
}
|
||||
async fn acquire_locks_batch_until(&self, requests: &[LockRequest], deadline: Instant) -> Vec<BatchAcquireOutcome> {
|
||||
self.calls
|
||||
.lock()
|
||||
.expect("call log")
|
||||
.push(requests.iter().map(|request| request.lock_id.clone()).collect());
|
||||
self.deadlines.lock().expect("deadline log").push(deadline);
|
||||
self.changed.notify_one();
|
||||
let plan = self.plans.lock().expect("plan queue").pop_front().unwrap_or(Plan::Acquire);
|
||||
match plan {
|
||||
Plan::Retryable => requests
|
||||
.iter()
|
||||
.map(|_| BatchAcquireOutcome::RetryableNoGrant {
|
||||
error: LockError::internal("explicit no grant"),
|
||||
})
|
||||
.collect(),
|
||||
Plan::Terminal => requests
|
||||
.iter()
|
||||
.map(|_| BatchAcquireOutcome::TerminalNoGrant {
|
||||
error: LockError::permission_denied("explicit refusal"),
|
||||
})
|
||||
.collect(),
|
||||
Plan::Ambiguous => requests
|
||||
.iter()
|
||||
.map(|_| BatchAcquireOutcome::Ambiguous {
|
||||
error: LockError::internal("unknown outcome"),
|
||||
cleanup_owned_by_client: true,
|
||||
})
|
||||
.collect(),
|
||||
Plan::Acquire | Plan::ReplyGate(_) => {
|
||||
let outcomes = self.backend.acquire_locks_batch_until(requests, deadline).await;
|
||||
self.grants.fetch_add(
|
||||
outcomes
|
||||
.iter()
|
||||
.filter(|outcome| matches!(outcome, BatchAcquireOutcome::Granted { .. }))
|
||||
.count(),
|
||||
Ordering::SeqCst,
|
||||
);
|
||||
self.changed.notify_one();
|
||||
if let Plan::ReplyGate(gate) = plan {
|
||||
gate.acquire().await.expect("response gate").forget();
|
||||
}
|
||||
outcomes
|
||||
}
|
||||
}
|
||||
}
|
||||
async fn cleanup_grant(&self, id: &LockId) -> crate::Result<LockCleanupOutcome> {
|
||||
self.cleanup_calls.fetch_add(1, Ordering::SeqCst);
|
||||
self.changed.notify_one();
|
||||
if let Some(gate) = &self.cleanup_gate {
|
||||
gate.acquire().await.expect("cleanup gate").forget();
|
||||
}
|
||||
if self
|
||||
.cleanup_failures
|
||||
.try_update(Ordering::SeqCst, Ordering::SeqCst, |count| count.checked_sub(1))
|
||||
.is_ok()
|
||||
{
|
||||
return Err(LockError::internal("injected cleanup failure"));
|
||||
}
|
||||
let result = self.backend.cleanup_grant(id).await?;
|
||||
if result == LockCleanupOutcome::Released {
|
||||
self.released.fetch_add(1, Ordering::SeqCst);
|
||||
self.changed.notify_one();
|
||||
}
|
||||
Ok(result)
|
||||
}
|
||||
async fn release(&self, id: &LockId) -> crate::Result<bool> {
|
||||
self.cleanup_grant(id)
|
||||
.await
|
||||
.map(|outcome| outcome == LockCleanupOutcome::Released)
|
||||
}
|
||||
async fn refresh(&self, id: &LockId) -> crate::Result<bool> {
|
||||
self.refreshes.fetch_add(1, Ordering::SeqCst);
|
||||
self.changed.notify_one();
|
||||
if self.refresh_allowed.load(Ordering::SeqCst) {
|
||||
self.backend.refresh(id).await
|
||||
} else {
|
||||
Ok(false)
|
||||
}
|
||||
}
|
||||
async fn force_release(&self, id: &LockId) -> crate::Result<bool> {
|
||||
self.backend.force_release(id).await
|
||||
}
|
||||
async fn check_status(&self, id: &LockId) -> crate::Result<Option<LockInfo>> {
|
||||
self.backend.check_status(id).await
|
||||
}
|
||||
async fn get_stats(&self) -> crate::Result<LockStats> {
|
||||
self.backend.get_stats().await
|
||||
}
|
||||
async fn close(&self) -> crate::Result<()> {
|
||||
self.backend.close().await
|
||||
}
|
||||
async fn is_online(&self) -> bool {
|
||||
true
|
||||
}
|
||||
async fn is_local(&self) -> bool {
|
||||
let gate = self.local_gate.lock().expect("local lookup gate").take();
|
||||
if let Some(gate) = gate {
|
||||
gate.acquire().await.expect("local lookup gate open").forget();
|
||||
}
|
||||
self.local
|
||||
}
|
||||
}
|
||||
|
||||
fn lock_for(clients: &[Arc<TypedClient>], quorum: usize) -> DistributedLock {
|
||||
DistributedLock::new(
|
||||
"batch-retry".to_owned(),
|
||||
clients.iter().cloned().map(|client| client as Arc<dyn LockClient>).collect(),
|
||||
quorum,
|
||||
)
|
||||
}
|
||||
|
||||
#[tokio::test(start_paused = true)]
|
||||
async fn healthy_thousand_key_batch_keeps_one_call_per_peer() {
|
||||
let peers: Vec<_> = (0..4).map(|_| TypedClient::new(Vec::new(), true, None)).collect();
|
||||
let resources: Vec<_> = (0..1000)
|
||||
.map(|index| ObjectKey::new("healthy-batch", format!("{index}")))
|
||||
.collect();
|
||||
let results = lock_for(&peers, 3)
|
||||
.acquire_guard_batch(&resources, "owner", Duration::from_secs(5), Duration::from_secs(30))
|
||||
.await;
|
||||
assert!(results.iter().all(Result::is_ok));
|
||||
for peer in &peers {
|
||||
peer.wait_calls(1).await;
|
||||
let calls = peer.calls.lock().expect("call log");
|
||||
assert_eq!(calls.len(), 1);
|
||||
assert_eq!(calls[0].len(), 1000);
|
||||
}
|
||||
drop(results);
|
||||
}
|
||||
|
||||
#[tokio::test(start_paused = true)]
|
||||
async fn third_pending_vote_is_waited_for_even_with_an_ambiguous_peer() {
|
||||
for before_deadline in [true, false] {
|
||||
let gate = Arc::new(Semaphore::new(0));
|
||||
let peers = vec![
|
||||
TypedClient::new(Vec::new(), true, None),
|
||||
TypedClient::new(Vec::new(), true, None),
|
||||
TypedClient::new(vec![Plan::ReplyGate(gate.clone())], false, None),
|
||||
TypedClient::new(vec![Plan::Ambiguous], false, None),
|
||||
];
|
||||
let lock = lock_for(&peers, 3);
|
||||
let start = Instant::now();
|
||||
let operation = tokio::spawn(async move {
|
||||
lock.acquire_guard_batch(
|
||||
&[ObjectKey::new("pending-third", "key")],
|
||||
"owner",
|
||||
Duration::from_secs(5),
|
||||
Duration::from_secs(30),
|
||||
)
|
||||
.await
|
||||
});
|
||||
for peer in &peers[..3] {
|
||||
peer.wait_grants(1).await;
|
||||
}
|
||||
assert!(!operation.is_finished(), "two votes do not discard a pending third vote");
|
||||
if before_deadline {
|
||||
gate.add_permits(1);
|
||||
let result = operation.await.expect("batch task");
|
||||
assert!(result[0].is_ok());
|
||||
assert!(start.elapsed() < Duration::from_secs(5));
|
||||
drop(result);
|
||||
} else {
|
||||
tokio::time::advance(Duration::from_secs(5)).await;
|
||||
let result = operation.await.expect("deadline returns without the delayed reply");
|
||||
assert!(result[0].is_err());
|
||||
gate.add_permits(1);
|
||||
}
|
||||
for peer in &peers[..3] {
|
||||
peer.wait_releases(1).await;
|
||||
assert!(peer.backend.list_lock_leases().await.is_empty());
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test(start_paused = true)]
|
||||
async fn ambiguous_peer_is_excluded_and_confirmed_subset_retries_with_fresh_ids() {
|
||||
for terminal in [false, true] {
|
||||
let peers = vec![
|
||||
TypedClient::new(vec![Plan::Ambiguous], false, None),
|
||||
TypedClient::new(Vec::new(), true, None),
|
||||
TypedClient::new(Vec::new(), true, None),
|
||||
TypedClient::new(vec![if terminal { Plan::Terminal } else { Plan::Retryable }], true, None),
|
||||
];
|
||||
peers[1].cleanup_failures.store(1, Ordering::SeqCst);
|
||||
let result = lock_for(&peers, 3)
|
||||
.acquire_guard_batch(
|
||||
&[ObjectKey::new("subset-retry", "key")],
|
||||
"owner",
|
||||
Duration::from_secs(5),
|
||||
Duration::from_secs(30),
|
||||
)
|
||||
.await;
|
||||
assert_eq!(result[0].is_err(), terminal);
|
||||
assert_eq!(
|
||||
peers[0].calls.lock().expect("ambiguous call log").len(),
|
||||
1,
|
||||
"uncertain peer never receives another attempt"
|
||||
);
|
||||
for peer in &peers[1..] {
|
||||
let calls = peer.calls.lock().expect("healthy call log");
|
||||
assert_eq!(calls.len(), if terminal { 1 } else { 2 });
|
||||
if !terminal {
|
||||
assert_ne!(calls[0][0], calls[1][0], "each retry uses a fresh exact ID");
|
||||
}
|
||||
}
|
||||
assert!(
|
||||
peers[1].cleanup_calls.load(Ordering::SeqCst) >= 2,
|
||||
"first failed cleanup is retried before a new acquire"
|
||||
);
|
||||
assert_eq!(peers[1].released.load(Ordering::SeqCst), 1);
|
||||
assert_eq!(peers[2].released.load(Ordering::SeqCst), 1);
|
||||
drop(result);
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test(start_paused = true)]
|
||||
async fn hot_key_retries_healthy_subset_without_waiting_for_silent_peer() {
|
||||
let peers: Vec<_> = (0..3).map(|_| TypedClient::new(Vec::new(), true, None)).collect();
|
||||
let holder_lock = lock_for(&peers, 3);
|
||||
let hot = ObjectKey::new("hot-subset", "hot");
|
||||
let holder = holder_lock
|
||||
.lock_guard(hot.clone(), "holder", Duration::from_secs(5), Duration::from_secs(30))
|
||||
.await
|
||||
.expect("legitimate distributed holder")
|
||||
.expect("holder reaches q3");
|
||||
let late_gate = Arc::new(Semaphore::new(0));
|
||||
let late = TypedClient::new(vec![Plan::ReplyGate(late_gate.clone())], false, None);
|
||||
let mut batch_peers = peers.clone();
|
||||
batch_peers.push(late.clone());
|
||||
let lock = lock_for(&batch_peers, 3);
|
||||
let start = Instant::now();
|
||||
let resources: Vec<_> = std::iter::once(hot.clone())
|
||||
.chain((0..999).map(|index| ObjectKey::new("hot-subset", format!("free-{index}"))))
|
||||
.collect();
|
||||
let release_holder = tokio::spawn(async move {
|
||||
tokio::time::sleep_until(start + Duration::from_millis(1500)).await;
|
||||
drop(holder);
|
||||
});
|
||||
let results = lock
|
||||
.acquire_guard_batch(&resources, "owner", Duration::from_secs(5), Duration::from_secs(30))
|
||||
.await;
|
||||
release_holder.await.expect("holder release");
|
||||
assert!(
|
||||
results.iter().all(Result::is_ok),
|
||||
"hot key recovers after the first local window, while the fourth peer is still silent"
|
||||
);
|
||||
assert!(start.elapsed() < Duration::from_secs(5));
|
||||
assert_eq!(late.calls.lock().expect("silent peer call log").len(), 1);
|
||||
for peer in &peers {
|
||||
let calls = peer.calls.lock().expect("call log");
|
||||
assert_eq!(calls[0].len(), 1000);
|
||||
assert!(calls.len() >= 2);
|
||||
assert!(
|
||||
calls[1..].iter().all(|batch| batch.len() == 1 && batch[0].resource == hot),
|
||||
"only the unresolved hot key is retried"
|
||||
);
|
||||
}
|
||||
late_gate.add_permits(1);
|
||||
late.wait_releases(1000).await;
|
||||
drop(results);
|
||||
}
|
||||
|
||||
#[tokio::test(start_paused = true)]
|
||||
async fn successful_keys_are_renewed_while_hot_key_uses_the_total_budget() {
|
||||
for refresh_allowed in [true, false] {
|
||||
let peers: Vec<_> = (0..3).map(|_| TypedClient::new(Vec::new(), true, None)).collect();
|
||||
let hot = ObjectKey::new("retained-guard", "hot");
|
||||
let holder = lock_for(&peers, 3)
|
||||
.lock_guard(hot.clone(), "holder", Duration::from_secs(5), Duration::from_secs(30))
|
||||
.await
|
||||
.expect("distributed holder")
|
||||
.expect("q3 holder");
|
||||
for peer in &peers {
|
||||
peer.refresh_allowed.store(refresh_allowed, Ordering::SeqCst);
|
||||
}
|
||||
let start = Instant::now();
|
||||
let results = lock_for(&peers, 3)
|
||||
.acquire_guard_batch(
|
||||
&[hot, ObjectKey::new("retained-guard", "free")],
|
||||
"owner",
|
||||
Duration::from_secs(5),
|
||||
Duration::from_secs(2),
|
||||
)
|
||||
.await;
|
||||
assert!(results[0].is_err(), "holder is never released inside the acquisition budget");
|
||||
assert!(
|
||||
peers.iter().any(|peer| peer.refreshes.load(Ordering::SeqCst) > 0),
|
||||
"cold key must form an early guard before D_total: calls={:?}, grants={:?}, refreshes={:?}",
|
||||
peers
|
||||
.iter()
|
||||
.map(|peer| peer.calls.lock().expect("call log").len())
|
||||
.collect::<Vec<_>>(),
|
||||
peers
|
||||
.iter()
|
||||
.map(|peer| peer.grants.load(Ordering::SeqCst))
|
||||
.collect::<Vec<_>>(),
|
||||
peers
|
||||
.iter()
|
||||
.map(|peer| peer.refreshes.load(Ordering::SeqCst))
|
||||
.collect::<Vec<_>>(),
|
||||
);
|
||||
assert_eq!(
|
||||
results[1].is_ok(),
|
||||
refresh_allowed,
|
||||
"the final result rejects a previously successful guard after quorum is lost"
|
||||
);
|
||||
assert_eq!(start.elapsed(), Duration::from_secs(5));
|
||||
if refresh_allowed {
|
||||
assert!(
|
||||
peers.iter().all(|peer| peer.refreshes.load(Ordering::SeqCst) > 1),
|
||||
"healthy early guards repeatedly renew every quorum peer during other keys' retries"
|
||||
);
|
||||
}
|
||||
drop(results);
|
||||
drop(holder);
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test(start_paused = true)]
|
||||
async fn rollback_deadline_detaches_cleanup_without_reacquiring() {
|
||||
let gate = Arc::new(Semaphore::new(0));
|
||||
let peers = vec![
|
||||
TypedClient::new(vec![Plan::Ambiguous], false, None),
|
||||
TypedClient::new(Vec::new(), true, Some(gate.clone())),
|
||||
TypedClient::new(Vec::new(), true, None),
|
||||
TypedClient::new(vec![Plan::Retryable], true, None),
|
||||
];
|
||||
let start = Instant::now();
|
||||
let result = lock_for(&peers, 3)
|
||||
.acquire_guard_batch(
|
||||
&[ObjectKey::new("rollback-deadline", "key")],
|
||||
"owner",
|
||||
Duration::from_millis(500),
|
||||
Duration::from_secs(30),
|
||||
)
|
||||
.await;
|
||||
assert!(result[0].is_err());
|
||||
assert_eq!(start.elapsed(), Duration::from_millis(500));
|
||||
assert!(peers.iter().all(|peer| peer.calls.lock().expect("call log").len() == 1));
|
||||
assert_eq!(peers[1].backend.list_lock_leases().await.len(), 1);
|
||||
gate.add_permits(1);
|
||||
peers[1].wait_releases(1).await;
|
||||
assert!(peers[1].backend.list_lock_leases().await.is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn batch_diagnostics_preserve_typed_no_grant_and_ambiguity() {
|
||||
let outcomes = vec![
|
||||
BatchAcquireOutcome::Granted {
|
||||
lock_info: None,
|
||||
wait_time: Duration::ZERO,
|
||||
},
|
||||
BatchAcquireOutcome::RetryableNoGrant {
|
||||
error: LockError::internal("explicit"),
|
||||
},
|
||||
BatchAcquireOutcome::TerminalNoGrant {
|
||||
error: LockError::internal("terminal"),
|
||||
},
|
||||
BatchAcquireOutcome::Ambiguous {
|
||||
error: LockError::internal("unknown"),
|
||||
cleanup_owned_by_client: false,
|
||||
},
|
||||
];
|
||||
let diagnostics = BatchLockPeerDiagnostics::from_outcomes(1, 5, &outcomes, Duration::ZERO);
|
||||
assert_eq!(diagnostics.granted_keys, 1);
|
||||
assert_eq!(diagnostics.retryable_no_grant_keys, 1);
|
||||
assert_eq!(diagnostics.terminal_no_grant_keys, 1);
|
||||
assert_eq!(diagnostics.ambiguous_keys, 1);
|
||||
assert_eq!(diagnostics.missing_response_keys, 1);
|
||||
}
|
||||
|
||||
#[derive(Debug)]
|
||||
struct LegacyHiddenClient {
|
||||
backend: LocalClient,
|
||||
requested: Mutex<Vec<LockId>>,
|
||||
release_attempts: Mutex<std::collections::HashMap<LockId, usize>>,
|
||||
cleanup_calls: AtomicUsize,
|
||||
changed: Notify,
|
||||
}
|
||||
|
||||
#[async_trait::async_trait]
|
||||
impl LockClient for LegacyHiddenClient {
|
||||
async fn acquire_lock(&self, request: &LockRequest) -> crate::Result<LockResponse> {
|
||||
self.requested.lock().expect("request IDs").push(request.lock_id.clone());
|
||||
if request.resource.object.as_ref() == "error" {
|
||||
return Err(LockError::internal("one legacy item failed"));
|
||||
}
|
||||
self.backend.acquire_lock(request).await
|
||||
}
|
||||
async fn release(&self, id: &LockId) -> crate::Result<bool> {
|
||||
let attempt = {
|
||||
let mut attempts = self.release_attempts.lock().expect("release attempts");
|
||||
let attempt = attempts.entry(id.clone()).or_default();
|
||||
*attempt += 1;
|
||||
*attempt
|
||||
};
|
||||
self.changed.notify_one();
|
||||
// An absence-looking first response cannot terminate uncertainty.
|
||||
if attempt == 1 {
|
||||
return Ok(false);
|
||||
}
|
||||
let result = self.backend.release(id).await;
|
||||
self.changed.notify_one();
|
||||
result
|
||||
}
|
||||
async fn cleanup_grant(&self, _id: &LockId) -> crate::Result<LockCleanupOutcome> {
|
||||
self.cleanup_calls.fetch_add(1, Ordering::SeqCst);
|
||||
Ok(LockCleanupOutcome::AlreadyAbsent)
|
||||
}
|
||||
async fn refresh(&self, id: &LockId) -> crate::Result<bool> {
|
||||
self.backend.refresh(id).await
|
||||
}
|
||||
async fn force_release(&self, id: &LockId) -> crate::Result<bool> {
|
||||
self.backend.force_release(id).await
|
||||
}
|
||||
async fn check_status(&self, id: &LockId) -> crate::Result<Option<LockInfo>> {
|
||||
self.backend.check_status(id).await
|
||||
}
|
||||
async fn get_stats(&self) -> crate::Result<LockStats> {
|
||||
self.backend.get_stats().await
|
||||
}
|
||||
async fn close(&self) -> crate::Result<()> {
|
||||
self.backend.close().await
|
||||
}
|
||||
async fn is_online(&self) -> bool {
|
||||
true
|
||||
}
|
||||
async fn is_local(&self) -> bool {
|
||||
true
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test(start_paused = true)]
|
||||
async fn legacy_collapsed_error_cleans_hidden_grant_without_accepting_absence() {
|
||||
let peer = Arc::new(LegacyHiddenClient {
|
||||
backend: LocalClient::with_manager(Arc::new(GlobalLockManager::new())),
|
||||
requested: Mutex::new(Vec::new()),
|
||||
release_attempts: Mutex::new(std::collections::HashMap::new()),
|
||||
cleanup_calls: AtomicUsize::new(0),
|
||||
changed: Notify::new(),
|
||||
});
|
||||
let lock = DistributedLock::new("legacy-hidden".to_owned(), vec![peer.clone()], 1);
|
||||
let results = lock
|
||||
.acquire_guard_batch(
|
||||
&[
|
||||
ObjectKey::new("legacy-hidden", "granted"),
|
||||
ObjectKey::new("legacy-hidden", "error"),
|
||||
],
|
||||
"owner",
|
||||
Duration::from_secs(5),
|
||||
Duration::from_secs(30),
|
||||
)
|
||||
.await;
|
||||
assert!(
|
||||
results.iter().all(Result::is_err),
|
||||
"the default collect error obscures the successful sibling"
|
||||
);
|
||||
tokio::time::timeout(Duration::from_millis(50), async {
|
||||
loop {
|
||||
let notified = peer.changed.notified();
|
||||
if peer.release_attempts.lock().expect("release attempts").len() == 2 {
|
||||
break;
|
||||
}
|
||||
notified.await;
|
||||
}
|
||||
})
|
||||
.await
|
||||
.expect("both uncertain IDs receive the first release pass");
|
||||
assert_eq!(
|
||||
peer.backend.list_lock_leases().await.len(),
|
||||
1,
|
||||
"false release does not prove this hidden grant absent"
|
||||
);
|
||||
assert_eq!(
|
||||
peer.cleanup_calls.load(Ordering::SeqCst),
|
||||
0,
|
||||
"uncertainty must never use authoritative-absence cleanup"
|
||||
);
|
||||
tokio::time::timeout(Duration::from_secs(1), async {
|
||||
loop {
|
||||
let notified = peer.changed.notified();
|
||||
if peer.backend.list_lock_leases().await.is_empty() {
|
||||
break;
|
||||
}
|
||||
notified.await;
|
||||
}
|
||||
})
|
||||
.await
|
||||
.expect("exact-ID release is retried and clears the hidden grant before its TTL");
|
||||
let requests = peer.requested.lock().expect("request IDs");
|
||||
let releases = peer.release_attempts.lock().expect("release attempts");
|
||||
assert_eq!(requests.len(), 2, "ambiguous requests are never reacquired");
|
||||
assert!(
|
||||
releases.keys().all(|id| requests.contains(id)),
|
||||
"cleanup only uses the original request IDs"
|
||||
);
|
||||
assert!(releases.values().all(|&attempts| attempts >= 2));
|
||||
}
|
||||
|
||||
#[tokio::test(start_paused = true)]
|
||||
async fn local_wait_deadline_is_fixed_before_scheduling_and_single_peer_keeps_total() {
|
||||
let gate = Arc::new(Semaphore::new(0));
|
||||
let local = TypedClient::new(Vec::new(), true, None);
|
||||
*local.local_gate.lock().expect("set lookup gate") = Some(gate.clone());
|
||||
let peers = vec![
|
||||
local.clone(),
|
||||
TypedClient::new(vec![Plan::Retryable], true, None),
|
||||
TypedClient::new(vec![Plan::Retryable], true, None),
|
||||
TypedClient::new(vec![Plan::Ambiguous], false, None),
|
||||
];
|
||||
let lock = lock_for(&peers, 3);
|
||||
let operation = tokio::spawn(async move {
|
||||
lock.acquire_guard_batch(
|
||||
&[ObjectKey::new("late-local", "key")],
|
||||
"owner",
|
||||
Duration::from_secs(5),
|
||||
Duration::from_secs(30),
|
||||
)
|
||||
.await
|
||||
});
|
||||
peers[1].wait_calls(1).await;
|
||||
peers[2].wait_calls(1).await;
|
||||
tokio::time::advance(Duration::from_millis(1250)).await;
|
||||
gate.add_permits(1);
|
||||
let result = operation.await.expect("retry after the delayed local lookup");
|
||||
assert!(result[0].is_ok());
|
||||
assert_eq!(
|
||||
local.calls.lock().expect("local calls").len(),
|
||||
1,
|
||||
"the expired first local round never calls acquire again with a fresh budget"
|
||||
);
|
||||
assert_eq!(peers[1].calls.lock().expect("peer calls").len(), 2);
|
||||
assert_eq!(peers[2].calls.lock().expect("peer calls").len(), 2);
|
||||
drop(result);
|
||||
|
||||
let single = TypedClient::new(Vec::new(), true, None);
|
||||
let start = Instant::now();
|
||||
let results = lock_for(std::slice::from_ref(&single), 1)
|
||||
.acquire_guard_batch(
|
||||
&[ObjectKey::new("single-local", "key")],
|
||||
"owner",
|
||||
Duration::from_secs(5),
|
||||
Duration::from_secs(30),
|
||||
)
|
||||
.await;
|
||||
assert!(results[0].is_ok());
|
||||
assert_eq!(single.deadlines.lock().expect("single deadline")[0], start + Duration::from_secs(5));
|
||||
drop(results);
|
||||
}
|
||||
|
||||
#[tokio::test(start_paused = true)]
|
||||
async fn total_deadline_never_dispatches_after_a_delayed_peer_lookup() {
|
||||
let gate = Arc::new(Semaphore::new(0));
|
||||
let peer = TypedClient::new(Vec::new(), false, None);
|
||||
*peer.local_gate.lock().expect("set lookup gate") = Some(gate.clone());
|
||||
let result = lock_for(std::slice::from_ref(&peer), 1)
|
||||
.acquire_guard_batch(
|
||||
&[ObjectKey::new("no-late-dispatch", "key")],
|
||||
"owner",
|
||||
Duration::from_secs(1),
|
||||
Duration::from_secs(30),
|
||||
)
|
||||
.await;
|
||||
assert!(result[0].is_err());
|
||||
gate.add_permits(1);
|
||||
// The lookup task is now runnable; yield until its gate permit was consumed.
|
||||
tokio::time::timeout(Duration::from_secs(1), async {
|
||||
while gate.available_permits() != 0 {
|
||||
tokio::task::yield_now().await;
|
||||
}
|
||||
})
|
||||
.await
|
||||
.expect("late lookup task resumes under its retained owner");
|
||||
assert!(peer.calls.lock().expect("peer calls").is_empty(), "no acquire call starts after D_total");
|
||||
assert!(peer.backend.list_lock_leases().await.is_empty());
|
||||
}
|
||||
@@ -38,7 +38,7 @@ pub mod types;
|
||||
// Re-export main types for easy access
|
||||
pub use crate::{
|
||||
// Client interfaces
|
||||
client::{BatchAcquireOutcome, LockCleanupOutcome, LockClient, local::LocalClient},
|
||||
client::{LockCleanupOutcome, LockClient, local::LocalClient},
|
||||
distributed_lock::DistributedLockGuard,
|
||||
// Error types
|
||||
error::{LockError, Result},
|
||||
|
||||
@@ -945,9 +945,6 @@ pub struct GenerallyLockResult {
|
||||
/// Per-ID result for unlock batches.
|
||||
#[prost(enumeration = "LockReleaseStatus", tag = "4")]
|
||||
pub release_status: i32,
|
||||
/// Only populated by lock batches.
|
||||
#[prost(enumeration = "LockAcquireStatus", tag = "5")]
|
||||
pub acquire_status: i32,
|
||||
}
|
||||
#[derive(Clone, PartialEq, ::prost::Message)]
|
||||
pub struct BatchGenerallyLockResponse {
|
||||
@@ -1915,36 +1912,6 @@ impl LockReleaseStatus {
|
||||
}
|
||||
}
|
||||
}
|
||||
/// Only explicit no-grant results permit a caller to retry without grant cleanup.
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash, PartialOrd, Ord, ::prost::Enumeration)]
|
||||
#[repr(i32)]
|
||||
pub enum LockAcquireStatus {
|
||||
Unspecified = 0,
|
||||
RetryableNoGrant = 1,
|
||||
RejectedNoGrant = 2,
|
||||
}
|
||||
impl LockAcquireStatus {
|
||||
/// String value of the enum field names used in the ProtoBuf definition.
|
||||
///
|
||||
/// The values are not transformed in any way and thus are considered stable
|
||||
/// (if the ProtoBuf definition does not change) and safe for programmatic use.
|
||||
pub fn as_str_name(&self) -> &'static str {
|
||||
match self {
|
||||
Self::Unspecified => "LOCK_ACQUIRE_STATUS_UNSPECIFIED",
|
||||
Self::RetryableNoGrant => "LOCK_ACQUIRE_STATUS_RETRYABLE_NO_GRANT",
|
||||
Self::RejectedNoGrant => "LOCK_ACQUIRE_STATUS_REJECTED_NO_GRANT",
|
||||
}
|
||||
}
|
||||
/// Creates an enum from field names used in the ProtoBuf definition.
|
||||
pub fn from_str_name(value: &str) -> ::core::option::Option<Self> {
|
||||
match value {
|
||||
"LOCK_ACQUIRE_STATUS_UNSPECIFIED" => Some(Self::Unspecified),
|
||||
"LOCK_ACQUIRE_STATUS_RETRYABLE_NO_GRANT" => Some(Self::RetryableNoGrant),
|
||||
"LOCK_ACQUIRE_STATUS_REJECTED_NO_GRANT" => Some(Self::RejectedNoGrant),
|
||||
_ => None,
|
||||
}
|
||||
}
|
||||
}
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash, PartialOrd, Ord, ::prost::Enumeration)]
|
||||
#[repr(i32)]
|
||||
pub enum TierMutationPeerState {
|
||||
|
||||
@@ -4093,7 +4093,7 @@ mod tests {
|
||||
|
||||
#[cfg(test)]
|
||||
mod lock_release_tests {
|
||||
use super::proto_gen::node_service::{GenerallyLockResponse, GenerallyLockResult, LockAcquireStatus, LockReleaseStatus};
|
||||
use super::proto_gen::node_service::{GenerallyLockResponse, GenerallyLockResult, LockReleaseStatus};
|
||||
use prost::Message;
|
||||
|
||||
/// The release response schema before structured outcomes were added.
|
||||
@@ -4140,7 +4140,6 @@ mod lock_release_tests {
|
||||
error_info: error.clone(),
|
||||
lock_info: None,
|
||||
release_status: status.into(),
|
||||
..Default::default()
|
||||
};
|
||||
for encoded in [response.encode_to_vec(), result.encode_to_vec()] {
|
||||
let legacy = LegacyReleaseResponse::decode(encoded.as_slice()).expect("old peer ignores additive outcome");
|
||||
@@ -4150,43 +4149,4 @@ mod lock_release_tests {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn batch_acquire_outcomes_are_additive_and_preserve_legacy_grants() {
|
||||
for success in [false, true] {
|
||||
let legacy = LegacyReleaseResponse {
|
||||
success,
|
||||
error_info: (!success).then(|| "legacy failure".to_string()),
|
||||
lock_info: None,
|
||||
};
|
||||
let result = GenerallyLockResult::decode(legacy.encode_to_vec().as_slice()).expect("decode legacy acquire");
|
||||
assert_eq!(result.success, success);
|
||||
assert_eq!(result.acquire_status(), LockAcquireStatus::Unspecified);
|
||||
}
|
||||
for status in [LockAcquireStatus::RetryableNoGrant, LockAcquireStatus::RejectedNoGrant] {
|
||||
let result = GenerallyLockResult {
|
||||
success: false,
|
||||
error_info: Some("not granted".to_string()),
|
||||
acquire_status: status.into(),
|
||||
..Default::default()
|
||||
};
|
||||
let legacy = LegacyReleaseResponse::decode(result.encode_to_vec().as_slice()).expect("decode additive acquire");
|
||||
assert!(!legacy.success);
|
||||
assert_eq!(legacy.error_info, result.error_info);
|
||||
let single =
|
||||
GenerallyLockResponse::decode(result.encode_to_vec().as_slice()).expect("single schema ignores batch field");
|
||||
assert_eq!(single.release_status(), LockReleaseStatus::Unspecified);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn batch_acquire_unknown_status_is_preserved_for_conservative_classification() {
|
||||
let result = GenerallyLockResult {
|
||||
acquire_status: 99,
|
||||
..Default::default()
|
||||
};
|
||||
let decoded = GenerallyLockResult::decode(result.encode_to_vec().as_slice()).expect("decode unknown enum");
|
||||
assert_eq!(decoded.acquire_status, 99);
|
||||
assert!(LockAcquireStatus::try_from(decoded.acquire_status).is_err());
|
||||
}
|
||||
}
|
||||
|
||||
@@ -662,19 +662,11 @@ message BatchGenerallyLockRequest {
|
||||
repeated string args = 1;
|
||||
}
|
||||
|
||||
// Only explicit no-grant results permit a caller to retry without grant cleanup.
|
||||
enum LockAcquireStatus {
|
||||
LOCK_ACQUIRE_STATUS_UNSPECIFIED = 0;
|
||||
LOCK_ACQUIRE_STATUS_RETRYABLE_NO_GRANT = 1;
|
||||
LOCK_ACQUIRE_STATUS_REJECTED_NO_GRANT = 2;
|
||||
}
|
||||
|
||||
message GenerallyLockResult {
|
||||
bool success = 1;
|
||||
optional string error_info = 2;
|
||||
optional string lock_info = 3; // JSON serialized LockInfo
|
||||
LockReleaseStatus release_status = 4; // Per-ID result for unlock batches.
|
||||
LockAcquireStatus acquire_status = 5; // Only populated by lock batches.
|
||||
}
|
||||
|
||||
message BatchGenerallyLockResponse {
|
||||
|
||||
@@ -8,6 +8,8 @@ collection:
|
||||
|
||||
## Operations
|
||||
|
||||
- [Candidate release pipeline](operations/release-pipeline.md): frozen inputs, staged acceptance, immutable publication and recovery.
|
||||
|
||||
- [Sampled local batch CPU diagnostics](operations/local-batch-cpu-diagnostics.md) — default-off request-correlated allocation and worker wall/CPU diagnosis.
|
||||
|
||||
- [Retained MRF responsibilities](operations/mrf-responsibilities.md) — unknown bucket identities, bounded probes, and unreadable checkpoint recovery.
|
||||
|
||||
@@ -0,0 +1,79 @@
|
||||
# Candidate acceptance and release publication
|
||||
|
||||
RustFS builds the final product version before publication. A candidate is identified by its Actions run and `<version>-preview.<run_number>`; this identifier is not a Git tag or the executable's displayed version. Failed candidates consume no public version. A public tag is created only after the complete artifact manifest is approved.
|
||||
|
||||
## Entry points
|
||||
|
||||
| Workflow | Input | Result | Public writes |
|
||||
|---|---|---|---|
|
||||
| `release-candidate.yml` | Exact final version and full source SHA | Four Linux archives, native checks, upgrade results, Linux manifest | None |
|
||||
| `release-complete.yml` | Successful candidate run, accepted Linux manifest SHA-256, backlog acceptance report | Desktop archives, DEB/RPM, OCI images, Helm chart, full certification and sealed manifest | None |
|
||||
| `release-publish.yml` | Successful completion run and approved final manifest SHA-256 | Annotated tag, GitHub Release, R2 files, registry images, stable pointers and Helm index | Explicit publication |
|
||||
|
||||
The builder workflows (`build.yml`, `package.yml`, `docker.yml`, `helm-package.yml`) are reusable stages. Only `release-publish.yml` has release write credentials. Development builds retain their artifact names and download channel. Daily development builds now select Linux only; macOS/Windows development artifacts are refreshed by an explicit manual build, and both remain mandatory for a completed release candidate. Tag pushes no longer start a second compilation.
|
||||
|
||||
## Freeze and build
|
||||
|
||||
Select a reviewed source commit whose Cargo version equals the intended final version. Finish exact-source CI before dispatching the candidate. The candidate workflow itself must run from upstream main. Unrelated main commits do not require rebuilding an accepted source.
|
||||
|
||||
```bash
|
||||
gh workflow run release-candidate.yml --ref main \
|
||||
-f version='<version>' -f source_sha='<full-source-sha>'
|
||||
```
|
||||
|
||||
The initial job records the source and workflow SHAs, release recipe, dependency lock, exact stable Rust version, Console tag/asset/SHA-256, CLI assets, previous stable release and validation plan. All platforms consume that record; none independently follows `latest`. The source commit must contain the same reviewed release recipe as the dispatcher. Relevant recipe changes require a new candidate; unrelated source changes on main do not invalidate the frozen record.
|
||||
|
||||
Linux is built first. Each archived executable must display the final version, expected target, source commit and compiler version, then start natively, serve the Console, and pass an authenticated S3 bucket/object round trip. The upgrade lane runs the existing pinned compatibility matrix plus the stable version frozen at candidate creation. It compiles the harness and runs the accepted executable; it does not rebuild a substitute server. A zero-test invocation fails.
|
||||
|
||||
The Linux manifest is partial evidence and cannot authorize publication. Download the exact artifacts for field acceptance. Record the source, manifest digest, run URL and actual tested binary digests in backlog. Required field acceptance includes Scanner/Heal and multi-pool behavior, major features, memory, high-concurrency 503s, performance against the agreed baseline, previous-version data integrity and upgrade recovery. The agreed stability duration is a release-specific requirement, not an implicit workflow default.
|
||||
|
||||
## Complete the accepted candidate
|
||||
|
||||
After the Linux report passes, dispatch `release-complete.yml` with the exact digest printed by the candidate run. The `release-linux-acceptance` environment is the operator gate. Configure its protection and allowed main branch in repository settings; a typed digest and report identify what is being accepted, while the workflow independently revalidates automated evidence.
|
||||
|
||||
```bash
|
||||
gh workflow run release-complete.yml --ref main \
|
||||
-f candidate_run_id='<run-id>' \
|
||||
-f linux_manifest_sha256='<accepted-linux-manifest-sha256>' \
|
||||
-f acceptance_report='https://github.com/rustfs/backlog/issues/<issue>#issuecomment-<comment>'
|
||||
```
|
||||
|
||||
macOS and Windows now build and run the same native checks. Packages and images wrap the accepted Linux binaries. DEB/RPM extraction must reproduce the recorded binary digest. Both OCI variants are checked on amd64 and arm64 and scanned before sealing. Vulnerability scans retain the repository's report-only policy; scanner execution errors still fail the stage. Image OS packages are resolved during packaging, then the actual resulting OCI bytes are accepted and preserved; this is not a promise of reproducible package-manager output.
|
||||
|
||||
The Helm chart is tested and packaged before sealing, with chart and application versions equal to the final product version. Full certification requires every platform and upgrade case from the original plan. The seal includes artifact hashes, raw evidence, source/input identities, acceptance report, checksums, SBOM and provenance. Successful Linux receipts remain valid while desktop results are added because each receipt binds its own exact archive and executable, the frozen plan, candidate identity and trusted producer.
|
||||
|
||||
Native smoke checks are not a substitute for Console UI and rc acceptance. Complete the release skill's UI/client checks against these exact artifacts, including desktop checks applicable to the release, before asking for final publication approval.
|
||||
|
||||
## Approve and publish
|
||||
|
||||
Review the complete run and final `release-manifest.json`. Human approval binds its SHA-256, source and version. A changed manifest requires renewed approval. No source version bump, compilation, package generation or image build occurs during publication.
|
||||
|
||||
```bash
|
||||
gh workflow run release-publish.yml --ref main \
|
||||
-f complete_run_id='<successful-completion-run-id>' \
|
||||
-f manifest_sha256='<approved-final-manifest-sha256>'
|
||||
```
|
||||
|
||||
The publisher downloads the artifact from the exact successful main workflow run, checks GitHub's artifact digest, recomputes file hashes and reruns complete certification. It creates an annotated tag whose message binds the source and manifest. An existing tag or asset with different bytes is a hard failure; tags are never moved and uploads never clobber assets.
|
||||
|
||||
Versioned GitHub assets, R2 objects and Docker Hub/GHCR/Quay manifests are uploaded and read back before the Release becomes public. OCI promotion uses `--preserve-digests`; registry conversion fails rather than silently accepting a new image. R2 creation uses atomic `If-None-Match: *`. The small evidence bundle persists with the public Release after Actions artifacts expire. Provenance is descriptive, not a claim of cryptographic attestation.
|
||||
|
||||
Publication uses a per-version lock and a shared channel lock, both without cancelling an active run. Stable channels advance only after versioned delivery succeeds. GitHub Latest acts as a monotonic marker during partial channel updates, so an older retry cannot downgrade Docker tags, R2 aliases or installer pointers. Prereleases do not advance stable channels. Helm publishes the already accepted chart after versioned images exist. GitHub's legacy `*-latest.zip` download names remain aliases of the accepted bytes; they become default downloads only when GitHub Latest advances.
|
||||
|
||||
Cross-registry publication is not atomic. A channel job can fail after some verified destinations advance. Resume the same run/seal: matching destinations are reused and conflicting ones stop the workflow. Retrying never recompiles. Do not claim complete publication until the channel receipt and all public endpoints are verified.
|
||||
|
||||
## Failure and retention
|
||||
|
||||
- Infrastructure failure: rerun failed jobs in the same run. Frozen inputs, successful platform/package/image artifacts and completed seals are reused. Missing or expired artifacts fail closed.
|
||||
- Product or build-input change before tagging: create a new candidate run and repeat affected acceptance. Do not mutate the old candidate.
|
||||
- Failure after tagging: resume publication of exactly that seal. A code fix requires a new patch version; never move the tag.
|
||||
- New main commits: retain evidence for unchanged source, recipe and inputs. Recheck only affected boundaries; do not restart the entire matrix because an unrelated PR merged.
|
||||
- Artifacts are retained for 30 days while awaiting acceptance. Public-repository Actions artifacts and staging identifiers are not confidential storage. Preserve important failure evidence in task-owned storage and backlog.
|
||||
|
||||
## Migration and workflow maintenance
|
||||
|
||||
Finish any in-flight legacy tag release, including its downstream Docker/package/Helm jobs, before enabling this cutover on main. Existing tags and Releases retain their identities. Do not feed a legacy public preview binary into the new promotion path: it embeds a different version and was not built from the frozen-input contract.
|
||||
|
||||
All workflow responsibilities, triggers and schedule ownership remain registered in `.github/scheduled-validations.json`. The migration does not re-enable workflows disabled in GitHub. PR checks, cache warming, nightly development builds, functional chains, lab tests and monitoring retain their distinct purposes; shared source or cache does not make different feature/platform tests redundant. The watchdog allocates a runner only for relevant unsuccessful scheduled/chained runs, while its registry still determines ownership.
|
||||
|
||||
Run `actionlint`, the release contract suite and the workflow registry validator after changes. The Security Audit workflow also exercises actual GitHub artifact transport, DEB/RPM package extraction, both multi-platform OCI builders, local registry promotion and archive scanning. Native packaging jobs run only for relevant changes or a manual audit. Their payloads are explicit transport fixtures, not product release acceptance. See the [release skill](../../.agents/skills/rustfs-release-publish/SKILL.md) for field acceptance, human confirmation and post-release follow-up.
|
||||
@@ -77,7 +77,7 @@
|
||||
|
||||
rustfs = rustPlatform.buildRustPackage {
|
||||
pname = "rustfs";
|
||||
version = "1.0.4";
|
||||
version = "1.0.5";
|
||||
|
||||
src = ./.;
|
||||
|
||||
|
||||
@@ -2,8 +2,8 @@ apiVersion: v2
|
||||
name: rustfs
|
||||
description: RustFS helm chart to deploy RustFS on kubernetes cluster.
|
||||
type: application
|
||||
version: "1.0.4"
|
||||
appVersion: "1.0.4"
|
||||
version: "1.0.5"
|
||||
appVersion: "1.0.5"
|
||||
home: https://rustfs.com
|
||||
icon: https://media.sys.truenas.net/apps/rustfs/icons/icon.svg
|
||||
maintainers:
|
||||
|
||||
+4
-1
@@ -1,7 +1,7 @@
|
||||
%global _enable_debug_packages 0
|
||||
%global _empty_manifest_terminate_build 0
|
||||
Name: rustfs
|
||||
Version: 1.0.4
|
||||
Version: 1.0.5
|
||||
Release: 1
|
||||
Summary: High-performance distributed object storage for MinIO alternative
|
||||
|
||||
@@ -57,6 +57,9 @@ install %_builddir/%{name}-%{version}/target/%_arch/%_arch-unknown-linux-gnu/rel
|
||||
%_bindir/rustfs
|
||||
|
||||
%changelog
|
||||
* Wed Nov 11 2026 overtrue <anzhengchao@gmail.com>
|
||||
- Update RPM package to RustFS 1.0.5
|
||||
|
||||
* Mon Oct 26 2026 overtrue <anzhengchao@gmail.com>
|
||||
- Update RPM package to RustFS 1.0.4
|
||||
|
||||
|
||||
@@ -14,13 +14,10 @@
|
||||
|
||||
//! DeleteObject / DeleteObjects path.
|
||||
|
||||
use super::shared::{LOG_COMPONENT_APP, LOG_SUBSYSTEM_OBJECT};
|
||||
use super::*;
|
||||
use crate::app::storage_api::object_usecase::DeleteObjectCondition;
|
||||
use s3s::dto::ObjectIdentifier;
|
||||
use std::collections::{BTreeMap, HashSet};
|
||||
|
||||
const EVENT_DELETE_OBJECTS_STORAGE_RESULT: &str = "delete_objects_storage_result";
|
||||
use std::collections::HashSet;
|
||||
|
||||
async fn authorize_recursive_delete<T>(
|
||||
req: &mut S3Request<T>,
|
||||
@@ -508,8 +505,9 @@ fn reduce_delete_objects_result<'a>(
|
||||
error: Option<&EcstoreError>,
|
||||
synthetic_version_id: bool,
|
||||
) -> Result<&'a StorageDeletedObject, s3s::dto::Error> {
|
||||
match delete_objects_reported_storage_error(object, error) {
|
||||
match error {
|
||||
None => Ok(deleted),
|
||||
Some(error) if object.delete_condition.is_none() && is_delete_objects_not_found(error) => Ok(deleted),
|
||||
Some(error) => {
|
||||
let api_error = ApiError::from(error.clone());
|
||||
Err(s3s::dto::Error {
|
||||
@@ -522,48 +520,6 @@ fn reduce_delete_objects_result<'a>(
|
||||
}
|
||||
}
|
||||
|
||||
fn delete_objects_reported_storage_error<'a>(
|
||||
object: &ObjectToDelete,
|
||||
error: Option<&'a EcstoreError>,
|
||||
) -> Option<&'a EcstoreError> {
|
||||
error.filter(|error| object.delete_condition.is_some() || !is_delete_objects_not_found(error))
|
||||
}
|
||||
|
||||
fn delete_objects_storage_error_counts(
|
||||
objects: &[ObjectToDelete],
|
||||
errors: &[Option<EcstoreError>],
|
||||
) -> BTreeMap<&'static str, usize> {
|
||||
let mut counts = BTreeMap::new();
|
||||
for (object, error) in objects.iter().zip(errors) {
|
||||
let Some(error) = delete_objects_reported_storage_error(object, error.as_ref()) else {
|
||||
continue;
|
||||
};
|
||||
// Classify the typed cause before S3 mapping collapses distinct failures.
|
||||
// Stages are recorded at their source, never inferred from this variant.
|
||||
let kind = match error {
|
||||
StorageError::NamespaceLockQuorumUnavailable { .. } => "namespace_lock_quorum",
|
||||
StorageError::Lock(_) => "lock",
|
||||
StorageError::ErasureReadQuorum | StorageError::InsufficientReadQuorum(_, _) => "read_quorum",
|
||||
StorageError::ErasureWriteQuorum | StorageError::InsufficientWriteQuorum(_, _) => "write_quorum",
|
||||
StorageError::PreconditionFailed | StorageError::ConditionalDeleteConflict => "condition",
|
||||
StorageError::DeleteRecoveryWorkLimit => "recovery_work_limit",
|
||||
StorageError::DeleteRecoveryLegacyAmbiguous => "recovery_legacy_ambiguous",
|
||||
StorageError::Timeout => "timeout",
|
||||
StorageError::RemoteClientUnavailable(_) => "remote_client_unavailable",
|
||||
StorageError::DiskNotFound | StorageError::FaultyDisk | StorageError::FaultyRemoteDisk => "disk_unavailable",
|
||||
StorageError::DiskOngoingReq => "disk_ongoing_request",
|
||||
StorageError::SlowDown => "slow_down",
|
||||
StorageError::Io(source) if source.kind() == std::io::ErrorKind::TimedOut => "io_timeout",
|
||||
error if is_err_bucket_not_found(error) => "bucket_not_found",
|
||||
error if is_delete_objects_not_found(error) => "not_found",
|
||||
StorageError::Io(_) => "io",
|
||||
_ => "other",
|
||||
};
|
||||
*counts.entry(kind).or_default() += 1;
|
||||
}
|
||||
counts
|
||||
}
|
||||
|
||||
impl DefaultObjectUsecase {
|
||||
#[instrument(level = "debug", skip(self, req))]
|
||||
pub async fn execute_delete_objects(&self, req: S3Request<DeleteObjectsInput>) -> S3Result<S3Response<DeleteObjectsOutput>> {
|
||||
@@ -894,39 +850,9 @@ impl DefaultObjectUsecase {
|
||||
..Default::default()
|
||||
};
|
||||
apply_bucket_generation_guard(&req, &bucket, &mut storage_delete_opts)?;
|
||||
let storage_started = std::time::Instant::now();
|
||||
let (dobjs, errs, accounting) = store
|
||||
.delete_objects_with_tier_delete_journal_and_accounting(&bucket, object_to_delete.clone(), storage_delete_opts)
|
||||
.await;
|
||||
let storage_elapsed_ms = storage_started.elapsed().as_millis();
|
||||
let storage_error_counts = delete_objects_storage_error_counts(&object_to_delete, &errs);
|
||||
let failed_object_count = storage_error_counts.values().sum::<usize>();
|
||||
if failed_object_count > 0 {
|
||||
warn!(
|
||||
event = EVENT_DELETE_OBJECTS_STORAGE_RESULT,
|
||||
component = LOG_COMPONENT_APP,
|
||||
subsystem = LOG_SUBSYSTEM_OBJECT,
|
||||
result = "failed",
|
||||
request_id = %request_context.request_id,
|
||||
attempted_object_count = object_to_delete.len(),
|
||||
failed_object_count,
|
||||
error_counts = ?storage_error_counts,
|
||||
elapsed_ms = storage_elapsed_ms,
|
||||
"Batch delete storage returned object failures"
|
||||
);
|
||||
} else {
|
||||
debug!(
|
||||
event = EVENT_DELETE_OBJECTS_STORAGE_RESULT,
|
||||
component = LOG_COMPONENT_APP,
|
||||
subsystem = LOG_SUBSYSTEM_OBJECT,
|
||||
result = "ok",
|
||||
request_id = %request_context.request_id,
|
||||
attempted_object_count = object_to_delete.len(),
|
||||
failed_object_count,
|
||||
elapsed_ms = storage_elapsed_ms,
|
||||
"Batch delete storage completed"
|
||||
);
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
super::super::storage_api::object_usecase::helper::pause_committed_mutation_for_test(&bucket).await;
|
||||
@@ -3087,67 +3013,6 @@ mod tests {
|
||||
assert_eq!(error.code.as_deref(), Some("NoSuchKey"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn delete_objects_storage_diagnostics_preserve_mixed_result_semantics() {
|
||||
let mut objects = (0..7)
|
||||
.map(|index| ObjectToDelete {
|
||||
object_name: format!("key-{index}"),
|
||||
..Default::default()
|
||||
})
|
||||
.collect::<Vec<_>>();
|
||||
objects[2].delete_condition = Some(DeleteObjectCondition::Exists);
|
||||
let errors = vec![
|
||||
None,
|
||||
Some(StorageError::FileNotFound),
|
||||
Some(StorageError::FileNotFound),
|
||||
Some(StorageError::NamespaceLockQuorumUnavailable {
|
||||
mode: "delete_objects_commit",
|
||||
bucket: "bucket".to_owned(),
|
||||
object: "key-3".to_owned(),
|
||||
required: 1,
|
||||
achieved: 0,
|
||||
}),
|
||||
Some(StorageError::ErasureWriteQuorum),
|
||||
Some(StorageError::Timeout),
|
||||
Some(StorageError::PreconditionFailed),
|
||||
];
|
||||
assert_eq!(
|
||||
delete_objects_storage_error_counts(&objects, &errors),
|
||||
BTreeMap::from([
|
||||
("condition", 1),
|
||||
("namespace_lock_quorum", 1),
|
||||
("not_found", 1),
|
||||
("timeout", 1),
|
||||
("write_quorum", 1),
|
||||
])
|
||||
);
|
||||
let deleted = StorageDeletedObject::default();
|
||||
let results = objects
|
||||
.iter()
|
||||
.zip(&errors)
|
||||
.map(|(object, error)| reduce_delete_objects_result(object, &deleted, error.as_ref(), false))
|
||||
.collect::<Vec<_>>();
|
||||
assert!(results[0].is_ok(), "a successful storage result remains successful");
|
||||
assert!(results[1].is_ok(), "ordinary missing objects remain idempotent success");
|
||||
assert_eq!(
|
||||
results[2]
|
||||
.as_ref()
|
||||
.expect_err("conditional missing object must fail")
|
||||
.code
|
||||
.as_deref(),
|
||||
Some("NoSuchKey")
|
||||
);
|
||||
assert_eq!(
|
||||
results[3]
|
||||
.as_ref()
|
||||
.expect_err("lost namespace fence must fail")
|
||||
.code
|
||||
.as_deref(),
|
||||
Some("ServiceUnavailable")
|
||||
);
|
||||
assert!(results[4..].iter().all(Result::is_err));
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[serial_test::serial]
|
||||
fn execute_delete_objects_conditions_authorize_per_entry_and_keep_quiet_errors() {
|
||||
|
||||
@@ -3608,9 +3608,6 @@ fn get_default_tcp_keepalive() -> TcpKeepalive {
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
#[path = "production_batch_lock_rpc_tests.rs"]
|
||||
mod production_batch_lock_rpc_tests;
|
||||
|
||||
use super::*;
|
||||
use crate::server::compress::RequestPathCategory;
|
||||
use crate::storage_api::server::http::ScannerScopedDirtyUsageAckEntry;
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -13,33 +13,16 @@
|
||||
// limitations under the License.
|
||||
|
||||
use super::NodeService;
|
||||
use rustfs_lock::{BatchAcquireOutcome, LockRequest};
|
||||
use rustfs_lock::LockRequest;
|
||||
use rustfs_protos::proto_gen::node_service::*;
|
||||
use tokio::time::Instant;
|
||||
use tonic::{Request, Response, Status};
|
||||
|
||||
fn lock_result_from_acquire(outcome: BatchAcquireOutcome) -> GenerallyLockResult {
|
||||
match outcome {
|
||||
BatchAcquireOutcome::Granted { lock_info, .. } => GenerallyLockResult {
|
||||
success: true,
|
||||
lock_info: lock_info.and_then(|info| serde_json::to_string(&info).ok()),
|
||||
..Default::default()
|
||||
},
|
||||
BatchAcquireOutcome::RetryableNoGrant { error } => lock_result_from_no_grant(error, LockAcquireStatus::RetryableNoGrant),
|
||||
BatchAcquireOutcome::TerminalNoGrant { error } => lock_result_from_no_grant(error, LockAcquireStatus::RejectedNoGrant),
|
||||
BatchAcquireOutcome::Ambiguous { error, .. } => lock_result_from_error(error.to_string()),
|
||||
}
|
||||
}
|
||||
|
||||
fn lock_result_from_no_grant(error: rustfs_lock::LockError, status: LockAcquireStatus) -> GenerallyLockResult {
|
||||
let message = match error {
|
||||
rustfs_lock::LockError::Timeout { .. } => "Lock acquisition timeout".to_string(),
|
||||
rustfs_lock::LockError::Internal { message } => message,
|
||||
error => error.to_string(),
|
||||
};
|
||||
fn lock_result_from_response(response: rustfs_lock::LockResponse) -> GenerallyLockResult {
|
||||
GenerallyLockResult {
|
||||
acquire_status: status.into(),
|
||||
..lock_result_from_error(message)
|
||||
success: response.success,
|
||||
error_info: response.error,
|
||||
lock_info: response.lock_info.and_then(|info| serde_json::to_string(&info).ok()),
|
||||
..Default::default()
|
||||
}
|
||||
}
|
||||
|
||||
@@ -100,7 +83,6 @@ fn lock_result_from_release(lock_id: &rustfs_lock::LockId, success: bool) -> Gen
|
||||
} else {
|
||||
LockReleaseStatus::AlreadyAbsent.into()
|
||||
},
|
||||
..Default::default()
|
||||
}
|
||||
}
|
||||
|
||||
@@ -263,7 +245,6 @@ impl NodeService {
|
||||
&self,
|
||||
request: Request<BatchGenerallyLockRequest>,
|
||||
) -> Result<Response<BatchGenerallyLockResponse>, Status> {
|
||||
let entered = Instant::now();
|
||||
#[cfg(feature = "e2e-test-hooks")]
|
||||
let request_id = super::e2e_rpc_delay::request_id(&request);
|
||||
let request = request.into_inner();
|
||||
@@ -278,10 +259,7 @@ impl NodeService {
|
||||
valid_indices.push(idx);
|
||||
}
|
||||
Err(err) => {
|
||||
results[idx] = GenerallyLockResult {
|
||||
acquire_status: LockAcquireStatus::RejectedNoGrant.into(),
|
||||
..lock_result_from_error(format!("can not decode args, err: {err}"))
|
||||
};
|
||||
results[idx] = lock_result_from_error(format!("can not decode args, err: {err}"));
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -293,17 +271,18 @@ impl NodeService {
|
||||
.await;
|
||||
}
|
||||
let lock_client = self.get_lock_client()?;
|
||||
// The payload's wait budget also pays for decoding and handler admission.
|
||||
let wait = valid_requests
|
||||
.iter()
|
||||
.map(|request| request.acquire_timeout)
|
||||
.max()
|
||||
.unwrap_or_default();
|
||||
let deadline = entered.checked_add(wait).unwrap_or(entered);
|
||||
let batch_results = lock_client.acquire_locks_batch_until(&valid_requests, deadline).await;
|
||||
for (result_idx, outcome) in batch_results.into_iter().enumerate() {
|
||||
if let Some(request_idx) = valid_indices.get(result_idx) {
|
||||
results[*request_idx] = lock_result_from_acquire(outcome);
|
||||
match lock_client.acquire_locks_batch(&valid_requests).await {
|
||||
Ok(batch_results) => {
|
||||
for (result_idx, response) in batch_results.into_iter().enumerate() {
|
||||
if let Some(request_idx) = valid_indices.get(result_idx) {
|
||||
results[*request_idx] = lock_result_from_response(response);
|
||||
}
|
||||
}
|
||||
}
|
||||
Err(err) => {
|
||||
for request_idx in valid_indices {
|
||||
results[request_idx] = lock_result_from_error(format!("can not batch lock, err: {err}"));
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -359,7 +338,7 @@ impl NodeService {
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::{lock_result_from_acquire, lock_result_from_release};
|
||||
use super::{lock_result_from_release, lock_result_from_response};
|
||||
|
||||
fn test_lock_id() -> rustfs_lock::LockId {
|
||||
rustfs_lock::LockRequest::new(rustfs_lock::ObjectKey::new("bucket", "object"), rustfs_lock::LockType::Exclusive, "owner")
|
||||
@@ -383,55 +362,14 @@ mod tests {
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn batch_acquire_mapping_preserves_explicit_and_ambiguous_failures() {
|
||||
for (outcome, status) in [
|
||||
(
|
||||
rustfs_lock::BatchAcquireOutcome::RetryableNoGrant {
|
||||
error: rustfs_lock::LockError::internal("lock conflict"),
|
||||
},
|
||||
super::LockAcquireStatus::RetryableNoGrant,
|
||||
),
|
||||
(
|
||||
rustfs_lock::BatchAcquireOutcome::TerminalNoGrant {
|
||||
error: rustfs_lock::LockError::internal("invalid request"),
|
||||
},
|
||||
super::LockAcquireStatus::RejectedNoGrant,
|
||||
),
|
||||
(
|
||||
rustfs_lock::BatchAcquireOutcome::Ambiguous {
|
||||
error: rustfs_lock::LockError::internal("legacy batch failed"),
|
||||
cleanup_owned_by_client: false,
|
||||
},
|
||||
super::LockAcquireStatus::Unspecified,
|
||||
),
|
||||
] {
|
||||
let result = lock_result_from_acquire(outcome);
|
||||
assert!(!result.success);
|
||||
assert!(result.error_info.is_some());
|
||||
assert!(result.lock_info.is_none());
|
||||
assert_eq!(result.acquire_status, status as i32);
|
||||
assert_eq!(result.release_status, super::LockReleaseStatus::Unspecified as i32);
|
||||
}
|
||||
}
|
||||
fn lock_result_from_response_preserves_lock_failure_error() {
|
||||
let response = rustfs_lock::LockResponse::failure("lock conflict", std::time::Duration::ZERO);
|
||||
let result = lock_result_from_response(response);
|
||||
|
||||
#[test]
|
||||
fn batch_acquire_timeout_preserves_legacy_business_error_text() {
|
||||
let result = lock_result_from_acquire(rustfs_lock::BatchAcquireOutcome::RetryableNoGrant {
|
||||
error: rustfs_lock::LockError::timeout("bucket/object", std::time::Duration::from_millis(50)),
|
||||
});
|
||||
assert_eq!(result.error_info.as_deref(), Some("Lock acquisition timeout"));
|
||||
assert_eq!(result.acquire_status, super::LockAcquireStatus::RetryableNoGrant as i32);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn batch_acquire_mapping_preserves_legacy_success() {
|
||||
let result = lock_result_from_acquire(rustfs_lock::BatchAcquireOutcome::Granted {
|
||||
lock_info: None,
|
||||
wait_time: std::time::Duration::ZERO,
|
||||
});
|
||||
assert!(result.success);
|
||||
assert!(result.error_info.is_none());
|
||||
assert_eq!(result.acquire_status, super::LockAcquireStatus::Unspecified as i32);
|
||||
assert!(!result.success);
|
||||
assert_eq!(result.error_info.as_deref(), Some("lock conflict"));
|
||||
assert!(result.lock_info.is_none());
|
||||
assert_eq!(result.release_status, super::LockReleaseStatus::Unspecified as i32);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
|
||||
@@ -203,8 +203,7 @@ pub(crate) mod server {
|
||||
pub(crate) use crate::storage::storage_api::{
|
||||
Endpoint, EndpointServerPools, Endpoints, PeerRestClient, PoolEndpoints, ScannerScopedDirtyUsageAckEntry,
|
||||
find_local_disk_by_ref, gen_signature_headers, gen_tonic_replay_scope_headers, gen_tonic_signature_headers,
|
||||
get_global_lock_client, get_global_lock_clients, init_local_disks, init_lock_clients,
|
||||
verify_tonic_boot_epoch_response,
|
||||
init_local_disks, verify_tonic_boot_epoch_response,
|
||||
};
|
||||
|
||||
pub(crate) mod ecfs {
|
||||
|
||||
+1
-1
@@ -40,7 +40,7 @@ their issue closes.
|
||||
| `check_unsafe_code_allowances.sh` | ci-gate | Unsafe-code allowance ledger guard | ci.yml Quick Checks |
|
||||
| `layer-dependency-baseline.txt` | ci-gate (data) | Committed baseline consumed by `check_layer_dependencies.sh` | [Architecture guard troubleshooting](../docs/operations/architecture-guard-troubleshooting.md) |
|
||||
| `static.sh` | ci-gate | Static-build helper executed inside image builds | `Dockerfile.source`, `Dockerfile.decommission-local` |
|
||||
| `helm_chart_version.sh` | ci-gate | Keeps the Helm chart version in sync with the release | helm-package.yml |
|
||||
| `helm_chart_version.sh` | dev-tool | Normalizes chart/app versions for installation references | rustfs-release-version-bump skill |
|
||||
| `test_helm_templates.sh` | ci-gate | Helm template rendering test | helm-package.yml |
|
||||
|
||||
## Test & e2e runners
|
||||
|
||||
@@ -17,12 +17,7 @@ case "$RAW" in
|
||||
esac
|
||||
|
||||
APP_VERSION="${RAW_TAG#v}"
|
||||
BETA_NUM=$(printf '%s\n' "$APP_VERSION" | sed -n -E 's/^.*-beta\.([0-9]+)$/\1/p')
|
||||
if [ -n "$BETA_NUM" ]; then
|
||||
CHART_VERSION="0.${BETA_NUM}.0"
|
||||
else
|
||||
CHART_VERSION="$APP_VERSION"
|
||||
fi
|
||||
CHART_VERSION="$APP_VERSION"
|
||||
|
||||
if [ -n "${GITHUB_OUTPUT:-}" ]; then
|
||||
{
|
||||
|
||||
Executable
+82
@@ -0,0 +1,82 @@
|
||||
#!/usr/bin/env bash
|
||||
# Copyright 2026 RustFS Team
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
set -euo pipefail
|
||||
|
||||
PKG_DIR="${DEB_FILE%.deb}"
|
||||
|
||||
echo "Building DEB: ${DEB_FILE}"
|
||||
|
||||
mkdir -p "${PKG_DIR}/DEBIAN"
|
||||
mkdir -p "${PKG_DIR}/usr/bin"
|
||||
mkdir -p "${PKG_DIR}/etc/default"
|
||||
mkdir -p "${PKG_DIR}/lib/systemd/system"
|
||||
mkdir -p "${PKG_DIR}/usr/share/doc/rustfs"
|
||||
|
||||
cp ./bin/rustfs "${PKG_DIR}/usr/bin/"
|
||||
chmod 755 "${PKG_DIR}/usr/bin/rustfs"
|
||||
|
||||
cp deploy/build/rustfs.service "${PKG_DIR}/lib/systemd/system/"
|
||||
|
||||
cat > "${PKG_DIR}/etc/default/rustfs" << 'ENVEOF'
|
||||
# RustFS Environment Configuration
|
||||
# See https://docs.rustfs.com for more information
|
||||
# RUSTFS_ACCESS_KEY=<changeme>
|
||||
# RUSTFS_SECRET_KEY=<changeme>
|
||||
# RUSTFS_VOLUMES=""
|
||||
# RUSTFS_ADDRESS=":9000"
|
||||
# RUSTFS_CONSOLE_ADDRESS=":9001"
|
||||
# RUSTFS_CONSOLE_ENABLE=true
|
||||
# RUSTFS_OBS_LOGGER_LEVEL=error
|
||||
# RUSTFS_OBS_LOG_DIRECTORY="/var/log/rustfs/"
|
||||
ENVEOF
|
||||
|
||||
cat > "${PKG_DIR}/DEBIAN/control" << EOF
|
||||
Package: rustfs
|
||||
Version: ${DEB_VERSION}
|
||||
Section: utils
|
||||
Priority: optional
|
||||
Architecture: ${DEB_ARCH}
|
||||
EOF
|
||||
|
||||
# gnu binaries dynamically link glibc and must declare it; musl
|
||||
# binaries are statically linked and need no libc dependency.
|
||||
if [[ -n "$DEB_DEPENDS" ]]; then
|
||||
echo "Depends: ${DEB_DEPENDS}" >> "${PKG_DIR}/DEBIAN/control"
|
||||
fi
|
||||
|
||||
cat >> "${PKG_DIR}/DEBIAN/control" << EOF
|
||||
Maintainer: RustFS Team <support@rustfs.com>
|
||||
Description: High-performance distributed object storage (${LIBC} build)
|
||||
RustFS is a high-performance distributed object storage software
|
||||
built using Rust. It is compatible with MinIO and S3 API.
|
||||
Homepage: https://rustfs.com
|
||||
EOF
|
||||
|
||||
# Declare /etc/default/rustfs as a conffile so dpkg preserves user
|
||||
# modifications on upgrade instead of silently overwriting them.
|
||||
cat > "${PKG_DIR}/DEBIAN/conffiles" << 'CONFFILES'
|
||||
/etc/default/rustfs
|
||||
CONFFILES
|
||||
|
||||
./scripts/release/package_service_scripts.sh after-install > "${PKG_DIR}/DEBIAN/postinst"
|
||||
chmod 755 "${PKG_DIR}/DEBIAN/postinst"
|
||||
|
||||
./scripts/release/package_service_scripts.sh before-remove > "${PKG_DIR}/DEBIAN/prerm"
|
||||
chmod 755 "${PKG_DIR}/DEBIAN/prerm"
|
||||
|
||||
./scripts/release/package_service_scripts.sh after-remove > "${PKG_DIR}/DEBIAN/postrm"
|
||||
chmod 755 "${PKG_DIR}/DEBIAN/postrm"
|
||||
|
||||
cp LICENSE "${PKG_DIR}/usr/share/doc/rustfs/"
|
||||
cp README.md "${PKG_DIR}/usr/share/doc/rustfs/"
|
||||
|
||||
fakeroot dpkg-deb --build "${PKG_DIR}" "$DEB_FILE"
|
||||
|
||||
[[ $(dpkg-deb -f "$DEB_FILE" Package) == rustfs ]]
|
||||
[[ $(dpkg-deb -f "$DEB_FILE" Version) == "$DEB_VERSION" ]]
|
||||
[[ $(dpkg-deb -f "$DEB_FILE" Architecture) == "$DEB_ARCH" ]]
|
||||
dpkg-deb --fsys-tarfile "$DEB_FILE" | tar -tf - | grep -Fx './usr/bin/rustfs' >/dev/null
|
||||
stat --printf='%n %s bytes\n' "$DEB_FILE"
|
||||
echo "deb_file=$DEB_FILE" >> "$GITHUB_OUTPUT"
|
||||
echo "✅ DEB built: $DEB_FILE"
|
||||
Executable
+69
@@ -0,0 +1,69 @@
|
||||
#!/usr/bin/env bash
|
||||
# Copyright 2026 RustFS Team
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
set -euo pipefail
|
||||
|
||||
echo "Building RPM for ${RPM_ARCH} (${LIBC})"
|
||||
|
||||
./scripts/test_package_versions.sh --require-package-managers
|
||||
|
||||
# gnu binaries dynamically link glibc and must declare it; musl
|
||||
# binaries are statically linked and need no libc dependency.
|
||||
depends_args=()
|
||||
if [[ -n "$RPM_DEPENDS" ]]; then
|
||||
depends_args+=(--depends "$RPM_DEPENDS")
|
||||
fi
|
||||
|
||||
# Create config file for fpm (DEB build creates it in its package dir structure,
|
||||
# but fpm needs the file to exist before packaging)
|
||||
mkdir -p ./tmp-pkg/etc/default
|
||||
cat > ./tmp-pkg/etc/default/rustfs << 'ENVEOF'
|
||||
# RustFS Environment Configuration
|
||||
# See https://docs.rustfs.com for more information
|
||||
# RUSTFS_ACCESS_KEY=<changeme>
|
||||
# RUSTFS_SECRET_KEY=<changeme>
|
||||
# RUSTFS_VOLUMES=""
|
||||
# RUSTFS_ADDRESS=":9000"
|
||||
# RUSTFS_CONSOLE_ADDRESS=":9001"
|
||||
# RUSTFS_CONSOLE_ENABLE=true
|
||||
# RUSTFS_OBS_LOGGER_LEVEL=error
|
||||
# RUSTFS_OBS_LOG_DIRECTORY="/var/log/rustfs/"
|
||||
ENVEOF
|
||||
|
||||
fpm -s dir -t rpm \
|
||||
--name rustfs \
|
||||
--version "$RPM_VERSION" \
|
||||
--iteration "$RPM_RELEASE" \
|
||||
--architecture "$RPM_ARCH" \
|
||||
--package "$RPM_FILE" \
|
||||
"${depends_args[@]}" \
|
||||
--description "High-performance distributed object storage (${LIBC} build)" \
|
||||
--maintainer "RustFS Team <support@rustfs.com>" \
|
||||
--url "https://rustfs.com" \
|
||||
--license "Apache-2.0" \
|
||||
--before-install <(./scripts/release/package_service_scripts.sh rpm-before-install) \
|
||||
--before-upgrade <(./scripts/release/package_service_scripts.sh rpm-before-upgrade) \
|
||||
--after-install <(./scripts/release/package_service_scripts.sh after-install) \
|
||||
--after-upgrade <(./scripts/release/package_service_scripts.sh after-install) \
|
||||
--before-remove <(./scripts/release/package_service_scripts.sh before-remove) \
|
||||
--after-remove <(./scripts/release/package_service_scripts.sh after-remove) \
|
||||
--rpm-posttrans <(./scripts/release/package_service_scripts.sh rpm-posttrans) \
|
||||
--config-files /etc/default/rustfs \
|
||||
./bin/rustfs=/usr/bin/rustfs \
|
||||
./tmp-pkg/etc/default/rustfs=/etc/default/rustfs \
|
||||
deploy/build/rustfs.service=/lib/systemd/system/rustfs.service \
|
||||
LICENSE=/usr/share/doc/rustfs/LICENSE \
|
||||
README.md=/usr/share/doc/rustfs/README.md
|
||||
|
||||
if [[ ! -f "$RPM_FILE" ]]; then
|
||||
echo "❌ RPM build failed"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
RPM_METADATA=$(rpm -qp --qf '%{NAME}\n%{VERSION}\n%{RELEASE}\n%{ARCH}\n' "$RPM_FILE")
|
||||
EXPECTED_METADATA=$(printf 'rustfs\n%s\n%s\n%s' "$RPM_VERSION" "$RPM_RELEASE" "$RPM_ARCH")
|
||||
[[ "$RPM_METADATA" == "$EXPECTED_METADATA" ]]
|
||||
rpm -qpl "$RPM_FILE" | grep -Fx '/usr/bin/rustfs' >/dev/null
|
||||
stat --printf='%n %s bytes\n' "$RPM_FILE"
|
||||
echo "rpm_file=$RPM_FILE" >> "$GITHUB_OUTPUT"
|
||||
echo "✅ RPM built: $RPM_FILE"
|
||||
@@ -0,0 +1,428 @@
|
||||
#!/usr/bin/env python3
|
||||
# Copyright 2026 RustFS Team
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
"""Frozen release inputs and artifact identity; publication never builds binaries."""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import tomllib
|
||||
import urllib.request
|
||||
from urllib.parse import urlsplit
|
||||
import zipfile
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
REPOSITORY = "rustfs/rustfs"
|
||||
SCHEMA = 1
|
||||
HEX40 = re.compile(r"[0-9a-f]{40}\Z")
|
||||
HEX64 = re.compile(r"[0-9a-f]{64}\Z")
|
||||
VERSION = re.compile(r"(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)(?:-(?:alpha|beta|rc)\.(0|[1-9][0-9]*))?\Z")
|
||||
TARGETS = ("linux-x86_64-musl", "linux-aarch64-musl", "linux-x86_64-gnu", "linux-aarch64-gnu", "macos-aarch64", "windows-x86_64")
|
||||
LINUX = TARGETS[:4]
|
||||
DESKTOP = TARGETS[4:]
|
||||
MAX_MEMBER = 2 * 1024**3
|
||||
RECIPE_PATHS = (
|
||||
".github/workflows/build.yml",
|
||||
".github/workflows/release-candidate.yml",
|
||||
".github/workflows/release-complete.yml",
|
||||
".github/workflows/release-publish.yml",
|
||||
".github/workflows/docker.yml",
|
||||
".github/workflows/package.yml",
|
||||
".github/actions/setup/action.yml",
|
||||
".github/actions/setup/download_release_asset.py",
|
||||
"scripts/release/candidate.py",
|
||||
"scripts/release/pipeline.py",
|
||||
"scripts/release/verify.py",
|
||||
"scripts/release/deliver.py",
|
||||
"scripts/release/publish.py",
|
||||
"scripts/release/release_gate.py",
|
||||
"scripts/release/package_versions.sh",
|
||||
"scripts/release/package_service_scripts.sh",
|
||||
"scripts/release/build_deb.sh",
|
||||
"scripts/release/build_rpm.sh",
|
||||
"scripts/release/check_glibc.py",
|
||||
".config/release-validation-plan.json",
|
||||
"Dockerfile", "Dockerfile.glibc", "entrypoint.sh", "deploy/build/rustfs.service",
|
||||
".github/workflows/helm-package.yml", "scripts/test_helm_templates.sh",
|
||||
"scripts/security/generate_release_supply_chain_assets.py",
|
||||
".github/workflows/ci.yml", "scripts/ci_gate.py",
|
||||
)
|
||||
|
||||
|
||||
def require(condition, message):
|
||||
if not condition:
|
||||
raise ValueError(message)
|
||||
|
||||
|
||||
def digest(value):
|
||||
return hashlib.sha256(json.dumps(value, sort_keys=True, separators=(",", ":"), ensure_ascii=False).encode()).hexdigest()
|
||||
|
||||
|
||||
def stream_digest(stream):
|
||||
result = hashlib.sha256()
|
||||
for chunk in iter(lambda: stream.read(1024 * 1024), b""):
|
||||
result.update(chunk)
|
||||
return result.hexdigest()
|
||||
|
||||
|
||||
def file_digest(path):
|
||||
with Path(path).open("rb") as stream:
|
||||
return stream_digest(stream)
|
||||
|
||||
|
||||
def read_json(path):
|
||||
path = Path(path)
|
||||
require(path.is_file() and path.stat().st_size <= 4 * 1024**2, "missing or oversized JSON record")
|
||||
def unique(pairs):
|
||||
value = {}
|
||||
for name, item in pairs:
|
||||
require(name not in value, "duplicate JSON field")
|
||||
value[name] = item
|
||||
return value
|
||||
return json.loads(path.read_text(encoding="utf-8"), object_pairs_hook=unique)
|
||||
|
||||
|
||||
def write_json(path, value):
|
||||
"""Existing identity records are immutable, including across job retries."""
|
||||
path = Path(path)
|
||||
if path.exists():
|
||||
require(read_json(path) == value, "refusing to replace an existing identity record")
|
||||
return
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with path.open("x", encoding="utf-8") as stream:
|
||||
json.dump(value, stream, indent=2, sort_keys=True, ensure_ascii=False)
|
||||
stream.write("\n")
|
||||
|
||||
|
||||
def command(args, *, input_text=None, cwd=None):
|
||||
return subprocess.run(args, input=input_text, text=True, capture_output=True, check=True, cwd=cwd, timeout=120).stdout.strip()
|
||||
|
||||
|
||||
class ArtifactRedirect(urllib.request.HTTPRedirectHandler):
|
||||
def redirect_request(self, request, fp, code, message, headers, url):
|
||||
require(urlsplit(url).scheme == "https", "artifact redirect must use HTTPS")
|
||||
redirected = super().redirect_request(request, fp, code, message, headers, url)
|
||||
if redirected is not None and urlsplit(request.full_url).netloc != urlsplit(url).netloc:
|
||||
redirected.remove_header("Authorization")
|
||||
return redirected
|
||||
|
||||
|
||||
def github_stream(path):
|
||||
require(path.startswith("repos/") and ":" not in path and not path.startswith("/"), "invalid GitHub API path")
|
||||
token = os.environ.get("GH_TOKEN") or command(["gh", "auth", "token"])
|
||||
request = urllib.request.Request("https://api.github.com/" + path,
|
||||
headers={"Authorization": "Bearer " + token, "Accept": "application/vnd.github+json",
|
||||
"X-GitHub-Api-Version": "2022-11-28", "User-Agent": "rustfs-release"})
|
||||
return urllib.request.build_opener(ArtifactRedirect()).open(request, timeout=600)
|
||||
|
||||
|
||||
def github(path):
|
||||
with github_stream(path) as response:
|
||||
return json.load(response)
|
||||
|
||||
|
||||
def git_file(revision, path, root=ROOT):
|
||||
require(HEX40.fullmatch(revision) is not None, "source must be a full commit SHA")
|
||||
return subprocess.check_output(["git", "show", f"{revision}:{path}"], cwd=root, timeout=60)
|
||||
|
||||
|
||||
def recipe_digest(revision, root=ROOT):
|
||||
files = {path: hashlib.sha256(git_file(revision, path, root)).hexdigest() for path in RECIPE_PATHS}
|
||||
for directory in ("bootstrap", "helm"):
|
||||
paths = command(["git", "ls-tree", "-r", "--name-only", revision, "--", directory], cwd=root).splitlines()
|
||||
require(bool(paths), "missing release recipe directory")
|
||||
files.update({path: hashlib.sha256(git_file(revision, path, root)).hexdigest() for path in paths})
|
||||
return digest(files)
|
||||
|
||||
|
||||
def console_asset(release):
|
||||
require(isinstance(release, dict) and release.get("draft") is False and release.get("prerelease") is False,
|
||||
"Console must be a published stable release")
|
||||
tag = release.get("tag_name")
|
||||
require(isinstance(tag, str) and re.fullmatch(r"v?[0-9]+\.[0-9]+\.[0-9]+", tag), "invalid Console version")
|
||||
assets = [asset for asset in release.get("assets", [])
|
||||
if re.fullmatch(r"rustfs-console-[A-Za-z0-9._-]+\.zip", asset.get("name", ""))]
|
||||
require(len(assets) == 1, "Console must have exactly one unambiguous ZIP asset")
|
||||
asset = assets[0]
|
||||
checksum = asset.get("digest", "")
|
||||
require(isinstance(checksum, str) and checksum.startswith("sha256:") and HEX64.fullmatch(checksum[7:]),
|
||||
"Console asset must have a SHA-256 digest")
|
||||
expected = f"https://github.com/rustfs/console/releases/download/{tag}/{asset['name']}"
|
||||
require(asset.get("browser_download_url") == expected, "Console URL does not match its release identity")
|
||||
return {"tag": tag, "name": asset["name"], "url": expected, "sha256": checksum[7:]}
|
||||
|
||||
|
||||
def release_asset(release, repository, name):
|
||||
require(release.get("draft") is False and release.get("prerelease") is False, "dependency must be a published stable release")
|
||||
tag = release.get("tag_name", "")
|
||||
require(re.fullmatch(r"v?[0-9]+\.[0-9]+\.[0-9]+", tag), "invalid dependency tag")
|
||||
matches = [a for a in release.get("assets", []) if a.get("name") == name]
|
||||
require(len(matches) == 1, "missing or ambiguous pinned dependency")
|
||||
asset = matches[0]
|
||||
url = f"https://github.com/{repository}/releases/download/{tag}/{name}"
|
||||
require(asset.get("browser_download_url") == url and re.fullmatch(r"sha256:[0-9a-f]{64}", asset.get("digest", "")), "invalid dependency identity")
|
||||
return {"version": tag, "name": name, "url": url, "sha256": asset["digest"][7:]}
|
||||
|
||||
|
||||
def validate_dependency(asset, repository):
|
||||
require(isinstance(asset, dict) and set(asset) == {"version", "name", "url", "sha256"}, "invalid dependency record")
|
||||
require(re.fullmatch(r"[A-Za-z0-9._-]+", asset["name"]), "unsafe dependency name")
|
||||
normalized = release_asset({"draft": False, "prerelease": False, "tag_name": asset["version"],
|
||||
"assets": [{"name": asset["name"], "browser_download_url": asset["url"],
|
||||
"digest": "sha256:" + asset["sha256"]}]}, repository, asset["name"])
|
||||
require(normalized == asset, "dependency record changed")
|
||||
|
||||
|
||||
def validate_inputs(value):
|
||||
require(isinstance(value, dict), "candidate inputs must be an object")
|
||||
fields = {"schema", "candidate_id", "version", "source_sha", "workflow_sha", "recipe_sha256", "lock_sha256",
|
||||
"plan_sha256", "toolchain", "console", "cli", "previous", "ci", "origin_run_id", "origin_run_number"}
|
||||
require(set(value) == fields and type(value["schema"]) is int and value["schema"] == SCHEMA,
|
||||
"unsupported candidate input schema")
|
||||
require(isinstance(value["version"], str) and VERSION.fullmatch(value["version"]), "invalid release version")
|
||||
for name in ("source_sha", "workflow_sha"):
|
||||
require(isinstance(value[name], str) and HEX40.fullmatch(value[name]), f"invalid {name}")
|
||||
for name in ("recipe_sha256", "lock_sha256", "plan_sha256"):
|
||||
require(isinstance(value[name], str) and HEX64.fullmatch(value[name]), f"invalid {name}")
|
||||
for name in ("origin_run_id", "origin_run_number"):
|
||||
require(type(value[name]) is int and value[name] > 0, f"invalid {name}")
|
||||
require(value["candidate_id"] == f"{value['version']}-preview.{value['origin_run_number']}", "candidate name differs from its immutable run identity")
|
||||
require(isinstance(value["toolchain"], str) and re.fullmatch(r"[1-9][0-9]*\.[0-9]+\.[0-9]+", value["toolchain"]),
|
||||
"candidate toolchain must be an exact stable Rust version")
|
||||
console = value["console"]
|
||||
require(isinstance(console, dict) and set(console) == {"tag", "name", "url", "sha256"}, "invalid Console identity")
|
||||
normalized = console_asset({"draft": False, "prerelease": False, "tag_name": console["tag"],
|
||||
"assets": [{"name": console["name"], "browser_download_url": console["url"],
|
||||
"digest": "sha256:" + str(console["sha256"])}]})
|
||||
require(normalized == console, "invalid pinned Console asset")
|
||||
require(isinstance(value["cli"], dict) and set(value["cli"]) == set(LINUX), "missing CLI platform")
|
||||
for asset in value["cli"].values():
|
||||
validate_dependency(asset, "rustfs/cli")
|
||||
validate_dependency(value["previous"], REPOSITORY)
|
||||
require(value["previous"]["version"].lstrip("v") != value["version"], "previous release must differ from the candidate")
|
||||
require(isinstance(value["ci"], dict) and set(value["ci"]) == {"run_id", "run_attempt"}
|
||||
and all(type(item) is int and item > 0 for item in value["ci"].values()), "invalid exact-source CI identity")
|
||||
return value
|
||||
|
||||
|
||||
def freeze(version, source_sha, workflow_sha, toolchain, release, run_id, run_number, cli, previous, ci, root=ROOT):
|
||||
require(VERSION.fullmatch(version) is not None, "invalid release version")
|
||||
source = tomllib.loads(git_file(source_sha, "Cargo.toml", root).decode())
|
||||
require(source["workspace"]["package"]["version"] == version, "Cargo workspace version differs from the target")
|
||||
recipe = recipe_digest(workflow_sha, root)
|
||||
require(recipe_digest(source_sha, root) == recipe, "candidate must contain the reviewed release recipe")
|
||||
plan = json.loads(git_file(workflow_sha, ".config/release-validation-plan.json", root))
|
||||
value = {"schema": SCHEMA, "version": version, "candidate_id": f"{version}-preview.{run_number}",
|
||||
"source_sha": source_sha, "workflow_sha": workflow_sha, "recipe_sha256": recipe,
|
||||
"lock_sha256": hashlib.sha256(git_file(source_sha, "Cargo.lock", root)).hexdigest(),
|
||||
"plan_sha256": digest(plan), "toolchain": toolchain, "console": console_asset(release),
|
||||
"origin_run_id": run_id, "origin_run_number": run_number, "ci": ci,
|
||||
"cli": {target: release_asset(cli, "rustfs/cli", f"rustfs-cli-linux-{'amd64' if 'x86_64' in target else 'arm64'}{'-gnu' if target.endswith('gnu') else ''}-{cli['tag_name']}.tar.gz") for target in LINUX},
|
||||
"previous": release_asset(previous, REPOSITORY, f"rustfs-linux-x86_64-gnu-v{previous['tag_name'].lstrip('v')}.zip")}
|
||||
return validate_inputs(value)
|
||||
|
||||
|
||||
def targets(selection):
|
||||
if selection == "linux":
|
||||
return LINUX
|
||||
if selection == "desktop":
|
||||
return DESKTOP
|
||||
if selection == "all":
|
||||
return TARGETS
|
||||
selected = tuple(selection.split(","))
|
||||
require(selected and len(set(selected)) == len(selected) and all(item in TARGETS for item in selected),
|
||||
"unknown or duplicate build targets")
|
||||
return selected
|
||||
|
||||
|
||||
def archive_record(path, target, inputs):
|
||||
validate_inputs(inputs)
|
||||
require(target in TARGETS, "unknown binary platform")
|
||||
path = Path(path)
|
||||
name = f"rustfs-{target}-v{inputs['version']}.zip"
|
||||
require(path.name == name and path.is_file() and not path.is_symlink(), "archive name does not match the target version")
|
||||
binary = "rustfs.exe" if target == "windows-x86_64" else "rustfs"
|
||||
expected = {binary}
|
||||
if target.endswith("-gnu"):
|
||||
expected.update(("rustfs.cpu-symbol-catalog.json", "rustfs.cpu-symbol-catalog.sha256"))
|
||||
with zipfile.ZipFile(path) as archive:
|
||||
members = archive.infolist()
|
||||
require(len(members) == len(expected) and {m.filename for m in members} == expected,
|
||||
"unexpected or duplicate release archive member")
|
||||
for member in members:
|
||||
require(not member.is_dir() and 0 < member.file_size <= MAX_MEMBER and (member.external_attr >> 16) & 0o170000 != 0o120000,
|
||||
"invalid release archive member")
|
||||
with archive.open(binary) as stream:
|
||||
binary_sha = stream_digest(stream)
|
||||
return {"target": target, "name": name, "kind": "zip", "sha256": file_digest(path), "size": path.stat().st_size,
|
||||
"binary_sha256": binary_sha, "inputs_sha256": digest(inputs)}
|
||||
|
||||
|
||||
def verify_records(inputs, records, directory, required):
|
||||
validate_inputs(inputs)
|
||||
require(isinstance(records, list) and len(records) == len(required), "candidate is missing required platform artifacts")
|
||||
seen = set()
|
||||
for record in records:
|
||||
require(isinstance(record, dict) and record.get("target") in required, "unexpected artifact platform")
|
||||
target = record["target"]
|
||||
require(target not in seen, "duplicate artifact platform")
|
||||
seen.add(target)
|
||||
expected = archive_record(Path(directory) / f"rustfs-{target}-v{inputs['version']}.zip", target, inputs)
|
||||
require(record == expected, "candidate artifact or input identity changed")
|
||||
require(seen == set(required), "incomplete candidate platform set")
|
||||
return sorted(records, key=lambda record: record["target"])
|
||||
|
||||
|
||||
def certification_candidate(inputs, records, directory, required=TARGETS):
|
||||
"""Candidate identity survives adding desktop files; test receipts bind bytes."""
|
||||
records = verify_records(inputs, records, directory, required)
|
||||
return {"schema_version": 1, "product_commit": inputs["source_sha"], "version": inputs["version"],
|
||||
"release_profile": "prerelease" if "-" in inputs["version"] else "stable", "features": ["pyroscope"],
|
||||
"plan_sha256": inputs["plan_sha256"], "workflow_sha": inputs["workflow_sha"],
|
||||
"validation_batch": {"run_id": inputs["candidate_id"], "run_attempt": 1},
|
||||
"deliverables": [{"kind": "zip", "name": r["name"], "sha256": r["sha256"], "size": r["size"],
|
||||
"binary_sha256": r["binary_sha256"],
|
||||
"location": f"https://github.com/{REPOSITORY}/actions/runs/{inputs['origin_run_id']}#artifact-{r['sha256']}"}
|
||||
for r in records]}
|
||||
|
||||
|
||||
def validate_run(run, workflow, *, completed=True):
|
||||
require(isinstance(run, dict) and (run.get("repository") or {}).get("full_name") == REPOSITORY,
|
||||
"build run belongs to another repository")
|
||||
require(run.get("path") == f".github/workflows/{workflow}" and run.get("event") == "workflow_dispatch"
|
||||
and run.get("head_branch") == "main", "candidate must originate from the trusted main workflow")
|
||||
require(type(run.get("id")) is int and run["id"] > 0 and type(run.get("run_attempt")) is int and run["run_attempt"] > 0,
|
||||
"invalid workflow run identity")
|
||||
require(isinstance(run.get("head_sha"), str) and HEX40.fullmatch(run["head_sha"]), "invalid workflow source SHA")
|
||||
if completed:
|
||||
require(run.get("status") == "completed" and run.get("conclusion") == "success", "candidate run did not succeed")
|
||||
return run
|
||||
|
||||
|
||||
def validate_artifact(artifact, run, name):
|
||||
require(artifact.get("name") == name and artifact.get("expired") is False, "missing or expired candidate artifact")
|
||||
origin = artifact.get("workflow_run") or {}
|
||||
require(origin.get("id") == run["id"] and origin.get("head_sha") == run["head_sha"], "artifact belongs to another build run")
|
||||
require(type(artifact.get("id")) is int and artifact["id"] > 0 and type(artifact.get("size_in_bytes")) is int
|
||||
and 0 < artifact["size_in_bytes"] <= 8 * MAX_MEMBER, "invalid artifact identity or size")
|
||||
require(isinstance(artifact.get("digest"), str) and re.fullmatch(r"sha256:[0-9a-f]{64}", artifact["digest"]),
|
||||
"artifact has no immutable digest")
|
||||
return artifact
|
||||
|
||||
|
||||
def artifact_list(run_id):
|
||||
require(type(run_id) is int and run_id > 0, "invalid run ID")
|
||||
result = []
|
||||
for page in range(1, 101):
|
||||
batch = github(f"repos/{REPOSITORY}/actions/runs/{run_id}/artifacts?per_page=100&page={page}")["artifacts"]
|
||||
result.extend(batch)
|
||||
if len(batch) < 100:
|
||||
return result
|
||||
raise ValueError("too many candidate artifacts")
|
||||
|
||||
|
||||
def download_artifact(run, name, destination):
|
||||
matches = [item for item in artifact_list(run["id"]) if item.get("name") == name]
|
||||
require(len(matches) == 1, "candidate artifact is absent or ambiguous")
|
||||
artifact = validate_artifact(matches[0], run, name)
|
||||
destination = Path(destination)
|
||||
destination.mkdir(parents=True, exist_ok=True)
|
||||
with tempfile.TemporaryDirectory() as temp:
|
||||
archive_path = Path(temp) / "artifact.zip"
|
||||
with archive_path.open("wb") as stream, github_stream(f"repos/{REPOSITORY}/actions/artifacts/{artifact['id']}/zip") as response:
|
||||
received = 0
|
||||
for chunk in iter(lambda: response.read(1024 * 1024), b""):
|
||||
received += len(chunk)
|
||||
require(received <= artifact["size_in_bytes"], "artifact response exceeds its declared size")
|
||||
stream.write(chunk)
|
||||
require(archive_path.stat().st_size == artifact["size_in_bytes"] and
|
||||
"sha256:" + file_digest(archive_path) == artifact["digest"], "downloaded artifact digest or size mismatch")
|
||||
with zipfile.ZipFile(archive_path) as archive:
|
||||
members = archive.infolist()
|
||||
require(0 < len(members) <= 256 and len({m.filename for m in members}) == len(members), "invalid artifact members")
|
||||
require(sum(member.file_size for member in members) <= 8 * MAX_MEMBER, "expanded artifact exceeds the candidate size limit")
|
||||
for member in members:
|
||||
require(re.fullmatch(r"[A-Za-z0-9][A-Za-z0-9._-]{0,200}", member.filename)
|
||||
and not member.is_dir() and 0 < member.file_size <= MAX_MEMBER
|
||||
and (member.external_attr >> 16) & 0o170000 != 0o120000, "unsafe artifact member")
|
||||
target = destination / member.filename
|
||||
require(not target.exists(), "artifact extraction would overwrite a file")
|
||||
with archive.open(member) as source, target.open("xb") as sink:
|
||||
import shutil
|
||||
shutil.copyfileobj(source, sink)
|
||||
return artifact
|
||||
|
||||
|
||||
def emit_output(name, value):
|
||||
require("\n" not in str(value) and "\r" not in str(value), "invalid workflow output")
|
||||
with open(os.environ["GITHUB_OUTPUT"], "a", encoding="utf-8") as stream:
|
||||
stream.write(f"{name}={value}\n")
|
||||
|
||||
|
||||
def main(argv=None):
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
sub = parser.add_subparsers(dest="command", required=True)
|
||||
p = sub.add_parser("freeze")
|
||||
for name in ("version", "source-sha", "workflow-sha", "toolchain", "console", "cli", "previous", "output"):
|
||||
p.add_argument("--" + name, required=True)
|
||||
p.add_argument("--run-id", type=int, required=True)
|
||||
p.add_argument("--run-number", type=int, required=True)
|
||||
p.add_argument("--ci-run-id", type=int, required=True)
|
||||
p.add_argument("--ci-run-attempt", type=int, required=True)
|
||||
p = sub.add_parser("validate-inputs")
|
||||
p.add_argument("--inputs", required=True)
|
||||
p.add_argument("--check-checkout", action="store_true")
|
||||
p = sub.add_parser("record")
|
||||
p.add_argument("--inputs", required=True)
|
||||
p.add_argument("--target", required=True)
|
||||
p.add_argument("--archive", required=True)
|
||||
p.add_argument("--output", required=True)
|
||||
p = sub.add_parser("assemble")
|
||||
p.add_argument("--inputs", required=True)
|
||||
p.add_argument("--directory", required=True)
|
||||
p.add_argument("--output", required=True)
|
||||
p.add_argument("--platforms", choices=("linux", "all"), default="all")
|
||||
p = sub.add_parser("download")
|
||||
p.add_argument("--run-id", type=int, required=True)
|
||||
p.add_argument("--workflow", choices=("release-candidate.yml", "release-complete.yml"), required=True)
|
||||
p.add_argument("--name", required=True)
|
||||
p.add_argument("--destination", required=True)
|
||||
args = parser.parse_args(argv)
|
||||
if args.command == "freeze":
|
||||
write_json(args.output, freeze(args.version, args.source_sha, args.workflow_sha, args.toolchain,
|
||||
read_json(args.console), args.run_id, args.run_number, read_json(args.cli), read_json(args.previous),
|
||||
{"run_id": args.ci_run_id, "run_attempt": args.ci_run_attempt}))
|
||||
elif args.command == "validate-inputs":
|
||||
inputs = validate_inputs(read_json(args.inputs))
|
||||
if args.check_checkout:
|
||||
head = command(["git", "rev-parse", "HEAD"], cwd=ROOT)
|
||||
require(head == inputs["source_sha"], "checkout differs from the frozen source")
|
||||
require(recipe_digest(head) == inputs["recipe_sha256"], "release recipe differs from frozen inputs")
|
||||
require(file_digest(ROOT / "Cargo.lock") == inputs["lock_sha256"], "dependency lock differs from frozen inputs")
|
||||
print(digest(inputs))
|
||||
elif args.command == "record":
|
||||
write_json(args.output, archive_record(args.archive, args.target, read_json(args.inputs)))
|
||||
elif args.command == "assemble":
|
||||
inputs = validate_inputs(read_json(args.inputs))
|
||||
directory = Path(args.directory)
|
||||
records = [read_json(path) for path in sorted(directory.glob("record-*.json"))]
|
||||
write_json(args.output, certification_candidate(inputs, records, directory, targets(args.platforms)))
|
||||
elif args.command == "download":
|
||||
run = validate_run(github(f"repos/{REPOSITORY}/actions/runs/{args.run_id}"), args.workflow)
|
||||
download_artifact(run, args.name, args.destination)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
raise SystemExit(main())
|
||||
except (ValueError, OSError, KeyError, TypeError, zipfile.BadZipFile, subprocess.CalledProcessError) as error:
|
||||
print(f"candidate rejected: {error}", file=sys.stderr)
|
||||
raise SystemExit(1) from None
|
||||
@@ -0,0 +1,200 @@
|
||||
#!/usr/bin/env python3
|
||||
# Copyright 2026 RustFS Team
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
"""Wrap accepted binaries, then inspect the actual packages and OCI images."""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import shutil
|
||||
import subprocess
|
||||
import tarfile
|
||||
import tempfile
|
||||
import urllib.request
|
||||
import time
|
||||
|
||||
import candidate as c
|
||||
import pipeline
|
||||
|
||||
|
||||
def run(args, **kwargs):
|
||||
return subprocess.run(args, check=True, timeout=1800, **kwargs)
|
||||
|
||||
|
||||
def binary(inputs, target, output):
|
||||
origin = pipeline.get_run(inputs["origin_run_id"], "release-candidate.yml")
|
||||
c.download_artifact(origin, f"candidate-{inputs['candidate_id']}-{target}", output)
|
||||
record = c.read_json(output / f"record-{target}.json")
|
||||
c.verify_records(inputs, [record], output, (target,))
|
||||
pipeline.verify_evidence(inputs, output, {origin["id"]: origin})
|
||||
import verify
|
||||
path, _ = verify.unpack(output / record["name"], target, inputs, output / "bin")
|
||||
return path, record
|
||||
|
||||
|
||||
def reuse(kind, target):
|
||||
inputs = pipeline.manifest_env()
|
||||
current = pipeline.current_run("release-complete.yml")
|
||||
name = f"{kind}-{target}"
|
||||
exists = any(a["name"] == name for a in c.artifact_list(current["id"]))
|
||||
if exists:
|
||||
with tempfile.TemporaryDirectory() as temp:
|
||||
c.download_artifact(current, name, temp)
|
||||
record = c.read_json(Path(temp) / f"{name}.json")
|
||||
c.require(record.get("inputs_sha256") == c.digest(inputs), "existing delivery belongs to another candidate")
|
||||
# Successful jobs are immutable. The final seal independently checks
|
||||
# all sizes and digests before they can be approved for publication.
|
||||
c.emit_output("reused", str(exists).lower())
|
||||
|
||||
|
||||
def package(inputs, target, output):
|
||||
output.mkdir(parents=True, exist_ok=True)
|
||||
with tempfile.TemporaryDirectory() as temp:
|
||||
temp = Path(temp)
|
||||
executable, record = binary(inputs, target, temp / "source")
|
||||
arch, libc = target.split("-")[1:]
|
||||
build_type = "prerelease" if "-" in inputs["version"] else "release"
|
||||
values = c.command(["bash", "scripts/release/package_versions.sh", build_type, inputs["version"], "", arch, libc], cwd=c.ROOT)
|
||||
versions = dict(line.split("=", 1) for line in values.splitlines())
|
||||
# Existing package recipes operate in a private working directory.
|
||||
for name in ("scripts", "deploy"):
|
||||
(temp / name).symlink_to(c.ROOT / name, target_is_directory=True)
|
||||
for name in ("LICENSE", "README.md"):
|
||||
shutil.copyfile(c.ROOT / name, temp / name)
|
||||
(temp / "bin").mkdir()
|
||||
shutil.copyfile(executable, temp / "bin/rustfs")
|
||||
(temp / "bin/rustfs").chmod(0o755)
|
||||
env = dict(os.environ, **{key.upper(): value for key, value in versions.items()}, LIBC=libc,
|
||||
DEB_ARCH="amd64" if arch == "x86_64" else "arm64", RPM_ARCH=arch,
|
||||
DEB_DEPENDS="libc6 (>= 2.31)" if libc == "gnu" else "",
|
||||
RPM_DEPENDS="glibc >= 2.31" if libc == "gnu" else "", GITHUB_OUTPUT=str(temp / "outputs"))
|
||||
for kind in ("deb", "rpm"):
|
||||
run(["bash", str(c.ROOT / f"scripts/release/build_{kind}.sh")], cwd=temp, env=env)
|
||||
path = temp / versions[f"{kind}_file"]
|
||||
unpacked = temp / f"unpacked-{kind}"
|
||||
unpacked.mkdir()
|
||||
if kind == "deb":
|
||||
run(["dpkg-deb", "--extract", str(path), str(unpacked)])
|
||||
else:
|
||||
with subprocess.Popen(["rpm2cpio", str(path)], stdout=subprocess.PIPE) as source:
|
||||
run(["cpio", "--extract", "--make-directories", "--quiet", "./usr/bin/rustfs"], stdin=source.stdout, cwd=unpacked)
|
||||
source.stdout.close()
|
||||
c.require(source.wait(timeout=60) == 0, "RPM extraction failed")
|
||||
c.require(c.file_digest(unpacked / "usr/bin/rustfs") == record["binary_sha256"], "package binary differs from accepted ZIP")
|
||||
shutil.copyfile(path, output / path.name)
|
||||
files = pipeline.file_records(output)
|
||||
c.write_json(output / f"packages-{target}.json", {"inputs_sha256": c.digest(inputs), "binary_sha256": record["binary_sha256"], "files": files})
|
||||
|
||||
|
||||
def image_prepare(inputs, variant, output):
|
||||
output.mkdir(parents=True, exist_ok=True)
|
||||
binaries = {}
|
||||
for arch, docker_arch in (("x86_64", "amd64"), ("aarch64", "arm64")):
|
||||
target = f"linux-{arch}-{variant}"
|
||||
with tempfile.TemporaryDirectory() as temp:
|
||||
temp = Path(temp)
|
||||
executable, record = binary(inputs, target, temp / "source")
|
||||
directory = c.ROOT / ".release-inputs" / docker_arch
|
||||
directory.mkdir(parents=True, exist_ok=True)
|
||||
shutil.copyfile(executable, directory / "rustfs")
|
||||
(directory / "rustfs").chmod(0o755)
|
||||
cli = inputs["cli"][target]
|
||||
archive = temp / "cli.tar.gz"
|
||||
run(["curl", "--fail", "--location", "--retry", "3", "--output", str(archive), cli["url"]])
|
||||
c.require(c.file_digest(archive) == cli["sha256"], "CLI dependency changed")
|
||||
with tarfile.open(archive, "r:gz") as package:
|
||||
members = [m for m in package.getmembers() if m.name == "rc"]
|
||||
c.require(len(members) == 1 and members[0].isfile() and 0 < members[0].size <= c.MAX_MEMBER, "invalid CLI archive")
|
||||
with package.extractfile(members[0]) as source, (directory / "rc").open("wb") as dest:
|
||||
shutil.copyfileobj(source, dest)
|
||||
(directory / "rc").chmod(0o755)
|
||||
binaries[arch] = record["binary_sha256"]
|
||||
c.write_json(output / f"image-{variant}-inputs.json", {"inputs_sha256": c.digest(inputs), "binaries": binaries})
|
||||
|
||||
|
||||
def image_smoke(inputs, variant, output):
|
||||
archive = output / f"image-{variant}.tar"
|
||||
record = c.read_json(output / f"image-{variant}-inputs.json")
|
||||
scan_directory = Path(str(output) + "-scan")
|
||||
scan_directory.mkdir(parents=True, exist_ok=True)
|
||||
opener = urllib.request.build_opener(urllib.request.ProxyHandler({}))
|
||||
for arch, docker_arch in (("x86_64", "amd64"), ("aarch64", "arm64")):
|
||||
image = f"rustfs-acceptance:{variant}-{docker_arch}"
|
||||
run(["skopeo", "copy", "--override-os", "linux", "--override-arch", docker_arch, f"oci-archive:{archive}", f"docker-daemon:{image}"])
|
||||
container = c.command(["docker", "create", "--platform", f"linux/{docker_arch}", image])
|
||||
try:
|
||||
with tempfile.TemporaryDirectory() as temp:
|
||||
path = Path(temp) / "rustfs"
|
||||
run(["docker", "cp", f"{container}:/usr/bin/rustfs", str(path)])
|
||||
c.require(c.file_digest(path) == record["binaries"][arch], "image wraps a different executable")
|
||||
finally:
|
||||
run(["docker", "rm", "-v", container])
|
||||
version = c.command(["docker", "run", "--rm", "--platform", f"linux/{docker_arch}", "--entrypoint", "/usr/bin/rustfs", image, "--version"])
|
||||
c.require(version.splitlines()[0] == f"rustfs {inputs['version']}", "container displays wrong version")
|
||||
container = c.command(["docker", "run", "-d", "--platform", f"linux/{docker_arch}", "-p", "127.0.0.1::9001",
|
||||
"-e", "RUSTFS_ACCESS_KEY=release-smoke", "-e", "RUSTFS_SECRET_KEY=release-smoke-test-secret",
|
||||
"-e", "RUSTFS_CONSOLE_ENABLE=true", image])
|
||||
log = output / f"image-{variant}-{docker_arch}.log"
|
||||
try:
|
||||
address = c.command(["docker", "port", container, "9001/tcp"]).strip()
|
||||
deadline = time.monotonic() + 60
|
||||
while time.monotonic() < deadline:
|
||||
try:
|
||||
with opener.open(f"http://{address}/rustfs/console/", timeout=2) as response:
|
||||
c.require(response.status == 200 and "text/html" in response.headers.get("Content-Type", ""), "image Console did not return HTML")
|
||||
break
|
||||
except OSError:
|
||||
time.sleep(0.5)
|
||||
else:
|
||||
raise ValueError("container startup timed out")
|
||||
finally:
|
||||
with log.open("w") as stream:
|
||||
subprocess.run(["docker", "logs", container], stdout=stream, stderr=subprocess.STDOUT, check=True, timeout=60)
|
||||
run(["docker", "rm", "-f", "-v", container])
|
||||
with log.open("a") as stream:
|
||||
stream.write("\nConsole: HTTP 200 text/html; wrapped executable SHA-256 and version verified\n")
|
||||
# Trivy's local archive reader does not reliably select a platform from
|
||||
# a multi-architecture OCI index. Export the image just verified above.
|
||||
scan_archive = scan_directory / f"image-{variant}-{docker_arch}.tar"
|
||||
run(["docker", "save", "--output", str(scan_archive), image])
|
||||
record.setdefault("scan_inputs", {})[docker_arch] = c.file_digest(scan_archive)
|
||||
raw = subprocess.check_output(["skopeo", "inspect", "--raw", f"oci-archive:{archive}"], timeout=60)
|
||||
import hashlib
|
||||
record.update(digest="sha256:" + hashlib.sha256(raw).hexdigest(), sha256=c.file_digest(archive), size=archive.stat().st_size, smoke="pass")
|
||||
c.write_json(output / f"image-{variant}-smoke.json", record)
|
||||
|
||||
|
||||
def image_record(inputs, variant, output):
|
||||
record = c.read_json(output / f"image-{variant}-smoke.json")
|
||||
c.require(record["inputs_sha256"] == c.digest(inputs), "image inputs changed")
|
||||
for arch in ("amd64", "arm64"):
|
||||
scan_archive = Path(str(output) + "-scan") / f"image-{variant}-{arch}.tar"
|
||||
c.require(record.get("scan_inputs", {}).get(arch) == c.file_digest(scan_archive), "scan input changed after image verification")
|
||||
report = c.read_json(output / f"scan-{variant}-{arch}.sarif")
|
||||
c.require(report.get("version") == "2.1.0" and report.get("runs"), "container scan did not produce a SARIF report")
|
||||
record["scan"] = "completed"
|
||||
c.write_json(output / f"image-{variant}.json", record)
|
||||
|
||||
|
||||
def main():
|
||||
p = argparse.ArgumentParser(description=__doc__)
|
||||
p.add_argument("mode", choices=("reuse", "package", "image-prepare", "image-smoke", "image-record"))
|
||||
p.add_argument("--kind", choices=("packages", "image"))
|
||||
p.add_argument("--target", required=True)
|
||||
p.add_argument("--output", type=Path, default=Path("delivery"))
|
||||
args = p.parse_args()
|
||||
inputs = pipeline.manifest_env()
|
||||
if args.mode == "reuse":
|
||||
reuse(args.kind, args.target)
|
||||
elif args.mode == "package":
|
||||
c.require(args.target in c.LINUX, "unknown package platform")
|
||||
package(inputs, args.target, args.output)
|
||||
else:
|
||||
c.require(args.target in ("musl", "gnu"), "unknown image variant")
|
||||
{"image-prepare": image_prepare, "image-smoke": image_smoke, "image-record": image_record}[args.mode](inputs, args.target, args.output)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,361 @@
|
||||
#!/usr/bin/env python3
|
||||
# Copyright 2026 RustFS Team
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
"""Candidate orchestration. GitHub run identity is checked before trusting bytes."""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
import tempfile
|
||||
import urllib.error
|
||||
import zipfile
|
||||
|
||||
import candidate as c
|
||||
import release_gate
|
||||
|
||||
|
||||
def current_run(workflow):
|
||||
c.require(os.environ.get("GITHUB_REPOSITORY") == c.REPOSITORY and os.environ.get("GITHUB_REF") == "refs/heads/main",
|
||||
"release orchestration must run on the upstream main branch")
|
||||
run = c.validate_run(c.github(f"repos/{c.REPOSITORY}/actions/runs/{int(os.environ['GITHUB_RUN_ID'])}"), workflow, completed=False)
|
||||
c.require(run["head_sha"] == os.environ["GITHUB_SHA"], "workflow source changed")
|
||||
return run
|
||||
|
||||
|
||||
def get_run(run_id, workflow):
|
||||
run = c.github(f"repos/{c.REPOSITORY}/actions/runs/{int(run_id)}")
|
||||
active = str(run_id) == os.environ.get("GITHUB_RUN_ID")
|
||||
return c.validate_run(run, workflow, completed=not active)
|
||||
|
||||
|
||||
def manifest_env():
|
||||
return c.validate_inputs(json.loads(os.environ["CANDIDATE_MANIFEST"]))
|
||||
|
||||
|
||||
def output_inputs(inputs):
|
||||
c.emit_output("manifest", json.dumps(inputs, separators=(",", ":")))
|
||||
for field in ("candidate_id", "source_sha", "workflow_sha", "version"):
|
||||
c.emit_output(field, inputs[field])
|
||||
|
||||
|
||||
def source_ci(source, version):
|
||||
runs = c.github(f"repos/{c.REPOSITORY}/actions/workflows/ci.yml/runs?head_sha={source}&status=success&per_page=10")["workflow_runs"]
|
||||
required = {"Quick Checks", "Test and Lint", "End-to-End Tests (full merge gate)"}
|
||||
for run in runs:
|
||||
if (run.get("head_sha") != source or run.get("path") != ".github/workflows/ci.yml"
|
||||
or run.get("head_branch") not in ("main", f"release/{version}")
|
||||
or run.get("event") not in ("push", "workflow_dispatch")
|
||||
or run.get("conclusion") != "success" or run.get("status") != "completed"
|
||||
or (run.get("repository") or {}).get("full_name") != c.REPOSITORY):
|
||||
continue
|
||||
jobs = []
|
||||
for page in range(1, 11):
|
||||
batch = c.github(f"repos/{c.REPOSITORY}/actions/runs/{run['id']}/attempts/{run['run_attempt']}/jobs?per_page=100&page={page}")["jobs"]
|
||||
jobs.extend(batch)
|
||||
if len(batch) < 100:
|
||||
break
|
||||
selected = {name: [job for job in jobs if job.get("name") == name] for name in required}
|
||||
if all(len(found) == 1 and found[0].get("status") == "completed" and found[0].get("conclusion") == "success" for found in selected.values()):
|
||||
return {"run_id": run["id"], "run_attempt": run["run_attempt"]}
|
||||
raise ValueError("no complete exact-source CI found; run ci.yml on main or the release branch before creating a candidate")
|
||||
|
||||
|
||||
def unused_version(version):
|
||||
# Reject a spent version before allocating native builders. A partial
|
||||
# publication resumes its existing seal instead of making a new candidate.
|
||||
for tag in (version, "v" + version):
|
||||
for endpoint in ("git/ref/tags", "releases/tags"):
|
||||
try:
|
||||
c.github(f"repos/{c.REPOSITORY}/{endpoint}/{tag}")
|
||||
except urllib.error.HTTPError as error:
|
||||
if error.code == 404:
|
||||
continue
|
||||
raise
|
||||
raise ValueError("version already has a tag or release; resume its approved seal or choose a new version")
|
||||
|
||||
|
||||
def prepare(version, source, directory):
|
||||
run = current_run("release-candidate.yml")
|
||||
c.require(c.HEX40.fullmatch(source) and c.VERSION.fullmatch(version), "use a full source commit and exact version")
|
||||
directory.mkdir(parents=True, exist_ok=True)
|
||||
name = "release-inputs"
|
||||
exists = any(a["name"] == name for a in c.artifact_list(run["id"]))
|
||||
if exists:
|
||||
c.download_artifact(run, name, directory)
|
||||
inputs = c.validate_inputs(c.read_json(directory / "inputs.json"))
|
||||
c.require(inputs["source_sha"] == source and inputs["version"] == version and inputs["workflow_sha"] == run["head_sha"], "retry changed candidate inputs")
|
||||
else:
|
||||
unused_version(version)
|
||||
c.command(["git", "fetch", "--no-tags", "origin", source], cwd=c.ROOT)
|
||||
ci = source_ci(source, version)
|
||||
c.command(["rustup", "update", "stable", "--no-self-update"])
|
||||
toolchain = c.command(["rustc", "+stable", "--version"]).split()[1]
|
||||
inputs = c.freeze(version, source, run["head_sha"], toolchain, c.github("repos/rustfs/console/releases/latest"),
|
||||
run["id"], int(os.environ["GITHUB_RUN_NUMBER"]), c.github("repos/rustfs/cli/releases/latest"),
|
||||
c.github("repos/rustfs/rustfs/releases/latest"), ci)
|
||||
c.write_json(directory / "inputs.json", inputs)
|
||||
output_inputs(inputs)
|
||||
c.emit_output("reused", str(exists).lower())
|
||||
|
||||
|
||||
def reuse_platform(target):
|
||||
inputs = manifest_env()
|
||||
workflow = os.environ["GITHUB_WORKFLOW_REF"].split("@", 1)[0].rsplit("/", 1)[-1]
|
||||
run = current_run(workflow)
|
||||
name = f"candidate-{inputs['candidate_id']}-{target}"
|
||||
exists = any(a["name"] == name for a in c.artifact_list(run["id"]))
|
||||
if exists:
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
c.download_artifact(run, name, directory)
|
||||
records = [c.read_json(Path(directory) / f"record-{target}.json")]
|
||||
c.verify_records(inputs, records, directory, (target,))
|
||||
verify_evidence(inputs, Path(directory), {run["id"]: run})
|
||||
c.emit_output("reused", str(exists).lower())
|
||||
|
||||
|
||||
def load_linux(run_id, expected, directory):
|
||||
run = get_run(run_id, "release-candidate.yml")
|
||||
c.download_artifact(run, "release-linux", directory)
|
||||
inputs = c.validate_inputs(c.read_json(directory / "inputs.json"))
|
||||
c.require(inputs["origin_run_id"] == run["id"] and inputs["workflow_sha"] == run["head_sha"], "candidate inputs belong to another run")
|
||||
c.require(c.HEX64.fullmatch(expected or "") and c.file_digest(directory / "linux-manifest.json") == expected,
|
||||
"Linux approval digest does not match the downloaded manifest")
|
||||
verify_seal_files(c.read_json(directory / "linux-manifest.json"), directory)
|
||||
check_recipe(inputs)
|
||||
return inputs, run
|
||||
|
||||
|
||||
def check_recipe(inputs):
|
||||
# Unrelated main changes do not invalidate a frozen candidate. A changed
|
||||
# producer/verifier/publisher does, even when the product source is unchanged.
|
||||
c.require(c.recipe_digest(os.environ["GITHUB_SHA"]) == inputs["recipe_sha256"], "release recipe changed; create a new candidate")
|
||||
|
||||
|
||||
def evidence_files(directory):
|
||||
return sorted(list(directory.glob("identity-*.json")) + list(directory.glob("upgrade-*.json")))
|
||||
|
||||
|
||||
def verify_evidence(inputs, directory, runs):
|
||||
for path in evidence_files(directory):
|
||||
value = c.read_json(path)
|
||||
workflow = value.get("workflow", {})
|
||||
run = runs.get(int(workflow.get("run_id", 0)))
|
||||
c.require(run is not None and workflow.get("repository") == c.REPOSITORY
|
||||
and workflow.get("workflow_id") == run["path"].rsplit("/", 1)[-1]
|
||||
and workflow.get("workflow_sha") == run["head_sha"]
|
||||
and type(workflow.get("run_attempt")) is int and 0 < workflow["run_attempt"] <= run["run_attempt"],
|
||||
"receipt was not produced by the downloaded trusted run")
|
||||
c.require(value.get("harness_commit") == inputs["workflow_sha"], "receipt harness changed")
|
||||
for item in value.get("evidence", []):
|
||||
name = item.get("path", "")
|
||||
c.require(re.fullmatch(r"[A-Za-z0-9][A-Za-z0-9._-]{0,200}", name), "unsafe evidence path")
|
||||
log = directory / name
|
||||
c.require(log.is_file() and not log.is_symlink() and log.stat().st_size == item.get("size")
|
||||
and c.file_digest(log) == item.get("sha256"), "receipt evidence was removed or substituted")
|
||||
|
||||
|
||||
def certify(inputs, directory, runs, scope):
|
||||
records = [c.read_json(directory / f"record-{target}.json") for target in c.targets("linux" if scope == "linux" else "all")]
|
||||
candidate = c.certification_candidate(inputs, records, directory, c.targets("linux" if scope == "linux" else "all"))
|
||||
plan = c.read_json(c.ROOT / ".config/release-validation-plan.json")
|
||||
c.require(c.digest(plan) == inputs["plan_sha256"], "validation plan changed")
|
||||
verify_evidence(inputs, directory, runs)
|
||||
with tempfile.TemporaryDirectory() as temp:
|
||||
temp = Path(temp)
|
||||
results = temp / "results"
|
||||
results.mkdir()
|
||||
for path in evidence_files(directory):
|
||||
shutil.copyfile(path, results / path.name)
|
||||
c.write_json(temp / "candidate.json", candidate)
|
||||
c.write_json(temp / "plan.json", plan)
|
||||
status = release_gate.run_validate(temp / "plan.json", temp / "candidate.json", results, temp / "gate", scope)
|
||||
summary = c.read_json(temp / "gate/release-gate-summary.json")
|
||||
c.require(status == 0, "certification failed: " + json.dumps(summary["findings"]))
|
||||
return candidate, summary
|
||||
|
||||
|
||||
def file_records(directory, excluded=()):
|
||||
result = []
|
||||
for path in sorted(directory.iterdir()):
|
||||
if path.name in excluded:
|
||||
continue
|
||||
c.require(path.is_file() and not path.is_symlink() and re.fullmatch(r"[A-Za-z0-9][A-Za-z0-9._-]{0,200}", path.name), "unexpected staged file")
|
||||
result.append({"name": path.name, "sha256": c.file_digest(path), "size": path.stat().st_size})
|
||||
return result
|
||||
|
||||
|
||||
def verify_seal_files(seal, directory):
|
||||
c.require(isinstance(seal, dict) and seal.get("schema") == 1 and seal.get("stage") in ("linux", "complete"), "invalid release seal")
|
||||
filename = "linux-manifest.json" if seal["stage"] == "linux" else "release-manifest.json"
|
||||
c.require(seal.get("files") == file_records(directory, (filename,)), "sealed files changed, are missing, or contain extras")
|
||||
|
||||
|
||||
def aliases(inputs):
|
||||
return {f"rustfs-{target}-latest.zip": f"rustfs-{target}-v{inputs['version']}.zip" for target in c.TARGETS}
|
||||
|
||||
|
||||
def linux_seal(directory):
|
||||
run = current_run("release-candidate.yml")
|
||||
if any(a["name"] == "release-linux" for a in c.artifact_list(run["id"])):
|
||||
c.download_artifact(run, "release-linux", directory)
|
||||
inputs = c.validate_inputs(c.read_json(directory / "inputs.json"))
|
||||
verify_seal_files(c.read_json(directory / "linux-manifest.json"), directory)
|
||||
certify(inputs, directory, {run["id"]: run}, "linux")
|
||||
c.emit_output("reused", "true")
|
||||
c.emit_output("manifest_sha256", c.file_digest(directory / "linux-manifest.json"))
|
||||
return
|
||||
c.download_artifact(run, "release-inputs", directory)
|
||||
inputs = c.validate_inputs(c.read_json(directory / "inputs.json"))
|
||||
for target in c.LINUX:
|
||||
c.download_artifact(run, f"candidate-{inputs['candidate_id']}-{target}", directory)
|
||||
c.download_artifact(run, "release-upgrade", directory)
|
||||
_, summary = certify(inputs, directory, {run["id"]: run}, "linux")
|
||||
c.require(not summary["publish_allowed"], "partial certification must not grant publication authority")
|
||||
c.write_json(directory / "linux-manifest.json", {"schema": 1, "stage": "linux", "inputs_sha256": c.digest(inputs),
|
||||
"files": file_records(directory), "summary": summary})
|
||||
c.emit_output("manifest_sha256", c.file_digest(directory / "linux-manifest.json"))
|
||||
|
||||
|
||||
def check_acceptance(url):
|
||||
c.require(re.fullmatch(r"https://github\.com/rustfs/backlog/issues/[1-9][0-9]*(?:#issuecomment-[1-9][0-9]*)?", url),
|
||||
"acceptance must link to a RustFS backlog report")
|
||||
|
||||
|
||||
def complete_prepare(run_id, expected, report, directory):
|
||||
current_run("release-complete.yml")
|
||||
check_acceptance(report)
|
||||
inputs, run = load_linux(run_id, expected, directory)
|
||||
certify(inputs, directory, {run["id"]: run}, "linux")
|
||||
output_inputs(inputs)
|
||||
|
||||
|
||||
def complete_seal(run_id, expected, report, directory):
|
||||
current = current_run("release-complete.yml")
|
||||
if any(a["name"] == "release-seal" for a in c.artifact_list(current["id"])):
|
||||
c.download_artifact(current, "release-seal", directory)
|
||||
inputs = c.validate_inputs(c.read_json(directory / "inputs.json"))
|
||||
seal = c.read_json(directory / "release-manifest.json")
|
||||
verify_seal_files(seal, directory)
|
||||
c.require(inputs["origin_run_id"] == run_id and seal["acceptance"] == {"linux_manifest_sha256": expected, "report": report}, "completion retry changed acceptance")
|
||||
origin = get_run(run_id, "release-candidate.yml")
|
||||
certify(inputs, directory, {origin["id"]: origin, current["id"]: current}, "full")
|
||||
verify_deliveries(inputs, directory)
|
||||
c.emit_output("reused", "true")
|
||||
c.emit_output("manifest_sha256", c.file_digest(directory / "release-manifest.json"))
|
||||
return
|
||||
inputs, origin = load_linux(run_id, expected, directory)
|
||||
check_acceptance(report)
|
||||
(directory / "linux-manifest.json").unlink()
|
||||
for target in c.DESKTOP:
|
||||
c.download_artifact(current, f"candidate-{inputs['candidate_id']}-{target}", directory)
|
||||
for target in c.LINUX:
|
||||
c.download_artifact(current, f"packages-{target}", directory)
|
||||
for variant in ("musl", "gnu"):
|
||||
c.download_artifact(current, f"image-{variant}", directory)
|
||||
c.download_artifact(current, "helm-package", directory)
|
||||
candidate, summary = certify(inputs, directory, {origin["id"]: origin, current["id"]: current}, "full")
|
||||
verify_deliveries(inputs, directory)
|
||||
# Persist the small proof bundle with the public release; Actions retention
|
||||
# must not be the only way to audit an already published version.
|
||||
evidence = directory / "release-evidence.zip"
|
||||
with zipfile.ZipFile(evidence, "x", compression=zipfile.ZIP_DEFLATED) as archive:
|
||||
for path in sorted(directory.iterdir()):
|
||||
if path.suffix in (".json", ".log", ".sarif"):
|
||||
archive.write(path, path.name)
|
||||
archive.writestr("certification.json", json.dumps(summary, indent=2, sort_keys=True))
|
||||
archive.writestr("validation-plan.json", (c.ROOT / ".config/release-validation-plan.json").read_bytes())
|
||||
with tempfile.TemporaryDirectory() as temp:
|
||||
lock = Path(temp) / "Cargo.lock"
|
||||
lock.write_bytes(c.git_file(inputs["source_sha"], "Cargo.lock"))
|
||||
c.require(c.file_digest(lock) == inputs["lock_sha256"], "SBOM dependency lock changed")
|
||||
subprocess.run(["python3", str(c.ROOT / "scripts/security/generate_release_supply_chain_assets.py"),
|
||||
"--asset-dir", str(directory), "--version", inputs["version"], "--tag", inputs["version"],
|
||||
"--build-type", candidate["release_profile"], "--repository", c.REPOSITORY,
|
||||
"--ref", "refs/heads/main", "--sha", inputs["source_sha"], "--run-id", str(current["id"]),
|
||||
"--run-attempt", str(current["run_attempt"]), "--cargo-lock", str(lock),
|
||||
"--workflow", "release-complete.yml"], check=True, timeout=60)
|
||||
provenance = directory / f"rustfs-{inputs['version']}.provenance.json"
|
||||
value = c.read_json(provenance)
|
||||
value["predicate"]["buildDefinition"]["externalParameters"]["candidateInputs"] = inputs
|
||||
provenance.write_text(json.dumps(value, indent=2, sort_keys=True) + "\n")
|
||||
assets = {p.name: p for p in directory.iterdir() if p.suffix in (".zip", ".deb", ".rpm", ".tgz") or p.name.endswith((".sbom.cdx.json", ".provenance.json"))}
|
||||
assets.update({name: directory / source for name, source in aliases(inputs).items()})
|
||||
for algorithm in ("sha256", "sha512"):
|
||||
checksums = []
|
||||
for name, path in sorted(assets.items()):
|
||||
with path.open("rb") as stream:
|
||||
checksum = hashlib.file_digest(stream, algorithm).hexdigest()
|
||||
checksums.append(f"{checksum} {name}")
|
||||
(directory / f"{algorithm.upper()}SUMS").write_text("\n".join(checksums) + "\n")
|
||||
c.write_json(directory / "release-manifest.json", {
|
||||
"schema": 1, "stage": "complete", "inputs_sha256": c.digest(inputs), "candidate": candidate,
|
||||
"summary": summary, "files": file_records(directory), "complete_run_id": current["id"],
|
||||
"aliases": aliases(inputs),
|
||||
"acceptance": {"linux_manifest_sha256": expected, "report": report},
|
||||
})
|
||||
c.emit_output("manifest_sha256", c.file_digest(directory / "release-manifest.json"))
|
||||
|
||||
|
||||
def verify_deliveries(inputs, directory):
|
||||
chart = c.read_json(directory / "chart.json")
|
||||
c.require(chart.get("version") == inputs["version"] and chart.get("inputs_sha256") == c.digest(inputs)
|
||||
and chart.get("name") == f"rustfs-{inputs['version']}.tgz", "chart identity changed")
|
||||
path = directory / chart["name"]
|
||||
c.require(path.stat().st_size == chart["size"] and c.file_digest(path) == chart["sha256"], "chart bytes changed")
|
||||
for target in c.LINUX:
|
||||
binary = c.read_json(directory / f"record-{target}.json")["binary_sha256"]
|
||||
record = c.read_json(directory / f"packages-{target}.json")
|
||||
c.require(record.get("inputs_sha256") == c.digest(inputs) and record.get("binary_sha256") == binary, "package wraps a different candidate")
|
||||
c.require(len(record.get("files", [])) == 2 and {Path(f["name"]).suffix for f in record["files"]} == {".deb", ".rpm"}, "missing package format")
|
||||
for item in record["files"]:
|
||||
path = directory / item["name"]
|
||||
c.require(path.name == item["name"] and path.is_file() and path.stat().st_size == item["size"] and c.file_digest(path) == item["sha256"], "package changed")
|
||||
for variant in ("musl", "gnu"):
|
||||
record = c.read_json(directory / f"image-{variant}.json")
|
||||
c.require(record.get("inputs_sha256") == c.digest(inputs), "image inputs changed")
|
||||
expected = {arch: c.read_json(directory / f"record-linux-{arch}-{variant}.json")["binary_sha256"] for arch in ("x86_64", "aarch64")}
|
||||
c.require(record.get("binaries") == expected, "image contains an unverified binary")
|
||||
path = directory / f"image-{variant}.tar"
|
||||
c.require(c.file_digest(path) == record.get("sha256") and path.stat().st_size == record.get("size")
|
||||
and re.fullmatch(r"sha256:[0-9a-f]{64}", record.get("digest", "")), "image archive changed")
|
||||
c.require(record.get("smoke") == "pass" and record.get("scan") == "completed", "image verification did not finish")
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
sub = parser.add_subparsers(dest="mode", required=True)
|
||||
p = sub.add_parser("prepare")
|
||||
p.add_argument("--version", required=True)
|
||||
p.add_argument("--source", required=True)
|
||||
p.add_argument("--directory", type=Path, required=True)
|
||||
p = sub.add_parser("reuse-platform")
|
||||
p.add_argument("--target", choices=c.TARGETS, required=True)
|
||||
p = sub.add_parser("linux-seal")
|
||||
p.add_argument("--directory", type=Path, required=True)
|
||||
for name in ("complete-prepare", "complete-seal"):
|
||||
p = sub.add_parser(name)
|
||||
p.add_argument("--run-id", type=int, required=True)
|
||||
p.add_argument("--digest", required=True)
|
||||
p.add_argument("--report", required=True)
|
||||
p.add_argument("--directory", type=Path, required=True)
|
||||
args = parser.parse_args()
|
||||
if args.mode == "prepare":
|
||||
prepare(args.version, args.source, args.directory)
|
||||
elif args.mode == "reuse-platform":
|
||||
reuse_platform(args.target)
|
||||
elif args.mode == "linux-seal":
|
||||
linux_seal(args.directory)
|
||||
elif args.mode == "complete-prepare":
|
||||
complete_prepare(args.run_id, args.digest, args.report, args.directory)
|
||||
else:
|
||||
complete_seal(args.run_id, args.digest, args.report, args.directory)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,280 @@
|
||||
#!/usr/bin/env python3
|
||||
# Copyright 2026 RustFS Team
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
"""Promote an approved seal. This module never compiles or packages a deliverable."""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
import tempfile
|
||||
|
||||
import candidate as c
|
||||
import pipeline
|
||||
|
||||
REGISTRIES = ("docker.io/rustfs/rustfs", "ghcr.io/rustfs/rustfs", "quay.io/rustfs/rustfs")
|
||||
|
||||
|
||||
def api(path, method="GET", body=None, missing=False):
|
||||
args = ["gh", "api", "--method", method, path]
|
||||
if body is not None:
|
||||
args += ["--input", "-"]
|
||||
result = subprocess.run(args, input=json.dumps(body) if body is not None else None, text=True,
|
||||
capture_output=True, timeout=120)
|
||||
if result.returncode:
|
||||
if missing and "(HTTP 404)" in result.stderr:
|
||||
return None
|
||||
raise ValueError(f"GitHub {method} failed for {path}: {result.stderr.strip()}")
|
||||
return json.loads(result.stdout) if result.stdout.strip() else None
|
||||
|
||||
|
||||
def verify(run_id, expected, directory):
|
||||
current = pipeline.current_run("release-publish.yml")
|
||||
complete = pipeline.get_run(run_id, "release-complete.yml")
|
||||
c.download_artifact(complete, "release-seal", directory)
|
||||
manifest_path = directory / "release-manifest.json"
|
||||
c.require(c.HEX64.fullmatch(expected or "") and c.file_digest(manifest_path) == expected, "publication approval does not match the sealed manifest")
|
||||
seal = c.read_json(manifest_path)
|
||||
pipeline.verify_seal_files(seal, directory)
|
||||
c.require(seal["stage"] == "complete" and seal.get("complete_run_id") == complete["id"], "seal belongs to another completion run")
|
||||
inputs = c.validate_inputs(c.read_json(directory / "inputs.json"))
|
||||
c.require(seal.get("inputs_sha256") == c.digest(inputs), "seal inputs changed")
|
||||
c.require(seal.get("aliases") == pipeline.aliases(inputs), "release aliases differ from the approved versioned archives")
|
||||
pipeline.check_recipe(inputs)
|
||||
origin = pipeline.get_run(inputs["origin_run_id"], "release-candidate.yml")
|
||||
c.require(origin["head_sha"] == inputs["workflow_sha"], "candidate producer changed")
|
||||
candidate, summary = pipeline.certify(inputs, directory, {origin["id"]: origin, complete["id"]: complete}, "full")
|
||||
c.require(candidate == seal.get("candidate") and summary == seal.get("summary") and summary["publish_allowed"], "full certification does not match the seal")
|
||||
pipeline.verify_deliveries(inputs, directory)
|
||||
pipeline.check_acceptance(seal["acceptance"]["report"])
|
||||
c.require(c.HEX64.fullmatch(seal["acceptance"].get("linux_manifest_sha256", "")), "missing Linux acceptance digest")
|
||||
c.emit_output("version", inputs["version"])
|
||||
c.emit_output("source_sha", inputs["source_sha"])
|
||||
return inputs, seal, current
|
||||
|
||||
|
||||
def ensure_tag(inputs, manifest_sha):
|
||||
version, source = inputs["version"], inputs["source_sha"]
|
||||
message = f"RustFS {version}\nRelease manifest SHA-256: {manifest_sha}\n"
|
||||
path = f"repos/{c.REPOSITORY}/git/ref/tags/{version}"
|
||||
reference = api(path, missing=True)
|
||||
if reference is None:
|
||||
tag = api(f"repos/{c.REPOSITORY}/git/tags", "POST", {"tag": version, "message": message, "object": source, "type": "commit"})
|
||||
api(f"repos/{c.REPOSITORY}/git/refs", "POST", {"ref": f"refs/tags/{version}", "sha": tag["sha"]})
|
||||
reference = api(path)
|
||||
c.require(reference["object"]["type"] == "tag", "existing release tag is not an annotated candidate tag")
|
||||
tag = api(f"repos/{c.REPOSITORY}/git/tags/{reference['object']['sha']}")
|
||||
c.require(tag.get("tag") == version and tag.get("object", {}).get("sha") == source and tag.get("object", {}).get("type") == "commit"
|
||||
and tag.get("message", "").rstrip() == message.rstrip(), "public tag already identifies different source or bytes; never move it")
|
||||
|
||||
|
||||
def public_files(directory):
|
||||
names = {"release-manifest.json", "SHA256SUMS", "SHA512SUMS", "release-evidence.zip"}
|
||||
return sorted(p for p in directory.iterdir() if p.name in names or p.suffix in (".zip", ".deb", ".rpm", ".tgz") or p.name.endswith((".sbom.cdx.json", ".provenance.json")))
|
||||
|
||||
|
||||
def release(inputs, manifest_sha):
|
||||
endpoint = f"repos/{c.REPOSITORY}/releases"
|
||||
value = api(endpoint + "/tags/" + inputs["version"], missing=True)
|
||||
marker = f"Release manifest SHA-256: {manifest_sha}"
|
||||
if value is None:
|
||||
notes = api(endpoint + "/generate-notes", "POST", {"tag_name": inputs["version"], "target_commitish": inputs["source_sha"],
|
||||
"previous_tag_name": inputs["previous"]["version"]})
|
||||
value = api(endpoint, "POST", {"tag_name": inputs["version"], "target_commitish": inputs["source_sha"],
|
||||
"name": f"RustFS {inputs['version']}", "body": marker + "\n\n" + notes["body"],
|
||||
"draft": True, "prerelease": "-" in inputs["version"], "make_latest": "false"})
|
||||
c.require(marker in value.get("body", ""), "existing release belongs to another sealed candidate")
|
||||
return value
|
||||
|
||||
|
||||
def github_assets(inputs, value, paths):
|
||||
assets = []
|
||||
for page in range(1, 101):
|
||||
batch = api(f"repos/{c.REPOSITORY}/releases/{value['id']}/assets?per_page=100&page={page}")
|
||||
assets.extend(batch)
|
||||
if len(batch) < 100:
|
||||
break
|
||||
known = {a["name"]: a for a in assets}
|
||||
c.require(len(known) == len(assets) and set(known) <= {p.name for p in paths}, "release has unsealed or duplicate assets")
|
||||
for path in paths:
|
||||
old = known.get(path.name)
|
||||
if old is None:
|
||||
# No --clobber: a race or interrupted upload must fail closed.
|
||||
subprocess.run(["gh", "release", "upload", inputs["version"], str(path), "--repo", c.REPOSITORY], check=True, timeout=600)
|
||||
else:
|
||||
c.require(old.get("digest") == "sha256:" + c.file_digest(path) and old.get("size") == path.stat().st_size,
|
||||
"existing release asset differs; never overwrite accepted bytes")
|
||||
actual = api(f"repos/{c.REPOSITORY}/releases/{value['id']}/assets?per_page=100")
|
||||
c.require(len(actual) == len(paths), "release asset set incomplete")
|
||||
for path in paths:
|
||||
match = [a for a in actual if a["name"] == path.name]
|
||||
c.require(len(match) == 1 and match[0].get("digest") == "sha256:" + c.file_digest(path)
|
||||
and match[0].get("size") == path.stat().st_size, "uploaded release asset verification failed")
|
||||
|
||||
|
||||
def aws(*args, capture=True):
|
||||
return subprocess.run(["aws", "--endpoint-url", os.environ["R2_ENDPOINT"], *args],
|
||||
text=True, capture_output=capture, check=False, timeout=600)
|
||||
|
||||
|
||||
def r2_file(path, key, *, mutable=False):
|
||||
bucket = os.environ["R2_BUCKET"]
|
||||
expected = c.file_digest(path)
|
||||
head = aws("s3api", "head-object", "--bucket", bucket, "--key", key)
|
||||
if head.returncode and "(404)" not in head.stderr:
|
||||
raise ValueError("R2 lookup failed; refusing to treat an unavailable destination as empty")
|
||||
if head.returncode == 0:
|
||||
with tempfile.TemporaryDirectory() as temp:
|
||||
old = Path(temp) / "object"
|
||||
result = aws("s3", "cp", f"s3://{bucket}/{key}", str(old), "--only-show-errors")
|
||||
c.require(result.returncode == 0, "could not verify existing R2 object")
|
||||
if c.file_digest(old) == expected:
|
||||
return
|
||||
c.require(mutable, "existing versioned R2 object differs; never overwrite it")
|
||||
args = ["s3api", "put-object", "--bucket", bucket, "--key", key, "--body", str(path), "--metadata", json.dumps({"sha256": expected})]
|
||||
if not mutable:
|
||||
args += ["--if-none-match", "*"]
|
||||
result = aws(*args)
|
||||
c.require(result.returncode == 0, "R2 upload failed; retry the same seal after checking destination state")
|
||||
with tempfile.TemporaryDirectory() as temp:
|
||||
actual = Path(temp) / "object"
|
||||
result = aws("s3", "cp", f"s3://{bucket}/{key}", str(actual), "--only-show-errors")
|
||||
c.require(result.returncode == 0 and c.file_digest(actual) == expected, "R2 read-back digest mismatch")
|
||||
|
||||
|
||||
def image_digest(reference, *, missing=False):
|
||||
result = subprocess.run(["skopeo", "inspect", "--authfile", os.environ["REGISTRY_AUTH_FILE"], "--raw", f"docker://{reference}"],
|
||||
capture_output=True, timeout=120)
|
||||
if result.returncode:
|
||||
error = result.stderr.decode(errors="replace").lower()
|
||||
if missing and ("manifest unknown" in error or "manifest_unknown" in error or "name unknown" in error):
|
||||
return None
|
||||
raise ValueError(f"cannot verify registry manifest for {reference}")
|
||||
return "sha256:" + hashlib.sha256(result.stdout).hexdigest()
|
||||
|
||||
|
||||
def copy_image(archive, reference, expected, *, mutable=False):
|
||||
old = image_digest(reference, missing=True)
|
||||
if old == expected:
|
||||
return
|
||||
c.require(old is None or mutable, "existing image tag identifies different bytes; never replace it")
|
||||
subprocess.run(["skopeo", "copy", "--authfile", os.environ["REGISTRY_AUTH_FILE"], "--all", "--preserve-digests",
|
||||
f"oci-archive:{archive}", f"docker://{reference}"], check=True, timeout=1800)
|
||||
c.require(image_digest(reference) == expected, "registry changed the accepted image digest")
|
||||
|
||||
|
||||
def images(inputs, directory, *, latest=False):
|
||||
for variant, suffix in (("musl", ""), ("gnu", "-glibc")):
|
||||
record = c.read_json(directory / f"image-{variant}.json")
|
||||
for registry in REGISTRIES:
|
||||
tag = ("latest" if latest else inputs["version"]) + suffix
|
||||
copy_image(directory / f"image-{variant}.tar", f"{registry}:{tag}", record["digest"], mutable=latest)
|
||||
|
||||
|
||||
def publish_version(inputs, seal, directory, manifest_sha):
|
||||
ensure_tag(inputs, manifest_sha)
|
||||
value = release(inputs, manifest_sha)
|
||||
paths = public_files(directory)
|
||||
with tempfile.TemporaryDirectory() as temp:
|
||||
alias_paths = []
|
||||
for name, source in seal["aliases"].items():
|
||||
path = Path(temp) / name
|
||||
shutil.copyfile(directory / source, path)
|
||||
alias_paths.append(path)
|
||||
github_assets(inputs, value, paths + alias_paths)
|
||||
for path in paths:
|
||||
prefix = "artifacts/rustfs/release/"
|
||||
if path.name in ("release-manifest.json", "release-evidence.zip", "SHA256SUMS", "SHA512SUMS"):
|
||||
prefix += inputs["version"] + "/"
|
||||
r2_file(path, prefix + path.name)
|
||||
images(inputs, directory)
|
||||
# Every versioned destination is verified before the draft is made public.
|
||||
if value["draft"]:
|
||||
api(f"repos/{c.REPOSITORY}/releases/{value['id']}", "PATCH",
|
||||
{"draft": False, "prerelease": "-" in inputs["version"], "make_latest": "false"})
|
||||
|
||||
|
||||
def stable_key(version):
|
||||
c.require(isinstance(version, str) and re.fullmatch(r"v?(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)", version), "invalid stable channel version")
|
||||
return tuple(map(int, version.lstrip("v").split(".")))
|
||||
|
||||
|
||||
def helm(inputs, directory):
|
||||
record = c.read_json(directory / "chart.json")
|
||||
package = directory / record["name"]
|
||||
c.require(c.file_digest(package) == record["sha256"], "Helm package changed")
|
||||
env = dict(os.environ, GH_TOKEN=os.environ["HELM_TOKEN"])
|
||||
subprocess.run(["gh", "auth", "setup-git", "--hostname", "github.com"], env=env, check=True, timeout=60)
|
||||
with tempfile.TemporaryDirectory() as temp:
|
||||
subprocess.run(["git", "clone", "--depth", "1", "--branch", "main", "https://github.com/rustfs/helm.git", temp], env=env, check=True, timeout=120)
|
||||
path = Path(temp) / package.name
|
||||
if path.exists():
|
||||
c.require(c.file_digest(path) == record["sha256"], "published Helm version already has different bytes")
|
||||
c.require((Path(temp) / "index.yaml").is_file(), "existing Helm chart has no index")
|
||||
return
|
||||
shutil.copyfile(package, path)
|
||||
c.command(["helm", "repo", "index", ".", "--url", "https://charts.rustfs.com"], cwd=temp)
|
||||
c.command(["git", "config", "user.name", "RustFS Release"], cwd=temp)
|
||||
c.command(["git", "config", "user.email", "dev@rustfs.com"], cwd=temp)
|
||||
c.command(["git", "add", package.name, "index.yaml"], cwd=temp)
|
||||
c.command(["git", "commit", "-m", f"chore: publish chart {inputs['version']}"], cwd=temp)
|
||||
subprocess.run(["git", "push", "origin", "HEAD:main"], cwd=temp, env=env, check=True, timeout=120)
|
||||
|
||||
|
||||
def channels(inputs, seal, directory, manifest_sha):
|
||||
# Recheck all immutable destinations, including when resuming a failed job.
|
||||
publish_version(inputs, seal, directory, manifest_sha)
|
||||
# The versioned Helm catalog entry must still be completed when a newer
|
||||
# release already owns the mutable channels.
|
||||
helm(inputs, directory)
|
||||
if "-" in inputs["version"]:
|
||||
return
|
||||
latest = api(f"repos/{c.REPOSITORY}/releases/latest", missing=True)
|
||||
if latest and stable_key(latest["tag_name"]) > stable_key(inputs["version"]):
|
||||
print("A newer stable release owns the channels; keeping its pointers.")
|
||||
return
|
||||
value = release(inputs, manifest_sha)
|
||||
# GitHub Latest also acts as a monotonic marker during partial channel
|
||||
# failures. Versioned assets and images are already fully available here.
|
||||
api(f"repos/{c.REPOSITORY}/releases/{value['id']}", "PATCH", {"make_latest": "true"})
|
||||
images(inputs, directory, latest=True)
|
||||
for target in c.TARGETS:
|
||||
path = directory / f"rustfs-{target}-v{inputs['version']}.zip"
|
||||
r2_file(path, f"artifacts/rustfs/release/rustfs-{target}-latest.zip", mutable=True)
|
||||
pointer = {"version": inputs["version"], "tag": inputs["version"], "release_date": value.get("published_at"),
|
||||
"release_type": "stable", "download_url": f"https://github.com/{c.REPOSITORY}/releases/tag/{inputs['version']}"}
|
||||
with tempfile.TemporaryDirectory() as temp:
|
||||
path = Path(temp) / "latest.json"
|
||||
c.write_json(path, pointer)
|
||||
subprocess.run([os.environ["OSSUTIL_BIN"], "cp", str(path), "oss://rustfs-version/latest.json", "--force"], check=True, timeout=120)
|
||||
actual = Path(temp) / "actual.json"
|
||||
subprocess.run([os.environ["OSSUTIL_BIN"], "cp", "oss://rustfs-version/latest.json", str(actual)], check=True, timeout=120)
|
||||
c.require(c.read_json(actual) == pointer, "installer pointer did not match the published release")
|
||||
|
||||
|
||||
def main():
|
||||
p = argparse.ArgumentParser(description=__doc__)
|
||||
p.add_argument("mode", choices=("verify", "version", "channels"))
|
||||
p.add_argument("--run-id", type=int, required=True)
|
||||
p.add_argument("--digest", required=True)
|
||||
p.add_argument("--directory", type=Path, required=True)
|
||||
args = p.parse_args()
|
||||
inputs, seal, current = verify(args.run_id, args.digest, args.directory)
|
||||
if args.mode != "verify":
|
||||
for name in ("R2_ENDPOINT", "R2_BUCKET", "AWS_ACCESS_KEY_ID", "AWS_SECRET_ACCESS_KEY", "REGISTRY_AUTH_FILE"):
|
||||
c.require(bool(os.environ.get(name)), f"missing publication setting: {name}")
|
||||
if args.mode == "channels":
|
||||
c.require(os.environ.get("HELM_TOKEN") and os.environ.get("OSSUTIL_BIN"), "missing channel publication settings")
|
||||
{"version": publish_version, "channels": channels}[args.mode](inputs, seal, args.directory, args.digest)
|
||||
c.write_json("publication.json", {"schema": 1, "version": inputs["version"], "manifest_sha256": args.digest,
|
||||
"source_sha": inputs["source_sha"], "complete_run_id": args.run_id,
|
||||
"publish_run_id": current["id"], "stage": args.mode})
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -263,9 +263,12 @@ def validate_plan(plan):
|
||||
harness_repository = require_string(
|
||||
entry, "harness_repository", code, f"plan lane {lane}"
|
||||
)
|
||||
harness_sha = require_hex(
|
||||
entry.get("harness_sha"), HEX40, code, "harness_sha", f"plan lane {lane}"
|
||||
)
|
||||
harness_sha = entry.get("harness_sha")
|
||||
if harness_sha == "candidate.workflow_sha":
|
||||
if harness_repository != "rustfs/rustfs":
|
||||
raise GateError(code, "dynamic harness identity is limited to the product repository")
|
||||
else:
|
||||
harness_sha = require_hex(harness_sha, HEX40, code, "harness_sha", f"plan lane {lane}")
|
||||
timeout_seconds = entry.get("timeout_seconds")
|
||||
if not isinstance(timeout_seconds, int) or timeout_seconds <= 0:
|
||||
raise GateError(
|
||||
@@ -500,7 +503,12 @@ def validate_candidate(candidate, plan_info, plan_sha256):
|
||||
)
|
||||
image = {"ref": ref, "digest": digest, "binary_sha256": image_binary}
|
||||
|
||||
workflow_sha = candidate.get("workflow_sha")
|
||||
if any(lane["harness_sha"] == "candidate.workflow_sha" for lane in plan_info["lanes"].values()):
|
||||
workflow_sha = require_hex(workflow_sha, HEX40, code, "workflow_sha", "candidate")
|
||||
|
||||
return {
|
||||
"workflow_sha": workflow_sha,
|
||||
"product_commit": product_commit,
|
||||
"version": version,
|
||||
"release_profile": release_profile,
|
||||
@@ -616,7 +624,10 @@ def validate_envelope(envelope, plan_info, candidate_info, source_name):
|
||||
|
||||
lane_info = plan_info["lanes"].get(lane)
|
||||
if lane_info is not None:
|
||||
if harness_commit != lane_info["harness_sha"]:
|
||||
expected_harness = lane_info["harness_sha"]
|
||||
if expected_harness == "candidate.workflow_sha":
|
||||
expected_harness = candidate_info["workflow_sha"]
|
||||
if harness_commit != expected_harness:
|
||||
raise GateError(
|
||||
"harness_mismatch",
|
||||
f"{context}: harness_commit does not match the fixed harness SHA for lane {lane}",
|
||||
@@ -658,6 +669,14 @@ def validate_envelope(envelope, plan_info, candidate_info, source_name):
|
||||
)
|
||||
|
||||
artifact = envelope.get("artifact")
|
||||
if state == "pass" and candidate_info.get("workflow_sha"):
|
||||
if not isinstance(artifact, dict) or not artifact.get("archive_sha256") or not artifact.get("binary_sha256"):
|
||||
raise GateError(code, f"{context}: candidate PASS must bind both archive and executable bytes")
|
||||
target = "-".join(part for part in platform_key(platform) if part not in ("none", "msvc"))
|
||||
name = f"rustfs-{target}-v{candidate_info['version']}.zip"
|
||||
matches = [d for d in candidate_info["deliverables"] if d["name"] == name]
|
||||
if len(matches) != 1 or matches[0]["sha256"] != artifact["archive_sha256"] or matches[0]["binary_sha256"] != artifact["binary_sha256"]:
|
||||
raise GateError("digest_mismatch", f"{context}: evidence belongs to a different platform artifact")
|
||||
if artifact is not None:
|
||||
if not isinstance(artifact, dict):
|
||||
raise GateError(code, f"{context}: artifact must be an object")
|
||||
@@ -734,7 +753,7 @@ def _validate_artifact_binding(artifact, candidate_info, context, case_id, lane)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def run_validate(plan_path, candidate_path, results_dir, output_dir):
|
||||
def run_validate(plan_path, candidate_path, results_dir, output_dir, scope="full"):
|
||||
findings = []
|
||||
plan_info = None
|
||||
candidate_info = None
|
||||
@@ -755,6 +774,10 @@ def run_validate(plan_path, candidate_path, results_dir, output_dir):
|
||||
return 1
|
||||
|
||||
required = expand_required(plan_info["lanes"], candidate_info["release_profile"])
|
||||
if scope == "linux":
|
||||
required = [case for case in required if case["platform"]["os"] == "linux"]
|
||||
elif scope != "full":
|
||||
raise GateError("plan_invalid", "unknown validation scope")
|
||||
expansion = {
|
||||
"schema_version": SCHEMA_VERSION,
|
||||
"plan_sha256": plan_info["plan_sha256"],
|
||||
@@ -915,7 +938,7 @@ def run_validate(plan_path, candidate_path, results_dir, output_dir):
|
||||
)
|
||||
)
|
||||
|
||||
_write_outputs(output, findings, plan_info, candidate_info, required, envelopes)
|
||||
_write_outputs(output, findings, plan_info, candidate_info, required, envelopes, scope=scope)
|
||||
return 1 if findings else 0
|
||||
|
||||
|
||||
@@ -939,12 +962,13 @@ def _validate_batch(document, candidate_info, source_name):
|
||||
return {"run_id": run_id, "run_attempt": run_attempt, "status": status}
|
||||
|
||||
|
||||
def _write_outputs(output, findings, plan_info, candidate_info, required, envelopes):
|
||||
def _write_outputs(output, findings, plan_info, candidate_info, required, envelopes, scope="full"):
|
||||
state = "blocked" if findings else "pass"
|
||||
summary = {
|
||||
"schema_version": SCHEMA_VERSION,
|
||||
"state": state,
|
||||
"publish_allowed": not findings,
|
||||
"scope": scope,
|
||||
"publish_allowed": not findings and scope == "full",
|
||||
"plan_sha256": plan_info["plan_sha256"] if plan_info else None,
|
||||
"candidate_sha256": candidate_info["candidate_sha256"] if candidate_info else None,
|
||||
"product_commit": candidate_info["product_commit"] if candidate_info else None,
|
||||
@@ -972,7 +996,7 @@ def _write_outputs(output, findings, plan_info, candidate_info, required, envelo
|
||||
"# Release gate summary",
|
||||
"",
|
||||
f"- state: {state}",
|
||||
f"- publish_allowed: {str(not findings).lower()}",
|
||||
f"- publish_allowed: {str(summary['publish_allowed']).lower()}",
|
||||
f"- plan_sha256: {summary['plan_sha256']}",
|
||||
f"- candidate_sha256: {summary['candidate_sha256']}",
|
||||
f"- required cases: {len(required)}",
|
||||
@@ -1190,6 +1214,7 @@ def main(argv=None):
|
||||
validate_parser.add_argument("--candidate", required=True)
|
||||
validate_parser.add_argument("--results", required=True)
|
||||
validate_parser.add_argument("--output", required=True)
|
||||
validate_parser.add_argument("--scope", choices=("full", "linux"), default="full")
|
||||
|
||||
promote_parser = subparsers.add_parser(
|
||||
"verify-promotion", help="re-check to-be-published bytes against the passed list"
|
||||
@@ -1216,6 +1241,7 @@ def main(argv=None):
|
||||
arguments.candidate,
|
||||
arguments.results,
|
||||
arguments.output,
|
||||
scope=arguments.scope,
|
||||
)
|
||||
except GateError as error:
|
||||
print(f"{error.code}: {error.message}", file=sys.stderr)
|
||||
|
||||
@@ -0,0 +1,279 @@
|
||||
#!/usr/bin/env python3
|
||||
# Copyright 2026 RustFS Team
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
"""Exercise release identity, substitution and retry boundaries without publishing."""
|
||||
from __future__ import annotations
|
||||
|
||||
import copy
|
||||
import io
|
||||
import os
|
||||
import json
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
import urllib.request
|
||||
import zipfile
|
||||
|
||||
import candidate
|
||||
|
||||
|
||||
class TransportTests(unittest.TestCase):
|
||||
def test_signed_download_redirect_does_not_leak_the_github_token(self):
|
||||
request = urllib.request.Request("https://api.github.com/repos/rustfs/rustfs/actions/artifacts/1/zip", headers={"Authorization": "Bearer test-token"})
|
||||
redirected = candidate.ArtifactRedirect().redirect_request(request, None, 302, "Found", {}, "https://example.blob.core.windows.net/artifact.zip?signature=fixture")
|
||||
self.assertIsNone(redirected.get_header("Authorization"))
|
||||
same_host = candidate.ArtifactRedirect().redirect_request(request, None, 302, "Found", {}, "https://api.github.com/another-path")
|
||||
self.assertEqual(same_host.get_header("Authorization"), "Bearer test-token")
|
||||
with self.assertRaises(ValueError):
|
||||
candidate.ArtifactRedirect().redirect_request(request, None, 302, "Found", {}, "http://example.invalid/artifact.zip")
|
||||
|
||||
def test_download_checks_real_bytes_and_rejects_traversal(self):
|
||||
import hashlib
|
||||
run = {"id": 70, "head_sha": "a" * 40}
|
||||
def blob(name):
|
||||
stream = io.BytesIO()
|
||||
with zipfile.ZipFile(stream, "w") as package:
|
||||
package.writestr(name, b"accepted")
|
||||
return stream.getvalue()
|
||||
for name, valid in (("record.json", True), ("../escape", False)):
|
||||
content = blob(name)
|
||||
artifact = {"id": 90, "name": "candidate", "expired": False, "workflow_run": run,
|
||||
"size_in_bytes": len(content), "digest": "sha256:" + hashlib.sha256(content).hexdigest()}
|
||||
with tempfile.TemporaryDirectory() as temp, patch.object(candidate, "artifact_list", return_value=[artifact]), patch.object(candidate, "github_stream", side_effect=lambda _: io.BytesIO(content)):
|
||||
if valid:
|
||||
candidate.download_artifact(run, "candidate", temp)
|
||||
self.assertEqual((Path(temp) / name).read_bytes(), b"accepted")
|
||||
artifact["digest"] = "sha256:" + "0" * 64
|
||||
with self.assertRaisesRegex(ValueError, "digest"):
|
||||
candidate.download_artifact(run, "candidate", Path(temp) / "changed")
|
||||
else:
|
||||
with self.assertRaisesRegex(ValueError, "unsafe"):
|
||||
candidate.download_artifact(run, "candidate", temp)
|
||||
|
||||
def test_duplicate_json_fields_are_not_silently_accepted(self):
|
||||
with tempfile.TemporaryDirectory() as temp:
|
||||
path = Path(temp) / "manifest.json"
|
||||
path.write_text('{"schema": 1, "schema": 2}')
|
||||
with self.assertRaisesRegex(ValueError, "duplicate"):
|
||||
candidate.read_json(path)
|
||||
|
||||
|
||||
class FreezeRepositoryTests(unittest.TestCase):
|
||||
def test_freeze_reads_real_commit_and_rejects_recipe_or_version_drift(self):
|
||||
with tempfile.TemporaryDirectory() as temp:
|
||||
root = Path(temp)
|
||||
for name in candidate.RECIPE_PATHS:
|
||||
path = root / name; path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_text("recipe\n" if name != ".config/release-validation-plan.json" else "{}\n")
|
||||
for directory in ("bootstrap", "helm"):
|
||||
(root / directory).mkdir()
|
||||
(root / directory / "input.txt").write_text("runtime input\n")
|
||||
(root / "Cargo.toml").write_text('[workspace.package]\nversion = "1.2.3"\n')
|
||||
(root / "Cargo.lock").write_text("version = 4\n")
|
||||
env = dict(os.environ, GIT_AUTHOR_NAME="Fixture", GIT_AUTHOR_EMAIL="fixture@example.invalid", GIT_COMMITTER_NAME="Fixture", GIT_COMMITTER_EMAIL="fixture@example.invalid")
|
||||
def git(*args):
|
||||
return subprocess.check_output(["git", *args], cwd=root, env=env, stderr=subprocess.DEVNULL, text=True).strip()
|
||||
git("init", "-q"); git("add", "."); git("commit", "-qm", "fixture")
|
||||
original = git("rev-parse", "HEAD")
|
||||
value = inputs()
|
||||
console = {"draft": False, "prerelease": False, "tag_name": value["console"]["tag"],
|
||||
"assets": [{"name": value["console"]["name"], "browser_download_url": value["console"]["url"], "digest": "sha256:" + value["console"]["sha256"]}]}
|
||||
cli = {"draft": False, "prerelease": False, "tag_name": "v0.1.0", "assets": []}
|
||||
for arch in ("amd64", "arm64"):
|
||||
for abi in ("", "-gnu"):
|
||||
name = f"rustfs-cli-linux-{arch}{abi}-v0.1.0.tar.gz"
|
||||
cli["assets"].append({"name": name, "browser_download_url": f"https://github.com/rustfs/cli/releases/download/v0.1.0/{name}", "digest": "sha256:" + "f" * 64})
|
||||
previous = {"draft": False, "prerelease": False, "tag_name": "1.2.2", "assets": [{"name": value["previous"]["name"], "browser_download_url": value["previous"]["url"], "digest": "sha256:" + value["previous"]["sha256"]}]}
|
||||
def freeze(source, workflow, version="1.2.3"):
|
||||
return candidate.freeze(version, source, workflow, "1.92.0", console, 70, 7, cli, previous, {"run_id": 60, "run_attempt": 1}, root)
|
||||
frozen = freeze(original, original)
|
||||
(root / "unrelated.txt").write_text("main may advance\n")
|
||||
git("add", "."); git("commit", "-qm", "unrelated")
|
||||
advanced = git("rev-parse", "HEAD")
|
||||
self.assertEqual(freeze(original, advanced)["recipe_sha256"], frozen["recipe_sha256"])
|
||||
(root / ".github/workflows/build.yml").write_text("different recipe\n")
|
||||
git("add", "."); git("commit", "-qm", "recipe change")
|
||||
with self.assertRaisesRegex(ValueError, "recipe"):
|
||||
freeze(original, git("rev-parse", "HEAD"))
|
||||
with self.assertRaisesRegex(ValueError, "version"):
|
||||
freeze(original, original, "1.2.4")
|
||||
|
||||
|
||||
def inputs():
|
||||
value = {
|
||||
"schema": 1, "candidate_id": "1.2.3-preview.7", "version": "1.2.3", "ci": {"run_id": 60, "run_attempt": 1},
|
||||
"source_sha": "a" * 40, "workflow_sha": "b" * 40, "recipe_sha256": "c" * 64,
|
||||
"lock_sha256": "d" * 64, "plan_sha256": "e" * 64, "toolchain": "1.92.0",
|
||||
"console": {"tag": "v0.1.41", "name": "rustfs-console-v0.1.41.zip",
|
||||
"url": "https://github.com/rustfs/console/releases/download/v0.1.41/rustfs-console-v0.1.41.zip",
|
||||
"sha256": "f" * 64}, "origin_run_id": 70, "origin_run_number": 7,
|
||||
}
|
||||
def asset(repo, version, name):
|
||||
return {"version": version, "name": name, "url": f"https://github.com/{repo}/releases/download/{version}/{name}", "sha256": "f" * 64}
|
||||
value["previous"] = asset("rustfs/rustfs", "1.2.2", "rustfs-linux-x86_64-gnu-v1.2.2.zip")
|
||||
value["cli"] = {target: asset("rustfs/cli", "v0.1.0", "rustfs-cli.tar.gz") for target in candidate.LINUX}
|
||||
return value
|
||||
|
||||
|
||||
def archive(directory, target, value=None, *, extra=None, binary=b"exact candidate bytes"):
|
||||
value = value or inputs()
|
||||
path = Path(directory) / f"rustfs-{target}-v{value['version']}.zip"
|
||||
members = {"rustfs.exe" if target == "windows-x86_64" else "rustfs": binary}
|
||||
if target.endswith("-gnu"):
|
||||
members.update({"rustfs.cpu-symbol-catalog.json": b"{}", "rustfs.cpu-symbol-catalog.sha256": b"catalog"})
|
||||
if extra:
|
||||
members.update(extra)
|
||||
with zipfile.ZipFile(path, "w") as output:
|
||||
for name, content in members.items():
|
||||
output.writestr(name, content)
|
||||
return path
|
||||
|
||||
|
||||
class InputIdentityTests(unittest.TestCase):
|
||||
def test_valid_frozen_candidate(self):
|
||||
self.assertEqual(candidate.validate_inputs(inputs()), inputs())
|
||||
|
||||
def test_moving_or_injected_inputs_rejected(self):
|
||||
changes = [("source_sha", "main"), ("version", "1.2.3\nTAG=bad"), ("version", "01.2.3"),
|
||||
("candidate_id", "1.2.3-preview.8"), ("toolchain", "stable"), ("origin_run_id", True),
|
||||
("schema", True), ("recipe_sha256", "c" * 63)]
|
||||
for key, value in changes:
|
||||
with self.subTest(key=key, value=value):
|
||||
record = inputs(); record[key] = value
|
||||
with self.assertRaises(ValueError):
|
||||
candidate.validate_inputs(record)
|
||||
|
||||
def test_console_asset_cannot_redirect_or_be_ambiguous(self):
|
||||
record = inputs()
|
||||
record["console"]["url"] = "https://example.invalid/console.zip"
|
||||
with self.assertRaises(ValueError):
|
||||
candidate.validate_inputs(record)
|
||||
asset = {"name": inputs()["console"]["name"], "browser_download_url": inputs()["console"]["url"], "digest": "sha256:" + "f" * 64}
|
||||
for release in ({"draft": True, "prerelease": False, "tag_name": "v0.1.41", "assets": [asset]},
|
||||
{"draft": False, "prerelease": False, "tag_name": "v0.1.41", "assets": [asset, asset]}):
|
||||
with self.assertRaises(ValueError):
|
||||
candidate.console_asset(release)
|
||||
|
||||
def test_frozen_record_is_retryable_but_not_replaceable(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
path = Path(directory) / "inputs.json"
|
||||
candidate.write_json(path, inputs())
|
||||
before = path.read_bytes()
|
||||
candidate.write_json(path, inputs())
|
||||
self.assertEqual(before, path.read_bytes())
|
||||
changed = inputs(); changed["source_sha"] = "f" * 40
|
||||
with self.assertRaisesRegex(ValueError, "replace"):
|
||||
candidate.write_json(path, changed)
|
||||
self.assertEqual(before, path.read_bytes())
|
||||
|
||||
def test_platform_selection_keeps_desktop_out_of_linux(self):
|
||||
self.assertEqual(candidate.targets("linux"), candidate.LINUX)
|
||||
self.assertEqual(candidate.targets("desktop"), candidate.DESKTOP)
|
||||
for value in ("", "linux-x86_64-gnu,linux-x86_64-gnu", "linux,windows", "all,linux"):
|
||||
with self.subTest(value=value), self.assertRaises(ValueError):
|
||||
candidate.targets(value)
|
||||
|
||||
|
||||
class ArtifactIdentityTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.temp = tempfile.TemporaryDirectory(); self.addCleanup(self.temp.cleanup)
|
||||
self.root = Path(self.temp.name)
|
||||
self.inputs = inputs()
|
||||
self.records = [candidate.archive_record(archive(self.root, target), target, self.inputs) for target in candidate.TARGETS]
|
||||
|
||||
def test_all_six_final_artifacts_bind_exact_bytes(self):
|
||||
value = candidate.certification_candidate(self.inputs, self.records, self.root)
|
||||
self.assertEqual(len(value["deliverables"]), 6)
|
||||
self.assertEqual(value["version"], "1.2.3")
|
||||
self.assertEqual(value["release_profile"], "stable")
|
||||
self.assertEqual(value["validation_batch"], {"run_id": "1.2.3-preview.7", "run_attempt": 1})
|
||||
self.assertNotIn("latest", json.dumps(value))
|
||||
|
||||
def test_partial_linux_bundle_cannot_be_published_as_complete(self):
|
||||
linux = self.records[:4]
|
||||
self.assertEqual(len(candidate.certification_candidate(self.inputs, linux, self.root, candidate.LINUX)["deliverables"]), 4)
|
||||
with self.assertRaisesRegex(ValueError, "missing"):
|
||||
candidate.certification_candidate(self.inputs, linux, self.root)
|
||||
|
||||
def test_same_version_different_bytes_rejected(self):
|
||||
archive(self.root, candidate.TARGETS[0], binary=b"replacement")
|
||||
with self.assertRaisesRegex(ValueError, "changed"):
|
||||
candidate.verify_records(self.inputs, self.records, self.root, candidate.TARGETS)
|
||||
|
||||
def test_different_inputs_cannot_share_receipts(self):
|
||||
changed = copy.deepcopy(self.inputs); changed["console"]["sha256"] = "0" * 64
|
||||
with self.assertRaisesRegex(ValueError, "changed"):
|
||||
candidate.verify_records(changed, self.records, self.root, candidate.TARGETS)
|
||||
|
||||
def test_duplicate_or_mislabeled_platform_rejected(self):
|
||||
for replacement in (self.records[:-1] + [self.records[0]], self.records + [self.records[0]]):
|
||||
with self.assertRaises(ValueError):
|
||||
candidate.verify_records(self.inputs, replacement, self.root, candidate.TARGETS)
|
||||
wrong = copy.deepcopy(self.records); wrong[0]["name"] = "../replacement.zip"
|
||||
with self.assertRaises(ValueError):
|
||||
candidate.verify_records(self.inputs, wrong, self.root, candidate.TARGETS)
|
||||
|
||||
def test_archive_members_are_strict(self):
|
||||
for name in ("../escape", "rustfs-cli", "nested/rustfs"):
|
||||
with self.subTest(name=name):
|
||||
path = archive(self.root, candidate.TARGETS[0], extra={name: b"extra"})
|
||||
with self.assertRaises(ValueError):
|
||||
candidate.archive_record(path, candidate.TARGETS[0], self.inputs)
|
||||
|
||||
def test_binary_symlink_rejected(self):
|
||||
path = self.root / f"rustfs-{candidate.TARGETS[0]}-v1.2.3.zip"
|
||||
with zipfile.ZipFile(path, "w") as output:
|
||||
entry = zipfile.ZipInfo("rustfs"); entry.external_attr = 0o120777 << 16
|
||||
output.writestr(entry, "../../untrusted")
|
||||
with self.assertRaises(ValueError):
|
||||
candidate.archive_record(path, candidate.TARGETS[0], self.inputs)
|
||||
|
||||
def test_real_cli_fails_without_overwriting_existing_output(self):
|
||||
candidate.write_json(self.root / "inputs.json", self.inputs)
|
||||
for item in self.records:
|
||||
candidate.write_json(self.root / f"record-{item['target']}.json", item)
|
||||
output = self.root / "candidate.json"
|
||||
command = [sys.executable, str(Path(candidate.__file__)), "assemble", "--inputs", str(self.root / "inputs.json"),
|
||||
"--directory", str(self.root), "--output", str(output)]
|
||||
good = subprocess.run(command, text=True, capture_output=True)
|
||||
self.assertEqual(good.returncode, 0, good.stderr)
|
||||
original = output.read_bytes()
|
||||
archive(self.root, candidate.TARGETS[1], binary=b"bad retry")
|
||||
bad = subprocess.run(command, text=True, capture_output=True)
|
||||
self.assertNotEqual(bad.returncode, 0)
|
||||
self.assertEqual(output.read_bytes(), original)
|
||||
|
||||
|
||||
class ProvenanceTests(unittest.TestCase):
|
||||
def run_record(self):
|
||||
return {"repository": {"full_name": "rustfs/rustfs"}, "path": ".github/workflows/release-candidate.yml",
|
||||
"event": "workflow_dispatch", "head_branch": "main", "id": 70, "run_attempt": 1,
|
||||
"head_sha": "b" * 40, "status": "completed", "conclusion": "success"}
|
||||
|
||||
def test_trusted_source_and_completion_required(self):
|
||||
run = self.run_record()
|
||||
self.assertEqual(candidate.validate_run(run, "release-candidate.yml"), run)
|
||||
for field, value in (("head_branch", "unreviewed"), ("event", "pull_request"), ("conclusion", "failure"),
|
||||
("status", "in_progress"), ("repository", {"full_name": "fork/rustfs"}),
|
||||
("path", ".github/workflows/other.yml")):
|
||||
with self.subTest(field=field):
|
||||
changed = {**run, field: value}
|
||||
with self.assertRaises(ValueError):
|
||||
candidate.validate_run(changed, "release-candidate.yml")
|
||||
|
||||
def test_artifact_must_match_exact_origin(self):
|
||||
run = self.run_record()
|
||||
artifact = {"name": "candidate", "expired": False, "id": 1, "size_in_bytes": 100,
|
||||
"digest": "sha256:" + "c" * 64, "workflow_run": {"id": 70, "head_sha": "b" * 40}}
|
||||
candidate.validate_artifact(artifact, run, "candidate")
|
||||
for field, value in (("expired", True), ("name", "other"), ("digest", ""), ("size_in_bytes", True),
|
||||
("workflow_run", {"id": 71, "head_sha": "b" * 40})):
|
||||
with self.subTest(field=field), self.assertRaises(ValueError):
|
||||
candidate.validate_artifact({**artifact, field: value}, run, "candidate")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,453 @@
|
||||
#!/usr/bin/env python3
|
||||
# Copyright 2026 RustFS Team
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
"""Offline contract rehearsals; these are not product acceptance receipts."""
|
||||
from __future__ import annotations
|
||||
|
||||
import copy
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
import zipfile
|
||||
|
||||
import candidate as c
|
||||
import pipeline
|
||||
import publish
|
||||
import release_gate
|
||||
import verify
|
||||
from test_candidate import archive, inputs
|
||||
|
||||
ROOT = c.ROOT
|
||||
|
||||
|
||||
def workflow(name):
|
||||
# Ruby/Psych is available on the developer host and hosted CI image. Tests
|
||||
# parse the actual workflow rather than a separately maintained job graph.
|
||||
result = subprocess.check_output(["ruby", "-ryaml", "-rjson", "-e",
|
||||
"puts JSON.generate(YAML.load_file(ARGV[0]))", str(ROOT / ".github/workflows" / name)], text=True)
|
||||
value = json.loads(result)
|
||||
value["on"] = value.pop("true", value.get("on"))
|
||||
return value
|
||||
|
||||
|
||||
def run_metadata(run_id, file, sha):
|
||||
return {"id": run_id, "run_attempt": 2, "head_sha": sha, "head_branch": "main", "event": "workflow_dispatch",
|
||||
"status": "completed", "conclusion": "success", "path": f".github/workflows/{file}",
|
||||
"repository": {"full_name": c.REPOSITORY}}
|
||||
|
||||
|
||||
def fixture(directory, selection="all"):
|
||||
value = inputs()
|
||||
value["plan_sha256"] = c.digest(c.read_json(ROOT / ".config/release-validation-plan.json"))
|
||||
runs = {70: run_metadata(70, "release-candidate.yml", value["workflow_sha"]),
|
||||
80: run_metadata(80, "release-complete.yml", "c" * 40)}
|
||||
for target in c.targets(selection):
|
||||
record = c.archive_record(archive(directory, target, value, binary=target.encode()), target, value)
|
||||
c.write_json(directory / f"record-{target}.json", record)
|
||||
run = runs[70 if target in c.LINUX else 80]
|
||||
env = {"GITHUB_WORKFLOW_REF": f"{c.REPOSITORY}/{run['path']}@refs/heads/main", "GITHUB_SHA": run["head_sha"],
|
||||
"GITHUB_RUN_ID": str(run["id"]), "GITHUB_RUN_ATTEMPT": "1"}
|
||||
with patch.dict(os.environ, env):
|
||||
for case in ("artifact-identity", "startup"):
|
||||
log = directory / f"{target}-{case}.log"
|
||||
log.write_text("contract fixture only\n")
|
||||
verify.envelope(value, record, "identity", case, [log], verify.now(), directory)
|
||||
if target == "linux-x86_64-gnu":
|
||||
for case in verify.UPGRADES:
|
||||
log = directory / f"upgrade-{case}.log"
|
||||
log.write_text("contract fixture only\n")
|
||||
verify.envelope(value, record, "upgrade", case, [log], verify.now(), directory)
|
||||
return value, runs
|
||||
|
||||
|
||||
class CertificationTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.temp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(self.temp.cleanup)
|
||||
self.path = Path(self.temp.name)
|
||||
self.value, self.runs = fixture(self.path)
|
||||
|
||||
def test_complete_original_bytes_certify(self):
|
||||
_, summary = pipeline.certify(self.value, self.path, self.runs, "full")
|
||||
self.assertTrue(summary["publish_allowed"])
|
||||
self.assertEqual(len(summary["required_cases"]), 18)
|
||||
|
||||
def test_linux_only_cannot_authorize_release(self):
|
||||
with tempfile.TemporaryDirectory() as temp:
|
||||
value, runs = fixture(Path(temp), "linux")
|
||||
candidate, summary = pipeline.certify(value, Path(temp), runs, "linux")
|
||||
self.assertEqual(summary["state"], "pass")
|
||||
self.assertFalse(summary["publish_allowed"])
|
||||
self.assertEqual(len(summary["required_cases"]), 14)
|
||||
self.assertEqual(len(candidate["deliverables"]), 4)
|
||||
with self.assertRaises((ValueError, OSError)):
|
||||
pipeline.certify(value, Path(temp), runs, "full")
|
||||
|
||||
def mutate_envelope(self, edit):
|
||||
path = self.path / "identity-linux-x86_64-gnu-startup.json"
|
||||
value = c.read_json(path)
|
||||
edit(value)
|
||||
path.write_text(json.dumps(value))
|
||||
|
||||
def test_digest_binding_is_required_and_platform_specific(self):
|
||||
original = (self.path / "identity-linux-x86_64-gnu-startup.json").read_bytes()
|
||||
arm = c.read_json(self.path / "record-linux-aarch64-gnu.json")
|
||||
edits = [lambda v: v.pop("artifact"), lambda v: v["artifact"].update(archive_sha256="0" * 64),
|
||||
lambda v: v["artifact"].update(archive_sha256=arm["sha256"], binary_sha256=arm["binary_sha256"])]
|
||||
for edit in edits:
|
||||
with self.subTest(edit=edit):
|
||||
(self.path / "identity-linux-x86_64-gnu-startup.json").write_bytes(original)
|
||||
self.mutate_envelope(edit)
|
||||
with self.assertRaises(ValueError):
|
||||
pipeline.certify(self.value, self.path, self.runs, "full")
|
||||
|
||||
def test_missing_changed_and_duplicate_evidence_are_rejected(self):
|
||||
path = self.path / "linux-x86_64-gnu-startup.log"
|
||||
path.write_text("substituted")
|
||||
with self.assertRaisesRegex(ValueError, "evidence"):
|
||||
pipeline.certify(self.value, self.path, self.runs, "full")
|
||||
|
||||
def test_receipt_cannot_claim_another_run_or_new_attempt(self):
|
||||
self.mutate_envelope(lambda v: v["workflow"].update(run_attempt=3))
|
||||
with self.assertRaisesRegex(ValueError, "trusted run"):
|
||||
pipeline.certify(self.value, self.path, self.runs, "full")
|
||||
|
||||
def test_old_successful_unit_survives_failed_job_retry(self):
|
||||
# The candidate batch is immutable; successful unit receipts from an
|
||||
# earlier attempt still bind the exact archive, plan and producer.
|
||||
_, summary = pipeline.certify(self.value, self.path, self.runs, "full")
|
||||
self.assertEqual(summary["state"], "pass")
|
||||
self.mutate_envelope(lambda v: v.update(run_id="different-candidate"))
|
||||
with self.assertRaises(ValueError):
|
||||
pipeline.certify(self.value, self.path, self.runs, "full")
|
||||
|
||||
def test_new_verifier_or_plan_cannot_reuse_old_acceptance(self):
|
||||
self.value["workflow_sha"] = "d" * 40
|
||||
with self.assertRaises(ValueError):
|
||||
pipeline.certify(self.value, self.path, self.runs, "full")
|
||||
|
||||
|
||||
class SourceCITests(unittest.TestCase):
|
||||
def test_spent_versions_and_unavailable_lookups_fail_before_building(self):
|
||||
import urllib.error
|
||||
missing = urllib.error.HTTPError('https://api.github.com/fixture', 404, 'Not Found', {}, None)
|
||||
with patch.object(c, "github", side_effect=missing) as api:
|
||||
pipeline.unused_version("1.2.3")
|
||||
self.assertEqual(api.call_count, 4)
|
||||
with patch.object(c, "github", return_value={"id": 1}):
|
||||
with self.assertRaisesRegex(ValueError, "already has a tag or release"):
|
||||
pipeline.unused_version("1.2.3")
|
||||
unavailable = urllib.error.HTTPError('https://api.github.com/fixture', 503, 'Unavailable', {}, None)
|
||||
with patch.object(c, "github", side_effect=unavailable):
|
||||
with self.assertRaises(urllib.error.HTTPError):
|
||||
pipeline.unused_version("1.2.3")
|
||||
|
||||
def test_only_complete_exact_source_ci_is_accepted(self):
|
||||
run = run_metadata(60, "ci.yml", "a" * 40)
|
||||
jobs = [{"name": name, "status": "completed", "conclusion": "success"} for name in
|
||||
("Quick Checks", "Test and Lint", "End-to-End Tests (full merge gate)")]
|
||||
def api(path):
|
||||
return {"jobs": jobs} if "/jobs?" in path else {"workflow_runs": [run]}
|
||||
with patch.object(c, "github", side_effect=api):
|
||||
self.assertEqual(pipeline.source_ci("a" * 40, "1.2.3"), {"run_id": 60, "run_attempt": 2})
|
||||
for key, wrong in (("head_sha", "b" * 40), ("head_branch", "unreviewed"), ("event", "pull_request"), ("conclusion", "failure")):
|
||||
old = run[key]; run[key] = wrong
|
||||
with self.subTest(key=key), self.assertRaises(ValueError):
|
||||
pipeline.source_ci("a" * 40, "1.2.3")
|
||||
run[key] = old
|
||||
for conclusion in ("skipped", "failure", None):
|
||||
jobs[2]["conclusion"] = conclusion
|
||||
with self.subTest(conclusion=conclusion), self.assertRaises(ValueError):
|
||||
pipeline.source_ci("a" * 40, "1.2.3")
|
||||
|
||||
def test_duplicate_required_ci_job_is_rejected(self):
|
||||
run = run_metadata(60, "ci.yml", "a" * 40)
|
||||
jobs = [{"name": name, "status": "completed", "conclusion": "success"} for name in
|
||||
("Quick Checks", "Quick Checks", "Test and Lint", "End-to-End Tests (full merge gate)")]
|
||||
with patch.object(c, "github", side_effect=lambda path: {"jobs": jobs} if "/jobs?" in path else {"workflow_runs": [run]}):
|
||||
with self.assertRaises(ValueError):
|
||||
pipeline.source_ci("a" * 40, "1.2.3")
|
||||
|
||||
|
||||
class SupplyChainTests(unittest.TestCase):
|
||||
def test_actual_generator_uses_frozen_lock_and_all_package_formats(self):
|
||||
with tempfile.TemporaryDirectory() as temp:
|
||||
root = Path(temp)
|
||||
assets = root / "assets"; assets.mkdir()
|
||||
for name in ("server.zip", "package.deb", "package.rpm", "chart.tgz"):
|
||||
(assets / name).write_bytes(name.encode())
|
||||
lock = root / "Cargo.lock"
|
||||
lock.write_text('version = 4\n[[package]]\nname = "frozen-fixture"\nversion = "1.2.3"\n')
|
||||
args = [sys.executable, str(ROOT / "scripts/security/generate_release_supply_chain_assets.py"), "--asset-dir", str(assets),
|
||||
"--version", "1.2.3", "--cargo-lock", str(lock), "--workflow", "release-complete.yml", "--sha", "a" * 40]
|
||||
result = subprocess.run(args, capture_output=True, text=True, timeout=10)
|
||||
self.assertEqual(result.returncode, 0, result.stderr)
|
||||
provenance = c.read_json(assets / "rustfs-1.2.3.provenance.json")
|
||||
self.assertEqual({s["name"] for s in provenance["subject"]}, {"server.zip", "package.deb", "package.rpm", "chart.tgz"})
|
||||
self.assertTrue(provenance["predicate"]["buildDefinition"]["externalParameters"]["workflow"].endswith("/release-complete.yml"))
|
||||
sbom = c.read_json(assets / "rustfs-1.2.3.sbom.cdx.json")
|
||||
self.assertIn("frozen-fixture", {item["name"] for item in sbom["components"]})
|
||||
lock.unlink()
|
||||
result = subprocess.run(args, capture_output=True, text=True, timeout=10)
|
||||
self.assertNotEqual(result.returncode, 0)
|
||||
|
||||
|
||||
class NativeEntryTests(unittest.TestCase):
|
||||
def test_real_native_entry_point_version_startup_and_failure(self):
|
||||
# A small child process exercises the verifier entry point and lifecycle;
|
||||
# it deliberately does not pretend to be a RustFS product test.
|
||||
target = "macos-aarch64"
|
||||
value = inputs()
|
||||
version = f"rustfs {value['version']}\nbuild target : {verify.TRIPLES[target]}\ngit commit : {value['source_sha']}\nrust version : {value['toolchain']}\n"
|
||||
source = f'''#!/usr/bin/env python3
|
||||
import sys, threading
|
||||
from http.server import BaseHTTPRequestHandler, HTTPServer
|
||||
store = {{}}
|
||||
if '--version' in sys.argv:
|
||||
print({version!r}, end='')
|
||||
else:
|
||||
class Handler(BaseHTTPRequestHandler):
|
||||
def do_GET(self):
|
||||
self.send_response(200)
|
||||
self.send_header('Content-Type', 'text/html')
|
||||
self.end_headers()
|
||||
self.wfile.write(store.get(self.path, b'<html>fixture</html>'))
|
||||
def do_PUT(self):
|
||||
store[self.path] = self.rfile.read(int(self.headers.get('Content-Length', '0')))
|
||||
self.send_response(200); self.end_headers()
|
||||
def do_DELETE(self):
|
||||
store.pop(self.path, None)
|
||||
self.send_response(204); self.end_headers()
|
||||
api = sys.argv[sys.argv.index('--address')+1].split(':')
|
||||
server = HTTPServer((api[0], int(api[1])), Handler)
|
||||
threading.Thread(target=server.serve_forever, daemon=True).start()
|
||||
address=sys.argv[sys.argv.index('--console-address')+1]
|
||||
host, port=address.split(':')
|
||||
HTTPServer((host,int(port)),Handler).serve_forever()
|
||||
'''
|
||||
with tempfile.TemporaryDirectory() as temp:
|
||||
path = Path(temp)
|
||||
c.write_json(path / "inputs.json", value)
|
||||
binary = archive(path, target, value, binary=source.encode())
|
||||
env = dict(os.environ, GITHUB_WORKFLOW_REF=f"{c.REPOSITORY}/.github/workflows/release-complete.yml@refs/heads/main",
|
||||
GITHUB_RUN_ID="80", GITHUB_RUN_ATTEMPT="1", GITHUB_SHA="c" * 40)
|
||||
args = [sys.executable, str(ROOT / "scripts/release/verify.py"), "native", "--inputs", str(path / "inputs.json"),
|
||||
"--archive", str(binary), "--target", target, "--output", str(path / "receipts")]
|
||||
result = subprocess.run(args, env=env, capture_output=True, text=True, timeout=30)
|
||||
self.assertEqual(result.returncode, 0, result.stderr)
|
||||
self.assertEqual(len(list((path / "receipts").glob("*.json"))), 2)
|
||||
archive(path, target, value, binary=source.replace("rustfs 1.2.3", "rustfs 1.2.4").encode())
|
||||
args[-1] = str(path / "rejected")
|
||||
result = subprocess.run(args, env=env, capture_output=True, text=True, timeout=30)
|
||||
self.assertNotEqual(result.returncode, 0)
|
||||
self.assertFalse(list((path / "rejected").glob("*.json")))
|
||||
|
||||
|
||||
class PromotionTests(unittest.TestCase):
|
||||
def test_tag_is_created_once_and_conflicting_tag_never_moved(self):
|
||||
value = inputs()
|
||||
state = {}
|
||||
calls = []
|
||||
def api(path, method="GET", body=None, missing=False):
|
||||
calls.append((path, method, body))
|
||||
if path.endswith("/git/tags"):
|
||||
state["tag"] = dict(body, object={"sha": body["object"], "type": body["type"]})
|
||||
return {"sha": "f" * 40}
|
||||
if path.endswith("/git/refs"):
|
||||
state["ref"] = {"object": {"type": "tag", "sha": body["sha"]}}
|
||||
return state["ref"]
|
||||
if "/git/ref/tags/" in path:
|
||||
return state.get("ref")
|
||||
return state["tag"]
|
||||
with patch.object(publish, "api", side_effect=api):
|
||||
publish.ensure_tag(value, "a" * 64)
|
||||
count = len([call for call in calls if call[1] == "POST"])
|
||||
publish.ensure_tag(value, "a" * 64)
|
||||
self.assertEqual(len([call for call in calls if call[1] == "POST"]), count)
|
||||
with self.assertRaisesRegex(ValueError, "never move"):
|
||||
publish.ensure_tag(value, "b" * 64)
|
||||
state["tag"]["object"]["sha"] = "e" * 40
|
||||
with self.assertRaises(ValueError):
|
||||
publish.ensure_tag(value, "a" * 64)
|
||||
self.assertFalse(any(method in ("PATCH", "DELETE") for _, method, _ in calls))
|
||||
|
||||
def test_image_promotion_preserves_digest_and_refuses_overwrite(self):
|
||||
with patch.dict(os.environ, REGISTRY_AUTH_FILE="auth.json"), patch.object(publish, "image_digest", return_value="sha256:" + "a" * 64), patch.object(subprocess, "run") as command:
|
||||
publish.copy_image(Path("image.tar"), "registry/image:1.2.3", "sha256:" + "a" * 64)
|
||||
command.assert_not_called()
|
||||
with self.assertRaisesRegex(ValueError, "never replace"):
|
||||
publish.copy_image(Path("image.tar"), "registry/image:1.2.3", "sha256:" + "b" * 64)
|
||||
command.assert_not_called()
|
||||
with patch.dict(os.environ, REGISTRY_AUTH_FILE="auth.json"), patch.object(publish, "image_digest", side_effect=[None, "sha256:" + "a" * 64]), patch.object(subprocess, "run") as command:
|
||||
publish.copy_image(Path("image.tar"), "registry/image:1.2.3", "sha256:" + "a" * 64)
|
||||
self.assertIn("--preserve-digests", command.call_args.args[0])
|
||||
self.assertIn("--all", command.call_args.args[0])
|
||||
|
||||
def test_newer_channel_cannot_be_downgraded_after_old_job_retry(self):
|
||||
with patch.object(publish, "publish_version") as versions, patch.object(publish, "api", return_value={"tag_name": "1.2.4"}) as api, patch.object(publish, "images") as images, patch.object(publish, "helm") as helm:
|
||||
publish.channels(inputs(), {}, Path("candidate"), "a" * 64)
|
||||
versions.assert_called_once()
|
||||
images.assert_not_called()
|
||||
helm.assert_called_once()
|
||||
self.assertTrue(all(call.args[1:] == () for call in api.call_args_list))
|
||||
|
||||
def test_prerelease_never_changes_stable_pointers(self):
|
||||
value = inputs(); value["version"] = "1.3.0-rc.1"
|
||||
with patch.object(publish, "publish_version"), patch.object(publish, "api") as api, patch.object(publish, "images") as images, patch.object(publish, "helm") as helm:
|
||||
publish.channels(value, {}, Path("candidate"), "a" * 64)
|
||||
api.assert_not_called(); images.assert_not_called(); helm.assert_called_once()
|
||||
|
||||
def test_interrupted_version_publish_never_advances_channels(self):
|
||||
with patch.object(publish, "publish_version", side_effect=ValueError("registry unavailable")), patch.object(publish, "api") as api:
|
||||
with self.assertRaises(ValueError):
|
||||
publish.channels(inputs(), {}, Path("candidate"), "a" * 64)
|
||||
api.assert_not_called()
|
||||
|
||||
def test_incomplete_helm_publication_does_not_advance_stable_channels(self):
|
||||
with patch.object(publish, "publish_version"), patch.object(publish, "helm", side_effect=ValueError("chart repository unavailable")), patch.object(publish, "api") as api, patch.object(publish, "images") as images:
|
||||
with self.assertRaises(ValueError):
|
||||
publish.channels(inputs(), {}, Path("candidate"), "a" * 64)
|
||||
api.assert_not_called()
|
||||
images.assert_not_called()
|
||||
|
||||
def test_r2_retry_checks_bytes_and_uses_conditional_creation(self):
|
||||
with tempfile.TemporaryDirectory() as temp, patch.dict(os.environ, R2_BUCKET="releases"):
|
||||
path = Path(temp) / "asset.zip"; path.write_bytes(b"accepted")
|
||||
objects = {}
|
||||
calls = []
|
||||
def aws(*args, **kwargs):
|
||||
calls.append(args)
|
||||
if args[:2] == ("s3api", "head-object"):
|
||||
exists = args[args.index("--key") + 1] in objects
|
||||
return subprocess.CompletedProcess(args, 0 if exists else 1, "{}", "" if exists else "(404)")
|
||||
if args[:2] == ("s3api", "put-object"):
|
||||
self.assertIn("--if-none-match", args)
|
||||
objects[args[args.index("--key") + 1]] = Path(args[args.index("--body") + 1]).read_bytes()
|
||||
else:
|
||||
Path(args[3]).write_bytes(objects[args[2].split("/", 3)[3]])
|
||||
return subprocess.CompletedProcess(args, 0, "{}", "")
|
||||
with patch.object(publish, "aws", side_effect=aws):
|
||||
publish.r2_file(path, "version/asset.zip")
|
||||
publish.r2_file(path, "version/asset.zip")
|
||||
self.assertEqual(sum(call[:2] == ("s3api", "put-object") for call in calls), 1)
|
||||
objects["version/asset.zip"] = b"changed"
|
||||
with self.assertRaisesRegex(ValueError, "never overwrite"):
|
||||
publish.r2_file(path, "version/asset.zip")
|
||||
|
||||
|
||||
class WorkflowContractTests(unittest.TestCase):
|
||||
def test_nightly_avoids_desktop_and_manual_builds_keep_platform_selection(self):
|
||||
value = workflow("build.yml")
|
||||
selection = value["jobs"]["prepare-platform-matrix"]["steps"][-1]
|
||||
self.assertEqual(selection["env"]["RAW_PLATFORMS"], "${{ inputs.platforms || 'linux' }}")
|
||||
self.assertEqual(value["on"]["workflow_dispatch"]["inputs"]["platforms"]["default"], "all")
|
||||
|
||||
def test_main_push_cannot_cancel_the_scheduled_oidc_gate(self):
|
||||
value = workflow("oidc-keycloak.yml")
|
||||
self.assertIn("github.event_name", value["concurrency"]["group"])
|
||||
self.assertIn("!= 'schedule'", value["concurrency"]["cancel-in-progress"])
|
||||
|
||||
def test_linux_is_the_only_initial_build(self):
|
||||
value = workflow("release-candidate.yml")
|
||||
self.assertEqual(set(value["on"]), {"workflow_dispatch"})
|
||||
self.assertEqual(value["jobs"]["linux"]["with"]["platforms"], "linux")
|
||||
self.assertNotIn("desktop", value["jobs"])
|
||||
self.assertEqual(set(value["jobs"]["seal-linux"]["needs"]), {"freeze", "linux", "upgrade"})
|
||||
|
||||
def test_desktop_and_deliveries_wait_for_linux_acceptance(self):
|
||||
value = workflow("release-complete.yml")
|
||||
self.assertEqual(value["jobs"]["acceptance"]["environment"], "release-linux-acceptance")
|
||||
for name in ("desktop", "packages", "images", "chart"):
|
||||
self.assertEqual(value["jobs"][name]["needs"], "acceptance")
|
||||
self.assertNotIn("secrets", value["jobs"][name])
|
||||
self.assertEqual(set(value["jobs"]["seal"]["needs"]), {"acceptance", "desktop", "packages", "images", "chart"})
|
||||
|
||||
def test_only_publisher_has_release_writes_and_never_builds(self):
|
||||
value = workflow("release-publish.yml")
|
||||
self.assertEqual(set(value["on"]), {"workflow_dispatch"})
|
||||
self.assertEqual(value["jobs"]["version"]["environment"], "release-publication")
|
||||
self.assertEqual(value["jobs"]["channels"]["needs"], ["verify", "version"])
|
||||
self.assertEqual(value["jobs"]["channels"]["concurrency"]["group"], "rustfs-release-channels")
|
||||
for job in ("version", "channels"):
|
||||
self.assertFalse(value["jobs"][job]["concurrency"]["cancel-in-progress"])
|
||||
for name in ("build.yml", "package.yml", "docker.yml", "helm-package.yml"):
|
||||
parsed = workflow(name)
|
||||
self.assertEqual(parsed["permissions"]["actions"], "read")
|
||||
self.assertNotIn("push", parsed["on"])
|
||||
self.assertNotIn("workflow_run", parsed["on"])
|
||||
self.assertNotIn("contents: write", (ROOT / ".github/workflows" / name).read_text())
|
||||
source = (ROOT / "scripts/release/publish.py").read_text()
|
||||
for forbidden in ("cargo build", "build-push", "helm package", "--clobber", "force=true"):
|
||||
self.assertNotIn(forbidden, source.replace("# No --clobber:", "# No replacement:"))
|
||||
|
||||
def test_stage_failures_do_not_have_a_continue_on_error_bypass(self):
|
||||
for name in ("release-candidate.yml", "release-complete.yml", "release-publish.yml", "docker.yml", "package.yml", "helm-package.yml"):
|
||||
value = workflow(name)
|
||||
for job in value["jobs"].values():
|
||||
self.assertNotIn("continue-on-error", job)
|
||||
for step in job.get("steps", []):
|
||||
self.assertNotIn("continue-on-error", step)
|
||||
|
||||
|
||||
class ImageWorkflowTests(unittest.TestCase):
|
||||
def test_substituting_a_single_platform_scan_archive_is_rejected(self):
|
||||
import deliver
|
||||
with tempfile.TemporaryDirectory() as temp:
|
||||
output = Path(temp) / "delivery"
|
||||
scans = Path(str(output) + "-scan")
|
||||
output.mkdir(); scans.mkdir()
|
||||
value = inputs()
|
||||
record = {"inputs_sha256": c.digest(value), "scan_inputs": {}}
|
||||
for arch in ("amd64", "arm64"):
|
||||
path = scans / f"image-musl-{arch}.tar"
|
||||
path.write_bytes(arch.encode())
|
||||
record["scan_inputs"][arch] = c.file_digest(path)
|
||||
c.write_json(output / f"scan-musl-{arch}.sarif", {"version": "2.1.0", "runs": [{}]})
|
||||
c.write_json(output / "image-musl-smoke.json", record)
|
||||
deliver.image_record(value, "musl", output)
|
||||
self.assertEqual(c.read_json(output / "image-musl.json")["scan"], "completed")
|
||||
(scans / "image-musl-arm64.tar").write_bytes(b"another architecture")
|
||||
with self.assertRaisesRegex(ValueError, "scan input changed"):
|
||||
deliver.image_record(value, "musl", output)
|
||||
|
||||
def test_both_architectures_scan_before_seal_without_registry_credentials(self):
|
||||
value = workflow("docker.yml")
|
||||
self.assertEqual(value["permissions"], {"contents": "read", "actions": "read"})
|
||||
steps = value["jobs"]["build-docker"]["steps"]
|
||||
build = next(s for s in steps if "docker/build-push-action@" in s.get("uses", ""))
|
||||
self.assertFalse(build["with"]["push"])
|
||||
self.assertIn("type=oci", build["with"]["outputs"])
|
||||
scans = [s for s in steps if "aquasecurity/trivy-action@" in s.get("uses", "")]
|
||||
self.assertEqual({s["env"]["TRIVY_PLATFORM"] for s in scans}, {"linux/amd64", "linux/arm64"})
|
||||
for scan in scans:
|
||||
self.assertEqual(scan["with"]["exit-code"], "0")
|
||||
self.assertIn("input", scan["with"])
|
||||
architecture = scan["env"]["TRIVY_PLATFORM"].split("/")[-1]
|
||||
self.assertTrue(scan["with"]["input"].endswith(f"-{architecture}.tar"))
|
||||
self.assertLess(steps.index(scan), next(i for i, s in enumerate(steps) if s.get("name") == "Record verified OCI identity"))
|
||||
|
||||
|
||||
class HelmWorkflowTests(unittest.TestCase):
|
||||
def test_installation_versions_match_candidate_chart_versions(self):
|
||||
subprocess.run(["bash", "scripts/test_helm_chart_version.sh"], cwd=ROOT,
|
||||
check=True, capture_output=True, text=True, timeout=15)
|
||||
|
||||
def test_chart_is_sealed_before_publish_and_does_not_push(self):
|
||||
value = workflow("helm-package.yml")
|
||||
self.assertEqual(set(value["on"]), {"workflow_call"})
|
||||
source = (ROOT / ".github/workflows/helm-package.yml").read_text()
|
||||
self.assertIn("scripts/test_helm_templates.sh", source)
|
||||
self.assertIn("('version', 'appVersion')", source)
|
||||
self.assertNotIn("git push", source)
|
||||
self.assertIn("inputs_sha256", source)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -111,7 +111,22 @@ class GateFixtureTests(unittest.TestCase):
|
||||
self.assertEqual(summary["findings"], [])
|
||||
expansion = json.loads((self.output / "pass" / "plan-expansion.json").read_text())
|
||||
self.assertEqual(len(expansion["required_cases"]), 5)
|
||||
promote = run_promote(self.output / "pass")
|
||||
import hashlib
|
||||
artifact = self.output / ARTIFACT_NAME
|
||||
artifact.write_bytes(b"release gate contract payload\n")
|
||||
candidate = json.loads(CANDIDATE.read_text())
|
||||
old_sha = candidate["deliverables"][0]["sha256"]
|
||||
new_sha = hashlib.sha256(artifact.read_bytes()).hexdigest()
|
||||
candidate["deliverables"][0].update(sha256=new_sha, size=artifact.stat().st_size)
|
||||
candidate_path = self.output / "candidate.json"
|
||||
candidate_path.write_text(json.dumps(candidate))
|
||||
results = self.output / "actual-results"
|
||||
results.mkdir()
|
||||
for path in (FIXTURES / "pass").glob("*.json"):
|
||||
(results / path.name).write_text(path.read_text().replace(old_sha, new_sha))
|
||||
result = run_validate(results, self.output / "actual-certification", candidate=candidate_path)
|
||||
self.assertEqual(result.returncode, 0, result.stderr)
|
||||
promote = run_promote(self.output / "actual-certification", candidate=candidate_path, artifact=artifact)
|
||||
self.assertEqual(promote.returncode, 0, promote.stderr)
|
||||
|
||||
def test_cross_platform_same_case_and_split_shas_accepted(self):
|
||||
@@ -306,70 +321,23 @@ class PlanContractTests(unittest.TestCase):
|
||||
|
||||
|
||||
class PublicationWiringTests(unittest.TestCase):
|
||||
"""Static checks that the real publication exits consume the gate."""
|
||||
"""The publisher re-runs full certification over trusted downloaded receipts."""
|
||||
|
||||
def read(self, relative):
|
||||
return (ROOT / relative).read_text()
|
||||
def test_single_publisher_revalidates_the_complete_seal(self):
|
||||
source = (ROOT / "scripts/release/publish.py").read_text()
|
||||
self.assertIn('pipeline.certify(inputs, directory,', source)
|
||||
self.assertIn('"full")', source)
|
||||
self.assertIn('summary["publish_allowed"]', source)
|
||||
self.assertIn('pipeline.verify_deliveries(inputs, directory)', source)
|
||||
workflow = (ROOT / ".github/workflows/release-publish.yml").read_text()
|
||||
self.assertIn('environment: release-publication', workflow)
|
||||
self.assertIn('needs: [verify, version]', workflow)
|
||||
|
||||
def job_block(self, workflow, job):
|
||||
match = re.search(
|
||||
rf"^ {re.escape(job)}:\n(.*?)(?=^ [A-Za-z#])",
|
||||
workflow,
|
||||
re.S | re.M,
|
||||
)
|
||||
self.assertIsNotNone(match, f"job {job} not found")
|
||||
return match.group(1)
|
||||
|
||||
def test_build_publish_requires_certification(self):
|
||||
workflow = self.read(".github/workflows/build.yml")
|
||||
self.assertIn("\n release-certification:", workflow)
|
||||
publish_block = self.job_block(workflow, "publish-release")
|
||||
self.assertIn("release-certification", publish_block)
|
||||
self.assertIn("release_gate.py validate", workflow)
|
||||
self.assertIn("release_gate.py verify-promotion", workflow)
|
||||
|
||||
def test_build_time_official_r2_writes_are_staged(self):
|
||||
workflow = self.read(".github/workflows/build.yml")
|
||||
self.assertIn("artifacts/rustfs/staging/", workflow)
|
||||
release_writes = [
|
||||
line for line in workflow.splitlines()
|
||||
if "artifacts/rustfs/release/" in line
|
||||
]
|
||||
# Only the certification-gated promotion job may write the official path.
|
||||
self.assertTrue(release_writes, "the promotion job must write the official R2 path")
|
||||
promotion_block = self.job_block(workflow, "promote-release-artifacts")
|
||||
self.assertIn("verify-promotion", promotion_block)
|
||||
self.assertIn("artifacts/rustfs/release/", promotion_block)
|
||||
build_block = self.job_block(workflow, "build-rustfs")
|
||||
self.assertNotIn('R2_PATH="s3://${R2_BUCKET}/artifacts/rustfs/release/"', build_block)
|
||||
|
||||
def test_downstream_workflows_verify_certification(self):
|
||||
docker = self.read(".github/workflows/docker.yml")
|
||||
self.assertIn("\n release-certification:", docker)
|
||||
build_docker_block = self.job_block(docker, "build-docker")
|
||||
self.assertIn("release-certification", build_docker_block)
|
||||
self.assertIn("verify-promotion", docker)
|
||||
package = self.read(".github/workflows/package.yml")
|
||||
self.assertIn("\n release-certification:", package)
|
||||
package_block = self.job_block(package, "package")
|
||||
self.assertIn("release-certification", package_block)
|
||||
helm = self.read(".github/workflows/helm-package.yml")
|
||||
self.assertIn("\n release-certification:", helm)
|
||||
helm_block = self.job_block(helm, "build-helm-package")
|
||||
self.assertIn("release-certification", helm_block)
|
||||
|
||||
def test_manual_downstream_dispatch_without_certification_fails(self):
|
||||
# Every downstream certification job must fail closed when the trusted
|
||||
# certification artifact is absent (e.g. a manual dispatch without one).
|
||||
for relative in (
|
||||
".github/workflows/docker.yml",
|
||||
".github/workflows/package.yml",
|
||||
".github/workflows/helm-package.yml",
|
||||
):
|
||||
workflow = self.read(relative)
|
||||
certification_block = self.job_block(workflow, "release-certification")
|
||||
self.assertIn("fail", certification_block.lower(), relative)
|
||||
self.assertIn("release-certification", certification_block, relative)
|
||||
def test_builders_do_not_have_publication_permissions(self):
|
||||
for name in ("build.yml", "docker.yml", "package.yml", "helm-package.yml"):
|
||||
source = (ROOT / ".github/workflows" / name).read_text()
|
||||
self.assertNotIn("contents: write", source)
|
||||
self.assertNotIn("packages: write", source)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
||||
@@ -0,0 +1,230 @@
|
||||
#!/usr/bin/env python3
|
||||
# Copyright 2026 RustFS Team
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
"""Run release checks against the archived executable, never a rebuilt server."""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
from datetime import datetime, timezone
|
||||
import json
|
||||
import hashlib
|
||||
import hmac
|
||||
import os
|
||||
from pathlib import Path
|
||||
import re
|
||||
import socket
|
||||
import subprocess
|
||||
import tempfile
|
||||
import time
|
||||
import urllib.request
|
||||
import zipfile
|
||||
|
||||
import candidate as c
|
||||
|
||||
TRIPLES = {
|
||||
"linux-x86_64-musl": "x86_64-unknown-linux-musl", "linux-aarch64-musl": "aarch64-unknown-linux-musl",
|
||||
"linux-x86_64-gnu": "x86_64-unknown-linux-gnu", "linux-aarch64-gnu": "aarch64-unknown-linux-gnu",
|
||||
"macos-aarch64": "aarch64-apple-darwin", "windows-x86_64": "x86_64-pc-windows-msvc",
|
||||
}
|
||||
UPGRADES = {
|
||||
"direct-upgrade": "direct_upgrade_from_previous_release_preserves_object_contracts",
|
||||
"mixed-version-upgrade": "rolling_upgrade_from_previous_release_preserves_mixed_version_contracts",
|
||||
"bucket-config-upgrade": "direct_upgrade_from_previous_release_preserves_bucket_configuration",
|
||||
"bucket-config-rollback": "rollback_to_previous_release_reads_current_bucket_metadata",
|
||||
"multipart-layout-upgrade": "direct_upgrade_from_previous_release_preserves_multipart_layouts",
|
||||
"multipart-layout-baseline": "previous_release_baseline_replicates_multipart_layouts",
|
||||
}
|
||||
|
||||
|
||||
def now():
|
||||
return datetime.now(timezone.utc).isoformat().replace("+00:00", "Z")
|
||||
|
||||
|
||||
def envelope(inputs, record, lane, case, logs, started, output):
|
||||
"""Only called after the real check succeeds; logs remain part of the seal."""
|
||||
target = record["target"].split("-")
|
||||
platform = dict(zip(("os", "arch", "abi"), target))
|
||||
platform.setdefault("abi", "msvc" if platform["os"] == "windows" else "none")
|
||||
workflow = os.environ["GITHUB_WORKFLOW_REF"].split("@", 1)[0].rsplit("/", 1)[-1]
|
||||
c.require(workflow in ("release-candidate.yml", "release-complete.yml"), "untrusted verifier workflow")
|
||||
value = {
|
||||
"schema_version": 1, "evidence_profile": "release-candidate", "tier": "release",
|
||||
"run_id": inputs["candidate_id"], "run_attempt": 1, "plan_sha256": inputs["plan_sha256"],
|
||||
"lane": lane, "case_id": case, "product_commit": inputs["source_sha"],
|
||||
"harness_commit": inputs["workflow_sha"], "target_fingerprint": record["binary_sha256"],
|
||||
"platform": platform, "started_at": started, "finished_at": now(), "state": "pass",
|
||||
"reason_code": "CHECK_PASSED", "reason_class": "NOT_APPLICABLE",
|
||||
"artifact": {"archive_sha256": record["sha256"], "binary_sha256": record["binary_sha256"]},
|
||||
"evidence": [{"path": p.name, "sha256": c.file_digest(p), "size": p.stat().st_size} for p in logs],
|
||||
"workflow": {"repository": c.REPOSITORY, "workflow_id": workflow, "run_id": os.environ["GITHUB_RUN_ID"],
|
||||
"run_attempt": int(os.environ["GITHUB_RUN_ATTEMPT"]), "workflow_sha": os.environ["GITHUB_SHA"],
|
||||
"status": "completed"},
|
||||
}
|
||||
c.write_json(output / f"{lane}-{record['target']}-{case}.json", value)
|
||||
|
||||
|
||||
def unpack(archive, target, inputs, directory):
|
||||
record = c.archive_record(archive, target, inputs)
|
||||
name = "rustfs.exe" if target.startswith("windows") else "rustfs"
|
||||
with zipfile.ZipFile(archive) as package:
|
||||
package.extract(name, directory)
|
||||
binary = (Path(directory) / name).resolve()
|
||||
binary.chmod(0o755)
|
||||
return binary, record
|
||||
|
||||
|
||||
def port():
|
||||
with socket.socket() as sock:
|
||||
sock.bind(("127.0.0.1", 0))
|
||||
return sock.getsockname()[1]
|
||||
|
||||
|
||||
def s3_request(opener, endpoint, method, path, access, secret, body=b""):
|
||||
timestamp = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%SZ")
|
||||
date = timestamp[:8]
|
||||
host = endpoint.removeprefix("http://")
|
||||
payload = hashlib.sha256(body).hexdigest()
|
||||
headers = f"host:{host}\nx-amz-content-sha256:{payload}\nx-amz-date:{timestamp}\n"
|
||||
signed = "host;x-amz-content-sha256;x-amz-date"
|
||||
canonical = f"{method}\n{path}\n\n{headers}\n{signed}\n{payload}"
|
||||
scope = f"{date}/us-east-1/s3/aws4_request"
|
||||
to_sign = f"AWS4-HMAC-SHA256\n{timestamp}\n{scope}\n{hashlib.sha256(canonical.encode()).hexdigest()}"
|
||||
key = ("AWS4" + secret).encode()
|
||||
for part in (date, "us-east-1", "s3", "aws4_request"):
|
||||
key = hmac.digest(key, part.encode(), "sha256")
|
||||
signature = hmac.new(key, to_sign.encode(), hashlib.sha256).hexdigest()
|
||||
request = urllib.request.Request(endpoint + path, data=body if method == "PUT" else None, method=method,
|
||||
headers={"Host": host, "X-Amz-Content-Sha256": payload, "X-Amz-Date": timestamp,
|
||||
"Authorization": f"AWS4-HMAC-SHA256 Credential={access}/{scope}, SignedHeaders={signed}, Signature={signature}"})
|
||||
with opener.open(request, timeout=10) as response:
|
||||
c.require(response.status in (200, 204), "S3 smoke request failed")
|
||||
return response.read(1024 * 1024)
|
||||
|
||||
|
||||
def native(inputs, archive, target, output):
|
||||
started = now()
|
||||
output.mkdir(parents=True, exist_ok=True)
|
||||
with tempfile.TemporaryDirectory() as temp:
|
||||
binary, record = unpack(archive, target, inputs, temp)
|
||||
version = subprocess.check_output([str(binary), "--version"], text=True, timeout=60)
|
||||
c.require(version.splitlines()[0] == f"rustfs {inputs['version']}", "binary displays a different product version")
|
||||
for label, expected in (("build target", TRIPLES[target]), ("git commit", inputs["source_sha"])):
|
||||
c.require(re.search(rf"(?m)^{label}\s*:\s*{re.escape(expected)}\s*$", version), f"binary {label} mismatch")
|
||||
c.require(re.search(rf"(?m)^rust version\s*:\s*(?:rustc )?{re.escape(inputs['toolchain'])}(?:\s|$)", version), "binary toolchain mismatch")
|
||||
log = output / f"identity-{target}.log"
|
||||
log.write_text(version, encoding="utf-8")
|
||||
envelope(inputs, record, "identity", "artifact-identity", [log], started, output)
|
||||
data = Path(temp) / "data"
|
||||
data.mkdir()
|
||||
api, console = port(), port()
|
||||
while console == api:
|
||||
console = port()
|
||||
log = output / f"startup-{target}.log"
|
||||
access, secret = "release-" + os.urandom(8).hex(), os.urandom(24).hex()
|
||||
env = dict(os.environ, NO_PROXY="127.0.0.1,localhost,::1", no_proxy="127.0.0.1,localhost,::1")
|
||||
opener = urllib.request.build_opener(urllib.request.ProxyHandler({}))
|
||||
with log.open("w", encoding="utf-8") as stream:
|
||||
process = subprocess.Popen([str(binary), "server", "--address", f"127.0.0.1:{api}", "--console-enable",
|
||||
"--console-address", f"127.0.0.1:{console}", "--access-key", access,
|
||||
"--secret-key", secret, str(data)],
|
||||
stdout=stream, stderr=subprocess.STDOUT, env=env)
|
||||
try:
|
||||
deadline = time.monotonic() + 60
|
||||
while time.monotonic() < deadline:
|
||||
c.require(process.poll() is None, "candidate exited before becoming ready")
|
||||
try:
|
||||
with opener.open(f"http://127.0.0.1:{api}/health/ready", timeout=2) as response:
|
||||
c.require(response.status == 200, "S3 listener is not ready")
|
||||
with opener.open(f"http://127.0.0.1:{console}/rustfs/console/", timeout=2) as response:
|
||||
c.require(response.status == 200 and "text/html" in response.headers.get("Content-Type", ""), "Console did not return HTML")
|
||||
break
|
||||
except OSError:
|
||||
time.sleep(0.5)
|
||||
else:
|
||||
raise ValueError("candidate did not become ready within 60 seconds")
|
||||
bucket = "/release-" + os.urandom(8).hex()
|
||||
endpoint = f"http://127.0.0.1:{api}"
|
||||
content = os.urandom(1024)
|
||||
s3_request(opener, endpoint, "PUT", bucket, access, secret)
|
||||
s3_request(opener, endpoint, "PUT", bucket + "/roundtrip", access, secret, content)
|
||||
c.require(s3_request(opener, endpoint, "GET", bucket + "/roundtrip", access, secret) == content, "S3 round-trip bytes differ")
|
||||
s3_request(opener, endpoint, "DELETE", bucket + "/roundtrip", access, secret)
|
||||
s3_request(opener, endpoint, "DELETE", bucket, access, secret)
|
||||
c.require(process.poll() is None, "candidate exited during startup verification")
|
||||
finally:
|
||||
process.terminate()
|
||||
try:
|
||||
process.wait(timeout=15)
|
||||
except subprocess.TimeoutExpired:
|
||||
process.kill()
|
||||
process.wait(timeout=15)
|
||||
c.require(c.file_digest(binary) == record["binary_sha256"], "server bytes changed during native verification")
|
||||
with log.open("a", encoding="utf-8") as stream:
|
||||
stream.write("\nConsole: HTTP 200 text/html; S3: ready, bucket/object CRUD and byte-identical round trip\n")
|
||||
envelope(inputs, record, "identity", "startup", [log], started, output)
|
||||
|
||||
|
||||
def upgrade(inputs, archive, output):
|
||||
output.mkdir(parents=True, exist_ok=True)
|
||||
plan = c.read_json(c.ROOT / ".config/release-validation-plan.json")
|
||||
c.require(c.digest(plan) == inputs["plan_sha256"], "upgrade plan changed")
|
||||
with tempfile.TemporaryDirectory() as temp:
|
||||
root = Path(temp)
|
||||
binary, record = unpack(archive, "linux-x86_64-gnu", inputs, root)
|
||||
metadata = binary.stat()
|
||||
receipt = root / "receipt.json"
|
||||
receipt.write_text(json.dumps({"schema": 1, "workspace": str(c.ROOT.resolve()), "binary": str(binary),
|
||||
"size": metadata.st_size, "modified_ns": metadata.st_mtime_ns, "features": ["pyroscope"]}))
|
||||
env = dict(os.environ, CARGO_BIN_EXE_rustfs=str(binary), RUSTFS_E2E_BINARY_RECEIPT=str(receipt),
|
||||
RUSTFS_BUILD_FEATURES="pyroscope", NO_PROXY="127.0.0.1,localhost,::1", no_proxy="127.0.0.1,localhost,::1",
|
||||
HTTP_PROXY="", HTTPS_PROXY="", ALL_PROXY="", http_proxy="", https_proxy="", all_proxy="")
|
||||
logs = {case: [] for case in UPGRADES}
|
||||
started = now()
|
||||
sources = list(plan["upgrade_sources"]["sources"])
|
||||
previous = inputs["previous"]
|
||||
if previous["version"] not in {s["version"] for s in sources}:
|
||||
sources.append({"version": previous["version"], "artifact_name": previous["name"], "artifact_sha256": previous["sha256"]})
|
||||
for source in sources:
|
||||
directory = root / source["version"]
|
||||
directory.mkdir()
|
||||
package = directory / source["artifact_name"]
|
||||
url = f"https://github.com/{c.REPOSITORY}/releases/download/{source['version']}/{source['artifact_name']}"
|
||||
subprocess.run(["curl", "--fail", "--location", "--retry", "3", "--output", str(package), url], check=True, timeout=600)
|
||||
c.require(c.file_digest(package) == source["artifact_sha256"], "previous release digest mismatch")
|
||||
with zipfile.ZipFile(package) as previous:
|
||||
c.require(previous.namelist().count("rustfs") == 1, "ambiguous previous server")
|
||||
previous.extract("rustfs", directory)
|
||||
(directory / "rustfs").chmod(0o755)
|
||||
env["RUSTFS_UPGRADE_SOURCE_BINARY"] = str(directory / "rustfs")
|
||||
for case, test in UPGRADES.items():
|
||||
log = output / f"upgrade-{source['version']}-{case}.log"
|
||||
env["RUSTFS_E2E_LOG_DIR"] = str(root / f"logs-{source['version']}-{case}")
|
||||
with log.open("w", encoding="utf-8") as stream:
|
||||
subprocess.run(["cargo", "test", "--locked", "-p", "e2e_test", f"upgrade_compatibility_test::{test}",
|
||||
"--", "--ignored", "--exact", "--nocapture"], cwd=c.ROOT, env=env,
|
||||
stdout=stream, stderr=subprocess.STDOUT, check=True, timeout=1800)
|
||||
c.require(re.search(r"test result: ok\. 1 passed; 0 failed;", log.read_text()), "upgrade command did not run exactly one passing test")
|
||||
c.require(c.file_digest(binary) == record["binary_sha256"], "candidate changed during upgrade tests")
|
||||
logs[case].append(log)
|
||||
for case, files in logs.items():
|
||||
envelope(inputs, record, "upgrade", case, files, started, output)
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("mode", choices=("native", "upgrade"))
|
||||
parser.add_argument("--inputs", required=True)
|
||||
parser.add_argument("--archive", required=True)
|
||||
parser.add_argument("--target", choices=c.TARGETS)
|
||||
parser.add_argument("--output", type=Path, required=True)
|
||||
args = parser.parse_args()
|
||||
inputs = c.validate_inputs(c.read_json(args.inputs))
|
||||
if args.mode == "native":
|
||||
c.require(args.target is not None, "native verification needs a platform")
|
||||
native(inputs, args.archive, args.target, args.output)
|
||||
else:
|
||||
upgrade(inputs, args.archive, args.output)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,27 +1,4 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
workflow=".github/workflows/docker.yml"
|
||||
|
||||
require_literal() {
|
||||
local needle="$1"
|
||||
local description="$2"
|
||||
|
||||
if ! grep -Fq "$needle" "$workflow"; then
|
||||
echo "missing container scan workflow contract: $description" >&2
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
require_literal "scan-docker-image:" "scan job"
|
||||
require_literal "needs: [ build-check, build-docker ]" "build dependency"
|
||||
require_literal "needs.build-check.outputs.should_build == 'true' && needs.build-check.outputs.should_push == 'true'" "release image push guard"
|
||||
require_literal "docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9" "pinned GHCR login action"
|
||||
require_literal "aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25" "pinned Trivy action"
|
||||
require_literal 'image-ref: ${{ env.REGISTRY_GHCR }}:${{ needs.build-check.outputs.version }}${{ matrix.suffix }}' "GHCR image reference"
|
||||
require_literal "format: sarif" "SARIF report format"
|
||||
require_literal 'exit-code: "0"' "report-only failure policy"
|
||||
require_literal "actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a" "pinned report upload action"
|
||||
require_literal "container-image-scan-" "scan report artifact name"
|
||||
|
||||
echo "Container scan workflow contract ok."
|
||||
cd "$(dirname "${BASH_SOURCE[0]}")/../.."
|
||||
python3 scripts/release/test_pipeline.py ImageWorkflowTests
|
||||
|
||||
@@ -1,420 +1,4 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
build_workflow=".github/workflows/build.yml"
|
||||
docker_workflow=".github/workflows/docker.yml"
|
||||
helm_workflow=".github/workflows/helm-package.yml"
|
||||
package_workflow=".github/workflows/package.yml"
|
||||
release_script="scripts/release/create_or_update_release.sh"
|
||||
|
||||
# shellcheck source=scripts/release/create_or_update_release.sh
|
||||
source "$release_script"
|
||||
|
||||
require_line() {
|
||||
local file="$1"
|
||||
local line="$2"
|
||||
local description="$3"
|
||||
|
||||
if ! grep -Fxq "$line" "$file"; then
|
||||
echo "missing preview release workflow contract: $description" >&2
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
require_absent() {
|
||||
local file="$1"
|
||||
local text="$2"
|
||||
local description="$3"
|
||||
|
||||
if grep -Fq -- "$text" "$file"; then
|
||||
echo "invalid preview release workflow contract: $description" >&2
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
require_line "$build_workflow" " cargo zigbuild --release --locked --target \"\$TARGET_ARG\" \"\${FEATURE_ARGS[@]}\" \"\${PROFILE_ARGS[@]}\" -p rustfs --bin rustfs" "cross builds must keep matrix features"
|
||||
require_line "$build_workflow" " cargo build --release --locked --target \"\$TARGET_ARG\" \"\${FEATURE_ARGS[@]}\" \"\${PROFILE_ARGS[@]}\" -p rustfs --bin rustfs" "native builds must keep matrix features"
|
||||
require_absent "$build_workflow" " cargo zigbuild --release --locked --target \"\$TARGET_ARG\" -p rustfs --bin rustfs" "cross builds must not drop matrix features"
|
||||
require_absent "$build_workflow" " cargo build --release --locked --target \"\$TARGET_ARG\" -p rustfs --bin rustfs" "native builds must not drop matrix features"
|
||||
|
||||
extract_job_if() {
|
||||
local file="$1"
|
||||
local job="$2"
|
||||
|
||||
awk -v job="$job" '
|
||||
$0 == " " job ":" { in_job = 1 }
|
||||
in_job && /^ if:/ { in_if = 1 }
|
||||
in_if && /^ [A-Za-z0-9_-]+:/ && $0 !~ /^ if:/ { exit }
|
||||
in_if { print }
|
||||
in_job && $0 ~ /^ [A-Za-z0-9_-]+:$/ && $0 != " " job ":" { exit }
|
||||
' "$file"
|
||||
}
|
||||
|
||||
require_job_if() {
|
||||
local file="$1"
|
||||
local job="$2"
|
||||
local expected="$3"
|
||||
local actual
|
||||
|
||||
actual=$(extract_job_if "$file" "$job")
|
||||
if [[ "$actual" != "$expected" ]]; then
|
||||
echo "missing preview release workflow contract: $job condition" >&2
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
require_assignment() {
|
||||
local block="$1"
|
||||
local name="$2"
|
||||
local expected="$3"
|
||||
local count
|
||||
|
||||
count=$(printf '%s\n' "$block" | grep -Ec "^[[:space:]]+${name}=")
|
||||
if [[ "$count" -ne 1 || "$block" != *"${name}=${expected}"* ]]; then
|
||||
echo "missing preview release workflow contract: $name=$expected" >&2
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
require_no_assignment() {
|
||||
local block="$1"
|
||||
local name="$2"
|
||||
|
||||
if printf '%s\n' "$block" | grep -Eq "^[[:space:]]+${name}="; then
|
||||
echo "invalid preview release workflow contract: unexpected $name override" >&2
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
tag_classification=$(awk '
|
||||
/if \[\[ "\$tag_name" =~ -preview\\\.\[0-9\]\+\$ \]\]; then/ { in_preview = 1 }
|
||||
in_preview { print }
|
||||
in_preview && $0 == " fi" { exit }
|
||||
' "$build_workflow")
|
||||
invalid_preview_condition="elif [[ \"\$tag_name\" == *\"-preview\"* ]]; then"
|
||||
prerelease_condition="elif [[ \"\$tag_name\" == *\"alpha\"* ]] || [[ \"\$tag_name\" == *\"beta\"* ]] || [[ \"\$tag_name\" == *\"rc\"* ]]; then"
|
||||
preview_branch=$(printf '%s\n' "$tag_classification" | awk -v boundary="$invalid_preview_condition" 'index($0, boundary) { exit } { print }')
|
||||
invalid_preview_branch=$(printf '%s\n' "$tag_classification" | awk -v start="$invalid_preview_condition" -v boundary="$prerelease_condition" 'index($0, start) { in_branch = 1 } in_branch && index($0, boundary) { exit } in_branch { print }')
|
||||
prerelease_branch=$(printf '%s\n' "$tag_classification" | awk -v start="$prerelease_condition" 'index($0, start) { in_branch = 1 } in_branch && $0 == " else" { exit } in_branch { print }')
|
||||
release_branch=$(printf '%s\n' "$tag_classification" | awk '$0 == " else" { in_branch = 1; next } in_branch && $0 == " fi" { exit } in_branch { print }')
|
||||
|
||||
if [[ -z "$preview_branch" || -z "$invalid_preview_branch" || -z "$prerelease_branch" || -z "$release_branch" ]]; then
|
||||
echo "missing preview release workflow contract: ordered preview, invalid-preview, prerelease, and release branches" >&2
|
||||
exit 1
|
||||
fi
|
||||
require_assignment "$preview_branch" "build_type" '"preview"'
|
||||
require_assignment "$preview_branch" "is_prerelease" "true"
|
||||
require_no_assignment "$invalid_preview_branch" "build_type"
|
||||
if [[ "$invalid_preview_branch" != *"exit 1"* ]]; then
|
||||
echo "missing preview release workflow contract: invalid preview tags must fail closed" >&2
|
||||
exit 1
|
||||
fi
|
||||
require_assignment "$prerelease_branch" "build_type" '"prerelease"'
|
||||
require_assignment "$prerelease_branch" "is_prerelease" "true"
|
||||
require_assignment "$release_branch" "build_type" '"release"'
|
||||
require_no_assignment "$release_branch" "is_prerelease"
|
||||
|
||||
initial_prerelease_line=$(grep -Fn " is_prerelease=false" "$build_workflow")
|
||||
preview_condition_line=$(grep -Fn " if [[ \"\$tag_name\" =~ -preview\\.[0-9]+\$ ]]; then" "$build_workflow")
|
||||
if [[ $(printf '%s\n' "$initial_prerelease_line" | wc -l) -ne 1 || "${initial_prerelease_line%%:*}" -ge "${preview_condition_line%%:*}" ]]; then
|
||||
echo "missing preview release workflow contract: is_prerelease=false must initialize before tag classification" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
tag_branch_tail=$(awk '
|
||||
/if \[\[ "\$tag_name" =~ -preview\\\.\[0-9\]\+\$ \]\]; then/ { in_classification = 1 }
|
||||
in_classification && $0 == " fi" { after_classification = 1; next }
|
||||
after_classification && /^ elif / { exit }
|
||||
after_classification { print }
|
||||
' "$build_workflow")
|
||||
post_strategy=$(awk '
|
||||
/# Determine build type based on trigger/ { in_strategy = 1 }
|
||||
in_strategy && $0 == " fi" { after_strategy = 1; next }
|
||||
after_strategy && $0 == " {" { exit }
|
||||
after_strategy { print }
|
||||
' "$build_workflow")
|
||||
for block in "$tag_branch_tail" "$post_strategy"; do
|
||||
require_no_assignment "$block" "build_type"
|
||||
require_no_assignment "$block" "is_prerelease"
|
||||
done
|
||||
require_line "$build_workflow" " if [[ \"\$BUILD_TYPE\" == \"release\" ]] || [[ \"\$BUILD_TYPE\" == \"prerelease\" ]]; then" "latest artifact guard"
|
||||
require_line "$build_workflow" " if: needs.build-check.outputs.should_publish == 'true' && env.R2_ACCESS_KEY_ID != '' && (needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease' || needs.build-check.outputs.build_type == 'development')" "R2 publication guard"
|
||||
require_line "$package_workflow" " if [[ \"\$TAG\" =~ ^\${semver_core}(-\${prerelease_id})?-preview\\.(0|[1-9][0-9]*)$ ]]; then" "stable and prerelease preview package classification"
|
||||
require_line "$package_workflow" " if: needs.resolve.outputs.build_type != 'preview' && env.R2_ACCESS_KEY_ID != ''" "preview packages must not publish to R2"
|
||||
release_guard="needs.build-check.outputs.should_publish == 'true' && startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'preview' || needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')"
|
||||
for job in create-release upload-release-assets publish-release; do
|
||||
require_job_if "$build_workflow" "$job" " if: $release_guard"
|
||||
done
|
||||
latest_guard="needs.build-check.outputs.should_publish == 'true' && startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')"
|
||||
require_job_if "$build_workflow" "update-latest-version" " if: $latest_guard"
|
||||
require_line "$build_workflow" " needs: [ build-check, publish-release ]" "latest update must follow release publication"
|
||||
|
||||
# Preview releases are internal validation artifacts: once the deliverable
|
||||
# release is published they are deleted, while their tags stay behind.
|
||||
require_job_if "$build_workflow" "cleanup-preview-releases" " if: $latest_guard"
|
||||
require_line "$build_workflow" " gh release delete \"\$preview_tag\" --repo \"\${GITHUB_REPOSITORY}\" --yes" "preview release cleanup after publication"
|
||||
require_line "$build_workflow" " | select(.tag_name | startswith(\$tag + \"-preview.\"))" "cleanup must match the target's own preview tags"
|
||||
require_line "$build_workflow" " | select(.tag_name | ltrimstr(\$tag + \"-preview.\") | test(\"^[0-9]+\$\"))" "cleanup must match a numeric preview iteration"
|
||||
require_absent "$build_workflow" "--cleanup-tag" "preview tags must survive their release cleanup"
|
||||
require_line "$build_workflow" " TARGET_COMMITISH=\$(git rev-parse --verify \"refs/tags/\${TAG}^{commit}\")" "release target commit resolution"
|
||||
require_line "$build_workflow" " ./scripts/release/create_or_update_release.sh \\" "managed release creation"
|
||||
require_absent "$build_workflow" "git tag -l --format='%(contents)'" "annotated tag messages must not become release notes"
|
||||
require_absent "$build_workflow" "--generate-notes" "GitHub must not choose the release notes baseline implicitly"
|
||||
require_line "$release_script" " gh api --method POST \"repos/\${GITHUB_REPOSITORY}/releases/generate-notes\" \\" "Generate Release Notes API"
|
||||
require_line "$release_script" " local create_args=(release create \"\$tag\" --title \"\$title\" --notes-file \"\$notes_file\" --latest=false --draft)" "draft release creation with a notes file"
|
||||
|
||||
release_channel_block=$(awk '
|
||||
$0 == " if [[ \"$BUILD_TYPE\" == \"release\" ]]; then" { in_block = 1 }
|
||||
in_block { print }
|
||||
in_block && $0 == " fi" { exit }
|
||||
' "$build_workflow")
|
||||
IFS= read -r -d '' expected_release_channel_block <<'EOF' || true
|
||||
if [[ "$BUILD_TYPE" == "release" ]]; then
|
||||
gh api --method PATCH "repos/${GITHUB_REPOSITORY}/releases/${RELEASE_ID}" \
|
||||
-F draft=false \
|
||||
-F prerelease=false \
|
||||
-f make_latest=true >/dev/null
|
||||
else
|
||||
gh api --method PATCH "repos/${GITHUB_REPOSITORY}/releases/${RELEASE_ID}" \
|
||||
-F draft=false \
|
||||
-F prerelease=true \
|
||||
-f make_latest=false >/dev/null
|
||||
fi
|
||||
EOF
|
||||
expected_release_channel_block=${expected_release_channel_block%$'\n'}
|
||||
if [[ "$release_channel_block" != "$expected_release_channel_block" ]]; then
|
||||
echo "missing preview release workflow contract: only stable releases may become GitHub Latest" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
publish_release_block=$(awk '
|
||||
$0 == " publish-release:" { in_job = 1 }
|
||||
in_job { print }
|
||||
in_job && /^ [A-Za-z0-9_-]+:$/ && $0 != " publish-release:" { exit }
|
||||
' "$build_workflow")
|
||||
if [[ "$publish_release_block" != *" - name: Publish release"* ]]; then
|
||||
echo "missing preview release workflow contract: publish job must only publish the prepared release" >&2
|
||||
exit 1
|
||||
fi
|
||||
if grep -Eq -- '--notes|generate-notes|create_or_update_release' <<<"$publish_release_block"; then
|
||||
echo "invalid preview release workflow contract: publish job must not replace release notes" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
IFS= read -r -d '' expected_docker_automatic_guard <<'EOF' || true
|
||||
if: >-
|
||||
github.event_name == 'workflow_dispatch' ||
|
||||
(github.event.workflow_run.conclusion == 'success' &&
|
||||
github.event.workflow_run.event == 'push' &&
|
||||
github.event.workflow_run.head_branch != 'main')
|
||||
EOF
|
||||
expected_docker_automatic_guard=${expected_docker_automatic_guard%$'\n'}
|
||||
require_job_if "$docker_workflow" "build-check" "$expected_docker_automatic_guard"
|
||||
require_line "$docker_workflow" ' source_ref: ${{ steps.check.outputs.source_ref }}' "Docker source ref output"
|
||||
require_line "$docker_workflow" ' source_ref="$HEAD_SHA"' "automatic Docker source ref"
|
||||
require_line "$docker_workflow" ' source_ref="$tag_ref"' "manual Docker source ref"
|
||||
require_line "$docker_workflow" ' ref: ${{ needs.build-check.outputs.source_ref }}' "Docker release source checkout"
|
||||
require_line "$docker_workflow" ' SOURCE_REVISION="$(git rev-parse HEAD)"' "Docker source revision resolution"
|
||||
require_line "$docker_workflow" ' LABELS="$LABELS,org.opencontainers.image.revision=$SOURCE_REVISION"' "Docker revision label"
|
||||
require_absent "$docker_workflow" 'org.opencontainers.image.revision=${{ github.sha }}' "Docker revision must not use the workflow branch SHA"
|
||||
|
||||
python3 scripts/test_docker_workflow.py
|
||||
|
||||
IFS= read -r -d '' expected_helm_guard <<'EOF' || true
|
||||
if: |
|
||||
(github.event_name == 'workflow_dispatch' && !contains(github.event.inputs.version, '-preview')) ||
|
||||
(
|
||||
github.event.workflow_run.conclusion == 'success' &&
|
||||
github.event.workflow_run.event == 'push' &&
|
||||
contains(github.event.workflow_run.head_branch, '.') &&
|
||||
!contains(github.event.workflow_run.head_branch, '-preview')
|
||||
)
|
||||
EOF
|
||||
expected_helm_guard=${expected_helm_guard%$'\n'}
|
||||
require_job_if "$helm_workflow" "build-helm-package" "$expected_helm_guard"
|
||||
require_line "$helm_workflow" " ref: \${{ github.event.workflow_run.head_sha || (startsWith(inputs.version, 'refs/tags/') && inputs.version || format('refs/tags/{0}', inputs.version)) }}" "Helm release source checkout"
|
||||
python3 scripts/test_helm_release_workflow.py
|
||||
|
||||
assert_equal() {
|
||||
local expected="$1"
|
||||
local actual="$2"
|
||||
local description="$3"
|
||||
|
||||
if [[ "$actual" != "$expected" ]]; then
|
||||
echo "invalid preview release workflow contract: $description (expected '$expected', got '$actual')" >&2
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
IFS= read -r -d '' release_fixture <<'EOF' || true
|
||||
[
|
||||
[
|
||||
{"tag_name":"1.0.0-beta.12-preview.1","created_at":"2026-07-30T02:30:00Z","published_at":"2026-07-30T02:36:43Z","draft":false},
|
||||
{"tag_name":"1.0.0-beta.12","created_at":"2026-07-30T04:00:00Z","published_at":"2026-07-30T04:13:04Z","draft":false},
|
||||
{"tag_name":"1.0.0-beta.12-preview.0","created_at":"2026-07-29T10:00:00Z","published_at":"2026-07-29T10:05:00Z","draft":false}
|
||||
],
|
||||
[
|
||||
{"tag_name":"1.0.0-beta.10","created_at":"2026-07-17T05:50:00Z","published_at":"2026-07-17T05:58:19Z","draft":false},
|
||||
{"tag_name":"1.0.0-beta.11","created_at":"2026-07-23T04:00:00Z","published_at":"2026-07-23T04:07:31Z","draft":false},
|
||||
{"tag_name":"9.0.0","created_at":"2025-01-01T00:00:00Z","published_at":"2025-01-01T00:00:00Z","draft":false},
|
||||
{"tag_name":"1.0.0-beta.13","created_at":"2026-08-01T00:00:00Z","published_at":null,"draft":true}
|
||||
]
|
||||
]
|
||||
EOF
|
||||
|
||||
preview_previous=$(select_previous_release_tag "1.0.0-beta.12-preview.1" "2026-07-30T02:36:43Z" <<<"$release_fixture")
|
||||
assert_equal "1.0.0-beta.11" "$preview_previous" "preview notes baseline must exclude preview releases and its final deliverable"
|
||||
|
||||
final_previous=$(select_previous_release_tag "1.0.0-beta.12" "2026-07-30T04:13:04Z" <<<"$release_fixture")
|
||||
assert_equal "1.0.0-beta.11" "$final_previous" "final notes baseline must exclude the same-commit preview release"
|
||||
|
||||
no_previous=$(select_previous_release_tag "1.0.0-beta.1-preview.1" "2026-01-01T00:00:00Z" <<'EOF'
|
||||
[[{"tag_name":"1.0.0-beta.1-preview.1","created_at":"2026-01-01T00:00:00Z","published_at":"2026-01-01T00:00:00Z","draft":false}]]
|
||||
EOF
|
||||
)
|
||||
assert_equal "" "$no_previous" "repositories without a previous deliverable must use GitHub's fallback baseline"
|
||||
|
||||
delayed_draft_previous=$(select_previous_release_tag "1.0.0-beta.12" "2026-07-30T04:00:00Z" <<'EOF'
|
||||
[[
|
||||
{"tag_name":"1.0.0-beta.12","created_at":"2026-07-30T04:00:00Z","published_at":null,"draft":true},
|
||||
{"tag_name":"1.0.0-beta.13","created_at":"2026-08-01T00:00:00Z","published_at":"2026-08-01T00:05:00Z","draft":false},
|
||||
{"tag_name":"1.0.0-beta.11","created_at":"2026-07-23T04:00:00Z","published_at":"2026-07-23T04:07:31Z","draft":false}
|
||||
]]
|
||||
EOF
|
||||
)
|
||||
assert_equal "1.0.0-beta.11" "$delayed_draft_previous" "draft rerun must exclude deliverables published after the draft was created"
|
||||
|
||||
release_test_dir=$(mktemp -d "${TMPDIR:-/tmp}/rustfs-release-workflow-test.XXXXXX")
|
||||
trap 'rm -rf -- "$release_test_dir"' EXIT
|
||||
request_json="${release_test_dir}/request.json"
|
||||
write_generate_notes_request "1.0.0-beta.12" "0123456789abcdef" "1.0.0-beta.11" "$request_json"
|
||||
assert_equal "1.0.0-beta.11" "$(jq -r .previous_tag_name "$request_json")" "explicit previous_tag_name request field"
|
||||
write_generate_notes_request "1.0.0-beta.1" "0123456789abcdef" "" "$request_json"
|
||||
assert_equal "false" "$(jq 'has("previous_tag_name")' "$request_json")" "fallback request must omit previous_tag_name"
|
||||
|
||||
existing_release_json="${release_test_dir}/existing-release.json"
|
||||
jq -n --arg body "Pre-release 1.0.0-beta.12 (beta)" '{body: $body}' > "$existing_release_json"
|
||||
assert_equal "update" "$(release_notes_action "$existing_release_json" "1.0.0-beta.12")" "rerun must repair legacy prerelease placeholder notes"
|
||||
jq -n --arg body "${MANAGED_NOTES_MARKER}
|
||||
## What's Changed" '{body: $body}' > "$existing_release_json"
|
||||
assert_equal "update" "$(release_notes_action "$existing_release_json" "1.0.0-beta.12")" "rerun must refresh managed notes"
|
||||
|
||||
generated_body="## What's Changed
|
||||
|
||||
* Fix release notes by @alice in #123
|
||||
* First contribution by @bob in #124
|
||||
|
||||
## New Contributors
|
||||
* @bob made their first contribution in #124
|
||||
|
||||
**Full Changelog**: https://github.com/rustfs/rustfs/compare/1.0.0-beta.11...1.0.0-beta.12"
|
||||
response_json="${release_test_dir}/response.json"
|
||||
notes_file="${release_test_dir}/notes.md"
|
||||
jq -n --arg body "$generated_body" '{body: $body}' > "$response_json"
|
||||
write_generated_release_notes "$response_json" "$notes_file" "rustfs/rustfs" "1.0.0-beta.12" "1.0.0-beta.11"
|
||||
expected_notes="${MANAGED_NOTES_MARKER}
|
||||
${generated_body}"
|
||||
assert_equal "$expected_notes" "$(<"$notes_file")" "generated multiline Markdown must remain intact"
|
||||
|
||||
mock_log="${release_test_dir}/gh.log"
|
||||
mock_notes="${release_test_dir}/captured-notes.md"
|
||||
mock_generate_calls="${release_test_dir}/generate-calls"
|
||||
MOCK_RELEASES_JSON="$release_fixture"
|
||||
MOCK_GENERATED_BODY="$generated_body"
|
||||
MOCK_RELEASE_EXISTS=true
|
||||
MOCK_RELEASE_BODY="Release 1.0.0-beta.12"
|
||||
MOCK_RELEASE_VIEW_ERROR=""
|
||||
|
||||
gh() {
|
||||
local input_file=""
|
||||
local notes_path=""
|
||||
|
||||
if [[ "$1" == "api" && "$*" == *"--paginate --slurp"* ]]; then
|
||||
printf '%s\n' "$MOCK_RELEASES_JSON"
|
||||
return
|
||||
fi
|
||||
|
||||
if [[ "$1" == "api" && "$*" == *"releases/generate-notes"* ]]; then
|
||||
while [[ "$#" -gt 0 ]]; do
|
||||
if [[ "$1" == "--input" ]]; then
|
||||
input_file="$2"
|
||||
break
|
||||
fi
|
||||
shift
|
||||
done
|
||||
jq -e '.tag_name == "1.0.0-beta.12" and .target_commitish == "0123456789abcdef" and .previous_tag_name == "1.0.0-beta.11"' "$input_file" >/dev/null
|
||||
printf '%s\n' called >> "$mock_generate_calls"
|
||||
jq -n --arg body "$MOCK_GENERATED_BODY" '{body: $body}'
|
||||
return
|
||||
fi
|
||||
|
||||
if [[ "$1" == "release" && "$2" == "view" ]]; then
|
||||
if [[ -n "$MOCK_RELEASE_VIEW_ERROR" ]]; then
|
||||
printf '%s\n' "$MOCK_RELEASE_VIEW_ERROR" >&2
|
||||
return 1
|
||||
fi
|
||||
if [[ "$MOCK_RELEASE_EXISTS" != "true" ]]; then
|
||||
echo "release not found" >&2
|
||||
return 1
|
||||
fi
|
||||
jq -n --arg body "$MOCK_RELEASE_BODY" '{databaseId: 123, url: "https://github.com/rustfs/rustfs/releases/tag/test", body: $body, createdAt: "2026-07-30T04:00:00Z", publishedAt: "2026-07-30T04:13:04Z"}'
|
||||
return
|
||||
fi
|
||||
|
||||
if [[ "$1" == "release" && ("$2" == "edit" || "$2" == "create") ]]; then
|
||||
local action="$2"
|
||||
shift 2
|
||||
while [[ "$#" -gt 0 ]]; do
|
||||
if [[ "$1" == "--notes-file" ]]; then
|
||||
notes_path="$2"
|
||||
break
|
||||
fi
|
||||
shift
|
||||
done
|
||||
cp "$notes_path" "$mock_notes"
|
||||
MOCK_RELEASE_BODY=$(<"$mock_notes")
|
||||
MOCK_RELEASE_EXISTS=true
|
||||
printf '%s\n' "$action" >> "$mock_log"
|
||||
return
|
||||
fi
|
||||
|
||||
echo "unexpected mock gh invocation: $*" >&2
|
||||
return 1
|
||||
}
|
||||
|
||||
GITHUB_REPOSITORY="rustfs/rustfs"
|
||||
GITHUB_OUTPUT="${release_test_dir}/github-output"
|
||||
RUNNER_TEMP="$release_test_dir"
|
||||
create_or_update_release "1.0.0-beta.12" "0123456789abcdef" "RustFS 1.0.0-beta.12 (beta)" true
|
||||
assert_equal "edit" "$(<"$mock_log")" "existing placeholder release must be updated on rerun"
|
||||
assert_equal "$expected_notes" "$(<"$mock_notes")" "rerun must update notes through --notes-file"
|
||||
|
||||
: > "$mock_log"
|
||||
MOCK_RELEASE_EXISTS=false
|
||||
MOCK_RELEASE_BODY=""
|
||||
create_or_update_release "1.0.0-beta.12" "0123456789abcdef" "RustFS 1.0.0-beta.12 (beta)" true
|
||||
assert_equal "create" "$(<"$mock_log")" "first run must create the draft release"
|
||||
|
||||
: > "$mock_log"
|
||||
: > "$mock_generate_calls"
|
||||
MOCK_RELEASE_EXISTS=true
|
||||
MOCK_RELEASE_BODY="$generated_body"
|
||||
create_or_update_release "1.0.0-beta.12" "0123456789abcdef" "RustFS 1.0.0-beta.12 (beta)" true
|
||||
assert_equal "" "$(<"$mock_log")" "manual notes must not trigger a release edit"
|
||||
assert_equal "" "$(<"$mock_generate_calls")" "manual notes must be preserved without regeneration"
|
||||
|
||||
MOCK_RELEASE_BODY="Manually curated release guidance"
|
||||
if create_or_update_release "1.0.0-beta.12" "0123456789abcdef" "RustFS 1.0.0-beta.12 (beta)" true; then
|
||||
echo "invalid preview release workflow contract: incomplete manual notes must block publication" >&2
|
||||
exit 1
|
||||
fi
|
||||
assert_equal "" "$(<"$mock_log")" "incomplete manual notes must not be overwritten"
|
||||
|
||||
MOCK_RELEASE_VIEW_ERROR="gh: service unavailable (HTTP 503)"
|
||||
if create_or_update_release "1.0.0-beta.12" "0123456789abcdef" "RustFS 1.0.0-beta.12 (beta)" true; then
|
||||
echo "invalid preview release workflow contract: transient release lookup failures must fail closed" >&2
|
||||
exit 1
|
||||
fi
|
||||
assert_equal "" "$(<"$mock_log")" "release lookup failures must not attempt create or edit"
|
||||
|
||||
echo "Preview release workflow contract ok."
|
||||
cd "$(dirname "${BASH_SOURCE[0]}")/../.."
|
||||
python3 scripts/release/test_pipeline.py WorkflowContractTests
|
||||
|
||||
@@ -25,6 +25,7 @@ def parse_args() -> argparse.Namespace:
|
||||
parser.add_argument("--run-id", default=os.getenv("GITHUB_RUN_ID", ""))
|
||||
parser.add_argument("--run-attempt", default=os.getenv("GITHUB_RUN_ATTEMPT", ""))
|
||||
parser.add_argument("--cargo-lock", default="Cargo.lock", help="Cargo.lock path.")
|
||||
parser.add_argument("--workflow", default="build.yml", help="Workflow that assembled these assets.")
|
||||
parser.add_argument("--metadata-file", help="Use an existing cargo metadata JSON file.")
|
||||
return parser.parse_args()
|
||||
|
||||
@@ -118,7 +119,7 @@ def build_sbom(packages: list[dict], args: argparse.Namespace, timestamp: str) -
|
||||
|
||||
def build_provenance(asset_dir: Path, args: argparse.Namespace, timestamp: str) -> dict:
|
||||
subjects = []
|
||||
for asset in sorted(asset_dir.glob("*.zip")):
|
||||
for asset in sorted(path for path in asset_dir.iterdir() if path.suffix in (".zip", ".deb", ".rpm", ".tgz")):
|
||||
subjects.append(
|
||||
{
|
||||
"name": asset.name,
|
||||
@@ -129,7 +130,7 @@ def build_provenance(asset_dir: Path, args: argparse.Namespace, timestamp: str)
|
||||
if not subjects:
|
||||
raise RuntimeError(f"no .zip release assets found in {asset_dir}")
|
||||
|
||||
workflow_ref = f"{args.repository}/.github/workflows/build.yml"
|
||||
workflow_ref = f"{args.repository}/.github/workflows/{args.workflow}"
|
||||
invocation = args.run_id
|
||||
if args.run_attempt:
|
||||
invocation = f"{invocation}/attempts/{args.run_attempt}" if invocation else args.run_attempt
|
||||
|
||||
@@ -1,199 +1,12 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Exercise the Docker workflow's version selection and tags without publishing."""
|
||||
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import tempfile
|
||||
import unittest
|
||||
"""Exercise the candidate-based replacement for the former release trigger."""
|
||||
import importlib
|
||||
import sys
|
||||
from pathlib import Path
|
||||
import unittest
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
WORKFLOW = ROOT / ".github/workflows/docker.yml"
|
||||
REGISTRIES = ("rustfs/rustfs", "ghcr.io/rustfs/rustfs", "quay.io/rustfs/rustfs")
|
||||
|
||||
|
||||
def run_body(name):
|
||||
lines = WORKFLOW.read_text().splitlines()
|
||||
start = lines.index(f" - name: {name}")
|
||||
start = lines.index(" run: |", start) + 1
|
||||
body = []
|
||||
for line in lines[start:]:
|
||||
if line.strip() and not line.startswith(" "):
|
||||
break
|
||||
body.append(line[10:])
|
||||
return "\n".join(body)
|
||||
|
||||
|
||||
class DockerWorkflowTests(unittest.TestCase):
|
||||
def run_step(self, name, context, **env):
|
||||
script = re.sub(
|
||||
r"\$\{\{\s*(.*?)\s*\}\}", lambda m: context[m[1]], run_body(name)
|
||||
)
|
||||
# Fail on unexpected Git calls so these tests cannot contact a remote.
|
||||
mock_git = """
|
||||
git() {
|
||||
if [[ "$1" == rev-parse ]]; then
|
||||
echo 0123456789abcdef
|
||||
elif [[ "$1 $2 $3" == "ls-remote --tags --refs" ]]; then
|
||||
if [[ "$STABLE_EXISTS" == true ]]; then
|
||||
echo "0123456789abcdef refs/tags/v1.0.0"
|
||||
fi
|
||||
elif [[ "$1 $2" == "ls-remote --exit-code" ]]; then
|
||||
[[ "$4" == "$EXISTING_TAG" ]]
|
||||
else
|
||||
return 99
|
||||
fi
|
||||
}
|
||||
"""
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
output = Path(directory) / "output"
|
||||
result = subprocess.run(
|
||||
["bash", "-e", "-o", "pipefail", "-c", mock_git + script],
|
||||
env={**os.environ, "GITHUB_OUTPUT": str(output), **env},
|
||||
text=True,
|
||||
capture_output=True,
|
||||
check=False,
|
||||
)
|
||||
values = (
|
||||
dict(line.split("=", 1) for line in output.read_text().splitlines())
|
||||
if output.exists()
|
||||
else {}
|
||||
)
|
||||
return result, values
|
||||
|
||||
def classify(
|
||||
self, version, event="workflow_run", push="true", stable="true", tag=None, **env
|
||||
):
|
||||
return self.run_step(
|
||||
"Check build conditions",
|
||||
{"github.event_name": event},
|
||||
GITHUB_SHA="workflow-sha",
|
||||
HEAD_SHA="release-sha",
|
||||
HEAD_BRANCH=version,
|
||||
INPUT_VERSION=version,
|
||||
INPUT_PUSH_IMAGES=push,
|
||||
INPUT_FORCE_REBUILD="false",
|
||||
STABLE_EXISTS=stable,
|
||||
EXISTING_TAG=tag or f"refs/tags/{version}",
|
||||
**{"CONCLUSION": "success", "TRIGGERING_EVENT": "push", **env},
|
||||
)
|
||||
|
||||
def assert_tags(self, values, suffix="", channels=()):
|
||||
context = {
|
||||
f"needs.build-check.outputs.{key}": value for key, value in values.items()
|
||||
}
|
||||
context.update(
|
||||
{
|
||||
"matrix.suffix": suffix,
|
||||
"env.REGISTRY_DOCKERHUB": REGISTRIES[0],
|
||||
"env.REGISTRY_GHCR": REGISTRIES[1],
|
||||
"env.REGISTRY_QUAY": REGISTRIES[2],
|
||||
"github.server_url": "https://github.com",
|
||||
"github.repository": "rustfs/rustfs",
|
||||
}
|
||||
)
|
||||
result, metadata = self.run_step("Extract metadata and generate tags", context)
|
||||
self.assertEqual(result.returncode, 0, result.stderr)
|
||||
expected = {
|
||||
f"{registry}:{tag}{suffix}"
|
||||
for registry in REGISTRIES
|
||||
for tag in (values["version"], *channels)
|
||||
}
|
||||
self.assertEqual(set(metadata["tags"].split(",")), expected)
|
||||
|
||||
def test_preview_version_tags_only(self):
|
||||
versions = (
|
||||
"1.0.1-preview.8",
|
||||
"v1.0.1-preview.8",
|
||||
"1.0.0-alpha.1-preview.2",
|
||||
"1.0.0-beta.1-preview.2",
|
||||
"v1.0.0-rc.1-preview.2",
|
||||
)
|
||||
for event in ("workflow_run", "workflow_dispatch"):
|
||||
for version in versions:
|
||||
for stable in ("true", "false"):
|
||||
with self.subTest(event=event, version=version, stable=stable):
|
||||
result, values = self.classify(version, event, stable=stable)
|
||||
self.assertEqual(result.returncode, 0, result.stderr)
|
||||
for key, expected in {
|
||||
"should_build": "true",
|
||||
"should_push": "true",
|
||||
"build_type": "preview",
|
||||
"is_prerelease": "true",
|
||||
"create_latest": "false",
|
||||
"version": version.removeprefix("v"),
|
||||
}.items():
|
||||
self.assertEqual(values[key], expected, key)
|
||||
expected_ref = (
|
||||
"release-sha"
|
||||
if event == "workflow_run"
|
||||
else f"refs/tags/{version}"
|
||||
)
|
||||
self.assertEqual(values["source_ref"], expected_ref)
|
||||
for suffix in ("", "-glibc"):
|
||||
self.assert_tags(values, suffix)
|
||||
|
||||
def test_manual_preview_dry_run_and_tag_fallback(self):
|
||||
result, values = self.classify(
|
||||
"1.0.1-preview.8",
|
||||
"workflow_dispatch",
|
||||
push="false",
|
||||
tag="refs/tags/v1.0.1-preview.8",
|
||||
)
|
||||
self.assertEqual(result.returncode, 0, result.stderr)
|
||||
self.assertEqual(values["should_build"], "true")
|
||||
self.assertEqual(values["should_push"], "false")
|
||||
self.assertEqual(values["source_ref"], "refs/tags/v1.0.1-preview.8")
|
||||
|
||||
def test_full_tag_ref(self):
|
||||
result, values = self.classify("refs/tags/v1.0.1-preview.8")
|
||||
self.assertEqual(result.returncode, 0, result.stderr)
|
||||
self.assertEqual(values["build_type"], "preview")
|
||||
self.assertEqual(values["version"], "1.0.1-preview.8")
|
||||
self.assert_tags(values)
|
||||
|
||||
def test_invalid_preview_and_missing_tag_fail(self):
|
||||
for event in ("workflow_run", "workflow_dispatch"):
|
||||
for version in (
|
||||
"1.0.1-preview",
|
||||
"1.0.1-preview.x",
|
||||
"1.0.1-preview.8-extra",
|
||||
):
|
||||
with self.subTest(event=event, version=version):
|
||||
result, _ = self.classify(version, event)
|
||||
self.assertNotEqual(result.returncode, 0)
|
||||
result, _ = self.classify(
|
||||
"1.0.1-preview.8", "workflow_dispatch", tag="refs/tags/missing"
|
||||
)
|
||||
self.assertNotEqual(result.returncode, 0)
|
||||
|
||||
def test_existing_release_channels(self):
|
||||
for event in ("workflow_run", "workflow_dispatch"):
|
||||
for stable in ("true", "false"):
|
||||
for channel in (None, "alpha", "beta", "rc"):
|
||||
version = f"1.0.1-{channel}.1" if channel else "1.0.1"
|
||||
with self.subTest(event=event, stable=stable, channel=channel):
|
||||
result, values = self.classify(version, event, stable=stable)
|
||||
self.assertEqual(result.returncode, 0, result.stderr)
|
||||
latest = channel is None or stable == "false"
|
||||
self.assertEqual(values["create_latest"], str(latest).lower())
|
||||
channels = ([channel] if channel else []) + (
|
||||
["latest"] if latest else []
|
||||
)
|
||||
self.assert_tags(values, channels=channels)
|
||||
|
||||
def test_non_release_builds_are_skipped(self):
|
||||
for version, env in (
|
||||
("main", {}),
|
||||
("feature/test", {}),
|
||||
("1.0.1-preview.8", {"CONCLUSION": "failure"}),
|
||||
("1.0.1-preview.8", {"TRIGGERING_EVENT": "workflow_dispatch"}),
|
||||
):
|
||||
result, values = self.classify(version, **env)
|
||||
self.assertEqual(result.returncode, 0, result.stderr)
|
||||
self.assertEqual(values["should_build"], "false")
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent / "release"))
|
||||
module = importlib.import_module("test_pipeline")
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
unittest.main(module=module, defaultTest=('PromotionTests', 'ImageWorkflowTests'))
|
||||
|
||||
@@ -21,6 +21,8 @@ assert_version() {
|
||||
rm -f "$output"
|
||||
}
|
||||
|
||||
assert_version "refs/tags/v1.0.0-beta.12" "v1.0.0-beta.12" "1.0.0-beta.12" "0.12.0"
|
||||
assert_version "refs/tags/v1.0.0-beta.12" "v1.0.0-beta.12" "1.0.0-beta.12" "1.0.0-beta.12"
|
||||
assert_version "2.0.0-beta.12" "2.0.0-beta.12" "2.0.0-beta.12" "2.0.0-beta.12"
|
||||
assert_version "1.0.6-rc.1" "1.0.6-rc.1" "1.0.6-rc.1" "1.0.6-rc.1"
|
||||
assert_version "v1.0.0" "v1.0.0" "1.0.0" "1.0.0"
|
||||
assert_version "refs/tags/1.0.0" "1.0.0" "1.0.0" "1.0.0"
|
||||
|
||||
@@ -1,135 +1,12 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Exercise the Helm publication image gate without contacting a registry."""
|
||||
|
||||
import os
|
||||
import subprocess
|
||||
import tempfile
|
||||
import unittest
|
||||
"""Exercise the candidate-based replacement for the former release trigger."""
|
||||
import importlib
|
||||
import sys
|
||||
from pathlib import Path
|
||||
import unittest
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
WORKFLOW = ROOT / ".github/workflows/helm-package.yml"
|
||||
|
||||
|
||||
def image_gate():
|
||||
lines = WORKFLOW.read_text().splitlines()
|
||||
start = lines.index(" - name: Wait for release image")
|
||||
start = lines.index(" run: |", start) + 1
|
||||
body = []
|
||||
for line in lines[start:]:
|
||||
if line.strip() and not line.startswith(" "):
|
||||
break
|
||||
body.append(line[10:])
|
||||
return "\n".join(body)
|
||||
|
||||
|
||||
class HelmReleaseWorkflowTests(unittest.TestCase):
|
||||
def run_gate(self, version="1.0.1", failures=0, auth="ready"):
|
||||
# Unexpected URLs fail closed; these functions cannot access the network.
|
||||
mocks = r"""
|
||||
curl() {
|
||||
case "${*: -1}" in
|
||||
'https://auth.docker.io/token?service=registry.docker.io&scope=repository:rustfs/rustfs:pull')
|
||||
echo auth >> "$CALLS"
|
||||
case "$AUTH_STATE" in
|
||||
unavailable) return 22 ;;
|
||||
empty) echo '{}' ;;
|
||||
ready) echo '{"token":"registry-test-token"}' ;;
|
||||
*) return 99 ;;
|
||||
esac
|
||||
;;
|
||||
"https://registry-1.docker.io/v2/rustfs/rustfs/manifests/$APP_VERSION")
|
||||
[[ " $* " == *" --head "* ]] || return 99
|
||||
[[ "$*" == *"Authorization: Bearer registry-test-token"* ]] || return 99
|
||||
echo image >> "$CALLS"
|
||||
[[ $(grep -c '^image$' "$CALLS") -gt "$IMAGE_FAILURES" ]]
|
||||
;;
|
||||
*) return 99 ;;
|
||||
esac
|
||||
}
|
||||
sleep() {
|
||||
[[ "$1" == 15 ]] || return 99
|
||||
echo sleep >> "$CALLS"
|
||||
}
|
||||
"""
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
calls = Path(directory) / "calls"
|
||||
result = subprocess.run(
|
||||
["bash", "-e", "-o", "pipefail", "-c", mocks + image_gate()],
|
||||
env={
|
||||
**os.environ,
|
||||
"APP_VERSION": version,
|
||||
"IMAGE_FAILURES": str(failures),
|
||||
"AUTH_STATE": auth,
|
||||
"CALLS": str(calls),
|
||||
},
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=False,
|
||||
timeout=30,
|
||||
)
|
||||
events = calls.read_text().splitlines() if calls.exists() else []
|
||||
return result, events
|
||||
|
||||
def test_published_stable_and_prerelease_images(self):
|
||||
for version in ("1.0.1", "1.1.0-beta.1", "1.1.0-rc.1"):
|
||||
with self.subTest(version=version):
|
||||
result, events = self.run_gate(version)
|
||||
self.assertEqual(result.returncode, 0, result.stderr)
|
||||
self.assertEqual(events, ["auth", "image"])
|
||||
|
||||
def test_image_published_after_binary_build(self):
|
||||
result, events = self.run_gate(failures=2)
|
||||
self.assertEqual(result.returncode, 0, result.stderr)
|
||||
self.assertEqual(events.count("image"), 3)
|
||||
self.assertEqual(events.count("sleep"), 2)
|
||||
|
||||
def test_missing_image_blocks_publication_after_bounded_retries(self):
|
||||
result, events = self.run_gate(failures=60)
|
||||
self.assertNotEqual(result.returncode, 0)
|
||||
self.assertIn("refusing to publish Helm chart", result.stderr)
|
||||
self.assertEqual(events.count("image"), 60)
|
||||
self.assertEqual(events.count("sleep"), 59)
|
||||
|
||||
def test_auth_failure_is_not_image_availability(self):
|
||||
for auth in ("unavailable", "empty"):
|
||||
with self.subTest(auth=auth):
|
||||
result, events = self.run_gate(auth=auth)
|
||||
self.assertNotEqual(result.returncode, 0)
|
||||
self.assertEqual(events.count("auth"), 60)
|
||||
self.assertNotIn("image", events)
|
||||
|
||||
def test_preview_and_invalid_versions_never_reach_registry(self):
|
||||
for version in ("1.0.1-preview.1", "1.1.0-beta.1-preview.2", "latest", "", "../tags"):
|
||||
with self.subTest(version=version):
|
||||
result, events = self.run_gate(version)
|
||||
self.assertNotEqual(result.returncode, 0)
|
||||
self.assertEqual(events, [])
|
||||
|
||||
def test_image_gate_precedes_packaging(self):
|
||||
workflow = WORKFLOW.read_text()
|
||||
gate = workflow.index(" - name: Wait for release image")
|
||||
package = workflow.index(" - name: Package Helm Chart")
|
||||
self.assertLess(gate, package)
|
||||
self.assertNotIn("continue-on-error", workflow[gate:package])
|
||||
|
||||
def test_upload_artifact_includes_chart_and_readme(self):
|
||||
workflow = WORKFLOW.read_text()
|
||||
copy_readme = workflow.index("cp helm/README.md helm/rustfs/")
|
||||
upload = workflow.index(" - name: Upload helm package as artifact")
|
||||
self.assertLess(copy_readme, upload)
|
||||
|
||||
publish = workflow.index(" publish-helm-package:", upload)
|
||||
upload_step = workflow[upload:publish]
|
||||
expected_path = "path: |\n helm/rustfs/*.tgz\n helm/rustfs/README.md\n"
|
||||
self.assertIn(expected_path, upload_step)
|
||||
|
||||
download = workflow.index(" - name: Download helm package", publish)
|
||||
push_step = workflow.index(" - name: Push helm package and index file", download)
|
||||
download_step = workflow[download:push_step]
|
||||
self.assertIn("name: helm-package", download_step)
|
||||
self.assertIn("path: ./", download_step)
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent / "release"))
|
||||
module = importlib.import_module("test_pipeline")
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
unittest.main(module=module, defaultTest=('HelmWorkflowTests',))
|
||||
|
||||
@@ -0,0 +1,106 @@
|
||||
#!/usr/bin/env python3
|
||||
# Copyright 2026 RustFS Team
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
"""Exercise real OCI builders and promotion with explicit non-product fixtures."""
|
||||
import importlib
|
||||
import os
|
||||
from pathlib import Path
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
import unittest
|
||||
import urllib.request
|
||||
from unittest.mock import patch
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent / "release"))
|
||||
c = importlib.import_module("candidate")
|
||||
deliver = importlib.import_module("deliver")
|
||||
publish = importlib.import_module("publish")
|
||||
fixtures = importlib.import_module("test_candidate")
|
||||
|
||||
|
||||
class ImageIntegrationTests(unittest.TestCase):
|
||||
def test_native_oci_smoke_and_digest_preserving_registry_promotion(self):
|
||||
value = fixtures.inputs()
|
||||
variant = os.environ["RELEASE_IMAGE_VARIANT"]
|
||||
self.assertIn(variant, ("musl", "gnu"))
|
||||
output = Path(os.environ["RELEASE_IMAGE_OUTPUT"]).resolve()
|
||||
output.mkdir(parents=True, exist_ok=True)
|
||||
run = subprocess.run
|
||||
with tempfile.TemporaryDirectory() as temp:
|
||||
root = Path(temp)
|
||||
for name in ("Dockerfile", "Dockerfile.glibc", "entrypoint.sh"):
|
||||
shutil.copyfile(c.ROOT / name, root / name)
|
||||
shutil.copytree(c.ROOT / "bootstrap", root / "bootstrap")
|
||||
(root / "scripts").mkdir()
|
||||
shutil.copyfile(c.ROOT / "scripts/download_rc.sh", root / "scripts/download_rc.sh")
|
||||
source = root / "fixture.go"
|
||||
source.write_text('''package main
|
||||
import ("fmt"; "net/http"; "os")
|
||||
func main() {
|
||||
if len(os.Args) > 1 && os.Args[1] == "--version" { fmt.Println("rustfs 1.2.3"); return }
|
||||
http.HandleFunc("/rustfs/console/", func(w http.ResponseWriter, r *http.Request) {
|
||||
w.Header().Set("Content-Type", "text/html"); fmt.Fprintln(w, "<html>transport fixture</html>")
|
||||
})
|
||||
if err := http.ListenAndServe(":9001", nil); err != nil { panic(err) }
|
||||
}
|
||||
''')
|
||||
binaries = {}
|
||||
for arch, docker_arch in (("x86_64", "amd64"), ("aarch64", "arm64")):
|
||||
directory = root / ".release-inputs" / docker_arch
|
||||
directory.mkdir(parents=True)
|
||||
env = dict(os.environ, GOOS="linux", GOARCH=docker_arch, CGO_ENABLED="0")
|
||||
run(["go", "build", "-o", str(directory / "rustfs"), str(source)], env=env, check=True, timeout=180)
|
||||
(directory / "rc").write_text("#!/bin/sh\nexit 0\n")
|
||||
(directory / "rc").chmod(0o755)
|
||||
binaries[arch] = c.file_digest(directory / "rustfs")
|
||||
archive = output / f"image-{variant}.tar"
|
||||
dockerfile = "Dockerfile" if variant == "musl" else "Dockerfile.glibc"
|
||||
run(["docker", "buildx", "build", "--platform", "linux/amd64,linux/arm64", "--provenance=false",
|
||||
"--build-arg", "BINARY_SOURCE=supplied", "--build-arg", "RELEASE=1.2.3", "-f", str(root / dockerfile),
|
||||
"--output", f"type=oci,dest={archive}", str(root)], check=True, timeout=1800)
|
||||
c.write_json(output / f"image-{variant}-inputs.json", {"inputs_sha256": c.digest(value), "binaries": binaries})
|
||||
deliver.image_smoke(value, variant, output)
|
||||
with self.assertRaisesRegex(ValueError, "wrong version"):
|
||||
deliver.image_smoke(dict(value, version="0.0.0"), variant, output)
|
||||
expected = c.read_json(output / f"image-{variant}-smoke.json")["digest"]
|
||||
registry = c.command(["docker", "run", "-d", "-p", "127.0.0.1::5000", "registry:3"])
|
||||
try:
|
||||
address = c.command(["docker", "port", registry, "5000/tcp"])
|
||||
opener = urllib.request.build_opener(urllib.request.ProxyHandler({}))
|
||||
deadline = time.monotonic() + 15
|
||||
while time.monotonic() < deadline:
|
||||
try:
|
||||
with opener.open(f"http://{address}/v2/", timeout=2) as response:
|
||||
self.assertEqual(response.status, 200)
|
||||
break
|
||||
except OSError:
|
||||
time.sleep(0.1)
|
||||
else:
|
||||
self.fail("local registry did not become ready")
|
||||
auth = root / "auth.json"
|
||||
c.write_json(auth, {"auths": {}})
|
||||
reference = f"{address}/fixture:1.2.3"
|
||||
def local_registry(args, **kwargs):
|
||||
# Only the ephemeral loopback registry uses HTTP. All OCI
|
||||
# copying, digest checks and conflict handling are real.
|
||||
if args[:2] == ["skopeo", "inspect"]:
|
||||
args = args[:2] + ["--tls-verify=false"] + args[2:]
|
||||
elif args[:2] == ["skopeo", "copy"]:
|
||||
args = args[:2] + ["--dest-tls-verify=false"] + args[2:]
|
||||
return run(args, **kwargs)
|
||||
with patch.dict(os.environ, REGISTRY_AUTH_FILE=str(auth)), patch.object(subprocess, "run", side_effect=local_registry):
|
||||
publish.copy_image(archive, reference, expected)
|
||||
publish.copy_image(archive, reference, expected)
|
||||
self.assertEqual(publish.image_digest(reference), expected)
|
||||
with self.assertRaisesRegex(ValueError, "never replace"):
|
||||
publish.copy_image(archive, reference, "sha256:" + "0" * 64)
|
||||
self.assertEqual(publish.image_digest(reference), expected)
|
||||
finally:
|
||||
run(["docker", "rm", "-f", "-v", registry], check=True, timeout=60)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,68 @@
|
||||
#!/usr/bin/env python3
|
||||
# Copyright 2026 RustFS Team
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
"""Linux integration tests using the real DEB/RPM recipes and package managers."""
|
||||
import hashlib
|
||||
import importlib
|
||||
import os
|
||||
from pathlib import Path
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent / "release"))
|
||||
deliver = importlib.import_module("deliver")
|
||||
c = importlib.import_module("candidate")
|
||||
fixtures = importlib.import_module("test_candidate")
|
||||
|
||||
|
||||
class PackageIntegrationTests(unittest.TestCase):
|
||||
@classmethod
|
||||
def setUpClass(cls):
|
||||
missing = [tool for tool in ("dpkg-deb", "fakeroot", "rpm", "rpm2cpio", "cpio", "fpm") if not shutil.which(tool)]
|
||||
if sys.platform != "linux" or missing:
|
||||
raise RuntimeError(f"Run on Linux with real packaging tools; missing: {missing}")
|
||||
|
||||
def test_each_target_wraps_its_accepted_bytes(self):
|
||||
for target in c.LINUX:
|
||||
with self.subTest(target=target), tempfile.TemporaryDirectory() as temp:
|
||||
root = Path(temp)
|
||||
executable = root / "rustfs"
|
||||
executable.write_bytes(b"package contract fixture only: " + target.encode())
|
||||
expected = hashlib.sha256(executable.read_bytes()).hexdigest()
|
||||
value = fixtures.inputs()
|
||||
with patch.object(deliver, "binary", return_value=(executable, {"binary_sha256": expected})):
|
||||
deliver.package(value, target, root / "output")
|
||||
record = c.read_json(root / "output" / f"packages-{target}.json")
|
||||
self.assertEqual(record["binary_sha256"], expected)
|
||||
self.assertEqual(record["inputs_sha256"], c.digest(value))
|
||||
self.assertEqual({Path(item["name"]).suffix for item in record["files"]}, {".deb", ".rpm"})
|
||||
|
||||
def test_real_package_with_substituted_payload_is_rejected(self):
|
||||
with tempfile.TemporaryDirectory() as temp:
|
||||
root = Path(temp)
|
||||
executable = root / "rustfs"
|
||||
executable.write_bytes(b"accepted fixture")
|
||||
expected = c.file_digest(executable)
|
||||
real_run = deliver.run
|
||||
def mutate_deb(args, **kwargs):
|
||||
result = real_run(args, **kwargs)
|
||||
if any(str(arg).endswith("build_deb.sh") for arg in args):
|
||||
work = Path(kwargs["cwd"])
|
||||
package = work / kwargs["env"]["DEB_FILE"]
|
||||
unpacked = work / "corrupted-package"
|
||||
subprocess.run(["dpkg-deb", "--raw-extract", str(package), str(unpacked)], check=True)
|
||||
(unpacked / "usr/bin/rustfs").write_bytes(b"wrong server")
|
||||
subprocess.run(["dpkg-deb", "--build", str(unpacked), str(package)], check=True)
|
||||
return result
|
||||
with patch.object(deliver, "binary", return_value=(executable, {"binary_sha256": expected})), patch.object(deliver, "run", side_effect=mutate_deb):
|
||||
with self.assertRaisesRegex(ValueError, "package binary differs"):
|
||||
deliver.package(fixtures.inputs(), "linux-x86_64-musl", root / "output")
|
||||
self.assertFalse(list((root / "output").glob("packages-*.json")))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
Reference in New Issue
Block a user