mirror of
https://github.com/rustfs/rustfs.git
synced 2026-11-10 20:28:47 +00:00
Compare commits
14 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 4ef63de06e | |||
| 8391c4f4f6 | |||
| 2780bba67b | |||
| 3a23122104 | |||
| 0b1ebeffec | |||
| 0f9258594c | |||
| 33fc735ee2 | |||
| 509869cfc5 | |||
| 395876ccd0 | |||
| 6083fbbeea | |||
| 42d3267d88 | |||
| 1d35e381b7 | |||
| 0b109b4885 | |||
| 1dea00ffa7 |
@@ -1,43 +1,38 @@
|
||||
---
|
||||
name: rustfs-release-publish
|
||||
description: Publish a RustFS version through Console, release-branch CI, preview acceptance, human confirmation, and final publication. Use only for an explicit release request.
|
||||
description: Publish a RustFS version through frozen Linux candidate acceptance, desktop completion, manifest approval, byte-preserving publication and post-release follow-up. Use only for an explicit release request.
|
||||
---
|
||||
# RustFS Release
|
||||
|
||||
Require the exact final SemVer target before version edits or publication. If unspecified, inspect current releases/tags and ask with concrete candidates; do not guess the channel or bump. A named bump type resolves to an exact version that still needs confirmation. Report the confirmed target verbatim; resolve any later mismatch before proceeding. Independent read-only preflight may continue.
|
||||
Require the exact final SemVer target before version edits or publication. If unspecified, inspect releases/tags and ask with concrete candidates; do not guess the channel or bump. Reuse the user's confirmed target and authorized delivery scope. Independent read-only preflight may continue.
|
||||
|
||||
## Invariants
|
||||
|
||||
- Cargo stores `<target>`, never `-preview.N`. Before publication, installation defaults retain the previous available deliverable. Final and preview binaries embed their own tag names, so version/asset names differ despite identical source.
|
||||
- Use annotated tags without `v`. Preview names are exactly `<target>-preview.<digits>`; choose the next unused iteration after fetching tags. Preview Releases are prereleases, never Latest; publish only versioned assets/exact Docker tags, never latest assets, R2, Helm, moving Docker channels, or installation/banner updates.
|
||||
- `release/<target>` owns the candidate. Retain its history, backport only reviewed fixes, and never merge main wholesale or force-push it. Pin `PREVIEW_HASH` to its exact commit with complete passing CI. Preview and final tags both point there; never tag a later main/branch HEAD or insert another bump.
|
||||
- Phase failures block dependents. Keep preview Releases until final publication and notes generation finish; CI removes Releases while retaining tags. Never move an existing preview or final tag.
|
||||
- Cargo and every candidate executable contain the final product version. `<target>-preview.<run_number>` is an immutable candidate identifier, not a public tag or displayed binary version.
|
||||
- Freeze the exact source, reviewed workflow recipe, toolchain, Console, CLI, dependency lock, previous stable version and validation plan. Newer unrelated main or Console releases do not change those inputs.
|
||||
- Retain `release/<target>` history when a release branch is used; backport only reviewed fixes. Pin `CANDIDATE_SHA` to its exact CI-validated commit. Never force-push a release branch or move a public tag.
|
||||
- Build Linux first. Complete field acceptance before macOS/Windows and packaging. Full certification and final acceptance are required before publication; a partial Linux PASS grants no release authority.
|
||||
- Promote the accepted bytes. Final publication never recompiles binaries or rebuilds packages/images. Infrastructure retries reuse successful artifacts; changed source or inputs require a new candidate and affected acceptance.
|
||||
- Phase failures block dependents. No latest/default-channel update before versioned delivery. Use the [pipeline runbook](../../../docs/operations/release-pipeline.md) for executable entry points, locks and recovery.
|
||||
|
||||
## Read by Phase
|
||||
|
||||
Read only the current phase's procedure; retain completed evidence across turns. Follow root verification, Git/PR, and authorization rules.
|
||||
## Read by phase
|
||||
|
||||
| Phase | Procedure |
|
||||
|---|---|
|
||||
| 0 | Clean tree, fetch main/tags, verify GitHub access, then complete the mandatory [Console release gate](references/console-gate.md). A green Console build alone is insufficient. |
|
||||
| 1–2 | [Prepare/reuse the release candidate and run exact-SHA branch CI](references/release-candidate.md). Source version changes use `rustfs-release-version-bump` with authorized delivery scope. |
|
||||
| 3 | [Verify the preview build, assets, channels, and notes](references/preview-release.md). |
|
||||
| 4–5 | [Run the downloaded binary, Console CRUD, and latest-rc matrix](references/preview-acceptance.md). Every check must pass. |
|
||||
| Confirmation | Complete the gate below and end the turn. |
|
||||
| 6 | After fresh confirmation, [publish and verify the final tag](references/final-release.md) at `PREVIEW_HASH`. |
|
||||
| 7 | [Reconcile Security Advisories, milestones, installation references, and the website; reconcile main and remove the completed release branch](references/post-release-updates.md) only after complete publication. |
|
||||
| 0 | Clean tree, fetch main/tags, verify access, then complete the mandatory [Console gate](references/console-gate.md). |
|
||||
| 1–2 | [Prepare the exact source and run branch CI](references/release-candidate.md). Version edits use `rustfs-release-version-bump`. |
|
||||
| 3 | [Build and verify the Linux candidate](references/preview-release.md). |
|
||||
| 4–5 | [Run artifact, field, Console and rc acceptance](references/preview-acceptance.md), then complete desktop/package/image/chart validation. |
|
||||
| Confirmation | Report the complete manifest and obtain the confirmation below. |
|
||||
| 6 | [Publish the accepted manifest](references/final-release.md). |
|
||||
| 7 | [Reconcile advisories, milestones, installation references, website and release branch](references/post-release-updates.md). |
|
||||
|
||||
## Human Confirmation
|
||||
## Human confirmation
|
||||
|
||||
After Phases 3–5 pass, recheck that the release branch still equals `PREVIEW_HASH` and that the current Console asset tag/digest match the preview's recorded build inputs. Report the target, branch, preview tag/hash, branch CI run, preview Release URL, accepted Console asset identity/result, and rc matrix. Explicitly ask whether to publish the final tag and end the turn without creating/pushing it.
|
||||
After all acceptance passes, report the target, source SHA, source CI run, Linux and completion run URLs, final manifest SHA-256, accepted Console/CLI identities, compatibility/performance results and backlog report. Explicitly ask whether to publish this exact manifest and end the turn without dispatching publication.
|
||||
|
||||
Only a new affirmative user reply to that exact report authorizes Phase 6. The original release request, earlier confirmation, silence, or automated follow-up does not count. Confirmation binds the target, preview tag, hash, and accepted build inputs; a changed value or repeated/failed acceptance cycle invalidates it. Revalidate and obtain fresh confirmation before continuing.
|
||||
Only a new affirmative reply to that exact report authorizes Phase 6. An earlier general release request, silence or automatic continuation does not count. Reuse confirmation for retries of that same manifest; do not ask again merely because a destination failed. A changed source, input, manifest or acceptance result invalidates it and requires renewed acceptance/confirmation.
|
||||
|
||||
## Recovery
|
||||
## Recovery and reporting
|
||||
|
||||
- Before the final tag exists, land fixes on main, backport to the candidate, verify target versions, and restart branch CI/acceptance. Use a new preview iteration once a preview was tagged; main may advance independently.
|
||||
- If only an external build input changed, record it, create a new preview iteration even when the source SHA is unchanged, and repeat the affected build and artifact acceptance before fresh confirmation. Reuse source checks whose inputs remain valid.
|
||||
- Once the final tag exists, retry publication jobs for that exact tag. A source fix requires a newly confirmed target, not retagging or a newer preview. Phase 7 retries recheck artifact availability and superseding releases.
|
||||
- If abandoned after the bump merged, report the unpublished version on main and record whether to revert or let the next release overwrite it. Deleting unpublished candidate branches or preview tags requires an explicit decision. Completed release branches follow Phase 7 cleanup; keep release tags.
|
||||
|
||||
Report completed phase results and incomplete work with evidence: Console previous/latest tags, release need, hash and run/Release URLs; target, branch CI/SHA, previews and hash, confirmation state; preview/final assets, latest-channel state, Console/rc results and cleanup; advisory reconciliation results and pending disclosure decisions; both milestone states, installation PR checks/merge state, and website text/link/PR/deployment state. Keep artifact publication separate from follow-up delivery; do not call an open PR a live update. Report deviations and their authorization.
|
||||
Resume failed infrastructure jobs without rebuilding successful units. Fix product defects on main, backport when needed, and start a new candidate. After a final tag exists, retry the same bytes or use a new patch version; never move the tag. Do not conflate implemented, CI-passed, field-accepted, approved, published and post-release-complete states. Report incomplete work and deviations with evidence; a tag or green build alone is not a completed release.
|
||||
|
||||
@@ -4,7 +4,7 @@ Read during Phase 0, before changing RustFS version files or tags. Follow the pa
|
||||
|
||||
### Console release gate
|
||||
|
||||
Complete this gate before changing any RustFS version file or creating any RustFS tag. RustFS `build.yml` downloads the asset returned by `repos/rustfs/console/releases/latest`, so a successful Console build alone is insufficient.
|
||||
Complete this gate before changing any RustFS version file or creating any RustFS tag. Candidate creation resolves `repos/rustfs/console/releases/latest` once and freezes the exact asset for every build, so a successful Console build alone is insufficient. After freezing, later Console releases do not silently change or invalidate the accepted inputs.
|
||||
|
||||
1. Read the latest published Console tag and compare it with Console `main`:
|
||||
|
||||
|
||||
@@ -1,22 +1,13 @@
|
||||
# Final Release Publication
|
||||
# Final release publication
|
||||
|
||||
Read only after fresh user confirmation of the preview acceptance report. The [parent confirmation gate](../SKILL.md#human-confirmation) applies.
|
||||
Read only after fresh confirmation of the complete candidate report. The [parent confirmation gate](../SKILL.md#human-confirmation) applies.
|
||||
|
||||
## Phase 6 — Publish the final tag on the validated commit
|
||||
Dispatch `release-publish.yml` on main with the successful completion run ID and approved `release-manifest.json` SHA-256. Do not create the tag manually. The workflow revalidates source, bytes and trusted certification, then creates the annotated final tag at `CANDIDATE_SHA`. A recipe change fails closed and requires a new candidate; unrelated main changes do not require a rebuild.
|
||||
|
||||
No second source version bump. Before creating the final tag, recheck the remote `release/<target>` HEAD equals `PREVIEW_HASH` and the current Console release tag/asset digest still match the accepted preview inputs. If either changed after confirmation, the approval no longer applies; validate the changed inputs with a new preview and obtain fresh confirmation. The final tag goes on the exact commit the preview validated:
|
||||
Require both versioned publication and channel publication jobs to succeed. Verify the final tag's commit and manifest identity; all six platform ZIPs and compatibility aliases; DEB/RPM, checksums, SBOM, provenance and evidence bundle; both multi-architecture image variants in Docker Hub, GHCR and Quay; and the accepted Helm chart in the chart repository. Download a host-compatible final artifact and confirm its digest is exactly the accepted candidate digest and its version/source output agrees.
|
||||
|
||||
```bash
|
||||
git fetch origin --tags
|
||||
git rev-parse "<preview-tag>^{commit}" # must equal PREVIEW_HASH — abort if not
|
||||
git tag -a "<target>" -m "Release <target>" "$PREVIEW_HASH"
|
||||
git push origin "<target>"
|
||||
```
|
||||
For stable releases verify GitHub Latest, R2 download aliases and `latest.json` unless a newer stable release already owns those channels. Prereleases must remain GitHub prereleases and leave stable pointers unchanged. Verify generated release notes compare against the frozen previous stable release, excluding internal previews.
|
||||
|
||||
- CI rebuilds from the same source with a different tag name, so the binary self-reports `<target>`. Other inputs can change: `build.yml` reads Console `releases/latest` during each build. Compare the actual Console tag, URL, and verified digest in every final build job with the accepted preview record, and inspect available build provenance for other relevant changes. A pre-tag check cannot prevent a later input change. On a mismatch or missing evidence, repeat affected acceptance against the final artifacts and report the limitation; failed acceptance leaves release validation incomplete even if publication already succeeded. Never move the final tag to conceal a failure.
|
||||
- Verify the final tag's complete publication path: all matrix and release jobs green; `gh release view "<target>"` shows the full versioned and `-latest` asset set plus checksums, SBOM, and provenance; Docker and Helm workflows succeed; `latest.json` points to `<target>`. A stable target must have `isPrerelease=false` and `isLatest=true`. An alpha/beta/rc target must have `isPrerelease=true`; GitHub does not permit prereleases to be Latest, but the project `latest.json` still advances to the final non-preview target.
|
||||
- Verify the final Release body contains `## What's Changed` and a Full Changelog link. When `PREVIOUS_DELIVERABLE` exists, the link MUST be `https://github.com/rustfs/rustfs/compare/<PREVIOUS_DELIVERABLE>...<target>` and the baseline MUST equal the preview Release baseline; for example, both `1.0.0-beta.12-preview.1` and `1.0.0-beta.12` compare from `1.0.0-beta.11`.
|
||||
- Verify the preview cleanup: `cleanup-preview-releases` must succeed and `gh release view "<preview-tag>"` must then report `release not found` for every preview iteration of this target. Each surviving preview tag must still resolve to its own recorded commit; the accepted preview resolves to `PREVIEW_HASH`. If the job failed, delete the leftover Releases manually with `gh release delete "<preview-tag>" --yes` and report it.
|
||||
- Verify a host-compatible final artifact's checksum and `./rustfs --version`; it must report `<target>` and the expected source SHA. This identity check does not replace affected acceptance when build inputs changed.
|
||||
On infrastructure failure retry the same workflow/seal; matching destinations are reused. Never clobber an asset, force a tag, rebuild the image or silently accept a changed digest. A source fix after the final tag requires a new patch version. Cross-registry writes are not atomic: report the unfinished destination rather than claiming publication complete.
|
||||
|
||||
Never pass `--cleanup-tag` when removing preview Releases. Remove them manually only if `cleanup-preview-releases` failed, after final publication and notes generation. Record deleted Releases and surviving tags.
|
||||
New candidates create no preview Releases to clean up. For a legacy preview release, preserve its tag and remove its Release only after final delivery and notes are verified. Do not pass `--cleanup-tag`. Complete the [post-release follow-up](post-release-updates.md) separately.
|
||||
|
||||
@@ -24,7 +24,7 @@ Reconcile advisories after every completed final release, including Console-only
|
||||
|
||||
1. Re-read the published deliverable state before edits or a retry. If a newer release has superseded this target, do not downgrade installation references or the website announcement. Report the superseding release and leave those defaults intact.
|
||||
2. Invoke `rustfs-release-version-bump` with stage `post-release`, the exact target, and the authorized delivery scope. Prepare the follow-up on current main; do not edit or move either validated release tag. Cargo versions may already be preparing the next target and must remain untouched.
|
||||
3. Verify the exact Docker image manifest, Release download links, RPM source archive, and Helm repository entry before publishing the installation PR. The chart workflow overrides chart/app versions from the final tag, pins the source checkout to that release, and waits for its default Docker image before packaging.
|
||||
3. Verify the exact Docker image manifest, Release download links, RPM source archive, and Helm repository entry before publishing the installation PR. The chart is packaged and tested before sealing; publication uploads those same bytes only after versioned Docker images are available.
|
||||
4. Report the installation PR and its merge state separately from the completed release. Reuse existing merge authorization and repository checks; do not infer new merge authority from this reference.
|
||||
|
||||
## rustfs.com top banner
|
||||
@@ -54,4 +54,4 @@ Use the existing announcement configuration from [website PR #112](https://githu
|
||||
1. Compare the exact published tag with current main. Audit every release-side commit since their merge base, including inherited backports. An ahead count is a history difference, not proof of missing changes. Use ancestry and patch equivalence, then inspect current code and regression evidence for squashed, adapted, superseded, or reverted changes. Record each commit as present, superseded, release-only, or missing; a positive `git cherry` result alone does not prove a missing fix.
|
||||
2. Bring missing shared changes to main through a reviewed PR and the required checks before closing the follow-up. Preserve main's later changes; decide version bumps and pinned Console configuration separately. If a history merge is needed, use a merge commit and verify the release commit is an ancestor of main; squash does not establish that ancestry. Do not merge main into the published release branch or move its tags. Report unresolved changes instead of claiming reconciliation is complete.
|
||||
3. After publication and authorized follow-ups are complete, delete the remote `release/<target>` branch. Immediately before deletion, verify its HEAD still equals the published tag's commit and no open PR targets or originates from it. Preserve a branch with newer work or open PRs and report the blocker. Use a deletion with an expected-old-SHA check to avoid removing a concurrently advanced branch.
|
||||
4. Verify the remote branch is absent and the final and preview tags still resolve to their recorded commits. Remove a local release branch only if it is task-owned, unchecked out, and has no unpreserved work. Record cleanup separately from artifact publication; an explicitly authorized deletion before reconciliation does not make unresolved main changes complete.
|
||||
4. Verify the remote branch is absent and the final tag and any legacy preview tags still resolve to their recorded commits. Remove a local release branch only if it is task-owned, unchecked out, and has no unpreserved work. Record cleanup separately from artifact publication; an explicitly authorized deletion before reconciliation does not make unresolved main changes complete.
|
||||
|
||||
@@ -1,10 +1,10 @@
|
||||
# Preview Artifact Acceptance
|
||||
# Candidate Artifact Acceptance
|
||||
|
||||
Read after Phase 3 succeeds. Complete every check below before the parent skill's manual confirmation gate. These checks cover the downloaded artifact, embedded Console, and latest rc client.
|
||||
Read after Phase 3 succeeds. Complete every check below before the parent skill's manual confirmation gate. These checks cover the downloaded artifact, embedded Console, and the rc release frozen in the candidate inputs.
|
||||
|
||||
## Phase 4 — Run the artifact locally, verify the console
|
||||
|
||||
Use a unique task-owned scratch directory. Select the versioned archive matching the host OS, architecture, and runtime, download its checksum file, and verify the archive before extraction. Do not substitute a different build when a matching artifact is unavailable; report that acceptance gap. The shell example below assumes the verified executable is at `RUSTFS_BINARY` (`rustfs.exe` on Windows), `SCRATCH` is the task directory, and `S3_PORT` and `CONSOLE_PORT` are distinct available ports. Use equivalent native commands when the shell differs.
|
||||
Use a unique task-owned scratch directory. Select the versioned archive matching the acceptance host, and verify it against the sealed manifest before extraction. Before desktop completion, use the Linux acceptance environment rather than rebuilding for the local desktop. Do not substitute a different build when a matching artifact is unavailable; report that acceptance gap. The shell example below assumes the verified executable is at `RUSTFS_BINARY` (`rustfs.exe` on Windows), `SCRATCH` is the task directory, and `S3_PORT` and `CONSOLE_PORT` are distinct available ports. Use equivalent native commands when the shell differs.
|
||||
|
||||
Bind both listeners to loopback. Choose task-only credentials, set loopback proxy exclusions for the server/client processes, and record the server PID. A port probe does not reserve a port; retry startup with fresh ports on a bind conflict rather than connecting to an existing service.
|
||||
|
||||
@@ -24,16 +24,16 @@ PREVIEW_SERVER_PID=$!
|
||||
|
||||
Checks (all must pass):
|
||||
|
||||
- `"$RUSTFS_BINARY" --version` reports the preview tag name and the short SHA of `PREVIEW_HASH`. Reporting `<target>` without the `-preview.N` suffix means the build did not embed the tag — treat as FAIL and investigate before proceeding.
|
||||
- `"$RUSTFS_BINARY" --version` reports the exact final `<target>` and full `CANDIDATE_SHA`. A preview suffix, another version or another source is a failure.
|
||||
- Wait with a bounded readiness deadline while checking the recorded server process remains alive; `curl --noproxy '*' -fsS "$S3_ENDPOINT/health/ready"` must return ready from this process's listener.
|
||||
- Startup log shows the embedded console being served (this was the regression that `fix(release): require embedded console assets` guards).
|
||||
- Open `CONSOLE_ENDPOINT` in the browser and log in with the task credentials. The dashboard must render without JS console errors; create a bucket, upload a file, download it back (byte-identical), and delete the object and bucket. Keep the server running for Phase 5.
|
||||
|
||||
## Phase 5 — Validate with the latest rc client
|
||||
## Phase 5 — Validate with the pinned rc client
|
||||
|
||||
`rc` is the RustFS CLI client from <https://github.com/rustfs/cli>.
|
||||
|
||||
- Resolve the latest release with `gh api repos/rustfs/cli/releases/latest --jq .tag_name`; verify the local client's version or download and verify a host-compatible client into the task directory. Set `RC_BIN` to that executable and record its version/digest. Avoid changing the user's installed client for this check.
|
||||
- Read the CLI release and asset digests from `inputs.json`; verify the local client's version or download a host-compatible asset from that exact release into the task directory. If the host is a desktop, resolve its asset within the same pinned CLI release and record the digest; do not independently follow `latest`. Set `RC_BIN` to that executable and record its version/digest. Avoid changing the user's installed client for this check.
|
||||
- Inspect the client's current help for an isolated configuration option and use it when supported. Otherwise choose a unique `PREVIEW_ALIAS` and confirm it is absent before creating it; never overwrite an existing alias. Use task-owned bucket/user names and input/output files. The names below are safe only inside the newly started isolated server.
|
||||
- Point the client at `S3_ENDPOINT` with loopback proxy exclusions and run the command matrix, recording PASS/FAIL per command:
|
||||
|
||||
@@ -60,3 +60,9 @@ cmp <local-file> "$SCRATCH/out"
|
||||
```
|
||||
|
||||
- Any FAIL blocks acceptance. On success or failure, stop only the recorded task process and remove only task-created aliases/data after retaining the needed failure evidence. Do not erase an existing client configuration or unrelated server data.
|
||||
|
||||
## Field acceptance and desktop completion
|
||||
|
||||
Before accepting the Linux manifest, record Scanner/Heal, multi-pool, memory, major features, high-concurrency 503, performance and previous-version integrity/upgrade results in backlog. Preserve the release-specific agreed limits and baseline; a PASS label without artifact identities or test results does not satisfy acceptance.
|
||||
|
||||
Dispatch `release-complete.yml` only after that report passes, using the exact Linux manifest digest and backlog URL. Require native macOS/Windows checks, package extraction, image startup/scans, chart tests and complete certification to pass. Complete any desktop Console/client checks applicable to the release against these same archives. Record the completion run and final manifest SHA-256 before entering the parent confirmation gate. Partial Linux acceptance does not authorize final publication.
|
||||
|
||||
@@ -1,25 +1,9 @@
|
||||
# Preview Release Verification
|
||||
# Linux candidate verification
|
||||
|
||||
Read for Phase 3 after the candidate passed exact-SHA CI. The [parent release invariants](../SKILL.md) apply.
|
||||
Read for Phase 3 after the source passed exact-SHA CI. The [parent invariants](../SKILL.md) apply; commands and recovery live in the [pipeline runbook](../../../../docs/operations/release-pipeline.md).
|
||||
|
||||
## Preview channels
|
||||
Dispatch `release-candidate.yml` on main with the confirmed final version and `CANDIDATE_SHA`. Record the run ID, frozen `inputs.json`, every Linux archive digest and `linux-manifest.json` SHA-256. Require the four Linux builds, native startup/S3 checks, previous-release upgrade matrix and Linux seal to succeed. Missing, skipped, cancelled or failed required work is incomplete acceptance.
|
||||
|
||||
- Use `<target>-preview.N` for every target, e.g. `1.0.0-beta.10-preview.3` or `1.1.0-preview.1`.
|
||||
- The canonical suffix is exactly `-preview.<digits>`. `build.yml` recognizes it before alpha/beta/rc classification and routes it to the preview-only path; any other tag containing `-preview` fails closed instead of being treated as a release.
|
||||
- A preview Release MUST be published with `isPrerelease=true` and `isLatest=false`. Any `*-latest` preview asset or preview-triggered `latest.json`, R2, Helm, or moving Docker channel tag is a pipeline failure. Docker images may use the exact preview version tag, including its existing variant suffixes.
|
||||
- Preview Releases are cleaned up by the `cleanup-preview-releases` job after `publish-release` succeeds for the deliverable tag. It deletes every Release whose tag is exactly `<target>-preview.<digits>` and never passes `--cleanup-tag`, so the tags survive.
|
||||
Verify the frozen Console identity equals the one accepted in Phase 0. Every matrix job consumes that same digest and exact toolchain. The binary must report the final product version and source commit; its candidate ID is separate. No public tag, Release, registry tag, R2 release path, Helm repository or latest pointer should change during this stage.
|
||||
|
||||
## Phase 3 — CI and preview Release verification
|
||||
|
||||
- Find and watch the tag build: `gh run list --workflow build.yml --branch "<preview-tag>" --limit 1` then `gh run watch <run-id>`. Every build matrix target must succeed (linux x86_64/aarch64 × musl/gnu, macos-aarch64, windows-x86_64).
|
||||
- Confirm the preview tag resolves to the Phase 2 CI-validated `PREVIEW_HASH`. If the release branch advances during acceptance, repeat Phases 2–5 with a new preview tag.
|
||||
- Confirm the Release publication jobs (`create-release`, `upload-release-assets`, and `publish-release`) succeed while `update-latest-version` is skipped.
|
||||
- Record the inputs actually consumed by each build job: source SHA and the embedded Console release tag, asset URL, and verified SHA256 from the download logs. The workflow reads Console `releases/latest` separately for each build; confirm the matrix used the Console asset accepted in Phase 0. A missing identity or differing asset blocks acceptance until resolved. Retain available toolchain and build provenance with the artifact checksums; the same source SHA alone does not establish equivalent builds.
|
||||
- Verify `gh release view "<preview-tag>" --json isPrerelease,assets,url`: `isPrerelease` must be `true`, and the Release must contain all 6 versioned platform zips, checksums, SBOM, and provenance with no `-latest` assets. Confirm `gh api repos/{owner}/{repo}/releases/latest --jq .tag_name` does not return `<preview-tag>`.
|
||||
- Record `PREVIOUS_DELIVERABLE`, selected from published Releases by `publishedAt` after excluding the current tag and every `-preview.N` tag. Verify `gh release view "<preview-tag>" --json body --jq .body` contains `## What's Changed` and, when `PREVIOUS_DELIVERABLE` exists, `**Full Changelog**: https://github.com/rustfs/rustfs/compare/<PREVIOUS_DELIVERABLE>...<preview-tag>`. For a repository with no previous deliverable, verify a Full Changelog link exists and record the GitHub baseline fallback.
|
||||
- Confirm Helm is skipped. If preview Docker images are published, confirm they use only exact preview tags (and variant suffixes), with no `latest`, `alpha`, `beta`, or `rc` channel updates. Preview validation covers the built RustFS binaries, embedded console, and rc compatibility; it does not authorize advancing any installation default.
|
||||
|
||||
## Release notes
|
||||
|
||||
- When a previous deliverable exists, GitHub Release notes for the preview and final tags MUST use it as their shared comparison baseline: the most recently published non-preview Release before the target. Internal `-preview.N` Releases are explicitly excluded from that selection, even when they point at the same commit as the final tag — cleanup runs after the notes are generated, so the preview Release is still present and would otherwise be picked as the baseline. If no previous deliverable exists, omit `previous_tag_name` and record that GitHub's default baseline fallback was used.
|
||||
- Generated Release notes carry a workflow-management marker so retries can repair them. Before manually curating a generated body, remove that marker; unmarked non-placeholder notes are preserved by later workflow runs.
|
||||
A green Linux run is ready for field acceptance, not publication. The partial certification must have `publish_allowed: false`. Retain the exact Linux manifest for the next stage; a new source/build-input candidate must not inherit the old approval. A newer unrelated main commit does not invalidate this evidence.
|
||||
|
||||
@@ -16,22 +16,13 @@ Read for Phases 1–2. The [parent release invariants](../SKILL.md) and Console
|
||||
- Only when neither the release branch, an unfinished preview tag, nor current main already has `<target>`: if main is at a later version, select and verify a historical commit containing `<target>` rather than downgrading main; block if none exists. Otherwise invoke the `rustfs-release-version-bump` skill with stage `prepare`, the final `<target>` (NOT a preview version), and full GitHub flow (commit/push/PR).
|
||||
- If a bump PR was needed, get it merged into main and record its merge commit. For a new release branch when main already has `<target>`, identify the main commit that introduced those versions. Phase 2 chooses a release candidate containing that commit; an existing release branch keeps its own history and a newer main HEAD is not required.
|
||||
|
||||
Do not wait for the moving latest main HEAD to become green. The release-branch CI in Phase 2 validates the chosen commit before a preview tag is created.
|
||||
Do not wait for the moving latest main HEAD to become green. The release-branch CI in Phase 2 validates the chosen commit before a candidate build is dispatched.
|
||||
|
||||
## Phase 2 — Pin and validate the release candidate, then publish the preview tag
|
||||
## Phase 2 — Pin and validate the release source
|
||||
|
||||
- Use `release/<target>`. On the first attempt, create it from a selected main commit containing the version bump and intended changes; it need not be the latest main HEAD. On a restart, retain the existing branch and backport only reviewed fixes from main. Verify the branch's Cargo versions still equal `<target>`. If adopting an existing preview, run this branch CI before continuing its acceptance; reuse the tag only when it points to the validated SHA.
|
||||
- The current `ci.yml` runs automatically on main pushes, not release-branch pushes. Dispatch it explicitly with `gh workflow run ci.yml --ref "release/<target>"`. Locate that dispatch run, verify its `head_sha` equals the release branch's remote HEAD, and require the full expected CI job set to pass, including `Quick Checks`, `Test and Lint`, and `End-to-End Tests (full merge gate)`. Treat a missing or skipped required lane as incomplete. A successful PR check or CI run for a different SHA does not qualify. If the branch advances or a run is cancelled, validate the new SHA again. Do not tag a candidate with failing or incomplete CI.
|
||||
- Use `release/<target>`. On the first attempt, create it from a selected main commit containing the version bump and intended changes; it need not be the latest main HEAD. On a restart, retain the existing branch and backport only reviewed fixes from main. Verify the branch's Cargo versions still equal `<target>`. A legacy preview can anchor source selection, but its tag and old artifacts do not constitute a candidate in the new protocol; freeze and verify a new candidate after the release recipe is aligned.
|
||||
- The current `ci.yml` runs automatically on main pushes, not release-branch pushes. Dispatch it explicitly with `gh workflow run ci.yml --ref "release/<target>"`. Locate that dispatch run, verify its `head_sha` equals the release branch's remote HEAD, and require the full expected CI job set to pass, including `Quick Checks`, `Test and Lint`, and `End-to-End Tests (full merge gate)`. Treat a missing or skipped required lane as incomplete. A successful PR check or CI run for a different SHA does not qualify. If the branch advances or a run is cancelled, validate the new SHA again. Do not dispatch a candidate with failing or incomplete CI.
|
||||
- Prefer protecting `release/*` against force pushes and deletion and requiring reviewed backport PRs. Check live rulesets rather than assuming the main ruleset covers release branches; report a missing release-branch rule as a gap.
|
||||
- Set `PREVIEW_HASH` to the CI-validated release-branch commit and recheck the remote branch HEAD immediately before tagging. Report the branch, commit, and CI run URL; do not derive `PREVIEW_HASH` from a later `origin/main` fetch. When adopting an existing preview tag at this SHA, skip tag creation and continue Phase 3.
|
||||
|
||||
```bash
|
||||
git tag -a "<preview-tag>" -m "Release <preview-tag>" "$PREVIEW_HASH"
|
||||
git push origin "<preview-tag>"
|
||||
```
|
||||
|
||||
Pushing the tag triggers `.github/workflows/build.yml` ("Build and Release"); `docker.yml` chains off it via `workflow_run`.
|
||||
|
||||
The preview run builds versioned artifacts and publishes them in a GitHub prerelease. Docker may publish exact preview image tags. Latest-channel, R2, and Helm publication must be skipped; installation references and the website announcement stay unchanged.
|
||||
|
||||
On a restart after a backport, re-run release-branch CI before setting the new `PREVIEW_HASH`. Use the next unused preview iteration when the validated commit differs from an existing preview tag; never move an existing tag.
|
||||
- Set `CANDIDATE_SHA` to the CI-validated release-branch commit. Record the branch, exact commit and CI run URL. The new candidate protocol starts from a commit whose release recipe matches the reviewed main dispatcher; backport the recipe before freezing an older release branch if needed.
|
||||
- Dispatch the Linux candidate as described in [candidate verification](preview-release.md). Do not create a preview or final tag here. The final executable already contains `<target>`; publication later promotes those same bytes.
|
||||
- After a source or build-input change, run the affected source checks and create a new candidate. Reuse the frozen candidate after an infrastructure failure. Do not restart acceptance for unrelated main advances.
|
||||
|
||||
@@ -4,18 +4,18 @@ description: Prepare an exact RustFS Cargo version or align installation referen
|
||||
---
|
||||
# Release Version Files
|
||||
|
||||
Require an exact target; ask if missing or ambiguous. Reject targets containing `-preview`: those are tag-only. Infer `post-release` only from an explicit request or parent workflow; otherwise use `prepare`. Delivery defaults to local edit/verify; commit, push, PR, and merge follow existing authorization and [Git rules](../../references/pull-requests.md).
|
||||
Require an exact target; ask if missing or ambiguous. Reject targets containing `-preview`: those identify candidates, not product versions or new public tags. Infer `post-release` only from an explicit request or parent workflow; otherwise use `prepare`. Delivery defaults to local edit/verify; commit, push, PR, and merge follow existing authorization and [Git rules](../../references/pull-requests.md).
|
||||
|
||||
| Stage | Files and constraints |
|
||||
|---|---|
|
||||
| `prepare` | Only `Cargo.toml` and `Cargo.lock`: workspace package, internal dependency, and workspace member versions become `<target>`. External dependencies stay unchanged. |
|
||||
| `post-release` | Only `README.md`, `README_ZH.md`, `flake.nix`, `helm/rustfs/Chart.yaml`, and `rustfs.spec`. Cargo may already target the next release; leave it untouched. |
|
||||
|
||||
Before installation edits, verify the exact non-preview Release, downloadable assets/source archive, pullable `rustfs/rustfs:<target>` image manifest, and published Helm chart. Block on missing artifacts. Do not downgrade defaults if a newer deliverable superseded the target. Preview preparation retains the previous published installation references; follow-up commits never replace either validated tag.
|
||||
Before installation edits, verify the exact non-preview Release, downloadable assets/source archive, pullable `rustfs/rustfs:<target>` image manifest, and published Helm chart. Block on missing artifacts. Do not downgrade defaults if a newer deliverable superseded the target. Candidate preparation retains the previous published installation references; follow-up commits never replace validated tags or sealed artifacts.
|
||||
|
||||
Packaging policy:
|
||||
|
||||
- Docker image tags have no `v` prefix. Derive chart/app versions with `scripts/helm_chart_version.sh <target>` (`beta.N -> 0.N.0`; other versions unchanged). Helm CI derives versions from the final tag, so no early chart bump is required.
|
||||
- Docker image tags have no `v` prefix. Derive chart/app versions with `scripts/helm_chart_version.sh <target>`; both equal the final product SemVer, including prerelease suffixes. Helm CI uses the frozen candidate version before sealing, so no early installation chart bump is required. Previously published chart versions remain unchanged.
|
||||
- RPM `Version` is numeric; `Release` is the prerelease suffix or `1` for stable. Expanded `Source0` and the unpack directory must resolve to the exact target including its suffix. Add the changelog entry using current git identity, date, and target; never copy an example identity/date.
|
||||
- Changing these mappings requires explicit confirmation.
|
||||
|
||||
|
||||
@@ -1,2 +1,2 @@
|
||||
sha256-darwin=219ac11e720b3f54dfee4d5d5e95171785ba159b2fe01db3037a8f5061c1ad32
|
||||
sha256-linux=2d34698cc8ec52d5b6c37ac612919a1d21f36cd56a1af539a7522a4eea518e26
|
||||
sha256-darwin=9901a4950099b94d3d2618524d540f1ed4bf6023044903886f98b6c4bf5f30fb
|
||||
sha256-linux=09fd5f48f5556c9fc576357222dccd7ebb0b90b790798f1834db2ec5d67b4952
|
||||
|
||||
@@ -22,6 +22,14 @@ TEST_THREADS ?= 1
|
||||
# differ from CI and .config/nextest.toml test-groups will NOT apply.
|
||||
|
||||
.PHONY: script-tests
|
||||
.PHONY: release-package-tests
|
||||
release-package-tests: ## Run release package integration on Linux with fpm, dpkg and rpm installed
|
||||
$(RUSTFS_PYTHON_BIN) ./scripts/test_release_packages.py
|
||||
|
||||
.PHONY: release-image-tests
|
||||
release-image-tests: ## Run OCI integration with Docker, Go and skopeo; set RELEASE_IMAGE_VARIANT and RELEASE_IMAGE_OUTPUT
|
||||
$(RUSTFS_PYTHON_BIN) ./scripts/test_release_images.py
|
||||
|
||||
script-tests: ## Run shell script tests
|
||||
@echo "Running script tests..."
|
||||
trap 'status=$$?; printf "ERROR: script-tests failed (exit %s): %s\n" "$$status" "$$BASH_COMMAND" >&2; exit "$$status"' ERR
|
||||
@@ -54,8 +62,10 @@ script-tests: ## Run shell script tests
|
||||
./scripts/check_embedded_secrets.sh --self-test
|
||||
$(RUSTFS_PYTHON_BIN) ./scripts/check_test_wiring.py --self-test
|
||||
$(RUSTFS_PYTHON_BIN) ./scripts/test_e2e_binary.py
|
||||
$(RUSTFS_PYTHON_BIN) ./scripts/test_download_release_asset.py
|
||||
$(RUSTFS_PYTHON_BIN) ./scripts/test_inspect_btf_prototypes.py
|
||||
$(RUSTFS_PYTHON_BIN) ./scripts/test_release_build_workflow.py
|
||||
$(RUSTFS_PYTHON_BIN) -m unittest discover -s scripts/release -p "test_*.py"
|
||||
$(RUSTFS_PYTHON_BIN) ./scripts/test_release_glibc.py
|
||||
$(RUSTFS_PYTHON_BIN) ./scripts/test_upgrade_workflow.py
|
||||
$(RUSTFS_PYTHON_BIN) ./scripts/test_migration_gate_evidence.py
|
||||
|
||||
@@ -1,44 +1,93 @@
|
||||
{
|
||||
"schema_version": 1,
|
||||
"plan_id": "rustfs-release-validation",
|
||||
"description": "Versioned certification plan for every public RustFS release exit (rustfs/backlog#2662, R01). preview, prerelease and stable all require this explicit plan. Scripts/release/release_gate.py fail-closes publication on any missing, duplicate, stale, substituted or non-PASS required item.",
|
||||
"release_types": ["preview", "prerelease", "stable"],
|
||||
"description": "Certification for immutable candidates. All shipped platforms require native identity and startup; stable and prerelease require the pinned upgrade matrix. Linux-only acceptance never authorizes publication.",
|
||||
"release_types": [
|
||||
"preview",
|
||||
"prerelease",
|
||||
"stable"
|
||||
],
|
||||
"targets": [
|
||||
{"os": "linux", "arch": "x86_64", "abi": "musl"},
|
||||
{"os": "linux", "arch": "aarch64", "abi": "musl"},
|
||||
{"os": "linux", "arch": "x86_64", "abi": "gnu"},
|
||||
{"os": "linux", "arch": "aarch64", "abi": "gnu"},
|
||||
{"os": "macos", "arch": "aarch64", "abi": "none"},
|
||||
{"os": "windows", "arch": "x86_64", "abi": "msvc"}
|
||||
{
|
||||
"os": "linux",
|
||||
"arch": "x86_64",
|
||||
"abi": "musl"
|
||||
},
|
||||
{
|
||||
"os": "linux",
|
||||
"arch": "aarch64",
|
||||
"abi": "musl"
|
||||
},
|
||||
{
|
||||
"os": "linux",
|
||||
"arch": "x86_64",
|
||||
"abi": "gnu"
|
||||
},
|
||||
{
|
||||
"os": "linux",
|
||||
"arch": "aarch64",
|
||||
"abi": "gnu"
|
||||
},
|
||||
{
|
||||
"os": "macos",
|
||||
"arch": "aarch64",
|
||||
"abi": "none"
|
||||
},
|
||||
{
|
||||
"os": "windows",
|
||||
"arch": "x86_64",
|
||||
"abi": "msvc"
|
||||
}
|
||||
],
|
||||
"targets_basis": "Platform matrix of .github/workflows/build.yml (prepare-platform-matrix). gnu and macOS targets build with the pyroscope feature; musl and Windows targets build with no extra features.",
|
||||
"features": ["pyroscope"],
|
||||
"trusted_workflows": [
|
||||
{"repository": "rustfs/s3chaos", "workflow_id": "release-gate.yml"}
|
||||
"features": [
|
||||
"pyroscope"
|
||||
],
|
||||
"trusted_workflows_basis": "The shared evidence envelope producer lives in rustfs/s3chaos (G01, fault/release_gate.rs) and its plan/live split workflow is .github/workflows/release-gate.yml there. Only results from these identities may enter a release certification; PR-uploaded JSON never gains release authority. R02 must extend this list when additional verifier workflows land, never widen it to untrusted sources.",
|
||||
"trusted_workflows": [
|
||||
{
|
||||
"repository": "rustfs/rustfs",
|
||||
"workflow_id": "release-candidate.yml"
|
||||
},
|
||||
{
|
||||
"repository": "rustfs/rustfs",
|
||||
"workflow_id": "release-complete.yml"
|
||||
}
|
||||
],
|
||||
"trusted_workflows_basis": "Receipts are produced by the reviewed main workflows and fetched by exact run ID with GitHub artifact digest verification. Candidate inputs freeze the harness commit separately from product source.",
|
||||
"lanes": [
|
||||
{
|
||||
"lane": "identity",
|
||||
"harness_repository": "rustfs/s3chaos",
|
||||
"harness_sha": "9a48f55c1d6d58ce63a8fb1ea7317e024f32f11d",
|
||||
"harness_basis": "Fixed G01 evidence-envelope producer revision (rustfs/s3chaos PR #102 head recorded in rustfs/backlog#2658). R02 re-pins this when the live verifier ships.",
|
||||
"harness_repository": "rustfs/rustfs",
|
||||
"harness_sha": "candidate.workflow_sha",
|
||||
"harness_basis": "The exact workflow commit is frozen in candidate inputs; no mutable branch is accepted as a harness identity.",
|
||||
"timeout_seconds": 1800,
|
||||
"environment_capabilities": ["process-exec"],
|
||||
"cases": ["artifact-identity", "startup"],
|
||||
"environment_capabilities": [
|
||||
"process-exec"
|
||||
],
|
||||
"cases": [
|
||||
"artifact-identity",
|
||||
"startup"
|
||||
],
|
||||
"applies_to": {
|
||||
"release_types": ["preview", "prerelease", "stable"],
|
||||
"release_types": [
|
||||
"preview",
|
||||
"prerelease",
|
||||
"stable"
|
||||
],
|
||||
"platforms": "all"
|
||||
},
|
||||
"applies_basis": "Artifact identity and startup checks are mandatory for every release type, preview included, on every shipped target (rustfs/backlog#2662 frozen design item 5)."
|
||||
},
|
||||
{
|
||||
"lane": "upgrade",
|
||||
"harness_repository": "rustfs/s3chaos",
|
||||
"harness_sha": "9a48f55c1d6d58ce63a8fb1ea7317e024f32f11d",
|
||||
"harness_basis": "Same fixed envelope producer as the identity lane; the workflow SHA and the product build SHA are bound separately and need not be equal.",
|
||||
"harness_repository": "rustfs/rustfs",
|
||||
"harness_sha": "candidate.workflow_sha",
|
||||
"harness_basis": "The exact workflow commit is frozen in candidate inputs; no mutable branch is accepted as a harness identity.",
|
||||
"timeout_seconds": 5400,
|
||||
"environment_capabilities": ["process-exec", "previous-release-artifact"],
|
||||
"environment_capabilities": [
|
||||
"process-exec",
|
||||
"previous-release-artifact"
|
||||
],
|
||||
"cases": [
|
||||
"direct-upgrade",
|
||||
"mixed-version-upgrade",
|
||||
@@ -49,8 +98,17 @@
|
||||
],
|
||||
"cases_basis": "Case IDs of the pinned previous-release matrix in .github/workflows/e2e-upgrade.yml (source.version/asset/sha256 and the case list mapped there).",
|
||||
"applies_to": {
|
||||
"release_types": ["prerelease", "stable"],
|
||||
"platforms": [{"os": "linux", "arch": "x86_64", "abi": "gnu"}]
|
||||
"release_types": [
|
||||
"prerelease",
|
||||
"stable"
|
||||
],
|
||||
"platforms": [
|
||||
{
|
||||
"os": "linux",
|
||||
"arch": "x86_64",
|
||||
"abi": "gnu"
|
||||
}
|
||||
]
|
||||
},
|
||||
"applies_basis": "preview releases use the pre-declared smaller set (identity lane only). The upgrade lane is only exercised on linux-x86_64-gnu, the platform of the pinned source assets."
|
||||
}
|
||||
|
||||
@@ -28,11 +28,12 @@ runs:
|
||||
|
||||
- name: Install actionlint
|
||||
shell: bash
|
||||
env:
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
ASSET_DOWNLOADER: ${{ github.action_path }}/../setup/download_release_asset.py
|
||||
run: |
|
||||
actionlint_dir="$(mktemp -d "${RUNNER_TEMP}/actionlint.XXXXXX")"
|
||||
curl --fail --location --silent --show-error \
|
||||
--output "$actionlint_dir/actionlint.tar.gz" \
|
||||
https://github.com/rhysd/actionlint/releases/download/v1.7.12/actionlint_1.7.12_linux_amd64.tar.gz
|
||||
python3 "$ASSET_DOWNLOADER" --repository rhysd/actionlint --tag v1.7.12 --asset actionlint_1.7.12_linux_amd64.tar.gz --output "$actionlint_dir/actionlint.tar.gz" --sha256 8aca8db96f1b94770f1b0d72b6dddcb1ebb8123cb3712530b08cc387b349a3d8
|
||||
echo "8aca8db96f1b94770f1b0d72b6dddcb1ebb8123cb3712530b08cc387b349a3d8 $actionlint_dir/actionlint.tar.gz" | sha256sum --check --status
|
||||
tar -xzf "$actionlint_dir/actionlint.tar.gz" -C "$actionlint_dir" actionlint
|
||||
rm "$actionlint_dir/actionlint.tar.gz"
|
||||
|
||||
@@ -66,10 +66,7 @@ inputs:
|
||||
runs:
|
||||
using: "composite"
|
||||
steps:
|
||||
# protobuf-compiler is deliberately absent: the setup-protoc step below
|
||||
# installs 35.1 into the tool cache and prepends it to PATH, so the apt
|
||||
# build (older, and never version-matched) was shadowed on every run and
|
||||
# simply never used.
|
||||
# Use the pinned code generators rather than Ubuntu's older packages.
|
||||
- name: Install system dependencies (Ubuntu)
|
||||
if: runner.os == 'Linux'
|
||||
shell: bash
|
||||
@@ -79,7 +76,9 @@ runs:
|
||||
build-essential \
|
||||
pkg-config \
|
||||
libssl-dev \
|
||||
ripgrep
|
||||
ripgrep \
|
||||
python3 \
|
||||
gh
|
||||
|
||||
# musl-gcc is needed by the native musl release leg, and zip/unzip by the
|
||||
# release packaging steps. No CI test lane touches any of them.
|
||||
@@ -88,13 +87,32 @@ runs:
|
||||
shell: bash
|
||||
run: sudo apt-get install -y musl-tools zip unzip
|
||||
|
||||
- name: Install pinned code generators through GitHub API (Linux x64)
|
||||
if: runner.os == 'Linux' && runner.arch == 'X64'
|
||||
shell: bash
|
||||
env:
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
ASSET_DOWNLOADER: ${{ github.action_path }}/download_release_asset.py
|
||||
run: |
|
||||
python3 "$ASSET_DOWNLOADER" --repository protocolbuffers/protobuf --tag v35.1 --asset protoc-35.1-linux-x86_64.zip --output "$RUNNER_TEMP/protoc.zip"
|
||||
python3 "$ASSET_DOWNLOADER" --repository google/flatbuffers --tag v25.12.19 --asset 'Linux.flatc.binary.g++-13.zip' --output "$RUNNER_TEMP/flatc.zip"
|
||||
python3 -m zipfile -e "$RUNNER_TEMP/protoc.zip" "$RUNNER_TEMP/protoc"
|
||||
python3 -m zipfile -e "$RUNNER_TEMP/flatc.zip" "$RUNNER_TEMP/flatc"
|
||||
chmod +x "$RUNNER_TEMP/protoc/bin/protoc" "$RUNNER_TEMP/flatc/flatc"
|
||||
"$RUNNER_TEMP/protoc/bin/protoc" --version
|
||||
"$RUNNER_TEMP/flatc/flatc" --version
|
||||
echo "$RUNNER_TEMP/protoc/bin" >> "$GITHUB_PATH"
|
||||
echo "$RUNNER_TEMP/flatc" >> "$GITHUB_PATH"
|
||||
|
||||
- name: Install protoc
|
||||
if: runner.os != 'Linux' || runner.arch != 'X64'
|
||||
uses: rustfs/setup-protoc@a3705324d8f9bf5b6c3573fb6cf8ae421db55dd6 # v3.0.1
|
||||
with:
|
||||
version: "35.1"
|
||||
repo-token: ${{ github.token }}
|
||||
|
||||
- name: Install flatc
|
||||
if: runner.os != 'Linux' || runner.arch != 'X64'
|
||||
uses: Nugine/setup-flatc@698800de72a96bfb22cf60431dc21a2ff9a7e07b # v1
|
||||
with:
|
||||
version: "25.12.19"
|
||||
|
||||
@@ -0,0 +1,84 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Download a named GitHub release asset through its API and verify its digest."""
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
from urllib.parse import quote
|
||||
|
||||
|
||||
def download(repository, tag, name, output, expected_sha256=None):
|
||||
if not re.fullmatch(r"[A-Za-z0-9][A-Za-z0-9_.-]*/[A-Za-z0-9][A-Za-z0-9_.-]*", repository):
|
||||
raise ValueError("invalid release repository")
|
||||
if not tag or not name or "/" in name or "\\" in name:
|
||||
raise ValueError("invalid release tag or asset name")
|
||||
if expected_sha256 is not None and not re.fullmatch(r"[0-9a-f]{64}", expected_sha256):
|
||||
raise ValueError("invalid pinned asset digest")
|
||||
result = subprocess.run(
|
||||
["gh", "api", "--hostname", "github.com", f"repos/{repository}/releases/tags/{quote(tag, safe='')}"],
|
||||
check=True, capture_output=True, text=True, timeout=60,
|
||||
)
|
||||
release = json.loads(result.stdout)
|
||||
if not isinstance(release, dict) or release.get("tag_name") != tag:
|
||||
raise ValueError("release metadata does not match the requested tag")
|
||||
assets = release.get("assets")
|
||||
if not isinstance(assets, list) or any(not isinstance(asset, dict) for asset in assets):
|
||||
raise ValueError("invalid release asset inventory")
|
||||
matches = [asset for asset in assets if asset.get("name") == name]
|
||||
if len(matches) != 1:
|
||||
raise ValueError("release must contain exactly one named asset")
|
||||
asset = matches[0]
|
||||
asset_id, size, digest = asset.get("id"), asset.get("size"), asset.get("digest")
|
||||
if type(asset_id) is not int or asset_id <= 0 or type(size) is not int or not 0 < size <= 512 * 1024 * 1024:
|
||||
raise ValueError("invalid release asset identity or size")
|
||||
if not isinstance(digest, str) or not re.fullmatch(r"sha256:[0-9a-f]{64}", digest):
|
||||
raise ValueError("release asset has no valid SHA-256 digest")
|
||||
sha256 = digest.removeprefix("sha256:")
|
||||
if expected_sha256 is not None and expected_sha256 != sha256:
|
||||
raise ValueError("release digest differs from the pinned digest")
|
||||
output = Path(output)
|
||||
output.parent.mkdir(parents=True, exist_ok=True)
|
||||
temporary = None
|
||||
try:
|
||||
with tempfile.NamedTemporaryFile(dir=output.parent, prefix=".release-asset-", delete=False) as stream:
|
||||
temporary = Path(stream.name)
|
||||
# gh authenticates to the API; its HTTP client strips credentials on
|
||||
# cross-host redirects. No browser URL or token is supplied in argv.
|
||||
subprocess.run(
|
||||
["gh", "api", "--hostname", "github.com", "--header", "Accept: application/octet-stream", f"repos/{repository}/releases/assets/{asset_id}"],
|
||||
check=True, stdout=stream, timeout=300,
|
||||
)
|
||||
with temporary.open("rb") as stream:
|
||||
actual = hashlib.file_digest(stream, "sha256").hexdigest()
|
||||
if temporary.stat().st_size != size or actual != sha256:
|
||||
raise ValueError("downloaded release asset failed size or SHA-256 verification")
|
||||
temporary.replace(output)
|
||||
print(f"Verified release asset: {repository} {tag} {name} sha256={actual}")
|
||||
finally:
|
||||
if temporary is not None:
|
||||
temporary.unlink(missing_ok=True)
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--repository", required=True)
|
||||
parser.add_argument("--tag", required=True)
|
||||
parser.add_argument("--asset", required=True)
|
||||
parser.add_argument("--output", required=True, type=Path)
|
||||
parser.add_argument("--sha256")
|
||||
args = parser.parse_args()
|
||||
try:
|
||||
download(args.repository, args.tag, args.asset, args.output, args.sha256)
|
||||
except (ValueError, KeyError, TypeError, json.JSONDecodeError, subprocess.SubprocessError, OSError) as error:
|
||||
print(f"Release asset download failed: {error}", file=sys.stderr)
|
||||
return 1
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -350,10 +350,10 @@
|
||||
"roles": [
|
||||
"periodic",
|
||||
"manual",
|
||||
"release"
|
||||
"reusable"
|
||||
],
|
||||
"events": [
|
||||
"push",
|
||||
"workflow_call",
|
||||
"schedule",
|
||||
"workflow_dispatch"
|
||||
],
|
||||
@@ -367,7 +367,7 @@
|
||||
"monitor": "workflow",
|
||||
"max_age_hours": 36,
|
||||
"result_owner": "build.yml",
|
||||
"disposition": "retained independent entry",
|
||||
"disposition": "Builds artifacts only; public release writes belong to release-publish.yml.",
|
||||
"failure_owner": "workflow",
|
||||
"origin": "baseline",
|
||||
"alert_contract": "scheduled-validation"
|
||||
@@ -745,14 +745,13 @@
|
||||
},
|
||||
{
|
||||
"workflow": ".github/workflows/docker.yml",
|
||||
"name": "Docker Images",
|
||||
"name": "Build candidate images",
|
||||
"roles": [
|
||||
"manual",
|
||||
"release"
|
||||
"release",
|
||||
"reusable"
|
||||
],
|
||||
"events": [
|
||||
"workflow_run",
|
||||
"workflow_dispatch"
|
||||
"workflow_call"
|
||||
],
|
||||
"cadence": "event",
|
||||
"timezone": "UTC",
|
||||
@@ -761,7 +760,7 @@
|
||||
"policy": "active",
|
||||
"monitor": "none",
|
||||
"result_owner": "docker.yml",
|
||||
"disposition": "retained independent entry",
|
||||
"disposition": "Builds artifacts only; public release writes belong to release-publish.yml.",
|
||||
"monitor_reason": "Event-driven or operator-requested work has no periodic freshness obligation.",
|
||||
"origin": "baseline"
|
||||
},
|
||||
@@ -954,14 +953,13 @@
|
||||
},
|
||||
{
|
||||
"workflow": ".github/workflows/helm-package.yml",
|
||||
"name": "Publish helm chart to artifacthub",
|
||||
"name": "Build candidate Helm chart",
|
||||
"roles": [
|
||||
"manual",
|
||||
"release"
|
||||
"release",
|
||||
"reusable"
|
||||
],
|
||||
"events": [
|
||||
"workflow_run",
|
||||
"workflow_dispatch"
|
||||
"workflow_call"
|
||||
],
|
||||
"cadence": "event",
|
||||
"timezone": "UTC",
|
||||
@@ -970,7 +968,7 @@
|
||||
"policy": "active",
|
||||
"monitor": "none",
|
||||
"result_owner": "helm-package.yml",
|
||||
"disposition": "retained independent entry",
|
||||
"disposition": "Builds artifacts only; public release writes belong to release-publish.yml.",
|
||||
"monitor_reason": "Event-driven or operator-requested work has no periodic freshness obligation.",
|
||||
"origin": "baseline"
|
||||
},
|
||||
@@ -1182,14 +1180,13 @@
|
||||
},
|
||||
{
|
||||
"workflow": ".github/workflows/package.yml",
|
||||
"name": "Package DEB/RPM",
|
||||
"name": "Build candidate packages",
|
||||
"roles": [
|
||||
"manual",
|
||||
"release"
|
||||
"release",
|
||||
"reusable"
|
||||
],
|
||||
"events": [
|
||||
"workflow_run",
|
||||
"workflow_dispatch"
|
||||
"workflow_call"
|
||||
],
|
||||
"cadence": "event",
|
||||
"timezone": "UTC",
|
||||
@@ -1198,7 +1195,7 @@
|
||||
"policy": "active",
|
||||
"monitor": "none",
|
||||
"result_owner": "package.yml",
|
||||
"disposition": "retained independent entry",
|
||||
"disposition": "Builds artifacts only; public release writes belong to release-publish.yml.",
|
||||
"monitor_reason": "Event-driven or operator-requested work has no periodic freshness obligation.",
|
||||
"origin": "baseline"
|
||||
},
|
||||
@@ -1727,6 +1724,69 @@
|
||||
"disposition": "retained independent entry",
|
||||
"monitor_reason": "Event-driven or operator-requested work has no periodic freshness obligation.",
|
||||
"origin": "baseline"
|
||||
},
|
||||
{
|
||||
"workflow": ".github/workflows/release-candidate.yml",
|
||||
"name": "Release candidate (Linux)",
|
||||
"roles": [
|
||||
"manual",
|
||||
"release"
|
||||
],
|
||||
"events": [
|
||||
"workflow_dispatch"
|
||||
],
|
||||
"cadence": "event",
|
||||
"timezone": "UTC",
|
||||
"schedule": [],
|
||||
"owner": "@overtrue",
|
||||
"policy": "active",
|
||||
"monitor": "none",
|
||||
"result_owner": "release-candidate.yml",
|
||||
"disposition": "Explicit immutable candidate stage",
|
||||
"monitor_reason": "Operator-requested release stage has no periodic freshness obligation.",
|
||||
"origin": "release-candidate-migration"
|
||||
},
|
||||
{
|
||||
"workflow": ".github/workflows/release-complete.yml",
|
||||
"name": "Release candidate (complete)",
|
||||
"roles": [
|
||||
"manual",
|
||||
"release"
|
||||
],
|
||||
"events": [
|
||||
"workflow_dispatch"
|
||||
],
|
||||
"cadence": "event",
|
||||
"timezone": "UTC",
|
||||
"schedule": [],
|
||||
"owner": "@overtrue",
|
||||
"policy": "active",
|
||||
"monitor": "none",
|
||||
"result_owner": "release-complete.yml",
|
||||
"disposition": "Explicit immutable candidate stage",
|
||||
"monitor_reason": "Operator-requested release stage has no periodic freshness obligation.",
|
||||
"origin": "release-candidate-migration"
|
||||
},
|
||||
{
|
||||
"workflow": ".github/workflows/release-publish.yml",
|
||||
"name": "Publish accepted release",
|
||||
"roles": [
|
||||
"manual",
|
||||
"release"
|
||||
],
|
||||
"events": [
|
||||
"workflow_dispatch"
|
||||
],
|
||||
"cadence": "event",
|
||||
"timezone": "UTC",
|
||||
"schedule": [],
|
||||
"owner": "@overtrue",
|
||||
"policy": "active",
|
||||
"monitor": "none",
|
||||
"result_owner": "release-publish.yml",
|
||||
"disposition": "Explicit immutable candidate stage",
|
||||
"monitor_reason": "Operator-requested release stage has no periodic freshness obligation.",
|
||||
"origin": "release-candidate-migration"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
@@ -5,7 +5,7 @@
|
||||
|
||||
## Billing facts
|
||||
|
||||
This repository is public. For a public repository, GitHub's standard hosted runners (`ubuntu-latest`, `windows-latest`, `macos-latest`) cost nothing and have no minute cap, whatever plan the organisation is on. GitHub bills only two things: larger hosted runners, which have custom labels and are not used here, and standard runners in private repositories.
|
||||
This repository is public. For a public repository, GitHub's standard hosted runners (`ubuntu-latest`, `windows-latest`, `macos-latest`) cost nothing and have no minute cap, whatever plan the organisation is on. GitHub bills only two things: larger hosted runners, including the `macos-latest-xlarge` release lane used here, and standard runners in private repositories.
|
||||
|
||||
Every other label in this repository is capacity RustFS pays for. ARC pods and lab machines accrue cost while the machine exists. A Ubicloud label is metered for the minutes a job holds its VM:
|
||||
|
||||
@@ -13,6 +13,7 @@ Every other label in this repository is capacity RustFS pays for. ARC pods and l
|
||||
|---|---|---|---|
|
||||
| `ubuntu-latest` | nobody | GitHub-hosted, Ubuntu 24.04, 4 vCPU, 16 GB RAM, about 14 GB of free disk. Ships a C toolchain, `docker`, `gh`, `curl`, `wget`, `jq`, Python. Can `mount` tmpfs and loop devices; io_uring works | the default for every job |
|
||||
| `windows-latest`, `macos-latest` | nobody | GitHub-hosted standard runners | Windows and macOS targets |
|
||||
| `macos-latest-xlarge` | RustFS | GitHub-hosted larger Apple Silicon runner | macOS release builds, deferred until Linux acceptance for candidates |
|
||||
| `sm-standard-2` | RustFS | ARC pod, Ubuntu 24.04, minimal image: no `gh`, no C toolchain; Docker only on the `dind-` label | only a job that a hosted runner cannot run (see the placement rule) |
|
||||
| `sm-standard-4` | RustFS | ARC pod, Ubuntu 24.04, 14 CPUs and 28 GB (the label is not the core count), large disk | workspace-scale Cargo builds |
|
||||
| `ubicloud-standard-16-arm` | RustFS | Ubicloud VM, alias of `ubicloud-standard-16-arm-ubuntu-2404`: Ubuntu 24.04 arm64, 16 vCPU, 48 GB, 300 GB disk. Reached through the `build.yml` matrix, so `actionlint.yaml` does not list it | native aarch64 release builds (`linux-aarch64-musl`, `linux-aarch64-gnu`) |
|
||||
@@ -32,7 +33,7 @@ A job runs on `ubuntu-latest` unless it has one of these reasons, and the reason
|
||||
|
||||
Jobs that do only the following have no reason to be self-hosted: shell or Python scripts, `cargo fmt`, `cargo deny`, typos, actionlint, API calls (`gh`, dependency review, issue and stale bots, CLA checks), downloading an artifact and running a prebuilt binary, or a `cargo build -p rustfs` with its own cache key. `e2e-distributed.yml` and `e2e-upgrade.yml` build the server and the `e2e_test` harness on `ubuntu-latest` inside their budgets, so "it compiles something" alone is not a reason.
|
||||
|
||||
The functional chain's `prepare` and `complete-chain` jobs, the performance runner availability check, and runner hygiene use `ubuntu-latest`: they inspect APIs or artifacts, not the lab hosts. The actual functional and performance suites remain on their lab runners. Functional-chain health collection uses `ubuntu-latest` so a lab outage cannot also stop its independent monitor. Docker image packaging also uses `ubuntu-latest` for both musl and glibc variants; it downloads release binaries rather than compiling the workspace, and still sets up QEMU and Buildx explicitly.
|
||||
The functional chain's `prepare` and `complete-chain` jobs, the performance runner availability check, and runner hygiene use `ubuntu-latest`: they inspect APIs or artifacts, not the lab hosts. The actual functional and performance suites remain on their lab runners. Functional-chain health collection uses `ubuntu-latest` so a lab outage cannot also stop its independent monitor. Docker image packaging also uses `ubuntu-latest` for both musl and glibc variants; it wraps verified candidate binaries rather than compiling the workspace, and still sets up QEMU and Buildx explicitly.
|
||||
|
||||
Moving these jobs off AKS reduces capacity demand, not necessarily the Azure bill immediately. VM costs fall only when nodes scale down or smaller capacity replaces them.
|
||||
|
||||
@@ -52,7 +53,7 @@ A hosted runner is also the only choice for a few things the pods cannot do: `mo
|
||||
These are the patterns that have cost real money here without validating anything more:
|
||||
|
||||
- An `if: always()` summary or alert job on a self-hosted label inside a `workflow_run`-triggered workflow. The upstream workflow completes many times a day, the real jobs skip, and the summary still starts a pod each time. Gate it on the upstream job having run, and keep it on `ubuntu-latest`.
|
||||
- Per-push workflows on `main`. A merge triggers the full `ci.yml` matrix, `cache-warm.yml` and the `build.yml` development build; with tens of merges a day that is as much self-hosted time as all pull requests together. Anything that does not verify the merged tree itself belongs on a schedule or behind a `paths` filter.
|
||||
- Per-push workflows on `main`. A merge triggers the applicable `ci.yml` matrix and cache warming; `build.yml` development builds are scheduled or manual; with tens of merges a day that is as much self-hosted time as all pull requests together. Anything that does not verify the merged tree itself belongs on a schedule or behind a `paths` filter.
|
||||
- `apt-get install` of tools the hosted image already ships. It is only needed on the minimal pod images.
|
||||
- A throttled `CARGO_BUILD_JOBS` on a 14-CPU pod. The limit exists for link-storm reasons documented in `ci.yml`; measure before raising it, and measure before lowering it further.
|
||||
- Tiny jobs on ARC pods. Each one pays roughly a minute of pod start-up before its first step; a compile-free job that takes ten seconds costs more to schedule than to run.
|
||||
@@ -66,7 +67,7 @@ gh api "repos/rustfs/rustfs/actions/runs/<run_id>/jobs?per_page=100" \
|
||||
--jq '.jobs[] | "\(.conclusion)\t\(.labels | join(","))\t\(.started_at)\t\(.completed_at)\t\(.name)"'
|
||||
```
|
||||
|
||||
Only rows whose label starts with `sm-`, `dind-`, `ubicloud-`, `smoke-` or `pf-` cost anything. At the October 2026 review, a full pull-request run of `ci.yml` spent about 150 self-hosted minutes, a third of it in jobs that never compiled; treat that as the scale of what a placement mistake costs, not as a current figure.
|
||||
Rows using `macos-latest-xlarge` or a label starting with `sm-`, `dind-`, `ubicloud-`, `smoke-` or `pf-` use paid capacity. At the October 2026 review, a full pull-request run of `ci.yml` spent about 150 self-hosted minutes, a third of it in jobs that never compiled; treat that as the scale of what a placement mistake costs, not as a current figure.
|
||||
|
||||
## Checklist for a new or moved job
|
||||
|
||||
@@ -76,3 +77,9 @@ Only rows whose label starts with `sm-`, `dind-`, `ubicloud-`, `smoke-` or `pf-`
|
||||
- A self-hosted `runs-on` carries a comment naming its reason from the placement rule.
|
||||
- Every `./.github/actions/setup` call states `cache-save-if`, and a key read from one pool is not written from the other.
|
||||
- Moving a job does not change its check name, so the `main` ruleset and `docs/testing/ci-gates.md` stay as they are; renaming it does, and then that document's change checklist applies.
|
||||
|
||||
## Release stages
|
||||
|
||||
The [candidate release pipeline](../../docs/operations/release-pipeline.md) builds Linux first and defers paid desktop capacity until acceptance. Daily development builds also select Linux; use manual platform selection for desktop development artifacts. All six platforms remain required before publication. The pipeline records and promotes the exact tested bytes. Builder workflows have no public-release credentials. Keep periodic development builds, PR verification and field-test chains separate when their contracts differ; do not re-enable disabled workflows as part of a source-file migration.
|
||||
|
||||
Larger runners remain billed for public repositories; see [GitHub runner billing](https://docs.github.com/en/actions/concepts/runners/larger-runners#billing). Actual spending comes from organization billing, not runner labels or an estimated build duration.
|
||||
|
||||
+149
-8
@@ -23,9 +23,16 @@ on:
|
||||
- 'deny.toml'
|
||||
- '.github/actions/**'
|
||||
- '.github/workflows/**'
|
||||
- 'scripts/release/create_or_update_release.sh'
|
||||
- 'scripts/release/package_service_scripts.sh'
|
||||
- 'scripts/release/package_versions.sh'
|
||||
- 'scripts/release/**'
|
||||
- 'scripts/security/generate_release_supply_chain_assets.py'
|
||||
- '.config/release-validation-plan.json'
|
||||
- '.github/scheduled-validations.json'
|
||||
- 'scripts/test_helm_release_workflow.py'
|
||||
- 'scripts/test_release_packages.py'
|
||||
- 'scripts/test_release_images.py'
|
||||
- 'Dockerfile*'
|
||||
- 'entrypoint.sh'
|
||||
- 'bootstrap/**'
|
||||
- 'scripts/test_package_service_scripts.sh'
|
||||
- 'scripts/test_package_versions.sh'
|
||||
- 'scripts/test_docker_workflow.py'
|
||||
@@ -41,9 +48,16 @@ on:
|
||||
- 'deny.toml'
|
||||
- '.github/actions/**'
|
||||
- '.github/workflows/**'
|
||||
- 'scripts/release/create_or_update_release.sh'
|
||||
- 'scripts/release/package_service_scripts.sh'
|
||||
- 'scripts/release/package_versions.sh'
|
||||
- 'scripts/release/**'
|
||||
- 'scripts/security/generate_release_supply_chain_assets.py'
|
||||
- '.config/release-validation-plan.json'
|
||||
- '.github/scheduled-validations.json'
|
||||
- 'scripts/test_helm_release_workflow.py'
|
||||
- 'scripts/test_release_packages.py'
|
||||
- 'scripts/test_release_images.py'
|
||||
- 'Dockerfile*'
|
||||
- 'entrypoint.sh'
|
||||
- 'bootstrap/**'
|
||||
- 'scripts/test_package_service_scripts.sh'
|
||||
- 'scripts/test_package_versions.sh'
|
||||
- 'scripts/test_docker_workflow.py'
|
||||
@@ -132,12 +146,38 @@ jobs:
|
||||
if: github.repository == 'rustfs/rustfs' && (github.event_name != 'pull_request' || github.event.action != 'closed')
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 5
|
||||
permissions:
|
||||
contents: read
|
||||
actions: read
|
||||
outputs:
|
||||
release_changed: ${{ steps.scope.outputs.release_changed }}
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
|
||||
with:
|
||||
fetch-depth: 0
|
||||
persist-credentials: false
|
||||
|
||||
- name: Scope native packaging checks to release changes
|
||||
id: scope
|
||||
env:
|
||||
BASE_SHA: ${{ github.event.pull_request.base.sha || github.event.before }}
|
||||
run: |
|
||||
python3 - <<'PY'
|
||||
import os, re, subprocess
|
||||
event, base = os.environ['GITHUB_EVENT_NAME'], os.environ.get('BASE_SHA', '')
|
||||
changed = event == 'workflow_dispatch'
|
||||
if event in ('push', 'pull_request'):
|
||||
if not re.fullmatch(r'[0-9a-f]{40}', base) or base == '0' * 40:
|
||||
changed = True
|
||||
else:
|
||||
paths = subprocess.check_output(['git', 'diff', '--name-only', base, os.environ['GITHUB_SHA']], text=True).splitlines()
|
||||
pattern = r'^(scripts/release/|scripts/test_release_|Dockerfile|entrypoint.sh$|bootstrap/|\.dockerignore$|\.config/release-validation-plan.json$|\.github/workflows/(audit|build|docker|package|helm-package|release-[^.]+)\.yml$)'
|
||||
changed = any(re.match(pattern, path) for path in paths)
|
||||
with open(os.environ['GITHUB_OUTPUT'], 'a') as output:
|
||||
output.write(f'release_changed={str(changed).lower()}\n')
|
||||
PY
|
||||
|
||||
- name: Report unpinned GitHub Actions
|
||||
run: ./scripts/security/check_workflow_pins.sh --enforce
|
||||
|
||||
@@ -150,8 +190,35 @@ jobs:
|
||||
- name: Check checkouts clear their credentials
|
||||
run: ./scripts/security/check_persist_credentials.sh
|
||||
|
||||
- name: Check preview release workflow policy
|
||||
run: ./scripts/security/check_preview_release_workflow.sh
|
||||
- name: Check release candidate and promotion contracts
|
||||
run: python3 -m unittest discover -s scripts/release -p 'test_*.py'
|
||||
|
||||
- name: Prepare artifact transport rehearsal
|
||||
run: |
|
||||
mkdir -p "$RUNNER_TEMP/release-transport"
|
||||
printf '{"purpose":"transport contract, not release evidence"}\n' > "$RUNNER_TEMP/release-transport/transport.json"
|
||||
|
||||
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
|
||||
with:
|
||||
name: release-transport-contract-${{ github.run_attempt }}
|
||||
path: ${{ runner.temp }}/release-transport/transport.json
|
||||
if-no-files-found: error
|
||||
retention-days: 1
|
||||
|
||||
- name: Verify actual GitHub artifact metadata and downloaded bytes
|
||||
env:
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
run: |
|
||||
python3 - <<'PY'
|
||||
import os, sys
|
||||
from pathlib import Path
|
||||
sys.path.insert(0, 'scripts/release')
|
||||
import candidate as c
|
||||
run = c.github(f"repos/{c.REPOSITORY}/actions/runs/{os.environ['GITHUB_RUN_ID']}")
|
||||
destination = Path(os.environ['RUNNER_TEMP']) / 'release-transport-downloaded'
|
||||
c.download_artifact(run, f"release-transport-contract-{os.environ['GITHUB_RUN_ATTEMPT']}", destination)
|
||||
c.require(c.read_json(destination / 'transport.json') == {'purpose': 'transport contract, not release evidence'}, 'artifact round trip changed')
|
||||
PY
|
||||
|
||||
- name: Check tier evidence workflow isolation
|
||||
run: ./scripts/security/check_tier_artifact_workflow.sh
|
||||
@@ -162,6 +229,80 @@ jobs:
|
||||
- name: Check package service lifecycle contract
|
||||
run: ./scripts/test_package_service_scripts.sh
|
||||
|
||||
release-package-contract:
|
||||
name: Release package contract
|
||||
needs: workflow-pin-report
|
||||
if: needs.workflow-pin-report.outputs.release_changed == 'true'
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 15
|
||||
steps:
|
||||
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
|
||||
with:
|
||||
persist-credentials: false
|
||||
- name: Install real package managers
|
||||
run: |
|
||||
sudo apt-get update
|
||||
sudo apt-get install -y ruby ruby-dev build-essential rpm fakeroot cpio
|
||||
sudo gem install fpm --version 1.17.0 --no-document
|
||||
- name: Exercise accepted and substituted payloads through both recipes
|
||||
run: python3 scripts/test_release_packages.py
|
||||
|
||||
release-image-contract:
|
||||
name: Release image contract (${{ matrix.variant }})
|
||||
needs: workflow-pin-report
|
||||
if: needs.workflow-pin-report.outputs.release_changed == 'true'
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 35
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
variant: [musl, gnu]
|
||||
env:
|
||||
RELEASE_IMAGE_VARIANT: ${{ matrix.variant }}
|
||||
NO_PROXY: 127.0.0.1,localhost,::1
|
||||
steps:
|
||||
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
|
||||
with:
|
||||
persist-credentials: false
|
||||
- name: Install OCI tools
|
||||
run: |
|
||||
echo "RELEASE_IMAGE_OUTPUT=$RUNNER_TEMP/image-contract" >> "$GITHUB_ENV"
|
||||
echo 'no_proxy=127.0.0.1,localhost,::1' >> "$GITHUB_ENV"
|
||||
sudo apt-get update
|
||||
sudo apt-get install -y skopeo
|
||||
go version
|
||||
- uses: docker/setup-qemu-action@c7c53464625b32c7a7e944ae62b3e17d2b600130 # v3
|
||||
- uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # v3
|
||||
- name: Exercise real OCI packaging, startup, registry retry and conflicts
|
||||
run: python3 scripts/test_release_images.py
|
||||
- name: Exercise amd64 OCI archive scanning
|
||||
uses: aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25
|
||||
env:
|
||||
TRIVY_PLATFORM: linux/amd64
|
||||
with:
|
||||
input: ${{ runner.temp }}/image-contract-scan/image-${{ matrix.variant }}-amd64.tar
|
||||
format: sarif
|
||||
output: ${{ runner.temp }}/image-contract/scan-${{ matrix.variant }}-amd64.sarif
|
||||
exit-code: "0"
|
||||
- name: Exercise arm64 OCI archive scanning
|
||||
uses: aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25
|
||||
env:
|
||||
TRIVY_PLATFORM: linux/arm64
|
||||
with:
|
||||
input: ${{ runner.temp }}/image-contract-scan/image-${{ matrix.variant }}-arm64.tar
|
||||
format: sarif
|
||||
output: ${{ runner.temp }}/image-contract/scan-${{ matrix.variant }}-arm64.sarif
|
||||
exit-code: "0"
|
||||
- name: Verify scan receipts before recording the archive
|
||||
run: |
|
||||
python3 - <<'PY'
|
||||
import os, sys
|
||||
from pathlib import Path
|
||||
sys.path.insert(0, 'scripts/release')
|
||||
import deliver, test_candidate
|
||||
deliver.image_record(test_candidate.inputs(), os.environ['RELEASE_IMAGE_VARIANT'], Path(os.environ['RELEASE_IMAGE_OUTPUT']))
|
||||
PY
|
||||
|
||||
dependency-review:
|
||||
name: Dependency Review
|
||||
runs-on: ubuntu-latest
|
||||
|
||||
+184
-765
File diff suppressed because it is too large
Load Diff
@@ -142,15 +142,23 @@ jobs:
|
||||
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
|
||||
with:
|
||||
persist-credentials: false
|
||||
- name: Install pinned typos through GitHub API
|
||||
env:
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
TYPOS_VERSION: 1.50.2
|
||||
run: |
|
||||
python3 .github/actions/setup/download_release_asset.py --repository crate-ci/typos --tag "v${TYPOS_VERSION}" --asset "typos-v${TYPOS_VERSION}-x86_64-unknown-linux-musl.tar.gz" --output "$RUNNER_TEMP/typos.tar.gz"
|
||||
mkdir -p "$RUNNER_TEMP/typos"
|
||||
tar -xzf "$RUNNER_TEMP/typos.tar.gz" -C "$RUNNER_TEMP/typos"
|
||||
- name: Typos check with custom config file
|
||||
uses: crate-ci/typos@512fc24f32f44ab01972217aaaf3dc86ec234d53 # v1.50.2
|
||||
run: '"$RUNNER_TEMP/typos/typos" .'
|
||||
|
||||
# Fail early with compile-free checks for every pull request.
|
||||
quick-checks:
|
||||
name: Quick Checks
|
||||
if: github.event_name != 'pull_request' || github.event.action != 'closed'
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 10
|
||||
timeout-minutes: 20
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
|
||||
@@ -1137,11 +1145,16 @@ jobs:
|
||||
|
||||
- name: Install mc
|
||||
env:
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
MC_VERSION: RELEASE.2025-08-13T08-35-41Z
|
||||
MC_SHA256: 01f866e9c5f9b87c2b09116fa5d7c06695b106242d829a8bb32990c00312e891
|
||||
run: |
|
||||
if ! command -v gh >/dev/null 2>&1; then
|
||||
sudo apt-get update
|
||||
sudo apt-get install -y gh
|
||||
fi
|
||||
MC_BINARY="mc.linux-amd64.${MC_VERSION}"
|
||||
curl -fsSLo "$RUNNER_TEMP/mc" "https://github.com/minio/mc/releases/download/${MC_VERSION}/${MC_BINARY}"
|
||||
python3 .github/actions/setup/download_release_asset.py --repository minio/mc --tag "$MC_VERSION" --asset "$MC_BINARY" --output "$RUNNER_TEMP/mc" --sha256 "$MC_SHA256"
|
||||
echo "${MC_SHA256} $RUNNER_TEMP/mc" | sha256sum --check --status
|
||||
chmod +x "$RUNNER_TEMP/mc"
|
||||
echo "$RUNNER_TEMP" >> "$GITHUB_PATH"
|
||||
|
||||
+71
-751
@@ -1,779 +1,99 @@
|
||||
# Copyright 2024 RustFS Team
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
# Docker Images Workflow
|
||||
#
|
||||
# This workflow builds Docker images using pre-built binaries from the build workflow.
|
||||
#
|
||||
# Trigger Types:
|
||||
# 1. workflow_run: Automatically triggered when "Build and Release" workflow completes
|
||||
# 2. workflow_dispatch: Manual trigger for standalone Docker builds
|
||||
#
|
||||
# Key Features:
|
||||
# - Only triggers when Linux builds (x86_64 + aarch64) are successful
|
||||
# - Independent of macOS/Windows build status
|
||||
# - Uses workflow_run event for precise control
|
||||
# - Only builds Docker images for releases and prereleases (development builds are skipped)
|
||||
|
||||
name: Docker Images
|
||||
|
||||
# Permissions needed for workflow_run event and Docker registry access
|
||||
# Copyright 2026 RustFS Team
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
name: Build candidate images
|
||||
on:
|
||||
workflow_call:
|
||||
inputs:
|
||||
candidate_manifest:
|
||||
type: string
|
||||
required: true
|
||||
permissions:
|
||||
contents: read
|
||||
packages: write
|
||||
|
||||
on:
|
||||
# Automatically triggered when build workflow completes
|
||||
workflow_run:
|
||||
workflows: [ "Build and Release" ]
|
||||
types: [ completed ]
|
||||
# Manual trigger with same parameters for consistency
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
push_images:
|
||||
description: "Push images to registries"
|
||||
required: false
|
||||
default: true
|
||||
type: boolean
|
||||
version:
|
||||
description: "Version to build (latest, v1.0.0, v1.0.0-alpha1, or v1.0.1-preview.1)"
|
||||
required: false
|
||||
default: "latest"
|
||||
type: string
|
||||
force_rebuild:
|
||||
description: "Force rebuild even if binary exists (useful for testing)"
|
||||
required: false
|
||||
default: false
|
||||
type: boolean
|
||||
|
||||
env:
|
||||
CONCLUSION: ${{ github.event.workflow_run.conclusion }}
|
||||
HEAD_BRANCH: ${{ github.event.workflow_run.head_branch }}
|
||||
HEAD_SHA: ${{ github.event.workflow_run.head_sha }}
|
||||
TRIGGERING_EVENT: ${{ github.event.workflow_run.event }}
|
||||
DOCKERHUB_USERNAME: rustfs
|
||||
CARGO_TERM_COLOR: always
|
||||
REGISTRY_DOCKERHUB: rustfs/rustfs
|
||||
REGISTRY_GHCR: ghcr.io/${{ github.repository }}
|
||||
REGISTRY_QUAY: quay.io/rustfs/rustfs
|
||||
DOCKER_PLATFORMS: linux/amd64,linux/arm64
|
||||
|
||||
actions: read
|
||||
jobs:
|
||||
# Check if we should build Docker images.
|
||||
# Short-circuit at job level so per-merge development builds (workflow_run
|
||||
# from a push to main) skip the whole run without spawning a checkout job:
|
||||
# images are only published for tag builds — build.yml push triggers cover
|
||||
# only main and tags, so a non-main head_branch is a tag name — and for
|
||||
# manual dispatch.
|
||||
build-check:
|
||||
name: Docker Build Check
|
||||
if: >-
|
||||
github.event_name == 'workflow_dispatch' ||
|
||||
(github.event.workflow_run.conclusion == 'success' &&
|
||||
github.event.workflow_run.event == 'push' &&
|
||||
github.event.workflow_run.head_branch != 'main')
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 30
|
||||
outputs:
|
||||
should_build: ${{ steps.check.outputs.should_build }}
|
||||
should_push: ${{ steps.check.outputs.should_push }}
|
||||
build_type: ${{ steps.check.outputs.build_type }}
|
||||
version: ${{ steps.check.outputs.version }}
|
||||
short_sha: ${{ steps.check.outputs.short_sha }}
|
||||
is_prerelease: ${{ steps.check.outputs.is_prerelease }}
|
||||
create_latest: ${{ steps.check.outputs.create_latest }}
|
||||
source_ref: ${{ steps.check.outputs.source_ref }}
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
|
||||
with:
|
||||
persist-credentials: false
|
||||
# For workflow_run events, checkout the specific commit that triggered the workflow
|
||||
ref: ${{ github.event.workflow_run.head_sha || github.sha }}
|
||||
|
||||
- name: Check build conditions
|
||||
id: check
|
||||
env:
|
||||
# dispatch inputs via env, not `${{ }}` interpolation: they are
|
||||
# free-form strings and would otherwise be evaluated by bash.
|
||||
INPUT_VERSION: ${{ github.event.inputs.version }}
|
||||
INPUT_PUSH_IMAGES: ${{ github.event.inputs.push_images }}
|
||||
INPUT_FORCE_REBUILD: ${{ github.event.inputs.force_rebuild }}
|
||||
run: |
|
||||
should_build=false
|
||||
should_push=false
|
||||
build_type="none"
|
||||
version=""
|
||||
short_sha=""
|
||||
is_prerelease=false
|
||||
create_latest=false
|
||||
source_ref="$GITHUB_SHA"
|
||||
|
||||
# Pre-GA policy: until the first stable (vX.Y.Z) tag exists, every
|
||||
# prerelease (alpha/beta/rc) also moves `latest`, so users pulling
|
||||
# `latest` get the newest test build. Once a stable tag is published
|
||||
# this returns false and `latest` follows stable releases only.
|
||||
prerelease_moves_latest() {
|
||||
local stable_tags
|
||||
stable_tags=$(git ls-remote --tags --refs origin 2>/dev/null \
|
||||
| awk '{print $2}' \
|
||||
| grep -E '^refs/tags/v?[0-9]+\.[0-9]+\.[0-9]+$' || true)
|
||||
if [[ -z "$stable_tags" ]]; then
|
||||
return 0
|
||||
fi
|
||||
echo "ℹ️ Stable release tag(s) already exist; prereleases no longer update latest"
|
||||
return 1
|
||||
}
|
||||
|
||||
if [[ "${{ github.event_name }}" == "workflow_run" ]]; then
|
||||
# Triggered by build workflow completion
|
||||
echo "🔗 Triggered by build workflow completion"
|
||||
|
||||
# Check if the triggering workflow was successful
|
||||
# If the workflow succeeded, it means ALL builds (including Linux x86_64 and aarch64) succeeded
|
||||
if [[ "$CONCLUSION" == "success" ]]; then
|
||||
echo "✅ Build workflow succeeded, all builds including Linux are successful"
|
||||
should_build=true
|
||||
should_push=true
|
||||
else
|
||||
echo "❌ Build workflow failed (conclusion: $CONCLUSION), skipping Docker build"
|
||||
should_build=false
|
||||
fi
|
||||
|
||||
# Extract version info from commit message or use commit SHA
|
||||
# Use Git to generate consistent short SHA (ensures uniqueness like build.yml)
|
||||
short_sha=$(git rev-parse --short "$HEAD_SHA")
|
||||
source_ref="$HEAD_SHA"
|
||||
|
||||
# Determine build type based on triggering workflow event and ref
|
||||
triggering_event="$TRIGGERING_EVENT"
|
||||
head_branch="$HEAD_BRANCH"
|
||||
|
||||
echo "🔍 Analyzing triggering workflow:"
|
||||
echo " 📋 Event: $triggering_event"
|
||||
echo " 🌿 Head branch: $head_branch"
|
||||
echo " 📎 Head SHA: $HEAD_SHA"
|
||||
|
||||
# Check if this was triggered by a tag push
|
||||
if [[ "$triggering_event" == "push" ]]; then
|
||||
# For tag pushes, head_branch will be like "refs/tags/v1.0.0" or just "v1.0.0"
|
||||
if [[ "$head_branch" == refs/tags/* ]]; then
|
||||
# Extract tag name from refs/tags/TAG_NAME
|
||||
tag_name="${head_branch#refs/tags/}"
|
||||
version="$tag_name"
|
||||
elif [[ "$head_branch" =~ ^v?[0-9]+\.[0-9]+\.[0-9]+ ]]; then
|
||||
# Direct tag name like "v1.0.0" or "1.0.0-alpha.1"
|
||||
version="$head_branch"
|
||||
elif [[ "$head_branch" == "main" ]]; then
|
||||
# Regular branch push to main
|
||||
build_type="development"
|
||||
version="dev-${short_sha}"
|
||||
should_build=false
|
||||
echo "⏭️ Skipping Docker build for development version (main branch push)"
|
||||
else
|
||||
# Other branch push
|
||||
build_type="development"
|
||||
version="dev-${short_sha}"
|
||||
should_build=false
|
||||
echo "⏭️ Skipping Docker build for development version (branch: $head_branch)"
|
||||
fi
|
||||
|
||||
# If we extracted a version (tag), determine release type
|
||||
if [[ -n "$version" ]] && [[ "$version" != "dev-${short_sha}" ]]; then
|
||||
# Remove 'v' prefix if present for consistent version format
|
||||
if [[ "$version" == v* ]]; then
|
||||
version="${version#v}"
|
||||
fi
|
||||
|
||||
if [[ "$version" =~ -preview\.[0-9]+$ ]]; then
|
||||
build_type="preview"
|
||||
is_prerelease=true
|
||||
echo "🔍 Building Docker image for preview: $version (version tags only)"
|
||||
elif [[ "$version" == *"-preview"* ]]; then
|
||||
echo "❌ Invalid preview tag: $version (expected suffix: -preview.<number>)" >&2
|
||||
exit 1
|
||||
elif [[ "$version" == *"alpha"* ]] || [[ "$version" == *"beta"* ]] || [[ "$version" == *"rc"* ]]; then
|
||||
build_type="prerelease"
|
||||
is_prerelease=true
|
||||
# Pre-GA policy: prereleases update latest until the first stable tag exists.
|
||||
if prerelease_moves_latest; then
|
||||
create_latest=true
|
||||
echo "🧪 Building Docker image for prerelease: $version (creating latest tag)"
|
||||
else
|
||||
echo "🧪 Building Docker image for prerelease: $version"
|
||||
fi
|
||||
else
|
||||
build_type="release"
|
||||
create_latest=true
|
||||
echo "🚀 Building Docker image for release: $version"
|
||||
fi
|
||||
fi
|
||||
else
|
||||
# Non-push events
|
||||
build_type="development"
|
||||
version="dev-${short_sha}"
|
||||
should_build=false
|
||||
echo "⏭️ Skipping Docker build for development version (event: $triggering_event)"
|
||||
fi
|
||||
|
||||
echo "🔄 Build triggered by workflow_run:"
|
||||
echo " 📋 Conclusion: $CONCLUSION"
|
||||
echo " 🌿 Branch: $HEAD_BRANCH"
|
||||
echo " 📎 SHA: $HEAD_SHA"
|
||||
echo " 🎯 Event: $TRIGGERING_EVENT"
|
||||
|
||||
elif [[ "${{ github.event_name }}" == "workflow_dispatch" ]]; then
|
||||
# Manual trigger
|
||||
input_version="$INPUT_VERSION"
|
||||
version="${input_version}"
|
||||
should_push="$INPUT_PUSH_IMAGES"
|
||||
should_build=true
|
||||
|
||||
# Get short SHA
|
||||
short_sha=$(git rev-parse --short HEAD)
|
||||
|
||||
echo "🎯 Manual Docker build triggered:"
|
||||
echo " 📋 Requested version: $input_version"
|
||||
echo " 🔧 Force rebuild: $INPUT_FORCE_REBUILD"
|
||||
echo " 🚀 Push images: $should_push"
|
||||
|
||||
case "$input_version" in
|
||||
"latest")
|
||||
build_type="release"
|
||||
create_latest=true
|
||||
echo "🚀 Building with latest stable release version"
|
||||
;;
|
||||
*-preview*)
|
||||
if [[ ! "$input_version" =~ ^v?[0-9]+\.[0-9]+\.[0-9]+(-[0-9A-Za-z.-]+)?-preview\.[0-9]+$ ]]; then
|
||||
echo "❌ Invalid preview version: $input_version" >&2
|
||||
exit 1
|
||||
fi
|
||||
build_type="preview"
|
||||
is_prerelease=true
|
||||
version="${input_version#v}"
|
||||
echo "🔍 Building with preview version: $version (version tags only)"
|
||||
;;
|
||||
# Prerelease versions (must match first, more specific)
|
||||
v*alpha*|v*beta*|v*rc*|*alpha*|*beta*|*rc*)
|
||||
build_type="prerelease"
|
||||
is_prerelease=true
|
||||
# Pre-GA policy: prereleases update latest until the first stable tag exists.
|
||||
if prerelease_moves_latest; then
|
||||
create_latest=true
|
||||
echo "🧪 Building with prerelease version: $input_version (creating latest tag)"
|
||||
else
|
||||
echo "🧪 Building with prerelease version: $input_version"
|
||||
fi
|
||||
;;
|
||||
# Release versions (match after prereleases, more general)
|
||||
v[0-9]*|[0-9]*.*.*)
|
||||
build_type="release"
|
||||
create_latest=true
|
||||
echo "📦 Building with specific release version: $input_version"
|
||||
;;
|
||||
*)
|
||||
# Invalid version for Docker build
|
||||
should_build=false
|
||||
echo "❌ Invalid version for Docker build: $input_version"
|
||||
echo "⚠️ Only release versions (latest, v1.0.0, 1.0.0) and prereleases (v1.0.0-alpha1, 1.0.0-beta2) are supported"
|
||||
;;
|
||||
esac
|
||||
|
||||
if [[ "$should_build" == true && "$input_version" != "latest" ]]; then
|
||||
tag_ref="refs/tags/$input_version"
|
||||
if ! git ls-remote --exit-code origin "$tag_ref" >/dev/null 2>&1; then
|
||||
if [[ "$input_version" == v* ]]; then
|
||||
tag_ref="refs/tags/${input_version#v}"
|
||||
else
|
||||
tag_ref="refs/tags/v$input_version"
|
||||
fi
|
||||
fi
|
||||
|
||||
if ! git ls-remote --exit-code origin "$tag_ref" >/dev/null 2>&1; then
|
||||
echo "❌ Release tag not found for Docker build: $input_version"
|
||||
exit 1
|
||||
fi
|
||||
source_ref="$tag_ref"
|
||||
fi
|
||||
fi
|
||||
|
||||
{
|
||||
echo "should_build=$should_build"
|
||||
echo "should_push=$should_push"
|
||||
echo "build_type=$build_type"
|
||||
echo "version=$version"
|
||||
echo "short_sha=$short_sha"
|
||||
echo "is_prerelease=$is_prerelease"
|
||||
echo "create_latest=$create_latest"
|
||||
echo "source_ref=$source_ref"
|
||||
} >> "$GITHUB_OUTPUT"
|
||||
|
||||
echo "🐳 Docker Build Summary:"
|
||||
echo " - Should build: $should_build"
|
||||
echo " - Should push: $should_push"
|
||||
echo " - Build type: $build_type"
|
||||
echo " - Version: $version"
|
||||
echo " - Short SHA: $short_sha"
|
||||
echo " - Is prerelease: $is_prerelease"
|
||||
echo " - Create latest: $create_latest"
|
||||
echo " - Source ref: $source_ref"
|
||||
|
||||
# Consume the trusted release certification for the exact version this
|
||||
# workflow wraps and verify the certified input digests. A manual dispatch
|
||||
# without that certification fails here and pushes no official tag; only
|
||||
# results from the trusted verification sources carry release authority.
|
||||
release-certification:
|
||||
name: Release Certification
|
||||
needs: build-check
|
||||
if: needs.build-check.outputs.should_build == 'true'
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 30
|
||||
permissions:
|
||||
contents: read
|
||||
actions: read
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
|
||||
with:
|
||||
persist-credentials: false
|
||||
ref: ${{ needs.build-check.outputs.source_ref }}
|
||||
|
||||
- name: Download and verify release certification
|
||||
env:
|
||||
GH_TOKEN: ${{ secrets.RELEASE_VALIDATION_TOKEN || github.token }}
|
||||
TRIGGERING_RUN_ID: ${{ github.event.workflow_run.id }}
|
||||
VERSION: ${{ needs.build-check.outputs.version }}
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
mkdir -p certification
|
||||
run_id="$TRIGGERING_RUN_ID"
|
||||
if [[ -z "$run_id" ]]; then
|
||||
run_id="$(gh api --method GET \
|
||||
"repos/${GITHUB_REPOSITORY}/actions/workflows/build.yml/runs" \
|
||||
-f branch="$VERSION" -f status=success -F per_page=1 \
|
||||
--jq '.workflow_runs[0].id // empty' || true)"
|
||||
fi
|
||||
if [[ -z "$run_id" ]]; then
|
||||
echo "No certified Build and Release run found for ${VERSION}; failing closed" >&2
|
||||
exit 1
|
||||
fi
|
||||
if ! gh run download "$run_id" -n release-certification -D certification; then
|
||||
echo "Build run ${run_id} published no release-certification artifact; failing closed" >&2
|
||||
exit 1
|
||||
fi
|
||||
python3 scripts/release/release_gate.py verify-promotion \
|
||||
--plan .config/release-validation-plan.json \
|
||||
--candidate certification/candidate.json \
|
||||
--summary certification/release-gate-output \
|
||||
--artifact-sha256 "$(jq -r '.deliverables[0].sha256' certification/candidate.json)" \
|
||||
--artifact-name "$(jq -r '.deliverables[0].name' certification/candidate.json)"
|
||||
|
||||
# Input digest binding: every zip this image wraps must be the
|
||||
# certified byte-for-byte content, never a mutable latest pointer.
|
||||
while IFS=$'\t' read -r name location expected; do
|
||||
echo "Verifying certified input ${name}"
|
||||
actual="$(curl -fsSL "$location" | sha256sum | awk '{print $1}')"
|
||||
if [[ "$actual" != "$expected" ]]; then
|
||||
echo "Certified input ${name} does not match its recorded digest" >&2
|
||||
exit 1
|
||||
fi
|
||||
done < <(jq -r '.deliverables[] | select(.name | contains("linux")) | [.name, .location, .sha256] | @tsv' certification/candidate.json)
|
||||
|
||||
# Build multi-arch Docker images
|
||||
# Strategy: Build images using pre-built binaries from dl.rustfs.com
|
||||
# Supports both release and dev channel binaries based on build context
|
||||
# Only runs when should_build is true (which includes workflow success check)
|
||||
build-docker:
|
||||
name: Build Docker Images
|
||||
needs: [ build-check, release-certification ]
|
||||
if: needs.build-check.outputs.should_build == 'true'
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 60
|
||||
timeout-minutes: 90
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
include:
|
||||
- variant: musl
|
||||
file: Dockerfile
|
||||
suffix: ""
|
||||
- variant: glibc
|
||||
- variant: gnu
|
||||
file: Dockerfile.glibc
|
||||
suffix: "-glibc"
|
||||
|
||||
env:
|
||||
CANDIDATE_MANIFEST: ${{ inputs.candidate_manifest }}
|
||||
VARIANT: ${{ matrix.variant }}
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
|
||||
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
|
||||
with:
|
||||
ref: ${{ fromJson(inputs.candidate_manifest).source_sha }}
|
||||
persist-credentials: false
|
||||
ref: ${{ needs.build-check.outputs.source_ref }}
|
||||
|
||||
- name: Login to Docker Hub
|
||||
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
|
||||
with:
|
||||
username: ${{ env.DOCKERHUB_USERNAME }}
|
||||
password: ${{ secrets.DOCKERHUB_TOKEN }}
|
||||
|
||||
- name: Login to GitHub Container Registry
|
||||
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
|
||||
with:
|
||||
registry: ghcr.io
|
||||
username: ${{ secrets.GHCR_USERNAME }}
|
||||
password: ${{ secrets.GHCR_PASSWORD }}
|
||||
|
||||
- name: Login to Quay.io
|
||||
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
|
||||
with:
|
||||
registry: quay.io
|
||||
username: ${{ secrets.QUAY_USERNAME }}
|
||||
password: ${{ secrets.QUAY_PASSWORD }}
|
||||
|
||||
- name: Set up QEMU
|
||||
uses: docker/setup-qemu-action@c7c53464625b32c7a7e944ae62b3e17d2b600130 # v3
|
||||
|
||||
- name: Set up Docker Buildx
|
||||
uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # v3
|
||||
|
||||
- name: Extract metadata and generate tags
|
||||
id: meta
|
||||
- name: Reuse completed image
|
||||
id: reuse
|
||||
env:
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
run: python3 scripts/release/deliver.py reuse --kind image --target "$VARIANT"
|
||||
- name: Install OCI tools
|
||||
if: steps.reuse.outputs.reused != 'true'
|
||||
run: |
|
||||
BUILD_TYPE="${{ needs.build-check.outputs.build_type }}"
|
||||
VERSION="${{ needs.build-check.outputs.version }}"
|
||||
CREATE_LATEST="${{ needs.build-check.outputs.create_latest }}"
|
||||
VARIANT_SUFFIX="${{ matrix.suffix }}"
|
||||
|
||||
# Convert version format for Dockerfile compatibility. The former
|
||||
# DOCKER_CHANNEL was "release" down every branch and was passed as a
|
||||
# build-arg no Dockerfile declares, so it is gone.
|
||||
case "$VERSION" in
|
||||
"latest")
|
||||
DOCKER_RELEASE="latest"
|
||||
;;
|
||||
v*)
|
||||
# For versioned releases (v1.0.0), remove 'v' prefix for Dockerfile
|
||||
DOCKER_RELEASE="${VERSION#v}"
|
||||
;;
|
||||
*)
|
||||
# For other versions, pass as-is
|
||||
DOCKER_RELEASE="${VERSION}"
|
||||
;;
|
||||
esac
|
||||
|
||||
echo "docker_release=$DOCKER_RELEASE" >> "$GITHUB_OUTPUT"
|
||||
|
||||
echo "🐳 Docker build parameters:"
|
||||
echo " - Original version: $VERSION"
|
||||
echo " - Docker RELEASE: $DOCKER_RELEASE"
|
||||
|
||||
# Generate tags based on build type
|
||||
# Only support release and prerelease builds (no development builds)
|
||||
TAG_BASE="${VERSION}${VARIANT_SUFFIX}"
|
||||
TAGS="${{ env.REGISTRY_DOCKERHUB }}:$TAG_BASE,${{ env.REGISTRY_GHCR }}:$TAG_BASE,${{ env.REGISTRY_QUAY }}:$TAG_BASE"
|
||||
|
||||
# Add latest when requested (stable releases, and prereleases before GA)
|
||||
if [[ "$CREATE_LATEST" == "true" ]]; then
|
||||
TAGS="$TAGS,${{ env.REGISTRY_DOCKERHUB }}:latest${VARIANT_SUFFIX},${{ env.REGISTRY_GHCR }}:latest${VARIANT_SUFFIX},${{ env.REGISTRY_QUAY }}:latest${VARIANT_SUFFIX}"
|
||||
fi
|
||||
|
||||
# Always add the channel tag for prereleases, independent of latest
|
||||
if [[ "$BUILD_TYPE" == "prerelease" ]]; then
|
||||
# Prerelease channel tags (alpha, beta, rc)
|
||||
if [[ "$VERSION" == *"alpha"* ]]; then
|
||||
CHANNEL="alpha"
|
||||
elif [[ "$VERSION" == *"beta"* ]]; then
|
||||
CHANNEL="beta"
|
||||
elif [[ "$VERSION" == *"rc"* ]]; then
|
||||
CHANNEL="rc"
|
||||
fi
|
||||
|
||||
if [[ -n "$CHANNEL" ]]; then
|
||||
TAGS="$TAGS,${{ env.REGISTRY_DOCKERHUB }}:${CHANNEL}${VARIANT_SUFFIX},${{ env.REGISTRY_GHCR }}:${CHANNEL}${VARIANT_SUFFIX},${{ env.REGISTRY_QUAY }}:${CHANNEL}${VARIANT_SUFFIX}"
|
||||
fi
|
||||
fi
|
||||
|
||||
# Output tags
|
||||
echo "tags=$TAGS" >> "$GITHUB_OUTPUT"
|
||||
|
||||
# Staging identity for this run's promotion: official tags are only
|
||||
# created after the staged image verifies against the certification,
|
||||
# and they point at this same digest (no rebuild-after-verify).
|
||||
STAGING_TAG="staging-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}${VARIANT_SUFFIX}"
|
||||
STAGING_TAGS="${{ env.REGISTRY_DOCKERHUB }}:${STAGING_TAG},${{ env.REGISTRY_GHCR }}:${STAGING_TAG},${{ env.REGISTRY_QUAY }}:${STAGING_TAG}"
|
||||
echo "staging_tags=$STAGING_TAGS" >> "$GITHUB_OUTPUT"
|
||||
echo "staging_tag=$STAGING_TAG" >> "$GITHUB_OUTPUT"
|
||||
|
||||
# Generate labels
|
||||
LABELS="org.opencontainers.image.title=RustFS"
|
||||
LABELS="$LABELS,org.opencontainers.image.description=RustFS distributed object storage system"
|
||||
LABELS="$LABELS,org.opencontainers.image.version=$VERSION"
|
||||
SOURCE_REVISION="$(git rev-parse HEAD)"
|
||||
LABELS="$LABELS,org.opencontainers.image.revision=$SOURCE_REVISION"
|
||||
LABELS="$LABELS,org.opencontainers.image.source=${{ github.server_url }}/${{ github.repository }}"
|
||||
LABELS="$LABELS,org.opencontainers.image.created=$(date -u +'%Y-%m-%dT%H:%M:%SZ')"
|
||||
LABELS="$LABELS,org.opencontainers.image.build-type=$BUILD_TYPE"
|
||||
|
||||
echo "labels=$LABELS" >> "$GITHUB_OUTPUT"
|
||||
|
||||
echo "🐳 Generated Docker tags:"
|
||||
echo "$TAGS" | tr ',' '\n' | sed 's/^/ - /'
|
||||
echo "📋 Build type: $BUILD_TYPE"
|
||||
echo "🔖 Version: $VERSION"
|
||||
|
||||
- name: Build and push Docker image
|
||||
sudo apt-get update
|
||||
sudo apt-get install -y skopeo
|
||||
- uses: docker/setup-qemu-action@c7c53464625b32c7a7e944ae62b3e17d2b600130 # v3
|
||||
if: steps.reuse.outputs.reused != 'true'
|
||||
- uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # v3
|
||||
if: steps.reuse.outputs.reused != 'true'
|
||||
- name: Supply accepted binaries and pinned CLI
|
||||
if: steps.reuse.outputs.reused != 'true'
|
||||
env:
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
run: python3 scripts/release/deliver.py image-prepare --target "$VARIANT"
|
||||
- name: Build OCI archive without registry writes
|
||||
if: steps.reuse.outputs.reused != 'true'
|
||||
uses: docker/build-push-action@10e90e3645eae34f1e60eeb005ba3a3d33f178e8 # v6
|
||||
with:
|
||||
context: .
|
||||
file: ${{ matrix.file }}
|
||||
platforms: ${{ env.DOCKER_PLATFORMS }}
|
||||
push: ${{ needs.build-check.outputs.should_push == 'true' }}
|
||||
# Push to staging tags only: official tags are created by the
|
||||
# certification-gated promotion step below from this same digest.
|
||||
tags: ${{ steps.meta.outputs.staging_tags }}
|
||||
labels: ${{ steps.meta.outputs.labels }}
|
||||
# No layer cache. This build compiles nothing — it downloads a
|
||||
# release zip and runs apk/apt — so the cache could only save the
|
||||
# minute or two those take, while creating a correctness problem: with
|
||||
# RELEASE=latest the binary URL is resolved by curl *inside* a RUN
|
||||
# layer, and the layer key does not include what that resolved to. A
|
||||
# rebuild at the same RELEASE value (dispatch with version=latest, or
|
||||
# a re-run of the same version) would hit the old layer and ship the
|
||||
# previous release's binary. mode=max also consumed the same 10GB
|
||||
# Actions cache quota the Rust lanes are fighting over.
|
||||
#
|
||||
# Only RELEASE is passed: it is the sole build-arg the Dockerfiles
|
||||
# declare besides TARGETARCH. BUILDTIME, VERSION, BUILD_TYPE, REVISION
|
||||
# and CHANNEL were never read by any stage (and BUILDTIME's $(date ...)
|
||||
# was a literal here, not a shell substitution). BUILD_DATE and VCS_REF
|
||||
# are declared by the Dockerfiles but deliberately left unset —
|
||||
# supplying them would change the published image labels.
|
||||
platforms: linux/amd64,linux/arm64
|
||||
push: false
|
||||
provenance: false
|
||||
outputs: type=oci,dest=delivery/image-${{ matrix.variant }}.tar
|
||||
build-args: |
|
||||
RELEASE=${{ steps.meta.outputs.docker_release }}
|
||||
provenance: true
|
||||
sbom: true
|
||||
# Add retry mechanism by splitting the build process
|
||||
no-cache: false
|
||||
pull: true
|
||||
|
||||
# A later-built wrapper: the staged image must complete this platform's
|
||||
# own binary identity verification before any official tag is written.
|
||||
# It may not inherit an unverified package's conclusion.
|
||||
- name: Verify staged image against certification
|
||||
if: needs.build-check.outputs.should_push == 'true'
|
||||
env:
|
||||
GH_TOKEN: ${{ secrets.RELEASE_VALIDATION_TOKEN || github.token }}
|
||||
TRIGGERING_RUN_ID: ${{ github.event.workflow_run.id }}
|
||||
VERSION: ${{ needs.build-check.outputs.version }}
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
mkdir -p certification
|
||||
run_id="$TRIGGERING_RUN_ID"
|
||||
if [[ -z "$run_id" ]]; then
|
||||
run_id="$(gh api --method GET \
|
||||
"repos/${GITHUB_REPOSITORY}/actions/workflows/build.yml/runs" \
|
||||
-f branch="$VERSION" -f status=success -F per_page=1 \
|
||||
--jq '.workflow_runs[0].id // empty' || true)"
|
||||
fi
|
||||
gh run download "$run_id" -n release-certification -D certification
|
||||
|
||||
STAGING_REF="${{ env.REGISTRY_DOCKERHUB }}:staging-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}${{ matrix.suffix }}"
|
||||
case "${{ matrix.variant }}" in
|
||||
musl) libc="musl" ;;
|
||||
glibc) libc="gnu" ;;
|
||||
esac
|
||||
for platform in linux/amd64 linux/arm64; do
|
||||
case "$platform" in
|
||||
linux/amd64) arch_sub="x86_64" ;;
|
||||
linux/arm64) arch_sub="aarch64" ;;
|
||||
esac
|
||||
docker pull --platform "$platform" "$STAGING_REF"
|
||||
container="$(docker create --platform "$platform" "$STAGING_REF")"
|
||||
docker cp "$container:/usr/bin/rustfs" "$RUNNER_TEMP/staged-rustfs"
|
||||
docker rm "$container" >/dev/null
|
||||
actual="$(sha256sum "$RUNNER_TEMP/staged-rustfs" | awk '{print $1}')"
|
||||
expected="$(jq -r --arg needle "linux-${arch_sub}-${libc}" \
|
||||
'[.deliverables[] | select(.name | contains($needle)) | .binary_sha256][0] // empty' \
|
||||
certification/candidate.json)"
|
||||
if [[ -z "$expected" || "$actual" != "$expected" ]]; then
|
||||
echo "Staged image binary for ${platform} does not match the certified binary digest" >&2
|
||||
exit 1
|
||||
fi
|
||||
python3 scripts/release/release_gate.py verify-promotion \
|
||||
--plan .config/release-validation-plan.json \
|
||||
--candidate certification/candidate.json \
|
||||
--summary certification/release-gate-output \
|
||||
--artifact "$RUNNER_TEMP/staged-rustfs" \
|
||||
--artifact-role binary
|
||||
done
|
||||
|
||||
- name: Promote staged image to official tags
|
||||
if: needs.build-check.outputs.should_push == 'true'
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
STAGING_REF="${{ env.REGISTRY_DOCKERHUB }}:staging-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}${{ matrix.suffix }}"
|
||||
IFS=',' read -r -a official_tags <<< "${{ steps.meta.outputs.tags }}"
|
||||
for tag in "${official_tags[@]}"; do
|
||||
docker buildx imagetools create -t "$tag" "$STAGING_REF"
|
||||
echo "Promoted $tag -> $STAGING_REF"
|
||||
done
|
||||
|
||||
# Downstream consumers (Helm) verify this record instead of inheriting
|
||||
# an unverified wrapper's conclusion.
|
||||
- name: Record image promotion
|
||||
if: needs.build-check.outputs.should_push == 'true'
|
||||
shell: bash
|
||||
env:
|
||||
GH_TOKEN: ${{ secrets.RELEASE_VALIDATION_TOKEN || github.token }}
|
||||
TRIGGERING_RUN_ID: ${{ github.event.workflow_run.id }}
|
||||
VERSION: ${{ needs.build-check.outputs.version }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
mkdir -p certification
|
||||
run_id="$TRIGGERING_RUN_ID"
|
||||
if [[ -z "$run_id" ]]; then
|
||||
run_id="$(gh api --method GET \
|
||||
"repos/${GITHUB_REPOSITORY}/actions/workflows/build.yml/runs" \
|
||||
-f branch="$VERSION" -f status=success -F per_page=1 \
|
||||
--jq '.workflow_runs[0].id // empty' || true)"
|
||||
fi
|
||||
gh run download "$run_id" -n release-certification -D certification
|
||||
STAGING_REF="${{ env.REGISTRY_DOCKERHUB }}:staging-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}${{ matrix.suffix }}"
|
||||
digest="$(docker buildx imagetools inspect "$STAGING_REF" --format '{{json .Manifest}}' | jq -r '.digest')"
|
||||
candidate_sha="$(python3 -c "import json,hashlib,sys; sys.path.insert(0,'scripts/release'); import release_gate; print(release_gate.canonical_sha256(json.load(open('certification/candidate.json'))))")"
|
||||
jq -n \
|
||||
--arg version "$VERSION" \
|
||||
--arg digest "$digest" \
|
||||
--arg staging_ref "$STAGING_REF" \
|
||||
--arg candidate_sha256 "$candidate_sha" \
|
||||
--arg tags "${{ steps.meta.outputs.tags }}" \
|
||||
--slurpfile candidate certification/candidate.json \
|
||||
'{schema_version: 1, version: $version, image_digest: $digest, staging_ref: $staging_ref,
|
||||
promoted_tags: ($tags | split(",")), candidate_sha256: $candidate_sha256,
|
||||
binary_sha256s: ($candidate[0].deliverables | map(.binary_sha256) | unique)}' \
|
||||
> image-promotion-record.json
|
||||
|
||||
- name: Upload image promotion record
|
||||
if: needs.build-check.outputs.should_push == 'true'
|
||||
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
|
||||
with:
|
||||
name: image-promotion-record${{ matrix.suffix }}
|
||||
path: image-promotion-record.json
|
||||
if-no-files-found: error
|
||||
retention-days: 30
|
||||
|
||||
# Note: Manifest creation is no longer needed as we only build one variant
|
||||
# Multi-arch manifests are automatically created by docker/build-push-action
|
||||
|
||||
# Report-only container image vulnerability scan
|
||||
scan-docker-image:
|
||||
name: Scan Docker Images
|
||||
needs: [ build-check, build-docker ]
|
||||
if: needs.build-check.outputs.should_build == 'true' && needs.build-check.outputs.should_push == 'true'
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 30
|
||||
permissions:
|
||||
contents: read
|
||||
security-events: write
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
include:
|
||||
- variant: musl
|
||||
suffix: ""
|
||||
- variant: glibc
|
||||
suffix: "-glibc"
|
||||
steps:
|
||||
- name: Login to GitHub Container Registry
|
||||
# docker/login-action v3
|
||||
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9
|
||||
with:
|
||||
registry: ghcr.io
|
||||
username: ${{ secrets.GHCR_USERNAME }}
|
||||
password: ${{ secrets.GHCR_PASSWORD }}
|
||||
|
||||
- name: Scan image with Trivy
|
||||
# aquasecurity/trivy-action v0.36.0
|
||||
BINARY_SOURCE=supplied
|
||||
RELEASE=${{ fromJson(inputs.candidate_manifest).version }}
|
||||
VCS_REF=${{ fromJson(inputs.candidate_manifest).source_sha }}
|
||||
- name: Verify wrapped binaries and startup on both architectures
|
||||
if: steps.reuse.outputs.reused != 'true'
|
||||
run: python3 scripts/release/deliver.py image-smoke --target "$VARIANT"
|
||||
- name: Scan amd64 image before sealing
|
||||
if: steps.reuse.outputs.reused != 'true'
|
||||
uses: aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25
|
||||
env:
|
||||
TRIVY_PLATFORM: linux/amd64
|
||||
with:
|
||||
image-ref: ${{ env.REGISTRY_GHCR }}:${{ needs.build-check.outputs.version }}${{ matrix.suffix }}
|
||||
input: delivery-scan/image-${{ matrix.variant }}-amd64.tar
|
||||
format: sarif
|
||||
output: trivy-${{ matrix.variant }}.sarif
|
||||
ignore-unfixed: true
|
||||
vuln-type: os,library
|
||||
severity: CRITICAL,HIGH
|
||||
output: delivery/scan-${{ matrix.variant }}-amd64.sarif
|
||||
exit-code: "0" # Keep the repository's report-only vulnerability policy.
|
||||
- name: Scan arm64 image before sealing
|
||||
if: steps.reuse.outputs.reused != 'true'
|
||||
uses: aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25
|
||||
env:
|
||||
TRIVY_PLATFORM: linux/arm64
|
||||
with:
|
||||
input: delivery-scan/image-${{ matrix.variant }}-arm64.tar
|
||||
format: sarif
|
||||
output: delivery/scan-${{ matrix.variant }}-arm64.sarif
|
||||
exit-code: "0"
|
||||
|
||||
# Surface findings in the Security tab; an artifact alone is write-only
|
||||
# reporting nobody reviews.
|
||||
- name: Upload scan results to code scanning
|
||||
uses: github/codeql-action/upload-sarif@99df26d4f13ea111d4ec1a7dddef6063f76b97e9 # v4.37.0
|
||||
- name: Record verified OCI identity
|
||||
if: steps.reuse.outputs.reused != 'true'
|
||||
run: python3 scripts/release/deliver.py image-record --target "$VARIANT"
|
||||
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
|
||||
if: steps.reuse.outputs.reused != 'true'
|
||||
with:
|
||||
sarif_file: trivy-${{ matrix.variant }}.sarif
|
||||
category: container-image-${{ matrix.variant }}
|
||||
|
||||
- name: Upload container scan report
|
||||
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
|
||||
with:
|
||||
name: container-image-scan-${{ matrix.variant }}
|
||||
path: trivy-${{ matrix.variant }}.sarif
|
||||
name: image-${{ matrix.variant }}
|
||||
path: delivery/*
|
||||
if-no-files-found: error
|
||||
retention-days: 30
|
||||
|
||||
# Docker build summary
|
||||
docker-summary:
|
||||
name: Docker Build Summary
|
||||
needs: [ build-check, build-docker ]
|
||||
if: always() && needs.build-check.outputs.should_build == 'true'
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 10
|
||||
steps:
|
||||
- name: Docker build completion summary
|
||||
run: |
|
||||
BUILD_TYPE="${{ needs.build-check.outputs.build_type }}"
|
||||
VERSION="${{ needs.build-check.outputs.version }}"
|
||||
CREATE_LATEST="${{ needs.build-check.outputs.create_latest }}"
|
||||
|
||||
echo "🐳 Docker build completed successfully!"
|
||||
echo "📦 Build type: $BUILD_TYPE"
|
||||
echo "🔢 Version: $VERSION"
|
||||
echo "🚀 Strategy: Images using pre-built binaries (release channel only)"
|
||||
echo ""
|
||||
|
||||
case "$BUILD_TYPE" in
|
||||
"preview")
|
||||
echo "🔍 Preview Docker image has been built with ${VERSION} tags"
|
||||
echo "⏭️ Preview images do not update latest or prerelease channels"
|
||||
;;
|
||||
"release")
|
||||
echo "🚀 Release Docker image has been built with ${VERSION} tags"
|
||||
echo "✅ This image is ready for production use"
|
||||
if [[ "$CREATE_LATEST" == "true" ]]; then
|
||||
echo "🏷️ Latest tag has been created for stable release"
|
||||
fi
|
||||
;;
|
||||
"prerelease")
|
||||
echo "🧪 Prerelease Docker image has been built with ${VERSION} tags"
|
||||
echo "⚠️ This is a prerelease image - use with caution"
|
||||
# Prereleases move latest until the first stable tag exists (pre-GA policy).
|
||||
if [[ "$CREATE_LATEST" == "true" ]]; then
|
||||
echo "🏷️ Latest tag has been created for prerelease: $VERSION"
|
||||
else
|
||||
echo "🚫 Latest tag NOT created for prerelease"
|
||||
fi
|
||||
;;
|
||||
*)
|
||||
echo "❌ Unexpected build type: $BUILD_TYPE"
|
||||
;;
|
||||
esac
|
||||
compression-level: 0
|
||||
|
||||
@@ -1,229 +1,26 @@
|
||||
# Copyright 2024 RustFS Team
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
name: Publish helm chart to artifacthub
|
||||
|
||||
# Copyright 2026 RustFS Team
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
name: Build candidate Helm chart
|
||||
on:
|
||||
workflow_run:
|
||||
workflows: [ "Build and Release" ]
|
||||
types: [ completed ]
|
||||
workflow_dispatch:
|
||||
workflow_call:
|
||||
inputs:
|
||||
version:
|
||||
description: "Release version to publish, e.g. 1.0.0-beta.1 or v1.0.0-beta.1"
|
||||
required: true
|
||||
default: "1.0.0-beta.1"
|
||||
candidate_manifest:
|
||||
type: string
|
||||
|
||||
required: true
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
actions: read
|
||||
jobs:
|
||||
# Consume the same trusted release certification as every other publication
|
||||
# exit and verify the image promotion record of the Docker workflow. The
|
||||
# chart wraps no binary of its own, so it may only reference an image whose
|
||||
# platform verification already passed; it must not inherit an unverified
|
||||
# package's conclusion. A run without that evidence fails closed here and
|
||||
# never reaches the chart repository.
|
||||
release-certification:
|
||||
package:
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 30
|
||||
permissions:
|
||||
contents: read
|
||||
actions: read
|
||||
outputs:
|
||||
version: ${{ steps.version.outputs.raw_tag }}
|
||||
timeout-minutes: 15
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
|
||||
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
|
||||
with:
|
||||
ref: ${{ fromJson(inputs.candidate_manifest).source_sha }}
|
||||
persist-credentials: false
|
||||
|
||||
- name: Normalize release version
|
||||
id: version
|
||||
env:
|
||||
RAW_INPUT: ${{ github.event.inputs.version }}
|
||||
RAW_BRANCH: ${{ github.event.workflow_run.head_branch }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then
|
||||
RAW="$RAW_INPUT"
|
||||
else
|
||||
RAW="$RAW_BRANCH"
|
||||
fi
|
||||
case "$RAW" in
|
||||
refs/tags/*) RAW_TAG="${RAW#refs/tags/}" ;;
|
||||
*) RAW_TAG="$RAW" ;;
|
||||
esac
|
||||
echo "raw_tag=$RAW_TAG" >> "$GITHUB_OUTPUT"
|
||||
|
||||
- name: Download and verify release certification and image record
|
||||
env:
|
||||
GH_TOKEN: ${{ secrets.RELEASE_VALIDATION_TOKEN || github.token }}
|
||||
TRIGGERING_RUN_ID: ${{ github.event.workflow_run.id }}
|
||||
RAW_TAG: ${{ steps.version.outputs.raw_tag }}
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
mkdir -p certification image-record
|
||||
|
||||
build_run="$TRIGGERING_RUN_ID"
|
||||
if [[ -z "$build_run" ]]; then
|
||||
build_run="$(gh api --method GET \
|
||||
"repos/${GITHUB_REPOSITORY}/actions/workflows/build.yml/runs" \
|
||||
-f branch="$RAW_TAG" -f status=success -F per_page=1 \
|
||||
--jq '.workflow_runs[0].id // empty' || true)"
|
||||
fi
|
||||
if [[ -z "$build_run" ]]; then
|
||||
echo "No certified Build and Release run found for ${RAW_TAG}; failing closed" >&2
|
||||
exit 1
|
||||
fi
|
||||
if ! gh run download "$build_run" -n release-certification -D certification; then
|
||||
echo "Build run ${build_run} published no release-certification artifact; failing closed" >&2
|
||||
exit 1
|
||||
fi
|
||||
python3 scripts/release/release_gate.py verify-promotion \
|
||||
--plan .config/release-validation-plan.json \
|
||||
--candidate certification/candidate.json \
|
||||
--summary certification/release-gate-output \
|
||||
--artifact-sha256 "$(jq -r '.deliverables[0].sha256' certification/candidate.json)" \
|
||||
--artifact-name "$(jq -r '.deliverables[0].name' certification/candidate.json)"
|
||||
|
||||
docker_run="$(gh api --method GET \
|
||||
"repos/${GITHUB_REPOSITORY}/actions/workflows/docker.yml/runs" \
|
||||
-f branch="$RAW_TAG" -f status=success -F per_page=1 \
|
||||
--jq '.workflow_runs[0].id // empty' || true)"
|
||||
if [[ -z "$docker_run" ]] || ! gh run download "$docker_run" -n image-promotion-record -D image-record; then
|
||||
echo "No image promotion record for ${RAW_TAG}; a chart may not wrap an unverified image" >&2
|
||||
exit 1
|
||||
fi
|
||||
candidate_sha="$(python3 -c "import json,hashlib,sys; sys.path.insert(0,'scripts/release'); import release_gate; print(release_gate.canonical_sha256(json.load(open('certification/candidate.json'))))")"
|
||||
record_sha="$(jq -r '.candidate_sha256 // empty' image-record/image-promotion-record.json)"
|
||||
if [[ "$record_sha" != "$candidate_sha" ]]; then
|
||||
echo "Image promotion record does not belong to this certification" >&2
|
||||
exit 1
|
||||
fi
|
||||
overlap="$(jq -r --slurpfile candidate certification/candidate.json \
|
||||
'[.binary_sha256s[]] - ($candidate[0].deliverables | map(.binary_sha256)) | length' \
|
||||
image-record/image-promotion-record.json)"
|
||||
if [[ "$overlap" -ne 0 ]]; then
|
||||
echo "Image promotion record carries binary digests outside the certification" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
build-helm-package:
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 30
|
||||
needs: [ release-certification ]
|
||||
if: |
|
||||
(github.event_name == 'workflow_dispatch' && !contains(github.event.inputs.version, '-preview')) ||
|
||||
(
|
||||
github.event.workflow_run.conclusion == 'success' &&
|
||||
github.event.workflow_run.event == 'push' &&
|
||||
contains(github.event.workflow_run.head_branch, '.') &&
|
||||
!contains(github.event.workflow_run.head_branch, '-preview')
|
||||
)
|
||||
|
||||
outputs:
|
||||
raw_tag: ${{ steps.version.outputs.raw_tag }}
|
||||
app_version: ${{ steps.version.outputs.app_version }}
|
||||
chart_version: ${{ steps.version.outputs.chart_version }}
|
||||
|
||||
steps:
|
||||
- name: Checkout release source
|
||||
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
|
||||
with:
|
||||
ref: ${{ github.event.workflow_run.head_sha || (startsWith(inputs.version, 'refs/tags/') && inputs.version || format('refs/tags/{0}', inputs.version)) }}
|
||||
persist-credentials: false
|
||||
|
||||
# Both inputs reach the shell through env rather than `${{ }}`
|
||||
# interpolation. A git ref name may contain `$(...)` — anything without a
|
||||
# space is a legal tag — and interpolation pastes it into the script
|
||||
# verbatim, where bash would run it. Reading "$RAW_INPUT" instead makes it
|
||||
# data.
|
||||
- name: Normalize release version
|
||||
id: version
|
||||
env:
|
||||
RAW_INPUT: ${{ github.event.inputs.version }}
|
||||
RAW_BRANCH: ${{ github.event.workflow_run.head_branch }}
|
||||
run: |
|
||||
set -eux
|
||||
|
||||
if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then
|
||||
RAW="$RAW_INPUT"
|
||||
else
|
||||
RAW="$RAW_BRANCH"
|
||||
fi
|
||||
|
||||
case "$RAW" in
|
||||
refs/tags/*)
|
||||
RAW_TAG="${RAW#refs/tags/}"
|
||||
;;
|
||||
*)
|
||||
RAW_TAG="$RAW"
|
||||
;;
|
||||
esac
|
||||
|
||||
./scripts/helm_chart_version.sh "$RAW_TAG"
|
||||
|
||||
# Docker and Helm share the binary-build trigger. Do not publish a chart
|
||||
# until the image selected by its default appVersion is pullable.
|
||||
- name: Wait for release image
|
||||
timeout-minutes: 20
|
||||
env:
|
||||
APP_VERSION: ${{ steps.version.outputs.app_version }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
if [[ ! "$APP_VERSION" =~ ^[0-9]+\.[0-9]+\.[0-9]+(-[0-9A-Za-z.-]+)?$ ]] ||
|
||||
[[ "$APP_VERSION" == *-preview* ]]; then
|
||||
echo "Invalid deliverable image version: $APP_VERSION" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
for attempt in {1..60}; do
|
||||
if token=$(curl --fail --silent --show-error --connect-timeout 10 --max-time 20 \
|
||||
'https://auth.docker.io/token?service=registry.docker.io&scope=repository:rustfs/rustfs:pull' \
|
||||
| jq -er '.token // empty') &&
|
||||
curl --fail --silent --show-error --head --connect-timeout 10 --max-time 20 \
|
||||
--header "Authorization: Bearer $token" \
|
||||
--header 'Accept: application/vnd.oci.image.index.v1+json, application/vnd.docker.distribution.manifest.list.v2+json' \
|
||||
"https://registry-1.docker.io/v2/rustfs/rustfs/manifests/$APP_VERSION" > /dev/null; then
|
||||
echo "Release image is available: rustfs/rustfs:$APP_VERSION"
|
||||
exit 0
|
||||
fi
|
||||
echo "Waiting for rustfs/rustfs:$APP_VERSION ($attempt/60)"
|
||||
if (( attempt < 60 )); then
|
||||
sleep 15
|
||||
fi
|
||||
done
|
||||
|
||||
echo "Release image is unavailable; refusing to publish Helm chart" >&2
|
||||
exit 1
|
||||
|
||||
- name: Replace chart version and app version
|
||||
env:
|
||||
CHART_VERSION: ${{ steps.version.outputs.chart_version }}
|
||||
APP_VERSION: ${{ steps.version.outputs.app_version }}
|
||||
run: |
|
||||
set -eux
|
||||
sed -i -E "s/^version:.*/version: \"${CHART_VERSION}\"/" helm/rustfs/Chart.yaml
|
||||
sed -i -E "s/^appVersion:.*/appVersion: \"${APP_VERSION}\"/" helm/rustfs/Chart.yaml
|
||||
|
||||
- name: Set up Helm
|
||||
uses: azure/setup-helm@b9e51907a09c216f16ebe8536097933489208112 # v4.3.0
|
||||
|
||||
- name: Install yq
|
||||
- uses: azure/setup-helm@b9e51907a09c216f16ebe8536097933489208112 # v4.3.0
|
||||
- name: Install pinned yq
|
||||
run: |
|
||||
set -euo pipefail
|
||||
yq_version="4.54.1"
|
||||
@@ -238,66 +35,43 @@ jobs:
|
||||
echo "$tools_dir" >> "$GITHUB_PATH"
|
||||
"$tools_dir/yq" --version
|
||||
|
||||
- name: Test Helm Chart Templates
|
||||
run: ./scripts/test_helm_templates.sh
|
||||
|
||||
- name: Package Helm Chart
|
||||
run: |
|
||||
set -eux
|
||||
cp helm/README.md helm/rustfs/
|
||||
helm package ./helm/rustfs \
|
||||
--destination helm/rustfs/ \
|
||||
--version "${{ steps.version.outputs.chart_version }}"
|
||||
|
||||
- name: Upload helm package as artifact
|
||||
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
|
||||
with:
|
||||
name: helm-package
|
||||
path: |
|
||||
helm/rustfs/*.tgz
|
||||
helm/rustfs/README.md
|
||||
retention-days: 1
|
||||
|
||||
publish-helm-package:
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 30
|
||||
needs: [ build-helm-package ]
|
||||
if: needs.build-helm-package.result == 'success'
|
||||
|
||||
steps:
|
||||
- name: Checkout helm package repo
|
||||
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
|
||||
with:
|
||||
# persist-credentials-exempt: this checkout's token IS the push credential —
|
||||
# the job git-pushes to rustfs/helm below. Clearing it breaks chart publishing.
|
||||
repository: rustfs/helm
|
||||
token: ${{ secrets.RUSTFS_HELM_PACKAGE }}
|
||||
|
||||
- name: Download helm package
|
||||
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
|
||||
with:
|
||||
name: helm-package
|
||||
path: ./
|
||||
|
||||
- name: Set up helm
|
||||
uses: azure/setup-helm@b9e51907a09c216f16ebe8536097933489208112 # v4.3.0
|
||||
|
||||
- name: Generate index
|
||||
run: helm repo index . --url https://charts.rustfs.com
|
||||
|
||||
# app_version is derived from the triggering tag name, and this job holds
|
||||
# the cross-repository push token with rustfs/helm already checked out —
|
||||
# the worst place in the repo to paste an attacker-influenced string into
|
||||
# a shell line. Passed through env so bash treats it as data.
|
||||
- name: Push helm package and index file
|
||||
- name: Test and package the exact application version
|
||||
id: chart
|
||||
env:
|
||||
GIT_USERNAME: ${{ secrets.USERNAME }}
|
||||
GIT_EMAIL: ${{ secrets.EMAIL_ADDRESS }}
|
||||
APP_VERSION: ${{ needs.build-helm-package.outputs.app_version }}
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
CANDIDATE_MANIFEST: ${{ inputs.candidate_manifest }}
|
||||
run: |
|
||||
set -eux
|
||||
git config --global user.name "${GIT_USERNAME}"
|
||||
git config --global user.email "${GIT_EMAIL}"
|
||||
git add .
|
||||
git commit -m "Update rustfs helm package with ${APP_VERSION}." || echo "No changes to commit"
|
||||
git push origin main
|
||||
python3 - <<'PYCHART'
|
||||
import sys, subprocess, shutil
|
||||
from pathlib import Path
|
||||
sys.path.insert(0, 'scripts/release')
|
||||
import candidate as c, pipeline
|
||||
inputs = pipeline.manifest_env()
|
||||
current = pipeline.current_run('release-complete.yml')
|
||||
if any(a['name'] == 'helm-package' for a in c.artifact_list(current['id'])):
|
||||
c.download_artifact(current, 'helm-package', 'chart')
|
||||
record = c.read_json('chart/chart.json')
|
||||
c.require(record['inputs_sha256'] == c.digest(inputs), 'chart belongs to another candidate')
|
||||
c.emit_output('reused', 'true')
|
||||
else:
|
||||
chart = Path('helm/rustfs/Chart.yaml')
|
||||
import re
|
||||
value = chart.read_text()
|
||||
for key in ('version', 'appVersion'):
|
||||
value, count = re.subn(rf'(?m)^{key}:.*$', f'{key}: "{inputs["version"]}"', value)
|
||||
c.require(count == 1, 'ambiguous chart metadata')
|
||||
chart.write_text(value)
|
||||
subprocess.run(['bash', 'scripts/test_helm_templates.sh'], check=True)
|
||||
shutil.copyfile('helm/README.md', 'helm/rustfs/README.md')
|
||||
subprocess.run(['helm', 'package', 'helm/rustfs', '--destination', 'chart'], check=True)
|
||||
path = Path('chart') / f"rustfs-{inputs['version']}.tgz"
|
||||
c.write_json('chart/chart.json', {'version': inputs['version'], 'inputs_sha256': c.digest(inputs),
|
||||
'name': path.name, 'sha256': c.file_digest(path), 'size': path.stat().st_size})
|
||||
PYCHART
|
||||
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
|
||||
if: steps.chart.outputs.reused != 'true'
|
||||
with:
|
||||
name: helm-package
|
||||
path: chart/*
|
||||
if-no-files-found: error
|
||||
retention-days: 30
|
||||
|
||||
@@ -47,8 +47,8 @@ permissions:
|
||||
contents: read
|
||||
|
||||
concurrency:
|
||||
group: oidc-keycloak-live-${{ github.ref }}
|
||||
cancel-in-progress: true
|
||||
group: oidc-keycloak-live-${{ github.event_name }}-${{ github.ref }}
|
||||
cancel-in-progress: ${{ github.event_name != 'schedule' }}
|
||||
|
||||
jobs:
|
||||
oidc-keycloak-live:
|
||||
|
||||
+38
-825
@@ -1,840 +1,53 @@
|
||||
# Copyright 2024 RustFS Team
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
# Package Workflow - Build DEB/RPM packages
|
||||
#
|
||||
# This workflow builds DEB and RPM packages from pre-built Linux binaries
|
||||
# and uploads them to Cloudflare R2 and the GitHub release.
|
||||
#
|
||||
# Trigger:
|
||||
# - workflow_run: automatically package after "Build and Release" completes
|
||||
# for a release tag (the mac/windows/linux binaries are already uploaded
|
||||
# to the GitHub release before packaging starts)
|
||||
# - workflow_dispatch: manual fallback with a release tag and/or exact build run ID
|
||||
#
|
||||
# Flow:
|
||||
# 1. Resolve and validate the selected Build workflow run and source identity
|
||||
# 2. Download Linux binaries (x86_64-gnu, aarch64-gnu, x86_64-musl,
|
||||
# aarch64-musl) from build artifacts
|
||||
# 3. Build DEB packages for amd64 and arm64, gnu and musl variants
|
||||
# 4. Build RPM packages for x86_64 and aarch64, gnu and musl variants
|
||||
# 5. Upload all packages to Cloudflare R2 and the GitHub release
|
||||
|
||||
name: Package DEB/RPM
|
||||
|
||||
permissions:
|
||||
# contents: write is required to upload packages to the GitHub release
|
||||
contents: write
|
||||
actions: read
|
||||
|
||||
# Copyright 2026 RustFS Team
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
name: Build candidate packages
|
||||
on:
|
||||
# Follows the same pattern as docker.yml: run after the release build
|
||||
# workflow completes, so packaging is triggered only by release tags
|
||||
# (e.g. 1.0.0-rc.2, 1.0.0-rc.3), never by development builds.
|
||||
workflow_run:
|
||||
workflows: [ "Build and Release" ]
|
||||
types: [ completed ]
|
||||
workflow_dispatch:
|
||||
workflow_call:
|
||||
inputs:
|
||||
tag:
|
||||
description: "Release tag to package (e.g. 1.0.0-beta.12). Leave empty for latest main build."
|
||||
required: false
|
||||
candidate_manifest:
|
||||
type: string
|
||||
build_run_id:
|
||||
description: "Build workflow run ID (when combined with tag, both must identify the same release commit)"
|
||||
required: false
|
||||
type: string
|
||||
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.event.workflow_run.head_branch || github.event.inputs.tag || github.run_id }}
|
||||
cancel-in-progress: true
|
||||
|
||||
env:
|
||||
HEAD_BRANCH: ${{ github.event.workflow_run.head_branch }}
|
||||
WORKFLOW_RUN_ID: ${{ github.event.workflow_run.id }}
|
||||
|
||||
required: true
|
||||
permissions:
|
||||
contents: read
|
||||
actions: read
|
||||
jobs:
|
||||
# Resolve which build run to use and extract version info
|
||||
resolve:
|
||||
name: Resolve Build
|
||||
# Auto-trigger only from successful tag builds of "Build and Release".
|
||||
# Tag pushes arrive as event == push with head_branch != main (a
|
||||
# non-main push head_branch is the release tag name). Manual dispatch
|
||||
# stays available as a fallback for backfills and re-runs.
|
||||
if: >-
|
||||
github.event_name == 'workflow_dispatch' ||
|
||||
(github.event.workflow_run.conclusion == 'success' &&
|
||||
github.event.workflow_run.event == 'push' &&
|
||||
github.event.workflow_run.head_branch != 'main')
|
||||
runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
|
||||
timeout-minutes: 10
|
||||
outputs:
|
||||
version: ${{ steps.resolve.outputs.version }}
|
||||
build_type: ${{ steps.resolve.outputs.build_type }}
|
||||
build_run_id: ${{ steps.resolve.outputs.build_run_id }}
|
||||
build_run_number: ${{ steps.resolve.outputs.build_run_number }}
|
||||
head_sha: ${{ steps.resolve.outputs.head_sha }}
|
||||
dev_sequence: ${{ steps.resolve.outputs.dev_sequence }}
|
||||
tag: ${{ steps.resolve.outputs.tag }}
|
||||
steps:
|
||||
- name: Resolve build run
|
||||
id: resolve
|
||||
shell: bash
|
||||
env:
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
EVENT_NAME: ${{ github.event_name }}
|
||||
REPOSITORY: ${{ github.repository }}
|
||||
INPUT_TAG: ${{ github.event.inputs.tag }}
|
||||
INPUT_RUN_ID: ${{ github.event.inputs.build_run_id }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
|
||||
fail() {
|
||||
echo "❌ $1" >&2
|
||||
exit 1
|
||||
}
|
||||
|
||||
TAG=""
|
||||
BUILD_RUN_ID=""
|
||||
case "$EVENT_NAME" in
|
||||
workflow_run)
|
||||
TAG="$HEAD_BRANCH"
|
||||
BUILD_RUN_ID="$WORKFLOW_RUN_ID"
|
||||
;;
|
||||
workflow_dispatch)
|
||||
TAG="$INPUT_TAG"
|
||||
BUILD_RUN_ID="$INPUT_RUN_ID"
|
||||
;;
|
||||
*) fail "unsupported event: $EVENT_NAME" ;;
|
||||
esac
|
||||
|
||||
# Validate and classify tags before using them in API paths or logs.
|
||||
semver_core='(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)'
|
||||
prerelease_id='(alpha|beta|rc)\.(0|[1-9][0-9]*)'
|
||||
if [[ -n "$TAG" ]]; then
|
||||
if [[ "$TAG" =~ ^${semver_core}(-${prerelease_id})?-preview\.(0|[1-9][0-9]*)$ ]]; then
|
||||
BUILD_TYPE=preview
|
||||
elif [[ "$TAG" =~ ^${semver_core}-${prerelease_id}$ ]]; then
|
||||
BUILD_TYPE=prerelease
|
||||
elif [[ "$TAG" =~ ^${semver_core}$ ]]; then
|
||||
BUILD_TYPE=release
|
||||
else
|
||||
fail "tag is not a supported strict package version"
|
||||
fi
|
||||
else
|
||||
BUILD_TYPE=development
|
||||
fi
|
||||
|
||||
if [[ -n "$BUILD_RUN_ID" ]]; then
|
||||
[[ "$BUILD_RUN_ID" =~ ^[1-9][0-9]*$ ]] || fail "build run ID must be a positive decimal integer"
|
||||
echo "Using selected build run: $BUILD_RUN_ID"
|
||||
elif [[ -n "$TAG" ]]; then
|
||||
echo "Looking for build run for tag: $TAG"
|
||||
BUILD_RUN_ID=$(gh api --method GET \
|
||||
"repos/${REPOSITORY}/actions/workflows/build.yml/runs" \
|
||||
-f branch="$TAG" -f status=success -F per_page=1 \
|
||||
--jq '.workflow_runs[0].id // empty' 2>/dev/null || true)
|
||||
|
||||
if [[ -z "$BUILD_RUN_ID" ]]; then
|
||||
BUILD_RUN_ID=$(gh api --method GET \
|
||||
"repos/${REPOSITORY}/actions/workflows/build.yml/runs" \
|
||||
-f event=push -f status=success -F per_page=100 2>/dev/null |
|
||||
jq -r --arg tag "$TAG" \
|
||||
'[.workflow_runs[] | select(.head_branch == $tag)][0].id // empty' || true)
|
||||
fi
|
||||
[[ "$BUILD_RUN_ID" =~ ^[1-9][0-9]*$ ]] || fail "no successful build run found for tag"
|
||||
echo "Found build run: $BUILD_RUN_ID"
|
||||
else
|
||||
echo "No tag specified, looking for latest main build"
|
||||
BUILD_RUN_ID=$(gh api --method GET \
|
||||
"repos/${REPOSITORY}/actions/workflows/build.yml/runs" \
|
||||
-f branch=main -f status=success -F per_page=1 \
|
||||
--jq '.workflow_runs[0].id // empty' 2>/dev/null || true)
|
||||
[[ "$BUILD_RUN_ID" =~ ^[1-9][0-9]*$ ]] || fail "no successful main build found"
|
||||
echo "Latest main build: $BUILD_RUN_ID"
|
||||
fi
|
||||
|
||||
# Fetch once and use the same immutable run metadata for identity,
|
||||
# ordering, workflow provenance, and release-channel validation.
|
||||
RUN_JSON=$(gh api "repos/${REPOSITORY}/actions/runs/${BUILD_RUN_ID}") ||
|
||||
fail "cannot read selected build run"
|
||||
RUN_ID=$(jq -r '.id // empty' <<<"$RUN_JSON")
|
||||
RUN_NUMBER=$(jq -r '.run_number // empty' <<<"$RUN_JSON")
|
||||
RUN_STATUS=$(jq -r '.status // empty' <<<"$RUN_JSON")
|
||||
RUN_CONCLUSION=$(jq -r '.conclusion // empty' <<<"$RUN_JSON")
|
||||
RUN_PATH=$(jq -r '.path // empty' <<<"$RUN_JSON")
|
||||
HEAD_SHA=$(jq -r '.head_sha // empty' <<<"$RUN_JSON")
|
||||
RUN_HEAD_BRANCH=$(jq -r '.head_branch // empty' <<<"$RUN_JSON")
|
||||
|
||||
[[ "$RUN_ID" == "$BUILD_RUN_ID" ]] || fail "run metadata ID mismatch"
|
||||
[[ "$RUN_NUMBER" =~ ^[1-9][0-9]*$ ]] || fail "build run number must be a positive decimal integer"
|
||||
[[ "$RUN_STATUS" == completed && "$RUN_CONCLUSION" == success ]] || fail "selected build run is not successful"
|
||||
[[ "$RUN_PATH" == .github/workflows/build.yml ]] || fail "selected run is not Build and Release"
|
||||
[[ "$HEAD_SHA" =~ ^[0-9a-f]{40}$ ]] || fail "selected build run has an invalid head SHA"
|
||||
[[ "$RUN_HEAD_BRANCH" != *$'\n'* && -n "$RUN_HEAD_BRANCH" ]] || fail "selected build run has an invalid head branch"
|
||||
|
||||
if [[ -n "$TAG" ]]; then
|
||||
[[ "$RUN_HEAD_BRANCH" == "$TAG" ]] || fail "tag and build run head branch do not match"
|
||||
|
||||
TAG_REF_JSON=$(gh api "repos/${REPOSITORY}/git/ref/tags/${TAG}") ||
|
||||
fail "cannot resolve release tag ref"
|
||||
TAG_OBJECT_TYPE=$(jq -r '.object.type // empty' <<<"$TAG_REF_JSON")
|
||||
TAG_OBJECT_SHA=$(jq -r '.object.sha // empty' <<<"$TAG_REF_JSON")
|
||||
depth=0
|
||||
while [[ "$TAG_OBJECT_TYPE" == tag && $depth -lt 5 ]]; do
|
||||
TAG_OBJECT_JSON=$(gh api "repos/${REPOSITORY}/git/tags/${TAG_OBJECT_SHA}") ||
|
||||
fail "cannot peel annotated release tag"
|
||||
TAG_OBJECT_TYPE=$(jq -r '.object.type // empty' <<<"$TAG_OBJECT_JSON")
|
||||
TAG_OBJECT_SHA=$(jq -r '.object.sha // empty' <<<"$TAG_OBJECT_JSON")
|
||||
depth=$((depth + 1))
|
||||
done
|
||||
[[ "$TAG_OBJECT_TYPE" == commit && "$TAG_OBJECT_SHA" =~ ^[0-9a-f]{40}$ ]] ||
|
||||
fail "release tag does not resolve to a commit"
|
||||
[[ "$TAG_OBJECT_SHA" == "$HEAD_SHA" ]] || fail "release tag commit and build run head SHA do not match"
|
||||
VERSION="$TAG"
|
||||
DEV_SEQUENCE=""
|
||||
else
|
||||
VERSION="dev-${HEAD_SHA}"
|
||||
DEV_SEQUENCE="$RUN_NUMBER"
|
||||
fi
|
||||
|
||||
{
|
||||
echo "version=$VERSION"
|
||||
echo "build_type=$BUILD_TYPE"
|
||||
echo "build_run_id=$BUILD_RUN_ID"
|
||||
echo "build_run_number=$RUN_NUMBER"
|
||||
echo "head_sha=$HEAD_SHA"
|
||||
echo "dev_sequence=$DEV_SEQUENCE"
|
||||
echo "tag=${TAG}"
|
||||
} >> "$GITHUB_OUTPUT"
|
||||
|
||||
echo "📊 Resolved:"
|
||||
echo " Version: $VERSION"
|
||||
echo " Build type: $BUILD_TYPE"
|
||||
echo " Build run ID: $BUILD_RUN_ID"
|
||||
echo " Build run number: $RUN_NUMBER"
|
||||
|
||||
# Consume the trusted release certification of the resolved build run and
|
||||
# verify the certified input digests before any package is wrapped. A run
|
||||
# without that certification fails closed here; packages built later must
|
||||
# complete their own verification and cannot inherit an unverified
|
||||
# conclusion.
|
||||
release-certification:
|
||||
name: Release Certification
|
||||
needs: resolve
|
||||
if: needs.resolve.result == 'success'
|
||||
runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
|
||||
timeout-minutes: 30
|
||||
permissions:
|
||||
contents: read
|
||||
actions: read
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
|
||||
with:
|
||||
persist-credentials: false
|
||||
|
||||
- name: Download and verify release certification
|
||||
env:
|
||||
GH_TOKEN: ${{ secrets.RELEASE_VALIDATION_TOKEN || github.token }}
|
||||
BUILD_RUN_ID: ${{ needs.resolve.outputs.build_run_id }}
|
||||
BUILD_TYPE: ${{ needs.resolve.outputs.build_type }}
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
mkdir -p certification
|
||||
if [[ "$BUILD_TYPE" == "development" ]]; then
|
||||
# Development artifacts are explicitly uncertified and never touch
|
||||
# official or latest channels; only the staging/dev paths remain.
|
||||
jq -n '{schema_version: 1, certified: false, reason: "development artifacts are explicitly uncertified"}' \
|
||||
> certification/development-uncertified.json
|
||||
exit 0
|
||||
fi
|
||||
if ! gh run download "$BUILD_RUN_ID" -n release-certification -D certification; then
|
||||
echo "Build run ${BUILD_RUN_ID} published no release-certification artifact; failing closed" >&2
|
||||
exit 1
|
||||
fi
|
||||
python3 scripts/release/release_gate.py verify-promotion \
|
||||
--plan .config/release-validation-plan.json \
|
||||
--candidate certification/candidate.json \
|
||||
--summary certification/release-gate-output \
|
||||
--artifact-sha256 "$(jq -r '.deliverables[0].sha256' certification/candidate.json)" \
|
||||
--artifact-name "$(jq -r '.deliverables[0].name' certification/candidate.json)"
|
||||
|
||||
# Build DEB and RPM packages for each architecture and libc variant
|
||||
package:
|
||||
name: Package (${{ matrix.arch }}-${{ matrix.libc }})
|
||||
needs: [ resolve, release-certification ]
|
||||
runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
|
||||
timeout-minutes: 30
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 45
|
||||
strategy:
|
||||
fail-fast: false
|
||||
max-parallel: 1
|
||||
matrix:
|
||||
include:
|
||||
- arch: x86_64
|
||||
libc: gnu
|
||||
deb_arch: amd64
|
||||
rpm_arch: x86_64
|
||||
artifact_name: "rustfs-linux-x86_64-gnu"
|
||||
# gnu binaries dynamically link glibc; musl binaries are static
|
||||
# and run without a libc dependency.
|
||||
deb_depends: "libc6 (>= 2.31)"
|
||||
rpm_depends: "glibc >= 2.31"
|
||||
- arch: aarch64
|
||||
libc: gnu
|
||||
deb_arch: arm64
|
||||
rpm_arch: aarch64
|
||||
artifact_name: "rustfs-linux-aarch64-gnu"
|
||||
deb_depends: "libc6 (>= 2.31)"
|
||||
rpm_depends: "glibc >= 2.31"
|
||||
- arch: x86_64
|
||||
libc: musl
|
||||
deb_arch: amd64
|
||||
rpm_arch: x86_64
|
||||
artifact_name: "rustfs-linux-x86_64-musl"
|
||||
deb_depends: ""
|
||||
rpm_depends: ""
|
||||
- arch: aarch64
|
||||
libc: musl
|
||||
deb_arch: arm64
|
||||
rpm_arch: aarch64
|
||||
artifact_name: "rustfs-linux-aarch64-musl"
|
||||
deb_depends: ""
|
||||
rpm_depends: ""
|
||||
target: [linux-x86_64-musl, linux-aarch64-musl, linux-x86_64-gnu, linux-aarch64-gnu]
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
|
||||
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
|
||||
with:
|
||||
ref: ${{ fromJson(inputs.candidate_manifest).source_sha }}
|
||||
persist-credentials: false
|
||||
|
||||
- name: Normalize package metadata
|
||||
id: versions
|
||||
shell: bash
|
||||
env:
|
||||
BUILD_TYPE: ${{ needs.resolve.outputs.build_type }}
|
||||
SOURCE_VERSION: ${{ needs.resolve.outputs.version }}
|
||||
DEV_SEQUENCE: ${{ needs.resolve.outputs.dev_sequence }}
|
||||
ARCH: ${{ matrix.rpm_arch }}
|
||||
LIBC: ${{ matrix.libc }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
|
||||
normalized=$(./scripts/release/package_versions.sh \
|
||||
"$BUILD_TYPE" "$SOURCE_VERSION" "$DEV_SEQUENCE" "$ARCH" "$LIBC")
|
||||
printf '%s\n' "$normalized" >> "$GITHUB_OUTPUT"
|
||||
|
||||
- name: Download binary artifact from build run
|
||||
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
|
||||
with:
|
||||
pattern: ${{ matrix.artifact_name }}*
|
||||
path: ./binary-artifact
|
||||
run-id: ${{ needs.resolve.outputs.build_run_id }}
|
||||
github-token: ${{ github.token }}
|
||||
merge-multiple: true
|
||||
|
||||
- name: Extract binary
|
||||
id: binary
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
|
||||
ZIP_FILE=$(find ./binary-artifact -name "*.zip" -type f | head -1)
|
||||
if [[ -z "$ZIP_FILE" ]]; then
|
||||
echo "❌ No binary artifact found"
|
||||
find ./binary-artifact -mindepth 1 -maxdepth 1 -print 2>/dev/null || true
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "Found artifact: $ZIP_FILE"
|
||||
|
||||
mkdir -p ./bin
|
||||
unzip -o "$ZIP_FILE" -d ./bin
|
||||
|
||||
if [[ ! -f ./bin/rustfs ]]; then
|
||||
echo "❌ rustfs binary not found in archive"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
chmod +x ./bin/rustfs
|
||||
stat --printf='%n %s bytes\n' ./bin/rustfs
|
||||
echo "✅ Binary extracted"
|
||||
|
||||
- name: Build DEB package
|
||||
id: deb
|
||||
shell: bash
|
||||
env:
|
||||
DEB_VERSION: ${{ steps.versions.outputs.deb_version }}
|
||||
DEB_ARCH: ${{ matrix.deb_arch }}
|
||||
DEB_FILE: ${{ steps.versions.outputs.deb_file }}
|
||||
DEB_DEPENDS: ${{ matrix.deb_depends }}
|
||||
LIBC: ${{ matrix.libc }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
|
||||
PKG_DIR="${DEB_FILE%.deb}"
|
||||
|
||||
echo "Building DEB: ${DEB_FILE}"
|
||||
|
||||
mkdir -p "${PKG_DIR}/DEBIAN"
|
||||
mkdir -p "${PKG_DIR}/usr/bin"
|
||||
mkdir -p "${PKG_DIR}/etc/default"
|
||||
mkdir -p "${PKG_DIR}/lib/systemd/system"
|
||||
mkdir -p "${PKG_DIR}/usr/share/doc/rustfs"
|
||||
|
||||
cp ./bin/rustfs "${PKG_DIR}/usr/bin/"
|
||||
chmod 755 "${PKG_DIR}/usr/bin/rustfs"
|
||||
|
||||
cp deploy/build/rustfs.service "${PKG_DIR}/lib/systemd/system/"
|
||||
|
||||
cat > "${PKG_DIR}/etc/default/rustfs" << 'ENVEOF'
|
||||
# RustFS Environment Configuration
|
||||
# See https://docs.rustfs.com for more information
|
||||
# RUSTFS_ACCESS_KEY=<changeme>
|
||||
# RUSTFS_SECRET_KEY=<changeme>
|
||||
# RUSTFS_VOLUMES=""
|
||||
# RUSTFS_ADDRESS=":9000"
|
||||
# RUSTFS_CONSOLE_ADDRESS=":9001"
|
||||
# RUSTFS_CONSOLE_ENABLE=true
|
||||
# RUSTFS_OBS_LOGGER_LEVEL=error
|
||||
# RUSTFS_OBS_LOG_DIRECTORY="/var/log/rustfs/"
|
||||
ENVEOF
|
||||
|
||||
cat > "${PKG_DIR}/DEBIAN/control" << EOF
|
||||
Package: rustfs
|
||||
Version: ${DEB_VERSION}
|
||||
Section: utils
|
||||
Priority: optional
|
||||
Architecture: ${DEB_ARCH}
|
||||
EOF
|
||||
|
||||
# gnu binaries dynamically link glibc and must declare it; musl
|
||||
# binaries are statically linked and need no libc dependency.
|
||||
if [[ -n "$DEB_DEPENDS" ]]; then
|
||||
echo "Depends: ${DEB_DEPENDS}" >> "${PKG_DIR}/DEBIAN/control"
|
||||
fi
|
||||
|
||||
cat >> "${PKG_DIR}/DEBIAN/control" << EOF
|
||||
Maintainer: RustFS Team <support@rustfs.com>
|
||||
Description: High-performance distributed object storage (${LIBC} build)
|
||||
RustFS is a high-performance distributed object storage software
|
||||
built using Rust. It is compatible with MinIO and S3 API.
|
||||
Homepage: https://rustfs.com
|
||||
EOF
|
||||
|
||||
# Declare /etc/default/rustfs as a conffile so dpkg preserves user
|
||||
# modifications on upgrade instead of silently overwriting them.
|
||||
cat > "${PKG_DIR}/DEBIAN/conffiles" << 'CONFFILES'
|
||||
/etc/default/rustfs
|
||||
CONFFILES
|
||||
|
||||
./scripts/release/package_service_scripts.sh after-install > "${PKG_DIR}/DEBIAN/postinst"
|
||||
chmod 755 "${PKG_DIR}/DEBIAN/postinst"
|
||||
|
||||
./scripts/release/package_service_scripts.sh before-remove > "${PKG_DIR}/DEBIAN/prerm"
|
||||
chmod 755 "${PKG_DIR}/DEBIAN/prerm"
|
||||
|
||||
./scripts/release/package_service_scripts.sh after-remove > "${PKG_DIR}/DEBIAN/postrm"
|
||||
chmod 755 "${PKG_DIR}/DEBIAN/postrm"
|
||||
|
||||
cp LICENSE "${PKG_DIR}/usr/share/doc/rustfs/"
|
||||
cp README.md "${PKG_DIR}/usr/share/doc/rustfs/"
|
||||
|
||||
fakeroot dpkg-deb --build "${PKG_DIR}" "$DEB_FILE"
|
||||
|
||||
[[ $(dpkg-deb -f "$DEB_FILE" Package) == rustfs ]]
|
||||
[[ $(dpkg-deb -f "$DEB_FILE" Version) == "$DEB_VERSION" ]]
|
||||
[[ $(dpkg-deb -f "$DEB_FILE" Architecture) == "$DEB_ARCH" ]]
|
||||
dpkg-deb --fsys-tarfile "$DEB_FILE" | tar -tf - | grep -Fx './usr/bin/rustfs' >/dev/null
|
||||
stat --printf='%n %s bytes\n' "$DEB_FILE"
|
||||
echo "deb_file=$DEB_FILE" >> "$GITHUB_OUTPUT"
|
||||
echo "✅ DEB built: $DEB_FILE"
|
||||
|
||||
- name: Build RPM package
|
||||
id: rpm
|
||||
shell: bash
|
||||
env:
|
||||
RPM_VERSION: ${{ steps.versions.outputs.rpm_version }}
|
||||
RPM_RELEASE: ${{ steps.versions.outputs.rpm_release }}
|
||||
RPM_ARCH: ${{ matrix.rpm_arch }}
|
||||
RPM_FILE: ${{ steps.versions.outputs.rpm_file }}
|
||||
RPM_DEPENDS: ${{ matrix.rpm_depends }}
|
||||
LIBC: ${{ matrix.libc }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
|
||||
echo "Building RPM for ${RPM_ARCH} (${LIBC})"
|
||||
|
||||
sudo apt-get update && sudo apt-get install -y ruby ruby-dev build-essential rpm
|
||||
sudo gem install fpm
|
||||
./scripts/test_package_versions.sh --require-package-managers
|
||||
|
||||
# gnu binaries dynamically link glibc and must declare it; musl
|
||||
# binaries are statically linked and need no libc dependency.
|
||||
depends_args=()
|
||||
if [[ -n "$RPM_DEPENDS" ]]; then
|
||||
depends_args+=(--depends "$RPM_DEPENDS")
|
||||
fi
|
||||
|
||||
# Create config file for fpm (DEB build creates it in its package dir structure,
|
||||
# but fpm needs the file to exist before packaging)
|
||||
mkdir -p ./tmp-pkg/etc/default
|
||||
cat > ./tmp-pkg/etc/default/rustfs << 'ENVEOF'
|
||||
# RustFS Environment Configuration
|
||||
# See https://docs.rustfs.com for more information
|
||||
# RUSTFS_ACCESS_KEY=<changeme>
|
||||
# RUSTFS_SECRET_KEY=<changeme>
|
||||
# RUSTFS_VOLUMES=""
|
||||
# RUSTFS_ADDRESS=":9000"
|
||||
# RUSTFS_CONSOLE_ADDRESS=":9001"
|
||||
# RUSTFS_CONSOLE_ENABLE=true
|
||||
# RUSTFS_OBS_LOGGER_LEVEL=error
|
||||
# RUSTFS_OBS_LOG_DIRECTORY="/var/log/rustfs/"
|
||||
ENVEOF
|
||||
|
||||
fpm -s dir -t rpm \
|
||||
--name rustfs \
|
||||
--version "$RPM_VERSION" \
|
||||
--iteration "$RPM_RELEASE" \
|
||||
--architecture "$RPM_ARCH" \
|
||||
--package "$RPM_FILE" \
|
||||
"${depends_args[@]}" \
|
||||
--description "High-performance distributed object storage (${LIBC} build)" \
|
||||
--maintainer "RustFS Team <support@rustfs.com>" \
|
||||
--url "https://rustfs.com" \
|
||||
--license "Apache-2.0" \
|
||||
--before-install <(./scripts/release/package_service_scripts.sh rpm-before-install) \
|
||||
--before-upgrade <(./scripts/release/package_service_scripts.sh rpm-before-upgrade) \
|
||||
--after-install <(./scripts/release/package_service_scripts.sh after-install) \
|
||||
--after-upgrade <(./scripts/release/package_service_scripts.sh after-install) \
|
||||
--before-remove <(./scripts/release/package_service_scripts.sh before-remove) \
|
||||
--after-remove <(./scripts/release/package_service_scripts.sh after-remove) \
|
||||
--rpm-posttrans <(./scripts/release/package_service_scripts.sh rpm-posttrans) \
|
||||
--config-files /etc/default/rustfs \
|
||||
./bin/rustfs=/usr/bin/rustfs \
|
||||
./tmp-pkg/etc/default/rustfs=/etc/default/rustfs \
|
||||
deploy/build/rustfs.service=/lib/systemd/system/rustfs.service \
|
||||
LICENSE=/usr/share/doc/rustfs/LICENSE \
|
||||
README.md=/usr/share/doc/rustfs/README.md
|
||||
|
||||
if [[ ! -f "$RPM_FILE" ]]; then
|
||||
echo "❌ RPM build failed"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
RPM_METADATA=$(rpm -qp --qf '%{NAME}\n%{VERSION}\n%{RELEASE}\n%{ARCH}\n' "$RPM_FILE")
|
||||
EXPECTED_METADATA=$(printf 'rustfs\n%s\n%s\n%s' "$RPM_VERSION" "$RPM_RELEASE" "$RPM_ARCH")
|
||||
[[ "$RPM_METADATA" == "$EXPECTED_METADATA" ]]
|
||||
rpm -qpl "$RPM_FILE" | grep -Fx '/usr/bin/rustfs' >/dev/null
|
||||
stat --printf='%n %s bytes\n' "$RPM_FILE"
|
||||
echo "rpm_file=$RPM_FILE" >> "$GITHUB_OUTPUT"
|
||||
echo "✅ RPM built: $RPM_FILE"
|
||||
|
||||
- name: Download certification evidence
|
||||
if: needs.resolve.outputs.build_type != 'development'
|
||||
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
|
||||
with:
|
||||
name: release-certification
|
||||
path: ./certification
|
||||
|
||||
# Later-built wrapper verification: the binary inside each package must
|
||||
# match the certified binary digest for this platform before anything is
|
||||
# uploaded, and the package itself stays on staging until promotion.
|
||||
- name: Verify wrapped binary against certification
|
||||
if: needs.resolve.outputs.build_type != 'development'
|
||||
shell: bash
|
||||
env:
|
||||
DEB_FILE: ${{ steps.deb.outputs.deb_file }}
|
||||
RPM_FILE: ${{ steps.rpm.outputs.rpm_file }}
|
||||
ARTIFACT_NAME: ${{ matrix.artifact_name }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
expected="$(jq -r --arg needle "$ARTIFACT_NAME" \
|
||||
'[.deliverables[] | select(.name | contains($needle)) | .binary_sha256][0] // empty' \
|
||||
certification/candidate.json)"
|
||||
if [[ -z "$expected" ]]; then
|
||||
echo "No certified binary digest recorded for ${ARTIFACT_NAME}" >&2
|
||||
exit 1
|
||||
fi
|
||||
if ! command -v rpm2cpio >/dev/null 2>&1; then
|
||||
sudo apt-get update && sudo apt-get install -y rpm2cpio
|
||||
fi
|
||||
if ! command -v cpio >/dev/null 2>&1; then
|
||||
sudo apt-get update && sudo apt-get install -y cpio
|
||||
fi
|
||||
for package_file in "$DEB_FILE" "$RPM_FILE"; do
|
||||
case "$package_file" in
|
||||
*.deb)
|
||||
dpkg-deb --fsys-tarfile "$package_file" | tar -xOf - ./usr/bin/rustfs > "$RUNNER_TEMP/wrapped-rustfs"
|
||||
;;
|
||||
*.rpm)
|
||||
rpm2cpio "$package_file" | cpio -i --to-stdout ./usr/bin/rustfs > "$RUNNER_TEMP/wrapped-rustfs"
|
||||
;;
|
||||
esac
|
||||
python3 scripts/release/release_gate.py verify-promotion \
|
||||
--plan .config/release-validation-plan.json \
|
||||
--candidate certification/candidate.json \
|
||||
--summary certification/release-gate-output \
|
||||
--artifact "$RUNNER_TEMP/wrapped-rustfs" \
|
||||
--artifact-role binary
|
||||
done
|
||||
|
||||
- name: Upload packages to artifacts
|
||||
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
|
||||
with:
|
||||
name: packages-${{ matrix.arch }}-${{ matrix.libc }}
|
||||
path: |
|
||||
*.deb
|
||||
*.rpm
|
||||
retention-days: 30
|
||||
|
||||
- name: Upload packages to Cloudflare R2
|
||||
if: needs.resolve.outputs.build_type != 'preview' && env.R2_ACCESS_KEY_ID != ''
|
||||
env:
|
||||
R2_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
|
||||
R2_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
|
||||
R2_ENDPOINT: ${{ secrets.R2_ENDPOINT }}
|
||||
R2_BUCKET: ${{ secrets.R2_BUCKET }}
|
||||
AWS_EC2_METADATA_DISABLED: true
|
||||
BUILD_TYPE: ${{ needs.resolve.outputs.build_type }}
|
||||
DEB_FILE: ${{ steps.deb.outputs.deb_file }}
|
||||
RPM_FILE: ${{ steps.rpm.outputs.rpm_file }}
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
|
||||
if [[ -z "$R2_ACCESS_KEY_ID" || -z "$R2_SECRET_ACCESS_KEY" || -z "$R2_ENDPOINT" || -z "$R2_BUCKET" ]]; then
|
||||
echo "⚠️ R2 credentials missing, skipping upload"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
if ! command -v aws >/dev/null 2>&1; then
|
||||
sudo apt-get update && sudo apt-get install -y awscli
|
||||
fi
|
||||
|
||||
export AWS_ACCESS_KEY_ID="$R2_ACCESS_KEY_ID"
|
||||
export AWS_SECRET_ACCESS_KEY="$R2_SECRET_ACCESS_KEY"
|
||||
export AWS_DEFAULT_REGION="auto"
|
||||
|
||||
# Build-phase writes stay on staging for release types: the official
|
||||
# packages/release and packages/latest paths are written only by the
|
||||
# certification-gated promote-packages job, from these same staged
|
||||
# objects. Development packages keep the uncertified dev channel.
|
||||
if [[ "$BUILD_TYPE" == "development" ]]; then
|
||||
R2_PREFIX="artifacts/rustfs/packages/dev"
|
||||
else
|
||||
R2_PREFIX="artifacts/rustfs/packages/staging/${GITHUB_RUN_ID}/${GITHUB_RUN_ATTEMPT}"
|
||||
fi
|
||||
R2_PATH="s3://${R2_BUCKET}/${R2_PREFIX}/"
|
||||
|
||||
echo "📤 Uploading to $R2_PATH"
|
||||
|
||||
for f in "$DEB_FILE" "$RPM_FILE"; do
|
||||
if [[ -n "$f" && -f "$f" ]]; then
|
||||
echo "Uploading: $f"
|
||||
aws s3 cp "$f" "$R2_PATH" --endpoint-url "$R2_ENDPOINT" --only-show-errors
|
||||
fi
|
||||
done
|
||||
|
||||
echo "✅ Upload complete"
|
||||
|
||||
- name: Upload packages to GitHub Release
|
||||
if: needs.resolve.outputs.tag != ''
|
||||
- name: Reuse completed package set
|
||||
id: reuse
|
||||
env:
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
TAG: ${{ needs.resolve.outputs.tag }}
|
||||
DEB_FILE: ${{ steps.deb.outputs.deb_file }}
|
||||
RPM_FILE: ${{ steps.rpm.outputs.rpm_file }}
|
||||
shell: bash
|
||||
CANDIDATE_MANIFEST: ${{ inputs.candidate_manifest }}
|
||||
TARGET: ${{ matrix.target }}
|
||||
run: python3 scripts/release/deliver.py reuse --kind packages --target "$TARGET"
|
||||
- name: Install packaging tools
|
||||
if: steps.reuse.outputs.reused != 'true'
|
||||
run: |
|
||||
set -euo pipefail
|
||||
|
||||
# Upload the packages, then refresh the release checksums so the new
|
||||
# assets are covered, matching the binary release flow.
|
||||
for f in "$DEB_FILE" "$RPM_FILE"; do
|
||||
if [[ -n "$f" && -f "$f" ]]; then
|
||||
echo "📤 Uploading $(basename "$f") to GitHub release ${TAG}..."
|
||||
gh release upload "$TAG" "$f" --clobber
|
||||
fi
|
||||
done
|
||||
|
||||
CHECKSUM_DIR="$(mktemp -d)"
|
||||
gh release download "$TAG" -p 'SHA256SUMS' -p 'SHA512SUMS' \
|
||||
-D "$CHECKSUM_DIR" --clobber
|
||||
|
||||
for spec in "SHA256SUMS:sha256sum" "SHA512SUMS:sha512sum"; do
|
||||
asset="${spec%%:*}"
|
||||
checksum_cmd="${spec##*:}"
|
||||
checksum_file="${CHECKSUM_DIR}/${asset}"
|
||||
|
||||
for f in "$DEB_FILE" "$RPM_FILE"; do
|
||||
if [[ -n "$f" && -f "$f" ]]; then
|
||||
base="$(basename "$f")"
|
||||
# GitHub stores release asset names with '~' normalized to '.'.
|
||||
# Current package file names embed the raw tag and contain no
|
||||
# '~', but older releases may still ship such assets, so the
|
||||
# checksum entries must reference the name as stored on the
|
||||
# release.
|
||||
github_base="${base//\~/.}"
|
||||
# Remove any stale entry (both naming variants), then append
|
||||
grep -Fv -- "$base" "$checksum_file" > "${checksum_file}.tmp" || true
|
||||
grep -Fv -- "$github_base" "${checksum_file}.tmp" > "${checksum_file}.tmp2" || true
|
||||
mv "${checksum_file}.tmp2" "$checksum_file"
|
||||
digest=$("$checksum_cmd" -- "$f" | awk '{print $1}')
|
||||
printf '%s %s\n' "$digest" "$github_base" >> "$checksum_file"
|
||||
fi
|
||||
done
|
||||
|
||||
echo "📤 Updating ${asset} for release ${TAG}..."
|
||||
gh release upload "$TAG" "$checksum_file" --clobber
|
||||
done
|
||||
|
||||
echo "✅ GitHub release assets updated"
|
||||
|
||||
# The only writer of the official packages/release and packages/latest R2
|
||||
# paths. Promotion re-checks each package's wrapped binary against the
|
||||
# passed certification list and copies the staged objects byte-for-byte:
|
||||
# no rebuild-after-verify, no mutable latest resolution downstream.
|
||||
promote-packages:
|
||||
name: Promote Packages
|
||||
needs: [ resolve, package, release-certification ]
|
||||
# Target-channel lock for the official package paths.
|
||||
concurrency:
|
||||
group: release-channel-r2-packages
|
||||
cancel-in-progress: false
|
||||
if: needs.resolve.outputs.tag != '' && (needs.resolve.outputs.build_type == 'release' || needs.resolve.outputs.build_type == 'prerelease')
|
||||
runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
|
||||
timeout-minutes: 30
|
||||
permissions:
|
||||
contents: read
|
||||
actions: read
|
||||
env:
|
||||
R2_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
|
||||
R2_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
|
||||
R2_ENDPOINT: ${{ secrets.R2_ENDPOINT }}
|
||||
R2_BUCKET: ${{ secrets.R2_BUCKET }}
|
||||
AWS_EC2_METADATA_DISABLED: true
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
|
||||
with:
|
||||
persist-credentials: false
|
||||
|
||||
- name: Download packages and certification
|
||||
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
|
||||
with:
|
||||
path: ./packages
|
||||
pattern: packages-*
|
||||
merge-multiple: true
|
||||
|
||||
- name: Download certification evidence
|
||||
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
|
||||
with:
|
||||
name: release-certification
|
||||
path: ./certification
|
||||
|
||||
- name: Verify and promote staged packages
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
|
||||
if [[ -z "$R2_ACCESS_KEY_ID" || -z "$R2_SECRET_ACCESS_KEY" || -z "$R2_ENDPOINT" || -z "$R2_BUCKET" ]]; then
|
||||
echo "❌ R2 credentials missing; refusing to promote to the official channel" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
STAGING_PREFIX="s3://${R2_BUCKET}/artifacts/rustfs/packages/staging/${GITHUB_RUN_ID}/${GITHUB_RUN_ATTEMPT}/"
|
||||
RELEASE_PATH="s3://${R2_BUCKET}/artifacts/rustfs/packages/release/"
|
||||
LATEST_PATH="s3://${R2_BUCKET}/artifacts/rustfs/packages/latest/"
|
||||
|
||||
if ! command -v aws >/dev/null 2>&1; then
|
||||
sudo apt-get update && sudo apt-get install -y awscli
|
||||
fi
|
||||
export AWS_ACCESS_KEY_ID="$R2_ACCESS_KEY_ID"
|
||||
export AWS_SECRET_ACCESS_KEY="$R2_SECRET_ACCESS_KEY"
|
||||
export AWS_DEFAULT_REGION="auto"
|
||||
|
||||
if ! command -v rpm2cpio >/dev/null 2>&1; then
|
||||
sudo apt-get install -y rpm2cpio
|
||||
fi
|
||||
if ! command -v cpio >/dev/null 2>&1; then
|
||||
sudo apt-get install -y cpio
|
||||
fi
|
||||
|
||||
promoted=0
|
||||
for package_file in packages/*.deb packages/*.rpm; do
|
||||
[[ -f "$package_file" ]] || continue
|
||||
name="$(basename "$package_file")"
|
||||
case "$package_file" in
|
||||
*.deb)
|
||||
dpkg-deb --fsys-tarfile "$package_file" | tar -xOf - ./usr/bin/rustfs > "$RUNNER_TEMP/wrapped-rustfs"
|
||||
;;
|
||||
*.rpm)
|
||||
rpm2cpio "$package_file" | cpio -i --to-stdout ./usr/bin/rustfs > "$RUNNER_TEMP/wrapped-rustfs"
|
||||
;;
|
||||
esac
|
||||
python3 scripts/release/release_gate.py verify-promotion \
|
||||
--plan .config/release-validation-plan.json \
|
||||
--candidate certification/candidate.json \
|
||||
--summary certification/release-gate-output \
|
||||
--artifact "$RUNNER_TEMP/wrapped-rustfs" \
|
||||
--artifact-role binary
|
||||
|
||||
staged="$(mktemp)"
|
||||
aws s3 cp "${STAGING_PREFIX}${name}" "$staged" --endpoint-url "$R2_ENDPOINT" --only-show-errors
|
||||
staged_sha="$(sha256sum "$staged" | awk '{print $1}')"
|
||||
built_sha="$(sha256sum "$package_file" | awk '{print $1}')"
|
||||
if [[ "$staged_sha" != "$built_sha" ]]; then
|
||||
echo "❌ Staged package ${name} does not match the verified bytes" >&2
|
||||
exit 1
|
||||
fi
|
||||
aws s3 cp "$staged" "${RELEASE_PATH}${name}" --endpoint-url "$R2_ENDPOINT" --only-show-errors
|
||||
aws s3 cp "$staged" "${LATEST_PATH}${name}" --endpoint-url "$R2_ENDPOINT" --only-show-errors
|
||||
rm -f "$staged"
|
||||
promoted=$((promoted + 1))
|
||||
echo "✅ Promoted ${name} (${built_sha})"
|
||||
done
|
||||
if [[ "$promoted" -eq 0 ]]; then
|
||||
echo "No staged packages found to promote" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Summary
|
||||
summary:
|
||||
name: Summary
|
||||
needs: [ resolve, package ]
|
||||
# Every completed Build and Release run triggers this workflow; when
|
||||
# resolve declined it (not a release tag), there is nothing to summarise.
|
||||
if: always() && needs.resolve.result != 'skipped'
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 5
|
||||
steps:
|
||||
- name: Print summary
|
||||
shell: bash
|
||||
sudo apt-get update
|
||||
sudo apt-get install -y ruby ruby-dev build-essential rpm fakeroot cpio
|
||||
sudo gem install fpm --version 1.17.0 --no-document
|
||||
- name: Package accepted executable and verify contents
|
||||
if: steps.reuse.outputs.reused != 'true'
|
||||
env:
|
||||
SUMMARY_VERSION: ${{ needs.resolve.outputs.version }}
|
||||
SUMMARY_BUILD_TYPE: ${{ needs.resolve.outputs.build_type }}
|
||||
SUMMARY_BUILD_RUN_ID: ${{ needs.resolve.outputs.build_run_id }}
|
||||
SUMMARY_PACKAGE_STATUS: ${{ needs.package.result }}
|
||||
run: |
|
||||
{
|
||||
echo "## 📦 Package Summary"
|
||||
echo ""
|
||||
echo "| Item | Value |"
|
||||
echo "|------|-------|"
|
||||
echo "| Version | \`${SUMMARY_VERSION}\` |"
|
||||
echo "| Build Type | ${SUMMARY_BUILD_TYPE} |"
|
||||
echo "| Build Run | #${SUMMARY_BUILD_RUN_ID} |"
|
||||
echo "| Package Status | ${SUMMARY_PACKAGE_STATUS} |"
|
||||
} >> "$GITHUB_STEP_SUMMARY"
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
CANDIDATE_MANIFEST: ${{ inputs.candidate_manifest }}
|
||||
TARGET: ${{ matrix.target }}
|
||||
run: python3 scripts/release/deliver.py package --target "$TARGET" --output delivery
|
||||
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
|
||||
if: steps.reuse.outputs.reused != 'true'
|
||||
with:
|
||||
name: packages-${{ matrix.target }}
|
||||
path: delivery/*
|
||||
if-no-files-found: error
|
||||
compression-level: 0
|
||||
retention-days: 30
|
||||
|
||||
@@ -0,0 +1,133 @@
|
||||
# Copyright 2026 RustFS Team
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
name: Release candidate (Linux)
|
||||
on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
version:
|
||||
description: Final product version (no preview suffix)
|
||||
type: string
|
||||
required: true
|
||||
source_sha:
|
||||
description: Full reviewed source commit
|
||||
type: string
|
||||
required: true
|
||||
permissions:
|
||||
contents: read
|
||||
actions: read
|
||||
concurrency:
|
||||
group: release-candidate-${{ inputs.version }}
|
||||
cancel-in-progress: false
|
||||
jobs:
|
||||
freeze:
|
||||
if: github.repository == 'rustfs/rustfs' && github.ref == 'refs/heads/main'
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 15
|
||||
outputs:
|
||||
manifest: ${{ steps.freeze.outputs.manifest }}
|
||||
steps:
|
||||
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
|
||||
with:
|
||||
fetch-depth: 0
|
||||
persist-credentials: false
|
||||
- name: Freeze or recover original inputs
|
||||
id: freeze
|
||||
env:
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
VERSION: ${{ inputs.version }}
|
||||
SOURCE_SHA: ${{ inputs.source_sha }}
|
||||
run: python3 scripts/release/pipeline.py prepare --version "$VERSION" --source "$SOURCE_SHA" --directory frozen
|
||||
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
|
||||
if: steps.freeze.outputs.reused != 'true'
|
||||
with:
|
||||
name: release-inputs
|
||||
path: frozen/inputs.json
|
||||
if-no-files-found: error
|
||||
retention-days: 30
|
||||
linux:
|
||||
needs: freeze
|
||||
uses: ./.github/workflows/build.yml
|
||||
with:
|
||||
candidate_manifest: ${{ needs.freeze.outputs.manifest }}
|
||||
platforms: linux
|
||||
upgrade:
|
||||
needs: [freeze, linux]
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 150
|
||||
env:
|
||||
RUSTUP_TOOLCHAIN: ${{ fromJson(needs.freeze.outputs.manifest).toolchain }}
|
||||
steps:
|
||||
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
|
||||
with:
|
||||
ref: ${{ fromJson(needs.freeze.outputs.manifest).workflow_sha }}
|
||||
persist-credentials: false
|
||||
- uses: ./.github/actions/setup
|
||||
with:
|
||||
rust-version: ${{ fromJson(needs.freeze.outputs.manifest).toolchain }}
|
||||
cache-shared-key: release-upgrade-harness-hosted
|
||||
cache-save-if: ${{ github.ref == 'refs/heads/main' }}
|
||||
install-build-packaging-tools: "false"
|
||||
install-test-tools: "false"
|
||||
- name: Verify archived server against previous releases
|
||||
env:
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
CANDIDATE_MANIFEST: ${{ needs.freeze.outputs.manifest }}
|
||||
run: |
|
||||
python3 - <<'PYVERIFY'
|
||||
import sys
|
||||
from pathlib import Path
|
||||
sys.path.insert(0, 'scripts/release')
|
||||
import candidate as c, pipeline, verify
|
||||
inputs = pipeline.manifest_env()
|
||||
run = pipeline.current_run('release-candidate.yml')
|
||||
if any(a['name'] == 'release-upgrade' for a in c.artifact_list(run['id'])):
|
||||
c.download_artifact(run, 'release-upgrade', 'receipts')
|
||||
pipeline.verify_evidence(inputs, Path('receipts'), {run['id']: run})
|
||||
c.emit_output('reused', 'true')
|
||||
else:
|
||||
target = 'linux-x86_64-gnu'
|
||||
c.download_artifact(run, f"candidate-{inputs['candidate_id']}-{target}", 'binary')
|
||||
verify.upgrade(inputs, Path('binary') / f"rustfs-{target}-v{inputs['version']}.zip", Path('receipts'))
|
||||
PYVERIFY
|
||||
id: verify
|
||||
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
|
||||
if: steps.verify.outputs.reused != 'true'
|
||||
with:
|
||||
name: release-upgrade
|
||||
path: receipts/*
|
||||
if-no-files-found: error
|
||||
retention-days: 30
|
||||
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
|
||||
if: failure()
|
||||
with:
|
||||
name: release-upgrade-failure-${{ github.run_attempt }}
|
||||
path: receipts/*
|
||||
retention-days: 7
|
||||
seal-linux:
|
||||
needs: [freeze, linux, upgrade]
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 30
|
||||
steps:
|
||||
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
|
||||
with:
|
||||
persist-credentials: false
|
||||
- name: Collect trusted receipts and seal Linux bytes
|
||||
id: seal
|
||||
env:
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
run: |
|
||||
python3 scripts/release/pipeline.py linux-seal --directory sealed
|
||||
{
|
||||
echo '## Linux acceptance'
|
||||
echo 'Use this run ID and manifest SHA-256 when starting Release candidate (complete).'
|
||||
sha256sum sealed/linux-manifest.json
|
||||
echo 'Record cluster, performance, integrity and upgrade acceptance in backlog before continuing.'
|
||||
} >> "$GITHUB_STEP_SUMMARY"
|
||||
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
|
||||
if: steps.seal.outputs.reused != 'true'
|
||||
with:
|
||||
name: release-linux
|
||||
path: sealed/*
|
||||
if-no-files-found: error
|
||||
compression-level: 0
|
||||
retention-days: 30
|
||||
@@ -0,0 +1,97 @@
|
||||
# Copyright 2026 RustFS Team
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
name: Release candidate (complete)
|
||||
on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
candidate_run_id:
|
||||
description: Successful Linux candidate run ID
|
||||
type: string
|
||||
required: true
|
||||
linux_manifest_sha256:
|
||||
description: Exact accepted Linux manifest SHA-256
|
||||
type: string
|
||||
required: true
|
||||
acceptance_report:
|
||||
description: Backlog report accepting cluster, performance, integrity and upgrade results
|
||||
type: string
|
||||
required: true
|
||||
permissions:
|
||||
contents: read
|
||||
actions: read
|
||||
concurrency:
|
||||
group: release-complete-${{ inputs.candidate_run_id }}
|
||||
cancel-in-progress: false
|
||||
jobs:
|
||||
acceptance:
|
||||
if: github.repository == 'rustfs/rustfs' && github.ref == 'refs/heads/main'
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 30
|
||||
environment: release-linux-acceptance
|
||||
outputs:
|
||||
manifest: ${{ steps.accept.outputs.manifest }}
|
||||
steps:
|
||||
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
|
||||
with:
|
||||
fetch-depth: 0
|
||||
persist-credentials: false
|
||||
- name: Bind acceptance to the exact Linux manifest
|
||||
id: accept
|
||||
env:
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
CANDIDATE_RUN: ${{ inputs.candidate_run_id }}
|
||||
MANIFEST_SHA: ${{ inputs.linux_manifest_sha256 }}
|
||||
REPORT: ${{ inputs.acceptance_report }}
|
||||
run: python3 scripts/release/pipeline.py complete-prepare --run-id "$CANDIDATE_RUN" --digest "$MANIFEST_SHA" --report "$REPORT" --directory linux
|
||||
desktop:
|
||||
needs: acceptance
|
||||
uses: ./.github/workflows/build.yml
|
||||
with:
|
||||
candidate_manifest: ${{ needs.acceptance.outputs.manifest }}
|
||||
platforms: desktop
|
||||
packages:
|
||||
needs: acceptance
|
||||
uses: ./.github/workflows/package.yml
|
||||
with:
|
||||
candidate_manifest: ${{ needs.acceptance.outputs.manifest }}
|
||||
images:
|
||||
needs: acceptance
|
||||
uses: ./.github/workflows/docker.yml
|
||||
with:
|
||||
candidate_manifest: ${{ needs.acceptance.outputs.manifest }}
|
||||
chart:
|
||||
needs: acceptance
|
||||
uses: ./.github/workflows/helm-package.yml
|
||||
with:
|
||||
candidate_manifest: ${{ needs.acceptance.outputs.manifest }}
|
||||
seal:
|
||||
needs: [acceptance, desktop, packages, images, chart]
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 45
|
||||
steps:
|
||||
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
|
||||
with:
|
||||
fetch-depth: 0
|
||||
persist-credentials: false
|
||||
- name: Verify every deliverable and seal the candidate
|
||||
id: seal
|
||||
env:
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
CANDIDATE_RUN: ${{ inputs.candidate_run_id }}
|
||||
MANIFEST_SHA: ${{ inputs.linux_manifest_sha256 }}
|
||||
REPORT: ${{ inputs.acceptance_report }}
|
||||
run: |
|
||||
python3 scripts/release/pipeline.py complete-seal --run-id "$CANDIDATE_RUN" --digest "$MANIFEST_SHA" --report "$REPORT" --directory sealed
|
||||
{
|
||||
echo '## Final publication approval'
|
||||
echo 'Review this manifest, then explicitly dispatch Publish accepted release with its SHA-256.'
|
||||
sha256sum sealed/release-manifest.json
|
||||
} >> "$GITHUB_STEP_SUMMARY"
|
||||
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
|
||||
if: steps.seal.outputs.reused != 'true'
|
||||
with:
|
||||
name: release-seal
|
||||
path: sealed/*
|
||||
if-no-files-found: error
|
||||
compression-level: 0
|
||||
retention-days: 30
|
||||
@@ -0,0 +1,193 @@
|
||||
# Copyright 2026 RustFS Team
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
name: Publish accepted release
|
||||
on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
complete_run_id:
|
||||
description: Successful completed candidate run ID
|
||||
type: string
|
||||
required: true
|
||||
manifest_sha256:
|
||||
description: Exact approved final manifest SHA-256 (authorizes publication)
|
||||
type: string
|
||||
required: true
|
||||
permissions:
|
||||
contents: read
|
||||
actions: read
|
||||
concurrency:
|
||||
group: release-publish-${{ inputs.complete_run_id }}
|
||||
cancel-in-progress: false
|
||||
jobs:
|
||||
verify:
|
||||
if: github.repository == 'rustfs/rustfs' && github.ref == 'refs/heads/main'
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 45
|
||||
outputs:
|
||||
version: ${{ steps.verify.outputs.version }}
|
||||
steps:
|
||||
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
|
||||
with:
|
||||
fetch-depth: 0
|
||||
persist-credentials: false
|
||||
- name: Recheck approved bytes and trusted certification
|
||||
id: verify
|
||||
env:
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
COMPLETE_RUN: ${{ inputs.complete_run_id }}
|
||||
MANIFEST_SHA: ${{ inputs.manifest_sha256 }}
|
||||
run: python3 scripts/release/publish.py verify --run-id "$COMPLETE_RUN" --digest "$MANIFEST_SHA" --directory sealed
|
||||
version:
|
||||
needs: verify
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 90
|
||||
concurrency:
|
||||
group: release-version-${{ needs.verify.outputs.version }}
|
||||
cancel-in-progress: false
|
||||
environment: release-publication
|
||||
permissions:
|
||||
contents: write
|
||||
actions: read
|
||||
packages: write
|
||||
security-events: write
|
||||
env:
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
COMPLETE_RUN: ${{ inputs.complete_run_id }}
|
||||
MANIFEST_SHA: ${{ inputs.manifest_sha256 }}
|
||||
AWS_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
|
||||
AWS_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
|
||||
AWS_DEFAULT_REGION: auto
|
||||
AWS_REQUEST_CHECKSUM_CALCULATION: when_required
|
||||
AWS_RESPONSE_CHECKSUM_VALIDATION: when_required
|
||||
AWS_EC2_METADATA_DISABLED: "true"
|
||||
R2_ENDPOINT: ${{ secrets.R2_ENDPOINT }}
|
||||
R2_BUCKET: ${{ secrets.R2_BUCKET }}
|
||||
steps:
|
||||
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
|
||||
with:
|
||||
fetch-depth: 0
|
||||
persist-credentials: false
|
||||
- name: Install promotion tools
|
||||
run: |
|
||||
sudo apt-get update
|
||||
sudo apt-get install -y skopeo
|
||||
aws --version
|
||||
- uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
|
||||
with:
|
||||
username: rustfs
|
||||
password: ${{ secrets.DOCKERHUB_TOKEN }}
|
||||
- uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
|
||||
with:
|
||||
registry: ghcr.io
|
||||
username: ${{ secrets.GHCR_USERNAME }}
|
||||
password: ${{ secrets.GHCR_PASSWORD }}
|
||||
- uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
|
||||
with:
|
||||
registry: quay.io
|
||||
username: ${{ secrets.QUAY_USERNAME }}
|
||||
password: ${{ secrets.QUAY_PASSWORD }}
|
||||
- name: Share registry credentials with the OCI copier
|
||||
run: |
|
||||
install -m 600 "$HOME/.docker/config.json" "$RUNNER_TEMP/registry-auth.json"
|
||||
echo "REGISTRY_AUTH_FILE=$RUNNER_TEMP/registry-auth.json" >> "$GITHUB_ENV"
|
||||
- name: Promote verified version
|
||||
run: python3 scripts/release/publish.py version --run-id "$COMPLETE_RUN" --digest "$MANIFEST_SHA" --directory sealed
|
||||
- name: Publish accepted musl amd64 scan
|
||||
uses: github/codeql-action/upload-sarif@99df26d4f13ea111d4ec1a7dddef6063f76b97e9 # v4
|
||||
with:
|
||||
sarif_file: sealed/scan-musl-amd64.sarif
|
||||
category: container-image-musl-amd64
|
||||
- name: Publish accepted musl arm64 scan
|
||||
uses: github/codeql-action/upload-sarif@99df26d4f13ea111d4ec1a7dddef6063f76b97e9 # v4
|
||||
with:
|
||||
sarif_file: sealed/scan-musl-arm64.sarif
|
||||
category: container-image-musl-arm64
|
||||
- name: Publish accepted GNU amd64 scan
|
||||
uses: github/codeql-action/upload-sarif@99df26d4f13ea111d4ec1a7dddef6063f76b97e9 # v4
|
||||
with:
|
||||
sarif_file: sealed/scan-gnu-amd64.sarif
|
||||
category: container-image-gnu-amd64
|
||||
- name: Publish accepted GNU arm64 scan
|
||||
uses: github/codeql-action/upload-sarif@99df26d4f13ea111d4ec1a7dddef6063f76b97e9 # v4
|
||||
with:
|
||||
sarif_file: sealed/scan-gnu-arm64.sarif
|
||||
category: container-image-gnu-arm64
|
||||
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
|
||||
with:
|
||||
name: release-version-receipt-${{ github.run_attempt }}
|
||||
path: publication.json
|
||||
if-no-files-found: error
|
||||
retention-days: 90
|
||||
channels:
|
||||
needs: [verify, version]
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 90
|
||||
concurrency:
|
||||
group: rustfs-release-channels
|
||||
cancel-in-progress: false
|
||||
permissions:
|
||||
contents: write
|
||||
actions: read
|
||||
packages: write
|
||||
env:
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
COMPLETE_RUN: ${{ inputs.complete_run_id }}
|
||||
MANIFEST_SHA: ${{ inputs.manifest_sha256 }}
|
||||
AWS_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
|
||||
AWS_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
|
||||
AWS_DEFAULT_REGION: auto
|
||||
AWS_REQUEST_CHECKSUM_CALCULATION: when_required
|
||||
AWS_RESPONSE_CHECKSUM_VALIDATION: when_required
|
||||
AWS_EC2_METADATA_DISABLED: "true"
|
||||
R2_ENDPOINT: ${{ secrets.R2_ENDPOINT }}
|
||||
R2_BUCKET: ${{ secrets.R2_BUCKET }}
|
||||
steps:
|
||||
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
|
||||
with:
|
||||
fetch-depth: 0
|
||||
persist-credentials: false
|
||||
- name: Install promotion tools
|
||||
run: |
|
||||
sudo apt-get update
|
||||
sudo apt-get install -y skopeo
|
||||
aws --version
|
||||
- uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
|
||||
with:
|
||||
username: rustfs
|
||||
password: ${{ secrets.DOCKERHUB_TOKEN }}
|
||||
- uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
|
||||
with:
|
||||
registry: ghcr.io
|
||||
username: ${{ secrets.GHCR_USERNAME }}
|
||||
password: ${{ secrets.GHCR_PASSWORD }}
|
||||
- uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
|
||||
with:
|
||||
registry: quay.io
|
||||
username: ${{ secrets.QUAY_USERNAME }}
|
||||
password: ${{ secrets.QUAY_PASSWORD }}
|
||||
- name: Share registry credentials with the OCI copier
|
||||
run: |
|
||||
install -m 600 "$HOME/.docker/config.json" "$RUNNER_TEMP/registry-auth.json"
|
||||
echo "REGISTRY_AUTH_FILE=$RUNNER_TEMP/registry-auth.json" >> "$GITHUB_ENV"
|
||||
- uses: azure/setup-helm@b9e51907a09c216f16ebe8536097933489208112 # v4.3.0
|
||||
- name: Install pinned OSS client
|
||||
run: |
|
||||
curl --fail --location --retry 3 --output "$RUNNER_TEMP/ossutil.zip" https://gosspublic.alicdn.com/ossutil/v2/2.1.1/ossutil-2.1.1-linux-amd64.zip
|
||||
printf '%s %s\n' 60f3a808cbbdfd4b5269b8f967c6a66f564c9dacff52d93a5d21a588976ab5a2 "$RUNNER_TEMP/ossutil.zip" | sha256sum --check -
|
||||
unzip -q "$RUNNER_TEMP/ossutil.zip" -d "$RUNNER_TEMP/ossutil-extract"
|
||||
install -m 755 "$RUNNER_TEMP/ossutil-extract/ossutil-2.1.1-linux-amd64/ossutil" "$RUNNER_TEMP/ossutil"
|
||||
echo "OSSUTIL_BIN=$RUNNER_TEMP/ossutil" >> "$GITHUB_ENV"
|
||||
- name: Promote verified channels
|
||||
env:
|
||||
HELM_TOKEN: ${{ secrets.RUSTFS_HELM_PACKAGE }}
|
||||
OSS_ACCESS_KEY_ID: ${{ secrets.ALICLOUDOSS_KEY_ID }}
|
||||
OSS_ACCESS_KEY_SECRET: ${{ secrets.ALICLOUDOSS_KEY_SECRET }}
|
||||
OSS_REGION: cn-beijing
|
||||
OSS_ENDPOINT: https://oss-cn-beijing.aliyuncs.com
|
||||
run: python3 scripts/release/publish.py channels --run-id "$COMPLETE_RUN" --digest "$MANIFEST_SHA" --directory sealed
|
||||
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
|
||||
with:
|
||||
name: release-channels-receipt-${{ github.run_attempt }}
|
||||
path: publication.json
|
||||
if-no-files-found: error
|
||||
retention-days: 90
|
||||
@@ -26,7 +26,13 @@ permissions:
|
||||
jobs:
|
||||
alert-on-incomplete-run:
|
||||
name: Alert on incomplete scheduled run
|
||||
if: github.repository == 'rustfs/rustfs'
|
||||
# The registry still owns responsibility and cadence selection. Successful
|
||||
# and PR runs can never produce an incomplete scheduled-run alert.
|
||||
if: >-
|
||||
github.repository == 'rustfs/rustfs' &&
|
||||
github.event.workflow_run.conclusion != 'success' &&
|
||||
(github.event.workflow_run.event == 'schedule' ||
|
||||
github.event.workflow_run.event == 'workflow_run')
|
||||
# Stay on GitHub-hosted runners: this monitor must keep running when the
|
||||
# self-hosted sm-standard-* pool is down, which is exactly when the
|
||||
# scheduled validations it watches stall.
|
||||
|
||||
Generated
+56
-52
@@ -4136,7 +4136,7 @@ checksum = "d0881ea181b1df73ff77ffaaf9c7544ecc11e82fba9b5f27b262a3c73a332555"
|
||||
|
||||
[[package]]
|
||||
name = "e2e_test"
|
||||
version = "1.0.2"
|
||||
version = "1.0.5"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"async-compression",
|
||||
@@ -9520,7 +9520,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "rustfs"
|
||||
version = "1.0.2"
|
||||
version = "1.0.5"
|
||||
dependencies = [
|
||||
"aes-gcm",
|
||||
"anyhow",
|
||||
@@ -9680,7 +9680,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "rustfs-audit"
|
||||
version = "1.0.2"
|
||||
version = "1.0.5"
|
||||
dependencies = [
|
||||
"const-str",
|
||||
"futures",
|
||||
@@ -9702,7 +9702,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "rustfs-checksums"
|
||||
version = "1.0.2"
|
||||
version = "1.0.5"
|
||||
dependencies = [
|
||||
"base64-simd",
|
||||
"bytes",
|
||||
@@ -9718,7 +9718,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "rustfs-common"
|
||||
version = "1.0.2"
|
||||
version = "1.0.5"
|
||||
dependencies = [
|
||||
"arc-swap",
|
||||
"hotpath",
|
||||
@@ -9733,7 +9733,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "rustfs-concurrency"
|
||||
version = "1.0.2"
|
||||
version = "1.0.5"
|
||||
dependencies = [
|
||||
"hotpath",
|
||||
"insta",
|
||||
@@ -9746,7 +9746,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "rustfs-config"
|
||||
version = "1.0.2"
|
||||
version = "1.0.5"
|
||||
dependencies = [
|
||||
"const-str",
|
||||
"hotpath",
|
||||
@@ -9756,7 +9756,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "rustfs-connect-symbol-catalog"
|
||||
version = "1.0.2"
|
||||
version = "1.0.5"
|
||||
dependencies = [
|
||||
"hex-simd",
|
||||
"object 0.40.0",
|
||||
@@ -9768,7 +9768,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "rustfs-credentials"
|
||||
version = "1.0.2"
|
||||
version = "1.0.5"
|
||||
dependencies = [
|
||||
"base64-simd",
|
||||
"hmac 0.13.0",
|
||||
@@ -9782,7 +9782,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "rustfs-crypto"
|
||||
version = "1.0.2"
|
||||
version = "1.0.5"
|
||||
dependencies = [
|
||||
"aes-gcm",
|
||||
"argon2",
|
||||
@@ -9803,7 +9803,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "rustfs-data-usage"
|
||||
version = "1.0.2"
|
||||
version = "1.0.5"
|
||||
dependencies = [
|
||||
"hotpath",
|
||||
"rmp-serde",
|
||||
@@ -9813,7 +9813,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "rustfs-ecstore"
|
||||
version = "1.0.2"
|
||||
version = "1.0.5"
|
||||
dependencies = [
|
||||
"arc-swap",
|
||||
"async-channel",
|
||||
@@ -9837,6 +9837,7 @@ dependencies = [
|
||||
"google-cloud-auth",
|
||||
"google-cloud-storage",
|
||||
"h2",
|
||||
"hashbrown 0.17.1",
|
||||
"hex-simd",
|
||||
"hmac 0.13.0",
|
||||
"hostname",
|
||||
@@ -9950,7 +9951,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "rustfs-extension-schema"
|
||||
version = "1.0.2"
|
||||
version = "1.0.5"
|
||||
dependencies = [
|
||||
"hotpath",
|
||||
"serde",
|
||||
@@ -9960,7 +9961,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "rustfs-filemeta"
|
||||
version = "1.0.2"
|
||||
version = "1.0.5"
|
||||
dependencies = [
|
||||
"arc-swap",
|
||||
"base64-simd",
|
||||
@@ -9989,7 +9990,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "rustfs-heal"
|
||||
version = "1.0.2"
|
||||
version = "1.0.5"
|
||||
dependencies = [
|
||||
"async-trait",
|
||||
"base64-simd",
|
||||
@@ -10030,7 +10031,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "rustfs-heal-contracts"
|
||||
version = "1.0.2"
|
||||
version = "1.0.5"
|
||||
dependencies = [
|
||||
"serde",
|
||||
"serde_json",
|
||||
@@ -10040,7 +10041,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "rustfs-iam"
|
||||
version = "1.0.2"
|
||||
version = "1.0.5"
|
||||
dependencies = [
|
||||
"arc-swap",
|
||||
"async-trait",
|
||||
@@ -10090,7 +10091,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "rustfs-io-core"
|
||||
version = "1.0.2"
|
||||
version = "1.0.5"
|
||||
dependencies = [
|
||||
"bytes",
|
||||
"hotpath",
|
||||
@@ -10103,7 +10104,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "rustfs-io-metrics"
|
||||
version = "1.0.2"
|
||||
version = "1.0.5"
|
||||
dependencies = [
|
||||
"criterion",
|
||||
"hotpath",
|
||||
@@ -10167,7 +10168,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "rustfs-keystone"
|
||||
version = "1.0.2"
|
||||
version = "1.0.5"
|
||||
dependencies = [
|
||||
"bytes",
|
||||
"futures",
|
||||
@@ -10193,7 +10194,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "rustfs-kms"
|
||||
version = "1.0.2"
|
||||
version = "1.0.5"
|
||||
dependencies = [
|
||||
"aes-gcm",
|
||||
"anyhow",
|
||||
@@ -10243,14 +10244,14 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "rustfs-license"
|
||||
version = "1.0.2"
|
||||
version = "1.0.5"
|
||||
dependencies = [
|
||||
"thiserror 2.0.21",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "rustfs-lifecycle"
|
||||
version = "1.0.2"
|
||||
version = "1.0.5"
|
||||
dependencies = [
|
||||
"async-trait",
|
||||
"hotpath",
|
||||
@@ -10272,7 +10273,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "rustfs-lock"
|
||||
version = "1.0.2"
|
||||
version = "1.0.5"
|
||||
dependencies = [
|
||||
"async-trait",
|
||||
"compact_str",
|
||||
@@ -10296,7 +10297,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "rustfs-log-analyzer"
|
||||
version = "1.0.2"
|
||||
version = "1.0.5"
|
||||
dependencies = [
|
||||
"chrono",
|
||||
"flate2",
|
||||
@@ -10315,7 +10316,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "rustfs-madmin"
|
||||
version = "1.0.2"
|
||||
version = "1.0.5"
|
||||
dependencies = [
|
||||
"hotpath",
|
||||
"http 1.5.0",
|
||||
@@ -10353,7 +10354,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "rustfs-notify"
|
||||
version = "1.0.2"
|
||||
version = "1.0.5"
|
||||
dependencies = [
|
||||
"arc-swap",
|
||||
"async-trait",
|
||||
@@ -10388,7 +10389,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "rustfs-object-capacity"
|
||||
version = "1.0.2"
|
||||
version = "1.0.5"
|
||||
dependencies = [
|
||||
"criterion",
|
||||
"futures",
|
||||
@@ -10407,7 +10408,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "rustfs-object-data-cache"
|
||||
version = "1.0.2"
|
||||
version = "1.0.5"
|
||||
dependencies = [
|
||||
"bytes",
|
||||
"criterion",
|
||||
@@ -10424,7 +10425,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "rustfs-obs"
|
||||
version = "1.0.2"
|
||||
version = "1.0.5"
|
||||
dependencies = [
|
||||
"chrono",
|
||||
"crossbeam-channel",
|
||||
@@ -10487,7 +10488,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "rustfs-policy"
|
||||
version = "1.0.2"
|
||||
version = "1.0.5"
|
||||
dependencies = [
|
||||
"async-trait",
|
||||
"base64-simd",
|
||||
@@ -10518,7 +10519,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "rustfs-protocols"
|
||||
version = "1.0.2"
|
||||
version = "1.0.5"
|
||||
dependencies = [
|
||||
"async-compression",
|
||||
"async-trait",
|
||||
@@ -10581,7 +10582,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "rustfs-protos"
|
||||
version = "1.0.2"
|
||||
version = "1.0.5"
|
||||
dependencies = [
|
||||
"flatbuffers",
|
||||
"hotpath",
|
||||
@@ -10606,7 +10607,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "rustfs-replication"
|
||||
version = "1.0.2"
|
||||
version = "1.0.5"
|
||||
dependencies = [
|
||||
"byteorder",
|
||||
"bytes",
|
||||
@@ -10624,7 +10625,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "rustfs-rio"
|
||||
version = "1.0.2"
|
||||
version = "1.0.5"
|
||||
dependencies = [
|
||||
"aes-gcm",
|
||||
"arc-swap",
|
||||
@@ -10670,7 +10671,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "rustfs-rio-v2"
|
||||
version = "1.0.2"
|
||||
version = "1.0.5"
|
||||
dependencies = [
|
||||
"aes-gcm",
|
||||
"bytes",
|
||||
@@ -10693,7 +10694,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "rustfs-s3-client"
|
||||
version = "1.0.2"
|
||||
version = "1.0.5"
|
||||
dependencies = [
|
||||
"base64-simd",
|
||||
"bytes",
|
||||
@@ -10737,7 +10738,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "rustfs-s3-ops"
|
||||
version = "1.0.2"
|
||||
version = "1.0.5"
|
||||
dependencies = [
|
||||
"hotpath",
|
||||
"rustfs-s3-types",
|
||||
@@ -10745,7 +10746,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "rustfs-s3-types"
|
||||
version = "1.0.2"
|
||||
version = "1.0.5"
|
||||
dependencies = [
|
||||
"hotpath",
|
||||
"serde",
|
||||
@@ -10754,22 +10755,25 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "rustfs-s3select-api"
|
||||
version = "1.0.2"
|
||||
version = "1.0.5"
|
||||
dependencies = [
|
||||
"arc-swap",
|
||||
"async-compression",
|
||||
"async-trait",
|
||||
"base64-simd",
|
||||
"brotli 9.1.1",
|
||||
"bytes",
|
||||
"chrono",
|
||||
"crc-fast",
|
||||
"datafusion",
|
||||
"flatbuffers",
|
||||
"flate2",
|
||||
"futures",
|
||||
"futures-core",
|
||||
"hotpath",
|
||||
"http 1.5.0",
|
||||
"lz4",
|
||||
"lz4_flex",
|
||||
"memchr",
|
||||
"metrics",
|
||||
"parking_lot 0.13.0",
|
||||
@@ -10792,7 +10796,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "rustfs-s3select-query"
|
||||
version = "1.0.2"
|
||||
version = "1.0.5"
|
||||
dependencies = [
|
||||
"async-trait",
|
||||
"bytes",
|
||||
@@ -10814,7 +10818,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "rustfs-scanner"
|
||||
version = "1.0.2"
|
||||
version = "1.0.5"
|
||||
dependencies = [
|
||||
"async-trait",
|
||||
"bytes",
|
||||
@@ -10865,7 +10869,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "rustfs-scanner-metrics"
|
||||
version = "1.0.2"
|
||||
version = "1.0.5"
|
||||
dependencies = [
|
||||
"chrono",
|
||||
"jiff",
|
||||
@@ -10880,7 +10884,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "rustfs-security-governance"
|
||||
version = "1.0.2"
|
||||
version = "1.0.5"
|
||||
dependencies = [
|
||||
"hotpath",
|
||||
"thiserror 2.0.21",
|
||||
@@ -10888,7 +10892,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "rustfs-signer"
|
||||
version = "1.0.2"
|
||||
version = "1.0.5"
|
||||
dependencies = [
|
||||
"base64-simd",
|
||||
"bytes",
|
||||
@@ -10907,7 +10911,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "rustfs-storage-api"
|
||||
version = "1.0.2"
|
||||
version = "1.0.5"
|
||||
dependencies = [
|
||||
"async-trait",
|
||||
"hotpath",
|
||||
@@ -10922,7 +10926,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "rustfs-targets"
|
||||
version = "1.0.2"
|
||||
version = "1.0.5"
|
||||
dependencies = [
|
||||
"arc-swap",
|
||||
"async-nats",
|
||||
@@ -10976,7 +10980,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "rustfs-test-utils"
|
||||
version = "1.0.2"
|
||||
version = "1.0.5"
|
||||
dependencies = [
|
||||
"hotpath",
|
||||
"rustfs-data-usage",
|
||||
@@ -10992,7 +10996,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "rustfs-tls-runtime"
|
||||
version = "1.0.2"
|
||||
version = "1.0.5"
|
||||
dependencies = [
|
||||
"arc-swap",
|
||||
"hotpath",
|
||||
@@ -11031,7 +11035,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "rustfs-trusted-proxies"
|
||||
version = "1.0.2"
|
||||
version = "1.0.5"
|
||||
dependencies = [
|
||||
"async-trait",
|
||||
"axum",
|
||||
@@ -11068,7 +11072,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "rustfs-utils"
|
||||
version = "1.0.2"
|
||||
version = "1.0.5"
|
||||
dependencies = [
|
||||
"aws-lc-rs",
|
||||
"base64-simd",
|
||||
@@ -11111,7 +11115,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "rustfs-zip"
|
||||
version = "1.0.2"
|
||||
version = "1.0.5"
|
||||
dependencies = [
|
||||
"async-compression",
|
||||
"futures",
|
||||
|
||||
+52
-51
@@ -75,7 +75,7 @@ edition = "2024"
|
||||
license = "Apache-2.0"
|
||||
repository = "https://github.com/rustfs/rustfs"
|
||||
rust-version = "1.99.0"
|
||||
version = "1.0.2"
|
||||
version = "1.0.5"
|
||||
homepage = "https://rustfs.com"
|
||||
description = "RustFS is a high-performance distributed object storage software built using Rust, one of the most popular languages worldwide. "
|
||||
keywords = ["RustFS", "Minio", "object-storage", "filesystem", "s3"]
|
||||
@@ -92,56 +92,56 @@ redundant_clone = "warn"
|
||||
|
||||
[workspace.dependencies]
|
||||
# RustFS Internal Crates
|
||||
rustfs = { path = "./rustfs", version = "1.0.2" }
|
||||
rustfs-heal = { path = "crates/heal", version = "1.0.2" }
|
||||
rustfs-heal-contracts = { path = "crates/heal-contracts", version = "1.0.2" }
|
||||
rustfs-scanner-metrics = { path = "crates/scanner-metrics", version = "1.0.2" }
|
||||
rustfs-audit = { path = "crates/audit", version = "1.0.2" }
|
||||
rustfs-checksums = { path = "crates/checksums", version = "1.0.2" }
|
||||
rustfs-common = { path = "crates/common", version = "1.0.2" }
|
||||
rustfs-data-usage = { path = "crates/data-usage", version = "1.0.2" }
|
||||
rustfs-config = { path = "./crates/config", version = "1.0.2" }
|
||||
rustfs-concurrency = { path = "./crates/concurrency", version = "1.0.2" }
|
||||
rustfs-credentials = { path = "crates/credentials", version = "1.0.2" }
|
||||
rustfs-crypto = { path = "crates/crypto", version = "1.0.2" }
|
||||
rustfs-ecstore = { path = "crates/ecstore", version = "1.0.2" }
|
||||
rustfs-filemeta = { path = "crates/filemeta", version = "1.0.2" }
|
||||
rustfs-iam = { path = "crates/iam", version = "1.0.2" }
|
||||
rustfs-keystone = { path = "crates/keystone", version = "1.0.2" }
|
||||
rustfs-license = { path = "crates/license", version = "1.0.2" }
|
||||
rustfs-lifecycle = { path = "crates/lifecycle", version = "1.0.2" }
|
||||
rustfs-kms = { path = "crates/kms", version = "1.0.2" }
|
||||
rustfs-lock = { path = "crates/lock", version = "1.0.2" }
|
||||
rustfs-madmin = { path = "crates/madmin", version = "1.0.2" }
|
||||
rustfs-notify = { path = "crates/notify", version = "1.0.2" }
|
||||
rustfs-io-metrics = { path = "crates/io-metrics", version = "1.0.2" }
|
||||
rustfs-io-core = { path = "crates/io-core", version = "1.0.2" }
|
||||
rustfs-object-capacity = { path = "crates/object-capacity", version = "1.0.2" }
|
||||
rustfs-object-data-cache = { path = "crates/object-data-cache", version = "1.0.2", default-features = false }
|
||||
rustfs-log-analyzer = { path = "crates/log-analyzer", version = "1.0.2" }
|
||||
rustfs-obs = { path = "crates/obs", version = "1.0.2" }
|
||||
rustfs-policy = { path = "crates/policy", version = "1.0.2" }
|
||||
rustfs-protos = { path = "crates/protos", version = "1.0.2" }
|
||||
rustfs-protocols = { path = "crates/protocols", version = "1.0.2" }
|
||||
rustfs-replication = { path = "crates/replication", version = "1.0.2" }
|
||||
rustfs-rio = { path = "crates/rio", version = "1.0.2" }
|
||||
rustfs-rio-v2 = { path = "crates/rio-v2", version = "1.0.2" }
|
||||
rustfs-s3-client = { path = "crates/s3-client", version = "1.0.2" }
|
||||
rustfs-s3-types = { path = "crates/s3-types", version = "1.0.2" }
|
||||
rustfs-s3-ops = { path = "crates/s3-ops", version = "1.0.2" }
|
||||
rustfs-s3select-api = { path = "crates/s3select-api", version = "1.0.2" }
|
||||
rustfs-s3select-query = { path = "crates/s3select-query", version = "1.0.2" }
|
||||
rustfs-scanner = { path = "crates/scanner", version = "1.0.2" }
|
||||
rustfs-security-governance = { path = "crates/security-governance", version = "1.0.2" }
|
||||
rustfs-extension-schema = { path = "crates/extension-schema", version = "1.0.2" }
|
||||
rustfs-signer = { path = "crates/signer", version = "1.0.2" }
|
||||
rustfs-storage-api = { path = "crates/storage-api", version = "1.0.2" }
|
||||
rustfs-trusted-proxies = { path = "crates/trusted-proxies", version = "1.0.2" }
|
||||
rustfs-targets = { path = "crates/targets", version = "1.0.2" }
|
||||
rustfs-test-utils = { path = "crates/test-utils", version = "1.0.2" }
|
||||
rustfs-tls-runtime = { path = "crates/tls-runtime", version = "1.0.2" }
|
||||
rustfs-utils = { path = "crates/utils", version = "1.0.2" }
|
||||
rustfs-zip = { path = "./crates/zip", version = "1.0.2" }
|
||||
rustfs = { path = "./rustfs", version = "1.0.5" }
|
||||
rustfs-heal = { path = "crates/heal", version = "1.0.5" }
|
||||
rustfs-heal-contracts = { path = "crates/heal-contracts", version = "1.0.5" }
|
||||
rustfs-scanner-metrics = { path = "crates/scanner-metrics", version = "1.0.5" }
|
||||
rustfs-audit = { path = "crates/audit", version = "1.0.5" }
|
||||
rustfs-checksums = { path = "crates/checksums", version = "1.0.5" }
|
||||
rustfs-common = { path = "crates/common", version = "1.0.5" }
|
||||
rustfs-data-usage = { path = "crates/data-usage", version = "1.0.5" }
|
||||
rustfs-config = { path = "./crates/config", version = "1.0.5" }
|
||||
rustfs-concurrency = { path = "./crates/concurrency", version = "1.0.5" }
|
||||
rustfs-credentials = { path = "crates/credentials", version = "1.0.5" }
|
||||
rustfs-crypto = { path = "crates/crypto", version = "1.0.5" }
|
||||
rustfs-ecstore = { path = "crates/ecstore", version = "1.0.5" }
|
||||
rustfs-filemeta = { path = "crates/filemeta", version = "1.0.5" }
|
||||
rustfs-iam = { path = "crates/iam", version = "1.0.5" }
|
||||
rustfs-keystone = { path = "crates/keystone", version = "1.0.5" }
|
||||
rustfs-license = { path = "crates/license", version = "1.0.5" }
|
||||
rustfs-lifecycle = { path = "crates/lifecycle", version = "1.0.5" }
|
||||
rustfs-kms = { path = "crates/kms", version = "1.0.5" }
|
||||
rustfs-lock = { path = "crates/lock", version = "1.0.5" }
|
||||
rustfs-madmin = { path = "crates/madmin", version = "1.0.5" }
|
||||
rustfs-notify = { path = "crates/notify", version = "1.0.5" }
|
||||
rustfs-io-metrics = { path = "crates/io-metrics", version = "1.0.5" }
|
||||
rustfs-io-core = { path = "crates/io-core", version = "1.0.5" }
|
||||
rustfs-object-capacity = { path = "crates/object-capacity", version = "1.0.5" }
|
||||
rustfs-object-data-cache = { path = "crates/object-data-cache", version = "1.0.5", default-features = false }
|
||||
rustfs-log-analyzer = { path = "crates/log-analyzer", version = "1.0.5" }
|
||||
rustfs-obs = { path = "crates/obs", version = "1.0.5" }
|
||||
rustfs-policy = { path = "crates/policy", version = "1.0.5" }
|
||||
rustfs-protos = { path = "crates/protos", version = "1.0.5" }
|
||||
rustfs-protocols = { path = "crates/protocols", version = "1.0.5" }
|
||||
rustfs-replication = { path = "crates/replication", version = "1.0.5" }
|
||||
rustfs-rio = { path = "crates/rio", version = "1.0.5" }
|
||||
rustfs-rio-v2 = { path = "crates/rio-v2", version = "1.0.5" }
|
||||
rustfs-s3-client = { path = "crates/s3-client", version = "1.0.5" }
|
||||
rustfs-s3-types = { path = "crates/s3-types", version = "1.0.5" }
|
||||
rustfs-s3-ops = { path = "crates/s3-ops", version = "1.0.5" }
|
||||
rustfs-s3select-api = { path = "crates/s3select-api", version = "1.0.5" }
|
||||
rustfs-s3select-query = { path = "crates/s3select-query", version = "1.0.5" }
|
||||
rustfs-scanner = { path = "crates/scanner", version = "1.0.5" }
|
||||
rustfs-security-governance = { path = "crates/security-governance", version = "1.0.5" }
|
||||
rustfs-extension-schema = { path = "crates/extension-schema", version = "1.0.5" }
|
||||
rustfs-signer = { path = "crates/signer", version = "1.0.5" }
|
||||
rustfs-storage-api = { path = "crates/storage-api", version = "1.0.5" }
|
||||
rustfs-trusted-proxies = { path = "crates/trusted-proxies", version = "1.0.5" }
|
||||
rustfs-targets = { path = "crates/targets", version = "1.0.5" }
|
||||
rustfs-test-utils = { path = "crates/test-utils", version = "1.0.5" }
|
||||
rustfs-tls-runtime = { path = "crates/tls-runtime", version = "1.0.5" }
|
||||
rustfs-utils = { path = "crates/utils", version = "1.0.5" }
|
||||
rustfs-zip = { path = "./crates/zip", version = "1.0.5" }
|
||||
|
||||
# Async Runtime and Networking
|
||||
async-channel = "2.5.0"
|
||||
@@ -296,6 +296,7 @@ local-ip-address = "0.6.13"
|
||||
log = "0.4"
|
||||
memmap2 = "0.9.12"
|
||||
lz4 = "1.28.1"
|
||||
lz4_flex = { version = "=0.14.0", default-features = false, features = ["std", "frame"] }
|
||||
matchit = "0.9.2"
|
||||
md-5 = "0.11.0"
|
||||
mime_guess = "2.0.5"
|
||||
|
||||
+11
-2
@@ -12,7 +12,9 @@
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
FROM alpine:3.24.2 AS build
|
||||
ARG BINARY_SOURCE=download
|
||||
|
||||
FROM alpine:3.24.2 AS download
|
||||
|
||||
ARG TARGETARCH
|
||||
ARG RELEASE=latest
|
||||
@@ -74,6 +76,14 @@ COPY scripts/download_rc.sh /usr/local/bin/download_rc.sh
|
||||
RUN sh /usr/local/bin/download_rc.sh "$TARGETARCH" musl "$RC_RELEASE" /build/rc
|
||||
|
||||
|
||||
# Candidate packaging supplies the accepted executables from verified archives.
|
||||
# The default download stage remains available for local image builds.
|
||||
FROM scratch AS supplied
|
||||
ARG TARGETARCH
|
||||
COPY .release-inputs/${TARGETARCH}/ /build/
|
||||
|
||||
FROM ${BINARY_SOURCE} AS build
|
||||
|
||||
FROM alpine:3.24.2
|
||||
|
||||
ARG RELEASE=latest
|
||||
@@ -103,7 +113,6 @@ RUN apk upgrade --no-cache && \
|
||||
tzdata \
|
||||
&& test "$(TZ=Asia/Kolkata date +%z)" = "+0530"
|
||||
|
||||
COPY --from=build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
|
||||
COPY --from=build /build/rustfs /usr/bin/rustfs
|
||||
COPY --from=build /build/rc /usr/bin/rc
|
||||
COPY entrypoint.sh /etc/rustfs/entrypoint.sh
|
||||
|
||||
+11
-1
@@ -12,7 +12,9 @@
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
FROM ubuntu:26.04 AS build
|
||||
ARG BINARY_SOURCE=download
|
||||
|
||||
FROM ubuntu:26.04 AS download
|
||||
|
||||
ARG TARGETARCH
|
||||
ARG RELEASE=latest
|
||||
@@ -80,6 +82,14 @@ RUN set -eux; \
|
||||
COPY scripts/download_rc.sh /usr/local/bin/download_rc.sh
|
||||
RUN sh /usr/local/bin/download_rc.sh "$TARGETARCH" gnu "$RC_RELEASE" /build/rc
|
||||
|
||||
# Candidate packaging supplies the accepted executables from verified archives.
|
||||
# The default download stage remains available for local image builds.
|
||||
FROM scratch AS supplied
|
||||
ARG TARGETARCH
|
||||
COPY .release-inputs/${TARGETARCH}/ /build/
|
||||
|
||||
FROM ${BINARY_SOURCE} AS build
|
||||
|
||||
FROM ubuntu:26.04
|
||||
|
||||
ARG RELEASE=latest
|
||||
|
||||
@@ -184,15 +184,27 @@ const _: () = assert!(!DEFAULT_INTERNODE_RPC_REPLAY_SCOPE_STRICT);
|
||||
/// Capacity (distinct nonces) of the process-local internode RPC replay cache that enforces
|
||||
/// one-time consumption of authenticated RPC signatures.
|
||||
///
|
||||
/// The cache retains each nonce for the ~10-minute signature freshness envelope. Once peers use
|
||||
/// replay-scoped v3 authentication, every authenticated RPC consumes one entry, so the steady
|
||||
/// state holds roughly `authenticated RPC RPS x 601s` entries. This default is the minimum floor:
|
||||
/// explicit operator values and resource-aware auto sizing both clamp upward to at least this
|
||||
/// value. Overflow fails closed — legitimate signed traffic is the only thing that can fill the
|
||||
/// cache (replays are rejected before insertion, and an attacker cannot mint valid nonces without
|
||||
/// the shared secret) — and increments
|
||||
/// `rustfs_system_network_internode_replay_cache_overflow_total`, so a sustained non-zero overflow
|
||||
/// counter means this capacity is undersized for the node's peak authenticated RPC rate.
|
||||
/// Nonces remain protected until their signed timestamp is outside the observed wall-clock
|
||||
/// validity window, including after clock rollback. Current-time signatures live about five
|
||||
/// minutes; the maximum accepted forward clock skew can extend that to ten minutes. A body-bound
|
||||
/// v3 request consumes both its v3 and compatibility-v2 nonces, so size for receiver nonce intake,
|
||||
/// not S3 request rate or CPU count.
|
||||
///
|
||||
/// Automatic admission budgets 13% of effective process memory for signed-time
|
||||
/// hash tables, allocator overhead and simultaneous old/new tables during growth.
|
||||
/// The reported count is an absolute UUID payload ceiling, not guaranteed throughput:
|
||||
/// actual byte admission can reject earlier depending on timestamp distribution.
|
||||
/// Tables grow with demand and are released together when their signed time expires.
|
||||
/// CPU count, a historical minimum, and a fixed 32M ceiling do not constrain this budget.
|
||||
/// If memory detection fails, the historical default remains a bounded count fallback.
|
||||
///
|
||||
/// Explicit values remain count ceilings with the historical minimum of 1,048,576
|
||||
/// for configuration compatibility, and bypass automatic byte admission. That
|
||||
/// compatibility floor does not apply to the automatically detected memory budget.
|
||||
/// Zero never disables replay protection.
|
||||
/// Overflow rejects the request without evicting a still-valid identity, increments
|
||||
/// `rustfs_system_network_internode_replay_cache_overflow_total`, and returns resource
|
||||
/// exhaustion rather than invalid credentials. Measure headroom before overriding.
|
||||
pub const ENV_INTERNODE_RPC_REPLAY_CACHE_CAPACITY: &str = "RUSTFS_INTERNODE_RPC_REPLAY_CACHE_CAPACITY";
|
||||
pub const DEFAULT_INTERNODE_RPC_REPLAY_CACHE_CAPACITY: usize = 1_048_576;
|
||||
|
||||
|
||||
@@ -121,6 +121,7 @@ test-util = []
|
||||
e2e-test-hooks = []
|
||||
|
||||
[dependencies]
|
||||
hashbrown.workspace = true
|
||||
hotpath.workspace = true
|
||||
rustfs-filemeta.workspace = true
|
||||
rustfs-utils = { workspace = true, features = ["full"] }
|
||||
|
||||
@@ -640,17 +640,17 @@ pub mod rpc {
|
||||
pub use crate::cluster::rpc::{
|
||||
AuthenticatedChannel, KMS_SIGNAL_SUBSYSTEM, LocalPeerS3Client, MAX_NETWORK_PROBE_BYTES, MAX_NETWORK_PROBE_DURATION,
|
||||
NetworkPeerProbeClient, NetworkPeerProbeError, NetworkPeerProbeMeasurement, NetworkPeerTarget, PEER_RESTDRY_RUN,
|
||||
PEER_RESTSIGNAL, PEER_RESTSUB_SYS, PeerRestClient, PeerS3Client, S3PeerSys, SERVICE_SIGNAL_REFRESH_CONFIG,
|
||||
SERVICE_SIGNAL_RELOAD_DYNAMIC, ScannerBucketListing, ScannerDirtyUsageAcknowledgement, ScannerPeerActivity,
|
||||
ScannerPeerDirtyUsageBucket, ScannerPeerDirtyUsageSnapshot, ScannerPublicationLease, ScannerScopedDirtyUsageAckEntry,
|
||||
TONIC_RPC_PREFIX, TonicInterceptor, build_put_file_auth_trailer, check_and_record_signed_rpc_nonce,
|
||||
decode_heal_bucket_rpc_options, encode_heal_bucket_rpc_options, gen_signature_headers, gen_tonic_replay_scope_headers,
|
||||
gen_tonic_signature_headers, gen_tonic_signature_interceptor, node_service_time_out_client,
|
||||
node_service_time_out_client_no_auth, normalize_tonic_rpc_audience, rpc_background_admission,
|
||||
set_tonic_canonical_body_digest, sign_ns_scanner_capability, sign_ns_scanner_capability_with_checkpoint_coverage,
|
||||
sign_ns_scanner_capability_with_tier_registry_generation, sign_put_file_capability, sign_tonic_rpc_response_proof,
|
||||
tonic_boot_epoch_challenge, tonic_boot_epoch_response_headers, tonic_rpc_auth_failure_reason,
|
||||
verify_ns_scanner_capability, verify_ns_scanner_capability_with_checkpoint_coverage,
|
||||
PEER_RESTSIGNAL, PEER_RESTSUB_SYS, PeerRestClient, PeerS3Client, RpcReplayCacheAdmissionError, S3PeerSys,
|
||||
SERVICE_SIGNAL_REFRESH_CONFIG, SERVICE_SIGNAL_RELOAD_DYNAMIC, ScannerBucketListing, ScannerDirtyUsageAcknowledgement,
|
||||
ScannerPeerActivity, ScannerPeerDirtyUsageBucket, ScannerPeerDirtyUsageSnapshot, ScannerPublicationLease,
|
||||
ScannerScopedDirtyUsageAckEntry, TONIC_RPC_PREFIX, TonicInterceptor, build_put_file_auth_trailer,
|
||||
check_and_record_signed_rpc_nonce, decode_heal_bucket_rpc_options, encode_heal_bucket_rpc_options, gen_signature_headers,
|
||||
gen_tonic_replay_scope_headers, gen_tonic_signature_headers, gen_tonic_signature_interceptor,
|
||||
node_service_time_out_client, node_service_time_out_client_no_auth, normalize_tonic_rpc_audience,
|
||||
rpc_auth_is_resource_exhausted, rpc_background_admission, set_tonic_canonical_body_digest, sign_ns_scanner_capability,
|
||||
sign_ns_scanner_capability_with_checkpoint_coverage, sign_ns_scanner_capability_with_tier_registry_generation,
|
||||
sign_put_file_capability, sign_tonic_rpc_response_proof, tonic_boot_epoch_challenge, tonic_boot_epoch_response_headers,
|
||||
tonic_rpc_auth_failure_reason, verify_ns_scanner_capability, verify_ns_scanner_capability_with_checkpoint_coverage,
|
||||
verify_ns_scanner_capability_with_tier_registry_generation, verify_put_file_auth_trailer, verify_put_file_capability,
|
||||
verify_rpc_signature, verify_tonic_boot_epoch_response, verify_tonic_canonical_body_digest,
|
||||
verify_tonic_mutation_body_digest, verify_tonic_mutation_body_digest_reject_unsigned, verify_tonic_rpc_response_proof,
|
||||
|
||||
@@ -229,6 +229,7 @@ pub struct ListPathRawOptions {
|
||||
pub hidden_entry_budget: usize,
|
||||
pub pending_prefix: Option<String>,
|
||||
pub filter_prefix: Option<String>,
|
||||
pub identifier_tree_leaf: Option<String>,
|
||||
pub forward_to: Option<String>,
|
||||
pub min_disks: usize,
|
||||
/// Deliver every replica to the partial callback, including matching headers.
|
||||
@@ -268,6 +269,7 @@ impl Clone for ListPathRawOptions {
|
||||
hidden_entry_budget: self.hidden_entry_budget,
|
||||
pending_prefix: self.pending_prefix.clone(),
|
||||
filter_prefix: self.filter_prefix.clone(),
|
||||
identifier_tree_leaf: self.identifier_tree_leaf.clone(),
|
||||
forward_to: self.forward_to.clone(),
|
||||
min_disks: self.min_disks,
|
||||
preserve_replica_metadata: self.preserve_replica_metadata,
|
||||
@@ -300,6 +302,7 @@ fn walk_dir_options(opts: &ListPathRawOptions) -> WalkDirOptions {
|
||||
pending_prefix: opts.pending_prefix.clone(),
|
||||
report_notfound: opts.report_not_found,
|
||||
filter_prefix: opts.filter_prefix.clone(),
|
||||
identifier_tree_leaf: opts.identifier_tree_leaf.clone(),
|
||||
forward_to: opts.forward_to.clone(),
|
||||
limit: opts.per_disk_limit,
|
||||
skip_total_timeout: opts.skip_walkdir_total_timeout,
|
||||
@@ -1233,6 +1236,21 @@ mod tests {
|
||||
assert!(!options.skip_total_timeout);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn identifier_tree_selection_survives_job_clone_and_disk_request() {
|
||||
let request = ListPathRawOptions {
|
||||
path: "catalog/namespaces/".to_string(),
|
||||
identifier_tree_leaf: Some("table-entry.json".to_string()),
|
||||
recursive: true,
|
||||
..Default::default()
|
||||
};
|
||||
let cloned = request.clone();
|
||||
let walk = walk_dir_options(&cloned);
|
||||
assert_eq!(walk.identifier_tree_leaf, request.identifier_tree_leaf);
|
||||
assert_eq!(walk.base_dir, request.path);
|
||||
assert!(walk.recursive);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn cloned_reader_options_preserve_hidden_scan_budget_and_pending_prefix() {
|
||||
let request = ListPathRawOptions {
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -35,17 +35,17 @@ pub use client::{
|
||||
// Re-exported through `api::rpc`; not every item is consumed inside this crate.
|
||||
#[allow(unused_imports)]
|
||||
pub use http_auth::{
|
||||
RPC_BACKGROUND_ADMISSION_HEADER, TONIC_RPC_PREFIX, build_auth_headers, build_put_file_auth_trailer,
|
||||
check_and_record_signed_rpc_nonce, gen_signature_headers, gen_tonic_replay_scope_headers, gen_tonic_signature_headers,
|
||||
normalize_tonic_rpc_audience, rpc_background_admission, set_tonic_canonical_body_digest, set_tonic_mutation_body_digest,
|
||||
set_tonic_rolling_canonical_body_digest, set_tonic_rolling_mutation_body_digest, sign_ns_scanner_capability,
|
||||
sign_ns_scanner_capability_with_checkpoint_coverage, sign_ns_scanner_capability_with_tier_registry_generation,
|
||||
sign_put_file_capability, sign_tonic_rpc_response_proof, tonic_boot_epoch_challenge, tonic_boot_epoch_response_headers,
|
||||
tonic_rpc_auth_failure_reason, verify_ns_scanner_capability, verify_ns_scanner_capability_with_checkpoint_coverage,
|
||||
verify_ns_scanner_capability_with_tier_registry_generation, verify_put_file_auth_trailer, verify_put_file_capability,
|
||||
verify_rpc_signature, verify_tonic_boot_epoch_response, verify_tonic_canonical_body_digest,
|
||||
verify_tonic_mutation_body_digest, verify_tonic_mutation_body_digest_reject_unsigned, verify_tonic_rpc_response_proof,
|
||||
verify_tonic_rpc_signature, verify_tonic_rpc_signature_with_bootstrap,
|
||||
RPC_BACKGROUND_ADMISSION_HEADER, RpcReplayCacheAdmissionError, TONIC_RPC_PREFIX, build_auth_headers,
|
||||
build_put_file_auth_trailer, check_and_record_signed_rpc_nonce, gen_signature_headers, gen_tonic_replay_scope_headers,
|
||||
gen_tonic_signature_headers, normalize_tonic_rpc_audience, rpc_auth_is_resource_exhausted, rpc_background_admission,
|
||||
set_tonic_canonical_body_digest, set_tonic_mutation_body_digest, set_tonic_rolling_canonical_body_digest,
|
||||
set_tonic_rolling_mutation_body_digest, sign_ns_scanner_capability, sign_ns_scanner_capability_with_checkpoint_coverage,
|
||||
sign_ns_scanner_capability_with_tier_registry_generation, sign_put_file_capability, sign_tonic_rpc_response_proof,
|
||||
tonic_boot_epoch_challenge, tonic_boot_epoch_response_headers, tonic_rpc_auth_failure_reason, verify_ns_scanner_capability,
|
||||
verify_ns_scanner_capability_with_checkpoint_coverage, verify_ns_scanner_capability_with_tier_registry_generation,
|
||||
verify_put_file_auth_trailer, verify_put_file_capability, verify_rpc_signature, verify_tonic_boot_epoch_response,
|
||||
verify_tonic_canonical_body_digest, verify_tonic_mutation_body_digest, verify_tonic_mutation_body_digest_reject_unsigned,
|
||||
verify_tonic_rpc_response_proof, verify_tonic_rpc_signature, verify_tonic_rpc_signature_with_bootstrap,
|
||||
};
|
||||
#[cfg(test)]
|
||||
pub(crate) use internode_data_transport::TcpHttpInternodeDataTransport;
|
||||
|
||||
@@ -18,7 +18,7 @@ use crate::cluster::rpc::client::{
|
||||
node_service_time_out_client,
|
||||
};
|
||||
use crate::cluster::rpc::set_tonic_mutation_body_digest;
|
||||
use crate::core::pools::{PoolMeta, PoolMetaWriteState};
|
||||
use crate::core::pools::{PoolMeta, PoolMetaSaveGate, PoolMetaSaveGateWriteGuard};
|
||||
use crate::disk::error::DiskError;
|
||||
use crate::disk::error::{Error, Result};
|
||||
use crate::disk::error_reduce::{BUCKET_OP_IGNORED_ERRS, is_all_buckets_not_found, reduce_write_quorum_errs};
|
||||
@@ -798,12 +798,12 @@ fn try_acquire_bucket_heal_movement_guard<'a>(
|
||||
}
|
||||
|
||||
async fn acquire_bucket_heal_write_guard<'a>(
|
||||
gate: Option<&'a tokio::sync::Mutex<PoolMetaWriteState>>,
|
||||
) -> Result<Option<tokio::sync::MutexGuard<'a, PoolMetaWriteState>>> {
|
||||
gate: Option<&'a PoolMetaSaveGate>,
|
||||
) -> Result<Option<PoolMetaSaveGateWriteGuard<'a>>> {
|
||||
let Some(gate) = gate else {
|
||||
return Ok(None);
|
||||
};
|
||||
let guard = gate.lock().await;
|
||||
let guard = gate.write().await;
|
||||
guard
|
||||
.ensure_write_safe("bucket heal cannot run while pool metadata requires recovery")
|
||||
.map_err(|e| e.narrow_to_disk().unwrap_or_else(Error::other))?;
|
||||
@@ -2388,8 +2388,8 @@ mod tests {
|
||||
|
||||
#[tokio::test]
|
||||
async fn local_bucket_heal_refuses_receiver_with_unsafe_pool_metadata() {
|
||||
let gate = tokio::sync::Mutex::new(PoolMetaWriteState::default());
|
||||
gate.lock().await.observe_replicas(PoolMetaReplicaState {
|
||||
let gate = PoolMetaSaveGate::default();
|
||||
gate.write().await.observe_replicas(PoolMetaReplicaState {
|
||||
needs_repair: true,
|
||||
repair_write_safe: false,
|
||||
});
|
||||
|
||||
@@ -628,7 +628,7 @@ async fn pool_meta_native_physical_runtime_fork_after_apply_latches_original_gat
|
||||
let fork = identity_epoch_fork(&fixture).await;
|
||||
let since = OffsetDateTime::from_unix_timestamp(1_700_000_000).expect("fixed original failure timestamp should be valid");
|
||||
{
|
||||
let mut state = fixture.store.pool_meta_save_gate.lock().await;
|
||||
let mut state = fixture.store.pool_meta_save_gate.write().await;
|
||||
assert_eq!(state.active_transactions.load(Ordering::SeqCst), 0);
|
||||
state.write_blocked = false;
|
||||
state.block_context = None;
|
||||
@@ -685,7 +685,7 @@ async fn pool_meta_native_physical_runtime_fork_after_apply_latches_original_gat
|
||||
);
|
||||
assert_committed(&fixture).await;
|
||||
{
|
||||
let state = fixture.store.pool_meta_save_gate.lock().await;
|
||||
let state = fixture.store.pool_meta_save_gate.write().await;
|
||||
assert!(
|
||||
state.write_blocked,
|
||||
"candidate validation must propagate its permanent latch to the original state"
|
||||
@@ -706,7 +706,7 @@ async fn pool_meta_native_physical_runtime_fork_after_apply_latches_original_gat
|
||||
.expect("permanently blocked recovery should return without mutation"),
|
||||
"ordinary-readable repaired heads must not clear the original known-fork latch"
|
||||
);
|
||||
let state = fixture.store.pool_meta_save_gate.lock().await;
|
||||
let state = fixture.store.pool_meta_save_gate.write().await;
|
||||
assert!(state.write_blocked);
|
||||
assert_eq!(
|
||||
state
|
||||
@@ -979,7 +979,7 @@ async fn pool_meta_native_heal_v1_with_all_upgrade_gates_disabled_keeps_exact_ra
|
||||
async {
|
||||
let (fixture, raw) = legacy_native_owner_fixture(POOL_META_V1_VERSION, false).await;
|
||||
let guard = owner(&fixture.store).await;
|
||||
let mut state = fixture.store.pool_meta_save_gate.lock().await;
|
||||
let mut state = fixture.store.pool_meta_save_gate.write().await;
|
||||
let result = tokio::time::timeout(
|
||||
std::time::Duration::from_secs(30),
|
||||
fixture.store.heal_pool_meta_replica_under_fence(0, guard, &mut state),
|
||||
@@ -1036,7 +1036,7 @@ async fn pool_meta_native_heal_rejects_legacy_source_with_trailing_bytes_without
|
||||
let before = snapshots(&fixture.store).await;
|
||||
let identities = owner_repair_identity_heads(&fixture.store).await;
|
||||
let guard = owner(&fixture.store).await;
|
||||
let mut state = fixture.store.pool_meta_save_gate.lock().await;
|
||||
let mut state = fixture.store.pool_meta_save_gate.write().await;
|
||||
let result = tokio::time::timeout(
|
||||
std::time::Duration::from_secs(30),
|
||||
fixture.store.heal_pool_meta_replica_under_fence(1, guard, &mut state),
|
||||
@@ -1110,7 +1110,7 @@ async fn pool_meta_native_heal_mixed_v1_v2_preserves_observed_format_floor() {
|
||||
assert_eq!(selection.revision.version, POOL_META_VERSION, "the observed V2 format floor must be retained");
|
||||
assert_eq!(selection.canonical.as_deref(), Some(semantic_v2.as_slice()));
|
||||
let guard = owner(&fixture.store).await;
|
||||
let mut state = fixture.store.pool_meta_save_gate.lock().await;
|
||||
let mut state = fixture.store.pool_meta_save_gate.write().await;
|
||||
tokio::time::timeout(std::time::Duration::from_secs(30),
|
||||
fixture.store.heal_pool_meta_replica_under_fence(target_pool, guard, &mut state))
|
||||
.await.expect("mixed legacy heal must terminate")
|
||||
@@ -1199,7 +1199,7 @@ async fn pool_meta_native_heal_rejects_v1_and_v3_prepare_in_either_pool_order()
|
||||
let before = snapshots(&fixture.store).await;
|
||||
let identities = owner_repair_identity_heads(&fixture.store).await;
|
||||
let guard = owner(&fixture.store).await;
|
||||
let mut state = fixture.store.pool_meta_save_gate.lock().await;
|
||||
let mut state = fixture.store.pool_meta_save_gate.write().await;
|
||||
let result = tokio::time::timeout(
|
||||
std::time::Duration::from_secs(30),
|
||||
fixture
|
||||
@@ -1301,7 +1301,7 @@ async fn pool_meta_native_heal_rejects_v1_legacy_dont_save_true_without_writes()
|
||||
let before = snapshots(&fixture.store).await;
|
||||
let identities = owner_repair_identity_heads(&fixture.store).await;
|
||||
let guard = owner(&fixture.store).await;
|
||||
let mut state = fixture.store.pool_meta_save_gate.lock().await;
|
||||
let mut state = fixture.store.pool_meta_save_gate.write().await;
|
||||
let result = tokio::time::timeout(
|
||||
std::time::Duration::from_secs(30),
|
||||
fixture.store.heal_pool_meta_replica_under_fence(1, guard, &mut state),
|
||||
@@ -1410,7 +1410,7 @@ async fn pool_meta_native_heal_rejects_target_v1_dont_save_true_without_writes()
|
||||
let before = snapshots(&fixture.store).await;
|
||||
let identities = owner_repair_identity_heads(&fixture.store).await;
|
||||
let guard = owner(&fixture.store).await;
|
||||
let mut state = fixture.store.pool_meta_save_gate.lock().await;
|
||||
let mut state = fixture.store.pool_meta_save_gate.write().await;
|
||||
let result = tokio::time::timeout(
|
||||
std::time::Duration::from_secs(30),
|
||||
fixture.store.heal_pool_meta_replica_under_fence(1, guard, &mut state),
|
||||
|
||||
+454
-137
File diff suppressed because it is too large
Load Diff
@@ -527,8 +527,8 @@ mod decommission_lock_order_tests {
|
||||
rebalance_meta: tokio::sync::RwLock::new(other_store.rebalance_meta.read().await.clone()),
|
||||
decommission_cancelers: tokio::sync::RwLock::new(vec![None; other_store.pools.len()]),
|
||||
start_gate: tokio::sync::Mutex::new(()),
|
||||
pool_meta_save_gate: tokio::sync::Mutex::new(
|
||||
other_store.pool_meta_save_gate.lock().await.independent_clone_for_test(),
|
||||
pool_meta_save_gate: crate::core::pools::PoolMetaSaveGate::new(
|
||||
other_store.pool_meta_save_gate.write().await.independent_clone_for_test(),
|
||||
),
|
||||
ctx,
|
||||
bucket_fence_registry: Arc::default(),
|
||||
|
||||
@@ -85,6 +85,7 @@ impl ScannerPauseBacklogRetirementPhase {
|
||||
pub struct ScannerPauseBacklogRetirementReplica {
|
||||
pub pool_index: usize,
|
||||
pub set_index: usize,
|
||||
pub set_id: Option<uuid::Uuid>,
|
||||
/// False for retiring members retained only to verify the old commit proof.
|
||||
pub writable: bool,
|
||||
pub data: Option<Vec<u8>>,
|
||||
@@ -140,6 +141,7 @@ async fn read_replica((set, writable): (Arc<SetDisks>, bool)) -> Result<ScannerP
|
||||
let mut replica = ScannerPauseBacklogRetirementReplica {
|
||||
pool_index: set.pool_index,
|
||||
set_index: set.set_index,
|
||||
set_id: Some(set.persistent_set_id()?),
|
||||
writable,
|
||||
data: None,
|
||||
};
|
||||
@@ -222,6 +224,7 @@ pub(crate) fn plan_scanner_pause_backlog_retirement(
|
||||
.map(|read| ScannerPauseBacklogRetirementReplica {
|
||||
pool_index: read.replica.pool_index,
|
||||
set_index: read.replica.set_index,
|
||||
set_id: read.replica.set_id,
|
||||
writable: read.replica.writable,
|
||||
data: read.replica.data.clone(),
|
||||
})
|
||||
|
||||
@@ -685,12 +685,8 @@ async fn clear_bucket_usage_memory(bucket: &str, guard: Option<&rustfs_lock::Nam
|
||||
}
|
||||
|
||||
pub async fn remove_bucket_usage_from_backend(store: Arc<ECStore>, bucket: &str) -> Result<(), Error> {
|
||||
remove_bucket_usage_for_namespace_change(store.as_ref(), bucket).await
|
||||
}
|
||||
|
||||
pub(crate) async fn remove_bucket_usage_for_namespace_change(store: &ECStore, bucket: &str) -> Result<(), Error> {
|
||||
prepare_bucket_usage_for_namespace_change(bucket, None).await?;
|
||||
remove_bucket_usage_from_backend_with_guard_fenced(store, bucket, None).await
|
||||
remove_bucket_usage_with_owned_namespace(store, bucket, None).await
|
||||
}
|
||||
|
||||
pub(crate) async fn prepare_bucket_usage_for_namespace_change(
|
||||
@@ -730,11 +726,37 @@ where
|
||||
pub(crate) async fn remove_bucket_usage_from_backend_with_guard_fenced(
|
||||
store: &ECStore,
|
||||
bucket: &str,
|
||||
guard: Option<&rustfs_lock::NamespaceLockGuard>,
|
||||
guard: Option<&Arc<rustfs_lock::NamespaceLockGuard>>,
|
||||
) -> Result<(), Error> {
|
||||
let result = remove_bucket_usage_from_backend_with_store_and_guard_and_publication(store, bucket, guard, Some(store)).await;
|
||||
invalidate_bucket_usage_snapshot_caches(guard, bucket).await?;
|
||||
result
|
||||
let owned_store = crate::bucket::metadata_sys::object_store_in(&store.ctx).await?;
|
||||
if !std::ptr::eq(store, owned_store.as_ref()) {
|
||||
return Err(Error::other("usage cleanup requires its own metadata store"));
|
||||
}
|
||||
remove_bucket_usage_with_owned_namespace(owned_store, bucket, guard.cloned()).await
|
||||
}
|
||||
|
||||
async fn remove_bucket_usage_with_owned_namespace(
|
||||
store: Arc<ECStore>,
|
||||
bucket: &str,
|
||||
guard: Option<Arc<rustfs_lock::NamespaceLockGuard>>,
|
||||
) -> Result<(), Error> {
|
||||
let bucket = bucket.to_owned();
|
||||
let ctx = Arc::clone(&store.ctx);
|
||||
// Retain the bucket generation and movement/object guards through every
|
||||
// physical commit even when the namespace request stops waiting.
|
||||
ctx.run_detached_mutation(async move {
|
||||
let result = remove_bucket_usage_from_backend_with_store_and_guard_and_publication(
|
||||
store.as_ref(),
|
||||
&bucket,
|
||||
guard.as_deref(),
|
||||
Some(store.as_ref()),
|
||||
)
|
||||
.await;
|
||||
invalidate_bucket_usage_snapshot_caches(guard.as_deref(), &bucket).await?;
|
||||
result
|
||||
})
|
||||
.await
|
||||
.map_err(Error::from)?
|
||||
}
|
||||
|
||||
async fn invalidate_bucket_usage_snapshot_caches(
|
||||
@@ -849,9 +871,9 @@ where
|
||||
let primary_seed_epoch = match publication_store {
|
||||
Some(publication_store) => Some(
|
||||
publication_store
|
||||
.scanner_data_usage_publication_epoch()
|
||||
.data_usage_namespace_cleanup_epoch()
|
||||
.await
|
||||
.ok_or_else(|| Error::other("data usage publication is blocked by data movement"))?,
|
||||
.ok_or_else(|| Error::other("data usage namespace cleanup epoch is unavailable"))?,
|
||||
),
|
||||
None => None,
|
||||
};
|
||||
@@ -871,9 +893,9 @@ where
|
||||
let backup_seed_epoch = match publication_store {
|
||||
Some(publication_store) => Some(
|
||||
publication_store
|
||||
.scanner_data_usage_publication_epoch()
|
||||
.data_usage_namespace_cleanup_epoch()
|
||||
.await
|
||||
.ok_or_else(|| Error::other("data usage publication is blocked by data movement"))?,
|
||||
.ok_or_else(|| Error::other("data usage namespace cleanup epoch is unavailable"))?,
|
||||
),
|
||||
None => None,
|
||||
};
|
||||
@@ -985,9 +1007,9 @@ where
|
||||
let read_epoch = match publication {
|
||||
Some((publication_store, expected_publication_epoch)) => {
|
||||
let epoch = publication_store
|
||||
.scanner_data_usage_publication_epoch()
|
||||
.data_usage_namespace_cleanup_epoch()
|
||||
.await
|
||||
.ok_or_else(|| Error::other("data usage publication is blocked by data movement"))?;
|
||||
.ok_or_else(|| Error::other("data usage namespace cleanup epoch is unavailable"))?;
|
||||
if expected_publication_epoch.is_some_and(|expected| expected != epoch) {
|
||||
return Err(Error::other("data usage publication epoch changed before snapshot read"));
|
||||
}
|
||||
@@ -1022,9 +1044,8 @@ where
|
||||
ensure_bucket_namespace_guard(guard, bucket, "data usage snapshot commit")?;
|
||||
let publication_guard = match publication {
|
||||
Some((publication_store, expected_publication_epoch)) => {
|
||||
let Some((guard, publication_epoch)) = publication_store.scanner_data_usage_publication_admission_guard().await
|
||||
else {
|
||||
return Err(Error::other("data usage publication is blocked by data movement"));
|
||||
let Some((guard, publication_epoch)) = publication_store.data_usage_namespace_cleanup_guard().await else {
|
||||
return Err(Error::other("data usage namespace cleanup epoch is unavailable"));
|
||||
};
|
||||
if expected_publication_epoch
|
||||
.or(read_epoch)
|
||||
@@ -1036,18 +1057,27 @@ where
|
||||
}
|
||||
None => None,
|
||||
};
|
||||
let save_result = store
|
||||
.put_object(
|
||||
RUSTFS_META_BUCKET,
|
||||
object,
|
||||
&mut put_data,
|
||||
&ObjectOptions {
|
||||
max_parity: true,
|
||||
http_preconditions: Some(http_preconditions),
|
||||
..Default::default()
|
||||
},
|
||||
)
|
||||
.await;
|
||||
let save_result = match publication {
|
||||
Some((publication_store, _)) => {
|
||||
publication_store
|
||||
.put_usage_snapshot_for_namespace_cleanup(object, revision.as_deref(), &mut put_data, guard)
|
||||
.await
|
||||
}
|
||||
None => {
|
||||
store
|
||||
.put_object(
|
||||
RUSTFS_META_BUCKET,
|
||||
object,
|
||||
&mut put_data,
|
||||
&ObjectOptions {
|
||||
max_parity: true,
|
||||
http_preconditions: Some(http_preconditions),
|
||||
..Default::default()
|
||||
},
|
||||
)
|
||||
.await
|
||||
}
|
||||
};
|
||||
drop(publication_guard);
|
||||
match save_result.map_err(|err| crate::config::com::map_system_metadata_write_error(err, object)) {
|
||||
Ok(_) => return Ok(()),
|
||||
@@ -3436,7 +3466,7 @@ mod tests {
|
||||
rebalance_meta: RwLock::new(None),
|
||||
decommission_cancelers: RwLock::new(Vec::new()),
|
||||
start_gate: TokioMutex::new(()),
|
||||
pool_meta_save_gate: TokioMutex::default(),
|
||||
pool_meta_save_gate: crate::core::pools::PoolMetaSaveGate::default(),
|
||||
ctx,
|
||||
bucket_fence_registry: Arc::default(),
|
||||
})
|
||||
|
||||
@@ -579,6 +579,7 @@ struct PreparedPartTxnReceipt {
|
||||
dst_meta_path: PathBuf,
|
||||
transaction_path: PathBuf,
|
||||
meta: Bytes,
|
||||
had_old_data: bool,
|
||||
had_old_meta: bool,
|
||||
}
|
||||
|
||||
@@ -586,6 +587,7 @@ struct PublishedPartTxnReceipt {
|
||||
current_data_path: PathBuf,
|
||||
current_meta_path: PathBuf,
|
||||
transaction_path: PathBuf,
|
||||
had_old_data: bool,
|
||||
had_old_meta: bool,
|
||||
}
|
||||
|
||||
@@ -977,6 +979,7 @@ async fn restore_delete_rollback_with_namespace_owner(
|
||||
.insert(backup_path.clone());
|
||||
os::rename_all_with_owner(&backup_path, xl_path, object_dir, publication_root, namespace_owner.clone()).await
|
||||
};
|
||||
let restored_backup = restored.is_ok();
|
||||
match restored {
|
||||
Ok(()) => {}
|
||||
Err(DiskError::FileNotFound) if remove_new_metadata => {
|
||||
@@ -1006,6 +1009,16 @@ async fn restore_delete_rollback_with_namespace_owner(
|
||||
os::fsync_dir_with_owner(object_dir, namespace_owner.clone())
|
||||
.await
|
||||
.map_err(to_file_error)?;
|
||||
}
|
||||
if restored_backup {
|
||||
// Renaming two hardlinks to the same inode can leave the backup in place.
|
||||
match os::remove_file_with_owner(&backup_path, namespace_owner.clone()).await {
|
||||
Ok(()) => {}
|
||||
Err(err) if err.kind() == ErrorKind::NotFound => {}
|
||||
Err(err) => return Err(to_file_error(err).into()),
|
||||
}
|
||||
}
|
||||
if effective_durability(volume).syncs_commit_metadata() {
|
||||
os::fsync_dir_with_owner(&rollback_path, namespace_owner.clone())
|
||||
.await
|
||||
.map_err(to_file_error)?;
|
||||
@@ -3794,6 +3807,25 @@ async fn read_dir_entries_with_walk_stall(path: &Path, count: i32, stall: Option
|
||||
const WALK_DIR_BATCH_SIZE: usize = 32;
|
||||
#[cfg(not(test))]
|
||||
const WALK_DIR_BATCH_SIZE: usize = 256;
|
||||
|
||||
async fn read_identifier_tree_entries(path: &Path, leaf: &str, stall: Option<Duration>) -> Result<Vec<String>> {
|
||||
let mut entries = read_dir_with_walk_stall(path.to_path_buf(), stall).await?;
|
||||
let mut names = Vec::new();
|
||||
loop {
|
||||
let (next, batch, eof) = read_dir_entry_batch_with_walk_stall(entries, READ_DIR_BATCH_SIZE, stall).await?;
|
||||
entries = next;
|
||||
// Filter before retaining or sorting the inventory. A report flood
|
||||
// costs directory iteration, not one retained name/metadata read per report.
|
||||
names.extend(batch.into_iter().filter(|name| {
|
||||
name.strip_suffix(SLASH_SEPARATOR)
|
||||
.is_some_and(|component| component == leaf || super::is_identifier_tree_component(component))
|
||||
}));
|
||||
if eof {
|
||||
return Ok(names);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const READ_DIR_BATCH_SIZE: usize = 512;
|
||||
|
||||
async fn read_dir_with_walk_stall(path: PathBuf, stall: Option<Duration>) -> Result<std::fs::ReadDir> {
|
||||
@@ -11472,9 +11504,17 @@ impl LocalDisk {
|
||||
admission: Option<multipart_staging::MultipartSourceAdmission>,
|
||||
) -> Result<()> {
|
||||
multipart_staging::reject_reserved_anchor(path)?;
|
||||
if volume == RUSTFS_META_MULTIPART_BUCKET {
|
||||
self.multipart_index.record_head(path, &buf).await?;
|
||||
}
|
||||
let fuse_with_admission = admission.is_some() && !crash_inject::is_armed(CrashPoint::MetaWriteAfterTmpBeforeRename, path);
|
||||
let prepared_index = if volume == RUSTFS_META_MULTIPART_BUCKET {
|
||||
if fuse_with_admission {
|
||||
self.multipart_index.prepare_record_head(path, &buf).await
|
||||
} else {
|
||||
self.multipart_index.record_head(path, &buf).await?;
|
||||
None
|
||||
}
|
||||
} else {
|
||||
None
|
||||
};
|
||||
let volume_dir = self.io_get_bucket_path(volume)?;
|
||||
let file_path = self.io_get_object_path(volume, path)?;
|
||||
check_path_length(file_path.to_string_lossy().as_ref())?;
|
||||
@@ -11490,11 +11530,14 @@ impl LocalDisk {
|
||||
{
|
||||
multipart_source_state::invalidate_active_multipart_source_head(admission.source());
|
||||
}
|
||||
if admission.is_some() && !crash_inject::is_armed(CrashPoint::MetaWriteAfterTmpBeforeRename, path) {
|
||||
if fuse_with_admission {
|
||||
let precondition = admission.as_ref().map(|a| a.precondition());
|
||||
let root = self.publication_root.clone();
|
||||
let lease = os::acquire_namespace_mutation_lease_with_owner(&file_path, namespace_owner).await;
|
||||
os::run_blocking_namespace_operation_with_sync(lease, do_sync, move || {
|
||||
if let Some(record) = prepared_index {
|
||||
record.run_std()?;
|
||||
}
|
||||
super::fs::create_dir_all_std(&tmp_volume_dir)?;
|
||||
#[cfg(test)]
|
||||
run_owned_file_before_open(&tmp_file_path);
|
||||
@@ -11746,6 +11789,14 @@ impl LocalDisk {
|
||||
W: AsyncWrite + Unpin + Send + 'a,
|
||||
{
|
||||
Box::pin(async move {
|
||||
if opts.identifier_tree_leaf.is_some() {
|
||||
let relative = current
|
||||
.strip_prefix(&opts.base_dir)
|
||||
.ok_or_else(|| DiskError::other("identifier-tree walk escaped its root"))?;
|
||||
if relative.split('/').filter(|component| !component.is_empty()).count() > 512 {
|
||||
return Err(DiskError::other("identifier-tree walk exceeds its directory depth limit"));
|
||||
}
|
||||
}
|
||||
// The part of forward_to below this directory, taken before `current`
|
||||
// loses its trailing slash.
|
||||
let forward_rest = opts
|
||||
@@ -11778,7 +11829,12 @@ impl LocalDisk {
|
||||
// as a single logical read in API semantics.
|
||||
let read_dir_started = rustfs_io_metrics::get_stage_metrics_enabled().then(std::time::Instant::now);
|
||||
let dir_path_abs = self.io_get_object_path(&opts.bucket, current.trim_start_matches(SLASH_SEPARATOR))?;
|
||||
let read_dir_result = match read_dir_entries_with_walk_stall(&dir_path_abs, -1, stall).await {
|
||||
let read_dir_result = if let Some(leaf) = opts.identifier_tree_leaf.as_deref() {
|
||||
read_identifier_tree_entries(&dir_path_abs, leaf, stall).await
|
||||
} else {
|
||||
read_dir_entries_with_walk_stall(&dir_path_abs, -1, stall).await
|
||||
};
|
||||
let read_dir_result = match read_dir_result {
|
||||
Err(err) if err == Error::FileNotFound && !skip_access_checks(&opts.bucket) => {
|
||||
let volume_dir = self.io_get_bucket_path(&opts.bucket)?;
|
||||
if let Err(access_err) = cached_access(&volume_dir).await {
|
||||
@@ -12075,6 +12131,16 @@ impl LocalDisk {
|
||||
return Ok(true);
|
||||
}
|
||||
|
||||
if let Some(leaf) = opts.identifier_tree_leaf.as_deref()
|
||||
&& entry != leaf
|
||||
{
|
||||
// Identifier directories can be namespaces at any depth,
|
||||
// including components named tables or maintenance. Their
|
||||
// own object metadata is outside this selected leaf scan.
|
||||
schedule_dir(&mut dir_stack, format!("{name}/"), true, None, true, false);
|
||||
continue;
|
||||
}
|
||||
|
||||
let mut meta = MetaCacheEntry {
|
||||
name,
|
||||
..Default::default()
|
||||
@@ -12124,7 +12190,8 @@ impl LocalDisk {
|
||||
} else {
|
||||
adaptive_batch_cap
|
||||
};
|
||||
let batch_names = if batch_limit > 1
|
||||
let batch_names = if opts.identifier_tree_leaf.is_none()
|
||||
&& batch_limit > 1
|
||||
&& dir_objes.is_empty()
|
||||
&& dir_stack.iter().all(|(_, _, _, scan_required, _)| !*scan_required)
|
||||
{
|
||||
@@ -12177,6 +12244,11 @@ impl LocalDisk {
|
||||
write_metacache_obj(out, &meta).await?;
|
||||
}
|
||||
|
||||
if opts.identifier_tree_leaf.is_some() {
|
||||
// A dotted leaf cannot contain identifier-tree descendants.
|
||||
continue;
|
||||
}
|
||||
|
||||
let has_namespace_child_candidate = read.has_namespace_child_candidate;
|
||||
let dir_to_skip: HashSet<String> = if has_namespace_child_candidate || !dir_objes.is_empty() {
|
||||
read.data_dirs.into_iter().map(|dir| dir.to_string()).collect()
|
||||
@@ -12223,6 +12295,9 @@ impl LocalDisk {
|
||||
}
|
||||
Err(err) => {
|
||||
if err == Error::FileNotFound || err == Error::IsNotRegular {
|
||||
if opts.identifier_tree_leaf.is_some() {
|
||||
continue;
|
||||
}
|
||||
adaptive_batch_cap = 1;
|
||||
// NOT an object, append to stack (with slash)
|
||||
// If dirObject, but no metadata (which is unexpected) we skip it.
|
||||
@@ -14363,6 +14438,7 @@ impl DiskAPI for LocalDisk {
|
||||
if bucket == RUSTFS_META_MULTIPART_BUCKET && !paths.is_empty() {
|
||||
let mut cached_parts = Vec::with_capacity(paths.len());
|
||||
let mut all_cached = true;
|
||||
let mut verified_source: Option<(&str, PathBuf)> = None;
|
||||
for path_str in paths {
|
||||
let Some(num) = Path::new(path_str)
|
||||
.file_name()
|
||||
@@ -14374,12 +14450,21 @@ impl DiskAPI for LocalDisk {
|
||||
all_cached = false;
|
||||
break;
|
||||
};
|
||||
let Ok(source) = self.io_get_object_open_path(bucket, multipart_staging::upload_scope_root(path_str, true))
|
||||
else {
|
||||
all_cached = false;
|
||||
break;
|
||||
let scope_root = multipart_staging::upload_scope_root(path_str, true);
|
||||
let (source, head_already_verified) = match verified_source.as_ref() {
|
||||
Some((prev_root, prev_source)) if *prev_root == scope_root => (prev_source, true),
|
||||
_ => {
|
||||
let Ok(source) = self.io_get_object_open_path(bucket, scope_root) else {
|
||||
all_cached = false;
|
||||
break;
|
||||
};
|
||||
verified_source = Some((scope_root, source));
|
||||
(&verified_source.as_ref().expect("just initialized").1, false)
|
||||
}
|
||||
};
|
||||
let Some(part) = multipart_source_state::get_active_multipart_source_part(&source, num) else {
|
||||
let Some(part) =
|
||||
multipart_source_state::get_active_multipart_source_part_with_head(source, num, head_already_verified)
|
||||
else {
|
||||
all_cached = false;
|
||||
break;
|
||||
};
|
||||
@@ -14563,6 +14648,7 @@ impl DiskAPI for LocalDisk {
|
||||
Ok(_) => return Ok(Err(DiskError::FileAccessDenied)),
|
||||
Err(err) => return Ok(Err(to_file_error(err).into())),
|
||||
}
|
||||
let mut created_transaction_parent = false;
|
||||
if let Some((integrity, source, destination, expected_len)) = integrity_prep {
|
||||
let stat = match std::fs::symlink_metadata(&source) {
|
||||
Ok(stat) => stat,
|
||||
@@ -14571,6 +14657,7 @@ impl DiskAPI for LocalDisk {
|
||||
if !stat.is_file() || stat.len() != expected_len {
|
||||
return Ok(Err(DiskError::FileCorrupt));
|
||||
}
|
||||
let presynced_proof = take_presynced_tmp_file(&source);
|
||||
let verify_and_sync = (|| -> std::io::Result<bool> {
|
||||
let mut file = os::open_file_for_sync_std(&source)?;
|
||||
let mut header = [0u8; rustfs_filemeta::shard_integrity::INDEX_HEADER_SIZE];
|
||||
@@ -14578,7 +14665,7 @@ impl DiskAPI for LocalDisk {
|
||||
if header != integrity.index_header() {
|
||||
return Ok(false);
|
||||
}
|
||||
if durability.syncs_data_shards() && !take_presynced_tmp_file(&source) {
|
||||
if durability.syncs_data_shards() && !presynced_proof {
|
||||
file.sync_all()?;
|
||||
}
|
||||
Ok(true)
|
||||
@@ -14588,7 +14675,13 @@ impl DiskAPI for LocalDisk {
|
||||
Ok(false) => return Ok(Err(DiskError::FileCorrupt)),
|
||||
Err(err) => return Ok(Err(to_file_error(err).into())),
|
||||
}
|
||||
let precondition = admission.as_ref().map(|check| check.precondition());
|
||||
#[cfg(all(any(test, feature = "test-util"), not(windows)))]
|
||||
let has_hooks = os::prepared_publication_test_hooks::has_any_hooks();
|
||||
#[cfg(not(all(any(test, feature = "test-util"), not(windows))))]
|
||||
let has_hooks = false;
|
||||
let precondition = has_hooks
|
||||
.then(|| admission.as_ref().map(|check| check.precondition()))
|
||||
.flatten();
|
||||
// Publish the immutable generation before preparing the part switch.
|
||||
// Rollback retains the old generation; an unreferenced new index is
|
||||
// reclaimed with the upload directory if preparation is interrupted.
|
||||
@@ -14605,12 +14698,13 @@ impl DiskAPI for LocalDisk {
|
||||
) {
|
||||
return Ok(Err(to_file_error(err).into()));
|
||||
}
|
||||
created_transaction_parent = same_parent;
|
||||
}
|
||||
|
||||
let Some(transaction_parent) = transaction_path.parent() else {
|
||||
return Ok(Err(DiskError::InvalidPath));
|
||||
};
|
||||
if let Err(err) = std::fs::create_dir_all(transaction_parent) {
|
||||
if !created_transaction_parent && let Err(err) = std::fs::create_dir_all(transaction_parent) {
|
||||
return Ok(Err(to_file_error(err).into()));
|
||||
}
|
||||
let staging_path = transaction_parent.join(format!(".part-txn-{}", Uuid::new_v4()));
|
||||
@@ -14618,17 +14712,35 @@ impl DiskAPI for LocalDisk {
|
||||
return Ok(Err(to_file_error(err).into()));
|
||||
}
|
||||
|
||||
let prepare_result = (|| -> std::io::Result<bool> {
|
||||
let prepare_result = (|| -> std::io::Result<(bool, bool)> {
|
||||
let old_data_absent = staging_path.join(PART_TRANSACTION_OLD_DATA_ABSENT);
|
||||
let old_meta_absent = staging_path.join(PART_TRANSACTION_OLD_META_ABSENT);
|
||||
let had_old_data = snapshot_part_transaction_file(
|
||||
&dst_file_path,
|
||||
&staging_path.join(PART_TRANSACTION_OLD_DATA),
|
||||
&staging_path.join(PART_TRANSACTION_OLD_DATA_ABSENT),
|
||||
)?;
|
||||
let had_old_meta = snapshot_part_transaction_file(
|
||||
&dst_meta_path,
|
||||
&staging_path.join(PART_TRANSACTION_OLD_META),
|
||||
&staging_path.join(PART_TRANSACTION_OLD_META_ABSENT),
|
||||
&old_data_absent,
|
||||
)?;
|
||||
let had_old_meta = match std::fs::symlink_metadata(&dst_meta_path) {
|
||||
Ok(metadata) if metadata.is_file() => {
|
||||
std::fs::hard_link(&dst_meta_path, staging_path.join(PART_TRANSACTION_OLD_META))?;
|
||||
true
|
||||
}
|
||||
Ok(metadata) if metadata.file_type().is_symlink() => {
|
||||
return Err(std::io::Error::new(ErrorKind::InvalidInput, "multipart transaction source is a symlink"));
|
||||
}
|
||||
Ok(_) => {
|
||||
return Err(std::io::Error::new(ErrorKind::InvalidData, "multipart transaction source is not a file"));
|
||||
}
|
||||
Err(err) if err.kind() == ErrorKind::NotFound => {
|
||||
if !had_old_data && std::fs::hard_link(&old_data_absent, &old_meta_absent).is_ok() {
|
||||
false
|
||||
} else {
|
||||
std::fs::write(&old_meta_absent, [])?;
|
||||
false
|
||||
}
|
||||
}
|
||||
Err(err) => return Err(err),
|
||||
};
|
||||
|
||||
let new_meta_path = staging_path.join(PART_TRANSACTION_NEW_META);
|
||||
let mut new_meta = std::fs::OpenOptions::new()
|
||||
@@ -14651,11 +14763,11 @@ impl DiskAPI for LocalDisk {
|
||||
if sync_overwrite_backup {
|
||||
os::fsync_dir_std(transaction_parent)?;
|
||||
}
|
||||
Ok(had_old_meta)
|
||||
Ok((had_old_data, had_old_meta))
|
||||
})();
|
||||
|
||||
let had_old_meta = match prepare_result {
|
||||
Ok(had_old_meta) => had_old_meta,
|
||||
let (had_old_data, had_old_meta) = match prepare_result {
|
||||
Ok(flags) => flags,
|
||||
Err(err) => {
|
||||
let _ = remove_dir_all_if_exists(&staging_path);
|
||||
return Ok(Err(to_file_error(err).into()));
|
||||
@@ -14671,6 +14783,7 @@ impl DiskAPI for LocalDisk {
|
||||
dst_meta_path,
|
||||
transaction_path: transaction_path.clone(),
|
||||
meta,
|
||||
had_old_data,
|
||||
had_old_meta,
|
||||
},
|
||||
);
|
||||
@@ -14697,23 +14810,25 @@ impl DiskAPI for LocalDisk {
|
||||
.io_get_object_open_path(volume, &txn_rel_path)
|
||||
.ok()
|
||||
.and_then(|txn_open| take_published_part_txn_receipt(&txn_open));
|
||||
let (current_data_path, current_meta_path, transaction_path, skip_obsolete_read_on_commit) = match cached_published {
|
||||
Some(entry) => (
|
||||
entry.current_data_path,
|
||||
entry.current_meta_path,
|
||||
entry.transaction_path,
|
||||
!entry.had_old_meta,
|
||||
),
|
||||
None => {
|
||||
let (_, current_data_path) = self.io_get_bucket_and_object_path(volume, path)?;
|
||||
let current_meta_path = self.io_get_object_path(volume, &format!("{path}.meta"))?;
|
||||
let transaction_path = self.io_get_object_path(volume, &txn_rel_path)?;
|
||||
for candidate in [¤t_data_path, ¤t_meta_path, &transaction_path] {
|
||||
check_path_length(candidate.to_string_lossy().as_ref())?;
|
||||
let (current_data_path, current_meta_path, transaction_path, skip_obsolete_read_on_commit, cached_old_flags) =
|
||||
match cached_published {
|
||||
Some(entry) => (
|
||||
entry.current_data_path,
|
||||
entry.current_meta_path,
|
||||
entry.transaction_path,
|
||||
!entry.had_old_meta,
|
||||
Some((entry.had_old_data, entry.had_old_meta)),
|
||||
),
|
||||
None => {
|
||||
let (_, current_data_path) = self.io_get_bucket_and_object_path(volume, path)?;
|
||||
let current_meta_path = self.io_get_object_path(volume, &format!("{path}.meta"))?;
|
||||
let transaction_path = self.io_get_object_path(volume, &txn_rel_path)?;
|
||||
for candidate in [¤t_data_path, ¤t_meta_path, &transaction_path] {
|
||||
check_path_length(candidate.to_string_lossy().as_ref())?;
|
||||
}
|
||||
(current_data_path, current_meta_path, transaction_path, false, None)
|
||||
}
|
||||
(current_data_path, current_meta_path, transaction_path, false)
|
||||
}
|
||||
};
|
||||
};
|
||||
let durability = effective_durability(volume);
|
||||
let sync = durability.syncs_commit_metadata();
|
||||
let part_number = current_data_path
|
||||
@@ -14812,7 +14927,26 @@ impl DiskAPI for LocalDisk {
|
||||
{
|
||||
return Ok(Err(to_file_error(err).into()));
|
||||
}
|
||||
if let Err(err) = remove_dir_all_if_exists(&cleanup_path) {
|
||||
if let Some((had_old_data, had_old_meta)) = cached_old_flags.filter(|_| action == PartTransactionAction::Commit) {
|
||||
let old_data_name = if had_old_data {
|
||||
PART_TRANSACTION_OLD_DATA
|
||||
} else {
|
||||
PART_TRANSACTION_OLD_DATA_ABSENT
|
||||
};
|
||||
let old_meta_name = if had_old_meta {
|
||||
PART_TRANSACTION_OLD_META
|
||||
} else {
|
||||
PART_TRANSACTION_OLD_META_ABSENT
|
||||
};
|
||||
for name in [PART_TRANSACTION_NEW_META, old_data_name, old_meta_name] {
|
||||
let _ = std::fs::remove_file(cleanup_path.join(name));
|
||||
}
|
||||
if std::fs::remove_dir(&cleanup_path).is_err()
|
||||
&& let Err(err) = remove_dir_all_if_exists(&cleanup_path)
|
||||
{
|
||||
return Ok(Err(to_file_error(err).into()));
|
||||
}
|
||||
} else if let Err(err) = remove_dir_all_if_exists(&cleanup_path) {
|
||||
return Ok(Err(to_file_error(err).into()));
|
||||
}
|
||||
Ok(Ok(true))
|
||||
@@ -14884,7 +15018,7 @@ impl DiskAPI for LocalDisk {
|
||||
dst_file_path,
|
||||
dst_meta_path,
|
||||
transaction_path,
|
||||
preverified_meta_and_old_flag,
|
||||
preverified_meta_and_old_flags,
|
||||
) = match cached_prepared {
|
||||
Some(entry) => (
|
||||
entry.src_volume_dir,
|
||||
@@ -14893,7 +15027,7 @@ impl DiskAPI for LocalDisk {
|
||||
entry.dst_file_path,
|
||||
Some(entry.dst_meta_path),
|
||||
Some(entry.transaction_path),
|
||||
Some(entry.had_old_meta),
|
||||
Some((entry.had_old_data, entry.had_old_meta)),
|
||||
),
|
||||
None => {
|
||||
let (src_volume_dir, src_file_path) = self.io_get_bucket_and_object_path(src_volume, src_path)?;
|
||||
@@ -14990,7 +15124,7 @@ impl DiskAPI for LocalDisk {
|
||||
let transaction_meta_path = transaction_path.join(PART_TRANSACTION_NEW_META);
|
||||
let publish_meta_path = transaction_path.join(PART_TRANSACTION_PUBLISH_META);
|
||||
if !cfg!(test)
|
||||
&& preverified_meta_and_old_flag.is_some()
|
||||
&& preverified_meta_and_old_flags.is_some()
|
||||
&& publish_meta_path.is_file()
|
||||
&& std::fs::metadata(&transaction_meta_path).is_ok_and(|m| m.is_file() && m.len() == meta.len() as u64)
|
||||
{
|
||||
@@ -15034,7 +15168,13 @@ impl DiskAPI for LocalDisk {
|
||||
os::sync_file(&src_file_path).map_err(to_file_error)?;
|
||||
}
|
||||
|
||||
let precondition = admission.as_ref().map(|check| check.precondition());
|
||||
#[cfg(all(any(test, feature = "test-util"), not(windows)))]
|
||||
let has_hooks = os::prepared_publication_test_hooks::has_any_hooks();
|
||||
#[cfg(not(all(any(test, feature = "test-util"), not(windows))))]
|
||||
let has_hooks = false;
|
||||
let precondition = has_hooks
|
||||
.then(|| admission.as_ref().map(|check| check.precondition()))
|
||||
.flatten();
|
||||
|
||||
os::rename_all_with_precondition_and_parent_sync_std(
|
||||
&src_file_path,
|
||||
@@ -15046,17 +15186,19 @@ impl DiskAPI for LocalDisk {
|
||||
)
|
||||
.map_err(to_file_error)?;
|
||||
|
||||
let dst_meta = lstat_std(&dst_file_path).map_err(to_file_error)?;
|
||||
if src_is_dir != dst_meta.is_dir() {
|
||||
warn!(
|
||||
event = EVENT_DISK_LOCAL_RENAME_REJECTED,
|
||||
component = LOG_COMPONENT_ECSTORE,
|
||||
subsystem = LOG_SUBSYSTEM_DISK_LOCAL,
|
||||
reason = "dst_type_changed_after_rename",
|
||||
path = ?dst_file_path,
|
||||
"Disk local rename rejected"
|
||||
);
|
||||
return Err(DiskError::FileAccessDenied.into());
|
||||
if cfg!(test) || src_is_dir {
|
||||
let dst_meta = lstat_std(&dst_file_path).map_err(to_file_error)?;
|
||||
if src_is_dir != dst_meta.is_dir() {
|
||||
warn!(
|
||||
event = EVENT_DISK_LOCAL_RENAME_REJECTED,
|
||||
component = LOG_COMPONENT_ECSTORE,
|
||||
subsystem = LOG_SUBSYSTEM_DISK_LOCAL,
|
||||
reason = "dst_type_changed_after_rename",
|
||||
path = ?dst_file_path,
|
||||
"Disk local rename rejected"
|
||||
);
|
||||
return Err(DiskError::FileAccessDenied.into());
|
||||
}
|
||||
}
|
||||
|
||||
if let Some(transaction_publish_meta) = transaction_publish_meta {
|
||||
@@ -15087,8 +15229,8 @@ impl DiskAPI for LocalDisk {
|
||||
}
|
||||
}
|
||||
}
|
||||
if let (Some(had_old_meta), Some(txn_path), Some(meta_path)) =
|
||||
(preverified_meta_and_old_flag, transaction_path.as_ref(), dst_meta_path.as_path().into())
|
||||
if let (Some((had_old_data, had_old_meta)), Some(txn_path), Some(meta_path)) =
|
||||
(preverified_meta_and_old_flags, transaction_path.as_ref(), dst_meta_path.as_path().into())
|
||||
{
|
||||
put_published_part_txn_receipt(
|
||||
txn_path,
|
||||
@@ -15096,6 +15238,7 @@ impl DiskAPI for LocalDisk {
|
||||
current_data_path: dst_file_path,
|
||||
current_meta_path: meta_path.to_path_buf(),
|
||||
transaction_path: txn_path.clone(),
|
||||
had_old_data,
|
||||
had_old_meta,
|
||||
},
|
||||
);
|
||||
@@ -15431,6 +15574,12 @@ impl DiskAPI for LocalDisk {
|
||||
super::lifecycle_test_io::observe(&opts.bucket, None);
|
||||
self.wait_for_startup_cleanup().await;
|
||||
|
||||
if let Some(leaf) = opts.identifier_tree_leaf.as_deref()
|
||||
&& (!opts.recursive || !opts.base_dir.ends_with('/') || !super::is_identifier_tree_leaf(leaf))
|
||||
{
|
||||
return Err(DiskError::other("invalid identifier-tree walk options"));
|
||||
}
|
||||
|
||||
// Callers that do not pin a stall budget inherit the configured default, so
|
||||
// every local walk keeps a liveness bound even when the wrapper-level total
|
||||
// timeout is skipped.
|
||||
@@ -15457,7 +15606,7 @@ impl DiskAPI for LocalDisk {
|
||||
|
||||
let mut skip_current_dir_object = false;
|
||||
let mut multipart_dir_to_skip: HashSet<String> = HashSet::new();
|
||||
if opts.base_dir.ends_with(SLASH_SEPARATOR) {
|
||||
if opts.identifier_tree_leaf.is_none() && opts.base_dir.ends_with(SLASH_SEPARATOR) {
|
||||
if let Ok(data) = with_walk_stall_timeout(
|
||||
stall,
|
||||
self.read_listing_metadata(
|
||||
@@ -19407,6 +19556,55 @@ mod test {
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn identifier_tree_walk_filters_report_batches_and_bounds_depth() {
|
||||
let dir = tempfile::tempdir().expect("create disk fixture");
|
||||
let root = dir.path().join("catalog/prefix");
|
||||
fs::create_dir_all(root.join("sales"))
|
||||
.await
|
||||
.expect("create identifier directory");
|
||||
fs::create_dir_all(root.join("table-entry.json"))
|
||||
.await
|
||||
.expect("create selected leaf");
|
||||
for index in 0..READ_DIR_BATCH_SIZE * 3 + 1 {
|
||||
fs::create_dir(root.join(format!("report-{index}.json")))
|
||||
.await
|
||||
.expect("create report directory");
|
||||
}
|
||||
let mut names = read_identifier_tree_entries(&root, "table-entry.json", Some(Duration::from_secs(5)))
|
||||
.await
|
||||
.expect("all directory batches should be filtered");
|
||||
names.sort();
|
||||
assert_eq!(names, ["sales/", "table-entry.json/"]);
|
||||
|
||||
let endpoint = Endpoint::try_from(dir.path().to_str().expect("UTF-8 fixture path")).expect("valid endpoint");
|
||||
let disk = LocalDisk::new(&endpoint, false).await.expect("create local disk");
|
||||
let options = WalkDirOptions {
|
||||
bucket: "catalog".to_string(),
|
||||
base_dir: "prefix/".to_string(),
|
||||
recursive: true,
|
||||
identifier_tree_leaf: Some("table-entry.json".to_string()),
|
||||
..Default::default()
|
||||
};
|
||||
let mut output = Vec::new();
|
||||
let mut writer = MetacacheWriter::new(&mut output);
|
||||
let mut count = 0;
|
||||
let error = disk
|
||||
.scan_dir(
|
||||
format!("prefix/{}", "a/".repeat(513)),
|
||||
String::new(),
|
||||
&options,
|
||||
&mut writer,
|
||||
&mut count,
|
||||
true,
|
||||
None,
|
||||
)
|
||||
.await
|
||||
.expect_err("excessive depth must fail before filesystem I/O");
|
||||
assert!(error.to_string().contains("directory depth limit"));
|
||||
assert_eq!(count, 0);
|
||||
}
|
||||
|
||||
// #4644: the stall budget bounds a single drive read, never the walk as a
|
||||
// whole. A walk that keeps making progress must survive even when the total
|
||||
// time spent blocked on a slow consumer dwarfs the stall timeout.
|
||||
|
||||
@@ -1461,7 +1461,12 @@ impl LocalDisk {
|
||||
}
|
||||
}
|
||||
}
|
||||
match publication_owned.finish_on_thread(sync_commit_metadata, !cfg!(test), staging_cleanup) {
|
||||
match publication_owned.finish_on_thread(
|
||||
sync_commit_metadata,
|
||||
!cfg!(test),
|
||||
staging_cleanup,
|
||||
Some(&directory_guard_owned),
|
||||
) {
|
||||
Ok(result) => Ok(MultipartFusedCommitOutcome::Committed(Box::new(RenameDataResp {
|
||||
multipart_source: Some(Box::new(result)),
|
||||
old_data_dir: has_old_data_dir,
|
||||
|
||||
@@ -1829,7 +1829,9 @@ impl LocalDisk {
|
||||
) -> Result<DeleteRecoveryInspection> {
|
||||
self.reject_multipart_delete_recovery_mutation(volume, object)?;
|
||||
if let Ok(root) = self.io_get_object_path(volume, object) {
|
||||
let _ = take_prepared_delete_receipt(&root.join(transaction.to_string()));
|
||||
let tx_dir = root.join(transaction.to_string());
|
||||
let _ = take_prepared_delete_receipt(&tx_dir);
|
||||
let _ = take_published_delete_receipt(&tx_dir);
|
||||
}
|
||||
|
||||
let (mut observed, mut has_data_dir_work) = self
|
||||
@@ -2493,8 +2495,8 @@ impl LocalDisk {
|
||||
Err(err) if err.kind() == ErrorKind::NotFound => return Ok(Ok(None)),
|
||||
Err(err) => return Err(err),
|
||||
};
|
||||
type PreparedDeleteTuple = (Option<Uuid>, Option<Vec<u8>>, Vec<u8>, DeleteRecoveryJournal, bool);
|
||||
let (old_dir, target, journal_raw, journal, sole_transaction) =
|
||||
type PreparedDeleteTuple = (Option<Uuid>, Option<Vec<u8>>, Vec<u8>, DeleteRecoveryJournal, bool, bool, bool);
|
||||
let (old_dir, target, journal_raw, journal, sole_transaction, destination_missing, old_data_dir_present) =
|
||||
match (|| -> Result<Option<PreparedDeleteTuple>> {
|
||||
let mut meta = FileMeta::load(&buf)?;
|
||||
let mut deleted_headers: Vec<_> = meta
|
||||
@@ -2529,7 +2531,11 @@ impl LocalDisk {
|
||||
let target = (!meta.versions.is_empty()).then(|| meta.marshal_msg()).transpose()?;
|
||||
let mut predecessors = BTreeSet::new();
|
||||
let mut sole_transaction = true;
|
||||
let mut destination_exists = false;
|
||||
let mut old_data_dir_exists = false;
|
||||
let mut scanned_file_path = false;
|
||||
if let Ok(entries) = std::fs::read_dir(&file_path) {
|
||||
scanned_file_path = true;
|
||||
for (visited, entry) in
|
||||
recovery_entries(entries, recovery_scope.as_ref().map(|scope| scope.budget.as_ref())).enumerate()
|
||||
{
|
||||
@@ -2542,13 +2548,19 @@ impl LocalDisk {
|
||||
sole_transaction = false;
|
||||
continue;
|
||||
};
|
||||
let Some(id) = Uuid::parse_str(name_str).ok().filter(|id| !id.is_nil() && *id != transaction)
|
||||
else {
|
||||
let Some(id) = Uuid::parse_str(name_str).ok().filter(|id| !id.is_nil()) else {
|
||||
continue;
|
||||
};
|
||||
if id == transaction {
|
||||
destination_exists = true;
|
||||
continue;
|
||||
}
|
||||
if !entry.file_type().map_err(to_file_error)?.is_dir() {
|
||||
return Err(DiskError::InvalidPath);
|
||||
}
|
||||
if old_dir == Some(id) && name_str == id.hyphenated().to_string() {
|
||||
old_data_dir_exists = true;
|
||||
}
|
||||
let dir_path = entry.path();
|
||||
let prepared = regular_bytes(&dir_path.join(PREPARED))?;
|
||||
let receipt = regular_bytes(&dir_path.join(DELETE_QUORUM_COMMITTED_FILE))?;
|
||||
@@ -2607,7 +2619,15 @@ impl LocalDisk {
|
||||
predecessors: predecessors.into_iter().collect(),
|
||||
};
|
||||
let raw = journal_bytes(&journal)?;
|
||||
Ok(Some((old_dir, target, raw, journal, sole_transaction)))
|
||||
Ok(Some((
|
||||
old_dir,
|
||||
target,
|
||||
raw,
|
||||
journal,
|
||||
sole_transaction,
|
||||
scanned_file_path && !destination_exists,
|
||||
scanned_file_path && old_data_dir_exists,
|
||||
)))
|
||||
})() {
|
||||
Ok(Some(tuple)) => tuple,
|
||||
Ok(None) => return Ok(Ok(None)),
|
||||
@@ -2641,23 +2661,27 @@ impl LocalDisk {
|
||||
if sync {
|
||||
os::fsync_dir_std(&staged_dir)?;
|
||||
}
|
||||
match std::fs::symlink_metadata(&destination) {
|
||||
Ok(metadata) if metadata.is_dir() => match std::fs::remove_dir(&destination) {
|
||||
Ok(()) => {}
|
||||
if !destination_missing {
|
||||
match std::fs::symlink_metadata(&destination) {
|
||||
Ok(metadata) if metadata.is_dir() => match std::fs::remove_dir(&destination) {
|
||||
Ok(()) => {}
|
||||
Err(err) if err.kind() == ErrorKind::NotFound => {}
|
||||
Err(err) if is_dir_not_empty_error(&err) => return Ok(Err(DiskError::OutdatedXLMeta)),
|
||||
Err(err) => return Err(err),
|
||||
},
|
||||
Ok(_) => return Ok(Err(DiskError::OutdatedXLMeta)),
|
||||
Err(err) if err.kind() == ErrorKind::NotFound => {}
|
||||
Err(err) if is_dir_not_empty_error(&err) => return Ok(Err(DiskError::OutdatedXLMeta)),
|
||||
Err(err) => return Err(err),
|
||||
},
|
||||
Ok(_) => return Ok(Err(DiskError::OutdatedXLMeta)),
|
||||
Err(err) if err.kind() == ErrorKind::NotFound => {}
|
||||
Err(err) => return Err(err),
|
||||
}
|
||||
}
|
||||
let staged_dir = staged.keep();
|
||||
if let Err(err) = std::fs::rename(&staged_dir, &destination) {
|
||||
let _ = std::fs::remove_dir_all(&staged_dir);
|
||||
return Err(err);
|
||||
}
|
||||
let old_data_path = old_dir.map(|uuid| file_path.join(uuid.to_string())).filter(|p| p.is_dir());
|
||||
let old_data_path = old_dir
|
||||
.map(|uuid| file_path.join(uuid.to_string()))
|
||||
.filter(|p| old_data_dir_present || p.is_dir());
|
||||
if sync && let Err(err) = os::fsync_dir_std(&file_path) {
|
||||
let _ = std::fs::remove_dir_all(&destination);
|
||||
return Err(err);
|
||||
@@ -2686,8 +2710,15 @@ impl LocalDisk {
|
||||
// A provisional head must stay outside the published recovery transaction.
|
||||
let target_in_rollback = destination.join(TARGET);
|
||||
let linked_xl = staging_root.join(format!("delete-xl-{transaction}"));
|
||||
let _ = std::fs::remove_file(&linked_xl);
|
||||
let write_res = if std::fs::hard_link(&target_in_rollback, &linked_xl).is_ok() {
|
||||
let linked = match std::fs::hard_link(&target_in_rollback, &linked_xl) {
|
||||
Ok(()) => true,
|
||||
Err(err) if err.kind() == ErrorKind::AlreadyExists => {
|
||||
let _ = std::fs::remove_file(&linked_xl);
|
||||
std::fs::hard_link(&target_in_rollback, &linked_xl).is_ok()
|
||||
}
|
||||
Err(_) => false,
|
||||
};
|
||||
let write_res = if linked {
|
||||
let rename_res = std::fs::rename(&linked_xl, &xl_path);
|
||||
if rename_res.is_err() {
|
||||
let _ = std::fs::remove_file(&linked_xl);
|
||||
@@ -3043,14 +3074,19 @@ impl LocalDisk {
|
||||
}
|
||||
|
||||
let data_dirs = journal.data_dirs.clone();
|
||||
let has_original = journal.original_digest.is_some();
|
||||
let has_target = journal.target_digest.is_some();
|
||||
let journal_for_receipt = cached_journal_raw.is_none().then(|| journal.clone());
|
||||
let lease = os::acquire_namespace_mutation_lease_with_owner(&receipt_path, owner).await;
|
||||
let prune_paths: Vec<_> = rollback
|
||||
.ancestors()
|
||||
.take_while(|dir| *dir != bucket_path.as_path() && dir.starts_with(&bucket_path))
|
||||
.map(Path::to_path_buf)
|
||||
.collect();
|
||||
let prune_paths: Vec<_> = if has_target {
|
||||
vec![rollback.clone()]
|
||||
} else {
|
||||
rollback
|
||||
.ancestors()
|
||||
.take_while(|dir| *dir != bucket_path.as_path() && dir.starts_with(&bucket_path))
|
||||
.map(Path::to_path_buf)
|
||||
.collect()
|
||||
};
|
||||
|
||||
let res = os::run_blocking_namespace_operation_with_sync(lease, sync, move || -> std::io::Result<()> {
|
||||
let persist_receipt = |finished: bool| -> std::io::Result<()> {
|
||||
@@ -3131,23 +3167,53 @@ impl LocalDisk {
|
||||
if had_physical_data_dirs && sync && root.is_dir() {
|
||||
os::fsync_dir_std(&root)?;
|
||||
}
|
||||
for name in [STORAGE_FORMAT_FILE_BACKUP, DELETE_MARKER_ROLLBACK_FILE] {
|
||||
if let Err(err) = ignore_not_found(std::fs::remove_file(rollback.join(name))) {
|
||||
let fast_cleanup = cached_journal_raw.is_some();
|
||||
if fast_cleanup {
|
||||
let backup_or_marker = if has_original {
|
||||
Some(STORAGE_FORMAT_FILE_BACKUP)
|
||||
} else if has_target {
|
||||
Some(DELETE_MARKER_ROLLBACK_FILE)
|
||||
} else {
|
||||
None
|
||||
};
|
||||
if let Some(name) = backup_or_marker
|
||||
&& let Err(err) = ignore_not_found(std::fs::remove_file(rollback.join(name)))
|
||||
{
|
||||
if has_target || had_physical_data_dirs {
|
||||
let _ = persist_receipt(true);
|
||||
}
|
||||
return Err(err);
|
||||
}
|
||||
}
|
||||
for name in [PREPARED, TARGET, FINISHED] {
|
||||
let path = rollback.join(name);
|
||||
#[cfg(all(any(test, feature = "test-util"), not(windows)))]
|
||||
os::prepared_publication_test_hooks::run(os::prepared_publication_test_hooks::Stage::Remove, &path);
|
||||
if let Err(err) = ignore_not_found(std::fs::remove_file(path)) {
|
||||
if has_target || had_physical_data_dirs {
|
||||
let _ = persist_receipt(true);
|
||||
for name in [PREPARED, FINISHED].into_iter().chain(has_target.then_some(TARGET)) {
|
||||
let path = rollback.join(name);
|
||||
#[cfg(all(any(test, feature = "test-util"), not(windows)))]
|
||||
os::prepared_publication_test_hooks::run(os::prepared_publication_test_hooks::Stage::Remove, &path);
|
||||
if let Err(err) = ignore_not_found(std::fs::remove_file(path)) {
|
||||
if has_target || had_physical_data_dirs {
|
||||
let _ = persist_receipt(true);
|
||||
}
|
||||
return Err(err);
|
||||
}
|
||||
}
|
||||
} else {
|
||||
for name in [STORAGE_FORMAT_FILE_BACKUP, DELETE_MARKER_ROLLBACK_FILE] {
|
||||
if let Err(err) = ignore_not_found(std::fs::remove_file(rollback.join(name))) {
|
||||
if has_target || had_physical_data_dirs {
|
||||
let _ = persist_receipt(true);
|
||||
}
|
||||
return Err(err);
|
||||
}
|
||||
}
|
||||
for name in [PREPARED, TARGET, FINISHED] {
|
||||
let path = rollback.join(name);
|
||||
#[cfg(all(any(test, feature = "test-util"), not(windows)))]
|
||||
os::prepared_publication_test_hooks::run(os::prepared_publication_test_hooks::Stage::Remove, &path);
|
||||
if let Err(err) = ignore_not_found(std::fs::remove_file(path)) {
|
||||
if has_target || had_physical_data_dirs {
|
||||
let _ = persist_receipt(true);
|
||||
}
|
||||
return Err(err);
|
||||
}
|
||||
return Err(err);
|
||||
}
|
||||
}
|
||||
if sync && let Err(err) = os::fsync_dir_std(&rollback) {
|
||||
@@ -7240,6 +7306,9 @@ mod tests {
|
||||
#[cfg(unix)]
|
||||
#[tokio::test]
|
||||
async fn delete_recovery_fused_prepare_checkpoint_reopens_before_head_publication() {
|
||||
use std::os::unix::fs::MetadataExt;
|
||||
|
||||
let _mode = durability_mode_override::set(DurabilityMode::Strict);
|
||||
let (_temp, disk, mut first, _) = fixture().await;
|
||||
first.version_id = Some(Uuid::new_v4());
|
||||
let mut second = first.clone();
|
||||
@@ -7263,12 +7332,25 @@ mod tests {
|
||||
let source = source_root.join(transaction.to_string());
|
||||
let destination = copy_root.join(transaction.to_string());
|
||||
std::fs::create_dir(&destination).expect("create checkpoint transaction");
|
||||
std::fs::copy(source_root.join(STORAGE_FORMAT_FILE), copy_root.join(STORAGE_FORMAT_FILE))
|
||||
.expect("capture head before publication");
|
||||
let source_head = source_root.join(STORAGE_FORMAT_FILE);
|
||||
let source_head_meta = std::fs::metadata(&source_head).expect("stat source head");
|
||||
let copy_head = copy_root.join(STORAGE_FORMAT_FILE);
|
||||
std::fs::copy(&source_head, ©_head).expect("capture head before publication");
|
||||
for entry in std::fs::read_dir(source).expect("read published preparation") {
|
||||
let entry = entry.expect("prepared file");
|
||||
assert!(entry.file_type().expect("prepared file type").is_file());
|
||||
std::fs::copy(entry.path(), destination.join(entry.file_name())).expect("capture prepared file");
|
||||
let dst = destination.join(entry.file_name());
|
||||
if entry.file_name() == STORAGE_FORMAT_FILE_BACKUP {
|
||||
let backup_meta = entry.metadata().expect("stat prepared backup");
|
||||
assert_eq!(
|
||||
(backup_meta.dev(), backup_meta.ino()),
|
||||
(source_head_meta.dev(), source_head_meta.ino()),
|
||||
"fused preparation must hardlink backup to live head"
|
||||
);
|
||||
std::fs::hard_link(©_head, &dst).expect("preserve hardlinked backup");
|
||||
} else {
|
||||
std::fs::copy(entry.path(), dst).expect("capture prepared file");
|
||||
}
|
||||
}
|
||||
});
|
||||
let opts = DeleteOptions {
|
||||
@@ -7292,10 +7374,42 @@ mod tests {
|
||||
.await
|
||||
.expect("actual preparation checkpoint must remain recoverable");
|
||||
assert_eq!(inspection.state, DeleteRecoveryState::NewPrepared);
|
||||
reopened
|
||||
.settle_delete_transaction("recovery", "object", transaction, inspection.fingerprint, DeleteRecoveryAction::Rollback)
|
||||
let reopened_tx = reopened
|
||||
.io_get_object_path("recovery", &format!("object/{transaction}"))
|
||||
.expect("reopened transaction path");
|
||||
os::fsync_dir_recorder::set_failure(&reopened_tx, ErrorKind::Other);
|
||||
assert!(
|
||||
reopened
|
||||
.settle_delete_transaction(
|
||||
"recovery",
|
||||
"object",
|
||||
transaction,
|
||||
inspection.fingerprint,
|
||||
DeleteRecoveryAction::Rollback
|
||||
)
|
||||
.await
|
||||
.is_err(),
|
||||
"failed transaction directory sync after backup removal must retain journal"
|
||||
);
|
||||
assert!(reopened_tx.join(PREPARED).is_file());
|
||||
assert!(!reopened_tx.join(STORAGE_FORMAT_FILE_BACKUP).exists());
|
||||
let retry_inspection = reopened
|
||||
.delete_recovery_inspect("recovery", "object", transaction)
|
||||
.await
|
||||
.expect("inspect interrupted rollback");
|
||||
assert_eq!(retry_inspection.state, DeleteRecoveryState::NewPrepared);
|
||||
let settled = reopened
|
||||
.settle_delete_transaction(
|
||||
"recovery",
|
||||
"object",
|
||||
transaction,
|
||||
retry_inspection.fingerprint,
|
||||
DeleteRecoveryAction::Rollback,
|
||||
)
|
||||
.await
|
||||
.expect("roll back interrupted preparation");
|
||||
assert_eq!(settled.state, DeleteRecoveryState::Absent, "same-inode backup must not survive rollback");
|
||||
assert!(!checkpoint_root.join(transaction.to_string()).exists());
|
||||
assert_eq!(
|
||||
fs::read(checkpoint_root.join(STORAGE_FORMAT_FILE))
|
||||
.await
|
||||
|
||||
@@ -429,7 +429,7 @@ impl IndexOwner {
|
||||
os::replace_multipart_source_metadata(&marker, b"1", &self.base, &root)?;
|
||||
os::sync_multipart_owned_record_parents(&self.base.join(".rustfs.sys"), &directory, true, &self.base, &root)?;
|
||||
}
|
||||
self.ensure_current_root()
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn ensure_current_root(&self) -> io::Result<()> {
|
||||
@@ -547,13 +547,10 @@ impl IndexOwner {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub(super) async fn record_head(self: &Arc<Self>, path: &str, bytes: &[u8]) -> Result<()> {
|
||||
let Some(upload) = path
|
||||
pub(super) async fn prepare_record_head(self: &Arc<Self>, path: &str, bytes: &[u8]) -> Option<PreparedHeadRecord> {
|
||||
let upload = path
|
||||
.strip_suffix("/xl.meta")
|
||||
.filter(|upload| canonical_upload(upload).is_some())
|
||||
else {
|
||||
return Ok(());
|
||||
};
|
||||
.filter(|upload| canonical_upload(upload).is_some())?;
|
||||
let info = match get_file_info(
|
||||
bytes,
|
||||
RUSTFS_META_MULTIPART_BUCKET,
|
||||
@@ -568,10 +565,33 @@ impl IndexOwner {
|
||||
Ok(info) => info,
|
||||
Err(_) => {
|
||||
self.disabled.store(true, Ordering::Release);
|
||||
return Ok(());
|
||||
return None;
|
||||
}
|
||||
};
|
||||
self.record_info(upload, &info).await
|
||||
let marker = info
|
||||
.metadata
|
||||
.get(crate::disk::multipart_index::RUSTFS_MULTIPART_BUCKET_KEY)
|
||||
.map(|bucket| {
|
||||
self.base
|
||||
.join(INDEX)
|
||||
.join(sha(bucket))
|
||||
.join(upload.split('/').next().unwrap_or_default())
|
||||
})
|
||||
.unwrap_or_else(|| self.base.join(INDEX));
|
||||
let lease = os::acquire_namespace_mutation_lease_with_owner(&marker, None).await;
|
||||
Some(PreparedHeadRecord {
|
||||
owner: Arc::clone(self),
|
||||
upload: upload.to_owned(),
|
||||
info,
|
||||
_lease: lease,
|
||||
})
|
||||
}
|
||||
|
||||
pub(super) async fn record_head(self: &Arc<Self>, path: &str, bytes: &[u8]) -> Result<()> {
|
||||
let Some(prepared) = self.prepare_record_head(path, bytes).await else {
|
||||
return Ok(());
|
||||
};
|
||||
prepared.run().await
|
||||
}
|
||||
|
||||
async fn page(self: &Arc<Self>, bucket: &str, marker: Option<&str>, disk_id: Uuid) -> Result<MultipartIndexPage> {
|
||||
@@ -693,6 +713,37 @@ impl IndexOwner {
|
||||
}
|
||||
}
|
||||
|
||||
pub(super) struct PreparedHeadRecord {
|
||||
owner: Arc<IndexOwner>,
|
||||
upload: String,
|
||||
info: FileInfo,
|
||||
_lease: Arc<os::NamespaceMutationLease>,
|
||||
}
|
||||
|
||||
impl PreparedHeadRecord {
|
||||
pub(super) fn run_std(self) -> io::Result<()> {
|
||||
let Self {
|
||||
owner,
|
||||
upload,
|
||||
info,
|
||||
_lease,
|
||||
} = self;
|
||||
match owner.record(&upload, &info) {
|
||||
Err(error) if error.kind() == io::ErrorKind::InvalidData => {
|
||||
owner.disabled.store(true, Ordering::Release);
|
||||
Ok(())
|
||||
}
|
||||
result => result,
|
||||
}
|
||||
}
|
||||
|
||||
async fn run(self) -> Result<()> {
|
||||
let lease = Arc::clone(&self._lease);
|
||||
os::run_blocking_namespace_operation_with_sync(lease, true, move || self.run_std()).await?;
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
impl LocalDisk {
|
||||
#[cfg(any(test, feature = "test-util"))]
|
||||
pub(in crate::disk) async fn wait_for_multipart_index_bootstrap(&self) {
|
||||
|
||||
@@ -428,7 +428,6 @@ impl LocalDisk {
|
||||
let root = store.publication_root.clone();
|
||||
let lease = os::acquire_namespace_mutation_lease_with_owner(&destination, Some(store.lease.clone())).await;
|
||||
os::run_blocking_namespace_operation_with_sync(lease, true, move || {
|
||||
permit.validate()?;
|
||||
os::rename_multipart_source_owned_std(&source, &destination, &base, &root, || permit.validate())
|
||||
})
|
||||
.await?;
|
||||
|
||||
@@ -295,10 +295,17 @@ pub(super) fn put_active_multipart_source_head(source: impl AsRef<Path>, raw: By
|
||||
}
|
||||
|
||||
fn stamp_matches_active_disk(source: &Path, expected_head_stamp: FileStamp) -> bool {
|
||||
FileStamp::from_path(&source.join(STORAGE_FORMAT_FILE)) == Some(expected_head_stamp)
|
||||
&& !source.join(STATE).exists()
|
||||
&& !source.join(crate::disk::MULTIPART_RECOVERY_DIR).exists()
|
||||
&& !source.join(".mpu-completions").exists()
|
||||
if FileStamp::from_path(&source.join(STORAGE_FORMAT_FILE)) != Some(expected_head_stamp) {
|
||||
return false;
|
||||
}
|
||||
#[cfg(test)]
|
||||
if source.join(STATE).exists()
|
||||
|| source.join(crate::disk::MULTIPART_RECOVERY_DIR).exists()
|
||||
|| source.join(".mpu-completions").exists()
|
||||
{
|
||||
return false;
|
||||
}
|
||||
true
|
||||
}
|
||||
|
||||
pub(super) fn get_active_multipart_source_head(source: &Path) -> Option<(Bytes, FileInfo)> {
|
||||
@@ -375,7 +382,11 @@ pub(super) fn finish_active_multipart_part_transaction(
|
||||
ACTIVE_MULTIPART_SOURCE_HEADS.remove_if_mut(source, |entry| !entry.unsettled_parts.remove(&part_number));
|
||||
}
|
||||
|
||||
pub(super) fn get_active_multipart_source_part(source: &Path, part_number: usize) -> Option<rustfs_filemeta::ObjectPartInfo> {
|
||||
pub(super) fn get_active_multipart_source_part_with_head(
|
||||
source: &Path,
|
||||
part_number: usize,
|
||||
head_already_verified: bool,
|
||||
) -> Option<rustfs_filemeta::ObjectPartInfo> {
|
||||
let (head_stamp, data_dir, expected_stamp, part) = ACTIVE_MULTIPART_SOURCE_HEADS
|
||||
.with_get(source, |entry| {
|
||||
if !entry.unsettled_parts.is_empty() {
|
||||
@@ -386,7 +397,7 @@ pub(super) fn get_active_multipart_source_part(source: &Path, part_number: usize
|
||||
Some((entry.head_stamp, data_dir, *expected_stamp, part.clone()))
|
||||
})
|
||||
.flatten()?;
|
||||
if !stamp_matches_active_disk(source, head_stamp) {
|
||||
if !head_already_verified && !stamp_matches_active_disk(source, head_stamp) {
|
||||
ACTIVE_MULTIPART_SOURCE_HEADS.remove(source);
|
||||
return None;
|
||||
}
|
||||
@@ -609,25 +620,45 @@ impl SourceStore {
|
||||
raw: &[u8],
|
||||
terminal: Option<&Path>,
|
||||
sync_source_before: bool,
|
||||
) -> std::io::Result<()> {
|
||||
Self::persist_state_with_guard_std(source, base, root, raw, terminal, sync_source_before, None)
|
||||
}
|
||||
fn persist_state_with_guard_std(
|
||||
source: &Path,
|
||||
base: &Path,
|
||||
root: &os::PublicationRoot,
|
||||
raw: &[u8],
|
||||
terminal: Option<&Path>,
|
||||
sync_source_before: bool,
|
||||
directory_guard: Option<&os::RenameCommitGuard>,
|
||||
) -> std::io::Result<()> {
|
||||
invalidate_active_multipart_source_head(source);
|
||||
#[cfg(unix)]
|
||||
let mut pinned = os::pin_multipart_source_directory(source, base, root)?;
|
||||
#[cfg(not(unix))]
|
||||
#[cfg(not(target_os = "linux"))]
|
||||
let _pinned = os::pin_multipart_source_directory(source, base, root)?;
|
||||
if sync_source_before {
|
||||
#[cfg(unix)]
|
||||
#[cfg(target_os = "linux")]
|
||||
{
|
||||
let mut pinned = os::pin_multipart_source_directory(source, base, root)?;
|
||||
let opened = std::fs::File::from(pinned.pop().ok_or_else(|| IoError::other("empty source directory guard"))?);
|
||||
os::fsync_directory_handle_std(source, Some(&opened))?;
|
||||
}
|
||||
#[cfg(all(unix, not(target_os = "linux")))]
|
||||
{
|
||||
let opened = std::fs::File::from(
|
||||
_pinned
|
||||
.last()
|
||||
.ok_or_else(|| IoError::other("empty source directory guard"))?
|
||||
.try_clone()?,
|
||||
);
|
||||
os::fsync_directory_handle_std(source, Some(&opened))?;
|
||||
}
|
||||
#[cfg(not(unix))]
|
||||
os::fsync_dir_std(source)?;
|
||||
}
|
||||
if let Some(path) = terminal {
|
||||
os::replace_multipart_source_metadata(path, raw, base, root)?;
|
||||
}
|
||||
os::replace_multipart_source_metadata(&source.join(STATE), raw, base, root)
|
||||
os::replace_multipart_source_metadata_with_guard(&source.join(STATE), raw, base, root, directory_guard)
|
||||
}
|
||||
async fn write_owned_backup_and_persist(&self, owner: Uuid, target: bool, bytes: &[u8], state: &SourceState) -> Result<()> {
|
||||
if owner.is_nil() {
|
||||
@@ -724,17 +755,39 @@ impl SourceStore {
|
||||
bytes: &[u8],
|
||||
sync_source_root: bool,
|
||||
) -> std::io::Result<()> {
|
||||
Self::write_owned_backup_inner_std(source, base, root, owner, target, bytes, sync_source_root, false, true, false)
|
||||
}
|
||||
#[expect(clippy::too_many_arguments)]
|
||||
fn write_owned_backup_inner_std(
|
||||
source: &Path,
|
||||
base: &Path,
|
||||
root: &os::PublicationRoot,
|
||||
owner: Uuid,
|
||||
target: bool,
|
||||
bytes: &[u8],
|
||||
sync_source_root: bool,
|
||||
fresh_owner: bool,
|
||||
sync_parents: bool,
|
||||
source_already_pinned: bool,
|
||||
) -> std::io::Result<()> {
|
||||
invalidate_active_multipart_source_head(source);
|
||||
let path = if target {
|
||||
target_backup_path(source, owner)
|
||||
} else {
|
||||
original_path(source, owner)
|
||||
};
|
||||
let _pin = os::pin_multipart_source_directory(source, base, root)?;
|
||||
let _pin = if source_already_pinned {
|
||||
None
|
||||
} else {
|
||||
Some(os::pin_multipart_source_directory(source, base, root)?)
|
||||
};
|
||||
let parent = path
|
||||
.parent()
|
||||
.ok_or_else(|| std::io::Error::other("source backup has no parent"))?;
|
||||
let _parents = os::mkdir_all_below_existing_base_std(parent, base, root)?;
|
||||
let existed = if target {
|
||||
let existed = if fresh_owner {
|
||||
false
|
||||
} else if target {
|
||||
match os::observe_multipart_source_metadata(&path, base, root, bytes.len() as u64) {
|
||||
Ok((length, sha)) if length == bytes.len() as u64 && sha == raw_sha(bytes) => true,
|
||||
Ok(_) => {
|
||||
@@ -754,7 +807,10 @@ impl SourceStore {
|
||||
if !existed {
|
||||
os::replace_multipart_source_metadata(&path, bytes, base, root)?;
|
||||
}
|
||||
os::sync_multipart_owned_record_parents_options(source, parent, existed, sync_source_root, base, root)
|
||||
if sync_parents {
|
||||
os::sync_multipart_owned_record_parents_options(source, parent, existed, sync_source_root, base, root)?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
pub async fn replace_head(&self, bytes: &[u8]) -> Result<()> {
|
||||
self.multipart_index
|
||||
@@ -2024,11 +2080,41 @@ impl NativeMultipartPublication {
|
||||
}
|
||||
let setup_res: std::io::Result<()> = (|| {
|
||||
if let Some((original_bytes, directory, anchor_bytes)) = deferred_initial {
|
||||
SourceStore::write_owned_backup_std(source, base, root, backup_owner, false, &original_bytes, false)?;
|
||||
SourceStore::write_owned_backup_inner_std(
|
||||
source,
|
||||
base,
|
||||
root,
|
||||
backup_owner,
|
||||
false,
|
||||
&original_bytes,
|
||||
false,
|
||||
true,
|
||||
old_metadata.is_none(),
|
||||
true,
|
||||
)?;
|
||||
if let Some(previous) = old_metadata {
|
||||
SourceStore::write_owned_backup_std(source, base, root, backup_owner, true, previous, false)?;
|
||||
SourceStore::write_owned_backup_inner_std(
|
||||
source,
|
||||
base,
|
||||
root,
|
||||
backup_owner,
|
||||
true,
|
||||
previous,
|
||||
false,
|
||||
true,
|
||||
true,
|
||||
true,
|
||||
)?;
|
||||
}
|
||||
SourceStore::persist_state_std(source, base, root, &next_state_raw, terminal.as_deref(), false)?;
|
||||
SourceStore::persist_state_with_guard_std(
|
||||
source,
|
||||
base,
|
||||
root,
|
||||
&next_state_raw,
|
||||
terminal.as_deref(),
|
||||
false,
|
||||
Some(directory_guard),
|
||||
)?;
|
||||
let _parent = os::mkdir_all_below_existing_base_std(&directory, base, root)?;
|
||||
os::replace_multipart_source_metadata(&directory.join(RETIREMENT_ANCHOR), &anchor_bytes, base, root)?;
|
||||
#[cfg(target_os = "linux")]
|
||||
@@ -2037,9 +2123,28 @@ impl NativeMultipartPublication {
|
||||
}
|
||||
} else {
|
||||
if let Some(previous) = old_metadata {
|
||||
SourceStore::write_owned_backup_std(source, base, root, backup_owner, true, previous, false)?;
|
||||
SourceStore::write_owned_backup_inner_std(
|
||||
source,
|
||||
base,
|
||||
root,
|
||||
backup_owner,
|
||||
true,
|
||||
previous,
|
||||
false,
|
||||
false,
|
||||
true,
|
||||
true,
|
||||
)?;
|
||||
}
|
||||
SourceStore::persist_state_std(source, base, root, &next_state_raw, terminal.as_deref(), false)?;
|
||||
SourceStore::persist_state_with_guard_std(
|
||||
source,
|
||||
base,
|
||||
root,
|
||||
&next_state_raw,
|
||||
terminal.as_deref(),
|
||||
false,
|
||||
Some(directory_guard),
|
||||
)?;
|
||||
}
|
||||
Ok(())
|
||||
})();
|
||||
@@ -2269,6 +2374,7 @@ impl NativeMultipartPublication {
|
||||
target_already_synced: bool,
|
||||
fused_in_place: bool,
|
||||
staging_cleanup: Option<(PathBuf, Arc<std::sync::atomic::AtomicBool>)>,
|
||||
directory_guard: Option<&os::RenameCommitGuard>,
|
||||
) -> Result<MultipartPublicationResult> {
|
||||
let mut state = self.state.lock().clone();
|
||||
let expected_state_raw = encode(&state)?;
|
||||
@@ -2354,7 +2460,15 @@ impl NativeMultipartPublication {
|
||||
})?;
|
||||
}
|
||||
}
|
||||
SourceStore::persist_state_std(source, base, root, &next_state_raw, terminal.as_deref(), !fused_in_place)?;
|
||||
SourceStore::persist_state_with_guard_std(
|
||||
source,
|
||||
base,
|
||||
root,
|
||||
&next_state_raw,
|
||||
terminal.as_deref(),
|
||||
!fused_in_place,
|
||||
directory_guard,
|
||||
)?;
|
||||
if let Some((io_path, cleaned)) = staging_cleanup {
|
||||
match std::fs::remove_dir(&io_path) {
|
||||
Ok(()) => cleaned.store(true, std::sync::atomic::Ordering::Release),
|
||||
@@ -2387,7 +2501,7 @@ impl NativeMultipartPublication {
|
||||
) -> Result<MultipartPublicationResult> {
|
||||
let this = Arc::clone(self);
|
||||
os::run_blocking_namespace_operation_with_sync(target_lease, true, move || {
|
||||
this.finish_on_thread(target_already_synced, false, staging_cleanup)
|
||||
this.finish_on_thread(target_already_synced, false, staging_cleanup, None)
|
||||
.map_err(IoError::other)
|
||||
})
|
||||
.await
|
||||
|
||||
@@ -455,8 +455,11 @@ impl LocalDisk {
|
||||
}
|
||||
let meta_name = format!("part.{}.meta", part.number);
|
||||
let actual = if let Some(cached_part) =
|
||||
super::multipart_source_state::get_active_multipart_source_part(&source_object, part.number)
|
||||
{
|
||||
super::multipart_source_state::get_active_multipart_source_part_with_head(
|
||||
&source_object,
|
||||
part.number,
|
||||
from_active_cache,
|
||||
) {
|
||||
cached_part
|
||||
} else {
|
||||
if pinned_source_data.is_none() {
|
||||
|
||||
@@ -2222,6 +2222,36 @@ pub struct WalkDirOptions {
|
||||
// Override the remote stream stall timeout for long background walks.
|
||||
#[serde(default)]
|
||||
pub stall_timeout_ms: Option<u64>,
|
||||
|
||||
/// Select this leaf below identifier-like directory components. The root is
|
||||
/// `base_dir`; unrelated object metadata must not be read or returned.
|
||||
/// Older peers ignore this additive field, so consumers must also filter.
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub identifier_tree_leaf: Option<String>,
|
||||
}
|
||||
|
||||
pub(crate) fn is_identifier_tree_component(value: &str) -> bool {
|
||||
!value.is_empty()
|
||||
&& value
|
||||
.bytes()
|
||||
.all(|byte| byte.is_ascii_alphanumeric() || matches!(byte, b'_' | b'-'))
|
||||
}
|
||||
|
||||
pub(crate) fn is_identifier_tree_leaf(value: &str) -> bool {
|
||||
value.len() <= 255
|
||||
&& value.contains('.')
|
||||
&& !matches!(value, "." | "..")
|
||||
&& value
|
||||
.bytes()
|
||||
.all(|byte| byte.is_ascii_alphanumeric() || matches!(byte, b'_' | b'-' | b'.'))
|
||||
}
|
||||
|
||||
pub(crate) fn identifier_tree_leaf_matches(prefix: &str, leaf: &str, object: &str) -> bool {
|
||||
let Some(relative) = object.strip_prefix(prefix) else {
|
||||
return false;
|
||||
};
|
||||
let mut components = relative.split('/');
|
||||
components.next_back() == Some(leaf) && components.all(is_identifier_tree_component)
|
||||
}
|
||||
|
||||
impl WalkDirOptions {
|
||||
@@ -3157,6 +3187,7 @@ mod tests {
|
||||
skip_total_timeout: false,
|
||||
timeout_ms: Some(10_000),
|
||||
stall_timeout_ms: Some(20_000),
|
||||
identifier_tree_leaf: None,
|
||||
};
|
||||
|
||||
assert_eq!(opts.bucket, "test-bucket");
|
||||
@@ -3191,6 +3222,69 @@ mod tests {
|
||||
assert!(!decoded.skip_hidden_prefix_check);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn identifier_tree_leaf_selection_preserves_nested_identifier_directories() {
|
||||
let prefix = "catalog/namespaces/";
|
||||
let leaf = "table-entry.json";
|
||||
for relative in [
|
||||
"sales/tables/orders/table-entry.json",
|
||||
"sales/tables/orders/maintenance/tables/owner/table-entry.json",
|
||||
"maintenance/tables/tables/table-entry.json",
|
||||
] {
|
||||
assert!(identifier_tree_leaf_matches(prefix, leaf, &format!("{prefix}{relative}")));
|
||||
}
|
||||
for relative in [
|
||||
"sales/tables/orders/not-table-entry.json",
|
||||
"sales/tables/orders/table-entry.json/child",
|
||||
"sales/report.json/tables/owner/table-entry.json",
|
||||
"sales//tables/owner/table-entry.json",
|
||||
"sales/../tables/owner/table-entry.json",
|
||||
] {
|
||||
assert!(!identifier_tree_leaf_matches(prefix, leaf, &format!("{prefix}{relative}")));
|
||||
}
|
||||
assert!(!identifier_tree_leaf_matches(prefix, leaf, "other/sales/tables/orders/table-entry.json"));
|
||||
assert!(is_identifier_tree_leaf(leaf));
|
||||
for invalid in ["", ".", "..", "table-entry", "table/entry.json", "table\\entry.json"] {
|
||||
assert!(!is_identifier_tree_leaf(invalid), "invalid selector {invalid}");
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn identifier_tree_walk_options_are_additive_on_json_and_messagepack_maps() {
|
||||
#[derive(serde::Deserialize)]
|
||||
struct LegacyView {
|
||||
bucket: String,
|
||||
base_dir: String,
|
||||
recursive: bool,
|
||||
}
|
||||
let options = WalkDirOptions {
|
||||
bucket: "catalog".to_string(),
|
||||
base_dir: "namespaces/".to_string(),
|
||||
recursive: true,
|
||||
identifier_tree_leaf: Some("table-entry.json".to_string()),
|
||||
..Default::default()
|
||||
};
|
||||
let json = serde_json::to_vec(&options).expect("encode current HTTP options");
|
||||
let messagepack = rmp_serde::to_vec_named(&options).expect("encode legacy RPC map");
|
||||
for old in [
|
||||
serde_json::from_slice::<LegacyView>(&json).expect("old JSON readers ignore the additive field"),
|
||||
rmp_serde::from_slice::<LegacyView>(&messagepack).expect("old MessagePack readers ignore the additive field"),
|
||||
] {
|
||||
assert_eq!(old.bucket, options.bucket);
|
||||
assert_eq!(old.base_dir, options.base_dir);
|
||||
assert!(old.recursive);
|
||||
}
|
||||
let mut old = serde_json::to_value(&options).expect("encode legacy options");
|
||||
old.as_object_mut().expect("options are a map").remove("identifier_tree_leaf");
|
||||
let from_json: WalkDirOptions = serde_json::from_value(old.clone()).expect("decode old HTTP options");
|
||||
let from_messagepack: WalkDirOptions =
|
||||
rmp_serde::from_slice(&rmp_serde::to_vec_named(&old).expect("encode old RPC map")).expect("decode old RPC map");
|
||||
assert!(from_json.identifier_tree_leaf.is_none());
|
||||
assert!(from_messagepack.identifier_tree_leaf.is_none());
|
||||
assert_eq!(from_json.base_dir, options.base_dir);
|
||||
assert_eq!(from_messagepack.base_dir, options.base_dir);
|
||||
}
|
||||
|
||||
/// Test DeleteOptions structure
|
||||
#[test]
|
||||
fn test_delete_options() {
|
||||
|
||||
@@ -611,6 +611,10 @@ pub(crate) mod prepare_rename_test_hooks {
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) fn has_after_component_opened() -> bool {
|
||||
!AFTER_COMPONENT_OPENED.lock().is_empty()
|
||||
}
|
||||
|
||||
/// Hooks keyed by a directory component name, run before a Windows
|
||||
/// directory pin retries a transient sharing conflict on it.
|
||||
#[cfg(windows)]
|
||||
@@ -3804,7 +3808,7 @@ pub(crate) fn rename_all_with_precondition_and_parent_sync_std(
|
||||
#[cfg(all(test, not(windows)))]
|
||||
prepared_publication_test_hooks::run_rename_destination(source, destination);
|
||||
#[cfg(all(any(test, feature = "test-util"), not(windows)))]
|
||||
{
|
||||
if prepared_publication_test_hooks::has_any_hooks() {
|
||||
prepared_publication_test_hooks::run(prepared_publication_test_hooks::Stage::Rename, source);
|
||||
prepared_publication_test_hooks::run(prepared_publication_test_hooks::Stage::Rename, destination);
|
||||
if let Some(check) = precondition {
|
||||
@@ -7198,6 +7202,39 @@ pub(crate) fn pin_multipart_source_directory(
|
||||
}
|
||||
}
|
||||
|
||||
fn same_parent_commit_guard(
|
||||
parent: &Path,
|
||||
base: &Path,
|
||||
publication_root: &PublicationRoot,
|
||||
directory_guard: Option<&RenameCommitGuard>,
|
||||
) -> io::Result<RenameCommitGuard> {
|
||||
#[cfg(target_os = "linux")]
|
||||
if let Some(directory_guard) = directory_guard.filter(|g| g.directories.source_path == parent) {
|
||||
let dir = &directory_guard.directories.source;
|
||||
return Ok(RenameCommitGuard {
|
||||
directories: Arc::new(LinuxCommitDirectories {
|
||||
root: directory_guard.directories.root.clone(),
|
||||
source_path: parent.to_path_buf(),
|
||||
destination_path: parent.to_path_buf(),
|
||||
source: LinuxCommitDirectory {
|
||||
file: Arc::clone(&dir.file),
|
||||
device: dir.device,
|
||||
inode: dir.inode,
|
||||
},
|
||||
destination: std::sync::OnceLock::from(LinuxCommitDirectory {
|
||||
file: Arc::clone(&dir.file),
|
||||
device: dir.device,
|
||||
inode: dir.inode,
|
||||
}),
|
||||
destination_base: base.to_path_buf(),
|
||||
}),
|
||||
});
|
||||
}
|
||||
#[cfg(not(target_os = "linux"))]
|
||||
let _ = directory_guard;
|
||||
prepare_rename_commit_guard(parent, parent, base, publication_root)
|
||||
}
|
||||
|
||||
/// A small control record always gets its own inode and an atomic replacement.
|
||||
/// The caller owns the upload SourceRW and keeps its namespace lease alive.
|
||||
pub(crate) fn replace_multipart_source_metadata(
|
||||
@@ -7205,11 +7242,21 @@ pub(crate) fn replace_multipart_source_metadata(
|
||||
bytes: &[u8],
|
||||
base: &Path,
|
||||
publication_root: &PublicationRoot,
|
||||
) -> io::Result<()> {
|
||||
replace_multipart_source_metadata_with_guard(path, bytes, base, publication_root, None)
|
||||
}
|
||||
|
||||
pub(crate) fn replace_multipart_source_metadata_with_guard(
|
||||
path: &Path,
|
||||
bytes: &[u8],
|
||||
base: &Path,
|
||||
publication_root: &PublicationRoot,
|
||||
directory_guard: Option<&RenameCommitGuard>,
|
||||
) -> io::Result<()> {
|
||||
let parent = path.parent().ok_or_else(|| io::Error::other("MPU record has no parent"))?;
|
||||
#[cfg(not(target_os = "linux"))]
|
||||
let _pinned = pin_multipart_source_directory(parent, base, publication_root)?;
|
||||
let guard = prepare_rename_commit_guard(parent, parent, base, publication_root)?;
|
||||
let guard = same_parent_commit_guard(parent, base, publication_root, directory_guard)?;
|
||||
let staged = parent.join(format!(".mpu-record-{}", uuid::Uuid::new_v4()));
|
||||
let mut prepared = create_prepared_rename_source_with_commit_guard(&staged, path, &guard)?;
|
||||
prepared.write_all(bytes, true)?;
|
||||
@@ -7458,37 +7505,9 @@ pub(crate) fn install_multipart_prepared_head(
|
||||
if original_path.parent() != Some(parent) || original_path == canonical_head {
|
||||
return Err(io::Error::new(io::ErrorKind::InvalidInput, "MPU proxy must be a private sibling"));
|
||||
}
|
||||
#[cfg(target_os = "linux")]
|
||||
let guard = if let Some(directory_guard) = directory_guard.filter(|g| g.directories.source_path == parent) {
|
||||
let dir = &directory_guard.directories.source;
|
||||
RenameCommitGuard {
|
||||
directories: Arc::new(LinuxCommitDirectories {
|
||||
root: directory_guard.directories.root.clone(),
|
||||
source_path: parent.to_path_buf(),
|
||||
destination_path: parent.to_path_buf(),
|
||||
source: LinuxCommitDirectory {
|
||||
file: Arc::clone(&dir.file),
|
||||
device: dir.device,
|
||||
inode: dir.inode,
|
||||
},
|
||||
destination: std::sync::OnceLock::from(LinuxCommitDirectory {
|
||||
file: Arc::clone(&dir.file),
|
||||
device: dir.device,
|
||||
inode: dir.inode,
|
||||
}),
|
||||
destination_base: base.to_path_buf(),
|
||||
}),
|
||||
}
|
||||
} else {
|
||||
prepare_rename_commit_guard(parent, parent, base, publication_root)?
|
||||
};
|
||||
#[cfg(not(target_os = "linux"))]
|
||||
let _pinned = pin_multipart_source_directory(parent, base, publication_root)?;
|
||||
#[cfg(not(target_os = "linux"))]
|
||||
let guard = {
|
||||
let _ = directory_guard;
|
||||
prepare_rename_commit_guard(parent, parent, base, publication_root)?
|
||||
};
|
||||
let guard = same_parent_commit_guard(parent, base, publication_root, directory_guard)?;
|
||||
rename_prepared_source_std(&prepared, &original_path, canonical_head, base, publication_root, &guard)?;
|
||||
prepared.path = canonical_head.to_path_buf();
|
||||
fsync_commit_directory_limited_std(parent, &guard)?;
|
||||
@@ -7790,8 +7809,7 @@ pub(crate) fn hardlink_multipart_source_files(
|
||||
}
|
||||
#[cfg(unix)]
|
||||
{
|
||||
use rustix::fs::{AtFlags, Mode, OFlags, linkat, openat, unlinkat};
|
||||
use std::os::unix::fs::MetadataExt;
|
||||
use rustix::fs::{AtFlags, FileType, linkat, statat, unlinkat};
|
||||
#[cfg(test)]
|
||||
let source_path = source.to_path_buf();
|
||||
#[cfg(test)]
|
||||
@@ -7806,20 +7824,26 @@ pub(crate) fn hardlink_multipart_source_files(
|
||||
.ok_or_else(|| io::Error::other("missing multipart attempt directory"))?;
|
||||
#[cfg(test)]
|
||||
prepared_publication_test_hooks::run_hardlink_destination(&source_path, &destination_path);
|
||||
let flags = OFlags::RDONLY | OFlags::NOFOLLOW | OFlags::CLOEXEC;
|
||||
for name in names {
|
||||
if cancelled() {
|
||||
return Err(io::Error::new(io::ErrorKind::Interrupted, "multipart staging cancelled"));
|
||||
}
|
||||
let original = std::fs::File::from(openat(source, name.as_str(), flags, Mode::empty())?);
|
||||
let expected = original.metadata()?;
|
||||
if !expected.is_file() {
|
||||
let expected = statat(source, name.as_str(), AtFlags::SYMLINK_NOFOLLOW)?;
|
||||
let expected_type = FileType::from_raw_mode(expected.st_mode);
|
||||
if expected_type.is_symlink() {
|
||||
return Err(io::Error::from_raw_os_error(libc::ELOOP));
|
||||
}
|
||||
if !expected_type.is_file() {
|
||||
return Err(io::Error::new(io::ErrorKind::InvalidData, "multipart shard is not a regular file"));
|
||||
}
|
||||
linkat(source, name.as_str(), destination, name.as_str(), AtFlags::empty())?;
|
||||
let linked = std::fs::File::from(openat(destination, name.as_str(), flags, Mode::empty())?);
|
||||
let actual = linked.metadata()?;
|
||||
if actual.dev() != expected.dev() || actual.ino() != expected.ino() {
|
||||
let actual = statat(destination, name.as_str(), AtFlags::SYMLINK_NOFOLLOW)?;
|
||||
if !FileType::from_raw_mode(actual.st_mode).is_file()
|
||||
|| actual.st_dev != expected.st_dev
|
||||
|| actual.st_ino != expected.st_ino
|
||||
|| actual.st_size != expected.st_size
|
||||
|| actual.st_mtime != expected.st_mtime
|
||||
{
|
||||
let _ = unlinkat(destination, name.as_str(), AtFlags::empty());
|
||||
return Err(io::Error::new(io::ErrorKind::InvalidData, "multipart shard changed during staging"));
|
||||
}
|
||||
@@ -8240,7 +8264,37 @@ pub(crate) fn mkdir_all_below_existing_base_std(
|
||||
let flags = OFlags::RDONLY | OFlags::DIRECTORY | OFlags::NOFOLLOW | OFlags::CLOEXEC;
|
||||
let mode = Mode::RWXU | Mode::RWXG | Mode::RWXO;
|
||||
#[cfg(target_os = "linux")]
|
||||
let base = publication_root.open_directory(base_dir)?.into();
|
||||
let base = {
|
||||
use rustix::fs::{ResolveFlags, openat2};
|
||||
use std::os::fd::AsFd;
|
||||
|
||||
let owned_base = (base_dir != publication_root.io_path.as_path() && base_dir != publication_root.path.as_path())
|
||||
.then(|| publication_root.open_directory(base_dir).map(std::os::fd::OwnedFd::from))
|
||||
.transpose()?;
|
||||
#[cfg(test)]
|
||||
let has_hook = prepare_rename_test_hooks::has_after_component_opened();
|
||||
#[cfg(not(test))]
|
||||
let has_hook = false;
|
||||
if !has_hook {
|
||||
let base_fd = owned_base
|
||||
.as_ref()
|
||||
.map_or_else(|| publication_root.directory.as_fd(), AsFd::as_fd);
|
||||
let target = if relative.as_os_str().is_empty() {
|
||||
Path::new(".")
|
||||
} else {
|
||||
relative
|
||||
};
|
||||
match openat2(base_fd, target, flags, Mode::empty(), ResolveFlags::BENEATH | ResolveFlags::NO_SYMLINKS) {
|
||||
Ok(leaf) => return Ok(vec![leaf]),
|
||||
Err(Errno::NOENT | Errno::NOSYS) => {}
|
||||
Err(err) => return Err(err.into()),
|
||||
}
|
||||
}
|
||||
match owned_base {
|
||||
Some(base) => base,
|
||||
None => publication_root.open_directory(base_dir)?.into(),
|
||||
}
|
||||
};
|
||||
#[cfg(not(target_os = "linux"))]
|
||||
let base = {
|
||||
// The configured endpoint is trusted; every component beneath it
|
||||
|
||||
@@ -593,7 +593,7 @@ impl ECStore {
|
||||
#[cfg(test)]
|
||||
crate::core::pools::observe_pool_activation_start_attempt(crate::core::pools::PoolActivationStartKind::Rebalance);
|
||||
let fleet_proof = acquire_pool_activation_fleet_proof(&self.ctx).await?;
|
||||
let mut pool_meta_guard = self.pool_meta_save_gate.lock().await;
|
||||
let mut pool_meta_guard = self.pool_meta_save_gate.write().await;
|
||||
pool_meta_guard.ensure_write_safe(stage)?;
|
||||
let activation_fence = acquire_pool_rebalance_activation_locks(pool.clone(), fleet_proof).await?;
|
||||
let pool_meta = self
|
||||
@@ -625,7 +625,7 @@ impl ECStore {
|
||||
{
|
||||
// Lock order: pool_meta_save_gate -> rebalance.bin -> pool.bin. These
|
||||
// shared fences admit a peer without waiting for the run's own entries.
|
||||
let mut pool_meta_guard = self.pool_meta_save_gate.lock().await;
|
||||
let mut pool_meta_guard = self.pool_meta_save_gate.write().await;
|
||||
pool_meta_guard.ensure_write_safe("rebalance worker activation")?;
|
||||
let mut worker_fence = acquire_pool_rebalance_worker_locks(pool.clone()).await?;
|
||||
let pool_meta = self
|
||||
@@ -654,7 +654,7 @@ impl ECStore {
|
||||
{
|
||||
// The admission read fences must already be dropped. Reacquire in the
|
||||
// original order and revalidate every durable boundary across that gap.
|
||||
let mut pool_meta_guard = self.pool_meta_save_gate.lock().await;
|
||||
let mut pool_meta_guard = self.pool_meta_save_gate.write().await;
|
||||
pool_meta_guard.ensure_write_safe("rebalance worker metadata update")?;
|
||||
let mut activation_fence = acquire_pool_rebalance_activation_locks(pool.clone(), None).await?;
|
||||
let pool_meta = self
|
||||
@@ -1714,7 +1714,7 @@ mod tests {
|
||||
persist_pool_meta_identity_for_startup(store.pools.clone(), &mut write_state, true)
|
||||
.await
|
||||
.expect("initialized pool metadata identity should persist");
|
||||
*store.pool_meta_save_gate.lock().await = write_state;
|
||||
*store.pool_meta_save_gate.write().await = write_state;
|
||||
// Simulate physical loss; raw deletion of authoritative pool.bin is
|
||||
// deliberately rejected by the native metadata mutation guard.
|
||||
for pool in &store.pools {
|
||||
|
||||
@@ -100,7 +100,7 @@ pub async fn test_store_with_persisted_rebalance_meta(
|
||||
rebalance_meta: tokio::sync::RwLock::new(Some(meta)),
|
||||
decommission_cancelers: tokio::sync::RwLock::new(vec![None]),
|
||||
start_gate: tokio::sync::Mutex::new(()),
|
||||
pool_meta_save_gate: tokio::sync::Mutex::default(),
|
||||
pool_meta_save_gate: crate::core::pools::PoolMetaSaveGate::default(),
|
||||
ctx,
|
||||
bucket_fence_registry: std::sync::Arc::default(),
|
||||
});
|
||||
@@ -335,7 +335,7 @@ async fn test_pool_stores_with_contexts(
|
||||
rebalance_meta: tokio::sync::RwLock::new(rebalance_meta.clone()),
|
||||
decommission_cancelers: tokio::sync::RwLock::new(vec![None; pool_count]),
|
||||
start_gate: tokio::sync::Mutex::new(()),
|
||||
pool_meta_save_gate: tokio::sync::Mutex::new(pool_meta_write_state.independent_clone_for_test()),
|
||||
pool_meta_save_gate: crate::core::pools::PoolMetaSaveGate::new(pool_meta_write_state.independent_clone_for_test()),
|
||||
ctx: store_ctx,
|
||||
bucket_fence_registry: std::sync::Arc::default(),
|
||||
})
|
||||
|
||||
@@ -3412,7 +3412,7 @@ fn test_store_with_rebalance_meta(meta: RebalanceMeta) -> Arc<crate::store::ECSt
|
||||
rebalance_meta: tokio::sync::RwLock::new(Some(meta)),
|
||||
decommission_cancelers: tokio::sync::RwLock::new(Vec::new()),
|
||||
start_gate: tokio::sync::Mutex::new(()),
|
||||
pool_meta_save_gate: tokio::sync::Mutex::default(),
|
||||
pool_meta_save_gate: crate::core::pools::PoolMetaSaveGate::default(),
|
||||
ctx: crate::runtime::instance::bootstrap_ctx(),
|
||||
bucket_fence_registry: std::sync::Arc::default(),
|
||||
})
|
||||
|
||||
@@ -93,6 +93,7 @@ use super::{
|
||||
delete_tier_mutation_intent_record_if_current, list_tier_coordinator_mutation_intent_records,
|
||||
list_tier_mutation_intent_records, load_tier_coordinator_mutation_intent_record_with_etag,
|
||||
load_tier_mutation_intent_record_with_etag, save_tier_coordinator_mutation_intent_record_if_absent,
|
||||
save_tier_mutation_intent_record_if_current,
|
||||
},
|
||||
warm_backend::WarmBackendImpl,
|
||||
};
|
||||
@@ -620,7 +621,11 @@ struct CommittedMutationRecoveryPlan {
|
||||
|
||||
impl TierMutationRecoverySnapshot {
|
||||
fn requires_config_lock(&self) -> bool {
|
||||
self.has_committed_mutation_blocks || self.intents.iter().any(|recovered| recovered.has_coordinator_record)
|
||||
self.has_committed_mutation_blocks
|
||||
|| self.intents.iter().any(|recovered| {
|
||||
recovered.has_coordinator_record
|
||||
|| (recovered.has_peer_record && recovered.intent.state == TierMutationIntentState::Prepared)
|
||||
})
|
||||
}
|
||||
|
||||
fn allowance(&self) -> MutationBlockAllowance {
|
||||
@@ -5102,6 +5107,9 @@ impl TierConfigMgr {
|
||||
|
||||
Self::reconcile_terminal_dual_prefix_mutation_intents(api.clone(), &mut snapshot).await?;
|
||||
Self::recover_prepared_coordinator_mutation_intents(api.clone(), &mut snapshot.coordinator_intents).await?;
|
||||
if config_lock.is_some() {
|
||||
Self::recover_expired_peer_only_prepares(api.clone(), &snapshot).await?;
|
||||
}
|
||||
|
||||
snapshot = Self::load_and_reconcile_mutation_recovery_snapshot(handle, api.clone(), true).await?;
|
||||
if snapshot.requires_config_lock() && config_lock.is_none() {
|
||||
@@ -5502,6 +5510,58 @@ impl TierConfigMgr {
|
||||
}
|
||||
}
|
||||
|
||||
async fn recover_expired_peer_only_prepares<S>(api: Arc<S>, snapshot: &TierMutationRecoverySnapshot) -> io::Result<()>
|
||||
where
|
||||
S: EcstoreObjectIO,
|
||||
{
|
||||
let now = i64::try_from(OffsetDateTime::now_utc().unix_timestamp_nanos()).unwrap_or(i64::MAX);
|
||||
let orphaned = snapshot
|
||||
.intents
|
||||
.iter()
|
||||
.filter(|recovered| {
|
||||
recovered.has_peer_record
|
||||
&& !recovered.has_coordinator_record
|
||||
&& recovered.intent.state == TierMutationIntentState::Prepared
|
||||
&& Self::tier_mutation_tombstone_retention_elapsed(&recovered.intent, now)
|
||||
})
|
||||
.map(|recovered| recovered.intent.clone())
|
||||
.collect::<Vec<_>>();
|
||||
if orphaned.is_empty() {
|
||||
return Ok(());
|
||||
}
|
||||
|
||||
// The caller holds the tier-config namespace lock. Only the exact
|
||||
// persisted candidate proves a commit; an old or different config
|
||||
// leaves the peer fence intact.
|
||||
let loaded = load_tier_config_for_recovery(api.clone()).await?;
|
||||
let Some(config_etag) = loaded.etag else {
|
||||
return Ok(());
|
||||
};
|
||||
for intent in orphaned {
|
||||
if loaded.persisted_digest != Some(intent.candidate_digest) {
|
||||
continue;
|
||||
}
|
||||
let _mutation_guard = acquire_tier_mutation_mutex(intent.mutation_id).await;
|
||||
let (mut current, record_etag) =
|
||||
match load_tier_mutation_intent_record_with_etag(api.clone(), intent.mutation_id).await {
|
||||
Ok(record) => record,
|
||||
Err(Error::ConfigNotFound) => continue,
|
||||
Err(err) => return Err(tier_mutation_replay_error(err)),
|
||||
};
|
||||
if current != intent {
|
||||
continue;
|
||||
}
|
||||
current
|
||||
.advance(TierMutationIntentState::Committed, Some(config_etag.clone()))
|
||||
.map_err(io::Error::other)?;
|
||||
match save_tier_mutation_intent_record_if_current(api.clone(), ¤t, &record_etag).await {
|
||||
Ok(()) | Err(Error::PreconditionFailed) => {}
|
||||
Err(err) => return Err(tier_mutation_replay_error(err)),
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
async fn recover_prepared_coordinator_mutation_intents<S>(api: Arc<S>, intents: &mut [TierMutationIntent]) -> io::Result<()>
|
||||
where
|
||||
S: EcstoreObjectIO,
|
||||
@@ -13379,6 +13439,208 @@ mod tests {
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn reload_recovers_expired_peer_only_prepare_for_persisted_add() {
|
||||
use crate::services::tier::tier_mutation_intent::save_tier_mutation_intent_record;
|
||||
|
||||
let store = Arc::new(CasConfigStore::default());
|
||||
let previous = empty_mgr();
|
||||
let mut candidate = empty_mgr();
|
||||
candidate.tiers.insert("COLD-A".to_string(), build_rustfs_tier("COLD-A"));
|
||||
let targets = build_tier_mutation_affected_targets(
|
||||
TierMutationIntentKind::Add,
|
||||
HashSet::from(["COLD-A".to_string()]),
|
||||
&previous,
|
||||
&candidate,
|
||||
)
|
||||
.expect("added tier should have a target");
|
||||
let mut intent = build_coordinator_tier_mutation_intent(TierMutationIntentKind::Add, None, &candidate, targets)
|
||||
.expect("intent should build")
|
||||
.expect("add should require an intent");
|
||||
intent.expires_at_unix_nanos = 1;
|
||||
candidate
|
||||
.save_tiering_config_if_current(store.clone(), None)
|
||||
.await
|
||||
.expect("candidate should persist");
|
||||
save_tier_mutation_intent_record(store.clone(), &intent)
|
||||
.await
|
||||
.expect("peer prepare should persist without a coordinator record");
|
||||
|
||||
let manager = TierConfigMgr::new();
|
||||
TIER_MUTATION_TEST_PEERS
|
||||
.scope(Vec::new(), async {
|
||||
TIER_DRIVER_TEST_FACTORY
|
||||
.scope(healthy_driver_factory(), async {
|
||||
TierConfigMgr::reload_handle_with(&manager, store.clone())
|
||||
.await
|
||||
.expect("matching saved candidate should recover");
|
||||
TierConfigMgr::acquire_operation_lease(&manager, "COLD-A")
|
||||
.await
|
||||
.expect("recovered tier should be readable");
|
||||
})
|
||||
.await;
|
||||
})
|
||||
.await;
|
||||
assert!(
|
||||
TierConfigMgr::load_tier_mutation_intents(store)
|
||||
.await
|
||||
.expect("peer scan should succeed")
|
||||
.iter()
|
||||
.all(|record| record.state != TierMutationIntentState::Prepared),
|
||||
"the orphaned prepare must no longer block later reloads"
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn reload_keeps_peer_only_prepare_without_commit_proof() {
|
||||
use crate::services::tier::tier_mutation_intent::{
|
||||
save_tier_mutation_intent_record, tier_mutation_intent_record_object_name,
|
||||
};
|
||||
|
||||
for case in [
|
||||
"unexpired",
|
||||
"recently_expired",
|
||||
"different_candidate",
|
||||
"missing_config",
|
||||
"concurrent_change",
|
||||
] {
|
||||
let store = Arc::new(CasConfigStore::default());
|
||||
let previous = empty_mgr();
|
||||
let mut candidate = empty_mgr();
|
||||
candidate.tiers.insert("COLD-A".to_string(), build_rustfs_tier("COLD-A"));
|
||||
let targets = build_tier_mutation_affected_targets(
|
||||
TierMutationIntentKind::Add,
|
||||
HashSet::from(["COLD-A".to_string()]),
|
||||
&previous,
|
||||
&candidate,
|
||||
)
|
||||
.expect("added tier should have a target");
|
||||
let mut intent = build_coordinator_tier_mutation_intent(TierMutationIntentKind::Add, None, &candidate, targets)
|
||||
.expect("intent should build")
|
||||
.expect("add should require an intent");
|
||||
intent.expires_at_unix_nanos = match case {
|
||||
"unexpired" => tier_mutation_intent_expiry_unix_nanos(),
|
||||
"recently_expired" => {
|
||||
i64::try_from(OffsetDateTime::now_utc().unix_timestamp_nanos()).expect("test clock should fit")
|
||||
- 1_000_000_000
|
||||
}
|
||||
_ => 1,
|
||||
};
|
||||
if case == "different_candidate" {
|
||||
intent.candidate_digest = [0x51; 32];
|
||||
}
|
||||
if case != "missing_config" {
|
||||
candidate
|
||||
.save_tiering_config_if_current(store.clone(), None)
|
||||
.await
|
||||
.expect("candidate should persist");
|
||||
}
|
||||
save_tier_mutation_intent_record(store.clone(), &intent)
|
||||
.await
|
||||
.expect("peer prepare should persist");
|
||||
let expected = if case == "concurrent_change" {
|
||||
let mut changed = intent.clone();
|
||||
changed.candidate_digest = [0x52; 32];
|
||||
let object = tier_mutation_intent_record_object_name(intent.mutation_id).expect("record path should build");
|
||||
store
|
||||
.rewrite_on_next_if_match(object, changed.encode().expect("changed intent should encode"))
|
||||
.await;
|
||||
changed
|
||||
} else {
|
||||
intent
|
||||
};
|
||||
|
||||
let manager = TierConfigMgr::new();
|
||||
TIER_MUTATION_TEST_PEERS
|
||||
.scope(Vec::new(), async {
|
||||
TierConfigMgr::reload_handle_with(&manager, store.clone())
|
||||
.await
|
||||
.expect("unproven prepare should remain fenced");
|
||||
})
|
||||
.await;
|
||||
let error = match TierConfigMgr::acquire_operation_lease(&manager, "COLD-A").await {
|
||||
Ok(_) => panic!("{case}: unproven prepare must block reads"),
|
||||
Err(error) => error,
|
||||
};
|
||||
assert!(error.message.contains("being replaced"), "{case}: {error}");
|
||||
let records = TierConfigMgr::load_tier_mutation_intents(store)
|
||||
.await
|
||||
.expect("peer scan should succeed");
|
||||
assert_eq!(records, vec![expected], "{case}: prepared evidence must remain unchanged");
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn peer_only_prepare_recovery_retains_fence_until_peer_commit_succeeds() {
|
||||
use crate::services::tier::tier_mutation_intent::save_tier_mutation_intent_record;
|
||||
|
||||
let store = Arc::new(CasConfigStore::default());
|
||||
let previous = empty_mgr();
|
||||
let mut candidate = empty_mgr();
|
||||
candidate.tiers.insert("COLD-A".to_string(), build_rustfs_tier("COLD-A"));
|
||||
let targets = build_tier_mutation_affected_targets(
|
||||
TierMutationIntentKind::Add,
|
||||
HashSet::from(["COLD-A".to_string()]),
|
||||
&previous,
|
||||
&candidate,
|
||||
)
|
||||
.expect("added tier should have a target");
|
||||
let mut intent = build_coordinator_tier_mutation_intent(TierMutationIntentKind::Add, None, &candidate, targets)
|
||||
.expect("intent should build")
|
||||
.expect("add should require an intent");
|
||||
intent.expires_at_unix_nanos = 1;
|
||||
candidate
|
||||
.save_tiering_config_if_current(store.clone(), None)
|
||||
.await
|
||||
.expect("candidate should persist");
|
||||
save_tier_mutation_intent_record(store.clone(), &intent)
|
||||
.await
|
||||
.expect("peer prepare should persist");
|
||||
|
||||
let manager = TierConfigMgr::new();
|
||||
let calls = Arc::new(Mutex::new(Vec::new()));
|
||||
TIER_MUTATION_TEST_PEERS
|
||||
.scope(vec![FakeTierMutationPeer::boxed("peer-a", calls.clone(), Err("offline"))], async {
|
||||
TierConfigMgr::reload_handle_with(&manager, store.clone())
|
||||
.await
|
||||
.expect_err("failed peer commit must leave recovery incomplete");
|
||||
})
|
||||
.await;
|
||||
let retained = TierConfigMgr::load_tier_mutation_intents(store.clone())
|
||||
.await
|
||||
.expect("peer scan should succeed");
|
||||
assert_eq!(retained.len(), 1);
|
||||
assert_eq!(retained[0].state, TierMutationIntentState::Committed);
|
||||
let error = match TierConfigMgr::acquire_operation_lease(&manager, "COLD-A").await {
|
||||
Ok(_) => panic!("failed peer commit must retain the fence"),
|
||||
Err(error) => error,
|
||||
};
|
||||
assert!(error.message.contains("being replaced"), "{error}");
|
||||
|
||||
TIER_MUTATION_TEST_PEERS
|
||||
.scope(
|
||||
vec![FakeTierMutationPeer::boxed(
|
||||
"peer-a",
|
||||
calls.clone(),
|
||||
Ok(PeerTierMutationState::Committed),
|
||||
)],
|
||||
async {
|
||||
TIER_DRIVER_TEST_FACTORY
|
||||
.scope(healthy_driver_factory(), async {
|
||||
TierConfigMgr::reload_handle_with(&manager, store.clone())
|
||||
.await
|
||||
.expect("peer commit retry should recover");
|
||||
TierConfigMgr::acquire_operation_lease(&manager, "COLD-A")
|
||||
.await
|
||||
.expect("tier should be readable after the retry");
|
||||
})
|
||||
.await;
|
||||
},
|
||||
)
|
||||
.await;
|
||||
assert_eq!(lock_unpoisoned(&calls).len(), 2, "recovery should retry the peer commit");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn coordinator_mutation_recovery_scan_blocks_new_operation_lease() {
|
||||
let store = Arc::new(CasConfigStore::default());
|
||||
|
||||
@@ -4608,6 +4608,19 @@ fn marker_purge_receipt_path(
|
||||
}
|
||||
|
||||
impl SetDisks {
|
||||
/// The first disk slot UUID is assigned when this erasure set is formatted.
|
||||
/// It survives pool reordering and replacement of a disk in that slot.
|
||||
pub fn persistent_set_id(&self) -> Result<Uuid> {
|
||||
self.format
|
||||
.erasure
|
||||
.sets
|
||||
.get(self.set_index)
|
||||
.and_then(|disks| disks.first())
|
||||
.copied()
|
||||
.filter(|id| !id.is_nil())
|
||||
.ok_or_else(|| Error::other("erasure set has no persistent format identity"))
|
||||
}
|
||||
|
||||
/// Persist a quorum DELETE receipt outside the object marker itself. The
|
||||
/// receipt is written to the metadata bucket so it survives a node restart
|
||||
/// while an inaccessible member still holds the stale marker.
|
||||
|
||||
@@ -443,6 +443,7 @@ impl ECStore {
|
||||
},
|
||||
other => StorageError::Lock(other),
|
||||
})?;
|
||||
let ns_guard = Arc::new(ns_guard);
|
||||
|
||||
let recover = Box::pin(async {
|
||||
self.ensure_bucket_volume_absent_on_all_sets(bucket).await?;
|
||||
@@ -930,15 +931,16 @@ impl ECStore {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
async fn cleanup_bucket_usage(&self, bucket: &str, guard: Option<&rustfs_lock::NamespaceLockGuard>) -> Result<()> {
|
||||
run_bucket_usage_cleanup(guard, bucket, async {
|
||||
crate::data_usage::prepare_bucket_usage_for_namespace_change(bucket, guard).await?;
|
||||
async fn cleanup_bucket_usage(&self, bucket: &str, guard: Option<&Arc<rustfs_lock::NamespaceLockGuard>>) -> Result<()> {
|
||||
let namespace_guard = guard.map(Arc::as_ref);
|
||||
run_bucket_usage_cleanup(namespace_guard, bucket, async {
|
||||
crate::data_usage::prepare_bucket_usage_for_namespace_change(bucket, namespace_guard).await?;
|
||||
crate::data_usage::remove_bucket_usage_from_backend_with_guard_fenced(self, bucket, guard).await
|
||||
})
|
||||
.await
|
||||
}
|
||||
|
||||
async fn cleanup_bucket_usage_best_effort(&self, bucket: &str, guard: Option<&rustfs_lock::NamespaceLockGuard>) {
|
||||
async fn cleanup_bucket_usage_best_effort(&self, bucket: &str, guard: Option<&Arc<rustfs_lock::NamespaceLockGuard>>) {
|
||||
if let Err(err) = self.cleanup_bucket_usage(bucket, guard).await {
|
||||
warn!(
|
||||
bucket = %bucket,
|
||||
@@ -1029,6 +1031,7 @@ impl ECStore {
|
||||
} else {
|
||||
None
|
||||
};
|
||||
let ns_guard = ns_guard.map(Arc::new);
|
||||
|
||||
let existing_bucket_info = match self.get_bucket_info_from_sets(bucket, &BucketOptions::default()).await {
|
||||
Ok(info) => Some(info),
|
||||
@@ -1146,7 +1149,7 @@ impl ECStore {
|
||||
metadata_sys::set_new_bucket_metadata_intent_in(&self.ctx, meta.clone()).await?;
|
||||
if bucket_lifecycle_guard.as_ref().is_some_and(|guard| guard.is_lock_lost())
|
||||
|| metadata_transaction_guard.as_ref().is_some_and(|guard| guard.is_lock_lost())
|
||||
|| ns_guard.as_ref().is_some_and(|guard| guard.is_lock_lost())
|
||||
|| ns_guard.as_deref().is_some_and(|guard| guard.is_lock_lost())
|
||||
{
|
||||
return Err(Error::other(format!("bucket metadata creation intent lock was lost: {bucket}")));
|
||||
}
|
||||
@@ -1162,7 +1165,7 @@ impl ECStore {
|
||||
|
||||
if let Err(err) = await_bucket_lifecycle_operation(
|
||||
bucket_lifecycle_guard.as_ref(),
|
||||
ns_guard.as_ref(),
|
||||
ns_guard.as_deref(),
|
||||
bucket,
|
||||
"physical bucket creation",
|
||||
await_bucket_namespace_operation(
|
||||
@@ -1191,9 +1194,9 @@ impl ECStore {
|
||||
{
|
||||
warn!("best-effort bucket heal after BucketExists failed: {heal_err}");
|
||||
}
|
||||
if confirmed_missing && !is_err_bucket_exists(&err) && ns_guard.as_ref().is_none_or(|guard| !guard.is_lock_lost()) {
|
||||
if confirmed_missing && !is_err_bucket_exists(&err) && ns_guard.as_deref().is_none_or(|guard| !guard.is_lock_lost()) {
|
||||
error!("make bucket failed: {err}");
|
||||
self.rollback_failed_bucket_creation(bucket, bucket_lifecycle_guard.as_ref(), ns_guard.as_ref())
|
||||
self.rollback_failed_bucket_creation(bucket, bucket_lifecycle_guard.as_ref(), ns_guard.as_deref())
|
||||
.await;
|
||||
}
|
||||
return Err(err);
|
||||
@@ -1214,10 +1217,10 @@ impl ECStore {
|
||||
.await;
|
||||
let metadata_lock_lost = bucket_lifecycle_guard.as_ref().is_some_and(|guard| guard.is_lock_lost())
|
||||
|| metadata_transaction_guard.as_ref().is_some_and(|guard| guard.is_lock_lost())
|
||||
|| ns_guard.as_ref().is_some_and(|guard| guard.is_lock_lost());
|
||||
|| ns_guard.as_deref().is_some_and(|guard| guard.is_lock_lost());
|
||||
if let Err(err) = metadata_result {
|
||||
if !metadata_lock_lost {
|
||||
self.rollback_failed_bucket_creation(bucket, bucket_lifecycle_guard.as_ref(), ns_guard.as_ref())
|
||||
self.rollback_failed_bucket_creation(bucket, bucket_lifecycle_guard.as_ref(), ns_guard.as_deref())
|
||||
.await;
|
||||
}
|
||||
return Err(err);
|
||||
@@ -1580,6 +1583,7 @@ impl ECStore {
|
||||
} else {
|
||||
None
|
||||
};
|
||||
let ns_guard = ns_guard.map(Arc::new);
|
||||
|
||||
let sr_mark_delete = opts.srdelete_op == SRBucketDeleteOp::MarkDelete;
|
||||
let sr_purge = opts.srdelete_op == SRBucketDeleteOp::Purge;
|
||||
@@ -1615,7 +1619,7 @@ impl ECStore {
|
||||
crate::bucket::metadata_sys::get_object_lock_config_and_incarnation_from_disk_with_guard_in(
|
||||
&self.ctx,
|
||||
bucket,
|
||||
Some(ns_guard.as_ref().unwrap_or(&guard)),
|
||||
Some(ns_guard.as_deref().unwrap_or(&guard)),
|
||||
)
|
||||
.await?;
|
||||
match state {
|
||||
@@ -1641,7 +1645,7 @@ impl ECStore {
|
||||
if sr_mark_delete {
|
||||
await_bucket_lifecycle_operation(
|
||||
bucket_lifecycle_guard.as_ref(),
|
||||
ns_guard.as_ref(),
|
||||
ns_guard.as_deref(),
|
||||
bucket,
|
||||
"bucket delete marker creation",
|
||||
await_bucket_namespace_operation(
|
||||
@@ -1680,7 +1684,7 @@ impl ECStore {
|
||||
object_lock_metadata_guard.as_ref().map(|(guard, _)| guard),
|
||||
bucket,
|
||||
"Object Lock configuration fence during bucket deletion",
|
||||
run_physical_bucket_deletion(ns_guard.as_ref(), bucket, async {
|
||||
run_physical_bucket_deletion(ns_guard.as_deref(), bucket, async {
|
||||
self.delete_bucket_on_sets(bucket, &delete_opts)
|
||||
.await
|
||||
.map_err(|err| to_object_err(err, vec![bucket]))
|
||||
@@ -1709,12 +1713,12 @@ impl ECStore {
|
||||
if let Some(guard) = bucket_lifecycle_guard.as_ref() {
|
||||
record_opts.add_bucket_lifecycle_lock_guard(guard);
|
||||
}
|
||||
if let Some(guard) = ns_guard.as_ref() {
|
||||
if let Some(guard) = ns_guard.as_deref() {
|
||||
record_opts.add_namespace_lock_guard(guard);
|
||||
}
|
||||
await_bucket_lifecycle_operation(
|
||||
bucket_lifecycle_guard.as_ref(),
|
||||
ns_guard.as_ref(),
|
||||
ns_guard.as_deref(),
|
||||
bucket,
|
||||
"bucket retirement publication",
|
||||
crate::bucket::retirement::commit_retirement(store.clone(), bucket, incarnation, &record_opts),
|
||||
@@ -1724,7 +1728,7 @@ impl ECStore {
|
||||
// V1 ledger must not block quota admission after same-name recreation.
|
||||
await_bucket_lifecycle_operation(
|
||||
bucket_lifecycle_guard.as_ref(),
|
||||
ns_guard.as_ref(),
|
||||
ns_guard.as_deref(),
|
||||
bucket,
|
||||
"retired bucket quota ledger cleanup",
|
||||
crate::bucket::quota::reservation::cleanup_retired_bucket_ledger(store, bucket, incarnation),
|
||||
@@ -1735,7 +1739,7 @@ impl ECStore {
|
||||
self.cleanup_bucket_usage_best_effort(bucket, ns_guard.as_ref()).await;
|
||||
|
||||
if let Err(err) = self
|
||||
.cleanup_deleted_bucket_metadata(bucket, sr_purge, ns_guard.as_ref())
|
||||
.cleanup_deleted_bucket_metadata(bucket, sr_purge, ns_guard.as_deref())
|
||||
.await
|
||||
{
|
||||
warn!(
|
||||
|
||||
@@ -527,7 +527,7 @@ impl ECStore {
|
||||
async fn acquire_heal_format_fence(
|
||||
&self,
|
||||
) -> Result<(
|
||||
tokio::sync::MutexGuard<'_, PoolMetaWriteState>,
|
||||
crate::core::pools::PoolMetaSaveGateWriteGuard<'_>,
|
||||
NamespaceLockGuard,
|
||||
NamespaceLockGuard,
|
||||
PoolMeta,
|
||||
@@ -538,7 +538,7 @@ impl ECStore {
|
||||
.first()
|
||||
.cloned()
|
||||
.ok_or_else(|| Error::other("heal format requires at least one storage pool"))?;
|
||||
let mut write_state = self.pool_meta_save_gate.lock().await;
|
||||
let mut write_state = self.pool_meta_save_gate.write().await;
|
||||
write_state.ensure_write_safe("heal format fence failed")?;
|
||||
|
||||
// Metadata fence order is part of the decommission/rebalance protocol:
|
||||
@@ -1079,7 +1079,7 @@ impl ECStore {
|
||||
pub(super) async fn handle_heal_bucket(&self, bucket: &str, opts: &HealOpts) -> Result<HealResultItem> {
|
||||
let movement_gate = self.ctx.data_movement_operation_gate();
|
||||
let _movement_guard = movement_gate.read().await;
|
||||
let save_guard = self.pool_meta_save_gate.lock().await;
|
||||
let save_guard = self.pool_meta_save_gate.write().await;
|
||||
save_guard.ensure_write_safe("bucket heal cannot run while pool metadata requires recovery")?;
|
||||
let mut fenced_pools = BTreeSet::new();
|
||||
{
|
||||
@@ -2527,7 +2527,7 @@ mod tests {
|
||||
rebalance_meta: RwLock::new(None),
|
||||
decommission_cancelers: RwLock::new(Vec::new()),
|
||||
start_gate: Mutex::new(()),
|
||||
pool_meta_save_gate: Mutex::default(),
|
||||
pool_meta_save_gate: crate::core::pools::PoolMetaSaveGate::default(),
|
||||
ctx: crate::runtime::instance::bootstrap_ctx(),
|
||||
bucket_fence_registry: std::sync::Arc::default(),
|
||||
}
|
||||
@@ -3550,10 +3550,14 @@ mod tests {
|
||||
async fn full_format_heal_preblocked_pool_metadata_never_writes_format() {
|
||||
let (temp_dir, store, shutdown) = multi_pool_heal_store().await;
|
||||
remove_heal_test_format(&temp_dir, &store, 0, 3).await;
|
||||
store.pool_meta_save_gate.lock().await.observe_replicas(PoolMetaReplicaState {
|
||||
needs_repair: true,
|
||||
repair_write_safe: false,
|
||||
});
|
||||
store
|
||||
.pool_meta_save_gate
|
||||
.write()
|
||||
.await
|
||||
.observe_replicas(PoolMetaReplicaState {
|
||||
needs_repair: true,
|
||||
repair_write_safe: false,
|
||||
});
|
||||
|
||||
let err = store
|
||||
.handle_heal_format(false)
|
||||
@@ -3688,7 +3692,7 @@ mod tests {
|
||||
let target = remove_heal_test_format(&temp_dir, &store, 0, 3).await;
|
||||
store.pools[0].disk_set[0].disks.write().await[3] = None;
|
||||
let barrier = HealFormatAfterSaveBarrier::install(&store.pools[0], 3);
|
||||
let recovery_latch = store.pool_meta_save_gate.lock().await.aborted_transaction_latch_for_test();
|
||||
let recovery_latch = store.pool_meta_save_gate.write().await.aborted_transaction_latch_for_test();
|
||||
let mut heal = tokio::spawn({
|
||||
let store = Arc::clone(&store);
|
||||
async move { store.handle_heal_format(false).await }
|
||||
@@ -3848,7 +3852,7 @@ mod tests {
|
||||
}
|
||||
}
|
||||
let endpoints: EndpointServerPools = pools.iter().map(|pool| pool.endpoints.clone()).collect::<Vec<_>>().into();
|
||||
let state = source.pool_meta_save_gate.lock().await.independent_clone_for_test();
|
||||
let state = source.pool_meta_save_gate.write().await.independent_clone_for_test();
|
||||
Arc::new(ECStore {
|
||||
id: Uuid::new_v4(),
|
||||
disk_map: Default::default(),
|
||||
@@ -3858,7 +3862,7 @@ mod tests {
|
||||
rebalance_meta: tokio::sync::RwLock::new(None),
|
||||
decommission_cancelers: tokio::sync::RwLock::new(vec![None; 2]),
|
||||
start_gate: tokio::sync::Mutex::new(()),
|
||||
pool_meta_save_gate: tokio::sync::Mutex::new(state),
|
||||
pool_meta_save_gate: crate::core::pools::PoolMetaSaveGate::new(state),
|
||||
ctx: Arc::clone(&source.ctx),
|
||||
bucket_fence_registry: Arc::default(),
|
||||
})
|
||||
@@ -4040,7 +4044,7 @@ mod tests {
|
||||
let missing = remove_pool_meta_shard(&store, 0).await;
|
||||
let (guard, admissions, mut state) = store.acquire_pool_meta_object_heal_fence(&[0]).await.expect("heal authority");
|
||||
assert!(admissions.into_iter().all(|admission| admission.is_ok()));
|
||||
assert!(store.pool_meta_save_gate.try_lock().is_err(), "heal must retain the earlier save gate");
|
||||
assert!(store.pool_meta_save_gate.try_write().is_err(), "heal must retain the earlier save gate");
|
||||
let writer_store = Arc::clone(&store);
|
||||
let writer = tokio::spawn(async move { writer_store.save_current_pool_meta_for_test(&[0]).await });
|
||||
tokio::task::yield_now().await;
|
||||
@@ -4687,10 +4691,14 @@ mod tests {
|
||||
#[tokio::test]
|
||||
async fn bucket_heal_blocks_before_dispatch_after_unreadable_pool_meta_replica() {
|
||||
let store = minimal_heal_store().await;
|
||||
store.pool_meta_save_gate.lock().await.observe_replicas(PoolMetaReplicaState {
|
||||
needs_repair: true,
|
||||
repair_write_safe: false,
|
||||
});
|
||||
store
|
||||
.pool_meta_save_gate
|
||||
.write()
|
||||
.await
|
||||
.observe_replicas(PoolMetaReplicaState {
|
||||
needs_repair: true,
|
||||
repair_write_safe: false,
|
||||
});
|
||||
|
||||
let err = store
|
||||
.handle_heal_bucket("bucket", &HealOpts::default())
|
||||
@@ -4724,7 +4732,7 @@ mod tests {
|
||||
.await
|
||||
.expect("peer dispatch should start");
|
||||
assert!(
|
||||
store.pool_meta_save_gate.try_lock().is_ok(),
|
||||
store.pool_meta_save_gate.try_write().is_ok(),
|
||||
"coordinator must release its local save gate before waiting for peers"
|
||||
);
|
||||
assert!(
|
||||
@@ -5116,7 +5124,7 @@ mod tests {
|
||||
rebalance_meta: RwLock::new(None),
|
||||
decommission_cancelers: RwLock::new(Vec::new()),
|
||||
start_gate: Mutex::new(()),
|
||||
pool_meta_save_gate: Mutex::default(),
|
||||
pool_meta_save_gate: crate::core::pools::PoolMetaSaveGate::default(),
|
||||
ctx: crate::runtime::instance::bootstrap_ctx(),
|
||||
bucket_fence_registry: std::sync::Arc::default(),
|
||||
};
|
||||
@@ -5136,7 +5144,7 @@ mod tests {
|
||||
.save_for_startup(store.pools.clone())
|
||||
.await
|
||||
.expect("pool metadata should be persisted before format heal");
|
||||
*store.pool_meta_save_gate.lock().await = PoolMetaWriteState::default();
|
||||
*store.pool_meta_save_gate.write().await = crate::core::pools::PoolMetaWriteState::default();
|
||||
|
||||
let replacement_endpoint = store.pools[1].endpoints.endpoints.as_ref()[2].clone();
|
||||
tokio::fs::remove_file(
|
||||
|
||||
@@ -676,7 +676,7 @@ impl ECStore {
|
||||
rebalance_meta: RwLock::new(None),
|
||||
decommission_cancelers,
|
||||
start_gate: Mutex::new(()),
|
||||
pool_meta_save_gate: Mutex::new(pool_meta_write_state),
|
||||
pool_meta_save_gate: crate::core::pools::PoolMetaSaveGate::new(pool_meta_write_state),
|
||||
// Adopt the caller's context (the process bootstrap one on the
|
||||
// legacy path) so startup writes (erasure type recorded before
|
||||
// this point) and later reads share one cell.
|
||||
@@ -725,7 +725,7 @@ impl ECStore {
|
||||
// Lock order is save gate -> distributed metadata writer, matching
|
||||
// runtime mutations. RPC locks are available before ECStore publication
|
||||
// and do not depend on the later fleet proof.
|
||||
let mut startup_write_state = self.pool_meta_save_gate.lock().await;
|
||||
let mut startup_write_state = self.pool_meta_save_gate.write().await;
|
||||
let startup_pool_meta_guard = if should_persist_pool_meta {
|
||||
let pool = self
|
||||
.pools
|
||||
|
||||
@@ -127,28 +127,30 @@ impl AuthoritativeListOperations for ECStore {
|
||||
marker: Option<String>,
|
||||
max_keys: i32,
|
||||
) -> Result<ListObjectVersionsInfo> {
|
||||
if max_keys <= 0 || max_keys > MAX_OBJECT_LIST {
|
||||
self.list_unversioned_authoritative_page(bucket, prefix, marker, max_keys, None)
|
||||
.await
|
||||
}
|
||||
|
||||
async fn list_identifier_tree_leaves_authoritative(
|
||||
self: Arc<Self>,
|
||||
bucket: &str,
|
||||
prefix: &str,
|
||||
leaf: &str,
|
||||
marker: Option<String>,
|
||||
max_keys: i32,
|
||||
) -> Result<ListObjectVersionsInfo> {
|
||||
let valid_prefix = prefix
|
||||
.strip_suffix('/')
|
||||
.is_some_and(|root| !root.is_empty() && root.split('/').all(|part| !part.is_empty() && !matches!(part, "." | "..")));
|
||||
if !valid_prefix || !crate::disk::is_identifier_tree_leaf(leaf) {
|
||||
return Err(StorageError::InvalidArgument(
|
||||
bucket.to_string(),
|
||||
prefix.to_string(),
|
||||
"authoritative page size is out of range".to_string(),
|
||||
"identifier-tree listing requires a directory prefix and a dotted leaf name".to_string(),
|
||||
));
|
||||
}
|
||||
let opts = ListPathOptions {
|
||||
bucket: bucket.to_string(),
|
||||
prefix: prefix.to_string(),
|
||||
marker: marker.clone(),
|
||||
limit: max_keys,
|
||||
recursive: true,
|
||||
incl_deleted: true,
|
||||
versioned: true,
|
||||
authoritative: true,
|
||||
caller: ListObjectsGatherCaller::S3ListVersions,
|
||||
..Default::default()
|
||||
};
|
||||
let mut page = self.list_path(&opts).await?;
|
||||
page.entries.forward_past(marker);
|
||||
unversioned_authoritative_page(bucket, page)
|
||||
self.list_unversioned_authoritative_page(bucket, prefix, marker, max_keys, Some(leaf.to_string()))
|
||||
.await
|
||||
}
|
||||
|
||||
async fn walk_authoritative(
|
||||
@@ -436,6 +438,8 @@ pub struct ListPathOptions {
|
||||
// Internal bounded pages require GET-equivalent per-set resolution.
|
||||
pub authoritative: bool,
|
||||
|
||||
pub identifier_tree_leaf: Option<String>,
|
||||
|
||||
pub stop_disk_at_limit: bool,
|
||||
|
||||
pub pool_idx: Option<usize>,
|
||||
@@ -1101,6 +1105,39 @@ impl ListPathOptions {
|
||||
}
|
||||
|
||||
impl ECStore {
|
||||
async fn list_unversioned_authoritative_page(
|
||||
self: Arc<Self>,
|
||||
bucket: &str,
|
||||
prefix: &str,
|
||||
marker: Option<String>,
|
||||
max_keys: i32,
|
||||
identifier_tree_leaf: Option<String>,
|
||||
) -> Result<ListObjectVersionsInfo> {
|
||||
if max_keys <= 0 || max_keys > MAX_OBJECT_LIST {
|
||||
return Err(StorageError::InvalidArgument(
|
||||
bucket.to_string(),
|
||||
prefix.to_string(),
|
||||
"authoritative page size is out of range".to_string(),
|
||||
));
|
||||
}
|
||||
let opts = ListPathOptions {
|
||||
bucket: bucket.to_string(),
|
||||
prefix: prefix.to_string(),
|
||||
marker: marker.clone(),
|
||||
limit: max_keys,
|
||||
recursive: true,
|
||||
incl_deleted: true,
|
||||
versioned: true,
|
||||
authoritative: true,
|
||||
identifier_tree_leaf,
|
||||
caller: ListObjectsGatherCaller::S3ListVersions,
|
||||
..Default::default()
|
||||
};
|
||||
let mut page = self.list_path(&opts).await?;
|
||||
page.entries.forward_past(marker);
|
||||
unversioned_authoritative_page(bucket, page)
|
||||
}
|
||||
|
||||
#[allow(clippy::too_many_arguments)]
|
||||
// @continuation_token marker
|
||||
// @start_after as marker when continuation_token empty
|
||||
@@ -2936,6 +2973,33 @@ impl SetDisks {
|
||||
Ok(page)
|
||||
}
|
||||
|
||||
fn resolve_authoritative_listed_entry(
|
||||
bucket: &str,
|
||||
selection: Option<(&str, &str)>,
|
||||
entries: MetaCacheEntries,
|
||||
errors: &[Option<DiskError>],
|
||||
disk_count: usize,
|
||||
parity: usize,
|
||||
) -> crate::disk::error::Result<Option<MetaCacheEntry>> {
|
||||
if disk_count == 0 || entries.0.len() > disk_count || entries.0.len() != errors.len() {
|
||||
return Err(DiskError::ErasureReadQuorum);
|
||||
}
|
||||
// An old peer may ignore the optional disk selector. Its noncandidate
|
||||
// rows must be ignored before metadata parsing. Selected keys retain
|
||||
// the original resolver and all of its quorum/corruption checks.
|
||||
if selection.is_some_and(|(prefix, leaf)| {
|
||||
entries
|
||||
.0
|
||||
.iter()
|
||||
.flatten()
|
||||
.next()
|
||||
.is_some_and(|entry| !crate::disk::identifier_tree_leaf_matches(prefix, leaf, &entry.name))
|
||||
}) {
|
||||
return Ok(None);
|
||||
}
|
||||
Self::resolve_listed_versions(bucket, entries, errors, disk_count, parity)
|
||||
}
|
||||
|
||||
/// Version scan for set-scoped callers that act on absence: set walks
|
||||
/// (tier reference proofs, tier-delete journals, pool maintenance) and the
|
||||
/// recursive prefix-delete scan. Every online drive is read and each
|
||||
@@ -2959,6 +3023,8 @@ impl SetDisks {
|
||||
let first_error = Arc::new(tokio::sync::Mutex::new(None));
|
||||
let error_sink = Arc::clone(&first_error);
|
||||
let bucket = opts.bucket.clone();
|
||||
let prefix = opts.prefix.clone();
|
||||
let identifier_tree_leaf = opts.identifier_tree_leaf.clone();
|
||||
let cancel = rx.clone();
|
||||
let result = list_path_raw_with_claim_tracker(
|
||||
rx,
|
||||
@@ -2976,6 +3042,7 @@ impl SetDisks {
|
||||
forward_to: opts.marker,
|
||||
min_disks: read_quorum,
|
||||
preserve_replica_metadata: true,
|
||||
identifier_tree_leaf: opts.identifier_tree_leaf,
|
||||
// A reader's local page may be consumed by stale entries. Only
|
||||
// the resolved, merged page may stop the authoritative walk.
|
||||
per_disk_limit: 0,
|
||||
@@ -2983,7 +3050,14 @@ impl SetDisks {
|
||||
walkdir_timeout: opts.walkdir_timeout,
|
||||
walkdir_stall_timeout: opts.walkdir_stall_timeout,
|
||||
partial: Some(Box::new(move |entries, errors| {
|
||||
let resolved = Self::resolve_listed_versions(&bucket, entries, errors, disk_count, parity);
|
||||
let resolved = Self::resolve_authoritative_listed_entry(
|
||||
&bucket,
|
||||
identifier_tree_leaf.as_deref().map(|leaf| (prefix.as_str(), leaf)),
|
||||
entries,
|
||||
errors,
|
||||
disk_count,
|
||||
parity,
|
||||
);
|
||||
let sender = sender.clone();
|
||||
let cancel = cancel.clone();
|
||||
let first_error = Arc::clone(&error_sink);
|
||||
@@ -6997,6 +7071,155 @@ mod test {
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn authoritative_identifier_tree_filters_old_peer_rows_before_metadata_resolution() {
|
||||
let prefix = "table-catalog/namespaces/";
|
||||
let leaf = "table-entry.json";
|
||||
let corrupt = |name: &str| {
|
||||
MetaCacheEntries(vec![
|
||||
Some(MetaCacheEntry {
|
||||
name: format!("{prefix}{name}"),
|
||||
metadata: b"corrupt metadata".to_vec(),
|
||||
..Default::default()
|
||||
});
|
||||
4
|
||||
])
|
||||
};
|
||||
let report = "sales/tables/orders/maintenance/hash/jobs/job.json";
|
||||
let errors = vec![None; 4];
|
||||
assert!(
|
||||
SetDisks::resolve_authoritative_listed_entry(RUSTFS_META_BUCKET, None, corrupt(report), &errors, 4, 2,).is_err(),
|
||||
"the control must actually parse and reject the corrupt metadata"
|
||||
);
|
||||
assert!(
|
||||
SetDisks::resolve_authoritative_listed_entry(
|
||||
RUSTFS_META_BUCKET,
|
||||
Some((prefix, leaf)),
|
||||
corrupt(report),
|
||||
&errors,
|
||||
4,
|
||||
2,
|
||||
)
|
||||
.expect("an old peer's unrelated report is outside the selected scan")
|
||||
.is_none()
|
||||
);
|
||||
assert!(
|
||||
SetDisks::resolve_authoritative_listed_entry(
|
||||
RUSTFS_META_BUCKET,
|
||||
Some((prefix, leaf)),
|
||||
corrupt("sales/tables/orders/maintenance/tables/owner/table-entry.json"),
|
||||
&errors,
|
||||
4,
|
||||
2,
|
||||
)
|
||||
.is_err(),
|
||||
"a matching nested owner must still fail closed on corruption"
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn authoritative_identifier_tree_pages_preserve_nested_owners_and_read_quorum() {
|
||||
let (dirs, store) = single_set_store(4).await;
|
||||
let prefix = "table-catalog/identifier-tree/";
|
||||
let leaf = "table-entry.json";
|
||||
for (root, selected) in [
|
||||
("table-catalog/identifier-tree", leaf),
|
||||
(prefix, "../entry.json"),
|
||||
(prefix, "table-entry"),
|
||||
] {
|
||||
assert!(matches!(
|
||||
store
|
||||
.clone()
|
||||
.list_identifier_tree_leaves_authoritative(RUSTFS_META_BUCKET, root, selected, None, 2)
|
||||
.await,
|
||||
Err(StorageError::InvalidArgument(..))
|
||||
));
|
||||
}
|
||||
let names = [
|
||||
format!("{prefix}a/tables/first/{leaf}"),
|
||||
format!("{prefix}sales/tables/orders/maintenance/tables/owner/{leaf}"),
|
||||
format!("{prefix}z/tables/last/{leaf}"),
|
||||
];
|
||||
for name in &names {
|
||||
write_version_replicas(&dirs, RUSTFS_META_BUCKET, name, |slot| Some(unversioned_catalog_metadata(name, slot))).await;
|
||||
}
|
||||
for name in [
|
||||
format!("{prefix}sales"),
|
||||
format!("{prefix}sales/tables/orders/maintenance/hash/jobs/job.json"),
|
||||
] {
|
||||
write_version_replicas(&dirs, RUSTFS_META_BUCKET, &name, |_| Some(b"corrupt noncandidate metadata".to_vec())).await;
|
||||
}
|
||||
assert!(
|
||||
store
|
||||
.clone()
|
||||
.list_unversioned_objects_authoritative(RUSTFS_META_BUCKET, prefix, None, 100)
|
||||
.await
|
||||
.is_err(),
|
||||
"the ordinary authoritative scan must expose the corrupt control objects"
|
||||
);
|
||||
let first = store
|
||||
.clone()
|
||||
.list_identifier_tree_leaves_authoritative(RUSTFS_META_BUCKET, prefix, leaf, None, 2)
|
||||
.await
|
||||
.expect("noncandidate metadata must not be read");
|
||||
assert!(first.is_truncated);
|
||||
assert_eq!(
|
||||
first.objects.iter().map(|object| &object.name).collect::<Vec<_>>(),
|
||||
names[..2].iter().collect::<Vec<_>>()
|
||||
);
|
||||
let second = store
|
||||
.clone()
|
||||
.list_identifier_tree_leaves_authoritative(RUSTFS_META_BUCKET, prefix, leaf, first.next_marker, 2)
|
||||
.await
|
||||
.expect("the next page must retain the final owner");
|
||||
assert!(!second.is_truncated);
|
||||
assert_eq!(second.objects.iter().map(|object| &object.name).collect::<Vec<_>>(), vec![&names[2]]);
|
||||
|
||||
write_version_replicas(&dirs, RUSTFS_META_BUCKET, &names[0], |_| Some(b"corrupt selected metadata".to_vec())).await;
|
||||
assert!(
|
||||
store
|
||||
.clone()
|
||||
.list_identifier_tree_leaves_authoritative(RUSTFS_META_BUCKET, prefix, leaf, None, 2)
|
||||
.await
|
||||
.is_err(),
|
||||
"corruption at a selected key cannot be skipped to fill the page"
|
||||
);
|
||||
write_version_replicas(&dirs, RUSTFS_META_BUCKET, &names[0], |slot| {
|
||||
Some(unversioned_catalog_metadata(&names[0], slot))
|
||||
})
|
||||
.await;
|
||||
write_version_replicas(&dirs, RUSTFS_META_BUCKET, &names[0], |slot| {
|
||||
Some(versioned_object_metadata(RUSTFS_META_BUCKET, &names[0], &[(2, 1_700_000_002)], slot).to_vec())
|
||||
})
|
||||
.await;
|
||||
assert!(
|
||||
store
|
||||
.clone()
|
||||
.list_identifier_tree_leaves_authoritative(RUSTFS_META_BUCKET, prefix, leaf, None, 2)
|
||||
.await
|
||||
.is_err(),
|
||||
"a versioned ownership entry cannot be accepted as an unversioned proof"
|
||||
);
|
||||
write_version_replicas(&dirs, RUSTFS_META_BUCKET, &names[0], |slot| {
|
||||
Some(unversioned_catalog_metadata(&names[0], slot))
|
||||
})
|
||||
.await;
|
||||
let set = &store.pools[0].disk_set[0];
|
||||
for disk in set.disks.write().await.iter_mut().skip(1) {
|
||||
*disk = None;
|
||||
}
|
||||
assert!(
|
||||
matches!(
|
||||
store
|
||||
.clone()
|
||||
.list_identifier_tree_leaves_authoritative(RUSTFS_META_BUCKET, prefix, leaf, None, 2)
|
||||
.await,
|
||||
Err(StorageError::ErasureReadQuorum)
|
||||
),
|
||||
"unavailable replicas cannot prove owner absence"
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn authoritative_store_walk_includes_reserved_prefix_versions() {
|
||||
let bucket = "catalog-cleanup-versions";
|
||||
|
||||
@@ -34,7 +34,7 @@ use crate::bucket::utils::check_put_object_part_args;
|
||||
use crate::bucket::utils::{check_valid_bucket_name, check_valid_bucket_name_strict, is_meta_bucketname};
|
||||
use crate::cluster::rpc::{RemoteClient, S3PeerSys};
|
||||
use crate::config::storageclass;
|
||||
use crate::core::pools::{DecommissionCanceler, PoolMeta, PoolMetaWriteState};
|
||||
use crate::core::pools::{DecommissionCanceler, PoolMeta, PoolMetaSaveGate};
|
||||
use crate::disk::endpoint::{Endpoint, EndpointType};
|
||||
use crate::disk::{DiskAPI, DiskInfo, DiskInfoOptions};
|
||||
use crate::error::{Error, Result};
|
||||
@@ -480,9 +480,13 @@ pub struct ECStore {
|
||||
/// Serializes full-document pool metadata saves and retains a fail-closed
|
||||
/// write block after startup observes an unreadable replica.
|
||||
///
|
||||
/// Immutable admission proofs share authority; publication, recovery and
|
||||
/// authority changes retain exclusive access through their existing I/O.
|
||||
/// Each writer publishes its final state on release, and health
|
||||
/// inspection reads that committed state without acquiring the gate.
|
||||
/// Lock order: acquire `pool_meta_save_gate`, then the distributed
|
||||
/// `pool.bin` fence, then clone `pool_meta` under a short read lock.
|
||||
pub(crate) pool_meta_save_gate: Mutex<PoolMetaWriteState>,
|
||||
pub(crate) pool_meta_save_gate: PoolMetaSaveGate,
|
||||
/// Per-instance runtime state (Phase 5, backlog#939).
|
||||
///
|
||||
/// Carries this instance's identity/runtime out of the process globals so
|
||||
@@ -555,7 +559,6 @@ impl Default for ScannerDataMovementPauseStatus {
|
||||
#[derive(Clone, Debug, PartialEq, Eq, serde::Serialize)]
|
||||
pub struct PoolMetaWriteGateStatus {
|
||||
pub writes_ready: bool,
|
||||
pub check_timed_out: bool,
|
||||
pub write_blocked: bool,
|
||||
pub transaction_aborted: bool,
|
||||
pub pool_meta_absent: bool,
|
||||
@@ -571,7 +574,6 @@ impl Default for PoolMetaWriteGateStatus {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
writes_ready: true,
|
||||
check_timed_out: false,
|
||||
write_blocked: false,
|
||||
transaction_aborted: false,
|
||||
pool_meta_absent: false,
|
||||
@@ -1118,6 +1120,20 @@ impl ECStore {
|
||||
Some(epoch)
|
||||
}
|
||||
|
||||
/// Namespace cleanup edits an existing usage baseline; it does not certify
|
||||
/// a scan. Retain the movement fence without requiring scanning to resume.
|
||||
pub(crate) async fn data_usage_namespace_cleanup_guard(&self) -> Option<(tokio::sync::OwnedRwLockReadGuard<()>, u64)> {
|
||||
let operation_guard = self.ctx.data_movement_operation_gate().read_owned().await;
|
||||
if self.ctx.data_movement_operation_epoch_exhausted() || self.ctx.data_movement_generation_exhausted() {
|
||||
return None;
|
||||
}
|
||||
Some((operation_guard, self.ctx.data_movement_operation_epoch()))
|
||||
}
|
||||
|
||||
pub(crate) async fn data_usage_namespace_cleanup_epoch(&self) -> Option<u64> {
|
||||
self.data_usage_namespace_cleanup_guard().await.map(|(_, epoch)| epoch)
|
||||
}
|
||||
|
||||
/// Acquire a storage-owned read admission for a coordinator's final
|
||||
/// scanner publication. The guard remains in the context's lease table,
|
||||
/// so a local movement writer cannot pass the peer while its authoritative
|
||||
@@ -1936,7 +1952,7 @@ mod tests {
|
||||
rebalance_meta: RwLock::new(None),
|
||||
decommission_cancelers: RwLock::new(Vec::new()),
|
||||
start_gate: Mutex::new(()),
|
||||
pool_meta_save_gate: Mutex::default(),
|
||||
pool_meta_save_gate: crate::core::pools::PoolMetaSaveGate::default(),
|
||||
ctx,
|
||||
bucket_fence_registry: Arc::default(),
|
||||
};
|
||||
@@ -2011,7 +2027,7 @@ mod tests {
|
||||
rebalance_meta: RwLock::new(None),
|
||||
decommission_cancelers: RwLock::new(Vec::new()),
|
||||
start_gate: Mutex::new(()),
|
||||
pool_meta_save_gate: Mutex::default(),
|
||||
pool_meta_save_gate: crate::core::pools::PoolMetaSaveGate::default(),
|
||||
ctx,
|
||||
bucket_fence_registry: Arc::default(),
|
||||
})
|
||||
|
||||
@@ -2133,7 +2133,7 @@ mod tests {
|
||||
rebalance_meta: RwLock::new(None),
|
||||
decommission_cancelers: RwLock::new(Vec::new()),
|
||||
start_gate: Mutex::new(()),
|
||||
pool_meta_save_gate: Mutex::default(),
|
||||
pool_meta_save_gate: crate::core::pools::PoolMetaSaveGate::default(),
|
||||
ctx: crate::runtime::instance::bootstrap_ctx(),
|
||||
bucket_fence_registry: std::sync::Arc::default(),
|
||||
}
|
||||
|
||||
@@ -4937,6 +4937,87 @@ impl ECStore {
|
||||
Ok((idx, result))
|
||||
}
|
||||
|
||||
/// Compare the logical revision across all pools before publishing a
|
||||
/// namespace-only usage update to a writable pool. A suspended source is
|
||||
/// retained for decommission to reconcile against the newer target.
|
||||
pub(crate) async fn put_usage_snapshot_for_namespace_cleanup(
|
||||
&self,
|
||||
object: &str,
|
||||
expected_revision: Option<&str>,
|
||||
data: &mut PutObjReader,
|
||||
namespace_guard: Option<&rustfs_lock::NamespaceLockGuard>,
|
||||
) -> Result<ObjectInfo> {
|
||||
// The caller owns the bucket namespace and movement guards. Acquire
|
||||
// the fixed object domain, then every physical domain in pool order.
|
||||
let guards = self
|
||||
.acquire_all_physical_object_write_locks("usage_namespace_cleanup", RUSTFS_META_BUCKET, object)
|
||||
.await?;
|
||||
let mut opts = ObjectOptions {
|
||||
no_lock: true,
|
||||
max_parity: true,
|
||||
write_completion: crate::object_api::WriteCompletion::TailDrained,
|
||||
suppress_read_repair: true,
|
||||
..Default::default()
|
||||
};
|
||||
for guard in &guards {
|
||||
guard.add_namespace_lock_fence(&mut opts);
|
||||
}
|
||||
if let Some(guard) = namespace_guard {
|
||||
opts.add_namespace_lock_guard(guard);
|
||||
}
|
||||
let source = match self
|
||||
.get_latest_object_info_for_conditional_write(RUSTFS_META_BUCKET, object, &opts)
|
||||
.await
|
||||
.and_then(|(info, idx)| resolve_latest_object_access(RUSTFS_META_BUCKET, object, info, idx, &opts))
|
||||
{
|
||||
Ok((info, _)) => Some(info),
|
||||
Err(err) if is_err_object_not_found(&err) || is_err_version_not_found(&err) => None,
|
||||
Err(err) => return Err(err),
|
||||
};
|
||||
if source.as_ref().and_then(|info| info.etag.as_deref()) != expected_revision {
|
||||
return Err(StorageError::PreconditionFailed);
|
||||
}
|
||||
if source.as_ref().is_some_and(|info| {
|
||||
info.version_id.is_some_and(|version| !version.is_nil()) || info.etag.as_ref().is_none_or(String::is_empty)
|
||||
}) {
|
||||
return Err(StorageError::other("usage cleanup requires an unversioned snapshot with a revision"));
|
||||
}
|
||||
let target = self.get_pool_idx_no_lock(RUSTFS_META_BUCKET, object, data.size()).await?;
|
||||
let target_info = match self.pools[target].get_object_info(RUSTFS_META_BUCKET, object, &opts).await {
|
||||
Ok(info) => Some(info),
|
||||
Err(err) if is_err_object_not_found(&err) || is_err_version_not_found(&err) => None,
|
||||
Err(err) => return Err(err),
|
||||
};
|
||||
opts.http_preconditions = Some(match target_info.as_ref() {
|
||||
Some(info) => crate::storage_api_contracts::object::HTTPPreconditions {
|
||||
if_match: Some(
|
||||
info.etag
|
||||
.as_ref()
|
||||
.filter(|etag| !etag.is_empty())
|
||||
.cloned()
|
||||
.ok_or_else(|| StorageError::other("usage cleanup target has no revision"))?,
|
||||
),
|
||||
..Default::default()
|
||||
},
|
||||
None => crate::storage_api_contracts::object::HTTPPreconditions {
|
||||
if_none_match: Some("*".to_string()),
|
||||
..Default::default()
|
||||
},
|
||||
});
|
||||
let latest = source
|
||||
.iter()
|
||||
.chain(target_info.iter())
|
||||
.filter_map(|info| info.mod_time)
|
||||
.fold(OffsetDateTime::now_utc(), std::cmp::max);
|
||||
opts.mod_time = Some(
|
||||
latest
|
||||
.checked_add(time::Duration::nanoseconds(1))
|
||||
.ok_or_else(|| StorageError::other("usage cleanup snapshot timestamp is exhausted"))?,
|
||||
);
|
||||
opts.decommission_capacity_admission = crate::bucket::metadata_sys::object_store_if_initialized_in(&self.ctx).await;
|
||||
self.pools[target].put_object(RUSTFS_META_BUCKET, object, data, &opts).await
|
||||
}
|
||||
|
||||
#[instrument(level = "debug", skip(self, data))]
|
||||
#[hotpath::measure(impl_type = "ECStore")]
|
||||
pub(super) async fn handle_put_object(
|
||||
@@ -7005,7 +7086,7 @@ mod tests {
|
||||
};
|
||||
use crate::set_disk::same_distributed_lock_domain;
|
||||
use crate::set_disk::{SetDisks, disk_call_counters};
|
||||
use crate::storage_api_contracts::bucket::MakeBucketOptions;
|
||||
use crate::storage_api_contracts::bucket::{BucketOptions, MakeBucketOptions};
|
||||
use crate::storage_api_contracts::lifecycle::TransitionedObject;
|
||||
use crate::storage_api_contracts::list::ListOperations as _;
|
||||
use bytes::Bytes;
|
||||
@@ -10003,7 +10084,7 @@ mod tests {
|
||||
rebalance_meta: RwLock::new(None),
|
||||
decommission_cancelers: RwLock::new(Vec::new()),
|
||||
start_gate: Mutex::new(()),
|
||||
pool_meta_save_gate: Mutex::default(),
|
||||
pool_meta_save_gate: crate::core::pools::PoolMetaSaveGate::default(),
|
||||
ctx: crate::runtime::instance::bootstrap_ctx(),
|
||||
bucket_fence_registry: std::sync::Arc::default(),
|
||||
}
|
||||
@@ -10095,7 +10176,7 @@ mod tests {
|
||||
rebalance_meta: RwLock::new(None),
|
||||
decommission_cancelers: RwLock::new(Vec::new()),
|
||||
start_gate: Mutex::new(()),
|
||||
pool_meta_save_gate: Mutex::default(),
|
||||
pool_meta_save_gate: crate::core::pools::PoolMetaSaveGate::default(),
|
||||
ctx,
|
||||
bucket_fence_registry: std::sync::Arc::default(),
|
||||
}
|
||||
@@ -10347,6 +10428,296 @@ mod tests {
|
||||
multipool_delete_test_store(bucket, true).await
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn create_fresh_bucket_after_canceled_decommission() {
|
||||
let (_dirs, store) = multipool_version_test_store("canceled-decommission-existing").await;
|
||||
let mut canceled = prepared_pool_test_status(1, false);
|
||||
canceled.decommission = Some(PoolDecommissionInfo {
|
||||
canceled: true,
|
||||
..Default::default()
|
||||
});
|
||||
*store.pool_meta.write().await = PoolMeta {
|
||||
pools: vec![prepared_pool_test_status(0, false), canceled],
|
||||
..Default::default()
|
||||
};
|
||||
store
|
||||
.handle_make_bucket("canceled-decommission-fresh", &MakeBucketOptions::default())
|
||||
.await
|
||||
.expect("canceled decommission must not block creation of a new bucket");
|
||||
assert!(
|
||||
store.scanner_data_usage_publication_blocked().await,
|
||||
"scanner publication must remain fenced"
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn create_bucket_after_failed_decommission() {
|
||||
let (_dirs, store) = multipool_version_test_store("decommission-existing-bucket").await;
|
||||
let bucket = "decommission-new-bucket";
|
||||
let snapshot_time = std::time::SystemTime::now() - Duration::from_secs(60);
|
||||
let mut snapshot = rustfs_data_usage::DataUsageInfo {
|
||||
last_update: Some(snapshot_time),
|
||||
scanner_cycle: Some(37),
|
||||
scanner_epoch: Some(2),
|
||||
buckets_count: 1,
|
||||
usage_snapshot_complete: true,
|
||||
..Default::default()
|
||||
};
|
||||
snapshot.buckets_usage.insert(
|
||||
bucket.to_string(),
|
||||
rustfs_data_usage::BucketUsageInfo {
|
||||
size: 42,
|
||||
objects_count: 1,
|
||||
versions_count: 1,
|
||||
..Default::default()
|
||||
},
|
||||
);
|
||||
snapshot.bucket_sizes.insert(bucket.to_string(), 42);
|
||||
snapshot.calculate_totals();
|
||||
crate::data_usage::store_data_usage_in_backend(snapshot, store.clone())
|
||||
.await
|
||||
.expect("store a stale bucket-usage baseline before the failed decommission");
|
||||
let (old_snapshot, usage_owner) = store
|
||||
.get_latest_accessible_object_info_with_idx(
|
||||
RUSTFS_META_BUCKET,
|
||||
crate::data_usage::DATA_USAGE_OBJ_NAME_PATH.as_str(),
|
||||
&ObjectOptions::default(),
|
||||
)
|
||||
.await
|
||||
.expect("identify the pool holding the old usage baseline");
|
||||
let mut pool_statuses = vec![prepared_pool_test_status(0, false), prepared_pool_test_status(1, false)];
|
||||
pool_statuses[usage_owner].decommission = Some(PoolDecommissionInfo {
|
||||
failed: true,
|
||||
..Default::default()
|
||||
});
|
||||
*store.pool_meta.write().await = PoolMeta {
|
||||
pools: pool_statuses,
|
||||
..Default::default()
|
||||
};
|
||||
store
|
||||
.handle_make_bucket(bucket, &MakeBucketOptions::default())
|
||||
.await
|
||||
.expect("a failed decommission must not block creation of an unrelated bucket");
|
||||
store
|
||||
.handle_get_bucket_info(bucket, &BucketOptions::default())
|
||||
.await
|
||||
.expect("the created bucket must be visible");
|
||||
let usage = crate::data_usage::load_data_usage_from_backend(store.clone())
|
||||
.await
|
||||
.expect("load the cleaned usage baseline");
|
||||
assert!(!usage.buckets_usage.contains_key(bucket), "the new bucket must not inherit old usage");
|
||||
assert_eq!(usage.scanner_cycle, Some(37), "namespace cleanup must not certify a new scan");
|
||||
assert_eq!(usage.scanner_epoch, Some(2), "namespace cleanup must preserve the scanner authority");
|
||||
let object = crate::data_usage::DATA_USAGE_OBJ_NAME_PATH.as_str();
|
||||
let (_, current_owner) = store
|
||||
.get_latest_accessible_object_info_with_idx(RUSTFS_META_BUCKET, object, &ObjectOptions::default())
|
||||
.await
|
||||
.expect("read the new snapshot authority");
|
||||
assert_ne!(current_owner, usage_owner, "cleanup must publish to a writable pool");
|
||||
let retained_source = store.pools[usage_owner]
|
||||
.get_object_info(RUSTFS_META_BUCKET, object, &ObjectOptions::default())
|
||||
.await
|
||||
.expect("decommission retains its original source for reconciliation");
|
||||
assert_eq!(retained_source.etag, old_snapshot.etag);
|
||||
let stale_update = store
|
||||
.put_usage_snapshot_for_namespace_cleanup(
|
||||
object,
|
||||
old_snapshot.etag.as_deref(),
|
||||
&mut PutObjReader::from_vec(b"stale snapshot".to_vec()),
|
||||
None,
|
||||
)
|
||||
.await;
|
||||
assert!(matches!(stale_update, Err(StorageError::PreconditionFailed)));
|
||||
let usage = crate::data_usage::load_data_usage_from_backend(store.clone())
|
||||
.await
|
||||
.expect("a stale source CAS must not replace the writable target");
|
||||
assert!(!usage.buckets_usage.contains_key(bucket));
|
||||
assert!(
|
||||
store.scanner_data_usage_publication_blocked().await,
|
||||
"scanner publication must remain fenced"
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn canceled_create_retains_usage_cleanup_namespace_owner() {
|
||||
use crate::set_disk::{PutObjectCommitBarrier, PutObjectCommitPause};
|
||||
|
||||
let (_dirs, store) = multipool_version_test_store("cleanup-cancellation-existing").await;
|
||||
let bucket = "cleanup-cancellation-new";
|
||||
let mut snapshot = rustfs_data_usage::DataUsageInfo {
|
||||
last_update: Some(std::time::SystemTime::now()),
|
||||
usage_snapshot_complete: true,
|
||||
..Default::default()
|
||||
};
|
||||
snapshot.buckets_usage.insert(
|
||||
bucket.to_owned(),
|
||||
rustfs_data_usage::BucketUsageInfo {
|
||||
size: 42,
|
||||
objects_count: 1,
|
||||
versions_count: 1,
|
||||
..Default::default()
|
||||
},
|
||||
);
|
||||
snapshot.bucket_sizes.insert(bucket.to_owned(), 42);
|
||||
snapshot.calculate_totals();
|
||||
crate::data_usage::store_data_usage_in_backend(snapshot, store.clone())
|
||||
.await
|
||||
.expect("seed the stale namespace usage");
|
||||
let object = crate::data_usage::DATA_USAGE_OBJ_NAME_PATH.as_str();
|
||||
let (_, source) = store
|
||||
.get_latest_accessible_object_info_with_idx(RUSTFS_META_BUCKET, object, &ObjectOptions::default())
|
||||
.await
|
||||
.expect("find the source snapshot");
|
||||
let mut pools = vec![prepared_pool_test_status(0, false), prepared_pool_test_status(1, false)];
|
||||
pools[source].decommission = Some(PoolDecommissionInfo {
|
||||
canceled: true,
|
||||
..Default::default()
|
||||
});
|
||||
*store.pool_meta.write().await = PoolMeta {
|
||||
pools,
|
||||
..Default::default()
|
||||
};
|
||||
let barrier = PutObjectCommitBarrier::install(RUSTFS_META_BUCKET, object, PutObjectCommitPause::AfterNamespace);
|
||||
let create_store = store.clone();
|
||||
let create = tokio::spawn(async move { create_store.handle_make_bucket(bucket, &MakeBucketOptions::default()).await });
|
||||
tokio::time::timeout(Duration::from_secs(10), barrier.wait_until_paused())
|
||||
.await
|
||||
.expect("cleanup reaches its physical commit with the namespace held");
|
||||
create.abort();
|
||||
assert!(create.await.expect_err("the request waiter was canceled").is_cancelled());
|
||||
let namespace = store
|
||||
.new_ns_lock(bucket, bucket)
|
||||
.await
|
||||
.expect("open the bucket namespace lock");
|
||||
assert!(
|
||||
tokio::time::timeout(Duration::from_millis(50), namespace.get_write_lock(Duration::from_secs(1)))
|
||||
.await
|
||||
.is_err(),
|
||||
"cancellation must not release the namespace before its cleanup finishes"
|
||||
);
|
||||
barrier.release();
|
||||
drop(barrier);
|
||||
tokio::time::timeout(Duration::from_secs(10), store.handle_make_bucket(bucket, &MakeBucketOptions::default()))
|
||||
.await
|
||||
.expect("the resumed create must not remain blocked")
|
||||
.expect("a new request resumes after the old cleanup releases its namespace");
|
||||
let usage = crate::data_usage::load_data_usage_from_backend(store.clone())
|
||||
.await
|
||||
.expect("read the cleaned baseline");
|
||||
assert!(!usage.buckets_usage.contains_key(bucket));
|
||||
assert!(store.scanner_data_usage_publication_blocked().await);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn usage_cleanup_rejects_an_unreadable_pool() {
|
||||
let (_dirs, store) = multipool_version_test_store("cleanup-unreadable-pool").await;
|
||||
let snapshot = rustfs_data_usage::DataUsageInfo {
|
||||
last_update: Some(std::time::SystemTime::now()),
|
||||
usage_snapshot_complete: true,
|
||||
..Default::default()
|
||||
};
|
||||
crate::data_usage::store_data_usage_in_backend(snapshot, store.clone())
|
||||
.await
|
||||
.expect("seed the usage baseline");
|
||||
let object = crate::data_usage::DATA_USAGE_OBJ_NAME_PATH.as_str();
|
||||
let (before, source) = store
|
||||
.get_latest_accessible_object_info_with_idx(RUSTFS_META_BUCKET, object, &ObjectOptions::default())
|
||||
.await
|
||||
.expect("find the baseline owner");
|
||||
let unavailable = 1 - source;
|
||||
for disk in store.pools[unavailable]
|
||||
.get_disks_by_key(object)
|
||||
.disks
|
||||
.read()
|
||||
.await
|
||||
.iter()
|
||||
.flatten()
|
||||
{
|
||||
disk.close().await.expect("stop disk health monitoring");
|
||||
disk.force_offline_for_test();
|
||||
}
|
||||
let update = store
|
||||
.put_usage_snapshot_for_namespace_cleanup(
|
||||
object,
|
||||
before.etag.as_deref(),
|
||||
&mut PutObjReader::from_vec(b"unproven authority".to_vec()),
|
||||
None,
|
||||
)
|
||||
.await;
|
||||
assert!(update.is_err(), "an unreadable pool may hold a newer snapshot");
|
||||
let after = store.pools[source]
|
||||
.get_object_info(RUSTFS_META_BUCKET, object, &ObjectOptions::default())
|
||||
.await
|
||||
.expect("the readable baseline remains intact");
|
||||
assert_eq!(after.etag, before.etag);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn usage_cleanup_rejects_namespace_loss_before_physical_commit() {
|
||||
use crate::set_disk::{PutObjectCommitBarrier, PutObjectCommitPause};
|
||||
use rustfs_lock::{LocalClient, LockRequest, LockType, NamespaceLock, ObjectKey};
|
||||
|
||||
let (_dirs, store) = multipool_version_test_store("cleanup-fence-existing").await;
|
||||
let snapshot = rustfs_data_usage::DataUsageInfo {
|
||||
last_update: Some(std::time::SystemTime::now()),
|
||||
usage_snapshot_complete: true,
|
||||
..Default::default()
|
||||
};
|
||||
crate::data_usage::store_data_usage_in_backend(snapshot, store.clone())
|
||||
.await
|
||||
.expect("seed the protected baseline");
|
||||
let object = crate::data_usage::DATA_USAGE_OBJ_NAME_PATH.as_str();
|
||||
let (before, _) = store
|
||||
.get_latest_accessible_object_info_with_idx(RUSTFS_META_BUCKET, object, &ObjectOptions::default())
|
||||
.await
|
||||
.expect("read the baseline revision");
|
||||
let ttl = Duration::from_secs(60);
|
||||
let namespace = NamespaceLock::new("usage-cleanup-commit-fence".to_owned(), Arc::new(LocalClient::new()));
|
||||
let request = LockRequest::new(ObjectKey::new("cleanup-fence-existing", ""), LockType::Exclusive, "usage-cleanup")
|
||||
.with_acquire_timeout(Duration::from_secs(1))
|
||||
.with_ttl(ttl)
|
||||
.with_refresh_interval(ttl);
|
||||
let guard = Arc::new(
|
||||
namespace
|
||||
.acquire_guard(&request)
|
||||
.await
|
||||
.expect("acquire the namespace lease")
|
||||
.expect("namespace lease is available"),
|
||||
);
|
||||
let barrier = PutObjectCommitBarrier::install(RUSTFS_META_BUCKET, object, PutObjectCommitPause::AfterNamespace);
|
||||
let update_store = store.clone();
|
||||
let update_guard = guard.clone();
|
||||
let revision = before.etag.clone();
|
||||
let update = tokio::spawn(async move {
|
||||
let _movement = update_store
|
||||
.data_usage_namespace_cleanup_guard()
|
||||
.await
|
||||
.expect("admit the namespace-only update");
|
||||
update_store
|
||||
.put_usage_snapshot_for_namespace_cleanup(
|
||||
object,
|
||||
revision.as_deref(),
|
||||
&mut PutObjReader::from_vec(b"must not commit".to_vec()),
|
||||
Some(update_guard.as_ref()),
|
||||
)
|
||||
.await
|
||||
});
|
||||
tokio::time::timeout(Duration::from_secs(10), barrier.wait_until_paused())
|
||||
.await
|
||||
.expect("the update reaches its commit barrier");
|
||||
tokio::time::pause();
|
||||
tokio::time::advance(ttl + Duration::from_millis(1)).await;
|
||||
assert!(guard.is_lock_lost(), "the parent namespace lease has expired");
|
||||
tokio::time::resume();
|
||||
barrier.release();
|
||||
assert!(update.await.expect("the cleanup task completes").is_err());
|
||||
let (after, _) = store
|
||||
.get_latest_accessible_object_info_with_idx(RUSTFS_META_BUCKET, object, &ObjectOptions::default())
|
||||
.await
|
||||
.expect("the original baseline remains readable");
|
||||
assert_eq!(after.etag, before.etag, "a lost namespace lease must fence the physical write");
|
||||
}
|
||||
|
||||
async fn multipool_delete_test_store(bucket: &str, versioned: bool) -> (Vec<tempfile::TempDir>, Arc<ECStore>) {
|
||||
let ctx = Arc::new(crate::runtime::instance::InstanceContext::new());
|
||||
let (mut dirs, first_set) = make_local_set_disks_with_ctx(4, 2, Arc::clone(&ctx)).await;
|
||||
|
||||
@@ -64,6 +64,8 @@ async-compression = { workspace = true, features = ["tokio", "gzip", "bzip2"] }
|
||||
async-trait.workspace = true
|
||||
arc-swap.workspace = true
|
||||
bytes = { workspace = true, features = ["serde"] }
|
||||
base64-simd.workspace = true
|
||||
flatbuffers.workspace = true
|
||||
chrono = { workspace = true, features = ["serde"] }
|
||||
crc-fast.workspace = true
|
||||
rustfs-common.workspace = true
|
||||
@@ -75,6 +77,7 @@ futures-core = { workspace = true }
|
||||
flate2.workspace = true
|
||||
brotli.workspace = true
|
||||
lz4.workspace = true
|
||||
lz4_flex.workspace = true
|
||||
http.workspace = true
|
||||
memchr.workspace = true
|
||||
s3s = { workspace = true, features = ["minio"] }
|
||||
|
||||
@@ -221,7 +221,7 @@ struct ThriftCompactCursor<'a> {
|
||||
remaining: usize,
|
||||
}
|
||||
|
||||
impl ThriftCompactCursor<'_> {
|
||||
impl<'a> ThriftCompactCursor<'a> {
|
||||
fn error() -> ParquetBudgetError {
|
||||
ParquetBudgetError::Invalid("Parquet thrift metadata is malformed".to_owned())
|
||||
}
|
||||
@@ -241,6 +241,14 @@ impl ThriftCompactCursor<'_> {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn binary(&mut self) -> Result<&'a [u8]> {
|
||||
let len = usize::try_from(self.varint()?).map_err(|_| Self::error())?;
|
||||
self.charge(len.checked_mul(2).ok_or_else(budget_error)?)?;
|
||||
let start = self.pos;
|
||||
self.skip_bytes(len)?;
|
||||
Ok(&self.bytes[start..self.pos])
|
||||
}
|
||||
|
||||
fn varint(&mut self) -> Result<u64> {
|
||||
let mut value = 0_u64;
|
||||
for shift in (0..64).step_by(7) {
|
||||
@@ -335,11 +343,7 @@ impl ThriftCompactCursor<'_> {
|
||||
THRIFT_COMPACT_BYTE => self.skip_bytes(1),
|
||||
THRIFT_COMPACT_I16 | THRIFT_COMPACT_I32 | THRIFT_COMPACT_I64 => self.varint().map(drop),
|
||||
THRIFT_COMPACT_DOUBLE => self.skip_bytes(8),
|
||||
THRIFT_COMPACT_BINARY => {
|
||||
let len = usize::try_from(self.varint()?).map_err(|_| Self::error())?;
|
||||
self.charge(len.checked_mul(2).ok_or_else(budget_error)?)?;
|
||||
self.skip_bytes(len)
|
||||
}
|
||||
THRIFT_COMPACT_BINARY => self.binary().map(drop),
|
||||
THRIFT_COMPACT_LIST | THRIFT_COMPACT_SET => {
|
||||
let (element, count) = self.list_header()?;
|
||||
for _ in 0..count {
|
||||
@@ -586,21 +590,98 @@ impl ThriftCompactCursor<'_> {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn arrow_schema_metadata(&mut self) -> Result<Option<&'a [u8]>> {
|
||||
let (element, count) = self.list_header()?;
|
||||
if element != THRIFT_COMPACT_STRUCT {
|
||||
return Err(Self::error());
|
||||
}
|
||||
let mut arrow_schema = None;
|
||||
for _ in 0..count {
|
||||
let mut field = 0;
|
||||
let mut key = None;
|
||||
let mut value = None;
|
||||
loop {
|
||||
let header = self.byte()?;
|
||||
if header == THRIFT_COMPACT_STOP {
|
||||
break;
|
||||
}
|
||||
field = self.field_id(field, header)?;
|
||||
let kind = header & 0x0f;
|
||||
if matches!(field, 1 | 2) && kind == THRIFT_COMPACT_BINARY {
|
||||
let bytes = self.binary()?;
|
||||
if field == 1 {
|
||||
key = Some(bytes);
|
||||
} else {
|
||||
value = Some(bytes);
|
||||
}
|
||||
} else {
|
||||
self.skip_field(MetadataStruct::KeyValue, field, kind, 1)?;
|
||||
}
|
||||
}
|
||||
// Match the reader's last key with a value; absent values are ignored.
|
||||
if key == Some(b"ARROW:schema".as_slice()) && value.is_some() {
|
||||
arrow_schema = value;
|
||||
}
|
||||
}
|
||||
Ok(arrow_schema)
|
||||
}
|
||||
|
||||
fn validate_arrow_schema(&mut self, encoded: &[u8]) -> Result<()> {
|
||||
use datafusion::arrow::ipc;
|
||||
let invalid = || ParquetBudgetError::Invalid("Parquet Arrow schema metadata is malformed".to_owned());
|
||||
// This reservation precedes base64 allocation. The binary field has
|
||||
// already reserved its owned metadata copies; decoding needs another buffer.
|
||||
self.charge(encoded.len())?;
|
||||
let bytes = base64_simd::STANDARD.decode_to_vec(encoded).map_err(|_| invalid())?;
|
||||
let message = if bytes.len() > 8 && bytes.starts_with(&[255; 4]) {
|
||||
&bytes[8..]
|
||||
} else {
|
||||
bytes.as_slice()
|
||||
};
|
||||
// The verifier visits shared offsets for every occurrence, before the
|
||||
// reader expands them into owned fields and metadata. Split the remaining
|
||||
// allowance between string/buffer copies and per-table Rust containers:
|
||||
// four owned copies per apparent byte and 1 KiB per visited table.
|
||||
let defaults = flatbuffers::VerifierOptions::default();
|
||||
let options = flatbuffers::VerifierOptions {
|
||||
max_apparent_size: defaults.max_apparent_size.min(self.remaining / 8),
|
||||
max_tables: defaults.max_tables.min(self.remaining / (2 * 1024)),
|
||||
..defaults
|
||||
};
|
||||
let message = ipc::root_as_message_with_opts(&options, message).map_err(|error| match error {
|
||||
flatbuffers::InvalidFlatbuffer::TooManyTables
|
||||
| flatbuffers::InvalidFlatbuffer::ApparentSizeTooLarge
|
||||
| flatbuffers::InvalidFlatbuffer::DepthLimitReached => budget_error(),
|
||||
_ => invalid(),
|
||||
})?;
|
||||
let schema = message.header_as_schema().ok_or_else(invalid)?;
|
||||
if schema.fields().is_none() {
|
||||
return Err(invalid());
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Walk the footer's `FileMetaData`. The pinned parquet reserves the
|
||||
/// declared `row_groups` capacity before reading one element; charge that
|
||||
/// reservation to the budget so a small footer cannot request gigabytes.
|
||||
fn file_metadata(&mut self, limit: usize) -> Result<()> {
|
||||
let mut field_id = 0_i16;
|
||||
let mut reserved = 0_usize;
|
||||
let mut arrow_schema = None;
|
||||
loop {
|
||||
let header = self.byte()?;
|
||||
if header == THRIFT_COMPACT_STOP {
|
||||
if let Some(encoded) = arrow_schema {
|
||||
self.validate_arrow_schema(encoded)?;
|
||||
}
|
||||
return Ok(());
|
||||
}
|
||||
let kind = header & 0x0f;
|
||||
field_id = self.field_id(field_id, header)?;
|
||||
if field_id == 2 && kind == THRIFT_COMPACT_LIST {
|
||||
self.schema()?;
|
||||
} else if field_id == 5 && kind == THRIFT_COMPACT_LIST {
|
||||
arrow_schema = self.arrow_schema_metadata()?;
|
||||
} else if field_id == FILE_METADATA_FIELD_ROW_GROUPS && kind == THRIFT_COMPACT_LIST {
|
||||
let (element, count) = self.list_header()?;
|
||||
if element != THRIFT_COMPACT_STRUCT {
|
||||
@@ -760,21 +841,34 @@ fn validate_compressed_page(payload: &[u8], sizes: PageHeaderSizes, codec: Compr
|
||||
bounded_stream_size(brotli::Decompressor::new(payload, 4096), expanded)?;
|
||||
Ok(window)
|
||||
}
|
||||
Compression::LZ4 if payload.starts_with(&[4, 34, 77, 24]) => {
|
||||
let descriptor = *payload.get(5).ok_or_else(ThriftCompactCursor::error)?;
|
||||
let block = match (descriptor >> 4) & 7 {
|
||||
4 => 64 * 1024,
|
||||
5 => 256 * 1024,
|
||||
6 => 1024 * 1024,
|
||||
7 => 4 * 1024 * 1024,
|
||||
_ => return Err(ThriftCompactCursor::error()),
|
||||
Compression::LZ4 if payload.starts_with(&[4, 34, 77, 24]) || payload.starts_with(&[2, 33, 76, 24]) => {
|
||||
let (block, linked) = if payload.starts_with(&[2, 33, 76, 24]) {
|
||||
(8 * 1024 * 1024, false)
|
||||
} else {
|
||||
let descriptor = *payload.get(5).ok_or_else(ThriftCompactCursor::error)?;
|
||||
let block = match (descriptor >> 4) & 7 {
|
||||
4 => 64 * 1024,
|
||||
5 => 256 * 1024,
|
||||
6 => 1024 * 1024,
|
||||
7 => 4 * 1024 * 1024,
|
||||
_ => return Err(ThriftCompactCursor::error()),
|
||||
};
|
||||
(block, payload[4] & 0x20 == 0)
|
||||
};
|
||||
let working = block * 2;
|
||||
// Match parquet's pinned fallback decoder: compressed input plus
|
||||
// output, with an extra output block and 64 KiB history when linked.
|
||||
let working = if linked { block * 3 + 64 * 1024 } else { block * 2 };
|
||||
if working > limit {
|
||||
return Err(budget_error());
|
||||
}
|
||||
bounded_stream_size(lz4::Decoder::new(payload)?, expanded)?;
|
||||
Ok(working)
|
||||
match bounded_stream_size(lz4_flex::frame::FrameDecoder::new(payload), expanded) {
|
||||
// Hadoop data can share the magic prefix, and parquet also
|
||||
// tries bounded RAW decoding after a frame error. The bounded
|
||||
// sink proved the frame produced no more than `expanded` bytes
|
||||
// before that error; let the actual reader select its fallback.
|
||||
Err(ParquetBudgetError::Codec(_)) => Ok(working),
|
||||
result => result.map(|()| working),
|
||||
}
|
||||
}
|
||||
_ => Ok(0),
|
||||
}
|
||||
@@ -851,6 +945,49 @@ mod tests {
|
||||
assert!(validate_compressed_page(&payload, compressed_page(&payload, 64), Compression::LZ4, 1024 * 1024).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn legacy_lz4_scratch_is_admitted_before_the_fallback_decoder() {
|
||||
let payload = [2, 33, 76, 24, 5, 0, 0, 0, 0x40, 42, 0, 0, 0];
|
||||
let header = compressed_page(&payload, 4);
|
||||
assert!(matches!(
|
||||
validate_compressed_page(&payload, header, Compression::LZ4, 1024 * 1024),
|
||||
Err(ParquetBudgetError::Exceeded)
|
||||
));
|
||||
let admitted = validate_compressed_page(&payload, header, Compression::LZ4, 32 * 1024 * 1024)
|
||||
.expect("legacy LZ4 remains supported with enough scratch memory");
|
||||
assert!(admitted >= 16 * 1024 * 1024);
|
||||
assert!(validate_compressed_page(&payload, compressed_page(&payload, 1), Compression::LZ4, 32 * 1024 * 1024).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn hadoop_lz4_can_share_the_legacy_magic_without_becoming_a_frame() {
|
||||
let size = usize::try_from(u32::from_be_bytes([2, 33, 76, 24])).expect("Hadoop size");
|
||||
let bytes = vec![b'x'; size];
|
||||
let block = lz4_flex::block::compress(&bytes);
|
||||
let mut payload = vec![2, 33, 76, 24];
|
||||
payload.extend(u32::try_from(block.len()).expect("compressed block").to_be_bytes());
|
||||
payload.extend(block);
|
||||
validate_compressed_page(&payload, compressed_page(&payload, size), Compression::LZ4, 64 * 1024 * 1024)
|
||||
.expect("Hadoop framing takes priority over the frame fallback");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn linked_lz4_accounts_for_the_history_and_both_block_buffers() {
|
||||
use lz4_flex::frame::{BlockMode, BlockSize, FrameEncoder, FrameInfo};
|
||||
let frame = FrameInfo::new().block_mode(BlockMode::Linked).block_size(BlockSize::Max64KB);
|
||||
let mut encoder = FrameEncoder::with_frame_info(frame, Vec::new());
|
||||
encoder.write_all(&vec![b'x'; 256 * 1024]).expect("linked input");
|
||||
let payload = encoder.finish().expect("linked frame");
|
||||
let header = compressed_page(&payload, 256 * 1024);
|
||||
assert!(matches!(
|
||||
validate_compressed_page(&payload, header, Compression::LZ4, 192 * 1024),
|
||||
Err(ParquetBudgetError::Exceeded)
|
||||
));
|
||||
let admitted = validate_compressed_page(&payload, header, Compression::LZ4, 1024 * 1024)
|
||||
.expect("linked frame with enough scratch memory");
|
||||
assert!(admitted >= 256 * 1024);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn uncompressed_v2_does_not_invoke_the_column_codec() {
|
||||
let payload = b"levels-and-uncompressed-values";
|
||||
@@ -941,6 +1078,170 @@ mod tests {
|
||||
validate_file_footer(&bytes, 1024 * 1024).expect("valid logical annotations remain accepted");
|
||||
}
|
||||
|
||||
fn shared_arrow_metadata_file(entries: usize) -> Vec<u8> {
|
||||
use datafusion::{
|
||||
arrow::ipc,
|
||||
parquet::{
|
||||
file::{properties::WriterProperties, writer::SerializedFileWriter},
|
||||
schema::parser::parse_message_type,
|
||||
},
|
||||
};
|
||||
use std::sync::Arc;
|
||||
let mut builder = flatbuffers::FlatBufferBuilder::new();
|
||||
let name = builder.create_string("v");
|
||||
let value = builder.create_string(&"x".repeat(64 * 1024));
|
||||
let data_type = ipc::Int::create(
|
||||
&mut builder,
|
||||
&ipc::IntArgs {
|
||||
bitWidth: 32,
|
||||
is_signed: true,
|
||||
},
|
||||
);
|
||||
let field = ipc::Field::create(
|
||||
&mut builder,
|
||||
&ipc::FieldArgs {
|
||||
name: Some(name),
|
||||
nullable: true,
|
||||
type_type: ipc::Type::Int,
|
||||
type_: Some(data_type.as_union_value()),
|
||||
..Default::default()
|
||||
},
|
||||
);
|
||||
let fields = builder.create_vector(&[field]);
|
||||
let mut metadata = Vec::new();
|
||||
for n in 0..entries {
|
||||
let key = builder.create_string(&format!("key-{n}"));
|
||||
metadata.push(ipc::KeyValue::create(
|
||||
&mut builder,
|
||||
&ipc::KeyValueArgs {
|
||||
key: Some(key),
|
||||
value: Some(value),
|
||||
},
|
||||
));
|
||||
}
|
||||
let metadata = builder.create_vector(&metadata);
|
||||
let schema = ipc::Schema::create(
|
||||
&mut builder,
|
||||
&ipc::SchemaArgs {
|
||||
fields: Some(fields),
|
||||
custom_metadata: Some(metadata),
|
||||
..Default::default()
|
||||
},
|
||||
);
|
||||
let message = ipc::Message::create(
|
||||
&mut builder,
|
||||
&ipc::MessageArgs {
|
||||
version: ipc::MetadataVersion::V5,
|
||||
header_type: ipc::MessageHeader::Schema,
|
||||
header: Some(schema.as_union_value()),
|
||||
..Default::default()
|
||||
},
|
||||
);
|
||||
builder.finish(message, None);
|
||||
let encoded = base64_simd::STANDARD.encode_to_string(builder.finished_data());
|
||||
let properties = WriterProperties::builder()
|
||||
.set_key_value_metadata(Some(vec![datafusion::parquet::file::metadata::KeyValue::new(
|
||||
"ARROW:schema".to_owned(),
|
||||
Some(encoded),
|
||||
)]))
|
||||
.build();
|
||||
let schema = Arc::new(parse_message_type("message root { OPTIONAL INT32 v; }").expect("physical schema"));
|
||||
let mut bytes = Vec::new();
|
||||
SerializedFileWriter::new(&mut bytes, schema, Arc::new(properties))
|
||||
.expect("writer")
|
||||
.close()
|
||||
.expect("footer");
|
||||
bytes
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn shared_arrow_metadata_is_admitted_before_owned_schema_conversion() {
|
||||
use datafusion::parquet::arrow::arrow_reader::ParquetRecordBatchReaderBuilder;
|
||||
let bytes = shared_arrow_metadata_file(256);
|
||||
assert!(bytes.len() < 100 * 1024, "shared offsets keep the encoded file small");
|
||||
let native = ParquetRecordBatchReaderBuilder::try_new(bytes::Bytes::from(bytes.clone())).expect("valid Arrow metadata");
|
||||
assert_eq!(native.schema().metadata().len(), 256);
|
||||
assert!(
|
||||
matches!(validate_file_footer(&bytes, 2 * 1024 * 1024), Err(ParquetBudgetError::Exceeded)),
|
||||
"32 MiB of owned metadata must not pass a 2 MiB preflight"
|
||||
);
|
||||
validate_file_footer(&bytes, 512 * 1024 * 1024).expect("sufficient budget preserves shared metadata");
|
||||
}
|
||||
|
||||
fn arrow_hint_footer(entries: &[(&[u8], Option<&[u8]>)]) -> Vec<u8> {
|
||||
let count = u8::try_from(entries.len()).expect("small key/value fixture");
|
||||
assert!(count < 15);
|
||||
let mut bytes = vec![0x59, (count << 4) | THRIFT_COMPACT_STRUCT];
|
||||
for (key, value) in entries {
|
||||
bytes.push(0x18);
|
||||
bytes.extend(varint(u64::try_from(key.len()).expect("key length")));
|
||||
bytes.extend(*key);
|
||||
if let Some(value) = value {
|
||||
bytes.push(0x18);
|
||||
bytes.extend(varint(u64::try_from(value.len()).expect("value length")));
|
||||
bytes.extend(*value);
|
||||
}
|
||||
bytes.push(0);
|
||||
}
|
||||
bytes.push(0);
|
||||
bytes
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn arrow_hint_framing_and_duplicate_keys_match_the_reader() {
|
||||
use datafusion::arrow::{
|
||||
datatypes::{DataType, Field, Schema},
|
||||
ipc,
|
||||
};
|
||||
let schema = Schema::new(vec![Field::new("v", DataType::Int32, true)]);
|
||||
let mut builder = flatbuffers::FlatBufferBuilder::new();
|
||||
let schema = ipc::convert::schema_to_fb_offset(&mut builder, &schema);
|
||||
let message = ipc::Message::create(
|
||||
&mut builder,
|
||||
&ipc::MessageArgs {
|
||||
header_type: ipc::MessageHeader::Schema,
|
||||
header: Some(schema.as_union_value()),
|
||||
..Default::default()
|
||||
},
|
||||
);
|
||||
builder.finish(message, None);
|
||||
let raw = builder.finished_data();
|
||||
let mut framed = vec![255; 4];
|
||||
framed.extend(u32::try_from(raw.len()).expect("small schema").to_le_bytes());
|
||||
framed.extend(raw);
|
||||
for message in [raw, framed.as_slice()] {
|
||||
let encoded = base64_simd::STANDARD.encode_to_string(message);
|
||||
let footer = arrow_hint_footer(&[
|
||||
(b"ARROW:schema", Some(b"invalid")),
|
||||
(b"ARROW:schema", Some(encoded.as_bytes())),
|
||||
(b"ARROW:schema", None),
|
||||
]);
|
||||
validate_footer(&footer, 1024 * 1024).expect("last present Arrow metadata wins, both IPC framings are supported");
|
||||
assert!(matches!(
|
||||
validate_footer(
|
||||
&arrow_hint_footer(&[
|
||||
(b"ARROW:schema", Some(encoded.as_bytes())),
|
||||
(b"ARROW:schema", Some(b"invalid"))
|
||||
]),
|
||||
1024 * 1024
|
||||
),
|
||||
Err(ParquetBudgetError::Invalid(_))
|
||||
));
|
||||
}
|
||||
validate_footer(&arrow_hint_footer(&[(b"other", Some(b"invalid"))]), 1024 * 1024).expect("other metadata is opaque");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn arrow_hint_value_before_key_is_still_admitted() {
|
||||
// Compact struct fields are identified by ID, not required to be ordered.
|
||||
let mut bytes = vec![0x59, 0x1c, 0x28, 3];
|
||||
bytes.extend(b"%%%");
|
||||
bytes.extend([0x08, 2, 12]);
|
||||
bytes.extend(b"ARROW:schema");
|
||||
bytes.extend([0, 0]);
|
||||
assert!(matches!(validate_footer(&bytes, 1024 * 1024), Err(ParquetBudgetError::Invalid(_))));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn footer_collections_are_bounded_before_decoder_allocation() {
|
||||
let mut huge = vec![0x49, 0xfc];
|
||||
|
||||
@@ -1106,6 +1106,98 @@ mod tests {
|
||||
assert_eq!(error.select_error(), SelectError::ResourceExhausted);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn parquet_legacy_lz4_checks_native_scratch_before_query_decode() {
|
||||
use datafusion::parquet::{basic::Compression, file::metadata::ParquetMetaDataWriter};
|
||||
let schema = Arc::new(Schema::new(vec![Field::new("v", DataType::Int32, false)]));
|
||||
let batch =
|
||||
RecordBatch::try_new(Arc::clone(&schema), vec![Arc::new(datafusion::arrow::array::Int32Array::from(vec![42]))])
|
||||
.expect("one integer row");
|
||||
let properties = datafusion::parquet::file::properties::WriterProperties::builder()
|
||||
.set_dictionary_enabled(false)
|
||||
.build();
|
||||
let mut original = Vec::new();
|
||||
let mut writer = ArrowWriter::try_new(&mut original, schema, Some(properties)).expect("writer");
|
||||
writer.write(&batch).expect("row");
|
||||
writer.close().expect("footer");
|
||||
let metadata = ParquetRecordBatchReaderBuilder::try_new(Bytes::from(original))
|
||||
.expect("metadata")
|
||||
.metadata()
|
||||
.clone();
|
||||
// A valid legacy frame encoding the four little-endian bytes for 42.
|
||||
let payload = [2, 33, 76, 24, 5, 0, 0, 0, 0x40, 42, 0, 0, 0];
|
||||
let mut page = vec![0x15, 0, 0x15, 8, 0x15, 26, 0x2c, 0x15, 2, 0x15, 0, 0x15, 6, 0x15, 6, 0, 0];
|
||||
page.extend(payload);
|
||||
let column = metadata
|
||||
.row_group(0)
|
||||
.column(0)
|
||||
.clone()
|
||||
.into_builder()
|
||||
.set_compression(Compression::LZ4)
|
||||
.set_data_page_offset(4)
|
||||
.set_total_compressed_size(i64::try_from(page.len()).expect("small page"))
|
||||
.build()
|
||||
.expect("legacy column");
|
||||
let group = metadata
|
||||
.row_group(0)
|
||||
.clone()
|
||||
.into_builder()
|
||||
.set_column_metadata(vec![column])
|
||||
.build()
|
||||
.expect("row group");
|
||||
let metadata = metadata.as_ref().clone().into_builder().set_row_groups(vec![group]).build();
|
||||
let mut encoded = b"PAR1".to_vec();
|
||||
encoded.extend(page);
|
||||
ParquetMetaDataWriter::new(&mut encoded, &metadata)
|
||||
.finish()
|
||||
.expect("legacy footer");
|
||||
assert!(encoded.len() < 4096);
|
||||
let mut native = ParquetRecordBatchReaderBuilder::try_new(Bytes::from(encoded.clone()))
|
||||
.expect("legacy metadata")
|
||||
.build()
|
||||
.expect("native reader");
|
||||
let decoded = native.next().expect("one batch").expect("legacy fallback decode");
|
||||
assert_eq!(
|
||||
decoded
|
||||
.column(0)
|
||||
.as_primitive::<datafusion::arrow::datatypes::Int32Type>()
|
||||
.value(0),
|
||||
42
|
||||
);
|
||||
let store = Arc::new(InMemory::new());
|
||||
store
|
||||
.put(&Path::from("legacy.parquet"), Bytes::from(encoded).into())
|
||||
.await
|
||||
.expect("fixture");
|
||||
let tight = parquet_session_with_memory(store.clone(), 2 * 1024 * 1024);
|
||||
let result = ParquetSelectTable::try_new(&tight.state(), &parquet_input("legacy.parquet"), None).await;
|
||||
let error = match result {
|
||||
Ok(_) => panic!("legacy decoder scratch exceeds the query budget"),
|
||||
Err(error) => error,
|
||||
};
|
||||
assert_eq!(error.select_error(), SelectError::ResourceExhausted);
|
||||
let roomy = parquet_session_with_memory(store, 128 * 1024 * 1024);
|
||||
let table = ParquetSelectTable::try_new(&roomy.state(), &parquet_input("legacy.parquet"), None)
|
||||
.await
|
||||
.expect("admitted legacy file");
|
||||
roomy.register_table("source", table).expect("table");
|
||||
let rows = roomy
|
||||
.sql("SELECT v FROM source")
|
||||
.await
|
||||
.expect("plan")
|
||||
.collect()
|
||||
.await
|
||||
.expect("query");
|
||||
assert_eq!(rows.iter().map(RecordBatch::num_rows).sum::<usize>(), 1);
|
||||
assert_eq!(
|
||||
rows[0]
|
||||
.column(0)
|
||||
.as_primitive::<datafusion::arrow::datatypes::Int32Type>()
|
||||
.value(0),
|
||||
42
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn parquet_nested_views_decode_through_the_actual_file_schema() {
|
||||
use datafusion::arrow::array::{ListArray, StringArray};
|
||||
|
||||
@@ -690,6 +690,19 @@ pub(super) enum ScannerPauseBacklogAttemptDecision {
|
||||
pub struct ScannerPauseBacklogReplicaId {
|
||||
pub pool_index: usize,
|
||||
pub set_index: usize,
|
||||
/// Stable format identity; indices are only the current routing location.
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub set_id: Option<uuid::Uuid>,
|
||||
}
|
||||
|
||||
impl ScannerPauseBacklogReplicaId {
|
||||
fn from_set(set: &SetDisks) -> Result<Self, EcstoreError> {
|
||||
Ok(Self {
|
||||
pool_index: set.pool_index,
|
||||
set_index: set.set_index,
|
||||
set_id: Some(set.persistent_set_id()?),
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, Deserialize, PartialEq, Eq, Serialize)]
|
||||
@@ -706,9 +719,23 @@ impl ScannerPauseBacklogCommitRecord {
|
||||
|
||||
fn validate(&self) -> Result<(), String> {
|
||||
self.ledger.validate()?;
|
||||
if self.replicas.is_empty() || !self.replicas.windows(2).all(|pair| pair[0] < pair[1]) {
|
||||
if self.replicas.is_empty()
|
||||
|| !self
|
||||
.replicas
|
||||
.windows(2)
|
||||
.all(|pair| (pair[0].pool_index, pair[0].set_index) < (pair[1].pool_index, pair[1].set_index))
|
||||
{
|
||||
return Err("scanner pause backlog commit has invalid replica membership".to_string());
|
||||
}
|
||||
let mut identities = BTreeSet::new();
|
||||
if self
|
||||
.replicas
|
||||
.iter()
|
||||
.filter_map(|id| id.set_id)
|
||||
.any(|id| id.is_nil() || !identities.insert(id))
|
||||
{
|
||||
return Err("scanner pause backlog commit has invalid persistent set identities".to_string());
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
@@ -747,6 +774,10 @@ impl ScannerPauseBacklogReplicaRecord {
|
||||
}
|
||||
if let Some(committed) = &self.committed {
|
||||
committed.validate()?;
|
||||
if committed.replicas.iter().any(|id| id.set_id.is_some()) && committed.replicas.iter().any(|id| id.set_id.is_none())
|
||||
{
|
||||
return Err("scanner pause backlog membership mixes positional and persistent identities".to_string());
|
||||
}
|
||||
if let Some(stable) = &self.stable {
|
||||
let stable_key = (stable.writer_epoch, stable.generation);
|
||||
let committed_key = (committed.ledger.writer_epoch, committed.ledger.generation);
|
||||
@@ -907,9 +938,19 @@ fn unix_now() -> u64 {
|
||||
}
|
||||
|
||||
async fn read_scanner_pause_backlog_replica(store: Arc<SetDisks>) -> ScannerPauseBacklogReplica {
|
||||
let id = ScannerPauseBacklogReplicaId {
|
||||
pool_index: store.pool_index,
|
||||
set_index: store.set_index,
|
||||
let id = match ScannerPauseBacklogReplicaId::from_set(&store) {
|
||||
Ok(id) => id,
|
||||
Err(error) => {
|
||||
return ScannerPauseBacklogReplica {
|
||||
id: ScannerPauseBacklogReplicaId {
|
||||
pool_index: store.pool_index,
|
||||
set_index: store.set_index,
|
||||
set_id: None,
|
||||
},
|
||||
revision: None,
|
||||
state: ScannerPauseBacklogReplicaState::Unavailable(error.to_string()),
|
||||
};
|
||||
}
|
||||
};
|
||||
let reader = match store
|
||||
.get_object_reader(
|
||||
@@ -1333,8 +1374,48 @@ fn select_scanner_pause_backlog_roll_forward(replicas: &[ScannerPauseBacklogRepl
|
||||
.then(|| newest.clone())
|
||||
}
|
||||
|
||||
/// Resolve stored membership against immutable set identities before checking
|
||||
/// commit proofs. A missing identity is never rebound to a new set at its old
|
||||
/// index. Legacy records retain their positional interpretation until an
|
||||
/// existing commit or rollback proof permits publishing the bound membership.
|
||||
fn normalize_scanner_pause_backlog_membership(
|
||||
replicas: &mut [ScannerPauseBacklogReplica],
|
||||
) -> Result<(), ScannerPauseBacklogSelectionError> {
|
||||
let mut by_location = HashMap::with_capacity(replicas.len());
|
||||
let mut by_identity = HashMap::with_capacity(replicas.len());
|
||||
for replica in replicas.iter() {
|
||||
let id = replica.id;
|
||||
if by_location.insert((id.pool_index, id.set_index), id).is_some()
|
||||
|| id
|
||||
.set_id
|
||||
.is_some_and(|identity| identity.is_nil() || by_identity.insert(identity, id).is_some())
|
||||
{
|
||||
return Err(ScannerPauseBacklogSelectionError::AuthorityConflict(
|
||||
"scanner pause backlog has duplicate or invalid storage identities".to_string(),
|
||||
));
|
||||
}
|
||||
}
|
||||
for replica in replicas {
|
||||
if let ScannerPauseBacklogReplicaState::Valid(record) = &mut replica.state
|
||||
&& let Some(commit) = record.committed.as_mut()
|
||||
{
|
||||
for member in &mut commit.replicas {
|
||||
let current = match member.set_id {
|
||||
Some(identity) => by_identity.get(&identity),
|
||||
None => by_location.get(&(member.pool_index, member.set_index)),
|
||||
};
|
||||
if let Some(current) = current {
|
||||
*member = *current;
|
||||
}
|
||||
}
|
||||
commit.replicas.sort_unstable();
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn select_scanner_pause_backlog_replicas(
|
||||
replicas: Vec<ScannerPauseBacklogReplica>,
|
||||
mut replicas: Vec<ScannerPauseBacklogReplica>,
|
||||
) -> Result<LoadedScannerPauseBacklog, ScannerPauseBacklogSelectionFailure> {
|
||||
if replicas.is_empty() {
|
||||
return Err(scanner_pause_backlog_selection_failure(
|
||||
@@ -1354,6 +1435,9 @@ fn select_scanner_pause_backlog_replicas(
|
||||
)),
|
||||
));
|
||||
}
|
||||
if let Err(error) = normalize_scanner_pause_backlog_membership(&mut replicas) {
|
||||
return Err(scanner_pause_backlog_selection_failure(replicas, error));
|
||||
}
|
||||
|
||||
let replica_ids = scanner_pause_backlog_replica_ids(&replicas);
|
||||
let authoritative_commit = match select_scanner_pause_backlog_commit(&replicas, &replica_ids) {
|
||||
@@ -1512,12 +1596,13 @@ fn decode_scanner_pause_backlog_retirement_replicas(
|
||||
) -> Result<(Vec<ScannerPauseBacklogReplica>, Vec<ScannerPauseBacklogReplica>), ScannerPauseBacklogRetirementError> {
|
||||
let mut ids = BTreeSet::new();
|
||||
let mut decoded = Vec::with_capacity(replicas.len());
|
||||
let mut surviving = Vec::new();
|
||||
let mut writable_ids = BTreeSet::new();
|
||||
let mut has_source = false;
|
||||
for replica in replicas {
|
||||
let id = ScannerPauseBacklogReplicaId {
|
||||
pool_index: replica.pool_index,
|
||||
set_index: replica.set_index,
|
||||
set_id: replica.set_id,
|
||||
};
|
||||
if !ids.insert(id) {
|
||||
return Err(scanner_pause_backlog_retirement_invalid_record(
|
||||
@@ -1548,7 +1633,7 @@ fn decode_scanner_pause_backlog_retirement_replicas(
|
||||
state,
|
||||
};
|
||||
if replica.pool_index != source_pool_index && replica.writable {
|
||||
surviving.push(decoded_replica.clone());
|
||||
writable_ids.insert(id);
|
||||
}
|
||||
decoded.push(decoded_replica);
|
||||
}
|
||||
@@ -1557,6 +1642,12 @@ fn decode_scanner_pause_backlog_retirement_replicas(
|
||||
"scanner pause backlog retirement has no native source record",
|
||||
));
|
||||
}
|
||||
normalize_scanner_pause_backlog_membership(&mut decoded).map_err(scanner_pause_backlog_retirement_selection_error)?;
|
||||
let surviving = decoded
|
||||
.iter()
|
||||
.filter(|replica| writable_ids.contains(&replica.id))
|
||||
.cloned()
|
||||
.collect();
|
||||
Ok((decoded, surviving))
|
||||
}
|
||||
|
||||
@@ -1874,11 +1965,9 @@ where
|
||||
let loaded_ids = loaded.replicas.iter().map(|replica| replica.id).collect::<BTreeSet<_>>();
|
||||
let writable_ids = writable
|
||||
.iter()
|
||||
.map(|set| ScannerPauseBacklogReplicaId {
|
||||
pool_index: set.pool_index,
|
||||
set_index: set.set_index,
|
||||
})
|
||||
.collect::<BTreeSet<_>>();
|
||||
.map(|set| ScannerPauseBacklogReplicaId::from_set(set))
|
||||
.collect::<Result<BTreeSet<_>, _>>()
|
||||
.map_err(|error| error.to_string())?;
|
||||
if writable_ids != loaded_ids {
|
||||
return Err("scanner pause backlog replica topology changed during commit".to_string());
|
||||
}
|
||||
@@ -1901,10 +1990,7 @@ where
|
||||
let results = tokio::spawn(async move {
|
||||
let mut results = Vec::with_capacity(writable.len());
|
||||
for set in writable {
|
||||
let id = ScannerPauseBacklogReplicaId {
|
||||
pool_index: set.pool_index,
|
||||
set_index: set.set_index,
|
||||
};
|
||||
let id = ScannerPauseBacklogReplicaId::from_set(&set).map_err(|error| error.to_string())?;
|
||||
if only.as_ref().is_some_and(|only| !only.contains(&id)) {
|
||||
continue;
|
||||
}
|
||||
@@ -1928,10 +2014,10 @@ where
|
||||
break;
|
||||
}
|
||||
}
|
||||
results
|
||||
Ok::<_, String>(results)
|
||||
})
|
||||
.await
|
||||
.map_err(|err| format!("scanner pause backlog publication owner failed: {err}"))?;
|
||||
.map_err(|err| format!("scanner pause backlog publication owner failed: {err}"))??;
|
||||
|
||||
let failures = results
|
||||
.iter()
|
||||
@@ -2444,6 +2530,142 @@ mod tests {
|
||||
(root, store)
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[serial_test::serial]
|
||||
fn native_legacy_upgrade_and_pool_reindex_preserve_backlog_authority() {
|
||||
assert_native_legacy_upgrade_and_pool_reindex_preserve_backlog_authority(false);
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[serial_test::serial]
|
||||
fn native_pool_reindex_with_failed_membership_publication_preserves_authority() {
|
||||
assert_native_legacy_upgrade_and_pool_reindex_preserve_backlog_authority(true);
|
||||
}
|
||||
|
||||
fn assert_native_legacy_upgrade_and_pool_reindex_preserve_backlog_authority(fail_publication: bool) {
|
||||
run_native_retirement_test(async move || {
|
||||
use crate::storage_api::owner::NativeScannerPauseBacklogWriteFault;
|
||||
|
||||
let (root, store) = native_pause_backlog_store(3, 1).await;
|
||||
let now = unix_now();
|
||||
let original = seed_native_expansion_ledger(&store, now).await;
|
||||
// Model bytes written by the previous positional-membership writer.
|
||||
for id in native_replica_ids(&store) {
|
||||
let (bytes, _) = native_replica_bytes(&store.pools[id.pool_index].disk_set[0]).await;
|
||||
let ScannerPauseBacklogReplicaState::Valid(mut record) = decode_scanner_pause_backlog_ledger(&bytes) else {
|
||||
panic!("native seed is valid");
|
||||
};
|
||||
record.replica_schema_version = 1;
|
||||
for member in &mut record.committed.as_mut().expect("seed commit").replicas {
|
||||
member.set_id = None;
|
||||
}
|
||||
record.validate().expect("previous schema fixture");
|
||||
replace_native_pause_backlog_replica(&store, id, *record).await;
|
||||
}
|
||||
let fault =
|
||||
NativeScannerPauseBacklogWriteFault::fail_before_write(Arc::clone(&store.pools[1].disk_set[0]), "publish", 1);
|
||||
assert!(
|
||||
ScannerPauseBacklogController::claim(Arc::clone(&store), now + 60)
|
||||
.await
|
||||
.is_err()
|
||||
);
|
||||
drop(fault);
|
||||
let interrupted = load_scanner_pause_backlog(Arc::clone(&store))
|
||||
.await
|
||||
.expect("mixed legacy and bound records retain the rollback proof");
|
||||
assert_eq!(interrupted.ledger, original);
|
||||
let upgraded = ScannerPauseBacklogController::claim(Arc::clone(&store), now + 61)
|
||||
.await
|
||||
.expect("retry finishes the format upgrade");
|
||||
let expected = upgraded.loaded.ledger.clone();
|
||||
drop(upgraded);
|
||||
for pool in &store.pools {
|
||||
let (bytes, _) = native_replica_bytes(&pool.disk_set[0]).await;
|
||||
let ScannerPauseBacklogReplicaState::Valid(record) = decode_scanner_pause_backlog_ledger(&bytes) else {
|
||||
panic!("upgraded native record is valid");
|
||||
};
|
||||
assert_eq!(record.replica_schema_version, 1);
|
||||
assert!(
|
||||
record
|
||||
.committed
|
||||
.expect("upgraded membership")
|
||||
.replicas
|
||||
.iter()
|
||||
.all(|id| id.set_id.is_some())
|
||||
);
|
||||
}
|
||||
let surviving_id = native_replica_id(&store, 2, 0).set_id;
|
||||
let (source, _) = native_replica_bytes(&store.pools[1].disk_set[0]).await;
|
||||
store
|
||||
.prepare_scanner_pause_backlog_retirement_for_test(1, source.len() * 2)
|
||||
.await
|
||||
.expect("start retiring the middle pool");
|
||||
store
|
||||
.retire_scanner_pause_backlog_for_test(1, 0)
|
||||
.await
|
||||
.expect("hand off the native backlog");
|
||||
store
|
||||
.complete_scanner_pause_backlog_retirement_for_test(1)
|
||||
.await
|
||||
.expect("persist the completed middle pool through the ILM receipt checks");
|
||||
shutdown_native_retirement_store(store).await;
|
||||
|
||||
let reordered = super::super::tests::setup_scanner_cycle_store_with_pool_order(
|
||||
root.path(),
|
||||
false,
|
||||
&[0, 2, 3],
|
||||
1,
|
||||
NATIVE_RETIREMENT_DRIVES_PER_SET,
|
||||
false,
|
||||
)
|
||||
.await;
|
||||
assert_eq!(native_replica_id(&reordered, 1, 0).set_id, surviving_id);
|
||||
assert_ne!(native_replica_id(&reordered, 2, 0).set_id, surviving_id);
|
||||
let loaded = load_scanner_pause_backlog(Arc::clone(&reordered))
|
||||
.await
|
||||
.expect("removing a middle pool and appending a new one preserves native authority");
|
||||
assert_eq!(loaded.ledger, expected);
|
||||
assert_eq!(loaded.persistence_state, "membership_repair_pending");
|
||||
if fail_publication {
|
||||
// Match the reported partial publication: the two existing
|
||||
// pools reject the new commit while the replacement accepts it.
|
||||
let faults = [0, 1].map(|pool| {
|
||||
NativeScannerPauseBacklogWriteFault::fail_before_write(
|
||||
Arc::clone(&reordered.pools[pool].disk_set[0]),
|
||||
"publish",
|
||||
2,
|
||||
)
|
||||
});
|
||||
let error = ScannerPauseBacklogController::claim(Arc::clone(&reordered), now + 120)
|
||||
.await
|
||||
.err()
|
||||
.expect("the incomplete publication must fail");
|
||||
assert!(error.contains("pool 0 set 0") && error.contains("pool 1 set 0"), "{error}");
|
||||
drop(faults);
|
||||
let recovered = load_scanner_pause_backlog(Arc::clone(&reordered))
|
||||
.await
|
||||
.expect("partial publication after reindexing retains proven authority");
|
||||
assert_eq!(recovered.ledger, expected);
|
||||
assert!(recovered.durable);
|
||||
}
|
||||
let (source, _) = native_replica_bytes(&reordered.pools[1].disk_set[0]).await;
|
||||
reordered
|
||||
.prepare_scanner_pause_backlog_retirement_for_test(1, source.len() * 2)
|
||||
.await
|
||||
.expect("retire the reindexed member before a scanner writer completes the membership change");
|
||||
reordered
|
||||
.retire_scanner_pause_backlog_for_test(1, 0)
|
||||
.await
|
||||
.expect("retirement uses the same bound authority");
|
||||
let surviving = load_scanner_pause_backlog(Arc::clone(&reordered))
|
||||
.await
|
||||
.expect("remaining members retain the complete ledger");
|
||||
assert_eq!(surviving.ledger, expected);
|
||||
assert!(surviving.all_current_replicas_stable());
|
||||
shutdown_native_retirement_store(reordered).await;
|
||||
});
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[serial_test::serial]
|
||||
fn native_single_pool_partial_commit_converges_across_24_sets() {
|
||||
@@ -2592,9 +2814,7 @@ mod tests {
|
||||
);
|
||||
|
||||
let stable = durable_ledger(unix_now());
|
||||
let members = (0..3)
|
||||
.flat_map(|pool_index| (0..2).map(move |set_index| replica_id(pool_index, set_index)))
|
||||
.collect::<Vec<_>>();
|
||||
let members = native_replica_ids(&store);
|
||||
let committed = ScannerPauseBacklogCommitRecord::new(stable.clone(), members);
|
||||
let record = ScannerPauseBacklogReplicaRecord::new(Some(stable), Some(committed));
|
||||
let bytes = encode_scanner_pause_backlog_record(&record).expect("stable replica should encode");
|
||||
@@ -3021,10 +3241,7 @@ mod tests {
|
||||
let now = unix_now().saturating_add(60);
|
||||
let replacement = claim_scanner_pause_backlog_writer(&original, now).expect("next writer generation");
|
||||
let new_commit = ScannerPauseBacklogCommitRecord::new(replacement.clone(), current_ids.clone());
|
||||
let failed_id = ScannerPauseBacklogReplicaId {
|
||||
pool_index: failed_pool,
|
||||
set_index: 0,
|
||||
};
|
||||
let failed_id = native_replica_id(&store, failed_pool, 0);
|
||||
let fault = NativeScannerPauseBacklogWriteFault::fail_before_write(
|
||||
Arc::clone(&store.pools[failed_pool].disk_set[0]),
|
||||
"publish",
|
||||
@@ -3553,10 +3770,8 @@ mod tests {
|
||||
.authoritative_commit
|
||||
.expect("old cohort commit");
|
||||
let now = unix_now().saturating_add(60);
|
||||
let ids = (0..3)
|
||||
.flat_map(|pool_index| (0..2).map(move |set_index| replica_id(pool_index, set_index)))
|
||||
.collect::<Vec<_>>();
|
||||
let newer_member = replica_id(2, 1);
|
||||
let ids = native_replica_ids(&store);
|
||||
let newer_member = native_replica_id(&store, 2, 1);
|
||||
let newer = claim_scanner_pause_backlog_writer(&original, now).expect("newer generation");
|
||||
let newer_commit = ScannerPauseBacklogCommitRecord::new(newer.clone(), ids.clone());
|
||||
for id in &ids {
|
||||
@@ -3617,14 +3832,12 @@ mod tests {
|
||||
let (_root, store, original) = native_expanded_backlog_store(1, false).await;
|
||||
let now = unix_now().saturating_add(60);
|
||||
let wedge = native_interleaved_expansion_wedge(&store, &original, now).await;
|
||||
let ids = (0..3)
|
||||
.flat_map(|pool_index| (0..2).map(move |set_index| replica_id(pool_index, set_index)))
|
||||
.collect::<Vec<_>>();
|
||||
let ids = native_replica_ids(&store);
|
||||
// The retiring source holds the newest stable ledger.
|
||||
let newest = claim_scanner_pause_backlog_writer(&wedge.acknowledged, now + 1).expect("newest generation");
|
||||
replace_native_pause_backlog_replica(
|
||||
&store,
|
||||
replica_id(0, 0),
|
||||
native_replica_id(&store, 0, 0),
|
||||
ScannerPauseBacklogReplicaRecord::new(
|
||||
Some(newest.clone()),
|
||||
Some(ScannerPauseBacklogCommitRecord::new(newest.clone(), ids)),
|
||||
@@ -3675,10 +3888,8 @@ mod tests {
|
||||
set_runtime_error(None);
|
||||
let (_root, store, original) = native_expanded_backlog_store(1, false).await;
|
||||
let now = unix_now().saturating_add(60);
|
||||
let ids = (0..3)
|
||||
.flat_map(|pool_index| (0..2).map(move |set_index| replica_id(pool_index, set_index)))
|
||||
.collect::<Vec<_>>();
|
||||
let newer_member = replica_id(2, 1);
|
||||
let ids = native_replica_ids(&store);
|
||||
let newer_member = native_replica_id(&store, 2, 1);
|
||||
let others = ids.iter().copied().filter(|id| *id != newer_member).collect::<Vec<_>>();
|
||||
let middle = claim_scanner_pause_backlog_writer(&original, now).expect("middle generation");
|
||||
let newest = claim_scanner_pause_backlog_writer(&middle, now + 1).expect("newest generation");
|
||||
@@ -3837,10 +4048,8 @@ mod tests {
|
||||
let (_root, store, original) = native_expanded_backlog_store(1, false).await;
|
||||
let now = unix_now().saturating_add(60);
|
||||
let wedge = native_interleaved_expansion_wedge(&store, &original, now).await;
|
||||
let ids = (0..3)
|
||||
.flat_map(|pool_index| (0..2).map(move |set_index| replica_id(pool_index, set_index)))
|
||||
.collect::<Vec<_>>();
|
||||
assert_eq!(wedge.last, replica_id(2, 1));
|
||||
let ids = native_replica_ids(&store);
|
||||
assert_eq!(wedge.last, native_replica_id(&store, 2, 1));
|
||||
|
||||
// A different stable ledger at the newest key is a fork.
|
||||
let fork = claim_scanner_pause_backlog_writer(&original, now + 7).expect("forked writer generation");
|
||||
@@ -3851,7 +4060,7 @@ mod tests {
|
||||
assert_ne!(fork, wedge.acknowledged);
|
||||
replace_native_pause_backlog_replica(
|
||||
&store,
|
||||
replica_id(1, 0),
|
||||
native_replica_id(&store, 1, 0),
|
||||
ScannerPauseBacklogReplicaRecord::new(
|
||||
Some(fork.clone()),
|
||||
Some(ScannerPauseBacklogCommitRecord::new(fork, ids.clone())),
|
||||
@@ -3868,7 +4077,7 @@ mod tests {
|
||||
"{error}"
|
||||
);
|
||||
assert_eq!(native_replica_snapshot(&store).await, forked, "a refused claim keeps every record");
|
||||
replace_native_pause_backlog_replica(&store, replica_id(1, 0), wedge.rollback.clone()).await;
|
||||
replace_native_pause_backlog_replica(&store, native_replica_id(&store, 1, 0), wedge.rollback.clone()).await;
|
||||
|
||||
// A missing member that a stored commit names may have lost a
|
||||
// newer stable ledger.
|
||||
@@ -3903,7 +4112,7 @@ mod tests {
|
||||
.expect_err("retirement must refuse an unproven authority");
|
||||
assert!(refused.to_string().contains("unsafe authority conflict"), "{refused}");
|
||||
assert_eq!(native_replica_bytes(&store.pools[0].disk_set[0]).await.0, source_bytes);
|
||||
replace_native_pause_backlog_replica(&store, replica_id(1, 1), wedge.rollback.clone()).await;
|
||||
replace_native_pause_backlog_replica(&store, native_replica_id(&store, 1, 1), wedge.rollback.clone()).await;
|
||||
|
||||
// The same records without the conflict converge and retire.
|
||||
for set in 0..2 {
|
||||
@@ -4020,7 +4229,7 @@ mod tests {
|
||||
.authoritative_commit
|
||||
.as_ref()
|
||||
.expect("seed retains the old cohort proof");
|
||||
assert_eq!(committed.replicas, vec![replica_id(0, 0), replica_id(0, 1)]);
|
||||
assert_eq!(committed.replicas, vec![native_replica_id(&store, 0, 0), native_replica_id(&store, 0, 1)]);
|
||||
assert_eq!(loaded.replica_count, 6);
|
||||
if loaded.healthy_replicas == loaded.replica_count {
|
||||
break loaded;
|
||||
@@ -4037,13 +4246,13 @@ mod tests {
|
||||
.authoritative_commit
|
||||
.as_ref()
|
||||
.expect("seed retains the old cohort proof");
|
||||
assert_eq!(committed.replicas, vec![replica_id(0, 0), replica_id(0, 1)]);
|
||||
assert_eq!(committed.replicas, vec![native_replica_id(&store, 0, 0), native_replica_id(&store, 0, 1)]);
|
||||
assert_eq!(loaded.replica_count, 6);
|
||||
assert_eq!(loaded.healthy_replicas, 6);
|
||||
let seeded = loaded
|
||||
.replicas
|
||||
.iter()
|
||||
.find(|replica| replica.id == replica_id(2, 0))
|
||||
.find(|replica| replica.id == native_replica_id(&store, 2, 0))
|
||||
.expect("the canceled caller's admitted seed replica");
|
||||
assert!(matches!(
|
||||
&seeded.state,
|
||||
@@ -4055,7 +4264,7 @@ mod tests {
|
||||
let remaining = loaded
|
||||
.replicas
|
||||
.iter()
|
||||
.find(|replica| replica.id == replica_id(2, 1))
|
||||
.find(|replica| replica.id == native_replica_id(&store, 2, 1))
|
||||
.expect("the remaining seed replica");
|
||||
assert!(matches!(
|
||||
&remaining.state,
|
||||
@@ -4268,7 +4477,12 @@ mod tests {
|
||||
.prepare_scanner_pause_backlog_retirement_for_test(0, source_before.len() * 2)
|
||||
.await
|
||||
.expect("retire the old cohort's first pool");
|
||||
let surviving = [replica_id(1, 0), replica_id(1, 1), replica_id(2, 0), replica_id(2, 1)];
|
||||
let surviving = [
|
||||
native_replica_id(&store, 1, 0),
|
||||
native_replica_id(&store, 1, 1),
|
||||
native_replica_id(&store, 2, 0),
|
||||
native_replica_id(&store, 2, 1),
|
||||
];
|
||||
let newer = claim_scanner_pause_backlog_writer(&old.ledger, now + 2).expect("surviving writer");
|
||||
replace_native_pause_backlog_replica(
|
||||
&store,
|
||||
@@ -4333,7 +4547,7 @@ mod tests {
|
||||
assert_eq!(loaded.ledger, original);
|
||||
assert!(loaded.durable && loaded.stable_matches_ledger);
|
||||
let committed = loaded.authoritative_commit.expect("survivor commit proof");
|
||||
assert_eq!(committed.replicas, vec![replica_id(2, 0), replica_id(2, 1)]);
|
||||
assert_eq!(committed.replicas, vec![native_replica_id(&store, 2, 0), native_replica_id(&store, 2, 1)]);
|
||||
}
|
||||
shutdown_native_retirement_store(store).await;
|
||||
});
|
||||
@@ -4911,7 +5125,135 @@ mod tests {
|
||||
}
|
||||
|
||||
fn replica_id(pool_index: usize, set_index: usize) -> ScannerPauseBacklogReplicaId {
|
||||
ScannerPauseBacklogReplicaId { pool_index, set_index }
|
||||
ScannerPauseBacklogReplicaId {
|
||||
pool_index,
|
||||
set_index,
|
||||
set_id: None,
|
||||
}
|
||||
}
|
||||
|
||||
fn native_replica_id(store: &ECStore, pool_index: usize, set_index: usize) -> ScannerPauseBacklogReplicaId {
|
||||
ScannerPauseBacklogReplicaId::from_set(&store.pools[pool_index].disk_set[set_index])
|
||||
.expect("native replica has an immutable format identity")
|
||||
}
|
||||
|
||||
fn native_replica_ids(store: &ECStore) -> Vec<ScannerPauseBacklogReplicaId> {
|
||||
store
|
||||
.pools
|
||||
.iter()
|
||||
.flat_map(|pool| pool.disk_set.iter())
|
||||
.map(|set| ScannerPauseBacklogReplicaId::from_set(set).expect("native replica format identity"))
|
||||
.collect()
|
||||
}
|
||||
|
||||
fn bound_replica_id(pool_index: usize, identity: u128) -> ScannerPauseBacklogReplicaId {
|
||||
ScannerPauseBacklogReplicaId {
|
||||
pool_index,
|
||||
set_index: 0,
|
||||
set_id: Some(uuid::Uuid::from_u128(identity)),
|
||||
}
|
||||
}
|
||||
|
||||
fn missing_replica(id: ScannerPauseBacklogReplicaId) -> ScannerPauseBacklogReplica {
|
||||
ScannerPauseBacklogReplica {
|
||||
id,
|
||||
revision: Some(DataUsageCacheRevision::Missing),
|
||||
state: ScannerPauseBacklogReplicaState::Missing,
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn bound_membership_survives_pool_reindexing() {
|
||||
let ledger = durable_ledger(50);
|
||||
let previous = bound_replica_id(2, 22);
|
||||
let current = bound_replica_id(1, 22);
|
||||
let added = bound_replica_id(2, 33);
|
||||
let record = replica_record_for_members(&ledger, &ledger, &[previous]);
|
||||
assert_eq!(record.replica_schema_version, 1);
|
||||
let loaded = select_scanner_pause_backlog_replicas(vec![
|
||||
missing_replica(bound_replica_id(0, 11)),
|
||||
decoded_replica(current, &record),
|
||||
missing_replica(added),
|
||||
])
|
||||
.expect("a surviving set keeps its commit proof after its routing index changes");
|
||||
assert_eq!(loaded.ledger, ledger);
|
||||
assert!(loaded.durable);
|
||||
assert_eq!(loaded.authoritative_commit.expect("surviving native proof").replicas, vec![current]);
|
||||
|
||||
let mut replicas = vec![retirement_replica(current, &record)];
|
||||
for id in [bound_replica_id(0, 11), added] {
|
||||
replicas.push(ScannerPauseBacklogRetirementReplica {
|
||||
pool_index: id.pool_index,
|
||||
set_index: id.set_index,
|
||||
set_id: id.set_id,
|
||||
writable: true,
|
||||
data: None,
|
||||
});
|
||||
}
|
||||
let plan = plan_scanner_pause_backlog_retirement(1, &replicas)
|
||||
.expect("reindexed native authority permits safe retirement")
|
||||
.expect("new survivors need the existing ledger");
|
||||
assert_eq!(planned_stable_ledger(&plan), ledger);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn bound_membership_does_not_rebind_a_missing_set_to_its_reused_index() {
|
||||
let ledger = durable_ledger(50);
|
||||
let missing = bound_replica_id(2, 22);
|
||||
let replacement = bound_replica_id(2, 33);
|
||||
let record = replica_record_for_members(&ledger, &ledger, &[missing]);
|
||||
assert!(
|
||||
select_scanner_pause_backlog_replicas(vec![
|
||||
missing_replica(bound_replica_id(0, 11)),
|
||||
decoded_replica(replacement, &record),
|
||||
])
|
||||
.is_err(),
|
||||
"copied bytes on a new set cannot impersonate the missing commit member"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn legacy_membership_binds_only_after_its_existing_proof_is_verified() {
|
||||
let ledger = durable_ledger(50);
|
||||
let legacy = replica_record_for_members(&ledger, &ledger, &[replica_id(2, 0)]);
|
||||
assert_eq!(legacy.replica_schema_version, 1);
|
||||
let current = bound_replica_id(2, 22);
|
||||
let loaded = select_scanner_pause_backlog_replicas(vec![decoded_replica(current, &legacy)])
|
||||
.expect("unchanged legacy membership remains readable");
|
||||
let upgraded = stable_scanner_pause_backlog_record(
|
||||
&loaded.ledger,
|
||||
loaded.authoritative_commit.as_ref(),
|
||||
&scanner_pause_backlog_replica_ids(&loaded.replicas),
|
||||
);
|
||||
upgraded
|
||||
.validate()
|
||||
.expect("the replacement record binds every committed member");
|
||||
assert_eq!(upgraded.replica_schema_version, 1);
|
||||
assert_eq!(upgraded.committed.expect("bound membership").replicas, vec![current]);
|
||||
|
||||
assert!(
|
||||
select_scanner_pause_backlog_replicas(vec![
|
||||
decoded_replica(bound_replica_id(1, 22), &legacy),
|
||||
missing_replica(bound_replica_id(2, 33)),
|
||||
])
|
||||
.is_err(),
|
||||
"an already stranded legacy proof must not guess its old member's identity"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn bound_membership_rejects_nil_duplicate_and_mixed_identities() {
|
||||
let ledger = durable_ledger(50);
|
||||
for members in [
|
||||
vec![bound_replica_id(0, 0)],
|
||||
vec![bound_replica_id(0, 11), bound_replica_id(1, 11)],
|
||||
vec![bound_replica_id(0, 11), bound_replica_id(0, 22)],
|
||||
vec![bound_replica_id(0, 11), replica_id(1, 0)],
|
||||
] {
|
||||
let record = replica_record_for_members(&ledger, &ledger, &members);
|
||||
assert_eq!(record.replica_schema_version, 1);
|
||||
assert!(record.validate().is_err());
|
||||
}
|
||||
}
|
||||
|
||||
fn decoded_replica(
|
||||
@@ -5155,6 +5497,7 @@ mod tests {
|
||||
.map(|set_index| ScannerPauseBacklogReplicaId {
|
||||
pool_index: 0,
|
||||
set_index,
|
||||
set_id: None,
|
||||
})
|
||||
.collect::<Vec<_>>();
|
||||
replica_record_for_members(stable, committed, &replicas)
|
||||
@@ -5167,6 +5510,7 @@ mod tests {
|
||||
ScannerPauseBacklogRetirementReplica {
|
||||
pool_index: id.pool_index,
|
||||
set_index: id.set_index,
|
||||
set_id: id.set_id,
|
||||
writable: true,
|
||||
data: Some(serde_json::to_vec(record).expect("native record bytes")),
|
||||
}
|
||||
@@ -5294,6 +5638,7 @@ mod tests {
|
||||
replicas.extend(targets.iter().map(|id| ScannerPauseBacklogRetirementReplica {
|
||||
pool_index: id.pool_index,
|
||||
set_index: id.set_index,
|
||||
set_id: id.set_id,
|
||||
writable: true,
|
||||
data: None,
|
||||
}));
|
||||
|
||||
@@ -113,17 +113,37 @@ pub(super) async fn setup_scanner_cycle_store_at_path_with_layout_and_disk_prein
|
||||
sets_per_pool: usize,
|
||||
drives_per_set: usize,
|
||||
preinitialize_disks: bool,
|
||||
) -> Arc<ECStore> {
|
||||
setup_scanner_cycle_store_with_pool_order(
|
||||
root,
|
||||
seed_usage_baseline,
|
||||
&(0..pool_count).collect::<Vec<_>>(),
|
||||
sets_per_pool,
|
||||
drives_per_set,
|
||||
preinitialize_disks,
|
||||
)
|
||||
.await
|
||||
}
|
||||
|
||||
pub(super) async fn setup_scanner_cycle_store_with_pool_order(
|
||||
root: &Path,
|
||||
seed_usage_baseline: bool,
|
||||
pool_order: &[usize],
|
||||
sets_per_pool: usize,
|
||||
drives_per_set: usize,
|
||||
preinitialize_disks: bool,
|
||||
) -> Arc<ECStore> {
|
||||
init_ecstore_config_for_scanner_tests();
|
||||
let pool_count = pool_order.len();
|
||||
let mut pools = Vec::with_capacity(pool_count);
|
||||
for pool_index in 0..pool_count {
|
||||
for (pool_index, physical_pool) in pool_order.iter().enumerate() {
|
||||
let mut endpoints = Vec::new();
|
||||
for set_index in 0..sets_per_pool {
|
||||
for disk_index in 0..drives_per_set {
|
||||
let disk_path = if sets_per_pool == 1 {
|
||||
root.join(format!("pool{pool_index}/disk{disk_index}"))
|
||||
root.join(format!("pool{physical_pool}/disk{disk_index}"))
|
||||
} else {
|
||||
root.join(format!("pool{pool_index}/set{set_index}/disk{disk_index}"))
|
||||
root.join(format!("pool{physical_pool}/set{set_index}/disk{disk_index}"))
|
||||
};
|
||||
tokio::fs::create_dir_all(&disk_path)
|
||||
.await
|
||||
@@ -144,7 +164,7 @@ pub(super) async fn setup_scanner_cycle_store_at_path_with_layout_and_disk_prein
|
||||
cmd_line: if pool_count == 1 && sets_per_pool == 1 {
|
||||
"scanner-cycle-metrics".to_string()
|
||||
} else {
|
||||
format!("scanner-cycle-metrics-pool-{pool_index}")
|
||||
format!("scanner-cycle-metrics-pool-{physical_pool}")
|
||||
},
|
||||
platform: format!("OS: {} | Arch: {}", std::env::consts::OS, std::env::consts::ARCH),
|
||||
});
|
||||
|
||||
@@ -591,6 +591,34 @@ pub trait AuthoritativeListOperations: ListOperations {
|
||||
max_keys: i32,
|
||||
) -> Result<Self::ListObjectVersionsInfo, Self::Error>;
|
||||
|
||||
/// Enumerate an exact dotted leaf below identifier-like directory components
|
||||
/// containing ASCII letters, digits, `_` or `-`. Noncandidate object metadata
|
||||
/// is outside the scan. Namespace components may occur at any depth; reserved
|
||||
/// words do not identify a subtree that can be skipped.
|
||||
///
|
||||
/// Returned keys retain the quorum, unversioned, EOF and cancellation
|
||||
/// guarantees of `list_unversioned_objects_authoritative`. Older peers may
|
||||
/// perform extra I/O; the coordinator must filter their extra rows before
|
||||
/// resolving metadata. Callers fence concurrent mutations separately.
|
||||
///
|
||||
/// # Errors
|
||||
/// Rejects an invalid directory prefix/leaf, invalid page size, uncertain
|
||||
/// matching metadata, versioned entries or insufficient read quorum. Backends
|
||||
/// without this capability return Unsupported; ordinary listing is not a fallback.
|
||||
async fn list_identifier_tree_leaves_authoritative(
|
||||
self: Arc<Self>,
|
||||
_bucket: &str,
|
||||
_prefix: &str,
|
||||
_leaf: &str,
|
||||
_marker: Option<String>,
|
||||
_max_keys: i32,
|
||||
) -> Result<Self::ListObjectVersionsInfo, Self::Error>
|
||||
where
|
||||
Self::Error: From<std::io::Error>,
|
||||
{
|
||||
Err(std::io::Error::new(std::io::ErrorKind::Unsupported, "authoritative identifier-tree listing is not supported").into())
|
||||
}
|
||||
|
||||
/// Resolves every set at its version read quorum, including reserved prefixes.
|
||||
///
|
||||
/// # Errors
|
||||
|
||||
@@ -8,6 +8,8 @@ collection:
|
||||
|
||||
## Operations
|
||||
|
||||
- [Candidate release pipeline](operations/release-pipeline.md): frozen inputs, staged acceptance, immutable publication and recovery.
|
||||
|
||||
- [Sampled local batch CPU diagnostics](operations/local-batch-cpu-diagnostics.md) — default-off request-correlated allocation and worker wall/CPU diagnosis.
|
||||
|
||||
- [Retained MRF responsibilities](operations/mrf-responsibilities.md) — unknown bucket identities, bounded probes, and unreadable checkpoint recovery.
|
||||
|
||||
@@ -72,6 +72,8 @@ Ordinary publication rechecks the selected pool against the durable pool metadat
|
||||
|
||||
Running, queued, failed, canceled, and completed decommission states exclude the source from new ordinary publication, including new multipart uploads. Previously created multipart uploads retain their drain path while the source remains non-terminal; terminal source states reject further multipart publication. Failed and canceled entries become writable for new ordinary publication only after an allowed clear operation removes that state. This check does not change repair admission or the separate fence for operations that only release capacity.
|
||||
|
||||
Bucket creation and deletion remove stale bucket usage through namespace maintenance, independently of the scanner publication pause. Failed or canceled decommission state alone does not reject that cleanup. The update checks the logical snapshot revision while holding the fixed and physical object locks, then conditionally publishes to a writable pool; a suspended source remains available for decommission to reconcile against the newer target. Cleanup preserves the scanner epoch and cycle, and retains the bucket namespace and movement guards until its physical writes finish even if the request is canceled. Namespace lease loss still fences the write. Scanner publication continues to require its existing movement admission.
|
||||
|
||||
For mixed batch deletes, only the pools selected to receive new delete markers are publication targets. Exact-version deletions on other pools remain protected by the same pool metadata read fence, without treating the retiring source or an unrelated reserved target as a destination for those markers.
|
||||
|
||||
### Shared Capacity On Healthy Targets
|
||||
@@ -143,6 +145,10 @@ Existing capacity-owner and mutation checks reconcile the pending intent before
|
||||
source cleanup; the replica exception does not clear an unknown intent, rewrite
|
||||
the native target, or change the scanner's committed-membership selection.
|
||||
|
||||
Backlog membership also carries an optional `set_id`, the first disk-slot UUID from the erasure set's persisted format. This identity survives pool removal, reordering, and disk replacement; `pool_index` and `set_index` remain routing locations. Readers resolve bound members by that UUID before checking the existing commit and rollback proofs. A new pool at a removed member's old index cannot impersonate that member. New publications bind every member, while positional records remain readable and gain identities only after their existing authority is verified. Replica schema 1 is retained; 1.0.4 readers ignore the additive identity field and still use positional membership.
|
||||
|
||||
Finish upgrading all scanner-capable nodes before changing pool membership. An older writer can replace a bound record with a positional one. If a legacy record was already stranded by pool reindexing, the missing identity cannot be guessed safely: restore the known topology in which its proof was readable, then let the updated writer bind the membership before changing it again. Do not delete or reset the backlog ledger to bypass this check. Partial writes, canceled callers, missing members, and conflicting authority remain subject to the native CAS and recovery rules above.
|
||||
|
||||
## MinIO Divergence Decisions
|
||||
|
||||
Behavior that is close to MinIO but not byte-for-byte identical. Changing either decision requires an operator compatibility note and updated characterization tests.
|
||||
|
||||
@@ -45,18 +45,9 @@ In a distributed deployment, startup publishes `FullReady` only after that local
|
||||
- Liveness reports process availability and must not depend on storage, IAM,
|
||||
lock quorum, or peer health.
|
||||
- Node readiness (`/health/ready`, `/minio/health/ready`) reports whether this process can serve reads: storage read quorum, IAM, and shared-lock quorum. It is the Kubernetes Service membership signal. Losing write quorum, the pool-metadata writer, or exclusive-lock quorum does not remove a node that can still read. The public health handler still returns 503 until startup publishes `FullReady`, so a process that has never opened S3 admission does not join the Service. Runtime storage diagnostics do not change that startup gate.
|
||||
- A blocked pool metadata writer degrades cluster-write readiness with
|
||||
`pool_meta_write_blocked`. It does not withdraw `/health/ready`, and the node
|
||||
probe omits that reason from `degradedReasons`. Metadata save-gate inspection
|
||||
is bounded to 100 ms; contention reports `pool_metadata_check_timeout` on the
|
||||
cluster-write probe without installing a block. The node probe keeps the last
|
||||
confirmed writable observation for `details.poolMetadata` during a transient
|
||||
inspection timeout, while preserving a confirmed block and failing that
|
||||
detail closed when no fresh observation exists.
|
||||
- The authenticated cluster snapshot extends its existing node-local metadata
|
||||
gate inspection with safe reason, failure phase, and original block time. It
|
||||
distinguishes timeout from a block and changes no admission or recovery
|
||||
decision. Runtime readiness and gate status are separate bounded observations.
|
||||
- Immutable pool metadata admission proofs share the local authority gate as readers. Publication, recovery, identity changes and operations requiring serialization retain exclusive ownership and their existing distributed namespace fences. A multi-phase operation releases its reader before acquiring the writer and revalidates under exclusive ownership; it never upgrades a held reader. The gate is fair, so a writer queued behind a reader that is waiting on namespace I/O also delays later admissions until that reader finishes. Every writer publishes the authority it leaves before releasing exclusive ownership. Health inspection reads that published authority without acquiring the gate, so neither a held nor a queued writer delays it. A writer's changes become visible when it releases the gate; a latch recorded by a transaction outcome after its writer released is shared with the published authority and visible immediately. Mutating paths still check authority under the gate.
|
||||
- A blocked pool metadata writer degrades cluster-write readiness with `pool_meta_write_blocked`. It does not withdraw `/health/ready`, and the node probe omits that reason from `degradedReasons`. Metadata authority inspection reads the state published by the last local writer and does not wait for the save gate, so a held or queued metadata writer does not degrade either probe; a block that writer installs appears when it releases the gate.
|
||||
- The authenticated cluster snapshot extends its existing node-local metadata gate inspection with safe reason, failure phase, and original block time. It changes no admission or recovery decision. Runtime readiness and gate status are separate observations.
|
||||
- Cluster write readiness (`/minio/health/cluster`) requires storage write quorum, the pool-metadata write gate, and exclusive-lock quorum. Use it when a caller must know that new writes can commit. It is not the Service membership probe.
|
||||
- Cluster read readiness uses the storage read-quorum path and cluster-health timeout behavior. Its lock dependency still uses the per-set majority/write-lock health check. Actual shared namespace locks require `n - n/2`, so the cluster read probe is conservative: a four-client set can still admit reads with two clients while that probe returns 503. Node `/health/ready` uses the shared-lock threshold instead, because that is what object reads acquire.
|
||||
- `HEAD` health probes keep header/status semantics and do not require response
|
||||
@@ -91,7 +82,7 @@ For a healthy IAM and metadata writer in a four-node, one-drive-per-node EC 2+2
|
||||
|
||||
With two of four nodes up, GET, HEAD, and List of a bucket that this process has not yet cached still succeed. PUT and other mutations fail because write quorum and exclusive locks are gone.
|
||||
|
||||
The node path reads local disk-handle health and reuses the same reachable-host observation as its lock dependency, including the existing `RUSTFS_HEALTH_READINESS_CACHE_TTL_MS` cache. Only `Online` drives count; a reachable host with a `Returning` drive does not yet prove data I/O has recovered. It does not call cluster `storage_info`, local `disk_info`, or add disk-info RPCs. The entire storage inventory snapshot has a separate 100 ms wait budget; expiry reports `storage_readiness_check_timeout` and fails closed. Pool metadata inspection retains its own 100 ms budget; a timeout is counted by `rustfs_pool_metadata_check_timeouts_total` and uses the last confirmed node-local gate state only within the same cache TTL. These observations are not an atomic cluster snapshot and do not bypass the existing lock-probe timing or cache policy.
|
||||
The node path reads local disk-handle health and reuses the same reachable-host observation as its lock dependency, including the existing `RUSTFS_HEALTH_READINESS_CACHE_TTL_MS` cache. Only `Online` drives count; a reachable host with a `Returning` drive does not yet prove data I/O has recovered. It does not call cluster `storage_info`, local `disk_info`, or add disk-info RPCs. The entire storage inventory snapshot has a separate 100 ms wait budget; expiry reports `storage_readiness_check_timeout` and fails closed. Pool metadata inspection reads the authority published by the last local writer and does not wait for the save gate. These observations are not an atomic cluster snapshot and do not bypass the existing lock-probe timing or cache policy.
|
||||
|
||||
The new fields are additive. Their absence in an older response is not evidence of storage quorum. Minimal responses still contain only the existing top-level fields, liveness remains dependency-independent, and HEAD responses remain bodyless.
|
||||
|
||||
|
||||
@@ -26,7 +26,7 @@ RustFS S3 Tables is an Iceberg REST Catalog and table-bucket implementation on t
|
||||
| `/_iceberg/v1` | Supported compatibility alias | MinIO AIStor-style alias; smoke profile defaults to signing name `s3tables`. |
|
||||
| S3 object data plane | Supported | Data, metadata, manifest, and delete files are ordinary S3 objects with table-aware policy checks on warehouse paths. |
|
||||
| S3 table-bucket listings | Supported | `ListObjects`, `ListObjectsV2`, `ListObjectVersions`, metadata-list extensions, and multipart-upload listings omit warehouse objects unless the caller has `admin:GetTableMetadata` for that table, and always omit internal `.rustfs-table` keys. Protected cursors are encrypted with a key derived from the cluster root credentials, so rotating those credentials invalidates cursors already issued. |
|
||||
| Object-backed warehouse-index recovery | Supported with bounded request fallback | Data-plane requests repair or verify a missing warehouse-prefix index by listing the catalog prefix. Only table entries count toward the 4,096-object limit, so maintenance reports and other per-table objects cannot exhaust it; the listing itself is bounded to 64 pages plus one page per table entry already listed, and after each page the cursor skips the rest of the maintenance subtree the page ended in, so a report flood under one table costs a bounded number of pages. Larger catalogs fail closed with a retryable service error and require operator backfill. An optional table-entry directory (see Maintenance Matrix) answers these lookups without the prefix listing. |
|
||||
| Object-backed warehouse-index recovery | Supported with bounded request fallback | Data-plane requests repair or verify a missing warehouse-prefix index using authoritative table-entry pages. The scan selects `table-entry.json` below identifier-like directory components at any depth; namespace components named `tables` or `maintenance` remain valid. Noncandidate report names are filtered in directory batches before inventory retention and metadata reads, so report history does not consume ownership pages. Selected entries retain GET-equivalent quorum and corruption checks. More than 4,096 table entries fails closed with a retryable service error and requires operator backfill. An optional table-entry directory (see Maintenance Matrix) answers these lookups without the prefix listing. |
|
||||
| Table bucket enablement | Supported | A regular bucket is enabled for catalog use and addressed as the REST catalog warehouse. |
|
||||
| Catalog-vended table credentials | Automated when enabled | Disabled by default. LoadTable vends credentials only when `X-Iceberg-Access-Delegation` contains the exact `vended-credentials` token; the dedicated credentials endpoint uses the same issuer path. |
|
||||
| AWS S3 Tables endpoint shape | Profile generator | Generates the AWS catalog URI and warehouse ARN shape for migration docs. API parity not claimed. |
|
||||
@@ -132,6 +132,8 @@ Object-backed commit publication identifies every catalog object it validated by
|
||||
|
||||
An optional per-bucket directory of table entries under `.rustfs.sys` answers data-plane index-miss lookups without listing the catalog prefix. It is off by default: `RUSTFS_TABLE_CATALOG_DIRECTORY_INDEX=true` and `RUSTFS_TABLE_CATALOG_DIRECTORY_INDEX_FLEET_CONFIRMED=true` on every node, then `POST /iceberg/v1/{warehouse}/catalog/directory` (`admin:MigrateTableCatalog`) advances a backfill of 128 catalog objects per call until the directory reports `Ready`; `GET` reports the phase and `DELETE` rolls the directory back. Entry writers add a directory candidate only while the directory exists, and the backfill covers earlier entries. The directory state occupies the backing-migration fence object: nodes without this feature answer catalog mutations on that bucket with a parse error until they are upgraded, which is what the fleet-confirmed flag attests, and restores require disabling the directory first. Object catalog backups exclude the derived directory state and candidates. Re-enable the directory after a restore to rebuild it from the restored entries.
|
||||
|
||||
Fallback ownership scans, including the check before metadata cleanup, use the optional `identifier_tree_leaf` walk selector. An absent selector preserves ordinary listing behavior. Older peers ignore the additive field; the coordinator filters their noncandidate rows before metadata resolution and retains read-quorum checks for every selected entry. Old peers may do more I/O or fail on unrelated corrupt metadata, so a mixed-version scan can refuse work until those peers are upgraded. Such failures never establish owner absence. This selector changes no catalog or on-disk format and requires no fleet-confirmation flag.
|
||||
|
||||
|
||||
|
||||
| Capability | Status | Claim |
|
||||
|
||||
@@ -77,7 +77,7 @@ An unreadable replica, lost fence, or conditional-write conflict leaves the orig
|
||||
|
||||
- The first block emits `decommission_state` with `state=pool_metadata_blocked`, `reason`, `phase`, and `blocked_since`. A change in recovery failure classification emits `state=pool_metadata_recovery_pending`; successful recovery emits `state=pool_metadata_recovered` with the original timestamp.
|
||||
- `rustfs_pool_metadata_blocks_total{reason}` and `rustfs_pool_metadata_recoveries_total` count block and recovery transitions. The original cause and phase remain attached to local typed errors; storage/RPC error numbers and on-disk formats are unchanged.
|
||||
- Node and cluster-write readiness include `pool_meta_write_blocked`. Waiting for the metadata save mutex is bounded to 100 ms and reports `pool_metadata_check_timeout`, not a persistent block. Cluster probes retain their existing cache and overall timeout behavior. Liveness and cluster-read quorum checks are unchanged. The authenticated node-local status and safe error fields are described in [S3 write failure diagnostics](s3-write-failure-diagnostics.md).
|
||||
- Cluster-write readiness reports `pool_meta_write_blocked`; node readiness reports the writer only through `details.poolMetadata`. Readiness reads the authority published by the last local metadata writer instead of waiting for the save gate, so an active or queued metadata transaction does not degrade readiness by itself. A block installed by a writer appears when that writer releases the gate. Cluster probes retain their existing cache and overall timeout behavior. Liveness and cluster-read quorum checks are unchanged. The authenticated node-local status and safe error fields are described in [S3 write failure diagnostics](s3-write-failure-diagnostics.md).
|
||||
|
||||
Pool metadata CAS explicitly selects `TailDrainedWithOwnerRepair`: it waits for the complete rename result and returns repair responsibility to its existing deployment and transaction owner when write quorum commits but a replica fails or signatures diverge. Cache invalidation and the committed cleanup handoff precede that error. Ordinary system metadata retains quorum-success semantics under `Quorum` and `TailDrained`; draining a tail does not require every disk to succeed. A repair-required completion does not enter rollback cleanup or wait for another temporary-data deletion. The committed bytes are not rolled back. Generic MRF and heal-channel requests cannot supply the deployment and transaction authority needed to repair system metadata; pool CAS retains the uncertain outcome and its existing recovery fence. Ordinary object repair keeps its bucket-incarnation fencing.
|
||||
|
||||
|
||||
@@ -0,0 +1,79 @@
|
||||
# Candidate acceptance and release publication
|
||||
|
||||
RustFS builds the final product version before publication. A candidate is identified by its Actions run and `<version>-preview.<run_number>`; this identifier is not a Git tag or the executable's displayed version. Failed candidates consume no public version. A public tag is created only after the complete artifact manifest is approved.
|
||||
|
||||
## Entry points
|
||||
|
||||
| Workflow | Input | Result | Public writes |
|
||||
|---|---|---|---|
|
||||
| `release-candidate.yml` | Exact final version and full source SHA | Four Linux archives, native checks, upgrade results, Linux manifest | None |
|
||||
| `release-complete.yml` | Successful candidate run, accepted Linux manifest SHA-256, backlog acceptance report | Desktop archives, DEB/RPM, OCI images, Helm chart, full certification and sealed manifest | None |
|
||||
| `release-publish.yml` | Successful completion run and approved final manifest SHA-256 | Annotated tag, GitHub Release, R2 files, registry images, stable pointers and Helm index | Explicit publication |
|
||||
|
||||
The builder workflows (`build.yml`, `package.yml`, `docker.yml`, `helm-package.yml`) are reusable stages. Only `release-publish.yml` has release write credentials. Development builds retain their artifact names and download channel. Daily development builds now select Linux only; macOS/Windows development artifacts are refreshed by an explicit manual build, and both remain mandatory for a completed release candidate. Tag pushes no longer start a second compilation.
|
||||
|
||||
## Freeze and build
|
||||
|
||||
Select a reviewed source commit whose Cargo version equals the intended final version. Finish exact-source CI before dispatching the candidate. The candidate workflow itself must run from upstream main. Unrelated main commits do not require rebuilding an accepted source.
|
||||
|
||||
```bash
|
||||
gh workflow run release-candidate.yml --ref main \
|
||||
-f version='<version>' -f source_sha='<full-source-sha>'
|
||||
```
|
||||
|
||||
The initial job records the source and workflow SHAs, release recipe, dependency lock, exact stable Rust version, Console tag/asset/SHA-256, CLI assets, previous stable release and validation plan. All platforms consume that record; none independently follows `latest`. The source commit must contain the same reviewed release recipe as the dispatcher. Relevant recipe changes require a new candidate; unrelated source changes on main do not invalidate the frozen record.
|
||||
|
||||
Linux is built first. Each archived executable must display the final version, expected target, source commit and compiler version, then start natively, serve the Console, and pass an authenticated S3 bucket/object round trip. The upgrade lane runs the existing pinned compatibility matrix plus the stable version frozen at candidate creation. It compiles the harness and runs the accepted executable; it does not rebuild a substitute server. A zero-test invocation fails.
|
||||
|
||||
The Linux manifest is partial evidence and cannot authorize publication. Download the exact artifacts for field acceptance. Record the source, manifest digest, run URL and actual tested binary digests in backlog. Required field acceptance includes Scanner/Heal and multi-pool behavior, major features, memory, high-concurrency 503s, performance against the agreed baseline, previous-version data integrity and upgrade recovery. The agreed stability duration is a release-specific requirement, not an implicit workflow default.
|
||||
|
||||
## Complete the accepted candidate
|
||||
|
||||
After the Linux report passes, dispatch `release-complete.yml` with the exact digest printed by the candidate run. The `release-linux-acceptance` environment is the operator gate. Configure its protection and allowed main branch in repository settings; a typed digest and report identify what is being accepted, while the workflow independently revalidates automated evidence.
|
||||
|
||||
```bash
|
||||
gh workflow run release-complete.yml --ref main \
|
||||
-f candidate_run_id='<run-id>' \
|
||||
-f linux_manifest_sha256='<accepted-linux-manifest-sha256>' \
|
||||
-f acceptance_report='https://github.com/rustfs/backlog/issues/<issue>#issuecomment-<comment>'
|
||||
```
|
||||
|
||||
macOS and Windows now build and run the same native checks. Packages and images wrap the accepted Linux binaries. DEB/RPM extraction must reproduce the recorded binary digest. Both OCI variants are checked on amd64 and arm64 and scanned before sealing. Vulnerability scans retain the repository's report-only policy; scanner execution errors still fail the stage. Image OS packages are resolved during packaging, then the actual resulting OCI bytes are accepted and preserved; this is not a promise of reproducible package-manager output.
|
||||
|
||||
The Helm chart is tested and packaged before sealing, with chart and application versions equal to the final product version. Full certification requires every platform and upgrade case from the original plan. The seal includes artifact hashes, raw evidence, source/input identities, acceptance report, checksums, SBOM and provenance. Successful Linux receipts remain valid while desktop results are added because each receipt binds its own exact archive and executable, the frozen plan, candidate identity and trusted producer.
|
||||
|
||||
Native smoke checks are not a substitute for Console UI and rc acceptance. Complete the release skill's UI/client checks against these exact artifacts, including desktop checks applicable to the release, before asking for final publication approval.
|
||||
|
||||
## Approve and publish
|
||||
|
||||
Review the complete run and final `release-manifest.json`. Human approval binds its SHA-256, source and version. A changed manifest requires renewed approval. No source version bump, compilation, package generation or image build occurs during publication.
|
||||
|
||||
```bash
|
||||
gh workflow run release-publish.yml --ref main \
|
||||
-f complete_run_id='<successful-completion-run-id>' \
|
||||
-f manifest_sha256='<approved-final-manifest-sha256>'
|
||||
```
|
||||
|
||||
The publisher downloads the artifact from the exact successful main workflow run, checks GitHub's artifact digest, recomputes file hashes and reruns complete certification. It creates an annotated tag whose message binds the source and manifest. An existing tag or asset with different bytes is a hard failure; tags are never moved and uploads never clobber assets.
|
||||
|
||||
Versioned GitHub assets, R2 objects and Docker Hub/GHCR/Quay manifests are uploaded and read back before the Release becomes public. OCI promotion uses `--preserve-digests`; registry conversion fails rather than silently accepting a new image. R2 creation uses atomic `If-None-Match: *`. The small evidence bundle persists with the public Release after Actions artifacts expire. Provenance is descriptive, not a claim of cryptographic attestation.
|
||||
|
||||
Publication uses a per-version lock and a shared channel lock, both without cancelling an active run. Stable channels advance only after versioned delivery succeeds. GitHub Latest acts as a monotonic marker during partial channel updates, so an older retry cannot downgrade Docker tags, R2 aliases or installer pointers. Prereleases do not advance stable channels. Helm publishes the already accepted chart after versioned images exist. GitHub's legacy `*-latest.zip` download names remain aliases of the accepted bytes; they become default downloads only when GitHub Latest advances.
|
||||
|
||||
Cross-registry publication is not atomic. A channel job can fail after some verified destinations advance. Resume the same run/seal: matching destinations are reused and conflicting ones stop the workflow. Retrying never recompiles. Do not claim complete publication until the channel receipt and all public endpoints are verified.
|
||||
|
||||
## Failure and retention
|
||||
|
||||
- Infrastructure failure: rerun failed jobs in the same run. Frozen inputs, successful platform/package/image artifacts and completed seals are reused. Missing or expired artifacts fail closed.
|
||||
- Product or build-input change before tagging: create a new candidate run and repeat affected acceptance. Do not mutate the old candidate.
|
||||
- Failure after tagging: resume publication of exactly that seal. A code fix requires a new patch version; never move the tag.
|
||||
- New main commits: retain evidence for unchanged source, recipe and inputs. Recheck only affected boundaries; do not restart the entire matrix because an unrelated PR merged.
|
||||
- Artifacts are retained for 30 days while awaiting acceptance. Public-repository Actions artifacts and staging identifiers are not confidential storage. Preserve important failure evidence in task-owned storage and backlog.
|
||||
|
||||
## Migration and workflow maintenance
|
||||
|
||||
Finish any in-flight legacy tag release, including its downstream Docker/package/Helm jobs, before enabling this cutover on main. Existing tags and Releases retain their identities. Do not feed a legacy public preview binary into the new promotion path: it embeds a different version and was not built from the frozen-input contract.
|
||||
|
||||
All workflow responsibilities, triggers and schedule ownership remain registered in `.github/scheduled-validations.json`. The migration does not re-enable workflows disabled in GitHub. PR checks, cache warming, nightly development builds, functional chains, lab tests and monitoring retain their distinct purposes; shared source or cache does not make different feature/platform tests redundant. The watchdog allocates a runner only for relevant unsuccessful scheduled/chained runs, while its registry still determines ownership.
|
||||
|
||||
Run `actionlint`, the release contract suite and the workflow registry validator after changes. The Security Audit workflow also exercises actual GitHub artifact transport, DEB/RPM package extraction, both multi-platform OCI builders, local registry promotion and archive scanning. Native packaging jobs run only for relevant changes or a manual audit. Their payloads are explicit transport fixtures, not product release acceptance. See the [release skill](../../.agents/skills/rustfs-release-publish/SKILL.md) for field acceptance, human confirmation and post-release follow-up.
|
||||
@@ -97,7 +97,7 @@ When the whole cluster (or several nodes) went down and nodes are brought back o
|
||||
|
||||
2. Logs say what the node waits for. The IAM recovery loop retries with backoff and logs `event="iam_bootstrap_retry_failed"` with an actionable `hint` field (for example, "storage read quorum not met yet; waiting for enough cluster nodes/disks to come online"). After repeated failures the level escalates from WARN to ERROR; this still does not kill the process.
|
||||
3. Recovery is automatic. Once storage read quorum is available, pending nodes can finish IAM bootstrap on the next retry. `/health/ready` returns `200` when startup has published `FullReady` and the node can still serve reads (storage read quorum, IAM, and shared-lock quorum). Write quorum is reported separately and is required by `/minio/health/cluster`, not by Service membership. Restarting pending nodes does not speed this up.
|
||||
4. Check readiness detail while waiting. `/health/ready` (and `/minio/health/ready`) separate `storage` / `poolMetadata` / `iam` / `lock` readiness. `storage.ready` follows read quorum (`readinessScope` is `read_quorum`); `storage.readQuorum` and `storage.writeQuorum` show the two observations. `poolMetadata.ready` is the metadata writer and does not by itself withdraw `/health/ready`. `degradedReasons` on this probe names read-path causes such as `storage_quorum_unavailable` and `storage_and_lock_unavailable`. `pool_meta_write_blocked` and `pool_metadata_check_timeout` stay on `/minio/health/cluster`. See the [storage detail contract](../architecture/readiness-matrix.md#storage-detail-contract) for probe scopes and sampling limits:
|
||||
4. Check readiness detail while waiting. `/health/ready` (and `/minio/health/ready`) separate `storage` / `poolMetadata` / `iam` / `lock` readiness. `storage.ready` follows read quorum (`readinessScope` is `read_quorum`); `storage.readQuorum` and `storage.writeQuorum` show the two observations. `poolMetadata.ready` is the metadata writer and does not by itself withdraw `/health/ready`. `degradedReasons` on this probe names read-path causes such as `storage_quorum_unavailable` and `storage_and_lock_unavailable`. `pool_meta_write_blocked` stays on `/minio/health/cluster`. See the [storage detail contract](../architecture/readiness-matrix.md#storage-detail-contract) for probe scopes and sampling limits:
|
||||
|
||||
```bash
|
||||
curl -s http://<node>:9000/health/ready | jq
|
||||
|
||||
@@ -45,15 +45,14 @@ The probe round timeout is configured independently; see [Admin peer probe timeo
|
||||
|
||||
## Correlate bounded diagnostics
|
||||
|
||||
Normal operation does not require success logs at WARN. Request counters remain available with WARN logging, while runtime readiness diagnostics distinguish `pool_meta_write_blocked`, `pool_metadata_check_timeout`, and insufficient storage quorum. Do not clear a metadata write fence merely to make readiness green.
|
||||
Normal operation does not require success logs at WARN. Request counters remain available with WARN logging, while runtime readiness diagnostics distinguish `pool_meta_write_blocked` from insufficient storage quorum. Do not clear a metadata write fence merely to make readiness green.
|
||||
|
||||
Query the authenticated `/rustfs/admin/v4/cluster/snapshot` endpoint on the affected node. `snapshot.pool_meta_write_gate` describes that node's metadata writer, not a fleet-wide aggregate. Its existing booleans are preserved; `state` and optional block details extend the same bounded, read-only gate inspection. This starts no recovery, disk reads, or additional RPCs. Runtime readiness and the gate are inspected separately, so a whole cluster snapshot is not atomic across sections.
|
||||
Query the authenticated `/rustfs/admin/v4/cluster/snapshot` endpoint on the affected node. `snapshot.pool_meta_write_gate` describes that node's metadata writer, not a fleet-wide aggregate. Its existing booleans are preserved; `state` and optional block details extend the same read-only inspection, which reads the authority published by the node's last metadata writer without waiting for the save gate. A block that an active writer installs appears when that writer releases the gate. Nodes running earlier releases can also report `check_timeout`, the `pool_metadata_check_timeout` readiness reason, and `rustfs_pool_metadata_check_timeouts_total` when their 100 ms gate inspection expired; current nodes emit none of them. This starts no recovery, disk reads, or additional RPCs. Runtime readiness and the gate are inspected separately, so a whole cluster snapshot is not atomic across sections.
|
||||
|
||||
| `state` | Meaning |
|
||||
| --- | --- |
|
||||
| `writable` | No metadata write block was observed. Other dependencies may still make the node unready. |
|
||||
| `blocked` | A metadata write block was observed; `reason`, `phase`, and `sinceUnixSecs` describe its typed failure context. |
|
||||
| `check_timeout` | The existing 100 ms inspection budget expired. Readiness remains false, but a write block is not asserted. |
|
||||
| `unavailable` | The object store or a recognized metadata observation was unavailable; no block details are invented. |
|
||||
|
||||
For `blocked`, `reason` reuses the metadata failure classification, `phase` identifies the operation stage that caused the block (not live recovery-worker progress), and `sinceUnixSecs` is the original block time in Unix seconds. Polling does not reset that time. Recovery replaces the observation with `writable` and removes the previous block details. An unavailable store reports `writesReady=false` with `state="unavailable"`. Older responses may omit `state` and block details; missing fields alone do not prove a healthy writer. Operation text, raw replica errors, disk paths, and credentials are excluded. See [Pool metadata recovery](pool-metadata-recovery.md) for recovery and escalation boundaries.
|
||||
|
||||
@@ -8,6 +8,8 @@ The classifier reads the pull request base revision's planner and policy. It com
|
||||
|
||||
The committed policy is `enforced`: ordinary Rust changes select their reverse dependency closure plus the fixed core inventory, and unrelated changes can omit the Connect specialization. Package-owned fixtures, non-Rust inputs, build scripts, manifests, unknown paths and non-PR events retain full validation. Existing documentation and isolated E2E scopes remain conservative. Changing a PR's own policy cannot authorize skipping its base revision's requirements. A base planner predating the current workflow contract clears all narrower outputs and selects the full matrix. Set the policy back to `shadow` to retain proposed selections while restoring full ordinary-code execution; do not remove core inventories or negative checks as a rollback.
|
||||
|
||||
Stable installation-version alignment has one content-based exception in enforced mode: literal version fields in `flake.nix`, `helm/rustfs/Chart.yaml` and `rustfs.spec`, plus a standard prepended RPM release note, can use the documentation scope. Quick Checks, Typos, CLA and the required aggregate remain. The planner compares both complete Git blobs and requires ordinary non-executable files; changes to build logic, CLI pins, chart settings, RPM macros or older changelog entries retain the full gate. New/deleted files, mode changes, prerelease syntax, unavailable history, shadow rollout and explicit full-path policy overrides also retain full validation. Product-code changes in the same PR retain their normal dependency and core coverage, and main pushes, merge groups, scheduled and manual runs still validate the complete candidate. This exception does not establish artifact availability or authorize a release; post-release checks still verify the published target before updating defaults.
|
||||
|
||||
`Test and Lint` remains the required aggregate name. Every verification producer and consumer is a direct dependency. The verifier accepts only the effective plan's exact expected success/skipped outcomes; failure, cancellation, missing results, extra jobs or a plan for a different candidate fail. Quick Checks and CLA retain their separate required names.
|
||||
|
||||
## Build and execute
|
||||
|
||||
@@ -211,9 +211,7 @@ pub(crate) struct ClusterPoolMetaWriteGateView {
|
||||
|
||||
impl From<ClusterPoolMetaWriteGateSnapshot> for ClusterPoolMetaWriteGateView {
|
||||
fn from(snapshot: ClusterPoolMetaWriteGateSnapshot) -> Self {
|
||||
let state = if snapshot.check_timed_out {
|
||||
"check_timeout"
|
||||
} else if snapshot.writes_ready {
|
||||
let state = if snapshot.writes_ready {
|
||||
"writable"
|
||||
} else if snapshot.write_blocked || snapshot.transaction_aborted {
|
||||
"blocked"
|
||||
@@ -706,7 +704,6 @@ fn summarize_storage_readiness(snapshot: &ClusterReadOnlySnapshot) -> Capability
|
||||
.filter_map(|reason| match reason {
|
||||
ReadinessDegradedReason::StorageQuorumUnavailable
|
||||
| ReadinessDegradedReason::PoolMetaWriteBlocked
|
||||
| ReadinessDegradedReason::PoolMetadataCheckTimeout
|
||||
| ReadinessDegradedReason::StorageAndIamUnavailable
|
||||
| ReadinessDegradedReason::StorageAndLockUnavailable
|
||||
| ReadinessDegradedReason::StorageIamAndLockUnavailable => Some(reason.as_str()),
|
||||
@@ -1120,14 +1117,6 @@ mod tests {
|
||||
|
||||
for (snapshot, state) in [
|
||||
(ClusterPoolMetaWriteGateSnapshot::default(), "writable"),
|
||||
(
|
||||
ClusterPoolMetaWriteGateSnapshot {
|
||||
writes_ready: false,
|
||||
check_timed_out: true,
|
||||
..Default::default()
|
||||
},
|
||||
"check_timeout",
|
||||
),
|
||||
(
|
||||
ClusterPoolMetaWriteGateSnapshot {
|
||||
writes_ready: false,
|
||||
|
||||
@@ -108,7 +108,6 @@ pub struct ClusterListingDiagnosticsSnapshot {
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub struct ClusterPoolMetaWriteGateSnapshot {
|
||||
pub writes_ready: bool,
|
||||
pub check_timed_out: bool,
|
||||
pub write_blocked: bool,
|
||||
pub transaction_aborted: bool,
|
||||
pub pool_meta_absent: bool,
|
||||
@@ -124,7 +123,6 @@ impl Default for ClusterPoolMetaWriteGateSnapshot {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
writes_ready: true,
|
||||
check_timed_out: false,
|
||||
write_blocked: false,
|
||||
transaction_aborted: false,
|
||||
pool_meta_absent: false,
|
||||
@@ -230,7 +228,6 @@ async fn current_pool_meta_write_gate_snapshot() -> ClusterPoolMetaWriteGateSnap
|
||||
let status = store.pool_meta_write_gate_status().await;
|
||||
ClusterPoolMetaWriteGateSnapshot {
|
||||
writes_ready: status.writes_ready,
|
||||
check_timed_out: status.check_timed_out,
|
||||
write_blocked: status.write_blocked,
|
||||
transaction_aborted: status.transaction_aborted,
|
||||
pool_meta_absent: status.pool_meta_absent,
|
||||
|
||||
+65
-3
@@ -28,6 +28,19 @@ const MAX_VERSIONS_EXCEEDED_MESSAGE: &str = "You've exceeded the limit on the nu
|
||||
const SLOW_DOWN_READ_CODE: &str = "SlowDownRead";
|
||||
const SLOW_DOWN_READ_MESSAGE: &str = "Resource requested is unreadable, please reduce your request rate";
|
||||
const NAMESPACE_LOCK_TIMEOUT_CAUSE: &str = "namespace_lock_timeout";
|
||||
const RPC_ADMISSION_METADATA_KEY: &str = "x-rustfs-rpc-admission";
|
||||
const RPC_REPLAY_ADMISSION_METADATA_VALUE: &str = "replay-cache";
|
||||
|
||||
/// Preserve temporary replay admission refusal through the RPC and S3 boundaries.
|
||||
pub(crate) fn rpc_replay_admission_exhausted_status() -> tonic::Status {
|
||||
let mut status = tonic::Status::resource_exhausted("RPC replay admission exhausted");
|
||||
status.metadata_mut().insert(
|
||||
RPC_ADMISSION_METADATA_KEY,
|
||||
tonic::metadata::MetadataValue::from_static(RPC_REPLAY_ADMISSION_METADATA_VALUE),
|
||||
);
|
||||
status
|
||||
}
|
||||
|
||||
/// Public message for `StorageError::UnsupportedDisk` on a request path. The
|
||||
/// only request-time producer is the delete-transaction protocol when a
|
||||
/// participant answers Unimplemented (a peer still on a release without the
|
||||
@@ -588,9 +601,14 @@ fn body_read_error_code(err: &(dyn std::error::Error + 'static)) -> Option<S3Err
|
||||
return Some(S3ErrorCode::BadDigest);
|
||||
}
|
||||
|
||||
rpc_unavailable |= error
|
||||
.downcast_ref::<tonic::Status>()
|
||||
.is_some_and(|status| status.code() == tonic::Code::Unavailable);
|
||||
rpc_unavailable |= error.downcast_ref::<tonic::Status>().is_some_and(|status| {
|
||||
status.code() == tonic::Code::Unavailable
|
||||
|| (status.code() == tonic::Code::ResourceExhausted
|
||||
&& status
|
||||
.metadata()
|
||||
.get(RPC_ADMISSION_METADATA_KEY)
|
||||
.is_some_and(|value| value.as_encoded_bytes() == RPC_REPLAY_ADMISSION_METADATA_VALUE.as_bytes()))
|
||||
});
|
||||
upload_limit_exceeded |= error.is::<UploadLimitExceeded>();
|
||||
server_source_failed |= error.is::<ServerSideSourceReadError>();
|
||||
client_body_timed_out |= error.is::<ClientBodyReadTimeout>();
|
||||
@@ -1742,6 +1760,50 @@ mod tests {
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn replay_admission_refusal_preserves_retryable_status_across_io_boundaries() {
|
||||
let status = rpc_replay_admission_exhausted_status;
|
||||
let original = StorageError::from(status());
|
||||
let cloned = original.clone();
|
||||
let wire = status().into_http::<()>();
|
||||
let received = tonic::Status::from_header_map(wire.headers()).expect("replay admission status on the wire");
|
||||
for error in [
|
||||
ApiError::from(original),
|
||||
ApiError::from(cloned),
|
||||
ApiError::from(StorageError::from(received)),
|
||||
ApiError::from(StorageError::Io(IoError::other(IoError::other(status())))),
|
||||
ApiError::from(IoError::other(status())),
|
||||
ApiError::from(IoError::other(StorageError::from(status()))),
|
||||
] {
|
||||
assert_eq!(error.code, S3ErrorCode::ServiceUnavailable);
|
||||
assert_eq!(error.diagnostic().rpc_code, Some(tonic::Code::ResourceExhausted));
|
||||
assert_eq!(S3Error::from(error).status_code(), Some(StatusCode::SERVICE_UNAVAILABLE));
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn replay_admission_marker_requires_its_exact_value_and_resource_code() {
|
||||
for code in [
|
||||
tonic::Code::Unauthenticated,
|
||||
tonic::Code::Internal,
|
||||
tonic::Code::ResourceExhausted,
|
||||
] {
|
||||
for marker in ["replay-cache", "unknown-admission"] {
|
||||
if code == tonic::Code::ResourceExhausted && marker == "replay-cache" {
|
||||
continue;
|
||||
}
|
||||
let mut status = tonic::Status::new(code, "RPC replay admission exhausted");
|
||||
status
|
||||
.metadata_mut()
|
||||
.insert(RPC_ADMISSION_METADATA_KEY, marker.parse().expect("valid test marker"));
|
||||
let error = ApiError::from(StorageError::from(status));
|
||||
assert_eq!(error.code, S3ErrorCode::InternalError);
|
||||
}
|
||||
}
|
||||
let error = ApiError::from(IoError::other("RPC replay admission exhausted"));
|
||||
assert_eq!(error.code, S3ErrorCode::InternalError);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn rpc_unavailable_mapping_does_not_match_other_codes_or_untyped_messages() {
|
||||
for code in [
|
||||
|
||||
@@ -614,22 +614,6 @@ mod tests {
|
||||
});
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[serial]
|
||||
fn writable_node_readiness_keeps_200_for_transient_pool_metadata_timeout() {
|
||||
let mut report = ready_report();
|
||||
report.degraded_reasons = vec![ReadinessDegradedReason::PoolMetadataCheckTimeout];
|
||||
|
||||
with_var(rustfs_config::ENV_HEALTH_MINIMAL_RESPONSE_ENABLE, Some("false"), || {
|
||||
let parts = build_health_response_parts(Method::GET, HealthProbe::Readiness, Some(&report), "rustfs", None, None);
|
||||
|
||||
assert_eq!(parts.status_code, StatusCode::OK);
|
||||
let payload = parts.payload.expect("GET readiness body");
|
||||
assert_eq!(payload["ready"], true);
|
||||
assert_eq!(payload["degradedReasons"], json!(["pool_metadata_check_timeout"]));
|
||||
});
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn readiness_collects_object_stalls_and_recovers_on_completion() {
|
||||
let object_traffic_health = ObjectTrafficHealth::enabled_for_test(Duration::ZERO);
|
||||
|
||||
@@ -3405,6 +3405,16 @@ fn hyper_body_write_close_kind(debug: &str) -> Option<&'static str> {
|
||||
}
|
||||
}
|
||||
|
||||
fn rpc_auth_failure_status(error: &std::io::Error) -> Status {
|
||||
if storage::rpc_auth_is_resource_exhausted(error) {
|
||||
crate::error::rpc_replay_admission_exhausted_status()
|
||||
} else if storage::tonic_rpc_auth_failure_reason(error) == "stale_boot_epoch" {
|
||||
Status::unavailable("RPC boot epoch changed")
|
||||
} else {
|
||||
Status::unauthenticated("No valid auth token")
|
||||
}
|
||||
}
|
||||
|
||||
#[allow(clippy::result_large_err)]
|
||||
fn check_auth(req: Request<()>) -> std::result::Result<Request<()>, Status> {
|
||||
let local_node_name =
|
||||
@@ -3467,6 +3477,12 @@ fn check_auth(req: Request<()>) -> std::result::Result<Request<()>, Status> {
|
||||
INTERNODE_TRANSPORT_BACKEND_GRPC,
|
||||
failure_reason,
|
||||
);
|
||||
let rejection = rpc_auth_failure_status(&e);
|
||||
if rejection.code() == tonic::Code::ResourceExhausted {
|
||||
// Existing scoped counters describe overload without producing an
|
||||
// authentication error log for every otherwise valid request.
|
||||
return rejection;
|
||||
}
|
||||
if failure_reason == "stale_boot_epoch" {
|
||||
// The signature is valid, but the peer restarted before this request. Reject it
|
||||
// before execution and let the client retry after its authenticated epoch refresh.
|
||||
@@ -3485,7 +3501,7 @@ fn check_auth(req: Request<()>) -> std::result::Result<Request<()>, Status> {
|
||||
error = %e,
|
||||
"RPC boot epoch changed; request rejected for retry"
|
||||
);
|
||||
Status::unavailable("RPC boot epoch changed")
|
||||
rejection
|
||||
} else {
|
||||
error!(
|
||||
event = EVENT_RPC_SIGNATURE_VERIFICATION_FAILED,
|
||||
@@ -3502,7 +3518,7 @@ fn check_auth(req: Request<()>) -> std::result::Result<Request<()>, Status> {
|
||||
error = %e,
|
||||
"RPC signature verification failed"
|
||||
);
|
||||
Status::unauthenticated("No valid auth token")
|
||||
rejection
|
||||
}
|
||||
})?;
|
||||
|
||||
@@ -7001,6 +7017,35 @@ mod tests {
|
||||
.await;
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn rpc_auth_resource_exhaustion_does_not_report_invalid_credentials() {
|
||||
for source in [
|
||||
storage::RpcReplayCacheAdmissionError::Capacity,
|
||||
storage::RpcReplayCacheAdmissionError::MemoryBudget,
|
||||
] {
|
||||
let error = std::io::Error::new(std::io::ErrorKind::WouldBlock, source);
|
||||
let status = rpc_auth_failure_status(&error);
|
||||
assert_eq!(status.code(), tonic::Code::ResourceExhausted);
|
||||
let api = crate::error::ApiError::from(crate::storage_api::error::StorageError::from(status));
|
||||
let response = s3s::S3Error::from(api);
|
||||
assert_eq!(response.status_code(), Some(http::StatusCode::SERVICE_UNAVAILABLE));
|
||||
}
|
||||
for message in [
|
||||
"RPC request replay detected",
|
||||
"Invalid RPC v2 signature",
|
||||
"Missing signature header",
|
||||
] {
|
||||
assert_eq!(
|
||||
rpc_auth_failure_status(&std::io::Error::other(message)).code(),
|
||||
tonic::Code::Unauthenticated
|
||||
);
|
||||
}
|
||||
assert_eq!(
|
||||
rpc_auth_failure_status(&std::io::Error::other("RPC boot epoch is stale")).code(),
|
||||
tonic::Code::Unavailable
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial_test::serial]
|
||||
async fn rpc_auth_rejection_records_failure_reason_metric() {
|
||||
|
||||
+51
-295
@@ -37,7 +37,7 @@ use std::task::{Context, Poll};
|
||||
use std::time::Duration;
|
||||
use std::{
|
||||
collections::{HashMap, HashSet},
|
||||
sync::{Mutex as StdMutex, OnceLock},
|
||||
sync::OnceLock,
|
||||
time::Instant,
|
||||
};
|
||||
use tokio::sync::Mutex;
|
||||
@@ -80,7 +80,6 @@ const LOG_COMPONENT_SERVER: &str = "server";
|
||||
const LOG_SUBSYSTEM_READINESS: &str = "readiness";
|
||||
const METRIC_RUNTIME_READINESS_READY: &str = "rustfs_runtime_readiness_ready";
|
||||
const METRIC_RUNTIME_READINESS_DEGRADED_TOTAL: &str = "rustfs_runtime_readiness_degraded_total";
|
||||
const METRIC_POOL_METADATA_CHECK_TIMEOUT_TOTAL: &str = "rustfs_pool_metadata_check_timeouts_total";
|
||||
|
||||
pub use crate::shared_types::{
|
||||
DependencyReadiness, DependencyReadinessReport, MissingReadinessDrive, ReadinessDegradedReason, StorageReadinessDetails,
|
||||
@@ -293,66 +292,6 @@ struct StorageWriteReadinessStatus {
|
||||
pool_metadata_reason: Option<ReadinessDegradedReason>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Copy, Default)]
|
||||
struct NodePoolMetadataReadinessCache {
|
||||
entry: Option<NodePoolMetadataReadinessCacheEntry>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Copy)]
|
||||
struct NodePoolMetadataReadinessCacheEntry {
|
||||
cached_at: Instant,
|
||||
observed_at: Instant,
|
||||
status: StorageWriteReadinessStatus,
|
||||
}
|
||||
|
||||
impl NodePoolMetadataReadinessCache {
|
||||
fn observe(
|
||||
&mut self,
|
||||
observed: StorageWriteReadinessStatus,
|
||||
observed_at: Instant,
|
||||
now: Instant,
|
||||
ttl: Duration,
|
||||
) -> StorageWriteReadinessStatus {
|
||||
if ttl.is_zero() {
|
||||
self.entry = None;
|
||||
return observed;
|
||||
}
|
||||
|
||||
if observed.pool_metadata_reason == Some(ReadinessDegradedReason::PoolMetadataCheckTimeout) {
|
||||
let Some(previous) = self.entry else {
|
||||
return observed;
|
||||
};
|
||||
if now.saturating_duration_since(previous.cached_at) > ttl {
|
||||
return observed;
|
||||
}
|
||||
|
||||
// A confirmed write block must never be hidden by a later timeout.
|
||||
if previous.status.pool_metadata_reason == Some(ReadinessDegradedReason::PoolMetaWriteBlocked) {
|
||||
return previous.status;
|
||||
}
|
||||
|
||||
// A transient inspection outage is diagnostic, not proof that the
|
||||
// writer became unsafe. Keep the last confirmed writable state.
|
||||
if previous.status.ready {
|
||||
return StorageWriteReadinessStatus {
|
||||
ready: true,
|
||||
pool_metadata_reason: Some(ReadinessDegradedReason::PoolMetadataCheckTimeout),
|
||||
};
|
||||
}
|
||||
return observed;
|
||||
}
|
||||
|
||||
if self.entry.is_none_or(|entry| entry.observed_at <= observed_at) {
|
||||
self.entry = Some(NodePoolMetadataReadinessCacheEntry {
|
||||
cached_at: now,
|
||||
observed_at,
|
||||
status: observed,
|
||||
});
|
||||
}
|
||||
observed
|
||||
}
|
||||
}
|
||||
|
||||
fn pool_metadata_write_readiness(result: Result<(), StorageError>) -> StorageWriteReadinessStatus {
|
||||
match result {
|
||||
Ok(()) => StorageWriteReadinessStatus {
|
||||
@@ -360,13 +299,9 @@ fn pool_metadata_write_readiness(result: Result<(), StorageError>) -> StorageWri
|
||||
pool_metadata_reason: None,
|
||||
},
|
||||
Err(error) => {
|
||||
let pool_metadata_reason = if error.pool_metadata_failure().is_some() {
|
||||
Some(ReadinessDegradedReason::PoolMetaWriteBlocked)
|
||||
} else if matches!(error, StorageError::Timeout) {
|
||||
Some(ReadinessDegradedReason::PoolMetadataCheckTimeout)
|
||||
} else {
|
||||
None
|
||||
};
|
||||
let pool_metadata_reason = error
|
||||
.pool_metadata_failure()
|
||||
.map(|_| ReadinessDegradedReason::PoolMetaWriteBlocked);
|
||||
StorageWriteReadinessStatus {
|
||||
ready: false,
|
||||
pool_metadata_reason,
|
||||
@@ -434,11 +369,6 @@ fn health_cluster_timeout() -> Duration {
|
||||
)
|
||||
}
|
||||
|
||||
fn node_pool_metadata_readiness_cache() -> &'static StdMutex<NodePoolMetadataReadinessCache> {
|
||||
static CACHE: OnceLock<StdMutex<NodePoolMetadataReadinessCache>> = OnceLock::new();
|
||||
CACHE.get_or_init(|| StdMutex::new(NodePoolMetadataReadinessCache::default()))
|
||||
}
|
||||
|
||||
fn storage_readiness_cache() -> &'static Mutex<Option<StorageReadinessCacheEntry>> {
|
||||
static CACHE: OnceLock<Mutex<Option<StorageReadinessCacheEntry>>> = OnceLock::new();
|
||||
CACHE.get_or_init(|| Mutex::new(None))
|
||||
@@ -543,26 +473,6 @@ async fn update_storage_readiness_cache(status: StorageWriteReadinessStatus) {
|
||||
});
|
||||
}
|
||||
|
||||
fn apply_node_pool_metadata_timeout_policy(
|
||||
observed: StorageWriteReadinessStatus,
|
||||
observed_at: Instant,
|
||||
) -> StorageWriteReadinessStatus {
|
||||
if observed.pool_metadata_reason == Some(ReadinessDegradedReason::PoolMetadataCheckTimeout) {
|
||||
counter!(METRIC_POOL_METADATA_CHECK_TIMEOUT_TOTAL).increment(1);
|
||||
}
|
||||
let mut cache = node_pool_metadata_readiness_cache()
|
||||
.lock()
|
||||
.unwrap_or_else(std::sync::PoisonError::into_inner);
|
||||
cache.observe(observed, observed_at, Instant::now(), health_readiness_cache_ttl())
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
fn reset_node_pool_metadata_readiness_cache() {
|
||||
*node_pool_metadata_readiness_cache()
|
||||
.lock()
|
||||
.unwrap_or_else(std::sync::PoisonError::into_inner) = NodePoolMetadataReadinessCache::default();
|
||||
}
|
||||
|
||||
async fn load_cached_lock_quorum_status() -> Option<LockQuorumObservation> {
|
||||
let ttl = health_readiness_cache_ttl();
|
||||
if ttl.is_zero() {
|
||||
@@ -855,9 +765,6 @@ fn record_readiness_report(report: &DependencyReadinessReport) {
|
||||
&& report.readiness.peer_health_ready;
|
||||
gauge!(METRIC_RUNTIME_READINESS_READY).set(if ready { 1.0 } else { 0.0 });
|
||||
for reason in &report.degraded_reasons {
|
||||
if ready && *reason == ReadinessDegradedReason::PoolMetadataCheckTimeout {
|
||||
continue;
|
||||
}
|
||||
counter!(METRIC_RUNTIME_READINESS_DEGRADED_TOTAL, "reason" => reason.as_str()).increment(1);
|
||||
}
|
||||
}
|
||||
@@ -944,7 +851,7 @@ pub async fn collect_node_readiness_report() -> DependencyReadinessReport {
|
||||
let mut storage_check_timed_out = false;
|
||||
if let Some(store) = runtime_sources::current_object_store_handle() {
|
||||
let pool_metadata_status = store.pool_meta_write_status().await;
|
||||
storage = apply_node_pool_metadata_timeout_policy(pool_metadata_write_readiness(pool_metadata_status), Instant::now());
|
||||
storage = pool_metadata_write_readiness(pool_metadata_status);
|
||||
match node_storage_snapshot(store.as_ref(), &lock_observation.online_hosts).await {
|
||||
Ok((info, unavailable_drives, missing_drives)) => {
|
||||
details.unavailable_drives = unavailable_drives;
|
||||
@@ -2927,7 +2834,7 @@ mod tests {
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn pool_metadata_observation_distinguishes_block_timeout_and_unavailable() {
|
||||
fn pool_metadata_observation_distinguishes_block_and_unavailable() {
|
||||
let blocked = blocked_pool_metadata_status();
|
||||
assert!(!blocked.ready);
|
||||
assert_eq!(blocked.pool_metadata_reason, Some(ReadinessDegradedReason::PoolMetaWriteBlocked));
|
||||
@@ -2935,227 +2842,76 @@ mod tests {
|
||||
let writable = pool_metadata_write_readiness(Ok(()));
|
||||
assert!(writable.ready);
|
||||
assert_eq!(writable.pool_metadata_reason, None);
|
||||
let timed_out = pool_metadata_write_readiness(Err(StorageError::Timeout));
|
||||
assert!(!timed_out.ready);
|
||||
assert_eq!(timed_out.pool_metadata_reason, Some(ReadinessDegradedReason::PoolMetadataCheckTimeout));
|
||||
let unavailable = pool_metadata_write_readiness(Err(StorageError::other("pool metadata writes remain blocked")));
|
||||
assert!(!unavailable.ready);
|
||||
assert_eq!(
|
||||
unavailable.pool_metadata_reason, None,
|
||||
"error text alone must not be interpreted as a typed write block"
|
||||
);
|
||||
}
|
||||
|
||||
let readiness = DependencyReadiness {
|
||||
storage_ready: false,
|
||||
#[test]
|
||||
fn readiness_report_records_ready_gauge_and_degraded_reasons() {
|
||||
let record = |report: DependencyReadinessReport| {
|
||||
let recorder = DebuggingRecorder::new();
|
||||
let snapshotter = recorder.snapshotter();
|
||||
metrics::with_local_recorder(&recorder, || record_readiness_report(&report));
|
||||
snapshotter.snapshot().into_vec()
|
||||
};
|
||||
let ready = DependencyReadiness {
|
||||
storage_ready: true,
|
||||
iam_ready: true,
|
||||
lock_quorum_ready: true,
|
||||
peer_health_ready: true,
|
||||
};
|
||||
let report = dependency_readiness_report_from_write_status(readiness, timed_out);
|
||||
assert!(!report.readiness.storage_ready, "inspection timeout must remain fail-closed");
|
||||
assert_eq!(report.degraded_reasons, vec![ReadinessDegradedReason::PoolMetadataCheckTimeout]);
|
||||
assert_eq!(report.degraded_reasons[0].as_str(), "pool_metadata_check_timeout");
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[serial]
|
||||
fn sticky_pool_metadata_timeout_uses_dedicated_metric_without_degraded_count() {
|
||||
reset_node_pool_metadata_readiness_cache();
|
||||
let recorder = DebuggingRecorder::new();
|
||||
let snapshotter = recorder.snapshotter();
|
||||
|
||||
metrics::with_local_recorder(&recorder, || {
|
||||
with_var(rustfs_config::ENV_HEALTH_READINESS_CACHE_TTL_MS, Some("60000"), || {
|
||||
let writable = StorageWriteReadinessStatus {
|
||||
ready: true,
|
||||
pool_metadata_reason: None,
|
||||
};
|
||||
assert_eq!(apply_node_pool_metadata_timeout_policy(writable, Instant::now()), writable);
|
||||
let sticky = apply_node_pool_metadata_timeout_policy(
|
||||
StorageWriteReadinessStatus {
|
||||
ready: false,
|
||||
pool_metadata_reason: Some(ReadinessDegradedReason::PoolMetadataCheckTimeout),
|
||||
},
|
||||
Instant::now(),
|
||||
);
|
||||
assert!(sticky.ready);
|
||||
record_readiness_report(&DependencyReadinessReport {
|
||||
readiness: DependencyReadiness {
|
||||
storage_ready: true,
|
||||
iam_ready: true,
|
||||
lock_quorum_ready: true,
|
||||
peer_health_ready: true,
|
||||
},
|
||||
degraded_reasons: vec![ReadinessDegradedReason::PoolMetadataCheckTimeout],
|
||||
storage_details: None,
|
||||
});
|
||||
});
|
||||
let entries = record(DependencyReadinessReport {
|
||||
readiness: ready,
|
||||
degraded_reasons: Vec::new(),
|
||||
storage_details: None,
|
||||
});
|
||||
reset_node_pool_metadata_readiness_cache();
|
||||
|
||||
let entries = snapshotter.snapshot().into_vec();
|
||||
let ready = entries.iter().find_map(|(composite, _, _, value)| {
|
||||
let gauge = entries.iter().find_map(|(composite, _, _, value)| {
|
||||
(composite.kind() == MetricKind::Gauge && composite.key().name() == METRIC_RUNTIME_READINESS_READY).then_some(value)
|
||||
});
|
||||
assert!(matches!(ready, Some(DebugValue::Gauge(value)) if value.into_inner() == 1.0));
|
||||
assert!(
|
||||
matches!(gauge, Some(DebugValue::Gauge(value)) if value.into_inner() == 1.0),
|
||||
"a ready report must set the readiness gauge: {gauge:?}"
|
||||
);
|
||||
assert!(
|
||||
entries
|
||||
.iter()
|
||||
.all(|(composite, _, _, _)| composite.key().name() != METRIC_RUNTIME_READINESS_DEGRADED_TOTAL),
|
||||
"a ready report without reasons must not count degradation"
|
||||
);
|
||||
|
||||
let entries = record(DependencyReadinessReport {
|
||||
readiness: DependencyReadiness {
|
||||
storage_ready: false,
|
||||
..ready
|
||||
},
|
||||
degraded_reasons: vec![ReadinessDegradedReason::PoolMetaWriteBlocked],
|
||||
storage_details: None,
|
||||
});
|
||||
let gauge = entries.iter().find_map(|(composite, _, _, value)| {
|
||||
(composite.kind() == MetricKind::Gauge && composite.key().name() == METRIC_RUNTIME_READINESS_READY).then_some(value)
|
||||
});
|
||||
assert!(
|
||||
matches!(gauge, Some(DebugValue::Gauge(value)) if value.into_inner() == 0.0),
|
||||
"a degraded report must clear the readiness gauge: {gauge:?}"
|
||||
);
|
||||
let degraded = entries.iter().find_map(|(composite, _, _, value)| {
|
||||
(composite.kind() == MetricKind::Counter
|
||||
&& composite.key().name() == METRIC_RUNTIME_READINESS_DEGRADED_TOTAL
|
||||
&& composite
|
||||
.key()
|
||||
.labels()
|
||||
.any(|label| label.key() == "reason" && label.value() == "pool_metadata_check_timeout"))
|
||||
.any(|label| label.key() == "reason" && label.value() == "pool_meta_write_blocked"))
|
||||
.then_some(value)
|
||||
});
|
||||
assert!(degraded.is_none());
|
||||
|
||||
let timeout = entries.iter().find_map(|(composite, _, _, value)| {
|
||||
(composite.kind() == MetricKind::Counter && composite.key().name() == METRIC_POOL_METADATA_CHECK_TIMEOUT_TOTAL)
|
||||
.then_some(value)
|
||||
});
|
||||
assert!(matches!(timeout, Some(DebugValue::Counter(1))));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn node_pool_metadata_timeout_preserves_recent_writable_observation() {
|
||||
reset_node_pool_metadata_readiness_cache();
|
||||
async_with_vars([(rustfs_config::ENV_HEALTH_READINESS_CACHE_TTL_MS, Some("60000"))], async {
|
||||
let writable = StorageWriteReadinessStatus {
|
||||
ready: true,
|
||||
pool_metadata_reason: None,
|
||||
};
|
||||
assert_eq!(apply_node_pool_metadata_timeout_policy(writable, Instant::now()), writable);
|
||||
|
||||
let timed_out = apply_node_pool_metadata_timeout_policy(
|
||||
StorageWriteReadinessStatus {
|
||||
ready: false,
|
||||
pool_metadata_reason: Some(ReadinessDegradedReason::PoolMetadataCheckTimeout),
|
||||
},
|
||||
Instant::now(),
|
||||
);
|
||||
|
||||
assert!(timed_out.ready);
|
||||
assert_eq!(timed_out.pool_metadata_reason, Some(ReadinessDegradedReason::PoolMetadataCheckTimeout));
|
||||
})
|
||||
.await;
|
||||
reset_node_pool_metadata_readiness_cache();
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn node_pool_metadata_timeout_fails_closed_with_empty_cache() {
|
||||
reset_node_pool_metadata_readiness_cache();
|
||||
async_with_vars([(rustfs_config::ENV_HEALTH_READINESS_CACHE_TTL_MS, Some("60000"))], async {
|
||||
let timed_out = apply_node_pool_metadata_timeout_policy(
|
||||
StorageWriteReadinessStatus {
|
||||
ready: false,
|
||||
pool_metadata_reason: Some(ReadinessDegradedReason::PoolMetadataCheckTimeout),
|
||||
},
|
||||
Instant::now(),
|
||||
);
|
||||
|
||||
assert!(!timed_out.ready);
|
||||
assert_eq!(timed_out.pool_metadata_reason, Some(ReadinessDegradedReason::PoolMetadataCheckTimeout));
|
||||
})
|
||||
.await;
|
||||
reset_node_pool_metadata_readiness_cache();
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn node_pool_metadata_timeout_fails_closed_after_cache_ttl() {
|
||||
let mut cache = NodePoolMetadataReadinessCache::default();
|
||||
let observed_at = Instant::now();
|
||||
let ttl = Duration::from_secs(60);
|
||||
let writable = StorageWriteReadinessStatus {
|
||||
ready: true,
|
||||
pool_metadata_reason: None,
|
||||
};
|
||||
assert_eq!(cache.observe(writable, observed_at, observed_at, ttl), writable);
|
||||
|
||||
let timed_out = cache.observe(
|
||||
StorageWriteReadinessStatus {
|
||||
ready: false,
|
||||
pool_metadata_reason: Some(ReadinessDegradedReason::PoolMetadataCheckTimeout),
|
||||
},
|
||||
observed_at + Duration::from_secs(61),
|
||||
observed_at + Duration::from_secs(61),
|
||||
ttl,
|
||||
assert!(
|
||||
matches!(degraded, Some(DebugValue::Counter(1))),
|
||||
"each degraded reason must be counted once: {degraded:?}"
|
||||
);
|
||||
|
||||
assert!(!timed_out.ready);
|
||||
assert_eq!(timed_out.pool_metadata_reason, Some(ReadinessDegradedReason::PoolMetadataCheckTimeout));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn node_pool_metadata_timeout_never_hides_confirmed_block() {
|
||||
reset_node_pool_metadata_readiness_cache();
|
||||
async_with_vars([(rustfs_config::ENV_HEALTH_READINESS_CACHE_TTL_MS, Some("60000"))], async {
|
||||
let blocked = StorageWriteReadinessStatus {
|
||||
ready: false,
|
||||
pool_metadata_reason: Some(ReadinessDegradedReason::PoolMetaWriteBlocked),
|
||||
};
|
||||
let newer_observation = Instant::now();
|
||||
assert_eq!(apply_node_pool_metadata_timeout_policy(blocked, newer_observation), blocked);
|
||||
assert_eq!(
|
||||
apply_node_pool_metadata_timeout_policy(
|
||||
StorageWriteReadinessStatus {
|
||||
ready: true,
|
||||
pool_metadata_reason: None,
|
||||
},
|
||||
newer_observation
|
||||
.checked_sub(Duration::from_secs(1))
|
||||
.expect("test instant has elapsed time"),
|
||||
),
|
||||
StorageWriteReadinessStatus {
|
||||
ready: true,
|
||||
pool_metadata_reason: None,
|
||||
}
|
||||
);
|
||||
|
||||
let timed_out = apply_node_pool_metadata_timeout_policy(
|
||||
StorageWriteReadinessStatus {
|
||||
ready: false,
|
||||
pool_metadata_reason: Some(ReadinessDegradedReason::PoolMetadataCheckTimeout),
|
||||
},
|
||||
Instant::now(),
|
||||
);
|
||||
|
||||
assert_eq!(timed_out, blocked);
|
||||
})
|
||||
.await;
|
||||
reset_node_pool_metadata_readiness_cache();
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial]
|
||||
async fn node_pool_metadata_timeout_fails_closed_without_fresh_observation() {
|
||||
reset_node_pool_metadata_readiness_cache();
|
||||
async_with_vars([(rustfs_config::ENV_HEALTH_READINESS_CACHE_TTL_MS, Some("60000"))], async {
|
||||
let writable = StorageWriteReadinessStatus {
|
||||
ready: true,
|
||||
pool_metadata_reason: None,
|
||||
};
|
||||
assert_eq!(apply_node_pool_metadata_timeout_policy(writable, Instant::now()), writable);
|
||||
|
||||
async_with_vars([(rustfs_config::ENV_HEALTH_READINESS_CACHE_TTL_MS, Some("0"))], async {
|
||||
let timed_out = apply_node_pool_metadata_timeout_policy(
|
||||
StorageWriteReadinessStatus {
|
||||
ready: false,
|
||||
pool_metadata_reason: Some(ReadinessDegradedReason::PoolMetadataCheckTimeout),
|
||||
},
|
||||
Instant::now(),
|
||||
);
|
||||
|
||||
assert!(!timed_out.ready);
|
||||
assert_eq!(timed_out.pool_metadata_reason, Some(ReadinessDegradedReason::PoolMetadataCheckTimeout));
|
||||
})
|
||||
.await;
|
||||
})
|
||||
.await;
|
||||
reset_node_pool_metadata_readiness_cache();
|
||||
}
|
||||
|
||||
#[test]
|
||||
|
||||
@@ -44,7 +44,6 @@ pub enum ReadinessDegradedReason {
|
||||
ObjectReadStalled,
|
||||
ObjectWriteStalled,
|
||||
PoolMetaWriteBlocked,
|
||||
PoolMetadataCheckTimeout,
|
||||
StorageReadinessCheckTimeout,
|
||||
ClusterHealthTimeout,
|
||||
PeerHealthUnavailable,
|
||||
@@ -65,7 +64,6 @@ impl ReadinessDegradedReason {
|
||||
ReadinessDegradedReason::ObjectReadStalled => "object_read_stalled",
|
||||
ReadinessDegradedReason::ObjectWriteStalled => "object_write_stalled",
|
||||
ReadinessDegradedReason::PoolMetaWriteBlocked => "pool_meta_write_blocked",
|
||||
ReadinessDegradedReason::PoolMetadataCheckTimeout => "pool_metadata_check_timeout",
|
||||
ReadinessDegradedReason::StorageReadinessCheckTimeout => "storage_readiness_check_timeout",
|
||||
ReadinessDegradedReason::ClusterHealthTimeout => "cluster_health_timeout",
|
||||
ReadinessDegradedReason::PeerHealthUnavailable => "peer_health_unavailable",
|
||||
|
||||
@@ -1664,6 +1664,9 @@ async fn handle_put_file(req: Request<Incoming>, require_auth: bool) -> Response
|
||||
INTERNODE_TRANSPORT_BACKEND_TCP_HTTP,
|
||||
)
|
||||
{
|
||||
if crate::storage::storage_api::rpc_auth_is_resource_exhausted(&e) {
|
||||
return response_with_status(StatusCode::SERVICE_UNAVAILABLE, "RPC replay admission exhausted");
|
||||
}
|
||||
log_internode_rpc_response_failure!(
|
||||
StatusCode::FORBIDDEN,
|
||||
&path,
|
||||
|
||||
@@ -621,7 +621,7 @@ pub(crate) mod ecstore_rpc {
|
||||
KMS_SIGNAL_SUBSYSTEM, LocalPeerS3Client, MAX_NETWORK_PROBE_BYTES, MAX_NETWORK_PROBE_DURATION, NetworkPeerProbeClient,
|
||||
NetworkPeerProbeError, PEER_RESTDRY_RUN, PEER_RESTSIGNAL, PEER_RESTSUB_SYS, PeerRestClient, PeerS3Client,
|
||||
SERVICE_SIGNAL_REFRESH_CONFIG, SERVICE_SIGNAL_RELOAD_DYNAMIC, TONIC_RPC_PREFIX, check_and_record_signed_rpc_nonce,
|
||||
decode_heal_bucket_rpc_options, normalize_tonic_rpc_audience, rpc_background_admission,
|
||||
decode_heal_bucket_rpc_options, normalize_tonic_rpc_audience, rpc_auth_is_resource_exhausted, rpc_background_admission,
|
||||
sign_ns_scanner_capability_with_checkpoint_coverage, sign_put_file_capability, sign_tonic_rpc_response_proof,
|
||||
tonic_boot_epoch_challenge, tonic_boot_epoch_response_headers, tonic_rpc_auth_failure_reason,
|
||||
verify_put_file_auth_trailer, verify_rpc_signature, verify_tonic_canonical_body_digest,
|
||||
@@ -630,8 +630,8 @@ pub(crate) mod ecstore_rpc {
|
||||
};
|
||||
#[cfg(test)]
|
||||
pub(crate) use rustfs_ecstore::api::rpc::{
|
||||
ScannerScopedDirtyUsageAckEntry, build_put_file_auth_trailer, gen_signature_headers, gen_tonic_replay_scope_headers,
|
||||
gen_tonic_signature_headers, set_tonic_canonical_body_digest, verify_put_file_capability,
|
||||
RpcReplayCacheAdmissionError, ScannerScopedDirtyUsageAckEntry, build_put_file_auth_trailer, gen_signature_headers,
|
||||
gen_tonic_replay_scope_headers, gen_tonic_signature_headers, set_tonic_canonical_body_digest, verify_put_file_capability,
|
||||
verify_tonic_boot_epoch_response, verify_tonic_rpc_response_proof,
|
||||
};
|
||||
}
|
||||
@@ -2433,6 +2433,10 @@ pub(crate) fn verify_tonic_rpc_signature_with_bootstrap(
|
||||
ecstore_rpc::verify_tonic_rpc_signature_with_bootstrap(audience, path, headers, allow_replay_scope_bootstrap)
|
||||
}
|
||||
|
||||
pub(crate) fn rpc_auth_is_resource_exhausted(error: &std::io::Error) -> bool {
|
||||
ecstore_rpc::rpc_auth_is_resource_exhausted(error)
|
||||
}
|
||||
|
||||
pub(crate) fn tonic_rpc_auth_failure_reason(error: &std::io::Error) -> &'static str {
|
||||
ecstore_rpc::tonic_rpc_auth_failure_reason(error)
|
||||
}
|
||||
|
||||
@@ -186,10 +186,13 @@ pub(crate) mod server {
|
||||
pub(crate) use crate::storage::storage_api::ecstore_disk::{
|
||||
DiskAPI, DiskOption, new_disk, with_background_resource_unit,
|
||||
};
|
||||
#[cfg(test)]
|
||||
pub(crate) use crate::storage::storage_api::ecstore_rpc::RpcReplayCacheAdmissionError;
|
||||
pub(crate) use crate::storage::storage_api::{
|
||||
RemoteBackgroundIo, ServerContextSlot, TONIC_RPC_PREFIX, normalize_tonic_rpc_audience, rpc_background_admission,
|
||||
tonic_boot_epoch_challenge, tonic_boot_epoch_response_headers, tonic_rpc_auth_failure_reason,
|
||||
verify_tonic_rpc_signature_with_bootstrap, with_remote_background_io,
|
||||
RemoteBackgroundIo, ServerContextSlot, TONIC_RPC_PREFIX, normalize_tonic_rpc_audience,
|
||||
rpc_auth_is_resource_exhausted, rpc_background_admission, tonic_boot_epoch_challenge,
|
||||
tonic_boot_epoch_response_headers, tonic_rpc_auth_failure_reason, verify_tonic_rpc_signature_with_bootstrap,
|
||||
with_remote_background_io,
|
||||
};
|
||||
|
||||
pub(crate) fn try_current_local_node_name() -> Option<String> {
|
||||
|
||||
@@ -181,10 +181,6 @@ const WAREHOUSE_INDEX_STATE_FILE: &str = "state.json";
|
||||
const TABLE_RENAME_ROOT: &str = "renames";
|
||||
const WAREHOUSE_INDEX_MAX_PREFIX_DEPTH: usize = 64;
|
||||
const TABLE_DATA_PLANE_INDEX_MISS_SCAN_MAX_CATALOG_OBJECTS: usize = 4096;
|
||||
/// Listing pages an index-miss scan may consume before failing closed. Only
|
||||
/// table entries count toward the object limit, so maintenance reports and
|
||||
/// other catalog objects cannot trip it; this bounds the listing work itself.
|
||||
const TABLE_DATA_PLANE_INDEX_MISS_SCAN_MAX_PAGES: usize = 64;
|
||||
const EXTERNAL_CATALOG_ROOT: &str = "external-catalog";
|
||||
const EXTERNAL_CATALOG_BRIDGE_FILE: &str = "bridge.json";
|
||||
const MAINTENANCE_ROOT: &str = "maintenance";
|
||||
|
||||
@@ -40,7 +40,7 @@ pub(super) use backup_types::{TableCatalogBackupStatus, TableCatalogRestoreStatu
|
||||
use manifest::table_catalog_backing_manifest;
|
||||
pub(crate) use object::ObjectTableCatalogStore;
|
||||
#[cfg(test)]
|
||||
pub(super) use object::{bounded_table_entry_objects_for_data_plane_scan, maintenance_subtree_skip_cursor};
|
||||
pub(super) use object::bounded_table_entry_objects_for_data_plane_scan;
|
||||
|
||||
struct PeriodicTableMaintenanceQueueGuard<'a> {
|
||||
table_id: &'a str,
|
||||
@@ -1055,6 +1055,20 @@ pub(crate) trait TableCatalogObjectBackend: Clone + Send + Sync + 'static {
|
||||
))
|
||||
}
|
||||
|
||||
/// GET-equivalent table ownership entries, selected before metadata reads.
|
||||
/// Report history must not consume pages. No ordinary S3 list fallback is safe.
|
||||
async fn list_table_entry_objects_page(
|
||||
&self,
|
||||
_bucket: &str,
|
||||
_prefix: &str,
|
||||
_start_after: Option<&str>,
|
||||
_limit: NonZeroUsize,
|
||||
) -> TableCatalogStoreResult<TableCatalogObjectListPage> {
|
||||
Err(TableCatalogStoreError::Unsupported(
|
||||
"authoritative table-entry pagination is not supported by this backend".to_string(),
|
||||
))
|
||||
}
|
||||
|
||||
/// GET-equivalent, bounded enumeration for publishing authoritative indexes.
|
||||
/// Backends without this contract must not fall back to an ordinary S3 list.
|
||||
async fn list_catalog_objects_page(
|
||||
@@ -2615,6 +2629,33 @@ where
|
||||
})
|
||||
}
|
||||
|
||||
async fn list_table_entry_objects_page(
|
||||
&self,
|
||||
bucket: &str,
|
||||
prefix: &str,
|
||||
start_after: Option<&str>,
|
||||
limit: NonZeroUsize,
|
||||
) -> TableCatalogStoreResult<TableCatalogObjectListPage> {
|
||||
let max_keys = i32::try_from(limit.get())
|
||||
.map_err(|_| TableCatalogStoreError::Invalid("catalog page size exceeds storage API range".to_string()))?;
|
||||
let page = self
|
||||
.store
|
||||
.clone()
|
||||
.list_identifier_tree_leaves_authoritative(
|
||||
bucket,
|
||||
prefix,
|
||||
TABLE_ENTRY_FILE,
|
||||
start_after.map(str::to_string),
|
||||
max_keys,
|
||||
)
|
||||
.await
|
||||
.map_err(|err| storage_error_to_catalog("list authoritative table entries", err))?;
|
||||
Ok(TableCatalogObjectListPage {
|
||||
objects: page.objects.into_iter().map(|object| object.name).collect(),
|
||||
is_truncated: page.is_truncated,
|
||||
})
|
||||
}
|
||||
|
||||
async fn list_catalog_objects_page(
|
||||
&self,
|
||||
bucket: &str,
|
||||
|
||||
@@ -430,20 +430,6 @@ pub(in crate::table_catalog) fn bounded_table_entry_objects_for_data_plane_scan(
|
||||
Ok(entries)
|
||||
}
|
||||
|
||||
/// When `object` lies in a table's `maintenance/` subtree, the cursor that skips
|
||||
/// the rest of that subtree. Only keys under that directory are skipped, never
|
||||
/// the table entry or any other table beside it.
|
||||
pub(in crate::table_catalog) fn maintenance_subtree_skip_cursor(prefix: &str, object: &str) -> Option<String> {
|
||||
let relative = object.strip_prefix(prefix)?;
|
||||
let mut segments = relative.splitn(5, '/');
|
||||
let namespace = segments.next()?;
|
||||
let root = segments.next()?;
|
||||
let table = segments.next()?;
|
||||
let maintenance = segments.next()?;
|
||||
(root == TABLE_ROOT && maintenance == MAINTENANCE_ROOT && segments.next().is_some())
|
||||
.then(|| format!("{prefix}{namespace}/{root}/{table}/{maintenance}/\u{10ffff}"))
|
||||
}
|
||||
|
||||
#[derive(Clone)]
|
||||
pub(crate) struct ObjectTableCatalogStore<B> {
|
||||
pub(in crate::table_catalog) backend: B,
|
||||
@@ -2861,41 +2847,21 @@ where
|
||||
max_catalog_objects: usize,
|
||||
) -> TableCatalogStoreResult<Vec<String>> {
|
||||
let mut objects = Vec::new();
|
||||
let mut entries = 0_usize;
|
||||
let mut pages = 0_usize;
|
||||
let mut cursor: Option<String> = None;
|
||||
let prefix = self.paths.namespace_entries_prefix(table_bucket);
|
||||
let limit = NonZeroUsize::new(TABLE_CATALOG_LIST_MAX_KEYS)
|
||||
.ok_or_else(|| TableCatalogStoreError::Internal("table catalog listing page size must be positive".to_string()))?;
|
||||
loop {
|
||||
// A page that ends inside a table's maintenance subtree skips the
|
||||
// rest of it, so each entry can waste at most one page: honest
|
||||
// catalogs scale with their entries while a report flood under one
|
||||
// table stays within the fixed budget.
|
||||
if pages >= TABLE_DATA_PLANE_INDEX_MISS_SCAN_MAX_PAGES.saturating_add(entries) {
|
||||
return Err(TableCatalogStoreError::Unavailable(format!(
|
||||
"table data-plane warehouse-index miss scan exceeds the {TABLE_DATA_PLANE_INDEX_MISS_SCAN_MAX_PAGES}-page listing budget"
|
||||
)));
|
||||
}
|
||||
let page = self
|
||||
.backend
|
||||
.list_objects_page(self.catalog_bucket(), &prefix, cursor.as_deref(), limit)
|
||||
.list_table_entry_objects_page(self.catalog_bucket(), &prefix, cursor.as_deref(), limit)
|
||||
.await?;
|
||||
pages += 1;
|
||||
let next_cursor = page
|
||||
.objects
|
||||
.last()
|
||||
.map(|last| maintenance_subtree_skip_cursor(&prefix, last).unwrap_or_else(|| last.clone()));
|
||||
entries += page
|
||||
.objects
|
||||
.iter()
|
||||
.filter(|object| object.ends_with(TABLE_ENTRY_FILE))
|
||||
.count();
|
||||
let next_cursor = page.objects.last().cloned();
|
||||
objects.extend(page.objects);
|
||||
if !page.is_truncated {
|
||||
return bounded_table_entry_objects_for_data_plane_scan(objects, false, max_catalog_objects);
|
||||
}
|
||||
if entries > max_catalog_objects {
|
||||
if objects.len() > max_catalog_objects {
|
||||
return bounded_table_entry_objects_for_data_plane_scan(objects, true, max_catalog_objects);
|
||||
}
|
||||
let Some(next_cursor) = next_cursor else {
|
||||
|
||||
@@ -594,6 +594,29 @@ impl TestCatalogObjectBackend {
|
||||
state.list_calls = 0;
|
||||
}
|
||||
|
||||
async fn overridden_list_page(&self, bucket: &str, prefix: &str) -> Option<TableCatalogObjectListPage> {
|
||||
let pause = self
|
||||
.state
|
||||
.lock()
|
||||
.await
|
||||
.list_page_pauses
|
||||
.get_mut(&(bucket.to_string(), prefix.to_string()))
|
||||
.and_then(std::collections::VecDeque::pop_front);
|
||||
if let Some(pause) = pause {
|
||||
pause.started.notify_one();
|
||||
pause.release.notified().await;
|
||||
}
|
||||
let mut state = self.state.lock().await;
|
||||
let page = state
|
||||
.list_page_overrides
|
||||
.get_mut(&(bucket.to_string(), prefix.to_string()))
|
||||
.and_then(std::collections::VecDeque::pop_front);
|
||||
if page.is_some() {
|
||||
state.list_calls += 1;
|
||||
}
|
||||
page
|
||||
}
|
||||
|
||||
pub(crate) async fn write_lock_acquisition_count(&self, bucket: &str, object: &str) -> usize {
|
||||
self.state
|
||||
.lock()
|
||||
@@ -1231,29 +1254,7 @@ impl TableCatalogObjectBackend for TestCatalogObjectBackend {
|
||||
start_after: Option<&str>,
|
||||
limit: NonZeroUsize,
|
||||
) -> TableCatalogStoreResult<TableCatalogObjectListPage> {
|
||||
let pause = self
|
||||
.state
|
||||
.lock()
|
||||
.await
|
||||
.list_page_pauses
|
||||
.get_mut(&(bucket.to_string(), prefix.to_string()))
|
||||
.and_then(std::collections::VecDeque::pop_front);
|
||||
if let Some(pause) = pause {
|
||||
pause.started.notify_one();
|
||||
pause.release.notified().await;
|
||||
}
|
||||
let injected = {
|
||||
let mut state = self.state.lock().await;
|
||||
let page = state
|
||||
.list_page_overrides
|
||||
.get_mut(&(bucket.to_string(), prefix.to_string()))
|
||||
.and_then(std::collections::VecDeque::pop_front);
|
||||
if page.is_some() {
|
||||
state.list_calls += 1;
|
||||
}
|
||||
page
|
||||
};
|
||||
if let Some(page) = injected {
|
||||
if let Some(page) = self.overridden_list_page(bucket, prefix).await {
|
||||
return Ok(page);
|
||||
}
|
||||
let mut objects = self.list_objects(bucket, prefix).await?;
|
||||
@@ -1266,6 +1267,34 @@ impl TableCatalogObjectBackend for TestCatalogObjectBackend {
|
||||
Ok(TableCatalogObjectListPage { objects, is_truncated })
|
||||
}
|
||||
|
||||
async fn list_table_entry_objects_page(
|
||||
&self,
|
||||
bucket: &str,
|
||||
prefix: &str,
|
||||
start_after: Option<&str>,
|
||||
limit: NonZeroUsize,
|
||||
) -> TableCatalogStoreResult<TableCatalogObjectListPage> {
|
||||
if let Some(page) = self.overridden_list_page(bucket, prefix).await {
|
||||
return Ok(page);
|
||||
}
|
||||
// This in-memory backend has no erasure replicas. Production selects
|
||||
// candidates before metadata I/O and then proves each key at read quorum.
|
||||
let mut inventory = self.list_objects_for_cleanup(bucket, prefix).await?;
|
||||
inventory.retain(|object| object.key.rsplit('/').next() == Some(TABLE_ENTRY_FILE));
|
||||
if inventory
|
||||
.iter()
|
||||
.any(|object| object.version_id.is_some() || object.delete_marker)
|
||||
{
|
||||
return Err(TableCatalogStoreError::Unavailable("versioned table ownership entry".to_string()));
|
||||
}
|
||||
let mut objects = inventory.into_iter().map(|object| object.key).collect::<Vec<_>>();
|
||||
objects.sort();
|
||||
objects.retain(|object| start_after.is_none_or(|after| object.as_str() > after));
|
||||
let is_truncated = objects.len() >= limit.get();
|
||||
objects.truncate(limit.get());
|
||||
Ok(TableCatalogObjectListPage { objects, is_truncated })
|
||||
}
|
||||
|
||||
async fn list_objects_for_cleanup(
|
||||
&self,
|
||||
bucket: &str,
|
||||
|
||||
@@ -1814,43 +1814,6 @@ fn table_data_plane_index_miss_scan_rejects_catalogs_above_its_object_limit() {
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn table_data_plane_index_miss_scan_skips_only_a_table_maintenance_subtree() {
|
||||
let prefix = "catalog/namespaces/";
|
||||
let skip = |object: &str| maintenance_subtree_skip_cursor(prefix, &format!("{prefix}{object}"));
|
||||
assert_eq!(
|
||||
skip("sales/tables/orders/maintenance/hash/jobs/job.json").as_deref(),
|
||||
Some("catalog/namespaces/sales/tables/orders/maintenance/\u{10ffff}")
|
||||
);
|
||||
assert_eq!(
|
||||
skip("sales/tables/orders/maintenance/hash/config.json").as_deref(),
|
||||
Some("catalog/namespaces/sales/tables/orders/maintenance/\u{10ffff}")
|
||||
);
|
||||
// A table or namespace literally named "maintenance" is not a subtree.
|
||||
assert_eq!(skip("sales/tables/maintenance/table-entry.json"), None);
|
||||
assert_eq!(skip("maintenance/tables/orders/table-entry.json"), None);
|
||||
assert_eq!(
|
||||
skip("maintenance/tables/maintenance/maintenance/hash/jobs/job.json").as_deref(),
|
||||
Some("catalog/namespaces/maintenance/tables/maintenance/maintenance/\u{10ffff}")
|
||||
);
|
||||
// Entries, bridges, views and a bare file named maintenance never jump.
|
||||
assert_eq!(skip("sales/tables/orders/table-entry.json"), None);
|
||||
assert_eq!(skip("sales/tables/orders/external-catalog/bridge.json"), None);
|
||||
assert_eq!(skip("sales/views/report/maintenance/hash/jobs/job.json"), None);
|
||||
assert_eq!(skip("sales/tables/orders/maintenance"), None);
|
||||
assert_eq!(skip("sales/namespace.json"), None);
|
||||
assert_eq!(maintenance_subtree_skip_cursor(prefix, "other/sales/tables/orders/maintenance/x/y"), None);
|
||||
// The jump lands before the entry that follows the subtree.
|
||||
assert!(
|
||||
"catalog/namespaces/sales/tables/orders/maintenance/\u{10ffff}"
|
||||
< "catalog/namespaces/sales/tables/orders/table-entry.json"
|
||||
);
|
||||
assert!(
|
||||
"catalog/namespaces/sales/tables/orders/maintenance/\u{10ffff}"
|
||||
< "catalog/namespaces/sales/tables/orders0/table-entry.json"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn catalog_lock_authority_failures_are_typed_as_unavailable() {
|
||||
for error in [
|
||||
@@ -6880,39 +6843,123 @@ async fn seed_noise_table_with_job_reports(
|
||||
|
||||
#[tokio::test]
|
||||
async fn table_data_plane_index_miss_scan_pages_scale_with_entries_not_job_reports() {
|
||||
let mut baseline_pages = None;
|
||||
for reports in [0, 80] {
|
||||
let backend = TestCatalogObjectBackend::default();
|
||||
let store = ObjectTableCatalogStore::new(backend.clone());
|
||||
let bucket = "analytics";
|
||||
let namespace = Namespace::parse("sales").expect("namespace should parse");
|
||||
let table = IdentifierSegment::parse("orders").expect("table should parse");
|
||||
let current = default_table_metadata_file_path(&namespace, &table, "00001.metadata.json");
|
||||
let object = "tables/table-id/data/part-00001.parquet";
|
||||
|
||||
seed_table_for_metadata_maintenance(&store, bucket, &namespace, &table, current).await;
|
||||
backend
|
||||
.delete_object(RUSTFS_META_BUCKET, &store.paths.warehouse_index_entry_path(bucket, "tables/table-id/"))
|
||||
.await
|
||||
.expect("warehouse index entry should be removed");
|
||||
// Keep the same table inventory with and without historical job reports.
|
||||
let noise_namespace = Namespace::parse("noise").expect("namespace should parse");
|
||||
for index in 0..1000 {
|
||||
let noise_table = IdentifierSegment::parse(format!("noise-{index:04}")).expect("table should parse");
|
||||
seed_noise_table_with_job_reports(&store, bucket, &noise_namespace, &noise_table, &format!("noise-{index}"), reports)
|
||||
.await;
|
||||
}
|
||||
backend.reset_call_counts().await;
|
||||
|
||||
let resource = table_data_plane_resource_for_object(&store, bucket, object)
|
||||
.await
|
||||
.expect("job history must not exhaust the index-miss listing budget")
|
||||
.expect("the scan must retain table-aware protection");
|
||||
assert_eq!(resource.table, "orders");
|
||||
let pages = backend.list_call_count().await;
|
||||
assert!(pages > 0, "the missing index must exercise enumeration");
|
||||
if let Some(expected) = baseline_pages {
|
||||
assert_eq!(pages, expected, "job history must not consume ownership listing pages");
|
||||
} else {
|
||||
baseline_pages = Some(pages);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn table_entry_scan_preserves_nested_namespaces_named_like_maintenance() {
|
||||
for name in ["sales", "sales.tables.orders.maintenance"] {
|
||||
let backend = TestCatalogObjectBackend::default();
|
||||
let store = ObjectTableCatalogStore::new(backend.clone());
|
||||
let bucket = "analytics";
|
||||
let namespace = Namespace::parse(name).expect("namespace should parse");
|
||||
let owner = IdentifierSegment::parse("zz-owner").expect("owner should parse");
|
||||
let current = default_table_metadata_file_path(&namespace, &owner, "00001.metadata.json");
|
||||
seed_table_for_metadata_maintenance(&store, bucket, &namespace, &owner, current).await;
|
||||
backend
|
||||
.delete_object(RUSTFS_META_BUCKET, &store.paths.warehouse_index_entry_path(bucket, "tables/table-id/"))
|
||||
.await
|
||||
.expect("remove the owner index to exercise catalog fallback");
|
||||
for index in 0..1000 {
|
||||
let table = IdentifierSegment::parse(format!("table-{index:04}")).expect("table should parse");
|
||||
seed_noise_table_with_job_reports(&store, bucket, &namespace, &table, &format!("id-{index}"), 0).await;
|
||||
}
|
||||
let resource = store
|
||||
.resolve_table_data_plane_resource_for_authorization(bucket, "tables/table-id/data/part.parquet")
|
||||
.await
|
||||
.expect("registered table entries must remain enumerable")
|
||||
.unwrap_or_else(|| panic!("namespace {name} lost its table owner across a page boundary"));
|
||||
assert_eq!(resource.namespace, name);
|
||||
assert_eq!(resource.table, "zz-owner");
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn object_maintenance_preserves_an_owner_after_nested_namespace_page_boundary() {
|
||||
let backend = TestCatalogObjectBackend::default();
|
||||
let store = ObjectTableCatalogStore::new(backend.clone());
|
||||
let bucket = "analytics";
|
||||
let namespace = Namespace::parse("sales").expect("namespace should parse");
|
||||
let table = IdentifierSegment::parse("orders").expect("table should parse");
|
||||
let current = default_table_metadata_file_path(&namespace, &table, "00001.metadata.json");
|
||||
let object = "tables/table-id/data/part-00001.parquet";
|
||||
|
||||
seed_table_for_metadata_maintenance(&store, bucket, &namespace, &table, current).await;
|
||||
let current = default_table_metadata_file_path(&namespace, &table, "00002.metadata.json");
|
||||
let orphan = default_table_metadata_file_path(&namespace, &table, "00001.metadata.json");
|
||||
seed_table_for_metadata_maintenance(&store, bucket, &namespace, &table, current.clone()).await;
|
||||
backend
|
||||
.delete_object(RUSTFS_META_BUCKET, &store.paths.warehouse_index_entry_path(bucket, "tables/table-id/"))
|
||||
.seed_object(bucket, ¤t, br#"{"metadata-log":[]}"#.to_vec())
|
||||
.await;
|
||||
backend.seed_object(bucket, &orphan, br#"{"metadata-log":[]}"#.to_vec()).await;
|
||||
let report = store
|
||||
.plan_table_metadata_maintenance(bucket, "sales", "orders", 0)
|
||||
.await
|
||||
.expect("warehouse index entry should be removed");
|
||||
// 1000 honest tables with 80 historical job reports each: ~80 pages of
|
||||
// objects, beyond the fixed page budget on its own, yet well within one
|
||||
// extra page per table entry.
|
||||
let noise_namespace = Namespace::parse("noise").expect("namespace should parse");
|
||||
for index in 0..1000 {
|
||||
let noise_table = IdentifierSegment::parse(format!("noise-{index:04}")).expect("table should parse");
|
||||
seed_noise_table_with_job_reports(&store, bucket, &noise_namespace, &noise_table, &format!("noise-{index}"), 80).await;
|
||||
}
|
||||
backend.reset_call_counts().await;
|
||||
.expect("the unowned old metadata should initially be a cleanup candidate");
|
||||
assert_eq!(report.deletable_metadata_locations, vec![orphan.clone()]);
|
||||
|
||||
let resource = table_data_plane_resource_for_object(&store, bucket, object)
|
||||
let nested = Namespace::parse("sales.tables.orders.maintenance").expect("nested namespace should parse");
|
||||
store
|
||||
.create_namespace(test_namespace_entry(bucket, &nested))
|
||||
.await
|
||||
.expect("job history must not exhaust the index-miss listing budget")
|
||||
.expect("the scan must retain table-aware protection");
|
||||
assert_eq!(resource.table, "orders");
|
||||
let pages = backend.list_call_count().await;
|
||||
.expect("nested namespace should be created");
|
||||
for index in 0..1100 {
|
||||
let noise = IdentifierSegment::parse(format!("table-{index:04}")).expect("table should parse");
|
||||
seed_noise_table_with_job_reports(&store, bucket, &nested, &noise, &format!("noise-{index}"), 0).await;
|
||||
}
|
||||
let owner = IdentifierSegment::parse("zz-owner").expect("owner should parse");
|
||||
let mut other = test_table_entry(bucket, &nested, &owner, orphan.clone());
|
||||
other.table_id = "other-id".to_string();
|
||||
other.table_uuid = "other-uuid".to_string();
|
||||
other.warehouse_location = format!("s3://{bucket}/tables/other-id");
|
||||
store
|
||||
.create_table(other)
|
||||
.await
|
||||
.expect("another table may adopt the metadata before cleanup");
|
||||
let deletes_before = backend.state.lock().await.delete_attempts.clone();
|
||||
let result = store
|
||||
.delete_table_metadata_maintenance_report(bucket, "sales", "orders", report)
|
||||
.await;
|
||||
assert!(
|
||||
pages > TABLE_DATA_PLANE_INDEX_MISS_SCAN_MAX_PAGES && pages <= TABLE_DATA_PLANE_INDEX_MISS_SCAN_MAX_PAGES + 1001,
|
||||
"the honest catalog needs {pages} pages: more than the fixed budget, at most one extra per entry"
|
||||
backend
|
||||
.object_exists(bucket, &orphan)
|
||||
.await
|
||||
.expect("read back the current owner's metadata")
|
||||
);
|
||||
assert_eq!(backend.state.lock().await.delete_attempts, deletes_before);
|
||||
assert_matches!(result, Err(TableCatalogStoreError::Conflict(message)) if message.contains("exclusive metadata"));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
@@ -6930,8 +6977,7 @@ async fn table_data_plane_index_miss_scan_bounds_a_job_report_flood_under_one_ta
|
||||
.delete_object(RUSTFS_META_BUCKET, &store.paths.warehouse_index_entry_path(bucket, "tables/table-id/"))
|
||||
.await
|
||||
.expect("warehouse index entry should be removed");
|
||||
// A table-scoped principal floods one table's job history: 70 reports per
|
||||
// page budget would have exhausted the old fixed budget bucket-wide.
|
||||
// Historical report volume must not consume ownership pagination.
|
||||
for job in 0..70_000 {
|
||||
backend
|
||||
.seed_object(
|
||||
@@ -6951,7 +6997,7 @@ async fn table_data_plane_index_miss_scan_bounds_a_job_report_flood_under_one_ta
|
||||
.expect("the scan must retain table-aware protection");
|
||||
assert_eq!(resource.table, "orders");
|
||||
let pages = backend.list_call_count().await;
|
||||
assert!(pages <= 2, "the page that ends inside the flood skips the rest of it, not {pages} pages");
|
||||
assert!(pages <= 2, "report history must not consume {pages} ownership pages");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
|
||||
+1
-1
@@ -40,7 +40,7 @@ their issue closes.
|
||||
| `check_unsafe_code_allowances.sh` | ci-gate | Unsafe-code allowance ledger guard | ci.yml Quick Checks |
|
||||
| `layer-dependency-baseline.txt` | ci-gate (data) | Committed baseline consumed by `check_layer_dependencies.sh` | [Architecture guard troubleshooting](../docs/operations/architecture-guard-troubleshooting.md) |
|
||||
| `static.sh` | ci-gate | Static-build helper executed inside image builds | `Dockerfile.source`, `Dockerfile.decommission-local` |
|
||||
| `helm_chart_version.sh` | ci-gate | Keeps the Helm chart version in sync with the release | helm-package.yml |
|
||||
| `helm_chart_version.sh` | dev-tool | Normalizes chart/app versions for installation references | rustfs-release-version-bump skill |
|
||||
| `test_helm_templates.sh` | ci-gate | Helm template rendering test | helm-package.yml |
|
||||
|
||||
## Test & e2e runners
|
||||
|
||||
+62
-3
@@ -80,6 +80,58 @@ def git_file(root: Path, revision: str, path: str) -> str:
|
||||
return git(root, "show", f"{revision}:{path}")
|
||||
|
||||
|
||||
def release_metadata_only(root: Path, base: str, head: str, path: str) -> bool:
|
||||
"""Recognize literal stable-version edits, never general packaging changes."""
|
||||
stable = r"(?:0|[1-9][0-9]*)\.(?:0|[1-9][0-9]*)\.(?:0|[1-9][0-9]*)"
|
||||
marker = "<version>"
|
||||
patterns = {
|
||||
"flake.nix": [rf'(?m)^([ \t]*pname = "rustfs";\n[ \t]*version = ")({stable})(";)$'],
|
||||
"helm/rustfs/Chart.yaml": [rf'(?m)^({key}: ")({stable})(")$' for key in ("version", "appVersion")],
|
||||
"rustfs.spec": [rf'(?m)^(Version:[ \t]+)({stable})()$'],
|
||||
}
|
||||
if path not in patterns or not SHA.fullmatch(base) or not SHA.fullmatch(head):
|
||||
return False
|
||||
try:
|
||||
normalized, versions = [], []
|
||||
for revision in (base, head):
|
||||
# A symlink, added/deleted file or executable-mode change is a
|
||||
# build input even if its apparent text resembles metadata.
|
||||
entry = git(root, "ls-tree", "-z", revision, "--", path)
|
||||
if not re.fullmatch(rf"100644 blob [0-9a-f]{{40}}\t{re.escape(path)}\x00", entry):
|
||||
return False
|
||||
content, values = git_file(root, revision, path), []
|
||||
# Existing markers could make edits to distinct fields collapse
|
||||
# to identical normalized text.
|
||||
if marker in content:
|
||||
return False
|
||||
for pattern in patterns[path]:
|
||||
matches = list(re.finditer(pattern, content))
|
||||
if len(matches) != 1:
|
||||
return False
|
||||
values.append(matches[0][2])
|
||||
content = re.sub(pattern, lambda match: match[1] + marker + match[3], content)
|
||||
if len(set(values)) != 1:
|
||||
return False
|
||||
normalized.append(content)
|
||||
versions.append(values[0])
|
||||
if path == "rustfs.spec":
|
||||
old, new = (content.split("%changelog\n") for content in normalized)
|
||||
if len(old) != 2 or len(new) != 2 or old[0] != new[0]:
|
||||
return False
|
||||
if old[1] == new[1]:
|
||||
return True
|
||||
# Only the standard prepended release note is non-executable;
|
||||
# RPM macro syntax and edits to older history retain the full gate.
|
||||
note = (r"\* (?:Mon|Tue|Wed|Thu|Fri|Sat|Sun) "
|
||||
r"(?:Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec) [0-9]{2} [0-9]{4} "
|
||||
r"[A-Za-z0-9 ._-]+ <[A-Za-z0-9._+-]+@[A-Za-z0-9.-]+>\n"
|
||||
rf"- Update RPM package to RustFS {re.escape(versions[1])}\n\n")
|
||||
return re.fullmatch(note + re.escape(old[1]), new[1]) is not None
|
||||
return normalized[0] == normalized[1]
|
||||
except (OSError, subprocess.CalledProcessError, UnicodeError, ValueError):
|
||||
return False
|
||||
|
||||
|
||||
def dependency_tables(document: dict):
|
||||
for key, value in document.items():
|
||||
if key in ("dependencies", "dev-dependencies", "build-dependencies"):
|
||||
@@ -265,10 +317,17 @@ def select_plan(event: str, base: str, head: str, ref: str, root: Path, action:
|
||||
base_packages, base_graph = head_packages, head_graph
|
||||
reasons.append(f"conservative fallback: {type(error).__name__}")
|
||||
graph = {name: head_graph.get(name, set()) | base_graph.get(name, set()) for name in set(head_graph) | set(base_graph)}
|
||||
affected = set()
|
||||
affected, non_runtime = set(), set()
|
||||
if not full and policy is not None:
|
||||
for path in changed:
|
||||
if documentation_path(path):
|
||||
non_runtime.add(path)
|
||||
continue
|
||||
if (policy["rollout"] == "enforced"
|
||||
and not any(fnmatch.fnmatchcase(path, pattern) for pattern in policy["full_paths"])
|
||||
and release_metadata_only(root, base, head, path)):
|
||||
non_runtime.add(path)
|
||||
reasons.append(f"literal stable release metadata: {path}")
|
||||
continue
|
||||
if path in E2E_SELECTION_FILES:
|
||||
affected.add("e2e_test")
|
||||
@@ -290,9 +349,9 @@ def select_plan(event: str, base: str, head: str, ref: str, root: Path, action:
|
||||
affected.update(owners)
|
||||
if not affected and not full:
|
||||
mode = "docs"
|
||||
reasons.append("all changes are documentation")
|
||||
reasons.append("all changes are documentation or literal stable release metadata")
|
||||
elif affected == {"e2e_test"} and not full and not (closure({"e2e_test"}, graph, reverse=True) - {"e2e_test"}) and all(
|
||||
documentation_path(path) or path in E2E_SELECTION_FILES or
|
||||
path in non_runtime or path in E2E_SELECTION_FILES or
|
||||
(path.startswith("crates/e2e_test/src/") and path.endswith(".rs")) for path in changed):
|
||||
mode = "e2e"
|
||||
reasons.append("isolated E2E harness changes")
|
||||
|
||||
@@ -17,12 +17,7 @@ case "$RAW" in
|
||||
esac
|
||||
|
||||
APP_VERSION="${RAW_TAG#v}"
|
||||
BETA_NUM=$(printf '%s\n' "$APP_VERSION" | sed -n -E 's/^.*-beta\.([0-9]+)$/\1/p')
|
||||
if [ -n "$BETA_NUM" ]; then
|
||||
CHART_VERSION="0.${BETA_NUM}.0"
|
||||
else
|
||||
CHART_VERSION="$APP_VERSION"
|
||||
fi
|
||||
CHART_VERSION="$APP_VERSION"
|
||||
|
||||
if [ -n "${GITHUB_OUTPUT:-}" ]; then
|
||||
{
|
||||
|
||||
Executable
+82
@@ -0,0 +1,82 @@
|
||||
#!/usr/bin/env bash
|
||||
# Copyright 2026 RustFS Team
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
set -euo pipefail
|
||||
|
||||
PKG_DIR="${DEB_FILE%.deb}"
|
||||
|
||||
echo "Building DEB: ${DEB_FILE}"
|
||||
|
||||
mkdir -p "${PKG_DIR}/DEBIAN"
|
||||
mkdir -p "${PKG_DIR}/usr/bin"
|
||||
mkdir -p "${PKG_DIR}/etc/default"
|
||||
mkdir -p "${PKG_DIR}/lib/systemd/system"
|
||||
mkdir -p "${PKG_DIR}/usr/share/doc/rustfs"
|
||||
|
||||
cp ./bin/rustfs "${PKG_DIR}/usr/bin/"
|
||||
chmod 755 "${PKG_DIR}/usr/bin/rustfs"
|
||||
|
||||
cp deploy/build/rustfs.service "${PKG_DIR}/lib/systemd/system/"
|
||||
|
||||
cat > "${PKG_DIR}/etc/default/rustfs" << 'ENVEOF'
|
||||
# RustFS Environment Configuration
|
||||
# See https://docs.rustfs.com for more information
|
||||
# RUSTFS_ACCESS_KEY=<changeme>
|
||||
# RUSTFS_SECRET_KEY=<changeme>
|
||||
# RUSTFS_VOLUMES=""
|
||||
# RUSTFS_ADDRESS=":9000"
|
||||
# RUSTFS_CONSOLE_ADDRESS=":9001"
|
||||
# RUSTFS_CONSOLE_ENABLE=true
|
||||
# RUSTFS_OBS_LOGGER_LEVEL=error
|
||||
# RUSTFS_OBS_LOG_DIRECTORY="/var/log/rustfs/"
|
||||
ENVEOF
|
||||
|
||||
cat > "${PKG_DIR}/DEBIAN/control" << EOF
|
||||
Package: rustfs
|
||||
Version: ${DEB_VERSION}
|
||||
Section: utils
|
||||
Priority: optional
|
||||
Architecture: ${DEB_ARCH}
|
||||
EOF
|
||||
|
||||
# gnu binaries dynamically link glibc and must declare it; musl
|
||||
# binaries are statically linked and need no libc dependency.
|
||||
if [[ -n "$DEB_DEPENDS" ]]; then
|
||||
echo "Depends: ${DEB_DEPENDS}" >> "${PKG_DIR}/DEBIAN/control"
|
||||
fi
|
||||
|
||||
cat >> "${PKG_DIR}/DEBIAN/control" << EOF
|
||||
Maintainer: RustFS Team <support@rustfs.com>
|
||||
Description: High-performance distributed object storage (${LIBC} build)
|
||||
RustFS is a high-performance distributed object storage software
|
||||
built using Rust. It is compatible with MinIO and S3 API.
|
||||
Homepage: https://rustfs.com
|
||||
EOF
|
||||
|
||||
# Declare /etc/default/rustfs as a conffile so dpkg preserves user
|
||||
# modifications on upgrade instead of silently overwriting them.
|
||||
cat > "${PKG_DIR}/DEBIAN/conffiles" << 'CONFFILES'
|
||||
/etc/default/rustfs
|
||||
CONFFILES
|
||||
|
||||
./scripts/release/package_service_scripts.sh after-install > "${PKG_DIR}/DEBIAN/postinst"
|
||||
chmod 755 "${PKG_DIR}/DEBIAN/postinst"
|
||||
|
||||
./scripts/release/package_service_scripts.sh before-remove > "${PKG_DIR}/DEBIAN/prerm"
|
||||
chmod 755 "${PKG_DIR}/DEBIAN/prerm"
|
||||
|
||||
./scripts/release/package_service_scripts.sh after-remove > "${PKG_DIR}/DEBIAN/postrm"
|
||||
chmod 755 "${PKG_DIR}/DEBIAN/postrm"
|
||||
|
||||
cp LICENSE "${PKG_DIR}/usr/share/doc/rustfs/"
|
||||
cp README.md "${PKG_DIR}/usr/share/doc/rustfs/"
|
||||
|
||||
fakeroot dpkg-deb --build "${PKG_DIR}" "$DEB_FILE"
|
||||
|
||||
[[ $(dpkg-deb -f "$DEB_FILE" Package) == rustfs ]]
|
||||
[[ $(dpkg-deb -f "$DEB_FILE" Version) == "$DEB_VERSION" ]]
|
||||
[[ $(dpkg-deb -f "$DEB_FILE" Architecture) == "$DEB_ARCH" ]]
|
||||
dpkg-deb --fsys-tarfile "$DEB_FILE" | tar -tf - | grep -Fx './usr/bin/rustfs' >/dev/null
|
||||
stat --printf='%n %s bytes\n' "$DEB_FILE"
|
||||
echo "deb_file=$DEB_FILE" >> "$GITHUB_OUTPUT"
|
||||
echo "✅ DEB built: $DEB_FILE"
|
||||
Executable
+69
@@ -0,0 +1,69 @@
|
||||
#!/usr/bin/env bash
|
||||
# Copyright 2026 RustFS Team
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
set -euo pipefail
|
||||
|
||||
echo "Building RPM for ${RPM_ARCH} (${LIBC})"
|
||||
|
||||
./scripts/test_package_versions.sh --require-package-managers
|
||||
|
||||
# gnu binaries dynamically link glibc and must declare it; musl
|
||||
# binaries are statically linked and need no libc dependency.
|
||||
depends_args=()
|
||||
if [[ -n "$RPM_DEPENDS" ]]; then
|
||||
depends_args+=(--depends "$RPM_DEPENDS")
|
||||
fi
|
||||
|
||||
# Create config file for fpm (DEB build creates it in its package dir structure,
|
||||
# but fpm needs the file to exist before packaging)
|
||||
mkdir -p ./tmp-pkg/etc/default
|
||||
cat > ./tmp-pkg/etc/default/rustfs << 'ENVEOF'
|
||||
# RustFS Environment Configuration
|
||||
# See https://docs.rustfs.com for more information
|
||||
# RUSTFS_ACCESS_KEY=<changeme>
|
||||
# RUSTFS_SECRET_KEY=<changeme>
|
||||
# RUSTFS_VOLUMES=""
|
||||
# RUSTFS_ADDRESS=":9000"
|
||||
# RUSTFS_CONSOLE_ADDRESS=":9001"
|
||||
# RUSTFS_CONSOLE_ENABLE=true
|
||||
# RUSTFS_OBS_LOGGER_LEVEL=error
|
||||
# RUSTFS_OBS_LOG_DIRECTORY="/var/log/rustfs/"
|
||||
ENVEOF
|
||||
|
||||
fpm -s dir -t rpm \
|
||||
--name rustfs \
|
||||
--version "$RPM_VERSION" \
|
||||
--iteration "$RPM_RELEASE" \
|
||||
--architecture "$RPM_ARCH" \
|
||||
--package "$RPM_FILE" \
|
||||
"${depends_args[@]}" \
|
||||
--description "High-performance distributed object storage (${LIBC} build)" \
|
||||
--maintainer "RustFS Team <support@rustfs.com>" \
|
||||
--url "https://rustfs.com" \
|
||||
--license "Apache-2.0" \
|
||||
--before-install <(./scripts/release/package_service_scripts.sh rpm-before-install) \
|
||||
--before-upgrade <(./scripts/release/package_service_scripts.sh rpm-before-upgrade) \
|
||||
--after-install <(./scripts/release/package_service_scripts.sh after-install) \
|
||||
--after-upgrade <(./scripts/release/package_service_scripts.sh after-install) \
|
||||
--before-remove <(./scripts/release/package_service_scripts.sh before-remove) \
|
||||
--after-remove <(./scripts/release/package_service_scripts.sh after-remove) \
|
||||
--rpm-posttrans <(./scripts/release/package_service_scripts.sh rpm-posttrans) \
|
||||
--config-files /etc/default/rustfs \
|
||||
./bin/rustfs=/usr/bin/rustfs \
|
||||
./tmp-pkg/etc/default/rustfs=/etc/default/rustfs \
|
||||
deploy/build/rustfs.service=/lib/systemd/system/rustfs.service \
|
||||
LICENSE=/usr/share/doc/rustfs/LICENSE \
|
||||
README.md=/usr/share/doc/rustfs/README.md
|
||||
|
||||
if [[ ! -f "$RPM_FILE" ]]; then
|
||||
echo "❌ RPM build failed"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
RPM_METADATA=$(rpm -qp --qf '%{NAME}\n%{VERSION}\n%{RELEASE}\n%{ARCH}\n' "$RPM_FILE")
|
||||
EXPECTED_METADATA=$(printf 'rustfs\n%s\n%s\n%s' "$RPM_VERSION" "$RPM_RELEASE" "$RPM_ARCH")
|
||||
[[ "$RPM_METADATA" == "$EXPECTED_METADATA" ]]
|
||||
rpm -qpl "$RPM_FILE" | grep -Fx '/usr/bin/rustfs' >/dev/null
|
||||
stat --printf='%n %s bytes\n' "$RPM_FILE"
|
||||
echo "rpm_file=$RPM_FILE" >> "$GITHUB_OUTPUT"
|
||||
echo "✅ RPM built: $RPM_FILE"
|
||||
@@ -0,0 +1,428 @@
|
||||
#!/usr/bin/env python3
|
||||
# Copyright 2026 RustFS Team
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
"""Frozen release inputs and artifact identity; publication never builds binaries."""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import tomllib
|
||||
import urllib.request
|
||||
from urllib.parse import urlsplit
|
||||
import zipfile
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
REPOSITORY = "rustfs/rustfs"
|
||||
SCHEMA = 1
|
||||
HEX40 = re.compile(r"[0-9a-f]{40}\Z")
|
||||
HEX64 = re.compile(r"[0-9a-f]{64}\Z")
|
||||
VERSION = re.compile(r"(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)(?:-(?:alpha|beta|rc)\.(0|[1-9][0-9]*))?\Z")
|
||||
TARGETS = ("linux-x86_64-musl", "linux-aarch64-musl", "linux-x86_64-gnu", "linux-aarch64-gnu", "macos-aarch64", "windows-x86_64")
|
||||
LINUX = TARGETS[:4]
|
||||
DESKTOP = TARGETS[4:]
|
||||
MAX_MEMBER = 2 * 1024**3
|
||||
RECIPE_PATHS = (
|
||||
".github/workflows/build.yml",
|
||||
".github/workflows/release-candidate.yml",
|
||||
".github/workflows/release-complete.yml",
|
||||
".github/workflows/release-publish.yml",
|
||||
".github/workflows/docker.yml",
|
||||
".github/workflows/package.yml",
|
||||
".github/actions/setup/action.yml",
|
||||
".github/actions/setup/download_release_asset.py",
|
||||
"scripts/release/candidate.py",
|
||||
"scripts/release/pipeline.py",
|
||||
"scripts/release/verify.py",
|
||||
"scripts/release/deliver.py",
|
||||
"scripts/release/publish.py",
|
||||
"scripts/release/release_gate.py",
|
||||
"scripts/release/package_versions.sh",
|
||||
"scripts/release/package_service_scripts.sh",
|
||||
"scripts/release/build_deb.sh",
|
||||
"scripts/release/build_rpm.sh",
|
||||
"scripts/release/check_glibc.py",
|
||||
".config/release-validation-plan.json",
|
||||
"Dockerfile", "Dockerfile.glibc", "entrypoint.sh", "deploy/build/rustfs.service",
|
||||
".github/workflows/helm-package.yml", "scripts/test_helm_templates.sh",
|
||||
"scripts/security/generate_release_supply_chain_assets.py",
|
||||
".github/workflows/ci.yml", "scripts/ci_gate.py",
|
||||
)
|
||||
|
||||
|
||||
def require(condition, message):
|
||||
if not condition:
|
||||
raise ValueError(message)
|
||||
|
||||
|
||||
def digest(value):
|
||||
return hashlib.sha256(json.dumps(value, sort_keys=True, separators=(",", ":"), ensure_ascii=False).encode()).hexdigest()
|
||||
|
||||
|
||||
def stream_digest(stream):
|
||||
result = hashlib.sha256()
|
||||
for chunk in iter(lambda: stream.read(1024 * 1024), b""):
|
||||
result.update(chunk)
|
||||
return result.hexdigest()
|
||||
|
||||
|
||||
def file_digest(path):
|
||||
with Path(path).open("rb") as stream:
|
||||
return stream_digest(stream)
|
||||
|
||||
|
||||
def read_json(path):
|
||||
path = Path(path)
|
||||
require(path.is_file() and path.stat().st_size <= 4 * 1024**2, "missing or oversized JSON record")
|
||||
def unique(pairs):
|
||||
value = {}
|
||||
for name, item in pairs:
|
||||
require(name not in value, "duplicate JSON field")
|
||||
value[name] = item
|
||||
return value
|
||||
return json.loads(path.read_text(encoding="utf-8"), object_pairs_hook=unique)
|
||||
|
||||
|
||||
def write_json(path, value):
|
||||
"""Existing identity records are immutable, including across job retries."""
|
||||
path = Path(path)
|
||||
if path.exists():
|
||||
require(read_json(path) == value, "refusing to replace an existing identity record")
|
||||
return
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with path.open("x", encoding="utf-8") as stream:
|
||||
json.dump(value, stream, indent=2, sort_keys=True, ensure_ascii=False)
|
||||
stream.write("\n")
|
||||
|
||||
|
||||
def command(args, *, input_text=None, cwd=None):
|
||||
return subprocess.run(args, input=input_text, text=True, capture_output=True, check=True, cwd=cwd, timeout=120).stdout.strip()
|
||||
|
||||
|
||||
class ArtifactRedirect(urllib.request.HTTPRedirectHandler):
|
||||
def redirect_request(self, request, fp, code, message, headers, url):
|
||||
require(urlsplit(url).scheme == "https", "artifact redirect must use HTTPS")
|
||||
redirected = super().redirect_request(request, fp, code, message, headers, url)
|
||||
if redirected is not None and urlsplit(request.full_url).netloc != urlsplit(url).netloc:
|
||||
redirected.remove_header("Authorization")
|
||||
return redirected
|
||||
|
||||
|
||||
def github_stream(path):
|
||||
require(path.startswith("repos/") and ":" not in path and not path.startswith("/"), "invalid GitHub API path")
|
||||
token = os.environ.get("GH_TOKEN") or command(["gh", "auth", "token"])
|
||||
request = urllib.request.Request("https://api.github.com/" + path,
|
||||
headers={"Authorization": "Bearer " + token, "Accept": "application/vnd.github+json",
|
||||
"X-GitHub-Api-Version": "2022-11-28", "User-Agent": "rustfs-release"})
|
||||
return urllib.request.build_opener(ArtifactRedirect()).open(request, timeout=600)
|
||||
|
||||
|
||||
def github(path):
|
||||
with github_stream(path) as response:
|
||||
return json.load(response)
|
||||
|
||||
|
||||
def git_file(revision, path, root=ROOT):
|
||||
require(HEX40.fullmatch(revision) is not None, "source must be a full commit SHA")
|
||||
return subprocess.check_output(["git", "show", f"{revision}:{path}"], cwd=root, timeout=60)
|
||||
|
||||
|
||||
def recipe_digest(revision, root=ROOT):
|
||||
files = {path: hashlib.sha256(git_file(revision, path, root)).hexdigest() for path in RECIPE_PATHS}
|
||||
for directory in ("bootstrap", "helm"):
|
||||
paths = command(["git", "ls-tree", "-r", "--name-only", revision, "--", directory], cwd=root).splitlines()
|
||||
require(bool(paths), "missing release recipe directory")
|
||||
files.update({path: hashlib.sha256(git_file(revision, path, root)).hexdigest() for path in paths})
|
||||
return digest(files)
|
||||
|
||||
|
||||
def console_asset(release):
|
||||
require(isinstance(release, dict) and release.get("draft") is False and release.get("prerelease") is False,
|
||||
"Console must be a published stable release")
|
||||
tag = release.get("tag_name")
|
||||
require(isinstance(tag, str) and re.fullmatch(r"v?[0-9]+\.[0-9]+\.[0-9]+", tag), "invalid Console version")
|
||||
assets = [asset for asset in release.get("assets", [])
|
||||
if re.fullmatch(r"rustfs-console-[A-Za-z0-9._-]+\.zip", asset.get("name", ""))]
|
||||
require(len(assets) == 1, "Console must have exactly one unambiguous ZIP asset")
|
||||
asset = assets[0]
|
||||
checksum = asset.get("digest", "")
|
||||
require(isinstance(checksum, str) and checksum.startswith("sha256:") and HEX64.fullmatch(checksum[7:]),
|
||||
"Console asset must have a SHA-256 digest")
|
||||
expected = f"https://github.com/rustfs/console/releases/download/{tag}/{asset['name']}"
|
||||
require(asset.get("browser_download_url") == expected, "Console URL does not match its release identity")
|
||||
return {"tag": tag, "name": asset["name"], "url": expected, "sha256": checksum[7:]}
|
||||
|
||||
|
||||
def release_asset(release, repository, name):
|
||||
require(release.get("draft") is False and release.get("prerelease") is False, "dependency must be a published stable release")
|
||||
tag = release.get("tag_name", "")
|
||||
require(re.fullmatch(r"v?[0-9]+\.[0-9]+\.[0-9]+", tag), "invalid dependency tag")
|
||||
matches = [a for a in release.get("assets", []) if a.get("name") == name]
|
||||
require(len(matches) == 1, "missing or ambiguous pinned dependency")
|
||||
asset = matches[0]
|
||||
url = f"https://github.com/{repository}/releases/download/{tag}/{name}"
|
||||
require(asset.get("browser_download_url") == url and re.fullmatch(r"sha256:[0-9a-f]{64}", asset.get("digest", "")), "invalid dependency identity")
|
||||
return {"version": tag, "name": name, "url": url, "sha256": asset["digest"][7:]}
|
||||
|
||||
|
||||
def validate_dependency(asset, repository):
|
||||
require(isinstance(asset, dict) and set(asset) == {"version", "name", "url", "sha256"}, "invalid dependency record")
|
||||
require(re.fullmatch(r"[A-Za-z0-9._-]+", asset["name"]), "unsafe dependency name")
|
||||
normalized = release_asset({"draft": False, "prerelease": False, "tag_name": asset["version"],
|
||||
"assets": [{"name": asset["name"], "browser_download_url": asset["url"],
|
||||
"digest": "sha256:" + asset["sha256"]}]}, repository, asset["name"])
|
||||
require(normalized == asset, "dependency record changed")
|
||||
|
||||
|
||||
def validate_inputs(value):
|
||||
require(isinstance(value, dict), "candidate inputs must be an object")
|
||||
fields = {"schema", "candidate_id", "version", "source_sha", "workflow_sha", "recipe_sha256", "lock_sha256",
|
||||
"plan_sha256", "toolchain", "console", "cli", "previous", "ci", "origin_run_id", "origin_run_number"}
|
||||
require(set(value) == fields and type(value["schema"]) is int and value["schema"] == SCHEMA,
|
||||
"unsupported candidate input schema")
|
||||
require(isinstance(value["version"], str) and VERSION.fullmatch(value["version"]), "invalid release version")
|
||||
for name in ("source_sha", "workflow_sha"):
|
||||
require(isinstance(value[name], str) and HEX40.fullmatch(value[name]), f"invalid {name}")
|
||||
for name in ("recipe_sha256", "lock_sha256", "plan_sha256"):
|
||||
require(isinstance(value[name], str) and HEX64.fullmatch(value[name]), f"invalid {name}")
|
||||
for name in ("origin_run_id", "origin_run_number"):
|
||||
require(type(value[name]) is int and value[name] > 0, f"invalid {name}")
|
||||
require(value["candidate_id"] == f"{value['version']}-preview.{value['origin_run_number']}", "candidate name differs from its immutable run identity")
|
||||
require(isinstance(value["toolchain"], str) and re.fullmatch(r"[1-9][0-9]*\.[0-9]+\.[0-9]+", value["toolchain"]),
|
||||
"candidate toolchain must be an exact stable Rust version")
|
||||
console = value["console"]
|
||||
require(isinstance(console, dict) and set(console) == {"tag", "name", "url", "sha256"}, "invalid Console identity")
|
||||
normalized = console_asset({"draft": False, "prerelease": False, "tag_name": console["tag"],
|
||||
"assets": [{"name": console["name"], "browser_download_url": console["url"],
|
||||
"digest": "sha256:" + str(console["sha256"])}]})
|
||||
require(normalized == console, "invalid pinned Console asset")
|
||||
require(isinstance(value["cli"], dict) and set(value["cli"]) == set(LINUX), "missing CLI platform")
|
||||
for asset in value["cli"].values():
|
||||
validate_dependency(asset, "rustfs/cli")
|
||||
validate_dependency(value["previous"], REPOSITORY)
|
||||
require(value["previous"]["version"].lstrip("v") != value["version"], "previous release must differ from the candidate")
|
||||
require(isinstance(value["ci"], dict) and set(value["ci"]) == {"run_id", "run_attempt"}
|
||||
and all(type(item) is int and item > 0 for item in value["ci"].values()), "invalid exact-source CI identity")
|
||||
return value
|
||||
|
||||
|
||||
def freeze(version, source_sha, workflow_sha, toolchain, release, run_id, run_number, cli, previous, ci, root=ROOT):
|
||||
require(VERSION.fullmatch(version) is not None, "invalid release version")
|
||||
source = tomllib.loads(git_file(source_sha, "Cargo.toml", root).decode())
|
||||
require(source["workspace"]["package"]["version"] == version, "Cargo workspace version differs from the target")
|
||||
recipe = recipe_digest(workflow_sha, root)
|
||||
require(recipe_digest(source_sha, root) == recipe, "candidate must contain the reviewed release recipe")
|
||||
plan = json.loads(git_file(workflow_sha, ".config/release-validation-plan.json", root))
|
||||
value = {"schema": SCHEMA, "version": version, "candidate_id": f"{version}-preview.{run_number}",
|
||||
"source_sha": source_sha, "workflow_sha": workflow_sha, "recipe_sha256": recipe,
|
||||
"lock_sha256": hashlib.sha256(git_file(source_sha, "Cargo.lock", root)).hexdigest(),
|
||||
"plan_sha256": digest(plan), "toolchain": toolchain, "console": console_asset(release),
|
||||
"origin_run_id": run_id, "origin_run_number": run_number, "ci": ci,
|
||||
"cli": {target: release_asset(cli, "rustfs/cli", f"rustfs-cli-linux-{'amd64' if 'x86_64' in target else 'arm64'}{'-gnu' if target.endswith('gnu') else ''}-{cli['tag_name']}.tar.gz") for target in LINUX},
|
||||
"previous": release_asset(previous, REPOSITORY, f"rustfs-linux-x86_64-gnu-v{previous['tag_name'].lstrip('v')}.zip")}
|
||||
return validate_inputs(value)
|
||||
|
||||
|
||||
def targets(selection):
|
||||
if selection == "linux":
|
||||
return LINUX
|
||||
if selection == "desktop":
|
||||
return DESKTOP
|
||||
if selection == "all":
|
||||
return TARGETS
|
||||
selected = tuple(selection.split(","))
|
||||
require(selected and len(set(selected)) == len(selected) and all(item in TARGETS for item in selected),
|
||||
"unknown or duplicate build targets")
|
||||
return selected
|
||||
|
||||
|
||||
def archive_record(path, target, inputs):
|
||||
validate_inputs(inputs)
|
||||
require(target in TARGETS, "unknown binary platform")
|
||||
path = Path(path)
|
||||
name = f"rustfs-{target}-v{inputs['version']}.zip"
|
||||
require(path.name == name and path.is_file() and not path.is_symlink(), "archive name does not match the target version")
|
||||
binary = "rustfs.exe" if target == "windows-x86_64" else "rustfs"
|
||||
expected = {binary}
|
||||
if target.endswith("-gnu"):
|
||||
expected.update(("rustfs.cpu-symbol-catalog.json", "rustfs.cpu-symbol-catalog.sha256"))
|
||||
with zipfile.ZipFile(path) as archive:
|
||||
members = archive.infolist()
|
||||
require(len(members) == len(expected) and {m.filename for m in members} == expected,
|
||||
"unexpected or duplicate release archive member")
|
||||
for member in members:
|
||||
require(not member.is_dir() and 0 < member.file_size <= MAX_MEMBER and (member.external_attr >> 16) & 0o170000 != 0o120000,
|
||||
"invalid release archive member")
|
||||
with archive.open(binary) as stream:
|
||||
binary_sha = stream_digest(stream)
|
||||
return {"target": target, "name": name, "kind": "zip", "sha256": file_digest(path), "size": path.stat().st_size,
|
||||
"binary_sha256": binary_sha, "inputs_sha256": digest(inputs)}
|
||||
|
||||
|
||||
def verify_records(inputs, records, directory, required):
|
||||
validate_inputs(inputs)
|
||||
require(isinstance(records, list) and len(records) == len(required), "candidate is missing required platform artifacts")
|
||||
seen = set()
|
||||
for record in records:
|
||||
require(isinstance(record, dict) and record.get("target") in required, "unexpected artifact platform")
|
||||
target = record["target"]
|
||||
require(target not in seen, "duplicate artifact platform")
|
||||
seen.add(target)
|
||||
expected = archive_record(Path(directory) / f"rustfs-{target}-v{inputs['version']}.zip", target, inputs)
|
||||
require(record == expected, "candidate artifact or input identity changed")
|
||||
require(seen == set(required), "incomplete candidate platform set")
|
||||
return sorted(records, key=lambda record: record["target"])
|
||||
|
||||
|
||||
def certification_candidate(inputs, records, directory, required=TARGETS):
|
||||
"""Candidate identity survives adding desktop files; test receipts bind bytes."""
|
||||
records = verify_records(inputs, records, directory, required)
|
||||
return {"schema_version": 1, "product_commit": inputs["source_sha"], "version": inputs["version"],
|
||||
"release_profile": "prerelease" if "-" in inputs["version"] else "stable", "features": ["pyroscope"],
|
||||
"plan_sha256": inputs["plan_sha256"], "workflow_sha": inputs["workflow_sha"],
|
||||
"validation_batch": {"run_id": inputs["candidate_id"], "run_attempt": 1},
|
||||
"deliverables": [{"kind": "zip", "name": r["name"], "sha256": r["sha256"], "size": r["size"],
|
||||
"binary_sha256": r["binary_sha256"],
|
||||
"location": f"https://github.com/{REPOSITORY}/actions/runs/{inputs['origin_run_id']}#artifact-{r['sha256']}"}
|
||||
for r in records]}
|
||||
|
||||
|
||||
def validate_run(run, workflow, *, completed=True):
|
||||
require(isinstance(run, dict) and (run.get("repository") or {}).get("full_name") == REPOSITORY,
|
||||
"build run belongs to another repository")
|
||||
require(run.get("path") == f".github/workflows/{workflow}" and run.get("event") == "workflow_dispatch"
|
||||
and run.get("head_branch") == "main", "candidate must originate from the trusted main workflow")
|
||||
require(type(run.get("id")) is int and run["id"] > 0 and type(run.get("run_attempt")) is int and run["run_attempt"] > 0,
|
||||
"invalid workflow run identity")
|
||||
require(isinstance(run.get("head_sha"), str) and HEX40.fullmatch(run["head_sha"]), "invalid workflow source SHA")
|
||||
if completed:
|
||||
require(run.get("status") == "completed" and run.get("conclusion") == "success", "candidate run did not succeed")
|
||||
return run
|
||||
|
||||
|
||||
def validate_artifact(artifact, run, name):
|
||||
require(artifact.get("name") == name and artifact.get("expired") is False, "missing or expired candidate artifact")
|
||||
origin = artifact.get("workflow_run") or {}
|
||||
require(origin.get("id") == run["id"] and origin.get("head_sha") == run["head_sha"], "artifact belongs to another build run")
|
||||
require(type(artifact.get("id")) is int and artifact["id"] > 0 and type(artifact.get("size_in_bytes")) is int
|
||||
and 0 < artifact["size_in_bytes"] <= 8 * MAX_MEMBER, "invalid artifact identity or size")
|
||||
require(isinstance(artifact.get("digest"), str) and re.fullmatch(r"sha256:[0-9a-f]{64}", artifact["digest"]),
|
||||
"artifact has no immutable digest")
|
||||
return artifact
|
||||
|
||||
|
||||
def artifact_list(run_id):
|
||||
require(type(run_id) is int and run_id > 0, "invalid run ID")
|
||||
result = []
|
||||
for page in range(1, 101):
|
||||
batch = github(f"repos/{REPOSITORY}/actions/runs/{run_id}/artifacts?per_page=100&page={page}")["artifacts"]
|
||||
result.extend(batch)
|
||||
if len(batch) < 100:
|
||||
return result
|
||||
raise ValueError("too many candidate artifacts")
|
||||
|
||||
|
||||
def download_artifact(run, name, destination):
|
||||
matches = [item for item in artifact_list(run["id"]) if item.get("name") == name]
|
||||
require(len(matches) == 1, "candidate artifact is absent or ambiguous")
|
||||
artifact = validate_artifact(matches[0], run, name)
|
||||
destination = Path(destination)
|
||||
destination.mkdir(parents=True, exist_ok=True)
|
||||
with tempfile.TemporaryDirectory() as temp:
|
||||
archive_path = Path(temp) / "artifact.zip"
|
||||
with archive_path.open("wb") as stream, github_stream(f"repos/{REPOSITORY}/actions/artifacts/{artifact['id']}/zip") as response:
|
||||
received = 0
|
||||
for chunk in iter(lambda: response.read(1024 * 1024), b""):
|
||||
received += len(chunk)
|
||||
require(received <= artifact["size_in_bytes"], "artifact response exceeds its declared size")
|
||||
stream.write(chunk)
|
||||
require(archive_path.stat().st_size == artifact["size_in_bytes"] and
|
||||
"sha256:" + file_digest(archive_path) == artifact["digest"], "downloaded artifact digest or size mismatch")
|
||||
with zipfile.ZipFile(archive_path) as archive:
|
||||
members = archive.infolist()
|
||||
require(0 < len(members) <= 256 and len({m.filename for m in members}) == len(members), "invalid artifact members")
|
||||
require(sum(member.file_size for member in members) <= 8 * MAX_MEMBER, "expanded artifact exceeds the candidate size limit")
|
||||
for member in members:
|
||||
require(re.fullmatch(r"[A-Za-z0-9][A-Za-z0-9._-]{0,200}", member.filename)
|
||||
and not member.is_dir() and 0 < member.file_size <= MAX_MEMBER
|
||||
and (member.external_attr >> 16) & 0o170000 != 0o120000, "unsafe artifact member")
|
||||
target = destination / member.filename
|
||||
require(not target.exists(), "artifact extraction would overwrite a file")
|
||||
with archive.open(member) as source, target.open("xb") as sink:
|
||||
import shutil
|
||||
shutil.copyfileobj(source, sink)
|
||||
return artifact
|
||||
|
||||
|
||||
def emit_output(name, value):
|
||||
require("\n" not in str(value) and "\r" not in str(value), "invalid workflow output")
|
||||
with open(os.environ["GITHUB_OUTPUT"], "a", encoding="utf-8") as stream:
|
||||
stream.write(f"{name}={value}\n")
|
||||
|
||||
|
||||
def main(argv=None):
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
sub = parser.add_subparsers(dest="command", required=True)
|
||||
p = sub.add_parser("freeze")
|
||||
for name in ("version", "source-sha", "workflow-sha", "toolchain", "console", "cli", "previous", "output"):
|
||||
p.add_argument("--" + name, required=True)
|
||||
p.add_argument("--run-id", type=int, required=True)
|
||||
p.add_argument("--run-number", type=int, required=True)
|
||||
p.add_argument("--ci-run-id", type=int, required=True)
|
||||
p.add_argument("--ci-run-attempt", type=int, required=True)
|
||||
p = sub.add_parser("validate-inputs")
|
||||
p.add_argument("--inputs", required=True)
|
||||
p.add_argument("--check-checkout", action="store_true")
|
||||
p = sub.add_parser("record")
|
||||
p.add_argument("--inputs", required=True)
|
||||
p.add_argument("--target", required=True)
|
||||
p.add_argument("--archive", required=True)
|
||||
p.add_argument("--output", required=True)
|
||||
p = sub.add_parser("assemble")
|
||||
p.add_argument("--inputs", required=True)
|
||||
p.add_argument("--directory", required=True)
|
||||
p.add_argument("--output", required=True)
|
||||
p.add_argument("--platforms", choices=("linux", "all"), default="all")
|
||||
p = sub.add_parser("download")
|
||||
p.add_argument("--run-id", type=int, required=True)
|
||||
p.add_argument("--workflow", choices=("release-candidate.yml", "release-complete.yml"), required=True)
|
||||
p.add_argument("--name", required=True)
|
||||
p.add_argument("--destination", required=True)
|
||||
args = parser.parse_args(argv)
|
||||
if args.command == "freeze":
|
||||
write_json(args.output, freeze(args.version, args.source_sha, args.workflow_sha, args.toolchain,
|
||||
read_json(args.console), args.run_id, args.run_number, read_json(args.cli), read_json(args.previous),
|
||||
{"run_id": args.ci_run_id, "run_attempt": args.ci_run_attempt}))
|
||||
elif args.command == "validate-inputs":
|
||||
inputs = validate_inputs(read_json(args.inputs))
|
||||
if args.check_checkout:
|
||||
head = command(["git", "rev-parse", "HEAD"], cwd=ROOT)
|
||||
require(head == inputs["source_sha"], "checkout differs from the frozen source")
|
||||
require(recipe_digest(head) == inputs["recipe_sha256"], "release recipe differs from frozen inputs")
|
||||
require(file_digest(ROOT / "Cargo.lock") == inputs["lock_sha256"], "dependency lock differs from frozen inputs")
|
||||
print(digest(inputs))
|
||||
elif args.command == "record":
|
||||
write_json(args.output, archive_record(args.archive, args.target, read_json(args.inputs)))
|
||||
elif args.command == "assemble":
|
||||
inputs = validate_inputs(read_json(args.inputs))
|
||||
directory = Path(args.directory)
|
||||
records = [read_json(path) for path in sorted(directory.glob("record-*.json"))]
|
||||
write_json(args.output, certification_candidate(inputs, records, directory, targets(args.platforms)))
|
||||
elif args.command == "download":
|
||||
run = validate_run(github(f"repos/{REPOSITORY}/actions/runs/{args.run_id}"), args.workflow)
|
||||
download_artifact(run, args.name, args.destination)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
raise SystemExit(main())
|
||||
except (ValueError, OSError, KeyError, TypeError, zipfile.BadZipFile, subprocess.CalledProcessError) as error:
|
||||
print(f"candidate rejected: {error}", file=sys.stderr)
|
||||
raise SystemExit(1) from None
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user