Compare commits

..

17 Commits

Author SHA1 Message Date
Hauser db375a8cf8 test(ecstore): avoid lock cleanup refcount race
Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
2026-11-01 14:42:52 +08:00
Hauser 234b05ecb8 test(connect): isolate cleanup deadline fixtures
Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
2026-11-01 12:14:56 +08:00
Hauser Zhi ea195b1d00 Merge branch 'main' into perf/cross-pool-selection 2026-11-01 12:12:44 +08:00
Chris An 890a463218 test(e2e): stabilize mixed-version coordinator assertion against leader-lock refresh gaps (#9103)
Replace the immediate post-replacement leader identity comparison with a
polling stabilization loop (up to 20 x 500ms). A leader-lock refresh gap
during old-binary replacement can make leader() briefly observe a different
node; the loop waits for the measured new coordinator to be confirmed. A
persistent change to an old-binary node still fails with the observed
identity in the error message.

Removes the redundant immediate assertion that the stabilization loop
supersedes. Also consolidates the non-mixed leader/pid resolution to avoid
a double leader query.
2026-11-01 07:03:54 +08:00
Joep 25f45f6309 test(console): strengthen brotli buffer middleware test coverage (#9098)
Extend test_buffer_brotli_responses to verify body bytes are preserved
and Content-Length matches the exact body length.  Add two new cases:
non-brotli responses pass through unmodified, and a body stream error
strips Content-Encoding so clients do not attempt to decode an empty
body as brotli (covers the fix from #9085).
2026-11-01 05:10:43 +08:00
Hauser 0b402cd661 test(e2e): isolate mc mirror fixture in full suite
Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
2026-11-01 02:40:44 +08:00
Ramakrishna Chilaka e3700614a1 perf(ecstore): optimize listing, versioned put, delete, and multipart (#9077) (#9094)
* perf(ecstore): optimize listing, versioned put, delete, and multipart (#9077)

- Buffer leaf object entries in LocalDisk::scan_dir and add zero-alloc
  FileMeta::buffers_match_strict, MetaCacheEntry::matches_strict_borrowed,
  and MetacacheReader::try_peek_buffered / skip_current for listing
- Add Linux PublicationRoot::open_directory_and_readonly_child and
  os::has_non_data_dir_entry_lazy_from_fd with RawDir stack buffer
- Fuse local versioned PUT overwrite commit, recoverable DELETE
  preparation/settlement, and multipart publication while preserving
  crash-consistency and fsync ordering invariants

Fixes #9077

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: d85aaf47-2a4b-4c2c-97a3-dbe8c7717e2a

* fix(app): use multipart storage domain facade

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): remove redundant multipart borrow

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Chris An <anzhengchao@gmail.com>
Co-authored-by: Hauser Zhi <housemecn@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Copilot-Session: d85aaf47-2a4b-4c2c-97a3-dbe8c7717e2a
2026-10-31 16:26:52 +00:00
Chris An d4ab7704b3 fix(scanner): rebuild unreadable usage floor instead of stopping leader iteration (#9101)
backlog#2918 retires the usage floor as a quota proof. After dual-corruption
recovery has been attempted, an unreadable floor is reported and rebuilt
through a CAS overwrite with a bootstrap marker rather than stopping the
leader iteration for a supervisor retry. Transient I/O errors that prevent
reading the floor revision fall through to the existing stop-and-retry path.

The identity-less floor regression from rustfs/rustfs#8670 now verifies
automatic rebuild and recovery instead of bounded backoff.
2026-11-01 01:38:39 +08:00
Hauser 884388163d perf(s3): streamline conditional multipart preflight
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
2026-10-31 22:38:26 +08:00
Chris An 422e724051 fix(ecstore): clear stale BucketMetadataSys on ECStore drop (#9097)
When sequential test fixtures share the bootstrap InstanceContext, the
BucketMetadataSys stores a Weak<ECStore> to avoid a reference cycle. The
ExpiryState holds an Arc<ECStore> that keeps the store alive past the
test's own Arc drop; the next fixture's resize_workers releases it and
the weak goes dangling before init_bucket_metadata_sys installs a fresh
BucketMetadataSys. Any object_store() call in that window panicked.

Add Drop for ECStore that clears the context's BucketMetadataSys slot
when its weak handle is stale, so the next fixture always finds a clean
slot. Fixes the heal absence_receipt_regressions and dangling_grace
test failures under sequential execution.
2026-10-31 22:25:08 +08:00
overtrue e0bf54197d fix(ecstore): clear stale BucketMetadataSys on ECStore drop
When sequential test fixtures share the bootstrap InstanceContext, the
BucketMetadataSys stores a Weak<ECStore> to avoid a reference cycle. The
ExpiryState holds an Arc<ECStore> that keeps the store alive past the
test's own Arc drop; the next fixture's resize_workers releases it and
the weak goes dangling before init_bucket_metadata_sys installs a fresh
BucketMetadataSys. Any object_store() call in that window panicked.

Add Drop for ECStore that clears the context's BucketMetadataSys slot
when its weak handle is stale, so the next fixture always finds a clean
slot. Fixes the heal absence_receipt_regressions and dangling_grace
test failures under sequential execution.
2026-10-31 21:58:51 +08:00
Hauser Zhi b4af43c884 fix(s3): guard conditional multipart pool errors (#9092)
* test(s3): cover conditional writes over delete markers

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* perf(s3): avoid repeated ETag regex compilation

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* perf(s3): borrow conditional write headers

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(s3): reject conditional completion on unknown pool

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(ci): update e2e smoke selection digest

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(ci): update full E2E selection for marker regression

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: Chris An <anzhengchao@gmail.com>
2026-10-31 20:48:34 +08:00
Chris An 1cb4c0ebe2 ci: stamp the plan contract and rename the workspace lint-and-build key (#9095)
* ci: gate the plan on a contract marker

A base planner predating the NIGHTLY_JOBS split still emits the old required
set, and the strict verifier would reject it. Select now stamps plan_contract=2
and the workflow refuses any plan without the current marker, falling back to
the full matrix, so a stale base planner can neither authorize the new narrower
selection nor force a job the new workflow no longer schedules.

* ci: rename the workspace lint-and-build job key and evidence path

The job key test-and-lint named a job that runs no tests: it does Clippy, the
test-binary archive, doctests and the log-rule anchors, while the tests live in
workspace-tests. Anyone reading needs: [test-and-lint] would conclude tests had
already gated that dependency. The key becomes workspace-lint-and-build to
match its display name, and the shared evidence directory becomes
artifacts/workspace so the path stops asserting the same thing. The aggregate's
display name stays Test and Lint because that is the live ruleset check
context. ci-gates.md also stops attributing the ECStore test run to the wrong
job.
2026-10-31 17:41:02 +08:00
Chris An e5563e6004 feat(release): gate publication on candidate certification (R01) (#9090)
Add a fail-closed release certification path so incomplete or substituted
evidence blocks every official publication exit. scripts/release/release_gate.py
validates the versioned release-validation plan, the candidate identity and the
G01 evidence-envelope set, then re-checks promoted bytes against the passed
list. build/docker/package/helm publication jobs require that certification and
stage artifacts before promoting identical bytes.
2026-10-31 16:53:18 +08:00
Chris An 0a72b29a34 chore(e2e): isolated harness event drill (close after evidence) (#9089)
chore(e2e): isolated harness drill
2026-10-31 16:52:19 +08:00
Chris An 51275cb7d8 ci: move specialized lanes off the pull-request path (#9093)
Connect Feature Boundaries, real io_uring and the rio-v2, Swift, SFTP and
HTTP/3 feature lanes no longer gate a merge. They run only in the scheduled
and manual full-validation runs, which report a regression within a day; on
the pull-request path they bought no earlier signal worth the runner
occupancy (Connect alone measured 53.5 median minutes and occasionally set
the wall). Workspace lint and tests, the short core contract, the debug
build, E2E smoke and the S3 lanes stay required on PRs and merge groups.
2026-10-31 16:52:03 +08:00
Chris An 3bf270b49f ci: record CI01 measurements, drill evidence and fix the OIDC live gate (#9091)
* ci: retain CI01 timing sample and fix the OIDC live gate tool

Record the measured PR feedback and runner occupancy for the CI plan's CI01
criterion, including the honest gap between the experimental targets and the
sampled medians, and replace the awscurl dependency in the live Keycloak gate
with the AWS CLI so the security suite runs on PEP 668 managed fleets.

* docs(testing): attach live run evidence to the event-drill index
2026-10-31 05:10:38 +00:00
144 changed files with 8340 additions and 549 deletions
+2 -2
View File
@@ -1,2 +1,2 @@
sha256-darwin=5d4509d69c3106c1b8a6c353d123440d6b4dbe5dd42612e42dcb864946beae6c
sha256-linux=414e1dae735e6d14821a992cebc09a545576e9a77e10a5e0ed671ab07b89df98
sha256-darwin=7c705942d3a9e583ebaa85845daa21c4f81fdbf36def24db931aaef38636c53e
sha256-linux=5c33dc395233671cb77452f8fb12900ecdbeb368632222b96835d310bce1aab8
+1 -1
View File
@@ -1 +1 @@
sha256=e9dc896774122a304e2b8b90f28c82d4d4e0af8ded36a82b147b30945ec994e3
sha256=00f8a56b75990b487a594b97c36610b94476a6944ba00473e37187ea654d42bf
+8 -2
View File
@@ -234,7 +234,7 @@ test-group = 'embedded-test-ports'
# processes cannot exhaust those budgets. Keep deadlines and assertions intact.
# The short HTTP fixtures are tracked in rustfs/backlog#2714.
[[profile.default.overrides]]
filter = 'package(rustfs) & (binary(/^connect_registration$/) | binary(connect_inventory) | binary(connect_perf_drive) | (binary(connect_perf_client) & test(=configured_proxy_and_direct_endpoint_failures_are_distinct)) | (binary(connect_perf_object) & test(/^(real_rustfs_endpoint_and_production_cli_support_bounded_get_and_put|cleanup_uses_a_budget_independent_of_the_measurement_window|oversized_response_is_rejected_without_buffering_it)$/)))'
filter = 'package(rustfs) & (binary(/^connect_registration$/) | binary(connect_inventory) | binary(connect_perf_drive) | (binary(connect_perf_client) & test(=configured_proxy_and_direct_endpoint_failures_are_distinct)) | (binary(connect_perf_object) & test(/^(real_rustfs_endpoint_and_production_cli_support_bounded_get_and_put|cleanup_uses_a_budget_independent_of_the_measurement_window|cleanup_accepts_missing_object_and_still_deletes_the_bucket|cleanup_diagnostics_identify_each_disconnected_transport_stage|cleanup_diagnostics_identify_each_failed_http_stage_without_private_context|cleanup_diagnostics_identify_timeouts_without_resetting_the_shared_deadline|oversized_response_is_rejected_without_buffering_it)$/)))'
threads-required = "num-test-threads"
# This cancellation probe hashes the serving executable before acquiring its
@@ -548,7 +548,7 @@ filter = 'package(rustfs) & (binary(/^embedded.*_test$/) | binary(admin_diagnost
test-group = 'embedded-test-ports'
[[profile.ci.overrides]]
filter = 'package(rustfs) & (binary(/^connect_registration$/) | binary(connect_inventory) | binary(connect_perf_drive) | (binary(connect_perf_client) & test(=configured_proxy_and_direct_endpoint_failures_are_distinct)) | (binary(connect_perf_object) & test(/^(real_rustfs_endpoint_and_production_cli_support_bounded_get_and_put|cleanup_uses_a_budget_independent_of_the_measurement_window|oversized_response_is_rejected_without_buffering_it)$/)))'
filter = 'package(rustfs) & (binary(/^connect_registration$/) | binary(connect_inventory) | binary(connect_perf_drive) | (binary(connect_perf_client) & test(=configured_proxy_and_direct_endpoint_failures_are_distinct)) | (binary(connect_perf_object) & test(/^(real_rustfs_endpoint_and_production_cli_support_bounded_get_and_put|cleanup_uses_a_budget_independent_of_the_measurement_window|cleanup_accepts_missing_object_and_still_deletes_the_bucket|cleanup_diagnostics_identify_each_disconnected_transport_stage|cleanup_diagnostics_identify_each_failed_http_stage_without_private_context|cleanup_diagnostics_identify_timeouts_without_resetting_the_shared_deadline|oversized_response_is_rejected_without_buffering_it)$/)))'
threads-required = "num-test-threads"
# Preserve the cancellation probe's startup budget and assertions in CI too.
@@ -976,6 +976,12 @@ test-group = 'e2e-vault'
filter = 'package(e2e_test) & (test(/^list_objects_v2_pagination_test::tests::(test_list_objects_v2_delimiter_small_page_traverses_all|test_list_objects_v2_max_keys_above_limit_returns_token|test_list_objects_v2_maxkeys_above_limit_with_delimiter)$/) | test(=reliant::tiering::test_hermetic_transition_restore_failure_expiry_and_retry))'
threads-required = "num-test-threads"
# The mc fixture uploads hundreds of objects concurrently before measuring listing.
# Reserve the runner so other server fixtures do not starve its disk operations.
[[profile.e2e-full.overrides]]
filter = 'package(e2e_test) & test(=mc_mirror_small_bucket_test::test_mc_mirror_small_bucket_completes_without_list_timeout)'
threads-required = "num-test-threads"
# Fixed, small contract for every executable server change. These existing
# scenarios exercise write/read, anonymous denial and retained bytes on restart.
[profile.core-contract]
+77
View File
@@ -0,0 +1,77 @@
{
"schema_version": 1,
"plan_id": "rustfs-release-validation",
"description": "Versioned certification plan for every public RustFS release exit (rustfs/backlog#2662, R01). preview, prerelease and stable all require this explicit plan. Scripts/release/release_gate.py fail-closes publication on any missing, duplicate, stale, substituted or non-PASS required item.",
"release_types": ["preview", "prerelease", "stable"],
"targets": [
{"os": "linux", "arch": "x86_64", "abi": "musl"},
{"os": "linux", "arch": "aarch64", "abi": "musl"},
{"os": "linux", "arch": "x86_64", "abi": "gnu"},
{"os": "linux", "arch": "aarch64", "abi": "gnu"},
{"os": "macos", "arch": "aarch64", "abi": "none"},
{"os": "windows", "arch": "x86_64", "abi": "msvc"}
],
"targets_basis": "Platform matrix of .github/workflows/build.yml (prepare-platform-matrix). gnu and macOS targets build with the pyroscope feature; musl and Windows targets build with no extra features.",
"features": ["pyroscope"],
"trusted_workflows": [
{"repository": "rustfs/s3chaos", "workflow_id": "release-gate.yml"}
],
"trusted_workflows_basis": "The shared evidence envelope producer lives in rustfs/s3chaos (G01, fault/release_gate.rs) and its plan/live split workflow is .github/workflows/release-gate.yml there. Only results from these identities may enter a release certification; PR-uploaded JSON never gains release authority. R02 must extend this list when additional verifier workflows land, never widen it to untrusted sources.",
"lanes": [
{
"lane": "identity",
"harness_repository": "rustfs/s3chaos",
"harness_sha": "9a48f55c1d6d58ce63a8fb1ea7317e024f32f11d",
"harness_basis": "Fixed G01 evidence-envelope producer revision (rustfs/s3chaos PR #102 head recorded in rustfs/backlog#2658). R02 re-pins this when the live verifier ships.",
"timeout_seconds": 1800,
"environment_capabilities": ["process-exec"],
"cases": ["artifact-identity", "startup"],
"applies_to": {
"release_types": ["preview", "prerelease", "stable"],
"platforms": "all"
},
"applies_basis": "Artifact identity and startup checks are mandatory for every release type, preview included, on every shipped target (rustfs/backlog#2662 frozen design item 5)."
},
{
"lane": "upgrade",
"harness_repository": "rustfs/s3chaos",
"harness_sha": "9a48f55c1d6d58ce63a8fb1ea7317e024f32f11d",
"harness_basis": "Same fixed envelope producer as the identity lane; the workflow SHA and the product build SHA are bound separately and need not be equal.",
"timeout_seconds": 5400,
"environment_capabilities": ["process-exec", "previous-release-artifact"],
"cases": [
"direct-upgrade",
"mixed-version-upgrade",
"bucket-config-upgrade",
"bucket-config-rollback",
"multipart-layout-upgrade",
"multipart-layout-baseline"
],
"cases_basis": "Case IDs of the pinned previous-release matrix in .github/workflows/e2e-upgrade.yml (source.version/asset/sha256 and the case list mapped there).",
"applies_to": {
"release_types": ["prerelease", "stable"],
"platforms": [{"os": "linux", "arch": "x86_64", "abi": "gnu"}]
},
"applies_basis": "preview releases use the pre-declared smaller set (identity lane only). The upgrade lane is only exercised on linux-x86_64-gnu, the platform of the pinned source assets."
}
],
"upgrade_sources": {
"basis": "Exactly the two previous-release sources pinned in .github/workflows/e2e-upgrade.yml are exercised today (source.version / source.asset / source.sha256). No other N-1/N-2 compatibility is claimed, and these digests are not inferred from tags.",
"sources": [
{
"version": "1.0.1",
"role": "N-1",
"artifact_name": "rustfs-linux-x86_64-gnu-v1.0.1.zip",
"artifact_sha256": "9742e183cb0dfb0d48fa909797af572fe47b7f827569036baa4156e0d3958ee5"
},
{
"version": "1.0.0-rc.5",
"role": "N-2",
"artifact_name": "rustfs-linux-x86_64-gnu-v1.0.0-rc.5.zip",
"artifact_sha256": "3ee8df71e8edcfada533be452c4135868f697bc515460ae97b027313eade7a3d"
}
],
"rollback_supported": false,
"rollback_basis": "Only the bucket-config-rollback case (rollback_to_previous_release_reads_current_bucket_metadata) is exercised. Full data-plane rollback is not certified and is never a default claim."
}
}
+311 -5
View File
@@ -814,13 +814,18 @@ jobs:
export AWS_SECRET_ACCESS_KEY="$R2_SECRET_ACCESS_KEY"
export AWS_DEFAULT_REGION="auto"
# Determine upload path based on build type
# Determine upload path based on build type. Release and prerelease
# builds only stage their bytes here: the official
# artifacts/rustfs/release/ path is written solely by the
# certification-gated promote-release-artifacts job, from these same
# staged objects. Development builds keep the uncertified dev channel
# and never touch official or latest channels.
if [[ "$BUILD_TYPE" == "development" ]]; then
R2_PATH="s3://${R2_BUCKET}/artifacts/rustfs/dev/"
echo "📤 Uploading development build to R2 dev directory"
else
R2_PATH="s3://${R2_BUCKET}/artifacts/rustfs/release/"
echo "📤 Uploading release build to R2 release directory"
R2_PATH="s3://${R2_BUCKET}/artifacts/rustfs/staging/${GITHUB_RUN_ID}/${GITHUB_RUN_ATTEMPT}/"
echo "📤 Staging release build for certification-gated promotion"
fi
# Upload all rustfs zip files to R2 using S3-compatible API
@@ -869,6 +874,221 @@ jobs:
echo "This run does not trigger Docker publishing."
fi
# Aggregate the parent candidate batch. This job always collects results and
# retains its evidence, including on failure, and every publication job
# below requires its success. Missing, stale, substituted or non-PASS
# evidence fails closed here: only the trusted verification sources
# (rustfs/s3chaos release-gate envelopes, rustfs/backlog#2658) can satisfy
# it, so a PR-uploaded JSON set never gains release authority.
release-certification:
name: Release Certification
needs: [ build-check, build-rustfs ]
if: ${{ !cancelled() && needs.build-check.outputs.should_publish == 'true' && startsWith(github.ref, 'refs/tags/') }}
runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
timeout-minutes: 30
permissions:
contents: read
actions: read
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Download all build artifacts
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
path: ./artifacts
pattern: rustfs-*
merge-multiple: true
- name: Write candidate and collect trusted validation results
env:
# Cross-repo read of the trusted verification run. Falls back to the
# workflow token, which cannot read rustfs/s3chaos; a missing token
# therefore fail-closes instead of certifying.
GH_TOKEN: ${{ secrets.RELEASE_VALIDATION_TOKEN || github.token }}
PRODUCT_COMMIT: ${{ github.sha }}
VERSION: ${{ needs.build-check.outputs.version }}
BUILD_TYPE: ${{ needs.build-check.outputs.build_type }}
FEATURES: ${{ needs.build-check.outputs.build_type == 'development' && '' || 'pyroscope' }}
shell: bash
run: |
set -euo pipefail
mkdir -p results release-gate-output
PLAN_SHA256=$(python3 -c "import hashlib,json,sys; sys.path.insert(0,'scripts/release'); import release_gate; print(release_gate.canonical_sha256(json.load(open('.config/release-validation-plan.json'))))")
python3 - "$PLAN_SHA256" > candidate.json <<'PY'
import hashlib, json, os, sys, zipfile
plan_sha256 = sys.argv[1]
profile = {"release": "stable", "prerelease": "prerelease", "preview": "preview"}.get(os.environ["BUILD_TYPE"])
if profile is None:
raise SystemExit("release certification only covers preview, prerelease and stable builds")
def sha256(path):
digest = hashlib.sha256()
with open(path, "rb") as handle:
for chunk in iter(lambda: handle.read(1024 * 1024), b""):
digest.update(chunk)
return digest.hexdigest()
deliverables = []
for name in sorted(os.listdir("artifacts")):
if not name.startswith("rustfs-") or not name.endswith(".zip"):
continue
path = os.path.join("artifacts", name)
members = zipfile.ZipFile(path).namelist()
binary_name = "rustfs.exe" if "rustfs.exe" in members else "rustfs"
with zipfile.ZipFile(path) as archive:
binary_bytes = archive.read(binary_name)
import tempfile
with tempfile.NamedTemporaryFile() as handle:
handle.write(binary_bytes)
handle.flush()
binary_sha = sha256(handle.name)
deliverables.append({
"kind": "zip",
"name": name,
"location": "https://github.com/rustfs/rustfs/releases/download/{}/{}".format(os.environ["VERSION"], name),
"sha256": sha256(path),
"size": os.path.getsize(path),
"binary_sha256": binary_sha,
})
if not deliverables:
raise SystemExit("no built deliverables found to certify")
candidate = {
"schema_version": 1,
"product_commit": os.environ["PRODUCT_COMMIT"],
"version": os.environ["VERSION"],
"release_profile": profile,
"features": [f for f in os.environ.get("FEATURES", "").split(",") if f],
"plan_sha256": plan_sha256,
"validation_batch": {"run_id": os.environ["GITHUB_RUN_ID"], "run_attempt": int(os.environ["GITHUB_RUN_ATTEMPT"])},
"deliverables": deliverables,
}
json.dump(candidate, sys.stdout, indent=2, sort_keys=True)
print()
PY
# Trusted verification evidence only: the release-gate envelopes of
# rustfs/s3chaos for this exact product commit. Absence blocks below.
TRUSTED_REPO="rustfs/s3chaos"
TRUSTED_WORKFLOW="release-gate.yml"
TRUSTED_RUN=$(gh api "repos/${TRUSTED_REPO}/actions/workflows/${TRUSTED_WORKFLOW}/runs?head_sha=${PRODUCT_COMMIT}&status=success&per_page=1" --jq '.workflow_runs[0].id // empty' || true)
if [[ -z "$TRUSTED_RUN" ]]; then
echo "❌ No trusted verification run of ${TRUSTED_REPO}/${TRUSTED_WORKFLOW} for ${PRODUCT_COMMIT}; refusing to certify" >&2
else
gh run download "$TRUSTED_RUN" -R "$TRUSTED_REPO" -n release-validation-evidence -D results || \
echo "❌ Trusted verification run ${TRUSTED_RUN} published no release-validation-evidence artifact" >&2
fi
python3 scripts/release/release_gate.py validate \
--plan .config/release-validation-plan.json \
--candidate candidate.json \
--results results \
--output release-gate-output
- name: Upload certification evidence
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: release-certification
path: |
candidate.json
results/
release-gate-output/
if-no-files-found: warn
retention-days: 30
# The only writer of the official R2 release path. Promotion re-checks the
# exact bytes about to be published against the passed certification list
# and copies the staged objects byte-for-byte: no rebuild-after-verify.
promote-release-artifacts:
name: Promote Release Artifacts
needs: [ build-check, build-rustfs, release-certification ]
# Target-channel lock for the official R2 release path.
concurrency:
group: release-channel-r2-release
cancel-in-progress: false
if: needs.build-check.outputs.should_publish == 'true' && startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
runs-on: ubuntu-latest
timeout-minutes: 30
permissions:
contents: read
actions: read
env:
R2_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
R2_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
R2_ENDPOINT: ${{ secrets.R2_ENDPOINT }}
R2_BUCKET: ${{ secrets.R2_BUCKET }}
AWS_EC2_METADATA_DISABLED: true
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Download build and certification artifacts
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
path: ./artifacts
pattern: rustfs-*
merge-multiple: true
- name: Download certification evidence
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: release-certification
path: ./certification
- name: Verify and promote staged release artifacts
shell: bash
run: |
set -euo pipefail
if [[ -z "$R2_ACCESS_KEY_ID" || -z "$R2_SECRET_ACCESS_KEY" || -z "$R2_ENDPOINT" || -z "$R2_BUCKET" ]]; then
echo "❌ R2 credentials missing; refusing to promote to the official channel" >&2
exit 1
fi
for zip_file in artifacts/rustfs-*.zip; do
[[ -f "$zip_file" ]] || continue
name="$(basename "$zip_file")"
python3 scripts/release/release_gate.py verify-promotion \
--plan .config/release-validation-plan.json \
--candidate certification/candidate.json \
--summary certification/release-gate-output \
--artifact "$zip_file" \
--artifact-name "$name"
done
if ! command -v aws >/dev/null 2>&1; then
sudo apt-get update && sudo apt-get install -y awscli
fi
export AWS_ACCESS_KEY_ID="$R2_ACCESS_KEY_ID"
export AWS_SECRET_ACCESS_KEY="$R2_SECRET_ACCESS_KEY"
export AWS_DEFAULT_REGION="auto"
STAGING_PATH="s3://${R2_BUCKET}/artifacts/rustfs/staging/${GITHUB_RUN_ID}/${GITHUB_RUN_ATTEMPT}/"
RELEASE_PATH="s3://${R2_BUCKET}/artifacts/rustfs/release/"
for zip_file in artifacts/rustfs-*.zip; do
[[ -f "$zip_file" ]] || continue
name="$(basename "$zip_file")"
staged="$(mktemp)"
aws s3 cp "${STAGING_PATH}${name}" "$staged" --endpoint-url "$R2_ENDPOINT" --only-show-errors
staged_sha="$(sha256sum "$staged" | awk '{print $1}')"
built_sha="$(sha256sum "$zip_file" | awk '{print $1}')"
if [[ "$staged_sha" != "$built_sha" ]]; then
echo "❌ Staged object ${name} does not match the verified bytes" >&2
exit 1
fi
aws s3 cp "$staged" "${RELEASE_PATH}${name}" --endpoint-url "$R2_ENDPOINT" --only-show-errors
rm -f "$staged"
echo "✅ Promoted ${name} (${built_sha})"
done
# Create GitHub Release for every valid release tag, including previews
create-release:
name: Create GitHub Release
@@ -1028,6 +1248,10 @@ jobs:
update-latest-version:
name: Update Latest Version
needs: [ build-check, publish-release ]
# Target-channel lock for the latest.json version pointer.
concurrency:
group: release-channel-oss-latest
cancel-in-progress: false
if: needs.build-check.outputs.should_publish == 'true' && startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
runs-on: ubuntu-latest
timeout-minutes: 30
@@ -1068,6 +1292,45 @@ jobs:
rm -rf "$OSSUTIL_DIR" "$OSSUTIL_ZIP"
chmod +x "$OSSUTIL_BIN"
# Candidate/pointer ordering: an older candidate must never overwrite
# a newer pointer, even if its promotion job finishes later.
CURRENT_LATEST="${RUNNER_TEMP}/current-latest.json"
if "$OSSUTIL_BIN" cp oss://rustfs-version/latest.json "$CURRENT_LATEST" 2>/dev/null; then
python3 - "$CURRENT_LATEST" "$VERSION" <<'PY'
import json
import re
import sys
def key(version):
match = re.match(r"^(\d+)\.(\d+)\.(\d+)(?:-([0-9A-Za-z.-]+))?$", version)
if not match:
raise SystemExit(3)
core = tuple(int(part) for part in match.groups()[:3])
prerelease = match.group(4)
if prerelease is None:
return core + (1,)
return core + (0, tuple(int(p) if p.isdigit() else p for p in re.split(r"[.-]", prerelease)))
try:
current = json.load(open(sys.argv[1], encoding="utf-8")).get("version", "")
proposed = sys.argv[2]
if current and key(current) > key(proposed):
raise SystemExit(1)
except SystemExit:
raise
except (OSError, ValueError, TypeError):
raise SystemExit(3)
PY
guard_status=$?
if [[ "$guard_status" -eq 1 ]]; then
echo "Current latest.json points at a newer version; refusing to downgrade" >&2
exit 0
elif [[ "$guard_status" -ne 0 ]]; then
echo "Existing latest.json is unparsable; refusing to overwrite it blindly" >&2
exit 1
fi
fi
# Create latest.json
cat > latest.json << EOF
{
@@ -1084,16 +1347,59 @@ jobs:
echo "✅ Updated latest.json for ${RELEASE_TYPE} release $VERSION"
# Publish release (remove draft status)
# Publish release (remove draft status). Requires the release-certification
# aggregate: without a complete legal success there is no public write.
publish-release:
name: Publish Release
needs: [ build-check, create-release, upload-release-assets ]
needs: [ build-check, create-release, upload-release-assets, release-certification ]
# Target-channel lock: concurrent releases serialize on this channel so an
# older candidate cannot interleave its publish with a newer one.
concurrency:
group: release-channel-github-release
cancel-in-progress: false
if: needs.build-check.outputs.should_publish == 'true' && startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'preview' || needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
timeout-minutes: 30
permissions:
contents: write
actions: read
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Download certification evidence
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: release-certification
path: ./certification
- name: Verify release assets against certification
env:
GH_TOKEN: ${{ github.token }}
shell: bash
run: |
set -euo pipefail
TAG="${{ needs.build-check.outputs.version }}"
mkdir -p "$RUNNER_TEMP/promotion-assets"
gh release download "$TAG" --dir "$RUNNER_TEMP/promotion-assets" --pattern '*.zip'
found=0
for zip_file in "$RUNNER_TEMP"/promotion-assets/*.zip; do
[[ -f "$zip_file" ]] || continue
found=1
python3 scripts/release/release_gate.py verify-promotion \
--plan .config/release-validation-plan.json \
--candidate certification/candidate.json \
--summary certification/release-gate-output \
--artifact "$zip_file" \
--artifact-name "$(basename "$zip_file")"
done
if [[ "$found" -ne 1 ]]; then
echo "No release zip assets to verify against the certification" >&2
exit 1
fi
- name: Publish release
env:
GH_TOKEN: ${{ github.token }}
+182 -2
View File
@@ -329,13 +329,75 @@ jobs:
echo " - Create latest: $create_latest"
echo " - Source ref: $source_ref"
# Consume the trusted release certification for the exact version this
# workflow wraps and verify the certified input digests. A manual dispatch
# without that certification fails here and pushes no official tag; only
# results from the trusted verification sources carry release authority.
release-certification:
name: Release Certification
needs: build-check
if: needs.build-check.outputs.should_build == 'true'
runs-on: ubuntu-latest
timeout-minutes: 30
permissions:
contents: read
actions: read
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
ref: ${{ needs.build-check.outputs.source_ref }}
- name: Download and verify release certification
env:
GH_TOKEN: ${{ secrets.RELEASE_VALIDATION_TOKEN || github.token }}
TRIGGERING_RUN_ID: ${{ github.event.workflow_run.id }}
VERSION: ${{ needs.build-check.outputs.version }}
shell: bash
run: |
set -euo pipefail
mkdir -p certification
run_id="$TRIGGERING_RUN_ID"
if [[ -z "$run_id" ]]; then
run_id="$(gh api --method GET \
"repos/${GITHUB_REPOSITORY}/actions/workflows/build.yml/runs" \
-f branch="$VERSION" -f status=success -F per_page=1 \
--jq '.workflow_runs[0].id // empty' || true)"
fi
if [[ -z "$run_id" ]]; then
echo "No certified Build and Release run found for ${VERSION}; failing closed" >&2
exit 1
fi
if ! gh run download "$run_id" -n release-certification -D certification; then
echo "Build run ${run_id} published no release-certification artifact; failing closed" >&2
exit 1
fi
python3 scripts/release/release_gate.py verify-promotion \
--plan .config/release-validation-plan.json \
--candidate certification/candidate.json \
--summary certification/release-gate-output \
--artifact-sha256 "$(jq -r '.deliverables[0].sha256' certification/candidate.json)" \
--artifact-name "$(jq -r '.deliverables[0].name' certification/candidate.json)"
# Input digest binding: every zip this image wraps must be the
# certified byte-for-byte content, never a mutable latest pointer.
while IFS=$'\t' read -r name location expected; do
echo "Verifying certified input ${name}"
actual="$(curl -fsSL "$location" | sha256sum | awk '{print $1}')"
if [[ "$actual" != "$expected" ]]; then
echo "Certified input ${name} does not match its recorded digest" >&2
exit 1
fi
done < <(jq -r '.deliverables[] | select(.name | contains("linux")) | [.name, .location, .sha256] | @tsv' certification/candidate.json)
# Build multi-arch Docker images
# Strategy: Build images using pre-built binaries from dl.rustfs.com
# Supports both release and dev channel binaries based on build context
# Only runs when should_build is true (which includes workflow success check)
build-docker:
name: Build Docker Images
needs: build-check
needs: [ build-check, release-certification ]
if: needs.build-check.outputs.should_build == 'true'
runs-on: ubuntu-latest
timeout-minutes: 60
@@ -443,6 +505,14 @@ jobs:
# Output tags
echo "tags=$TAGS" >> "$GITHUB_OUTPUT"
# Staging identity for this run's promotion: official tags are only
# created after the staged image verifies against the certification,
# and they point at this same digest (no rebuild-after-verify).
STAGING_TAG="staging-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}${VARIANT_SUFFIX}"
STAGING_TAGS="${{ env.REGISTRY_DOCKERHUB }}:${STAGING_TAG},${{ env.REGISTRY_GHCR }}:${STAGING_TAG},${{ env.REGISTRY_QUAY }}:${STAGING_TAG}"
echo "staging_tags=$STAGING_TAGS" >> "$GITHUB_OUTPUT"
echo "staging_tag=$STAGING_TAG" >> "$GITHUB_OUTPUT"
# Generate labels
LABELS="org.opencontainers.image.title=RustFS"
LABELS="$LABELS,org.opencontainers.image.description=RustFS distributed object storage system"
@@ -467,7 +537,9 @@ jobs:
file: ${{ matrix.file }}
platforms: ${{ env.DOCKER_PLATFORMS }}
push: ${{ needs.build-check.outputs.should_push == 'true' }}
tags: ${{ steps.meta.outputs.tags }}
# Push to staging tags only: official tags are created by the
# certification-gated promotion step below from this same digest.
tags: ${{ steps.meta.outputs.staging_tags }}
labels: ${{ steps.meta.outputs.labels }}
# No layer cache. This build compiles nothing — it downloads a
# release zip and runs apk/apt — so the cache could only save the
@@ -493,6 +565,114 @@ jobs:
no-cache: false
pull: true
# A later-built wrapper: the staged image must complete this platform's
# own binary identity verification before any official tag is written.
# It may not inherit an unverified package's conclusion.
- name: Verify staged image against certification
if: needs.build-check.outputs.should_push == 'true'
env:
GH_TOKEN: ${{ secrets.RELEASE_VALIDATION_TOKEN || github.token }}
TRIGGERING_RUN_ID: ${{ github.event.workflow_run.id }}
VERSION: ${{ needs.build-check.outputs.version }}
shell: bash
run: |
set -euo pipefail
mkdir -p certification
run_id="$TRIGGERING_RUN_ID"
if [[ -z "$run_id" ]]; then
run_id="$(gh api --method GET \
"repos/${GITHUB_REPOSITORY}/actions/workflows/build.yml/runs" \
-f branch="$VERSION" -f status=success -F per_page=1 \
--jq '.workflow_runs[0].id // empty' || true)"
fi
gh run download "$run_id" -n release-certification -D certification
STAGING_REF="${{ env.REGISTRY_DOCKERHUB }}:staging-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}${{ matrix.suffix }}"
case "${{ matrix.variant }}" in
musl) libc="musl" ;;
glibc) libc="gnu" ;;
esac
for platform in linux/amd64 linux/arm64; do
case "$platform" in
linux/amd64) arch_sub="x86_64" ;;
linux/arm64) arch_sub="aarch64" ;;
esac
docker pull --platform "$platform" "$STAGING_REF"
container="$(docker create --platform "$platform" "$STAGING_REF")"
docker cp "$container:/usr/bin/rustfs" "$RUNNER_TEMP/staged-rustfs"
docker rm "$container" >/dev/null
actual="$(sha256sum "$RUNNER_TEMP/staged-rustfs" | awk '{print $1}')"
expected="$(jq -r --arg needle "linux-${arch_sub}-${libc}" \
'[.deliverables[] | select(.name | contains($needle)) | .binary_sha256][0] // empty' \
certification/candidate.json)"
if [[ -z "$expected" || "$actual" != "$expected" ]]; then
echo "Staged image binary for ${platform} does not match the certified binary digest" >&2
exit 1
fi
python3 scripts/release/release_gate.py verify-promotion \
--plan .config/release-validation-plan.json \
--candidate certification/candidate.json \
--summary certification/release-gate-output \
--artifact "$RUNNER_TEMP/staged-rustfs" \
--artifact-role binary
done
- name: Promote staged image to official tags
if: needs.build-check.outputs.should_push == 'true'
shell: bash
run: |
set -euo pipefail
STAGING_REF="${{ env.REGISTRY_DOCKERHUB }}:staging-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}${{ matrix.suffix }}"
IFS=',' read -r -a official_tags <<< "${{ steps.meta.outputs.tags }}"
for tag in "${official_tags[@]}"; do
docker buildx imagetools create -t "$tag" "$STAGING_REF"
echo "Promoted $tag -> $STAGING_REF"
done
# Downstream consumers (Helm) verify this record instead of inheriting
# an unverified wrapper's conclusion.
- name: Record image promotion
if: needs.build-check.outputs.should_push == 'true'
shell: bash
env:
GH_TOKEN: ${{ secrets.RELEASE_VALIDATION_TOKEN || github.token }}
TRIGGERING_RUN_ID: ${{ github.event.workflow_run.id }}
VERSION: ${{ needs.build-check.outputs.version }}
run: |
set -euo pipefail
mkdir -p certification
run_id="$TRIGGERING_RUN_ID"
if [[ -z "$run_id" ]]; then
run_id="$(gh api --method GET \
"repos/${GITHUB_REPOSITORY}/actions/workflows/build.yml/runs" \
-f branch="$VERSION" -f status=success -F per_page=1 \
--jq '.workflow_runs[0].id // empty' || true)"
fi
gh run download "$run_id" -n release-certification -D certification
STAGING_REF="${{ env.REGISTRY_DOCKERHUB }}:staging-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}${{ matrix.suffix }}"
digest="$(docker buildx imagetools inspect "$STAGING_REF" --format '{{json .Manifest}}' | jq -r '.digest')"
candidate_sha="$(python3 -c "import json,hashlib,sys; sys.path.insert(0,'scripts/release'); import release_gate; print(release_gate.canonical_sha256(json.load(open('certification/candidate.json'))))")"
jq -n \
--arg version "$VERSION" \
--arg digest "$digest" \
--arg staging_ref "$STAGING_REF" \
--arg candidate_sha256 "$candidate_sha" \
--arg tags "${{ steps.meta.outputs.tags }}" \
--slurpfile candidate certification/candidate.json \
'{schema_version: 1, version: $version, image_digest: $digest, staging_ref: $staging_ref,
promoted_tags: ($tags | split(",")), candidate_sha256: $candidate_sha256,
binary_sha256s: ($candidate[0].deliverables | map(.binary_sha256) | unique)}' \
> image-promotion-record.json
- name: Upload image promotion record
if: needs.build-check.outputs.should_push == 'true'
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: image-promotion-record${{ matrix.suffix }}
path: image-promotion-record.json
if-no-files-found: error
retention-days: 30
# Note: Manifest creation is no longer needed as we only build one variant
# Multi-arch manifests are automatically created by docker/build-push-action
+93
View File
@@ -30,9 +30,102 @@ permissions:
contents: read
jobs:
# Consume the same trusted release certification as every other publication
# exit and verify the image promotion record of the Docker workflow. The
# chart wraps no binary of its own, so it may only reference an image whose
# platform verification already passed; it must not inherit an unverified
# package's conclusion. A run without that evidence fails closed here and
# never reaches the chart repository.
release-certification:
runs-on: ubuntu-latest
timeout-minutes: 30
permissions:
contents: read
actions: read
outputs:
version: ${{ steps.version.outputs.raw_tag }}
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Normalize release version
id: version
env:
RAW_INPUT: ${{ github.event.inputs.version }}
RAW_BRANCH: ${{ github.event.workflow_run.head_branch }}
run: |
set -euo pipefail
if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then
RAW="$RAW_INPUT"
else
RAW="$RAW_BRANCH"
fi
case "$RAW" in
refs/tags/*) RAW_TAG="${RAW#refs/tags/}" ;;
*) RAW_TAG="$RAW" ;;
esac
echo "raw_tag=$RAW_TAG" >> "$GITHUB_OUTPUT"
- name: Download and verify release certification and image record
env:
GH_TOKEN: ${{ secrets.RELEASE_VALIDATION_TOKEN || github.token }}
TRIGGERING_RUN_ID: ${{ github.event.workflow_run.id }}
RAW_TAG: ${{ steps.version.outputs.raw_tag }}
shell: bash
run: |
set -euo pipefail
mkdir -p certification image-record
build_run="$TRIGGERING_RUN_ID"
if [[ -z "$build_run" ]]; then
build_run="$(gh api --method GET \
"repos/${GITHUB_REPOSITORY}/actions/workflows/build.yml/runs" \
-f branch="$RAW_TAG" -f status=success -F per_page=1 \
--jq '.workflow_runs[0].id // empty' || true)"
fi
if [[ -z "$build_run" ]]; then
echo "No certified Build and Release run found for ${RAW_TAG}; failing closed" >&2
exit 1
fi
if ! gh run download "$build_run" -n release-certification -D certification; then
echo "Build run ${build_run} published no release-certification artifact; failing closed" >&2
exit 1
fi
python3 scripts/release/release_gate.py verify-promotion \
--plan .config/release-validation-plan.json \
--candidate certification/candidate.json \
--summary certification/release-gate-output \
--artifact-sha256 "$(jq -r '.deliverables[0].sha256' certification/candidate.json)" \
--artifact-name "$(jq -r '.deliverables[0].name' certification/candidate.json)"
docker_run="$(gh api --method GET \
"repos/${GITHUB_REPOSITORY}/actions/workflows/docker.yml/runs" \
-f branch="$RAW_TAG" -f status=success -F per_page=1 \
--jq '.workflow_runs[0].id // empty' || true)"
if [[ -z "$docker_run" ]] || ! gh run download "$docker_run" -n image-promotion-record -D image-record; then
echo "No image promotion record for ${RAW_TAG}; a chart may not wrap an unverified image" >&2
exit 1
fi
candidate_sha="$(python3 -c "import json,hashlib,sys; sys.path.insert(0,'scripts/release'); import release_gate; print(release_gate.canonical_sha256(json.load(open('certification/candidate.json'))))")"
record_sha="$(jq -r '.candidate_sha256 // empty' image-record/image-promotion-record.json)"
if [[ "$record_sha" != "$candidate_sha" ]]; then
echo "Image promotion record does not belong to this certification" >&2
exit 1
fi
overlap="$(jq -r --slurpfile candidate certification/candidate.json \
'[.binary_sha256s[]] - ($candidate[0].deliverables | map(.binary_sha256)) | length' \
image-record/image-promotion-record.json)"
if [[ "$overlap" -ne 0 ]]; then
echo "Image promotion record carries binary digests outside the certification" >&2
exit 1
fi
build-helm-package:
runs-on: ubuntu-latest
timeout-minutes: 30
needs: [ release-certification ]
if: |
(github.event_name == 'workflow_dispatch' && !contains(github.event.inputs.version, '-preview')) ||
(
+210 -14
View File
@@ -225,10 +225,57 @@ jobs:
echo " Build run ID: $BUILD_RUN_ID"
echo " Build run number: $RUN_NUMBER"
# Consume the trusted release certification of the resolved build run and
# verify the certified input digests before any package is wrapped. A run
# without that certification fails closed here; packages built later must
# complete their own verification and cannot inherit an unverified
# conclusion.
release-certification:
name: Release Certification
needs: resolve
if: needs.resolve.result == 'success'
runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
timeout-minutes: 30
permissions:
contents: read
actions: read
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Download and verify release certification
env:
GH_TOKEN: ${{ secrets.RELEASE_VALIDATION_TOKEN || github.token }}
BUILD_RUN_ID: ${{ needs.resolve.outputs.build_run_id }}
BUILD_TYPE: ${{ needs.resolve.outputs.build_type }}
shell: bash
run: |
set -euo pipefail
mkdir -p certification
if [[ "$BUILD_TYPE" == "development" ]]; then
# Development artifacts are explicitly uncertified and never touch
# official or latest channels; only the staging/dev paths remain.
jq -n '{schema_version: 1, certified: false, reason: "development artifacts are explicitly uncertified"}' \
> certification/development-uncertified.json
exit 0
fi
if ! gh run download "$BUILD_RUN_ID" -n release-certification -D certification; then
echo "Build run ${BUILD_RUN_ID} published no release-certification artifact; failing closed" >&2
exit 1
fi
python3 scripts/release/release_gate.py verify-promotion \
--plan .config/release-validation-plan.json \
--candidate certification/candidate.json \
--summary certification/release-gate-output \
--artifact-sha256 "$(jq -r '.deliverables[0].sha256' certification/candidate.json)" \
--artifact-name "$(jq -r '.deliverables[0].name' certification/candidate.json)"
# Build DEB and RPM packages for each architecture and libc variant
package:
name: Package (${{ matrix.arch }}-${{ matrix.libc }})
needs: resolve
needs: [ resolve, release-certification ]
runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
timeout-minutes: 30
strategy:
@@ -494,6 +541,55 @@ jobs:
echo "rpm_file=$RPM_FILE" >> "$GITHUB_OUTPUT"
echo "✅ RPM built: $RPM_FILE"
- name: Download certification evidence
if: needs.resolve.outputs.build_type != 'development'
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: release-certification
path: ./certification
# Later-built wrapper verification: the binary inside each package must
# match the certified binary digest for this platform before anything is
# uploaded, and the package itself stays on staging until promotion.
- name: Verify wrapped binary against certification
if: needs.resolve.outputs.build_type != 'development'
shell: bash
env:
DEB_FILE: ${{ steps.deb.outputs.deb_file }}
RPM_FILE: ${{ steps.rpm.outputs.rpm_file }}
ARTIFACT_NAME: ${{ matrix.artifact_name }}
run: |
set -euo pipefail
expected="$(jq -r --arg needle "$ARTIFACT_NAME" \
'[.deliverables[] | select(.name | contains($needle)) | .binary_sha256][0] // empty' \
certification/candidate.json)"
if [[ -z "$expected" ]]; then
echo "No certified binary digest recorded for ${ARTIFACT_NAME}" >&2
exit 1
fi
if ! command -v rpm2cpio >/dev/null 2>&1; then
sudo apt-get update && sudo apt-get install -y rpm2cpio
fi
if ! command -v cpio >/dev/null 2>&1; then
sudo apt-get update && sudo apt-get install -y cpio
fi
for package_file in "$DEB_FILE" "$RPM_FILE"; do
case "$package_file" in
*.deb)
dpkg-deb --fsys-tarfile "$package_file" | tar -xOf - ./usr/bin/rustfs > "$RUNNER_TEMP/wrapped-rustfs"
;;
*.rpm)
rpm2cpio "$package_file" | cpio -i --to-stdout ./usr/bin/rustfs > "$RUNNER_TEMP/wrapped-rustfs"
;;
esac
python3 scripts/release/release_gate.py verify-promotion \
--plan .config/release-validation-plan.json \
--candidate certification/candidate.json \
--summary certification/release-gate-output \
--artifact "$RUNNER_TEMP/wrapped-rustfs" \
--artifact-role binary
done
- name: Upload packages to artifacts
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
@@ -531,10 +627,14 @@ jobs:
export AWS_SECRET_ACCESS_KEY="$R2_SECRET_ACCESS_KEY"
export AWS_DEFAULT_REGION="auto"
# Build-phase writes stay on staging for release types: the official
# packages/release and packages/latest paths are written only by the
# certification-gated promote-packages job, from these same staged
# objects. Development packages keep the uncertified dev channel.
if [[ "$BUILD_TYPE" == "development" ]]; then
R2_PREFIX="artifacts/rustfs/packages/dev"
else
R2_PREFIX="artifacts/rustfs/packages/release"
R2_PREFIX="artifacts/rustfs/packages/staging/${GITHUB_RUN_ID}/${GITHUB_RUN_ATTEMPT}"
fi
R2_PATH="s3://${R2_BUCKET}/${R2_PREFIX}/"
@@ -549,18 +649,6 @@ jobs:
echo "✅ Upload complete"
# Also upload as latest for release/prerelease
if [[ "$BUILD_TYPE" == "release" || "$BUILD_TYPE" == "prerelease" ]]; then
LATEST_PATH="s3://${R2_BUCKET}/artifacts/rustfs/packages/latest/"
for f in "$DEB_FILE" "$RPM_FILE"; do
if [[ -n "$f" && -f "$f" ]]; then
echo "Uploading latest: $(basename "$f")"
aws s3 cp "$f" "$LATEST_PATH" --endpoint-url "$R2_ENDPOINT" --only-show-errors
fi
done
echo "✅ Latest packages updated"
fi
- name: Upload packages to GitHub Release
if: needs.resolve.outputs.tag != ''
env:
@@ -614,6 +702,114 @@ jobs:
echo "✅ GitHub release assets updated"
# The only writer of the official packages/release and packages/latest R2
# paths. Promotion re-checks each package's wrapped binary against the
# passed certification list and copies the staged objects byte-for-byte:
# no rebuild-after-verify, no mutable latest resolution downstream.
promote-packages:
name: Promote Packages
needs: [ resolve, package, release-certification ]
# Target-channel lock for the official package paths.
concurrency:
group: release-channel-r2-packages
cancel-in-progress: false
if: needs.resolve.outputs.tag != '' && (needs.resolve.outputs.build_type == 'release' || needs.resolve.outputs.build_type == 'prerelease')
runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
timeout-minutes: 30
permissions:
contents: read
actions: read
env:
R2_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
R2_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
R2_ENDPOINT: ${{ secrets.R2_ENDPOINT }}
R2_BUCKET: ${{ secrets.R2_BUCKET }}
AWS_EC2_METADATA_DISABLED: true
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Download packages and certification
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
path: ./packages
pattern: packages-*
merge-multiple: true
- name: Download certification evidence
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: release-certification
path: ./certification
- name: Verify and promote staged packages
shell: bash
run: |
set -euo pipefail
if [[ -z "$R2_ACCESS_KEY_ID" || -z "$R2_SECRET_ACCESS_KEY" || -z "$R2_ENDPOINT" || -z "$R2_BUCKET" ]]; then
echo "❌ R2 credentials missing; refusing to promote to the official channel" >&2
exit 1
fi
STAGING_PREFIX="s3://${R2_BUCKET}/artifacts/rustfs/packages/staging/${GITHUB_RUN_ID}/${GITHUB_RUN_ATTEMPT}/"
RELEASE_PATH="s3://${R2_BUCKET}/artifacts/rustfs/packages/release/"
LATEST_PATH="s3://${R2_BUCKET}/artifacts/rustfs/packages/latest/"
if ! command -v aws >/dev/null 2>&1; then
sudo apt-get update && sudo apt-get install -y awscli
fi
export AWS_ACCESS_KEY_ID="$R2_ACCESS_KEY_ID"
export AWS_SECRET_ACCESS_KEY="$R2_SECRET_ACCESS_KEY"
export AWS_DEFAULT_REGION="auto"
if ! command -v rpm2cpio >/dev/null 2>&1; then
sudo apt-get install -y rpm2cpio
fi
if ! command -v cpio >/dev/null 2>&1; then
sudo apt-get install -y cpio
fi
promoted=0
for package_file in packages/*.deb packages/*.rpm; do
[[ -f "$package_file" ]] || continue
name="$(basename "$package_file")"
case "$package_file" in
*.deb)
dpkg-deb --fsys-tarfile "$package_file" | tar -xOf - ./usr/bin/rustfs > "$RUNNER_TEMP/wrapped-rustfs"
;;
*.rpm)
rpm2cpio "$package_file" | cpio -i --to-stdout ./usr/bin/rustfs > "$RUNNER_TEMP/wrapped-rustfs"
;;
esac
python3 scripts/release/release_gate.py verify-promotion \
--plan .config/release-validation-plan.json \
--candidate certification/candidate.json \
--summary certification/release-gate-output \
--artifact "$RUNNER_TEMP/wrapped-rustfs" \
--artifact-role binary
staged="$(mktemp)"
aws s3 cp "${STAGING_PREFIX}${name}" "$staged" --endpoint-url "$R2_ENDPOINT" --only-show-errors
staged_sha="$(sha256sum "$staged" | awk '{print $1}')"
built_sha="$(sha256sum "$package_file" | awk '{print $1}')"
if [[ "$staged_sha" != "$built_sha" ]]; then
echo "❌ Staged package ${name} does not match the verified bytes" >&2
exit 1
fi
aws s3 cp "$staged" "${RELEASE_PATH}${name}" --endpoint-url "$R2_ENDPOINT" --only-show-errors
aws s3 cp "$staged" "${LATEST_PATH}${name}" --endpoint-url "$R2_ENDPOINT" --only-show-errors
rm -f "$staged"
promoted=$((promoted + 1))
echo "✅ Promoted ${name} (${built_sha})"
done
if [[ "$promoted" -eq 0 ]]; then
echo "No staged packages found to promote" >&2
exit 1
fi
# Summary
summary:
name: Summary
@@ -16,8 +16,193 @@
mod tests {
use crate::common::{RustFSTestEnvironment, init_logging};
use aws_sdk_s3::Client;
use aws_sdk_s3::config::http::HttpResponse;
use aws_sdk_s3::error::{ProvideErrorMetadata, SdkError};
use aws_sdk_s3::primitives::ByteStream;
use aws_sdk_s3::types::{BucketVersioningStatus, VersioningConfiguration};
use aws_sdk_s3::types::{BucketVersioningStatus, CompletedMultipartUpload, CompletedPart, VersioningConfiguration};
fn assert_not_found<E: ProvideErrorMetadata + std::fmt::Debug>(error: &SdkError<E, HttpResponse>) {
assert_eq!(error.raw_response().map(|r| r.status().as_u16()), Some(404), "{error:?}");
assert_eq!(
error.as_service_error().and_then(ProvideErrorMetadata::code),
Some("NoSuchKey"),
"{error:?}"
);
}
#[tokio::test]
async fn conditional_writes_cannot_replace_delete_marker_with_if_match() {
init_logging();
let mut env = RustFSTestEnvironment::new().await.expect("create test environment");
env.start_rustfs_server(vec![]).await.expect("start RustFS");
let client = env.create_s3_client();
let bucket = "conditional-delete-marker-proof";
create_versioned_bucket(&client, bucket).await;
client
.put_object()
.bucket(bucket)
.key("source")
.body(ByteStream::from_static(b"source"))
.send()
.await
.expect("write copy source");
for operation in ["put", "copy", "multipart"] {
let old = client
.put_object()
.bucket(bucket)
.key(operation)
.body(ByteStream::from_static(b"old"))
.send()
.await
.expect("write historical version");
let old_etag = old.e_tag().expect("historical ETag");
let marker = client
.delete_object()
.bucket(bucket)
.key(operation)
.send()
.await
.expect("create delete marker");
let marker_id = marker.version_id().expect("delete marker version");
let (upload_id, completed) = if operation == "multipart" {
let upload = client
.create_multipart_upload()
.bucket(bucket)
.key(operation)
.send()
.await
.expect("initiate multipart upload");
let upload_id = upload.upload_id().expect("upload ID").to_owned();
let part = client
.upload_part()
.bucket(bucket)
.key(operation)
.upload_id(&upload_id)
.part_number(1)
.body(ByteStream::from_static(b"new"))
.send()
.await
.expect("upload part");
let completed = CompletedMultipartUpload::builder()
.parts(
CompletedPart::builder()
.part_number(1)
.set_e_tag(part.e_tag().map(str::to_owned))
.build(),
)
.build();
(Some(upload_id), Some(completed))
} else {
(None, None)
};
for etag in [old_etag, "\"00000000000000000000000000000000\""] {
match operation {
"put" => {
let err = client
.put_object()
.bucket(bucket)
.key(operation)
.if_match(etag)
.body(ByteStream::from_static(b"new"))
.send()
.await
.expect_err("If-Match must reject marker");
assert_not_found(&err);
}
"copy" => {
let err = client
.copy_object()
.bucket(bucket)
.key(operation)
.copy_source(format!("{bucket}/source"))
.if_match(etag)
.send()
.await
.expect_err("If-Match copy must reject marker");
assert_not_found(&err);
}
_ => {
let err = client
.complete_multipart_upload()
.bucket(bucket)
.key(operation)
.upload_id(upload_id.as_deref().expect("upload ID"))
.multipart_upload(completed.clone().expect("completed parts"))
.if_match(etag)
.send()
.await
.expect_err("If-Match completion must reject marker");
assert_not_found(&err);
}
}
let versions = client
.list_object_versions()
.bucket(bucket)
.prefix(operation)
.send()
.await
.expect("list versions after rejected write");
assert_eq!(versions.versions().len(), 1, "{operation}: failed write created a version");
assert_eq!(versions.delete_markers().len(), 1);
assert_eq!(versions.delete_markers()[0].version_id(), Some(marker_id));
assert_eq!(versions.delete_markers()[0].is_latest(), Some(true));
}
match operation {
"put" => {
client
.put_object()
.bucket(bucket)
.key(operation)
.if_none_match("*")
.body(ByteStream::from_static(b"new"))
.send()
.await
.expect("If-None-Match PUT over marker");
}
"copy" => {
client
.copy_object()
.bucket(bucket)
.key(operation)
.copy_source(format!("{bucket}/source"))
.if_none_match("*")
.send()
.await
.expect("If-None-Match copy over marker");
}
_ => {
client
.complete_multipart_upload()
.bucket(bucket)
.key(operation)
.upload_id(upload_id.as_deref().expect("upload ID"))
.multipart_upload(completed.expect("completed parts"))
.if_none_match("*")
.send()
.await
.expect("If-None-Match completion over marker");
}
}
let current = client
.get_object()
.bucket(bucket)
.key(operation)
.send()
.await
.expect("read successful conditional write")
.body
.collect()
.await
.expect("read object body")
.into_bytes();
let expected: &[u8] = if operation == "copy" { b"source" } else { b"new" };
assert_eq!(current.as_ref(), expected, "{operation}: unexpected current object");
}
}
async fn create_versioned_bucket(client: &Client, bucket: &str) {
client
@@ -1636,20 +1636,53 @@ async fn hot_window_with_old_followers(
let mixed_initial_physical = if mixed { json!(converged_shards(&run)?) } else { Value::Null };
let scheduler_config = run.assert_default_cadence().await?;
let baseline = run.quiet_baseline().await?;
let leader = run.leader().await?;
let leader_pid = run.pid(leader)?;
// A leader-lock refresh gap during old-binary replacement can make
// `leader()` briefly observe a different node. Poll until the coordinator
// is stable before comparing against the measured fleet. A persistent
// change to an old-binary node is a genuine election outcome and fails.
let (leader, leader_pid) = if mixed {
let expected_leader = usize::try_from(
mixed_fleet["leader"]
.as_u64()
.ok_or("mixed fleet omitted its measured leader")?,
)?;
let expected_pid = mixed_fleet["leader_pid"]
.as_u64()
.ok_or("mixed fleet omitted its measured leader pid")?;
let mut resolved = None;
for _ in 0..20 {
let candidate = run.leader().await?;
let candidate_pid = u64::from(run.pid(candidate)?);
if Some(u64::try_from(candidate)?) == mixed_fleet["leader"].as_u64() && Some(candidate_pid) == Some(expected_pid) {
resolved = Some((candidate, run.pid(candidate)?));
break;
}
tokio::time::sleep(std::time::Duration::from_millis(500)).await;
}
match resolved {
Some(pair) => pair,
None => {
let current = match run.leader().await {
Ok(node) => Some((node, run.pid(node).ok())),
Err(_) => None,
};
return Err(format!(
"the hot window did not retain its measured new coordinator after stabilization \
(expected node {expected_leader} pid {expected_pid}, observed {current:?})"
)
.into());
}
}
} else {
let leader = run.leader().await?;
(leader, run.pid(leader)?)
};
let producer = (leader + 1) % 4;
let producers = if fault.is_some() {
vec![producer, (leader + 2) % 4]
} else {
vec![producer]
};
if mixed
&& (mixed_fleet["leader"].as_u64() != Some(u64::try_from(leader)?)
|| mixed_fleet["leader_pid"].as_u64() != Some(u64::from(leader_pid)))
{
return Err("the hot window did not retain its measured new coordinator".into());
}
let producer_pid = run.pid(producer)?;
let producer_address = run.peer_address(producer);
let [cold_a, cold_b, hot] = run.buckets.clone();
+1
View File
@@ -2146,3 +2146,4 @@ mod tests {
);
}
}
// event-drill: isolated harness-only change
@@ -1901,6 +1901,15 @@ impl BucketMetadataSys {
self.api.upgrade()
}
/// Whether the backing `ECStore` has been dropped.
///
/// Used by `InstanceContext::clear_bucket_metadata_sys_if_stale` during
/// `ECStore::drop` so a shared bootstrap context does not retain a
/// `BucketMetadataSys` whose weak handle is dangling.
pub(crate) fn is_store_stale(&self) -> bool {
self.api.upgrade().is_none()
}
fn metadata_publish_lock(&self, bucket: &str) -> Arc<Mutex<MetadataPublishLockState>> {
Self::bucket_lock_in(&self.metadata_publish_locks, bucket)
}
@@ -747,6 +747,16 @@ async fn list_path_raw_inner(
// Start every missing head read in the same round so one stalled
// disk cannot multiply the wait budget by the erasure-set width.
// Outcomes are still consumed below in stable disk-index order.
if !revjob_rx.is_cancelled() {
for (i, reader) in readers.iter_mut().enumerate() {
if errs[i].is_none()
&& pending_entries[i].is_none()
&& let Some(Ok(Some(entry))) = reader.try_peek_buffered()
{
pending_entries[i] = Some(entry);
}
}
}
let concurrent_peeks = readers.iter_mut().enumerate().filter_map(|(i, reader)| {
if errs[i].is_some() || pending_entries[i].is_some() {
return None;
@@ -878,7 +888,7 @@ async fn list_path_raw_inner(
}
if entry.name == current.name {
// If exact match, we agree.
if current.matches_strict(&entry) {
if current.matches_strict_borrowed(&entry) {
top_entries[i] = Some(entry);
agree += 1;
} else {
@@ -986,7 +996,9 @@ async fn list_path_raw_inner(
if agree == readers.len() && !opts.preserve_replica_metadata {
for r in readers.iter_mut() {
let _ = r.skip(1).await;
if !r.skip_current() {
let _ = r.skip(1).await;
}
}
if let Some(agreed_fn) = opts.agreed.as_ref() {
@@ -1009,7 +1021,7 @@ async fn list_path_raw_inner(
}
for (i, r) in readers.iter_mut().enumerate() {
if top_entries[i].is_some() {
if top_entries[i].is_some() && !r.skip_current() {
let _ = r.skip(1).await;
}
}
+1
View File
@@ -3506,6 +3506,7 @@ mod tests {
replication_decision: String::new(),
checksum: None,
multipart_completion_replayed: false,
backfilled_old_current_size: None,
}
}
}
+248 -118
View File
@@ -459,22 +459,62 @@ fn read_listing_metadata_entry_std(
root: &os::PublicationRoot,
object_dir: &Path,
) -> core::result::Result<ListingMetadataRead, ReadAllError> {
let metadata_path = object_dir.join(STORAGE_FORMAT_FILE);
let (bytes, _) = read_listing_metadata_bytes_std(root.open_readonly(&metadata_path))?;
let (file_meta, data_dirs, counts_toward_limit) = parse_listing_metadata_bytes(&bytes);
#[cfg(not(test))]
let _ = file_meta;
let has_namespace_child_candidate =
os::has_non_data_dir_entry(object_dir, &data_dirs).map_err(|err| ReadAllError::Disk(to_file_error(err).into()))?;
Ok(ListingMetadataRead {
bytes,
#[cfg(target_os = "linux")]
{
let (dir_fd, meta_file) = root
.open_directory_and_readonly_child(object_dir, STORAGE_FORMAT_FILE)
.map_err(ReadAllError::Open)?;
let (bytes, _) = read_listing_metadata_bytes_std(Ok(meta_file))?;
let mut parsed_counts: Option<bool> = None;
let (data_dirs, has_namespace_child_candidate) = os::has_non_data_dir_entry_lazy_from_fd(&dir_fd, || {
let (_, data_dirs, counts) = parse_listing_metadata_bytes(&bytes);
parsed_counts = Some(counts);
data_dirs
})
.map_err(|err| ReadAllError::Disk(to_file_error(err).into()))?;
let counts_toward_limit = parsed_counts.unwrap_or_else(|| FileMeta::bytes_count_toward_listing_limit(&bytes));
Ok(ListingMetadataRead {
#[cfg(test)]
file_meta: FileMeta::load(&bytes).ok(),
bytes,
data_dirs,
has_namespace_child_candidate,
counts_toward_limit,
})
}
#[cfg(not(target_os = "linux"))]
{
let metadata_path = object_dir.join(STORAGE_FORMAT_FILE);
let (bytes, _) = read_listing_metadata_bytes_std(root.open_readonly(&metadata_path))?;
#[cfg(test)]
file_meta,
data_dirs,
has_namespace_child_candidate,
counts_toward_limit,
})
let (file_meta, data_dirs, counts_toward_limit, has_namespace_child_candidate) = {
let (file_meta, data_dirs, counts_toward_limit) = parse_listing_metadata_bytes(&bytes);
let has_namespace_child_candidate = os::has_non_data_dir_entry(object_dir, &data_dirs)
.map_err(|err| ReadAllError::Disk(to_file_error(err).into()))?;
(file_meta, data_dirs, counts_toward_limit, has_namespace_child_candidate)
};
#[cfg(not(test))]
let (data_dirs, counts_toward_limit, has_namespace_child_candidate) = {
let mut parsed_counts: Option<bool> = None;
let (data_dirs, has_namespace_child_candidate) = os::has_non_data_dir_entry_lazy(object_dir, || {
let (_, data_dirs, counts) = parse_listing_metadata_bytes(&bytes);
parsed_counts = Some(counts);
data_dirs
})
.map_err(|err| ReadAllError::Disk(to_file_error(err).into()))?;
let counts_toward_limit = parsed_counts.unwrap_or_else(|| FileMeta::bytes_count_toward_listing_limit(&bytes));
(data_dirs, counts_toward_limit, has_namespace_child_candidate)
};
Ok(ListingMetadataRead {
bytes,
#[cfg(test)]
file_meta,
data_dirs,
has_namespace_child_candidate,
counts_toward_limit,
})
}
}
pub(crate) fn inline_metadata_rollback_dir(version_id: Uuid, meta: &FileMeta) -> Uuid {
@@ -3563,9 +3603,9 @@ async fn read_dir_entries_with_walk_stall(path: &Path, count: i32, stall: Option
loop {
let result_limit = if remaining > 0 {
(remaining as usize).min(WALK_DIR_BATCH_SIZE)
(remaining as usize).min(READ_DIR_BATCH_SIZE)
} else {
WALK_DIR_BATCH_SIZE
READ_DIR_BATCH_SIZE
};
let (next_entries, next_names, reached_eof) = read_dir_entry_batch_with_walk_stall(entries, result_limit, stall).await?;
entries = next_entries;
@@ -3579,7 +3619,11 @@ async fn read_dir_entries_with_walk_stall(path: &Path, count: i32, stall: Option
Ok(names)
}
#[cfg(test)]
const WALK_DIR_BATCH_SIZE: usize = 32;
#[cfg(not(test))]
const WALK_DIR_BATCH_SIZE: usize = 256;
const READ_DIR_BATCH_SIZE: usize = 512;
async fn read_dir_with_walk_stall(path: PathBuf, stall: Option<Duration>) -> Result<std::fs::ReadDir> {
let io_permit = acquire_context_disk_io_permit().await.map_err(DiskError::Io)?;
@@ -3598,7 +3642,7 @@ async fn read_dir_entry_batch_with_walk_stall(
let batch = spawn_blocking_with_disk_io_permit(io_permit, move || {
let mut names = Vec::with_capacity(result_limit);
let mut reached_eof = false;
for _ in 0..WALK_DIR_BATCH_SIZE {
for _ in 0..READ_DIR_BATCH_SIZE {
if names.len() == result_limit {
break;
}
@@ -3606,8 +3650,8 @@ async fn read_dir_entry_batch_with_walk_stall(
reached_eof = true;
break;
};
let _ = progress_tx.send(());
let entry = entry.map_err(to_file_error)?;
let _ = progress_tx.send(());
let name = entry.file_name().to_string_lossy().to_string();
if name.is_empty() || name == "." || name == ".." {
continue;
@@ -9574,13 +9618,8 @@ impl LocalDisk {
.and_then(|v| v.strip_suffix(".meta"))
.and_then(|v| v.parse::<usize>().ok())
.unwrap_or_default();
let data_path = self.io_get_object_path(
bucket,
&path_join_buf(&[
path.parent().unwrap_or_else(|| Path::new("")).to_string_lossy().as_ref(),
&format!("part.{num}"),
]),
)?;
let parent_str = path.parent().unwrap_or_else(|| Path::new("")).to_string_lossy();
let data_path = self.io_get_object_path(bucket, &path_join_buf(&[parent_str.as_ref(), &format!("part.{num}")]))?;
let metadata_path = self.io_get_object_path(bucket, path.to_string_lossy().as_ref());
// A part's existence check, metadata read and decode share one dispatch.
// Keep open errors unmapped for the existing missing-volume fallback.
@@ -9646,12 +9685,29 @@ impl LocalDisk {
stream_listing_metadata_batch(&object_names, tx, |object_name| {
let (bucket_path, object_dir) = build_local_disk_object_path(&io_root, &volume_str, object_name);
#[cfg(target_os = "linux")]
let path_check = check_local_disk_valid_object_path_at(&io_root, &mount_lease, &bucket_path, &object_dir);
{
if let Err(err) = check_local_disk_object_path_components(&io_root, &bucket_path, &object_dir) {
return Err(ReadAllError::Disk(err));
}
match read_listing_metadata_entry_std(&root, &object_dir) {
Ok(read) => Ok(read),
Err(err) => {
if let Err(path_err) =
check_local_disk_valid_object_path_at(&io_root, &mount_lease, &bucket_path, &object_dir)
{
Err(ReadAllError::Disk(path_err))
} else {
Err(err)
}
}
}
}
#[cfg(not(target_os = "linux"))]
let path_check = check_local_disk_valid_object_path(&io_root, &bucket_path, &object_dir);
match path_check {
Ok(()) => read_listing_metadata_entry_std(&root, &object_dir),
Err(err) => Err(ReadAllError::Disk(err)),
{
match check_local_disk_valid_object_path(&io_root, &bucket_path, &object_dir) {
Ok(()) => read_listing_metadata_entry_std(&root, &object_dir),
Err(err) => Err(ReadAllError::Disk(err)),
}
}
});
}));
@@ -10928,14 +10984,25 @@ impl LocalDisk {
}
let (pop, skip_object, dir_to_skip, scan_required) = dir_stack.pop().expect("operation should succeed");
write_metacache_obj(
out,
&MetaCacheEntry {
let more_ready_in_batch = !(opts.recursive && scan_required)
&& prefetched_batch.as_ref().is_some_and(|b| b.remaining > 0 && !b.rx.is_empty());
if more_ready_in_batch {
out.write_obj_buffered(&MetaCacheEntry {
name: pop.clone(),
..Default::default()
},
)
.await?;
})
.await
.map_err(metacache_write_error)?;
} else {
write_metacache_obj(
out,
&MetaCacheEntry {
name: pop.clone(),
..Default::default()
},
)
.await?;
}
let scan_path = pop.clone();
if opts.recursive && scan_required {
@@ -10992,7 +11059,14 @@ impl LocalDisk {
let mut spawned = false;
let (volume_dir, raw_res) = loop {
if let Some(batch) = prefetched_batch.as_mut() {
if let Some(raw_res) = with_walk_stall_deadline(stall, batch.rx.recv()).await? {
let next_res = match batch.rx.try_recv() {
Ok(res) => Some(res),
Err(tokio::sync::mpsc::error::TryRecvError::Empty) => {
with_walk_stall_deadline(stall, batch.rx.recv()).await?
}
Err(tokio::sync::mpsc::error::TryRecvError::Disconnected) => None,
};
if let Some(raw_res) = next_res {
batch.remaining -= 1;
let stop = batch.remaining == 0
|| raw_res.as_ref().map_or(true, |r| r.has_namespace_child_candidate);
@@ -11030,7 +11104,13 @@ impl LocalDisk {
};
prefetched_batch = Some(self.spawn_listing_metadata_batch(&opts.bucket, batch_names, stall).await?);
};
with_walk_stall_timeout(stall, self.resolve_read_all_result(&opts.bucket, &volume_dir, raw_res)).await
match raw_res {
Ok(read) => Ok(read),
Err(err) => {
with_walk_stall_timeout(stall, self.resolve_read_all_result(&opts.bucket, &volume_dir, Err(err)))
.await
}
}
}
.await
} else {
@@ -11063,12 +11143,21 @@ impl LocalDisk {
meta.metadata = read.bytes;
write_metacache_obj(out, &meta).await?;
if opts.limit <= 0 || read.counts_toward_limit {
*objs_returned += 1;
}
let reached_limit = opts.limit > 0 && *objs_returned >= opts.limit;
let more_ready_in_batch = !reached_limit
&& !read.has_namespace_child_candidate
&& dir_stack.is_empty()
&& prefetched_batch.as_ref().is_some_and(|b| b.remaining > 0 && !b.rx.is_empty());
if more_ready_in_batch {
out.write_obj_buffered(&meta).await.map_err(metacache_write_error)?;
} else {
write_metacache_obj(out, &meta).await?;
}
let has_namespace_child_candidate = read.has_namespace_child_candidate;
let dir_to_skip: HashSet<String> = if has_namespace_child_candidate || !dir_objes.is_empty() {
read.data_dirs.into_iter().map(|dir| dir.to_string()).collect()
@@ -11550,7 +11639,7 @@ fn is_root_path(path: impl AsRef<Path>) -> bool {
}
fn metadata_counts_toward_limit(metadata: &[u8]) -> bool {
FileMeta::load(metadata).map_or(true, |meta| file_meta_counts_toward_limit(&meta))
FileMeta::bytes_count_toward_listing_limit(metadata)
}
fn file_meta_counts_toward_limit(meta: &FileMeta) -> bool {
@@ -13227,7 +13316,7 @@ impl DiskAPI for LocalDisk {
let durability = effective_durability(dst_volume);
let part = ObjectPartInfo::unmarshal(&meta)?;
if let Some(integrity) = part.integrity {
let integrity_prep = if let Some(integrity) = part.integrity {
integrity.validate()?;
if usize::try_from(integrity.number).map_err(|_| DiskError::FileCorrupt)? != part.number
|| usize::try_from(integrity.size).map_err(|_| DiskError::FileCorrupt)? != part.size
@@ -13240,55 +13329,84 @@ impl DiskAPI for LocalDisk {
let destination = dst_file_path.parent().ok_or(DiskError::FileCorrupt)?.join(&proof_name);
check_path_length(source.to_string_lossy().as_ref())?;
check_path_length(destination.to_string_lossy().as_ref())?;
let stat = fs::symlink_metadata(&source).await.map_err(to_file_error)?;
if !stat.is_file() || stat.len() != u64::try_from(integrity.index_size()?).map_err(|_| DiskError::FileCorrupt)? {
return Err(DiskError::FileCorrupt);
let expected_len = u64::try_from(integrity.index_size()?).map_err(|_| DiskError::FileCorrupt)?;
Some((integrity, source, destination, expected_len))
} else {
None
};
let sync = durability.syncs_data_shards() || durability.syncs_commit_metadata();
let publication_root = self.publication_root.clone();
let lease = os::acquire_namespace_mutation_lease_with_owner(&transaction_path, source_owner).await;
os::run_blocking_namespace_operation_with_sync(lease, sync, move || -> std::io::Result<Result<()>> {
if let Some(ref admission) = admission
&& let Err(err) = admission.validate()
{
return Ok(Err(to_file_error(err).into()));
}
if durability.syncs_data_shards() {
let source = source.clone();
let retained_owner = source_owner.clone();
tokio::task::spawn_blocking(move || {
let _owner = retained_owner;
os::sync_file(&source)
})
.await
.map_err(DiskError::from)?
.map_err(to_file_error)?;
}
// Publish the immutable generation before preparing the part switch.
// Rollback retains the old generation; an unreferenced new index is
// reclaimed with the upload directory if preparation is interrupted.
os::rename_all_with_precondition_and_parent_sync(
&source,
&destination,
&dst_volume_dir,
&self.publication_root,
source_owner.clone(),
admission.as_ref().map(|check| check.precondition()),
durability.syncs_commit_metadata(),
)
.await?;
}
let retained_owner = source_owner.clone();
tokio::task::spawn_blocking(move || {
let _owner = retained_owner;
if let Some(admission) = admission {
admission.validate().map_err(to_file_error)?;
}
let source = std::fs::symlink_metadata(&src_file_path).map_err(to_file_error)?;
let source = match std::fs::symlink_metadata(&src_file_path) {
Ok(source) => source,
Err(err) => return Ok(Err(to_file_error(err).into())),
};
if !source.is_file() {
return Err(DiskError::FileAccessDenied);
return Ok(Err(DiskError::FileAccessDenied));
}
if transaction_path.exists() {
return Err(DiskError::FileAccessDenied);
return Ok(Err(DiskError::FileAccessDenied));
}
if let Some((integrity, source, destination, expected_len)) = integrity_prep {
let stat = match std::fs::symlink_metadata(&source) {
Ok(stat) => stat,
Err(err) => return Ok(Err(to_file_error(err).into())),
};
if !stat.is_file() || stat.len() != expected_len {
return Ok(Err(DiskError::FileCorrupt));
}
let verify_and_sync = (|| -> std::io::Result<bool> {
let mut file = std::fs::File::open(&source)?;
let mut header = [0u8; rustfs_filemeta::shard_integrity::INDEX_HEADER_SIZE];
std::io::Read::read_exact(&mut file, &mut header)?;
if header != integrity.index_header() {
return Ok(false);
}
if durability.syncs_data_shards() {
file.sync_all()?;
}
Ok(true)
})();
match verify_and_sync {
Ok(true) => {}
Ok(false) => return Ok(Err(DiskError::FileCorrupt)),
Err(err) => return Ok(Err(to_file_error(err).into())),
}
let precondition = admission.as_ref().map(|check| check.precondition());
// Publish the immutable generation before preparing the part switch.
// Rollback retains the old generation; an unreferenced new index is
// reclaimed with the upload directory if preparation is interrupted.
// Defer the parent directory fsync to the transaction_parent fsync
// below when both live in the same parent directory.
let same_parent = destination.parent() == transaction_path.parent();
if let Err(err) = os::rename_all_with_precondition_and_parent_sync_std(
&source,
&destination,
&dst_volume_dir,
&publication_root,
precondition.as_deref(),
durability.syncs_commit_metadata() && !same_parent,
) {
return Ok(Err(to_file_error(err).into()));
}
}
let Some(transaction_parent) = transaction_path.parent() else {
return Err(DiskError::InvalidPath);
return Ok(Err(DiskError::InvalidPath));
};
std::fs::create_dir_all(transaction_parent).map_err(to_file_error)?;
if let Err(err) = std::fs::create_dir_all(transaction_parent) {
return Ok(Err(to_file_error(err).into()));
}
let staging_path = transaction_parent.join(format!(".part-txn-{}", Uuid::new_v4()));
std::fs::create_dir(&staging_path).map_err(to_file_error)?;
if let Err(err) = std::fs::create_dir(&staging_path) {
return Ok(Err(to_file_error(err).into()));
}
let prepare_result = (|| -> std::io::Result<()> {
snapshot_part_transaction_file(
@@ -13325,13 +13443,13 @@ impl DiskAPI for LocalDisk {
if let Err(err) = prepare_result {
let _ = remove_dir_all_if_exists(&staging_path);
return Err(to_file_error(err).into());
return Ok(Err(to_file_error(err).into()));
}
Ok(())
Ok(Ok(()))
})
.await
.map_err(DiskError::from)?
.map_err(to_file_error)?
}
#[tracing::instrument(level = "trace", skip_all)]
@@ -13346,47 +13464,57 @@ impl DiskAPI for LocalDisk {
check_path_length(candidate.to_string_lossy().as_ref())?;
}
let durability = effective_durability(volume);
let sync = durability.syncs_commit_metadata();
let lease = os::acquire_namespace_mutation_lease_with_owner(&transaction_path, source_owner).await;
tokio::task::spawn_blocking(move || {
let _owner = source_owner;
os::run_blocking_namespace_operation_with_sync(lease, sync, move || -> std::io::Result<Result<()>> {
match std::fs::symlink_metadata(&transaction_path) {
Ok(metadata) if metadata.is_dir() => {}
Ok(_) => return Err(DiskError::FileCorrupt),
Err(err) if err.kind() == ErrorKind::NotFound => return Ok(()),
Err(err) => return Err(to_file_error(err).into()),
Ok(_) => return Ok(Err(DiskError::FileCorrupt)),
Err(err) if err.kind() == ErrorKind::NotFound => return Ok(Ok(())),
Err(err) => return Ok(Err(to_file_error(err).into())),
}
if let Some(admission) = admission {
admission.validate().map_err(to_file_error)?;
if let Some(admission) = admission
&& let Err(err) = admission.validate()
{
return Ok(Err(to_file_error(err).into()));
}
if action == PartTransactionAction::Rollback {
std::fs::write(transaction_path.join(PART_TRANSACTION_ROLLBACK), []).map_err(to_file_error)?;
if durability.syncs_commit_metadata() {
os::fsync_dir_std(&transaction_path).map_err(to_file_error)?;
if let Err(err) = std::fs::write(transaction_path.join(PART_TRANSACTION_ROLLBACK), []) {
return Ok(Err(to_file_error(err).into()));
}
restore_part_transaction_file(
if durability.syncs_commit_metadata()
&& let Err(err) = os::fsync_dir_std(&transaction_path)
{
return Ok(Err(to_file_error(err).into()));
}
if let Err(err) = restore_part_transaction_file(
&current_data_path,
&transaction_path.join(PART_TRANSACTION_OLD_DATA),
&transaction_path.join(PART_TRANSACTION_OLD_DATA_ABSENT),
&transaction_path.join("restore.data"),
)
.map_err(to_file_error)?;
restore_part_transaction_file(
) {
return Ok(Err(to_file_error(err).into()));
}
if let Err(err) = restore_part_transaction_file(
&current_meta_path,
&transaction_path.join(PART_TRANSACTION_OLD_META),
&transaction_path.join(PART_TRANSACTION_OLD_META_ABSENT),
&transaction_path.join("restore.meta"),
)
.map_err(to_file_error)?;
) {
return Ok(Err(to_file_error(err).into()));
}
if durability.syncs_commit_metadata()
&& let Some(parent) = current_data_path.parent()
&& let Err(err) = os::fsync_dir_std(parent)
{
os::fsync_dir_std(parent).map_err(to_file_error)?;
return Ok(Err(to_file_error(err).into()));
}
}
let Some(parent) = transaction_path.parent() else {
return Err(DiskError::InvalidPath);
return Ok(Err(DiskError::InvalidPath));
};
// Delete only the generation made obsolete by this settled switch.
// Check the published metadata before removing either proof, so a
@@ -13413,25 +13541,27 @@ impl DiskAPI for LocalDisk {
{
let obsolete_path = parent.join(integrity.file_name());
match std::fs::remove_file(&obsolete_path) {
Ok(()) => {
if durability.syncs_commit_metadata() {
os::fsync_dir_std(parent).map_err(to_file_error)?;
}
}
Ok(()) => {}
Err(error) if error.kind() == ErrorKind::NotFound => {}
Err(error) => return Err(to_file_error(error).into()),
Err(error) => return Ok(Err(to_file_error(error).into())),
}
}
let cleanup_path = parent.join(format!(".part-txn-settled-{}", Uuid::new_v4()));
std::fs::rename(&transaction_path, &cleanup_path).map_err(to_file_error)?;
if durability.syncs_commit_metadata() {
os::fsync_dir_std(parent).map_err(to_file_error)?;
if let Err(err) = std::fs::rename(&transaction_path, &cleanup_path) {
return Ok(Err(to_file_error(err).into()));
}
remove_dir_all_if_exists(&cleanup_path).map_err(to_file_error)?;
Ok(())
if durability.syncs_commit_metadata()
&& let Err(err) = os::fsync_dir_std(parent)
{
return Ok(Err(to_file_error(err).into()));
}
if let Err(err) = remove_dir_all_if_exists(&cleanup_path) {
return Ok(Err(to_file_error(err).into()));
}
Ok(Ok(()))
})
.await
.map_err(DiskError::from)??;
.map_err(to_file_error)??;
self.io_backend.invalidate_cached_fd(volume, path).await;
Ok(())
@@ -13776,7 +13906,7 @@ impl DiskAPI for LocalDisk {
}
pool_meta_native::reject_raw_path(&self.root, volume, path)?;
crate::hp_guard!("LocalDisk::create_file");
if !origvolume.is_empty() {
if !origvolume.is_empty() && !(volume == RUSTFS_META_TMP_BUCKET && path.ends_with("/part.1")) {
let origvolume_dir = self.io_get_bucket_path(origvolume)?;
if !skip_access_checks(origvolume) {
cached_access(origvolume_dir)
@@ -15850,7 +15980,7 @@ mod test {
std::fs::write(&xl_path, old_metadata).expect("old metadata should be written");
set_local_inline_rollback_hardlink_failure(&xl_path);
let rollback_path = create_local_inline_rollback_backup(&xl_path, &staging_path, old_metadata)
let rollback_path = create_local_inline_rollback_backup(&xl_path, &staging_path, old_metadata, false)
.expect("copy fallback should create rollback backup");
let backup = std::fs::read(&rollback_path).expect("fallback backup should be readable");
+98 -72
View File
@@ -175,17 +175,28 @@ pub(super) fn create_local_inline_rollback_backup(
dst_file_path: &Path,
staging_file_path: &Path,
old_metadata: &[u8],
sync: bool,
) -> std::io::Result<PathBuf> {
let Some(staging_parent) = staging_file_path.parent() else {
return Err(std::io::Error::new(ErrorKind::InvalidInput, "missing staging metadata parent"));
};
let backup_path = staging_parent.join(STORAGE_FORMAT_FILE_BACKUP);
remove_file_if_exists(&backup_path)?;
if (should_fail_local_inline_rollback_hardlink(dst_file_path) || std::fs::hard_link(dst_file_path, &backup_path).is_err())
&& let Err(err) = std::fs::write(&backup_path, old_metadata)
{
let _ = remove_file_if_exists(&backup_path);
return Err(err);
if should_fail_local_inline_rollback_hardlink(dst_file_path) || std::fs::hard_link(dst_file_path, &backup_path).is_err() {
let write_res = (|| -> std::io::Result<()> {
let mut file = std::fs::File::create(&backup_path)?;
std::io::Write::write_all(&mut file, old_metadata)?;
if sync && !old_metadata.is_empty() {
file.sync_data()?;
}
Ok(())
})();
if let Err(err) = write_res {
let _ = remove_file_if_exists(&backup_path);
return Err(err);
}
} else if sync && !old_metadata.is_empty() {
std::fs::File::open(&backup_path)?.sync_data()?;
}
Ok(backup_path)
}
@@ -785,9 +796,10 @@ impl LocalDisk {
&dst_volume_dir_owned,
&publication_root_owned,
)?;
// An existing xl.meta can share versions or require rollback
// restoration when a post-rename directory sync fails.
if is_cancelled() || (sync_commit_metadata && has_dst_buf.is_some()) {
// An existing xl.meta that replaces an existing version slot
// requires a rollback directory; appending a new version keeps
// an inline rollback backup for post-rename directory sync errors.
if is_cancelled() {
return Ok(FusedCommitAttempt::Fallback(guard, has_dst_buf, fi_moved));
}
let mut xlmeta = FileMeta::new();
@@ -834,6 +846,13 @@ impl LocalDisk {
os::create_prepared_rename_source_with_commit_guard(&src_file_path_owned, &dst_file_path_owned, &guard)?;
prepared_metadata_source.write_all(&new_dst_buf, sync_commit_metadata)?;
let mut staged_rollback_path = None;
if sync_commit_metadata && let Some(ref old_metadata) = has_dst_buf {
let backup_path =
create_local_inline_rollback_backup(&dst_file_path_owned, &src_file_path_owned, old_metadata, true)?;
staged_rollback_path = Some(StagedRollbackBackup(backup_path));
}
if no_inline {
if sync_data_shards && let Some((src_data_path, _)) = has_data_dir_path_owned.as_ref() {
let sync_result = if super::take_presynced_tmp_data_dir(src_data_path) {
@@ -846,12 +865,18 @@ impl LocalDisk {
if let Err(err) = sync_result
&& err.kind() != ErrorKind::NotFound
{
if let Some(backup_path) = staged_rollback_path.as_ref().map(StagedRollbackBackup::path) {
let _ = remove_file_if_exists(backup_path);
}
return Err(err);
}
}
if let Some((src_data_path, dst_data_path)) = has_data_dir_path_owned.as_ref()
&& let Err(err) = os::rename_all_with_commit_guard_on_thread(src_data_path, dst_data_path, &guard)
{
if let Some(backup_path) = staged_rollback_path.as_ref().map(StagedRollbackBackup::path) {
let _ = remove_file_if_exists(backup_path);
}
return Ok(FusedCommitAttempt::CommitErr(err));
}
#[cfg(test)]
@@ -868,6 +893,9 @@ impl LocalDisk {
&publication_root_owned,
&guard,
) {
if let Some(backup_path) = staged_rollback_path.as_ref().map(StagedRollbackBackup::path) {
let _ = remove_file_if_exists(backup_path);
}
if no_inline && let Some((src_data_path, dst_data_path)) = has_data_dir_path_owned.as_ref() {
let _ = std::fs::rename(dst_data_path, src_data_path);
}
@@ -875,13 +903,22 @@ impl LocalDisk {
}
if sync_commit_metadata
&& let Err(err) =
fsync_inline_commit_std(&dst_file_path_owned, &dst_volume_dir_owned, destination_was_absent, None, &guard)
&& let Err(err) = fsync_inline_commit_std(
&dst_file_path_owned,
&dst_volume_dir_owned,
destination_was_absent,
staged_rollback_path.as_ref(),
&guard,
)
{
if no_inline && let Some((src_data_path, dst_data_path)) = has_data_dir_path_owned.as_ref() {
let _ = std::fs::rename(dst_data_path, src_data_path);
}
return Err(err);
return Ok(FusedCommitAttempt::CommitErr(DiskError::from(err)));
}
if let Some(backup_path) = staged_rollback_path.as_ref().map(StagedRollbackBackup::path) {
let _ = remove_file_if_exists(backup_path);
}
if let Some(parent) = src_file_path_owned.parent() {
@@ -1053,52 +1090,6 @@ impl LocalDisk {
// after the commit rename, and a crash before the rename means the
// PUT was never acknowledged. A metadata commit: relaxed tiers
// leave it to the page cache.
if let Some(publication) = multipart_publication.as_ref() {
pin_multipart_future(|| publication.prepare_metadata(&new_dst_buf, has_dst_buf.as_deref())).await?;
}
let tmp_meta_sync = if durability.syncs_commit_metadata() || multipart_publication.is_some() {
SyncMode::FileOnly
} else {
SyncMode::None
};
// The tmp xl.meta write and the shard-file fdatasync are independent
// (disjoint paths) and both only need to be durable before the commit
// renames below, so run them concurrently to drop a blocking
// round-trip from the PUT commit critical path (rustfs/backlog#922
// step 2). The "contents durable -> rename -> dst dir fsync" ordering
// is unchanged — both futures complete before any rename — which the
// rename_data crash-consistency harness (backlog#935) exercises.
//
// Shard durability: once rename_data succeeds the write is
// acknowledged, so data must not live only in the page cache.
// Multipart parts were already synced during rename_part, so their
// fdatasync here is a cheap no-op. A missing source dir is left for the
// rename below to report through the existing rollback path. Payload
// durability is kept by both strict and relaxed.
let tmp_meta_write = {
let src_file_path = multipart_publication
.as_ref()
.map_or_else(|| commit.source_metadata.to_path_buf(), |publication| publication.prepared_path.clone());
let dst_file_path = dst_file_path.clone();
let owned_directory_guard = commit.directory_guard.clone();
let owned_mutation_lease = commit.mutation_lease.clone();
async move {
os::run_blocking_namespace_operation(owned_mutation_lease, move || {
#[cfg(test)]
run_owned_file_before_open(&src_file_path);
let mut prepared_metadata_source = os::create_prepared_rename_source_with_commit_guard(
&src_file_path,
&dst_file_path,
&owned_directory_guard,
)?;
prepared_metadata_source.write_all(&new_dst_buf, tmp_meta_sync != SyncMode::None)?;
Ok(prepared_metadata_source)
})
.await
.map_err(to_file_error)
.map_err(DiskError::from)
}
};
let shard_sync = async {
if durability.syncs_data_shards()
&& let Some((src_data_path, _)) = has_data_dir_path.as_ref()
@@ -1109,15 +1100,55 @@ impl LocalDisk {
}
Ok(())
};
let (tmp_meta_res, shard_sync_res) = tokio::join!(tmp_meta_write, shard_sync);
// Surface a tmp-meta failure first (its prior serial position), then a
// shard-sync failure; either aborts before any rename, exactly as the
// sequential version did.
let prepared_metadata_source = tmp_meta_res?;
shard_sync_res?;
let prepared_metadata_source = match multipart_publication.as_ref() {
Some(publication) => pin_multipart_future(|| publication.install_head(prepared_metadata_source)).await?,
None => prepared_metadata_source,
let prepared_metadata_source = if let Some(publication) = multipart_publication.as_ref() {
let dst_file_path_ref = &dst_file_path;
let dir_guard_ref = &commit.directory_guard;
let lease_clone = commit.mutation_lease.clone();
let prepare_fut = pin_multipart_future(|| {
publication.prepare_and_install_head(
&new_dst_buf,
has_dst_buf.as_deref(),
dst_file_path_ref,
dir_guard_ref,
lease_clone,
)
});
let (tmp_meta_res, shard_sync_res) = tokio::join!(prepare_fut, shard_sync);
let prepared_metadata_source = tmp_meta_res?;
shard_sync_res?;
prepared_metadata_source
} else {
let tmp_meta_sync = if durability.syncs_commit_metadata() {
SyncMode::FileOnly
} else {
SyncMode::None
};
let tmp_meta_write = {
let src_file_path = commit.source_metadata.to_path_buf();
let dst_file_path = dst_file_path.clone();
let owned_directory_guard = commit.directory_guard.clone();
let owned_mutation_lease = commit.mutation_lease.clone();
async move {
os::run_blocking_namespace_operation(owned_mutation_lease, move || {
#[cfg(test)]
run_owned_file_before_open(&src_file_path);
let mut prepared_metadata_source = os::create_prepared_rename_source_with_commit_guard(
&src_file_path,
&dst_file_path,
&owned_directory_guard,
)?;
prepared_metadata_source.write_all(&new_dst_buf, tmp_meta_sync != SyncMode::None)?;
Ok(prepared_metadata_source)
})
.await
.map_err(to_file_error)
.map_err(DiskError::from)
}
};
let (tmp_meta_res, shard_sync_res) = tokio::join!(tmp_meta_write, shard_sync);
let prepared_metadata_source = tmp_meta_res?;
shard_sync_res?;
prepared_metadata_source
};
commit.directory_guard = remove_dst_base_before_commit(
dst_path,
@@ -1663,13 +1694,8 @@ impl LocalDisk {
#[cfg(not(windows))]
let backup_path = {
let _backup = os::StageTimer::new(native_timing, "pool_meta_native_backup_create");
create_local_inline_rollback_backup(&dst, &src, old_metadata)?
create_local_inline_rollback_backup(&dst, &src, old_metadata, sync)?
};
#[cfg(not(windows))]
if sync {
let _backup_sync = os::StageTimer::new(native_timing, "pool_meta_native_backup_file_sync");
std::fs::File::open(&backup_path)?.sync_data()?;
}
staged_rollback_path = Some(StagedRollbackBackup(backup_path));
}
+263 -36
View File
@@ -2132,12 +2132,13 @@ impl LocalDisk {
Err(err) if err.kind() == ErrorKind::NotFound => return Ok(Ok(None)),
Err(err) => return Err(err),
};
let (old_dir, journal_raw) = match (|| -> Result<Option<(Option<Uuid>, Vec<u8>)>> {
type PreparedDeleteTuple = (Option<Uuid>, Option<Vec<u8>>, Vec<u8>);
let (old_dir, target, journal_raw) = match (|| -> Result<Option<PreparedDeleteTuple>> {
let mut meta = FileMeta::load(&buf)?;
let deleted_versions: Vec<_> = meta
let mut deleted_headers: Vec<_> = meta
.versions
.iter()
.map(|v| v.header.version_id.unwrap_or_default())
.map(|v| (v.header.version_id, std::mem::discriminant(&v.header.version_type)))
.collect();
let original_versions: Vec<_> = meta
.versions
@@ -2156,9 +2157,14 @@ impl LocalDisk {
return Ok(None);
}
}
if !meta.versions.is_empty() {
return Ok(None);
}
let surviving: HashSet<_> = meta
.versions
.iter()
.map(|v| (v.header.version_id, std::mem::discriminant(&v.header.version_type)))
.collect();
deleted_headers.retain(|header| !surviving.contains(header));
let deleted_versions = deleted_headers.into_iter().map(|(id, _)| id.unwrap_or_default()).collect();
let target = (!meta.versions.is_empty()).then(|| meta.marshal_msg()).transpose()?;
let mut predecessors = BTreeSet::new();
if let Ok(entries) = std::fs::read_dir(&file_path) {
for (visited, entry) in entries.enumerate() {
@@ -2216,16 +2222,16 @@ impl LocalDisk {
transaction,
bucket_incarnation: incarnation,
original_digest: Some(digest(&buf)),
target_digest: None,
target_digest: target.as_deref().map(digest),
data_dirs: old_dir.into_iter().collect(),
deleted_versions,
requests: vec![sparse],
original_versions,
predecessors: predecessors.into_iter().collect(),
};
Ok(Some((old_dir, journal_bytes(&journal)?)))
Ok(Some((old_dir, target, journal_bytes(&journal)?)))
})() {
Ok(Some(pair)) => pair,
Ok(Some(tuple)) => tuple,
Ok(None) => return Ok(Ok(None)),
Err(err) => return Ok(Err(err)),
};
@@ -2233,10 +2239,18 @@ impl LocalDisk {
let staged = tempfile::Builder::new()
.prefix("delete-preparation-")
.tempdir_in(&staging_root)?;
for (name, bytes) in [
(STORAGE_FORMAT_FILE_BACKUP, buf.as_slice()),
(PREPARED, journal_raw.as_slice()),
] {
let backup_path = staged.path().join(STORAGE_FORMAT_FILE_BACKUP);
if std::fs::hard_link(&xl_path, &backup_path).is_err() {
let mut file = std::fs::File::create(&backup_path)?;
std::io::Write::write_all(&mut file, &buf)?;
if sync && !buf.is_empty() {
file.sync_data()?;
}
}
for (name, bytes) in [(TARGET, target.as_deref()), (PREPARED, Some(journal_raw.as_slice()))] {
let Some(bytes) = bytes else {
continue;
};
let mut file = std::fs::File::create(staged.path().join(name))?;
std::io::Write::write_all(&mut file, bytes)?;
if sync && !bytes.is_empty() {
@@ -2261,23 +2275,57 @@ impl LocalDisk {
std::fs::create_dir_all(parent)?;
}
std::fs::rename(staged.path(), &destination)?;
if sync {
os::fsync_dir_std(&file_path)?;
if sync && let Err(err) = os::fsync_dir_std(&file_path) {
let _ = std::fs::remove_dir_all(&destination);
return Err(err);
}
let mut reserved_data_path = None;
if let Some(uuid) = old_dir {
let old_path = file_path.join(uuid.to_string());
if old_path.is_dir() {
let reserved_marker = old_path.join(format!("{RESERVED_DELETE_DATA_DIR_MARKER_PREFIX}{transaction}"));
let marker = std::fs::File::create(&reserved_marker)?;
if sync {
marker.sync_all()?;
os::fsync_dir_std(&old_path)?;
let reserve_res = (|| -> std::io::Result<()> {
let marker = std::fs::File::create(&reserved_marker)?;
if sync {
marker.sync_all()?;
os::fsync_dir_std(&old_path)?;
}
Ok(())
})();
if let Err(err) = reserve_res {
let _ = std::fs::remove_file(&reserved_marker);
let _ = std::fs::remove_dir_all(&destination);
return Err(err);
}
reserved_data_path = Some((old_path, reserved_marker));
}
}
if let Err(err) = std::fs::remove_file(&xl_path)
if let Some(ref target_bytes) = target {
let mut staged_xl = match tempfile::NamedTempFile::new_in(&staging_root) {
Ok(staged_xl) => staged_xl,
Err(err) => {
if let Some((_, reserved_marker)) = &reserved_data_path {
let _ = std::fs::remove_file(reserved_marker);
}
let _ = std::fs::remove_dir_all(&destination);
return Err(err);
}
};
let write_res = (|| -> std::io::Result<()> {
std::io::Write::write_all(staged_xl.as_file_mut(), target_bytes)?;
if sync && !target_bytes.is_empty() {
staged_xl.as_file().sync_data()?;
}
std::fs::rename(staged_xl.path(), &xl_path)
})();
if let Err(err) = write_res {
if let Some((_, reserved_marker)) = &reserved_data_path {
let _ = std::fs::remove_file(reserved_marker);
}
let _ = std::fs::remove_dir_all(&destination);
return Err(err);
}
} else if let Err(err) = std::fs::remove_file(&xl_path)
&& err.kind() != ErrorKind::NotFound
{
if let Some((_, reserved_marker)) = &reserved_data_path {
@@ -2360,12 +2408,18 @@ impl LocalDisk {
{
return Ok(Err(DiskError::FileCorrupt));
}
let current_exists = match regular_file_exists(&root.join(STORAGE_FORMAT_FILE)) {
Ok(exists) => exists,
Err(e) => return Ok(Err(e)),
let target_satisfied = match journal.target_digest {
None => match regular_file_exists(&root.join(STORAGE_FORMAT_FILE)) {
Ok(exists) => !exists,
Err(e) => return Ok(Err(e)),
},
Some(expected_target) => match regular_bytes(&root.join(STORAGE_FORMAT_FILE)) {
Ok(Some(bytes)) => digest(&bytes) == expected_target,
Ok(None) => false,
Err(e) => return Ok(Err(e)),
},
};
let unversioned_target_absent = journal.target_digest.is_none() && !current_exists;
if !unversioned_target_absent {
if !target_satisfied {
return Ok(Ok(None));
}
let mut sole_transaction = journal.predecessors.is_empty();
@@ -2382,8 +2436,15 @@ impl LocalDisk {
&& id != transaction
&& !journal.data_dirs.contains(&id)
{
sole_transaction = false;
break;
let dir_path = entry.path();
let is_surviving_data_dir = journal.original_versions.iter().any(|v| v.data_dir == Some(id))
&& !dir_path.join(PREPARED).exists()
&& !dir_path.join(DELETE_QUORUM_COMMITTED_FILE).exists()
&& !dir_path.join(STORAGE_FORMAT_FILE_BACKUP).exists();
if !is_surviving_data_dir {
sole_transaction = false;
break;
}
}
}
}
@@ -2406,7 +2467,7 @@ impl LocalDisk {
if sync {
os::fsync_dir_std(&rollback)?;
}
Ok(Ok(Some((journal, sole_transaction, unversioned_target_absent))))
Ok(Ok(Some((journal, sole_transaction, target_satisfied))))
},
)
.await
@@ -2458,12 +2519,15 @@ impl LocalDisk {
}
let bucket_path = self.io_get_bucket_path(volume)?;
let staging_root = self.io_get_bucket_path(RUSTFS_META_TMP_BUCKET)?;
let root = self.io_get_object_path(volume, object)?;
let rollback = root.join(transaction.to_string());
let receipt_path = rollback.join(DELETE_QUORUM_COMMITTED_FILE);
let sync = effective_durability(volume).syncs_commit_metadata();
let lease = os::acquire_namespace_mutation_lease_with_owner(&receipt_path, owner).await;
let data_dirs = journal.data_dirs.clone();
let has_target = journal.target_digest.is_some();
let journal_for_receipt = has_target.then(|| journal.clone());
let lease = os::acquire_namespace_mutation_lease_with_owner(&receipt_path, owner).await;
let prune_paths: Vec<_> = rollback
.ancestors()
.take_while(|dir| *dir != bucket_path.as_path() && dir.starts_with(&bucket_path))
@@ -2522,13 +2586,25 @@ impl LocalDisk {
if !data_dirs.is_empty() && sync && root.is_dir() {
os::fsync_dir_std(&root)?;
}
for name in [
STORAGE_FORMAT_FILE_BACKUP,
DELETE_MARKER_ROLLBACK_FILE,
TARGET,
PREPARED,
FINISHED,
] {
for name in [STORAGE_FORMAT_FILE_BACKUP, DELETE_MARKER_ROLLBACK_FILE] {
ignore_not_found(std::fs::remove_file(rollback.join(name)))?;
}
if let Some(journal) = journal_for_receipt {
let finished_receipt = CommitReceipt { journal, finished: true };
let receipt_bytes = serde_json::to_vec(&finished_receipt).map_err(std::io::Error::other)?;
std::fs::create_dir_all(&staging_root)?;
let mut staged_receipt = tempfile::NamedTempFile::new_in(&staging_root)?;
std::io::Write::write_all(staged_receipt.as_file_mut(), &receipt_bytes)?;
if sync && !receipt_bytes.is_empty() {
staged_receipt.as_file().sync_data()?;
}
std::fs::rename(staged_receipt.path(), &receipt_path)?;
let _ = std::fs::File::create(rollback.join(FINISHED))?;
if sync {
os::fsync_dir_std(&rollback)?;
}
}
for name in [TARGET, PREPARED, FINISHED] {
ignore_not_found(std::fs::remove_file(rollback.join(name)))?;
}
if sync {
@@ -6282,6 +6358,157 @@ mod tests {
b"outside bytes"
);
}
#[tokio::test]
async fn delete_recovery_fused_versioned_delete_marker_and_publish_fast_path() {
let (_temp, disk, info, _) = fixture().await;
let root = disk.io_get_object_path("recovery", "object").expect("object root");
let transaction = Uuid::new_v4();
let mut marker = FileInfo {
name: "object".to_owned(),
version_id: Some(Uuid::new_v4()),
deleted: true,
mark_deleted: true,
mod_time: info.mod_time.map(|t| t + time::Duration::seconds(1)),
..Default::default()
};
let opts = DeleteOptions {
old_data_dir: Some(transaction),
recoverable_delete: true,
..Default::default()
};
let fused = disk
.try_fused_recoverable_delete_version("recovery", "object", &marker, &opts, transaction, None)
.await
.expect("fused delete marker preparation must succeed");
assert_eq!(fused, Some(()), "versioned delete marker must use fused path");
assert!(root.join(transaction.to_string()).join(TARGET).is_file());
assert!(root.join(STORAGE_FORMAT_FILE).is_file());
let published = disk
.try_publish_prepared_delete_local("recovery", "object", transaction, None)
.await
.expect("local publish must succeed")
.expect("versioned target must satisfy local publish");
assert!(published.1, "sole transaction must be true");
assert!(published.2, "target_satisfied must be true");
let finalized = disk
.try_finish_and_finalize_prepared_delete_local("recovery", "object", transaction, &published.0, None)
.await
.expect("local finalize must succeed");
assert!(finalized);
assert!(!root.join(transaction.to_string()).exists());
let final_meta = FileMeta::load(&fs::read(root.join(STORAGE_FORMAT_FILE)).await.expect("read final xl.meta"))
.expect("decode final xl.meta");
assert_eq!(final_meta.versions.len(), 2);
assert_eq!(final_meta.versions[0].header.version_id, marker.version_id);
marker.deleted = false;
}
#[tokio::test]
async fn delete_recovery_fused_versioned_delete_with_surviving_version_and_interrupted_target_cleanup() {
let (_temp, disk, mut v1, _) = fixture().await;
let root = disk.io_get_object_path("recovery", "object").expect("object path");
// Configure v1 and v2 with distinct physical data directories so deleting v2
// exercises both `old_dir` reservation and `is_surviving_data_dir`.
v1.data = None;
rustfs_utils::http::metadata_compat::remove_str(
&mut v1.metadata,
rustfs_utils::http::metadata_compat::SUFFIX_INLINE_DATA,
);
let v1_dir = Uuid::new_v4();
v1.data_dir = Some(v1_dir);
fs::create_dir_all(root.join(v1_dir.to_string()))
.await
.expect("create v1 dir");
fs::write(root.join(v1_dir.to_string()).join("part.1"), b"v1-data")
.await
.expect("write v1 part");
let mut v2 = v1.clone();
v2.version_id = Some(Uuid::new_v4());
let v2_dir = Uuid::new_v4();
v2.data_dir = Some(v2_dir);
fs::create_dir_all(root.join(v2_dir.to_string()))
.await
.expect("create v2 dir");
fs::write(root.join(v2_dir.to_string()).join("part.1"), b"v2-data")
.await
.expect("write v2 part");
let mut meta = FileMeta::new();
meta.add_version(v1.clone()).expect("add v1");
meta.add_version(v2.clone()).expect("add v2");
fs::write(root.join(STORAGE_FORMAT_FILE), meta.marshal_msg().expect("marshal meta"))
.await
.expect("write xl.meta");
let transaction = Uuid::new_v4();
let opts = DeleteOptions {
recoverable_delete: true,
..Default::default()
};
let fused = disk
.try_fused_recoverable_delete_version("recovery", "object", &v2, &opts, transaction, None)
.await
.expect("fused versioned delete preparation must succeed");
assert_eq!(fused, Some(()));
assert!(
root.join(v2_dir.to_string())
.join(format!("{RECOVERY_DELETE_DATA_DIR_MARKER_PREFIX}{transaction}"))
.is_file(),
"deleted version data_dir must have recovery marker"
);
let (journal, sole_transaction, target_satisfied) = disk
.try_publish_prepared_delete_local("recovery", "object", transaction, None)
.await
.expect("local publish must succeed")
.expect("versioned target must satisfy local publish");
assert!(sole_transaction, "surviving v1 data_dir must not clear sole_transaction");
assert!(target_satisfied);
assert!(journal.target_digest.is_some());
// Simulate interruption during finish-before-finalize right after TARGET is removed
// (after finished receipt + FINISHED marker were persisted, before receipt_path is removed).
let rollback = root.join(transaction.to_string());
let finished_receipt = CommitReceipt {
journal: journal.clone(),
finished: true,
};
fs::write(
rollback.join(DELETE_QUORUM_COMMITTED_FILE),
serde_json::to_vec(&finished_receipt).expect("serialize finished receipt"),
)
.await
.expect("write finished receipt");
fs::write(rollback.join(FINISHED), []).await.expect("write FINISHED");
fs::remove_file(rollback.join(STORAGE_FORMAT_FILE_BACKUP))
.await
.expect("remove backup");
fs::remove_file(rollback.join(TARGET)).await.expect("remove TARGET");
let inspected = disk
.inspect_delete_transaction("recovery", "object", transaction)
.await
.expect("interrupted post-TARGET state must remain inspectable without FileCorrupt");
assert_eq!(inspected.state, DeleteRecoveryState::CommitFinished);
// Complete cleanup via try_finish_and_finalize_prepared_delete_local.
let finalized = disk
.try_finish_and_finalize_prepared_delete_local("recovery", "object", transaction, &journal, None)
.await
.expect("local finalize must succeed after interruption");
assert!(finalized);
assert!(!rollback.exists(), "rollback directory must be pruned");
assert!(!root.join(v2_dir.to_string()).exists(), "deleted v2 data_dir must be removed");
assert!(
root.join(v1_dir.to_string()).join("part.1").is_file(),
"surviving v1 data_dir must remain intact"
);
}
}
#[cfg(test)]
@@ -1323,41 +1323,6 @@ impl LocalDisk {
}
impl SourceStore {
/// Only native source validation may create provenance. Ordinary TMP/heal
/// publication can retain a valid old local anchor, never issue/rebind one.
pub async fn write_retirement_anchor(
&self,
state: &SourceState,
publication: &PublicationIdentity,
directory: &Path,
lease: Arc<os::NamespaceMutationLease>,
) -> Result<()> {
if state.lineage != publication.lineage
|| state.generation != publication.source_generation
|| state.disk_id != self.disk_id
|| state.physical_slot != self.physical_slot
{
return Err(DiskError::ConditionalDeleteConflict);
}
let record = NativeRetirementAnchor {
protocol_version: MULTIPART_SOURCE_PROTOCOL_VERSION,
disk_id: self.disk_id,
physical_slot: self.physical_slot,
publication: publication.clone(),
source_original_sha256: state.original_sha256,
source_validation_receipt_id: state.backup_owner,
};
let bytes = encode(&record)?;
let directory = directory.to_path_buf();
let base = self.base.clone();
let root = self.publication_root.clone();
os::run_blocking_namespace_operation(lease, move || {
let _parent = os::mkdir_all_below_existing_base_std(&directory, &base, &root)?;
os::replace_multipart_source_metadata(&directory.join(RETIREMENT_ANCHOR), &bytes, &base, &root)
})
.await
.map_err(DiskError::from)
}
/// This caller already owns SourceRW and destination metadata ownership.
/// An Aborting survivor can confirm a historical target only with its
/// independent, same-disk publication anchor. Closure never creates one.
@@ -1569,7 +1534,31 @@ impl LocalDisk {
};
// Copy the small source FI to an independent immutable inode. Its
// canonical inode is never truncated to build the destination proxy.
store.write_owned_backup_and_persist(owner, false, original, &state).await?;
let directory = staging.private_directory(publication.final_data_dir);
let anchor_record = NativeRetirementAnchor {
protocol_version: MULTIPART_SOURCE_PROTOCOL_VERSION,
disk_id: store.disk_id,
physical_slot: store.physical_slot,
publication: publication.clone(),
source_original_sha256: state.original_sha256,
source_validation_receipt_id: state.backup_owner,
};
let anchor_bytes = encode(&anchor_record)?;
state.validate(store.disk_id, &store.path)?;
let state_raw = encode(&state)?;
let source_dir = store.source.clone();
let base_dir = store.base.clone();
let pub_root = store.publication_root.clone();
let terminal = terminal_record_path(&source_dir, &state);
let original_bytes = original.to_vec();
os::run_blocking_namespace_operation_with_sync(staging.lease.clone(), true, move || {
SourceStore::write_owned_backup_std(&source_dir, &base_dir, &pub_root, owner, false, &original_bytes)?;
SourceStore::persist_state_std(&source_dir, &base_dir, &pub_root, &state_raw, terminal.as_deref(), false)?;
let _parent = os::mkdir_all_below_existing_base_std(&directory, &base_dir, &pub_root)?;
os::replace_multipart_source_metadata(&directory.join(RETIREMENT_ANCHOR), &anchor_bytes, &base_dir, &pub_root)
})
.await
.map_err(DiskError::from)?;
let prepared_path = store.source.join(format!(".mpu-prepared-{owner}"));
let context = NativeMultipartPublication {
store,
@@ -1580,12 +1569,6 @@ impl LocalDisk {
validated_prepared: parking_lot::Mutex::new(None),
legacy,
};
let directory = staging.private_directory(context.publication.final_data_dir);
let reserved = context.state.lock().clone();
context
.store
.write_retirement_anchor(&reserved, &context.publication, &directory, staging.lease.clone())
.await?;
Ok(context)
}
}
@@ -1594,18 +1577,15 @@ impl NativeMultipartPublication {
pub fn canonical_head(&self) -> PathBuf {
self.store.source.join(STORAGE_FORMAT_FILE)
}
#[cfg(test)]
pub async fn prepare_metadata(&self, new_metadata: &[u8], old_metadata: Option<&[u8]>) -> Result<()> {
let mut state = self.state.lock().clone();
let current = self.store.load().await?;
let current_state = current.state.as_ref().ok_or(DiskError::ConditionalDeleteConflict)?;
if encode(current_state)? != encode(&state)? || !current.head_matches_state {
return Err(DiskError::ConditionalDeleteConflict);
}
// Validate the actual persisted representation once, after FileMeta's
// canonicalization. No unvalidated input can install a source proxy.
if !self.store.target_matches(&self.publication, new_metadata)? {
return Err(DiskError::ConditionalDeleteConflict);
}
let expected_state_raw = encode(&state)?;
let validated = ValidatedPreparedPublication {
publication: self.publication.clone(),
sha256: raw_sha(new_metadata),
@@ -1618,17 +1598,109 @@ impl NativeMultipartPublication {
state.prepared_sha256 = Some(validated.sha256);
state.prepared_length = Some(validated.length);
state.advance(MultipartSourcePhase::Prepared)?;
if let Some(previous) = old_metadata {
self.store
.write_owned_backup_and_persist(state.backup_owner, true, previous, &state)
.await?;
} else {
self.store.persist(&state).await?;
}
state.validate(self.store.disk_id, &self.store.path)?;
let next_state_raw = encode(&state)?;
let source = self.store.source.clone();
let base = self.store.base.clone();
let root = self.store.publication_root.clone();
let path = self.store.path.clone();
let disk_id = self.store.disk_id;
let physical_slot = self.store.physical_slot;
let disk_ids = self.store.disk_ids.clone();
let backup_owner = state.backup_owner;
let old_bytes = old_metadata.map(|b| b.to_vec());
let terminal = terminal_record_path(&source, &state);
os::run_blocking_namespace_operation_with_sync(self.store.lease.clone(), true, move || {
let current = load_source(&source, &base, &root, &path, disk_id, physical_slot, &disk_ids).map_err(IoError::other)?;
let current_state = current
.state
.as_ref()
.ok_or_else(|| IoError::other(DiskError::ConditionalDeleteConflict))?;
let current_raw = encode(current_state).map_err(IoError::other)?;
if current_raw != expected_state_raw || !current.head_matches_state {
return Err(IoError::other(DiskError::ConditionalDeleteConflict));
}
if let Some(previous) = old_bytes.as_deref() {
SourceStore::write_owned_backup_std(&source, &base, &root, backup_owner, true, previous)?;
}
SourceStore::persist_state_std(&source, &base, &root, &next_state_raw, terminal.as_deref(), false)
})
.await
.map_err(DiskError::from)?;
*self.state.lock() = state;
*self.validated_prepared.lock() = Some(validated);
Ok(())
}
pub async fn prepare_and_install_head(
&self,
new_metadata: &[u8],
old_metadata: Option<&[u8]>,
dst_file_path: &Path,
directory_guard: &os::RenameCommitGuard,
mutation_lease: Arc<os::NamespaceMutationLease>,
) -> Result<os::PreparedRenameSource> {
let mut state = self.state.lock().clone();
if !self.store.target_matches(&self.publication, new_metadata)? {
return Err(DiskError::ConditionalDeleteConflict);
}
let expected_state_raw = encode(&state)?;
let validated = ValidatedPreparedPublication {
publication: self.publication.clone(),
sha256: raw_sha(new_metadata),
length: new_metadata.len() as u64,
};
if let Some(previous) = old_metadata {
state.previous_target_sha256 = Some(raw_sha(previous));
state.previous_target_length = Some(previous.len() as u64);
}
state.prepared_sha256 = Some(validated.sha256);
state.prepared_length = Some(validated.length);
state.advance(MultipartSourcePhase::Prepared)?;
state.validate(self.store.disk_id, &self.store.path)?;
let next_state_raw = encode(&state)?;
let source = self.store.source.clone();
let base = self.store.base.clone();
let root = self.store.publication_root.clone();
let path = self.store.path.clone();
let disk_id = self.store.disk_id;
let physical_slot = self.store.physical_slot;
let disk_ids = self.store.disk_ids.clone();
let backup_owner = state.backup_owner;
let old_bytes = old_metadata.map(|b| b.to_vec());
let new_bytes = new_metadata.to_vec();
let terminal = terminal_record_path(&source, &state);
let prepared_path = self.prepared_path.clone();
let canonical = self.canonical_head();
let dst_file_path = dst_file_path.to_path_buf();
let owned_directory_guard = directory_guard.clone();
let prepared = os::run_blocking_namespace_operation_with_sync(mutation_lease, true, move || {
let current = load_source(&source, &base, &root, &path, disk_id, physical_slot, &disk_ids).map_err(IoError::other)?;
let current_state = current
.state
.as_ref()
.ok_or_else(|| IoError::other(DiskError::ConditionalDeleteConflict))?;
let current_raw = encode(current_state).map_err(IoError::other)?;
if current_raw != expected_state_raw || !current.head_matches_state {
return Err(IoError::other(DiskError::ConditionalDeleteConflict));
}
if let Some(previous) = old_bytes.as_deref() {
SourceStore::write_owned_backup_std(&source, &base, &root, backup_owner, true, previous)?;
}
SourceStore::persist_state_std(&source, &base, &root, &next_state_raw, terminal.as_deref(), false)?;
#[cfg(test)]
super::run_owned_file_before_open(&prepared_path);
let mut prepared_metadata_source =
os::create_prepared_rename_source_with_commit_guard(&prepared_path, &dst_file_path, &owned_directory_guard)?;
prepared_metadata_source.write_all(&new_bytes, true)?;
os::install_multipart_prepared_head(prepared_metadata_source, &canonical, &base, &root)
})
.await
.map_err(DiskError::from)?;
*self.state.lock() = state;
*self.validated_prepared.lock() = Some(validated);
Ok(prepared)
}
#[cfg(test)]
pub async fn install_head(&self, prepared: os::PreparedRenameSource) -> Result<os::PreparedRenameSource> {
{
let guard = self.validated_prepared.lock();
@@ -1701,11 +1773,7 @@ impl NativeMultipartPublication {
}
pub async fn finish(&self, target_lease: Arc<os::NamespaceMutationLease>) -> Result<MultipartPublicationResult> {
let mut state = self.state.lock().clone();
let current = self.store.load().await?;
let current_state = current.state.as_ref().ok_or(DiskError::ConditionalDeleteConflict)?;
if current.head.is_some() || encode(current_state)? != encode(&state)? {
return Err(DiskError::ConditionalDeleteConflict);
}
let expected_state_raw = encode(&state)?;
let (expected_sha256, expected_length) = {
let guard = self.validated_prepared.lock();
let validated = guard.as_ref().ok_or(DiskError::ConditionalDeleteConflict)?;
@@ -1717,16 +1785,52 @@ impl NativeMultipartPublication {
}
(validated.sha256, validated.length)
};
self.store
.read_and_sync_target_with_lease(&self.publication, target_lease, move |target| {
if target.len() as u64 != expected_length || raw_sha(target) != expected_sha256 {
return Err(DiskError::ConditionalDeleteConflict);
}
Ok(())
})
.await?;
let target_dir = self.store.target_path(&self.publication)?;
state.advance(MultipartSourcePhase::ConsumedUnsealed)?;
let snapshot = self.store.persist_with_source_sync(&state, true).await?;
state.validate(self.store.disk_id, &self.store.path)?;
let next_state_raw = encode(&state)?;
let snapshot = snapshot_for_state(self.store.disk_id, &state, &next_state_raw, None)?;
let source = self.store.source.clone();
let base = self.store.base.clone();
let root = self.store.publication_root.clone();
let path = self.store.path.clone();
let disk_id = self.store.disk_id;
let physical_slot = self.store.physical_slot;
let disk_ids = self.store.disk_ids.clone();
let terminal = terminal_record_path(&source, &state);
os::run_blocking_namespace_operation_with_sync(target_lease, true, move || {
let current = load_source(&source, &base, &root, &path, disk_id, physical_slot, &disk_ids).map_err(IoError::other)?;
let current_state = current
.state
.as_ref()
.ok_or_else(|| IoError::other(DiskError::ConditionalDeleteConflict))?;
let current_raw = encode(current_state).map_err(IoError::other)?;
if current.head.is_some() || current_raw != expected_state_raw {
return Err(IoError::other(DiskError::ConditionalDeleteConflict));
}
os::read_and_sync_multipart_target_metadata(
&target_dir,
std::ffi::OsStr::new(STORAGE_FORMAT_FILE),
&base,
&root,
|target| {
if target.len() as u64 != expected_length || raw_sha(target) != expected_sha256 {
return Err(IoError::other(DiskError::ConditionalDeleteConflict));
}
Ok(())
},
)
.map_err(|error| {
if error.kind() == std::io::ErrorKind::NotFound {
IoError::other(DiskError::ConditionalDeleteConflict)
} else {
error
}
})?;
SourceStore::persist_state_std(&source, &base, &root, &next_state_raw, terminal.as_deref(), true)
})
.await
.map_err(DiskError::from)?;
*self.state.lock() = state;
let proof = self
.store
+187 -23
View File
@@ -2752,22 +2752,114 @@ pub(crate) fn read_dir_probe(path: impl AsRef<Path>, entry_limit: usize) -> io::
})
}
#[cfg(test)]
pub(crate) fn has_non_data_dir_entry(path: impl AsRef<Path>, data_dirs: &[uuid::Uuid]) -> io::Result<bool> {
match has_non_data_dir_entry_inner(path.as_ref(), data_dirs) {
Ok(res) => Ok(res),
Err(err) if err.kind() == io::ErrorKind::NotFound => Ok(false),
Err(err) => Err(err),
}
has_non_data_dir_entry_lazy(path, || data_dirs.to_vec()).map(|(_, res)| res)
}
fn has_non_data_dir_entry_inner(path: &Path, data_dirs: &[uuid::Uuid]) -> io::Result<bool> {
let mut dir = std::fs::read_dir(path)?;
let entry_limit = data_dirs.len().saturating_add(2);
#[cfg(target_os = "linux")]
pub(crate) fn has_non_data_dir_entry_lazy_from_fd(
dir_fd: &std::fs::File,
load_data_dirs: impl FnOnce() -> Vec<uuid::Uuid>,
) -> io::Result<(Vec<uuid::Uuid>, bool)> {
use rustix::fs::{AtFlags, FileType, RawDir, statat};
let mut raw_buf = [std::mem::MaybeUninit::<u8>::uninit(); 1024];
let mut dir = RawDir::new(dir_fd, &mut raw_buf);
let mut data_dirs: Option<Vec<uuid::Uuid>> = None;
let mut load = Some(load_data_dirs);
let mut valid_entries = 0usize;
for _ in 0..READ_DIR_PROBE_RAW_LIMIT {
let Some(entry) = dir.next() else {
return Ok(false);
return Ok((data_dirs.unwrap_or_default(), false));
};
let entry = match entry {
Ok(entry) => entry,
Err(rustix::io::Errno::NOENT) => return Ok((data_dirs.unwrap_or_default(), false)),
Err(err) => return Err(io::Error::from(err)),
};
let c_name = entry.file_name();
let name_bytes = c_name.to_bytes();
if name_bytes.is_empty() || name_bytes == b"." || name_bytes == b".." {
continue;
}
let file_type = match entry.file_type() {
FileType::Unknown => match statat(dir_fd, c_name, AtFlags::SYMLINK_NOFOLLOW) {
Ok(st) => FileType::from_raw_mode(st.st_mode),
Err(rustix::io::Errno::NOENT) => continue,
Err(err) => return Err(io::Error::from(err)),
},
ft => ft,
};
if file_type == FileType::Directory {
let parsed_uuid = if name_bytes.len() == uuid::fmt::Hyphenated::LENGTH {
std::str::from_utf8(name_bytes).ok().and_then(|s| {
let parsed = uuid::Uuid::try_parse(s).ok()?;
let mut buf = [0u8; uuid::fmt::Hyphenated::LENGTH];
(parsed.hyphenated().encode_lower(&mut buf) == s).then_some(parsed)
})
} else {
None
};
let Some(parsed) = parsed_uuid else {
let dirs = data_dirs
.take()
.unwrap_or_else(|| (load.take().expect("load_data_dirs called once"))());
return Ok((dirs, true));
};
let dirs = data_dirs.get_or_insert_with(|| (load.take().expect("load_data_dirs called once"))());
if dirs.binary_search(&parsed).is_err() {
return Ok((std::mem::take(dirs), true));
}
valid_entries += 1;
} else if file_type == FileType::RegularFile {
valid_entries += 1;
} else {
continue;
}
let entry_limit = data_dirs.as_ref().map_or(2, |dirs| dirs.len().saturating_add(2));
if valid_entries == entry_limit {
let dirs = data_dirs.get_or_insert_with(|| (load.take().expect("load_data_dirs called once"))());
if valid_entries == dirs.len().saturating_add(2) {
return Ok((std::mem::take(dirs), true));
}
}
}
let dirs = data_dirs
.take()
.unwrap_or_else(|| (load.take().expect("load_data_dirs called once"))());
Ok((dirs, true))
}
#[cfg(any(not(target_os = "linux"), test))]
pub(crate) fn has_non_data_dir_entry_lazy(
path: impl AsRef<Path>,
load_data_dirs: impl FnOnce() -> Vec<uuid::Uuid>,
) -> io::Result<(Vec<uuid::Uuid>, bool)> {
match has_non_data_dir_entry_lazy_inner(path.as_ref(), load_data_dirs) {
Ok(res) => Ok(res),
Err(err) if err.kind() == io::ErrorKind::NotFound => Ok((Vec::new(), false)),
Err(err) => Err(err),
}
}
#[cfg(any(not(target_os = "linux"), test))]
fn has_non_data_dir_entry_lazy_inner(
path: &Path,
load_data_dirs: impl FnOnce() -> Vec<uuid::Uuid>,
) -> io::Result<(Vec<uuid::Uuid>, bool)> {
let mut dir = std::fs::read_dir(path)?;
let mut data_dirs: Option<Vec<uuid::Uuid>> = None;
let mut load = Some(load_data_dirs);
let mut valid_entries = 0usize;
for _ in 0..READ_DIR_PROBE_RAW_LIMIT {
let Some(entry) = dir.next() else {
return Ok((data_dirs.unwrap_or_default(), false));
};
let entry = entry?;
let file_name = entry.file_name();
@@ -2777,18 +2869,24 @@ fn has_non_data_dir_entry_inner(path: &Path, data_dirs: &[uuid::Uuid]) -> io::Re
let file_type = entry.file_type()?;
if file_type.is_dir() {
let is_data_dir = file_name.len() == uuid::fmt::Hyphenated::LENGTH
&& file_name
.to_str()
.and_then(|s| {
let parsed = uuid::Uuid::try_parse(s).ok()?;
let mut buf = [0u8; uuid::fmt::Hyphenated::LENGTH];
(parsed.hyphenated().encode_lower(&mut buf) == s && data_dirs.binary_search(&parsed).is_ok())
.then_some(())
})
.is_some();
if !is_data_dir {
return Ok(true);
let parsed_uuid = if file_name.len() == uuid::fmt::Hyphenated::LENGTH {
file_name.to_str().and_then(|s| {
let parsed = uuid::Uuid::try_parse(s).ok()?;
let mut buf = [0u8; uuid::fmt::Hyphenated::LENGTH];
(parsed.hyphenated().encode_lower(&mut buf) == s).then_some(parsed)
})
} else {
None
};
let Some(parsed) = parsed_uuid else {
let dirs = data_dirs
.take()
.unwrap_or_else(|| (load.take().expect("load_data_dirs called once"))());
return Ok((dirs, true));
};
let dirs = data_dirs.get_or_insert_with(|| (load.take().expect("load_data_dirs called once"))());
if dirs.binary_search(&parsed).is_err() {
return Ok((std::mem::take(dirs), true));
}
valid_entries += 1;
} else if file_type.is_file() {
@@ -2797,12 +2895,19 @@ fn has_non_data_dir_entry_inner(path: &Path, data_dirs: &[uuid::Uuid]) -> io::Re
continue;
}
let entry_limit = data_dirs.as_ref().map_or(2, |dirs| dirs.len().saturating_add(2));
if valid_entries == entry_limit {
return Ok(true);
let dirs = data_dirs.get_or_insert_with(|| (load.take().expect("load_data_dirs called once"))());
if valid_entries == dirs.len().saturating_add(2) {
return Ok((std::mem::take(dirs), true));
}
}
}
Ok(true)
let dirs = data_dirs
.take()
.unwrap_or_else(|| (load.take().expect("load_data_dirs called once"))());
Ok((dirs, true))
}
// read_dir count read limit. when count == 0 unlimit.
@@ -4832,6 +4937,17 @@ impl PublicationRoot {
)
}
#[cfg(target_os = "linux")]
pub(crate) fn open_directory_and_readonly_child(
&self,
dir_path: &Path,
child_name: &str,
) -> io::Result<(std::fs::File, std::fs::File)> {
let dir = self.open_directory(dir_path)?;
let child = open_linux_relative(&dir, Path::new(child_name), rustix::fs::OFlags::RDONLY, rustix::fs::Mode::empty())?;
Ok((dir, child))
}
#[cfg(target_os = "linux")]
fn relative_path<'a>(&self, path: &'a Path) -> io::Result<&'a Path> {
path.strip_prefix(self.io_path.as_path())
@@ -10985,6 +11101,54 @@ mod tests {
assert_eq!(err.kind(), io::ErrorKind::NotFound);
}
#[test]
fn has_non_data_dir_entry_lazy_skips_loader_when_only_xl_meta_exists() {
let temp_dir = tempdir().expect("create temp dir");
std::fs::write(temp_dir.path().join("xl.meta"), b"meta").expect("write xl.meta");
let mut called = 0usize;
let (dirs, has_child) = has_non_data_dir_entry_lazy(temp_dir.path(), || {
called += 1;
vec![uuid::Uuid::new_v4()]
})
.expect("lazy check must succeed");
assert_eq!(called, 0, "inline object dir without subdirs must not invoke loader");
assert!(dirs.is_empty());
assert!(!has_child);
#[cfg(target_os = "linux")]
{
let pub_root = PublicationRoot::new(temp_dir.path()).expect("open pub root");
let (dir_fd, _meta) = pub_root
.open_directory_and_readonly_child(temp_dir.path(), "xl.meta")
.expect("open dir and child");
let mut fd_called = 0usize;
let (fd_dirs, fd_has_child) = has_non_data_dir_entry_lazy_from_fd(&dir_fd, || {
fd_called += 1;
vec![uuid::Uuid::new_v4()]
})
.expect("fd lazy check must succeed");
assert_eq!(fd_called, 0);
assert!(fd_dirs.is_empty());
assert!(!fd_has_child);
}
let data_dir = uuid::Uuid::new_v4();
std::fs::create_dir(temp_dir.path().join(data_dir.to_string())).expect("create data dir");
let (dirs, has_child) = has_non_data_dir_entry_lazy(temp_dir.path(), || {
called += 1;
vec![data_dir]
})
.expect("lazy check with data dir must succeed");
assert_eq!(called, 1, "data dir presence must invoke loader once");
assert_eq!(dirs, vec![data_dir]);
assert!(!has_child);
std::fs::create_dir(temp_dir.path().join("nested-key")).expect("create nested child dir");
let (dirs, has_child) = has_non_data_dir_entry_lazy(temp_dir.path(), || vec![data_dir]).expect("lazy check child");
assert_eq!(dirs, vec![data_dir], "child presence must still return data_dirs for skip set");
assert!(has_child);
}
#[tokio::test]
#[serial_test::serial(file_sync_probe)]
async fn resource_grouped_fdatasync_retired_limiter_keeps_its_budget_until_worker_cleanup() {
+15 -1
View File
@@ -828,13 +828,27 @@ pub async fn create_bitrot_writer(
length: i64,
shard_size: usize,
checksum_algo: HashAlgorithm,
) -> disk::error::Result<BitrotWriterWrapper> {
create_bitrot_writer_with_orig_volume(is_inline_buffer, disk, "", volume, path, length, shard_size, checksum_algo).await
}
#[allow(clippy::too_many_arguments)]
pub async fn create_bitrot_writer_with_orig_volume(
is_inline_buffer: bool,
disk: Option<&DiskStore>,
origvolume: &str,
volume: &str,
path: &str,
length: i64,
shard_size: usize,
checksum_algo: HashAlgorithm,
) -> disk::error::Result<BitrotWriterWrapper> {
let writer = if is_inline_buffer {
CustomWriter::new_inline_buffer()
} else if let Some(disk) = disk {
let length = bitrot_create_file_size(length, shard_size, &checksum_algo);
let file = disk.create_file("", volume, path, length).await?;
let file = disk.create_file(origvolume, volume, path, length).await?;
#[cfg(feature = "hotpath")]
let file = instrument_raw_shard_writer(file, disk.is_local());
CustomWriter::new_tokio_writer(file)
+4
View File
@@ -1543,6 +1543,8 @@ pub struct ObjectInfo {
/// published by the same upload. Callers must not repeat quota, replication,
/// lifecycle, or object-created side effects for that response.
pub multipart_completion_replayed: bool,
/// Previous current version's size observed during `rename_data` commit.
pub backfilled_old_current_size: Option<crate::disk::OldCurrentSize>,
}
impl Default for ObjectInfo {
@@ -1585,6 +1587,7 @@ impl Default for ObjectInfo {
replication_decision: String::new(),
checksum: None,
multipart_completion_replayed: false,
backfilled_old_current_size: None,
}
}
}
@@ -1628,6 +1631,7 @@ impl Clone for ObjectInfo {
replication_decision: self.replication_decision.clone(),
checksum: self.checksum.clone(),
multipart_completion_replayed: self.multipart_completion_replayed,
backfilled_old_current_size: self.backfilled_old_current_size,
expires: self.expires,
}
}
+26
View File
@@ -941,6 +941,32 @@ impl InstanceContext {
.clone()
}
/// Remove the `BucketMetadataSys` when its backing `ECStore` is gone.
///
/// Called from `ECStore::drop`. When test fixtures share the bootstrap
/// context, the `BucketMetadataSys` stores a `Weak<ECStore>` to prevent a
/// reference cycle; once the store drops the weak is dangling and any
/// `object_store()` call panics. Clearing the slot here lets the next
/// fixture install a fresh `BucketMetadataSys` without hitting the stale one.
pub fn clear_bucket_metadata_sys_if_stale(&self) {
let mut guard = self.bucket_metadata_sys.lock().expect("bucket_metadata_sys lock poisoned");
if let Some(sys) = guard.as_ref() {
// `read()` may block briefly; the async RwLock is acceptable here
// because Drop is sync and the lock is held only long enough to
// check the weak reference.
let is_stale = {
let s = sys.try_read();
match s {
Ok(s) => s.is_store_stale(),
Err(_) => true, // contended: treat as stale-safe to clear
}
};
if is_stale {
*guard = None;
}
}
}
/// Set this instance's background-services cancellation token (once).
pub fn init_background_cancel_token(&self, token: CancellationToken) -> Result<(), CancellationToken> {
self.background_cancel_token.set(token)
@@ -5598,10 +5598,11 @@ impl SetDisks {
write_quorum: usize,
quorum_context: Option<MultipartWriteQuorumContext<'_>>,
) -> disk::error::Result<Vec<Option<DiskStore>>> {
self.recover_part_transaction(dst_object, write_quorum).await?;
let part = ObjectPartInfo::unmarshal(&meta)?;
let integrity = part.integrity.map(Arc::new);
self.recover_part_transaction(dst_object, write_quorum).await?;
let src_bucket = Arc::new(src_bucket.to_string());
let src_object = Arc::new(src_object.to_string());
let dst_bucket = Arc::new(dst_bucket.to_string());
@@ -5623,7 +5624,9 @@ impl SetDisks {
if let Err(error) = prepared {
return Some(Err(error));
}
if let Some(integrity) = integrity {
if let Some(integrity) = integrity
&& !disk.is_local()
{
let Some((directory, _)) = dst_object.rsplit_once('/') else {
return Some(Err(DiskError::FileCorrupt));
};
@@ -5740,7 +5743,36 @@ impl SetDisks {
return Err(err);
}
let committed = self.recover_part_transaction(&dst_object, write_quorum).await?;
let all_prepared_succeeded = errs
.iter()
.zip(prepared_disks.iter())
.all(|(err, disk)| disk.is_none() || err.is_none());
let committed = if all_prepared_succeeded {
let settles = prepared_disks.iter().map(|disk| {
let disk = disk.clone();
let dst_object = dst_object.clone();
async move {
let Some(disk) = disk else {
return Err(DiskError::DiskNotFound);
};
disk.settle_part_transaction(RUSTFS_META_MULTIPART_BUCKET, &dst_object, PartTransactionAction::Commit)
.await
}
});
let settle_results = join_all(settles).await;
if settle_results.iter().filter(|res| res.is_ok()).count() >= write_quorum
&& settle_results
.iter()
.zip(prepared_disks.iter())
.all(|(res, disk)| disk.is_none() || res.is_ok())
{
true
} else {
self.recover_part_transaction(&dst_object, write_quorum).await?
}
} else {
self.recover_part_transaction(&dst_object, write_quorum).await?
};
if !committed {
let err = DiskError::ErasureWriteQuorum;
if let Some(context) = quorum_context {
@@ -6737,9 +6769,7 @@ impl SetDisks {
object,
);
}
let if_none_match = http_preconditions.if_none_match_value().map(str::to_owned);
let if_match = http_preconditions.if_match_value().map(str::to_owned);
if should_prevent_write(&oi, if_none_match, if_match)
if should_prevent_write(&oi, http_preconditions.if_none_match_value(), http_preconditions.if_match_value())
&& !crate::data_movement::can_replace_stale_data_movement_target(&oi, opts)
{
return Some(StorageError::PreconditionFailed);
+63 -36
View File
@@ -80,7 +80,7 @@ use crate::erasure::codec::bridge::{
use crate::erasure::coding;
use crate::error::{Error, Result, is_err_version_not_found};
use crate::error::{GenericError, ObjectApiError, is_err_object_not_found};
use crate::io_support::bitrot::{create_bitrot_reader_from_bytes, create_bitrot_writer};
use crate::io_support::bitrot::{create_bitrot_reader_from_bytes, create_bitrot_writer, create_bitrot_writer_with_orig_volume};
use crate::object_api::ObjectOptions;
use crate::object_api::get_object_body_cache_hook;
use crate::object_api::object_api_utils::get_raw_etag;
@@ -120,7 +120,6 @@ use bytesize::ByteSize;
use futures::future::join_all;
use http::HeaderMap;
use md5::{Digest as Md5Digest, Md5};
use regex::Regex;
use rustfs_config::MI_B;
use rustfs_filemeta::metadata_keys;
use rustfs_filemeta::{
@@ -8036,16 +8035,29 @@ fn parts_after_marker(part_numbers: &[usize], part_number_marker: usize) -> Opti
Some(&part_numbers[index..])
}
pub fn canonicalize_etag(etag: &str) -> String {
let re = Regex::new("\"*?([^\"]*?)\"*?$").unwrap();
re.replace_all(etag, "$1").to_string()
fn canonicalized_etag(etag: &str) -> Cow<'_, str> {
// Preserve the legacy suffix-matching behavior for embedded quotes.
let trimmed = etag.trim_end_matches('"');
let Some(last_quote) = trimmed.rfind('"') else {
return Cow::Borrowed(trimmed);
};
let prefix = trimmed[..last_quote].trim_end_matches('"');
let suffix = &trimmed[last_quote + 1..];
if prefix.is_empty() {
Cow::Borrowed(suffix)
} else {
let mut result = String::with_capacity(prefix.len() + suffix.len());
result.push_str(prefix);
result.push_str(suffix);
Cow::Owned(result)
}
}
pub fn e_tag_matches(etag: &str, condition: &str) -> bool {
if condition.trim() == "*" {
return true;
}
canonicalize_etag(etag) == canonicalize_etag(condition)
canonicalized_etag(etag) == canonicalized_etag(condition)
}
fn weak_etag_matches(etag: &str, condition: &str) -> bool {
@@ -8060,15 +8072,12 @@ fn weak_etag_matches(etag: &str, condition: &str) -> bool {
.or_else(|| trimmed.strip_prefix("w/"))
.unwrap_or(trimmed)
}
canonicalize_etag(strip_weak(etag)) == canonicalize_etag(strip_weak(condition))
canonicalized_etag(strip_weak(etag)) == canonicalized_etag(strip_weak(condition))
}
pub fn should_prevent_write(oi: &ObjectInfo, if_none_match: Option<String>, if_match: Option<String>) -> bool {
let if_none_match = if_none_match
.as_deref()
.map(str::trim)
.filter(|condition| !condition.is_empty());
let if_match = if_match.as_deref().map(str::trim).filter(|condition| !condition.is_empty());
pub fn should_prevent_write(oi: &ObjectInfo, if_none_match: Option<&str>, if_match: Option<&str>) -> bool {
let if_none_match = if_none_match.map(str::trim).filter(|condition| !condition.is_empty());
let if_match = if_match.map(str::trim).filter(|condition| !condition.is_empty());
match &oi.etag {
Some(etag) => {
@@ -9029,6 +9038,13 @@ mod tests {
"constructing an object lock must not clone each client Arc"
);
// Check shared ownership before acquisition can spawn pending cleanup.
let read_lock = set
.new_ns_lock("bucket", "read-object")
.await
.expect("second namespace lock should be created");
assert_eq!(Arc::strong_count(&set.shared_lockers), clients_before + 2);
let write_error = write_lock
.get_write_lock(Duration::from_millis(500))
.await
@@ -9043,11 +9059,6 @@ mod tests {
),
"the shared client representation must preserve the exact write quorum result: {write_error}"
);
let read_lock = set
.new_ns_lock("bucket", "read-object")
.await
.expect("second namespace lock should be created");
assert_eq!(Arc::strong_count(&set.shared_lockers), clients_before + 2);
let read_guard = read_lock
.get_read_lock(Duration::from_millis(500))
.await
@@ -13772,6 +13783,22 @@ mod tests {
assert!(result2.iter().all(|d| d.is_none()));
}
#[test]
fn canonicalized_etag_preserves_legacy_regex_semantics() {
let regex = regex::Regex::new("\"*?([^\"]*?)\"*?$").expect("valid legacy ETag pattern");
let alphabet = ['a', '"', '\n', 'é', '\0'];
for length in 0..=6_u32 {
for mut number in 0..alphabet.len().pow(length) {
let mut input = String::new();
for _ in 0..length {
input.push(alphabet[number % alphabet.len()]);
number /= alphabet.len();
}
assert_eq!(canonicalized_etag(&input), regex.replace_all(&input, "$1"), "input: {input:?}");
}
}
}
#[test]
fn test_etag_matches() {
assert!(e_tag_matches("abc", "abc"));
@@ -14347,61 +14374,61 @@ mod tests {
etag: Some("abc".to_string()),
..Default::default()
};
let if_none_match = Some("abc".to_string());
let if_none_match = Some("abc");
let if_match = None;
assert!(should_prevent_write(&oi, if_none_match, if_match));
let if_none_match = Some("*".to_string());
let if_none_match = Some("*");
let if_match = None;
assert!(should_prevent_write(&oi, if_none_match, if_match));
let if_none_match = None;
let if_match = Some("def".to_string());
let if_match = Some("def");
assert!(should_prevent_write(&oi, if_none_match, if_match));
let if_none_match = None;
let if_match = Some("*".to_string());
let if_match = Some("*");
assert!(!should_prevent_write(&oi, if_none_match, if_match));
let if_none_match = Some("def".to_string());
let if_none_match = Some("def");
let if_match = None;
assert!(!should_prevent_write(&oi, if_none_match, if_match));
let if_none_match = Some("def".to_string());
let if_match = Some("*".to_string());
let if_none_match = Some("def");
let if_match = Some("*");
assert!(!should_prevent_write(&oi, if_none_match, if_match));
let if_none_match = Some("def".to_string());
let if_match = Some("\"abc\"".to_string());
let if_none_match = Some("def");
let if_match = Some("\"abc\"");
assert!(!should_prevent_write(&oi, if_none_match, if_match));
let if_none_match = Some("*".to_string());
let if_match = Some("\"abc\"".to_string());
let if_none_match = Some("*");
let if_match = Some("\"abc\"");
assert!(should_prevent_write(&oi, if_none_match, if_match));
let oi = ObjectInfo {
etag: None,
..Default::default()
};
let if_none_match = Some("*".to_string());
let if_match = Some("\"abc\"".to_string());
let if_none_match = Some("*");
let if_match = Some("\"abc\"");
assert!(should_prevent_write(&oi, if_none_match, if_match));
let if_none_match = None;
let if_match = None;
assert!(!should_prevent_write(&oi, if_none_match, if_match));
let if_none_match = Some(String::new());
let if_match = Some(" ".to_string());
let if_none_match = Some("");
let if_match = Some(" ");
assert!(!should_prevent_write(&oi, if_none_match, if_match));
let oi = ObjectInfo {
etag: Some("abc".to_string()),
..Default::default()
};
assert!(should_prevent_write(&oi, Some("W/\"abc\"".to_string()), None));
assert!(!should_prevent_write(&oi, Some("W/\"def\"".to_string()), None));
assert!(should_prevent_write(&oi, None, Some("W/\"abc\"".to_string())));
assert!(should_prevent_write(&oi, Some("W/\"abc\""), None));
assert!(!should_prevent_write(&oi, Some("W/\"def\""), None));
assert!(should_prevent_write(&oi, None, Some("W/\"abc\"")));
}
#[test]
+8 -5
View File
@@ -3471,10 +3471,6 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
bind_multipart_proof_snapshot(&disks, &self.multipart_proof_verification_context())?;
let part_path = format!("{}/{}/", upload_id_path, fi.data_dir.unwrap_or(Uuid::nil()));
self.recover_part_transactions(&part_path, read_quorum, write_quorum)
.await
.map_err(|err| to_object_err(err.into(), vec![bucket, object]))?;
let part_meta_paths = uploaded_parts
.iter()
.map(|v| format!("{part_path}part.{0}.meta", v.part_num))
@@ -3482,6 +3478,10 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
let part_numbers = uploaded_parts.iter().map(|v| v.part_num).collect::<Vec<usize>>();
self.recover_part_transactions(&part_path, read_quorum, write_quorum)
.await
.map_err(|err| to_object_err(err.into(), vec![bucket, object]))?;
let object_parts = Self::read_parts(&disks, RUSTFS_META_MULTIPART_BUCKET, &part_meta_paths, &part_numbers, read_quorum)
.await
.map_err(|err| to_object_err(err.into(), vec![bucket, object]))?;
@@ -4389,6 +4389,7 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
let needs_immediate_heal = rename_commit.needs_immediate_heal();
let op_old_dir = rename_commit.data_dir;
let cleanup_disks = rename_commit.cleanup_disks;
let old_current_size = rename_commit.old_current_size;
let committed_file_info = rename_commit.committed_file_info;
if needs_immediate_heal {
@@ -4503,7 +4504,9 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
drop(_upload_guard.take());
Ok(ObjectInfo::from_file_info(&fi, &commit_bucket, &commit_object, commit_is_versioned))
let mut obj_info = ObjectInfo::from_file_info(&fi, &commit_bucket, &commit_object, commit_is_versioned);
obj_info.backfilled_old_current_size = old_current_size;
Ok(obj_info)
};
let result = if detach_commit_owner {
+8 -5
View File
@@ -46,9 +46,9 @@ use super::super::{
build_get_object_info, build_inline_bitrot_readers, build_inline_bitrot_readers_from_refs, can_try_inline_data_shards_direct,
check_object_lock_delete, check_object_lock_for_deletion_with_state, check_object_lock_retention_update,
classify_get_codec_streaming_object_class, classify_put_write_path, classify_storage_error,
collect_inline_data_shard_fileinfos_by_index, contains_key_str, create_bitrot_writer, debug, delete_file_info_version_id,
disk, ensure_delete_commit_locks_held, error, explicit_delete_removed_marker, finish_set_disk_read_lock,
get_codec_streaming_reader_gate_with_plan, get_object_body_cache_hook, get_raw_etag,
collect_inline_data_shard_fileinfos_by_index, contains_key_str, create_bitrot_writer_with_orig_volume, debug,
delete_file_info_version_id, disk, ensure_delete_commit_locks_held, error, explicit_delete_removed_marker,
finish_set_disk_read_lock, get_codec_streaming_reader_gate_with_plan, get_object_body_cache_hook, get_raw_etag,
get_small_object_direct_memory_decision_with_threshold_and_plan, get_small_object_direct_memory_threshold,
get_stage_timer_if_enabled, get_str, get_transitioned_object_reader_with_tier_manager, inline_admission_shard_size,
inline_erasure_shard_file_offset, inline_erasure_shard_size, insert_str, is_deadlock_detection_enabled,
@@ -4459,9 +4459,10 @@ impl SetDisks {
if let Some(disk) = disk_op
&& disk.is_online().await
{
match create_bitrot_writer(
match create_bitrot_writer_with_orig_volume(
is_inline_buffer,
Some(disk),
bucket,
RUSTFS_META_TMP_BUCKET,
&tmp_obj,
shard_file_size,
@@ -11064,7 +11065,9 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
if let Some(err) = &gerr
&& goi.name.is_empty()
{
if should_force_delete_marker_for_missing_version(&opts) {
if (is_err_object_not_found(err) || is_err_version_not_found(err))
&& should_force_delete_marker_for_missing_version(&opts)
{
version_found = false;
} else {
if is_err_object_not_found(err) {
+197
View File
@@ -5340,6 +5340,203 @@ mod tests {
shutdown.cancel();
}
#[cfg(feature = "test-util")]
#[tokio::test]
#[serial_test::serial(storage_class_env)]
async fn conditional_multipart_must_not_commit_against_an_older_readable_pool() {
use crate::storage_api_contracts::multipart::CompletePart;
use crate::storage_api_contracts::object::HTTPPreconditions;
let temp_dir = tempfile::tempdir().expect("create conditional multipart test directory");
let (_ctx, store, shutdown) =
without_storage_class_env(build_isolated_test_store(temp_dir.path(), "conditional-mpu-unreadable", &[4, 4])).await;
crate::bucket::metadata_sys::init_bucket_metadata_sys(store.clone(), Vec::new()).await;
let bucket = format!("conditional-mpu-{}", uuid::Uuid::new_v4().simple());
let object = "object";
store
.make_bucket(&bucket, &MakeBucketOptions::default())
.await
.expect("create conditional multipart test bucket");
let mut old_body = PutObjReader::from_vec(b"older-pool".to_vec());
let old = store.pools[0]
.put_object(
&bucket,
object,
&mut old_body,
&ObjectOptions {
mod_time: Some(OffsetDateTime::UNIX_EPOCH + time::Duration::seconds(1)),
..Default::default()
},
)
.await
.expect("write old object to staging pool");
let upload = store.pools[0]
.new_multipart_upload(&bucket, object, &ObjectOptions::default())
.await
.expect("create multipart upload in pool 0");
let mut part_body = PutObjReader::from_vec(b"stale-writer".to_vec());
let part = store.pools[0]
.put_object_part(&bucket, object, &upload.upload_id, 1, &mut part_body, &ObjectOptions::default())
.await
.expect("stage multipart part in pool 0");
let mut new_body = PutObjReader::from_vec(b"newer-pool".to_vec());
let new = store.pools[1]
.put_object(
&bucket,
object,
&mut new_body,
&ObjectOptions {
mod_time: Some(OffsetDateTime::UNIX_EPOCH + time::Duration::seconds(2)),
..Default::default()
},
)
.await
.expect("write newer object to pool 1");
let old_etag = old.etag.clone().expect("old object must have an ETag");
let before = store.get_object_info(&bucket, object, &ObjectOptions::default()).await;
let mut metadata = Vec::new();
for disk_index in 0..4 {
let path = temp_dir
.path()
.join(format!("pool1/set0/disk{disk_index}/{bucket}/{object}/xl.meta"));
let original = tokio::fs::read(&path)
.await
.expect("save original object metadata before fault injection");
metadata.push((path, original));
}
let mut fault_error = None;
for (path, _) in &metadata {
if let Err(err) = tokio::fs::write(path, b"invalid object metadata").await {
fault_error = Some(err);
break;
}
}
let mut fault_contents = Vec::new();
if fault_error.is_none() {
for (path, _) in &metadata {
fault_contents.push(tokio::fs::read(path).await);
}
store.pools[1]
.get_disks_by_key(object)
.invalidate_get_object_metadata_cache(&bucket, object)
.await;
}
let unreadable = if fault_error.is_none() {
Some(
store.pools[1]
.get_object_info(
&bucket,
object,
&ObjectOptions {
no_lock: true,
metadata_cache_safe: false,
suppress_read_repair: true,
..Default::default()
},
)
.await,
)
} else {
None
};
let completed = if fault_error.is_none() {
let completion_store = Arc::clone(&store);
let completion_bucket = bucket.clone();
let upload_id = upload.upload_id.clone();
let part_etag = part.etag.clone();
Some(
tokio::spawn(async move {
completion_store
.complete_multipart_upload(
&completion_bucket,
object,
&upload_id,
vec![CompletePart {
part_num: 1,
etag: part_etag,
..Default::default()
}],
&ObjectOptions {
http_preconditions: Some(HTTPPreconditions {
if_match: Some(old_etag),
..Default::default()
}),
..Default::default()
},
)
.await
})
.await,
)
} else {
None
};
let mut restore_error = None;
for (path, original) in &metadata {
if let Err(err) = tokio::fs::write(path, original).await {
restore_error = Some(err);
}
}
store.pools[1]
.get_disks_by_key(object)
.invalidate_get_object_metadata_cache(&bucket, object)
.await;
let after = store.get_object_info(&bucket, object, &ObjectOptions::default()).await;
let recovered_body = match store
.get_object_reader(&bucket, object, None, HeaderMap::new(), &ObjectOptions::default())
.await
{
Ok(mut reader) => {
let mut body = Vec::new();
let result = reader
.read_to_end(&mut body)
.await
.map(|_| body)
.map_err(|err| err.to_string());
drop(reader);
result
}
Err(err) => Err(err.to_string()),
};
let parts = store
.list_object_parts(&bucket, object, &upload.upload_id, None, 100, &ObjectOptions::default())
.await;
shutdown.cancel();
assert!(fault_error.is_none(), "failed to inject metadata fault: {fault_error:?}");
assert!(restore_error.is_none(), "failed to restore metadata: {restore_error:?}");
assert_eq!(fault_contents.len(), 4, "all four pool 1 metadata files must be inspected");
for observed in fault_contents {
assert_eq!(
observed.expect("read injected metadata"),
b"invalid object metadata",
"each pool 1 metadata file must contain the injected corruption"
);
}
assert_eq!(before.expect("latest object before fault").etag, new.etag);
assert_ne!(old.etag, new.etag, "fixture bodies must produce different ETags");
let read_error = unreadable
.expect("metadata fault should have been injected")
.expect_err("the newer pool must fail its object metadata read during the fault");
assert!(
!crate::error::is_err_object_not_found(&read_error) && !crate::error::is_err_version_not_found(&read_error),
"corrupt metadata must produce a non-not-found read error: {read_error:?}"
);
let completed = completed
.expect("completion should have been attempted")
.expect("completion task should not panic");
let Err(completion_error) = completed else {
panic!("stale If-Match must not commit while the newer object's metadata is unreadable");
};
assert_eq!(completion_error, read_error, "completion must propagate the unreadable pool's error");
assert_eq!(after.expect("newer object after recovery").etag, new.etag);
assert_eq!(recovered_body.expect("newer body after recovery"), b"newer-pool");
let parts = parts.expect("failed completion must leave the upload retryable");
assert_eq!(parts.parts.len(), 1);
assert_eq!(parts.parts[0].etag, part.etag);
}
async fn migrate_versioned_decommission_test_object(
store: &Arc<crate::store::ECStore>,
bucket: &str,
+32 -1
View File
@@ -1292,6 +1292,37 @@ impl ECStore {
);
let rx = rx.child_token();
if self.pools.len() == 1 && self.pools[0].disk_set.len() == 1 {
rustfs_io_metrics::record_list_objects_merge(LIST_OBJECTS_SOURCE_WALKER, 1, 1);
let res = self.pools[0].disk_set[0].list_path(rx, opts, sender).await;
let mut outcome = ListPathRawOutcome::default();
let err = match res {
Ok(set_outcome) => {
outcome.limit_reached = set_outcome.limit_reached;
None
}
Err(err) => Some(err),
};
let errs = [err];
if is_all_not_found(&errs) {
return Ok(outcome);
}
if let Some(err) = errs.into_iter().flatten().next() {
return Err(err);
}
rustfs_io_metrics::record_stage_duration(
"store_list_objects_list_merged",
merge_started.elapsed().as_secs_f64() * 1000.0,
);
debug!(
set_pair_count = 1,
limit_reached = outcome.limit_reached,
error_count = 0,
"store list_merged finished"
);
return Ok(outcome);
}
let mut futures = Vec::new();
let mut inputs = Vec::new();
@@ -1811,7 +1842,7 @@ async fn gather_results(
if opts.versioned
&& is_object
&& entry
.xl_meta()
.xl_meta_ref()
.is_ok_and(|meta| !meta.versions.is_empty() && meta.versions.iter().all(|version| version.header.free_version()))
{
continue;
+11
View File
@@ -498,6 +498,17 @@ pub struct ECStore {
pub(crate) bucket_fence_registry: Arc<bucket_fence::BucketFenceRegistry>,
}
impl Drop for ECStore {
/// Clear the instance context's `BucketMetadataSys` when its backing store
/// is dropped. Without this, the shared bootstrap context retains a
/// `BucketMetadataSys` whose `Weak<ECStore>` is dangling after the store
/// goes away (e.g. between sequential test fixtures that share the
/// bootstrap), and the next `object_store()` call panics.
fn drop(&mut self) {
self.ctx.clear_bucket_metadata_sys_if_stale();
}
}
const METRIC_SCANNER_DATA_MOVEMENT_PAUSED: &str = "rustfs_scanner_data_movement_paused";
const METRIC_SCANNER_DATA_MOVEMENT_PAUSE_DURATION_SECONDS: &str = "rustfs_scanner_data_movement_pause_duration_seconds";
const METRIC_SCANNER_DATA_MOVEMENT_BACKLOG_WORK_ITEMS: &str = "rustfs_scanner_data_movement_backlog_work_items";
+30 -8
View File
@@ -19,6 +19,7 @@ use crate::set_disk::get_lock_acquire_timeout;
use crate::storage_api_contracts::multipart::MultipartOperations as _;
use crate::storage_api_contracts::object::ObjectOperations as _;
use futures::{StreamExt, stream};
use rustfs_utils::path::encode_dir_object_ref;
use std::collections::HashSet;
const MULTIPART_LIST_SET_CONCURRENCY: usize = 4;
@@ -612,7 +613,7 @@ impl ECStore {
}
if opts.http_preconditions.is_some()
&& !opts.data_movement
&& let Some(err) = self.check_write_precondition(bucket, object, &opts).await
&& let Some(err) = self.check_write_precondition(bucket, object, &opts, false).await
{
return Err(err);
}
@@ -1154,12 +1155,31 @@ impl ECStore {
Ok(())
}
async fn check_write_precondition(&self, bucket: &str, object: &str, opts: &ObjectOptions) -> Option<StorageError> {
async fn check_write_precondition(
&self,
bucket: &str,
object: &str,
opts: &ObjectOptions,
require_all_pools: bool,
) -> Option<StorageError> {
let mut lookup_opts = opts.clone();
let http_preconditions = lookup_opts.http_preconditions?;
lookup_opts.http_preconditions = None;
lookup_opts.no_lock = true;
let oi = self.get_object_info_for_delete(bucket, object, &lookup_opts).await;
let oi = if require_all_pools {
lookup_opts.metadata_cache_safe = false;
lookup_opts.suppress_read_repair = true;
// Only completion uses this branch, after validating bucket and object.
let encoded_object = encode_dir_object_ref(object);
self.get_latest_object_info_for_conditional_write(bucket, encoded_object.as_ref(), &lookup_opts)
.await
.and_then(|(info, _)| {
lookup_opts.precondition_check(&info)?;
Ok(info)
})
} else {
self.get_object_info_for_delete(bucket, object, &lookup_opts).await
};
crate::set_disk::SetDisks::evaluate_write_precondition(oi, &http_preconditions, bucket, object, opts)
}
@@ -1186,11 +1206,13 @@ impl ECStore {
.await;
return enqueue_transition_after_write(self.as_ref(), result, LcEventSrc::S3CompleteMultipartUpload, opts).await;
}
if opts.http_preconditions.is_some()
&& !opts.data_movement
&& let Some(err) = self.check_write_precondition(bucket, object, opts).await
{
return Err(err);
if opts.http_preconditions.is_some() && !opts.data_movement {
// Preserve explicitly scoped internal reads and borrowed-lock callers.
let require_all_pools =
!opts.replication_request && !opts.no_lock && !opts.skip_decommissioned && !opts.skip_rebalancing;
if let Some(err) = self.check_write_precondition(bucket, object, opts, require_all_pools).await {
return Err(err);
}
}
let result = match self.multipart_upload_pool_idx(bucket, object, upload_id, opts).await {
+5 -3
View File
@@ -5637,13 +5637,11 @@ impl ECStore {
}
if self.single_pool()
&& !creates_latest_marker
&& !opts.data_movement
&& opts.http_preconditions.is_none()
&& !opts.replication_request
&& !opts.versioned
&& !opts.version_suspended
&& opts.version_id.is_none()
&& !opts.delete_marker
&& !self.is_suspended(0).await
&& !self.is_pool_rebalancing(0).await
{
@@ -5655,6 +5653,10 @@ impl ECStore {
.await
{
Ok(mut res) => {
#[cfg(test)]
if creates_latest_marker {
pause_versioned_delete_marker_after_commit(bucket, object).await;
}
res.name = decode_dir_object(res.name.as_str());
return Ok(res);
}
+27 -2
View File
@@ -837,7 +837,29 @@ impl ECStore {
object: &str,
opts: &ObjectOptions,
) -> Result<(ObjectInfo, usize)> {
let suspended_pools = if opts.skip_decommissioned {
self.get_latest_object_info_with_pool_policy(bucket, object, opts, false)
.await
}
/// Observe every pool for a conditional-write preflight. This does not
/// hold the namespace locks needed to prove a condition at publication.
pub(super) async fn get_latest_object_info_for_conditional_write(
&self,
bucket: &str,
object: &str,
opts: &ObjectOptions,
) -> Result<(ObjectInfo, usize)> {
self.get_latest_object_info_with_pool_policy(bucket, object, opts, true).await
}
async fn get_latest_object_info_with_pool_policy(
&self,
bucket: &str,
object: &str,
opts: &ObjectOptions,
require_all_pools: bool,
) -> Result<(ObjectInfo, usize)> {
let suspended_pools = if opts.skip_decommissioned && !require_all_pools {
let pool_meta = self.pool_meta.read().await;
Some(
(0..self.pools.len())
@@ -853,7 +875,7 @@ impl ECStore {
continue;
}
if opts.skip_rebalancing && self.is_pool_rebalancing(idx).await {
if opts.skip_rebalancing && !require_all_pools && self.is_pool_rebalancing(idx).await {
continue;
}
@@ -873,6 +895,9 @@ impl ECStore {
});
}
Err(e) => {
if require_all_pools && !is_err_object_not_found(&e) && !is_err_version_not_found(&e) {
return Err(e);
}
candidates.push(LatestObjectInfoCandidate {
info: None,
idx,
+47
View File
@@ -2070,6 +2070,53 @@ mod test {
assert!(meta.versions.iter().all(|version| !version.header.free_version()));
}
#[test]
fn bytes_count_toward_listing_limit_matches_fileinfo_visibility() {
let mut meta = FileMeta::new();
let mut obj = FileInfo::new("object", 2, 2);
obj.version_id = Some(Uuid::new_v4());
obj.mod_time = Some(OffsetDateTime::now_utc());
meta.add_version(obj.clone()).expect("add object version");
let obj_bytes = meta.marshal_msg().expect("marshal object");
assert!(FileMeta::bytes_count_toward_listing_limit(&obj_bytes));
let dm = FileInfo {
name: "object".to_string(),
version_id: Some(Uuid::new_v4()),
deleted: true,
mark_deleted: true,
mod_time: obj.mod_time.map(|t| t + time::Duration::seconds(1)),
..Default::default()
};
meta.delete_version(&dm).expect("add delete marker");
let dm_bytes = meta.marshal_msg().expect("marshal delete marker");
assert!(!FileMeta::bytes_count_toward_listing_limit(&dm_bytes));
let (_, mut source) = tier_overwrite_fixture(crate::TransitionVersionState::Exact);
source.version_id = Some(Uuid::new_v4());
let mut transitioned = FileMeta::new();
transitioned.add_version(source.clone()).expect("seed transitioned version");
let mut del_req = FileInfo {
name: "object".to_string(),
version_id: source.version_id,
mod_time: source.mod_time,
..Default::default()
};
del_req.set_tier_free_version_id(&Uuid::new_v4().to_string());
transitioned
.delete_version(&del_req)
.expect("delete transitioned version leaving free version");
assert!(!FileMeta::bytes_count_toward_listing_limit(
&transitioned.marshal_msg().expect("marshal free-only")
));
transitioned.add_version(obj).expect("add live object version");
assert!(FileMeta::bytes_count_toward_listing_limit(
&transitioned.marshal_msg().expect("marshal free + live object")
));
assert!(FileMeta::bytes_count_toward_listing_limit(b"corrupt-xl-meta"));
}
#[test]
fn add_version_filemata_uses_canonical_equal_time_order() {
let mod_time = OffsetDateTime::from_unix_timestamp(1_700_000_000).expect("valid test timestamp");
+100
View File
@@ -324,6 +324,63 @@ impl FileMeta {
true
}
pub(crate) fn buffers_match_strict(self_buf: &[u8], other_buf: &[u8]) -> bool {
let Ok((s_meta, _)) = Self::parse_validated_xl_envelope(self_buf) else {
return false;
};
let Ok((o_meta, _)) = Self::parse_validated_xl_envelope(other_buf) else {
return false;
};
if s_meta.is_empty() || o_meta.is_empty() {
return s_meta.is_empty() && o_meta.is_empty();
}
let Ok((s_len, s_header_ver, _s_meta_ver, s_meta)) = Self::decode_validated_xl_headers(s_meta) else {
return false;
};
let Ok((o_len, o_header_ver, _o_meta_ver, o_meta)) = Self::decode_validated_xl_headers(o_meta) else {
return false;
};
if s_len != o_len {
return false;
}
let mut s_cur: Cursor<&[u8]> = Cursor::new(s_meta);
let mut o_cur: Cursor<&[u8]> = Cursor::new(o_meta);
for _ in 0..s_len {
let Ok((s_header_raw, _)) = Self::read_next_raw_slices(s_meta, &mut s_cur) else {
return false;
};
let Ok((o_header_raw, _)) = Self::read_next_raw_slices(o_meta, &mut o_cur) else {
return false;
};
if s_header_ver == XL_HEADER_VERSION
&& o_header_ver == XL_HEADER_VERSION
&& s_header_raw.len() == 38
&& o_header_raw.len() == 38
&& s_header_raw == o_header_raw
&& parse_canonical_v3_header_fields(s_header_raw).is_some()
{
continue;
}
let mut s_header = FileMetaVersionHeader::default();
if s_header.unmarshal_v(s_header_ver, s_header_raw).is_err() {
return false;
}
let mut o_header = FileMetaVersionHeader::default();
if o_header.unmarshal_v(o_header_ver, o_header_raw).is_err() {
return false;
}
if !Self::headers_match_strict(&s_header, &o_header) {
return false;
}
}
true
}
#[inline]
fn header_matches_canonical_v3_raw_strict(s_header: &FileMetaVersionHeader, buf: &[u8]) -> Option<bool> {
if buf.len() != 38 {
@@ -625,6 +682,49 @@ impl FileMeta {
}
}
pub fn bytes_count_toward_listing_limit(buf: &[u8]) -> bool {
let Ok((meta, _inline_buf)) = Self::parse_validated_xl_envelope(buf) else {
return true;
};
if meta.is_empty() {
return false;
}
let Ok((versions_len, header_ver, _meta_ver, meta)) = Self::decode_validated_xl_headers(meta) else {
return true;
};
if versions_len == 0 {
return false;
}
let mut cur = Cursor::new(meta);
let mut selected: Option<(FileMetaVersionHeader, &[u8])> = None;
let mut any_non_hidden = false;
for _ in 0..versions_len {
let Ok((header, ver_meta)) = Self::read_next_shallow_slice(meta, &mut cur, header_ver) else {
return true;
};
if header.version_type != VersionType::Delete && !header.free_version() {
any_non_hidden = true;
}
if selected.is_some() {
continue;
}
if header.free_version() && header.version_id.is_some_and(|id| !id.is_nil()) {
continue;
}
selected = Some((header, ver_meta));
}
let Some((header, ver_meta)) = selected else {
return false;
};
if !header.free_version() && header.version_type != VersionType::Delete {
return true;
}
if !any_non_hidden {
return false;
}
FileMetaVersion::decode_fileinfo_from_slice(ver_meta, "", "", true, false).is_err()
}
#[hotpath::measure(impl_type = "FileMeta")]
pub fn marshal_msg(&self) -> Result<Vec<u8>> {
let mut wr =
+197
View File
@@ -352,6 +352,22 @@ impl MetaCacheEntry {
(prefer, true)
}
pub fn matches_strict_borrowed(&mut self, other: &MetaCacheEntry) -> bool {
if self.name != other.name {
return false;
}
if other.is_dir() || self.is_dir() {
return other.is_dir() == self.is_dir();
}
if self.cached.is_none() && other.cached.is_none() {
return FileMeta::buffers_match_strict(&self.metadata, &other.metadata);
}
self.matches_strict(other)
}
pub fn matches_strict(&mut self, other: &MetaCacheEntry) -> bool {
if self.name != other.name {
return false;
@@ -989,6 +1005,17 @@ impl<W: AsyncWrite + Unpin> MetacacheWriter<W> {
Ok(())
}
pub async fn write_obj_buffered(&mut self, obj: &MetaCacheEntry) -> Result<()> {
self.init().await?;
Self::encode_obj(&mut self.buf, obj)?;
if self.buf.len() >= METACACHE_READ_CHUNK_SIZE {
self.flush().await?;
}
Ok(())
}
pub async fn close(&mut self) -> Result<()> {
self.close_with_limit_reached(false).await
}
@@ -1282,6 +1309,85 @@ impl<R: AsyncRead + Unpin> MetacacheReader<R> {
Ok(u32::from_be_bytes(buf.try_into().expect("Slice with incorrect length")))
}
pub fn skip_current(&mut self) -> bool {
if self.has_current && self.init && self.err.is_none() {
self.has_current = false;
true
} else {
false
}
}
pub fn try_peek_buffered(&mut self) -> Option<Result<Option<MetaCacheEntry>>> {
if !self.init || self.err.is_some() || self.terminated {
return None;
}
let avail = &self.buf[self.pos..];
let &first = avail.first()?;
if first != 0xc3 {
return None;
}
let str_marker = *avail.get(1)?;
let (str_hdr, str_len) = match str_marker {
0xa0..=0xbf => (1usize, usize::from(str_marker & 0x1f)),
0xd9 => (2usize, usize::from(*avail.get(2)?)),
0xda => (3usize, usize::from(u16::from_be_bytes([*avail.get(2)?, *avail.get(3)?]))),
0xdb => (
5usize,
usize::try_from(u32::from_be_bytes([*avail.get(2)?, *avail.get(3)?, *avail.get(4)?, *avail.get(5)?])).ok()?,
),
_ => return None,
};
let bin_offset = 1usize.checked_add(str_hdr)?.checked_add(str_len)?;
let bin_marker = *avail.get(bin_offset)?;
let (bin_hdr, bin_len) = match bin_marker {
0xc4 => (2usize, usize::from(*avail.get(bin_offset + 1)?)),
0xc5 => (
3usize,
usize::from(u16::from_be_bytes([*avail.get(bin_offset + 1)?, *avail.get(bin_offset + 2)?])),
),
0xc6 => (
5usize,
usize::try_from(u32::from_be_bytes([
*avail.get(bin_offset + 1)?,
*avail.get(bin_offset + 2)?,
*avail.get(bin_offset + 3)?,
*avail.get(bin_offset + 4)?,
]))
.ok()?,
),
_ => return None,
};
let total_len = bin_offset.checked_add(bin_hdr)?.checked_add(bin_len)?;
if avail.len() < total_len {
return None;
}
let Ok(decoded) = from_utf8(&avail[1 + str_hdr..bin_offset]) else {
return None;
};
let name = decoded.to_owned();
let metadata = if bin_len == 0 {
Vec::new()
} else {
avail[bin_offset + bin_hdr..total_len].to_vec()
};
self.pos += total_len;
self.reset();
self.has_current = true;
Some(Ok(Some(MetaCacheEntry {
name,
metadata,
cached: None,
reusable: false,
})))
}
pub async fn skip(&mut self, size: usize) -> Result<()> {
self.check_init().await?;
@@ -4257,4 +4363,95 @@ mod tests {
.expect("resolve mixed empty");
assert_eq!(res_mixed_empty.cached.expect("cached").versions.len(), 1);
}
#[test]
fn matches_strict_borrowed_matches_shallow_headers_and_rejects_corrupt_or_divergent_buffers() {
let base_bytes = create_real_xlmeta().expect("create real xlmeta");
let mut peer_fm = FileMeta::load(&base_bytes).expect("load base xlmeta");
// Mutate inner version payload (ver.meta) without changing shallow header.
let mut ver = FileMetaVersion::try_from(peer_fm.versions[1].meta.as_slice()).expect("decode version");
ver.object.as_mut().expect("object version").erasure_index += 1;
peer_fm.versions[1].meta = ver.marshal_msg().expect("marshal peer version");
let peer_bytes = peer_fm.marshal_msg().expect("marshal peer xlmeta");
assert_ne!(base_bytes, peer_bytes, "peer shard has distinct erasure index payload");
let mut e1 = MetaCacheEntry {
name: "obj-a".to_owned(),
metadata: base_bytes.clone(),
cached: None,
reusable: false,
};
let e1_clone = e1.clone();
let e2 = MetaCacheEntry {
name: "obj-a".to_owned(),
metadata: peer_bytes,
cached: None,
reusable: false,
};
assert!(e1.matches_strict_borrowed(&e2));
assert!(e1.matches_strict_borrowed(&e1_clone));
// Different shallow header must not match.
let mut newer_fm = FileMeta::load(&base_bytes).expect("load base xlmeta");
newer_fm.versions[0].header.version_id = Some(Uuid::new_v4());
let newer_entry = MetaCacheEntry {
name: "obj-a".to_owned(),
metadata: newer_fm.marshal_msg().expect("marshal newer xlmeta"),
cached: None,
reusable: false,
};
assert!(!e1.matches_strict_borrowed(&newer_entry));
// Identical corrupt buffers must return false, never true via byte-equality shortcut.
let mut corrupt = MetaCacheEntry {
name: "obj-a".to_owned(),
metadata: b"XL2 corrupt-payload".to_vec(),
cached: None,
reusable: false,
};
let corrupt_peer = corrupt.clone();
assert!(!corrupt.matches_strict_borrowed(&corrupt_peer));
}
#[tokio::test]
async fn try_peek_buffered_and_skip_current_match_async_peek_and_skip() {
let entries: Vec<_> = (0..5)
.map(|i| MetaCacheEntry {
name: format!("buffered-key-{i}"),
metadata: vec![i as u8; 16],
cached: None,
reusable: false,
})
.collect();
let mut buf = Cursor::new(Vec::new());
let mut writer = MetacacheWriter::new(&mut buf);
writer.write(&entries).await.expect("write entries");
writer.close_with_limit_reached(false).await.expect("close stream");
let mut reader = MetacacheReader::new(Cursor::new(buf.into_inner()));
assert!(reader.try_peek_buffered().is_none(), "uninitialized reader must defer to async peek");
assert!(!reader.skip_current(), "no current entry before first peek");
let first = reader.peek().await.expect("first peek").expect("entry 0");
assert_eq!(first, entries[0]);
assert!(reader.skip_current(), "skip_current must consume the peeked entry");
assert!(!reader.skip_current(), "second skip_current without peek must return false");
for expected in &entries[1..] {
let next = reader
.try_peek_buffered()
.expect("remaining entries are already buffered")
.expect("valid entry")
.expect("present entry");
assert_eq!(&next, expected);
assert!(reader.skip_current());
}
assert!(
reader.try_peek_buffered().is_none(),
"stream terminator must defer to async peek for trailer handling"
);
assert!(reader.peek().await.expect("eof peek").is_none());
assert_eq!(reader.limit_reached(), Some(false));
}
}
+47 -1
View File
@@ -1062,6 +1062,31 @@ fn prepare_cycle_for_usage_floor_bootstrap(
}
}
/// Overwrite an unreadable usage floor with a bootstrap marker using the
/// file's current revision for CAS. A successful publish leaves a
/// `BootstrapPending` floor that the normal load path picks up on retry.
/// Returns `Err` when the floor is not even readable (transient I/O) so the
/// caller keeps the supervisor-retry behavior instead of rebuilding.
async fn rebuild_unreadable_usage_floor(
storeapi: Arc<impl ScannerObjectIO + ScannerConfigObjectDelete>,
) -> Result<(), ScannerError> {
let Some(expected_epoch) = scanner_publication_epoch(storeapi.clone()).await else {
return Err(ScannerError::Other("scanner usage floor rebuild is blocked by data movement".to_string()));
};
let (_data, revision) = read_config_with_revision(storeapi.clone(), DATA_USAGE_OBJ_NAME_PATH.as_str())
.await
.map_err(|err| ScannerError::Other(format!("scanner usage floor revision unreadable for rebuild: {err}")))?;
publish_scanner_usage_bootstrap_primary(
storeapi,
&revision,
expected_epoch,
None,
ScannerUsageBootstrapPublishContext::Recovery,
|| true,
)
.await
}
pub async fn init_data_scanner(ctx: CancellationToken, storeapi: Arc<ECStore>) {
init_data_scanner_with_storage(ctx, storeapi, ScannerRuntimeGuard::new()).await;
}
@@ -3090,6 +3115,27 @@ where
Err(err) => usage_floor_result = Err(err),
}
}
// backlog#2918: after dual-corruption recovery has been attempted an
// unreadable floor is no longer a quota proof. Report the failure, then
// rebuild through a CAS overwrite before giving up. Transient I/O errors
// fail the rebuild and fall through to the supervisor-retry stop below.
if usage_floor_result.is_err() {
let error = usage_floor_result.as_ref().unwrap_err().to_string();
error!(
target: "rustfs::scanner",
event = EVENT_SCANNER_PERSIST_STATE,
component = LOG_COMPONENT_SCANNER,
subsystem = LOG_SUBSYSTEM_RUNTIME,
path = %DATA_USAGE_OBJ_NAME_PATH.as_str(),
state = "usage_floor_load_failed",
error = %error,
"Scanner usage floor is unreadable after recovery; attempting rebuild"
);
record_scanner_usage_floor_failure(error);
if rebuild_unreadable_usage_floor(storeapi.clone()).await.is_ok() {
usage_floor_result = persisted_usage_floor_for_startup(storeapi.clone(), true).await;
}
}
let (usage_floor, usage_floor_startup) = match usage_floor_result {
Ok(result) => result,
Err(err) => {
@@ -3102,7 +3148,7 @@ where
path = %DATA_USAGE_OBJ_NAME_PATH.as_str(),
state = "usage_floor_load_failed",
error = %err,
"Scanner stopped because the persisted usage floor could not be loaded"
"Scanner usage floor could not be loaded or rebuilt; stopping leader iteration for supervisor retry"
);
record_scanner_usage_floor_failure(error.clone());
global_metrics().set_cycle(None).await;
@@ -41,20 +41,6 @@ fn reset_recovery_status() {
.expect("test must be able to reset the local recovery observation");
}
async fn wait_for_recovery_attempt(attempt: u32) -> ScannerCycleRecoveryStatus {
tokio::time::timeout(WAIT_BUDGET, async {
loop {
let status = scanner_cycle_recovery_status();
if status.retry_count >= u64::from(attempt) {
return status;
}
tokio::time::sleep(Duration::from_millis(10)).await;
}
})
.await
.unwrap_or_else(|_| panic!("recovery attempt {attempt} was never recorded: {:?}", scanner_cycle_recovery_status()))
}
async fn wait_for_recovery_state(state: &str) -> ScannerCycleRecoveryStatus {
tokio::time::timeout(WAIT_BUDGET, async {
loop {
@@ -130,20 +116,19 @@ fn started_rebalance() -> EcstoreRebalanceMeta {
}
}
/// rustfs/rustfs#8670: a usage floor that is present but untrusted must not
/// drive the supervisor into a terminal `paused, retryable=false` observation.
/// Every attempt stays retryable with a bounded backoff, and repairing the
/// floor recovers the scanner without a process restart.
/// rustfs/rustfs#8670 + backlog#2918: a usage floor that is present but
/// untrusted must not stop the scanner. The unreadable floor is reported and
/// rebuilt through a CAS overwrite so the scanner recovers without a process
/// restart or a terminal `paused, retryable=false` observation.
#[tokio::test]
#[serial]
async fn usage_floor_load_failure_backs_off_and_recovers_without_restart() {
async fn usage_floor_load_failure_rebuilds_and_recovers_without_restart() {
temp_env::async_with_vars(SUPERVISOR_ENV, async {
crate::runtime_config::refresh_scanner_runtime_config_for_tests();
crate::scanner_io::clear_dirty_usage_buckets_for_tests();
reset_recovery_status();
let (_dir, store) = setup_scanner_cycle_store().await;
// Present but without a baseline identity: the shape #8670 reported,
// whose error points the operator at the usage-state reset.
// Present but without a baseline identity: the shape #8670 reported.
save_config(
store.clone(),
DATA_USAGE_OBJ_NAME_PATH.as_str(),
@@ -156,41 +141,11 @@ async fn usage_floor_load_failure_backs_off_and_recovers_without_restart() {
let _cancel_on_drop = ctx.clone().drop_guard();
init_data_scanner_with_storage(ctx.clone(), store.clone(), ScannerRuntimeGuard::new()).await;
let attempts = MAX_SCANNER_CYCLE_RECOVERY_RETRIES + 2;
let mut first_detected = None;
for attempt in 1..=attempts {
let status = wait_for_recovery_attempt(attempt).await;
assert_eq!(status.classification.as_deref(), Some("usage_floor_load_failed"), "attempt {attempt}");
assert!(status.retryable, "attempt {attempt} must stay retryable: {status:?}");
assert_ne!(status.state, "paused", "attempt {attempt} must not report a terminal pause");
if attempt >= MAX_SCANNER_CYCLE_RECOVERY_RETRIES {
assert_eq!(status.state, "backoff", "attempt {attempt}: {status:?}");
} else {
assert_eq!(status.state, "usage_floor_load_failed", "attempt {attempt}: {status:?}");
}
assert!(
status
.reason
.as_deref()
.is_some_and(|reason| reason.contains("usage-state/reset") && reason.contains("next automatic attempt")),
"the operator hint and the next attempt must both be visible: {status:?}"
);
assert!(status.next_attempt_at_unix_secs.is_some(), "attempt {attempt}: {status:?}");
match first_detected {
None => first_detected = status.first_detected_at_unix_secs,
Some(first) => assert_eq!(status.first_detected_at_unix_secs, Some(first), "attempt {attempt}"),
}
if attempt == attempts {
save_config(store.clone(), DATA_USAGE_OBJ_NAME_PATH.as_str(), valid_usage_floor(0, 0))
.await
.expect("repaired usage floor should persist");
}
notify_scanner_cycle_recovery_wake();
}
// backlog#2918: the unreadable floor is rebuilt automatically; the
// scanner must reach healthy without a manual repair or terminal pause.
let healthy = wait_for_recovery_state("healthy").await;
assert_eq!(healthy.retry_count, 0, "a successful attempt clears the retry budget: {healthy:?}");
assert!(!healthy.retryable);
assert!(!healthy.retryable, "rebuild must clear the retry budget: {healthy:?}");
assert_ne!(healthy.state, "paused", "rebuild must not report a terminal pause");
wait_for_persisted_cycle(&store, |cycle, _| cycle.next >= 1).await;
assert_eq!(scanner_cycle_schedule_status().execution_role, "leader");
stop_supervisor(ctx).await;
+2 -2
View File
@@ -33,8 +33,8 @@ Run URLs are `https://github.com/rustfs/rustfs/actions/runs/<id>`. Re-open the r
| `workflow_dispatch` | manual full validation | [41152382212](https://github.com/rustfs/rustfs/actions/runs/41152382212) | full suite set ran and the aggregate accepted the complete plan (85 min) |
| `workflow_run` | `Scheduled Validation Watchdog` | [41161431618](https://github.com/rustfs/rustfs/actions/runs/41161431618) | independent filter/monitor path executed against the registry |
| `merge_group` | first queued candidate after queue enablement | see below | live evidence attached after the first queue entry completes |
| Documentation-only PR | this document's own PR | see below | `docs` mode: expensive jobs required **skipped** |
| Isolated E2E PR | drill branch confined to `crates/e2e_test/src/` | see below | `e2e` mode: Connect/ILM/protocol lanes required **skipped** |
| Documentation-only PR | PR #9088, this document's own PR | [41162871854](https://github.com/rustfs/rustfs/actions/runs/41162871854) | `docs` mode: `Select CI scope` recorded `mode: docs`; `Test and Lint` accepted in 8 s with every expensive job required **skipped** |
| Isolated E2E PR | PR #9089, comment-only change under `crates/e2e_test/src/` | [41163203959](https://github.com/rustfs/rustfs/actions/runs/41163203959) | `e2e` mode: Connect boundary, serial ILM, rio-v2/protocol lanes and real io_uring required **skipped**; workspace build, E2E and both S3 lanes retained |
## Repeating a drill
+29 -7
View File
@@ -117,18 +117,40 @@ Functional-chain health runs on a hosted runner independently of the lab fleet.
## Release validation
Post-merge and tag-driven; not a substitute for a PR gate.
Post-merge and tag-driven; not a substitute for a PR gate. Every public publication exit consumes the release certification described here; this certification path is separate from the reusable CI artifact contract in [verification plans](ci-verification-pipeline.md), whose archives expire after one day and are never used as release certification.
The versioned plan is `.config/release-validation-plan.json` (`schema_version=1`): release types, target OS/arch/ABI, supported features, a fixed harness SHA per lane, required case IDs, environment capabilities, timeouts, applicability conditions, and an explicit `upgrade_sources` list carrying the real artifact digests of the supported N-1/N-2 sources plus a stated rollback basis — compatibility and rollback are never invented or defaulted. `scripts/release/release_gate.py validate` expands and freezes `plan_sha256` and the required case set before reading results, then binds that plan, a `candidate.json` (product 40-character SHA, version, release profile, features, immutable deliverable locations with SHA256 and wrapped-binary SHA256, image digest) and the shared G01 evidence-envelope set (rustfs/backlog#2658) of one parent candidate batch, and writes machine JSON, JUnit and a summary. Within a batch the unique key is `(lane, platform{os,arch,abi}, case_id)`; the top-level run/attempt is the parent batch and the nested workflow run/attempt is the child workflow identity, and both are validated separately (workflow SHA and product build SHA are bound independently and need not be equal). Missing, duplicate or extra cases, stale attempts, wrong digests, wrong platforms, wrong harness SHAs, zero tests, cancellation, timeout, untrusted sources and any non-PASS required item block with classified errors; only a complete legal success returns 0. `verify-promotion` re-checks the exact bytes about to be published against the passed list and fails on a swapped package or an edited plan after verification. preview, prerelease and stable all require an explicit plan; preview may use the plan's smaller pre-declared set but never bypasses artifact identity and startup checks. development artifacts are explicitly uncertified and never touch official or latest channels.
| Trigger | Workflow / job | Result |
|---|---|---|
| Nightly schedule, dispatch | `build.yml` `build-rustfs` (development build over the full platform matrix) | build artifacts and the dev download channel; no release publication |
| Valid release or preview tag | `build.yml` `build-rustfs`, `create-release`, `upload-release-assets`, `publish-release` | draft release, checksummed assets, publish |
| Successful non-preview release-tag build (`workflow_run`) | `docker.yml` `build-docker`, `scan-docker-image` | multi-architecture images and vulnerability report |
| Successful release-tag build (`workflow_run`) | `package.yml` `package` | DEB/RPM packages and checksums uploaded to the release |
| Successful non-preview release-tag build (`workflow_run`) | `helm-package.yml` `build-helm-package`, `publish-helm-package` | versioned chart and repository index |
| Nightly schedule, dispatch | `build.yml` `build-rustfs` (development build over the full platform matrix) | build artifacts and the uncertified dev download channel; no release publication |
| Valid release or preview tag | `build.yml` `build-rustfs`, `release-certification`, `create-release`, `upload-release-assets`, `publish-release`, `promote-release-artifacts` | staged binaries, draft release, checksummed assets, then certified publish and promotion of the same bytes |
| Successful non-preview release-tag build (`workflow_run`) | `docker.yml` `release-certification`, `build-docker`, `scan-docker-image` | multi-architecture images staged, verified against the certification and promoted, plus vulnerability report |
| Successful release-tag build (`workflow_run`) | `package.yml` `release-certification`, `package`, `promote-packages` | DEB/RPM packages with wrapped-binary verification, staged then promoted with checksums |
| Successful non-preview release-tag build (`workflow_run`) | `helm-package.yml` `release-certification`, `build-helm-package`, `publish-helm-package` | versioned chart and repository index, only over a certified image promotion record |
| Final tag's release published | `build.yml` `cleanup-preview-releases` | deletes every `<target>-preview.<N>` Release for that target; the tags are kept |
Use an exact preview tag for an end-to-end release rehearsal. Manual dispatches are backfill/debug paths and do not prove the automatic `workflow_run` chain.
Publication exits and their gate:
| Exit | Writer | Certification consumed |
|---|---|---|
| GitHub Release draft → public | `build.yml` `publish-release` | `release-certification` success and per-asset `verify-promotion` |
| Official R2 binary path `artifacts/rustfs/release/` | `build.yml` `promote-release-artifacts` | `release-certification` plus `verify-promotion`; the build job itself only writes `artifacts/rustfs/staging/<run>/<attempt>/` |
| `latest.json` version pointer | `build.yml` `update-latest-version` | transitively through `publish-release` |
| Image channels (Docker Hub, GHCR, Quay tags) | `docker.yml` `build-docker` promotion step | certified input digests, staged image binary identity and `verify-promotion`, then promotion of the same digest with `imagetools create` |
| Official R2 package paths `packages/release` and `packages/latest` | `package.yml` `promote-packages` | `release-certification` and wrapped-binary `verify-promotion`; the package job only stages |
| GitHub release package assets | `package.yml` `package` | `release-certification` and wrapped-binary `verify-promotion` before upload |
| Helm chart repository (`rustfs/helm` push) | `helm-package.yml` `publish-helm-package` | `release-certification` plus the Docker workflow's image-promotion record bound to the same candidate |
Downstream workflows fail closed when the trusted certification is absent: a manual dispatch without certification fails and pushes no official tag. Only results from the trusted verification sources listed in the plan (currently the `rustfs/s3chaos` `release-gate.yml` envelope runs) carry release authority; a PR-uploaded JSON set cannot certify a release. Retries reuse the same candidate, only the staged bytes are promoted (no rebuild-after-verify), and no downstream step re-resolves a mutable `latest`.
Use an exact preview tag for an end-to-end release rehearsal. Manual dispatches are backfill/debug paths and do not prove the automatic `workflow_run` chain. The committed fixtures under `testdata/release-gate/` and the commands below are contract rehearsals of this orchestration; they are explicitly not product release evidence and not live verifier results.
```bash
python3 -m unittest discover -s scripts/release -p 'test_release_gate.py'
python3 scripts/release/release_gate.py validate --plan testdata/release-gate/plan.json --candidate testdata/release-gate/candidate.json --results testdata/release-gate/pass --output target/release-gate-check
bash scripts/security/check_preview_release_workflow.sh
```
## Change checklist
@@ -0,0 +1,72 @@
# CI timing sample 2026-10-31 (backlog #2963 CI01 measurement)
Operational measurement of ordinary-code PR feedback and runner occupancy against the CI01 experimental targets (feedback median 15–20 min, P95 ≤ 30 min, paid occupancy median 50–60 runner minutes). Methodology: [ci-timing.md](ci-timing.md) and [ci-resource-accounting.md](ci-resource-accounting.md). Raw snapshots: `/tmp/ci-timing.json` (limit 30 reproduction), `/tmp/ci-timing-200.json` (analysis sample), `/tmp/ci-acc-w{1,3}.json`. Nothing in this report is an acceptance result; the targets are **not met** by the measured sample.
## Sample window and n
| Item | Value |
|---|---|
| Collection time | 2026-10-31 ~02:50–03:15 UTC |
| Window | completed `pull_request` runs of `ci.yml` created in the last 7 days |
| Runs sampled | 30 (reproduction) and 200 (analysis, script max) |
| Successful-code runs (classifier) | 1 of 30; **39 of 200** |
| Other outcomes in 200-run sample | 68 cancelled (34 %), 36 failed, remainder non-code greens/closure handlers |
| Lane shapes in sample | 24 combined-lane runs (`Workspace Test and Lint`), 15 split-lane runs (`Workspace Lint and Build` + `Workspace tests` + `Core contract`) |
The limit-30 reproduction reports median wall 76.85 min and median job-sum 133.33 runner-min on n=1 (per methodology, an absent/small median is not zero). All medians/P95 below are from the n=39 analysis sample. Two of the 39 are rerun continuations (run_attempt > 1, wall 7.0 and 22.6 min re-executing only failed jobs under the rerun-start convention); they do not move the median.
## Measured feedback and occupancy (ordinary successful code runs, n=39)
| Metric | Median | P90 | P95 | Max |
|---|---|---|---|---|
| Wall feedback (created/rerun-start → last job end) | **76.3 min** | 94.5 | **97.0** | 103.2 |
| Runner-minute job sum per run | **179.5** | 206.5 | **213.6** | 220.4 |
Independent accounting cross-check (`scripts/ci/actions_cost_report.py`, 2026-10-30T00–12Z window, 547 attempts, collection **incomplete** — see gaps): successful-code PR feedback median 88.34 min (n=12), runner_sum median 199.68 min (n=12). A complete half-day window (2026-10-31T00–12Z, 78 attempts) shows total known runner occupancy 1379.72 min with 42 jobs of unknown occupancy; queue wait summed 28.43 min, so queueing is not the dominant cost.
## Per-job critical path breakdown
Execution minutes across the 39 runs (known only):
| Job | n | Median | P95 |
|---|---|---|---|
| Workspace Test and Lint (combined lane) | 24 | 63.6 | 73.9 |
| Connect Feature Boundaries | 38 | 53.5 | 70.4 |
| Workspace Lint and Build (split lane) | 15 | 43.3 | 56.5 |
| Workspace tests (split lane) | 15 | 29.2 | 43.0 |
| Build RustFS Debug Binary | 39 | 18.5 | 24.9 |
| End-to-End Tests | 39 | 11.4 | 16.7 |
| S3 Implemented Tests | 39 | 9.8 | 11.1 |
| io_uring Integration (real) | 39 | 7.7 | 8.8 |
| Quick Checks | 39 | 7.2 | 9.2 |
Per-run chain timeline (start = attempt start; see `/tmp/per-run-jobs.json`):
- Split lane (15 runs): `Quick Checks` (~7–10) → `Workspace Lint and Build` (ends ~46–67) → `Workspace tests` (ends ~71–103) → shard evidence → aggregator `Test and Lint`/`required-checks`. The lint→tests chain sets the wall in most split runs.
- Combined lane (24 runs): `Quick Checks` → `Workspace Test and Lint` alone runs 63–86 min and sets the wall.
- `Connect Feature Boundaries` (median 53.5, P95 70.4) runs parallel to the lint chain and occasionally becomes the tail (e.g. runs 41021787700, 40750122159: connect end within 0.2 min of wall).
- `Build RustFS Debug Binary` → E2E/Core-contract/S3 chain finishes with 30–60 min of slack in every sampled run.
- The required-check aggregator `Test and Lint` (`required-checks`, `ci_gate.py verify`) finishes last by construction (~0.2 min).
## Against the CI01 targets
| Target | Measured | Verdict |
|---|---|---|
| Feedback median 15–20 min | 76.3 min (n=39) | **not met** (~3.8–5× over) |
| Feedback P95 ≤ 30 min | 97.0 min | **not met** (~3.2× over) |
| Occupancy median 50–60 runner-min | 179.5 runner-min | **not met** (~3–3.6× over) |
These are the experimental-target measurements. The full merge-lifecycle cost is separate and was not established here (the accounting PR-allocation buckets were empty in the partial window; see gaps).
## Bottlenecks (honest list)
1. **Workspace lint+test compile/test chain (critical path):** combined `Workspace Test and Lint` median 63.6 min, or split `Workspace Lint and Build` 43.3 + `Workspace tests` 29.2 ≈ 72.5 min after Quick Checks. This is the wall clock.
2. **Connect Feature Boundaries (parallel near-critical):** median 53.5 min on its own lane with `CARGO_BUILD_JOBS: "2"` limiting build concurrency on the credential-test step; becomes the tail when the lint chain is fast.
3. **Duplicate compilation across lanes:** `Workspace Lint and Build` builds workspace test binaries; `Build RustFS Debug Binary` separately builds the debug binary + E2E binaries on another runner; Connect builds again. `cache-save-if: 'false'` on the Connect lane means that lane never warms the shared `ci-dev` cache.
4. Runner queueing is small (28.43 queue-min vs 1379.72 runner-min in the complete half-day window) — not the bottleneck.
## Gaps and caveats
- Accounting windows: 2026-10-30T00–12Z collected 547 attempts but is flagged incomplete; 2026-10-30T12–24Z collection did not finish (timeout) and has no report; 2026-10-31T12–24Z was empty at collection time. Occupancy figures are per-window partial sums and must not be summed as a bill.
- No CI workflow or selection change was made in this pass; strict gate semantics unchanged. No savings are claimed. Candidate fixes (build-artifact reuse between lanes, enabling cache save on the Connect lane, finer affected-selection granularity in shadow mode per ci-timing.md) are **unmeasured proposals**, not results.
- Timing data alone does not establish functional coverage or escaped-regression rate; the 15–20 min target remains an experiment that currently misses by a wide margin.
+115 -12
View File
@@ -43,9 +43,9 @@ mod tests {
.map(|value| value.to_str().expect("allow-origin header must be valid UTF-8").to_string())
}
/// Runs a request through a router that returns a brotli-encoded response without
/// `Content-Length` and returns the response's `content-length` header value, if any.
async fn brotli_response_content_length(has_content_length: bool) -> Option<String> {
/// Sends a brotli-encoded response through `buffer_brotli_responses` and returns
/// the `Content-Length` header value and body bytes.
async fn run_brotli_middleware(body: &'static str, has_content_length: bool) -> (Option<String>, bytes::Bytes) {
use crate::admin::console::buffer_brotli_responses;
use axum::middleware;
use axum::{Router, body::Body, routing::get};
@@ -57,9 +57,9 @@ mod tests {
get(move || async move {
let mut builder = http::Response::builder().header(http::header::CONTENT_ENCODING, "br");
if has_content_length {
builder = builder.header(http::header::CONTENT_LENGTH, "4");
builder = builder.header(http::header::CONTENT_LENGTH, body.len().to_string());
}
builder.body(Body::from("data")).expect("response must build")
builder.body(Body::from(body)).expect("response must build")
}),
)
.layer(middleware::from_fn(buffer_brotli_responses));
@@ -74,27 +74,130 @@ mod tests {
.await
.expect("request must not fail");
response
let content_length = response
.headers()
.get(http::header::CONTENT_LENGTH)
.map(|value| value.to_str().expect("content-length header must be valid UTF-8").to_string())
.map(|value| value.to_str().expect("content-length header must be valid UTF-8").to_string());
let body_bytes = axum::body::to_bytes(response.into_body(), 1024 * 1024)
.await
.expect("body must be readable");
(content_length, body_bytes)
}
#[tokio::test]
#[serial]
async fn test_buffer_brotli_responses() {
// Brotli response without Content-Length must have Content-Length added.
assert!(
brotli_response_content_length(false).await.is_some(),
"brotli response without Content-Length must have Content-Length added"
// Brotli response without Content-Length must have Content-Length added with correct value.
let (content_length, body) = run_brotli_middleware("fake-brotli-data", false).await;
assert_eq!(
content_length.as_deref(),
Some("16"),
"brotli response without Content-Length must have Content-Length set to the body length"
);
assert_eq!(
body.as_ref(),
b"fake-brotli-data",
"body bytes must be preserved through the buffer round-trip"
);
// Brotli response that already carries Content-Length must pass through unchanged.
let (content_length, body) = run_brotli_middleware("data", true).await;
assert_eq!(
brotli_response_content_length(true).await.as_deref(),
content_length.as_deref(),
Some("4"),
"brotli response that already has Content-Length must not be re-buffered"
);
assert_eq!(body.as_ref(), b"data", "body bytes must be preserved when passing through unchanged");
}
#[tokio::test]
#[serial]
async fn test_buffer_brotli_responses_non_brotli_passthrough() {
use crate::admin::console::buffer_brotli_responses;
use axum::middleware;
use axum::{Router, body::Body, routing::get};
use tower::ServiceExt;
// Non-brotli response must pass through with no modification.
let app = Router::new()
.route(
"/",
get(|| async {
http::Response::builder()
.header(http::header::CONTENT_ENCODING, "gzip")
.body(Body::from("gzip-data"))
.expect("response must build")
}),
)
.layer(middleware::from_fn(buffer_brotli_responses));
let response = app
.oneshot(
http::Request::builder()
.uri("/")
.body(Body::empty())
.expect("request must build"),
)
.await
.expect("request must not fail");
assert_eq!(
response
.headers()
.get(http::header::CONTENT_ENCODING)
.and_then(|v| v.to_str().ok()),
Some("gzip"),
"non-brotli Content-Encoding must pass through unchanged"
);
let body_bytes = axum::body::to_bytes(response.into_body(), 1024 * 1024)
.await
.expect("body must be readable");
assert_eq!(body_bytes.as_ref(), b"gzip-data", "body bytes must pass through unchanged");
}
#[tokio::test]
#[serial]
async fn test_buffer_brotli_responses_body_stream_error() {
use crate::admin::console::buffer_brotli_responses;
use axum::middleware;
use axum::{Router, body::Body, routing::get};
use tower::ServiceExt;
// A brotli response whose body stream errors must have Content-Encoding stripped
// so the client does not attempt to decode an empty body as brotli.
let app = Router::new()
.route(
"/",
get(|| async {
let broken = futures::stream::once(async {
Err::<bytes::Bytes, std::io::Error>(std::io::Error::other("simulated stream error"))
});
http::Response::builder()
.header(http::header::CONTENT_ENCODING, "br")
.body(Body::from_stream(broken))
.expect("response must build")
}),
)
.layer(middleware::from_fn(buffer_brotli_responses));
let response = app
.oneshot(
http::Request::builder()
.uri("/")
.body(Body::empty())
.expect("request must build"),
)
.await
.expect("request must not fail");
assert!(
response.headers().get(http::header::CONTENT_ENCODING).is_none(),
"Content-Encoding must be stripped when body buffering fails so the client does not decode an empty body as brotli"
);
let body_bytes = axum::body::to_bytes(response.into_body(), 1024 * 1024)
.await
.expect("body must be readable");
assert!(body_bytes.is_empty(), "body must be empty after buffering failure");
}
#[tokio::test]
+51 -39
View File
@@ -39,6 +39,7 @@ use super::storage_api::multipart_usecase::contract::object::ObjectOperations as
use super::storage_api::multipart_usecase::contract::range::HTTPRangeSpec;
use super::storage_api::multipart_usecase::data_usage::{
quota_object_size, record_bucket_object_version_write_memory, record_bucket_object_write_memory,
record_bucket_object_write_unknown_previous_memory,
};
use super::storage_api::multipart_usecase::error::{
StorageError, is_err_invalid_upload_id, is_err_object_not_found, is_err_version_not_found,
@@ -690,25 +691,45 @@ impl DefaultMultipartUsecase {
return Err(S3Error::with_message(S3ErrorCode::InternalError, "Not init".to_string()));
};
let current_opts = internal_object_info_lookup_opts(
get_opts(&bucket, &key, None, None, &req.headers)
.await
.map_err(ApiError::from)?,
);
let quota_metadata_sys = self.bucket_metadata_sys();
let quota_tracking = quota_metadata_sys.is_some();
let mut quota_enabled = false;
if let Some(metadata_sys) = quota_metadata_sys.as_ref() {
let quota_checker = QuotaChecker::new(metadata_sys.clone());
let check_result =
map_quota_check_outcome(&bucket, quota_checker.check_quota(&bucket, QuotaOperation::PutObject, 0).await)?;
quota_enabled = check_result.quota_limit.is_some();
apply_quota_admission(&mut opts, &check_result)?;
}
let object_lock_config_state = Box::pin(load_bucket_object_lock_config_state(&bucket)).await?;
let previous_current_sizes = match store.get_object_info(&bucket, &key, &current_opts).await {
Ok(existing_obj_info) => {
validate_existing_object_lock_for_write(&object_lock_config_state, &existing_obj_info, &current_opts)?;
let physical_size = existing_obj_info.size.max(0) as u64;
let logical_size = quota_object_size(&existing_obj_info);
Some((physical_size, logical_size))
}
Err(err) => {
if !is_err_object_not_found(&err) && !is_err_version_not_found(&err) {
return Err(ApiError::from(err).into());
let prelookup_required = quota_enabled
|| (!super::object::shared::put_like_write_creates_new_version(&opts)
&& super::object::shared::object_lock_checks_required_for_state(&object_lock_config_state));
let prelookup_previous_current_size: Option<Option<u64>> = if prelookup_required {
let current_opts = internal_object_info_lookup_opts(
get_opts(&bucket, &key, None, None, &req.headers)
.await
.map_err(ApiError::from)?,
);
match store.get_object_info(&bucket, &key, &current_opts).await {
Ok(existing_obj_info) => {
validate_existing_object_lock_for_write(&object_lock_config_state, &existing_obj_info, &current_opts)?;
Some(Some(if quota_enabled {
quota_object_size(&existing_obj_info).map_err(ApiError::from)?
} else {
existing_obj_info.size.max(0) as u64
}))
}
Err(err) => {
if !is_err_object_not_found(&err) && !is_err_version_not_found(&err) {
return Err(ApiError::from(err).into());
}
Some(None)
}
None
}
} else {
None
};
// A lost Complete response retries the same upload id after staging is
@@ -746,24 +767,6 @@ impl DefaultMultipartUsecase {
let upload_user_defined = multipart_info.as_ref().map(|info| info.user_defined.clone());
let table_data_plane_publication_guards = retained_table_data_plane_publication_guards(&req);
let quota_metadata_sys = self.bucket_metadata_sys();
let quota_tracking = quota_metadata_sys.is_some();
let mut quota_enabled = false;
if let Some(metadata_sys) = quota_metadata_sys.as_ref() {
let quota_checker = QuotaChecker::new(metadata_sys.clone());
let check_result =
map_quota_check_outcome(&bucket, quota_checker.check_quota(&bucket, QuotaOperation::PutObject, 0).await)?;
quota_enabled = check_result.quota_limit.is_some();
apply_quota_admission(&mut opts, &check_result)?;
}
let previous_current_size = match previous_current_sizes {
Some((_, Ok(logical_size))) if quota_enabled => Some(logical_size),
Some((_, Err(err))) if quota_enabled => return Err(ApiError::from(err).into()),
Some((physical_size, _)) => Some(physical_size),
None => None,
};
// Replication admission is decided at the object-creation commit, not
// at multipart-session creation. Configuration and matching targets
// can change while parts are uploaded, so persist the exact decision's
@@ -831,11 +834,20 @@ impl DefaultMultipartUsecase {
if !obj_info.multipart_completion_replayed {
if quota_tracking {
let committed_size = quota_accounting_object_size(&obj_info, quota_enabled)?;
if versioned {
record_bucket_object_version_write_memory(&bucket, previous_current_size, committed_size).await;
} else {
record_bucket_object_write_memory(&bucket, previous_current_size, committed_size).await;
match prelookup_previous_current_size
.or_else(|| super::object::previous_current_size_from_backfill(obj_info.backfilled_old_current_size))
{
Some(previous_current_size) => {
if versioned {
record_bucket_object_version_write_memory(&bucket, previous_current_size, committed_size)
.await;
} else {
record_bucket_object_write_memory(&bucket, previous_current_size, committed_size).await;
}
}
None => {
record_bucket_object_write_unknown_previous_memory(&bucket, committed_size, versioned).await;
}
}
}
+4 -1
View File
@@ -230,7 +230,10 @@ pub(crate) use self::on_demand_migration_put::*;
#[cfg(test)]
pub(crate) use self::put::PollCountingBody;
use self::put::*;
pub(crate) use self::put::{guard_put_object_body_read_timeout, put_object_body_read_timeout, reject_oversize_single_upload};
pub(crate) use self::put::{
guard_put_object_body_read_timeout, previous_current_size_from_backfill, put_object_body_read_timeout,
reject_oversize_single_upload,
};
#[cfg(test)]
pub(crate) use self::restore::RestoreStatusCommitBarrier;
pub(crate) use self::shared::*;
+1 -1
View File
@@ -2324,7 +2324,7 @@ impl DefaultObjectUsecase {
/// `previous_current_size` value the usage-accounting helpers expect. Outer
/// `None` = unknown (no quorum agreement, or a peer predates the field) — the
/// caller must fall back to the degraded accounting path.
pub(super) fn previous_current_size_from_backfill(backfill: Option<OldCurrentSize>) -> Option<Option<u64>> {
pub(crate) fn previous_current_size_from_backfill(backfill: Option<OldCurrentSize>) -> Option<Option<u64>> {
backfill.map(|observation| match observation {
OldCurrentSize::Present(size) => Some(size.max(0) as u64),
OldCurrentSize::Absent => None,
+2 -2
View File
@@ -874,7 +874,7 @@ pub(crate) fn build_put_like_object_lock_metadata(
Ok(Some(eval_metadata))
}
pub(super) fn put_like_write_creates_new_version(opts: &ObjectOptions) -> bool {
pub(in crate::app) fn put_like_write_creates_new_version(opts: &ObjectOptions) -> bool {
opts.version_id.is_none() && opts.versioned && !opts.version_suspended
}
@@ -997,7 +997,7 @@ pub(crate) async fn object_lock_checks_required(bucket: &str) -> bool {
.map_or(true, |metadata| metadata.object_lock_checks_required())
}
pub(super) fn object_lock_checks_required_for_state(state: &metadata_sys::ObjectLockConfigState) -> bool {
pub(in crate::app) fn object_lock_checks_required_for_state(state: &metadata_sys::ObjectLockConfigState) -> bool {
match state {
metadata_sys::ObjectLockConfigState::Configured { .. } | metadata_sys::ObjectLockConfigState::Fabricated => true,
metadata_sys::ObjectLockConfigState::ConfirmedAbsent => false,
File diff suppressed because it is too large Load Diff
+376
View File
@@ -0,0 +1,376 @@
#!/usr/bin/env python3
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Contract tests for scripts/release/release_gate.py (rustfs/backlog#2662).
Every fixture under testdata/release-gate/ is a contract rehearsal only: these
tests prove the gate's fail-closed orchestration and must never be read as
product release evidence or as R02 live verification results.
"""
import json
import re
import subprocess
import sys
import tempfile
import unittest
from pathlib import Path
ROOT = Path(__file__).resolve().parents[2]
GATE = ROOT / "scripts" / "release" / "release_gate.py"
FIXTURES = ROOT / "testdata" / "release-gate"
PLAN = FIXTURES / "plan.json"
CANDIDATE = FIXTURES / "candidate.json"
ARTIFACT = FIXTURES / "artifacts" / "rustfs-linux-x86_64-gnu-v9.9.9-fixture.zip"
ARTIFACT_NAME = "rustfs-linux-x86_64-gnu-v9.9.9-fixture.zip"
REAL_PLAN = ROOT / ".config" / "release-validation-plan.json"
def run_gate(*arguments):
result = subprocess.run(
[sys.executable, str(GATE), *map(str, arguments)],
cwd=ROOT,
text=True,
capture_output=True,
check=False,
)
return result
def run_validate(results, output, plan=PLAN, candidate=CANDIDATE):
return run_gate(
"validate",
"--plan", plan,
"--candidate", candidate,
"--results", results,
"--output", output,
)
def run_promote(output, plan=PLAN, candidate=CANDIDATE, summary=None, artifact=ARTIFACT,
artifact_name=ARTIFACT_NAME, artifact_sha256=None):
arguments = [
"verify-promotion",
"--plan", plan,
"--candidate", candidate,
"--summary", summary or output,
]
if artifact is not None:
arguments += ["--artifact", artifact]
if artifact_name is not None:
arguments += ["--artifact-name", artifact_name]
if artifact_sha256 is not None:
arguments += ["--artifact-sha256", artifact_sha256]
if output is not None:
arguments += ["--output", output]
return run_gate(*arguments)
def load_summary(output):
return json.loads((Path(output) / "release-gate-summary.json").read_text())
def finding_codes(summary):
return {finding["code"] for finding in summary["findings"]}
class GateFixtureTests(unittest.TestCase):
"""Assertion matrix rows from rustfs/backlog#2662 exercised on the fixtures."""
def setUp(self):
self._temporary = tempfile.TemporaryDirectory(prefix="release-gate-test.")
self.addCleanup(self._temporary.cleanup)
self.output = Path(self._temporary.name)
def validate_fixture(self, name, expect):
output = self.output / name
result = run_validate(FIXTURES / name, output)
self.assertEqual(result.returncode, expect, result.stderr)
return result, load_summary(output)
def test_all_required_pass_and_identity_consistent(self):
# Matrix: all required PASS and identity consistent -> validate and
# verify-promotion return 0.
result, summary = self.validate_fixture("pass", 0)
self.assertEqual(summary["state"], "pass")
self.assertTrue(summary["publish_allowed"])
self.assertEqual(len(summary["required_cases"]), 5)
self.assertEqual(len(summary["observed"]), 5)
self.assertEqual(summary["findings"], [])
expansion = json.loads((self.output / "pass" / "plan-expansion.json").read_text())
self.assertEqual(len(expansion["required_cases"]), 5)
promote = run_promote(self.output / "pass")
self.assertEqual(promote.returncode, 0, promote.stderr)
def test_cross_platform_same_case_and_split_shas_accepted(self):
# Matrix: the same case name on two platforms, and a workflow SHA that
# differs from the (legally bound) product SHA, must not be rejected.
result, summary = self.validate_fixture("cross-platform", 0)
self.assertEqual(summary["state"], "pass")
product_commit = json.loads(CANDIDATE.read_text())["product_commit"]
workflows = []
for name in sorted((FIXTURES / "cross-platform").glob("*.json")):
if name.name == "batch.json":
continue
envelope = json.loads(name.read_text())
workflows.append(envelope["workflow"]["workflow_sha"])
self.assertNotEqual(envelope["workflow"]["workflow_sha"], product_commit)
self.assertEqual(len(workflows), 5)
case_names = [entry["case_id"] for entry in summary["observed"]]
self.assertEqual(case_names.count("startup"), 2)
self.assertEqual(case_names.count("artifact-identity"), 2)
def test_missing_case_names_the_case(self):
result, summary = self.validate_fixture("missing-case", 1)
self.assertIn("missing_case", finding_codes(summary))
missing = [f for f in summary["findings"] if f["code"] == "missing_case"]
self.assertTrue(any(f.get("case_id") == "startup" for f in missing), missing)
self.assertFalse(summary["publish_allowed"])
def test_duplicate_key_names_the_case(self):
result, summary = self.validate_fixture("duplicate-key", 1)
self.assertIn("duplicate_key", finding_codes(summary))
duplicates = [f for f in summary["findings"] if f["code"] == "duplicate_key"]
self.assertTrue(any(f.get("case_id") == "startup" for f in duplicates), duplicates)
self.assertFalse(summary["publish_allowed"])
def test_extra_unplanned_case_names_the_case(self):
result, summary = self.validate_fixture("extra-case", 1)
self.assertIn("extra_case", finding_codes(summary))
extra = [f for f in summary["findings"] if f["code"] == "extra_case"]
self.assertTrue(any(f.get("case_id") == "surprise-case" for f in extra), extra)
self.assertFalse(summary["publish_allowed"])
def test_non_pass_required_items_block(self):
# Matrix: any skipped / infra_error / quarantined required item blocks
# and no official publish write is allowed.
for name in ("non-pass-skipped", "non-pass-infra-error", "non-pass-quarantined"):
with self.subTest(name=name):
result, summary = self.validate_fixture(name, 1)
self.assertIn("non_pass", finding_codes(summary))
self.assertFalse(summary["publish_allowed"])
def test_child_workflow_cancellation_blocks(self):
result, summary = self.validate_fixture("child-cancelled", 1)
self.assertIn("cancelled", finding_codes(summary))
self.assertFalse(summary["publish_allowed"])
def test_batch_timeout_blocks(self):
result, summary = self.validate_fixture("timeout", 1)
self.assertIn("timeout", finding_codes(summary))
self.assertFalse(summary["publish_allowed"])
def test_same_tag_different_bytes_is_rejected(self):
result, summary = self.validate_fixture("same-tag-different-bytes", 1)
self.assertIn("digest_mismatch", finding_codes(summary))
self.assertFalse(summary["publish_allowed"])
def test_stale_attempt_cannot_be_reused(self):
result, summary = self.validate_fixture("stale-attempt", 1)
self.assertIn("stale_attempt", finding_codes(summary))
self.assertFalse(summary["publish_allowed"])
def test_wrong_platform_is_rejected(self):
result, summary = self.validate_fixture("wrong-platform", 1)
self.assertIn("platform_mismatch", finding_codes(summary))
self.assertFalse(summary["publish_allowed"])
def test_wrong_harness_sha_cannot_be_reused(self):
result, summary = self.validate_fixture("wrong-harness", 1)
self.assertIn("harness_mismatch", finding_codes(summary))
self.assertFalse(summary["publish_allowed"])
def test_zero_tests_block(self):
result, summary = self.validate_fixture("zero-tests", 1)
self.assertIn("zero_tests", finding_codes(summary))
self.assertFalse(summary["publish_allowed"])
def test_untrusted_source_blocks(self):
# A PR-uploaded or foreign JSON set must never gain release authority.
result, summary = self.validate_fixture("untrusted-source", 1)
self.assertIn("untrusted_source", finding_codes(summary))
self.assertFalse(summary["publish_allowed"])
def test_swapped_package_after_verification_fails_promotion(self):
# Matrix: swapping a published package after verification must fail
# verify-promotion.
output = self.output / "certified"
self.assertEqual(run_validate(FIXTURES / "pass", output).returncode, 0)
promote = run_promote(
output,
artifact=FIXTURES / "promotion-swap" / ARTIFACT_NAME,
summary=output,
)
self.assertEqual(promote.returncode, 1, promote.stdout)
self.assertIn("artifact_mismatch", promote.stderr)
report = json.loads((output / "promotion-check.json").read_text())
self.assertEqual(report["state"], "blocked")
self.assertEqual(finding_codes(report), {"artifact_mismatch"})
def test_edited_plan_after_verification_fails_promotion(self):
# Matrix: editing the plan after verification must fail
# verify-promotion; the old certification cannot be reused.
output = self.output / "certified"
self.assertEqual(run_validate(FIXTURES / "pass", output).returncode, 0)
promote = run_promote(
output,
plan=FIXTURES / "promotion-edit" / "plan.json",
summary=output,
)
self.assertEqual(promote.returncode, 1, promote.stdout)
self.assertIn("plan_edited", promote.stderr)
def test_promotion_without_certification_fails(self):
output = self.output / "blocked"
self.assertEqual(run_validate(FIXTURES / "missing-case", output).returncode, 1)
promote = run_promote(output, summary=output)
self.assertEqual(promote.returncode, 1, promote.stdout)
self.assertIn("certification_missing", promote.stderr)
def test_summary_junit_and_expansion_are_written(self):
output = self.output / "pass"
self.assertEqual(run_validate(FIXTURES / "pass", output).returncode, 0)
summary = load_summary(output)
self.assertEqual(summary["plan_sha256"], json.loads(CANDIDATE.read_text())["plan_sha256"])
junit = (output / "release-gate.junit.xml").read_text()
self.assertIn("release-gate", junit)
self.assertIn("identity/linux-x86_64-gnu/startup", junit)
markdown = (output / "release-gate-summary.md").read_text()
self.assertIn("state: pass", markdown)
def test_error_messages_carry_no_credentials(self):
for name in ("missing-case", "stale-attempt", "untrusted-source", "zero-tests"):
output = self.output / name
run_validate(FIXTURES / name, output)
text = output.joinpath("release-gate-summary.json").read_text()
text += (output / "release-gate-summary.md").read_text()
lowered = text.lower()
for marker in ("secret", "password", "access_key", "akia", "bearer "):
self.assertNotIn(marker, lowered, f"{name} leaked a credential-shaped string")
class PlanContractTests(unittest.TestCase):
def test_real_plan_is_explicit_about_sources_and_rollback(self):
sys.path.insert(0, str(GATE.parent))
import release_gate
plan = json.loads(REAL_PLAN.read_text())
info = release_gate.validate_plan(plan)
sources = info["upgrade_sources"]
self.assertGreaterEqual(len(sources["sources"]), 1)
self.assertFalse(sources["rollback_supported"])
self.assertTrue(sources["basis"].strip())
self.assertTrue(sources["rollback_basis"].strip())
for source in sources["sources"]:
self.assertEqual(len(source["artifact_sha256"]), 64)
def test_plan_requires_trusted_workflows_and_frozen_harness(self):
sys.path.insert(0, str(GATE.parent))
import release_gate
plan = json.loads(REAL_PLAN.read_text())
plan["trusted_workflows"] = []
with self.assertRaises(release_gate.GateError):
release_gate.validate_plan(plan)
plan = json.loads(REAL_PLAN.read_text())
plan["lanes"][0]["harness_sha"] = "not-a-sha"
with self.assertRaises(release_gate.GateError):
release_gate.validate_plan(plan)
plan = json.loads(REAL_PLAN.read_text())
plan["upgrade_sources"].pop("rollback_supported")
with self.assertRaises(release_gate.GateError):
release_gate.validate_plan(plan)
def test_plan_sha256_is_semantic_and_detects_required_set_edits(self):
sys.path.insert(0, str(GATE.parent))
import release_gate
plan = json.loads(PLAN.read_text())
baseline = release_gate.canonical_sha256(plan)
self.assertEqual(release_gate.canonical_sha256(json.loads(json.dumps(plan))), baseline)
edited = json.loads(PLAN.read_text())
edited["lanes"][0]["cases"] = ["artifact-identity"]
self.assertNotEqual(release_gate.canonical_sha256(edited), baseline)
class PublicationWiringTests(unittest.TestCase):
"""Static checks that the real publication exits consume the gate."""
def read(self, relative):
return (ROOT / relative).read_text()
def job_block(self, workflow, job):
match = re.search(
rf"^ {re.escape(job)}:\n(.*?)(?=^ [A-Za-z#])",
workflow,
re.S | re.M,
)
self.assertIsNotNone(match, f"job {job} not found")
return match.group(1)
def test_build_publish_requires_certification(self):
workflow = self.read(".github/workflows/build.yml")
self.assertIn("\n release-certification:", workflow)
publish_block = self.job_block(workflow, "publish-release")
self.assertIn("release-certification", publish_block)
self.assertIn("release_gate.py validate", workflow)
self.assertIn("release_gate.py verify-promotion", workflow)
def test_build_time_official_r2_writes_are_staged(self):
workflow = self.read(".github/workflows/build.yml")
self.assertIn("artifacts/rustfs/staging/", workflow)
release_writes = [
line for line in workflow.splitlines()
if "artifacts/rustfs/release/" in line
]
# Only the certification-gated promotion job may write the official path.
self.assertTrue(release_writes, "the promotion job must write the official R2 path")
promotion_block = self.job_block(workflow, "promote-release-artifacts")
self.assertIn("verify-promotion", promotion_block)
self.assertIn("artifacts/rustfs/release/", promotion_block)
build_block = self.job_block(workflow, "build-rustfs")
self.assertNotIn('R2_PATH="s3://${R2_BUCKET}/artifacts/rustfs/release/"', build_block)
def test_downstream_workflows_verify_certification(self):
docker = self.read(".github/workflows/docker.yml")
self.assertIn("\n release-certification:", docker)
build_docker_block = self.job_block(docker, "build-docker")
self.assertIn("release-certification", build_docker_block)
self.assertIn("verify-promotion", docker)
package = self.read(".github/workflows/package.yml")
self.assertIn("\n release-certification:", package)
package_block = self.job_block(package, "package")
self.assertIn("release-certification", package_block)
helm = self.read(".github/workflows/helm-package.yml")
self.assertIn("\n release-certification:", helm)
helm_block = self.job_block(helm, "build-helm-package")
self.assertIn("release-certification", helm_block)
def test_manual_downstream_dispatch_without_certification_fails(self):
# Every downstream certification job must fail closed when the trusted
# certification artifact is absent (e.g. a manual dispatch without one).
for relative in (
".github/workflows/docker.yml",
".github/workflows/package.yml",
".github/workflows/helm-package.yml",
):
workflow = self.read(relative)
certification_block = self.job_block(workflow, "release-certification")
self.assertIn("fail", certification_block.lower(), relative)
self.assertIn("release-certification", certification_block, relative)
if __name__ == "__main__":
unittest.main()
+14 -7
View File
@@ -27,7 +27,7 @@ cleanup() {
}
trap cleanup EXIT INT TERM
for command in curl docker python3 awscurl; do
for command in curl docker python3 aws; do
command -v "${command}" >/dev/null || {
echo "missing required command: ${command}" >&2
exit 1
@@ -192,12 +192,19 @@ print("\t".join(values[field] for field in ("AccessKeyId", "SecretAccessKey", "S
PY
)
awscurl --fail-with-body --service s3 --region us-east-1 \
--access_key "${STS_ACCESS_KEY}" \
--secret_key "${STS_SECRET_KEY}" \
--security_token "${STS_SESSION_TOKEN}" \
"${RUSTFS_ORIGIN}/" >"${WORK_DIR}/list-buckets.xml"
grep -q '<ListAllMyBucketsResult' "${WORK_DIR}/list-buckets.xml"
# Sign with the STS session to prove the web-identity credentials are usable.
# The AWS CLI replaces awscurl here: the functional fleet's system Python is
# PEP 668 managed, so the live gate must not require pip-installed tools. The
# security suite already requires the AWS CLI. Strip proxy env explicitly
# (the runners define it) so this localhost request never leaves the host.
env -u http_proxy -u https_proxy -u HTTP_PROXY -u HTTPS_PROXY \
AWS_ACCESS_KEY_ID="${STS_ACCESS_KEY}" \
AWS_SECRET_ACCESS_KEY="${STS_SECRET_KEY}" \
AWS_SESSION_TOKEN="${STS_SESSION_TOKEN}" \
AWS_EC2_METADATA_DISABLED=true \
aws --endpoint-url "${RUSTFS_ORIGIN}" --region us-east-1 \
s3api list-buckets --output json >"${WORK_DIR}/list-buckets.json"
grep -q '"Buckets"' "${WORK_DIR}/list-buckets.json"
TAMPERED_TOKEN="$(python3 - "${GOOD_TOKEN}" <<'PY'
import sys
+29
View File
@@ -0,0 +1,29 @@
{
"schema_version": 1,
"product_commit": "3333333333333333333333333333333333333333",
"version": "9.9.9-fixture",
"release_profile": "stable",
"features": [
"pyroscope"
],
"plan_sha256": "578bb57e65df8ba7a63e40aafc254fa644163270b7ec288fe3368bcd7c10ff86",
"validation_batch": {
"run_id": "fixture-batch-1",
"run_attempt": 2
},
"deliverables": [
{
"kind": "zip",
"name": "rustfs-linux-x86_64-gnu-v9.9.9-fixture.zip",
"location": "https://example.invalid/artifacts/runs/fixture-batch-1/2/rustfs-linux-x86_64-gnu-v9.9.9-fixture.zip",
"sha256": "ea3ac1fb0ec4aa8fbec650f64ccdbc2543843c8d5734c3915d1b57d7cbb92ec8",
"size": 36,
"binary_sha256": "7b77e78cc7b8f63a60f8a7101bebed6502f82e03016f4d4e11450f927e0a52fa"
}
],
"image": {
"ref": "example.invalid/rustfs:9.9.9-fixture",
"digest": "sha256:5555555555555555555555555555555555555555555555555555555555555555",
"binary_sha256": "7b77e78cc7b8f63a60f8a7101bebed6502f82e03016f4d4e11450f927e0a52fa"
}
}
+6
View File
@@ -0,0 +1,6 @@
{
"schema_version": 1,
"run_id": "fixture-batch-1",
"run_attempt": 2,
"status": "completed"
}
@@ -0,0 +1,42 @@
{
"schema_version": 1,
"evidence_profile": "release-artifact",
"run_id": "fixture-batch-1",
"run_attempt": 2,
"plan_sha256": "578bb57e65df8ba7a63e40aafc254fa644163270b7ec288fe3368bcd7c10ff86",
"tier": "release-certification",
"lane": "identity",
"case_id": "artifact-identity",
"product_commit": "3333333333333333333333333333333333333333",
"harness_commit": "1111111111111111111111111111111111111111",
"target_fingerprint": "rustfs-9.9.9-fixture-linux-aarch64-gnu",
"platform": {
"os": "linux",
"arch": "aarch64",
"abi": "gnu"
},
"started_at": "2026-09-26T10:02:00Z",
"finished_at": "2026-09-26T10:02:30Z",
"state": "pass",
"reason_code": "ok",
"reason_class": "NOT_APPLICABLE",
"evidence": [
{
"path": "cases/identity/artifact-identity-linux-aarch64-gnu.json",
"sha256": "fa138d5891e5098b97137b0edbb4779c4350285fdc76cec7e6214809b34b3075",
"size": 64
}
],
"workflow": {
"repository": "rustfs/s3chaos",
"workflow_id": "release-gate.yml",
"run_id": "child-identity-linux-aarch64-gnu",
"run_attempt": 1,
"workflow_sha": "1111111111111111111111111111111111111111"
},
"artifact": {
"archive_sha256": "ea3ac1fb0ec4aa8fbec650f64ccdbc2543843c8d5734c3915d1b57d7cbb92ec8",
"binary_sha256": "7b77e78cc7b8f63a60f8a7101bebed6502f82e03016f4d4e11450f927e0a52fa",
"container_image_digest": "sha256:5555555555555555555555555555555555555555555555555555555555555555"
}
}
@@ -0,0 +1,42 @@
{
"schema_version": 1,
"evidence_profile": "release-artifact",
"run_id": "fixture-batch-1",
"run_attempt": 2,
"plan_sha256": "578bb57e65df8ba7a63e40aafc254fa644163270b7ec288fe3368bcd7c10ff86",
"tier": "release-certification",
"lane": "identity",
"case_id": "startup",
"product_commit": "3333333333333333333333333333333333333333",
"harness_commit": "1111111111111111111111111111111111111111",
"target_fingerprint": "rustfs-9.9.9-fixture-linux-aarch64-gnu",
"platform": {
"os": "linux",
"arch": "aarch64",
"abi": "gnu"
},
"started_at": "2026-09-26T10:03:00Z",
"finished_at": "2026-09-26T10:03:30Z",
"state": "pass",
"reason_code": "ok",
"reason_class": "NOT_APPLICABLE",
"evidence": [
{
"path": "cases/identity/startup-linux-aarch64-gnu.json",
"sha256": "314b19c4fee5f406cbf7ee0d6a436059115f1ff06f6864830045f9d452ab91c9",
"size": 64
}
],
"workflow": {
"repository": "rustfs/s3chaos",
"workflow_id": "release-gate.yml",
"run_id": "child-identity-linux-aarch64-gnu",
"run_attempt": 1,
"workflow_sha": "1111111111111111111111111111111111111111"
},
"artifact": {
"archive_sha256": "ea3ac1fb0ec4aa8fbec650f64ccdbc2543843c8d5734c3915d1b57d7cbb92ec8",
"binary_sha256": "7b77e78cc7b8f63a60f8a7101bebed6502f82e03016f4d4e11450f927e0a52fa",
"container_image_digest": "sha256:5555555555555555555555555555555555555555555555555555555555555555"
}
}
@@ -0,0 +1,42 @@
{
"schema_version": 1,
"evidence_profile": "release-artifact",
"run_id": "fixture-batch-1",
"run_attempt": 2,
"plan_sha256": "578bb57e65df8ba7a63e40aafc254fa644163270b7ec288fe3368bcd7c10ff86",
"tier": "release-certification",
"lane": "identity",
"case_id": "artifact-identity",
"product_commit": "3333333333333333333333333333333333333333",
"harness_commit": "1111111111111111111111111111111111111111",
"target_fingerprint": "rustfs-9.9.9-fixture-linux-x86_64-gnu",
"platform": {
"os": "linux",
"arch": "x86_64",
"abi": "gnu"
},
"started_at": "2026-09-26T10:00:00Z",
"finished_at": "2026-09-26T10:00:30Z",
"state": "pass",
"reason_code": "ok",
"reason_class": "NOT_APPLICABLE",
"evidence": [
{
"path": "cases/identity/artifact-identity-linux-x86_64-gnu.json",
"sha256": "37e58ec3fc6fe3ac7ff2fcbeebdb62f6a6aaaa34ec771a0f3306c821da0336f7",
"size": 64
}
],
"workflow": {
"repository": "rustfs/s3chaos",
"workflow_id": "release-gate.yml",
"run_id": "child-identity-linux-x86_64-gnu",
"run_attempt": 1,
"workflow_sha": "1111111111111111111111111111111111111111"
},
"artifact": {
"archive_sha256": "ea3ac1fb0ec4aa8fbec650f64ccdbc2543843c8d5734c3915d1b57d7cbb92ec8",
"binary_sha256": "7b77e78cc7b8f63a60f8a7101bebed6502f82e03016f4d4e11450f927e0a52fa",
"container_image_digest": "sha256:5555555555555555555555555555555555555555555555555555555555555555"
}
}
@@ -0,0 +1,43 @@
{
"schema_version": 1,
"evidence_profile": "release-artifact",
"run_id": "fixture-batch-1",
"run_attempt": 2,
"plan_sha256": "578bb57e65df8ba7a63e40aafc254fa644163270b7ec288fe3368bcd7c10ff86",
"tier": "release-certification",
"lane": "identity",
"case_id": "startup",
"product_commit": "3333333333333333333333333333333333333333",
"harness_commit": "1111111111111111111111111111111111111111",
"target_fingerprint": "rustfs-9.9.9-fixture-linux-x86_64-gnu",
"platform": {
"os": "linux",
"arch": "x86_64",
"abi": "gnu"
},
"started_at": "2026-09-26T10:01:00Z",
"finished_at": "2026-09-26T10:01:30Z",
"state": "infra_error",
"reason_code": "child-cancelled",
"reason_class": "ENV",
"evidence": [
{
"path": "cases/identity/startup-linux-x86_64-gnu.json",
"sha256": "8b8774175c55c400b337610a89fe725ad2637af8fcc4973a82ab218f9182b80d",
"size": 64
}
],
"workflow": {
"repository": "rustfs/s3chaos",
"workflow_id": "release-gate.yml",
"run_id": "child-identity-linux-x86_64-gnu",
"run_attempt": 1,
"workflow_sha": "1111111111111111111111111111111111111111",
"status": "cancelled"
},
"artifact": {
"archive_sha256": "ea3ac1fb0ec4aa8fbec650f64ccdbc2543843c8d5734c3915d1b57d7cbb92ec8",
"binary_sha256": "7b77e78cc7b8f63a60f8a7101bebed6502f82e03016f4d4e11450f927e0a52fa",
"container_image_digest": "sha256:5555555555555555555555555555555555555555555555555555555555555555"
}
}
@@ -0,0 +1,42 @@
{
"schema_version": 1,
"evidence_profile": "release-artifact",
"run_id": "fixture-batch-1",
"run_attempt": 2,
"plan_sha256": "578bb57e65df8ba7a63e40aafc254fa644163270b7ec288fe3368bcd7c10ff86",
"tier": "release-certification",
"lane": "upgrade",
"case_id": "upgrade-smoke",
"product_commit": "3333333333333333333333333333333333333333",
"harness_commit": "2222222222222222222222222222222222222222",
"target_fingerprint": "rustfs-9.9.9-fixture-linux-x86_64-gnu",
"platform": {
"os": "linux",
"arch": "x86_64",
"abi": "gnu"
},
"started_at": "2026-09-26T10:04:00Z",
"finished_at": "2026-09-26T10:04:30Z",
"state": "pass",
"reason_code": "ok",
"reason_class": "NOT_APPLICABLE",
"evidence": [
{
"path": "cases/upgrade/upgrade-smoke-linux-x86_64-gnu.json",
"sha256": "c1ce56808b52859d2590b34b5990aed7bf560b134e44bd0c09f1873c9f764b38",
"size": 64
}
],
"workflow": {
"repository": "rustfs/s3chaos",
"workflow_id": "release-gate.yml",
"run_id": "child-upgrade-linux-x86_64-gnu",
"run_attempt": 1,
"workflow_sha": "2222222222222222222222222222222222222222"
},
"artifact": {
"archive_sha256": "ea3ac1fb0ec4aa8fbec650f64ccdbc2543843c8d5734c3915d1b57d7cbb92ec8",
"binary_sha256": "7b77e78cc7b8f63a60f8a7101bebed6502f82e03016f4d4e11450f927e0a52fa",
"container_image_digest": "sha256:5555555555555555555555555555555555555555555555555555555555555555"
}
}
+6
View File
@@ -0,0 +1,6 @@
{
"schema_version": 1,
"run_id": "fixture-batch-1",
"run_attempt": 2,
"status": "completed"
}
@@ -0,0 +1,42 @@
{
"schema_version": 1,
"evidence_profile": "release-artifact",
"run_id": "fixture-batch-1",
"run_attempt": 2,
"plan_sha256": "578bb57e65df8ba7a63e40aafc254fa644163270b7ec288fe3368bcd7c10ff86",
"tier": "release-certification",
"lane": "identity",
"case_id": "artifact-identity",
"product_commit": "3333333333333333333333333333333333333333",
"harness_commit": "1111111111111111111111111111111111111111",
"target_fingerprint": "rustfs-9.9.9-fixture-linux-aarch64-gnu",
"platform": {
"os": "linux",
"arch": "aarch64",
"abi": "gnu"
},
"started_at": "2026-09-26T10:02:00Z",
"finished_at": "2026-09-26T10:02:30Z",
"state": "pass",
"reason_code": "ok",
"reason_class": "NOT_APPLICABLE",
"evidence": [
{
"path": "cases/identity/artifact-identity-linux-aarch64-gnu.json",
"sha256": "fa138d5891e5098b97137b0edbb4779c4350285fdc76cec7e6214809b34b3075",
"size": 64
}
],
"workflow": {
"repository": "rustfs/s3chaos",
"workflow_id": "release-gate.yml",
"run_id": "child-cross-identity-linux-aarch64-gnu-artifact-identity",
"run_attempt": 1,
"workflow_sha": "4444444444444444444444444444444444444444"
},
"artifact": {
"archive_sha256": "ea3ac1fb0ec4aa8fbec650f64ccdbc2543843c8d5734c3915d1b57d7cbb92ec8",
"binary_sha256": "7b77e78cc7b8f63a60f8a7101bebed6502f82e03016f4d4e11450f927e0a52fa",
"container_image_digest": "sha256:5555555555555555555555555555555555555555555555555555555555555555"
}
}
@@ -0,0 +1,42 @@
{
"schema_version": 1,
"evidence_profile": "release-artifact",
"run_id": "fixture-batch-1",
"run_attempt": 2,
"plan_sha256": "578bb57e65df8ba7a63e40aafc254fa644163270b7ec288fe3368bcd7c10ff86",
"tier": "release-certification",
"lane": "identity",
"case_id": "startup",
"product_commit": "3333333333333333333333333333333333333333",
"harness_commit": "1111111111111111111111111111111111111111",
"target_fingerprint": "rustfs-9.9.9-fixture-linux-aarch64-gnu",
"platform": {
"os": "linux",
"arch": "aarch64",
"abi": "gnu"
},
"started_at": "2026-09-26T10:03:00Z",
"finished_at": "2026-09-26T10:03:30Z",
"state": "pass",
"reason_code": "ok",
"reason_class": "NOT_APPLICABLE",
"evidence": [
{
"path": "cases/identity/startup-linux-aarch64-gnu.json",
"sha256": "314b19c4fee5f406cbf7ee0d6a436059115f1ff06f6864830045f9d452ab91c9",
"size": 64
}
],
"workflow": {
"repository": "rustfs/s3chaos",
"workflow_id": "release-gate.yml",
"run_id": "child-cross-identity-linux-aarch64-gnu-startup",
"run_attempt": 1,
"workflow_sha": "4444444444444444444444444444444444444444"
},
"artifact": {
"archive_sha256": "ea3ac1fb0ec4aa8fbec650f64ccdbc2543843c8d5734c3915d1b57d7cbb92ec8",
"binary_sha256": "7b77e78cc7b8f63a60f8a7101bebed6502f82e03016f4d4e11450f927e0a52fa",
"container_image_digest": "sha256:5555555555555555555555555555555555555555555555555555555555555555"
}
}
@@ -0,0 +1,42 @@
{
"schema_version": 1,
"evidence_profile": "release-artifact",
"run_id": "fixture-batch-1",
"run_attempt": 2,
"plan_sha256": "578bb57e65df8ba7a63e40aafc254fa644163270b7ec288fe3368bcd7c10ff86",
"tier": "release-certification",
"lane": "identity",
"case_id": "artifact-identity",
"product_commit": "3333333333333333333333333333333333333333",
"harness_commit": "1111111111111111111111111111111111111111",
"target_fingerprint": "rustfs-9.9.9-fixture-linux-x86_64-gnu",
"platform": {
"os": "linux",
"arch": "x86_64",
"abi": "gnu"
},
"started_at": "2026-09-26T10:00:00Z",
"finished_at": "2026-09-26T10:00:30Z",
"state": "pass",
"reason_code": "ok",
"reason_class": "NOT_APPLICABLE",
"evidence": [
{
"path": "cases/identity/artifact-identity-linux-x86_64-gnu.json",
"sha256": "37e58ec3fc6fe3ac7ff2fcbeebdb62f6a6aaaa34ec771a0f3306c821da0336f7",
"size": 64
}
],
"workflow": {
"repository": "rustfs/s3chaos",
"workflow_id": "release-gate.yml",
"run_id": "child-cross-identity-linux-x86_64-gnu-artifact-identity",
"run_attempt": 1,
"workflow_sha": "4444444444444444444444444444444444444444"
},
"artifact": {
"archive_sha256": "ea3ac1fb0ec4aa8fbec650f64ccdbc2543843c8d5734c3915d1b57d7cbb92ec8",
"binary_sha256": "7b77e78cc7b8f63a60f8a7101bebed6502f82e03016f4d4e11450f927e0a52fa",
"container_image_digest": "sha256:5555555555555555555555555555555555555555555555555555555555555555"
}
}
@@ -0,0 +1,42 @@
{
"schema_version": 1,
"evidence_profile": "release-artifact",
"run_id": "fixture-batch-1",
"run_attempt": 2,
"plan_sha256": "578bb57e65df8ba7a63e40aafc254fa644163270b7ec288fe3368bcd7c10ff86",
"tier": "release-certification",
"lane": "identity",
"case_id": "startup",
"product_commit": "3333333333333333333333333333333333333333",
"harness_commit": "1111111111111111111111111111111111111111",
"target_fingerprint": "rustfs-9.9.9-fixture-linux-x86_64-gnu",
"platform": {
"os": "linux",
"arch": "x86_64",
"abi": "gnu"
},
"started_at": "2026-09-26T10:01:00Z",
"finished_at": "2026-09-26T10:01:30Z",
"state": "pass",
"reason_code": "ok",
"reason_class": "NOT_APPLICABLE",
"evidence": [
{
"path": "cases/identity/startup-linux-x86_64-gnu.json",
"sha256": "8b8774175c55c400b337610a89fe725ad2637af8fcc4973a82ab218f9182b80d",
"size": 64
}
],
"workflow": {
"repository": "rustfs/s3chaos",
"workflow_id": "release-gate.yml",
"run_id": "child-cross-identity-linux-x86_64-gnu-startup",
"run_attempt": 1,
"workflow_sha": "4444444444444444444444444444444444444444"
},
"artifact": {
"archive_sha256": "ea3ac1fb0ec4aa8fbec650f64ccdbc2543843c8d5734c3915d1b57d7cbb92ec8",
"binary_sha256": "7b77e78cc7b8f63a60f8a7101bebed6502f82e03016f4d4e11450f927e0a52fa",
"container_image_digest": "sha256:5555555555555555555555555555555555555555555555555555555555555555"
}
}
@@ -0,0 +1,42 @@
{
"schema_version": 1,
"evidence_profile": "release-artifact",
"run_id": "fixture-batch-1",
"run_attempt": 2,
"plan_sha256": "578bb57e65df8ba7a63e40aafc254fa644163270b7ec288fe3368bcd7c10ff86",
"tier": "release-certification",
"lane": "upgrade",
"case_id": "upgrade-smoke",
"product_commit": "3333333333333333333333333333333333333333",
"harness_commit": "2222222222222222222222222222222222222222",
"target_fingerprint": "rustfs-9.9.9-fixture-linux-x86_64-gnu",
"platform": {
"os": "linux",
"arch": "x86_64",
"abi": "gnu"
},
"started_at": "2026-09-26T10:04:00Z",
"finished_at": "2026-09-26T10:04:30Z",
"state": "pass",
"reason_code": "ok",
"reason_class": "NOT_APPLICABLE",
"evidence": [
{
"path": "cases/upgrade/upgrade-smoke-linux-x86_64-gnu.json",
"sha256": "c1ce56808b52859d2590b34b5990aed7bf560b134e44bd0c09f1873c9f764b38",
"size": 64
}
],
"workflow": {
"repository": "rustfs/s3chaos",
"workflow_id": "release-gate.yml",
"run_id": "child-cross-upgrade-linux-x86_64-gnu-upgrade-smoke",
"run_attempt": 1,
"workflow_sha": "4444444444444444444444444444444444444444"
},
"artifact": {
"archive_sha256": "ea3ac1fb0ec4aa8fbec650f64ccdbc2543843c8d5734c3915d1b57d7cbb92ec8",
"binary_sha256": "7b77e78cc7b8f63a60f8a7101bebed6502f82e03016f4d4e11450f927e0a52fa",
"container_image_digest": "sha256:5555555555555555555555555555555555555555555555555555555555555555"
}
}
+6
View File
@@ -0,0 +1,6 @@
{
"schema_version": 1,
"run_id": "fixture-batch-1",
"run_attempt": 2,
"status": "completed"
}
@@ -0,0 +1,42 @@
{
"schema_version": 1,
"evidence_profile": "release-artifact",
"run_id": "fixture-batch-1",
"run_attempt": 2,
"plan_sha256": "578bb57e65df8ba7a63e40aafc254fa644163270b7ec288fe3368bcd7c10ff86",
"tier": "release-certification",
"lane": "identity",
"case_id": "artifact-identity",
"product_commit": "3333333333333333333333333333333333333333",
"harness_commit": "1111111111111111111111111111111111111111",
"target_fingerprint": "rustfs-9.9.9-fixture-linux-aarch64-gnu",
"platform": {
"os": "linux",
"arch": "aarch64",
"abi": "gnu"
},
"started_at": "2026-09-26T10:02:00Z",
"finished_at": "2026-09-26T10:02:30Z",
"state": "pass",
"reason_code": "ok",
"reason_class": "NOT_APPLICABLE",
"evidence": [
{
"path": "cases/identity/artifact-identity-linux-aarch64-gnu.json",
"sha256": "fa138d5891e5098b97137b0edbb4779c4350285fdc76cec7e6214809b34b3075",
"size": 64
}
],
"workflow": {
"repository": "rustfs/s3chaos",
"workflow_id": "release-gate.yml",
"run_id": "child-identity-linux-aarch64-gnu",
"run_attempt": 1,
"workflow_sha": "1111111111111111111111111111111111111111"
},
"artifact": {
"archive_sha256": "ea3ac1fb0ec4aa8fbec650f64ccdbc2543843c8d5734c3915d1b57d7cbb92ec8",
"binary_sha256": "7b77e78cc7b8f63a60f8a7101bebed6502f82e03016f4d4e11450f927e0a52fa",
"container_image_digest": "sha256:5555555555555555555555555555555555555555555555555555555555555555"
}
}
@@ -0,0 +1,42 @@
{
"schema_version": 1,
"evidence_profile": "release-artifact",
"run_id": "fixture-batch-1",
"run_attempt": 2,
"plan_sha256": "578bb57e65df8ba7a63e40aafc254fa644163270b7ec288fe3368bcd7c10ff86",
"tier": "release-certification",
"lane": "identity",
"case_id": "startup",
"product_commit": "3333333333333333333333333333333333333333",
"harness_commit": "1111111111111111111111111111111111111111",
"target_fingerprint": "rustfs-9.9.9-fixture-linux-aarch64-gnu",
"platform": {
"os": "linux",
"arch": "aarch64",
"abi": "gnu"
},
"started_at": "2026-09-26T10:03:00Z",
"finished_at": "2026-09-26T10:03:30Z",
"state": "pass",
"reason_code": "ok",
"reason_class": "NOT_APPLICABLE",
"evidence": [
{
"path": "cases/identity/startup-linux-aarch64-gnu.json",
"sha256": "314b19c4fee5f406cbf7ee0d6a436059115f1ff06f6864830045f9d452ab91c9",
"size": 64
}
],
"workflow": {
"repository": "rustfs/s3chaos",
"workflow_id": "release-gate.yml",
"run_id": "child-identity-linux-aarch64-gnu",
"run_attempt": 1,
"workflow_sha": "1111111111111111111111111111111111111111"
},
"artifact": {
"archive_sha256": "ea3ac1fb0ec4aa8fbec650f64ccdbc2543843c8d5734c3915d1b57d7cbb92ec8",
"binary_sha256": "7b77e78cc7b8f63a60f8a7101bebed6502f82e03016f4d4e11450f927e0a52fa",
"container_image_digest": "sha256:5555555555555555555555555555555555555555555555555555555555555555"
}
}
@@ -0,0 +1,42 @@
{
"schema_version": 1,
"evidence_profile": "release-artifact",
"run_id": "fixture-batch-1",
"run_attempt": 2,
"plan_sha256": "578bb57e65df8ba7a63e40aafc254fa644163270b7ec288fe3368bcd7c10ff86",
"tier": "release-certification",
"lane": "identity",
"case_id": "artifact-identity",
"product_commit": "3333333333333333333333333333333333333333",
"harness_commit": "1111111111111111111111111111111111111111",
"target_fingerprint": "rustfs-9.9.9-fixture-linux-x86_64-gnu",
"platform": {
"os": "linux",
"arch": "x86_64",
"abi": "gnu"
},
"started_at": "2026-09-26T10:00:00Z",
"finished_at": "2026-09-26T10:00:30Z",
"state": "pass",
"reason_code": "ok",
"reason_class": "NOT_APPLICABLE",
"evidence": [
{
"path": "cases/identity/artifact-identity-linux-x86_64-gnu.json",
"sha256": "37e58ec3fc6fe3ac7ff2fcbeebdb62f6a6aaaa34ec771a0f3306c821da0336f7",
"size": 64
}
],
"workflow": {
"repository": "rustfs/s3chaos",
"workflow_id": "release-gate.yml",
"run_id": "child-identity-linux-x86_64-gnu",
"run_attempt": 1,
"workflow_sha": "1111111111111111111111111111111111111111"
},
"artifact": {
"archive_sha256": "ea3ac1fb0ec4aa8fbec650f64ccdbc2543843c8d5734c3915d1b57d7cbb92ec8",
"binary_sha256": "7b77e78cc7b8f63a60f8a7101bebed6502f82e03016f4d4e11450f927e0a52fa",
"container_image_digest": "sha256:5555555555555555555555555555555555555555555555555555555555555555"
}
}
@@ -0,0 +1,42 @@
{
"schema_version": 1,
"evidence_profile": "release-artifact",
"run_id": "fixture-batch-1",
"run_attempt": 2,
"plan_sha256": "578bb57e65df8ba7a63e40aafc254fa644163270b7ec288fe3368bcd7c10ff86",
"tier": "release-certification",
"lane": "identity",
"case_id": "startup",
"product_commit": "3333333333333333333333333333333333333333",
"harness_commit": "1111111111111111111111111111111111111111",
"target_fingerprint": "rustfs-9.9.9-fixture-linux-x86_64-gnu",
"platform": {
"os": "linux",
"arch": "x86_64",
"abi": "gnu"
},
"started_at": "2026-09-26T10:01:00Z",
"finished_at": "2026-09-26T10:01:30Z",
"state": "pass",
"reason_code": "ok",
"reason_class": "NOT_APPLICABLE",
"evidence": [
{
"path": "cases/identity/startup-linux-x86_64-gnu.json",
"sha256": "8b8774175c55c400b337610a89fe725ad2637af8fcc4973a82ab218f9182b80d",
"size": 64
}
],
"workflow": {
"repository": "rustfs/s3chaos",
"workflow_id": "release-gate.yml",
"run_id": "child-identity-linux-x86_64-gnu",
"run_attempt": 1,
"workflow_sha": "1111111111111111111111111111111111111111"
},
"artifact": {
"archive_sha256": "ea3ac1fb0ec4aa8fbec650f64ccdbc2543843c8d5734c3915d1b57d7cbb92ec8",
"binary_sha256": "7b77e78cc7b8f63a60f8a7101bebed6502f82e03016f4d4e11450f927e0a52fa",
"container_image_digest": "sha256:5555555555555555555555555555555555555555555555555555555555555555"
}
}
@@ -0,0 +1,42 @@
{
"schema_version": 1,
"evidence_profile": "release-artifact",
"run_id": "fixture-batch-1",
"run_attempt": 2,
"plan_sha256": "578bb57e65df8ba7a63e40aafc254fa644163270b7ec288fe3368bcd7c10ff86",
"tier": "release-certification",
"lane": "identity",
"case_id": "startup",
"product_commit": "3333333333333333333333333333333333333333",
"harness_commit": "1111111111111111111111111111111111111111",
"target_fingerprint": "rustfs-9.9.9-fixture-linux-x86_64-gnu",
"platform": {
"os": "linux",
"arch": "x86_64",
"abi": "gnu"
},
"started_at": "2026-09-26T10:01:00Z",
"finished_at": "2026-09-26T10:01:30Z",
"state": "pass",
"reason_code": "ok",
"reason_class": "NOT_APPLICABLE",
"evidence": [
{
"path": "cases/identity/startup-linux-x86_64-gnu.json",
"sha256": "8b8774175c55c400b337610a89fe725ad2637af8fcc4973a82ab218f9182b80d",
"size": 64
}
],
"workflow": {
"repository": "rustfs/s3chaos",
"workflow_id": "release-gate.yml",
"run_id": "child-identity-linux-x86_64-gnu",
"run_attempt": 1,
"workflow_sha": "1111111111111111111111111111111111111111"
},
"artifact": {
"archive_sha256": "ea3ac1fb0ec4aa8fbec650f64ccdbc2543843c8d5734c3915d1b57d7cbb92ec8",
"binary_sha256": "7b77e78cc7b8f63a60f8a7101bebed6502f82e03016f4d4e11450f927e0a52fa",
"container_image_digest": "sha256:5555555555555555555555555555555555555555555555555555555555555555"
}
}
@@ -0,0 +1,42 @@
{
"schema_version": 1,
"evidence_profile": "release-artifact",
"run_id": "fixture-batch-1",
"run_attempt": 2,
"plan_sha256": "578bb57e65df8ba7a63e40aafc254fa644163270b7ec288fe3368bcd7c10ff86",
"tier": "release-certification",
"lane": "upgrade",
"case_id": "upgrade-smoke",
"product_commit": "3333333333333333333333333333333333333333",
"harness_commit": "2222222222222222222222222222222222222222",
"target_fingerprint": "rustfs-9.9.9-fixture-linux-x86_64-gnu",
"platform": {
"os": "linux",
"arch": "x86_64",
"abi": "gnu"
},
"started_at": "2026-09-26T10:04:00Z",
"finished_at": "2026-09-26T10:04:30Z",
"state": "pass",
"reason_code": "ok",
"reason_class": "NOT_APPLICABLE",
"evidence": [
{
"path": "cases/upgrade/upgrade-smoke-linux-x86_64-gnu.json",
"sha256": "c1ce56808b52859d2590b34b5990aed7bf560b134e44bd0c09f1873c9f764b38",
"size": 64
}
],
"workflow": {
"repository": "rustfs/s3chaos",
"workflow_id": "release-gate.yml",
"run_id": "child-upgrade-linux-x86_64-gnu",
"run_attempt": 1,
"workflow_sha": "2222222222222222222222222222222222222222"
},
"artifact": {
"archive_sha256": "ea3ac1fb0ec4aa8fbec650f64ccdbc2543843c8d5734c3915d1b57d7cbb92ec8",
"binary_sha256": "7b77e78cc7b8f63a60f8a7101bebed6502f82e03016f4d4e11450f927e0a52fa",
"container_image_digest": "sha256:5555555555555555555555555555555555555555555555555555555555555555"
}
}
+6
View File
@@ -0,0 +1,6 @@
{
"schema_version": 1,
"run_id": "fixture-batch-1",
"run_attempt": 2,
"status": "completed"
}
@@ -0,0 +1,42 @@
{
"schema_version": 1,
"evidence_profile": "release-artifact",
"run_id": "fixture-batch-1",
"run_attempt": 2,
"plan_sha256": "578bb57e65df8ba7a63e40aafc254fa644163270b7ec288fe3368bcd7c10ff86",
"tier": "release-certification",
"lane": "identity",
"case_id": "artifact-identity",
"product_commit": "3333333333333333333333333333333333333333",
"harness_commit": "1111111111111111111111111111111111111111",
"target_fingerprint": "rustfs-9.9.9-fixture-linux-aarch64-gnu",
"platform": {
"os": "linux",
"arch": "aarch64",
"abi": "gnu"
},
"started_at": "2026-09-26T10:02:00Z",
"finished_at": "2026-09-26T10:02:30Z",
"state": "pass",
"reason_code": "ok",
"reason_class": "NOT_APPLICABLE",
"evidence": [
{
"path": "cases/identity/artifact-identity-linux-aarch64-gnu.json",
"sha256": "fa138d5891e5098b97137b0edbb4779c4350285fdc76cec7e6214809b34b3075",
"size": 64
}
],
"workflow": {
"repository": "rustfs/s3chaos",
"workflow_id": "release-gate.yml",
"run_id": "child-identity-linux-aarch64-gnu",
"run_attempt": 1,
"workflow_sha": "1111111111111111111111111111111111111111"
},
"artifact": {
"archive_sha256": "ea3ac1fb0ec4aa8fbec650f64ccdbc2543843c8d5734c3915d1b57d7cbb92ec8",
"binary_sha256": "7b77e78cc7b8f63a60f8a7101bebed6502f82e03016f4d4e11450f927e0a52fa",
"container_image_digest": "sha256:5555555555555555555555555555555555555555555555555555555555555555"
}
}
@@ -0,0 +1,42 @@
{
"schema_version": 1,
"evidence_profile": "release-artifact",
"run_id": "fixture-batch-1",
"run_attempt": 2,
"plan_sha256": "578bb57e65df8ba7a63e40aafc254fa644163270b7ec288fe3368bcd7c10ff86",
"tier": "release-certification",
"lane": "identity",
"case_id": "startup",
"product_commit": "3333333333333333333333333333333333333333",
"harness_commit": "1111111111111111111111111111111111111111",
"target_fingerprint": "rustfs-9.9.9-fixture-linux-aarch64-gnu",
"platform": {
"os": "linux",
"arch": "aarch64",
"abi": "gnu"
},
"started_at": "2026-09-26T10:03:00Z",
"finished_at": "2026-09-26T10:03:30Z",
"state": "pass",
"reason_code": "ok",
"reason_class": "NOT_APPLICABLE",
"evidence": [
{
"path": "cases/identity/startup-linux-aarch64-gnu.json",
"sha256": "314b19c4fee5f406cbf7ee0d6a436059115f1ff06f6864830045f9d452ab91c9",
"size": 64
}
],
"workflow": {
"repository": "rustfs/s3chaos",
"workflow_id": "release-gate.yml",
"run_id": "child-identity-linux-aarch64-gnu",
"run_attempt": 1,
"workflow_sha": "1111111111111111111111111111111111111111"
},
"artifact": {
"archive_sha256": "ea3ac1fb0ec4aa8fbec650f64ccdbc2543843c8d5734c3915d1b57d7cbb92ec8",
"binary_sha256": "7b77e78cc7b8f63a60f8a7101bebed6502f82e03016f4d4e11450f927e0a52fa",
"container_image_digest": "sha256:5555555555555555555555555555555555555555555555555555555555555555"
}
}
@@ -0,0 +1,42 @@
{
"schema_version": 1,
"evidence_profile": "release-artifact",
"run_id": "fixture-batch-1",
"run_attempt": 2,
"plan_sha256": "578bb57e65df8ba7a63e40aafc254fa644163270b7ec288fe3368bcd7c10ff86",
"tier": "release-certification",
"lane": "identity",
"case_id": "artifact-identity",
"product_commit": "3333333333333333333333333333333333333333",
"harness_commit": "1111111111111111111111111111111111111111",
"target_fingerprint": "rustfs-9.9.9-fixture-linux-x86_64-gnu",
"platform": {
"os": "linux",
"arch": "x86_64",
"abi": "gnu"
},
"started_at": "2026-09-26T10:00:00Z",
"finished_at": "2026-09-26T10:00:30Z",
"state": "pass",
"reason_code": "ok",
"reason_class": "NOT_APPLICABLE",
"evidence": [
{
"path": "cases/identity/artifact-identity-linux-x86_64-gnu.json",
"sha256": "37e58ec3fc6fe3ac7ff2fcbeebdb62f6a6aaaa34ec771a0f3306c821da0336f7",
"size": 64
}
],
"workflow": {
"repository": "rustfs/s3chaos",
"workflow_id": "release-gate.yml",
"run_id": "child-identity-linux-x86_64-gnu",
"run_attempt": 1,
"workflow_sha": "1111111111111111111111111111111111111111"
},
"artifact": {
"archive_sha256": "ea3ac1fb0ec4aa8fbec650f64ccdbc2543843c8d5734c3915d1b57d7cbb92ec8",
"binary_sha256": "7b77e78cc7b8f63a60f8a7101bebed6502f82e03016f4d4e11450f927e0a52fa",
"container_image_digest": "sha256:5555555555555555555555555555555555555555555555555555555555555555"
}
}
@@ -0,0 +1,42 @@
{
"schema_version": 1,
"evidence_profile": "release-artifact",
"run_id": "fixture-batch-1",
"run_attempt": 2,
"plan_sha256": "578bb57e65df8ba7a63e40aafc254fa644163270b7ec288fe3368bcd7c10ff86",
"tier": "release-certification",
"lane": "identity",
"case_id": "startup",
"product_commit": "3333333333333333333333333333333333333333",
"harness_commit": "1111111111111111111111111111111111111111",
"target_fingerprint": "rustfs-9.9.9-fixture-linux-x86_64-gnu",
"platform": {
"os": "linux",
"arch": "x86_64",
"abi": "gnu"
},
"started_at": "2026-09-26T10:01:00Z",
"finished_at": "2026-09-26T10:01:30Z",
"state": "pass",
"reason_code": "ok",
"reason_class": "NOT_APPLICABLE",
"evidence": [
{
"path": "cases/identity/startup-linux-x86_64-gnu.json",
"sha256": "8b8774175c55c400b337610a89fe725ad2637af8fcc4973a82ab218f9182b80d",
"size": 64
}
],
"workflow": {
"repository": "rustfs/s3chaos",
"workflow_id": "release-gate.yml",
"run_id": "child-identity-linux-x86_64-gnu",
"run_attempt": 1,
"workflow_sha": "1111111111111111111111111111111111111111"
},
"artifact": {
"archive_sha256": "ea3ac1fb0ec4aa8fbec650f64ccdbc2543843c8d5734c3915d1b57d7cbb92ec8",
"binary_sha256": "7b77e78cc7b8f63a60f8a7101bebed6502f82e03016f4d4e11450f927e0a52fa",
"container_image_digest": "sha256:5555555555555555555555555555555555555555555555555555555555555555"
}
}
@@ -0,0 +1,42 @@
{
"schema_version": 1,
"evidence_profile": "release-artifact",
"run_id": "fixture-batch-1",
"run_attempt": 2,
"plan_sha256": "578bb57e65df8ba7a63e40aafc254fa644163270b7ec288fe3368bcd7c10ff86",
"tier": "release-certification",
"lane": "identity",
"case_id": "surprise-case",
"product_commit": "3333333333333333333333333333333333333333",
"harness_commit": "1111111111111111111111111111111111111111",
"target_fingerprint": "rustfs-9.9.9-fixture-linux-x86_64-gnu",
"platform": {
"os": "linux",
"arch": "x86_64",
"abi": "gnu"
},
"started_at": "2026-09-26T10:05:00Z",
"finished_at": "2026-09-26T10:05:30Z",
"state": "pass",
"reason_code": "ok",
"reason_class": "NOT_APPLICABLE",
"evidence": [
{
"path": "cases/identity/surprise-case-linux-x86_64-gnu.json",
"sha256": "f82eedac16c983bd426ca24a6cbf3acfa032dc722e9c4a5ed7ef21c7eff0bcb1",
"size": 64
}
],
"workflow": {
"repository": "rustfs/s3chaos",
"workflow_id": "release-gate.yml",
"run_id": "child-identity-linux-x86_64-gnu",
"run_attempt": 1,
"workflow_sha": "1111111111111111111111111111111111111111"
},
"artifact": {
"archive_sha256": "ea3ac1fb0ec4aa8fbec650f64ccdbc2543843c8d5734c3915d1b57d7cbb92ec8",
"binary_sha256": "7b77e78cc7b8f63a60f8a7101bebed6502f82e03016f4d4e11450f927e0a52fa",
"container_image_digest": "sha256:5555555555555555555555555555555555555555555555555555555555555555"
}
}
@@ -0,0 +1,42 @@
{
"schema_version": 1,
"evidence_profile": "release-artifact",
"run_id": "fixture-batch-1",
"run_attempt": 2,
"plan_sha256": "578bb57e65df8ba7a63e40aafc254fa644163270b7ec288fe3368bcd7c10ff86",
"tier": "release-certification",
"lane": "upgrade",
"case_id": "upgrade-smoke",
"product_commit": "3333333333333333333333333333333333333333",
"harness_commit": "2222222222222222222222222222222222222222",
"target_fingerprint": "rustfs-9.9.9-fixture-linux-x86_64-gnu",
"platform": {
"os": "linux",
"arch": "x86_64",
"abi": "gnu"
},
"started_at": "2026-09-26T10:04:00Z",
"finished_at": "2026-09-26T10:04:30Z",
"state": "pass",
"reason_code": "ok",
"reason_class": "NOT_APPLICABLE",
"evidence": [
{
"path": "cases/upgrade/upgrade-smoke-linux-x86_64-gnu.json",
"sha256": "c1ce56808b52859d2590b34b5990aed7bf560b134e44bd0c09f1873c9f764b38",
"size": 64
}
],
"workflow": {
"repository": "rustfs/s3chaos",
"workflow_id": "release-gate.yml",
"run_id": "child-upgrade-linux-x86_64-gnu",
"run_attempt": 1,
"workflow_sha": "2222222222222222222222222222222222222222"
},
"artifact": {
"archive_sha256": "ea3ac1fb0ec4aa8fbec650f64ccdbc2543843c8d5734c3915d1b57d7cbb92ec8",
"binary_sha256": "7b77e78cc7b8f63a60f8a7101bebed6502f82e03016f4d4e11450f927e0a52fa",
"container_image_digest": "sha256:5555555555555555555555555555555555555555555555555555555555555555"
}
}
+6
View File
@@ -0,0 +1,6 @@
{
"schema_version": 1,
"run_id": "fixture-batch-1",
"run_attempt": 2,
"status": "completed"
}
@@ -0,0 +1,42 @@
{
"schema_version": 1,
"evidence_profile": "release-artifact",
"run_id": "fixture-batch-1",
"run_attempt": 2,
"plan_sha256": "578bb57e65df8ba7a63e40aafc254fa644163270b7ec288fe3368bcd7c10ff86",
"tier": "release-certification",
"lane": "identity",
"case_id": "artifact-identity",
"product_commit": "3333333333333333333333333333333333333333",
"harness_commit": "1111111111111111111111111111111111111111",
"target_fingerprint": "rustfs-9.9.9-fixture-linux-aarch64-gnu",
"platform": {
"os": "linux",
"arch": "aarch64",
"abi": "gnu"
},
"started_at": "2026-09-26T10:02:00Z",
"finished_at": "2026-09-26T10:02:30Z",
"state": "pass",
"reason_code": "ok",
"reason_class": "NOT_APPLICABLE",
"evidence": [
{
"path": "cases/identity/artifact-identity-linux-aarch64-gnu.json",
"sha256": "fa138d5891e5098b97137b0edbb4779c4350285fdc76cec7e6214809b34b3075",
"size": 64
}
],
"workflow": {
"repository": "rustfs/s3chaos",
"workflow_id": "release-gate.yml",
"run_id": "child-identity-linux-aarch64-gnu",
"run_attempt": 1,
"workflow_sha": "1111111111111111111111111111111111111111"
},
"artifact": {
"archive_sha256": "ea3ac1fb0ec4aa8fbec650f64ccdbc2543843c8d5734c3915d1b57d7cbb92ec8",
"binary_sha256": "7b77e78cc7b8f63a60f8a7101bebed6502f82e03016f4d4e11450f927e0a52fa",
"container_image_digest": "sha256:5555555555555555555555555555555555555555555555555555555555555555"
}
}
@@ -0,0 +1,42 @@
{
"schema_version": 1,
"evidence_profile": "release-artifact",
"run_id": "fixture-batch-1",
"run_attempt": 2,
"plan_sha256": "578bb57e65df8ba7a63e40aafc254fa644163270b7ec288fe3368bcd7c10ff86",
"tier": "release-certification",
"lane": "identity",
"case_id": "artifact-identity",
"product_commit": "3333333333333333333333333333333333333333",
"harness_commit": "1111111111111111111111111111111111111111",
"target_fingerprint": "rustfs-9.9.9-fixture-linux-x86_64-gnu",
"platform": {
"os": "linux",
"arch": "x86_64",
"abi": "gnu"
},
"started_at": "2026-09-26T10:00:00Z",
"finished_at": "2026-09-26T10:00:30Z",
"state": "pass",
"reason_code": "ok",
"reason_class": "NOT_APPLICABLE",
"evidence": [
{
"path": "cases/identity/artifact-identity-linux-x86_64-gnu.json",
"sha256": "37e58ec3fc6fe3ac7ff2fcbeebdb62f6a6aaaa34ec771a0f3306c821da0336f7",
"size": 64
}
],
"workflow": {
"repository": "rustfs/s3chaos",
"workflow_id": "release-gate.yml",
"run_id": "child-identity-linux-x86_64-gnu",
"run_attempt": 1,
"workflow_sha": "1111111111111111111111111111111111111111"
},
"artifact": {
"archive_sha256": "ea3ac1fb0ec4aa8fbec650f64ccdbc2543843c8d5734c3915d1b57d7cbb92ec8",
"binary_sha256": "7b77e78cc7b8f63a60f8a7101bebed6502f82e03016f4d4e11450f927e0a52fa",
"container_image_digest": "sha256:5555555555555555555555555555555555555555555555555555555555555555"
}
}
@@ -0,0 +1,42 @@
{
"schema_version": 1,
"evidence_profile": "release-artifact",
"run_id": "fixture-batch-1",
"run_attempt": 2,
"plan_sha256": "578bb57e65df8ba7a63e40aafc254fa644163270b7ec288fe3368bcd7c10ff86",
"tier": "release-certification",
"lane": "identity",
"case_id": "startup",
"product_commit": "3333333333333333333333333333333333333333",
"harness_commit": "1111111111111111111111111111111111111111",
"target_fingerprint": "rustfs-9.9.9-fixture-linux-x86_64-gnu",
"platform": {
"os": "linux",
"arch": "x86_64",
"abi": "gnu"
},
"started_at": "2026-09-26T10:01:00Z",
"finished_at": "2026-09-26T10:01:30Z",
"state": "pass",
"reason_code": "ok",
"reason_class": "NOT_APPLICABLE",
"evidence": [
{
"path": "cases/identity/startup-linux-x86_64-gnu.json",
"sha256": "8b8774175c55c400b337610a89fe725ad2637af8fcc4973a82ab218f9182b80d",
"size": 64
}
],
"workflow": {
"repository": "rustfs/s3chaos",
"workflow_id": "release-gate.yml",
"run_id": "child-identity-linux-x86_64-gnu",
"run_attempt": 1,
"workflow_sha": "1111111111111111111111111111111111111111"
},
"artifact": {
"archive_sha256": "ea3ac1fb0ec4aa8fbec650f64ccdbc2543843c8d5734c3915d1b57d7cbb92ec8",
"binary_sha256": "7b77e78cc7b8f63a60f8a7101bebed6502f82e03016f4d4e11450f927e0a52fa",
"container_image_digest": "sha256:5555555555555555555555555555555555555555555555555555555555555555"
}
}
@@ -0,0 +1,42 @@
{
"schema_version": 1,
"evidence_profile": "release-artifact",
"run_id": "fixture-batch-1",
"run_attempt": 2,
"plan_sha256": "578bb57e65df8ba7a63e40aafc254fa644163270b7ec288fe3368bcd7c10ff86",
"tier": "release-certification",
"lane": "upgrade",
"case_id": "upgrade-smoke",
"product_commit": "3333333333333333333333333333333333333333",
"harness_commit": "2222222222222222222222222222222222222222",
"target_fingerprint": "rustfs-9.9.9-fixture-linux-x86_64-gnu",
"platform": {
"os": "linux",
"arch": "x86_64",
"abi": "gnu"
},
"started_at": "2026-09-26T10:04:00Z",
"finished_at": "2026-09-26T10:04:30Z",
"state": "pass",
"reason_code": "ok",
"reason_class": "NOT_APPLICABLE",
"evidence": [
{
"path": "cases/upgrade/upgrade-smoke-linux-x86_64-gnu.json",
"sha256": "c1ce56808b52859d2590b34b5990aed7bf560b134e44bd0c09f1873c9f764b38",
"size": 64
}
],
"workflow": {
"repository": "rustfs/s3chaos",
"workflow_id": "release-gate.yml",
"run_id": "child-upgrade-linux-x86_64-gnu",
"run_attempt": 1,
"workflow_sha": "2222222222222222222222222222222222222222"
},
"artifact": {
"archive_sha256": "ea3ac1fb0ec4aa8fbec650f64ccdbc2543843c8d5734c3915d1b57d7cbb92ec8",
"binary_sha256": "7b77e78cc7b8f63a60f8a7101bebed6502f82e03016f4d4e11450f927e0a52fa",
"container_image_digest": "sha256:5555555555555555555555555555555555555555555555555555555555555555"
}
}
+6
View File
@@ -0,0 +1,6 @@
{
"schema_version": 1,
"run_id": "fixture-batch-1",
"run_attempt": 2,
"status": "completed"
}
@@ -0,0 +1,42 @@
{
"schema_version": 1,
"evidence_profile": "release-artifact",
"run_id": "fixture-batch-1",
"run_attempt": 2,
"plan_sha256": "578bb57e65df8ba7a63e40aafc254fa644163270b7ec288fe3368bcd7c10ff86",
"tier": "release-certification",
"lane": "identity",
"case_id": "artifact-identity",
"product_commit": "3333333333333333333333333333333333333333",
"harness_commit": "1111111111111111111111111111111111111111",
"target_fingerprint": "rustfs-9.9.9-fixture-linux-aarch64-gnu",
"platform": {
"os": "linux",
"arch": "aarch64",
"abi": "gnu"
},
"started_at": "2026-09-26T10:02:00Z",
"finished_at": "2026-09-26T10:02:30Z",
"state": "pass",
"reason_code": "ok",
"reason_class": "NOT_APPLICABLE",
"evidence": [
{
"path": "cases/identity/artifact-identity-linux-aarch64-gnu.json",
"sha256": "fa138d5891e5098b97137b0edbb4779c4350285fdc76cec7e6214809b34b3075",
"size": 64
}
],
"workflow": {
"repository": "rustfs/s3chaos",
"workflow_id": "release-gate.yml",
"run_id": "child-identity-linux-aarch64-gnu",
"run_attempt": 1,
"workflow_sha": "1111111111111111111111111111111111111111"
},
"artifact": {
"archive_sha256": "ea3ac1fb0ec4aa8fbec650f64ccdbc2543843c8d5734c3915d1b57d7cbb92ec8",
"binary_sha256": "7b77e78cc7b8f63a60f8a7101bebed6502f82e03016f4d4e11450f927e0a52fa",
"container_image_digest": "sha256:5555555555555555555555555555555555555555555555555555555555555555"
}
}
@@ -0,0 +1,42 @@
{
"schema_version": 1,
"evidence_profile": "release-artifact",
"run_id": "fixture-batch-1",
"run_attempt": 2,
"plan_sha256": "578bb57e65df8ba7a63e40aafc254fa644163270b7ec288fe3368bcd7c10ff86",
"tier": "release-certification",
"lane": "identity",
"case_id": "startup",
"product_commit": "3333333333333333333333333333333333333333",
"harness_commit": "1111111111111111111111111111111111111111",
"target_fingerprint": "rustfs-9.9.9-fixture-linux-aarch64-gnu",
"platform": {
"os": "linux",
"arch": "aarch64",
"abi": "gnu"
},
"started_at": "2026-09-26T10:03:00Z",
"finished_at": "2026-09-26T10:03:30Z",
"state": "pass",
"reason_code": "ok",
"reason_class": "NOT_APPLICABLE",
"evidence": [
{
"path": "cases/identity/startup-linux-aarch64-gnu.json",
"sha256": "314b19c4fee5f406cbf7ee0d6a436059115f1ff06f6864830045f9d452ab91c9",
"size": 64
}
],
"workflow": {
"repository": "rustfs/s3chaos",
"workflow_id": "release-gate.yml",
"run_id": "child-identity-linux-aarch64-gnu",
"run_attempt": 1,
"workflow_sha": "1111111111111111111111111111111111111111"
},
"artifact": {
"archive_sha256": "ea3ac1fb0ec4aa8fbec650f64ccdbc2543843c8d5734c3915d1b57d7cbb92ec8",
"binary_sha256": "7b77e78cc7b8f63a60f8a7101bebed6502f82e03016f4d4e11450f927e0a52fa",
"container_image_digest": "sha256:5555555555555555555555555555555555555555555555555555555555555555"
}
}
@@ -0,0 +1,42 @@
{
"schema_version": 1,
"evidence_profile": "release-artifact",
"run_id": "fixture-batch-1",
"run_attempt": 2,
"plan_sha256": "578bb57e65df8ba7a63e40aafc254fa644163270b7ec288fe3368bcd7c10ff86",
"tier": "release-certification",
"lane": "identity",
"case_id": "artifact-identity",
"product_commit": "3333333333333333333333333333333333333333",
"harness_commit": "1111111111111111111111111111111111111111",
"target_fingerprint": "rustfs-9.9.9-fixture-linux-x86_64-gnu",
"platform": {
"os": "linux",
"arch": "x86_64",
"abi": "gnu"
},
"started_at": "2026-09-26T10:00:00Z",
"finished_at": "2026-09-26T10:00:30Z",
"state": "pass",
"reason_code": "ok",
"reason_class": "NOT_APPLICABLE",
"evidence": [
{
"path": "cases/identity/artifact-identity-linux-x86_64-gnu.json",
"sha256": "37e58ec3fc6fe3ac7ff2fcbeebdb62f6a6aaaa34ec771a0f3306c821da0336f7",
"size": 64
}
],
"workflow": {
"repository": "rustfs/s3chaos",
"workflow_id": "release-gate.yml",
"run_id": "child-identity-linux-x86_64-gnu",
"run_attempt": 1,
"workflow_sha": "1111111111111111111111111111111111111111"
},
"artifact": {
"archive_sha256": "ea3ac1fb0ec4aa8fbec650f64ccdbc2543843c8d5734c3915d1b57d7cbb92ec8",
"binary_sha256": "7b77e78cc7b8f63a60f8a7101bebed6502f82e03016f4d4e11450f927e0a52fa",
"container_image_digest": "sha256:5555555555555555555555555555555555555555555555555555555555555555"
}
}
@@ -0,0 +1,42 @@
{
"schema_version": 1,
"evidence_profile": "release-artifact",
"run_id": "fixture-batch-1",
"run_attempt": 2,
"plan_sha256": "578bb57e65df8ba7a63e40aafc254fa644163270b7ec288fe3368bcd7c10ff86",
"tier": "release-certification",
"lane": "identity",
"case_id": "startup",
"product_commit": "3333333333333333333333333333333333333333",
"harness_commit": "1111111111111111111111111111111111111111",
"target_fingerprint": "rustfs-9.9.9-fixture-linux-x86_64-gnu",
"platform": {
"os": "linux",
"arch": "x86_64",
"abi": "gnu"
},
"started_at": "2026-09-26T10:01:00Z",
"finished_at": "2026-09-26T10:01:30Z",
"state": "infra_error",
"reason_code": "fixture-infra_error",
"reason_class": "NOT_APPLICABLE",
"evidence": [
{
"path": "cases/identity/startup-linux-x86_64-gnu.json",
"sha256": "8b8774175c55c400b337610a89fe725ad2637af8fcc4973a82ab218f9182b80d",
"size": 64
}
],
"workflow": {
"repository": "rustfs/s3chaos",
"workflow_id": "release-gate.yml",
"run_id": "child-identity-linux-x86_64-gnu",
"run_attempt": 1,
"workflow_sha": "1111111111111111111111111111111111111111"
},
"artifact": {
"archive_sha256": "ea3ac1fb0ec4aa8fbec650f64ccdbc2543843c8d5734c3915d1b57d7cbb92ec8",
"binary_sha256": "7b77e78cc7b8f63a60f8a7101bebed6502f82e03016f4d4e11450f927e0a52fa",
"container_image_digest": "sha256:5555555555555555555555555555555555555555555555555555555555555555"
}
}
@@ -0,0 +1,42 @@
{
"schema_version": 1,
"evidence_profile": "release-artifact",
"run_id": "fixture-batch-1",
"run_attempt": 2,
"plan_sha256": "578bb57e65df8ba7a63e40aafc254fa644163270b7ec288fe3368bcd7c10ff86",
"tier": "release-certification",
"lane": "upgrade",
"case_id": "upgrade-smoke",
"product_commit": "3333333333333333333333333333333333333333",
"harness_commit": "2222222222222222222222222222222222222222",
"target_fingerprint": "rustfs-9.9.9-fixture-linux-x86_64-gnu",
"platform": {
"os": "linux",
"arch": "x86_64",
"abi": "gnu"
},
"started_at": "2026-09-26T10:04:00Z",
"finished_at": "2026-09-26T10:04:30Z",
"state": "pass",
"reason_code": "ok",
"reason_class": "NOT_APPLICABLE",
"evidence": [
{
"path": "cases/upgrade/upgrade-smoke-linux-x86_64-gnu.json",
"sha256": "c1ce56808b52859d2590b34b5990aed7bf560b134e44bd0c09f1873c9f764b38",
"size": 64
}
],
"workflow": {
"repository": "rustfs/s3chaos",
"workflow_id": "release-gate.yml",
"run_id": "child-upgrade-linux-x86_64-gnu",
"run_attempt": 1,
"workflow_sha": "2222222222222222222222222222222222222222"
},
"artifact": {
"archive_sha256": "ea3ac1fb0ec4aa8fbec650f64ccdbc2543843c8d5734c3915d1b57d7cbb92ec8",
"binary_sha256": "7b77e78cc7b8f63a60f8a7101bebed6502f82e03016f4d4e11450f927e0a52fa",
"container_image_digest": "sha256:5555555555555555555555555555555555555555555555555555555555555555"
}
}
+6
View File
@@ -0,0 +1,6 @@
{
"schema_version": 1,
"run_id": "fixture-batch-1",
"run_attempt": 2,
"status": "completed"
}
@@ -0,0 +1,42 @@
{
"schema_version": 1,
"evidence_profile": "release-artifact",
"run_id": "fixture-batch-1",
"run_attempt": 2,
"plan_sha256": "578bb57e65df8ba7a63e40aafc254fa644163270b7ec288fe3368bcd7c10ff86",
"tier": "release-certification",
"lane": "identity",
"case_id": "artifact-identity",
"product_commit": "3333333333333333333333333333333333333333",
"harness_commit": "1111111111111111111111111111111111111111",
"target_fingerprint": "rustfs-9.9.9-fixture-linux-aarch64-gnu",
"platform": {
"os": "linux",
"arch": "aarch64",
"abi": "gnu"
},
"started_at": "2026-09-26T10:02:00Z",
"finished_at": "2026-09-26T10:02:30Z",
"state": "pass",
"reason_code": "ok",
"reason_class": "NOT_APPLICABLE",
"evidence": [
{
"path": "cases/identity/artifact-identity-linux-aarch64-gnu.json",
"sha256": "fa138d5891e5098b97137b0edbb4779c4350285fdc76cec7e6214809b34b3075",
"size": 64
}
],
"workflow": {
"repository": "rustfs/s3chaos",
"workflow_id": "release-gate.yml",
"run_id": "child-identity-linux-aarch64-gnu",
"run_attempt": 1,
"workflow_sha": "1111111111111111111111111111111111111111"
},
"artifact": {
"archive_sha256": "ea3ac1fb0ec4aa8fbec650f64ccdbc2543843c8d5734c3915d1b57d7cbb92ec8",
"binary_sha256": "7b77e78cc7b8f63a60f8a7101bebed6502f82e03016f4d4e11450f927e0a52fa",
"container_image_digest": "sha256:5555555555555555555555555555555555555555555555555555555555555555"
}
}
@@ -0,0 +1,42 @@
{
"schema_version": 1,
"evidence_profile": "release-artifact",
"run_id": "fixture-batch-1",
"run_attempt": 2,
"plan_sha256": "578bb57e65df8ba7a63e40aafc254fa644163270b7ec288fe3368bcd7c10ff86",
"tier": "release-certification",
"lane": "identity",
"case_id": "startup",
"product_commit": "3333333333333333333333333333333333333333",
"harness_commit": "1111111111111111111111111111111111111111",
"target_fingerprint": "rustfs-9.9.9-fixture-linux-aarch64-gnu",
"platform": {
"os": "linux",
"arch": "aarch64",
"abi": "gnu"
},
"started_at": "2026-09-26T10:03:00Z",
"finished_at": "2026-09-26T10:03:30Z",
"state": "pass",
"reason_code": "ok",
"reason_class": "NOT_APPLICABLE",
"evidence": [
{
"path": "cases/identity/startup-linux-aarch64-gnu.json",
"sha256": "314b19c4fee5f406cbf7ee0d6a436059115f1ff06f6864830045f9d452ab91c9",
"size": 64
}
],
"workflow": {
"repository": "rustfs/s3chaos",
"workflow_id": "release-gate.yml",
"run_id": "child-identity-linux-aarch64-gnu",
"run_attempt": 1,
"workflow_sha": "1111111111111111111111111111111111111111"
},
"artifact": {
"archive_sha256": "ea3ac1fb0ec4aa8fbec650f64ccdbc2543843c8d5734c3915d1b57d7cbb92ec8",
"binary_sha256": "7b77e78cc7b8f63a60f8a7101bebed6502f82e03016f4d4e11450f927e0a52fa",
"container_image_digest": "sha256:5555555555555555555555555555555555555555555555555555555555555555"
}
}
@@ -0,0 +1,42 @@
{
"schema_version": 1,
"evidence_profile": "release-artifact",
"run_id": "fixture-batch-1",
"run_attempt": 2,
"plan_sha256": "578bb57e65df8ba7a63e40aafc254fa644163270b7ec288fe3368bcd7c10ff86",
"tier": "release-certification",
"lane": "identity",
"case_id": "artifact-identity",
"product_commit": "3333333333333333333333333333333333333333",
"harness_commit": "1111111111111111111111111111111111111111",
"target_fingerprint": "rustfs-9.9.9-fixture-linux-x86_64-gnu",
"platform": {
"os": "linux",
"arch": "x86_64",
"abi": "gnu"
},
"started_at": "2026-09-26T10:00:00Z",
"finished_at": "2026-09-26T10:00:30Z",
"state": "pass",
"reason_code": "ok",
"reason_class": "NOT_APPLICABLE",
"evidence": [
{
"path": "cases/identity/artifact-identity-linux-x86_64-gnu.json",
"sha256": "37e58ec3fc6fe3ac7ff2fcbeebdb62f6a6aaaa34ec771a0f3306c821da0336f7",
"size": 64
}
],
"workflow": {
"repository": "rustfs/s3chaos",
"workflow_id": "release-gate.yml",
"run_id": "child-identity-linux-x86_64-gnu",
"run_attempt": 1,
"workflow_sha": "1111111111111111111111111111111111111111"
},
"artifact": {
"archive_sha256": "ea3ac1fb0ec4aa8fbec650f64ccdbc2543843c8d5734c3915d1b57d7cbb92ec8",
"binary_sha256": "7b77e78cc7b8f63a60f8a7101bebed6502f82e03016f4d4e11450f927e0a52fa",
"container_image_digest": "sha256:5555555555555555555555555555555555555555555555555555555555555555"
}
}
@@ -0,0 +1,42 @@
{
"schema_version": 1,
"evidence_profile": "release-artifact",
"run_id": "fixture-batch-1",
"run_attempt": 2,
"plan_sha256": "578bb57e65df8ba7a63e40aafc254fa644163270b7ec288fe3368bcd7c10ff86",
"tier": "release-certification",
"lane": "identity",
"case_id": "startup",
"product_commit": "3333333333333333333333333333333333333333",
"harness_commit": "1111111111111111111111111111111111111111",
"target_fingerprint": "rustfs-9.9.9-fixture-linux-x86_64-gnu",
"platform": {
"os": "linux",
"arch": "x86_64",
"abi": "gnu"
},
"started_at": "2026-09-26T10:01:00Z",
"finished_at": "2026-09-26T10:01:30Z",
"state": "quarantined",
"reason_code": "fixture-quarantined",
"reason_class": "NOT_APPLICABLE",
"evidence": [
{
"path": "cases/identity/startup-linux-x86_64-gnu.json",
"sha256": "8b8774175c55c400b337610a89fe725ad2637af8fcc4973a82ab218f9182b80d",
"size": 64
}
],
"workflow": {
"repository": "rustfs/s3chaos",
"workflow_id": "release-gate.yml",
"run_id": "child-identity-linux-x86_64-gnu",
"run_attempt": 1,
"workflow_sha": "1111111111111111111111111111111111111111"
},
"artifact": {
"archive_sha256": "ea3ac1fb0ec4aa8fbec650f64ccdbc2543843c8d5734c3915d1b57d7cbb92ec8",
"binary_sha256": "7b77e78cc7b8f63a60f8a7101bebed6502f82e03016f4d4e11450f927e0a52fa",
"container_image_digest": "sha256:5555555555555555555555555555555555555555555555555555555555555555"
}
}
@@ -0,0 +1,42 @@
{
"schema_version": 1,
"evidence_profile": "release-artifact",
"run_id": "fixture-batch-1",
"run_attempt": 2,
"plan_sha256": "578bb57e65df8ba7a63e40aafc254fa644163270b7ec288fe3368bcd7c10ff86",
"tier": "release-certification",
"lane": "upgrade",
"case_id": "upgrade-smoke",
"product_commit": "3333333333333333333333333333333333333333",
"harness_commit": "2222222222222222222222222222222222222222",
"target_fingerprint": "rustfs-9.9.9-fixture-linux-x86_64-gnu",
"platform": {
"os": "linux",
"arch": "x86_64",
"abi": "gnu"
},
"started_at": "2026-09-26T10:04:00Z",
"finished_at": "2026-09-26T10:04:30Z",
"state": "pass",
"reason_code": "ok",
"reason_class": "NOT_APPLICABLE",
"evidence": [
{
"path": "cases/upgrade/upgrade-smoke-linux-x86_64-gnu.json",
"sha256": "c1ce56808b52859d2590b34b5990aed7bf560b134e44bd0c09f1873c9f764b38",
"size": 64
}
],
"workflow": {
"repository": "rustfs/s3chaos",
"workflow_id": "release-gate.yml",
"run_id": "child-upgrade-linux-x86_64-gnu",
"run_attempt": 1,
"workflow_sha": "2222222222222222222222222222222222222222"
},
"artifact": {
"archive_sha256": "ea3ac1fb0ec4aa8fbec650f64ccdbc2543843c8d5734c3915d1b57d7cbb92ec8",
"binary_sha256": "7b77e78cc7b8f63a60f8a7101bebed6502f82e03016f4d4e11450f927e0a52fa",
"container_image_digest": "sha256:5555555555555555555555555555555555555555555555555555555555555555"
}
}
+6
View File
@@ -0,0 +1,6 @@
{
"schema_version": 1,
"run_id": "fixture-batch-1",
"run_attempt": 2,
"status": "completed"
}
@@ -0,0 +1,42 @@
{
"schema_version": 1,
"evidence_profile": "release-artifact",
"run_id": "fixture-batch-1",
"run_attempt": 2,
"plan_sha256": "578bb57e65df8ba7a63e40aafc254fa644163270b7ec288fe3368bcd7c10ff86",
"tier": "release-certification",
"lane": "identity",
"case_id": "artifact-identity",
"product_commit": "3333333333333333333333333333333333333333",
"harness_commit": "1111111111111111111111111111111111111111",
"target_fingerprint": "rustfs-9.9.9-fixture-linux-aarch64-gnu",
"platform": {
"os": "linux",
"arch": "aarch64",
"abi": "gnu"
},
"started_at": "2026-09-26T10:02:00Z",
"finished_at": "2026-09-26T10:02:30Z",
"state": "pass",
"reason_code": "ok",
"reason_class": "NOT_APPLICABLE",
"evidence": [
{
"path": "cases/identity/artifact-identity-linux-aarch64-gnu.json",
"sha256": "fa138d5891e5098b97137b0edbb4779c4350285fdc76cec7e6214809b34b3075",
"size": 64
}
],
"workflow": {
"repository": "rustfs/s3chaos",
"workflow_id": "release-gate.yml",
"run_id": "child-identity-linux-aarch64-gnu",
"run_attempt": 1,
"workflow_sha": "1111111111111111111111111111111111111111"
},
"artifact": {
"archive_sha256": "ea3ac1fb0ec4aa8fbec650f64ccdbc2543843c8d5734c3915d1b57d7cbb92ec8",
"binary_sha256": "7b77e78cc7b8f63a60f8a7101bebed6502f82e03016f4d4e11450f927e0a52fa",
"container_image_digest": "sha256:5555555555555555555555555555555555555555555555555555555555555555"
}
}
@@ -0,0 +1,42 @@
{
"schema_version": 1,
"evidence_profile": "release-artifact",
"run_id": "fixture-batch-1",
"run_attempt": 2,
"plan_sha256": "578bb57e65df8ba7a63e40aafc254fa644163270b7ec288fe3368bcd7c10ff86",
"tier": "release-certification",
"lane": "identity",
"case_id": "startup",
"product_commit": "3333333333333333333333333333333333333333",
"harness_commit": "1111111111111111111111111111111111111111",
"target_fingerprint": "rustfs-9.9.9-fixture-linux-aarch64-gnu",
"platform": {
"os": "linux",
"arch": "aarch64",
"abi": "gnu"
},
"started_at": "2026-09-26T10:03:00Z",
"finished_at": "2026-09-26T10:03:30Z",
"state": "pass",
"reason_code": "ok",
"reason_class": "NOT_APPLICABLE",
"evidence": [
{
"path": "cases/identity/startup-linux-aarch64-gnu.json",
"sha256": "314b19c4fee5f406cbf7ee0d6a436059115f1ff06f6864830045f9d452ab91c9",
"size": 64
}
],
"workflow": {
"repository": "rustfs/s3chaos",
"workflow_id": "release-gate.yml",
"run_id": "child-identity-linux-aarch64-gnu",
"run_attempt": 1,
"workflow_sha": "1111111111111111111111111111111111111111"
},
"artifact": {
"archive_sha256": "ea3ac1fb0ec4aa8fbec650f64ccdbc2543843c8d5734c3915d1b57d7cbb92ec8",
"binary_sha256": "7b77e78cc7b8f63a60f8a7101bebed6502f82e03016f4d4e11450f927e0a52fa",
"container_image_digest": "sha256:5555555555555555555555555555555555555555555555555555555555555555"
}
}
@@ -0,0 +1,42 @@
{
"schema_version": 1,
"evidence_profile": "release-artifact",
"run_id": "fixture-batch-1",
"run_attempt": 2,
"plan_sha256": "578bb57e65df8ba7a63e40aafc254fa644163270b7ec288fe3368bcd7c10ff86",
"tier": "release-certification",
"lane": "identity",
"case_id": "artifact-identity",
"product_commit": "3333333333333333333333333333333333333333",
"harness_commit": "1111111111111111111111111111111111111111",
"target_fingerprint": "rustfs-9.9.9-fixture-linux-x86_64-gnu",
"platform": {
"os": "linux",
"arch": "x86_64",
"abi": "gnu"
},
"started_at": "2026-09-26T10:00:00Z",
"finished_at": "2026-09-26T10:00:30Z",
"state": "pass",
"reason_code": "ok",
"reason_class": "NOT_APPLICABLE",
"evidence": [
{
"path": "cases/identity/artifact-identity-linux-x86_64-gnu.json",
"sha256": "37e58ec3fc6fe3ac7ff2fcbeebdb62f6a6aaaa34ec771a0f3306c821da0336f7",
"size": 64
}
],
"workflow": {
"repository": "rustfs/s3chaos",
"workflow_id": "release-gate.yml",
"run_id": "child-identity-linux-x86_64-gnu",
"run_attempt": 1,
"workflow_sha": "1111111111111111111111111111111111111111"
},
"artifact": {
"archive_sha256": "ea3ac1fb0ec4aa8fbec650f64ccdbc2543843c8d5734c3915d1b57d7cbb92ec8",
"binary_sha256": "7b77e78cc7b8f63a60f8a7101bebed6502f82e03016f4d4e11450f927e0a52fa",
"container_image_digest": "sha256:5555555555555555555555555555555555555555555555555555555555555555"
}
}
@@ -0,0 +1,42 @@
{
"schema_version": 1,
"evidence_profile": "release-artifact",
"run_id": "fixture-batch-1",
"run_attempt": 2,
"plan_sha256": "578bb57e65df8ba7a63e40aafc254fa644163270b7ec288fe3368bcd7c10ff86",
"tier": "release-certification",
"lane": "identity",
"case_id": "startup",
"product_commit": "3333333333333333333333333333333333333333",
"harness_commit": "1111111111111111111111111111111111111111",
"target_fingerprint": "rustfs-9.9.9-fixture-linux-x86_64-gnu",
"platform": {
"os": "linux",
"arch": "x86_64",
"abi": "gnu"
},
"started_at": "2026-09-26T10:01:00Z",
"finished_at": "2026-09-26T10:01:30Z",
"state": "skipped",
"reason_code": "fixture-skipped",
"reason_class": "NOT_APPLICABLE",
"evidence": [
{
"path": "cases/identity/startup-linux-x86_64-gnu.json",
"sha256": "8b8774175c55c400b337610a89fe725ad2637af8fcc4973a82ab218f9182b80d",
"size": 64
}
],
"workflow": {
"repository": "rustfs/s3chaos",
"workflow_id": "release-gate.yml",
"run_id": "child-identity-linux-x86_64-gnu",
"run_attempt": 1,
"workflow_sha": "1111111111111111111111111111111111111111"
},
"artifact": {
"archive_sha256": "ea3ac1fb0ec4aa8fbec650f64ccdbc2543843c8d5734c3915d1b57d7cbb92ec8",
"binary_sha256": "7b77e78cc7b8f63a60f8a7101bebed6502f82e03016f4d4e11450f927e0a52fa",
"container_image_digest": "sha256:5555555555555555555555555555555555555555555555555555555555555555"
}
}
@@ -0,0 +1,42 @@
{
"schema_version": 1,
"evidence_profile": "release-artifact",
"run_id": "fixture-batch-1",
"run_attempt": 2,
"plan_sha256": "578bb57e65df8ba7a63e40aafc254fa644163270b7ec288fe3368bcd7c10ff86",
"tier": "release-certification",
"lane": "upgrade",
"case_id": "upgrade-smoke",
"product_commit": "3333333333333333333333333333333333333333",
"harness_commit": "2222222222222222222222222222222222222222",
"target_fingerprint": "rustfs-9.9.9-fixture-linux-x86_64-gnu",
"platform": {
"os": "linux",
"arch": "x86_64",
"abi": "gnu"
},
"started_at": "2026-09-26T10:04:00Z",
"finished_at": "2026-09-26T10:04:30Z",
"state": "pass",
"reason_code": "ok",
"reason_class": "NOT_APPLICABLE",
"evidence": [
{
"path": "cases/upgrade/upgrade-smoke-linux-x86_64-gnu.json",
"sha256": "c1ce56808b52859d2590b34b5990aed7bf560b134e44bd0c09f1873c9f764b38",
"size": 64
}
],
"workflow": {
"repository": "rustfs/s3chaos",
"workflow_id": "release-gate.yml",
"run_id": "child-upgrade-linux-x86_64-gnu",
"run_attempt": 1,
"workflow_sha": "2222222222222222222222222222222222222222"
},
"artifact": {
"archive_sha256": "ea3ac1fb0ec4aa8fbec650f64ccdbc2543843c8d5734c3915d1b57d7cbb92ec8",
"binary_sha256": "7b77e78cc7b8f63a60f8a7101bebed6502f82e03016f4d4e11450f927e0a52fa",
"container_image_digest": "sha256:5555555555555555555555555555555555555555555555555555555555555555"
}
}
+6
View File
@@ -0,0 +1,6 @@
{
"schema_version": 1,
"run_id": "fixture-batch-1",
"run_attempt": 2,
"status": "completed"
}
@@ -0,0 +1,42 @@
{
"schema_version": 1,
"evidence_profile": "release-artifact",
"run_id": "fixture-batch-1",
"run_attempt": 2,
"plan_sha256": "578bb57e65df8ba7a63e40aafc254fa644163270b7ec288fe3368bcd7c10ff86",
"tier": "release-certification",
"lane": "identity",
"case_id": "artifact-identity",
"product_commit": "3333333333333333333333333333333333333333",
"harness_commit": "1111111111111111111111111111111111111111",
"target_fingerprint": "rustfs-9.9.9-fixture-linux-aarch64-gnu",
"platform": {
"os": "linux",
"arch": "aarch64",
"abi": "gnu"
},
"started_at": "2026-09-26T10:02:00Z",
"finished_at": "2026-09-26T10:02:30Z",
"state": "pass",
"reason_code": "ok",
"reason_class": "NOT_APPLICABLE",
"evidence": [
{
"path": "cases/identity/artifact-identity-linux-aarch64-gnu.json",
"sha256": "fa138d5891e5098b97137b0edbb4779c4350285fdc76cec7e6214809b34b3075",
"size": 64
}
],
"workflow": {
"repository": "rustfs/s3chaos",
"workflow_id": "release-gate.yml",
"run_id": "child-identity-linux-aarch64-gnu",
"run_attempt": 1,
"workflow_sha": "1111111111111111111111111111111111111111"
},
"artifact": {
"archive_sha256": "ea3ac1fb0ec4aa8fbec650f64ccdbc2543843c8d5734c3915d1b57d7cbb92ec8",
"binary_sha256": "7b77e78cc7b8f63a60f8a7101bebed6502f82e03016f4d4e11450f927e0a52fa",
"container_image_digest": "sha256:5555555555555555555555555555555555555555555555555555555555555555"
}
}

Some files were not shown because too many files have changed in this diff Show More