Compare commits

...

11 Commits

Author SHA1 Message Date
Zhengchao An 7e071dc0fa test(site-replication): cover retry endpoint matching 2026-08-16 06:17:58 +08:00
Zhengchao An e117a5d09f test(site-replication): preserve collapsed retry liability 2026-08-16 06:14:35 +08:00
Zhengchao An a26e97bbff fix(site-replication): gate test-only retry helper 2026-08-16 05:34:49 +08:00
overtrue 96a2f9c67f fix(site-replication): preserve retry accountability 2026-08-16 04:54:11 +08:00
唐小鸭 af802756c5 fix(site-replication): only a repair settles snapshot-escalated retry entries
Second review round: every iam-item / bucket-meta delivery shares a
constant path, so any later successful single-item delivery (a Bob
update) dequeued the escalated marker recording a possibly-unreplayed
deletion (a failed Alice delete) while the entity still existed
remotely. Ordinary settlement now skips escalated entries; only the
repair path — the operator's explicit accountability transfer — clears
them via dequeue_..._including_escalated. A new hook failure still
overwrites the marker and re-arms the drain. Regression covers
survive-ordinary-dequeue and repair-clears.
2026-08-16 01:02:42 +08:00
唐小鸭 971addca6e fix(site-replication): escalate snapshot-replayed retry entries instead of clearing them
Review: the bootstrap-plan snapshot cannot replay deletions — a deleted
IAM entity or absent bucket config produces no task, so clearing the
collapsed iam-item / bucket-meta entry after a successful snapshot
resend silently lost a failed delete and the peer kept stale state
permanently.

The drain now keeps those entries until remote absence is proven:
after a successful snapshot resend the entry is escalated
(failed=true, marker last_error) so it stays operator-visible in
retry_stats, and classification skips marked entries so the
once-per-failure-episode snapshot is not re-sent daily. A newer hook
failure overwrites the marker and re-arms the drain; a later full
delivery or a manual repair settles the entry. Escalation is
conditional on the snapshot timestamp, preserving the earlier
review's in-flight-failure guarantee. Bucket ops and peer edits keep
auto-settle — their replays are faithful.
2026-08-15 19:14:23 +08:00
唐小鸭 2c7d1f1f9f fix(site-replication): harden the retry drain against review findings
Adversarial review of the drain surfaced one real race and three cheap
hardenings:

- Conditional settlement for collapsed (constant-path) iam-item /
  bucket-meta entries: the snapshot resend proves delivery of the state
  as of plan-build time, so a hook failure stamped during the delivery
  window (a newer local commit the snapshot did not contain) must
  survive the snapshot's success instead of being cleared — previously
  the peer would silently diverge until the next same-path failure.
  (The operator repair path shares this collapse but keeps its existing
  unconditional settle; the drain runs every tick and needed the guard
  first.)
- Re-check the pending_* gates on the freshly loaded state: an endpoint
  refresh can commit its pending marker mid-tick without the lifecycle
  guard.
- Do not upsert retry events for peers that already left the state —
  remove_sites pruned their entries and they can never drain again.
- Correct the mutual-exclusion comment: the repair-execution lock does
  NOT close the dry-run -> execute window (dry-run takes no lock); that
  window fails safe via the preflight token, which hashes the
  replayable entries.

New tests pin the (Some settled, None failed) generation-settle
combination the peer-edit drain depends on, and the snapshot-relative
conditional settlement.
2026-08-15 11:01:58 +08:00
唐小鸭 5328e8b958 feat(site-replication): drain the retry queue from the reconcile tick
The retry queue recorded every failed peer delivery but had no
consumer — missed IAM/bucket metadata updates drifted until an operator
ran a manual repair (backlog#1675 P1-3). The 600s reconcile tick now
drains the queue behind the existing lifecycle guard and pending_*
gates.

Replay discipline (pinned by the red-light tests):
- IAM / bucket-meta entries collapse per (peer, path) and their bodies
  are not persisted, so the only faithful replay is the current
  bootstrap-plan snapshot (SiteReplicationRepairTask reuse) sent to the
  affected peer.
- make-with-versioning / configure-replication ops are re-derived from
  the CURRENT plan for their bucket — never the recorded path, whose
  query can carry an expired one-shot bootstrap token; an op whose
  bucket left the plan is provably stale and is settled.
- Peer edits are re-sent as the current peer records under a freshly
  allocated generation inside one state transaction — the recorded
  generation is stale by definition and the receiver would fence it.
- Destructive bucket ops (delete-bucket / force-delete-bucket) and
  internal: marker records (pending-endpoint-refresh backup store) are
  never background-replayed.
- Exponential backoff (600s * 2^(n-1), 24h ceiling) gates each attempt;
  an unreachable peer's transport failure re-queues its events so the
  backoff still advances.

Concurrency: the drain takes the repair-execution config lock — the
operator repair preflight token hashes the replayable retry events, so
settling them between dry-run and execute would strand the operator on
a stale preflight. Lock order matches repair (lifecycle guard -> repair
execution lock -> state object lock); success/failure settlement reuses
the generation-fenced upsert/settle semantics from #5882/#6097.

Scope note: the drain settles 'delivered once and failed' entries. A
hook that never fired (crash between local commit and send) leaves no
entry; a low-frequency plan-diff catch-all remains follow-up work, and
manual repair stays authoritative.
2026-08-15 10:42:36 +08:00
唐小鸭 1e16e06f8a test(site-replication): pin the background retry-drain discipline
Red-light evidence for backlog#1675 P1-3: the retry queue has no
background consumer — every failed peer delivery waits for a manual
repair. The new tests specify the drain rules before the drain exists:

- classification: IAM / bucket-meta entries collapse per (peer, path)
  with no persisted body, so only a bootstrap-plan snapshot resend is a
  faithful replay; make-with-versioning / configure-replication are
  re-derivable per bucket; peer edits are re-sent under a fresh
  generation; destructive bucket ops and internal: marker records
  (pending-endpoint-refresh backup store) are never background-replayed
- exponential backoff (600s * 2^(n-1), 24h ceiling) gates every
  attempt, otherwise a dead peer's entries hit the failed threshold
  within 30 minutes of reconcile ticks
- the actionable subset respects classification, current peer
  membership and backoff

All fail against the placeholder implementations (no consumer).
2026-08-15 10:19:28 +08:00
Zhengchao An 72fd7339c9 test(utils): allow ephemeral port reuse (#6122)
* test(utils): allow ephemeral port reuse

* test(kms): allow any ciphertext prefix
2026-08-15 08:32:10 +08:00
Zhengchao An 71e83aeec4 fix(ci): pin Docker images to release source (#6121) 2026-08-15 07:13:37 +08:00
6 changed files with 1083 additions and 18 deletions
+25 -1
View File
@@ -94,6 +94,7 @@ jobs:
short_sha: ${{ steps.check.outputs.short_sha }}
is_prerelease: ${{ steps.check.outputs.is_prerelease }}
create_latest: ${{ steps.check.outputs.create_latest }}
source_ref: ${{ steps.check.outputs.source_ref }}
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
@@ -118,6 +119,7 @@ jobs:
short_sha=""
is_prerelease=false
create_latest=false
source_ref="$GITHUB_SHA"
if [[ "${{ github.event_name }}" == "workflow_run" ]]; then
# Triggered by build workflow completion
@@ -137,6 +139,7 @@ jobs:
# Extract version info from commit message or use commit SHA
# Use Git to generate consistent short SHA (ensures uniqueness like build.yml)
short_sha=$(git rev-parse --short "$HEAD_SHA")
source_ref="$HEAD_SHA"
# Determine build type based on triggering workflow event and ref
triggering_event="$TRIGGERING_EVENT"
@@ -261,6 +264,23 @@ jobs:
echo "⚠️ Only release versions (latest, v1.0.0, 1.0.0) and prereleases (v1.0.0-alpha1, 1.0.0-beta2) are supported"
;;
esac
if [[ "$should_build" == true && "$input_version" != "latest" ]]; then
tag_ref="refs/tags/$input_version"
if ! git ls-remote --exit-code origin "$tag_ref" >/dev/null 2>&1; then
if [[ "$input_version" == v* ]]; then
tag_ref="refs/tags/${input_version#v}"
else
tag_ref="refs/tags/v$input_version"
fi
fi
if ! git ls-remote --exit-code origin "$tag_ref" >/dev/null 2>&1; then
echo "❌ Release tag not found for Docker build: $input_version"
exit 1
fi
source_ref="$tag_ref"
fi
fi
{
@@ -271,6 +291,7 @@ jobs:
echo "short_sha=$short_sha"
echo "is_prerelease=$is_prerelease"
echo "create_latest=$create_latest"
echo "source_ref=$source_ref"
} >> "$GITHUB_OUTPUT"
echo "🐳 Docker Build Summary:"
@@ -281,6 +302,7 @@ jobs:
echo " - Short SHA: $short_sha"
echo " - Is prerelease: $is_prerelease"
echo " - Create latest: $create_latest"
echo " - Source ref: $source_ref"
# Build multi-arch Docker images
# Strategy: Build images using pre-built binaries from dl.rustfs.com
@@ -308,6 +330,7 @@ jobs:
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
ref: ${{ needs.build-check.outputs.source_ref }}
- name: Login to Docker Hub
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
@@ -397,7 +420,8 @@ jobs:
LABELS="org.opencontainers.image.title=RustFS"
LABELS="$LABELS,org.opencontainers.image.description=RustFS distributed object storage system"
LABELS="$LABELS,org.opencontainers.image.version=$VERSION"
LABELS="$LABELS,org.opencontainers.image.revision=${{ github.sha }}"
SOURCE_REVISION="$(git rev-parse HEAD)"
LABELS="$LABELS,org.opencontainers.image.revision=$SOURCE_REVISION"
LABELS="$LABELS,org.opencontainers.image.source=${{ github.server_url }}/${{ github.repository }}"
LABELS="$LABELS,org.opencontainers.image.created=$(date -u +'%Y-%m-%dT%H:%M:%SZ')"
LABELS="$LABELS,org.opencontainers.image.build-type=$BUILD_TYPE"
-2
View File
@@ -225,8 +225,6 @@ async fn nothing_readable_leaves_the_bundle_unwrapped() {
"artifact {} carries the raw on-disk record",
artifact.path
);
// A cheap structural check too: an encrypted payload is not JSON.
assert_ne!(payload.first(), Some(&b'{'), "artifact {} looks like plaintext JSON", artifact.path);
}
// The manifest itself is not encrypted, so assert directly that it carries
+3 -3
View File
@@ -258,7 +258,7 @@ pub struct SRLDAPUser {
pub api_version: Option<String>,
}
#[derive(Debug, Serialize, Deserialize, Default)]
#[derive(Debug, Clone, Serialize, Deserialize, Default)]
pub struct SRIAMUser {
#[serde(rename = "accessKey", default)]
pub access_key: String,
@@ -270,7 +270,7 @@ pub struct SRIAMUser {
pub api_version: Option<String>,
}
#[derive(Debug, Serialize, Deserialize, Default)]
#[derive(Debug, Clone, Serialize, Deserialize, Default)]
pub struct SRGroupInfo {
#[serde(rename = "updateReq", default)]
pub update_req: GroupAddRemove,
@@ -346,7 +346,7 @@ pub struct SRCredInfo {
pub api_version: Option<String>,
}
#[derive(Debug, Serialize, Deserialize, Default)]
#[derive(Debug, Clone, Serialize, Deserialize, Default)]
pub struct SRIAMItem {
#[serde(default)]
pub r#type: String,
-3
View File
@@ -659,9 +659,6 @@ mod test {
// Port should be in valid range (u16 max is always <= 65535)
assert!(port1 > 0);
assert!(port2 > 0);
// Different calls should typically return different ports
assert_ne!(port1, port2);
}
#[test]
File diff suppressed because it is too large Load Diff
@@ -195,6 +195,13 @@ IFS= read -r -d '' expected_docker_automatic_guard <<'EOF' || true
EOF
expected_docker_automatic_guard=${expected_docker_automatic_guard%$'\n'}
require_job_if "$docker_workflow" "build-check" "$expected_docker_automatic_guard"
require_line "$docker_workflow" ' source_ref: ${{ steps.check.outputs.source_ref }}' "Docker source ref output"
require_line "$docker_workflow" ' source_ref="$HEAD_SHA"' "automatic Docker source ref"
require_line "$docker_workflow" ' source_ref="$tag_ref"' "manual Docker source ref"
require_line "$docker_workflow" ' ref: ${{ needs.build-check.outputs.source_ref }}' "Docker release source checkout"
require_line "$docker_workflow" ' SOURCE_REVISION="$(git rev-parse HEAD)"' "Docker source revision resolution"
require_line "$docker_workflow" ' LABELS="$LABELS,org.opencontainers.image.revision=$SOURCE_REVISION"' "Docker revision label"
require_absent "$docker_workflow" 'org.opencontainers.image.revision=${{ github.sha }}' "Docker revision must not use the workflow branch SHA"
docker_manual_guard=$(awk '
$0 == " *-preview*)" { in_preview = 1 }