Compare commits

...

9 Commits

Author SHA1 Message Date
loverustfs 43a7f94f14 fix(scanner): preserve failed usage and reliable heal sampling 2026-09-05 09:57:10 +08:00
Zhengchao An 9ed1d46090 docs(ecstore): define generation authority and recovery boundary (#7159) 2026-09-05 09:29:00 +08:00
cxymds 6eb60f8e72 feat(tier): add durable probe intent protocol (#7151)
* feat(tier): add durable probe intent protocol

* test(tier): remove redundant intent clones
2026-09-05 09:06:05 +08:00
cxymds 8dd3cabd41 test(ecstore): stabilize transition generation fixture (#7140) 2026-09-05 00:46:19 +00:00
Zhengchao An e648f683bf fix: use BTreeMap for deterministic encryption context serialization (#7154)
SealScope::encryption_context() returned a HashMap whose key order is
non-deterministic. The FakeSealer test round-trips the context through
JSON serialization, and HashMap's random iteration order caused the
prefix comparison to intermittently fail with 'encryption context mismatch'.

Switch to BTreeMap which guarantees stable key ordering.
2026-09-05 07:21:25 +08:00
cxymds 193b1b7d3f test(ecstore): stabilize sealed context encoding (#7149) 2026-09-05 07:18:37 +08:00
hector 3b920c7999 ci: render step results and version in workflow reports (#7141)
* ci(upgrade): render an upgrade matrix in the report; fix from default

The upgrade report only ever showed the requested deb URLs and a case
table. The nightly chain runs died installing the OLD package (default
from_version 1.0.0-rc.4-preview.1 has no .deb asset on its release, and
release 1.0.0-rc.4 ships none either), leaving an empty Total: 0 report
with no indication of what was upgraded.

- Default from_version is now 1.0.0-rc.3 (ships rustfs_1.0.0.rc.3_amd64.deb).
  Matches the auto-testing default from PR #32.
- The report generator also parses the [UPG-TOPO] lines the suite now
  emits and renders an 'Upgrade Matrix' section: per topology and KMS
  backend, the versions actually in place before/after (captured via
  'rustfs --version' on the node) and the aggregated result. When the
  suite dies before any topology completes, the matrix says so instead
  of silently showing nothing.

* fix(ci): use English headers in the upgrade matrix table

* ci(heal,pool): render step results and version in the reports

The heal and pool-expansion reports were only a raw log tail: no
structured indication of which steps passed, no overall verdict, and no
version information for the cluster under test.

The suites now emit machine-readable lines (auto-testing PR):
  [HEAL-STEP] <n> <desc> PASS|FAIL     [POOL-STEP] <n> <desc> PASS|FAIL
  [HEAL-VERSION] <ver> (node <n>)      [POOL-VERSION] <ver> (node <n>)
  [HEAL-RESULT] PASS|FAIL <detail>     [POOL-RESULT] PASS|FAIL <detail>

Both report generators parse them and emit a '## Step Results' section
before the log tail: the version captured in place via 'rustfs --version'
on a node, the overall verdict, and a per-step table. When a run dies
before any step reports (old script or early crash), the table shows a
NOT RUN placeholder row instead of silently showing nothing.
2026-09-05 02:28:14 +08:00
cxymds 5f8b097172 fix(tier): bound distributed mutation latency (#7150) 2026-09-05 02:28:08 +08:00
cxymds 445114577f docs(ilm): approve bounded recovery disposition (#7155) 2026-09-05 02:28:02 +08:00
19 changed files with 4079 additions and 320 deletions
+56
View File
@@ -152,6 +152,60 @@ jobs:
else
PACKAGE_SOURCE="${RUSTFS_NIGHTLY_PACKAGE_URL}"
fi
STEPS_TABLE="/tmp/rustfs-heal-steps.md"
python3 - "${LOG_FILE}" "${STEPS_TABLE}" <<'PY'
import re
import sys
log_file, out_file = sys.argv[1], sys.argv[2]
ansi = re.compile(r'\x1b\[[0-9;]*m')
step_re = re.compile(r'^\[HEAL-STEP\]\s+(\d+)\s+(.+?)\s+(PASS|FAIL|SKIP)\s*$')
ver_re = re.compile(r'^\[HEAL-VERSION\]\s+(\S+)(?:\s+\(node\s+(\S+)\))?\s*$')
result_re = re.compile(r'^\[HEAL-RESULT\]\s+(PASS|FAIL)\s+(.*)$')
steps = {}
order = []
version = None
version_node = None
verdict = None
verdict_detail = ''
try:
with open(log_file, 'r', encoding='utf-8', errors='replace') as fh:
for raw in fh:
line = ansi.sub('', raw).strip()
m = step_re.match(line)
if m:
n, desc, status = m.group(1), m.group(2), m.group(3)
if n not in steps:
order.append(n)
steps[n] = (desc, status) # later lines win (fail after pass)
continue
m = ver_re.match(line)
if m:
version, version_node = m.group(1), m.group(2)
continue
m = result_re.match(line)
if m:
verdict, verdict_detail = m.group(1), m.group(2)
except FileNotFoundError:
pass
with open(out_file, 'w', encoding='utf-8') as out:
out.write('## Step Results\n\n')
if version:
node_note = f' (captured via `rustfs --version` on {version_node})' if version_node else ''
out.write(f'- Version under test: **{version}**{node_note}\n')
if verdict:
out.write(f'- Overall result: **{verdict}** — {verdict_detail}\n')
out.write('\n')
out.write('| Step | Description | Result |\n')
out.write('| --- | --- | --- |\n')
for n in sorted(order, key=int):
desc, status = steps[n]
out.write(f'| {n} | {desc} | {status} |\n')
if not order:
out.write('| - | - | NOT RUN (no step result lines found) |\n')
PY
{
echo "# RustFS heal test report"
echo ""
@@ -160,6 +214,8 @@ jobs:
echo "- Package: ${PACKAGE_SOURCE}"
echo "- Test Step Outcome: ${{ steps.test.outcome }}"
echo ""
cat "${STEPS_TABLE}" || true
echo ""
echo "## Log tail"
echo '```text'
tail -n 200 "${LOG_FILE}" || true
@@ -380,6 +380,60 @@ jobs:
else
PACKAGE_SOURCE="${RUSTFS_NIGHTLY_PACKAGE_URL}"
fi
STEPS_TABLE="${POOL_ARTIFACT_DIR}/pool-steps.md"
python3 - "${LOG_FILE}" "${STEPS_TABLE}" <<'PY'
import re
import sys
log_file, out_file = sys.argv[1], sys.argv[2]
ansi = re.compile(r'\x1b\[[0-9;]*m')
step_re = re.compile(r'^\[POOL-STEP\]\s+(\d+)\s+(.+?)\s+(PASS|FAIL|SKIP)\s*$')
ver_re = re.compile(r'^\[POOL-VERSION\]\s+(\S+)(?:\s+\(node\s+(\S+)\))?\s*$')
result_re = re.compile(r'^\[POOL-RESULT\]\s+(PASS|FAIL)\s+(.*)$')
steps = {}
order = []
version = None
version_node = None
verdict = None
verdict_detail = ''
try:
with open(log_file, 'r', encoding='utf-8', errors='replace') as fh:
for raw in fh:
line = ansi.sub('', raw).strip()
m = step_re.match(line)
if m:
n, desc, status = m.group(1), m.group(2), m.group(3)
if n not in steps:
order.append(n)
steps[n] = (desc, status) # later lines win (fail after pass)
continue
m = ver_re.match(line)
if m:
version, version_node = m.group(1), m.group(2)
continue
m = result_re.match(line)
if m:
verdict, verdict_detail = m.group(1), m.group(2)
except FileNotFoundError:
pass
with open(out_file, 'w', encoding='utf-8') as out:
out.write('## Step Results\n\n')
if version:
node_note = f' (captured via `rustfs --version` on {version_node})' if version_node else ''
out.write(f'- Version under test: **{version}**{node_note}\n')
if verdict:
out.write(f'- Overall result: **{verdict}** — {verdict_detail}\n')
out.write('\n')
out.write('| Step | Description | Result |\n')
out.write('| --- | --- | --- |\n')
for n in sorted(order, key=int):
desc, status = steps[n]
out.write(f'| {n} | {desc} | {status} |\n')
if not order:
out.write('| - | - | NOT RUN (no step result lines found) |\n')
PY
{
echo "# RustFS pool expansion test report"
echo ""
@@ -389,6 +443,8 @@ jobs:
echo "- Warp concurrent: ${{ inputs.warp_concurrent || '32' }}"
echo "- Test Step Outcome: ${{ steps.pool_test.outcome }}"
echo ""
cat "${STEPS_TABLE}" || true
echo ""
echo "## Log tail"
echo '```text'
tail -n 200 "${LOG_FILE}" || true
+57 -34
View File
@@ -18,9 +18,9 @@ on:
workflow_dispatch:
inputs:
from_version:
description: 'OLD RustFS release tag (e.g. 1.0.0-rc.4-preview.1)'
description: 'OLD RustFS release tag (must ship a .deb asset, e.g. 1.0.0-rc.3)'
required: false
default: '1.0.0-rc.4-preview.1'
default: '1.0.0-rc.3'
from_url:
description: 'OLD .deb URL. Overrides from_version.'
required: false
@@ -203,54 +203,75 @@ jobs:
TO_SOURCE="${RUSTFS_NIGHTLY_PACKAGE_URL}"
fi
CASE_TABLE="/tmp/rustfs-upgrade-cases.md"
python3 - "${LOG_FILE}" "${CASE_TABLE}" <<'PY'
MATRIX_TABLE="/tmp/rustfs-upgrade-matrix.md"
python3 - "${LOG_FILE}" "${CASE_TABLE}" "${MATRIX_TABLE}" <<'PY'
import re
import sys
log_file, out_file = sys.argv[1], sys.argv[2]
log_file, out_file, matrix_file = sys.argv[1], sys.argv[2], sys.argv[3]
ansi = re.compile(r'\x1b\[[0-9;]*m')
start_re = re.compile(r'^---\s+([A-Z]+-[0-9]+)\s+(.+?)\s+---$')
done_re = re.compile(r'^\[(PASS|FAIL|UNSUPPORTED)\]\s+([A-Z]+-[0-9]+)\b')
topo_re = re.compile(
r'^\[UPG-TOPO\]\s+(\S+)\s+(\S+)\s+(\S+)\s+(\S+)\s+PASS=(\d+)\s+FAIL=(\d+)\s*$')
rows = []
index = {}
topo_rows = []
try:
with open(log_file, 'r', encoding='utf-8', errors='replace') as fh:
for raw in fh:
line = ansi.sub('', raw).strip()
m = start_re.match(line)
if m:
case_id, name = m.group(1), m.group(2)
if case_id not in index:
index[case_id] = len(rows)
rows.append([case_id, name, 'RUNNING'])
continue
m = done_re.match(line)
if m:
status, case_id = m.group(1), m.group(2)
if case_id in index:
rows[index[case_id]][2] = status
else:
rows.append([case_id, case_id, status])
index[case_id] = len(rows) - 1
with open(log_file, 'r', encoding='utf-8', errors='replace') as fh:
for raw in fh:
line = ansi.sub('', raw).strip()
m = topo_re.match(line)
if m:
topo_rows.append(m.groups())
continue
m = start_re.match(line)
if m:
case_id, name = m.group(1), m.group(2)
if case_id not in index:
index[case_id] = len(rows)
rows.append([case_id, name, 'RUNNING'])
continue
m = done_re.match(line)
if m:
status, case_id = m.group(1), m.group(2)
if case_id in index:
rows[index[case_id]][2] = status
else:
rows.append([case_id, case_id, status])
index[case_id] = len(rows) - 1
except FileNotFoundError:
rows = []
rows = []
counts = {'PASS': 0, 'FAIL': 0, 'UNSUPPORTED': 0, 'RUNNING': 0}
for _, _, status in rows:
counts[status] = counts.get(status, 0) + 1
counts[status] = counts.get(status, 0) + 1
with open(out_file, 'w', encoding='utf-8') as out:
out.write('## Case Summary\n\n')
out.write(f"- Total: {len(rows)}\\n")
out.write(f"- PASS: {counts.get('PASS', 0)}\\n")
out.write(f"- FAIL: {counts.get('FAIL', 0)}\\n")
out.write(f"- UNSUPPORTED: {counts.get('UNSUPPORTED', 0)}\\n")
out.write('\\n')
out.write('| Case | Name | Status |\\n')
out.write('| --- | --- | --- |\\n')
for case_id, name, status in rows:
out.write(f'| {case_id} | {name} | {status} |\\n')
out.write('## Case Summary\n\n')
out.write(f"- Total: {len(rows)}\\n")
out.write(f"- PASS: {counts.get('PASS', 0)}\\n")
out.write(f"- FAIL: {counts.get('FAIL', 0)}\\n")
out.write(f"- UNSUPPORTED: {counts.get('UNSUPPORTED', 0)}\\n")
out.write('\\n')
out.write('| Case | Name | Status |\\n')
out.write('| --- | --- | --- |\\n')
for case_id, name, status in rows:
out.write(f'| {case_id} | {name} | {status} |\\n')
# Upgrade matrix: one row per topology/backend with the versions
# captured on the nodes (rustfs --version) and the aggregated
# result. The dashboard renders this table directly.
with open(matrix_file, 'w', encoding='utf-8') as out:
out.write('## Upgrade Matrix\n\n')
out.write('| Topology | KMS Backend | From Version | To Version | Result |\n')
out.write('| --- | --- | --- | --- | --- |\n')
for topo, backend, old_v, new_v, npass, nfail in topo_rows:
result = 'PASS' if nfail == '0' else 'FAIL'
out.write(f'| {topo} | {backend} | {old_v} | {new_v} | {result} (PASS={npass} FAIL={nfail}) |\n')
if not topo_rows:
out.write('| - | - | - | - | NOT RUN (suite failed before upgrade) |\n')
PY
{
echo "# RustFS upgrade compatibility report"
@@ -261,6 +282,8 @@ jobs:
echo "- To: ${TO_SOURCE}"
echo "- Test Step Outcome: ${{ steps.test.outcome }}"
echo ""
cat "${MATRIX_TABLE}" || true
echo ""
cat "${CASE_TABLE}" || true
echo ""
echo "## Log tail"
@@ -25,6 +25,7 @@ use super::{
manual_transition_job, tier_delete_journal, transition_transaction,
};
use crate::error::{Error, Result};
use crate::services::tier::tier_probe_intent;
pub(crate) const ILM_META_PREFIX: &str = "ilm";
const ILM_META_OBJECT_PREFIX: &str = "ilm/";
@@ -35,6 +36,7 @@ pub(crate) enum DurableIlmRecordKind {
TierDeleteJournal,
TierDeleteDispatchManifest,
TransitionTransaction,
TierProbeIntent,
ManualTransitionJob,
ManualTransitionScope,
ManualTransitionTask,
@@ -73,6 +75,12 @@ pub(crate) const TRANSITION_TRANSACTION_NAMESPACE: DurableIlmNamespace = Durable
max_record_size: transition_transaction::MAX_TRANSITION_TRANSACTION_SIZE,
kind: DurableIlmRecordKind::TransitionTransaction,
};
pub(crate) const TIER_PROBE_INTENT_NAMESPACE: DurableIlmNamespace = DurableIlmNamespace {
name: "tier-probe-intent",
prefix: tier_probe_intent::TIER_PROBE_INTENT_RECORD_PREFIX,
max_record_size: tier_probe_intent::MAX_TIER_PROBE_INTENT_SIZE,
kind: DurableIlmRecordKind::TierProbeIntent,
};
pub(crate) const MANUAL_TRANSITION_JOB_NAMESPACE: DurableIlmNamespace = DurableIlmNamespace {
name: "manual-transition-job",
prefix: "ilm/manual-transition/jobs",
@@ -98,11 +106,12 @@ pub(crate) const MANUAL_TRANSITION_WORKER_RESULT_NAMESPACE: DurableIlmNamespace
kind: DurableIlmRecordKind::ManualTransitionWorkerResult,
};
pub(crate) const DURABLE_ILM_NAMESPACES: [DurableIlmNamespace; 8] = [
pub(crate) const DURABLE_ILM_NAMESPACES: [DurableIlmNamespace; 9] = [
TIER_DELETE_JOURNAL_NAMESPACE,
TIER_DELETE_JOURNAL_V6_NAMESPACE,
TIER_DELETE_DISPATCH_MANIFEST_NAMESPACE,
TRANSITION_TRANSACTION_NAMESPACE,
TIER_PROBE_INTENT_NAMESPACE,
MANUAL_TRANSITION_JOB_NAMESPACE,
MANUAL_TRANSITION_SCOPE_NAMESPACE,
MANUAL_TRANSITION_TASK_NAMESPACE,
@@ -200,6 +209,15 @@ pub(crate) enum DurableIlmRecordCheckpoint {
revision: u64,
state: transition_transaction::TransitionTransactionState,
},
TierProbeIntent {
content_sha256: String,
identity_sha256: String,
remote_version_sha256: String,
remote_version_known: bool,
owner_fence_sha256: String,
revision: u64,
state: tier_probe_intent::TierProbeIntentState,
},
ManualTransitionJob {
content_sha256: String,
identity_sha256: String,
@@ -232,6 +250,7 @@ impl DurableIlmRecordCheckpoint {
| Self::TierDeleteDispatchManifest { content_sha256, .. }
| Self::TierDeleteDispatchParent { content_sha256, .. }
| Self::TransitionTransaction { content_sha256, .. }
| Self::TierProbeIntent { content_sha256, .. }
| Self::ManualTransitionJob { content_sha256, .. }
| Self::ManualTransitionScope { content_sha256, .. }
| Self::ManualTransitionTask { content_sha256 }
@@ -421,6 +440,32 @@ impl DurableIlmRecordCheckpoint {
.is_some_and(|expected_revision| *next_revision == expected_revision)
&& (!previous_remote_version_known || previous_remote_version == next_remote_version)
}
(
Self::TierProbeIntent {
identity_sha256: previous_identity,
remote_version_sha256: previous_remote_version,
remote_version_known: previous_remote_version_known,
owner_fence_sha256: previous_owner_fence,
revision: previous_revision,
state: previous_state,
..
},
Self::TierProbeIntent {
identity_sha256: next_identity,
remote_version_sha256: next_remote_version,
owner_fence_sha256: next_owner_fence,
revision: next_revision,
state: next_state,
..
},
) => {
previous_identity == next_identity
&& previous_owner_fence == next_owner_fence
&& next_revision
.checked_sub(*previous_revision)
.is_some_and(|distance| distance == 1 && tier_probe_state_reaches(*previous_state, *next_state, distance))
&& (!previous_remote_version_known || previous_remote_version == next_remote_version)
}
(
Self::ManualTransitionJob {
content_sha256: previous_content,
@@ -500,6 +545,14 @@ impl DurableIlmRecordCheckpoint {
/// after the exact terminal ETag and terminal receipt were committed, to
/// purge older object versions exposed by that deletion.
pub(crate) fn is_predecessor_of_terminal(&self, terminal: &Self) -> bool {
if let Self::TierProbeIntent { state, .. } = terminal
&& !matches!(
state,
tier_probe_intent::TierProbeIntentState::AbortedNoRemote | tier_probe_intent::TierProbeIntentState::Completed
)
{
return false;
}
if self == terminal || self.validate_successor(terminal).is_ok() {
return true;
}
@@ -568,6 +621,37 @@ impl DurableIlmRecordCheckpoint {
}
})
}
(
Self::TierProbeIntent {
identity_sha256: previous_identity,
remote_version_sha256: previous_remote_version,
remote_version_known: previous_remote_version_known,
owner_fence_sha256: previous_owner_fence,
revision: previous_revision,
state: previous_state,
..
},
Self::TierProbeIntent {
identity_sha256: terminal_identity,
remote_version_sha256: terminal_remote_version,
owner_fence_sha256: terminal_owner_fence,
revision: terminal_revision,
state: terminal_state,
..
},
) => {
previous_identity == terminal_identity
&& previous_owner_fence == terminal_owner_fence
&& matches!(
terminal_state,
tier_probe_intent::TierProbeIntentState::AbortedNoRemote
| tier_probe_intent::TierProbeIntentState::Completed
)
&& terminal_revision
.checked_sub(*previous_revision)
.is_some_and(|distance| tier_probe_state_reaches(*previous_state, *terminal_state, distance))
&& (!previous_remote_version_known || previous_remote_version == terminal_remote_version)
}
_ => false,
}
}
@@ -606,6 +690,23 @@ fn transition_state_distance(
}
}
fn tier_probe_state_reaches(
from: tier_probe_intent::TierProbeIntentState,
to: tier_probe_intent::TierProbeIntentState,
revision_distance: u64,
) -> bool {
use tier_probe_intent::TierProbeIntentState::{AbortedNoRemote, CleanupPending, Completed, UploadOutcomeUnknown, Uploaded};
match (from, to) {
(UploadOutcomeUnknown, Uploaded | CleanupPending | AbortedNoRemote) => revision_distance == 1,
(UploadOutcomeUnknown, Completed) => matches!(revision_distance, 2 | 3),
(Uploaded, CleanupPending) => revision_distance == 1,
(Uploaded, Completed) => revision_distance == 2,
(CleanupPending, Completed) => revision_distance == 1,
_ => false,
}
}
fn manual_job_state_reaches(
from: manual_transition_job::ManualTransitionJobState,
to: manual_transition_job::ManualTransitionJobState,
@@ -1082,6 +1183,42 @@ pub(crate) fn validate_durable_ilm_record(path: &str, data: &[u8]) -> Result<Val
},
)
}
DurableIlmRecordKind::TierProbeIntent => {
let probe_id = tier_probe_intent::tier_probe_intent_id_from_record_object_name(path)
.map_err(|err| Error::other(err.to_string()))?;
let intent =
tier_probe_intent::TierProbeIntent::decode(probe_id, data).map_err(|err| Error::other(err.to_string()))?;
let canonical =
tier_probe_intent::tier_probe_intent_record_object_name(probe_id).map_err(|err| Error::other(err.to_string()))?;
if canonical != path {
return Err(Error::other("tier probe intent path is not canonical"));
}
let identity_sha256 = checkpoint_hash(&(
intent.probe_id,
&intent.operation,
&intent.tier_name,
intent.destination_id,
&intent.probe_object,
&intent.creator_id,
intent.creator_epoch,
intent.created_at_unix_nanos,
))?;
let remote_version_sha256 = checkpoint_hash(&intent.remote_version)?;
let owner_fence_sha256 = checkpoint_hash(&intent.owner)?;
(
"probe_id",
probe_id.to_string(),
DurableIlmRecordCheckpoint::TierProbeIntent {
content_sha256,
identity_sha256,
remote_version_sha256,
remote_version_known: !intent.remote_version.is_unknown(),
owner_fence_sha256,
revision: intent.revision,
state: intent.state,
},
)
}
DurableIlmRecordKind::ManualTransitionJob => {
let job_id = manual_transition_job::manual_transition_job_id_from_record_object_name(path)
.map_err(|err| Error::other(err.to_string()))?;
@@ -1237,6 +1374,102 @@ mod tests {
}
}
fn tier_probe_intent_fixture() -> tier_probe_intent::TierProbeIntent {
let probe_id = Uuid::parse_str("36e2220e-9ad2-495b-b3bc-c4d2caf70a31").expect("fixture uuid should parse");
tier_probe_intent::TierProbeIntent {
probe_id,
revision: 1,
state: tier_probe_intent::TierProbeIntentState::UploadOutcomeUnknown,
operation: tier_probe_intent::TierProbeOperationIdentity::Verify {
config_etag: "config-etag".to_string(),
backend_identity: [1; 32],
},
tier_name: "COLD-A".to_string(),
destination_id: [1; 32],
probe_object: tier_probe_intent::tier_probe_object_name(probe_id),
creator_id: "node-a".to_string(),
creator_epoch: Uuid::parse_str("76746062-c05a-40b7-9e38-d2722d7e0332").expect("fixture creator epoch should parse"),
created_at_unix_nanos: 1_780_000_000_000_000_000,
owner: tier_probe_intent::TierProbeOwnerFence {
owner_id: "node-a".to_string(),
owner_epoch: Uuid::parse_str("76746062-c05a-40b7-9e38-d2722d7e0332").expect("fixture owner epoch should parse"),
not_after_unix_nanos: 1_780_000_900_000_000_000,
},
remote_version: tier_probe_intent::TierProbeRemoteVersion::default(),
}
}
fn tier_probe_checkpoint(intent: &tier_probe_intent::TierProbeIntent) -> DurableIlmRecordCheckpoint {
let path =
tier_probe_intent::tier_probe_intent_record_object_name(intent.probe_id).expect("tier probe path should build");
let encoded = intent.encode().expect("tier probe intent should encode");
let namespace = classify_durable_ilm_record(&path)
.expect("tier probe namespace should classify")
.expect("tier probe intent should be durable");
assert_eq!(namespace, &TIER_PROBE_INTENT_NAMESPACE);
validate_durable_ilm_record(&path, &encoded)
.expect("tier probe intent should validate")
.checkpoint
}
#[test]
fn tier_probe_intent_checkpoint_tracks_exact_monotonic_generations() {
let initial_intent = tier_probe_intent_fixture();
let initial = tier_probe_checkpoint(&initial_intent);
let mut uploaded_intent = initial_intent;
uploaded_intent
.advance(
tier_probe_intent::TierProbeIntentState::Uploaded,
tier_probe_intent::TierProbeRemoteVersion::versioned("opaque-v1"),
)
.expect("uploaded state should advance");
let uploaded = tier_probe_checkpoint(&uploaded_intent);
initial
.validate_successor(&uploaded)
.expect("durable receipt may adopt the exact uploaded generation");
let mut cleanup_intent = uploaded_intent.clone();
cleanup_intent
.advance(
tier_probe_intent::TierProbeIntentState::CleanupPending,
uploaded_intent.remote_version.clone(),
)
.expect("cleanup state should advance");
let cleanup = tier_probe_checkpoint(&cleanup_intent);
uploaded
.validate_successor(&cleanup)
.expect("durable receipt may adopt the exact cleanup generation");
let mut completed_intent = cleanup_intent.clone();
completed_intent
.advance(tier_probe_intent::TierProbeIntentState::Completed, cleanup_intent.remote_version.clone())
.expect("completed state should advance");
let completed = tier_probe_checkpoint(&completed_intent);
cleanup
.validate_successor(&completed)
.expect("durable receipt may adopt the exact terminal generation");
assert!(
initial.is_predecessor_of_terminal(&completed),
"terminal cleanup must recognize the full acknowledged-PUT path"
);
assert!(
initial.validate_successor(&completed).is_err(),
"ordinary receipt advancement must not skip intermediate generations"
);
assert!(
!initial.is_predecessor_of_terminal(&uploaded),
"a nonterminal generation must not be accepted as terminal proof"
);
let mut rebound = uploaded_intent;
rebound.owner.owner_epoch = Uuid::new_v4();
assert!(
rebound.encode().is_err(),
"dormant v1 must reject owner takeover before producing a checkpoint"
);
}
#[test]
fn tier_delete_dispatch_manifest_namespace_validates_monotonic_branches() {
use tier_delete_journal::TierDeleteDispatchManifestState::{Aborted, Aborting, Completed, DispatchAuthorized, Preparing};
@@ -28,7 +28,7 @@
use async_trait::async_trait;
use serde::{Deserialize, Serialize};
use std::collections::HashMap;
use std::collections::BTreeMap;
use std::fmt;
use std::sync::{Arc, OnceLock};
@@ -81,8 +81,8 @@ impl SealScope {
/// The encryption context handed to the sealer. Keys are stable: they are
/// part of the on-disk contract, because a ciphertext only decrypts under
/// the same context.
pub fn encryption_context(&self) -> HashMap<String, String> {
HashMap::from([
pub fn encryption_context(&self) -> BTreeMap<String, String> {
BTreeMap::from([
("rustfs:store".to_string(), self.store.as_str().to_string()),
("rustfs:owner".to_string(), self.owner.clone()),
("rustfs:field".to_string(), self.field.to_string()),
@@ -201,12 +201,18 @@ pub async fn unseal_secret(sealed: &SealedCredential, scope: &SealScope) -> Resu
mod tests {
use super::*;
use parking_lot::Mutex;
use std::collections::BTreeMap;
fn encode_context(context: &HashMap<String, String>) -> String {
let ordered = context.iter().collect::<BTreeMap<_, _>>();
serde_json::to_string(&ordered).expect("context serializes")
}
/// Stands in for the KMS-backed sealer: records the context it was called
/// with, and refuses a ciphertext presented under a different one.
#[derive(Default)]
struct FakeSealer {
sealed_contexts: Mutex<Vec<HashMap<String, String>>>,
sealed_contexts: Mutex<Vec<BTreeMap<String, String>>>,
}
#[async_trait]
@@ -214,7 +220,7 @@ mod tests {
async fn seal(&self, plaintext: &str, scope: &SealScope) -> Result<SealedCredential, SealedCredentialError> {
let context = scope.encryption_context();
self.sealed_contexts.lock().push(context.clone());
let mut bound = serde_json::to_string(&context).expect("context serializes");
let mut bound = encode_context(&context);
bound.push('|');
bound.push_str(plaintext);
Ok(SealedCredential {
@@ -231,7 +237,7 @@ mod tests {
.decode_to_vec(sealed.ct.as_bytes())
.map_err(|err| SealedCredentialError::Malformed(err.to_string()))?;
let bound = String::from_utf8(raw).map_err(|err| SealedCredentialError::Malformed(err.to_string()))?;
let expected = serde_json::to_string(&scope.encryption_context()).expect("context serializes");
let expected = encode_context(&scope.encryption_context());
bound
.strip_prefix(&expected)
.and_then(|rest| rest.strip_prefix('|'))
+1
View File
@@ -21,6 +21,7 @@ pub mod tier_gen;
pub mod tier_handlers;
pub(crate) mod tier_mutation_intent;
pub mod tier_mutation_peer;
pub(crate) mod tier_probe_intent;
pub mod warm_backend;
pub mod warm_backend_aliyun;
pub mod warm_backend_azure;
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
@@ -25,19 +25,24 @@ use rustfs_filemeta::{RestoreStatusOps as _, parse_restore_obj_status};
use tokio::io::AsyncReadExt;
async fn prime_metadata_generation(set_disks: &SetDisks, bucket: &str, object: &str) -> GetObjectMetadataCacheKey {
set_disks
.get_object_fileinfo(bucket, object, &ObjectOptions::default(), true, false)
.await
.expect("object metadata should resolve");
let generation = set_disks
.get_object_metadata_cache_generation(bucket, object)
.expect("metadata generation should be active");
let key = GetObjectMetadataCacheKey::new(bucket, object, generation);
assert!(
set_disks.get_object_metadata_cache.get(&key).await.is_some(),
"metadata read should publish the generation under test"
);
key
tokio::time::timeout(Duration::from_secs(30), async {
loop {
set_disks
.get_object_fileinfo(bucket, object, &ObjectOptions::default(), true, false)
.await
.expect("object metadata should resolve");
let generation = set_disks
.get_object_metadata_cache_generation(bucket, object)
.expect("metadata generation should be active");
let key = GetObjectMetadataCacheKey::new(bucket, object, generation);
if set_disks.get_object_metadata_cache.get(&key).await.is_some() {
return key;
}
tokio::task::yield_now().await;
}
})
.await
.expect("metadata read should publish the generation under test")
}
async fn assert_generation_reclaimed(set_disks: &SetDisks, key: &GetObjectMetadataCacheKey) {
@@ -60,8 +65,17 @@ async fn transition_and_restore_reclaim_prior_metadata_generations() {
.await
.expect("bucket should be created");
let mut reader = PutObjReader::from_vec(payload.clone());
// Cache priming must not race a quorum-acknowledged PUT's remaining rename tail.
let original = set_disks
.put_object(bucket, object, &mut reader, &ObjectOptions::default())
.put_object(
bucket,
object,
&mut reader,
&ObjectOptions {
no_lock: true,
..Default::default()
},
)
.await
.expect("source object should be written");
let source_generation = prime_metadata_generation(&set_disks, bucket, object).await;
@@ -164,8 +178,17 @@ async fn prepared_snapshot_transition_duplicate_and_late_get_use_committed_remot
.await
.expect("bucket should be created");
let mut reader = PutObjReader::from_vec(payload.clone());
// Cache priming must not race a quorum-acknowledged PUT's remaining rename tail.
let original = set_disks
.put_object(bucket, object, &mut reader, &ObjectOptions::default())
.put_object(
bucket,
object,
&mut reader,
&ObjectOptions {
no_lock: true,
..Default::default()
},
)
.await
.expect("source object should be written");
+146
View File
@@ -864,6 +864,11 @@ mod tests {
save_tier_mutation_intent_record, save_tier_mutation_intent_record_if_current,
},
tier_mutation_peer::{TierMutationPeerError, TierMutationPeerState, handle_tier_mutation_peer_request},
tier_probe_intent::{
TierProbeIntent, TierProbeIntentState, TierProbeOperationIdentity, TierProbeOwnerFence, TierProbeRemoteVersion,
delete_tier_probe_intent_record_if_current, load_tier_probe_intent_record,
save_tier_probe_intent_record_if_absent, save_tier_probe_intent_record_if_current,
},
warm_backend::{TransitionCandidateProbe, WarmBackend},
},
set_disk::SetDiskTransitionUploadedCommitBarrier as TransitionUploadedCommitBarrier,
@@ -17196,6 +17201,147 @@ mod tests {
assert!(matches!(err, Error::ConfigNotFound));
}
#[cfg(feature = "test-util")]
#[tokio::test]
#[serial_test::serial(storage_class_env)]
async fn tier_probe_intent_store_enforces_create_cas_and_terminal_delete_preconditions() {
let temp_dir = tempfile::tempdir().expect("create temp store dir");
let (_ctx, store, _shutdown) =
without_storage_class_env(build_isolated_test_store(temp_dir.path(), "tier-probe-intent-cas", &[4])).await;
let probe_id = uuid::Uuid::new_v4();
let creator_epoch = uuid::Uuid::new_v4();
let initial = TierProbeIntent {
probe_id,
revision: 1,
state: TierProbeIntentState::UploadOutcomeUnknown,
operation: TierProbeOperationIdentity::Verify {
config_etag: "config-etag".to_string(),
backend_identity: [1; 32],
},
tier_name: "COLD-A".to_string(),
destination_id: [1; 32],
probe_object: format!("rustfs-tier-probe-{probe_id}"),
creator_id: "node-a".to_string(),
creator_epoch,
created_at_unix_nanos: 1_780_000_000_000_000_000,
owner: TierProbeOwnerFence {
owner_id: "node-a".to_string(),
owner_epoch: creator_epoch,
not_after_unix_nanos: 1_780_000_900_000_000_000,
},
remote_version: TierProbeRemoteVersion::default(),
};
save_tier_probe_intent_record_if_absent(store.clone(), &initial)
.await
.expect("initial probe intent should persist with create-only semantics");
let duplicate = save_tier_probe_intent_record_if_absent(store.clone(), &initial)
.await
.expect_err("duplicate create must fail closed");
assert!(matches!(duplicate, Error::PreconditionFailed));
let observed_initial = load_tier_probe_intent_record(store.clone(), probe_id)
.await
.expect("initial probe intent should load with an ETag");
assert_eq!(observed_initial.intent(), &initial);
let nonterminal_delete = delete_tier_probe_intent_record_if_current(store.clone(), &observed_initial)
.await
.expect_err("nonterminal evidence must not be deleted");
assert!(nonterminal_delete.to_string().contains("must be terminal"));
let mut fabricated_current_intent = initial.clone();
fabricated_current_intent.tier_name = "COLD-B".to_string();
let mut fabricated_successor = fabricated_current_intent.clone();
fabricated_successor
.advance(
TierProbeIntentState::Uploaded,
TierProbeRemoteVersion::versioned(uuid::Uuid::new_v4().to_string()),
)
.expect("fabricated successor should be internally valid");
let fabricated_current = observed_initial.with_intent_for_test(fabricated_current_intent.clone());
let crossed_cas = save_tier_probe_intent_record_if_current(store.clone(), &fabricated_current, &fabricated_successor)
.await
.expect_err("a live ETag must not authorize a different caller record");
assert!(matches!(crossed_cas, Error::PreconditionFailed));
assert_eq!(
load_tier_probe_intent_record(store.clone(), probe_id)
.await
.expect("crossed CAS must retain the authoritative record")
.intent(),
&initial
);
let mut fabricated_terminal_intent = fabricated_current_intent;
fabricated_terminal_intent
.advance(TierProbeIntentState::AbortedNoRemote, TierProbeRemoteVersion::default())
.expect("fabricated terminal should be internally valid");
let fabricated_terminal = observed_initial.with_intent_for_test(fabricated_terminal_intent);
let crossed_delete = delete_tier_probe_intent_record_if_current(store.clone(), &fabricated_terminal)
.await
.expect_err("a live ETag must not delete for a different caller record");
assert!(matches!(crossed_delete, Error::PreconditionFailed));
assert_eq!(
load_tier_probe_intent_record(store.clone(), probe_id)
.await
.expect("crossed delete must retain the authoritative record")
.intent(),
&initial
);
let remote_version = TierProbeRemoteVersion::versioned(uuid::Uuid::new_v4().to_string());
let mut uploaded = observed_initial.intent().clone();
uploaded
.advance(TierProbeIntentState::Uploaded, remote_version.clone())
.expect("known PUT result should advance");
save_tier_probe_intent_record_if_current(store.clone(), &observed_initial, &uploaded)
.await
.expect("the matching initial ETag should admit one successor");
let stale_cas = save_tier_probe_intent_record_if_current(store.clone(), &observed_initial, &uploaded)
.await
.expect_err("a consumed ETag must not overwrite the current generation");
assert!(matches!(stale_cas, Error::PreconditionFailed));
let observed_uploaded = load_tier_probe_intent_record(store.clone(), probe_id)
.await
.expect("uploaded generation should load");
assert_eq!(observed_uploaded.intent(), &uploaded);
let mut cleanup = observed_uploaded.intent().clone();
cleanup
.advance(TierProbeIntentState::CleanupPending, remote_version.clone())
.expect("known candidate should become cleanup-pending");
save_tier_probe_intent_record_if_current(store.clone(), &observed_uploaded, &cleanup)
.await
.expect("cleanup generation should persist by exact ETag");
let observed_cleanup = load_tier_probe_intent_record(store.clone(), probe_id)
.await
.expect("cleanup generation should load");
let mut completed = observed_cleanup.intent().clone();
completed
.advance(TierProbeIntentState::Completed, remote_version)
.expect("exact cleanup should become terminal");
save_tier_probe_intent_record_if_current(store.clone(), &observed_cleanup, &completed)
.await
.expect("terminal generation should persist by exact ETag");
let stale_terminal = observed_cleanup.with_intent_for_test(completed.clone());
let stale_delete = delete_tier_probe_intent_record_if_current(store.clone(), &stale_terminal)
.await
.expect_err("a stale ETag must not delete terminal evidence");
assert!(matches!(stale_delete, Error::PreconditionFailed));
let observed_completed = load_tier_probe_intent_record(store.clone(), probe_id)
.await
.expect("terminal generation should remain after stale delete");
assert_eq!(observed_completed.intent(), &completed);
delete_tier_probe_intent_record_if_current(store.clone(), &observed_completed)
.await
.expect("the exact terminal ETag should delete the record");
assert!(matches!(load_tier_probe_intent_record(store, probe_id).await, Err(Error::ConfigNotFound)));
}
#[cfg(feature = "test-util")]
#[tokio::test]
#[serial_test::serial(storage_class_env)]
+47 -15
View File
@@ -230,7 +230,7 @@ fn scanner_abandoned_child_list_options() -> ListPathRawOptions {
}
pub fn data_usage_update_dir_cycles() -> u32 {
rustfs_utils::get_env_u32(ENV_DATA_USAGE_UPDATE_DIR_CYCLES, DATA_USAGE_UPDATE_DIR_CYCLES)
rustfs_utils::get_env_u32(ENV_DATA_USAGE_UPDATE_DIR_CYCLES, DATA_USAGE_UPDATE_DIR_CYCLES).max(1)
}
pub fn heal_object_select_prob() -> u32 {
@@ -806,6 +806,7 @@ impl FolderScanner {
fn prune_failed_objects_cache(&mut self) {
let ttl = self.failed_object_ttl_secs;
if ttl == 0 {
self.new_cache.info.failed_objects.clear();
return;
}
@@ -963,6 +964,27 @@ impl FolderScanner {
}
}
async fn preserve_failed_child(
&mut self,
parent: &Option<DataUsageHash>,
child_hash: &DataUsageHash,
parent_entry: &mut DataUsageEntry,
child_entry: &DataUsageEntry,
) {
// A failed walk proves neither deletion nor a complete replacement.
// Keep the previous subtree and mark this snapshot incomplete even
// when the failed-object retry cache is disabled or at capacity.
parent_entry.failed_objects = parent_entry.failed_objects.saturating_add(1);
if self.old_cache.cache.contains_key(&child_hash.key()) {
self.new_cache.delete_recursive(child_hash);
self.new_cache.copy_with_children(&self.old_cache, child_hash, parent);
parent_entry.add_child(child_hash);
} else {
self.preserve_partial_child_progress(parent, child_hash, parent_entry, child_entry)
.await;
}
}
fn alert_excessive_folders(&self, folder: &str, total_folders: usize) {
let threshold = scanner_excess_folders_threshold();
if u64::try_from(total_folders).unwrap_or(u64::MAX) <= threshold {
@@ -1177,8 +1199,6 @@ impl FolderScanner {
return Err(ScannerError::Other("Operation cancelled".to_string()));
}
self.prune_failed_objects_cache();
let mut abandoned_children: DataUsageHashMap = HashSet::new();
if !into.compacted {
abandoned_children = self.old_cache.find_children_copy(this_hash.clone());
@@ -1221,7 +1241,9 @@ impl FolderScanner {
};
let active_object_lock = self.old_cache.info.object_lock.clone();
self.sleeper.sleep_folder().await;
ctx.run_until_cancelled(self.sleeper.sleep_folder())
.await
.ok_or_else(|| ScannerError::Other("Operation cancelled".to_string()))?;
let mut existing_folders: Vec<CachedFolder> = Vec::new();
let mut new_folders: Vec<CachedFolder> = Vec::new();
@@ -1448,7 +1470,7 @@ impl FolderScanner {
let heal_enabled = this_hash.mod_alt(
self.old_cache.info.next_cycle as u32 / folder.object_heal_prob_div,
self.heal_object_select / folder.object_heal_prob_div,
(self.heal_object_select / folder.object_heal_prob_div).max(1),
) && self.should_heal().await;
let mut item = ScannerItem {
@@ -1465,12 +1487,10 @@ impl FolderScanner {
file_type: entry_type,
};
// If this path is already known as failed, just skip it.
// We intentionally do NOT call `record_failed` or bump `failed_objects` here,
// because the failure was recorded when the original error occurred
// (e.g. in the get_size error branch below). This branch only accounts
// for subsequent skips of already-failed paths.
// Count unresolved objects in each snapshot without extending
// the retry TTL or emitting another failure event.
if self.should_skip_failed(&item.path) {
into.failed_objects = into.failed_objects.saturating_add(1);
continue;
}
@@ -1485,7 +1505,7 @@ impl FolderScanner {
if failure_action != GetSizeFailureAction::Skip {
// Track failed objects to prevent infinite retry loops
into.failed_objects += 1;
into.failed_objects = into.failed_objects.saturating_add(1);
self.record_failed(&item.path);
if should_log_failed_object(into.failed_objects) {
@@ -1564,12 +1584,15 @@ impl FolderScanner {
}
}
timer.sleep().await;
ctx.run_until_cancelled(timer.sleep())
.await
.ok_or_else(|| ScannerError::Other("Operation cancelled".to_string()))?;
continue;
}
};
found_object_metadata = true;
self.new_cache.info.failed_objects.remove(&item.path);
item.transform_meta_dir();
@@ -1581,7 +1604,9 @@ impl FolderScanner {
object_count += 1;
self.budget.record_object_scanned();
timer.sleep().await;
ctx.run_until_cancelled(timer.sleep())
.await
.ok_or_else(|| ScannerError::Other("Operation cancelled".to_string()))?;
if ctx.is_cancelled() {
return Err(ScannerError::Other("Operation cancelled".to_string()));
@@ -1622,9 +1647,9 @@ impl FolderScanner {
if self.is_erasure_mode && found_erasure_data_directory && !found_object_metadata {
found_object_metadata = true;
let metadata_path = path_join_buf(&[&dir_path, STORAGE_FORMAT_FILE]);
into.failed_objects = into.failed_objects.saturating_add(1);
if !self.should_skip_failed(&metadata_path) {
into.failed_objects = into.failed_objects.saturating_add(1);
self.record_failed(&metadata_path);
let failed_cache_entries = self.new_cache.info.failed_objects.len();
@@ -1835,6 +1860,7 @@ impl FolderScanner {
error = %e,
"Scanner child folder scan failed"
);
self.preserve_failed_child(&folder_item.parent, &h, into, &dst).await;
continue;
}
tokio::task::yield_now().await;
@@ -2230,6 +2256,7 @@ impl FolderScanner {
error = %e,
"Scanner heal child folder scan failed"
);
self.preserve_failed_child(&folder_item.parent, &h, into, &dst).await;
continue;
}
tokio::task::yield_now().await;
@@ -2396,6 +2423,9 @@ pub async fn scan_data_folder(
};
let now = FolderScanner::now_secs();
// Prune once per bucket walk, not once per directory. Per-path TTL checks
// still allow retries during long scans, and insertions enforce the cap.
scanner.prune_failed_objects_cache();
prune_size_reconciliation(&mut scanner.new_cache.info, now);
prune_size_reconciliation(&mut scanner.update_cache.info, now);
@@ -2422,7 +2452,9 @@ pub async fn scan_data_folder(
new_cache.force_compact(DATA_SCANNER_COMPACT_AT_CHILDREN);
new_cache.info.last_update = Some(SystemTime::now());
new_cache.info.next_cycle = cache.info.next_cycle;
let unresolved_objects = root.failed_objects > 0
let unresolved_objects = new_cache
.size_recursive(&cache.info.name)
.is_none_or(|root| root.failed_objects > 0)
|| !new_cache.info.failed_objects.is_empty()
|| !new_cache.info.size_reconciliation.is_empty();
new_cache.info.snapshot_complete = !unresolved_objects;
@@ -1366,6 +1366,104 @@ mod tests {
assert_eq!(item.object_path(), "object");
}
#[tokio::test]
#[serial_test::serial]
async fn scanner_blocked_expiry_preserves_usage_replication_and_integrity_work() {
use s3s::dto::{LifecycleExpiration, LifecycleRule};
let lifecycle = Arc::new(BucketLifecycleConfiguration {
rules: vec![LifecycleRule {
status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
expiration: Some(LifecycleExpiration {
days: Some(1),
..Default::default()
}),
abort_incomplete_multipart_upload: None,
del_marker_expiration: None,
filter: None,
id: None,
noncurrent_version_expiration: None,
noncurrent_version_transitions: None,
prefix: None,
transitions: None,
}],
..Default::default()
});
let attempts = |report: &rustfs_scanner_metrics::metrics::ScannerMetricsReport, source: ScannerWorkSource| {
report
.source_work
.iter()
.filter(|work| work.source == source.as_str())
.map(|work| work.queued + work.skipped + work.missed)
.sum::<u64>()
};
for with_lifecycle in [false, true] {
for scan_mode in [HealScanMode::Normal, HealScanMode::Deep] {
for guard in ["pending", "failed", "legal_hold"] {
let mut metadata = HashMap::new();
let replication_status = match guard {
"pending" => ReplicationStatusType::Pending,
"failed" => ReplicationStatusType::Failed,
_ => {
metadata.insert("x-amz-object-lock-legal-hold".to_string(), "ON".to_string());
ReplicationStatusType::Completed
}
};
let object = ObjectInfo {
bucket: "bucket".to_string(),
name: "object".to_string(),
version_id: Some(uuid::Uuid::new_v4()),
num_versions: 1,
is_latest: true,
mod_time: Some(OffsetDateTime::now_utc() - time::Duration::days(90)),
size: 4096,
actual_size: 4096,
replication_status,
user_defined: Arc::new(metadata),
..Default::default()
};
let events = Evaluator::new(lifecycle.clone())
.eval(&[crate::ecstore_object_opts_from_object_info(&object)])
.await
.expect("evaluate expiry guard");
assert_eq!(events[0].action, IlmAction::NoneAction, "expiry must be blocked by {guard}");
let mut item = scanner_item_with_prefix("");
item.object_name = "object".to_string();
item.lifecycle = with_lifecycle.then(|| lifecycle.clone());
item.replication = Some(Arc::new(ReplicationConfig::new(None, None)));
item.heal_enabled = true;
item.heal_bitrot = scan_mode == HealScanMode::Deep;
let before = global_metrics().report().await;
let mut summary = SizeSummary::default();
item.apply_actions(vec![object], None, VersioningConfiguration::default(), &[], &mut summary)
.await;
let after = global_metrics().report().await;
assert_eq!(summary.total_size, 4096, "blocked expiry must retain bytes for {guard}");
assert_eq!(summary.versions, 1);
assert_eq!(summary.delete_markers, 0);
assert!(summary.size_reconciliation.is_empty());
assert_eq!(
attempts(&after, scanner_heal_source(scan_mode)) - attempts(&before, scanner_heal_source(scan_mode)),
1,
"integrity work must continue with lifecycle={with_lifecycle}, guard={guard}"
);
assert_eq!(
attempts(&after, ScannerWorkSource::BucketReplication)
- attempts(&before, ScannerWorkSource::BucketReplication),
1,
"replication inspection must continue with lifecycle={with_lifecycle}, guard={guard}"
);
assert_eq!(
attempts(&after, ScannerWorkSource::Lifecycle) - attempts(&before, ScannerWorkSource::Lifecycle),
0,
"blocked expiry must not enqueue destructive lifecycle work"
);
}
}
}
}
#[test]
fn unknown_tier_never_triggers_transition() {
let object = ObjectInfo {
+319 -2
View File
@@ -1883,6 +1883,323 @@ async fn test_scan_folder_skips_unreadable_child_directory() {
assert!(result.is_ok(), "expected unreadable child directory to be skipped");
}
#[tokio::test]
#[serial]
async fn scanner_failed_child_retains_usage_and_scans_healthy_sibling() {
for with_prior in [false, true] {
let (mut scanner, temp_dir) = build_test_scanner().await;
let _guard = TestGuard::new(0, 0, &mut scanner, temp_dir.clone());
let bad_dir = temp_dir.join("bucket/bad");
tokio::fs::create_dir_all(&bad_dir).await.expect("create failing directory");
write_test_object_metadata_bytes(
&temp_dir,
"bucket",
"good",
&metadata_for_object_version("bucket", "good", Some(Uuid::new_v4())),
)
.await;
scanner.old_cache.info.name = "bucket".to_string();
scanner.new_cache.info.name = "bucket".to_string();
scanner.update_cache.info.name = "bucket".to_string();
let root_hash = hash_path("bucket");
let bad_hash = hash_path("bucket/bad");
let mut prior = DataUsageEntry {
size: 4096,
objects: 2,
versions: 3,
delete_markers: 1,
..Default::default()
};
prior.replication_stats = Some(rustfs_data_usage::ReplicationAllStats {
replica_size: 4096,
replica_count: 2,
..Default::default()
});
prior.add_tier_sizes(&HashMap::from([(
"WARM".to_string(),
TierStats {
total_size: 4096,
num_versions: 3,
num_objects: 2,
},
)]));
scanner
.old_cache
.replace_hashed(&root_hash, &None, &DataUsageEntry::default());
if with_prior {
scanner.old_cache.replace_hashed(&bad_hash, &Some(root_hash.clone()), &prior);
} else {
prior = DataUsageEntry::default();
}
scanner.update_current_path = Arc::new(move |path| {
if path == "bucket/bad" {
// Replace the directory after enumeration but before descent. This
// injects a real read_dir error even when tests run as root.
std::fs::remove_dir(&bad_dir).expect("remove enumerated directory");
std::fs::write(&bad_dir, b"not a directory").expect("replace enumerated directory");
}
Box::pin(async {})
});
let mut root = DataUsageEntry::default();
scanner
.scan_folder(
CancellationToken::new(),
CachedFolder {
name: "bucket".to_string(),
parent: None,
object_heal_prob_div: 1,
},
&mut root,
)
.await
.expect("one failed directory must not stop healthy siblings");
let total = scanner.new_cache.size_recursive(&root_hash.key()).expect("root usage");
assert_eq!(total.size, prior.size + 1, "unreadable child must retain its previous bytes");
assert_eq!(total.objects, prior.objects + 1, "healthy sibling must still be counted");
assert_eq!(total.versions, prior.versions + 1);
assert_eq!(total.delete_markers, prior.delete_markers);
assert_eq!(total.failed_objects, 1, "walk error must keep the snapshot incomplete");
assert_eq!(
serde_json::to_value(&total.replication_stats).expect("serialize replication usage"),
serde_json::to_value(&prior.replication_stats).expect("serialize prior replication usage")
);
assert_eq!(
serde_json::to_value(&total.all_tier_stats).expect("serialize tier usage"),
serde_json::to_value(&prior.all_tier_stats).expect("serialize prior tier usage")
);
}
}
#[tokio::test]
#[serial]
async fn scanner_nested_metadata_failure_without_retry_cache_is_partial_then_recovers() {
let (scanner, temp_dir) = build_test_scanner().await;
let _guard = TestGuard {
temp_dir: Some(temp_dir.clone()),
};
write_test_object_metadata_bytes(&temp_dir, "bucket", "prefix/bad", b"").await;
write_test_object_metadata_bytes(
&temp_dir,
"bucket",
"prefix/good",
&metadata_for_object_version("bucket", "prefix/good", Some(Uuid::new_v4())),
)
.await;
temp_env::async_with_vars([(ENV_FAILED_OBJECT_TTL_SECS, Some("0"))], async {
for inherited_failure in [false, true] {
write_test_object_metadata_bytes(&temp_dir, "bucket", "prefix/bad", b"").await;
let mut cache = DataUsageCache {
info: DataUsageCacheInfo {
name: "bucket".to_string(),
next_cycle: u64::from(
(0..DATA_USAGE_UPDATE_DIR_CYCLES)
.find(|cycle| !hash_path("bucket/prefix").mod_(*cycle, DATA_USAGE_UPDATE_DIR_CYCLES))
.expect("cycle outside the prefix compaction sample"),
),
..Default::default()
},
..Default::default()
};
if inherited_failure {
cache
.info
.failed_objects
.insert("removed-object/xl.meta".to_string(), FolderScanner::now_secs());
}
let budget = ScannerCycleBudget::new(&CancellationToken::new(), Default::default());
let result = scan_data_folder(
budget.token(),
budget,
vec![scanner.local_disk.clone()],
scanner.local_disk.clone(),
cache,
None,
HealScanMode::Normal,
DynamicSleeper::new(rustfs_config::ScannerSpeed::Fastest),
)
.await;
let mut partial = match result {
Err(ScannerError::PartialCache(cache)) => *cache,
other => panic!("nested failure must never publish a complete snapshot: {other:?}"),
};
assert!(!partial.info.snapshot_complete);
assert!(partial.info.failed_objects.is_empty(), "TTL zero disables only the retry cache");
let total = partial.size_recursive("bucket").expect("partial root");
assert_eq!(total.objects, 1);
assert_eq!(total.failed_objects, 1);
// Reusing the partial compacted subtree must remain partial even
// without a retry ledger. Recovery happens on its next selected cycle.
let budget = ScannerCycleBudget::new(&CancellationToken::new(), Default::default());
let reused = scan_data_folder(
budget.token(),
budget,
vec![scanner.local_disk.clone()],
scanner.local_disk.clone(),
partial.clone(),
None,
HealScanMode::Normal,
DynamicSleeper::new(rustfs_config::ScannerSpeed::Fastest),
)
.await;
assert!(matches!(reused, Err(ScannerError::PartialCache(_))));
partial.info.next_cycle = u64::from(
(0..DATA_USAGE_UPDATE_DIR_CYCLES)
.find(|cycle| hash_path("bucket/prefix").mod_(*cycle, DATA_USAGE_UPDATE_DIR_CYCLES))
.expect("next selected directory cycle"),
);
write_test_object_metadata_bytes(
&temp_dir,
"bucket",
"prefix/bad",
&metadata_for_object_version("bucket", "prefix/bad", Some(Uuid::new_v4())),
)
.await;
let budget = ScannerCycleBudget::new(&CancellationToken::new(), Default::default());
let recovered = scan_data_folder(
budget.token(),
budget,
vec![scanner.local_disk.clone()],
scanner.local_disk.clone(),
partial,
None,
HealScanMode::Normal,
DynamicSleeper::new(rustfs_config::ScannerSpeed::Fastest),
)
.await
.expect("repaired subtree must converge on its next selected cycle");
assert!(recovered.info.snapshot_complete);
let total = recovered.size_recursive("bucket").expect("recovered root");
assert_eq!(total.objects, 2);
assert_eq!(total.size, 2);
assert_eq!(total.versions, 2);
assert_eq!(total.failed_objects, 0);
}
})
.await;
}
#[tokio::test]
#[serial]
async fn scanner_compacted_directory_keeps_aggressive_heal_and_bitrot_sampling() {
for scan_mode in [HealScanMode::Normal, HealScanMode::Deep] {
for select_prob in [0, 1, 8, 16] {
let (mut scanner, temp_dir) = build_test_scanner().await;
let _guard = TestGuard::new(0, 0, &mut scanner, temp_dir.clone());
write_test_object_metadata_bytes(
&temp_dir,
"bucket",
"object",
&metadata_for_object_version("bucket", "object", Some(Uuid::new_v4())),
)
.await;
scanner.old_cache.info.name = "bucket".to_string();
scanner.new_cache.info.name = "bucket".to_string();
scanner.update_cache.info.name = "bucket".to_string();
scanner.is_erasure_mode = true;
scanner.heal_object_select = select_prob;
scanner.scan_mode = scan_mode;
let root_hash = hash_path("bucket");
let object_hash = hash_path("bucket/object");
scanner.old_cache.info.next_cycle = u64::from(
(0..DATA_USAGE_UPDATE_DIR_CYCLES)
.find(|cycle| object_hash.mod_(*cycle, DATA_USAGE_UPDATE_DIR_CYCLES))
.expect("selected directory cycle"),
);
scanner
.old_cache
.replace_hashed(&root_hash, &None, &DataUsageEntry::default());
scanner.old_cache.replace_hashed(
&object_hash,
&Some(root_hash),
&DataUsageEntry {
compacted: true,
objects: 1,
versions: 1,
..Default::default()
},
);
let attempts = |report: rustfs_scanner_metrics::metrics::ScannerMetricsReport| {
report
.source_work
.iter()
.filter(|work| work.source == scanner_heal_source(scan_mode).as_str())
.map(|work| work.queued + work.skipped + work.missed)
.sum::<u64>()
};
temp_env::async_with_vars([(ENV_SCANNER_DEEP_VERIFY_COOLDOWN_SECS, Some("0"))], async {
let before = attempts(global_metrics().report().await);
let mut root = DataUsageEntry::default();
scanner
.scan_folder(
CancellationToken::new(),
CachedFolder {
name: "bucket".to_string(),
parent: None,
object_heal_prob_div: 1,
},
&mut root,
)
.await
.expect("scan selected compacted object");
assert_eq!(
attempts(global_metrics().report().await) - before,
u64::from(select_prob != 0),
"selected compacted object must reach {scan_mode:?} admission with divisor {select_prob}"
);
let total = scanner.new_cache.size_recursive("bucket").expect("usage root");
assert_eq!(total.objects, 1);
assert_eq!(total.versions, 1);
})
.await;
}
}
}
#[tokio::test(start_paused = true)]
#[serial]
async fn scanner_cancellation_interrupts_folder_throttle() {
use futures::{FutureExt, poll};
let (mut scanner, temp_dir) = build_test_scanner().await;
let _guard = TestGuard::new(0, 0, &mut scanner, temp_dir);
scanner.sleeper = DynamicSleeper::new(rustfs_config::ScannerSpeed::Slowest);
let previous_idle = crate::sleeper::SCANNER_IDLE_MODE.swap(true, std::sync::atomic::Ordering::Relaxed);
let ctx = CancellationToken::new();
let mut root = DataUsageEntry::default();
let mut scan = scanner
.scan_folder(
ctx.clone(),
CachedFolder {
name: "bucket".to_string(),
parent: None,
object_heal_prob_div: 1,
},
&mut root,
)
.boxed();
assert!(poll!(scan.as_mut()).is_pending(), "scan should be waiting in its folder throttle");
ctx.cancel();
let outcome = scan.now_or_never();
crate::sleeper::SCANNER_IDLE_MODE.store(previous_idle, std::sync::atomic::Ordering::Relaxed);
assert!(
matches!(outcome, Some(Err(_))),
"cancellation must finish without advancing the sleep clock"
);
}
#[test]
#[serial]
fn scanner_zero_directory_cycle_keeps_rescanning_enabled() {
temp_env::with_var(ENV_DATA_USAGE_UPDATE_DIR_CYCLES, Some("0"), || {
for cycle in 0..32 {
assert!(
hash_path("bucket/object").mod_(cycle, data_usage_update_dir_cycles()),
"zero must not leave compacted usage stale forever"
);
}
});
}
#[tokio::test]
#[serial]
async fn test_scan_folder_exits_when_abandoned_child_listing_finishes() {
@@ -2153,7 +2470,7 @@ async fn test_scan_folder_corrupt_xl_meta_stops_erasure_data_dir_descent() {
.await
.expect("cached metadata failure must still stop erasure data directory descent");
assert_eq!(retry_into.failed_objects, 0, "cached failure should not be counted twice");
assert_eq!(retry_into.failed_objects, 1, "cached failure must remain visible in each snapshot");
assert!(!retry_budget.budget_elapsed());
assert_eq!(retry_budget.reason(), None);
@@ -2278,7 +2595,7 @@ async fn test_scan_folder_missing_xl_meta_stops_erasure_data_dir_descent() {
.await
.expect("cached missing metadata must still stop erasure data directory descent");
assert_eq!(retry_into.failed_objects, 0, "cached failure should not be counted twice");
assert_eq!(retry_into.failed_objects, 1, "cached failure must remain visible in each snapshot");
assert!(!retry_budget.budget_elapsed());
assert_eq!(retry_budget.reason(), None);
}
+1 -1
View File
@@ -7,7 +7,7 @@ For crate ownership, read [crate-boundaries.md](crate-boundaries.md): ECStore ow
## Model
Heal and every foreground or background write path serialize on the same object-level namespace write lock (a quorum lock RPC in distributed mode, the in-process lock manager on a single node; granularity is the object, the version component is always `None`), and heal holds its guard across the whole rename commit. MinIO's `x-minio-healing` marker is an out-of-lock defence against version-cleanup logic inside `RenameData` interleaving with a heal commit; RustFS's commit model has no such interleaving, so no persistent marker exists (`x-minio-healing` does not occur in `crates/` or `rustfs/`) and none is needed. Three layers replace it:
Heal and every foreground or background write path serialize on the same object-level namespace write lock (a quorum lock RPC in distributed mode, the in-process lock manager on a single node; granularity is the object, the version component is always `None`), and heal holds its guard across the whole rename commit. This describes the intended lock scope while the guard remains valid; it does not prove rejection of an already-dispatched disk syscall after distributed lease loss. The authority, delayed-mutation, and recovery boundary is specified in [unified-object-generation.md](unified-object-generation.md). MinIO's `x-minio-healing` marker is an out-of-lock defence against version-cleanup logic inside `RenameData` interleaving with a heal commit; RustFS's commit model has no such interleaving, so no persistent marker exists (`x-minio-healing` does not occur in `crates/` or `rustfs/`) and none is needed. Three layers replace it:
| Layer | Mechanism | Owner |
| --- | --- | --- |
@@ -1,7 +1,7 @@
# ILM And Tiering Persistence Contracts
**Use this when:** changing an ILM transition, tier configuration mutation, manual transition job, tier-delete recovery path, pool decommission, or any code that can create, transfer, or destroy ownership of a remote-tier object.
**Source of truth:** `TransitionTransaction` and `process_transition_transaction_record` in `crates/ecstore/src/bucket/lifecycle/transition_transaction.rs`; `TierMutationIntent` and its conditional store helpers in `crates/ecstore/src/services/tier/tier_mutation_intent.rs`; `TierConfigMgr::update_candidate_with_config_lock` and mutation recovery in `crates/ecstore/src/services/tier/tier.rs`; `handle_tier_mutation_peer_request` in `crates/ecstore/src/services/tier/tier_mutation_peer.rs`; the record encoders and CAS helpers in `crates/ecstore/src/bucket/lifecycle/manual_transition_job.rs`; manual-job execution/recovery and `cleanup_free_version_exact` in `crates/ecstore/src/bucket/lifecycle/bucket_lifecycle_ops.rs`; `process_tier_delete_journal_entry` and manifest recovery in `crates/ecstore/src/bucket/lifecycle/tier_delete_journal.rs`; the free-version scan/re-enqueue path `recover_tier_free_versions` in `crates/ecstore/src/bucket/lifecycle/tier_free_version_recovery.rs`; `DURABLE_ILM_NAMESPACES` and `validate_durable_ilm_record` in `crates/ecstore/src/bucket/lifecycle/durable_namespace.rs`; and `record_durable_ilm_decommission_progress`, receipt verification, and receipt cleanup in `crates/ecstore/src/core/pools.rs`.
**Source of truth:** `TransitionTransaction` and `process_transition_transaction_record` in `crates/ecstore/src/bucket/lifecycle/transition_transaction.rs`; `TierMutationIntent` and its conditional store helpers in `crates/ecstore/src/services/tier/tier_mutation_intent.rs`; the dormant validation-probe record and conditional primitives in `crates/ecstore/src/services/tier/tier_probe_intent.rs`; `TierConfigMgr::update_candidate_with_config_lock` and mutation recovery in `crates/ecstore/src/services/tier/tier.rs`; `handle_tier_mutation_peer_request` in `crates/ecstore/src/services/tier/tier_mutation_peer.rs`; the record encoders and CAS helpers in `crates/ecstore/src/bucket/lifecycle/manual_transition_job.rs`; manual-job execution/recovery and `cleanup_free_version_exact` in `crates/ecstore/src/bucket/lifecycle/bucket_lifecycle_ops.rs`; `process_tier_delete_journal_entry` and manifest recovery in `crates/ecstore/src/bucket/lifecycle/tier_delete_journal.rs`; the free-version scan/re-enqueue path `recover_tier_free_versions` in `crates/ecstore/src/bucket/lifecycle/tier_free_version_recovery.rs`; `DURABLE_ILM_NAMESPACES` and `validate_durable_ilm_record` in `crates/ecstore/src/bucket/lifecycle/durable_namespace.rs`; and `record_durable_ilm_decommission_progress`, receipt verification, and receipt cleanup in `crates/ecstore/src/core/pools.rs`.
This document separates three kinds of statement:
@@ -41,9 +41,10 @@ All keys below are objects in the internal metadata bucket. The table gives the
| Protocol | Current schema/version | Canonical key | Creator and cleanup owner | Authoritative identity and mutable fields | Current durability point |
|---|---|---|---|---|---|
| Transition transaction | `rustfs-transition-transaction-v1` | `ilm/transition-transactions/records/<aa>/<bb>/<transaction-id>.json` | The transition attempt creates it; transition commit/recovery cleans it | Immutable/fence identity: deployment, transaction, fixed `owner_epoch`, write, source identity, tier/backend fingerprint, canonical remote object, deadline. Mutable: state, remote version, revision. `TransitionCleanupProof` is only a transient admission input to `mark_cleanup_pending`; it is not persisted in the record | Maximum-parity config write. Current create/update/delete calls do not use ETag preconditions |
| Transition transaction | `rustfs-transition-transaction-v1`; successor v2 is approved below but not implemented | `ilm/transition-transactions/records/<aa>/<bb>/<transaction-id>.json` | The transition attempt creates it; transition commit/recovery cleans it | Immutable/fence identity: deployment, transaction, fixed v1 `owner_epoch`, write, source identity, tier/backend fingerprint, canonical remote object, deadline. Mutable: state, remote version, revision. `TransitionCleanupProof` is only a transient admission input to `mark_cleanup_pending`; it is not persisted in the record | Create-only maximum-parity write; exact record and ETag read before successor `If-Match`; terminal receipt followed by exact ETag conditional delete |
| Tier mutation peer intent | `rustfs-tier-mutation-intent-v1` | `tier/mutation-intents/records/<aa>/<bb>/<mutation-id>.json` | The receiving peer creates and converges it; the mutation recovery path cleans it | Immutable: mutation ID/kind, old config ETag, candidate digest, sorted affected target identities, expiry. Mutable: revision, state, committed config ETag | Create with `If-None-Match: *`; transition/delete with ETag `If-Match`; maximum parity |
| Tier mutation coordinator intent | `rustfs-tier-mutation-intent-v1` | `tier/mutation-intents/coordinators/<aa>/<bb>/<mutation-id>.json` | The initiating node creates it; coordinator recovery cleans it after peer convergence | Same mutation identity and mutable fields as the peer record | Same conditional-write contract as the peer intent |
| Tier validation probe intent | Dormant `rustfs-tier-probe-intent-v1`; no writer or recovery is enabled | `ilm/tier-probe-intents/records/<aa>/<bb>/<probe-id>.json` | No current runtime owner because no path creates the record; v1 permits only the immutable creator as owner | Immutable probe, operation-generation, destination, random remote object, creator identity, and v1 owner fence. Mutable: revision, state, and monotonic remote-version proof | Conditional create/CAS/delete primitives exist but are not called by Add/Edit/Verify or recovery |
| Manual job | `rustfs-manual-transition-job-v1` | `ilm/manual-transition/jobs/<aa>/<bb>/<job-id>.json` | The admin run creates it; the active owner or recovery lease advances it. There is no current record GC owner | Immutable: job ID, bucket-level scope, options, creation time. Mutable: owner/lease, state, cancel bit, cursor, progress/report, queue snapshot, timestamps/error | Initial UUID-key write uses maximum parity without create-only precondition; later updates use ETag CAS |
| Manual scope admission | `rustfs-manual-transition-job-v1` | `ilm/manual-transition/scopes/<aa>/<bb>/<scope-digest>.json` | Job admission creates/renews it; the job owner removes it after terminalization | Immutable bucket/run-vs-dry-run scope; mutable job/lease ownership and expiry | Create-only, renew/delete by ETag CAS |
| Manual task | `rustfs-manual-transition-task-v1` | `ilm/manual-transition/tasks/<job shards>/<job-id>/<task-key>.json` | The scanner persists it before queue admission; no current GC owner | Immutable job plus exact bucket/object/version/tier work identity | Append-only create with `If-None-Match: *` and maximum parity |
@@ -54,8 +55,9 @@ All keys below are objects in the internal metadata bucket. The table gives the
| Tier-delete chunk parent | Version 1 with `record_type = "chunked_parent"` | `ilm/tier-delete-dispatch-manifests/<scope-digest>.json` | The over-limit prefix-delete coordinator creates and advances it; parent recovery advances completed children and removes the terminal parent | Immutable operation, bucket/incarnation/prefix/topology; mutable monotonic revision, next child sequence, completed journal count, one optional exact child binding, and `Active`/`Completed` state | Create-only and fenced ETag CAS. The parent binds a `Preparing` child before it can become `DispatchAuthorized`; final `Completed` follows an error-free, non-truncated empty-candidate rescan and local prefix deletion |
| Decommission durable-namespace receipt | `v2` | `decommission/ilm-receipts/<run-token>/<source-path>/<id-kind>/<id>.json` | The decommission coordinator writes target/source proof and is the only cleanup owner for that run | Source path, namespace and record identity, monotonic checkpoint, optional terminal checkpoint, optional v6 topology generation | Create-only then ETag CAS merge; checksum envelope; maximum parity |
| Decommission expected-receipt manifest | `v1` | `decommission/ilm-manifests/<run-token>.json` | The source-pool decommission coordinator creates and cleans it | Run token plus exact sorted receipt-path count/digest | Create-only, exact readback, and verification before pool removal |
| Recovery control, export, and disposition | `rustfs-ilm-recovery-control-v1`, `rustfs-ilm-recovery-export-v1`, and `rustfs-ilm-recovery-disposition-v1` are approved below but not implemented | `ilm/recovery-controls/...`, `ilm/recovery-exports/...`, and `ilm/recovery-dispositions/...` under protocol/shard/operation identities | Recovery owns one control for an exact source generation; the authenticated operator creates immutable export/disposition evidence; their collectors never own remote DELETE | Source protocol/path, all-pool copy-set manifest, ETags/content digests, owner lease, retry state, redacted error code, action, actor/reason, and terminal proof | Create-only, ETag CAS, all-pool strong readback, terminal receipt when covered by decommission, and exact conditional cleanup |
`durable_namespace.rs` registers exactly the two tier-journal namespaces, the dispatch-record namespace shared by single manifests, chunk children, and chunk parents, the transaction namespace, and four manual-job namespaces. A path beginning with `ilm/` that is not in that registry is an error during decommission rather than an ignorable object.
`durable_namespace.rs` registers exactly the two tier-journal namespaces, the dispatch-record namespace shared by single manifests, chunk children, and chunk parents, the transition-transaction namespace, the dormant tier-validation-probe namespace, and four manual-job namespaces. A path beginning with `ilm/` that is not in that registry is an error during decommission rather than an ignorable object.
## Durable fences and write primitives
@@ -71,7 +73,7 @@ The internal config layer supplies maximum-parity writes, create-only writes, ET
| Conditional delete | Removes only the verified terminal generation; if an active decommission covers it, its terminal receipt is written first |
| Strong readback | Resolves a lost response only when key, schema, full immutable identity, state, and expected successor all match |
Tier mutation intents and v6 journal/manifest records use the conditional primitives. Manual job updates, scope admission, and decommission receipts also use CAS after creation. The transition transaction currently carries a fixed `owner_epoch` fence identity and a mutable `revision`, but persists with unconditional writes and deletes; those fields therefore detect some in-memory misuse but are not yet a durable exclusion fence. Changing `owner_epoch` during takeover is not current behavior and remains an open design. The manual job's initial UUID-key write has the same create-only gap, although all later owner/lease updates are CAS-protected.
Tier mutation intents and v6 journal/manifest records use the conditional primitives. Manual job updates, scope admission, and decommission receipts also use CAS after creation. Transition transaction v1 now uses create-only installation, exact record plus ETag read before each successor CAS, and exact ETag terminal deletion. Its `owner_epoch` and `not_after_unix_nanos` remain immutable, however, so an expired recovery worker claims only the next state generation rather than a renewable durable owner lease. The manual job's initial UUID-key write still lacks create-only installation, although all later owner/lease updates are CAS-protected.
### Approved target
@@ -81,7 +83,7 @@ Tier mutation intents and v6 journal/manifest records use the conditional primit
- After a timeout, connection loss, or quorum-uncertain response, the caller must strongly reread. Only the exact intended successor is success; predecessor, absence, conflict, corruption, or unavailable readback retains the record and blocks destructive action.
- A process-local mutex, cancellation token, task registry, or cached generation may reduce duplicate work but cannot authorize publication, rollback, or remote deletion.
The exact transition-transaction lease/takeover fields and whether the existing `not_after` becomes the owner expiry are an **open design**. They must be settled with upgrade/downgrade behavior before the v1 schema changes.
The approved transition-transaction successor, lease/takeover fields, v1 migration, and upgrade/downgrade gates are specified in [Bounded recovery control and operator disposition](#bounded-recovery-control-and-operator-disposition). They require implementation and fleet gating before any v2 writer or destructive v1 takeover is enabled.
## Lock and operation order
@@ -89,18 +91,19 @@ Lock ordering is part of the recovery contract. Callers acquire only the locks n
| Path | Current acquisition order | Operations allowed while held | Operations forbidden while held |
|---|---|---|---|
| Tier edit/remove/clear | Tier-config namespace WRITE lock; dedicated owned `admin_updates` serialization mutex; short `TierConfigMgr` state locks only while accessing manager/runtime state | The dedicated `admin_updates` guard intentionally spans awaited backend validation/probes, peer Prepare/Commit/Abort RPC, reference scans, config CAS, and candidate publication in the current protocol | Ordinary manager `RwLock` and runtime-state `Mutex` guards must not cross awaited network I/O; that rule does not prohibit the dedicated `admin_updates` guard from spanning those awaits. Remote object DELETE is never part of mutation |
| Tier add/edit/remove/clear | A short tier-config namespace WRITE lock captures the persisted config ETag, then releases before backend validation. After validation, namespace WRITE then `admin_updates` protect the ETag check and durable coordinator Prepared write. Both guards are released for lease drain, peer Prepare, and reference proof, then reacquired in the same order for final identity checks and config CAS. Both are released again after the coordinator becomes durably Committed | Backend validation, peer fanout, and reference proof run without either exclusive guard. Immediately before config CAS the coordinator revalidates the ETag, candidate digest, exact Prepared intent identity, and intent expiry. The durable Committed intent is recovery authority while peer Commit and local publication finish without the exclusive guards | Ordinary manager `RwLock` and runtime-state `Mutex` guards must not cross awaited network I/O. Recovery must retain an unexpired Prepared coordinator whose old ETag is still current; the old ETag alone is not abandonment proof. Remote object DELETE is never part of mutation |
| v6 manifest prepare | Caller already holds the bucket-lifecycle WRITE fence; caller acquires a bucket-metadata transaction READ guard covering the Object Lock and bucket-incarnation snapshot and keeps it through local mutation; exact tier-generation leases; fleet/topology proof; for a single dispatch, synthetic manifest-operation WRITE; for a child, parent-operation WRITE then child-operation WRITE | Build and write one immutable bounded journal set and manifest, validate exact set/digest, then authorize local dispatch while both caller-held bucket guards and all leases remain current. A parent binding is durable before child authorization | Remote tier DELETE; per-object worker cleanup; releasing the metadata guard or a required lease before the authorized local mutation completes; child-to-parent nested lock acquisition |
| v6 manifest/parent recovery | Fleet/topology proof; bucket-lifecycle WRITE lock; then exactly one synthetic manifest- or parent-operation WRITE lock | Read/write manifest, parent, and journal metadata; verify exact set/digest/binding; converge or roll back child records; advance a parent only after child completion | Remote tier DELETE; per-object worker cleanup; rollback after authorization; taking a child lock while holding a parent lock in background recovery |
| v5 journal destructive recovery | Synthetic per-journal recovery lock; bucket-lifecycle READ lock; exact tier-generation lease; all physical object READ locks in stable pool/set order | Authoritative source/free-version scan; fenced state CAS; for an eligible terminal state, one bounded remote DELETE; conditional record cleanup | Any delete when a lock or lease is lost; publishing local metadata; selecting an arbitrary backend/version |
| v6 journal destructive recovery | Synthetic per-journal recovery lock; fleet/topology proof; bucket-lifecycle READ lock; exact tier-generation lease; all physical object READ locks in stable pool/set order | Immutable manifest/topology validation, authoritative source/free-version scan, fenced state CAS, and, for an eligible terminal state, one bounded remote DELETE followed by record cleanup | Any delete when a lock, lease, or fleet proof is lost; publishing local metadata; selecting an arbitrary backend/version |
| Free-version cleanup | Bucket-lifecycle READ lock; exact tier-generation lease; all physical object WRITE locks in stable pool/set order | Exact all-pool scan; bounded remote DELETE; local marker removal; post-delete rescan | Deleting before the free-version is the sole owner or after any fence changes |
| Transition commit | Existing object commit locks plus exact source identity and tier-generation checks in the transition path | Publish the exact remote tuple into the matching local version | Publishing a tuple after the source identity or generation changes |
| Transition transaction cleanup | Record validation; exact backend-generation lease inside the probe/delete helper. Current recovery has no explicit bucket-lifecycle/physical-set ownership fence or durable takeover CAS | Identity-bound provider probe and deletion of a known canonical candidate | A tier lease alone does not fence the creator. The approved target requires expired ownership, durable takeover, and exact local/source reread before DELETE |
| Transition transaction cleanup | Record validation; expiry check; a next-state ETag CAS for cleanup ownership; exact backend-generation lease inside the probe/delete helper. Current recovery has no explicit renewable owner lease or bucket-lifecycle/physical-set ownership fence | Identity-bound provider probe and deletion of a known canonical candidate | The cleanup-state CAS fences a stale predecessor, but the approved target also requires an explicit recovery lease and exact all-pool source reread before DELETE |
| Recovery artifact quota admission (approved target) | Cluster-scoped recovery-admission WRITE lock first; then the one canonical control/source operation lock and any source-protocol metadata/physical locks in that protocol's existing stable order | Bounded internal artifact inventory, exact source/control validation, and create-only installation plus strong readback of one fully encoded export or disposition candidate | Acquiring admission while holding a control, source, disposition, bucket, physical, migration, or decommission guard; remote backend I/O; source-journal deletion; disposition `Applying`; releasing admission before candidate installation/readback converges |
| Manual job | Initial maximum-parity job write; then persisted bucket-level scope create/CAS; later short job/task/result metadata operations | List, checkpoint, append tasks before enqueue, append results after work, renew/take over lease | Holding metadata guards across remote transition PUT; treating the local active-job map as cluster authority. A crash between the job write and scope claim can leave `Running` without a scope record |
| Decommission receipt | Decommission coordinator's source/target record workflow; record-specific conditional writes | Copy/validate durable record, advance receipts, construct and verify expected manifest, conditionally clean exact covered source | Remote tier DELETE; deleting an uncovered or divergent source record |
The tier mutation lock scope is intentionally recorded as **current**, not ideal. Reducing it is allowed only after a durable `Prepared` intent blocks new reference creators across the fleet, existing tier-operation leases drain, and recovery can reconstruct that block without the initiating process. Which network validation can move outside the namespace lock is an **open design**.
Tier mutation backend validation is outside both exclusive guards and is bound to the persisted ETag snapshot. The initiating task is detached from the admin request so cancellation cannot interrupt validation cleanup. Once the coordinator Prepared record and local fence are durable, lease drain, all-node peer Prepare, and reference proof run without the tier-config namespace or `admin_updates` guard. Recovery treats an unexpired Prepared record as active even while the old config ETag remains current. Both guards are reacquired in namespace-then-admin order, and the ETag, candidate digest, exact intent identity, and expiry are revalidated immediately before config CAS. After the coordinator advances to Committed, the guards are released again for peer Commit and local publication.
## Transition transaction
@@ -114,7 +117,7 @@ UploadStarted -> Uploaded -> LocalCommitStarted -> Committed
\-> AbortedNoRemote
```
Separately, `mark_cleanup_pending` permits proof-checked model edges from `Uploaded`, `UploadOutcomeUnknown`, and `LocalCommitStarted`. Current production code emits `CleanupPending` only when recovery probes `UploadOutcomeUnknown` as `UnversionedPresent` or as `VersionedPresent` with a non-nil identifier. The `Uploaded` abort/recovery path deletes its candidate and transaction record directly, and `LocalCommitStarted` mismatch or missing-source recovery retains the record. The `Uploaded` and `LocalCommitStarted` cleanup edges are currently exercised through the state-machine API and tests, not produced by runtime recovery. States that require a remote delete still require a known `TransitionRemoteVersion` kind. A probed versioned candidate whose identifier parses as a nil UUID is another current special case: recovery exact-deletes it and removes the record without first persisting `CleanupPending`.
Separately, `mark_cleanup_pending` permits proof-checked model edges from `Uploaded`, `UploadOutcomeUnknown`, and `LocalCommitStarted`. Current production recovery emits `CleanupPending` after an expired `Uploaded` record wins the exact successor CAS, or when an expired `UploadOutcomeUnknown` probe returns `UnversionedPresent` or `VersionedPresent` with a non-nil identifier. `LocalCommitStarted` mismatch or missing-source recovery retains the record; that cleanup edge is currently exercised through the state-machine API and tests, not produced by runtime recovery. States that require a remote delete still require a known `TransitionRemoteVersion` kind. A probed versioned candidate whose identifier parses as a nil UUID is retained and never authorizes remote deletion.
The remote candidate itself is named by `canonical_transition_remote_object` under `ilm/transition-transactions/<bucket-hash>/<transaction shards>/<transaction-id>/<write-id>`. That deterministic identity is what a provider probe or exact cleanup must bind; it is distinct from the internal transaction-record key.
@@ -124,13 +127,13 @@ The creator owns the canonical remote candidate until local metadata commits the
| Observed durable state/input | Unique current owner | Current recovery decision | Approved destructive admission |
|---|---|---|---|
| `UploadStarted` | Originating transition attempt; current durable exclusion is incomplete | Retain | No delete. The upload may still publish |
| `UploadOutcomeUnknown`; exact provider probe says missing | Transaction recovery, logically; current record writes do not durably exclude a concurrent worker | Delete the record | Strong probe identity must match transaction/backend; no remote delete occurs |
| `UploadStarted` | Originating transition attempt; no current recovery successor is emitted | Retain | No delete. The upload may still publish |
| `UploadOutcomeUnknown`; exact provider probe says missing | Transaction recovery under the exact record generation and recovery lock | Conditionally delete the record | Strong probe identity must match transaction/backend; no remote delete occurs |
| `UploadOutcomeUnknown`; probe returns `UnversionedPresent` | Transaction recovery, with operator reconcile available after expiry | Persist `CleanupPending`, delete the unversioned candidate, delete the record | Exact transaction/canonical object/backend identity, explicitly unversioned state, durable takeover after owner expiry, current tier lease, and exact reread before cleanup |
| `UploadOutcomeUnknown`; probe returns `VersionedPresent` with a non-nil exact identifier | Transaction recovery, with operator reconcile available after expiry | Persist `CleanupPending`, exact-delete that versioned candidate, delete the record | Exact transaction/canonical object/backend identity and remote version, durable takeover after owner expiry, current tier lease, and exact reread before cleanup |
| `UploadOutcomeUnknown`; probe returns `VersionedPresent` whose identifier is a nil UUID | Transaction recovery | Current code directly exact-deletes that versioned candidate and deletes the record; it does not persist `CleanupPending` | This remains a versioned exact-delete candidate and must not be treated as `UnversionedPresent`. The approved target still requires durable takeover, a current tier lease, and exact reread |
| `UploadOutcomeUnknown`; probe returns `VersionedPresent` whose identifier is a nil UUID | Transaction recovery retains ownership evidence | Retain | A nil identifier is invalid exact-version evidence and never becomes unversioned or remote-delete authority |
| `UploadOutcomeUnknown`; probe ambiguous, unsupported, or errors | Transaction recovery retains ownership evidence | Retain | No destructive action; operator reconcile may inspect after expiry |
| `Uploaded` | Originating transition attempt; current recovery can race it because the persisted fence is not CAS-protected | Current code immediately deletes the candidate and record | **Current safety gap:** approved behavior must first prove the creator cannot still commit by expired ownership plus durable takeover/CAS, then recheck that no matching local commit exists |
| `Uploaded` | Originating transition attempt until expiry; after expiry, the worker that wins `Uploaded -> CleanupPending` by exact ETag CAS | Retain while active; after expiry, persist `CleanupPending`, then recheck and delete the unreferenced candidate or record | Current CAS fences the predecessor, but approved v2 also requires a durable recovery lease, full all-pool source/free-version proof, and before/after fence checks |
| `LocalCommitStarted`; logical source lookup returns `TRANSITION_COMPLETE` with the same remote object, tier, and remote version | Transition committer until ownership transfers to `xl.meta` | Delete transaction record | Current recovery treats this tuple as ownership transfer. The approved target additionally compares recorded source version ID, data directory, modification time, size, and ETag before conditional terminal cleanup |
| `LocalCommitStarted`; logical source is missing, its transition tuple differs, or the read is uncertain | Transaction record/recovery | Retain | No remote delete without a separate durable cleanup proof |
| `CleanupPending`; logical source lookup returns the same current transition predicate | `xl.meta` is remote reachability owner; recovery owns only record cleanup | Delete transaction record | `xl.meta` is owner; do not delete remote. The approved target adds the full recorded source comparison |
@@ -148,12 +151,12 @@ The current background loop runs every 60 seconds, scans at most 1,000 records p
### Approved target and open design
- Replace unconditional transaction create/update/delete with create-only, ETag CAS, and conditional terminal delete.
- Preserve the current create-only transaction installation, exact record/ETag successor CAS, and conditional terminal delete, and add mandatory exact-successor strong readback for lost or uncertain responses. No v2 work may regress the existing v1 protections.
- Recovery of `Uploaded` and cleanup-capable states must acquire durable ownership only after the prior owner's expiry. The recovery worker must reread the exact generation after takeover and before remote DELETE.
- Preserve and compare source version ID, data directory, modification time, size, and ETag through local commit and recovery before accepting ownership transfer.
- Recompute and require the exact lowercase sharded path in both transition-transaction and manual-job runtime recovery, and validate a truncated page's continuation token before processing any record from that page.
- **Open:** owner lease duration, clock-skew allowance, takeover revision encoding, and compatibility for existing v1 records that have only `not_after`/`owner_epoch`.
- **Open:** bounded retention and an operator disposition for permanently ambiguous records. Until defined, retained ambiguity is safer than collection.
- The approved v2 owner lease uses the duration, clock-skew allowance, takeover revision, and v1 mapping in [Transition transaction v2 and v1 migration](#transition-transaction-v2-and-v1-migration).
- Active automatic retries are bounded by the recovery-control policy below. Ambiguous evidence becomes explicit `retained_ambiguous` or `operator_required`; source evidence is never collected merely because it is old.
## Tier mutation intent
@@ -163,14 +166,15 @@ The current background loop runs every 60 seconds, scans at most 1,000 records p
New intents use a 15-minute expiry. A peer-only terminal tombstone is retained until that expiry plus five minutes of clock-skew allowance and until no coordinator record remains. Expiry bounds replay protection; it is not config commit/abort evidence.
The coordinator creates its durable record and peer `Prepare` blocks new reference creation, drains exact tier-operation leases, and proves that edit/remove/clear will not strand authoritative references. The coordinator then conditionally writes tier config, commits peers, publishes the runtime candidate, and clears the block. Per-mutation sharded mutexes serialize local phases only; persisted intent plus tier-config ETag is authoritative.
The coordinator creates its durable record and peer `Prepare` blocks new reference creation, drains exact tier-operation leases, and proves that edit/remove/clear will not strand authoritative references. Prepare, Commit, and Abort use all-node fanout rather than quorum: independent peer calls use a work-conserving concurrency limit of four, a 30-second per-peer deadline, and a 30-second fanout-wide deadline; Prepare is additionally capped by the intent expiry. The coordinator collects every completed outcome. A timed-out or otherwise ambiguous started Prepare is included in compensating Abort because cancellation does not prove the peer failed to persist its fence; peers not started before the fanout deadline make Prepare fail but do not require Abort. The coordinator then conditionally writes tier config, durably commits the coordinator intent, releases its exclusive guards, requires every prepared peer to commit, publishes the runtime candidate, and clears the block. Per-mutation sharded mutexes serialize local phases only; persisted intent plus tier-config ETag is authoritative.
### Recovery decisions
| Observed durable state/input | Unique current owner | Current recovery decision | Destructive/config admission |
|---|---|---|---|
| `Prepared`; current tier-config digest equals candidate | Coordinator recovery; each peer recovery owns only its matching peer record/block | CAS to `Committed`, replay peer Commit and publish | Exact mutation identity and candidate digest; never infer from expiry |
| `Prepared`; current config still proves the old ETag/config | Coordinator recovery | Fan out canonical Abort, then CAS `Aborted` | Abort only the matching intent; a delayed matching Prepare converges to the tombstone |
| `Prepared`; intent is unexpired and current config still proves the old ETag/config | The live coordinator remains owner | Retain `Prepared` and the runtime block | Recovery must not abort work that may still be in lock-free peer Prepare or reference proof |
| `Prepared`; intent is expired and current config still proves the old ETag/config | Coordinator recovery | Fan out canonical Abort, then CAS `Aborted` | Abort only the matching intent; a delayed matching Prepare converges to the tombstone |
| `Prepared`; config is a third generation, unreadable, or peer outcome is ambiguous | Coordinator record remains owner of the block | Retain `Prepared` and runtime block | No commit, abort, cleanup, or unblock |
| `Committed` | Coordinator recovery, with peers owning convergence of their local records | Replay peer Commit/runtime publication; clean exact converged records | Config ETag/digest and peer identity must match |
| `Aborted` | Coordinator recovery; peer recovery retains the local tombstone | Replay/confirm Abort and clear matching block; retain peer tombstone until expiry plus clock skew and no coordinator | Never roll back config based on timeout alone |
@@ -183,10 +187,39 @@ Intent transitions retry an ETag race at most three times before returning a ret
### Approved target and open design
- Keep create-only, ETag CAS, exact identity comparison, canonical Abort tombstones, and lost-response readback.
- Any shorter configuration-lock window must leave a durable `Prepared` fence installed on every required peer before releasing the broad exclusion scope and must prove recovery restores that fence before admitting reference creators.
- **Open:** the exact split between backend validation, peer fanout, reference scan, config CAS, and publication; expiry must never replace config-generation proof.
- The phase split is fixed: ETag snapshot, lock-free backend validation, ETag revalidation, all-node Prepare, reference proof, ETag/digest/intent revalidation, config CAS, all-node Commit, and local publication. Backend validation uses `(old_config_etag, candidate_digest)` as its stable config generation; the durable mutation identity adds `mutation_id`. Runtime driver revisions are not persistence identities and Add does not require one before publication.
- Lease drain, peer Prepare, and reference proof run outside both exclusive guards after the coordinator Prepared record and local fence are durable. The commit path reacquires namespace WRITE then `admin_updates` and repeats the full generation/identity proof. Expiry is checked as an additional rejection boundary and never replaces config-generation proof.
- **Open:** a dedicated operator reconcile/status surface and bounded retention for irreconcilable coordinator/peer records.
## Tier validation probe intent
### Dormant current contract
`TierProbeIntent` defines a strict, checksum-protected `rustfs-tier-probe-intent-v1` envelope and the canonical key `ilm/tier-probe-intents/records/<aa>/<bb>/<probe-id>.json`. The same non-nil probe UUID also determines the remote object name `rustfs-tier-probe-<probe-id>`. The path parser rejects uppercase UUID aliases, wrong shards, extra components, and path/payload/object-name disagreement. The durable namespace registry validates this record so pool decommission cannot silently treat it as an ordinary object.
The format binds Add and Edit to the same durable mutation identity tuple `(mutation_id, old_config_etag, candidate_digest)`. The old config ETag is required even for Add because it identifies the complete persisted tier configuration generation, not whether the named destination tier already exists. Verify instead requires the current persisted config ETag and credential-independent backend identity. These are mutually exclusive tagged variants. Every record also requires the tier name, matching destination identity, immutable creator identity/epoch, positive creation time, and an owner fence with nonempty owner, non-nil epoch, and later `not_after` timestamp. In v1 that owner identity must remain exactly equal to the immutable creator identity. It never persists the process-local driver revision or credential-bearing driver fingerprint.
The dormant state graph is:
```text
UploadOutcomeUnknown -> Uploaded -> CleanupPending -> Completed
\-> CleanupPending -> Completed
\-> AbortedNoRemote
```
`UploadOutcomeUnknown` and `AbortedNoRemote` carry no remote version. `Uploaded`, `CleanupPending`, and `Completed` carry either explicit unversioned semantics or one exact nonempty opaque version. Once known, that remote version cannot change. Revision advances by one for each edge: `UploadOutcomeUnknown` is revision 1, `Uploaded` and `AbortedNoRemote` are revision 2, `CleanupPending` is revision 2 or 3, and `Completed` is revision 3 or 4. The strict decoder rejects any other state/revision pairing. The direct `UploadOutcomeUnknown -> CleanupPending` edge is reserved for a future authoritative provider probe that discovers the exact cleanup candidate after the original PUT response was lost.
Create-only, ETag CAS, exact-ETag delete, and read-with-ETag primitives exist for the record, plus a crate-level read-only inspection result that always reports both writer and destructive recovery as disabled. No Add, Edit, Verify, startup loop, periodic loop, admin HTTP route, or remote backend operation currently calls these mutation primitives. Consequently this version creates no records and authorizes no remote PUT or DELETE.
### Activation requirements
- Add/Edit must create the mutation identity before validation and reread the same `(mutation_id, old_config_etag, candidate_digest)` before every probe-intent successor. Verify must reread the same config ETag, tier, and backend identity. A credential rotation may supply usable current credentials only when the persisted destination identity remains exact; it may not weaken operation-generation checks.
- Before enabling any writer, every required node must advertise a probe-intent-specific read/retain/recovery capability. This protocol does not reuse the legacy transition-state reconciliation capability or its token. Unknown or older nodes keep validation in the current process-local mode and no durable v1 record is written.
- The v1 owner fence is immutable and must equal the creator identity. Any future takeover requires a new schema with explicit takeover proof, plus an approved lease duration, clock-skew allowance, durable owner/epoch CAS, and revalidation order. Expiry alone never permits remote DELETE.
- Remote-version discovery and deletion must use the provider-bound, destination-bound implementation and bounded request API approved for that target. Unknown, multiple, changing, unsupported, or unavailable results retain the record.
- A successful or response-lost state write must strongly reread the exact record. Remote DELETE requires a current operation-generation proof, exact destination, current credentials for that same destination, a valid fleet fence, durable takeover, and the same known remote version immediately before and after the call.
- Terminal retention, bounded scanning, metrics, and any HTTP inspect/reconcile route remain unapproved. Raw age is never cleanup evidence, and this dormant core API must not be presented as an operator endpoint.
## Manual transition job, task, result, and checkpoint
### Current contract
@@ -311,7 +344,149 @@ Single and child manifest preparation is bounded by 200,000 journals and a 32 Mi
- New destructive prefix paths use only v6 plus either one byte-compatible manifest or one parent-bound sequence of byte-compatible child manifests. No new v1-v5 sole-owner records may be created.
- Preserve the two-phase authorization barrier: all prepared records, durable barrier, all dispatched records, durable `DispatchAuthorized`, local mutation, journals committed, durable `Completed`, then remote DELETE.
- Do not downgrade every v6-aware recovery worker while v6 records remain. v5-and-older readers reject and retain v6 records; older nodes may continue producing fallback free-versions until the fleet is homogeneous.
- **Open:** bounded age/count policy and operator disposition for quarantined v1/v2, incomplete manifests, and repeatedly failing exact deletes. Capacity rejection and recovery throughput must not be “fixed” by weakening ownership proof.
- Quarantined v1/v2 records, incomplete manifests, and repeatedly failing exact deletes use the bounded retry, explicit operator state, and single-record control surface approved below. Capacity and recovery throughput must not be “fixed” by weakening ownership proof or age-deleting source evidence.
## Bounded recovery control and operator disposition
This section is an **approved target that is not implemented yet**. It closes the ownership, retry, and operator-disposition design required before backlog recovery work can change destructive behavior. It does not authorize an implementation to enable a v2 writer, take over a v1 transaction, or remove a legacy journal until the fleet and storage gates below exist.
### Recovery-control record
Retry scheduling is persisted separately from the source transaction or journal so legacy bytes remain readable and their cleanup ownership does not move. The control record uses schema `rustfs-ilm-recovery-control-v1`, a checksum envelope, a 16 KiB encoded-size limit, strict unknown-field rejection, and this canonical key:
```text
ilm/recovery-controls/<protocol>/<aa>/<bb>/<source-operation-digest>.json
```
Export envelopes use `ilm/recovery-exports/<protocol>/<aa>/<bb>/<export-id>.json`; disposition receipts use `ilm/recovery-dispositions/<protocol>/<aa>/<bb>/<disposition-id>.json`. `export-id` is the SHA-256 of the control ID plus exact observed source content/copy-set digests; `disposition-id` is the SHA-256 of the export ID plus the bounded action. Repeating an identical export or disposition reuses and strongly validates the same canonical object; different bytes at that ID are a conflict. Both use strict checksum envelopes and create-only installation. A control or disposition is limited to 16 KiB. An export is limited to the source protocol's own maximum encoded record size plus 64 KiB for its copy manifest and envelope; it stores the source bytes once rather than duplicating identical replica bytes.
Admission is fail-closed and cluster-wide. One control may have at most one export creation and one disposition application in flight. The immutable candidate is fully encoded before quota admission. The cluster-scoped recovery-admission WRITE lock is always outermost; a caller already holding any control, source, disposition, bucket, physical, migration, or decommission guard must release it and restart in the order above. While admission is held, a complete artifact inventory must prove that the projected totals, including the candidate, are at most 10,000 export envelopes, 10,000 disposition receipts, 1 GiB of encoded export data, and 256 MiB of encoded control/disposition data. The create-only candidate installation and exact strong readback complete before the lock is released, so neither a single candidate nor a concurrent node can oversubscribe the pre-create snapshot. A crash before installation leaves no artifact; a lost create response is resolved by exact canonical readback while admission remains serialized or after reacquiring it in the same order. Replaying an existing canonical ID consumes no new count, byte, or rate token. New creations are limited to ten per authenticated actor per minute and 100 cluster-wide per minute, with at most 32 export creations and eight disposition applications executing cluster-wide. Exceeding a count, byte, rate, or concurrency limit returns a retryable capacity result before source mutation; it never evicts evidence, interrupts an admitted operation, or blocks ordinary object I/O. The collector examines at most 100 terminal artifacts per minute and never acquires the admission lock while holding an artifact/source guard.
`source-operation-digest` is SHA-256 over the length-delimited source protocol, canonical source path, and stable semantic operation identity: transaction UUID, journal identity, or manifest operation ID. For corrupt bytes whose semantic ID cannot be trusted, the canonical path plus a `corrupt` domain separator is the stable identity; a replacement at that path conflicts with the existing control instead of resetting its history. The control's immutable identity repeats the protocol, path, stable identity, and record class. Its mutable `observed_source_generation` contains the source schema, source ETag/content SHA-256, and sorted all-pool copy-set digest. The copy set records each authoritative pool/set identity, canonical path, ETag, byte length, and content SHA-256; an unreachable pool, missing ETag, divergent copy, or incomplete listing cannot produce an actionable generation. The remaining mutable generation contains:
- `revision`, an ETag-CAS successor counter;
- `classification`: `retrying`, `retained_ambiguous`, `corrupt`, `operator_required`, `abandoned`, or `terminal`;
- `owner_id`, `owner_epoch`, `lease_acquired_at_unix_nanos`, and `lease_expires_at_unix_nanos` while an attempt is owned;
- monotonic `attempt_count`, `consecutive_failure_count`, `first_failure_at_unix_nanos`, `last_failure_at_unix_nanos`, and `next_attempt_at_unix_nanos`;
- one bounded enum `last_error_code`; no free-form provider error, endpoint, credential material, request payload, or response body is persisted;
- for an operator disposition only, the authenticated actor identifier, reason code, confirmation time, exported payload digest, and exact source/control ETags that were confirmed.
The control record is a scheduler and audit fence, not a remote-object owner. It never substitutes for the source record's version, backend, manifest, source-identity, or absence proof. Before every source CAS, local cleanup, or remote request, the worker strongly rereads every authoritative source copy and the control, requires the current observed generation to match, and then applies the source protocol's own locks and proof. A missing, stale, corrupt, divergent, or unavailable control read cannot authorize work.
Control creation is `If-None-Match: *`; every update is exact ETag `If-Match`; response loss requires exact strong readback. A legal source-state CAS does not create a new control. The stable control CAS advances `observed_source_generation` only from the exact predecessor to one source-protocol successor while preserving `first_failure_at_unix_nanos`, lifetime `attempt_count`, and first-seen lineage. If the source CAS succeeded before a crash, recovery accepts the new bytes only after validating that exact legal successor and then converges the old control generation by CAS. A multi-edge jump, semantic-identity change, replacement ETag/content, or unavailable predecessor proof is a conflict and cannot reset counters. Recovery conditionally removes a terminal control only after strongly proving the stable source operation resolved or absent, recording any active decommission terminal receipt, and confirming that no in-flight operator request still names its ETag. `durable_namespace.rs` must register every new namespace, path parser, decoder, size bound, successor relation, and terminal checkpoint before rollout.
### Transition transaction v2 and v1 migration
The successor envelope is `rustfs-transition-transaction-v2`. It preserves the v1 transaction ID, deployment ID, write ID, complete source identity, tier/backend identity, canonical remote object, remote-version state, operation state, and revision. It also records an immutable `origin_format` (`native_v2` or `migrated_v1`), the state-entry revision and predecessor state/revision, and, for migration, the exact source v1 state/revision. These fields make the state history needed for destructive authorization independently checkable instead of inferring it from the current state. The envelope and body reject unknown fields, checksum every field, use no default for a required v2 value, require the existing exact lowercase canonical path, and reject nil UUIDs, nonpositive timestamps, revision zero/overflow, lease inversion, impossible state/revision history, and illegal state/version combinations. It replaces the fixed ownership pair with:
- immutable `creator_epoch` and `creator_not_after_unix_nanos`, copied exactly from v1 `owner_epoch` and `not_after_unix_nanos` during migration;
- optional `created_at_unix_nanos`; migrated v1 records use `None` rather than inventing an age;
- mutable `owner_role` (`creator` or `recovery`), `owner_id`, `owner_epoch`, `owner_generation`, `lease_acquired_at_unix_nanos`, and `lease_expires_at_unix_nanos`. `owner_id` is the authenticated stable fleet-node identity; `owner_epoch` is a fresh UUID for one process claim. A node without both identities cannot create, renew, take over, or act on v2.
A native v2 create uses revision and owner generation 1, a fresh non-nil creator owner/epoch, `UploadStarted`, and an unknown remote version. The first rollout keeps the current seven-day creator ownership window. Creator leases are not renewable; a creator that cannot finish inside the safety window stops publishing and leaves the record for recovery. Recovery-owner leases are 15 minutes. The persisted clock-skew allowance is five minutes: an action may start only when its bounded deadline fits before `lease_expires_at - 5 minutes`, and another owner cannot take over until its local time is at least `lease_expires_at + 5 minutes`. A recovery attempt remains capped at five minutes and every remote call remains subject to its narrower client deadline. A recovery renewal is a same-owner, next-revision CAS that strictly extends expiry; a takeover changes `owner_role`, `owner_id`, and `owner_epoch`, increments `owner_generation` and `revision`, and uses the exact observed ETag. Takeover and state advancement are separate CAS operations; one revision cannot both acquire ownership and claim a recovery outcome.
Before migration can be enabled, the fleet must first deploy a v1 creator fence that strongly rereads the exact transaction path, ETag, state, owner epoch, and source generation immediately before local metadata publication and refuses to publish after any migration/takeover change. The fleet then durably disables new v1 admission and proves that every captured creator/recovery process epoch has either acknowledged quiescence or terminated. A paused or unreachable epoch prevents migration. This drain barrier is distinct from format capability advertisement and remains in force until v2 writer admission is enabled.
Only these checksum-valid v1 state/revision pairs are migration inputs: `UploadStarted@1`; `UploadOutcomeUnknown@2`; `AbortedNoRemote@2`; `Uploaded@2` or `Uploaded@3`; `LocalCommitStarted@3` or `LocalCommitStarted@4`; `Committed@4` or `Committed@5`; and `CleanupPending@3`, `CleanupPending@4`, or `CleanupPending@5`. Any other pair is `corrupt`, inspect-only, and cannot be migrated or authorize a probe, local cleanup, or remote DELETE. A native v2 record must prove a legal predecessor edge at its recorded state-entry revision; ownership-only revisions may increase the outer revision but cannot change the recorded state-entry history. Missing, contradictory, or skipped history is corrupt.
An identity-preserving `v1 -> v2` conversion is one legal successor with `revision + 1` and an unchanged remote tuple. The state is unchanged except that every v1 `UploadStarted` maps conservatively to v2 `UploadOutcomeUnknown`. Historical v1 writers could issue PUT while still in `UploadStarted`; no age, fleet version, or current process observation proves that a retained record came from the later pre-PUT-fence writer. It is permitted only when:
1. every node that can create, commit, recover, heal, or decommission the record advertises both `transition_transaction_v2` and `ilm_recovery_control_v1` for the captured fleet/topology generation, the durable v1-admission stop is active, and the process-epoch drain barrier above is complete;
2. current time is at least the v1 `not_after_unix_nanos` plus five minutes of skew;
3. the canonical path, checksum, full immutable identity, state, remote-version invariant, source record, and ETag all match the observed v1 generation;
4. the migration CAS and strong readback install one fresh recovery lease before any state transition or side effect.
The original creator must successfully CAS `UploadStarted -> UploadOutcomeUnknown` before issuing remote PUT, and must CAS the exact current owner generation to `LocalCommitStarted` before publishing local metadata. A v2 takeover therefore fences a delayed creator. An implementation that can issue PUT or publish after losing this CAS is not compatible with this protocol.
After takeover, recovery applies this matrix:
| State | Approved recovery after exact takeover | Required proof before side effect |
|---|---|---|
| native-v2 `UploadStarted` | CAS `AbortedNoRemote`, then conditionally remove the terminal transaction | Valid native-v2 history proves the creator was fenced before the mandatory pre-PUT `UploadOutcomeUnknown` CAS; migrated v1 never enters this row and no remote request is made |
| `UploadOutcomeUnknown` | Probe under the exact backend lease. Missing becomes terminal cleanup; proven unversioned presence or one nonempty, non-nil exact version becomes `CleanupPending`; nil, ambiguous, or unsupported results become `retained_ambiguous` | Exact transaction/control generations, bounded live probe, current tier destination and lease |
| `Uploaded` | If the exact transitioned tuple or its free-version owns the candidate, remove only the transaction. If the complete original source is still unchanged and no local commit/free-version exists, CAS `CleanupPending`. Otherwise retain | All-pool source/free-version read, full source tuple, bucket incarnation, tier generation, object locks, and post-probe revalidation |
| `LocalCommitStarted` | Exact committed tuple/free-version means record-only cleanup. A fully unchanged original source with no partial committed tuple may move to `CleanupPending`. Missing, divergent, partial, or unavailable metadata is `retained_ambiguous` | Same all-pool proof, including data directory, modification time, size, ETag, transition transaction ID, remote tuple, and destination identity |
| `CleanupPending` | Resume the same exact idempotent candidate delete, or remove only the transaction when local ownership transfer is proven | Current owner lease, source/free-version proof, exact tier lease, physical locks, and before/after fence checks |
| `Committed` | Conditionally remove only the exact terminal transaction when complete local ownership transfer is proven; otherwise classify the record as corrupt or retain it as ambiguous | All-pool strong read matches the complete logical `xl.meta` source identity, transaction ID, remote tuple, destination identity, and any required decommission terminal receipt; no remote DELETE |
| `AbortedNoRemote` | Conditionally remove the exact terminal transaction | Valid native-v2 pre-PUT history or exact migrated v1 `AbortedNoRemote@2`, exact terminal generation, and any required decommission terminal receipt; no remote DELETE |
Remote DELETE is never admitted directly from `UploadStarted`, `UploadOutcomeUnknown`, `Uploaded`, or `LocalCommitStarted`; it first requires a CAS-protected `CleanupPending` generation with known remote-version semantics. A lost source read, mixed all-pool result, expired lease, failed renewal, or source/control CAS conflict retains the evidence and performs no destructive action.
New writers emit v2 only after the homogeneous fleet gate, durable v1-admission stop, and process-epoch drain barrier are complete. During a rolling upgrade, new readers accept v1 but all writers continue v1 and no v1 takeover/migration occurs. A v1 reader rejects and retains v2. Downgrade is blocked until v2 creation is disabled and all v2 transactions and recovery-control records are drained or exported; live v2 bytes are never rewritten to v1.
### Retry and retention policy
An attempt that loses a CAS or discovers a newer source generation reloads instead of recording a remote failure. A retryable transport timeout, backend 5xx/throttle, metadata quorum outage, or bounded remote-delete failure increments the persisted counters and schedules:
```text
min(60 seconds * 2^min(consecutive_failure_count - 1, 6), 1 hour)
```
A deterministic multiplier from 80 to 100 percent, derived from the source-generation digest and attempt count, is applied to that capped base. The jitter can only shorten the delay and therefore never exceeds the one-hour cap; restarts reproduce the same deadline without synchronizing a fleet. Success or a proven source-state advance resets `consecutive_failure_count` but never decreases `attempt_count`. `next_attempt_at_unix_nanos` is only a not-before scheduler hint; ownership and destructive authority still require the lease and source proofs.
After 32 consecutive retryable failures or seven days since `first_failure_at_unix_nanos`, whichever occurs first, the control CAS moves to `operator_required` and automatic attempts stop. Unsupported probes and unknown remote-version semantics move directly to `retained_ambiguous`; corrupt source bytes use `corrupt`; incomplete destructive evidence uses `operator_required`. None is periodically hot-looped. An operator may explicitly request another bounded attempt after the underlying capability or configuration changes, but the request creates a new owner lease and preserves the lifetime attempt count.
This policy bounds automatic work, not evidence lifetime. A source transaction, journal, or manifest is never deleted solely because it is old, numerous, or over a byte threshold. `operator_required` source evidence remains until its protocol reaches a proven terminal state or the legacy-journal disposition below is completed.
Resolved control tombstones are retained for at least 30 days, immutable export envelopes for at least 90 days, and compact completed disposition receipts for at least 365 days. A collector may conditionally remove only a terminal artifact past its floor after proving that the bound source generation is absent where required, no nonterminal successor or active decommission references it, and the terminal audit checkpoint is durable. Capacity pressure blocks new export/disposition work rather than evicting unexpired or nonterminal evidence. Uncertainty retains the artifact; collection never authorizes source or remote deletion.
### Legacy journal and manifest disposition
Journal v1 has neither backend identity nor remote-version authority; v2 has backend identity but still lacks remote-version semantics. Their automatic classification is `retained_ambiguous`, and neither recovery nor an operator action may instantiate a backend or issue remote PUT, GET, probe, or DELETE from those bytes. The approved single-record actions are:
- **inspect**: strictly decode a server-reconstructed canonical journal identity, perform an all-pool strong read, and return a redacted copy-set/content digest, version, quarantine reason, control classification, age information when known, topology readiness, and decommission coverage; it changes nothing and does not return raw object/version fields by default;
- **export**: after a fresh exact inspect, create-only persist an immutable `rustfs-ilm-recovery-export-v1` envelope containing the raw source bytes and sorted copy manifest, then strongly read it back. The response downloads that envelope rather than rereading the live journal, uses no-store/attachment semantics, and never adds credentials or backend configuration;
- **abandon after export**: v1 or v2 only; create a `Prepared` `rustfs-ilm-recovery-disposition-v1` receipt bound to the immutable export and every source copy, conditionally remove only those exact local journal generations, prove every bound copy absent with no replacement, and advance the receipt through `Applying` to `Completed`. This accepts a possible remote storage leak and never asserts that cleanup occurred.
Export and abandon require a fresh all-member capability/topology proof including each member's current process epoch; inspect may remain available in a mixed fleet but returns not-ready for mutation. `abandon after export` uses POST and requires `confirm: true`, `action: abandon_remote_cleanup`, `acknowledge_remote_cleanup_abandoned: true`, the export operation ID/digest, source content and copy-set digests, every source ETag, control ETag, and a bounded operator reason code. It is refused while an active decommission or migration receipt covers either record, while physical copy discovery is incomplete, or when the implementation cannot target every discovered copy with its own `If-Match` condition.
The disposition receipt has immutable action/export/control identities and an immutable sorted copy manifest. Its ETag-CAS generation contains state `Prepared`, `Applying`, or `Completed` and a monotonic sorted `confirmed_absent` set naming only entries from that manifest. Before `Prepared -> Applying`, a fresh all-pool read must find every bound copy at its exact ETag/content digest and repeat the fleet, lock, migration, and decommission checks. The manifest can never be widened, reordered, or replaced.
During `Applying`, recovery treats each manifest entry independently while retaining the original all-pool boundary. An entry already in `confirmed_absent` must still be strongly absent with no successor or replacement generation. For an unconfirmed entry, an exact ETag/content match may be conditionally deleted and then added to `confirmed_absent` only after strong absence readback. If a crash or lost response left that exact path absent before the progress CAS, recovery may add it only after the same strong absence, stable source/control generation, topology, process-epoch, migration, and decommission proofs establish that no replacement exists. A different ETag/content, an unbound copy, an unreadable member, or loss of any proof is a conflict and preserves the current progress. Thus a crash after deleting copy A but before recording its progress can converge and continue with copy B without requiring deleted copy A to reappear.
Immediately before each local metadata deletion, the server repeats the applicable all-pool/fleet/lock checks and conditionally targets only the still-unconfirmed exact ETag. Completion requires every immutable manifest entry in `confirmed_absent`, a fresh all-pool proof that all remain absent without replacement, unchanged fleet/process epochs, and no active decommission or migration coverage. A lost final response is success only when strong readback proves the canonical receipt `Completed`. Recovery may repeat only this canonical operation and never creates a tier client or issues a backend request.
Malformed or unsupported bytes whose outer v1/v2 identity cannot be proven are inspect-only and cannot use abandon. Versions v3-v6 never use `abandon after export`. Their known candidate or manifest ownership must converge through the normal exact protocol. An operator may inspect/export and request a bounded retry, but cannot bypass source/free-version proof, manifest membership, topology, or remote-version validation. `Preparing`/`Aborting` manifests may use their existing whole-set rollback; `DispatchAuthorized`/`Completed`, a missing member, a nonempty operation namespace, or any uncertain binding cannot be manually removed.
### Admin and metrics contract
The approved surface is single-record and uses a protocol-specific expected tuple; it does not reuse the legacy metadata-reconcile digest or create a bucket/prefix job:
```text
GET /rustfs/admin/v3/ilm/recovery/records?protocol=<protocol>&classification=<classification>&limit=<n>&marker=<opaque>
GET /rustfs/admin/v3/ilm/recovery/records/<control-id>
POST /rustfs/admin/v3/ilm/recovery/records/<control-id>
GET /rustfs/admin/v3/ilm/recovery/exports/<export-id>
```
List and redacted inspect require `admin:ListTier`. Raw export creation/download, retry, or abandon require `admin:SetTier` because legacy bytes can reveal bucket, object, tier, and remote-version information. The default page is 100 records and the hard maximum is 1,000. A truncated page without a continuation marker is an error, and counts from an incomplete scan are labeled incomplete rather than reported as zero or complete.
Inspect returns a 15-minute opaque observation receipt bound to the authenticated actor, canonical record identity, source/copy ETags and digests, topology/fleet generation, every member process epoch, action class, issue/expiry time, and nonce. It is observation evidence, not mutation authority. POST requires that receipt and `confirm: true` for terminal actions, and binds the control ETag/revision/classification, requested action, and export digest when applicable. The server repeats all live proofs; a restarted member, membership change, or process-epoch mismatch invalidates the receipt. Client fields are concurrency guards, not authority. Export download reads the immutable envelope, uses TLS plus `Cache-Control: no-store` and attachment disposition, and never logs the raw payload.
Metrics use bounded labels only:
- `rustfs_ilm_recovery_records{protocol,classification,schema}` and `rustfs_ilm_recovery_oldest_age_seconds{protocol,classification,schema}`;
- `rustfs_ilm_recovery_attempts_total{protocol,outcome,error_code}`;
- `rustfs_ilm_recovery_operator_actions_total{protocol,action,outcome}`;
- scan completeness, corrupt-record, and orphan-control counters.
Every label value is a closed enum. `schema` exposes recognized schema identifiers only; an unrecognized raw value maps to `unknown`, while a recognized future schema disabled by the current fleet maps to `unsupported`. `protocol`, `classification`, `outcome`, `error_code`, and `action` likewise map unknown input to one bounded fallback and never expose decoded or operator-provided text.
Object names, bucket names, tier names, transaction IDs, control IDs, endpoints, ETags, error text, and credentials are never metric labels. Admin output may identify the selected record but redacts credentials and raw backend configuration. Audit/log events use the repository ILM event fields, stable reason codes, authenticated actor, source/control generations, action, and outcome; they do not persist or log provider response bodies.
One canonical source generation counts once regardless of its physical copy count or how many recovery passes observed it. Attempt counters increment once per coordinator attempt, not per replica, CAS retry, or page revisit. Aggregate totals and oldest age are authoritative only after a complete all-pool scan; partial coverage reports `incomplete` and never publishes a false zero. A legacy record's first-seen time comes from its durable control record rather than an inferred object modification time.
### Required protocol fixtures
Implementation acceptance requires deterministic crash/restart and mixed-version fixtures, not timing-only tests. At minimum they cover:
- a historical v1 `UploadStarted@1` whose PUT may have reached the provider, proving migration yields `UploadOutcomeUnknown` and never `AbortedNoRemote` or a direct delete;
- every accepted v1 state/revision pair above plus checksum-valid impossible pairs such as `CleanupPending@1`, proving impossible history is inspect-only and makes zero backend calls;
- an in-flight v1 creator interleaved with admission stop, process-epoch drain, migration, takeover, and local publication, proving no stale creator can publish after takeover;
- `Committed` with complete, missing, partial, divergent, and unavailable all-pool `xl.meta` ownership proof, proving only the complete exact tuple permits record cleanup;
- an old reader retaining v2, a mixed fleet blocking v2 writer and migration, and downgrade refusing until v2/control records are drained or exported;
- operator abandon across a crash after one per-copy delete, a lost delete response, a lost progress CAS, a replacement ETag, incomplete topology, and active decommission, proving progress is monotonic, replacements survive, unsafe cases retain evidence, and every case issues zero backend PUT, GET, probe, or DELETE calls;
- canonical export replay, a crash before candidate installation, a lost create response, concurrent admission at the remaining-byte boundary, and actor/cluster count, byte, rate, and concurrency exhaustion, proving duplicate IDs consume no new quota, projected totals never oversubscribe, and admission failure mutates no source evidence.
## `xl.meta` free-version boundary
@@ -417,7 +592,7 @@ A bucket/prefix/fleet batch reconcile is still an **open design**. It requires a
Decommission cannot treat durable ILM objects as ordinary configuration blobs. `validate_durable_ilm_record` validates namespace, size, schema/checksum, identity, and a protocol-specific checkpoint, and most protocol branches recompute the canonical path. Its transition-transaction branch currently inherits the weaker final-component parser: mismatched shard directories, extra components, and uppercase hex can pass when the final UUID and record contents agree. Exact transition-path validation is therefore an approved target, not a current decommission guarantee. Checkpoint successors enforce journal/manifest legal states, chunk-parent revision/sequence/count/binding progression, transition identity and revision progression, monotonic manual-job progress, scope ownership, and immutable task/result payloads.
The decommission coordinator copies and validates a durable record on a target, persists a receipt for that exact source path/identity/checkpoint, and records the expected receipt set on the source. While a matching decommission operation is active, protocol writers advance receipts as records change and terminal cleanup records a terminal checkpoint before deleting a covered record. Without an active decommission operation, the receipt helper creates no terminal receipt and ordinary protocol recovery proceeds with that protocol's current delete primitive: v6 journal/manifest/parent cleanup uses the exact ETag, while transition-transaction cleanup remains unconditional as documented above. Completion verifies every expected receipt and target checkpoint before the source pool can be removed.
The decommission coordinator copies and validates a durable record on a target, persists a receipt for that exact source path/identity/checkpoint, and records the expected receipt set on the source. While a matching decommission operation is active, protocol writers advance receipts as records change and terminal cleanup records a terminal checkpoint before deleting a covered record. Without an active decommission operation, the receipt helper creates no terminal receipt and ordinary protocol recovery proceeds with that protocol's current exact ETag conditional-delete primitive. Completion verifies every expected receipt and target checkpoint before the source pool can be removed.
A terminal receipt is proof that an exact target copy reached a terminal checkpoint. It may authorize conditional removal of the matching source record when every active target copy is covered; it never authorizes remote DELETE. A terminal receipt on one target cannot hide a later nonterminal receipt on another target.
@@ -427,7 +602,7 @@ Receipts have no enum state. Their legal evolution is `absent -> checkpoint -> m
| Observed state | Unique current owner | Current recovery decision | Destructive admission |
|---|---|---|---|
| No active decommission run | Underlying protocol owner | Protocol recovery proceeds normally and no receipt is created. An eligible v6 journal/manifest record is directly removed by exact ETag; transition-transaction cleanup follows its documented current unconditional path | Receipt state grants no remote-delete authority |
| No active decommission run | Underlying protocol owner | Protocol recovery proceeds normally and no receipt is created. Eligible v6 journal/manifest and transition-transaction records are removed only by their exact observed ETag | Receipt state grants no remote-delete authority |
| Source and target exact identity/checkpoint agree | Decommission coordinator for the run token | Create or CAS-advance the run-scoped receipt | Successor must be monotonic and topology-bound where required |
| Receipt already covers the same successor | Decommission coordinator for the run token | Treat as idempotent | Exact identity/checkpoint only |
| Conflicting receipt, checksum/schema/path error, divergent target record, missing ETag, or non-successor checkpoint | No decommission actor acquires cleanup authority | Fail decommission and retain source | Never overwrite or guess |
@@ -451,7 +626,7 @@ Receipt and expected-manifest create/CAS conflicts retry at most three times. Ex
### Approved target failure matrix
The matrix below is the normative approved target, not a blanket description of current implementation. Current exceptions are authoritative only where each protocol section above labels them explicitly. In particular, transition-transaction initial and successor writes and deletes are currently unconditional, while the transition transaction's initial write and the manual job's initial write have neither create-only installation nor mandatory lost-response strong-readback convergence.
The matrix below is the normative approved target, not a blanket description of current implementation. Current exceptions are authoritative only where each protocol section above labels them explicitly. Transition transaction v1 now has create-only installation, exact ETag successor CAS, and conditional terminal deletion, but it still lacks mandatory lost-response exact-successor readback and a renewable durable recovery lease. The manual job's initial write remains non-create-only.
| Event | Approved result |
|---|---|
@@ -462,19 +637,20 @@ The matrix below is the normative approved target, not a blanket description of
| Crash after remote DELETE but before journal/free-version cleanup | Retry the same exact idempotent DELETE under the same fences, then conditionally clean local evidence |
| Cancellation | Stop issuing new work, persist monotonic cancellation where the protocol has it, and leave ambiguous durable records for recovery. Cancellation is never rollback proof after authorization |
| Rolling upgrade | Gate writers on the minimum capability required by the format. Known older journal/RPC versions follow their explicit compatibility rule; unknown formats are retained |
| Downgrade | Drain v6 journals before removing all v6-aware workers. Do not write a new format until its downgrade reader behavior and writer gate are specified |
| Downgrade | Drain v6 journals and any enabled transition-v2/control protocol before removing their capable workers. Do not write a new format until its downgrade reader behavior and writer gate are specified |
| Corrupt or unknown input | Record a diagnosable failure, retain bytes, and block destructive action/completion |
Transition transaction v1, manual job/task/result v1, and receipt v2 do not currently have a complete persisted-format negotiation for rolling downgrade. Until one is designed, caller/operator orchestration must not enable writers whose records required recovery nodes cannot decode. The manual async endpoint does not enforce that fleet gate and a direct request proceeds to job creation. This caller-side fail-closed rule is stricter than treating an unknown record as absent.
Transition transaction v1, manual job/task/result v1, and receipt v2 do not currently have an implemented persisted-format negotiation for rolling downgrade. The approved transition-v2/control gate above is not current behavior. Until the applicable gate is implemented, caller/operator orchestration must not enable writers whose records required recovery nodes cannot decode. The manual async endpoint does not enforce that fleet gate and a direct request proceeds to job creation. This caller-side fail-closed rule is stricter than treating an unknown record as absent.
### Current format compatibility decisions
| Family/version | Current reader and writer behavior | Upgrade, downgrade, and ignore rule |
|---|---|---|
| Transition transaction v1 | Writers emit v1; the payload decoder rejects another schema, bad checksum, unknown state, or inconsistent transaction/remote identity. The current record-path parser accepts any shard/extra-component layout and uppercase hex when the final 32-hex UUID parses and matches the payload | There is no intentional ignore path, but exact lowercase canonical-path rejection remains an approved fix. A future schema needs a fleet writer gate and an old-reader retention test before rollout; downgrade behavior is open |
| Transition transaction v1 | Writers emit v1; the payload decoder rejects another schema, bad checksum, unknown state, or inconsistent transaction/remote identity. The current record-path parser accepts any shard/extra-component layout and uppercase hex when the final 32-hex UUID parses and matches the payload | There is no intentional ignore path, but exact lowercase canonical-path rejection remains an approved fix. V1 remains the only writer format until the approved v2 fleet gate is implemented; a v2 reader never rewrites an active v1 record |
| Transition transaction v2 and recovery-control/export/disposition v1 | Approved target only; no current reader or writer emits these formats | Roll out read support before the homogeneous writer gate; old readers reject and retain. Disable creation and prove all active records drained before downgrade; never rewrite v2 to v1 |
| Tier mutation intent v1; peer RPC v3/v4 | Durable readers/writers require intent v1. New peers accept signed/canonical v3 and v4 RPC; old v3 peers return an exact authenticated unsupported response to v4 | Pause and drain edit/remove/clear across the mixed interval; do not automatically retry v4 as v3. Unknown durable intent is retained and blocks recovery |
| Manual job/scope/task/result v1 | Writers emit the v1 family. Manual-job runtime recovery accepts an uppercase UUID path when both shard strings match its uppercase prefix, then loads the lowercase canonical job by UUID; the decommission validator recomputes the canonical path and rejects that alias. Other decoder/path/checksum failures stop reconciliation. Runtime capabilities advertise `enqueue_only` and `async`, but the async run handler does not consult a fleet capability gate and a direct request creates a job | Runtime recovery still needs exact lowercase canonical-path validation to prevent alias-driven duplicate work. Caller/operator orchestration must verify every required node and fail closed when capability is unknown or unsupported. An automatic server-side fleet gate and persisted downgrade negotiation remain open; unknown records are never ignored as completed work |
| Journal v1/v2 | Readers decode but quarantine because remote-version authority is missing; compatibility writers can preserve these forms | Retain indefinitely unless a separately approved, authoritative repair protocol resolves them; never translate empty version ID to known-disabled |
| Journal v1/v2 | Readers decode but quarantine because remote-version authority is missing; compatibility writers can preserve these forms | Never translate empty version ID to known-disabled or authorize remote DELETE. Retain unless the approved exact inspect/export/abandon protocol conditionally removes only the local journal generation |
| Journal v3/v4 | Readers recover supported committed records according to exact or explicit version-state semantics; current compatible writes use v4 for known state | Unknown/inconsistent state is retained. These legacy paths are not evidence that a new sole-owner operation may omit v5/v6 source proof |
| Journal v5 | Readers use stable source/all-pool proof; decoded v5 can be checkpointed, while new online sole-owner transactions are not emitted as v5 | Retain and recover conservatively during upgrade. Do not manufacture v5 from older records or use it to bypass v6 manifest authorization |
| Journal v6, dispatch manifest v1, and chunk parent v1 | v6-aware writers/readers require immutable manifest membership and topology. Complete sets at or below 200,000 retain the legacy root manifest bytes; larger sets install a strict parent at that root and operation-scoped v1 child payloads. Pre-chunking v6 readers reject the parent schema and child paths, while v5-and-older readers reject and retain v6 journals | Gate writers on the current fleet capability and retain the root parent for the entire active chunk sequence. Drain v6 before removing all v6-aware workers; do not downgrade by rewriting a live v6 operation |
@@ -484,10 +660,10 @@ Transition transaction v1, manual job/task/result v1, and receipt v2 do not curr
| Protocol | Current operator/telemetry surface | Current retention | Required follow-up |
|---|---|---|---|
| Transition transaction | Expired unknown-upload inspect/delete/finalize routes; `lifecycle_transition_transaction_recovery` diagnostics | Terminal records are deleted; ambiguous and unsafe states may remain indefinitely | Backlog age/count/state metrics, bounded policy, and durable takeover status |
| Transition transaction | Expired unknown-upload inspect/delete/finalize routes; `lifecycle_transition_transaction_recovery` diagnostics | Terminal records are deleted; ambiguous and unsafe states may remain indefinitely | Implement the approved v2 lease/takeover, recovery-control status, bounded retry, and backlog metrics |
| Tier mutation intent | Admin mutation response plus recovery diagnostics; no dedicated reconcile API | Peer aborted tombstone through expiry plus skew; ambiguous coordinator/peer records retained | Status/reconcile view for mutation, peer convergence, config generation, and blocked tiers |
| Manual job | POST run response, GET status, DELETE cancel; runtime capabilities advertise both modes | Job/task/result history is indefinite. Terminalizers only best-effort delete the exact scope; startup skips a terminal job with a leftover scope, which remains until a later admission claimant lazily replaces it | Age/count/bytes limit and a terminal-history/scope GC protocol that preserves recovery evidence |
| Tier-delete journal/manifest | `lifecycle_tier_delete_journal` events, quarantined counter, remote-delete failure/breaker/inflight metrics | Terminal records converge; quarantined/ambiguous records are unbounded by age | Safe operator inspection/disposition, backlog age/count by version/state, bounded recovery without evidence loss |
| Tier-delete journal/manifest | `lifecycle_tier_delete_journal` events, quarantined counter, remote-delete failure/breaker/inflight metrics | Terminal records converge; quarantined/ambiguous records are unbounded by age | Implement the approved single-record inspect/export/disposition, bounded retry controls, and logical backlog metrics |
| Decommission receipt | Decommission state/events including `receipt_cleanup_failed` | Completion triggers only best-effort receipt/manifest cleanup. Delete failures reported as `receipt_cleanup_failed`, as well as abandoned runs, can leave run-scoped records behind | Run-scoped retention and resume-safe cleanup policy |
Retention is a protocol transition, not raw deletion. Any collector must name its unique owner, minimum age/count/bytes bound, exact terminal or quarantine predicate, readback behavior, decommission interaction, and audit/metric output. It may not collect a record solely because it is old.
+146 -81
View File
@@ -1,112 +1,177 @@
# Object Transaction UUID And Generation-Fencing Contract
# Object Generation Authority And Recovery Contract
**Use this when:** adding or changing anything that fences a commit, scopes a read lease, gates old-directory cleanup, binds prepared pool reads, or settles quota against "the current version of an object", or when adding a field that rides internode RPC or `xl.meta`.
**Source of truth:** `assign_object_transaction_epoch` in `crates/ecstore/src/set_disk/ops/object.rs` and `crates/ecstore/src/set_disk/ops/multipart.rs`; `FileInfo::set_object_transaction_epoch` in `crates/filemeta/src/fileinfo.rs`; `commit_rename_data_dir` and `RenameConvergence` in `crates/ecstore/src/set_disk/core/io_primitives.rs`; `PreparedPoolReadFallbackBarrier` in `crates/ecstore/src/store/rebalance.rs`; `crates/protos/src/node.proto`; env constants in `crates/config/src/constants/object.rs` and `crates/config/src/constants/internode.rs`.
**Use this when:** changing object commit fencing, rollback, old-directory cleanup, prepared reads, quota settlement, or the metadata and RPC fields used by those operations.
**Source of truth:** `crates/ecstore/src/set_disk/ops/object.rs` (`assign_object_transaction_epoch`, `verify_object_transaction_epoch_fence`); `crates/ecstore/src/set_disk/core/io_primitives.rs` (`rename_data_owned_with_fence`, `commit_rename_data_dir`); `crates/ecstore/src/disk/local.rs` (`rename_data`, `write_all_meta`); `crates/lock/src/distributed_lock.rs` (`DistributedLockGuard`, `LockLostSignal`). The implementation boundary below distinguishes existing behavior from the selected design.
Design tracking lives in `rustfs/backlog#1326`. This document holds only the invariants.
## Decision And Implementation Boundary
## Authority
The selected minimum authority is a **durable, ordered per-object decision protocol attached to the existing namespace-lock participant group**. The object transaction UUID remains an opaque operation/idempotency identifier. It is not an ordered lock epoch. Extending the existing lock group requires durable promises, accepted values, quorum decisions, and recovery; adding a counter to today's lock response is insufficient.
The target contract requires **one per-object commit identity** consumed by commit fencing, read leases, cleanup, prepared reads, and quota settlement. No consumer may mint a second value and call it the same generation.
An independent service holding every object's full manifest is not selected. It would add a new routing, membership, availability, and metadata ownership system and require a wider read/write migration. The selected protocol stores the current decision and recoverable outstanding successor with the existing lock participants; object payload and prepared metadata remain on the existing storage disks. This is still new consensus and persistence work, not a small `RenameData` patch.
What exists today is an **object transaction UUID**, not the target authority:
**Implementation status:** this document does not implement or claim distributed generation authority. Existing fencing remains an opt-in coordinator equality recheck. `rustfs/backlog#2251` cannot be completed by forwarding the UUID to disks and adding local CAS. Its implementation must be split at the protocol boundaries in [Required Implementation Boundaries](#required-implementation-boundaries), with the availability and rollout changes reviewed before strict activation. The two original requirements “commit with a quorum while a disk is unreachable” and “every disk immediately rejects every older request” cannot both hold; the precise target below preserves quorum availability.
| Property | Current implementation |
Related contracts remain authoritative for their domains: [erasure-coding.md](erasure-coding.md) defines data durability and voting, [heal-concurrency-model.md](heal-concurrency-model.md) defines namespace-lock scope, [placement-repair-invariants.md](placement-repair-invariants.md) defines placement and repair admission, and [minio-file-format-compat.md](minio-file-format-compat.md) defines format interoperability.
## Current Guarantee And Counterexamples
`assign_object_transaction_epoch` generates a random UUID for gated PUT and CompleteMultipartUpload. `FileInfo::set_object_transaction_epoch` in `crates/filemeta/src/fileinfo.rs` stores it under both internal metadata prefixes. `verify_object_transaction_epoch_fence` re-reads quorum metadata before the rename fanout, outside the eventual per-disk mutation critical section. `ObjectTransactionEpochFence::Absent` currently covers both an absent object and existing metadata without a UUID. Cleanup receipts compare UUID equality. None of these operations is a durable distributed CAS.
The lock implementation already bounds lease validity. `LockLostSignal::is_lost` includes the conservative deadline; `DistributedLockGuard::run_heartbeat` retains prior deadlines after transient RPC failure and reports loss when refresh quorum is no longer valid. `LocalClient` in `crates/lock/src/client/local.rs` keeps `LocalGuardEntry` in an in-memory map; `LockResponse` in `crates/lock/src/types.rs` contains no persisted ballot or accepted object decision. Restarting a lock participant therefore cannot supply the durable order required here.
These schedules disprove a local UUID-CAS replacement, even assuming a perfect local mutex and atomic metadata replacement. They are protocol counterexamples, not claims that a multi-node fault test has already been run.
| Schedule | Result and implication |
|---|---|
| Minting | `assign_object_transaction_epoch` mints a random non-nil UUID for PUT and CompleteMultipartUpload when the object-transaction gate is active. |
| Persistence | Written through `FileInfo::set_object_transaction_epoch` into the version's internal metadata map under the dual-key contract (`x-rustfs-internal-*` / `x-minio-internal-*`). |
| Fence check | The coordinator reads the current UUID (or `Absent`) and revalidates exact equality immediately before `rename_data`. |
| Cleanup | Old-data cleanup receipts carry the committed UUID; reconciliation deletes only when the receipt UUID still equals the current object UUID. |
| Four disks start at X; A captures expected X and stalls. B commits B on d1d3, satisfying W=3. d4 has not heard from B. A reaches d4 with expected X. | d4's exact-CAS accepts A. A durable local `highest_ballot` also accepts if d4 never received B's ballot. Neither mechanism proves rejection on every disk after a quorum commit. |
| A changes d1,d2 from X to A; after lock loss B changes d3,d4 from X to B. Each refuses the other's disks because expected X no longer matches. | Neither reaches W=3, even with all disks now reachable. Equality-CAS alone has no rule for choosing recovery, retaining uncertain work, or safely retiring it. This is a recovery/liveness counterexample, not proof of two successful intersecting write quorums. |
| A reaches W=3; its reply is lost. A's coordinator restarts and sees a partial or changed disk view. A's rollback runs after B has replaced A. | Timeout is not evidence of abort. Restoring A's backup can erase B unless rollback names its own committed effect and consults a durable decision. |
| A checks a lease/UUID; B later commits; A's blocking syscall resumes. | Another coordinator-side check, cancellation token, or process-local mutex cannot establish an atomic cross-node order. The mutation itself must consume the protocol state. |
This is an equality-CAS fence and cleanup identity. It is not a monotonic epoch, is not minted by the distributed lock grant, and is not compared atomically at each disk's `xl.meta` commit point. Documents and issues must call it the *object transaction UUID*, not proof that the generation authority exists.
A lower-ballot write on an isolated stale disk cannot become the authoritative object. A disk that has applied B must never replace B with A. An uncontacted disk may retain older committed materialization until recovery; it must not vote that state as a newer decision or authorize cleanup. Requiring all disks to learn B before acknowledging it would change W to N or require successful isolation of every unreachable disk. That availability change is rejected for the selected design and must not be hidden inside E03's tests.
### Authority modes (one must be selected)
## Authority, Identity, And State
| Mode | Contract | Persistence requirement |
|---|---|---|
| Total-ordered fencing epoch | A lock grant returns a durable per-object `(term, counter)`; every disk rejects a lower epoch at the atomic metadata commit point; the value never regresses across lock-plane restart, failover, or minority recovery. | Quorum-persisted before grant, or derived from a durable term whose full comparison cannot regress. The in-memory distributed lock entry alone is insufficient. |
| Opaque commit-generation identity | Consumers compare exact identity only; no `<` / `>` semantics. The authoritative commit performs an atomic expected-generation CAS; lease, cleanup, prepared-read, and quota contracts are phrased as "references this exact generation". | Atomic expected-identity comparison plus durable crash recovery. |
The authority key is `(bucket incarnation, bucket, object key)`, covering the whole object version set. It is not the S3 version ID. Deleting a noncurrent version, updating tags, or recording replication status can change the authoritative revision while the current S3 version remains the same. The bucket incarnation prevents reuse after bucket deletion/recreation.
The current UUID proves neither a durable total order nor a per-disk atomic CAS, so it does not decide between the modes.
The protocol has separate typed values:
## Consumer Binding
- `Ballot = (configuration epoch, counter, durable proposer ID)`, compared lexicographically only within the specified authority configuration. The proposer persists a counter before use and raises it above every observed promise. Restart never resets it; exhaustion is an error. A different process boot gets a new transport epoch, not permission to reuse a ballot for different bytes.
- `Generation = (object revision, operation UUID)`, allocated by a chosen successor decision. Revisions increase from the committed predecessor; UUIDs are compared for equality only. Never infer generation order from modification time, version ID, or UUID bytes.
- `DecisionValue = (authority key, predecessor generation, successor generation, operation kind, semantic metadata digest, per-disk prepared metadata/data receipts, outcome identity)`. Disk-specific erasure indices, checksums, and metadata blobs are bound by individual receipts, not assumed byte-identical across disks. The semantic digest includes the full version set and relevant metadata, including fields omitted from ordinary read voting.
| Consumer | Binds generation how | Key invariant | Current state |
|---|---|---|---|
| Commit fence (PUT / CompleteMultipartUpload) | Checked at `rename`, rollback restore/delete, and cleanup mutation points using the selected rule | A stale writer is rejected on **all** disks; an already-ACK'd write is never rolled back | Opt-in UUID equality recheck before rename; no per-disk atomic comparison |
| Read lease | Lease binds the exact generation observed at read time; GC runs only after every lease on that generation is released | Lease visible across nodes; crashed reader's lease reclaimed by TTL | Streaming/multipart GET holds the namespace read lock through EOF/drop (part-boundary coverage: `#6887`); no cross-node generation-bound registry |
| Old-dir GC | Cleanup job carries the committed generation and confirms no lease owns `old_dir` before deleting | `old_dir != committed_dir`; a still-referenced directory is never deleted | UUID receipt equality (`#6077`); no lease consultation |
| Prepared pool read | The prepared bundle carries the generation resolved during pool lookup; the chosen pool reuses it only after a match | Mismatch forces fallback to full metadata fanout | `PreparedPoolReadFallbackBarrier` (`#6889`) is a pool-local identity that fails closed / refetches on pool state change; it is not a cross-pool authority |
| Quota reservation | Reserve / settle record binds the exact object generation (and the ordered epoch too, if selected) | A late commit cannot settle quota for a different committed generation | Durable per-bucket ledger with independent snapshot-lease fence tokens (`#6058`); not bound to the transaction UUID |
A decision does not require a new `FileInfo` positional field. The operation UUID stays in the existing metadata map. Durable authority records and local recovery records carry the revision, predecessor, ballot, and complete decision identity. They use separately versioned records; they cannot be inferred from a version's UUID alone. `xl.meta` remains a recoverable materialization of the chosen decision in strict mode.
## Fence Coverage: Three Disk-Write Points
Checking generation only before the `rename` fanout is insufficient. The commit sequence is `tmp sync → data-dir rename → xl.meta commit → directory sync` in `crates/ecstore/src/disk/local.rs`, and `crates/ecstore/src/set_disk/core/io_primitives.rs` has two further detachable disk-write points:
1. **Rollback restore/delete.** On quorum failure each disk can restore backup metadata or delete the failed version. A stale writer's rollback must compare the expected generation, or it can overwrite or delete the winner's committed metadata. Panic, cancel, and timeout outcomes must be reaped into coordinator convergence rather than skip rollback through an early return.
2. **`commit_rename_data_dir`.** A cancel-then-detach disk-write point; the coordinator's "reap all child tasks" must include it so a cancelled writer cannot bypass fence or lease and keep deleting directories.
If generation is validated only after the data-dir rename, a fenced writer may already have renamed its data-dir into the object path, leaving a staged orphan. Either move the fence ahead of the data-dir rename, or declare that orphan an accepted residue accounted for by GC metrics.
`RenameConvergence` (`AllSuccessIdentical` / `PartialCommit` / `SignatureDivergent` / `Unknown`) is a *post-commit* heal signal on the same `rename_data` path; the fence is a *commit* gate. They compose: the fence decides whether a convergence is produced, `RenameConvergence` classifies it. A fence-aware convergence variant would be an additive enum change.
## Transport And Security
Generation and derived tokens (lease, reservation) cross node boundaries in internode RPC bodies; every such flow must be signature-bound.
| Rule | Detail |
| Input/state | Required treatment |
|---|---|
| HMAC scope | Target audience, exact service/method, timestamp, nonce, canonical body digest, receiver replay (boot) epoch. The receiver consumes the nonce in a bounded replay cache; a transmitted-but-unconsumed nonce is not replay protection. |
| Current substrate | RPC v2/v3 in `crates/ecstore/src/cluster/rpc/http_auth.rs` binds all of the above. Body-bound policy covers mutating disk RPCs including `RenameData`, whose versioned canonical body includes every `RenameDataRequest` field, so the `FileInfo` metadata map carrying the UUID is authenticated. |
| Strict switches | `RUSTFS_INTERNODE_RPC_SIGNATURE_STRICT`, `RUSTFS_INTERNODE_RPC_BODY_DIGEST_STRICT`, `RUSTFS_INTERNODE_RPC_REPLAY_SCOPE_STRICT` (`crates/config/src/constants/internode.rs`) are default-off rollout gates governed by [compat-cleanup-register.md](compat-cleanup-register.md). A generation capability may claim strong transport binding only after the relevant strict modes have converged fleet-wide. |
| Acceptance tests per consumer | Method substitution, canonical body tamper, nonce replay, receiver restart, stripped-strict-metadata negatives. |
| Object never present | An explicit absent bootstrap state, established during strict cutover; no client-supplied `None` may authorize creation. |
| Object deleted, including deletion of its last version | A durable tombstone head with its own revision; never revert to never-present. This prevents ABA and resurrection after a laggard rejoins. |
| Null version | A real version-set member, distinct from absent. Replacing/removing it changes the object head. |
| Delete marker | A real version-set member and, when latest, a deleted-current state. Preserve marker type and existing S3 behavior. |
| Noncurrent version mutation | Compare the whole-object predecessor and the selected version's identity; commit a successor even if the latest S3 version is unchanged. |
| Valid legacy metadata with no UUID | Import once during fenced cutover after ordinary quorum/format validation; assign a bootstrap generation in the authority. Never equate this with absent or silently import while old writers are still admitted. |
| Missing, malformed, nil, or conflicting dual generation keys after strict enrollment | Typed corrupt/unsupported-state error; retain data for repair. No default to zero, absent, or a new UUID. |
| Missing/lagging/replacement disk | A non-authoritative materialization target. Recover the chosen decision and validate its data before it can contribute; do not demand that it already equals the predecessor or blindly overwrite it. |
## Encoding Rules
## Durable Decision Protocol
| Rule | Reason |
Use the existing namespace-lock participant identities and routing, with an explicitly persisted authority configuration. Lock voters and erasure disks are different sets: `Qlock = floor(lock_participants / 2) + 1` decides authority; the existing per-operation `Wdata` decides recoverable object durability. A lock vote is not a shard receipt. One-node deployments still persist their single voter's state. Membership changes cannot be inferred from whichever RPC endpoints answered.
Each participant persists, per authority key and successor slot, its highest promise, highest accepted `(ballot, DecisionValue)`, and the last learned committed head. Promises and accepted records survive unlock, TTL expiry, restart, and log compaction. Durable records are written through a storage boundary below the object API; writing them through PUT would recursively acquire the same authority. `LocalClient` must not become a filesystem implementation: the lock crate consumes an injected durability interface, while the storage owner implements it.
The following is a protocol contract, not pseudocode to paste into the current rename implementation:
1. Acquire the existing object namespace write lock for admission. Read/recover the latest authority head through a fresh quorum promise/recovery barrier; reading only cached learned-head markers is insufficient because a quorum may have accepted a value before its commit notification arrived. Resolve any accepted successor before returning a head or allocating another slot. An unavailable decision quorum is an explicit failure. The lock still prevents ordinary competing work, but it is not the safety proof after lease loss.
2. Stage the new shards and exact replacement metadata in transaction-owned paths. Obtain Wdata receipts only after the required file and directory syncs. Receipts bind disk identity/incarnation, authority configuration, key, operation UUID, blob digests, and data directories. Preparing must not replace live metadata, remove old directories, or reuse a winner's directory. Metadata-only/delete operations also stage a recoverable replacement version set.
3. For successor slot `head.revision + 1`, obtain Qlock durable promises for a unique ballot. Each response returns its accepted value, if any. Adopt the value with the highest accepted ballot among the promise quorum. Only if none was accepted may the proposer offer its own candidate with that predecessor. A recovered candidate is not replaced merely because its coordinator timed out or its lease expired.
4. Validate the candidate's Wdata preparation receipts and predecessor, then obtain Qlock durable accepts of the **same** value at that ballot. A participant accepts only at or above its promise and must reject a different value at the same ballot. A value becomes chosen at Qlock acceptance. Quorum intersection plus adoption of the highest accepted value prevents a different value from being chosen in that slot. Retrying the same operation cannot allocate a second successor.
5. Learn/persist the chosen decision and publish it on storage disks through the guarded local recovery protocol below. Return S3 success only after both a durable chosen decision and Wdata durable materializations satisfy the existing operation's rules. Decision chosen but publication incomplete is `OutcomeUnknown/PendingRecovery`, never authorization to roll back the decision. A later proposer first resolves the prior slot before allocating the next one.
6. On a lost ACK, resolve by operation UUID and exact request digest. The results are `NotChosen`, `ChosenPendingPublication`, `Committed`, or `SupersededAfterCommit`. Reusing a UUID for different input is invalid. A timeout without a recovered decision stays unknown. Record idempotency outcomes until the client retry horizon and all dependent cleanup/accounting records have passed a durable retirement watermark; old requests then fail as expired rather than being treated as new.
The decision and payload retention lifetimes are coupled. Accepted/staged data is not garbage merely because no coordinator is alive. A new promise quorum can adopt a previously accepted value and finish it. If the required payload has been physically lost, fail closed and repair; never choose a different value for an already chosen slot. Persisted voter state that is lost or corrupt requires catch-up/replacement, not an empty voter with the same identity.
This is the minimum extension that turns the lock grant into a recoverable authority. A promise-only grant lacks outcome recovery; per-disk promises without a common decision allow minority uncertainty to escape into reads. The protocol requires review and executable state-machine tests before production integration. It does not imply that the current lock RPC is already a consensus implementation.
## Single-Disk Publication And Recovery
Local and remote disks execute the same guarded primitive. A remote RPC handler must decode and authenticate the request, then call that primitive; checking only in the RPC handler leaves local callers and deferred syscalls uncovered. All operations touching the object's metadata, backup, or referenced directories participate.
Under one object mutation guard, re-read local durable recovery state, verify the decision/configuration and local nonregression condition, record a write-ahead intent, sync it, perform data-directory rename and atomic metadata replacement, sync affected directories, then persist applied outcome. The guard, including ownership of any namespace/deletion lease, stays with the blocking syscall until it completes, even if its caller is cancelled. An async task disappearing must not release a guard while its syscall still runs.
A single filesystem rename does not atomically commit a sidecar plus `xl.meta`. The write-ahead record binds predecessor/successor identities and exact metadata bytes; startup recovery runs before disk readiness. Recovery replays a chosen intent forward and completes syncs. An unchosen staged operation stays private until authority recovery makes its retirement safe. Old snapshots never overwrite a newer applied local revision. Conflicting bytes for the same decision are corruption. Treat write/fsync errors and torn records as unknown until decoded and reconciled, not as successful rollback.
A laggard need not contain the predecessor. Recovery fetches the chosen decision and its verified metadata, reconstructs or validates its shards under [erasure-coding.md](erasure-coding.md), and installs that state. An empty replacement uses a fresh disk incarnation and cannot reuse old preparation receipts. A disk whose durable state claims a later decision than the supplied one rejects the operation; a conflicting same-revision digest is quarantined. Never erase a divergent disk merely because it is in the minority.
`rollback_committed_rename_std`, `rollback_inline_metadata_commit_std`, and `restore_metadata_backup` in `crates/ecstore/src/disk/local.rs` must become decision-aware before strict mode includes them. The permitted rollback is limited to a transaction's unchosen private preparation, or restoration proven by recovery to be necessary before any newer local effect. A chosen operation is repaired forward. Neither a client timeout nor a rename-tail error permits reverting an acknowledged decision.
`RenameConvergence` remains a post-publication repair signal. `PartialCommit`, `SignatureDivergent`, and `Unknown` do not decide which transaction won. Keep their diagnostics and quorum accounting; resolve authority first. Early ACK may still precede minority-tail completion after the two quorum conditions hold. Tests that inspect all disks must synchronize the tail or assert the permitted minority residue separately.
## Writer Participation
Every semantic metadata change advances the whole-object generation, including metadata-only writes. A physical repair that reproduces exactly the already chosen bytes preserves the generation and consumes that chosen decision; it must not create a new semantic value. The table specifies participation, not a generated inventory of every call site.
| Writer and current code boundary | Required generation behavior |
|---|---|
| **Do not bump `XL_META_VERSION` or `XL_HEADER_VERSION`** (`crates/filemeta/src/filemeta.rs`). | `decode_xl_headers` in `crates/filemeta/src/filemeta/codec.rs` rejects newer values outright; a bump makes every new `xl.meta` unreadable by rolling-upgrade old nodes and by MinIO. See [minio-file-format-compat.md](minio-file-format-compat.md). |
| **Do not add generation as a `FileInfo` struct field.** | Internode RPC serializes `FileInfo` with two msgpack encoders: positional-array encoding for the `read_version` family (a new positional field breaks mixed-version decode) and `encode_msgpack_named` (named-map) for `rename_data` in `rustfs/src/storage/rpc/node_service/disk.rs`. A field would have to be correct under both plus the JSON compatibility twin. Use the metadata map, which rides every encoder unchanged. |
| **Metadata-map dual key.** | The UUID lives under `x-rustfs-internal-*` / `x-minio-internal-*`; missing, malformed, nil, or conflicting dual values fail closed when fencing is active. |
| **No sidecar unless atomic.** | An epoch sidecar outside `xl.meta` is admissible only if it commits at the same atomic/CAS point as `xl.meta` with a specified crash-recovery protocol. None is implemented. |
| **Regression guard.** | The real-MinIO `xl.meta` interop fixtures in `crates/filemeta/src/filemeta.rs` must keep passing: objects written by a new node stay readable by old RustFS nodes and by MinIO in both upgrade directions. |
| PUT / data COPY: `put_object_with_old_current_size_inner`, `copy_object` in `crates/ecstore/src/set_disk/ops/object.rs` | Stage, choose, publish a successor; preserve source read protection. A metadata-only COPY is also a semantic successor, even if data directories are shared. |
| MPU: `complete_multipart_upload`, `new_multipart_upload`, `abort_multipart_upload` in `crates/ecstore/src/set_disk/ops/multipart.rs` | Complete chooses the destination object's successor. Part staging/upload metadata and abort remain in the upload namespace; they cannot delete a directory transferred to a chosen object decision. |
| DELETE, batch DELETE, null/marker removal: `delete_object`, `delete_objects_with_accounting`, `delete_object_version` in `crates/ecstore/src/set_disk/ops/object.rs`; lifecycle callers in `crates/ecstore/src/bucket/lifecycle/bucket_lifecycle_ops.rs` | Each object has its own predecessor/decision; retain tombstone authority after the last version. Prefix deletion must enumerate decisions or prove a bucket-incarnation retirement barrier; a recursive bypass is forbidden in strict mode. |
| Heal: `heal_object_with_explicit_version_regen` in `crates/ecstore/src/set_disk/ops/heal.rs` | Exact repair preserves the chosen generation and verifies full metadata identity. A version-list or semantic metadata change needs a successor. `no_lock` may skip admission only; it cannot bypass authority. In-place directory repair cannot remove a reader's live directory. |
| Transition / restore: `transition_object`, `restore_transitioned_object`, `put_object_metadata` in `crates/ecstore/src/set_disk/ops/object.rs`; `finalize_restore_metadata`, `update_restore_metadata` in `crates/ecstore/src/set_disk/replication.rs` | Each metadata transition is a successor, preserving existing operation-ID, remote tuple, and tier lease checks. Bind the transition transaction to the exact predecessor/successor; a late finalizer cannot rebase onto another restore operation. |
| Replication status and metadata/tag/retention writeback: `put_object_metadata`, `put_object_tags`, `delete_object_tags`, `merge_replication_metadata_lww` in `crates/ecstore/src/set_disk/ops/object.rs`; callers in `crates/ecstore/src/bucket/replication/replication_resyncer.rs` | Commit a field-scoped successor conditional on the exact version/content identity. On conflict, reload and revalidate the mutation; never replay a full stale `FileInfo`. Existing LWW category rules remain applicable within that validation. |
| Rebalance/decommission: `migrate_entry_version` in `crates/ecstore/src/services/rebalance/migration.rs`; `decommission_tier_free_version`, `decommission_tiered_object` in `crates/ecstore/src/set_disk/mod.rs`; `crates/ecstore/src/data_movement/mod.rs` | The authority key and lock group remain stable across pools. Stage the destination, choose the location/ownership successor, then retire the exact source receipt. Do not mint independent source and destination authorities. Existing placement and tier ownership fences remain required. |
| Generic metadata entry points: `write_unique_file_info`, `update_object_meta_with_opts` in `crates/ecstore/src/set_disk/core/io_primitives.rs`; `LocalDisk::write_metadata`, `update_metadata`, `delete_version`, `delete_versions_internal`, `write_all_meta` in `crates/ecstore/src/disk/local.rs` | Consume a validated decision/recovery context or reject strict writes to enrolled objects. None may invent a generation, reset it through `fresh`, replace corrupt metadata with an unproven empty version set, or bypass durability with `no_persistence`. |
| Rollback and GC: `rename_data_owned_with_fence`, `commit_rename_data_dir`, `reclaim_orphan_data_dirs` in `crates/ecstore/src/set_disk/core/io_primitives.rs`; `reconcile_old_data_cleanup_receipts` in `crates/ecstore/src/set_disk/ops/object.rs` | Consume the owning decision and exact directory references. Cleanup does not change object contents or grant a new semantic generation. It must use a durable retirement decision and local reader/deletion guards. |
### Wire-encoding window (JSON and msgpack)
Strict capability is withheld until every raw writer in `DiskAPI`, its local implementation, `DiskStore`, remote adapters, and server handlers has an enforced path. Internal authority persistence must use its own narrow storage primitive, not evade this rule by recursively calling generic object metadata writes.
- Dual-encoded RPC fields exist twice in `crates/protos/src/node.proto`: a JSON `string` field and a msgpack `bytes *_bin` field (e.g. `file_info` and `file_info_bin` on `RenameDataRequest`). Senders emit both; receivers (`decode_msgpack_or_json` in `crates/ecstore/src/cluster/rpc/remote_disk.rs`) prefer `_bin` and fall back to JSON only when `_bin` is empty.
- `rustfs_protos::internode_rpc_msgpack_only()` drops the JSON copy only when both `RUSTFS_INTERNODE_RPC_MSGPACK_ONLY` and `RUSTFS_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED` are set after the JSON-fallback metric reads zero fleet-wide.
- Generation inside the `FileInfo` metadata map is carried in both copies automatically. Any new *top-level* generation datum must be added to both encodings and be safe under both msgpack encoders; a field in only one encoding is silently lost when a peer falls back.
- `RenameDataRequest` has a versioned, injective canonical-body encoder covering both compatibility fields; a strict generation-capable request must reject missing or mismatched canonical-body metadata rather than downgrade to the unauthenticated JSON twin.
## Reads, Garbage Collection, And Accounting
### Proto evolution
Strict reads need the chosen head, not a majority of arbitrary prepared/live UUIDs. Resolve the decision under the namespace read lock before accepting object metadata; validate the selected current or explicit version against it, and wait for or repair missing materialization. HEAD, GET, ListObjects/ListObjectVersions, scanner reads used for deletion, and prepared pool reads all need this distinction. A query may return an error while a chosen write is recovering; it must not expose an unchosen candidate or resurrect a retired version. This read-decision adapter is part of the strict-mode scope and is a reason E03 is larger than disk CAS.
No top-level proto field is required by the metadata-map UUID. If an ordered epoch or explicit expected-generation is ever added to proto, it uses **proto3 `optional`** (explicit presence). A non-optional scalar is forbidden: an old coordinator talking to a new disk decodes absence as a plausible zero.
Keep the current namespace read lock through EOF/drop, including multipart part boundaries. This design does not replace it with a new cross-node generation lease registry. A strict implementation must also bind every deferred local/remote part open to the resolved generation and acquire protection on the disk that owns the directory before handing the read capability out. A reader that loses authority or cannot renew its disk protection must fail before another open; it cannot continue on an unvalidated cached pathname.
## Mixed-Version Gate: One Direction
`LocalDisk::acquire_snapshot_lease`, `renew_snapshot_lease`, `release_snapshot_lease`, and `delete_data_dir` in `crates/ecstore/src/disk/local.rs` provide disk-local path protection and deletion deferral. They are not proof of a fleet-wide object generation. Before reuse, their token must bind disk incarnation and exact generation/directory, and strict reads must reject a pre-restart token. Existing open file descriptors may finish reading an unlinked inode, but later part opens need a valid protected generation. This preserves streaming behavior without assuming a local mutex protects another node.
When generation enforcement is not explicitly requested, or fleet confirmation is absent, behavior falls back to current semantics. Fail-closed is reserved for an explicit administrator-confirmed strict rollout.
GC consumes a durable retirement authorization for exact directories no longer referenced by **any** retained version or pending accepted decision. Include retirement in the chosen successor that removes the last reference; if it was not recorded there, choose a metadata-neutral successor that records it before deletion. That successor advances the authority revision while preserving the S3 version contents; the cleanup syscall itself never mints an identity. Retirement prevents future repairs/reads from creating new references; a new reference requires a new decision and cannot revive a retired directory. At the destructive syscall, hold the object/directory guard, recheck the local chosen metadata references, `old_dir != committed_dir`, retirement/configuration identity, and local snapshot protection. Any uncertainty defers deletion. A stale cleanup receipt matching an earlier UUID is not enough. Across executor restart, replay the same retirement ID idempotently; do not convert a lost reply into a broader recursive delete.
| Flag (`crates/config/src/constants/object.rs`) | Default | Effect |
|---|---|---|
| `RUSTFS_OBJECT_TRANSACTION_FENCING_WRITE` | false | With either flag absent, PUT/MPU neither persists nor consumes the transaction UUID. |
| `RUSTFS_OBJECT_TRANSACTION_FENCING_FLEET_CONFIRMED` | false | With both enabled, failure to obtain or retain the live fleet proof rejects the commit before rename. |
Prepared pool reads remain a separate optimization domain. In `crates/ecstore/src/store/rebalance.rs`, `prepare_latest_object_metadata_with_idx` collects candidates and revalidates a refetched winner with `validate_prepared_pool_refetch_identity` from `crates/ecstore/src/store/rebalance/support.rs`. `PreparedPoolReadFallbackBarrier` is a `#[cfg(test)]` scheduling fixture, not a production identity. Keep the all-pool resolution rules until a chosen location decision supplies equivalent evidence. The prepared bundle binds authority generation plus pool identity; mismatch requires a complete refetch or a typed failure, never selection of a different generation using the old bundle.
The fleet proof is currently borrowed from the remote-version-state writer rollout. It proves membership/process-epoch convergence for that feature only; it does not prove an epoch type, per-disk CAS support, or RPC strict-mode convergence, and must not be treated as the final generation handshake.
Quota remains a separate per-bucket arbitration domain. `QuotaLedger` and `settle` in `crates/ecstore/src/bucket/quota/reservation.rs` key reservations by operation UUID, validate object/size, and update under the ledger fence. A late settlement cannot remove a different reservation key; an absent/mismatched key fails or follows the existing idempotent abort rule. That proves key isolation, not that current generation A committed or that a ledger storage write is immune to stale-disk mutation.
### Capability negotiation (target)
For strict mode, add the decision identity to the reservation/settlement binding. Settle a committed **historical** decision even if it has since been superseded, but only against its original reservation and recorded old/new sizes; demanding that it still be current would leak valid reservations. Abort only a recovered unchosen/retired operation. Unknown outcomes stay reserved and reconcile. Retain the existing conservative usage floor and `commit_started` recovery behavior. Ledger writes themselves require the authority protocol, with a documented lock order and no recursion through their own reservation path. No quota token is compared numerically to an object ballot or used to revoke a newer object's leases.
Generation enforcement requires one **live fleet proof** containing at least: the selected authority version and comparison mode; the current membership/topology fingerprint and process epochs; support for every required disk mutation point; RPC signature/body/replay strict convergence; and the on-disk encoding version (the metadata-map UUID is version 1). Membership change or an old-node rejoin revokes the proof; revocation before commit fails an explicitly strict request and never rewrites or lowers a persisted generation. The proof may extend the authenticated fleet-proof machinery in `notification_sys` or the runtime capability contract; this document requires one shared token, not a mechanism.
## Restart, Membership, And Rollout
## Open Decisions
Authority configuration is durable and includes participant identities, routing, bucket incarnation, protocol version, and quorum rules. Changing storage pool placement must not remap the authority key. A replaced voter starts as a non-voter, catches up durable promise/accepted/chosen state, and only joins through a quorum-approved configuration transition. Configuration change requires intersecting old/new decision quorums; losing the old quorum is a recovery incident, not permission to bootstrap a new empty authority. Offline data disks rejoin through generation-aware catch-up, independent of voter admission.
Blockers for calling the contract implemented:
A live capability proof must bind the authority configuration, topology, every participant process boot epoch, disk incarnations, writer/read/recovery protocol support, encoding version, and RPC signature/body/replay strictness. Restart, membership change, disk replacement, protocol downgrade, or a strict-transport setting change revokes it. Receivers revalidate before entering the publication critical section; accepted durable decisions survive proof revocation and are recovered under a fresh valid proof, never replayed as unvalidated requests. The remote-version-state fleet proof does not prove any of these generation capabilities.
1. **Authority mode.** Total order or opaque exact-CAS. Do not retrofit ordering semantics onto the existing random UUID.
2. **Complete `xl.meta`-writer coverage.** Enumerate commit rename, rollback restore/delete, cleanup, heal, transition, restore, replication, and data movement; each path compares/carries the selected generation or is proved incapable of replacing the authoritative identity.
3. **Rollback as expected-generation CAS.** The quorum-failure rollback in `rename_data` restores backup metadata, not just a private temp file; it must run only when the stored generation still matches the failed writer's expectation.
4. **Generation capability proof.** Extend the fleet proof or the runtime capability contract; one revalidatable token.
5. **Read-lease and GC crash recovery.** Cross-node registry, TTL reclamation, lease-holder crash behavior, GC-executor recovery.
6. **Quota reserve → commit → settle binding.** Relate the ledger's independent mutation tokens to the selected generation, with a concrete late-settle rejection test, or prove the fence is a separate arbitration domain that cannot cross-settle.
7. **Prepared reads stay pool-local.** `PreparedPoolReadFallbackBarrier` validates freshness only within the pool that produced it; cross-pool ordering requires a common authority, and the multi-pool wait cannot be short-circuited without one.
8. **Hot-path cost is a blocking metric.** Measure any added consensus write, fsync, fleet-proof lookup, lease operation, or centralized serialization under 4 KiB and hot-key/hot-bucket A/B.
9. **Test infrastructure.** Multi-node, multi-pool, directed network-fault, and large-object budget for restart, mixed-version, and cross-node lease acceptance.
The existing `RUSTFS_OBJECT_TRANSACTION_FENCING_WRITE` and `RUSTFS_OBJECT_TRANSACTION_FENCING_FLEET_CONFIRMED` flags in `crates/config/src/constants/object.rs` retain their current default-off behavior. They do not become a claim that the new protocol exists. If generation strictness is explicitly selected, missing capability is an error, never silent downgrade. No new environment variable is introduced by this document; a production gate must be documented with its implementation.
Strict enrollment requires quiescing old writers and readers for the enrolled namespace, recovering ambiguous operations, validating/importing legacy heads, persisting a strict-format/protocol marker, and enabling the complete fleet. New disks reject unbound legacy mutation RPCs for that namespace. Old binaries must be prevented from opening a strict-enrolled drive by a startup compatibility gate they understand before enrollment; an environment flag known only to new binaries is insufficient. Until that prerequisite is deployed, do not activate strict mode in a mixed fleet. Disabling flags after enrollment cannot drop durable authority; downgrade requires a separately verified quiescent materialization/export operation. Ordinary un-enrolled compatibility deployments keep their current behavior.
## Encoding And Transport
- Do not bump `XL_META_VERSION` or `XL_HEADER_VERSION` in `crates/filemeta/src/filemeta.rs`. Do not add fields to positional-msgpack `FileInfo`; carry the UUID through the metadata map and protocol records through explicit versioned envelopes.
- Write RustFS/MinIO internal metadata dual keys using `crates/utils/src/http/metadata_compat.rs`. Reject conflicting, nil, or malformed generation values; validate every persisted/RPC record again at consumption.
- New proto values in `crates/protos/src/node.proto` require explicit presence (`optional` scalars or a present message), including absent/tombstone state. Bind expected/new generation, ballot, receipts, configuration, and outcome identity in the canonical body. The JSON and msgpack representations must carry identical semantics; absent data from an old peer cannot decode as a valid zero ballot.
- `crates/ecstore/src/cluster/rpc/remote_disk.rs` and `rustfs/src/storage/rpc/node_service/disk.rs` must share local protocol behavior. Extend canonical encoders for every affected mutation, not only `RenameData`. Authenticate both compatibility representations and reject disagreement rather than falling back to a weaker JSON twin.
- `crates/ecstore/src/cluster/rpc/http_auth.rs` supplies signature, canonical-body, and replay-scope checks. Strict generation capability requires fleet convergence of `RUSTFS_INTERNODE_RPC_SIGNATURE_STRICT`, `RUSTFS_INTERNODE_RPC_BODY_DIGEST_STRICT`, and `RUSTFS_INTERNODE_RPC_REPLAY_SCOPE_STRICT` from `crates/config/src/constants/internode.rs`. Cover method substitution, body tamper, stripped metadata, consumed nonce replay, and receiver restart.
- Preserve real-MinIO metadata fixture decoding and supported old-RustFS compatibility before strict enrollment. Do not promise that a live MinIO binary can start a RustFS-written drive set; [minio-file-format-compat.md](minio-file-format-compat.md) explicitly excludes that direction. Strict authority records also impose a new deployment boundary even though the `xl.meta` container version is unchanged.
## Required Failure Outcomes
Every scenario must first prove its intended barrier, quorum, or crash point was reached, then inspect authority records, decoded metadata, directory references, return/error categories, and full GET bytes. Single-process barriers are insufficient evidence for the network-partition cases.
| Interleaving | Unique permitted outcome |
|---|---|
| A pauses after coordinator verification; A loses lock quorum while data RPC remains reachable; B chooses and publishes; A resumes rename. | A cannot become the chosen successor for B's predecessor or overwrite B on a disk that applied B. Isolated older materialization cannot vote as current. Read B exactly; reconcile laggards under B's decision. |
| A's publication result is unknown; B succeeds; A's rollback/cleanup resumes. | Recover A's decision. Never restore/delete B's metadata or referenced data; retire only A-owned unchosen paths or separately authorized dead directories. |
| Reader acquires generation G, consumes part 1; replacement retires G; GC attempts to delete part 2's directory. | The reader's valid disk protection defers GC; otherwise the reader fails before its next open. Never silently serve another generation or delete a directory still covered by valid protection. Repeat across reader and disk restarts. |
| A and B each stage or partially publish on two of four disks; one coordinator dies; all disks return. | The promise/accept recovery rule preserves any chosen value or adopts the highest accepted candidate and finishes that slot. No guessing from UUID order, no permanent exact-CAS split, no replacement of a chosen value. |
| Process dies after staging sync, intent sync, data rename, metadata replace, directory sync, or accepted/decision reply. | Reopen durable records before readiness. Unchosen work stays private; chosen work is replayed forward; lost ACK resolves to the original operation identity. Torn/insufficient evidence fails closed. |
| Null version becomes a delete marker; a noncurrent version is deleted; delayed heal/metadata write resumes. | Whole-object predecessor no longer matches. Exact repair uses the chosen version set; no resurrection, marker-to-object conversion, or reset to never-present. |
| Old coordinator reaches a new strict disk, new coordinator reaches an old disk, voter restarts, or transport strictness changes. | Compatibility behavior only in an un-enrolled namespace. Strict admission fails until a fresh complete proof and supported startup gate exist; no zero/missing-field fallback. |
| Replacement disk is empty, or a restored minority disk has an old promise and old metadata. | It cannot vote as an initialized authority. Catch up chosen state and reconstruct data under a fresh incarnation; stale requests cannot bypass enrollment by presenting absent metadata. |
| Quota settlement for A arrives after B commits; prepared pool refetch sees B instead of A. | Only A's original chosen outcome may settle A's reservation; B is unaffected. Prepared A cannot supply metadata/data for B without a new validated preparation. |
## Required Implementation Boundaries
These are durable ownership and acceptance boundaries, not permission to close the disk-fencing work before the protocol exists. The decision-model and availability changes require architecture review before production implementation. No unrelated external consensus service or full-manifest rewrite is authorized by this contract.
| Boundary | Required implementation and exit evidence |
|---|---|
| Durable authority substrate | Injected lock-participant persistence; typed ballot/configuration/decision values; prepare/accept/recover state machine; restart-safe proposer identities; corruption and voter replacement handling. Model/exhaustively test two competing proposers, lost replies, minority recovery, and every durable transition. The same-slot different-value property must be impossible. |
| Disk publication boundary | Separate private preparation from publication; implement write-ahead intent, decision receipts, atomic guarded mutation, idempotent recovery, and directory retirement. Include inline/non-inline, every crash point, canceled blocking syscalls, ACK loss, and empty/lagging disks. Existing rollback helpers cannot remain an unguarded alternate route. |
| Writer and read integration | Route every writer in the table and every strict read/scan decision through the authority; preserve data quorum and S3 version semantics. Bind prepared reads, MPU ownership transfer, tier operations, and quota outcomes. Demonstrate no raw metadata entry point bypasses strict mode. |
| Fleet activation | Deploy the startup downgrade barrier first; import legacy/absent heads during quiescence; implement configuration/proof revocation and both RPC encodings. Run real multi-node lock/data-plane partitions and mixed-binary/restart tests. Only then can strict E03 acceptance run and activation be considered. |
The conservative immediate action is to keep the existing compatibility behavior and improve its local convergence/recovery independently. Those fixes must describe their smaller guarantee and must not advertise E03's distributed safety. A strict-only local CAS helper can be built behind the inactive capability boundary, but it cannot enable the feature or close the authority work.
## Performance And Activation Criteria
Measure the existing implementation and the full proposed path on identical machines, disk/filesystem, durability settings, network, object population, concurrency, and warmup. Include single hot-key and many-key 4 KiB PUT, 1 MiB PUT, metadata-only writes, and CompleteMultipartUpload with fixed part counts. Report throughput, p50/p95/p99, peak retained preparation/recovery bytes, recovery time, per-operation RPCs/fsyncs, and the object mutation critical-section duration. Include a slow minority disk, one voter loss, and restart recovery; a throughput result alone is insufficient.
The unoptimized proposal adds a lock-quorum promise round and an accept round with durable writes, plus decision learning/publication and local intent/applied-state persistence. Wdata staging remains separate. Read resolution may add an authority quorum round. Record actual overlapping rounds and fsync group commits; do not claim these costs disappear because the existing lock RPC is reused. Never hold a global lock across shard I/O, wait for all disks on the successful path, or weaken fsync/bitrot/quorum to recover throughput.
Activation requires all failure scenarios to pass with no acknowledged-data loss or wrong-generation read; no unexplained RPC/fsync amplification beyond the implemented phase budget; and an explicit performance acceptance recorded with the review. Use a conservative review trigger of more than 10% throughput loss or 15% p99 growth in any fixed-workload comparison: exceeding it blocks default activation until the architecture/operations owners accept the measured tradeoff or the implementation removes it. These are proposed rollout budgets, not measurements or performance claims. Without a reproducible baseline, leave strict mode unavailable.
@@ -34,6 +34,22 @@ The `scanner` and `heal` subsystems are served by `GetConfigKVHandler` (`rustfs/
## Test Matrix
### Deterministic regression checks
Run the scanner regressions before collecting host-pressure measurements:
```bash
cargo nextest run -p rustfs-scanner --lib
```
Most tests in `crates/scanner/tests/lifecycle_integration_test.rs` are ignored in the default lane because they require serial execution. Run the scanner portion of the `ILM Integration (serial)` selection in `.github/workflows/ci.yml` with `-j1 --run-ignored all` as well; preserve its documented exclusions for known noncurrent transition/expiry failures.
The folder regressions exercise real directory enumeration and metadata decoding. `scanner_failed_child_retains_usage_and_scans_healthy_sibling` replaces an enumerated directory before descent, so its I/O failure is reproducible without depending on Unix permission enforcement. `scanner_nested_metadata_failure_without_retry_cache_is_partial_then_recovers` checks fresh and inherited failure state with retry caching disabled, reuse of a partial compacted subtree, and recovery on the next selected directory cycle. Neither a failed subtree nor an expired retry ledger proves zero usage.
`scanner_compacted_directory_keeps_aggressive_heal_and_bitrot_sampling` covers disabled, sub-interval, and exact-interval heal divisors in normal and deep modes. `scanner_cancellation_interrupts_folder_throttle` uses a paused clock to require immediate cooperative cancellation. `scanner_blocked_expiry_preserves_usage_replication_and_integrity_work` covers lifecycle enabled/disabled with pending replication, failed replication, and Legal Hold; retained bytes and integrity/replication inspection must survive blocked expiry.
These checks complement the sampling and cancellation design in [MinIO's scanner implementation](https://github.com/minio/minio/blob/master/cmd/data-scanner.go), especially `scanDataFolder`, `folderScanner.scanFolder`, and `dynamicSleeper.Sleep`. Scanner admission counters prove that work reaches the admission boundary; they do not prove remote replication delivery or a completed shard repair. The deployment matrix below remains necessary for those claims and for measured CPU, memory, IOPS, and foreground-latency comparisons.
Collect at least two runs on the same RustFS commit and the same workload. Keep hardware, commit, object count, object size, bucket count, scanner-enabled state, and foreground workload constant between runs.
| Run | Purpose | Example scanner settings |
+3 -3
View File
@@ -70,9 +70,9 @@ These have no persistent key and are read from the environment only.
| `RUSTFS_SCANNER_ENABLED` (deprecated alias `RUSTFS_ENABLE_SCANNER`) | `true` (`scanner_enabled_from_env`, `rustfs/src/module_switches.rs`) | Starts the data scanner at all. The heal manager is initialized whenever heal or scanner is enabled, because scanner-produced heal candidates need a consumer. |
| `RUSTFS_SCANNER_ALERT_COOLDOWN_SECS` | `86400` (`DEFAULT_SCANNER_ALERT_COOLDOWN_SECS`, `scanner_folder.rs`) | Per-(kind, bucket, object) cooldown between S3 excess-alert events; `0` emits every cycle. See [Scanner Excess Alerts](scanner-excess-alerts.md). |
| `RUSTFS_SCANNER_DEEP_VERIFY_COOLDOWN_SECS` | `60` (`DEFAULT_SCANNER_DEEP_VERIFY_COOLDOWN_SECS`, `scanner_folder.rs`) | Objects modified within this window are skipped by deep (bitrot) verification in the current cycle. |
| `RUSTFS_HEAL_OBJECT_SELECT_PROB` | `1024` (`DEFAULT_HEAL_OBJECT_SELECT_PROB`, `scanner_folder.rs`) | Sampling divisor for scanner-originated heal checks: roughly one object in N per cycle is selected for a low-priority heal check. |
| `RUSTFS_DATA_USAGE_UPDATE_DIR_CYCLES` | `16` (`DATA_USAGE_UPDATE_DIR_CYCLES`, `scanner_folder.rs`) | Every N cycles a compacted directory is re-descended instead of reusing its cached usage. `1` forces re-descent every cycle (used by lifecycle e2e lanes). |
| `RUSTFS_DATA_USAGE_FAILED_OBJECT_TTL_SECS` | `86400` (`DEFAULT_FAILED_OBJECT_TTL_SECS`, `scanner_folder.rs`) | Retention of per-bucket failed-object entries in the usage cache. |
| `RUSTFS_HEAL_OBJECT_SELECT_PROB` | `1024` (`DEFAULT_HEAL_OBJECT_SELECT_PROB`, `scanner_folder.rs`) | Sampling divisor for scanner-originated heal checks: roughly one object in N per cycle is selected for a low-priority heal check. `0` disables sampled checks. When N is smaller than the compacted-directory interval, every object in a selected directory is eligible; compaction must not round the sampling probability to zero. |
| `RUSTFS_DATA_USAGE_UPDATE_DIR_CYCLES` | `16` (`DATA_USAGE_UPDATE_DIR_CYCLES`, `scanner_folder.rs`) | Every N cycles a compacted directory is re-descended instead of reusing its cached usage. `1` forces re-descent every cycle (used by lifecycle e2e lanes); `0` is normalized to `1`. |
| `RUSTFS_DATA_USAGE_FAILED_OBJECT_TTL_SECS` | `86400` (`DEFAULT_FAILED_OBJECT_TTL_SECS`, `scanner_folder.rs`) | Retention of per-bucket failed-object retry entries in the usage cache. `0` disables and clears the retry cache; it does not allow failed scans to publish complete usage. Cached failures remain visible in each partial snapshot without extending their retry deadline. |
| `RUSTFS_DATA_USAGE_FAILED_OBJECTS_MAX` | `10000` (`DEFAULT_FAILED_OBJECTS_MAX`, `scanner_folder.rs`) | Cap on retained failed-object entries per bucket. |
### Cycle budgets and cadence
+61
View File
@@ -16,6 +16,7 @@
| Persisted free-version scan and re-enqueue after local-first expiry | `crates/ecstore/src/bucket/lifecycle/tier_free_version_recovery.rs` |
| Fenced free-version remote delete, local-marker cleanup, and rescan | `crates/ecstore/src/bucket/lifecycle/bucket_lifecycle_ops.rs` (`cleanup_free_version_exact`) |
| Durable manual transition job/task/result records | `crates/ecstore/src/bucket/lifecycle/manual_transition_job.rs` |
| Dormant tier validation probe intent format and read-only core inspection | `crates/ecstore/src/services/tier/tier_probe_intent.rs` |
| Manual run/status/cancel and transition-transaction reconcile admin routes | `rustfs/src/admin/handlers/ilm_transition.rs` |
| `ObjectInfo` / `TransitionedObject` types | `crates/ecstore/src/object_api/types.rs` |
| `FileMeta` / `FileInfo` / version metadata | `crates/filemeta/src/` |
@@ -119,6 +120,12 @@ rc admin ilm transition run local/mybucket --prefix logs/ --tier cold --dry-run
rc admin ilm transition run local/mybucket --prefix logs/ --tier cold --max-objects 1000 --max-duration-seconds 30
```
## Validation probe crash recovery status
Tier Add, Edit, and Verify currently validate a destination with a unique `rustfs-tier-probe-<uuid>` object and perform bounded compensation while the process remains alive. The `rustfs-tier-probe-intent-v1` decoder, canonical durable namespace, conditional storage primitives, state machine, and crate-level inspection type are present only as a dormant foundation. No validation path writes this record, no startup or periodic recovery scans it, and no admin HTTP route exposes it. V1 requires the owner to remain exactly equal to the immutable creator; takeover would require a new schema with explicit proof. Both durable writing and destructive recovery remain disabled until the fleet capability, operation-generation revalidation, provider timeout, retention, and operator contracts are approved.
Do not search the internal metadata bucket for these records as evidence that validation is crash recoverable: a current server does not create them. If a process is killed after the remote probe PUT but before cleanup, inspect the destination provider manually and retain ambiguous candidates. Never delete an empty or guessed version, and do not hand-create a probe intent to authorize cleanup.
Inspect the aggregate counters before widening scope. Full object-key lists are intentionally not returned. If `RUSTFS_RPC_SECRET` or other credentials were pasted into an issue, chat, log, or ticket while debugging tiering, rotate them on every node, restart the cluster with the new value, and redact the exposed copy before sharing more diagnostics.
## Reconcile an unknown transition upload
@@ -154,6 +161,60 @@ Historical transition transactions in `upload_outcome_unknown` state can use an
`finalize_missing` re-runs the provider probe and fails closed for `unversioned_present`, `versioned_present`, `ambiguous`, `unsupported`, or probe errors. It never accepts an operator assertion in place of a live `missing` result. Providers without an authoritative probe or exact version deletion remain pending; the endpoint does not infer provider capabilities, accept external absence assertions, or select a candidate automatically.
## Inspect and disposition retained recovery records
This section describes an **approved target that is not implemented yet**. Current servers do not expose the routes below and continue to quarantine tier-delete journal v1/v2 records. Do not remove internal metadata objects by hand: that loses ETag, all-pool, decommission, export, and audit guarantees.
The approved read-only inventory is bounded and paginated:
```text
GET /rustfs/admin/v3/ilm/recovery/records?protocol=<protocol>&classification=<classification>&limit=<n>&marker=<opaque>
GET /rustfs/admin/v3/ilm/recovery/records/<control-id>
```
List and redacted inspect require `admin:ListTier`. The server reconstructs the canonical source identity, strongly reads every authoritative copy, and reports one logical record with its schema, classification (`retrying`, `retained_ambiguous`, `corrupt`, `operator_required`, `abandoned`, or `terminal`), stable reason code, copy/content digests, retry deadline/counters, fleet readiness, scan completeness, and decommission coverage. It does not return raw legacy bytes, object/version names, endpoints, credentials, or provider error text in the default JSON. Incomplete pool coverage, divergent copies, a missing ETag, corruption, or a truncated page without a continuation marker is fail-closed and cannot produce an actionable receipt.
Inspect returns a 15-minute opaque observation receipt. It binds the authenticated actor, canonical record, every source copy/ETag/digest, topology/fleet generation, requested action class, issue/expiry time, and nonce. The receipt prevents a stale request from widening its target; it is not cleanup authority.
For a strictly decoded v1/v2 tier-delete journal, the approved evidence-preserving flow is:
1. Inspect the exact record and independently decide whether retaining the local cleanup obligation is still useful.
2. With `admin:SetTier`, create an immutable server-side export from the current observation receipt. The export contains the exact raw journal bytes and copy manifest, is installed create-only at the canonical digest-derived export ID, strongly read back, and downloaded through a no-store attachment response:
```text
POST /rustfs/admin/v3/ilm/recovery/records/<control-id>
{ "action": "export", "observation_receipt": "<opaque>" }
GET /rustfs/admin/v3/ilm/recovery/exports/<export-id>
```
3. Only after preserving that export, submit a fresh exact disposition with `admin:SetTier`:
```json
POST /rustfs/admin/v3/ilm/recovery/records/<control-id>
{
"action": "abandon_remote_cleanup",
"confirm": true,
"acknowledge_remote_cleanup_abandoned": true,
"observation_receipt": "<opaque>",
"export_id": "<export-id>",
"export_sha256": "<sha256>",
"reason_code": "<bounded-operator-reason>"
}
```
The last action removes only the exact local v1/v2 journal generations by per-copy `If-Match` after a durable `Prepared` disposition receipt and fresh all-member capability proof. The receipt advances `Prepared -> Applying -> Completed` and records a monotonic per-copy `confirmed_absent` set. If the server deletes copy A and crashes before recording progress, recovery may confirm A absent under the unchanged source/control, topology, process-epoch, migration, and decommission proofs, persist that progress, and continue with still-exact copy B. A replacement ETag is always a conflict; recovery never widens the immutable copy manifest.
The action never creates a tier client, probes a backend, or issues remote PUT/GET/DELETE. Its meaning is deliberately narrow: the operator accepts that remote storage may leak and abandons RustFS cleanup after preserving evidence. A changed copy, active decommission, missing member, topology/process restart, incomplete read, or uncertain replacement proof retains the evidence. Success requires every bound copy be durably confirmed absent, a fresh all-member/decommission proof, and the disposition receipt durably `Completed`; response loss resumes only the same canonical operation ID.
Canonical replay of an identical export/disposition consumes no new quota. New operations require a complete artifact inventory and are refused before source mutation when the projected retained total, including the fully encoded candidate, would exceed 10,000 exports, 10,000 disposition receipts, 1 GiB of encoded export data, or 256 MiB of encoded control/disposition data. The quota decision, create-only installation, and exact readback share one cluster-scoped admission WRITE lock. That lock is always acquired before control/source/disposition and physical metadata locks and is released before disposition `Applying` or any source deletion; callers never acquire it while holding those inner guards. A crash before installation consumes no capacity, and lost installation response is resolved by canonical readback under the same serialized order, so concurrent nodes cannot oversubscribe a stale snapshot. Admission is also limited to ten new creations per actor per minute, 100 cluster-wide per minute, 32 concurrent exports, and eight concurrent dispositions. Capacity pressure never evicts recovery evidence or blocks ordinary object I/O; the collector examines at most 100 terminal artifacts per minute.
Malformed/unsupported records and journal v3-v6 cannot use abandon. Known-version and v6 manifest ownership must converge through their normal exact recovery protocol. Operators may inspect, export, and request a bounded retry, but cannot bypass source/free-version proof, manifest membership, topology, or version semantics.
Automatic retry state survives restart. Retryable transport/quorum failures use a 60-second exponential base capped at one hour and a deterministic 80-to-100-percent multiplier, so jitter never increases the capped delay. After 32 consecutive failures or seven days from the first persisted failure, automatic work stops at `operator_required`. Unsupported or ambiguous evidence goes directly to `retained_ambiguous`/`operator_required`; age alone never deletes it. Resolved controls, immutable exports, and completed disposition receipts have minimum 30-day, 90-day, and 365-day retention respectively, and are collected only after exact source absence, decommission, successor, and audit checks.
The full schema, lease, mixed-version, retry, privacy, and metric requirements are in [../architecture/ilm-tiering-persistence-contracts.md](../architecture/ilm-tiering-persistence-contracts.md#bounded-recovery-control-and-operator-disposition).
## Reconcile legacy transition-version metadata
This section describes an **approved target that is not implemented yet**. The current server has no admin route that backfills a missing `transitioned-version-state` in `xl.meta`. Do not use the transaction reconcile route above for this purpose: that route owns an upload transaction candidate and may delete it, while legacy metadata reconciliation is non-destructive and may update only the exact local metadata version.