gsadmin/rustfs
1
0
Fork 0
mirror of https://github.com/rustfs/rustfs.git synced 2026-09-07 12:35:54 +00:00
Code Issues Packages Projects Releases Wiki Activity

perf(ilm): reduce transition transaction mutations (#7320)

Browse Source 
* perf(ilm): reduce transition transaction mutations

* test(ilm): rename transition kill points

---------

Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
This commit is contained in:
cxymds
2026-09-07 10:40:57 +08:00
committed by GitHub
parent d633a635ec
commit 6018dd372f
11 changed files with 1135 additions and 128 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/architecture/ilm-tiering-persistence-contracts.md
+16 -6
View File
@@ -41,7 +41,7 @@ All keys below are objects in the internal metadata bucket. The table gives the
| Protocol | Current schema/version | Canonical key | Creator and cleanup owner | Authoritative identity and mutable fields | Current durability point |
|---|---|---|---|---|---|
| Transition transaction | `rustfs-transition-transaction-v1`; successor v2 is approved below but not implemented | `ilm/transition-transactions/records/<aa>/<bb>/<transaction-id>.json` | The transition attempt creates it; transition commit/recovery cleans it | Immutable/fence identity: deployment, transaction, fixed v1 `owner_epoch`, write, source identity, tier/backend fingerprint, canonical remote object, deadline. Mutable: state, remote version, revision. `TransitionCleanupProof` is only a transient admission input to `mark_cleanup_pending`; it is not persisted in the record | Create-only maximum-parity write; exact record and ETag read before successor `If-Match`; terminal receipt followed by exact ETag conditional delete |
| Transition transaction | `rustfs-transition-transaction-v1`; the compact v1 state profile is fleet-gated; successor v2 is approved below but not implemented | `ilm/transition-transactions/records/<aa>/<bb>/<transaction-id>.json` | The transition attempt creates it; transition commit/recovery cleans it | Immutable/fence identity: deployment, transaction, fixed v1 `owner_epoch`, write, source identity, tier/backend fingerprint, canonical remote object, deadline. Mutable: state, remote version, revision. `TransitionCleanupProof` is only a transient admission input to `mark_cleanup_pending`; it is not persisted in the record | Create-only maximum-parity write; exact record and ETag read before successor `If-Match`; terminal receipt followed by exact ETag conditional delete. A compact success uses two saves and one delete instead of five saves and one delete |
| Tier mutation peer intent | `rustfs-tier-mutation-intent-v1` | `tier/mutation-intents/records/<aa>/<bb>/<mutation-id>.json` | The receiving peer creates and converges it; the mutation recovery path cleans it | Immutable: mutation ID/kind, old config ETag, candidate digest, sorted affected target identities, expiry. Mutable: revision, state, committed config ETag | Create with `If-None-Match: *`; transition/delete with ETag `If-Match`; maximum parity |
| Tier mutation coordinator intent | `rustfs-tier-mutation-intent-v1` | `tier/mutation-intents/coordinators/<aa>/<bb>/<mutation-id>.json` | The initiating node creates it; coordinator recovery cleans it after peer convergence | Same mutation identity and mutable fields as the peer record | Same conditional-write contract as the peer intent |
| Tier validation probe intent | Dormant `rustfs-tier-probe-intent-v1`; no writer or recovery is enabled | `ilm/tier-probe-intents/records/<aa>/<bb>/<probe-id>.json` | No current runtime owner because no path creates the record; v1 permits only the immutable creator as owner | Immutable probe, operation-generation, destination, random remote object, creator identity, and v1 owner fence. Mutable: revision, state, and monotonic remote-version proof | Conditional create/CAS/delete primitives exist but are not called by Add/Edit/Verify or recovery |
@@ -109,7 +109,7 @@ Tier mutation backend validation is outside both exclusive guards and is bound t
### Current contract
`TransitionTransaction` binds the canonical candidate name to a transaction UUID, write UUID, source identity, tier name, backend fingerprint, remote-version state, deadline, fixed `owner_epoch`, and mutable `revision`. The state model permits these ordinary edges:
`TransitionTransaction` binds the canonical candidate name to a transaction UUID, write UUID, source identity, tier name, backend fingerprint, remote-version state, deadline, fixed `owner_epoch`, and mutable `revision`. Without a current homogeneous compaction capability proof, writers use the legacy state profile:
```text
UploadStarted -> Uploaded -> LocalCommitStarted -> Committed
@@ -117,6 +117,16 @@ UploadStarted -> Uploaded -> LocalCommitStarted -> Committed
\-> AbortedNoRemote
```
When every current topology member answers the exact `transition_transaction_compaction_v1` capability challenge, a writer may hold that generation's non-cloneable proof permit and emit the compact v1 profile:
```text
UploadOutcomeUnknown@1 -> LocalCommitStarted@2 -> conditional record delete
```
The create-only `UploadOutcomeUnknown@1` record is durable before the remote PUT. After PUT succeeds, the writer revalidates the source, Object Lock decision, tier generation, and fleet proof before one exact successor CAS to `LocalCommitStarted@2`; that successor carries the known remote version. After the local metadata commit, the writer conditionally deletes that exact record rather than persisting a redundant `Committed` generation. A crash before the PUT leaves an unknown record whose provider probe can prove absence. A crash after PUT leaves the canonical candidate under the unknown record. A crash after the commit fence leaves the exact remote tuple under `LocalCommitStarted`, and a crash after local commit lets recovery prove ownership transfer and remove only the record.
The `UploadOutcomeUnknown@1 -> LocalCommitStarted@2` pair is the only compact direct edge. Legacy `UploadOutcomeUnknown@2 -> LocalCommitStarted@3` remains invalid, while historical receipt jumps keep their separately enumerated revision distances. This distinction lets existing v1 payload readers decode compact records without treating arbitrary skipped history as valid. If any peer is absent, old, unreachable, restarted with a different process epoch, or the topology changes, proof publication/revalidation fails closed and new attempts use the legacy profile. Planned downgrade first disables compact admission and drains live compact records; an older runtime reader retains or safely reconciles the v1 states, but an older decommission checkpoint validator can reject the compact successor and block pool completion.
Separately, `mark_cleanup_pending` permits proof-checked model edges from `Uploaded`, `UploadOutcomeUnknown`, and `LocalCommitStarted`. Current production recovery emits `CleanupPending` after an expired `Uploaded` record wins the exact successor CAS, or when an expired `UploadOutcomeUnknown` probe returns `UnversionedPresent` or `VersionedPresent` with a non-nil identifier. `LocalCommitStarted` mismatch or missing-source recovery retains the record; that cleanup edge is currently exercised through the state-machine API and tests, not produced by runtime recovery. States that require a remote delete still require a known `TransitionRemoteVersion` kind. A probed versioned candidate whose identifier parses as a nil UUID is retained and never authorizes remote deletion.
The remote candidate itself is named by `canonical_transition_remote_object` under `ilm/transition-transactions/<bucket-hash>/<transaction shards>/<transaction-id>/<write-id>`. That deterministic identity is what a provider probe or exact cleanup must bind; it is distinct from the internal transaction-record key.
@@ -638,17 +648,17 @@ The matrix below is the normative approved target, not a blanket description of
| Crash after local transition commit | The exact logical `xl.meta` reference, full recorded source identity (version ID, data directory, modification time, size, and ETag), and remote tuple prove ownership transfer; cleanup only the terminal transaction record |
| Crash after remote DELETE but before journal/free-version cleanup | Retry the same exact idempotent DELETE under the same fences, then conditionally clean local evidence |
| Cancellation | Stop issuing new work, persist monotonic cancellation where the protocol has it, and leave ambiguous durable records for recovery. Cancellation is never rollback proof after authorization |
| Rolling upgrade | Gate writers on the minimum capability required by the format. Known older journal/RPC versions follow their explicit compatibility rule; unknown formats are retained |
| Downgrade | Drain v6 journals and any enabled transition-v2/control protocol before removing their capable workers. Do not write a new format until its downgrade reader behavior and writer gate are specified |
| Rolling upgrade | Gate writers on the minimum capability required by the format or state profile. Transition compaction falls back to the legacy v1 profile until every current topology member answers the exact capability challenge. Known older journal/RPC versions follow their explicit compatibility rule; unknown formats are retained |
| Downgrade | Disable transition compaction admission and drain compact v1 records before removing capable checkpoint validators. Drain v6 journals and any enabled transition-v2/control protocol before removing their capable workers. Do not write a new format until its downgrade reader behavior and writer gate are specified |
| Corrupt or unknown input | Record a diagnosable failure, retain bytes, and block destructive action/completion |
Transition transaction v1, manual job/task/result v1, and receipt v2 do not currently have an implemented persisted-format negotiation for rolling downgrade. The approved transition-v2/control gate above is not current behavior. Until the applicable gate is implemented, caller/operator orchestration must not enable writers whose records required recovery nodes cannot decode. The manual async endpoint does not enforce that fleet gate and a direct request proceeds to job creation. This caller-side fail-closed rule is stricter than treating an unknown record as absent.
The compact transition-transaction v1 state profile has an implemented live homogeneous-fleet gate, but transition v2, manual job/task/result v1, and receipt v2 do not currently have a complete persisted-format negotiation for rolling downgrade. The approved transition-v2/control gate above is not current behavior. Until the applicable gate is implemented, caller/operator orchestration must not enable writers whose records required recovery nodes cannot decode. The manual async endpoint does not enforce that fleet gate and a direct request proceeds to job creation. This caller-side fail-closed rule is stricter than treating an unknown record as absent.
### Current format compatibility decisions
| Family/version | Current reader and writer behavior | Upgrade, downgrade, and ignore rule |
|---|---|---|
| Transition transaction v1 | Writers emit v1; the payload decoder rejects another schema, bad checksum, unknown state, or inconsistent transaction/remote identity. The current record-path parser accepts any shard/extra-component layout and uppercase hex when the final 32-hex UUID parses and matches the payload | There is no intentional ignore path, but exact lowercase canonical-path rejection remains an approved fix. V1 remains the only writer format until the approved v2 fleet gate is implemented; a v2 reader never rewrites an active v1 record |
| Transition transaction v1 | Writers emit v1. A homogeneous live capability permits the compact `UploadOutcomeUnknown@1 -> LocalCommitStarted@2` profile; otherwise writers retain the legacy profile. The payload decoder rejects another schema, bad checksum, unknown state, or inconsistent transaction/remote identity. The current record-path parser accepts any shard/extra-component layout and uppercase hex when the final 32-hex UUID parses and matches the payload | There is no intentional ignore path, but exact lowercase canonical-path rejection remains an approved fix. During rolling upgrade, an unsupported or unavailable peer forces legacy writes. Before downgrade, disable compact admission and drain compact records because older decommission validators may conservatively reject the direct successor. V1 remains the only writer format until the approved v2 fleet gate is implemented; a v2 reader never rewrites an active v1 record |
| Transition transaction v2 and recovery-control/export/disposition v1 | Approved target only; no current reader or writer emits these formats | Roll out read support before the homogeneous writer gate; old readers reject and retain. Disable creation and prove all active records drained before downgrade; never rewrite v2 to v1 |
| Tier mutation intent v1; peer RPC v3/v4 | Durable readers/writers require intent v1. New peers accept signed/canonical v3 and v4 RPC; old v3 peers return an exact authenticated unsupported response to v4 | Pause and drain edit/remove/clear across the mixed interval; do not automatically retry v4 as v3. Unknown durable intent is retained and blocks recovery |
| Manual job/scope/task/result v1 | Writers emit the v1 family. Manual-job runtime recovery accepts an uppercase UUID path when both shard strings match its uppercase prefix, then loads the lowercase canonical job by UUID; the decommission validator recomputes the canonical path and rejects that alias. Other decoder/path/checksum failures stop reconciliation. Runtime capabilities advertise `enqueue_only` and `async`, but the async run handler does not consult a fleet capability gate and a direct request creates a job | Runtime recovery still needs exact lowercase canonical-path validation to prevent alias-driven duplicate work. Caller/operator orchestration must verify every required node and fail closed when capability is unknown or unsupported. An automatic server-side fleet gate and persisted downgrade negotiation remain open; unknown records are never ignored as completed work |