fix(odm): preserve cursor compatibility and native source semantics (#7238)

docs/operations/on-demand-migration.md

@@ -23,13 +23,25 @@ Both builds can read, redact and preserve GCS configuration. A build without `gc
 
 ## List continuation token rollout
 
-`RUSTFS_ON_DEMAND_MIGRATION_LIST_V2_TOKENS` defaults to `false`; unset or invalid boolean values also keep it off. It controls only whether a v1 listing may first issue a v2 continuation token after an empty truncated merged page. Every node with this reader support accepts existing v2 tokens and continues their budget even with the switch off. Ordinary pages that consume an object or common prefix retain the original v1 token shape.
+The two rollout switches have different defaults. Unset or invalid boolean values use the stated default. Both are node environment variables, not bucket settings:
 
-Leave the switch off while deploying v2 reader support to every node that can receive a continuation request, including nodes behind other load-balancer routes. Then set it to `true` in each node's environment and restart those nodes to enable issuance. A v1-only binary rejects v2 with `400 InvalidArgument` before the source-error policy runs; neither `not_found` nor turning off list-through makes that old reader compatible. With issuance still off, a new v1 chain retains the existing limitation: an empty source cursor cycle spanning requests can continue indefinitely. The default rollout does not claim to fix that chain until issuance is enabled.
+- `RUSTFS_ON_DEMAND_MIGRATION_LIST_V2_TOKENS` defaults to `false` and allows a v1 listing to first issue a v2 token after an empty truncated merged page. Existing v2 tokens keep their budget even on reader-only nodes. Consuming an object/common prefix or reaching a new EOF resets the budget to v1 without changing the chain's framing.
+- `RUSTFS_ON_DEMAND_MIGRATION_LIST_FRAMED_TOKENS` defaults to `true`, preserving the framed output of the #7187 / `e1608fbd9` generation. It allows a bare/new merged listing to first issue a NUL-prefixed JSON envelope inside the existing base64 encoding. Existing framed chains stay framed even with this switch off, including a reset to v1 and local continuation after list-through is disabled. With framing issuance off, new bare v1 output keeps its historical bytes; ordinary local listings remain unchanged. This switch does not enable the v2 budget.
 
-An active v2 budget rejects the sixteenth consecutive merged page that consumes no new object/common prefix and reaches no new end-of-list state. The first fifteen empty pages can be resumed; with the existing two-fetch-per-side limit, that interval costs at most 32 fetches per side, including the failing request. A key, common prefix, or a newly exhausted side on the sixteenth request succeeds and resets the budget. A side that was already exhausted does not reset it again. This is a resource bound, not proof of a cursor cycle: an unusually long but valid empty source-page chain also reaches the limit. Tokens are unsigned base64 JSON, so this budget applies to clients that continue with the returned token unchanged; replaying or editing a token can reset it, and it is not a malicious-client defense or a global request quota. The two-fetch-per-side request limit and existing source rate limiter still apply. A source failure follows `policy.source_error`: `propagate` returns `424 SourceUnavailable` with `invalid_pagination`; `not_found` returns the fetched local listing with `x-rustfs-on-demand-migration-list: local_only`. A blocking local-side failure returns `InternalError`, without silently discarding local entries.
+Choose the upgrade path from the binaries currently serving LIST requests, including every load-balancer route. This build reads complete historical bare envelopes and framed v1/v2 envelopes with the same strict version/count validation. There is no single writer format understood by both bare-only and framed-only readers. A bare-v1-only binary rejects bare v2 with `400 InvalidArgument`; a bare-only reader mistakes framed input for a local marker, while a framed-only reader mistakes bare input for one. These framing mismatches can restart a merged scan and lose its budget without returning an error. For example, with local keys `b,d` and source keys `a,c`, a new node issuing bare after `a` followed by an `e1608fbd9` reader can return `a` again. That old reader then emits framing, so the symptom need not be an infinite loop.
 
-For rollback, first turn issuance off on every node. Keep v2-capable readers available for outstanding v2 chains: switching issuance off does not erase their budgets, and tokens have no expiration that proves those chains have drained. Route those continuations to compatible readers or have clients explicitly restart their listings before restoring v1-only binaries. Restarting a listing is a new scan and can repeat entries. Do not roll back readers while assuming the issuance switch makes existing v2 tokens disappear.
+- **Upgrading from the framed-only #7187 / `e1608fbd9` generation:** before deploying the first new node, set `RUSTFS_ON_DEMAND_MIGRATION_LIST_FRAMED_TOKENS=true` in the new nodes' deployment environment, or leave it unset to use this build's `true` default. Remove any previous explicit `false` override. Existing framed-only nodes ignore this new variable and already emit framing. Keep framing enabled while any of those nodes serves continuations. While list-through remains active, new and existing framed v1/v2 chains then retain their cursors and any active budget in both directions. Only after all serving nodes are dual readers may you choose `false`; existing framed chains still stay framed, while newly started bare chains must stay on dual readers.
+- **Upgrading from older, pre-#7187 bare-only binaries:** explicitly set `RUSTFS_ON_DEMAND_MIGRATION_LIST_FRAMED_TOKENS=false` on every new node **before its first start**. Keep it false until all serving readers accept both formats; this preserves bare v1 bytes and active bare cursors during that rollout. Keep `RUSTFS_ON_DEMAND_MIGRATION_LIST_V2_TOKENS=false` until every reader also supports v2. After reader convergence, you may enable framing and then the v2 budget; enabling framing also frames the next nonzero merged continuation of an existing bare chain, so do not do this while bare-only readers remain.
+
+Restart nodes after changing their environment. Do not directly mix bare-only and framed-only binaries on the same continuation routes. Existing bare tokens must be routed to dual readers while any framed-only nodes remain. The v2 issuance switch is independent: keep it off until all readers support v2, but turning it off never removes an existing v2 budget.
+
+Partial JSON-shaped object keys remain local markers. To retain already issued cursors, a bare JSON object with the ODM tag and every historical writer field (`v`, `local`, `local_done`, `source`, `source_done`, `last_key`) is treated as an envelope, then strictly validated. A valid object key can be identical to that complete envelope: the two byte strings are indistinguishable, so legacy compatibility necessarily gives the envelope interpretation precedence. Framing identifies new merged tokens unambiguously, but dual-format readers do not eliminate this old full-envelope key collision. There is no signature, session store, or automatic format negotiation.
+
+An active v2 budget rejects the sixteenth consecutive merged page that consumes no new object/common prefix and reaches no new end-of-list state. The first fifteen empty pages can be resumed; with the existing two-fetch-per-side limit, that interval costs at most 32 fetches per side, including the failing request. A key, common prefix, or a newly exhausted side on the sixteenth request succeeds and resets the budget. A side that was already exhausted does not reset it again. A zero-sized request does not spend an existing budget. This is a resource bound, not proof of a cursor cycle: an unusually long but valid empty source-page chain also reaches the limit. Tokens are unsigned base64-encoded JSON, optionally framed, so this budget applies to clients that continue with the returned token unchanged; replaying or editing a token can reset it, and it is not a malicious-client defense or a global request quota. The two-fetch-per-side request limit and existing source rate limiter still apply. A source failure follows `policy.source_error`: `propagate` returns `424 SourceUnavailable` with `invalid_pagination`; `not_found` returns the fetched local listing with `x-rustfs-on-demand-migration-list: local_only`. A blocking local-side failure returns `InternalError`, without silently discarding local entries.
+
+With budget issuance off, a new v1 chain retains the existing limitation: an empty source cursor cycle spanning requests can continue indefinitely. Default rollout does not fix that chain until the v2 switch is enabled. Framing alone does not impose the budget.
+
+For rollback, turn v2 issuance off, but choose framing for the readers being restored. When returning to the framed-only generation, keep framing `true` and route any outstanding bare tokens only to dual readers. Before restoring bare-only readers, set framing `false` on the remaining dual readers and deal with all outstanding framed tokens; v1-only readers also cannot resume v2 tokens. Neither switch rewrites existing framed or v2 chains, and tokens have no expiration that proves they have drained. Retain compatible readers for those continuations or have clients explicitly restart their listings before restoring incompatible binaries. Restarting a listing is a new scan and can repeat entries. Switching issuance off alone does not make outstanding tokens safe for older readers.
 
 ## Positioning
 
@@ -188,9 +200,9 @@ No write, delete, ACL or versioning permission is required or used. Scope the po
 
 Behaviour a client can observe. The "Test" column names the case that pins it: `*_test.rs` files live under `crates/e2e_test/src/on_demand_migration/`, and the unit tests live next to the code in `rustfs/src/app/object/get.rs`, `head.rs` and `shared.rs`.
 
-ODM merged continuation tokens use a NUL-prefixed JSON envelope inside the existing base64 encoding. NUL is not valid in a local object key, so a legitimate JSON-shaped key can never be mistaken for a merged cursor. Upgrade every node before using list-through, and restart any in-progress ODM listing issued by an older build: its unframed JSON tokens cannot be distinguished from legitimate local keys. Ordinary local listing tokens remain unchanged. Tokens issued by this build can still resume the local side after list-through is disabled.
+ODM merged continuation tokens use bare or NUL-prefixed JSON inside the existing base64 encoding. The default writer preserves framed output; an explicit `RUSTFS_ON_DEMAND_MIGRATION_LIST_FRAMED_TOKENS=false` keeps historical bare output during older-reader rollouts. Compatible readers accept both formats and retain existing budgets. See [List continuation token rollout](#list-continuation-token-rollout) for the independent issuance switches, rolling-upgrade requirements, and the unavoidable ambiguity between a complete historical envelope and an identically named local key.
 
-Source `HEAD` responses with status 404 require a successful bucket probe before being negative-cached. The source credential therefore needs permission for `HeadBucket` (S3 `ListBucket`); a prefix-restricted ListBucket policy can deny that probe, in which case the response is a source failure rather than a cached miss. A missing/inaccessible source bucket, a missing source version, or an ambiguous GET 404 is not proof that the requested key is absent. Conditional GET validators are checked against the actual source GET metadata as well as the advisory HEAD; a missing required validator fails with 424. Source LIST entries without a key or a non-negative size fail the page rather than fabricating an empty object.
+Source `HEAD` responses with status 404 require a successful bucket probe before being negative-cached. The source credential therefore needs permission for `HeadBucket` (S3 `ListBucket`); a prefix-restricted ListBucket policy can deny that probe, in which case the response is a source failure rather than a cached miss. A missing/inaccessible source bucket or a missing source version is not proof that the requested key is absent. Native GCS verifies the bucket after either HEAD or GET returns 404 and preserves a failed probe as a source error. Azure accepts explicit `BlobNotFound` only on an unversioned object read with status 404; an ambiguous HEAD may make one container probe, while an ambiguous GET remains a source error. Native probes add at most one request and retain the existing per-request timeouts, rather than a single deadline for the pair. Conditional GET validators are checked against the actual source GET metadata as well as the advisory HEAD; a missing required validator fails with 424. Source LIST entries without a key or a non-negative size fail the page rather than fabricating an empty object.
 
 Write-back currently requires namespace locking enabled and exactly one pool with one erasure set. Other topologies fail write-back explicitly as `unsupported`: source reads remain available, but backfill cannot complete successfully or certify cutover. This restriction avoids relying on a set-local condition across distinct pool or lock domains; it does not restrict ordinary S3 writes. Full cross-pool migration requires a globally fenced commit protocol.