buckit/docs/single-trip-get-design.md

Single-Trip FastOpen GET - Design Note

Status: Implemented in this branch

Summary: Buckit's final FastOpen design does not add current/ shadow directories, versions/<id>/ direct paths, or write-side shadow repair. Instead, it adds a read-side storage protocol that lets each disk answer a GET with a single streamed response:

1. read that disk's xl.meta;
2. encode the chosen object version into a compact FastOpen frame;
3. immediately continue with the shard body bytes from the same stream when the object is streamable.

The landing node opens FastOpen streams on a first wave of disks, reads only the frame headers to establish the winning object/version/layout, then reuses the already-open body streams for decode. If the first wave is not sufficient, it opens additional disks or falls back to canonical GET. xl.meta remains canonical; FastOpen is an opportunistic read optimization only.

This note describes the shipped FastOpen design. It supersedes earlier proposals based on current/part.1, versions/<versionId>/, and write-side shadow copies.

Read request-flow.md first; this note assumes its vocabulary (xl.meta, erasure sets, parts, FileInfo, quorum, distribution, local vs remote disk access).

---

1. Goal

Canonical GET pays two logical storage phases:

1. read xl.meta enough times to establish the visible version and layout;
2. open shard data files and decode from the selected readers.

FastOpen keeps the same correctness rules but collapses those into one per-disk storage request. Each participating disk returns metadata and, when possible, body bytes from the same open stream.

This is a GET-only optimization. It does not change the canonical write format, object version layout, or healing authority model.

---

2. High-level architecture

2.1 Canonical state stays unchanged

FastOpen does not introduce a second on-disk object namespace.

• xl.meta remains the source of truth for visible versions, delete markers, transition state, erasure layout, and object metadata.
• The ordinary shard files under the version's existing DataDir remain the only physical shard bytes.
• PUT/DELETE do not create current/ shadow copies or versions/<id>/ direct paths for FastOpen.

FastOpen is therefore always allowed to fall back to the normal xl.meta path without any repair or reconciliation step.

2.2 New storage primitive

Each disk implements:

• StorageAPI.FastOpenPart

This operation:

1. reads the disk-local xl.meta;
2. resolves the requested object version (latest or an explicit VersionID);
3. converts the chosen object version into a compact FastOpenGETMeta;
4. returns a stream whose first bytes are a checked FastOpen frame;
5. appends either:
   • shard bytes,
   • inline bytes,
   • no body for metadata-only outcomes,
   • or a transitioned marker.

This works both locally and over storage REST. The landing node therefore uses the same abstraction for local and remote disks.

---

3. FastOpen frame protocol

The protocol is defined in cmd/fastopen-frame.go.

Each successful FastOpen stream begins with:

• a fixed prelude:
  • magic
  • protocol version
  • payload length
  • CRC
  • body mode
• a compact encoded payload:
  • object/version identity
  • part metadata
  • erasure metadata
  • transition metadata when relevant

Body modes are:

• FastOpenBodyShard
• FastOpenBodyInline
• FastOpenBodyMetadataOnly
• FastOpenBodyTransitioned

Frame statuses are:

• FastOpenStatusOK
• FastOpenStatusDeleteMarker
• FastOpenStatusNotFound
• FastOpenStatusVersionNotFound
• FastOpenStatusUnsupported

Important design point:

• FastOpen does not bypass xl.meta on the storage node.
• It bypasses a second request phase from the landing node by coalescing metadata and body into one stream per disk.

---

4. Request eligibility

FastOpen is intentionally narrow. The request-level gate is implemented in cmd/fastopen-get.go.

FastOpen is attempted only when:

• BUCKIT_FAST_GET=1
• opts.FastGetObjInfo is set by the caller
• the bucket is not the internal metadata bucket
• the request is not a PartNumber GET
• the request is not a range GET
• the request is not a replication or proxy request
• SSE-C is not requested

At object/layout selection time, FastOpen may still reject the object and fall back. Important fallback cases include:

• unsupported frame status from any selected disk
• unsupported checksum algorithm
• inability to assemble enough valid readers
• body mode/layout combinations not handled by the FastOpen reader path
• non-zero body offset after NewGetObjectReader

Delete markers, zero-byte objects, and transitioned objects are still handled via FastOpen metadata when possible, but they return metadata-only object-level results rather than streamed shard decode.

---

5. Landing-node read flow

The read path lives in cmd/fastopen-get.go.

5.1 First wave

tryFastOpenGET calls openFastOpenGETInfo, which:

1. chooses an initial wave of online disks;
2. opens FastOpenPart on each selected disk;
3. reads only the FastOpen frame from each stream;
4. leaves the stream positioned immediately after the frame.

Disk selection uses two modes:

• default: local-first, then stable disk order
• spread mode: rotate the first wave by object hash when BUCKIT_FASTGET_SPREAD=1

The initial open count is based on the set's configured data/parity shape, not the object's eventual winning layout. If the object's actual layout needs more help than the first wave provides, FastOpen opens additional disks.

5.2 Quorum and winning object selection

FastOpen does not invent new correctness rules. After reading the per-disk frames, it rebuilds FileInfo values and reuses the same quorum and version selection rules as canonical GET:

• objectQuorumFromMeta
• reduceReadQuorumErrs
• listOnlineDisks
• pickValidFileInfo
• filterOnlineDisksInplace

That means:

• NotFound and VersionNotFound are still object-level outcomes only after normal quorum rules are applied.
• stale but individually valid disk metadata does not win the read.
• xl.meta semantics remain canonical even though the landing node did not call the normal metadata fan-out helper.

5.3 Replacement wave

If the first wave is not enough before response commit:

1. FastOpen records a replacement-path attempt.
2. It opens the remaining online disks.
3. It retries object selection with the larger set.

If selection still cannot succeed, the request falls back to canonical GET unless BUCKIT_FASTGET_NO_FALLBACK=1, in which case the request returns an error.

---

6. Reader assembly and decode

6.1 Reusing the opened body streams

Once the winning FileInfo is known, FastOpen maps the already-open per-disk streams into erasure-slot readers.

For directly usable body streams:

• shard mode becomes a fastOpenStreamingBitrotReader
• inline mode is also accepted when it can be validated safely

Readers are placed by Erasure.Index, not by response order, so they match the same slot semantics canonical decode expects.

6.2 Lazy replacement readers

If the initially selected body streams do not cover enough slots, FastOpen can fill missing positions with lazy replacement readers.

The lazy replacement path:

1. records which disks are already engaged;
2. chooses the one disk that can satisfy each missing erasure slot;
3. opens a new FastOpenPart stream only when that slot is actually read;
4. validates that the replacement frame still matches the winning object/layout;
5. resumes decode from the requested shard offset.

This allows FastOpen to recover some pre-commit selection failures and some decode-time missing-reader situations without abandoning the request before body streaming begins.

6.3 Streaming bitrot verification

The FastOpen body path uses HighwayHash256S only. Other algorithms currently force fallback.

The streaming reader verifies the existing on-disk bitrot layout inline:

1. read block hash
2. read block bytes
3. recompute hash
4. compare

No new shard-body format is introduced. FastOpen only changes how the landing node gets to the shard bytes.

6.4 Range behavior

The current FastOpen path is effectively full-object only:

• request-level range GETs are rejected up front
• if NewGetObjectReader computes a non-zero offset, FastOpen falls back

Multipart part selection is also not supported in the FastOpen path.

---

7. Fallback model

FastOpen is allowed to fail only before it commits to a streaming response as the request's object-selection path.

Pre-commit failures fall into two categories:

• ok=false: abandon FastOpen and fall back to canonical GET
• ok=true, err!=nil: FastOpen determined the object-level result, including delete marker or quorum failure cases

Once FastOpen has started the body goroutine and returned a GetObjectReader, mid-stream failures are reported as stream errors, not by restarting through the canonical path. This matches the fact that HTTP response commit has already happened.

This is the central simplification versus the abandoned current/ design:

• no request-time repair
• no direct-path reconciliation
• no crash-state cleanup
• no write-side invalidation rules

Fallback is always safe because the canonical layout was never changed.

---

8. Observability

FastOpen observability is exposed under /minio/metrics/v3/api/requests in cmd/metrics-v3-api.go.

Counters include:

• fast_open_attempted_total
• fast_open_hits_total
• fast_open_unsupported_total
• fast_open_replacement_path_total
• fast_open_streams_opened_total
• fast_open_replacement_opens_total
• fast_open_selected_set_failures_total
• fast_open_stream_cancellations_total
• fast_open_final_errors_total
• fast_open_httptrace_connections_total
• fast_open_httptrace_reused_connections_total
• fast_open_httptrace_fresh_connections_total
• fast_open_httptrace_was_idle_connections_total

Timing metrics include:

• fast_open_try_seconds_total
• fast_open_try_seconds_count
• fast_open_open_info_seconds_total
• fast_open_open_info_seconds_count
• fast_open_body_decode_seconds_total
• fast_open_body_decode_seconds_count

The old BUCKIT_FASTOPEN_PROFILE stderr logging path has been removed. Timing is now metrics-based.

---

9. What the final implementation is not

The current FastOpen implementation does not do any of the following:

• no current/part.N shadow copy
• no versions/<versionId>/part.N direct namespace
• no renameat2(RENAME_EXCHANGE)-based latest swap
• no write-side direct-path install/invalidate protocol
• no crash-state repair for direct data paths
• no scanner-driven direct-path reconciliation
• no arbitrary range or multipart FastOpen read path
• no support for all checksum algorithms

Those ideas belonged to earlier design exploration. They are not part of the implemented FastOpen shipped in this branch.

---

10. Expected behavior and tradeoffs

FastOpen improves the healthy full-object GET path by removing one landing-node request phase to each participating disk. It does not remove the disk-local xl.meta lookup itself.

Benefits:

• lower first-byte latency when the request stays on the FastOpen path
• fewer landing-node storage RPC phases
• opportunistic reuse of already-open body streams
• no write-path format migration

Costs and limitations:

• narrow eligibility
• fallback is still common for unsupported objects/requests
• some recovery cases require opening additional disks
• mid-stream errors are not converted into a new canonical GET
• observability is required to know whether a benchmark actually hit the path

This is therefore best understood as a conservative read-path optimization layered on top of Buckit's existing metadata and erasure semantics.

---

11. References

• cmd/fastopen-frame.go - FastOpen frame protocol and compact metadata encoding
• cmd/fastopen-part.go - disk-side FastOpenPart implementation
• cmd/fastopen-get.go - landing-node FastOpen GET path
• cmd/fastget-config.go - runtime flags
• cmd/metrics-v3-api.go - exported metrics
• docs/single-trip-get-phase1-implementation.md - historical prototype/implementation notes