diff --git a/docs/single-trip-get-design.md b/docs/single-trip-get-design.md index 8d12d64ce..0508831bf 100644 --- a/docs/single-trip-get-design.md +++ b/docs/single-trip-get-design.md @@ -1,502 +1,387 @@ -# Single-Trip Direct GET - Design Note +# Single-Trip FastOpen GET - Design Note -**Status:** Draft / discussion notes +**Status:** Implemented in this branch -**Summary:** Buckit currently serves a normal shard-backed GET in two storage -phases: first read `xl.meta` across the erasure set to choose the visible version -and layout, then read shard bytes from the selected data files. This note proposes -adding **direct data paths** on top of the current metadata model: `xl.meta` -remains the canonical version index, while the physical shard directory for the -latest version is named `current` and older versions are named by version ID. The -shard files carry a small checked header before the existing bitrot-protected -shard bytes. A plain latest GET can open -`bucket/object/current/part.1`, read header then shard in one trip, and fall back -to today's `xl.meta` path whenever the fast path is missing, stale, inconsistent, -or not applicable. +**Summary:** Buckit's final FastOpen design does **not** add `current/` shadow +directories, `versions//` direct paths, or write-side shadow repair. Instead, +it adds a **read-side storage protocol** that lets each disk answer a GET with a +single streamed response: -This is not a replacement for `xl.meta`, and `current` is not an extra copy of -the latest shard data. There is one physical data location for a version: -`current` while it is latest, then `versions/` after it is superseded. -The first target is plain latest GET; the same mechanism can also accelerate -explicit version GET. +1. read that disk's `xl.meta`; +2. encode the chosen object version into a compact FastOpen frame; +3. immediately continue with the shard body bytes from the same stream when the + object is streamable. -Read `request-flow.md` first; this note assumes its vocabulary (set, shard, -`xl.meta`, `DataDir`, quorum, `ErasureDist`). +The landing node opens FastOpen streams on a first wave of disks, reads only the +frame headers to establish the winning object/version/layout, then reuses the +already-open body streams for decode. If the first wave is not sufficient, it +opens additional disks or falls back to canonical GET. `xl.meta` remains +canonical; FastOpen is an opportunistic read optimization only. + +This note describes the **shipped FastOpen design**. It supersedes earlier +proposals based on `current/part.1`, `versions//`, and write-side +shadow copies. + +Read `request-flow.md` first; this note assumes its vocabulary (`xl.meta`, +erasure sets, parts, `FileInfo`, quorum, distribution, local vs remote disk +access). --- -## 1. Context and motivation +## 1. Goal -### 1.1 Why this matters: making HDD more viable +Canonical GET pays two logical storage phases: -The architecture Buckit inherited from MinIO is optimized for flash media. HDDs -are cheaper per terabyte and strong at sequential I/O, but weak at random metadata -IOPS. This matters for capacity-oriented deployments such as backups, archives, -media stores, and data lakes. +1. read `xl.meta` enough times to establish the visible version and layout; +2. open shard data files and decode from the selected readers. -The HDD issue is broader than GET alone: small-object workloads, listing, -versioning, healing, scanning, and degraded reads can all create random I/O. This -proposal targets one concrete hot-path contributor: latest GET currently pays a -metadata-file read phase before it can open shard files. +FastOpen keeps the same correctness rules but collapses those into one per-disk +storage request. Each participating disk returns metadata and, when possible, +body bytes from the same open stream. -### 1.2 Current GET path - -For a healthy 16-disk set with `EC:4` (M=12 data, N=4 parity), a normal large -object GET generally does: - -1. **Metadata fan-out:** read `xl.meta` from disks in the set to establish quorum - on the visible version and learn its layout. -2. **Shard read:** open the selected `DataDir/part.N` files and Reed-Solomon - decode from M available shard streams, pulling extra shards only on failures. - -For large shard-backed objects, this is roughly one metadata-file seek plus one -data-file seek on the disks that participate in the read. On HDD, those seeks and -the second storage request phase dominate first-byte latency. On flash, the seek -cost is small, but the extra request phase still costs latency. - -This statement has important exceptions: - -- Small objects can already be returned from the metadata read path when inline - data is present or when `ReadData` inlines a small single-part shard. -- Zero-byte and transitioned objects take special paths. -- The decoder does not eagerly read all M+N shards; it starts with M readers per - stripe and falls back to more only when needed. - -The target is therefore not "all GETs." The first target is the common large, -local, shard-backed latest GET, with explicit version GET as a natural extension -when the version ID maps directly to a stable data directory. +This is a **GET-only** optimization. It does not change the canonical write +format, object version layout, or healing authority model. --- -## 2. Core design: `xl.meta` canonical, direct data paths +## 2. High-level architecture -The proposed layout adds stable directly-openable data directories alongside the -canonical metadata file: +### 2.1 Canonical state stays unchanged -``` -bucket/object/xl.meta # canonical version chain and metadata -bucket/object/current/part.1 # physical data for the latest version -bucket/object/versions//part.1 # physical data for older version -bucket/object/versions//part.1 -``` +FastOpen does not introduce a second on-disk object namespace. -For non-versioned or null-version objects, the direct version path needs an -encoded reserved name rather than the literal empty/null value. +- `xl.meta` remains the source of truth for visible versions, delete markers, + transition state, erasure layout, and object metadata. +- The ordinary shard files under the version's existing `DataDir` remain the only + physical shard bytes. +- PUT/DELETE do not create `current/` shadow copies or `versions//` direct + paths for FastOpen. -There is no duplicate `part.1` under both `current` and `versions/` for -the same version. When a latest version is superseded, its data directory is -renamed or exchanged out of `current` into its deterministic version path. +FastOpen is therefore always allowed to fall back to the normal `xl.meta` path +without any repair or reconciliation step. -`xl.meta` remains the source of truth for: +### 2.2 New storage primitive -- version existence and ordering; -- delete markers; -- fallback for explicit `GET ?versionId=...`; -- listing and version listing; -- lifecycle, replication, healing, scanner, and repair decisions; -- compatibility with existing objects. +Each disk implements: -`current` is the direct path for plain GET without `versionId`. -`versions/` is the direct path for explicit GET of non-current -versions. If a direct path is missing or disagrees with quorum, the request falls -back to the existing `xl.meta`-driven path. Repair can later reconstruct missing -or corrupt direct data from erasure quorum, with `xl.meta` deciding what should -exist. +- `StorageAPI.FastOpenPart` -### 2.1 Current shard file format +This operation: -Each direct-path `part.N` file starts with a small header, then the existing shard -payload: +1. reads the disk-local `xl.meta`; +2. resolves the requested object version (`latest` or an explicit `VersionID`); +3. converts the chosen object version into a compact `FastOpenGETMeta`; +4. returns a stream whose first bytes are a checked FastOpen frame; +5. appends either: + - shard bytes, + - inline bytes, + - no body for metadata-only outcomes, + - or a transitioned marker. -``` -current/part.1 (one disk's current shard for one S3 part) -+--------------+-----------------------------------------------------------+ -| HEADER | SHARD DATA | -| (metadata) | hash_0 | block_0 | hash_1 | block_1 | ... | hash_k | block_k | -+--------------+-----------------------------------------------------------+ - ^ length-prefixed, checksummed ^ existing per-block bitrot layout -``` - -The header carries enough information for the landing node to validate and decode -the fast path: - -- object name, version ID, modtime, etag, object size; -- whether the current entry is an object or delete marker; -- direct path identity (`current` or `versions/`) and canonical - version identity; -- `ErasureM`, `ErasureN`, `ErasureIndex`, `ErasureDist`, block size; -- part number, part size, actual shard file size, checksum algorithm; -- version/header signature used for cross-disk quorum comparison; -- format version and header checksum. - -The shard-data region should remain byte-for-byte compatible with the existing -bitrot block layout after accounting for the header offset. Existing bitrot -verification and erasure decode should not need semantic changes, but the reader -must know that shard offsets are relative to the payload start, not file offset 0. - -The header should be **variable-length with a self-describing payload** (a -length-prefixed, version-tagged frame followed by tagged/keyed fields, e.g. -TLV or msgpack), not a fixed-size positional record, so fields can be -added/removed across releases without lockstep: the reader takes the payload -length from the frame and the shard data begins immediately after, preserving -the single trip. Because `current` is a rebuildable cache of `xl.meta`, an -unrecognized format version or any decode failure simply falls back to the -canonical path and the shadow is lazily rewritten — so the schema can evolve with -no in-place migration. (The Phase 1 prototype uses a fixed 1024-byte header with a -positional payload for simplicity; that is an implementation shortcut, not the -intended on-disk contract.) - -### 2.2 Delete-marker current - -Latest can be a delete marker. A plain GET must return not found when the quorum -latest entry is a delete marker, even if older data versions exist. - -Therefore `current` must also represent "latest is deleted." Options: - -- a small `current/marker` file with the same checked header and no shard data; -- a reserved `current/part.1` header marked delete-marker with empty payload; -- no `current` data plus a separate marker file. - -The fast path must never silently fall through to older shard data when latest is -a delete marker. If the delete-marker fast path is missing or inconsistent, fall -back to `xl.meta`. +This works both locally and over storage REST. The landing node therefore uses +the same abstraction for local and remote disks. --- -## 3. Fast GET protocol +## 3. FastOpen frame protocol -### 3.1 Plain latest GET +The protocol is defined in [cmd/fastopen-frame.go](/Users/rooseveltlai/develop/buckit-io/buckit/cmd/fastopen-frame.go:1). -For `GET /bucket/object` without `versionId`: +Each successful FastOpen stream begins with: -1. The landing node opens `current/part.N` on each disk in the set for the S3 part - needed by the request. -2. Each disk returns the checked header first. -3. The landing node compares headers and establishes quorum on the same current - version/signature. -4. If the quorum current is a delete marker, return not found. -5. If the quorum current is object data, stream from the selected M shard readers. -6. If any check fails, fall back to the existing `xl.meta` path and queue repair - of the direct data path. +- a fixed prelude: + - magic + - protocol version + - payload length + - CRC + - body mode +- a compact encoded payload: + - object/version identity + - part metadata + - erasure metadata + - transition metadata when relevant -The common healthy path removes the separate `xl.meta` read phase from latest -GET. The correctness rule is simple: `current` can accelerate the answer, but it -cannot override `xl.meta`. +Body modes are: -### 3.2 Header-pause vs speculative stream +- `FastOpenBodyShard` +- `FastOpenBodyInline` +- `FastOpenBodyMetadataOnly` +- `FastOpenBodyTransitioned` -For large objects, the safest protocol is header-pause: +Frame statuses are: -``` -each disk -> stream HEADER -> pause -landing node: - collect headers and establish quorum - continue selected M shard streams - stop the rest -``` +- `FastOpenStatusOK` +- `FastOpenStatusDeleteMarker` +- `FastOpenStatusNotFound` +- `FastOpenStatusVersionNotFound` +- `FastOpenStatusUnsupported` -For small objects or low-latency flash deployments, the implementation may choose -to stream header and data speculatively and cancel losers after quorum selection. -This trades some wasted bandwidth for lower control-plane complexity. +Important design point: -The existing decoder already starts with M readers per stripe and reads additional -shards only after missing/corrupt reads. The new protocol should preserve that -behavior rather than eagerly reading all M+N shards. - -### 3.3 Explicit version GET - -Explicit version GET can use the same direct-path mechanism, but the lookup must -account for the no-duplicate rule: - -``` -GET /bucket/object?versionId= --> try bucket/object/current/part.N and accept it only if the header says --> otherwise try bucket/object/versions//part.N --> read header, establish quorum for , stream selected M shards -``` - -This removes the `xl.meta` lookup for healthy explicit version GETs when the -requested version is either current or already under `versions/`. If the direct -paths are missing, corrupt, stale, or not supported by an old object, fall back to -today's `xl.meta` path and queue repair if enough shard data exists. - -The directory name must be a safe encoding of the S3 version ID. The header still -needs to carry and validate the canonical version ID; the path name alone is not -sufficient for correctness. - -### 3.4 Multipart and range GET - -In the current layout, `part.N` is the S3 multipart part number, not the erasure -shard number. Most single-part objects only have `part.1`; multipart objects may -have `part.2`, `part.3`, and so on. - -Today's multipart read flow is: - -1. read `xl.meta` and choose the visible version; -2. use that version's part list to map the requested object byte range to a start - part, offset within that part, and end part; -3. for each needed S3 part, open `bucket/object//part.N` across disks; -4. Reed-Solomon decode that part's shard streams, then continue to the next - `part.N` until the requested range is complete. - -Range GET complicates the fast path because the starting byte may map to a later -S3 part. To keep the fast path correct, every direct-path `part.N` that can be -opened directly must carry enough header information to validate the requested -version and that part's layout. - -In the direct-path layout, a multipart current version would look like: - -``` -bucket/object/current/part.1 -bucket/object/current/part.2 -bucket/object/current/part.3 -``` - -Each file needs its own header so a range GET that starts in `part.3` can validate -the version and part layout without first reading `part.1` or `xl.meta`. - -Initial implementation can reasonably limit the fast path to: - -- full-object or range reads that begin in `part.1`; -- single-part objects; -- non-transitioned shard-backed data. - -Other requests fall back to `xl.meta` until the multipart/range protocol is -explicitly designed. +- FastOpen does **not** bypass `xl.meta` on the storage node. +- It bypasses a **second request phase** from the landing node by coalescing + metadata and body into one stream per disk. --- -## 4. Write and crash consistency +## 4. Request eligibility -### 4.1 Canonical commit +FastOpen is intentionally narrow. The request-level gate is implemented in +[cmd/fastopen-get.go](/Users/rooseveltlai/develop/buckit-io/buckit/cmd/fastopen-get.go:175). -PUT/DELETE correctness remains based on today's canonical metadata commit: +FastOpen is attempted only when: -1. write shard data to temporary files; -2. commit the new latest data to `current`; -3. if an older latest version existed, move that old `current` directory to - `versions/`; -4. update `xl.meta` as the canonical visible version chain. +- `BUCKIT_FAST_GET=1` +- `opts.FastGetObjInfo` is set by the caller +- the bucket is not the internal metadata bucket +- the request is not a `PartNumber` GET +- the request is not a range GET +- the request is not a replication or proxy request +- SSE-C is not requested -The ordering can be adjusted as long as recovery has a deterministic rule. The -important invariant is that `xl.meta` decides which versions are visible and which -direct path should contain each version. If a crash leaves a mismatch, fast GET -falls back to `xl.meta`; repair either reconstructs the missing direct path from -erasure quorum or removes uncommitted data. +At object/layout selection time, FastOpen may still reject the object and fall +back. Important fallback cases include: -If a crash leaves `current` newer than `xl.meta`, `xl.meta` wins. The fast path -must detect that by quorum/header validation or by falling back when the `current` -headers do not form a valid committed quorum. +- unsupported frame status from any selected disk +- unsupported checksum algorithm +- inability to assemble enough valid readers +- body mode/layout combinations not handled by the FastOpen reader path +- non-zero body offset after `NewGetObjectReader` -### 4.2 Updating `current` - -A stable `current` directory is attractive because latest GET can open: - -``` -bucket/object/current/part.1 -``` - -without resolving an opaque `DataDir` through `xl.meta`. - -The important rule is to replace `current` by rename/exchange, not by overwriting -shard bytes in place. A reader that already opened `current/part.1` holds a file -descriptor to the old inode, so it can keep streaming even after the `current` -path is atomically swapped to a new directory. The swapped-out old directory then -becomes the immutable `versions/` directory; it is not a temporary -duplicate. - -Updating that directory is not identical to today's `RenameData` flow. Today a -fresh immutable `DataDir` is committed under an opaque name and then made visible -through `xl.meta`. In the new layout, the latest version's physical data must land -at: - -``` -bucket/object/current -``` - -and the previous latest data must be preserved under: - -``` -bucket/object/versions/ -``` - -There is no second copy. The filesystem operation is a rename/exchange of -directories, not a copy into `current`. - -On Linux, `renameat2(RENAME_EXCHANGE)` can atomically swap two existing pathnames, -including directories on filesystems that support it: - -``` -bucket/object/current <-> bucket/object/.staging-new -``` - -After the exchange: - -``` -bucket/object/current # new latest data -bucket/object/.staging-new # old latest data, to rename to versions/ -``` - -This can make latest replacement atomic at the per-disk namespace level, but it is -not portable POSIX behavior and still needs: - -- startup capability detection for kernel and filesystem support; -- correct file and parent-directory fsync ordering; -- recovery of the old swapped-out directory into `versions/`; -- handling when latest is a delete marker; -- fallback for platforms/filesystems without `RENAME_EXCHANGE`. - -The post-exchange rename of old data to `versions/` is another -crash point. Recovery must be able to recognize a swapped-out old-current staging -directory from its header and finish the rename or discard it if `xl.meta` says it -is not a committed version. +Delete markers, zero-byte objects, and transitioned objects are still handled via +FastOpen metadata when possible, but they return metadata-only object-level +results rather than streamed shard decode. --- -## 5. Durability risks and remediations +## 5. Landing-node read flow -The new direct paths do not weaken erasure coding, but they add crash states that -the current opaque-`DataDir` layout does not have. The safe rule is: +The read path lives in [cmd/fastopen-get.go](/Users/rooseveltlai/develop/buckit-io/buckit/cmd/fastopen-get.go:1). -``` -xl.meta remains canonical. -Direct paths are served only when their headers form quorum for a committed version. -Staging/current/version mismatches are reconciled against xl.meta. -``` +### 5.1 First wave -### 5.1 Crash-state risks +`tryFastOpenGET` calls `openFastOpenGETInfo`, which: -| Risk | Example state | Remediation | -|---|---|---| -| `current` ahead of `xl.meta` | `current` header says `v2`, but `xl.meta` still says latest is `v1` | Fast GET falls back to `xl.meta`; scanner/repair quarantines or removes uncommitted `v2` unless quorum metadata later confirms it. | -| `xl.meta` ahead of `current` | `xl.meta` says latest is `v2`, but `current` still contains `v1` | Fast GET detects header mismatch and falls back; repair moves/reconstructs `v2` into `current`. | -| old latest stranded in staging | after exchange, `.staging-new` contains `v1`, but `versions/v1` is missing | Recovery reads the staging header, verifies `v1` is committed in `xl.meta`, and renames it to `versions/v1`. | -| delete marker torn update | `xl.meta` latest is delete marker, but `current` still contains old data | Fast GET must not serve old data unless `current` headers form quorum for the committed latest; fallback returns 404 and repair installs the delete-marker current state. | -| partial per-disk commit | some disks have `current=v2`, some `current=v1`, some staging leftovers | Landing node groups headers by signature; serve only a committed quorum, otherwise fall back and queue per-disk repair. | -| missing fsync after rename/exchange | syscall returned success, but crash loses a directory entry | Follow strict fsync ordering for files and parent directories; recovery treats missing direct paths as repairable if `xl.meta` has quorum. | -| stale or orphan direct dirs | directory contains a version not present in `xl.meta` | Scanner quarantines or deletes after confirming it is not referenced by canonical metadata. | +1. chooses an initial wave of online disks; +2. opens `FastOpenPart` on each selected disk; +3. reads only the FastOpen frame from each stream; +4. leaves the stream positioned immediately after the frame. -### 5.2 Request-time repair trigger +Disk selection uses two modes: -The fast GET path can detect direct-path divergence cheaply from headers: +- default: local-first, then stable disk order +- spread mode: rotate the first wave by object hash when + `BUCKIT_FASTGET_SPREAD=1` -1. The landing node asks all disks for `current/part.N` or `versions//part.N`. -2. It groups returned headers by version/signature. -3. It reads from the quorum group if the group is valid for the request. -4. Disks whose headers are missing, corrupt, or outside the quorum group are - excluded from the read. -5. The landing node queues a repair for those disk/object paths. +The initial open count is based on the set's configured data/parity shape, not +the object's eventual winning layout. If the object's actual layout needs more +help than the first wave provides, FastOpen opens additional disks. -The landing node should not tell a disk "your version is wrong" based only on the -header mismatch. It should tell the disk to reconcile that object against -canonical `xl.meta`. The disk-local repair then decides whether its direct path is -ahead, behind, stranded in staging, or uncommitted. +### 5.2 Quorum and winning object selection -### 5.3 Disk-local reconciliation algorithm +FastOpen does not invent new correctness rules. After reading the per-disk +frames, it rebuilds `FileInfo` values and reuses the same quorum and version +selection rules as canonical GET: -Per-disk repair extends the existing healing/scanner model: +- `objectQuorumFromMeta` +- `reduceReadQuorumErrs` +- `listOnlineDisks` +- `pickValidFileInfo` +- `filterOnlineDisksInplace` -1. Read local `xl.meta`. -2. Build the expected direct-path map: - `current` -> latest committed object version or delete-marker marker, and - `versions/` -> each committed non-current object version. -3. Inspect `current`, `versions/*`, and staging directories by reading their - checked headers. -4. Keep directories whose header matches the version expected at that path. -5. If a staging/current/version directory contains a committed version expected - elsewhere, rename it into the expected path. -6. If a directory contains an uncommitted version not present in `xl.meta`, - quarantine/delete it. -7. If committed shard data is missing or corrupt but recoverable from other disks, - invoke normal erasure healing to reconstruct it. +That means: -This is the same authority model Buckit uses today: metadata decides what should -exist; shard directories are repaired or cleaned up to match it. +- `NotFound` and `VersionNotFound` are still object-level outcomes only after + normal quorum rules are applied. +- stale but individually valid disk metadata does not win the read. +- `xl.meta` semantics remain canonical even though the landing node did not call + the normal metadata fan-out helper. + +### 5.3 Replacement wave + +If the first wave is not enough before response commit: + +1. FastOpen records a replacement-path attempt. +2. It opens the remaining online disks. +3. It retries object selection with the larger set. + +If selection still cannot succeed, the request falls back to canonical GET unless +`BUCKIT_FASTGET_NO_FALLBACK=1`, in which case the request returns an error. --- -## 6. Expected performance +## 6. Reader assembly and decode -For a healthy 16-disk EC:4 set serving large shard-backed latest GETs, the current -path is roughly: +### 6.1 Reusing the opened body streams -``` -16 xl.meta reads + 12 shard reads -``` +Once the winning `FileInfo` is known, FastOpen maps the already-open per-disk +streams into erasure-slot readers. -The fast path is roughly: +For directly usable body streams: -``` -16 current header reads, then M shard streams from the same opened files -``` +- shard mode becomes a `fastOpenStreamingBitrotReader` +- inline mode is also accepted when it can be validated safely -On HDD, the optimistic seek-bound ceiling is about: +Readers are placed by `Erasure.Index`, not by response order, so they match the +same slot semantics canonical decode expects. -``` -28 / 16 ~= 1.75x -``` +### 6.2 Lazy replacement readers -In practice, expected gains are lower: +If the initially selected body streams do not cover enough slots, FastOpen can +fill missing positions with lazy replacement readers. -- seek-bound small/medium shard-backed GET throughput: up to about 1.5x-1.75x; -- first-byte latency on HDD: roughly 30%-50% lower in the healthy fast path; -- large sequential transfers: usually 0%-10% total transfer improvement, mostly - faster first byte; -- flash/NVMe: mostly one fewer storage request phase, often a smaller latency win. +The lazy replacement path: -These are estimates, not measured results. The win shrinks or disappears when the -request falls back to `xl.meta`, targets old objects that have not been migrated -to direct paths, targets transitioned objects, needs multipart/range behavior not -covered by the fast path, or runs from warm metadata cache where the `xl.meta` -seek is already cheap. +1. records which disks are already engaged; +2. chooses the one disk that can satisfy each missing erasure slot; +3. opens a new `FastOpenPart` stream only when that slot is actually read; +4. validates that the replacement frame still matches the winning object/layout; +5. resumes decode from the requested shard offset. + +This allows FastOpen to recover some pre-commit selection failures and some +decode-time missing-reader situations without abandoning the request before body +streaming begins. + +### 6.3 Streaming bitrot verification + +The FastOpen body path uses `HighwayHash256S` only. Other algorithms currently +force fallback. + +The streaming reader verifies the existing on-disk bitrot layout inline: + +1. read block hash +2. read block bytes +3. recompute hash +4. compare + +No new shard-body format is introduced. FastOpen only changes how the landing +node gets to the shard bytes. + +### 6.4 Range behavior + +The current FastOpen path is effectively full-object only: + +- request-level range GETs are rejected up front +- if `NewGetObjectReader` computes a non-zero offset, FastOpen falls back + +Multipart part selection is also not supported in the FastOpen path. --- -## 7. Fallback and repair rules +## 7. Fallback model -Fast GET must fall back to the current implementation when: +FastOpen is allowed to fail only **before** it commits to a streaming response as +the request's object-selection path. -- the direct path (`current` or `versions/`) is missing on too many - disks; -- headers do not form quorum on version/signature; -- a header checksum fails; -- the current entry is ambiguous relative to delete-marker semantics; -- requested range maps to a `part.N` not supported by the fast path; -- bitrot verification fails and quorum cannot be reconstructed from available - fast-path shards; -- the object is transitioned, restored, or otherwise not local shard-backed data. +Pre-commit failures fall into two categories: -After fallback succeeds through `xl.meta`, the system should queue a repair to -finish any interrupted rename/exchange, reconstruct missing shards from erasure -quorum, or delete uncommitted direct data. +- `ok=false`: abandon FastOpen and fall back to canonical GET +- `ok=true, err!=nil`: FastOpen determined the object-level result, including + delete marker or quorum failure cases + +Once FastOpen has started the body goroutine and returned a `GetObjectReader`, +mid-stream failures are reported as stream errors, not by restarting through the +canonical path. This matches the fact that HTTP response commit has already +happened. + +This is the central simplification versus the abandoned `current/` design: + +- no request-time repair +- no direct-path reconciliation +- no crash-state cleanup +- no write-side invalidation rules + +Fallback is always safe because the canonical layout was never changed. --- -## 8. Open decisions +## 8. Observability -1. **Direct-path layout:** `current/part.N` for the latest version plus - `versions//part.N` for non-current versions, stable regular files, - or another no-duplicate path scheme. -2. **Current update protocol:** `renameat2(RENAME_EXCHANGE)` vs. staged - rename/recovery fallback for filesystems without atomic directory exchange. -3. **Delete-marker representation:** marker file, empty current shard header, or - separate current state file. -4. **Multipart/range scope:** whether v1 supports only single-part `part.1` reads - or duplicates headers into every direct-path `part.N`. -5. **Transport:** header-pause control protocol vs. speculative stream/cancel. -6. **Migration:** old objects simply miss direct paths and use `xl.meta`; scanner - or write/read repair can migrate them opportunistically. +FastOpen observability is exposed under +`/minio/metrics/v3/api/requests` in +[cmd/metrics-v3-api.go](/Users/rooseveltlai/develop/buckit-io/buckit/cmd/metrics-v3-api.go:1). + +Counters include: + +- `fast_get_hits_total` +- `fast_get_fallbacks_total` +- `fast_open_attempted_total` +- `fast_open_hits_total` +- `fast_open_unsupported_total` +- `fast_open_replacement_path_total` +- `fast_open_streams_opened_total` +- `fast_open_replacement_opens_total` +- `fast_open_selected_set_failures_total` +- `fast_open_stream_cancellations_total` +- `fast_open_final_errors_total` +- `fast_open_httptrace_connections_total` +- `fast_open_httptrace_reused_connections_total` +- `fast_open_httptrace_fresh_connections_total` +- `fast_open_httptrace_was_idle_connections_total` + +Timing metrics include: + +- `fast_open_try_seconds_total` +- `fast_open_try_seconds_count` +- `fast_open_open_info_seconds_total` +- `fast_open_open_info_seconds_count` +- `fast_open_body_decode_seconds_total` +- `fast_open_body_decode_seconds_count` + +The old `BUCKIT_FASTOPEN_PROFILE` stderr logging path has been removed. Timing is +now metrics-based. --- -## 9. References +## 9. What the final implementation is not -- `docs/request-flow.md` - current GET/PUT/DELETE flows, `xl.meta` format, - quorum, and local-vs-distributed transport split. -- `cmd/erasure-object.go` - `GetObjectNInfo`, `getObjectFileInfo`, and the decode - loop. -- `cmd/erasure-decode.go` - M-reader decode behavior and degraded fallback. -- `cmd/xl-storage.go` - `ReadXL`, `ReadVersion`, inline-data behavior, and - `ReadFileStream`. -- `cmd/storage-interface.go` - storage API signatures for metadata and file - reads. +The current FastOpen implementation does **not** do any of the following: + +- no `current/part.N` shadow copy +- no `versions//part.N` direct namespace +- no `renameat2(RENAME_EXCHANGE)`-based latest swap +- no write-side direct-path install/invalidate protocol +- no crash-state repair for direct data paths +- no scanner-driven direct-path reconciliation +- no arbitrary range or multipart FastOpen read path +- no support for all checksum algorithms + +Those ideas belonged to earlier design exploration. They are not part of the +implemented FastOpen shipped in this branch. + +--- + +## 10. Expected behavior and tradeoffs + +FastOpen improves the healthy full-object GET path by removing one landing-node +request phase to each participating disk. It does **not** remove the disk-local +`xl.meta` lookup itself. + +Benefits: + +- lower first-byte latency when the request stays on the FastOpen path +- fewer landing-node storage RPC phases +- opportunistic reuse of already-open body streams +- no write-path format migration + +Costs and limitations: + +- narrow eligibility +- fallback is still common for unsupported objects/requests +- some recovery cases require opening additional disks +- mid-stream errors are not converted into a new canonical GET +- observability is required to know whether a benchmark actually hit the path + +This is therefore best understood as a conservative read-path optimization layered +on top of Buckit's existing metadata and erasure semantics. + +--- + +## 11. References + +- [cmd/fastopen-frame.go](/Users/rooseveltlai/develop/buckit-io/buckit/cmd/fastopen-frame.go:1) - FastOpen frame protocol and compact metadata encoding +- [cmd/fastopen-part.go](/Users/rooseveltlai/develop/buckit-io/buckit/cmd/fastopen-part.go:1) - disk-side `FastOpenPart` implementation +- [cmd/fastopen-get.go](/Users/rooseveltlai/develop/buckit-io/buckit/cmd/fastopen-get.go:1) - landing-node FastOpen GET path +- [cmd/fastget-config.go](/Users/rooseveltlai/develop/buckit-io/buckit/cmd/fastget-config.go:1) - runtime flags +- [cmd/metrics-v3-api.go](/Users/rooseveltlai/develop/buckit-io/buckit/cmd/metrics-v3-api.go:1) - exported metrics +- [docs/single-trip-get-phase1-implementation.md](/Users/rooseveltlai/develop/buckit-io/buckit/docs/single-trip-get-phase1-implementation.md:1) - historical prototype/implementation notes diff --git a/docs/single-trip-get-full-feature-gaps.md b/docs/single-trip-get-full-feature-gaps.md deleted file mode 100644 index 0cfcdb38e..000000000 --- a/docs/single-trip-get-full-feature-gaps.md +++ /dev/null @@ -1,874 +0,0 @@ -# FastGet Full-Feature Design Gaps - -**Status:** design backlog after Phase 1 prototype - -**Inputs:** -- `docs/single-trip-get-design.md` - intended production direction. -- `docs/single-trip-get-phase1-implementation.md` - measurement prototype plan. -- Current `feature/single-trip-get-phase1` prototype - additive `current/part.1` - shadow, fixed header, EAGERSTABLE default, stable shard selection, parity - reconstruction, and prototype metrics. - -This document lists the remaining design work needed to turn FastGet from a -measurement prototype into a full system feature. It is not an implementation -plan yet. Each section should produce a concrete design decision, invariants, -failure handling, and tests before code lands. - ---- - -## 1. Current Prototype Boundary - -The current branch proves the main latency mechanism: a latest GET can skip the -`xl.meta` fan-out and reconstruct `FileInfo` from a checked header next to shard -bytes. - -Important prototype properties: - -- `current/part.1` is an additive shadow copy, not the only physical data path. -- FastGet is gated by `BUCKIT_FAST_GET`. -- Default candidate is EAGERSTABLE: eager first-block prefetch is on, stable - spread selection is on, hedging is off. -- Only latest, non-versioned, single-part, non-inline, local, unencrypted, - uncompressed, non-object-lock objects are eligible. -- FastGet is true no-`xl.meta` only on the `SinglePool()` path. Multi-pool GET - still resolves the owner through metadata first. -- The fixed 1024-byte header carries only a bounded response-metadata subset. -- There is no request-time repair, scanner reconciliation, fsync protocol, or - production migration path. - -Design implication: the prototype is good evidence for the GET-side mechanism, -but it is not production-safe and does not define the final on-disk contract. - ---- - -## 2. Product Scope and Compatibility - -### Gaps - -- Decide which GET API surfaces FastGet must cover beyond Phase 1. FastGet is - GET-only for all planned phases; HEAD stays on the existing metadata-only - `GetObjectInfo` path and is out of FastGet scope. -- Decide whether FastGet is a supported persistent feature or still an - experimental server flag. -- Define compatibility behavior for old objects without `current` aliases or - embedded direct headers, old header versions, mixed-version clusters, and - downgrade. -- Define user-visible behavior: FastGet must not change response bytes, response - headers, errors, version semantics, lifecycle behavior, replication behavior, or - object-lock semantics. - -### Design tasks - -- Define the supported request matrix and fallback matrix. -- Specify config shape: environment-only, config subsystem, dynamic cluster-wide - toggle, or per-bucket policy. -- Define upgrade/downgrade rules and minimum mixed-cluster safety constraints. -- Decide whether FastGet can be enabled by default, and under what storage/media - conditions. -- Define production observability: hits, fallbacks, fallback reasons, repair - queue stats, stale-`current` repair stats, and latency counters without noisy - per-request profiling. - ---- - -## 3. Direct-Path Layout - -### Gaps - -The production design wants no duplicate latest data: - -``` -bucket/object/current/part.N -bucket/object//part.N -bucket/object/xl.meta -``` - -The prototype instead writes: - -``` -bucket/object//part.1 -bucket/object/current/part.1 -bucket/object/xl.meta -``` - -That doubles write cost and storage, makes PUT numbers invalid, and leaves the -system without a design for old-version direct GETs. - -### Discussed direction - -Keep the existing Buckit `xl.meta` plus opaque `DataDir` layout as the durable -layout. Add `current` only as a latest-version GET acceleration alias: - -``` -bucket/object/xl.meta -bucket/object//part.N -bucket/object/current/part.N -``` - -The important invariant is that `xl.meta` remains the source-of-truth metadata and -version index, and each version's shard data remains under the `DataDir` -recorded in `xl.meta`. `current` is only an acceleration path for latest-version -GET. Explicit old-version GET, such as `GET ?versionId=`, should use -the coalesced `xl.meta` route. That is slightly slower than a -direct version path, but old-version GET is not the common hot path and does not -justify a new `versions/` directory layout in the initial full -feature. - -Two `current` materializations should be designed and benchmarked behind the -same FastGet read path: - -1. **Reserved real DataDir:** `current/part.N` contains the latest version's - shard files directly. In this mode, `current` is a special `DataDir` name for - the latest version. This avoids symlink semantics but makes PUT/overwrite - harder because the previous latest `current` directory must be renamed to a - generated `DataDir` and recorded in `xl.meta` without exposing mixed crash - states. -2. **Object-level pointer:** `current` points to the latest generated `DataDir`, - for example as a directory symlink on POSIX filesystems. This makes PUT - simpler: write a new generated `DataDir`, update `xl.meta`, then atomically - replace the single `current` pointer. - -Avoid per-part symlinks such as: - -``` -bucket/object/current/part.1 -> ..//part.1 -``` - -A per-part pointer can expose mixed state when an object has multiple S3 parts, -header sidecars, checksums, or future layout files. If indirection is used, it -should be a single object-level pointer: - -``` -bucket/object/current -> -``` - -The symlink/pointer mode is acceptable only if it is scoped as a rebuildable -GET acceleration index: - -- PUT, DELETE, scanner, heal, lifecycle, replication, GC, and version listing - must continue to use `xl.meta` and recorded `DataDir` paths. -- FastGet may open `current/part.N`, but it must validate the direct header and - fall back to the coalesced `xl.meta` FastOpen tier if the pointer is missing, - stale, invalid, or points at a header that does not represent the latest - committed version. -- Delete-marker latest state must remove or invalidate `current`, or otherwise - make FastGet fall back. It must never serve the previous data version as - latest after a committed delete marker. -- Pointer targets must be relative, object-local, and unable to escape the - object directory. -- The implementation should hide the choice behind a storage-layer publish - operation, not expose generic symlink creation to unrelated code. - -This keeps the FastGet reader stable (`current/part.N`) while allowing a -controlled comparison between real-directory and object-level-pointer publish -protocols. - -### Design tasks - -- Decide final directory names for: - - latest object data; - - generated `DataDir` directories; - - null versions and non-versioned objects; - - delete-marker current state; - - staging directories; - - quarantine/orphan directories. -- Decide whether production `current` is a real directory, an object-level - pointer, or a runtime/feature-gated choice for benchmarking. -- If `current` is a pointer, define the storage API operation for atomic pointer - replacement and the fallback behavior on platforms where symlink creation is - unavailable or unreliable. -- Define whether generated `DataDir` names stay unchanged or whether `current` - as a reserved real `DataDir` requires additional reserved-name handling. -- Define how `current` aliases and embedded direct headers coexist with existing - opaque `DataDir` objects during migration. -- Define scanner behavior for `current` aliases, old `DataDir` paths, orphan - staging paths, and partial migrations. - ---- - -## 4. Header Format and Metadata Fidelity - -### Gaps - -The prototype header is fixed-size and intentionally scoped down. A full feature -needs a durable schema and enough metadata to make FastGet byte/header-equivalent -to existing GET behavior. - -Metadata still needing design: - -- user metadata: `x-amz-meta-*`; -- object tags: `x-amz-tagging`, tag count, optional tag response; -- object-lock metadata: retention and legal hold, permission-filtered on GET; -- checksum metadata for `x-amz-checksum-mode: ENABLED`; -- server-side encryption metadata for SSE-S3 and SSE-KMS, including enough - sealed-key/KMS metadata for the landing node to run the existing decryption - path after erasure decode; -- compression metadata and decompression behavior, with range-specific - compressed-offset planning deferred until range FastGet is in scope; -- transition/restore metadata and remote-tier behavior; -- replication status and version purge status; -- multipart part metadata and per-part checksums; -- non-STANDARD storage class and lifecycle prediction headers. - -### Discussed direction - -Use a two-tier storage-node open protocol: - -1. **Embedded direct header:** the storage node opens `current/part.1`, reads the - embedded direct header, validates it, and streams the shard body after the - header. This is the highest-performance FastGet path because it avoids - request-time `xl.meta` reads. -2. **Coalesced `xl.meta` open:** if the direct header is missing, invalid, - stale, or cannot represent the object metadata, the storage node reads - `xl.meta`, selects the version/data directory, and returns a - metadata frame followed by the shard stream in the same storage response. - -The landing node should see both tiers through one framed `FastOpenPart` -response stream. The first frame identifies the tier and carries either direct -header data or coalesced `xl.meta` metadata; shard bytes follow in the same -stream: - -``` -FastOpenPart(bucket, object, version selector, part.N) - -> DirectHeaderFrame, then direct shard bytes - -> CoalescedMetadataFrame, then shard bytes - -> miss/error -``` - -This means the embedded header does not need to encode every object case in the -first full version. It can stay optimized for the high-value direct path, while -the coalesced `xl.meta` open preserves full metadata fidelity and keeps one -storage RPC for broader object coverage. - -The embedded direct header should be a variable-length frame, not a fixed -reserved slot. A fixed 4 KiB or 16 KiB header area would permanently add padding -to every eligible object, which is especially wasteful for small objects. The -preferred shape is: - -``` -part.1 = fixed small prelude + variable metadata frame + shard body -``` - -The prelude must be just large enough to identify the format and locate the -frame, for example magic, format version, header length, checksum, and flags. -After decoding the prelude and frame, the reader computes: - -``` -bodyOffset = preludeLen + headerFrameLen -``` - -EAGERSTABLE still works with this model, but it cannot assume the prototype's -fixed 1024-byte body offset. Eager prefetch, bounded local body reads, and -streaming bitrot readers must use the decoded `bodyOffset`. - -For multipart objects, the embedded direct header is anchored in `part.1` only. -Other S3 part files should remain ordinary shard files unless a future design -adds a direct part index. Requests that need to enter at `part.2` or later, such -as `GET ?partNumber=2` or ranges that start beyond `part.1`, should use the -coalesced `xl.meta` tier of `FastOpenPart`. - -`xl.meta` remains the fallback source of truth, but reading it on the storage -node and appending the shard stream is different from the landing node doing a -separate metadata phase and then a separate shard phase. The coalesced path -optimizes transport shape; the embedded header path optimizes both transport and -disk access shape. - -### Design tasks - -- Replace the fixed positional 1024-byte prototype header with a small fixed - prelude plus variable-length evolvable frame: length-prefixed, versioned, - checksummed, and self-describing. -- Define the decoded header body offset and update Eager/read protocols to use - that offset instead of a constant header length. -- Decide whether the header contains full metadata or a scoped subset plus - fallback flags. -- Define the `CoalescedMetadataFrame` returned by storage-node coalesced open - and prove it can reconstruct existing GET response metadata and `ObjectInfo`. - For Phase 1, this frame must include SSE-S3/SSE-KMS metadata instead of - treating encrypted objects as ineligible, because discovering that encryption - state already required the storage node to read `xl.meta`; doing another - landing-node `xl.meta` read would defeat the coalesced-open goal. - It must also include compression metadata, actual uncompressed size, and any - single-part compression index data needed for the landing node to reconstruct - `ObjectInfo` and run the existing decompression wrapper after erasure decode. -- Define how the landing node merges direct-header results and coalesced - `xl.meta` results when different disks return different tiers for the same - request. -- Define a stable signature over all fields that affect decode, version - identity, response headers, and fallback eligibility. -- Specify exact behavior for over-cap metadata. -- Define how metadata-only mutations update or invalidate `current` and embedded - direct headers: - `PutObjectTags`, `DeleteObjectTags`, `PutObjectMetadata`, legal hold, - retention, replication metadata, restore metadata, and lifecycle metadata. -- Define tests proving FastGet response headers match existing GET behavior for - all supported metadata. - ---- - -## 5. Write, Delete, and Crash Protocol - -### Gaps - -The prototype uses invalidate-before-commit and post-commit shadow install. That -is safe enough for measurement, but production needs a durable `current` publish -protocol with crash recovery and no duplicate latest shard. The existing write -path should still commit `xl.meta` plus the version's `DataDir`; `current` is a -derived latest GET index. - -Open design decisions: - -- whether production `current` is a reserved real `DataDir`, an object-level - pointer to a generated `DataDir`, or a benchmark-gated choice. -- atomic publish protocol for `current`: - - real-directory mode may need a safe rename from `current` to a generated - old-version `DataDir` before publishing the new latest; - - pointer mode needs atomic pointer replacement after the new `DataDir` and - `xl.meta` are durable. -- fsync ordering for shard files, `DataDir` directories, `xl.meta`, `current`, - and staging paths. -- how to publish or invalidate `current` when the latest version is a delete - marker, without serving older data. -- whether failed `current` publish aborts writes or commits `xl.meta` metadata - and lets GET fall back to the coalesced `xl.meta` tier of `FastOpenPart` - until repair rebuilds `current`. -- how to handle platforms where symlink/pointer replacement is unavailable or - unreliable. - -Current delete-marker representation: - -- Delete markers are metadata-only `xl.meta` journal entries, not shard - directories. -- The xl.meta entry has `Type: DeleteType` and an `xlMetaV2DeleteMarker` - containing `VersionID`, `ModTime`, and `MetaSys`. -- When converted to `FileInfo`, a delete marker sets `Deleted: true`, carries the - delete marker `VersionID` and `ModTime`, and exposes `MetaSys` as metadata. -- `MetaSys` is primarily used for internal state such as replication, - version-purge, and tier/free-version metadata. - -Because a delete marker has no `DataDir` and no `part.1`, the initial FastGet -rule should be to remove or invalidate `current` when the latest version is a -delete marker. Latest GET then uses the coalesced `xl.meta` tier of -`FastOpenPart`, reads `xl.meta` on the storage node, sees `Deleted: true`, and -returns the existing delete-marker response. - -Metadata-only header replacement: - -`PutObjectMetadata`, `PutObjectTags`, legal hold, retention, replication -metadata, restore metadata, and lifecycle metadata can change fields that the -embedded direct header would otherwise cache. For latest-version metadata -mutations, treat the operation like a `PutObject` commit that reuses the same -version, `DataDir`, and shard body. Do not rely on loose async repair that only -recreates `current`; that would republish a `part.1` whose embedded header still -contains old metadata. - -1. Lock the object. -2. Read `xl.meta` and pick the target `FileInfo`. -3. Apply the metadata mutation to produce the updated `FileInfo`. -4. Temporarily invalidate `current` so FastGet cannot serve either the old header - or a mixed header/metadata state during the commit. -5. If the updated latest version is no longer eligible for embedded-header - FastGet, commit updated `xl.meta` and leave `current` invalidated. -6. If still eligible, rewrite the embedded `part.1` header synchronously using a - temporary file and the same shard body. -7. Commit updated `xl.meta` with write quorum. -8. Republish `current` only after both the embedded header and `xl.meta` reflect - the same updated `FileInfo`. - -If the embedded header is stored in the source-of-truth `part.1`, do not patch it in -place. Use a staged rewrite: - -``` -/part.1 -/part.1.fastget.tmp -``` - -The synchronous refresh decodes the old header to find `oldBodyOffset`, writes -the new variable header to the temporary file, stream-copies the existing shard -body from `oldBodyOffset`, fsyncs the temporary file, atomically renames it over -`part.1`, and fsyncs the parent directory. - -If the header refresh fails, the safest initial behavior is to fail the -metadata-only operation before publishing `current`. A degraded mode that commits -`xl.meta` metadata and leaves `current` invalidated can be designed later, but it -must never republish `current` until the embedded header matches `xl.meta`. - -This keeps correctness simple but makes metadata-only direct-header refresh a -file rewrite. A separate derived file would make refresh cheaper but would -reintroduce data duplication, so it is not the initial direction. - -### Design tasks - -- Specify the write-state machine for: - - first PUT of a key; - - overwrite in non-versioned bucket; - - overwrite in versioned bucket; - - overwrite from eligible to ineligible; - - multipart complete; - - copy object; - - delete marker creation; - - permanent delete by version ID. -- Specify crash states and deterministic recovery actions for every state. -- Define stale-read prevention invariants. `xl.meta` remains authoritative, but the - `current` alias must never serve an older committed version as latest. -- Define delete-marker `current` invalidation and repair behavior. -- Define `current`/header publish rules: eligible writes publish a fresh header - generated from the committed `FileInfo`; ineligible writes and metadata-only - mutations invalidate or rebuild `current`; scanner repairs stale `current` - aliases only after verifying that the existing embedded `part.1` header - matches `xl.meta`. -- Define synchronous metadata-only header refresh and its failure behavior. -- Define a FastGet repair queue or scanner hook for stale/missing `current` - discovered outside the locked metadata mutation path. -- Add design-level proof that FastGet's matching-header read quorum is safe - under offline-disk rejoin. The proof should rely on normal erasure quorum - intersection: once a newer version is committed to write quorum, stale - `current` aliases from disks that missed the write cannot form a disjoint read - quorum for the older version. Mixed old/new headers must not be combined, and - scanner/repair only shortens the stale-`current` window; it should not be - required for correctness when stale aliases are below read quorum. - ---- - -## 6. Read Protocol - -### Gaps - -The prototype evolved from "open all to EOF once" into EAGERSTABLE: open exactly -the quorum, use stable spread selection, prefetch local first blocks, and use -bounded body reads for local multi-block eager. This is better for the measured -candidate. The production read protocol should keep the tight initial open set -and add deterministic before-first-byte replacement plus explicit cancellation. - -Recommended read/open order: - -1. Select only disks where `disk != nil && disk.IsOnline()`, matching the - existing GET metadata read path. -2. Open the maximum configured read quorum for the erasure set, with no initial - hedge. This should cover the largest possible data-block count across - configured storage classes and write modes. For example, in a 16-drive set, - if the lowest configured parity is 2, the initial open count is 14. If data - and parity can be equal, apply the existing strict-majority rule. -3. Give every `FastOpenPart` call its own cancellable child context. -4. Ask each selected disk for `FastOpenPart`. -5. For latest GET that can enter through `part.1`, the disk first tries the - embedded direct path: `current/part.1`. -6. If direct open fails before streaming, the disk may fall back locally to the - coalesced `xl.meta` tier: read `xl.meta`, select the version and - data directory, then return a metadata frame followed by the shard stream. -7. The landing node applies the existing Buckit metadata quorum, erasure - distribution ordering, and decode semantics. FastGet must not invent a new - metadata matching or shard scoring policy. -8. If the initial selected set fails before the first response byte, open all - remaining online disks as replacements, then apply the same existing Buckit - quorum/decode semantics over the combined responses. -9. After a winning read set is selected, close and cancel every unused stream - immediately. Do not drain unused remote streams. -10. If the client disconnects, or FastOpen exits before response bytes are sent, - cancel all outstanding child contexts so remote storage reads stop promptly. -11. After the first response byte is committed, do not switch to a semantic - fallback path. Surface errors through the existing read/decode behavior and - cancel all remaining streams on exit. - -Explicit `versionId` GET and delete-marker responses should use the coalesced -`xl.meta` `FastOpenPart` tier starting in Phase 1. Requests that enter at -non-`part.1` multipart data should not require `current` or a direct embedded -header; they can stay on the current non-FastGet implementation until multipart -FastGet scope is added. - -The embedded direct tier is the performance ceiling. The coalesced `xl.meta` tier -is the compatibility and fidelity tier. Both should use the same shard selection, -decode, cancellation, and before-first-byte fallback framework where possible. - -### Design tasks - -- Factor the existing Buckit metadata quorum and erasure decode selection so - `FastOpenPart` responses feed the same logic as current GET. -- Define the framed `FastOpenPart` response shape and body modes. -- Define tests for initial selected-set failure, replacement opens, unused-stream - cancellation, client disconnect, and post-first-byte stream errors. - ---- - -## 7. Versioning - -### Gaps - -The prototype disables FastGet for versioned and version-suspended buckets. The -initial full-feature design should support versioned single-part GET through the -coalesced `xl.meta` route, including latest-version GET, explicit old-version -GET, and delete-marker responses. - -Design constraints: - -- Metadata is version-scoped in `xl.meta`. -- Tags and object-lock state can be updated for an old version. -- A plain GET without `versionId` must see only the latest version. -- A latest delete marker must return the same error and headers as existing GET - behavior. -- `GET ?versionId=` must not read metadata from a newer version. -- Only mutations that change the latest visible version should publish, refresh, - or invalidate `current`. -- Mutations to non-latest versions should update `xl.meta` and leave - `current` untouched. - -### Design tasks - -- Define latest-version direct lookup through `current`. -- Define explicit `versionId` GET through the coalesced `xl.meta` tier of - `FastOpenPart`. -- Define old-version metadata mutation behavior: update `xl.meta` for that - `VersionID`, do not refresh `part.1`, and do not touch `current`. -- Define how a new version publishes or invalidates `current` without changing - the old-version `DataDir` contract. -- Define how permanent delete removes or quarantines recorded `DataDir` - directories. If permanent delete changes the latest visible version, the - simplest initial rule is to invalidate `current`; scanner/repair may rebuild - it later after verifying the exposed latest version and header. -- Define versioned bucket tests covering tags, user metadata, object lock, - delete markers, null versions, and permanent delete. - ---- - -## 8. Multipart and Range - -### Discussed direction - -Embed the direct header only in `part.1`: - -``` -current/part.1 # direct header + shard body -current/part.2 # shard body only -current/part.3 # shard body only -``` - -This avoids duplicating object metadata into every S3 part and limits -metadata-only mutation work. Full-object multipart GET can use the `part.1` -header as the direct metadata anchor, then stream remaining parts from the same -latest `current` alias or recorded `DataDir`. - -Requests whose first required shard is not `part.1` should use coalesced -`xl.meta` open unless a later design adds a direct multipart index. That includes -`GET ?partNumber=N` for `N > 1` and ranges that start in a later S3 part. - -The initial Phase 1 implementation does not need to support multipart objects or -range GET. Multipart GET, `partNumber`, and ranges can stay on the existing -non-FastGet implementation until the single-part `FastOpenPart` protocol is -proven. - -When multipart support is added, keep existing landing-node range planning. The -landing node already converts `Range` and `partNumber` into logical offsets, -maps those offsets to physical `part.N` spans, and decodes each physical part. -The multipart extension should replace each per-part remote shard open with one -`FastOpenPart` call for that physical part span. A range crossing multiple -multipart parts may issue multiple `FastOpenPart` calls. That is acceptable -because multipart parts are large, so the extra open cost is negligible compared -with transfer and decode time, and it keeps the remote storage node from needing -a cross-part object-range planner. - -### Design tasks - -- Define what the `part.1` direct header must carry for full-object multipart - GET without reading `xl.meta`. -- Define the `FastOpenPart` per-part call contract for logical ranges that map - to multiple physical `part.N` spans. -- Define which Phase 2 direct-header multipart/range requests must use - coalesced `xl.meta` open because they do not enter through `part.1`. -- Decide whether a future direct multipart index is needed for non-`part.1` - entrypoints. -- Define behavior for checksum-mode on full-object vs. range reads. -- Define tests for ranges crossing part boundaries and for `partNumber` GET. - ---- - -## 9. Pool Routing and Cluster Topology - -### Gaps - -Current FastGet only avoids `xl.meta` in the single-pool path. Multi-pool -deployments still call pool-resolution code that reads metadata before the set -FastGet path can run. - -### Discussed direction - -Keep the current metadata-based pool selection for correctness, but reduce the -initial metadata fanout per pool with a quorum-intersection probe. - -For each pool's hashed set, probe: - -``` -safeProbeCount = N - writeQuorum + 1 -``` - -where `N` is the set drive count and `writeQuorum` is the minimum write quorum -for object layouts supported in that set. This count intersects any committed -version's write quorum. For example, in a 16-drive set with write quorum 12, -at most 4 drives can be stale/missing for a committed version, so probing 5 -drives is enough to see that committed version if it exists. - -Pool selection flow: - -1. Probe `safeProbeCount` stable-spread drives in each pool's hashed set. -2. If exactly one pool reports a candidate and every other pool is proven absent - by the probe, select that pool. -3. If multiple pools report candidates, run full `xl.meta` pool arbitration. -4. If any pool returns ambiguous results, errors, or insufficient proof of - absence, expand that pool or run full `xl.meta` pool arbitration. -5. After selecting a pool, run `FastOpenPart` in that pool. - -This reduces the common single-owner case without requiring cross-pool -`FastOpenPart` arbitration. It does not change GET availability: a 16-drive set -that has lost 6 drives can only serve objects whose actual read quorum can be -satisfied by the remaining 10 drives. - -### Design tasks - -- Define safe-probe pool selection and the expansion rules for ambiguity, - multiple candidates, delete markers, versioned requests, and pool errors. -- Decide how to handle objects that may exist in multiple pools during rebalance - or pool expansion. -- Define interaction with decommission, rebalance, site replication, and - data-movement code. -- Add multi-pool correctness and performance tests. - ---- - -## 10. Storage API and Transport - -### Gaps - -The prototype reuses `ReadFileStream`, `CreateFile`, `RenameFile`, and `Delete`. -Production likely needs `FastOpenPart` for reads and must extend existing -commit-style storage mutations so they update variable headers and `current` -atomically inside the storage node. - -### Design tasks - -- Decide whether to add storage APIs for: - - `FastOpenPart` that can return either an embedded direct header plus shard - stream or a coalesced metadata frame plus shard stream; - - extensions to existing mutation APIs such as `RenameData`, `UpdateMetadata`, - and delete-version paths so header refresh and `current` publish/invalidate - happen atomically inside each storage node; - - `WritePrefixedShard` or equivalent local helper to write variable-header - `part.1` without cross-node shadow copy; - - direct-path header read plus paused body stream, if this is not folded into - `FastOpenPart`; - - `current` reconcile/repair; - - delete/invalidate `current` for latest delete markers, as part of the delete - storage mutation. -- Define REST/grid wire behavior and cancellation semantics. -- Define how local `xlStorage` and remote `storageRESTClient` implementations - differ, if at all. -- Define capability detection for symlink/pointer support, variable direct - headers, and mixed nodes. - ---- - -## 11. Repair, Scanner, and Migration - -### Gaps - -The prototype has no scanner or repair path for `current`. In the full design, -objects without a valid `current` alias or embedded direct header can still use -the coalesced `xl.meta` tier of `FastOpenPart`, but stale `current` must not -serve old latest data. - -Scanner/request-time repair should only reconcile the `current` alias. It should -not create or rewrite embedded `part.1` headers. Header creation and refresh are -owned by write-side mutation paths such as PUT, multipart complete, copy, and -latest-version metadata updates. If scanner finds a latest `DataDir` whose -`part.1` header is missing, stale, or invalid, it must leave `current` absent and -let GET use the coalesced `xl.meta` tier. - -### Design tasks - -- Define request-time repair triggers when FastGet falls back. -- Define disk-local reconciliation of `current` against `xl.meta` and the - latest version's `DataDir`. -- Define scanner migration for existing objects as optional installation or - repair of `current` only when the latest `DataDir` already has a valid - embedded `part.1` header matching `xl.meta`. -- Define scanner cleanup for stale `current`, staging directories, orphan - `DataDir`s, and uncommitted direct data. -- Define repair priority and throttling so FastGet repair does not starve normal - healing. -- Define metrics for direct-path health: present, missing, stale, repaired, - quarantined, and migrated. - ---- - -## 12. Security and Correctness - -### Gaps - -The embedded direct tier reconstructs `ObjectInfo` without `xl.meta`; the -coalesced `xl.meta` tier reconstructs it from `xl.meta` on the storage node. The -full design must prove that neither tier can bypass checks performed by the -existing metadata path. - -### Design tasks - -- Audit every GET decision that depends on `ObjectInfo` or `FileInfo`. -- Ensure FastGet preserves: - - IAM and bucket policy behavior; - - SSE-C/SSE-S3/SSE-KMS behavior; - - object lock retention/legal hold filtering; - - lifecycle expiry and transition behavior; - - replication proxy behavior; - - conditional request behavior; - - checksum response behavior; - - delete-marker errors and headers. -- Define what must be in the direct header vs. what forces fallback. -- Define fuzz/corruption tests for malformed headers, signature collisions, - mismatched metadata, duplicated erasure indexes, and path/header disagreement. - ---- - -## 13. Test and Validation Plan - -### Required design outputs - -- A correctness matrix covering request type, bucket versioning state, object - type, metadata type, and expected FastGet/fallback behavior. -- A crash-state matrix covering every write/delete stage and recovery result. -- A migration matrix covering old objects, mixed object layouts, and mixed - software versions. Mixed object layouts here means objects with and without - `current` aliases or variable direct headers, not a new dedicated version - directory layout. -- A performance matrix covering HDD, SSD/NVMe, local single-pool, distributed - single-pool, and multi-pool. - -### Required test groups - -- Header encode/decode compatibility and corruption tests. -- End-to-end GET byte and header equality tests. -- Versioned object tests for tags, metadata, delete markers, and old-version GET. -- Multipart/range tests for the later multipart FastGet extension. -- Metadata mutation tests. -- Crash/fault-injection tests around `DataDir` commit, `current` publish, pointer - replacement, invalidation, and fsync points. -- Offline-disk rejoin and scanner repair tests. -- Multi-pool routing tests. -- HDD load tests that compare FastGet with existing GET under cold and warm - cache conditions. - ---- - -## 14. Suggested Design Sequence - -### Phase 1: FastOpenPart with coalesced xl.meta frames - -Goal: improve the transport shape without changing the on-disk object layout. - -- Add `FastOpenPart` as a framed response stream: - `CoalescedMetadataFrame`, then body bytes when the selected object has a - FastOpen-streamable local body. -- On the storage node, read `xl.meta`, select the requested single-part object - version or latest visible version, then return the selected version metadata - before any body bytes. For local shard-backed objects, open the selected - version's `DataDir` shard and stream it after the frame. For metadata-only or - non-shard cases, the frame identifies the case and the body stream is omitted - or handled by the landing node through the existing path. -- Support all non-range, non-multipart GET cases through this tier in Phase 1, - including latest GET, explicit `versionId` GET, versioned/suspended buckets, - null versions, and delete-marker responses. -- Support inline objects in Phase 1. Since the storage node already reads - `xl.meta`, inline data can be returned through the coalesced frame/body path - without a local shard open. -- Support object-lock metadata in Phase 1. The frame must carry retention and - legal-hold metadata, and the landing node must run the existing permission - filtering before setting response headers. -- Support `x-amz-checksum-mode: ENABLED` for non-range Phase 1 GET. The frame - must carry checksum metadata so the landing node can run the existing checksum - response-header logic. Range checksum behavior remains deferred with range - FastGet. -- Support non-STANDARD storage classes in Phase 1 when the selected object is - still local or restored locally. The frame must carry the actual storage class - and erasure layout; the landing node must use the returned `ErasureM` instead - of assuming the default class. -- Support SSE-S3 and SSE-KMS for single-part GET in Phase 1. The remote storage - node streams encrypted shard bytes; the landing node reconstructs `ObjectInfo` - from `CoalescedMetadataFrame`, runs existing encryption validation/response - header logic, erasure-decodes, then decrypts with the existing landing-node - stream path. -- Support compressed single-part full-object GET in Phase 1. The remote storage - node streams stored compressed shard bytes; the landing node reconstructs - `ObjectInfo` from `CoalescedMetadataFrame`, erasure-decodes, then uses the - existing decompression path. Compressed range and compressed multipart GET stay - out of scope until range/multipart FastGet is added. -- Support restore/transition metadata in Phase 1. Restored-on-disk objects can - use the local shard/inline path while preserving `x-amz-restore` headers. - Remote-tier objects that are not restored locally should return a coalesced - metadata frame without a shard stream so the landing node can use the existing - transitioned-object reader; FastOpen must not try to open a missing local - `DataDir` shard for them. -- Keep SSE-C out of Phase 1. It is request-header/customer-key sensitive and can - be rejected before FastOpen from request headers. -- Use maximum-configured-read-quorum initial drive selection over online disks, - with no initial hedge. If the selected set fails before the first response - byte, open all remaining online disks as replacements and then apply existing - Buckit quorum/decode semantics. -- Give every `FastOpenPart` stream its own cancellable child context. Cancel - unused streams immediately after selecting the winning read set, and propagate - client disconnect/fallback cancellation to every outstanding remote read. -- Keep current multi-pool metadata selection, with the safe-probe optimization as - a separate pool-selection improvement. - -Phase 1 should establish the framed stream protocol, cancellation semantics, -metadata fidelity against existing GET behavior, selected-drive fallback, and -parity reconstruction without introducing `current` or embedded direct headers. - -### Phase 2: current symlink plus embedded part.1 header - -Goal: add true no-`xl.meta` latest GET for eligible objects while preserving -the Phase 1 coalesced `xl.meta` tier as fallback. - -- Keep `xl.meta` plus generated `DataDir` directories as the durable layout. Do - not add `versions/` directories. -- Materialize `current` as an object-level pointer/symlink to the latest - generated `DataDir`. -- Store a variable direct header only in `part.1`: - small fixed prelude, variable metadata frame, then shard body. -- `FastOpenPart` tries `current/part.1` first for eligible latest GET. - If the direct header is missing, stale, invalid, or ineligible, the storage - node falls back to the Phase 1 coalesced `xl.meta` frame. -- Update write-side mutations to maintain the direct header and `current`: - - `PutObject`, `CompleteMultipartUpload`, and `CopyObject` write eligible - `part.1` headers from committed `FileInfo` and publish or invalidate - `current`. - - latest delete markers invalidate/remove `current`. - - permanent delete invalidates `current` if it changes the latest visible - version. - - latest metadata-only mutations synchronously refresh `part.1` using the same - `DataDir` and shard body, update `xl.meta`, and republish or invalidate - `current`. - - old-version metadata mutations update only `xl.meta` and do not touch - `current`. -- Scanner/request-time repair only reconciles the `current` alias when the - latest `DataDir` already has a valid embedded `part.1` header. Scanner must - not create or rewrite embedded headers. -- Prove quorum safety: matching direct headers form the read quorum, mixed - versions are not combined, and read/write quorum intersection prevents stale - `current` aliases below read quorum from serving an older committed version. - -Phase 2 is the main production FastGet feature. - -### Phase 3: optional real current directory - -Goal: benchmark whether avoiding symlink resolution is worth the additional -write/crash complexity. - -- Materialize `current` as a reserved real latest `DataDir` instead of a symlink - to a generated `DataDir`. -- Preserve the same `FastOpenPart` response stream and Phase 1 coalesced fallback. -- Preserve the same variable `part.1` header format from Phase 2. -- Design the overwrite protocol that safely turns the previous real `current` - directory into a generated old-version `DataDir` before publishing the new - latest. -- Compare real-directory and symlink modes for PUT cost, crash recovery, - scanner behavior, platform support, cold-cache GET latency, and warmed-cache GET - latency. - -Phase 3 should remain optional unless benchmarks show that symlink resolution is -a material bottleneck or platform constraints require a non-symlink mode. diff --git a/docs/single-trip-get-phase1-fastopen-plan.md b/docs/single-trip-get-phase1-fastopen-plan.md deleted file mode 100644 index 7c704ea92..000000000 --- a/docs/single-trip-get-phase1-fastopen-plan.md +++ /dev/null @@ -1,770 +0,0 @@ -# FastGet Phase 1 FastOpen Implementation Plan - -**Status:** implementation plan for full-feature Phase 1 - -**Companion docs:** - -- `docs/single-trip-get-full-feature-gaps.md` -- `docs/single-trip-get-phase1-implementation.md` for prototype history only - ---- - -## 1. Goal - -Implement Phase 1 FastGet as a coalesced `xl.meta` FastOpen path: - -``` -FastOpenPart -> CoalescedMetadataFrame -> optional body stream -``` - -The goal is to remove the separate landing-node metadata fan-out before eligible -GET reads while preserving existing GET response bytes, headers, errors, -authorization behavior, version semantics, and erasure-read correctness. - -Phase 1 must not introduce `current`, embedded `part.1` headers, symlinks, real -`current` directories, scanner repair, or write-path layout changes. - ---- - -## 2. Scope - -### Supported - -- GET only. HEAD remains on existing `GetObjectInfo`. -- All non-range, non-multipart GET cases. -- Latest GET, explicit `versionId` GET, versioned/suspended buckets, null - versions, and delete-marker responses. -- Single-part local shard-backed objects. -- Inline objects, with inline bytes streamed after the frame. -- Zero-byte objects, with metadata frame and no required body stream. -- Restored-on-disk objects, preserving `x-amz-restore`. -- Remote-tier objects not restored locally, with metadata frame only and existing - transitioned-object reader on the landing node. -- Object-lock metadata, permission-filtered on the landing node. -- `x-amz-checksum-mode: ENABLED` for non-range GET. -- Non-STANDARD storage class when the object is local or restored locally. -- SSE-S3 and SSE-KMS, with encrypted shard bytes streamed and existing - landing-node decrypt path preserved. -- Compressed single-part full-object GET, with existing landing-node - decompression path preserved. - -### Out Of Scope - -- HEAD. -- SSE-C. -- Range GET. -- `partNumber`. -- Multipart object body FastOpen. -- Multi-pool safe-probe optimization. -- Mixed-version or rolling-upgrade FastOpen compatibility. Phase 1 assumes the - cluster is running a homogeneous FastOpen-capable binary. -- `current` path, embedded direct headers, write-path publish/invalidation, and - scanner repair. - -Out-of-scope requests use the current non-FastGet implementation. -On multi-pool deployments, existing pool/owner resolution remains unchanged and -still happens before FastOpen. The true no-extra-`xl.meta` win for Phase 1 is the -single-pool path; multi-pool may still benefit after pool selection, but the -pool-selection metadata fan-out is not optimized in this phase. - ---- - -## 3. Core Design Rules - -- Reuse existing Buckit GET semantics. Do not invent a new metadata quorum, - version selection, erasure ordering, decode, auth, object-lock, checksum, - encryption, compression, or lifecycle behavior. -- The storage node may read `xl.meta` and open/stream local body bytes, but the - landing node remains responsible for request-context behavior such as auth, - preconditions, object-lock permission filtering, checksum headers, encryption - response headers, decryption, decompression, and final HTTP response shaping. -- The frame carries a compact GET-only metadata shape. It must be sufficient to - reconstruct the minimal `FileInfo`/`ObjectInfo` state used by existing GET, but - it must not carry `FileInfo` fields that are only used by write, list, repair, - or metadata-mutation paths. -- FastOpen failures before the first response byte are handled inside FastOpen by - opening all remaining online disks as replacements. Do not fall back to a - second landing-node metadata fan-out for supported Phase 1 requests. -- After the first response byte is committed, do not switch semantic paths. - Surface errors through existing read/decode behavior. -- Body bytes in shard and inline modes are per-disk encoded data. The landing - node must wrap selected body streams in the same bitrot-verifying decode path - before exposing any bytes to `NewGetObjectReader`. - ---- - -## 4. Interfaces And Data Types - -This section is the review boundary before coding. Names are proposed Go names; -field layout can be adjusted during implementation, but the semantics should be -settled here first. - -### Storage Interface - -Add a streaming storage operation alongside existing `ReadVersion` and -`ReadFileStream`: - -```go -type StorageAPI interface { - FastOpenPart(ctx context.Context, volume, object string, req FastOpenPartRequest) (io.ReadCloser, error) -} -``` - -The returned stream starts with a `FastOpenFramePrelude`, followed by an encoded -`CoalescedMetadataFrame`, followed by body bytes only when `BodyMode` requires a -local body stream. Errors returned from `FastOpenPart` happen before any frame is -available. Per-object semantic results such as delete marker are represented in -the frame, not as transport errors. - -### Request - -```go -type FastOpenPartRequest struct { - Version uint16 - VersionID string - PartNumber int - Offset int64 - Length int64 - Flags FastOpenPartFlags -} -``` - -Request semantics: - -- `Version` is the request protocol version. -- `VersionID == ""` means latest visible version. -- Phase 1 uses `PartNumber == 1` for shard-backed body streams. -- Initial Phase 1 body reads use `Offset == 0`, `Length == -1`. -- For shard-backed replacement reads, `Offset` and `Length` are encoded - part-file byte coordinates for the body bytes after the frame, not object - byte coordinates. `Offset` may be nonzero to open a new forward-only body - stream at an encoded shard-body offset. `Length == -1` means to EOF; a bounded - length may be used as an optimization. -- Replacement offsets are only for erasure-decode replacement. They do not make - range GET or `partNumber` in scope. -- Range, `partNumber`, multipart, and SSE-C requests are rejected before - FastOpen by the landing node. -- Future phases may add direct/current-path flags, but Phase 1 is coalesced - `xl.meta` only. - -### Prelude - -```go -type FastOpenFramePrelude struct { - Magic [4]byte - Version uint16 - HeaderLen uint32 - HeaderCRC32 uint32 - BodyMode FastOpenBodyMode -} -``` - -Prelude rules: - -- The prelude is fixed-size and appears at byte 0 of every successful - `FastOpenPart` stream. -- `Magic` is `"BFG1"`; the `Version` field carries the protocol version. -- `HeaderLen` is the byte length of the encoded `CoalescedMetadataFrame`. -- `HeaderCRC32` covers only the encoded frame bytes, not body bytes. -- Unknown `Version` or invalid checksum is a FastOpen failure before client body - bytes are sent. - -### Body Mode - -```go -type FastOpenBodyMode uint8 - -const ( - FastOpenBodyShard FastOpenBodyMode = iota + 1 - FastOpenBodyInline - FastOpenBodyMetadataOnly - FastOpenBodyTransitioned -) -``` - -Body mode semantics: - -- `FastOpenBodyShard`: frame followed by this disk's encoded shard bytes from - `DataDir/part.1`. -- `FastOpenBodyInline`: frame followed by this disk's encoded inline body bytes - from `xl.meta`; the landing node decodes it through the same path as shard - mode. -- `FastOpenBodyMetadataOnly`: frame only. Used for delete markers and zero-byte - objects. -- `FastOpenBodyTransitioned`: frame only. Landing node uses existing - transitioned-object reader. - -### Coalesced Metadata Frame - -```go -type CoalescedMetadataFrame struct { - Status FastOpenFrameStatus - Meta FastOpenGETMeta - BodyMode FastOpenBodyMode - BodyLen int64 -} - -type FastOpenGETMeta struct { - VersionID string - IsLatest bool - Legacy bool - ModTimeUnixNano int64 - Size int64 - Metadata map[string]string - Transition FastOpenTransitionMeta - Part FastOpenPartMeta - Erasure FastOpenErasureMeta - Checksum []byte - NumVersions int - SuccessorModTimeNanos int64 -} - -type FastOpenTransitionMeta struct { - Status string - Object string - Tier string - VersionID string -} - -type FastOpenPartMeta struct { - Number int - Size int64 - ActualSize int64 -} - -type FastOpenErasureMeta struct { - DataBlocks int - ParityBlocks int - BlockSize int64 - Index int - Distribution []int - Bitrot FastOpenBitrotMeta -} - -type FastOpenBitrotMeta struct { - PartNumber int - Algorithm uint8 - Hash []byte -} -``` - -Frame rules: - -- `Meta` describes one selected object version on one disk, after the storage - node has read `xl.meta` and applied the existing latest-version or explicit - `versionId` selection. -- `Meta` is not raw `FileInfo`. It carries only fields needed by GET selection, - response metadata, erasure decode, and bitrot verification. -- The landing node converts `Meta` into minimal `FileInfo` values before calling - existing metadata quorum, erasure ordering, and decode helpers. This keeps - quorum behavior aligned with current GET while avoiding full `FileInfo` wire - size. -- `Metadata` is the raw, unfiltered metadata map from the selected `FileInfo`. - The storage node must not call `cleanMetadata`, strip reserved keys, or apply - object-lock permission filtering. The landing node reconstructs - `ReplicationState` with `getInternalReplicationState(Metadata)`, then performs - the same cleaning, object-lock filtering, encryption, compression, restore, - storage-class, tag, ETag, and checksum handling as canonical GET. -- `Part` contains exactly the selected single part for Phase 1. Multipart body - paths, range GET, and `partNumber` are out of scope, so the frame does not - carry a full multipart part list. -- `Checksum` is the object checksum blob used by `x-amz-checksum-mode` and by - encrypted checksum handling. -- `NumVersions` and `SuccessorModTime` are carried because current GET response - header logic feeds `ObjectInfo.ToLifecycleOpts()` into lifecycle prediction - headers. -- `BodyLen` is the number of bytes following the frame for `shard` and - `inline` body modes. It is `0` for metadata-only and transitioned modes. -- `Status` distinguishes normal object, delete marker, not found/version not - found, and unsupported/capability cases when those should be represented as a - frame. -- `Status` is the authoritative delete-marker signal. During frame-to-`FileInfo` - conversion, `FileInfo.Deleted` is set only when - `Status == FastOpenStatusDeleteMarker`. Version-purge state is reconstructed - separately from raw metadata into `ReplicationState` and - `VersionPurgeStatus`. -- Wire fields use stable protocol encodings, not Go implementation details: - times are Unix nanoseconds, `FastOpenBitrotMeta.Algorithm` is a stable - FastOpen bitrot algorithm code mapped to/from Buckit's `BitrotAlgorithm`, and - enum values are protocol constants. -- Reconstruct `ModTimeUnixNano` and `SuccessorModTimeNanos` with - `time.Unix(0, n).UTC()` so quorum `time.Time.Equal` comparisons match - canonical metadata reads and carry no monotonic clock component. -- FastOpen bitrot algorithm codes are: - - `1`: `SHA256` - - `2`: `HighwayHash256` - - `3`: `HighwayHash256S` - - `4`: `BLAKE2b512` - Unknown algorithm codes are fail-safe: treat the frame as unsupported/corrupt - for quorum selection and never attempt decode with a guessed algorithm. -- `FastOpenBitrotMeta.Hash` is equivalent to existing `ChecksumInfo.Hash` for - the selected part. For streaming bitrot, per-block hashes remain in the body - stream and this hash may be empty, matching canonical verifier setup. -- The frame must not carry `Volume`, `Name`, `DataDir`, inline `Data`, full - multipart `Parts`, `Mode`, `WrittenByVersion`, `MarkDeleted`, `Fresh`, `Idx`, - or `Versioned`. The landing node already knows bucket/object/versioning - context, Phase 1 streams body bytes after the frame, and these fields are not - needed for GET. -- For the plain single-part objects used in the performance tests, the target - frame size is approximately the selected-version `xl.meta` size plus small - framing overhead. With observed `xl.meta` around 368-384 bytes, the compact - frame should remain in the same range rather than growing to a multi-KB - payload. This is not a hard bound; metadata size is data-dependent. - -### Status - -```go -type FastOpenFrameStatus uint8 - -const ( - FastOpenStatusOK FastOpenFrameStatus = iota - FastOpenStatusDeleteMarker - FastOpenStatusNotFound - FastOpenStatusVersionNotFound - FastOpenStatusUnsupported -) -``` - -Status rules: - -- Transport/storage failures before frame creation return `error`. -- Delete marker is a valid frame status so the landing node can return existing - delete-marker behavior with metadata. -- `FastOpenStatusNotFound` and `FastOpenStatusVersionNotFound` are disk-local - results. They do not immediately become a 404; the landing node applies the - same not-found/read-quorum logic as canonical GET, after any required - replacement opens. -- Unsupported means the request or selected object cannot be served by - FastOpen; the landing node may use current non-FastGet implementation only if - the request type is out of Phase 1 scope or capability is missing. Supported - Phase 1 requests should exhaust online-disk FastOpen replacement before - returning an error. - -### Encoding Choice - -Use a small fixed prelude plus msgp-encoded frame payload unless implementation -review finds a strong reason to use another existing Buckit codec. The frame -must be versioned and checksummed from the first implementation so future direct -headers can reuse the framing shape. - ---- - -## 5. Storage Node Work - -Add local `xlStorage` support for `FastOpenPart`: - -1. Read `xl.meta`. -2. Select latest visible version or explicit `versionId` using existing - `xl.meta`/`FileInfo` helpers. -3. Build compact `FastOpenGETMeta` from the selected `FileInfo`, omitting fields - that are not used by GET. Preserve the raw metadata map exactly as canonical - `ToFileInfo` would expose it. -4. Decide body mode: - - delete marker: metadata-only; - - zero-byte: metadata-only; - - inline: inline body; - - transitioned and not restored: transitioned metadata-only; - - local shard-backed: open selected `DataDir/part.1` and stream shard bytes. -5. For shard-backed replacement requests with nonzero `Offset`, return the same - compact frame followed by this disk's encoded shard bytes starting at the - requested encoded part-file byte offset. The returned body is still a - forward-only stream. `BodyLen` is the number of body bytes following the - frame for this offset read: either the remaining encoded body length or the - requested bounded length. -6. Reject unsupported offsets before body bytes. Offset support is required only - for encoded shard-body replacement streams, not inline objects, metadata-only - statuses, range GET, multipart, or `partNumber`. -7. Return errors before any body bytes when metadata cannot be read or the - version cannot be selected. - -Add remote storage transport support through the existing storage REST/grid -pattern. The final endpoint choice can follow the local code ownership and -streaming constraints, but it must support request-context cancellation. - -Storage timeout behavior must follow the current GET path: - -- `xl.meta` read and frame construction are bounded by the same drive timeout - used by `ReadXL`/`ReadVersion` (`globalDriveConfig.GetMaxTimeout()` through - the existing storage wrappers). -- Body stream open/read follows `ReadFileStream` plus existing bitrot/decode - behavior. Phase 1 must not add a new per-body-read timeout policy. -- FastOpen adds cancellation for streams it opened before final shard selection. - This cancellation is for abandoning unused/replaced streams, not for changing - canonical body-read timeout semantics. - ---- - -## 6. Landing Node Work - -Add a Phase 1 FastOpen path in `erasureObjects.GetObjectNInfo` before the -current `getObjectFileInfo` fan-out for eligible requests. - -Eligibility before FastOpen: - -- FastGet enabled; -- GET path only; -- not SSE-C request; -- no range; -- no `partNumber`; -- request is not replication/proxy special case. - -FastOpen read flow: - -1. Select only disks where `disk != nil && disk.IsOnline()`. -2. Open maximum configured read quorum over online disks, with no hedge. -3. Give every `FastOpenPart` call its own cancellable child context. -4. Convert returned `FastOpenGETMeta` frames into minimal `FileInfo`/metadata - arrays used by current Buckit GET selection. - - Rebuild `ReplicationState` from raw metadata with - `getInternalReplicationState`. - - Rebuild `Parts` as a one-entry slice from `FastOpenPartMeta`. - - Rebuild inline state, free-version state, encryption/compression state, - restore state, tags, ETag, and object-lock metadata from the raw metadata - map, matching canonical `FileInfo.ToObjectInfo`. -5. Apply existing Buckit metadata quorum, erasure distribution ordering, and - decode semantics. -6. If the initial selected set fails before first response byte, open all - remaining online disks as replacements and reapply the same selection logic. -7. Cancel every unused stream immediately after selecting the winning read set. -8. For shard body mode, wrap selected encoded shard streams in a streaming - bitrot-verifying `ReaderAt`, erasure-decode with the existing decode path, - and wrap with existing `NewGetObjectReader` behavior. -9. For inline body mode, use the same bitrot-verifying erasure-decode path as - shard body mode. -10. For metadata-only delete marker/zero-byte, return existing GET behavior. -11. For transitioned mode, call existing transitioned-object reader using the - reconstructed `ObjectInfo`. - -No HTTP body bytes are committed until FastOpen has selected a metadata/body -quorum and the first decode attempt can start from offset 0. If the initial -selected set fails during frame processing or block-0 decode before any client -body byte is written, FastOpen opens all remaining online disks and restarts -selection/decode from offset 0. After the first client body byte is written, -FastOpen follows existing stream/decode error behavior, including same-offset -erasure replacement where possible. - -### Mid-Stream Replacement - -FastOpen must preserve canonical GET availability for post-commit shard -failures. A single disk stream that becomes missing, corrupt, or unreadable -after response bytes have started must not fail the customer GET if the erasure -set still has enough healthy shards to reconstruct the object. - -Canonical GET achieves this because each bitrot reader is an `io.ReaderAt` that -opens its underlying `ReadFileStream` lazily. When the erasure decoder first -needs a spare shard at shard offset `N`, the reader opens that disk's part stream -directly at the encoded bitrot offset for `N`, then reads forward from there. - -FastOpen should use the same model: - -1. Initial selected readers are normal FastOpen streams opened at body offset 0. -2. Spare readers are lazy replacement readers. They do not need an opened body - stream before decode starts. A spare may reuse frame metadata if that disk was - already contacted as validation context, or it may fetch frame and body - together on first use. A replacement offset-read frame is always decoded and - validated; cached frame metadata never bypasses validation. -3. A lazy replacement reader holds only the information needed to open and - validate a later replacement read: disk, disk index, object identity, version - ID, selected object metadata identity, selected erasure layout, bitrot - algorithm, and body-length expectations. -4. On first `ReadAt(buf, shardOffset)`, a lazy replacement reader computes the - encoded body offset: - - ``` - bodyOffset = (shardOffset / shardSize) * hashSize + shardOffset - ``` - - It then calls `FastOpenPart` for the same object/version/part with - `Offset == bodyOffset` and a bounded or to-EOF `Length`, validates the - returned frame against the already selected object version and erasure - layout, and reads forward from the returned body stream. -5. Replacement validation is a hard gate. A frame is rejected before reading its - body if any required property differs from the winning object selection: - version identity (`VersionID`, or canonical null-version identity using - `ModTime`/ETag), the same winning-version filters used by initial FastOpen - selection (`IsValid`, selected `ModTime` or ETag, and `Erasure.Equal` over - `DataBlocks`, `ParityBlocks`, `BlockSize`, and `Distribution`), per-disk - `Erasure.Index`, slot availability at `Erasure.Index - 1`, part - number/size/actual size, body mode, body length expectations, and bitrot - algorithm. `Erasure.Equal` is not sufficient by itself because it does not - validate the per-disk shard index. -6. The replacement stream is still forward-only after open. It is not a random - access handle. The random-start behavior is provided by opening a new - `FastOpenPart` body stream at the requested encoded offset. -7. If the erasure decoder later asks the same replacement reader for the next - contiguous shard offset, the reader continues on the same stream. If it asks - for a backward or non-contiguous offset, the reader returns `errUnexpected`. - The reader initializes its internal current offset to the requested - `shardOffset`, not to zero. -8. If an active body stream fails, it is closed and retired for the rest of that - GET, matching canonical decode behavior. FastOpen does not reopen the same - failed disk as a later lazy replacement for the same request. -9. If the replacement frame is missing, stale, unsupported, corrupt, or belongs - to a different selected version/layout, that reader fails and the existing - erasure decoder may try another spare. If decode quorum cannot be reached, - the GET returns the same class of read error as canonical GET. - -The extra frame validation is required because FastOpen lazy replacement re-runs -`FastOpenPart`, which reads `xl.meta` again and may observe a different selected -version or disk state. Canonical GET does not have this extra validation step -because it builds each spare's file path and layout from the already quorum'd -metadata before decode. - -The block-0 prefix remains useful after lazy replacement exists. Its purpose is -pre-commit validation: FastOpen should prove that the selected metadata and body -streams can decode the first object block before `GetObjectReader` is returned -and client-visible bytes can be sent. Lazy replacement handles failures after -commit; block-0 prefix keeps first-block failure inside the pre-commit retry -window. Lazy replacement could also handle a block-0 spare open with -`bodyOffset == 0`, but the pre-commit path can afford to close streams, reopen -or revalidate the whole online set, and fail before committing any response -bytes. Post-commit recovery cannot restart the response and therefore uses -same-offset lazy replacement. - -Lazy replacement happens below decryption and decompression, so SSE-S3/SSE-KMS -and compressed objects use the same shard-layer replacement behavior as plain -objects. Altered parity and non-STANDARD storage class are handled through the -selected object's actual erasure layout. Inline, metadata-only, and transitioned -objects do not use offset replacement: inline bodies fit in the pre-commit -decode window, and metadata-only/transitioned responses have no local shard body -stream to replace. - -An engaged lazy replacement stream follows the same cancellation and cleanup -rules as any other `FastOpenPart` stream. It is closed/canceled on decode end, -client disconnect, or GET teardown. A lazy spare that is never engaged holds no -body stream and has nothing to close. - -The initial open count is intentionally the maximum configured read quorum only, -with no hedge. If a disk is reachable and reports online but consistently times -out or returns stale/corrupt data, requests may take two waves: selected set -first, then all remaining online disks. This is accepted for Phase 1 correctness; -later tuning may widen the initial open set when the erasure set is known -degraded. - ---- - -## 7. Cancellation Contract - -- One cancellable child context per `FastOpenPart` stream. -- Client request cancellation cancels every child context. -- Unused streams are closed and canceled immediately after a winner is selected. -- Replacement losers are closed and canceled immediately. -- If FastOpen exits before client bytes are sent, all opened streams are closed - and canceled. -- Remote storage handlers must stop reading local files and stop writing the - response when the request context is canceled. -- Do not drain unused remote streams. - ---- - -## 8. Error And Fallback Rules - -- Unsupported request type: use current non-FastGet implementation. -- Missing FastOpen capability on any required storage node/set is a deployment - incompatibility. Phase 1 does not support rolling upgrade to this version. The - implementation may disable FastOpen and use the current implementation if this - is detected, but it does not need request-time mixed-version negotiation. -- Supported request, selected set fails or times out before first client byte: - open all remaining online disks as replacements. -- Supported request, all online disks still cannot satisfy existing Buckit - metadata/read quorum: return the same class of error as existing GET. -- After first client byte: no semantic fallback and no new timeout behavior; - return existing stream/decode error behavior. - ---- - -## 9. Testing Plan - -### Functional Equivalence - -- Gating golden test: for each supported scenario below, run canonical GET and - FastOpen GET against the same object and byte-compare the resulting - `ObjectInfo`, response headers, status/error, and body. -- Latest single-part GET body/header equality. -- Explicit `versionId` GET. -- Versioned bucket latest GET. -- Version-suspended/null version GET. -- Delete marker latest and explicit delete marker version. -- Zero-byte object. -- Inline object. -- Restored-on-disk object. -- Remote-tier not restored. -- Object-lock metadata with and without retention/legal-hold permissions. -- Tags and user metadata. -- `x-amz-checksum-mode: ENABLED`. -- Non-STANDARD storage class / altered parity layout. -- SSE-S3 and SSE-KMS. -- Compressed single-part full-object GET. -- Replication-configured source object, including replication status and version - purge status headers. -- Lifecycle-eligible object where prediction headers are emitted. -- Legacy/XLV1 object if Phase 1 keeps XLV1 in scope. - -### Failure And Availability - -- Initial selected disk missing object/version. -- Initial selected disk stale `xl.meta`. -- Initial selected disk missing `DataDir/part.1`. -- Initial selected disk returns corrupt body and bitrot verification detects the - corruption before or during decode. -- Initial selected set cannot reach quorum, replacement opens all remaining - online disks. -- Initial selected stream fails during block-0 decode before response commit, - replacement restarts from offset 0 and succeeds when quorum is available. -- Post-commit selected stream fails at a later erasure block and a lazy - replacement reader opens a spare shard at the same encoded body offset; GET - succeeds when decode quorum is still available. -- Lazy replacement frame mismatch or unsupported status is rejected and another - spare can be tried. -- Replacement set reaches quorum and response succeeds. -- All online disks cannot reach quorum and error matches existing GET. -- Client disconnect cancels remote reads. -- Unused selected streams are canceled and not drained. -- Post-first-byte stream failure with insufficient replacement quorum follows - existing decode/read error behavior. -- Inline object follows the same decode behavior as shard-backed objects. - -### Regression - -- Range, `partNumber`, multipart, HEAD, and SSE-C continue through current - non-FastGet implementation. -- Multi-pool selection behavior is unchanged; FastOpen runs only after existing - pool/owner resolution. -- Mixed-version cluster with missing FastOpen support disables FastOpen or uses - the current implementation; rolling-upgrade compatibility is not provided. -- Existing FastGet prototype tests are either updated for the new Phase 1 path or - retired if they only validate obsolete `current` shadow behavior. - ---- - -## 10. Milestone Task List - -### Milestone 1: Frame And Conversion Foundation - -- [ ] Add `FastOpenFramePrelude`, `CoalescedMetadataFrame`, - `FastOpenGETMeta`, body-mode/status enums, and stable bitrot algorithm code - mapping. -- [ ] Add frame encode/decode helpers with `"BFG1"` magic, protocol version, - header length, and frame CRC validation. -- [ ] Add `fileInfoToFastOpenGETMeta` and `fastOpenGETMetaToFileInfo` - conversion helpers. -- [ ] In conversion tests, verify raw metadata is preserved, `ReplicationState` - is rebuilt with `getInternalReplicationState`, `Deleted` comes only from - `FastOpenStatusDeleteMarker`, time values round-trip through Unix nanos, and - unknown bitrot algorithm codes fail safely. -- [ ] Add golden conversion tests comparing canonical `FileInfo.ToObjectInfo` - output with compact-frame reconstructed output for representative metadata - combinations. - -### Milestone 2: Storage-Node xl.meta FastOpenPart - -- [ ] Add disk-local `xlStorage.FastOpenPart` support that reads this disk's - `xl.meta`, selects latest or explicit `versionId`, and emits a compact - `CoalescedMetadataFrame`. -- [ ] Add metadata-only frame results for not found, version not found, delete - marker, zero-byte object, and transitioned object not restored locally. -- [ ] Add inline object support by returning the frame plus encoded inline body - bytes from `xl.meta`. -- [ ] Add single-part shard-backed support by returning the frame plus an opened - `DataDir/part.1` encoded shard stream. -- [ ] Preserve canonical storage timeout behavior: `xl.meta`/frame construction - uses the existing drive timeout, while body stream behavior follows - `ReadFileStream` plus existing bitrot/decode semantics. -- [ ] Reject or return unsupported for out-of-scope requests before body bytes: - HEAD, SSE-C, range, `partNumber`, and multipart body paths. - -### Milestone 3: Remote Transport And Cancellation - -- [ ] Add storage REST/grid transport for `FastOpenPart` with the same framed - response stream shape as disk-local storage. -- [ ] Ensure the remote handler invokes the storage-node `xl.meta` - `FastOpenPart` operation on the host where the target disk is mounted. -- [ ] Give every opened `FastOpenPart` stream a cancellable child context. -- [ ] Ensure closing unused or replacement-loser streams cancels remote work - without draining. -- [ ] Ensure client disconnect cancels all open FastOpen child contexts. -- [ ] Add transport tests for frame decode errors, context cancellation, and - remote handler cleanup. - -### Milestone 4: Landing-Node Selection And Decode - -- [ ] Add eligible GET-only FastOpen entrypoint in `erasureObjects.GetObjectNInfo` - before the current `getObjectFileInfo` fan-out. -- [ ] Select only `disk != nil && disk.IsOnline()` and initially open the maximum - configured read quorum over online disks, with no hedge. -- [ ] Convert frames into minimal `FileInfo` arrays and reuse existing - `objectQuorumFromMeta`, not-found quorum handling, `pickValidFileInfo`, - erasure distribution ordering, and decode helpers. -- [ ] Wrap selected shard/inline body streams in a streaming bitrot-verifying - `ReaderAt` before erasure decode. -- [ ] Do not commit HTTP body bytes until frame quorum and block-0 decode can - start from offset 0. - -### Milestone 5: Replacement, Errors, And Fallback - -- [ ] If the initial selected set fails, times out, or cannot form quorum before - the first client byte, open all remaining online disks and restart - selection/decode from offset 0. -- [ ] Add lazy replacement readers for spare disks so mid-stream shard failures - can open a new `FastOpenPart` body stream at the failed encoded shard-body - offset and continue decode when quorum is still available. -- [ ] Extend disk-local and remote `FastOpenPart` to support encoded shard-body - offset reads for replacement streams while keeping returned bodies - forward-only. -- [ ] Apply per-disk `NotFound` and `VersionNotFound` through canonical quorum - logic rather than returning immediate 404. -- [ ] After first client byte, preserve canonical stream/decode error behavior - and do not switch semantic paths; use erasure replacement before surfacing a - read/decode failure. -- [ ] Treat missing FastOpen capability as deployment incompatibility for this - version; rolling-upgrade compatibility is out of scope. -- [ ] Add failure tests for stale metadata, missing body, corrupt body with - bitrot detection, block-0 replacement, mid-stream replacement, replacement - failure, client disconnect, and unused-stream cancellation. - -### Milestone 6: Golden Equivalence And Scope Gates - -- [ ] Add the gating canonical-vs-FastOpen golden GET test across the full - supported matrix in Section 9. -- [ ] Verify exact equality for `ObjectInfo`, response headers, status/error, - and body bytes. -- [ ] Include replication-configured source objects, version purge status, - object lock permissions, tags, checksum mode, SSE-S3/SSE-KMS, compression, - restored-on-disk objects, remote-tier metadata-only objects, versioned/latest - and explicit-version reads, inline objects, zero-byte objects, and altered - storage class/parity. -- [ ] Verify out-of-scope requests continue through the current non-FastGet path: - HEAD, SSE-C, range, `partNumber`, multipart body paths, and multi-pool pool - selection. -- [ ] Add lazy-replacement hardening cases not required for the M5 commit: - non-block-aligned object with mid-stream replacement, multi-block continuation - after replacement engagement, post-commit replacement exhaustion returning the - canonical read-quorum error class, and stale/mismatched replacement candidate - rejection before body bytes are used. - -### Milestone 7: Observability And Cleanup - -- [ ] Add minimal metrics listed in Section 11. -- [ ] Remove or retire obsolete prototype-only FastGet tests that validate - `current` shadow behavior rather than Phase 1 coalesced FastOpen behavior. -- [ ] Keep noisy profiling/tracing logs out of the final path. -- [ ] Run focused `go test -tags kqueue,dev ./cmd` coverage for new FastOpen - tests, then broaden as needed for touched shared helpers. - ---- - -## 11. Metrics - -Keep observability minimal: - -- FastOpen attempted. -- FastOpen hit. -- FastOpen unsupported request. -- FastOpen replacement path used. -- FastOpen streams opened per GET. -- FastOpen replacement opens per GET. -- FastOpen selected-set failure reason. -- FastOpen stream cancellation count. -- FastOpen final error category. - -Do not reintroduce per-phase profiling logs or high-cardinality diagnostics. diff --git a/docs/single-trip-get-phase1-handoff.md b/docs/single-trip-get-phase1-handoff.md deleted file mode 100644 index d2b870509..000000000 --- a/docs/single-trip-get-phase1-handoff.md +++ /dev/null @@ -1,284 +0,0 @@ -# Single-Trip GET Phase 1 Handoff - -## Scope - -This note summarizes the work completed on the two-host HDD-style benchmark setup for the phase-1 single-trip GET prototype, plus the code change made to fix misleading fast-get fallback metrics. - -## Hosts - -- `buckit_node1` - - public IP: `3.81.144.6` - - private IP: `172.31.44.123` -- `buckit_node2` - - public IP: `35.173.238.193` - - private IP: `172.31.37.19` - -SSH used: - -```sh -ssh -i /Users/rooseveltlai/Downloads/buckit.pem ubuntu@3.81.144.6 -ssh -i /Users/rooseveltlai/Downloads/buckit.pem ubuntu@35.173.238.193 -``` - -## Host Setup Completed - -Each host now has: - -- three XFS-formatted instance-store mounts: - - `/mnt/data01` - - `/mnt/data02` - - `/mnt/data03` -- benchmark workspace: - - `/home/ubuntu/singletrip-bench` -- deployed Buckit binary: - - `/home/ubuntu/singletrip-bench/bin/buckit` -- helper scripts copied from repo: - - `/home/ubuntu/singletrip-bench/run/start-buckit.sh` - - `/home/ubuntu/singletrip-bench/run/cold-curl-ttfb.sh` -- `mc` installed on `buckit_node1`: - - `/home/ubuntu/singletrip-bench/bin/mc` - -## Cluster Topology - -Important detail: Buckit rejected endpoint hostnames with underscores. - -This failed: - -- `buckit_node1` -- `buckit_node2` - -because Buckit reported: - -- `FATAL Unable to split host port buckit_node1:9000: invalid hostname` - -So the distributed single pool was run with DNS-valid aliases: - -- `buckit-node1 -> 172.31.44.123` -- `buckit-node2 -> 172.31.37.19` - -Those mappings were written to `/etc/hosts` on both nodes. - -Distributed server command shape used on both nodes: - -```sh -/home/ubuntu/singletrip-bench/bin/buckit \ - --config-dir /home/ubuntu/singletrip-bench/config \ - server \ - --address :9000 \ - --console-address :9001 \ - http://buckit-node1/mnt/data01 \ - http://buckit-node1/mnt/data02 \ - http://buckit-node1/mnt/data03 \ - http://buckit-node2/mnt/data01 \ - http://buckit-node2/mnt/data02 \ - http://buckit-node2/mnt/data03 -``` - -Live backend info observed via `mc admin info`: - -- one pool -- one set -- `6` drives per set -- standard parity `3` -- RRS parity `1` - -So the live setup is effectively `EC:3` for standard objects, not `EC:2`. - -## Datasets Created - -Buckets loaded on the live cluster: - -- `singletrip-cold-640k-6d` -- `singletrip-cold-1m-6d` -- `singletrip-cold-2m-6d` - -Object counts: - -- `10` objects per bucket - -Sizes: - -- `640 KiB` -- `1 MiB` -- `2 MiB` - -Source data on node1: - -- `/home/ubuntu/singletrip-bench/data/640k` -- `/home/ubuntu/singletrip-bench/data/1m` -- `/home/ubuntu/singletrip-bench/data/2m` - -Presigned URL lists on node1: - -- `/home/ubuntu/singletrip-bench/results/urls-640k.txt` -- `/home/ubuntu/singletrip-bench/results/urls-1m.txt` -- `/home/ubuntu/singletrip-bench/results/urls-2m.txt` - -The `640 KiB` and `1 MiB` buckets were loaded while `FAST_GET=1` was enabled, and `current/part.1` shadow files were verified to exist. - -## Functional Validation Completed - -### Byte-for-byte correctness - -Validated by downloading objects back on `buckit_node1` and using `cmp` against the original upload files. - -Passed: - -- `FAST_GET=1`: all 10 objects matched exactly -- `FAST_GET=0`: all 10 objects matched exactly - -### Dependency split validation - -Validated on `obj-01.bin` across all six drives: - -- `FAST_GET=1` - - renamed away all `xl.meta` - - GET still returned `HTTP 200` - - bytes still matched original file - -- `FAST_GET=0` - - renamed away all `current/` - - GET still returned `HTTP 200` - - bytes still matched original file - -This confirms: - -- ON path can serve without `xl.meta` -- OFF path can serve without `current/` - -## Code Change Made - -### Problem - -The fast-get fallback counter was misleading. - -Observed before fix: - -- immediately after a fresh `FAST_GET=1` restart, before external GETs, `mc` metrics already showed: - - `fast_get_fallbacks_total = 9` - -Root cause: - -- internal `.minio.sys` reads used `GetObjectNInfo` -- those reads passed the coarse `fastGetRequestEligible(...)` gate -- they could not use the single-trip path -- so they incremented `fastGetFallbacks` - -This made the metric unsuitable for interpreting user-facing GET fallback. - -### Fix - -Minimal fix implemented: - -- reject `minioMetaBucket` in `fastGetRequestEligible(...)` - -Files changed: - -- [cmd/singletrip-get.go](/private/tmp/buckit-single-trip-get-phase1/cmd/singletrip-get.go) -- [cmd/erasure-object.go](/private/tmp/buckit-single-trip-get-phase1/cmd/erasure-object.go) -- [cmd/singletrip-header_test.go](/private/tmp/buckit-single-trip-get-phase1/cmd/singletrip-header_test.go) - -Behavior after fix: - -- fresh `FAST_GET=1` restart shows no startup fallback noise -- after one real GET: - - `fast_get_hits_total = 1` - - no fallback line emitted - -After a fresh 100-request ON-only run: - -- `GetObject total = 100` -- `fast_get_hits_total = 100` -- `fast_get_fallbacks_total = 0` effectively (no line emitted) - -## Metrics / Benchmark Notes - -### Reading metrics - -Useful command: - -```sh -/home/ubuntu/singletrip-bench/bin/mc admin prometheus metrics bench api --api-version v3 -``` - -Relevant series after fix: - -- `minio_api_requests_total{name="GetObject",...}` -- `minio_api_requests_fast_get_hits_total{...}` -- `minio_api_requests_fast_get_fallbacks_total{...}` (only appears when non-zero) - -### Earlier benchmark notes - -Several A/B runs were performed before the metric fix, including: - -- 10-run ON/OFF passes for `640 KiB`, `1 MiB`, `2 MiB` -- reversed-order runs -- 100-run `2 MiB` ON/OFF percentile runs - -Takeaway: - -- results were not uniformly in favor of `FAST_GET=1` -- `2 MiB` in particular behaved inconsistently across different run orders -- after the metric fix, the ON-only 100-run counter check confirmed: - - `100/100` requests hit the fast path - - `0/100` requests fell back - -### Latest percentile numbers captured - -From the latest fresh patched-build 100-run `2 MiB` curl TTFB comparison: - -ON: - -- `p95 47.417 ms` -- `p90 29.137 ms` -- `p80 24.092 ms` -- `p70 22.538 ms` -- `p50 13.761 ms` - -OFF: - -- `p95 37.950 ms` -- `p90 28.294 ms` -- `p80 21.372 ms` -- `p70 12.067 ms` -- `p50 6.452 ms` - -These are only for the latest fresh 100-run `2 MiB` comparison. - -## Local Result Files - -Not checked in; available in the local workspace under: - -- `.tmp/bench2/` -- `.tmp/bench2_rerun_2m/` -- `.tmp/bench100_2m/` -- `.tmp/bench100_on_counts/` -- `.tmp/bench100_off_counts/` - -These contain raw curl timings and, where captured, trace JSON. - -## Current State - -At the time of hand-off: - -- the patched binary has been rebuilt and deployed to both nodes -- the cluster was last switched to `FAST_GET=1` and then later to `FAST_GET=0`/`FAST_GET=1` multiple times during experiments -- one `640 KiB` 100-run test was started but the user interrupted the turn - -Because the last turn was interrupted, do not assume any in-flight benchmark loop completed cleanly. - -Before resuming, check: - -```sh -ssh -i /Users/rooseveltlai/Downloads/buckit.pem ubuntu@3.81.144.6 'ps -ef | grep -E "buckit|curl --http1.1" | grep -v grep' -ssh -i /Users/rooseveltlai/Downloads/buckit.pem ubuntu@35.173.238.193 'ps -ef | grep -E "buckit|curl --http1.1" | grep -v grep' -``` - -and re-establish whether the cluster is currently ON or OFF. - -## Recommended Next Steps - -1. Confirm current cluster mode (`FAST_GET=1` or `FAST_GET=0`) and kill any stray benchmark loops left by the interrupted turn. -2. Re-run the `640 KiB` 100-request comparison cleanly on the patched build. -3. If useful, add admin-trace capture to the fresh patched 100-run size comparisons, not just curl TTFB. -4. If the benchmark note should be preserved, convert key results from this hand-off into a checked-in benchmark README similar to `docs/single-trip-hdd-bench/README.md`. diff --git a/docs/single-trip-get-phase1-implementation.md b/docs/single-trip-get-phase1-implementation.md deleted file mode 100644 index 2b9e1826d..000000000 --- a/docs/single-trip-get-phase1-implementation.md +++ /dev/null @@ -1,1121 +0,0 @@ -# Single-Trip Direct GET — Phase 1 Prototype Implementation Plan - -**Status:** Implementation plan / prototype scope - -**Companion to:** `docs/single-trip-get-design.md` (read it first for the layout, -header, quorum, and durability model). This document is the concrete build plan -for a **measurement prototype** of the latest-GET fast path, not the production -design. - ---- - -## Goal - -Validate the performance claim in design §6 — that removing the `xl.meta` -fan-out read phase from a healthy latest GET buys lower first-byte latency and -higher seek-bound throughput — with the smallest faithful change to the real -server. Everything not relevant to the GET-side measurement is deliberately -faked or skipped. - -### What the prototype measures, and what it does not - -The §6 claim splits into two physically distinct effects that need different -measurements: - -- **First-byte latency** is governed by *round-trip phases*. Today: metadata - fan-out (≈ one parallel seek across the set) **then** shard fan-out (≈ one - parallel seek) = two sequential phases. The fast path collapses this to one - phase. This is only observable when the disks are genuinely parallel — i.e. - on a real multi-spindle cluster, never on a serial single-host benchmark. -- **Saturated throughput** is governed by *seeks-per-GET-per-disk*. A data disk - does a metadata seek plus a data seek (2) today, versus header + data in one - open (1). This is per-spindle and IOPS-bound under concurrency. - -A single-host benchmark that reads 16 `xl.meta` files serially does **not** -model either effect — in a real deployment those 16 reads fan out concurrently -across 16 separate drives, so the metadata phase costs roughly one seek of -wall-clock latency, not sixteen. The prototype is therefore exercised on the -cluster rig (`testing/cluster`, 4 nodes × 4 drives = 16 drive paths in one EC:4 -set), with an optional single-spindle IOPS sanity check as the only honest -single-host experiment. - -**Caveat on the rig — read before trusting absolute numbers.** The rig's drives -are XFS-on-loopback files inside Docker (on macOS, inside a LinuxKit VM). They -are 16 *independent drive paths* and give a correct functional A/B, but they are -**not 16 independent physical HDD spindles**: they share backing storage and an -extra page-cache layer, so they do not faithfully reproduce HDD seek latency. -The rig is therefore good for *relative* off-vs-on comparison and for proving the -mechanism, but **HDD seek-behavior evidence requires running the same build on a -host with real, separate spinning disks** (see §9.2 and §9.9). Treat container -results as ratios between arms, not as absolute HDD figures. - -### Deliberate prototype shortcuts - -Stated up front because they bound what the numbers mean: - -- `current/part.1` is an **additive shadow copy** of the shard, not a moved - directory. This doubles write cost and storage. **Only GET numbers are valid; - PUT numbers are meaningless.** -- Single-part, latest-only, non-versioned, non-transitioned, non-encrypted, - non-compressed, non-object-lock, non-inline; **offset-0 reads only** (full - object or a range starting at byte 0 — required for the true single-trip stream - reuse in §4). Everything else falls through to the existing path. -- No crash consistency, no repair, no delete-marker reconstruction, no - `renameat2` exchange, no `versions//`. `xl.meta` stays canonical, so - fallback is always correct. - ---- - -## 0. Feature flag & scope guard - -- Read `BUCKIT_FAST_GET=1` once at startup into `globalFastGetEnabled` - (prototype-only gate; no config subsystem wiring). -- Eligibility is checked in **two stages**, because some predicates are knowable - from the request alone but others require the object's metadata — which on the - fast path we only learn *from the direct header*, not from `xl.meta`. Mixing - them into one pre-`getObjectFileInfo` check would be wrong (we'd be testing - encryption/transition/part-count before we have any metadata). - - **Stage 1 — request-level precheck (before any read), `fastGetRequestEligible`:** - - flag on; `opts.VersionID == ""` (latest); - - bucket versioning is Unversioned (from bucket metadata, available without - object metadata); - - no SSE-C request headers (request-detectable); not a replication request; - - range is absent or **starts at byte offset 0** (see §4 — true single-trip - requires an offset-0 read; non-zero offsets fall back in Phase 1). - - **Stage 2 — header-level validation (after reading the direct header), - `fastGetHeaderValid`:** confirm from the quorum header that the object is - single-part, not a delete marker, not transitioned, not SSE-S3/KMS encrypted, - not compressed, not object-locked, not inline, and (under the default - response-metadata scoping of §1.1) carries no custom `x-amz-meta-*` user - metadata or object tags, and that its bounded response headers did not exceed - the §1.1 byte budget (over-cap objects are marked ineligible at write time and - must fall back, never truncate). Any failure → fall back to `getObjectFileInfo`. - These facts are carried in the header (§1) precisely so we can validate them - without `xl.meta`. -- The existing RLock path is unchanged (we still take the read lock as today). - ---- - -## 1. On-disk header format - -New file `cmd/singletrip-header.go`. A fixed-size, self-describing, CRC'd header -prefixed onto each disk's `current/part.1`: - -``` -current/part.1 = [ singleTripHeader (fixed size, prototype: 1024B) ][ existing shard bytes: hash_0|block_0|... ] -``` - -`singleTripHeader` carries the design §2.1 minimum needed to rebuild a -`FileInfo` without reading `xl.meta`: - -- magic + format version + `headerLen`; header CRC (xxhash) over the rest; -- a **direct-path signature** `[4]byte` for cross-disk quorum grouping — - **computed explicitly** at write time, *not* read from `FileInfo` (which does - not expose `xlMetaV2VersionHeader.Signature`). The signature must cover **every - field that affects decode correctness**, not just identity: - `directSig = xxhash32(versionID ‖ modTimeNanos ‖ ErasureM ‖ ErasureN ‖ - ErasureBlockSize ‖ ErasureDist ‖ bitrotAlgo ‖ partCount ‖ PartSize ‖ - ActualPartSize ‖ Size ‖ ETag ‖ flags ‖ contentType ‖ contentEncoding ‖ - cacheControl ‖ expires ‖ storageClass)` — the bounded response-metadata fields (§1.1) are - included so copies that would send *different* response headers cannot group - together, and the §3.1 quorum additionally checks them for byte-equality. - `ErasureIndex` is deliberately - **excluded** (it differs per disk). A 4-byte digest can collide, so the quorum - step (§3.1) does **not** trust the signature alone: it groups by `directSig` - and then additionally requires the common layout fields above to be - byte-equal across the group, and the per-disk `ErasureIndex` values to be - **distinct and valid**. **`ErasureIndex` is 1-based in Buckit** — valid values - are `1..M+N` (`cmd/erasure-metadata.go:81` checks `Index > 0 && Index <= M+N`), - the **data** shards are indices `1..M`, and the on-disk array position is - `Index-1`. Signature first for cheap grouping, field equality + index validity - for correctness; -- `ErasureM, ErasureN, ErasureIndex, ErasureBlockSize, ErasureDist []uint8`; -- `Size, ModTime, PartSize, ActualPartSize`, bitrot algorithm; -- ETag, VersionID; -- validation flags needed by Stage-2 eligibility (§0): `isDeleteMarker`, - `isInline`, `isTransitioned`, `isEncrypted`, `isCompressed`, - `isObjectLocked`, `partCount`, and — for the §1.1 default scoping — - `hasUserMeta` and `hasObjectTags` (so the read path can reject objects with - custom `x-amz-meta-*`/tags without `xl.meta`). These flags are covered by - `directSig` and the §3.1 byte-equality check like the other layout fields; -- **S3 response-metadata subset** (see §1.1) — without it the fast path would - send wrong/missing response headers. - -Prototype implementation note: Phase 1 uses a small fixed-binary payload encoded -by `cmd/singletrip-header.go`, not generated `msgp`, to keep the measurement -patch self-contained. It is still **zero-padded to a fixed -`singleTripHeaderLen`** so the payload offset is a compile-time constant — no -second read is needed to discover where the shard begins. Production can switch -the inner payload to generated `msgp` if we want schema evolution support after -the measurement proves useful. - -### 1.1 Response-metadata fidelity (don't just decode the bytes) - -Decoding the payload correctly is necessary but not sufficient: a normal GET also -returns response headers that `ToObjectInfo` derives from `fi.Metadata` and -`setObjectHeaders` sends to the client (`cmd/erasure-metadata.go:97`, -`cmd/api-headers.go:125,149`) — `Content-Type`, `Content-Encoding`, -`Cache-Control`, `Expires`, **a non-STANDARD `x-amz-storage-class`** (kept in -`UserDefined` by `cleanMetadata`, `cmd/object-api-utils.go:401`, and emitted by -`setObjectHeaders`), object tags, and arbitrary `x-amz-meta-*` user metadata. The -fast path synthesizes `ObjectInfo` from the header, not from `xl.meta`, so it must -carry enough to reproduce these or it will silently drop/garble them — and the §6 -byte-and-header equality test (which compares against a flag-off GET) would catch -it as a diff. - -Three parts, by boundedness: - -- **Bounded common headers — carried in the fixed header, with explicit caps.** - `Content-Type`, `Content-Encoding`, `Cache-Control`, `Expires`, storage class - (and `ETag`, already present). These have **no inherent small bound** (a - `Content-Type` or `Cache-Control` can be long), so the header reserves a fixed - byte budget per field and a total budget within `singleTripHeaderLen`, e.g. - `Content-Type ≤ 128`, `Content-Encoding ≤ 64`, `Cache-Control ≤ 128`, - `Expires ≤ 40`, `storage-class ≤ 32` (tune to fit). **Over-cap is not truncated - and not a silent failure: an object whose encoded subset exceeds the budget is - marked Stage-2 ineligible (`shadowEligible` false on write, header-invalid on - read) and falls back.** All of these fields are included in `directSig` and the - §3.1 byte-equality check (so storage class can't be lost or mixed across a - group). -- **Storage class specifically.** STANDARD is the default and emits no header; - only a *non-STANDARD* `x-amz-storage-class` round-trips. Carry the value in the - bounded subset above (covered by `directSig`/equality). If you prefer not to - carry it, the alternative is to mark any non-STANDARD-storage-class object - Stage-2 ineligible — but do not simply ignore it, or the fast path would drop - the header. -- **Unbounded `x-amz-meta-*` user metadata and object tags.** These have no size - bound, so a fixed header cannot hold them. Phase 1 picks one of: - - **(default) Scope it out, with a caveat.** The benchmark/correctness objects - carry no custom user metadata or tags; the §6 header-equality test asserts the - bounded subset only. Documented limitation: objects *with* user metadata/tags - are not fast-served faithfully and must be excluded from the fast path (treat - presence of user-meta/tags as a Stage-2 ineligibility, falling back). This - keeps the constant-offset design and is fine for a perf prototype. - - **(upgrade) Variable-length metadata region.** Carry `metaLen` in the fixed - header, followed by a msgpack blob of the full `MetaUser`/relevant `MetaSys`, - then the shard. The payload offset becomes `fixedLen + metaLen` (read from the - fixed part) — still consumed from the *same* single open, so the single-trip - invariant holds; only the "compile-time-constant offset" simplification is - relaxed. Use this only if full-fidelity GETs matter for the evaluation. - - Phase 1 uses the default (scope-out) unless the perf objects need custom - metadata. - -- **Compression and object-lock metadata are also scoped out.** Compression is - not only response metadata: normal GET uses the internal compression marker and - actual-size metadata to wrap the object stream in decompression. If the fast - path synthesized an `ObjectInfo` without those fields, it would stream raw - compressed shard bytes and compute ranges against the wrong size. Therefore any - object carrying `X-Minio-Internal-compression` is Stage-2 ineligible and falls - back. Object-lock retention/legal-hold metadata similarly affects - `x-amz-object-lock-*` response headers; Phase 1 does not carry it, so locked - objects are ineligible and fall back. - -**Bitrot.** The per-block bitrot hashes stay **interleaved in the shard payload -exactly as today** (they are not moved into the header), and the fast path uses -the **streaming** bitrot reader exclusively (`newStreamingBitrotReader`), which -reads each block's hash inline from the stream. This is deliberate: the streaming -reader needs only the algorithm (carried in the header), **not** a per-block -checksum slice, so we do not have to serialize bitrot sums into the header. The -non-streaming/whole reader (`newWholeBitrotReader`, which *does* require a `sum`) -is never used on the fast path. `erasure.Decode` and verification are therefore -byte-for-byte unchanged; only the file's start offset shifts by -`singleTripHeaderLen`. - ---- - -## 2. Write side — shadow `current/part.1` - -Hook into `putObject` around the canonical commit (`commitRenameDataDir` / -`RenameData`, `cmd/erasure-object.go:1577`), gated on the flag. Note the ordering -required by §2.1: prior-shadow **invalidation runs *before* the canonical commit** -and new-shadow **installation runs *after*** it — it is not a single -post-commit step. This is an **additive shadow copy**, not a directory move -(prototype shortcut; GET-perf only). - -**The disks are not all local.** The evaluation rig is distributed, so -`onlineDisks[i]` is a `storageRESTClient` (remote) for most shards, not a local -`xlStorage`. A local-only helper on `xlStorage` cannot run on remote disks. The -shadow write must therefore go through the `StorageAPI` interface, which is -transparently local or remote. Two viable approaches: - -- **Prototype approach (no new RPC):** from the landing node, for each - `onlineDisks[i]` (local or remote) compose the shadow file from the bytes that - were just written, reusing **existing** `StorageAPI` calls — read the committed - shard via `ReadFileStream(bucket, DataDir/part.1)` and write the shadow with a - **streaming** `CreateFile(..., fileSize, io.MultiReader(bytes.NewReader(header), shardStream))`. - `CreateFile` requires the exact `fileSize` up front and `xlStorage` validates - the bytes written against it, so compute it explicitly: - `fileSize = singleTripHeaderLen + fi.Erasure.ShardFileSize(fi.Parts[0].Size)` - (`cmd/erasure-metadata.go:54`) — the source `DataDir/part.1` is the *encoded* - shard file (it already includes the interleaved bitrot hashes), so - `ShardFileSize` gives its on-disk length, and the header adds a fixed - `singleTripHeaderLen`. - **`CreateFile` is `O_EXCL` (`cmd/xl-storage.go:2128`) — it will not overwrite.** - So write to a fresh temp path (`current/part.1.tmp-`) and then - `RenameFile` it over `current/part.1`, which replaces atomically per disk and - works local+remote. This is also why the stale-shadow rule below is mandatory. - Do **not** use `WriteAll` for this: it materializes the whole prefixed shard in - memory, which is fine only for tiny test fixtures, not for the multi-MiB - objects the perf run uses. `CreateFile` streams, so memory stays bounded. Both - calls already work over storage REST/grid, so no protocol change is needed. - Cost: an extra full read+write per disk at PUT time — an explicit PUT - distortion, acceptable because **only GET numbers count** (see shortcuts). -- **Cleaner approach (if PUT distortion matters):** add a - `WritePrefixedShard(...)` method to `StorageAPI` with storage REST/grid - plumbing so each disk writes its own shadow locally from its already-present - shard. More code; avoids the cross-node copy. - -Phase 1 uses the prototype approach. - -- Build `singleTripHeader` per disk from `fi` (plus that disk's `ErasureIndex`), - then write `pathJoin(object, "current", "part.1")` = header followed by the - shard bytes. No `renameat2`, no fsync ordering — out of scope. -- **Decide eligibility, then either install or invalidate — never "do nothing."** - A single `shadowEligible(fi, bucket, opts)` guard, mirroring the read-side - eligibility, returns false for: multipart (`len(fi.Parts) != 1`), inline / - zero-byte (no `DataDir/part.1` exists), any encryption (SSE-C/S3/KMS), - compressed objects, object-lock retention/legal-hold metadata, - transitioned/remote, versioned or version-suspended buckets, delete markers, - legacy-v1 objects, objects carrying custom `x-amz-meta-*` user metadata or - object tags (under §1.1 default scoping), and objects whose bounded response - headers (§1.1) exceed the per-field/total byte budget. **But "ineligible" must not mean "leave `current/` - untouched"** — see §2.1: if a prior eligible version left a shadow, doing - nothing leaves it readable and fast GET returns stale bytes. So on every - successful canonical PUT, eligible ⇒ install the shadow (after commit); - ineligible ⇒ **invalidate** the prior `current/` to the §2.1 `oldN+1` - threshold-confirmed level *before* the canonical commit, or **abort** — never - the unsafe "best-effort, ignore failures" path §2.1 rules out. Writing direct - headers for - objects the prototype must never fast-serve would also waste storage and risk a - header that disagrees with what the read path can validate; the read side still - re-validates from the header (§0 Stage 2) — defense in depth, not a substitute. - -### 2.1 Stale-shadow invalidation (correctness-critical) - -`current/part.1` is a **stable, reused path** — every overwrite and delete of the -key targets the same path. Unlike the per-version `DataDir`, it is *not* -write-once. The hard invariant is: - -> **A stale `current/` shadow must never survive as a valid quorum after a newer -> canonical commit (overwrite or delete).** If the fresh shadow cannot be -> installed to quorum, no old shadow may remain that could form one — GET must be -> forced to fall back to `xl.meta`. - -The corollary that's easy to miss: **every successful canonical PUT must take a -definite action on `current/` — install or delete — never "do nothing."** Doing -nothing is only safe when no prior shadow exists, which the write path cannot -assume. If this is violated, fast GET can return bytes (or existence) that -disagree with the canonical `xl.meta`, which is exactly the failure the whole -"xl.meta stays canonical" design forbids. Concretely: - -- **Overwrite, new version eligible.** All steps run under the object write lock - already held by `putObject`. Follow the uniform ordering: **(1)** invalidate the - prior `current/` to the `oldN+1` threshold (§2.1, old-layout-derived); **(2)** canonical commit - (`RenameData`); **(3)** install the new shadow (temp-write-then-`RenameFile`). - Do **not** install-the-new-shadow-before-commit: if the commit then failed, a - fresh shadow with the *new* `directSig` would be readable while `xl.meta` still - points at the old version — a stale-*new* read, the mirror of the stale-old bug. - Deleting first also means step 3 lands on an empty path, sidestepping the - `O_EXCL` overwrite problem; the temp+`RenameFile` is retained only to maximize - new-shadow coverage on disks that may still hold a leftover old file. -- **Overwrite, new version *ineligible* (the asymmetric case).** When an eligible - object is overwritten by an ineligible one — inline/zero-byte, multipart, - compressed, object-locked, encrypted, transitioned, versioned/suspended, - delete-marker-like, or legacy-v1 — the write path produces no new shadow, so - the **prior** eligible shadow would remain readable and fast GET would pass - Stage 2 on *old* metadata and return stale bytes. Therefore: on every - successful canonical PUT where - `shadowEligible(...) == false`, **delete `current/` to the invalidation - threshold below before returning success.** -- **Delete (non-versioned).** A delete in scope actually removes the object, so - add an **invalidation hook in the delete path** (`deleteObject` / - `DeleteObjects`) that deletes `current/` to the invalidation threshold below, - under the write lock. (Versioned deletes create a delete marker and are out of - scope per §2 — they never had a shadow.) - -#### Invalidation success condition (there is no read-side staleness backstop) - -"Best-effort" is **not** a correctness condition here, and this is the subtle -part: the read path deliberately never consults `xl.meta`, so **nothing on the -read side can detect that a surviving old shadow is stale.** If an overwrite/ -delete removes `current/` from only some disks and the old shadow still has -enough shards to satisfy a fast read, `tryFastGet` will group the old `directSig`, -reach quorum, and return stale bytes. Invalidation must therefore *prove* the old -shadow can no longer form a fast-path read: - -- A fast read requires (§3.1) ≥ `oldM` headers agreeing on one `directSig`, - **and** all `oldM` data-index shards present. So the old shadow is provably - unreadable once **at most `oldM-1` of its `current/` files remain** — i.e. - invalidation must **confirm removal (or confirmed absence) of `current/` on at - least `oldN + 1` disks**, where `oldM`/`oldN` are the **stale shadow's own - erasure layout**, `oldM+oldN − (oldN+1) = oldM−1 < oldM`, which simultaneously - drops the old `directSig` group below quorum *and* guarantees at least one - data-index shard is gone (only `oldN` parity positions exist). -- **The threshold is keyed to the *old* layout, not the new object's.** This is a - trap: an overwrite can change the erasure layout — storage class, max-parity - config, or availability optimization can give the *new* object a different - parity than the old (`cmd/erasure-object.go:1298`). Using the new object's `N` - (or a default `EC:` value) can under-delete and leave a readable stale shadow. - So derive `oldN` from the **existing `current/` headers** (each carries - `ErasureN`, §1) before deleting; if those headers can't be read reliably on - enough disks, fall back to a **conservative threshold = `maxPossibleParity + 1`** - for the set (e.g. `setDriveCount/2 + 1`, ≥ any layout the old shadow could have - had). Never assume the old parity equals the new parity. This threshold is - stricter than the canonical write quorum, which is required because — unlike a - normal write — a stale fast read has no second check. -- **Ordering is what makes failure cluster-safe — invalidate *before* the - canonical commit.** A process-local kill switch is **not** sufficient: this is a - distributed cluster, and another serving node would still fast-serve the stale - `current/` shadow. The fix is ordering, not a flag. Under the object write lock, - delete the prior `current/` to the `oldN+1` threshold **before** the canonical - metadata commit (`RenameData`) that supersedes it (for delete, before the - tombstone/removal commit). Then: - - **Threshold met →** proceed with the canonical commit; for an eligible - overwrite, install the new shadow afterward (a failed install just yields - fallback, never staleness). - - **Threshold *not* met →** **abort the operation before the canonical commit - and return an error** (client retries). Because nothing newer was committed, - the old `current/` shadow still agrees with the still-current old `xl.meta` on - *every* node — the cluster stays consistent, with no window where new canonical - state coexists with a readable old shadow anywhere. This is the cluster-safe - behavior the local kill switch failed to provide. - - Optional heavier alternative, only if you want to keep serving while degraded: - disable fast GET **cluster-wide** via a peer/config broadcast that every - serving node observes (e.g. a peer-REST notification flipping `globalFastGetEnabled` - on all nodes) before reporting success. Phase 1 prefers abort-before-commit; - the cluster-wide switch is noted only for completeness. -- **Known residual (prototype-only):** a disk that was offline during invalidation - can rejoin later still carrying its old `current/` file. If enough such disks - rejoin to re-cross quorum before the key is next written, a stale fast read - becomes possible again. Phase 1 does not add the scanner/repair that would - reconcile this (design §5.3); it is an accepted prototype limitation, called out - as **not production-safe**, and is why the production design keeps `xl.meta` - authoritative with active reconciliation. - -**Crash residuals under the new ordering (invalidate → commit → install).** With -invalidation required *before* the canonical commit (§2.1), the crash windows are -benign: - -- **Crash after invalidation, before commit:** the old `current/` was already - reduced below the `oldN+1` threshold, and `xl.meta` still names the *old* version - (not yet committed). A torn/partial old shadow can no longer form a fast quorum, - so GET falls back to `xl.meta` and serves the old canonical version — consistent. -- **Crash after commit, before/while installing the new shadow:** `xl.meta` names - the new version. Either the new shadow isn't installed yet (GET falls back to - `xl.meta` → new version, consistent) or it's partially installed; partial - installs can't reach the new-version quorum, so GET still falls back. A stale - *old* shadow cannot reappear because it was invalidated pre-commit. - -In all cases the read path's quorum + `directSig` check plus "fewer than `M` -agreeing ⇒ fallback" is the backstop, and `xl.meta` stays authoritative. The one -residual that *does* remain is the offline-disk rejoin above. Phase 1 adds no -scanner/repair (design §4/§5); this is acceptable for a measurement prototype but -is **not production-safe** — the production design's `renameat2`/repair model -(design §4–5) is what closes it. - ---- - -## 3. Read side — single-trip fast path - -In `GetObjectNInfo` (`cmd/erasure-object.go:203`), insert before the -`getObjectFileInfo` call at line 239: - -```go -if fastGetRequestEligible(ctx, bucket, object, rs, opts) { // Stage 1 (§0) - if gr, ok := er.tryFastGet(ctx, bucket, object, rs, h, opts, nsUnlocker, &unlockOnDefer); ok { - return gr, nil - } - // not ok → fall through to existing path unchanged -} -fi, metaArr, onlineDisks, err := er.getObjectFileInfo(...) -``` - -### 3.1 The single-trip invariant (the point of the whole experiment) - -The fast path must be **one open / one storage request per disk**: the same -stream that returns the header continues directly into the shard bytes. If we -instead read the header with one `ReadFileStream(...,0,headerLen)` and then let -the decoder open `current/part.1` a *second* time, we have preserved two request -phases and two seeks — which is exactly the cost the design claims to remove, so -the measurement would be meaningless. The implementation below therefore opens -each disk's `current/part.1` **once** and threads that one open reader through -both header validation and decode. - -`tryFastGet` (new code in `cmd/singletrip-get.go`): - -1. **One open per disk — but open to EOF, not a precomputed length.** The encoded - shard length we'd want as the read length depends on header fields we have not - read yet (size, part size, erasure layout, bitrot block count), so we *cannot* - pass `headerLen + tillOffset` up front — that's a chicken-and-egg. Open instead - with **`length = -1`** (stream to EOF): - `rc, _ := disk.ReadFileStream(bucket, currentPartPath, 0, -1)`. One local open, - or one storage-REST/grid request for remote disks. The decoder reads only as - far as it needs and we then close; we never have to know the exact length in - advance. -2. **Header off the front of that stream.** `io.ReadFull(rc, hdrBuf[:headerLen])` - (`headerLen` is the fixed `singleTripHeaderLen`), verify the header CRC. `rc` - is now positioned exactly at the shard payload (byte `headerLen`); for the - streams we keep it is **not** closed and **not** reopened. -3. **Quorum** — group validated headers by the direct-path signature (§1) **and** - require, within the group, byte-equal **layout *and* bounded response-metadata - fields** (§1/§1.1 — `Content-Type`, `Content-Encoding`, `Cache-Control`, - `Expires`, storage class, `ETag`, `Size`, `ModTime`) plus distinct valid `ErasureIndex` values - (§1). The response-metadata fields are part of the equality check, not merely - "present," so the synthesized `ObjectInfo` cannot be assembled from a group - that disagrees on what headers to send. Require ≥ `DataBlocks` agreeing. If not, - cancel all open streams (step 6) and return `ok=false` (fallback). -4. **Stage-2 header validation** (§0): single-part, not delete-marker, not - transitioned, not inline, not compressed, not object-locked, not SSE-S3/KMS, - and (default §1.1 scoping) **no custom `x-amz-meta-*` user metadata or object - tags**. Any failure → cancel + fallback. -5. **Keep only the data-shard streams; wrap them into an index-ordered slice.** - Phase 1 serves only the *fully healthy* M-data case: select the streams whose - `ErasureIndex` is a **data index `1..M`**, and wrap each in a streaming bitrot - reader that reads *from the already-open `rc`* (see §4). If any data index - `1..M` is missing/invalid, **fall back** (we do not pull parity on the fast - path; degraded reads go through `xl.meta`). This fallback boundary is only - before response streaming starts. Once the fast path has emitted bytes to the - client, it cannot restart through `xl.meta` without corrupting HTTP semantics. - The Phase 1 from-stream bitrot reader is forward-only (`ReadAt` must be at the - current stream offset), so a mid-stream disk error or bad block at stripe K can - fail/truncate a GET that the normal path could have reconstructed by opening a - parity shard at the correct offset. This is an availability limitation of the - healthy-path prototype, not a data-integrity issue: bitrot verification still - prevents wrong bytes from being served. - - **Ordering is a correctness requirement, not a detail.** Reed-Solomon decode - is only correct if `readers[i]` corresponds to shard index `i`. The production - path enforces this via `shuffleDisksAndPartsMetadataByIndex` - (`cmd/erasure-metadata-utils.go:222`), which places each disk at - `shuffled[Index-1]` using `Erasure.Distribution`. The fast path must do the - same: build a **full `M+N`-length `readers` slice** and assign - `readers[ErasureIndex-1] = fromStreamReader(...)` for each kept data shard - (1-based data indices `1..M` → zero-based positions `0..M-1`), leaving the - parity positions (zero-based `M..M+N-1`) `nil`. **Never** - append readers in disk-response order — that can decode a valid quorum into - corrupted bytes. The cleanest implementation reuses the production machinery: - synthesize `fi.Erasure.Distribution` from the header, build per-disk - `onlineDisks`/`metaArr` in set order, and let the existing - `shuffleDisksAndPartsMetadataByIndex` + decode loop order everything (with the - readers carrying their pre-opened streams). -6. **Close with cancel, never drain — for *all* fast-path streams.** Because - every stream was opened with `length = -1` (to EOF), draining on close pulls - *whatever is left of the shard*. That is wrong for two distinct sets of - streams: - - **Non-kept streams** (parity, minority/loser, and all streams on fallback): - nearly the whole shard is undrained, so a drain pulls the entire unwanted - shard. - - **Kept streams that are not fully consumed** — an offset-0 range shorter - than the object, or a client that disconnects early — still have a tail of - the EOF stream pending; draining that tail wastes bandwidth and distorts - range/early-abort measurements. - - The normal `streamingBitrotReader.Close` calls `xhttp.DrainBody` - (`cmd/bitrot-streaming.go:153-156`) precisely to reuse the connection, which - is the opposite of what we want here. So the from-stream reader (§4) and the - header-only readers must both expose a `cancelClose()` that, for remote disks, - cancels the request context (or closes the underlying body without draining) - and for local disks just closes the fd. **No fast-path stream is ever - `Close`d via the drain path — kept or not.** (To keep the headline perf - numbers free of this subtlety, §9 benchmarks full-object GETs for the main - latency/throughput cells and treats short ranges as a correctness case.) -7. **Synthesize `FileInfo` + the index-ordered disk/meta slices.** Build `fi` - from the quorum header (Erasure params incl. `Distribution`, single Part, - Size, ModTime, ETag) — enough for `NewGetObjectReader` - (`cmd/erasure-object.go:283`) and the decode loop. `onlineDisks` and `metaArr` - must be **full `M+N`-length slices in erasure-index order** (the same shape the - decode loop expects after `shuffleDisksAndPartsMetadataByIndex`), with each - kept data shard at position `ErasureIndex-1` and **every non-kept/parity - position set to `OfflineDisk`/invalid `FileInfo`** — *not* a compacted slice of - only the kept disks. The decode path is index-sensitive (§3.1 step 5); a - compacted slice would misalign shards and decode to garbage. -8. **Lock cleanup on success — mirror the non-inline path exactly.** The fast path - only serves non-inline objects, so on a successful stream `tryFastGet` must do - what `cmd/erasure-object.go:288-304` does for non-inline reads: set - `*unlockOnDefer = false` and attach `nsUnlocker` to the returned - `GetObjectReader`'s cleanup (i.e. `fn(pr, h, pipeCloser, nsUnlocker)`), so the - RLock is released when the reader is closed — **not** by the caller's `defer`. - Getting this wrong either unlocks while bytes are still streaming or leaks the - RLock. On the fallback path `tryFastGet` returns `ok=false` having touched - neither, and the existing code at line 239+ takes over unchanged. - -On any fallback, every stream opened in step 1 is `cancelClose`d before calling -`getObjectFileInfo`. The wasted cost is the `headerLen` bytes read **plus -whatever the transport and the remote `ReadFileStreamHandler` already read, -buffered, or put on the wire before the cancel landed** — for remote disks the -server begins copying the EOF stream immediately (`cmd/storage-rest-server.go`), -so cancellation bounds but does not zero this. It is still far less than a drained -full shard, and never a leaked fd. If you need the exact wasted-bytes figure for -the writeup, instrument the cancel path rather than assuming it is just the -header. - -**Observability (required for the evaluation).** Maintain two process counters, -`fastGetHits` and `fastGetFallbacks` (atomic, prototype-only), incremented on the -fast-path success and on every fallback respectively. Phase 1 exposes them under -the existing `/api/requests` metrics group as `fast_get_hits_total` and -`fast_get_fallbacks_total`, labeled `type="s3"`. Without this you cannot tell -whether the "ON" arm actually exercised the fast path or silently fell back to -`xl.meta`, which would make a null result unfalsifiable. The evaluation playbook -(§9) checks these counters after every run. - ---- - -## 4. Decode — reusing the already-open stream (the surgical reader change) - -The decode loop in `getObjectWithFileInfo` (`cmd/erasure-object.go:351-390`) -builds a `streamingBitrotReader` per disk, and that reader **opens its own** -`ReadFileStream` on first `ReadAt` (`cmd/bitrot-streaming.go:168-176`). For the -fast path we must *not* let it open a second time — it has to consume the stream -`tryFastGet` already opened in §3.1. - -Why this works for Phase 1's scope: eligibility restricts the fast path to -**offset-0 reads** of single-part objects, so the decoder reads the payload -sequentially from byte 0. The header sits at file byte 0; once we have consumed -`headerLen` bytes, the open stream is positioned exactly where the streaming -bitrot reader's first `ReadAt(offset=0)` expects to begin. No seek, no reopen. -(Non-zero-offset ranges would require repositioning the shared stream; those are -excluded in Stage 1 and fall back — this is the honest boundary of the -single-trip claim.) - -**Length semantics — raw shard offset vs encoded file bytes.** Be precise about -the two different "lengths" here, because conflating them under-reads the file: - -- The decoder's `partOffset/partLength/tillOffset` are **raw shard** quantities - (object bytes, no hashes). -- The **physical `current/part.1`** is `headerLen` + the *encoded* bitrot stream, - where the encoded stream interleaves a hash before every `shardSize` block. - `newStreamingBitrotReader` already performs this conversion internally - (`cmd/bitrot-streaming.go:210`: `tillOffset = ceilFrac(tillOffset, shardSize)*hashSize + tillOffset`). - -Because §3.1 opens the stream with **`length = -1` (to EOF)**, `tryFastGet` -never has to compute the encoded length itself — it just consumes `headerLen` -raw bytes and hands the rest to the bitrot reader, which reads the correct number -of hash+block bytes per its existing math. The header still carries `Size`, -`PartSize`, and `ActualPartSize` (raw quantities) so the synthesized `FileInfo` -and the range/offset arithmetic in `getObjectWithFileInfo` are correct; only the -*open length* is delegated to EOF. - -Change to `streamingBitrotReader` (`cmd/bitrot-streaming.go`): - -- Add an optional pre-opened reader field, e.g. `rc io.ReadCloser` supplied at - construction via a new `newStreamingBitrotReaderFromStream(rc, ...)`. -- In `ReadAt`, when `b.rc` is already set (line 168 branch), skip the - `disk.ReadFileStream(...)` open entirely and read straight from the supplied - stream; initialize `currOffset = 0`. The existing lazy-open path is untouched - for all other callers. -- The `streamOffset`/`tillOffset` math (lines 171, 210) and the per-block - hash-then-block verification (lines 185-199) are **unchanged** — the only - difference is *where the bytes come from* (a handed-in stream vs. a fresh - open). -- **`Close` must cancel, not drain.** The default `Close` - (`cmd/bitrot-streaming.go:153-156`) calls `xhttp.DrainBody` for connection - reuse. The from-stream reader was opened to EOF (§3.1), so if it is only - partially consumed (short range, early client abort, or it's a parity/loser - stream) draining pulls the rest of the shard. The from-stream variant must - therefore override `Close` to cancel the underlying request/body **without** - draining — this is the `cancelClose()` referenced in §3.1 step 6. - -Factor a `fastGetObjectStream` that clones the decode-loop body but feeds the -pre-opened readers, rather than overloading `getObjectWithFileInfo` with -conditionals — keeps the production path pristine and the prototype cleanly -removable. - -**This is the change that makes or breaks the experiment:** verify by tracing -syscalls (or counting `ReadFileStream` calls) that an ON-arm GET issues exactly -**one** open per participating disk, not two. If you see two, the measurement is -invalid regardless of the numbers. - ---- - -## 5. Fallback correctness - -Every failure mode returns `ok=false`, and the caller runs today's -`getObjectFileInfo` path against untouched `xl.meta`: missing/short -`current/part.1`, header CRC failure, no signature quorum, out-of-scope range, -decode/bitrot failure. Because the shadow path is purely additive and `xl.meta` -is canonical, fallback is always correct. No request-time repair queue in the -prototype (design §5.2 is deferred). - ---- - -## 6. Correctness tests (before any perf run) - -`cmd/singletrip-get_test.go`, run with `-tags kqueue,dev`. Each test asserts on -the `fastGetHits`/`fastGetFallbacks` counters (§3) so "correct bytes" and "took -the path we think it did" are checked separately. - -- **Shadow written:** PUT with flag on of a **non-inline** object → assert - `current/part.1` exists with a valid header on quorum disks. -- **Byte- and header-identical on the fast path:** GET with flag on returns - identical **body bytes and response headers** to a flag-off GET — assert the - §1.1 bounded subset (`Content-Type`, `Content-Encoding`, `Cache-Control`, - `Expires`, `x-amz-storage-class`, `ETag`, `Content-Length`, `Last-Modified`) - matches, not just the body — with `fastGetHits` incremented. Sizes **above the - inline cutoff only**: 64 KiB / 1 MiB / 16 MiB, full object + range starting at - offset 0. (4 KiB is omitted here — see the inline case below.) Set a non-default - `Content-Type` **and** a non-STANDARD storage class on at least one fixture so - both header paths are actually exercised. -- **User-metadata/tags, compression, and object-lock fall back (default scoping, - §1.1):** an object carrying `x-amz-meta-*`, object tags, compression metadata, - or object-lock retention/legal-hold metadata must take the fallback path - (`fastGetHits` unchanged) and return all bytes/metadata correctly — unless the - variable-length/full-metadata header upgrade (§1.1) is implemented, in which - case assert full fidelity on the fast path instead. -- **Over-cap headers fall back (§1.1):** an object whose `Content-Type`/ - `Cache-Control` exceeds the bounded byte budget must take the fallback path - (`fastGetHits` unchanged) and return the full header — never truncated. -- **Single open:** assert the ON-arm GET issues exactly one `ReadFileStream` per - participating disk (counter or trace), proving the single-trip invariant - (§3.1/§4). -- **No RLock leak / early unlock:** after a fast-path GET (both fully consumed and - early-closed by the client), assert the object's namespace RLock is released - exactly once on reader close — guards the §3.1 step 8 cleanup. A subsequent - `Lock()` on the same key must succeed without timeout. -- **Quorum/repair tolerance:** corrupt or delete one disk's `current/part.1` → - still correct, either via the remaining quorum (hit) or via fallback. -- **Overwrite never serves stale (§2.1):** PUT object A (assert fast hit returns - A), overwrite the same key with object B, then GET must return **B** (fast hit - or fallback) — **never A**. Repeat with B larger and smaller than A. -- **Eligible → ineligible overwrite (the asymmetric case, §2.1):** PUT eligible - object A (assert fast hit), overwrite the same key with a **4 KiB inline** - object B; GET must return **B via fallback** (zero fast hits), **never A**. - Assert `current/` was deleted across the set. Repeat cheaply for at least one - more ineligible kind (multipart or SSE-C) to cover the non-inline ineligible - transition. -- **Delete never serves stale (§2.1):** PUT object A so a shadow exists, then - delete the key; GET must 404 / fall back — **never serve the old shadow**. - Assert `current/` is gone (or no longer forms a quorum) across the set. -- **Old-layout invalidation threshold (§2.1, the heterogeneous-parity case):** - PUT object A whose shadow uses one erasure layout `oldM/oldN`, then - overwrite/delete the key under a condition that yields a *different* parity for - the new object — e.g. a different storage class, a changed max-parity config, or - an availability-optimized layout (`cmd/erasure-object.go:1298`) such that - `newN < oldN`. Assert the invalidation deletes to **`oldN+1`** (derived from the - old `current/` headers), **not** `newN+1`; and that a subsequent GET never - returns A — this is the **production assertion** and it must pass. - - *Mutation / fault-injection sub-case (a positive, passing assertion):* via a - test-only seam, swap in an invalidator that deletes only `newN+1` disks (the - wrong, too-small count) and **assert that this faulty version leaves stale A - readable** — i.e. the test positively detects the stale read produced by the - bug. This proves the threshold is load-bearing (the real test isn't passing by - luck) while remaining a normal green assertion about the injected fault, not an - "expect the suite to fail" construct. - - *Conservative fallback:* when the old headers can't be read on enough disks, - assert deletion uses `maxPossibleParity+1`. -- **Inline small object (4 KiB):** no `DataDir/part.1` exists, so no shadow is - written; assert the GET is byte-correct **via fallback** with `fastGetHits` - *not* incremented. This is expected behavior, not a defect. -- **Out-of-scope → fallback, zero fast hits:** versioned bucket (incl. a - delete-marker object: assert normal 404 via the `xl.meta` path, `fastGetHits` - unchanged), multipart, SSE, and non-zero-offset ranges each confirm the - request takes the fallback path. (Phase 1 implements **no** delete-marker fast - path; the marker case is purely a fallback assertion.) - ---- - -## 7. Deliverables checklist - -| Item | File | -|---|---| -| Flag + eligibility guard | `cmd/singletrip-get.go` | -| Header struct + encode/CRC | `cmd/singletrip-header.go` (+ generated `_gen.go`) | -| Write-side shadow install + replace (temp+`RenameFile`) | `cmd/erasure-object.go` (`putObject`) | -| Shadow invalidation on overwrite/delete (§2.1) | `cmd/erasure-object.go` (`deleteObject`, `DeleteObjects`) | -| Read-side single-trip fast path + decode clone | `cmd/singletrip-get.go` | -| Stream-reuse bitrot reader (`...FromStream`) | `cmd/bitrot-streaming.go` | -| Hit/fallback counters | `cmd/singletrip-get.go`, `cmd/metrics-v3-api.go`, `cmd/metrics-v3.go` | -| Correctness tests | `cmd/singletrip-get_test.go` | - ---- - -## 7.1 Phase 1 implementation task list - -Use this checklist as the execution order. Keep each step independently reviewable; -do not mix the measurement harness, write-side shadowing, and read-side stream -reuse in one change. - -### A. Scaffolding and header model - -- [x] Add startup feature gate `BUCKIT_FAST_GET=1` and process counters - `fastGetHits` / `fastGetFallbacks`. -- [x] Add `cmd/singletrip-header.go` with fixed-size header encode/decode, - CRC validation, direct signature computation, bounded response metadata caps, - and explicit Stage-2 flags (`hasUserMeta`, `hasObjectTags`, compression, - object-lock, over-cap marker). -- [x] Add header unit tests for valid round-trip, CRC failure, bad magic/version, - signature/equality grouping, response-metadata caps, and 1-based - `ErasureIndex` validation. -- [x] Add request/header eligibility helpers without wiring them into the hot path - yet. - -### B. Write-side shadow creation and invalidation - -- [x] Implement shadow eligibility using the Phase 1 scope: single-part, - non-inline, non-versioned, non-transitioned, non-encrypted, non-compressed, - non-object-lock, no user metadata or tags under default §1.1 scoping, and - bounded response metadata within cap. -- [x] Implement old-shadow invalidation under the object write lock before - canonical commit, deriving `oldN` from existing `current/` headers or using the - conservative `maxPossibleParity+1` threshold when old headers are not reliable. - The prototype derives old parity from the strongest agreeing `current/` header - group and requires `oldN+1` delete-or-absence confirmations; if no old header is - readable, it falls back to the caller's conservative quorum. -- [x] Implement post-commit shadow install through `StorageAPI`: read committed - `DataDir/part.1`, stream fixed header + shard bytes into temp - `current./part.1`, then `RenameFile` to `current/part.1`. -- [x] Make failed post-commit shadow install performance-only: log and leave the - object correct via fallback, but never leave an old readable shadow after a - newer canonical commit. - -### C. Read-side fast path - -- [x] Insert `tryFastGet` before `getObjectFileInfo` in `GetObjectNInfo`, behind - Stage-1 request eligibility. -- [x] Open `current/part.1` once per disk with `length=-1`, read and validate the - fixed header, then keep the same stream positioned at the shard payload. -- [x] Group headers by direct signature and byte-equal layout/metadata fields; - require distinct valid 1-based erasure indices and all data indices `1..M`. -- [x] Synthesize `FileInfo`, `ObjectInfo`, and full `M+N` index-ordered - `onlineDisks`/reader slices; never compact by response order. -- [x] Add stream-reuse bitrot reader variant with cancel-not-drain close semantics. -- [x] Add an end-to-end assertion that fast GET uses one `ReadFileStream` per - participating disk. -- [x] Mirror current non-inline RLock cleanup semantics: attach `nsUnlocker` to the - returned `GetObjectReader` cleanup and clear caller defer on successful fast - stream. - -### D. Correctness and measurement - -- [x] Add correctness tests from §6, including byte/header equality, fallback - counters, stale-overwrite/delete cases, eligible-to-ineligible overwrite, - old-layout invalidation threshold, over-cap metadata fallback, and single-open - assertions. Byte/ObjectInfo equality, no-shadow fallback counters, single-open - assertions, offset-0 range reads, stale overwrite/delete, - eligible-to-ineligible overwrite, and over-cap fallback are covered; - heterogeneous parity/old-layout threshold is covered by a targeted invalidation - quorum test. Mid-stream corruption is covered as a documented availability - limitation: canonical read succeeds, while fast read returns a partial errored - stream. -- [x] Expose or log `fastGetHits` / `fastGetFallbacks` enough for the §9 playbook. -- [x] Run focused `go test -tags kqueue,dev ./cmd -run SingleTrip` plus any - touched-package tests. -- [x] Run a local single-process 16-drive smoke validation to confirm shadow - install, byte correctness, and fast-path counters before attempting the full - A/B benchmark. -- [x] Fix and run the Docker cluster rig far enough to execute a container - cold-TTFB A/B pilot on 4 nodes x 4 XFS loopback drives. -- [x] Run the §9 A/B benchmark playbook on the container rig (`warp` saturated - throughput + cold-TTFB, OFF vs ON) after all correctness tests pass. With the - rig corrected to a single 16-drive pool and 2 MiB (non-inlined) objects, the ON - arm is provably single-trip (serves with `xl.meta` deleted) and cold server-side - TTFB drops ~18% vs OFF; saturated throughput is flat. Direction matches §6; - absolute magnitude still requires §9.2's bare-metal host. See results below. - -Local smoke result (2026-06-04): with `BUCKIT_FAST_GET=1`, a disposable -single-process erasure server using 16 local drive directories accepted an 8 MiB -non-uniform object, installed `current/part.1` on all 16 drives, served a GET -whose bytes matched the uploaded payload, and reported -`minio_api_requests_fast_get_hits_total=1`. This is a mechanism check only; it -does not measure the Phase 1 performance delta because all drives are directories -on the same local filesystem and the run does not compare cold-cache OFF vs ON -arms. - -Docker cluster A/B results (2026-06-04): getting a *valid* measurement out of the -4-node x 4-drive loopback-XFS rig (`testing/cluster`, `--memory 512M ---drive-size 2G`) required correcting **two rig defects** that each silently -neutered the experiment. Both are worth recording because they are easy traps: - -> **Trap 1 — multi-pool routing reads `xl.meta` before the fast path.** As shipped, -> `cluster.sh` emitted one endpoint arg per node, so the deployment came up as -> **four independent server pools** (`totalSets:[1,1,1,1]`). On a multi-pool GET, -> `erasureServerPools.GetObjectNInfo` calls `getLatestObjectInfoWithIdx` first — -> which runs `GetObjectInfo` (an `xl.meta` read) across *every* pool to find the -> owner — and only then dispatches to the set-level shadow fast path. So with -> `FAST_GET=1` the server still read `xl.meta` on every GET; the run was *not* -> single-trip. The fast path bypasses `xl.meta` only on the `z.SinglePool()` -> branch (`erasure-server-pool.go`). Fix: `cluster.sh` now emits a **single pool -> spanning all nodes** (`http://node{1...4}:9000/data/drive{0...3}`, -> `totalSets:[1]`, one 16-drive EC:4 set), so GET goes straight to the set. -> -> **Trap 2 — inline cutoff gates eligibility.** A shard is inlined into `xl.meta` -> when `ShardFileSize(ActualSize) <= 128 KiB`. With 12 data shards (EC:4 over 16 -> drives) an object must exceed ~1.5 MiB to have a standalone `DataDir/part.1`; -> below that it is inlined, `writeSingleTripShadow` skips it (`fi.InlineData()` -> guard), and every GET falls back. (A first 64 KiB run on the broken 4-pool/EC:2 -> layout hit exactly this: `fast_get_hits_total=0` vs `fallbacks≈57 k`.) The runs -> below use **2 MiB objects** (~174 KiB shard, comfortably above the cutoff; -> `xl.meta` stays at 382 B with a separate data shard + shadow, verified on disk). - -With both fixed, the dataset (2,000 x 2 MiB, distinct keys) was loaded once under -`BUCKIT_FAST_GET=1` so shadows exist on every drive, then the same volumes were -restarted for the OFF and ON arms. (`api requests_max` was raised to 10000 — the -512 MiB nodes auto-tune a ~75 req/s limit that throttles the single-set load.) - -**Single-trip invariant — direct proof.** For one 2 MiB object, `xl.meta` and the -canonical `DataDir/part.1` were deleted on **all 16 drives**, leaving only -`current/part.1`. Under `FAST_GET=1` a pre-signed GET still returned **HTTP 200 -with byte-exact md5** — served entirely from the shadow with no `xl.meta` on disk. -The identical test on the old 4-pool layout returned **404** (pool resolution -needs `xl.meta`). This is the decisive confirmation that the ON arm reads no -`xl.meta` in single-pool mode. - -**Measurement A — saturated throughput** (`warp get --list-existing`, 2,000 -objects, 48 concurrent, 30 s, read-only on identical data): - -| Arm | Throughput | obj/s | req p50 | req p90 | req p99 | TTFB median | Fast-path counters | -|---|---:|---:|---:|---:|---:|---:|---| -| `FAST_GET=0` | 147.46 MiB/s | 73.7 | 543.3 ms | 1074.1 ms | 1716.9 ms | 75 ms | n/a (path disabled) | -| `FAST_GET=1` | 149.86 MiB/s | 74.9 | 570.8 ms | 1011.7 ms | 1558.5 ms | 70 ms | 2375 hits / 14 fallbacks (99.4%) | - -**Measurement B — cold single-stream TTFB** (40 distinct objects, page cache -dropped on all four nodes before each batch, single connection, 3 runs each arm). -Both instruments were captured **in one pass per run on the identical requests**: -a `mc admin trace --verbose` stream recorded the server-side `TTFB` field on each -`s3.GetObject` `[RESPONSE]` (request-receipt to first body byte) while `curl --w '%{time_starttransfer}'` issued and timed the same GETs client-side. The OFF -and ON arms were run back-to-back in the same settled session. Per-run median TTFB: - -| Arm | trace R1/R2/R3 | trace median | curl R1/R2/R3 | curl median | Fast-path counters | -|---|---|---:|---|---:|---| -| `FAST_GET=0` | 7.25 / 6.39 / 7.06 ms | **7.06 ms** | 8.80 / 7.82 / 8.47 ms | **8.47 ms** | n/a (path disabled) | -| `FAST_GET=1` | 6.16 / 5.25 / 5.23 ms | **5.25 ms** | 7.55 / 6.51 / 6.67 ms | **6.67 ms** | hits == GET count, ~0 fallbacks | - -Because trace and curl are paired per request, the expected invariant `curl ≥ -trace` holds in **every** run (curl median ~1.3–1.4 ms above trace — the loopback -TCP-connect/send cost the server-side timer never sees). Read the OFF-vs-ON delta -within an instrument; the absolute numbers carry `mc admin trace`'s per-request -observation overhead (both arms equally, so the delta is unaffected). - -**Reading the result.** With a genuine single-trip ON arm, the metadata-collapse -signal appears in both instruments: cold TTFB drops from ~7.1 ms to ~5.3 ms on the -server-side trace (**~26%**) and from ~8.5 ms to ~6.7 ms on the client-side curl -cross-check (**~21%**). Saturated throughput is essentially flat (+1.6%), which is expected at 2 MiB on a -cache-backed medium: transfer dominates first-byte cost and there is no physical -seek for the saved open to eliminate, so bandwidth is unchanged. The win is a -per-request *latency* effect (one fewer open), exactly where the design predicts -it. - -Caveat unchanged: loopback-XFS on Docker Desktop has no real seek latency, so the -saved open costs only syscall/RPC overhead (~1.5 ms here), not an 8–10 ms HDD -seek. The **direction** now matches design §6/§9.10 (cold-TTFB reduction with -hits ≫ fallbacks and a within-noise throughput control); the **absolute -magnitude** still requires the same binary on a Linux host with real, separate -spinning disks (§9.2), where the eliminated open is a seek and the cold-TTFB delta -should widen well past the ~26% seen here. Raw artifacts: `warp` archives -`/private/tmp/warp-singletrip-2m-{load-on,off,on}.csv.zst` and the paired -curl+trace cold-TTFB captures `/tmp/st-both-{curl,trace}-{off,on}.txt`. - ---- - -## 8. Out of scope for Phase 1 - -Deferred to the production design (`docs/single-trip-get-design.md`) and called -out in the measurement writeup: - -- `renameat2(RENAME_EXCHANGE)` directory swap; directory move-not-copy. -- `versions//` for non-current versions. -- Crash recovery, request-time/disk-local repair, fsync ordering. -- Multipart and arbitrary-range fast path. -- Mid-stream degraded-read recovery from parity on the fast path. Phase 1 can - fall back before streaming if required data shard headers are missing, but once - streaming starts a later shard error returns a truncated/errored GET instead of - reopening parity at the failed offset like the canonical `xl.meta` path can. -- Migration of pre-existing objects to direct paths. - -Because the shadow copy makes PUT cost unrepresentative, **only GET latency and -throughput numbers are valid** from this prototype. - ---- - -## 9. Evaluation playbook — using the Phase 1 build to measure the difference - -This section is the operational recipe: how to drive the prototype and read a -number out of it. The whole evaluation is a clean **A/B on identical on-disk data -and one identical binary**, because `BUCKIT_FAST_GET` is a *runtime* switch — the -only variable that changes between arms is whether GET takes the single-trip path -or the `xl.meta` path. - -### 9.1 Why this is a clean A/B - -- The shadow `current/part.1` is written at PUT time whenever the flag is on. So - load the dataset **once with the flag on**; every object then has both - `xl.meta` + `DataDir/part.1` (canonical) and `current/part.1` (shadow). -- **Baseline arm (OFF):** GET ignores `current/` and reads `xl.meta` — exactly - today's two-phase path. The unused shadow files do not affect the `xl.meta` - read cost. -- **Fast arm (ON):** GET reads `current/part.1` headers and skips the `xl.meta` - fan-out. -- Same data, same binary, same keys, same client — only the read path differs. - -### 9.2 Bring up the rig - -```sh -cd testing/cluster -./cluster.sh create --fast-get 1 # 4 nodes × 4 drives = 16 drive paths in one EC:4 set -# bigger drives if your dataset needs it (default 1G/drive): -# ./cluster.sh create --fast-get 1 --drive-size 4G -``` - -The rig's drives are XFS-on-loopback files, **not** independent physical -spindles (see the caveat in the overview). That makes this a sound *relative* -off-vs-on rig but a poor model of HDD seek latency. **For absolute HDD evidence, -run the same Phase 1 binary on a Linux host whose `BUCKIT_ENDPOINTS` point at -mount paths backed by real, separate spinning disks** and repeat §9.4–§9.5 -there; the container rig is for mechanism validation and quick relative checks. - -Facts the playbook relies on (from `cluster.sh`): - -| Thing | Value | -|---|---| -| S3 API endpoints | `http://localhost:9000` (node1), `9002`, `9004`, `9006` | -| Console | odd ports `9001`, `9003`, … | -| Credentials | `buckitadmin` / `buckitadmin` | -| SSH (for `drop_caches`) | `localhost:2201`–`2204`, root password `buckitadmin` | -| Drive mounts inside a node | `/data/drive0` … `/data/drive3` (XFS loopback) | - -`cluster.sh create` builds the current repo's Phase 1 binary for Linux, copies it -into the node image, and starts it through systemd. Toggle the arm with -`--fast-get`: use `1` while loading the dataset so shadows are written, then use -`0` for the baseline arm and `1` for the fast arm when regenerating/restarting -the rig. The same source binary is used; only the `BUCKIT_FAST_GET` service -environment changes. - -Configure an `mc` alias once: - -```sh -mc alias set lab http://localhost:9000 buckitadmin buckitadmin -``` - -### 9.3 Load the dataset (flag ON, so shadows exist) - -Use distinct keys per object so neither arm gets a trivial cache hit. Spread -across size classes; keep total within usable capacity -(`16 × drive_size / 1.33`). Example with `warp`: - -```sh -mc mb lab/perf -# one run per size class; --obj.randsize off, fixed sizes: -warp put --host localhost:9000 --access-key buckitadmin --secret-key buckitadmin \ - --bucket perf --obj.size 1MiB --objects 4000 --concurrent 32 --noclear -# repeat with --obj.size 4KiB / 64KiB / 16MiB into separate prefixes -``` - -(`mc cp`/a small PUT loop works too; `warp` is just convenient.) Sanity-check -that shadows were actually written: - -```sh -sshpass -p buckitadmin ssh -p 2201 root@localhost \ - 'ls /data/drive0/perf/*/current/part.1 2>/dev/null | head' -``` - -If that path is empty, the write-side hook didn't fire — fix that before -measuring, or the ON arm will silently fall back. - -### 9.4 Measurement 1 — cold first-byte latency (the phase-collapse win) - -This is where the design's §6 latency claim lives (≈30–50% on HDD). It is only -visible **cold**, because a warm `xl.meta` makes the first phase free. - -Drop the page cache on every node before each cold GET: - -```sh -drop_caches() { - for p in 2201 2202 2203 2204; do - sshpass -p buckitadmin ssh -p $p root@localhost 'sync; echo 3 > /proc/sys/vm/drop_caches' - done -} -``` - -Measure time-to-first-byte for a single stream over a sweep of *distinct* cold -objects: - -```sh -# presign once, then drop caches and GET, capturing TTFB: -url=$(mc share download --json lab/perf/ | jq -r .share) -drop_caches -curl -s -o /dev/null -w 'ttfb=%{time_starttransfer}s total=%{time_total}s\n' "$url" -``` - -Loop over ~100 distinct objects per size class, dropping caches before each (or -drop once, then GET a batch of never-touched keys). Record **median and p95 -TTFB** per size class, for both arms. The delta is the latency result. - -### 9.5 Measurement 2 — saturated throughput (the per-disk seek win) - -This is the §6 throughput claim (≈1.5–1.75× for seek-bound small/medium GETs). -Drive many concurrent GETs over a working set larger than the VM's RAM so the -drives actually seek and the metadata cache is pressured (most meaningful on the -real-disk host; on the container rig read it as a relative number): - -```sh -warp get --host localhost:9000 --access-key buckitadmin --secret-key buckitadmin \ - --bucket perf --obj.size 64KiB --objects 4000 --concurrent 64 --duration 2m --noclear -``` - -Record aggregate **GET req/s** and **p50/p99 latency**, both arms. Stay in the -seek-bound size regime (64 KiB–1 MiB); at 16 MiB transfer dominates and the -expected win shrinks to ~0–10%. - -### 9.6 Measurement 3 — warm-cache control (the null check) - -Repeatedly GET a small hot set **without** dropping caches. Both arms should land -within noise of each other, because `xl.meta` is already cached so the baseline -pays no extra seek. **If the ON arm shows a large win here, the harness is -measuring something other than the metadata seek — stop and investigate before -trusting Measurements 1–2.** - -### 9.7 Confirm the fast path actually fired - -After each ON-arm run, read the `fast_get_hits_total` / -`fast_get_fallbacks_total` metrics from `/api/requests` (§3). A meaningful ON -result requires hits ≫ fallbacks. A near-zero hit count means requests fell out -of eligibility (wrong size/range/versioning) or the shadow was missing — the -measured "no difference" would be an artifact, not a verdict on the design. - -### 9.8 Optional single-spindle IOPS pre-check - -The only honest single-host experiment, and it covers **throughput only, not -latency**: saturate one loopback XFS device with GET-shaped I/O and count ops/sec -for a 2-seek-per-op access pattern (open+read meta, then open+read data) versus a -1-seek-per-op pattern (open+read the prefixed shard). This models §6's per-disk -seek argument directly, on a single disk, without the full server. A small -`fio` job-pair or a ~50-line Go harness suffices. Do **not** use it to reason -about latency — that needs the parallel cluster. - -### 9.9 Pitfalls and how to read the result - -- **Silent fallback** — always check §9.7 counters; a null result with zero hits - is meaningless. -- **Cancel-close churn** — the fast path deliberately closes EOF streams without - draining so it does not pull whole unused shards from remote disks. That may - reduce connection reuse and add setup cost to subsequent requests, so ON-arm - latency numbers are conservative with respect to connection reuse. -- **Not actually cold** — if the working set fits in the Docker VM's RAM, page - cache hides the seek and Measurement 1 flatlines. Make the set larger than RAM - and always `drop_caches`. -- **Docker on macOS is not bare-metal HDD** — Docker Desktop runs a LinuxKit VM - and loopback-XFS-on-overlay has its own caching; treat the numbers as **ratios - between arms, not absolute HDD figures**. For absolute numbers, run the rig on - a Linux host with real spinning disks. -- **Inline small objects** — objects below the inline cutoff already return from - the metadata read (design §1.2), so 4 KiB GETs should show ≈0 gain in both - measurements; that is expected, not a failure. -- **Hold everything else fixed** — identical key set, concurrency, and request - order across arms; run each cell ≥3× and report median + spread. - -### 9.10 Decision criteria - -The prototype supports the design's §6 claims if, on the parallel cluster: - -- **Latency:** cold-cache median TTFB on ≥1 MiB objects drops by roughly **≥25%** - in the ON arm, while the warm-cache control (§9.6) stays within noise; **and** -- **Throughput:** saturated GET req/s on 64 KiB–1 MiB objects improves by roughly - **≥1.4×** in the ON arm, with fast-path hits ≫ fallbacks. - -If the warm control shows the same gain as cold, or hits ≈ 0, the result is a -measurement artifact and must be fixed before drawing a conclusion. Report all -numbers against the §6 predictions with the caveats above (shadow-copy write cost -excluded; single-part only; container loopback ≠ bare-metal HDD). diff --git a/docs/single-trip-hdd-bench/README.md b/docs/single-trip-hdd-bench/README.md deleted file mode 100644 index 7a64723fc..000000000 --- a/docs/single-trip-hdd-bench/README.md +++ /dev/null @@ -1,256 +0,0 @@ -# Single-Trip HDD Bench Notes - -This document captures the ad hoc single-HDD test rig used on June 4-5, 2026 for the phase-1 single-trip GET prototype. - -The goal of this run was not to prove the final HDD benefit. The host had only one rotational disk, so this was a control rig to: - -- validate the host-side test procedure -- validate basic path dependency assumptions -- get an early latency signal before a real multi-node HDD run - -## Host - -- Host: `root@192.184.90.116` -- OS: Ubuntu 24.04.1 -- Disk layout: one rotational root disk (`/dev/vda1`) -- Buckit paths used as pseudo-drives: - - `/root/singletrip-bench/data4/d01` - - `/root/singletrip-bench/data4/d02` - - `/root/singletrip-bench/data4/d03` - - `/root/singletrip-bench/data4/d04` - -These are four directories on the same HDD, not four independent disks. - -## Binaries And Scripts - -Host-side paths: - -- `buckit`: `/root/singletrip-bench/bin/buckit` -- `warp`: `/root/singletrip-bench/bin/warp` -- server launcher copied from repo: `testing/singletrip-hdd/start-buckit.sh` - -Checked-in rig assets live in: - -- [testing/singletrip-hdd/README.md](/Users/rooseveltlai/develop/buckit-io/buckit/testing/singletrip-hdd/README.md) - -Local helper state used during the runs: - -- `mc` config dir: `/private/tmp/buckit-hdd-mc` -- `640 KiB` presigned URL file: `/private/tmp/hdd-640k-obj-urls.jsonl` -- `2 MiB` presigned URL file: `/private/tmp/hdd-obj-urls.jsonl` - -## Server Start - -Use `MINIO_CI_CD=1` because the test data lives on the root disk. - -`FAST_GET=1`: - -```sh -ssh root@192.184.90.116 ' - env MINIO_CI_CD=1 FAST_GET=1 \ - MINIO_ROOT_USER=buckitadmin \ - MINIO_ROOT_PASSWORD=buckitadmin \ - /root/singletrip-bench/run/start-buckit.sh \ - /root/singletrip-bench/data4/d01 \ - /root/singletrip-bench/data4/d02 \ - /root/singletrip-bench/data4/d03 \ - /root/singletrip-bench/data4/d04 -' -``` - -`FAST_GET=0`: - -```sh -ssh root@192.184.90.116 ' - env MINIO_CI_CD=1 FAST_GET=0 \ - MINIO_ROOT_USER=buckitadmin \ - MINIO_ROOT_PASSWORD=buckitadmin \ - /root/singletrip-bench/run/start-buckit.sh \ - /root/singletrip-bench/data4/d01 \ - /root/singletrip-bench/data4/d02 \ - /root/singletrip-bench/data4/d03 \ - /root/singletrip-bench/data4/d04 -' -``` - -Stop the server: - -```sh -ssh root@192.184.90.116 'pkill -f "/root/singletrip-bench/bin/buckit"' -``` - -## Benchmark Method - -The method that gave the cleanest signal was: - -1. Use 10 different objects of the same size. -2. Run the GETs from the host itself, not from a remote client. -3. Before each GET: - - `sync` - - `echo 3 > /proc/sys/vm/drop_caches` -4. Measure: - - client-visible TTFB with `curl -w '%{time_starttransfer}'` - - server-side TTFB with `mc admin trace --verbose` -5. Compare either: - - last 5 of 10 runs, or - - lowest 5 of 10 runs when the run is noisy - -Example host-local loop: - -```sh -ssh root@192.184.90.116 ' - set -e - for u in "$@"; do - sync - echo 3 > /proc/sys/vm/drop_caches - curl --http1.1 -sS -o /dev/null -w "%{time_starttransfer}\n" "$u" - done -' sh "" "" ... -``` - -Server-side trace: - -```sh -mc --config-dir /private/tmp/buckit-hdd-mc admin trace --verbose hdd -``` - -## Datasets - -### 2 MiB objects - -- Bucket: `singletrip-cold-2m-4d` -- Object count: 20 total, 10 used in the clean comparison -- Object size: `2 MiB` -- Shadow verified: `current/part.1` present on all 4 pseudo-drives - -### 640 KiB objects - -- Bucket: `singletrip-cold-640k-4d` -- Object count: 10 -- Object size: `640 KiB` -- Shadow verified: `current/part.1` present on all 4 pseudo-drives -- `current/part.1` size on disk: `328736` bytes -- `xl.meta` size on disk: `368` bytes - -## Functional Validation - -These checks passed on the host: - -- `FAST_GET=1`: rename `xl.meta` away on all 4 object directories before GET, GET still returns `200` -- `FAST_GET=0`: rename `current/` away on all 4 object directories before GET, GET still returns `200` - -This validates the basic dependency split: - -- ON path can serve without `xl.meta` -- OFF path can serve without `current/` - -## Results - -### 2 MiB, 10 different objects, OFF first then ON - -Using the last 5 of 10 runs: - -- `FAST_GET=0` - - `curl` median: `17.032 ms` - - trace median: `15.301 ms` -- `FAST_GET=1` - - `curl` median: `12.073 ms` - - trace median: `11.698 ms` - -Improvement: - -- `curl`: `29.1%` -- trace: `23.5%` - -### 2 MiB, 10 different objects, ON first then OFF - -Using the last 5 of 10 runs: - -- `FAST_GET=1` - - `curl` median: `30.449 ms` - - trace median: `30.214 ms` -- `FAST_GET=0` - - `curl` median: `43.962 ms` - - trace median: `43.241 ms` - -Improvement: - -- `curl`: `30.7%` -- trace: `30.1%` - -This was the most convincing control result from the single-HDD host. - -### 640 KiB, OFF first then ON - -Using the last 5 of 10 runs: - -- `FAST_GET=0` - - `curl` median: `20.905 ms` - - trace median: `19.100 ms` -- `FAST_GET=1` - - `curl` median: `42.286 ms` - - trace median: `39.010 ms` - -This run inverted and was not trustworthy as a clean fast-path comparison. - -Observed counters after the ON arm: - -- `fast_get_hits_total=10` -- `fast_get_fallbacks_total=15` - -That indicates fallback noise or unrelated contamination during the `640 KiB` run. - -### 640 KiB, ON first then OFF - -Full 10-run lists: - -- `FAST_GET=1` `curl` ms: - - `157.507`, `112.020`, `50.408`, `70.818`, `267.515`, `39.806`, `53.513`, `35.694`, `109.684`, `140.253` -- `FAST_GET=1` trace ms: - - `153.795`, `112.089`, `48.038`, `70.438`, `265.378`, `37.215`, `51.709`, `34.376`, `108.788`, `139.557` -- `FAST_GET=0` `curl` ms: - - `291.295`, `69.004`, `70.546`, `54.955`, `70.527`, `73.920`, `215.505`, `55.148`, `65.311`, `33.009` -- `FAST_GET=0` trace ms: - - `288.341`, `68.346`, `69.096`, `54.154`, `69.758`, `72.806`, `167.278`, `42.117`, `64.080`, `30.531` - -Using the lowest 5 runs in each group: - -- `FAST_GET=1` - - `curl` mean: `50.048 ms` - - `curl` median: `50.408 ms` - - trace mean: `48.355 ms` - - trace median: `48.038 ms` -- `FAST_GET=0` - - `curl` mean: `55.485 ms` - - `curl` median: `55.148 ms` - - trace mean: `51.846 ms` - - trace median: `54.154 ms` - -Difference from lowest-5 slices: - -- `curl` median improvement: `8.6%` -- trace median improvement: `11.3%` - -This is weaker and much noisier than the `2 MiB` result. - -## Interpretation - -Current conclusion from the single-HDD host: - -- The host rig works and is reusable. -- The `2 MiB` different-object cold test produced a repeatable positive signal around `24-31%` TTFB improvement. -- The `640 KiB` case is noisy and not yet a stable signal. -- Because all four paths are on the same physical disk, none of these numbers should be presented as proof of the final HDD benefit. - -## Recommended Reuse For Tomorrow - -For the true multi-node HDD run: - -- keep the host-local `curl` plus `mc admin trace` method -- keep the "10 different objects" pattern -- keep cache-drop before each GET -- prefer object sizes that cleanly produce `current/part.1` -- record both full 10-run lists and the summary slice used for comparison -- check `fast_get_hits_total` and `fast_get_fallbacks_total` after every ON arm - -If the multi-node rig is clean, the `2 MiB` method from this document should be the baseline procedure. diff --git a/docs/single-trip-hdd-bench/two-node-d3-2mib.md b/docs/single-trip-hdd-bench/two-node-d3-2mib.md deleted file mode 100644 index 4faaf5d17..000000000 --- a/docs/single-trip-hdd-bench/two-node-d3-2mib.md +++ /dev/null @@ -1,127 +0,0 @@ -# Two-Node D3 HDD Bench — 2 MiB Cold TTFB - -This document records the June 5, 2026 single-trip GET phase-1 cold-TTFB A/B run on -a **two-node AWS `d3.xlarge` cluster with real local HDD storage**. It supersedes -the earlier, confounded 2 MiB numbers in `docs/single-trip-get-phase1-handoff.md` -(which repeated a small object set and was polluted by buckit's in-process metadata -cache — see "Method corrections" below). - -## Headline - -Pooled across **300 cold first-byte samples per arm** (3 rounds × 100 distinct -objects), enabling the single-trip fast path (`FAST_GET=1`) reduced **median cold -TTFB by ~14%** versus the canonical `xl.meta` path (`FAST_GET=0`), with a ~8–12% -reduction across p75–p95. The bottom decile is within noise. The signal is real but -**smaller than the design §6 HDD prediction (≥25%)** and the rig is very noisy, so -only the pooled aggregate is trustworthy. - -## Rig - -- Two AWS `d3.xlarge` nodes: - - node1 — public `3.81.144.6`, private `172.31.44.123` - - node2 — public `35.173.238.193`, private `172.31.37.19` -- Storage: three `/mnt/data0{1,2,3}` XFS mounts per node, **real dense HDD** local - instance storage (D3 family). - - **`rotational=0` is a Nitro artifact, not SSD.** D3/D3en expose spinning HDDs - through the NVMe interface, so the kernel reports the `/dev/nvme*` devices as - non-rotational even though the media is HDD. Confirm via IMDSv2 instance-type - (`d3.xlarge`), not the rotational flag. -- Topology: one pool, one set, 6 drives, **EC:3** (standard parity 3, RRS 1). -- Endpoints use DNS aliases `buckit-node1` / `buckit-node2` in `/etc/hosts` - (buckit rejects hostnames with underscores). -- SSH: `ssh -i /Users/rooseveltlai/Downloads/buckit.pem ubuntu@`. - -## Dataset - -- Bucket: `singletrip-cold-2m-big` -- 100 distinct objects, 2 MiB each, independent random content. -- Loaded once under `FAST_GET=1` so the `current/part.1` shadow exists on every - drive (verified: `current/part.1` = 700140 B per drive vs `xl.meta` ~382 B). -- Presigned URLs (12 h) generated for all 100 objects; `curl` issues them - host-local on node1 against `http://127.0.0.1:9000`. - -## Method - -The whole experiment is a runtime A/B on **identical on-disk data and one binary**; -the only variable is `BUCKIT_FAST_GET`. - -1. **Distinct cold objects.** Each of the 100 objects is GET once per sweep — a - genuine first-touch. (Re-reading the same object is not an independent cold - sample, even with caches dropped.) -2. **Cold on both nodes.** Before every GET, page cache is dropped on **both** - nodes (`sudo sh -c "sync; echo 3 > /proc/sys/vm/drop_caches"`). This is a - distributed cluster, so dropping only node1 would leave shards warm on node2. -3. **Fresh restart per arm.** The server is restarted before each arm to clear - buckit's **in-process metadata cache** (which `drop_caches` does not clear). -4. **Measurement.** Client-side TTFB only, via - `curl --http1.1 -sS -o /dev/null -w '%{time_starttransfer}'`, executed on node1 - against the loopback endpoint. **No `mc admin trace` server-side TTFB was - captured in this run** — these are curl numbers. -5. **Rounds.** 3 alternating rounds of [ON sweep, OFF sweep], 100 GETs each, - pooled to 300 samples per arm. -6. **Fast-path verification.** After each ON sweep the counters confirmed the path - actually fired: `minio_api_requests_fast_get_hits_total` == GET count and - `fast_get_fallbacks_total` == 0 (line absent when zero). OFF sweeps show 0 hits. - -## Results - -### Pooled cold TTFB, 2 MiB (n = 300 per arm) - -| percentile | OFF (ms) | ON (ms) | ON vs OFF | -|---|---:|---:|---:| -| p10 | 18.03 | 19.59 | +8.7% | -| p25 | 24.48 | 23.74 | −3.0% | -| **p50** | **36.68** | **31.62** | **−13.8%** | -| p75 | 47.43 | 42.78 | −9.8% | -| p90 | 56.26 | 52.04 | −7.5% | -| p95 | 65.49 | 57.40 | −12.4% | -| mean | 37.19 | 34.15 | −8.2% | - -### Per-round medians (illustrating rig noise) - -| round | ON median (ms) | OFF median (ms) | -|---|---:|---:| -| 1 | 37.69 | 33.67 | -| 2 | 25.72 | 43.17 | -| 3 | 31.35 | 29.09 | - -Round-to-round variance (ON 25.7–37.7; OFF 29.1–43.2) **exceeds the ON/OFF gap**, -so no single round is meaningful — only the pooled aggregate is. Round 1 even shows -ON slower than OFF. This matches the run-order inconsistency noted in the handoff. - -## Method corrections (why earlier 2 MiB numbers were wrong) - -An initial attempt repeated the **same 10 objects** across cycles and reported a -~31–54% "win." That was an artifact: - -- **In-process metadata cache.** `drop_caches` clears only the Linux page cache; - buckit keeps its own metadata cache that survives it. Re-GETting the same object - was served partly from that cache, so cycles 2–3 were ~5–8 ms while cycle 1 was - ~20–30 ms — a warmup curve, not a fast-path effect. -- **Fix:** many distinct objects (touch each once) + fresh server restart per arm. - The honest signal then collapsed to the ~14% median above. - -## Caveats - -- **curl-side only.** Server-side `mc admin trace` TTFB was not captured this run; - curl TTFB includes connect/request-send overhead the server timer would not see - (small over loopback, but present). -- **Real HDD, but noisy.** Despite being genuine HDD, the D3-behind-Nitro path has - its own caching/queueing; treat per-round numbers as noise and the pooled median - as the result. -- **Magnitude below §6.** ~14% median vs the design's ≥25% HDD prediction. Plausible - causes: EC:3 metadata fan-out is parallel across spindles (≈ one seek of - wall-clock, not six), and fixed RPC/processing overhead dilutes the single saved - open/seek as a percentage of TTFB. -- **GET only.** The shadow doubles write cost; PUT numbers from this rig are - meaningless by design. - -## Raw artifacts - -Local (not checked in), under `/tmp/st-hdd-bench/` on the operator's Mac: - -- `agg-on.txt`, `agg-off.txt` — 300 pooled samples per arm -- `r{1,2,3}-{ON,OFF}.txt` — per-round 100-sample sweeps -- `urls-2m-big-signed.txt` — the 100 presigned URLs -- driver scripts: `cold-sweep.sh`, `multi-round.sh`, `launch.sh` (on the nodes), - `stats.sh` diff --git a/docs/single-trip-hdd-bench/two-node-d3-640kib-ec2.md b/docs/single-trip-hdd-bench/two-node-d3-640kib-ec2.md deleted file mode 100644 index 517a464a8..000000000 --- a/docs/single-trip-hdd-bench/two-node-d3-640kib-ec2.md +++ /dev/null @@ -1,104 +0,0 @@ -# Two-Node D3 HDD Bench — 640 KiB Cold TTFB (EC:2) - -June 5, 2026 single-trip GET phase-1 cold-TTFB A/B on the two-node AWS `d3.xlarge` -real-HDD cluster, reconfigured to **EC:2** so the 6-drive set has **4 data drives -+ 2 parity**. Companion to `docs/single-trip-hdd-bench/two-node-d3-2mib.md` (same -rig, same method); read that first for the rig and method details. - -## Headline - -At 640 KiB the fast path is **not a clean win — it is a crossover**. Pooled across -300 cold samples per arm: - -- **Low percentiles favor OFF, dramatically:** p10 ON +288%, p25 ON +210% (OFF - floor ~2 ms vs ON floor ~4 ms). -- **Median is roughly a wash, slightly worse for ON:** p50 +11.8%. -- **The tail favors ON:** p90 −26%, p95 −26%, mean −18%. - -The fast path's per-request overhead (EOF-stream open + cancel-not-drain close, no -connection reuse — design §9.9) sets a ~4 ms floor that dominates small/fast GETs, -while its saved metadata seek only pays off on the seek-bound tail. At 2 MiB the -saved seek dominated (clean ~14% median win); at 640 KiB the fixed overhead and the -seek-saving roughly cancel in the middle and the result splits by percentile. - -## Config change vs the 2 MiB run - -- Standard storage class set to **`EC:2`** via `MINIO_STORAGE_CLASS_STANDARD=EC:2` - (env, both nodes; verified `mc admin info` → `EC:2` and process environ). -- 6 drives → **4 data + 2 parity**. A 640 KiB object splits into 4 × 160 KiB data - shards (> 128 KiB inline cutoff), so a standalone `DataDir/part.1` and shadow are - written. Verified on disk: `xl.meta` 370 B, `current/part.1` 164,896 B - (160 KiB shard + header). -- Dataset: bucket `singletrip-cold-640k-ec2`, 100 distinct 640 KiB objects, random - content, loaded under `FAST_GET=1`. - -## Method - -Identical to the 2 MiB run: 100 distinct cold first-touch objects per sweep, page -cache dropped on **both** nodes before each GET, **fresh server restart per arm** -(clears buckit's in-process metadata cache), client-side **curl** TTFB -(`time_starttransfer`) host-local on node1 against loopback, 3 alternating rounds → -300 samples/arm. No `mc admin trace` (curl-side only). - -**Fast-path verified:** a 30-GET spot check on this dataset returned -`fast_get_hits_total = 30`, `fast_get_fallbacks_total = 0` — the ON arm is genuinely -single-trip, so the crossover below is not a silent-fallback artifact. - -## Results - -### Pooled cold TTFB, 640 KiB EC:2 (n = 300 per arm) - -| percentile | OFF (ms) | ON (ms) | ON vs OFF | -|---|---:|---:|---:| -| p10 | 2.23 | 8.66 | +287.8% | -| p25 | 5.45 | 16.89 | +210.1% | -| p50 | 20.25 | 22.63 | +11.8% | -| p75 | 33.31 | 29.71 | −10.8% | -| p90 | 53.96 | 39.76 | −26.3% | -| p95 | 61.28 | 45.58 | −25.6% | -| mean | 30.22 | 24.71 | −18.2% | - -Floors (5 fastest of 300): OFF ~1.7–1.9 ms, ON ~3.9–4.2 ms. - -### Per-round medians (rig noise) - -| round | ON median (ms) | OFF median (ms) | -|---|---:|---:| -| 1 | 21.47 | 10.30 | -| 2 | 19.66 | 28.52 | -| 3 | 27.67 | 19.75 | - -As at 2 MiB, round-to-round variance swamps the per-round ON/OFF gap (round 2 OFF -even had a single 677 ms outlier). Only the pooled aggregate is meaningful. - -## Interpretation - -- **The ~2 ms OFF floor is not a fully-cold read.** 640 KiB with a 370 B `xl.meta` - is tiny; the D3-behind-Nitro device layer appears to serve a chunk of these from - a cache `drop_caches` cannot clear, giving OFF a large population of ~2 ms - responses. The OFF path can return that fast because its cached-metadata + small - read has little fixed machinery. -- **The ON floor is set by the fast path itself (~4 ms).** Opening `current/part.1` - as an EOF stream per disk and closing with cancel-not-drain (no connection reuse) - is a heavier per-request path than a warm small `xl.meta` read. For small objects - where the disk work is cheap, this fixed cost makes ON *slower* at the fast end. -- **ON wins only where a real seek happens** — the p75–p95 tail and the mean — - because there the one saved metadata seek outweighs the fixed overhead. -- **Net:** at 640 KiB on this rig the fast path is a tail/mean improvement - (−18% mean, −26% p90) bought at the cost of low-percentile and median latency. - This is the opposite balance from 2 MiB and is consistent with §9.9's - cancel-close-churn caveat scaling worse as object size shrinks. - -## Caveats - -Same as the 2 MiB doc: curl-side only; real-HDD but very noisy (pooled only); -GET-only; magnitudes are rig-specific. Additionally: the ~2 ms OFF floor suggests -imperfect cold isolation on the D3/Nitro device cache, which inflates OFF's -low-percentile advantage — a server-side `mc admin trace` cross-check would help -separate device-cache effects from the path difference. - -## Raw artifacts - -Local under `/tmp/st-hdd-bench/` (not checked in): `agg-on.txt`/`agg-off.txt` -(300 samples/arm), `r{1,2,3}-{ON,OFF}.txt`, `urls-640k-ec2-clean.txt`, -`multi-round-640k.log`. diff --git a/docs/single-trip-parallelreader-deadlock-handoff.md b/docs/single-trip-parallelreader-deadlock-handoff.md deleted file mode 100644 index f99618a32..000000000 --- a/docs/single-trip-parallelreader-deadlock-handoff.md +++ /dev/null @@ -1,145 +0,0 @@ -# Single-Trip `parallelReader` Deadlock Handoff - -## Context - -Branch: `fix/singletrip-recon-parallelreader` - -The eager single-trip GET path prefetches the first encoded block from local -shards, keeps remote shard streams positioned after the direct header, and calls -`Erasure.Decode` with local readers marked as preferred. - -Claude's current uncommitted `cmd/singletrip-read.go` change already replaces the -old sparse, preselected M-reader layout with the complete agreeing M+N reader -layout. The hang remained after that change. - -Observed blocked stack: - -```text -parallelReader.Read at cmd/erasure-decode.go:161 (chanrecv) -Erasure.Decode -getObjectWithSingleTripInfo at cmd/singletrip-read.go:434 -``` - -## Root Cause - -`parallelReader.preferReaders` reordered `p.readers` to put preferred local -readers first, but did not apply the same permutation correctly to -`p.readerToBuf`. - -The old code was: - -```go -p.readers[next], p.readers[i] = p.readers[i], p.readers[next] -p.readerToBuf[next] = i -p.readerToBuf[i] = next -``` - -`readerToBuf` may already contain a permutation from an earlier swap. Assigning -the current indices instead of swapping the existing mapping values can create -duplicate output-buffer mappings. - -One relevant six-drive EC:2 case is M=4, N=2 with three local preferred readers. -For preferred positions `[1, 2, 3]`, the old logic produces: - -```text -readers: [1, 2, 3, 0, 4, 5] -readerToBuf: [1, 2, 3, 2, 4, 5] -``` - -The first four reads can all succeed, but readers at positions 1 and 3 both -write to output buffer 2. Only three distinct buffers become non-empty. -`canDecode()` therefore remains false for M=4. - -Successful reads send `false` to `readTriggerCh`. Once those notifications are -consumed, no goroutine sends another trigger and the channel remains open. -The loop at `cmd/erasure-decode.go:161` then waits forever on an empty channel. - -## Fix - -File: `cmd/erasure-decode.go` - -Apply the exact same swap to the mapping as to the readers: - -```go -p.readers[next], p.readers[i] = p.readers[i], p.readers[next] -p.readerToBuf[next], p.readerToBuf[i] = p.readerToBuf[i], p.readerToBuf[next] -``` - -This preserves a one-to-one permutation from reordered reader positions to the -readers' original erasure-shard buffer positions. - -## Regression Test - -File: `cmd/erasure-decode_test.go` - -Added `TestParallelReaderPreferReadersDoesNotDeadlock`: - -- Creates six one-byte readers with M=4. -- Marks positions `[1, 2, 3]` preferred, matching three local disks in the - two-host six-drive layout. -- Calls `parallelReader.Read` in a goroutine. -- Fails if it does not return within one second. -- Verifies that decode does not return an error. - -The existing uncommitted -`TestSingleTripDecodeMixedReadersReconstructNoHang` in -`cmd/singletrip-get_test.go` was also run. It exercises M=4/N=2 with three fast -local readers, one slow remote parity reader, and reconstruction of a missing -data shard. - -Targeted command: - -```sh -GOCACHE=/tmp/buckit-go-cache CGO_ENABLED=0 \ - go test -tags kqueue,dev ./cmd \ - -run 'TestParallelReaderPreferReadersDoesNotDeadlock|TestSingleTripDecodeMixedReadersReconstructNoHang' \ - -count=1 -timeout=45s -``` - -Result: PASS. - -`git diff --check` also passed. - -## Broader Test Caveat - -The existing `TestErasureDecode` suite was attempted separately: - -```sh -GOCACHE=/tmp/buckit-go-cache CGO_ENABLED=0 \ - go test -tags kqueue,dev ./cmd -run '^TestErasureDecode$' \ - -count=1 -timeout=2m -``` - -It did not reach a useful result because an unrelated goroutine panicked with an -integer divide by zero in `internal/ringbuffer/ring_buffer.go:212`, reached from -`xlStorage.writeAllDirect`. This occurred during test storage setup, outside the -changed `parallelReader` logic. - -## Validation Still Needed - -1. Review the full-layout changes already present in the dirty working tree and - keep them separate from this mapping fix when committing. -2. Build and deploy the patched binary to both benchmark hosts. -3. Repeat the stress workload that previously hung, ideally with goroutine dumps - enabled so any remaining block can be compared with the old line-161 stack. -4. Validate returned bytes and fast-path hit/fallback counters after stress. -5. Run longer ON/OFF tests for 640 KiB, 1 MiB, and 2 MiB only after the hang is - shown to be resolved. -6. Investigate remote `ReadFileStream` cancellation separately. It can cause - resource pressure under load, but it does not explain the specific empty - `readTriggerCh` receive described above. - -## Working Tree Warning - -At the time of this handoff, these files already contain uncommitted work: - -```text -cmd/erasure-decode.go -cmd/erasure-decode_test.go -cmd/singletrip-get_test.go -cmd/singletrip-read.go -.tmp/ -``` - -Do not discard the existing `singletrip-read.go` or `singletrip-get_test.go` -changes when isolating or committing this fix. diff --git a/docs/single-trip-recon-review.md b/docs/single-trip-recon-review.md deleted file mode 100644 index 565dc06da..000000000 --- a/docs/single-trip-recon-review.md +++ /dev/null @@ -1,431 +0,0 @@ -# Single-Trip Fast GET — Prefer-Local + Parity Reconstruction: Review Guide - -Audience: a reviewer (Claude Code or human) picking up the change cold. This -explains **what changed, why, and how it was validated**, and points at the parts -that most need scrutiny. Companion doc: `single-trip-parallelreader-deadlock-handoff.md` -(Codex's root-cause note). - -Branch: `fix/singletrip-recon-parallelreader` (built on the single-trip phase-1 work). - ---- - -## TL;DR - -1. **A general erasure-coding bug was found and fixed** in - `parallelReader.preferReaders` (`cmd/erasure-decode.go`) — a one-line fix. It is - independent of single-trip and should be reviewed/committed on its own. -2. **The single-trip fast GET path gained prefer-local shard selection + parity - reconstruction** (and an opt-in eager first-block prefetch). This is what first - *exercised* the latent `preferReaders` bug. -3. **Result (2-node `d3.xlarge` real-HDD rig, EC:2, reboot-cold, n=300/arm/size):** - the fast path is a **modest, size-dependent** cold-TTFB win, not the −63% an - earlier `drop_caches` run suggested (that was a cold-cache artifact). Best variant - **ON-eager**: median **−11% / −5% / −12% / −7%** at 640 KiB / 1 MB / 2 MB / 4 MB, with the best - tail and mean at every size. `ON-stream` is erratic (−25% at 640 KiB but **+3%** - at 1 MB). See §3 "Rig reboot-cold A/B — AUTHORITATIVE." - ---- - -## 1. The core fix — `parallelReader.preferReaders` (review first) - -File: `cmd/erasure-decode.go` (1 line) + `cmd/erasure-decode_test.go` (regression test). - -```diff - p.readers[next], p.readers[i] = p.readers[i], p.readers[next] --p.readerToBuf[next] = i --p.readerToBuf[i] = next -+p.readerToBuf[next], p.readerToBuf[i] = p.readerToBuf[i], p.readerToBuf[next] -``` - -**Bug:** `preferReaders` reorders `p.readers` to put preferred (local) readers first. -It must apply the *same permutation* to `p.readerToBuf` (reader-position → -output-buffer-index). The old code *assigned current indices* instead of *swapping -the existing mapping values*. Across chained swaps this produces **duplicate -output-buffer mappings** — two readers write the same buffer, so fewer than -`dataBlocks` distinct buffers fill, `canDecode()` never becomes true, and -`parallelReader.Read` blocks forever on its trigger channel -(`cmd/erasure-decode.go:161`). - -Concrete EC:2 (M=4,N=2) case, preferred positions `[1,2,3]`: -``` -readers: [1, 2, 3, 0, 4, 5] -readerToBuf: [1, 2, 3, 2, 4, 5] ← buf 2 mapped twice; only 3 distinct buffers fill -``` - -**Why it was dormant:** `preferReaders` only reorders when some `prefer[i]` is true. -The canonical decode path sets `prefer[i] = disk.Hostname() == ""`, which is a -**no-op in distributed deployments with hostnamed endpoints** (`http://node/path` -→ local disks still report a non-empty `Hostname()`), so nothing was ever marked -preferred and the buggy branch never ran. The single-trip change below sets -`prefer[i] = disk.IsLocal()` (the correct signal), which finally exercised it. - -**Regression test:** `TestParallelReaderPreferReadersDoesNotDeadlock` — 6 readers, -M=4, prefer `[1,2,3]`, watchdog-fails if `Read` doesn't return in 1s. Verified -load-bearing: reverting the one-liner makes the test deadlock/fail; the fix makes -it pass. - -**Note:** consider whether the canonical path's `prefer[i] = disk.Hostname()==""` -(`cmd/erasure-object.go`, `cmd/erasure-healing.go`) should also be `IsLocal()` — -prefer-local is currently a no-op there in hostnamed-endpoint clusters. Out of scope -for this change, but worth a follow-up. - ---- - -## 2. Single-trip fast path changes - -All in `cmd/singletrip-read.go` unless noted. The fast path reads `current/part.1` -(direct header + shard, co-located in one file) instead of `xl.meta` + `DataDir/part.1`. - -What changed from the original single-trip fast path: - -- **Quorum relaxed to "any decodable M"** (`pickSingleTripFastInfo`): accept a - `directSig`-agreeing group with **≥ M distinct valid indices** (data and/or - parity), instead of requiring all M *data* indices. This is what allows parity - reconstruction. -- **Full M+N reader layout handed to the decoder** (`buildSingleTripFastInfo`): - builds a reader for **every agreeing shard** at its `erasureIndex-1` position — - the layout `parallelReader` is designed for. (An earlier attempt built a sparse - M-only/nil-padded slice; that was a dead end and is gone.) -- **Prefer-local selection via the decoder**: `getObjectWithSingleTripInfo` sets - `prefer[i] = disk.IsLocal()` (line ~432). `parallelReader` then reads the cheapest - M shards (locals first) and **reconstructs missing data from parity** if a local - shard is parity / a data shard is remote. This minimizes cross-node reads - (3 local + 1 remote on a 3+3 layout) instead of chasing data shards that may be - remote. -- **Close-after-response ordering**: only non-agreeing reads are closed in-path - (`closeSingleTripHeaderReadsKeepGroup`); the selected readers' streams are owned - by `info.readers` and closed after the response (`closeBitrotReaders`), matching - the original path. (Closing partially-read remote streams *in-path before the - response* was investigated as a hang cause and ruled out; the real cause was §1.) -- **Eager first-block prefetch (opt-in, `BUCKIT_FASTGET_EAGER=1`)**: for a LOCAL - shard, `readSingleTripHeader` reads the header **and first encoded block** in one - go off the open stream, then decode serves block 0 from memory and streams the - rest (`multiReadCloser` = first-block buffer + remaining stream). Removes the - header→body barrier so local shards decode instantly. Remote shards keep a live - stream. **Default OFF** (see §3 for why). - -Supporting: -- `cmd/singletrip-get.go`: `globalFastGetEager` env gate. - ---- - -## 3. Validation - -### Unit tests (`-tags kqueue,dev`, both `BUCKIT_FASTGET_EAGER` on and off) - -- `TestParallelReaderPreferReadersDoesNotDeadlock` — the core-fix regression guard. -- `TestSingleTripFastGetReconstructsFromParity` — deletes a data shard's shadow so - the fast path must reconstruct from parity; asserts bytes + fast-hit. -- `TestSingleTripDecodeMixedReadersReconstructNoHang` — `erasure.Decode` with - instant + slow readers + forced reconstruction, watchdog'd. (Note: this passes - even *without* the §1 fix — the deadlock needs the real `preferReaders` reorder - path; kept as a decode-level guard.) -- Full `SingleTrip|FastGet` suite passes both modes. - -### Rig reboot-cold A/B — AUTHORITATIVE (2× `d3.xlarge`, EC:2 = 4 data + 2 parity, 3 local + 3 remote) - -This is the trustworthy result; the `drop_caches` pooled section further below is -**superseded** (see "Why reboot-cold" — small objects didn't re-cool under -`drop_caches`, inflating the 640 KiB win). - -Method: fresh AWS `d3.xlarge` pair, one 6-drive EC:2 set, 100 distinct objects/size -(640 KiB, 1 MB, 2 MB, 4 MB) seeded once. Per arm-cycle: **reboot both hosts** (clears all -RAM cache; instance-store survives reboot, fstab auto-remounts), relaunch the arm, -wait for `6 drives online` + a readiness probe on a *throwaway* object (so the -post-launch 503 window never touches measured objects and they stay genuinely cold), -then one cold pass = 100 distinct first-touch GETs. **3 reboot cycles × 100 = -300 cold samples/arm/size.** Arms on one binary: **OFF** `BUCKIT_FAST_GET=0`; -**ON-stream** `FAST_GET=1, EAGER=0` (prefer-local + reconstruction); **ON-eager** -`FAST_GET=1, EAGER=1` (adds first-block prefetch). 0 × 503 across all runs. - -Cold first-byte TTFB medians (ms): - -| size | OFF | ON-stream | ON-eager | -|---|---:|---:|---:| -| 640 KiB | 14.75 | **11.12 (−25%)** | 13.20 (−11%) | -| 1 MB | 22.48 | **23.26 (+3%)** ⚠ | 21.24 (−5%) | -| 2 MB | 21.16 | 19.47 (−8%) | **18.56 (−12%)** | -| 4 MB | 23.15 | 21.86 (−6%) | **21.60 (−7%)** | - -Cold first-byte TTFB **p90** (ms) — the tail, where the variants diverge most: - -| size | OFF | ON-stream | ON-eager | -|---|---:|---:|---:| -| 640 KiB | 24.66 | 26.02 (**+6%**) | **22.59 (−8%)** | -| 1 MB | 37.46 | 34.51 (−8%) | **28.19 (−25%)** | -| 2 MB | 28.77 | 32.77 (**+14%**) | **24.78 (−14%)** | -| 4 MB | 37.52 | 28.93 (−23%) | **27.53 (−27%)** | - -At p90 **ON-eager wins at every size** (−8 / −25 / −14 / −27%), and by *more* than at -the median — the eager block-0-from-memory decode is what protects against the -slow-shard-open tail. **ON-stream regresses at p90 for 640 KiB (+6%) and 2 MB (+14%)** -(lazy streaming has no tail protection), so on tail latency it is often *worse than -OFF*. This is the strongest argument for the ON-eager default. - -Full distributions (ms), `n=300/arm`: - -**640 KiB** (single block, 160 KiB shard): - -| pct | OFF | ON-stream | ON-eager | -|---|---:|---:|---:| -| p10 | 2.98 | 3.01 | 2.63 | -| p25 | 3.12 | 3.20 | 2.78 | -| **p50** | 14.75 | **11.12** | 13.20 | -| p75 | 19.19 | 18.86 | 19.27 | -| p90 | 24.66 | 26.02 | **22.59** | -| p95 | 27.69 | 32.17 | **24.48** | -| p99 | 61.33 | 68.18 | **45.77** | -| mean | 13.36 | 12.51 | **12.07** | -| max | 68.59 | 68.78 | 65.14 | - -per-cycle p50: OFF 14.1/15.5/14.5 · ONS 10.7/11.3/11.3 · ONE 14.4/11.6/13.0 - -**1 MB** (exactly one full block = `blockSizeV2`, 256 KiB shard — the largest single-block): - -| pct | OFF | ON-stream | ON-eager | -|---|---:|---:|---:| -| p10 | 14.83 | 13.06 | 14.09 | -| p25 | 18.79 | 17.77 | 17.34 | -| **p50** | 22.48 | 23.26 | **21.24** | -| p75 | 27.51 | 28.23 | **24.62** | -| p90 | 37.46 | 34.51 | **28.19** | -| p95 | 43.01 | 38.64 | **30.72** | -| p99 | 90.03 | 77.00 | 76.74 | -| mean | 24.23 | 23.57 | **21.55** | -| max | 121.54 | 78.19 | 81.82 | - -per-cycle p50: OFF 22.0/22.2/22.9 · ONS 23.3/23.7/22.7 · ONE 21.1/21.2/21.6 - -**2 MB** (2 blocks, 512 KiB shard): - -| pct | OFF | ON-stream | ON-eager | -|---|---:|---:|---:| -| p10 | 9.75 | 9.86 | 8.36 | -| p25 | 17.06 | 15.96 | 14.89 | -| **p50** | 21.16 | 19.47 | **18.56** | -| p75 | 23.48 | 23.61 | **21.42** | -| p90 | 28.77 | 32.77 | **24.78** | -| p95 | 34.17 | 38.62 | **25.61** | -| p99 | 42.75 | 52.13 | **40.32** | -| mean | 20.23 | 20.42 | **17.93** | -| max | 43.62 | 55.56 | 47.37 | - -per-cycle p50: OFF 19.8/21.5/21.5 · ONS 19.3/20.0/19.6 · ONE 18.4/18.6/18.8 - -**4 MB** (4 blocks, 1 MiB shard; eager prefetches only block 0 = 25% of shard): - -| pct | OFF | ON-stream | ON-eager | -|---|---:|---:|---:| -| p10 | 16.82 | 17.64 | 17.06 | -| p25 | 20.43 | 19.91 | 19.44 | -| **p50** | 23.15 | 21.86 | **21.60** | -| p75 | 30.96 | 25.24 | **24.83** | -| p90 | 37.52 | 28.93 | **27.53** | -| p95 | 40.90 | 35.56 | **31.58** | -| p99 | 92.94 | 67.37 | 69.42 | -| mean | 26.00 | 23.48 | **22.52** | -| max | 127.37 | 97.66 | 70.38 | - -per-cycle p50: OFF 23.1/23.3/23.4 · ONS 21.9/22.1/21.8 · ONE 21.6/21.6/21.5 - -**Reading the reboot-cold results:** - -- **The win is real but modest and *non-monotonic* in object size** — not the −63% - the `drop_caches` method suggested. The fast path saves the small metadata-read - phase (~2–3 ms, `xl.meta` is ~366 B); the dominant cost (cold block-0 *data* read, - paid by both OFF and ON) is unchanged. So the relative win tracks "how big is the - cheap metadata phase vs the data read," which isn't monotonic. -- **1 MB is the weakest case** (one full single block): **ON-stream regresses to - +3% vs OFF** (real — per-cycle 23.3/23.7/22.7 vs 22.0/22.2/22.9), while ON-eager - still wins −5%. Lazy streaming loses its edge exactly at the full-block boundary; - the eager block-0-from-memory decode is what keeps ON-eager ahead there. -- **ON-eager is the only consistent performer:** −11% / −5% / −12% / −7% across - 640 KiB / 1 MB / 2 MB / 4 MB, and it has the **best tail (p90/p95/p99) and mean at - *every* size** (e.g. 4 MB p90 27.5 vs OFF 37.5). The earlier "eager has a bad - single-block tail" was a transient under `drop_caches`; under clean reboot-cold its - tail is the best. -- **ON-stream is erratic:** −25% (640 KiB), **+3% (1 MB, regression)**, −8% (2 MB), - −6% (4 MB), with worse tails at the smaller sizes. The +3% at 1 MB is a **localized - full-single-block-boundary** effect — it recovers to a modest win at 2–4 MB — but - it shows lazy streaming has no reliable edge across sizes. -- Per-cycle medians are tight across the 3 reboots → reproducible cold; **0 × 503** - (readiness probe worked). - -**Recommended default: ON-eager** — the only variant that wins or ties at every size -and owns the tail/mean. ON-stream's 640 KiB median edge isn't worth its 1 MB -regression and worse tails. - -**Why reboot-cold (and why the `drop_caches` numbers below are superseded):** with -`drop_caches=3` before each GET, *small* objects (640 KiB) did **not** actually -re-cool — per-round medians ran 20.6 → 7.3 → 5.9 ms (round 1 cold, then effectively -warm), which inflated the headline 640 KiB win to −63%. Rebooting both hosts forces -a true cold cache every cycle; per-cycle medians are now stable (e.g. OFF 640 KiB -14.1/15.5/14.5), so these are the numbers to trust. - ---- - -### [SUPERSEDED] Rig pooled A/B (`drop_caches` method) - -> Kept for history. The 640 KiB figures here are inflated by the cold-cache issue -> described above; use the reboot-cold table instead. - -Method: cold first-byte TTFB, page cache dropped on **both** nodes before each GET, -single connection, `curl -w %{time_starttransfer}` issued host-local on node1 -against the loopback endpoint; 100 distinct objects/size; **3 alternating rounds × -100 = 300 samples/arm** (fresh server restart per arm to clear in-process metadata -cache). Arms: **OFF** = canonical `xl.meta` path; **ON-stream** = fast path, -`BUCKIT_FASTGET_EAGER=0` (prefer-local + reconstruction, no prefetch); **ON-eager** -= fast path, `BUCKIT_FASTGET_EAGER=1` (adds first-block prefetch). - -All six arms returned **300/300 HTTP 200, zero timeouts**. (Pre-fix, the -prefer-local/eager arms hung ~15% with 30 s stalls; see §1.) Full cold-TTFB -distribution (ms): - -**640 KiB (single-block):** - -| metric | OFF | ON-stream | ON-eager | -|---|---:|---:|---:| -| n | 300 | 300 | 300 | -| min | 4.11 | 3.91 | 3.51 | -| p10 | 4.48 | 4.36 | 3.90 | -| p25 | 4.92 | 4.56 | 4.07 | -| **p50** | **13.95** | **5.10** | **4.83** | -| p75 | 21.65 | 19.35 | 18.41 | -| p90 | 28.94 | 24.39 | 25.81 | -| p95 | 36.77 | 31.62 | 69.06 ⚠ | -| p99 | 47.90 | 50.25 | 707.64 ⚠ | -| max | 120.55 | 152.11 | 962.04 ⚠ | -| mean | 15.18 | 12.80 | 29.73 ⚠ | -| **median vs OFF** | — | **−63%** | **−65%** | - -**2 MiB (multi-block):** - -| metric | OFF | ON-stream | ON-eager | -|---|---:|---:|---:| -| n | 300 | 300 | 300 | -| min | 4.82 | 4.79 | 4.30 | -| p10 | 14.69 | 12.40 | 11.83 | -| p25 | 20.22 | 17.51 | 17.46 | -| **p50** | **23.87** | **21.94** | **20.40** | -| p75 | 30.98 | 29.12 | 23.95 | -| p90 | 41.88 | 39.45 | 29.37 | -| p95 | 47.09 | 46.48 | 39.46 | -| p99 | 79.56 | 63.40 | 72.18 | -| max | 192.44 | 131.99 | 102.88 | -| mean | 27.14 | 24.33 | 21.83 | -| **median vs OFF** | — | **−8%** | **−15%** | - -Per-round medians (rounds of 100 — shows the rig's run-to-run noise; read the -pooled distribution above, not any single round): - -| | OFF | ON-stream | ON-eager | -|---|---|---|---| -| 640 KiB | 20.6 / 7.3 / 5.9 | 12.4 / 12.8 / 4.7 | 4.7 / 15.9 / 4.3 | -| 2 MiB | 23.1 / 23.8 / 24.2 | 21.8 / 24.8 / 20.6 | 20.0 / 21.3 / 19.9 | - -How to read it: - -- **Both fast-path arms beat OFF at the median for both sizes.** The single-block - win is large (−63%/−65%); the multi-block win is smaller at the median but - ON-eager widens across the upper percentiles (p90 41.9 → 29.4, mean 27.1 → 21.8). -- **ON-eager's single-block tail (p95 69 / p99 708 / max 962 ms) is a single - temporal stall, not steady overhead.** All 10 of its >100 ms outliers fall in - round 2, positions 128–159 (a contiguous burst; six were 624–962 ms). Per-round - means: 10.4 / **69.2** / 9.6 ms. OFF and ON-stream had a mild round-2 bump too - (1–2 outliers each), so that window was generally noisy — but the arm order was - fixed OFF → ON-stream → ON-eager every round, so ON-eager always ran last and - caught the worst of it. The tail is therefore **temporally confounded**, not shown - to be steady eager overhead. Likely mechanism if eager *does* amplify such stalls: - `readSingleTripHeader` synchronously reads the first block from **all three local - disks** and `openSingleTripFastInfo` waits for **every** disk goroutine - (`g.Wait`), so one slow local HDD blocks the whole open phase, and the required - remote shard is only read *after* — making local-tail and remote-transfer - additive. ON-stream starts its selected body reads together after the header - barrier, allowing more overlap. (Drain-on-close is *not* the leading explanation: - cleanup runs after first-byte, the custom reader's `Close()` doesn't drain, and - ON-stream has the same unused remote streams without the tail.) - - **Rerun result (ON-eager, cold 640 KiB, n=300):** the - severe tail did **not** reproduce — median 9.2 ms, p95 37 ms, **max 206 ms, only - 2 GETs >100 ms** (vs the pooled run's 10 up to 962 ms), confirming a one-time - transient, not steady eager overhead. Per-GET profiling of the slow ones shows - they are **open-phase-bound and pinned to the slowest *remote* `ReadFileStream` - open**: `maxRemoteOpen` ≈ `openphase` (e.g. 204.2 ≈ 204.4 ms), while - `maxLocalBody` stays ≤4 ms on the slow GETs (56 ms max overall) and `firstbyte` is - ~0.3 ms. So the cause is **cross-node grid-open latency variance** — *not* a local - body stall (Codex's predicted 700–900 ms local `body_ms` did not appear) and not - drain-on-close. The open phase `g.Wait`s on **all 6** disks, so one slow remote - open blocks the whole GET; this exposure is **shared by ON-stream** (also opens all - 6), so it is not eager-specific. Mitigation if pursued: early-quorum — proceed once - the needed M shards (prefer-local) are ready instead of waiting on slow *unneeded* - remote opens. -- **640 KiB is noisy round-to-round** (OFF medians 20.6/7.3/5.9). The pooled p50 is - the right summary; don't over-read a single round. 2 MiB is much steadier. -- The −8% vs −63% gap between sizes is expected: at 640 KiB metadata/open dominates - cold TTFB (so collapsing it helps a lot), while at 2 MiB transfer is a larger - share of first-byte. - -Byte-correctness: md5 MATCH on both sizes (`current/part.1` served with `xl.meta` -absent on disk). Single-open-per-disk invariant preserved (one `ReadFileStream` per -participating disk; asserted by the end-to-end test). - -**Caveat:** the rig HDDs are AWS d3 (real spinning disks behind Nitro NVMe; the -`rotational=0` flag is a Nitro artifact). They share backing storage and a page-cache -layer, so treat these as **relative off-vs-on ratios**, not absolute HDD figures. -Raw per-sample data: `/tmp/st-hdd-bench/pool-{640k,2m}-{OFF,ONS,ONE}.txt`. - ---- - -## 4. Recommended default & open items - -- **Default = ON-eager** (`FAST_GET=1, EAGER=1`): per the authoritative reboot-cold - results (§3) it's the only variant that wins or ties at **every** size (−11% / −5% - / −12% / −7% at 640 KiB / 1 MB / 2 MB / 4 MB) and has the **best tail and mean at every size**. - The earlier "eager has a bad single-block tail" was a `drop_caches`-era transient; - under clean reboot-cold its tail is the best. -- **ON-stream (eager off) is not recommended as default:** erratic across sizes - (−25% at 640 KiB but **+3% regression at 1 MB**, −8% at 2 MB) with worse tails at - the smaller sizes. The full-block (1 MB) boundary is where lazy streaming loses to - OFF; eager's block-0-from-memory decode is what avoids that. -- **Magnitude expectation:** the cold-TTFB win is **modest (~5–12% at the median for - ON-eager, larger in the tail)**, because the fast path only elides the small - metadata-read phase (~2–3 ms; `xl.meta` ≈ 366 B) while the dominant cost — the cold - block-0 *data* read — is paid by both arms. Don't expect the (artifactual) −63%. -- **Earlier eager-tail investigation (now moot for the default but still informative):** - - **Rerun done (ON-eager, cold 640 KiB, n=300, profiling on):** tail did not - reproduce (median 9.2, p95 37, max 206 ms, 2 outliers). Slow GETs are - open-phase-bound with `maxRemoteOpen` ≈ `openphase` (up to 204 ms) and - `maxLocalBody` ≤4 ms — i.e. **cross-node `ReadFileStream`-open latency variance**, - blocking the whole GET because the open phase `g.Wait`s on all 6 disks. Shared by - ON-stream (also opens all 6), so not eager-specific. (Codex's predicted 700–900 ms - local `body_ms` did not appear; drain-on-close already ruled out.) - - **Mitigation:** early-quorum — proceed once the needed M shards (3 local + fastest - remote) are ready instead of waiting on the 2 slow *unneeded* remote opens. Helps - both fast-path arms. Also worth: investigate why the remote grid open occasionally - spikes to 100s of ms (node2/grid/network). -- **Prototype-only code to strip before merge if landing for real:** - the `BUCKIT_FASTGET_*` diagnostic env gates. -- **Out-of-scope follow-ups:** canonical-path `prefer` should use `IsLocal()` (§1 - note); remote `ReadFileStream` cancellation under load (separate from the deadlock). - ---- - -## 5. Reproduce - -Unit: -```sh -CGO_ENABLED=0 go test -tags kqueue,dev ./cmd \ - -run 'TestParallelReaderPreferReadersDoesNotDeadlock|SingleTrip|FastGet' -count=1 -``` - -Rig (scripts under `/tmp/st-hdd-bench/` on the operator's Mac, IPs hardcoded): -- `pooled-both.sh` → pooled OFF/ON-stream/ON-eager for 640 KiB + 2 MiB. -- Per-arm: `run/launch.sh EC:2` on both nodes. Raw data: - `pool-{640k,2m}-{OFF,ONS,ONE}.txt`. - -## 6. Suggested commit split - -1. `cmd/erasure-decode.go` + `cmd/erasure-decode_test.go` — the `preferReaders` - permutation fix (general; standalone). -2. `cmd/singletrip-*.go` (+ tests) — prefer-local + reconstruction + eager prefetch - + profiler, and the bench docs. diff --git a/docs/single-trip-warp-load-handoff.md b/docs/single-trip-warp-load-handoff.md deleted file mode 100644 index 6d951e7a7..000000000 --- a/docs/single-trip-warp-load-handoff.md +++ /dev/null @@ -1,1302 +0,0 @@ -# Single-Trip GET — Warp Load-Test Handoff (for Codex) - -Audience: Codex picking this up cold. Scope: **load-test the single-trip fast GET -path under concurrency** (warp) and characterize how it compares to the canonical -path. All AWS hosts from the last session were **terminated** — you must -re-provision (steps in §6). This doc records what was observed, the rig/method, how -to reproduce, and Codex's current troubleshooting theory and isolation plan (§9). - -Repo: `github.com/buckit-io/buckit`, working dir as checked out. Branches/binaries in §5. - ---- - -## 1. Results observed so far - -Warp GET, concurrency 64, 640 KiB non-inlined objects, EC:2, reboot-cold, working set -~5–7 GB reused via `--list-existing`: - -| arm | obj/s | vs OFF | -|---|---:|---:| -| **OFF** (canonical) | ~593–610 | — | -| **ON, open all 6 shadows** (committed baseline) | ~309 | −48% | -| **ON, early-release** (open 6, cancel unused immediately) | ~309 (cold) | −48% (no change cold) | -| **ON, exactly-read-quorum** (open only 4) | 208 (stream) / 254 (eager) | −58…−66% | - -Two raw observations, stated without interpretation: -- Under this concurrent load the single-trip path (ON) sustains roughly **half** the - throughput of the canonical path (OFF). -- The **exactly-read-quorum** variant (which opens *fewer* shadows) measured **lower** - throughput than the open-all-6 baseline, not higher. - -Separately (different workload, for context only): single-request **cold-latency** -A/B is documented in `docs/single-trip-recon-review.md` — there single-trip is a -modest win. That is a single-request test, not this concurrent-load test. - ---- - -## 2. What the two paths do (factual cost model) - -Canonical GET (OFF): read `xl.meta` from all disks (≈366 B, caches quickly), resolve -quorum, then read the M **data shards** from `DataDir/part.1` via **bounded reads** -(`ReadFileStream` with an exact byte length) on the data disks. There is a subtle but -important reason this cluster normally reads data indices 1..M: OFF does construct a -`prefer` mask, but uses `disk.Hostname() == ""`. With URL/hostnamed distributed -endpoints, even the local `xlStorage` disks have a non-empty endpoint host, so the -mask is all false. `parallelReader` therefore keeps erasure-index order and starts -the first M readers, which are the M data indices. OFF only reconstructs when one of -those data reads is missing/corrupt (or in a path-endpoint deployment where the old -locality test actually reorders readers). This corrects an earlier verbal analysis -that treated OFF as prefer-local on this rig. - -Single-trip GET (ON, `BUCKIT_FAST_GET=1`, `cmd/singletrip-read.go`): a co-located -shadow `object/current/part.1` = `[1 KiB fixed header][erasure shard]` exists. The -fast path opens it with **`ReadFileStream(..., 0, -1)` (to-EOF)**, reads the header -to rebuild `FileInfo` without `xl.meta`, then decodes. The server side of a to-EOF -`ReadFileStream` runs `xioutil.Copy` (128 KiB buffer) over a held connection (grid -for remote disks). `BUCKIT_FASTGET_EAGER=1` additionally reads the first block in the -open goroutine. - -On-disk layout for a **non-inlined** object (verified on disk): both -`object//part.1` (e.g. 262176 B for a 1 MiB object — the canonical shard) -**and** `object/current/part.1` (263200 B = header + shard) exist; `xl.meta` ≈ 366 B. - ---- - -## 3. Measurements established during the session (facts) - -- **Controller is not the bottleneck.** During load the warp controller was ~82% - idle (first a 2-vCPU box, later a 4-vCPU t3.xlarge). Node-side iowait was high and - the data HDDs were ~100% util. -- The reboot-cold load run is a **cold→warm mix**: a ~5–7 GB working set caches in - ~13 s at the observed ~430 MiB/s, so a 40 s run starts disk-bound and warms. -- During the exactly-read-quorum run, `fast_get_fallbacks` stayed ~0 (checked - mid-run: 5 total) and `fast_get_hits` climbed — i.e. the fast path was engaging, - not silently falling back. -- A **fully warm** (no-reboot) ON smoke reached ~744–944 obj/s; the throughput gap - vs OFF appears when the run involves disk I/O / streaming, less so fully cached. -- md5 of returned bytes matched the source for all valid (non-inlined) arms tested. - ---- - -## 4. Gotchas that cost us time (do not repeat) - -1. **Object size must be ≥ 640 KiB (non-inlined).** `smallFileThreshold = 128 KiB` - (`cmd/xl-storage.go`); MinIO inlines when `ShardFileSize ≤ 128 KiB`. For EC:2 - (4 data) that means objects **≤ 512 KiB are inlined**: stored entirely in - `xl.meta`, with **no `current/part.1` shadow** — so the fast path **cannot - engage**; every ON GET probes 6 shadows (3 of them remote RPCs), gets "not found", - and **falls back** to canonical. Our first warp run used 512 KiB and was - **invalid** (all three arms ran identical canonical code). Always confirm - `fast_get_hits` climbs and `fast_get_fallbacks` stays ~0 before trusting an ON - number. (Observed side effect: with `FAST_GET=1`, inlined/small objects each pay a - failed cross-node shadow probe + fallback.) - -2. **Cold cache requires a reboot, not `drop_caches`.** Small objects do not re-cool - under `echo 3 > drop_caches` (640 KiB ran 20→7→6 ms across rounds). **Reboot both - nodes** before each arm. Instance-store **survives reboot** (only stop/terminate - wipes it); mount the XFS volumes via `/etc/fstab` **by UUID** with `nofail` so a - reboot auto-remounts them. - -3. **Post-launch 503 settling:** the cluster reports "6 drives online" a few seconds - before it actually serves. Probe a **throwaway** object until HTTP 200 before - measuring, so the 503 window does not pollute samples and measured objects stay - cold. - -4. **Volume spec = ONE combined ellipsis:** `buckit ... server - "http://buckit-node{1...2}:9000/mnt/data0{1...3}"` → one pool, one 6-drive EC:2 - set. Passing two separate args (one per node) makes **two pools of 3 drives** and - EC:2 fails at startup with *"parity 2 should be ≤ 1"*. - -5. **Warp has no v1.5.0 release binary.** Cross-build on a Mac: - `GOOS=linux GOARCH=amd64 go install github.com/minio/warp@latest` → - `$(go env GOPATH)/bin/linux_amd64/warp`; scp to the controller. - -6. **PUT is slow on HDD EC:2** (~9–15 obj/s sustained). Build the warp working set - **once** and reuse it with `warp get --list-existing` across all arms; do not - re-PUT per arm. - -7. **Warm vs cold are very different.** A fully-warm smoke (no reboot) reads far - higher than a reboot-cold run. Compare arms only under the **same** protocol - (the reboot-cold orchestrator), never a warm smoke of one arm vs a reboot-cold - number of another. - ---- - -## 5. Code: branches, binaries, variants - -Committed baseline: **`fix/singletrip-recon-parallelreader`** — the working -single-trip fast path (prefer-local + parity reconstruction) plus the general -`parallelReader.preferReaders` fix; 3 clean commits. This is the **open-all-6** -version (the −48% arm). Its internals are described in -`docs/single-trip-recon-review.md`. - -Perf experiments — **uncommitted working tree** on **`perf/singletrip-early-release`** -(`cmd/singletrip-read.go`, `cmd/singletrip-get_test.go`). The tree currently holds -**both** layered changes: -- **Early-release**: reintroduced `cancelReadCloser` (per-stream context cancel); - selection picks M prefer-local and **cancels the unused N immediately** rather than - after the response. -- **Exactly-read-quorum** (on top): `openSingleTripFastInfo` opens **only the read - quorum** of disks (prefer-local), computed from `setDriveCount - defaultParityCount` - (+1 when `data == parity`); decode `prefer` is all-false (exactly M readers). -- Tests updated accordingly: `TestSingleTripFastGetEndToEnd` asserts exactly - read-quorum opens (`assertSingleTripQuorumOpens`); the former reconstruct test is - now `TestSingleTripFastGetFallsBackWhenQuorumShadowMissing`. All single-trip + - `ParallelReader` unit tests pass with eager on and off. - -Prebuilt linux/amd64 binaries (operator Mac, `/tmp/st-hdd-bench/`): -- `buckit-fixed` — committed baseline (open all 6) = the −48% arm. -- `buckit-earlyrel` — early-release. -- `buckit-rquorum` — exactly-read-quorum = the −58…−66% arm. -(Other `buckit-*` are older cold-latency experiments; ignore for the load test.) - -Build: `GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -tags kqueue -trimpath -o .` -Unit tests: `CGO_ENABLED=0 go test -tags kqueue,dev -run 'SingleTrip|FastGet|ParallelReader' ./cmd/` -(run with and without `BUCKIT_FASTGET_EAGER=1`). - ---- - -## 6. Re-provision the rig (hosts are gone) - -Need: **2× d3.xlarge** (x86 — dense-HDD D3 is x86-only; `rotational=0` is a Nitro -artifact, the media is real HDD) and **1 controller** (t3.xlarge). SSH key: -`/Users/rooseveltlai/Downloads/buckit.pem`, user `ubuntu`. The operator launches EC2 -and provides public + private IPs. - -Per cluster node: -1. `mkfs.xfs -f` each of `/dev/nvme1n1 nvme2n1 nvme3n1`; mount to `/mnt/data0{1,2,3}`; - add to `/etc/fstab` **by UUID** with `defaults,noatime,nofail`; `chown -R ubuntu`. - (`/tmp/st-hdd-bench/provision-node.sh`.) -2. `/etc/hosts` on both nodes: ` buckit-node1`, ` buckit-node2` - (underscores rejected; use hyphen names). -3. `mkdir -p /home/ubuntu/singletrip-bench/{bin,run,config,logs,results,data}`. -4. scp `bin/buckit` (a §5 binary) + `bin/mc` - (`curl -sSL https://dl.min.io/client/mc/release/linux-amd64/mc`) + `run/launch.sh`. - -`run/launch.sh` (in `/tmp/st-hdd-bench/launch.sh`) — args ` [SC]`, honors -`BUCKIT_FASTGET_EAGER`: -``` -export BUCKIT_FAST_GET=$FG BUCKIT_FASTGET_EAGER=$EAGER -export MINIO_ROOT_USER=buckitadmin MINIO_ROOT_PASSWORD=buckitadmin -export MINIO_STORAGE_CLASS_STANDARD=EC:2 MINIO_CONFIG_ENV_FILE= -nohup buckit --config-dir $ROOT/config server --address :9000 --console-address :9001 \ - "http://buckit-node{1...2}:9000/mnt/data0{1...3}" > $LOG 2>&1 & -``` -Run the **same** command on **both** nodes; wait for `mc admin info` → -"6 drives online, EC:2". Controller: scp the cross-built `warp` to `~/warp`. - ---- - -## 7. Run the load A/B - -Working set (once, cluster up): -``` -warp put --host=:9000,:9000 --access-key=buckitadmin --secret-key=buckitadmin \ - --obj.size=640KiB --concurrent=32 --duration=10m --noclear --no-color # ~8k objects (~5GB) -``` -Create a throwaway probe object + presigned URL for readiness; verify engagement with -a short `warp get --list-existing` (check `fast_get_hits` climbs, `fast_get_fallbacks` -~0). - -Reboot-cold A/B orchestrator: **`/tmp/st-hdd-bench/warp-ab-multi.sh`** (edit the 3 IPs -at top). Per arm-cycle: reboot both → remount → launch arm → wait healthy → -wait-serving (probe) → `warp get --list-existing --concurrent=64 --duration=40s`. -3 rounds, **alternating arm order**. Arms: OFF (`fg=0`), ONS (`fg=1 eager=0`), -ONE (`fg=1 eager=1`). Output: `/tmp/st-hdd-bench/warp/multi.txt`. To test a variant, -overwrite `bin/buckit` with that binary and re-run. - -Production cleanup note: the prototype per-request fast-get profiler was removed -from the production candidate. Use external CPU/heap profiles, `iostat`, and regular -fast-get hit/fallback counters for future validation. - ---- - -## 8. Pointers - -- `docs/single-trip-recon-review.md` — fast-path internals + the - `parallelReader.preferReaders` fix + the cold-latency results. -- `docs/single-trip-parallelreader-deadlock-handoff.md` — earlier decode-deadlock - root cause (already fixed). -- `cmd/singletrip-read.go` (fast path), `cmd/singletrip-get.go` (env gates, - eligibility). -- `/tmp/st-hdd-bench/` (operator Mac) — binaries, scripts, `warp/multi.txt` - (last results). Scripts hardcode the now-dead IPs; re-point before use. - ---- - -## 9. Codex theory and troubleshooting plan - -This section is intentionally self-contained so a new Codex/Claude session can -resume without the previous conversation. - -### 9.1 What is and is not equivalent to OFF - -The current exactly-read-quorum experiment selects all 3 local physical disks and -then the first remote physical disk. Adding a hash/random rotation for the remote -disk fixes a serious hotspot, but it still does **not** make the selected shard set -identical to OFF: - -- OFF normally reads erasure data indices 1..4. Across objects, those four indices - rotate over all six physical disks according to `fi.Erasure.Distribution`. It - averages roughly 2 local + 2 remote shards for a request handled by either node. -- ON prefer-local reads 3 local + 1 remote regardless of erasure index. Some of the - local shards are parity, so ON commonly calls `ReconstructData`; OFF normally does - not. More importantly for HDD load, every request handled by a node fans out to - **all three of that node's HDDs**. -- With a uniformly hash-selected remote drive and warp balanced across both hosts, - the long-run read count per physical HDD should become approximately balanced and - similar to OFF. The per-request scheduling pattern remains different, however: - ON synchronizes three local disks on every request, while OFF's four data shards - rotate across all six disks. - -The uncommitted exactly-quorum code currently chooses the same first remote entry -from `er.getDisks()` for every request. That concentrates all cross-node reads from -node 1 onto one HDD on node 2, and vice versa. This is the leading explanation for -exactly-quorum becoming worse (208/254 obj/s) than open-all (309 obj/s). Do not draw -conclusions about exactly-quorum until the remote disk is selected deterministically -but evenly, for example `hash(object) % numberOfRemoteDisks`. - -### 9.2 Ranked hypotheses for the remaining ON-vs-OFF gap - -1. **Fixed remote-HDD hotspot in the current exactly-quorum prototype.** Expected - signature: one remote HDD per node has much higher `r/s`, queue depth, `await`, - and utilization than its two peers. Hash-rotating the remote selection should - materially improve exactly-quorum without changing CPU usage much. - -2. **Unbounded remote streams destroy connection reuse.** OFF opens each shard with - a known finite length. ON opens `current/part.1` using - `ReadFileStream(..., 0, -1)`, so the HTTP response is to-EOF/unknown-length. The - decoder reads exactly the expected shard bytes but may never perform the extra - read that observes response EOF/chunk termination. `cancelReadCloser.Close()` - then cancels even a successfully consumed selected stream, likely discarding the - inter-node connection rather than returning it to the idle pool. One new remote - connection per GET is expensive at hundreds of GET/s. Open-all is potentially - worse because it opens three remote streams and leaves two mostly unused. - Expected signature: high TCP connection creation/TIME_WAIT and a large gain when - selected full-object streams are read through EOF and closed normally rather - than canceled. Cancellation should remain for genuinely unused streams. - -3. **Eager serializes the local and remote body stages.** For a 640 KiB object the - entire ~160 KiB local shard is the first block. Each local open goroutine reads - header + full shard, and `openSingleTripFastInfo` waits for all selected goroutines - with `g.Wait()`. The selected remote goroutine reads only its header; its body is - consumed later by decode. Unless HTTP/socket buffering has already pulled the - remote body, the critical path becomes `slowest of 3 local body reads` followed - by `remote body read`. OFF starts its selected body reads together. This explains - why eager can improve TTFB yet still reduce saturated throughput. - -4. **Eager adds allocation and copy bandwidth.** At 640 KiB, eager allocates about - 3 x 160 KiB of temporary local shard buffers per GET. `parallelReader` then owns - its normal pooled decode buffers, and `io.MultiReader` copies the eager buffers - into them. At ~300 GET/s this is roughly 140 MiB/s of extra short-lived allocation - and copy traffic before accounting for metadata objects/maps. Expected signature: - higher `alloc_space`, GC CPU, `runtime.memmove`, and heap churn in ON-eager than - ON-stream. This likely explains only the eager-vs-stream delta, not the entire - ~50% ON-vs-OFF gap. - -5. **Prefer-local fan-out and reconstruction change HDD queueing.** Even after - remote rotation, every ON request handled by a node needs all three local HDDs. - A slow/queued local disk gates every such request. OFF rotates data indices over - all six drives and normally avoids Reed-Solomon reconstruction. Expected - signature: an OFF diagnostic changed to `prefer = IsLocal()` falls toward ON, or - an ON diagnostic that selects data indices 1..M rises toward OFF. - -6. **The duplicate shadow files may have worse physical placement.** - `writeSingleTripShadow` copies each canonical shard into a separately allocated - `current/part.1`. The shadow workload may be more fragmented or laid out less - favorably than canonical `DataDir/part.1` files on XFS/HDD. This is lower - confidence, but it is testable with `filefrag` samples and a direct read-only - microbenchmark over matched canonical/shadow files. - -7. **Warm-cache behavior masks the issue.** The fully warm ON smoke was fast, while - reboot-cold mixed runs were poor and HDDs reached ~100% utilization. That points - primarily to disk/stream scheduling rather than the header parser itself. Keep - cold, warm, and cold-to-warm results separate. - -### 9.3 Isolation variants — run in this order - -Use the same 640 KiB working set, concurrency 64, reboot protocol, alternating arm -order, and at least three rounds. Change one dimension per binary. - -1. **R0: reproduce controls.** Re-run OFF and committed open-all ON-stream/ON-eager - to verify the new hosts reproduce ~600 vs ~300 obj/s before testing changes. - -2. **R1: exactly-quorum + hash-rotated remote, stream mode.** Select all local disks - plus one remote using a stable object hash. Do not use process-global randomness; - reproducibility and even object-to-drive distribution matter. This isolates the - fixed-HDD hotspot without eager allocation/copying. - -3. **R2: same as R1, eager mode.** The R2-R1 delta is the eager prefetch cost/benefit - after disk selection is balanced. - -4. **R3: ON exact-data-indices.** Use the object's deterministic erasure - distribution to open the physical disks holding erasure indices 1..M, rather - than 3-local+1-remote. Keep the shadow file/header path. This most closely matches - OFF's shard set while retaining ON's to-EOF stream implementation. If R3 remains - slow, shard selection/reconstruction is not the main cause. - -5. **R4: OFF with `prefer[index] = disk.IsLocal()` (diagnostic only).** This makes - canonical files use the same 3-local+1-remote/reconstruction policy as ON while - retaining OFF's metadata and bounded stream path. If this OFF variant falls near - R1, prefer-local HDD scheduling/reconstruction is the cause. If it stays near - normal OFF, investigate ON's stream lifecycle and shadow layout. - -6. **R5: selected-stream EOF/connection-reuse diagnostic.** For a full-object GET, - do not cancel selected streams after the expected body is consumed. Read through - response EOF (or otherwise give the response a correct content length), then - close normally. Continue canceling unused streams. A large improvement identifies - connection churn. A two-request implementation (fixed 1 KiB header read followed - by a bounded body read) is acceptable as a diagnostic even though it is not the - final single-trip design. - -7. **R6: shadow-vs-canonical direct-read microbenchmark.** Bypass metadata and decode; - issue matched bounded reads of the same shard bytes from `current/part.1` and - `DataDir/part.1`. This isolates physical placement/path effects. - -Decision table: - -| result | implication | -|---|---| -| R1 greatly exceeds current exactly-quorum | fixed remote HDD hotspot confirmed | -| R3 approaches OFF, R1 does not | prefer-local/reconstruction/fan-out is costly | -| R4 falls near R1 | canonical path is fast partly because it reads data indices, not local shards | -| R5 approaches OFF | unknown-length stream close/cancel and connection reuse are primary | -| R6 shadow materially slower | duplicate-file placement/fragmentation is primary | -| Only R2 is worse than R1 | eager allocation/copy/barrier is the remaining issue | - -### 9.4 Capture alongside every arm - -- `iostat -dx 1` on both nodes, preserving per-device `r/s`, `rkB/s`, `await`, - `aqu-sz`, and `%util`. The per-HDD distribution is more important than aggregate - throughput. -- CPU plus allocation profiles for OFF, R1, and R2. Look for Reed-Solomon routines, - hashing, `runtime.memmove`, allocation/GC, HTTP transport, and syscall time. -- `ss -s`, TCP state counts, and connection-rate evidence. Compare selected-stream - cancel behavior with the EOF/drain diagnostic. -- Network bytes per node. OFF may read about two remote shards/request on average; - prefer-local ON should read one. If ON sends more bytes despite that, to-EOF - prestream/read amplification is occurring. -- Existing fast-path hit/fallback counters before and after each run. Record exact - deltas, not only snapshots. -- Add temporary counters/log sampling for selected physical disk, erasure index, - data-vs-parity, local-vs-remote, reconstruction invoked, streams opened, streams - canceled, and bytes consumed. Avoid per-request logging during the measured run; - use counters or sampling so instrumentation does not become the bottleneck. -- Warp per-host throughput. A large node1/node2 asymmetry points to selection or - disk imbalance rather than generic decoder CPU. - -### 9.5 Current recommendation before more data - -Do not merge the uncommitted exactly-quorum implementation as written. Its fixed -remote selection invalidates its throughput result and reduces failure tolerance: -one missing selected shadow causes fallback because there is no spare. First run R1 -through R5. The most promising production shape is likely early-quorum/open-racing: -start enough candidates to tolerate a slow/missing shard, proceed with the first -balanced decodable M, cancel only unused streams, and let selected streams complete -normally so transport connections remain reusable. - ---- - -## 10. New rig and quick isolation results (2026-06-07) - -Current hosts: - -| role | public IP | private IP | -|---|---|---| -| buckit_node1 | `54.235.32.111` | `172.31.34.45` | -| buckit_node2 | `98.83.23.248` | `172.31.37.173` | -| controller | `100.24.17.87` | `172.31.42.212` | - -Both storage nodes have three 1.8 TB instance-store HDDs formatted XFS and mounted -at `/mnt/data01..03` by UUID. The cluster is one six-drive pool, one erasure set, -EC:2. A reusable Warp set was seeded with 9,883 objects of 640 KiB (6.0 GiB logical). -The seed completed at 16.39 obj/s and was balanced across both hosts. Reboot/remount -and fast-path hit validation passed. - -Quick protocol: one reboot-cold arm, Warp GET `--list-existing`, concurrency 64, -duration 15 seconds (about 12-13 measured seconds). These short runs are fully cold -and therefore lower than the earlier 40-second cold-to-warm numbers, but repeated -controls are stable enough for directional isolation. - -| arm | obj/s | result | -|---|---:|---| -| OFF | 159.4 | initial control | -| OFF repeat | 165.9 | control reproduced | -| committed ON-stream/open-all | 87.4 | -45% vs first OFF | -| exactly-quorum, fixed remote | 85.5 | no improvement | -| exactly-quorum, object-hash rotated remote | 88.7 | small +3.7%; hotspot is secondary | -| exactly-quorum, exact data indices 1..M | 87.8 | reconstruction/locality is not primary | -| exact-data, bounded header + bounded body | 87.0 | to-EOF cancellation is not primary | -| exact-data, full contiguous shard prefetch + EOF drain | 87.0 | header/body barrier is not primary | - -Direct matched-file cold read on node1, 3,000 files (1,000 per HDD): canonical -`DataDir/part.1` took 36.19 s; shadow `current/part.1` took 38.25 s. The shadow was -only 5.7% slower, so physical placement/fragmentation cannot explain the ~45% S3 -gap. - -What is now ruled out as the dominant cause: fixed remote-HDD selection, -prefer-local parity reconstruction, selected shard count, unknown-length stream -cancellation/connection reuse, shadow-file physical placement, and eager/full-body -prefetch. The next high-value step is a paired OFF/ON CPU + allocation profile and -per-device `iostat` capture under the same short cold run. In particular, compare -the canonical `streamingBitrotReader` path with `singleTripStreamingBitrotReader`, -and count actual storage operations/bytes per request; the remaining gap is in the -Buckit fast-read pipeline rather than the files or EC2 hosts. - -Local artifacts: `/tmp/st-hdd-bench/warp-quick.sh`, `buckit-rquorum-rotated`, -`buckit-data-quorum`, `buckit-bounded-shadow`, and `buckit-full-prefetch`. The -worktree was restored to the pre-test exactly-quorum prototype after these failed -diagnostics; the experimental binaries remain for reference only. - ---- - -## 11. Concurrency sweep and ON-eager phase profile (2026-06-07) - -Additional local artifacts: - -- `/tmp/st-hdd-bench/warp-concurrency-sweep.sh` -- `/tmp/st-hdd-bench/warp-profile-point.sh` -- `/tmp/st-hdd-bench/profile-helper.go` -- `/tmp/st-hdd-bench/analyze-profiles.sh` -- `/tmp/st-hdd-bench/fastget-prof-c32/` - -The cluster was restored after profiling to ON-eager; `mc admin info` showed -`6 drives online, EC:2`. - -### 11.1 Concurrency sweep - -One reboot-cold arm per point, Warp GET `--list-existing`, 640 KiB objects, -duration 10 seconds: - -| concurrency | OFF obj/s | ON-eager obj/s | interpretation | -|---:|---:|---:|---| -| 1 | 18.93 | 23.32 | eager helps single sequential TTFB | -| 4 | 58.05 | 56.74 | roughly tied | -| 16 | 114.60 | 89.29 | ON-eager hits a scaling knee | -| 32 | 148.73 | 82.37 | OFF keeps scaling; ON plateaus | -| 64 | 129.43 | 81.49 | OFF overdriven but still higher | - -This confirms the problem is not single-request latency. ON-eager improves c=1, but -its saturated service time roughly doubles once concurrent random HDD reads build -queues. - -### 11.2 pprof/iostat controls - -Profile point: c32, Warp duration 12 seconds, profile window 18 seconds. - -| arm | obj/s | avg TTFB | median TTFB | p90 TTFB | p99 TTFB | -|---|---:|---:|---:|---:|---:| -| OFF | 148.90 | 192 ms | 167 ms | 433 ms | 814 ms | -| ON-eager | 86.00 | 373 ms | 341 ms | 718 ms | 1.131 s | - -CPU profiles sampled only about 3% CPU on both arms. Mutex profiles were effectively -empty. Heap/alloc profiles were dominated by startup/background allocations and did -not show a request-path allocation explanation. This is not CPU, GC, or mutex -limited. - -`iostat -dx` showed all six data HDDs active in both arms. OFF averaged roughly -160-171 read ops/s per disk with about 96-99% util; ON-eager averaged roughly -152-157 read ops/s per disk with about 87-94% util. The ON disk rate is lower, but -not low enough by itself to explain the full object-throughput gap. Also note the -profile run had substantial cache effects: Warp reported ~93 MiB/s logical OFF -while iostat only saw ~10 MiB/s raw disk reads, so treat this as a relative -request-latency profile rather than a pure cold-HDD bandwidth profile. - -### 11.3 Per-request ON-eager profiling - -Short ON-eager profile run: c32, Warp duration 8 seconds, reproduced the issue at -79.06 obj/s. Historical per-request logs were filtered to Warp `.rnd` objects. -The log field `remote=` was unreliable because it used `disk.Hostname() != ""`; -classify remote by `host="http://..."` instead when interpreting those artifacts. - -Node1 Warp requests: - -| phase | n | avg | p50 | p90 | p95 | p99 | -|---|---:|---:|---:|---:|---:|---:| -| `openphase_ms` | 410 | 195 ms | 171 ms | 355 ms | 411 ms | 581 ms | -| local eager body read | 1230 | 29 ms | 10 ms | 65 ms | 111 ms | 312 ms | -| remote open/header | 410 | 149 ms | 132 ms | 261 ms | 343 ms | 507 ms | -| decode/first-byte | 410 | 120 ms | 116 ms | 226 ms | 284 ms | 434 ms | -| open + decode | 410 | 316 ms | 307 ms | 492 ms | 582 ms | 724 ms | - -Node2 Warp requests: - -| phase | n | avg | p50 | p90 | p95 | p99 | -|---|---:|---:|---:|---:|---:|---:| -| `openphase_ms` | 239 | 543 ms | 553 ms | 832 ms | 931 ms | 1154 ms | -| local eager body read | 717 | 144 ms | 17 ms | 470 ms | 549 ms | 740 ms | -| remote open/header | 239 | 82 ms | 53 ms | 219 ms | 306 ms | 385 ms | -| decode/first-byte | 239 | 3 ms | 0 ms | 1 ms | 1 ms | 80 ms | -| open + decode | 239 | 546 ms | 556 ms | 834 ms | 932 ms | 1155 ms | - -Important observation: each successful GET opens exactly four shadows. In eager -mode, the three local selected shard bodies are read inside the header-open -goroutines before `openSingleTripFastInfo` returns; the selected remote shard is -kept as a live stream and may be read later by decode. That creates a staged -barrier: - -1. wait for remote header/open and three local body reads; -2. build `singleTripFastInfo`; -3. run decode, which may still need to read the remote body. - -On node1 this is visible as `openphase` plus an additional ~116 ms median decode -delay. In other words, eager does not necessarily read the four required shard -bodies in one fully overlapped decode wave. OFF's canonical path pays metadata -first, but the body reads happen together in `parallelReader`; that can win under -concurrency even though OFF uses more trips. - -Node2 is worse mostly because the max of three local eager body reads is very high -under HDD queueing. A single local-body read has p50 only 17 ms, but p90 470 ms; a -GET waits for three of them, so the per-request max lands near the long tail. - -### 11.4 Current theory - -The dominant ON-eager under-load cost is the fast path's fork/join structure, not -network round trips alone: - -- Eager moves full first-block shard reads into the metadata/open phase, so the S3 - response cannot start until those local HDD reads complete. -- The selected remote shard body is not prefetched in the same wave, so some - requests pay local-body wait and then remote-body/decode wait serially. -- At c16+, HDD random-read tail latency makes the max-of-three-local-body barrier - expensive; throughput becomes `concurrency / longer service time`. -- OFF has two logical trips, but its large body reads are driven by the decoder in a - single parallel wave and can keep more independent work in flight. - -This does not fully explain why ON-stream is also poor; ON-stream still has a -header-open barrier before decode, uses the shadow to-EOF stream shape, and may be -less able to overlap remote body work than canonical bounded `DataDir/part.1` -reads. But for ON-eager specifically, the phase profile shows the eager body -barrier is real and large. - -### 11.5 Next troubleshooting plan - -1. Add low-overhead counters, not per-request logs, for selected erasure indices, - local/remote classification using `IsLocal()`, whether decode reads a remote - body, and per-request selected shard count. -2. Test an "eager all selected M" diagnostic: after header quorum/selection, - prefetch the first block for all selected shards, including the selected remote - shard, concurrently. This should remove the node1 `openphase + remote decode` - serial cost. If c32 improves materially, the staged barrier is confirmed. -3. Test a "no eager body before selection" diagnostic: read headers only, select M, - then let decode read all selected bodies in parallel. This should approximate - OFF's body scheduling while preserving single-trip metadata. If it beats eager, - the local prefetch barrier is the main load issue. -4. Compare ON-stream, ON-eager-current, eager-all-selected, and no-eager-body at - c16/c32 only. Long multi-round tests are not needed until one variant moves the - plateau. -5. Fix profiling labels: replace `disk.Hostname() != ""` with `!disk.IsLocal()` in - the profile log, or log both fields explicitly. - -### 11.6 Diagnostic results: header-only and selected-eager - -Built current worktree as `/tmp/st-hdd-bench/buckit-headeronly-current`: - -- Linux amd64, build id `f2c3a73ca137717022bedeae1c91c74bf35d6671` -- sha256 `f8e9ccf7d4828b8c430173df0c26923e967570d930d653c499358f3cd0c1a613` - -Quick comparison, same binary, one reboot-cold arm, Warp duration 10 seconds: - -| concurrency | OFF | ON-stream/header-only | ON-eager | -|---:|---:|---:|---:| -| 16 | 114.29 obj/s | 79.09 obj/s | 92.15 obj/s | -| 32 | 145.21 obj/s | 82.98 obj/s | 82.02 obj/s | - -Conclusion: simply removing eager body prefetch is not a fix. Header-only still -plateaus at about the same level. Eager helps at c16 but not at c32. - -Then added diagnostic env `BUCKIT_FASTGET_EAGER_SELECTED=1` and built -`/tmp/st-hdd-bench/buckit-eager-selected`: - -- Linux amd64, build id `99646f61385ed72bf9436c5d988080a371ae549b` -- sha256 `be9f685d74b9ae005e4b99ee889d45e2b41554468a039d1693661f7cfaf22c2d` - -In selected-eager mode, the code reads headers first, selects the M decode shards, -then prefetches the first block for all selected shards concurrently. Focused tests -passed: - -```sh -GOCACHE=/tmp/st-hdd-bench/gocache GOTMPDIR=/tmp/st-hdd-bench/gotmp \ - CGO_ENABLED=0 go test -tags kqueue,dev -run 'SingleTrip|FastGet|ParallelReader' ./cmd/ - -BUCKIT_FASTGET_EAGER=1 BUCKIT_FASTGET_EAGER_SELECTED=1 \ - GOCACHE=/tmp/st-hdd-bench/gocache GOTMPDIR=/tmp/st-hdd-bench/gotmp \ - CGO_ENABLED=0 go test -tags kqueue,dev -run 'SingleTrip|FastGet|ParallelReader' ./cmd/ -``` - -Selected-eager quick result: - -| concurrency | ON-selected-eager | -|---:|---:| -| 16 | 90.76 obj/s | -| 32 | 86.42 obj/s | - -Conclusion: prefetching all selected M shards after selection also does not move the -plateau. It slightly improves c32 vs current eager/header-only, but remains far from -OFF. This weakens the earlier "local eager barrier" theory as the dominant cause. -The more likely remaining cause is the shadow stream path itself: unknown-length -`ReadFileStream(..., 0, -1)` plus `singleTripStreamingBitrotReader` and remote grid -stream behavior are materially worse under concurrent small-object reads than the -canonical bounded `DataDir/part.1` bitrot readers, even when the selected shard count -and eager scheduling are changed. - -Next best diagnostic: make the single-trip path use bounded reads for selected -shadow shards after reading the 1 KiB header, i.e. two storage reads per selected -shard (`offset=0,length=1024` for header, then bounded body range). This loses the -single-trip property but isolates whether `to EOF` streaming/connection handling is -the real under-load cost. If bounded shadow reads approach OFF, the production -single-trip design needs a bounded-length shadow stream or protocol support to keep -one trip without the current to-EOF behavior. - -Current cluster state after diagnostics: relaunched with regular ON-eager, -`BUCKIT_FASTGET_EAGER_SELECTED` unset. `mc admin info` reported `6 drives online, -0 drives offline, EC:2`. - -### 11.7 Diagnostic phase metrics quick validation - -Added low-cardinality diagnostic phase metrics exported through the existing API -metrics endpoint during isolation. - -Phases instrumented during that validation: - -| phase | meaning | -|---|---| -| `metadata` | OFF `getObjectFileInfo` / normal xl.meta path | -| `reader_setup` | `NewGetObjectReader` setup | -| `shadow_open` | ON shadow `ReadFileStream(..., current/part.1, 0, -1)` open | -| `shadow_header` | ON 1 KiB single-trip header read | -| `shadow_prefetch` | ON eager first-block body prefetch | -| `readat` | body shard `ReaderAt.ReadAt` calls used by decode | -| `decode` | `erasure.Decode` plus response write path | -| `decode_firstbyte` | time from decode entry to first write | - -Built `/tmp/st-hdd-bench/buckit-diag-metrics-v2`: - -- Linux amd64, build id `1faa13badc29e7ea7fdbd7b7cb26613ebba7a19b` -- sha256 `9d43761a4718f0e7ac7ccae3d948591a791458e46fca77373eafdb8f915a0c5b` - -Focused tests passed: - -```sh -GOCACHE=/tmp/st-hdd-bench/gocache GOTMPDIR=/tmp/st-hdd-bench/gotmp \ - CGO_ENABLED=0 go test -tags kqueue,dev -run 'SingleTrip|FastGet|ParallelReader' ./cmd/ -``` - -One-shot validation, c32, Warp duration 8 seconds, one reboot-cold arm each: - -| arm | obj/s | -|---|---:| -| OFF | 146.15 | -| ON-eager | 81.91 | - -Phase diff from before/after metrics snapshots, summed across both nodes: - -| arm | phase | count | avg ms | bytes | p50 | p90 | p99 | -|---|---|---:|---:|---:|---:|---:|---:| -| OFF | metadata | 1163 | 1.00 | 0 | <=1 ms | <=1 ms | <=20 ms | -| OFF | readat | 4650 | 109.24 | 758,842,346 | <=100 ms | <=500 ms | <=1 s | -| OFF | decode | 1164 | 219.16 | 0 | <=500 ms | <=500 ms | <=1 s | -| ON-eager | shadow_open | 2720 | 29.01 | 0 | <=0.5 ms | <=200 ms | <=500 ms | -| ON-eager | shadow_header | 2720 | 40.73 | 2,785,280 | <=20 ms | <=200 ms | <=500 ms | -| ON-eager | shadow_prefetch | 2040 | 71.76 | 334,298,880 | <=20 ms | <=500 ms | <=1 s | -| ON-eager | readat | 2742 | 16.84 | 446,235,629 | <=0.5 ms | <=100 ms | <=500 ms | -| ON-eager | decode | 687 | 68.25 | 0 | <=5 ms | <=200 ms | <=500 ms | - -Interpretation: - -- OFF spends almost all request time in the decode/body-read phase; metadata is not - a bottleneck. -- ON-eager's decode/body phase is faster than OFF once it reaches decode. -- The missing throughput is before decode: per GET, ON-eager performs four shadow - opens, four header reads, and three local eager prefetch reads. These phases have - HDD-tail latencies under c32 and extend request service time before response - streaming begins. -- This confirmed the metric idea was useful for quick validation, but the phase - metrics were removed from the production candidate to avoid permanent diagnostic - series. - -Artifacts: - -- Metrics run: `/tmp/st-hdd-bench/diag-metrics-c32-130132` -- Runner: `.tmp/fastget-diag-metrics-c32.sh` - -Current cluster state after this metrics validation: ON-eager diagnostic metrics -binary running, `BUCKIT_FASTGET_EAGER_SELECTED` unset. -`mc admin info` reported `6 drives online, 0 drives offline, EC:2`. - -PR cleanup note: these diagnostic phase metrics were removed from the production -candidate. Keep the regular fast-get hit/fallback counters. - -### 11.8 No-fallback validation - -Added diagnostic env `BUCKIT_FASTGET_NO_FALLBACK=1`. For fast-get-eligible -requests, if `tryFastGet` returns `ok=false`, the request returns an error instead -of falling back to the canonical path. This is diagnostic-only and was added to rule -out mixed ON/OFF work during the measured ON arm. - -Important operational note: object-based readiness probes fail in this mode if the -probe object lacks a single-trip shadow. Use `/minio/health/ready` for readiness -checks instead. - -Built `/tmp/st-hdd-bench/buckit-diag-nofallback`: - -- Linux amd64, build id `04293489805a8f00ad6bc60089ab831b4bf9175e` -- sha256 `c0f2d7e6d9859b18c348772b2c2caf0eda92b74570b3dd862309e806436e6877` - -Focused tests passed with no-fallback unset: - -```sh -GOCACHE=/tmp/st-hdd-bench/gocache GOTMPDIR=/tmp/st-hdd-bench/gotmp \ - CGO_ENABLED=0 go test -tags kqueue,dev -run 'SingleTrip|FastGet|ParallelReader' ./cmd/ -``` - -One-shot c32 ON-eager validation, Warp duration 8 seconds: - -| arm | obj/s | -|---|---:| -| ON-eager, no fallback | 84.11 | - -Metrics diff across both nodes: - -| metric | delta | -|---|---:| -| fast-get hits | 694 | -| fast-get fallbacks | 0 | - -Phase averages: - -| phase | count | avg ms | bytes | -|---|---:|---:|---:| -| decode | 701 | 63.53 | 0 | -| readat | 2798 | 15.62 | 455,410,687 | -| shadow_header | 2776 | 41.45 | 2,842,624 | -| shadow_open | 2776 | 30.48 | 0 | -| shadow_prefetch | 2082 | 71.44 | 341,181,504 | - -Conclusion: fallback is not causing the ON-eager plateau. With fallback disabled, -throughput remains in the same range and the phase profile is unchanged. Current -cluster state after validation: relaunched with `BUCKIT_FASTGET_NO_FALLBACK` unset, -regular ON-eager, diagnostic metrics binary still running, 6 drives online EC:2. - -### 11.9 Request-level metrics fix - -The first phase metrics missed the per-request fork/join barrier. Per-shard averages -for `shadow_open`, `shadow_header`, and `shadow_prefetch` cannot be summed to explain -request service time because those operations happen concurrently and the request -waits for the slowest required shard plus selection work. Added request-level phases: - -- `fast_open`: wraps the full `openSingleTripFastInfo` call, including all selected - shadow goroutines, `g.Wait()`, header grouping, optional eager prefetch, and M-shard - selection. -- `request_firstbyte`: from fast-path entry to first write from the decode goroutine. -- `request_total`: from fast-path entry to decode goroutine completion. - -Built `/tmp/st-hdd-bench/buckit-diag-metrics-v3`: - -- Linux amd64, build id `ab8c0bdcfcae992d5c0ec8f230d54421a96f0ffc` -- sha256 `f6747eb9bc139ab5a512f8fd4fc71c8966e1233764d1c34f07f4da3f4b457fc5` - -ON-only c32 validation, no-fallback enabled, Warp duration 8 seconds: - -| metric | value | -|---|---:| -| throughput | 81.53 obj/s | -| throughput-derived service time (`32 / obj/s`) | 392.49 ms | -| fast-get hits | 685 | -| fast-get fallbacks | 0 | - -Phase averages: - -| phase | count | avg ms | bytes | -|---|---:|---:|---:| -| `fast_open` | 685 | 340.03 | 0 | -| `request_firstbyte` | 692 | 378.95 | 0 | -| `request_total` | 692 | 378.95 | 0 | -| `decode` | 692 | 42.10 | 0 | -| `readat` | 2762 | 10.24 | 449,512,453 | -| `shadow_open` | 2740 | 30.42 | 0 | -| `shadow_header` | 2740 | 41.93 | 2,805,760 | -| `shadow_prefetch` | 2055 | 80.19 | 336,756,960 | - -This closes the accounting gap. `request_total` (~379 ms) aligns with -throughput-derived service time (~392 ms). The missing time was inside the full -`openSingleTripFastInfo` barrier, not an uninstrumented post-decode wait. The -per-shard shadow phase averages are still useful, but `fast_open` is the critical -request-level metric: ON-eager spends about 340 ms before entering decode. - -Current cluster state after this validation: relaunched with `BUCKIT_FASTGET_NO_FALLBACK` -unset, regular ON-eager, 6 drives online EC:2. - -## 12. Final load-test root cause and fix: M-spread selection - -Date: June 7, 2026. Branch/worktree: `perf/singletrip-early-release`. - -### 12.1 What was actually wrong - -The concurrent Warp throughput regression was not caused by fallback and was not an -unmeasured post-decode wait. Request-scoped profile logs showed that -`openSingleTripFastInfo` time was almost exactly `g.Wait()` time, and `g.Wait()` was -almost exactly the slowest selected shard's active time. - -The exact-quorum prototype had a load-balance bug: - -1. EC:2 has `M=4`, `N=2`. -2. Each node has 3 local drives. -3. The fast path opened exactly `M` shadows by selecting all local disks first, then - appending remote disks in fixed erasure-set order, then truncating at `M`. -4. On each landing node this became `3 local + first remote`. -5. Under c32 Warp load this pinned every remote quorum read from node1 to - `buckit-node2:/mnt/data01`, and every remote quorum read from node2 to - `buckit-node1:/mnt/data01`. - -Trace proof from `/tmp/st-hdd-bench/fastopen-trace-c32-141323`: - -```text -landing node1 remote selected: -893 times read=3 disk=1 http://buckit-node2:9000/mnt/data01 - -landing node2 remote selected: -497 times read=3 disk=0 http://buckit-node1:9000/mnt/data01 -``` - -For a representative bad request (`18B6DF3891AD18BA`): - -```text -fastopen total = 353.523 ms -g.Wait = 353.501 ms -slowest shard = remote node2/data01 = 353.473 ms -decode total = 110.002 ms -computed body production = 463.525 ms -``` - -The remote selected shard was not prefetched in the local-only eager arm because -`BUCKIT_FASTGET_EAGER_SELECTED` was unset. That meant the request paid for slow -remote stream acquisition during fast-open and then could pay again during decode -when the remote body was needed. - -### 12.2 OFF baseline caveat - -OFF is also not truly local-prefer on this hostnamed distributed setup. The canonical -decode path used: - -```go -prefer[index] = disk.Hostname() == "" -``` - -But profile logs showed local disks have non-empty hostnames: - -```text -host="/mnt/data02" hostname="buckit-node2:9000" local=true -``` - -Therefore OFF's `prefer` mask is effectively all false in this rig. A short OFF -profile run (`/tmp/st-hdd-bench/off-readat-locality-c32-144054`) captured 516 normal -GETs where decode read `2 local + 2 remote` shards, not `3 local + 1 remote`. - -This matters for comparisons: OFF spreads body reads better than the original -exact-quorum ON selector because OFF opens all valid readers and does not pin to the -first remote disk. OFF also benefits from `xl.meta` being warm in bounded working-set -load tests, so OFF is partly advantaged versus a true cold-metadata workload. - -### 12.3 Implemented fix / diagnostic mode - -Added `BUCKIT_FASTGET_SPREAD=1` as a diagnostic selector mode. With spread enabled, -the fast path still opens exactly `M` shadows, but it rotates across the full -erasure-set disk order instead of local-first truncating. On this 2-node EC:2 layout -that produces `2 local + 2 remote` per GET and spreads remote reads across all three -remote drives. - -Relevant code: - -- `cmd/singletrip-get.go`: `globalFastGetSpreadSelection` -- `cmd/singletrip-read.go`: `selectSingleTripFastOpenDisks` and - `selectSingleTripSpreadDisks` -- `cmd/singletrip-get_test.go`: - `TestSingleTripFastOpenRemoteSelectionRotates` and - `TestSingleTripFastOpenSpreadSelectionRotatesAcrossSet` - -The remote-rotation fix for local-first mode remains useful: - -- Default selector: all local disks first, then rotated remote candidates. -- Spread selector: rotated full-set selection, used only when - `BUCKIT_FASTGET_SPREAD=1`. - -### 12.4 Additional diagnostics added - -Request-scoped profile logs exposed the full fast-open critical path during -isolation. Those profiler logs were later removed from the production candidate. - -Aggregate diagnostic metrics also include: - -- `fast_open` -- `fast_open_wait` -- `fast_open_pick` -- `fast_open_build` -- `fast_open_close` -- `request_firstbyte` -- `request_total` - -`BUCKIT_FASTGET_NO_FALLBACK=1` remains a diagnostic guard: eligible fast-get misses -return an error instead of silently using OFF, so ON-only validation cannot be -polluted by fallback work. - -### 12.5 Validation results - -Focused tests: - -```sh -GOCACHE=/tmp/st-hdd-bench/gocache GOTMPDIR=/tmp/st-hdd-bench/gotmp \ - CGO_ENABLED=0 go test -tags kqueue,dev -run 'SingleTrip|FastGet|ParallelReader' ./cmd/ -``` - -Result: - -```text -ok github.com/buckit-io/buckit/cmd 6.759s -``` - -ON-spread profiled c32 run: - -```text -artifacts: /tmp/st-hdd-bench/fastopen-trace-c32-144904 -throughput: 80.46 MiB/s, 128.73 obj/s -``` - -Selection distribution: - -```text -1054 requests: all were 2 local + 2 remote -``` - -Remote selected-shard distribution: - -```text -node1 -> node2/data01: 348 -node1 -> node2/data02: 391 -node1 -> node2/data03: 339 - -node2 -> node1/data01: 346 -node2 -> node1/data02: 323 -node2 -> node1/data03: 361 -``` - -Side-by-side c32 throughput, profiling disabled: - -```text -artifacts: /tmp/st-hdd-bench/off-vs-onspread-c32-145331 -OFF 66.10 MiB/s, 105.76 obj/s -ON-spread 73.47 MiB/s, 117.55 obj/s -``` - -`ON-spread` was +11.1% obj/s over OFF in that run. - -Side-by-side c32 throughput with selected-eager enabled: - -```text -artifacts: /tmp/st-hdd-bench/off-vs-onspread-c32-150400 -OFF 67.38 MiB/s, 107.81 obj/s -ON-spread + selected-eager 73.58 MiB/s, 117.72 obj/s -``` - -The selected-eager variant was essentially identical to local-only eager in spread -mode. That suggests the large win came from shard-selection/load distribution, not -from moving remote body reads from decode into fast-open. - -### 12.6 Current recommendation - -For this EC:2 concurrent-load workload, keep the `M-spread` selector as the leading -candidate for further validation. It matches OFF's effective shard spread while still -skipping `xl.meta`, avoids the remote-disk hot spot, and measured faster than OFF in -quick side-by-side runs. - -Open follow-ups: - -1. Decide whether `BUCKIT_FASTGET_SPREAD=1` stays diagnostic or becomes the default - exact-quorum selector for distributed EC:2-style pools. -2. Fix OFF local preference separately by changing `disk.Hostname() == ""` to - `disk != nil && disk.IsLocal()` and then rerun OFF vs ON-spread. That is a - separate behavioral change and should not be mixed into this commit unless the - goal is to redefine the baseline. -3. Run longer multi-round alternating order tests after the quick validation phase. -4. Consider hedged `M+1` only after the spread selector is fully characterized. - -## 13. Critical cache discovery: stable selection, not lazy body, fixes hot repeats - -The earlier duration-based warp results were heavily shaped by repeated-object page -cache behavior. To isolate this, `testing/singletrip-hdd/once_get.py` now supports: - -```sh ---repeat=2 --repeat-mode=passes -``` - -This lists a fixed key set once, shuffles it deterministically, then issues the -entire key set once followed by the same key set a second time. This makes pass 1 a -cold-to-warm read and pass 2 a repeated-object read. - -Important result: OFF's hot-cache collapse was not mainly caused by its lazy bounded -body reader. It was caused by stable per-object shard selection. The original -single-trip spread selector rotated selected shards per request; repeated GETs for -the same object often read a different M-shard set, so pass 2 missed the warmed -shadow shard bodies. - -Validation workload: - -- Two-node EC:2 pool, 6 HDD drives, EC:2. -- 9,883 object keys from `/home/ubuntu/once-get-keys-seed17.txt`. -- `--repeat=2 --repeat-mode=passes`, concurrency 32. -- Reboot before each arm. - -Key artifacts: - -- OFF vs current Eager: `/tmp/st-hdd-bench/repeat2-passes-off-eager-213512` -- OFF vs lazy-body diagnostic: `/tmp/st-hdd-bench/repeat2-passes-off-lazy-214959` -- Lazy body + stable selection: `/tmp/st-hdd-bench/repeat2-passes-lazystable-220024` -- Current Eager + stable selection: `/tmp/st-hdd-bench/repeat2-passes-eagerstable-220630` - -Summary: - -| arm | throughput | pass 1 p50 TTFB | pass 2 p50 TTFB | -|---|---:|---:|---:| -| OFF | 120.15 obj/s | 441.0 ms | 11.9 ms | -| EAGER unstable | 147.23 obj/s | 247.3 ms | 118.0 ms | -| LAZYSTABLE | 175.90 obj/s | 295.6 ms | 11.9 ms | -| EAGERSTABLE | 178.62 obj/s | 271.5 ms | 11.4 ms | - -Conclusions: - -- `BUCKIT_FASTGET_LAZY_BODY=1` was useful as a diagnostic and is preserved in commit - `2752daca0`, but it is not needed for the hot-cache fix. -- Stable selection is the critical fix: the same object must select the same decode - shard set across requests if we want repeated-object cache reuse. -- Current Eager + stable selection is the best tested candidate so far. It preserves - Eager's lower cold-pass TTFB while matching OFF's hot-pass TTFB collapse. -- The final code drops the lazy-body diagnostic and keeps stable selection as the - actionable behavior to validate further. - -Implementation note: - -- Stable selection keeps the object-hash start position and has no per-request - rotation in either local-first remote selection or spread selection. -- The repeat-pass generator remains checked in because it is the simplest way to - catch cache-shard-selection regressions. - -Availability note: - -- The default fast path should not need to open more than M shards when the chosen - drives are known online. -- Before selecting the stable primary M, exclude drives that cluster/admin state - already knows are offline, unreachable, or otherwise unavailable. -- If one of the selected M header/body reads fails quickly despite being considered - online, retry with a deterministic backup shard. This is cheaper than paying the - M+1 hedge cost on every GET. -- Hedging should remain an optional tail-latency mode, not the steady-state default, - unless future tests show the tail win is worth the extra HDD pressure. - -## 14. M+1 hedge diagnostic - -A follow-up diagnostic tested stable primary `M` plus one deterministic hedge shard: - -- Env: `BUCKIT_FASTGET_EAGER=1`, `BUCKIT_FASTGET_SPREAD=1`, optional - `BUCKIT_FASTGET_HEDGE=1`. -- The hedge opens one extra stable candidate beyond the primary M. -- The fast-open path prefers the stable primary M if all are ready and agreeing. -- If one primary is slow or bad, any agreeing M among the M+1 can proceed. -- Unused/in-flight candidates are cancel-closed/drained asynchronously. - -Validation workload: - -- Same 9,883-key repeat-pass run as section 13. -- Reboot before each arm. -- Artifacts: `/tmp/st-hdd-bench/repeat2-passes-eagerstable-hedge-223008`. - -Summary: - -| arm | throughput | pass 1 p50 TTFB | pass 1 p95 TTFB | pass 1 p99 TTFB | pass 2 p50 TTFB | -|---|---:|---:|---:|---:|---:| -| EAGERSTABLE | 196.86 obj/s | 250.3 ms | 590.3 ms | 853.7 ms | 11.4 ms | -| EAGERSTABLEHEDGE | 181.83 obj/s | 288.4 ms | 589.0 ms | 792.2 ms | 12.1 ms | - -Interpretation: - -- The hedge preserved hot-cache reuse: pass 2 p50 TTFB stayed near 12 ms. -- It reduced throughput by about 8% versus EAGERSTABLE in this run and worsened pass - 1 p50. -- It was still much faster than OFF in the same repeat-pass methodology: - EAGERSTABLEHEDGE was 181.83 obj/s versus OFF's 120.15 obj/s from section 13 - (`+51%`), with pass 2 p50 TTFB still near OFF's hot-cache 12 ms behavior. -- It slightly improved pass 1 p99/max, suggesting it can rescue some slow-primary - tails, but the extra candidate open/read pressure is not free on HDDs. -- Current recommendation: do not make M+1 hedge the default from this one run. Keep - it diagnostic and only revisit if future workloads show tail latency is worth the - throughput/median tradeoff. - -## 15. Two MiB repeat-pass validation and multi-block Eager fix - -The 2 MiB run initially exposed a correctness bug before the benchmark could run: -local multi-block Eager opened the shadow body as: - -```text -ReadFileStream(current/part.1, offset=singleTripHeaderLen, length=-1) -``` - -For local `xlStorage.ReadFileStream`, `length < 0` returns the opened `*os.File` -directly and does not seek to `offset`. That means the local multi-block Eager body -stream started at byte 0, reread the 1 KiB single-trip header as if it were bitrot -body data, and returned 503 on 2 MiB FastGet reads. The 640 KiB tests did not hit -this because single-block Eager already used a bounded body read. - -Fix: local multi-block Eager now opens the body with an exact bounded bitrot length: - -```text -ReadFileStream(current/part.1, offset=singleTripHeaderLen, length=singleTripBitrotBodyLen(header)) -``` - -Targeted tests after the fix: - -```sh -GOCACHE=/tmp/st-hdd-bench/gocache GOTMPDIR=/tmp/st-hdd-bench/gotmp \ - CGO_ENABLED=0 go test -tags kqueue,dev -run 'SingleTrip|FastGet|ParallelReader' ./cmd/ -``` - -Result: - -```text -ok github.com/buckit-io/buckit/cmd 6.874s -``` - -2 MiB validation workload: - -- Seeded 1,002 objects under prefix `2mbench/` using `warp put --obj.size=2MiB`. -- Key file: `/home/ubuntu/once-get-keys-2mbench.txt`. -- `--repeat=2 --repeat-mode=passes`, concurrency 32. -- Reboot before each arm. -- Artifacts: `/tmp/st-hdd-bench/repeat2-passes-2m-off-eagerstable-hedge-231116`. - -Aggregate: - -| arm | throughput | p50 TTFB | p95 TTFB | -|---|---:|---:|---:| -| OFF | 110.31 obj/s, 220.62 MiB/s | 56.6 ms | 602.0 ms | -| EAGERSTABLE | 126.99 obj/s, 253.98 MiB/s | 49.5 ms | 565.7 ms | -| EAGERSTABLEHEDGE | 120.88 obj/s, 241.76 MiB/s | 76.0 ms | 442.7 ms | - -Important caveat: this aggregate is a 50/50 cold-pass + hot-pass mix by construction -and is not a production throughput estimate for a 50 TB+ per-host working set. Pass 2 -assumes every object was just read once and remains hot in the file-system page cache; -that creates the ~11 ms p50 TTFB hot pass and inflates the aggregate, especially for -OFF. For large AI/object-storage deployments where the working set is far larger than -RAM, the pass 1 / exactly-once view below is the more relevant comparison. - -Pass split: - -| arm | pass 1 p50 TTFB | pass 1 p95 TTFB | pass 2 p50 TTFB | pass 2 p95 TTFB | -|---|---:|---:|---:|---:| -| OFF | 307.2 ms | 743.0 ms | 11.0 ms | 20.1 ms | -| EAGERSTABLE | 253.9 ms | 783.3 ms | 11.3 ms | 19.5 ms | -| EAGERSTABLEHEDGE | 243.4 ms | 564.9 ms | 12.0 ms | 52.2 ms | - -Interpretation: - -- EAGERSTABLE is the best 2 MiB throughput result in this run: +15% obj/s over OFF. -- Stable selection again preserves the hot pass: pass 2 p50 TTFB is about 11-12 ms - for all arms. -- Do not use the aggregate mixed-pass throughput to estimate production behavior for - very large datasets; it overweights hot-cache reads. For such deployments, use - pass 1, exactly-once key traversal, or a working set larger than RAM/cache. -- The M+1 hedge improves cold-pass TTFB tail for 2 MiB (`p95 564.9 ms` vs - EAGERSTABLE `783.3 ms`), but loses throughput and has worse hot-pass p95 TTFB. -- Recommendation remains unchanged: default should be stable spread exactly M; - M+1 hedge is a tail-latency diagnostic/option, not the default. - -## 16. Recommended next steps - -### 16.1 Make the default candidate explicit - -Current best default candidate: - -```text -FAST_GET=1 -BUCKIT_FASTGET_EAGER=1 -BUCKIT_FASTGET_SPREAD=1 -BUCKIT_FASTGET_HEDGE=0 -``` - -Stable selection is now unconditional. This is `EAGERSTABLE`: stable spread exactly -M, no M+1 hedge. It is the best tested balance so far: - -- Preserves repeated-object cache reuse. -- Avoids local-HDD hot spotting. -- Avoids paying the extra M+1 open/read cost on every GET. -- Keeps the hedge path available as a diagnostic/tail-latency option. - -### 16.2 Add offline-drive replacement without steady-state hedging - -The default path should not open more than M shards when selected drives are known -online. Availability should be handled before and after selection: - -- Before selecting stable primary M, exclude drives that cluster/admin state already - knows are offline, unreachable, healing-unavailable, or otherwise unavailable. -- If one selected M header/body read fails quickly despite being considered online, - retry once with a deterministic backup shard. -- Keep backup choice stable per object so cache reuse remains predictable. -- Do not pay the M+1 hedge cost on every GET unless an operator explicitly enables - tail-latency mode. - -This should provide most of the availability benefit without adding HDD pressure to -the healthy steady state. - -### 16.3 Run production-relevant benchmarks - -Do not use repeat-pass aggregate throughput as the production estimate. It is useful -for cache diagnostics but overweights hot-cache reads. - -Next benchmark set should use exactly-once traversal / pass 1 style results: - -- 2 MiB exactly-once, concurrency 32 and/or 64. -- 8 MiB or 16 MiB exactly-once to check whether Eager benefit diminishes as body - transfer dominates. -- 64 MiB if the target includes AI-training-style packed shards. -- Dataset larger than RAM/page cache if feasible; otherwise avoid repeated keys and - report the limitation clearly. - -Key metrics to report: - -- obj/s and MiB/s. -- TTFB p50/p90/p95/p99. -- total request p50/p90/p95/p99. -- pass1/exactly-once throughput estimate, not mixed hot/cold aggregate. -- drive-level distribution and saturation if available. - -### 16.4 Decide large-object positioning - -Expected trend: Eager's relative throughput win shrinks as object size grows because -large-object GETs are dominated by body read, network transfer, erasure decode, and -client streaming. - -If 8-64 MiB results show small throughput gains but persistent TTFB gains, position -FastGet/Eager as: - -- strong for small/medium objects and metadata-heavy access, -- useful for TTFB-sensitive GETs, -- less central for AI-training large-shard sustained throughput. - -For AI-training workloads, also benchmark packed-shard patterns separately: - -- 64-512 MiB sequential GETs, -- range reads if loaders sample within large shard objects, -- aggregate sustained throughput over a working set larger than cache, -- input-pipeline stall behavior if integrated with a training loader. - -### 16.5 PR cleanup guidance - -Before a production PR, decide what remains in the final patch: - -- Keep stable selection. -- Keep the multi-block Eager bounded-body fix. -- Keep the repeat-pass generator if benchmark tooling is part of the branch. -- Keep `BUCKIT_FASTGET_HEDGE` only if diagnostic flags are acceptable; otherwise - split it into a separate experimental commit/branch. -- Remove or gate noisy low-level profiling/tracing if it is not suitable for - production builds. -- Document the default env combination and the exact benchmark methodology used to - justify it.