gsadmin/buckit
1
0
Fork 0
mirror of https://github.com/buckit-io/buckit.git synced 2026-09-24 19:31:59 +00:00
Code Issues Packages Projects Releases Wiki Activity

Align FastOpen design doc and trim docs

Browse Source
This commit is contained in:
abuckit
2026-06-13 21:36:51 -04:00
parent 94e4b8ab7b
commit 0fe7bd110a
11 changed files with 301 additions and 5830 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/single-trip-get-design.md
+301 -416
View File
@@ -1,502 +1,387 @@
# Single-Trip Direct GET - Design Note
# Single-Trip FastOpen GET - Design Note
**Status:** Draft / discussion notes
**Status:** Implemented in this branch
**Summary:** Buckit currently serves a normal shard-backed GET in two storage
phases: first read `xl.meta` across the erasure set to choose the visible version
and layout, then read shard bytes from the selected data files. This note proposes
adding **direct data paths** on top of the current metadata model: `xl.meta`
remains the canonical version index, while the physical shard directory for the
latest version is named `current` and older versions are named by version ID. The
shard files carry a small checked header before the existing bitrot-protected
shard bytes. A plain latest GET can open
`bucket/object/current/part.1`, read header then shard in one trip, and fall back
to today's `xl.meta` path whenever the fast path is missing, stale, inconsistent,
or not applicable.
**Summary:** Buckit's final FastOpen design does **not** add `current/` shadow
directories, `versions/<id>/` direct paths, or write-side shadow repair. Instead,
it adds a **read-side storage protocol** that lets each disk answer a GET with a
single streamed response:
This is not a replacement for `xl.meta`, and `current` is not an extra copy of
the latest shard data. There is one physical data location for a version:
`current` while it is latest, then `versions/<versionId>` after it is superseded.
The first target is plain latest GET; the same mechanism can also accelerate
explicit version GET.
1. read that disk's `xl.meta`;
2. encode the chosen object version into a compact FastOpen frame;
3. immediately continue with the shard body bytes from the same stream when the
object is streamable.
Read `request-flow.md` first; this note assumes its vocabulary (set, shard,
`xl.meta`, `DataDir`, quorum, `ErasureDist`).
The landing node opens FastOpen streams on a first wave of disks, reads only the
frame headers to establish the winning object/version/layout, then reuses the
already-open body streams for decode. If the first wave is not sufficient, it
opens additional disks or falls back to canonical GET. `xl.meta` remains
canonical; FastOpen is an opportunistic read optimization only.
This note describes the **shipped FastOpen design**. It supersedes earlier
proposals based on `current/part.1`, `versions/<versionId>/`, and write-side
shadow copies.
Read `request-flow.md` first; this note assumes its vocabulary (`xl.meta`,
erasure sets, parts, `FileInfo`, quorum, distribution, local vs remote disk
access).
---
## 1. Context and motivation
## 1. Goal
### 1.1 Why this matters: making HDD more viable
Canonical GET pays two logical storage phases:
The architecture Buckit inherited from MinIO is optimized for flash media. HDDs
are cheaper per terabyte and strong at sequential I/O, but weak at random metadata
IOPS. This matters for capacity-oriented deployments such as backups, archives,
media stores, and data lakes.
1. read `xl.meta` enough times to establish the visible version and layout;
2. open shard data files and decode from the selected readers.
The HDD issue is broader than GET alone: small-object workloads, listing,
versioning, healing, scanning, and degraded reads can all create random I/O. This
proposal targets one concrete hot-path contributor: latest GET currently pays a
metadata-file read phase before it can open shard files.
FastOpen keeps the same correctness rules but collapses those into one per-disk
storage request. Each participating disk returns metadata and, when possible,
body bytes from the same open stream.
### 1.2 Current GET path
For a healthy 16-disk set with `EC:4` (M=12 data, N=4 parity), a normal large
object GET generally does:
1. **Metadata fan-out:** read `xl.meta` from disks in the set to establish quorum
on the visible version and learn its layout.
2. **Shard read:** open the selected `DataDir/part.N` files and Reed-Solomon
decode from M available shard streams, pulling extra shards only on failures.
For large shard-backed objects, this is roughly one metadata-file seek plus one
data-file seek on the disks that participate in the read. On HDD, those seeks and
the second storage request phase dominate first-byte latency. On flash, the seek
cost is small, but the extra request phase still costs latency.
This statement has important exceptions:
- Small objects can already be returned from the metadata read path when inline
data is present or when `ReadData` inlines a small single-part shard.
- Zero-byte and transitioned objects take special paths.
- The decoder does not eagerly read all M+N shards; it starts with M readers per
stripe and falls back to more only when needed.
The target is therefore not "all GETs." The first target is the common large,
local, shard-backed latest GET, with explicit version GET as a natural extension
when the version ID maps directly to a stable data directory.
This is a **GET-only** optimization. It does not change the canonical write
format, object version layout, or healing authority model.
---
## 2. Core design: `xl.meta` canonical, direct data paths
## 2. High-level architecture
The proposed layout adds stable directly-openable data directories alongside the
canonical metadata file:
### 2.1 Canonical state stays unchanged
```
bucket/object/xl.meta # canonical version chain and metadata
bucket/object/current/part.1 # physical data for the latest version
bucket/object/versions/<versionId-1>/part.1 # physical data for older version
bucket/object/versions/<versionId-2>/part.1
```
FastOpen does not introduce a second on-disk object namespace.
For non-versioned or null-version objects, the direct version path needs an
encoded reserved name rather than the literal empty/null value.
- `xl.meta` remains the source of truth for visible versions, delete markers,
transition state, erasure layout, and object metadata.
- The ordinary shard files under the version's existing `DataDir` remain the only
physical shard bytes.
- PUT/DELETE do not create `current/` shadow copies or `versions/<id>/` direct
paths for FastOpen.
There is no duplicate `part.1` under both `current` and `versions/<versionId>` for
the same version. When a latest version is superseded, its data directory is
renamed or exchanged out of `current` into its deterministic version path.
FastOpen is therefore always allowed to fall back to the normal `xl.meta` path
without any repair or reconciliation step.
`xl.meta` remains the source of truth for:
### 2.2 New storage primitive
- version existence and ordering;
- delete markers;
- fallback for explicit `GET ?versionId=...`;
- listing and version listing;
- lifecycle, replication, healing, scanner, and repair decisions;
- compatibility with existing objects.
Each disk implements:
`current` is the direct path for plain GET without `versionId`.
`versions/<versionId>` is the direct path for explicit GET of non-current
versions. If a direct path is missing or disagrees with quorum, the request falls
back to the existing `xl.meta`-driven path. Repair can later reconstruct missing
or corrupt direct data from erasure quorum, with `xl.meta` deciding what should
exist.
- `StorageAPI.FastOpenPart`
### 2.1 Current shard file format
This operation:
Each direct-path `part.N` file starts with a small header, then the existing shard
payload:
1. reads the disk-local `xl.meta`;
2. resolves the requested object version (`latest` or an explicit `VersionID`);
3. converts the chosen object version into a compact `FastOpenGETMeta`;
4. returns a stream whose first bytes are a checked FastOpen frame;
5. appends either:
- shard bytes,
- inline bytes,
- no body for metadata-only outcomes,
- or a transitioned marker.
```
current/part.1 (one disk's current shard for one S3 part)
+--------------+-----------------------------------------------------------+
| HEADER | SHARD DATA |
| (metadata) | hash_0 | block_0 | hash_1 | block_1 | ... | hash_k | block_k |
+--------------+-----------------------------------------------------------+
^ length-prefixed, checksummed ^ existing per-block bitrot layout
```
The header carries enough information for the landing node to validate and decode
the fast path:
- object name, version ID, modtime, etag, object size;
- whether the current entry is an object or delete marker;
- direct path identity (`current` or `versions/<versionId>`) and canonical
version identity;
- `ErasureM`, `ErasureN`, `ErasureIndex`, `ErasureDist`, block size;
- part number, part size, actual shard file size, checksum algorithm;
- version/header signature used for cross-disk quorum comparison;
- format version and header checksum.
The shard-data region should remain byte-for-byte compatible with the existing
bitrot block layout after accounting for the header offset. Existing bitrot
verification and erasure decode should not need semantic changes, but the reader
must know that shard offsets are relative to the payload start, not file offset 0.
The header should be **variable-length with a self-describing payload** (a
length-prefixed, version-tagged frame followed by tagged/keyed fields, e.g.
TLV or msgpack), not a fixed-size positional record, so fields can be
added/removed across releases without lockstep: the reader takes the payload
length from the frame and the shard data begins immediately after, preserving
the single trip. Because `current` is a rebuildable cache of `xl.meta`, an
unrecognized format version or any decode failure simply falls back to the
canonical path and the shadow is lazily rewritten — so the schema can evolve with
no in-place migration. (The Phase 1 prototype uses a fixed 1024-byte header with a
positional payload for simplicity; that is an implementation shortcut, not the
intended on-disk contract.)
### 2.2 Delete-marker current
Latest can be a delete marker. A plain GET must return not found when the quorum
latest entry is a delete marker, even if older data versions exist.
Therefore `current` must also represent "latest is deleted." Options:
- a small `current/marker` file with the same checked header and no shard data;
- a reserved `current/part.1` header marked delete-marker with empty payload;
- no `current` data plus a separate marker file.
The fast path must never silently fall through to older shard data when latest is
a delete marker. If the delete-marker fast path is missing or inconsistent, fall
back to `xl.meta`.
This works both locally and over storage REST. The landing node therefore uses
the same abstraction for local and remote disks.
---
## 3. Fast GET protocol
## 3. FastOpen frame protocol
### 3.1 Plain latest GET
The protocol is defined in [cmd/fastopen-frame.go](/Users/rooseveltlai/develop/buckit-io/buckit/cmd/fastopen-frame.go:1).
For `GET /bucket/object` without `versionId`:
Each successful FastOpen stream begins with:
1. The landing node opens `current/part.N` on each disk in the set for the S3 part
needed by the request.
2. Each disk returns the checked header first.
3. The landing node compares headers and establishes quorum on the same current
version/signature.
4. If the quorum current is a delete marker, return not found.
5. If the quorum current is object data, stream from the selected M shard readers.
6. If any check fails, fall back to the existing `xl.meta` path and queue repair
of the direct data path.
- a fixed prelude:
- magic
- protocol version
- payload length
- CRC
- body mode
- a compact encoded payload:
- object/version identity
- part metadata
- erasure metadata
- transition metadata when relevant
The common healthy path removes the separate `xl.meta` read phase from latest
GET. The correctness rule is simple: `current` can accelerate the answer, but it
cannot override `xl.meta`.
Body modes are:
### 3.2 Header-pause vs speculative stream
- `FastOpenBodyShard`
- `FastOpenBodyInline`
- `FastOpenBodyMetadataOnly`
- `FastOpenBodyTransitioned`
For large objects, the safest protocol is header-pause:
Frame statuses are:
```
each disk -> stream HEADER -> pause
landing node:
collect headers and establish quorum
continue selected M shard streams
stop the rest
```
- `FastOpenStatusOK`
- `FastOpenStatusDeleteMarker`
- `FastOpenStatusNotFound`
- `FastOpenStatusVersionNotFound`
- `FastOpenStatusUnsupported`
For small objects or low-latency flash deployments, the implementation may choose
to stream header and data speculatively and cancel losers after quorum selection.
This trades some wasted bandwidth for lower control-plane complexity.
Important design point:
The existing decoder already starts with M readers per stripe and reads additional
shards only after missing/corrupt reads. The new protocol should preserve that
behavior rather than eagerly reading all M+N shards.
### 3.3 Explicit version GET
Explicit version GET can use the same direct-path mechanism, but the lookup must
account for the no-duplicate rule:
```
GET /bucket/object?versionId=<v>
-> try bucket/object/current/part.N and accept it only if the header says <v>
-> otherwise try bucket/object/versions/<v>/part.N
-> read header, establish quorum for <v>, stream selected M shards
```
This removes the `xl.meta` lookup for healthy explicit version GETs when the
requested version is either current or already under `versions/<v>`. If the direct
paths are missing, corrupt, stale, or not supported by an old object, fall back to
today's `xl.meta` path and queue repair if enough shard data exists.
The directory name must be a safe encoding of the S3 version ID. The header still
needs to carry and validate the canonical version ID; the path name alone is not
sufficient for correctness.
### 3.4 Multipart and range GET
In the current layout, `part.N` is the S3 multipart part number, not the erasure
shard number. Most single-part objects only have `part.1`; multipart objects may
have `part.2`, `part.3`, and so on.
Today's multipart read flow is:
1. read `xl.meta` and choose the visible version;
2. use that version's part list to map the requested object byte range to a start
part, offset within that part, and end part;
3. for each needed S3 part, open `bucket/object/<DataDir>/part.N` across disks;
4. Reed-Solomon decode that part's shard streams, then continue to the next
`part.N` until the requested range is complete.
Range GET complicates the fast path because the starting byte may map to a later
S3 part. To keep the fast path correct, every direct-path `part.N` that can be
opened directly must carry enough header information to validate the requested
version and that part's layout.
In the direct-path layout, a multipart current version would look like:
```
bucket/object/current/part.1
bucket/object/current/part.2
bucket/object/current/part.3
```
Each file needs its own header so a range GET that starts in `part.3` can validate
the version and part layout without first reading `part.1` or `xl.meta`.
Initial implementation can reasonably limit the fast path to:
- full-object or range reads that begin in `part.1`;
- single-part objects;
- non-transitioned shard-backed data.
Other requests fall back to `xl.meta` until the multipart/range protocol is
explicitly designed.
- FastOpen does **not** bypass `xl.meta` on the storage node.
- It bypasses a **second request phase** from the landing node by coalescing
metadata and body into one stream per disk.
---
## 4. Write and crash consistency
## 4. Request eligibility
### 4.1 Canonical commit
FastOpen is intentionally narrow. The request-level gate is implemented in
[cmd/fastopen-get.go](/Users/rooseveltlai/develop/buckit-io/buckit/cmd/fastopen-get.go:175).
PUT/DELETE correctness remains based on today's canonical metadata commit:
FastOpen is attempted only when:
1. write shard data to temporary files;
2. commit the new latest data to `current`;
3. if an older latest version existed, move that old `current` directory to
`versions/<oldVersionId>`;
4. update `xl.meta` as the canonical visible version chain.
- `BUCKIT_FAST_GET=1`
- `opts.FastGetObjInfo` is set by the caller
- the bucket is not the internal metadata bucket
- the request is not a `PartNumber` GET
- the request is not a range GET
- the request is not a replication or proxy request
- SSE-C is not requested
The ordering can be adjusted as long as recovery has a deterministic rule. The
important invariant is that `xl.meta` decides which versions are visible and which
direct path should contain each version. If a crash leaves a mismatch, fast GET
falls back to `xl.meta`; repair either reconstructs the missing direct path from
erasure quorum or removes uncommitted data.
At object/layout selection time, FastOpen may still reject the object and fall
back. Important fallback cases include:
If a crash leaves `current` newer than `xl.meta`, `xl.meta` wins. The fast path
must detect that by quorum/header validation or by falling back when the `current`
headers do not form a valid committed quorum.
- unsupported frame status from any selected disk
- unsupported checksum algorithm
- inability to assemble enough valid readers
- body mode/layout combinations not handled by the FastOpen reader path
- non-zero body offset after `NewGetObjectReader`
### 4.2 Updating `current`
A stable `current` directory is attractive because latest GET can open:
```
bucket/object/current/part.1
```
without resolving an opaque `DataDir` through `xl.meta`.
The important rule is to replace `current` by rename/exchange, not by overwriting
shard bytes in place. A reader that already opened `current/part.1` holds a file
descriptor to the old inode, so it can keep streaming even after the `current`
path is atomically swapped to a new directory. The swapped-out old directory then
becomes the immutable `versions/<oldVersionId>` directory; it is not a temporary
duplicate.
Updating that directory is not identical to today's `RenameData` flow. Today a
fresh immutable `DataDir` is committed under an opaque name and then made visible
through `xl.meta`. In the new layout, the latest version's physical data must land
at:
```
bucket/object/current
```
and the previous latest data must be preserved under:
```
bucket/object/versions/<oldVersionId>
```
There is no second copy. The filesystem operation is a rename/exchange of
directories, not a copy into `current`.
On Linux, `renameat2(RENAME_EXCHANGE)` can atomically swap two existing pathnames,
including directories on filesystems that support it:
```
bucket/object/current <-> bucket/object/.staging-new
```
After the exchange:
```
bucket/object/current # new latest data
bucket/object/.staging-new # old latest data, to rename to versions/<oldVersionId>
```
This can make latest replacement atomic at the per-disk namespace level, but it is
not portable POSIX behavior and still needs:
- startup capability detection for kernel and filesystem support;
- correct file and parent-directory fsync ordering;
- recovery of the old swapped-out directory into `versions/<oldVersionId>`;
- handling when latest is a delete marker;
- fallback for platforms/filesystems without `RENAME_EXCHANGE`.
The post-exchange rename of old data to `versions/<oldVersionId>` is another
crash point. Recovery must be able to recognize a swapped-out old-current staging
directory from its header and finish the rename or discard it if `xl.meta` says it
is not a committed version.
Delete markers, zero-byte objects, and transitioned objects are still handled via
FastOpen metadata when possible, but they return metadata-only object-level
results rather than streamed shard decode.
---
## 5. Durability risks and remediations
## 5. Landing-node read flow
The new direct paths do not weaken erasure coding, but they add crash states that
the current opaque-`DataDir` layout does not have. The safe rule is:
The read path lives in [cmd/fastopen-get.go](/Users/rooseveltlai/develop/buckit-io/buckit/cmd/fastopen-get.go:1).
```
xl.meta remains canonical.
Direct paths are served only when their headers form quorum for a committed version.
Staging/current/version mismatches are reconciled against xl.meta.
```
### 5.1 First wave
### 5.1 Crash-state risks
`tryFastOpenGET` calls `openFastOpenGETInfo`, which:
| Risk | Example state | Remediation |
|---|---|---|
| `current` ahead of `xl.meta` | `current` header says `v2`, but `xl.meta` still says latest is `v1` | Fast GET falls back to `xl.meta`; scanner/repair quarantines or removes uncommitted `v2` unless quorum metadata later confirms it. |
| `xl.meta` ahead of `current` | `xl.meta` says latest is `v2`, but `current` still contains `v1` | Fast GET detects header mismatch and falls back; repair moves/reconstructs `v2` into `current`. |
| old latest stranded in staging | after exchange, `.staging-new` contains `v1`, but `versions/v1` is missing | Recovery reads the staging header, verifies `v1` is committed in `xl.meta`, and renames it to `versions/v1`. |
| delete marker torn update | `xl.meta` latest is delete marker, but `current` still contains old data | Fast GET must not serve old data unless `current` headers form quorum for the committed latest; fallback returns 404 and repair installs the delete-marker current state. |
| partial per-disk commit | some disks have `current=v2`, some `current=v1`, some staging leftovers | Landing node groups headers by signature; serve only a committed quorum, otherwise fall back and queue per-disk repair. |
| missing fsync after rename/exchange | syscall returned success, but crash loses a directory entry | Follow strict fsync ordering for files and parent directories; recovery treats missing direct paths as repairable if `xl.meta` has quorum. |
| stale or orphan direct dirs | directory contains a version not present in `xl.meta` | Scanner quarantines or deletes after confirming it is not referenced by canonical metadata. |
1. chooses an initial wave of online disks;
2. opens `FastOpenPart` on each selected disk;
3. reads only the FastOpen frame from each stream;
4. leaves the stream positioned immediately after the frame.
### 5.2 Request-time repair trigger
Disk selection uses two modes:
The fast GET path can detect direct-path divergence cheaply from headers:
- default: local-first, then stable disk order
- spread mode: rotate the first wave by object hash when
`BUCKIT_FASTGET_SPREAD=1`
1. The landing node asks all disks for `current/part.N` or `versions/<id>/part.N`.
2. It groups returned headers by version/signature.
3. It reads from the quorum group if the group is valid for the request.
4. Disks whose headers are missing, corrupt, or outside the quorum group are
excluded from the read.
5. The landing node queues a repair for those disk/object paths.
The initial open count is based on the set's configured data/parity shape, not
the object's eventual winning layout. If the object's actual layout needs more
help than the first wave provides, FastOpen opens additional disks.
The landing node should not tell a disk "your version is wrong" based only on the
header mismatch. It should tell the disk to reconcile that object against
canonical `xl.meta`. The disk-local repair then decides whether its direct path is
ahead, behind, stranded in staging, or uncommitted.
### 5.2 Quorum and winning object selection
### 5.3 Disk-local reconciliation algorithm
FastOpen does not invent new correctness rules. After reading the per-disk
frames, it rebuilds `FileInfo` values and reuses the same quorum and version
selection rules as canonical GET:
Per-disk repair extends the existing healing/scanner model:
- `objectQuorumFromMeta`
- `reduceReadQuorumErrs`
- `listOnlineDisks`
- `pickValidFileInfo`
- `filterOnlineDisksInplace`
1. Read local `xl.meta`.
2. Build the expected direct-path map:
`current` -> latest committed object version or delete-marker marker, and
`versions/<id>` -> each committed non-current object version.
3. Inspect `current`, `versions/*`, and staging directories by reading their
checked headers.
4. Keep directories whose header matches the version expected at that path.
5. If a staging/current/version directory contains a committed version expected
elsewhere, rename it into the expected path.
6. If a directory contains an uncommitted version not present in `xl.meta`,
quarantine/delete it.
7. If committed shard data is missing or corrupt but recoverable from other disks,
invoke normal erasure healing to reconstruct it.
That means:
This is the same authority model Buckit uses today: metadata decides what should
exist; shard directories are repaired or cleaned up to match it.
- `NotFound` and `VersionNotFound` are still object-level outcomes only after
normal quorum rules are applied.
- stale but individually valid disk metadata does not win the read.
- `xl.meta` semantics remain canonical even though the landing node did not call
the normal metadata fan-out helper.
### 5.3 Replacement wave
If the first wave is not enough before response commit:
1. FastOpen records a replacement-path attempt.
2. It opens the remaining online disks.
3. It retries object selection with the larger set.
If selection still cannot succeed, the request falls back to canonical GET unless
`BUCKIT_FASTGET_NO_FALLBACK=1`, in which case the request returns an error.
---
## 6. Expected performance
## 6. Reader assembly and decode
For a healthy 16-disk EC:4 set serving large shard-backed latest GETs, the current
path is roughly:
### 6.1 Reusing the opened body streams
```
16 xl.meta reads + 12 shard reads
```
Once the winning `FileInfo` is known, FastOpen maps the already-open per-disk
streams into erasure-slot readers.
The fast path is roughly:
For directly usable body streams:
```
16 current header reads, then M shard streams from the same opened files
```
- shard mode becomes a `fastOpenStreamingBitrotReader`
- inline mode is also accepted when it can be validated safely
On HDD, the optimistic seek-bound ceiling is about:
Readers are placed by `Erasure.Index`, not by response order, so they match the
same slot semantics canonical decode expects.
```
28 / 16 ~= 1.75x
```
### 6.2 Lazy replacement readers
In practice, expected gains are lower:
If the initially selected body streams do not cover enough slots, FastOpen can
fill missing positions with lazy replacement readers.
- seek-bound small/medium shard-backed GET throughput: up to about 1.5x-1.75x;
- first-byte latency on HDD: roughly 30%-50% lower in the healthy fast path;
- large sequential transfers: usually 0%-10% total transfer improvement, mostly
faster first byte;
- flash/NVMe: mostly one fewer storage request phase, often a smaller latency win.
The lazy replacement path:
These are estimates, not measured results. The win shrinks or disappears when the
request falls back to `xl.meta`, targets old objects that have not been migrated
to direct paths, targets transitioned objects, needs multipart/range behavior not
covered by the fast path, or runs from warm metadata cache where the `xl.meta`
seek is already cheap.
1. records which disks are already engaged;
2. chooses the one disk that can satisfy each missing erasure slot;
3. opens a new `FastOpenPart` stream only when that slot is actually read;
4. validates that the replacement frame still matches the winning object/layout;
5. resumes decode from the requested shard offset.
This allows FastOpen to recover some pre-commit selection failures and some
decode-time missing-reader situations without abandoning the request before body
streaming begins.
### 6.3 Streaming bitrot verification
The FastOpen body path uses `HighwayHash256S` only. Other algorithms currently
force fallback.
The streaming reader verifies the existing on-disk bitrot layout inline:
1. read block hash
2. read block bytes
3. recompute hash
4. compare
No new shard-body format is introduced. FastOpen only changes how the landing
node gets to the shard bytes.
### 6.4 Range behavior
The current FastOpen path is effectively full-object only:
- request-level range GETs are rejected up front
- if `NewGetObjectReader` computes a non-zero offset, FastOpen falls back
Multipart part selection is also not supported in the FastOpen path.
---
## 7. Fallback and repair rules
## 7. Fallback model
Fast GET must fall back to the current implementation when:
FastOpen is allowed to fail only **before** it commits to a streaming response as
the request's object-selection path.
- the direct path (`current` or `versions/<versionId>`) is missing on too many
disks;
- headers do not form quorum on version/signature;
- a header checksum fails;
- the current entry is ambiguous relative to delete-marker semantics;
- requested range maps to a `part.N` not supported by the fast path;
- bitrot verification fails and quorum cannot be reconstructed from available
fast-path shards;
- the object is transitioned, restored, or otherwise not local shard-backed data.
Pre-commit failures fall into two categories:
After fallback succeeds through `xl.meta`, the system should queue a repair to
finish any interrupted rename/exchange, reconstruct missing shards from erasure
quorum, or delete uncommitted direct data.
- `ok=false`: abandon FastOpen and fall back to canonical GET
- `ok=true, err!=nil`: FastOpen determined the object-level result, including
delete marker or quorum failure cases
Once FastOpen has started the body goroutine and returned a `GetObjectReader`,
mid-stream failures are reported as stream errors, not by restarting through the
canonical path. This matches the fact that HTTP response commit has already
happened.
This is the central simplification versus the abandoned `current/` design:
- no request-time repair
- no direct-path reconciliation
- no crash-state cleanup
- no write-side invalidation rules
Fallback is always safe because the canonical layout was never changed.
---
## 8. Open decisions
## 8. Observability
1. **Direct-path layout:** `current/part.N` for the latest version plus
`versions/<versionId>/part.N` for non-current versions, stable regular files,
or another no-duplicate path scheme.
2. **Current update protocol:** `renameat2(RENAME_EXCHANGE)` vs. staged
rename/recovery fallback for filesystems without atomic directory exchange.
3. **Delete-marker representation:** marker file, empty current shard header, or
separate current state file.
4. **Multipart/range scope:** whether v1 supports only single-part `part.1` reads
or duplicates headers into every direct-path `part.N`.
5. **Transport:** header-pause control protocol vs. speculative stream/cancel.
6. **Migration:** old objects simply miss direct paths and use `xl.meta`; scanner
or write/read repair can migrate them opportunistically.
FastOpen observability is exposed under
`/minio/metrics/v3/api/requests` in
[cmd/metrics-v3-api.go](/Users/rooseveltlai/develop/buckit-io/buckit/cmd/metrics-v3-api.go:1).
Counters include:
- `fast_get_hits_total`
- `fast_get_fallbacks_total`
- `fast_open_attempted_total`
- `fast_open_hits_total`
- `fast_open_unsupported_total`
- `fast_open_replacement_path_total`
- `fast_open_streams_opened_total`
- `fast_open_replacement_opens_total`
- `fast_open_selected_set_failures_total`
- `fast_open_stream_cancellations_total`
- `fast_open_final_errors_total`
- `fast_open_httptrace_connections_total`
- `fast_open_httptrace_reused_connections_total`
- `fast_open_httptrace_fresh_connections_total`
- `fast_open_httptrace_was_idle_connections_total`
Timing metrics include:
- `fast_open_try_seconds_total`
- `fast_open_try_seconds_count`
- `fast_open_open_info_seconds_total`
- `fast_open_open_info_seconds_count`
- `fast_open_body_decode_seconds_total`
- `fast_open_body_decode_seconds_count`
The old `BUCKIT_FASTOPEN_PROFILE` stderr logging path has been removed. Timing is
now metrics-based.
---
## 9. References
## 9. What the final implementation is not
- `docs/request-flow.md` - current GET/PUT/DELETE flows, `xl.meta` format,
quorum, and local-vs-distributed transport split.
- `cmd/erasure-object.go` - `GetObjectNInfo`, `getObjectFileInfo`, and the decode
loop.
- `cmd/erasure-decode.go` - M-reader decode behavior and degraded fallback.
- `cmd/xl-storage.go` - `ReadXL`, `ReadVersion`, inline-data behavior, and
`ReadFileStream`.
- `cmd/storage-interface.go` - storage API signatures for metadata and file
reads.
The current FastOpen implementation does **not** do any of the following:
- no `current/part.N` shadow copy
- no `versions/<versionId>/part.N` direct namespace
- no `renameat2(RENAME_EXCHANGE)`-based latest swap
- no write-side direct-path install/invalidate protocol
- no crash-state repair for direct data paths
- no scanner-driven direct-path reconciliation
- no arbitrary range or multipart FastOpen read path
- no support for all checksum algorithms
Those ideas belonged to earlier design exploration. They are not part of the
implemented FastOpen shipped in this branch.
---
## 10. Expected behavior and tradeoffs
FastOpen improves the healthy full-object GET path by removing one landing-node
request phase to each participating disk. It does **not** remove the disk-local
`xl.meta` lookup itself.
Benefits:
- lower first-byte latency when the request stays on the FastOpen path
- fewer landing-node storage RPC phases
- opportunistic reuse of already-open body streams
- no write-path format migration
Costs and limitations:
- narrow eligibility
- fallback is still common for unsupported objects/requests
- some recovery cases require opening additional disks
- mid-stream errors are not converted into a new canonical GET
- observability is required to know whether a benchmark actually hit the path
This is therefore best understood as a conservative read-path optimization layered
on top of Buckit's existing metadata and erasure semantics.
---
## 11. References
- [cmd/fastopen-frame.go](/Users/rooseveltlai/develop/buckit-io/buckit/cmd/fastopen-frame.go:1) - FastOpen frame protocol and compact metadata encoding
- [cmd/fastopen-part.go](/Users/rooseveltlai/develop/buckit-io/buckit/cmd/fastopen-part.go:1) - disk-side `FastOpenPart` implementation
- [cmd/fastopen-get.go](/Users/rooseveltlai/develop/buckit-io/buckit/cmd/fastopen-get.go:1) - landing-node FastOpen GET path
- [cmd/fastget-config.go](/Users/rooseveltlai/develop/buckit-io/buckit/cmd/fastget-config.go:1) - runtime flags
- [cmd/metrics-v3-api.go](/Users/rooseveltlai/develop/buckit-io/buckit/cmd/metrics-v3-api.go:1) - exported metrics
- [docs/single-trip-get-phase1-implementation.md](/Users/rooseveltlai/develop/buckit-io/buckit/docs/single-trip-get-phase1-implementation.md:1) - historical prototype/implementation notes
docs/single-trip-get-full-feature-gaps.md
-874
View File
@@ -1,874 +0,0 @@
# FastGet Full-Feature Design Gaps
**Status:** design backlog after Phase 1 prototype
**Inputs:**
- `docs/single-trip-get-design.md` - intended production direction.
- `docs/single-trip-get-phase1-implementation.md` - measurement prototype plan.
- Current `feature/single-trip-get-phase1` prototype - additive `current/part.1`
shadow, fixed header, EAGERSTABLE default, stable shard selection, parity
reconstruction, and prototype metrics.
This document lists the remaining design work needed to turn FastGet from a
measurement prototype into a full system feature. It is not an implementation
plan yet. Each section should produce a concrete design decision, invariants,
failure handling, and tests before code lands.
---
## 1. Current Prototype Boundary
The current branch proves the main latency mechanism: a latest GET can skip the
`xl.meta` fan-out and reconstruct `FileInfo` from a checked header next to shard
bytes.
Important prototype properties:
- `current/part.1` is an additive shadow copy, not the only physical data path.
- FastGet is gated by `BUCKIT_FAST_GET`.
- Default candidate is EAGERSTABLE: eager first-block prefetch is on, stable
spread selection is on, hedging is off.
- Only latest, non-versioned, single-part, non-inline, local, unencrypted,
uncompressed, non-object-lock objects are eligible.
- FastGet is true no-`xl.meta` only on the `SinglePool()` path. Multi-pool GET
still resolves the owner through metadata first.
- The fixed 1024-byte header carries only a bounded response-metadata subset.
- There is no request-time repair, scanner reconciliation, fsync protocol, or
production migration path.
Design implication: the prototype is good evidence for the GET-side mechanism,
but it is not production-safe and does not define the final on-disk contract.
---
## 2. Product Scope and Compatibility
### Gaps
- Decide which GET API surfaces FastGet must cover beyond Phase 1. FastGet is
GET-only for all planned phases; HEAD stays on the existing metadata-only
`GetObjectInfo` path and is out of FastGet scope.
- Decide whether FastGet is a supported persistent feature or still an
experimental server flag.
- Define compatibility behavior for old objects without `current` aliases or
embedded direct headers, old header versions, mixed-version clusters, and
downgrade.
- Define user-visible behavior: FastGet must not change response bytes, response
headers, errors, version semantics, lifecycle behavior, replication behavior, or
object-lock semantics.
### Design tasks
- Define the supported request matrix and fallback matrix.
- Specify config shape: environment-only, config subsystem, dynamic cluster-wide
toggle, or per-bucket policy.
- Define upgrade/downgrade rules and minimum mixed-cluster safety constraints.
- Decide whether FastGet can be enabled by default, and under what storage/media
conditions.
- Define production observability: hits, fallbacks, fallback reasons, repair
queue stats, stale-`current` repair stats, and latency counters without noisy
per-request profiling.
---
## 3. Direct-Path Layout
### Gaps
The production design wants no duplicate latest data:
```
bucket/object/current/part.N
bucket/object/<DataDir>/part.N
bucket/object/xl.meta
```
The prototype instead writes:
```
bucket/object/<DataDir>/part.1
bucket/object/current/part.1
bucket/object/xl.meta
```
That doubles write cost and storage, makes PUT numbers invalid, and leaves the
system without a design for old-version direct GETs.
### Discussed direction
Keep the existing Buckit `xl.meta` plus opaque `DataDir` layout as the durable
layout. Add `current` only as a latest-version GET acceleration alias:
```
bucket/object/xl.meta
bucket/object/<DataDir>/part.N
bucket/object/current/part.N
```
The important invariant is that `xl.meta` remains the source-of-truth metadata and
version index, and each version's shard data remains under the `DataDir`
recorded in `xl.meta`. `current` is only an acceleration path for latest-version
GET. Explicit old-version GET, such as `GET ?versionId=<id>`, should use
the coalesced `xl.meta` route. That is slightly slower than a
direct version path, but old-version GET is not the common hot path and does not
justify a new `versions/<versionId>` directory layout in the initial full
feature.
Two `current` materializations should be designed and benchmarked behind the
same FastGet read path:
1. **Reserved real DataDir:** `current/part.N` contains the latest version's
shard files directly. In this mode, `current` is a special `DataDir` name for
the latest version. This avoids symlink semantics but makes PUT/overwrite
harder because the previous latest `current` directory must be renamed to a
generated `DataDir` and recorded in `xl.meta` without exposing mixed crash
states.
2. **Object-level pointer:** `current` points to the latest generated `DataDir`,
for example as a directory symlink on POSIX filesystems. This makes PUT
simpler: write a new generated `DataDir`, update `xl.meta`, then atomically
replace the single `current` pointer.
Avoid per-part symlinks such as:
```
bucket/object/current/part.1 -> ../<DataDir>/part.1
```
A per-part pointer can expose mixed state when an object has multiple S3 parts,
header sidecars, checksums, or future layout files. If indirection is used, it
should be a single object-level pointer:
```
bucket/object/current -> <DataDir>
```
The symlink/pointer mode is acceptable only if it is scoped as a rebuildable
GET acceleration index:
- PUT, DELETE, scanner, heal, lifecycle, replication, GC, and version listing
must continue to use `xl.meta` and recorded `DataDir` paths.
- FastGet may open `current/part.N`, but it must validate the direct header and
fall back to the coalesced `xl.meta` FastOpen tier if the pointer is missing,
stale, invalid, or points at a header that does not represent the latest
committed version.
- Delete-marker latest state must remove or invalidate `current`, or otherwise
make FastGet fall back. It must never serve the previous data version as
latest after a committed delete marker.
- Pointer targets must be relative, object-local, and unable to escape the
object directory.
- The implementation should hide the choice behind a storage-layer publish
operation, not expose generic symlink creation to unrelated code.
This keeps the FastGet reader stable (`current/part.N`) while allowing a
controlled comparison between real-directory and object-level-pointer publish
protocols.
### Design tasks
- Decide final directory names for:
- latest object data;
- generated `DataDir` directories;
- null versions and non-versioned objects;
- delete-marker current state;
- staging directories;
- quarantine/orphan directories.
- Decide whether production `current` is a real directory, an object-level
pointer, or a runtime/feature-gated choice for benchmarking.
- If `current` is a pointer, define the storage API operation for atomic pointer
replacement and the fallback behavior on platforms where symlink creation is
unavailable or unreliable.
- Define whether generated `DataDir` names stay unchanged or whether `current`
as a reserved real `DataDir` requires additional reserved-name handling.
- Define how `current` aliases and embedded direct headers coexist with existing
opaque `DataDir` objects during migration.
- Define scanner behavior for `current` aliases, old `DataDir` paths, orphan
staging paths, and partial migrations.
---
## 4. Header Format and Metadata Fidelity
### Gaps
The prototype header is fixed-size and intentionally scoped down. A full feature
needs a durable schema and enough metadata to make FastGet byte/header-equivalent
to existing GET behavior.
Metadata still needing design:
- user metadata: `x-amz-meta-*`;
- object tags: `x-amz-tagging`, tag count, optional tag response;
- object-lock metadata: retention and legal hold, permission-filtered on GET;
- checksum metadata for `x-amz-checksum-mode: ENABLED`;
- server-side encryption metadata for SSE-S3 and SSE-KMS, including enough
sealed-key/KMS metadata for the landing node to run the existing decryption
path after erasure decode;
- compression metadata and decompression behavior, with range-specific
compressed-offset planning deferred until range FastGet is in scope;
- transition/restore metadata and remote-tier behavior;
- replication status and version purge status;
- multipart part metadata and per-part checksums;
- non-STANDARD storage class and lifecycle prediction headers.
### Discussed direction
Use a two-tier storage-node open protocol:
1. **Embedded direct header:** the storage node opens `current/part.1`, reads the
embedded direct header, validates it, and streams the shard body after the
header. This is the highest-performance FastGet path because it avoids
request-time `xl.meta` reads.
2. **Coalesced `xl.meta` open:** if the direct header is missing, invalid,
stale, or cannot represent the object metadata, the storage node reads
`xl.meta`, selects the version/data directory, and returns a
metadata frame followed by the shard stream in the same storage response.
The landing node should see both tiers through one framed `FastOpenPart`
response stream. The first frame identifies the tier and carries either direct
header data or coalesced `xl.meta` metadata; shard bytes follow in the same
stream:
```
FastOpenPart(bucket, object, version selector, part.N)
-> DirectHeaderFrame, then direct shard bytes
-> CoalescedMetadataFrame, then shard bytes
-> miss/error
```
This means the embedded header does not need to encode every object case in the
first full version. It can stay optimized for the high-value direct path, while
the coalesced `xl.meta` open preserves full metadata fidelity and keeps one
storage RPC for broader object coverage.
The embedded direct header should be a variable-length frame, not a fixed
reserved slot. A fixed 4 KiB or 16 KiB header area would permanently add padding
to every eligible object, which is especially wasteful for small objects. The
preferred shape is:
```
part.1 = fixed small prelude + variable metadata frame + shard body
```
The prelude must be just large enough to identify the format and locate the
frame, for example magic, format version, header length, checksum, and flags.
After decoding the prelude and frame, the reader computes:
```
bodyOffset = preludeLen + headerFrameLen
```
EAGERSTABLE still works with this model, but it cannot assume the prototype's
fixed 1024-byte body offset. Eager prefetch, bounded local body reads, and
streaming bitrot readers must use the decoded `bodyOffset`.
For multipart objects, the embedded direct header is anchored in `part.1` only.
Other S3 part files should remain ordinary shard files unless a future design
adds a direct part index. Requests that need to enter at `part.2` or later, such
as `GET ?partNumber=2` or ranges that start beyond `part.1`, should use the
coalesced `xl.meta` tier of `FastOpenPart`.
`xl.meta` remains the fallback source of truth, but reading it on the storage
node and appending the shard stream is different from the landing node doing a
separate metadata phase and then a separate shard phase. The coalesced path
optimizes transport shape; the embedded header path optimizes both transport and
disk access shape.
### Design tasks
- Replace the fixed positional 1024-byte prototype header with a small fixed
prelude plus variable-length evolvable frame: length-prefixed, versioned,
checksummed, and self-describing.
- Define the decoded header body offset and update Eager/read protocols to use
that offset instead of a constant header length.
- Decide whether the header contains full metadata or a scoped subset plus
fallback flags.
- Define the `CoalescedMetadataFrame` returned by storage-node coalesced open
and prove it can reconstruct existing GET response metadata and `ObjectInfo`.
For Phase 1, this frame must include SSE-S3/SSE-KMS metadata instead of
treating encrypted objects as ineligible, because discovering that encryption
state already required the storage node to read `xl.meta`; doing another
landing-node `xl.meta` read would defeat the coalesced-open goal.
It must also include compression metadata, actual uncompressed size, and any
single-part compression index data needed for the landing node to reconstruct
`ObjectInfo` and run the existing decompression wrapper after erasure decode.
- Define how the landing node merges direct-header results and coalesced
`xl.meta` results when different disks return different tiers for the same
request.
- Define a stable signature over all fields that affect decode, version
identity, response headers, and fallback eligibility.
- Specify exact behavior for over-cap metadata.
- Define how metadata-only mutations update or invalidate `current` and embedded
direct headers:
`PutObjectTags`, `DeleteObjectTags`, `PutObjectMetadata`, legal hold,
retention, replication metadata, restore metadata, and lifecycle metadata.
- Define tests proving FastGet response headers match existing GET behavior for
all supported metadata.
---
## 5. Write, Delete, and Crash Protocol
### Gaps
The prototype uses invalidate-before-commit and post-commit shadow install. That
is safe enough for measurement, but production needs a durable `current` publish
protocol with crash recovery and no duplicate latest shard. The existing write
path should still commit `xl.meta` plus the version's `DataDir`; `current` is a
derived latest GET index.
Open design decisions:
- whether production `current` is a reserved real `DataDir`, an object-level
pointer to a generated `DataDir`, or a benchmark-gated choice.
- atomic publish protocol for `current`:
- real-directory mode may need a safe rename from `current` to a generated
old-version `DataDir` before publishing the new latest;
- pointer mode needs atomic pointer replacement after the new `DataDir` and
`xl.meta` are durable.
- fsync ordering for shard files, `DataDir` directories, `xl.meta`, `current`,
and staging paths.
- how to publish or invalidate `current` when the latest version is a delete
marker, without serving older data.
- whether failed `current` publish aborts writes or commits `xl.meta` metadata
and lets GET fall back to the coalesced `xl.meta` tier of `FastOpenPart`
until repair rebuilds `current`.
- how to handle platforms where symlink/pointer replacement is unavailable or
unreliable.
Current delete-marker representation:
- Delete markers are metadata-only `xl.meta` journal entries, not shard
directories.
- The xl.meta entry has `Type: DeleteType` and an `xlMetaV2DeleteMarker`
containing `VersionID`, `ModTime`, and `MetaSys`.
- When converted to `FileInfo`, a delete marker sets `Deleted: true`, carries the
delete marker `VersionID` and `ModTime`, and exposes `MetaSys` as metadata.
- `MetaSys` is primarily used for internal state such as replication,
version-purge, and tier/free-version metadata.
Because a delete marker has no `DataDir` and no `part.1`, the initial FastGet
rule should be to remove or invalidate `current` when the latest version is a
delete marker. Latest GET then uses the coalesced `xl.meta` tier of
`FastOpenPart`, reads `xl.meta` on the storage node, sees `Deleted: true`, and
returns the existing delete-marker response.
Metadata-only header replacement:
`PutObjectMetadata`, `PutObjectTags`, legal hold, retention, replication
metadata, restore metadata, and lifecycle metadata can change fields that the
embedded direct header would otherwise cache. For latest-version metadata
mutations, treat the operation like a `PutObject` commit that reuses the same
version, `DataDir`, and shard body. Do not rely on loose async repair that only
recreates `current`; that would republish a `part.1` whose embedded header still
contains old metadata.
1. Lock the object.
2. Read `xl.meta` and pick the target `FileInfo`.
3. Apply the metadata mutation to produce the updated `FileInfo`.
4. Temporarily invalidate `current` so FastGet cannot serve either the old header
or a mixed header/metadata state during the commit.
5. If the updated latest version is no longer eligible for embedded-header
FastGet, commit updated `xl.meta` and leave `current` invalidated.
6. If still eligible, rewrite the embedded `part.1` header synchronously using a
temporary file and the same shard body.
7. Commit updated `xl.meta` with write quorum.
8. Republish `current` only after both the embedded header and `xl.meta` reflect
the same updated `FileInfo`.
If the embedded header is stored in the source-of-truth `part.1`, do not patch it in
place. Use a staged rewrite:
```
<DataDir>/part.1
<DataDir>/part.1.fastget.tmp
```
The synchronous refresh decodes the old header to find `oldBodyOffset`, writes
the new variable header to the temporary file, stream-copies the existing shard
body from `oldBodyOffset`, fsyncs the temporary file, atomically renames it over
`part.1`, and fsyncs the parent directory.
If the header refresh fails, the safest initial behavior is to fail the
metadata-only operation before publishing `current`. A degraded mode that commits
`xl.meta` metadata and leaves `current` invalidated can be designed later, but it
must never republish `current` until the embedded header matches `xl.meta`.
This keeps correctness simple but makes metadata-only direct-header refresh a
file rewrite. A separate derived file would make refresh cheaper but would
reintroduce data duplication, so it is not the initial direction.
### Design tasks
- Specify the write-state machine for:
- first PUT of a key;
- overwrite in non-versioned bucket;
- overwrite in versioned bucket;
- overwrite from eligible to ineligible;
- multipart complete;
- copy object;
- delete marker creation;
- permanent delete by version ID.
- Specify crash states and deterministic recovery actions for every state.
- Define stale-read prevention invariants. `xl.meta` remains authoritative, but the
`current` alias must never serve an older committed version as latest.
- Define delete-marker `current` invalidation and repair behavior.
- Define `current`/header publish rules: eligible writes publish a fresh header
generated from the committed `FileInfo`; ineligible writes and metadata-only
mutations invalidate or rebuild `current`; scanner repairs stale `current`
aliases only after verifying that the existing embedded `part.1` header
matches `xl.meta`.
- Define synchronous metadata-only header refresh and its failure behavior.
- Define a FastGet repair queue or scanner hook for stale/missing `current`
discovered outside the locked metadata mutation path.
- Add design-level proof that FastGet's matching-header read quorum is safe
under offline-disk rejoin. The proof should rely on normal erasure quorum
intersection: once a newer version is committed to write quorum, stale
`current` aliases from disks that missed the write cannot form a disjoint read
quorum for the older version. Mixed old/new headers must not be combined, and
scanner/repair only shortens the stale-`current` window; it should not be
required for correctness when stale aliases are below read quorum.
---
## 6. Read Protocol
### Gaps
The prototype evolved from "open all to EOF once" into EAGERSTABLE: open exactly
the quorum, use stable spread selection, prefetch local first blocks, and use
bounded body reads for local multi-block eager. This is better for the measured
candidate. The production read protocol should keep the tight initial open set
and add deterministic before-first-byte replacement plus explicit cancellation.
Recommended read/open order:
1. Select only disks where `disk != nil && disk.IsOnline()`, matching the
existing GET metadata read path.
2. Open the maximum configured read quorum for the erasure set, with no initial
hedge. This should cover the largest possible data-block count across
configured storage classes and write modes. For example, in a 16-drive set,
if the lowest configured parity is 2, the initial open count is 14. If data
and parity can be equal, apply the existing strict-majority rule.
3. Give every `FastOpenPart` call its own cancellable child context.
4. Ask each selected disk for `FastOpenPart`.
5. For latest GET that can enter through `part.1`, the disk first tries the
embedded direct path: `current/part.1`.
6. If direct open fails before streaming, the disk may fall back locally to the
coalesced `xl.meta` tier: read `xl.meta`, select the version and
data directory, then return a metadata frame followed by the shard stream.
7. The landing node applies the existing Buckit metadata quorum, erasure
distribution ordering, and decode semantics. FastGet must not invent a new
metadata matching or shard scoring policy.
8. If the initial selected set fails before the first response byte, open all
remaining online disks as replacements, then apply the same existing Buckit
quorum/decode semantics over the combined responses.
9. After a winning read set is selected, close and cancel every unused stream
immediately. Do not drain unused remote streams.
10. If the client disconnects, or FastOpen exits before response bytes are sent,
cancel all outstanding child contexts so remote storage reads stop promptly.
11. After the first response byte is committed, do not switch to a semantic
fallback path. Surface errors through the existing read/decode behavior and
cancel all remaining streams on exit.
Explicit `versionId` GET and delete-marker responses should use the coalesced
`xl.meta` `FastOpenPart` tier starting in Phase 1. Requests that enter at
non-`part.1` multipart data should not require `current` or a direct embedded
header; they can stay on the current non-FastGet implementation until multipart
FastGet scope is added.
The embedded direct tier is the performance ceiling. The coalesced `xl.meta` tier
is the compatibility and fidelity tier. Both should use the same shard selection,
decode, cancellation, and before-first-byte fallback framework where possible.
### Design tasks
- Factor the existing Buckit metadata quorum and erasure decode selection so
`FastOpenPart` responses feed the same logic as current GET.
- Define the framed `FastOpenPart` response shape and body modes.
- Define tests for initial selected-set failure, replacement opens, unused-stream
cancellation, client disconnect, and post-first-byte stream errors.
---
## 7. Versioning
### Gaps
The prototype disables FastGet for versioned and version-suspended buckets. The
initial full-feature design should support versioned single-part GET through the
coalesced `xl.meta` route, including latest-version GET, explicit old-version
GET, and delete-marker responses.
Design constraints:
- Metadata is version-scoped in `xl.meta`.
- Tags and object-lock state can be updated for an old version.
- A plain GET without `versionId` must see only the latest version.
- A latest delete marker must return the same error and headers as existing GET
behavior.
- `GET ?versionId=<old>` must not read metadata from a newer version.
- Only mutations that change the latest visible version should publish, refresh,
or invalidate `current`.
- Mutations to non-latest versions should update `xl.meta` and leave
`current` untouched.
### Design tasks
- Define latest-version direct lookup through `current`.
- Define explicit `versionId` GET through the coalesced `xl.meta` tier of
`FastOpenPart`.
- Define old-version metadata mutation behavior: update `xl.meta` for that
`VersionID`, do not refresh `part.1`, and do not touch `current`.
- Define how a new version publishes or invalidates `current` without changing
the old-version `DataDir` contract.
- Define how permanent delete removes or quarantines recorded `DataDir`
directories. If permanent delete changes the latest visible version, the
simplest initial rule is to invalidate `current`; scanner/repair may rebuild
it later after verifying the exposed latest version and header.
- Define versioned bucket tests covering tags, user metadata, object lock,
delete markers, null versions, and permanent delete.
---
## 8. Multipart and Range
### Discussed direction
Embed the direct header only in `part.1`:
```
current/part.1 # direct header + shard body
current/part.2 # shard body only
current/part.3 # shard body only
```
This avoids duplicating object metadata into every S3 part and limits
metadata-only mutation work. Full-object multipart GET can use the `part.1`
header as the direct metadata anchor, then stream remaining parts from the same
latest `current` alias or recorded `DataDir`.
Requests whose first required shard is not `part.1` should use coalesced
`xl.meta` open unless a later design adds a direct multipart index. That includes
`GET ?partNumber=N` for `N > 1` and ranges that start in a later S3 part.
The initial Phase 1 implementation does not need to support multipart objects or
range GET. Multipart GET, `partNumber`, and ranges can stay on the existing
non-FastGet implementation until the single-part `FastOpenPart` protocol is
proven.
When multipart support is added, keep existing landing-node range planning. The
landing node already converts `Range` and `partNumber` into logical offsets,
maps those offsets to physical `part.N` spans, and decodes each physical part.
The multipart extension should replace each per-part remote shard open with one
`FastOpenPart` call for that physical part span. A range crossing multiple
multipart parts may issue multiple `FastOpenPart` calls. That is acceptable
because multipart parts are large, so the extra open cost is negligible compared
with transfer and decode time, and it keeps the remote storage node from needing
a cross-part object-range planner.
### Design tasks
- Define what the `part.1` direct header must carry for full-object multipart
GET without reading `xl.meta`.
- Define the `FastOpenPart` per-part call contract for logical ranges that map
to multiple physical `part.N` spans.
- Define which Phase 2 direct-header multipart/range requests must use
coalesced `xl.meta` open because they do not enter through `part.1`.
- Decide whether a future direct multipart index is needed for non-`part.1`
entrypoints.
- Define behavior for checksum-mode on full-object vs. range reads.
- Define tests for ranges crossing part boundaries and for `partNumber` GET.
---
## 9. Pool Routing and Cluster Topology
### Gaps
Current FastGet only avoids `xl.meta` in the single-pool path. Multi-pool
deployments still call pool-resolution code that reads metadata before the set
FastGet path can run.
### Discussed direction
Keep the current metadata-based pool selection for correctness, but reduce the
initial metadata fanout per pool with a quorum-intersection probe.
For each pool's hashed set, probe:
```
safeProbeCount = N - writeQuorum + 1
```
where `N` is the set drive count and `writeQuorum` is the minimum write quorum
for object layouts supported in that set. This count intersects any committed
version's write quorum. For example, in a 16-drive set with write quorum 12,
at most 4 drives can be stale/missing for a committed version, so probing 5
drives is enough to see that committed version if it exists.
Pool selection flow:
1. Probe `safeProbeCount` stable-spread drives in each pool's hashed set.
2. If exactly one pool reports a candidate and every other pool is proven absent
by the probe, select that pool.
3. If multiple pools report candidates, run full `xl.meta` pool arbitration.
4. If any pool returns ambiguous results, errors, or insufficient proof of
absence, expand that pool or run full `xl.meta` pool arbitration.
5. After selecting a pool, run `FastOpenPart` in that pool.
This reduces the common single-owner case without requiring cross-pool
`FastOpenPart` arbitration. It does not change GET availability: a 16-drive set
that has lost 6 drives can only serve objects whose actual read quorum can be
satisfied by the remaining 10 drives.
### Design tasks
- Define safe-probe pool selection and the expansion rules for ambiguity,
multiple candidates, delete markers, versioned requests, and pool errors.
- Decide how to handle objects that may exist in multiple pools during rebalance
or pool expansion.
- Define interaction with decommission, rebalance, site replication, and
data-movement code.
- Add multi-pool correctness and performance tests.
---
## 10. Storage API and Transport
### Gaps
The prototype reuses `ReadFileStream`, `CreateFile`, `RenameFile`, and `Delete`.
Production likely needs `FastOpenPart` for reads and must extend existing
commit-style storage mutations so they update variable headers and `current`
atomically inside the storage node.
### Design tasks
- Decide whether to add storage APIs for:
- `FastOpenPart` that can return either an embedded direct header plus shard
stream or a coalesced metadata frame plus shard stream;
- extensions to existing mutation APIs such as `RenameData`, `UpdateMetadata`,
and delete-version paths so header refresh and `current` publish/invalidate
happen atomically inside each storage node;
- `WritePrefixedShard` or equivalent local helper to write variable-header
`part.1` without cross-node shadow copy;
- direct-path header read plus paused body stream, if this is not folded into
`FastOpenPart`;
- `current` reconcile/repair;
- delete/invalidate `current` for latest delete markers, as part of the delete
storage mutation.
- Define REST/grid wire behavior and cancellation semantics.
- Define how local `xlStorage` and remote `storageRESTClient` implementations
differ, if at all.
- Define capability detection for symlink/pointer support, variable direct
headers, and mixed nodes.
---
## 11. Repair, Scanner, and Migration
### Gaps
The prototype has no scanner or repair path for `current`. In the full design,
objects without a valid `current` alias or embedded direct header can still use
the coalesced `xl.meta` tier of `FastOpenPart`, but stale `current` must not
serve old latest data.
Scanner/request-time repair should only reconcile the `current` alias. It should
not create or rewrite embedded `part.1` headers. Header creation and refresh are
owned by write-side mutation paths such as PUT, multipart complete, copy, and
latest-version metadata updates. If scanner finds a latest `DataDir` whose
`part.1` header is missing, stale, or invalid, it must leave `current` absent and
let GET use the coalesced `xl.meta` tier.
### Design tasks
- Define request-time repair triggers when FastGet falls back.
- Define disk-local reconciliation of `current` against `xl.meta` and the
latest version's `DataDir`.
- Define scanner migration for existing objects as optional installation or
repair of `current` only when the latest `DataDir` already has a valid
embedded `part.1` header matching `xl.meta`.
- Define scanner cleanup for stale `current`, staging directories, orphan
`DataDir`s, and uncommitted direct data.
- Define repair priority and throttling so FastGet repair does not starve normal
healing.
- Define metrics for direct-path health: present, missing, stale, repaired,
quarantined, and migrated.
---
## 12. Security and Correctness
### Gaps
The embedded direct tier reconstructs `ObjectInfo` without `xl.meta`; the
coalesced `xl.meta` tier reconstructs it from `xl.meta` on the storage node. The
full design must prove that neither tier can bypass checks performed by the
existing metadata path.
### Design tasks
- Audit every GET decision that depends on `ObjectInfo` or `FileInfo`.
- Ensure FastGet preserves:
- IAM and bucket policy behavior;
- SSE-C/SSE-S3/SSE-KMS behavior;
- object lock retention/legal hold filtering;
- lifecycle expiry and transition behavior;
- replication proxy behavior;
- conditional request behavior;
- checksum response behavior;
- delete-marker errors and headers.
- Define what must be in the direct header vs. what forces fallback.
- Define fuzz/corruption tests for malformed headers, signature collisions,
mismatched metadata, duplicated erasure indexes, and path/header disagreement.
---
## 13. Test and Validation Plan
### Required design outputs
- A correctness matrix covering request type, bucket versioning state, object
type, metadata type, and expected FastGet/fallback behavior.
- A crash-state matrix covering every write/delete stage and recovery result.
- A migration matrix covering old objects, mixed object layouts, and mixed
software versions. Mixed object layouts here means objects with and without
`current` aliases or variable direct headers, not a new dedicated version
directory layout.
- A performance matrix covering HDD, SSD/NVMe, local single-pool, distributed
single-pool, and multi-pool.
### Required test groups
- Header encode/decode compatibility and corruption tests.
- End-to-end GET byte and header equality tests.
- Versioned object tests for tags, metadata, delete markers, and old-version GET.
- Multipart/range tests for the later multipart FastGet extension.
- Metadata mutation tests.
- Crash/fault-injection tests around `DataDir` commit, `current` publish, pointer
replacement, invalidation, and fsync points.
- Offline-disk rejoin and scanner repair tests.
- Multi-pool routing tests.
- HDD load tests that compare FastGet with existing GET under cold and warm
cache conditions.
---
## 14. Suggested Design Sequence
### Phase 1: FastOpenPart with coalesced xl.meta frames
Goal: improve the transport shape without changing the on-disk object layout.
- Add `FastOpenPart` as a framed response stream:
`CoalescedMetadataFrame`, then body bytes when the selected object has a
FastOpen-streamable local body.
- On the storage node, read `xl.meta`, select the requested single-part object
version or latest visible version, then return the selected version metadata
before any body bytes. For local shard-backed objects, open the selected
version's `DataDir` shard and stream it after the frame. For metadata-only or
non-shard cases, the frame identifies the case and the body stream is omitted
or handled by the landing node through the existing path.
- Support all non-range, non-multipart GET cases through this tier in Phase 1,
including latest GET, explicit `versionId` GET, versioned/suspended buckets,
null versions, and delete-marker responses.
- Support inline objects in Phase 1. Since the storage node already reads
`xl.meta`, inline data can be returned through the coalesced frame/body path
without a local shard open.
- Support object-lock metadata in Phase 1. The frame must carry retention and
legal-hold metadata, and the landing node must run the existing permission
filtering before setting response headers.
- Support `x-amz-checksum-mode: ENABLED` for non-range Phase 1 GET. The frame
must carry checksum metadata so the landing node can run the existing checksum
response-header logic. Range checksum behavior remains deferred with range
FastGet.
- Support non-STANDARD storage classes in Phase 1 when the selected object is
still local or restored locally. The frame must carry the actual storage class
and erasure layout; the landing node must use the returned `ErasureM` instead
of assuming the default class.
- Support SSE-S3 and SSE-KMS for single-part GET in Phase 1. The remote storage
node streams encrypted shard bytes; the landing node reconstructs `ObjectInfo`
from `CoalescedMetadataFrame`, runs existing encryption validation/response
header logic, erasure-decodes, then decrypts with the existing landing-node
stream path.
- Support compressed single-part full-object GET in Phase 1. The remote storage
node streams stored compressed shard bytes; the landing node reconstructs
`ObjectInfo` from `CoalescedMetadataFrame`, erasure-decodes, then uses the
existing decompression path. Compressed range and compressed multipart GET stay
out of scope until range/multipart FastGet is added.
- Support restore/transition metadata in Phase 1. Restored-on-disk objects can
use the local shard/inline path while preserving `x-amz-restore` headers.
Remote-tier objects that are not restored locally should return a coalesced
metadata frame without a shard stream so the landing node can use the existing
transitioned-object reader; FastOpen must not try to open a missing local
`DataDir` shard for them.
- Keep SSE-C out of Phase 1. It is request-header/customer-key sensitive and can
be rejected before FastOpen from request headers.
- Use maximum-configured-read-quorum initial drive selection over online disks,
with no initial hedge. If the selected set fails before the first response
byte, open all remaining online disks as replacements and then apply existing
Buckit quorum/decode semantics.
- Give every `FastOpenPart` stream its own cancellable child context. Cancel
unused streams immediately after selecting the winning read set, and propagate
client disconnect/fallback cancellation to every outstanding remote read.
- Keep current multi-pool metadata selection, with the safe-probe optimization as
a separate pool-selection improvement.
Phase 1 should establish the framed stream protocol, cancellation semantics,
metadata fidelity against existing GET behavior, selected-drive fallback, and
parity reconstruction without introducing `current` or embedded direct headers.
### Phase 2: current symlink plus embedded part.1 header
Goal: add true no-`xl.meta` latest GET for eligible objects while preserving
the Phase 1 coalesced `xl.meta` tier as fallback.
- Keep `xl.meta` plus generated `DataDir` directories as the durable layout. Do
not add `versions/<versionId>` directories.
- Materialize `current` as an object-level pointer/symlink to the latest
generated `DataDir`.
- Store a variable direct header only in `part.1`:
small fixed prelude, variable metadata frame, then shard body.
- `FastOpenPart` tries `current/part.1` first for eligible latest GET.
If the direct header is missing, stale, invalid, or ineligible, the storage
node falls back to the Phase 1 coalesced `xl.meta` frame.
- Update write-side mutations to maintain the direct header and `current`:
- `PutObject`, `CompleteMultipartUpload`, and `CopyObject` write eligible
`part.1` headers from committed `FileInfo` and publish or invalidate
`current`.
- latest delete markers invalidate/remove `current`.
- permanent delete invalidates `current` if it changes the latest visible
version.
- latest metadata-only mutations synchronously refresh `part.1` using the same
`DataDir` and shard body, update `xl.meta`, and republish or invalidate
`current`.
- old-version metadata mutations update only `xl.meta` and do not touch
`current`.
- Scanner/request-time repair only reconciles the `current` alias when the
latest `DataDir` already has a valid embedded `part.1` header. Scanner must
not create or rewrite embedded headers.
- Prove quorum safety: matching direct headers form the read quorum, mixed
versions are not combined, and read/write quorum intersection prevents stale
`current` aliases below read quorum from serving an older committed version.
Phase 2 is the main production FastGet feature.
### Phase 3: optional real current directory
Goal: benchmark whether avoiding symlink resolution is worth the additional
write/crash complexity.
- Materialize `current` as a reserved real latest `DataDir` instead of a symlink
to a generated `DataDir`.
- Preserve the same `FastOpenPart` response stream and Phase 1 coalesced fallback.
- Preserve the same variable `part.1` header format from Phase 2.
- Design the overwrite protocol that safely turns the previous real `current`
directory into a generated old-version `DataDir` before publishing the new
latest.
- Compare real-directory and symlink modes for PUT cost, crash recovery,
scanner behavior, platform support, cold-cache GET latency, and warmed-cache GET
latency.
Phase 3 should remain optional unless benchmarks show that symlink resolution is
a material bottleneck or platform constraints require a non-symlink mode.
docs/single-trip-get-phase1-fastopen-plan.md
-770
View File
@@ -1,770 +0,0 @@
# FastGet Phase 1 FastOpen Implementation Plan
**Status:** implementation plan for full-feature Phase 1
**Companion docs:**
- `docs/single-trip-get-full-feature-gaps.md`
- `docs/single-trip-get-phase1-implementation.md` for prototype history only
---
## 1. Goal
Implement Phase 1 FastGet as a coalesced `xl.meta` FastOpen path:
```
FastOpenPart -> CoalescedMetadataFrame -> optional body stream
```
The goal is to remove the separate landing-node metadata fan-out before eligible
GET reads while preserving existing GET response bytes, headers, errors,
authorization behavior, version semantics, and erasure-read correctness.
Phase 1 must not introduce `current`, embedded `part.1` headers, symlinks, real
`current` directories, scanner repair, or write-path layout changes.
---
## 2. Scope
### Supported
- GET only. HEAD remains on existing `GetObjectInfo`.
- All non-range, non-multipart GET cases.
- Latest GET, explicit `versionId` GET, versioned/suspended buckets, null
versions, and delete-marker responses.
- Single-part local shard-backed objects.
- Inline objects, with inline bytes streamed after the frame.
- Zero-byte objects, with metadata frame and no required body stream.
- Restored-on-disk objects, preserving `x-amz-restore`.
- Remote-tier objects not restored locally, with metadata frame only and existing
transitioned-object reader on the landing node.
- Object-lock metadata, permission-filtered on the landing node.
- `x-amz-checksum-mode: ENABLED` for non-range GET.
- Non-STANDARD storage class when the object is local or restored locally.
- SSE-S3 and SSE-KMS, with encrypted shard bytes streamed and existing
landing-node decrypt path preserved.
- Compressed single-part full-object GET, with existing landing-node
decompression path preserved.
### Out Of Scope
- HEAD.
- SSE-C.
- Range GET.
- `partNumber`.
- Multipart object body FastOpen.
- Multi-pool safe-probe optimization.
- Mixed-version or rolling-upgrade FastOpen compatibility. Phase 1 assumes the
cluster is running a homogeneous FastOpen-capable binary.
- `current` path, embedded direct headers, write-path publish/invalidation, and
scanner repair.
Out-of-scope requests use the current non-FastGet implementation.
On multi-pool deployments, existing pool/owner resolution remains unchanged and
still happens before FastOpen. The true no-extra-`xl.meta` win for Phase 1 is the
single-pool path; multi-pool may still benefit after pool selection, but the
pool-selection metadata fan-out is not optimized in this phase.
---
## 3. Core Design Rules
- Reuse existing Buckit GET semantics. Do not invent a new metadata quorum,
version selection, erasure ordering, decode, auth, object-lock, checksum,
encryption, compression, or lifecycle behavior.
- The storage node may read `xl.meta` and open/stream local body bytes, but the
landing node remains responsible for request-context behavior such as auth,
preconditions, object-lock permission filtering, checksum headers, encryption
response headers, decryption, decompression, and final HTTP response shaping.
- The frame carries a compact GET-only metadata shape. It must be sufficient to
reconstruct the minimal `FileInfo`/`ObjectInfo` state used by existing GET, but
it must not carry `FileInfo` fields that are only used by write, list, repair,
or metadata-mutation paths.
- FastOpen failures before the first response byte are handled inside FastOpen by
opening all remaining online disks as replacements. Do not fall back to a
second landing-node metadata fan-out for supported Phase 1 requests.
- After the first response byte is committed, do not switch semantic paths.
Surface errors through existing read/decode behavior.
- Body bytes in shard and inline modes are per-disk encoded data. The landing
node must wrap selected body streams in the same bitrot-verifying decode path
before exposing any bytes to `NewGetObjectReader`.
---
## 4. Interfaces And Data Types
This section is the review boundary before coding. Names are proposed Go names;
field layout can be adjusted during implementation, but the semantics should be
settled here first.
### Storage Interface
Add a streaming storage operation alongside existing `ReadVersion` and
`ReadFileStream`:
```go
type StorageAPI interface {
FastOpenPart(ctx context.Context, volume, object string, req FastOpenPartRequest) (io.ReadCloser, error)
}
```
The returned stream starts with a `FastOpenFramePrelude`, followed by an encoded
`CoalescedMetadataFrame`, followed by body bytes only when `BodyMode` requires a
local body stream. Errors returned from `FastOpenPart` happen before any frame is
available. Per-object semantic results such as delete marker are represented in
the frame, not as transport errors.
### Request
```go
type FastOpenPartRequest struct {
Version uint16
VersionID string
PartNumber int
Offset int64
Length int64
Flags FastOpenPartFlags
}
```
Request semantics:
- `Version` is the request protocol version.
- `VersionID == ""` means latest visible version.
- Phase 1 uses `PartNumber == 1` for shard-backed body streams.
- Initial Phase 1 body reads use `Offset == 0`, `Length == -1`.
- For shard-backed replacement reads, `Offset` and `Length` are encoded
part-file byte coordinates for the body bytes after the frame, not object
byte coordinates. `Offset` may be nonzero to open a new forward-only body
stream at an encoded shard-body offset. `Length == -1` means to EOF; a bounded
length may be used as an optimization.
- Replacement offsets are only for erasure-decode replacement. They do not make
range GET or `partNumber` in scope.
- Range, `partNumber`, multipart, and SSE-C requests are rejected before
FastOpen by the landing node.
- Future phases may add direct/current-path flags, but Phase 1 is coalesced
`xl.meta` only.
### Prelude
```go
type FastOpenFramePrelude struct {
Magic [4]byte
Version uint16
HeaderLen uint32
HeaderCRC32 uint32
BodyMode FastOpenBodyMode
}
```
Prelude rules:
- The prelude is fixed-size and appears at byte 0 of every successful
`FastOpenPart` stream.
- `Magic` is `"BFG1"`; the `Version` field carries the protocol version.
- `HeaderLen` is the byte length of the encoded `CoalescedMetadataFrame`.
- `HeaderCRC32` covers only the encoded frame bytes, not body bytes.
- Unknown `Version` or invalid checksum is a FastOpen failure before client body
bytes are sent.
### Body Mode
```go
type FastOpenBodyMode uint8
const (
FastOpenBodyShard FastOpenBodyMode = iota + 1
FastOpenBodyInline
FastOpenBodyMetadataOnly
FastOpenBodyTransitioned
)
```
Body mode semantics:
- `FastOpenBodyShard`: frame followed by this disk's encoded shard bytes from
`DataDir/part.1`.
- `FastOpenBodyInline`: frame followed by this disk's encoded inline body bytes
from `xl.meta`; the landing node decodes it through the same path as shard
mode.
- `FastOpenBodyMetadataOnly`: frame only. Used for delete markers and zero-byte
objects.
- `FastOpenBodyTransitioned`: frame only. Landing node uses existing
transitioned-object reader.
### Coalesced Metadata Frame
```go
type CoalescedMetadataFrame struct {
Status FastOpenFrameStatus
Meta FastOpenGETMeta
BodyMode FastOpenBodyMode
BodyLen int64
}
type FastOpenGETMeta struct {
VersionID string
IsLatest bool
Legacy bool
ModTimeUnixNano int64
Size int64
Metadata map[string]string
Transition FastOpenTransitionMeta
Part FastOpenPartMeta
Erasure FastOpenErasureMeta
Checksum []byte
NumVersions int
SuccessorModTimeNanos int64
}
type FastOpenTransitionMeta struct {
Status string
Object string
Tier string
VersionID string
}
type FastOpenPartMeta struct {
Number int
Size int64
ActualSize int64
}
type FastOpenErasureMeta struct {
DataBlocks int
ParityBlocks int
BlockSize int64
Index int
Distribution []int
Bitrot FastOpenBitrotMeta
}
type FastOpenBitrotMeta struct {
PartNumber int
Algorithm uint8
Hash []byte
}
```
Frame rules:
- `Meta` describes one selected object version on one disk, after the storage
node has read `xl.meta` and applied the existing latest-version or explicit
`versionId` selection.
- `Meta` is not raw `FileInfo`. It carries only fields needed by GET selection,
response metadata, erasure decode, and bitrot verification.
- The landing node converts `Meta` into minimal `FileInfo` values before calling
existing metadata quorum, erasure ordering, and decode helpers. This keeps
quorum behavior aligned with current GET while avoiding full `FileInfo` wire
size.
- `Metadata` is the raw, unfiltered metadata map from the selected `FileInfo`.
The storage node must not call `cleanMetadata`, strip reserved keys, or apply
object-lock permission filtering. The landing node reconstructs
`ReplicationState` with `getInternalReplicationState(Metadata)`, then performs
the same cleaning, object-lock filtering, encryption, compression, restore,
storage-class, tag, ETag, and checksum handling as canonical GET.
- `Part` contains exactly the selected single part for Phase 1. Multipart body
paths, range GET, and `partNumber` are out of scope, so the frame does not
carry a full multipart part list.
- `Checksum` is the object checksum blob used by `x-amz-checksum-mode` and by
encrypted checksum handling.
- `NumVersions` and `SuccessorModTime` are carried because current GET response
header logic feeds `ObjectInfo.ToLifecycleOpts()` into lifecycle prediction
headers.
- `BodyLen` is the number of bytes following the frame for `shard` and
`inline` body modes. It is `0` for metadata-only and transitioned modes.
- `Status` distinguishes normal object, delete marker, not found/version not
found, and unsupported/capability cases when those should be represented as a
frame.
- `Status` is the authoritative delete-marker signal. During frame-to-`FileInfo`
conversion, `FileInfo.Deleted` is set only when
`Status == FastOpenStatusDeleteMarker`. Version-purge state is reconstructed
separately from raw metadata into `ReplicationState` and
`VersionPurgeStatus`.
- Wire fields use stable protocol encodings, not Go implementation details:
times are Unix nanoseconds, `FastOpenBitrotMeta.Algorithm` is a stable
FastOpen bitrot algorithm code mapped to/from Buckit's `BitrotAlgorithm`, and
enum values are protocol constants.
- Reconstruct `ModTimeUnixNano` and `SuccessorModTimeNanos` with
`time.Unix(0, n).UTC()` so quorum `time.Time.Equal` comparisons match
canonical metadata reads and carry no monotonic clock component.
- FastOpen bitrot algorithm codes are:
- `1`: `SHA256`
- `2`: `HighwayHash256`
- `3`: `HighwayHash256S`
- `4`: `BLAKE2b512`
Unknown algorithm codes are fail-safe: treat the frame as unsupported/corrupt
for quorum selection and never attempt decode with a guessed algorithm.
- `FastOpenBitrotMeta.Hash` is equivalent to existing `ChecksumInfo.Hash` for
the selected part. For streaming bitrot, per-block hashes remain in the body
stream and this hash may be empty, matching canonical verifier setup.
- The frame must not carry `Volume`, `Name`, `DataDir`, inline `Data`, full
multipart `Parts`, `Mode`, `WrittenByVersion`, `MarkDeleted`, `Fresh`, `Idx`,
or `Versioned`. The landing node already knows bucket/object/versioning
context, Phase 1 streams body bytes after the frame, and these fields are not
needed for GET.
- For the plain single-part objects used in the performance tests, the target
frame size is approximately the selected-version `xl.meta` size plus small
framing overhead. With observed `xl.meta` around 368-384 bytes, the compact
frame should remain in the same range rather than growing to a multi-KB
payload. This is not a hard bound; metadata size is data-dependent.
### Status
```go
type FastOpenFrameStatus uint8
const (
FastOpenStatusOK FastOpenFrameStatus = iota
FastOpenStatusDeleteMarker
FastOpenStatusNotFound
FastOpenStatusVersionNotFound
FastOpenStatusUnsupported
)
```
Status rules:
- Transport/storage failures before frame creation return `error`.
- Delete marker is a valid frame status so the landing node can return existing
delete-marker behavior with metadata.
- `FastOpenStatusNotFound` and `FastOpenStatusVersionNotFound` are disk-local
results. They do not immediately become a 404; the landing node applies the
same not-found/read-quorum logic as canonical GET, after any required
replacement opens.
- Unsupported means the request or selected object cannot be served by
FastOpen; the landing node may use current non-FastGet implementation only if
the request type is out of Phase 1 scope or capability is missing. Supported
Phase 1 requests should exhaust online-disk FastOpen replacement before
returning an error.
### Encoding Choice
Use a small fixed prelude plus msgp-encoded frame payload unless implementation
review finds a strong reason to use another existing Buckit codec. The frame
must be versioned and checksummed from the first implementation so future direct
headers can reuse the framing shape.
---
## 5. Storage Node Work
Add local `xlStorage` support for `FastOpenPart`:
1. Read `xl.meta`.
2. Select latest visible version or explicit `versionId` using existing
`xl.meta`/`FileInfo` helpers.
3. Build compact `FastOpenGETMeta` from the selected `FileInfo`, omitting fields
that are not used by GET. Preserve the raw metadata map exactly as canonical
`ToFileInfo` would expose it.
4. Decide body mode:
- delete marker: metadata-only;
- zero-byte: metadata-only;
- inline: inline body;
- transitioned and not restored: transitioned metadata-only;
- local shard-backed: open selected `DataDir/part.1` and stream shard bytes.
5. For shard-backed replacement requests with nonzero `Offset`, return the same
compact frame followed by this disk's encoded shard bytes starting at the
requested encoded part-file byte offset. The returned body is still a
forward-only stream. `BodyLen` is the number of body bytes following the
frame for this offset read: either the remaining encoded body length or the
requested bounded length.
6. Reject unsupported offsets before body bytes. Offset support is required only
for encoded shard-body replacement streams, not inline objects, metadata-only
statuses, range GET, multipart, or `partNumber`.
7. Return errors before any body bytes when metadata cannot be read or the
version cannot be selected.
Add remote storage transport support through the existing storage REST/grid
pattern. The final endpoint choice can follow the local code ownership and
streaming constraints, but it must support request-context cancellation.
Storage timeout behavior must follow the current GET path:
- `xl.meta` read and frame construction are bounded by the same drive timeout
used by `ReadXL`/`ReadVersion` (`globalDriveConfig.GetMaxTimeout()` through
the existing storage wrappers).
- Body stream open/read follows `ReadFileStream` plus existing bitrot/decode
behavior. Phase 1 must not add a new per-body-read timeout policy.
- FastOpen adds cancellation for streams it opened before final shard selection.
This cancellation is for abandoning unused/replaced streams, not for changing
canonical body-read timeout semantics.
---
## 6. Landing Node Work
Add a Phase 1 FastOpen path in `erasureObjects.GetObjectNInfo` before the
current `getObjectFileInfo` fan-out for eligible requests.
Eligibility before FastOpen:
- FastGet enabled;
- GET path only;
- not SSE-C request;
- no range;
- no `partNumber`;
- request is not replication/proxy special case.
FastOpen read flow:
1. Select only disks where `disk != nil && disk.IsOnline()`.
2. Open maximum configured read quorum over online disks, with no hedge.
3. Give every `FastOpenPart` call its own cancellable child context.
4. Convert returned `FastOpenGETMeta` frames into minimal `FileInfo`/metadata
arrays used by current Buckit GET selection.
- Rebuild `ReplicationState` from raw metadata with
`getInternalReplicationState`.
- Rebuild `Parts` as a one-entry slice from `FastOpenPartMeta`.
- Rebuild inline state, free-version state, encryption/compression state,
restore state, tags, ETag, and object-lock metadata from the raw metadata
map, matching canonical `FileInfo.ToObjectInfo`.
5. Apply existing Buckit metadata quorum, erasure distribution ordering, and
decode semantics.
6. If the initial selected set fails before first response byte, open all
remaining online disks as replacements and reapply the same selection logic.
7. Cancel every unused stream immediately after selecting the winning read set.
8. For shard body mode, wrap selected encoded shard streams in a streaming
bitrot-verifying `ReaderAt`, erasure-decode with the existing decode path,
and wrap with existing `NewGetObjectReader` behavior.
9. For inline body mode, use the same bitrot-verifying erasure-decode path as
shard body mode.
10. For metadata-only delete marker/zero-byte, return existing GET behavior.
11. For transitioned mode, call existing transitioned-object reader using the
reconstructed `ObjectInfo`.
No HTTP body bytes are committed until FastOpen has selected a metadata/body
quorum and the first decode attempt can start from offset 0. If the initial
selected set fails during frame processing or block-0 decode before any client
body byte is written, FastOpen opens all remaining online disks and restarts
selection/decode from offset 0. After the first client body byte is written,
FastOpen follows existing stream/decode error behavior, including same-offset
erasure replacement where possible.
### Mid-Stream Replacement
FastOpen must preserve canonical GET availability for post-commit shard
failures. A single disk stream that becomes missing, corrupt, or unreadable
after response bytes have started must not fail the customer GET if the erasure
set still has enough healthy shards to reconstruct the object.
Canonical GET achieves this because each bitrot reader is an `io.ReaderAt` that
opens its underlying `ReadFileStream` lazily. When the erasure decoder first
needs a spare shard at shard offset `N`, the reader opens that disk's part stream
directly at the encoded bitrot offset for `N`, then reads forward from there.
FastOpen should use the same model:
1. Initial selected readers are normal FastOpen streams opened at body offset 0.
2. Spare readers are lazy replacement readers. They do not need an opened body
stream before decode starts. A spare may reuse frame metadata if that disk was
already contacted as validation context, or it may fetch frame and body
together on first use. A replacement offset-read frame is always decoded and
validated; cached frame metadata never bypasses validation.
3. A lazy replacement reader holds only the information needed to open and
validate a later replacement read: disk, disk index, object identity, version
ID, selected object metadata identity, selected erasure layout, bitrot
algorithm, and body-length expectations.
4. On first `ReadAt(buf, shardOffset)`, a lazy replacement reader computes the
encoded body offset:
```
bodyOffset = (shardOffset / shardSize) * hashSize + shardOffset
```
It then calls `FastOpenPart` for the same object/version/part with
`Offset == bodyOffset` and a bounded or to-EOF `Length`, validates the
returned frame against the already selected object version and erasure
layout, and reads forward from the returned body stream.
5. Replacement validation is a hard gate. A frame is rejected before reading its
body if any required property differs from the winning object selection:
version identity (`VersionID`, or canonical null-version identity using
`ModTime`/ETag), the same winning-version filters used by initial FastOpen
selection (`IsValid`, selected `ModTime` or ETag, and `Erasure.Equal` over
`DataBlocks`, `ParityBlocks`, `BlockSize`, and `Distribution`), per-disk
`Erasure.Index`, slot availability at `Erasure.Index - 1`, part
number/size/actual size, body mode, body length expectations, and bitrot
algorithm. `Erasure.Equal` is not sufficient by itself because it does not
validate the per-disk shard index.
6. The replacement stream is still forward-only after open. It is not a random
access handle. The random-start behavior is provided by opening a new
`FastOpenPart` body stream at the requested encoded offset.
7. If the erasure decoder later asks the same replacement reader for the next
contiguous shard offset, the reader continues on the same stream. If it asks
for a backward or non-contiguous offset, the reader returns `errUnexpected`.
The reader initializes its internal current offset to the requested
`shardOffset`, not to zero.
8. If an active body stream fails, it is closed and retired for the rest of that
GET, matching canonical decode behavior. FastOpen does not reopen the same
failed disk as a later lazy replacement for the same request.
9. If the replacement frame is missing, stale, unsupported, corrupt, or belongs
to a different selected version/layout, that reader fails and the existing
erasure decoder may try another spare. If decode quorum cannot be reached,
the GET returns the same class of read error as canonical GET.
The extra frame validation is required because FastOpen lazy replacement re-runs
`FastOpenPart`, which reads `xl.meta` again and may observe a different selected
version or disk state. Canonical GET does not have this extra validation step
because it builds each spare's file path and layout from the already quorum'd
metadata before decode.
The block-0 prefix remains useful after lazy replacement exists. Its purpose is
pre-commit validation: FastOpen should prove that the selected metadata and body
streams can decode the first object block before `GetObjectReader` is returned
and client-visible bytes can be sent. Lazy replacement handles failures after
commit; block-0 prefix keeps first-block failure inside the pre-commit retry
window. Lazy replacement could also handle a block-0 spare open with
`bodyOffset == 0`, but the pre-commit path can afford to close streams, reopen
or revalidate the whole online set, and fail before committing any response
bytes. Post-commit recovery cannot restart the response and therefore uses
same-offset lazy replacement.
Lazy replacement happens below decryption and decompression, so SSE-S3/SSE-KMS
and compressed objects use the same shard-layer replacement behavior as plain
objects. Altered parity and non-STANDARD storage class are handled through the
selected object's actual erasure layout. Inline, metadata-only, and transitioned
objects do not use offset replacement: inline bodies fit in the pre-commit
decode window, and metadata-only/transitioned responses have no local shard body
stream to replace.
An engaged lazy replacement stream follows the same cancellation and cleanup
rules as any other `FastOpenPart` stream. It is closed/canceled on decode end,
client disconnect, or GET teardown. A lazy spare that is never engaged holds no
body stream and has nothing to close.
The initial open count is intentionally the maximum configured read quorum only,
with no hedge. If a disk is reachable and reports online but consistently times
out or returns stale/corrupt data, requests may take two waves: selected set
first, then all remaining online disks. This is accepted for Phase 1 correctness;
later tuning may widen the initial open set when the erasure set is known
degraded.
---
## 7. Cancellation Contract
- One cancellable child context per `FastOpenPart` stream.
- Client request cancellation cancels every child context.
- Unused streams are closed and canceled immediately after a winner is selected.
- Replacement losers are closed and canceled immediately.
- If FastOpen exits before client bytes are sent, all opened streams are closed
and canceled.
- Remote storage handlers must stop reading local files and stop writing the
response when the request context is canceled.
- Do not drain unused remote streams.
---
## 8. Error And Fallback Rules
- Unsupported request type: use current non-FastGet implementation.
- Missing FastOpen capability on any required storage node/set is a deployment
incompatibility. Phase 1 does not support rolling upgrade to this version. The
implementation may disable FastOpen and use the current implementation if this
is detected, but it does not need request-time mixed-version negotiation.
- Supported request, selected set fails or times out before first client byte:
open all remaining online disks as replacements.
- Supported request, all online disks still cannot satisfy existing Buckit
metadata/read quorum: return the same class of error as existing GET.
- After first client byte: no semantic fallback and no new timeout behavior;
return existing stream/decode error behavior.
---
## 9. Testing Plan
### Functional Equivalence
- Gating golden test: for each supported scenario below, run canonical GET and
FastOpen GET against the same object and byte-compare the resulting
`ObjectInfo`, response headers, status/error, and body.
- Latest single-part GET body/header equality.
- Explicit `versionId` GET.
- Versioned bucket latest GET.
- Version-suspended/null version GET.
- Delete marker latest and explicit delete marker version.
- Zero-byte object.
- Inline object.
- Restored-on-disk object.
- Remote-tier not restored.
- Object-lock metadata with and without retention/legal-hold permissions.
- Tags and user metadata.
- `x-amz-checksum-mode: ENABLED`.
- Non-STANDARD storage class / altered parity layout.
- SSE-S3 and SSE-KMS.
- Compressed single-part full-object GET.
- Replication-configured source object, including replication status and version
purge status headers.
- Lifecycle-eligible object where prediction headers are emitted.
- Legacy/XLV1 object if Phase 1 keeps XLV1 in scope.
### Failure And Availability
- Initial selected disk missing object/version.
- Initial selected disk stale `xl.meta`.
- Initial selected disk missing `DataDir/part.1`.
- Initial selected disk returns corrupt body and bitrot verification detects the
corruption before or during decode.
- Initial selected set cannot reach quorum, replacement opens all remaining
online disks.
- Initial selected stream fails during block-0 decode before response commit,
replacement restarts from offset 0 and succeeds when quorum is available.
- Post-commit selected stream fails at a later erasure block and a lazy
replacement reader opens a spare shard at the same encoded body offset; GET
succeeds when decode quorum is still available.
- Lazy replacement frame mismatch or unsupported status is rejected and another
spare can be tried.
- Replacement set reaches quorum and response succeeds.
- All online disks cannot reach quorum and error matches existing GET.
- Client disconnect cancels remote reads.
- Unused selected streams are canceled and not drained.
- Post-first-byte stream failure with insufficient replacement quorum follows
existing decode/read error behavior.
- Inline object follows the same decode behavior as shard-backed objects.
### Regression
- Range, `partNumber`, multipart, HEAD, and SSE-C continue through current
non-FastGet implementation.
- Multi-pool selection behavior is unchanged; FastOpen runs only after existing
pool/owner resolution.
- Mixed-version cluster with missing FastOpen support disables FastOpen or uses
the current implementation; rolling-upgrade compatibility is not provided.
- Existing FastGet prototype tests are either updated for the new Phase 1 path or
retired if they only validate obsolete `current` shadow behavior.
---
## 10. Milestone Task List
### Milestone 1: Frame And Conversion Foundation
- [ ] Add `FastOpenFramePrelude`, `CoalescedMetadataFrame`,
`FastOpenGETMeta`, body-mode/status enums, and stable bitrot algorithm code
mapping.
- [ ] Add frame encode/decode helpers with `"BFG1"` magic, protocol version,
header length, and frame CRC validation.
- [ ] Add `fileInfoToFastOpenGETMeta` and `fastOpenGETMetaToFileInfo`
conversion helpers.
- [ ] In conversion tests, verify raw metadata is preserved, `ReplicationState`
is rebuilt with `getInternalReplicationState`, `Deleted` comes only from
`FastOpenStatusDeleteMarker`, time values round-trip through Unix nanos, and
unknown bitrot algorithm codes fail safely.
- [ ] Add golden conversion tests comparing canonical `FileInfo.ToObjectInfo`
output with compact-frame reconstructed output for representative metadata
combinations.
### Milestone 2: Storage-Node xl.meta FastOpenPart
- [ ] Add disk-local `xlStorage.FastOpenPart` support that reads this disk's
`xl.meta`, selects latest or explicit `versionId`, and emits a compact
`CoalescedMetadataFrame`.
- [ ] Add metadata-only frame results for not found, version not found, delete
marker, zero-byte object, and transitioned object not restored locally.
- [ ] Add inline object support by returning the frame plus encoded inline body
bytes from `xl.meta`.
- [ ] Add single-part shard-backed support by returning the frame plus an opened
`DataDir/part.1` encoded shard stream.
- [ ] Preserve canonical storage timeout behavior: `xl.meta`/frame construction
uses the existing drive timeout, while body stream behavior follows
`ReadFileStream` plus existing bitrot/decode semantics.
- [ ] Reject or return unsupported for out-of-scope requests before body bytes:
HEAD, SSE-C, range, `partNumber`, and multipart body paths.
### Milestone 3: Remote Transport And Cancellation
- [ ] Add storage REST/grid transport for `FastOpenPart` with the same framed
response stream shape as disk-local storage.
- [ ] Ensure the remote handler invokes the storage-node `xl.meta`
`FastOpenPart` operation on the host where the target disk is mounted.
- [ ] Give every opened `FastOpenPart` stream a cancellable child context.
- [ ] Ensure closing unused or replacement-loser streams cancels remote work
without draining.
- [ ] Ensure client disconnect cancels all open FastOpen child contexts.
- [ ] Add transport tests for frame decode errors, context cancellation, and
remote handler cleanup.
### Milestone 4: Landing-Node Selection And Decode
- [ ] Add eligible GET-only FastOpen entrypoint in `erasureObjects.GetObjectNInfo`
before the current `getObjectFileInfo` fan-out.
- [ ] Select only `disk != nil && disk.IsOnline()` and initially open the maximum
configured read quorum over online disks, with no hedge.
- [ ] Convert frames into minimal `FileInfo` arrays and reuse existing
`objectQuorumFromMeta`, not-found quorum handling, `pickValidFileInfo`,
erasure distribution ordering, and decode helpers.
- [ ] Wrap selected shard/inline body streams in a streaming bitrot-verifying
`ReaderAt` before erasure decode.
- [ ] Do not commit HTTP body bytes until frame quorum and block-0 decode can
start from offset 0.
### Milestone 5: Replacement, Errors, And Fallback
- [ ] If the initial selected set fails, times out, or cannot form quorum before
the first client byte, open all remaining online disks and restart
selection/decode from offset 0.
- [ ] Add lazy replacement readers for spare disks so mid-stream shard failures
can open a new `FastOpenPart` body stream at the failed encoded shard-body
offset and continue decode when quorum is still available.
- [ ] Extend disk-local and remote `FastOpenPart` to support encoded shard-body
offset reads for replacement streams while keeping returned bodies
forward-only.
- [ ] Apply per-disk `NotFound` and `VersionNotFound` through canonical quorum
logic rather than returning immediate 404.
- [ ] After first client byte, preserve canonical stream/decode error behavior
and do not switch semantic paths; use erasure replacement before surfacing a
read/decode failure.
- [ ] Treat missing FastOpen capability as deployment incompatibility for this
version; rolling-upgrade compatibility is out of scope.
- [ ] Add failure tests for stale metadata, missing body, corrupt body with
bitrot detection, block-0 replacement, mid-stream replacement, replacement
failure, client disconnect, and unused-stream cancellation.
### Milestone 6: Golden Equivalence And Scope Gates
- [ ] Add the gating canonical-vs-FastOpen golden GET test across the full
supported matrix in Section 9.
- [ ] Verify exact equality for `ObjectInfo`, response headers, status/error,
and body bytes.
- [ ] Include replication-configured source objects, version purge status,
object lock permissions, tags, checksum mode, SSE-S3/SSE-KMS, compression,
restored-on-disk objects, remote-tier metadata-only objects, versioned/latest
and explicit-version reads, inline objects, zero-byte objects, and altered
storage class/parity.
- [ ] Verify out-of-scope requests continue through the current non-FastGet path:
HEAD, SSE-C, range, `partNumber`, multipart body paths, and multi-pool pool
selection.
- [ ] Add lazy-replacement hardening cases not required for the M5 commit:
non-block-aligned object with mid-stream replacement, multi-block continuation
after replacement engagement, post-commit replacement exhaustion returning the
canonical read-quorum error class, and stale/mismatched replacement candidate
rejection before body bytes are used.
### Milestone 7: Observability And Cleanup
- [ ] Add minimal metrics listed in Section 11.
- [ ] Remove or retire obsolete prototype-only FastGet tests that validate
`current` shadow behavior rather than Phase 1 coalesced FastOpen behavior.
- [ ] Keep noisy profiling/tracing logs out of the final path.
- [ ] Run focused `go test -tags kqueue,dev ./cmd` coverage for new FastOpen
tests, then broaden as needed for touched shared helpers.
---
## 11. Metrics
Keep observability minimal:
- FastOpen attempted.
- FastOpen hit.
- FastOpen unsupported request.
- FastOpen replacement path used.
- FastOpen streams opened per GET.
- FastOpen replacement opens per GET.
- FastOpen selected-set failure reason.
- FastOpen stream cancellation count.
- FastOpen final error category.
Do not reintroduce per-phase profiling logs or high-cardinality diagnostics.
docs/single-trip-get-phase1-handoff.md
-284
View File
@@ -1,284 +0,0 @@
# Single-Trip GET Phase 1 Handoff
## Scope
This note summarizes the work completed on the two-host HDD-style benchmark setup for the phase-1 single-trip GET prototype, plus the code change made to fix misleading fast-get fallback metrics.
## Hosts
- `buckit_node1`
- public IP: `3.81.144.6`
- private IP: `172.31.44.123`
- `buckit_node2`
- public IP: `35.173.238.193`
- private IP: `172.31.37.19`
SSH used:
```sh
ssh -i /Users/rooseveltlai/Downloads/buckit.pem ubuntu@3.81.144.6
ssh -i /Users/rooseveltlai/Downloads/buckit.pem ubuntu@35.173.238.193
```
## Host Setup Completed
Each host now has:
- three XFS-formatted instance-store mounts:
- `/mnt/data01`
- `/mnt/data02`
- `/mnt/data03`
- benchmark workspace:
- `/home/ubuntu/singletrip-bench`
- deployed Buckit binary:
- `/home/ubuntu/singletrip-bench/bin/buckit`
- helper scripts copied from repo:
- `/home/ubuntu/singletrip-bench/run/start-buckit.sh`
- `/home/ubuntu/singletrip-bench/run/cold-curl-ttfb.sh`
- `mc` installed on `buckit_node1`:
- `/home/ubuntu/singletrip-bench/bin/mc`
## Cluster Topology
Important detail: Buckit rejected endpoint hostnames with underscores.
This failed:
- `buckit_node1`
- `buckit_node2`
because Buckit reported:
- `FATAL Unable to split host port buckit_node1:9000: invalid hostname`
So the distributed single pool was run with DNS-valid aliases:
- `buckit-node1 -> 172.31.44.123`
- `buckit-node2 -> 172.31.37.19`
Those mappings were written to `/etc/hosts` on both nodes.
Distributed server command shape used on both nodes:
```sh
/home/ubuntu/singletrip-bench/bin/buckit \
--config-dir /home/ubuntu/singletrip-bench/config \
server \
--address :9000 \
--console-address :9001 \
http://buckit-node1/mnt/data01 \
http://buckit-node1/mnt/data02 \
http://buckit-node1/mnt/data03 \
http://buckit-node2/mnt/data01 \
http://buckit-node2/mnt/data02 \
http://buckit-node2/mnt/data03
```
Live backend info observed via `mc admin info`:
- one pool
- one set
- `6` drives per set
- standard parity `3`
- RRS parity `1`
So the live setup is effectively `EC:3` for standard objects, not `EC:2`.
## Datasets Created
Buckets loaded on the live cluster:
- `singletrip-cold-640k-6d`
- `singletrip-cold-1m-6d`
- `singletrip-cold-2m-6d`
Object counts:
- `10` objects per bucket
Sizes:
- `640 KiB`
- `1 MiB`
- `2 MiB`
Source data on node1:
- `/home/ubuntu/singletrip-bench/data/640k`
- `/home/ubuntu/singletrip-bench/data/1m`
- `/home/ubuntu/singletrip-bench/data/2m`
Presigned URL lists on node1:
- `/home/ubuntu/singletrip-bench/results/urls-640k.txt`
- `/home/ubuntu/singletrip-bench/results/urls-1m.txt`
- `/home/ubuntu/singletrip-bench/results/urls-2m.txt`
The `640 KiB` and `1 MiB` buckets were loaded while `FAST_GET=1` was enabled, and `current/part.1` shadow files were verified to exist.
## Functional Validation Completed
### Byte-for-byte correctness
Validated by downloading objects back on `buckit_node1` and using `cmp` against the original upload files.
Passed:
- `FAST_GET=1`: all 10 objects matched exactly
- `FAST_GET=0`: all 10 objects matched exactly
### Dependency split validation
Validated on `obj-01.bin` across all six drives:
- `FAST_GET=1`
- renamed away all `xl.meta`
- GET still returned `HTTP 200`
- bytes still matched original file
- `FAST_GET=0`
- renamed away all `current/`
- GET still returned `HTTP 200`
- bytes still matched original file
This confirms:
- ON path can serve without `xl.meta`
- OFF path can serve without `current/`
## Code Change Made
### Problem
The fast-get fallback counter was misleading.
Observed before fix:
- immediately after a fresh `FAST_GET=1` restart, before external GETs, `mc` metrics already showed:
- `fast_get_fallbacks_total = 9`
Root cause:
- internal `.minio.sys` reads used `GetObjectNInfo`
- those reads passed the coarse `fastGetRequestEligible(...)` gate
- they could not use the single-trip path
- so they incremented `fastGetFallbacks`
This made the metric unsuitable for interpreting user-facing GET fallback.
### Fix
Minimal fix implemented:
- reject `minioMetaBucket` in `fastGetRequestEligible(...)`
Files changed:
- [cmd/singletrip-get.go](/private/tmp/buckit-single-trip-get-phase1/cmd/singletrip-get.go)
- [cmd/erasure-object.go](/private/tmp/buckit-single-trip-get-phase1/cmd/erasure-object.go)
- [cmd/singletrip-header_test.go](/private/tmp/buckit-single-trip-get-phase1/cmd/singletrip-header_test.go)
Behavior after fix:
- fresh `FAST_GET=1` restart shows no startup fallback noise
- after one real GET:
- `fast_get_hits_total = 1`
- no fallback line emitted
After a fresh 100-request ON-only run:
- `GetObject total = 100`
- `fast_get_hits_total = 100`
- `fast_get_fallbacks_total = 0` effectively (no line emitted)
## Metrics / Benchmark Notes
### Reading metrics
Useful command:
```sh
/home/ubuntu/singletrip-bench/bin/mc admin prometheus metrics bench api --api-version v3
```
Relevant series after fix:
- `minio_api_requests_total{name="GetObject",...}`
- `minio_api_requests_fast_get_hits_total{...}`
- `minio_api_requests_fast_get_fallbacks_total{...}` (only appears when non-zero)
### Earlier benchmark notes
Several A/B runs were performed before the metric fix, including:
- 10-run ON/OFF passes for `640 KiB`, `1 MiB`, `2 MiB`
- reversed-order runs
- 100-run `2 MiB` ON/OFF percentile runs
Takeaway:
- results were not uniformly in favor of `FAST_GET=1`
- `2 MiB` in particular behaved inconsistently across different run orders
- after the metric fix, the ON-only 100-run counter check confirmed:
- `100/100` requests hit the fast path
- `0/100` requests fell back
### Latest percentile numbers captured
From the latest fresh patched-build 100-run `2 MiB` curl TTFB comparison:
ON:
- `p95 47.417 ms`
- `p90 29.137 ms`
- `p80 24.092 ms`
- `p70 22.538 ms`
- `p50 13.761 ms`
OFF:
- `p95 37.950 ms`
- `p90 28.294 ms`
- `p80 21.372 ms`
- `p70 12.067 ms`
- `p50 6.452 ms`
These are only for the latest fresh 100-run `2 MiB` comparison.
## Local Result Files
Not checked in; available in the local workspace under:
- `.tmp/bench2/`
- `.tmp/bench2_rerun_2m/`
- `.tmp/bench100_2m/`
- `.tmp/bench100_on_counts/`
- `.tmp/bench100_off_counts/`
These contain raw curl timings and, where captured, trace JSON.
## Current State
At the time of hand-off:
- the patched binary has been rebuilt and deployed to both nodes
- the cluster was last switched to `FAST_GET=1` and then later to `FAST_GET=0`/`FAST_GET=1` multiple times during experiments
- one `640 KiB` 100-run test was started but the user interrupted the turn
Because the last turn was interrupted, do not assume any in-flight benchmark loop completed cleanly.
Before resuming, check:
```sh
ssh -i /Users/rooseveltlai/Downloads/buckit.pem ubuntu@3.81.144.6 'ps -ef | grep -E "buckit|curl --http1.1" | grep -v grep'
ssh -i /Users/rooseveltlai/Downloads/buckit.pem ubuntu@35.173.238.193 'ps -ef | grep -E "buckit|curl --http1.1" | grep -v grep'
```
and re-establish whether the cluster is currently ON or OFF.
## Recommended Next Steps
1. Confirm current cluster mode (`FAST_GET=1` or `FAST_GET=0`) and kill any stray benchmark loops left by the interrupted turn.
2. Re-run the `640 KiB` 100-request comparison cleanly on the patched build.
3. If useful, add admin-trace capture to the fresh patched 100-run size comparisons, not just curl TTFB.
4. If the benchmark note should be preserved, convert key results from this hand-off into a checked-in benchmark README similar to `docs/single-trip-hdd-bench/README.md`.
docs/single-trip-get-phase1-implementation.md
-1121
View File
File diff suppressed because it is too large Load Diff
docs/single-trip-hdd-bench/README.md
-256
View File
@@ -1,256 +0,0 @@
# Single-Trip HDD Bench Notes
This document captures the ad hoc single-HDD test rig used on June 4-5, 2026 for the phase-1 single-trip GET prototype.
The goal of this run was not to prove the final HDD benefit. The host had only one rotational disk, so this was a control rig to:
- validate the host-side test procedure
- validate basic path dependency assumptions
- get an early latency signal before a real multi-node HDD run
## Host
- Host: `root@192.184.90.116`
- OS: Ubuntu 24.04.1
- Disk layout: one rotational root disk (`/dev/vda1`)
- Buckit paths used as pseudo-drives:
- `/root/singletrip-bench/data4/d01`
- `/root/singletrip-bench/data4/d02`
- `/root/singletrip-bench/data4/d03`
- `/root/singletrip-bench/data4/d04`
These are four directories on the same HDD, not four independent disks.
## Binaries And Scripts
Host-side paths:
- `buckit`: `/root/singletrip-bench/bin/buckit`
- `warp`: `/root/singletrip-bench/bin/warp`
- server launcher copied from repo: `testing/singletrip-hdd/start-buckit.sh`
Checked-in rig assets live in:
- [testing/singletrip-hdd/README.md](/Users/rooseveltlai/develop/buckit-io/buckit/testing/singletrip-hdd/README.md)
Local helper state used during the runs:
- `mc` config dir: `/private/tmp/buckit-hdd-mc`
- `640 KiB` presigned URL file: `/private/tmp/hdd-640k-obj-urls.jsonl`
- `2 MiB` presigned URL file: `/private/tmp/hdd-obj-urls.jsonl`
## Server Start
Use `MINIO_CI_CD=1` because the test data lives on the root disk.
`FAST_GET=1`:
```sh
ssh root@192.184.90.116 '
env MINIO_CI_CD=1 FAST_GET=1 \
MINIO_ROOT_USER=buckitadmin \
MINIO_ROOT_PASSWORD=buckitadmin \
/root/singletrip-bench/run/start-buckit.sh \
/root/singletrip-bench/data4/d01 \
/root/singletrip-bench/data4/d02 \
/root/singletrip-bench/data4/d03 \
/root/singletrip-bench/data4/d04
'
```
`FAST_GET=0`:
```sh
ssh root@192.184.90.116 '
env MINIO_CI_CD=1 FAST_GET=0 \
MINIO_ROOT_USER=buckitadmin \
MINIO_ROOT_PASSWORD=buckitadmin \
/root/singletrip-bench/run/start-buckit.sh \
/root/singletrip-bench/data4/d01 \
/root/singletrip-bench/data4/d02 \
/root/singletrip-bench/data4/d03 \
/root/singletrip-bench/data4/d04
'
```
Stop the server:
```sh
ssh root@192.184.90.116 'pkill -f "/root/singletrip-bench/bin/buckit"'
```
## Benchmark Method
The method that gave the cleanest signal was:
1. Use 10 different objects of the same size.
2. Run the GETs from the host itself, not from a remote client.
3. Before each GET:
- `sync`
- `echo 3 > /proc/sys/vm/drop_caches`
4. Measure:
- client-visible TTFB with `curl -w '%{time_starttransfer}'`
- server-side TTFB with `mc admin trace --verbose`
5. Compare either:
- last 5 of 10 runs, or
- lowest 5 of 10 runs when the run is noisy
Example host-local loop:
```sh
ssh root@192.184.90.116 '
set -e
for u in "$@"; do
sync
echo 3 > /proc/sys/vm/drop_caches
curl --http1.1 -sS -o /dev/null -w "%{time_starttransfer}\n" "$u"
done
' sh "<presigned-url-1>" "<presigned-url-2>" ...
```
Server-side trace:
```sh
mc --config-dir /private/tmp/buckit-hdd-mc admin trace --verbose hdd
```
## Datasets
### 2 MiB objects
- Bucket: `singletrip-cold-2m-4d`
- Object count: 20 total, 10 used in the clean comparison
- Object size: `2 MiB`
- Shadow verified: `current/part.1` present on all 4 pseudo-drives
### 640 KiB objects
- Bucket: `singletrip-cold-640k-4d`
- Object count: 10
- Object size: `640 KiB`
- Shadow verified: `current/part.1` present on all 4 pseudo-drives
- `current/part.1` size on disk: `328736` bytes
- `xl.meta` size on disk: `368` bytes
## Functional Validation
These checks passed on the host:
- `FAST_GET=1`: rename `xl.meta` away on all 4 object directories before GET, GET still returns `200`
- `FAST_GET=0`: rename `current/` away on all 4 object directories before GET, GET still returns `200`
This validates the basic dependency split:
- ON path can serve without `xl.meta`
- OFF path can serve without `current/`
## Results
### 2 MiB, 10 different objects, OFF first then ON
Using the last 5 of 10 runs:
- `FAST_GET=0`
- `curl` median: `17.032 ms`
- trace median: `15.301 ms`
- `FAST_GET=1`
- `curl` median: `12.073 ms`
- trace median: `11.698 ms`
Improvement:
- `curl`: `29.1%`
- trace: `23.5%`
### 2 MiB, 10 different objects, ON first then OFF
Using the last 5 of 10 runs:
- `FAST_GET=1`
- `curl` median: `30.449 ms`
- trace median: `30.214 ms`
- `FAST_GET=0`
- `curl` median: `43.962 ms`
- trace median: `43.241 ms`
Improvement:
- `curl`: `30.7%`
- trace: `30.1%`
This was the most convincing control result from the single-HDD host.
### 640 KiB, OFF first then ON
Using the last 5 of 10 runs:
- `FAST_GET=0`
- `curl` median: `20.905 ms`
- trace median: `19.100 ms`
- `FAST_GET=1`
- `curl` median: `42.286 ms`
- trace median: `39.010 ms`
This run inverted and was not trustworthy as a clean fast-path comparison.
Observed counters after the ON arm:
- `fast_get_hits_total=10`
- `fast_get_fallbacks_total=15`
That indicates fallback noise or unrelated contamination during the `640 KiB` run.
### 640 KiB, ON first then OFF
Full 10-run lists:
- `FAST_GET=1` `curl` ms:
- `157.507`, `112.020`, `50.408`, `70.818`, `267.515`, `39.806`, `53.513`, `35.694`, `109.684`, `140.253`
- `FAST_GET=1` trace ms:
- `153.795`, `112.089`, `48.038`, `70.438`, `265.378`, `37.215`, `51.709`, `34.376`, `108.788`, `139.557`
- `FAST_GET=0` `curl` ms:
- `291.295`, `69.004`, `70.546`, `54.955`, `70.527`, `73.920`, `215.505`, `55.148`, `65.311`, `33.009`
- `FAST_GET=0` trace ms:
- `288.341`, `68.346`, `69.096`, `54.154`, `69.758`, `72.806`, `167.278`, `42.117`, `64.080`, `30.531`
Using the lowest 5 runs in each group:
- `FAST_GET=1`
- `curl` mean: `50.048 ms`
- `curl` median: `50.408 ms`
- trace mean: `48.355 ms`
- trace median: `48.038 ms`
- `FAST_GET=0`
- `curl` mean: `55.485 ms`
- `curl` median: `55.148 ms`
- trace mean: `51.846 ms`
- trace median: `54.154 ms`
Difference from lowest-5 slices:
- `curl` median improvement: `8.6%`
- trace median improvement: `11.3%`
This is weaker and much noisier than the `2 MiB` result.
## Interpretation
Current conclusion from the single-HDD host:
- The host rig works and is reusable.
- The `2 MiB` different-object cold test produced a repeatable positive signal around `24-31%` TTFB improvement.
- The `640 KiB` case is noisy and not yet a stable signal.
- Because all four paths are on the same physical disk, none of these numbers should be presented as proof of the final HDD benefit.
## Recommended Reuse For Tomorrow
For the true multi-node HDD run:
- keep the host-local `curl` plus `mc admin trace` method
- keep the "10 different objects" pattern
- keep cache-drop before each GET
- prefer object sizes that cleanly produce `current/part.1`
- record both full 10-run lists and the summary slice used for comparison
- check `fast_get_hits_total` and `fast_get_fallbacks_total` after every ON arm
If the multi-node rig is clean, the `2 MiB` method from this document should be the baseline procedure.
docs/single-trip-hdd-bench/two-node-d3-2mib.md
-127
View File
@@ -1,127 +0,0 @@
# Two-Node D3 HDD Bench — 2 MiB Cold TTFB
This document records the June 5, 2026 single-trip GET phase-1 cold-TTFB A/B run on
a **two-node AWS `d3.xlarge` cluster with real local HDD storage**. It supersedes
the earlier, confounded 2 MiB numbers in `docs/single-trip-get-phase1-handoff.md`
(which repeated a small object set and was polluted by buckit's in-process metadata
cache — see "Method corrections" below).
## Headline
Pooled across **300 cold first-byte samples per arm** (3 rounds × 100 distinct
objects), enabling the single-trip fast path (`FAST_GET=1`) reduced **median cold
TTFB by ~14%** versus the canonical `xl.meta` path (`FAST_GET=0`), with a ~8–12%
reduction across p75–p95. The bottom decile is within noise. The signal is real but
**smaller than the design §6 HDD prediction (≥25%)** and the rig is very noisy, so
only the pooled aggregate is trustworthy.
## Rig
- Two AWS `d3.xlarge` nodes:
- node1 — public `3.81.144.6`, private `172.31.44.123`
- node2 — public `35.173.238.193`, private `172.31.37.19`
- Storage: three `/mnt/data0{1,2,3}` XFS mounts per node, **real dense HDD** local
instance storage (D3 family).
- **`rotational=0` is a Nitro artifact, not SSD.** D3/D3en expose spinning HDDs
through the NVMe interface, so the kernel reports the `/dev/nvme*` devices as
non-rotational even though the media is HDD. Confirm via IMDSv2 instance-type
(`d3.xlarge`), not the rotational flag.
- Topology: one pool, one set, 6 drives, **EC:3** (standard parity 3, RRS 1).
- Endpoints use DNS aliases `buckit-node1` / `buckit-node2` in `/etc/hosts`
(buckit rejects hostnames with underscores).
- SSH: `ssh -i /Users/rooseveltlai/Downloads/buckit.pem ubuntu@<public-ip>`.
## Dataset
- Bucket: `singletrip-cold-2m-big`
- 100 distinct objects, 2 MiB each, independent random content.
- Loaded once under `FAST_GET=1` so the `current/part.1` shadow exists on every
drive (verified: `current/part.1` = 700140 B per drive vs `xl.meta` ~382 B).
- Presigned URLs (12 h) generated for all 100 objects; `curl` issues them
host-local on node1 against `http://127.0.0.1:9000`.
## Method
The whole experiment is a runtime A/B on **identical on-disk data and one binary**;
the only variable is `BUCKIT_FAST_GET`.
1. **Distinct cold objects.** Each of the 100 objects is GET once per sweep — a
genuine first-touch. (Re-reading the same object is not an independent cold
sample, even with caches dropped.)
2. **Cold on both nodes.** Before every GET, page cache is dropped on **both**
nodes (`sudo sh -c "sync; echo 3 > /proc/sys/vm/drop_caches"`). This is a
distributed cluster, so dropping only node1 would leave shards warm on node2.
3. **Fresh restart per arm.** The server is restarted before each arm to clear
buckit's **in-process metadata cache** (which `drop_caches` does not clear).
4. **Measurement.** Client-side TTFB only, via
`curl --http1.1 -sS -o /dev/null -w '%{time_starttransfer}'`, executed on node1
against the loopback endpoint. **No `mc admin trace` server-side TTFB was
captured in this run** — these are curl numbers.
5. **Rounds.** 3 alternating rounds of [ON sweep, OFF sweep], 100 GETs each,
pooled to 300 samples per arm.
6. **Fast-path verification.** After each ON sweep the counters confirmed the path
actually fired: `minio_api_requests_fast_get_hits_total` == GET count and
`fast_get_fallbacks_total` == 0 (line absent when zero). OFF sweeps show 0 hits.
## Results
### Pooled cold TTFB, 2 MiB (n = 300 per arm)
| percentile | OFF (ms) | ON (ms) | ON vs OFF |
|---|---:|---:|---:|
| p10 | 18.03 | 19.59 | +8.7% |
| p25 | 24.48 | 23.74 | −3.0% |
| **p50** | **36.68** | **31.62** | **−13.8%** |
| p75 | 47.43 | 42.78 | −9.8% |
| p90 | 56.26 | 52.04 | −7.5% |
| p95 | 65.49 | 57.40 | −12.4% |
| mean | 37.19 | 34.15 | −8.2% |
### Per-round medians (illustrating rig noise)
| round | ON median (ms) | OFF median (ms) |
|---|---:|---:|
| 1 | 37.69 | 33.67 |
| 2 | 25.72 | 43.17 |
| 3 | 31.35 | 29.09 |
Round-to-round variance (ON 25.7–37.7; OFF 29.1–43.2) **exceeds the ON/OFF gap**,
so no single round is meaningful — only the pooled aggregate is. Round 1 even shows
ON slower than OFF. This matches the run-order inconsistency noted in the handoff.
## Method corrections (why earlier 2 MiB numbers were wrong)
An initial attempt repeated the **same 10 objects** across cycles and reported a
~31–54% "win." That was an artifact:
- **In-process metadata cache.** `drop_caches` clears only the Linux page cache;
buckit keeps its own metadata cache that survives it. Re-GETting the same object
was served partly from that cache, so cycles 2–3 were ~5–8 ms while cycle 1 was
~20–30 ms — a warmup curve, not a fast-path effect.
- **Fix:** many distinct objects (touch each once) + fresh server restart per arm.
The honest signal then collapsed to the ~14% median above.
## Caveats
- **curl-side only.** Server-side `mc admin trace` TTFB was not captured this run;
curl TTFB includes connect/request-send overhead the server timer would not see
(small over loopback, but present).
- **Real HDD, but noisy.** Despite being genuine HDD, the D3-behind-Nitro path has
its own caching/queueing; treat per-round numbers as noise and the pooled median
as the result.
- **Magnitude below §6.** ~14% median vs the design's ≥25% HDD prediction. Plausible
causes: EC:3 metadata fan-out is parallel across spindles (≈ one seek of
wall-clock, not six), and fixed RPC/processing overhead dilutes the single saved
open/seek as a percentage of TTFB.
- **GET only.** The shadow doubles write cost; PUT numbers from this rig are
meaningless by design.
## Raw artifacts
Local (not checked in), under `/tmp/st-hdd-bench/` on the operator's Mac:
- `agg-on.txt`, `agg-off.txt` — 300 pooled samples per arm
- `r{1,2,3}-{ON,OFF}.txt` — per-round 100-sample sweeps
- `urls-2m-big-signed.txt` — the 100 presigned URLs
- driver scripts: `cold-sweep.sh`, `multi-round.sh`, `launch.sh` (on the nodes),
`stats.sh`
docs/single-trip-hdd-bench/two-node-d3-640kib-ec2.md
-104
View File
@@ -1,104 +0,0 @@
# Two-Node D3 HDD Bench — 640 KiB Cold TTFB (EC:2)
June 5, 2026 single-trip GET phase-1 cold-TTFB A/B on the two-node AWS `d3.xlarge`
real-HDD cluster, reconfigured to **EC:2** so the 6-drive set has **4 data drives
+ 2 parity**. Companion to `docs/single-trip-hdd-bench/two-node-d3-2mib.md` (same
rig, same method); read that first for the rig and method details.
## Headline
At 640 KiB the fast path is **not a clean win — it is a crossover**. Pooled across
300 cold samples per arm:
- **Low percentiles favor OFF, dramatically:** p10 ON +288%, p25 ON +210% (OFF
floor ~2 ms vs ON floor ~4 ms).
- **Median is roughly a wash, slightly worse for ON:** p50 +11.8%.
- **The tail favors ON:** p90 −26%, p95 −26%, mean −18%.
The fast path's per-request overhead (EOF-stream open + cancel-not-drain close, no
connection reuse — design §9.9) sets a ~4 ms floor that dominates small/fast GETs,
while its saved metadata seek only pays off on the seek-bound tail. At 2 MiB the
saved seek dominated (clean ~14% median win); at 640 KiB the fixed overhead and the
seek-saving roughly cancel in the middle and the result splits by percentile.
## Config change vs the 2 MiB run
- Standard storage class set to **`EC:2`** via `MINIO_STORAGE_CLASS_STANDARD=EC:2`
(env, both nodes; verified `mc admin info` → `EC:2` and process environ).
- 6 drives → **4 data + 2 parity**. A 640 KiB object splits into 4 × 160 KiB data
shards (> 128 KiB inline cutoff), so a standalone `DataDir/part.1` and shadow are
written. Verified on disk: `xl.meta` 370 B, `current/part.1` 164,896 B
(160 KiB shard + header).
- Dataset: bucket `singletrip-cold-640k-ec2`, 100 distinct 640 KiB objects, random
content, loaded under `FAST_GET=1`.
## Method
Identical to the 2 MiB run: 100 distinct cold first-touch objects per sweep, page
cache dropped on **both** nodes before each GET, **fresh server restart per arm**
(clears buckit's in-process metadata cache), client-side **curl** TTFB
(`time_starttransfer`) host-local on node1 against loopback, 3 alternating rounds →
300 samples/arm. No `mc admin trace` (curl-side only).
**Fast-path verified:** a 30-GET spot check on this dataset returned
`fast_get_hits_total = 30`, `fast_get_fallbacks_total = 0` — the ON arm is genuinely
single-trip, so the crossover below is not a silent-fallback artifact.
## Results
### Pooled cold TTFB, 640 KiB EC:2 (n = 300 per arm)
| percentile | OFF (ms) | ON (ms) | ON vs OFF |
|---|---:|---:|---:|
| p10 | 2.23 | 8.66 | +287.8% |
| p25 | 5.45 | 16.89 | +210.1% |
| p50 | 20.25 | 22.63 | +11.8% |
| p75 | 33.31 | 29.71 | −10.8% |
| p90 | 53.96 | 39.76 | −26.3% |
| p95 | 61.28 | 45.58 | −25.6% |
| mean | 30.22 | 24.71 | −18.2% |
Floors (5 fastest of 300): OFF ~1.7–1.9 ms, ON ~3.9–4.2 ms.
### Per-round medians (rig noise)
| round | ON median (ms) | OFF median (ms) |
|---|---:|---:|
| 1 | 21.47 | 10.30 |
| 2 | 19.66 | 28.52 |
| 3 | 27.67 | 19.75 |
As at 2 MiB, round-to-round variance swamps the per-round ON/OFF gap (round 2 OFF
even had a single 677 ms outlier). Only the pooled aggregate is meaningful.
## Interpretation
- **The ~2 ms OFF floor is not a fully-cold read.** 640 KiB with a 370 B `xl.meta`
is tiny; the D3-behind-Nitro device layer appears to serve a chunk of these from
a cache `drop_caches` cannot clear, giving OFF a large population of ~2 ms
responses. The OFF path can return that fast because its cached-metadata + small
read has little fixed machinery.
- **The ON floor is set by the fast path itself (~4 ms).** Opening `current/part.1`
as an EOF stream per disk and closing with cancel-not-drain (no connection reuse)
is a heavier per-request path than a warm small `xl.meta` read. For small objects
where the disk work is cheap, this fixed cost makes ON *slower* at the fast end.
- **ON wins only where a real seek happens** — the p75–p95 tail and the mean —
because there the one saved metadata seek outweighs the fixed overhead.
- **Net:** at 640 KiB on this rig the fast path is a tail/mean improvement
(−18% mean, −26% p90) bought at the cost of low-percentile and median latency.
This is the opposite balance from 2 MiB and is consistent with §9.9's
cancel-close-churn caveat scaling worse as object size shrinks.
## Caveats
Same as the 2 MiB doc: curl-side only; real-HDD but very noisy (pooled only);
GET-only; magnitudes are rig-specific. Additionally: the ~2 ms OFF floor suggests
imperfect cold isolation on the D3/Nitro device cache, which inflates OFF's
low-percentile advantage — a server-side `mc admin trace` cross-check would help
separate device-cache effects from the path difference.
## Raw artifacts
Local under `/tmp/st-hdd-bench/` (not checked in): `agg-on.txt`/`agg-off.txt`
(300 samples/arm), `r{1,2,3}-{ON,OFF}.txt`, `urls-640k-ec2-clean.txt`,
`multi-round-640k.log`.
docs/single-trip-parallelreader-deadlock-handoff.md
-145
View File
@@ -1,145 +0,0 @@
# Single-Trip `parallelReader` Deadlock Handoff
## Context
Branch: `fix/singletrip-recon-parallelreader`
The eager single-trip GET path prefetches the first encoded block from local
shards, keeps remote shard streams positioned after the direct header, and calls
`Erasure.Decode` with local readers marked as preferred.
Claude's current uncommitted `cmd/singletrip-read.go` change already replaces the
old sparse, preselected M-reader layout with the complete agreeing M+N reader
layout. The hang remained after that change.
Observed blocked stack:
```text
parallelReader.Read at cmd/erasure-decode.go:161 (chanrecv)
Erasure.Decode
getObjectWithSingleTripInfo at cmd/singletrip-read.go:434
```
## Root Cause
`parallelReader.preferReaders` reordered `p.readers` to put preferred local
readers first, but did not apply the same permutation correctly to
`p.readerToBuf`.
The old code was:
```go
p.readers[next], p.readers[i] = p.readers[i], p.readers[next]
p.readerToBuf[next] = i
p.readerToBuf[i] = next
```
`readerToBuf` may already contain a permutation from an earlier swap. Assigning
the current indices instead of swapping the existing mapping values can create
duplicate output-buffer mappings.
One relevant six-drive EC:2 case is M=4, N=2 with three local preferred readers.
For preferred positions `[1, 2, 3]`, the old logic produces:
```text
readers: [1, 2, 3, 0, 4, 5]
readerToBuf: [1, 2, 3, 2, 4, 5]
```
The first four reads can all succeed, but readers at positions 1 and 3 both
write to output buffer 2. Only three distinct buffers become non-empty.
`canDecode()` therefore remains false for M=4.
Successful reads send `false` to `readTriggerCh`. Once those notifications are
consumed, no goroutine sends another trigger and the channel remains open.
The loop at `cmd/erasure-decode.go:161` then waits forever on an empty channel.
## Fix
File: `cmd/erasure-decode.go`
Apply the exact same swap to the mapping as to the readers:
```go
p.readers[next], p.readers[i] = p.readers[i], p.readers[next]
p.readerToBuf[next], p.readerToBuf[i] = p.readerToBuf[i], p.readerToBuf[next]
```
This preserves a one-to-one permutation from reordered reader positions to the
readers' original erasure-shard buffer positions.
## Regression Test
File: `cmd/erasure-decode_test.go`
Added `TestParallelReaderPreferReadersDoesNotDeadlock`:
- Creates six one-byte readers with M=4.
- Marks positions `[1, 2, 3]` preferred, matching three local disks in the
two-host six-drive layout.
- Calls `parallelReader.Read` in a goroutine.
- Fails if it does not return within one second.
- Verifies that decode does not return an error.
The existing uncommitted
`TestSingleTripDecodeMixedReadersReconstructNoHang` in
`cmd/singletrip-get_test.go` was also run. It exercises M=4/N=2 with three fast
local readers, one slow remote parity reader, and reconstruction of a missing
data shard.
Targeted command:
```sh
GOCACHE=/tmp/buckit-go-cache CGO_ENABLED=0 \
go test -tags kqueue,dev ./cmd \
-run 'TestParallelReaderPreferReadersDoesNotDeadlock|TestSingleTripDecodeMixedReadersReconstructNoHang' \
-count=1 -timeout=45s
```
Result: PASS.
`git diff --check` also passed.
## Broader Test Caveat
The existing `TestErasureDecode` suite was attempted separately:
```sh
GOCACHE=/tmp/buckit-go-cache CGO_ENABLED=0 \
go test -tags kqueue,dev ./cmd -run '^TestErasureDecode$' \
-count=1 -timeout=2m
```
It did not reach a useful result because an unrelated goroutine panicked with an
integer divide by zero in `internal/ringbuffer/ring_buffer.go:212`, reached from
`xlStorage.writeAllDirect`. This occurred during test storage setup, outside the
changed `parallelReader` logic.
## Validation Still Needed
1. Review the full-layout changes already present in the dirty working tree and
keep them separate from this mapping fix when committing.
2. Build and deploy the patched binary to both benchmark hosts.
3. Repeat the stress workload that previously hung, ideally with goroutine dumps
enabled so any remaining block can be compared with the old line-161 stack.
4. Validate returned bytes and fast-path hit/fallback counters after stress.
5. Run longer ON/OFF tests for 640 KiB, 1 MiB, and 2 MiB only after the hang is
shown to be resolved.
6. Investigate remote `ReadFileStream` cancellation separately. It can cause
resource pressure under load, but it does not explain the specific empty
`readTriggerCh` receive described above.
## Working Tree Warning
At the time of this handoff, these files already contain uncommitted work:
```text
cmd/erasure-decode.go
cmd/erasure-decode_test.go
cmd/singletrip-get_test.go
cmd/singletrip-read.go
.tmp/
```
Do not discard the existing `singletrip-read.go` or `singletrip-get_test.go`
changes when isolating or committing this fix.
docs/single-trip-recon-review.md
-431
View File
@@ -1,431 +0,0 @@
# Single-Trip Fast GET — Prefer-Local + Parity Reconstruction: Review Guide
Audience: a reviewer (Claude Code or human) picking up the change cold. This
explains **what changed, why, and how it was validated**, and points at the parts
that most need scrutiny. Companion doc: `single-trip-parallelreader-deadlock-handoff.md`
(Codex's root-cause note).
Branch: `fix/singletrip-recon-parallelreader` (built on the single-trip phase-1 work).
---
## TL;DR
1. **A general erasure-coding bug was found and fixed** in
`parallelReader.preferReaders` (`cmd/erasure-decode.go`) — a one-line fix. It is
independent of single-trip and should be reviewed/committed on its own.
2. **The single-trip fast GET path gained prefer-local shard selection + parity
reconstruction** (and an opt-in eager first-block prefetch). This is what first
*exercised* the latent `preferReaders` bug.
3. **Result (2-node `d3.xlarge` real-HDD rig, EC:2, reboot-cold, n=300/arm/size):**
the fast path is a **modest, size-dependent** cold-TTFB win, not the −63% an
earlier `drop_caches` run suggested (that was a cold-cache artifact). Best variant
**ON-eager**: median **−11% / −5% / −12% / −7%** at 640 KiB / 1 MB / 2 MB / 4 MB, with the best
tail and mean at every size. `ON-stream` is erratic (−25% at 640 KiB but **+3%**
at 1 MB). See §3 "Rig reboot-cold A/B — AUTHORITATIVE."
---
## 1. The core fix — `parallelReader.preferReaders` (review first)
File: `cmd/erasure-decode.go` (1 line) + `cmd/erasure-decode_test.go` (regression test).
```diff
p.readers[next], p.readers[i] = p.readers[i], p.readers[next]
-p.readerToBuf[next] = i
-p.readerToBuf[i] = next
+p.readerToBuf[next], p.readerToBuf[i] = p.readerToBuf[i], p.readerToBuf[next]
```
**Bug:** `preferReaders` reorders `p.readers` to put preferred (local) readers first.
It must apply the *same permutation* to `p.readerToBuf` (reader-position →
output-buffer-index). The old code *assigned current indices* instead of *swapping
the existing mapping values*. Across chained swaps this produces **duplicate
output-buffer mappings** — two readers write the same buffer, so fewer than
`dataBlocks` distinct buffers fill, `canDecode()` never becomes true, and
`parallelReader.Read` blocks forever on its trigger channel
(`cmd/erasure-decode.go:161`).
Concrete EC:2 (M=4,N=2) case, preferred positions `[1,2,3]`:
```
readers: [1, 2, 3, 0, 4, 5]
readerToBuf: [1, 2, 3, 2, 4, 5] ← buf 2 mapped twice; only 3 distinct buffers fill
```
**Why it was dormant:** `preferReaders` only reorders when some `prefer[i]` is true.
The canonical decode path sets `prefer[i] = disk.Hostname() == ""`, which is a
**no-op in distributed deployments with hostnamed endpoints** (`http://node/path`
→ local disks still report a non-empty `Hostname()`), so nothing was ever marked
preferred and the buggy branch never ran. The single-trip change below sets
`prefer[i] = disk.IsLocal()` (the correct signal), which finally exercised it.
**Regression test:** `TestParallelReaderPreferReadersDoesNotDeadlock` — 6 readers,
M=4, prefer `[1,2,3]`, watchdog-fails if `Read` doesn't return in 1s. Verified
load-bearing: reverting the one-liner makes the test deadlock/fail; the fix makes
it pass.
**Note:** consider whether the canonical path's `prefer[i] = disk.Hostname()==""`
(`cmd/erasure-object.go`, `cmd/erasure-healing.go`) should also be `IsLocal()` —
prefer-local is currently a no-op there in hostnamed-endpoint clusters. Out of scope
for this change, but worth a follow-up.
---
## 2. Single-trip fast path changes
All in `cmd/singletrip-read.go` unless noted. The fast path reads `current/part.1`
(direct header + shard, co-located in one file) instead of `xl.meta` + `DataDir/part.1`.
What changed from the original single-trip fast path:
- **Quorum relaxed to "any decodable M"** (`pickSingleTripFastInfo`): accept a
`directSig`-agreeing group with **≥ M distinct valid indices** (data and/or
parity), instead of requiring all M *data* indices. This is what allows parity
reconstruction.
- **Full M+N reader layout handed to the decoder** (`buildSingleTripFastInfo`):
builds a reader for **every agreeing shard** at its `erasureIndex-1` position —
the layout `parallelReader` is designed for. (An earlier attempt built a sparse
M-only/nil-padded slice; that was a dead end and is gone.)
- **Prefer-local selection via the decoder**: `getObjectWithSingleTripInfo` sets
`prefer[i] = disk.IsLocal()` (line ~432). `parallelReader` then reads the cheapest
M shards (locals first) and **reconstructs missing data from parity** if a local
shard is parity / a data shard is remote. This minimizes cross-node reads
(3 local + 1 remote on a 3+3 layout) instead of chasing data shards that may be
remote.
- **Close-after-response ordering**: only non-agreeing reads are closed in-path
(`closeSingleTripHeaderReadsKeepGroup`); the selected readers' streams are owned
by `info.readers` and closed after the response (`closeBitrotReaders`), matching
the original path. (Closing partially-read remote streams *in-path before the
response* was investigated as a hang cause and ruled out; the real cause was §1.)
- **Eager first-block prefetch (opt-in, `BUCKIT_FASTGET_EAGER=1`)**: for a LOCAL
shard, `readSingleTripHeader` reads the header **and first encoded block** in one
go off the open stream, then decode serves block 0 from memory and streams the
rest (`multiReadCloser` = first-block buffer + remaining stream). Removes the
header→body barrier so local shards decode instantly. Remote shards keep a live
stream. **Default OFF** (see §3 for why).
Supporting:
- `cmd/singletrip-get.go`: `globalFastGetEager` env gate.
---
## 3. Validation
### Unit tests (`-tags kqueue,dev`, both `BUCKIT_FASTGET_EAGER` on and off)
- `TestParallelReaderPreferReadersDoesNotDeadlock` — the core-fix regression guard.
- `TestSingleTripFastGetReconstructsFromParity` — deletes a data shard's shadow so
the fast path must reconstruct from parity; asserts bytes + fast-hit.
- `TestSingleTripDecodeMixedReadersReconstructNoHang` — `erasure.Decode` with
instant + slow readers + forced reconstruction, watchdog'd. (Note: this passes
even *without* the §1 fix — the deadlock needs the real `preferReaders` reorder
path; kept as a decode-level guard.)
- Full `SingleTrip|FastGet` suite passes both modes.
### Rig reboot-cold A/B — AUTHORITATIVE (2× `d3.xlarge`, EC:2 = 4 data + 2 parity, 3 local + 3 remote)
This is the trustworthy result; the `drop_caches` pooled section further below is
**superseded** (see "Why reboot-cold" — small objects didn't re-cool under
`drop_caches`, inflating the 640 KiB win).
Method: fresh AWS `d3.xlarge` pair, one 6-drive EC:2 set, 100 distinct objects/size
(640 KiB, 1 MB, 2 MB, 4 MB) seeded once. Per arm-cycle: **reboot both hosts** (clears all
RAM cache; instance-store survives reboot, fstab auto-remounts), relaunch the arm,
wait for `6 drives online` + a readiness probe on a *throwaway* object (so the
post-launch 503 window never touches measured objects and they stay genuinely cold),
then one cold pass = 100 distinct first-touch GETs. **3 reboot cycles × 100 =
300 cold samples/arm/size.** Arms on one binary: **OFF** `BUCKIT_FAST_GET=0`;
**ON-stream** `FAST_GET=1, EAGER=0` (prefer-local + reconstruction); **ON-eager**
`FAST_GET=1, EAGER=1` (adds first-block prefetch). 0 × 503 across all runs.
Cold first-byte TTFB medians (ms):
| size | OFF | ON-stream | ON-eager |
|---|---:|---:|---:|
| 640 KiB | 14.75 | **11.12 (−25%)** | 13.20 (−11%) |
| 1 MB | 22.48 | **23.26 (+3%)** ⚠ | 21.24 (−5%) |
| 2 MB | 21.16 | 19.47 (−8%) | **18.56 (−12%)** |
| 4 MB | 23.15 | 21.86 (−6%) | **21.60 (−7%)** |
Cold first-byte TTFB **p90** (ms) — the tail, where the variants diverge most:
| size | OFF | ON-stream | ON-eager |
|---|---:|---:|---:|
| 640 KiB | 24.66 | 26.02 (**+6%**) | **22.59 (−8%)** |
| 1 MB | 37.46 | 34.51 (−8%) | **28.19 (−25%)** |
| 2 MB | 28.77 | 32.77 (**+14%**) | **24.78 (−14%)** |
| 4 MB | 37.52 | 28.93 (−23%) | **27.53 (−27%)** |
At p90 **ON-eager wins at every size** (−8 / −25 / −14 / −27%), and by *more* than at
the median — the eager block-0-from-memory decode is what protects against the
slow-shard-open tail. **ON-stream regresses at p90 for 640 KiB (+6%) and 2 MB (+14%)**
(lazy streaming has no tail protection), so on tail latency it is often *worse than
OFF*. This is the strongest argument for the ON-eager default.
Full distributions (ms), `n=300/arm`:
**640 KiB** (single block, 160 KiB shard):
| pct | OFF | ON-stream | ON-eager |
|---|---:|---:|---:|
| p10 | 2.98 | 3.01 | 2.63 |
| p25 | 3.12 | 3.20 | 2.78 |
| **p50** | 14.75 | **11.12** | 13.20 |
| p75 | 19.19 | 18.86 | 19.27 |
| p90 | 24.66 | 26.02 | **22.59** |
| p95 | 27.69 | 32.17 | **24.48** |
| p99 | 61.33 | 68.18 | **45.77** |
| mean | 13.36 | 12.51 | **12.07** |
| max | 68.59 | 68.78 | 65.14 |
per-cycle p50: OFF 14.1/15.5/14.5 · ONS 10.7/11.3/11.3 · ONE 14.4/11.6/13.0
**1 MB** (exactly one full block = `blockSizeV2`, 256 KiB shard — the largest single-block):
| pct | OFF | ON-stream | ON-eager |
|---|---:|---:|---:|
| p10 | 14.83 | 13.06 | 14.09 |
| p25 | 18.79 | 17.77 | 17.34 |
| **p50** | 22.48 | 23.26 | **21.24** |
| p75 | 27.51 | 28.23 | **24.62** |
| p90 | 37.46 | 34.51 | **28.19** |
| p95 | 43.01 | 38.64 | **30.72** |
| p99 | 90.03 | 77.00 | 76.74 |
| mean | 24.23 | 23.57 | **21.55** |
| max | 121.54 | 78.19 | 81.82 |
per-cycle p50: OFF 22.0/22.2/22.9 · ONS 23.3/23.7/22.7 · ONE 21.1/21.2/21.6
**2 MB** (2 blocks, 512 KiB shard):
| pct | OFF | ON-stream | ON-eager |
|---|---:|---:|---:|
| p10 | 9.75 | 9.86 | 8.36 |
| p25 | 17.06 | 15.96 | 14.89 |
| **p50** | 21.16 | 19.47 | **18.56** |
| p75 | 23.48 | 23.61 | **21.42** |
| p90 | 28.77 | 32.77 | **24.78** |
| p95 | 34.17 | 38.62 | **25.61** |
| p99 | 42.75 | 52.13 | **40.32** |
| mean | 20.23 | 20.42 | **17.93** |
| max | 43.62 | 55.56 | 47.37 |
per-cycle p50: OFF 19.8/21.5/21.5 · ONS 19.3/20.0/19.6 · ONE 18.4/18.6/18.8
**4 MB** (4 blocks, 1 MiB shard; eager prefetches only block 0 = 25% of shard):
| pct | OFF | ON-stream | ON-eager |
|---|---:|---:|---:|
| p10 | 16.82 | 17.64 | 17.06 |
| p25 | 20.43 | 19.91 | 19.44 |
| **p50** | 23.15 | 21.86 | **21.60** |
| p75 | 30.96 | 25.24 | **24.83** |
| p90 | 37.52 | 28.93 | **27.53** |
| p95 | 40.90 | 35.56 | **31.58** |
| p99 | 92.94 | 67.37 | 69.42 |
| mean | 26.00 | 23.48 | **22.52** |
| max | 127.37 | 97.66 | 70.38 |
per-cycle p50: OFF 23.1/23.3/23.4 · ONS 21.9/22.1/21.8 · ONE 21.6/21.6/21.5
**Reading the reboot-cold results:**
- **The win is real but modest and *non-monotonic* in object size** — not the −63%
the `drop_caches` method suggested. The fast path saves the small metadata-read
phase (~2–3 ms, `xl.meta` is ~366 B); the dominant cost (cold block-0 *data* read,
paid by both OFF and ON) is unchanged. So the relative win tracks "how big is the
cheap metadata phase vs the data read," which isn't monotonic.
- **1 MB is the weakest case** (one full single block): **ON-stream regresses to
+3% vs OFF** (real — per-cycle 23.3/23.7/22.7 vs 22.0/22.2/22.9), while ON-eager
still wins −5%. Lazy streaming loses its edge exactly at the full-block boundary;
the eager block-0-from-memory decode is what keeps ON-eager ahead there.
- **ON-eager is the only consistent performer:** −11% / −5% / −12% / −7% across
640 KiB / 1 MB / 2 MB / 4 MB, and it has the **best tail (p90/p95/p99) and mean at
*every* size** (e.g. 4 MB p90 27.5 vs OFF 37.5). The earlier "eager has a bad
single-block tail" was a transient under `drop_caches`; under clean reboot-cold its
tail is the best.
- **ON-stream is erratic:** −25% (640 KiB), **+3% (1 MB, regression)**, −8% (2 MB),
−6% (4 MB), with worse tails at the smaller sizes. The +3% at 1 MB is a **localized
full-single-block-boundary** effect — it recovers to a modest win at 2–4 MB — but
it shows lazy streaming has no reliable edge across sizes.
- Per-cycle medians are tight across the 3 reboots → reproducible cold; **0 × 503**
(readiness probe worked).
**Recommended default: ON-eager** — the only variant that wins or ties at every size
and owns the tail/mean. ON-stream's 640 KiB median edge isn't worth its 1 MB
regression and worse tails.
**Why reboot-cold (and why the `drop_caches` numbers below are superseded):** with
`drop_caches=3` before each GET, *small* objects (640 KiB) did **not** actually
re-cool — per-round medians ran 20.6 → 7.3 → 5.9 ms (round 1 cold, then effectively
warm), which inflated the headline 640 KiB win to −63%. Rebooting both hosts forces
a true cold cache every cycle; per-cycle medians are now stable (e.g. OFF 640 KiB
14.1/15.5/14.5), so these are the numbers to trust.
---
### [SUPERSEDED] Rig pooled A/B (`drop_caches` method)
> Kept for history. The 640 KiB figures here are inflated by the cold-cache issue
> described above; use the reboot-cold table instead.
Method: cold first-byte TTFB, page cache dropped on **both** nodes before each GET,
single connection, `curl -w %{time_starttransfer}` issued host-local on node1
against the loopback endpoint; 100 distinct objects/size; **3 alternating rounds ×
100 = 300 samples/arm** (fresh server restart per arm to clear in-process metadata
cache). Arms: **OFF** = canonical `xl.meta` path; **ON-stream** = fast path,
`BUCKIT_FASTGET_EAGER=0` (prefer-local + reconstruction, no prefetch); **ON-eager**
= fast path, `BUCKIT_FASTGET_EAGER=1` (adds first-block prefetch).
All six arms returned **300/300 HTTP 200, zero timeouts**. (Pre-fix, the
prefer-local/eager arms hung ~15% with 30 s stalls; see §1.) Full cold-TTFB
distribution (ms):
**640 KiB (single-block):**
| metric | OFF | ON-stream | ON-eager |
|---|---:|---:|---:|
| n | 300 | 300 | 300 |
| min | 4.11 | 3.91 | 3.51 |
| p10 | 4.48 | 4.36 | 3.90 |
| p25 | 4.92 | 4.56 | 4.07 |
| **p50** | **13.95** | **5.10** | **4.83** |
| p75 | 21.65 | 19.35 | 18.41 |
| p90 | 28.94 | 24.39 | 25.81 |
| p95 | 36.77 | 31.62 | 69.06 ⚠ |
| p99 | 47.90 | 50.25 | 707.64 ⚠ |
| max | 120.55 | 152.11 | 962.04 ⚠ |
| mean | 15.18 | 12.80 | 29.73 ⚠ |
| **median vs OFF** | — | **−63%** | **−65%** |
**2 MiB (multi-block):**
| metric | OFF | ON-stream | ON-eager |
|---|---:|---:|---:|
| n | 300 | 300 | 300 |
| min | 4.82 | 4.79 | 4.30 |
| p10 | 14.69 | 12.40 | 11.83 |
| p25 | 20.22 | 17.51 | 17.46 |
| **p50** | **23.87** | **21.94** | **20.40** |
| p75 | 30.98 | 29.12 | 23.95 |
| p90 | 41.88 | 39.45 | 29.37 |
| p95 | 47.09 | 46.48 | 39.46 |
| p99 | 79.56 | 63.40 | 72.18 |
| max | 192.44 | 131.99 | 102.88 |
| mean | 27.14 | 24.33 | 21.83 |
| **median vs OFF** | — | **−8%** | **−15%** |
Per-round medians (rounds of 100 — shows the rig's run-to-run noise; read the
pooled distribution above, not any single round):
| | OFF | ON-stream | ON-eager |
|---|---|---|---|
| 640 KiB | 20.6 / 7.3 / 5.9 | 12.4 / 12.8 / 4.7 | 4.7 / 15.9 / 4.3 |
| 2 MiB | 23.1 / 23.8 / 24.2 | 21.8 / 24.8 / 20.6 | 20.0 / 21.3 / 19.9 |
How to read it:
- **Both fast-path arms beat OFF at the median for both sizes.** The single-block
win is large (−63%/−65%); the multi-block win is smaller at the median but
ON-eager widens across the upper percentiles (p90 41.9 → 29.4, mean 27.1 → 21.8).
- **ON-eager's single-block tail (p95 69 / p99 708 / max 962 ms) is a single
temporal stall, not steady overhead.** All 10 of its >100 ms outliers fall in
round 2, positions 128–159 (a contiguous burst; six were 624–962 ms). Per-round
means: 10.4 / **69.2** / 9.6 ms. OFF and ON-stream had a mild round-2 bump too
(1–2 outliers each), so that window was generally noisy — but the arm order was
fixed OFF → ON-stream → ON-eager every round, so ON-eager always ran last and
caught the worst of it. The tail is therefore **temporally confounded**, not shown
to be steady eager overhead. Likely mechanism if eager *does* amplify such stalls:
`readSingleTripHeader` synchronously reads the first block from **all three local
disks** and `openSingleTripFastInfo` waits for **every** disk goroutine
(`g.Wait`), so one slow local HDD blocks the whole open phase, and the required
remote shard is only read *after* — making local-tail and remote-transfer
additive. ON-stream starts its selected body reads together after the header
barrier, allowing more overlap. (Drain-on-close is *not* the leading explanation:
cleanup runs after first-byte, the custom reader's `Close()` doesn't drain, and
ON-stream has the same unused remote streams without the tail.)
**Rerun result (ON-eager, cold 640 KiB, n=300):** the
severe tail did **not** reproduce — median 9.2 ms, p95 37 ms, **max 206 ms, only
2 GETs >100 ms** (vs the pooled run's 10 up to 962 ms), confirming a one-time
transient, not steady eager overhead. Per-GET profiling of the slow ones shows
they are **open-phase-bound and pinned to the slowest *remote* `ReadFileStream`
open**: `maxRemoteOpen` ≈ `openphase` (e.g. 204.2 ≈ 204.4 ms), while
`maxLocalBody` stays ≤4 ms on the slow GETs (56 ms max overall) and `firstbyte` is
~0.3 ms. So the cause is **cross-node grid-open latency variance** — *not* a local
body stall (Codex's predicted 700–900 ms local `body_ms` did not appear) and not
drain-on-close. The open phase `g.Wait`s on **all 6** disks, so one slow remote
open blocks the whole GET; this exposure is **shared by ON-stream** (also opens all
6), so it is not eager-specific. Mitigation if pursued: early-quorum — proceed once
the needed M shards (prefer-local) are ready instead of waiting on slow *unneeded*
remote opens.
- **640 KiB is noisy round-to-round** (OFF medians 20.6/7.3/5.9). The pooled p50 is
the right summary; don't over-read a single round. 2 MiB is much steadier.
- The −8% vs −63% gap between sizes is expected: at 640 KiB metadata/open dominates
cold TTFB (so collapsing it helps a lot), while at 2 MiB transfer is a larger
share of first-byte.
Byte-correctness: md5 MATCH on both sizes (`current/part.1` served with `xl.meta`
absent on disk). Single-open-per-disk invariant preserved (one `ReadFileStream` per
participating disk; asserted by the end-to-end test).
**Caveat:** the rig HDDs are AWS d3 (real spinning disks behind Nitro NVMe; the
`rotational=0` flag is a Nitro artifact). They share backing storage and a page-cache
layer, so treat these as **relative off-vs-on ratios**, not absolute HDD figures.
Raw per-sample data: `/tmp/st-hdd-bench/pool-{640k,2m}-{OFF,ONS,ONE}.txt`.
---
## 4. Recommended default & open items
- **Default = ON-eager** (`FAST_GET=1, EAGER=1`): per the authoritative reboot-cold
results (§3) it's the only variant that wins or ties at **every** size (−11% / −5%
/ −12% / −7% at 640 KiB / 1 MB / 2 MB / 4 MB) and has the **best tail and mean at every size**.
The earlier "eager has a bad single-block tail" was a `drop_caches`-era transient;
under clean reboot-cold its tail is the best.
- **ON-stream (eager off) is not recommended as default:** erratic across sizes
(−25% at 640 KiB but **+3% regression at 1 MB**, −8% at 2 MB) with worse tails at
the smaller sizes. The full-block (1 MB) boundary is where lazy streaming loses to
OFF; eager's block-0-from-memory decode is what avoids that.
- **Magnitude expectation:** the cold-TTFB win is **modest (~5–12% at the median for
ON-eager, larger in the tail)**, because the fast path only elides the small
metadata-read phase (~2–3 ms; `xl.meta` ≈ 366 B) while the dominant cost — the cold
block-0 *data* read — is paid by both arms. Don't expect the (artifactual) −63%.
- **Earlier eager-tail investigation (now moot for the default but still informative):**
- **Rerun done (ON-eager, cold 640 KiB, n=300, profiling on):** tail did not
reproduce (median 9.2, p95 37, max 206 ms, 2 outliers). Slow GETs are
open-phase-bound with `maxRemoteOpen` ≈ `openphase` (up to 204 ms) and
`maxLocalBody` ≤4 ms — i.e. **cross-node `ReadFileStream`-open latency variance**,
blocking the whole GET because the open phase `g.Wait`s on all 6 disks. Shared by
ON-stream (also opens all 6), so not eager-specific. (Codex's predicted 700–900 ms
local `body_ms` did not appear; drain-on-close already ruled out.)
- **Mitigation:** early-quorum — proceed once the needed M shards (3 local + fastest
remote) are ready instead of waiting on the 2 slow *unneeded* remote opens. Helps
both fast-path arms. Also worth: investigate why the remote grid open occasionally
spikes to 100s of ms (node2/grid/network).
- **Prototype-only code to strip before merge if landing for real:**
the `BUCKIT_FASTGET_*` diagnostic env gates.
- **Out-of-scope follow-ups:** canonical-path `prefer` should use `IsLocal()` (§1
note); remote `ReadFileStream` cancellation under load (separate from the deadlock).
---
## 5. Reproduce
Unit:
```sh
CGO_ENABLED=0 go test -tags kqueue,dev ./cmd \
-run 'TestParallelReaderPreferReadersDoesNotDeadlock|SingleTrip|FastGet' -count=1
```
Rig (scripts under `/tmp/st-hdd-bench/` on the operator's Mac, IPs hardcoded):
- `pooled-both.sh` → pooled OFF/ON-stream/ON-eager for 640 KiB + 2 MiB.
- Per-arm: `run/launch.sh <FAST_GET> EC:2` on both nodes. Raw data:
`pool-{640k,2m}-{OFF,ONS,ONE}.txt`.
## 6. Suggested commit split
1. `cmd/erasure-decode.go` + `cmd/erasure-decode_test.go` — the `preferReaders`
permutation fix (general; standalone).
2. `cmd/singletrip-*.go` (+ tests) — prefer-local + reconstruction + eager prefetch
+ profiler, and the bench docs.
docs/single-trip-warp-load-handoff.md
-1302
View File
File diff suppressed because it is too large Load Diff