overtrue
524c774e2f
Scan findings in the Iceberg table catalog:
- Metadata maintenance deleted every unreferenced *.metadata.json in a
table's directory without checking whether a renamed table in another
namespace still owned those files (backlog#3060); it now resolves the
owning table through the directory index before deleting.
- Manifest lists and manifests were decoded with the embedded Avro
writer schema and no recursion depth limit; a self-referencing record
schema overflowed the stack (backlog#3080). Schemas are validated for
depth before decoding.
- Compaction decoded every input Parquet file into Arrow batches held in
memory; a decompression bomb OOM-killed the server (backlog#3085). Row
groups are streamed through a byte budget and oversized inputs are
refused.
- An S3 request for a key outside an indexed table prefix listed the
bucket's whole namespaces/ catalog prefix and failed once it held more
than 4096 objects (backlog#3086); a per-bucket directory index answers
index misses without the listing.
- The maintenance planner read whole orphan files to obtain their
modification time (backlog#3087); it reads metadata only.
- A client disconnect between persisting active_rename_id and clearing it
left the bucket's catalog entry locked at 503 (backlog#3088); the rename
runs detached from the request future and clears the flag on every exit.
- Table commits held the bucket publication fence until publication,
starving every object write (backlog#3111); commits now hold it under a
deadline and object writes are admitted between commits.
The directory index is managed through three new admin routes under
{warehouse}/catalog/directory, registered in the admin route policy.
The durable-strong store is left as is (its snapshot partitioning for
backlog#3109 and the strong-path deadline are not ported): that backing
is being retired separately, so the new tests cover the object backing.
Documentation
Use the focused indexes rather than treating this directory as an unordered collection:
Operations
-
Sampled local batch CPU diagnostics — default-off request-correlated allocation and worker wall/CPU diagnosis.
-
Retained MRF responsibilities — unknown bucket identities, bounded probes, and unreadable checkpoint recovery.
-
Read-only BTF declaration inspection — bounded prototypes and scalar member layouts, explicit void handling, and independent native references.
-
Internode HTTP streaming transport — protocol selection, pooled connection isolation, and distributed read validation.
-
Persisted config compatibility — ignored MinIO settings, bounded diagnostics, alias retention, and safe resets.
-
Table maintenance periodic queueing — opt-in scheduling, limits, worker handoff, and recovery.
For PUT cleanup admission, blocking-pool isolation, and optional CPU affinity, see PUT cleanup isolation.
-
External certificate issuance and TLS rotation — publication, native reload, trust separation and failure handling.
-
Architecture guard troubleshooting — diagnose repository guard failures.
-
Logging governance — broad event audits, migrations, and guardrail changes.
-
Multipart upload memory diagnosis — allocator attribution, Docker reproduction and sustained upload checks.
For the logical per-operation io_uring read cap, see io_uring read chunk size.
For bounded local read-backend startup and cancellation, see io_uring backend initialization.
For legacy protection assessment and bounded protected copies, see Object integrity inventory, audit, and migration.
For encrypted listing compatibility and older stored fingerprints, see Encrypted object ETags.
For durable hard quota accounting, node-outage behavior, and fence protocol upgrades, see Durable hard quotas.
Operational runbooks live under operations/. Replication
operators should start with:
| Runbook | Use it for |
|---|---|
| Site replication operations | Health fields, pending operations, outage recovery, re-pair admission, IAM/SSE boundaries, and upgrades. |
| Replication target check | Validating an S3 destination and version fidelity before enabling replication. |
| Replication object size limits | Multipart routing, large-object limits, and retry characteristics. |
| Replication outbound transport | Integrity headers, generic target behavior, and transport knobs. |
For the erasure-coded cluster lifecycle (planning, parity and EC:0,
expansion, rebalance, decommission, heal, drive replacement, restart
recovery, and the rc CLI mapping), start with
Cluster and erasure-coding lifecycle operations.
For concurrent bucket creation and deletion, see Bucket operation admission.
For persisted administrator bucket tasks and bucket recreation, see Bucket heal recovery.
For interrupted object deletions and returning drives, see Delete transaction recovery.
For disk replacement across VM restarts and schema 5/6 maintenance migration, see Replacement generation recovery.
For historical GET timeouts during PUT or Heal, see Object lock contention diagnostics.
Other runbooks remain grouped by filename in operations/;
architecture pages link to the relevant runbook where a cross-boundary
procedure is required.
For storage dashboards, see Storage metrics and observer selection: drive ownership, snapshot freshness, counter queries, and rolling upgrades.
For per-bucket capacity and object counts, see Bucket usage metrics: OTLP export, cluster deduplication, accounting semantics, and scanner freshness.
For optional shard commitments, see Independent shard integrity rollout: activation, legacy repair results, multipart mode changes, and rollback limits.
For crates.io publication of workspace crates, see Workspace Cargo Publish: dependency ordering, dry-run, publish, and failure handling.