* fix(ecstore): preserve uncertainty after partial multipart abort Keep recovery callers from reporting NoSuchUpload while minority primary metadata remains. Retain all quorum and deletion checks. Exercise actual partial Abort and explicitly retained late RPC packets after caller retirement, staging cleanup and bucket recreation. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): prevent retired multipart staging resurrection Require the upload root to exist before preparing recovery publication. Preserve rooted directory walks for nested bases on non-Linux Unix and cover missing-source native work, configured aliases and pinned mixed-version coordinator recovery. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(e2e): use IPv4 peer aliases for pinned mixed recovery Keep direct IPv6 S3 listeners behind IPv4 RPC proxies for the pinned previous release, whose peer health resolver rejects bracketed IPv6 hosts. Preserve signed authorities, ports and strict recovery assertions. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(e2e): separate pinned startup from MPU fault budgets Use the old fleet defaults during bootstrap, then apply the original short fault budgets to the current replacement. Keep direct S3 URLs aligned with IPv6 listeners and preserve all quorum and physical cleanup assertions. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(e2e): seed legacy MPU through transparent transport Retire raw proxy connections before enabling method faults so legacy streaming setup stays byte-transparent and every later scoped mutation must traverse the fault barrier. Verify raw and adapted protocol authority, bodies and trailers. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(e2e): retain direct listeners for legacy MPU peers Advertise same-port localhost peers for the pinned resolver while retaining direct IPv4 S3 listeners. Exact scoped RPC hit counts still reject bypassed injection and all physical authority assertions remain unchanged. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(server): keep startup gRPC on the authenticated router Use the hybrid protocol predicate for startup RPC admission so early control responses retain production authentication and owner validation. Keep REST FullReady gating and pinned replay capability rejection intact. Add deterministic regressions, scoped diagnostics and bounded legacy fixture resources. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(ci): verify MPU recovery across upgrade source versions Run the exact sixteen-drive mixed MPU case for every checksum-verified upgrade source, independently of generic compatibility failures. Require one listed test and retain per-version logs while preserving the historical distributed source pin and every recovery assertion. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(ci): trigger upgrade MPU checks for fixture changes Include distributed fixtures and the RPC fault proxy in the upgrade workflow paths so test-only changes schedule the strict recovery gate. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(storage): verify resend owners across MPU finalization Hold real part-transaction RPCs while another coordinator completes or aborts the upload, with both live and cancelled client response waiters. Require full object bytes, terminal upload state, and all staging roots to remain consistent after the part owner drains. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Documentation
Use the focused indexes rather than treating this directory as an unordered collection:
Operations
-
Retained MRF responsibilities — unknown bucket identities, bounded probes, and unreadable checkpoint recovery.
-
Internode HTTP streaming transport — protocol selection, pooled connection isolation, and distributed read validation.
-
Persisted config compatibility — ignored MinIO settings, bounded diagnostics, alias retention, and safe resets.
-
Table maintenance periodic queueing — opt-in scheduling, limits, worker handoff, and recovery.
For PUT cleanup admission, blocking-pool isolation, and optional CPU affinity, see PUT cleanup isolation.
-
External certificate issuance and TLS rotation — publication, native reload, trust separation and failure handling.
-
Architecture guard troubleshooting — diagnose repository guard failures.
-
Logging governance — broad event audits, migrations, and guardrail changes.
-
Multipart upload memory diagnosis — allocator attribution, Docker reproduction and sustained upload checks.
For the logical per-operation io_uring read cap, see io_uring read chunk size.
For bounded local read-backend startup and cancellation, see io_uring backend initialization.
For legacy protection assessment and bounded protected copies, see Object integrity inventory, audit, and migration.
For encrypted listing compatibility and older stored fingerprints, see Encrypted object ETags.
For durable hard quota accounting, node-outage behavior, and fence protocol upgrades, see Durable hard quotas.
Operational runbooks live under operations/. Replication
operators should start with:
| Runbook | Use it for |
|---|---|
| Site replication operations | Health fields, pending operations, outage recovery, re-pair admission, IAM/SSE boundaries, and upgrades. |
| Replication target check | Validating an S3 destination and version fidelity before enabling replication. |
| Replication object size limits | Multipart routing, large-object limits, and retry characteristics. |
| Replication outbound transport | Integrity headers, generic target behavior, and transport knobs. |
For the erasure-coded cluster lifecycle (planning, parity and EC:0,
expansion, rebalance, decommission, heal, drive replacement, restart
recovery, and the rc CLI mapping), start with
Cluster and erasure-coding lifecycle operations.
For concurrent bucket creation and deletion, see Bucket operation admission.
For persisted administrator bucket tasks and bucket recreation, see Bucket heal recovery.
For interrupted object deletions and returning drives, see Delete transaction recovery.
For disk replacement across VM restarts and schema 5/6 maintenance migration, see Replacement generation recovery.
For historical GET timeouts during PUT or Heal, see Object lock contention diagnostics.
Other runbooks remain grouped by filename in operations/;
architecture pages link to the relevant runbook where a cross-boundary
procedure is required.
For storage dashboards, see Storage metrics and observer selection: drive ownership, snapshot freshness, counter queries, and rolling upgrades.
For per-bucket capacity and object counts, see Bucket usage metrics: OTLP export, cluster deduplication, accounting semantics, and scanner freshness.
For optional shard commitments, see Independent shard integrity rollout: activation, legacy repair results, multipart mode changes, and rollback limits.
For crates.io publication of workspace crates, see Workspace Cargo Publish: dependency ordering, dry-run, publish, and failure handling.