* perf(protos): balance peer channels and synchronize TLS eviction Use bounded per-peer bulk cursors and shared TLS generation reads. Keep eviction and generation removal coherent with concurrent publication, and expose opt-in bulk transport tuning with checked capacities. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * feat(metrics): observe internode unary request lifetimes Record client attempts through errors and cancellation, preserve the published server-duration family, and add gated activity and completion metrics. Cache enabled instruments after recorder publication while retaining startup duration recording. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(perf): make internode gRPC experiments reproducible Reset phase-owned settings before applying overrides, separate lifecycle experiments, and provide a bounded loopback workload with independently reported control and bulk latency. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(perf): isolate transport knobs and redact dry runs Add single-parameter P0 presets with separate artifacts while preserving the legacy combined default. Mask credential arguments in dry-run output without changing actual dispatch, and document fleet stop barriers and measurement boundaries. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * feat(observability): qualify gRPC windows and trace shutdown Reject stale, incomplete or maintenance-active measurement evidence using producer timestamps and independent process identities. Preserve raw inputs, bind evidence hashes, and exercise failure paths through the script entry point. Record HTTP/2 graceful, child-join and TLS reloader retirement phases without changing shared deadlines, join ownership or unclean-shutdown marker policy. Document RPC coverage, fixed-window behavior and separate maintenance workloads. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(perf): add authenticated ReadAll and fixed-rate Ping probe Exercise production remote-disk and signed control clients against task-owned temporary fixtures. Preserve all offered control arrivals, dispatch delay, payload validation and bounded raw request samples, with optional external measurement markers. Register probe regressions with the default package test discovery and document its telemetry, retry and storage-acceptance boundaries. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(perf): resolve RPC probe disks by explicit UUID Allow a complete non-nil disk UUID mapping to be installed through the existing remote-disk client API before fixture preflight. This supports test services that register local paths instead of the probe's transport URLs, without modifying server metadata. Validate mapping shape and assert the actual wire target in the loopback regression. Keep preflight diagnostics limited to the numeric disk-error classification. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * docs(perf): reseed temporary RPC fixtures after restart Separate startup-cleaned RPC temporary files from confirmed S3 durability fixtures. Revalidate readiness and disk identity before reseeding the probe after recovery. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(server): cancel unfinished TLS handshakes during shutdown Cancel pre-request TLS acceptance when the listener retires so silent peers cannot retain graceful watchers beyond the shared drain budget. Keep established TLS/H2 response draining and original child joins intact. Add real TCP/TLS shutdown regressions and document fresh native OTLP observation fences and exec-confirmed diagnostic test units. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * perf(grpc): reuse bulk keys and fix IPv6 TLS verification Reuse bounded per-peer slot keys and retain the existing cache identities for larger pools. Delegate custom-CA certificate names to tonic's URI parser and cover real IPv6 control/bulk TLS connections with a wrong-name rejection regression. Classify ReadVersion failure boundaries while preserving error totals, raise the bounded raw probe budget, and align the Linux-only rename test helper with its caller's platform condition. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(grpc): keep eviction race behind cache helpers Pause eviction on the owned TLS-generation lock and inspect the channel through the common cache API. Assert lock ordering while both mutations are queued, then retain the replacement-generation assertion after they complete. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(grpc): load native trust and preserve mTLS client identity Select explicit private trust or platform roots, then attach the published client identity for either mode. Build the TLS connector on the blocking pool under the existing acquisition deadline. Cover real control/bulk TLS connections, required client authentication, private-root scope and queued configuration timeout without late connection establishment. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Documentation
Use the focused indexes rather than treating this directory as an unordered collection:
Operations
-
Candidate release pipeline: frozen inputs, staged acceptance, immutable publication and recovery.
-
Sampled local batch CPU diagnostics — default-off request-correlated allocation and worker wall/CPU diagnosis.
-
Retained MRF responsibilities — unknown bucket identities, bounded probes, and unreadable checkpoint recovery.
-
Read-only BTF declaration inspection — bounded prototypes and scalar member layouts, explicit void handling, and independent native references.
-
Internode HTTP streaming transport — protocol selection, pooled connection isolation, and distributed read validation.
-
Persisted config compatibility — ignored MinIO settings, bounded diagnostics, alias retention, and safe resets.
-
Table maintenance periodic queueing — opt-in scheduling, limits, worker handoff, and recovery.
-
S3 Tables catalog lifecycle — disablement, fenced empty-table cleanup, and registration-cache boundaries.
For PUT cleanup admission, blocking-pool isolation, and optional CPU affinity, see PUT cleanup isolation.
-
External certificate issuance and TLS rotation — publication, native reload, trust separation and failure handling.
-
Architecture guard troubleshooting — diagnose repository guard failures.
-
Logging governance — broad event audits, migrations, and guardrail changes.
-
Multipart upload memory diagnosis — allocator attribution, Docker reproduction and sustained upload checks.
For the logical per-operation io_uring read cap, see io_uring read chunk size.
For bounded local read-backend startup and cancellation, see io_uring backend initialization.
For legacy protection assessment and bounded protected copies, see Object integrity inventory, audit, and migration.
For encrypted listing compatibility and older stored fingerprints, see Encrypted object ETags.
For durable hard quota accounting, node-outage behavior, and fence protocol upgrades, see Durable hard quotas.
Operational runbooks live under operations/. Replication
operators should start with:
| Runbook | Use it for |
|---|---|
| Site replication operations | Health fields, pending operations, outage recovery, re-pair admission, IAM/SSE boundaries, and upgrades. |
| Replication target check | Validating an S3 destination and version fidelity before enabling replication. |
| Replication object size limits | Multipart routing, large-object limits, and retry characteristics. |
| Replication outbound transport | Integrity headers, generic target behavior, and transport knobs. |
For the erasure-coded cluster lifecycle (planning, parity and EC:0,
expansion, rebalance, decommission, heal, drive replacement, restart
recovery, and the rc CLI mapping), start with
Cluster and erasure-coding lifecycle operations.
For concurrent bucket creation and deletion, see Bucket operation admission.
For persisted administrator bucket tasks and bucket recreation, see Bucket heal recovery.
For interrupted object deletions and returning drives, see Delete transaction recovery.
For disk replacement across VM restarts and schema 5/6 maintenance migration, see Replacement generation recovery.
For historical GET timeouts during PUT or Heal, see Object lock contention diagnostics.
Other runbooks remain grouped by filename in operations/;
architecture pages link to the relevant runbook where a cross-boundary
procedure is required.
For storage dashboards, see Storage metrics and observer selection: drive ownership, snapshot freshness, counter queries, and rolling upgrades.
For per-bucket capacity and object counts, see Bucket usage metrics: OTLP export, cluster deduplication, accounting semantics, and scanner freshness.
For optional shard commitments, see Independent shard integrity rollout: activation, legacy repair results, multipart mode changes, and rollback limits.
For crates.io publication of workspace crates, see Workspace Cargo Publish: dependency ordering, dry-run, publish, and failure handling.