* fix(object-data-cache): bound cache config and make memory container-aware
Harden the object data cache configuration path so a single bad input
degrades to the disabled adapter instead of OOM-killing pods, panicking
at boot, or silently mis-sizing the cache (backlog#1110/1113/1114/1115/
1127/1140/1130).
- ODC-05: resolve capacity and the fill gate from the effective
(container-aware) memory. Prefer sysinfo cgroup_limits() and use
min(host, cgroup) as the total plus cgroup-derived availability, falling
back to host values when no cgroup limit constrains. Log the resolved
capacity and its basis once at startup.
- ODC-08: cap ttl/time_to_idle at 30 days in validate() so an unbounded
Duration can no longer trip moka's ~1000-year builder assertion.
- ODC-09: drop the max_entry_bytes floor from the derived-capacity clamp so
MAX_ENTRY_BYTES can no longer inflate total capacity above the safety
clamp; reject an entry larger than the resolved capacity instead.
- ODC-10: require an explicit max_bytes to clear max_entry_bytes plus the
weigher overhead so a fillable-but-unretainable cache is rejected.
- ODC-22: reject max_entry_bytes at/above the u32 weigher boundary.
- ODC-35: warn (not reject) when time_to_idle exceeds ttl and document the
min(ttl, time_to_idle) expiry interaction.
- ODC-25: give numeric env overrides two-valued semantics via a new
rustfs_utils::get_env_parse_outcome; a malformed value now disables the
whole cache with one aggregated warning instead of silently keeping
defaults.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(object-data-cache): require identity_keys_max of at least 2
The identity index admits a new key by evicting the oldest one, so a
budget of 1 evicts the previous key on every fill and can never hold two
live keys of one object at once — a versioned bucket alternating two
versions then hits a permanent 0% hit rate.
Reject the degenerate value at config validation time, where the adapter
turns it into a disabled cache with a warning, since the bounded-eviction
policy cannot rescue it at runtime.
Handed off from the identity-index batch (backlog#1128), which changed the
overflow policy but could not touch validate().
Refs: backlog#1115, backlog#1128
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(object-data-cache): let a zero free-memory floor opt out of the gate
Making the memory gate container-aware turned it live in CI: the runners
are Kubernetes pods, so the gate now reads the pod's cgroup free memory,
finds it below the 20% floor, and refuses the fill. Tests that assert a
fill succeeds then failed on CI while passing on a developer host, which
has no cgroup and falls back to host memory.
The gate is behaving correctly — the tests were the ones depending on a
live memory reading. Treat min_free_memory_percent = 0 as a deliberate
opt-out rather than an invalid value: allows_fill returns early before it
touches any snapshot, so admission becomes independent of where the suite
runs. Operators gain the same escape hatch.
Every test that requires a fill to succeed now sets the floor to 0. The
tests that exercise the gate keep it enabled via memory_gated_config, so
the ODC-05 coverage they provide is preserved rather than short-circuited.
Refs: backlog#1110
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(object-data-cache): opt the usecase fill tests out of the memory gate
The previous commit exempted the fill-dependent tests it could find, but
missed the six adapters built inside object_usecase.rs. Fixing the first
batch moved nextest's fail-fast point forward and CI surfaced them:
build_get_object_body_with_cache_materializes_once_and_hits_later and
..._uses_cached_body_without_reader_preread both assert a fill lands, and
both ran with the default 20% free-memory floor.
Set the floor to 0 on all six, matching the sibling test modules. Verified
by pinning the gate's snapshot to 0% available — harsher than any CI pod —
and confirming these tests still pass, which shows the exemption path never
reads memory at all.
An exhaustive sweep over every fill-enabled ObjectDataCacheConfig in the
tree now shows no remaining site: the only unexempted ones are
fill_enabled()'s matches! arm and two adapter tests that assert the adapter
is disabled and therefore never fill.
Refs: backlog#1110
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Multi-node startups (Kubernetes StatefulSets, bare-metal, VMs) can hit
transient DNS/local-host resolution failures while peers and headless
service records are still coming up. Previously endpoint resolution
retried DNS for a fixed 90s window and then returned an error, which
propagated out of run() and exited the process; in Kubernetes this drove
repeated kubelet restarts even though the cluster eventually converged.
Introduce a startup topology convergence policy with three modes:
- orchestrated: wait effectively indefinitely for recoverable
DNS/topology errors so the process stays Running (readiness stays
false) instead of exiting; defer the DNS-IP cross-port validation
because headless-service records can still be flapping.
- bounded: wait a finite window (default 3m) then fail with an
action-oriented error listing host/stage/elapsed and remediation hints.
- fail-fast: fail on the first non-transient resolution error.
Mode is auto-detected (Kubernetes -> orchestrated, distributed URL
endpoints -> bounded, local path endpoints -> fail-fast) and can be
overridden via RUSTFS_STARTUP_TOPOLOGY_WAIT_MODE,
RUSTFS_STARTUP_TOPOLOGY_WAIT_TIMEOUT and
RUSTFS_STARTUP_TOPOLOGY_RETRY_MAX_DELAY.
Also normalize divergent local/remote verdicts for endpoints that share
a host:port, throttle retry warnings, and keep the DNS-independent local
same-path checks in every mode. Configuration error handling is
unchanged: malformed volumes, mixed styles/schemes, duplicate or root
paths, and non-transient errors still fail fast.
Read the topology env vars through rustfs_utils::get_env_opt_str for
consistent alias handling, and simplify the log_once poisoned-lock branch
to unwrap_or_else(PoisonError::into_inner).
Co-authored-by: heihutu <heihutu@gmail.com>
* improve code for metrics
* improve code for metrics
* fix
* fix
* Refactor telemetry initialization and environment functions ordering
- Reorder functions in envs.rs by type size (8-bit to 64-bit, signed before unsigned) and add missing variants like get_env_opt_u16.
- Optimize init_telemetry to support three modes: stdout logging (default error level with span tracing), file rolling logs (size-based with retention), and HTTP-based observability with sub-endpoints (trace, metric, log) falling back to unified endpoint.
- Fix stdout logging issue by retaining WorkerGuard in OtelGuard to prevent premature release of async writer threads.
- Enhance observability mode with HTTP protocol, compression, and proper resource management.
- Update OtelGuard to include tracing_guard for stdout and flexi_logger_handles for file logging.
- Improve error handling and configuration extraction in OtelConfig.
* fix
* up
* fix
* fix
* improve code for obs
* fix
* fix
Zhengchao An