feat(scanner): expose prefix-level bucket usage via admin API
The scanner's per-bucket, per-set usage caches already hold a path-keyed
prefix tree, but dui() flattened it only to bucket names — consoles and
operators had no way to ask "what does this prefix hold" without an S3
listing sweep (rustfs/backlog#1872, MinIO loadPrefixUsageFromBackend
parity).
Add:
- data-usage: prefix_usage_in_cache — a shared aggregation over the
entry map (arbitrary prefix, full counters, one-level sub-prefix
breakdown with names recovered from the literal-path cache keys),
hardened like the scanner's checked flatten: cycles, dangling child
links, over-deep trees, and overflowing counters yield None rather
than unbounded recursion or wrapped totals.
- ecstore: ECStore::all_set_disks — iterate every erasure set so a
query can read each set's own cache copy; the hash-routed store path
would always land on one set.
- scanner: bucket_prefix_usage — per-set loads (5s budget each, a slow
set degrades to not-reporting instead of stalling the caller),
merged across sets with partial/compacted/truncated flags, served
from a bounded 30s cache (128 entries, hard-capped) that bucket
writes invalidate through the dirty-usage hook.
- admin: GET /rustfs/admin/v3/usage/{bucket}?prefix=&max-entries=
behind the same any-of gate as datausageinfo (DataUsageInfoAdminAction
OR ListBucketAction), rejecting unknown query parameters and
clamping max-entries to 1..=10000. Route registered in the policy
table (deferred MultipleActions, matching datausageinfo) and the
route matrix test.
Closesrustfs/backlog#1872.
Co-authored-by: heihutu <heihutu@gmail.com>
feat(ecstore): pin bitrot algorithms with a startup self-test
A drifted HighwayHash implementation fails silently: every shard reads
back corrupt, heal rewrites healthy data, and cross-platform clusters
disagree about which copy is good. Mirror MinIO's bitrotSelfTest by
verifying, once at process start:
- known-answer digests for HighwayHash256S / HighwayHash256SLegacy over
a deterministic 4096-byte xorshift64* payload, plus the externally
verifiable FIPS SHA-256 "abc" vector guarding the HashAlgorithm
plumbing itself;
- an end-to-end roundtrip per streaming variant (encode -> size formula
-> bitrot_verify -> BitrotReader read-back), over full blocks and a
partial tail;
- tamper detection: one flipped byte in the final data block and one in
the leading hash must both be rejected as a hash mismatch, not by an
incidental read error.
The check costs microseconds and runs inline in
init_background_service_runtime before any shard can be written or
verified. Outcome surfaces as one structured bitrot_selftest log event,
the rustfs_bitrot_selftest_status gauge (1=passed / 0=failed / 2=skipped),
a bitrotSelftest field on the admin server-info response, and
RUSTFS_BITROT_SELFTEST_STRICT=on turns a failure into a startup error
(MinIO Fatal parity; the default only degrades the status so a bad build
cannot brick an existing fleet on upgrade).
Closesrustfs/backlog#1873 (HS-11).
Co-authored-by: heihutu <heihutu@gmail.com>
* test(replication): accept madmin nanosecond healthCheckDuration payloads
Red-phase TDD tests for P0-7: mc 'replicate add' sends the madmin default
healthCheckDuration=60s as a Go time.Duration nanosecond integer
(60000000000), which RustFS currently rejects as an unsupported field and
would misread as seconds. Also pins the defensive seconds-or-nanos read
for persisted bucket-targets metadata and the capability contract listing
healthCheckDuration as writable.
Currently failing (red):
- remote_target_request_accepts_go_duration_wire_values
- remote_target_request_accepts_legacy_seconds_health_check
- remote_target_health_check_duration_is_declared_writable
- bucket_target_reads_go_nanosecond_durations_defensively
- runtime_capabilities_response_reports_missing_topology_before_storage_init
* fix(replication): accept remote target healthCheckDuration nanoseconds
mc 'replicate add' always sends the madmin default healthcheck-seconds=60
serialized as a Go time.Duration nanosecond integer (60000000000), so the
default mc link-creation path (and 'mc replicate update') failed with
InvalidRequest. Move healthCheckDuration from the unsupported to the
writable remote-target field list; the capability contract in the runtime
capabilities response follows the constants automatically.
Fix the unit mismatch in both directions:
- Request parsing and persisted bucket-targets reads decode the value
defensively: below 10^7 it is legacy RustFS seconds, otherwise Go
time.Duration nanoseconds (also covers MinIO-written metadata).
totalDowntime shares the same wire shape and gets the same handling.
- The list-remote-targets admin response re-encodes only these two fields
as nanoseconds via a dedicated serialization path, leaving the persisted
seconds-based wire format untouched for existing readers.
The per-target health-check interval is accepted for mc compatibility but
not yet applied; the heartbeat keeps its global env-configured interval,
and the explicit 'healthcheck' update op stays rejected. disableProxy,
edge, and edgeSyncBeforeExpiry remain explicitly rejected.
backlog#1052 S2, second slice: admin request dispatch.
Admin operations are registered as &'static dyn Operation, so unlike FS
they cannot carry per-server state. Instead, the (per-server) S3Router now
holds the server's ServerContextSlot and injects it into every request's
extensions at both dispatch points (check_access and call) — the same
extensions channel already used for ReqInfo/RemoteAddr.
- make_admin_route binds the slot; the HTTP builder passes the same slot
the S3 service (FS) uses, so both paths dispatch to the same server.
- admin/runtime_sources gains object_store_from_req / an extensions-based
variant for handlers that have already moved request fields out; both
fall back to the ambient process context when no slot was injected
(direct handler tests, non-router paths) — single-instance unchanged.
- 27 handler resolution sites across 12 files now resolve per-request;
helper functions without request access (11 sites: site_replication
state helpers, quota/table_catalog internals, object_zip_download
workers) stay on the ambient path until app subsystems become
per-server (backlog#1052 S3).
- One site (site-replication resync) resolves before the request body is
consumed, keeping the original error precedence.
New test: the router injects its slot into request extensions by pointer
identity. Embedded e2e (basic S3 + deferred-IAM recovery) still green.
feat(admin): add MinIO-compatible diagnostics and service-control endpoints
Register and implement MinIO admin API compatibility routes:
- POST /v3/service: restart/stop trigger the existing graceful-shutdown
path; freeze/unfreeze record advisory state (request admission is not
yet gated, surfaced honestly via effective=false)
- profiling/trace and healthinfo/obdinfo/log diagnostics endpoints
- GET /v3/top/locks and POST /v3/force-unlock, backed by new
FastObjectLockManager::list_locks()/force_unlock() introspection
- speedtest routes where feasible
Refs rustfs/backlog#604 #606#607#615
Zhengchao An