d5df66ac4f
fix(scanner): require authoritative usage snapshots ( #5333 )
2026-07-28 07:29:57 +08:00
a63b79004c
fix(scanner): make distributed usage convergence authoritative ( #5151 )
...
* fix(scanner): make distributed usage cycles authoritative
* fix(scanner): close distributed refresh races
* fix(config): align scanner reload integration
* fix(admin): scope config test helpers
* fix(scanner): harden distributed usage convergence
* fix(scanner): preserve rolling activity compatibility
* fix(admin): expose non-secret optional config values
* fix(scanner): acknowledge distributed dirty usage
* fix(ecstore): make bucket mutations cancellation safe
* fix(scanner): preserve pending dirty acknowledgements
* test(obs): account for superseded scanner metric
* fix(api): reject excess detached bucket mutations
* test: close scanner convergence coverage gaps
* fix(scanner): make path tracking cleanup one-shot
---------
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com >
Co-authored-by: houseme <housemecn@gmail.com >
2026-07-25 18:45:16 +08:00
889a45ad4d
fix(scanner): back off clean idle scans across erasure clusters ( #4984 )
...
* fix(scanner): back off clean single-disk cycles
* fix(scanner): extend idle backoff across erasure clusters
---------
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com >
2026-07-18 10:49:45 +08:00
627c396649
fix(scanner): allow usage snapshot save when existing timestamp is future-dated beyond clock tolerance ( #4982 )
2026-07-17 16:04:43 +00:00
6780140318
fix(object-data-cache): make the GET key write-unique and dedup the lookup ( #4693 )
...
fix(object-data-cache): make GET body cache key write-unique and dedup lookups
Address four object-data-cache GET-path findings (backlog#1107 batch):
ODC-06 (backlog#1111): the cache key was content-unique, not write-unique.
Extend ObjectDataCacheKey with the resolved version's modification time
(i128 unix nanoseconds, None -> 0), derived once in the shared planner so the
ecstore hook and the usecase layer produce an identical key. An unversioned
overwrite advances mod_time, so a stale node can no longer serve old bytes for
up to the TTL under an MD5 collision; etag + size stay as belt-and-braces.
ODC-16 (backlog#1121): every cacheable GET planned and looked up twice (once in
the ecstore hook, once in the usecase layer), double-counting hits, hit_bytes
and lookups. GetObjectReader now carries a GetObjectBodySource marker
(Unprobed / HookMissed / HookServed); the hook stamps it, and
build_get_object_body_with_cache serves a hook-served body directly and skips
its lookup whenever the hook already probed. One hook-served GET now records
exactly one lookup.
ODC-19 (backlog#1124): ENABLE=true with no explicit mode defaulted to HitOnly,
which never fills and keeps a permanent 0% hit rate. Default to
FillBufferedOnly, log the resolved mode at startup, and warn when HitOnly is
selected explicitly.
ODC-24 (backlog#1129): max_entry_bytes above the in-memory GET fill limits was
silently inert. Clamp the planner's size eligibility to
min(max_entry_bytes, seek-support threshold, 64 MiB buffer cap) so ineligible
sizes plan SkipTooLarge instead of being reported eligible, and warn at startup
when the excess is inert.
Co-authored-by: heihutu <heihutu@gmail.com >
2026-07-10 18:22:20 +00:00
54496f2796
fix(scanner): advance cycle counter when a cycle ends on budget ( #4163 )
2026-07-02 17:15:51 +08:00
b21b27dddd
fix: preserve scanner usage backups ( #4179 )
2026-07-02 16:33:59 +08:00
b16120dbcc
feat: optimize small GET read paths ( #4022 )
2026-07-01 15:40:00 +08:00
0ff261864a
fix(scanner): harden data scanner integrity handling ( #4012 )
...
* fix(scanner): heal metadata scan failures
* fix(scanner): preserve dirty buckets after scan failures
* feat(scanner): record leader lock liveness
* fix(scanner): preserve failed deep scan state
* fix(scanner): reject untimestamped stale usage
* fix(scanner): report topology-derived admission limit
* fix(scanner): accumulate tier usage stats
* fix(scanner): guard data usage cache recursion
* feat(scanner): expose startup enabled status
* fix(scanner): continue after heal admission rejection
* fix(scanner): avoid cyclic root usage double count
* fix(scanner): preserve dirty markers on cache save failure
* feat(scanner): expose leader liveness status
* fix(scanner): avoid heal escalation for transient metadata reads
* feat(scanner): persist rejected heal retry candidates
* test(scanner): satisfy freshness status clippy
* test(app): avoid global store reinit in context test
* test(app): gate global store helper to tests
2026-06-29 08:19:58 +08:00
1f7e159388
refactor: segment external storage api boundaries ( #3903 )
2026-06-26 15:10:49 +08:00
4d6ea453a7
refactor: route scanner heal storage boundaries ( #3888 )
2026-06-26 05:24:41 +08:00
30f781871d
fix(scanner): preserve dirty usage convergence ( #3844 )
...
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com >
2026-06-25 21:09:12 +08:00
a935854b32
refactor: centralize scanner runtime source helpers ( #3827 )
...
* refactor: batch lifecycle runtime source handles
* refactor: centralize scanner runtime source helpers
2026-06-24 22:05:03 +08:00
5c60f0cae9
refactor: centralize owner server config reads ( #3793 )
2026-06-23 21:27:30 +08:00
d3796d6c10
refactor: remove external owner compat bridges ( #3741 )
2026-06-22 18:28:35 +08:00
cca9e83a8b
refactor: use relative standalone compat consumers ( #3734 )
2026-06-22 15:07:58 +08:00
7eb9a4759e
refactor: wrap bucket compat trait methods ( #3706 )
2026-06-22 04:32:37 +08:00
6b3d96fde3
refactor: prune trait import compat re-exports ( #3699 )
2026-06-21 23:17:58 +08:00
a689ff3c39
feat(scanner): wake cycles for dirty usage refresh ( #3617 )
2026-06-19 18:36:48 +08:00
80144ec886
feat(scanner): expose usage freshness status ( #3577 )
...
* feat(scanner): expose usage freshness status
* fix(scanner): reset dirty usage cycle state
---------
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com >
2026-06-18 21:14:55 +08:00
bdb2461b55
refactor: clean scanner heal runtime boundaries ( #3571 )
2026-06-18 16:12:27 +08:00
acdf439371
refactor: clean external DTO consumers ( #3566 )
...
* refactor: clean external DTO consumers
* fix: handle empty erasure shard recovery
2026-06-18 14:22:44 +08:00
99941f7e7c
refactor: clean external operation consumers ( #3565 )
2026-06-18 12:39:41 +08:00
57403525ee
refactor: move heal and namespace contracts ( #3560 )
2026-06-18 11:00:21 +08:00
e5cad7ed20
refactor: move object operation contracts ( #3559 )
2026-06-18 09:48:13 +08:00
53337a6f71
refactor: move HTTP range helper contracts ( #3533 )
2026-06-18 00:47:23 +08:00
a9aba323c6
fix(scanner): restore prompt bucket usage scans ( #3516 )
...
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com >
Co-authored-by: houseme <housemecn@gmail.com >
2026-06-17 21:49:56 +08:00
ed55857bca
refactor: move bucket operations contract ( #3507 )
2026-06-17 09:10:38 +08:00
c2e0792f6f
fix(scanner): skip startup delay when replication is active so failed… ( #3455 )
...
fix(scanner): skip startup delay when replication is active so failed objects heal promptly after restart
Co-authored-by: houseme <housemecn@gmail.com >
2026-06-15 09:02:21 +08:00
e8012bd1ba
refactor(logging): normalize admin telemetry and error messages ( #3430 )
2026-06-14 13:27:10 +08:00
7da10db852
refactor(logging): standardize heal and scanner events ( #3414 )
...
* refactor(logging): standardize heal and scanner events
* chore(git): untrack local logging governance note
* chore(git): ignore local logging governance note
2026-06-14 01:47:39 +08:00
19ff378e9b
fix(scanner): preserve newer usage snapshots ( #3370 )
...
* fix(scanner): preserve newer usage snapshots
* fix(table-catalog): require namespace locking backend
---------
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com >
2026-06-12 07:06:08 +08:00
a85cc0354c
refactor(storage-api): remove namespace lock from StorageAPI ( #3365 )
...
* refactor(storage-api): remove namespace lock from StorageAPI
* test(scanner): wait for runtime budget cancellation
---------
Co-authored-by: loverustfs <hello@rustfs.com >
2026-06-11 22:42:12 +08:00
7146c893cb
refactor(storage-api): remove legacy bucket DTO re-export ( #3364 )
2026-06-11 21:24:56 +08:00
b4524033e3
refactor(config): move global config accessors ( #3360 )
2026-06-11 19:16:02 +08:00
f40dd2f93c
fix(scanner): publish partial usage for compacted scans ( #3277 )
...
* fix(scanner): publish partial usage for compacted scans
* fix(scanner): publish first partial usage immediately
* fix(scanner): skip startup delay for cold usage cache
* fix(scanner): tighten cold usage publish gate
---------
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com >
Co-authored-by: houseme <housemecn@gmail.com >
2026-06-08 06:18:35 +00:00
4ae070c8cc
feat(scanner): expose scan partial source status ( #3247 )
...
* feat(scanner): expose scan partial source status
* fix(scanner): expose partial source in aggregated metrics
---------
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com >
Co-authored-by: cxymds <Cxymds@qq.com >
Co-authored-by: houseme <housemecn@gmail.com >
2026-06-07 06:40:04 +00:00
6850457707
feat(scanner): add runtime scanner controls and status ( #3203 )
...
* feat(scanner): add runtime scanner controls and status
* fix(scanner): validate persisted scanner config
* docs(scanner): clarify start delay behavior
---------
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com >
Co-authored-by: houseme <housemecn@gmail.com >
2026-06-04 05:16:26 +00:00
ad1a489f75
feat(scanner): add scanner budget progress controls ( #3185 )
...
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com >
2026-06-03 14:37:58 +00:00
cc07946782
feat(scanner): add cycle budget observability ( #3166 )
...
* feat(scanner): add cycle budget observability
* fix(scanner): clear cycle state after budget stop
* test(scanner): stabilize timeout-based scanner tests
* fix(scanner): keep cycle ILM counts scanner-only
* test(scanner): avoid test-only pending import
---------
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com >
Co-authored-by: houseme <housemecn@gmail.com >
Co-authored-by: 安正超 <anzhengchao@gmail.com >
2026-06-03 03:32:41 +00:00
1d46047d6f
feat(scanner): expand scanner observability metrics ( #3159 )
...
* feat(scanner): expand scanner observability metrics
* chore(scanner): align bucket-drive metric wording
---------
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com >
2026-06-02 01:29:53 +00:00
f3bd838925
feat(scanner): expose cycle progress metrics ( #3152 )
...
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com >
Co-authored-by: houseme <housemecn@gmail.com >
2026-06-01 07:54:39 +00:00
76da2a48d0
feat(scanner): expose cycle observability controls ( #3147 )
...
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com >
2026-05-31 21:55:46 +00:00
a99ef64db2
feat(scanner): add scanner budgets and progress metrics ( #3145 )
...
* fix(scanner): preserve maintenance scan cadence
* feat(scanner): add scanner concurrency budget
* feat(scanner): expose scanner runtime progress
* fix(scanner): address scanner review feedback
---------
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com >
Co-authored-by: houseme <housemecn@gmail.com >
2026-05-31 16:42:38 +00:00
92104cb354
fix(scanner): reduce single-disk scanner churn ( #3135 )
...
* fix(scanner): slow single-disk default scans
* fix(scanner): reduce single-disk scanner churn
---------
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com >
Co-authored-by: houseme <housemecn@gmail.com >
2026-05-31 05:05:04 +00:00
d921af6ef7
fix(heal): normalize completed root heal state ( #3140 )
...
Clear persisted background heal active state after a deep scanner cycle completes so root heal status does not remain in progress between cycles.
Treat path-based stop for an already missing heal task as stopped while keeping client-token cancellation strict.
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com >
2026-05-31 04:47:36 +00:00
62d1f80dbf
fix(data-usage): refresh admin usage after object changes ( #3081 )
...
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com >
2026-05-25 14:24:00 +00:00
69345fe059
fix(scanner): preserve background heal compatibility ( #3041 )
2026-05-22 14:38:35 +08:00
50d03ef021
perf(memory): add reclaim signals and cache controls ( #2689 )
2026-04-26 16:42:35 +00:00
6ce24f3b63
feat(lifecycle): improve ILM compatibility and scanner runtime config ( #2534 )
...
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com >
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com >
Co-authored-by: cxymds <Cxymds@qq.com >
2026-04-16 10:28:03 +00:00
Henry Guo