6780140318
fix(object-data-cache): make the GET key write-unique and dedup the lookup ( #4693 )
...
fix(object-data-cache): make GET body cache key write-unique and dedup lookups
Address four object-data-cache GET-path findings (backlog#1107 batch):
ODC-06 (backlog#1111): the cache key was content-unique, not write-unique.
Extend ObjectDataCacheKey with the resolved version's modification time
(i128 unix nanoseconds, None -> 0), derived once in the shared planner so the
ecstore hook and the usecase layer produce an identical key. An unversioned
overwrite advances mod_time, so a stale node can no longer serve old bytes for
up to the TTL under an MD5 collision; etag + size stay as belt-and-braces.
ODC-16 (backlog#1121): every cacheable GET planned and looked up twice (once in
the ecstore hook, once in the usecase layer), double-counting hits, hit_bytes
and lookups. GetObjectReader now carries a GetObjectBodySource marker
(Unprobed / HookMissed / HookServed); the hook stamps it, and
build_get_object_body_with_cache serves a hook-served body directly and skips
its lookup whenever the hook already probed. One hook-served GET now records
exactly one lookup.
ODC-19 (backlog#1124): ENABLE=true with no explicit mode defaulted to HitOnly,
which never fills and keeps a permanent 0% hit rate. Default to
FillBufferedOnly, log the resolved mode at startup, and warn when HitOnly is
selected explicitly.
ODC-24 (backlog#1129): max_entry_bytes above the in-memory GET fill limits was
silently inert. Clamp the planner's size eligibility to
min(max_entry_bytes, seek-support threshold, 64 MiB buffer cap) so ineligible
sizes plan SkipTooLarge instead of being reported eligible, and warn at startup
when the excess is inert.
Co-authored-by: heihutu <heihutu@gmail.com >
2026-07-10 18:22:20 +00:00
6ab8636734
fix(obs): count scanned versions independent of ILM ( #4516 )
...
versions_scanned for both the scanner and ILM collectors was read from
the Lifecycle work source's `checked` counter, which is never recorded on
the production scan path — so rustfs_scanner_versions_scanned_total and
rustfs_ilm_versions_scanned_total sat at zero even while objects_scanned
climbed. The two metrics also have distinct intended meanings that were
conflated: "versions scanned" (all versions, any bucket) vs "versions
checked for ILM actions" (lifecycle-configured buckets only).
Add a lifetime `versions_scanned` counter recorded for every version the
scanner walks (independent of ILM), and record the Lifecycle source's
`checked` counter from the ILM evaluator so the ILM metric reflects the
real checked subset. The scanner collector now reports total scanned
versions; the ILM collector keeps the ILM-checked subset.
Closes backlog#995 (OBS-09).
Co-authored-by: heihutu <heihutu@gmail.com >
2026-07-08 16:50:51 +00:00
506cd156bb
fix(scanner): scope long walk timeouts ( #4376 )
...
* fix(scanner): scope long walk timeouts
* fix(scanner): bound IAM config walks
---------
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com >
2026-07-08 17:06:51 +08:00
e0972f19ac
fix(scanner): cache object lock config ( #4422 )
...
* fix(scanner): cache object lock config
* test(scanner): update lifecycle scanner items
2026-07-08 15:01:57 +08:00
d0792b87be
refactor(lifecycle): extract core rule contracts ( #4258 )
2026-07-04 20:05:56 +08:00
4883d3eb50
refactor(replication): route runtime facades through ecstore ( #4248 )
2026-07-03 23:49:24 +08:00
769549dd2c
refactor(replication): isolate status contract boundaries ( #4236 )
2026-07-03 19:40:49 +08:00
83c32a7a40
fix(lifecycle): avoid blocking expiry enqueue ( #4197 )
2026-07-03 00:45:34 +08:00
f8f373a236
fix: preserve compacted bucket counts ( #4183 )
...
Co-authored-by: overtrue <anzhengchao@gmail.com >
2026-07-02 22:34:47 +08:00
54496f2796
fix(scanner): advance cycle counter when a cycle ends on budget ( #4163 )
2026-07-02 17:15:51 +08:00
902cf1d3dc
fix: handle scanner task failures ( #4185 )
2026-07-02 16:34:56 +08:00
b21b27dddd
fix: preserve scanner usage backups ( #4179 )
2026-07-02 16:33:59 +08:00
d6a9826949
fix: preserve scanner usage state ( #4177 )
2026-07-02 16:32:51 +08:00
18d89d1e31
refactor(replication): route metadata contracts through crate ( #4160 )
2026-07-02 08:48:42 +08:00
14016dbe8c
refactor(scanner): hide replication queue DTOs ( #4150 )
2026-07-02 00:17:03 +08:00
fc806679fb
refactor(ecstore): add replication work bridges ( #4137 )
2026-07-01 18:37:01 +08:00
b16120dbcc
feat: optimize small GET read paths ( #4022 )
2026-07-01 15:40:00 +08:00
fab78f8402
refactor(scanner): use expiry runtime boundary ( #4079 )
2026-06-30 02:09:52 +08:00
c200dc10d2
fix(scanner): preserve partial scan progress ( #3996 )
2026-06-29 10:43:25 +08:00
0ff261864a
fix(scanner): harden data scanner integrity handling ( #4012 )
...
* fix(scanner): heal metadata scan failures
* fix(scanner): preserve dirty buckets after scan failures
* feat(scanner): record leader lock liveness
* fix(scanner): preserve failed deep scan state
* fix(scanner): reject untimestamped stale usage
* fix(scanner): report topology-derived admission limit
* fix(scanner): accumulate tier usage stats
* fix(scanner): guard data usage cache recursion
* feat(scanner): expose startup enabled status
* fix(scanner): continue after heal admission rejection
* fix(scanner): avoid cyclic root usage double count
* fix(scanner): preserve dirty markers on cache save failure
* feat(scanner): expose leader liveness status
* fix(scanner): avoid heal escalation for transient metadata reads
* feat(scanner): persist rejected heal retry candidates
* test(scanner): satisfy freshness status clippy
* test(app): avoid global store reinit in context test
* test(app): gate global store helper to tests
2026-06-29 08:19:58 +08:00
82bbef0b60
test: cover runtime and repair preservation ( #3964 )
2026-06-27 23:34:59 +08:00
1b3dea012e
refactor: route ecstore runtime globals through facade ( #3941 )
2026-06-27 12:27:03 +08:00
675597ec16
fix(ecstore): handle stalled recovery reads and listings ( #3790 )
...
* fix(ecstore): handle stalled recovery reads and listings
* fix(rio): start HTTP stall timeout on read
* fix(ecstore): handle stalled reads and partial lists
* fix(ecstore): retire stalled shards and list errors
* fix(ecstore): preserve list merge lookahead entries
* fix(ecstore): bound zero-copy shard reads
* fix(ecstore): hedge stalled shard reads
* fix(ecstore): retire abandoned shard reads
* fix(ecstore): include part identity in metadata quorum
* fix(ecstore): validate heal shard sources
* fix(ecstore): verify reconstructed read shards
* chore(ecstore): log slow object read stages
* fix(heal): throttle auto heal during recovery
* fix(scanner): yield to foreground reads
* fix(scanner): track streaming object reads
* fix(ecstore): avoid false read heal fanout
* fix(ecstore): verify codec streaming reconstruction sources
* fix(ecstore): preserve quorum progress on slow shards
* fix(storage): restore read timeout facade
* fix(ecstore): retain fallback readers after quorum
* chore: allow decode helper argument lists
---------
Co-authored-by: overtrue <anzhengchao@gmail.com >
2026-06-27 10:21:09 +08:00
a8327f8901
fix(scanner): account versioned delete markers in usage ( #3904 )
...
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com >
2026-06-26 18:34:21 +08:00
72ae43cb90
refactor: segment external storage contract imports ( #3908 )
2026-06-26 16:44:59 +08:00
1f7e159388
refactor: segment external storage api boundaries ( #3903 )
2026-06-26 15:10:49 +08:00
0b9c8a7731
fix(heal): align scanner repair safety with quorum errors ( #3887 )
...
* fix(heal): avoid task-level delete on repair failure
* fix(scanner): avoid recreate for scanner heal checks
* fix(heal): preserve typed quorum failures
* test(heal): satisfy clippy bool assertion
2026-06-26 08:58:27 +08:00
4d6ea453a7
refactor: route scanner heal storage boundaries ( #3888 )
2026-06-26 05:24:41 +08:00
86e9362459
fix(heal): canonicalize scanner object-dir repairs ( #3864 )
2026-06-25 22:07:38 +08:00
30f781871d
fix(scanner): preserve dirty usage convergence ( #3844 )
...
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com >
2026-06-25 21:09:12 +08:00
4cb28abfc9
fix(scanner): reduce non-actionable scan noise ( #3840 )
2026-06-25 14:35:10 +08:00
a935854b32
refactor: centralize scanner runtime source helpers ( #3827 )
...
* refactor: batch lifecycle runtime source handles
* refactor: centralize scanner runtime source helpers
2026-06-24 22:05:03 +08:00
eef8f43251
fix: harden lock timeouts, health, and heal paths ( #3815 )
2026-06-24 15:48:06 +08:00
55b402ec2e
fix(scanner): avoid retrying usage-cache lock failures ( #3809 )
...
Avoid retrying scanner cache lock failures
2026-06-24 10:22:43 +08:00
7bdb25ae9d
feat(scanner): define replication boundary contract ( #3630 )
...
* feat(scanner): define replication boundary contract
* fix(scanner): propagate replication boundary metrics
* fix(scanner): preserve repair metadata on merge
---------
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com >
Co-authored-by: houseme <housemecn@gmail.com >
2026-06-23 21:35:36 +08:00
5c60f0cae9
refactor: centralize owner server config reads ( #3793 )
2026-06-23 21:27:30 +08:00
30559f7e1b
refactor: collapse test fuzz ecstore thin bridges ( #3765 )
2026-06-23 07:04:51 +08:00
198fd4f150
refactor(runtime): route RustFS runtime consumers through storage owner ( #3756 )
2026-06-23 05:17:56 +08:00
ec5f112205
refactor: expose scanner ECStore owner symbols ( #3749 )
2026-06-22 22:26:25 +08:00
d3796d6c10
refactor: remove external owner compat bridges ( #3741 )
2026-06-22 18:28:35 +08:00
cca9e83a8b
refactor: use relative standalone compat consumers ( #3734 )
2026-06-22 15:07:58 +08:00
de158743c3
refactor: split compat facade imports ( #3716 )
2026-06-22 09:53:22 +08:00
c13f42e9a0
refactor: wrap disk rpc compat trait methods ( #3707 )
2026-06-22 06:31:24 +08:00
7eb9a4759e
refactor: wrap bucket compat trait methods ( #3706 )
2026-06-22 04:32:37 +08:00
d39815470e
refactor: prune consumer storage compat paths ( #3704 )
2026-06-22 02:31:30 +08:00
6b3d96fde3
refactor: prune trait import compat re-exports ( #3699 )
2026-06-21 23:17:58 +08:00
30b46640b4
refactor: prune edge compat aliases ( #3692 )
2026-06-21 17:41:07 +08:00
0985225448
refactor: consolidate ecstore public facade ( #3678 )
...
* refactor: expand ecstore compatibility facade
* test: enforce ecstore api facade imports
* refactor: hide legacy ecstore layout modules
* refactor: hide ecstore facade root modules
2026-06-21 09:09:11 +08:00
c47dea0009
fix(scanner): preserve complete usage snapshots ( #3654 )
2026-06-21 08:21:20 +08:00
9b3fda9e30
refactor: route ecstore public surfaces through api ( #3673 )
2026-06-21 05:23:23 +08:00
houseme