e2a921bc16
fix(storage): harden ODM and scanner publication ( #7187 )
...
* fix(storage): harden ODM and scanner publication
* fix(app): simplify absent SSE configuration matching
* test(heal): settle PUT rename tails before disk-wipe fixtures
* fix(ecstore): remove duplicate local rename implementation
Keep the canonical commit module after concurrent storage changes merged.
The control-write and rollback changes are already present there.
Co-Authored-By: heihutu <heihutu@gmail.com >
Co-Authored-By: zhi22915 <qiuzgang@gmail.com >
* fix(ci): satisfy new clippy lints
* style(scanner): order merged test imports
* fix(scanner): invalidate bucket work after namespace completion
* fix(scanner): fence cached snapshots by scan execution
---------
Co-authored-by: houseme <housemecn@gmail.com >
Co-authored-by: heihutu <heihutu@gmail.com >
Co-authored-by: zhi22915 <qiuzgang@gmail.com >
2026-09-05 13:47:12 +00:00
acfeef55ab
feat(scanner): add bounded incarnation-scoped ACK receiver ( #7182 )
...
* chore(deps): refresh SDKs and pin clock skew regression coverage
Refresh compatible dependencies for Scanner/Heal V2 batch 1 and verify
the production S3 retry/signing path with a deterministic clock.
Co-Authored-By: heihutu <heihutu@gmail.com >
Co-Authored-By: zhi22915 <qiuzgang@gmail.com >
* feat(scanner): add bounded incarnation-scoped ACK receiver
Refs rustfs/backlog#2265 and rustfs/backlog#2240 .
Co-Authored-By: heihutu <heihutu@gmail.com >
Co-Authored-By: zhi22915 <qiuzgang@gmail.com >
---------
Co-authored-by: heihutu <heihutu@gmail.com >
Co-authored-by: zhi22915 <qiuzgang@gmail.com >
2026-09-05 08:54:55 +00:00
2d159635ed
feat(scanner): plan dirty bucket cache refreshes ( #7146 )
...
* feat(scanner): plan dirty bucket cache refreshes
* fix(scanner): route peer snapshot through storage boundary
---------
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com >
Co-authored-by: cxymds <cxymds@gmail.com >
Co-authored-by: Zhengchao An <anzhengchao@gmail.com >
2026-09-05 16:32:02 +08:00
f16a30b231
feat(scanner): expose authenticated dirty bucket snapshots ( #7122 )
...
* feat(scanner): add peer bucket dirty snapshots
* fix(scanner): keep dirty snapshot errors stable
* test(protos): satisfy dirty snapshot clippy
* fix(scanner): satisfy dirty snapshot clippy
---------
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com >
2026-09-04 18:23:13 +08:00
80c88a9031
fix(ilm): delete historical null versions by exact identity ( #7109 )
2026-09-04 08:14:47 +08:00
62e66baf89
feat(scanner): add storage seam ( #7105 )
2026-09-03 20:31:59 +08:00
281e40f1cc
fix(scanner): fit usage persistence within publication lease ( #6967 )
...
Lower the default scanner cache save timeout so the derived usage persistence budget stays inside the effective distributed publication lease window.
Add focused regressions for the default publication budget and bootstrap-pending observational baselines, and update operator docs with the new default.
Co-authored-by: heihutu <heihutu@gmail.com >
2026-08-31 22:58:29 +08:00
ee39e4fccb
fix(scanner): own publication mutations through storage drain ( #6867 )
...
* fix(scanner): own publication mutations through storage drain
Co-Authored-By: heihutu <heihutu@gmail.com >
* fix(storage): remove unused rename data shim
Co-Authored-By: heihutu <heihutu@gmail.com >
---------
Co-authored-by: heihutu <heihutu@gmail.com >
2026-08-30 02:39:07 +00:00
ff3ad30f0c
fix(scanner): bound publication proof retries on main ( #6870 )
...
* fix(scanner): retain completed publication candidates
* fix(scanner): export publication activity helper
* test(ecstore): retain activity snapshot across retries
* fix(scanner): rebase publication proof retry onto main
Co-Authored-By: heihutu <heihutu@gmail.com >
* fix(scanner): resolve publication proof retry conflicts
Co-Authored-By: heihutu <heihutu@gmail.com >
---------
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com >
Co-authored-by: heihutu <heihutu@gmail.com >
2026-08-29 22:41:36 +00:00
7d3f5545e7
fix(ci): repair post-merge build gates
2026-08-24 16:10:24 +08:00
eec0e0e056
fix(scanner): fence movement generation publication ( #6461 )
...
* feat(scanner): add movement generation fencing
* fix(scanner): prioritize unverified cycle deferral
* feat(ecstore): add scanner publication lease fence
* feat(rpc): add scanner publication lease protocol
* feat(scanner): hold remote leases through usage publish
* test(scanner): cover publication lease fencing
* fix(scanner): fence remote leases across restart and delay
* feat(rpc): fence scanner publication rename writes
* fix(scanner): fence observed cleanup deletes
* fix(proto): qualify lease release test types
* fix(scanner): pin movement notifications
* fix(scanner): clean publication imports
* fix(ecstore): satisfy scanner fence clippy
* refactor(scanner): group wait and publication options
* fix(scanner): satisfy final lint and facade guards
* fix(rpc): resolve facade export conflicts
* fix(ci): remove unused decommission and healing facades
* fix(ci): cfg-gate test-only usage overlay import
* fix(scanner): wake on remote scanner restart
2026-08-24 14:17:35 +08:00
adb90fc6e1
fix(scanner): defer usage publication during pool recovery ( #6333 )
...
* fix(scanner): defer usage publication during pool recovery
* fix(scanner): preserve metrics when publication is deferred
* fix(scanner): route test types through storage boundary
* fix(scanner): keep cache floor deferred during movement
2026-08-21 17:32:59 +08:00
81332718e6
perf(scanner): cut per-object allocations in the scan hot path ( #6274 )
2026-08-19 23:33:49 +08:00
35a30cd614
feat(scanner): emit excess alerts as S3 notification events (HS-04) ( #6176 )
...
* feat(scanner): emit excess alerts as S3 notification events
The excess-versions / excess-version-size / excess-folders alerts were
metrics-and-logs only; consoles and external auditors had no way to hear
them (rustfs/backlog#1868 , HS-04). MinIO emits s3:ObjectManyVersions /
s3:ObjectLargeVersions / s3:PrefixManyFolders for the same conditions —
RustFS carries those as EventName::Scanner* with s3:Scanner:* wire names
that already existed unpublished.
The three alert sites now also dispatch through the standard event
pipeline (send_event via the storage_api owner facade), carrying the
actual values and thresholds in req_params and UserAgent "Scanner".
Without a cooldown a single over-threshold object would re-emit on every
~60s scan cycle, so emissions are edge-held per (kind, bucket, object)
for 24h (RUSTFS_SCANNER_ALERT_COOLDOWN_SECS, 0 = every cycle), backed by
a process-global map with a 4096-key hard cap that clears rather than
grows. Metrics and structured logs stay level-triggered every cycle;
only the notification events are held back. A restart resets the
cooldown deliberately: one re-emission per still-hot key buys back
visibility after the restarts that accompany incident response.
Tests pin the edge-hold semantics (first fires, immediate re-check held,
independent keys, cooldown expiry re-fires, zero cooldown always emits,
hard bound) in one sequential test for the process-global map, and pin
the emitted wire names against EventName's canonical string forms so a
subscribed bucket notification can never silently stop matching.
docs/operations/scanner-excess-alerts.md documents the three events,
the metric-vs-event cadence difference, and the HS-15 threshold deltas
(alert_excess_folders 65538 vs MinIO 50000 is deliberate: Proxmox
Backup Server chunk layout compatibility).
Closes rustfs/backlog#1868 .
Co-Authored-By: heihutu <heihutu@gmail.com >
* docs(operations): split scanner excess alerts into English and Chinese pages
The page shipped Chinese-only; keep it as scanner-excess-alerts_zh.md and
add a faithful English translation at the original path, cross-linked at
the top of both.
Co-Authored-By: heihutu <heihutu@gmail.com >
---------
Co-authored-by: heihutu <heihutu@gmail.com >
2026-08-18 08:46:32 +08:00
6f29431a65
test(ecstore): isolate rename publication hooks ( #6106 )
2026-08-14 11:07:34 +00:00
4c5e73b2f2
fix(scanner): defer cycles during data movement ( #5970 )
...
* fix(scanner): defer cycles during data movement
* fix(scanner): distinguish deferred scan cycles
---------
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com >
Co-authored-by: houseme <housemecn@gmail.com >
2026-08-12 20:46:06 +08:00
706a8b6061
fix(scanner): publish bounded observational usage ( #5742 )
...
* fix(scanner): publish bounded observational usage
* test(ci): serialize embedded integration ports
* test(cache): isolate generation-change timeout
* fix(scanner): address observational usage review
Co-Authored-By: heihutu <heihutu@gmail.com >
---------
Co-authored-by: Anthony Martin <949506+anthonymartin@users.noreply.github.com >
Co-authored-by: houseme <housemecn@gmail.com >
Co-authored-by: heihutu <heihutu@gmail.com >
2026-08-07 08:52:34 +08:00
d5df66ac4f
fix(scanner): require authoritative usage snapshots ( #5333 )
2026-07-28 07:29:57 +08:00
a63b79004c
fix(scanner): make distributed usage convergence authoritative ( #5151 )
...
* fix(scanner): make distributed usage cycles authoritative
* fix(scanner): close distributed refresh races
* fix(config): align scanner reload integration
* fix(admin): scope config test helpers
* fix(scanner): harden distributed usage convergence
* fix(scanner): preserve rolling activity compatibility
* fix(admin): expose non-secret optional config values
* fix(scanner): acknowledge distributed dirty usage
* fix(ecstore): make bucket mutations cancellation safe
* fix(scanner): preserve pending dirty acknowledgements
* test(obs): account for superseded scanner metric
* fix(api): reject excess detached bucket mutations
* test: close scanner convergence coverage gaps
* fix(scanner): make path tracking cleanup one-shot
---------
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com >
Co-authored-by: houseme <housemecn@gmail.com >
2026-07-25 18:45:16 +08:00
d0792b87be
refactor(lifecycle): extract core rule contracts ( #4258 )
2026-07-04 20:05:56 +08:00
4883d3eb50
refactor(replication): route runtime facades through ecstore ( #4248 )
2026-07-03 23:49:24 +08:00
769549dd2c
refactor(replication): isolate status contract boundaries ( #4236 )
2026-07-03 19:40:49 +08:00
18d89d1e31
refactor(replication): route metadata contracts through crate ( #4160 )
2026-07-02 08:48:42 +08:00
14016dbe8c
refactor(scanner): hide replication queue DTOs ( #4150 )
2026-07-02 00:17:03 +08:00
fc806679fb
refactor(ecstore): add replication work bridges ( #4137 )
2026-07-01 18:37:01 +08:00
fab78f8402
refactor(scanner): use expiry runtime boundary ( #4079 )
2026-06-30 02:09:52 +08:00
1b3dea012e
refactor: route ecstore runtime globals through facade ( #3941 )
2026-06-27 12:27:03 +08:00
72ae43cb90
refactor: segment external storage contract imports ( #3908 )
2026-06-26 16:44:59 +08:00
1f7e159388
refactor: segment external storage api boundaries ( #3903 )
2026-06-26 15:10:49 +08:00
4d6ea453a7
refactor: route scanner heal storage boundaries ( #3888 )
2026-06-26 05:24:41 +08:00
Zhengchao An