Commit Graph

2424 Commits

Author SHA1 Message Date
overtrue 6233466c5d chore: integrate current main for namespace target validation 2026-09-06 15:49:36 +08:00
Zhengchao An a40ec8a6f3 fix(odm): preserve progress when list-through is disabled (#7278)
* test(odm): cover disabled list-through continuation progress

* test(odm): cover literal cache tags in disabled list cursors

* fix(odm): preserve progress when list-through is disabled
2026-09-06 15:24:59 +08:00
Zhengchao An c99efd9477 test(admin): box remote target repair scenarios (#7277) 2026-09-06 15:23:30 +08:00
Zhengchao An 27d593fa35 test(odm): bypass loopback proxies in native list fixtures (#7276) 2026-09-06 15:22:50 +08:00
overtrue fdb3d9f9bb test(startup): encode observation digests with shared hex helper 2026-09-06 15:10:28 +08:00
Zhengchao An 1a88870809 fix(odm): preserve cursor compatibility and native source semantics (#7238) 2026-09-06 14:12:56 +08:00
唐小鸭 760c9d65be fix(replication): close IAM snapshot, marker purge and broadcast gaps (#7195) 2026-09-06 14:12:25 +08:00
houseme 6a323c3e91 test(heal): cover start retry and deadline outcome contracts (#7242) 2026-09-06 14:11:24 +08:00
overtrue 44333e136b test(rpc): use native part size in signed fixture 2026-09-06 13:57:07 +08:00
Zhengchao An 38611d2510 fix(admin): compile metadata test helper only in tests (#7266)
fix(admin): compile metadata update facade only for tests
2026-09-06 13:28:58 +08:00
Zhengchao An 5aef1796cc fix(odm): reject ambiguous native source dot segments (#7263)
* fix(odm): reject ambiguous native source dot segments

* docs(odm): align native provider limitations with implementation
2026-09-06 13:11:35 +08:00
overtrue 9e24d23c30 test(rpc): observe bootstrap CAS during fresh startup 2026-09-06 13:01:22 +08:00
Zhengchao An a8aaadb886 fix(replication): preserve concurrent remote target changes (#7262)
* test(replication): cover concurrent remote target writes

* fix(replication): merge remote target writes under transactions

* test(replication): preserve concurrent repairs during target updates

* fix(replication): retain removal guards after request cancellation

* test(replication): enable loopback in ordinary target fixtures

* test(replication): isolate remote target mutation scenarios

* refactor(admin): keep target writes within existing boundaries

* test(replication): assert removed target cache state

* test(replication): inspect target cache before client refresh
2026-09-06 12:51:44 +08:00
overtrue 6277287399 chore(test): merge main before startup CAS coverage 2026-09-06 12:45:50 +08:00
Zhengchao An 71f1dcf859 fix(admin): follow recovery facade and error boundaries (#7259) 2026-09-06 12:35:54 +08:00
Zhengchao An d44244f60f fix(admin): preserve metadata during export and target repair (#7258)
* fix(admin): reject incomplete metadata backups

* fix(admin): repair remote targets from locked disk state

* test(admin): fence target repair against source changes

* fix(admin): report unreadable XML in metadata exports

* test(admin): enable loopback in target repair fixtures

* refactor(admin): remove unused metadata getter forwards

* test(admin): box direct target repair futures

* test(admin): box target repair scenarios at env boundary
2026-09-06 12:18:00 +08:00
Zhengchao An d3884ed3ea test(odm): use fixed-size Azure request chunks (#7254) 2026-09-06 12:10:39 +08:00
overtrue 358ff0f0e6 test(rpc): select fixture disks through the store topology 2026-09-06 11:42:53 +08:00
overtrue 1cea5fa1c4 test(rpc): retain bootstrap authority across delayed requests 2026-09-06 11:42:53 +08:00
overtrue e0663c11af test(rpc): cover signed listener binding across startup 2026-09-06 11:13:09 +08:00
Zhengchao An 282d6d5efe fix(odm): decode encoded Azure blob names exactly once (#7251)
* fix(odm): decode encoded Azure blob names exactly once

* test(odm): cover Azure encoded name transport matrix

* test(odm): keep Azure cursor continuation query stable
2026-09-06 11:10:40 +08:00
overtrue 11c9fd64ce fix(rpc): bind local mutations to the listener instance 2026-09-06 10:58:11 +08:00
Zhengchao An 61e0edce16 test(odm): pin gcs error classes and require the backend contracts in ci (#7250)
* test(odm): pin gcs source status-to-error-class mapping

The native GCS backend classifies every failure from the HTTP status
alone, because GCS states its error code in a body this backend never
reads. Only NotFound is negative-cached and only a retryable class may be
re-sent, so cover 401/403 -> AccessDenied, 429/503 -> Throttled,
500/502 -> ServerError and 404 -> NotFound over both HEAD and GET.

* ci(odm): require the source-backend contract tests in test-and-lint

The shared contract tests already run in ci/test-and-lint, but only
because gcs is a rustfs default feature; nothing failed if that
selection went away. Pin the S3, Azure and native GCS contracts in the
core required-test manifest so a lost selection fails the lane.
2026-09-06 10:45:09 +08:00
overtrue 7f631ec378 test(rpc): mark instance regression request as v2 authenticated 2026-09-06 10:21:55 +08:00
overtrue 3205f85c2a test(rpc): preserve bootstrap metadata in instance snapshot 2026-09-06 10:21:55 +08:00
overtrue ae87ddbe2f test(rpc): reproduce same-UUID cross-instance rename 2026-09-06 10:21:55 +08:00
Zhengchao An 5dca076efe fix(connect): sync protocol fixture consumers (#7167)
* fix(connect): sync protocol fixture consumers

* fix(connect): preserve enrollment validation order

* fix(connect): satisfy base64 length lint

* fix(connect): restore signature validation order

* fix(connect): preserve signature precedence across chain parsing

* fix(connect): preserve signature error classification

* test(ci): provide log path in workflow harness
2026-09-06 10:12:51 +08:00
Zhengchao An 6655272c90 fix(ilm): reject invalid retention counts and validate lifecycle filters (#7132)
* fix(ilm): reject invalid retention counts and validate lifecycle filters

`NewerNoncurrentVersions` had no lower bound at PUT, and evaluation read a
negative count through `usize::try_from(...).unwrap_or(usize::MAX)`. An
HTTP-accepted rule therefore retained (almost) everything and silently
stopped expiring versions — the one outcome a retention rule must never
produce by accident.

Reject a negative count during validation, and stop reading one as
"retain everything" anywhere it can still arrive from older persistence
or an import: evaluation takes no action for such a rule and says so in a
diagnostic, the batch limit path yields no event, and `Evaluator::eval`
reports a typed corruption error to callers that can surface one.

A count-only noncurrent expiration is a MinIO extension, not an AWS form.
It used to be rejected as an actionless rule and was never executed. It
is now accepted and honoured with the semantics MinIO gives it: the
newest N noncurrent versions are kept and every older one is due as soon
as it became noncurrent. Zero keeps the meaning the batch limit path has
always given it — no count constraint — so a zero-count rule with no age
condition still has no action.

`LifecycleRuleFilter` is an all-`Option` DTO, so the schema constraints
were not checked anywhere: validate at most one top-level predicate, an
`And` that combines at least two, no repeated tag key, tag key/value
limits, non-negative sizes, and `ObjectSizeGreaterThan <
ObjectSizeLessThan`. An empty filter stays valid — AWS documents it as
"every object in the bucket".

Schema-shape violations are reported with a distinct `ErrorKind` so the
S3 boundary answers them with `MalformedXML`; rejected values keep the
`InvalidArgument` this path has always returned.

backlog#2201

* fix(ilm): satisfy lifecycle clippy checks

* fix(ilm): fail closed on invalid lifecycle rules

* fix: initialize optional migration source fields

---------

Co-authored-by: cxymds <cxymds@gmail.com>
2026-09-06 10:11:05 +08:00
cxymds 0a5d4cef0e fix(tier): drain cleanup before tier removal (#7213)
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
2026-09-06 10:10:48 +08:00
cxymds 941fae61a0 fix(ilm): persist bounded transition recovery controls (#7200)
* fix(ilm): persist bounded transition recovery controls

* test(ilm): keep expiry sentinel within control range

* fix(ilm): repair recovery control CI regressions

---------

Co-authored-by: overtrue <anzhengchao@gmail.com>
2026-09-06 10:10:24 +08:00
Zhengchao An 03fa62cc7d fix(admin): keep exporting past unreadable configs and allow explicit target repair (#7247)
* test(ecstore): pin MinIO array-shaped targets blob as unreadable

* fix(admin): mark unreadable configs instead of aborting export

* feat(admin): opt-in replacement of unreadable bucket targets
2026-09-06 10:03:32 +08:00
Zhengchao An bde4b78a9f test(odm): run multipart race on large stack (#7239) 2026-09-06 04:46:44 +08:00
Zhengchao An eb1b17802c test(odm): provide the source region in access fixture (#7235) 2026-09-06 02:34:56 +08:00
Zhengchao An 112f70914d fix(build): scope migration helpers to their features (#7234)
* test(odm): keep listing header import test scoped

* fix(build): gate GCS-only migration HTTP helpers
2026-09-06 02:24:04 +08:00
Zhengchao An dd368f0f5b fix(odm): fence source work against bucket recreation (#7231)
* fix(odm): fence backfill checkpoints by bucket incarnation

* fix(odm): bind source work to the bucket incarnation

* fix(odm): retain checkpoint fences through owned commit tails

* docs(odm): explain application service and incarnation boundaries

* test(odm): probe lifecycle fence after checkpoint waiter aborts

* fix(odm): defer source identity errors past local reads

* docs(metadata): clarify MinIO target recovery limits

* fix(odm): keep source-free reads independent of capture errors

* fix(odm): retain one source policy snapshot across lookup

* test(odm): name recorded metadata hook snapshots
2026-09-06 02:05:43 +08:00
Zhengchao An 6d8606412e fix(odm): compile relocated instance-bound backfill service (#7230) 2026-09-06 01:43:14 +08:00
Zhengchao An 037354cec0 fix(build): declare relocated migration service dependencies (#7229) 2026-09-06 01:37:02 +08:00
Zhengchao An 30ab919bb3 fix(ci): route v1 listing test types through application bridge (#7227) 2026-09-06 01:35:05 +08:00
Zhengchao An 8fc1c9281e refactor(odm): move migration orchestration into application (#7226)
* refactor(odm): move migration orchestration into application

* style(odm): format relocated listing test imports
2026-09-06 01:31:58 +08:00
Zhengchao An 14cef91423 fix(admin): add isolated bucket metadata diagnostics (#7225) 2026-09-06 01:30:18 +08:00
Zhengchao An 955d491174 feat(build): make native GCS backends optional (#7223)
* feat(build): make native GCS backends optional

* test(odm): cover native Azure runtime credentials
2026-09-06 01:27:36 +08:00
Zhengchao An 35aefbb2a5 fix(s3): keep ListObjects v1 local during migration (#7220)
* fix(s3): keep ListObjects v1 local during migration

* test(s3): use the v1 listing request DTO directly
2026-09-06 01:25:44 +08:00
houseme f54323b062 chore(deps): preserve scanner and heal validation compatibility (#7209)
* chore(deps): refresh scanner heal batch dependency baseline

Regenerate compatible lockfile selections before the next implementation
batch. Cargo upgrade leaves direct requirements unchanged.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): remove duplicate local rename implementation

Keep the canonical commit module after concurrent storage changes merged.
The control-write and rollback changes are already present there.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* chore(deps): refresh profiling dependencies for the next batch

Update hotpath and its macro crate to the compatible patch release before
the next dependency-ready implementation tasks.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(deps): preserve supported hotpath focus expressions

Keep the profiler runtime before its regex-lite compatibility regression.
Track the opt-in validation required to remove this constraint in backlog.

Refs rustfs/backlog#2302.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(rustfs): complete list-through source config

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
2026-09-05 23:50:43 +08:00
Zhengchao An 188f380b3b feat(ecstore): add native azure blob and gcs migration sources (#7211)
* feat(ecstore): add a native azure blob odm source backend

* feat(ecstore): add a native gcs odm source backend and one backend contract

* fix(ecstore): refuse an empty azure account key at client build

* fix(ecstore): probe gcs sources with the listing permission

* fix(app): drop a redundant match guard on the sse config lookup

* fix(ecstore): drop stale rename commit duplicates from local.rs

* test(ecstore): use the sanctioned placeholder key in the gcs fixture
2026-09-05 22:06:30 +08:00
Zhengchao An e2a921bc16 fix(storage): harden ODM and scanner publication (#7187)
* fix(storage): harden ODM and scanner publication

* fix(app): simplify absent SSE configuration matching

* test(heal): settle PUT rename tails before disk-wipe fixtures

* fix(ecstore): remove duplicate local rename implementation

Keep the canonical commit module after concurrent storage changes merged.
The control-write and rollback changes are already present there.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ci): satisfy new clippy lints

* style(scanner): order merged test imports

* fix(scanner): invalidate bucket work after namespace completion

* fix(scanner): fence cached snapshots by scan execution

---------

Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-09-05 13:47:12 +00:00
Zhengchao An 9e2545244c fix(odm): bound empty pagination chains with staged tokens (#7197)
* fix(odm): add staged cross-request pagination progress budgets

* fix(ecstore): remove duplicate local rename implementation

Keep the canonical commit module after concurrent storage changes merged.
The control-write and rollback changes are already present there.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(app): simplify absent SSE configuration matching

* fix(tests): satisfy new clippy lints

---------

Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-09-05 11:44:49 +00:00
houseme e8a7f4bc4a fix(ecstore): remove duplicate local rename implementation (#7190)
* fix(ecstore): remove duplicate local rename implementation

Keep the canonical commit module after concurrent storage changes merged.
The control-write and rollback changes are already present there.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ci): satisfy new clippy lints

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
2026-09-05 10:44:01 +00:00
houseme acfeef55ab feat(scanner): add bounded incarnation-scoped ACK receiver (#7182)
* chore(deps): refresh SDKs and pin clock skew regression coverage

Refresh compatible dependencies for Scanner/Heal V2 batch 1 and verify
the production S3 retry/signing path with a deterministic clock.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* feat(scanner): add bounded incarnation-scoped ACK receiver

Refs rustfs/backlog#2265 and rustfs/backlog#2240.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-09-05 08:54:55 +00:00
houseme e6bf2a4646 fix(admin): report partial background heal coverage (#7178)
* chore(deps): refresh SDKs and pin clock skew regression coverage

Refresh compatible dependencies for Scanner/Heal V2 batch 1 and verify
the production S3 retry/signing path with a deterministic clock.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(admin): report partial background heal coverage

Refs rustfs/backlog#2035 and rustfs/backlog#2240.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-09-05 08:45:45 +00:00
cxymds bbd7b9ef17 fix(site-replication): bound and order outage recovery (#7148)
* fix(site-replication): wake retry drain after peer recovery

* fix(site-replication): replay configure after bucket make

* fix(site-replication): serialize retry replay state

* fix(site-replication): persist destructive retry intents

* fix(site-replication): bound retry recovery rounds

* fix(site-replication): keep recovery replay live

* fix(site-replication): preserve retry ordering

* fix(site-replication): bound retry coordination

* fix(site-replication): serialize topology replay

* fix(site-replication): fence distributed retry state

* fix(site-replication): bound outage retry drain

* fix(site-replication): drop unsafe delete retry intents

* fix(site-replication): order bucket mutation replay

* fix(site-replication): harden outage retry replay

* fix(site-replication): fence destructive peer delivery

* fix(site-replication): avoid peer edit retry deadlock

* fix(site-replication): fence retry error classification

* fix(site-replication): classify connect timeouts

* fix(site-replication): close recovery review races

* test(site-replication): cover timeout endpoint text

* fix(site-replication): close destructive recovery gaps

* fix(site-replication): fence recovery revisions

* fix(site-replication): replay bucket metadata on recovery

* fix(site-replication): preserve s3gate boundary

---------

Co-authored-by: overtrue <anzhengchao@gmail.com>
2026-09-05 15:55:54 +08:00