Commit Graph

6399 Commits

Author SHA1 Message Date
houseme 51982be687 Merge branch 'main' into houseme/fix/scanner-heal-v2-scanner-integration 2026-09-06 01:33:21 +08:00
Zhengchao An 8fc1c9281e refactor(odm): move migration orchestration into application (#7226)
* refactor(odm): move migration orchestration into application

* style(odm): format relocated listing test imports
2026-09-06 01:31:58 +08:00
Zhengchao An 14cef91423 fix(admin): add isolated bucket metadata diagnostics (#7225) 2026-09-06 01:30:18 +08:00
Zhengchao An c9acc33720 test(odm): verify rc5 rollback configuration recovery (#7224)
* test(odm): verify rc5 rollback configuration recovery

* ci(e2e): run the ODM rollback recovery scenario
2026-09-06 01:28:54 +08:00
Zhengchao An 955d491174 feat(build): make native GCS backends optional (#7223)
* feat(build): make native GCS backends optional

* test(odm): cover native Azure runtime credentials
2026-09-06 01:27:36 +08:00
Zhengchao An 1c4e9f1b65 fix(odm): order initial installation against configuration removal (#7222)
* fix(odm): retain removal generation before initial install

* fix(odm): reserve generations only for configured buckets
2026-09-06 01:26:57 +08:00
houseme a9f01dbbdb fix(ecstore): restore odm source contract tests (#7215)
* fix(ecstore): restore odm source contract tests

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(ci): initialize replication evidence in chain test

Run the replication workflow's evidence initialization before the chain handoff self-test executes the suite step. This keeps the test model aligned with the workflow-provided LOG_FILE and TMPDIR values.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(odm): distinguish missing GCS buckets from object misses (#7221)

---------

Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
2026-09-06 01:26:08 +08:00
Zhengchao An 35aefbb2a5 fix(s3): keep ListObjects v1 local during migration (#7220)
* fix(s3): keep ListObjects v1 local during migration

* test(s3): use the v1 listing request DTO directly
2026-09-06 01:25:44 +08:00
Zhengchao An 8fb335cf19 test(e2e): pin upgrade compatibility to rc.5 and cover bucket configuration (#7217)
test(e2e): prove bucket config survives rc.5 upgrade and rollback

Add two upgrade-compatibility scenarios pinned to the on-demand-migration
series' on-disk surfaces: BucketMetadata's 44 -> 46 msgpack keys, the
fail-closed bucket-config reads of rustfs#7172, the encryption-gated PUT
path of rustfs#7183, and the default-on migration module of rustfs#7089.

The upgrade case writes versioning, SSE-S3 default encryption, a validated
replication target plus rule, lifecycle, tags, quota, a public access block,
a bucket policy and an object lock configuration with the pinned previous
release, then asserts each one reads back unchanged on the current build,
that list-remote-targets still reports the target, that writes to the
encrypted and plain buckets keep their encryption posture, that every
pre-upgrade object including a multipart one is byte-identical, and that an
unconfigured bucket reports no migration and still answers NoSuchKey.

The rollback case is the reverse: the current build writes the 46-key blob
and the previous release must decode it by skipping the two unknown keys.
2026-09-06 01:08:22 +08:00
Zhengchao An 8f763fb1a2 fix(ci): run existing script contracts in quick checks (#7203)
* fix(ci): share quick checks and lint workflows

* fix(ci): install actionlint from its verified release

* fix(ci): reject dependencies on required quick checks

* fix(ci): run existing script contracts in quick checks
2026-09-06 00:50:06 +08:00
Zhengchao An a6b5da64f2 fix(ci): serialize performance on shared functional VMs (#7204) 2026-09-06 00:15:03 +08:00
Zhengchao An 1210428b6d fix(ci): publish immutable nightly package candidates (#7202) 2026-09-06 00:14:37 +08:00
Zhengchao An d5426f59ec fix(ci): isolate functional evidence and preserve every result (#7201)
* fix(ci): preserve reported functional suite failures

* fix(ci): isolate functional evidence and preserve every result

* fix(ci): exclude sensitive scratch files from suite artifacts
2026-09-06 00:14:22 +08:00
houseme b8f0119320 Merge branch 'main' into houseme/fix/scanner-heal-v2-scanner-integration 2026-09-05 23:53:05 +08:00
houseme f54323b062 chore(deps): preserve scanner and heal validation compatibility (#7209)
* chore(deps): refresh scanner heal batch dependency baseline

Regenerate compatible lockfile selections before the next implementation
batch. Cargo upgrade leaves direct requirements unchanged.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): remove duplicate local rename implementation

Keep the canonical commit module after concurrent storage changes merged.
The control-write and rollback changes are already present there.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* chore(deps): refresh profiling dependencies for the next batch

Update hotpath and its macro crate to the compatible patch release before
the next dependency-ready implementation tasks.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(deps): preserve supported hotpath focus expressions

Keep the profiler runtime before its regex-lite compatibility regression.
Track the opt-in validation required to remove this constraint in backlog.

Refs rustfs/backlog#2302.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(rustfs): complete list-through source config

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
2026-09-05 23:50:43 +08:00
houseme 20ca988e85 Merge branch 'main' into houseme/fix/scanner-heal-v2-scanner-integration 2026-09-05 23:33:13 +08:00
hector cc5060ac20 ci(functional): fix dashboard report upload argv overflow and security checkout clobbering (#7212)
Two fixes for the functional test chain:

1. Report upload fails with 'jq: Argument list too long' when the base64
   report is passed through '--arg content' (pool reports exceed the OS
   argv limit; last night's pool run lost its Step Results report this
   way). Write the base64 payload to a temp file and load it in jq via
   --rawfile instead. Applied uniformly to all nine suite workflows
   that share this upload step.

2. The security workflow cloned rustfs/auto-testing into the workspace
   and then ran actions/checkout at the workspace root for the OIDC
   live gate script, which wiped the auto-testing clone and killed the
   suite with 'chmod: cannot access auto-testing/rustfs-security-test.sh'.
   Check out the repository into the rustfs-repo/ subdirectory instead
   and point RUSTFS_SECURITY_OIDC_LIVE_SCRIPT there.

Co-authored-by: rustfs-ci <ci@rustfs.com>
2026-09-05 23:06:28 +08:00
houseme 045770d09d fix(scanner): satisfy cache prefix sort lint
Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
2026-09-05 22:34:20 +08:00
houseme 278cbaefd5 Merge branch 'main' into houseme/fix/scanner-heal-v2-scanner-integration 2026-09-05 22:16:44 +08:00
houseme 982bebf6a5 Merge remote-tracking branch 'origin/main' into houseme/fix/scanner-heal-v2-scanner-integration
Resolved scanner cache publication conflicts after the main branch added
execution identity fencing and post-lease activity proof coverage.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
2026-09-05 22:10:39 +08:00
Zhengchao An 188f380b3b feat(ecstore): add native azure blob and gcs migration sources (#7211)
* feat(ecstore): add a native azure blob odm source backend

* feat(ecstore): add a native gcs odm source backend and one backend contract

* fix(ecstore): refuse an empty azure account key at client build

* fix(ecstore): probe gcs sources with the listing permission

* fix(app): drop a redundant match guard on the sse config lookup

* fix(ecstore): drop stale rename commit duplicates from local.rs

* test(ecstore): use the sanctioned placeholder key in the gcs fixture
2026-09-05 22:06:30 +08:00
Zhengchao An e2a921bc16 fix(storage): harden ODM and scanner publication (#7187)
* fix(storage): harden ODM and scanner publication

* fix(app): simplify absent SSE configuration matching

* test(heal): settle PUT rename tails before disk-wipe fixtures

* fix(ecstore): remove duplicate local rename implementation

Keep the canonical commit module after concurrent storage changes merged.
The control-write and rollback changes are already present there.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ci): satisfy new clippy lints

* style(scanner): order merged test imports

* fix(scanner): invalidate bucket work after namespace completion

* fix(scanner): fence cached snapshots by scan execution

---------

Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-09-05 13:47:12 +00:00
houseme 447f3c704b feat(heal): add explicit committed MRF snapshot reader (#7179)
* chore(deps): refresh SDKs and pin clock skew regression coverage

Refresh compatible dependencies for Scanner/Heal V2 batch 1 and verify
the production S3 retry/signing path with a deterministic clock.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* feat(heal): add explicit committed MRF snapshot reader

Refs rustfs/backlog#2263 and rustfs/backlog#2240.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* docs(heal): register legacy MRF inspection cleanup

State the compatibility removal condition on the source marker and in
the architecture cleanup register.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-09-05 13:33:41 +00:00
Zhengchao An d915f9565e fix(ci): preserve reported functional suite failures (#7199) 2026-09-05 13:26:23 +00:00
Zhengchao An 55ad7508b9 fix(tier): persist coordinator intent before waking refresh (#7171) 2026-09-05 13:24:00 +00:00
houseme 1aafe477ec test(scanner): verify joint checkpoint coverage metadata
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
2026-09-05 21:05:13 +08:00
houseme c33a42c5ab test(scanner): supply explicit coverage in publication fixtures
Keep the confirmed-empty namespace fixture authoritative under the required
coverage contract and qualify the bucket cache metadata test type.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
2026-09-05 20:55:19 +08:00
houseme e3fd1ae19a fix(scanner): fence same-cycle caches with full activity coverage
Keep structural baseline identity separate from the full activity coverage
required by bucket admission and set publication. Require complete set
coverage proofs while retaining revision CAS and epoch regression checks.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
2026-09-05 20:55:19 +08:00
houseme 6c1d401215 test(scanner): reproduce same-cycle dirty aggregate replay
Cover a Normal-to-Normal retry with a new dirty bucket generation after
bucket persistence and root delivery failure.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
2026-09-05 20:53:20 +08:00
houseme c80ff445c0 fix(scanner): fence set snapshot reuse with the scan work proof
Prevent same-cycle set publication from replacing freshly scanned maintenance
results with an older Normal aggregate. Recognize uniform completed
maintenance baselines when planning later ordinary dirty-bucket work.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
2026-09-05 20:53:20 +08:00
houseme 3d4c05ddf9 fix(scanner): bind bucket cache reuse to scan work requirements
Carry stable scan mode and full-maintenance requirements in the existing
opaque bucket digest before local and remote cache admission. Different
requirements cannot replay a same-cycle Normal cache after root delivery
failure; matching requirements remain reusable for the same intent.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
2026-09-05 20:52:15 +08:00
houseme f9dd0388fa fix(scanner): refresh scope safety independently of idle backoff
Inspect maintenance on multi-disk startup and refresh changed or failed
evidence even when explicit bitrot configuration disables idle backoff.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
2026-09-05 20:50:13 +08:00
houseme 527860d71e fix(scanner): keep maintenance cycles outside dirty bucket scopes
Force complete bucket scope for deep scans and scheduled maintenance while
preserving the existing planner for verified ordinary dirty work.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
2026-09-05 20:50:12 +08:00
houseme 7d2c120073 test(scanner): use valid modification times in checkpoint fixtures
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
2026-09-05 20:49:50 +08:00
houseme 489c3276a4 fix(scanner): verify coverage receipts and scan strength
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
2026-09-05 20:49:50 +08:00
houseme b09a4f3706 fix(scanner): keep stable snapshot rescan behavior
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
2026-09-05 20:49:50 +08:00
houseme a0c32e7c6b fix(scanner): retain scoped partial coverage across dirty plans
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
2026-09-05 20:49:50 +08:00
Zhengchao An 33fd056000 fix(ecstore): release heal disk snapshot before nested reads (#7189)
* fix(ecstore): release heal disk snapshot before nested reads

* fix(ecstore): remove duplicate local rename implementation

Keep the canonical commit module after concurrent storage changes merged.
The control-write and rollback changes are already present there.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(app): simplify absent SSE configuration matching

* fix(tests): satisfy new clippy lints

---------

Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-09-05 12:49:40 +00:00
houseme d5a1530409 fix(scanner): require complete publication coverage
Refs rustfs/backlog#2261 and rustfs/backlog#2240.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
2026-09-05 20:48:01 +08:00
houseme e7b0349a4f Merge remote-tracking branch 'origin/main' into houseme/fix/scanner-heal-v2-scanner-integration 2026-09-05 20:43:53 +08:00
RustFS af2e9df821 fix(lifecycle): correct expiration and transition evaluation (#7169) 2026-09-05 12:42:22 +00:00
cxymds 0a92a7d98c fix(tier): bound remote transition requests (#7147)
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
2026-09-05 12:41:11 +00:00
Zhengchao An c589fd2439 fix(dev): install a lightweight formatting commit hook (#7198) 2026-09-05 12:22:56 +00:00
Zhengchao An 3e5d4ebb09 fix(ecstore): release multipart disk snapshot before nested reads (#7184)
* fix(ecstore): release multipart disk snapshot before nested reads

* fix(ecstore): remove duplicate local rename implementation

Keep the canonical commit module after concurrent storage changes merged.
The control-write and rollback changes are already present there.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(app): simplify absent SSE configuration matching

* fix(tests): satisfy new clippy lints

---------

Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-09-05 12:20:50 +00:00
houseme 3677871468 chore(deps): bump zstd to 0.14 (#7173)
Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-09-05 12:16:46 +00:00
Zhengchao An cc1ec6b992 fix(ci): share quick checks and lint workflows (#7194)
* fix(ci): share quick checks and lint workflows

* fix(ci): install actionlint from its verified release

* fix(ci): reject dependencies on required quick checks
2026-09-05 12:03:59 +00:00
houseme bdbdca07c8 fix(deps): preserve supported hotpath focus expressions
Keep the profiler runtime before its regex-lite compatibility regression.
Track the opt-in validation required to remove this constraint in backlog.

Refs rustfs/backlog#2302.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
2026-09-05 19:50:46 +08:00
Zhengchao An 9e2545244c fix(odm): bound empty pagination chains with staged tokens (#7197)
* fix(odm): add staged cross-request pagination progress budgets

* fix(ecstore): remove duplicate local rename implementation

Keep the canonical commit module after concurrent storage changes merged.
The control-write and rollback changes are already present there.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(app): simplify absent SSE configuration matching

* fix(tests): satisfy new clippy lints

---------

Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-09-05 11:44:49 +00:00
houseme 53efaa2b8f chore(deps): refresh profiling dependencies for the next batch
Update hotpath and its macro crate to the compatible patch release before
the next dependency-ready implementation tasks.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
2026-09-05 19:10:54 +08:00
Zhengchao An f053862aad docs: request concrete behavior evidence in pull requests (#7196) 2026-09-05 18:56:26 +08:00