mirror of
https://github.com/rustfs/rustfs.git
synced 2026-09-05 19:55:37 +00:00
Zhengchao An
a5bde8b0af
* feat(ecstore): add on-demand migration bucket config model Introduce OnDemandMigrationConfig (deny_unknown_fields, version 1) with typed validation, credential redaction, a secret-free Debug impl, and the OnceLock publish hook the runtime registers into. Exported through the api facade. * feat(ecstore): persist on-demand migration config in bucket metadata Store the config as a RustFS extension entry (on-demand-migration.json) with its update time in .metadata.bin, add the typed BucketMetadataSys accessor, and publish the config through the hook on every cache-install path alongside the durability sync. * refactor(ecstore): extract shared remote S3 client builder Move the aws_sdk_s3 client construction out of bucket_target_sys into bucket/remote_s3_client.rs: endpoint assembly, credential provider, path-style selection, custom CA / skip-TLS transports and the outbound SSRF gate now build from a neutral RemoteS3EndpointSpec so replication targets and the upcoming on-demand migration source client share one policy. Replication builds its client through From<&BucketTarget>; the gate keeps its relaxed semantics (private allowed, loopback only behind RUSTFS_REPLICATION_ALLOW_LOOPBACK_TARGET) verbatim. The builder also gains optional connect/read timeouts and a User-Agent suffix interceptor, both unset for replication. Refs rustfs/backlog#2149 * feat(ecstore): add on-demand migration SourceClient Add bucket/on_demand_migration/source_client.rs on top of the shared remote S3 builder: HEAD, ranged streaming GET, ListObjectsV2 with source-prefix mapping, GetObjectTagging and an admin probe. Every request carries the x-rustfs-/x-minio-source-proxy-request anti-loop markers and a RustFS-OnDemandMigration/<version> User-Agent suffix; SSE-C source objects are rejected as unsupported. SourceError classifies SDK failures (not found, access denied, throttled, timeout, connect, server error) with retryability and a stable metrics label. Debug output redacts credentials. Refs rustfs/backlog#2149 * docs(operations): point outbound policy at shared remote S3 client builder * chore: integrate ODM-01 and ODM-02 as B1 base (fix facade merge) * feat(admin): add on-demand migration bucket admin API Add the management plane for On-Demand Migration (ODM-07, rustfs/backlog#2154): PUT/GET/DELETE /v3/on-demand-migration/{bucket}, PUT ?dry-run=true, and a GET .../status skeleton. - PUT authorizes SetBucketOnDemandMigration, checks the bucket, the RUSTFS_ON_DEMAND_MIGRATION_ENABLED switch and the license, validates the ODM-01 config against local endpoints and replication targets, probes the source with SourceClient::probe(), then persists through the incarnation gate and asks peers to reload. Responses carry the redacted config and a probe summary; probe failures name only the error class. - GET answers 404 NoSuchConfiguration when unset; DELETE is idempotent (204). - New AdminAction variants admin:SetBucketOnDemandMigration and admin:GetBucketOnDemandMigration, route policy matrix rows, registration and MinIO alias coverage, and a doc row for the extra handler gates. - rustfs-madmin gains on_demand_migration wire types and client methods; golden fixtures under crates/madmin/fixtures/on_demand_migration/ are asserted byte-for-byte by both the handler and the client tests. Anonymous sources still map to a 400 naming source.credentials until the runtime slice adds the credential-less path. * refactor(admin): route on-demand migration handler errors through the s3 facade
Architecture Documentation
Use this when: you need the contract, invariant, or boundary rule that governs a change, and you want the one document that owns it.
Source of truth: the code and the guards. scripts/check_architecture_migration_rules.sh enforces the CI-anchored documents below; scripts/check_doc_paths.sh fails the pre-commit gate when any doc under docs/ cites a repository path that no longer exists.
Two rules keep this directory healthy:
- Durable reference only. One-shot plans, task trackers, dated analyses, status snapshots, and PR-scoped notes do not belong in the repository; keep them in the issue tracker or a local worktree and delete them when the work closes.
- No copies of other sources of truth. Crate lists come from
Cargo.toml, CI steps from.github/workflows/, code structure from the code. Cite a file path plus a symbol name, never a line number, and never paste counts or tables that a command can regenerate.
Every document starts with a **Use this when:** line so an agent can decide in one glance whether to read further. The index below repeats those lines.
CI-anchored core
Required headings and strings in these files are asserted by scripts/check_architecture_migration_rules.sh; rename a heading only together with the guard.
| Document | Use this when |
|---|---|
| crate-boundaries.md | you add a crate dependency, move code across crates, touch a storage_api.rs boundary file, or need the change-type vocabulary the architecture guard enforces |
| runtime-lifecycle.md | moving or reordering anything in rustfs/src/startup_*.rs, changing readiness publication, or touching shutdown ordering |
| readiness-matrix.md | changing what a request surface does before storage or IAM is ready, changing probe semantics, or adding a runtime dependency that readiness must wait for |
| storage-control-data-plane.md | adding a storage API surface, a cluster read model, or a background-service status/reconcile surface, and you need to know which layer owns it |
| global-state-crate-split-plan.md | business logic needs runtime state (object store, endpoints, lock clients, lifecycle state, config) and you must pick the right boundary, or you are evaluating a crate split out of ECStore |
| global-state-inventory.md | you meet a GLOBAL_* static or an OnceLock and need to know whether it is a runtime ownership handle, an owner-local static, or process-global by design |
| ecstore-module-split-plan.md | you add lifecycle or replication logic and need to know which crate it belongs in, plan to move an operation family out of SetDisks, or the guard fails on one of the split rules |
| ecstore-api-facade-inventory.md | you need something from rustfs_ecstore in another crate, you are narrowing a rustfs_ecstore::api facade group, or the guard reports a facade bypass |
| obs-ecstore-dependency-inventory.md | adding, removing, or moving any rustfs_ecstore or rustfs_storage_api reference inside crates/obs |
| compat-cleanup-register.md | you add, review, or remove a temporary compatibility path and need the RUSTFS_COMPAT_TODO marker format and its removal condition |
| overview.md | you need the historical framing of the architecture-migration program or the phase names that other contracts refer to |
Contracts and invariants
| Document | Use this when |
|---|---|
| erasure-coding.md | changing anything under crates/ecstore/src/erasure/, crates/filemeta/, crates/ecstore/src/set_disk/, storage-class or layout code, or any decode, quorum, or heal boundary (normative spec) |
| placement-repair-invariants.md | changing anything that resolves an object to a pool, set, or disk, or that admits scanner or heal work |
| heal-concurrency-model.md | changing heal, PUT/multipart commit, delete, lifecycle expiry, or data-movement code that shares the (bucket, object) commit surface, or asking whether RustFS needs a persistent healing marker |
| unified-object-generation.md | adding or changing anything that fences a commit, scopes a read lease, gates old-directory cleanup, binds prepared pool reads, or settles quota against the current object version |
| decommission-compatibility.md | changing pool decommission or rebalance behavior, its admin API shape, the persisted PoolMeta fields, or how tier free versions move between pools |
| ecstore-layout-boundary.md | touching endpoint expansion, FormatV3, pool/set layout, or moving files between ECStore's internal directories |
| runtime-capability-contracts.md | changing the read-only observability or topology snapshot contracts in rustfs-storage-api, their providers, or the storage_classes payload of GET /rustfs/admin/v4/runtime/capabilities |
| workload-admission-contracts.md | adding a workload class or snapshot provider, or consuming admission state from a background job |
| background-controller-contract.md | adding a status snapshot or reconcile surface for a background service, or being tempted to fold several services into a generic controller |
| config-model-boundary-adr.md | touching the server-config model (Config, KV, KVS) or its persistence, or asking which crate owns which part of server configuration |
| admin-route-action-snapshot.md | adding, moving, or re-authorizing an admin route and needing to know where the route → handler → AdminAction contract is enforced |
| kms-bulk-rekey-contract.md | changing the bulk envelope re-wrap sweep, its admin endpoints, the re-wrap primitive, or which objects a rekey may touch |
Support and compatibility matrices (release-facing, keep current)
| Document | Use this when |
|---|---|
| s3-compatibility-matrix.md | writing or checking a user-facing S3 compatibility claim, or moving a Ceph s3tests case between lists |
| s3-tables-support-matrix.md | writing a release note or client-compatibility statement about S3 Tables / Iceberg REST Catalog (cutover procedure: ../operations/s3-tables-cutover-runbook.md) |
| minio-rustfs-router-compatibility.md | a client or mc call that works against MinIO fails against RustFS and you need to know whether the endpoint is missing, stubbed, or deliberately different |
| minio-file-format-compat.md | deciding whether a MinIO drive set, bucket-metadata blob, or SSE object can be read or imported by a given RustFS build, or before touching a listed version anchor |
Operations runbooks live in ../operations/ and testing references in ../testing/README.md.