Commit Graph

6 Commits

Author SHA1 Message Date
Zhengchao An df4fdef1d8 docs(odm): add the on-demand migration operations guide (#7086)
* docs(odm): add the on-demand migration operations guide

Positioning against replication, site replication, tiering and the
replication read-proxy; the admin API walkthrough; the full config field
table with defaults and bounds; the semantics table with the test that
pins each row; provider presets and least-privilege source permissions;
integrity, ETag and metadata mapping; protections, error codes,
observability with PromQL, troubleshooting, known limitations and
security notes.

Adds docs/architecture/background-services-inventory.md with the
write-back pipeline, backfill job and backfill recovery loop rows, and
registers it in the architecture index.

Refs rustfs/backlog#2160, rustfs/backlog#2147.

* docs(odm): note migration sources on the replication egress guard

Refs rustfs/backlog#2160.
2026-09-03 08:48:31 +08:00
Zhengchao An 7e1f261e38 refactor(ecstore): shared remote S3 client builder and ODM source client (#7067)
* refactor(ecstore): extract shared remote S3 client builder

Move the aws_sdk_s3 client construction out of bucket_target_sys into
bucket/remote_s3_client.rs: endpoint assembly, credential provider,
path-style selection, custom CA / skip-TLS transports and the outbound
SSRF gate now build from a neutral RemoteS3EndpointSpec so replication
targets and the upcoming on-demand migration source client share one
policy. Replication builds its client through From<&BucketTarget>; the
gate keeps its relaxed semantics (private allowed, loopback only behind
RUSTFS_REPLICATION_ALLOW_LOOPBACK_TARGET) verbatim. The builder also
gains optional connect/read timeouts and a User-Agent suffix
interceptor, both unset for replication.

Refs rustfs/backlog#2149

* feat(ecstore): add on-demand migration SourceClient

Add bucket/on_demand_migration/source_client.rs on top of the shared
remote S3 builder: HEAD, ranged streaming GET, ListObjectsV2 with
source-prefix mapping, GetObjectTagging and an admin probe. Every request
carries the x-rustfs-/x-minio-source-proxy-request anti-loop markers and
a RustFS-OnDemandMigration/<version> User-Agent suffix; SSE-C source
objects are rejected as unsupported. SourceError classifies SDK failures
(not found, access denied, throttled, timeout, connect, server error)
with retryability and a stable metrics label. Debug output redacts
credentials.

Refs rustfs/backlog#2149

* docs(operations): point outbound policy at shared remote S3 client builder
2026-09-02 21:59:11 +08:00
Zhengchao An 0a975f2fe2 docs(knowledge-base): prune stale content and add agent-facing index (#7035) 2026-09-02 08:26:59 +08:00
GatewayJ 1b1b217826 fix(iam): preserve OIDC outbound policy errors (#5762) 2026-08-08 19:28:47 +08:00
houseme 0a2370c024 docs: clarify outbound allowlist scope (#5336) 2026-07-27 19:57:22 +08:00
Heracles 0c9d721910 docs: document RUSTFS_OUTBOUND_ALLOW_ORIGINS outbound policy (#5334) 2026-07-27 19:50:47 +08:00