5 Commits

Author	SHA1	Message	Date
Zhengchao An	a5bde8b0af	feat(admin): add on-demand migration bucket admin API (#7076) ... * feat(ecstore): add on-demand migration bucket config model Introduce OnDemandMigrationConfig (deny_unknown_fields, version 1) with typed validation, credential redaction, a secret-free Debug impl, and the OnceLock publish hook the runtime registers into. Exported through the api facade. * feat(ecstore): persist on-demand migration config in bucket metadata Store the config as a RustFS extension entry (on-demand-migration.json) with its update time in .metadata.bin, add the typed BucketMetadataSys accessor, and publish the config through the hook on every cache-install path alongside the durability sync. * refactor(ecstore): extract shared remote S3 client builder Move the aws_sdk_s3 client construction out of bucket_target_sys into bucket/remote_s3_client.rs: endpoint assembly, credential provider, path-style selection, custom CA / skip-TLS transports and the outbound SSRF gate now build from a neutral RemoteS3EndpointSpec so replication targets and the upcoming on-demand migration source client share one policy. Replication builds its client through From<&BucketTarget>; the gate keeps its relaxed semantics (private allowed, loopback only behind RUSTFS_REPLICATION_ALLOW_LOOPBACK_TARGET) verbatim. The builder also gains optional connect/read timeouts and a User-Agent suffix interceptor, both unset for replication. Refs rustfs/backlog#2149 * feat(ecstore): add on-demand migration SourceClient Add bucket/on_demand_migration/source_client.rs on top of the shared remote S3 builder: HEAD, ranged streaming GET, ListObjectsV2 with source-prefix mapping, GetObjectTagging and an admin probe. Every request carries the x-rustfs-/x-minio-source-proxy-request anti-loop markers and a RustFS-OnDemandMigration/<version> User-Agent suffix; SSE-C source objects are rejected as unsupported. SourceError classifies SDK failures (not found, access denied, throttled, timeout, connect, server error) with retryability and a stable metrics label. Debug output redacts credentials. Refs rustfs/backlog#2149 * docs(operations): point outbound policy at shared remote S3 client builder * chore: integrate ODM-01 and ODM-02 as B1 base (fix facade merge) * feat(admin): add on-demand migration bucket admin API Add the management plane for On-Demand Migration (ODM-07, rustfs/backlog#2154): PUT/GET/DELETE /v3/on-demand-migration/{bucket}, PUT ?dry-run=true, and a GET .../status skeleton. - PUT authorizes SetBucketOnDemandMigration, checks the bucket, the RUSTFS_ON_DEMAND_MIGRATION_ENABLED switch and the license, validates the ODM-01 config against local endpoints and replication targets, probes the source with SourceClient::probe(), then persists through the incarnation gate and asks peers to reload. Responses carry the redacted config and a probe summary; probe failures name only the error class. - GET answers 404 NoSuchConfiguration when unset; DELETE is idempotent (204). - New AdminAction variants admin:SetBucketOnDemandMigration and admin:GetBucketOnDemandMigration, route policy matrix rows, registration and MinIO alias coverage, and a doc row for the extra handler gates. - rustfs-madmin gains on_demand_migration wire types and client methods; golden fixtures under crates/madmin/fixtures/on_demand_migration/ are asserted byte-for-byte by both the handler and the client tests. Anonymous sources still map to a 400 naming source.credentials until the runtime slice adds the credential-less path. * refactor(admin): route on-demand migration handler errors through the s3 facade	2026-09-03 01:58:49 +08:00
Sinan Eldem	b93e7b2355	feat(admin): self-service account management and TOTP two-factor authentication (#6596) ... * feat(madmin): add account and two-factor wire contract Defines the self-service account and MFA API shapes in one place so the console and the `rc` CLI decode identical payloads instead of each carrying its own copy of the contract. `AccountMutability` is part of the contract on purpose: a client needs to know whether the server will accept a password change for this identity before offering the control, rather than discovering it from a rejected request. * feat(s3-types): add IAM identity audit events Adds `iam:Identity:CredentialChanged` and `iam:Identity:AuthChallenge` so account and authentication activity reaches the audit pipeline in its own namespace, the way the KMS events already do. Neither is reachable from a bucket notification config. Two variants for the whole surface rather than one per operation: `mask()` gives every variant its own bit in a `u64`, and the budget is nearly spent (63 of 64 used after this). The per-operation detail lives in `AuditEntry::api.name` and the `iamOperation` tag, which is what a SIEM filters on anyway. Splitting these further needs `mask()` widened first. * feat(iam): add two-factor authentication primitives Implements the state machine behind TOTP enrollment and verification in the IAM domain, so the admin handlers stay HTTP plumbing and the console and CLI drive identical logic. * `totp`: RFC 6238 over the workspace's existing hmac/sha1, pinned to the published Appendix B vectors. SHA-1, 6 digits, 30s: the parameters every mainstream authenticator app implements. Verification returns the matched time step so the caller can burn it. * `recovery`: ten single-use codes, 100 bits each, in a Crockford base32 alphabet without I/L/O/U. Stored as domain-separated SHA-256 digests — a password KDF would have to run once per stored code on every attempt, turning each guess into an attacker-controlled cost, and with uniform 100-bit input there is no dictionary for it to defend against. * `challenge`: stateless HMAC tokens. A TTL cache would be node-local, so a cluster without session affinity would issue on one node and verify on another; nothing here needs replicating. * `record`: two-phase enrollment, replay high-water mark, and lockout. Pending enrollment never gates a login, so a mis-scanned QR cannot lock an operator out, and re-configuring keeps the old factor working until the new one is confirmed. * `store`: one object per identity under `config/mfa/`, a sibling of `config/iam/` so the IAM cache loader's startup walk does not sweep it up. Optimistic `If-Match` writes; deliberately uncached, because a cache would need cluster-wide invalidation to keep the replay mark and the lockout counter honest. * `qr`: server-side rendering, so neither client needs a QR encoder. Enrollment is refused without `RUSTFS_IAM_MASTER_KEY`. A TOTP secret is credential-equivalent, and one written in plaintext could be lifted off a disk — worse than no second factor, because the user believes they have one. IAM identities tolerate a missing master key for backward compatibility; a new feature has no such history to honour. Also adds `IamSys::revoke_sts_sessions_for_parent`, so a credential rotation can invalidate the sessions minted under the old secret. * feat(admin): add self-service account endpoints and the two-factor login gate Adds the account surface (`/v3/account/*`), the second-factor endpoints, the administrative reset (`/v3/user/mfa`), and `PUT /v3/set-user-secret-key`, plus the gate on `AssumeRole`. What the gate covers, and what it deliberately does not: * `AssumeRole` is the only interactive login RustFS has, so it is where a second factor can be enforced. With one enrolled it requires `TokenCode`; without an enrollment the code path is unchanged, so existing deployments are untouched. * A request signed directly with a long-term access key stays ungated. Gating it would break every script and CLI the moment a human enabled 2FA on their own account, and would add no protection: whoever holds the secret key already has full access without presenting a code. This is the division AWS draws; making 2FA meaningful for API access needs an `aws:MultiFactorAuthPresent` policy condition, tracked separately. `SerialNumber`/`TokenCode` are STS's own parameters, so an SDK or script authenticates the same way the console does. `caller_identity` resolves who a request acts as. The console signs with a short-lived STS session, so "the caller" is almost never the key that signed. It reports two separate capabilities: root cannot rotate its secret (a process-wide `OnceLock` that also derives the internode RPC secret) but *can* enroll a second factor — conflating the two would leave the default deployment's console login unprotectable. The self-service routes carry no admin action. Giving them one would be wrong in both directions: it would stop an ordinary user from changing their own password, and let any holder of that action change someone else's. They gate on possession of the credential plus, for the mutations, knowledge of the current secret — a signature only proves a credential was used, so without that a hijacked tab could rewrite the account's credentials or strip its second factor. `set-user-secret-key` exists because the only prior way to change a password was to re-POST the whole user through `add-user`, which rewrote `status` and dropped the policy field — a password reset that silently re-enabled a disabled account. Wrong, replayed and malformed codes are indistinguishable on the wire; the distinction survives only in the audit trail, where no submitted value, secret or code is ever recorded. * test(e2e): cover the two-factor lifecycle and its regressions Unit tests cover the state machine at its edges; only an end-to-end test proves the pieces are wired together and that the existing authentication paths still behave. Asserts, against a real server: enrollment is refused without a master key; the full enroll/activate flow works with a genuine RFC 6238 code; `AssumeRole` refuses without a factor and accepts a valid one; a recovery code works exactly once; a direct SigV4 admin request keeps working with a factor enrolled; `AssumeRole` for an unenrolled identity is unchanged; and a password rotation invalidates the old secret. The test computes TOTP codes itself rather than calling the server's implementation — a shared helper could agree with a bug on both sides. This suite caught a real defect during development: enrollment was refused for root because its *password* is immutable, which would have left the default deployment — an administrator signing into the console as root — unable to protect the one login the feature exists for. * docs(operations): document the two-factor authentication model Records what the second factor protects and what it deliberately does not, because several of the boundaries look like gaps until the alternative is spelled out: why direct SigV4 access stays ungated, why root credentials cannot be rotated at runtime, why secret keys cannot be hashed in an S3 server, and why at-rest protection is mandatory for a TOTP secret but optional for an IAM identity. Also states the limitations plainly, including that GHSA-m77q-r63m-pj89 is unaffected: a holder of the root secret can still forge a session token, 2FA claim included. Placed alongside the other authentication and KMS security documents rather than under a new `docs/security/`, which `.gitignore` excludes. * fix(admin): route the new account handlers through the admin s3 facade Two of the guardrails in the CI "Quick Checks" job rejected the previous commits, so the required check would have gone red as soon as a maintainer approved the workflow run. `check_architecture_migration_rules.sh` requires everything under `rustfs/src/admin` to reach `ECStore` through a domain module rather than the root of `storage_api`. The MFA handler and the two `AssumeRole` signatures now use `storage_api::runtime::ECStore`, which is where the other ten admin handlers already take it from. `check_s3s_footprint.sh` ratchets two counters that new code may not grow: files referencing `s3s` and error-macro invocation lines. This branch added four files and thirty-two lines to them. The ratchet is lower-only and its header forbids raising a baseline to get green, so the construction moves behind the facade instead: `storage_api::s3` now re-exports the request and body types these handlers need and gains an `error` constructor over `S3Error::with_message`. That is the same constructor the macro expands to and the one `handlers/mod.rs`, `rebalance_internal_error` and `invalid_object_lock_configuration` already call, so this is the existing practice rather than a new one, and it keeps the `s3s` dependency in the boundary file the s3gate migration replaces. Every error code and message is carried over unchanged. In `sts.rs` only the call site this branch added is converted; the sixteen that predate it are left alone, because rewriting them would put unrelated churn in a feature PR and push the counter below the baseline it is meant to hold.	2026-08-26 09:35:29 +08:00
houseme	23b17c2d5a	feat(madmin): add a SigV4-signed admin client for heal and scanner APIs (HS-05) (#6166) ... feat(madmin): add a SigV4-signed admin client for heal and scanner APIs The madmin crate held only wire types; automation and mc-style tooling had no way to drive the heal/scanner admin surface without hand-rolled HTTP. Add `AdminClient`, which signs with the same rustfs-signer path the server authenticates (UNSIGNED-PAYLOAD marker, matching RustFS peer admin calls) and wraps: - heal_start / heal_status / heal_stop over POST /rustfs/admin/v3/heal/ (bucket/prefix path params percent-encoded per segment; stop models the server's two cancel branches: token-scoped task status vs path-scoped start-success receipt); - background_heal_status, scanner_status (freshness typed), plus ilm_expiry_status / replacement_recovery_status passthroughs; - a public get_json escape hatch for endpoints not wrapped yet. Wire types follow the madmin-go model (SDK-owned mirrors pinned by round-trip tests): HealOpts with serde defaults so partial settings objects decode, HealScanMode accepting both the numeric and name encodings, and status structs that type the fields operators branch on while flattening unknown nested payloads verbatim so server additions cannot break the client. Errors map to a closed AdminClientError enum (InvalidEndpoint / Transport / HttpStatus with body / Decode). Tests cover wire round-trips, path building, both stop branches, error mapping, and — via a dependency-free raw-TCP test server — that signed requests carry a SigV4 Authorization header, the right method/path/ query, and the expected JSON body. Closes rustfs/backlog#1869 (first increment; single-sourcing the wire structs server-side and an embedded-server e2e roundtrip are noted as follow-ups there). Co-authored-by: heihutu <heihutu@gmail.com>	2026-08-17 15:04:49 +08:00
weisd	15995aae14	feat(admin): complete site replication support (#2346)	2026-03-31 13:01:38 +08:00
houseme	5826396cd0	refactor: Restructure project layout and clean up dependencies (#30) ... This commit introduces a significant reorganization of the project structure to improve maintainability and clarity. Key changes include: - Adjusted the directory layout for a more logical module organization. - Removed unused crate dependencies, reducing the overall project size and potentially speeding up build times. - Updated import paths and configuration files to reflect the structural changes.	2025-07-02 19:33:12 +08:00