* feat(policy): support object creation operation conditions
Derive the condition from the resolved S3 operation and keep multipart staging separate from object publication. Reject request and claim spoofing and cover policy evaluation and HTTP upload paths.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(protocols): preserve multipart staging authorization
Keep the trusted protocol operation until the shared policy context is built, so FTPS and SFTP honor object creation conditions on multipart staging. Reuse the server-only marker and test real backend decisions for IAM and bucket policies.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(e2e): cover ObjectCreationOperation in STS session policies
Run AWS's conditional-write example 1 as an inline AssumeRole session
policy: object publication needs a request header, while multipart
staging is exempt through s3:ObjectCreationOperation.
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: oc7o <oc7o@sloow.de>
Co-authored-by: overtrue <anzhengchao@gmail.com>
* feat(policy): support s3:if-none-match and s3:if-match condition keys
Bucket, IAM, and STS session policies that enforce conditional writes
with s3:if-none-match or s3:if-match were rejected with "invalid key
name". Accept both keys and populate them from the If-None-Match and
If-Match request headers. A blank header is left out of the condition
values because the write ignores it, so it cannot satisfy a Null check
for a write that runs unconditionally.
* fix(policy): classify conditional write keys as request-derived
s3:if-match and s3:if-none-match are filled from request headers, but
is_server_derived() reported them as server-derived because their names
lack the x-amz- prefix. Add an explicit arm so header input is never
refused for these keys if they join COMMON_KEYS.
* fix(s3): reject non-wildcard If-None-Match on writes
S3 accepts If-None-Match on PutObject, CopyObject and
CompleteMultipartUpload only as "*" and answers other values with 501
NotImplemented. RustFS compared such a value as an ETag, so a write with
If-None-Match: "<other etag>" overwrote the key while satisfying a
policy that requires s3:if-none-match.
Reject ETag values in the three handlers before any I/O. The store-level
check now treats any non-blank If-None-Match as create-only, so a caller
that skips the handler check cannot weaken the precondition. Drop the
CompleteMultipartUpload ETag pre-check that the rejection supersedes.
* test: tighten conditional write policy coverage
Pin trimming and the non-UTF-8 case in the condition values, and drop
the auth test that re-ran policy evaluation already covered by the
policy crate. Run the three policy e2e scenarios on one server, assert
that ETag If-None-Match cannot overwrite under a Null: false policy and
that CompleteMultipartUpload with If-None-Match: * returns 412. Share
the AssumeRole response parser through common.rs.
* test(ecstore): isolate manual delete recovery from workers
Stop fixture background services before seeding recovery journals so the
explicit recovery pass owns the decisions its assertions inspect.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(s3): enforce request-derived conditional write guards
Validate raw If-None-Match headers as well as DTO fields so POST form
uploads cannot skip rejection of unsupported ETag preconditions. Derive
conditional policy keys after identity claims and clear colliding claims
when the actual request has no usable precondition.
Align multipart and pinned Ceph regressions with wildcard-only writes,
retaining If-Match, object-content, version and delete-marker assertions.
Document that 501 is the implementation's unsupported-value response.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* perf(s3): eliminate conditional write request copies
Borrow request headers during policy condition collection and exclude
conditional keys at the identity-claim boundary instead of inserting and
removing them. Preserve existing aliases and multivalue header behavior.
Move the CompleteMultipartUpload input out of its request, compare strong
ETags by reference, and remove the ETag If-None-Match branch made unreachable
by the write guard. Keep both form/HTTP input checks and storage commit
preconditions intact.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(s3): align ignored conditional PUT and COPY cases
Use If-None-Match: * for the existing-object 412 regressions and assert 501
for matching and nonmatching ETag values. Cover both header-only callers
and parsed DTO-plus-header requests in the serial ILM fixtures.
Check object bytes and live version counts after each rejected operation.
Keep the existing test names and ignored-test lane selection unchanged.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: overtrue <anzhengchao@gmail.com>
Co-authored-by: Hauser <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* feat(tls): reload external certificate snapshots coherently
Pin projected certificate generations, validate complete TLS material, coordinate server consumers, and invalidate outbound RPC caches coherently. Add externally managed TLS Secret support and rotation guidance.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* feat(tls): publish server certificates through a fenced admin API
Add opt-in server-leaf publication with dedicated IAM actions, actual TLS
transport checks, immutable generations, durable captured-token CAS and
idempotent receipts. Preserve independent trust and client identity sources
and distinguish visible publication state from the full serving chain.
Validate 64 TLS tests, 41 TLS-runtime tests, the focused IAM policy test,
and eight live local API scenarios including least privilege and restart.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(admin): route certificate handler types through S3 facade
Use the existing admin S3 facade to keep direct s3s imports within the footprint ratchet.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(admin): include TLS certificate routes in route matrix
Account for the registered certificate-state GET and certificate publication POST without changing route behavior or weakening matrix assertions.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(tls): reconcile startup trust before skipping reloads
Compare the initialized listener bundle with coherently published outbound material, retain startup trust and client identity when sources disappear, and cover independent startup rotations through real reload passes.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* refactor(tls): reuse pinned generations and prepared listener state
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(tls): preserve startup trust source requirements
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>
Reuse the existing UTF-8 byte minimum for credential creation so long service account secrets remain intact. Keep generated secrets at 40 bytes and document older receiver and import constraints.
* feat(iam): expose stored bucket tags to OPA
* fix(ci): lint bucket tag fixtures and register new e2e tests
* fix(ecstore): restore strict Clippy compatibility on Rust 1.99
Use try_update without changing atomic ordering or overflow behavior. Keep
recursive storage futures boxed once at each frame and remove the redundant
async-recursion macro, including its non-recursive SQL planner use. Remove
needless closure borrows and orphaned dependency entries.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
(cherry picked from commit 537c15277a)
* fix(deps): replace yanked yoke-derive release
(cherry picked from commit b54b32b8a9)
* perf(iam): borrow unchanged OPA bucket tag conditions
Reuse clean conditions for bucketless and untagged OPA decisions. Keep the existing clone-and-sanitize path for tagged requests, reserve enrichment capacity, and preserve authoritative lookup errors. Cover borrowed input, reserved namespaces, immutable request state, and both paths in the OPA input regression.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(iam): clarify OPA bucket policy fallback semantics
Document that an OPA tag mismatch remains an implicit IAM denial which a bucket-policy Allow can supplement. Extend the existing S3 contract test to contrast a normal OPA denial with corrupt bucket metadata, whose lookup error must abort the same Allow fallback.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ci): isolate durable progress fixtures from competing IO
Reserve the nextest execution budget for schedule, streaming lock, retirement, MRF checkpoint/crash, and native scanner progress scenarios. Preserve their internal concurrency, existing deadlines, typed failure assertions, and zero-retry policy in default and CI profiles.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: rjregenold <214054+rjregenold@users.noreply.github.com>
Co-authored-by: Hauser <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>
* fix(iam): require an explicit permission for force-delete
A force-delete header no longer inherits s3:* or consoleAdmin. Bucket
force-delete requires s3:ForceDeleteBucket whenever the header is present,
and recursive object force-delete requires s3:ForceDeleteObject. A plain
delete keeps the existing checks.
Co-authored-by: RustFS <hello@rustfs.com>
Signed-off-by: loverustfs <155562731+loverustfs@users.noreply.github.com>
* test(e2e): keep force-delete header names static
The bucket force-delete helper must pass a static header name into the
SDK request mutator.
Co-authored-by: RustFS <hello@rustfs.com>
Signed-off-by: loverustfs <155562731+loverustfs@users.noreply.github.com>
* test(e2e): move the force-delete header into the request mutator
The SDK request customizer requires a static header name owned by the
closure.
Co-authored-by: RustFS <hello@rustfs.com>
Signed-off-by: loverustfs <155562731+loverustfs@users.noreply.github.com>
* fix(iam): keep force-delete out of NotAction grants
NotAction now uses plain wildcard matching, so NotAction "s3:*" still
excludes force-delete. An Allow statement grants s3:ForceDeleteObject or
s3:ForceDeleteBucket only when its Action list names the action; a
NotAction-only Allow never does. The rule applies to both IAM and bucket
policy statements.
Also build the invalid-header errors with S3Error::with_message to keep
the s3s footprint at its baseline, and fix a clippy single_match.
---------
Signed-off-by: loverustfs <155562731+loverustfs@users.noreply.github.com>
Co-authored-by: Hauser <housemecn@gmail.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>
* feat(ecstore): add on-demand migration bucket config model
Introduce OnDemandMigrationConfig (deny_unknown_fields, version 1) with typed validation, credential redaction, a secret-free Debug impl, and the OnceLock publish hook the runtime registers into. Exported through the api facade.
* feat(ecstore): persist on-demand migration config in bucket metadata
Store the config as a RustFS extension entry (on-demand-migration.json) with its update time in .metadata.bin, add the typed BucketMetadataSys accessor, and publish the config through the hook on every cache-install path alongside the durability sync.
* refactor(ecstore): extract shared remote S3 client builder
Move the aws_sdk_s3 client construction out of bucket_target_sys into
bucket/remote_s3_client.rs: endpoint assembly, credential provider,
path-style selection, custom CA / skip-TLS transports and the outbound
SSRF gate now build from a neutral RemoteS3EndpointSpec so replication
targets and the upcoming on-demand migration source client share one
policy. Replication builds its client through From<&BucketTarget>; the
gate keeps its relaxed semantics (private allowed, loopback only behind
RUSTFS_REPLICATION_ALLOW_LOOPBACK_TARGET) verbatim. The builder also
gains optional connect/read timeouts and a User-Agent suffix
interceptor, both unset for replication.
Refs rustfs/backlog#2149
* feat(ecstore): add on-demand migration SourceClient
Add bucket/on_demand_migration/source_client.rs on top of the shared
remote S3 builder: HEAD, ranged streaming GET, ListObjectsV2 with
source-prefix mapping, GetObjectTagging and an admin probe. Every request
carries the x-rustfs-/x-minio-source-proxy-request anti-loop markers and
a RustFS-OnDemandMigration/<version> User-Agent suffix; SSE-C source
objects are rejected as unsupported. SourceError classifies SDK failures
(not found, access denied, throttled, timeout, connect, server error)
with retryability and a stable metrics label. Debug output redacts
credentials.
Refs rustfs/backlog#2149
* docs(operations): point outbound policy at shared remote S3 client builder
* chore: integrate ODM-01 and ODM-02 as B1 base (fix facade merge)
* feat(admin): add on-demand migration bucket admin API
Add the management plane for On-Demand Migration (ODM-07,
rustfs/backlog#2154): PUT/GET/DELETE /v3/on-demand-migration/{bucket},
PUT ?dry-run=true, and a GET .../status skeleton.
- PUT authorizes SetBucketOnDemandMigration, checks the bucket, the
RUSTFS_ON_DEMAND_MIGRATION_ENABLED switch and the license, validates the
ODM-01 config against local endpoints and replication targets, probes the
source with SourceClient::probe(), then persists through the incarnation
gate and asks peers to reload. Responses carry the redacted config and a
probe summary; probe failures name only the error class.
- GET answers 404 NoSuchConfiguration when unset; DELETE is idempotent (204).
- New AdminAction variants admin:SetBucketOnDemandMigration and
admin:GetBucketOnDemandMigration, route policy matrix rows, registration
and MinIO alias coverage, and a doc row for the extra handler gates.
- rustfs-madmin gains on_demand_migration wire types and client methods;
golden fixtures under crates/madmin/fixtures/on_demand_migration/ are
asserted byte-for-byte by both the handler and the client tests.
Anonymous sources still map to a 400 naming source.credentials until the
runtime slice adds the credential-less path.
* refactor(admin): route on-demand migration handler errors through the s3 facade
Backlog#1845 step 8 prerequisite. policy::error::Error carried six variants with zero construction and zero match sites anywhere in the workspace: ErrCredMalformed, CredNotInitialized, NoAccessKey, InvalidToken, InvalidAccessKey, InvalidExpiration. Their only reference was the grouped fallthrough arm in iam's From<policy::error::Error>, whose own dead same-name twins were already removed in backlog#1831 (#6030).
Delete the variants and their display-message test rows; the iam mapping's grouped arm shrinks from eight variants to the two that are actually produced (InvalidServiceType from service_type parsing, JWTError via #[from]). This clears the way for folding the remaining 25-arm hand-written mapping (backlog#1845 step 8).
Ref rustfs/backlog#1845
backlog#1823 step 10, batch 1 of the repo-wide item-allow sweep. 227 bare #[allow(dead_code)] remain across 83 files; this takes the 19 in utils, notify, checksums, policy, keystone and trusted-proxies, which are small enough to verify end to end.
Removing all 19 first, before writing any reason, matters: 8 of them suppress nothing. Every allow in utils, one in policy and three in notify sit on items that are publicly reachable, so dead_code never applied to them — the same shape as the swift module and kms's dek.rs. Writing a reason onto a no-op allow would dress noise up as considered judgement, so those are simply deleted.
Three items are genuinely dead and go with their allows: notify's new_target_id_set, the AWS metadata fetcher's get_metadata_token, and policy's empty `pub struct Value;`, none of which is referenced anywhere in the tree.
The remaining eight keep an allow, now saying why the item survives rather than who calls it. Two are exercised only by their own crate's tests (checksums' MD5_HEADER_NAME, policy's is_match_as_pattern_prefix). Four are fields written but never read back: keystone's verify_ssl, parsed from config after the reqwest client is already built; keystone's client handle, which keeps the Keystone client alive for the mapper's lifetime; the AWS IMDS endpoint, kept beside the client while requests build their own URLs; and notify's rules_map, whose own comment retains it for snapshot-time judgements no code performs.
checksums' Md5 needed the most care. Crc32, Sha256 and seven others each have an arm in ChecksumAlgorithm::into_impl, and Md5 has none, which reads like a missing algorithm. It is not: ChecksumAlgorithm has no Md5 variant at all. S3 carries Content-MD5 as its own header, separate from the x-amz-checksum-* family, and this impl exists so both paths share the Checksum trait. The reason records that, so the next reader does not re-derive it.
One measurement note for anyone continuing this sweep: cargo does not re-emit warnings for cached compilations, so a per-crate loop of `cargo check -p <crate>` under-reports. checksums showed zero that way while actually carrying three. Touch the sources and check the crates in one invocation, then attribute by path.
Verification: the six crates are warning-free under cargo check --tests; clippy --lib --tests -D warnings clean; cargo nextest run 1096 passed; make pre-commit exit 0.
Ref rustfs/backlog#1823 (step 10).
The policy crate's Go path.Clean port and rustfs-utils' Windows-aware clean look like duplicates but are not interchangeable: S3 ARN/resource matching must treat backslashes as object-name data, never as separators, so adopting the utils version would change policy evaluation semantics on Windows — a security-adjacent behavior change. Record that judgment as bidirectional do-not-merge notes on both implementations, per the issue's adversarial ruling.
Comment-only change.
Ref rustfs/backlog#1833 (PR7).
Deletes three blocks of commented-out code that can never be revived: seven dead tokio::tests plus ~20 commented use statements at the tail of ecstore's store/list_objects.rs test module (all hardcoded to a developer's personal machine path), the commented test_extract_claims in policy/utils.rs, and the commented-out pre-strum AdminAction enum draft in policy/action.rs (the live enum below it is untouched).
Also rewords two doc comments on the bucket-metadata inline-data interop test to drop the personal attribution while keeping the technical content, so the corpse check (rg weisd) now returns zero across the repo.
Ref rustfs/backlog#1836 (PR2).
* refactor(time): migrate audit and notify timestamps to jiff
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(ecstore): initialize heal walk decode error
Co-Authored-By: heihutu <heihutu@gmail.com>
* refactor(targets): parse MySQL event time with jiff
Preserve MySQL DATETIME(6) wall-time formatting for RFC3339 eventTime values while removing the direct chrono dependency from rustfs-targets.
Co-Authored-By: heihutu <heihutu@gmail.com>
* chore(deps): prune unused workspace dependencies
Apply cargo shear --fix to remove unused path-clean and s3select-api tempfile entries after the scoped jiff migration.
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(ecstore): remove duplicate heal walk decode error init
Remove the duplicate decode_error field from the heal walk test collector initializer so lib-test clippy compiles on CI.
Co-Authored-By: heihutu <heihutu@gmail.com>
* refactor(policy): emit OPA timestamps with jiff
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
* feat(policy): add built-in KMS role policies
KMSKeyAdministrator, KMSKeyUser and KMSAuditor ship as canned identity
policies so operators can express KMS role separation without hand-writing
the resource grammar. They grant only kms actions, so they compose with an
existing data-plane policy, and none of them confers kms:Configure,
kms:ServiceControl, kms:ClearCache, kms:Backup or kms:Restore.
* docs(kms): document per-key KMS authorization and the role templates
* test(kms): add an end-to-end negative authorization matrix
Covers the admin and SSE-KMS planes for a wrong identity, a wrong key, a
wrong action and an explicit Deny, each preceded by a positive control so a
denial cannot be an unpropagated policy. SSE-S3 and unencrypted objects are
asserted to stay exempt.
* test(replication): pin the SSE-KMS contract with per-key authorization on
The replication worker carries no request identity, so it must stay exempt
from SSE-KMS key authorization. Running the existing contract with the
switch enabled makes a regression in that exemption visible here.
* feat(kms): add admin endpoints for key description and tag updates
Wire the KMS key metadata updates landed at the service layer to the admin
API: POST /v3/kms/keys/{update-description,tag,untag}. Each endpoint gates on
a dedicated KMS action scoped to the key its body names, records a handler-owned
audit entry for both the authorization denial and the outcome, and maps a
backend that cannot update key metadata to 501 rather than 404.
Refs rustfs/backlog#1586 (part of rustfs/backlog#1562)
* fix(admin): audit metadata attempts refused by an unavailable KMS
* test(admin): register the new KMS metadata routes in the matrix
* feat(kms): add backup and restore admin API
Wires the merged KMS backup contract, Local export and Local restore into
the admin API: export a sealed bundle, run a zero-write restore preflight,
execute a confirmed restore, roll an interrupted restore back, and report
subsystem readiness.
- Dedicated kms:Backup / kms:Restore actions, recorded in the admin route
matrix. Neither is reachable through any other KMS action.
- Restore requires two independent confirmations: an echo of the bundle
manifest's backup id, and an explicitly named conflict policy (the
default never writes).
- The backup KEK comes from the environment and is refused when it reuses
a secret of the configured backend, compared both as the literal value
and as raw key bytes.
- No endpoint accepts a path: bundles are addressed by a validated name
under a configured root, and the restore target is always the server's
own configured key directory.
- Bundles now carry a sanitized configuration artifact built as an
allowlist projection, so a future backend credential field cannot leak
into a bundle by default. Restore verifies it and never applies it.
- Audit entries go through the existing KMS admin wiring and carry
identifiers only.
* test(kms): pin the backup admin API gates
Fixes the test KEK to a real 32-byte value and drives the export refusal
from the configured backend rather than from the handle that happens to
be available, so a Local handle cannot export on behalf of a backend
whose material RustFS does not own.
overtrue