mirror of
https://github.com/rustfs/rustfs.git
synced 2026-09-09 13:46:05 +00:00
唐小鸭
ae9fe62fb1
* fix(sse): resolve bucket default encryption per request PUT and the POST-object/extract path resolved a bucket's default encryption with a hard-coded "no explicit SSE-C" flag, so the default was layered onto a request that already carried an SSE-C header triple and then tripped that request's own mutual-exclusion check. Every bucket with default encryption refused SSE-C single PUTs with 400 InvalidArgument, while CreateMultipartUpload on the same bucket succeeded because it resolves SSE elsewhere. Both call sites now derive the flag from the request headers, as COPY already did. The bucket default's KMS key id was also inherited independently of the effective algorithm, so an explicit AES256 request against an aws:kms default bucket produced a self-contradictory algorithm/key-id pair and was rejected. The key id is now inherited only when the effective algorithm is aws:kms, matching the storage-layer resolver. Refs backlog#2368 B1, B2. * fix(sse): refuse SSE-KMS without a running KMS service A write requesting aws:kms on a node with no KMS service fell back to the node-local SSE-S3 provider: the data key was wrapped with RUSTFS_SSE_S3_MASTER_KEY while the object metadata still recorded aws:kms and the requested KMS key id. The stored object claimed a KMS protection it never had, under a key that was never consulted, and no signal distinguished it from a genuine SSE-KMS object. The managed-encryption path now asks the resolved DEK provider whether it wraps with a node-local master key and refuses SSE-KMS in that case: InvalidRequest when KMS was never configured, ServiceUnavailable when a configured service is not running. The check sits after the per-key authorization gate so an unauthorized caller still receives AccessDenied whatever the KMS runtime state is, and asks the provider rather than a parallel availability signal because the provider is what actually wraps the key. A missing master key no longer answers an SSE-KMS request with an SSE-S3-worded configuration error. The SSE-S3 local fallback is unchanged. Refs backlog#2368 B4. * fix(ecstore): restore and archive tiers in stored coordinates Multipart restore addressed the remote tier in plaintext coordinates while the copy-back reads the stored representation. Each part received a misaligned slice of the remote object whose length still satisfied the range, the hash reader and the completion size check, so the restore reported success and silently replaced the object's bytes. Encrypted and compressed multipart objects were both affected. Restore now accumulates stored part sizes, passes the stored length to the hash reader alongside the plaintext length, and validates against the stored size. The copy-back digests stored bytes, so its computed MD5 is not the object's public ETag. Restore now preserves the object ETag on both the single-part and multipart paths, and gives each restored part its own recorded part ETag rather than the object-level value. Transition also handed the tier the object's SSE headers and its RustFS-wrapped data key as request headers. Any S3 target rejected an SSE-C archive outright, an SSE-KMS archive asked the target to encrypt a second time under a key id it does not own, and the wrapped DEK left the cluster. The archive request now strips every SSE header and encryption marker with the predicate the replication path already uses; the local xl.meta keeps all of it, so read-through and restore are unaffected. Objects restored by an affected release are not detected or repaired retroactively and must be re-restored from the tier. Refs backlog#2368 B3, B5; backlog#2369 P7.1. * fix(rio): lock the v1 nonce layout within a segment Decrypting a v1 segment tried three historical nonce layouts per frame, independently for every frame. The last of them exists for streams written before 1.0.0-alpha.91, which reused a segment's part nonce for every block in it; because block zero's derived nonce equals that base nonce, a frame encrypted at index zero authenticated at any position. An attacker able to rewrite the underlying shards could replay it and have the forged plaintext returned with 200 and an unchanged length. Shard integrity uses a keyed-hash-free checksum, which such an attacker can recompute, so it is not a barrier. A segment now locks onto whichever layout decoded its first non-zero-index frame and rejects any later frame needing a different one. That leaves one residual shape: a stream built purely from repeats of frame zero has no later frame to disagree. New RUSTFS_ENCRYPTION_LEGACY_NONCE_FALLBACK (default true, so pre-alpha.91 objects keep decrypting) drops the third layout entirely when set to false, which closes it. Turning it off refuses pre-alpha.91 objects, so migrate them first by rewriting in place. Refs backlog#2369 P2. * fix(kms): reload a service that failed to start POST /rustfs/admin/v3/kms/reload short-circuited whenever the persisted configuration matched the in-memory one byte for byte. A node whose KMS failed to start keeps that configuration and sits in Error, so the documented recovery call returned "reloaded successfully" while leaving the node down. Peers reached the same path through the reload broadcast, so a cluster that lost Vault during a rolling restart had no working recovery route other than the node-local start endpoint. Reload now short-circuits only for a service that is actually running, and otherwise reconfigures, which starts a service that is not running. The AWS backend also advertised key-version enumeration through kms/status, which its own documentation says it cannot do; the capability and its golden snapshot now say false. Refs backlog#2369 P1, P7.3. * docs: record the SSE and KMS changes for 1.0.0 The Unreleased changelog section carried no entry for any encryption work merged since 1.0.0-rc.5, including three items with operational impact: the config-secret variable whose absence persists secrets in cleartext with only a warning, the v2 frame write switch and its rolling-upgrade constraint, and per-key authorization making a public bucket incompatible with SSE-KMS objects. Adds those plus this batch, including the SSE-KMS refusal as a breaking change with both routes out. Also corrects four places where documentation contradicted the code: the cleanup register still called encrypted range seek opt-in after its default flipped, the Helm README claimed vault_mount_path only applies to Transit while the template also feeds the KV2 mount, the disaster-recovery drill listed bundle contents for backends whose export is refused with 501, and the Chinese README capability table predated most of the feature set. Documents the SSE-S3 local master key as a first-class operational mode with its rotation dead end, and what the v1 frame layout does and does not authenticate. Refs backlog#2369 P5. * fix(kms): classify data-path KMS failures by what the caller can do Only "key not found" and a backend outage were classified; every other KMS failure that reached the S3 data path fell through to 500 InternalError with a generic message. A disabled or pending-deletion key, a denied KMS grant, an encryption-context mismatch, an unsupported algorithm, a credential or timeout failure, and a capability the configured backend does not have all looked identical to a server fault. SDKs therefore applied exponential backoff to configuration errors no retry can fix, and monitoring counted every one of them against the server's own error rate. Unusable-key and request-side failures now answer 400, a denied grant 403, transient backend failures 503, and a missing backend capability 501. Damaged, unreadable, or unknown-format key material keeps its 500: it is a server-side integrity fault, and existing tests pin it. The classifier is deliberately separate from the admin lifecycle mapping, which answers 404 for a missing key because there a key id is the resource being addressed; on the data path it arrives inside a request header or a bucket default. Messages either name what the caller asked for or stay generic, with deployment-side detail left on the error source the way the storage-IO mapping already does. Refs backlog#2368 B6. * fix(kms): track and renew static Vault tokens Token authentication hard-coded "this token carries no lease", so the renewal task never started, the remaining-TTL gauge was never published, and nothing looked wrong. `vault token create` grants a 768-hour TTL by default, so a cluster that had been healthy for a month turned every KMS call into a 403 and could not recover without a restart or a reconfigure. Production configuration validation only rejects the literal dev-token, so an ordinary expiring token reaches a whole cluster. The source now reads `auth/token/lookup-self` at login and adopts what Vault reports. A token with no expiry behaves exactly as before. An expiring renewable one is picked up by the existing renewal loop and renewed at half TTL like every other auth method. An expiring non-renewable one warns with its remaining lifetime and publishes the gauge, so the fail-closed window is visible before it arrives. The probe never fails the login: a policy that omits lookup-self, or a Vault that is briefly unreachable, warns and falls back to exactly the previous behaviour rather than taking down a deployment that works today. The scripted Vault test double answers the lookup out of band so existing scripts keep describing only the protocol under test. Refs backlog#2369 P3. * feat(sse): report SSE-C requests that arrive without TLS An SSE-C request carries the customer's AES key in a request header, so AWS S3 and MinIO both refuse one that did not arrive over TLS. RustFS accepted them on any transport: a plaintext hop hands the key to anyone on the path, and since the object cannot be read without that same key, the exposure lasts as long as the object does. Refusing outright is the correct end state but not a safe default to adopt inside a release window, because the project's own s3-tests and e2e lanes and most staging deployments speak plain HTTP. This release reports instead: each such request increments rustfs_ssec_plaintext_requests_total and logs one warning per process, so an operator can confirm nothing would break before the default flips. RUSTFS_SSE_C_REQUIRE_TLS=true opts into the AWS 400 now. The verdict is per connection rather than per deployment: the layer is built with whether this listener terminated TLS, and additionally accepts an https protocol forwarded by a proxy the trusted-proxy configuration already vetted. It sits beside the rate limiter, after the layer that makes a forwarded protocol trustworthy and after the request context, so a rejection can echo the request id. Refs backlog#2369 P7.2. * fix(kms): say what a node-local backend means for a cluster The Local backend keeps key material on each node's own disk and generates its Argon2id salt per node, so two nodes derive different keys from the same master_key and an object encrypted on one node cannot be decrypted on another. Behind a load balancer that surfaces as intermittent 500s on reads that succeeded moments earlier, with nothing tying the symptom to the cause: the only signal was a generic "development, testing and demos only" positioning warning that says nothing about what actually breaks. Configuring or reconfiguring Local while the deployment is distributed now logs a dedicated event and appends the consequence to the configure response, so the operator who made the change sees it. The product decision to warn rather than refuse is unchanged. Refs backlog#2369 P7.4. * docs: record the remaining SSE and KMS changes for 1.0.0 Adds changelog entries for the KMS data-path status classification, the Vault static-token lease probe, the SSE-C plaintext-transport report and its switch, and the node-local backend warning. Documents two things the backend security guide never stated: that SSE-C belongs on a secure transport, with the counter and switch to plan the change around, and that the Local backend cannot be shared by a multi-node deployment because each node derives different keys from the same master key. Refs backlog#2368 B6; backlog#2369 P3, P5, P7.2, P7.4. * fix(kms): report an unreadable key store as an outage on the S3 path A backend now distinguishes a key store it could not read from a key that is genuinely absent, but the S3 boundary collapsed the first one back onto 500 InternalError through the fallthrough for integrity faults. The distinction was therefore invisible to the client: a temporary key-directory outage looked exactly like a permanently damaged key record, and neither the status nor the metric said the request was worth retrying. An unreadable key store joins the retryable class and answers 503, next to a backend error and a credential failure. Damaged, unreadable or unknown-format key material keeps its 500. Refs backlog#2368 B6; builds on rustfs/rustfs#7470.
Architecture Documentation
Use this when: you need the contract, invariant, or boundary rule that governs a change, and you want the one document that owns it.
Source of truth: the code and the guards. scripts/check_architecture_migration_rules.sh enforces the CI-anchored documents below; scripts/check_doc_paths.sh fails the pre-commit gate when any doc under docs/ cites a repository path that no longer exists.
Two rules keep this directory healthy:
- Durable reference only. One-shot plans, task trackers, dated analyses, status snapshots, and PR-scoped notes do not belong in the repository; keep them in the issue tracker or a local worktree and delete them when the work closes.
- No copies of other sources of truth. Crate lists come from
Cargo.toml, CI steps from.github/workflows/, code structure from the code. Cite a file path plus a symbol name, never a line number, and never paste counts or tables that a command can regenerate.
Every document starts with a **Use this when:** line so an agent can decide in one glance whether to read further. The index below repeats those lines.
CI-anchored core
Required headings and strings in these files are asserted by scripts/check_architecture_migration_rules.sh; rename a heading only together with the guard.
| Document | Use this when |
|---|---|
| crate-boundaries.md | you add a crate dependency, move code across crates, touch a storage_api.rs boundary file, or need the change-type vocabulary the architecture guard enforces |
| runtime-lifecycle.md | moving or reordering anything in rustfs/src/startup_*.rs, changing readiness publication, or touching shutdown ordering |
| readiness-matrix.md | changing what a request surface does before storage or IAM is ready, changing probe semantics, or adding a runtime dependency that readiness must wait for |
| storage-control-data-plane.md | adding a storage API surface, a cluster read model, or a background-service status/reconcile surface, and you need to know which layer owns it |
| global-state-crate-split-plan.md | business logic needs runtime state (object store, endpoints, lock clients, lifecycle state, config) and you must pick the right boundary, or you are evaluating a crate split out of ECStore |
| global-state-inventory.md | you meet a GLOBAL_* static or an OnceLock and need to know whether it is a runtime ownership handle, an owner-local static, or process-global by design |
| ecstore-module-split-plan.md | you add lifecycle or replication logic and need to know which crate it belongs in, plan to move an operation family out of SetDisks, or the guard fails on one of the split rules |
| ecstore-api-facade-inventory.md | you need something from rustfs_ecstore in another crate, you are narrowing a rustfs_ecstore::api facade group, or the guard reports a facade bypass |
| obs-ecstore-dependency-inventory.md | adding, removing, or moving any rustfs_ecstore or rustfs_storage_api reference inside crates/obs |
| compat-cleanup-register.md | you add, review, or remove a temporary compatibility path and need the RUSTFS_COMPAT_TODO marker format and its removal condition |
| overview.md | you need the historical framing of the architecture-migration program or the phase names that other contracts refer to |
Contracts and invariants
| Document | Use this when |
|---|---|
| erasure-coding.md | changing anything under crates/ecstore/src/erasure/, crates/filemeta/, crates/ecstore/src/set_disk/, storage-class or layout code, or any decode, quorum, or heal boundary (normative spec) |
| placement-repair-invariants.md | changing anything that resolves an object to a pool, set, or disk, or that admits scanner or heal work |
| heal-concurrency-model.md | changing heal, PUT/multipart commit, delete, lifecycle expiry, or data-movement code that shares the (bucket, object) commit surface, or asking whether RustFS needs a persistent healing marker |
| unified-object-generation.md | adding or changing anything that fences a commit, scopes a read lease, gates old-directory cleanup, binds prepared pool reads, or settles quota against the current object version |
| ilm-tiering-persistence-contracts.md | changing an ILM transition, tier configuration mutation, manual transition job, tier-delete recovery path, pool decommission, or any code that can create, transfer, or destroy ownership of a remote-tier object |
| decommission-compatibility.md | changing pool decommission or rebalance behavior, its admin API shape, the persisted PoolMeta fields, or how tier free versions move between pools |
| ecstore-layout-boundary.md | touching endpoint expansion, FormatV3, pool/set layout, or moving files between ECStore's internal directories |
| runtime-capability-contracts.md | changing the read-only observability or topology snapshot contracts in rustfs-storage-api, their providers, or the storage_classes payload of GET /rustfs/admin/v4/runtime/capabilities |
| workload-admission-contracts.md | adding a workload class or snapshot provider, or consuming admission state from a background job |
| background-controller-contract.md | adding a status snapshot or reconcile surface for a background service, or being tempted to fold several services into a generic controller |
| background-services-inventory.md | you need one audited background service's desired source, current-status inputs, status surface, and declared side effects |
| scanner-usage-publication.md | changing scanner data-usage cache publication, quota-visible usage snapshots, scanner cycle recovery, or the persisted scanner usage artifacts |
| scanner-usage-authority-decision.md | deciding whether quota admission depends on scanner data usage, removing scanner publication layers, or designing a scanner storage boundary |
| config-model-boundary-adr.md | touching the server-config model (Config, KV, KVS) or its persistence, or asking which crate owns which part of server configuration |
| admin-route-action-snapshot.md | adding, moving, or re-authorizing an admin route and needing to know where the route → handler → AdminAction contract is enforced |
| kms-bulk-rekey-contract.md | changing the bulk envelope re-wrap sweep, its admin endpoints, the re-wrap primitive, or which objects a rekey may touch |
| remote-credential-sealing-adr.md | adding, reading, or persisting a stored remote credential (replication target, remote tier, on-demand migration source), or needing the sealed-envelope format, the mixed-version rules, or the reason this is worth doing in one deployment and not in another |
| tier-stats-contract.md | changing what GET /rustfs/admin/v3/tier-stats returns, adding a tier accounting source, or wiring a metric to a remote tier request |
Support and compatibility matrices (release-facing, keep current)
| Document | Use this when |
|---|---|
| s3-compatibility-matrix.md | writing or checking a user-facing S3 compatibility claim, or moving a Ceph s3tests case between lists |
| s3-tables-support-matrix.md | writing a release note or client-compatibility statement about S3 Tables / Iceberg REST Catalog (cutover procedure: ../operations/s3-tables-cutover-runbook.md) |
| minio-rustfs-router-compatibility.md | a client or mc call that works against MinIO fails against RustFS and you need to know whether the endpoint is missing, stubbed, or deliberately different |
| minio-file-format-compat.md | deciding whether a MinIO drive set, bucket-metadata blob, or SSE object can be read or imported by a given RustFS build, or before touching a listed version anchor |
Operations runbooks live in ../operations/ and testing references in ../testing/README.md.
For per-node HTTP failure ratios and cached storage probe provenance, see S3 write failure diagnostics.