mirror of
https://github.com/rustfs/rustfs.git
synced 2026-10-13 08:02:03 +00:00
feat(tls): reload and publish external certificate generations (#8467)
* feat(tls): reload external certificate snapshots coherently Pin projected certificate generations, validate complete TLS material, coordinate server consumers, and invalidate outbound RPC caches coherently. Add externally managed TLS Secret support and rotation guidance. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * feat(tls): publish server certificates through a fenced admin API Add opt-in server-leaf publication with dedicated IAM actions, actual TLS transport checks, immutable generations, durable captured-token CAS and idempotent receipts. Preserve independent trust and client identity sources and distinguish visible publication state from the full serving chain. Validate 64 TLS tests, 41 TLS-runtime tests, the focused IAM policy test, and eight live local API scenarios including least privilege and restart. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(admin): route certificate handler types through S3 facade Use the existing admin S3 facade to keep direct s3s imports within the footprint ratchet. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(admin): include TLS certificate routes in route matrix Account for the registered certificate-state GET and certificate publication POST without changing route behavior or weakening matrix assertions. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(tls): reconcile startup trust before skipping reloads Compare the initialized listener bundle with coherently published outbound material, retain startup trust and client identity when sources disappear, and cover independent startup rotations through real reload passes. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * refactor(tls): reuse pinned generations and prepared listener state Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(tls): preserve startup trust source requirements --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> Co-authored-by: overtrue <anzhengchao@gmail.com>
This commit is contained in:
Hauser
@@ -8,6 +8,8 @@ collection:
|
||||
|
||||
## Operations
|
||||
|
||||
- [External certificate issuance and TLS rotation](operations/tls-certificate-rotation.md) — publication, native reload, trust separation and failure handling.
|
||||
|
||||
- [Architecture guard troubleshooting](operations/architecture-guard-troubleshooting.md) — diagnose repository guard failures.
|
||||
- [Logging governance](operations/logging-governance.md) — broad event audits, migrations, and guardrail changes.
|
||||
|
||||
|
||||
@@ -0,0 +1,204 @@
|
||||
# External certificate issuance and TLS rotation
|
||||
|
||||
Use an independent certificate controller to obtain and renew certificates.
|
||||
RustFS consumes certificate material; it does not register ACME accounts or
|
||||
change DNS records. The independent
|
||||
[rustfs-cert-controller](https://github.com/houseme/rustfs-cert-controller)
|
||||
owns issuance, renewal scheduling and publication. `plugin-runner` starts its
|
||||
container; it does not own certificate schedules or CA credentials. Progress
|
||||
and deployment acceptance are tracked in
|
||||
[backlog #2806](https://github.com/rustfs/backlog/issues/2806).
|
||||
|
||||
## Deployment boundaries
|
||||
|
||||
For a distributed Kubernetes deployment, prefer public TLS at an
|
||||
Ingress/Gateway and private TLS/mTLS between the ingress and RustFS. Public
|
||||
CA certificates cannot cover internal names such as `*.svc.cluster.local`;
|
||||
DNS-01 can cover names under a controlled public domain even when the service
|
||||
is private. A listener
|
||||
that requires mTLS also requires client certificates from browsers and S3
|
||||
clients; selecting a different SNI certificate does not change that policy.
|
||||
|
||||
Native server TLS can consume an externally managed `kubernetes.io/tls`
|
||||
Secret with only `tls.crt` and `tls.key`:
|
||||
|
||||
```yaml
|
||||
mode:
|
||||
distributed:
|
||||
enabled: false
|
||||
standalone:
|
||||
enabled: true
|
||||
ingress:
|
||||
enabled: false
|
||||
tls:
|
||||
enabled: true
|
||||
existingSecret: rustfs-serving-tls
|
||||
reloadInterval: 30
|
||||
```
|
||||
|
||||
Create and renew that Secret using cert-manager or an independently managed
|
||||
publisher. Configure the issuing resource with the actual server DNS/IP SANs
|
||||
and explicit private-key rotation. The chart does not install cert-manager,
|
||||
create an ACME account or grant DNS permissions. For distributed native TLS,
|
||||
the certificate must also cover the configured peer hostnames; explicitly
|
||||
configure independent outbound trust when a private server CA is used.
|
||||
|
||||
`tls.trustSecret` optionally names a separate Secret with an explicitly chosen
|
||||
`ca.crt`, mounted as outbound `public.crt`. It does not enable mTLS or add a
|
||||
public issuing CA to client authentication trust. Do not write a fabricated
|
||||
`ca.crt` into the Secret managed by an ACME issuer.
|
||||
|
||||
`tls.enabled` and `mtls.enabled` are mutually exclusive. Existing mTLS mode
|
||||
keeps its separate server/client certificate Secrets and projects their
|
||||
contents into one read-only directory. The chart enables certificate reload
|
||||
for either mode and disables TLS key logging. Explicit container environment
|
||||
variables can override these settings.
|
||||
|
||||
## Material layout and publication
|
||||
|
||||
`RUSTFS_TLS_PATH` contains `rustfs_cert.pem` and `rustfs_key.pem`. SNI
|
||||
identities can reside in visible domain subdirectories with the same two
|
||||
filenames. Outbound trust uses `public.crt` and `ca.crt`; inbound mTLS client
|
||||
trust prefers `client_ca.crt`, falling back to `ca.crt` only at initial load.
|
||||
The default outbound client identity is `client_cert.pem`/`client_key.pem`.
|
||||
|
||||
Runtime loading validates existing PEM/DER, key pairing, leaf validity and
|
||||
explicit authentication EKUs. Each named SNI identity must match its domain.
|
||||
Chain structure validation does not replace issuer/trust-policy verification
|
||||
by the issuing controller and client PKI verifier.
|
||||
|
||||
Kubernetes mounts must project the entire directory; single-file `subPath`
|
||||
mounts do not receive Secret updates. RustFS pins one physical `..data`
|
||||
generation before loading its material. Separate controllers may update the
|
||||
serving identity and trust independently, so private CA migration still needs
|
||||
an old-plus-new trust overlap.
|
||||
|
||||
For VMs, publish immutable version directories and atomically change the
|
||||
directory pointer after syncing the complete candidate and its containing
|
||||
directories. Keep current/accepted/in-use versions during cleanup. Certbot
|
||||
file symlinks remain compatible, but independent file replacements do not
|
||||
provide an atomic directory generation; key mismatch is rejected, and a
|
||||
publisher should use a complete immutable bundle for coherent publication.
|
||||
|
||||
`RUSTFS_MTLS_CLIENT_CERT`/`RUSTFS_MTLS_CLIENT_KEY` overrides are validated as
|
||||
one identity. Explicit missing, malformed, mismatched, expired or unsuitable
|
||||
override material rejects the update; it does not silently select another
|
||||
identity. Keep default identity files valid when they are present as well.
|
||||
|
||||
## Controller publication API
|
||||
|
||||
For a VM or Docker installation with one default serving certificate, enable
|
||||
reload and configure `RUSTFS_TLS_PUBLISH_DIR` to a dedicated persistent directory
|
||||
owned by the RustFS process with mode `0700`. Keep `RUSTFS_TLS_PATH` configured
|
||||
for the initial serving identity and independent trust/client material. The
|
||||
publication directory must be separate from the original TLS source. A valid
|
||||
initial certificate and trusted management credentials must be provisioned
|
||||
before a controller can connect to the HTTPS API. That bootstrap identity must
|
||||
cover the names the configured issuer will continue to renew; publication
|
||||
preserves existing SAN coverage.
|
||||
|
||||
The controller signs requests using the existing S3 SigV4 authentication:
|
||||
|
||||
- `GET /rustfs/admin/v3/tls/certificate-state` requires
|
||||
`admin:TLSCertificateStatus`.
|
||||
- `POST /rustfs/admin/v3/tls/certificate` requires
|
||||
`admin:TLSCertificatePublish` and a connection terminated by RustFS TLS.
|
||||
|
||||
Grant only these two actions to the controller identity. Read permissions,
|
||||
general configuration permissions and forwarded HTTPS headers do not authorize
|
||||
private-key publication. Keep API requests and credentials out of access-body
|
||||
logs. The API does not add MinIO-compatible aliases.
|
||||
|
||||
The `rustfs.tls-publish/v1alpha1` request fixes `resource_id` to `server-default`
|
||||
and carries an `issuance_id`, the captured `expected_target_revision`,
|
||||
`certificate_pem` and `private_key_pem`. Read the target token before issuance;
|
||||
retain it with that immutable candidate. Repeating the same issuance and payload
|
||||
recovers a lost receipt. A conflict stops that candidate; fetching a new token
|
||||
and replaying the old candidate would defeat this concurrency protection.
|
||||
An HTTP 503 or uncertain network result is retryable with the same issuance,
|
||||
payload and token; HTTP 409 indicates a revision or payload conflict.
|
||||
|
||||
RustFS validates the complete candidate before publishing an immutable leaf
|
||||
generation and syncing its atomic pointer. It does not copy or overwrite CA
|
||||
trust or client credentials. This mode rejects projected Kubernetes sources,
|
||||
multiple SNI identities and configurations that trust the serving leaf as a CA.
|
||||
Use the existing external-file/cert-manager integration for those deployments.
|
||||
|
||||
The state reports the visible target revision and public leaf fingerprint. A
|
||||
successful POST receipt proves the publication was synced; a GET during an
|
||||
uncertain write does not replace that receipt. Retry the identical POST to
|
||||
confirm acceptance. `observed_certificate_fingerprint` is present only when
|
||||
this process's serving reload group has the exact current certificate chain;
|
||||
matching the leaf alone is insufficient. A durable receipt can precede that
|
||||
observation. Check convergence and a fresh TLS handshake on every relevant
|
||||
endpoint. One process's response does not prove another node has converged.
|
||||
|
||||
Use a stable local publication volume with one owner. Captured-token CAS and
|
||||
local process locking do not implement HA leadership, restore ordering or PITR
|
||||
protection. Do not remove active generation directories; automated generation
|
||||
cleanup is not part of this interface.
|
||||
|
||||
## Activation and failure handling
|
||||
|
||||
Native `RUSTFS_TLS_RELOAD_ENABLE` defaults to false. Set it to `1` outside the
|
||||
chart. `RUSTFS_TLS_RELOAD_INTERVAL` defaults to 30 seconds and is clamped to a
|
||||
minimum of 5 seconds.
|
||||
|
||||
Listeners sharing the same configured TLS directory use one reload group.
|
||||
The group prepares TCP and enabled HTTP/3 configurations from the same
|
||||
snapshot before swapping them. A late listener first joins the accepted
|
||||
configuration, even if the current files have become invalid. TCP, QUIC and
|
||||
separate processes still have a short activation window; there is no global
|
||||
atomic certificate switch.
|
||||
TCP and QUIC derive their protocol settings from one prepared certificate
|
||||
resolver and client verifier for that generation. Publication state and retries
|
||||
load each immutable generation once per operation; the next operation validates
|
||||
its disk contents again. Reload uses the verified serving identity with the
|
||||
independently pinned trust/client source, without reopening its certificate and
|
||||
key files. Candidate staging still performs its separate disk validation.
|
||||
|
||||
Unchanged material is skipped. A serving certificate renewal does not advance
|
||||
the outbound generation unless outbound trust or identity actually changes.
|
||||
Outbound state is published as one immutable trust/identity/generation
|
||||
snapshot. Control and bulk RPC cache acquisition checks that generation;
|
||||
existing in-flight calls retain their channels.
|
||||
|
||||
Reload rejects the disappearance of accepted CA/identity source components,
|
||||
SNI entries or accepted DNS/IP SAN coverage. Removing or renaming an identity
|
||||
or trust source requires an explicit configuration change and restart;
|
||||
ordinary renewal must preserve those requirements. Rejected updates retain
|
||||
the last accepted in-memory configuration. New handshakes reject certificates
|
||||
outside their validity interval. Normal renewal does not terminate existing
|
||||
uploads, downloads or sessions.
|
||||
|
||||
The chart uses TCP listener liveness for TLS deployments so certificate expiry
|
||||
does not trigger a restart loop. Readiness continues to check the TLS service.
|
||||
This liveness probe detects listener availability, not complete application
|
||||
health. Keep ingress traffic admission separate from internal peer routing
|
||||
and storage quorum. Do not restart every node because a CA is unavailable.
|
||||
|
||||
There is no automatic persisted private-key fallback in Kubernetes. On
|
||||
restart, invalid explicitly configured TLS material prevents startup. Key
|
||||
compromise, explicit certificate revocation, session invalidation, per-purpose
|
||||
profile coordination and deployment-boundary fencing require the separate
|
||||
operator/control-plane workflow; ordinary reload does not implement them.
|
||||
|
||||
## Verification
|
||||
|
||||
Check a fresh TLS connection to each endpoint and compare the leaf serial,
|
||||
public certificate fingerprint, SANs and expiry. Include HTTP/2, enabled
|
||||
HTTP/3, Console and internal RPC endpoints. Do not infer served certificates
|
||||
from Secret contents or an outbound generation gauge alone.
|
||||
|
||||
Focused local checks:
|
||||
|
||||
```sh
|
||||
RUSTUP_TOOLCHAIN=stable cargo test -p rustfs-tls-runtime
|
||||
RUSTUP_TOOLCHAIN=stable cargo test -p rustfs-protos --lib
|
||||
bash scripts/test_helm_tls_reload.sh
|
||||
```
|
||||
|
||||
Production acceptance also requires real Kubernetes projection, CA/DNS
|
||||
issuance and renewal, outage/expiry, long-transfer continuity, private CA
|
||||
overlap and distributed convergence tests. These are separate from protocol
|
||||
and local loader tests.
|
||||
Reference in New Issue
Block a user