fix(ecstore): start with unresolved Kubernetes peers (#5460)

* fix(ecstore): start with unresolved Kubernetes peers

* fix(ecstore): infer Kubernetes endpoint identity safely

* fix(ecstore): fail closed on unsafe format migration

* fix(ecstore): reject poisoned format heal candidates

* fix(ecstore): reject unsafe legacy migration outliers

* fix(ecstore): resume interrupted format migrations

* fix(ecstore): preserve Kubernetes startup compatibility
This commit is contained in:
cxymds
2026-07-30 11:14:38 +08:00
committed by GitHub
parent 2e5cef513f
commit 67904a6c18
30 changed files with 3618 additions and 461 deletions
+68 -14
View File
@@ -47,6 +47,60 @@ without manual intervention:
If you previously set `replicaCount=16` and now want a different topology,
set both `replicaCount` and `drivesPerNode` explicitly.
For distributed deployments that use the chart-generated
`RUSTFS_VOLUMES`, `localEndpointHost.autoInject` defaults to automatic
selection. Without `secret.existingSecret`, the chart injects a private
Downward API variable, `RUSTFS_CHART_POD_NAME`, and uses it to build the pod's
fully qualified hostname in
`RUSTFS_LOCAL_ENDPOINT_HOST`. RustFS can then identify local drives without
waiting for every peer's DNS record or TCP listener. A user-defined `POD_NAME`
is preserved and does not interfere with the private chart variable. Setting
`RUSTFS_STARTUP_TOPOLOGY_WAIT_MODE` explicitly to `bounded`, `fail-fast`,
`failfast`, or `strict` also keeps legacy DNS-based locality discovery. A
dynamically sourced wait mode keeps the legacy path because its value cannot be
validated while rendering. Otherwise, Kubernetes auto-detection selects
orchestrated startup when RustFS consumes the generated anchor.
An existing Secret is opaque to the chart and may historically contain more
than credentials, so the chart does not inject an anchor whenever
`secret.existingSecret` is set. In Kubernetes auto/orchestrated mode, RustFS
then derives a DNS-free identity from the kernel hostname when exactly one
domain endpoint at the server port has the same full hostname or first label.
All-IP topologies retain direct IP locality detection. A domain topology with
zero matches retains legacy DNS locality discovery; implicit auto mode bounds
that compatibility path by `RUSTFS_STARTUP_TOPOLOGY_WAIT_TIMEOUT` (180 seconds
by default). Multiple matching candidates remain an error. Set
`RUSTFS_LOCAL_ENDPOINT_HOST` explicitly to avoid DNS discovery. For a
credentials-only Secret, set
`localEndpointHost.autoInject=true` to add the chart anchor without changing
the historical ConfigMap-then-Secret `envFrom` precedence. If injection is
explicitly enabled with an incompatible hidden `RUSTFS_VOLUMES`,
`RUSTFS_ADDRESS`, or `RUSTFS_STARTUP_TOPOLOGY_WAIT_MODE`, RustFS also fails
during endpoint construction; it does not silently fall back to a different
topology.
When `config.rustfs.volumes` is set explicitly, the chart does not infer a
local endpoint identity. RustFS applies the same kernel-hostname inference to
custom domain topologies in Kubernetes auto/orchestrated mode; aliases that do
not match the Pod hostname retain legacy DNS locality, with the bounded auto
fallback described above. They may provide `RUSTFS_LOCAL_ENDPOINT_HOST`
through `extraEnv` for DNS-free startup. An explicit `RUSTFS_VOLUMES`, explicit
`RUSTFS_LOCAL_ENDPOINT_HOST`, bounded/dynamic or unrecognized startup mode, or
`localEndpointHost.autoInject=false`, also disables chart injection. A
`RUSTFS_ADDRESS` override alone does not disable it; the effective address and
generated topology must agree on the endpoint port. Custom anchor-based
configurations must resolve to `orchestrated` startup mode and must not receive
a conflicting mode from an `envFrom` source.
`startupWaitTimeoutSeconds` is retained for values-file compatibility but is
deprecated and ignored.
Historical `RUSTFS_STARTUP_TOPOLOGY_RETRY_MAX_DELAY` values of `0` or `0ms`
are replaced with the safe default retry cap instead of causing a busy loop or
blocking a direct upgrade.
Upgrade the chart and RustFS image together. An older image that does not
recognize `RUSTFS_LOCAL_ENDPOINT_HOST` retains its previous DNS-based startup
behavior.
---
# Parameters Overview
@@ -57,6 +111,7 @@ set both `replicaCount` and `drivesPerNode` explicitly.
| affinity.podAntiAffinity.enabled | bool | `true` | |
| affinity.podAntiAffinity.topologyKey | string | `"kubernetes.io/hostname"` | |
| clusterDomain | string | `"cluster.local"` | Kubernetes cluster DNS domain used to build in-cluster FQDNs for `RUSTFS_VOLUMES` (distributed mode) and mTLS server certificate SANs. Override for clusters not using the default `cluster.local`. Provide the DNS root only, without a `svc.` prefix or leading/trailing dots. |
| localEndpointHost.autoInject | bool or null | `null` | Automatically inject `RUSTFS_LOCAL_ENDPOINT_HOST` for chart-generated distributed topologies unless `secret.existingSecret` is set. Use `true` for a credentials-only existing Secret or `false` to preserve legacy DNS locality explicitly. |
| commonLabels | object | `{}` | Labels to add to all deployed objects. |
| config.rustfs.address | string | `":9000"` | |
| config.rustfs.console_address | string | `":9001"` | |
@@ -66,7 +121,7 @@ set both `replicaCount` and `drivesPerNode` explicitly.
| config.rustfs.obs_environment | string | `"development"` | |
| config.rustfs.obs_log_directory | string | `"/logs"` | Log directory inside the RustFS container. Set to `""` to disable log PVCs and mounts. |
| config.rustfs.region | string | `"us-east-1"` | |
| config.rustfs.volumes | string | `""` | |
| config.rustfs.volumes | string | `""` | Explicit distributed volume topology. When empty, the chart generates the topology and normally injects `RUSTFS_LOCAL_ENDPOINT_HOST`; custom topologies must configure local endpoint identity explicitly when needed. |
| config.rustfs.log_rotation.size | int | `"100"` | Default log rotation size mb for rustfs. |
| config.rustfs.log_rotation.time | string | `"hour"` | Default log rotation time for rustfs. |
| config.rustfs.log_rotation.keep_files | int | `"30"` | Default log keep files for rustfs. |
@@ -107,7 +162,7 @@ set both `replicaCount` and `drivesPerNode` explicitly.
| config.rustfs.kms.vault.vault_token | string | `""`| The vault token. Rendered into a dedicated Secret (`<fullname>-kms-secret`), never into the ConfigMap. |
| config.rustfs.kms.vault.vault_mount_path | string | `"transit"`| The vault mount path, only works if `vault_backend` equals `vault-transit` . |
| config.rustfs.kms.vault.default_key | string | `"transit"`| The master key id for RustFS. |
| extraEnv | map | `[]` | Extra environment variables for RustFS container. |
| extraEnv | list | `[]` | Extra environment variables for the RustFS container. An explicit `RUSTFS_LOCAL_ENDPOINT_HOST` or `RUSTFS_VOLUMES`, or a bounded, dynamic, or unrecognized startup mode, disables generated anchor injection. `POD_NAME` and `RUSTFS_ADDRESS` remain independent overrides. |
| extraVolumes | list | `[]` | Extra volumes to add to the pod spec. Supported in both standalone (Deployment) and distributed (StatefulSet) modes. |
| extraVolumeMounts | list | `[]` | Extra volume mounts to add to the RustFS container. Supported in both standalone (Deployment) and distributed (StatefulSet) modes. |
| containerSecurityContext.capabilities.drop[0] | string | `"ALL"` | |
@@ -190,7 +245,7 @@ uer. `ClusterIssuer` or `Issuer`. |
| resources.limits.memory | string | `"512Mi"` | |
| resources.requests.cpu | string | `"100m"` | |
| resources.requests.memory | string | `"128Mi"` | |
| secret.existingSecret | string | `""` | Use existing secret with a credentials. |
| secret.existingSecret | string | `""` | Use an existing Secret. Automatic endpoint-anchor injection is disabled because the Secret is opaque; set `localEndpointHost.autoInject=true` only after confirming it contains credentials rather than runtime topology, address, or startup-mode overrides. |
| secret.rustfs.access_key | string | `"rustfsadmin"` | RustFS Access Key ID |
| secret.rustfs.secret_key | string | `"rustfsadmin"` | RustFS Secret Key ID |
| service.type | string | `"ClusterIP"` | |
@@ -202,6 +257,7 @@ uer. `ClusterIssuer` or `Issuer`. |
| serviceAccount.automount | bool | `true` | |
| serviceAccount.create | bool | `true` | |
| serviceAccount.name | string | `""` | |
| startupWaitTimeoutSeconds | int | `300` | Deprecated and ignored; retained for values-file compatibility. |
| storageclass.dataStorageSize | string | `"256Mi"` | The storage size for data PVC. |
| storageclass.logStorageSize | string | `"256Mi"` | The storage size for logs PVC. |
| storageclass.name | string | `"local-path"` | The name for StorageClass. |
@@ -291,20 +347,17 @@ Notes:
* **Pools are append-only.** The list index determines the StatefulSet name —
never remove or reorder entries. Retire a pool with
`rc admin decommission` before removing it from the list.
* During the expansion rollout, pods restart until every pod of every pool is
resolvable — the server refuses to start with unresolvable peers, so expect
a few crash/restart cycles before the cluster converges. This is harmless.
* With chart-generated volumes, each pod receives an explicit local endpoint
identity. An unavailable peer no longer blocks a pod from reaching RustFS's
own startup and quorum checks.
* After the cluster converges, run `rc admin rebalance start <alias>` to
spread existing objects across the new pool.
* Pod anti-affinity in pool mode is scoped per pool and **preferred**
(soft), not required: two pools can share nodes, and each pool's own pods
spread across distinct nodes when capacity allows. Soft affinity is
load-bearing for in-place expansion — with required rules, the
not-yet-rolled pods of the existing pool block the new pool's pods from
their nodes while the rolled pods crash on the unresolvable (Pending)
peers, deadlocking the rollout on any cluster with fewer nodes than the
total pod count. Single-pool deployments (`pools.enabled=false`) keep the
chart's existing required anti-affinity unchanged.
spread across distinct nodes when capacity allows. Preferred affinity keeps
additional pools schedulable when the cluster has fewer nodes than total
pods. Single-pool deployments (`pools.enabled=false`) keep the chart's
existing required anti-affinity unchanged.
* The PodDisruptionBudget spans all pools: with the default
`pdb.maxUnavailable: 1`, at most one pod of the whole cluster may be
evicted at a time. This is deliberately conservative — quorum safety
@@ -315,7 +368,8 @@ Notes:
## Requirement
* Helm V3
* RustFS >= 1.0.0-alpha.69
* The RustFS image from the same release as the chart. If `image.rustfs.tag`
is overridden, that image must support `RUSTFS_LOCAL_ENDPOINT_HOST`.
Due to the traefik and ingress has different session sticky/affinity annotations, and rustfs support both those two controller, you should specify parameter `ingress.className` to select the right one which suits for you.
+10 -8
View File
@@ -322,21 +322,23 @@ Render RUSTFS_SERVER_DOMAINS
{{- join "," $domains -}}
{{- end -}}
{{/* Render probe command for liveness and readiness
{{/* Render an mTLS probe command
*/}}
{{- define "rustfs.probeCommand" -}}
{{- $endpoint_port := .Values.service.endpoint.port | default 9000 -}}
{{- $console_port := .Values.service.console.port | default 9001 -}}
{{- $root := .root -}}
{{- $endpointPath := .endpointPath -}}
{{- $endpoint_port := $root.Values.service.endpoint.port | default 9000 -}}
{{- $console_port := $root.Values.service.console.port | default 9001 -}}
{{- $args := "-skf" -}}
{{- if and .Values.mtls.enabled -}}
{{- $args = printf "%s --cert %s --key %s" $args .Values.mtls.clientCertPath .Values.mtls.clientKeyPath -}}
{{- if and $root.Values.mtls.enabled -}}
{{- $args = printf "%s --cert %s --key %s" $args $root.Values.mtls.clientCertPath $root.Values.mtls.clientKeyPath -}}
{{- end -}}
- /bin/sh
- -c
- |
curl {{ $args }} https://127.0.0.1:{{ $endpoint_port }}/health/ready && \
curl {{ $args }} https://127.0.0.1:{{ $endpoint_port }}{{ $endpointPath }} && \
curl {{ $args }} https://127.0.0.1:{{ $console_port }}/rustfs/console/health
{{- end -}}
@@ -350,7 +352,7 @@ livenessProbe:
{{- if .Values.mtls.enabled }}
exec:
command:
{{ include "rustfs.probeCommand" . | nindent 6 }}
{{ include "rustfs.probeCommand" (dict "root" . "endpointPath" "/health") | nindent 6 }}
{{- else }}
httpGet:
path: /health
@@ -369,7 +371,7 @@ readinessProbe:
{{- if .Values.mtls.enabled }}
exec:
command:
{{ include "rustfs.probeCommand" . | nindent 6 }}
{{ include "rustfs.probeCommand" (dict "root" . "endpointPath" "/health/ready") | nindent 6 }}
{{- else }}
httpGet:
path: /health/ready
+43 -53
View File
@@ -1,12 +1,37 @@
{{- $logDir := .Values.config.rustfs.obs_log_directory }}
{{- $logDirEnabled := ne $logDir "" }}
{{- $poolsEnabled := and .Values.pools .Values.pools.enabled }}
{{- $generatedVolumes := empty .Values.config.rustfs.volumes }}
{{- $localEndpointHostAutoInject := empty .Values.secret.existingSecret }}
{{- with .Values.localEndpointHost }}
{{- if and (hasKey . "autoInject") (ne .autoInject nil) }}
{{- if not (kindIs "bool" .autoInject) }}
{{- fail "localEndpointHost.autoInject must be a boolean or null" }}
{{- end }}
{{- $localEndpointHostAutoInject = .autoInject }}
{{- end }}
{{- end }}
{{- $injectLocalEndpointHost := and $generatedVolumes $localEndpointHostAutoInject }}
{{- range $env := .Values.extraEnv -}}
{{- $name := default "" $env.name -}}
{{- if eq $name "RUSTFS_VOLUMES" -}}
{{- $injectLocalEndpointHost = false -}}
{{- else if eq $name "RUSTFS_LOCAL_ENDPOINT_HOST" -}}
{{- $injectLocalEndpointHost = false -}}
{{- else if eq $name "RUSTFS_STARTUP_TOPOLOGY_WAIT_MODE" -}}
{{- if hasKey $env "value" -}}
{{- $mode := lower (trim (toString $env.value)) -}}
{{- if and (ne $mode "") (ne $mode "auto") (ne $mode "orchestrated") -}}
{{- $injectLocalEndpointHost = false -}}
{{- end -}}
{{- else -}}
{{- $injectLocalEndpointHost = false -}}
{{- end -}}
{{- end -}}
{{- end -}}
{{- if and .Values.mode.distributed.enabled (le (int .Values.replicaCount) 1) -}}
{{- fail "Distributed mode requires replicaCount >= 2" -}}
{{- end -}}
{{- if and .Values.mode.distributed.enabled (le (int .Values.startupWaitTimeoutSeconds) 0) -}}
{{- fail "startupWaitTimeoutSeconds must be greater than 0 in distributed mode" -}}
{{- end -}}
{{- if .Values.mode.distributed.enabled }}
{{- $pools := include "rustfs.pools" . | fromJsonArray }}
@@ -74,13 +99,10 @@ spec:
{{- if $.Values.affinity.podAntiAffinity.enabled }}
podAntiAffinity:
{{- if $poolsEnabled }}
{{- /* Pool-scoped and PREFERRED, not required: during in-place
expansion the not-yet-rolled pods' required rules block new
pods from their nodes, while the new pods' DNS must resolve
before the roll can proceed - required rules deadlock
expansion whenever the cluster has fewer nodes than total
pods. Soft anti-affinity converges (the scheduler still
spreads when capacity allows). */}}
{{- /* Pool-scoped and PREFERRED, not required: additional pools
must remain schedulable when the cluster has fewer nodes
than total pods. The scheduler still spreads them when
capacity allows. */}}
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
@@ -127,10 +149,6 @@ spec:
env:
- name: DRIVES_PER_NODE
value: {{ $drivesPerNode | quote }}
- name: ENDPOINT_PORT
value: {{ $.Values.service.endpoint.port | quote }}
- name: STARTUP_WAIT_TIMEOUT_SECONDS
value: {{ $.Values.startupWaitTimeoutSeconds | quote }}
command:
- sh
- -c
@@ -146,44 +164,6 @@ spec:
mkdir -p /mnt/rustfs/logs
chmod 755 /mnt/rustfs/logs
{{- end }}
deadline=$(($(date +%s) + STARTUP_WAIT_TIMEOUT_SECONDS))
wait_for_peer() {
description=$1
shift
until "$@" >/dev/null 2>&1; do
if [ "$(date +%s)" -ge "$deadline" ]; then
echo "Timed out after ${STARTUP_WAIT_TIMEOUT_SECONDS}s waiting for ${description}" >&2
exit 1
fi
sleep 2
done
}
echo "Waiting for distributed peer DNS records"
{{- range $peerPool := $pools }}
{{- range $i := until (int $peerPool.replicaCount) }}
wait_for_peer "DNS for {{ $peerPool.fullname }}-{{ $i }}" \
nslookup "{{ $peerPool.fullname }}-{{ $i }}.{{ include "rustfs.fullname" $ }}-headless.{{ $.Release.Namespace }}.svc.{{ include "rustfs.clusterDomain" $ }}"
{{- end }}
{{- end }}
ordinal=${HOSTNAME##*-}
echo "Waiting for earlier distributed peers to listen on port ${ENDPOINT_PORT}"
{{- range $peerPool := $pools }}
{{- if lt (int $peerPool.index) (int $pool.index) }}
{{- range $i := until (int $peerPool.replicaCount) }}
wait_for_peer "{{ $peerPool.fullname }}-{{ $i }}:${ENDPOINT_PORT}" \
nc -z -w 2 "{{ $peerPool.fullname }}-{{ $i }}.{{ include "rustfs.fullname" $ }}-headless.{{ $.Release.Namespace }}.svc.{{ include "rustfs.clusterDomain" $ }}" "$ENDPOINT_PORT"
{{- end }}
{{- else if eq (int $peerPool.index) (int $pool.index) }}
i=0
while [ "$i" -lt "$ordinal" ]; do
wait_for_peer "{{ $peerPool.fullname }}-${i}:${ENDPOINT_PORT}" \
nc -z -w 2 "{{ $peerPool.fullname }}-${i}.{{ include "rustfs.fullname" $ }}-headless.{{ $.Release.Namespace }}.svc.{{ include "rustfs.clusterDomain" $ }}" "$ENDPOINT_PORT"
i=$((i + 1))
done
{{- end }}
{{- end }}
volumeMounts:
{{- if gt $drivesPerNode 1 }}
{{- range $i := until $drivesPerNode }}
@@ -210,9 +190,19 @@ spec:
containerPort: {{ $.Values.service.endpoint.port }}
- name: console
containerPort: {{ $.Values.service.console.port }}
{{- with $.Values.extraEnv }}
{{- if or $injectLocalEndpointHost (not (empty $.Values.extraEnv)) }}
env:
{{- if $injectLocalEndpointHost }}
- name: RUSTFS_CHART_POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: RUSTFS_LOCAL_ENDPOINT_HOST
value: {{ printf "$(RUSTFS_CHART_POD_NAME).%s-headless.%s.svc.%s" (include "rustfs.fullname" $) $.Release.Namespace (include "rustfs.clusterDomain" $) | quote }}
{{- end }}
{{- with $.Values.extraEnv }}
{{- toYaml . | nindent 12 }}
{{- end }}
{{- end }}
envFrom:
- configMapRef:
+26 -4
View File
@@ -30,11 +30,22 @@ drivesPerNode: null
# Provide the DNS root only, without a `svc.` prefix or leading/trailing dots.
clusterDomain: cluster.local
# Maximum time an init container waits for distributed peers to publish DNS
# and for earlier peers to start listening. The wait is shared across all
# peers, not applied once per peer.
# Deprecated and ignored. Retained so existing values files continue to
# render after peer startup coordination moved into RustFS.
# RUSTFS_COMPAT_TODO(rustfs-5416): Keep the removed init-gate setting visible to existing values files. Remove after the minimum supported direct-upgrade chart includes localEndpointHost.autoInject and no longer renders peer gates.
startupWaitTimeoutSeconds: 300
# Inject a pod-specific endpoint identity for chart-generated distributed
# topologies. The default enables injection unless secret.existingSecret is
# set, because the chart cannot inspect runtime overrides hidden in that
# Secret. Without injection, the server infers a unique matching domain
# endpoint from its kernel hostname in Kubernetes auto/orchestrated mode; custom
# aliases retain bounded legacy DNS locality when no hostname matches. Configure
# RUSTFS_LOCAL_ENDPOINT_HOST explicitly for DNS-free startup. Set true for a
# credentials-only existing Secret or false to suppress chart injection.
localEndpointHost:
autoInject: null
# This sets the container image more information can be found here: https://kubernetes.io/docs/concepts/containers/images/
image:
rustfs: # This sets the rustfs image repository and tag.
@@ -223,7 +234,18 @@ config:
default_key: ""
extraEnv: [] # This is for setting extra environment variables in the rustfs container. It should be a list of key value pairs. For example:
# Extra environment variables for the RustFS container.
# In distributed mode with chart-generated volumes (config.rustfs.volumes is
# empty), the chart injects a private Downward API pod name and
# RUSTFS_LOCAL_ENDPOINT_HOST to identify the pod's local endpoints. An explicit
# RUSTFS_LOCAL_ENDPOINT_HOST or RUSTFS_VOLUMES entry disables automatic anchor
# injection. An explicit bounded or fail-fast
# RUSTFS_STARTUP_TOPOLOGY_WAIT_MODE, or an unrecognized static value, also keeps
# legacy DNS-based locality discovery. Otherwise Kubernetes auto-detection
# selects orchestrated startup when RustFS consumes the generated anchor.
# With custom volumes, the chart does not inject them and extraEnv may define
# the local endpoint identity explicitly.
extraEnv: []
# extraEnv:
# - name: RUSTFS_ERASURE_SET_DRIVE_COUNT
# value: "16"