fix(ecstore): start with unresolved Kubernetes peers

This commit is contained in:
马登山
2026-07-30 08:06:20 +08:00
parent 3991a1d73c
commit a1e3ee9f40
26 changed files with 2368 additions and 371 deletions
+55 -14
View File
@@ -47,6 +47,47 @@ without manual intervention:
If you previously set `replicaCount=16` and now want a different topology,
set both `replicaCount` and `drivesPerNode` explicitly.
For distributed deployments that use the chart-generated
`RUSTFS_VOLUMES`, `localEndpointHost.autoInject` defaults to automatic
selection. Without `secret.existingSecret`, the chart injects a private
Downward API variable, `RUSTFS_CHART_POD_NAME`, and uses it to build the pod's
fully qualified hostname in
`RUSTFS_LOCAL_ENDPOINT_HOST`. RustFS can then identify local drives without
waiting for every peer's DNS record or TCP listener. A user-defined `POD_NAME`
is preserved and does not interfere with the private chart variable. Setting
`RUSTFS_STARTUP_TOPOLOGY_WAIT_MODE` explicitly to `bounded`, `fail-fast`,
`failfast`, or `strict` also keeps legacy DNS-based locality discovery. A
dynamically sourced wait mode keeps the legacy path because its value cannot be
validated while rendering. Otherwise, Kubernetes auto-detection selects
orchestrated startup when RustFS consumes the generated anchor.
An existing Secret is opaque to the chart and may historically contain more
than credentials, so automatic selection preserves the legacy DNS-based
locality path whenever `secret.existingSecret` is set. For a credentials-only
Secret, set `localEndpointHost.autoInject=true` to add the anchor without
changing the historical ConfigMap-then-Secret `envFrom` precedence. Set it to
`false` to preserve the legacy path explicitly. If injection is explicitly
enabled with an incompatible hidden `RUSTFS_VOLUMES`, `RUSTFS_ADDRESS`, or
`RUSTFS_STARTUP_TOPOLOGY_WAIT_MODE`, RustFS fails during endpoint construction
before format initialization; it does not silently fall back to a different
topology.
When `config.rustfs.volumes` is set explicitly, the chart does not infer a
local endpoint identity. Custom topologies that need one must provide
`RUSTFS_LOCAL_ENDPOINT_HOST` through `extraEnv`. An explicit
`RUSTFS_VOLUMES`, explicit `RUSTFS_LOCAL_ENDPOINT_HOST`, bounded/dynamic
or unrecognized startup mode, or `localEndpointHost.autoInject=false`, also
disables automatic injection. A `RUSTFS_ADDRESS` override alone does not disable
it; the effective address and generated topology must agree on the endpoint
port. Custom anchor-based configurations must resolve to `orchestrated` startup
mode and must not receive a conflicting mode from an `envFrom` source.
`startupWaitTimeoutSeconds` is retained for values-file compatibility but is
deprecated and ignored.
Upgrade the chart and RustFS image together. An older image that does not
recognize `RUSTFS_LOCAL_ENDPOINT_HOST` retains its previous DNS-based startup
behavior.
---
# Parameters Overview
@@ -57,6 +98,7 @@ set both `replicaCount` and `drivesPerNode` explicitly.
| affinity.podAntiAffinity.enabled | bool | `true` | |
| affinity.podAntiAffinity.topologyKey | string | `"kubernetes.io/hostname"` | |
| clusterDomain | string | `"cluster.local"` | Kubernetes cluster DNS domain used to build in-cluster FQDNs for `RUSTFS_VOLUMES` (distributed mode) and mTLS server certificate SANs. Override for clusters not using the default `cluster.local`. Provide the DNS root only, without a `svc.` prefix or leading/trailing dots. |
| localEndpointHost.autoInject | bool or null | `null` | Automatically inject `RUSTFS_LOCAL_ENDPOINT_HOST` for chart-generated distributed topologies unless `secret.existingSecret` is set. Use `true` for a credentials-only existing Secret or `false` to preserve legacy DNS locality explicitly. |
| commonLabels | object | `{}` | Labels to add to all deployed objects. |
| config.rustfs.address | string | `":9000"` | |
| config.rustfs.console_address | string | `":9001"` | |
@@ -66,7 +108,7 @@ set both `replicaCount` and `drivesPerNode` explicitly.
| config.rustfs.obs_environment | string | `"development"` | |
| config.rustfs.obs_log_directory | string | `"/logs"` | Log directory inside the RustFS container. Set to `""` to disable log PVCs and mounts. |
| config.rustfs.region | string | `"us-east-1"` | |
| config.rustfs.volumes | string | `""` | |
| config.rustfs.volumes | string | `""` | Explicit distributed volume topology. When empty, the chart generates the topology and normally injects `RUSTFS_LOCAL_ENDPOINT_HOST`; custom topologies must configure local endpoint identity explicitly when needed. |
| config.rustfs.log_rotation.size | int | `"100"` | Default log rotation size mb for rustfs. |
| config.rustfs.log_rotation.time | string | `"hour"` | Default log rotation time for rustfs. |
| config.rustfs.log_rotation.keep_files | int | `"30"` | Default log keep files for rustfs. |
@@ -107,7 +149,7 @@ set both `replicaCount` and `drivesPerNode` explicitly.
| config.rustfs.kms.vault.vault_token | string | `""`| The vault token. Rendered into a dedicated Secret (`<fullname>-kms-secret`), never into the ConfigMap. |
| config.rustfs.kms.vault.vault_mount_path | string | `"transit"`| The vault mount path, only works if `vault_backend` equals `vault-transit` . |
| config.rustfs.kms.vault.default_key | string | `"transit"`| The master key id for RustFS. |
| extraEnv | map | `[]` | Extra environment variables for RustFS container. |
| extraEnv | list | `[]` | Extra environment variables for the RustFS container. An explicit `RUSTFS_LOCAL_ENDPOINT_HOST` or `RUSTFS_VOLUMES`, or a bounded, dynamic, or unrecognized startup mode, disables generated anchor injection. `POD_NAME` and `RUSTFS_ADDRESS` remain independent overrides. |
| extraVolumes | list | `[]` | Extra volumes to add to the pod spec. Supported in both standalone (Deployment) and distributed (StatefulSet) modes. |
| extraVolumeMounts | list | `[]` | Extra volume mounts to add to the RustFS container. Supported in both standalone (Deployment) and distributed (StatefulSet) modes. |
| containerSecurityContext.capabilities.drop[0] | string | `"ALL"` | |
@@ -190,7 +232,7 @@ uer. `ClusterIssuer` or `Issuer`. |
| resources.limits.memory | string | `"512Mi"` | |
| resources.requests.cpu | string | `"100m"` | |
| resources.requests.memory | string | `"128Mi"` | |
| secret.existingSecret | string | `""` | Use existing secret with a credentials. |
| secret.existingSecret | string | `""` | Use an existing Secret. Automatic endpoint-anchor injection is disabled because the Secret is opaque; set `localEndpointHost.autoInject=true` only after confirming it contains credentials rather than runtime topology, address, or startup-mode overrides. |
| secret.rustfs.access_key | string | `"rustfsadmin"` | RustFS Access Key ID |
| secret.rustfs.secret_key | string | `"rustfsadmin"` | RustFS Secret Key ID |
| service.type | string | `"ClusterIP"` | |
@@ -202,6 +244,7 @@ uer. `ClusterIssuer` or `Issuer`. |
| serviceAccount.automount | bool | `true` | |
| serviceAccount.create | bool | `true` | |
| serviceAccount.name | string | `""` | |
| startupWaitTimeoutSeconds | int | `300` | Deprecated and ignored; retained for values-file compatibility. |
| storageclass.dataStorageSize | string | `"256Mi"` | The storage size for data PVC. |
| storageclass.logStorageSize | string | `"256Mi"` | The storage size for logs PVC. |
| storageclass.name | string | `"local-path"` | The name for StorageClass. |
@@ -291,20 +334,17 @@ Notes:
* **Pools are append-only.** The list index determines the StatefulSet name —
never remove or reorder entries. Retire a pool with
`rc admin decommission` before removing it from the list.
* During the expansion rollout, pods restart until every pod of every pool is
resolvable — the server refuses to start with unresolvable peers, so expect
a few crash/restart cycles before the cluster converges. This is harmless.
* With chart-generated volumes, each pod receives an explicit local endpoint
identity. An unavailable peer no longer blocks a pod from reaching RustFS's
own startup and quorum checks.
* After the cluster converges, run `rc admin rebalance start <alias>` to
spread existing objects across the new pool.
* Pod anti-affinity in pool mode is scoped per pool and **preferred**
(soft), not required: two pools can share nodes, and each pool's own pods
spread across distinct nodes when capacity allows. Soft affinity is
load-bearing for in-place expansion — with required rules, the
not-yet-rolled pods of the existing pool block the new pool's pods from
their nodes while the rolled pods crash on the unresolvable (Pending)
peers, deadlocking the rollout on any cluster with fewer nodes than the
total pod count. Single-pool deployments (`pools.enabled=false`) keep the
chart's existing required anti-affinity unchanged.
spread across distinct nodes when capacity allows. Preferred affinity keeps
additional pools schedulable when the cluster has fewer nodes than total
pods. Single-pool deployments (`pools.enabled=false`) keep the chart's
existing required anti-affinity unchanged.
* The PodDisruptionBudget spans all pools: with the default
`pdb.maxUnavailable: 1`, at most one pod of the whole cluster may be
evicted at a time. This is deliberately conservative — quorum safety
@@ -315,7 +355,8 @@ Notes:
## Requirement
* Helm V3
* RustFS >= 1.0.0-alpha.69
* The RustFS image from the same release as the chart. If `image.rustfs.tag`
is overridden, that image must support `RUSTFS_LOCAL_ENDPOINT_HOST`.
Due to the traefik and ingress has different session sticky/affinity annotations, and rustfs support both those two controller, you should specify parameter `ingress.className` to select the right one which suits for you.
+10 -8
View File
@@ -322,21 +322,23 @@ Render RUSTFS_SERVER_DOMAINS
{{- join "," $domains -}}
{{- end -}}
{{/* Render probe command for liveness and readiness
{{/* Render an mTLS probe command
*/}}
{{- define "rustfs.probeCommand" -}}
{{- $endpoint_port := .Values.service.endpoint.port | default 9000 -}}
{{- $console_port := .Values.service.console.port | default 9001 -}}
{{- $root := .root -}}
{{- $endpointPath := .endpointPath -}}
{{- $endpoint_port := $root.Values.service.endpoint.port | default 9000 -}}
{{- $console_port := $root.Values.service.console.port | default 9001 -}}
{{- $args := "-skf" -}}
{{- if and .Values.mtls.enabled -}}
{{- $args = printf "%s --cert %s --key %s" $args .Values.mtls.clientCertPath .Values.mtls.clientKeyPath -}}
{{- if and $root.Values.mtls.enabled -}}
{{- $args = printf "%s --cert %s --key %s" $args $root.Values.mtls.clientCertPath $root.Values.mtls.clientKeyPath -}}
{{- end -}}
- /bin/sh
- -c
- |
curl {{ $args }} https://127.0.0.1:{{ $endpoint_port }}/health/ready && \
curl {{ $args }} https://127.0.0.1:{{ $endpoint_port }}{{ $endpointPath }} && \
curl {{ $args }} https://127.0.0.1:{{ $console_port }}/rustfs/console/health
{{- end -}}
@@ -350,7 +352,7 @@ livenessProbe:
{{- if .Values.mtls.enabled }}
exec:
command:
{{ include "rustfs.probeCommand" . | nindent 6 }}
{{ include "rustfs.probeCommand" (dict "root" . "endpointPath" "/health") | nindent 6 }}
{{- else }}
httpGet:
path: /health
@@ -369,7 +371,7 @@ readinessProbe:
{{- if .Values.mtls.enabled }}
exec:
command:
{{ include "rustfs.probeCommand" . | nindent 6 }}
{{ include "rustfs.probeCommand" (dict "root" . "endpointPath" "/health/ready") | nindent 6 }}
{{- else }}
httpGet:
path: /health/ready
+43 -53
View File
@@ -1,12 +1,37 @@
{{- $logDir := .Values.config.rustfs.obs_log_directory }}
{{- $logDirEnabled := ne $logDir "" }}
{{- $poolsEnabled := and .Values.pools .Values.pools.enabled }}
{{- $generatedVolumes := empty .Values.config.rustfs.volumes }}
{{- $localEndpointHostAutoInject := empty .Values.secret.existingSecret }}
{{- with .Values.localEndpointHost }}
{{- if and (hasKey . "autoInject") (ne .autoInject nil) }}
{{- if not (kindIs "bool" .autoInject) }}
{{- fail "localEndpointHost.autoInject must be a boolean or null" }}
{{- end }}
{{- $localEndpointHostAutoInject = .autoInject }}
{{- end }}
{{- end }}
{{- $injectLocalEndpointHost := and $generatedVolumes $localEndpointHostAutoInject }}
{{- range $env := .Values.extraEnv -}}
{{- $name := default "" $env.name -}}
{{- if eq $name "RUSTFS_VOLUMES" -}}
{{- $injectLocalEndpointHost = false -}}
{{- else if eq $name "RUSTFS_LOCAL_ENDPOINT_HOST" -}}
{{- $injectLocalEndpointHost = false -}}
{{- else if eq $name "RUSTFS_STARTUP_TOPOLOGY_WAIT_MODE" -}}
{{- if hasKey $env "value" -}}
{{- $mode := lower (trim (toString $env.value)) -}}
{{- if and (ne $mode "") (ne $mode "auto") (ne $mode "orchestrated") -}}
{{- $injectLocalEndpointHost = false -}}
{{- end -}}
{{- else -}}
{{- $injectLocalEndpointHost = false -}}
{{- end -}}
{{- end -}}
{{- end -}}
{{- if and .Values.mode.distributed.enabled (le (int .Values.replicaCount) 1) -}}
{{- fail "Distributed mode requires replicaCount >= 2" -}}
{{- end -}}
{{- if and .Values.mode.distributed.enabled (le (int .Values.startupWaitTimeoutSeconds) 0) -}}
{{- fail "startupWaitTimeoutSeconds must be greater than 0 in distributed mode" -}}
{{- end -}}
{{- if .Values.mode.distributed.enabled }}
{{- $pools := include "rustfs.pools" . | fromJsonArray }}
@@ -74,13 +99,10 @@ spec:
{{- if $.Values.affinity.podAntiAffinity.enabled }}
podAntiAffinity:
{{- if $poolsEnabled }}
{{- /* Pool-scoped and PREFERRED, not required: during in-place
expansion the not-yet-rolled pods' required rules block new
pods from their nodes, while the new pods' DNS must resolve
before the roll can proceed - required rules deadlock
expansion whenever the cluster has fewer nodes than total
pods. Soft anti-affinity converges (the scheduler still
spreads when capacity allows). */}}
{{- /* Pool-scoped and PREFERRED, not required: additional pools
must remain schedulable when the cluster has fewer nodes
than total pods. The scheduler still spreads them when
capacity allows. */}}
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
@@ -127,10 +149,6 @@ spec:
env:
- name: DRIVES_PER_NODE
value: {{ $drivesPerNode | quote }}
- name: ENDPOINT_PORT
value: {{ $.Values.service.endpoint.port | quote }}
- name: STARTUP_WAIT_TIMEOUT_SECONDS
value: {{ $.Values.startupWaitTimeoutSeconds | quote }}
command:
- sh
- -c
@@ -146,44 +164,6 @@ spec:
mkdir -p /mnt/rustfs/logs
chmod 755 /mnt/rustfs/logs
{{- end }}
deadline=$(($(date +%s) + STARTUP_WAIT_TIMEOUT_SECONDS))
wait_for_peer() {
description=$1
shift
until "$@" >/dev/null 2>&1; do
if [ "$(date +%s)" -ge "$deadline" ]; then
echo "Timed out after ${STARTUP_WAIT_TIMEOUT_SECONDS}s waiting for ${description}" >&2
exit 1
fi
sleep 2
done
}
echo "Waiting for distributed peer DNS records"
{{- range $peerPool := $pools }}
{{- range $i := until (int $peerPool.replicaCount) }}
wait_for_peer "DNS for {{ $peerPool.fullname }}-{{ $i }}" \
nslookup "{{ $peerPool.fullname }}-{{ $i }}.{{ include "rustfs.fullname" $ }}-headless.{{ $.Release.Namespace }}.svc.{{ include "rustfs.clusterDomain" $ }}"
{{- end }}
{{- end }}
ordinal=${HOSTNAME##*-}
echo "Waiting for earlier distributed peers to listen on port ${ENDPOINT_PORT}"
{{- range $peerPool := $pools }}
{{- if lt (int $peerPool.index) (int $pool.index) }}
{{- range $i := until (int $peerPool.replicaCount) }}
wait_for_peer "{{ $peerPool.fullname }}-{{ $i }}:${ENDPOINT_PORT}" \
nc -z -w 2 "{{ $peerPool.fullname }}-{{ $i }}.{{ include "rustfs.fullname" $ }}-headless.{{ $.Release.Namespace }}.svc.{{ include "rustfs.clusterDomain" $ }}" "$ENDPOINT_PORT"
{{- end }}
{{- else if eq (int $peerPool.index) (int $pool.index) }}
i=0
while [ "$i" -lt "$ordinal" ]; do
wait_for_peer "{{ $peerPool.fullname }}-${i}:${ENDPOINT_PORT}" \
nc -z -w 2 "{{ $peerPool.fullname }}-${i}.{{ include "rustfs.fullname" $ }}-headless.{{ $.Release.Namespace }}.svc.{{ include "rustfs.clusterDomain" $ }}" "$ENDPOINT_PORT"
i=$((i + 1))
done
{{- end }}
{{- end }}
volumeMounts:
{{- if gt $drivesPerNode 1 }}
{{- range $i := until $drivesPerNode }}
@@ -210,9 +190,19 @@ spec:
containerPort: {{ $.Values.service.endpoint.port }}
- name: console
containerPort: {{ $.Values.service.console.port }}
{{- with $.Values.extraEnv }}
{{- if or $injectLocalEndpointHost (not (empty $.Values.extraEnv)) }}
env:
{{- if $injectLocalEndpointHost }}
- name: RUSTFS_CHART_POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: RUSTFS_LOCAL_ENDPOINT_HOST
value: {{ printf "$(RUSTFS_CHART_POD_NAME).%s-headless.%s.svc.%s" (include "rustfs.fullname" $) $.Release.Namespace (include "rustfs.clusterDomain" $) | quote }}
{{- end }}
{{- with $.Values.extraEnv }}
{{- toYaml . | nindent 12 }}
{{- end }}
{{- end }}
envFrom:
- configMapRef:
+23 -4
View File
@@ -30,11 +30,19 @@ drivesPerNode: null
# Provide the DNS root only, without a `svc.` prefix or leading/trailing dots.
clusterDomain: cluster.local
# Maximum time an init container waits for distributed peers to publish DNS
# and for earlier peers to start listening. The wait is shared across all
# peers, not applied once per peer.
# Deprecated and ignored. Retained so existing values files continue to
# render after peer startup coordination moved into RustFS.
# RUSTFS_COMPAT_TODO(rustfs-5416): Keep the removed init-gate setting visible to existing values files. Remove after the minimum supported direct-upgrade chart includes localEndpointHost.autoInject and no longer renders peer gates.
startupWaitTimeoutSeconds: 300
# Inject a pod-specific endpoint identity for chart-generated distributed
# topologies. The default enables injection unless secret.existingSecret is
# set, because the chart cannot inspect runtime overrides hidden in that
# Secret. Set true for a credentials-only existing Secret or false to keep
# legacy DNS-based locality explicitly.
localEndpointHost:
autoInject: null
# This sets the container image more information can be found here: https://kubernetes.io/docs/concepts/containers/images/
image:
rustfs: # This sets the rustfs image repository and tag.
@@ -223,7 +231,18 @@ config:
default_key: ""
extraEnv: [] # This is for setting extra environment variables in the rustfs container. It should be a list of key value pairs. For example:
# Extra environment variables for the RustFS container.
# In distributed mode with chart-generated volumes (config.rustfs.volumes is
# empty), the chart injects a private Downward API pod name and
# RUSTFS_LOCAL_ENDPOINT_HOST to identify the pod's local endpoints. An explicit
# RUSTFS_LOCAL_ENDPOINT_HOST or RUSTFS_VOLUMES entry disables automatic anchor
# injection. An explicit bounded or fail-fast
# RUSTFS_STARTUP_TOPOLOGY_WAIT_MODE, or an unrecognized static value, also keeps
# legacy DNS-based locality discovery. Otherwise Kubernetes auto-detection
# selects orchestrated startup when RustFS consumes the generated anchor.
# With custom volumes, the chart does not inject them and extraEnv may define
# the local endpoint identity explicitly.
extraEnv: []
# extraEnv:
# - name: RUSTFS_ERASURE_SET_DRIVE_COUNT
# value: "16"