gsadmin/rustfs
1
0
Fork 0
mirror of https://github.com/rustfs/rustfs.git synced 2026-08-10 07:06:53 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(helm): support multiple server pools for capacity expansion (#3325)

Browse Source 
* feat(helm): support multiple server pools for capacity expansion

The server already understands multiple pools (RUSTFS_VOLUMES is split on
spaces, one pool expression each; rc admin pool/expand/rebalance/
decommission exist), but the chart could only render a single StatefulSet.

Add an opt-in pools mode:

- pools.enabled=false (default) renders byte-identical output to the
  current chart (verified against five values variants)
- pools.list renders one StatefulSet per entry; entries may set
  replicaCount (4 or 16) and a storageclass block, everything else is
  inherited from the top-level values
- pool 0 keeps the legacy StatefulSet/pod/PVC names and its (immutable)
  selector, so existing single-pool deployments can be expanded in place
  without renaming or data loss; additional pools render as
  <fullname>-poolN with a rustfs.com/pool label in their selector
- RUSTFS_VOLUMES and RUSTFS_SERVER_DOMAINS enumerate every pool
- pod anti-affinity is scoped per pool so pools may share nodes while
  each pool still spreads its own pods across distinct nodes
- template validation fails fast on unsupported replicaCount, an empty
  pools.list, or pools in standalone mode
- pools are append-only by design (list index = identity), documented in
  values.yaml and the README

* fix(helm): truncate pool names DNS-safely, document PDB pool scope

Truncate <fullname>-poolN to 63 chars by shortening the base name rather
than the suffix, so the pool index always survives and two pools of a
max-length release cannot collide. Document that the single
PodDisruptionBudget deliberately spans all pools.

* fix(helm): pool-mode anti-affinity must be preferred, not required

Found by live-testing in-place expansion on a 5-node cluster (4-replica
pool 0 + 4-replica pool 1): with requiredDuringScheduling anti-affinity
the expansion deadlocks in a cycle that cannot self-heal -

  1. the not-yet-rolled pool-0 pods still carry the unscoped required
     rule and repel every rustfs pod from their nodes, so part of pool 1
     stays Pending (no IP, no headless-DNS record);
  2. every pod that already has the new RUSTFS_VOLUMES exits fatally on
     'failed to lookup address information' for those Pending peers;
  3. the StatefulSet rolling update is gated on the crashing pod going
     Ready, so the remaining pool-0 pods never roll and keep repelling.

Because StatefulSets assign revisions per ordinal, even a subsequent
template fix cannot rescue a wedged rollout (Pending ordinals are only
recreated with the new template after the higher ordinals go Ready) -
the new pool's StatefulSet has to be recreated. Shipping the soft
affinity from the start avoids the state entirely; expansion was
re-tested end-to-end with it and converges.

Pool-mode rendering only - pools.enabled=false still renders the
existing required anti-affinity byte-identically.

---------

Co-authored-by: cxymds <Cxymds@qq.com>
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: cxymds <cxymds@gmail.com>
This commit is contained in:
Rohmilchkaese
2026-07-09 09:58:06 +02:00
committed by GitHub
parent 10ba2273c4
commit 8bcffd8a04
4 changed files with 260 additions and 81 deletions
Download Patch File Download Diff File Expand all files Collapse all files
helm/rustfs/templates/_helpers.tpl
+71 -12
View File
@@ -168,22 +168,80 @@ falling back to cluster.local when the value is empty or only dots.
{{- .Values.clusterDomain | default "cluster.local" | trimAll "." | default "cluster.local" -}}
{{- end -}}
{{/*
Return the fully qualified name of a server pool.
Pool 0 keeps the legacy single-pool name so that existing deployments can be
expanded in place without renaming their StatefulSet, pods or PVCs.
Expects a dict with keys "root" (the chart root context) and "index".
*/}}
{{- define "rustfs.poolFullname" -}}
{{- if eq (int .index) 0 -}}
{{- include "rustfs.fullname" .root -}}
{{- else -}}
{{- /* Truncate the base name, not the suffix: the pool index must survive
truncation or two long-named pools could collide. */ -}}
{{- $suffix := printf "-pool%d" (int .index) -}}
{{- printf "%s%s" (include "rustfs.fullname" .root | trunc (int (sub 63 (len $suffix))) | trimSuffix "-") $suffix -}}
{{- end -}}
{{- end }}
{{/*
Return the normalized list of server pools as JSON.
With pools disabled this is a single pool built from the top-level
replicaCount/storageclass values, which keeps all rendered output identical
to the previous single-pool chart. With pools enabled, each entry of
pools.list becomes one pool; omitted fields inherit the top-level values.
Pools are strictly append-only: the index determines the StatefulSet name,
so entries must never be removed or reordered.
*/}}
{{- define "rustfs.pools" -}}
{{- $pools := list -}}
{{- if and .Values.pools .Values.pools.enabled -}}
{{- if not .Values.mode.distributed.enabled -}}
{{- fail "pools.enabled requires mode.distributed.enabled=true" -}}
{{- end -}}
{{- if not .Values.pools.list -}}
{{- fail "pools.enabled is true but pools.list is empty" -}}
{{- end -}}
{{- range $i, $p := .Values.pools.list -}}
{{- $p = default (dict) $p -}}
{{- $replicas := int (default $.Values.replicaCount $p.replicaCount) -}}
{{- if and (ne $replicas 4) (ne $replicas 16) -}}
{{- fail (printf "pools.list[%d].replicaCount must be 4 or 16, got %d" $i $replicas) -}}
{{- end -}}
{{- $sc := mergeOverwrite (deepCopy $.Values.storageclass) (default (dict) $p.storageclass) -}}
{{- $pools = append $pools (dict "index" $i "fullname" (include "rustfs.poolFullname" (dict "root" $ "index" $i)) "replicaCount" $replicas "storageclass" $sc) -}}
{{- end -}}
{{- else -}}
{{- $pools = append $pools (dict "index" 0 "fullname" (include "rustfs.fullname" .) "replicaCount" (int .Values.replicaCount) "storageclass" .Values.storageclass) -}}
{{- end -}}
{{- toJson $pools -}}
{{- end }}
{{/*
Render RUSTFS_VOLUMES
One volume expression per server pool, joined with spaces (the server splits
RUSTFS_VOLUMES on spaces, one pool per expression).
*/}}
{{- define "rustfs.volumes" -}}
{{- $protocol := "http" -}}
{{- if .Values.mtls.enabled -}}
{{- $protocol = "https" -}}
{{- end -}}
{{- if eq (int .Values.replicaCount) 4 }}
{{- printf "%s://%s-{0...%d}.%s-headless.%s.svc.%s:%d/data/rustfs{0...%d}" $protocol (include "rustfs.fullname" .) (sub (.Values.replicaCount | int) 1) (include "rustfs.fullname" . ) .Release.Namespace (include "rustfs.clusterDomain" .) (.Values.service.endpoint.port | int) (sub (.Values.replicaCount | int) 1) }}
{{- end }}
{{- if eq (int .Values.replicaCount) 16 }}
{{- printf "%s://%s-{0...%d}.%s-headless.%s.svc.%s:%d/data" $protocol (include "rustfs.fullname" .) (sub (.Values.replicaCount | int) 1) (include "rustfs.fullname" .) .Release.Namespace (include "rustfs.clusterDomain" .) (.Values.service.endpoint.port | int) }}
{{- end }}
{{- $headless := printf "%s-headless" (include "rustfs.fullname" .) -}}
{{- $ns := .Release.Namespace -}}
{{- $domain := include "rustfs.clusterDomain" . -}}
{{- $port := .Values.service.endpoint.port | int -}}
{{- $exprs := list -}}
{{- range $pool := include "rustfs.pools" . | fromJsonArray -}}
{{- $n := int $pool.replicaCount -}}
{{- if eq $n 4 -}}
{{- $exprs = append $exprs (printf "%s://%s-{0...%d}.%s.%s.svc.%s:%d/data/rustfs{0...%d}" $protocol $pool.fullname (sub $n 1) $headless $ns $domain $port (sub $n 1)) -}}
{{- else if eq $n 16 -}}
{{- $exprs = append $exprs (printf "%s://%s-{0...%d}.%s.%s.svc.%s:%d/data" $protocol $pool.fullname (sub $n 1) $headless $ns $domain $port) -}}
{{- end -}}
{{- end -}}
{{- join " " $exprs -}}
{{- end }}
{{/*
@@ -192,13 +250,14 @@ Render RUSTFS_SERVER_DOMAINS
{{- define "rustfs.serverDomains" -}}
{{- $domains := list .Values.config.rustfs.domains -}}
{{- $fullname := include "rustfs.fullname" . -}}
{{- $replicaCount := int .Values.replicaCount -}}
{{- $headless := printf "%s-headless" (include "rustfs.fullname" .) -}}
{{- $servicePort := .Values.service.endpoint.port | default 9000 -}}
{{- range $i := until $replicaCount -}}
{{- $podDomain := printf "%s-%d.%s-headless:%d" $fullname $i $fullname (int $servicePort) -}}
{{- range $pool := include "rustfs.pools" . | fromJsonArray -}}
{{- range $i := until (int $pool.replicaCount) -}}
{{- $podDomain := printf "%s-%d.%s:%d" $pool.fullname $i $headless (int $servicePort) -}}
{{- $domains = append $domains $podDomain -}}
{{- end -}}
{{- end -}}
{{- join "," $domains -}}
{{- end -}}
helm/rustfs/templates/statefulset.yaml
+107 -69
View File
@@ -1,87 +1,124 @@
{{- $logDir := .Values.config.rustfs.obs_log_directory }}
{{- $logDirEnabled := ne $logDir "" }}
{{- $poolsEnabled := and .Values.pools .Values.pools.enabled }}
{{- if .Values.mode.distributed.enabled }}
{{- range $pool := include "rustfs.pools" . | fromJsonArray }}
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: {{ include "rustfs.fullname" . }}
namespace: {{ .Release.Namespace }}
name: {{ $pool.fullname }}
namespace: {{ $.Release.Namespace }}
labels:
{{- include "rustfs.labels" . | nindent 4 }}
{{- with .Values.commonLabels }}
{{- include "rustfs.labels" $ | nindent 4 }}
{{- if $poolsEnabled }}
rustfs.com/pool: {{ $pool.index | quote }}
{{- end }}
{{- with $.Values.commonLabels }}
{{- toYaml . | nindent 4 }}
{{- end }}
spec:
serviceName: {{ include "rustfs.fullname" . }}-headless
replicas: {{ .Values.replicaCount }}
serviceName: {{ include "rustfs.fullname" $ }}-headless
replicas: {{ $pool.replicaCount }}
podManagementPolicy: Parallel
selector:
matchLabels:
{{- include "rustfs.selectorLabels" . | nindent 6 }}
{{- include "rustfs.selectorLabels" $ | nindent 6 }}
{{- if gt (int $pool.index) 0 }}
{{- /* Pool 0 keeps the legacy selector: it is immutable on existing
StatefulSets and must not change when pools are enabled. */}}
rustfs.com/pool: {{ $pool.index | quote }}
{{- end }}
template:
metadata:
labels:
{{- include "rustfs.selectorLabels" . | nindent 8 }}
{{- with .Values.podLabels }}
{{- include "rustfs.selectorLabels" $ | nindent 8 }}
{{- if $poolsEnabled }}
rustfs.com/pool: {{ $pool.index | quote }}
{{- end }}
{{- with $.Values.podLabels }}
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.podAnnotations }}
{{- with $.Values.podAnnotations }}
annotations:
{{- toYaml . | nindent 8 }}
{{- end }}
spec:
serviceAccountName: {{ include "rustfs.serviceAccountName" . }}
enableServiceLinks: {{ .Values.enableServiceLinks }}
{{- with include "chart.imagePullSecrets" . }}
serviceAccountName: {{ include "rustfs.serviceAccountName" $ }}
enableServiceLinks: {{ $.Values.enableServiceLinks }}
{{- with include "chart.imagePullSecrets" $ }}
imagePullSecrets:
{{- . | nindent 8 }}
{{- end }}
{{- if and .Values.nodeSelector (not .Values.affinity.nodeAffinity) }}
{{- if and $.Values.nodeSelector (not $.Values.affinity.nodeAffinity) }}
nodeSelector:
{{- toYaml .Values.nodeSelector | nindent 8 }}
{{- toYaml $.Values.nodeSelector | nindent 8 }}
{{- end }}
{{- if .Values.affinity }}
{{- if $.Values.affinity }}
affinity:
nodeAffinity:
{{- if .Values.affinity.nodeAffinity }}
{{- toYaml .Values.affinity.nodeAffinity | nindent 10 }}
{{- if $.Values.affinity.nodeAffinity }}
{{- toYaml $.Values.affinity.nodeAffinity | nindent 10 }}
{{- else }}
{}
{{- end }}
{{- if .Values.affinity.podAntiAffinity.enabled }}
{{- if $.Values.affinity.podAntiAffinity.enabled }}
podAntiAffinity:
{{- if $poolsEnabled }}
{{- /* Pool-scoped and PREFERRED, not required: during in-place
expansion the not-yet-rolled pods' required rules block new
pods from their nodes, while the new pods' DNS must resolve
before the roll can proceed - required rules deadlock
expansion whenever the cluster has fewer nodes than total
pods. Soft anti-affinity converges (the scheduler still
spreads when capacity allows). */}}
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchExpressions:
- key: app.kubernetes.io/name
operator: In
values:
- {{ include "rustfs.name" $ }}
- key: rustfs.com/pool
operator: In
values:
- {{ $pool.index | quote }}
topologyKey: {{ $.Values.affinity.podAntiAffinity.topologyKey }}
{{- else }}
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: app.kubernetes.io/name
operator: In
values:
- {{ include "rustfs.name" . }}
topologyKey: {{ .Values.affinity.podAntiAffinity.topologyKey }}
- {{ include "rustfs.name" $ }}
topologyKey: {{ $.Values.affinity.podAntiAffinity.topologyKey }}
{{- end }}
{{- end }}
{{- end }}
{{- if .Values.tolerations }}
{{- if $.Values.tolerations }}
tolerations:
{{- toYaml .Values.tolerations | nindent 8 }}
{{- toYaml $.Values.tolerations | nindent 8 }}
{{- end }}
{{- if and .Values.topologySpreadConstraints.enabled .Values.topologySpreadConstraints.constraints }}
{{- if and $.Values.topologySpreadConstraints.enabled $.Values.topologySpreadConstraints.constraints }}
topologySpreadConstraints:
{{- toYaml .Values.topologySpreadConstraints.constraints | nindent 8 }}
{{- toYaml $.Values.topologySpreadConstraints.constraints | nindent 8 }}
{{- end }}
priorityClassName: {{ .Values.priorityClassName }}
priorityClassName: {{ $.Values.priorityClassName }}
securityContext:
{{- toYaml .Values.podSecurityContext | nindent 8 }}
{{- toYaml $.Values.podSecurityContext | nindent 8 }}
initContainers:
- name: init-step
image: "{{ .Values.image.initImage.repository }}:{{ .Values.image.initImage.tag }}"
imagePullPolicy: {{ .Values.image.initImage.pullPolicy }}
image: "{{ $.Values.image.initImage.repository }}:{{ $.Values.image.initImage.tag }}"
imagePullPolicy: {{ $.Values.image.initImage.pullPolicy }}
securityContext:
{{- toYaml .Values.containerSecurityContext | nindent 12 }}
{{- toYaml $.Values.containerSecurityContext | nindent 12 }}
env:
- name: REPLICA_COUNT
value: {{ .Values.replicaCount | quote }}
value: {{ $pool.replicaCount | quote }}
command:
- sh
- -c
@@ -98,12 +135,12 @@ spec:
chmod 755 /mnt/rustfs/logs
{{- end }}
volumeMounts:
{{- if eq (int .Values.replicaCount) 4 }}
{{- range $i := until (int .Values.replicaCount) }}
{{- if eq (int $pool.replicaCount) 4 }}
{{- range $i := until (int $pool.replicaCount) }}
- name: data-rustfs-{{ $i }}
mountPath: /data/rustfs{{ $i }}
{{- end }}
{{- else if eq (int .Values.replicaCount) 16 }}
{{- else if eq (int $pool.replicaCount) 16 }}
- name: data
mountPath: /data
{{- end }}
@@ -112,31 +149,31 @@ spec:
mountPath: /mnt/rustfs
{{- end }}
containers:
- name: {{ .Chart.Name }}
image: "{{ .Values.image.rustfs.repository }}:{{ .Values.image.rustfs.tag | default .Chart.AppVersion }}"
- name: {{ $.Chart.Name }}
image: "{{ $.Values.image.rustfs.repository }}:{{ $.Values.image.rustfs.tag | default $.Chart.AppVersion }}"
command: ["/usr/bin/rustfs"]
imagePullPolicy: {{ .Values.image.rustfs.pullPolicy }}
imagePullPolicy: {{ $.Values.image.rustfs.pullPolicy }}
securityContext:
{{- toYaml .Values.containerSecurityContext | nindent 12 }}
{{- toYaml $.Values.containerSecurityContext | nindent 12 }}
ports:
- name: endpoint
containerPort: {{ .Values.service.endpoint.port }}
containerPort: {{ $.Values.service.endpoint.port }}
- name: console
containerPort: {{ .Values.service.console.port }}
{{- with .Values.extraEnv }}
containerPort: {{ $.Values.service.console.port }}
{{- with $.Values.extraEnv }}
env:
{{- toYaml . | nindent 12 }}
{{- end }}
envFrom:
- configMapRef:
name: {{ include "rustfs.fullname" . }}-config
name: {{ include "rustfs.fullname" $ }}-config
- secretRef:
name: {{ include "rustfs.secretName" . }}
name: {{ include "rustfs.secretName" $ }}
resources:
{{- toYaml .Values.resources | nindent 12 }}
{{- include "rustfs.probes" . | nindent 10 }}
{{- toYaml $.Values.resources | nindent 12 }}
{{- include "rustfs.probes" $ | nindent 10 }}
volumeMounts:
{{- if .Values.mtls.enabled }}
{{- if $.Values.mtls.enabled }}
- name: client-cert
mountPath: /opt/tls/client_cert.pem
subPath: client_cert.pem
@@ -156,7 +193,7 @@ spec:
mountPath: /opt/tls/ca.crt
subPath: ca.crt
{{- end }}
{{- with .Values.extraVolumeMounts }}
{{- with $.Values.extraVolumeMounts }}
{{- toYaml . | nindent 12 }}
{{- end }}
{{- if $logDirEnabled }}
@@ -164,21 +201,21 @@ spec:
mountPath: {{ $logDir }}
subPath: logs
{{- end }}
{{- if eq (int .Values.replicaCount) 4 }}
{{- range $i := until (int .Values.replicaCount) }}
{{- if eq (int $pool.replicaCount) 4 }}
{{- range $i := until (int $pool.replicaCount) }}
- name: data-rustfs-{{ $i }}
mountPath: /data/rustfs{{ $i }}
{{- end }}
{{- else if eq (int .Values.replicaCount) 16 }}
{{- else if eq (int $pool.replicaCount) 16 }}
- name: data
mountPath: /data
{{- end }}
{{- if or .Values.mtls.enabled .Values.extraVolumes }}
{{- if or $.Values.mtls.enabled $.Values.extraVolumes }}
volumes:
{{- if .Values.mtls.enabled }}
{{- if $.Values.mtls.enabled }}
- name: server-cert
secret:
secretName: {{ include "rustfs.fullname" . }}-server-tls
secretName: {{ include "rustfs.fullname" $ }}-server-tls
items:
- key: tls.crt
path: rustfs_cert.pem
@@ -188,7 +225,7 @@ spec:
path: ca.crt
- name: client-cert
secret:
secretName: {{ include "rustfs.fullname" . }}-client-tls
secretName: {{ include "rustfs.fullname" $ }}-client-tls
items:
- key: tls.crt
path: client_cert.pem
@@ -197,7 +234,7 @@ spec:
- key: ca.crt
path: client_ca.crt
{{- end }}
{{- with .Values.extraVolumes }}
{{- with $.Values.extraVolumes }}
{{- toYaml . | nindent 8 }}
{{- end }}
{{- end }}
@@ -208,18 +245,18 @@ spec:
metadata:
name: logs
labels:
{{- toYaml .Values.commonLabels | nindent 10 }}
{{- toYaml $.Values.commonLabels | nindent 10 }}
annotations:
{{- toYaml .Values.storageclass.pvcAnnotations.logs | nindent 10 }}
{{- toYaml $pool.storageclass.pvcAnnotations.logs | nindent 10 }}
spec:
accessModes: ["ReadWriteOnce"]
storageClassName: {{ .Values.storageclass.name }}
storageClassName: {{ $pool.storageclass.name }}
resources:
requests:
storage: {{ .Values.storageclass.logStorageSize }}
storage: {{ $pool.storageclass.logStorageSize }}
{{- end }}
{{- if eq (int .Values.replicaCount) 4 }}
{{- range $i := until (int .Values.replicaCount) }}
{{- if eq (int $pool.replicaCount) 4 }}
{{- range $i := until (int $pool.replicaCount) }}
- apiVersion: v1
kind: PersistentVolumeClaim
metadata:
@@ -227,28 +264,29 @@ spec:
labels:
{{- toYaml $.Values.commonLabels | nindent 10 }}
annotations:
{{- toYaml $.Values.storageclass.pvcAnnotations.data | nindent 10 }}
{{- toYaml $pool.storageclass.pvcAnnotations.data | nindent 10 }}
spec:
accessModes: ["ReadWriteOnce"]
storageClassName: {{ $.Values.storageclass.name }}
storageClassName: {{ $pool.storageclass.name }}
resources:
requests:
storage: {{ $.Values.storageclass.dataStorageSize }}
storage: {{ $pool.storageclass.dataStorageSize }}
{{- end }}
{{- else if eq (int .Values.replicaCount) 16 }}
{{- else if eq (int $pool.replicaCount) 16 }}
- apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: data
labels:
{{- toYaml .Values.commonLabels | nindent 10 }}
{{- toYaml $.Values.commonLabels | nindent 10 }}
annotations:
{{- toYaml .Values.storageclass.pvcAnnotations.data | nindent 10 }}
{{- toYaml $pool.storageclass.pvcAnnotations.data | nindent 10 }}
spec:
accessModes: ["ReadWriteOnce"]
storageClassName: {{ .Values.storageclass.name }}
storageClassName: {{ $pool.storageclass.name }}
resources:
requests:
storage: {{ .Values.storageclass.dataStorageSize }}
storage: {{ $pool.storageclass.dataStorageSize }}
{{- end }}
{{- end }}
{{- end }}
helm/rustfs/values.yaml
+27
View File
@@ -53,6 +53,33 @@ mode:
distributed:
enabled: true
# Server pools for horizontal capacity expansion (distributed mode only).
# Disabled by default: the chart then behaves exactly like the classic
# single-pool layout driven by the top-level replicaCount/storageclass.
#
# With pools.enabled=true, one StatefulSet is rendered per entry of
# pools.list and RUSTFS_VOLUMES contains every pool's volume expression
# (space separated), which is how the server discovers multiple pools.
# Each entry may set replicaCount (4 or 16) and/or a storageclass block;
# omitted fields inherit the top-level values.
#
# IMPORTANT:
# - Pools are append-only. The list index determines the StatefulSet name
# (pool 0 keeps the legacy name so existing single-pool deployments can be
# expanded in place); never remove or reorder entries. To retire a pool,
# use `rc admin decommission` first.
# - After adding a pool, run `rc admin rebalance start <alias>` to spread
# existing data across the new capacity.
pools:
enabled: false
list: []
# Example: expand an existing 4-node deployment with a second, larger pool:
# list:
# - {} # pool 0: inherits top-level values, keeps existing PVCs
# - replicaCount: 4 # pool 1: new capacity
# storageclass:
# dataStorageSize: 10Gi
secret:
existingSecret: ""
# SECURITY: rendering fails by default unless one of the following is true: