mirror of
https://github.com/rustfs/rustfs.git
synced 2026-07-26 08:18:18 +00:00
feat(helm): flexible drivesPerNode topology with backward-compatible per-pool defaults (#4901)
* helm chart: one data drive per node - fix1 * refactor(helm): prevent negative or 0 replicaCount Co-authored-by: Copilot <copilot@github.com> * refactor(helm): remove env REPLICA_COUNT * feat(helm): default no logs directory to force stdout Co-authored-by: Copilot <copilot@github.com> * feat(helm): add drivesPerNode * feat(helm): correct default parity with 4 nodes / 1 drive per node * conditional render of RUSTFS_OBS_LOG_DIRECTORY * feat(chart): add table doc for parity * fix(chart): handle invalid annotation objects * fix(chart): move logging and obsevability options together; default for kubernetes output to stdout * feat(chart): better table doc for parity * fix(chart): leave RUSTFS_OBS_LOG_DIRECTORY empty, or the defaults will attempt to write to readonly fs * fix(chart): chart defaults as the previous version: 4 nodes with 4 drives per node * feat(chart): render RUSTFS_STORAGE_CLASS_STANDARD in configmap * minor fix for pvcAnnotations Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Signed-off-by: Cristian Chiru <cristi.chiru@gmail.com> * fix(chart): better drivesPerNode handling * fix(chart): obs_log_directory default non empty again to preserve previous deployments compatibility * fix(chart): proper drivesPerNode impl * fix(chart): values clarification for empty vs unset `obs_log_directory` * feat(chart): add service externalIPs * feat(helm): add optional service labels * fix(helm): merge service labels * fix(helm): storageclass pvcAnnotations * fix(chart): move logging and obsevability options together; default for kubernetes output to stdout * fix(helm): remove duplicate keys * fix(helm): default drivesPerNode null, keep legacy chart behavior * feat(helm): add template test for externalIPs * fix(helm): per-pool drivesPerNode inference, restore regression tests Repairs the drivesPerNode feature from #2693 so it renders and stays upgrade-safe: - define the missing drives inference (rustfs.poolDrives) and compute it per pool inside rustfs.pools, so mixed 4x4 + 16x1 pool deployments keep their exact legacy volumeClaimTemplates when drivesPerNode is unset - restore the $poolsEnabled definition dropped in the rebase (chart failed to render at all) - fix .Values references inside the pool range (dot is the pool there) and keep per-pool storageclass pvcAnnotations overrides working - make rustfs.volumes derive the drive range from pool drives instead of the pod count, and relax the pools.list 4-or-16 restriction to >= 2 - reject drivesPerNode=0 explicitly instead of silently inferring - keep default rendering identical to main: storage_class_standard stays unrendered by default, obs_endpoint.use_stdout stays false, clusterDomain value restored - restore the clusterDomain/mTLS SAN/explicit-volumes regression tests that the branch deleted, keeping the new topology tests --------- Signed-off-by: Cristian Chiru <cristi.chiru@gmail.com> Co-authored-by: Cristian Chiru <cristi.chiru@gmail.com> Co-authored-by: Copilot <copilot@github.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
This commit is contained in:
Zhengchao An
+49
-10
@@ -1,13 +1,51 @@
|
||||
# RustFS Helm Mode
|
||||
|
||||
RustFS helm chart supports **standalone and distributed mode**. For standalone mode, there is only one pod and one pvc; for distributed mode, there are two styles, 4 pods and 16 pvcs(each pod has 4 pvcs), 16 pods and 16 pvcs(each pod has 1 pvc). You should decide which mode and style suits for your situation. You can specify the parameters `mode` and `replicaCount` to install different mode and style.
|
||||
RustFS helm chart supports **standalone** and **distributed** mode.
|
||||
|
||||
- **For standalone mode**: Only one pod and one pvc acts as single node single disk; Specify parameters `mode.standalone.enabled="true",mode.distributed.enabled="false"` to install.
|
||||
- **For distributed mode**(**default**): Multiple pods and multiple pvcs, acts as multiple nodes multiple disks, there are two styles:
|
||||
- 4 pods and each pods has 4 pvcs(**default**)
|
||||
- 16 pods and each pods has 1 pvc: Specify parameters `replicaCount` with `--set replicaCount="16"` to install.
|
||||
- **Standalone mode**: one pod with one PVC (single node, single disk).
|
||||
- **Distributed mode** (**default**): multiple pods with multiple PVCs (multiple nodes, multiple disks).
|
||||
|
||||
**NOTE**: Please make sure which mode suits for you situation and specify the right parameter to install rustfs on kubernetes.
|
||||
## Distributed topology
|
||||
|
||||
The distributed topology is defined by two parameters:
|
||||
|
||||
- `replicaCount` — number of pods (nodes) in the StatefulSet.
|
||||
- `drivesPerNode` — number of data PVCs mounted on each pod.
|
||||
|
||||
Total drives in the cluster = `replicaCount * drivesPerNode`.
|
||||
|
||||
When `drivesPerNode` is left unset, the chart automatically infers a
|
||||
backward-compatible value from each pool's replica count (with pools
|
||||
disabled there is a single pool driven by the top-level `replicaCount`):
|
||||
|
||||
| `replicaCount` | Inferred `drivesPerNode` | Legacy equivalent |
|
||||
|----------------|--------------------------|-------------------|
|
||||
| 4 | 4 | old default 4×4 |
|
||||
| anything else | 1 | old 16×1, etc. |
|
||||
|
||||
You can override the inference by setting `drivesPerNode` explicitly, e.g.
|
||||
`--set drivesPerNode=2` for an 8×2 cluster.
|
||||
|
||||
**IMPORTANT**: Kubernetes does **not** allow changes to
|
||||
`volumeClaimTemplates` in an existing StatefulSet. If you want to change
|
||||
`drivesPerNode` after installation you must delete the StatefulSet
|
||||
(with `--cascade=orphan` to keep pods and PVCs) and recreate it, or perform a
|
||||
full reinstall.
|
||||
|
||||
---
|
||||
|
||||
## Upgrade notes
|
||||
|
||||
Upgrading from chart versions that did **not** have `drivesPerNode` is safe
|
||||
without manual intervention:
|
||||
|
||||
- Existing 4×4 deployments (default `replicaCount=4`) continue to receive 4
|
||||
drives per node because the chart infers `drivesPerNode=4`.
|
||||
- Existing 16×1 deployments (`replicaCount=16`) continue to receive 1 drive
|
||||
per node because the chart infers `drivesPerNode=1`.
|
||||
|
||||
If you previously set `replicaCount=16` and now want a different topology,
|
||||
set both `replicaCount` and `drivesPerNode` explicitly.
|
||||
|
||||
---
|
||||
|
||||
@@ -134,7 +172,7 @@ uer. `ClusterIssuer` or `Issuer`. |
|
||||
| pdb.minAvailable | string | `""` | |
|
||||
| podAnnotations | object | `{}` | |
|
||||
| pools.enabled | bool | `false` | Enable multiple server pools (capacity expansion, distributed mode only). |
|
||||
| pools.list | list | `[]` | One entry per pool; entries may set `replicaCount` (4 or 16) and `storageclass`, omitted fields inherit top-level values. Append-only. |
|
||||
| pools.list | list | `[]` | One entry per pool; entries may set `replicaCount` (>= 2) and `storageclass`, omitted fields inherit top-level values. Append-only. |
|
||||
| podLabels | object | `{}` | |
|
||||
| podSecurityContext.fsGroup | int | `10001` | |
|
||||
| podSecurityContext.runAsGroup | int | `10001` | |
|
||||
@@ -146,7 +184,8 @@ uer. `ClusterIssuer` or `Issuer`. |
|
||||
| readinessProbe.periodSeconds | int | `5` | |
|
||||
| readinessProbe.successThreshold | int | `1` | |
|
||||
| readinessProbe.timeoutSeconds | int | `3` | |
|
||||
| replicaCount | int | `4` | Number of cluster nodes. |
|
||||
| replicaCount | int | `4` | Number of cluster nodes. Distributed mode requires >= 2. |
|
||||
| drivesPerNode | int | `null` | Number of data PVCs per pod. Inferred from replicaCount when unset (see Distributed topology above). |
|
||||
| resources.limits.cpu | string | `"200m"` | |
|
||||
| resources.limits.memory | string | `"512Mi"` | |
|
||||
| resources.requests.cpu | string | `"100m"` | |
|
||||
@@ -237,12 +276,12 @@ pools:
|
||||
list:
|
||||
- {} # pool 0: inherits top-level values and keeps the
|
||||
# existing StatefulSet/pod/PVC names and data
|
||||
- replicaCount: 4 # pool 1: new capacity (4 or 16)
|
||||
- replicaCount: 4 # pool 1: new capacity
|
||||
storageclass:
|
||||
dataStorageSize: 10Gi
|
||||
```
|
||||
|
||||
Each entry may set `replicaCount` (4 or 16) and/or a `storageclass` block;
|
||||
Each entry may set `replicaCount` (>= 2) and/or a `storageclass` block;
|
||||
omitted fields inherit the top-level values. Additional pools render as
|
||||
`<fullname>-pool<N>` StatefulSets; all pools share the headless service,
|
||||
the main service, the configuration and the credentials.
|
||||
|
||||
@@ -42,6 +42,19 @@ app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
|
||||
app.kubernetes.io/managed-by: {{ .Release.Service }}
|
||||
{{- end }}
|
||||
|
||||
{{/*
|
||||
Render extra labels for the main Service resource.
|
||||
Merges (in order of increasing precedence):
|
||||
- commonLabels
|
||||
- service.labels
|
||||
*/}}
|
||||
{{- define "rustfs.serviceLabels" -}}
|
||||
{{- $labels := mergeOverwrite (dict) (default (dict) .Values.commonLabels) (default (dict) .Values.service.labels) }}
|
||||
{{- if $labels }}
|
||||
{{- toYaml $labels }}
|
||||
{{- end }}
|
||||
{{- end }}
|
||||
|
||||
{{/*
|
||||
Selector labels
|
||||
*/}}
|
||||
@@ -185,6 +198,30 @@ Expects a dict with keys "root" (the chart root context) and "index".
|
||||
{{- end -}}
|
||||
{{- end }}
|
||||
|
||||
{{/*
|
||||
Return the number of data drives (PVCs) per pod for a pool.
|
||||
When .Values.drivesPerNode is set it applies to every pool. When unset the
|
||||
value is inferred per pool from its replica count so that rendered output
|
||||
stays identical to the pre-drivesPerNode chart: 4 replicas keep the legacy
|
||||
4-drive layout, everything else (16x1 and any new topology) gets one drive.
|
||||
Expects a dict with keys "root" (the chart root context) and "replicas".
|
||||
*/}}
|
||||
{{- define "rustfs.poolDrives" -}}
|
||||
{{- if kindIs "invalid" .root.Values.drivesPerNode -}}
|
||||
{{- if eq (int .replicas) 4 -}}
|
||||
4
|
||||
{{- else -}}
|
||||
1
|
||||
{{- end -}}
|
||||
{{- else -}}
|
||||
{{- $d := int .root.Values.drivesPerNode -}}
|
||||
{{- if lt $d 1 -}}
|
||||
{{- fail "drivesPerNode must be >= 1 when set" -}}
|
||||
{{- end -}}
|
||||
{{- $d -}}
|
||||
{{- end -}}
|
||||
{{- end }}
|
||||
|
||||
{{/*
|
||||
Return the normalized list of server pools as JSON.
|
||||
With pools disabled this is a single pool built from the top-level
|
||||
@@ -206,14 +243,16 @@ so entries must never be removed or reordered.
|
||||
{{- range $i, $p := .Values.pools.list -}}
|
||||
{{- $p = default (dict) $p -}}
|
||||
{{- $replicas := int (default $.Values.replicaCount $p.replicaCount) -}}
|
||||
{{- if and (ne $replicas 4) (ne $replicas 16) -}}
|
||||
{{- fail (printf "pools.list[%d].replicaCount must be 4 or 16, got %d" $i $replicas) -}}
|
||||
{{- if lt $replicas 2 -}}
|
||||
{{- fail (printf "pools.list[%d].replicaCount must be >= 2, got %d" $i $replicas) -}}
|
||||
{{- end -}}
|
||||
{{- $sc := mergeOverwrite (deepCopy $.Values.storageclass) (default (dict) $p.storageclass) -}}
|
||||
{{- $pools = append $pools (dict "index" $i "fullname" (include "rustfs.poolFullname" (dict "root" $ "index" $i)) "replicaCount" $replicas "storageclass" $sc) -}}
|
||||
{{- $drives := int (include "rustfs.poolDrives" (dict "root" $ "replicas" $replicas)) -}}
|
||||
{{- $pools = append $pools (dict "index" $i "fullname" (include "rustfs.poolFullname" (dict "root" $ "index" $i)) "replicaCount" $replicas "drives" $drives "storageclass" $sc) -}}
|
||||
{{- end -}}
|
||||
{{- else -}}
|
||||
{{- $pools = append $pools (dict "index" 0 "fullname" (include "rustfs.fullname" .) "replicaCount" (int .Values.replicaCount) "storageclass" .Values.storageclass) -}}
|
||||
{{- $drives := int (include "rustfs.poolDrives" (dict "root" $ "replicas" (int .Values.replicaCount))) -}}
|
||||
{{- $pools = append $pools (dict "index" 0 "fullname" (include "rustfs.fullname" .) "replicaCount" (int .Values.replicaCount) "drives" $drives "storageclass" .Values.storageclass) -}}
|
||||
{{- end -}}
|
||||
{{- toJson $pools -}}
|
||||
{{- end }}
|
||||
@@ -235,9 +274,10 @@ RUSTFS_VOLUMES on spaces, one pool per expression).
|
||||
{{- $exprs := list -}}
|
||||
{{- range $pool := include "rustfs.pools" . | fromJsonArray -}}
|
||||
{{- $n := int $pool.replicaCount -}}
|
||||
{{- if eq $n 4 -}}
|
||||
{{- $exprs = append $exprs (printf "%s://%s-{0...%d}.%s.%s.svc.%s:%d/data/rustfs{0...%d}" $protocol $pool.fullname (sub $n 1) $headless $ns $domain $port (sub $n 1)) -}}
|
||||
{{- else if eq $n 16 -}}
|
||||
{{- $d := int $pool.drives -}}
|
||||
{{- if gt $d 1 -}}
|
||||
{{- $exprs = append $exprs (printf "%s://%s-{0...%d}.%s.%s.svc.%s:%d/data/rustfs{0...%d}" $protocol $pool.fullname (sub $n 1) $headless $ns $domain $port (sub $d 1)) -}}
|
||||
{{- else -}}
|
||||
{{- $exprs = append $exprs (printf "%s://%s-{0...%d}.%s.%s.svc.%s:%d/data" $protocol $pool.fullname (sub $n 1) $headless $ns $domain $port) -}}
|
||||
{{- end -}}
|
||||
{{- end -}}
|
||||
|
||||
@@ -19,6 +19,9 @@ data:
|
||||
{{- if .domains }}
|
||||
RUSTFS_SERVER_DOMAINS: {{ include "rustfs.serverDomains" $ | quote }}
|
||||
{{- end }}
|
||||
{{- if .ec.storage_class_standard }}
|
||||
RUSTFS_STORAGE_CLASS_STANDARD: {{ .ec.storage_class_standard | quote }}
|
||||
{{- end }}
|
||||
{{- with .log_rotation }}
|
||||
{{- if .size }}
|
||||
RUSTFS_OBS_LOG_ROTATION_SIZE_MB: {{ .size | quote }}
|
||||
|
||||
@@ -131,7 +131,7 @@ spec:
|
||||
subPath: ca.crt
|
||||
- name: client-cert
|
||||
mountPath: /opt/tls/client_cert.pem
|
||||
subPath: client_cert.pem
|
||||
subPath: client_cert.pem
|
||||
- name: client-cert
|
||||
mountPath: /opt/tls/client_key.pem
|
||||
subPath: client_key.pem
|
||||
|
||||
@@ -39,8 +39,8 @@ metadata:
|
||||
{{- end }}
|
||||
labels:
|
||||
{{- include "rustfs.labels" . | nindent 4 }}
|
||||
{{- with .Values.commonLabels }}
|
||||
{{- toYaml . | nindent 4 }}
|
||||
{{- with (include "rustfs.serviceLabels" .) }}
|
||||
{{- . | nindent 4 }}
|
||||
{{- end }}
|
||||
spec:
|
||||
{{- if eq $serviceType "ClusterIP" }}
|
||||
@@ -64,6 +64,10 @@ spec:
|
||||
{{- toYaml . | nindent 4 }}
|
||||
{{- end }}
|
||||
{{- end }}
|
||||
{{- with .Values.service.externalIPs }}
|
||||
externalIPs:
|
||||
{{- toYaml . | nindent 4 }}
|
||||
{{- end }}
|
||||
ports:
|
||||
- name: endpoint
|
||||
port: {{ .Values.service.endpoint.port }}
|
||||
|
||||
@@ -1,9 +1,13 @@
|
||||
{{- $logDir := .Values.config.rustfs.obs_log_directory }}
|
||||
{{- $logDirEnabled := ne $logDir "" }}
|
||||
{{- $poolsEnabled := and .Values.pools .Values.pools.enabled }}
|
||||
{{- if and .Values.mode.distributed.enabled (le (int .Values.replicaCount) 1) -}}
|
||||
{{- fail "Distributed mode requires replicaCount >= 2" -}}
|
||||
{{- end -}}
|
||||
|
||||
{{- if .Values.mode.distributed.enabled }}
|
||||
{{- range $pool := include "rustfs.pools" . | fromJsonArray }}
|
||||
{{- $drivesPerNode := int $pool.drives }}
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: StatefulSet
|
||||
@@ -117,17 +121,17 @@ spec:
|
||||
securityContext:
|
||||
{{- toYaml $.Values.containerSecurityContext | nindent 12 }}
|
||||
env:
|
||||
- name: REPLICA_COUNT
|
||||
value: {{ $pool.replicaCount | quote }}
|
||||
- name: DRIVES_PER_NODE
|
||||
value: {{ $drivesPerNode | quote }}
|
||||
command:
|
||||
- sh
|
||||
- -c
|
||||
- |
|
||||
if [ "$REPLICA_COUNT" -eq 4 ]; then
|
||||
for i in $(seq 0 $(($REPLICA_COUNT - 1))); do
|
||||
if [ "$DRIVES_PER_NODE" -gt 1 ]; then
|
||||
for i in $(seq 0 $(($DRIVES_PER_NODE - 1))); do
|
||||
mkdir -p /data/rustfs$i
|
||||
done;
|
||||
elif [ "$REPLICA_COUNT" -eq 16 ]; then
|
||||
done
|
||||
else
|
||||
mkdir -p /data
|
||||
fi
|
||||
{{- if $logDirEnabled }}
|
||||
@@ -135,12 +139,12 @@ spec:
|
||||
chmod 755 /mnt/rustfs/logs
|
||||
{{- end }}
|
||||
volumeMounts:
|
||||
{{- if eq (int $pool.replicaCount) 4 }}
|
||||
{{- range $i := until (int $pool.replicaCount) }}
|
||||
{{- if gt $drivesPerNode 1 }}
|
||||
{{- range $i := until $drivesPerNode }}
|
||||
- name: data-rustfs-{{ $i }}
|
||||
mountPath: /data/rustfs{{ $i }}
|
||||
{{- end }}
|
||||
{{- else if eq (int $pool.replicaCount) 16 }}
|
||||
{{- else }}
|
||||
- name: data
|
||||
mountPath: /data
|
||||
{{- end }}
|
||||
@@ -201,12 +205,12 @@ spec:
|
||||
mountPath: {{ $logDir }}
|
||||
subPath: logs
|
||||
{{- end }}
|
||||
{{- if eq (int $pool.replicaCount) 4 }}
|
||||
{{- range $i := until (int $pool.replicaCount) }}
|
||||
{{- if gt $drivesPerNode 1 }}
|
||||
{{- range $i := until $drivesPerNode }}
|
||||
- name: data-rustfs-{{ $i }}
|
||||
mountPath: /data/rustfs{{ $i }}
|
||||
{{- end }}
|
||||
{{- else if eq (int $pool.replicaCount) 16 }}
|
||||
{{- else }}
|
||||
- name: data
|
||||
mountPath: /data
|
||||
{{- end }}
|
||||
@@ -246,8 +250,11 @@ spec:
|
||||
name: logs
|
||||
labels:
|
||||
{{- toYaml $.Values.commonLabels | nindent 10 }}
|
||||
{{- $logAnn := (default (dict) $pool.storageclass.pvcAnnotations).logs | default (dict) -}}
|
||||
{{- if gt (len $logAnn) 0 }}
|
||||
annotations:
|
||||
{{- toYaml $pool.storageclass.pvcAnnotations.logs | nindent 10 }}
|
||||
{{- toYaml $logAnn | nindent 10 }}
|
||||
{{- end }}
|
||||
spec:
|
||||
accessModes: ["ReadWriteOnce"]
|
||||
storageClassName: {{ $pool.storageclass.name }}
|
||||
@@ -255,16 +262,19 @@ spec:
|
||||
requests:
|
||||
storage: {{ $pool.storageclass.logStorageSize }}
|
||||
{{- end }}
|
||||
{{- if eq (int $pool.replicaCount) 4 }}
|
||||
{{- range $i := until (int $pool.replicaCount) }}
|
||||
{{- if gt $drivesPerNode 1 }}
|
||||
{{- range $i := until $drivesPerNode }}
|
||||
- apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: data-rustfs-{{ $i }}
|
||||
labels:
|
||||
{{- toYaml $.Values.commonLabels | nindent 10 }}
|
||||
{{- $dataAnn := (default (dict) $pool.storageclass.pvcAnnotations).data | default (dict) -}}
|
||||
{{- if gt (len $dataAnn) 0 }}
|
||||
annotations:
|
||||
{{- toYaml $pool.storageclass.pvcAnnotations.data | nindent 10 }}
|
||||
{{- toYaml $dataAnn | nindent 10 }}
|
||||
{{- end }}
|
||||
spec:
|
||||
accessModes: ["ReadWriteOnce"]
|
||||
storageClassName: {{ $pool.storageclass.name }}
|
||||
@@ -272,15 +282,18 @@ spec:
|
||||
requests:
|
||||
storage: {{ $pool.storageclass.dataStorageSize }}
|
||||
{{- end }}
|
||||
{{- else if eq (int $pool.replicaCount) 16 }}
|
||||
{{- else }}
|
||||
- apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: data
|
||||
labels:
|
||||
{{- toYaml $.Values.commonLabels | nindent 10 }}
|
||||
{{- $dataAnn := (default (dict) $pool.storageclass.pvcAnnotations).data | default (dict) -}}
|
||||
{{- if gt (len $dataAnn) 0 }}
|
||||
annotations:
|
||||
{{- toYaml $pool.storageclass.pvcAnnotations.data | nindent 10 }}
|
||||
{{- toYaml $dataAnn | nindent 10 }}
|
||||
{{- end }}
|
||||
spec:
|
||||
accessModes: ["ReadWriteOnce"]
|
||||
storageClassName: {{ $pool.storageclass.name }}
|
||||
|
||||
+53
-18
@@ -2,9 +2,27 @@
|
||||
# This is a YAML-formatted file.
|
||||
# Declare variables to be passed into your templates.
|
||||
|
||||
# This will set the replicaset count more information can be found here: https://kubernetes.io/docs/concepts/workloads/controllers/replicaset/
|
||||
# Number of StatefulSet pods (nodes in the distributed cluster).
|
||||
# In distributed mode this must be >= 2 (RustFS requires at least two nodes
|
||||
# for an erasure-coded cluster).
|
||||
replicaCount: 4
|
||||
|
||||
# Number of data drives (PVCs) per pod. Combined with replicaCount this
|
||||
# determines the total drive count seen by rustfs.
|
||||
# - drivesPerNode > 1: each pod gets N volumes mounted at /data/rustfs0.../data/rustfsN-1
|
||||
# - drivesPerNode = 1: each pod gets a single volume mounted at /data
|
||||
#
|
||||
# When left unset (null) the chart infers a backward-compatible value per
|
||||
# pool from that pool's replica count:
|
||||
# replicaCount = 4 -> drivesPerNode = 4 (legacy default)
|
||||
# replicaCount != 4 -> drivesPerNode = 1 (legacy 16x1 and any other topology)
|
||||
#
|
||||
# WARNING: changing drivesPerNode on an existing StatefulSet is not allowed
|
||||
# because Kubernetes forbids updates to volumeClaimTemplates. To change the
|
||||
# drive count you must delete the StatefulSet (with --cascade=orphan to keep
|
||||
# pods/PVCs) and recreate it, or perform a full reinstall.
|
||||
drivesPerNode: null
|
||||
|
||||
# Kubernetes cluster DNS domain used to build in-cluster FQDNs for
|
||||
# RUSTFS_VOLUMES (distributed mode) and mTLS server certificate SANs.
|
||||
# Override this only if your cluster does not use the default `cluster.local`
|
||||
@@ -60,7 +78,7 @@ mode:
|
||||
# With pools.enabled=true, one StatefulSet is rendered per entry of
|
||||
# pools.list and RUSTFS_VOLUMES contains every pool's volume expression
|
||||
# (space separated), which is how the server discovers multiple pools.
|
||||
# Each entry may set replicaCount (4 or 16) and/or a storageclass block;
|
||||
# Each entry may set replicaCount (>= 2) and/or a storageclass block;
|
||||
# omitted fields inherit the top-level values.
|
||||
#
|
||||
# IMPORTANT:
|
||||
@@ -99,23 +117,31 @@ config:
|
||||
# Examples
|
||||
# volumes: "/data/rustfs0,/data/rustfs1,/data/rustfs2,/data/rustfs3"
|
||||
# volumes: "http://rustfs-{0...3}.rustfs-headless:9000/data/rustfs{0...3}"
|
||||
# NOTE: When overriding volumes manually, the mount paths and drive ranges
|
||||
# must match the number of pods (replicaCount) and drives per pod
|
||||
# (drivesPerNode). Mismatched values will cause broken volume discovery.
|
||||
volumes: ""
|
||||
address: ":9000"
|
||||
console_enable: "true"
|
||||
console_address: ":9001"
|
||||
log_level: "info"
|
||||
region: "us-east-1"
|
||||
obs_log_directory: "/logs" # Set to "" to disable log PVCs and mounts.
|
||||
obs_environment: "development"
|
||||
# Optionally enable support for virtual-hosted-style requests.
|
||||
# See more information: https://docs.rustfs.com/integration/virtual.html
|
||||
domains: "" # e.g. "example.com"
|
||||
ec:
|
||||
storage_class_standard: "EC:4" # Storage class for standard storage class in erasure coding mode, default is "STANDARD"
|
||||
log_rotation: # Specify log rotation settings
|
||||
# size: 100 # Default value: 100 MB
|
||||
# time: hour # Default value: hour, eg: day,hour,minute,second
|
||||
# keep_files: 30 # number of rotated log files to keep
|
||||
# Erasure-coding parity for the STANDARD storage class, e.g. "EC:4".
|
||||
# Left empty by default: the server then picks a parity suited to the
|
||||
# drive count. If you set it, it must be valid for the total drive
|
||||
# count (total drives = replicaCount * drivesPerNode):
|
||||
#
|
||||
# | replicaCount | drivesPerNode | Total drives | Valid parity |
|
||||
# | ------------ | ------------- | ------------ | ------------ |
|
||||
# | 4 | 1 | 4 | 0-2 |
|
||||
# | 4 | 2 | 8 | 0-4 |
|
||||
# | 4 | 4 | 16 | 0-8 |
|
||||
# | 8 | 1 | 8 | 0-4 |
|
||||
# | 16 | 1 | 16 | 0-8 |
|
||||
storage_class_standard: ""
|
||||
scanner:
|
||||
# Scanner speed preset: fastest|fast|default|slow|slowest
|
||||
speed: ""
|
||||
@@ -155,6 +181,16 @@ config:
|
||||
# - default: keep current timeout defaults.
|
||||
# - high_latency: use higher timeout defaults for scanner-sensitive operations.
|
||||
timeout_profile: ""
|
||||
log_level: "info"
|
||||
log_rotation: # Specify log rotation settings
|
||||
# size: 100 # Default value: 100 MB
|
||||
# time: hour # Default value: hour, eg: day,hour,minute,second
|
||||
# keep_files: 30 # number of rotated log files to keep
|
||||
## If obs_log_directory is not empty, a separate volume for logs will be created and the logs will NOT be sent to stdout. If not set, will default to default chart value "/logs"
|
||||
obs_log_directory: "/logs"
|
||||
## Supported values include `production`, `development`, `test`, and `staging`;
|
||||
## refer to the RustFS configuration documentation/source for the authoritative list.
|
||||
obs_environment: "development"
|
||||
obs_endpoint:
|
||||
enabled: false # If true, rustfs will export metrics, traces, logs and profiling data to the specified OTLP endpoints. If false, the individual settings for metrics, traces, logs and profiling endpoints will be ignored and all data will not be exported.
|
||||
base_endpoint: "" #Root OTLP/HTTP endpoint, e.g. http://otel-collector:4318
|
||||
@@ -175,7 +211,7 @@ config:
|
||||
enabled: false
|
||||
type: "vault" # Only Support vault currently.
|
||||
vault:
|
||||
vault_backend: "" # Only support vault kv2 and vault transit.
|
||||
vault_backend: "" # Only support vault kv2 and vault transit.
|
||||
vault_address: ""
|
||||
vault_token: ""
|
||||
vault_mount_path: ""
|
||||
@@ -235,6 +271,7 @@ priorityClassName: ""
|
||||
|
||||
service:
|
||||
annotations: {}
|
||||
labels: {} # Additional labels
|
||||
headlessAnnotations: {} # Applied to the headless Service when mode.distributed.enabled=true
|
||||
traefikAnnotations: # Applied to the Service when mode.distributed.enabled=true and ingress.className=traefik
|
||||
traefik.ingress.kubernetes.io/service.sticky.cookie: "true"
|
||||
@@ -247,6 +284,7 @@ service:
|
||||
loadBalancerIP: "" # Request a specific IP from the load balancer (e.g. for Cilium LB-IPAM or MetalLB)
|
||||
loadBalancerClass: "" # Specify a load balancer implementation (e.g. "io.cilium/bgp", "metallb")
|
||||
loadBalancerSourceRanges: [] # Restrict client IPs (e.g. ["10.0.0.0/8"])
|
||||
externalIPs: [] # Expose service via specific external IPs (e.g. ["203.0.113.1"])
|
||||
endpoint:
|
||||
port: 9000
|
||||
nodePort: 32000
|
||||
@@ -297,7 +335,7 @@ ingress:
|
||||
gatewayApi:
|
||||
enabled: false
|
||||
gatewayClass: traefik # Only support for traefik and contour gatewayClass at the moment.
|
||||
listeners: # Specify which listeners to create on the Gateway.
|
||||
listeners: # Specify which listeners to create on the Gateway.
|
||||
http:
|
||||
name: web
|
||||
port: 8000
|
||||
@@ -380,12 +418,9 @@ storageclass:
|
||||
name: local-path
|
||||
dataStorageSize: 256Mi
|
||||
logStorageSize: 256Mi
|
||||
pvcAnnotations: {}
|
||||
#pvcAnnotations:
|
||||
# data:
|
||||
# key: value
|
||||
# logs:
|
||||
# key: value
|
||||
pvcAnnotations:
|
||||
data: {}
|
||||
logs: {}
|
||||
|
||||
pdb:
|
||||
create: false
|
||||
|
||||
@@ -303,3 +303,70 @@ if grep -q 'svc\.cluster\.local"' <<<"$cert_custom"; then
|
||||
echo "Custom clusterDomain must fully replace cluster.local in mTLS server cert SANs" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Legacy topology compatibility: default replicaCount=4 (no drivesPerNode set)
|
||||
# must render the old 4x4 PVC names (data-rustfs-0 .. data-rustfs-3).
|
||||
legacy_four_by_four=$(render_distributed_statefulset)
|
||||
for i in 0 1 2 3; do
|
||||
if ! grep -q "name: data-rustfs-${i}" <<<"$legacy_four_by_four"; then
|
||||
echo "Legacy 4x4 topology must contain PVC data-rustfs-${i}" >&2
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
if grep -q "name: data$" <<<"$legacy_four_by_four"; then
|
||||
echo "Legacy 4x4 topology must NOT contain a single 'data' PVC" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Legacy topology compatibility: replicaCount=16 (no drivesPerNode set)
|
||||
# must render a single 'data' PVC (old 16x1 behaviour).
|
||||
legacy_sixteen_by_one=$(render_distributed_statefulset --set replicaCount=16)
|
||||
if ! grep -q "name: data$" <<<"$legacy_sixteen_by_one"; then
|
||||
echo "Legacy 16x1 topology must contain a single 'data' PVC" >&2
|
||||
exit 1
|
||||
fi
|
||||
if grep -q "name: data-rustfs-" <<<"$legacy_sixteen_by_one"; then
|
||||
echo "Legacy 16x1 topology must NOT contain data-rustfs-* PVCs" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Generic topology: explicit replicaCount=8 drivesPerNode=2 must render
|
||||
# exactly two PVCs per pod.
|
||||
generic_eight_by_two=$(render_distributed_statefulset --set replicaCount=8 --set drivesPerNode=2)
|
||||
for i in 0 1; do
|
||||
if ! grep -q "name: data-rustfs-${i}" <<<"$generic_eight_by_two"; then
|
||||
echo "Generic 8x2 topology must contain PVC data-rustfs-${i}" >&2
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
if grep -q "name: data$" <<<"$generic_eight_by_two"; then
|
||||
echo "Generic 8x2 topology must NOT contain a single 'data' PVC" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# volumeClaimTemplates must not contain empty annotations when pvcAnnotations are unset,
|
||||
# because Kubernetes treats annotations: {} as a mutation of the immutable field.
|
||||
no_ann_output=$(render_distributed_statefulset)
|
||||
if grep -A1 'kind: PersistentVolumeClaim' <<<"$no_ann_output" | grep -q 'annotations:'; then
|
||||
echo "Empty pvcAnnotations must not render an annotations key in volumeClaimTemplates" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Distributed mode with replicaCount < 2 must fail rendering.
|
||||
low_replica_status=0
|
||||
render_distributed_statefulset --set replicaCount=1 >/dev/null 2>&1 || low_replica_status=$?
|
||||
if [[ $low_replica_status -eq 0 ]]; then
|
||||
echo "Distributed mode with replicaCount=1 must fail rendering" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# service.externalIPs must render correctly when supplied.
|
||||
external_ips_output=$(helm template rustfs "$CHART_DIR" \
|
||||
--namespace rustfs \
|
||||
--set secret.rustfs.access_key=test-access-key \
|
||||
--set secret.rustfs.secret_key=test-secret-key \
|
||||
--set 'service.externalIPs[0]=203.0.113.1')
|
||||
if ! grep -A2 'externalIPs:' <<<"$external_ips_output" | grep -q '203.0.113.1'; then
|
||||
echo "service.externalIPs must contain 203.0.113.1" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
Reference in New Issue
Block a user