mirror of
https://github.com/shankar0123/certctl.git
synced 2026-06-07 21:41:39 +00:00
f1fa311191
Bundle 3 closure (2026-05-12 acquisition diligence audit). Closes the
"chart claims production-ready but lying-fields silently break it"
hazard cluster: README install command had wrong key, required secrets
weren't fail-fast, external Postgres rendered the bundled StatefulSet
hostname, container-only security hardening fields landed at pod scope
(silently dropped by K8s API), and three advertised template surfaces
(ServiceMonitor, PodDisruptionBudget, NetworkPolicy) didn't render at
all even when their values.yaml toggles were on.
Source findings closed:
C2 C3 D1 D2 D3 D5 D7 D11 D12 (repo audit)
OPS-L1 OPS-L2 (cowork audit)
Source findings explicitly deferred (tracked in WORKSPACE-ROADMAP.md):
D6 OPS-H1 (backup automation — operator must choose target storage)
D10 (digest pinning of latest `:latest` tags)
OPS-M1 (prometheus/client_golang migration)
OPS-M2 (distributed tracing instrumentation)
Chart truth table (rendered with helm 3.16.3):
-f values.yaml + tls.existingSecret + auth.apiKey + pg.auth.password
→ 12 resources (default mode, no monitoring/PDB/networkpolicy)
+ postgresql.enabled=false + externalDatabase.url=…
→ NO StatefulSet, NO postgres-secret, NO postgres-service (D2)
+ server.tls.certManager.enabled=true
→ +1 Certificate (cert-manager mode)
+ replicas=3 + monitoring.enabled=true + serviceMonitor.enabled=true
+ podDisruptionBudget.enabled=true + networkPolicy.enabled=true
→ +1 ServiceMonitor + 1 PodDisruptionBudget + 1 NetworkPolicy (D5+D11)
tls.existingSecret AND tls.certManager.enabled both set
→ REFUSED with "EXACTLY ONE TLS ownership path" error (D7)
Missing required secrets (apiKey / pg password / external URL)
→ REFUSED at template time with operator-actionable guidance (D1)
Closures by source ID:
C2 — README Helm install example fixed. Was `--set postgresql.password=…`
(does not exist); now `--set postgresql.auth.password=…` matching
the chart key. README install block also wires TLS, mentions
fail-fast at template time, and links the external-Postgres example.
C3 — Kubernetes Secrets connector annotated PREVIEW in values.yaml.
The chart still exposes `kubernetesSecrets.enabled` for the RBAC
preview wiring, but the values block now states clearly that the
production K8s client at internal/connector/target/k8ssecret/
k8ssecret.go::realK8sClient is a stub (verified — go.mod imports
zero k8s.io/client-go packages). Production landing tracked in
WORKSPACE-ROADMAP.md.
D1 — `certctl.requiredSecrets` template helper. Fail-fasts at render
time when (a) server.auth.type=api-key + apiKey empty, (b)
postgresql.enabled=true + pg.auth.password empty, (c)
postgresql.enabled=false + externalDatabase.url + legacy env
CERTCTL_DATABASE_URL all empty. Each branch emits an
operator-actionable diagnostic with the openssl rand command or
values override needed. postgres-secret template additionally
uses Helm's `required` builtin so it can't render with the empty
fallback that pre-Bundle-3 produced ("changeme" literal).
D2 — externalDatabase.url first-class. New top-level values block.
certctl.databaseURL helper now branches on postgresql.enabled:
bundled path uses the helper-emitted in-cluster URL; external
path uses externalDatabase.url verbatim. postgres-secret,
postgres-statefulset, and postgres-service ALL gate on
postgresql.enabled — external mode renders ZERO postgres-*
resources. POSTGRES_PASSWORD env in server-deployment also gates.
D3 — Container-vs-pod security context split. K8s API silently drops
readOnlyRootFilesystem / allowPrivilegeEscalation / capabilities /
privileged when they land at pod scope (`spec.securityContext`);
they only work at container scope (`spec.containers[].securityContext`).
Pre-Bundle-3 all fields sat at pod scope so the chart's documented
"read-only rootfs + drop-all caps" hardening was effectively
unenforced. New certctl.podSecurityContext + containerSecurityContext
helpers split the operator-facing securityContext map by field-name
whitelist so existing values keep working byte-for-byte while
fields render at the K8s-valid scope. Applied to both
server-deployment.yaml and agent-daemonset.yaml (DaemonSet + Deployment
branches).
D5 — Prometheus ServiceMonitor template. New
templates/servicemonitor.yaml. Renders when monitoring.enabled AND
monitoring.serviceMonitor.enabled. Scrapes /api/v1/metrics/prometheus
(rbac-gated on metrics.read — needs bearerTokenSecret with an API
key holding that perm). values.yaml block extended with bearerTokenSecret,
tlsConfig, and relabelings knobs and the operator-facing comment
documenting the auth requirement.
D7 — TLS both-set rejection. certctl.tls.required helper extended.
Pre-Bundle-3 only the NEITHER-set case was caught; setting BOTH
rendered a dangling cert-manager Certificate alongside an
existing-Secret mount, two conflicting TLS sources of truth.
Now refuses with "EXACTLY ONE TLS ownership path" + remediation
steps for both possible operator intents.
D11 — PodDisruptionBudget + NetworkPolicy templates. New
templates/pdb.yaml (renders when podDisruptionBudget.enabled +
server.replicas > 1) + templates/networkpolicy.yaml (renders when
networkPolicy.enabled). PDB uses minAvailable / maxUnavailable
exclusivity per K8s spec. NetworkPolicy default-allows in-namespace
agent → server traffic, kube-DNS egress, and bundled-postgres
egress (when postgresql.enabled), with operator-extensible
extraIngress / extraEgress for CA / OIDC / SMTP egress. Both
default off so existing deploys don't lose network reach
unannounced.
D12 — Database max-conn config wired. Pre-Bundle-3
internal/repository/postgres/db.go::NewDB hard-coded
SetMaxOpenConns(25). config.go loaded CERTCTL_DATABASE_MAX_CONNS,
Validate() enforced the >= 1 floor, values.yaml documented it,
and docs/reference/configuration.md surfaced it — but the pool
ignored every operator setting. New NewDBWithMaxConns threads
the operator value into the pool with maxIdle = maxOpen / 5
(≥ 1) so the historical ratio carries forward. cmd/server/main.go
calls the new constructor; NewDB stays for compat at the default 25.
OPS-L1 — Chart version 0.1.0 → 1.0.0. Chart has shipped through 8 audit
closures since 2026-02 (M-018, U-1, U-2, U-3, H-1, G-1, B1, B2);
pre-1.0 version was implying instability the chart no longer has.
OPS-L2 — External-Postgres path is now properly documented in values.yaml
(externalDatabase block with mode-2 example), README install command
links the existing examples/values-external-db.yaml, and the chart
truth table above proves the external mode renders cleanly.
Receipts:
helm lint deploy/helm/certctl/ # clean
helm template c deploy/helm/certctl/ \
--set server.tls.existingSecret=ci \
--set postgresql.auth.password=p \
--set server.auth.apiKey=k # 12 kinds, default
helm template c deploy/helm/certctl/ \
--set server.tls.existingSecret=ci \
--set postgresql.enabled=false \
--set externalDatabase.url='postgres://u:p@h:5432/db?sslmode=require' \
--set server.auth.apiKey=k # 9 kinds, no postgres-*
helm template c deploy/helm/certctl/ \
--set server.tls.certManager.enabled=true \
--set server.tls.certManager.issuerRef.name=letsencrypt \
--set postgresql.auth.password=p --set server.auth.apiKey=k
# +1 Certificate (cert-manager)
helm template c deploy/helm/certctl/ \
--set server.tls.existingSecret=ci \
--set postgresql.auth.password=p --set server.auth.apiKey=k \
--set server.replicas=3 \
--set monitoring.enabled=true \
--set monitoring.serviceMonitor.enabled=true \
--set podDisruptionBudget.enabled=true \
--set networkPolicy.enabled=true # +ServiceMonitor +PDB +NetworkPolicy
(TLS both-set + missing apiKey + missing pg password + missing extDb URL all REFUSED.)
gofmt -l # clean
go vet ./internal/repository/postgres ./cmd/server # clean
go build ./cmd/server # clean
bash scripts/ci-guards/B3-helm-chart-coherence.sh # clean
Remaining operator warnings (deferred, tracked in WORKSPACE-ROADMAP.md):
- Backup CronJob + restore script (D6 + OPS-H1): operator chooses
target (S3, GCS, Azure Blob, NFS). Sample CronJob yaml may ship
in deploy/helm/examples/ once an operator workstation has run
one full backup-restore cycle.
- Distributed tracing (OPS-M2): otel/* are go.mod indirect deps,
not actively instrumented. Adding spans is a v3 work item.
- Prometheus client_golang migration (OPS-M1): the hand-rolled
/metrics/prometheus exposition format works today; client_golang
migration unlocks histograms + exemplars + native label sets.
Audit-Closes: BUNDLE-3 C2 C3 D1 D2 D3 D5 D7 D11 D12 OPS-L1 OPS-L2
Audit-Defers: D6 D10 OPS-H1 OPS-M1 OPS-M2
Certctl Helm Chart
Production-ready Helm chart for deploying certctl (self-hosted certificate lifecycle management platform) on Kubernetes.
Table of Contents
- Quick Start
- Chart Features
- Prerequisites
- Installation
- Configuration
- Usage Examples
- Upgrading
- Uninstalling
- Architecture
- Security Considerations
- Troubleshooting
Quick Start
# Add the chart repository (when available)
helm repo add certctl https://charts.example.com
helm repo update
# Install with default values
helm install certctl certctl/certctl \
--set server.auth.apiKey="your-secure-api-key" \
--set postgresql.auth.password="your-secure-password"
# Check installation status
kubectl get pods -l app.kubernetes.io/instance=certctl
Chart Features
- Server Deployment — certctl control plane with configurable replicas
- PostgreSQL StatefulSet — Persistent database with automatic schema migration
- Agent DaemonSet or Deployment — Flexible agent deployment (per-node or custom replicas)
- Ingress Support — Optional HTTPS ingress with cert-manager integration
- Security Contexts — Non-root containers, read-only filesystems, minimal capabilities
- Resource Limits — Configurable CPU and memory requests/limits
- Health Checks — Liveness and readiness probes on all containers
- ConfigMaps and Secrets — Centralized configuration management
- Service Account and RBAC — Optional cluster role bindings
- Pod Disruption Budgets — HA-ready with configurable disruption budgets
- Monitoring — Optional Prometheus ServiceMonitor support
Prerequisites
- Kubernetes 1.19 or later
- Helm 3.0 or later
- Optional: cert-manager (for automatic TLS certificate provisioning)
- Optional: Prometheus (for metrics scraping)
Installation
1. Using Chart from Repository
helm repo add certctl https://charts.example.com
helm repo update
helm install certctl certctl/certctl -f my-values.yaml
2. Using Local Chart
cd deploy/helm
helm install certctl certctl/ \
--set server.auth.apiKey="$(openssl rand -base64 32)" \
--set postgresql.auth.password="$(openssl rand -base64 32)"
3. Minimal Production Installation
helm install certctl certctl/certctl \
--namespace certctl \
--create-namespace \
--set server.auth.apiKey="change-me" \
--set postgresql.auth.password="change-me" \
--set server.replicas=2 \
--set server.resources.requests.cpu=200m \
--set server.resources.requests.memory=256Mi \
--set ingress.enabled=true \
--set ingress.className=nginx \
--set ingress.hosts[0].host=certctl.example.com
Configuration
Server Configuration
server:
replicas: 1 # Number of server replicas
port: 8443 # Service port
auth:
type: api-key # Authentication type
apiKey: "your-api-key" # REQUIRED for production
logging:
level: info # Log level (debug, info, warn, error)
format: json # Output format
issuer:
local:
enabled: true # Enable local CA issuer
acme:
enabled: false # Enable ACME issuer
directoryURL: "" # ACME directory URL
email: "" # ACME registration email
challengeType: "http-01" # Challenge type (http-01, dns-01, dns-persist-01)
PostgreSQL Configuration
postgresql:
enabled: true # Use managed PostgreSQL
auth:
database: certctl
username: certctl
password: "your-password" # REQUIRED
storage:
size: 10Gi # PVC size
storageClass: "" # Use default StorageClass
Agent Configuration
agent:
enabled: true # Deploy agents
kind: DaemonSet # DaemonSet (one per node) or Deployment
replicas: 1 # For Deployment kind only
discoveryDirs: "" # Comma-separated cert discovery paths
nodeSelector: {} # Node affinity for DaemonSet
Ingress Configuration
ingress:
enabled: false
className: nginx
annotations:
cert-manager.io/cluster-issuer: letsencrypt-prod
hosts:
- host: certctl.example.com
paths:
- path: /
pathType: Prefix
tls:
- secretName: certctl-tls
hosts:
- certctl.example.com
See values.yaml for all available configuration options.
Usage Examples
Example 1: High Availability Setup
# ha-values.yaml
server:
replicas: 3
resources:
requests:
cpu: 250m
memory: 256Mi
limits:
cpu: 1000m
memory: 512Mi
postgresql:
storage:
size: 50Gi
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: app.kubernetes.io/component
operator: In
values: [server]
topologyKey: kubernetes.io/hostname
Deploy with:
helm install certctl certctl/certctl -f ha-values.yaml
Example 2: External PostgreSQL Database
# external-db-values.yaml
postgresql:
enabled: false
server:
env:
CERTCTL_DATABASE_URL: "postgres://user:password@rds.example.com:5432/certctl?sslmode=require"
Deploy with:
helm install certctl certctl/certctl -f external-db-values.yaml
Example 3: ACME + Let's Encrypt
# acme-values.yaml
server:
issuer:
acme:
enabled: true
directoryURL: https://acme-v02.api.letsencrypt.org/directory
email: admin@example.com
challengeType: dns-01
dnsPresentScript: /scripts/dns-present.sh
dnsCleanupScript: /scripts/dns-cleanup.sh
dnsPropagationWait: 30s
Example 4: Email Notifications via Slack + SMTP
# notifications-values.yaml
server:
smtp:
enabled: true
host: smtp.example.com
port: 587
username: certctl@example.com
password: "smtp-password"
fromAddress: certctl@example.com
useTLS: true
notifiers:
slack:
enabled: true
webhookUrl: https://hooks.slack.com/services/YOUR/WEBHOOK/URL
channel: "#certificates"
Upgrading
# Update chart repository
helm repo update
# Upgrade release
helm upgrade certctl certctl/certctl -f values.yaml
# View upgrade history
helm history certctl
# Rollback to previous version
helm rollback certctl 1
Uninstalling
# Delete the release (keeps data by default)
helm uninstall certctl
# Also delete persistent data
kubectl delete pvc --all -l app.kubernetes.io/instance=certctl
# Delete namespace
kubectl delete namespace certctl
Architecture
Components
┌──────────────────────────────────────────────────────────────┐
│ Kubernetes Cluster │
├──────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────────┐ ┌──────────────────┐ │
│ │ Ingress/LB │ │ Agent Pod 1 │ │
│ │ (optional) │ │ (DaemonSet) │ │
│ └────────┬────────┘ └──────────────────┘ │
│ │ │
│ ▼ ┌──────────────────┐ │
│ ┌─────────────────────────┐ │ Agent Pod 2 │ │
│ │ Server Deployment │ │ (DaemonSet) │ │
│ │ (1 to N replicas) │ └──────────────────┘ │
│ │ - REST API │ │
│ │ - Scheduler │ ┌──────────────────┐ │
│ │ - UI Dashboard │ │ Agent Pod N │ │
│ └────────┬────────────────┘ │ (DaemonSet) │ │
│ │ └──────────────────┘ │
│ │ │
│ ▼ │
│ ┌──────────────────────────┐ │
│ │ PostgreSQL StatefulSet │ │
│ │ - Database │ │
│ │ - PVC (persistent) │ │
│ └──────────────────────────┘ │
│ │
└──────────────────────────────────────────────────────────────┘
Network Communication
- Server → PostgreSQL: Internal cluster DNS (
certctl-postgres:5432) - Agent → Server: Internal cluster DNS (
certctl-server:8443) - External → Server: Via Ingress or Service (ClusterIP/LoadBalancer/NodePort)
Security Considerations
1. Secrets Management
All sensitive data is stored in Kubernetes Secrets:
- PostgreSQL credentials
- API keys
- SMTP passwords
- ACME account secrets
Best Practices:
- Use sealed-secrets or external-secrets operator
- Enable encryption at rest in etcd
- Rotate secrets regularly
# Example: Using sealed-secrets
kubectl create secret generic certctl-api-key --from-literal=api-key="$(openssl rand -base64 32)" --dry-run=client -o yaml | kubeseal -f - | kubectl apply -f -
2. RBAC
The chart creates minimal RBAC by default:
- ServiceAccount per release
- ClusterRole (empty, extensible)
- ClusterRoleBinding
To restrict further:
rbac:
create: true
# Add specific rules here
3. Pod Security
All containers run with:
- Non-root user (UID 1000)
- Read-only root filesystem
- No privilege escalation
- Dropped capabilities (ALL)
4. Network Policies
Restrict pod-to-pod communication:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: certctl-default-deny
spec:
podSelector:
matchLabels:
app.kubernetes.io/instance: certctl
policyTypes:
- Ingress
- Egress
ingress:
- from:
- namespaceSelector:
matchLabels:
name: certctl
egress:
- to:
- namespaceSelector:
matchLabels:
name: certctl
- to:
- podSelector: {}
ports:
- protocol: TCP
port: 53 # DNS
- protocol: UDP
port: 53
5. TLS/HTTPS
Enable HTTPS with cert-manager:
helm install cert-manager jetstack/cert-manager \
--namespace cert-manager \
--create-namespace \
--set installCRDs=true
Then configure Ingress with TLS.
6. API Key Security
For production:
- Generate a strong API key:
openssl rand -base64 32 - Store securely (Vault, sealed-secrets, etc.)
- Never commit to Git
- Rotate periodically
# Generate and deploy API key
NEW_KEY=$(openssl rand -base64 32)
kubectl patch secret certctl-server -p "{\"data\":{\"api-key\":\"$(echo -n $NEW_KEY | base64)\"}}"
Troubleshooting
1. Pods Not Starting
# Check pod status
kubectl get pods -l app.kubernetes.io/instance=certctl
kubectl describe pod <pod-name>
kubectl logs <pod-name>
2. Database Connection Issues
# Verify PostgreSQL is running
kubectl get pods -l app.kubernetes.io/component=postgres
kubectl logs -l app.kubernetes.io/component=postgres
# Test connection from server pod
kubectl exec -it <server-pod> -- \
psql postgres://certctl:password@certctl-postgres:5432/certctl
3. Agent Not Connecting
# Check agent logs
kubectl logs -l app.kubernetes.io/component=agent
# Verify server is reachable
kubectl exec -it <agent-pod> -- \
wget -q -O - http://certctl-server:8443/health
4. Persistent Data Loss
# Check PVC status
kubectl get pvc
# Verify data is being stored
kubectl exec -it <postgres-pod> -- \
ls -lah /var/lib/postgresql/data/postgres
5. Permission Denied Errors
The chart runs containers as non-root (UID 1000). If you see permission errors:
# Temporarily allow root for debugging
server:
securityContext:
runAsUser: 0 # NOT FOR PRODUCTION
6. Out of Memory
Increase resource limits:
helm upgrade certctl certctl/certctl \
--set server.resources.limits.memory=1Gi \
--set postgresql.resources.limits.memory=2Gi
7. Certificate Validation Issues
For self-signed certificates:
kubectl exec -it <pod> -- \
CERTCTL_TLS_INSECURE_SKIP_VERIFY=true <command>
Common Issues and Solutions
| Issue | Solution |
|---|---|
ImagePullBackOff |
Update server.image.repository to your registry |
CrashLoopBackOff |
Check logs with kubectl logs <pod> |
Pending PVC |
Check storage class availability |
| Connection timeout | Verify network policies and service DNS |
| High memory usage | Adjust postgresql.resources.limits and server.resources.limits |
Support and Contributing
For issues, questions, or contributions, visit:
- GitHub: https://github.com/certctl-io/certctl
- Documentation: https://github.com/certctl-io/certctl/tree/main/docs
License
BSL-1.1