mirror of
https://github.com/shankar0123/certctl.git
synced 2026-06-07 16:51:31 +00:00
8191b1ee64
Phase 6 of the certctl architecture diligence remediation. Five
findings across the same scheduler-and-DB-pool surface.
SCALE-M1 (Med) — DB pool default bumped 25 → 50
internal/config/config.go line 1972:
MaxConnections: getEnvInt("CERTCTL_DATABASE_MAX_CONNS", 50)
Postgres default max_connections is 100; 50 leaves headroom for
pg_dump + ad-hoc psql + a server replica without exhausting the
DB-side cap. Operator override env var unchanged. Operator-tune
ladder for larger fleets (5K / 50K certs) lives in
docs/operator/scale.md as starter values pending Phase 8 load
tests — explicitly marked TBD.
SCALE-M3 (Med) — async-CA poll budget operator-configurable
Live state was partially-already-shipped: all 4 async-CA
connectors (digicert, entrust, globalsign, sectigo) already have
per-connector CERTCTL_<NAME>_POLL_MAX_WAIT_SECONDS (Audit fix #5
closed pre-Phase-6). What was missing: a global package-default
override. Shipped:
- internal/connector/issuer/asyncpoll/asyncpoll.go gains
SetDefaultMaxWait(d) + effectiveDefaultMaxWait var + the
currentDefaultMaxWait() priority resolver.
- cmd/server/main.go reads CERTCTL_ASYNC_POLL_MAX_WAIT_SECONDS
at boot and calls SetDefaultMaxWait.
- deploy/ENVIRONMENTS.md documents the new env var (G-3 guard
green).
Naming deviation from the prompt's CERTCTL_ASYNC_POLL_MAX_ATTEMPTS:
the live code tracks wall-clock time (MaxWait), not attempt count.
Matched the existing per-connector nomenclature (_POLL_MAX_WAIT_SECONDS)
so the priority chain reads naturally.
SCALE-M5 (Med) — JitteredTicker wrapper for all 15 scheduler loops
internal/scheduler/jitter.go ships NewJitteredTicker(interval,
jitterPct) + DefaultSchedulerJitter (±10%). All 15 sites in
internal/scheduler/scheduler.go migrated from bare time.NewTicker
to NewJitteredTicker(interval, DefaultSchedulerJitter). Base
intervals unchanged; only the per-tick envelope adds ±10%
randomized delay so multiple loops with the same nominal cadence
don't co-fire and spike CPU + DB at wall-clock boundaries.
internal/scheduler/jitter_test.go pins:
- Bounded envelope (each tick within ±jitterPct of interval)
- Mean drift < 30% of nominal (sign-bug detector)
- Stop() releases the goroutine + closes C
- Stop() idempotent (no panic on repeat)
- Zero-jitter behaves like time.NewTicker
- Negative and >=1 jitterPct values clamped defensively
CI guard scripts/ci-guards/no-bare-newticker-in-scheduler.sh blocks
any future bare time.NewTicker in scheduler.go.
SCALE-L1 (Low) — renewal-sweep semaphore behavior documented
docs/operator/scale.md "Scheduler tick budgets" section explains
the per-tick concurrency semaphore (CERTCTL_RENEWAL_CONCURRENCY=25
default), the ctx-cancellation drain on tick-budget overrun, and
operator tuning advice (raise concurrency + DB pool together).
No code change — the behavior is defensible as-is per the audit.
SCALE-L2 (Low) — ETag middleware for top-5 read endpoints
internal/api/middleware/etag.go computes SHA-256 ETag over the
buffered response body, respects If-None-Match, short-circuits
to 304 Not Modified on match. GET/HEAD only; non-2xx responses
pass through unchanged. 64 KiB buffer cap degrades gracefully on
oversized responses (no caching, body still flushes intact).
Wired around the top-5 read endpoints via etagged() helper in
internal/api/router/router.go:
GET /api/v1/certificates
GET /api/v1/agents
GET /api/v1/jobs
GET /api/v1/audit
GET /api/v1/discovered-certificates
internal/api/middleware/etag_test.go pins 11 behaviors including
304-on-repeat, 200-after-mutation-with-new-ETag, POST bypass,
4xx/5xx pass-through, oversized-response degradation, wildcard
match, HEAD-treated-like-GET, byte-equal pass-through.
Cross-cutting fixes:
- internal/config/config_test.go::TestLoad_DefaultValues updated
to assert the new 50 default (was 25).
- deploy/helm/certctl/values.yaml comment corrected — agent
pollInterval is hardcoded 30s, not env-configurable; the
Phase 4 comment mistakenly referenced CERTCTL_AGENT_POLL_INTERVAL
which G-3 caught as a phantom env var.
- asyncpoll.go reformatted by gofmt; functionally unchanged.
Verification (all pass):
grep -nE 'SetMaxOpenConns' internal/repository/postgres/db.go # finds 1 site
grep -nE 'CERTCTL_DATABASE_MAX_CONNS.*50' internal/config/config.go # config default is 50
grep -rnE 'CERTCTL_ASYNC_POLL_MAX_WAIT_SECONDS' internal/ deploy/ENVIRONMENTS.md # wired
grep -cE 'time\.NewTicker\(' internal/scheduler/scheduler.go # 0 (all migrated)
grep -cE 'JitteredTicker' internal/scheduler/scheduler.go # 15
ls internal/scheduler/jitter.go internal/api/middleware/etag.go # both exist
ls docs/operator/scale.md # exists
bash scripts/ci-guards/no-bare-newticker-in-scheduler.sh # clean
bash scripts/ci-guards/G-3-env-docs-drift.sh # clean
go test ./internal/scheduler/ ./internal/api/middleware/ \
./internal/connector/issuer/asyncpoll/ ./internal/config/ # 4/4 packages green
Closes: cowork/certctl-architecture-diligence-audit.html#fix-SCALE-M1
cowork/certctl-architecture-diligence-audit.html#fix-SCALE-M3
cowork/certctl-architecture-diligence-audit.html#fix-SCALE-M5
cowork/certctl-architecture-diligence-audit.html#fix-SCALE-L1
cowork/certctl-architecture-diligence-audit.html#fix-SCALE-L2
Certctl Helm Chart
Production-ready Helm chart for deploying certctl (self-hosted certificate lifecycle management platform) on Kubernetes.
Table of Contents
- Quick Start
- Chart Features
- Prerequisites
- Installation
- Configuration
- Usage Examples
- Upgrading
- Uninstalling
- Architecture
- Security Considerations
- Troubleshooting
Quick Start
# Add the chart repository (when available)
helm repo add certctl https://charts.example.com
helm repo update
# Install with default values
helm install certctl certctl/certctl \
--set server.auth.apiKey="your-secure-api-key" \
--set postgresql.auth.password="your-secure-password"
# Check installation status
kubectl get pods -l app.kubernetes.io/instance=certctl
Chart Features
- Server Deployment — certctl control plane with configurable replicas
- PostgreSQL StatefulSet — Persistent database with automatic schema migration
- Agent DaemonSet or Deployment — Flexible agent deployment (per-node or custom replicas)
- Ingress Support — Optional HTTPS ingress with cert-manager integration
- Security Contexts — Non-root containers, read-only filesystems, minimal capabilities
- Resource Limits — Configurable CPU and memory requests/limits
- Health Checks — Liveness and readiness probes on all containers
- ConfigMaps and Secrets — Centralized configuration management
- Service Account and RBAC — Optional cluster role bindings
- Pod Disruption Budgets — HA-ready with configurable disruption budgets
- Monitoring — Optional Prometheus ServiceMonitor support
Prerequisites
- Kubernetes 1.19 or later
- Helm 3.0 or later
- Optional: cert-manager (for automatic TLS certificate provisioning)
- Optional: Prometheus (for metrics scraping)
Installation
1. Using Chart from Repository
helm repo add certctl https://charts.example.com
helm repo update
helm install certctl certctl/certctl -f my-values.yaml
2. Using Local Chart
cd deploy/helm
helm install certctl certctl/ \
--set server.auth.apiKey="$(openssl rand -base64 32)" \
--set postgresql.auth.password="$(openssl rand -base64 32)"
3. Minimal Production Installation
helm install certctl certctl/certctl \
--namespace certctl \
--create-namespace \
--set server.auth.apiKey="change-me" \
--set postgresql.auth.password="change-me" \
--set server.replicas=2 \
--set server.resources.requests.cpu=200m \
--set server.resources.requests.memory=256Mi \
--set ingress.enabled=true \
--set ingress.className=nginx \
--set ingress.hosts[0].host=certctl.example.com
Configuration
Server Configuration
server:
replicas: 1 # Number of server replicas
port: 8443 # Service port
auth:
type: api-key # Authentication type
apiKey: "your-api-key" # REQUIRED for production
logging:
level: info # Log level (debug, info, warn, error)
format: json # Output format
issuer:
local:
enabled: true # Enable local CA issuer
acme:
enabled: false # Enable ACME issuer
directoryURL: "" # ACME directory URL
email: "" # ACME registration email
challengeType: "http-01" # Challenge type (http-01, dns-01, dns-persist-01)
PostgreSQL Configuration
postgresql:
enabled: true # Use managed PostgreSQL
auth:
database: certctl
username: certctl
password: "your-password" # REQUIRED
storage:
size: 10Gi # PVC size
storageClass: "" # Use default StorageClass
Agent Configuration
agent:
enabled: true # Deploy agents
kind: DaemonSet # DaemonSet (one per node) or Deployment
replicas: 1 # For Deployment kind only
discoveryDirs: "" # Comma-separated cert discovery paths
nodeSelector: {} # Node affinity for DaemonSet
Ingress Configuration
ingress:
enabled: false
className: nginx
annotations:
cert-manager.io/cluster-issuer: letsencrypt-prod
hosts:
- host: certctl.example.com
paths:
- path: /
pathType: Prefix
tls:
- secretName: certctl-tls
hosts:
- certctl.example.com
See values.yaml for all available configuration options.
Usage Examples
Example 1: High Availability Setup
# ha-values.yaml
server:
replicas: 3
resources:
requests:
cpu: 250m
memory: 256Mi
limits:
cpu: 1000m
memory: 512Mi
postgresql:
storage:
size: 50Gi
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: app.kubernetes.io/component
operator: In
values: [server]
topologyKey: kubernetes.io/hostname
Deploy with:
helm install certctl certctl/certctl -f ha-values.yaml
Example 2: External PostgreSQL Database
# external-db-values.yaml
postgresql:
enabled: false
server:
env:
CERTCTL_DATABASE_URL: "postgres://user:password@rds.example.com:5432/certctl?sslmode=require"
Deploy with:
helm install certctl certctl/certctl -f external-db-values.yaml
Example 3: ACME + Let's Encrypt
# acme-values.yaml
server:
issuer:
acme:
enabled: true
directoryURL: https://acme-v02.api.letsencrypt.org/directory
email: admin@example.com
challengeType: dns-01
dnsPresentScript: /scripts/dns-present.sh
dnsCleanupScript: /scripts/dns-cleanup.sh
dnsPropagationWait: 30s
Example 4: Email Notifications via Slack + SMTP
# notifications-values.yaml
server:
smtp:
enabled: true
host: smtp.example.com
port: 587
username: certctl@example.com
password: "smtp-password"
fromAddress: certctl@example.com
useTLS: true
notifiers:
slack:
enabled: true
webhookUrl: https://hooks.slack.com/services/YOUR/WEBHOOK/URL
channel: "#certificates"
Upgrading
# Update chart repository
helm repo update
# Upgrade release
helm upgrade certctl certctl/certctl -f values.yaml
# View upgrade history
helm history certctl
# Rollback to previous version
helm rollback certctl 1
Uninstalling
# Delete the release (keeps data by default)
helm uninstall certctl
# Also delete persistent data
kubectl delete pvc --all -l app.kubernetes.io/instance=certctl
# Delete namespace
kubectl delete namespace certctl
Architecture
Components
┌──────────────────────────────────────────────────────────────┐
│ Kubernetes Cluster │
├──────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────────┐ ┌──────────────────┐ │
│ │ Ingress/LB │ │ Agent Pod 1 │ │
│ │ (optional) │ │ (DaemonSet) │ │
│ └────────┬────────┘ └──────────────────┘ │
│ │ │
│ ▼ ┌──────────────────┐ │
│ ┌─────────────────────────┐ │ Agent Pod 2 │ │
│ │ Server Deployment │ │ (DaemonSet) │ │
│ │ (1 to N replicas) │ └──────────────────┘ │
│ │ - REST API │ │
│ │ - Scheduler │ ┌──────────────────┐ │
│ │ - UI Dashboard │ │ Agent Pod N │ │
│ └────────┬────────────────┘ │ (DaemonSet) │ │
│ │ └──────────────────┘ │
│ │ │
│ ▼ │
│ ┌──────────────────────────┐ │
│ │ PostgreSQL StatefulSet │ │
│ │ - Database │ │
│ │ - PVC (persistent) │ │
│ └──────────────────────────┘ │
│ │
└──────────────────────────────────────────────────────────────┘
Network Communication
- Server → PostgreSQL: Internal cluster DNS (
certctl-postgres:5432) - Agent → Server: Internal cluster DNS (
certctl-server:8443) - External → Server: Via Ingress or Service (ClusterIP/LoadBalancer/NodePort)
Security Considerations
1. Secrets Management
All sensitive data is stored in Kubernetes Secrets:
- PostgreSQL credentials
- API keys
- SMTP passwords
- ACME account secrets
Best Practices:
- Use sealed-secrets or external-secrets operator
- Enable encryption at rest in etcd
- Rotate secrets regularly
# Example: Using sealed-secrets
kubectl create secret generic certctl-api-key --from-literal=api-key="$(openssl rand -base64 32)" --dry-run=client -o yaml | kubeseal -f - | kubectl apply -f -
2. RBAC
The chart creates minimal RBAC by default:
- ServiceAccount per release
- ClusterRole (empty, extensible)
- ClusterRoleBinding
To restrict further:
rbac:
create: true
# Add specific rules here
3. Pod Security
All containers run with:
- Non-root user (UID 1000)
- Read-only root filesystem
- No privilege escalation
- Dropped capabilities (ALL)
4. Network Policies
Restrict pod-to-pod communication:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: certctl-default-deny
spec:
podSelector:
matchLabels:
app.kubernetes.io/instance: certctl
policyTypes:
- Ingress
- Egress
ingress:
- from:
- namespaceSelector:
matchLabels:
name: certctl
egress:
- to:
- namespaceSelector:
matchLabels:
name: certctl
- to:
- podSelector: {}
ports:
- protocol: TCP
port: 53 # DNS
- protocol: UDP
port: 53
5. TLS/HTTPS
Enable HTTPS with cert-manager:
helm install cert-manager jetstack/cert-manager \
--namespace cert-manager \
--create-namespace \
--set installCRDs=true
Then configure Ingress with TLS.
6. API Key Security
For production:
- Generate a strong API key:
openssl rand -base64 32 - Store securely (Vault, sealed-secrets, etc.)
- Never commit to Git
- Rotate periodically
# Generate and deploy API key
NEW_KEY=$(openssl rand -base64 32)
kubectl patch secret certctl-server -p "{\"data\":{\"api-key\":\"$(echo -n $NEW_KEY | base64)\"}}"
Troubleshooting
1. Pods Not Starting
# Check pod status
kubectl get pods -l app.kubernetes.io/instance=certctl
kubectl describe pod <pod-name>
kubectl logs <pod-name>
2. Database Connection Issues
# Verify PostgreSQL is running
kubectl get pods -l app.kubernetes.io/component=postgres
kubectl logs -l app.kubernetes.io/component=postgres
# Test connection from server pod
kubectl exec -it <server-pod> -- \
psql postgres://certctl:password@certctl-postgres:5432/certctl
3. Agent Not Connecting
# Check agent logs
kubectl logs -l app.kubernetes.io/component=agent
# Verify server is reachable
kubectl exec -it <agent-pod> -- \
wget -q -O - http://certctl-server:8443/health
4. Persistent Data Loss
# Check PVC status
kubectl get pvc
# Verify data is being stored
kubectl exec -it <postgres-pod> -- \
ls -lah /var/lib/postgresql/data/postgres
5. Permission Denied Errors
The chart runs containers as non-root (UID 1000). If you see permission errors:
# Temporarily allow root for debugging
server:
securityContext:
runAsUser: 0 # NOT FOR PRODUCTION
6. Out of Memory
Increase resource limits:
helm upgrade certctl certctl/certctl \
--set server.resources.limits.memory=1Gi \
--set postgresql.resources.limits.memory=2Gi
7. Certificate Validation Issues
For self-signed certificates:
kubectl exec -it <pod> -- \
CERTCTL_TLS_INSECURE_SKIP_VERIFY=true <command>
Common Issues and Solutions
| Issue | Solution |
|---|---|
ImagePullBackOff |
Update server.image.repository to your registry |
CrashLoopBackOff |
Check logs with kubectl logs <pod> |
Pending PVC |
Check storage class availability |
| Connection timeout | Verify network policies and service DNS |
| High memory usage | Adjust postgresql.resources.limits and server.resources.limits |
Support and Contributing
For issues, questions, or contributions, visit:
- GitHub: https://github.com/certctl-io/certctl
- Documentation: https://github.com/certctl-io/certctl/tree/main/docs
License
BSL-1.1