mirror of
https://github.com/shankar0123/certctl.git
synced 2026-06-07 17:51:29 +00:00
feat(pre-2.1.0): demo data overhaul, examples, migration guides, install script
Pre-2.1.0 adoption polish delivering all four milestones: A) Demo Data Overhaul — seed_demo.sql rewritten with 35 certs across 5 issuers, 8 agents, 8 targets, 50+ jobs spanning 90 days, 55+ audit events, discovery scans, network scan targets, S/MIME cert. B) Examples Directory — 5 turnkey docker-compose configs: acme-nginx, acme-wildcard-dns01, private-ca-traefik, step-ca-haproxy, multi-issuer. C) Migration Guides — migrate-from-certbot.md, migrate-from-acmesh.md, certctl-for-cert-manager-users.md. D) Agent Install Script — install-agent.sh with cross-platform support (Linux systemd + macOS launchd), release.yml updated for 6-target cross-compilation. Triple-audited against codebase: 22 factual corrections applied across docs, examples, and config (env var names, CLI flags, ports, DNS hook interface, scheduler loop counts, license conversion date). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Shankar
@@ -0,0 +1,370 @@
|
||||
# certctl + NGINX + Let's Encrypt
|
||||
|
||||
This example demonstrates certctl's core use case: **automatically manage TLS certificates for NGINX using Let's Encrypt (ACME HTTP-01 challenges).**
|
||||
|
||||
## What This Does
|
||||
|
||||
- Deploys certctl server (control plane) with PostgreSQL
|
||||
- Deploys certctl agent on the same network (in production: on your NGINX server)
|
||||
- Configures Let's Encrypt as the certificate issuer via ACME v2
|
||||
- Demonstrates HTTP-01 challenge solving (requires port 80 open to the internet)
|
||||
- Shows how to set up 3 example domains for certificate enrollment and renewal
|
||||
- Automatically renews certificates 30 days before expiration
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
Your Domain (example.com)
|
||||
↓ [HTTP-01 validation, port 80]
|
||||
Let's Encrypt ACME
|
||||
↓ [CSR submission]
|
||||
certctl Server (control plane)
|
||||
↓ [API polling]
|
||||
certctl Agent (on NGINX server)
|
||||
↓ [deploy cert+key]
|
||||
NGINX Reverse Proxy
|
||||
```
|
||||
|
||||
## Prerequisites
|
||||
|
||||
1. **Docker & Docker Compose** (v20.10+)
|
||||
2. **A domain name** pointing to your server (e.g., `example.com`)
|
||||
3. **Ports 80 and 443 open** to the internet (ACME HTTP-01 needs port 80)
|
||||
4. **Valid email address** for Let's Encrypt account (errors and renewal notices)
|
||||
|
||||
If you don't have a real domain or can't open port 80, see [Customization Tips](#customization-tips) below.
|
||||
|
||||
## Quick Start
|
||||
|
||||
### 1. Clone or copy this example
|
||||
|
||||
```bash
|
||||
cd examples/acme-nginx
|
||||
```
|
||||
|
||||
### 2. Create a `.env` file with your settings
|
||||
|
||||
```bash
|
||||
cat > .env <<'EOF'
|
||||
# Your email for Let's Encrypt account
|
||||
ACME_EMAIL=admin@example.com
|
||||
|
||||
# Database password (change this in production!)
|
||||
DB_PASSWORD=certctl-demo-password
|
||||
|
||||
# Agent API key (generate a real one in production)
|
||||
AGENT_API_KEY=agent-demo-key
|
||||
|
||||
# Server port (certctl listens here internally on 8443; expose as needed)
|
||||
SERVER_PORT=8443
|
||||
EOF
|
||||
```
|
||||
|
||||
### 3. (Optional) Create an NGINX config
|
||||
|
||||
If you have a real domain and want NGINX to route traffic:
|
||||
|
||||
```bash
|
||||
cat > nginx.conf <<'EOF'
|
||||
events {
|
||||
worker_connections 1024;
|
||||
}
|
||||
|
||||
http {
|
||||
# HTTP block for ACME challenges
|
||||
server {
|
||||
listen 80;
|
||||
server_name example.com www.example.com api.example.com;
|
||||
|
||||
# ACME challenge directory (certctl writes validation files here)
|
||||
location /.well-known/acme-challenge/ {
|
||||
root /var/www/certbot;
|
||||
}
|
||||
|
||||
# Redirect HTTP to HTTPS
|
||||
location / {
|
||||
return 301 https://$server_name$request_uri;
|
||||
}
|
||||
}
|
||||
|
||||
# HTTPS block (certificates deployed here by certctl agent)
|
||||
server {
|
||||
listen 443 ssl http2;
|
||||
server_name example.com www.example.com api.example.com;
|
||||
|
||||
ssl_certificate /etc/nginx/ssl/example.com.crt;
|
||||
ssl_certificate_key /etc/nginx/ssl/example.com.key;
|
||||
ssl_protocols TLSv1.2 TLSv1.3;
|
||||
ssl_ciphers HIGH:!aNULL:!MD5;
|
||||
|
||||
location / {
|
||||
proxy_pass http://upstream-service;
|
||||
}
|
||||
}
|
||||
}
|
||||
EOF
|
||||
```
|
||||
|
||||
Or just accept the default empty NGINX config for demonstration.
|
||||
|
||||
### 4. Start the stack
|
||||
|
||||
```bash
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
Monitor logs:
|
||||
```bash
|
||||
docker compose logs -f certctl-server certctl-agent
|
||||
```
|
||||
|
||||
### 5. Access the dashboard
|
||||
|
||||
Navigate to `http://localhost:8443` (or your `SERVER_PORT`)
|
||||
|
||||
You should see:
|
||||
- An empty certificate inventory (no certs issued yet)
|
||||
- One ACME issuer ("iss-acme") configured and ready
|
||||
- One agent ("nginx-agent-01") online and heartbeating
|
||||
|
||||
### 6. Create a certificate profile
|
||||
|
||||
In the certctl dashboard:
|
||||
1. Go to **Profiles** (sidebar)
|
||||
2. Click **New Profile**
|
||||
3. Set:
|
||||
- Name: `acme-prod`
|
||||
- Key Type: `RSA-2048` (or `ECDSA-P256`)
|
||||
- Max TTL: `90 days`
|
||||
- Allowed Key Types: `RSA-2048, ECDSA-P256`
|
||||
4. Save
|
||||
|
||||
### 7. Request a certificate
|
||||
|
||||
In the certctl dashboard:
|
||||
1. Go to **Certificates** (sidebar)
|
||||
2. Click **Request New Certificate**
|
||||
3. Set:
|
||||
- Common Name: `example.com`
|
||||
- SANs: `www.example.com`, `api.example.com` (optional)
|
||||
- Issuer: `iss-acme` (Let's Encrypt)
|
||||
- Profile: `acme-prod`
|
||||
4. Click **Request**
|
||||
|
||||
Behind the scenes:
|
||||
- Server creates an `Issuance` job
|
||||
- Agent polls for work, fetches the job
|
||||
- Agent generates a P-256 key (never sent to server)
|
||||
- Agent submits CSR to server
|
||||
- Server sends CSR to Let's Encrypt ACME
|
||||
- Let's Encrypt provides HTTP-01 challenge token
|
||||
- Server downloads ACME challenge, returns to agent
|
||||
- Agent deploys challenge file to NGINX `/.well-known/acme-challenge/`
|
||||
- Let's Encrypt validates (HTTP GET to `http://example.com/.well-known/acme-challenge/...`)
|
||||
- Let's Encrypt issues certificate
|
||||
- Server receives certificate, passes to agent
|
||||
- Agent deploys cert+key to `/etc/nginx/ssl/example.com.crt` + `.key`
|
||||
- Agent reloads NGINX (`nginx -s reload`)
|
||||
- Certificate is now active
|
||||
|
||||
### 8. View the certificate
|
||||
|
||||
In the dashboard:
|
||||
1. Go to **Certificates**
|
||||
2. Click the certificate to see:
|
||||
- Common name, SANs, serial number
|
||||
- Issuer (Let's Encrypt), not-before/after dates
|
||||
- Status (Active, Expiring in N days, Expired)
|
||||
- Deployment history (timestamps, agent name, target)
|
||||
- Next auto-renewal date (30 days before expiration)
|
||||
|
||||
### 9. Set up automatic renewal
|
||||
|
||||
The server automatically checks for certificates expiring within 30 days and triggers renewal. You can:
|
||||
- Adjust the threshold in the certificate's policy
|
||||
- Manually trigger renewal via dashboard button
|
||||
- View renewal job status and history
|
||||
|
||||
## How It Works
|
||||
|
||||
### Certificate Lifecycle
|
||||
|
||||
1. **Request** — Operator creates certificate request via dashboard or API
|
||||
2. **CSR Generation** — Agent generates private key locally, submits CSR to server
|
||||
3. **ACME Challenge** — Server communicates with Let's Encrypt ACME, obtains challenge
|
||||
4. **Challenge Proof** — Agent deploys challenge proof to NGINX
|
||||
5. **Issuance** — Let's Encrypt validates, issues certificate
|
||||
6. **Deployment** — Agent receives certificate, deploys to NGINX SSL directory
|
||||
7. **Reload** — Agent signals NGINX to reload (`nginx -s reload`)
|
||||
8. **Verification** — Agent optionally verifies the live TLS endpoint (handshake fingerprint)
|
||||
9. **Renewal** — 30 days before expiration, process repeats automatically
|
||||
|
||||
### HTTP-01 Challenge
|
||||
|
||||
ACME HTTP-01 works like this:
|
||||
1. Let's Encrypt generates random token (e.g., `abc123def456`)
|
||||
2. Server returns token to agent
|
||||
3. Agent writes file: `/.well-known/acme-challenge/abc123def456` with value (random key material)
|
||||
4. Let's Encrypt performs HTTP GET to `http://example.com/.well-known/acme-challenge/abc123def456`
|
||||
5. If content matches, domain ownership is proven
|
||||
6. Certificate is issued
|
||||
|
||||
**Requirements:**
|
||||
- Port 80 must be open to the internet
|
||||
- DNS must resolve your domain to your server
|
||||
- NGINX must serve `/.well-known/acme-challenge/` (or certctl mounts a separate directory)
|
||||
|
||||
### Agent Key Generation
|
||||
|
||||
Keys are generated **on the agent**, never on the server:
|
||||
1. Agent creates ECDSA P-256 keypair using `crypto/ecdsa`
|
||||
2. Private key is stored locally on agent at `/var/lib/certctl/keys/` (readable only by certctl process)
|
||||
3. Agent creates CSR (certificate signing request) with private key
|
||||
4. Agent submits CSR to server
|
||||
5. Server never sees the private key
|
||||
6. Certificate is returned, agent stores it alongside key
|
||||
7. Both key and cert used for NGINX deployment
|
||||
|
||||
This keeps private keys in the infrastructure where they're used, following zero-trust principles.
|
||||
|
||||
## Adding More Domains
|
||||
|
||||
### Option 1: Additional SANs on Same Certificate
|
||||
|
||||
Edit the existing certificate in the dashboard:
|
||||
1. Click the certificate
|
||||
2. Edit SANs to add `mail.example.com`, `ftp.example.com`, etc.
|
||||
3. Trigger renewal
|
||||
4. Agent generates new CSR with all SANs
|
||||
5. Let's Encrypt validates each SAN (HTTP-01 for each)
|
||||
6. Single certificate with multiple SANs is issued
|
||||
|
||||
### Option 2: Separate Certificates per Domain
|
||||
|
||||
If you want separate certificates (different issuance schedules, different targets):
|
||||
1. Dashboard → **Certificates** → **Request New Certificate**
|
||||
2. Common Name: `subdomain.example.com`
|
||||
3. Set same issuer and profile
|
||||
4. Request
|
||||
|
||||
Each domain gets its own cert, key, and renewal schedule.
|
||||
|
||||
### Wildcard Certificates (Not HTTP-01)
|
||||
|
||||
HTTP-01 does **not** support wildcard (`*.example.com`). To issue wildcards, use DNS-01 challenge (see [acme-wildcard-dns01](../acme-wildcard-dns01/) example).
|
||||
|
||||
## Customization Tips
|
||||
|
||||
### Using Let's Encrypt Staging (for testing)
|
||||
|
||||
Staging has higher rate limits and doesn't require real domains:
|
||||
|
||||
```bash
|
||||
# In .env or docker-compose.yml override:
|
||||
CERTCTL_ACME_DIRECTORY_URL=https://acme-staging-v02.api.letsencrypt.org/directory
|
||||
```
|
||||
|
||||
Staging certificates won't be trusted by browsers (fake CA), but you can test the full flow without hitting production rate limits.
|
||||
|
||||
### Disabling Port 80 Requirement (Demo Mode)
|
||||
|
||||
If you can't open port 80, use ACME DNS-01 instead (requires DNS provider integration). See [acme-wildcard-dns01](../acme-wildcard-dns01/) example.
|
||||
|
||||
Or use Local CA for internal testing:
|
||||
```bash
|
||||
# Switch issuer to Local CA (not public-trusted, but no challenge needed)
|
||||
CERTCTL_ACME_DIRECTORY_URL= # Leave empty to disable ACME
|
||||
# (then configure Local CA instead)
|
||||
```
|
||||
|
||||
### Custom NGINX Config
|
||||
|
||||
Replace `nginx.conf` with your own before `docker compose up`. The agent doesn't manage the NGINX config — it only deploys certificates. You're responsible for:
|
||||
- Configuring SSL paths (`ssl_certificate`, `ssl_certificate_key`)
|
||||
- Setting up challenge directory (`/.well-known/acme-challenge/`)
|
||||
- Pointing NGINX to agent-deployed certificates
|
||||
|
||||
### Database Persistence
|
||||
|
||||
PostgreSQL data is stored in the `postgres_data` volume. To reset:
|
||||
```bash
|
||||
docker compose down -v # Destroy all volumes
|
||||
```
|
||||
|
||||
### Viewing Agent Logs
|
||||
|
||||
```bash
|
||||
docker compose logs -f certctl-agent
|
||||
```
|
||||
|
||||
Look for:
|
||||
- `Heartbeat successful` — agent is communicating with server
|
||||
- `CSR submitted` — key generation and CSR submission worked
|
||||
- `Deployment succeeded` — certificate deployed to NGINX
|
||||
- `NGINX reload` — signal sent to reload
|
||||
|
||||
### Testing ACME Without Real Domain
|
||||
|
||||
Use `nip.io` (free DNS service):
|
||||
1. Deploy to a server with a public IP
|
||||
2. Use domain: `<your-ip>.nip.io` (e.g., `203.0.113.45.nip.io`)
|
||||
3. Let's Encrypt will validate to that IP
|
||||
4. Change ACME_EMAIL to a real email you control
|
||||
|
||||
## Production Checklist
|
||||
|
||||
Before running in production:
|
||||
|
||||
- [ ] Change `DB_PASSWORD` to a strong random password
|
||||
- [ ] Generate a real API key for the agent (don't use the demo key)
|
||||
- [ ] Enable `CERTCTL_AUTH_TYPE=api-key` and enforce authentication
|
||||
- [ ] Use Let's Encrypt production directory (not staging)
|
||||
- [ ] Configure `CERTCTL_CORS_ORIGINS` to restrict cross-origin access
|
||||
- [ ] Use `CERTCTL_KEYGEN_MODE=agent` (default, but verify)
|
||||
- [ ] Set `CERTCTL_LOG_LEVEL=warn` to reduce log noise
|
||||
- [ ] Configure email notifications for certificate expiration alerts
|
||||
- [ ] Set up log aggregation (Datadog, ELK, Splunk, etc.)
|
||||
- [ ] Use docker secrets or external secret manager for credentials (not .env)
|
||||
- [ ] Run agent on actual NGINX servers (not co-located with server for HA)
|
||||
- [ ] Set up monitoring and alerting on agent heartbeat and job completion
|
||||
- [ ] Implement backup/restore for PostgreSQL
|
||||
- [ ] Use TLS for certctl server (terminate at reverse proxy or load balancer)
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Agent heartbeat failing
|
||||
```bash
|
||||
docker compose logs certctl-agent
|
||||
# Check: CERTCTL_SERVER_URL, CERTCTL_API_KEY, network connectivity
|
||||
```
|
||||
|
||||
### ACME challenge failing
|
||||
```bash
|
||||
# Ensure port 80 is open: curl http://example.com/.well-known/acme-challenge/test
|
||||
# Check NGINX is running and serving /.well-known/acme-challenge/
|
||||
# Verify DNS resolves domain to your server: dig example.com
|
||||
```
|
||||
|
||||
### NGINX reload failing
|
||||
Check agent permissions on NGINX socket and that NGINX is reachable from agent container.
|
||||
|
||||
### Let's Encrypt rate limited
|
||||
Let's Encrypt has rate limits (50 certs per domain per week). Use staging to test, or wait a week.
|
||||
|
||||
### Certificate not deployed to NGINX
|
||||
Check agent logs for deployment errors. Verify `/etc/nginx/ssl` volume is writable by agent container.
|
||||
|
||||
## Next Steps
|
||||
|
||||
- **Wildcard certificates**: See [acme-wildcard-dns01](../acme-wildcard-dns01/) example
|
||||
- **Multiple issuers**: See [multi-issuer](../multi-issuer/) example
|
||||
- **Private CA**: See [private-ca-traefik](../private-ca-traefik/) example
|
||||
- **Dashboard deep dive**: Read [docs/quickstart.md](../../docs/quickstart.md)
|
||||
- **REST API**: Explore [api/openapi.yaml](../../api/openapi.yaml)
|
||||
|
||||
## Support
|
||||
|
||||
For issues or questions:
|
||||
- Check [docs/troubleshooting.md](../../docs/troubleshooting.md)
|
||||
- Open an issue on GitHub
|
||||
- Review server and agent logs: `docker compose logs -f`
|
||||
@@ -0,0 +1,146 @@
|
||||
version: '3.8'
|
||||
|
||||
services:
|
||||
# PostgreSQL database for certctl
|
||||
postgres:
|
||||
image: postgres:16-alpine
|
||||
container_name: certctl-postgres-acme-nginx
|
||||
environment:
|
||||
POSTGRES_DB: certctl
|
||||
POSTGRES_USER: certctl
|
||||
POSTGRES_PASSWORD: ${DB_PASSWORD:-certctl-dev-password}
|
||||
volumes:
|
||||
- postgres_data:/var/lib/postgresql/data
|
||||
healthcheck:
|
||||
test: ['CMD-SHELL', 'pg_isready -U certctl -d certctl']
|
||||
interval: 5s
|
||||
timeout: 5s
|
||||
retries: 5
|
||||
networks:
|
||||
- certctl-network
|
||||
restart: unless-stopped
|
||||
|
||||
# certctl server (control plane)
|
||||
certctl-server:
|
||||
image: ghcr.io/shankar0123/certctl-server:latest
|
||||
container_name: certctl-server-acme-nginx
|
||||
environment:
|
||||
# Database
|
||||
DATABASE_URL: postgres://certctl:${DB_PASSWORD:-certctl-dev-password}@postgres:5432/certctl?sslmode=disable
|
||||
|
||||
# Server settings
|
||||
CERTCTL_SERVER_PORT: 8443
|
||||
CERTCTL_SERVER_HOST: 0.0.0.0
|
||||
|
||||
# Auth (disabled for demo; production should use API keys)
|
||||
CERTCTL_AUTH_TYPE: none
|
||||
|
||||
# CORS (allow agent communication)
|
||||
CERTCTL_CORS_ORIGINS: '*'
|
||||
|
||||
# Key generation mode (agent-side in production, server-side for demo)
|
||||
CERTCTL_KEYGEN_MODE: agent
|
||||
|
||||
# ACME issuer configuration
|
||||
# This registers the Let's Encrypt ACME issuer
|
||||
CERTCTL_ACME_DIRECTORY_URL: https://acme-v02.api.letsencrypt.org/directory
|
||||
CERTCTL_ACME_EMAIL: ${ACME_EMAIL:-admin@example.com}
|
||||
CERTCTL_ACME_CHALLENGE_TYPE: http-01
|
||||
|
||||
# Local CA as fallback for internal services (optional)
|
||||
CERTCTL_CA_CERT_PATH: /etc/certctl/ca.crt
|
||||
CERTCTL_CA_KEY_PATH: /etc/certctl/ca.key
|
||||
|
||||
# Logging
|
||||
CERTCTL_LOG_LEVEL: info
|
||||
ports:
|
||||
- '${SERVER_PORT:-8443}:8443'
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
networks:
|
||||
- certctl-network
|
||||
healthcheck:
|
||||
test: ['CMD-SHELL', 'curl -sf http://localhost:8443/api/v1/health || exit 1']
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
restart: unless-stopped
|
||||
|
||||
# certctl agent (runs on the target machine with NGINX)
|
||||
# In this example, the agent is in the same compose file for simplicity.
|
||||
# In production, the agent runs on each server that needs certificates.
|
||||
certctl-agent:
|
||||
image: ghcr.io/shankar0123/certctl-agent:latest
|
||||
container_name: certctl-agent-acme-nginx
|
||||
environment:
|
||||
# Control plane connection
|
||||
CERTCTL_SERVER_URL: http://certctl-server:8443
|
||||
CERTCTL_API_KEY: ${AGENT_API_KEY:-agent-demo-key}
|
||||
|
||||
# Key generation (agent-side keys, never sent to server)
|
||||
CERTCTL_KEYGEN_MODE: agent
|
||||
CERTCTL_KEY_DIR: /var/lib/certctl/keys
|
||||
|
||||
# Discovery (scan existing certs so operator knows what's already deployed)
|
||||
CERTCTL_DISCOVERY_DIRS: /etc/nginx/ssl
|
||||
|
||||
# Heartbeat interval
|
||||
CERTCTL_HEARTBEAT_INTERVAL: 30s
|
||||
|
||||
# Agent metadata (self-reported)
|
||||
CERTCTL_AGENT_NAME: nginx-agent-01
|
||||
|
||||
# Logging
|
||||
CERTCTL_LOG_LEVEL: info
|
||||
volumes:
|
||||
# Mount NGINX config and cert directories
|
||||
# In production, these would be the actual NGINX paths
|
||||
- nginx_certs:/etc/nginx/ssl
|
||||
- nginx_conf:/etc/nginx/conf.d
|
||||
# Agent key storage (persisted across restarts)
|
||||
- agent_keys:/var/lib/certctl/keys
|
||||
depends_on:
|
||||
certctl-server:
|
||||
condition: service_healthy
|
||||
networks:
|
||||
- certctl-network
|
||||
restart: unless-stopped
|
||||
|
||||
# NGINX reverse proxy / web server
|
||||
# This is where certificates will be deployed
|
||||
nginx:
|
||||
image: nginx:alpine
|
||||
container_name: certctl-nginx-acme-nginx
|
||||
ports:
|
||||
- '80:80'
|
||||
- '443:443'
|
||||
volumes:
|
||||
- nginx_conf:/etc/nginx/conf.d
|
||||
- nginx_certs:/etc/nginx/ssl
|
||||
# Default NGINX config (if not provided by agent)
|
||||
- ./nginx.conf:/etc/nginx/nginx.conf:ro
|
||||
depends_on:
|
||||
- certctl-agent
|
||||
networks:
|
||||
- certctl-network
|
||||
healthcheck:
|
||||
test: ['CMD-SHELL', 'wget --quiet --tries=1 --spider http://localhost/ || exit 1']
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
restart: unless-stopped
|
||||
|
||||
networks:
|
||||
certctl-network:
|
||||
driver: bridge
|
||||
|
||||
volumes:
|
||||
postgres_data:
|
||||
driver: local
|
||||
nginx_certs:
|
||||
driver: local
|
||||
nginx_conf:
|
||||
driver: local
|
||||
agent_keys:
|
||||
driver: local
|
||||
@@ -0,0 +1,306 @@
|
||||
# ACME Wildcard DNS-01 Example
|
||||
|
||||
**What this does:** Issues wildcard certificates (e.g., `*.example.com`) from Let's Encrypt using DNS-01 challenge validation.
|
||||
|
||||
This example is ideal for:
|
||||
- Issuing wildcard certificates (`*.example.com`)
|
||||
- Services behind NAT, firewalls, or non-public networks
|
||||
- Batch issuance of multiple domains in parallel
|
||||
- Internal PKI with public DNS names
|
||||
- Scenarios where you have programmatic access to your DNS provider's API
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Before running this example, you need:
|
||||
|
||||
1. **A domain name** (e.g., `example.com`) that you control and can manage DNS records for
|
||||
2. **DNS provider credentials:**
|
||||
- **Cloudflare** (example included): API token with DNS:write permission + Zone ID
|
||||
- **Route53 (AWS)**: AWS access key + secret key
|
||||
- **Azure DNS**: Azure subscription ID + credentials
|
||||
- **Other providers**: See "Adapting for Other DNS Providers" below
|
||||
3. **Docker and Docker Compose** installed
|
||||
|
||||
## Quick Start (Cloudflare)
|
||||
|
||||
### Step 1: Get Cloudflare Credentials
|
||||
|
||||
1. Log in to [Cloudflare Dashboard](https://dash.cloudflare.com)
|
||||
2. Select your domain (e.g., `example.com`)
|
||||
3. In the sidebar, find **Zone ID** (copy this)
|
||||
4. Go to **Account Settings > API Tokens**
|
||||
5. Create a new token with these scopes:
|
||||
- **Zone > Zone:Read** (to list DNS records)
|
||||
- **Zone > DNS:Write** (to create/delete challenge records)
|
||||
6. Copy the API token
|
||||
|
||||
### Step 2: Set Environment Variables
|
||||
|
||||
Create a `.env` file in this directory:
|
||||
|
||||
```bash
|
||||
# .env
|
||||
CLOUDFLARE_API_TOKEN=your-api-token-here
|
||||
CLOUDFLARE_ZONE_ID=your-zone-id-here
|
||||
ACME_EMAIL=admin@example.com
|
||||
DB_PASSWORD=your-secure-db-password
|
||||
```
|
||||
|
||||
Or export them in your shell:
|
||||
|
||||
```bash
|
||||
export CLOUDFLARE_API_TOKEN="your-api-token-here"
|
||||
export CLOUDFLARE_ZONE_ID="your-zone-id-here"
|
||||
export ACME_EMAIL="admin@example.com"
|
||||
export DB_PASSWORD="your-secure-db-password"
|
||||
```
|
||||
|
||||
### Step 3: Make DNS Scripts Executable
|
||||
|
||||
```bash
|
||||
chmod +x dns-hooks/*.sh
|
||||
```
|
||||
|
||||
### Step 4: Start the Stack
|
||||
|
||||
```bash
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
This starts:
|
||||
- **certctl-server** (port 8443): Control plane and ACME orchestrator
|
||||
- **postgres**: Certificate metadata database
|
||||
- **certctl-agent**: Certificate deployment agent
|
||||
|
||||
### Step 5: Access the Dashboard
|
||||
|
||||
Open your browser to `http://localhost:8443`
|
||||
|
||||
### Step 6: Create a Wildcard Certificate
|
||||
|
||||
1. Go to **Issuers** page
|
||||
2. Verify the ACME issuer is registered
|
||||
3. Go to **Certificates** > **New Certificate**
|
||||
4. Fill in:
|
||||
- **Issuer:** ACME (Let's Encrypt)
|
||||
- **Common Name:** `*.example.com`
|
||||
- **Subject Alt Names:** `example.com` (to also cover the root domain)
|
||||
5. Click **Request**
|
||||
|
||||
The renewal job will:
|
||||
1. Send a request to Let's Encrypt
|
||||
2. Run `dns-hooks/cloudflare-present.sh` to create `_acme-challenge.example.com` TXT record
|
||||
3. Wait for Let's Encrypt to verify the TXT record
|
||||
4. Issue the certificate
|
||||
5. Run `dns-hooks/cloudflare-cleanup.sh` to delete the temporary TXT record
|
||||
|
||||
### Step 7: Monitor the Job
|
||||
|
||||
Go to **Jobs** page to see the renewal progress:
|
||||
- **AwaitingCSR**: Agent is generating the CSR
|
||||
- **Running**: ACME challenge in progress (DNS record being validated)
|
||||
- **Completed**: Certificate issued and stored
|
||||
- **Failed**: Check logs for errors (e.g., DNS provider API issues)
|
||||
|
||||
## How DNS-01 Works
|
||||
|
||||
The DNS-01 challenge proves you own a domain by creating a DNS TXT record:
|
||||
|
||||
```
|
||||
_acme-challenge.example.com TXT "acme-validation-token-xxxxx"
|
||||
```
|
||||
|
||||
Let's Encrypt then queries this TXT record. Once verified, it issues the certificate and certctl cleans up the TXT record.
|
||||
|
||||
**Why DNS-01 is better than HTTP-01 for wildcards:**
|
||||
- HTTP-01 requires a public web server; DNS-01 works anywhere
|
||||
- Wildcard certificates require DNS proof (not HTTP)
|
||||
- DNS challenges can be solved for multiple domains in parallel
|
||||
- No need for public IP or inbound port 80/443
|
||||
|
||||
## Adapting for Other DNS Providers
|
||||
|
||||
The example uses Cloudflare, but certctl supports **any DNS provider via pluggable shell scripts**.
|
||||
|
||||
### AWS Route53
|
||||
|
||||
Replace the `CERTCTL_ACME_DNS_PRESENT_SCRIPT` and `CERTCTL_ACME_DNS_CLEANUP_SCRIPT` in `docker-compose.yml` with:
|
||||
- `./dns-hooks/route53-present.sh`
|
||||
- `./dns-hooks/route53-cleanup.sh`
|
||||
|
||||
Example script outline (using AWS CLI):
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
DOMAIN="$1"
|
||||
VALIDATION_TOKEN="$2"
|
||||
|
||||
# Get Route53 hosted zone ID for the domain
|
||||
ZONE_ID=$(aws route53 list-hosted-zones --query \
|
||||
"HostedZones[?Name=='$DOMAIN.'].Id" --output text | cut -d'/' -f3)
|
||||
|
||||
# Create TXT record
|
||||
aws route53 change-resource-record-sets \
|
||||
--hosted-zone-id "$ZONE_ID" \
|
||||
--change-batch "{
|
||||
\"Changes\": [{
|
||||
\"Action\": \"CREATE\",
|
||||
\"ResourceRecordSet\": {
|
||||
\"Name\": \"_acme-challenge.$DOMAIN\",
|
||||
\"Type\": \"TXT\",
|
||||
\"TTL\": 120,
|
||||
\"ResourceRecords\": [{\"Value\": \"\\\"$VALIDATION_TOKEN\\\"\"}]
|
||||
}
|
||||
}]
|
||||
}"
|
||||
```
|
||||
|
||||
### Azure DNS
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
DOMAIN="$1"
|
||||
VALIDATION_TOKEN="$2"
|
||||
|
||||
# Set Azure credentials via environment variables
|
||||
# AZURE_SUBSCRIPTION_ID, AZURE_RESOURCE_GROUP, AZURE_TENANT_ID, etc.
|
||||
|
||||
az network dns record-set txt create \
|
||||
--resource-group "$AZURE_RESOURCE_GROUP" \
|
||||
--zone-name "$DOMAIN" \
|
||||
--name "_acme-challenge" \
|
||||
--ttl 120 \
|
||||
--txt-value "$VALIDATION_TOKEN"
|
||||
```
|
||||
|
||||
### Generic DNS Provider (using dig + TSIG)
|
||||
|
||||
If your DNS provider supports NSUPDATE (RFC 2136):
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
DOMAIN="$1"
|
||||
VALIDATION_TOKEN="$2"
|
||||
|
||||
nsupdate <<EOF
|
||||
zone $DOMAIN
|
||||
update add _acme-challenge.$DOMAIN 120 TXT "$VALIDATION_TOKEN"
|
||||
send
|
||||
EOF
|
||||
```
|
||||
|
||||
### Manual DNS (for testing)
|
||||
|
||||
Replace scripts with no-ops during testing:
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
echo "Please create: _acme-challenge.$1 TXT $2"
|
||||
sleep 60 # Manual wait for you to create the record
|
||||
```
|
||||
|
||||
## Alternative: DNS-PERSIST-01 (Standing Records)
|
||||
|
||||
If your DNS provider supports it, use **DNS-PERSIST-01** for zero-maintenance renewals.
|
||||
|
||||
Instead of creating a new TXT record for each renewal, you create one standing record once:
|
||||
|
||||
```
|
||||
_validation-persist.example.com TXT "letsencrypt.org; accounturi=https://acme-v02.api.letsencrypt.org/acme/acct/12345678"
|
||||
```
|
||||
|
||||
Then every renewal uses the same record — no cleanup scripts needed!
|
||||
|
||||
To enable in `docker-compose.yml`:
|
||||
|
||||
```yaml
|
||||
CERTCTL_ACME_CHALLENGE_TYPE: dns-persist-01
|
||||
CERTCTL_ACME_DNS_PERSIST_ISSUER_DOMAIN: letsencrypt.org
|
||||
```
|
||||
|
||||
Certctl will:
|
||||
1. Fetch your ACME account URI
|
||||
2. Create the standing `_validation-persist` record once
|
||||
3. Reuse it for all future renewals (no per-renewal DNS updates)
|
||||
|
||||
## Security Notes
|
||||
|
||||
1. **API Token Scope:** Restrict Cloudflare/AWS tokens to DNS:write only (not full account access)
|
||||
2. **Key Generation:** This example uses agent-side key generation (`CERTCTL_KEYGEN_MODE=agent`), which is production-standard. Private keys never leave the agent.
|
||||
3. **Script Safety:** The DNS scripts run in the certctl-server container. For production:
|
||||
- Validate script inputs (already done in certctl code)
|
||||
- Log all API calls
|
||||
- Monitor for failed DNS operations
|
||||
- Use a separate proxy agent for DNS operations if needed
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### DNS record not created
|
||||
|
||||
Check the server logs:
|
||||
|
||||
```bash
|
||||
docker logs certctl-server-dns01
|
||||
```
|
||||
|
||||
Look for lines like:
|
||||
- `[certctl DNS-01] Creating DNS record: _acme-challenge.example.com`
|
||||
- `Error: Cloudflare API failed: ...`
|
||||
|
||||
**Common issues:**
|
||||
- Missing or invalid `CLOUDFLARE_API_TOKEN`
|
||||
- Invalid `CLOUDFLARE_ZONE_ID`
|
||||
- API token doesn't have DNS:write permission
|
||||
- Domain not in your Cloudflare account
|
||||
|
||||
### DNS propagation timeout
|
||||
|
||||
If the TLS negotiation fails, it might be DNS caching. Increase the wait time in the script:
|
||||
|
||||
```bash
|
||||
sleep 30 # Increase from 10 to 30 seconds
|
||||
```
|
||||
|
||||
### Let's Encrypt rate limits
|
||||
|
||||
Let's Encrypt has strict rate limits:
|
||||
- 50 certificates per registered domain per week
|
||||
- 5 duplicate certificates per domain per week
|
||||
|
||||
For testing, use the **staging directory**:
|
||||
|
||||
```yaml
|
||||
CERTCTL_ACME_DIRECTORY_URL: https://acme-staging-v02.api.letsencrypt.org/directory
|
||||
```
|
||||
|
||||
(Staging certs won't be trusted by browsers, but don't count against rate limits.)
|
||||
|
||||
### Job fails with "CSR generation timeout"
|
||||
|
||||
If your DNS provider is very slow, increase the timeout in the cleanup script or add a longer wait time:
|
||||
|
||||
```bash
|
||||
sleep 60 # Wait 1 minute for DNS propagation
|
||||
```
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. **Monitor renewals:** Set up notifications (email, Slack, PagerDuty) for renewal events
|
||||
2. **Deploy certificates:** Configure target connectors (NGINX, HAProxy, Traefik) to automatically deploy issued certs
|
||||
3. **Multi-domain:** Use certificate profiles to group wildcard + subdomain certs
|
||||
4. **Backup DNS scripts:** Version control your DNS provider scripts in git
|
||||
|
||||
## Files in This Example
|
||||
|
||||
- **docker-compose.yml** — Container stack definition with ACME DNS-01 configuration
|
||||
- **dns-hooks/cloudflare-present.sh** — Creates `_acme-challenge` TXT record (Cloudflare)
|
||||
- **dns-hooks/cloudflare-cleanup.sh** — Deletes `_acme-challenge` TXT record (Cloudflare)
|
||||
- **README.md** — This file
|
||||
|
||||
## Additional Resources
|
||||
|
||||
- [certctl Documentation](../../docs/)
|
||||
- [ACME Specification (RFC 8555)](https://tools.ietf.org/html/rfc8555)
|
||||
- [DNS-01 Challenge Details](https://letsencrypt.org/docs/challenge-types/#dns-01)
|
||||
- [DNS-PERSIST-01 (IETF Draft)](https://datatracker.ietf.org/doc/html/draft-ietf-acme-dns-persist)
|
||||
- [Let's Encrypt Documentation](https://letsencrypt.org/docs/)
|
||||
@@ -0,0 +1,86 @@
|
||||
#!/bin/bash
|
||||
|
||||
#
|
||||
# Cloudflare DNS-01 Challenge Script (CLEANUP)
|
||||
#
|
||||
# This script removes a DNS TXT record after ACME DNS-01 challenge validation.
|
||||
# Called by certctl after certificate issuance to clean up temporary challenge records.
|
||||
#
|
||||
# certctl sets these environment variables before invoking this script:
|
||||
# CERTCTL_DNS_DOMAIN - Base domain (e.g., "example.com")
|
||||
# CERTCTL_DNS_FQDN - Full challenge FQDN (e.g., "_acme-challenge.example.com")
|
||||
# CERTCTL_DNS_VALUE - Challenge value/token that was in the TXT record
|
||||
#
|
||||
# You must set these environment variables before running:
|
||||
# CLOUDFLARE_API_TOKEN - Cloudflare API token with DNS:write permission
|
||||
# CLOUDFLARE_ZONE_ID - Cloudflare zone ID for your domain
|
||||
#
|
||||
# Error Handling:
|
||||
# This script exits 0 on success, non-zero on failure.
|
||||
# If cleanup fails, certctl logs the error but doesn't block renewals.
|
||||
#
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
# Get values from certctl environment variables
|
||||
DOMAIN="${CERTCTL_DNS_DOMAIN:-}"
|
||||
RECORD_NAME="${CERTCTL_DNS_FQDN:-}"
|
||||
VALIDATION_TOKEN="${CERTCTL_DNS_VALUE:-}"
|
||||
|
||||
# Validate inputs
|
||||
if [[ -z "$DOMAIN" || -z "$RECORD_NAME" || -z "$VALIDATION_TOKEN" ]]; then
|
||||
echo "Error: Required certctl environment variables not set (CERTCTL_DNS_DOMAIN, CERTCTL_DNS_FQDN, CERTCTL_DNS_VALUE)" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Validate environment
|
||||
if [[ -z "${CLOUDFLARE_API_TOKEN:-}" ]]; then
|
||||
echo "Error: CLOUDFLARE_API_TOKEN environment variable not set" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [[ -z "${CLOUDFLARE_ZONE_ID:-}" ]]; then
|
||||
echo "Error: CLOUDFLARE_ZONE_ID environment variable not set" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Validate RECORD_NAME (set by certctl above)
|
||||
RECORD_TYPE="TXT"
|
||||
|
||||
# Cloudflare API endpoint
|
||||
CF_API="https://api.cloudflare.com/client/v4"
|
||||
CF_ZONE="$CLOUDFLARE_ZONE_ID"
|
||||
CF_TOKEN="$CLOUDFLARE_API_TOKEN"
|
||||
|
||||
echo "[certctl DNS-01] Cleaning up DNS record: $RECORD_NAME"
|
||||
|
||||
# Step 1: Find the record ID
|
||||
RECORD_ID=$(curl -s -X GET \
|
||||
"$CF_API/zones/$CF_ZONE/dns_records?name=$RECORD_NAME&type=$RECORD_TYPE" \
|
||||
-H "Authorization: Bearer $CF_TOKEN" \
|
||||
-H "Content-Type: application/json" \
|
||||
| jq -r '.result | if length > 0 then .[0].id else "" end')
|
||||
|
||||
if [[ -z "$RECORD_ID" ]]; then
|
||||
echo "[certctl DNS-01] Record not found (already deleted?). Skipping cleanup."
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# Step 2: Delete the record (DELETE /zones/{zone_id}/dns_records/{record_id})
|
||||
echo "[certctl DNS-01] Deleting DNS record (ID: $RECORD_ID)..."
|
||||
RESPONSE=$(curl -s -X DELETE \
|
||||
"$CF_API/zones/$CF_ZONE/dns_records/$RECORD_ID" \
|
||||
-H "Authorization: Bearer $CF_TOKEN" \
|
||||
-H "Content-Type: application/json")
|
||||
|
||||
# Check response success
|
||||
SUCCESS=$(echo "$RESPONSE" | jq -r '.success')
|
||||
if [[ "$SUCCESS" != "true" ]]; then
|
||||
ERROR=$(echo "$RESPONSE" | jq -r '.errors[0].message // "Unknown error"')
|
||||
echo "Warning: Cloudflare API failed to delete record: $ERROR" >&2
|
||||
# Don't exit 1 here — DNS cleanup is best-effort; cleanup failures shouldn't block certs
|
||||
exit 0
|
||||
fi
|
||||
|
||||
echo "[certctl DNS-01] Successfully deleted DNS record"
|
||||
exit 0
|
||||
@@ -0,0 +1,108 @@
|
||||
#!/bin/bash
|
||||
|
||||
#
|
||||
# Cloudflare DNS-01 Challenge Script (PRESENT)
|
||||
#
|
||||
# This script creates a DNS TXT record for ACME DNS-01 challenge validation.
|
||||
# Called by certctl during the renewal process to prove domain ownership.
|
||||
#
|
||||
# certctl sets these environment variables before invoking this script:
|
||||
# CERTCTL_DNS_DOMAIN - Base domain (e.g., "example.com")
|
||||
# CERTCTL_DNS_FQDN - Full challenge FQDN (e.g., "_acme-challenge.example.com")
|
||||
# CERTCTL_DNS_VALUE - Challenge value/token to place in the TXT record
|
||||
#
|
||||
# You must set these environment variables before running:
|
||||
# CLOUDFLARE_API_TOKEN - Cloudflare API token with DNS:write permission
|
||||
# CLOUDFLARE_ZONE_ID - Cloudflare zone ID for your domain
|
||||
# (Find at: https://dash.cloudflare.com > Select Domain > Zone ID in sidebar)
|
||||
#
|
||||
# Error Handling:
|
||||
# This script exits 0 on success, non-zero on failure.
|
||||
# certctl will retry the renewal if this script fails.
|
||||
#
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
# Get values from certctl environment variables
|
||||
DOMAIN="${CERTCTL_DNS_DOMAIN:-}"
|
||||
RECORD_NAME="${CERTCTL_DNS_FQDN:-}"
|
||||
VALIDATION_TOKEN="${CERTCTL_DNS_VALUE:-}"
|
||||
|
||||
# Validate inputs
|
||||
if [[ -z "$DOMAIN" || -z "$RECORD_NAME" || -z "$VALIDATION_TOKEN" ]]; then
|
||||
echo "Error: Required certctl environment variables not set (CERTCTL_DNS_DOMAIN, CERTCTL_DNS_FQDN, CERTCTL_DNS_VALUE)" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Validate environment
|
||||
if [[ -z "${CLOUDFLARE_API_TOKEN:-}" ]]; then
|
||||
echo "Error: CLOUDFLARE_API_TOKEN environment variable not set" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [[ -z "${CLOUDFLARE_ZONE_ID:-}" ]]; then
|
||||
echo "Error: CLOUDFLARE_ZONE_ID environment variable not set" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Validate RECORD_NAME (set by certctl above)
|
||||
RECORD_TYPE="TXT"
|
||||
RECORD_TTL=120 # Short TTL for challenge records (1-2 min)
|
||||
|
||||
# Cloudflare API endpoint
|
||||
CF_API="https://api.cloudflare.com/client/v4"
|
||||
CF_ZONE="$CLOUDFLARE_ZONE_ID"
|
||||
CF_TOKEN="$CLOUDFLARE_API_TOKEN"
|
||||
|
||||
echo "[certctl DNS-01] Creating DNS record: $RECORD_NAME = $VALIDATION_TOKEN"
|
||||
|
||||
# Step 1: Check if record already exists (GET /zones/{zone_id}/dns_records)
|
||||
# This is optional but helps with idempotency
|
||||
EXISTING=$(curl -s -X GET \
|
||||
"$CF_API/zones/$CF_ZONE/dns_records?name=$RECORD_NAME&type=$RECORD_TYPE" \
|
||||
-H "Authorization: Bearer $CF_TOKEN" \
|
||||
-H "Content-Type: application/json" \
|
||||
| jq -r '.result | if length > 0 then .[0].id else "" end')
|
||||
|
||||
if [[ -n "$EXISTING" ]]; then
|
||||
echo "[certctl DNS-01] Record already exists (ID: $EXISTING). Updating..."
|
||||
# Update existing record
|
||||
RESPONSE=$(curl -s -X PUT \
|
||||
"$CF_API/zones/$CF_ZONE/dns_records/$EXISTING" \
|
||||
-H "Authorization: Bearer $CF_TOKEN" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d "{
|
||||
\"type\": \"$RECORD_TYPE\",
|
||||
\"name\": \"$RECORD_NAME\",
|
||||
\"content\": \"$VALIDATION_TOKEN\",
|
||||
\"ttl\": $RECORD_TTL
|
||||
}")
|
||||
else
|
||||
echo "[certctl DNS-01] Creating new DNS record..."
|
||||
# Create new record
|
||||
RESPONSE=$(curl -s -X POST \
|
||||
"$CF_API/zones/$CF_ZONE/dns_records" \
|
||||
-H "Authorization: Bearer $CF_TOKEN" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d "{
|
||||
\"type\": \"$RECORD_TYPE\",
|
||||
\"name\": \"$RECORD_NAME\",
|
||||
\"content\": \"$VALIDATION_TOKEN\",
|
||||
\"ttl\": $RECORD_TTL
|
||||
}")
|
||||
fi
|
||||
|
||||
# Check response success
|
||||
SUCCESS=$(echo "$RESPONSE" | jq -r '.success')
|
||||
if [[ "$SUCCESS" != "true" ]]; then
|
||||
ERROR=$(echo "$RESPONSE" | jq -r '.errors[0].message // "Unknown error"')
|
||||
echo "Error: Cloudflare API failed: $ERROR" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
RECORD_ID=$(echo "$RESPONSE" | jq -r '.result.id')
|
||||
echo "[certctl DNS-01] Successfully created/updated DNS record (ID: $RECORD_ID)"
|
||||
echo "[certctl DNS-01] Waiting for DNS propagation..."
|
||||
sleep 10
|
||||
|
||||
exit 0
|
||||
@@ -0,0 +1,171 @@
|
||||
version: '3.8'
|
||||
|
||||
# ACME Wildcard DNS-01 Example
|
||||
#
|
||||
# This example demonstrates how to use certctl with Let's Encrypt to issue wildcard
|
||||
# certificates (*.example.com) using DNS-01 challenge validation.
|
||||
#
|
||||
# DNS-01 is ideal for:
|
||||
# - Wildcard certificates (*.domain.com)
|
||||
# - Services behind NAT or non-public networks
|
||||
# - Batch certificate issuance (multiple domains in parallel)
|
||||
#
|
||||
# It works by:
|
||||
# 1. certctl creates a renewal job for a wildcard certificate
|
||||
# 2. Let's Encrypt sends an ACME challenge: "create _acme-challenge TXT record with value X"
|
||||
# 3. certctl runs the dns-present.sh script to create the TXT record via your DNS provider API
|
||||
# 4. Let's Encrypt verifies the TXT record exists
|
||||
# 5. Certificate is issued
|
||||
# 6. certctl runs dns-cleanup.sh to remove the TXT record
|
||||
#
|
||||
# This compose file also demonstrates:
|
||||
# - ACME issuer with DNS-01 challenge type
|
||||
# - Pluggable DNS provider scripts (Cloudflare example included; adapt for Route53, Azure DNS, etc.)
|
||||
# - Wildcard and multi-SAN certificate support
|
||||
# - Agent-side key generation (production-ready)
|
||||
|
||||
services:
|
||||
# PostgreSQL database for certctl metadata
|
||||
postgres:
|
||||
image: postgres:16-alpine
|
||||
container_name: certctl-postgres-dns01
|
||||
environment:
|
||||
POSTGRES_DB: certctl
|
||||
POSTGRES_USER: certctl
|
||||
POSTGRES_PASSWORD: ${DB_PASSWORD:-certctl-dev-password}
|
||||
volumes:
|
||||
- postgres_data:/var/lib/postgresql/data
|
||||
healthcheck:
|
||||
test: ['CMD-SHELL', 'pg_isready -U certctl -d certctl']
|
||||
interval: 5s
|
||||
timeout: 5s
|
||||
retries: 5
|
||||
networks:
|
||||
- certctl-network
|
||||
restart: unless-stopped
|
||||
|
||||
# certctl server (control plane + ACME orchestration)
|
||||
certctl-server:
|
||||
image: ghcr.io/shankar0123/certctl-server:latest
|
||||
container_name: certctl-server-dns01
|
||||
environment:
|
||||
# Database
|
||||
DATABASE_URL: postgres://certctl:${DB_PASSWORD:-certctl-dev-password}@postgres:5432/certctl?sslmode=disable
|
||||
|
||||
# Server settings
|
||||
CERTCTL_SERVER_PORT: 8443
|
||||
CERTCTL_SERVER_HOST: 0.0.0.0
|
||||
|
||||
# Auth (disabled for demo; production should use API keys with CERTCTL_AUTH_TYPE=api-key)
|
||||
CERTCTL_AUTH_TYPE: none
|
||||
|
||||
# CORS (allow agent communication)
|
||||
CERTCTL_CORS_ORIGINS: '*'
|
||||
|
||||
# Key generation mode (agent-side: keys never leave agents; production standard)
|
||||
CERTCTL_KEYGEN_MODE: agent
|
||||
|
||||
# ===== ACME Issuer Configuration (DNS-01 Wildcard) =====
|
||||
# Let's Encrypt production directory (ACME v2)
|
||||
CERTCTL_ACME_DIRECTORY_URL: https://acme-v02.api.letsencrypt.org/directory
|
||||
|
||||
# Email for certificate expiration notices and account recovery
|
||||
CERTCTL_ACME_EMAIL: ${ACME_EMAIL:-admin@example.com}
|
||||
|
||||
# Challenge type: dns-01 (not http-01, which doesn't support wildcards)
|
||||
CERTCTL_ACME_CHALLENGE_TYPE: dns-01
|
||||
|
||||
# DNS present script: creates _acme-challenge TXT record
|
||||
# The script is mounted from ./dns-hooks/cloudflare-present.sh
|
||||
# Arguments: $1 = domain (e.g., "example.com"), $2 = validation token
|
||||
CERTCTL_ACME_DNS_PRESENT_SCRIPT: /etc/certctl/dns-hooks/cloudflare-present.sh
|
||||
|
||||
# DNS cleanup script: removes _acme-challenge TXT record
|
||||
# Arguments: $1 = domain, $2 = validation token
|
||||
CERTCTL_ACME_DNS_CLEANUP_SCRIPT: /etc/certctl/dns-hooks/cloudflare-cleanup.sh
|
||||
|
||||
# Optional: DNS propagation wait time (seconds) before proceeding to next challenge
|
||||
# Default is 30s; increase if your DNS propagates slowly
|
||||
# Set via CERTCTL_ACME_DNS_PROPAGATION_WAIT in code, or rely on default
|
||||
|
||||
# Optional: Let's Encrypt Renewal Information (RFC 9702) for CA-directed renewal timing
|
||||
# CERTCTL_ACME_ARI_ENABLED: "true"
|
||||
|
||||
# Local CA as fallback for internal services (optional)
|
||||
CERTCTL_CA_CERT_PATH: /etc/certctl/ca.crt
|
||||
CERTCTL_CA_KEY_PATH: /etc/certctl/ca.key
|
||||
|
||||
# Logging
|
||||
CERTCTL_LOG_LEVEL: info
|
||||
|
||||
ports:
|
||||
- '${SERVER_PORT:-8443}:8443'
|
||||
|
||||
volumes:
|
||||
# Mount DNS provider scripts (adapt these for your DNS provider)
|
||||
- ./dns-hooks:/etc/certctl/dns-hooks:ro
|
||||
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
|
||||
networks:
|
||||
- certctl-network
|
||||
|
||||
healthcheck:
|
||||
test: ['CMD-SHELL', 'curl -sf http://localhost:8443/api/v1/health || exit 1']
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
|
||||
restart: unless-stopped
|
||||
|
||||
# certctl agent (manages certificate deployment on target hosts)
|
||||
# In production, run agents on each host that needs certificates.
|
||||
# For demo, we include one agent in this compose.
|
||||
certctl-agent:
|
||||
image: ghcr.io/shankar0123/certctl-agent:latest
|
||||
container_name: certctl-agent-dns01
|
||||
environment:
|
||||
# Control plane connection
|
||||
CERTCTL_SERVER_URL: http://certctl-server:8443
|
||||
CERTCTL_API_KEY: ${AGENT_API_KEY:-agent-demo-key}
|
||||
|
||||
# Key generation (agent-side keys: production-standard security model)
|
||||
CERTCTL_KEYGEN_MODE: agent
|
||||
CERTCTL_KEY_DIR: /var/lib/certctl/keys
|
||||
|
||||
# Discovery (scan existing certs so operator knows what's already deployed)
|
||||
CERTCTL_DISCOVERY_DIRS: /etc/letsencrypt/live:/etc/ssl/certs
|
||||
|
||||
# Heartbeat interval (how often agent checks for work)
|
||||
CERTCTL_HEARTBEAT_INTERVAL: 30s
|
||||
|
||||
# Agent metadata (self-reported to server)
|
||||
CERTCTL_AGENT_NAME: wildcard-agent-01
|
||||
|
||||
# Logging
|
||||
CERTCTL_LOG_LEVEL: info
|
||||
|
||||
volumes:
|
||||
# Agent persistent key storage (survives restarts)
|
||||
- agent_keys:/var/lib/certctl/keys
|
||||
|
||||
depends_on:
|
||||
certctl-server:
|
||||
condition: service_healthy
|
||||
|
||||
networks:
|
||||
- certctl-network
|
||||
|
||||
restart: unless-stopped
|
||||
|
||||
networks:
|
||||
certctl-network:
|
||||
driver: bridge
|
||||
|
||||
volumes:
|
||||
postgres_data:
|
||||
driver: local
|
||||
agent_keys:
|
||||
driver: local
|
||||
@@ -0,0 +1,242 @@
|
||||
# Multi-Issuer Example: ACME + Local CA
|
||||
|
||||
This example demonstrates certctl managing **both public and internal certificates from a single dashboard**. Public-facing services use Let's Encrypt (ACME), while internal services use a private Local CA — all visible and managed in one place.
|
||||
|
||||
## The Use Case
|
||||
|
||||
You have:
|
||||
- **Public-facing services** (web app, API, etc.) that need TLS certs signed by a trusted public CA (Let's Encrypt)
|
||||
- **Internal services** (databases, microservices, middleware) that need TLS certs but don't require public trust
|
||||
- **One team** managing certs across both, needing unified visibility and automated renewal
|
||||
|
||||
With certctl, both issuer types are configured and available. You assign each certificate to the appropriate issuer via its profile or at enrollment time. The dashboard shows all certs together, with renewal status, expiration timelines, and audit trails — regardless of which CA issued them.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ certctl Server (Control Plane) │
|
||||
│ - Let's Encrypt ACME issuer (HTTP-01 challenges) │
|
||||
│ - Local CA issuer (self-signed or sub-CA mode) │
|
||||
│ - PostgreSQL database (cert inventory, audit, jobs) │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
▲
|
||||
│ API polling
|
||||
│
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ certctl Agent │
|
||||
│ - Discovers existing certs in /etc/nginx/ssl and /etc/app/ssl │
|
||||
│ - Polls server for renewal/issuance/deployment jobs │
|
||||
│ - Generates keys locally (agent-side crypto) │
|
||||
│ - Deploys certs to NGINX and app service directories │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
│ │
|
||||
▼ ▼
|
||||
NGINX (public TLS) App Services (internal TLS)
|
||||
(Let's Encrypt certs) (Local CA certs)
|
||||
```
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- **Docker & Docker Compose** — containers run everything
|
||||
- **Port access** — 80 (HTTP-01 challenges) and 443 (HTTPS) for Let's Encrypt
|
||||
- **Domain for ACME** (optional) — if using real Let's Encrypt, not needed for demo
|
||||
- **Internet connectivity** — to reach Let's Encrypt's API (demo can use staging directory)
|
||||
|
||||
## Quick Start
|
||||
|
||||
### 1. Clone or navigate to this directory
|
||||
|
||||
```bash
|
||||
cd examples/multi-issuer
|
||||
```
|
||||
|
||||
### 2. Set environment variables (optional, defaults provided)
|
||||
|
||||
```bash
|
||||
# Email for Let's Encrypt account
|
||||
export ACME_EMAIL="your-email@example.com"
|
||||
|
||||
# Database password (for demo, default is fine)
|
||||
export DB_PASSWORD="certctl-dev-password"
|
||||
|
||||
# Agent API key
|
||||
export AGENT_API_KEY="agent-demo-key"
|
||||
|
||||
# Server port (default 8443)
|
||||
export SERVER_PORT="8443"
|
||||
```
|
||||
|
||||
### 3. Start the services
|
||||
|
||||
```bash
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
This spins up:
|
||||
- **PostgreSQL** database (certctl data store)
|
||||
- **certctl server** with ACME and Local CA issuers configured
|
||||
- **certctl agent** discovering existing certs and polling for work
|
||||
- **NGINX** web server (target for public TLS certs)
|
||||
|
||||
### 4. Access the dashboard
|
||||
|
||||
Open your browser to **http://localhost:8443** (or your configured SERVER_PORT)
|
||||
|
||||
You should see:
|
||||
- Empty cert inventory (fresh start)
|
||||
- Two configured issuers: "ACME" and "Local CA"
|
||||
- One registered agent ("multi-issuer-agent-01")
|
||||
|
||||
### 5. Create test certificates
|
||||
|
||||
In the dashboard:
|
||||
|
||||
**For a public cert (Let's Encrypt):**
|
||||
1. Go to **Certificates** > **+ New Certificate**
|
||||
2. Common Name: `example.com` (or a test domain you control)
|
||||
3. Issuer: Select "ACME"
|
||||
4. Profile: Select default or create one (key type: RSA 2048, TTL: 90 days)
|
||||
5. Create → The server submits an ACME order
|
||||
|
||||
**For an internal cert (Local CA):**
|
||||
1. Go to **Certificates** > **+ New Certificate**
|
||||
2. Common Name: `internal-api.internal` (or any internal name)
|
||||
3. Issuer: Select "Local CA"
|
||||
4. Profile: Select default
|
||||
5. Create → The server issues immediately from the private CA
|
||||
|
||||
### 6. Monitor in the dashboard
|
||||
|
||||
- **Dashboard** — see cert counts by status and issuer
|
||||
- **Certificates** page — filter by issuer, see renewal status, expiration timeline
|
||||
- **Audit Trail** — track all operations (issuance, renewals, deployments)
|
||||
- **Agents** — view agent health and pending work
|
||||
|
||||
## How Issuer Assignment Works
|
||||
|
||||
### Via Profiles
|
||||
Create a profile for each issuer type:
|
||||
- Profile **public-tls** → Issuer: ACME, TTL: 90 days, allowed domains: `*.example.com`
|
||||
- Profile **internal-tls** → Issuer: Local CA, TTL: 1 year, allowed SANs: internal DNS names
|
||||
|
||||
Then create certificates using the appropriate profile.
|
||||
|
||||
### Via Direct Assignment
|
||||
When creating a certificate, explicitly select the issuer. The certificate remembers which issuer it belongs to.
|
||||
|
||||
## ACME Configuration
|
||||
|
||||
The server is configured with Let's Encrypt's production directory:
|
||||
|
||||
```yaml
|
||||
CERTCTL_ACME_DIRECTORY_URL: https://acme-v02.api.letsencrypt.org/directory
|
||||
CERTCTL_ACME_EMAIL: admin@example.com
|
||||
CERTCTL_ACME_CHALLENGE_TYPE: http-01
|
||||
```
|
||||
|
||||
**For testing without a real domain**, use Let's Encrypt's staging directory:
|
||||
|
||||
```bash
|
||||
# Edit docker-compose.yml and change:
|
||||
CERTCTL_ACME_DIRECTORY_URL: https://acme-staging-v02.api.letsencrypt.org/directory
|
||||
```
|
||||
|
||||
Staging certs are untrusted (for testing only) but unlimited rate limits.
|
||||
|
||||
## Local CA Configuration
|
||||
|
||||
The Local CA issuer can operate in two modes:
|
||||
|
||||
### Mode 1: Self-Signed (Default)
|
||||
Leave `CERTCTL_CA_CERT_PATH` and `CERTCTL_CA_KEY_PATH` empty. The server generates a self-signed root CA on first run.
|
||||
|
||||
```yaml
|
||||
CERTCTL_CA_CERT_PATH: ""
|
||||
CERTCTL_CA_KEY_PATH: ""
|
||||
```
|
||||
|
||||
**Use case:** Development, testing, internal services that trust a self-signed root.
|
||||
|
||||
### Mode 2: Sub-CA (Enterprise)
|
||||
Provide an existing CA cert + key (e.g., from your organization's PKI). The Local CA issues certs signed by that intermediate.
|
||||
|
||||
```bash
|
||||
# In docker-compose.yml, volume-mount your CA cert+key:
|
||||
volumes:
|
||||
- /path/to/ca.crt:/etc/certctl/ca.crt:ro
|
||||
- /path/to/ca.key:/etc/certctl/ca.key:ro
|
||||
|
||||
# And set env vars:
|
||||
CERTCTL_CA_CERT_PATH: /etc/certctl/ca.crt
|
||||
CERTCTL_CA_KEY_PATH: /etc/certctl/ca.key
|
||||
```
|
||||
|
||||
**Use case:** Enterprise internal PKI where certs need to chain to a trusted root (e.g., Windows ADCS, OpenSSL, Vault PKI).
|
||||
|
||||
## Deployment Flow
|
||||
|
||||
When you create a certificate and assign it for deployment:
|
||||
|
||||
1. **Issuance** — Server calls the issuer connector (ACME or Local CA)
|
||||
- ACME: submit challenge, poll until DNS/HTTP validated, retrieve cert
|
||||
- Local CA: generate and sign immediately
|
||||
|
||||
2. **Agent picks up work** — Agent polls `/api/v1/agents/{id}/work`
|
||||
|
||||
3. **Agent deployment** — Agent places cert+key in the target directory
|
||||
- NGINX: `/etc/nginx/ssl/` (mounted volume)
|
||||
- App services: `/etc/app/ssl/` (mounted volume)
|
||||
|
||||
4. **Service reload** — Agent triggers reload (NGINX: `nginx -s reload`, etc.)
|
||||
|
||||
5. **Dashboard reflects status** — Job transitions from `Running` → `Completed`, cert shows as `Active`
|
||||
|
||||
## Scaling Beyond Docker Compose
|
||||
|
||||
In production:
|
||||
|
||||
- **Deploy certctl server** on a single node (or HA cluster with external PostgreSQL)
|
||||
- **Deploy certctl agents** on each server needing cert management
|
||||
- **Point agents to server URL** via `CERTCTL_SERVER_URL` env var
|
||||
- **Configure issuers on server** via env vars or (in V3+) the dashboard UI
|
||||
- **Use profiles to segment issuers** — operators select a profile at cert creation time
|
||||
|
||||
Each agent independently manages its local cert inventory and deployments. The server coordinates all agent work and provides the unified dashboard.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Certs aren't being issued
|
||||
- Check server logs: `docker compose logs certctl-server`
|
||||
- Verify issuer configuration: Dashboard → Issuers, click "Test Connection"
|
||||
- For ACME, ensure ports 80/443 are open and your domain resolves
|
||||
|
||||
### Agent can't reach server
|
||||
- Check network: `docker compose exec certctl-agent curl http://certctl-server:8443/api/v1/health`
|
||||
- Verify `CERTCTL_SERVER_URL` environment variable
|
||||
|
||||
### No issuers showing up
|
||||
- Ensure env vars are set on the server container
|
||||
- Restart server: `docker compose restart certctl-server`
|
||||
- Check server logs for validation errors
|
||||
|
||||
### Let's Encrypt rate limits
|
||||
- Use the staging directory for testing (unlimited, untrusted certs)
|
||||
- Production directory: 50 certs per domain per week
|
||||
- Read more: https://letsencrypt.org/docs/rate-limits/
|
||||
|
||||
## Next Steps
|
||||
|
||||
- **Create a certificate profile** — Dashboard → Profiles → + New Profile
|
||||
- **Configure team ownership** — Dashboard → Owners/Teams (assign certs to teams)
|
||||
- **Set renewal policies** — Dashboard → Policies (expiration thresholds, auto-renewal)
|
||||
- **Enable notifications** — Configure Slack/Teams webhook to get alerts on renewals and expirations
|
||||
- **Explore discovery** — Agent scans `/etc/nginx/ssl` and `/etc/app/ssl`, Dashboard → Discovery shows what's already deployed
|
||||
|
||||
## Further Reading
|
||||
|
||||
- [certctl Architecture](../../docs/architecture.md)
|
||||
- [ACME Connector Docs](../../docs/connectors.md#acme-letsencrypt)
|
||||
- [Local CA Connector Docs](../../docs/connectors.md#local-ca)
|
||||
- [Agent Configuration](../../docs/agent.md)
|
||||
- [Deployment Targets](../../docs/connectors.md#deployment-targets)
|
||||
@@ -0,0 +1,150 @@
|
||||
version: '3.8'
|
||||
|
||||
services:
|
||||
# PostgreSQL database for certctl
|
||||
postgres:
|
||||
image: postgres:16-alpine
|
||||
container_name: certctl-postgres-multi-issuer
|
||||
environment:
|
||||
POSTGRES_DB: certctl
|
||||
POSTGRES_USER: certctl
|
||||
POSTGRES_PASSWORD: ${DB_PASSWORD:-certctl-dev-password}
|
||||
volumes:
|
||||
- postgres_data:/var/lib/postgresql/data
|
||||
healthcheck:
|
||||
test: ['CMD-SHELL', 'pg_isready -U certctl -d certctl']
|
||||
interval: 5s
|
||||
timeout: 5s
|
||||
retries: 5
|
||||
networks:
|
||||
- certctl-network
|
||||
restart: unless-stopped
|
||||
|
||||
# certctl server (control plane)
|
||||
# Configured with BOTH ACME (Let's Encrypt) and Local CA issuers
|
||||
certctl-server:
|
||||
image: ghcr.io/shankar0123/certctl-server:latest
|
||||
container_name: certctl-server-multi-issuer
|
||||
environment:
|
||||
# Database
|
||||
DATABASE_URL: postgres://certctl:${DB_PASSWORD:-certctl-dev-password}@postgres:5432/certctl?sslmode=disable
|
||||
|
||||
# Server settings
|
||||
CERTCTL_SERVER_PORT: 8443
|
||||
CERTCTL_SERVER_HOST: 0.0.0.0
|
||||
|
||||
# Auth (disabled for demo; production should use API keys)
|
||||
CERTCTL_AUTH_TYPE: none
|
||||
|
||||
# CORS (allow agent communication)
|
||||
CERTCTL_CORS_ORIGINS: '*'
|
||||
|
||||
# Key generation mode (agent-side in production, server-side for demo)
|
||||
CERTCTL_KEYGEN_MODE: server
|
||||
|
||||
# ACME issuer (Let's Encrypt for public-facing services)
|
||||
# Change CERTCTL_ACME_EMAIL to your email and CERTCTL_ACME_CHALLENGE_TYPE as needed
|
||||
CERTCTL_ACME_DIRECTORY_URL: https://acme-v02.api.letsencrypt.org/directory
|
||||
CERTCTL_ACME_EMAIL: ${ACME_EMAIL:-admin@example.com}
|
||||
CERTCTL_ACME_CHALLENGE_TYPE: http-01
|
||||
|
||||
# Local CA issuer (for internal services - self-signed or sub-CA)
|
||||
# Set these paths if you have an existing CA cert+key for sub-CA mode
|
||||
# Otherwise, leave empty for self-signed CA generation
|
||||
CERTCTL_CA_CERT_PATH: ${CA_CERT_PATH:-}
|
||||
CERTCTL_CA_KEY_PATH: ${CA_KEY_PATH:-}
|
||||
|
||||
# Logging
|
||||
CERTCTL_LOG_LEVEL: info
|
||||
ports:
|
||||
- '${SERVER_PORT:-8443}:8443'
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
networks:
|
||||
- certctl-network
|
||||
healthcheck:
|
||||
test: ['CMD-SHELL', 'curl -sf http://localhost:8443/api/v1/health || exit 1']
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
restart: unless-stopped
|
||||
|
||||
# certctl agent (manages certificates on NGINX and application servers)
|
||||
certctl-agent:
|
||||
image: ghcr.io/shankar0123/certctl-agent:latest
|
||||
container_name: certctl-agent-multi-issuer
|
||||
environment:
|
||||
# Control plane connection
|
||||
CERTCTL_SERVER_URL: http://certctl-server:8443
|
||||
CERTCTL_API_KEY: ${AGENT_API_KEY:-agent-demo-key}
|
||||
|
||||
# Key generation (agent-side keys, never sent to server)
|
||||
CERTCTL_KEYGEN_MODE: server
|
||||
CERTCTL_KEY_DIR: /var/lib/certctl/keys
|
||||
|
||||
# Discovery (scan existing certs to track what's already deployed)
|
||||
CERTCTL_DISCOVERY_DIRS: /etc/nginx/ssl:/etc/app/ssl
|
||||
|
||||
# Heartbeat interval
|
||||
CERTCTL_HEARTBEAT_INTERVAL: 30s
|
||||
|
||||
# Agent metadata
|
||||
CERTCTL_AGENT_NAME: multi-issuer-agent-01
|
||||
|
||||
# Logging
|
||||
CERTCTL_LOG_LEVEL: info
|
||||
volumes:
|
||||
# Mount NGINX cert directories
|
||||
- nginx_certs:/etc/nginx/ssl
|
||||
- nginx_conf:/etc/nginx/conf.d
|
||||
# Mount application service cert directory
|
||||
- app_certs:/etc/app/ssl
|
||||
# Agent key storage (persisted across restarts)
|
||||
- agent_keys:/var/lib/certctl/keys
|
||||
depends_on:
|
||||
certctl-server:
|
||||
condition: service_healthy
|
||||
networks:
|
||||
- certctl-network
|
||||
restart: unless-stopped
|
||||
|
||||
# NGINX reverse proxy / web server
|
||||
# This is where public TLS certs (from ACME) will be deployed
|
||||
nginx:
|
||||
image: nginx:alpine
|
||||
container_name: certctl-nginx-multi-issuer
|
||||
ports:
|
||||
- '80:80'
|
||||
- '443:443'
|
||||
volumes:
|
||||
- nginx_conf:/etc/nginx/conf.d
|
||||
- nginx_certs:/etc/nginx/ssl
|
||||
# Default NGINX config
|
||||
- ./nginx.conf:/etc/nginx/nginx.conf:ro
|
||||
depends_on:
|
||||
- certctl-agent
|
||||
networks:
|
||||
- certctl-network
|
||||
healthcheck:
|
||||
test: ['CMD-SHELL', 'wget --quiet --tries=1 --spider http://localhost/ || exit 1']
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
restart: unless-stopped
|
||||
|
||||
networks:
|
||||
certctl-network:
|
||||
driver: bridge
|
||||
|
||||
volumes:
|
||||
postgres_data:
|
||||
driver: local
|
||||
nginx_certs:
|
||||
driver: local
|
||||
nginx_conf:
|
||||
driver: local
|
||||
app_certs:
|
||||
driver: local
|
||||
agent_keys:
|
||||
driver: local
|
||||
@@ -0,0 +1,358 @@
|
||||
# Private CA + Traefik Example
|
||||
|
||||
This example demonstrates certctl managing certificates for **internal services without public CA dependency**. Ideal for enterprise environments where:
|
||||
|
||||
- All services are internal (VPN, private networks)
|
||||
- You need unified certificate lifecycle management across multiple internal apps
|
||||
- You want automatic cert deployment to your reverse proxy
|
||||
- You may have an existing enterprise root CA (ADCS, OpenCA, etc.)
|
||||
|
||||
## What's Included
|
||||
|
||||
- **certctl server** with Local CA issuer (self-signed or sub-CA mode)
|
||||
- **certctl agent** that deploys certificates to Traefik
|
||||
- **Traefik** reverse proxy with file provider for dynamic cert discovery
|
||||
- **PostgreSQL** database for certificate storage and audit trail
|
||||
- Automatic certificate discovery for existing certs in Traefik
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌──────────────────┐
|
||||
│ certctl-server │ (Local CA issuer)
|
||||
│ (control │
|
||||
│ plane) │
|
||||
└────────┬─────────┘
|
||||
│
|
||||
│ REST API (job polling)
|
||||
│
|
||||
┌────────▼──────────┐
|
||||
│ certctl-agent │ (certificate deployer)
|
||||
└────────┬──────────┘
|
||||
│
|
||||
│ Write cert/key files
|
||||
│
|
||||
┌────────▼──────────────────────┐
|
||||
│ Traefik │
|
||||
│ (watches cert directory) │
|
||||
└────────────────────────────────┘
|
||||
│
|
||||
│ TLS handshakes
|
||||
│
|
||||
[Internal Services]
|
||||
```
|
||||
|
||||
## Quick Start (Self-Signed CA)
|
||||
|
||||
The simplest way to get running in 2 minutes:
|
||||
|
||||
```bash
|
||||
# 1. Create directory structure
|
||||
mkdir -p traefik-config ca-certs
|
||||
|
||||
# 2. Create a minimal Traefik dynamic config
|
||||
cat > traefik-config/default.yaml << 'EOF'
|
||||
# Traefik will auto-load certificates from /etc/traefik/certs
|
||||
# Certctl deploys {cert-id}.crt and {cert-id}.key files here
|
||||
http:
|
||||
routers:
|
||||
api:
|
||||
rule: "Host(`api.internal.local`)"
|
||||
service: api-service
|
||||
tls: {}
|
||||
services:
|
||||
api-service:
|
||||
loadBalancer:
|
||||
servers:
|
||||
- url: "http://localhost:3000"
|
||||
EOF
|
||||
|
||||
# 3. Start the stack
|
||||
docker compose up -d
|
||||
|
||||
# 4. Access the dashboards
|
||||
# - certctl: http://localhost:8443 (API only, use the CLI or direct HTTP calls)
|
||||
# - Traefik dashboard: http://localhost:8080
|
||||
```
|
||||
|
||||
The self-signed CA will be automatically generated on first startup.
|
||||
|
||||
## Using Sub-CA Mode (Enterprise Root CA)
|
||||
|
||||
If you have an existing enterprise CA (ADCS, OpenCA, etc.) and want issued certs to chain to your root:
|
||||
|
||||
```bash
|
||||
# 1. Create directory structure
|
||||
mkdir -p traefik-config ca-certs
|
||||
|
||||
# 2. Copy your enterprise CA cert and key
|
||||
cp /path/to/your/enterprise-ca.crt ca-certs/ca-cert.pem
|
||||
cp /path/to/your/enterprise-ca-key.pem ca-certs/ca-key.pem
|
||||
|
||||
# 3. Edit docker-compose.yml and uncomment the sub-CA env vars:
|
||||
# CERTCTL_CA_CERT_PATH: /etc/certctl/ca-cert.pem
|
||||
# CERTCTL_CA_KEY_PATH: /etc/certctl/ca-key.pem
|
||||
|
||||
# 4. Create the dynamic config (same as above)
|
||||
mkdir -p traefik-config
|
||||
cat > traefik-config/default.yaml << 'EOF'
|
||||
http:
|
||||
routers:
|
||||
api:
|
||||
rule: "Host(`api.internal.local`)"
|
||||
service: api-service
|
||||
tls: {}
|
||||
services:
|
||||
api-service:
|
||||
loadBalancer:
|
||||
servers:
|
||||
- url: "http://localhost:3000"
|
||||
EOF
|
||||
|
||||
# 5. Start the stack
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
**Requirements for sub-CA mode:**
|
||||
- CA certificate must have `X509v3 Basic Constraints: CA:TRUE`
|
||||
- CA certificate must have `X509v3 Key Usage: Certificate Sign`
|
||||
- Key format: RSA, ECDSA, or PKCS#8
|
||||
- Paths: must be absolute paths to mounted files
|
||||
|
||||
## Creating a Certificate
|
||||
|
||||
Once the stack is running:
|
||||
|
||||
```bash
|
||||
# 1. Create a certificate profile in certctl (defines allowed key types, TTL, etc.)
|
||||
curl -X POST http://localhost:8443/api/v1/profiles \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"id": "prof-internal",
|
||||
"name": "Internal Services",
|
||||
"description": "For internal APIs and web apps",
|
||||
"max_ttl_hours": 8760,
|
||||
"key_types": ["rsa-2048", "ecdsa-p256"]
|
||||
}'
|
||||
|
||||
# 2. Create a renewal policy (defines issuer, renewal thresholds, etc.)
|
||||
curl -X POST http://localhost:8443/api/v1/policies \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"id": "pol-internal",
|
||||
"name": "Internal Renewal Policy",
|
||||
"issuer_id": "iss-local",
|
||||
"profile_id": "prof-internal",
|
||||
"renewal_threshold_days": 30,
|
||||
"alert_thresholds_days": [30, 14, 7, 0]
|
||||
}'
|
||||
|
||||
# 3. Create a certificate (triggers issuance immediately)
|
||||
curl -X POST http://localhost:8443/api/v1/certificates \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"common_name": "api.internal.local",
|
||||
"sans": ["app.internal.local", "www.internal.local"],
|
||||
"policy_id": "pol-internal"
|
||||
}'
|
||||
|
||||
# 4. Create a Traefik target (agent will deploy to this)
|
||||
curl -X POST http://localhost:8443/api/v1/targets \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"id": "target-traefik-01",
|
||||
"name": "Traefik Primary",
|
||||
"type": "traefik",
|
||||
"config": {
|
||||
"cert_dir": "/etc/traefik/certs"
|
||||
}
|
||||
}'
|
||||
|
||||
# 5. Create a deployment job (agent picks this up and deploys)
|
||||
curl -X POST http://localhost:8443/api/v1/certificates/{cert-id}/deploy \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"target_ids": ["target-traefik-01"]
|
||||
}'
|
||||
```
|
||||
|
||||
Once deployed, Traefik automatically loads the new certificate from the certs directory.
|
||||
|
||||
## How It Works
|
||||
|
||||
### Certificate Lifecycle
|
||||
|
||||
1. **Issue** — certctl-server generates certificate from Local CA (self-signed or sub-CA)
|
||||
2. **Store** — certificate stored in PostgreSQL with full audit trail
|
||||
3. **Deploy** — certctl-agent writes `{cert-id}.crt` + `{cert-id}.key` to `/etc/traefik/certs`
|
||||
4. **Reload** — Traefik file provider detects new files and hot-loads them (zero downtime)
|
||||
5. **Monitor** — certctl tracks deployment status and renewal timelines
|
||||
|
||||
### Self-Signed CA
|
||||
|
||||
- Generated automatically on first startup if `CERTCTL_CA_CERT_PATH` and `CERTCTL_CA_KEY_PATH` are not set
|
||||
- Certificate stored in server's in-memory state (not persisted)
|
||||
- All issued certs chain to this self-signed root
|
||||
- Use this for: demos, development, internal labs
|
||||
|
||||
### Sub-CA Mode
|
||||
|
||||
- Requires you to provide an existing CA certificate and key
|
||||
- Issued certificates chain to your enterprise root CA
|
||||
- All issued certs are trustworthy to systems with your root CA in their trust store
|
||||
- Use this for: production internal services, compliance requirements, enterprise PKI
|
||||
|
||||
## File Organization
|
||||
|
||||
```
|
||||
private-ca-traefik/
|
||||
├── docker-compose.yml # Stack definition
|
||||
├── traefik-config/ # Traefik dynamic config (you create)
|
||||
│ └── default.yaml # Routing rules and TLS settings
|
||||
├── ca-certs/ # CA certificate and key (for sub-CA mode)
|
||||
│ ├── ca-cert.pem # Your enterprise CA certificate
|
||||
│ └── ca-key.pem # Your enterprise CA private key
|
||||
└── README.md # This file
|
||||
```
|
||||
|
||||
## Monitoring
|
||||
|
||||
### certctl Dashboard
|
||||
The server provides a REST API on port 8443. Example queries:
|
||||
|
||||
```bash
|
||||
# List all certificates
|
||||
curl http://localhost:8443/api/v1/certificates
|
||||
|
||||
# Check certificate status
|
||||
curl http://localhost:8443/api/v1/certificates/{cert-id}
|
||||
|
||||
# View audit trail
|
||||
curl http://localhost:8443/api/v1/audit
|
||||
|
||||
# Check renewal policy compliance
|
||||
curl http://localhost:8443/api/v1/policies/{policy-id}
|
||||
```
|
||||
|
||||
### Traefik Dashboard
|
||||
http://localhost:8080 shows:
|
||||
- HTTP routers and services
|
||||
- TLS certificates currently loaded
|
||||
- Request/response metrics
|
||||
|
||||
### Logs
|
||||
```bash
|
||||
# certctl server logs
|
||||
docker compose logs certctl-server
|
||||
|
||||
# certctl agent logs
|
||||
docker compose logs certctl-agent
|
||||
|
||||
# Traefik logs
|
||||
docker compose logs traefik
|
||||
```
|
||||
|
||||
## Customizing Traefik Config
|
||||
|
||||
Edit `traefik-config/default.yaml` to add routers for your services:
|
||||
|
||||
```yaml
|
||||
http:
|
||||
routers:
|
||||
# Internal API
|
||||
api:
|
||||
rule: "Host(`api.internal.local`)"
|
||||
service: api-service
|
||||
tls: {}
|
||||
|
||||
# Web application
|
||||
webapp:
|
||||
rule: "Host(`app.internal.local`)"
|
||||
service: webapp-service
|
||||
tls: {}
|
||||
|
||||
services:
|
||||
api-service:
|
||||
loadBalancer:
|
||||
servers:
|
||||
- url: "http://api-backend:3000"
|
||||
|
||||
webapp-service:
|
||||
loadBalancer:
|
||||
servers:
|
||||
- url: "http://webapp-backend:3001"
|
||||
```
|
||||
|
||||
Changes are picked up automatically (file watcher enabled).
|
||||
|
||||
## Production Considerations
|
||||
|
||||
1. **Use sub-CA mode** — chain to your enterprise root for full trust
|
||||
2. **Enable API key authentication** — set `CERTCTL_AUTH_TYPE: api-key` and `CERTCTL_API_KEY`
|
||||
3. **Use agent-side key generation** — set `CERTCTL_KEYGEN_MODE: agent` (keys never leave agents)
|
||||
4. **Back up PostgreSQL** — certificate data is authoritative; database loss means certificate loss
|
||||
5. **Monitor renewal windows** — set up alerts on policy thresholds
|
||||
6. **Rotate CA keys regularly** — plan for future CA refresh (sub-CA mode)
|
||||
7. **Audit certificate usage** — review `certctl_audit_events` for compliance
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Certificates not deploying
|
||||
```bash
|
||||
# Check agent is healthy
|
||||
docker compose logs certctl-agent | grep heartbeat
|
||||
|
||||
# Check deployment job status
|
||||
curl http://localhost:8443/api/v1/jobs | jq '.[] | select(.type == "Deployment")'
|
||||
|
||||
# Check Traefik is watching the directory
|
||||
docker compose exec traefik ls -la /etc/traefik/certs/
|
||||
```
|
||||
|
||||
### Traefik not reloading certs
|
||||
```bash
|
||||
# Verify file provider is enabled (check docker-compose.yml command)
|
||||
# Verify certs volume is mounted at /etc/traefik/certs
|
||||
# Check Traefik logs
|
||||
docker compose logs traefik | grep "file"
|
||||
```
|
||||
|
||||
### CA cert not loading in sub-CA mode
|
||||
```bash
|
||||
# Verify file permissions
|
||||
docker compose exec certctl-server ls -la /etc/certctl/
|
||||
|
||||
# Check server logs for CA loading errors
|
||||
docker compose logs certctl-server | grep -i "ca\|cert"
|
||||
|
||||
# Verify CA certificate format
|
||||
openssl x509 -in ca-certs/ca-cert.pem -text -noout | grep -A 3 "Basic Constraints"
|
||||
```
|
||||
|
||||
## Cleanup
|
||||
|
||||
```bash
|
||||
# Stop all services
|
||||
docker compose down
|
||||
|
||||
# Remove all data (certificates, database, etc.)
|
||||
docker compose down -v
|
||||
|
||||
# Remove CA cert files (if using custom CA)
|
||||
rm -rf ca-certs/
|
||||
```
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. **Add more services** — create additional routers and backends in `traefik-config/default.yaml`
|
||||
2. **Set up renewal automation** — configure renewal policies with thresholds
|
||||
3. **Integrate with monitoring** — expose certctl metrics to Prometheus
|
||||
4. **Enable notifications** — configure email/Slack alerts on certificate events
|
||||
5. **Scale to multiple environments** — deploy separate certctl stacks per environment (dev/staging/prod)
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [certctl Architecture](../../docs/architecture.md)
|
||||
- [Traefik File Provider](https://doc.traefik.io/traefik/providers/file/)
|
||||
- [Local CA Sub-CA Mode](../../docs/connectors.md#local-ca)
|
||||
- [Certificate Profiles](../../docs/quickstart.md#profiles)
|
||||
@@ -0,0 +1,182 @@
|
||||
version: '3.8'
|
||||
|
||||
services:
|
||||
# PostgreSQL database for certctl
|
||||
postgres:
|
||||
image: postgres:16-alpine
|
||||
container_name: certctl-postgres-private-ca
|
||||
environment:
|
||||
POSTGRES_DB: certctl
|
||||
POSTGRES_USER: certctl
|
||||
POSTGRES_PASSWORD: ${DB_PASSWORD:-certctl-dev-password}
|
||||
volumes:
|
||||
- postgres_data:/var/lib/postgresql/data
|
||||
healthcheck:
|
||||
test: ['CMD-SHELL', 'pg_isready -U certctl -d certctl']
|
||||
interval: 5s
|
||||
timeout: 5s
|
||||
retries: 5
|
||||
networks:
|
||||
- certctl-network
|
||||
restart: unless-stopped
|
||||
|
||||
# certctl server (control plane) with Local CA in sub-CA mode
|
||||
certctl-server:
|
||||
image: ghcr.io/shankar0123/certctl-server:latest
|
||||
container_name: certctl-server-private-ca
|
||||
environment:
|
||||
# Database
|
||||
DATABASE_URL: postgres://certctl:${DB_PASSWORD:-certctl-dev-password}@postgres:5432/certctl?sslmode=disable
|
||||
|
||||
# Server settings
|
||||
CERTCTL_SERVER_PORT: 8443
|
||||
CERTCTL_SERVER_HOST: 0.0.0.0
|
||||
|
||||
# Auth (disabled for demo; production should use API keys)
|
||||
CERTCTL_AUTH_TYPE: none
|
||||
|
||||
# CORS (allow agent and Traefik communication)
|
||||
CERTCTL_CORS_ORIGINS: '*'
|
||||
|
||||
# Key generation mode (agent-side in production, server-side for demo)
|
||||
CERTCTL_KEYGEN_MODE: server
|
||||
|
||||
# Local CA configuration
|
||||
# For self-signed CA (default, no paths set):
|
||||
# - CA generates a self-signed root certificate
|
||||
# - All issued certificates chain to this root
|
||||
#
|
||||
# For sub-CA mode (provide both paths):
|
||||
# - Load pre-signed CA certificate and key from these paths
|
||||
# - All issued certificates chain to your enterprise root CA
|
||||
# - Requires: CA cert must have IsCA=true and KeyUsageCertSign
|
||||
# - Supports: RSA, ECDSA, PKCS#8 key formats
|
||||
#
|
||||
# To use sub-CA mode:
|
||||
# 1. Place your enterprise CA cert at ./ca-cert.pem
|
||||
# 2. Place your enterprise CA key at ./ca-key.pem
|
||||
# 3. Uncomment the two lines below
|
||||
# 4. Restart the service
|
||||
#
|
||||
# CERTCTL_CA_CERT_PATH: /etc/certctl/ca-cert.pem
|
||||
# CERTCTL_CA_KEY_PATH: /etc/certctl/ca-key.pem
|
||||
|
||||
# Logging
|
||||
CERTCTL_LOG_LEVEL: info
|
||||
ports:
|
||||
- '${SERVER_PORT:-8443}:8443'
|
||||
volumes:
|
||||
# Mount directory for CA cert/key (for sub-CA mode)
|
||||
# Copy your enterprise CA cert+key here:
|
||||
# cp /path/to/your/ca.pem ./ca-cert.pem
|
||||
# cp /path/to/your/ca-key.pem ./ca-key.pem
|
||||
- ./ca-certs:/etc/certctl:ro
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
networks:
|
||||
- certctl-network
|
||||
healthcheck:
|
||||
test: ['CMD-SHELL', 'curl -sf http://localhost:8443/api/v1/health || exit 1']
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
restart: unless-stopped
|
||||
|
||||
# certctl agent (deploys certs to Traefik)
|
||||
certctl-agent:
|
||||
image: ghcr.io/shankar0123/certctl-agent:latest
|
||||
container_name: certctl-agent-private-ca
|
||||
environment:
|
||||
# Control plane connection
|
||||
CERTCTL_SERVER_URL: http://certctl-server:8443
|
||||
CERTCTL_API_KEY: ${AGENT_API_KEY:-agent-demo-key}
|
||||
|
||||
# Key generation (agent-side keys, never sent to server)
|
||||
CERTCTL_KEYGEN_MODE: server
|
||||
CERTCTL_KEY_DIR: /var/lib/certctl/keys
|
||||
|
||||
# Discovery (scan for existing certs in Traefik's directory)
|
||||
CERTCTL_DISCOVERY_DIRS: /etc/traefik/certs
|
||||
|
||||
# Heartbeat interval
|
||||
CERTCTL_HEARTBEAT_INTERVAL: 30s
|
||||
|
||||
# Agent metadata (self-reported)
|
||||
CERTCTL_AGENT_NAME: traefik-agent-01
|
||||
|
||||
# Logging
|
||||
CERTCTL_LOG_LEVEL: info
|
||||
volumes:
|
||||
# Mount Traefik cert directory for deployment
|
||||
- traefik_certs:/etc/traefik/certs
|
||||
# Agent key storage (persisted across restarts)
|
||||
- agent_keys:/var/lib/certctl/keys
|
||||
depends_on:
|
||||
certctl-server:
|
||||
condition: service_healthy
|
||||
networks:
|
||||
- certctl-network
|
||||
restart: unless-stopped
|
||||
|
||||
# Traefik reverse proxy / edge router
|
||||
# Certificates deployed by certctl-agent are automatically loaded from the certs directory
|
||||
traefik:
|
||||
image: traefik:v3.0
|
||||
container_name: certctl-traefik-private-ca
|
||||
command:
|
||||
# Enable dashboard and API
|
||||
- '--api.insecure=true'
|
||||
- '--api.dashboard=true'
|
||||
|
||||
# File provider: watch the certs directory for dynamic config updates
|
||||
- '--providers.file.directory=/etc/traefik/dynamic'
|
||||
- '--providers.file.watch=true'
|
||||
|
||||
# Entry points (HTTP and HTTPS)
|
||||
- '--entrypoints.web.address=:80'
|
||||
- '--entrypoints.websecure.address=:443'
|
||||
- '--entrypoints.websecure.http.tls=true'
|
||||
|
||||
# Global TLS settings
|
||||
- '--entryPoints.websecure.http.tls.certResolver=internal'
|
||||
|
||||
# Logging
|
||||
- '--log.level=info'
|
||||
- '--accesslog=true'
|
||||
ports:
|
||||
# HTTP
|
||||
- '80:80'
|
||||
# HTTPS
|
||||
- '443:443'
|
||||
# Dashboard (http://localhost:8080)
|
||||
- '8080:8080'
|
||||
volumes:
|
||||
# Mount Traefik config directory
|
||||
- ./traefik-config:/etc/traefik/dynamic:ro
|
||||
# Mount cert directory (where certctl deploys certs)
|
||||
- traefik_certs:/etc/traefik/certs:ro
|
||||
# Allow Traefik to read Docker socket (optional, for container labeling)
|
||||
- /var/run/docker.sock:/var/run/docker.sock:ro
|
||||
networks:
|
||||
- certctl-network
|
||||
depends_on:
|
||||
- certctl-agent
|
||||
healthcheck:
|
||||
test: ['CMD-SHELL', 'curl -sf http://localhost:8080/ping || exit 1']
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
restart: unless-stopped
|
||||
|
||||
networks:
|
||||
certctl-network:
|
||||
driver: bridge
|
||||
|
||||
volumes:
|
||||
postgres_data:
|
||||
driver: local
|
||||
traefik_certs:
|
||||
driver: local
|
||||
agent_keys:
|
||||
driver: local
|
||||
@@ -0,0 +1,355 @@
|
||||
# step-ca + HAProxy Example
|
||||
|
||||
This example demonstrates certctl managing certificates issued by **Smallstep step-ca** and deploying them to **HAProxy**.
|
||||
|
||||
## Scenario
|
||||
|
||||
You're a Smallstep user running step-ca as your internal PKI. You have HAProxy load balancers that need certificates. This setup:
|
||||
|
||||
1. **step-ca** issues certificates (via JWK provisioner, no challenge solving)
|
||||
2. **certctl** manages the certificate lifecycle (renewal policies, deployment, audit)
|
||||
3. **HAProxy** serves HTTPS with certificates managed by certctl
|
||||
|
||||
This is the natural choice if you're already invested in step-ca and want to consolidate certificate lifecycle management without learning Let's Encrypt, DNS-01 challenges, or external integrations.
|
||||
|
||||
## What's Included
|
||||
|
||||
| Service | Image | Purpose |
|
||||
|---------|-------|---------|
|
||||
| **step-ca** | `smallstep/step-ca:latest` | Private internal CA |
|
||||
| **certctl-server** | `ghcr.io/shankar0123/certctl-server:latest` | Certificate management control plane |
|
||||
| **certctl-agent** | `ghcr.io/shankar0123/certctl-agent:latest` | Agent running on HAProxy server |
|
||||
| **haproxy** | `haproxy:2.9-alpine` | Reverse proxy / load balancer |
|
||||
| **postgres** | `postgres:16-alpine` | certctl audit trail + config storage |
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- Docker and Docker Compose
|
||||
- Curl (to interact with APIs)
|
||||
|
||||
### 1. Start Everything
|
||||
|
||||
```bash
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
This will:
|
||||
- Initialize step-ca with a self-signed root CA
|
||||
- Create a JWK provisioner named `certctl` (pre-configured credentials)
|
||||
- Start certctl-server (connected to step-ca)
|
||||
- Start the certctl-agent (ready to deploy certs to HAProxy)
|
||||
- Start HAProxy with a placeholder config
|
||||
|
||||
Monitor logs:
|
||||
|
||||
```bash
|
||||
docker compose logs -f certctl-server
|
||||
```
|
||||
|
||||
Wait for all services to reach healthy state:
|
||||
|
||||
```bash
|
||||
docker compose ps
|
||||
```
|
||||
|
||||
Expected output:
|
||||
```
|
||||
NAME STATUS
|
||||
certctl-postgres-... healthy
|
||||
certctl-server-... healthy
|
||||
step-ca-... healthy
|
||||
certctl-agent-... running
|
||||
certctl-haproxy-... healthy
|
||||
```
|
||||
|
||||
### 2. Access certctl Dashboard
|
||||
|
||||
Open your browser to:
|
||||
|
||||
```
|
||||
http://localhost:8443
|
||||
```
|
||||
|
||||
You should see an empty dashboard. This is expected — no certificates issued yet.
|
||||
|
||||
### 3. Create a Certificate Profile
|
||||
|
||||
This defines what certificates certctl can issue (key algorithm, max TTL, allowed names).
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:8443/api/v1/profiles \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{
|
||||
"name": "internal-web",
|
||||
"key_type": "rsa-2048",
|
||||
"max_ttl_days": 90,
|
||||
"description": "Internal web services"
|
||||
}'
|
||||
```
|
||||
|
||||
### 4. Create an HAProxy Deployment Target
|
||||
|
||||
This tells certctl where to deploy certificates on the HAProxy server.
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:8443/api/v1/targets \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{
|
||||
"name": "haproxy-01",
|
||||
"type": "haproxy",
|
||||
"enabled": true,
|
||||
"config": {
|
||||
"pem_path": "/etc/haproxy/ssl/cert.pem",
|
||||
"reload_command": "systemctl reload haproxy",
|
||||
"validate_command": "haproxy -c -f /etc/haproxy/haproxy.cfg"
|
||||
}
|
||||
}'
|
||||
```
|
||||
|
||||
Note: In the Docker Compose environment, reload command can be `kill -HUP $(pidof haproxy)` instead of `systemctl reload haproxy`.
|
||||
|
||||
### 5. Create a Renewal Policy
|
||||
|
||||
This ties a certificate profile to a deployment target and sets renewal thresholds.
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:8443/api/v1/renewal-policies \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{
|
||||
"name": "haproxy-internal-web",
|
||||
"profile_id": "<profile_id_from_step_3>",
|
||||
"issuer_id": "iss-stepca",
|
||||
"enabled": true,
|
||||
"renewal_days_before_expiry": 30,
|
||||
"alert_thresholds_days": [30, 14, 7, 0]
|
||||
}'
|
||||
```
|
||||
|
||||
Get the issuer ID:
|
||||
|
||||
```bash
|
||||
curl http://localhost:8443/api/v1/issuers | jq '.'
|
||||
```
|
||||
|
||||
You should see `iss-stepca` in the list.
|
||||
|
||||
### 6. Issue a Certificate
|
||||
|
||||
Request a certificate via the API. The server will sign it via step-ca.
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:8443/api/v1/certificates \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{
|
||||
"common_name": "api.internal.example.com",
|
||||
"sans": ["api.internal.example.com", "api.staging.example.com"],
|
||||
"issuer_id": "iss-stepca",
|
||||
"profile_id": "<profile_id_from_step_3>"
|
||||
}'
|
||||
```
|
||||
|
||||
### 7. Deploy to HAProxy
|
||||
|
||||
Get the certificate ID and trigger deployment:
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:8443/api/v1/certificates/<cert_id>/deploy \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{
|
||||
"target_id": "<target_id_from_step_4>"
|
||||
}'
|
||||
```
|
||||
|
||||
The agent will:
|
||||
1. Fetch the deployment job
|
||||
2. Generate a combined PEM (cert + chain + key) locally
|
||||
3. Write it to `/etc/haproxy/ssl/cert.pem` on HAProxy
|
||||
4. Reload HAProxy
|
||||
5. Report status back to certctl
|
||||
|
||||
### 8. Verify in Dashboard
|
||||
|
||||
Refresh http://localhost:8443 and you should see:
|
||||
- 1 certificate (status: Active, expiry in 90 days)
|
||||
- 1 deployment job (status: Completed)
|
||||
- 1 agent (heartbeat: recent)
|
||||
|
||||
## Configuration Details
|
||||
|
||||
### step-ca Integration
|
||||
|
||||
step-ca is configured with:
|
||||
|
||||
- **Root CA Name**: `certctl-demo-ca`
|
||||
- **Provisioner**: `certctl` (JWK type)
|
||||
- **Default Password**: `certctl-provisioner-demo` (override with `STEP_CA_PROVISIONER_PASSWORD`)
|
||||
|
||||
To inspect step-ca:
|
||||
|
||||
```bash
|
||||
docker compose exec step-ca step ca provisioner list
|
||||
docker compose exec step-ca step ca health --insecure
|
||||
```
|
||||
|
||||
### HAProxy Combined PEM Format
|
||||
|
||||
HAProxy requires a single file with certificate, chain, and key concatenated:
|
||||
|
||||
```
|
||||
-----BEGIN CERTIFICATE-----
|
||||
[leaf certificate]
|
||||
-----END CERTIFICATE-----
|
||||
-----BEGIN CERTIFICATE-----
|
||||
[intermediate CA]
|
||||
-----END CERTIFICATE-----
|
||||
-----BEGIN RSA PRIVATE KEY-----
|
||||
[private key]
|
||||
-----END RSA PRIVATE KEY-----
|
||||
```
|
||||
|
||||
The agent automatically constructs this file from the issued certificate and step-ca-provided chain.
|
||||
|
||||
**Security**: The combined PEM is written with `0600` permissions (owner-readable only) because it contains the private key.
|
||||
|
||||
### Environment Variables
|
||||
|
||||
Customize behavior with:
|
||||
|
||||
| Variable | Default | Purpose |
|
||||
|----------|---------|---------|
|
||||
| `DB_PASSWORD` | `certctl-dev-password` | PostgreSQL password |
|
||||
| `STEP_CA_PASSWORD` | `stepca-demo-password` | step-ca root key password |
|
||||
| `STEP_CA_PROVISIONER_PASSWORD` | `certctl-provisioner-demo` | certctl JWK provisioner password |
|
||||
| `AGENT_API_KEY` | `agent-demo-key` | Agent authentication token |
|
||||
| `SERVER_PORT` | `8443` | certctl server external port |
|
||||
|
||||
Example:
|
||||
|
||||
```bash
|
||||
STEP_CA_PASSWORD=myca-password AGENT_API_KEY=secret-key docker compose up -d
|
||||
```
|
||||
|
||||
## Integrating with an Existing step-ca Instance
|
||||
|
||||
If you already run step-ca elsewhere (not in this Compose file):
|
||||
|
||||
1. **Extract the root certificate** from your step-ca:
|
||||
|
||||
```bash
|
||||
step ca root /tmp/step-ca-root.crt --ca-url https://ca.internal:9000 --insecure
|
||||
```
|
||||
|
||||
2. **Create or retrieve the certctl JWK provisioner key**:
|
||||
|
||||
```bash
|
||||
step ca provisioner list --ca-url https://ca.internal:9000 --insecure
|
||||
step ca provisioner describe certctl --ca-url https://ca.internal:9000 --insecure
|
||||
```
|
||||
|
||||
3. **Update docker-compose.yml**:
|
||||
|
||||
```yaml
|
||||
certctl-server:
|
||||
environment:
|
||||
CERTCTL_STEPCA_URL: https://ca.internal:9000
|
||||
CERTCTL_STEPCA_ROOT_CERT_PATH: /etc/certctl/step-ca-root.crt
|
||||
CERTCTL_STEPCA_PROVISIONER_NAME: certctl
|
||||
CERTCTL_STEPCA_PROVISIONER_KEY_PATH: /etc/certctl/step-ca-provisioner.json
|
||||
CERTCTL_STEPCA_PROVISIONER_PASSWORD: <your-password>
|
||||
```
|
||||
|
||||
4. **Mount the cert and key**:
|
||||
|
||||
```yaml
|
||||
volumes:
|
||||
- /path/to/step-ca-root.crt:/etc/certctl/step-ca-root.crt:ro
|
||||
- /path/to/provisioner.json:/etc/certctl/step-ca-provisioner.json:ro
|
||||
```
|
||||
|
||||
## Cleanup
|
||||
|
||||
```bash
|
||||
docker compose down -v
|
||||
```
|
||||
|
||||
This removes all containers and volumes (step-ca config, certificates, database).
|
||||
|
||||
## Next Steps
|
||||
|
||||
### Production Deployment
|
||||
|
||||
- Replace image tags (`latest` → specific version)
|
||||
- Use real TLS certificates for step-ca (self-signed is fine internally, but use proper roots for verification)
|
||||
- Configure persistent storage for step-ca keys (HSM or encrypted filesystem)
|
||||
- Set `CERTCTL_AUTH_TYPE: api-key` and rotate API keys regularly
|
||||
- Enable audit trail export for compliance
|
||||
- Configure renewal alerts (Slack, email, PagerDuty)
|
||||
- Run agents on separate machines (not in Compose)
|
||||
|
||||
### Advanced Features
|
||||
|
||||
- **Multiple HAProxy instances**: Create additional targets and agents
|
||||
- **Policy-based renewal**: Set different renewal windows per environment (staging vs. production)
|
||||
- **Approval workflows**: Require manual approval before deploying to production
|
||||
- **Discovery**: Scan existing HAProxy certs and bring them under management
|
||||
- **Network scanning**: Discover TLS endpoints in your network and inventory them
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### step-ca fails to initialize
|
||||
|
||||
Check logs:
|
||||
|
||||
```bash
|
||||
docker compose logs step-ca
|
||||
```
|
||||
|
||||
Common issues:
|
||||
- Permissions on `/home/step/step-ca` volume
|
||||
- Port 9000 already in use
|
||||
|
||||
### Agent can't reach server
|
||||
|
||||
Verify network:
|
||||
|
||||
```bash
|
||||
docker compose exec certctl-agent curl http://certctl-server:8443/api/v1/health
|
||||
```
|
||||
|
||||
### HAProxy config validation fails
|
||||
|
||||
Check HAProxy config syntax:
|
||||
|
||||
```bash
|
||||
docker compose exec haproxy haproxy -c -f /etc/haproxy/haproxy.cfg
|
||||
```
|
||||
|
||||
### Deployment job stays in "Running" state
|
||||
|
||||
Check agent logs:
|
||||
|
||||
```bash
|
||||
docker compose logs certctl-agent
|
||||
```
|
||||
|
||||
Likely causes:
|
||||
- Agent can't write to `/etc/haproxy/ssl/cert.pem` (permissions)
|
||||
- Reload command is misconfigured
|
||||
- HAProxy container is not accessible
|
||||
|
||||
## Documentation
|
||||
|
||||
- [certctl Architecture](../../docs/architecture.md)
|
||||
- [step-ca Connector Docs](../../docs/connectors.md#step-ca)
|
||||
- [HAProxy Target Docs](../../docs/connectors.md#haproxy)
|
||||
- [API Reference](../../api/openapi.yaml)
|
||||
|
||||
## Support
|
||||
|
||||
For issues or questions:
|
||||
|
||||
1. Check the [troubleshooting guide](../../docs/troubleshooting.md)
|
||||
2. Review service logs: `docker compose logs <service>`
|
||||
3. Open an issue on GitHub
|
||||
@@ -0,0 +1,204 @@
|
||||
version: '3.8'
|
||||
|
||||
services:
|
||||
# PostgreSQL database for certctl
|
||||
postgres:
|
||||
image: postgres:16-alpine
|
||||
container_name: certctl-postgres-stepca-haproxy
|
||||
environment:
|
||||
POSTGRES_DB: certctl
|
||||
POSTGRES_USER: certctl
|
||||
POSTGRES_PASSWORD: ${DB_PASSWORD:-certctl-dev-password}
|
||||
volumes:
|
||||
- postgres_data:/var/lib/postgresql/data
|
||||
healthcheck:
|
||||
test: ['CMD-SHELL', 'pg_isready -U certctl -d certctl']
|
||||
interval: 5s
|
||||
timeout: 5s
|
||||
retries: 5
|
||||
networks:
|
||||
- certctl-network
|
||||
restart: unless-stopped
|
||||
|
||||
# Smallstep step-ca (internal private CA)
|
||||
# Initialized with default admin token and provisioner configuration
|
||||
step-ca:
|
||||
image: smallstep/step-ca:latest
|
||||
container_name: step-ca-stepca-haproxy
|
||||
environment:
|
||||
# step-ca root password (for key encryption)
|
||||
STEPPATH: /home/step/step-ca
|
||||
# Provisioner password will be set up below
|
||||
volumes:
|
||||
# Persist step-ca configuration and keys
|
||||
- step_ca_data:/home/step/step-ca
|
||||
- ./step-ca-init.sh:/opt/step-ca-init.sh:ro
|
||||
entrypoint: /bin/sh
|
||||
command:
|
||||
- -c
|
||||
- |
|
||||
# Initialize step-ca if not already done
|
||||
if [ ! -f /home/step/step-ca/config/ca.json ]; then
|
||||
echo "Initializing step-ca..."
|
||||
step ca init \
|
||||
--name="certctl-demo-ca" \
|
||||
--dns=step-ca \
|
||||
--address=0.0.0.0:9000 \
|
||||
--provisioner=admin \
|
||||
--provisioner-password-file=<(echo "${STEP_CA_PASSWORD:-stepca-demo-password}") \
|
||||
--password-file=<(echo "${STEP_CA_PASSWORD:-stepca-demo-password}") \
|
||||
--deployment-type=standalone \
|
||||
--acme 2>&1 || true
|
||||
fi
|
||||
|
||||
# Add a JWK provisioner for certctl if not present
|
||||
if ! step ca provisioner list 2>/dev/null | grep -q "certctl"; then
|
||||
echo "Adding certctl JWK provisioner..."
|
||||
step ca provisioner add certctl \
|
||||
--type=JWK \
|
||||
--password-file=<(echo "${STEP_CA_PROVISIONER_PASSWORD:-certctl-provisioner-demo}") \
|
||||
2>&1 || true
|
||||
fi
|
||||
|
||||
# Start step-ca
|
||||
echo "Starting step-ca..."
|
||||
step-ca /home/step/step-ca/config/ca.json \
|
||||
--password-file=<(echo "${STEP_CA_PASSWORD:-stepca-demo-password}")
|
||||
ports:
|
||||
- '9000:9000'
|
||||
healthcheck:
|
||||
test: ['CMD-SHELL', 'step ca health --insecure || exit 1']
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
networks:
|
||||
- certctl-network
|
||||
restart: unless-stopped
|
||||
|
||||
# certctl server (control plane)
|
||||
certctl-server:
|
||||
image: ghcr.io/shankar0123/certctl-server:latest
|
||||
container_name: certctl-server-stepca-haproxy
|
||||
environment:
|
||||
# Database
|
||||
DATABASE_URL: postgres://certctl:${DB_PASSWORD:-certctl-dev-password}@postgres:5432/certctl?sslmode=disable
|
||||
|
||||
# Server settings
|
||||
CERTCTL_SERVER_PORT: 8443
|
||||
CERTCTL_SERVER_HOST: 0.0.0.0
|
||||
|
||||
# Auth (disabled for demo; production should use API keys)
|
||||
CERTCTL_AUTH_TYPE: none
|
||||
|
||||
# CORS (allow agent communication)
|
||||
CERTCTL_CORS_ORIGINS: '*'
|
||||
|
||||
# Key generation mode (agent-side in production, server-side for demo)
|
||||
CERTCTL_KEYGEN_MODE: agent
|
||||
|
||||
# step-ca issuer configuration
|
||||
# step-ca runs on step-ca:9000 in this compose network
|
||||
CERTCTL_STEPCA_URL: https://step-ca:9000
|
||||
CERTCTL_STEPCA_ROOT_CERT_PATH: /etc/certctl/step-ca-root.crt
|
||||
CERTCTL_STEPCA_PROVISIONER: certctl
|
||||
CERTCTL_STEPCA_KEY_PATH: /etc/certctl/step-ca-provisioner.json
|
||||
CERTCTL_STEPCA_PASSWORD: ${STEP_CA_PROVISIONER_PASSWORD:-certctl-provisioner-demo}
|
||||
|
||||
# Logging
|
||||
CERTCTL_LOG_LEVEL: info
|
||||
volumes:
|
||||
# Mount step-ca certs for TLS verification (auto-generated by step-ca init)
|
||||
- step_ca_data:/home/step/step-ca/config:ro
|
||||
ports:
|
||||
- '${SERVER_PORT:-8443}:8443'
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
step-ca:
|
||||
condition: service_healthy
|
||||
networks:
|
||||
- certctl-network
|
||||
healthcheck:
|
||||
test: ['CMD-SHELL', 'curl -sf http://localhost:8443/api/v1/health || exit 1']
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
restart: unless-stopped
|
||||
|
||||
# certctl agent (runs on the target machine with HAProxy)
|
||||
certctl-agent:
|
||||
image: ghcr.io/shankar0123/certctl-agent:latest
|
||||
container_name: certctl-agent-stepca-haproxy
|
||||
environment:
|
||||
# Control plane connection
|
||||
CERTCTL_SERVER_URL: http://certctl-server:8443
|
||||
CERTCTL_API_KEY: ${AGENT_API_KEY:-agent-demo-key}
|
||||
|
||||
# Key generation (agent-side keys, never sent to server)
|
||||
CERTCTL_KEYGEN_MODE: agent
|
||||
CERTCTL_KEY_DIR: /var/lib/certctl/keys
|
||||
|
||||
# Discovery (scan existing certs so operator knows what's already deployed)
|
||||
CERTCTL_DISCOVERY_DIRS: /etc/haproxy/ssl
|
||||
|
||||
# Heartbeat interval
|
||||
CERTCTL_HEARTBEAT_INTERVAL: 30s
|
||||
|
||||
# Agent metadata (self-reported)
|
||||
CERTCTL_AGENT_NAME: haproxy-agent-01
|
||||
|
||||
# Logging
|
||||
CERTCTL_LOG_LEVEL: info
|
||||
volumes:
|
||||
# Mount HAProxy config and cert directories
|
||||
# In production, these would be the actual HAProxy paths
|
||||
- haproxy_certs:/etc/haproxy/ssl
|
||||
- haproxy_conf:/etc/haproxy
|
||||
# Agent key storage (persisted across restarts)
|
||||
- agent_keys:/var/lib/certctl/keys
|
||||
depends_on:
|
||||
certctl-server:
|
||||
condition: service_healthy
|
||||
networks:
|
||||
- certctl-network
|
||||
restart: unless-stopped
|
||||
|
||||
# HAProxy reverse proxy / load balancer
|
||||
# This is where certificates will be deployed
|
||||
haproxy:
|
||||
image: haproxy:2.9-alpine
|
||||
container_name: certctl-haproxy-stepca-haproxy
|
||||
ports:
|
||||
- '80:80'
|
||||
- '443:443'
|
||||
volumes:
|
||||
- haproxy_conf:/etc/haproxy
|
||||
- haproxy_certs:/etc/haproxy/ssl
|
||||
# Default HAProxy config
|
||||
- ./haproxy.cfg:/etc/haproxy/haproxy.cfg:ro
|
||||
depends_on:
|
||||
- certctl-agent
|
||||
networks:
|
||||
- certctl-network
|
||||
healthcheck:
|
||||
test: ['CMD-SHELL', 'wget --quiet --tries=1 --spider http://localhost:8080/stats || exit 1']
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
restart: unless-stopped
|
||||
|
||||
networks:
|
||||
certctl-network:
|
||||
driver: bridge
|
||||
|
||||
volumes:
|
||||
postgres_data:
|
||||
driver: local
|
||||
step_ca_data:
|
||||
driver: local
|
||||
haproxy_certs:
|
||||
driver: local
|
||||
haproxy_conf:
|
||||
driver: local
|
||||
agent_keys:
|
||||
driver: local
|
||||
@@ -0,0 +1,69 @@
|
||||
global
|
||||
log stdout local0
|
||||
log stdout local1 notice
|
||||
chroot /var/lib/haproxy
|
||||
stats socket /run/haproxy/admin.sock mode 660 level admin
|
||||
stats timeout 30s
|
||||
user haproxy
|
||||
group haproxy
|
||||
daemon
|
||||
|
||||
# Default SSL options for modern TLS
|
||||
tune.ssl.default-dh-param 2048
|
||||
ssl-default-bind-ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384
|
||||
ssl-default-bind-options ssl-min-ver TLSv1.2
|
||||
|
||||
defaults
|
||||
mode http
|
||||
log global
|
||||
option httplog
|
||||
option dontlognull
|
||||
timeout connect 5000
|
||||
timeout client 50000
|
||||
timeout server 50000
|
||||
errorfile 400 /etc/haproxy/errors/400.http
|
||||
errorfile 403 /etc/haproxy/errors/403.http
|
||||
errorfile 408 /etc/haproxy/errors/408.http
|
||||
errorfile 500 /etc/haproxy/errors/500.http
|
||||
errorfile 502 /etc/haproxy/errors/502.http
|
||||
errorfile 503 /etc/haproxy/errors/503.http
|
||||
errorfile 504 /etc/haproxy/errors/504.http
|
||||
|
||||
# Statistics endpoint (accessible on port 8080)
|
||||
listen stats
|
||||
bind *:8080
|
||||
stats enable
|
||||
stats uri /stats
|
||||
stats refresh 30s
|
||||
stats admin if TRUE
|
||||
|
||||
# Example HTTPS frontend with certificate from certctl
|
||||
# This frontend will serve HTTPS on port 443 using a combined PEM file
|
||||
# deployed by certctl to /etc/haproxy/ssl/cert.pem
|
||||
frontend https_in
|
||||
# HTTP redirect to HTTPS
|
||||
bind *:80
|
||||
mode http
|
||||
acl is_http hdr(X-Forwarded-Proto) http
|
||||
redirect scheme https code 301 if !is_https
|
||||
|
||||
# HTTPS with certificate
|
||||
# In production, certctl will manage cert.pem and reload HAProxy after deployment
|
||||
bind *:443 ssl crt /etc/haproxy/ssl/cert.pem strict-sni
|
||||
mode http
|
||||
option httplog
|
||||
|
||||
# Default backend
|
||||
default_backend http_backend
|
||||
|
||||
# Example backend (simple web service placeholder)
|
||||
backend http_backend
|
||||
mode http
|
||||
option httpchk GET /
|
||||
server local_app 127.0.0.1:8000 check disabled
|
||||
|
||||
# Health endpoint (useful for certctl agent deployment verification)
|
||||
frontend health
|
||||
bind *:9999
|
||||
mode http
|
||||
monitor-uri /health
|
||||
Reference in New Issue
Block a user