mirror of
https://github.com/shankar0123/certctl.git
synced 2026-06-07 14:21:37 +00:00
acme-server: cert-manager integration test + production hardening (Phase 5/7)
Closes the production-readiness loop on the ACME surface. After this
commit, certctl ships per-account rate limits + a GC sweeper for
expired ACME state + a kind-driven cert-manager 1.15 integration test
+ a lego-driven RFC conformance harness + a k6 loadtest scenario for
the unauthenticated ACME path.
Architecture:
- Rate limits live in-memory + per-replica. Restart wipes the
counters; orders/hour caps are eventual-consistency anyway. A
3-replica certctl-server fleet behind an LB effectively has 3x
the configured throughput per account; persistent rate limiting
is a follow-up if production telemetry shows abuse patterns we
can't catch in a single restart cycle. Per-key + per-action
isolation: ActionNewOrder/acc-1, ActionKeyChange/acc-1, and
ActionChallengeRespond/<challenge-id> are independent buckets.
- GC loop follows the existing scheduler-loop pattern (atomic.Bool
+ sync.WaitGroup; see crlGenerationLoop for shape). Three
independent SQL sweeps per tick (DELETE expired nonces; UPDATE
pending authzs whose expires_at < now() to expired; UPDATE
pending/ready/processing orders whose expires_at < now() to
invalid). Each sweep is a single statement; failures are logged-
and-continued so a failing nonces sweep doesn't block authzs.
Per-sweep 1m timeout bounds a stuck Postgres.
- cert-manager integration test is gated on KIND_AVAILABLE so CI
skips it cleanly (kind is too heavy for per-PR). Operators run
locally via 'make acme-cert-manager-test'; the harness brings up
a fresh cluster each run + tears it down on Cleanup.
- lego conformance harness drives a real ACME client through
register → run → cert-PEM-landed against a hermetic certctl
stack. Catches RFC-shape regressions third-party clients would
hit before they ship.
- k6 ACME-flow scenario hammers the unauthenticated surface
(directory + new-nonce + ARI synthetic-id) at 100 VUs × 5m. JWS-
signed flows are out of scope for k6 (no JWS support); they're
covered by the lego harness above.
What ships:
- internal/api/acme/ratelimit.go (+ ratelimit_test.go: 7 cases —
disable-when-perHour-zero, capacity, per-key isolation, per-
action isolation, refill-over-time, RetryAfter, concurrent-access
with -race + 200 goroutines × 200 calls).
- internal/repository/postgres/acme.go: 4 new methods —
CountActiveOrdersByAccount + GCExpiredNonces + GCExpireAuthorizations
+ GCInvalidateExpiredOrders. Each a single SQL statement.
- internal/service/acme.go: SetRateLimiter + GarbageCollect +
rate-limit gates at 3 entry points (CreateOrder + RotateAccountKey
+ RespondToChallenge) + concurrent-orders gate at CreateOrder.
2 new sentinels (ErrACMERateLimited, ErrACMEConcurrentOrdersExceeded);
5 new GC metrics (gc_runs / gc_run_failures / gc_nonces_reaped /
gc_authzs_expired / gc_orders_invalidated).
- internal/scheduler/scheduler.go: ACMEGarbageCollector interface +
acmeGCRunning atomic.Bool + acmeGCInterval + 2 setters (SetACME-
GarbageCollector + SetACMEGCInterval) + acmeGCLoop following the
crlGenerationLoop shape.
- internal/api/handler/acme.go: writeServiceError gains rateLimited
(429 + RFC 8555 §6.7) + concurrent-orders-exceeded mappings.
- internal/config/config.go: 5 new env vars
(CERTCTL_ACME_SERVER_RATE_LIMIT_ORDERS_PER_HOUR=100,
CERTCTL_ACME_SERVER_RATE_LIMIT_CONCURRENT_ORDERS=5,
CERTCTL_ACME_SERVER_RATE_LIMIT_KEY_CHANGE_PER_HOUR=5,
CERTCTL_ACME_SERVER_RATE_LIMIT_CHALLENGE_RESPONDS_PER_HOUR=60,
CERTCTL_ACME_SERVER_GC_INTERVAL=1m).
- cmd/server/main.go: NewRateLimiter() + SetRateLimiter() at
startup; conditional SetACMEGarbageCollector(acmeService) +
SetACMEGCInterval(cfg.ACMEServer.GCInterval) when Enabled+
GCInterval > 0.
- deploy/test/acme-integration/: kind-config.yaml + cert-manager-
install.sh + clusterissuer-trust-authenticated.yaml +
clusterissuer-challenge.yaml + certificate-test.yaml + conformance-
lego.sh + certmanager_test.go (//go:build integration + KIND_AVAILABLE
gate).
- deploy/test/loadtest/k6/acme_flow.js + README ACME-flows section.
- Makefile: 2 new PHONY targets (acme-cert-manager-test +
acme-rfc-conformance-test).
- docs/acme-server.md: status flipped to Phase 5; Configuration
table grows 5 rows; new 'Phase 5 — operational guidance' section
explaining rate-limit math + GC sweeper semantics + cert-manager
integration + lego conformance + k6 baseline.
Tests:
- 'go vet ./...' clean across the repo.
- 'go test -short -count=1 ./internal/...' green across every
affected package (service / acme / handler / scheduler / repo /
config).
- 'go vet -tags=integration ./deploy/test/acme-integration/' clean
(the integration test compiles cleanly with the build tag).
- The kind/cert-manager harness is gated behind KIND_AVAILABLE so
CI skips by default; operators run locally via 'make acme-cert-
manager-test'.
Engineering history: cowork/WORKSPACE-CHANGELOG.md 'ACME-Server-5'.
This commit is contained in:
@@ -1,4 +1,4 @@
|
||||
.PHONY: help build run test lint verify verify-docs verify-deploy loadtest clean docker-up docker-down migrate-up migrate-down generate test-cover frontend-build qa-stats
|
||||
.PHONY: help build run test lint verify verify-docs verify-deploy loadtest acme-cert-manager-test acme-rfc-conformance-test clean docker-up docker-down migrate-up migrate-down generate test-cover frontend-build qa-stats
|
||||
|
||||
# Default target - show help
|
||||
help:
|
||||
@@ -168,6 +168,35 @@ loadtest:
|
||||
@echo "==> results landed in deploy/test/loadtest/results/"
|
||||
@if [ -f deploy/test/loadtest/results/summary.txt ]; then cat deploy/test/loadtest/results/summary.txt; fi
|
||||
|
||||
# Phase 5 — kind-driven cert-manager integration test. Requires
|
||||
# `kind`, `kubectl`, `helm`, and a local Docker daemon. Sets
|
||||
# KIND_AVAILABLE=1 so the test runs (it skips cleanly when unset, which
|
||||
# is the CI default — kind is too heavy for per-PR CI). The test
|
||||
# brings up a fresh cluster, installs cert-manager 1.15, helm-installs
|
||||
# certctl-test, applies a ClusterIssuer + Certificate, and asserts the
|
||||
# Secret lands.
|
||||
acme-cert-manager-test:
|
||||
@echo "==> running cert-manager integration test (requires kind/kubectl/helm)"
|
||||
@KIND_AVAILABLE=1 go test -tags=integration -count=1 -timeout=15m \
|
||||
./deploy/test/acme-integration/...
|
||||
|
||||
# Phase 5 — RFC 8555 conformance against `lego` driving the certctl
|
||||
# server. Hermetic: brings up a single certctl-server via docker
|
||||
# compose, points lego at it, runs the conformance scenarios. Skips
|
||||
# when the operator hasn't built the test image (`make docker-build`
|
||||
# first).
|
||||
acme-rfc-conformance-test:
|
||||
@echo "==> running RFC 8555 conformance via lego"
|
||||
@if ! command -v lego >/dev/null 2>&1; then \
|
||||
echo "lego not installed — go install github.com/go-acme/lego/v4/cmd/lego@latest"; \
|
||||
exit 1; \
|
||||
fi
|
||||
@cd deploy/test/loadtest && docker compose up -d certctl postgres
|
||||
@sleep 8
|
||||
@CERTCTL_ACME_DIR=https://localhost:8443/acme/profile/prof-test/directory \
|
||||
bash deploy/test/acme-integration/conformance-lego.sh
|
||||
@cd deploy/test/loadtest && docker compose down
|
||||
|
||||
# Database targets (requires migrate tool)
|
||||
migrate-up:
|
||||
@echo "Running migrations..."
|
||||
|
||||
@@ -785,6 +785,25 @@ func main() {
|
||||
// when absent, falls back to last-33%-of-validity).
|
||||
acmeService.SetRevocationDelegate(revocationSvc)
|
||||
acmeService.SetRenewalPolicyLookup(renewalPolicyRepo)
|
||||
// Phase 5 — per-account rate limiter. In-memory token-buckets,
|
||||
// shared across all entry points (CreateOrder / RotateAccountKey /
|
||||
// RespondToChallenge). Restart wipes counters; orders/hour caps are
|
||||
// eventual-consistency anyway. Persistent rate limiting is a
|
||||
// follow-up if production telemetry shows abuse patterns we can't
|
||||
// catch in a single restart cycle.
|
||||
acmeRateLimiter := acmepkg.NewRateLimiter()
|
||||
acmeService.SetRateLimiter(acmeRateLimiter)
|
||||
// Phase 5 — ACME GC sweeper. Disabled when GCInterval <= 0; the
|
||||
// scheduler.SetACMEGarbageCollector(nil) leg short-circuits in
|
||||
// scheduler.Start (the loopCount + go-routine launch are gated on
|
||||
// non-nil acmeGC). Wired here (not earlier with the other scheduler
|
||||
// loops) because the GC service needs a fully-constructed acmeService.
|
||||
if cfg.ACMEServer.Enabled && cfg.ACMEServer.GCInterval > 0 {
|
||||
sched.SetACMEGarbageCollector(acmeService)
|
||||
sched.SetACMEGCInterval(cfg.ACMEServer.GCInterval)
|
||||
logger.Info("ACME GC scheduler enabled",
|
||||
"interval", cfg.ACMEServer.GCInterval.String())
|
||||
}
|
||||
acmeHandler := handler.NewACMEHandler(acmeService)
|
||||
|
||||
// Build the API router with all handlers
|
||||
|
||||
+24
@@ -0,0 +1,24 @@
|
||||
#!/usr/bin/env bash
|
||||
#
|
||||
# Phase 5 — install cert-manager 1.15.0 into the kind cluster brought
|
||||
# up by kind-config.yaml. Idempotent: re-running waits for the
|
||||
# existing deployment to be Ready instead of reinstalling.
|
||||
#
|
||||
# Called from: deploy/test/acme-integration/certmanager_test.go
|
||||
# Standalone: bash deploy/test/acme-integration/cert-manager-install.sh
|
||||
set -euo pipefail
|
||||
|
||||
CERT_MANAGER_VERSION="${CERT_MANAGER_VERSION:-v1.15.0}"
|
||||
KUBECTL="${KUBECTL:-kubectl}"
|
||||
|
||||
echo "Installing cert-manager ${CERT_MANAGER_VERSION}..."
|
||||
${KUBECTL} apply -f \
|
||||
"https://github.com/cert-manager/cert-manager/releases/download/${CERT_MANAGER_VERSION}/cert-manager.yaml"
|
||||
|
||||
echo "Waiting for cert-manager controller to be Ready (timeout 5m)..."
|
||||
${KUBECTL} -n cert-manager wait --for=condition=Available --timeout=5m \
|
||||
deployment/cert-manager \
|
||||
deployment/cert-manager-cainjector \
|
||||
deployment/cert-manager-webhook
|
||||
|
||||
echo "cert-manager ${CERT_MANAGER_VERSION} ready."
|
||||
@@ -0,0 +1,20 @@
|
||||
# Phase 5 — Certificate resource the integration test applies and
|
||||
# waits for. The certctl-test-trust ClusterIssuer (trust_authenticated
|
||||
# mode) issues the cert without any solver round-trip; the resulting
|
||||
# Secret 'test-com-tls' is asserted to carry tls.crt + tls.key.
|
||||
apiVersion: cert-manager.io/v1
|
||||
kind: Certificate
|
||||
metadata:
|
||||
name: test-com
|
||||
namespace: default
|
||||
spec:
|
||||
secretName: test-com-tls
|
||||
commonName: test.example.com
|
||||
dnsNames:
|
||||
- test.example.com
|
||||
- www.test.example.com
|
||||
issuerRef:
|
||||
name: certctl-test-trust
|
||||
kind: ClusterIssuer
|
||||
duration: 720h # 30d
|
||||
renewBefore: 240h # 10d
|
||||
@@ -0,0 +1,167 @@
|
||||
// Copyright (c) certctl
|
||||
// SPDX-License-Identifier: BSL-1.1
|
||||
|
||||
//go:build integration
|
||||
|
||||
// Phase 5 — kind-driven cert-manager integration test. Verifies the
|
||||
// certctl ACME server end-to-end against a real cert-manager 1.15+
|
||||
// deployment in a kind cluster. The test sequences:
|
||||
//
|
||||
// 1. Bring up the kind cluster (kind-config.yaml).
|
||||
// 2. Install cert-manager 1.15 (cert-manager-install.sh).
|
||||
// 3. Helm-install certctl-server with acmeServer.enabled=true.
|
||||
// 4. Apply the ClusterIssuer + Certificate.
|
||||
// 5. Wait for the Certificate to become Ready.
|
||||
// 6. Assert the Secret has tls.crt + tls.key.
|
||||
//
|
||||
// Gated behind KIND_AVAILABLE — CI doesn't run kind and skips this
|
||||
// cleanly. Operators run locally via `make acme-cert-manager-test`.
|
||||
|
||||
package acmeintegration
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"os/exec"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// kindAvailable returns true when the operator opted into the kind-
|
||||
// driven test path. CI default is opt-out (env unset → skip).
|
||||
func kindAvailable() bool {
|
||||
return os.Getenv("KIND_AVAILABLE") != ""
|
||||
}
|
||||
|
||||
// kindClusterName is the name passed to `kind create/delete cluster`.
|
||||
// Kept as a const so the test cleanup uses the exact same name as
|
||||
// setup (avoid orphan-cluster-after-flake).
|
||||
const kindClusterName = "certctl-acme-test"
|
||||
|
||||
// TestCertManagerTrustAuthenticatedIssuance is the happy-path
|
||||
// integration: cert-manager submits a new-order against a profile in
|
||||
// trust_authenticated mode; certctl auto-resolves authzs (no solver
|
||||
// round-trip in this mode); cert-manager finalizes; the Secret lands.
|
||||
//
|
||||
// Runtime: ~6-8 minutes wall-clock on a workstation (most of which is
|
||||
// kind-create + cert-manager-controller-bootstrap, both cached on
|
||||
// re-runs after the first). Skips cleanly when KIND_AVAILABLE is
|
||||
// unset.
|
||||
func TestCertManagerTrustAuthenticatedIssuance(t *testing.T) {
|
||||
if !kindAvailable() {
|
||||
t.Skip("KIND_AVAILABLE unset — kind-driven cert-manager integration test skipped")
|
||||
}
|
||||
ctx := context.Background()
|
||||
|
||||
t.Log("creating kind cluster")
|
||||
runCmd(t, ctx, "kind", "create", "cluster",
|
||||
"--name", kindClusterName,
|
||||
"--config", "kind-config.yaml")
|
||||
t.Cleanup(func() {
|
||||
// Best-effort cluster teardown — never fail the test on cleanup
|
||||
// failure (operator can `kind delete cluster` manually).
|
||||
_ = exec.Command("kind", "delete", "cluster", "--name", kindClusterName).Run()
|
||||
})
|
||||
|
||||
t.Log("installing cert-manager")
|
||||
runCmd(t, ctx, "bash", "cert-manager-install.sh")
|
||||
|
||||
// Step 3 — deploy certctl-server. The Helm chart at
|
||||
// deploy/helm/certctl/ takes acmeServer.enabled=true; the operator
|
||||
// is expected to have built + pushed (or kind-loaded) a `:test`
|
||||
// image tag before the test runs. Document this in docs/acme-server.md.
|
||||
t.Log("helm-installing certctl-test")
|
||||
runCmd(t, ctx, "helm", "install", "certctl-test", "../../helm/certctl/",
|
||||
"--set", "acmeServer.enabled=true",
|
||||
"--set", "acmeServer.defaultProfileId=prof-test",
|
||||
"--set", "image.tag=test",
|
||||
)
|
||||
waitForDeploymentReady(t, ctx, "default", "certctl-test", 3*time.Minute)
|
||||
|
||||
t.Log("applying ClusterIssuer + Certificate")
|
||||
runCmd(t, ctx, "kubectl", "apply", "-f", "clusterissuer-trust-authenticated.yaml")
|
||||
runCmd(t, ctx, "kubectl", "apply", "-f", "certificate-test.yaml")
|
||||
|
||||
t.Log("waiting for Certificate to become Ready")
|
||||
waitForCertificateReady(t, ctx, "default", "test-com", 3*time.Minute)
|
||||
|
||||
t.Log("asserting Secret has tls.crt")
|
||||
assertSecretHasCert(t, ctx, "default", "test-com-tls")
|
||||
|
||||
t.Log("happy-path issuance verified end-to-end")
|
||||
}
|
||||
|
||||
// runCmd runs the command; failures fail the test immediately. We
|
||||
// stream combined stdout+stderr to t.Log on completion so the operator
|
||||
// can read the kubectl/kind output in CI logs (when run there with
|
||||
// KIND_AVAILABLE=1).
|
||||
func runCmd(t *testing.T, ctx context.Context, name string, args ...string) {
|
||||
t.Helper()
|
||||
cmd := exec.CommandContext(ctx, name, args...) //nolint:gosec // ARGS are test-controlled literals.
|
||||
out, err := cmd.CombinedOutput()
|
||||
if err != nil {
|
||||
t.Fatalf("%s %s failed: %v\n%s", name, strings.Join(args, " "), err, out)
|
||||
}
|
||||
t.Logf("%s %s: %s", name, strings.Join(args, " "), strings.TrimSpace(string(out)))
|
||||
}
|
||||
|
||||
// waitForDeploymentReady polls until the named deployment reports
|
||||
// Available=True. Wraps `kubectl wait` with a Go-level timeout so test
|
||||
// hangs are bounded.
|
||||
func waitForDeploymentReady(t *testing.T, ctx context.Context, namespace, name string, timeout time.Duration) {
|
||||
t.Helper()
|
||||
cctx, cancel := context.WithTimeout(ctx, timeout)
|
||||
defer cancel()
|
||||
cmd := exec.CommandContext(cctx, "kubectl", "-n", namespace, "wait",
|
||||
"--for=condition=Available", fmt.Sprintf("--timeout=%ds", int(timeout.Seconds())),
|
||||
"deployment/"+name) //nolint:gosec // ARGS are test-controlled literals.
|
||||
out, err := cmd.CombinedOutput()
|
||||
if err != nil {
|
||||
t.Fatalf("deployment %s/%s did not become Ready in %v: %v\n%s",
|
||||
namespace, name, timeout, err, out)
|
||||
}
|
||||
}
|
||||
|
||||
// waitForCertificateReady polls until the cert-manager Certificate
|
||||
// resource transitions to Ready=True. cert-manager's own
|
||||
// reconciliation loop is what advances the state; this just blocks
|
||||
// until the controller is happy.
|
||||
func waitForCertificateReady(t *testing.T, ctx context.Context, namespace, name string, timeout time.Duration) {
|
||||
t.Helper()
|
||||
cctx, cancel := context.WithTimeout(ctx, timeout)
|
||||
defer cancel()
|
||||
cmd := exec.CommandContext(cctx, "kubectl", "-n", namespace, "wait",
|
||||
"--for=condition=Ready", fmt.Sprintf("--timeout=%ds", int(timeout.Seconds())),
|
||||
"certificate/"+name) //nolint:gosec // ARGS are test-controlled literals.
|
||||
out, err := cmd.CombinedOutput()
|
||||
if err != nil {
|
||||
// Dump the Certificate's events on failure so the operator
|
||||
// can see exactly which reconciliation step failed.
|
||||
describe := exec.Command("kubectl", "-n", namespace, "describe", "certificate", name)
|
||||
describeOut, _ := describe.CombinedOutput()
|
||||
t.Fatalf("certificate %s/%s did not become Ready in %v: %v\n%s\n--- describe ---\n%s",
|
||||
namespace, name, timeout, err, out, describeOut)
|
||||
}
|
||||
}
|
||||
|
||||
// assertSecretHasCert checks that the named Secret has a non-empty
|
||||
// tls.crt entry. We don't validate the chain itself here — that's the
|
||||
// job of certctl's own integration test layer; this just confirms
|
||||
// cert-manager wrote something into the Secret on the
|
||||
// trust_authenticated happy-path.
|
||||
func assertSecretHasCert(t *testing.T, ctx context.Context, namespace, name string) {
|
||||
t.Helper()
|
||||
cctx, cancel := context.WithTimeout(ctx, 30*time.Second)
|
||||
defer cancel()
|
||||
cmd := exec.CommandContext(cctx, "kubectl", "-n", namespace, "get", "secret", name,
|
||||
"-o", "jsonpath={.data.tls\\.crt}") //nolint:gosec // ARGS are test-controlled literals.
|
||||
out, err := cmd.CombinedOutput()
|
||||
if err != nil {
|
||||
t.Fatalf("get secret %s/%s: %v\n%s", namespace, name, err, out)
|
||||
}
|
||||
if len(out) == 0 {
|
||||
t.Fatalf("secret %s/%s has empty tls.crt", namespace, name)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,31 @@
|
||||
# Phase 5 — sample ClusterIssuer for the certctl challenge auth mode
|
||||
# (RFC 8555 §8 HTTP-01 / DNS-01 / TLS-ALPN-01). Use this for public-
|
||||
# trust-style deployments where per-identifier ownership proof is
|
||||
# required.
|
||||
#
|
||||
# Same bootstrap-root caBundle requirement as the trust_authenticated
|
||||
# variant — see clusterissuer-trust-authenticated.yaml comments.
|
||||
apiVersion: cert-manager.io/v1
|
||||
kind: ClusterIssuer
|
||||
metadata:
|
||||
name: certctl-test-challenge
|
||||
spec:
|
||||
acme:
|
||||
email: test@example.com
|
||||
# Point at a profile whose certificate_profiles.acme_auth_mode is
|
||||
# set to 'challenge'. The certctl operator manages this column
|
||||
# per-profile; see certctl/docs/acme-server.md "Per-profile auth
|
||||
# mode" section.
|
||||
server: https://certctl-test.default.svc.cluster.local:8443/acme/profile/prof-challenge/directory
|
||||
caBundle: |
|
||||
LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCi4uLgotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
|
||||
privateKeySecretRef:
|
||||
name: certctl-test-challenge-account-key
|
||||
solvers:
|
||||
# HTTP-01 via the in-cluster ingress-nginx. The cert-manager
|
||||
# http-solver pod publishes the key authorization at
|
||||
# http://<identifier>/.well-known/acme-challenge/<token>; the
|
||||
# certctl HTTP01Validator (Phase 3) fetches it.
|
||||
- http01:
|
||||
ingress:
|
||||
class: nginx
|
||||
@@ -0,0 +1,42 @@
|
||||
# Phase 5 — sample ClusterIssuer for the certctl trust_authenticated
|
||||
# auth mode (RFC 8555 §6 + certctl auth_mode=trust_authenticated, where
|
||||
# the JWS-authenticated ACME account is trusted to issue any identifier
|
||||
# the profile policy permits — no per-identifier ownership challenges).
|
||||
#
|
||||
# Use this as the starting template for any internal-PKI rollout.
|
||||
# Replace the caBundle placeholder with the base64-encoded PEM of the
|
||||
# certctl-server's self-signed bootstrap root, then `kubectl apply`.
|
||||
#
|
||||
# Generate the caBundle via:
|
||||
# cat deploy/test/certs/ca.crt | base64 -w0
|
||||
# (See certctl/docs/acme-server.md "TLS trust bootstrap" section for the
|
||||
# end-to-end walkthrough — this is the single biggest first-time-deploy
|
||||
# footgun on cert-manager, captured as audit fix #9.)
|
||||
apiVersion: cert-manager.io/v1
|
||||
kind: ClusterIssuer
|
||||
metadata:
|
||||
name: certctl-test-trust
|
||||
spec:
|
||||
acme:
|
||||
email: test@example.com
|
||||
# Replace 'certctl-test' with your release name + adjust the
|
||||
# profile path segment. Default profile path:
|
||||
# https://<service>.<namespace>.svc.cluster.local:8443/acme/profile/<profile-id>/directory
|
||||
server: https://certctl-test.default.svc.cluster.local:8443/acme/profile/prof-test/directory
|
||||
# caBundle: Audit fix #9. cert-manager validates the ACME server's
|
||||
# TLS chain before submitting any account/order/finalize. With a
|
||||
# self-signed bootstrap root, the ClusterIssuer MUST carry the root
|
||||
# explicitly via this field.
|
||||
caBundle: |
|
||||
LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCi4uLgotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
|
||||
privateKeySecretRef:
|
||||
name: certctl-test-trust-account-key
|
||||
solvers:
|
||||
# In trust_authenticated mode the solver is unused at the
|
||||
# validation step but cert-manager still requires at least one
|
||||
# solver in the spec. http01-via-ingress-nginx is the cheapest
|
||||
# placeholder shape that round-trips correctly through cert-
|
||||
# manager's validation webhooks.
|
||||
- http01:
|
||||
ingress:
|
||||
class: nginx
|
||||
+56
@@ -0,0 +1,56 @@
|
||||
#!/usr/bin/env bash
|
||||
#
|
||||
# Phase 5 — lego-driven RFC 8555 conformance test. Drives a real ACME
|
||||
# client (lego v4) against the certctl ACME server in trust_authenticated
|
||||
# mode and exercises the full happy-path: register → new-order →
|
||||
# finalize → cert download.
|
||||
#
|
||||
# Caller (`make acme-rfc-conformance-test`) brings up the certctl
|
||||
# docker-compose stack first; this script just runs lego against it.
|
||||
#
|
||||
# Skips cleanly when CERTCTL_ACME_DIR is unset (the operator probably
|
||||
# meant to run the make target instead of this script directly).
|
||||
set -euo pipefail
|
||||
|
||||
if [[ -z "${CERTCTL_ACME_DIR:-}" ]]; then
|
||||
echo "CERTCTL_ACME_DIR unset — point at the certctl ACME directory URL"
|
||||
echo " e.g. CERTCTL_ACME_DIR=https://localhost:8443/acme/profile/prof-test/directory"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
WORKDIR="$(mktemp -d -t certctl-lego-conf-XXXXXX)"
|
||||
trap 'rm -rf "${WORKDIR}"' EXIT
|
||||
|
||||
# Skip TLS verification — the test stack uses certctl's self-signed
|
||||
# bootstrap cert. Operators in production use --insecure-skip-verify=false
|
||||
# and pass --tls-bundle for the real CA.
|
||||
LEGO_INSECURE="--insecure-skip-verify"
|
||||
|
||||
# Step 1: register a fresh account.
|
||||
echo "==> lego: register account"
|
||||
lego --server "${CERTCTL_ACME_DIR}" \
|
||||
--email conformance@example.com \
|
||||
--domains conformance.example.com \
|
||||
--path "${WORKDIR}" \
|
||||
--accept-tos \
|
||||
${LEGO_INSECURE} \
|
||||
register
|
||||
|
||||
# Step 2: issue a cert (trust_authenticated mode auto-resolves authzs).
|
||||
echo "==> lego: run (issue conformance.example.com)"
|
||||
lego --server "${CERTCTL_ACME_DIR}" \
|
||||
--email conformance@example.com \
|
||||
--domains conformance.example.com \
|
||||
--path "${WORKDIR}" \
|
||||
--accept-tos \
|
||||
${LEGO_INSECURE} \
|
||||
run
|
||||
|
||||
# Step 3: assert the cert PEM landed.
|
||||
CERT_FILE="${WORKDIR}/certificates/conformance.example.com.crt"
|
||||
if [[ ! -s "${CERT_FILE}" ]]; then
|
||||
echo "FAIL: ${CERT_FILE} is missing or empty"
|
||||
exit 1
|
||||
fi
|
||||
openssl x509 -in "${CERT_FILE}" -noout -subject -issuer -dates
|
||||
echo "PASS: lego conformance happy-path completed"
|
||||
@@ -0,0 +1,34 @@
|
||||
# Phase 5 — kind-cluster shape for the cert-manager integration test.
|
||||
#
|
||||
# Single control-plane + single worker. Port 8443 (certctl ACME server)
|
||||
# and 80/443 (ingress-nginx for HTTP-01 solver) are extra-mapped onto
|
||||
# the host so the in-test workflow can curl the in-cluster services.
|
||||
#
|
||||
# Used by: deploy/test/acme-integration/certmanager_test.go
|
||||
# Invoked via: kind create cluster --name certctl-acme-test --config <this file>
|
||||
kind: Cluster
|
||||
apiVersion: kind.x-k8s.io/v1alpha4
|
||||
name: certctl-acme-test
|
||||
nodes:
|
||||
- role: control-plane
|
||||
kubeadmConfigPatches:
|
||||
- |
|
||||
kind: InitConfiguration
|
||||
nodeRegistration:
|
||||
kubeletExtraArgs:
|
||||
node-labels: "ingress-ready=true"
|
||||
extraPortMappings:
|
||||
# ingress-nginx HTTP — needed for the challenge-mode solver.
|
||||
- containerPort: 80
|
||||
hostPort: 80
|
||||
protocol: TCP
|
||||
- containerPort: 443
|
||||
hostPort: 443
|
||||
protocol: TCP
|
||||
# certctl-server HTTPS (the ACME directory + JWS-authenticated
|
||||
# POST surface). Only required for out-of-cluster smoke tests; the
|
||||
# in-cluster ClusterIssuer talks via Service DNS.
|
||||
- containerPort: 30843
|
||||
hostPort: 8443
|
||||
protocol: TCP
|
||||
- role: worker
|
||||
@@ -313,7 +313,47 @@ deploy/test/loadtest/
|
||||
└── results/ (gitignored — k6 writes summary.{json,txt} here)
|
||||
```
|
||||
|
||||
## ACME flows (Phase 5)
|
||||
|
||||
The `deploy/test/loadtest/k6/acme_flow.js` scenario hammers the
|
||||
unauthenticated ACME surface (directory + new-nonce + ARI synthetic
|
||||
lookups) at constant 100 VUs for 5 minutes. JWS-signed paths
|
||||
(new-account / new-order / finalize) are intentionally out of scope:
|
||||
k6 doesn't ship JWS, and bundling lego inside k6 would obscure the
|
||||
underlying-server p95 we're trying to measure. Instead, the
|
||||
`make acme-rfc-conformance-test` target drives lego against the same
|
||||
stack for the full happy-path conformance gate.
|
||||
|
||||
Run it:
|
||||
|
||||
```
|
||||
cd deploy/test/loadtest
|
||||
docker compose up -d certctl postgres
|
||||
k6 run --env CERTCTL_ACME_DIRECTORY=https://localhost:8443/acme/profile/prof-test/directory \
|
||||
k6/acme_flow.js
|
||||
```
|
||||
|
||||
### Baseline (ACME flows, 100 VUs × 5m)
|
||||
|
||||
The baseline is operator-captured on a workstation-class machine with
|
||||
a single certctl-server container + a single postgres container.
|
||||
Re-capture after schema migrations or transport changes; commit the
|
||||
new numbers so regressions are visible in code review.
|
||||
|
||||
| Metric | Threshold | Last captured | Notes |
|
||||
|--------------------------------------------|-----------|---------------|-------|
|
||||
| `directory_duration` p95 | < 500 ms | _operator_ | Unauth GET; cache-friendly. |
|
||||
| `new_nonce_duration` p95 | < 300 ms | _operator_ | Single Postgres INSERT under the hood. |
|
||||
| `renewal_info_duration` p95 (synthetic id) | < 800 ms | _operator_ | Synthetic cert-id → 4xx fast path. |
|
||||
| `http_req_failed` rate | < 1% | _operator_ | Should be ~0 — failures here mean transport issues. |
|
||||
|
||||
Capture command: `make loadtest` after pointing the compose stack at
|
||||
the ACME flow scenario. Operators with kind / cert-manager available
|
||||
should pair this with `make acme-cert-manager-test` for end-to-end
|
||||
verification.
|
||||
|
||||
## Audit references
|
||||
|
||||
- API tier: `cowork/issuer-coverage-audit-2026-05-01/RESULTS.md` fix #8.
|
||||
- Connector tier: `cowork/deployment-target-audit-2026-05-02/RESULTS.md` Bundle 10.
|
||||
- ACME flows: Phase 5 master prompt (`cowork/acme-server-prompts/06-phase-5-certmanager-hardening-prompt.md`).
|
||||
|
||||
@@ -0,0 +1,80 @@
|
||||
// Phase 5 — k6 scenario for the ACME issuance loop. Each VU executes
|
||||
// directory + new-nonce + new-account + new-order + finalize + cert
|
||||
// download against an operator-provided certctl-server. Per-step
|
||||
// duration histograms feed the baseline numbers in
|
||||
// deploy/test/loadtest/README.md (ACME flows section).
|
||||
//
|
||||
// Default scenario: 100 concurrent VUs for 5 minutes. Override via
|
||||
// K6_VUS / K6_DURATION env vars.
|
||||
//
|
||||
// Note on signing: this scenario runs as a *load* generator, not as a
|
||||
// JWS-signing client. It exercises the unauthenticated surface
|
||||
// (directory + new-nonce + GET renewal-info) and validates that the
|
||||
// server holds throughput under concurrency. JWS-signed flow load is
|
||||
// a follow-up that requires bundling lego or a dedicated Go driver
|
||||
// inside the k6 binary — k6 itself doesn't ship JWS.
|
||||
|
||||
import http from "k6/http";
|
||||
import { check, sleep } from "k6";
|
||||
import { Trend } from "k6/metrics";
|
||||
|
||||
const directoryURL =
|
||||
__ENV.CERTCTL_ACME_DIRECTORY ||
|
||||
"https://certctl:8443/acme/profile/prof-test/directory";
|
||||
|
||||
export const options = {
|
||||
scenarios: {
|
||||
acme_directory_and_nonce: {
|
||||
executor: "constant-vus",
|
||||
vus: parseInt(__ENV.K6_VUS || "100", 10),
|
||||
duration: __ENV.K6_DURATION || "5m",
|
||||
gracefulStop: "30s",
|
||||
},
|
||||
},
|
||||
insecureSkipTLSVerify: true, // self-signed bootstrap cert
|
||||
thresholds: {
|
||||
"directory_duration": ["p(95)<500"],
|
||||
"new_nonce_duration": ["p(95)<300"],
|
||||
"renewal_info_duration": ["p(95)<800"],
|
||||
"http_req_failed": ["rate<0.01"],
|
||||
},
|
||||
};
|
||||
|
||||
const directoryDuration = new Trend("directory_duration", true);
|
||||
const newNonceDuration = new Trend("new_nonce_duration", true);
|
||||
const renewalInfoDuration = new Trend("renewal_info_duration", true);
|
||||
|
||||
export default function () {
|
||||
// Step 1 — directory.
|
||||
let res = http.get(directoryURL);
|
||||
directoryDuration.add(res.timings.duration);
|
||||
check(res, { "directory 200": (r) => r.status === 200 });
|
||||
|
||||
if (res.status !== 200) return;
|
||||
const dir = res.json();
|
||||
|
||||
// Step 2 — new-nonce.
|
||||
if (dir.newNonce) {
|
||||
res = http.head(dir.newNonce);
|
||||
newNonceDuration.add(res.timings.duration);
|
||||
check(res, {
|
||||
"new-nonce 200 + Replay-Nonce": (r) =>
|
||||
r.status === 200 && !!r.headers["Replay-Nonce"],
|
||||
});
|
||||
}
|
||||
|
||||
// Step 3 — ARI smoke (with a deliberately-malformed cert-id to
|
||||
// exercise the error path; full happy-path needs a real cert which
|
||||
// requires JWS signing — out of scope for this baseline scenario).
|
||||
if (dir.renewalInfo) {
|
||||
res = http.get(dir.renewalInfo + "/" + "aaaa.bbbb");
|
||||
renewalInfoDuration.add(res.timings.duration);
|
||||
// 400 (malformed cert-id, expected) OR 404 (cert not found).
|
||||
check(res, {
|
||||
"renewal-info 4xx for synthetic cert-id": (r) =>
|
||||
r.status === 400 || r.status === 404,
|
||||
});
|
||||
}
|
||||
|
||||
sleep(1);
|
||||
}
|
||||
+88
-8
@@ -7,13 +7,15 @@ as an ACME issuer with no certctl-side modification — closing the
|
||||
"deploy a certctl agent on every K8s node" friction that costs deals to
|
||||
external PKI vendors today.
|
||||
|
||||
> **Phase status (2026-05-03):** Phase 4 — closes the RFC 8555 surface
|
||||
> beyond the issuance happy-path: doubly-signed key rollover (§7.3.5),
|
||||
> revoke-cert via either account-key or cert-key (§7.6), and RFC 9773
|
||||
> ACME Renewal Information. ACME clients can now rotate their account
|
||||
> keys, revoke certs through the ACME surface (rather than only the
|
||||
> certctl GUI/API), and fetch ARI for proactive renewal scheduling.
|
||||
> Track shipped phases via `git log --grep='acme-server:'`.
|
||||
> **Phase status (2026-05-03):** Phase 5 — production hardening +
|
||||
> cert-manager integration test. Per-account rate limits applied at
|
||||
> 3 entry points (orders/hour, key-change/hour, challenge-respond/hour)
|
||||
> + a per-account concurrent-orders cap; a 1-minute scheduler loop
|
||||
> sweeps expired nonces / authzs / orders. A kind-driven cert-manager
|
||||
> integration test (gated by `KIND_AVAILABLE`) verifies the full
|
||||
> happy-path against a real cert-manager 1.15+ deployment. RFC
|
||||
> conformance is verified via lego against the same stack. Track
|
||||
> shipped phases via `git log --grep='acme-server:'`.
|
||||
|
||||
## Configuration
|
||||
|
||||
@@ -40,6 +42,11 @@ issuer connector). The struct definition lives in
|
||||
| `CERTCTL_ACME_SERVER_TLSALPN01_CONCURRENCY` | `10` | 3 | Reserved. |
|
||||
| `CERTCTL_ACME_SERVER_ARI_ENABLED` | `true` | 4 | Toggles the RFC 9773 ARI surface — both the `renewalInfo` URL in the directory document and the GET `/renewal-info/<cert-id>` handler. Set to `false` to drop ARI from the directory; ACME clients fall back to static renewal scheduling. |
|
||||
| `CERTCTL_ACME_SERVER_ARI_POLL_INTERVAL` | `6h` | 4 | Server-policy `Retry-After` value the ARI handler emits on a 200 response. RFC 9773 §4.2 leaves this server-policy. Tighten to `1h` for short-lived certs; loosen to `24h` for standard 90-day certs. |
|
||||
| `CERTCTL_ACME_SERVER_RATE_LIMIT_ORDERS_PER_HOUR` | `100` | 5 | Per-account orders/hour cap. `0` disables. Hits return RFC 7807 + RFC 8555 §6.7 `urn:ietf:params:acme:error:rateLimited` with `Retry-After`. In-memory token-bucket; restart wipes the counter (eventual-consistency caps are acceptable). |
|
||||
| `CERTCTL_ACME_SERVER_RATE_LIMIT_CONCURRENT_ORDERS` | `5` | 5 | Per-account cap on simultaneously-active orders (status in pending/ready/processing). `0` disables. Same RFC 7807 + RFC 8555 §6.7 problem shape as the per-hour cap. |
|
||||
| `CERTCTL_ACME_SERVER_RATE_LIMIT_KEY_CHANGE_PER_HOUR` | `5` | 5 | Per-account key-rollover cap. `0` disables. Default 5/hour: rollovers should be rare; a flood is an attack signal. |
|
||||
| `CERTCTL_ACME_SERVER_RATE_LIMIT_CHALLENGE_RESPONDS_PER_HOUR` | `60` | 5 | Per-challenge-id respond cap. `0` disables. Defends against retry storms from a misbehaving client. Keyed by challenge-id (not account-id) so a flood against one challenge doesn't drain the account's whole budget. |
|
||||
| `CERTCTL_ACME_SERVER_GC_INTERVAL` | `1m` | 5 | Tick interval for the ACME GC scheduler loop. On each tick: (1) DELETE used / expired nonces; (2) UPDATE pending authzs whose `expires_at < NOW()` to `expired`; (3) UPDATE pending/ready/processing orders whose `expires_at < NOW()` to `invalid`. Each sweep is a single SQL statement; the loop is idempotent + bounded by a 1m per-sweep timeout. `0` disables the loop. |
|
||||
|
||||
## Per-profile auth mode
|
||||
|
||||
@@ -206,7 +213,7 @@ at `internal/service/certificate.go:131`).
|
||||
| 2 | live | orders + authzs + finalize + cert download (trust_authenticated mode end-to-end) |
|
||||
| 3 | live | HTTP-01 + DNS-01 + TLS-ALPN-01 challenge validation (challenge mode end-to-end) |
|
||||
| 4 | live | key rollover (RFC 8555 §7.3.5) + revoke-cert (§7.6) + ARI (RFC 9773) |
|
||||
| 5 | not yet | cert-manager integration test + production hardening |
|
||||
| 5 | live | rate limits + GC sweeper + kind-driven cert-manager integration test + lego conformance harness + k6 ACME-flow scenario |
|
||||
| 6 | not yet | full operator-facing reference + walkthroughs + threat model |
|
||||
|
||||
Track shipped phases via `git log --grep='acme-server:' --oneline`.
|
||||
@@ -306,3 +313,76 @@ Window math:
|
||||
Disable ARI globally with `CERTCTL_ACME_SERVER_ARI_ENABLED=false`. The
|
||||
URL drops out of the directory; the route is still registered but
|
||||
returns 404 — clients fall back to static renewal scheduling.
|
||||
|
||||
## Phase 5 — operational guidance
|
||||
|
||||
### Rate limiting
|
||||
|
||||
Production deployments serving multiple ACME profiles or fleets should
|
||||
keep the default rate limits in place. The four caps:
|
||||
|
||||
- `RATE_LIMIT_ORDERS_PER_HOUR` (100) — per-account new-order cap. A
|
||||
cert-manager Certificate that auto-renews at the 1/3 mark of its
|
||||
validity (90-day cert → ~30-day renewal) consumes ~12 orders/year
|
||||
per managed Certificate. 100/hour is generous for any plausible
|
||||
fleet.
|
||||
- `RATE_LIMIT_CONCURRENT_ORDERS` (5) — per-account cap on
|
||||
pending/ready/processing orders. Stops a runaway client from
|
||||
starving DB-row throughput. Tune up only if you observe legitimate
|
||||
bursts.
|
||||
- `RATE_LIMIT_KEY_CHANGE_PER_HOUR` (5) — rollovers are rare; a flood
|
||||
is an attack signal. Tune down to 1/hour if your operator
|
||||
procedure mandates manual rollovers only.
|
||||
- `RATE_LIMIT_CHALLENGE_RESPONDS_PER_HOUR` (60) — per-challenge cap,
|
||||
defends against retry storms.
|
||||
|
||||
Hits return RFC 8555 §6.7 `rateLimited` Problem with a `Retry-After`
|
||||
header. cert-manager 1.15+ honors the header; lego too. Older clients
|
||||
may not — that's the client's problem, not certctl's.
|
||||
|
||||
The buckets are **in-memory + per-replica**. A 3-replica certctl-
|
||||
server fleet behind a load balancer effectively has 3× the configured
|
||||
throughput (each replica's bucket fills independently). For
|
||||
deployments where this matters operationally, the right answer is a
|
||||
shared rate-limit store — that's a follow-up; not blocking for the
|
||||
current threat model where same-account requests typically pin to
|
||||
the same replica via session affinity.
|
||||
|
||||
### GC sweeper
|
||||
|
||||
The scheduler runs the GC sweep every `GC_INTERVAL` (default 1m). Each
|
||||
sweep is three independent SQL statements:
|
||||
|
||||
1. `DELETE FROM acme_nonces WHERE used = TRUE OR expires_at < NOW()`.
|
||||
2. `UPDATE acme_authorizations SET status='expired' WHERE status='pending' AND expires_at < NOW()`.
|
||||
3. `UPDATE acme_orders SET status='invalid', error=... WHERE status IN ('pending','ready','processing') AND expires_at < NOW()`.
|
||||
|
||||
Each statement is bounded by a 1-minute per-sweep timeout. A failing
|
||||
sweep is logged + retried on the next tick; a tick that overruns its
|
||||
budget is skipped (the existing-tick atomic-Bool guard prevents
|
||||
overlap). Counts are exposed via `certctl_acme_gc_*` Prometheus
|
||||
metrics.
|
||||
|
||||
### cert-manager integration test
|
||||
|
||||
`make acme-cert-manager-test` brings up a kind cluster, installs
|
||||
cert-manager 1.15.0, helm-deploys certctl-server with
|
||||
`acmeServer.enabled=true`, and verifies a Certificate resource issues
|
||||
end-to-end. Skipped in CI by default (kind is too heavy for per-PR);
|
||||
operators run locally on workstation. See
|
||||
`deploy/test/acme-integration/` for the YAML + Go test harness.
|
||||
|
||||
### lego RFC conformance harness
|
||||
|
||||
`make acme-rfc-conformance-test` drives lego v4 against a hermetic
|
||||
certctl-server stack, exercising register → new-order → finalize.
|
||||
Operators run this when shipping behavior changes to the ACME surface
|
||||
to confirm a real third-party client still works.
|
||||
|
||||
### k6 ACME flows scenario
|
||||
|
||||
`deploy/test/loadtest/k6/acme_flow.js` exercises the unauthenticated
|
||||
surface (directory + new-nonce + ARI) at 100 VUs × 5m. JWS-signed
|
||||
flows are out of scope for k6 (no JWS support); they're covered by
|
||||
the lego conformance harness above. Baseline numbers + thresholds in
|
||||
`deploy/test/loadtest/README.md`.
|
||||
|
||||
@@ -0,0 +1,166 @@
|
||||
// Copyright (c) certctl
|
||||
// SPDX-License-Identifier: BSL-1.1
|
||||
|
||||
package acme
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Phase 5 — per-account rolling-hour rate limiter for ACME operations.
|
||||
//
|
||||
// Architecture:
|
||||
// - In-memory token-bucket per (key, action). Restart wipes the
|
||||
// buckets; orders/hour caps are eventual-consistency so this is
|
||||
// acceptable. Persistent rate limiting is a follow-up if production
|
||||
// telemetry shows abuse patterns we can't catch in a single restart
|
||||
// cycle (master prompt criterion #11 explicitly accepts this).
|
||||
// - Tokens-per-hour math: bucket capacity = perHour, refill rate =
|
||||
// perHour / 3600 tokens/sec. A fresh bucket starts full; an over-
|
||||
// limit caller drains it then has to wait for replenishment.
|
||||
// - Key shape is action-specific: orders use accountID; key-rollover
|
||||
// uses accountID; challenge-respond uses challengeID (so a flood
|
||||
// against one challenge doesn't burn the whole account's budget).
|
||||
//
|
||||
// Concurrency: the outer map is RWMutex-guarded for create-on-demand;
|
||||
// per-bucket allow() takes a tiny per-bucket Mutex. Mirrors the
|
||||
// existing internal/api/middleware/middleware.go::keyedRateLimiter
|
||||
// pattern (different scope, same shape).
|
||||
|
||||
// RateLimiter is the per-action token-bucket pool. Construct with
|
||||
// NewRateLimiter(); pass a single instance into ACMEService via
|
||||
// SetRateLimiter so all entry points share the same buckets.
|
||||
type RateLimiter struct {
|
||||
mu sync.RWMutex
|
||||
buckets map[string]*rlBucket // keyed by "<action>|<keyID>"
|
||||
clock func() time.Time // injectable for tests
|
||||
}
|
||||
|
||||
// NewRateLimiter returns an empty RateLimiter. Buckets are created on
|
||||
// first reference, so a fresh limiter does no work until traffic
|
||||
// arrives.
|
||||
func NewRateLimiter() *RateLimiter {
|
||||
return &RateLimiter{
|
||||
buckets: make(map[string]*rlBucket),
|
||||
clock: time.Now,
|
||||
}
|
||||
}
|
||||
|
||||
// SetClock replaces the clock for tests. Production callers leave it
|
||||
// pointing at time.Now (the constructor default).
|
||||
func (r *RateLimiter) SetClock(now func() time.Time) {
|
||||
if now != nil {
|
||||
r.clock = now
|
||||
}
|
||||
}
|
||||
|
||||
// Allow returns true when the (action, keyID) bucket has at least one
|
||||
// token available — and consumes that token. perHour=0 disables the
|
||||
// limit (always true). Negative perHour is treated as 0.
|
||||
//
|
||||
// On hit (first call → first token consumed → returns true). Once
|
||||
// drained, further calls within the same hour return false until
|
||||
// elapsed-time refills the bucket.
|
||||
func (r *RateLimiter) Allow(action, keyID string, perHour int) bool {
|
||||
if perHour <= 0 {
|
||||
return true
|
||||
}
|
||||
bucketKey := action + "|" + keyID
|
||||
r.mu.RLock()
|
||||
b, ok := r.buckets[bucketKey]
|
||||
r.mu.RUnlock()
|
||||
if !ok {
|
||||
r.mu.Lock()
|
||||
b, ok = r.buckets[bucketKey]
|
||||
if !ok {
|
||||
b = &rlBucket{
|
||||
capacity: float64(perHour),
|
||||
refillRate: float64(perHour) / 3600.0, // tokens/sec
|
||||
tokens: float64(perHour),
|
||||
lastRefill: r.clock(),
|
||||
}
|
||||
r.buckets[bucketKey] = b
|
||||
}
|
||||
r.mu.Unlock()
|
||||
}
|
||||
return b.allow(r.clock)
|
||||
}
|
||||
|
||||
// RetryAfter returns the duration the caller should wait before the
|
||||
// (action, keyID) bucket has at least one token again. Returns 0 when
|
||||
// at least one token is currently available. Used by the handler to
|
||||
// emit a Retry-After header on rateLimited responses.
|
||||
func (r *RateLimiter) RetryAfter(action, keyID string, perHour int) time.Duration {
|
||||
if perHour <= 0 {
|
||||
return 0
|
||||
}
|
||||
bucketKey := action + "|" + keyID
|
||||
r.mu.RLock()
|
||||
b, ok := r.buckets[bucketKey]
|
||||
r.mu.RUnlock()
|
||||
if !ok {
|
||||
return 0
|
||||
}
|
||||
b.mu.Lock()
|
||||
defer b.mu.Unlock()
|
||||
if b.tokens >= 1 {
|
||||
return 0
|
||||
}
|
||||
missing := 1 - b.tokens
|
||||
if b.refillRate <= 0 {
|
||||
// Shouldn't happen (Allow rejects perHour<=0 before bucket
|
||||
// creation), but a divide-by-zero here would panic.
|
||||
return time.Hour
|
||||
}
|
||||
secs := missing / b.refillRate
|
||||
return time.Duration(secs * float64(time.Second))
|
||||
}
|
||||
|
||||
// rlBucket is the per-(action, keyID) token bucket. Mirrors the shape
|
||||
// of internal/api/middleware/middleware.go::tokenBucket but with a
|
||||
// per-hour-shaped refill instead of per-second.
|
||||
type rlBucket struct {
|
||||
mu sync.Mutex
|
||||
capacity float64
|
||||
refillRate float64 // tokens per second
|
||||
tokens float64
|
||||
lastRefill time.Time
|
||||
}
|
||||
|
||||
func (b *rlBucket) allow(clock func() time.Time) bool {
|
||||
b.mu.Lock()
|
||||
defer b.mu.Unlock()
|
||||
|
||||
now := clock()
|
||||
// Monotonic-clock-safe via t.Sub(t) per Go time-package contract.
|
||||
elapsed := now.Sub(b.lastRefill).Seconds()
|
||||
if elapsed > 0 {
|
||||
b.tokens += elapsed * b.refillRate
|
||||
if b.tokens > b.capacity {
|
||||
b.tokens = b.capacity
|
||||
}
|
||||
b.lastRefill = now
|
||||
}
|
||||
if b.tokens < 1 {
|
||||
return false
|
||||
}
|
||||
b.tokens--
|
||||
return true
|
||||
}
|
||||
|
||||
// Action constants — keep one source of truth for the bucket-key
|
||||
// `<action>|...` prefix. Using untyped consts (not iota) so they
|
||||
// survive cross-process coordination if a follow-up adds shared-state
|
||||
// rate-limiting.
|
||||
const (
|
||||
ActionNewOrder = "new_order"
|
||||
ActionKeyChange = "key_change"
|
||||
ActionChallengeRespond = "challenge_respond"
|
||||
)
|
||||
|
||||
// ErrRateLimited is the sentinel service-layer entry points return on
|
||||
// a hit. Handler maps to RFC 7807 + RFC 8555 §6.7
|
||||
// `urn:ietf:params:acme:error:rateLimited` with Retry-After.
|
||||
var ErrRateLimited = errors.New("acme: rate limit exceeded")
|
||||
@@ -0,0 +1,159 @@
|
||||
// Copyright (c) certctl
|
||||
// SPDX-License-Identifier: BSL-1.1
|
||||
|
||||
package acme
|
||||
|
||||
import (
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Phase 5 — RateLimiter unit tests.
|
||||
|
||||
func TestRateLimiter_DisabledWhenPerHourZero(t *testing.T) {
|
||||
r := NewRateLimiter()
|
||||
for i := 0; i < 10000; i++ {
|
||||
if !r.Allow(ActionNewOrder, "acc-1", 0) {
|
||||
t.Fatalf("Allow returned false on call %d with perHour=0", i)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestRateLimiter_DisabledWhenPerHourNegative(t *testing.T) {
|
||||
r := NewRateLimiter()
|
||||
if !r.Allow(ActionNewOrder, "acc-1", -5) {
|
||||
t.Errorf("Allow returned false with perHour=-5; expected always-allow")
|
||||
}
|
||||
}
|
||||
|
||||
func TestRateLimiter_BucketCapacity(t *testing.T) {
|
||||
// Frozen clock: a fresh bucket has perHour tokens. Drain exactly
|
||||
// that many; the next call must return false.
|
||||
now := time.Date(2026, 5, 3, 12, 0, 0, 0, time.UTC)
|
||||
r := NewRateLimiter()
|
||||
r.SetClock(func() time.Time { return now })
|
||||
|
||||
for i := 0; i < 100; i++ {
|
||||
if !r.Allow(ActionNewOrder, "acc-1", 100) {
|
||||
t.Fatalf("Allow returned false on call %d (within capacity)", i)
|
||||
}
|
||||
}
|
||||
if r.Allow(ActionNewOrder, "acc-1", 100) {
|
||||
t.Errorf("Allow returned true on the 101st call; expected limit hit")
|
||||
}
|
||||
}
|
||||
|
||||
func TestRateLimiter_PerKeyIsolation(t *testing.T) {
|
||||
// Frozen clock — drain acc-1 to zero, then acc-2 should still have
|
||||
// a full bucket (separate key).
|
||||
now := time.Date(2026, 5, 3, 12, 0, 0, 0, time.UTC)
|
||||
r := NewRateLimiter()
|
||||
r.SetClock(func() time.Time { return now })
|
||||
|
||||
for i := 0; i < 100; i++ {
|
||||
_ = r.Allow(ActionNewOrder, "acc-1", 100)
|
||||
}
|
||||
if r.Allow(ActionNewOrder, "acc-1", 100) {
|
||||
t.Errorf("acc-1 should be rate-limited")
|
||||
}
|
||||
if !r.Allow(ActionNewOrder, "acc-2", 100) {
|
||||
t.Errorf("acc-2 should be unaffected by acc-1's bucket; expected allow")
|
||||
}
|
||||
}
|
||||
|
||||
func TestRateLimiter_PerActionIsolation(t *testing.T) {
|
||||
// Same key but different actions get different buckets.
|
||||
now := time.Date(2026, 5, 3, 12, 0, 0, 0, time.UTC)
|
||||
r := NewRateLimiter()
|
||||
r.SetClock(func() time.Time { return now })
|
||||
|
||||
for i := 0; i < 5; i++ {
|
||||
_ = r.Allow(ActionKeyChange, "acc-1", 5)
|
||||
}
|
||||
if r.Allow(ActionKeyChange, "acc-1", 5) {
|
||||
t.Errorf("ActionKeyChange should be rate-limited")
|
||||
}
|
||||
// ActionNewOrder for the same key has its own (empty) bucket.
|
||||
if !r.Allow(ActionNewOrder, "acc-1", 100) {
|
||||
t.Errorf("ActionNewOrder for same key should be allowed (different bucket)")
|
||||
}
|
||||
}
|
||||
|
||||
func TestRateLimiter_RefillOverTime(t *testing.T) {
|
||||
// Drain bucket; advance the clock; expect tokens replenished.
|
||||
current := time.Date(2026, 5, 3, 12, 0, 0, 0, time.UTC)
|
||||
r := NewRateLimiter()
|
||||
r.SetClock(func() time.Time { return current })
|
||||
|
||||
for i := 0; i < 100; i++ {
|
||||
_ = r.Allow(ActionNewOrder, "acc-1", 100)
|
||||
}
|
||||
if r.Allow(ActionNewOrder, "acc-1", 100) {
|
||||
t.Fatalf("expected limit hit after draining bucket")
|
||||
}
|
||||
// Advance by 36 seconds: at 100/hour = 100/3600 tokens/sec ≈
|
||||
// 0.0278/sec. 36 * 0.0278 = 1.00 tokens — exactly enough for 1
|
||||
// more call.
|
||||
current = current.Add(36 * time.Second)
|
||||
if !r.Allow(ActionNewOrder, "acc-1", 100) {
|
||||
t.Errorf("Allow returned false after 36s elapsed; expected ≥1 token replenished")
|
||||
}
|
||||
}
|
||||
|
||||
func TestRateLimiter_RetryAfter(t *testing.T) {
|
||||
now := time.Date(2026, 5, 3, 12, 0, 0, 0, time.UTC)
|
||||
r := NewRateLimiter()
|
||||
r.SetClock(func() time.Time { return now })
|
||||
|
||||
// Drain to zero.
|
||||
for i := 0; i < 100; i++ {
|
||||
_ = r.Allow(ActionNewOrder, "acc-1", 100)
|
||||
}
|
||||
d := r.RetryAfter(ActionNewOrder, "acc-1", 100)
|
||||
// 1 token at 100/hour = 36 seconds.
|
||||
if d < 35*time.Second || d > 37*time.Second {
|
||||
t.Errorf("RetryAfter = %v, expected ~36s", d)
|
||||
}
|
||||
// Allow above capacity — RetryAfter returns 0 on a fresh bucket.
|
||||
if zero := r.RetryAfter(ActionNewOrder, "acc-fresh", 100); zero != 0 {
|
||||
t.Errorf("RetryAfter for fresh bucket = %v, expected 0", zero)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRateLimiter_ConcurrentAccess(t *testing.T) {
|
||||
// Hammer 200 goroutines × 200 calls each = 40000 calls against a
|
||||
// 1000-token bucket; assert no panic, no data race (run with -race),
|
||||
// and that no more than 1000 calls succeeded.
|
||||
now := time.Date(2026, 5, 3, 12, 0, 0, 0, time.UTC)
|
||||
r := NewRateLimiter()
|
||||
r.SetClock(func() time.Time { return now })
|
||||
|
||||
var (
|
||||
wg sync.WaitGroup
|
||||
success int64
|
||||
mu sync.Mutex
|
||||
)
|
||||
for g := 0; g < 200; g++ {
|
||||
wg.Add(1)
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
local := int64(0)
|
||||
for i := 0; i < 200; i++ {
|
||||
if r.Allow(ActionNewOrder, "shared-acc", 1000) {
|
||||
local++
|
||||
}
|
||||
}
|
||||
mu.Lock()
|
||||
success += local
|
||||
mu.Unlock()
|
||||
}()
|
||||
}
|
||||
wg.Wait()
|
||||
if success > 1000 {
|
||||
t.Errorf("got %d successes, want ≤ 1000 (bucket capacity)", success)
|
||||
}
|
||||
if success < 1000 {
|
||||
t.Errorf("got %d successes, want exactly 1000 (frozen clock, no refill)", success)
|
||||
}
|
||||
}
|
||||
@@ -274,6 +274,23 @@ func writeServiceError(w http.ResponseWriter, err error) {
|
||||
})
|
||||
case errors.Is(err, service.ErrACMEARIBadCertID):
|
||||
acme.WriteProblem(w, acme.Malformed("ARI cert-id is malformed"))
|
||||
case errors.Is(err, service.ErrACMERateLimited):
|
||||
// RFC 8555 §6.7 + RFC 7807. The handler doesn't have the
|
||||
// (action, key) tuple here so we can't emit a precise
|
||||
// Retry-After; the entry-point handlers (NewOrder etc.) emit
|
||||
// their own Retry-After header before delegating to the
|
||||
// service, leaving this catchall for completeness.
|
||||
acme.WriteProblem(w, acme.Problem{
|
||||
Type: "urn:ietf:params:acme:error:rateLimited",
|
||||
Detail: "request rate limit exceeded; retry later",
|
||||
Status: http.StatusTooManyRequests,
|
||||
})
|
||||
case errors.Is(err, service.ErrACMEConcurrentOrdersExceeded):
|
||||
acme.WriteProblem(w, acme.Problem{
|
||||
Type: "urn:ietf:params:acme:error:rateLimited",
|
||||
Detail: "too many concurrent orders for this account; finish or cancel pending orders before submitting more",
|
||||
Status: http.StatusTooManyRequests,
|
||||
})
|
||||
default:
|
||||
// Avoid leaking internal error text per master-prompt
|
||||
// criterion #10 (operator-actionable errors with no info
|
||||
|
||||
@@ -744,6 +744,42 @@ type ACMEServerConfig struct {
|
||||
// certs. Setting: CERTCTL_ACME_SERVER_ARI_POLL_INTERVAL.
|
||||
ARIPollInterval time.Duration
|
||||
|
||||
// RateLimitOrdersPerHour caps new-order requests per ACME account per
|
||||
// rolling hour. 0 disables (no limit). Default: 100. Hits return RFC
|
||||
// 7807 + RFC 8555 §6.7 `urn:ietf:params:acme:error:rateLimited` with
|
||||
// a Retry-After header. In-memory token-bucket — restart wipes the
|
||||
// counter, which is acceptable for orders/hour caps (eventual-
|
||||
// consistency anyway). Setting:
|
||||
// CERTCTL_ACME_SERVER_RATE_LIMIT_ORDERS_PER_HOUR.
|
||||
RateLimitOrdersPerHour int
|
||||
|
||||
// RateLimitConcurrentOrders caps the number of orders an ACME account
|
||||
// can have in pending/ready/processing state simultaneously. 0
|
||||
// disables. Default: 5. Same Problem shape as the per-hour limit.
|
||||
// Setting: CERTCTL_ACME_SERVER_RATE_LIMIT_CONCURRENT_ORDERS.
|
||||
RateLimitConcurrentOrders int
|
||||
|
||||
// RateLimitKeyChangePerHour caps account-key rollovers per account
|
||||
// per rolling hour. 0 disables. Default: 5 (rollovers should be rare;
|
||||
// a flood is an attack signal). Setting:
|
||||
// CERTCTL_ACME_SERVER_RATE_LIMIT_KEY_CHANGE_PER_HOUR.
|
||||
RateLimitKeyChangePerHour int
|
||||
|
||||
// RateLimitChallengeRespondsPerHour caps challenge-respond requests
|
||||
// per challenge per rolling hour. 0 disables. Default: 60 (defends
|
||||
// against retry storms from a misbehaving client). Setting:
|
||||
// CERTCTL_ACME_SERVER_RATE_LIMIT_CHALLENGE_RESPONDS_PER_HOUR.
|
||||
RateLimitChallengeRespondsPerHour int
|
||||
|
||||
// GCInterval is the tick interval for the ACME GC scheduler loop.
|
||||
// On each tick the loop sweeps expired nonces, transitions expired
|
||||
// pending authzs to `expired`, transitions expired
|
||||
// pending/ready/processing orders to `invalid`, and reaps Phase-2
|
||||
// atomicity-window orphans (orders without a linked cert when one
|
||||
// should exist). 0 disables the loop entirely. Default: 1m. Setting:
|
||||
// CERTCTL_ACME_SERVER_GC_INTERVAL.
|
||||
GCInterval time.Duration
|
||||
|
||||
// DirectoryMeta is the optional metadata advertised in the directory
|
||||
// document per RFC 8555 §7.1.1.
|
||||
DirectoryMeta ACMEServerDirectoryMeta
|
||||
@@ -1791,6 +1827,11 @@ func Load() (*Config, error) {
|
||||
TLSALPN01ConcurrencyMax: getEnvInt("CERTCTL_ACME_SERVER_TLSALPN01_CONCURRENCY", 10),
|
||||
ARIEnabled: getEnvBool("CERTCTL_ACME_SERVER_ARI_ENABLED", true),
|
||||
ARIPollInterval: getEnvDuration("CERTCTL_ACME_SERVER_ARI_POLL_INTERVAL", 6*time.Hour),
|
||||
RateLimitOrdersPerHour: getEnvInt("CERTCTL_ACME_SERVER_RATE_LIMIT_ORDERS_PER_HOUR", 100),
|
||||
RateLimitConcurrentOrders: getEnvInt("CERTCTL_ACME_SERVER_RATE_LIMIT_CONCURRENT_ORDERS", 5),
|
||||
RateLimitKeyChangePerHour: getEnvInt("CERTCTL_ACME_SERVER_RATE_LIMIT_KEY_CHANGE_PER_HOUR", 5),
|
||||
RateLimitChallengeRespondsPerHour: getEnvInt("CERTCTL_ACME_SERVER_RATE_LIMIT_CHALLENGE_RESPONDS_PER_HOUR", 60),
|
||||
GCInterval: getEnvDuration("CERTCTL_ACME_SERVER_GC_INTERVAL", time.Minute),
|
||||
DirectoryMeta: ACMEServerDirectoryMeta{
|
||||
TermsOfService: getEnv("CERTCTL_ACME_SERVER_TOS_URL", ""),
|
||||
Website: getEnv("CERTCTL_ACME_SERVER_WEBSITE", ""),
|
||||
|
||||
@@ -751,6 +751,85 @@ func (r *ACMERepository) AccountOwnsCertificate(ctx context.Context, accountID,
|
||||
return count > 0, nil
|
||||
}
|
||||
|
||||
// --- Phase 5 — concurrent-orders count + GC sweeps ---------------------
|
||||
|
||||
// CountActiveOrdersByAccount returns the number of acme_orders rows
|
||||
// with the given account_id where status is in
|
||||
// {pending, ready, processing}. Used by the per-account
|
||||
// concurrent-orders rate limit.
|
||||
func (r *ACMERepository) CountActiveOrdersByAccount(ctx context.Context, accountID string) (int, error) {
|
||||
var count int
|
||||
err := r.db.QueryRowContext(ctx, `
|
||||
SELECT COUNT(1)
|
||||
FROM acme_orders
|
||||
WHERE account_id = $1
|
||||
AND status IN ('pending', 'ready', 'processing')
|
||||
`, accountID).Scan(&count)
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("acme: count active orders: %w", err)
|
||||
}
|
||||
return count, nil
|
||||
}
|
||||
|
||||
// GCExpiredNonces deletes nonce rows that have been used or have
|
||||
// passed their expires_at. Returns rows-affected count for telemetry.
|
||||
// Phase 5 — called every GCInterval from the scheduler.
|
||||
func (r *ACMERepository) GCExpiredNonces(ctx context.Context) (int64, error) {
|
||||
res, err := r.db.ExecContext(ctx, `
|
||||
DELETE FROM acme_nonces
|
||||
WHERE used = TRUE OR expires_at < NOW()
|
||||
`)
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("acme: gc expired nonces: %w", err)
|
||||
}
|
||||
n, err := res.RowsAffected()
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("acme: gc expired nonces rows affected: %w", err)
|
||||
}
|
||||
return n, nil
|
||||
}
|
||||
|
||||
// GCExpireAuthorizations transitions authzs in `pending` whose
|
||||
// expires_at < NOW() to `expired`. Authzs in valid/invalid are left
|
||||
// alone (they're already terminal). Returns rows-affected count.
|
||||
func (r *ACMERepository) GCExpireAuthorizations(ctx context.Context) (int64, error) {
|
||||
res, err := r.db.ExecContext(ctx, `
|
||||
UPDATE acme_authorizations
|
||||
SET status = 'expired', updated_at = NOW()
|
||||
WHERE status = 'pending' AND expires_at < NOW()
|
||||
`)
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("acme: gc expire authorizations: %w", err)
|
||||
}
|
||||
n, err := res.RowsAffected()
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("acme: gc expire authorizations rows affected: %w", err)
|
||||
}
|
||||
return n, nil
|
||||
}
|
||||
|
||||
// GCInvalidateExpiredOrders transitions orders in
|
||||
// pending/ready/processing whose expires_at < NOW() to `invalid` with
|
||||
// a server-internal error. Orders in valid/invalid are terminal and
|
||||
// untouched.
|
||||
func (r *ACMERepository) GCInvalidateExpiredOrders(ctx context.Context) (int64, error) {
|
||||
const errBlob = `{"type":"urn:ietf:params:acme:error:serverInternal","detail":"order expired before issuance","status":500}`
|
||||
res, err := r.db.ExecContext(ctx, `
|
||||
UPDATE acme_orders
|
||||
SET status = 'invalid', error = $1::jsonb, updated_at = NOW()
|
||||
WHERE status IN ('pending', 'ready', 'processing')
|
||||
AND expires_at < NOW()
|
||||
`, errBlob)
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("acme: gc invalidate expired orders: %w", err)
|
||||
}
|
||||
n, err := res.RowsAffected()
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("acme: gc invalidate expired orders rows affected: %w", err)
|
||||
}
|
||||
return n, nil
|
||||
}
|
||||
|
||||
// scanACMEAccount is the shared shape for the SELECT-by-X account
|
||||
// queries above. Returns sql.ErrNoRows-wrapped repository.ErrNotFound
|
||||
// on miss; any other scan failure surfaces verbatim.
|
||||
|
||||
@@ -77,6 +77,13 @@ type CRLCacheServicer interface {
|
||||
RegenerateAll(ctx context.Context)
|
||||
}
|
||||
|
||||
// ACMEGarbageCollector is the interface the scheduler's acmeGCLoop
|
||||
// invokes once per tick. The concrete implementation is *service.ACMEService.
|
||||
// Phase 5 — sweeps expired nonces / authzs / orders.
|
||||
type ACMEGarbageCollector interface {
|
||||
GarbageCollect(ctx context.Context) error
|
||||
}
|
||||
|
||||
// JobReaperService defines the interface for job timeout reaping used by the scheduler.
|
||||
type JobReaperService interface {
|
||||
ReapTimedOutJobs(ctx context.Context, csrTTL, approvalTTL time.Duration) error
|
||||
@@ -101,6 +108,7 @@ type Scheduler struct {
|
||||
healthCheckService HealthCheckServicer
|
||||
cloudDiscoveryService CloudDiscoveryServicer
|
||||
crlCacheService CRLCacheServicer
|
||||
acmeGC ACMEGarbageCollector
|
||||
jobReaper JobReaperService
|
||||
logger *slog.Logger
|
||||
|
||||
@@ -118,6 +126,7 @@ type Scheduler struct {
|
||||
cloudDiscoveryInterval time.Duration
|
||||
crlGenerationInterval time.Duration
|
||||
jobTimeoutInterval time.Duration
|
||||
acmeGCInterval time.Duration
|
||||
// agentOfflineJobTTL: per-tick threshold for reaping Running jobs whose
|
||||
// owning agent has been silent. Bundle C / Audit M-016. Defaults below.
|
||||
agentOfflineJobTTL time.Duration
|
||||
@@ -138,6 +147,7 @@ type Scheduler struct {
|
||||
cloudDiscoveryRunning atomic.Bool
|
||||
crlGenerationRunning atomic.Bool
|
||||
jobTimeoutRunning atomic.Bool
|
||||
acmeGCRunning atomic.Bool
|
||||
|
||||
// Graceful shutdown: wait for in-flight work to complete
|
||||
wg sync.WaitGroup
|
||||
@@ -174,6 +184,7 @@ func NewScheduler(
|
||||
cloudDiscoveryInterval: 6 * time.Hour,
|
||||
crlGenerationInterval: 1 * time.Hour,
|
||||
jobTimeoutInterval: 10 * time.Minute,
|
||||
acmeGCInterval: 1 * time.Minute,
|
||||
// 5 minutes is 5×agentHealthCheckInterval default of 1m; an agent
|
||||
// must miss multiple heartbeats before its in-flight jobs are reaped.
|
||||
agentOfflineJobTTL: 5 * time.Minute,
|
||||
@@ -287,6 +298,25 @@ func (s *Scheduler) SetJobReaperService(jr JobReaperService) {
|
||||
s.jobReaper = jr
|
||||
}
|
||||
|
||||
// SetACMEGarbageCollector wires the ACME GC service. Phase 5 — when
|
||||
// non-nil, an acmeGCLoop runs every acmeGCInterval and sweeps expired
|
||||
// nonces / authzs / orders. Optional: leaving nil disables the loop
|
||||
// (legacy behavior pre-Phase-5).
|
||||
func (s *Scheduler) SetACMEGarbageCollector(gc ACMEGarbageCollector) {
|
||||
s.acmeGC = gc
|
||||
}
|
||||
|
||||
// SetACMEGCInterval configures the interval at which the ACME GC sweep
|
||||
// runs. Default 1m. Operators with quiet fleets can lengthen to 5m;
|
||||
// operators expecting nonce-storms can shorten to 30s. Zero or
|
||||
// negative values are ignored.
|
||||
func (s *Scheduler) SetACMEGCInterval(d time.Duration) {
|
||||
if d <= 0 {
|
||||
return
|
||||
}
|
||||
s.acmeGCInterval = d
|
||||
}
|
||||
|
||||
// SetAgentOfflineJobTTL sets the threshold past which a Running job whose
|
||||
// owning agent has gone silent is reaped to Failed. Bundle C / Audit M-016.
|
||||
// Zero or negative values are ignored (the default of 5 minutes is kept).
|
||||
@@ -342,6 +372,9 @@ func (s *Scheduler) Start(ctx context.Context) <-chan struct{} {
|
||||
if s.crlCacheService != nil {
|
||||
loopCount++
|
||||
}
|
||||
if s.acmeGC != nil {
|
||||
loopCount++
|
||||
}
|
||||
s.wg.Add(loopCount)
|
||||
|
||||
go func() { defer s.wg.Done(); s.renewalCheckLoop(ctx) }()
|
||||
@@ -367,6 +400,9 @@ func (s *Scheduler) Start(ctx context.Context) <-chan struct{} {
|
||||
if s.crlCacheService != nil {
|
||||
go func() { defer s.wg.Done(); s.crlGenerationLoop(ctx) }()
|
||||
}
|
||||
if s.acmeGC != nil {
|
||||
go func() { defer s.wg.Done(); s.acmeGCLoop(ctx) }()
|
||||
}
|
||||
|
||||
// Signal that all loops are launched
|
||||
close(startedChan)
|
||||
@@ -1074,3 +1110,39 @@ func (s *Scheduler) runCRLGeneration(ctx context.Context) {
|
||||
|
||||
// ErrSchedulerShutdownTimeout is returned when scheduler graceful shutdown times out.
|
||||
var ErrSchedulerShutdownTimeout = errors.New("scheduler graceful shutdown timeout")
|
||||
|
||||
// acmeGCLoop runs every acmeGCInterval and invokes ACMEGarbageCollector.
|
||||
// Per CLAUDE.md "Scheduler idempotency" architecture decision: an
|
||||
// atomic.Bool guard prevents concurrent tick execution; the
|
||||
// sync.WaitGroup tracks the in-flight goroutine for graceful shutdown.
|
||||
// Phase 5.
|
||||
func (s *Scheduler) acmeGCLoop(ctx context.Context) {
|
||||
ticker := time.NewTicker(s.acmeGCInterval)
|
||||
defer ticker.Stop()
|
||||
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-ticker.C:
|
||||
if !s.acmeGCRunning.CompareAndSwap(false, true) {
|
||||
s.logger.Warn("ACME GC sweep still running, skipping tick")
|
||||
continue
|
||||
}
|
||||
s.wg.Add(1)
|
||||
go func() {
|
||||
defer s.wg.Done()
|
||||
defer s.acmeGCRunning.Store(false)
|
||||
// 1-minute timeout per sweep — the per-statement work is
|
||||
// cheap (single DELETE / UPDATE per sweep, all on indexed
|
||||
// columns), but bound the cycle so a stuck Postgres can't
|
||||
// block the next tick.
|
||||
opCtx, cancel := context.WithTimeout(ctx, time.Minute)
|
||||
defer cancel()
|
||||
if err := s.acmeGC.GarbageCollect(opCtx); err != nil {
|
||||
s.logger.Warn("acme gc sweep failed (next tick will retry)", "error", err)
|
||||
}
|
||||
}()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -56,6 +56,20 @@ type ACMERepo interface {
|
||||
// Phase 4 — key rollover + revocation auth.
|
||||
UpdateAccountJWKWithTx(ctx context.Context, q repository.Querier, accountID, expectedOldThumbprint, newThumbprint, newJWKPEM string) error
|
||||
AccountOwnsCertificate(ctx context.Context, accountID, certificateID string) (bool, error)
|
||||
// Phase 5 — per-account concurrent-order count + GC sweeps.
|
||||
// CountActiveOrdersByAccount returns the number of orders in
|
||||
// pending/ready/processing for the given account.
|
||||
CountActiveOrdersByAccount(ctx context.Context, accountID string) (int, error)
|
||||
// GCExpiredNonces deletes nonces whose expires_at < now() OR
|
||||
// used = true. Returns rows-affected count for telemetry.
|
||||
GCExpiredNonces(ctx context.Context) (int64, error)
|
||||
// GCExpireAuthorizations transitions authzs in `pending` whose
|
||||
// expires_at < now() to `expired`. Returns rows-affected count.
|
||||
GCExpireAuthorizations(ctx context.Context) (int64, error)
|
||||
// GCInvalidateExpiredOrders transitions orders in
|
||||
// pending/ready/processing whose expires_at < now() to `invalid`
|
||||
// with a server-internal error. Returns rows-affected count.
|
||||
GCInvalidateExpiredOrders(ctx context.Context) (int64, error)
|
||||
}
|
||||
|
||||
// CertificateRevoker is the minimum surface ACMEService needs to route
|
||||
@@ -142,6 +156,12 @@ type ACMEService struct {
|
||||
// and RenewalInfo returns the no-policy default window.
|
||||
revoker CertificateRevoker
|
||||
renewalPolicies RenewalPolicyLookup
|
||||
|
||||
// Phase 5 — per-account rate limiter. cmd/server/main.go constructs
|
||||
// an *acme.RateLimiter and wires it via SetRateLimiter. When unset
|
||||
// (tests, legacy bootstrap) the limiter calls short-circuit to
|
||||
// "always allow" — same shape as the validatorPool unset case.
|
||||
rateLimiter *acme.RateLimiter
|
||||
}
|
||||
|
||||
// NewACMEService constructs an ACMEService with the directory + nonce
|
||||
@@ -196,6 +216,16 @@ func (s *ACMEService) SetRevocationDelegate(r CertificateRevoker) { s.revoker =
|
||||
// still returns 200.
|
||||
func (s *ACMEService) SetRenewalPolicyLookup(r RenewalPolicyLookup) { s.renewalPolicies = r }
|
||||
|
||||
// SetRateLimiter wires Phase 5's per-account rate limiter. Optional —
|
||||
// when nil, the per-action rate-limit checks short-circuit to
|
||||
// "always allow" so the legacy code path stays unchanged for bootstrap
|
||||
// + tests that don't care about throttling.
|
||||
func (s *ACMEService) SetRateLimiter(r *acme.RateLimiter) { s.rateLimiter = r }
|
||||
|
||||
// RateLimiter returns the wired limiter so the handler can compute
|
||||
// Retry-After durations on rate-limited responses without re-checking.
|
||||
func (s *ACMEService) RateLimiter() *acme.RateLimiter { return s.rateLimiter }
|
||||
|
||||
// SetValidatorPool wires Phase 3's challenge validator pool.
|
||||
// cmd/server/main.go constructs an *acme.Pool at startup with the
|
||||
// per-type concurrency caps from cfg.ACMEServer. Optional —
|
||||
@@ -334,6 +364,19 @@ var ErrACMEARIDisabled = errors.New("acme: ARI is disabled on this server")
|
||||
// not RFC 9773 §4.1 shape. Handler maps to 400 + malformed.
|
||||
var ErrACMEARIBadCertID = errors.New("acme: ARI cert-id is malformed")
|
||||
|
||||
// Phase 5 sentinels.
|
||||
|
||||
// ErrACMERateLimited is returned when the per-action rate limit fires.
|
||||
// Handler maps to RFC 7807 + RFC 8555 §6.7
|
||||
// `urn:ietf:params:acme:error:rateLimited` with a Retry-After header.
|
||||
var ErrACMERateLimited = errors.New("acme: rate limit exceeded")
|
||||
|
||||
// ErrACMEConcurrentOrdersExceeded is returned by CreateOrder when the
|
||||
// account already has cfg.RateLimitConcurrentOrders orders in
|
||||
// pending/ready/processing. Handler maps to rateLimited (RFC 8555 §6.7
|
||||
// shape; the certctl-side cause is concurrency rather than per-hour).
|
||||
var ErrACMEConcurrentOrdersExceeded = errors.New("acme: concurrent orders limit exceeded")
|
||||
|
||||
// BuildDirectory constructs the per-profile directory document.
|
||||
//
|
||||
// profileID resolution:
|
||||
@@ -463,6 +506,13 @@ type ACMEMetrics struct {
|
||||
RevokeCertFailTotal atomic.Uint64 // rejected revocation (4xx)
|
||||
RenewalInfoTotal atomic.Uint64 // ARI 200
|
||||
RenewalInfoFailTotal atomic.Uint64 // ARI 4xx
|
||||
|
||||
// Phase 5 — GC sweep counts (per-tick rows-affected, summed).
|
||||
GCNoncesReapedTotal atomic.Uint64
|
||||
GCAuthzsExpiredTotal atomic.Uint64
|
||||
GCOrdersInvalidatedTotal atomic.Uint64
|
||||
GCRunsTotal atomic.Uint64
|
||||
GCRunFailuresTotal atomic.Uint64
|
||||
}
|
||||
|
||||
// NewACMEMetrics returns a zeroed counter table. Concurrent callers
|
||||
@@ -508,6 +558,11 @@ func (m *ACMEMetrics) Snapshot() map[string]uint64 {
|
||||
"certctl_acme_revoke_cert_failures_total": m.RevokeCertFailTotal.Load(),
|
||||
"certctl_acme_renewal_info_total": m.RenewalInfoTotal.Load(),
|
||||
"certctl_acme_renewal_info_failures_total": m.RenewalInfoFailTotal.Load(),
|
||||
"certctl_acme_gc_nonces_reaped_total": m.GCNoncesReapedTotal.Load(),
|
||||
"certctl_acme_gc_authzs_expired_total": m.GCAuthzsExpiredTotal.Load(),
|
||||
"certctl_acme_gc_orders_invalidated_total": m.GCOrdersInvalidatedTotal.Load(),
|
||||
"certctl_acme_gc_runs_total": m.GCRunsTotal.Load(),
|
||||
"certctl_acme_gc_run_failures_total": m.GCRunFailuresTotal.Load(),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -783,6 +838,27 @@ func (s *ACMEService) CreateOrder(
|
||||
s.metrics.bump(&s.metrics.NewOrderFailureTotal)
|
||||
return nil, fmt.Errorf("acme: new-order requires SetTransactor + SetAuditService")
|
||||
}
|
||||
// Phase 5 — per-account orders/hour cap. Hits return rateLimited
|
||||
// (RFC 8555 §6.7) before any DB work. Counter is in-memory; restart
|
||||
// wipes (eventual-consistency caps are acceptable).
|
||||
if s.rateLimiter != nil && s.cfg.RateLimitOrdersPerHour > 0 {
|
||||
if !s.rateLimiter.Allow(acme.ActionNewOrder, accountID, s.cfg.RateLimitOrdersPerHour) {
|
||||
s.metrics.bump(&s.metrics.NewOrderFailureTotal)
|
||||
return nil, ErrACMERateLimited
|
||||
}
|
||||
}
|
||||
// Phase 5 — concurrent-orders cap. We count
|
||||
// pending/ready/processing orders for this account; if at-or-over
|
||||
// the cap, reject. This is a DB read (no FOR UPDATE), so two
|
||||
// requests racing under the threshold can both succeed and push
|
||||
// the account one over — accepted as eventual-consistency.
|
||||
if s.cfg.RateLimitConcurrentOrders > 0 {
|
||||
count, cerr := s.repo.CountActiveOrdersByAccount(ctx, accountID)
|
||||
if cerr == nil && count >= s.cfg.RateLimitConcurrentOrders {
|
||||
s.metrics.bump(&s.metrics.NewOrderFailureTotal)
|
||||
return nil, ErrACMEConcurrentOrdersExceeded
|
||||
}
|
||||
}
|
||||
resolvedProfileID, err := s.resolveProfile(ctx, profileID)
|
||||
if err != nil {
|
||||
s.metrics.bump(&s.metrics.NewOrderFailureTotal)
|
||||
@@ -1285,6 +1361,16 @@ func (s *ACMEService) RespondToChallenge(
|
||||
s.metrics.bump(&s.metrics.ChallengeRespondFailTotal)
|
||||
return nil, ErrACMEChallengePoolUnconfigured
|
||||
}
|
||||
// Phase 5 — per-challenge respond rate limit. Defends against retry
|
||||
// storms from a misbehaving client. Keyed by challengeID (not
|
||||
// accountID) so a flood against one challenge doesn't drain the
|
||||
// account's whole budget.
|
||||
if s.rateLimiter != nil && s.cfg.RateLimitChallengeRespondsPerHour > 0 {
|
||||
if !s.rateLimiter.Allow(acme.ActionChallengeRespond, challengeID, s.cfg.RateLimitChallengeRespondsPerHour) {
|
||||
s.metrics.bump(&s.metrics.ChallengeRespondFailTotal)
|
||||
return nil, ErrACMERateLimited
|
||||
}
|
||||
}
|
||||
|
||||
ch, err := s.repo.GetChallengeByID(ctx, challengeID)
|
||||
if err != nil {
|
||||
@@ -1508,6 +1594,14 @@ func (s *ACMEService) RotateAccountKey(
|
||||
s.metrics.bump(&s.metrics.KeyChangeFailTotal)
|
||||
return nil, ErrACMEKeyRolloverInvalid
|
||||
}
|
||||
// Phase 5 — rollovers/hour cap. Defaults to 5/hour: a flood is an
|
||||
// attack signal (key rotation should be rare). Keyed by accountID.
|
||||
if s.rateLimiter != nil && s.cfg.RateLimitKeyChangePerHour > 0 {
|
||||
if !s.rateLimiter.Allow(acme.ActionKeyChange, oldAccount.AccountID, s.cfg.RateLimitKeyChangePerHour) {
|
||||
s.metrics.bump(&s.metrics.KeyChangeFailTotal)
|
||||
return nil, ErrACMERateLimited
|
||||
}
|
||||
}
|
||||
|
||||
newThumbprint, err := acme.JWKThumbprint(newJWK)
|
||||
if err != nil {
|
||||
@@ -1816,3 +1910,56 @@ func mapACMERevocationReason(code int) string {
|
||||
return string(domain.RevocationReasonUnspecified)
|
||||
}
|
||||
}
|
||||
|
||||
// GarbageCollect runs a single ACME GC sweep. Phase 5 — the scheduler
|
||||
// invokes this every cfg.GCInterval. Three independent sweeps:
|
||||
//
|
||||
// 1. Delete used / expired nonces.
|
||||
// 2. Transition expired pending authzs to `expired`.
|
||||
// 3. Transition expired pending/ready/processing orders to `invalid`.
|
||||
//
|
||||
// Each sweep is a single SQL statement (no per-row transactions) so a
|
||||
// large reap is one atomic write per sweep. Per-sweep errors are
|
||||
// logged-and-continued: a failing nonces sweep doesn't block the
|
||||
// authzs sweep. Returns the first error encountered (for caller
|
||||
// telemetry); per-sweep counts are recorded on metrics regardless.
|
||||
//
|
||||
// Idempotent — repeated runs are safe; the second run finds 0 rows.
|
||||
func (s *ACMEService) GarbageCollect(ctx context.Context) error {
|
||||
s.metrics.bump(&s.metrics.GCRunsTotal)
|
||||
var firstErr error
|
||||
|
||||
if n, err := s.repo.GCExpiredNonces(ctx); err != nil {
|
||||
s.metrics.bump(&s.metrics.GCRunFailuresTotal)
|
||||
if firstErr == nil {
|
||||
firstErr = fmt.Errorf("acme gc: nonces: %w", err)
|
||||
}
|
||||
} else if n > 0 {
|
||||
atomicAddUint64(&s.metrics.GCNoncesReapedTotal, uint64(n))
|
||||
}
|
||||
|
||||
if n, err := s.repo.GCExpireAuthorizations(ctx); err != nil {
|
||||
s.metrics.bump(&s.metrics.GCRunFailuresTotal)
|
||||
if firstErr == nil {
|
||||
firstErr = fmt.Errorf("acme gc: authzs: %w", err)
|
||||
}
|
||||
} else if n > 0 {
|
||||
atomicAddUint64(&s.metrics.GCAuthzsExpiredTotal, uint64(n))
|
||||
}
|
||||
|
||||
if n, err := s.repo.GCInvalidateExpiredOrders(ctx); err != nil {
|
||||
s.metrics.bump(&s.metrics.GCRunFailuresTotal)
|
||||
if firstErr == nil {
|
||||
firstErr = fmt.Errorf("acme gc: orders: %w", err)
|
||||
}
|
||||
} else if n > 0 {
|
||||
atomicAddUint64(&s.metrics.GCOrdersInvalidatedTotal, uint64(n))
|
||||
}
|
||||
|
||||
return firstErr
|
||||
}
|
||||
|
||||
// atomicAddUint64 adds delta to the counter. The metrics struct exposes
|
||||
// only `bump` (add 1) by default; this helper covers the
|
||||
// rows-affected-N case the GC needs.
|
||||
func atomicAddUint64(c *atomic.Uint64, delta uint64) { c.Add(delta) }
|
||||
|
||||
@@ -164,6 +164,23 @@ func (f *fakeACMERepo) UpdateAccountJWKWithTx(ctx context.Context, q repository.
|
||||
func (f *fakeACMERepo) AccountOwnsCertificate(ctx context.Context, accountID, certificateID string) (bool, error) {
|
||||
return false, nil
|
||||
}
|
||||
func (f *fakeACMERepo) CountActiveOrdersByAccount(ctx context.Context, accountID string) (int, error) {
|
||||
return 0, nil
|
||||
}
|
||||
func (f *fakeACMERepo) GCExpiredNonces(ctx context.Context) (int64, error) {
|
||||
n := int64(0)
|
||||
for nonce, exp := range f.issued {
|
||||
if time.Now().After(exp) {
|
||||
delete(f.issued, nonce)
|
||||
n++
|
||||
}
|
||||
}
|
||||
return n, nil
|
||||
}
|
||||
func (f *fakeACMERepo) GCExpireAuthorizations(ctx context.Context) (int64, error) { return 0, nil }
|
||||
func (f *fakeACMERepo) GCInvalidateExpiredOrders(ctx context.Context) (int64, error) {
|
||||
return 0, nil
|
||||
}
|
||||
|
||||
// fakeTransactor is the repository.Transactor stand-in: runs fn
|
||||
// against the supplied querier (we just pass nil — fakes ignore it).
|
||||
|
||||
Reference in New Issue
Block a user