mirror of
https://github.com/shankar0123/certctl.git
synced 2026-06-07 15:51:30 +00:00
shankar0123
bee47f0318
Closes the production-readiness loop on the ACME surface. After this
commit, certctl ships per-account rate limits + a GC sweeper for
expired ACME state + a kind-driven cert-manager 1.15 integration test
+ a lego-driven RFC conformance harness + a k6 loadtest scenario for
the unauthenticated ACME path.
Architecture:
- Rate limits live in-memory + per-replica. Restart wipes the
counters; orders/hour caps are eventual-consistency anyway. A
3-replica certctl-server fleet behind an LB effectively has 3x
the configured throughput per account; persistent rate limiting
is a follow-up if production telemetry shows abuse patterns we
can't catch in a single restart cycle. Per-key + per-action
isolation: ActionNewOrder/acc-1, ActionKeyChange/acc-1, and
ActionChallengeRespond/<challenge-id> are independent buckets.
- GC loop follows the existing scheduler-loop pattern (atomic.Bool
+ sync.WaitGroup; see crlGenerationLoop for shape). Three
independent SQL sweeps per tick (DELETE expired nonces; UPDATE
pending authzs whose expires_at < now() to expired; UPDATE
pending/ready/processing orders whose expires_at < now() to
invalid). Each sweep is a single statement; failures are logged-
and-continued so a failing nonces sweep doesn't block authzs.
Per-sweep 1m timeout bounds a stuck Postgres.
- cert-manager integration test is gated on KIND_AVAILABLE so CI
skips it cleanly (kind is too heavy for per-PR). Operators run
locally via 'make acme-cert-manager-test'; the harness brings up
a fresh cluster each run + tears it down on Cleanup.
- lego conformance harness drives a real ACME client through
register → run → cert-PEM-landed against a hermetic certctl
stack. Catches RFC-shape regressions third-party clients would
hit before they ship.
- k6 ACME-flow scenario hammers the unauthenticated surface
(directory + new-nonce + ARI synthetic-id) at 100 VUs × 5m. JWS-
signed flows are out of scope for k6 (no JWS support); they're
covered by the lego harness above.
What ships:
- internal/api/acme/ratelimit.go (+ ratelimit_test.go: 7 cases —
disable-when-perHour-zero, capacity, per-key isolation, per-
action isolation, refill-over-time, RetryAfter, concurrent-access
with -race + 200 goroutines × 200 calls).
- internal/repository/postgres/acme.go: 4 new methods —
CountActiveOrdersByAccount + GCExpiredNonces + GCExpireAuthorizations
+ GCInvalidateExpiredOrders. Each a single SQL statement.
- internal/service/acme.go: SetRateLimiter + GarbageCollect +
rate-limit gates at 3 entry points (CreateOrder + RotateAccountKey
+ RespondToChallenge) + concurrent-orders gate at CreateOrder.
2 new sentinels (ErrACMERateLimited, ErrACMEConcurrentOrdersExceeded);
5 new GC metrics (gc_runs / gc_run_failures / gc_nonces_reaped /
gc_authzs_expired / gc_orders_invalidated).
- internal/scheduler/scheduler.go: ACMEGarbageCollector interface +
acmeGCRunning atomic.Bool + acmeGCInterval + 2 setters (SetACME-
GarbageCollector + SetACMEGCInterval) + acmeGCLoop following the
crlGenerationLoop shape.
- internal/api/handler/acme.go: writeServiceError gains rateLimited
(429 + RFC 8555 §6.7) + concurrent-orders-exceeded mappings.
- internal/config/config.go: 5 new env vars
(CERTCTL_ACME_SERVER_RATE_LIMIT_ORDERS_PER_HOUR=100,
CERTCTL_ACME_SERVER_RATE_LIMIT_CONCURRENT_ORDERS=5,
CERTCTL_ACME_SERVER_RATE_LIMIT_KEY_CHANGE_PER_HOUR=5,
CERTCTL_ACME_SERVER_RATE_LIMIT_CHALLENGE_RESPONDS_PER_HOUR=60,
CERTCTL_ACME_SERVER_GC_INTERVAL=1m).
- cmd/server/main.go: NewRateLimiter() + SetRateLimiter() at
startup; conditional SetACMEGarbageCollector(acmeService) +
SetACMEGCInterval(cfg.ACMEServer.GCInterval) when Enabled+
GCInterval > 0.
- deploy/test/acme-integration/: kind-config.yaml + cert-manager-
install.sh + clusterissuer-trust-authenticated.yaml +
clusterissuer-challenge.yaml + certificate-test.yaml + conformance-
lego.sh + certmanager_test.go (//go:build integration + KIND_AVAILABLE
gate).
- deploy/test/loadtest/k6/acme_flow.js + README ACME-flows section.
- Makefile: 2 new PHONY targets (acme-cert-manager-test +
acme-rfc-conformance-test).
- docs/acme-server.md: status flipped to Phase 5; Configuration
table grows 5 rows; new 'Phase 5 — operational guidance' section
explaining rate-limit math + GC sweeper semantics + cert-manager
integration + lego conformance + k6 baseline.
Tests:
- 'go vet ./...' clean across the repo.
- 'go test -short -count=1 ./internal/...' green across every
affected package (service / acme / handler / scheduler / repo /
config).
- 'go vet -tags=integration ./deploy/test/acme-integration/' clean
(the integration test compiles cleanly with the build tag).
- The kind/cert-manager harness is gated behind KIND_AVAILABLE so
CI skips by default; operators run locally via 'make acme-cert-
manager-test'.
Engineering history: cowork/WORKSPACE-CHANGELOG.md 'ACME-Server-5'.
81 lines
2.7 KiB
JavaScript
81 lines
2.7 KiB
JavaScript
// Phase 5 — k6 scenario for the ACME issuance loop. Each VU executes
|
|
// directory + new-nonce + new-account + new-order + finalize + cert
|
|
// download against an operator-provided certctl-server. Per-step
|
|
// duration histograms feed the baseline numbers in
|
|
// deploy/test/loadtest/README.md (ACME flows section).
|
|
//
|
|
// Default scenario: 100 concurrent VUs for 5 minutes. Override via
|
|
// K6_VUS / K6_DURATION env vars.
|
|
//
|
|
// Note on signing: this scenario runs as a *load* generator, not as a
|
|
// JWS-signing client. It exercises the unauthenticated surface
|
|
// (directory + new-nonce + GET renewal-info) and validates that the
|
|
// server holds throughput under concurrency. JWS-signed flow load is
|
|
// a follow-up that requires bundling lego or a dedicated Go driver
|
|
// inside the k6 binary — k6 itself doesn't ship JWS.
|
|
|
|
import http from "k6/http";
|
|
import { check, sleep } from "k6";
|
|
import { Trend } from "k6/metrics";
|
|
|
|
const directoryURL =
|
|
__ENV.CERTCTL_ACME_DIRECTORY ||
|
|
"https://certctl:8443/acme/profile/prof-test/directory";
|
|
|
|
export const options = {
|
|
scenarios: {
|
|
acme_directory_and_nonce: {
|
|
executor: "constant-vus",
|
|
vus: parseInt(__ENV.K6_VUS || "100", 10),
|
|
duration: __ENV.K6_DURATION || "5m",
|
|
gracefulStop: "30s",
|
|
},
|
|
},
|
|
insecureSkipTLSVerify: true, // self-signed bootstrap cert
|
|
thresholds: {
|
|
"directory_duration": ["p(95)<500"],
|
|
"new_nonce_duration": ["p(95)<300"],
|
|
"renewal_info_duration": ["p(95)<800"],
|
|
"http_req_failed": ["rate<0.01"],
|
|
},
|
|
};
|
|
|
|
const directoryDuration = new Trend("directory_duration", true);
|
|
const newNonceDuration = new Trend("new_nonce_duration", true);
|
|
const renewalInfoDuration = new Trend("renewal_info_duration", true);
|
|
|
|
export default function () {
|
|
// Step 1 — directory.
|
|
let res = http.get(directoryURL);
|
|
directoryDuration.add(res.timings.duration);
|
|
check(res, { "directory 200": (r) => r.status === 200 });
|
|
|
|
if (res.status !== 200) return;
|
|
const dir = res.json();
|
|
|
|
// Step 2 — new-nonce.
|
|
if (dir.newNonce) {
|
|
res = http.head(dir.newNonce);
|
|
newNonceDuration.add(res.timings.duration);
|
|
check(res, {
|
|
"new-nonce 200 + Replay-Nonce": (r) =>
|
|
r.status === 200 && !!r.headers["Replay-Nonce"],
|
|
});
|
|
}
|
|
|
|
// Step 3 — ARI smoke (with a deliberately-malformed cert-id to
|
|
// exercise the error path; full happy-path needs a real cert which
|
|
// requires JWS signing — out of scope for this baseline scenario).
|
|
if (dir.renewalInfo) {
|
|
res = http.get(dir.renewalInfo + "/" + "aaaa.bbbb");
|
|
renewalInfoDuration.add(res.timings.duration);
|
|
// 400 (malformed cert-id, expected) OR 404 (cert not found).
|
|
check(res, {
|
|
"renewal-info 4xx for synthetic cert-id": (r) =>
|
|
r.status === 400 || r.status === 404,
|
|
});
|
|
}
|
|
|
|
sleep(1);
|
|
}
|