mirror of
https://github.com/shankar0123/certctl.git
synced 2026-06-07 15:01:32 +00:00
scheduler+db: close Phase 6 — scale hardening across pool, jitter, ETag, asyncpoll
Phase 6 of the certctl architecture diligence remediation. Five
findings across the same scheduler-and-DB-pool surface.
SCALE-M1 (Med) — DB pool default bumped 25 → 50
internal/config/config.go line 1972:
MaxConnections: getEnvInt("CERTCTL_DATABASE_MAX_CONNS", 50)
Postgres default max_connections is 100; 50 leaves headroom for
pg_dump + ad-hoc psql + a server replica without exhausting the
DB-side cap. Operator override env var unchanged. Operator-tune
ladder for larger fleets (5K / 50K certs) lives in
docs/operator/scale.md as starter values pending Phase 8 load
tests — explicitly marked TBD.
SCALE-M3 (Med) — async-CA poll budget operator-configurable
Live state was partially-already-shipped: all 4 async-CA
connectors (digicert, entrust, globalsign, sectigo) already have
per-connector CERTCTL_<NAME>_POLL_MAX_WAIT_SECONDS (Audit fix #5
closed pre-Phase-6). What was missing: a global package-default
override. Shipped:
- internal/connector/issuer/asyncpoll/asyncpoll.go gains
SetDefaultMaxWait(d) + effectiveDefaultMaxWait var + the
currentDefaultMaxWait() priority resolver.
- cmd/server/main.go reads CERTCTL_ASYNC_POLL_MAX_WAIT_SECONDS
at boot and calls SetDefaultMaxWait.
- deploy/ENVIRONMENTS.md documents the new env var (G-3 guard
green).
Naming deviation from the prompt's CERTCTL_ASYNC_POLL_MAX_ATTEMPTS:
the live code tracks wall-clock time (MaxWait), not attempt count.
Matched the existing per-connector nomenclature (_POLL_MAX_WAIT_SECONDS)
so the priority chain reads naturally.
SCALE-M5 (Med) — JitteredTicker wrapper for all 15 scheduler loops
internal/scheduler/jitter.go ships NewJitteredTicker(interval,
jitterPct) + DefaultSchedulerJitter (±10%). All 15 sites in
internal/scheduler/scheduler.go migrated from bare time.NewTicker
to NewJitteredTicker(interval, DefaultSchedulerJitter). Base
intervals unchanged; only the per-tick envelope adds ±10%
randomized delay so multiple loops with the same nominal cadence
don't co-fire and spike CPU + DB at wall-clock boundaries.
internal/scheduler/jitter_test.go pins:
- Bounded envelope (each tick within ±jitterPct of interval)
- Mean drift < 30% of nominal (sign-bug detector)
- Stop() releases the goroutine + closes C
- Stop() idempotent (no panic on repeat)
- Zero-jitter behaves like time.NewTicker
- Negative and >=1 jitterPct values clamped defensively
CI guard scripts/ci-guards/no-bare-newticker-in-scheduler.sh blocks
any future bare time.NewTicker in scheduler.go.
SCALE-L1 (Low) — renewal-sweep semaphore behavior documented
docs/operator/scale.md "Scheduler tick budgets" section explains
the per-tick concurrency semaphore (CERTCTL_RENEWAL_CONCURRENCY=25
default), the ctx-cancellation drain on tick-budget overrun, and
operator tuning advice (raise concurrency + DB pool together).
No code change — the behavior is defensible as-is per the audit.
SCALE-L2 (Low) — ETag middleware for top-5 read endpoints
internal/api/middleware/etag.go computes SHA-256 ETag over the
buffered response body, respects If-None-Match, short-circuits
to 304 Not Modified on match. GET/HEAD only; non-2xx responses
pass through unchanged. 64 KiB buffer cap degrades gracefully on
oversized responses (no caching, body still flushes intact).
Wired around the top-5 read endpoints via etagged() helper in
internal/api/router/router.go:
GET /api/v1/certificates
GET /api/v1/agents
GET /api/v1/jobs
GET /api/v1/audit
GET /api/v1/discovered-certificates
internal/api/middleware/etag_test.go pins 11 behaviors including
304-on-repeat, 200-after-mutation-with-new-ETag, POST bypass,
4xx/5xx pass-through, oversized-response degradation, wildcard
match, HEAD-treated-like-GET, byte-equal pass-through.
Cross-cutting fixes:
- internal/config/config_test.go::TestLoad_DefaultValues updated
to assert the new 50 default (was 25).
- deploy/helm/certctl/values.yaml comment corrected — agent
pollInterval is hardcoded 30s, not env-configurable; the
Phase 4 comment mistakenly referenced CERTCTL_AGENT_POLL_INTERVAL
which G-3 caught as a phantom env var.
- asyncpoll.go reformatted by gofmt; functionally unchanged.
Verification (all pass):
grep -nE 'SetMaxOpenConns' internal/repository/postgres/db.go # finds 1 site
grep -nE 'CERTCTL_DATABASE_MAX_CONNS.*50' internal/config/config.go # config default is 50
grep -rnE 'CERTCTL_ASYNC_POLL_MAX_WAIT_SECONDS' internal/ deploy/ENVIRONMENTS.md # wired
grep -cE 'time\.NewTicker\(' internal/scheduler/scheduler.go # 0 (all migrated)
grep -cE 'JitteredTicker' internal/scheduler/scheduler.go # 15
ls internal/scheduler/jitter.go internal/api/middleware/etag.go # both exist
ls docs/operator/scale.md # exists
bash scripts/ci-guards/no-bare-newticker-in-scheduler.sh # clean
bash scripts/ci-guards/G-3-env-docs-drift.sh # clean
go test ./internal/scheduler/ ./internal/api/middleware/ \
./internal/connector/issuer/asyncpoll/ ./internal/config/ # 4/4 packages green
Closes: cowork/certctl-architecture-diligence-audit.html#fix-SCALE-M1
cowork/certctl-architecture-diligence-audit.html#fix-SCALE-M3
cowork/certctl-architecture-diligence-audit.html#fix-SCALE-M5
cowork/certctl-architecture-diligence-audit.html#fix-SCALE-L1
cowork/certctl-architecture-diligence-audit.html#fix-SCALE-L2
This commit is contained in:
@@ -112,6 +112,49 @@ const (
|
||||
DefaultJitterPct = 0.2
|
||||
)
|
||||
|
||||
// Phase 6 SCALE-M3 closure (2026-05-14): operator-overridable global
|
||||
// default for the package-level MaxWait fallback. Priority chain for
|
||||
// every Poll() call:
|
||||
//
|
||||
// 1. cfg.MaxWait > 0 → per-call value (set by the caller, usually
|
||||
// from a per-connector env like CERTCTL_DIGICERT_POLL_MAX_WAIT_SECONDS)
|
||||
// 2. effectiveDefaultMaxWait != nil → process-wide override set via
|
||||
// SetDefaultMaxWait (from CERTCTL_ASYNC_POLL_MAX_WAIT_SECONDS at
|
||||
// server boot)
|
||||
// 3. DefaultMaxWait constant (10 minutes)
|
||||
//
|
||||
// Pre-Phase-6, paths (1) + (3) existed. Path (2) lets an operator tune
|
||||
// the global fallback in one place without setting four per-connector
|
||||
// envs (digicert, entrust, globalsign, sectigo).
|
||||
var effectiveDefaultMaxWait *time.Duration
|
||||
|
||||
// SetDefaultMaxWait overrides the package-level DefaultMaxWait
|
||||
// fallback for the rest of the process lifetime. Intended to be
|
||||
// called exactly once at boot from cmd/server/main.go after reading
|
||||
// CERTCTL_ASYNC_POLL_MAX_WAIT_SECONDS. Subsequent calls overwrite the
|
||||
// previous override. A zero or negative duration clears the override
|
||||
// (restoring the constant default).
|
||||
//
|
||||
// Per-connector overrides (caller-provided cfg.MaxWait) take
|
||||
// precedence over this global default.
|
||||
func SetDefaultMaxWait(d time.Duration) {
|
||||
if d <= 0 {
|
||||
effectiveDefaultMaxWait = nil
|
||||
return
|
||||
}
|
||||
effectiveDefaultMaxWait = &d
|
||||
}
|
||||
|
||||
// currentDefaultMaxWait returns the effective default — the
|
||||
// SetDefaultMaxWait override if one is in place, else the package's
|
||||
// DefaultMaxWait constant.
|
||||
func currentDefaultMaxWait() time.Duration {
|
||||
if effectiveDefaultMaxWait != nil {
|
||||
return *effectiveDefaultMaxWait
|
||||
}
|
||||
return DefaultMaxWait
|
||||
}
|
||||
|
||||
// Poll runs fn with exponential backoff + jitter until Done, Failed,
|
||||
// MaxWait, or ctx cancellation.
|
||||
//
|
||||
@@ -132,7 +175,7 @@ const (
|
||||
// error in case MaxWait or ctx-cancel later fires.
|
||||
func Poll(ctx context.Context, cfg Config, fn PollFunc) (Result, error) {
|
||||
if cfg.MaxWait <= 0 {
|
||||
cfg.MaxWait = DefaultMaxWait
|
||||
cfg.MaxWait = currentDefaultMaxWait()
|
||||
}
|
||||
if cfg.InitialWait <= 0 {
|
||||
cfg.InitialWait = DefaultInitialWait
|
||||
|
||||
Reference in New Issue
Block a user