2 Commits

Author	SHA1	Message	Date
shankar0123	264015059d	ci(guards): fix G-3 (CERTCTL_MCP_READ_ONLY phantom) + S-1 (hardcoded 45) ... Two CI guards tripped on the B4 + B5 closure commits: 1. G-3 env-docs-drift caught `CERTCTL_MCP_READ_ONLY` mentioned in docs/operator/security-bundle-5-audit-closure.md (Bundle 5 S8 row) without a corresponding entry in internal/config/config.go. The env var is a v3 idea, not a shipped feature — the doc now describes the future gate without naming the literal env var, matching the G-3 phantom-env-var contract. 2. S-1 hardcoded-source-counts caught "all 45 migrations" in docs/operator/scheduler-ha.md (Bundle 4 D8 closure prose). Per the CLAUDE.md operating rule "Numeric claims about current state rot", swapped the literal count for the rebuild command `ls migrations/*.up.sql | wc -l`. Both fixes are doc-only — no code change, no test change. The underlying Bundle 4 + Bundle 5 closures stand. Verification: bash scripts/ci-guards/G-3-env-docs-drift.sh # clean bash scripts/ci-guards/S-1-hardcoded-source-counts.sh # clean	2026-05-13 01:24:06 +00:00
shankar0123	750478a6fe	fix(scale): close BUNDLE 4 — migrations, scheduler HA, rate-limits, scale receipts ... Bundle 4 closure (2026-05-13 acquisition diligence audit). Closes the "what happens under multi-replica" question cluster: migration runner had no concurrency control + no applied-version ledger, 15 scheduler loops had per-process idempotency but no cross-replica documentation, rate limits were process-local without an operator-facing scope statement, load-test scope explicitly omitted four hot paths without linking them to a roadmap. Source findings closed: HIGH-1 + D4 + finding 4 (migration tracking) D8 (scheduler loop ownership) MED-1 + MED-2 (rate-limit scope) T9 + LOW-7 + finding 7 (load-test receipt scope) Closures by source ID: HIGH-1 + D4 + finding 4 — Migration tracking + advisory lock. internal/repository/postgres/db.go::RunMigrations now wraps every migration execution in: 1. A dedicated *sql.Conn pinned to one connection for the entire scan + apply lifecycle (pg_advisory_lock is connection-scoped). 2. pg_advisory_lock(migrationAdvisoryLockID) — fixed int64 key derived from "certctl-migrations" so the same constant resolves across deployments without colliding with operator advisory locks. Blocks the second replica until the first finishes. 3. CREATE TABLE IF NOT EXISTS schema_migrations(version TEXT PK, applied_at TIMESTAMPTZ DEFAULT NOW()) — audit ledger. 4. Skip-applied loop: SELECT version FROM schema_migrations → map[string]struct{} → skip every .up.sql whose filename is in the map. INSERT after successful execute, ON CONFLICT (version) DO NOTHING for defense in depth. Pre-Bundle-4 every server boot re-ran all 45 .up.sql files. The "idempotency via IF NOT EXISTS / ON CONFLICT" contract in CLAUDE.md held per-migration but offered no protection when two Helm replicas raced on schema DDL. Post-Bundle-4 single-replica deploys see zero behavior change beyond the audit-table population; multi-replica deploys get HA-safe schema bootstrap. D8 — Scheduler HA semantics documented. New docs/operator/scheduler-ha.md with per-loop inventory of all 15 loops in internal/scheduler/scheduler.go. Classification: - HA-safe (jobProcessorLoop, jobRetryLoop) — FOR UPDATE SKIP LOCKED via ClaimPendingJobs (Bundle 1 H-6 closure, 3e78ecb). - HA-safe-ish (jobTimeoutLoop) — atomic UPDATE-WHERE-status. - Idempotent under N>1 replicas (renewalCheckLoop, agentHealthCheckLoop, shortLivedExpiryCheckLoop, networkScanLoop, healthCheckLoop, acmeGCLoop, sessionGCLoop) — duplicate ticks produce idempotent side effects. - Side-effect-duplicating under N>1 replicas (notificationProcessLoop, notificationRetryLoop, digestLoop, cloudDiscoveryLoop, crlGenerationLoop) — duplicate webhook/email/AWS-API/CRL-signing operations. Operators running multi-replica accept N× side effects or pin to server.replicas: 1. Leader-election work tracked in WORKSPACE-ROADMAP.md as v3. MED-1 + MED-2 — Rate-limit scope. New docs/operator/rate-limit-scope.md states the contract verbatim: process-local sync.Mutex-guarded sliding-window log, effective cluster-wide cap = configured-per-replica × server.replicas, restart-safe (no persistent state, no shared store), bounded (50k/100k key cap with eviction). Five call sites documented: ocspLimiter (1m/IP), exportLimiter (1h/actor), EST per-principal (24h/CN), EST failed-auth (1h/IP), Intune dispatcher (24h/Subject+Issuer), plus the HTTP middleware token-bucket (RPS+Burst per replica). Cluster-wide shared limits via Redis or Postgres-backed bucket are tracked in WORKSPACE-ROADMAP.md as v3. T9 + LOW-7 + finding 7 — Load-test receipt scope. The existing harness at deploy/test/loadtest/ already self-documents the gap ("What it explicitly does NOT measure"). No code change needed for this finding; Bundle 4 cross-references scheduler-ha.md and rate-limit-scope.md from those gap callouts so the four deferred coverage classes (issuer connector, scheduler throughput, agent fleet, DB p99) land in the same place an acquirer reads about HA semantics and rate limits. Tests: internal/repository/postgres/migrations_test.go (new, 4 tests): - TestRunMigrations_PopulatesSchemaMigrations: audit table exists and is non-empty after the first migration run. - TestRunMigrations_SkipsAppliedOnSecondCall: second call is observable no-op on row count. - TestRunMigrations_ConcurrentCallsSerialized: two goroutines racing the migrator both return without error; row count unchanged; no duplicate versions. - TestRunMigrations_FreshDatabaseHappyPath: ≥ 30 migrations land on a fresh schema. Gated by testcontainers via the existing repo_test.go getTestDB pattern; skipped under -short. The integration lane runs them. Verification: gofmt -l # clean go vet ./internal/repository/postgres ./cmd/server # clean go build ./cmd/server ./internal/repository/postgres # clean go test -short -count=1 ./internal/repository/postgres ./internal/ratelimit # PASS Operator follow-up: full integration run on workstation: go test -count=1 ./internal/repository/postgres -run TestRunMigrations_ Receipts (paths for the audit packet): Migration runner evidence: internal/repository/postgres/db.go L135-340 (advisory-lock + ledger + skip-applied loop) + internal/repository/postgres/migrations_test.go (4 tests). Scheduler loop inventory: docs/operator/scheduler-ha.md (15-loop table with HA classification per loop). Rate-limit storage matrix: docs/operator/rate-limit-scope.md. Load-test baseline: deploy/test/loadtest/README.md (already self-documenting), cross-linked from scheduler-ha.md. Remaining operator warnings (deferred, tracked in WORKSPACE-ROADMAP.md): - Leader election for the four duplicate-side-effect loops (notificationProcessLoop, notificationRetryLoop, digestLoop, cloudDiscoveryLoop, crlGenerationLoop). v3 work item. - Shared rate-limits across replicas (Redis / Postgres token bucket). v3 work item. - Issuer-connector + scheduler-throughput + agent-fleet + DB-p99 load-test coverage. Tracked separately; per-issuer Prometheus histograms already capture issuer round-trip latency in production runs. Audit-Closes: BUNDLE-4 HIGH-1 D4 D8 MED-1 MED-2 T9 LOW-7 finding-4 finding-7	2026-05-13 01:00:39 +00:00