Bundle C: Renewal/reliability cluster — 7 findings closed

Closes M-006 + M-007 + M-008 + M-015 + M-016 + M-019 + M-020 from
comprehensive-audit-2026-04-25. M-028 was already closed by the
Bundle B CI follow-up.

M-006 (CWE-913) — Idempotent migration 000014
  migrations/000014_policy_violation_severity_check.up.sql:
    Prepended ALTER TABLE ... DROP CONSTRAINT IF EXISTS before the
    ADD. Mirrors the down migration's existing IF EXISTS shape and
    the M-7 idempotent-index idiom. Re-runs against partially-applied
    DBs now succeed.

M-007 — Bulk-op partial-failure tests (3 new)
  internal/api/handler/bulk_partial_failure_test.go:
    TestBulkRevoke_PartialFailure_ReportsBoth
    TestBulkRenew_PartialFailure_ReportsBoth
    TestBulkReassign_PartialFailure_ReportsBoth
  Each asserts HTTP 200 + both success/failure counters round-trip
  + per-cert errors[] preserved with non-empty messages so operators
  can correlate each failure to its certificate ID.

M-008 — Admin-gated handler enumeration pin (verified-already-clean)
  Recon: only one admin-gated handler — bulk_revocation.go — with
  full 3-branch test triplet already in place. health.go calls
  IsAdmin informationally to surface the flag to the GUI without
  gating.
  internal/api/handler/m008_admin_gate_test.go:
    Walks every handler .go file, asserts every middleware.IsAdmin
    call site is in AdminGatedHandlers (with required test triplet)
    or InformationalIsAdminCallers (justified). Adding a new admin
    gate without updating both the constant AND adding the test
    triplet fails CI.

M-015 — Single-profile cardinality pin (verified-already-clean)
  Audit claim 'no cardinality validation' was wrong — enforced at
  struct level. domain.ManagedCertificate.{CertificateProfileID,
  RenewalPolicyID,IssuerID,OwnerID} and RenewalPolicy.
  CertificateProfileID are bare strings, not slices.
  internal/domain/m015_cardinality_test.go:
    reflect-based pin on kind=String. Schema change to N:N would
    have to update renewal.go's lookup loop in the same commit.

M-016 (CWE-754) — Reap stale-agent jobs
  internal/repository/postgres/job.go::ListJobsWithOfflineAgents:
    JOIN jobs to agents on agent_id, filter (status=Running AND
    a.last_heartbeat_at < cutoff), exclude server-keygen jobs.
  internal/service/job.go::ReapJobsWithOfflineAgents:
    Flips matched jobs to Failed reason agent_offline so I-001
    retry loop re-queues them on a healthy agent. Records audit
    event per reap.
  internal/scheduler/scheduler.go:
    Scheduler.runJobTimeout cycle now calls both reaper arms.
    agentOfflineJobTTL default 5min (5x agent-health-check default);
    SetAgentOfflineJobTTL knob for operator override.
  internal/service/job_offline_agent_reaper_test.go: 6 unit tests
  cover happy path, server-keygen-skip, non-Running-skip, non-
  positive-TTL fail-loud, repo-error propagation, audit-event
  recording.

M-019 — Configurable ARI HTTP timeout
  Audit claim 'no fallback timeout' was wrong — ari.go:52 already
  had a 15s timeout. Bundle C makes it configurable.
  internal/connector/issuer/acme/acme.go:
    Config.ARIHTTPTimeoutSeconds field with env path
    CERTCTL_ACME_ARI_HTTP_TIMEOUT_SECONDS.
  internal/connector/issuer/acme/ari.go:
    Both HTTP clients (GetRenewalInfo + getARIEndpoint) now use the
    new ariHTTPTimeout() helper. Zero / negative / nil-config all
    fall back to the historic 15s default.
  ari_timeout_test.go: 4 dispatch arm tests.

M-020 (CWE-770) — OCSP DoS hardening
  Pre-bundle the noAuthHandler chain had no rate limit. An attacker
  could DoS the OCSP responder, which for fail-open relying parties
  is a revocation bypass.
  cmd/server/main.go:
    noAuthHandler refactored from fixed middleware.Chain(...) to a
    conditional slice that appends middleware.NewRateLimiter when
    cfg.RateLimit.Enabled. Per-IP keying applies; OCSP/CRL/EST/SCEP
    are unauth.
  docs/security.md (NEW):
    Operator runbook documenting Must-Staple TLS Feature extension
    RFC 7633 as the architectural fix for fail-open relying parties.
    Profile-flip guidance + nginx/Apache/HAProxy/Envoy stapling
    snippets + explicit scope statement on what the rate limiter
    alone does NOT solve.

Audit deliverables:
  cowork/comprehensive-audit-2026-04-25/audit-report.md: score
    31/55 -> 38/55 closed (Medium 13/27 -> 20/27).
  cowork/comprehensive-audit-2026-04-25/findings.yaml: 7 status
    flips open -> closed with closure notes citing the Bundle C
    mechanism.
  certctl/CHANGELOG.md: Bundle C section under [unreleased].

Verification:
  go vet ./internal/service ./internal/scheduler ./internal/connector/issuer/acme
    ./internal/api/handler ./internal/domain ./cmd/server     clean
  go test -count=1 -short on the same packages              all green
  helm template + helm lint                                 clean
  internal/repository/postgres setup-fail                   sandbox disk
    pressure (same on master HEAD before this branch)

internal/scheduler/scheduler.go

@@ -67,6 +67,12 @@ type CloudDiscoveryServicer interface {
 // JobReaperService defines the interface for job timeout reaping used by the scheduler.
 type JobReaperService interface {
 	ReapTimedOutJobs(ctx context.Context, csrTTL, approvalTTL time.Duration) error
+	// Bundle C / Audit M-016 (CWE-754): closes the gap left by ReapTimedOutJobs
+	// (which only handles AwaitingCSR / AwaitingApproval). Jobs in Running
+	// status whose owning agent has been silent for longer than agentTTL get
+	// transitioned to Failed with reason "agent_offline" so I-001's retry
+	// loop can re-queue them on a healthy agent.
+	ReapJobsWithOfflineAgents(ctx context.Context, agentTTL time.Duration) error
 }
 
 // Scheduler manages background jobs and periodic tasks for the certificate control plane.
@@ -97,6 +103,9 @@ type Scheduler struct {
 	healthCheckInterval           time.Duration
 	cloudDiscoveryInterval        time.Duration
 	jobTimeoutInterval            time.Duration
+	// agentOfflineJobTTL: per-tick threshold for reaping Running jobs whose
+	// owning agent has been silent. Bundle C / Audit M-016. Defaults below.
+	agentOfflineJobTTL time.Duration
 	awaitingCSRTimeout            time.Duration
 	awaitingApprovalTimeout       time.Duration
 
@@ -148,6 +157,9 @@ func NewScheduler(
 		healthCheckInterval:           60 * time.Second,
 		cloudDiscoveryInterval:        6 * time.Hour,
 		jobTimeoutInterval:            10 * time.Minute,
+		// 5 minutes is 5×agentHealthCheckInterval default of 1m; an agent
+		// must miss multiple heartbeats before its in-flight jobs are reaped.
+		agentOfflineJobTTL: 5 * time.Minute,
 	}
 }
 
@@ -233,6 +245,16 @@ func (s *Scheduler) SetJobReaperService(jr JobReaperService) {
 	s.jobReaper = jr
 }
 
+// SetAgentOfflineJobTTL sets the threshold past which a Running job whose
+// owning agent has gone silent is reaped to Failed. Bundle C / Audit M-016.
+// Zero or negative values are ignored (the default of 5 minutes is kept).
+func (s *Scheduler) SetAgentOfflineJobTTL(d time.Duration) {
+	if d <= 0 {
+		return
+	}
+	s.agentOfflineJobTTL = d
+}
+
 // SetJobTimeoutInterval sets the job timeout reaper tick interval (I-003).
 func (s *Scheduler) SetJobTimeoutInterval(d time.Duration) {
 	s.jobTimeoutInterval = d
@@ -503,6 +525,15 @@ func (s *Scheduler) jobTimeoutLoop(ctx context.Context) {
 // When no JobReaperService has been wired (e.g. in tests that don't exercise
 // I-003) the call is a safe no-op, preserving the always-on loop topology
 // described in I-003 without forcing every consumer to wire a reaper.
+//
+// Bundle C / Audit M-016: the reaping cycle now has TWO arms:
+//
+//  1. ReapTimedOutJobs handles AwaitingCSR / AwaitingApproval timeouts (I-003).
+//  2. ReapJobsWithOfflineAgents handles Running jobs whose owning agent has
+//     gone silent (M-016). Reuses the same agentHealthCheckTimeout as the
+//     mark-stale-agents-offline path for consistency: if the agent is judged
+//     offline by AgentService.MarkStaleAgentsOffline, its in-flight jobs
+//     should be reaped on the same cadence.
 func (s *Scheduler) runJobTimeout(ctx context.Context) {
 	if s.jobReaper == nil {
 		return
@@ -516,6 +547,20 @@ func (s *Scheduler) runJobTimeout(ctx context.Context) {
 	} else {
 		s.logger.Debug("job timeout reaper completed")
 	}
+	// Second arm: offline-agent reaper. Uses agentOfflineTimeout (defaults to
+	// 5 minutes — same value the agent-health-check path uses to flip an
+	// agent to Offline). A sensible default of 5×agentHealthCheckInterval
+	// catches agents that miss multiple consecutive heartbeats while leaving
+	// a single missed beat as a transient blip that does NOT reap.
+	offlineCtx, offlineCancel := context.WithTimeout(ctx, 2*time.Minute)
+	defer offlineCancel()
+	if err := s.jobReaper.ReapJobsWithOfflineAgents(offlineCtx, s.agentOfflineJobTTL); err != nil {
+		s.logger.Error("offline-agent job reaper failed",
+			"error", err,
+			"agent_offline_ttl", s.agentOfflineJobTTL.String())
+	} else {
+		s.logger.Debug("offline-agent job reaper completed")
+	}
 }
 
 // agentHealthCheckLoop runs every agentHealthCheckInterval and marks stale agents as offline.

internal/scheduler/scheduler_test.go

@@ -165,6 +165,15 @@ func (m *mockJobService) ReapTimedOutJobs(ctx context.Context, csrTTL, approvalT
 	return nil
 }
 
+// ReapJobsWithOfflineAgents is the Bundle C / Audit M-016 stub. The
+// existing scheduler tests do not exercise this path; the offline-agent
+// reaper has its own end-to-end test in internal/service. Here we just
+// satisfy the JobReaperService interface so the scheduler tests still
+// compile.
+func (m *mockJobService) ReapJobsWithOfflineAgents(ctx context.Context, agentTTL time.Duration) error {
+	return nil
+}
+
 // mockAgentService is a mock implementation for testing.
 type mockAgentService struct {
 	mu          sync.Mutex