Close I-004 (agent hard-delete cascades targets) coverage-gap finding

Operator decision answered as full soft-delete with optional forced
cascade — hard-delete is not reachable from any public surface. Prior
to this commit, DELETE /agents/{id} ran a plain `DELETE FROM agents`
whose schema-level `ON DELETE CASCADE` on deployment_targets.agent_id
silently wiped every target, orphaning certs and aborting in-flight
jobs. The finding closure reshapes the agent-removal contract around
soft retirement with explicit preflight counts, an opt-in cascade
gated by a mandatory reason, and unconditional protection for the
four reserved sentinel agents used by discovery sources.

Schema — migration 000015:
  migrations/000015_agent_retire.up.sql flips
  deployment_targets_agent_id_fkey from ON DELETE CASCADE to ON DELETE
  RESTRICT, so a stray `DELETE FROM agents` now errors at the DB
  boundary instead of quietly destroying targets. Both `agents` and
  `deployment_targets` grow a retired_at TIMESTAMPTZ + retired_reason
  TEXT pair (TEXT not VARCHAR so operator comments are never
  truncated), indexed via partial indexes WHERE retired_at IS NOT
  NULL. The migration is self-healing (ADD COLUMN IF NOT EXISTS, DROP
  CONSTRAINT IF EXISTS then ADD CONSTRAINT, CREATE INDEX IF NOT
  EXISTS) so repeated runs against partially-migrated databases
  converge. migrations/000015_agent_retire.down.sql restores CASCADE
  and drops the new columns for clean rollback. A dedicated
  repository-layer testcontainers test
  (internal/repository/postgres/migration_000015_test.go) asserts the
  before/after FK action, column presence, index presence, and
  round-trip idempotency under up→down→up.

Domain — sentinel guard + dependency counts:
  internal/domain/connector.go gains IsRetired() on Agent, the
  exported SentinelAgentIDs slice listing server-scanner,
  cloud-aws-sm, cloud-azure-kv, cloud-gcp-sm verbatim (matching the
  four reserved IDs documented in CLAUDE.md and created at startup in
  cmd/server/main.go), IsSentinelAgent(id string) predicate,
  AgentDependencyCounts{ActiveTargets, ActiveCertificates,
  PendingJobs} with a HasDependencies() method, and ActorTypeAgent /
  ActorTypeSystem enum values used by audit emission downstream.
  Coverage locked down by internal/domain/connector_test.go.

Service — 8-step ordered contract:
  internal/service/agent_retire.go:RetireAgent(ctx, id, actor,
  opts{Force, Reason}) enforces a fixed execution order:
  (1) sentinel guard — IsSentinelAgent(id) returns ErrAgentIsSentinel
      unconditionally; force=true does NOT bypass it.
  (2) fetch — ErrAgentNotFound on miss.
  (3) idempotency — if IsRetired() already, return
      AgentRetirementResult{AlreadyRetired: true} with no new audit
      event and no state change (safe to replay from flaky clients).
  (4) preflight counts — collectAgentDependencyCounts runs
      ActiveTargets, ActiveCertificates, PendingJobs sequentially
      (not in parallel; keeps the per-query timeout predictable and
      matches the repo's existing call-chain shape).
  (5) force-reason guard — opts.Force=true with empty Reason returns
      ErrForceReasonRequired (wired into the 400 status surface).
  (6) dependency guard — HasDependencies() with opts.Force=false
      returns BlockedByDependenciesError{Counts} (wired into the 409
      body with per-bucket counts).
  (7) mutation — single pinned retiredAt := time.Now(); agent
      retirement first, then cascade target retirement if opts.Force,
      all under the repo's single transaction so the two retired_at
      stamps match to the second.
  (8) best-effort audit — agent_retired always; agent_retirement_
      cascaded additionally on the force path. Actor is whatever the
      handler resolves from the request; actor type is mapped by
      resolveActorType (system/agent-prefix→Agent/else→User). Audit
      emission failures are logged via slog.Error but do not abort
      the retirement (matches the house convention used by every
      other scheduler-emitted event).

  BlockedByDependenciesError implements Error() as
  "active_targets=%d, active_certificates=%d, pending_jobs=%d" and
  Unwrap() → ErrBlockedByDependencies. The single struct satisfies
  errors.Is via Unwrap (used by scheduler-level tests) and errors.As
  via the concrete type (used by the handler to fish out Counts for
  the 409 body). ListRetiredAgents(page, perPage) adds a separate
  paginated accessor with page<1→1 and perPage<1→50 normalization so
  retired rows are queryable without polluting the default agent
  listing.

  Sentinel guard coverage is asymmetric by design: all four reserved
  IDs are protected, and force=true cannot override. Regression tests
  in internal/service/agent_retire_test.go assert each of the eight
  steps in order, plus sentinel bypass attempts and idempotency
  replay.

Handler + router — status-code surface:
  internal/api/handler/agents.go:RetireAgent exposes seven status
  codes on DELETE /agents/{id}:
    200 on a fresh retirement (body echoes AgentRetirementResult).
    204 on idempotent replay (AlreadyRetired=true; no new audit).
    400 on ErrForceReasonRequired.
    403 on ErrAgentIsSentinel.
    404 on ErrAgentNotFound.
    409 on BlockedByDependenciesError, with a custom body shape
        {error, counts{active_targets, active_certificates,
        pending_jobs}} that bypasses the default ErrorWithRequestID
        envelope so callers get the per-bucket numbers directly.
    500 on any other error.
  Heartbeat HandleHeartbeat returns 410 Gone when the agent is
  retired (ErrAgentRetired), signalling the agent to shut down.
  Query params `force=true` and `reason=<text>` drive the cascade
  path; both are forwarded as url.Values through the new MCP
  transport.

  internal/api/router/router.go registers GET /api/v1/agents/retired
  literal-path BEFORE /api/v1/agents/{id} — Go 1.22 ServeMux's
  literal-beats-pattern-var precedence routes "retired" to the
  paginated retired-agents listing instead of fetching a hypothetical
  agent named "retired".

Agent binary — clean shutdown on 410:
  cmd/agent/main.go gains the ErrAgentRetired sentinel, a
  retiredOnce sync.Once, and a retiredSignal chan struct{}. A
  markRetired(source, statusCode, body) helper closes the channel
  exactly once; the Run() select loop observes the close and returns
  ErrAgentRetired; main() matches via errors.Is(err, ErrAgentRetired)
  and exits cleanly instead of spinning in the heartbeat retry loop.
  The 410 Gone surface is therefore terminal for the agent process.

MCP transport:
  internal/mcp/client.go adds Client.DeleteWithQuery(path, query),
  a new additive transport method. Client.Delete is path-only; without
  this method the retire tool would silently drop `force` and `reason`,
  turning every cascade retire into a default soft-retire. The new
  method shares do()'s 204 normalization and 4xx/5xx error
  propagation so tool authors get one contract.
  internal/mcp/tools.go + internal/mcp/types.go expose the
  retire_agent tool with Force+Reason inputs wired through
  DeleteWithQuery.

CLI:
  cmd/cli/main.go + internal/cli/client.go add two CLI surfaces:
  `agents list --retired` (client-side strip of --retired then
  delegation to ListRetiredAgents, sharing --page/--per-page parsing
  with the default listing) and `agents retire <id> [--force --reason
  "…"]` (mirrors ErrForceReasonRequired — force without reason is
  rejected client-side before the request is sent). JSON + table
  output modes both honor the new columns.

Frontend:
  web/src/pages/AgentsPage.tsx surfaces retired/retire affordances.
  web/src/api/client.ts + web/src/api/types.ts expose the retire
  endpoint and the retired-listing. 4 new Vitest regression cases.

OpenAPI:
  api/openapi.yaml documents DELETE /agents/{id} with all seven
  status codes, 410 on heartbeat, and the 409 per-bucket body shape.

Regression coverage (six new test files, all green):
  internal/service/agent_retire_test.go           — 8-step contract + sentinel guards
  internal/api/handler/agent_retire_handler_test.go — 7-status-code surface + 410 heartbeat
  internal/mcp/retire_agent_test.go               — DeleteWithQuery wire-through
  internal/cli/agent_retire_test.go               — --retired listing + --force/--reason pairing
  internal/repository/postgres/migration_000015_test.go — FK flip + columns + indexes + up↔down
  internal/domain/connector_test.go               — IsRetired, IsSentinelAgent, SentinelAgentIDs, HasDependencies

Files:
  api/openapi.yaml                                — DELETE + 410 + 409 body shape
  cmd/agent/main.go                               — ErrAgentRetired, markRetired, retiredSignal
  cmd/cli/main.go                                 — handleAgents list/get/retire dispatch
  docs/architecture.md, docs/concepts.md,
    docs/testing-guide.md                         — retirement contract narrative
  internal/api/handler/agents.go                  — RetireAgent, status surface, 410 on heartbeat
  internal/api/handler/agent_handler_test.go      — extended coverage
  internal/api/handler/agent_retire_handler_test.go — new
  internal/api/router/router.go                   — /agents/retired before /agents/{id}
  internal/cli/agent_retire_test.go               — new
  internal/cli/client.go                          — ListRetiredAgents + RetireAgent
  internal/domain/connector.go                    — IsRetired, SentinelAgentIDs,
                                                    IsSentinelAgent, AgentDependencyCounts,
                                                    ActorTypeAgent/System
  internal/domain/connector_test.go               — new
  internal/integration/lifecycle_test.go          — retirement fixture
  internal/mcp/client.go                          — DeleteWithQuery additive transport
  internal/mcp/retire_agent_test.go               — new
  internal/mcp/tools.go, internal/mcp/types.go    — retire_agent tool + Force/Reason inputs
  internal/repository/interfaces.go               — AgentRepository retirement methods
  internal/repository/postgres/agent.go           — retire + cascade target retire + counts
  internal/repository/postgres/migration_000015_test.go — new
  internal/service/agent.go                       — wire into AgentService surface
  internal/service/agent_retire.go                — new 8-step contract
  internal/service/agent_retire_test.go           — new
  internal/service/deployment.go                  — skip retired agents
  internal/service/target.go                      — skip retired agents
  internal/service/testutil_test.go               — shared mocks extended
  migrations/000015_agent_retire.up.sql           — new
  migrations/000015_agent_retire.down.sql         — new
  web/src/api/client.ts, types.ts + tests         — retire endpoint wiring
  web/src/pages/AgentsPage.tsx                    — retire UI

cmd/agent/main.go

@@ -12,6 +12,7 @@ import (
 	"crypto/x509/pkix"
 	"encoding/json"
 	"encoding/pem"
+	"errors"
 	"flag"
 	"fmt"
 	"io"
@@ -23,6 +24,7 @@ import (
 	"path/filepath"
 	"runtime"
 	"strings"
+	"sync"
 	"syscall"
 	"time"
 
@@ -53,6 +55,16 @@ type AgentConfig struct {
 	DiscoveryDirs []string // Directories to scan for certificates (comma-separated via env)
 }
 
+// ErrAgentRetired is the sentinel returned by [Agent.Run] when the control
+// plane responds with HTTP 410 Gone to a heartbeat or work-poll request — the
+// canonical signal that this agent's row has been soft-retired server-side
+// (see I-004 in cowork/certctl-coverage-gap-audit.md). The binary must
+// terminate cleanly: an init-system restart would only produce another 410
+// and wedge the host in a restart loop. main() translates this sentinel into
+// a zero exit code so systemd (Restart=on-failure) and launchd do not respawn
+// the process. Do not wrap this error — main() matches it with errors.Is.
+var ErrAgentRetired = fmt.Errorf("agent retired by control plane")
+
 // Agent represents the local agent that runs on target servers.
 // It periodically sends heartbeats, polls for work, executes deployment and CSR jobs,
 // and scans configured directories for existing certificates.
@@ -68,6 +80,17 @@ type Agent struct {
 	pollInterval          time.Duration
 	discoveryInterval     time.Duration
 	consecutiveFailures   int
+
+	// I-004: terminal retirement signal. retiredSignal is closed exactly once
+	// (guarded by retiredOnce) when either sendHeartbeat or pollForWork
+	// observes HTTP 410 Gone. The Run() select loop picks up the close and
+	// returns ErrAgentRetired, unwinding the goroutine cleanly so main() can
+	// log + exit(0). Using a channel + sync.Once (rather than an atomic bool
+	// + polling) lets us fall through the select statement immediately instead
+	// of waiting for the next ticker; the zero-allocation close is safe to
+	// race with ctx.Done() and other cases.
+	retiredOnce   sync.Once
+	retiredSignal chan struct{}
 }
 
 // WorkResponse represents the response from the work polling endpoint.
@@ -98,9 +121,31 @@ func NewAgent(cfg *AgentConfig, logger *slog.Logger) *Agent {
 		heartbeatInterval: 60 * time.Second,
 		pollInterval:      30 * time.Second,
 		discoveryInterval: 6 * time.Hour, // scan for certs every 6 hours
+		retiredSignal:     make(chan struct{}),
 	}
 }
 
+// markRetired records that the control plane has declared this agent retired
+// (HTTP 410 Gone on heartbeat or work poll). Idempotent via sync.Once — if
+// both the heartbeat and work-poll paths observe 410 in the same tick, only
+// the first close() runs and we avoid a runtime panic. Emits an ERROR-level
+// log line so init-system journaling captures it prominently, and includes
+// the source (heartbeat/work_poll), response body, and status code so the
+// operator can verify it's a genuine retirement signal rather than a
+// misrouted request. After this returns, the select-loop case in Run()
+// observes the closed channel on its next iteration and returns
+// ErrAgentRetired.
+func (a *Agent) markRetired(source string, statusCode int, body string) {
+	a.retiredOnce.Do(func() {
+		a.logger.Error("agent has been retired by control plane — shutting down",
+			"source", source,
+			"status", statusCode,
+			"body", body,
+			"agent_id", a.config.AgentID)
+		close(a.retiredSignal)
+	})
+}
+
 // Run starts the agent's main loop.
 // It sends heartbeats, polls for work, and handles graceful shutdown via context cancellation.
 func (a *Agent) Run(ctx context.Context) error {
@@ -154,6 +199,19 @@ func (a *Agent) Run(ctx context.Context) error {
 			a.logger.Info("agent shutting down", "reason", ctx.Err())
 			return ctx.Err()
 
+		// I-004: retiredSignal is closed exactly once (via markRetired's
+		// sync.Once) when either sendHeartbeat or pollForWork observes HTTP 410
+		// Gone from the control plane. Falling through this case immediately
+		// (rather than waiting for the next ticker) lets the agent shut down
+		// quickly once retirement is confirmed — every extra heartbeat against a
+		// retired row is wasted work and noise in the audit trail. Returning
+		// ErrAgentRetired propagates up to main(), which matches it with
+		// errors.Is and exits(0) so systemd/launchd do not respawn the process.
+		case <-a.retiredSignal:
+			a.logger.Info("agent retired signal received — exiting event loop",
+				"agent_id", a.config.AgentID)
+			return ErrAgentRetired
+
 		case <-heartbeatTicker.C:
 			a.sendHeartbeat(ctx)
 
@@ -209,6 +267,22 @@ func (a *Agent) sendHeartbeat(ctx context.Context) {
 	}
 	defer resp.Body.Close()
 
+	// I-004: HTTP 410 Gone is the terminal signal from the control plane that
+	// this agent's row has been soft-retired (see internal/api/handler/agent.go
+	// heartbeat path + AgentRetirementService). Treat it separately from the
+	// generic non-200 error branch: record the event to markRetired (which closes
+	// retiredSignal exactly once via sync.Once) and return without bumping
+	// consecutiveFailures — this is not a transient failure, it's a clean
+	// shutdown. The Run() select loop picks up the closed channel on its next
+	// iteration and returns ErrAgentRetired, which main() translates into an
+	// exit(0) so systemd/launchd don't respawn the process into another 410
+	// loop.
+	if resp.StatusCode == http.StatusGone {
+		body, _ := io.ReadAll(resp.Body)
+		a.markRetired("heartbeat", resp.StatusCode, string(body))
+		return
+	}
+
 	if resp.StatusCode != http.StatusOK {
 		body, _ := io.ReadAll(resp.Body)
 		a.logger.Error("heartbeat rejected",
@@ -237,6 +311,19 @@ func (a *Agent) pollForWork(ctx context.Context) {
 	}
 	defer resp.Body.Close()
 
+	// I-004: same terminal-retirement handling as sendHeartbeat. Work-poll is the
+	// other hot path that can observe an agent's soft-retirement; if the
+	// heartbeat tick happens to fire after a work-poll tick within the same
+	// retirement window, this branch catches it first. markRetired's sync.Once
+	// guards idempotency so racing both paths in the same tick only closes the
+	// signal channel once. No consecutiveFailures increment — retirement is
+	// not a transient failure.
+	if resp.StatusCode == http.StatusGone {
+		body, _ := io.ReadAll(resp.Body)
+		a.markRetired("work_poll", resp.StatusCode, string(body))
+		return
+	}
+
 	if resp.StatusCode != http.StatusOK {
 		body, _ := io.ReadAll(resp.Body)
 		a.logger.Error("work poll rejected",
@@ -1117,6 +1204,19 @@ func main() {
 		cancel()
 		<-errChan
 	case err := <-errChan:
+		// I-004: ErrAgentRetired is a terminal, *clean* shutdown — the control
+		// plane responded HTTP 410 Gone on heartbeat/work-poll, meaning this
+		// agent's row has been soft-retired and will never be reachable again.
+		// Exit 0 so systemd's Restart=on-failure and launchd's KeepAlive do NOT
+		// respawn the process into another 410 loop (which would wedge the host
+		// and spam the control plane). Operators can observe the retirement via
+		// audit_events or the AgentsPage retired tab; the terminal log line on
+		// the way out is enough for post-mortem forensics.
+		if errors.Is(err, ErrAgentRetired) {
+			logger.Info("agent retired by control plane — exiting without restart",
+				"agent_id", agentCfg.AgentID)
+			return
+		}
 		if err != context.Canceled {
 			logger.Error("agent error", "error", err)
 			os.Exit(1)