feat: M18b Filesystem Certificate Discovery — agent scanning, server dedup, triage API

Agent-side: - Filesystem scanner walks configured directories (CERTCTL_DISCOVERY_DIRS) - Parses PEM (.pem, .crt, .cer, .cert) and DER (.der) certificate files - Extracts CN, SANs, serial, issuer/subject DN, validity, key info, SHA-256 fingerprint - Reports discoveries to control plane on startup + every 6 hours - Skips files >1MB and private key files Server-side: - Migration 000006: discovered_certificates + discovery_scans tables - Domain model: DiscoveredCertificate, DiscoveryScan, DiscoveryReport - Three triage states: Unmanaged, Managed (claimed), Dismissed - Repository with upsert dedup (fingerprint + agent + path) - Service layer: process reports, claim, dismiss, list, summary - 7 new API endpoints (84 total): POST /agents/{id}/discoveries, GET /discovered-certificates, GET /discovered-certificates/{id}, POST .../claim, POST .../dismiss, GET /discovery-scans, GET /discovery-summary - Audit trail: scan_completed, cert_claimed, cert_dismissed events Tests: 28 new test functions (domain, handler, service layers) Docs: README, quickstart, demo-guide, demo-advanced, architecture, concepts, connectors, features.md all updated Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-06-07 22:11:38 +00:00 · 2026-03-24 00:25:00 -04:00
parent 5ac1bbbc4c
commit 27d2f8b95e
23 changed files with 2916 additions and 24 deletions
@@ -703,6 +703,61 @@ flowchart TB

 For production, you would also add an ingress controller, TLS termination for the certctl API itself, and external PostgreSQL (RDS, Cloud SQL, etc.).

+## Discovery Data Flow (M18b)
+
+Certificate discovery enables operators to build a complete inventory of existing certificates before managing them with certctl. Here's how data flows through the system:
+
+```mermaid
+flowchart TB
+    AGENT["certctl-agent\n(on infrastructure)"]
+    SCAN["Filesystem Scanner\n(CERTCTL_DISCOVERY_DIRS)"]
+    EXTRACT["Extract Metadata\n(CN, SANs, serial, issuer, expiry, fingerprint)"]
+    REPORT["POST /api/v1/agents/{id}/discoveries\n(submit scan results)"]
+    HANDLER["Discovery Handler\n(parse request)"]
+    SERVICE["Discovery Service\n(ProcessDiscoveryReport)"]
+    REPO["Discovery Repository\n(upsert with fingerprint dedup)"]
+    DB["PostgreSQL\ndiscovered_certificates\ndiscovery_scans tables"]
+    AUDIT["Audit Service\n(RecordDiscoveryScanCompleted)"]
+    API_LIST["GET /api/v1/discovered-certificates\n(list for triage)"]
+    API_CLAIM["POST /discovered-certificates/{id}/claim\n(operator claims cert)"]
+    API_DISMISS["POST /discovered-certificates/{id}/dismiss\n(operator dismisses)"]
+    UPDATE_STATUS["Update Status\n(Unmanaged → Managed/Dismissed)"]
+
+    AGENT -->|"Scan loop\n(startup + 6h)"| SCAN
+    SCAN --> EXTRACT
+    EXTRACT --> REPORT
+    REPORT --> HANDLER
+    HANDLER --> SERVICE
+    SERVICE --> REPO
+    REPO -->|"Dedup by fingerprint\n+ agent + path"| DB
+    SERVICE --> AUDIT
+    AUDIT -->|"discovery_scan_completed"| DB
+    DB -->|"query unmanaged"| API_LIST
+    API_LIST -->|"operator reviews"| API_CLAIM
+    API_LIST -->|"operator reviews"| API_DISMISS
+    API_CLAIM --> UPDATE_STATUS
+    API_DISMISS --> UPDATE_STATUS
+    UPDATE_STATUS -->|"RecordDiscoveryCertClaimed\nRecordDiscoveryCertDismissed"| AUDIT
+    AUDIT --> DB
+```
+
+**Key steps:**
+
+1. **Agent-side discovery** — Agent scans `CERTCTL_DISCOVERY_DIRS` on startup and every 6 hours, walking directories recursively and parsing PEM/DER files
+2. **Metadata extraction** — For each certificate found, extract: common name, SANs, serial number, issuer DN, subject DN, expiration date, key algorithm, key size, is_ca flag, SHA-256 fingerprint (used as dedup key)
+3. **Server submission** — Agent POSTs scan results as `DiscoveryReport` to `POST /api/v1/agents/{id}/discoveries`
+4. **Deduplication** — Server uses fingerprint + agent ID + filesystem path as unique key; prevents duplicate records of the same cert on the same agent
+5. **Storage** — Records stored in `discovered_certificates` table with status = "Unmanaged"
+6. **Audit** — `discovery_scan_completed` event logged with agent ID, cert count, scan timestamp
+7. **Operator triage** — Operator queries `GET /api/v1/discovered-certificates?status=Unmanaged` to see new findings
+8. **Claim or dismiss** — For each unmanaged cert, operator either:
+   - **Claims it** via `POST /discovered-certificates/{id}/claim` — links to existing managed cert or creates new enrollment
+   - **Dismisses it** via `POST /discovered-certificates/{id}/dismiss` — removes from triage, marked as "Dismissed"
+9. **Status tracking** — `discovery_cert_claimed` and `discovery_cert_dismissed` events audit the operator's decision
+10. **Summary** — `GET /api/v1/discovery-summary` returns count of Unmanaged, Managed, and Dismissed certs (useful for compliance reporting)
+
+This data flow is pull-based and non-blocking. Agents discover at their own pace; the server stores results for later review. There's no pressure to claim or dismiss; operators can leave certificates in "Unmanaged" status indefinitely.
+
 ## Testing Strategy

 certctl uses a layered testing approach aligned with the handler → service → repository architecture, with 860+ tests across five layers (service, handler, integration, connector, and frontend). The goal is high-confidence regression prevention at the service and handler layers, where the most complex business logic lives, combined with integration tests that exercise the full request path from HTTP to database.
@@ -190,6 +190,19 @@ certctl includes an MCP (Model Context Protocol) server that exposes all 76 API

 The MCP server is a separate binary (`cmd/mcp-server/`) that communicates via stdio transport and acts as a stateless HTTP proxy to the certctl REST API. It requires no additional infrastructure — just point it at your certctl server URL and API key.

+### Certificate Discovery
+
+Certificate discovery is the process of automatically finding existing certificates in your infrastructure — certificates you didn't issue through certctl, possibly issued by other CAs or tools. This is essential for building a complete inventory before you can manage everything.
+
+**How it works:** Agents can scan configured directories (configured via `CERTCTL_DISCOVERY_DIRS`) for certificate files. On startup and every 6 hours, the agent walks these directories recursively, parses PEM and DER files, extracts metadata (common name, SANs, expiration, issuer, key algorithm), and reports all findings to the control plane. The server deduplicates by fingerprint (prevents duplicate reports of the same cert) and stores them with a status: **Unmanaged** (discovered but not yet managed), **Managed** (linked to a control plane cert), or **Dismissed** (operator decided not to manage it).
+
+This gives you a three-step triage workflow:
+1. **Discover** — Agents find all existing certs on your infrastructure
+2. **Triage** — Operators review discoveries and decide: claim it (enroll for management), or dismiss it (not worth managing)
+3. **Baseline** — Once triaged, you have a complete baseline of what's deployed, what you're managing, and what's unmanaged
+
+This is a prerequisite for multi-CA migration, compliance audits, and building confidence that you've found all the certificates that matter.
+
 ### Observability

 certctl exposes a JSON metrics endpoint at `GET /api/v1/metrics` with gauges (certificate totals by status, agent counts, pending jobs), counters (completed/failed jobs), and uptime. Five stats endpoints power the dashboard charts: summary statistics, certificates by status, expiration timeline, job trends, and issuance rate.
@@ -581,6 +581,64 @@ docker rm -f nginx
 6. **Idempotent operations** — Deploying the same certificate twice should succeed, not fail
 7. **Report metadata** — Return deployment duration, target address, and other useful data in results

+## Agent Discovery Scanner
+
+Agents include a built-in certificate discovery scanner that walks configured directories and reports unmanaged certificates to the control plane. This is useful for discovering existing certificates already deployed in your infrastructure, so you can bring them under certctl's management.
+
+### Configuration
+
+Enable discovery on an agent by setting `CERTCTL_DISCOVERY_DIRS` to a comma-separated list of directories:
+
+```bash
+export CERTCTL_DISCOVERY_DIRS="/etc/nginx/certs,/etc/ssl/certs,/etc/apache2/ssl"
+```
+
+Or via command-line flag:
+
+```bash
+./agent --agent-id agent-nginx-01 --discovery-dirs "/etc/nginx/certs,/etc/ssl/certs"
+```
+
+The agent scans these directories on startup and every 6 hours, looking for certificate files in PEM or DER format (extensions: `.pem`, `.crt`, `.cer`, `.cert`, `.der`).
+
+### How It Works
+
+1. **Scan**: Agent recursively walks directories, extracts certificates
+2. **Deduplicate**: Control plane deduplicates by SHA-256 fingerprint (same cert in multiple locations is one discovery)
+3. **Store**: Discovered certificates stored with metadata (agent ID, file path, found date, fingerprint)
+4. **Triage**: Operators query discovered certs via API, claim to link to managed certificates, or dismiss false positives
+
+### API Endpoints
+
+```bash
+# List discovered certificates (filter by agent, status)
+curl -s "http://localhost:8443/api/v1/discovered-certificates?agent_id=agent-nginx-01&status=new" | jq .
+
+# Get discovery detail
+curl -s http://localhost:8443/api/v1/discovered-certificates/DISCOVERY_ID | jq .
+
+# Claim a discovered cert (link to managed certificate)
+curl -s -X POST http://localhost:8443/api/v1/discovered-certificates/DISCOVERY_ID/claim \
+  -H "Content-Type: application/json" \
+  -d '{"managed_certificate_id": "mc-api-prod"}' | jq .
+
+# Dismiss a discovery
+curl -s -X POST http://localhost:8443/api/v1/discovered-certificates/DISCOVERY_ID/dismiss | jq .
+
+# View discovery scan history
+curl -s http://localhost:8443/api/v1/discovery-scans | jq .
+
+# Summary counts (new, claimed, dismissed)
+curl -s http://localhost:8443/api/v1/discovery-summary | jq .
+```
+
+### Use Cases
+
+- **Inventory audit** — Find all TLS certificates running in your infrastructure
+- **Migration** — Onboard existing certificates that were issued outside certctl
+- **Compliance** — Detect rogue/unauthorized certificates in monitored directories
+- **Integration** — Pull certificate data from systems that pre-generate certs (e.g., Kubernetes CertManager)
+
 ## What's Next

 - [Architecture Guide](architecture.md) — Understanding the full system design
@@ -675,6 +675,57 @@ curl -s "$API/api/v1/certificates/mc-demo-api/deployments" | jq .

 ---

+## Part 14: Certificate Discovery (M18b)
+
+Agents can automatically discover existing certificates already deployed in your infrastructure. This is useful for building a baseline inventory before you start managing everything with certctl.
+
+First, configure the demo agent to scan for certificates. In the Docker Compose setup, agents have a `/tmp/certs` directory (created by the seed script). Restart the agent with discovery enabled:
+
+```bash
+# Stop the existing agent
+docker compose -f deploy/docker-compose.yml stop agent
+
+# Restart with discovery enabled (scans /tmp/certs every 6 hours, or on startup)
+docker compose -f deploy/docker-compose.yml run -e CERTCTL_DISCOVERY_DIRS=/tmp/certs agent certctl-agent
+```
+
+Or with the CLI flag:
+
+```bash
+certctl-agent --agent-id a-demo-1 --key-dir /tmp/keys --discovery-dirs /tmp/certs --server http://localhost:8443 --api-key test-key-123
+```
+
+Now check what the agent discovered:
+
+```bash
+# List discovered certificates (should show unmanaged certs found on the agent)
+curl -s "$API/api/v1/discovered-certificates?status=Unmanaged" | jq '.data[] | {id, common_name, expires_at, issuer_dn, status}'
+
+# Get a summary of all discoveries
+curl -s $API/api/v1/discovery-summary | jq .
+```
+
+If the agent found certificates, you'll see entries with `status: "Unmanaged"`. Now triage them — claim the ones you want to manage or dismiss the ones you don't:
+
+```bash
+# Claim a certificate (link it to a managed cert, or create new enrollment)
+DISCOVERED_ID=$(curl -s "$API/api/v1/discovered-certificates?status=Unmanaged" | jq -r '.data[0].id')
+curl -s -X POST "$API/api/v1/discovered-certificates/$DISCOVERED_ID/claim" \
+  -H "Content-Type: application/json" \
+  -d '{"reason": "Migrating from external CA to certctl"}' | jq .
+
+# Or dismiss a certificate
+curl -s -X POST "$API/api/v1/discovered-certificates/$DISCOVERED_ID/dismiss" \
+  -H "Content-Type: application/json" \
+  -d '{"reason": "Self-signed test cert, not production"}' | jq .
+```
+
+**How it works:** The agent scans `CERTCTL_DISCOVERY_DIRS` on startup and every 6 hours, extracts metadata (common name, SANs, issuer, expiration, key type, fingerprint) from all PEM and DER files, and POSTs the findings to `POST /api/v1/agents/{id}/discoveries`. The server deduplicates by fingerprint (prevents duplicate records) and stores results with a status: **Unmanaged** (discovered, not yet managed), **Managed** (linked to a control plane cert), or **Dismissed** (operator decided not to manage). This gives you a triage workflow: discover → review → claim or dismiss.
+
+**In the dashboard**, the Discovery page (coming in future V2.x) will provide a visual triage interface for claiming and dismissing discovered certificates.
+
+---
+
 ## End-to-End Architecture Summary

 Here's what we just walked through, mapped to the system architecture:
@@ -69,6 +69,9 @@ On the Certificates page, select multiple certificates using the checkboxes. A b
 **10. "How do I see the deployment history?"**
 Click any certificate, then scroll to the deployment timeline. A visual 4-step timeline shows the lifecycle: Requested → Issued → Deploying → Active. Previous versions show a rollback button.

+**11. "What about certificates already running in production?"**
+Enable discovery on agents by setting `CERTCTL_DISCOVERY_DIRS` to directories containing certificates (e.g., `/etc/nginx/certs`). Agents scan on startup and every 6 hours, report findings to the control plane. Click "Discovered Certificates" to see what agents found — claim unmanaged certs to bring them under certctl's management, or dismiss them.
+
 ## API Walkthrough

 The dashboard is backed by a real REST API. Try these while the demo is running:
@@ -111,6 +114,12 @@ curl -s http://localhost:8443/api/v1/agent-groups | jq .
 curl -s -X POST http://localhost:8443/api/v1/certificates/mc-api-prod/revoke \
  -H "Content-Type: application/json" \
  -d '{"reason": "superseded"}' | jq .
+
+# List discovered certificates
+curl -s http://localhost:8443/api/v1/discovered-certificates | jq .
+
+# Discovery summary (counts by status)
+curl -s http://localhost:8443/api/v1/discovery-summary | jq .
 ```

 ## Demo Without Docker
@@ -147,7 +156,8 @@ If you're demoing to a team or customer, here's a suggested flow:
 7. **Show profiles** — "Certificate profiles enforce crypto constraints — key types, max TTL, compliance requirements"
 8. **Show policies** — "Guardrails prevent teams from going outside approved scope"
 9. **Show bulk operations** — "Select multiple certs, trigger renewal or revoke in bulk with progress tracking"
-10. **Show the API** — "Everything you see here is API-first. We also have a CLI tool and an MCP server for AI assistant integration"
+10. **Show certificate discovery** — "Agents scan your infrastructure for existing certificates you're not managing yet. We automatically deduplicate by fingerprint, show you what we found, and let you claim them or dismiss them"
+11. **Show the API** — "Everything you see here is API-first. We also have a CLI tool and an MCP server for AI assistant integration"

 The whole walkthrough takes 5-10 minutes.

@@ -7,7 +7,7 @@ Complete reference of all features shipped in the V2 release (as of March 2026).
 ## API Surface

 ### Overview
- **77 endpoints** across 16 resource domains under `/api/v1/`
+- **84 endpoints** across 17 resource domains under `/api/v1/`
 - REST API with HTTP semantics (GET, POST, PUT, DELETE)
 - All endpoints require authentication by default (configurable)
 - OpenAPI 3.1 spec with full schema documentation
@@ -54,6 +54,7 @@ Complete reference of all features shipped in the V2 release (as of March 2026).
 | **Teams** | 5 | List, create, get, update, delete |
 | **Owners** | 5 | List, create, get, update, delete |
 | **Agent Groups** | 6 | List, create, get, update, delete, list agents in group |
+| **Discovery** | 7 | Submit scan results, list discovered certs, get detail, claim, dismiss, list scans, summary stats |
 | **Audit** | 3 | List events, list by resource, export (CSV/JSON) |
 | **Notifications** | 3 | List, get, mark as read |
 | **Stats** | 5 | Dashboard summary, certificates by status, expiration timeline, job trends, issuance rate |
@@ -346,6 +347,70 @@ Agents report to `/api/v1/agents/{id}/work` with supported target types and issu

 ---

+## Certificate Discovery (M18b)
+
+### Overview
+Agents automatically discover existing certificates in the infrastructure — on filesystem, in key stores, or elsewhere — report findings to the control plane, and operators triage them for enrollment.
+
+### Agent-Side Discovery
+- **Configuration** — `CERTCTL_DISCOVERY_DIRS` env var (comma-separated list) or `--discovery-dirs` CLI flag
+- **Scan Execution** — Runs on agent startup and every 6 hours in background
+- **Supported Formats** — PEM (.pem, .crt, .cer, .cert) and DER (.der) files
+- **Recursive Walk** — Scans directory trees to find all certificates
+- **File Filtering** — Skips files > 1MB and obvious key files
+
+### Certificate Extraction
+Each discovered certificate is parsed and its metadata extracted:
+
+| Field | Source | Example |
+|-------|--------|---------|
+| **Common Name** | X.509 Subject CN | api.example.com |
+| **SANs** | X.509 SubjectAltNames | api.example.com, *.api.example.com |
+| **Serial** | Certificate serial number | 0x123abc... |
+| **Issuer DN** | X.509 Issuer | CN=Internal CA, O=Acme Inc |
+| **Subject DN** | X.509 Subject | CN=api.example.com, O=Acme Inc |
+| **Not Before** | Validity start | 2024-01-15T00:00:00Z |
+| **Not After** | Validity end | 2026-01-15T00:00:00Z |
+| **Key Algorithm** | Key type | RSA, ECDSA, Ed25519 |
+| **Key Size** | Bits | 2048, 256, 4096 |
+| **Is CA** | CA flag in extensions | true/false |
+| **Fingerprint** | SHA-256 hash (dedup key) | a1b2c3d4e5f6... |
+
+### Server-Side Processing
+- **Deduplication** — Uses fingerprint + agent ID + path as unique key; prevents duplicates
+- **Status Tracking** — Three statuses: **Unmanaged** (discovered, not yet claimed), **Managed** (linked to control plane cert), **Dismissed** (operator decided not to manage)
+- **Audit Trail** — `discovery_scan_completed`, `discovery_cert_claimed`, `discovery_cert_dismissed` events logged with actor and reason
+- **Storage** — `discovered_certificates` and `discovery_scans` tables in PostgreSQL
+
+### Triage Workflow
+1. Agent submits scan results via `POST /api/v1/agents/{id}/discoveries`
+2. Server deduplicates and stores discovery records
+3. Operator views `GET /api/v1/discovered-certificates?status=Unmanaged`
+4. For each unmanaged cert:
+   - **Claim it** — `POST /api/v1/discovered-certificates/{id}/claim` links to managed cert or creates new enrollment
+   - **Dismiss it** — `POST /api/v1/discovered-certificates/{id}/dismiss` removes from triage queue
+5. Tracking enables visibility into what's deployed vs. what's managed
+
+### Discovery API Endpoints (M18b)
+| Endpoint | Method | Purpose |
+|----------|--------|---------|
+| `/api/v1/agents/{id}/discoveries` | POST | Agent submits scan results |
+| `/api/v1/discovered-certificates` | GET | List discovered certs (with ?agent_id, ?status filters) |
+| `/api/v1/discovered-certificates/{id}` | GET | Get single discovered cert detail |
+| `/api/v1/discovered-certificates/{id}/claim` | POST | Link to managed cert or create enrollment |
+| `/api/v1/discovered-certificates/{id}/dismiss` | POST | Dismiss from triage |
+| `/api/v1/discovery-scans` | GET | List scan history with timestamps |
+| `/api/v1/discovery-summary` | GET | Aggregate status counts (Unmanaged, Managed, Dismissed) |
+
+### Use Cases
+- **Inventory Baseline** — Scan production servers at deployment time to establish baseline of existing certificates
+- **Compliance Discovery** — Find all TLS certs before renewing certificate policies
+- **Migration Planning** — Discover unmanaged certs to plan migration from other CA/platforms
+- **Audit Preparation** — Triage discovered certs into managed and dismissed for compliance reports
+- **Multi-CA Migration** — Find all certs currently issued by old CA, claim them for renewal under new issuer
+
+---
+
 ## Ownership & Accountability

 ### Teams
@@ -840,7 +905,7 @@ The web dashboard is the primary operational interface for certctl. Built with *

 | Category | Count |
 |----------|-------|
-| **API Endpoints** | 77 (under /api/v1/) |
+| **API Endpoints** | 84 (under /api/v1/) |
 | **Dashboard Pages** | 19 |
 | **Issuer Connectors** | 4 (Local CA, ACME, step-ca, OpenSSL) |
 | **Target Connectors** | 5 (3 impl: NGINX, Apache, HAProxy; 2 stubs: F5, IIS) |
@@ -850,9 +915,10 @@ The web dashboard is the primary operational interface for certctl. Built with *
 | **Policy Rule Types** | 5 (AllowedIssuers, AllowedDomains, RequiredMetadata, AllowedEnvironments, RenewalLeadTime) |
 | **Certificate States** | 8 (Pending, Active, Expiring, Expired, RenewalInProgress, Failed, Revoked, Archived) |
 | **Revocation Reason Codes** | 8 (RFC 5280 compliant) |
-| **MCP Tools** | 76 (16 resource domains) |
+| **Discovery Statuses** | 3 (Unmanaged, Managed, Dismissed) |
+| **MCP Tools** | 83 (17 resource domains) |
 | **CLI Subcommands** | 10 |
-| **Database Tables** | 18+ |
-| **Test Suite** | 860+ tests |
-| **Environment Variables** | 40+ configuration options |
+| **Database Tables** | 20+ |
+| **Test Suite** | 881+ tests |
+| **Environment Variables** | 41+ configuration options |

@@ -338,6 +338,32 @@ docker compose -f deploy/docker-compose.yml down -v

 The `-v` flag removes the PostgreSQL data volume so you get a clean slate next time.

+### Certificate Discovery
+
+Agents can scan your infrastructure for existing certificates you're not yet managing:
+
+```bash
+# Configure agent to scan directories
+export CERTCTL_DISCOVERY_DIRS="/etc/nginx/certs,/etc/ssl/certs,/var/lib/certs"
+
+# Agent scans on startup + every 6 hours, reports findings to control plane
+```
+
+Query discovered certificates:
+
+```bash
+# List all discovered certs from a specific agent
+curl -s "http://localhost:8443/api/v1/discovered-certificates?agent_id=agent-nginx-prod" | jq .
+
+# Get discovery summary (counts by status)
+curl -s http://localhost:8443/api/v1/discovery-summary | jq .
+
+# Claim a discovered cert (link to managed cert)
+curl -s -X POST "http://localhost:8443/api/v1/discovered-certificates/DISCOVERY_ID/claim" \
+  -H "Content-Type: application/json" \
+  -d '{"managed_certificate_id": "mc-api-prod"}' | jq .
+```
+
 ## What's Next

 - **[Advanced Demo](demo-advanced.md)** — Issue a real certificate via the Local CA and watch it appear in the dashboard