feat(M50): cloud secret manager discovery — AWS SM, Azure KV, GCP SM

Extend certificate discovery from filesystem + network to cloud secret
managers. Three pluggable DiscoverySource connectors feed into the
existing discovery pipeline via sentinel agent pattern, with a 9th
scheduler loop for periodic cloud scanning.

- AWS Secrets Manager: aws-sdk-go-v2, tag/prefix filtering, 10 tests
- Azure Key Vault: stdlib HTTP + OAuth2, base64 DER/PEM, 16 tests
- GCP Secret Manager: stdlib HTTP + JWT OAuth2, label filter, 14 tests
- CloudDiscoveryService orchestrator with 9 tests
- 9th scheduler loop (6h default, atomic.Bool idempotency)
- Discovery page: color-coded source type badges
- 14 new env vars across CloudDiscoveryConfig structs
- Docs: connectors.md, architecture.md, features.md, README updated

49 new tests. All CI checks pass (go vet, race, lint, coverage).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
shankar0123
2026-04-15 23:01:00 -04:00
parent 3f619bcaac
commit e1bcde4cf1
19 changed files with 3791 additions and 24 deletions
+14 -3
View File
@@ -959,9 +959,9 @@ See `deploy/helm/certctl/values.yaml` for the full configuration reference and `
For production, you would also add an ingress controller, TLS termination for the certctl API itself, and external PostgreSQL (RDS, Cloud SQL, etc.).
## Discovery Data Flow (M18b + M21)
## Discovery Data Flow (M18b + M21 + M50)
Certificate discovery enables operators to build a complete inventory of existing certificates before managing them with certctl. There are two discovery modes that feed into the same pipeline:
Certificate discovery enables operators to build a complete inventory of existing certificates before managing them with certctl. There are three discovery modes that feed into the same pipeline:
```mermaid
flowchart TB
@@ -970,6 +970,7 @@ flowchart TB
SCAN["Filesystem Scanner\n(CERTCTL_DISCOVERY_DIRS)"]
SERVER["certctl-server\n(network discovery)"]
NETSCAN["TLS Scanner\n(CIDR ranges + ports)"]
CLOUD["Cloud Discovery\n(AWS SM / Azure KV / GCP SM)"]
end
EXTRACT["Extract Metadata\n(CN, SANs, serial, issuer, expiry, fingerprint)"]
@@ -985,6 +986,7 @@ flowchart TB
SCAN --> EXTRACT
SERVER -->|"Scheduler loop\n(every 6h)"| NETSCAN
NETSCAN -->|"crypto/tls.Dial\n50 goroutines"| EXTRACT
CLOUD -->|"Scheduler loop\n(every 6h)"| EXTRACT
EXTRACT --> SERVICE
SERVICE --> REPO
REPO -->|"Dedup by fingerprint\n+ agent_id + source_path"| DB
@@ -1011,7 +1013,16 @@ flowchart TB
5. **Sentinel agent** — Results submitted using `server-scanner` as virtual agent ID, with `source_path` set to `ip:port` and `source_format` set to `network`
6. **Same pipeline** — Feeds into the same `DiscoveryService.ProcessDiscoveryReport()` as filesystem discovery — same dedup, same audit trail, same triage workflow
**Common triage workflow (both sources):**
**Cloud Secret Manager Discovery (M50):**
1. **Pluggable sources** — Each cloud provider implements the `DiscoverySource` interface (Name, Type, Discover, ValidateConfig). Three built-in sources: AWS Secrets Manager, Azure Key Vault, GCP Secret Manager
2. **CloudDiscoveryService orchestrator** — Iterates registered sources, calls `Discover()` on each, feeds reports into `ProcessDiscoveryReport()`. Errors from one source don't prevent other sources from running
3. **Scheduler integration** — 9th scheduler loop (6h default), runs immediately on startup, `atomic.Bool` idempotency guard
4. **Sentinel agents** — Each source uses its own sentinel agent ID (`cloud-aws-sm`, `cloud-azure-kv`, `cloud-gcp-sm`) for dedup and triage filtering
5. **Source path format**`aws-sm://{region}/{secret}`, `azure-kv://{cert-name}/{version}`, `gcp-sm://{project}/{secret}`
6. **No new schema** — Reuses existing `discovered_certificates` and `discovery_scans` tables. Sentinel agent IDs leverage existing `(fingerprint_sha256, agent_id, source_path)` dedup constraint
**Common triage workflow (all sources):**
1. **Storage** — Records stored in `discovered_certificates` table with status = "Unmanaged"
2. **Audit**`discovery_scan_completed` event logged with agent ID, cert count, scan timestamp
+57
View File
@@ -1396,6 +1396,63 @@ When `CERTCTL_NETWORK_SCAN_ENABLED=true`, the server runs a 6th scheduler loop (
- **Migration assessment** — Scan a network range before onboarding to certctl management
- **Expiration monitoring** — Discover soon-to-expire certs on network endpoints before they cause outages
## Cloud Secret Manager Discovery
certctl extends the existing filesystem and network discovery pipeline to cloud secret managers. Certificates stored in cloud vaults are automatically discovered, inventoried, and available for triage in the Discovery page.
Each cloud source runs as a pluggable `DiscoverySource` with its own sentinel agent ID. Discovered certificates flow through the same `ProcessDiscoveryReport` pipeline used by filesystem and network discovery — dedup by fingerprint, audit trail, status tracking.
### AWS Secrets Manager
Discovers certificates stored as secrets in AWS Secrets Manager. Filters by tag (`type=certificate` by default) and optional name prefix.
| Variable | Description | Default |
|---|---|---|
| `CERTCTL_CLOUD_DISCOVERY_ENABLED` | Enable cloud discovery scheduler | `false` |
| `CERTCTL_AWS_SM_DISCOVERY_ENABLED` | Enable AWS SM source | `false` |
| `CERTCTL_AWS_SM_REGION` | AWS region (e.g., `us-east-1`) | — |
| `CERTCTL_AWS_SM_TAG_FILTER` | Tag key=value filter | `type=certificate` |
| `CERTCTL_AWS_SM_NAME_PREFIX` | Secret name prefix filter | — |
Source path format: `aws-sm://{region}/{secret-name}`. Sentinel agent: `cloud-aws-sm`.
### Azure Key Vault
Discovers certificates from Azure Key Vault using OAuth2 client credentials authentication. No Azure SDK dependency — uses stdlib HTTP with Azure AD token exchange.
| Variable | Description | Default |
|---|---|---|
| `CERTCTL_AZURE_KV_DISCOVERY_ENABLED` | Enable Azure KV source | `false` |
| `CERTCTL_AZURE_KV_VAULT_URL` | Vault URL (e.g., `https://myvault.vault.azure.net`) | — |
| `CERTCTL_AZURE_KV_TENANT_ID` | Azure AD tenant ID | — |
| `CERTCTL_AZURE_KV_CLIENT_ID` | Azure AD application (client) ID | — |
| `CERTCTL_AZURE_KV_CLIENT_SECRET` | Azure AD application secret | — |
Source path format: `azure-kv://{cert-name}/{version}`. Sentinel agent: `cloud-azure-kv`.
### GCP Secret Manager
Discovers certificates stored in GCP Secret Manager. Filters by label (`type=certificate`). Uses JWT-based OAuth2 service account auth — no Google SDK dependency.
| Variable | Description | Default |
|---|---|---|
| `CERTCTL_GCP_SM_DISCOVERY_ENABLED` | Enable GCP SM source | `false` |
| `CERTCTL_GCP_SM_PROJECT` | GCP project ID | — |
| `CERTCTL_GCP_SM_CREDENTIALS` | Path to service account JSON file | — |
Source path format: `gcp-sm://{project}/{secret-name}`. Sentinel agent: `cloud-gcp-sm`.
### Cloud Discovery Scheduler
All enabled cloud sources run on a shared scheduler loop (9th loop). The interval is configurable:
| Variable | Description | Default |
|---|---|---|
| `CERTCTL_CLOUD_DISCOVERY_ENABLED` | Master switch | `false` |
| `CERTCTL_CLOUD_DISCOVERY_INTERVAL` | Scan interval | `6h` |
The loop runs immediately on startup and then on each tick. Each source runs sequentially within the loop. Errors from one source do not prevent other sources from running.
## What's Next
- [Architecture Guide](architecture.md) — Understanding the full system design
+25
View File
@@ -852,6 +852,31 @@ Server-side active TLS scanning of CIDR ranges. Concurrent probing with semaphor
| `/api/v1/network-scan-targets/{id}` | DELETE | Delete |
| `/api/v1/network-scan-targets/{id}/scan` | POST | Trigger immediate scan |
### Cloud Secret Manager Discovery
<!-- Source: internal/connector/discovery/awssm/, azurekv/, gcpsm/, internal/service/cloud_discovery.go -->
Discovers certificates stored in cloud secret managers and brings them into the certctl inventory. Extends the existing discovery pipeline with pluggable `DiscoverySource` implementations. Each source runs as part of the 9th scheduler loop (6h default).
**Supported sources:**
- **AWS Secrets Manager** — filters by tag (`type=certificate`) and name prefix. Uses `aws-sdk-go-v2`. Sentinel agent: `cloud-aws-sm`
- **Azure Key Vault** — OAuth2 client credentials auth, no Azure SDK. Lists certificates from vault. Sentinel agent: `cloud-azure-kv`
- **GCP Secret Manager** — JWT-based OAuth2 service account auth, no Google SDK. Filters by label (`type=certificate`). Sentinel agent: `cloud-gcp-sm`
| Env Var | Default | Description |
|---|---|---|
| `CERTCTL_CLOUD_DISCOVERY_ENABLED` | `false` | Enable cloud discovery scheduler |
| `CERTCTL_CLOUD_DISCOVERY_INTERVAL` | `6h` | Scheduler loop interval |
| `CERTCTL_AWS_SM_DISCOVERY_ENABLED` | `false` | Enable AWS SM source |
| `CERTCTL_AWS_SM_REGION` | — | AWS region |
| `CERTCTL_AWS_SM_TAG_FILTER` | `type=certificate` | Tag filter for secrets |
| `CERTCTL_AZURE_KV_DISCOVERY_ENABLED` | `false` | Enable Azure KV source |
| `CERTCTL_AZURE_KV_VAULT_URL` | — | Key Vault URL |
| `CERTCTL_GCP_SM_DISCOVERY_ENABLED` | `false` | Enable GCP SM source |
| `CERTCTL_GCP_SM_PROJECT` | — | GCP project ID |
| `CERTCTL_GCP_SM_CREDENTIALS` | — | Service account JSON path |
### Continuous TLS Health Monitoring
<!-- Source: internal/domain/health_check.go, internal/service/health_check.go -->