mirror of
https://github.com/shankar0123/certctl.git
synced 2026-06-07 13:41:30 +00:00
feat(M50): cloud secret manager discovery — AWS SM, Azure KV, GCP SM
Extend certificate discovery from filesystem + network to cloud secret managers. Three pluggable DiscoverySource connectors feed into the existing discovery pipeline via sentinel agent pattern, with a 9th scheduler loop for periodic cloud scanning. - AWS Secrets Manager: aws-sdk-go-v2, tag/prefix filtering, 10 tests - Azure Key Vault: stdlib HTTP + OAuth2, base64 DER/PEM, 16 tests - GCP Secret Manager: stdlib HTTP + JWT OAuth2, label filter, 14 tests - CloudDiscoveryService orchestrator with 9 tests - 9th scheduler loop (6h default, atomic.Bool idempotency) - Discovery page: color-coded source type badges - 14 new env vars across CloudDiscoveryConfig structs - Docs: connectors.md, architecture.md, features.md, README updated 49 new tests. All CI checks pass (go vet, race, lint, coverage). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
+14
-3
@@ -959,9 +959,9 @@ See `deploy/helm/certctl/values.yaml` for the full configuration reference and `
|
||||
|
||||
For production, you would also add an ingress controller, TLS termination for the certctl API itself, and external PostgreSQL (RDS, Cloud SQL, etc.).
|
||||
|
||||
## Discovery Data Flow (M18b + M21)
|
||||
## Discovery Data Flow (M18b + M21 + M50)
|
||||
|
||||
Certificate discovery enables operators to build a complete inventory of existing certificates before managing them with certctl. There are two discovery modes that feed into the same pipeline:
|
||||
Certificate discovery enables operators to build a complete inventory of existing certificates before managing them with certctl. There are three discovery modes that feed into the same pipeline:
|
||||
|
||||
```mermaid
|
||||
flowchart TB
|
||||
@@ -970,6 +970,7 @@ flowchart TB
|
||||
SCAN["Filesystem Scanner\n(CERTCTL_DISCOVERY_DIRS)"]
|
||||
SERVER["certctl-server\n(network discovery)"]
|
||||
NETSCAN["TLS Scanner\n(CIDR ranges + ports)"]
|
||||
CLOUD["Cloud Discovery\n(AWS SM / Azure KV / GCP SM)"]
|
||||
end
|
||||
|
||||
EXTRACT["Extract Metadata\n(CN, SANs, serial, issuer, expiry, fingerprint)"]
|
||||
@@ -985,6 +986,7 @@ flowchart TB
|
||||
SCAN --> EXTRACT
|
||||
SERVER -->|"Scheduler loop\n(every 6h)"| NETSCAN
|
||||
NETSCAN -->|"crypto/tls.Dial\n50 goroutines"| EXTRACT
|
||||
CLOUD -->|"Scheduler loop\n(every 6h)"| EXTRACT
|
||||
EXTRACT --> SERVICE
|
||||
SERVICE --> REPO
|
||||
REPO -->|"Dedup by fingerprint\n+ agent_id + source_path"| DB
|
||||
@@ -1011,7 +1013,16 @@ flowchart TB
|
||||
5. **Sentinel agent** — Results submitted using `server-scanner` as virtual agent ID, with `source_path` set to `ip:port` and `source_format` set to `network`
|
||||
6. **Same pipeline** — Feeds into the same `DiscoveryService.ProcessDiscoveryReport()` as filesystem discovery — same dedup, same audit trail, same triage workflow
|
||||
|
||||
**Common triage workflow (both sources):**
|
||||
**Cloud Secret Manager Discovery (M50):**
|
||||
|
||||
1. **Pluggable sources** — Each cloud provider implements the `DiscoverySource` interface (Name, Type, Discover, ValidateConfig). Three built-in sources: AWS Secrets Manager, Azure Key Vault, GCP Secret Manager
|
||||
2. **CloudDiscoveryService orchestrator** — Iterates registered sources, calls `Discover()` on each, feeds reports into `ProcessDiscoveryReport()`. Errors from one source don't prevent other sources from running
|
||||
3. **Scheduler integration** — 9th scheduler loop (6h default), runs immediately on startup, `atomic.Bool` idempotency guard
|
||||
4. **Sentinel agents** — Each source uses its own sentinel agent ID (`cloud-aws-sm`, `cloud-azure-kv`, `cloud-gcp-sm`) for dedup and triage filtering
|
||||
5. **Source path format** — `aws-sm://{region}/{secret}`, `azure-kv://{cert-name}/{version}`, `gcp-sm://{project}/{secret}`
|
||||
6. **No new schema** — Reuses existing `discovered_certificates` and `discovery_scans` tables. Sentinel agent IDs leverage existing `(fingerprint_sha256, agent_id, source_path)` dedup constraint
|
||||
|
||||
**Common triage workflow (all sources):**
|
||||
|
||||
1. **Storage** — Records stored in `discovered_certificates` table with status = "Unmanaged"
|
||||
2. **Audit** — `discovery_scan_completed` event logged with agent ID, cert count, scan timestamp
|
||||
|
||||
@@ -1396,6 +1396,63 @@ When `CERTCTL_NETWORK_SCAN_ENABLED=true`, the server runs a 6th scheduler loop (
|
||||
- **Migration assessment** — Scan a network range before onboarding to certctl management
|
||||
- **Expiration monitoring** — Discover soon-to-expire certs on network endpoints before they cause outages
|
||||
|
||||
## Cloud Secret Manager Discovery
|
||||
|
||||
certctl extends the existing filesystem and network discovery pipeline to cloud secret managers. Certificates stored in cloud vaults are automatically discovered, inventoried, and available for triage in the Discovery page.
|
||||
|
||||
Each cloud source runs as a pluggable `DiscoverySource` with its own sentinel agent ID. Discovered certificates flow through the same `ProcessDiscoveryReport` pipeline used by filesystem and network discovery — dedup by fingerprint, audit trail, status tracking.
|
||||
|
||||
### AWS Secrets Manager
|
||||
|
||||
Discovers certificates stored as secrets in AWS Secrets Manager. Filters by tag (`type=certificate` by default) and optional name prefix.
|
||||
|
||||
| Variable | Description | Default |
|
||||
|---|---|---|
|
||||
| `CERTCTL_CLOUD_DISCOVERY_ENABLED` | Enable cloud discovery scheduler | `false` |
|
||||
| `CERTCTL_AWS_SM_DISCOVERY_ENABLED` | Enable AWS SM source | `false` |
|
||||
| `CERTCTL_AWS_SM_REGION` | AWS region (e.g., `us-east-1`) | — |
|
||||
| `CERTCTL_AWS_SM_TAG_FILTER` | Tag key=value filter | `type=certificate` |
|
||||
| `CERTCTL_AWS_SM_NAME_PREFIX` | Secret name prefix filter | — |
|
||||
|
||||
Source path format: `aws-sm://{region}/{secret-name}`. Sentinel agent: `cloud-aws-sm`.
|
||||
|
||||
### Azure Key Vault
|
||||
|
||||
Discovers certificates from Azure Key Vault using OAuth2 client credentials authentication. No Azure SDK dependency — uses stdlib HTTP with Azure AD token exchange.
|
||||
|
||||
| Variable | Description | Default |
|
||||
|---|---|---|
|
||||
| `CERTCTL_AZURE_KV_DISCOVERY_ENABLED` | Enable Azure KV source | `false` |
|
||||
| `CERTCTL_AZURE_KV_VAULT_URL` | Vault URL (e.g., `https://myvault.vault.azure.net`) | — |
|
||||
| `CERTCTL_AZURE_KV_TENANT_ID` | Azure AD tenant ID | — |
|
||||
| `CERTCTL_AZURE_KV_CLIENT_ID` | Azure AD application (client) ID | — |
|
||||
| `CERTCTL_AZURE_KV_CLIENT_SECRET` | Azure AD application secret | — |
|
||||
|
||||
Source path format: `azure-kv://{cert-name}/{version}`. Sentinel agent: `cloud-azure-kv`.
|
||||
|
||||
### GCP Secret Manager
|
||||
|
||||
Discovers certificates stored in GCP Secret Manager. Filters by label (`type=certificate`). Uses JWT-based OAuth2 service account auth — no Google SDK dependency.
|
||||
|
||||
| Variable | Description | Default |
|
||||
|---|---|---|
|
||||
| `CERTCTL_GCP_SM_DISCOVERY_ENABLED` | Enable GCP SM source | `false` |
|
||||
| `CERTCTL_GCP_SM_PROJECT` | GCP project ID | — |
|
||||
| `CERTCTL_GCP_SM_CREDENTIALS` | Path to service account JSON file | — |
|
||||
|
||||
Source path format: `gcp-sm://{project}/{secret-name}`. Sentinel agent: `cloud-gcp-sm`.
|
||||
|
||||
### Cloud Discovery Scheduler
|
||||
|
||||
All enabled cloud sources run on a shared scheduler loop (9th loop). The interval is configurable:
|
||||
|
||||
| Variable | Description | Default |
|
||||
|---|---|---|
|
||||
| `CERTCTL_CLOUD_DISCOVERY_ENABLED` | Master switch | `false` |
|
||||
| `CERTCTL_CLOUD_DISCOVERY_INTERVAL` | Scan interval | `6h` |
|
||||
|
||||
The loop runs immediately on startup and then on each tick. Each source runs sequentially within the loop. Errors from one source do not prevent other sources from running.
|
||||
|
||||
## What's Next
|
||||
|
||||
- [Architecture Guide](architecture.md) — Understanding the full system design
|
||||
|
||||
@@ -852,6 +852,31 @@ Server-side active TLS scanning of CIDR ranges. Concurrent probing with semaphor
|
||||
| `/api/v1/network-scan-targets/{id}` | DELETE | Delete |
|
||||
| `/api/v1/network-scan-targets/{id}/scan` | POST | Trigger immediate scan |
|
||||
|
||||
### Cloud Secret Manager Discovery
|
||||
|
||||
<!-- Source: internal/connector/discovery/awssm/, azurekv/, gcpsm/, internal/service/cloud_discovery.go -->
|
||||
|
||||
Discovers certificates stored in cloud secret managers and brings them into the certctl inventory. Extends the existing discovery pipeline with pluggable `DiscoverySource` implementations. Each source runs as part of the 9th scheduler loop (6h default).
|
||||
|
||||
**Supported sources:**
|
||||
|
||||
- **AWS Secrets Manager** — filters by tag (`type=certificate`) and name prefix. Uses `aws-sdk-go-v2`. Sentinel agent: `cloud-aws-sm`
|
||||
- **Azure Key Vault** — OAuth2 client credentials auth, no Azure SDK. Lists certificates from vault. Sentinel agent: `cloud-azure-kv`
|
||||
- **GCP Secret Manager** — JWT-based OAuth2 service account auth, no Google SDK. Filters by label (`type=certificate`). Sentinel agent: `cloud-gcp-sm`
|
||||
|
||||
| Env Var | Default | Description |
|
||||
|---|---|---|
|
||||
| `CERTCTL_CLOUD_DISCOVERY_ENABLED` | `false` | Enable cloud discovery scheduler |
|
||||
| `CERTCTL_CLOUD_DISCOVERY_INTERVAL` | `6h` | Scheduler loop interval |
|
||||
| `CERTCTL_AWS_SM_DISCOVERY_ENABLED` | `false` | Enable AWS SM source |
|
||||
| `CERTCTL_AWS_SM_REGION` | — | AWS region |
|
||||
| `CERTCTL_AWS_SM_TAG_FILTER` | `type=certificate` | Tag filter for secrets |
|
||||
| `CERTCTL_AZURE_KV_DISCOVERY_ENABLED` | `false` | Enable Azure KV source |
|
||||
| `CERTCTL_AZURE_KV_VAULT_URL` | — | Key Vault URL |
|
||||
| `CERTCTL_GCP_SM_DISCOVERY_ENABLED` | `false` | Enable GCP SM source |
|
||||
| `CERTCTL_GCP_SM_PROJECT` | — | GCP project ID |
|
||||
| `CERTCTL_GCP_SM_CREDENTIALS` | — | Service account JSON path |
|
||||
|
||||
### Continuous TLS Health Monitoring
|
||||
|
||||
<!-- Source: internal/domain/health_check.go, internal/service/health_check.go -->
|
||||
|
||||
Reference in New Issue
Block a user