From cacde5a0ceb19cec1724ff58db74756a70b44070 Mon Sep 17 00:00:00 2001 From: Shankar Date: Sat, 4 Apr 2026 00:20:22 -0400 Subject: [PATCH] docs: remove hardcoded test counts from public-facing docs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Replace brittle test count numbers (1,554+, 1,088+, 211, etc.) with descriptions of testing approach and CI-enforced coverage gates. Counts go stale every milestone — coverage thresholds are machine- verified and never drift. Co-Authored-By: Claude Opus 4.6 --- README.md | 4 ++-- docs/architecture.md | 22 ++++++++++------------ docs/features.md | 10 +++++----- docs/why-certctl.md | 2 +- 4 files changed, 18 insertions(+), 20 deletions(-) diff --git a/README.md b/README.md index e938393..fed7afd 100644 --- a/README.md +++ b/README.md @@ -36,7 +36,7 @@ gantt 47 days :crit, 2020-01-01, 47d ``` -> **Actively maintained — shipping weekly.** Found something? [Open a GitHub issue](https://github.com/shankar0123/certctl/issues) — issues get triaged same-day. CI runs 1,554+ tests with race detection, static analysis, and vulnerability scanning on every commit. +> **Actively maintained — shipping weekly.** Found something? [Open a GitHub issue](https://github.com/shankar0123/certctl/issues) — issues get triaged same-day. CI runs the full test suite with race detection, static analysis, and vulnerability scanning on every commit. ## Why certctl Exists @@ -294,7 +294,7 @@ CI runs on every push: `go vet`, `go test -race`, `golangci-lint`, `govulncheck` Core lifecycle management — Local CA + ACME v2 issuers, NGINX target connector, agent-side key generation, API auth + rate limiting, React dashboard, CI pipeline with coverage gates, Docker images on GHCR. ### V2: Operational Maturity — Shipped -30+ milestones, 1,554+ tests. Sub-CA mode, ACME DNS-01/DNS-PERSIST-01, step-ca, Vault PKI, DigiCert CertCentral, OpenSSL/Custom CA issuers. NGINX, Apache, HAProxy, Traefik, Caddy, Envoy, Postfix, Dovecot, IIS targets. RFC 5280 revocation with CRL + OCSP. Certificate profiles, ownership tracking, approval workflows. Filesystem and network certificate discovery. Prometheus metrics, dashboard charts, agent fleet overview. EST server (RFC 7030), ACME ARI (RFC 9702), certificate export, S/MIME support, Helm chart, MCP server, CLI, scheduled digest emails. Slack, Teams, PagerDuty, OpsGenie, SMTP notifications. Compliance mapping (SOC 2, PCI-DSS 4.0, NIST SP 800-57). See the [Feature Inventory](docs/features.md) for details. +30+ milestones, extensively tested with CI-enforced coverage gates. Sub-CA mode, ACME DNS-01/DNS-PERSIST-01, step-ca, Vault PKI, DigiCert CertCentral, OpenSSL/Custom CA issuers. NGINX, Apache, HAProxy, Traefik, Caddy, Envoy, Postfix, Dovecot, IIS targets. RFC 5280 revocation with CRL + OCSP. Certificate profiles, ownership tracking, approval workflows. Filesystem and network certificate discovery. Prometheus metrics, dashboard charts, agent fleet overview. EST server (RFC 7030), ACME ARI (RFC 9702), certificate export, S/MIME support, Helm chart, MCP server, CLI, scheduled digest emails. Slack, Teams, PagerDuty, OpsGenie, SMTP notifications. Compliance mapping (SOC 2, PCI-DSS 4.0, NIST SP 800-57). See the [Feature Inventory](docs/features.md) for details. **Coming in v2.1.0:** Dynamic issuer and target configuration via GUI (no env var restarts), first-run onboarding wizard. diff --git a/docs/architecture.md b/docs/architecture.md index 65e4f9d..e97c592 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -964,27 +964,25 @@ This data flow is pull-based and non-blocking. Agents discover at their own pace ## Testing Strategy -certctl uses a layered testing approach aligned with the handler → service → repository architecture, with 1050+ tests across six layers (service, handler, integration, connector, frontend, and scheduler). The goal is high-confidence regression prevention at the service and handler layers, where the most complex business logic lives, combined with integration tests that exercise the full request path from HTTP to database. +certctl is extensively tested across eight layers with CI-enforced coverage gates that act as regression floors. The goal is high-confidence regression prevention at the service and handler layers (where the most complex business logic lives), combined with integration tests that exercise the full request path from HTTP to database. -**Service layer unit tests** (`internal/service/*_test.go`) — ~238 test functions across 15 files with mock repositories. These test all business logic in isolation: certificate CRUD with validation, certificate revocation (success, already-revoked, archived, invalid reason, all RFC 5280 reason codes, issuer notification, notification service integration, OCSP/CRL generation), agent lifecycle (registration, heartbeat, CSR submission with both keygen modes), job state machine (creation, processing, cancellation, retry logic), policy evaluation (all 5 rule types, violation creation), renewal and issuance flow (server-side and agent-side keygen paths), notification deduplication (threshold tag matching, channel routing), team/owner/agent group CRUD with pagination and audit recording, issuer service CRUD with connection testing, and the issuer connector adapter (type translation between connector and service layers including revocation). Mock repositories are simple structs with function fields, avoiding heavy mocking frameworks — this keeps tests readable and avoids coupling to mock library APIs. +**Service layer unit tests** (`internal/service/*_test.go`) — Mock-based tests across all service files covering certificate CRUD, revocation (all RFC 5280 reason codes, OCSP/CRL generation), agent lifecycle, job state machine, policy evaluation, renewal/issuance flow (both keygen modes), notification deduplication, team/owner/agent group CRUD, issuer service CRUD with connection testing, and the issuer connector adapter. Mock repositories are simple structs with function fields — no heavy mocking frameworks. -**Handler layer tests** (`internal/api/handler/*_test.go`) — ~257 test functions across 11 files using Go's `httptest` package. Every handler file has a corresponding test file: certificates (50 tests including revocation, DER CRL, and OCSP), agents (28 tests), jobs (21 tests including approve/reject), notifications (11 tests), policies (19 tests), profiles (18 tests), issuers (17 tests), targets (17 tests), agent groups (12 tests), teams (26 tests), and owners (21 tests). Each test file follows the same pattern: a mock service struct with function fields, `httptest.NewRecorder` for capturing responses, and a shared `contextWithRequestID()` helper. Tests cover the happy path, input validation (missing fields, invalid JSON, empty IDs, name length limits), error propagation from the service layer, method-not-allowed responses, and pagination parameters. +**Handler layer tests** (`internal/api/handler/*_test.go`) — Every handler file has a corresponding test file using Go's `httptest` package: certificates (including revocation, DER CRL, OCSP), agents, jobs (including approve/reject), notifications, policies, profiles, issuers, targets, agent groups, teams, owners, discovery, network scan, verification, export, EST, digest, stats, and metrics. Tests cover the happy path, input validation, error propagation, method-not-allowed, and pagination. -**Integration tests** (`internal/integration/`) — Two test files exercising the full stack from HTTP request through router, handler, service, and postgres repository layers. `lifecycle_test.go` has 11 subtests covering the complete certificate lifecycle: team/owner creation, certificate creation, issuer verification, renewal trigger, job verification, agent registration, CSR submission, deployment, and status reporting. `negative_test.go` has 14 subtests covering error paths, 19 M11b endpoint tests, and 8 revocation endpoint tests (M15a+M15b): nonexistent resource lookups (404s), invalid request bodies (malformed JSON, missing required fields), invalid CSR submission, heartbeat for nonexistent agents, wrong HTTP methods on list endpoints, empty list responses, renewal on nonexistent certificates, expired certificate lifecycle, team/owner/agent group CRUD validation, revocation success, already-revoked rejection, not-found revocation, JSON CRL retrieval, DER CRL retrieval, OCSP response retrieval, and short-lived cert exemption. Both use a shared `setupTestServer()` that builds a fully-wired server with real postgres repositories and the Local CA issuer connector. A third file, `e2e_test.go`, contains 8 cross-milestone test functions with 48+ subtests that exercise features across milestones end-to-end: M10 agent metadata via heartbeat, M11 profiles/teams/owners/agent-groups CRUD, M12 issuer registry verification, M13 GUI operation endpoints, M14 stats and metrics, M15 revocation and CRL, M16 notification channels, and M20 enhanced query API (sorting, cursor pagination, sparse fields, time-range filters). +**Integration tests** (`internal/integration/`) — Three test files exercising the full stack from HTTP request through router, handler, service, and repository layers. `lifecycle_test.go` covers the complete certificate lifecycle (team/owner creation through deployment and status reporting). `negative_test.go` covers error paths, endpoint validation, and revocation scenarios. `e2e_test.go` exercises cross-milestone features end-to-end (agent metadata, profiles, issuer registry, GUI operations, stats, revocation, notifications, enhanced query API). -**Frontend tests** (`web/src/api/client.test.ts`, `web/src/api/utils.test.ts`) — 86 Vitest tests covering the API client, stats/metrics endpoints, and utility functions. The API client tests mock `globalThis.fetch` and verify all endpoint functions (certificates, agents, jobs, policies, issuers, targets, notifications, audit, stats, metrics, health) send correct HTTP methods, URLs, headers, and request bodies. They also test API key management (store/retrieve/clear), auth header propagation, 401 event dispatching, and error handling (server messages, error fields, status text fallback). The stats/metrics endpoint tests verify correct query parameter handling and response shape validation. The utility tests use `vi.useFakeTimers()` for deterministic date testing and cover `formatDate`, `formatDateTime`, `timeAgo`, `daysUntil`, and `expiryColor`. The test environment uses jsdom with `@testing-library/jest-dom` matchers. +**Go integration tests** (`deploy/test/integration_test.go`) — Runs against the live Docker Compose test environment with real CA backends (Local CA, Pebble ACME, step-ca). Covers health checks, agent heartbeat, issuance, renewal, revocation, CRL/OCSP, EST enrollment, S/MIME, discovery, network scanning, and deployment verification using `crypto/x509` for cert parsing and `crypto/tls` for live TLS verification. -**CLI tests** (`internal/cli/client_test.go`) — 14 tests covering all 10 CLI subcommands with httptest mock servers, PEM parsing for bulk import, auth header verification, and JSON/table output formatting. +**Frontend tests** (`web/src/api/`) — Vitest tests covering the full API client (all endpoint functions with fetch mocking), stats/metrics endpoints, utility functions, and auth flows. Test environment uses jsdom with `@testing-library/jest-dom` matchers. -**CI pipeline** (`.github/workflows/ci.yml`) — Two parallel jobs: Go (build, vet, race detection, static analysis, vulnerability scanning, test with coverage, coverage threshold enforcement) and Frontend (TypeScript type check, Vitest test suite, Vite production build). The Go job runs `go test -race` on service, handler, middleware, and scheduler packages to catch data races. It runs `golangci-lint` with 11 linters (errcheck, govet, staticcheck, unused, gosimple, ineffassign, typecheck, gocritic, gosec, bodyclose, noctx) configured in `.golangci.yml`. It runs `govulncheck ./...` to scan dependencies for known CVEs. Coverage thresholds are enforced per-layer: service 60%, handler 60%, domain 40%, middleware 50%. These thresholds act as regression floors — they can only go up. Connector tests are included via `./internal/connector/issuer/...` and `./internal/connector/target/...` (covers Local CA, ACME, step-ca, NGINX, Apache, HAProxy, Traefik, and Caddy packages with unit tests for certificate signing logic, DNS solver, issuer validation, and deployment flows). The Frontend job runs `npx vitest run` between the TypeScript check and production build steps. +**Connector tests** (`internal/connector/`) — Issuer connectors (Local CA self-signed/sub-CA modes, ACME DNS-01/DNS-PERSIST-01, step-ca, OpenSSL, Vault PKI, DigiCert, Sectigo, Google CAS — all with httptest mock servers). Target connectors (NGINX, Apache, HAProxy, Traefik, Caddy, Envoy, IIS with mock PowerShell executor, F5 BIG-IP with mock iControl client, Postfix/Dovecot). Notifier connectors (Slack, Teams, PagerDuty, OpsGenie). -**Connector tests** (`internal/connector/`) — 57 test functions covering issuer, target, and notifier connectors. The Local CA connector has tests for self-signed and sub-CA modes (RSA, ECDSA, config validation, non-CA cert rejection). The ACME DNS solver has 10 tests for script-based DNS-01 and DNS-PERSIST-01 challenges (6 DNS-01 tests + 4 DNS-PERSIST-01 tests covering `PresentPersist` success, no-script error, script failure, and wildcard domain handling). The step-ca connector has tests with a mock HTTP server for issuance, renewal, revocation, and error paths. The OpenSSL/Custom CA connector has 14 tests covering config validation, issuance success/failure/timeout, renewal, revocation, and CRL generation. The NGINX target connector has 13 tests covering config validation, certificate deployment (file writing, permissions, validate/reload commands), and deployment validation. Apache httpd and HAProxy connectors each have 3 tests covering config validation, deployment, and validation flows. Traefik and Caddy connectors have tests covering file-based deployment and (for Caddy) dual-mode API/file configuration. Notifier connector tests span 20 tests across Slack (5), Teams (4), PagerDuty (6), and OpsGenie (5) — verifying channel identity, payload formatting, HTTP error handling, connection failures, auth headers, and configuration defaults. +**Scheduler tests** (`internal/scheduler/scheduler_test.go`) — Idempotency guards (`sync/atomic.Bool`), `WaitForCompletion` success and timeout paths, and multi-loop concurrency safety. -**Scheduler tests** (`internal/scheduler/scheduler_test.go`) — Tests for idempotency guards (`sync/atomic.Bool` CompareAndSwap prevents concurrent loop ticks), `WaitForCompletion` success and timeout paths, and multi-loop idempotency. +**Fuzz tests** (`internal/validation/`, `internal/domain/`) — Go native fuzz tests for command validation (`ValidateShellCommand`, `ValidateDomainName`, `ValidateACMEToken`) and revocation domain parsing. -**Fuzz tests** (`internal/validation/command_fuzz_test.go`, `internal/domain/revocation_fuzz_test.go`) — Go native fuzz tests (`testing/fuzz`) for command validation functions and revocation domain parsing. These exercise `ValidateShellCommand`, `ValidateDomainName`, and `ValidateACMEToken` with random inputs to discover edge cases. - -**What's not tested and why:** Postgres repository implementations (`internal/repository/postgres/`) require a real database and are tested only through integration tests, not unit tests — a `testcontainers-go` scaffolding for isolated PostgreSQL instances is planned. Target connectors for F5 BIG-IP and IIS are interface stubs (implementation planned for V3). The ACME connector requires a real ACME server (tested manually against Let's Encrypt staging). These are all candidates for future expansion as the test infrastructure matures. +**CI pipeline** (`.github/workflows/ci.yml`) — Two parallel jobs. Go: build, vet, `go test -race`, `golangci-lint` (11 linters), `govulncheck`, test with coverage, per-layer coverage threshold enforcement (service 60%, handler 60%, domain 40%, middleware 50%). Frontend: TypeScript type check, Vitest, Vite production build. ## What's Next diff --git a/docs/features.md b/docs/features.md index 9cb3cfb..0df4ab1 100644 --- a/docs/features.md +++ b/docs/features.md @@ -1286,11 +1286,11 @@ The web dashboard is the primary operational interface for certctl. Built with * - **Docker Tags** — `:latest`, `:v{version}` (`shankar0123.docker.scarf.sh/certctl-server`, `shankar0123.docker.scarf.sh/certctl-agent`) ### Test Suite -- **Unit Tests** — 1,088+ test functions across service, handler, middleware, domain layers -- **Integration Tests** — End-to-end workflows (issuance→renewal→deployment) +- **Unit Tests** — Extensive coverage across service, handler, middleware, domain, and connector layers +- **Integration Tests** — End-to-end workflows (issuance→renewal→deployment) against live Docker Compose environment - **Negative Tests** — Malformed input, nonexistent resources, error conditions -- **Frontend Tests** — 211 Vitest tests (API client, utilities, stats/metrics, full endpoint coverage) -- **Total Coverage** — 1,554+ tests (Go + frontend combined) +- **Frontend Tests** — Vitest suite covering API client, utilities, stats/metrics, and full endpoint coverage +- **CI Gates** — Per-layer coverage thresholds (service 60%, handler 60%, domain 40%, middleware 50%), race detection, static analysis, vulnerability scanning ### Licensing - **License** — Business Source License 1.1 (BSL 1.1) @@ -1492,6 +1492,6 @@ Each guide includes an evidence summary table mapping specific criteria to certc | **MCP Tools** | 76 (16 resource domains) | | **CLI Subcommands** | 10 | | **Database Tables** | 19 | -| **Test Suite** | 1,554+ tests (Go backend + frontend) | +| **Test Suite** | Extensively tested with CI-enforced coverage gates | | **Environment Variables** | 41+ configuration options | diff --git a/docs/why-certctl.md b/docs/why-certctl.md index 2af0daa..796929f 100644 --- a/docs/why-certctl.md +++ b/docs/why-certctl.md @@ -70,7 +70,7 @@ The three differentiators above get the headlines, but the feature surface is wi **Full REST API** — 97 OpenAPI 3.1-documented operations. CLI tool with 10 subcommands. Helm chart for Kubernetes deployment. Scheduled certificate digest emails. Certificate export in PEM and PKCS#12. S/MIME support with EKU-aware issuance. -**1,554 tests** — Go backend with race detection, static analysis (golangci-lint), and vulnerability scanning (govulncheck) on every commit. Frontend test suite. CI runs on every push. +**Extensively tested** — Go backend with race detection, static analysis (golangci-lint), and vulnerability scanning (govulncheck) on every commit. CI-enforced per-layer coverage thresholds. Frontend test suite. Every push is gated. ## How certctl Compares