mirror of
https://github.com/shankar0123/certctl.git
synced 2026-06-07 15:51:30 +00:00
metrics: add per-issuer-type issuance counters, histogram, and failure classifier
Closes the #4 acquisition-readiness blocker from the 2026-05-01 issuer coverage audit. Before this commit, certctl's Prometheus exposition had zero per-issuer-type signal — operators answering "is DigiCert slow?" or "is Sectigo failing more than ACME?" had to grep logs by issuer name. This commit adds three series labelled by issuer type: certctl_issuance_total{issuer_type, outcome} certctl_issuance_duration_seconds{issuer_type} (histogram) certctl_issuance_failures_total{issuer_type, error_class} The histogram covers 0.05–120 second buckets to span the local-issuer fast path and async-CA slow path (DigiCert/Sectigo/Entrust polling can take minutes). error_class is a closed enum of eight values (timeout, auth, rate_limited, validation, upstream_5xx, upstream_4xx, network, other) classified once in service.ClassifyError. Cardinality budget is ~276 new series, well within Prometheus's comfortable range. Implementation: - service.IssuanceMetrics is the thread-safe counter + histogram table. Three independent views (counters / failures / durations) exposed via SnapshotCounters / SnapshotFailures / SnapshotDurations. sync.RWMutex protects the map shape; per-key sync/atomic.Uint64 primitives keep the recording hot path lock-free under concurrent service-layer goroutines. - service.IssuanceCounterEntry / IssuanceFailureEntry / IssuanceDurationEntry / IssuanceMetricsSnapshotter live in service (not handler) to avoid an import cycle: handler already imports service for admin_est.go etc., so service can't import handler back. Handler's exposer takes the snapshotter via the service-defined interface. - service.ClassifyError pure function maps error → error_class. context.DeadlineExceeded / context.Canceled → timeout; *net.OpError → network; substring matches against canonical AWS / DigiCert / Sectigo error shapes for auth / rate_limited / validation / upstream_5xx / upstream_4xx / network; unknown → other. Each branch has at least one representative test case in TestClassifyError. - IssuerConnectorAdapter.SetMetrics wires per-adapter recording (issuerType + metrics). Existing 28+ test call sites of NewIssuerConnectorAdapter keep their one-arg signature; production wiring goes through SetMetrics post-construction. - IssuerRegistry.SetIssuanceMetrics + Rebuild type-asserts to *IssuerConnectorAdapter and calls SetMetrics with the issuer type string. nil-guarded — tests that hand-build adapters without metrics get no-op recording. - IssuerConnectorAdapter.IssueCertificate / RenewCertificate wrap the underlying connector call with start := time.Now() and recordIssuance(start, err). Renewal is recorded into the same certctl_issuance_* series as initial issuance — operationally, renewal IS issuance from the connector's perspective (matches the audit prompt's guidance on series naming). - handler/metrics.go GetPrometheusMetrics gains a new exposer block emitting all three series in stable label order with correct Prometheus format (_bucket / _sum / _count for the histogram, +Inf bucket appended). Sorted via sort.Slice for stable output. nil- guarded so deploys without the wire produce clean exposition. - formatLE helper trims trailing zeros from histogram bucket labels via strconv.FormatFloat(le, 'f', -1, 64) so the `le` labels match Prometheus client conventions ("0.05", "30", "120", not "0.0500" etc.). - cmd/server/main.go wires a single IssuanceMetrics instance into both the IssuerRegistry (recording) and the MetricsHandler (exposer) using DefaultIssuanceBucketBoundaries. Tests: - TestIssuanceMetrics_RecordAndSnapshot — happy-path counter + histogram + failure recording, BucketBoundaries returns a copy (not shared storage). - TestIssuanceMetrics_HistogramCumulative — pins the cumulative-buckets contract. 100ms observation lands in 0.1 bucket and every larger bucket; 750ms only in the 1.0 bucket. Off-by-one here would corrupt every quantile query downstream. - TestIssuanceMetrics_Concurrency — 100 goroutines × 1000 ops under the race detector. Asserts atomic counter integrity across contended writes. - TestClassifyError — 17 cases covering every branch of the closed enum plus the nil-error special case. Implementation chooses the existing hand-rolled fmt.Fprintf exposition pattern (no prometheus/client_golang dependency added) to stay consistent with the OCSP / deploy counter blocks already in the file. Out of scope (separate follow-ups): - Revocation metrics (certctl_revocation_*) — symmetric to issuance but the audit didn't ask; explicit follow-up commit. - Discovery / health-check duration histograms. - prometheus/client_golang migration. Verified locally: - gofmt clean - go vet ./... clean - staticcheck ./... clean - golangci-lint run --timeout 5m ./... → 0 issues - go test -short -count=1 ./internal/service/ green - go test -short -count=1 -race -run TestIssuanceMetrics ./internal/service/ green - go test -short -count=1 ./internal/api/handler/ green - go build ./... success Audit reference: cowork/issuer-coverage-audit-2026-05-01/RESULTS.md Top-10 fix #4 (Part 3, narrative section).
This commit is contained in:
shankar0123
@@ -6,9 +6,11 @@ import (
|
||||
"fmt"
|
||||
"net/http"
|
||||
"sort"
|
||||
"strconv"
|
||||
"time"
|
||||
|
||||
"github.com/shankar0123/certctl/internal/api/middleware"
|
||||
"github.com/shankar0123/certctl/internal/service"
|
||||
)
|
||||
|
||||
// MetricsService defines the service interface for metrics collection.
|
||||
@@ -51,6 +53,13 @@ type DeployCounterSnapshotter interface {
|
||||
Snapshot() []DeploySnapshotEntry
|
||||
}
|
||||
|
||||
// IssuanceCounterEntry / IssuanceFailureEntry / IssuanceDurationEntry
|
||||
// and the IssuanceMetricsSnapshotter interface live in
|
||||
// internal/service (issuance_metrics.go). Handler can't define them
|
||||
// locally because internal/api/handler is imported by service — the
|
||||
// reverse import would create a cycle. The exposer below takes the
|
||||
// types via the interface defined in service.
|
||||
|
||||
// MetricsHandler handles HTTP requests for metrics.
|
||||
// Supports both JSON format (GET /api/v1/metrics) and Prometheus exposition format
|
||||
// (GET /api/v1/metrics/prometheus) for integration with Prometheus, Grafana, Datadog, etc.
|
||||
@@ -64,6 +73,12 @@ type MetricsHandler struct {
|
||||
ocspCounters CounterSnapshotter
|
||||
// Phase 10 (deploy-hardening I) — per-target-type deploy counters.
|
||||
deployCounters DeployCounterSnapshotter
|
||||
// Per-issuer-type issuance metrics (audit fix #4). nil disables
|
||||
// the new metric block; main.go wires the instance at startup.
|
||||
// The interface lives in service to avoid an import cycle (handler
|
||||
// imports service for admin_est.go etc., so service can't import
|
||||
// handler back).
|
||||
issuanceCounters service.IssuanceMetricsSnapshotter
|
||||
}
|
||||
|
||||
// NewMetricsHandler creates a new MetricsHandler with a service dependency.
|
||||
@@ -89,6 +104,14 @@ func (h *MetricsHandler) SetDeployCounters(c DeployCounterSnapshotter) {
|
||||
h.deployCounters = c
|
||||
}
|
||||
|
||||
// SetIssuanceCounters wires the per-issuer-type issuance metrics for
|
||||
// the Prometheus exposition. nil disables the block. Closes the #4
|
||||
// acquisition-readiness blocker from the 2026-05-01 issuer coverage
|
||||
// audit (per-issuer-type metrics).
|
||||
func (h *MetricsHandler) SetIssuanceCounters(c service.IssuanceMetricsSnapshotter) {
|
||||
h.issuanceCounters = c
|
||||
}
|
||||
|
||||
// MetricsResponse represents the JSON metrics response for V2.
|
||||
type MetricsResponse struct {
|
||||
Gauge MetricsGauge `json:"gauge"`
|
||||
@@ -344,6 +367,71 @@ func (h MetricsHandler) GetPrometheusMetrics(w http.ResponseWriter, r *http.Requ
|
||||
fmt.Fprintf(w, "certctl_deploy_idempotent_skip_total{target_type=%q} %d\n", s.TargetType, s.IdempotentSkips)
|
||||
}
|
||||
}
|
||||
|
||||
// Per-issuer-type issuance metrics (audit fix #4). Three series:
|
||||
// certctl_issuance_total{issuer_type, outcome} counter
|
||||
// certctl_issuance_duration_seconds{issuer_type} histogram
|
||||
// certctl_issuance_failures_total{issuer_type, error_class} counter
|
||||
//
|
||||
// Cardinality: 12 issuer_types × 2 outcomes (24) +
|
||||
// 12 × 11 buckets+sum+count (~156) +
|
||||
// 12 × 8 error_classes (96) = ~276 series. Comfortable
|
||||
// for any Prometheus instance.
|
||||
if h.issuanceCounters != nil {
|
||||
// certctl_issuance_total
|
||||
fmt.Fprintf(w, "\n# HELP certctl_issuance_total Total certificate issuance attempts, labelled by issuer type and outcome.\n")
|
||||
fmt.Fprintf(w, "# TYPE certctl_issuance_total counter\n")
|
||||
counters := h.issuanceCounters.SnapshotCounters()
|
||||
sort.Slice(counters, func(i, j int) bool {
|
||||
if counters[i].IssuerType != counters[j].IssuerType {
|
||||
return counters[i].IssuerType < counters[j].IssuerType
|
||||
}
|
||||
return counters[i].Outcome < counters[j].Outcome
|
||||
})
|
||||
for _, c := range counters {
|
||||
fmt.Fprintf(w, "certctl_issuance_total{issuer_type=%q,outcome=%q} %d\n", c.IssuerType, c.Outcome, c.Count)
|
||||
}
|
||||
|
||||
// certctl_issuance_duration_seconds histogram
|
||||
fmt.Fprintf(w, "\n# HELP certctl_issuance_duration_seconds Certificate issuance duration in seconds, labelled by issuer type. Cumulative histogram with +Inf.\n")
|
||||
fmt.Fprintf(w, "# TYPE certctl_issuance_duration_seconds histogram\n")
|
||||
durations := h.issuanceCounters.SnapshotDurations()
|
||||
boundaries := h.issuanceCounters.BucketBoundaries()
|
||||
sort.Slice(durations, func(i, j int) bool { return durations[i].IssuerType < durations[j].IssuerType })
|
||||
for _, d := range durations {
|
||||
for i, le := range boundaries {
|
||||
if i < len(d.Buckets) {
|
||||
fmt.Fprintf(w, "certctl_issuance_duration_seconds_bucket{issuer_type=%q,le=%q} %d\n",
|
||||
d.IssuerType, formatLE(le), d.Buckets[i])
|
||||
}
|
||||
}
|
||||
fmt.Fprintf(w, "certctl_issuance_duration_seconds_bucket{issuer_type=%q,le=\"+Inf\"} %d\n", d.IssuerType, d.Count)
|
||||
fmt.Fprintf(w, "certctl_issuance_duration_seconds_sum{issuer_type=%q} %g\n", d.IssuerType, d.Sum)
|
||||
fmt.Fprintf(w, "certctl_issuance_duration_seconds_count{issuer_type=%q} %d\n", d.IssuerType, d.Count)
|
||||
}
|
||||
|
||||
// certctl_issuance_failures_total
|
||||
fmt.Fprintf(w, "\n# HELP certctl_issuance_failures_total Issuance failures by issuer type and error class. error_class is a closed enum (timeout, auth, rate_limited, validation, upstream_5xx, upstream_4xx, network, other).\n")
|
||||
fmt.Fprintf(w, "# TYPE certctl_issuance_failures_total counter\n")
|
||||
failures := h.issuanceCounters.SnapshotFailures()
|
||||
sort.Slice(failures, func(i, j int) bool {
|
||||
if failures[i].IssuerType != failures[j].IssuerType {
|
||||
return failures[i].IssuerType < failures[j].IssuerType
|
||||
}
|
||||
return failures[i].ErrorClass < failures[j].ErrorClass
|
||||
})
|
||||
for _, f := range failures {
|
||||
fmt.Fprintf(w, "certctl_issuance_failures_total{issuer_type=%q,error_class=%q} %d\n", f.IssuerType, f.ErrorClass, f.Count)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// formatLE formats a histogram bucket boundary the way Prometheus
|
||||
// expects: no trailing zeros, no scientific notation for typical
|
||||
// sub-second / sub-minute values. Used for the `le` label in the
|
||||
// issuance-duration histogram exposer.
|
||||
func formatLE(v float64) string {
|
||||
return strconv.FormatFloat(v, 'f', -1, 64)
|
||||
}
|
||||
|
||||
// DashboardSummary mirrors the service.DashboardSummary for JSON unmarshaling.
|
||||
|
||||
Reference in New Issue
Block a user