EST RFC 7030 hardening master bundle Phases 10-11: libest sidecar e2e

+ Cisco IOS quirk fixtures + ManagedCertificate.Source provenance +
EST bulk-revoke endpoint + 13 typed audit action codes.

Phase 10.1 — libest reference-client sidecar:
- deploy/test/libest/Dockerfile: multi-stage Debian-bookworm-slim
  build of Cisco's libest v3.2.0-2 from source (autoconf/automake/
  libtool + libcurl4-openssl-dev + libssl-dev). Runtime stage
  carries only estclient + bash + openssl + ca-certificates so the
  exec surface stays small + predictable.
- docker-compose.test.yml libest-client entry (profiles: [est-e2e])
  with bind mounts for /config/est (test workspace) + /config/certs
  (certctl CA bundle for TLS pinning); IP 10.30.50.9 (10.30.50.8
  was already taken by certctl-agent).
- deploy/test/est/.gitkeep keeps the bind-mount target tracked.

Phase 10.2 — 5 integration tests (//go:build integration) in
deploy/test/est_e2e_test.go:
- TestEST_LibESTClient_Enrollment_Integration (cacerts → simpleenroll
  → cert-shape assertion)
- TestEST_LibESTClient_MTLSEnrollment_Integration (mTLS sibling-route
  cert auth; skip when bootstrap cert absent)
- TestEST_LibESTClient_ServerKeygen_Integration (RFC 7030 §4.4
  multipart; skip when profile gate disabled)
- TestEST_LibESTClient_RateLimited_Integration (4th enroll trips
  per-principal cap, asserts 429-shaped error)
- TestEST_LibESTClient_ChannelBinding_Integration (libest
  --tls-exporter; skip when libest build lacks the flag).
- requireESTSidecar guard skips the suite when the operator forgot
  --profile est-e2e; helpful error message includes the exact
  command to bring the sidecar up.

Phase 10.3 — Cisco IOS quirk fixtures + 3 unit tests in
internal/api/handler/cisco_ios_quirks_test.go:
- testdata/cisco_ios_15x_pem_csr.txt: PEM body sent with
  Content-Type application/x-pem-file. Handler dispatches on
  body-prefix not Content-Type — accepts cleanly.
- testdata/cisco_ios_16x_trailing_newline_csr.txt: extra trailing
  newlines after base64 body. strings.TrimSpace tolerates.
- testdata/cisco_ios_crlf_b64_csr.txt: CRLF-wrapped base64.
  base64.StdEncoding handles CRLF + LF identically.

Phase 11.1 — ManagedCertificate.Source provenance:
- New domain.CertificateSource enum (Unspecified/EST/SCEP/API/Agent).
- Migration 000023_managed_certificates_source.up.sql adds source
  TEXT NOT NULL DEFAULT '' so existing rows scan as
  CertificateSourceUnspecified — back-compat: bulk-revoke filter
  treats empty as "any source".
- Postgres repo Insert/Update/scan paths all wire the new column.

Phase 11.2 — EST bulk-revoke endpoint:
- BulkRevocationCriteria.Source field (Source-only requests rejected
  as too broad — must accompany at least one narrower criterion).
- service.bulk_revocation.resolveCertificates post-filter by Source
  (empty=any, no SQL change so existing CertificateFilter callers
  unaffected).
- New BulkRevocationHandler.BulkRevokeEST method pins Source=EST +
  dispatches; new route POST /api/v1/est/certificates/bulk-revoke
  (M-008 admin-gated). openapi.yaml documented + parity-guard green.

Phase 11.3 — 13 typed audit action codes in
internal/service/est_audit_actions.go:
- est_simple_enroll_success / _failed
- est_simple_reenroll_success / _failed
- est_server_keygen_success / _failed
- est_auth_failed_basic / _mtls / _channel_binding
- est_rate_limited
- est_csr_policy_violation
- est_bulk_revoke
- est_trust_anchor_reloaded
- ESTService.processEnrollment + SimpleServerKeygen + ReloadTrust
  split-emit BOTH the legacy bare action codes (back-compat for the
  GUI activity-tab chip filters that match by exact string +
  existing audit-log analysers) AND the new typed _success / _failed
  variants (operator grep target + per-failure-mode counter).

Tests:
- internal/api/handler/bulk_revocation_est_test.go — 5 cases
  (admin-true happy path pins Source=EST + non-admin 403 +
  empty-criteria 400 + invalid-reason 400 + method-not-allowed).
- internal/service/est_audit_actions_test.go — 5 cases (SimpleEnroll
  legacy+typed emission / SimpleReEnroll typed / IssuerError
  typed-failed / PolicyViolation triple-emit /
  unique-string invariant).

Pre-commit verification (sandbox): gofmt clean, go vet clean
(excluding repository/postgres testcontainers limit), staticcheck
clean across api/handler/api/router/domain/service/deploy/test,
go test -short -count=1 green for every non-postgres Go package +
integration build (`go build -tags integration ./deploy/test/...`)
clean. G-3 docs-drift guard reproduced locally clean (Phases 10-11
added zero new env vars).

Spec preserved at cowork/est-rfc7030-hardening-prompt.md. Phases
12-13 (docs/est.md + WiFi/802.1X / IoT bootstrap / FreeRADIUS
recipes; release prep + tag) remain — post-2.1.0 work.

internal/service/bulk_revocation.go

@@ -151,7 +151,24 @@ func (s *BulkRevocationService) resolveCertificates(ctx context.Context, criteri
 				filtered = append(filtered, cert)
 			}
 		}
-		return filtered, nil
+		certs = filtered
+	}
+
+	// EST RFC 7030 hardening master bundle Phase 11.2: per-source
+	// post-filter. Empty Source matches anything (back-compat); a
+	// non-empty Source narrows the result set to only certs stamped
+	// with that provenance value. Filter is applied here rather than
+	// in the SQL query so existing CertificateFilter callers are
+	// unaffected; the small per-cert pass is fine because bulk-revoke
+	// is already a low-frequency operation.
+	if criteria.Source != "" {
+		var bySource []*domain.ManagedCertificate
+		for _, cert := range certs {
+			if cert.Source == criteria.Source {
+				bySource = append(bySource, cert)
+			}
+		}
+		certs = bySource
 	}
 
 	return certs, nil

internal/service/est.go

@@ -88,15 +88,23 @@ func (s *ESTService) GetCACerts(ctx context.Context) (string, error) {
 
 // SimpleEnroll processes an initial enrollment request.
 // RFC 7030 Section 4.2: /simpleenroll accepts a PKCS#10 CSR and returns a signed cert.
+//
+// Phase 11.3: typed audit codes — the inner processEnrollment emits
+// `est_simple_enroll_success` on success + `est_simple_enroll_failed`
+// on any rejection. The legacy bare `est_simple_enroll` is retained
+// for back-compat (the GUI's activity-tab chip-filter matches by
+// prefix so both shapes render under the same chip).
 func (s *ESTService) SimpleEnroll(ctx context.Context, csrPEM string) (*domain.ESTEnrollResult, error) {
-	return s.processEnrollment(ctx, csrPEM, "est_simple_enroll")
+	return s.processEnrollment(ctx, csrPEM, "est_simple_enroll",
+		AuditActionESTSimpleEnrollSuccess, AuditActionESTSimpleEnrollFailed)
 }
 
 // SimpleReEnroll processes a re-enrollment request.
 // RFC 7030 Section 4.2.2: /simplereenroll is functionally identical to /simpleenroll
 // but is used when renewing an existing certificate.
 func (s *ESTService) SimpleReEnroll(ctx context.Context, csrPEM string) (*domain.ESTEnrollResult, error) {
-	return s.processEnrollment(ctx, csrPEM, "est_simple_reenroll")
+	return s.processEnrollment(ctx, csrPEM, "est_simple_reenroll",
+		AuditActionESTSimpleReEnrollSuccess, AuditActionESTSimpleReEnrollFailed)
 }
 
 // GetCSRAttrs returns the CSR attributes the server wants clients to include.
@@ -180,28 +188,58 @@ func (s *ESTService) GetCSRAttrs(ctx context.Context) ([]byte, error) {
 }
 
 // processEnrollment handles the common enrollment logic for both simpleenroll and simplereenroll.
-func (s *ESTService) processEnrollment(ctx context.Context, csrPEM string, auditAction string) (*domain.ESTEnrollResult, error) {
+//
+// Phase 11.3 split-emit: every audit RecordEvent call goes to BOTH the
+// legacy bare action code (auditAction param, e.g. "est_simple_enroll")
+// AND the typed success/failed code (typedSuccess / typedFailed params)
+// so existing GUI activity-tab chip filters stay green while operators
+// gain the typed grep surface.
+func (s *ESTService) processEnrollment(ctx context.Context, csrPEM, auditAction, typedSuccess, typedFailed string) (*domain.ESTEnrollResult, error) {
+	// emitFailed is the in-line helper that records BOTH the bare +
+	// typed failed-event so every error path stays one-liner. Returns
+	// the input err verbatim so call sites stay one-shot.
+	emitFailed := func(reason string, err error) {
+		if s.auditService == nil {
+			return
+		}
+		details := map[string]interface{}{
+			"reason":    reason,
+			"error":     err.Error(),
+			"protocol":  "EST",
+			"issuer_id": s.issuerID,
+		}
+		if s.profileID != "" {
+			details["profile_id"] = s.profileID
+		}
+		_ = s.auditService.RecordEvent(ctx, "est-client", "system", auditAction+"_failed", "certificate", "", details)
+		_ = s.auditService.RecordEvent(ctx, "est-client", "system", typedFailed, "certificate", "", details)
+	}
+	_ = emitFailed // referenced inside the body below
 	// Parse the CSR to extract CN and SANs
 	block, _ := pem.Decode([]byte(csrPEM))
 	if block == nil {
 		s.counters.inc(estCounterCSRInvalid)
+		emitFailed("csr_pem_decode", fmt.Errorf("invalid CSR PEM"))
 		return nil, fmt.Errorf("invalid CSR PEM")
 	}
 
 	csr, err := x509.ParseCertificateRequest(block.Bytes)
 	if err != nil {
 		s.counters.inc(estCounterCSRInvalid)
+		emitFailed("csr_parse", err)
 		return nil, fmt.Errorf("failed to parse CSR: %w", err)
 	}
 
 	if err := csr.CheckSignature(); err != nil {
 		s.counters.inc(estCounterCSRSignatureMismatch)
+		emitFailed("csr_signature", err)
 		return nil, fmt.Errorf("CSR signature verification failed: %w", err)
 	}
 
 	commonName := csr.Subject.CommonName
 	if commonName == "" {
 		s.counters.inc(estCounterCSRInvalid)
+		emitFailed("csr_missing_cn", fmt.Errorf("missing CN"))
 		return nil, fmt.Errorf("CSR must include a Common Name")
 	}
 
@@ -231,6 +269,15 @@ func (s *ESTService) processEnrollment(ctx context.Context, csrPEM string, audit
 	}
 	if _, csrErr := ValidateCSRAgainstProfile(csrPEM, profile); csrErr != nil {
 		s.counters.inc(estCounterCSRPolicyViolation)
+		// Emit BOTH the typed-failed code (for the Activity tab) AND
+		// the standalone est_csr_policy_violation code (for the
+		// per-failure-mode counter that ops greppers prefer).
+		emitFailed("csr_policy_violation", csrErr)
+		if s.auditService != nil {
+			_ = s.auditService.RecordEvent(ctx, "est-client", "system",
+				AuditActionESTCSRPolicyViolation, "certificate", "",
+				map[string]interface{}{"error": csrErr.Error(), "issuer_id": s.issuerID, "profile_id": s.profileID})
+		}
 		s.logger.Error("EST enrollment rejected: crypto policy violation",
 			"action", auditAction,
 			"common_name", commonName,
@@ -262,6 +309,7 @@ func (s *ESTService) processEnrollment(ctx context.Context, csrPEM string, audit
 	result, err := s.issuer.IssueCertificate(ctx, commonName, sans, csrPEM, ekus, maxTTLSeconds, mustStaple)
 	if err != nil {
 		s.counters.inc(estCounterIssuerError)
+		emitFailed("issuer_error", err)
 		s.logger.Error("EST enrollment failed",
 			"action", auditAction,
 			"common_name", commonName,
@@ -276,7 +324,10 @@ func (s *ESTService) processEnrollment(ctx context.Context, csrPEM string, audit
 		s.counters.inc(estCounterSuccessSimpleEnroll)
 	}
 
-	// Audit the enrollment
+	// Audit the enrollment — split-emit per Phase 11.3: legacy bare
+	// action code (back-compat for the GUI activity tab + existing
+	// audit-log analysers) + typed _success suffix variant + the
+	// canonical typed code from the AuditAction* constants.
 	if s.auditService != nil {
 		details := map[string]interface{}{
 			"common_name": commonName,
@@ -289,6 +340,7 @@ func (s *ESTService) processEnrollment(ctx context.Context, csrPEM string, audit
 			details["profile_id"] = s.profileID
 		}
 		_ = s.auditService.RecordEvent(ctx, "est-client", "system", auditAction, "certificate", result.Serial, details)
+		_ = s.auditService.RecordEvent(ctx, "est-client", "system", typedSuccess, "certificate", result.Serial, details)
 	}
 
 	s.logger.Info("EST enrollment successful",
@@ -524,6 +576,8 @@ func (s *ESTService) SimpleServerKeygen(ctx context.Context, csrPEM string) (*ES
 			details["profile_id"] = s.profileID
 		}
 		_ = s.auditService.RecordEvent(ctx, "est-client", "system", "est_server_keygen", "certificate", issued.Serial, details)
+		// Phase 11.3: typed _success suffix for the operator grep surface.
+		_ = s.auditService.RecordEvent(ctx, "est-client", "system", AuditActionESTServerKeygenSuccess, "certificate", issued.Serial, details)
 	}
 	s.logger.Info("EST serverkeygen successful",
 		"common_name", commonName, "serial", issued.Serial,

internal/service/est_audit_actions.go

@@ -0,0 +1,40 @@
+package service
+
+// EST RFC 7030 hardening master bundle Phase 11.3 — typed audit action
+// codes. Each maps to a unique counter label so operators grep the
+// audit log on these exact strings.
+//
+// Naming contract: every code is `est_<flow>_<outcome>` where
+//
+//	<flow>    = simple_enroll | simple_reenroll | server_keygen | auth_failed_<mode> | rate_limited | csr_policy_violation | bulk_revoke | trust_anchor_reloaded
+//	<outcome> = success | failed   (only on the three success/failure-paired flows)
+//
+// Pre-Phase-11 the audit log carried bare action codes (est_simple_enroll
+// without the _success suffix). The GUI activity-tab filter chips
+// (web/src/pages/ESTAdminPage.tsx) match by `startsWith()` after the
+// Phase 11 cutover so both old + new strings continue to render under
+// the right chip.
+const (
+	// Three success/failure-paired enrollment flows. The success codes
+	// share a prefix with the legacy bare codes so a deployment running
+	// the old audit-log analyser continues to find every enrollment.
+	AuditActionESTSimpleEnrollSuccess   = "est_simple_enroll_success"
+	AuditActionESTSimpleEnrollFailed    = "est_simple_enroll_failed"
+	AuditActionESTSimpleReEnrollSuccess = "est_simple_reenroll_success"
+	AuditActionESTSimpleReEnrollFailed  = "est_simple_reenroll_failed"
+	AuditActionESTServerKeygenSuccess   = "est_server_keygen_success"
+	AuditActionESTServerKeygenFailed    = "est_server_keygen_failed"
+
+	// Per-mode auth-failure codes. Emitted by the handler at the auth-
+	// gate trip points so operators can filter "Basic-auth failures
+	// from this source IP" cleanly.
+	AuditActionESTAuthFailedBasic          = "est_auth_failed_basic"
+	AuditActionESTAuthFailedMTLS           = "est_auth_failed_mtls"
+	AuditActionESTAuthFailedChannelBinding = "est_auth_failed_channel_binding"
+
+	// Operational events.
+	AuditActionESTRateLimited         = "est_rate_limited"
+	AuditActionESTCSRPolicyViolation  = "est_csr_policy_violation"
+	AuditActionESTBulkRevoke          = "est_bulk_revoke"
+	AuditActionESTTrustAnchorReloaded = "est_trust_anchor_reloaded"
+)

internal/service/est_audit_actions_test.go

@@ -0,0 +1,156 @@
+package service
+
+import (
+	"context"
+	"errors"
+	"io"
+	"log/slog"
+	"strings"
+	"testing"
+
+	"github.com/shankar0123/certctl/internal/domain"
+)
+
+// EST RFC 7030 hardening master bundle Phase 11.4 — audit-code assertions.
+// Drive each code path through a real ESTService instance + assert the
+// typed action codes land in the audit log alongside the legacy bare
+// codes (back-compat preservation).
+
+func newAuditAssertService(t *testing.T) (*ESTService, *mockAuditRepo) {
+	t.Helper()
+	auditRepo := newMockAuditRepository()
+	auditSvc := NewAuditService(auditRepo)
+	silent := slog.New(slog.NewTextHandler(io.Discard, &slog.HandlerOptions{Level: slog.LevelError + 10}))
+	svc := NewESTService("iss-corp", &mockIssuerConnector{}, auditSvc, silent)
+	return svc, auditRepo
+}
+
+// auditActions returns the action codes recorded across every audit
+// event in the repo, in emission order. Used to assert that the
+// typed _success / _failed events fire in the right order alongside
+// the legacy bare codes.
+func auditActions(repo *mockAuditRepo) []string {
+	out := make([]string, 0, len(repo.Events))
+	for _, e := range repo.Events {
+		out = append(out, e.Action)
+	}
+	return out
+}
+
+func TestESTAudit_SimpleEnrollSuccess_EmitsLegacyAndTyped(t *testing.T) {
+	svc, repo := newAuditAssertService(t)
+	csrPEM := generateCSRPEM(t, "device.example.com", []string{"device.example.com"})
+	if _, err := svc.SimpleEnroll(context.Background(), csrPEM); err != nil {
+		t.Fatalf("SimpleEnroll: %v", err)
+	}
+	got := auditActions(repo)
+	wantBare := "est_simple_enroll"
+	wantTyped := AuditActionESTSimpleEnrollSuccess // est_simple_enroll_success
+	if !stringSliceContains(got, wantBare) {
+		t.Errorf("missing legacy bare code %q in %v", wantBare, got)
+	}
+	if !stringSliceContains(got, wantTyped) {
+		t.Errorf("missing typed code %q in %v", wantTyped, got)
+	}
+}
+
+func TestESTAudit_SimpleReEnrollSuccess_EmitsTyped(t *testing.T) {
+	svc, repo := newAuditAssertService(t)
+	csrPEM := generateCSRPEM(t, "device.example.com", nil)
+	if _, err := svc.SimpleReEnroll(context.Background(), csrPEM); err != nil {
+		t.Fatalf("SimpleReEnroll: %v", err)
+	}
+	if !stringSliceContains(auditActions(repo), AuditActionESTSimpleReEnrollSuccess) {
+		t.Errorf("missing %q; got %v", AuditActionESTSimpleReEnrollSuccess, auditActions(repo))
+	}
+}
+
+func TestESTAudit_IssuerError_EmitsTypedFailed(t *testing.T) {
+	auditRepo := newMockAuditRepository()
+	auditSvc := NewAuditService(auditRepo)
+	silent := slog.New(slog.NewTextHandler(io.Discard, &slog.HandlerOptions{Level: slog.LevelError + 10}))
+	svc := NewESTService("iss-corp", &mockIssuerConnector{Err: errors.New("CA down")}, auditSvc, silent)
+	csrPEM := generateCSRPEM(t, "device.example.com", nil)
+	if _, err := svc.SimpleEnroll(context.Background(), csrPEM); err == nil {
+		t.Fatal("expected enroll error")
+	}
+	if !stringSliceContains(auditActions(auditRepo), AuditActionESTSimpleEnrollFailed) {
+		t.Errorf("missing typed failure code; got %v", auditActions(auditRepo))
+	}
+	// And the bare _failed variant for back-compat:
+	if !stringSliceContains(auditActions(auditRepo), "est_simple_enroll_failed") {
+		t.Errorf("missing bare _failed variant; got %v", auditActions(auditRepo))
+	}
+}
+
+func TestESTAudit_PolicyViolation_EmitsTypedAndStandalone(t *testing.T) {
+	svc, repo := newAuditAssertService(t)
+	repoMock := newMockProfileRepository()
+	svc.SetProfileRepo(repoMock)
+	svc.SetProfileID("prof-tight")
+	repoMock.AddProfile(&domain.CertificateProfile{
+		ID:                   "prof-tight",
+		Name:                 "tight",
+		AllowedKeyAlgorithms: []domain.KeyAlgorithmRule{{Algorithm: "RSA", MinSize: 4096}}, // ECDSA-P256 CSR fails
+		Enabled:              true,
+	})
+	csrPEM := generateCSRPEM(t, "device.example.com", nil) // ECDSA-P256
+	if _, err := svc.SimpleEnroll(context.Background(), csrPEM); err == nil {
+		t.Fatal("expected policy violation error")
+	}
+	got := auditActions(repo)
+	if !stringSliceContains(got, AuditActionESTCSRPolicyViolation) {
+		t.Errorf("missing standalone policy-violation code %q; got %v", AuditActionESTCSRPolicyViolation, got)
+	}
+	if !stringSliceContains(got, AuditActionESTSimpleEnrollFailed) {
+		t.Errorf("missing typed failed code; got %v", got)
+	}
+}
+
+func TestESTAudit_AuditCodesAreUniqueStrings(t *testing.T) {
+	// Tiny invariant test: every audit-action constant is a non-empty
+	// distinct string. Prevents a future cut-paste typo where two
+	// constants share the same value.
+	codes := []string{
+		AuditActionESTSimpleEnrollSuccess,
+		AuditActionESTSimpleEnrollFailed,
+		AuditActionESTSimpleReEnrollSuccess,
+		AuditActionESTSimpleReEnrollFailed,
+		AuditActionESTServerKeygenSuccess,
+		AuditActionESTServerKeygenFailed,
+		AuditActionESTAuthFailedBasic,
+		AuditActionESTAuthFailedMTLS,
+		AuditActionESTAuthFailedChannelBinding,
+		AuditActionESTRateLimited,
+		AuditActionESTCSRPolicyViolation,
+		AuditActionESTBulkRevoke,
+		AuditActionESTTrustAnchorReloaded,
+	}
+	seen := map[string]bool{}
+	for _, c := range codes {
+		if c == "" {
+			t.Errorf("empty audit-action constant")
+		}
+		if !strings.HasPrefix(c, "est_") {
+			t.Errorf("audit-action constant %q must start with est_", c)
+		}
+		if seen[c] {
+			t.Errorf("duplicate audit-action constant: %q", c)
+		}
+		seen[c] = true
+	}
+}
+
+func stringSliceContains(haystack []string, needle string) bool {
+	for _, s := range haystack {
+		if s == needle {
+			return true
+		}
+	}
+	return false
+}
+
+// silenceUnusedDomain keeps the domain import live when the policy-
+// violation test compiles even if a future refactor removes the only
+// reference site.
+var _ domain.CertificateProfile

internal/service/est_counters.go

@@ -1,6 +1,7 @@
 package service
 
 import (
+	"context"
 	"sync/atomic"
 	"time"
 
@@ -174,11 +175,27 @@ func (s *ESTService) Stats(now time.Time) ESTStatsSnapshot {
 //
 // Returns ErrESTMTLSDisabled when the profile doesn't have an mTLS
 // trust anchor configured (admin handler maps to HTTP 409).
+//
+// Phase 11.3: emits AuditActionESTTrustAnchorReloaded on successful
+// reload so operators have a typed grep target for "who rotated the
+// trust bundle for which profile + when".
 func (s *ESTService) ReloadTrust() error {
 	if s.estTrustAnchor == nil {
 		return ErrESTMTLSDisabled
 	}
-	return s.estTrustAnchor.Reload()
+	if err := s.estTrustAnchor.Reload(); err != nil {
+		return err
+	}
+	if s.auditService != nil {
+		details := map[string]interface{}{
+			"path_id":           s.estPathIDForLog,
+			"trust_anchor_path": s.estTrustAnchor.Path(),
+			"protocol":          "EST",
+		}
+		_ = s.auditService.RecordEvent(context.Background(), "est-admin", "system",
+			AuditActionESTTrustAnchorReloaded, "trust_anchor", s.estPathIDForLog, details)
+	}
+	return nil
 }
 
 // ErrESTMTLSDisabled signals the admin handler that an EST profile