Close I-001 (RetryFailedJobs never invoked) coverage-gap finding

Operator decision answered as Option A: JobService.RetryFailedJobs is
now wired into the scheduler as an always-on 10th loop. Prior to this
commit the method was implemented, unit-tested, and exported but had
zero runtime callers — any job that transitioned to status=Failed stayed
Failed forever regardless of how many attempts it had remaining.

Scheduler — 10th loop:
  internal/scheduler/scheduler.go grows a jobRetryLoop alongside the
  existing nine loops (renewal, jobs, health, notifications, short-lived,
  network scan, digest, health check, cloud discovery). The loop follows
  the established run-immediately-then-tick pattern (same shape as
  jobProcessorLoop), gated by a sync/atomic.Bool idempotency guard and
  joined into the scheduler's sync.WaitGroup so WaitForCompletion drains
  it on graceful shutdown. Each tick runs under a 2-minute context
  timeout mirroring jobProcessorLoop's opCtx budget. The runJobRetry
  helper invokes jobService.RetryFailedJobs(ctx, 3) — the advisory
  maxRetries cap is belt-and-suspenders; per-job eligibility is still
  enforced inside the service via Attempts < MaxAttempts.

  The JobServicer scheduler-interface gains RetryFailedJobs so the
  scheduler's dependency surface stays explicit and mockable.

Service — audit trail per retry:
  internal/service/job.go:RetryFailedJobs now emits an audit event for
  every Failed→Pending transition. Following the house convention used
  by all scheduler-emitted events, actor='system' and actorType=
  domain.ActorTypeSystem; action='job_retry'; details capture
  old_status, new_status, attempts, max_attempts. JobService carries an
  optional *AuditService (SetAuditService) that nil-guards to preserve
  test-wiring ergonomics — existing tests that construct JobService
  without an audit service continue to pass unchanged.

Config — env var with sane default:
  internal/config/config.go:SchedulerConfig grows RetryInterval, wired
  to CERTCTL_SCHEDULER_RETRY_INTERVAL with a 5-minute default. Validate
  rejects intervals below 1 second (matches other scheduler interval
  validators).

Server wiring:
  cmd/server/main.go calls jobService.SetAuditService(auditService)
  after JobService construction and sched.SetJobRetryInterval(
  cfg.Scheduler.RetryInterval) alongside the other SetXxxInterval calls.

Regression coverage:
  internal/service/job_test.go (3 new)
    - TestJobService_RetryFailedJobs_EligibleJobTransitionsAndAudits
    - TestJobService_RetryFailedJobs_SkipsJobsAtMaxAttempts
    - TestJobService_RetryFailedJobs_NoAuditServiceOK
  internal/scheduler/scheduler_test.go (3 new)
    - TestScheduler_JobRetryLoop_CallsService
    - TestScheduler_JobRetryLoop_IdempotencyGuard
    - TestScheduler_JobRetryLoop_WaitForCompletion

  The service tests assert status transitions, attempt-cap short-
  circuiting, and audit event shape (actor='system', action='job_retry',
  details keys). The scheduler tests assert the loop invokes the service,
  the atomic.Bool guard skips overlapping ticks with the expected
  'still running, skipping tick' log, and WaitForCompletion drains the
  in-flight tick on Stop.

Residual follow-up (not in scope for this commit):
  internal/service/renewal.go:RetryFailedJobs is a parallel dead-code
  duplicate of the same logic on RenewalService — untested and has no
  runtime caller. The audit finding called this out as 'implemented
  twice'. Removing it is a separate cleanup and does not block the
  Option-A wiring this commit delivers.

Files:
  cmd/server/main.go                     — SetAuditService + SetJobRetryInterval
  internal/config/config.go              — RetryInterval field + env + validate
  internal/scheduler/scheduler.go        — 10th loop, interface, field, setter
  internal/scheduler/scheduler_test.go   — 3 new scheduler-loop tests
  internal/service/job.go                — RetryFailedJobs audit emission + SetAuditService
  internal/service/job_test.go           — 3 new service-layer tests

internal/service/job.go

@@ -26,6 +26,7 @@ type JobService struct {
 	ownerRepo         repository.OwnerRepository
 	renewalService    *RenewalService
 	deploymentService *DeploymentService
+	auditService      *AuditService
 	logger            *slog.Logger
 }
 
@@ -54,6 +55,15 @@ func NewJobService(
 	}
 }
 
+// SetAuditService wires an optional audit service for emitting lifecycle
+// events (e.g., scheduler-driven job_retry transitions recorded by
+// RetryFailedJobs). Construction keeps the audit dependency optional so
+// bootstrap/test wiring that doesn't exercise the retry path can omit it;
+// production wiring in cmd/server/main.go should always call this.
+func (s *JobService) SetAuditService(a *AuditService) {
+	s.auditService = a
+}
+
 // ProcessPendingJobs fetches and processes all pending jobs.
 // It routes jobs to the appropriate service based on job type and handles errors gracefully.
 //
@@ -163,6 +173,16 @@ func (s *JobService) processValidationJob(ctx context.Context, job *domain.Job)
 
 // RetryFailedJobs finds failed jobs and resets them for retry.
 // It only retries jobs that haven't exceeded max attempts.
+//
+// Audit trail (I-001): each successful Failed → Pending transition emits a
+// "job_retry" audit event with actor "system" (ActorTypeSystem), capturing
+// the old→new state and attempt counters so operators can reconstruct
+// scheduler-driven retry activity. The audit service is optional — callers
+// that haven't wired it via SetAuditService simply skip emission.
+//
+// maxRetries is retained for interface compatibility with
+// scheduler.JobServicer but is advisory: per-job eligibility is governed by
+// each job's own Attempts vs. MaxAttempts, not this parameter.
 func (s *JobService) RetryFailedJobs(ctx context.Context, maxRetries int) error {
 	s.logger.Debug("retrying failed jobs", "max_retries", maxRetries)
 
@@ -191,6 +211,21 @@ func (s *JobService) RetryFailedJobs(ctx context.Context, maxRetries int) error
 			continue
 		}
 
+		if s.auditService != nil {
+			if auditErr := s.auditService.RecordEvent(ctx, "system", domain.ActorTypeSystem,
+				"job_retry", "job", job.ID,
+				map[string]interface{}{
+					"old_status":   string(domain.JobStatusFailed),
+					"new_status":   string(domain.JobStatusPending),
+					"attempts":     job.Attempts,
+					"max_attempts": job.MaxAttempts,
+				}); auditErr != nil {
+				s.logger.Error("failed to record job retry audit event",
+					"job_id", job.ID,
+					"error", auditErr)
+			}
+		}
+
 		retriedCount++
 	}
 

internal/service/job_test.go

@@ -2,6 +2,7 @@ package service
 
 import (
 	"context"
+	"encoding/json"
 	"errors"
 	"log/slog"
 	"os"
@@ -398,3 +399,180 @@ func TestRejectJob_SelfRejection_Permitted(t *testing.T) {
 			jobRepo.StatusUpdates["job-self"])
 	}
 }
+
+// --- I-001: scheduler-driven retry emits audit events ---
+//
+// These regression tests prove that RetryFailedJobs (a) transitions eligible
+// Failed jobs to Pending, (b) skips jobs that have exhausted their max
+// attempts, and (c) records a "job_retry" audit event per transition when the
+// audit service is wired. A separate variant (_NoAuditServiceOK) confirms the
+// nil-guard path so test/bootstrap wiring that skips the setter still works.
+
+// newTestJobServiceWithAudit wires the optional audit dependency onto the
+// standard test JobService so retry assertions can inspect recorded events.
+// Mirrors newTestJobServiceWithRepos but also returns the mock audit repo
+// holding any emitted events.
+func newTestJobServiceWithAudit(jobRepo *mockJobRepo) (*JobService, *mockAuditRepo) {
+	svc, _, _ := newTestJobServiceWithRepos(jobRepo)
+	auditRepo := &mockAuditRepo{}
+	svc.SetAuditService(NewAuditService(auditRepo))
+	return svc, auditRepo
+}
+
+func TestJobService_RetryFailedJobs_EligibleJobTransitionsAndAudits(t *testing.T) {
+	ctx := context.Background()
+
+	now := time.Now()
+	failed := &domain.Job{
+		ID:            "job-retry-1",
+		Type:          domain.JobTypeRenewal,
+		CertificateID: "cert-001",
+		Status:        domain.JobStatusFailed,
+		Attempts:      1,
+		MaxAttempts:   3,
+		CreatedAt:     now,
+		ScheduledAt:   now,
+	}
+	jobRepo := &mockJobRepo{
+		Jobs:          map[string]*domain.Job{failed.ID: failed},
+		StatusUpdates: make(map[string]domain.JobStatus),
+	}
+
+	svc, auditRepo := newTestJobServiceWithAudit(jobRepo)
+
+	if err := svc.RetryFailedJobs(ctx, 3); err != nil {
+		t.Fatalf("RetryFailedJobs failed: %v", err)
+	}
+
+	if got := jobRepo.StatusUpdates[failed.ID]; got != domain.JobStatusPending {
+		t.Fatalf("expected job %s status Pending after retry, got %s", failed.ID, got)
+	}
+
+	if len(auditRepo.Events) != 1 {
+		t.Fatalf("expected 1 audit event, got %d", len(auditRepo.Events))
+	}
+
+	ev := auditRepo.Events[0]
+	if ev.Action != "job_retry" {
+		t.Errorf("expected action job_retry, got %s", ev.Action)
+	}
+	if ev.Actor != "system" {
+		t.Errorf("expected actor system, got %s", ev.Actor)
+	}
+	if ev.ActorType != domain.ActorTypeSystem {
+		t.Errorf("expected actor type System, got %s", ev.ActorType)
+	}
+	if ev.ResourceType != "job" {
+		t.Errorf("expected resource type job, got %s", ev.ResourceType)
+	}
+	if ev.ResourceID != failed.ID {
+		t.Errorf("expected resource ID %s, got %s", failed.ID, ev.ResourceID)
+	}
+
+	// Details are stored as json.RawMessage — decode and verify the state
+	// transition + attempt counters were captured.
+	var details map[string]interface{}
+	if err := json.Unmarshal(ev.Details, &details); err != nil {
+		t.Fatalf("failed to decode audit event details: %v", err)
+	}
+	if got, want := details["old_status"], string(domain.JobStatusFailed); got != want {
+		t.Errorf("expected details.old_status=%s, got %v", want, got)
+	}
+	if got, want := details["new_status"], string(domain.JobStatusPending); got != want {
+		t.Errorf("expected details.new_status=%s, got %v", want, got)
+	}
+	// JSON numerics round-trip as float64.
+	if got, want := details["attempts"], float64(1); got != want {
+		t.Errorf("expected details.attempts=%v, got %v", want, got)
+	}
+	if got, want := details["max_attempts"], float64(3); got != want {
+		t.Errorf("expected details.max_attempts=%v, got %v", want, got)
+	}
+}
+
+func TestJobService_RetryFailedJobs_SkipsJobsAtMaxAttempts(t *testing.T) {
+	ctx := context.Background()
+
+	now := time.Now()
+	// Eligible: Attempts=0, MaxAttempts=3.
+	eligible := &domain.Job{
+		ID:            "job-retry-eligible",
+		Type:          domain.JobTypeRenewal,
+		CertificateID: "cert-001",
+		Status:        domain.JobStatusFailed,
+		Attempts:      0,
+		MaxAttempts:   3,
+		CreatedAt:     now,
+		ScheduledAt:   now,
+	}
+	// Exhausted: Attempts >= MaxAttempts must be skipped.
+	exhausted := &domain.Job{
+		ID:            "job-retry-exhausted",
+		Type:          domain.JobTypeDeployment,
+		CertificateID: "cert-002",
+		Status:        domain.JobStatusFailed,
+		Attempts:      3,
+		MaxAttempts:   3,
+		CreatedAt:     now,
+		ScheduledAt:   now,
+	}
+	jobRepo := &mockJobRepo{
+		Jobs: map[string]*domain.Job{
+			eligible.ID:  eligible,
+			exhausted.ID: exhausted,
+		},
+		StatusUpdates: make(map[string]domain.JobStatus),
+	}
+
+	svc, auditRepo := newTestJobServiceWithAudit(jobRepo)
+
+	if err := svc.RetryFailedJobs(ctx, 3); err != nil {
+		t.Fatalf("RetryFailedJobs failed: %v", err)
+	}
+
+	if got := jobRepo.StatusUpdates[eligible.ID]; got != domain.JobStatusPending {
+		t.Errorf("expected eligible job to transition to Pending, got %s", got)
+	}
+	if _, flipped := jobRepo.StatusUpdates[exhausted.ID]; flipped {
+		t.Errorf("expected exhausted job to be skipped, but status was updated")
+	}
+
+	if len(auditRepo.Events) != 1 {
+		t.Fatalf("expected 1 audit event (only for eligible job), got %d", len(auditRepo.Events))
+	}
+	if auditRepo.Events[0].ResourceID != eligible.ID {
+		t.Errorf("expected audit event for eligible job %s, got %s",
+			eligible.ID, auditRepo.Events[0].ResourceID)
+	}
+}
+
+func TestJobService_RetryFailedJobs_NoAuditServiceOK(t *testing.T) {
+	ctx := context.Background()
+
+	now := time.Now()
+	failed := &domain.Job{
+		ID:            "job-retry-no-audit",
+		Type:          domain.JobTypeRenewal,
+		CertificateID: "cert-001",
+		Status:        domain.JobStatusFailed,
+		Attempts:      0,
+		MaxAttempts:   3,
+		CreatedAt:     now,
+		ScheduledAt:   now,
+	}
+	jobRepo := &mockJobRepo{
+		Jobs:          map[string]*domain.Job{failed.ID: failed},
+		StatusUpdates: make(map[string]domain.JobStatus),
+	}
+
+	// Intentionally skip SetAuditService: the nil-guard must prevent a panic
+	// and still transition the job.
+	svc := newTestJobService(jobRepo)
+
+	if err := svc.RetryFailedJobs(ctx, 3); err != nil {
+		t.Fatalf("RetryFailedJobs failed without audit wiring: %v", err)
+	}
+	if got := jobRepo.StatusUpdates[failed.ID]; got != domain.JobStatusPending {
+		t.Errorf("expected status Pending after retry, got %s", got)
+	}
+}