mirror of
https://github.com/shankar0123/certctl.git
synced 2026-06-14 14:18:52 +00:00
Close I-001 (RetryFailedJobs never invoked) coverage-gap finding
Operator decision answered as Option A: JobService.RetryFailedJobs is
now wired into the scheduler as an always-on 10th loop. Prior to this
commit the method was implemented, unit-tested, and exported but had
zero runtime callers — any job that transitioned to status=Failed stayed
Failed forever regardless of how many attempts it had remaining.
Scheduler — 10th loop:
internal/scheduler/scheduler.go grows a jobRetryLoop alongside the
existing nine loops (renewal, jobs, health, notifications, short-lived,
network scan, digest, health check, cloud discovery). The loop follows
the established run-immediately-then-tick pattern (same shape as
jobProcessorLoop), gated by a sync/atomic.Bool idempotency guard and
joined into the scheduler's sync.WaitGroup so WaitForCompletion drains
it on graceful shutdown. Each tick runs under a 2-minute context
timeout mirroring jobProcessorLoop's opCtx budget. The runJobRetry
helper invokes jobService.RetryFailedJobs(ctx, 3) — the advisory
maxRetries cap is belt-and-suspenders; per-job eligibility is still
enforced inside the service via Attempts < MaxAttempts.
The JobServicer scheduler-interface gains RetryFailedJobs so the
scheduler's dependency surface stays explicit and mockable.
Service — audit trail per retry:
internal/service/job.go:RetryFailedJobs now emits an audit event for
every Failed→Pending transition. Following the house convention used
by all scheduler-emitted events, actor='system' and actorType=
domain.ActorTypeSystem; action='job_retry'; details capture
old_status, new_status, attempts, max_attempts. JobService carries an
optional *AuditService (SetAuditService) that nil-guards to preserve
test-wiring ergonomics — existing tests that construct JobService
without an audit service continue to pass unchanged.
Config — env var with sane default:
internal/config/config.go:SchedulerConfig grows RetryInterval, wired
to CERTCTL_SCHEDULER_RETRY_INTERVAL with a 5-minute default. Validate
rejects intervals below 1 second (matches other scheduler interval
validators).
Server wiring:
cmd/server/main.go calls jobService.SetAuditService(auditService)
after JobService construction and sched.SetJobRetryInterval(
cfg.Scheduler.RetryInterval) alongside the other SetXxxInterval calls.
Regression coverage:
internal/service/job_test.go (3 new)
- TestJobService_RetryFailedJobs_EligibleJobTransitionsAndAudits
- TestJobService_RetryFailedJobs_SkipsJobsAtMaxAttempts
- TestJobService_RetryFailedJobs_NoAuditServiceOK
internal/scheduler/scheduler_test.go (3 new)
- TestScheduler_JobRetryLoop_CallsService
- TestScheduler_JobRetryLoop_IdempotencyGuard
- TestScheduler_JobRetryLoop_WaitForCompletion
The service tests assert status transitions, attempt-cap short-
circuiting, and audit event shape (actor='system', action='job_retry',
details keys). The scheduler tests assert the loop invokes the service,
the atomic.Bool guard skips overlapping ticks with the expected
'still running, skipping tick' log, and WaitForCompletion drains the
in-flight tick on Stop.
Residual follow-up (not in scope for this commit):
internal/service/renewal.go:RetryFailedJobs is a parallel dead-code
duplicate of the same logic on RenewalService — untested and has no
runtime caller. The audit finding called this out as 'implemented
twice'. Removing it is a separate cleanup and does not block the
Option-A wiring this commit delivers.
Files:
cmd/server/main.go — SetAuditService + SetJobRetryInterval
internal/config/config.go — RetryInterval field + env + validate
internal/scheduler/scheduler.go — 10th loop, interface, field, setter
internal/scheduler/scheduler_test.go — 3 new scheduler-loop tests
internal/service/job.go — RetryFailedJobs audit emission + SetAuditService
internal/service/job_test.go — 3 new service-layer tests
This commit is contained in:
@@ -26,6 +26,7 @@ type JobService struct {
|
||||
ownerRepo repository.OwnerRepository
|
||||
renewalService *RenewalService
|
||||
deploymentService *DeploymentService
|
||||
auditService *AuditService
|
||||
logger *slog.Logger
|
||||
}
|
||||
|
||||
@@ -54,6 +55,15 @@ func NewJobService(
|
||||
}
|
||||
}
|
||||
|
||||
// SetAuditService wires an optional audit service for emitting lifecycle
|
||||
// events (e.g., scheduler-driven job_retry transitions recorded by
|
||||
// RetryFailedJobs). Construction keeps the audit dependency optional so
|
||||
// bootstrap/test wiring that doesn't exercise the retry path can omit it;
|
||||
// production wiring in cmd/server/main.go should always call this.
|
||||
func (s *JobService) SetAuditService(a *AuditService) {
|
||||
s.auditService = a
|
||||
}
|
||||
|
||||
// ProcessPendingJobs fetches and processes all pending jobs.
|
||||
// It routes jobs to the appropriate service based on job type and handles errors gracefully.
|
||||
//
|
||||
@@ -163,6 +173,16 @@ func (s *JobService) processValidationJob(ctx context.Context, job *domain.Job)
|
||||
|
||||
// RetryFailedJobs finds failed jobs and resets them for retry.
|
||||
// It only retries jobs that haven't exceeded max attempts.
|
||||
//
|
||||
// Audit trail (I-001): each successful Failed → Pending transition emits a
|
||||
// "job_retry" audit event with actor "system" (ActorTypeSystem), capturing
|
||||
// the old→new state and attempt counters so operators can reconstruct
|
||||
// scheduler-driven retry activity. The audit service is optional — callers
|
||||
// that haven't wired it via SetAuditService simply skip emission.
|
||||
//
|
||||
// maxRetries is retained for interface compatibility with
|
||||
// scheduler.JobServicer but is advisory: per-job eligibility is governed by
|
||||
// each job's own Attempts vs. MaxAttempts, not this parameter.
|
||||
func (s *JobService) RetryFailedJobs(ctx context.Context, maxRetries int) error {
|
||||
s.logger.Debug("retrying failed jobs", "max_retries", maxRetries)
|
||||
|
||||
@@ -191,6 +211,21 @@ func (s *JobService) RetryFailedJobs(ctx context.Context, maxRetries int) error
|
||||
continue
|
||||
}
|
||||
|
||||
if s.auditService != nil {
|
||||
if auditErr := s.auditService.RecordEvent(ctx, "system", domain.ActorTypeSystem,
|
||||
"job_retry", "job", job.ID,
|
||||
map[string]interface{}{
|
||||
"old_status": string(domain.JobStatusFailed),
|
||||
"new_status": string(domain.JobStatusPending),
|
||||
"attempts": job.Attempts,
|
||||
"max_attempts": job.MaxAttempts,
|
||||
}); auditErr != nil {
|
||||
s.logger.Error("failed to record job retry audit event",
|
||||
"job_id", job.ID,
|
||||
"error", auditErr)
|
||||
}
|
||||
}
|
||||
|
||||
retriedCount++
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user