Files
sencho/backend
Anso 57cd8422fb feat(gitops): poll scheduling, source policies, and retry evidence (#1910)
* feat(gitops): migrate legacy auto-apply boolean to source policy

* feat(git-sources): source policy compatibility for legacy boolean clients

* feat(gitops): poll scheduling transition and fetch failure evidence

* feat(gitops): controller retry scheduling and automatic acceptance

* feat(git-sources): node-scoped polling settings endpoint

* perf(gitops): partial indexes for poll and retry due queries

* test(gitops): drop unused reconcile handle in polling suite

* fix(gitops): poll eligibility defers to a future retry cursor

A source inside a fetch backoff window kept its poll cursor visible to
the due scan, so a cadence change or an unlucky tick could refetch the
remote before the backoff expired. The poll scan now excludes rows with
a retry cursor, the matching partial index mirrors the term, and
rescheduleAll leaves backoff rows alone so the retry cursor stays the
next wake. Retry-due rows still arrive through the retry scan and are
deduplicated per application.

* fix(gitops): persist source policy when upserting an existing source

PUT accepted source_policy and the legacy boolean but persisted them only
through the create path, so an operator editing an already-linked source
could not leave or return to the automatic policy; the field resolved and
then vanished. The existing-app branch of the upsert transaction now
records a source_policy_changed transition when the resolved policy
differs from the stored one. A boolean 0 or omitted still resolves to the
existing policy, so the no-silent-conversion rule is unchanged.

* refactor(gitops): allow policy edits on a suspended source

Suspension gates unattended fetching and applying, not configuration:
the operator must be able to re-policy a suspended source before
resuming it. The in-flight guard stays, since the settle path reads the
policy when deciding acceptance.

* docs(gitops): correct comment claims about retry-deferral and policy persistence

The rescheduleAll skip comment attributed mid-backoff refetch prevention
to the wrong guard (the poll-scan SQL exclusion does that); it now states
what the skip itself delivers: no inert poll cursor, no misreported next
wake in the polling projection, no spurious audit line. The retry-query
test comment describes the defensive property it actually pins instead
of an unreachable origin story, the listSourcesDueForPoll doc names the
retry-cursor exclusion, the sourcePolicyChanged doc drops the liveness
claim the method does not enforce (callers resolve the live row), the
duplicated create-path narrative is kept in the method doc alone, and a
temporal marker is removed from a test comment.

* test(gitops): pin suspended-source policy edits, audit row, and index rebuild

Three coverage gaps from the test-analysis pass: a policy PUT on a
suspended source must succeed (suspension gates work, not configuration,
and a future hardening refactor that re-adds the guard would 500 operator
edits); the source_policy_changed history row is asserted so a dropped
audit insert fails loudly; and a PATCH with a backoff row exercises the
rescheduleAll skip through the HTTP route. The poll due index is dropped
before creation because CREATE INDEX IF NOT EXISTS never updates an
existing index, so upgraded installs would keep the pre-retry_at WHERE
clause; the schema SQL now forces the rebuild.

* test(gitops): unit-test sourcePolicyChanged persistence and in-flight guard

The route suite covers the upsert path end to end, but the transition
itself had no direct tests: one pins that the policy row and history
land, the other pins that a fetch in flight refuses the edit, which is
the conflict the settle path depends on.

* fix(gitops): map an in-flight refusal during source save to 409

A PUT that changes the source policy while a fetch is in flight threw
GitOpsTransitionError out of the upsert transaction. The rollback was
correct, but the route mapped the throw to a generic 500, hiding the
actionable reason. Suspend and dismiss already translate this error to
an OPERATION_IN_FLIGHT GitSourceError; the save path now does the same,
and a route test pins the 409 and the full rollback.

* fix(gitops): consume the poll cursor when a source moves to manual

The policy-flip path made manual-with-armed-cursor reachable for the
first time: the poll scan picked the row every tick, the controller's
manual guard declined it without consuming the cursor, and the
projection kept reporting a scheduled poll that could never run. Moving
to manual now clears next_poll_at, matching how fetchStarted consumes
the cursor; non-manual policies keep it.

* fix(gitops): skip reschedule for any unconsumed retry cursor

The guard only skipped a future retry_at, so a fired-but-unconsumed
cursor still got a poll cursor armed that the poll scan would never
honor: the exact inert-cursor audit line and misreported next wake the
skip exists to prevent. Aligning the guard with the poll scan's
any-cursor exclusion closes the mismatch; the retry scan still owns the
row either way.

* test(gitops): drive the backoff deferral test through the real due queries

The test replaced both due queries with JS filters that mirrored the
intended SQL predicate, so a regression in the query itself would pass
it; the real SQL was only covered by the separate eligibility suite.
Eligibility is the behavior under test here, so the real store queries
now run against the seeded row and the deferral claim holds end to end.

* fix(gitops): consume the retry cursor too when a source moves to manual

The poll-cursor clear missed the retry cursor: an automatic source in a
backoff window that was flipped to manual would stay in the retry-due
set forever, since the retry scan has no policy filter and the
controller's manual guard declines without consuming. Moving to manual
now withdraws both schedules while leaving failure evidence untouched,
and the rescheduleAll docstring matches the guard the previous commit
widened.

* style(gitops): re-indent the upsert transaction body under its new try wrapper

Whitespace only: the save transaction gained a try/catch in dcc645c2 but
its body kept the old indentation, leaving the block misaligned.

* test(gitops): pin the source-row rollback in the 409 race test

The 409 test asserted only the application row; the source row is
written by the same transaction, so its survival is what proves the
save rolled back whole.

* fix(gitops): refuse acceptance of a blocked candidate and a raced policy

Automatic acceptance now runs only on a success-shaped staged-candidate
outcome, so a conflict-blocked candidate never reaches acceptance from
the controller tick. The acceptance boundary backs this with two durable
refusals that re-read the row inside the mutation transaction: a
candidate_plan_blocked row is refused whatever the caller reports, and a
configured_policy acceptance is refused once the source policy has left
automatic, so a policy change completing while an evaluation is in
flight is honored rather than bypassed. Operator authority is
unconstrained.

* feat(gitops): arm initial poll cursors when a source becomes eligible

An eligible source joins the unattended cadence the moment it becomes
eligible, not after the next global settings PATCH happens to re-scan
it. The interval rule moves to backoff.ts as effectivePollIntervalSecs
so the controller re-arm, the save path, and create recovery all
compute the same wake.

Cursors are armed on create-from-git, on first linking an existing
stack, on a flip from manual into review or automatic, and on resume
when a source comes back with no cursor (a manual source suspended,
re-policed, and resumed would otherwise stay out of the cadence
forever). A crash-recovered create arms the cursor the same way the
live create path does. The arming pre-check mirrors every refusal
sourcePollScheduled itself makes plus the due-scan eligibility rules,
so a suspended row, a row already holding a wake, and a row under a
live retry cursor are all left alone: a poll cursor armed underneath a
retry cursor would sit inert until the retry discards it.

* feat(gitops): round-trip source_policy through projection and create

The tri-state policy is now part of the public source projection, so a
client can read back what the source actually runs under: reading it
and PUTting it again is a no-op, not a silent conversion. A source with
no application row still projects its policy from the legacy boolean
(true is automatic, false is review, the historical webhook behavior).

Create-from-git accepts and validates an explicit source_policy the
same way the upsert path does, so a stack can be created directly into
manual or review without a follow-up edit to walk the policy back.

* fix(gitops): scope the retry due-scan to active Direct applications

The retry scan selected any row with a due retry cursor regardless of
target mode or lifecycle. A detached Direct row, or a Blueprint-mode row
with no stack name, would have run the Direct fetch path against a
target it cannot address. The scan now carries the same Direct/Active
terms as the poll scan, the retry partial index mirrors them, and the
index is dropped and recreated on schema apply so upgraded installs
rebuild it.

* feat(gitops): record security-policy evidence on automatic acceptance

Configured-policy acceptance evaluated the candidate against the
security policy but stored no durable record of what allowed it: the
verdict lived only in the audit trail and the evaluation's lifetime, so
a restart left an accepted generation with no persisted proof of the
policy that authorized it.

The controller now builds redacted evidence (deciding policy id, name,
and decision inputs, evaluation time, image refs) and the acceptance
transition writes it onto the generation row inside the same
transaction that commits the acceptance. Held candidates record
nothing: a generation must never imply a verdict it does not have.

* docs(gitops): correct inaccurate retry-scan and evidence comments

Detachment does not clear cursors: applicationTombstoned leaves retry_at
alone, so a detached row can still carry a due retry cursor and the scan
must exclude it on its own terms. A Blueprint-mode wake reaches evaluate()
with no stack name, which returns early, so selecting such a row would
only re-wake it every tick with a misleading warning; the old comment
overstated the harm. The listApplicationsDueForRetry docstring gains the
active-Direct scoping the SQL already enforces, and the policyRow()
docstring says shaped instead of projected, since the helper hand-builds
a row rather than transforming one.

* fix(gitops): restore generation immutability across acceptance

Automatic acceptance updated the generation row in place to record the
security-policy verdict, which violated the authoritative contract that
generations are never updated after insert: a generation is immutable
candidate-time evidence, and rewriting it makes the same row observable
first without and later with evidence. Acceptance now moves the
application pointer only; the policy verdict remains in the audit
history, and the accepted-generation contract keeps reporting missing
security-policy evidence as an explicit limitation until the dispatch
delivery builds the append-only revalidation flow that records current
evidence on a new generation.

A regression test pins the contract: acceptance must leave the
generation row byte-for-byte unchanged.
2026-09-11 16:10:44 -04:00
..