mirror of
https://github.com/Studio-Saelix/sencho.git
synced 2026-09-09 17:05:51 +00:00
Anso
69e76090cc
* feat(gitops): add reconcile trigger normalization and coalescing keys The source controller needs a single normalized shape for every trigger that can request evaluation (manual, API, webhook, poll, retry, config change, startup, resume, and future provider/schedule/binding-change producers), and a way to decide whether two concurrent submissions describe the same work. Add ReconcileTrigger, the discriminated ReconcileRequest (fetch vs apply), and coalesceKey/deliveryKey. A fetch coalesces by application alone, since it has one live outcome regardless of trigger; an apply coalesces only when the commit, plan fingerprint, and deploy choice all match, so two applies that differ in any of those can never be joined and have one silently receive the other's result. * feat(gitops): add exhaustive failure classification and retry backoff The controller needs to tell a transient network condition from a permanent configuration one from a broad range of Git-source, policy, and target failures, since several distinct causes collapse onto the same public error code and a wrong call either retries forever on a bad URL or gives up on a DNS blip. Add classifyFailure, built on two total lookup records (one keyed by TransportFailureReason, one by GitSourceErrorCode) so adding a new value to either source union fails the build until this classifier accounts for it, rather than silently defaulting. A tip-changed race classifies as supersession, not backoff; an unrecognized exit-coded git error gets a lower retry ceiling than a plain network timeout; a deploy or health failure after a successful apply is its own class that must never refetch or reapply. Add nextRetryAt: bounded exponential backoff (60s doubling, capped at one hour) with +-10% jitter, honoring a provider-supplied retry floor when it is larger than the computed delay. * feat(gitops): derive normalized reconcile outcomes from the source facet Silence is not an acceptable GitOps result: every reconcile attempt needs to settle into one named outcome an operator can act on, not a bare success/failure. Deriving that outcome from the existing source-facet projection (rather than a second, parallel status source) is what keeps "no source change" from silently collapsing into "converged" the way a bare commit-SHA comparison would. Add outcomeFromSourceFacet, exhaustive over all 17 SourceFacet statuses, each mapped to a ReconcileOutcome, a human reason, and a next action (review, resume, retry, resolve_conflict, configure_credentials, view_target_results, or none). converged is deliberately never produced here: it requires target and health evidence this source-only projection does not carry, so a later composition over source + target + health is the only thing allowed to report it. * feat(gitops): add the portable accepted-generation contract Direct and Blueprint dispatch need to consume the exact same description of what an accepted generation contains, without either side inventing a node id, local candidate path, target project name, or placement field into it, since that is exactly the kind of drift that would let a stale acceptance authorize a routing decision made after it. Add six additive, nullable columns to gitops_generations (portable manifest, Compose inputs, source/security policy evidence, support and compatibility requirements), all decoded honestly: a legacy row missing one records an explicit limitation rather than inventing evidence, and a legacy pending candidate lacking the new contract must be re-evaluated before it can be accepted or dispatched. Add gitops/handoff.ts: the AcceptedGeneration type built from a generation row via buildAcceptedGeneration, a compile-time assertion that the contract cannot carry a target-mode-specific field, and the TargetAdapter boundary with BlueprintTargetAdapter failing closed until Blueprint rollout orchestration exists. Current target mode and binding travel separately in DispatchContext, re-read under the dispatch lock rather than carried on the generation itself. * feat(gitops): add controller-owned bookkeeping columns to gitops_applications The source controller needs somewhere to persist source policy, poll cadence, and a durable attempt sequence per application, and it needs to work identically for Direct and Blueprint applications. stack_git_sources (the existing home for auto_apply_on_webhook/auto_deploy_on_apply) is Direct-only and keyed by stack name, so it cannot represent a Blueprint application at all. Add source_policy (manual|review|automatic, default manual), poll_interval_secs (NULL inherits the global default, 0 disables), next_poll_at (the durable scheduling cursor), and attempt_seq to gitops_applications instead. All default to values that start no unattended work: an upgraded installation begins polling nothing and stays on manual policy until explicitly migrated or configured. * feat(gitops): add durable attempt reservation and poll/retry queries History dedupe alone is not execution idempotency: mutateApp writes the application row and only then inserts history, so a dedupe conflict still commits the row write. The controller needs a reservation that runs before any side effect and touches nothing else, so a duplicate or restarted submission can be told apart from new work without repeating it. Add reserveReconcileAttempt/settleReconcileAttempt: a bare history insert in its own transaction, deliberately not through mutateApp, using the existing history dedupe index as the reservation/idempotency check itself. A repeated reservation or settlement for the same operation is a no-op, never overwriting the first settled result. Add the store queries a controller needs to drive this: getSettledAttempt and latestSettledAttempt (exact vs. most-recent lookup, the latter tie-broken by rowid since the id column is a random UUID unrelated to recency), listUnsettledReconcileAttempts (a reservation with no matching settled row, for startup recovery), and listSourcesDueForPoll / listApplicationsDueForRetry (excluding suspended, in-flight, and Blueprint-mode applications). * feat(gitops): add controller-facing reconcile entry point to GitSourceService Adds GitSourceService.reconcile(), a single normalized entry point that takes a fetch or apply request and returns a normalized outcome, next action, and reason instead of a thrown error or a raw boolean. It owns the git mutex for the whole evaluation and calls the same private fetch/apply bodies the existing pull()/apply() routes use, so it never nests locks. Resolves the live application for the stack first and fails closed, without attempting any work, when the request's application id no longer matches it or when no application exists at all. A fetch or apply failure that the underlying transition never persisted (missing config, a stale commit, lock contention) is classified through the existing retry disposition table so an unretryable failure is never reported as retryable, and a stale-looking success is never reported in place of a real failure. * feat(gitops): dispatch accepted generations to their target Adds GitSourceService.dispatchAcceptedGeneration(), which routes a portable accepted-generation contract to its target: Blueprint mode delegates to the existing BlueprintTargetAdapter (rollout orchestration is not built yet, so it always blocks), and Direct mode dispatches by driving reconcile() with the generation's commit, since there is no separate generation-based promotion pipeline yet. The deploy flag is read from the stack's own auto_deploy_on_apply setting, matching how every other producer decides it. Also fixes a case reconcile() got wrong: when a promotion succeeds but the following deploy fails, the source itself has genuinely changed, so reporting it as an unchanged failure was as untruthful as reporting it a plain success would have been. It now reports a result that neither claims, pointing at the target instead of asking for a source retry that could never succeed. * feat(gitops): add background poll and retry driver for source reconciliation Adds SourceController, a self-rescheduling background timer that finds sources whose poll interval or retry time has arrived and evaluates each through the existing reconcile() entry point. A per-application in-flight set is what keeps one slow evaluation from blocking the rest of the fleet; the tick never waits on any evaluation before scheduling the next one, and a scan that throws still reschedules rather than stopping the driver permanently. This delivery only covers detection: a tick issues a fetch, the same step a manual pull performs, so a new candidate is staged for review but not automatically accepted or dispatched, and a retry re-issues a plain fetch rather than resuming whatever stage previously failed. Both are documented as open follow-on work rather than silent gaps. Not yet wired into the process lifecycle; that follows separately. * feat(gitops): start and stop the source reconciliation driver with the process Wires SourceController into the background-service lifecycle: it starts after the existing GitOps recovery and orphan-sweep steps, so it never scans an application still carrying a stale in-progress marker from a killed process, and stops alongside every other background timer on shutdown. * feat(gitops): add suspend, resume, and explicit retry for a source Adds GitSourceService.suspend()/resume()/retry(), the service-layer methods behind an upcoming suspend/resume/retry control surface. Suspending a source now genuinely stops it: pullLocked and applyLockedBody both check suspension before doing any work, not just inside the transition bookkeeping, which used to reject but then let the fetch or apply proceed anyway. A refused suspend surfaces as a real error rather than a silent no-op, since an operator believing a source is suspended when it isn't is the exact failure this exists to prevent; a refused resume stays silent, since the row read back after the attempt already reports the true state either way. A webhook delivery to a suspended source is reported as skipped rather than a failed pull, so a long suspension does not read to the Git host as a broken webhook. * feat(gitops): add suspend, resume, and retry routes for a git source Adds POST endpoints for suspending a source (with an optional, length- capped reason), resuming it, and explicitly retrying it, wired to the existing service-layer methods behind the same stack:edit permission as pull and apply. Neither route can trigger a deploy today, so unlike apply and webhook-pull they need no conditional stack:deploy check. * fix(gitops): close path-injection gaps at two candidate-file sinks Adds the inline resolve-and-prefix-check barrier this codebase already uses at other filesystem sinks derived from a stack name or a stored candidate path, closing two sinks that lacked it: the manifest write in GitProjectManifestService, and the pending-candidate access check before promoting an apply. The apply-side check now shares the same strict candidate-path validator the registry-delivery path already uses on the identical stored field, rather than a looser check, so both call sites treat a tampered candidate reference the same way. Fixed two test fixtures that had never matched the shape a real candidate path takes, which the stricter check would otherwise have rejected. * fix(gitops): confine the manifest write directory to the managed area The prior fix checked the manifest write/rename targets against the resolved managed directory, but left the mkdir call on that directory itself unconfined and checked it against the data root rather than the narrower managed area every sibling barrier in this file uses. Aligns it with the established convention and the actual boundary that matters. * feat(gitops): wire durable attempt reservation and coalescing into reconcile() reconcile() now reserves a durable attempt before doing any fetch or apply and settles it with the normalized result once execution finishes, using the reservation and settlement primitives that already existed but had no caller. A request carrying a stable external delivery id reserves under a producer- and intent-namespaced key, so a redelivery reuses the same attempt instead of minting a second one; a request with no such identity gets a freshly allocated attemptSeq- based id, bumped atomically with its reservation. Concurrent submissions that would do the same work now coalesce: the first becomes the leader and actually runs, and any submission that joins while it is still in flight awaits the leader's real result instead of running a duplicate fetch or apply. Each still gets its own durable attempt and its own settled row, including a concurrent redelivery of the same external event, which joins the in-flight leader rather than falling back to a snapshot of the row from before the leader's work landed. Startup gains an attempt-recovery phase, run before the managed-area sweep and before SourceController starts: every attempt reserved but never settled, most likely from a crash between the two, is resolved from durable state without re-executing anything. One row failing to recover no longer blocks the rest; it is skipped and logged, and recovery keeps paging until nothing unsettled remains. A settlement failure is caught and logged rather than turning an already-successful fetch or apply into a thrown error for the caller, and a settled attempt's stored result is now decoded through a validated outcome/next-action check instead of a blind cast, logging rather than silently reporting unknown when a stored row is corrupt or unreadable. * fix(gitops): remove unused store variable from coalescing test * fix(gitops): make reconcile-attempt recovery leader-aware and cursor-paginated A fresh audit found real gaps in the reservation/coalescing wiring from the previous commit: recovery paged by "still unsettled" status rather than a cursor, so once a permanently unrecoverable row occupied every slot in a page, every genuinely recoverable row beyond it was silently never reached; a follower's outcome was reconstructed independently from row state rather than from its leader's actual stored result, so a leader and its follower could durably disagree; and an attempt resolved on the live path was returned but never actually settled, leaving it open indefinitely until the next restart. listUnsettledReconcileAttempts now takes an optional (created_at, id) cursor, matching the pagination shape queryHistoryRows already uses, so paging always advances regardless of which rows settle. Recovery is now two-pass: independent attempts settle first from row state, followers are deferred, then each deferred follower settles from its leader's now-settled result. A follower whose leader is a real but still-unresolved reservation is left unsettled rather than guessed at independently, since the leader could still settle to something else later, including when the leader simply failed to settle in this same pass rather than "never will". The same leader-aware resolution now backs the live reconcile() path too, via a shared helper, so an already-reserved attempt is durably settled instead of merely returning a value. Also fixes a narrower race in reconcile() itself: two submissions can share an operation id (a stable external delivery id) while running under different coalesce keys, since an apply's coalesce key includes its commit sha, plan fingerprint, and deploy flag, which the delivery id does not carry. reconcile() now checks in-process executions by operation id directly before falling back to a durable-state resolution, so such a submission joins the real in-flight leader instead of settling a stale pre-execution snapshot ahead of it. * feat(gitops): route pull, apply, and webhook producers through reservation A fresh audit found the same "primitive built, real caller does not use it" pattern one layer deeper than the previous commit fixed: reconcile(), the controller-facing entry point, reserved and coalesced durable attempts correctly, but the actual production producers, the manual pull button, the manual apply button, and the webhook route, all called pullLocked/applyWithSharedLock directly, bypassing reservation entirely. pullLocked also minted its own independent operation id rather than using a reserved attempt's, breaking the "one operation id spans an attempt and its stage evidence" invariant. pullLocked, applyWithSharedLock, applyLocked, and applyLockedBody now accept an optional operation id, using it in place of their own default when a caller supplies one. pull() and apply() reserve a durable attempt and coalesce with a concurrent call to themselves whenever a real GitOps application exists for the stack, the same definition pullLocked itself already used to decide whether it has any GitOps bookkeeping to do at all. handleWebhookPull()'s fetch step and its conditional auto-apply step reserve too, without a coalescing map: the route's own debounce window plus its single per-stack lock acquisition already prevent a concurrent duplicate from reaching that point, so there is nothing to coalesce there. Fixed along the way: apply()'s coalesce key computed its deploy flag differently than the code that actually executed the apply, so two concurrent applies that genuinely differed in deploy behavior could share a key and one could silently receive the other's result; deploy is now resolved once, the same way applyLockedBody itself resolves it, before it drives either the key or the execution. A policy-bypassing apply is now routed through a fresh, never-shared coalescing map, since bypassPolicy changes behavior but was not part of the key. A coalesced follower's own reservation is now settled in a finally rather than only after a successful await, so a rejecting leader (the ordinary path for a producer that preserves its own throw contract, unlike reconcile()'s internal error handling) no longer leaves the follower's attempt open until the next restart. A reservation bookkeeping failure no longer turns a manual pull or apply that would otherwise have succeeded into a hard failure; it logs and falls through to unreserved execution instead. * fix(gitops): make the boot sweep consult claimant pointers before reaping a candidate The boot-time managed-area sweep decided whether to delete a staged candidate directory purely from file age and a completeness marker, with no awareness of the database. A fresh audit gave the concrete failure: a reconcile stages a candidate, the process crashes before settlement, the installation stays down more than a day, startup settles the attempt from a snapshot rather than real stage evidence, and the sweep then deletes the still-needed candidate out from under it. sweepManagedArea now takes the set of candidate directory basenames still referenced by the stack, and never reaps one of them regardless of age or completeness. GitSourceService computes that set from three independent sources: the live application's current candidate generation, its accepted-but-not-yet-promoted generation (the sourceAccepted-committed, targetApplied-not-yet-committed window; that path has no production caller yet, so this is forward-looking coverage for it), and the pending fetch record's own candidate reference, which is written outside the transaction that mints a generation and can therefore be the only claimant for a candidate that failed validation or was staged while no live application existed to read a pointer from. * fix(gitops): fail closed on reservation failure and a torn-down application A fresh audit found that a reservation-bookkeeping failure (a transient DB error, an application torn down in the window between resolving it and reserving against it) fell through to unreserved execution. That directly defeated this delivery's own purpose: a manual apply could still promote Compose files and deploy with zero durable record of it happening. Reservation failure now fails closed for pull, apply, and the webhook route: the operation is refused, logged, and recorded to the stack's own activity history, distinguishing a torn-down application (never retryable) from a transient failure (worth retrying). Closing that hole surfaced a second, related gap: pull, apply, and the webhook route only checked for a live (active) application before deciding whether to reserve at all, so a stack whose GitOps tracking was explicitly torn down while its Git source configuration survived could still run fully untracked, the same failure mode reached a different way. Refusing this case took two attempts to get right, both caught by review before landing: the first version refused on any tombstoned state, which would have permanently and unrecoverably blocked pull and apply for an application deliberately tombstoned as deleted while its config survives for a future rebuild, a state two existing production paths produce on purpose; narrowing to detached only still misfired on a routine, fully completed detach, since that same operation deletes the source row in the same transaction, so the refusal must also confirm the source row actually survived before firing. Both are covered by regression tests now, alongside the original reservation-failure fix. * fix(gitops): unify fetch-intent coalescing across pull and reconcile A manual pull and a concurrently poll-triggered reconcile for the same application previously ran their own separate in-flight maps and could each start a clone for the same fetch. They now share one coalescing map, with reconcile's fetch path classifying its own outcome instead of relying on generic row-state derivation, so a pre-transition failure the row does not yet reflect is never durably recorded as a plain success. Apply-intent coalescing stays producer-local for now; unifying it needs deploy-failure awareness threaded into the shared settlement path first, which is a separately scoped follow-up. * feat(gitops): recognize a webhook delivery id for traceability The real webhook trigger endpoint is generic and HMAC-signed, with no delivery identity of its own. It now extracts one from a recognized provider header (GitHub, GitLab, Bitbucket, or a generic fallback) when the caller sends one, and threads it through to the git-pull execution path as a plain traceability breadcrumb on failure logs. Redelivery dedup is deliberately not implemented here: an earlier attempt routed the delivery id through the durable attempt reservation itself, which silently dropped a redelivery's history through the existing dedupe index instead of recording it. Building real dedup needs settlement to reflect classified outcomes rather than generic row-state derivation for both a settled and a crashed-mid-flight prior attempt, which is a wider, separately scoped change shared with the same gap already deferred for apply-intent reconcile. * fix(gitops): thread the reserved attempt's operation id into the pending fetch record The pending fetch record stamped its own independent random operation id instead of the reserved attempt's real one, so an apply falling back to it (when it holds no reservation of its own) inherited an identity unrelated to the fetch that actually produced the candidate. One id now spans reservation, fetch, generation, and the pending record. Also fixes the short "op" token rendered in activity log lines: a fixed prefix stopped discriminating between attempts once operation ids became structured (<applicationId>:attempt:<seq>), since the prefix is now the same applicationId every time. A shared helper renders the actual attempt-discriminating suffix instead, applied consistently across pull, apply, and create so each event's logged identity matches what its own durable history actually recorded. * test(gitops): cover the reconcile-recovery-then-sweep startup ordering Reconcile-attempt recovery and the managed-area sweep must run in that fixed order before the source controller's own poll loop starts, but the guarantee lived only in a comment inside startServer, a function with roughly two dozen unrelated service initializations that makes it impractical to exercise end to end in a test. Extracted the two steps into their own function so they're directly testable in isolation, without moving the source controller's own start call: that stays exactly where it was, since pulling it earlier would have crossed a separate, already-documented ordering requirement for registry delivery recovery. A structural test guards the one property that can't be covered by driving the function directly: that the real startServer body still calls the extracted function before starting the controller. * fix(gitops): complete durable reconciliation execution * fix(gitops): resolve static analysis findings