mirror of
https://github.com/Studio-Saelix/sencho.git
synced 2026-08-30 03:59:41 +00:00
feat(git): add GitOps revision store and Direct apply transitions (#1835)
* feat(git): add GitOps revision store and Direct apply transitions
Adds the canonical GitOps revision state model: schema, store, history,
approval resolution, Direct apply transitions, and a first-cut read
projection. Also widens recovery capture with generation, artifact, and
source-acceptance bindings, rejects credentials and query strings in new
Git repository URLs, and keeps Blueprint and node-label routes on the hub.
* feat(git): add create-from-Git activation and teardown transitions
Adds the durable half of create-from-Git: the single transaction that persists
the application, resolved commit, generation, checkpoint, and candidate
together, and the teardown that tombstones a create which never reached its
success boundary. Adds the staging marker and operation-owned cleanup rules
that decide what a crashed create is allowed to delete.
* feat(git): make create-from-Git crash-safe
Wires the create path through the GitOps state model: the staging marker is
written before the candidate is built, the activation transaction persists the
application, generation, and checkpoint together, and the source row plus
accepted pointers now commit as one success boundary. A create that fails
before that boundary removes only what it staged, then tombstones itself.
The managed-area sweep no longer deletes a directory it cannot prove it owns,
counts in-flight creates as claimed, and is awaited at startup so an
interrupted create is settled before any mutation service runs.
* feat(git): settle interrupted create-from-Git at startup
Adds boot-time recovery for creates a previous process left in flight. A create
whose project was already committed to disk is finished, including its source
row and accepted pointers. Anything earlier has its files removed first and is
then tombstoned, so the stack name becomes usable again instead of staying
locked by a half-created application. A source row that outlived its
application is always preserved.
* fix(git): close create-from-Git ownership and reporting gaps
Removes two ways a failed create could destroy files it did not own: boot
recovery no longer deletes a stack directory the create never recorded making,
and the managed-root teardown is gated on having created that root. Cleanup
now rejects a path that is not a generation directory, so a malformed row
cannot widen the blast radius.
A create that fails during materialization now clears its staging marker, which
previously stayed behind and made the stack name uncreatable until restart.
Generation paths no longer resolve through a module cycle that left them
pointing at a directory that never existed.
Boot recovery restores the deploy spec and manifest cache a finished create
needs, settles each application independently so one bad row cannot strand the
others, and distinguishes a missing directory from one it could not read. A
create that succeeds now clears its checkpoint, and a failure after the success
boundary says the stack was created rather than reporting a bare error.
* fix(git): reap managed areas that nothing has ever claimed
The boot sweep now distinguishes a missing staging marker from an unreadable
one. A missing marker means no stack, no create, and no marker claims the
directory, which is the ordinary orphan a crashed stack deletion leaves behind,
so it is removed as it always was. An unreadable marker is evidence of a claim
that cannot be verified, so the area is preserved.
Without this an orphaned managed area survived every boot forever, since a
completed create deletes its marker and nothing else would ever claim it.
* feat(git): complete the Direct source and target transitions
Adds the remaining Direct events the state model owed: an invalid fetch that
advances the resolved commit without minting a candidate, a blocked candidate
that is visible but cannot apply, dismissal that leaves the running workload
alone, material configuration change that invalidates a staged candidate while
keeping accepted and applied pointers, the two deploy failure classes, and
application and target tombstones.
Widens the application update to every mutable column. It previously wrote
about half of them, so a transition assigning a pause, suspend, recovery, or
intent field would have type-checked, appeared in the history snapshot, and
been silently dropped at commit.
* feat(git): record Direct Git operations in the revision state
Wires fetch, apply, and detach through the state model. A pull records the
fetch and, when it produces a candidate, the generation behind it, marking a
blocked change plan as a blocker that cannot apply. An apply records the
acceptance that binds the generation to the workload, and closes its operation
on failure so a throw cannot leave the source reporting work in progress.
Detach tombstones the application and its target in the same transaction that
removes the source row.
Every producer is a no-op for a stack with no live application, so installs
whose Git stacks predate this model are untouched until migration runs.
* test(git): cover the Direct Git producers end to end
Adds a harness that stubs only the git transport and rollback capture, so
fetch, candidate materialization, change-plan classification, apply, and detach
all run for real and are asserted against the resulting revision state.
Covers the seam the transition-level tests could not: that a fetch advances the
resolved commit without moving the accepted generation, that the apply binds
the exact candidate the fetch recorded and its acceptance proves that
generation and no other, that detach tombstones while keeping configured
identity as a frozen fact, that a failed fetch closes its operation and leaves
every pointer alone, and that a stack with no application produces no writes.
* feat(git): bind the deployed generation from the Compose adapter
Makes ComposeService the sole producer of deploy events. Every deploy path
funnels through deployStack, so recording it there keeps one start and one
terminal row per mutation instead of each caller reporting its own.
A successful deploy binds the applied generation, a failure records the class
and leaves the deployed pointer alone. The class is conservative: once the
compose command is handed off we cannot prove the workload was untouched, and
claiming it was intact is the more dangerous error. A stack with no live
application, or one with nothing applied, has no deploy identity to record and
is skipped entirely.
* feat(git): promote healthy and last-known-good from health verdicts
A stack health run now records which generation it observed, and its verdict
drives the revision state. Promotion is narrow by design: healthy and
last-known-good move only when the run passed, watched the whole stack, and the
generation it watched is still deployed. A stale or service-scoped verdict
records history and moves nothing.
Last-known-good keeps the artifact expectation only when that expectation
belongs to the promoted generation. Otherwise the generation is still good and
its executable identity simply is not proven, so the pointer is left null
rather than borrowed from another generation.
beginStack now takes the deployed generation explicitly rather than reading
current state, because a verdict is only meaningful for the generation the run
actually observed. Update-path callers pass null until updateStack reports what
it deployed.
* feat(git): bind the deployed generation on the update path
Gives updateStack the same treatment deployStack has: it opens a deploy
operation at the point Compose is handed the mutation, closes it on both
outcomes, and reports the generation it bound. The orchestrator carries that
binding through, and all five update call sites now pass it to the health gate
instead of null, so a passed health run after an update promotes healthy and
last-known-good exactly as it does after a deploy.
The operation is opened at the compose call rather than at entry, so an update
that fails during capture or classification records nothing: it never touched
the workload, and reporting a deploy failure for it would make the projection
claim something that did not happen.
* feat(git): track linked stacks and material config changes
Linking an existing stack to a Git source now brings it into the revision
state: the application starts live with nothing desired or accepted, so the
projection asks for a fetch rather than claiming a state it has not observed.
Editing a source's material configuration invalidates its staged candidate in
the same transaction that writes the row and clears the pending pull. A
candidate built from a different repository, ref, or file set can no longer be
applied, and clearing pending without invalidating it would leave the model
offering an apply the operator cannot produce. Credential-only and policy-only
edits change nothing material and leave the candidate alone.
* feat(git): retire the application when a stack is deleted
Stack deletion now tombstones its GitOps application and targets in the same
transaction that commits the deletion. A deleted stack that kept a live
application would go on claiming the stack name and block re-creating it
through the unique live-application index.
The tombstone is driven from the deletion service rather than from inside
DatabaseService, so the store keeps its transitions and their history in one
place and the two modules do not form an import cycle.
* feat(git): retire node targets when a node is removed
Deleting a node now tombstones the GitOps targets that lived on it, in the same
transaction as the delete and while those rows still exist. Otherwise a
part-way failure would leave targets pointing at a node that is gone.
Applications stay live: a Direct application still describes a real stack, and
a Blueprint one may have targets on other nodes. Both the local and remote node
paths go through the same helper, so neither can skip the retirement.
* feat(git): add proof-bound recovery transitions
A restore moves a target back to an older generation, the one case where a
target and its application legitimately disagree about what is current. These
transitions decide what may move with it.
Pointers move only when the restore is provable: the recovery point named a
generation, that generation still exists, and it belongs to this application.
The artifact expectation comes from what the recovery point captured, never
from what the application expects now, and the acceptance is kept only if it
still proves the restored generation rather than borrowing one that authorized
a newer one. A last-known-good survives unless the generation behind it is gone
or turns out to belong elsewhere, and then the reason is recorded so the
projection can say unavailable rather than none.
An unproven restore is still recorded as a real operational recovery and moves
nothing, because there is no evidence to move pointers to.
* fix(git): close GitOps operations that cannot be recorded
Recording a transition still never fails the operation it describes, but a
rejected terminal event no longer leaves that operation open. A start that
never terminates reported work in flight for ever and offered no actions, and
because a new fetch refuses to open a second operation, one rejection silently
stopped the model tracking that stack until a restart.
Startup now reclassifies operations the previous process left open, which the
transition already supported but nothing called. An interrupted restore is
closed there too, since only the terminal recovery events clear it.
The pull records its fetch, generation, and candidate as one transaction, and
the apply verifies the candidate was built from the commit being applied.
Separately those two allowed an apply to accept a generation whose files were
never on disk while the projection reported the older commit as current.
Also: the candidate now records its own invocation rather than the one it
replaces, changing material configuration is refused while an operation is in
flight, a tombstoned target cannot be repopulated by a late health verdict, a
deploy handle is not returned when its start was not recorded, the mutation
handoff is marked only once Compose is genuinely about to receive it, and a
rejected tombstone fails one deletion rather than aborting the boot sweep.
* feat(git): record why unprovable evidence was dropped
The projection carried a limitations array that only read-time derivation ever
wrote to. When a transition cleared a pointer it could not prove, the result
was indistinguishable from that pointer never having existed, and in two cases
it made the target read healthier: a dropped artifact expectation silently
disables the runtime drift check, and an unproven restore left every pointer
agreeing with itself so the target reported as synced and healthy.
Adds a persisted, fail-closed evidence record on the application and target
rows. The transitions that drop an artifact expectation, a last-known-good
artifact, or a source acceptance now say why, an unproven restore is marked as
such, and the deriver folds all of it into the limitations the projection
already exposes. Clearing a code when the evidence becomes provable again is
part of the contract, so a stale doubt cannot outlive its cause.
This lands before the migration matrix because migration is the largest
producer of evidence that cannot be proven, and the plan requires those to
surface as bounded limitation evidence rather than as fabricated pointers.
* feat(git): migrate pre-existing Git stacks into the revision state
Git stacks created before this model had no application at all, so every
producer was a no-op for them and the projection could not describe them. They
are now brought in at boot.
The governing rule is that a canonical pointer is written only when the
evidence proves that exact generation under the repository and ref configured
now. A legacy applied commit is not that proof on its own: the manifest may be
gone, unreadable, or stamped for a repository the stack no longer points at. In
each of those the commit is kept as recorded limitation evidence and the
pointers stay null, so the stack asks for a fetch instead of asserting a state
nobody verified. Deployed, healthy, and last-known-good are never invented,
because a manifest proves what was materialized, not what is running, and no
source acceptance is written because nobody approved through the model.
Replay is decided by scope, schema version, and configuration fingerprint. A
changed fingerprint re-runs the matrix, but a stack that already has an
application is skipped rather than rebuilt, so migration cannot overwrite
pointers written with proof it does not have. A stack whose directory has
vanished migrates to a tombstone rather than claiming a name it cannot back.
* feat(git): record rollbacks in the revision state
A restore now opens a recovery in the model before any file moves, so a crash
mid-restore leaves a target that says what it was doing rather than one that
merely looks broken, and closes it on both outcomes. The failure is classified
by whether the files had already been restored, because only a failure before
that leaves the previous workload provably intact.
Pointers move only when the restore is provable: the recovery point named a
generation, that generation still exists, and the manifest actually restored
carries the same commit and manifest version. Anything less is still recorded
as a real recovery, it simply has nothing to bind, and the transition marks it
unproven rather than guessing.
The deployed pointer is never claimed here. This path drives Compose through a
callback that reports nothing back, so binding cannot be proven and applied
moves without it.
* feat(git): add the deferred rollout-state transitions
Retry scheduling, suspension, pause, and partial rollout complete the
transition store. They have no production writer by design, but implementing
them now means the deriver has no branch a writer cannot reach, and the shape a
future producer has to satisfy is pinned rather than inferred from the read
side.
None of them is a statement about health. A scheduled retry leaves the failure
that caused it visible, so a stack that keeps failing does not read as merely
busy. A suspended source keeps everything it had accepted, and an operation in
flight is interrupted rather than abandoned so it cannot report as running for
ever. A paused or partially rolled out target keeps whatever was deployed, and
the partial record never stands in for a deployed pointer.
* test(git): stop the Compose producer tests reading Docker from the host
Both deploy cases asserted that the compose command would reject, which was
only true on a workstation with no Docker daemon. On Linux the command
succeeded, the promise resolved, and the assertions failed.
The compose subprocess now reports an exit code each test chooses, so the
adapter is what decides the outcome rather than the machine. Only the verbs
that need a daemon and travel through spawn are answered; `config` still runs
for real, because a host with the CLI and no daemon parses compose files
exactly as CI does, and failing that call breaks every create in the file.
That also makes the bound path reachable, so the deploy test now covers it:
a failed compose leaves the deployed pointer alone and classifies
post-mutation, and a successful one binds the applied generation, returns it
for health to bind against, and clears the earlier failure.
* fix(git): contain every managed-area path at the call that uses it
A stack name reaches the managed root without passing through
`isValidStackName` on this path, so the marker, the cleanup, and the create's
root probe all built filesystem paths from unvalidated input. Each call now
resolves its target against the managed area and checks containment in its own
scope, which is also the form the security scan credits: it does not follow the
barrier through the shared `isPathWithinBase` helper, so the check has to sit
with the call it protects.
`cleanupUnclaimedManagedRoot` removes a whole root recursively and gets the
same check, even though it was not among the reported calls.
Clearing a staging marker can now fail, so the two boot-recovery branches that
dropped the checkpoint first were reordered. Losing the checkpoint while the
marker survives would report the area as settled and leave the stack name
uncreatable.
The deploy adapter's two failure logs built their format string from the stack
name; both now use a constant format with the name as an argument.
The marker and cleanup fixtures stood a bare temp directory in for a managed
root, so nothing exercised the invariant these checks enforce. Both now build a
real managed area over a scoped data directory.
* fix(git): refuse to claim a managed area whose marker cannot be read
A marker that exists but will not parse is still someone's claim, yet the write
path refused only a readable foreign marker and wrote straight over a corrupt
one, discarding the reason and logging nothing. A transient permission error, a
marker truncated by a crash between write and rename, or a full disk all read
as corrupt, and each one let a second create take ownership of an area the
first still owned. Every other path in the module preserves on corrupt; this
one now does too, and says why.
Clearing a settled create's staging marker can fail, and reporting that as a
retained create sent a reader looking for an unfinished create that had
finished. It reports `marker_retained` instead. The checkpoint is still kept
for the retry, so its encrypted token can outlive the create while the clear
keeps failing; that tradeoff is stated at the call, because the alternative
leaves a marker no later create can get past.
The reaper also computed why it was preserving an area and threw the reason
away, so a directory could survive every boot with nothing said about it.
The containment checks these paths rest on had no tests and were silently
deletable, so the out-of-area read, write, delete, and reap now have them.
* feat(git): bind and observe a recovered generation
A proven restore could never claim the deployed pointer, because the restore
path drives Compose through a callback that reported nothing back: a caller
that restored some other way resolved identically, so binding on a resolved
promise would have claimed a workload nobody launched. The Compose wrapper now
returns what it did, and only that answer binds.
With binding reachable, a bound recovery claims its health run inside the same
transaction as the pointers it describes, and arms the timer once that
transaction lands. Committing the two together is the point: a crash between
them would otherwise leave a restored workload nothing was watching. Anything
that stops the timer starting writes the run off immediately, so an observing
row never outlives the timer meant to watch it, and a reservation is never
armed across a restart.
The reservation is handed to the transition rather than reached for, because
the health gate reports its verdicts back through the transition store and
importing it there would close a module cycle.
Startup now finalizes each interrupted observation on its own instead of
sweeping them with one update, so the revision state hears a verdict for every
run rather than watching the rows change under it.
* feat(git): add the rollout-scoped rollback transitions
Completes the deferred-state set. The three aliases write the same recovery
columns under the same rules as the recovery events, differing only in
provenance: Direct Git recovery emits `recovery_*`, and a later rollout
producer emits these. Nothing in this PR writes them, which is why they are
tested directly rather than through a caller.
`partial` is the one failure class they add, for a rollback that reached some
targets and not others. Completing has no unproven variant on purpose: a
rollback nobody can bind to a generation has nothing to complete against, so
it refuses without one and refuses a generation another application owns.
Also covers the recovery health reservation end to end: reserving writes and
links the row without arming anything, a replay reuses the run rather than
opening a second observation of the same restore, arming inserts nothing and
supersedes a conflicting stack gate, an unarmed reservation is written off
once, and a reservation that outlived its process is finalized rather than
armed. The per-row startup sweep is pinned too, since one row whose write
fails leaving the others finalized is the entire reason it replaced a bulk
update.
* feat(git): add the Blueprint source and deployment transitions
The first slice of the Blueprint work: the store side, with no production
caller yet. The routes and the reconciler are wired to these next, and pinning
the contract first means a caller cannot quietly satisfy a different one.
What these enforce is that a terminal event names the request it answers. A
deploy records the intent and candidate it was launched for, and a terminal is
accepted only against that same request, matching stage and identity on one
side rather than either in isolation. Without that pairing a deploy could be
acknowledged out of an in-flight withdraw, or one request's intent stored
against another's candidate. An acknowledgement carries the candidate from the
matched request rather than from its own payload, for the same reason.
Releasing an operation now releases its identity with it, and a resolved
interruption is retired rather than left to match again. Both were leaking
through the shared helpers: a stale identity let a later start resurrect a
superseded intent as live, and a surviving interruption both reported
completion as unknown for ever and let a late acknowledgement regress the
target after two later deploys had succeeded. Fixing the helpers fixes the
Direct paths too.
A start refuses to displace an unrelated operation, which would otherwise
abandon it with no terminal event and no history saying so. Observations write
the latest stage, since the placement facet reads it to decide whether a
stateful deployment is waiting on confirmation, and history alone could never
reach a reader.
Minting is left to the caller. A no-op Blueprint edit must mint no intent,
because a fresh identity would invalidate acknowledgements that are still
accurate.
* feat(git): record Blueprint create, edit, pin, and delete
Wires the Blueprint routes to the revision state. Each producer writes the
Blueprint source row and its GitOps rows in one transaction, so an operator
never sees a Blueprint that exists with nothing describing what it means, or an
intent for a Blueprint that failed to save.
The question these answer is when an edit invalidates what the fleet already
acknowledged. Changing the name, the compose content, the selector, the drift
mode, or whether it is enabled changes what nodes run or where, so each mints a
new intent and candidate. Changing the description or the classification
changes how the Blueprint reads and nothing a node can observe, so it updates
the source row alone: minting there would make every acknowledgement report as
stale over a reworded sentence.
An edit is measured by value, not by which fields were submitted. The editor
sends every field on every save, and the source layer decides what to
invalidate from which keys are present, so only the keys that genuinely differ
are passed down. Without that the two disagreed: rewording a description
advanced the revision past the one the current intent describes and cleared the
approval, while this layer classified it as metadata and minted nothing.
Selectors compare by value too, so reordering a list that names the same nodes
is not a placement change.
That measurement also changes one behaviour: a save that alters nothing now
leaves the approval alone, where before any save cleared it. Nothing changed,
so nothing is invalidated.
Pinning revises placement the same way a selector edit does, and re-pinning the
node already pinned is not a change. Deleting tombstones the application and
its live targets so the Blueprint stops claiming its slot, and withdraws
nothing itself, because the route has already done that and doing it twice
would record removals that never happened.
The required node set is stored in a canonical order, so reordering it is not
mistaken for a placement change. A Blueprint that predates the model has no
application yet and is left for migration rather than given a first intent
here, which would claim a starting point for deployments nobody has reconciled.
Desired nodes are passed in rather than computed here: the reconciler that
knows how to compute them reaches this layer, and importing it back would close
a module cycle.
* feat(git): record the placement a label or cordon moves
A label and a cordon say nothing about any one Blueprint, but both change which
nodes a selector matches. Each now revises placement for whichever Blueprints
the change actually moved, by comparing the desired nodes either side of the
write rather than reacting to the event.
That comparison is the whole point. Labelling a node no selector mentions, or
cordoning one no Blueprint wanted, moves nothing, and minting an intent for it
would invalidate every acknowledgement in the fleet over an edit no node can
observe. The same comparison covers a Blueprint pinned to a cordoned node: a
cordon governs automatic placement only, so the pinned target still wants that
node and its set does not move, without needing a case of its own. Nodes are
compared as a set, so the same nodes returned in a different order is not a
change either.
Each route wraps its existing write and the recording in one transaction, so a
recording failure cannot leave a fleet selecting on a label nothing recorded.
The write itself is untouched, which keeps label validation and the per-node
limit where they already live rather than restating them in a producer.
Node deletion already retires its targets, so this adds nothing there.
* feat(git): record Blueprint deployments by what caused them
Every production write to a Blueprint deployment row now goes through one
funnel that records the cause. The cause is carried rather than inferred from
the resulting status, because several causes land on the same one: a deploy
that failed and a withdraw that failed both read `failed`, and they mean
opposite things about whether the deployment is still on the node.
Recording is skipped when the status did not move, so a reconciler tick that
re-asserts a state it already reported does not append a second event
describing the same fact. The write still happens either way.
A terminal answers the request the target says it was given, not whatever the
Blueprint currently wants: an acknowledgement matched against the current
intent would accept work for a revision that node was never sent. The first
deploy to a node creates its target, since a Blueprint application has no
targets until something is sent somewhere.
Recording never fails the deployment. The rollout already happened, and turning
a bookkeeping problem into a stuck rollout would be the worse outcome.
The four reconciler observations record what was seen and nothing else. Preview
cleanup stays outside this path deliberately: it reverses a projection nobody
deployed, so recording it would report removals that never happened.
* fix(git): record an Inline Blueprint coming into existence
Creating a Blueprint inserted its application row directly, so the history
began at the first intent and described an application nothing recorded coming
into existence. Every other application-creation path emits the activation
event, and the event's own definition covers this case: an Inline Blueprint
inserts the application, the first intent, and the first candidate.
Activation goes through a transition now, which also enforces the one thing the
direct insert could not: a Blueprint gets one live application, refused rather
than silently duplicated.
No target is created. A Blueprint application has no targets until something is
deployed somewhere, unlike a Direct one which always has the node its stack
lives on.
* fix(git): file withdraw failures as withdraw failures
Four of the five withdraw failure paths were recorded as failed deploys. Only
the thrown one was tagged correctly, and it is the rare case: a delete-lock
conflict, a network error, a remote 409, and a remote non-200 all return rather
than throw, and all four were filed under deploy.
That inverted the thing this funnel exists to prevent. Recording a deploy
failure clears the in-flight operation, so a routine "another operation is
already in progress" wiped the withdraw that had just started and left the
target reporting a failed deploy while the deployment was still on the node.
A withdraw refused because the on-disk stack is not ours is now its own cause
too, rather than borrowing the deploy-side name conflict.
The status guard no longer suppresses starts. A start writes the identity that
terminals are matched against, so skipping one because the row already read
`deploying` let a later acknowledgement answer a request that had been
superseded, reporting a node as converged on a revision it was not running. The
guard now covers observations only, which is where repetition is the reconciler
re-asserting a state it already reported. A newer deploy supersedes an older
one so a redeploy of a stuck request can take over; anything else in flight is
still refused rather than displaced with no terminal event.
A rejection is logged differently from an infrastructure error, because a
target the model keeps refusing holds its active slot and stops recording
anything further, and that is worth seeing.
* feat(git): migrate pre-existing Blueprints into the revision state
Blueprints that predate the model are brought in at boot: an application, an
intent describing what the Blueprint currently asks for, and a candidate marked
as coming from the legacy inline record.
None of it is an acknowledgement. The Blueprint revision is carried for display
and nothing else, because a revision number is not evidence that any node is
running what it names, and recording it as agreement would report a fleet as
converged on an intent nobody verified. No targets are created for the same
reason, and the candidate's required set is empty: migration records what is
asked for, never which nodes currently satisfy it.
An approval authorizes the intent it was given for, so a Blueprint that was
never approved, or edited since, is recorded as needing reapproval rather than
left blank. Blank would read the same as an approval that is merely absent.
Replay is decided by the same scope, schema version, and fingerprint the Git
migration uses, and a Blueprint the new path already described is skipped
outright, since its rows were written with proof this pass does not have.
Blueprints migrate in their own pass. Coupling them to the Git migration would
let one unreadable Git stack keep every Blueprint outside the model.
* fix(git): contain the blueprint stack-directory probe at its call
The ownership probe resolved a path from the blueprint name and checked it
through the shared helper, which the security scan does not credit as a
barrier. The check now sits with the call it protects, matching the form used
elsewhere for the same reason.
The path was already validated, so this changes no behaviour. It surfaced now
because an earlier commit in this branch rewrote the file's line endings, which
made every line read as new and brought a pre-existing call into the scan's
changed-code window.
* feat(git): read GitOps history and carry revision state on source reads
Adds the instance-local history API and the additive revision fields the
source reads have been missing.
`GET /api/git-sources` and `GET /api/stacks/:stackName/git-source` now
carry `gitopsRevision` and `stackResourcePresent`. Only the instance that
owns the files can say whether a stack directory is really there, so that
answer travels with the response rather than being inferred by a reader
that has never seen the filesystem. It is read through the strict stack
listing: the lenient one answers a failed directory read with an empty
list, which here would read as every stack having vanished and would turn
an unreadable disk into an empty audit trail.
Two history routes land with them: `GET /api/git-sources/history` across
stacks, and `GET /api/stacks/:stackName/git-source/history` for one. Both
page on `(created_at, id)` so rows written inside the same millisecond
survive a page boundary, and both bound how far one request will scan. A
malformed cursor or an unusable filter value is refused rather than
ignored, because answering "show me the failures" with everything reads
as an answer instead of a non-answer.
Authorization is decided per row. A row reduces to a stack read only when
it names a stack, its application is live, and the stack is present on
disk; anything unprovable stays with Admin. The cross-stack routes
classify every row, while the per-stack route is authorized whole by
name, so reading a stack shows its full history including entries from an
earlier application. That distinction is carried in the scope type rather
than a flag, so skipping the row classifier without pinning the query to
the authorized stack cannot be expressed.
History entries record the fields each transition moved rather than a
whole revision, so lifecycle for the classifier comes from the owning
application row. That keeps authorization off the recorded payload
entirely: an entry whose detail cannot be read still returns its identity,
stage, and outcome with a stated limitation, and the decode failure is
logged rather than passed over in silence.
* feat(git): let auditors read GitOps history they cannot tie to a stack
A history entry that cannot be tied to a readable stack fell to Admin,
which left the auditor role seeing exactly what a viewer sees on the one
surface it exists to read.
History entries are an audit trail: insert-only, and recording the actor,
trigger, stage, and outcome of every transition. The request audit log is
already gated on the audit permission, so an entry whose audience cannot
be narrowed now falls there too. Withholding it protected nothing in any
case, since the request log already records that a Git-source mutation
happened and who made it.
The source collection deliberately keeps its Admin fallback. Those rows
are live Git configuration, not a record of events, and a mandate to
audit does not extend to reading the repository, ref, and credentials
policy of stacks that were deleted or never finished being created.
The fallback is now part of the requirement type rather than a role
comparison, so each surface states which audience it falls back to and
the compiler requires every case to be answered.
* feat(git): correct remote node identities on GitOps reads
A remote instance numbers its own nodes and has never heard of the hub's
numbering, so every node id it reports is a statement in its own
namespace. Read straight through, a hub joining two nodes showed two
different machines as the same node.
Four GETs now take a second hop that buffers the response and corrects
those identities: the git-source list and detail, and both history
routes. Everything else keeps streaming exactly as before, which is the
reason this is a separate hop rather than a mode of the existing one:
logs, downloads, and event streams must never be buffered, and the
enumerated positions rewritten here do not appear in them.
Only JSON numbers are replaced. A null node is preserved, since "no node"
is a fact the remote is entitled to state and inventing one there would
claim a placement that does not exist. Strings, application ids, and
stack names travel untouched.
The hub re-authorizes what comes back. A remote authorized its rows for
the machine account the hub proxies with, which says nothing about the
person behind the request, so every row is classified again against the
signed-in user. To make that possible without the hub holding another
instance's database, each history entry now carries the lifecycle of its
owning application alongside the stack-presence answer already there.
Both are validated fail-closed, and a verdict a remote might volunteer is
ignored: the hub decides, the remote supplies evidence.
Asking one node for another node's history is refused before the hop
rather than forwarded, because the remote would answer about itself and
the page would read as an answer to a question nobody asked. When the
node asked for is the node being talked to, the hub translates that into
a flag the remote resolves to its own default node. Only the hub may set
that flag, and a remote honours it only on a proxied hop.
Every way this hop can end converges on one terminal answer: rewritten,
passed through, too large, undecodable, failed upstream, or abandoned by
the client. A body the hub could not read never borrows the remote's
success status.
* test(git): assert the unlinked git-source response field by field
The detach test compared the whole response to `{ linked: false }`, which
stopped holding once that route started carrying the additive revision
fields. The equivalent backend route test was updated when those fields
landed; this one was missed.
Asserted field by field rather than loosened to a partial match, so the
two new fields are actually checked: a detached stack has no application
to project while its directory is still on disk.
* feat(git): carry GitOps revisions on blueprint, node, and drift responses
Completes the additive JSON half of the revision-state read contract, so the
Blueprint, node, and drift surfaces report GitOps state through the same
projection the Git-source routes already return instead of leaving it
unreadable.
Blueprint list, detail, create, update, and pin carry gitopsRevision. Node-label
add, cordon, uncordon, and node delete carry gitopsRevisions, ordered by
blueprint id and covering only the Blueprints the mutation actually moved: an
edit no selector reacts to reports an empty list rather than invalidating the
whole catalog. Node deletion reads its Blueprint owners inside the deletion
transaction and before the tombstone, which is the last moment a target row can
be traced back to an application, and an orphaned target is logged rather than
being silently indistinguishable from a Direct one. Both DELETEs that answered
204 still answer 204.
The drift GET and re-check gain the same field alongside the existing ledger,
which is untouched; no GitOps class is written into stack_drift_findings.
Reads and mutations treat a projection fault differently, on purpose. On a read
the revision is part of the answer, so a fault surfaces. On a mutation the write
has already committed, so the revision is decoration and is built defensively:
letting it throw would answer a successful cordon or a completed node deletion
with a 500, sending the operator to retry a hard delete that already happened
and be told the node does not exist. It degrades to an empty list and logs.
Because both drift routes answer with that projection, the identity hop now
intercepts them too, matching its route table per method instead of rejecting
every non-GET. Left as they were, one object would carry the hub's node
numbering or the remote's depending only on which route asked for it. The
allowlist names a single mutation rather than opening a verb, and the hub's
collection filter still keys off the git-source paths alone, so drift payloads
are rewritten without being re-authorized: they are per-stack and already
authorized by name before the hop.
Tests cover the exact shapes, the preserved 204s, revision ordering, the
empty-list cases, node deletion reporting its tombstoned Blueprints and
surviving a projection failure, and the proxy's new route matching, drift
rewrite, and filter exclusion.
* fix(git): resolve every application a GitOps surface can own
Three gaps found while building the additive JSON, all the same shape: an id or
a sentinel stood in for a row, so "there is nothing here" and "the thing that
should be here is unreachable" collapsed into one answer nobody could tell
apart.
A Blueprint application is stored with stack_name NULL, so no lookup by stack
name could reach it, while the reconciler materializes every Blueprint as a
stack directory of that name. A stack's Drift tab and Git panel therefore
reported no GitOps at all for a stack GitOps was actively managing, while the
Blueprint page reported a live application for the same thing. Stack surfaces
now bridge the two through the deployment row for the asking node. The
deployment row is what makes it safe: Blueprint and stack names share one
namespace, so matching on name alone would let a Blueprint claim an unrelated
stack of the same name on a node it never targeted.
A retired application was unreachable for the same reason, because every entry
point filtered to the live rows. Tombstoning deliberately keeps the configured
identity and SHA pointers as frozen facts so the projection can still say what
an application was, and the source deriver has a not_live status waiting for
exactly that, but nothing could hand it one. A detached Blueprint or stack now
reports what it was instead of reading as one that never existed.
The not-applicable projection was typed so it could not carry a reason, which
made a row that vanished between resolving it and re-reading it by id identical
to a stack the model was never asked about. That one field widens to hold
limitations, and the missing case now says so with the id as evidence. Every
other missing-row path in the deriver already worked this way.
Also logs, rather than silently skipping, a live application whose blueprint row
is gone: with cascade off that is a real integrity fault, the placement did
move, and the response would otherwise report that nothing had.
Cordon and uncordon are deliberately left reporting nothing. A cordon governs
whether new placements may be made, not what a Blueprint asks for, and the
reconciler applies it only to new placements, so revising intent for it would
invalidate acknowledgements fleet-wide over a change that evicts nothing. Two
comments implied otherwise and now state the contract.
* fix(git): keep stack-state resolution out of the Git-source read path
Review of the previous commit found the fallback it added had reached further
than intended. The resolver is now split by the question it answers.
projectStackRevision answers "what Git source is attached to this stack" and
stays Direct-only. projectManagedStackRevision answers "what manages this stack"
and is what the drift routes use. The Git-source routes must not be answered
with another application's identity, and there was a second reason: the row
classifier takes the stack name from the Git-source row but the lifecycle from
whatever the projection resolved, so a Blueprint application arriving there
would have turned an Admin-only row into one any stack grant could read. The
Blueprint that unlocked it could be the very one holding name_conflict because
of that stack.
The Blueprint-to-stack bridge also needed the right predicate rather than a
present deployment row. name_conflict is written precisely when a stack of that
name already exists on the node and Sencho does not own it, so guarding only
against withdrawn admitted exactly the collision the bridge exists to prevent.
It now requires last_deployed_at, which proves this Blueprint wrote the
directory, and excludes name_conflict and withdrawn: the same predicate the
delete and withdraw paths use. Only a live Blueprint application qualifies,
since a retired one has no claim on the directory.
The detached fallback no longer resolves deleted applications. Detached means
the files are still on disk and still describe that stack; deleted means the
stack is gone, so any directory of that name now belongs to something else and
reporting the old repository and SHA against it would disclose one stack's Git
identity through another's name. The authorization side already refused deleted
rows for that reason, so the two now agree. Ordering breaks ties on rowid rather
than a random UUID, and two partial indexes cover the new lookups, which the
existing live-only unique indexes could not serve.
The not-applicable variant's limitations are readonly, so pushing onto the
shared frozen instance is a compile error again rather than a runtime throw.
Tests cover a Blueprint claiming a stack it never deployed onto and one it hit a
name conflict on, a detached Direct source projecting not_live, and a deleted
one staying unresolved for a reused name. Drift fixtures move to afterEach so a
failing assertion cannot leak state into the next test.
* fix(git): stop the stack resolver falling through proven ownership
A second review round found the resolution chain could answer with the wrong
application, and that part of the previous commit was reading a state nothing
writes.
The Blueprint bridge returned one value for two different facts. Once the
deployment row proved a Blueprint had written the directory, a missing
application row for it still answered "no Blueprint here", so the chain
continued to the detached-Direct lookup. A stack that once had Direct Git, was
detached, and whose directory a Blueprint later took over would then report the
old Direct application's repository, ref, and SHA as that directory's state.
The bridge now separates "not mine" from "mine, but broken": the second logs and
returns its own limitation, so the fault is visible and no unrelated identity is
offered in its place.
A Blueprint projection also carried the whole placement roster into a route
authorized by a grant on one stack name. Targets are now scoped to the node
being asked about, which is both the safe answer and the accurate one, since the
question is what manages this directory here.
The detached-Blueprint lookup, its index, and its fallback are removed. Blueprint
retirement writes deleted, never detached, and the only detached writer is the
Direct detach path, so the getter could never match and the index covered an
empty set. The test that seemed to cover it built the state by hand and was
green against something the product cannot produce; it now drives the real
delete route and asserts what that path actually writes.
The Git-source resolver goes back to live applications only. Detach removes the
source row in the same transaction that tombstones the application, so a source
row beside a detached one is not producible, and the route's row classifier
takes its stack name from the source row but its lifecycle from the projection.
Keeping that resolver narrow is what stops a resolution change from quietly
moving who may read a row. The comment saying so is restored and now explains
why it must stay true.
* refactor(git): share the GitOps application fixture across the route tests
Two test files carried near-identical 58-line GitOpsApplicationRow literals that
differed only in a repo URL nothing asserted on. Both now import one helper.
The drift tests also repeated a Blueprint seed and its activation across four
cases, and re-imported DatabaseService per test; those become local helpers and
one hoisted import, which lets the cleanup hook go synchronous. Node lookup
throws on a missing default instead of casting.
projectBlueprintRevisions is module-private again: its only caller is the
committed-projection wrapper beside it.
A third copy of the same fixture remains in git-source-routes.test.ts. It is
left alone deliberately: it pins timestamps at 1 rather than now, and that file
has ordering-sensitive history assertions that would need auditing first.
* feat(git): mirror the GitOps revision contract on the frontend
Adds the client-side read contract for the GitOps revision projection, with
no consumers yet.
types/gitops.ts is a hand-written mirror of the backend read contract, in the
same convention as the other domain type files: the frontend never imports
backend. The projection's two arms are named separately so a component can
take the live one as a prop without re-narrowing, and the absent arm keeps its
absent keys rather than nulls so reaching for a lifecycle status without
narrowing is a compile error. The limitation code stays an open string, since
the backend adds codes without a schema bump and an exhaustive switch would
silently stop rendering the newest ones.
lib/gitopsState.ts is the one place a facet status becomes words and a colour,
so a state reads the same in a sidebar tooltip, a panel banner and the drift
tab. Both maps are keyed on the closed status unions, so widening the mirror
without naming the new state fails this build.
Every line of copy states the condition the deriver actually tests, which is
not always what the status name suggests. recovery_required is a recovery
already running rather than one that is needed. candidate_ready is reached only
when review is not required, so it is ready to apply, not ready to review.
synced_and_healthy is also reached with the health gate switched off and never
compares the deployed generation against the accepted one, so it claims
neither. never_reconciled means nothing has been accepted; a fetch that
produced no materialization still records its commit.
pendingSourceStatus keys "an update is waiting" on the candidate pointer rather
than the status name, because source_reconcile_required is reachable both from
a stale candidate and from an accepted generation with no candidate at all.
Retirement is excluded first: tombstoning keeps the candidate pointer as a
frozen fact, so a stack detached mid-review still carries one.
absentFault separates the two facts the empty projection carries. Empty
limitations means the model was never asked about this stack and the right
rendering is nothing; a non-empty list means an application that was expected
could not be reached. A live application's limitations are caveats on state
that is being reported, not faults, so they are deliberately excluded.
* feat(git): name the Git source state on the stack and sidebar surfaces
The sidebar indicator and the anatomy source row both read a raw pending
commit pointer, so a candidate blocked by local conflicts, held for review, or
stale against the configuration in force all render identically to one that is
ready to apply. The projection distinguishes them; nothing was reading it.
refreshGitSourcePending now derives each stack's state from the projection and
the pending map carries that state rather than a boolean. Presence in the map
still means exactly what it meant before, because it is keyed on the candidate
pointer rather than the status name: source_reconcile_required is reachable
both from a stale candidate and from an accepted generation with no candidate,
and only the first is something to review. One narrow fallback remains, for a
row with no projection at all: a failed GitOps write is logged and swallowed
while the pending commit still commits, so the flat pointer is the only thing
that can answer there. Wherever a projection exists it is the sole authority.
The sidebar keeps one indicator, in the same slot, at the same size and colour,
with the same position in the priority ladder. Only the tooltip changes, so a
blocked plan now says so instead of reading as an ordinary update. A test
asserts the rendered indicator is identical across states, which is what keeps
the rendered sidebar unchanged.
The anatomy source row keeps its pulsing dot alone for the ordinary case and
adds the state word only for the states the dot cannot express. Inline rather
than in a tooltip, since this is the reason something is stuck.
* feat(git): surface derived GitOps state in the Git source and Drift panels
The Git source panel's pending banner had two things it could say, read from a
raw commit pointer and a single blocked boolean. It now names which of four
states the candidate is actually in, so a plan blocked by local conflicts, one
held for review, and one gone stale against the configuration in force stop
rendering as the same ordinary update. The short commit sha stays.
A new card above it reports an application the projection could not reach. That
case renders as nothing at all today, which is indistinguishable from a stack
the model was never asked about. Empty limitations stays silent, because that
is the ordinary answer for most stacks and a header over an empty block would
be worse than nothing.
The summary block gains one row naming the source state. It is the first place
the panel can show applying, retry scheduled, suspended, recovering, or a
detached source, none of which have a pending commit and so never reach the
banner. Both existing rows are untouched and the last applied commit stays a
display fact.
The revision lives in its own state rather than on the Git source type, because
the PUT that saves this panel answers with a bare source and no revision. A
save drops it instead of rendering a state the write has already moved.
The Drift tab gains a third block below the two it already has. The compose
versus runtime card and the since-last-deploy card keep their exact positions
and copy; the new block answers a different question, which generation this node
was asked to run and whether it got there. A Blueprint-owned stack shows its
target rows and no source card, since a Blueprint application has no Git source
and inventing one would be a claim the model never made. The drift class list
renders against the type and expects no rows, because nothing populates it yet.
* feat(git): declare revision fields on the blueprint and node clients
Types first. The blueprint list, detail, create, update and pin responses all
carry a GitOps revision, and the node label add, cordon, uncordon and delete
responses carry a list of them. None of that was modelled, so the fields were
being dropped silently and the next reader would have taken these for bare
records.
Most of it is declared and deliberately unread, with the reason written where
the type lives. Create, update and pin are followed immediately by a re-read of
the catalog or the detail that carries the same projection, so rendering the
mutation's copy would show the same fact twice with one of them stale. Cordon
and uncordon always answer with an empty list by design, because a cordon
governs whether new placements may be made rather than what a Blueprint asks
for, and the reconciler leaves existing deployments where they are. The comment
on that type is the point: the risk there is a future reader building a
consumer for a list that is never populated.
Two places do report something. Deleting a node and adding a node label both
re-place Blueprints, and both now say how many. The count only, and only when
it is not zero: an empty list means both that nothing moved and that the
projection faulted after the write had already committed, so it can never be
reported as the first.
Blueprint detail gains the same unreachable-application card the Git source
panel has. A Blueprint with no live application row is a fact nothing in the
product could previously express, which is a different problem from having two
ways to say the same thing.
* fix(git): close the gaps review found in the GitOps frontend consumers
Six defects, each verified against the code before being fixed.
The pending-map read assumed every /git-sources row carries a projection. That
route is proxied, so a node predating the model answers rows without one, and
dereferencing it threw inside the loop. The catch swallowed the throw, the map
was never set, and every stack's Git indicator froze at its previous value for
as long as that node was selected. The field is optional now and a row without
one falls through to the same branch as a row with nothing to project.
That fallback also branched on the arm alone, so a projection reporting an
unreachable application plus a stale commit pointer was reported as a candidate
ready to apply. It now excludes a faulted projection: a fault means an
application was expected and could not be read, so the pointer is not evidence
that anything is ready, and naming a state there would be a guess.
Detaching a Git source cleared the source but not the revision, and the pending
card is derived from the revision alone. The panel kept advertising a waiting
commit for a stack Git no longer managed, behind a Review button that silently
did nothing. A read that threw had the same problem across stacks: the panel is
reused, so stack A's state could render under stack B's header.
The panel also lost the flat-pointer fallback the sidebar keeps, so the two
surfaces disagreed: the sidebar would show an indicator and clicking through
showed no card. Both now apply the same rule.
Node delete parsed the success body unguarded. The delete has already committed
at that point, so a malformed body would have reported a completed deletion as
a failure and skipped the refresh, leaving the deleted node on screen.
Tests: five mutations that previously survived now fail. The zero-pixel claim
compared the wrong element and passed when the indicator vanished entirely; the
stale-read test asserted during the loading window, when the body is skeletons
regardless; the sidebar passthrough test proved only that an indicator existed,
not that the state reached it. Added coverage for the drift row, an unknown
node, the not-applicable source guard, the save clearing, the Review button, and
a live application carrying a caveat, which must not read as a fault.
The fixture set drops two source statuses whose identity defaults describe a
state they cannot be in, and gains a drift-item builder.
* refactor(git): share the GitOps fault card and live-facet derivation
The unreachable-application card was built inline on three surfaces, and the
copies had already drifted: the Git source panel used one icon while the Drift
tab and the Blueprint sheet used another, so the same failure rendered two
different ways. It is one component now, which also owns the state key and the
test id those surfaces assert on.
Deriving the live source facet was likewise repeated, along with the two
semantic decisions behind it: the absent arm has no facets, and a source facet
of not applicable means a Blueprint owns the stack rather than that something
is missing. Both now live next to pendingSourceStatus, which already encoded
the same exclusion.
The Git source panel's seven derived values collapse to three, with the
pending-commit rule extracted to a named function so the four cases read
straight through instead of as nested ternaries. Behaviour is unchanged in
every case.
The two flat-pointer fallbacks are deliberately not unified: they look alike
but the panel also treats a live application whose source facet is not
applicable as unanswered, and merging the predicates would change what that
case renders.
* feat(git): count and announce committed GitOps transitions
Every history row that is actually inserted now produces one metric
increment and one state-invalidate event, so the surfaces that read GitOps
state hear about a change instead of waiting for the next poll.
Announcement is buffered and drained on a macrotask rather than fired
inline. better-sqlite3 is synchronous, so waiting for the macrotask puts
the drain after the transaction that wrote the row, and after any outer
transaction wrapping it, without having to detect which nesting depth it
is in. The drain confirms each row is still present before announcing it,
so a rolled-back transaction says nothing on its own, and the insert
declines to queue a dedupe replay, so a retry says nothing either. The
broadcaster is injected at startup rather than imported, keeping the
GitOps layer free of a cycle back into the notification stack.
Counters are process-local and in-memory, and their keyspace is finite by
construction: history stages are now a closed union that the build
enforces, and outcomes were already a closed set. The payload names no
stack, node, repository, or actor, because a counter carrying those would
be an audit trail with no retention rules and no per-row authorization,
which is what the history routes provide. GET /api/gitops-metrics is
Admin-only and instance-local, so selecting a node answers with that
node's counters.
On the client, a gitops-scoped invalidate refreshes the derived state
through a 250ms trailing window, matching the existing stack refresh. One
operation commits several transitions in a row and a first-boot migration
commits a great many, so refetching per event would thrash the API for a
picture that only settles at the end.
* feat(git): show GitOps source state on the stack dashboards
The dashboards list every stack on a node, which is where a fleet-wide
reading of Git state is most useful and where, until now, a Git-backed
stack looked exactly like a local one. Each row that the model has
something to say about carries a chip naming its source state.
The chip reads from the same status vocabulary the panels and the sidebar
use, so a stack cannot be "pending update" in one place and something else
in another. It sits beside the stack name rather than replacing the source
column: the column says where the files come from, the chip says what
GitOps makes of them, and those are different facts. The label is a word
and the title is a whole sentence, so the state never rests on colour
alone; the phone rows keep the word on screen, since touch has no hover.
The join is by stack name, which is what the dashboards have, and its
source is the same proxied route the sidebar reads. A row without a
revision, a Blueprint-owned application, and a projection fault all leave
the stack unbadged rather than inventing a state for it. State arrives by
announcement rather than by poll, since it only moves on a transition, and
a node switch blanks the map first: stack names repeat across nodes, so a
slow answer for the node just left is discarded rather than allowed to
label the wrong stacks.
* feat(git): say which part of a GitOps state could not be proven
Eighteen conditions can qualify a live projection: a manifest that does
not match the branch configured now, an approval that could not be
restored after a recovery, an artifact record that has gone. Every one was
already recorded and none of them reached the operator, so a state with a
hole in it read exactly like one without.
Each now has operator wording under the state it qualifies, checked
against the site that emits it rather than against the code's name, since
several names describe something narrower or wider than the condition
actually tested. The stored messages stay out of the UI: they are written
for a log reader, and some are raw decoder errors.
The presentation is deliberately quiet. A caveat is not a failure: the
state above it is real, and one piece of evidence behind it is missing, so
the reader learns which part to distrust without being told the whole
thing is broken. Faults keep their own card, because those replace the
state rather than qualifying it, and the two arms are read through
separate helpers so neither can be rendered as the other.
A code with no wording names itself rather than vanishing, so a node
running ahead of this build degrades to something honest instead of
reporting full confidence in a state its own backend flagged.
* fix(git): record a stateful first placement and keep a half-built Blueprint editable
Two faults that today's product cannot reach, and that the Git-backed
Blueprint mode would reach immediately.
The live-application lookup answers with an application that is active or
still being created, because its other callers ask whether the Blueprint
already holds the live slot, where a half-built row counts. Three
producers passed that answer straight into transitions that accept only an
active application and reject anything else by throwing, inside the
caller's own transaction. A Blueprint edited or pinned while its
application was still being created would have failed with a server error
and rolled the Blueprint write back with it. The producers now narrow to
what they actually require, through one shared predicate, leaving the slot
check honest.
Separately, a stateful placement is held for operator review before
anything is deployed, so no target exists when that hold is recorded, and
the observation was dropped for want of one. The hold left no trace: no
history row, and nothing on the target to say the node had been asked to
hold anything. First contact now creates the target, exactly as the first
deploy does. Nothing else changes with it: no intent, no generation, no
operation, and connectivity stays unset, because a node that has only been
asked to hold something has not been contacted. Observations that follow a
deploy already have a target, so they are unaffected, and a drift or evict
report for a node nothing was ever sent to is still dropped.
* fix(git): keep the Git source panel's state after a save
The save answers with the source row and no revision, so the panel had
nothing to replace its copy with and dropped it. The result was a stack
that had just been reconfigured showing no GitOps state at all until the
panel was reopened, which reads as a stack the model knows nothing about
rather than one whose state has just moved.
It re-reads instead. Keeping the old copy was not an option either: a
material configuration change clears the staged candidate server side, so
the state genuinely has moved and the panel would have gone on offering a
commit that is no longer there. Only the server can say what replaced it.
The alternative was returning the revision on the save itself. That route
is proxied, and the identity hop rewrites node numbering for an enumerated
set of routes, so adding a revision to a response outside that set would
hand back a remote node's numbering unrewritten. Re-reading costs one
request on a low-frequency action and needs no change to the hop.
* docs(git): document what GitOps state means for an operator
A commit SHA says which files Sencho wrote. It does not say whether that
commit was accepted, whether something newer is waiting, or whether an
operation was interrupted halfway, and until now the docs had no words for
any of that.
Git Sources gains a source-state table and a section on what happens when
part of a state could not be proven, since a stack with missing evidence
looks identical to one with complete evidence unless the product says
otherwise. Drift Detection gains the gitops section and explains why it
answers a different question from the two signals above it: those compare
files and containers as they are now, this reports what has been proven
over time. The dashboard page describes the state chip and, importantly,
that it and the row tint are independent, so a healthy row carrying
"pending update" is not a contradiction.
The Blueprint page now separates its revision counter from GitOps state.
The counter labels the spec; it says nothing about whether a given node
has got there, and reading it as fleet truth is the mistake the note
prevents.
The tutorial's verification step names the source state it should show,
so a reader can tell the difference between files written and a commit
accepted.
Audited and left unchanged: health-gated updates and fleet federation.
Neither presents a SHA or a revision as canonical GitOps truth, so neither
needed correcting.
* docs(git): name which saves actually clear a staged commit
The note claimed saving the form always clears the staged commit and moves
the state to reconcile required. Only a change to what gets materialized
does that: the repository, the ref, the compose paths, the project
directory, or the env sync. Changing the token or the apply behavior
leaves a staged commit alone, because neither changes what would be
written, and telling an operator otherwise would have them pull again for
nothing.
* fix(git): say when a committed transition has nobody to announce it to
Two silences worth breaking, both found reviewing the error paths in this
slice.
A server that never installs the event sink still counts every transition
and still writes every history row, so the only symptom is that no client
ever refreshes and the UI is quietly as stale as it was before any of this
existed. It now says so, once rather than per row, since a boot migration
would otherwise fill the log with one fact.
The dashboard's source-state fetch dropped a non-ok response without a
word. A refusal there freezes every badge at its last value, which looks
exactly like a fleet where nothing has changed, so the status code is
logged rather than inferred from badges that stopped moving.
* fix(git): close the defects review found in the announcement slice
Five findings, two of them able to reach an operator.
The event sink was installed one line after the deletion reconcile, which
tombstones applications and targets and awaits inside its own loop. A
drain therefore landed while the sink was still absent, so those boot-time
transitions were counted and never announced, and the warning added for an
unwired sink would have fired on every boot with a prepared deletion
intent: the fastest way to teach an operator to ignore it when it means
something. The sink now precedes every reconcile and migration pass.
The first-placement target was created outside the transition that records
it. The observation runs in its own savepoint and can refuse, and the
caller deliberately lets the deployment commit whatever the record says,
so a refusal left an active target with no generation, no stage and no
history: a placement relationship the model never established, which the
delete path would later tombstone as if it were real. Both writes now
succeed or fail together.
The badge looked its status up in a map closed at compile time, behind two
casts, while the value arrives over a proxy from a node that may run a
newer vocabulary. An unmapped key dereferenced undefined inside a stack
row and would have taken the whole table down with it. The facet and its
status are now a discriminated union, the lookup is optional, and an
unrecognised status renders nothing, which is what the join already does
for a stack it has no state for.
The dashboard's source-state loop derived each row inside the shared
try, so one row in an unexpected shape abandoned the loop and froze every
badge at its last value with nothing on screen to say so. Each row is now
guarded on its own and the count of unreadable ones is logged.
Two tests claimed things they did not check: both asserted a word was on
screen with a matcher that also matches screen-reader-only text, so a
compact badge would have kept them green, and one compared a label by
containment where the shorter label is a prefix of the longer. Both now
assert the visible node by equality. Added coverage for the window event
the dashboard actually refetches on, an unknown status, a row that cannot
be read, node-switch blanking, and a caveat recorded twice. One fixture
built a status pair the deriver cannot emit and was corrected.
Also: the guard narrowing Blueprint producers to an active application
excludes nothing today, because every Blueprint-mode application is
inserted as active and only Direct mode can be creating. The guard stays,
since the getter's slot semantics and the transitions' requirement have
drifted apart once already, but its rationale and its tests now say
plainly that they pin a defensive guard rather than reachable behaviour.
* refactor(git): make three runtime guards visible to the compiler
Type review found the same shape three times: a correct guard whose
necessity was asserted in a comment the type system could not see. This
project does not set noUncheckedIndexedAccess, so each of these read as
dead code to anything that trusts the types, and the guards protect the
behaviours that matter most here.
The limitation copy map typed its values optional, so the fallback for an
unrecognised code is now something the compiler requires rather than
something a comment explains. The badge reads its status through partial
views of the two vocabularies, so a miss is a fact TypeScript derives:
deleting the guard that stops one unknown status taking down a stack list
now fails the build, where before it compiled. Neither needed a cast.
The event payload has to be a type alias rather than an interface, because
only the former gets an implicit index signature and the broadcaster takes
an open envelope. That requirement now fails at the declaration instead of
surfacing in the startup wiring as an unexplained index-signature error.
Also added the reverse of the copy-coverage check: a code retired from the
backend leaving stale wording behind was invisible, since the fallback
only fires for entries that are missing rather than ones that linger.
* fix(git): make the state cards survive an unknown status, and correct the copy
Comment review found the protection added for the dashboard chip was
claimed more widely than it held. Five other surfaces indexed the status
vocabularies directly and passed the result straight to a card that
dereferences it, so the exact input the comments said would take a list
down still threw in the Git source panel, the Drift tab, the sidebar
tooltip and the stack anatomy row. The card now takes an optional state
and renders nothing without one, which also means a new surface cannot
reintroduce the dereference by rendering a card the ordinary way, and the
remaining four read through the lookups.
Copy corrections, each against the site that emits it rather than the
name of the code:
The manifest-identity caveat said the manifest named a different
repository or branch. It also fires when the manifest has no identity
block at all, or identifies another node or another stack, so it now says
it does not identify this stack on this node from the repository
configured now. The last-known-good artifact caveat covered a mismatch but
not the row being gone, which is the other half of the same condition.
Doc corrections: a Blueprint-delivered stack was described as carrying the
dashboard chip and a source state on the Drift tab, and it carries
neither, because a Blueprint has no Git source of its own. Per-node
Blueprint state was attributed to the detail sheet's GitOps section, which
only reports what could not be read or proven; the Deployments table is
where per-node status lives. Clearing a staged commit was said to always
land on reconcile required, but a stack that has never had a commit
accepted lands on never reconciled. The caveat lines were described as
sitting under the state when they render above the form. And the pending
banner was still documented as webhook-only, which this branch changed:
it now appears for any staged commit and its heading is the source state.
Also corrected two claims in the code: the observation branch does not
skip for a Blueprint that migration brought in, because migration creates
no targets, and the only refusal reachable from that call site is a
tombstoned target.
* refactor(git): simplify the announcement slice without moving any behaviour
A simplification sweep over the new code, all of it behaviour-preserving
and re-checked by breaking each guard afterwards.
The drain now separates policy from mechanism: it decides whether a row
survived, counts it, and either warns or hands it to a small announce
helper that owns the envelope and its error handling. The row-to-state
pass in the dashboard hook lifts out of the fetch, so one function does
the request and another does the derivation. The caveat dedupe collapses
to a set, which preserves first-insertion order and so keeps the ordering
its own docstring promises. The badge renders one span with a conditional
class rather than two that differed only by class. The metrics service
drops an internal type that was byte-identical to the one it exports.
One assertion goes: the observation branch tested membership and then
re-asserted the same fact to index the stage map, and a type predicate
lets the compiler carry it instead.
Everything the sweep was told to leave alone is intact: the buffered
drain, the row-existence re-check, the injected sink, the one-shot
warning, the partial-view lookups and their guards, the optional copy map
with its fallback, and the generation counter with its per-row guard.
* fix(git): close audit findings in the GitOps revision model
Five corrections from the pre-merge audit of this branch.
Migration no longer certifies a commit it cannot prove. The manifest read
now carries its resolved commit, and a manifest that names a different
commit than the source row records as applied is refused: the applied
directory materializes one commit while the row names another, so
trusting them together would mint a generation pointing at files that are
not the ones it claims. A manifest that names no commit at all, which is
what adoption writes, is a separate answer rather than a disagreement,
because reporting it as one would name a commit the manifest does not
contain. Both are recorded as evidence and the projection asks for a
fetch.
Create cleanup proves containment against the real filesystem before it
deletes. Every check on these paths was lexical, so a symlink or Windows
junction above the target read as contained while the recursive delete
followed it out of the managed area. Resolution keeps "is not there"
apart from "could not be read": only a genuinely missing path lets the
walk climb to an ancestor, because treating an unreadable one as absent
would infer containment for the single path whose link status could not
be established. Both sides are resolved, so relocating the data directory
onto another volume keeps working, and an area that does not exist at all
is nothing to delete rather than a suspected escape. The marker write
takes the same barrier as the marker delete, so a link cannot be written
through and then refused on the way out, wedging the stack name.
Startup stops while an interrupted create is unresolved, matching the
restore reconcile above it. A create that could not be settled leaves a
stack directory the deploy path cannot tell apart from a finished one, so
the alternative to stopping is letting a scheduler, webhook or operator
act on a half-built stack. Only that blocks: once the staged directories
are gone the create is torn down, and a staging marker nobody could
unlink is reported rather than thrown, so one failed unlink is not the
difference between an instance that boots and one that does not.
Per-stack history no longer exposes a predecessor through a reused stack
name. The grant covers the application holding the name now, which is
what keeps a stack's own entries readable while it is still being
created; every other row on that name is classified per row, so entries
belonging to an application that held the name earlier need the audit
permission, as they already did across stacks. A detached predecessor is
still readable, which is the classifier's own standing decision about
detach rather than a gap here, and is now pinned by a test.
Blueprint observations reach a reader. The reconciler recorded state
review, evict blocked, drifted and correcting against the target and
nothing projected them, so a deployed Blueprint could report itself as
never applied. The runtime facet now projects all four, below the states
a live or failed mutation puts the target in and above the pointer
checks, and any later transition supersedes the observation. The map is
declared total over the stages the reconciler can record, so a stage
added without a projection fails the build rather than silently reading
as never applied again.
* fix(git): confine managed-area cleanup to each stack's own location
Cleanup proved only that a resolved path landed somewhere inside the
managed area. That is satisfied by every other stack and every other node
in it, so a link from one stack's generations directory into another's
passed the check while the delete took a generation belonging to someone
else. The check is now positional: the managed area is resolved once, so
relocating the data directory onto another volume still works, and every
segment below it must be reached without redirection. The create-path
sinks use it on the write as well as the delete, so a claim cannot be
written through a link and then refused by the hardened delete, and the
manifest service's pre-existing deletion sinks (generation pruning,
boot-sweep orphan reaping, detach staging and finalization, and
whole-area removal, including after a restored snapshot) run it before
their recursive deletes too. Each of those sinks also keeps a literal
containment comparison beside the positional check, because static
analysis credits only a comparison at the call and reports the delete
otherwise. A refusal names both paths in the log, because it can hold
the boot gate.
Boot recovery dropped the checkpoint of an application that had left the
creating state without first clearing its staging marker. That left a
claim on the stack name with nothing to retry it, and every later create
for that name was refused by a marker nothing could remove. The settled
branches now share one exit, so the marker ordering holds for all of them.
A detached application's history was readable on a stack grant because
its files are still the stack standing at that name. Nothing in these
tables can prove it still is: a Blueprint successor records the name off
the application row, and a plain Compose stack recreated at the name
leaves no trace at all, so no detection-based allowance can be made
sound. Detach now moves an application's trail to the audit audience
outright, the same answer deleted and creating predecessors get, while
reading the stack itself stays where it was.
* fix(gitops): close four audit findings in revision state and identity proxy
A1: derive runtime and health against target desired_generation_id
- deriveRuntime returns 'applied_not_deployed' when desired != deployed, so a
stale deployment stays deploy-pending instead of reading synced_and_healthy
- deriveHealth judges against desired_generation_id falling back to deployed
when null; control case proves null-desired rows unchanged
- collectRuntimeDrift emits the plan-pinned runtime drift item for exact/
qualified observation mismatches; equal/non-comparable observations emit
nothing; ordering pin keeps deploy question first
A2: stop upstream 304 responses from bypassing hub rewrite and reauthorization
- identity proxy answers no-store on every terminal (rewrite, 204, 304,
generated failures); validators (etag, last-modified, cache-control,
expires, vary) no longer forwarded
- conditional request headers (if-none-match, if-modified-since, if-match,
if-unmodified-since) stripped on the identity-hop branch of the shared
proxyReq handler; streaming hop untouched (optimistic-concurrency file writes
depend on If-Match/If-Unmodified-Since)
- supertest integration test drives the real middleware through a loopback
capture server and asserts if-none-match absent while accept survives
S1: migrate legacy query/userinfo URLs via secret-free path
- parseLegacyRepoUrl strips userinfo, query, fragment instead of refusing;
parseHttpsRepoUrl remains the strict gate for user-driven paths
- migrationDirectSourceIdentity uses the tolerant parser; operational
stack_git_sources.repo_url untouched; fingerprint convergence proven
- trusted-manifest-on-legacy-URL test covers the worst real-world instance
S2: emit runtime artifact drift for current evidence
- top-level drift array carries the seven-class runtime item; frontend
comments updated to reflect backend now emits runtime drift
All regression tests added and passing. CI green (7,374 backend + 2,790
frontend tests; only pre-existing Windows EBUSY teardown failure unrelated).
* test(frontend): fix DriftPanel drift item rendering test
Update test assertions to match the new backend drift item format.
The function now renders artifact_set expected
identities as 'artifact <id> · <qualification>' and runtime_artifact
observed identities as the raw identity string.
* fix(gitops): emit a runtime drift item for desired-versus-deployed mismatch
The projection reported applied_not_deployed with no entry in drift, so
the canonical drift list contradicted the runtime facet it travels with.
collectRuntimeDrift now emits the r27 generation-mismatch item for that
state: desired generation as expected, deployed generation as observed,
ComposeService as owner. The action mirrors what availableActions offers:
deploy, unless an application-level fetch or apply is in flight or a
recovery is in progress, which withhold deploying without removing the
fact of the mismatch. The item clears once the desired generation deploys.
The stale-deployment test pins the exact item shape, its stability across
re-derivation, its convergence removal, and the withheld-action case; the
ordering pin keeps the artifact observation suppressed while the deploy
question stands. Frontend fixture aligns its artifact-mismatch example
with the backend's none action.
* fix(gitops): keep generation drift visible across failure and pause states
The desired-versus-deployed drift item was keyed to the applied_not_deployed
runtime status, but paused, failed, recovering, interrupted, and in-flight
statuses all outrank the pointer comparison in the deriver. A failed deploy
of a new generation over a running older one therefore dropped the drift
report exactly while the old workload was still serving.
The item is now judged from the pointers themselves: known and different
means reported, whatever presentation status the target carries. Retired
targets are excluded because nothing can rebind them, so their surviving
pointer divergence would be permanently unresolvable noise. The action
follows what the payload offers: deploy only when no application-level
operation or recovery withholds it and availableActions lists deploying,
none otherwise.
The failed-redeploy regression test pins the exact item shape after a
pre-mutation deploy failure, its stability across re-derivation, continued
artifact suppression for the replaced workload, and convergence removal;
a second pin keeps retirement silent.
* fix(gitops): complete the available-action legality matrix
The action list transcribed only part of the approved rules. An
interrupted apply offered apply without checking that its recorded
generation was still the current candidate; deploy fired for any target
reading applied_not_deployed regardless of target mode, with no retry
for an interrupted Direct deploy and none of the Blueprint interruption
rules; approve_legacy sat in the type union with no producer, leaving a
pending Inline placement review permanently unactionable through the
projection; and drift items inherited whatever deploy recommendation any
sibling target earned.
Actions now follow the matrix. Deploy is decided per target through one
predicate shared by deriveActions and the drift items: a paused or
failed sibling can no longer inherit another target's legal deploy, a
Blueprint-mode divergence never advertises Direct deployment, an
interrupted Direct deploy retries against the generation still applied,
and an interrupted Blueprint deploy or withdraw repeats only while its
recorded intent revision and rollout candidate still equal what the
application requires, where an absent pair matches because inline
Blueprints carry no candidate until the later-phase producer lands.
Interrupted apply requires the recorded generation to still be the
current candidate and the source not to have been suspended meanwhile.
A reachable Inline placement review now offers approve_legacy.
Six tests pin the matrix: sibling isolation across failed and paused
targets, the Blueprint mode guard, vacuous and matched Blueprint retry
identities plus the superseded case, matching-versus-stale Direct deploy
and apply interruptions, and the legacy review action.
* fix(gitops): withhold apply retry while the candidate is blocked
The interrupted-apply retry checked that the recorded generation still
matched the current candidate and that the source was not suspended, but
a later classification can also block that candidate, and applyStarted
refuses a blocked one outright. The retry is offered only when every
precondition the transition enforces still holds.
* fix(gitops): prove apply preconditions and limit fetch to Direct
The interrupted-apply retry checked identity, suspension, and blockage
but never loaded the candidate generation, so apply could be recommended
for a row that was missing, owned by another application, or built from
a superseded materialization fingerprint; applyStarted refuses all
three, and the positive fixture itself described a state no transition
would accept. The gate now proves existence, ownership, and fingerprint
before offering apply. Its fixture is rebuilt around a transition-legal
candidate whose projected action is executed against applyStarted, with
negatives for every refusal including suspension.
Fetch was offered to any application whose source looked unreconciled,
but the approved rules reserve fetch for live Direct applications; a
Git-backed Blueprint divergence now advertises nothing of the sort.
Controls pin both modes against equivalent source state.
* fix(gitops): fail closed when a candidate row is missing or foreign
Ordinary candidate_ready checked a fingerprint only when the candidate
generation row existed, so a dangling id or one owned by another
application fell through to ready and offered an apply that
applyStarted would refuse on sight. The source now requires the row to
exist under this application before readiness, records a limitation
for the anomaly with its operator copy mirrored in the frontend, and
reports reconcile-required instead. The Blueprint retry test also pins
that a superseded rollout candidate alone, with the intent still
matching, suppresses deployment.
* fix(gitops): fail accepted-generation derivation closed
The acceptance branch compared fingerprint and sha only when the
accepted generation row loaded, so a dangling id or one owned by
another application fell through and reported
application_generation_accepted with no evidence behind it, while also
withholding fetch as the recovery. The source now requires the row to
exist under this application before any comparison, records an
accepted_generation_invalid limitation naming the pointer with its
operator copy and inventory entry mirrored on the frontend, and
reports reconcile-required instead. Five scenarios pin valid, missing,
foreign, fingerprint-mismatched, and sha-mismatched acceptances.
* fix(gitops): enforce canonical state in dismiss, reconcile, and pulls
- Route dismiss-pending through the canonical dismissed transition so a
refusal while an operation is in flight surfaces as 409 instead of
clearing staged state behind the model's back
- Treat tombstoned targets as authoritative in both reconciler decision
surfaces so ticks never redeploy onto a placement the model severed;
an explicit deploy revives the target and records the revival delta
- Keep application-level rows in node-scoped history pages so proxied
hub views do not read as if the application never came into being
- Expose repoIdentity and configuredRef on history items to match the
server-side identity filters
- Stand down when a pull resolves to exactly the live candidate (same
commit, source fingerprint, plan verdict); pulls after an acceptance
still open a fresh staging generation as the apply target
* fix(rbac): classify git-source manifest reads as stack:read
GET /stacks/:name/git-source/manifest had no rule in the hub route
classifier, so proxied requests were refused with 403 before reaching
the remote node even for callers holding stack read access.
* fix(gitops): surface an error toast when dismiss is refused as in-flight
Dismissing a pending Git update while a fetch or apply is still running
now returns 409 OPERATION_IN_FLIGHT, but the frontend handler had no
else branch for a non-2xx response, so the refusal was swallowed with
no operator feedback at all. Add the same error-toast pattern already
used by the sibling apply handler, and order the success toast before
its side effects so a downstream failure cannot invert the outcome.
This commit is contained in:
@@ -21,6 +21,8 @@ import {
|
||||
applyClearStaleGuard,
|
||||
buildBlueprintPreview,
|
||||
} from './blueprintPreviewProjection';
|
||||
import { commitBlueprintDeploymentCause } from './gitops/blueprintDeploymentProducers';
|
||||
import { GitOpsStore } from './gitops/store';
|
||||
|
||||
const RECONCILER_INTERVAL_MS = 60_000;
|
||||
const RECONCILER_INITIAL_DELAY_MS = 5_000;
|
||||
@@ -134,6 +136,8 @@ export interface ReconcileDecision {
|
||||
check: Node[];
|
||||
stateReview: Node[];
|
||||
evictBlocked: Node[];
|
||||
/** Nodes whose canonical target is severed; no automatic action may run. */
|
||||
severedNodeIds: number[];
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -324,25 +328,21 @@ export class BlueprintReconciler {
|
||||
switch (action) {
|
||||
case 'await_state_review': {
|
||||
const existing = DatabaseService.getInstance().getDeployment(blueprint.id, node.id);
|
||||
DatabaseService.getInstance().upsertDeployment({
|
||||
blueprint_id: blueprint.id,
|
||||
node_id: node.id,
|
||||
commitBlueprintDeploymentCause('await_state_review', blueprint.id, node.id, {
|
||||
status: 'pending_state_review',
|
||||
last_checked_at: Date.now(),
|
||||
drift_summary: existing
|
||||
? 'Stateful blueprint revision change awaits operator confirmation'
|
||||
: 'Stateful blueprint awaiting operator confirmation before first deploy',
|
||||
});
|
||||
}, null);
|
||||
return { ...base, status: 'ok' };
|
||||
}
|
||||
case 'await_evict_confirm': {
|
||||
DatabaseService.getInstance().upsertDeployment({
|
||||
blueprint_id: blueprint.id,
|
||||
node_id: node.id,
|
||||
commitBlueprintDeploymentCause('await_evict_confirm', blueprint.id, node.id, {
|
||||
status: 'evict_blocked',
|
||||
last_checked_at: Date.now(),
|
||||
drift_summary: 'Stateful blueprint eviction requires operator confirmation',
|
||||
});
|
||||
}, null);
|
||||
return { ...base, status: 'ok' };
|
||||
}
|
||||
case 'clear_reversed_evict':
|
||||
@@ -365,14 +365,12 @@ export class BlueprintReconciler {
|
||||
const driftResult = await svc.checkForDrift(blueprint, node);
|
||||
if (!driftResult.drifted) return { ...base, status: 'ok' };
|
||||
const reason = driftResult.reason ?? 'unknown drift';
|
||||
DatabaseService.getInstance().upsertDeployment({
|
||||
blueprint_id: blueprint.id,
|
||||
node_id: node.id,
|
||||
commitBlueprintDeploymentCause('drift_observed', blueprint.id, node.id, {
|
||||
status: 'drifted',
|
||||
last_checked_at: Date.now(),
|
||||
last_drift_at: Date.now(),
|
||||
drift_summary: reason,
|
||||
});
|
||||
}, null);
|
||||
// observe/suggest/enforce: notify path via handleDrift still respects drift_mode
|
||||
await this.handleDrift(blueprint, node, reason);
|
||||
return { ...base, status: 'ok' };
|
||||
@@ -511,18 +509,34 @@ export class BlueprintReconciler {
|
||||
const deploymentByNode = new Map<number, BlueprintDeployment>();
|
||||
for (const dep of existingDeployments) deploymentByNode.set(dep.node_id, dep);
|
||||
|
||||
// A tombstoned target is a placement the model has severed (withdraw,
|
||||
// node delete). Redeploying onto one would run the workload while the
|
||||
// projection insists the target is gone, so automatic placement skips
|
||||
// it. Only an explicit deploy re-opens the placement, and that revival
|
||||
// is recorded by the transition itself.
|
||||
const gitopsApp = GitOpsStore.getInstance().getLiveBlueprintApplication(blueprint.id);
|
||||
const severedNodes = new Set<number>(
|
||||
gitopsApp
|
||||
? GitOpsStore.getInstance().listTargets(gitopsApp.id)
|
||||
.filter((t) => t.target_status === 'tombstoned')
|
||||
.map((t) => t.node_id)
|
||||
: [],
|
||||
);
|
||||
|
||||
const decision: ReconcileDecision = {
|
||||
deploy: [],
|
||||
withdraw: [],
|
||||
check: [],
|
||||
stateReview: [],
|
||||
evictBlocked: [],
|
||||
severedNodeIds: [...severedNodes],
|
||||
};
|
||||
|
||||
// Desired but not active or stale
|
||||
for (const node of desiredNodes) {
|
||||
const dep = deploymentByNode.get(node.id);
|
||||
if (!dep) {
|
||||
if (severedNodes.has(node.id)) continue;
|
||||
// Cordon filter: skip new placements onto cordoned nodes.
|
||||
// Pin always wins, so the pinned node is exempt. Existing
|
||||
// deployments below are untouched: cordon does not evict.
|
||||
@@ -553,6 +567,7 @@ export class BlueprintReconciler {
|
||||
continue;
|
||||
}
|
||||
if (dep.applied_revision !== blueprint.revision) {
|
||||
if (severedNodes.has(node.id)) continue;
|
||||
if (blueprint.classification === 'stateful' || blueprint.classification === 'unknown') {
|
||||
decision.stateReview.push(node);
|
||||
} else {
|
||||
@@ -561,6 +576,7 @@ export class BlueprintReconciler {
|
||||
continue;
|
||||
}
|
||||
if (dep.status === 'failed' || dep.status === 'pending') {
|
||||
if (severedNodes.has(node.id)) continue;
|
||||
decision.deploy.push(node);
|
||||
continue;
|
||||
}
|
||||
@@ -623,12 +639,10 @@ export class BlueprintReconciler {
|
||||
return;
|
||||
}
|
||||
}
|
||||
DatabaseService.getInstance().upsertDeployment({
|
||||
blueprint_id: blueprint.id,
|
||||
node_id: node.id,
|
||||
commitBlueprintDeploymentCause('drift_enforce_start', blueprint.id, node.id, {
|
||||
status: 'correcting',
|
||||
last_checked_at: Date.now(),
|
||||
});
|
||||
}, null);
|
||||
const result = await BlueprintService.getInstance().deployToNode(blueprint, node);
|
||||
if (result.status !== 'active') {
|
||||
notifications.dispatchAlert(
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -24,6 +24,9 @@ import { authoredComposeFileArgs, authoredComposeEnvFileArgs } from '../utils/au
|
||||
import type { RollbackInvocationRecord } from '../types/rollbackGeneration';
|
||||
import { parseMissingRequiredVars } from '../helpers/envVarParse';
|
||||
import { redactSensitiveText, sanitizeForLog } from '../utils/safeLog';
|
||||
import { randomUUID } from 'crypto';
|
||||
import { GitOpsStore } from './gitops/store';
|
||||
import { GitOpsTransitions } from './gitops/transitions';
|
||||
import { pathsMatch, resolveHostBindPath } from '../utils/composePathMapping';
|
||||
import { loadStackBuildServices } from './ImageUpdateService';
|
||||
import { resolveMissingExternalNetworks } from './network/resolveMissingExternalNetworks';
|
||||
@@ -139,6 +142,16 @@ function getComposeStallTimeoutMs(): number {
|
||||
* In the Distributed API model, remote node compose operations are handled
|
||||
* by the remote Sencho instance. This service only executes commands locally.
|
||||
*/
|
||||
/**
|
||||
* Evidence that a Compose mutation actually ran.
|
||||
*
|
||||
* Returned rather than inferred from a resolved promise because the recovery
|
||||
* path takes its Compose step as a callback: a caller that restores some other
|
||||
* way resolves identically, and binding the deployed pointer on that would
|
||||
* claim a workload nobody launched.
|
||||
*/
|
||||
export type ComposeMutationResult = { mutatedByCompose: true };
|
||||
|
||||
export class ComposeService {
|
||||
private baseDir: string;
|
||||
private nodeId: number;
|
||||
@@ -659,12 +672,70 @@ export class ComposeService {
|
||||
recordCreatedNetworks('info');
|
||||
}
|
||||
|
||||
/**
|
||||
* Open a GitOps deploy operation for this stack, or nothing when there is no
|
||||
* generation to bind.
|
||||
*
|
||||
* Returns closures rather than ids so the caller cannot terminate an
|
||||
* operation it never started. A stack with no live application, or one whose
|
||||
* target has nothing applied, has no deploy identity to record, so the whole
|
||||
* thing is a no-op. Recording never fails the deploy: the store describes
|
||||
* what happened, it does not make it happen.
|
||||
*/
|
||||
private beginGitOpsDeploy(stackName: string): {
|
||||
generationId: string;
|
||||
bound: () => void;
|
||||
failed: (failureClass: 'pre_mutation' | 'post_mutation') => void;
|
||||
} | null {
|
||||
try {
|
||||
const app = GitOpsStore.getInstance().getLiveDirectApplication(stackName);
|
||||
if (!app || app.lifecycle_status !== 'active') return null;
|
||||
const target = GitOpsStore.getInstance().getTarget(app.id, this.nodeId);
|
||||
const generationId = target?.applied_generation_id;
|
||||
if (!target || target.target_status !== 'active' || !generationId) return null;
|
||||
|
||||
const tx = GitOpsTransitions.getInstance();
|
||||
const envelope = { operationId: randomUUID(), actor: 'system:compose', trigger: 'deploy', at: Date.now() };
|
||||
const record = (what: string, write: () => void): boolean => {
|
||||
try {
|
||||
write();
|
||||
return true;
|
||||
} catch (error) {
|
||||
console.error(
|
||||
'[GitOps] Could not record deploy %s for %s (application %s, generation %s):',
|
||||
what,
|
||||
sanitizeForLog(stackName),
|
||||
app.id,
|
||||
generationId,
|
||||
error instanceof Error ? error.stack ?? error.message : String(error),
|
||||
);
|
||||
return false;
|
||||
}
|
||||
};
|
||||
if (!record('start', () => tx.deployStarted(app.id, this.nodeId, generationId, envelope))) {
|
||||
return null;
|
||||
}
|
||||
return {
|
||||
generationId,
|
||||
bound: () => record('binding', () => tx.deployBound(app.id, this.nodeId, generationId, envelope)),
|
||||
failed: (failureClass) => record('failure', () => tx.deployFailed(app.id, this.nodeId, failureClass, envelope)),
|
||||
};
|
||||
} catch (error) {
|
||||
console.error(
|
||||
'[GitOps] Could not open a deploy operation for %s:',
|
||||
sanitizeForLog(stackName),
|
||||
error instanceof Error ? error.message : String(error),
|
||||
);
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
async deployStack(
|
||||
stackName: string,
|
||||
ws?: WebSocket,
|
||||
atomic?: boolean,
|
||||
ctx?: DeployInvocationContext,
|
||||
): Promise<{ recoveryId: string | null }> {
|
||||
): Promise<{ recoveryId: string | null; deployedGenerationId: string | null }> {
|
||||
await this.assertRequiredEnvPresent(stackName);
|
||||
await this.assertSafePilotBindMapping(stackName);
|
||||
await this.ensureExternalNetworksForDeploy(stackName, ctx);
|
||||
@@ -705,6 +776,12 @@ export class ComposeService {
|
||||
}
|
||||
}
|
||||
|
||||
// ComposeService is the only producer of deploy events: every deploy path
|
||||
// (manual, bulk, Git auto-deploy, App Store, scheduler, webhook) funnels
|
||||
// through here, so recording it anywhere else would double-count.
|
||||
const gitopsDeploy = this.beginGitOpsDeploy(stackName);
|
||||
let composeHandedOff = false;
|
||||
|
||||
try {
|
||||
try {
|
||||
const dockerController = DockerController.getInstance(this.nodeId);
|
||||
@@ -718,7 +795,9 @@ export class ComposeService {
|
||||
}
|
||||
|
||||
await this.withRegistryAuth(async (env) => {
|
||||
await this.execute('docker', await this.authoredComposeArgs(stackName, ['up', '-d', '--remove-orphans']), stackDir, ws, true, env, getComposeStallTimeoutMs());
|
||||
const args = await this.authoredComposeArgs(stackName, ['up', '-d', '--remove-orphans']);
|
||||
composeHandedOff = true;
|
||||
await this.execute('docker', args, stackDir, ws, true, env, getComposeStallTimeoutMs());
|
||||
}, sendOutput);
|
||||
|
||||
// Post-Deploy Health Probe
|
||||
@@ -751,7 +830,11 @@ export class ComposeService {
|
||||
}
|
||||
}
|
||||
if (debug) console.debug(`[ComposeService:debug] deployStack completed in ${Date.now() - t0}ms`, { stackName });
|
||||
gitopsDeploy?.bound();
|
||||
} catch (deployError) {
|
||||
// Classified by whether Compose was handed the mutation. Only a failure
|
||||
// before that leaves the previous workload provably intact.
|
||||
gitopsDeploy?.failed(composeHandedOff ? 'post_mutation' : 'pre_mutation');
|
||||
if (atomic && recoverySvc && handedOff && recoveryId) {
|
||||
sendOutput('\n=== Deployment failed - restoring previous runtime from recovery generation ===\n');
|
||||
const generationId = recoveryId;
|
||||
@@ -793,7 +876,7 @@ export class ComposeService {
|
||||
console.warn('[ComposeService] Exposure refresh failed after deploy for %s:',
|
||||
sanitizeForLog(stackName), sanitizeForLog(getErrorMessage(err, 'unknown')));
|
||||
}
|
||||
return { recoveryId };
|
||||
return { recoveryId, deployedGenerationId: gitopsDeploy?.generationId ?? null };
|
||||
}
|
||||
|
||||
streamLogs(stackName: string, ws: WebSocket) {
|
||||
@@ -942,7 +1025,7 @@ export class ComposeService {
|
||||
overridePath: string,
|
||||
ws?: WebSocket,
|
||||
invocation?: RollbackInvocationRecord | null,
|
||||
): Promise<void> {
|
||||
): Promise<ComposeMutationResult> {
|
||||
const stackDir = path.join(this.baseDir, stackName);
|
||||
const sendOutput = (data: string) => {
|
||||
if (ws && ws.readyState === WebSocket.OPEN) ws.send(data);
|
||||
@@ -963,6 +1046,7 @@ export class ComposeService {
|
||||
getComposeStallTimeoutMs(),
|
||||
);
|
||||
}, sendOutput);
|
||||
return { mutatedByCompose: true };
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -1121,7 +1205,7 @@ export class ComposeService {
|
||||
stackName: string,
|
||||
ws?: WebSocket,
|
||||
atomic?: boolean,
|
||||
): Promise<{ recoveryId: string | null }> {
|
||||
): Promise<{ recoveryId: string | null; deployedGenerationId: string | null }> {
|
||||
await this.assertRequiredEnvPresent(stackName);
|
||||
await this.assertSafePilotBindMapping(stackName);
|
||||
const stackDir = path.join(this.baseDir, stackName);
|
||||
@@ -1132,6 +1216,12 @@ export class ComposeService {
|
||||
if (ws && ws.readyState === WebSocket.OPEN) ws.send(data);
|
||||
};
|
||||
|
||||
// Opened once the update is committed to recreating containers, not at the
|
||||
// top: an update that fails during capture or classification never reached
|
||||
// Compose, so there is no deploy to record.
|
||||
let gitopsDeploy: ReturnType<ComposeService['beginGitOpsDeploy']> = null;
|
||||
let composeHandedOff = false;
|
||||
|
||||
// Dynamic import avoids a static cycle (recovery imports getComposeCommandTimeoutMs).
|
||||
const { StackUpdateRecoveryService } = await import('./StackUpdateRecoveryService');
|
||||
const recoverySvc = StackUpdateRecoveryService.getInstance();
|
||||
@@ -1218,13 +1308,15 @@ export class ComposeService {
|
||||
}
|
||||
}
|
||||
|
||||
gitopsDeploy = this.beginGitOpsDeploy(stackName);
|
||||
await this.withRegistryAuth(async (env) => {
|
||||
sendOutput('=== Recreating containers ===\n');
|
||||
await this.execute(
|
||||
'docker',
|
||||
await this.authoredComposeArgs(stackName, ['up', '-d', '--remove-orphans']),
|
||||
stackDir, ws, true, env, getComposeStallTimeoutMs(),
|
||||
);
|
||||
const args = await this.authoredComposeArgs(stackName, ['up', '-d', '--remove-orphans']);
|
||||
// Set only once Compose is genuinely about to receive the mutation:
|
||||
// reading compose args or resolving registry auth can still fail with
|
||||
// the previous workload provably intact.
|
||||
composeHandedOff = true;
|
||||
await this.execute('docker', args, stackDir, ws, true, env, getComposeStallTimeoutMs());
|
||||
}, sendOutput);
|
||||
|
||||
// Immediate verification probe
|
||||
@@ -1279,7 +1371,9 @@ export class ComposeService {
|
||||
if (debug) {
|
||||
console.debug(`[ComposeService:debug] updateStack completed in ${Date.now() - t0}ms`, { stackName });
|
||||
}
|
||||
gitopsDeploy?.bound();
|
||||
} catch (updateError) {
|
||||
gitopsDeploy?.failed(composeHandedOff ? 'post_mutation' : 'pre_mutation');
|
||||
if (!handedOff && recoveryId) {
|
||||
await recoverySvc.abandon(recoveryId);
|
||||
recoveryId = null;
|
||||
@@ -1315,7 +1409,7 @@ export class ComposeService {
|
||||
sanitizeForLog(getErrorMessage(err, 'unknown')),
|
||||
);
|
||||
}
|
||||
return { recoveryId };
|
||||
return { recoveryId, deployedGenerationId: gitopsDeploy?.generationId ?? null };
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
@@ -20,6 +20,7 @@ import { sanitizeForLog } from '../utils/safeLog';
|
||||
import type { GitSourceManifestState } from '../types/gitProjectManifest';
|
||||
import type { RollbackOperationKind } from '../types/rollbackGeneration';
|
||||
import { collectImageIds, parseServicesJsonStrict } from './recoveryServicesJson';
|
||||
import { GITOPS_SCHEMA_SQL } from './gitops/schema';
|
||||
|
||||
export type { SnapshotFileReadResult } from '../helpers/snapshotFileDecrypt';
|
||||
export type { RollbackOperationKind } from '../types/rollbackGeneration';
|
||||
@@ -220,13 +221,13 @@ export interface StackExposureRow {
|
||||
computed_at: number;
|
||||
}
|
||||
|
||||
/** One post-update health gate observation run. */
|
||||
/** One health-gate observation run, keyed by `trigger_action`. */
|
||||
export interface HealthGateRunRow {
|
||||
id: string;
|
||||
node_id: number;
|
||||
stack_name: string;
|
||||
/** Named trigger_action because TRIGGER is reserved in SQLite. */
|
||||
trigger_action: 'update' | 'deploy' | 'service_update' | 'service_restore';
|
||||
trigger_action: 'update' | 'deploy' | 'service_update' | 'service_restore' | 'recovery';
|
||||
status: 'observing' | 'passed' | 'failed' | 'unknown';
|
||||
reason: string | null;
|
||||
window_seconds: number;
|
||||
@@ -239,6 +240,11 @@ export interface HealthGateRunRow {
|
||||
target_scope: 'stack' | 'service';
|
||||
service_name: string | null;
|
||||
failure_source: 'primary' | 'collateral' | null;
|
||||
/**
|
||||
* Reserved for the GitOps deploy path: the generation live when this run
|
||||
* started. No writer populates it yet, so it is currently always null.
|
||||
*/
|
||||
deployed_generation_id?: string | null;
|
||||
}
|
||||
|
||||
/** Pre-update image snapshot enabling a manual per-service restore after a service-scoped update. */
|
||||
@@ -269,6 +275,9 @@ export interface StackUpdateRecoveryGenerationRow {
|
||||
/** Set when an operator manually released rollback protection early (see releaseStackUpdateRecoveryGeneration). */
|
||||
released_at: number | null;
|
||||
released_by: string | null;
|
||||
gitops_generation_id?: string | null;
|
||||
gitops_artifact_set_id?: string | null;
|
||||
gitops_source_acceptance_ref?: string | null;
|
||||
}
|
||||
|
||||
/** Durable cleanup tombstone for stack/node deletion artifact sweep. */
|
||||
@@ -1161,6 +1170,7 @@ export class DatabaseService {
|
||||
this.migrateGitSourceMultiFile();
|
||||
this.migrateGitSourceManifest();
|
||||
this.migrateGitSourceChangePlan();
|
||||
this.migrateGitOpsRecoveryColumns();
|
||||
this.migrateNodeUpdateSkips();
|
||||
this.migrateStackAlertServiceScope();
|
||||
|
||||
@@ -1777,7 +1787,7 @@ export class DatabaseService {
|
||||
id TEXT PRIMARY KEY,
|
||||
node_id INTEGER NOT NULL,
|
||||
stack_name TEXT NOT NULL,
|
||||
trigger_action TEXT NOT NULL CHECK (trigger_action IN ('update','deploy','service_update','service_restore')),
|
||||
trigger_action TEXT NOT NULL CHECK (trigger_action IN ('update','deploy','service_update','service_restore','recovery')),
|
||||
status TEXT NOT NULL CHECK (status IN ('observing','passed','failed','unknown')),
|
||||
reason TEXT,
|
||||
window_seconds INTEGER NOT NULL,
|
||||
@@ -1906,6 +1916,8 @@ export class DatabaseService {
|
||||
CREATE INDEX IF NOT EXISTS idx_secret_pushes_node ON secret_pushes(node_id, stack_name);
|
||||
`);
|
||||
|
||||
this.db.exec(GITOPS_SCHEMA_SQL);
|
||||
|
||||
// Apply migrations safely (ignore if columns already exist)
|
||||
const maybeAddCol = (table: string, col: string, def: string) => {
|
||||
try { this.db.prepare(`ALTER TABLE ${table} ADD COLUMN ${col} ${def}`).run(); } catch (e) { /* ignore */ }
|
||||
@@ -2098,6 +2110,7 @@ export class DatabaseService {
|
||||
// behave); admins who want a strict absolute session ceiling can turn
|
||||
// it off in Settings > Users.
|
||||
stmt.run('session_sliding_refresh', '1');
|
||||
stmt.run('gitops_schema_version', '1');
|
||||
|
||||
// Seed the default local node if none exists
|
||||
const nodeCount = (this.db.prepare('SELECT COUNT(*) as count FROM nodes').get() as any)?.count || 0;
|
||||
@@ -2177,10 +2190,11 @@ export class DatabaseService {
|
||||
}
|
||||
|
||||
/**
|
||||
* Rebuild health_gate_runs when the installed CHECK still only allows
|
||||
* update|deploy or target/failure columns are missing. Idempotent and
|
||||
* restart-safe: drops a stale temporary table, then rebuilds in one
|
||||
* better-sqlite3 transaction so an interrupted startup cannot leave
|
||||
* Rebuild health_gate_runs when the CHECK lacks service_update,
|
||||
* service_restore, or recovery, or when target/failure columns are missing.
|
||||
* After the CHECK is current, ensure deployed_generation_id exists.
|
||||
* Idempotent and restart-safe: drops a stale temporary table, then rebuilds
|
||||
* in one better-sqlite3 transaction so an interrupted startup cannot leave
|
||||
* CREATE TABLE health_gate_runs_new blocking the next boot.
|
||||
*/
|
||||
private migrateHealthGateTargetSchema(): void {
|
||||
@@ -2192,7 +2206,11 @@ export class DatabaseService {
|
||||
const hasTarget = colNames.has('target_scope');
|
||||
const hasFailure = colNames.has('failure_source');
|
||||
const hasWideTrigger = tableSql.includes('service_update') && tableSql.includes('service_restore');
|
||||
if (hasTarget && hasFailure && hasWideTrigger) return;
|
||||
const hasRecoveryTrigger = tableSql.includes("'recovery'");
|
||||
if (hasTarget && hasFailure && hasWideTrigger && hasRecoveryTrigger) {
|
||||
this.ensureHealthGateDeployedGenerationColumn();
|
||||
return;
|
||||
}
|
||||
|
||||
// A previous crash between CREATE and RENAME leaves this temp table behind.
|
||||
this.db.exec('DROP TABLE IF EXISTS health_gate_runs_new');
|
||||
@@ -2200,6 +2218,7 @@ export class DatabaseService {
|
||||
const targetExpr = hasTarget ? 'target_scope' : "'stack'";
|
||||
const serviceExpr = colNames.has('service_name') ? 'service_name' : 'NULL';
|
||||
const failureExpr = hasFailure ? 'failure_source' : 'NULL';
|
||||
const deployedExpr = colNames.has('deployed_generation_id') ? 'deployed_generation_id' : 'NULL';
|
||||
|
||||
this.db.transaction(() => {
|
||||
this.db.exec(`
|
||||
@@ -2207,7 +2226,7 @@ export class DatabaseService {
|
||||
id TEXT PRIMARY KEY,
|
||||
node_id INTEGER NOT NULL,
|
||||
stack_name TEXT NOT NULL,
|
||||
trigger_action TEXT NOT NULL CHECK (trigger_action IN ('update','deploy','service_update','service_restore')),
|
||||
trigger_action TEXT NOT NULL CHECK (trigger_action IN ('update','deploy','service_update','service_restore','recovery')),
|
||||
status TEXT NOT NULL CHECK (status IN ('observing','passed','failed','unknown')),
|
||||
reason TEXT,
|
||||
window_seconds INTEGER NOT NULL,
|
||||
@@ -2217,23 +2236,36 @@ export class DatabaseService {
|
||||
created_by TEXT,
|
||||
target_scope TEXT NOT NULL DEFAULT 'stack' CHECK (target_scope IN ('stack','service')),
|
||||
service_name TEXT,
|
||||
failure_source TEXT CHECK (failure_source IS NULL OR failure_source IN ('primary','collateral'))
|
||||
failure_source TEXT CHECK (failure_source IS NULL OR failure_source IN ('primary','collateral')),
|
||||
deployed_generation_id TEXT NULL
|
||||
);
|
||||
INSERT INTO health_gate_runs_new (
|
||||
id, node_id, stack_name, trigger_action, status, reason, window_seconds,
|
||||
containers_json, started_at, ended_at, created_by, target_scope, service_name, failure_source
|
||||
containers_json, started_at, ended_at, created_by, target_scope, service_name,
|
||||
failure_source, deployed_generation_id
|
||||
)
|
||||
SELECT
|
||||
id, node_id, stack_name, trigger_action, status, reason, window_seconds,
|
||||
containers_json, started_at, ended_at, created_by,
|
||||
${targetExpr}, ${serviceExpr}, ${failureExpr}
|
||||
${targetExpr}, ${serviceExpr}, ${failureExpr}, ${deployedExpr}
|
||||
FROM health_gate_runs;
|
||||
DROP TABLE health_gate_runs;
|
||||
ALTER TABLE health_gate_runs_new RENAME TO health_gate_runs;
|
||||
CREATE INDEX IF NOT EXISTS idx_health_gate_runs_node_stack
|
||||
ON health_gate_runs(node_id, stack_name, started_at);
|
||||
CREATE INDEX IF NOT EXISTS idx_health_gate_runs_deployed_gen
|
||||
ON health_gate_runs(node_id, stack_name, deployed_generation_id);
|
||||
`);
|
||||
})();
|
||||
this.ensureHealthGateDeployedGenerationColumn();
|
||||
}
|
||||
|
||||
private ensureHealthGateDeployedGenerationColumn(): void {
|
||||
this.tryAddColumn('health_gate_runs', 'deployed_generation_id', 'TEXT');
|
||||
this.db.exec(`
|
||||
CREATE INDEX IF NOT EXISTS idx_health_gate_runs_deployed_gen
|
||||
ON health_gate_runs(node_id, stack_name, deployed_generation_id);
|
||||
`);
|
||||
}
|
||||
|
||||
private migrateEncryptNodeTokens(): void {
|
||||
@@ -2550,6 +2582,12 @@ export class DatabaseService {
|
||||
this.tryAddColumn('stack_git_sources', 'last_plan_outcome', 'TEXT');
|
||||
}
|
||||
|
||||
private migrateGitOpsRecoveryColumns(): void {
|
||||
this.tryAddColumn('stack_update_recovery_generations', 'gitops_generation_id', 'TEXT');
|
||||
this.tryAddColumn('stack_update_recovery_generations', 'gitops_artifact_set_id', 'TEXT');
|
||||
this.tryAddColumn('stack_update_recovery_generations', 'gitops_source_acceptance_ref', 'TEXT');
|
||||
}
|
||||
|
||||
private migrateGitSourceMultiFile(): void {
|
||||
this.tryAddColumn('stack_git_sources', 'compose_paths', 'TEXT');
|
||||
this.tryAddColumn('stack_git_sources', 'context_dir', 'TEXT');
|
||||
@@ -4054,12 +4092,14 @@ export class DatabaseService {
|
||||
this.db.prepare(
|
||||
`INSERT INTO health_gate_runs
|
||||
(id, node_id, stack_name, trigger_action, status, reason, window_seconds, containers_json,
|
||||
started_at, ended_at, created_by, target_scope, service_name, failure_source)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`
|
||||
started_at, ended_at, created_by, target_scope, service_name, failure_source,
|
||||
deployed_generation_id)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`
|
||||
).run(
|
||||
run.id, run.node_id, run.stack_name, run.trigger_action, run.status, run.reason,
|
||||
run.window_seconds, run.containers_json, run.started_at, run.ended_at, run.created_by,
|
||||
run.target_scope, run.service_name, run.failure_source,
|
||||
run.deployed_generation_id ?? null,
|
||||
);
|
||||
// Bounded history: keep only the 10 most recent runs per stack.
|
||||
this.db.prepare(
|
||||
@@ -4099,11 +4139,17 @@ export class DatabaseService {
|
||||
}
|
||||
|
||||
/** Finalize runs left observing by a previous process (startup sweep). */
|
||||
public markInterruptedHealthGateRuns(reason: string, endedAt: number): number {
|
||||
const result = this.db.prepare(
|
||||
"UPDATE health_gate_runs SET status = 'unknown', reason = ?, ended_at = ? WHERE status = 'observing'"
|
||||
).run(reason, endedAt);
|
||||
return result.changes;
|
||||
/**
|
||||
* Runs a previous process left observing.
|
||||
*
|
||||
* Returned as rows rather than swept with one UPDATE because each has to be
|
||||
* finalized individually: the verdict is what the revision state listens
|
||||
* for, and a bulk update moves the rows while telling the model nothing.
|
||||
*/
|
||||
public listObservingHealthGateRuns(): HealthGateRunRow[] {
|
||||
return this.db.prepare(
|
||||
"SELECT * FROM health_gate_runs WHERE status = 'observing'"
|
||||
).all() as HealthGateRunRow[];
|
||||
}
|
||||
|
||||
// --- Service Update Recovery ---
|
||||
@@ -4234,14 +4280,17 @@ export class DatabaseService {
|
||||
id, node_id, stack_name, status, phase, is_current, backup_slot_id, content_path,
|
||||
operation_kind, override_path, services_json, health_gate_id, gate_retain_until,
|
||||
artifact_expires_at, operation_lease_expires_at, created_at, updated_at,
|
||||
created_by, artifacts_retired
|
||||
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`
|
||||
created_by, artifacts_retired, gitops_generation_id, gitops_artifact_set_id,
|
||||
gitops_source_acceptance_ref
|
||||
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`
|
||||
).run(
|
||||
row.id, row.node_id, row.stack_name, row.status, row.phase, row.is_current,
|
||||
row.backup_slot_id, row.content_path ?? null, row.operation_kind ?? null,
|
||||
row.override_path, row.services_json, row.health_gate_id,
|
||||
row.gate_retain_until, row.artifact_expires_at, row.operation_lease_expires_at,
|
||||
row.created_at, row.updated_at, row.created_by, row.artifacts_retired ?? 0,
|
||||
row.gitops_generation_id ?? null, row.gitops_artifact_set_id ?? null,
|
||||
row.gitops_source_acceptance_ref ?? null,
|
||||
);
|
||||
}
|
||||
|
||||
|
||||
@@ -28,6 +28,8 @@ import {
|
||||
BLUEPRINT_MARKER_FILENAME,
|
||||
parseBlueprintMarker,
|
||||
} from '../helpers/blueprintMarker';
|
||||
import { GitOpsStore } from './gitops/store';
|
||||
import { GitOpsTransitions } from './gitops/transitions';
|
||||
import { scrapeRollbackTagsLenient } from './recoveryServicesJson';
|
||||
|
||||
/**
|
||||
@@ -329,6 +331,54 @@ export class DeployedStackDeletionService {
|
||||
}
|
||||
|
||||
/** Ready transaction, secondary DB/RBAC cleanup, mesh opt-out, sweep, invalidate. */
|
||||
/**
|
||||
* Commit the deletion and retire the stack's GitOps application together.
|
||||
*
|
||||
* One transaction, because a deleted stack with a live application would keep
|
||||
* claiming a stack name that no longer exists, and would block re-creating it
|
||||
* through the unique live-application index. The tombstone is driven from
|
||||
* here rather than from inside DatabaseService so the store keeps its
|
||||
* transitions, and its history, in one place.
|
||||
*/
|
||||
private commitDeletionReady(intentId: string, nodeId: number, stackName: string): boolean {
|
||||
const db = DatabaseService.getInstance();
|
||||
return db.getDb().transaction(() => {
|
||||
if (!db.commitStackDeletionReadyTransaction(intentId, nodeId, stackName)) return false;
|
||||
const app = GitOpsStore.getInstance().getLiveDirectApplication(stackName);
|
||||
if (!app) return true;
|
||||
const tx = GitOpsTransitions.getInstance();
|
||||
const envelope = {
|
||||
operationId: intentId,
|
||||
actor: 'system:stack-deletion',
|
||||
trigger: 'delete',
|
||||
at: Date.now(),
|
||||
};
|
||||
// The files are already gone by the time this runs, and the startup
|
||||
// reconciler loops over every prepared intent. A rejected tombstone must
|
||||
// fail this one deletion, not throw an opaque driver error out of a
|
||||
// deletion that already succeeded on disk, and not abandon the intents
|
||||
// that follow it.
|
||||
try {
|
||||
for (const target of GitOpsStore.getInstance().listTargets(app.id)) {
|
||||
if (target.target_status !== 'active') continue;
|
||||
tx.targetTombstoned(app.id, target.node_id, envelope);
|
||||
}
|
||||
tx.applicationTombstoned(app.id, 'deleted', envelope);
|
||||
} catch (error) {
|
||||
console.error(
|
||||
'[GitOps] Could not retire the application for deleted stack %s (application %s):',
|
||||
sanitizeForLog(stackName), app.id,
|
||||
error instanceof Error ? error.stack ?? error.message : String(error),
|
||||
);
|
||||
return false;
|
||||
}
|
||||
// A create that never settled leaves a checkpoint whose application is
|
||||
// now gone; drop it so boot recovery does not retry it for ever.
|
||||
GitOpsStore.getInstance().deleteCreateCheckpoint(app.id);
|
||||
return true;
|
||||
})();
|
||||
}
|
||||
|
||||
private async finalizeLogicalDeletion(
|
||||
input: DeleteDeployedStackInput,
|
||||
intentId: string,
|
||||
@@ -336,7 +386,7 @@ export class DeployedStackDeletionService {
|
||||
const { nodeId, stackName } = input;
|
||||
const db = DatabaseService.getInstance();
|
||||
|
||||
if (!db.commitStackDeletionReadyTransaction(intentId, nodeId, stackName)) {
|
||||
if (!this.commitDeletionReady(intentId, nodeId, stackName)) {
|
||||
return {
|
||||
ok: false,
|
||||
code: 'db_failed',
|
||||
@@ -524,17 +574,67 @@ export class DeployedStackDeletionService {
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* Remove a node and retire the GitOps targets that lived on it, together.
|
||||
*
|
||||
* The tombstones have to be written while the target rows still exist, and in
|
||||
* the same transaction as the delete, or a failure part-way through would
|
||||
* leave targets pointing at a node that is gone. Applications are left live:
|
||||
* a Direct application still describes a real stack, and a Blueprint one may
|
||||
* have targets on other nodes.
|
||||
*
|
||||
* Returns the Blueprints that lost a target here, so the caller can report
|
||||
* what the deletion moved. Read inside the transaction and before the
|
||||
* tombstone, because afterwards no active target row remains to trace back to
|
||||
* an application. Direct applications contribute nothing: they carry no
|
||||
* `blueprint_id`.
|
||||
*/
|
||||
public deleteNodeWithGitOps(
|
||||
nodeId: number,
|
||||
localCleanup?: { tombstoneId: string; tags: string[]; overridePaths: string[] },
|
||||
): number[] {
|
||||
const db = DatabaseService.getInstance();
|
||||
return db.getDb().transaction(() => {
|
||||
const store = GitOpsStore.getInstance();
|
||||
const blueprintIds: number[] = [];
|
||||
for (const target of store.listActiveTargetsForNode(nodeId)) {
|
||||
const application = store.getApplication(target.application_id);
|
||||
if (!application) {
|
||||
// No foreign key backs this column and the database runs without
|
||||
// cascade, so an orphaned target is possible, and this is the only
|
||||
// path that would ever look at one. Collapsing it into the Direct
|
||||
// case below would make a referential fault read as the normal
|
||||
// outcome. The tombstone still retires the row either way.
|
||||
console.error(
|
||||
'[DeployedStackDeletion] Orphaned GitOps target on node %s: application %s is missing.',
|
||||
sanitizeForLog(nodeId),
|
||||
sanitizeForLog(target.application_id),
|
||||
);
|
||||
continue;
|
||||
}
|
||||
if (application.blueprint_id !== null) blueprintIds.push(application.blueprint_id);
|
||||
}
|
||||
GitOpsTransitions.getInstance().tombstoneNodeTargets(nodeId, {
|
||||
operationId: localCleanup?.tombstoneId ?? randomUUID(),
|
||||
actor: 'system:node-deletion',
|
||||
trigger: 'node_delete',
|
||||
at: Date.now(),
|
||||
});
|
||||
db.deleteNode(nodeId, localCleanup);
|
||||
return blueprintIds;
|
||||
})();
|
||||
}
|
||||
|
||||
/**
|
||||
* Delete a local-socket node with an atomic ready tombstone, then sweep.
|
||||
* Remote node records call DatabaseService.deleteNode without cleanup.
|
||||
*/
|
||||
public async deleteLocalNode(nodeId: number): Promise<void> {
|
||||
public async deleteLocalNode(nodeId: number): Promise<number[]> {
|
||||
const db = DatabaseService.getInstance();
|
||||
const node = db.getNode(nodeId);
|
||||
if (!node) throw new Error('Node not found');
|
||||
if (node.type !== 'local') {
|
||||
db.deleteNode(nodeId);
|
||||
return;
|
||||
return this.deleteNodeWithGitOps(nodeId);
|
||||
}
|
||||
// Preserve Docker + compose dir before the row disappears so sweep never
|
||||
// targets a remote default node.
|
||||
@@ -542,7 +642,7 @@ export class DeployedStackDeletionService {
|
||||
const composeDir = FileSystemService.getInstance(nodeId).getBaseDir();
|
||||
const { tags, overridePaths } = this.collectNodeArtifacts(nodeId);
|
||||
const tombstoneId = randomUUID();
|
||||
db.deleteNode(nodeId, { tombstoneId, tags, overridePaths });
|
||||
const blueprintIds = this.deleteNodeWithGitOps(nodeId, { tombstoneId, tags, overridePaths });
|
||||
NodeRegistry.getInstance().evictConnection(nodeId);
|
||||
try {
|
||||
await this.sweepReadyIntent(tombstoneId, { docker, composeDir });
|
||||
@@ -554,6 +654,7 @@ export class DeployedStackDeletionService {
|
||||
sanitizeForLog(getErrorMessage(error, 'unknown')),
|
||||
);
|
||||
}
|
||||
return blueprintIds;
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -585,7 +686,7 @@ export class DeployedStackDeletionService {
|
||||
}
|
||||
|
||||
if (!dirExists) {
|
||||
if (!db.commitStackDeletionReadyTransaction(intent.id, nodeId, stackName)) {
|
||||
if (!this.commitDeletionReady(intent.id, nodeId, stackName)) {
|
||||
console.warn(
|
||||
'[DeployedStackDeletion] Startup ready commit failed for %s/%s',
|
||||
nodeId,
|
||||
|
||||
@@ -0,0 +1,70 @@
|
||||
/**
|
||||
* Counters for GitOps revision transitions, one increment per history row that
|
||||
* was actually inserted.
|
||||
*
|
||||
* Process-local and in-memory, the same bargain StackOpMetricsService makes: a
|
||||
* restart clears the counters, and persisting them would put a write on every
|
||||
* transition for very little operator value. The durable record is
|
||||
* `gitops_history` itself, which these counters only summarise.
|
||||
*
|
||||
* The keyspace is finite by construction. `GitOpsHistoryStage` is a closed
|
||||
* union of everything a producer can write and `HistoryOutcome` is a closed
|
||||
* CHECK set of six, so the map cannot exceed their product however much traffic
|
||||
* arrives. No identity, stack name, node, actor, or repository is recorded:
|
||||
* a counter that carried those would be an audit trail with no retention rules
|
||||
* and no authorization, which is what the history API is for.
|
||||
*/
|
||||
import type { GitOpsHistoryStage, HistoryOutcome } from './gitops/history';
|
||||
|
||||
export interface GitOpsMetricEntry {
|
||||
stage: GitOpsHistoryStage;
|
||||
outcome: HistoryOutcome;
|
||||
count: number;
|
||||
}
|
||||
|
||||
export class GitOpsMetricsService {
|
||||
private static instance: GitOpsMetricsService;
|
||||
private readonly buckets = new Map<string, GitOpsMetricEntry>();
|
||||
|
||||
public static getInstance(): GitOpsMetricsService {
|
||||
if (!GitOpsMetricsService.instance) {
|
||||
GitOpsMetricsService.instance = new GitOpsMetricsService();
|
||||
}
|
||||
return GitOpsMetricsService.instance;
|
||||
}
|
||||
|
||||
public static resetForTests(): void {
|
||||
this.instance = new GitOpsMetricsService();
|
||||
}
|
||||
|
||||
/**
|
||||
* Count one transition.
|
||||
*
|
||||
* Called once per newly inserted history row, never on a dedupe replay: a
|
||||
* replay is the same transition arriving twice, and counting it would report
|
||||
* retries as activity.
|
||||
*/
|
||||
public record(stage: GitOpsHistoryStage, outcome: HistoryOutcome): void {
|
||||
const key = `${stage}:${outcome}`;
|
||||
const bucket = this.buckets.get(key);
|
||||
if (bucket) {
|
||||
bucket.count += 1;
|
||||
return;
|
||||
}
|
||||
this.buckets.set(key, { stage, outcome, count: 1 });
|
||||
}
|
||||
|
||||
/**
|
||||
* Every bucket that has been touched, ordered by stage then outcome.
|
||||
*
|
||||
* Untouched pairs are absent rather than zero. Ordering is stable so an
|
||||
* operator pulling this twice can diff the two responses directly. Each
|
||||
* bucket is copied, so a reader cannot edit the counters through the
|
||||
* snapshot it was handed.
|
||||
*/
|
||||
public snapshot(): GitOpsMetricEntry[] {
|
||||
return [...this.buckets.values()]
|
||||
.map((bucket) => ({ ...bucket }))
|
||||
.sort((a, b) => a.stage.localeCompare(b.stage) || a.outcome.localeCompare(b.outcome));
|
||||
}
|
||||
}
|
||||
@@ -27,6 +27,7 @@ import { authoredComposeFileArgs, authoredComposeEnvFileArgs } from '../utils/au
|
||||
import { collectManifestFilePaths } from '../helpers/manifestFilePaths';
|
||||
import { sanitizeForLog } from '../utils/safeLog';
|
||||
import { isPathWithinBase, isValidStackName } from '../utils/validation';
|
||||
import { isRealPathAtManagedLocation, managedAreaBase } from './gitops/managedPaths';
|
||||
import type {
|
||||
BuildContextPlan,
|
||||
ComposeInputEntry,
|
||||
@@ -1226,10 +1227,29 @@ export class GitProjectManifestService {
|
||||
// says nothing about recency, and the manifest's previousDir is what a
|
||||
// crash restore reads from.
|
||||
const keep = new Set([keepBase, previousDir ? path.basename(previousDir) : null].filter((v): v is string => v !== null));
|
||||
const areaBase = managedAreaBase();
|
||||
for (const entry of entries) {
|
||||
if (!entry.isDirectory() || !entry.name.startsWith('applied-')) continue;
|
||||
if (keep.has(entry.name)) continue;
|
||||
await fs.promises.rm(path.join(dir, entry.name), { recursive: true, force: true });
|
||||
const abs = path.resolve(dir, entry.name);
|
||||
// Inline containment barrier at the removal sink (see
|
||||
// `managedAreaBase`): the analyzer credits this literal comparison,
|
||||
// not the positional check below it.
|
||||
if (!abs.startsWith(areaBase + path.sep)) {
|
||||
console.warn(`[GitManifest] refusing to prune ${sanitizeForLog(entry.name)} for ${sanitizeForLog(stackName)}: it resolves outside the managed area`);
|
||||
continue;
|
||||
}
|
||||
// Positional barrier at the recursive-delete sink. This runs on
|
||||
// every successful apply, exactly where a link planted over one
|
||||
// generation name would do the most damage, and lexical
|
||||
// containment cannot see it. Refusal skips the entry rather than
|
||||
// failing an apply that already committed; retention keeps
|
||||
// everything recoverable either way.
|
||||
if (!await isRealPathAtManagedLocation(abs)) {
|
||||
console.warn(`[GitManifest] refusing to prune ${sanitizeForLog(entry.name)} for ${sanitizeForLog(stackName)}: it is not at its own location in the managed area`);
|
||||
continue;
|
||||
}
|
||||
await fs.promises.rm(abs, { recursive: true, force: true });
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1344,9 +1364,24 @@ export class GitProjectManifestService {
|
||||
try {
|
||||
const entries = await fs.promises.readdir(dir, { withFileTypes: true });
|
||||
const now = Date.now();
|
||||
const areaBase = managedAreaBase();
|
||||
for (const entry of entries) {
|
||||
if (!entry.isDirectory() || !entry.name.startsWith('candidate-')) continue;
|
||||
const abs = path.join(dir, entry.name);
|
||||
const abs = path.resolve(dir, entry.name);
|
||||
// Inline containment barrier at the removal sink (see
|
||||
// `managedAreaBase`): the analyzer credits this literal
|
||||
// comparison, not the positional check below it.
|
||||
if (!abs.startsWith(areaBase + path.sep)) {
|
||||
console.warn(`[GitManifest] refusing to reap orphan candidate ${sanitizeForLog(entry.name)} for ${sanitizeForLog(stackName)}: it resolves outside the managed area`);
|
||||
continue;
|
||||
}
|
||||
// Same positional barrier as generation pruning: the boot sweep
|
||||
// reaps candidate directories nobody claims, which is precisely
|
||||
// the kind of unattended delete a planted link would steer.
|
||||
if (!await isRealPathAtManagedLocation(abs)) {
|
||||
console.warn(`[GitManifest] refusing to reap orphan candidate ${sanitizeForLog(entry.name)} for ${sanitizeForLog(stackName)}: it is not at its own location in the managed area`);
|
||||
continue;
|
||||
}
|
||||
const complete = await fs.promises
|
||||
.access(path.join(abs, CANDIDATE_COMPLETE_MARKER))
|
||||
.then(() => true)
|
||||
@@ -1475,7 +1510,17 @@ export class GitProjectManifestService {
|
||||
}
|
||||
await fs.promises.rm(markerPath, { force: true });
|
||||
if (!parsed.managedAreaExisted) {
|
||||
await fs.promises.rm(root, { recursive: true, force: true });
|
||||
// Inline containment barrier at the removal sink (see
|
||||
// `managedAreaBase`), then the positional check. Refusal leaves the
|
||||
// area in place for the operator rather than deleting through a
|
||||
// redirected root; the restore itself already succeeded, so
|
||||
// recovery still reports success.
|
||||
if (!path.resolve(root).startsWith(managedAreaBase() + path.sep)
|
||||
|| !await isRealPathAtManagedLocation(root)) {
|
||||
console.warn(`[GitManifest] restored ${sanitizeForLog(stackName)} but left its managed area in place: it is not at its own location in the managed area`);
|
||||
} else {
|
||||
await fs.promises.rm(root, { recursive: true, force: true });
|
||||
}
|
||||
}
|
||||
return true;
|
||||
}
|
||||
@@ -1509,6 +1554,18 @@ export class GitProjectManifestService {
|
||||
async stageManagedAreaForDetach(stackName: string): Promise<boolean> {
|
||||
const root = this.managedRoot(stackName);
|
||||
const staged = this.detachStagedRoot(stackName);
|
||||
// Inline containment barrier at the removal sink (see
|
||||
// `managedAreaBase`): the analyzer credits this literal comparison,
|
||||
// not the positional check below it.
|
||||
if (!path.resolve(staged).startsWith(managedAreaBase() + path.sep)) {
|
||||
throw new Error(`refusing to restage the managed area for ${sanitizeForLog(stackName)}: a stale staged area resolves outside the managed area`);
|
||||
}
|
||||
// Positional barrier on the stale-staged cleanup. Thrown rather than
|
||||
// skipped because the rename below needs this path back, and detaching
|
||||
// over an unverifiable directory silently is the outcome to avoid.
|
||||
if (!await isRealPathAtManagedLocation(staged)) {
|
||||
throw new Error(`refusing to restage the managed area for ${sanitizeForLog(stackName)}: a stale staged area is not at its own location in the managed area`);
|
||||
}
|
||||
await fs.promises.rm(staged, { recursive: true, force: true });
|
||||
try {
|
||||
await fs.promises.rename(root, staged);
|
||||
@@ -1529,8 +1586,19 @@ export class GitProjectManifestService {
|
||||
|
||||
/** Delete a staged managed area after the database row is gone. */
|
||||
async finalizeStagedDetach(stackName: string): Promise<boolean> {
|
||||
const staged = this.detachStagedRoot(stackName);
|
||||
// Same refusal semantics as `deleteManagedArea`: false leaves the area
|
||||
// in place for an operator rather than deleting through a redirected
|
||||
// root now that nothing else references it. The inline containment
|
||||
// barrier (see `managedAreaBase`) is what the analyzer credits; the
|
||||
// positional check carries the property.
|
||||
if (!path.resolve(staged).startsWith(managedAreaBase() + path.sep)
|
||||
|| !await isRealPathAtManagedLocation(staged)) {
|
||||
console.warn(`[GitManifest] refusing to delete staged area for ${sanitizeForLog(stackName)}: it is not at its own location in the managed area`);
|
||||
return false;
|
||||
}
|
||||
try {
|
||||
await fs.promises.rm(this.detachStagedRoot(stackName), { recursive: true, force: true });
|
||||
await fs.promises.rm(staged, { recursive: true, force: true });
|
||||
return true;
|
||||
} catch (e) {
|
||||
console.warn('[GitManifest] staged detach cleanup failed:', sanitizeForLog(stackName), (e as Error).message);
|
||||
@@ -1546,6 +1614,16 @@ export class GitProjectManifestService {
|
||||
*/
|
||||
async deleteManagedArea(stackName: string): Promise<boolean> {
|
||||
const root = this.managedRoot(stackName);
|
||||
// Same refusal semantics as the rm failure below: false means the area
|
||||
// survived, which is what keeps a detach from dropping its row while
|
||||
// generations it cannot account for are still on disk. The inline
|
||||
// containment barrier (see `managedAreaBase`) is what the analyzer
|
||||
// credits; the positional check carries the property.
|
||||
if (!path.resolve(root).startsWith(managedAreaBase() + path.sep)
|
||||
|| !await isRealPathAtManagedLocation(root)) {
|
||||
console.warn(`[GitManifest] refusing to delete managed area for ${sanitizeForLog(stackName)}: it is not at its own location in the managed area`);
|
||||
return false;
|
||||
}
|
||||
try {
|
||||
await fs.promises.rm(root, { recursive: true, force: true });
|
||||
return true;
|
||||
|
||||
@@ -28,6 +28,20 @@ import type { ComposeInputEntry, GitProjectManifest, GitSourceManifestState, Inv
|
||||
import type { GitChangePlan, PublicGitChangePlan, GitChangePlanCounts, PublicGitChangePlanOperation } from '../types/gitChangePlan';
|
||||
import { GIT_CHANGE_PLAN_SCHEMA_VERSION } from '../types/gitChangePlan';
|
||||
import type { NotificationCategory } from './NotificationService';
|
||||
import { GitOpsStore } from './gitops/store';
|
||||
import { GitOpsTransitions, GitOpsTransitionError } from './gitops/transitions';
|
||||
import {
|
||||
buildCreateCheckpointRow,
|
||||
buildDirectApplicationRow,
|
||||
buildGenerationRow,
|
||||
directSourceIdentity,
|
||||
newGitOpsId,
|
||||
stackManagedRoot,
|
||||
} from './gitops/directApplication';
|
||||
import type { GitOpsApplicationRow } from './gitops/types';
|
||||
import { appliedRelPathFor, candidateRelPathForSha, deleteStagingMarker, readStagingMarker, writeStagingMarker } from './gitops/createStagingMarker';
|
||||
import { cleanupUnclaimedManagedRoot, removeOperationOwnedPaths } from './gitops/createCleanup';
|
||||
import { managedAreaBase } from './gitops/managedPaths';
|
||||
import type { GitHttpRequest, GitHttpResponse, HttpClient } from 'isomorphic-git/http/node';
|
||||
|
||||
// isomorphic-git is the heaviest dependency in the backend (~5 MB) and only
|
||||
@@ -180,7 +194,8 @@ export type GitSourceErrorCode =
|
||||
| 'PLAN_FINGERPRINT_REQUIRED'
|
||||
| 'PLAN_BLOCKED'
|
||||
| 'LEGACY_PENDING'
|
||||
| 'PLAN_UNAVAILABLE';
|
||||
| 'PLAN_UNAVAILABLE'
|
||||
| 'OPERATION_IN_FLIGHT';
|
||||
|
||||
export class GitSourceError extends Error {
|
||||
constructor(
|
||||
@@ -758,31 +773,88 @@ export class GitSourceService {
|
||||
(existing.context_dir ?? null) !== input.contextDir
|
||||
);
|
||||
|
||||
db.upsertGitSource({
|
||||
stack_name: input.stackName,
|
||||
repo_url: input.repoUrl,
|
||||
const gitopsConfig = {
|
||||
repoUrl: input.repoUrl,
|
||||
branch: input.branch,
|
||||
compose_path: input.composePaths[0],
|
||||
compose_paths: input.composePaths,
|
||||
context_dir: input.contextDir,
|
||||
sync_env: input.syncEnv,
|
||||
env_path: resolvedEnvPath,
|
||||
auth_type: input.authType,
|
||||
encrypted_token: encryptedToken,
|
||||
auto_apply_on_webhook: input.autoApplyOnWebhook,
|
||||
auto_deploy_on_apply: input.autoDeployOnApply,
|
||||
last_applied_commit_sha: existing?.last_applied_commit_sha ?? null,
|
||||
last_applied_content_hash: existing?.last_applied_content_hash ?? null,
|
||||
pending_commit_sha: existing?.pending_commit_sha ?? null,
|
||||
pending_compose_content: existing?.pending_compose_content ?? null,
|
||||
pending_env_content: existing?.pending_env_content ?? null,
|
||||
pending_fetched_at: existing?.pending_fetched_at ?? null,
|
||||
last_debounce_at: existing?.last_debounce_at ?? null,
|
||||
});
|
||||
composePaths: input.composePaths,
|
||||
contextDir: input.contextDir,
|
||||
syncEnv: input.syncEnv,
|
||||
envPath: resolvedEnvPath,
|
||||
};
|
||||
const gitopsIdentity = directSourceIdentity(gitopsConfig);
|
||||
|
||||
if (configChanged) {
|
||||
db.clearGitSourcePending(input.stackName);
|
||||
}
|
||||
// The source row, the pending clear, and the GitOps transition commit
|
||||
// together. Clearing pending without invalidating the candidate would
|
||||
// leave the model offering an apply for files the operator can no
|
||||
// longer produce.
|
||||
db.getDb().transaction(() => {
|
||||
db.upsertGitSource({
|
||||
stack_name: input.stackName,
|
||||
repo_url: input.repoUrl,
|
||||
branch: input.branch,
|
||||
compose_path: input.composePaths[0],
|
||||
compose_paths: input.composePaths,
|
||||
context_dir: input.contextDir,
|
||||
sync_env: input.syncEnv,
|
||||
env_path: resolvedEnvPath,
|
||||
auth_type: input.authType,
|
||||
encrypted_token: encryptedToken,
|
||||
auto_apply_on_webhook: input.autoApplyOnWebhook,
|
||||
auto_deploy_on_apply: input.autoDeployOnApply,
|
||||
last_applied_commit_sha: existing?.last_applied_commit_sha ?? null,
|
||||
last_applied_content_hash: existing?.last_applied_content_hash ?? null,
|
||||
pending_commit_sha: existing?.pending_commit_sha ?? null,
|
||||
pending_compose_content: existing?.pending_compose_content ?? null,
|
||||
pending_env_content: existing?.pending_env_content ?? null,
|
||||
pending_fetched_at: existing?.pending_fetched_at ?? null,
|
||||
last_debounce_at: existing?.last_debounce_at ?? null,
|
||||
});
|
||||
|
||||
if (configChanged) {
|
||||
db.clearGitSourcePending(input.stackName);
|
||||
}
|
||||
|
||||
const app = this.gitopsApplicationFor(input.stackName);
|
||||
const envelope = this.gitopsEnvelope(crypto.randomUUID(), 'system:git-source', 'configure');
|
||||
if (!app && !existing && !this.gitopsNameHeld(input.stackName)) {
|
||||
// Linking a stack that already exists. Nothing is fetched or
|
||||
// accepted yet, so the application starts live with no desired
|
||||
// commit and the projection asks for a fetch.
|
||||
GitOpsTransitions.getInstance().activateDirect({
|
||||
application: buildDirectApplicationRow({
|
||||
id: newGitOpsId(),
|
||||
stackName: input.stackName,
|
||||
config: gitopsConfig,
|
||||
identity: gitopsIdentity,
|
||||
lifecycleStatus: 'active',
|
||||
at: envelope.at,
|
||||
}),
|
||||
nodeId: NodeRegistry.getInstance().getDefaultNodeId(),
|
||||
envelope,
|
||||
});
|
||||
return;
|
||||
}
|
||||
// Credential-only and policy-only edits change nothing material, so
|
||||
// they leave the candidate and every accepted pointer alone.
|
||||
if (app && configChanged) {
|
||||
GitOpsTransitions.getInstance().configChangedPendingCleared({
|
||||
applicationId: app.id,
|
||||
identity: {
|
||||
repoUrl: gitopsIdentity.repoUrl,
|
||||
repoIdentityJson: JSON.stringify(gitopsIdentity.identity),
|
||||
configuredRef: input.branch,
|
||||
},
|
||||
material: {
|
||||
composePathsJson: JSON.stringify([...input.composePaths]),
|
||||
contextDir: input.contextDir,
|
||||
syncEnv: input.syncEnv ? 1 : 0,
|
||||
envPath: resolvedEnvPath,
|
||||
fingerprint: gitopsIdentity.fingerprint,
|
||||
},
|
||||
envelope,
|
||||
});
|
||||
}
|
||||
})();
|
||||
|
||||
return this.get(input.stackName)!;
|
||||
}
|
||||
@@ -933,7 +1005,24 @@ export class GitSourceService {
|
||||
await rollbackAndThrow('Managed project data disappeared during detach');
|
||||
}
|
||||
try {
|
||||
DatabaseService.getInstance().deleteGitSource(stackName);
|
||||
// The source row and the GitOps tombstones commit together, so
|
||||
// a detached stack can never leave a live application pointing
|
||||
// at a source that no longer exists. Configured identity and
|
||||
// SHA pointers survive on the tombstone as frozen facts, and a
|
||||
// later reattach mints a new application rather than reviving
|
||||
// this one.
|
||||
const gitopsApp = this.gitopsApplicationFor(stackName);
|
||||
DatabaseService.getInstance().getDb().transaction(() => {
|
||||
DatabaseService.getInstance().deleteGitSource(stackName);
|
||||
if (!gitopsApp) return;
|
||||
const tx = GitOpsTransitions.getInstance();
|
||||
const envelope = this.gitopsEnvelope(crypto.randomUUID(), 'system:git-source', 'detach');
|
||||
for (const target of GitOpsStore.getInstance().listTargets(gitopsApp.id)) {
|
||||
if (target.target_status !== 'active') continue;
|
||||
tx.targetTombstoned(gitopsApp.id, target.node_id, envelope);
|
||||
}
|
||||
tx.applicationTombstoned(gitopsApp.id, 'detached', envelope);
|
||||
})();
|
||||
} catch (e) {
|
||||
await rollbackAndThrow('Could not commit the Git source removal', e);
|
||||
}
|
||||
@@ -1777,12 +1866,131 @@ export class GitSourceService {
|
||||
* reads last_debounce_at while it is still unset on every request, slips
|
||||
* past the gate, and clones once per request.
|
||||
*/
|
||||
/**
|
||||
* The GitOps application tracking this stack, or null when there is none.
|
||||
*
|
||||
* Stacks that predate the revision-state model have no application until
|
||||
* migration runs, so every producer is a no-op for them rather than
|
||||
* inventing an application from configuration alone.
|
||||
*/
|
||||
private gitopsApplicationFor(stackName: string): GitOpsApplicationRow | null {
|
||||
const app = GitOpsStore.getInstance().getLiveDirectApplication(stackName);
|
||||
return app && app.lifecycle_status === 'active' ? app : null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether any application still holds this stack name.
|
||||
*
|
||||
* Wider than `gitopsApplicationFor`, which only reports usable applications.
|
||||
* A `creating` row left by a crash the boot sweep could not settle still
|
||||
* occupies the unique live-application index, so activating over it would
|
||||
* fail the whole save with an internal constraint message.
|
||||
*/
|
||||
private gitopsNameHeld(stackName: string): boolean {
|
||||
return !!GitOpsStore.getInstance().getLiveDirectApplication(stackName);
|
||||
}
|
||||
|
||||
private gitopsEnvelope(operationId: string, actor: string, trigger: string) {
|
||||
return { operationId, actor, trigger, at: Date.now() };
|
||||
}
|
||||
|
||||
/**
|
||||
* Record a GitOps transition without letting it break the operation it
|
||||
* describes.
|
||||
*
|
||||
* The store is the record of what happened, not the mechanism that makes it
|
||||
* happen, so a rejected transition must not fail a fetch or an apply that
|
||||
* has already touched the filesystem. The rejection is logged loudly
|
||||
* because it means the recorded state has drifted from reality.
|
||||
*/
|
||||
private recordGitOps(stackName: string, what: string, write: () => void): boolean {
|
||||
try {
|
||||
write();
|
||||
return true;
|
||||
} catch (error) {
|
||||
console.error(
|
||||
`[GitOps] Could not record ${what} for ${sanitizeForLog(stackName)}:`,
|
||||
error instanceof Error ? error.stack ?? error.message : String(error),
|
||||
);
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Close an operation whose terminal transition was rejected.
|
||||
*
|
||||
* A start that never terminates leaves the source reporting work in
|
||||
* progress and offering no actions, permanently. When the real terminal
|
||||
* event cannot be recorded, the next best truth is that we lost track:
|
||||
* clear the operation and stamp a failure the projection can render, so the
|
||||
* operator sees an error they can retry instead of a spinner.
|
||||
*/
|
||||
private abandonGitOpsOperation(
|
||||
stackName: string,
|
||||
applicationId: string,
|
||||
envelope: ReturnType<GitSourceService['gitopsEnvelope']>,
|
||||
): void {
|
||||
this.recordGitOps(stackName, 'lost-track fallback', () => {
|
||||
GitOpsTransitions.getInstance().applyFailed(applicationId, 'bookkeeping_rejected', envelope);
|
||||
});
|
||||
}
|
||||
|
||||
private async pullLocked(stackName: string, actor: string): Promise<PullResult> {
|
||||
const db = DatabaseService.getInstance();
|
||||
const src = db.getGitSource(stackName);
|
||||
if (!src) throw new GitSourceError('GIT_ERROR', 'No Git source configured for this stack.');
|
||||
const gitopsApp = this.gitopsApplicationFor(stackName);
|
||||
const gitopsOperationId = crypto.randomUUID();
|
||||
const gitopsEnv = this.gitopsEnvelope(gitopsOperationId, actor, 'pull');
|
||||
// A fetch that starts and never terminates is worse than one that is
|
||||
// never recorded: fetchStarted refuses to open a second operation, so
|
||||
// every later pull silently stops being tracked until a restart.
|
||||
let fetchOpen = false;
|
||||
if (gitopsApp) {
|
||||
fetchOpen = this.recordGitOps(stackName, 'fetch start', () => {
|
||||
GitOpsTransitions.getInstance().fetchStarted(gitopsApp.id, gitopsEnv);
|
||||
});
|
||||
}
|
||||
const closeFetch = (): void => {
|
||||
if (!gitopsApp || !fetchOpen) return;
|
||||
fetchOpen = false;
|
||||
this.recordGitOps(stackName, 'fetch failure', () => {
|
||||
GitOpsTransitions.getInstance().fetchFailed(gitopsApp.id, gitopsEnv);
|
||||
});
|
||||
};
|
||||
|
||||
try {
|
||||
return await this.pullLockedBody(stackName, actor, src, {
|
||||
app: gitopsApp,
|
||||
operationId: gitopsOperationId,
|
||||
envelope: gitopsEnv,
|
||||
markSettled: () => { fetchOpen = false; },
|
||||
abandon: closeFetch,
|
||||
});
|
||||
} catch (e) {
|
||||
closeFetch();
|
||||
throw e;
|
||||
}
|
||||
}
|
||||
|
||||
private async pullLockedBody(
|
||||
stackName: string,
|
||||
actor: string,
|
||||
src: StackGitSource,
|
||||
gitops: {
|
||||
app: GitOpsApplicationRow | null;
|
||||
operationId: string;
|
||||
envelope: ReturnType<GitSourceService['gitopsEnvelope']>;
|
||||
markSettled: () => void;
|
||||
abandon: () => void;
|
||||
},
|
||||
): Promise<PullResult> {
|
||||
const db = DatabaseService.getInstance();
|
||||
const diag = isDebugEnabled();
|
||||
const gitopsApp = gitops.app;
|
||||
const gitopsOperationId = gitops.operationId;
|
||||
const gitopsEnv = gitops.envelope;
|
||||
|
||||
if (diag) {
|
||||
console.log(`[GitSource:diag] pull start stack=${stackName} branch=${src.branch} host=${repoHost(src.repo_url)}`);
|
||||
}
|
||||
@@ -1792,7 +2000,9 @@ export class GitSourceService {
|
||||
// Object holder: property access is not narrowed by control-flow
|
||||
// analysis, so the closure assignment below stays visible.
|
||||
const materialization: { value: MaterializationResult | null } = { value: null };
|
||||
const fetched = await this.fetchFromGit({
|
||||
// Every throw from here on, including this fetch, is closed by the
|
||||
// caller's handler, so nothing is recorded locally.
|
||||
const fetched: FetchResult = await this.fetchFromGit({
|
||||
repoUrl: src.repo_url,
|
||||
branch: src.branch,
|
||||
composePaths: src.compose_paths,
|
||||
@@ -1835,6 +2045,80 @@ export class GitSourceService {
|
||||
});
|
||||
}
|
||||
|
||||
// Record what this fetch resolved before the pending blob is written,
|
||||
// so the durable pointers and the operational pending store agree.
|
||||
if (gitopsApp) {
|
||||
const outcomeRecorded = this.recordGitOps(stackName, 'fetch outcome', () => {
|
||||
const tx = GitOpsTransitions.getInstance();
|
||||
// One transaction: a candidate that exists without the fetch
|
||||
// that produced it would let a later apply accept the wrong
|
||||
// generation while the projection reports the older commit.
|
||||
DatabaseService.getInstance().getDb().transaction(() => {
|
||||
if (!validation.ok) {
|
||||
tx.fetchedInvalid(gitopsApp.id, fetched.commitSha, gitopsEnv);
|
||||
return;
|
||||
}
|
||||
tx.fetched(gitopsApp.id, fetched.commitSha, gitopsEnv);
|
||||
if (!materialization.value) return;
|
||||
const identity = directSourceIdentity({
|
||||
repoUrl: src.repo_url,
|
||||
branch: src.branch,
|
||||
composePaths: src.compose_paths,
|
||||
contextDir: src.context_dir,
|
||||
syncEnv: src.sync_env,
|
||||
envPath: src.env_path,
|
||||
});
|
||||
// A pull that resolves to exactly what the live candidate
|
||||
// already proposes (same commit, source fingerprint, plan
|
||||
// verdict) must not mint a lookalike generation and rewrite the
|
||||
// candidate pointers. The staged generation stands; only the
|
||||
// fetch above is new. A candidate for a different commit, or no
|
||||
// candidate at all, mints anew: staging after an apply is a new
|
||||
// dispatch cycle and needs its own generation to accept.
|
||||
const staged = gitopsApp.candidate_generation_id
|
||||
? GitOpsStore.getInstance().getGeneration(gitopsApp.candidate_generation_id)
|
||||
: undefined;
|
||||
if (
|
||||
staged &&
|
||||
staged.commit_sha === fetched.commitSha &&
|
||||
staged.materialization_fingerprint === identity.fingerprint &&
|
||||
staged.plan_blocked === (plan?.blocked === true ? 1 : 0)
|
||||
) {
|
||||
return;
|
||||
}
|
||||
const generationId = newGitOpsId();
|
||||
const nextManifestVersion = (prior?.manifestVersion ?? 0) + 1;
|
||||
GitOpsStore.getInstance().insertGeneration(buildGenerationRow({
|
||||
id: generationId,
|
||||
applicationId: gitopsApp.id,
|
||||
commitSha: fetched.commitSha,
|
||||
identity,
|
||||
configuredRef: src.branch,
|
||||
candidateRelPath: materialization.value.candidateRelPath,
|
||||
appliedRelPath: appliedRelPathFor(fetched.commitSha, nextManifestVersion),
|
||||
manifestVersion: nextManifestVersion,
|
||||
// The candidate's own invocation. Recording the prior
|
||||
// generation's would attribute one generation's facts to
|
||||
// another, which is the whole failure this model prevents.
|
||||
expectedInvocation: plan?.candidateInvocation ?? prior?.project.invocation ?? null,
|
||||
changePlanFingerprint: plan?.fingerprint ?? null,
|
||||
operationId: gitopsOperationId,
|
||||
trigger: gitopsEnv.trigger,
|
||||
actor,
|
||||
at: gitopsEnv.at,
|
||||
planBlocked: plan?.blocked === true,
|
||||
}));
|
||||
if (plan?.blocked) tx.sourceConflictBlocker(gitopsApp.id, generationId, gitopsEnv);
|
||||
else tx.candidateReady(gitopsApp.id, generationId, false, gitopsEnv);
|
||||
})();
|
||||
gitops.markSettled();
|
||||
});
|
||||
// The pull itself succeeded; the files and the pending blob are
|
||||
// real. Closing the operation is what stops the source reporting a
|
||||
// fetch in flight for ever and locking out every later pull.
|
||||
if (!outcomeRecorded) gitops.abandon();
|
||||
}
|
||||
|
||||
const publicPlan = plan ? GitChangePlanService.getInstance().toPublic(plan) : null;
|
||||
const summary = plan ? GitChangePlanService.getInstance().toPendingSummary(plan) : null;
|
||||
db.setGitSourcePending(
|
||||
@@ -1947,10 +2231,47 @@ export class GitSourceService {
|
||||
}
|
||||
|
||||
/** Body of apply(); assumes the caller already holds Git mutex + shared stack lock. */
|
||||
/**
|
||||
* Apply a pending pull, recording the attempt as a GitOps operation.
|
||||
*
|
||||
* The wrapper exists so a throw anywhere in the body still closes the
|
||||
* operation. An apply that started and never terminated would leave the
|
||||
* source projecting `applying` until the next restart reclassified it as an
|
||||
* interruption, which reads as "still working" when nothing is.
|
||||
*/
|
||||
private async applyLocked(
|
||||
stackName: string,
|
||||
commitSha: string,
|
||||
opts: GitApplyOpts,
|
||||
): Promise<{ applied: boolean; deployed: boolean; deployError?: string; recoveryId?: string }> {
|
||||
const started: { app: GitOpsApplicationRow | null; env: ReturnType<GitSourceService['gitopsEnvelope']> | null; settled: boolean } = {
|
||||
app: null,
|
||||
env: null,
|
||||
settled: false,
|
||||
};
|
||||
try {
|
||||
return await this.applyLockedBody(stackName, commitSha, opts, started);
|
||||
} catch (e) {
|
||||
if (started.app && started.env && !started.settled) {
|
||||
const app = started.app;
|
||||
const env = started.env;
|
||||
this.recordGitOps(stackName, 'apply failure', () => {
|
||||
GitOpsTransitions.getInstance().applyFailed(
|
||||
app.id,
|
||||
e instanceof GitSourceError ? e.code : 'apply',
|
||||
env,
|
||||
);
|
||||
});
|
||||
}
|
||||
throw e;
|
||||
}
|
||||
}
|
||||
|
||||
private async applyLockedBody(
|
||||
stackName: string,
|
||||
commitSha: string,
|
||||
opts: GitApplyOpts,
|
||||
started: { app: GitOpsApplicationRow | null; env: ReturnType<GitSourceService['gitopsEnvelope']> | null; settled: boolean },
|
||||
): Promise<{ applied: boolean; deployed: boolean; deployError?: string; recoveryId?: string }> {
|
||||
const diag = isDebugEnabled();
|
||||
const db = DatabaseService.getInstance();
|
||||
@@ -1994,6 +2315,34 @@ export class GitSourceService {
|
||||
const actor = opts.actor ?? 'system:git-source';
|
||||
let recoveryId: string | undefined;
|
||||
|
||||
// The apply is bound to the candidate the pull recorded. Without one
|
||||
// there is nothing to accept, so the acceptance below is skipped rather
|
||||
// than inventing a generation from the pending blob.
|
||||
const gitopsApp = this.gitopsApplicationFor(stackName);
|
||||
// The candidate must be the generation built from the commit being
|
||||
// applied. Without this check a candidate left behind by a swallowed
|
||||
// fetch outcome would be accepted for files from a different commit,
|
||||
// and the projection would confidently report the wrong one.
|
||||
const candidateId = gitopsApp?.candidate_generation_id ?? null;
|
||||
const candidateGeneration = candidateId
|
||||
? GitOpsStore.getInstance().getGeneration(candidateId)
|
||||
: undefined;
|
||||
const gitopsGenerationId = candidateGeneration?.commit_sha === commitSha ? candidateId : null;
|
||||
if (candidateId && !gitopsGenerationId) {
|
||||
console.warn(
|
||||
'[GitOps] Skipping acceptance for %s: the recorded candidate is not built from %s',
|
||||
sanitizeForLog(stackName), sanitizeForLog(commitSha.slice(0, 7)),
|
||||
);
|
||||
}
|
||||
const gitopsEnv = this.gitopsEnvelope(pending.operationId, actor, 'apply');
|
||||
if (gitopsApp && gitopsGenerationId) {
|
||||
this.recordGitOps(stackName, 'apply start', () => {
|
||||
GitOpsTransitions.getInstance().applyStarted(gitopsApp.id, gitopsGenerationId, gitopsEnv);
|
||||
started.app = gitopsApp;
|
||||
started.env = gitopsEnv;
|
||||
});
|
||||
}
|
||||
|
||||
let appliedSpec: GitSourceAppliedSpec | null;
|
||||
if (pending.candidateRelPath !== null && pending.inventory !== null) {
|
||||
// ── Complete-project path (v4 pending) ───────────────────────────
|
||||
@@ -2255,6 +2604,25 @@ export class GitSourceService {
|
||||
db.markGitSourceApplied(stackName, commitSha, hash);
|
||||
db.setGitSourceAppliedSpec(stackName, appliedSpec);
|
||||
|
||||
// The files are on disk and the source row now points at this commit,
|
||||
// so this is where the generation becomes the accepted one.
|
||||
if (gitopsApp && gitopsGenerationId) {
|
||||
const recorded = this.recordGitOps(stackName, 'acceptance', () => {
|
||||
GitOpsTransitions.getInstance().applied({
|
||||
applicationId: gitopsApp.id,
|
||||
generationId: gitopsGenerationId,
|
||||
artifactSetId: newGitOpsId(),
|
||||
sourceAcceptanceId: newGitOpsId(),
|
||||
authority: actor === 'system:webhook' ? 'configured_policy' : 'operator',
|
||||
envelope: gitopsEnv,
|
||||
});
|
||||
});
|
||||
started.settled = true;
|
||||
// The files are on disk either way. What we can still control is
|
||||
// not leaving the operation open when the acceptance was rejected.
|
||||
if (!recorded) this.abandonGitOpsOperation(stackName, gitopsApp.id, gitopsEnv);
|
||||
}
|
||||
|
||||
const shouldDeploy = opts.deploy ?? src.auto_deploy_on_apply;
|
||||
if (diag) console.log('[GitSource:diag] apply wrote stack=%s sha=%s deploy=%s', sanitizeForLog(stackName), sanitizeForLog(commitSha.slice(0, 7)), sanitizeForLog(shouldDeploy));
|
||||
|
||||
@@ -2289,7 +2657,7 @@ export class GitSourceService {
|
||||
await finalizeRecoveryCurrent(recoveryId, false);
|
||||
}
|
||||
// Shared stack lock already held as git_apply for capture→deploy.
|
||||
await ComposeService.getInstance(nodeId).deployStack(
|
||||
const autoDeploy = await ComposeService.getInstance(nodeId).deployStack(
|
||||
stackName,
|
||||
undefined,
|
||||
undefined,
|
||||
@@ -2305,6 +2673,7 @@ export class GitSourceService {
|
||||
stackName,
|
||||
'deploy',
|
||||
'system:git-source',
|
||||
{ deployedGenerationId: autoDeploy.deployedGenerationId },
|
||||
);
|
||||
if (recoveryId) {
|
||||
recoverySvc.linkGateOrRetain(recoveryId, healthGateId);
|
||||
@@ -2354,7 +2723,28 @@ export class GitSourceService {
|
||||
return { applied: true, deployed: false, recoveryId };
|
||||
}
|
||||
|
||||
public dismissPending(stackName: string): void {
|
||||
public dismissPending(stackName: string, actor?: string): void {
|
||||
const app = GitOpsStore.getInstance().getLiveDirectApplication(stackName);
|
||||
if (app?.candidate_generation_id) {
|
||||
try {
|
||||
GitOpsTransitions.getInstance().dismissed(
|
||||
app.id,
|
||||
this.gitopsEnvelope(crypto.randomUUID(), actor ?? 'system:git-source', 'dismiss'),
|
||||
);
|
||||
} catch (error) {
|
||||
// The refusal is the outcome the operator must see. Swallowing it
|
||||
// here would leave the projection offering a candidate they just
|
||||
// declined, which is exactly the stale state dismissal exists to
|
||||
// prevent.
|
||||
if (error instanceof GitOpsTransitionError) {
|
||||
throw new GitSourceError(
|
||||
'OPERATION_IN_FLIGHT',
|
||||
`Cannot dismiss the pending update for ${stackName}: ${error.message}`,
|
||||
);
|
||||
}
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
DatabaseService.getInstance().clearGitSourcePending(stackName);
|
||||
}
|
||||
|
||||
@@ -2380,25 +2770,69 @@ export class GitSourceService {
|
||||
throw new GitSourceError('GIT_ERROR', 'Auto-deploy requires auto-apply-on-webhook to be enabled.');
|
||||
}
|
||||
|
||||
const gitopsOperationId = crypto.randomUUID();
|
||||
// Inline containment barrier at the stat sink. CodeQL does not
|
||||
// credit the wrapped isPathWithinBase helper, so resolve against the
|
||||
// managed-area base and check containment right here.
|
||||
const areaBase = managedAreaBase();
|
||||
const managedRoot = path.resolve(stackManagedRoot(input.stackName));
|
||||
if (!managedRoot.startsWith(areaBase + path.sep)) {
|
||||
throw new GitSourceError('GIT_ERROR', 'Invalid stack path');
|
||||
}
|
||||
// Whether the managed root is ours to delete is decided once, here,
|
||||
// before anything can create it. Cleanup later reads this answer
|
||||
// rather than re-probing a directory it may itself have made.
|
||||
const rootPreexisted = existsSync(managedRoot);
|
||||
const gitopsIdentity = directSourceIdentity({
|
||||
repoUrl: input.repoUrl,
|
||||
branch: input.branch,
|
||||
composePaths: input.composePaths,
|
||||
contextDir: input.contextDir,
|
||||
syncEnv: input.syncEnv,
|
||||
envPath: input.envPath,
|
||||
});
|
||||
const staged: { candidateRelPath: string | null } = { candidateRelPath: null };
|
||||
|
||||
// 1. Fetch from git BEFORE touching disk or DB. If the fetch
|
||||
// fails there is nothing to clean up. The onClone hook stages
|
||||
// the complete-project candidate inside the clone lifecycle.
|
||||
const manifestSvc = GitProjectManifestService.getInstance();
|
||||
const materialization: { value: MaterializationResult | null } = { value: null };
|
||||
const fetched = await this.fetchFromGit({
|
||||
repoUrl: input.repoUrl,
|
||||
branch: input.branch,
|
||||
composePaths: input.composePaths,
|
||||
envPath: input.syncEnv ? input.envPath : null,
|
||||
token: input.token,
|
||||
onClone: async (cloneDir, commitSha, envContent) => {
|
||||
materialization.value = await this.buildMaterialization(input.stackName, cloneDir, commitSha, {
|
||||
compose_paths: input.composePaths,
|
||||
context_dir: input.contextDir,
|
||||
sync_env: input.syncEnv,
|
||||
}, envContent);
|
||||
},
|
||||
});
|
||||
let fetched: FetchResult;
|
||||
try {
|
||||
fetched = await this.fetchFromGit({
|
||||
repoUrl: input.repoUrl,
|
||||
branch: input.branch,
|
||||
composePaths: input.composePaths,
|
||||
envPath: input.syncEnv ? input.envPath : null,
|
||||
token: input.token,
|
||||
onClone: async (cloneDir, commitSha, envContent) => {
|
||||
// The candidate path is recorded before the build that
|
||||
// creates it, so a crash mid-build still names exactly one
|
||||
// directory this operation owns.
|
||||
staged.candidateRelPath = candidateRelPathForSha(commitSha);
|
||||
await writeStagingMarker(managedRoot, {
|
||||
schemaVersion: 1,
|
||||
operationId: gitopsOperationId,
|
||||
rootPreexisted,
|
||||
candidateRelPath: staged.candidateRelPath,
|
||||
createdAt: Date.now(),
|
||||
});
|
||||
materialization.value = await this.buildMaterialization(input.stackName, cloneDir, commitSha, {
|
||||
compose_paths: input.composePaths,
|
||||
context_dir: input.contextDir,
|
||||
sync_env: input.syncEnv,
|
||||
}, envContent);
|
||||
},
|
||||
});
|
||||
} catch (e) {
|
||||
// Materialization refuses routinely, not just on crashes. The
|
||||
// marker has to come off with the staged files, or it would
|
||||
// claim this managed area against every later attempt and make
|
||||
// the stack name uncreatable until the next restart.
|
||||
await this.cleanupStagedCreate(managedRoot, staged.candidateRelPath, rootPreexisted);
|
||||
throw e;
|
||||
}
|
||||
|
||||
// 2. Validate against the same `docker compose config` check the
|
||||
// apply path uses. Reject before creating anything on disk.
|
||||
@@ -2406,6 +2840,7 @@ export class GitSourceService {
|
||||
? materialization.value.validation
|
||||
: await this.validateCompose(fetched.composeFiles, fetched.envContent, input.contextDir);
|
||||
if (!validation.ok) {
|
||||
await this.cleanupStagedCreate(managedRoot, staged.candidateRelPath, rootPreexisted);
|
||||
throw new GitSourceError('GIT_ERROR', `Compose validation failed: ${validation.error}`);
|
||||
}
|
||||
|
||||
@@ -2420,6 +2855,12 @@ export class GitSourceService {
|
||||
// state instead of 'absent'.
|
||||
let completeProjectManifest: GitProjectManifest | null = null;
|
||||
let recordedCreatePlan: GitChangePlan | null = null;
|
||||
// Set once the activation transaction commits. Before that there is
|
||||
// nothing in the database to tear down; after it, cleanup has to go
|
||||
// through create_failed so the checkpoint and tombstone stay
|
||||
// consistent with what was removed from disk.
|
||||
let gitopsApplicationId: string | null = null;
|
||||
let gitopsCommitted = false;
|
||||
try {
|
||||
let appliedSpec: GitSourceAppliedSpec | null;
|
||||
if (materialization.value) {
|
||||
@@ -2511,10 +2952,88 @@ export class GitSourceService {
|
||||
completeProjectManifest = manifest;
|
||||
}
|
||||
|
||||
// 3b. Persist the GitOps identity of this create. Everything
|
||||
// above is still reversible by deleting files; from here on
|
||||
// recovery is driven by the checkpoint instead of guesswork.
|
||||
if (completeProjectManifest && materialization.value && staged.candidateRelPath) {
|
||||
const applicationId = newGitOpsId();
|
||||
const envelope = {
|
||||
operationId: gitopsOperationId,
|
||||
actor: 'system:git-source',
|
||||
trigger: 'create',
|
||||
at: Date.now(),
|
||||
};
|
||||
const sourceConfig = {
|
||||
repoUrl: input.repoUrl,
|
||||
branch: input.branch,
|
||||
composePaths: input.composePaths,
|
||||
contextDir: input.contextDir,
|
||||
syncEnv: input.syncEnv,
|
||||
envPath: input.envPath,
|
||||
};
|
||||
GitOpsTransitions.getInstance().activateCreateFromGit({
|
||||
application: buildDirectApplicationRow({
|
||||
id: applicationId,
|
||||
stackName: input.stackName,
|
||||
config: sourceConfig,
|
||||
identity: gitopsIdentity,
|
||||
lifecycleStatus: 'creating',
|
||||
at: envelope.at,
|
||||
}),
|
||||
nodeId: NodeRegistry.getInstance().getDefaultNodeId(),
|
||||
commitSha: fetched.commitSha,
|
||||
generation: buildGenerationRow({
|
||||
id: newGitOpsId(),
|
||||
applicationId,
|
||||
commitSha: fetched.commitSha,
|
||||
identity: gitopsIdentity,
|
||||
configuredRef: input.branch,
|
||||
candidateRelPath: staged.candidateRelPath,
|
||||
appliedRelPath: appliedRelPathFor(fetched.commitSha, completeProjectManifest.manifestVersion),
|
||||
manifestVersion: completeProjectManifest.manifestVersion,
|
||||
expectedInvocation: completeProjectManifest.project.invocation,
|
||||
changePlanFingerprint: recordedCreatePlan?.fingerprint ?? null,
|
||||
operationId: gitopsOperationId,
|
||||
trigger: envelope.trigger,
|
||||
actor: envelope.actor,
|
||||
at: envelope.at,
|
||||
}),
|
||||
checkpoint: buildCreateCheckpointRow({
|
||||
applicationId,
|
||||
stackName: input.stackName,
|
||||
operationId: gitopsOperationId,
|
||||
config: sourceConfig,
|
||||
identity: gitopsIdentity,
|
||||
authType: input.authType,
|
||||
encryptedToken: input.authType === 'token' && input.token
|
||||
? this.crypto.encrypt(input.token)
|
||||
: null,
|
||||
autoApplyOnWebhook: input.autoApplyOnWebhook,
|
||||
autoDeployOnApply: input.autoDeployOnApply,
|
||||
commitSha: fetched.commitSha,
|
||||
createdManagedRoot: !rootPreexisted,
|
||||
at: envelope.at,
|
||||
}),
|
||||
envelope,
|
||||
});
|
||||
gitopsApplicationId = applicationId;
|
||||
await deleteStagingMarker(managedRoot);
|
||||
}
|
||||
|
||||
await fsSvc.createStack(input.stackName);
|
||||
stackCreated = true;
|
||||
if (gitopsApplicationId) {
|
||||
GitOpsStore.getInstance().updateCreateCheckpoint(
|
||||
gitopsApplicationId, { phase: 'stack_created' }, Date.now(),
|
||||
);
|
||||
}
|
||||
|
||||
if (completeProjectManifest && materialization.value) {
|
||||
if (gitopsApplicationId) {
|
||||
GitOpsStore.getInstance().updateCreateCheckpoint(
|
||||
gitopsApplicationId, { phase: 'promoting' }, Date.now(),
|
||||
);
|
||||
}
|
||||
await manifestSvc.promoteGeneration(input.stackName, {
|
||||
sha: fetched.commitSha,
|
||||
candidateRelPath: materialization.value.candidateRelPath,
|
||||
@@ -2523,6 +3042,13 @@ export class GitSourceService {
|
||||
adoptExistingMaterializedPaths: 'all',
|
||||
});
|
||||
appliedSpec = this.deriveAppliedSpec(input.composePaths, input.contextDir);
|
||||
if (gitopsApplicationId) {
|
||||
GitOpsStore.getInstance().updateCreateCheckpoint(
|
||||
gitopsApplicationId,
|
||||
{ phase: 'manifest_committed', appliedSpecJson: JSON.stringify(appliedSpec) },
|
||||
Date.now(),
|
||||
);
|
||||
}
|
||||
} else {
|
||||
appliedSpec = await this.materialize(
|
||||
input.stackName, fetched.composeFiles, input.contextDir, input.syncEnv, fetched.envContent, null,
|
||||
@@ -2537,6 +3063,11 @@ export class GitSourceService {
|
||||
? this.crypto.encrypt(input.token)
|
||||
: null;
|
||||
const hash = this.hashContent(fetched.composeFiles, fetched.envContent);
|
||||
// The source row, the applied pointers, and the checkpoint
|
||||
// advance together. This commit is the success boundary: once
|
||||
// it lands the stack is live, and any later error is reported
|
||||
// without deleting anything.
|
||||
const commitCreate = db.getDb().transaction(() => {
|
||||
db.upsertGitSource({
|
||||
stack_name: input.stackName,
|
||||
repo_url: input.repoUrl,
|
||||
@@ -2570,6 +3101,38 @@ export class GitSourceService {
|
||||
completeProjectManifest.generation.appliedDir,
|
||||
);
|
||||
}
|
||||
if (gitopsApplicationId) {
|
||||
const checkpoint = GitOpsStore.getInstance().getCreateCheckpoint(gitopsApplicationId);
|
||||
if (!checkpoint?.generation_id) {
|
||||
throw new GitSourceError('GIT_ERROR', 'Create checkpoint lost its generation before acceptance.');
|
||||
}
|
||||
GitOpsTransitions.getInstance().applied({
|
||||
applicationId: gitopsApplicationId,
|
||||
generationId: checkpoint.generation_id,
|
||||
artifactSetId: newGitOpsId(),
|
||||
sourceAcceptanceId: newGitOpsId(),
|
||||
authority: 'operator',
|
||||
envelope: {
|
||||
operationId: gitopsOperationId,
|
||||
actor: 'system:git-source',
|
||||
trigger: 'create',
|
||||
at: Date.now(),
|
||||
},
|
||||
activateCreating: true,
|
||||
});
|
||||
GitOpsStore.getInstance().updateCreateCheckpoint(
|
||||
gitopsApplicationId, { phase: 'pointers_committed' }, Date.now(),
|
||||
);
|
||||
}
|
||||
});
|
||||
commitCreate();
|
||||
gitopsCommitted = true;
|
||||
// The checkpoint has done its job. Dropping it here keeps the
|
||||
// boot sweep reporting only genuine interruptions, and stops a
|
||||
// copy of the encrypted token living past the create.
|
||||
if (gitopsApplicationId) {
|
||||
GitOpsStore.getInstance().deleteCreateCheckpoint(gitopsApplicationId);
|
||||
}
|
||||
|
||||
rowInserted = true;
|
||||
const operationId = crypto.randomUUID();
|
||||
@@ -2601,6 +3164,31 @@ export class GitSourceService {
|
||||
}
|
||||
return { source, commitSha: fetched.commitSha, envWritten, warnings: fetched.warnings };
|
||||
} catch (e) {
|
||||
// Past the success boundary the stack is live and owned by the
|
||||
// operator. A later error is reported, never compensated: the
|
||||
// leftover marker or checkpoint is finished by the boot sweep.
|
||||
if (gitopsCommitted) {
|
||||
const detail = e instanceof Error ? e.message : String(e);
|
||||
console.error(
|
||||
`[GitSource] Create for ${sanitizeForLog(input.stackName)} succeeded but a later step failed:`,
|
||||
detail,
|
||||
);
|
||||
this.recordGitActivity(
|
||||
input.stackName,
|
||||
'git_create',
|
||||
`Git create for ${input.stackName} completed, but a follow-up step failed: ${detail}`,
|
||||
'system:git-source',
|
||||
'warning',
|
||||
);
|
||||
// Say plainly that the stack exists. The raw downstream
|
||||
// error reads as a failed create, and an operator acting on
|
||||
// it retries and hits "stack already exists", which looks
|
||||
// like corruption rather than success.
|
||||
throw new GitSourceError(
|
||||
'GIT_ERROR',
|
||||
`The stack was created from Git, but a follow-up step failed: ${detail}`,
|
||||
);
|
||||
}
|
||||
// Roll back any partial on-disk state so the caller can retry
|
||||
// cleanly. The DB row is only inserted at step 4, so an error
|
||||
// earlier leaves nothing to clean in the DB.
|
||||
@@ -2616,7 +3204,10 @@ export class GitSourceService {
|
||||
// for a non-existence reason) must never lose its previous
|
||||
// applied generations to someone else's rollback: when the stack
|
||||
// dir was NOT created by us, remove only the candidate we staged.
|
||||
if (stackCreated) {
|
||||
if (stackCreated && !rootPreexisted) {
|
||||
// Only legal because this operation created the managed
|
||||
// root. A root that predated the create holds retained
|
||||
// generations of its own and is cleaned path by path below.
|
||||
await GitProjectManifestService.getInstance().deleteManagedArea(input.stackName);
|
||||
} else if (materialization.value) {
|
||||
const dataDir = process.env.DATA_DIR || path.join(process.cwd(), 'data');
|
||||
@@ -2632,11 +3223,61 @@ export class GitSourceService {
|
||||
if (rowInserted) {
|
||||
db.deleteGitSource(input.stackName);
|
||||
}
|
||||
// Filesystem cleanup has to succeed before the tombstone, so a
|
||||
// create whose files could not be removed keeps its checkpoint
|
||||
// and is retried by the next boot rather than being recorded as
|
||||
// cleanly failed.
|
||||
if (gitopsApplicationId) {
|
||||
try {
|
||||
await removeOperationOwnedPaths({
|
||||
stackManagedRoot: managedRoot,
|
||||
candidateRelPath: staged.candidateRelPath,
|
||||
appliedRelPath: completeProjectManifest?.generation.appliedDir ?? null,
|
||||
ownsManagedRoot: !rootPreexisted,
|
||||
});
|
||||
GitOpsTransitions.getInstance().createFailed(
|
||||
gitopsApplicationId,
|
||||
e instanceof GitSourceError ? e.code : 'create',
|
||||
{ operationId: gitopsOperationId, actor: 'system:git-source', trigger: 'create', at: Date.now() },
|
||||
);
|
||||
} catch (cleanupErr) {
|
||||
console.error(
|
||||
`[GitSource] Could not finish tearing down the failed create for ${sanitizeForLog(input.stackName)}; leaving it for the next boot sweep:`,
|
||||
cleanupErr instanceof Error ? cleanupErr.message : String(cleanupErr),
|
||||
);
|
||||
}
|
||||
}
|
||||
throw e;
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Remove what a create staged before any GitOps row existed.
|
||||
*
|
||||
* Best-effort by design: the caller is already throwing the real error, and
|
||||
* a leftover directory here is picked up by the boot sweep, which has the
|
||||
* marker to tell it what this operation owned.
|
||||
*/
|
||||
private async cleanupStagedCreate(
|
||||
managedRoot: string,
|
||||
candidateRelPath: string | null,
|
||||
rootPreexisted: boolean,
|
||||
): Promise<void> {
|
||||
try {
|
||||
await removeOperationOwnedPaths({
|
||||
stackManagedRoot: managedRoot,
|
||||
candidateRelPath,
|
||||
ownsManagedRoot: !rootPreexisted,
|
||||
});
|
||||
} catch (error) {
|
||||
console.warn(
|
||||
'[GitSource] Could not remove the staged create area:',
|
||||
error instanceof Error ? error.message : String(error),
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Boot sweep for every managed-project area, under the per-stack lock:
|
||||
* crash-recovery restore, orphan candidates, and areas whose stack no
|
||||
@@ -2730,13 +3371,63 @@ export class GitSourceService {
|
||||
}
|
||||
return;
|
||||
}
|
||||
// A managed area with no git-source row is not automatically an orphan.
|
||||
// An in-flight or crashed create owns its area through a checkpoint or
|
||||
// a creating application before the source row exists, so both count as
|
||||
// known and must survive the sweep.
|
||||
for (const checkpoint of GitOpsStore.getInstance().listCreateCheckpoints()) {
|
||||
known.add(checkpoint.stack_name);
|
||||
}
|
||||
for (const app of GitOpsStore.getInstance().listCreatingDirectApplications()) {
|
||||
if (app.stack_name) known.add(app.stack_name);
|
||||
}
|
||||
|
||||
for (const entry of entries) {
|
||||
if (!entry.isDirectory() || known.has(entry.name)) continue;
|
||||
if (entry.name.startsWith('.detach-') && known.has(entry.name.slice('.detach-'.length))) continue;
|
||||
// Detach staging areas carry their own ownership proof in the
|
||||
// detach journal, not a create marker. They are reaped exactly as
|
||||
// before once their stack is gone.
|
||||
if (entry.name.startsWith('.detach-')) {
|
||||
if (known.has(entry.name.slice('.detach-'.length))) continue;
|
||||
try {
|
||||
await fsPromises.rm(path.join(managedRoot, entry.name), { recursive: true, force: true });
|
||||
} catch (e) {
|
||||
console.error(`[GitManifest] could not remove staged detach area ${sanitizeForLog(entry.name)}:`, (e as Error).message);
|
||||
}
|
||||
continue;
|
||||
}
|
||||
// Nothing in the database claims this directory: no git-source row,
|
||||
// no create checkpoint, no creating application. What happens next
|
||||
// turns on the staging marker, and the distinction between its two
|
||||
// failure states is the whole rule.
|
||||
//
|
||||
// valid an in-flight create owns this area. Remove only what
|
||||
// that operation staged.
|
||||
// missing nothing ever claimed it. This is the ordinary orphan a
|
||||
// crashed stack deletion leaves behind, and reaping it is
|
||||
// the long-standing behavior that keeps managed data from
|
||||
// outliving its stack.
|
||||
// corrupt something claimed it and we cannot read the claim.
|
||||
// Preserve: an unexplained directory is far cheaper than a
|
||||
// wrongly deleted generation.
|
||||
const area = path.join(managedRoot, entry.name);
|
||||
try {
|
||||
await fsPromises.rm(path.join(managedRoot, entry.name), { recursive: true, force: true });
|
||||
const marker = await readStagingMarker(area);
|
||||
if (marker.state === 'corrupt') {
|
||||
console.warn(
|
||||
`[GitManifest] preserving unclaimed managed area ${sanitizeForLog(entry.name)}: its staging marker is unreadable (${marker.reason}), so ownership cannot be established`,
|
||||
);
|
||||
continue;
|
||||
}
|
||||
if (marker.state === 'missing') {
|
||||
await fsPromises.rm(area, { recursive: true, force: true });
|
||||
console.log(`[GitManifest] removed orphaned managed area ${sanitizeForLog(entry.name)}: no stack, no create, no marker claims it`);
|
||||
continue;
|
||||
}
|
||||
const outcome = await cleanupUnclaimedManagedRoot(area, marker.marker);
|
||||
console.log(`[GitManifest] unclaimed managed area ${sanitizeForLog(entry.name)}: ${outcome}`);
|
||||
} catch (e) {
|
||||
console.error(`[GitManifest] could not remove orphaned area ${sanitizeForLog(entry.name)}:`, (e as Error).message);
|
||||
console.error(`[GitManifest] could not clean unclaimed area ${sanitizeForLog(entry.name)}:`, (e as Error).message);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -9,6 +9,8 @@ import { isCleanOneShotCompletion } from '../utils/oneShotCompletion';
|
||||
import { declaredFromEffectiveModel } from '../helpers/effectiveToDeclaredCompose';
|
||||
import { parseEffectiveModel } from './preflight/effectiveModel';
|
||||
import type { HealthGateContainer, HealthGateReport } from './updateGuard/types';
|
||||
import { GitOpsStore } from './gitops/store';
|
||||
import { GitOpsTransitions } from './gitops/transitions';
|
||||
import { ComposeService, getComposeCommandTimeoutMs } from './ComposeService';
|
||||
|
||||
const POLL_INTERVAL_MS = 5_000;
|
||||
@@ -79,7 +81,7 @@ interface ActiveGate {
|
||||
/** 'stack' for the legacy post-mutation gate, 'service' for prepared gates. */
|
||||
targetScope: 'stack' | 'service';
|
||||
/** Named trigger persisted on the row. */
|
||||
trigger: 'update' | 'deploy' | 'service_update' | 'service_restore';
|
||||
trigger: 'update' | 'deploy' | 'service_update' | 'service_restore' | 'recovery';
|
||||
/** Service gates only. */
|
||||
serviceName: string | null;
|
||||
/** Single image id every primary replica must converge on (service gates). */
|
||||
@@ -224,18 +226,40 @@ export class HealthGateService {
|
||||
return HealthGateService.instance;
|
||||
}
|
||||
|
||||
/** Sweep runs left observing by a previous process, then accept begin() calls. */
|
||||
/**
|
||||
* Sweep runs left observing by a previous process, then accept begin() calls.
|
||||
*
|
||||
* Each row is finalized on its own rather than swept with one UPDATE, so the
|
||||
* revision state hears a verdict for every one of them. A reserved recovery
|
||||
* run is finalized here like any other: a reservation is only ever armed by
|
||||
* the process that made it, so one that survived a restart has no timer and
|
||||
* nothing left to observe.
|
||||
*/
|
||||
public start(): void {
|
||||
this.started = true;
|
||||
let interrupted: HealthGateRunRow[];
|
||||
try {
|
||||
const swept = DatabaseService.getInstance().markInterruptedHealthGateRuns(
|
||||
'Sencho restarted during observation', Date.now(),
|
||||
);
|
||||
if (swept > 0) {
|
||||
console.log(`[HealthGate] Marked ${swept} interrupted observation(s) as unknown`);
|
||||
}
|
||||
interrupted = DatabaseService.getInstance().listObservingHealthGateRuns();
|
||||
} catch (error) {
|
||||
console.error('[HealthGate] Startup sweep failed:', getErrorMessage(error, 'unknown'));
|
||||
return;
|
||||
}
|
||||
let finalized = 0;
|
||||
for (const run of interrupted) {
|
||||
// Per row, so one unreadable row cannot leave every later one observing
|
||||
// for ever.
|
||||
try {
|
||||
this.finalizePersistedRun(run, 'Sencho restarted during observation');
|
||||
finalized++;
|
||||
} catch (error) {
|
||||
console.error(
|
||||
'[HealthGate] Could not finalize interrupted run %s for %s:',
|
||||
run.id, sanitizeForLog(run.stack_name), getErrorMessage(error, 'unknown'),
|
||||
);
|
||||
}
|
||||
}
|
||||
if (finalized > 0) {
|
||||
console.log(`[HealthGate] Marked ${finalized} interrupted observation(s) as unknown`);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -303,11 +327,22 @@ export class HealthGateService {
|
||||
* every gated update path gets the timeline marker even when the gate
|
||||
* itself is disabled.
|
||||
*/
|
||||
/**
|
||||
* Start a stack-scoped health observation.
|
||||
*
|
||||
* `binding.deployedGenerationId` is the generation the mutation that preceded
|
||||
* this call actually deployed, or null when there was none. It is a required
|
||||
* argument rather than something this method reads from current state,
|
||||
* because a verdict is only meaningful for the generation the run watched:
|
||||
* reading it later could bind a run to whatever happens to be deployed by
|
||||
* then, and a pass would then promote a generation this run never observed.
|
||||
*/
|
||||
public beginStack(
|
||||
nodeId: number,
|
||||
stackName: string,
|
||||
trigger: 'update' | 'deploy',
|
||||
actor: string | null,
|
||||
binding: { deployedGenerationId: string | null },
|
||||
): string | null {
|
||||
// Refuses work outside the start()/stop() lifecycle so a late call during
|
||||
// shutdown cannot leave a dangling poll timer.
|
||||
@@ -343,6 +378,7 @@ export class HealthGateService {
|
||||
target_scope: 'stack',
|
||||
service_name: null,
|
||||
failure_source: null,
|
||||
deployed_generation_id: binding.deployedGenerationId,
|
||||
};
|
||||
|
||||
if (this.active.size >= MAX_CONCURRENT_GATES) {
|
||||
@@ -386,14 +422,136 @@ export class HealthGateService {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Claim a health run for a proven, bound recovery, inside the caller's open
|
||||
* transaction.
|
||||
*
|
||||
* Writes the row and links it to the recovery generation, and touches nothing
|
||||
* in memory. Committing the reservation alongside the recovery is the point:
|
||||
* a crash between the two would otherwise leave a restored workload that no
|
||||
* run was ever recorded against. Arming the timer is the caller's separate
|
||||
* step after its transaction commits.
|
||||
*
|
||||
* Idempotent on the recovery generation's `health_gate_id`, so a replayed
|
||||
* recovery reuses its run rather than opening a second one.
|
||||
*/
|
||||
public reserveRecoveryRun(args: {
|
||||
recoveryRef: string;
|
||||
nodeId: number;
|
||||
stackName: string;
|
||||
deployedGenerationId: string;
|
||||
actor: string | null;
|
||||
}): { outcome: 'reserved' | 'replayed' | 'disabled'; runId: string | null } {
|
||||
const db = DatabaseService.getInstance();
|
||||
if (!this.readSettings().enabled) return { outcome: 'disabled', runId: null };
|
||||
|
||||
const linked = db.getStackUpdateRecoveryGeneration(args.recoveryRef)?.health_gate_id ?? null;
|
||||
if (linked) return { outcome: 'replayed', runId: linked };
|
||||
|
||||
const runId = randomUUID();
|
||||
db.insertHealthGateRun({
|
||||
id: runId,
|
||||
node_id: args.nodeId,
|
||||
stack_name: args.stackName,
|
||||
trigger_action: 'recovery',
|
||||
status: 'observing',
|
||||
reason: null,
|
||||
window_seconds: this.readSettings().windowSeconds,
|
||||
containers_json: '[]',
|
||||
started_at: Date.now(),
|
||||
ended_at: null,
|
||||
created_by: args.actor,
|
||||
target_scope: 'stack',
|
||||
service_name: null,
|
||||
failure_source: null,
|
||||
deployed_generation_id: args.deployedGenerationId,
|
||||
});
|
||||
db.updateStackUpdateRecoveryGeneration(args.recoveryRef, { health_gate_id: runId });
|
||||
return { outcome: 'reserved', runId };
|
||||
}
|
||||
|
||||
/**
|
||||
* Start observing a run that was reserved in a committed transaction.
|
||||
*
|
||||
* Inserts nothing: the row already exists, and creating a second one would
|
||||
* give the same recovery two verdicts. Throws on anything unexpected so the
|
||||
* caller can finalize the reservation unknown rather than leave an observing
|
||||
* row with no timer behind it.
|
||||
*
|
||||
* Same-process only. A reservation that outlived its process is finalized by
|
||||
* `start`, never armed here.
|
||||
*/
|
||||
public armReservedRun(runId: string, nodeId: number, stackName: string): void {
|
||||
if (!this.started) throw new Error('health gate service is not started');
|
||||
|
||||
const run = DatabaseService.getInstance().getHealthGateRun(nodeId, stackName, runId);
|
||||
if (!run) throw new Error(`reserved health run ${runId} was not found`);
|
||||
if (run.status !== 'observing' || run.trigger_action !== 'recovery' || run.target_scope !== 'stack') {
|
||||
throw new Error(`health run ${runId} is not a reserved stack recovery observation`);
|
||||
}
|
||||
|
||||
const key = this.gateKey(nodeId, stackName, 'stack', null);
|
||||
if (this.active.get(key)?.runId === runId) return;
|
||||
this.supersedeGatesForStack(nodeId, stackName);
|
||||
if (this.active.size >= MAX_CONCURRENT_GATES) {
|
||||
throw new Error('too many concurrent observations');
|
||||
}
|
||||
|
||||
const gate: ActiveGate = {
|
||||
runId,
|
||||
nodeId,
|
||||
stackName,
|
||||
windowSeconds: run.window_seconds,
|
||||
startedAt: run.started_at,
|
||||
timer: null,
|
||||
expected: null,
|
||||
consecutivePollErrors: 0,
|
||||
missingLastPoll: new Set(),
|
||||
restartingLastPoll: new Set(),
|
||||
finalized: false,
|
||||
targetScope: 'stack',
|
||||
trigger: 'recovery',
|
||||
serviceName: null,
|
||||
expectedImageId: null,
|
||||
expectedReplicas: 0,
|
||||
collateralEligibleNames: new Set(),
|
||||
collateralBaselineByName: new Map(),
|
||||
roleByName: new Map(),
|
||||
declaredRestartByService: null,
|
||||
};
|
||||
this.active.set(key, gate);
|
||||
this.scheduleNextPoll(gate);
|
||||
}
|
||||
|
||||
/**
|
||||
* Write off a reservation this process could not arm.
|
||||
*
|
||||
* Public because the reservation is made inside the recovery transaction and
|
||||
* armed after it commits, so the window where arming can fail belongs to the
|
||||
* caller, not to this service.
|
||||
*/
|
||||
public abandonReservedRun(runId: string, nodeId: number, stackName: string, reason: string): void {
|
||||
try {
|
||||
const run = DatabaseService.getInstance().getHealthGateRun(nodeId, stackName, runId);
|
||||
if (!run || run.status !== 'observing') return;
|
||||
this.finalizePersistedRun(run, reason);
|
||||
} catch (error) {
|
||||
console.error(
|
||||
'[HealthGate] Could not finalize unarmed reservation %s for %s:',
|
||||
runId, sanitizeForLog(stackName), getErrorMessage(error, 'unknown'),
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/** @deprecated Prefer beginStack; retained as a one-PR alias for callers under migration. */
|
||||
public begin(
|
||||
nodeId: number,
|
||||
stackName: string,
|
||||
trigger: 'update' | 'deploy',
|
||||
actor: string | null,
|
||||
binding: { deployedGenerationId: string | null },
|
||||
): string | null {
|
||||
return this.beginStack(nodeId, stackName, trigger, actor);
|
||||
return this.beginStack(nodeId, stackName, trigger, actor, binding);
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -1018,6 +1176,60 @@ export class HealthGateService {
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Hand a finalized verdict to the GitOps state model.
|
||||
*
|
||||
* The generation is read back from the persisted run row rather than taken
|
||||
* from memory, so the verdict is attributed to what this run was recorded as
|
||||
* observing. The transition decides whether that is still promotable; this
|
||||
* method only reports.
|
||||
*
|
||||
* Never throws: a health gate is an observer, and a bookkeeping failure must
|
||||
* not change the verdict that was just written.
|
||||
*/
|
||||
private recordGitOpsHealthVerdict(
|
||||
nodeId: number,
|
||||
stackName: string,
|
||||
runId: string,
|
||||
status: 'passed' | 'failed' | 'unknown',
|
||||
): void {
|
||||
try {
|
||||
const run = DatabaseService.getInstance().getHealthGateRun(nodeId, stackName, runId);
|
||||
if (!run) return;
|
||||
const app = GitOpsStore.getInstance().getLiveDirectApplication(stackName);
|
||||
if (!app || app.lifecycle_status !== 'active') return;
|
||||
if (!GitOpsStore.getInstance().getTarget(app.id, nodeId)) return;
|
||||
GitOpsTransitions.getInstance().healthFinalized({
|
||||
applicationId: app.id,
|
||||
nodeId,
|
||||
healthRunId: runId,
|
||||
healthStatus: status,
|
||||
deployedGenerationId: run.deployed_generation_id ?? null,
|
||||
targetScope: run.target_scope,
|
||||
envelope: { operationId: runId, actor: 'system:health-gate', trigger: 'health', at: Date.now() },
|
||||
});
|
||||
} catch (error) {
|
||||
console.error(
|
||||
'[GitOps] Could not record the health verdict for %s:',
|
||||
sanitizeForLog(stackName), getErrorMessage(error, 'unknown'),
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Write an unknown verdict for a run this process is not observing.
|
||||
*
|
||||
* Covers a row a previous process left behind and a reservation this process
|
||||
* could not arm. Both are the same situation: an observing row with no timer
|
||||
* behind it, which would otherwise sit unresolved for ever.
|
||||
*/
|
||||
private finalizePersistedRun(run: HealthGateRunRow, reason: string): void {
|
||||
DatabaseService.getInstance().finalizeHealthGateRun(
|
||||
run.id, 'unknown', reason, Date.now(), run.containers_json ?? '[]', null,
|
||||
);
|
||||
this.recordGitOpsHealthVerdict(run.node_id, run.stack_name, run.id, 'unknown');
|
||||
}
|
||||
|
||||
private finalize(
|
||||
gate: ActiveGate,
|
||||
status: 'passed' | 'failed' | 'unknown',
|
||||
@@ -1047,6 +1259,7 @@ export class HealthGateService {
|
||||
DatabaseService.getInstance().finalizeHealthGateRun(
|
||||
gate.runId, status, reason, Date.now(), JSON.stringify(containers), failureSource,
|
||||
);
|
||||
this.recordGitOpsHealthVerdict(gate.nodeId, gate.stackName, gate.runId, status);
|
||||
} catch (error) {
|
||||
// The verdict is lost from the DB (the startup sweep will later rewrite
|
||||
// the row as unknown), so log everything needed to reconstruct it.
|
||||
|
||||
@@ -1342,8 +1342,8 @@ export class SchedulerService {
|
||||
|
||||
// Health observation starts immediately after Compose; registry recheck is
|
||||
// isolated so a verification failure cannot turn Compose success into a failure.
|
||||
const healthGateId = HealthGateService.getInstance().beginStack(nodeId, stackName, 'update', 'system:scheduler');
|
||||
const orchResult = lock.result;
|
||||
const healthGateId = HealthGateService.getInstance().beginStack(nodeId, stackName, 'update', 'system:scheduler', { deployedGenerationId: orchResult && orchResult.kind === 'stack_compose_done' ? orchResult.deployedGenerationId : null });
|
||||
const recoveryId = orchResult && orchResult.kind === 'stack_compose_done' ? orchResult.recoveryId : null;
|
||||
if (recoveryId) {
|
||||
const { StackUpdateRecoveryService } = await import('./StackUpdateRecoveryService');
|
||||
|
||||
@@ -47,7 +47,7 @@ export interface ServiceUpdateOptions {
|
||||
}
|
||||
|
||||
export type OrchestratorResult =
|
||||
| { kind: 'stack_compose_done'; recoveryId: string | null }
|
||||
| { kind: 'stack_compose_done'; recoveryId: string | null; deployedGenerationId: string | null }
|
||||
| {
|
||||
kind: 'service_done';
|
||||
serviceName: string;
|
||||
@@ -186,7 +186,11 @@ export class StackUpdateOrchestrator {
|
||||
ctx.stackName, options.terminalWs ?? undefined, options.atomic,
|
||||
);
|
||||
const recoveryId = updateResult?.recoveryId ?? null;
|
||||
return { kind: 'stack_compose_done', recoveryId };
|
||||
return {
|
||||
kind: 'stack_compose_done',
|
||||
recoveryId,
|
||||
deployedGenerationId: updateResult?.deployedGenerationId ?? null,
|
||||
};
|
||||
}
|
||||
|
||||
private async executeServiceUpdate(
|
||||
|
||||
@@ -34,10 +34,20 @@ import {
|
||||
type StackRecoveryServiceCapture,
|
||||
} from './recoveryServicesJson';
|
||||
import { getComposeCommandTimeoutMs } from './ComposeService';
|
||||
import type { ComposeMutationResult } from './ComposeService';
|
||||
import { HealthGateService } from './HealthGateService';
|
||||
import type { HealthRunReservation } from './gitops/transitions';
|
||||
import { assessGenerationEligibility } from './rollbackEligibility';
|
||||
import { enforcePolicyForImageRefs, type PolicyEnforcementOptions } from './PolicyEnforcement';
|
||||
import { describePolicyBlock } from '../helpers/policyGate';
|
||||
import type { GitSourceAppliedSpec } from './DatabaseService';
|
||||
import {
|
||||
captureGitOpsRecoveryBinding,
|
||||
EMPTY_GITOPS_RECOVERY_CAPTURE,
|
||||
type GitOpsRecoveryCapture,
|
||||
} from './gitops/recoveryCapture';
|
||||
import { GitOpsStore } from './gitops/store';
|
||||
import { GitOpsTransitions } from './gitops/transitions';
|
||||
import type { GitSourceManifestState } from '../types/gitProjectManifest';
|
||||
import type {
|
||||
RollbackGenerationManifest,
|
||||
@@ -568,6 +578,7 @@ export class StackUpdateRecoveryService {
|
||||
artifacts_retired: 0,
|
||||
released_at: null,
|
||||
released_by: null,
|
||||
...this.captureGitOpsBindingOrEmpty(stackName, nodeId),
|
||||
};
|
||||
DatabaseService.getInstance().insertStackUpdateRecoveryGeneration(row);
|
||||
return row;
|
||||
@@ -592,6 +603,29 @@ export class StackUpdateRecoveryService {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Read the GitOps binding for this recovery point, degrading to no binding
|
||||
* if that lookup fails.
|
||||
*
|
||||
* These three columns are advisory: they let a later restore rebind the
|
||||
* generation, artifact, and acceptance pointers instead of guessing. Rollback
|
||||
* protection itself does not depend on them, so a failure here must not abort
|
||||
* the capture and block the update it protects. The failure is logged rather
|
||||
* than swallowed, and the degraded shape is the same one legacy rows carry.
|
||||
*/
|
||||
private captureGitOpsBindingOrEmpty(stackName: string, nodeId: number): GitOpsRecoveryCapture {
|
||||
try {
|
||||
return captureGitOpsRecoveryBinding(stackName, nodeId);
|
||||
} catch (error) {
|
||||
console.warn(
|
||||
'[StackUpdateRecovery] Could not capture the GitOps binding for %s; recovery point stored without it: %s',
|
||||
sanitizeForLog(stackName),
|
||||
sanitizeForLog(getErrorMessage(error, 'unknown')),
|
||||
);
|
||||
return { ...EMPTY_GITOPS_RECOVERY_CAPTURE };
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Capture the live authored project as the current recovery generation.
|
||||
* Shares captureCandidate with deploy/update (files, holds, override), then
|
||||
@@ -892,12 +926,148 @@ export class StackUpdateRecoveryService {
|
||||
* Post-handoff compensation: restore files + pinned up, then probe before
|
||||
* reporting restored_current / immediate_verified.
|
||||
*/
|
||||
/**
|
||||
* Open a GitOps recovery for this restore, or nothing when the stack is not
|
||||
* modelled.
|
||||
*
|
||||
* Returns closures rather than ids so a terminal event cannot be recorded for
|
||||
* an operation that was never opened. Recording never fails the restore: the
|
||||
* files and containers are the real work, and the store describes it.
|
||||
*
|
||||
* The proof rule is deliberately strict. Pointers move only when the recovery
|
||||
* point named a generation, that generation still exists, and the manifest
|
||||
* that was actually restored carries the same commit and manifest version. A
|
||||
* restore we cannot tie to a generation is still a real recovery, it just has
|
||||
* nothing to bind, and the transition records it as unproven rather than
|
||||
* guessing.
|
||||
*/
|
||||
/**
|
||||
* Start observing a run the recovery transaction reserved.
|
||||
*
|
||||
* Runs after that transaction commits, because arming is in-memory work that
|
||||
* a rollback could not undo. Anything that stops the timer starting writes
|
||||
* the run off immediately: an observing row with nothing behind it would
|
||||
* otherwise report a restore as still being watched for ever.
|
||||
*/
|
||||
private armReservedRecoveryRun(
|
||||
reservation: HealthRunReservation | null,
|
||||
row: StackUpdateRecoveryGenerationRow,
|
||||
): void {
|
||||
if (!reservation?.runId || reservation.outcome === 'disabled') return;
|
||||
const gate = HealthGateService.getInstance();
|
||||
try {
|
||||
gate.armReservedRun(reservation.runId, row.node_id, row.stack_name);
|
||||
} catch (error) {
|
||||
const reason = getErrorMessage(error, 'unknown');
|
||||
console.warn(
|
||||
'[HealthGate] Could not arm the reserved recovery run %s for %s: %s',
|
||||
reservation.runId, sanitizeForLog(row.stack_name), sanitizeForLog(reason),
|
||||
);
|
||||
gate.abandonReservedRun(reservation.runId, row.node_id, row.stack_name, `could not arm: ${reason}`);
|
||||
}
|
||||
}
|
||||
|
||||
private beginGitOpsRecovery(row: StackUpdateRecoveryGenerationRow): {
|
||||
succeeded: (
|
||||
restored: RollbackGenerationManifest | null,
|
||||
binding: 'bound' | 'unbound',
|
||||
) => HealthRunReservation | null;
|
||||
failed: (failureClass: 'pre_mutation' | 'post_mutation') => void;
|
||||
} | null {
|
||||
// Returns what the write produced, or null when it could not be recorded.
|
||||
// Recording never fails the restore: the store describes what happened, it
|
||||
// does not make it happen.
|
||||
const record = <T>(what: string, write: () => T): T | null => {
|
||||
try {
|
||||
return write();
|
||||
} catch (error) {
|
||||
console.error(
|
||||
'[GitOps] Could not record recovery %s for %s (recovery %s):',
|
||||
what, sanitizeForLog(row.stack_name), row.id,
|
||||
error instanceof Error ? error.stack ?? error.message : String(error),
|
||||
);
|
||||
return null;
|
||||
}
|
||||
};
|
||||
try {
|
||||
const store = GitOpsStore.getInstance();
|
||||
const app = store.getLiveDirectApplication(row.stack_name);
|
||||
if (!app || app.lifecycle_status !== 'active') return null;
|
||||
const target = store.getTarget(app.id, row.node_id);
|
||||
if (!target || target.target_status !== 'active') return null;
|
||||
|
||||
const tx = GitOpsTransitions.getInstance();
|
||||
const envelope = {
|
||||
operationId: row.id,
|
||||
actor: 'system:recovery',
|
||||
trigger: 'recovery',
|
||||
at: Date.now(),
|
||||
};
|
||||
const capturedGenerationId = row.gitops_generation_id ?? null;
|
||||
record('start', () => tx.recoveryStarted({
|
||||
applicationId: app.id,
|
||||
nodeId: row.node_id,
|
||||
recoveryRef: row.id,
|
||||
recoveryGenerationId: capturedGenerationId,
|
||||
envelope,
|
||||
}));
|
||||
|
||||
return {
|
||||
succeeded: (restored, binding) => record('success', () => {
|
||||
const generation = capturedGenerationId
|
||||
? store.getGeneration(capturedGenerationId)
|
||||
: undefined;
|
||||
const proven = !!generation
|
||||
&& !!restored
|
||||
&& restored.git?.commitSha === generation.commit_sha
|
||||
&& restored.git?.manifestVersion === generation.manifest_version;
|
||||
const result = tx.recoverySucceeded({
|
||||
applicationId: app.id,
|
||||
nodeId: row.node_id,
|
||||
recoveryRef: row.id,
|
||||
recoveryGenerationId: capturedGenerationId,
|
||||
proven,
|
||||
gitopsBinding: binding,
|
||||
capturedArtifactSetId: row.gitops_artifact_set_id ?? null,
|
||||
capturedSourceAcceptanceRef: row.gitops_source_acceptance_ref ?? null,
|
||||
envelope: { ...envelope, at: Date.now() },
|
||||
// Claimed inside the recovery transaction so the run and the
|
||||
// pointers it describes commit together. Arming it is a separate
|
||||
// step once that transaction has landed.
|
||||
reserveHealthRun: (deployedGenerationId) => HealthGateService.getInstance().reserveRecoveryRun({
|
||||
recoveryRef: row.id,
|
||||
nodeId: row.node_id,
|
||||
stackName: row.stack_name,
|
||||
deployedGenerationId,
|
||||
actor: envelope.actor,
|
||||
}),
|
||||
});
|
||||
return result.healthReservation;
|
||||
}),
|
||||
failed: (failureClass) => record('failure', () => tx.recoveryFailed({
|
||||
applicationId: app.id,
|
||||
nodeId: row.node_id,
|
||||
recoveryRef: row.id,
|
||||
failureClass,
|
||||
envelope: { ...envelope, at: Date.now() },
|
||||
})),
|
||||
};
|
||||
} catch (error) {
|
||||
console.error(
|
||||
'[GitOps] Could not open a recovery for %s:',
|
||||
sanitizeForLog(row.stack_name),
|
||||
error instanceof Error ? error.stack ?? error.message : String(error),
|
||||
);
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
public async compensateWithCandidate(
|
||||
generationId: string,
|
||||
composeUp: (
|
||||
overridePath: string,
|
||||
invocation: RollbackInvocationRecord | null,
|
||||
) => Promise<void>,
|
||||
) => Promise<ComposeMutationResult | void>,
|
||||
policyOptions?: PolicyEnforcementOptions,
|
||||
): Promise<boolean> {
|
||||
const row = this.get(generationId);
|
||||
@@ -931,6 +1101,11 @@ export class StackUpdateRecoveryService {
|
||||
const generationContentPath =
|
||||
expectsGenerationContent(row) && row.content_path ? row.content_path : null;
|
||||
|
||||
// Open the recovery in the model before any file moves, so a crash mid
|
||||
// restore leaves a target that says what it was doing rather than one that
|
||||
// merely looks broken.
|
||||
const gitopsRecovery = this.beginGitOpsRecovery(row);
|
||||
|
||||
let filesRestored = false;
|
||||
try {
|
||||
// Eligibility (integrity + held images + security posture) before mutation.
|
||||
@@ -985,7 +1160,10 @@ export class StackUpdateRecoveryService {
|
||||
if (!row.override_path) {
|
||||
throw new Error('Recovery generation has no override path');
|
||||
}
|
||||
await composeUp(row.override_path, restoredInvocation);
|
||||
// Only a callback that reports a Compose mutation licenses the deployed
|
||||
// pointer. A restore driven some other way resolves the same, and binding
|
||||
// on that would claim a workload nobody launched.
|
||||
const composeResult = await composeUp(row.override_path, restoredInvocation);
|
||||
const probeOk = await this.probeRecoveredStack(
|
||||
row.node_id,
|
||||
row.stack_name,
|
||||
@@ -1023,9 +1201,17 @@ export class StackUpdateRecoveryService {
|
||||
is_current: 1,
|
||||
artifact_expires_at: null,
|
||||
});
|
||||
const reservation = gitopsRecovery?.succeeded(
|
||||
restoredManifest ?? null,
|
||||
composeResult?.mutatedByCompose ? 'bound' : 'unbound',
|
||||
) ?? null;
|
||||
this.armReservedRecoveryRun(reservation, row);
|
||||
return true;
|
||||
} catch (error) {
|
||||
const code = (error as { code?: string }).code;
|
||||
// Classified by whether the files had already moved. Only a failure
|
||||
// before that leaves the previous workload provably intact.
|
||||
gitopsRecovery?.failed(filesRestored ? 'post_mutation' : 'pre_mutation');
|
||||
if (filesRestored && generationContentPath) {
|
||||
try {
|
||||
await RollbackGenerationStore.reconcileInterruptedRestore(
|
||||
|
||||
@@ -163,7 +163,7 @@ export class WebhookService {
|
||||
atomic,
|
||||
{ source: 'webhook', actor: 'system:webhook' },
|
||||
);
|
||||
const healthGateId = HealthGateService.getInstance().beginStack(nodeId, stackName, 'deploy', 'system:webhook');
|
||||
const healthGateId = HealthGateService.getInstance().beginStack(nodeId, stackName, 'deploy', 'system:webhook', { deployedGenerationId: deployResult.deployedGenerationId });
|
||||
if (deployResult.recoveryId) {
|
||||
const { StackUpdateRecoveryService } = await import('./StackUpdateRecoveryService');
|
||||
StackUpdateRecoveryService.getInstance().linkGateOrRetain(deployResult.recoveryId, healthGateId);
|
||||
@@ -189,7 +189,7 @@ export class WebhookService {
|
||||
{ nodeId, stackName, target: { scope: 'stack' }, trigger: 'webhook', actor: 'system:webhook' },
|
||||
{ atomic: atomic ?? false, terminalWs: null },
|
||||
);
|
||||
const healthGateId = HealthGateService.getInstance().beginStack(nodeId, stackName, 'update', 'system:webhook');
|
||||
const healthGateId = HealthGateService.getInstance().beginStack(nodeId, stackName, 'update', 'system:webhook', { deployedGenerationId: orchResult.kind === 'stack_compose_done' ? orchResult.deployedGenerationId : null });
|
||||
const recoveryId = orchResult.kind === 'stack_compose_done' ? orchResult.recoveryId : null;
|
||||
if (recoveryId) {
|
||||
const { StackUpdateRecoveryService } = await import('./StackUpdateRecoveryService');
|
||||
|
||||
@@ -337,6 +337,14 @@ function projectActions(
|
||||
seen.add(node.id);
|
||||
};
|
||||
|
||||
// A severed canonical target is invisible to every automatic action. The
|
||||
// decision arrays already refuse it, and marking it seen here keeps the
|
||||
// deployment-status fallbacks below from resurrecting a retry behind
|
||||
// the model's back.
|
||||
for (const nodeId of decision.severedNodeIds) {
|
||||
seen.add(nodeId);
|
||||
}
|
||||
|
||||
// Status-precedence pass: in-flight, name conflict, and clear_* must win over
|
||||
// decision.withdraw / evictBlocked so Confirm never authorizes mid-flight mutates
|
||||
// and never-deployed guards stay on the remove-only clear_stale path.
|
||||
|
||||
@@ -0,0 +1,245 @@
|
||||
/**
|
||||
* Blueprint deployment-state writes, recorded by what caused them.
|
||||
*
|
||||
* Every production write to a Blueprint deployment row comes through here, so
|
||||
* the revision state hears one event per real change rather than one per call.
|
||||
* The cause is passed in rather than inferred from the resulting status,
|
||||
* because several causes land on the same status: a deploy that failed and a
|
||||
* withdraw that failed both read `failed`, and telling them apart afterwards is
|
||||
* impossible.
|
||||
*
|
||||
* Preview cleanup deliberately does not come through here. It reverses a
|
||||
* projection nobody deployed, so recording it would report removals that never
|
||||
* happened.
|
||||
*/
|
||||
import { DatabaseService, type BlueprintDeployment } from '../DatabaseService';
|
||||
import { GitOpsStore, emptyTargetRow } from './store';
|
||||
import { GitOpsTransitions, GitOpsTransitionError } from './transitions';
|
||||
import { envelopeFor, recordableApplication } from './blueprintProducers';
|
||||
|
||||
/** Why a deployment row moved. */
|
||||
export type BlueprintDeploymentCause =
|
||||
| 'deploy_start'
|
||||
| 'deploy_ack'
|
||||
| 'deploy_fail'
|
||||
| 'name_conflict'
|
||||
| 'withdraw_start'
|
||||
| 'withdraw_success'
|
||||
| 'withdraw_fail'
|
||||
| 'withdraw_name_conflict'
|
||||
| 'await_state_review'
|
||||
| 'await_evict_confirm'
|
||||
| 'drift_observed'
|
||||
| 'drift_enforce_start';
|
||||
|
||||
/** Causes that only observe, and must never acknowledge or mint anything. */
|
||||
const OBSERVATION_STAGE = {
|
||||
await_state_review: 'blueprint_state_review',
|
||||
await_evict_confirm: 'blueprint_evict_blocked',
|
||||
drift_observed: 'blueprint_drifted',
|
||||
drift_enforce_start: 'blueprint_correcting',
|
||||
} as const;
|
||||
|
||||
type ObservationCause = keyof typeof OBSERVATION_STAGE;
|
||||
|
||||
/**
|
||||
* Narrows to the observation causes, so the stage lookup below reads as a fact
|
||||
* the compiler derives rather than one an assertion claims.
|
||||
*/
|
||||
function isObservation(cause: BlueprintDeploymentCause): cause is ObservationCause {
|
||||
return cause in OBSERVATION_STAGE;
|
||||
}
|
||||
|
||||
type DeploymentFields = Omit<Parameters<DatabaseService['upsertDeployment']>[0], 'blueprint_id' | 'node_id'>;
|
||||
|
||||
/**
|
||||
* Write a deployment row and record what caused it.
|
||||
*
|
||||
* The write happens either way. Recording is skipped when the effective status
|
||||
* did not move, so a reconciler tick that re-asserts a state it already
|
||||
* reported does not append a second event describing the same fact.
|
||||
*/
|
||||
export function commitBlueprintDeploymentCause(
|
||||
cause: BlueprintDeploymentCause,
|
||||
blueprintId: number,
|
||||
nodeId: number,
|
||||
fields: DeploymentFields,
|
||||
actor: string | null,
|
||||
): BlueprintDeployment {
|
||||
const db = DatabaseService.getInstance();
|
||||
|
||||
return db.getDb().transaction(() => {
|
||||
const previous = db.getDeployment(blueprintId, nodeId);
|
||||
const deployment = db.upsertDeployment({ blueprint_id: blueprintId, node_id: nodeId, ...fields });
|
||||
const statusMoved = previous?.status !== deployment.status;
|
||||
|
||||
try {
|
||||
record(cause, blueprintId, nodeId, statusMoved, actor);
|
||||
} catch (error) {
|
||||
// The deployment happened whatever the record says. Failing the write
|
||||
// here would turn a bookkeeping problem into a stuck rollout.
|
||||
//
|
||||
// A rejection is louder than an infrastructure error on purpose: it means
|
||||
// the model refused this as invalid, and a target that keeps refusing
|
||||
// holds its active slot and stops recording anything further.
|
||||
const rejected = error instanceof GitOpsTransitionError;
|
||||
console.error(
|
||||
'[GitOps] %s recording blueprint %s for blueprint %d on node %d:',
|
||||
rejected ? 'Rejected' : 'Could not record', cause, blueprintId, nodeId,
|
||||
error instanceof Error ? error.stack ?? error.message : String(error),
|
||||
);
|
||||
}
|
||||
return deployment;
|
||||
})();
|
||||
}
|
||||
|
||||
function record(
|
||||
cause: BlueprintDeploymentCause,
|
||||
blueprintId: number,
|
||||
nodeId: number,
|
||||
statusMoved: boolean,
|
||||
actor: string | null,
|
||||
): void {
|
||||
const store = GitOpsStore.getInstance();
|
||||
const tx = GitOpsTransitions.getInstance();
|
||||
const app = store.getLiveBlueprintApplication(blueprintId);
|
||||
// A Blueprint that predates the model has nothing to record against.
|
||||
if (!recordableApplication(app)) return;
|
||||
|
||||
const envelope = envelopeFor(actor, `blueprint_${cause}`);
|
||||
|
||||
if (isObservation(cause)) {
|
||||
// Observations are the only causes the status guard applies to. A start
|
||||
// writes the identity terminals are matched against, so suppressing one
|
||||
// because the row already read `deploying` would let a later
|
||||
// acknowledgement answer a request that had been superseded.
|
||||
if (!statusMoved) return;
|
||||
// A stateful first placement is held for review before anything deploys,
|
||||
// so there is no target yet and nothing to observe against. Creating it
|
||||
// here is the same first-contact write `deploy_start` does below: the
|
||||
// node has been asked to hold this Blueprint, which is exactly what the
|
||||
// observation is about. Any other cause arriving without a target is
|
||||
// dropped, which is also what happens to a drift or evict report for a
|
||||
// Blueprint that migration brought in: migration records the application,
|
||||
// its intent and its candidate, but no targets, so a fleet that predates
|
||||
// this model reports nothing here until its next deploy creates one.
|
||||
const firstPlacement = !store.getTarget(app.id, nodeId);
|
||||
if (firstPlacement && cause !== 'await_state_review') return;
|
||||
const stage = OBSERVATION_STAGE[cause];
|
||||
// Both writes in one transaction so they succeed or fail together. The
|
||||
// observation refuses a tombstoned target, and it runs in its own
|
||||
// savepoint, so creating the target outside this would leave an active
|
||||
// target with no generation, no stage and no history behind a refusal: a
|
||||
// placement relationship the model never established, which the delete
|
||||
// path would later tombstone as if it were real.
|
||||
DatabaseService.getInstance().getDb().transaction(() => {
|
||||
if (firstPlacement) store.upsertTarget(emptyTargetRow(app.id, nodeId, envelope.at));
|
||||
tx.blueprintObservation({ applicationId: app.id, nodeId, stage, envelope });
|
||||
})();
|
||||
return;
|
||||
}
|
||||
|
||||
if (cause === 'deploy_start') {
|
||||
// First deploy to this node: the target is created here, because a
|
||||
// Blueprint application has no targets until something is sent somewhere.
|
||||
if (!store.getTarget(app.id, nodeId)) {
|
||||
store.upsertTarget(emptyTargetRow(app.id, nodeId, envelope.at));
|
||||
}
|
||||
if (!app.intent_revision_id) return;
|
||||
tx.blueprintDeployStarted({
|
||||
applicationId: app.id,
|
||||
nodeId,
|
||||
intentRevisionId: app.intent_revision_id,
|
||||
rolloutCandidateId: app.rollout_candidate_id,
|
||||
envelope,
|
||||
});
|
||||
return;
|
||||
}
|
||||
|
||||
const target = store.getTarget(app.id, nodeId);
|
||||
if (!target) return;
|
||||
|
||||
// Terminals answer the request the target says it was given, not whatever the
|
||||
// Blueprint currently wants. An ack matched against the current intent would
|
||||
// accept work for a revision this node was never sent.
|
||||
const requested = target.active_operation_stage !== null
|
||||
? target.active_intent_revision_id
|
||||
: target.interruption_intent_revision_id;
|
||||
|
||||
switch (cause) {
|
||||
case 'deploy_ack':
|
||||
if (!requested) return;
|
||||
tx.blueprintAckRecorded({
|
||||
applicationId: app.id,
|
||||
nodeId,
|
||||
intentRevisionId: requested,
|
||||
rolloutCandidateId: target.active_operation_stage !== null
|
||||
? target.active_rollout_candidate_id
|
||||
: target.interruption_rollout_candidate_id,
|
||||
legacyAppliedRevision: null,
|
||||
envelope,
|
||||
});
|
||||
return;
|
||||
case 'deploy_fail':
|
||||
case 'name_conflict':
|
||||
tx.blueprintDeployFailed({
|
||||
applicationId: app.id,
|
||||
nodeId,
|
||||
failureClass: cause === 'name_conflict' ? 'name_conflict' : 'post_mutation',
|
||||
envelope,
|
||||
});
|
||||
return;
|
||||
case 'withdraw_start':
|
||||
if (!target.intent_revision_id) return;
|
||||
tx.blueprintWithdrawStarted({
|
||||
applicationId: app.id,
|
||||
nodeId,
|
||||
// The intent being removed is the one this node acknowledged, never a
|
||||
// later replacement.
|
||||
intentRevisionId: target.intent_revision_id,
|
||||
envelope,
|
||||
});
|
||||
return;
|
||||
case 'withdraw_success':
|
||||
if (!requested) return;
|
||||
tx.blueprintWithdrawn({ applicationId: app.id, nodeId, intentRevisionId: requested, envelope });
|
||||
return;
|
||||
case 'withdraw_fail':
|
||||
case 'withdraw_name_conflict':
|
||||
tx.blueprintWithdrawFailed({
|
||||
applicationId: app.id,
|
||||
nodeId,
|
||||
failureClass: cause === 'withdraw_name_conflict' ? 'name_conflict' : 'post_mutation',
|
||||
envelope,
|
||||
});
|
||||
return;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Record a withdraw that removed the deployment row entirely.
|
||||
*
|
||||
* Split from the cause above because the row is deleted rather than updated, so
|
||||
* there is no status to compare.
|
||||
*/
|
||||
export function commitBlueprintDeploymentRemoved(
|
||||
blueprintId: number,
|
||||
nodeId: number,
|
||||
actor: string | null,
|
||||
): void {
|
||||
const db = DatabaseService.getInstance();
|
||||
db.getDb().transaction(() => {
|
||||
const existed = db.getDeployment(blueprintId, nodeId) !== undefined;
|
||||
db.deleteDeployment(blueprintId, nodeId);
|
||||
if (!existed) return;
|
||||
try {
|
||||
record('withdraw_success', blueprintId, nodeId, true, actor);
|
||||
} catch (error) {
|
||||
console.error(
|
||||
'[GitOps] Could not record blueprint withdrawal for blueprint %d on node %d:',
|
||||
blueprintId, nodeId,
|
||||
error instanceof Error ? error.stack ?? error.message : String(error),
|
||||
);
|
||||
}
|
||||
})();
|
||||
}
|
||||
@@ -0,0 +1,401 @@
|
||||
/**
|
||||
* The Blueprint operations that write to the revision state.
|
||||
*
|
||||
* Each one wraps the Blueprint source write and its GitOps rows in a single
|
||||
* transaction, so an operator never sees a Blueprint that exists with nothing
|
||||
* describing what it means, or an intent for a Blueprint that failed to save.
|
||||
*
|
||||
* Desired node ids arrive as an argument rather than being computed here. The
|
||||
* reconciler that knows how to compute them reaches this layer, so importing it
|
||||
* back would close a module cycle, and a cycle in this package has already
|
||||
* produced one silent defect on this branch.
|
||||
*/
|
||||
import { createHash, randomUUID } from 'crypto';
|
||||
import { DatabaseService, type Blueprint, type BlueprintSelector } from '../DatabaseService';
|
||||
import { GitOpsStore } from './store';
|
||||
import { GitOpsTransitions, type EventEnvelope } from './transitions';
|
||||
import type { GitOpsApplicationRow, GitOpsIntentRevisionRow, GitOpsRolloutCandidateRow } from './types';
|
||||
|
||||
/** What an operator changed, which decides whether a new intent is minted. */
|
||||
export type BlueprintChangeKind = 'operational' | 'metadata_only' | 'none';
|
||||
|
||||
function sha256(value: string): string {
|
||||
return createHash('sha256').update(value).digest('hex');
|
||||
}
|
||||
|
||||
export function envelopeFor(actor: string | null, trigger: string): EventEnvelope {
|
||||
return { operationId: randomUUID(), actor: actor ?? 'system:blueprint', trigger, at: Date.now() };
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether an application can be recorded against.
|
||||
*
|
||||
* `getLiveBlueprintApplication` answers with `active` or `creating`, because
|
||||
* its other callers ask "does this Blueprint already hold the live slot",
|
||||
* where a half-built row counts. The transitions that mint intents and
|
||||
* candidates accept only `active` and reject anything else by throwing, and
|
||||
* they run inside the caller's transaction, so a `creating` row reaching one
|
||||
* would fail the operator's edit and roll back the Blueprint write with it.
|
||||
*
|
||||
* **No current path produces that row.** Every Blueprint-mode application is
|
||||
* inserted through `blankInlineApplication`, which hardcodes `active`, and
|
||||
* `creating` is reachable only in `direct` mode. So this narrowing excludes
|
||||
* nothing today and is deliberately defensive: it exists because the getter's
|
||||
* slot-check semantics and the transitions' stricter requirement have already
|
||||
* drifted apart once, and the Git-backed `blueprint` mode is what will insert
|
||||
* a Blueprint application that is still being created. Narrowing here rather
|
||||
* than in the getter keeps the slot check correct for its own callers.
|
||||
*/
|
||||
export function recordableApplication(
|
||||
app: GitOpsApplicationRow | undefined,
|
||||
): app is GitOpsApplicationRow {
|
||||
return !!app && app.lifecycle_status === 'active';
|
||||
}
|
||||
|
||||
export type BlueprintUpdates = Parameters<DatabaseService['updateBlueprint']>[1];
|
||||
|
||||
/**
|
||||
* Which fields changed, in the only terms that matter here.
|
||||
*
|
||||
* Operational fields describe what gets deployed and where, so changing one
|
||||
* makes every existing acknowledgement stale. Description and classification
|
||||
* describe the Blueprint to a reader and change nothing a node runs, so they
|
||||
* must not mint an intent: a fresh identity would invalidate acknowledgements
|
||||
* that are still accurate.
|
||||
*/
|
||||
export function classifyBlueprintChange(
|
||||
before: Blueprint,
|
||||
updates: BlueprintUpdates,
|
||||
): BlueprintChangeKind {
|
||||
const changedKeys = changedBlueprintKeys(before, updates);
|
||||
if (OPERATIONAL_KEYS.some((key) => changedKeys.has(key))) return 'operational';
|
||||
return changedKeys.size > 0 ? 'metadata_only' : 'none';
|
||||
}
|
||||
|
||||
const OPERATIONAL_KEYS = ['name', 'compose_content', 'selector', 'drift_mode', 'enabled'] as const;
|
||||
|
||||
/**
|
||||
* A selector compared by value rather than by how it was written.
|
||||
*
|
||||
* The lists inside carry request order, so a multi-select that emits click
|
||||
* order would otherwise read as a placement change and invalidate every
|
||||
* acknowledgement over a reorder that selects the same nodes.
|
||||
*/
|
||||
function canonicalSelector(selector: BlueprintSelector): string {
|
||||
return selector.type === 'nodes'
|
||||
? JSON.stringify({ type: 'nodes', ids: [...selector.ids].sort((a, b) => a - b) })
|
||||
: JSON.stringify({
|
||||
type: 'labels',
|
||||
any: [...selector.any].sort(),
|
||||
all: [...selector.all].sort(),
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* The keys whose submitted value actually differs from what is stored.
|
||||
*
|
||||
* The editor submits every field on every save, and the source layer decides
|
||||
* what to invalidate from which keys are *present*. Comparing values here and
|
||||
* handing that layer the untouched payload made the two disagree: a
|
||||
* description edit bumped the revision and cleared the approval while this
|
||||
* layer classified it as metadata and minted nothing, leaving the current
|
||||
* intent describing a revision that no longer existed.
|
||||
*/
|
||||
function changedBlueprintKeys(
|
||||
before: Blueprint,
|
||||
updates: BlueprintUpdates,
|
||||
): Set<keyof BlueprintUpdates> {
|
||||
const changed = new Set<keyof BlueprintUpdates>();
|
||||
const differs = (key: keyof BlueprintUpdates): boolean => {
|
||||
const next = updates[key];
|
||||
if (next === undefined) return false;
|
||||
if (key === 'selector') {
|
||||
return canonicalSelector(next as BlueprintSelector) !== canonicalSelector(before.selector);
|
||||
}
|
||||
if (key === 'classification_reasons') {
|
||||
return JSON.stringify(next) !== JSON.stringify(before.classification_reasons);
|
||||
}
|
||||
return next !== before[key as keyof Blueprint];
|
||||
};
|
||||
for (const key of ['name', 'compose_content', 'selector', 'drift_mode', 'enabled', 'description', 'classification', 'classification_reasons'] as const) {
|
||||
if (differs(key)) changed.add(key);
|
||||
}
|
||||
return changed;
|
||||
}
|
||||
|
||||
/** Only the keys that changed, so presence and difference mean the same thing. */
|
||||
function prunedUpdates(before: Blueprint, updates: BlueprintUpdates): BlueprintUpdates {
|
||||
const changedKeys = changedBlueprintKeys(before, updates);
|
||||
const pruned: BlueprintUpdates = {};
|
||||
for (const key of changedKeys) {
|
||||
Object.assign(pruned, { [key]: updates[key] });
|
||||
}
|
||||
// The revision bump rides on the compose content. Pruned out, it would still
|
||||
// advance a revision no intent describes.
|
||||
if (changedKeys.has('compose_content') && updates.bumpRevision) pruned.bumpRevision = true;
|
||||
return pruned;
|
||||
}
|
||||
|
||||
export function intentRowFor(
|
||||
applicationId: string,
|
||||
blueprint: Blueprint,
|
||||
operationId: string,
|
||||
actor: string | null,
|
||||
at: number,
|
||||
): GitOpsIntentRevisionRow {
|
||||
return {
|
||||
id: randomUUID(),
|
||||
application_id: applicationId,
|
||||
blueprint_id: blueprint.id,
|
||||
compose_content_sha256: sha256(blueprint.compose_content),
|
||||
blueprint_revision: blueprint.revision,
|
||||
deploy_stack_name: blueprint.name,
|
||||
selector_json: JSON.stringify(blueprint.selector),
|
||||
pinned_node_id: blueprint.pinned_node_id,
|
||||
cordon_implications_json: JSON.stringify({ pinnedOverridesCordon: blueprint.pinned_node_id !== null }),
|
||||
rollout_strategy_json: JSON.stringify({ driftMode: blueprint.drift_mode, enabled: blueprint.enabled }),
|
||||
runtime_drift_policy: blueprint.drift_mode,
|
||||
stateful_policy_json: null,
|
||||
health_failure_rollback_policy_json: null,
|
||||
operation_id: operationId,
|
||||
actor,
|
||||
created_at: at,
|
||||
};
|
||||
}
|
||||
|
||||
export function candidateRowFor(
|
||||
applicationId: string,
|
||||
intent: GitOpsIntentRevisionRow,
|
||||
desiredNodeIds: number[],
|
||||
provenance: GitOpsRolloutCandidateRow['provenance'],
|
||||
operationId: string,
|
||||
at: number,
|
||||
): GitOpsRolloutCandidateRow {
|
||||
return {
|
||||
id: randomUUID(),
|
||||
application_id: applicationId,
|
||||
intent_revision_id: intent.id,
|
||||
compose_content_sha256: intent.compose_content_sha256,
|
||||
accepted_generation_id: null,
|
||||
artifact_set_id: null,
|
||||
// Canonical: the required set is compared across revisions, so an order
|
||||
// change must not read as a placement change.
|
||||
required_targets_json: JSON.stringify({ nodeIds: [...desiredNodeIds].sort((a, b) => a - b) }),
|
||||
authoritative: 1,
|
||||
provenance,
|
||||
operation_id: operationId,
|
||||
created_at: at,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Record a new Blueprint: the source row, its application, and the first
|
||||
* intent and candidate describing what it currently asks for.
|
||||
*/
|
||||
export function commitBlueprintCreate(
|
||||
input: Parameters<DatabaseService['createBlueprint']>[0],
|
||||
desiredNodeIdsFor: (blueprint: Blueprint) => number[],
|
||||
): Blueprint {
|
||||
const db = DatabaseService.getInstance();
|
||||
const tx = GitOpsTransitions.getInstance();
|
||||
|
||||
return db.getDb().transaction(() => {
|
||||
const blueprint = db.createBlueprint(input);
|
||||
const envelope = envelopeFor(input.created_by, 'blueprint_create');
|
||||
const applicationId = randomUUID();
|
||||
|
||||
tx.activateInlineBlueprint({
|
||||
application: blankInlineApplication(applicationId, blueprint.id, envelope.at),
|
||||
envelope,
|
||||
});
|
||||
|
||||
const intent = intentRowFor(applicationId, blueprint, envelope.operationId, input.created_by, envelope.at);
|
||||
tx.intentRevised({ applicationId, intent, envelope });
|
||||
tx.rolloutCandidateOpened({
|
||||
applicationId,
|
||||
candidate: candidateRowFor(
|
||||
applicationId, intent, desiredNodeIdsFor(blueprint), 'intent_change', envelope.operationId, envelope.at,
|
||||
),
|
||||
envelope,
|
||||
});
|
||||
return blueprint;
|
||||
})();
|
||||
}
|
||||
|
||||
/**
|
||||
* Record an edit to a Blueprint.
|
||||
*
|
||||
* A change that alters nothing writes nothing at all, and a change that only
|
||||
* alters how the Blueprint reads updates the source row alone. Only an
|
||||
* operational change mints a new intent, because only that makes what the
|
||||
* fleet already acknowledged out of date.
|
||||
*/
|
||||
export function commitBlueprintUpdate(
|
||||
blueprintId: number,
|
||||
updates: BlueprintUpdates,
|
||||
actor: string | null,
|
||||
desiredNodeIdsFor: (blueprint: Blueprint) => number[],
|
||||
): { blueprint: Blueprint | undefined; change: BlueprintChangeKind } {
|
||||
const db = DatabaseService.getInstance();
|
||||
const store = GitOpsStore.getInstance();
|
||||
const tx = GitOpsTransitions.getInstance();
|
||||
|
||||
return db.getDb().transaction(() => {
|
||||
const before = db.getBlueprint(blueprintId);
|
||||
if (!before) return { blueprint: undefined, change: 'none' as BlueprintChangeKind };
|
||||
|
||||
const change = classifyBlueprintChange(before, updates);
|
||||
if (change === 'none') return { blueprint: before, change };
|
||||
|
||||
const blueprint = db.updateBlueprint(blueprintId, prunedUpdates(before, updates));
|
||||
if (!blueprint || change === 'metadata_only') return { blueprint, change };
|
||||
|
||||
const app = store.getLiveBlueprintApplication(blueprintId);
|
||||
// A Blueprint that predates the model has no application yet. Migration
|
||||
// brings it in; inventing one here would claim a first intent for a
|
||||
// Blueprint whose deployments nobody has reconciled.
|
||||
if (!recordableApplication(app)) return { blueprint, change };
|
||||
|
||||
const envelope = envelopeFor(actor, 'blueprint_update');
|
||||
const intent = intentRowFor(app.id, blueprint, envelope.operationId, actor, envelope.at);
|
||||
tx.intentRevised({ applicationId: app.id, intent, envelope });
|
||||
tx.rolloutCandidateOpened({
|
||||
applicationId: app.id,
|
||||
candidate: candidateRowFor(
|
||||
app.id, intent, desiredNodeIdsFor(blueprint), 'intent_change', envelope.operationId, envelope.at,
|
||||
),
|
||||
envelope,
|
||||
});
|
||||
return { blueprint, change };
|
||||
})();
|
||||
}
|
||||
|
||||
/**
|
||||
* Record a pin change.
|
||||
*
|
||||
* Pinning moves where a Blueprint is allowed to run, so it revises placement
|
||||
* the same way a selector edit does. Re-pinning to the node already pinned
|
||||
* changes nothing and writes nothing.
|
||||
*/
|
||||
export function commitBlueprintPin(
|
||||
blueprintId: number,
|
||||
nodeId: number | null,
|
||||
actor: string | null,
|
||||
desiredNodeIdsFor: (blueprint: Blueprint) => number[],
|
||||
): { blueprint: Blueprint | undefined; changed: boolean } {
|
||||
const db = DatabaseService.getInstance();
|
||||
const store = GitOpsStore.getInstance();
|
||||
const tx = GitOpsTransitions.getInstance();
|
||||
|
||||
return db.getDb().transaction(() => {
|
||||
const before = db.getBlueprint(blueprintId);
|
||||
if (!before) return { blueprint: undefined, changed: false };
|
||||
if (before.pinned_node_id === nodeId) return { blueprint: before, changed: false };
|
||||
|
||||
const blueprint = db.setBlueprintPinnedNode(blueprintId, nodeId);
|
||||
if (!blueprint) return { blueprint: undefined, changed: false };
|
||||
|
||||
const app = store.getLiveBlueprintApplication(blueprintId);
|
||||
if (!recordableApplication(app)) return { blueprint, changed: true };
|
||||
|
||||
const envelope = envelopeFor(actor, 'blueprint_pin');
|
||||
const intent = intentRowFor(app.id, blueprint, envelope.operationId, actor, envelope.at);
|
||||
tx.intentRevised({ applicationId: app.id, intent, envelope });
|
||||
tx.rolloutCandidateOpened({
|
||||
applicationId: app.id,
|
||||
candidate: candidateRowFor(
|
||||
app.id, intent, desiredNodeIdsFor(blueprint), 'roster_change', envelope.operationId, envelope.at,
|
||||
),
|
||||
envelope,
|
||||
});
|
||||
return { blueprint, changed: true };
|
||||
})();
|
||||
}
|
||||
|
||||
/**
|
||||
* Retire a Blueprint's application after its deployments have been withdrawn.
|
||||
*
|
||||
* Tombstones only. A deleted Blueprint must stop claiming its live-application
|
||||
* slot, or the name cannot be used again, but nothing here withdraws anything:
|
||||
* the caller has already done that, and doing it twice would report removals
|
||||
* that never happened.
|
||||
*/
|
||||
export function commitBlueprintDelete(blueprintId: number, actor: string | null): boolean {
|
||||
const db = DatabaseService.getInstance();
|
||||
const store = GitOpsStore.getInstance();
|
||||
const tx = GitOpsTransitions.getInstance();
|
||||
|
||||
return db.getDb().transaction(() => {
|
||||
const app = store.getLiveBlueprintApplication(blueprintId);
|
||||
const removed = db.deleteBlueprint(blueprintId);
|
||||
if (!removed || !app) return removed;
|
||||
|
||||
const envelope = envelopeFor(actor, 'blueprint_delete');
|
||||
for (const target of store.listTargets(app.id)) {
|
||||
if (target.target_status !== 'active') continue;
|
||||
tx.targetTombstoned(app.id, target.node_id, envelope);
|
||||
}
|
||||
tx.applicationTombstoned(app.id, 'deleted', envelope);
|
||||
return removed;
|
||||
})();
|
||||
}
|
||||
|
||||
/** A Blueprint application before anything has been asked of it. */
|
||||
export function blankInlineApplication(id: string, blueprintId: number, at: number) {
|
||||
return {
|
||||
id,
|
||||
lifecycle_key: `blueprint:${blueprintId}`,
|
||||
lifecycle_status: 'active' as const,
|
||||
target_mode: 'inline_blueprint' as const,
|
||||
stack_name: null,
|
||||
blueprint_id: blueprintId,
|
||||
configured_repo_url: null,
|
||||
repo_identity_json: null,
|
||||
configured_ref: null,
|
||||
compose_paths_json: null,
|
||||
context_dir: null,
|
||||
sync_env: 0,
|
||||
env_path: null,
|
||||
materialization_fingerprint: null,
|
||||
desired_commit_sha: null,
|
||||
fetched_commit_sha: null,
|
||||
candidate_generation_id: null,
|
||||
accepted_generation_id: null,
|
||||
candidate_plan_blocked: 0,
|
||||
review_required: 0,
|
||||
artifact_set_id: null,
|
||||
latest_artifact_set_id: null,
|
||||
intent_revision_id: null,
|
||||
rollout_candidate_id: null,
|
||||
rollout_generation_id: null,
|
||||
source_acceptance_ref: null,
|
||||
placement_approval_ref: null,
|
||||
rollout_authorization_ref: null,
|
||||
legacy_combined_approval_ref: null,
|
||||
preflight_fingerprint: null,
|
||||
latest_operation_id: null,
|
||||
active_operation_id: null,
|
||||
active_operation_stage: null,
|
||||
active_operation_at: null,
|
||||
active_generation_id: null,
|
||||
pause_at: null,
|
||||
pause_reason: null,
|
||||
partial_json: null,
|
||||
failure_stage: null,
|
||||
failure_class: null,
|
||||
failure_at: null,
|
||||
retry_at: null,
|
||||
retry_count: 0,
|
||||
suspended_at: null,
|
||||
recovery_ref: null,
|
||||
recovery_phase: null,
|
||||
interruption_stage: null,
|
||||
interruption_at: null,
|
||||
interruption_operation_id: null,
|
||||
interruption_generation_id: null,
|
||||
evidence_fresh_at: null,
|
||||
evidence_limitations_json: null,
|
||||
created_at: at,
|
||||
updated_at: at,
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,171 @@
|
||||
import fs from 'fs/promises';
|
||||
import path from 'path';
|
||||
import { isPathWithinBase } from '../../utils/validation';
|
||||
import { sanitizeForLog } from '../../utils/safeLog';
|
||||
import { deleteStagingMarker, validateCandidateRelPath } from './createStagingMarker';
|
||||
import { isRealPathAtManagedLocation, managedAreaBase } from './managedPaths';
|
||||
|
||||
/**
|
||||
* Appended to positional-containment refusals so an operator staring at one
|
||||
* knows there is no override to flip: something under the managed area is not
|
||||
* where its own name says it is, and the fix is repairing or removing that
|
||||
* directory, not relocating the data directory (a whole-area move resolves
|
||||
* cleanly and never trips this).
|
||||
*/
|
||||
const RELOCATION_REMEDIATION = 'repair or remove the redirected directory under DATA_DIR/git-managed, then retry';
|
||||
|
||||
export type OperationOwnedCleanup = {
|
||||
/** Absolute path of the stack's managed root. */
|
||||
stackManagedRoot: string;
|
||||
/** Candidate directory this operation staged, relative to the managed root. */
|
||||
candidateRelPath: string | null;
|
||||
/** Applied directory this operation promoted, relative to the managed root. */
|
||||
appliedRelPath?: string | null;
|
||||
/**
|
||||
* True only when this operation created the managed root itself. Nothing
|
||||
* else authorizes deleting the whole root, because a root that predated the
|
||||
* operation may hold another generation's retained content.
|
||||
*/
|
||||
ownsManagedRoot: boolean;
|
||||
};
|
||||
|
||||
/**
|
||||
* Remove only what one create operation put on disk.
|
||||
*
|
||||
* The rule this enforces is that a failed create must never cost an unrelated
|
||||
* generation its files. When the operation created the managed root, the whole
|
||||
* root is ours and goes. Otherwise the blast radius is exactly the directories
|
||||
* the operation staged, resolved and containment-checked against the root
|
||||
* before anything is removed.
|
||||
*
|
||||
* Throws on the first failed *directory* removal rather than continuing, because
|
||||
* the caller uses success here as the precondition for tombstoning: a partially
|
||||
* cleaned area must keep its checkpoint so the next boot can retry.
|
||||
*
|
||||
* The staging marker is reported rather than thrown. Once the directories are
|
||||
* gone the create is torn down, and a marker file nobody could delete is the
|
||||
* same condition the settled path already treats as non-fatal. Throwing here
|
||||
* would make one unlink failure the difference between an instance that boots
|
||||
* and one that does not.
|
||||
*/
|
||||
export async function removeOperationOwnedPaths(
|
||||
input: OperationOwnedCleanup,
|
||||
): Promise<'cleared' | 'marker_retained'> {
|
||||
const base = path.resolve(input.stackManagedRoot);
|
||||
// Inline containment barrier at the removal sink (see `managedAreaBase`).
|
||||
const areaBase = managedAreaBase();
|
||||
if (!base.startsWith(areaBase + path.sep)) {
|
||||
throw new Error('refusing to remove a managed root outside the managed area');
|
||||
}
|
||||
|
||||
if (input.ownsManagedRoot) {
|
||||
if (!await isRealPathAtManagedLocation(base)) {
|
||||
throw new Error(
|
||||
'refusing to remove a managed root that links outside its managed location. '
|
||||
+ RELOCATION_REMEDIATION,
|
||||
);
|
||||
}
|
||||
await fs.rm(base, { recursive: true, force: true });
|
||||
return 'cleared';
|
||||
}
|
||||
|
||||
for (const relPath of [input.candidateRelPath, input.appliedRelPath ?? null]) {
|
||||
if (!relPath) continue;
|
||||
const resolved = path.resolve(base, relPath);
|
||||
// A strict descendant, not merely "within": `.` and `./` resolve to the
|
||||
// base itself, and containment alone would let them wipe the whole managed
|
||||
// root on the branch whose entire purpose is to protect it.
|
||||
if (resolved === base || !isPathWithinBase(resolved, base)) {
|
||||
throw new Error('refusing to remove a path outside the managed root');
|
||||
}
|
||||
// Paths that arrive from a persisted row get the same shape check as
|
||||
// marker paths, so a malformed generation row cannot widen the blast
|
||||
// radius to a whole generations directory.
|
||||
if (!/^(generations)[\\/](candidate|applied)-/.test(relPath)) {
|
||||
throw new Error(`refusing to remove a path that is not a generation directory: ${relPath}`);
|
||||
}
|
||||
// Redundant as a security check: `resolved` is already a strict descendant
|
||||
// of `base`, and `base` of `areaBase`. Present because it is this variable
|
||||
// that reaches the removal below, and the barrier has to sit at the call.
|
||||
if (!resolved.startsWith(areaBase + path.sep)) {
|
||||
throw new Error('refusing to remove a path outside the managed area');
|
||||
}
|
||||
// The checks above are lexical, so a link above this path would still pass
|
||||
// them while the delete below followed it somewhere else, including into
|
||||
// another stack's generations directory inside this same managed area.
|
||||
if (!await isRealPathAtManagedLocation(resolved)) {
|
||||
throw new Error(
|
||||
'refusing to remove a path that links outside its managed location. '
|
||||
+ RELOCATION_REMEDIATION,
|
||||
);
|
||||
}
|
||||
await fs.rm(resolved, { recursive: true, force: true });
|
||||
}
|
||||
|
||||
try {
|
||||
await deleteStagingMarker(base);
|
||||
} catch (error) {
|
||||
console.warn(
|
||||
'[GitOps] Removed the staged directories under %s but could not clear its staging marker: %s',
|
||||
sanitizeForLog(base),
|
||||
error instanceof Error ? error.message : String(error),
|
||||
);
|
||||
return 'marker_retained';
|
||||
}
|
||||
return 'cleared';
|
||||
}
|
||||
|
||||
/**
|
||||
* Cleanup for a managed root found at startup with no checkpoint and no live
|
||||
* application, driven entirely by its staging marker.
|
||||
*
|
||||
* Returns what was done so the caller can log it. A corrupt or missing marker
|
||||
* is not an error: it means nothing proves who owns this directory, so the
|
||||
* only safe action is to leave it alone.
|
||||
*/
|
||||
export async function cleanupUnclaimedManagedRoot(
|
||||
stackManagedRoot: string,
|
||||
marker: { operationId: string; rootPreexisted: boolean; candidateRelPath: string } | null,
|
||||
): Promise<'removed_root' | 'removed_candidate' | 'preserved'> {
|
||||
if (!marker) return 'preserved';
|
||||
const reason = validateCandidateRelPath(marker.candidateRelPath, stackManagedRoot);
|
||||
if (reason) {
|
||||
// Said out loud, because the caller only logs the outcome. A directory
|
||||
// that survives every boot with no stated reason is indistinguishable
|
||||
// from one nothing has looked at.
|
||||
console.warn(
|
||||
'[GitOps] Preserving unclaimed managed area %s: %s',
|
||||
sanitizeForLog(stackManagedRoot), reason,
|
||||
);
|
||||
return 'preserved';
|
||||
}
|
||||
|
||||
if (!marker.rootPreexisted) {
|
||||
// Same inline containment barrier as the sinks above: this one removes a
|
||||
// whole managed root, so it gets the check even though the analyzer has not
|
||||
// reported it. Reported as `preserved` rather than thrown, because every
|
||||
// other unprovable case here answers that way and the caller reads the
|
||||
// outcome; the warning is what says this one is anomalous rather than
|
||||
// merely unproven.
|
||||
const root = path.resolve(stackManagedRoot);
|
||||
if (!root.startsWith(managedAreaBase() + path.sep) || !await isRealPathAtManagedLocation(root)) {
|
||||
console.warn(
|
||||
'[GitOps] Refusing to reap a managed root that links outside its managed location: %s',
|
||||
sanitizeForLog(stackManagedRoot),
|
||||
);
|
||||
return 'preserved';
|
||||
}
|
||||
await fs.rm(root, { recursive: true, force: true });
|
||||
return 'removed_root';
|
||||
}
|
||||
|
||||
// The marker outcome is not reported onward: this sweep has no checkpoint to
|
||||
// keep, so a marker it could not clear is already said out loud by the
|
||||
// warning inside the call and there is nothing further for a caller to do.
|
||||
await removeOperationOwnedPaths({
|
||||
stackManagedRoot,
|
||||
candidateRelPath: marker.candidateRelPath,
|
||||
ownsManagedRoot: false,
|
||||
});
|
||||
return 'removed_candidate';
|
||||
}
|
||||
@@ -0,0 +1,355 @@
|
||||
import fs from 'fs/promises';
|
||||
import path from 'path';
|
||||
import { DatabaseService, type GitSourceAppliedSpec } from '../DatabaseService';
|
||||
import { FileSystemService } from '../FileSystemService';
|
||||
import { GitProjectManifestService } from '../GitProjectManifestService';
|
||||
import { sanitizeForLog } from '../../utils/safeLog';
|
||||
import { removeOperationOwnedPaths } from './createCleanup';
|
||||
import { deleteStagingMarker } from './createStagingMarker';
|
||||
import { newGitOpsId, stackManagedRoot } from './directApplication';
|
||||
import { GitOpsStore } from './store';
|
||||
import { GitOpsTransitions } from './transitions';
|
||||
import type { GitOpsCreateCheckpointRow } from './types';
|
||||
|
||||
/** What the sweep decided about one interrupted create. */
|
||||
export type CreateRecoveryOutcome =
|
||||
| 'completed'
|
||||
| 'tombstoned'
|
||||
| 'checkpoint_cleared'
|
||||
| 'source_preserved'
|
||||
| 'retained'
|
||||
/**
|
||||
* The create itself is settled; only clearing its staging marker failed.
|
||||
* Distinct from `retained` so the boot log does not send an operator looking
|
||||
* for an unfinished create that finished.
|
||||
*/
|
||||
| 'marker_retained';
|
||||
|
||||
export type CreateRecoveryResult = {
|
||||
stackName: string;
|
||||
applicationId: string;
|
||||
outcome: CreateRecoveryOutcome;
|
||||
};
|
||||
|
||||
/**
|
||||
* Refuse to continue while any create is still unresolved.
|
||||
*
|
||||
* A create that could not be settled leaves a stack directory the deploy path
|
||||
* cannot tell apart from a finished one, so the alternative to stopping is
|
||||
* letting a scheduler, webhook or operator act on a half-built stack. Startup
|
||||
* calls this before the background mutators and the HTTP bind.
|
||||
*
|
||||
* Only `retained` counts. `marker_retained` means the create itself is settled
|
||||
* and a leftover marker file is all that survived, which decides nothing about
|
||||
* ownership and must not cost an operator their instance.
|
||||
*/
|
||||
export function assertCreatesSettled(settled: readonly CreateRecoveryResult[]): void {
|
||||
const unresolved = settled.filter((entry) => entry.outcome === 'retained');
|
||||
if (unresolved.length === 0) return;
|
||||
const named = unresolved.map((entry) => sanitizeForLog(entry.stackName || entry.applicationId)).join(', ');
|
||||
throw new Error(
|
||||
`${unresolved.length} interrupted create(s) could not be settled: ${named}. `
|
||||
+ 'Sencho does not start while a create is unresolved, because a half-built stack '
|
||||
+ 'is indistinguishable from a finished one. The cause is logged above; clearing it '
|
||||
+ 'lets the next start finish the recovery.',
|
||||
);
|
||||
}
|
||||
|
||||
function envelopeFor(checkpoint: GitOpsCreateCheckpointRow) {
|
||||
return {
|
||||
operationId: checkpoint.operation_id,
|
||||
actor: 'system:startup',
|
||||
trigger: 'create_recovery',
|
||||
at: Date.now(),
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Three states, because the two callers need opposite fail-safe directions.
|
||||
*
|
||||
* Teardown must not treat "cannot tell" as absent, or it would skip a directory
|
||||
* that is really there. Completion must not treat it as present, or it would
|
||||
* mark a create live on the strength of a failed stat.
|
||||
*/
|
||||
/**
|
||||
* Clear a settled create's staging marker, reporting rather than throwing.
|
||||
*
|
||||
* The checkpoint is only dropped once this succeeds, because a marker left
|
||||
* behind with no checkpoint has nothing to retry it and refuses every later
|
||||
* create for that stack name. The cost is that a marker this keeps failing on
|
||||
* also keeps the checkpoint row, and its encrypted token, alive across boots.
|
||||
* That is the lesser harm: the row is encrypted at rest with the same key as
|
||||
* the source it came from, and it is what makes the retry possible at all.
|
||||
*/
|
||||
async function clearSettledMarker(stackName: string, managedRoot: string): Promise<boolean> {
|
||||
try {
|
||||
await deleteStagingMarker(managedRoot);
|
||||
return true;
|
||||
} catch (error) {
|
||||
console.warn(
|
||||
`[GitOps] Settled the create for ${sanitizeForLog(stackName)} but could not clear its staging marker:`,
|
||||
error instanceof Error ? error.message : String(error),
|
||||
);
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
async function stackDirState(stackName: string): Promise<'present' | 'absent' | 'unknown'> {
|
||||
try {
|
||||
const base = FileSystemService.getInstance().getBaseDir();
|
||||
await fs.stat(path.join(base, stackName));
|
||||
return 'present';
|
||||
} catch (error) {
|
||||
if ((error as NodeJS.ErrnoException).code === 'ENOENT') return 'absent';
|
||||
console.warn(
|
||||
`[GitOps] Cannot determine whether stack ${sanitizeForLog(stackName)} exists on disk:`,
|
||||
error instanceof Error ? error.message : String(error),
|
||||
);
|
||||
return 'unknown';
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Settle every create that a previous process left in flight.
|
||||
*
|
||||
* Runs at boot, before any mutation service, and is idempotent: each row is
|
||||
* decided from the durable checkpoint phase plus what is actually on disk, so
|
||||
* a crash during recovery itself just replays on the next boot.
|
||||
*
|
||||
* The two rules that shape every branch: a create is only finished when its
|
||||
* manifest is already committed on disk, and it is only torn down after its
|
||||
* files are gone. A create whose files cannot be removed keeps its checkpoint
|
||||
* rather than being recorded as cleanly failed.
|
||||
*/
|
||||
export async function resolveInterruptedCreates(): Promise<CreateRecoveryResult[]> {
|
||||
const store = GitOpsStore.getInstance();
|
||||
const db = DatabaseService.getInstance();
|
||||
const results: CreateRecoveryResult[] = [];
|
||||
|
||||
for (const checkpoint of store.listCreateCheckpoints()) {
|
||||
try {
|
||||
results.push(await resolveOne(checkpoint));
|
||||
} catch (error) {
|
||||
console.error(
|
||||
`[GitOps] Could not settle the interrupted create for ${sanitizeForLog(checkpoint.stack_name)}; retrying next boot:`,
|
||||
error instanceof Error ? error.stack ?? error.message : String(error),
|
||||
);
|
||||
results.push({
|
||||
stackName: checkpoint.stack_name,
|
||||
applicationId: checkpoint.application_id,
|
||||
outcome: 'retained',
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
// A creating application with no checkpoint at all cannot be finished: the
|
||||
// facts needed to complete it are gone. Tombstone it so the stack name is
|
||||
// usable again, and never touch a source row that outlived it, because the
|
||||
// conservative migration can still build a live application from that row.
|
||||
for (const app of store.listCreatingDirectApplications()) {
|
||||
if (store.getCreateCheckpoint(app.id)) continue;
|
||||
const stackName = app.stack_name ?? '';
|
||||
// Guarded per row for the same reason as the loop above: one application
|
||||
// that cannot be tombstoned must not strand the rest. A creating row that
|
||||
// survives keeps matching the live-application lookup, so every later
|
||||
// create for that name would fail until it is cleared.
|
||||
try {
|
||||
const sourceRow = stackName ? db.getGitSource(stackName) : null;
|
||||
GitOpsTransitions.getInstance().createFailed(app.id, 'create_checkpoint_missing', {
|
||||
operationId: app.latest_operation_id ?? newGitOpsId(),
|
||||
actor: 'system:startup',
|
||||
trigger: 'create_recovery',
|
||||
at: Date.now(),
|
||||
});
|
||||
results.push({
|
||||
stackName,
|
||||
applicationId: app.id,
|
||||
outcome: sourceRow ? 'source_preserved' : 'tombstoned',
|
||||
});
|
||||
} catch (error) {
|
||||
console.error(
|
||||
`[GitOps] Could not tombstone the checkpointless create for ${sanitizeForLog(stackName)}; retrying next boot:`,
|
||||
error instanceof Error ? error.stack ?? error.message : String(error),
|
||||
);
|
||||
results.push({ stackName, applicationId: app.id, outcome: 'retained' });
|
||||
}
|
||||
}
|
||||
|
||||
return results;
|
||||
}
|
||||
|
||||
/**
|
||||
* Reclassify every operation the previous process left open.
|
||||
*
|
||||
* An operation that started and never terminated keeps reporting as in flight,
|
||||
* and the deriver offers no actions while it does, so without this a single
|
||||
* interrupted fetch or apply strands a stack until someone notices. The
|
||||
* interruption is recorded as unknown rather than failed: we genuinely do not
|
||||
* know whether the work completed.
|
||||
*
|
||||
* Runs at boot, after create recovery and before any mutation service, and is
|
||||
* guarded per application so one bad row cannot strand the rest.
|
||||
*/
|
||||
export function reclassifyInterruptedOperations(): number {
|
||||
const store = GitOpsStore.getInstance();
|
||||
let reclassified = 0;
|
||||
for (const app of store.listApplicationsWithOpenOperations()) {
|
||||
try {
|
||||
GitOpsTransitions.getInstance().interruptActiveOperations(app.id, {
|
||||
operationId: app.latest_operation_id ?? newGitOpsId(),
|
||||
actor: 'system:startup',
|
||||
trigger: 'startup_reconcile',
|
||||
at: Date.now(),
|
||||
});
|
||||
reclassified += 1;
|
||||
} catch (error) {
|
||||
console.error(
|
||||
`[GitOps] Could not reclassify the interrupted operation for ${sanitizeForLog(app.stack_name ?? app.id)}:`,
|
||||
error instanceof Error ? error.stack ?? error.message : String(error),
|
||||
);
|
||||
}
|
||||
}
|
||||
return reclassified;
|
||||
}
|
||||
|
||||
async function resolveOne(checkpoint: GitOpsCreateCheckpointRow): Promise<CreateRecoveryResult> {
|
||||
const store = GitOpsStore.getInstance();
|
||||
const db = DatabaseService.getInstance();
|
||||
const app = store.getApplication(checkpoint.application_id);
|
||||
const stackName = checkpoint.stack_name;
|
||||
const managedRoot = stackManagedRoot(stackName);
|
||||
|
||||
// Nothing left to decide: the checkpoint outlived its application, the create
|
||||
// already reached its success boundary, or the application has since moved out
|
||||
// of `creating` on some other path. Every one of them means this row is stale
|
||||
// bookkeeping, so they settle identically and share one exit, which is what
|
||||
// keeps the marker ordering below true of all of them rather than of whichever
|
||||
// branch last remembered it.
|
||||
if (!app || checkpoint.phase === 'pointers_committed' || app.lifecycle_status !== 'creating') {
|
||||
// Marker first: clearing it can fail, and dropping the checkpoint before
|
||||
// that would leave a marker that makes the stack name uncreatable with
|
||||
// nothing left to retry it.
|
||||
if (!await clearSettledMarker(stackName, managedRoot)) {
|
||||
return { stackName, applicationId: checkpoint.application_id, outcome: 'marker_retained' };
|
||||
}
|
||||
store.deleteCreateCheckpoint(checkpoint.application_id);
|
||||
return { stackName, applicationId: checkpoint.application_id, outcome: 'checkpoint_cleared' };
|
||||
}
|
||||
|
||||
// The manifest is committed on disk, so the authored project the operator
|
||||
// asked for exists. Finish the create rather than destroying it: this is the
|
||||
// same source-row plus acceptance commit the live path performs.
|
||||
if (
|
||||
checkpoint.phase === 'manifest_committed'
|
||||
&& checkpoint.generation_id
|
||||
&& await stackDirState(stackName) === 'present'
|
||||
) {
|
||||
const generationId = checkpoint.generation_id;
|
||||
const appliedSpec = checkpoint.applied_spec_json
|
||||
? JSON.parse(checkpoint.applied_spec_json) as GitSourceAppliedSpec
|
||||
: null;
|
||||
const manifest = await GitProjectManifestService.getInstance().readManifest(
|
||||
stackName, checkpoint.repo_url, checkpoint.branch,
|
||||
);
|
||||
db.getDb().transaction(() => {
|
||||
if (!db.getGitSource(stackName)) {
|
||||
db.upsertGitSource({
|
||||
stack_name: stackName,
|
||||
repo_url: checkpoint.repo_url,
|
||||
branch: checkpoint.branch,
|
||||
compose_path: checkpoint.compose_path,
|
||||
compose_paths: JSON.parse(checkpoint.compose_paths_json) as string[],
|
||||
context_dir: checkpoint.context_dir,
|
||||
sync_env: checkpoint.sync_env === 1,
|
||||
env_path: checkpoint.env_path,
|
||||
auth_type: checkpoint.auth_type as 'none' | 'token',
|
||||
encrypted_token: checkpoint.encrypted_token,
|
||||
auto_apply_on_webhook: checkpoint.auto_apply_on_webhook === 1,
|
||||
auto_deploy_on_apply: checkpoint.auto_deploy_on_apply === 1,
|
||||
last_applied_commit_sha: checkpoint.commit_sha,
|
||||
last_applied_content_hash: null,
|
||||
pending_commit_sha: null,
|
||||
pending_compose_content: null,
|
||||
pending_env_content: null,
|
||||
pending_fetched_at: null,
|
||||
last_debounce_at: null,
|
||||
});
|
||||
}
|
||||
// The insert does not carry the applied pointers, so stamp them the way
|
||||
// the live create path does. The content hash is left empty because the
|
||||
// bytes that produced it belonged to the process that crashed; the first
|
||||
// pull after recovery re-establishes it.
|
||||
db.markGitSourceApplied(stackName, checkpoint.commit_sha ?? '', '');
|
||||
// The live path also stamps the deploy spec and the manifest cache. Both
|
||||
// are load-bearing: a null spec silently reverts a multi-file stack to
|
||||
// single-file auto-discovery, and an unset manifest state makes a managed
|
||||
// stack render as unmanaged and suppresses its rollback disclosure.
|
||||
if (appliedSpec) db.setGitSourceAppliedSpec(stackName, appliedSpec);
|
||||
if (manifest && 'manifestVersion' in manifest) {
|
||||
db.setGitSourceManifestState(
|
||||
stackName,
|
||||
manifest.manifestVersion,
|
||||
manifest.state,
|
||||
manifest.generation.appliedDir,
|
||||
);
|
||||
}
|
||||
GitOpsTransitions.getInstance().applied({
|
||||
applicationId: checkpoint.application_id,
|
||||
generationId,
|
||||
artifactSetId: newGitOpsId(),
|
||||
sourceAcceptanceId: newGitOpsId(),
|
||||
authority: 'operator',
|
||||
envelope: envelopeFor(checkpoint),
|
||||
activateCreating: true,
|
||||
});
|
||||
store.updateCreateCheckpoint(checkpoint.application_id, { phase: 'pointers_committed' }, Date.now());
|
||||
})();
|
||||
// Marker before checkpoint, for the reason given on the branch above. The
|
||||
// create is live either way by this point: the transaction above committed.
|
||||
if (!await clearSettledMarker(stackName, managedRoot)) {
|
||||
return { stackName, applicationId: checkpoint.application_id, outcome: 'marker_retained' };
|
||||
}
|
||||
store.deleteCreateCheckpoint(checkpoint.application_id);
|
||||
return { stackName, applicationId: checkpoint.application_id, outcome: 'completed' };
|
||||
}
|
||||
|
||||
// Everything else stopped before the project was durable. Remove exactly what
|
||||
// this operation put on disk, then record the failure. Filesystem first: if it
|
||||
// throws, the checkpoint survives and the next boot retries.
|
||||
const generation = checkpoint.generation_id ? store.getGeneration(checkpoint.generation_id) : undefined;
|
||||
// `pre_stack` is durable proof that createStack had not returned, so a
|
||||
// directory present now was not necessarily made by this operation. It could
|
||||
// be the operator's own stack that appeared while the create was fetching.
|
||||
// Removing it on that evidence is the one mistake this path cannot take back,
|
||||
// so an orphaned directory is left behind instead.
|
||||
const stackDir = await stackDirState(stackName);
|
||||
if (checkpoint.phase === 'pre_stack') {
|
||||
if (stackDir === 'present') {
|
||||
console.warn(
|
||||
`[GitOps] Leaving the directory for ${sanitizeForLog(stackName)} in place: the interrupted create never recorded creating it.`,
|
||||
);
|
||||
}
|
||||
} else if (stackDir === 'present') {
|
||||
await FileSystemService.getInstance().deleteStack(stackName);
|
||||
}
|
||||
const cleanup = await removeOperationOwnedPaths({
|
||||
stackManagedRoot: managedRoot,
|
||||
candidateRelPath: generation?.candidate_dir ?? null,
|
||||
appliedRelPath: generation?.applied_dir ?? null,
|
||||
ownsManagedRoot: checkpoint.created_managed_root === 1,
|
||||
});
|
||||
// The staged directories are gone, so nothing deployable survives, but the
|
||||
// marker still claims the name. Keep the checkpoint and report the same
|
||||
// non-fatal outcome the settled path uses, rather than letting one unlink
|
||||
// failure read as an unresolved create and stop the instance booting.
|
||||
if (cleanup === 'marker_retained') {
|
||||
return { stackName, applicationId: checkpoint.application_id, outcome: 'marker_retained' };
|
||||
}
|
||||
|
||||
GitOpsTransitions.getInstance().createFailed(
|
||||
checkpoint.application_id,
|
||||
'interrupted_create',
|
||||
envelopeFor(checkpoint),
|
||||
);
|
||||
return { stackName, applicationId: checkpoint.application_id, outcome: 'tombstoned' };
|
||||
}
|
||||
@@ -0,0 +1,201 @@
|
||||
import fs from 'fs/promises';
|
||||
import path from 'path';
|
||||
import { isPathWithinBase } from '../../utils/validation';
|
||||
import { GENERATIONS_DIR, isRealPathAtManagedLocation, managedAreaBase } from './managedPaths';
|
||||
|
||||
export const CREATE_STAGING_MARKER_FILENAME = '.create-staging.v1.json';
|
||||
|
||||
const CANDIDATE_PREFIX = `${GENERATIONS_DIR}/candidate-`;
|
||||
|
||||
export type CreateStagingMarker = {
|
||||
schemaVersion: 1;
|
||||
operationId: string;
|
||||
/** True when the managed root already existed before this operation ran. */
|
||||
rootPreexisted: boolean;
|
||||
/** Never null: the candidate path is computed before any managed-root mutation. */
|
||||
candidateRelPath: string;
|
||||
/** Diagnostic only. Never deletion authority. */
|
||||
createdAt: number;
|
||||
};
|
||||
|
||||
export type ReadStagingMarkerResult =
|
||||
| { state: 'valid'; marker: CreateStagingMarker }
|
||||
| { state: 'missing' }
|
||||
| { state: 'corrupt'; reason: string };
|
||||
|
||||
/**
|
||||
* The candidate directory this operation will stage, computed the same way the
|
||||
* manifest service computes it, before anything touches the managed root.
|
||||
*
|
||||
* Recording the path up front is what makes crash cleanup exact: a process that
|
||||
* dies part-way through building the candidate still left a marker naming the
|
||||
* one directory it owned.
|
||||
*/
|
||||
export function candidateRelPathForSha(commitSha: string): string {
|
||||
return `${CANDIDATE_PREFIX}${commitSha}`;
|
||||
}
|
||||
|
||||
/**
|
||||
* The applied directory promotion will move this candidate into.
|
||||
*
|
||||
* Computed here rather than read from the manifest because the generation row
|
||||
* is written before promotion runs, and the manifest carries an empty applied
|
||||
* path until then. Storing that empty value would make "not promoted yet"
|
||||
* indistinguishable from "nothing to clean up" during teardown.
|
||||
*
|
||||
* Must stay in step with the promotion path in GitProjectManifestService.
|
||||
*/
|
||||
export function appliedRelPathFor(commitSha: string, manifestVersion: number): string {
|
||||
return `${GENERATIONS_DIR}/applied-${commitSha}-${manifestVersion}`;
|
||||
}
|
||||
|
||||
function isSafeRelPath(value: unknown): value is string {
|
||||
if (typeof value !== 'string' || value.length === 0) return false;
|
||||
if (path.isAbsolute(value)) return false;
|
||||
const segments = value.split(/[\\/]/);
|
||||
return !segments.some((seg) => seg === '..' || seg === '.' || seg === '');
|
||||
}
|
||||
|
||||
/**
|
||||
* Validate a candidate path against the stack's own managed root.
|
||||
*
|
||||
* Every read re-runs this, not just the write, because the marker is a
|
||||
* filesystem artifact an operator or a bug could have rewritten between the
|
||||
* crash and the sweep. A path that fails any check makes the marker corrupt,
|
||||
* and a corrupt marker preserves the root rather than authorizing a delete.
|
||||
*/
|
||||
export function validateCandidateRelPath(candidateRelPath: unknown, stackManagedRoot: string): string | null {
|
||||
if (!isSafeRelPath(candidateRelPath)) return 'candidateRelPath is not a safe relative path';
|
||||
if (!candidateRelPath.startsWith(CANDIDATE_PREFIX)) {
|
||||
return `candidateRelPath must start with ${CANDIDATE_PREFIX}`;
|
||||
}
|
||||
const base = path.resolve(stackManagedRoot);
|
||||
const resolved = path.resolve(base, candidateRelPath);
|
||||
if (!isPathWithinBase(resolved, base)) return 'candidateRelPath escapes the managed root';
|
||||
return null;
|
||||
}
|
||||
|
||||
/** Test-only. Production call sites resolve this path inline at their own sink. */
|
||||
export function stagingMarkerPath(stackManagedRoot: string): string {
|
||||
return path.join(stackManagedRoot, CREATE_STAGING_MARKER_FILENAME);
|
||||
}
|
||||
|
||||
export async function readStagingMarker(stackManagedRoot: string): Promise<ReadStagingMarkerResult> {
|
||||
// Inline containment barrier at the read sink (see `managedAreaBase`). A root
|
||||
// outside the managed area is treated as corrupt rather than thrown, matching
|
||||
// this module's rule that an unreadable claim preserves rather than deletes.
|
||||
const areaBase = managedAreaBase();
|
||||
const markerPath = path.resolve(stackManagedRoot, CREATE_STAGING_MARKER_FILENAME);
|
||||
if (!markerPath.startsWith(areaBase + path.sep)) {
|
||||
return { state: 'corrupt', reason: 'managed root escapes the managed area' };
|
||||
}
|
||||
let raw: string;
|
||||
try {
|
||||
raw = await fs.readFile(markerPath, 'utf8');
|
||||
} catch (error) {
|
||||
if ((error as NodeJS.ErrnoException).code === 'ENOENT') return { state: 'missing' };
|
||||
return { state: 'corrupt', reason: (error as Error).message };
|
||||
}
|
||||
let parsed: unknown;
|
||||
try {
|
||||
parsed = JSON.parse(raw);
|
||||
} catch {
|
||||
return { state: 'corrupt', reason: 'marker is not valid JSON' };
|
||||
}
|
||||
if (typeof parsed !== 'object' || parsed === null || Array.isArray(parsed)) {
|
||||
return { state: 'corrupt', reason: 'marker is not an object' };
|
||||
}
|
||||
const marker = parsed as Record<string, unknown>;
|
||||
if (marker.schemaVersion !== 1) return { state: 'corrupt', reason: 'unsupported marker schemaVersion' };
|
||||
if (typeof marker.operationId !== 'string' || marker.operationId.length === 0) {
|
||||
return { state: 'corrupt', reason: 'marker operationId is missing' };
|
||||
}
|
||||
if (typeof marker.rootPreexisted !== 'boolean') {
|
||||
return { state: 'corrupt', reason: 'marker rootPreexisted is missing' };
|
||||
}
|
||||
const pathReason = validateCandidateRelPath(marker.candidateRelPath, stackManagedRoot);
|
||||
if (pathReason) return { state: 'corrupt', reason: pathReason };
|
||||
const createdAt = typeof marker.createdAt === 'number' && Number.isFinite(marker.createdAt)
|
||||
? marker.createdAt
|
||||
: 0;
|
||||
return {
|
||||
state: 'valid',
|
||||
marker: {
|
||||
schemaVersion: 1,
|
||||
operationId: marker.operationId,
|
||||
rootPreexisted: marker.rootPreexisted,
|
||||
candidateRelPath: marker.candidateRelPath as string,
|
||||
createdAt,
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
export class CreateStagingMarkerError extends Error {
|
||||
constructor(message: string) {
|
||||
super(message);
|
||||
this.name = 'CreateStagingMarkerError';
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Write the marker atomically, refusing to trample another operation's.
|
||||
*
|
||||
* A live marker for a different operation means a concurrent or abandoned
|
||||
* create still owns this managed root. Overwriting it would hand our operation
|
||||
* id deletion authority over their staged directory, so we refuse instead.
|
||||
*/
|
||||
export async function writeStagingMarker(
|
||||
stackManagedRoot: string,
|
||||
marker: CreateStagingMarker,
|
||||
): Promise<void> {
|
||||
const reason = validateCandidateRelPath(marker.candidateRelPath, stackManagedRoot);
|
||||
if (reason) throw new CreateStagingMarkerError(reason);
|
||||
|
||||
const existing = await readStagingMarker(stackManagedRoot);
|
||||
// A marker that exists but cannot be read is still a claim. Overwriting it
|
||||
// would hand this operation deletion authority over whatever the last one
|
||||
// staged, which is the opposite of what the rest of this module does with a
|
||||
// corrupt marker, and it would do so with nothing said about why.
|
||||
if (existing.state === 'corrupt') {
|
||||
throw new CreateStagingMarkerError(
|
||||
`this managed area has an unreadable staging marker (${existing.reason}); refusing to claim it`,
|
||||
);
|
||||
}
|
||||
if (existing.state === 'valid' && existing.marker.operationId !== marker.operationId) {
|
||||
throw new CreateStagingMarkerError(
|
||||
'another create operation already owns this managed area',
|
||||
);
|
||||
}
|
||||
|
||||
// Inline containment barrier at each write sink (see `managedAreaBase`).
|
||||
const areaBase = managedAreaBase();
|
||||
const root = path.resolve(stackManagedRoot);
|
||||
const target = path.resolve(stackManagedRoot, CREATE_STAGING_MARKER_FILENAME);
|
||||
const temp = path.resolve(stackManagedRoot, `${CREATE_STAGING_MARKER_FILENAME}.${marker.operationId}.tmp`);
|
||||
if (
|
||||
!root.startsWith(areaBase + path.sep)
|
||||
|| !target.startsWith(areaBase + path.sep)
|
||||
|| !temp.startsWith(areaBase + path.sep)
|
||||
// Checked on the write and not only on the delete, so a link cannot be
|
||||
// written through and then refused by the hardened delete below. That
|
||||
// asymmetry would wedge the stack name: a valid marker naming another
|
||||
// operation refuses every later create, and nothing could remove it.
|
||||
|| !await isRealPathAtManagedLocation(root)
|
||||
) {
|
||||
throw new CreateStagingMarkerError('managed root links outside its managed location');
|
||||
}
|
||||
|
||||
await fs.mkdir(root, { recursive: true });
|
||||
await fs.writeFile(temp, JSON.stringify(marker), 'utf8');
|
||||
await fs.rename(temp, target);
|
||||
}
|
||||
|
||||
export async function deleteStagingMarker(stackManagedRoot: string): Promise<void> {
|
||||
// Inline containment barrier at the removal sink (see `managedAreaBase`).
|
||||
const areaBase = managedAreaBase();
|
||||
const markerPath = path.resolve(stackManagedRoot, CREATE_STAGING_MARKER_FILENAME);
|
||||
if (!markerPath.startsWith(areaBase + path.sep) || !await isRealPathAtManagedLocation(markerPath)) {
|
||||
throw new CreateStagingMarkerError('managed root links outside its managed location');
|
||||
}
|
||||
await fs.rm(markerPath, { force: true });
|
||||
}
|
||||
@@ -0,0 +1,914 @@
|
||||
import {
|
||||
decodeArtifactEvidenceJson,
|
||||
decodeGitOpsEvidenceLimitations,
|
||||
decodeObservedArtifactIdentity,
|
||||
GitOpsJsonError,
|
||||
} from './json';
|
||||
import { GitOpsStore } from './store';
|
||||
import type { BlueprintObservationStage } from './transitions';
|
||||
import type {
|
||||
ArtifactExpectedIdentity,
|
||||
ArtifactFacet,
|
||||
ArtifactLatestEvidence,
|
||||
ArtifactQualification,
|
||||
FutureGitOpsEvidence,
|
||||
GitOpsApplicationRow,
|
||||
GitOpsAvailableAction,
|
||||
GitOpsLimitation,
|
||||
GitOpsRevisionProjection,
|
||||
GitOpsDriftItem,
|
||||
GitOpsTargetCurrentRow,
|
||||
GitOpsTargetProjection,
|
||||
HealthFacet,
|
||||
LkgFacet,
|
||||
PlacementFacet,
|
||||
RolloutFacet,
|
||||
RuntimeFacet,
|
||||
SourceFacet,
|
||||
SourceIdentityFields,
|
||||
} from './types';
|
||||
|
||||
/**
|
||||
* The projection for anything that carries no GitOps application.
|
||||
*
|
||||
* One shared instance, so its collections are frozen alongside it: a caller
|
||||
* that pushed a limitation onto this projection would otherwise corrupt every
|
||||
* later response in the process. The separate declaration is what gives the
|
||||
* literal its contextual type; freezing it inline widens the empty tuples to
|
||||
* `never[]` and fails to typecheck.
|
||||
*/
|
||||
const NOT_APPLICABLE: GitOpsRevisionProjection = {
|
||||
schemaVersion: 1,
|
||||
targetMode: 'not_applicable',
|
||||
applicationId: null,
|
||||
facets: null,
|
||||
targets: [],
|
||||
drift: [],
|
||||
limitations: [],
|
||||
availableActions: [],
|
||||
approvals: null,
|
||||
};
|
||||
// Frozen after construction rather than inline: the collections stay mutable
|
||||
// types so the projection union still matches, while the shared instance
|
||||
// refuses writes at runtime.
|
||||
for (const collection of [
|
||||
NOT_APPLICABLE.targets,
|
||||
NOT_APPLICABLE.drift,
|
||||
NOT_APPLICABLE.limitations,
|
||||
NOT_APPLICABLE.availableActions,
|
||||
]) {
|
||||
Object.freeze(collection);
|
||||
}
|
||||
export const NOT_APPLICABLE_REVISION: GitOpsRevisionProjection = Object.freeze(NOT_APPLICABLE);
|
||||
|
||||
export type DeriveFacts = {
|
||||
application: GitOpsApplicationRow | null;
|
||||
targets: GitOpsTargetCurrentRow[];
|
||||
healthDisabled: boolean;
|
||||
};
|
||||
|
||||
export function deriveGitOpsRevision(
|
||||
facts: DeriveFacts,
|
||||
futureEvidence: FutureGitOpsEvidence | null,
|
||||
): GitOpsRevisionProjection {
|
||||
// Future-only facets are typed here so callers share one deriver; this slice
|
||||
// projects only persisted current evidence.
|
||||
void futureEvidence;
|
||||
const app = facts.application;
|
||||
if (!app) return NOT_APPLICABLE_REVISION;
|
||||
const limitations: GitOpsLimitation[] = [];
|
||||
mergePersistedLimitations(app.evidence_limitations_json, limitations);
|
||||
const source = deriveSource(app, limitations);
|
||||
const artifact = deriveArtifact(app, app.accepted_generation_id, app.artifact_set_id, app.latest_artifact_set_id, limitations);
|
||||
const placement = derivePlacement(app);
|
||||
const targets = facts.targets
|
||||
.slice()
|
||||
.sort((a, b) => a.node_id - b.node_id)
|
||||
.map((target) => deriveTarget(app, target, facts.healthDisabled, limitations));
|
||||
const rollout = deriveRollout(app, targets);
|
||||
const availableActions = deriveActions(app, source, placement, targets);
|
||||
return {
|
||||
schemaVersion: 1,
|
||||
targetMode: app.target_mode,
|
||||
applicationId: app.id,
|
||||
lifecycleStatus: app.lifecycle_status,
|
||||
stackName: app.stack_name,
|
||||
blueprintId: app.blueprint_id,
|
||||
rolloutGenerationId: app.rollout_generation_id,
|
||||
approvals: {
|
||||
sourceAcceptanceRef: app.source_acceptance_ref,
|
||||
placementApprovalRef: app.placement_approval_ref,
|
||||
rolloutAuthorizationRef: app.rollout_authorization_ref,
|
||||
legacyCombinedApprovalRef: app.legacy_combined_approval_ref,
|
||||
},
|
||||
facets: { source, artifact, placement, rollout },
|
||||
targets,
|
||||
drift: collectRuntimeDrift(app, targets),
|
||||
limitations,
|
||||
availableActions,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* The two drift classes current evidence can confirm on its own.
|
||||
*
|
||||
* Most of the seven classes need producers this model deliberately defers, but
|
||||
* a comparable runtime artifact mismatch and a desired-versus-deployed
|
||||
* generation mismatch rest entirely on rows that exist now. Leaving `drift`
|
||||
* empty while a facet says `runtime_artifact_drift` would report one fault
|
||||
* twice with only one copy readable; the same holds whenever the known
|
||||
* pointers disagree, whatever presentation status outranks them.
|
||||
*
|
||||
* The generation-mismatch item carries the desired generation as expected and
|
||||
* the deployed generation as observed, owned by ComposeService. It clears
|
||||
* once the desired generation is deployed.
|
||||
*
|
||||
* The artifact item is emitted only for an exact or qualified expectation
|
||||
* against an exact or qualified observation whose identity strings differ.
|
||||
* Every other observation kind stays `artifact_verification_pending`, and
|
||||
* equal identities emit nothing. Policy composition has no producer yet, so
|
||||
* items carry null rather than a policy nothing wrote.
|
||||
*/
|
||||
function collectRuntimeDrift(
|
||||
app: GitOpsApplicationRow,
|
||||
targets: GitOpsTargetProjection[],
|
||||
): GitOpsDriftItem[] {
|
||||
const items: GitOpsDriftItem[] = [];
|
||||
for (const target of targets) {
|
||||
// Generation mismatch: the target's contract is its desired generation,
|
||||
// and a different generation is running. Judged from the pointers
|
||||
// themselves, not from any one runtime status: paused, failed, recovering,
|
||||
// interrupted, and in-flight states all outrank the pointer comparison in
|
||||
// deriveRuntime, so keying the item to `applied_not_deployed` would drop
|
||||
// the report exactly when a failed deploy leaves the old workload serving.
|
||||
// A retired target is excluded: its pointers survive retirement on
|
||||
// purpose, but nothing can ever rebind it, so its mismatch would be a
|
||||
// permanently unresolvable item rather than a live divergence.
|
||||
if (
|
||||
!target.tombstoned
|
||||
&& target.desiredGenerationId !== null
|
||||
&& target.deployedGenerationId !== null
|
||||
&& target.desiredGenerationId !== target.deployedGenerationId
|
||||
) {
|
||||
items.push({
|
||||
class: 'runtime',
|
||||
expected: { kind: 'generation', id: target.desiredGenerationId },
|
||||
observed: { kind: 'generation', id: target.deployedGenerationId },
|
||||
freshnessAt: null,
|
||||
owner: 'ComposeService',
|
||||
reason: 'the target is running a different generation than the one it was asked to run',
|
||||
configuredPolicy: null,
|
||||
affectedTargets: [{ nodeId: target.nodeId, stackName: app.stack_name }],
|
||||
// The action answers for this affected target alone, using the same
|
||||
// legality predicate that puts deploy into availableActions, so a
|
||||
// sibling's clean convergence can never recommend deploying this one
|
||||
// while it is paused, failed, or otherwise unable to act.
|
||||
action: targetDeployLegal(app, target) ? 'deploy' : 'none',
|
||||
});
|
||||
}
|
||||
// Artifact mismatch: comparable exact/qualified expectation vs observation.
|
||||
if (target.runtime.status !== 'runtime_artifact_drift') continue;
|
||||
const expected = target.artifact.status !== 'not_applicable' && 'expected' in target.artifact
|
||||
? target.artifact.expected
|
||||
: null;
|
||||
if (!expected || expected.identity === null) continue;
|
||||
const observed = target.observedArtifactIdentity;
|
||||
if ((observed.kind !== 'exact' && observed.kind !== 'qualified') || observed.identity === expected.identity) {
|
||||
continue;
|
||||
}
|
||||
items.push({
|
||||
class: 'runtime',
|
||||
expected: {
|
||||
kind: 'artifact_set',
|
||||
id: expected.artifactSetId,
|
||||
qualification: expected.qualification,
|
||||
evidenceVersion: expected.evidenceVersion,
|
||||
},
|
||||
observed: { kind: 'runtime_artifact', identity: observed.identity, observedAt: observed.observedAt },
|
||||
freshnessAt: observed.observedAt,
|
||||
owner: 'observed_artifact_identity',
|
||||
reason: 'the running workload reports an artifact identity other than the expected artifact set',
|
||||
configuredPolicy: null,
|
||||
affectedTargets: [{ nodeId: target.nodeId, stackName: app.stack_name }],
|
||||
action: 'none',
|
||||
});
|
||||
}
|
||||
return items;
|
||||
}
|
||||
|
||||
function deriveSource(app: GitOpsApplicationRow, limitations: GitOpsLimitation[]): SourceFacet {
|
||||
if (app.target_mode === 'inline_blueprint') return { status: 'not_applicable' };
|
||||
const identity = sourceIdentity(app, limitations);
|
||||
if (app.lifecycle_status === 'detached' || app.lifecycle_status === 'deleted') {
|
||||
return { ...identity, status: 'not_live', lifecycleStatus: app.lifecycle_status };
|
||||
}
|
||||
if (app.recovery_phase === 'restoring' || app.recovery_phase === 'compensating') {
|
||||
return { ...identity, status: 'recovery_required', recoveryRef: app.recovery_ref, recoveryGenerationId: null };
|
||||
}
|
||||
if (app.recovery_phase === 'failed' || app.failure_stage === 'recovery') {
|
||||
return {
|
||||
...identity,
|
||||
status: 'recovery_failed',
|
||||
recoveryRef: app.recovery_ref,
|
||||
recoveryGenerationId: null,
|
||||
failureClass: app.failure_class ?? 'unknown',
|
||||
failureAt: app.failure_at ?? 0,
|
||||
};
|
||||
}
|
||||
if (app.active_operation_stage === 'fetch_started') return { ...identity, status: 'checking_fetching' };
|
||||
if (app.active_operation_stage === 'apply_started') {
|
||||
return {
|
||||
...identity,
|
||||
status: 'applying',
|
||||
activeOperationId: app.active_operation_id ?? '',
|
||||
activeGenerationId: app.active_generation_id ?? '',
|
||||
};
|
||||
}
|
||||
if (app.interruption_stage === 'fetch_started' || app.interruption_stage === 'apply_started') {
|
||||
return {
|
||||
...identity,
|
||||
status: 'source_unknown',
|
||||
interruptedStage: app.interruption_stage,
|
||||
interruptedAt: app.interruption_at ?? 0,
|
||||
interruptedOperationId: app.interruption_operation_id,
|
||||
interruptedGenerationId: app.interruption_generation_id,
|
||||
};
|
||||
}
|
||||
if (app.suspended_at) return { ...identity, status: 'source_suspended', suspendedAt: app.suspended_at };
|
||||
if (app.failure_stage === 'fetch' || app.failure_stage === 'validation' || app.failure_stage === 'apply' || app.failure_stage === 'create') {
|
||||
return {
|
||||
...identity,
|
||||
status: 'source_failed',
|
||||
failureStage: app.failure_stage,
|
||||
failureClass: app.failure_class ?? app.failure_stage,
|
||||
failureAt: app.failure_at ?? 0,
|
||||
retryAt: app.retry_at,
|
||||
retryCount: app.retry_count,
|
||||
};
|
||||
}
|
||||
if (app.retry_at) {
|
||||
return { ...identity, status: 'source_retry_scheduled', retryAt: app.retry_at, retryCount: app.retry_count };
|
||||
}
|
||||
const store = GitOpsStore.getInstance();
|
||||
if (app.candidate_generation_id) {
|
||||
const generation = store.getGeneration(app.candidate_generation_id);
|
||||
// A candidate only counts as ready when its generation row exists under
|
||||
// this application and the application's materialization fingerprint is
|
||||
// still the one the generation was built from; applyStarted refuses
|
||||
// anything else, so a dangling or foreign row is a reconcile problem,
|
||||
// not a ready one.
|
||||
if (!generation || generation.application_id !== app.id) {
|
||||
limitations.push({
|
||||
code: 'candidate_generation_invalid',
|
||||
message: 'candidate generation row is missing or belongs to another application',
|
||||
evidence: app.candidate_generation_id,
|
||||
});
|
||||
return { ...identity, status: 'source_reconcile_required' };
|
||||
}
|
||||
if (generation.materialization_fingerprint !== app.materialization_fingerprint) {
|
||||
return { ...identity, status: 'source_reconcile_required' };
|
||||
}
|
||||
if (app.candidate_plan_blocked === 1) return { ...identity, status: 'source_conflict_blocker' };
|
||||
if (app.review_required === 1) return { ...identity, status: 'source_review_pending' };
|
||||
return { ...identity, status: 'candidate_ready' };
|
||||
}
|
||||
if (app.accepted_generation_id) {
|
||||
const accepted = store.getGeneration(app.accepted_generation_id);
|
||||
// An acceptance only counts when its generation row exists under this
|
||||
// application; missing evidence cannot establish the fingerprint and sha
|
||||
// agreement (the latter when a desired commit is configured) that the
|
||||
// success claim rests on, so a dangling or foreign pointer is a reconcile
|
||||
// problem, not an accepted one.
|
||||
if (!accepted || accepted.application_id !== app.id) {
|
||||
limitations.push({
|
||||
code: 'accepted_generation_invalid',
|
||||
message: 'accepted generation row is missing or belongs to another application',
|
||||
evidence: app.accepted_generation_id,
|
||||
});
|
||||
return { ...identity, status: 'source_reconcile_required' };
|
||||
}
|
||||
if (
|
||||
!app.desired_commit_sha
|
||||
|| accepted.materialization_fingerprint !== app.materialization_fingerprint
|
||||
|| accepted.commit_sha !== app.desired_commit_sha
|
||||
) {
|
||||
return { ...identity, status: 'source_reconcile_required' };
|
||||
}
|
||||
return { ...identity, status: 'application_generation_accepted' };
|
||||
}
|
||||
return { ...identity, status: 'never_reconciled' };
|
||||
}
|
||||
|
||||
function sourceIdentity(app: GitOpsApplicationRow, limitations: GitOpsLimitation[]): SourceIdentityFields {
|
||||
let repoIdentity = { host: '', pathname: '' };
|
||||
if (app.repo_identity_json) {
|
||||
try {
|
||||
const parsed = JSON.parse(app.repo_identity_json) as { host?: unknown; pathname?: unknown };
|
||||
if (typeof parsed.host === 'string' && typeof parsed.pathname === 'string') {
|
||||
repoIdentity = { host: parsed.host, pathname: parsed.pathname };
|
||||
} else {
|
||||
limitations.push({ code: 'repo_identity_invalid', message: 'repo identity json is invalid', evidence: null });
|
||||
}
|
||||
} catch {
|
||||
limitations.push({ code: 'repo_identity_invalid', message: 'repo identity json is invalid', evidence: null });
|
||||
}
|
||||
}
|
||||
return {
|
||||
configuredRepoUrl: app.configured_repo_url ?? '',
|
||||
repoIdentity,
|
||||
configuredRef: app.configured_ref ?? '',
|
||||
desiredCommitSha: app.desired_commit_sha,
|
||||
fetchedCommitSha: app.fetched_commit_sha,
|
||||
candidateGenerationId: app.candidate_generation_id,
|
||||
acceptedGenerationId: app.accepted_generation_id,
|
||||
};
|
||||
}
|
||||
|
||||
function deriveArtifact(
|
||||
app: GitOpsApplicationRow,
|
||||
generationId: string | null,
|
||||
expectedId: string | null,
|
||||
latestId: string | null,
|
||||
limitations: GitOpsLimitation[],
|
||||
): ArtifactFacet {
|
||||
if (app.target_mode === 'inline_blueprint' || !generationId) return { status: 'not_applicable' };
|
||||
const store = GitOpsStore.getInstance();
|
||||
const expected = expectedId ? toExpected(store, expectedId, limitations) : null;
|
||||
if (!latestId) {
|
||||
return {
|
||||
status: 'artifact_unresolved',
|
||||
generationId,
|
||||
expected,
|
||||
latestEvidence: null,
|
||||
limitation: 'artifact_pointer_missing',
|
||||
};
|
||||
}
|
||||
const latestRow = store.getArtifactSet(latestId);
|
||||
if (!latestRow) {
|
||||
limitations.push({ code: 'artifact_pointer_missing', message: 'latest artifact row is missing', evidence: latestId });
|
||||
return {
|
||||
status: 'artifact_unresolved',
|
||||
generationId,
|
||||
expected,
|
||||
latestEvidence: null,
|
||||
limitation: 'artifact_pointer_missing',
|
||||
};
|
||||
}
|
||||
let latestEvidence: ArtifactLatestEvidence;
|
||||
try {
|
||||
const decoded = decodeArtifactEvidenceJson(latestRow.evidence_json);
|
||||
latestEvidence = {
|
||||
artifactSetId: latestRow.id,
|
||||
evidenceVersion: latestRow.evidence_version,
|
||||
qualification: latestRow.qualification,
|
||||
identity: 'identity' in decoded ? decoded.identity : null,
|
||||
};
|
||||
} catch {
|
||||
limitations.push({ code: 'artifact_evidence_json_invalid', message: 'latest artifact evidence is invalid', evidence: latestId });
|
||||
latestEvidence = {
|
||||
artifactSetId: latestRow.id,
|
||||
evidenceVersion: latestRow.evidence_version,
|
||||
qualification: latestRow.qualification,
|
||||
identity: null,
|
||||
};
|
||||
return {
|
||||
status: 'artifact_unresolved',
|
||||
artifactSetId: latestRow.id,
|
||||
generationId,
|
||||
evidenceVersion: latestRow.evidence_version,
|
||||
qualification: latestRow.qualification,
|
||||
freshnessAt: latestRow.created_at,
|
||||
expected,
|
||||
latestEvidence,
|
||||
};
|
||||
}
|
||||
const status = artifactStatus(latestRow.qualification, expected, latestEvidence);
|
||||
return {
|
||||
status,
|
||||
artifactSetId: latestRow.id,
|
||||
generationId,
|
||||
evidenceVersion: latestRow.evidence_version,
|
||||
qualification: latestRow.qualification,
|
||||
freshnessAt: latestRow.created_at,
|
||||
expected,
|
||||
latestEvidence,
|
||||
};
|
||||
}
|
||||
|
||||
function artifactStatus(
|
||||
qualification: ArtifactQualification,
|
||||
expected: ArtifactExpectedIdentity | null,
|
||||
latest: ArtifactLatestEvidence,
|
||||
): Exclude<ArtifactFacet, { status: 'not_applicable' } | { latestEvidence: null }>['status'] {
|
||||
if (qualification === 'unresolved') return expected ? 'artifact_resolution_pending' : 'artifact_unresolved';
|
||||
if (qualification === 'stale') return 'artifact_stale';
|
||||
if (qualification === 'unavailable') return 'artifact_unavailable';
|
||||
if (qualification === 'local_build_unverified') return 'artifact_local_build_unverified';
|
||||
if (
|
||||
expected
|
||||
&& (expected.qualification === 'exact' || expected.qualification === 'qualified')
|
||||
&& latest.identity
|
||||
&& expected.identity
|
||||
&& latest.identity !== expected.identity
|
||||
) {
|
||||
return 'artifact_identity_changed';
|
||||
}
|
||||
return qualification === 'qualified' ? 'artifact_qualified' : 'artifact_exact';
|
||||
}
|
||||
|
||||
function toExpected(
|
||||
store: GitOpsStore,
|
||||
id: string,
|
||||
limitations: GitOpsLimitation[],
|
||||
): ArtifactExpectedIdentity | null {
|
||||
const row = store.getArtifactSet(id);
|
||||
if (!row) {
|
||||
limitations.push({ code: 'artifact_pointer_missing', message: 'expected artifact row is missing', evidence: id });
|
||||
return null;
|
||||
}
|
||||
try {
|
||||
const decoded = decodeArtifactEvidenceJson(row.evidence_json);
|
||||
return {
|
||||
artifactSetId: row.id,
|
||||
evidenceVersion: row.evidence_version,
|
||||
qualification: row.qualification,
|
||||
identity: 'identity' in decoded ? decoded.identity : null,
|
||||
};
|
||||
} catch {
|
||||
limitations.push({ code: 'artifact_evidence_json_invalid', message: 'expected artifact evidence is invalid', evidence: id });
|
||||
return {
|
||||
artifactSetId: row.id,
|
||||
evidenceVersion: row.evidence_version,
|
||||
qualification: row.qualification,
|
||||
identity: null,
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
function derivePlacement(app: GitOpsApplicationRow): PlacementFacet {
|
||||
if (app.target_mode === 'direct') return { status: 'unbound_direct' };
|
||||
if (!app.intent_revision_id) return { status: 'unknown', limitation: 'missing_intent' };
|
||||
if (app.legacy_combined_approval_ref && !app.placement_approval_ref) {
|
||||
return { status: 'placement_review_pending' };
|
||||
}
|
||||
return { status: 'blueprint_bound', completion: 'unknown' };
|
||||
}
|
||||
|
||||
function deriveRollout(app: GitOpsApplicationRow, targets: GitOpsTargetProjection[]): RolloutFacet {
|
||||
if (app.recovery_phase === 'restoring' || app.recovery_phase === 'compensating') {
|
||||
return { status: 'rollback_in_progress', recoveryRef: app.recovery_ref ?? '', recoveryGenerationId: null };
|
||||
}
|
||||
const failed = targets.find((target) => target.runtime.status === 'recovery_failed');
|
||||
if (failed && failed.runtime.status === 'recovery_failed') {
|
||||
return {
|
||||
status: 'rollback_partial_failed',
|
||||
recoveryRef: failed.runtime.recoveryRef ?? app.recovery_ref ?? '',
|
||||
recoveryGenerationId: failed.runtime.recoveryGenerationId,
|
||||
failureClass: failed.runtime.failureClass,
|
||||
failureAt: failed.runtime.failureAt,
|
||||
};
|
||||
}
|
||||
if (targets.some((target) => target.connectivity === 'unreachable')) return { status: 'target_unreachable' };
|
||||
if (targets.some((target) => target.connectivity === 'stale')) return { status: 'target_stale' };
|
||||
if (app.pause_at) return { status: 'rollout_paused', pauseAt: app.pause_at, pauseReason: app.pause_reason };
|
||||
if (app.partial_json) return { status: 'partially_rolled_out', partial: app.partial_json };
|
||||
if (app.target_mode === 'direct') return { status: 'not_applicable' };
|
||||
if (app.rollout_candidate_id) return { status: 'rollout_not_executable', rolloutCandidateId: app.rollout_candidate_id };
|
||||
return { status: 'not_applicable' };
|
||||
}
|
||||
|
||||
function deriveTarget(
|
||||
app: GitOpsApplicationRow,
|
||||
target: GitOpsTargetCurrentRow,
|
||||
healthDisabled: boolean,
|
||||
limitations: GitOpsLimitation[],
|
||||
): GitOpsTargetProjection {
|
||||
let connectivity: GitOpsTargetProjection['connectivity'] = 'unknown';
|
||||
if (
|
||||
target.connectivity === 'unknown'
|
||||
|| target.connectivity === 'reachable'
|
||||
|| target.connectivity === 'unreachable'
|
||||
|| target.connectivity === 'stale'
|
||||
) {
|
||||
connectivity = target.connectivity;
|
||||
} else if (target.connectivity) {
|
||||
limitations.push({ code: 'connectivity_invalid', message: 'stored connectivity is illegal', evidence: target.connectivity });
|
||||
}
|
||||
mergePersistedLimitations(target.evidence_limitations_json, limitations);
|
||||
const observed = decodeObservedSafe(target.observed_artifact_identity_json, limitations);
|
||||
const artifact = deriveArtifact(app, target.desired_generation_id, target.expected_artifact_set_id, target.latest_artifact_set_id, limitations);
|
||||
const runtime = deriveRuntime(target, artifact, observed, healthDisabled);
|
||||
return {
|
||||
nodeId: target.node_id,
|
||||
stackName: app.stack_name,
|
||||
desiredGenerationId: target.desired_generation_id,
|
||||
candidateGenerationId: target.candidate_generation_id,
|
||||
appliedGenerationId: target.applied_generation_id,
|
||||
deployedGenerationId: target.deployed_generation_id,
|
||||
healthyGenerationId: target.healthy_generation_id,
|
||||
lkgGenerationId: target.lkg_generation_id,
|
||||
lkgArtifactSetId: target.lkg_artifact_set_id,
|
||||
lkgUnavailableAt: target.lkg_unavailable_at,
|
||||
lkgUnavailableReason: target.lkg_unavailable_reason,
|
||||
expectedArtifactSetId: target.expected_artifact_set_id,
|
||||
latestArtifactSetId: target.latest_artifact_set_id,
|
||||
artifact,
|
||||
observedArtifactIdentity: observed,
|
||||
intentRevisionId: target.intent_revision_id,
|
||||
rolloutCandidateId: target.rollout_candidate_id,
|
||||
rolloutGenerationId: target.rollout_generation_id,
|
||||
approvals: {
|
||||
sourceAcceptanceRef: target.source_acceptance_ref,
|
||||
placementApprovalRef: target.placement_approval_ref,
|
||||
rolloutAuthorizationRef: target.rollout_authorization_ref,
|
||||
legacyCombinedApprovalRef: target.legacy_combined_approval_ref,
|
||||
},
|
||||
connectivity,
|
||||
legacyAppliedRevision: target.legacy_applied_revision,
|
||||
runtime,
|
||||
health: deriveHealth(target, healthDisabled),
|
||||
lkg: deriveLkg(target, limitations),
|
||||
tombstoned: target.target_status === 'tombstoned',
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* The runtime status each Blueprint observation stage projects as.
|
||||
*
|
||||
* The reconciler records what it saw against the target rather than acting on
|
||||
* it, so this is the only route those observations have into a derived status.
|
||||
*
|
||||
* Two type obligations, and they pull in opposite directions. The declared type
|
||||
* is keyed on an open string because the value looked up is `latest_stage`,
|
||||
* which holds whichever stage was recorded last: anything that is not an
|
||||
* observation must be absent here and fall through to the states below, which
|
||||
* is exactly how a later transition supersedes an earlier observation. The
|
||||
* `satisfies` closes the other side, making the map total over the stages the
|
||||
* reconciler can actually record, so a new observation stage that nothing
|
||||
* projects fails this build instead of silently reading as never applied.
|
||||
*
|
||||
* The `| undefined` is load-bearing: this project does not set
|
||||
* `noUncheckedIndexedAccess`, so without it a miss would type as a status and
|
||||
* the guard at the call site would look like dead code.
|
||||
*/
|
||||
type ObservationRuntimeStatus = 'pending_state_review' | 'evict_blocked' | 'drifted' | 'correcting';
|
||||
|
||||
const BLUEPRINT_OBSERVATION_STATUS: Record<string, ObservationRuntimeStatus | undefined> = {
|
||||
blueprint_state_review: 'pending_state_review',
|
||||
blueprint_evict_blocked: 'evict_blocked',
|
||||
blueprint_drifted: 'drifted',
|
||||
blueprint_correcting: 'correcting',
|
||||
} satisfies Record<BlueprintObservationStage, ObservationRuntimeStatus>;
|
||||
|
||||
function deriveRuntime(
|
||||
target: GitOpsTargetCurrentRow,
|
||||
artifact: ArtifactFacet,
|
||||
observed: ReturnType<typeof decodeObservedSafe>,
|
||||
healthDisabled: boolean,
|
||||
): RuntimeFacet {
|
||||
if (target.target_status === 'tombstoned') return { status: 'tombstoned' };
|
||||
if (target.recovery_phase === 'restoring' || target.recovery_phase === 'compensating') {
|
||||
return { status: 'recovery_required' };
|
||||
}
|
||||
if (target.recovery_phase === 'failed' || target.failure_stage === 'recovery') {
|
||||
return {
|
||||
status: 'recovery_failed',
|
||||
recoveryRef: target.recovery_ref,
|
||||
recoveryGenerationId: target.recovery_generation_id,
|
||||
failureClass: target.failure_class ?? 'unknown',
|
||||
failureAt: target.failure_at ?? 0,
|
||||
};
|
||||
}
|
||||
if (target.active_operation_stage === 'deploy_started') return { status: 'deploying' };
|
||||
if (
|
||||
target.interruption_stage === 'deploy_started'
|
||||
|| target.interruption_stage === 'blueprint_deploy_started'
|
||||
|| target.interruption_stage === 'blueprint_withdraw_started'
|
||||
) {
|
||||
return {
|
||||
status: 'completion_unknown',
|
||||
interruptedStage: target.interruption_stage,
|
||||
interruptedAt: target.interruption_at ?? 0,
|
||||
interruptedOperationId: target.interruption_operation_id,
|
||||
interruptedGenerationId: target.interruption_generation_id,
|
||||
interruptedIntentRevisionId: target.interruption_intent_revision_id,
|
||||
interruptedRolloutCandidateId: target.interruption_rollout_candidate_id,
|
||||
};
|
||||
}
|
||||
if (target.pause_at) return { status: 'paused', pauseAt: target.pause_at, pauseReason: target.pause_reason };
|
||||
if (target.partial_json) return { status: 'partially_rolled_out' };
|
||||
if (target.failure_stage === 'deploy' && (target.failure_class === 'pre_mutation' || target.failure_class === 'unbound')) {
|
||||
return { status: 'failed_previous_workload_intact' };
|
||||
}
|
||||
if (target.failure_stage === 'deploy' && target.failure_class === 'post_mutation') {
|
||||
return { status: 'failed_after_mutation' };
|
||||
}
|
||||
// Placed after every state a live, interrupted or failed mutation puts the
|
||||
// target in, and before the applied and deployed pointer checks. So an
|
||||
// observation cannot mask an in-flight deploy or a failure, but does outrank
|
||||
// pointers that predate it. The case that is easy to miss is the last one: a
|
||||
// target with no applied generation that has been observed now reports what
|
||||
// was seen rather than `never_applied`, which is what a deployed Blueprint
|
||||
// that drifted used to report.
|
||||
const blueprintStage = BLUEPRINT_OBSERVATION_STATUS[target.latest_stage ?? ''];
|
||||
if (blueprintStage) return { status: blueprintStage };
|
||||
if (!target.applied_generation_id) return { status: 'never_applied' };
|
||||
if (!target.deployed_generation_id) return { status: 'applied_not_deployed' };
|
||||
// The target's contract is its desired generation, so a populated deployed
|
||||
// pointer alone proves nothing: a newer applied generation with the old one
|
||||
// still running stays deploy-pending, or a stack awaiting its deploy would
|
||||
// read as synced and healthy off the previous workload's pointers. A null
|
||||
// desired id is the unknown case (legacy rows, recovered targets), where the
|
||||
// deployed pointer remains the only basis to judge.
|
||||
if (
|
||||
target.desired_generation_id !== null
|
||||
&& target.deployed_generation_id !== target.desired_generation_id
|
||||
) {
|
||||
return { status: 'applied_not_deployed' };
|
||||
}
|
||||
if (artifact.status !== 'not_applicable' && 'expected' in artifact && artifact.expected
|
||||
&& (artifact.expected.qualification === 'exact' || artifact.expected.qualification === 'qualified')) {
|
||||
if (
|
||||
observed.kind === 'unknown'
|
||||
|| observed.kind === 'missing'
|
||||
|| observed.kind === 'unavailable'
|
||||
|| observed.kind === 'stale'
|
||||
|| observed.kind === 'local_build_unverified'
|
||||
) {
|
||||
return { status: 'artifact_verification_pending' };
|
||||
}
|
||||
if (
|
||||
(observed.kind === 'exact' || observed.kind === 'qualified')
|
||||
&& artifact.expected.identity
|
||||
&& observed.identity !== artifact.expected.identity
|
||||
) {
|
||||
return { status: 'runtime_artifact_drift' };
|
||||
}
|
||||
}
|
||||
if (target.retry_at) return { status: 'retry_scheduled' };
|
||||
if (healthDisabled) return { status: 'synced_and_healthy' };
|
||||
if (target.healthy_generation_id === target.deployed_generation_id) return { status: 'synced_and_healthy' };
|
||||
return { status: 'fully_deployed_health_pending' };
|
||||
}
|
||||
|
||||
function deriveHealth(target: GitOpsTargetCurrentRow, healthDisabled: boolean): HealthFacet {
|
||||
if (healthDisabled) return { status: 'not_applicable' };
|
||||
if (!target.deployed_generation_id) return { status: 'unbound' };
|
||||
// A passing run answers for the generation the target was asked to run, so
|
||||
// it is judged against the desired id and only falls back to the deployed
|
||||
// pointer when no desired id is recorded. Judging against whatever is
|
||||
// deployed would let the previous workload's green run vouch for a newer
|
||||
// generation nobody has watched.
|
||||
const expectedGeneration = target.desired_generation_id ?? target.deployed_generation_id;
|
||||
if (target.healthy_generation_id === expectedGeneration) {
|
||||
return { status: 'passed', runId: '', deployedGenerationId: target.deployed_generation_id };
|
||||
}
|
||||
return { status: 'pending', runId: null };
|
||||
}
|
||||
|
||||
function deriveLkg(target: GitOpsTargetCurrentRow, limitations: GitOpsLimitation[]): LkgFacet {
|
||||
if (!target.lkg_generation_id && !target.lkg_unavailable_at) return { status: 'none' };
|
||||
if (target.lkg_unavailable_at) return { status: 'unavailable' };
|
||||
const generation = target.lkg_generation_id
|
||||
? GitOpsStore.getInstance().getGeneration(target.lkg_generation_id)
|
||||
: undefined;
|
||||
if (target.lkg_generation_id && !generation) {
|
||||
limitations.push({ code: 'lkg_generation_missing', message: 'LKG generation row is gone', evidence: target.lkg_generation_id });
|
||||
return { status: 'unavailable' };
|
||||
}
|
||||
if (target.lkg_artifact_set_id) {
|
||||
const artifact = GitOpsStore.getInstance().getArtifactSet(target.lkg_artifact_set_id);
|
||||
if (!artifact || artifact.generation_id !== target.lkg_generation_id) {
|
||||
limitations.push({ code: 'lkg_artifact_invalid', message: 'captured LKG artifact is invalid', evidence: target.lkg_artifact_set_id });
|
||||
return { status: 'available', generationId: target.lkg_generation_id!, artifactSetId: target.lkg_artifact_set_id };
|
||||
}
|
||||
if (artifact.qualification === 'qualified') {
|
||||
return { status: 'qualified', generationId: target.lkg_generation_id!, artifactSetId: artifact.id };
|
||||
}
|
||||
return { status: 'available', generationId: target.lkg_generation_id!, artifactSetId: artifact.id };
|
||||
}
|
||||
return { status: 'available', generationId: target.lkg_generation_id!, artifactSetId: null };
|
||||
}
|
||||
|
||||
/**
|
||||
* Application-level conditions under which deploying is withheld outright: an
|
||||
* operation or recovery already owns the stack, so nothing may start a deploy
|
||||
* even where a target's own state would make one legal.
|
||||
*/
|
||||
function appDeployWithheld(app: GitOpsApplicationRow): boolean {
|
||||
return app.active_operation_stage === 'fetch_started'
|
||||
|| app.active_operation_stage === 'apply_started'
|
||||
|| app.recovery_phase === 'restoring'
|
||||
|| app.recovery_phase === 'compensating'
|
||||
|| app.recovery_phase === 'failed'
|
||||
|| app.failure_stage === 'recovery';
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether deploying this exact target is legal right now, per the revision-state
|
||||
* plan's available-action rules. Deliberately per target: the application-wide
|
||||
* action list is a union across targets, so keying an item to it would tell a
|
||||
* paused or failed sibling to deploy because a healthy sibling diverged.
|
||||
*
|
||||
* Direct targets converge a known divergence outright; an interrupted deploy is
|
||||
* retried only against the generation still applied, exactly what deployStarted
|
||||
* will demand. Writers keep applied and desired equal today, so keying on
|
||||
* applied can never advertise an action the transition would refuse.
|
||||
*
|
||||
* Targets of Blueprint modes have no Direct deploy at all: their sole retry
|
||||
* repeats an interrupted deploy or withdraw, legal only while both persisted
|
||||
* identities equal what the application currently requires. An absent pair
|
||||
* counts as matching: rollout candidates come from a later-phase producer, so
|
||||
* inline Blueprints carry no candidate id on either side yet, and demanding one
|
||||
* here would leave every interrupted inline deploy permanently unactionable. A
|
||||
* superseded value on either side fails the comparison.
|
||||
*
|
||||
* disk_invocation_drift is deliberately absent: no producer reaches that
|
||||
* status in this slice, and until one lands the predicate fails safe to a
|
||||
* reported mismatch with no deploy recommendation.
|
||||
*/
|
||||
function targetDeployLegal(app: GitOpsApplicationRow, target: GitOpsTargetProjection): boolean {
|
||||
if (appDeployWithheld(app) || target.tombstoned) return false;
|
||||
if (app.target_mode !== 'direct') {
|
||||
if (target.runtime.status !== 'completion_unknown') return false;
|
||||
const stage = target.runtime.interruptedStage;
|
||||
if (stage !== 'blueprint_deploy_started' && stage !== 'blueprint_withdraw_started') return false;
|
||||
return (
|
||||
target.runtime.interruptedIntentRevisionId === app.intent_revision_id
|
||||
&& target.runtime.interruptedRolloutCandidateId === app.rollout_candidate_id
|
||||
);
|
||||
}
|
||||
if (target.runtime.status === 'applied_not_deployed') return true;
|
||||
return (
|
||||
target.runtime.status === 'completion_unknown'
|
||||
&& target.runtime.interruptedStage === 'deploy_started'
|
||||
&& target.runtime.interruptedGenerationId !== null
|
||||
&& target.runtime.interruptedGenerationId === target.appliedGenerationId
|
||||
);
|
||||
}
|
||||
|
||||
function deriveActions(
|
||||
app: GitOpsApplicationRow,
|
||||
source: SourceFacet,
|
||||
placement: PlacementFacet,
|
||||
targets: GitOpsTargetProjection[],
|
||||
): GitOpsAvailableAction[] {
|
||||
if (source.status === 'applying' || source.status === 'checking_fetching') return ['none'];
|
||||
if (source.status === 'recovery_required' || source.status === 'recovery_failed') return ['none'];
|
||||
const actions = new Set<GitOpsAvailableAction>();
|
||||
// Fetch is offered only to live Direct applications: Blueprint Git-source
|
||||
// integration ships later, until then no Blueprint mode advertises fetch.
|
||||
if (
|
||||
app.target_mode === 'direct'
|
||||
&& (
|
||||
source.status === 'never_reconciled'
|
||||
|| source.status === 'source_reconcile_required'
|
||||
|| source.status === 'source_retry_scheduled'
|
||||
|| (source.status === 'source_unknown' && source.interruptedStage === 'fetch_started')
|
||||
|| (source.status === 'source_failed' && (source.failureStage === 'fetch' || source.failureStage === 'validation'))
|
||||
)
|
||||
) {
|
||||
actions.add('fetch');
|
||||
}
|
||||
if (source.status === 'candidate_ready') actions.add('apply');
|
||||
// An interrupted apply may be finished only while its recorded generation is
|
||||
// still the current candidate and nothing has since suspended the source or
|
||||
// blocked that candidate, all of which applyStarted would refuse. No shipped
|
||||
// producer pairs blockage with a matching interruption today, because
|
||||
// blocking always mints a fresh candidate id; these clauses hold the gate to
|
||||
// the transition table's contract regardless of what future producers do.
|
||||
if (
|
||||
source.status === 'source_unknown'
|
||||
&& source.interruptedStage === 'apply_started'
|
||||
&& source.interruptedGenerationId !== null
|
||||
&& source.interruptedGenerationId === app.candidate_generation_id
|
||||
&& !app.suspended_at
|
||||
&& app.candidate_plan_blocked !== 1
|
||||
) {
|
||||
// applyStarted also demands the candidate generation exist under this
|
||||
// application with an unchanged materialization fingerprint; prove all of
|
||||
// it here rather than recommend a transition that would refuse.
|
||||
const candidate = GitOpsStore.getInstance().getGeneration(app.candidate_generation_id);
|
||||
if (
|
||||
candidate !== undefined
|
||||
&& candidate.application_id === app.id
|
||||
&& candidate.materialization_fingerprint === app.materialization_fingerprint
|
||||
) {
|
||||
actions.add('apply');
|
||||
}
|
||||
}
|
||||
if (app.candidate_generation_id && !app.active_operation_stage) actions.add('dismiss');
|
||||
if (targets.some((target) => targetDeployLegal(app, target))) actions.add('deploy');
|
||||
if (placement.status === 'placement_review_pending') actions.add('approve_legacy');
|
||||
if (actions.size === 0) return ['none'];
|
||||
return Array.from(actions);
|
||||
}
|
||||
|
||||
/**
|
||||
* Fold the limitations a writer recorded into the ones derived here.
|
||||
*
|
||||
* These cannot be re-derived: they describe evidence that was dropped because
|
||||
* it could not be proven, and once dropped the row looks the same as one that
|
||||
* never had it. Decoded fail-closed, so a corrupt record surfaces as its own
|
||||
* limitation rather than disappearing.
|
||||
*/
|
||||
function mergePersistedLimitations(raw: string | null, limitations: GitOpsLimitation[]): void {
|
||||
if (!raw) return;
|
||||
try {
|
||||
for (const item of decodeGitOpsEvidenceLimitations(raw)) {
|
||||
limitations.push({
|
||||
code: item.code,
|
||||
message: 'evidence recorded at write time could not be proven',
|
||||
evidence: item.detail,
|
||||
});
|
||||
}
|
||||
} catch (err) {
|
||||
limitations.push({
|
||||
code: 'evidence_limitations_invalid',
|
||||
message: err instanceof Error ? err.message : String(err),
|
||||
evidence: raw,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
function decodeObservedSafe(
|
||||
raw: string | null,
|
||||
limitations: GitOpsLimitation[],
|
||||
): ReturnType<typeof decodeObservedArtifactIdentity> {
|
||||
try {
|
||||
return decodeObservedArtifactIdentity(raw);
|
||||
} catch (err) {
|
||||
// Any failure here means the runtime observation is unusable, so it must
|
||||
// surface as a limitation. Returning a clean 'unknown' without one would
|
||||
// read as "nothing observed yet" and quietly downgrade a real artifact
|
||||
// drift to a pending check.
|
||||
limitations.push({
|
||||
code: err instanceof GitOpsJsonError ? 'artifact_observation_invalid' : 'artifact_observation_decode_failed',
|
||||
message: err instanceof Error ? err.message : String(err),
|
||||
evidence: raw,
|
||||
});
|
||||
return { kind: 'unknown' };
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* The not-applicable shape, carrying why an application we expected was absent.
|
||||
*
|
||||
* Distinct from `NOT_APPLICABLE_REVISION` on purpose. That one means "nothing
|
||||
* here", which is the honest answer for a stack or Blueprint the model was
|
||||
* never asked about. This one means "something should have been here and was
|
||||
* not", which is a fault. Returning the shared sentinel for both would make a
|
||||
* vanished row indistinguishable from one that never existed, and the reader
|
||||
* has no third source to tell them apart.
|
||||
*/
|
||||
function unreachableApplicationRevision(limitation: GitOpsLimitation): GitOpsRevisionProjection {
|
||||
return {
|
||||
schemaVersion: 1,
|
||||
targetMode: 'not_applicable',
|
||||
applicationId: null,
|
||||
facets: null,
|
||||
targets: [],
|
||||
drift: [],
|
||||
limitations: [limitation],
|
||||
availableActions: [],
|
||||
approvals: null,
|
||||
};
|
||||
}
|
||||
|
||||
function missingApplicationRevision(applicationId: string): GitOpsRevisionProjection {
|
||||
return unreachableApplicationRevision({
|
||||
code: 'application_row_missing',
|
||||
message: 'The application this projection was resolved from is no longer present.',
|
||||
evidence: { applicationId },
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* A Blueprint proven to manage a stack directory, with no application row.
|
||||
*
|
||||
* Its own code, not `application_row_missing`, because the evidence differs:
|
||||
* there is no application id to name, only the Blueprint and the stack whose
|
||||
* deployment row proved the ownership.
|
||||
*/
|
||||
export function missingBlueprintApplicationRevision(blueprintId: number, stackName: string): GitOpsRevisionProjection {
|
||||
return unreachableApplicationRevision({
|
||||
code: 'blueprint_application_missing',
|
||||
message: 'A Blueprint deployed this stack but has no live application to describe it.',
|
||||
evidence: { blueprintId, stackName },
|
||||
});
|
||||
}
|
||||
|
||||
export function projectApplication(applicationId: string, healthDisabled: boolean): GitOpsRevisionProjection {
|
||||
const store = GitOpsStore.getInstance();
|
||||
const application = store.getApplication(applicationId);
|
||||
// The caller resolved this id from a row it had just read, so a miss here is
|
||||
// not "no application": it is a row that went away between the two reads,
|
||||
// which are deliberately not in one transaction. Say so rather than reporting
|
||||
// the same answer an unmodelled stack gets.
|
||||
if (!application) return missingApplicationRevision(applicationId);
|
||||
return deriveGitOpsRevision({
|
||||
application,
|
||||
targets: store.listTargets(applicationId),
|
||||
healthDisabled,
|
||||
}, null);
|
||||
}
|
||||
@@ -0,0 +1,248 @@
|
||||
import { randomUUID } from 'crypto';
|
||||
import path from 'path';
|
||||
import { NodeRegistry } from '../NodeRegistry';
|
||||
import { MANAGED_ROOT_NAME } from './managedPaths';
|
||||
import { encodeGitOpsJson } from './json';
|
||||
import { materializationFingerprint } from './fingerprint';
|
||||
import { parseHttpsRepoUrl, parseLegacyRepoUrl, secretFreeRepoUrl, serializeRepoIdentity, type RepoIdentity } from './repoIdentity';
|
||||
import type {
|
||||
GitOpsApplicationRow,
|
||||
GitOpsCreateCheckpointRow,
|
||||
GitOpsGenerationRow,
|
||||
} from './types';
|
||||
|
||||
/** The material source configuration a Direct application is bound to. */
|
||||
export type DirectSourceConfig = {
|
||||
repoUrl: string;
|
||||
branch: string;
|
||||
composePaths: readonly string[];
|
||||
contextDir: string | null;
|
||||
syncEnv: boolean;
|
||||
envPath: string | null;
|
||||
};
|
||||
|
||||
export type DirectSourceIdentity = {
|
||||
/** Secret-free `https://host/pathname`, safe to persist and to project. */
|
||||
repoUrl: string;
|
||||
identity: RepoIdentity;
|
||||
fingerprint: string;
|
||||
};
|
||||
|
||||
export class GitOpsIdentityError extends Error {
|
||||
constructor(message: string) {
|
||||
super(message);
|
||||
this.name = 'GitOpsIdentityError';
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Derive the storable identity and materialization fingerprint for a source.
|
||||
*
|
||||
* The fingerprint is what later decides whether a staged candidate still
|
||||
* matches the configuration it was built from, so it is computed from the same
|
||||
* secret-free identity that gets persisted, never from the raw operational URL.
|
||||
*/
|
||||
export function directSourceIdentity(config: DirectSourceConfig): DirectSourceIdentity {
|
||||
const parsed = parseHttpsRepoUrl(config.repoUrl);
|
||||
if (!parsed.ok) throw new GitOpsIdentityError(`repository URL is not storable: ${parsed.reason}`);
|
||||
return directSourceIdentityFromUrl(config, parsed.url);
|
||||
}
|
||||
|
||||
/**
|
||||
* The same derivation, for URLs that predate strict ingress.
|
||||
*
|
||||
* Migration is its only caller. A legacy operational row may still carry
|
||||
* userinfo or a query string that fetch needs, so the storable identity strips
|
||||
* them instead of refusing the stack; the strict helper above stays the gate
|
||||
* for every path a user can drive.
|
||||
*/
|
||||
export function migrationDirectSourceIdentity(config: DirectSourceConfig): DirectSourceIdentity {
|
||||
const parsed = parseLegacyRepoUrl(config.repoUrl);
|
||||
if (!parsed.ok) throw new GitOpsIdentityError(`repository URL is not storable: ${parsed.reason}`);
|
||||
return directSourceIdentityFromUrl(config, parsed.url);
|
||||
}
|
||||
|
||||
function directSourceIdentityFromUrl(config: DirectSourceConfig, url: URL): DirectSourceIdentity {
|
||||
const identity = serializeRepoIdentity(url);
|
||||
const material = {
|
||||
repoIdentity: identity,
|
||||
configuredRef: config.branch,
|
||||
composePaths: config.composePaths,
|
||||
contextDir: config.contextDir,
|
||||
syncEnv: config.syncEnv,
|
||||
envPath: config.envPath,
|
||||
};
|
||||
return {
|
||||
repoUrl: secretFreeRepoUrl(identity),
|
||||
identity,
|
||||
fingerprint: materializationFingerprint(material),
|
||||
};
|
||||
}
|
||||
|
||||
/** Absolute managed root for one stack on the local node. */
|
||||
export function stackManagedRoot(stackName: string): string {
|
||||
const dataDir = process.env.DATA_DIR || path.join(process.cwd(), 'data');
|
||||
const nodeId = NodeRegistry.getInstance().getDefaultNodeId();
|
||||
return path.join(dataDir, MANAGED_ROOT_NAME, String(nodeId), stackName);
|
||||
}
|
||||
|
||||
export function newGitOpsId(): string {
|
||||
return randomUUID();
|
||||
}
|
||||
|
||||
/**
|
||||
* Build a Direct application row.
|
||||
*
|
||||
* `creating` is for create-from-Git, where the stack does not exist yet and the
|
||||
* checkpoint decides what happens if the process dies. `active` is for linking
|
||||
* a stack that already exists: there is nothing to recover, so it is live from
|
||||
* the moment the source row commits.
|
||||
*/
|
||||
export function buildDirectApplicationRow(args: {
|
||||
id: string;
|
||||
stackName: string;
|
||||
config: DirectSourceConfig;
|
||||
identity: DirectSourceIdentity;
|
||||
lifecycleStatus: 'creating' | 'active';
|
||||
at: number;
|
||||
}): GitOpsApplicationRow {
|
||||
return {
|
||||
id: args.id,
|
||||
lifecycle_key: `direct:${args.stackName}`,
|
||||
lifecycle_status: args.lifecycleStatus,
|
||||
target_mode: 'direct',
|
||||
stack_name: args.stackName,
|
||||
blueprint_id: null,
|
||||
configured_repo_url: args.identity.repoUrl,
|
||||
repo_identity_json: encodeGitOpsJson(args.identity.identity),
|
||||
configured_ref: args.config.branch,
|
||||
compose_paths_json: encodeGitOpsJson([...args.config.composePaths]),
|
||||
context_dir: args.config.contextDir,
|
||||
sync_env: args.config.syncEnv ? 1 : 0,
|
||||
env_path: args.config.syncEnv ? args.config.envPath : null,
|
||||
materialization_fingerprint: args.identity.fingerprint,
|
||||
desired_commit_sha: null,
|
||||
fetched_commit_sha: null,
|
||||
candidate_generation_id: null,
|
||||
accepted_generation_id: null,
|
||||
candidate_plan_blocked: 0,
|
||||
review_required: 0,
|
||||
artifact_set_id: null,
|
||||
latest_artifact_set_id: null,
|
||||
intent_revision_id: null,
|
||||
rollout_candidate_id: null,
|
||||
rollout_generation_id: null,
|
||||
source_acceptance_ref: null,
|
||||
placement_approval_ref: null,
|
||||
rollout_authorization_ref: null,
|
||||
legacy_combined_approval_ref: null,
|
||||
preflight_fingerprint: null,
|
||||
latest_operation_id: null,
|
||||
active_operation_id: null,
|
||||
active_operation_stage: null,
|
||||
active_operation_at: null,
|
||||
active_generation_id: null,
|
||||
pause_at: null,
|
||||
pause_reason: null,
|
||||
partial_json: null,
|
||||
failure_stage: null,
|
||||
failure_class: null,
|
||||
failure_at: null,
|
||||
retry_at: null,
|
||||
retry_count: 0,
|
||||
suspended_at: null,
|
||||
recovery_ref: null,
|
||||
recovery_phase: null,
|
||||
interruption_stage: null,
|
||||
interruption_at: null,
|
||||
interruption_operation_id: null,
|
||||
interruption_generation_id: null,
|
||||
evidence_fresh_at: null,
|
||||
evidence_limitations_json: null,
|
||||
created_at: args.at,
|
||||
updated_at: args.at,
|
||||
};
|
||||
}
|
||||
|
||||
export function buildGenerationRow(args: {
|
||||
id: string;
|
||||
applicationId: string;
|
||||
commitSha: string;
|
||||
identity: DirectSourceIdentity;
|
||||
configuredRef: string;
|
||||
candidateRelPath: string;
|
||||
appliedRelPath: string;
|
||||
manifestVersion: number;
|
||||
expectedInvocation: unknown;
|
||||
changePlanFingerprint: string | null;
|
||||
operationId: string;
|
||||
trigger: string;
|
||||
actor: string | null;
|
||||
at: number;
|
||||
/** A blocked change plan is recorded, but such a generation can never apply. */
|
||||
planBlocked?: boolean;
|
||||
}): GitOpsGenerationRow {
|
||||
return {
|
||||
id: args.id,
|
||||
application_id: args.applicationId,
|
||||
commit_sha: args.commitSha,
|
||||
repo_url: args.identity.repoUrl,
|
||||
configured_ref: args.configuredRef,
|
||||
repo_identity_json: encodeGitOpsJson(args.identity.identity),
|
||||
manifest_version: args.manifestVersion,
|
||||
candidate_dir: args.candidateRelPath,
|
||||
applied_dir: args.appliedRelPath,
|
||||
expected_invocation_json: encodeGitOpsJson(args.expectedInvocation),
|
||||
materialization_fingerprint: args.identity.fingerprint,
|
||||
validation_ok: 1,
|
||||
plan_blocked: args.planBlocked ? 1 : 0,
|
||||
change_plan_fingerprint: args.changePlanFingerprint,
|
||||
operation_id: args.operationId,
|
||||
trigger: args.trigger,
|
||||
actor: args.actor,
|
||||
previous_generation_id: null,
|
||||
redacted_limitations_json: '[]',
|
||||
created_at: args.at,
|
||||
};
|
||||
}
|
||||
|
||||
export function buildCreateCheckpointRow(args: {
|
||||
applicationId: string;
|
||||
stackName: string;
|
||||
operationId: string;
|
||||
config: DirectSourceConfig;
|
||||
identity: DirectSourceIdentity;
|
||||
authType: string;
|
||||
encryptedToken: string | null;
|
||||
autoApplyOnWebhook: boolean;
|
||||
autoDeployOnApply: boolean;
|
||||
commitSha: string;
|
||||
createdManagedRoot: boolean;
|
||||
at: number;
|
||||
}): GitOpsCreateCheckpointRow {
|
||||
return {
|
||||
application_id: args.applicationId,
|
||||
stack_name: args.stackName,
|
||||
phase: 'pre_stack',
|
||||
generation_id: null,
|
||||
operation_id: args.operationId,
|
||||
// Operational URL for fetch compatibility during create. Copies into
|
||||
// generations and history always go through the secret-free identity.
|
||||
repo_url: args.config.repoUrl,
|
||||
branch: args.config.branch,
|
||||
compose_path: args.config.composePaths[0] ?? '',
|
||||
compose_paths_json: encodeGitOpsJson([...args.config.composePaths]),
|
||||
context_dir: args.config.contextDir,
|
||||
sync_env: args.config.syncEnv ? 1 : 0,
|
||||
env_path: args.config.syncEnv ? args.config.envPath : null,
|
||||
auth_type: args.authType,
|
||||
encrypted_token: args.encryptedToken,
|
||||
auto_apply_on_webhook: args.autoApplyOnWebhook ? 1 : 0,
|
||||
auto_deploy_on_apply: args.autoDeployOnApply ? 1 : 0,
|
||||
commit_sha: args.commitSha,
|
||||
applied_spec_json: null,
|
||||
created_managed_root: args.createdManagedRoot ? 1 : 0,
|
||||
created_at: args.at,
|
||||
updated_at: args.at,
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,39 @@
|
||||
import { createHash } from 'crypto';
|
||||
import { encodeGitOpsJson } from './json';
|
||||
import type { RepoIdentity } from './repoIdentity';
|
||||
|
||||
export type MaterialConfigInput = {
|
||||
repoIdentity: RepoIdentity;
|
||||
configuredRef: string;
|
||||
composePaths: readonly string[];
|
||||
contextDir: string | null;
|
||||
syncEnv: boolean;
|
||||
envPath: string | null;
|
||||
};
|
||||
|
||||
function emptyToNull(value: string | null | undefined): string | null {
|
||||
if (value === null || value === undefined) return null;
|
||||
const trimmed = value.trim();
|
||||
return trimmed.length === 0 ? null : trimmed;
|
||||
}
|
||||
|
||||
export function canonicalMaterialConfigJson(input: MaterialConfigInput): string {
|
||||
const contextDir = emptyToNull(input.contextDir);
|
||||
const syncEnv = input.syncEnv === true;
|
||||
const envPath = syncEnv ? emptyToNull(input.envPath) : null;
|
||||
return encodeGitOpsJson({
|
||||
composePaths: [...input.composePaths],
|
||||
contextDir,
|
||||
syncEnv,
|
||||
envPath,
|
||||
repoIdentity: {
|
||||
host: input.repoIdentity.host,
|
||||
pathname: input.repoIdentity.pathname,
|
||||
},
|
||||
configuredRef: input.configuredRef,
|
||||
});
|
||||
}
|
||||
|
||||
export function materializationFingerprint(input: MaterialConfigInput): string {
|
||||
return createHash('sha256').update(canonicalMaterialConfigJson(input)).digest('hex');
|
||||
}
|
||||
@@ -0,0 +1,430 @@
|
||||
import { randomUUID } from 'crypto';
|
||||
import type Database from 'better-sqlite3';
|
||||
import { decodeGitOpsJson, encodeGitOpsJson, isRecord, GitOpsJsonError } from './json';
|
||||
import { enqueueHistoryPublication } from './publish';
|
||||
import { sanitizeForLog } from '../../utils/safeLog';
|
||||
import type {
|
||||
GitOpsApplicationRow,
|
||||
GitOpsApprovalRefs,
|
||||
GitOpsHistoryRow,
|
||||
GitOpsLimitation,
|
||||
GitOpsTargetMode,
|
||||
} from './types';
|
||||
|
||||
export type HistoryOutcome = GitOpsHistoryRow['outcome'];
|
||||
|
||||
/**
|
||||
* Every stage a transition can record.
|
||||
*
|
||||
* The column itself is open TEXT, because a stage is an audit label rather than
|
||||
* a state and a future producer must be able to add one without a migration.
|
||||
* This union constrains the *writers*: it is what makes the set finite at
|
||||
* compile time, so the metrics keyspace is bounded by the type rather than by
|
||||
* whatever strings happen to reach the insert. Adding a producer stage without
|
||||
* adding it here fails the build, which is the point.
|
||||
*/
|
||||
export type GitOpsHistoryStage =
|
||||
| 'application_activated'
|
||||
| 'application_tombstoned'
|
||||
| 'applied'
|
||||
| 'apply_failed'
|
||||
| 'apply_started'
|
||||
| 'artifact_evidence_recorded'
|
||||
| 'artifact_expectation_accepted'
|
||||
| 'blueprint_ack_recorded'
|
||||
| 'blueprint_correcting'
|
||||
| 'blueprint_deploy_failed'
|
||||
| 'blueprint_deploy_started'
|
||||
| 'blueprint_drifted'
|
||||
| 'blueprint_evict_blocked'
|
||||
| 'blueprint_state_review'
|
||||
| 'blueprint_withdraw_failed'
|
||||
| 'blueprint_withdraw_started'
|
||||
| 'blueprint_withdrawn'
|
||||
| 'candidate_ready'
|
||||
| 'candidate_superseded'
|
||||
| 'config_changed_pending_cleared'
|
||||
| 'create_failed'
|
||||
| 'deploy_bound'
|
||||
| 'deploy_failed'
|
||||
| 'deploy_started'
|
||||
| 'deploy_unbound'
|
||||
| 'dismissed'
|
||||
| 'fetch_failed'
|
||||
| 'fetch_started'
|
||||
| 'fetched'
|
||||
| 'fetched_invalid'
|
||||
| 'health_finalized'
|
||||
| 'intent_revised'
|
||||
| 'operation_interrupted'
|
||||
| 'partial_cleared'
|
||||
| 'partially_rolled_out'
|
||||
| 'recovery_failed'
|
||||
| 'recovery_started'
|
||||
| 'recovery_succeeded'
|
||||
| 'rollback_completed'
|
||||
| 'rollback_in_progress'
|
||||
| 'rollback_partial_failed'
|
||||
| 'rollout_candidate_opened'
|
||||
| 'rollout_paused'
|
||||
| 'rollout_unpaused'
|
||||
| 'source_conflict_blocker'
|
||||
| 'source_retry_scheduled'
|
||||
| 'source_suspended'
|
||||
| 'source_unsuspended'
|
||||
| 'target_tombstoned';
|
||||
|
||||
export type HistoryInsert = {
|
||||
application: GitOpsApplicationRow;
|
||||
nodeId: number | null;
|
||||
dedupeTarget: string;
|
||||
operationId: string;
|
||||
stage: GitOpsHistoryStage;
|
||||
outcome: HistoryOutcome;
|
||||
trigger: string;
|
||||
actor: string | null;
|
||||
// Objects, not `unknown`: the read path reports a non-object payload as an
|
||||
// unreadable audit record, so a producer must not be able to author one.
|
||||
before: Record<string, unknown>;
|
||||
after: Record<string, unknown>;
|
||||
generationId?: string | null;
|
||||
artifactSetId?: string | null;
|
||||
commitSha?: string | null;
|
||||
intentRevisionId?: string | null;
|
||||
rolloutCandidateId?: string | null;
|
||||
sourceAcceptanceRef?: string | null;
|
||||
placementApprovalRef?: string | null;
|
||||
rolloutAuthorizationRef?: string | null;
|
||||
legacyCombinedApprovalRef?: string | null;
|
||||
requiredTargetsJson?: string | null;
|
||||
recoveryRef?: string | null;
|
||||
redactedReasonClass?: string | null;
|
||||
at: number;
|
||||
};
|
||||
|
||||
/**
|
||||
* Append one history row, returning null when this exact operation already
|
||||
* wrote its row (a replay).
|
||||
*
|
||||
* The conflict clause names the dedupe index deliberately rather than using
|
||||
* `INSERT OR IGNORE`: `OR IGNORE` also swallows NOT NULL and CHECK violations,
|
||||
* which would drop an audit row while the state change committed and report it
|
||||
* to the caller as a harmless replay. Only a duplicate of the dedupe tuple is
|
||||
* tolerated here; every other constraint failure throws and rolls the
|
||||
* transaction back. Callers turn a null return into `replayed: true`, so the
|
||||
* dedupe index is load-bearing for idempotency, not just for storage hygiene.
|
||||
*
|
||||
* An inserted row is also queued for announcement here rather than at the
|
||||
* transition call sites, because this is the only place that can tell an
|
||||
* insert from a replay: the callers see a null and turn it into `replayed`,
|
||||
* by which point the distinction has already been made once.
|
||||
*/
|
||||
export function insertHistory(db: Database.Database, row: HistoryInsert): string | null {
|
||||
const id = randomUUID();
|
||||
const result = db.prepare(
|
||||
`INSERT INTO gitops_history (
|
||||
id, created_at, application_id, target_mode, lifecycle_key, stack_name, blueprint_id,
|
||||
node_id, dedupe_target, repo_url, configured_ref, repo_identity_json, commit_sha,
|
||||
generation_id, artifact_set_id, intent_revision_id, rollout_candidate_id, rollout_generation_id,
|
||||
source_acceptance_ref, placement_approval_ref, rollout_authorization_ref,
|
||||
legacy_combined_approval_ref, operation_id, stage, outcome, trigger, actor,
|
||||
before_json, after_json, required_targets_json, validation_json, per_target_results_json,
|
||||
health_run_id, health_snapshot_json, invocation_observed_json, recovery_ref, redacted_reason_class
|
||||
) VALUES (${Array(37).fill('?').join(', ')})
|
||||
ON CONFLICT(application_id, operation_id, stage, dedupe_target) DO NOTHING`,
|
||||
).run(
|
||||
id,
|
||||
row.at,
|
||||
row.application.id,
|
||||
row.application.target_mode as GitOpsTargetMode,
|
||||
row.application.lifecycle_key,
|
||||
row.application.stack_name,
|
||||
row.application.blueprint_id,
|
||||
row.nodeId,
|
||||
row.dedupeTarget,
|
||||
row.application.configured_repo_url,
|
||||
row.application.configured_ref,
|
||||
row.application.repo_identity_json,
|
||||
row.commitSha ?? row.application.desired_commit_sha,
|
||||
row.generationId ?? null,
|
||||
row.artifactSetId ?? null,
|
||||
row.intentRevisionId ?? row.application.intent_revision_id,
|
||||
row.rolloutCandidateId ?? row.application.rollout_candidate_id,
|
||||
row.application.rollout_generation_id,
|
||||
row.sourceAcceptanceRef ?? row.application.source_acceptance_ref,
|
||||
row.placementApprovalRef ?? row.application.placement_approval_ref,
|
||||
row.rolloutAuthorizationRef ?? row.application.rollout_authorization_ref,
|
||||
row.legacyCombinedApprovalRef ?? row.application.legacy_combined_approval_ref,
|
||||
row.operationId,
|
||||
row.stage,
|
||||
row.outcome,
|
||||
row.trigger,
|
||||
row.actor,
|
||||
encodeGitOpsJson(row.before),
|
||||
encodeGitOpsJson(row.after),
|
||||
row.requiredTargetsJson ?? null,
|
||||
null,
|
||||
null,
|
||||
null,
|
||||
null,
|
||||
null,
|
||||
row.recoveryRef ?? null,
|
||||
row.redactedReasonClass ?? null,
|
||||
);
|
||||
if (result.changes !== 1) return null;
|
||||
enqueueHistoryPublication({
|
||||
db,
|
||||
id,
|
||||
stage: row.stage,
|
||||
outcome: row.outcome,
|
||||
applicationId: row.application.id,
|
||||
targetMode: row.application.target_mode,
|
||||
stackName: row.application.stack_name,
|
||||
blueprintId: row.application.blueprint_id,
|
||||
nodeId: row.nodeId,
|
||||
at: row.at,
|
||||
});
|
||||
return id;
|
||||
}
|
||||
|
||||
/** Page size when the caller does not ask for one. */
|
||||
export const HISTORY_DEFAULT_LIMIT = 50;
|
||||
/** Hard ceiling on page size, whatever the caller asks for. */
|
||||
export const HISTORY_MAX_LIMIT = 100;
|
||||
/**
|
||||
* Rows examined per request before the page is cut short.
|
||||
*
|
||||
* Authorization runs per row after the query, so a caller with narrow grants
|
||||
* could otherwise walk the whole table looking for rows they may read. This
|
||||
* bounds that per-row loop, and the cursor still advances past every examined
|
||||
* row so the next request resumes rather than rescanning. It is not a bound on
|
||||
* database work: the query itself is bounded by the index over
|
||||
* `(created_at, id)`, not by this constant.
|
||||
*/
|
||||
export const HISTORY_SCAN_CAP = 1000;
|
||||
|
||||
export type GitOpsHistoryFilters = {
|
||||
applicationId?: string;
|
||||
stackName?: string;
|
||||
/** Secret-free `repo_url`. Credentials never reach this column. */
|
||||
repoIdentity?: string;
|
||||
configuredRef?: string;
|
||||
commitSha?: string;
|
||||
generationId?: string;
|
||||
artifactSetId?: string;
|
||||
blueprintId?: number;
|
||||
rolloutCandidateId?: string;
|
||||
rolloutGenerationId?: string;
|
||||
nodeId?: number;
|
||||
trigger?: string;
|
||||
actor?: string;
|
||||
outcome?: HistoryOutcome;
|
||||
};
|
||||
|
||||
/**
|
||||
* One history row as the API returns it.
|
||||
*
|
||||
* `before` and `after` are the producer's delta for that transition, not a full
|
||||
* revision projection: each transition records only the fields it moved. They
|
||||
* are display evidence. Authorization never reads them, so a row whose JSON is
|
||||
* unreadable still returns its identity, stage, and outcome alongside a
|
||||
* `history_json_invalid` limitation rather than vanishing from the audit trail.
|
||||
*/
|
||||
export type GitOpsHistoryItem = {
|
||||
id: string;
|
||||
createdAt: number;
|
||||
applicationId: string;
|
||||
targetMode: GitOpsTargetMode;
|
||||
stackName: string | null;
|
||||
/** Secret-free repository URL as recorded with the event, matching the `repoIdentity` filter. */
|
||||
repoIdentity: string | null;
|
||||
/** Ref as configured when the event ran, matching the `configuredRef` filter. */
|
||||
configuredRef: string | null;
|
||||
blueprintId: number | null;
|
||||
nodeId: number | null;
|
||||
commitSha: string | null;
|
||||
generationId: string | null;
|
||||
artifactSetId: string | null;
|
||||
intentRevisionId: string | null;
|
||||
rolloutCandidateId: string | null;
|
||||
rolloutGenerationId: string | null;
|
||||
approvals: GitOpsApprovalRefs;
|
||||
operationId: string;
|
||||
stage: string;
|
||||
outcome: HistoryOutcome;
|
||||
trigger: string;
|
||||
actor: string | null;
|
||||
before: Record<string, unknown> | null;
|
||||
after: Record<string, unknown> | null;
|
||||
limitations: GitOpsLimitation[];
|
||||
};
|
||||
|
||||
/**
|
||||
* Page cursor over the `(created_at, id)` ordering.
|
||||
*
|
||||
* Both halves are needed: rows written by one transaction share a single
|
||||
* timestamp by construction, so paginating on the timestamp alone would drop or
|
||||
* repeat rows at a page boundary.
|
||||
*
|
||||
* Encoded in plain text, so treat it as readable and forgeable rather than
|
||||
* opaque. That is tolerable because it only picks a start position in a scan
|
||||
* whose rows are authorized individually afterwards, but it does mean the
|
||||
* cursor discloses one row's timestamp and id to a caller who may not read that
|
||||
* row.
|
||||
*/
|
||||
export type GitOpsHistoryCursor = { createdAt: number; id: string };
|
||||
|
||||
export function encodeHistoryCursor(cursor: GitOpsHistoryCursor): string {
|
||||
return `${cursor.createdAt}.${cursor.id}`;
|
||||
}
|
||||
|
||||
/** History ids are minted with `randomUUID()`, so anything else is not one. */
|
||||
const HISTORY_ID_RE = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/;
|
||||
|
||||
/**
|
||||
* Parse a caller-supplied cursor, returning null for anything malformed.
|
||||
*
|
||||
* Both halves are validated. The id half matters as much as the timestamp: it
|
||||
* is compared as a string in the page query, so an unvalidated one would not
|
||||
* fail, it would silently include or exclude an arbitrary slice of the rows
|
||||
* sharing that millisecond. Callers turn null into a 400 rather than starting
|
||||
* over, because quietly serving page one to someone who asked to resume is the
|
||||
* kind of wrong answer an audit reader cannot detect.
|
||||
*/
|
||||
export function decodeHistoryCursor(raw: string): GitOpsHistoryCursor | null {
|
||||
const separator = raw.indexOf('.');
|
||||
if (separator <= 0 || separator === raw.length - 1) return null;
|
||||
const createdAt = Number(raw.slice(0, separator));
|
||||
if (!Number.isSafeInteger(createdAt) || createdAt < 0) return null;
|
||||
const id = raw.slice(separator + 1);
|
||||
if (!HISTORY_ID_RE.test(id)) return null;
|
||||
return { createdAt, id };
|
||||
}
|
||||
|
||||
/**
|
||||
* Decode one history JSON column into its delta object.
|
||||
*
|
||||
* `null` means the column could not be read as an object. The caller turns that
|
||||
* into the row's `history_json_invalid` limitation so the entry survives, and
|
||||
* the reason is logged here because a corrupt audit column is a storage problem
|
||||
* an operator needs to see rather than a routine response variation.
|
||||
*/
|
||||
function decodeHistoryDelta(rowId: string, column: string, raw: string): Record<string, unknown> | null {
|
||||
let value: unknown;
|
||||
try {
|
||||
value = decodeGitOpsJson(raw);
|
||||
} catch (error) {
|
||||
if (!(error instanceof GitOpsJsonError)) throw error;
|
||||
console.error(`[GitOps] history ${sanitizeForLog(rowId)}.${column} is not decodable JSON: ${error.message}`);
|
||||
return null;
|
||||
}
|
||||
if (!isRecord(value)) {
|
||||
console.error(`[GitOps] history ${sanitizeForLog(rowId)}.${column} decoded to ${typeof value}, expected an object`);
|
||||
return null;
|
||||
}
|
||||
return value;
|
||||
}
|
||||
|
||||
export function toHistoryItem(row: GitOpsHistoryRow): GitOpsHistoryItem {
|
||||
const before = decodeHistoryDelta(row.id, 'before', row.before_json);
|
||||
const after = decodeHistoryDelta(row.id, 'after', row.after_json);
|
||||
const limitations: GitOpsLimitation[] = [];
|
||||
if (before === null || after === null) {
|
||||
limitations.push({
|
||||
code: 'history_json_invalid',
|
||||
message: 'Recorded change detail for this entry could not be read.',
|
||||
evidence: { before: before === null, after: after === null },
|
||||
});
|
||||
}
|
||||
return {
|
||||
id: row.id,
|
||||
createdAt: row.created_at,
|
||||
applicationId: row.application_id,
|
||||
targetMode: row.target_mode,
|
||||
stackName: row.stack_name,
|
||||
repoIdentity: row.repo_url,
|
||||
configuredRef: row.configured_ref,
|
||||
blueprintId: row.blueprint_id,
|
||||
nodeId: row.node_id,
|
||||
commitSha: row.commit_sha,
|
||||
generationId: row.generation_id,
|
||||
artifactSetId: row.artifact_set_id,
|
||||
intentRevisionId: row.intent_revision_id,
|
||||
rolloutCandidateId: row.rollout_candidate_id,
|
||||
rolloutGenerationId: row.rollout_generation_id,
|
||||
approvals: {
|
||||
sourceAcceptanceRef: row.source_acceptance_ref,
|
||||
placementApprovalRef: row.placement_approval_ref,
|
||||
rolloutAuthorizationRef: row.rollout_authorization_ref,
|
||||
legacyCombinedApprovalRef: row.legacy_combined_approval_ref,
|
||||
},
|
||||
operationId: row.operation_id,
|
||||
stage: row.stage,
|
||||
outcome: row.outcome,
|
||||
trigger: row.trigger,
|
||||
actor: row.actor,
|
||||
before,
|
||||
after,
|
||||
limitations,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Read one scan window of history rows, newest first.
|
||||
*
|
||||
* Returns raw rows rather than a finished page because authorization is decided
|
||||
* per row by the caller, which owns the permission context. The caller stops
|
||||
* once its page is full and uses the last row it *examined* (not the last it
|
||||
* kept) as the next cursor, so skipped rows are never revisited.
|
||||
*/
|
||||
export function queryHistoryRows(
|
||||
db: Database.Database,
|
||||
filters: GitOpsHistoryFilters,
|
||||
cursor: GitOpsHistoryCursor | null,
|
||||
scanLimit: number,
|
||||
): GitOpsHistoryRow[] {
|
||||
const clauses: string[] = [];
|
||||
const params: Array<string | number> = [];
|
||||
const eq = (column: string, value: string | number | undefined): void => {
|
||||
if (value === undefined) return;
|
||||
clauses.push(`${column} = ?`);
|
||||
params.push(value);
|
||||
};
|
||||
|
||||
eq('application_id', filters.applicationId);
|
||||
eq('stack_name', filters.stackName);
|
||||
eq('repo_url', filters.repoIdentity);
|
||||
eq('configured_ref', filters.configuredRef);
|
||||
eq('commit_sha', filters.commitSha);
|
||||
eq('generation_id', filters.generationId);
|
||||
eq('artifact_set_id', filters.artifactSetId);
|
||||
eq('blueprint_id', filters.blueprintId);
|
||||
eq('rollout_candidate_id', filters.rolloutCandidateId);
|
||||
// Distinct columns on purpose: a candidate is a proposal, a generation is a
|
||||
// rollout that ran. Answering one filter from the other's column would report
|
||||
// a proposal as executed.
|
||||
eq('rollout_generation_id', filters.rolloutGenerationId);
|
||||
// A node-scoped page keeps application-level rows: activation and similar
|
||||
// stages carry no node, and a plain `node_id = ?` would make a proxied hub
|
||||
// view read as if the application never came into being.
|
||||
if (filters.nodeId !== undefined) {
|
||||
clauses.push('(node_id = ? OR node_id IS NULL)');
|
||||
params.push(filters.nodeId);
|
||||
}
|
||||
eq('trigger', filters.trigger);
|
||||
eq('actor', filters.actor);
|
||||
eq('outcome', filters.outcome);
|
||||
|
||||
if (cursor) {
|
||||
clauses.push('(created_at < ? OR (created_at = ? AND id < ?))');
|
||||
params.push(cursor.createdAt, cursor.createdAt, cursor.id);
|
||||
}
|
||||
|
||||
const where = clauses.length ? `WHERE ${clauses.join(' AND ')}` : '';
|
||||
params.push(scanLimit);
|
||||
return db.prepare(
|
||||
`SELECT * FROM gitops_history ${where}
|
||||
ORDER BY created_at DESC, id DESC
|
||||
LIMIT ?`,
|
||||
).all(...params) as GitOpsHistoryRow[];
|
||||
}
|
||||
@@ -0,0 +1,303 @@
|
||||
export class GitOpsJsonError extends Error {
|
||||
constructor(message: string) {
|
||||
super(message);
|
||||
this.name = 'GitOpsJsonError';
|
||||
}
|
||||
}
|
||||
|
||||
export function encodeGitOpsJson(value: unknown): string {
|
||||
let encoded: string | undefined;
|
||||
try {
|
||||
encoded = JSON.stringify(value);
|
||||
} catch (err) {
|
||||
const message = err instanceof Error ? err.message : String(err);
|
||||
throw new GitOpsJsonError(`gitops json encode failed: ${message}`);
|
||||
}
|
||||
// JSON.stringify returns undefined (it does not throw) for undefined, a
|
||||
// function, or a symbol. Every JSON column is NOT NULL, so letting that
|
||||
// through would bind SQL NULL and lose the row.
|
||||
if (typeof encoded !== 'string') {
|
||||
throw new GitOpsJsonError('gitops json encode produced no output');
|
||||
}
|
||||
return encoded;
|
||||
}
|
||||
|
||||
export function decodeGitOpsJson(raw: string): unknown {
|
||||
try {
|
||||
return JSON.parse(raw);
|
||||
} catch {
|
||||
throw new GitOpsJsonError('gitops json decode failed');
|
||||
}
|
||||
}
|
||||
|
||||
export function isRecord(value: unknown): value is Record<string, unknown> {
|
||||
return typeof value === 'object' && value !== null && !Array.isArray(value);
|
||||
}
|
||||
|
||||
export function isFiniteInteger(value: unknown): value is number {
|
||||
return typeof value === 'number' && Number.isInteger(value) && Number.isFinite(value);
|
||||
}
|
||||
|
||||
export function isPositiveInteger(value: unknown): value is number {
|
||||
return isFiniteInteger(value) && value > 0;
|
||||
}
|
||||
|
||||
export const PREFLIGHT_FINGERPRINT_RE = /^[0-9a-f]{64}$/;
|
||||
|
||||
export function isPreflightFingerprint(value: unknown): value is string {
|
||||
return typeof value === 'string' && PREFLIGHT_FINGERPRINT_RE.test(value);
|
||||
}
|
||||
|
||||
export type GitOpsRequiredTargetsJson = { nodeIds: number[] };
|
||||
|
||||
export function canonicalizeNodeIds(nodeIds: readonly number[]): number[] {
|
||||
return Array.from(new Set(nodeIds)).sort((a, b) => a - b);
|
||||
}
|
||||
|
||||
export function decodeGitOpsRequiredTargetsJson(raw: string): GitOpsRequiredTargetsJson {
|
||||
const decoded = decodeGitOpsJson(raw);
|
||||
if (!isRecord(decoded)) {
|
||||
throw new GitOpsJsonError('required_targets_json must be an object');
|
||||
}
|
||||
const keys = Object.keys(decoded);
|
||||
if (keys.length !== 1 || keys[0] !== 'nodeIds') {
|
||||
throw new GitOpsJsonError('required_targets_json must have only nodeIds');
|
||||
}
|
||||
if (!Array.isArray(decoded.nodeIds)) {
|
||||
throw new GitOpsJsonError('required_targets_json.nodeIds must be an array');
|
||||
}
|
||||
const nodeIds: number[] = [];
|
||||
for (const item of decoded.nodeIds) {
|
||||
if (!isFiniteInteger(item)) {
|
||||
throw new GitOpsJsonError('required_targets_json.nodeIds must be integers');
|
||||
}
|
||||
nodeIds.push(item);
|
||||
}
|
||||
const canonical = canonicalizeNodeIds(nodeIds);
|
||||
if (canonical.length !== nodeIds.length) {
|
||||
throw new GitOpsJsonError('required_targets_json.nodeIds must be unique');
|
||||
}
|
||||
for (let i = 0; i < nodeIds.length; i += 1) {
|
||||
if (nodeIds[i] !== canonical[i]) {
|
||||
throw new GitOpsJsonError('required_targets_json.nodeIds must be sorted unique');
|
||||
}
|
||||
}
|
||||
return { nodeIds };
|
||||
}
|
||||
|
||||
export function encodeGitOpsRequiredTargetsJson(nodeIds: readonly number[]): string {
|
||||
const canonical = canonicalizeNodeIds(nodeIds);
|
||||
if (canonical.length !== nodeIds.length) {
|
||||
throw new GitOpsJsonError('required_targets_json.nodeIds must be unique');
|
||||
}
|
||||
for (let i = 0; i < nodeIds.length; i += 1) {
|
||||
if (nodeIds[i] !== canonical[i]) {
|
||||
throw new GitOpsJsonError('required_targets_json.nodeIds must be sorted unique');
|
||||
}
|
||||
if (!isFiniteInteger(nodeIds[i])) {
|
||||
throw new GitOpsJsonError('required_targets_json.nodeIds must be integers');
|
||||
}
|
||||
}
|
||||
return encodeGitOpsJson({ nodeIds: [...nodeIds] });
|
||||
}
|
||||
|
||||
/**
|
||||
* Why a writer could not prove something, recorded on the row it affected.
|
||||
*
|
||||
* Distinct from the limitations the deriver computes at read time: those are
|
||||
* re-derivable from current rows, these are facts only the transition that
|
||||
* dropped a pointer knew. Without them a pointer that was cleared because it
|
||||
* could not be proven is indistinguishable from one that never existed.
|
||||
*/
|
||||
export type GitOpsEvidenceLimitation = { code: string; detail: string | null };
|
||||
|
||||
export function decodeGitOpsEvidenceLimitations(raw: string | null): GitOpsEvidenceLimitation[] {
|
||||
if (raw === null) return [];
|
||||
const decoded = decodeGitOpsJson(raw);
|
||||
if (!Array.isArray(decoded)) {
|
||||
throw new GitOpsJsonError('evidence_limitations_json must be an array');
|
||||
}
|
||||
return decoded.map((item) => {
|
||||
if (!isRecord(item)) throw new GitOpsJsonError('evidence limitation must be an object');
|
||||
const keys = Object.keys(item);
|
||||
if (keys.length !== 2 || !('code' in item) || !('detail' in item)) {
|
||||
throw new GitOpsJsonError('evidence limitation must have exactly code and detail');
|
||||
}
|
||||
if (typeof item.code !== 'string' || item.code.length === 0) {
|
||||
throw new GitOpsJsonError('evidence limitation code must be a non-empty string');
|
||||
}
|
||||
if (item.detail !== null && typeof item.detail !== 'string') {
|
||||
throw new GitOpsJsonError('evidence limitation detail must be a string or null');
|
||||
}
|
||||
return { code: item.code, detail: item.detail };
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Replace the limitations for one code, keeping every other code intact.
|
||||
*
|
||||
* Returns null when nothing remains, so a row that has recovered its evidence
|
||||
* stores NULL rather than an empty array.
|
||||
*/
|
||||
export function encodeGitOpsEvidenceLimitations(
|
||||
existing: GitOpsEvidenceLimitation[],
|
||||
code: string,
|
||||
next: GitOpsEvidenceLimitation | null,
|
||||
): string | null {
|
||||
const kept = existing.filter((item) => item.code !== code);
|
||||
if (next) kept.push(next);
|
||||
if (kept.length === 0) return null;
|
||||
const encoded = encodeGitOpsJson(kept);
|
||||
decodeGitOpsEvidenceLimitations(encoded);
|
||||
return encoded;
|
||||
}
|
||||
|
||||
export type GitOpsApprovedTargetEffectJson = Array<{ nodeId: number; outcome: 'place' | 'remove' }>;
|
||||
|
||||
export function decodeGitOpsApprovedTargetEffectJson(raw: string): GitOpsApprovedTargetEffectJson {
|
||||
const decoded = decodeGitOpsJson(raw);
|
||||
if (!Array.isArray(decoded)) {
|
||||
throw new GitOpsJsonError('blast_json must be an array');
|
||||
}
|
||||
const out: GitOpsApprovedTargetEffectJson = [];
|
||||
const seen = new Set<number>();
|
||||
let lastNodeId = Number.NEGATIVE_INFINITY;
|
||||
for (const item of decoded) {
|
||||
if (!isRecord(item)) {
|
||||
throw new GitOpsJsonError('blast_json entries must be objects');
|
||||
}
|
||||
const keys = Object.keys(item);
|
||||
if (keys.length !== 2 || !('nodeId' in item) || !('outcome' in item)) {
|
||||
throw new GitOpsJsonError('blast_json entries must have exactly nodeId and outcome');
|
||||
}
|
||||
if (!isPositiveInteger(item.nodeId)) {
|
||||
throw new GitOpsJsonError('blast_json.nodeId must be a positive integer');
|
||||
}
|
||||
if (item.outcome !== 'place' && item.outcome !== 'remove') {
|
||||
throw new GitOpsJsonError('blast_json.outcome must be place or remove');
|
||||
}
|
||||
if (seen.has(item.nodeId)) {
|
||||
throw new GitOpsJsonError('blast_json node ids must be unique');
|
||||
}
|
||||
if (item.nodeId <= lastNodeId) {
|
||||
throw new GitOpsJsonError('blast_json must be strictly increasing by nodeId');
|
||||
}
|
||||
seen.add(item.nodeId);
|
||||
lastNodeId = item.nodeId;
|
||||
out.push({ nodeId: item.nodeId, outcome: item.outcome });
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
export function encodeGitOpsApprovedTargetEffectJson(
|
||||
effect: GitOpsApprovedTargetEffectJson,
|
||||
): string {
|
||||
const encoded = encodeGitOpsJson(effect);
|
||||
// Round-trip through the decoder so an invalid shape throws here rather than
|
||||
// reaching SQLite. The decoded value is deliberately discarded.
|
||||
decodeGitOpsApprovedTargetEffectJson(encoded);
|
||||
return encoded;
|
||||
}
|
||||
|
||||
export type ArtifactEvidenceJson =
|
||||
| { kind: 'unresolved' }
|
||||
| { kind: 'exact'; identity: string }
|
||||
| { kind: 'qualified'; identity: string }
|
||||
| { kind: 'stale'; identity: string | null }
|
||||
| { kind: 'unavailable' }
|
||||
| { kind: 'local_build_unverified'; identity: string | null };
|
||||
|
||||
function requireNonEmptyIdentity(value: unknown): string {
|
||||
if (typeof value !== 'string' || value.length === 0) {
|
||||
throw new GitOpsJsonError('artifact identity must be a non-empty string');
|
||||
}
|
||||
return value;
|
||||
}
|
||||
|
||||
export function decodeArtifactEvidenceJson(raw: string): ArtifactEvidenceJson {
|
||||
const decoded = decodeGitOpsJson(raw);
|
||||
if (!isRecord(decoded) || typeof decoded.kind !== 'string') {
|
||||
throw new GitOpsJsonError('evidence_json must have a kind');
|
||||
}
|
||||
const keys = Object.keys(decoded);
|
||||
switch (decoded.kind) {
|
||||
case 'unresolved':
|
||||
case 'unavailable':
|
||||
if (keys.length !== 1 || 'identity' in decoded) {
|
||||
throw new GitOpsJsonError(`${decoded.kind} evidence forbids identity`);
|
||||
}
|
||||
return { kind: decoded.kind };
|
||||
case 'exact':
|
||||
case 'qualified':
|
||||
if (keys.length !== 2) {
|
||||
throw new GitOpsJsonError(`${decoded.kind} evidence requires identity only`);
|
||||
}
|
||||
return { kind: decoded.kind, identity: requireNonEmptyIdentity(decoded.identity) };
|
||||
case 'stale':
|
||||
case 'local_build_unverified':
|
||||
if (keys.length !== 2 || !('identity' in decoded)) {
|
||||
throw new GitOpsJsonError(`${decoded.kind} evidence requires identity`);
|
||||
}
|
||||
if (decoded.identity !== null && typeof decoded.identity !== 'string') {
|
||||
throw new GitOpsJsonError(`${decoded.kind} identity must be string or null`);
|
||||
}
|
||||
return { kind: decoded.kind, identity: decoded.identity };
|
||||
default:
|
||||
throw new GitOpsJsonError('unknown artifact evidence kind');
|
||||
}
|
||||
}
|
||||
|
||||
export function encodeArtifactEvidenceJson(value: ArtifactEvidenceJson): string {
|
||||
const encoded = encodeGitOpsJson(value);
|
||||
// Round-trip through the decoder so an invalid shape throws here rather than
|
||||
// reaching SQLite. The decoded value is deliberately discarded.
|
||||
decodeArtifactEvidenceJson(encoded);
|
||||
return encoded;
|
||||
}
|
||||
|
||||
export type ObservedArtifactIdentity =
|
||||
| { kind: 'unknown' }
|
||||
| { kind: 'missing' }
|
||||
| { kind: 'unavailable' }
|
||||
| { kind: 'exact'; identity: string; observedAt: number }
|
||||
| { kind: 'qualified'; identity: string; observedAt: number }
|
||||
| { kind: 'stale'; identity: string; observedAt: number }
|
||||
| { kind: 'local_build_unverified'; identity: string; observedAt: number };
|
||||
|
||||
export function decodeObservedArtifactIdentity(raw: string | null): ObservedArtifactIdentity {
|
||||
if (raw === null) return { kind: 'unknown' };
|
||||
const decoded = decodeGitOpsJson(raw);
|
||||
if (!isRecord(decoded) || typeof decoded.kind !== 'string') {
|
||||
throw new GitOpsJsonError('observed artifact identity must have a kind');
|
||||
}
|
||||
const keys = Object.keys(decoded);
|
||||
switch (decoded.kind) {
|
||||
case 'unknown':
|
||||
if (keys.length !== 1) {
|
||||
throw new GitOpsJsonError('unknown observation forbids extra fields');
|
||||
}
|
||||
return { kind: 'unknown' };
|
||||
case 'missing':
|
||||
case 'unavailable':
|
||||
if (keys.length !== 1 || 'identity' in decoded) {
|
||||
throw new GitOpsJsonError(`${decoded.kind} observation forbids identity`);
|
||||
}
|
||||
return { kind: decoded.kind };
|
||||
case 'exact':
|
||||
case 'qualified':
|
||||
case 'stale':
|
||||
case 'local_build_unverified':
|
||||
if (keys.length !== 3 || !('identity' in decoded) || !('observedAt' in decoded)) {
|
||||
throw new GitOpsJsonError(`${decoded.kind} observation requires identity and observedAt`);
|
||||
}
|
||||
if (typeof decoded.identity !== 'string' || decoded.identity.length === 0) {
|
||||
throw new GitOpsJsonError(`${decoded.kind} observation identity must be a non-empty string`);
|
||||
}
|
||||
if (typeof decoded.observedAt !== 'number' || !Number.isFinite(decoded.observedAt)) {
|
||||
throw new GitOpsJsonError(`${decoded.kind} observation observedAt must be a finite number`);
|
||||
}
|
||||
return { kind: decoded.kind, identity: decoded.identity, observedAt: decoded.observedAt };
|
||||
default:
|
||||
throw new GitOpsJsonError('unknown observed artifact identity kind');
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,164 @@
|
||||
/**
|
||||
* Managed-area path vocabulary, defined here rather than imported from
|
||||
* GitProjectManifestService.
|
||||
*
|
||||
* The manifest service reaches these modules through GitSourceService, so
|
||||
* importing its constants back into the gitops layer forms a cycle. Under that
|
||||
* cycle the binding can still be uninitialized when these modules evaluate,
|
||||
* which silently yields paths like `undefined/candidate-<sha>`: they pass a
|
||||
* containment check against the managed root and name a directory that does
|
||||
* not exist, so cleanup removes nothing and leaves the real one behind.
|
||||
*
|
||||
* These values must stay in step with the manifest service's layout.
|
||||
*/
|
||||
import fs from 'fs/promises';
|
||||
import path from 'path';
|
||||
import { sanitizeForLog } from '../../utils/safeLog';
|
||||
|
||||
export const MANAGED_ROOT_NAME = 'git-managed';
|
||||
export const GENERATIONS_DIR = 'generations';
|
||||
|
||||
/**
|
||||
* The directory every stack's managed area lives under.
|
||||
*
|
||||
* Exists so each filesystem call on a managed path can resolve its target and
|
||||
* check containment in its own scope. CodeQL does not credit the wrapped
|
||||
* `isPathWithinBase` helper as a barrier, so `js/path-injection` reports the
|
||||
* call even when the path was already validated; the check has to be inline at
|
||||
* the call to be recognised. The duplication is deliberate.
|
||||
*
|
||||
* The base is node-agnostic, since callers here are given a root rather than a
|
||||
* node id. On its own it proves only that a path names somewhere inside the
|
||||
* managed area, which is why the real-path check below pins the path to its own
|
||||
* position under this base rather than to the base itself.
|
||||
*/
|
||||
export function managedAreaBase(): string {
|
||||
const dataDir = process.env.DATA_DIR || path.join(process.cwd(), 'data');
|
||||
return path.resolve(dataDir, MANAGED_ROOT_NAME);
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolve one path, keeping "is not there" apart from "could not be read".
|
||||
*
|
||||
* The distinction is the whole safety property. Collapsing both to "absent"
|
||||
* would let the walk below climb past a path it could not resolve and infer
|
||||
* containment from an ancestor, which is how a junction that throws `EPERM` or
|
||||
* `ELOOP` instead of resolving would be treated as though it were not there at
|
||||
* all. Only `ENOENT` means absent; everything else is a failure to establish
|
||||
* what a path points at, and a containment check that cannot see a path must
|
||||
* not pass it.
|
||||
*/
|
||||
async function resolveRealPath(target: string): Promise<
|
||||
| { kind: 'resolved'; real: string }
|
||||
| { kind: 'absent' }
|
||||
| { kind: 'unreadable'; code: string }
|
||||
> {
|
||||
try {
|
||||
return { kind: 'resolved', real: await fs.realpath(target) };
|
||||
} catch (error) {
|
||||
const code = (error as NodeJS.ErrnoException).code;
|
||||
if (code === 'ENOENT') return { kind: 'absent' };
|
||||
return { kind: 'unreadable', code: code ?? 'UNKNOWN' };
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether a path resolves to its own place in the managed area once links are
|
||||
* followed.
|
||||
*
|
||||
* `path.resolve` never touches the filesystem, so lexical containment says
|
||||
* nothing about a symlink or Windows junction sitting above the target: the
|
||||
* string stays inside the managed area while a recursive delete walks straight
|
||||
* out of it. Every sink that creates or removes a managed path asks this before
|
||||
* it acts: the create teardown and staging-marker sinks, and the manifest
|
||||
* service's generation pruning, boot-sweep orphan reaping, detach staging,
|
||||
* staged-area finalization, and whole-area deletion.
|
||||
*
|
||||
* The property is positional, not membership. Asking only whether the resolved
|
||||
* path lands somewhere under the managed area is satisfied by every other node
|
||||
* and every other stack in it, so a junction from one stack's `generations` into
|
||||
* another's passes while the delete takes a generation that belongs to someone
|
||||
* else. What is checked instead is that the path resolves to exactly the
|
||||
* location its own name claims: the managed area is resolved once, and the
|
||||
* segments below it must be reached without redirection.
|
||||
*
|
||||
* Resolving the area separately is what keeps an operator's relocation working.
|
||||
* Pointing the data directory at another volume moves the whole area and stays
|
||||
* legal; a link *inside* the area does not, because nothing under it has any
|
||||
* reason to live somewhere other than where it is named.
|
||||
*
|
||||
* The target itself is resolved when it exists, so a managed root or generation
|
||||
* directory that turns out to be a link elsewhere is rejected rather than
|
||||
* deleted through. Only when it is genuinely missing does the walk climb to the
|
||||
* nearest ancestor that exists, appending the absent segments lexically:
|
||||
* nothing can be linked at a path that is not there. A path that cannot be read
|
||||
* at all stops the walk and refuses, because climbing past it would infer a
|
||||
* location for the one path whose link status could not be established.
|
||||
*
|
||||
* Callers still run their own lexical containment check at their sink, because
|
||||
* the analyzer only credits a barrier it can see at the call. This does not rely
|
||||
* on them: it establishes the target's own position before resolving anything.
|
||||
*
|
||||
* What this does not close is the window between answering and acting. Node
|
||||
* exposes no directory-relative remove, so a link swapped into an intermediate
|
||||
* segment after this returns is still followed by the caller's `fs.rm`. Whoever
|
||||
* could do that already has write access inside the managed area, which is the
|
||||
* same position they would need to plant the link this rejects, so the check is
|
||||
* worth having and the window is accepted rather than overlooked.
|
||||
*/
|
||||
export async function isRealPathAtManagedLocation(target: string): Promise<boolean> {
|
||||
const areaLexical = managedAreaBase();
|
||||
const resolved = path.resolve(target);
|
||||
// The position the target claims for itself, read off the lexical path before
|
||||
// any link is followed. This is what the real path has to agree with.
|
||||
const relative = path.relative(areaLexical, resolved);
|
||||
// The area root itself has no position under the area, and a relative path
|
||||
// that climbs out never had one. `path.relative` normalizes, so `..` can only
|
||||
// lead; it never appears in the middle of what this returns.
|
||||
if (relative === '' || path.isAbsolute(relative)
|
||||
|| relative === '..' || relative.startsWith(`..${path.sep}`)) {
|
||||
return false;
|
||||
}
|
||||
|
||||
const area = await resolveRealPath(areaLexical);
|
||||
if (area.kind === 'unreadable') {
|
||||
console.warn('[GitOps] Cannot resolve the managed area (%s); refusing to act on anything under it', area.code);
|
||||
return false;
|
||||
}
|
||||
// No managed area on disk means nothing under it exists either, so there is
|
||||
// no link anywhere beneath it to be misled by, and the check above already
|
||||
// proved the target names a path inside it. A removal is then a no-op, and
|
||||
// the one non-removal sink creates the area as it goes.
|
||||
if (area.kind === 'absent') return true;
|
||||
|
||||
const expected = path.resolve(area.real, relative);
|
||||
const trailing: string[] = [];
|
||||
let probe = resolved;
|
||||
for (;;) {
|
||||
const real = await resolveRealPath(probe);
|
||||
if (real.kind === 'resolved') {
|
||||
const actual = path.resolve(real.real, ...trailing);
|
||||
if (actual === expected) return true;
|
||||
// The single most useful fact for an operator staring at a refusal, and
|
||||
// the only place it exists: the thrown message names neither path, and a
|
||||
// refusal here can hold the boot gate.
|
||||
console.warn(
|
||||
'[GitOps] %s resolves to %s, not to %s; refusing to act on it',
|
||||
sanitizeForLog(resolved), sanitizeForLog(actual), sanitizeForLog(expected),
|
||||
);
|
||||
return false;
|
||||
}
|
||||
if (real.kind === 'unreadable') {
|
||||
console.warn(
|
||||
'[GitOps] Cannot resolve %s (%s); refusing to act on it rather than assuming where it points',
|
||||
sanitizeForLog(probe), real.code,
|
||||
);
|
||||
return false;
|
||||
}
|
||||
const parent = path.dirname(probe);
|
||||
// Reached the filesystem root without finding anything that exists.
|
||||
if (parent === probe) return false;
|
||||
trailing.unshift(path.basename(probe));
|
||||
probe = parent;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,508 @@
|
||||
import { createHash } from 'crypto';
|
||||
import fs from 'fs';
|
||||
import path from 'path';
|
||||
import { DatabaseService, type Blueprint, type StackGitSource } from '../DatabaseService';
|
||||
import { FileSystemService } from '../FileSystemService';
|
||||
import { GitProjectManifestService } from '../GitProjectManifestService';
|
||||
import { NodeRegistry } from '../NodeRegistry';
|
||||
import { sanitizeForLog } from '../../utils/safeLog';
|
||||
import { encodeGitOpsEvidenceLimitations, type GitOpsEvidenceLimitation } from './json';
|
||||
import { buildDirectApplicationRow, migrationDirectSourceIdentity, newGitOpsId } from './directApplication';
|
||||
import { emptyTargetRow, GitOpsStore } from './store';
|
||||
import { GitOpsTransitions, type EventEnvelope } from './transitions';
|
||||
import { blankInlineApplication } from './blueprintProducers';
|
||||
import { evaluateEffectiveApproval, intentFingerprint } from '../blueprintApproval';
|
||||
import type { GitOpsApplicationRow, GitOpsGenerationRow, GitOpsTargetCurrentRow } from './types';
|
||||
|
||||
/** Schema version this migration writes. Bumping it replays every scope. */
|
||||
const MIGRATION_SCHEMA_VERSION = 1;
|
||||
|
||||
/**
|
||||
* What the on-disk manifest proves about a stack.
|
||||
*
|
||||
* `trusted` is the only classification that licenses a canonical commit
|
||||
* pointer, and it means the manifest parsed, validated, carries an identity
|
||||
* stamp matching the repository and ref the source row configures *now*, and
|
||||
* names the same commit the source row records as applied. The five failure
|
||||
* kinds are kept apart because they tell an operator different things, and
|
||||
* each implies a different next step: nothing was ever written, something was
|
||||
* written and is unreadable, something was written for a different repository,
|
||||
* the manifest records no commit at all, or the two records name different
|
||||
* commits.
|
||||
*
|
||||
* The last two are deliberately separate. A manifest adopted from an existing
|
||||
* directory is written with an empty commit and `state: 'migrated'`, which the
|
||||
* validator permits, so "no commit yet" is an ordinary state for a stack that
|
||||
* has never been fetched. Reporting it as a disagreement would name a commit
|
||||
* the manifest does not contain.
|
||||
*/
|
||||
type ManifestTrust =
|
||||
| { kind: 'trusted'; commitSha: string; manifestVersion: number; appliedDir: string }
|
||||
| { kind: 'absent' }
|
||||
| { kind: 'corrupt'; reason: string }
|
||||
| { kind: 'identity_invalid'; reason: string }
|
||||
| { kind: 'commit_unresolved' }
|
||||
| { kind: 'commit_mismatch'; manifestCommitSha: string };
|
||||
|
||||
export type MigrationOutcome =
|
||||
| 'skipped_current'
|
||||
| 'skipped_live_application'
|
||||
| 'migrated_accepted'
|
||||
| 'migrated_unreconciled'
|
||||
| 'tombstoned_missing_stack'
|
||||
| 'migrated_inline'
|
||||
| 'failed';
|
||||
|
||||
export type MigrationResult = { stackName: string; outcome: MigrationOutcome };
|
||||
|
||||
/**
|
||||
* Bring Git stacks that predate the revision state model into it.
|
||||
*
|
||||
* The governing rule is that a pointer is written only when the evidence proves
|
||||
* that exact generation under the repository and ref configured now. A legacy
|
||||
* applied commit is not that proof on its own: the manifest may be gone, may be
|
||||
* unreadable, may be stamped for a repository the stack no longer points at, or
|
||||
* may name a different commit than the one the source row records as applied.
|
||||
* In every one of those cases the canonical pointers stay null and the legacy
|
||||
* commit survives as recorded limitation evidence, so the projection asks for a
|
||||
* fetch instead of asserting a state nobody verified.
|
||||
*
|
||||
* Idempotent by checkpoint. Replay after a configuration change re-runs the
|
||||
* matrix but never upgrades an already-justified pointer to a stronger claim.
|
||||
*/
|
||||
export function migrateDirectGitStacks(): MigrationResult[] {
|
||||
const db = DatabaseService.getInstance();
|
||||
const results: MigrationResult[] = [];
|
||||
for (const source of db.getGitSources()) {
|
||||
try {
|
||||
results.push(migrateOne(source));
|
||||
} catch (error) {
|
||||
console.error(
|
||||
`[GitOps] Could not migrate the Git stack ${sanitizeForLog(source.stack_name)}; retrying next boot:`,
|
||||
error instanceof Error ? error.stack ?? error.message : String(error),
|
||||
);
|
||||
results.push({ stackName: source.stack_name, outcome: 'failed' });
|
||||
}
|
||||
}
|
||||
return results;
|
||||
}
|
||||
|
||||
function migrateOne(source: StackGitSource): MigrationResult {
|
||||
const store = GitOpsStore.getInstance();
|
||||
const stackName = source.stack_name;
|
||||
// Migration-only identity derivation: a legacy operational URL may still
|
||||
// carry userinfo or a query string that fetch needs, so the storable
|
||||
// identity strips them and the source row is never rewritten.
|
||||
const identity = migrationDirectSourceIdentity({
|
||||
repoUrl: source.repo_url,
|
||||
branch: source.branch,
|
||||
composePaths: source.compose_paths,
|
||||
contextDir: source.context_dir,
|
||||
syncEnv: source.sync_env,
|
||||
envPath: source.env_path,
|
||||
});
|
||||
|
||||
const scope = `direct:${stackName}`;
|
||||
const checkpoint = store.getMigrationCheckpoint(scope);
|
||||
if (
|
||||
checkpoint
|
||||
&& checkpoint.schema_version === MIGRATION_SCHEMA_VERSION
|
||||
&& checkpoint.fingerprint === identity.fingerprint
|
||||
) {
|
||||
return { stackName, outcome: 'skipped_current' };
|
||||
}
|
||||
|
||||
// A stack created through the new path already describes itself. Migration
|
||||
// never touches it: its pointers were written with proof this pass does not
|
||||
// have.
|
||||
if (store.getLiveDirectApplication(stackName)) {
|
||||
store.upsertMigrationCheckpoint(scope, MIGRATION_SCHEMA_VERSION, identity.fingerprint, Date.now());
|
||||
return { stackName, outcome: 'skipped_live_application' };
|
||||
}
|
||||
|
||||
const trust = classifyManifest(stackName, source);
|
||||
const limitations = collectLimitations(source, trust);
|
||||
const nodeId = NodeRegistry.getInstance().getDefaultNodeId();
|
||||
const stackPresent = stackDirectoryPresent(stackName, nodeId);
|
||||
const at = Date.now();
|
||||
const envelope: EventEnvelope = {
|
||||
operationId: newGitOpsId(),
|
||||
actor: 'system:migration',
|
||||
trigger: 'migrate',
|
||||
at,
|
||||
};
|
||||
|
||||
const application = buildDirectApplicationRow({
|
||||
id: newGitOpsId(),
|
||||
stackName,
|
||||
config: {
|
||||
repoUrl: source.repo_url,
|
||||
branch: source.branch,
|
||||
composePaths: source.compose_paths,
|
||||
contextDir: source.context_dir,
|
||||
syncEnv: source.sync_env,
|
||||
envPath: source.env_path,
|
||||
},
|
||||
identity,
|
||||
// A stack whose directory is gone is recorded as detached rather than live:
|
||||
// it describes something that no longer exists, and a live application
|
||||
// would go on claiming the name.
|
||||
lifecycleStatus: 'active',
|
||||
at,
|
||||
});
|
||||
|
||||
return DatabaseService.getInstance().getDb().transaction((): MigrationResult => {
|
||||
if (trust.kind === 'trusted' && stackPresent) {
|
||||
migrateAccepted(application, source, trust, identity.fingerprint, limitations, envelope, nodeId);
|
||||
store.upsertMigrationCheckpoint(scope, MIGRATION_SCHEMA_VERSION, identity.fingerprint, at);
|
||||
return { stackName, outcome: 'migrated_accepted' };
|
||||
}
|
||||
|
||||
application.evidence_limitations_json = encodeLimitations(limitations);
|
||||
GitOpsTransitions.getInstance().activateDirect({ application, nodeId, envelope });
|
||||
if (!stackPresent) {
|
||||
GitOpsTransitions.getInstance().targetTombstoned(application.id, nodeId, envelope);
|
||||
GitOpsTransitions.getInstance().applicationTombstoned(application.id, 'deleted', envelope);
|
||||
store.upsertMigrationCheckpoint(scope, MIGRATION_SCHEMA_VERSION, identity.fingerprint, at);
|
||||
return { stackName, outcome: 'tombstoned_missing_stack' };
|
||||
}
|
||||
store.upsertMigrationCheckpoint(scope, MIGRATION_SCHEMA_VERSION, identity.fingerprint, at);
|
||||
return { stackName, outcome: 'migrated_unreconciled' };
|
||||
})();
|
||||
}
|
||||
|
||||
/**
|
||||
* The one path that writes canonical pointers, because the manifest proves the
|
||||
* applied commit under the configuration in force now.
|
||||
*
|
||||
* Deployed, healthy, and last-known-good stay null regardless: a manifest
|
||||
* proves what was materialized, not what is running, and inventing those is how
|
||||
* a migration would claim a health record nobody observed. No source acceptance
|
||||
* is written either, because nobody approved this generation through the model.
|
||||
*/
|
||||
function migrateAccepted(
|
||||
application: GitOpsApplicationRow,
|
||||
source: StackGitSource,
|
||||
trust: Extract<ManifestTrust, { kind: 'trusted' }>,
|
||||
fingerprint: string,
|
||||
limitations: GitOpsEvidenceLimitation[],
|
||||
envelope: EventEnvelope,
|
||||
nodeId: number,
|
||||
): void {
|
||||
const store = GitOpsStore.getInstance();
|
||||
const generationId = newGitOpsId();
|
||||
const artifactSetId = newGitOpsId();
|
||||
|
||||
application.desired_commit_sha = trust.commitSha;
|
||||
application.fetched_commit_sha = trust.commitSha;
|
||||
application.accepted_generation_id = generationId;
|
||||
application.artifact_set_id = artifactSetId;
|
||||
application.latest_artifact_set_id = artifactSetId;
|
||||
application.evidence_limitations_json = encodeLimitations(limitations);
|
||||
|
||||
GitOpsTransitions.getInstance().activateDirect({ application, nodeId, envelope });
|
||||
|
||||
const generation: GitOpsGenerationRow = {
|
||||
id: generationId,
|
||||
application_id: application.id,
|
||||
commit_sha: trust.commitSha,
|
||||
repo_url: application.configured_repo_url ?? '',
|
||||
configured_ref: source.branch,
|
||||
repo_identity_json: application.repo_identity_json ?? '{}',
|
||||
manifest_version: trust.manifestVersion,
|
||||
candidate_dir: `generations/candidate-${trust.commitSha}`,
|
||||
applied_dir: trust.appliedDir,
|
||||
expected_invocation_json: '{"composeFileOrder":[],"projectName":null,"projectDirectory":null,"envFileOrder":[]}',
|
||||
// Equal to the application's, so the accepted generation is not immediately
|
||||
// reported as stale against its own configuration.
|
||||
materialization_fingerprint: fingerprint,
|
||||
validation_ok: 1,
|
||||
plan_blocked: 0,
|
||||
change_plan_fingerprint: null,
|
||||
operation_id: envelope.operationId,
|
||||
trigger: envelope.trigger,
|
||||
actor: envelope.actor,
|
||||
previous_generation_id: null,
|
||||
redacted_limitations_json: '[]',
|
||||
created_at: envelope.at,
|
||||
};
|
||||
store.insertGeneration(generation);
|
||||
store.insertArtifactSet({
|
||||
id: artifactSetId,
|
||||
generation_id: generationId,
|
||||
evidence_version: 1,
|
||||
authoritative: 0,
|
||||
qualification: 'unresolved',
|
||||
evidence_json: '{"kind":"unresolved"}',
|
||||
created_at: envelope.at,
|
||||
});
|
||||
|
||||
const target: GitOpsTargetCurrentRow = {
|
||||
...emptyTargetRow(application.id, nodeId, envelope.at),
|
||||
desired_generation_id: generationId,
|
||||
applied_generation_id: generationId,
|
||||
expected_artifact_set_id: artifactSetId,
|
||||
latest_artifact_set_id: artifactSetId,
|
||||
};
|
||||
store.upsertTarget(target);
|
||||
store.writeApplicationPointers(application);
|
||||
}
|
||||
|
||||
/** Whether the manifest licenses a canonical commit pointer. */
|
||||
function classifyManifest(stackName: string, source: StackGitSource): ManifestTrust {
|
||||
if (!source.last_applied_commit_sha) return { kind: 'absent' };
|
||||
const read = readManifestSync(stackName, source);
|
||||
if (read === null) return { kind: 'absent' };
|
||||
if ('corrupt' in read) {
|
||||
// An identity mismatch is not the same fault as unreadable content: the
|
||||
// file is fine, it just belongs to a different repository or ref.
|
||||
return read.corrupt.toLowerCase().includes('identity') || read.corrupt.toLowerCase().includes('mismatch')
|
||||
? { kind: 'identity_invalid', reason: read.corrupt }
|
||||
: { kind: 'corrupt', reason: read.corrupt };
|
||||
}
|
||||
// The two records must name the same commit. The applied directory comes from
|
||||
// the manifest and the commit from the source row, so trusting them together
|
||||
// while they disagree would mint a generation that claims one commit and
|
||||
// points at another's files, which is the exact false proof this migration
|
||||
// exists to avoid. An adopted manifest carries no commit at all, which is a
|
||||
// different fact about a different situation and gets its own answer.
|
||||
if (read.resolvedRevision.commitSha.length === 0) return { kind: 'commit_unresolved' };
|
||||
if (read.resolvedRevision.commitSha !== source.last_applied_commit_sha) {
|
||||
return { kind: 'commit_mismatch', manifestCommitSha: read.resolvedRevision.commitSha };
|
||||
}
|
||||
return {
|
||||
kind: 'trusted',
|
||||
commitSha: source.last_applied_commit_sha,
|
||||
manifestVersion: read.manifestVersion,
|
||||
appliedDir: read.generation.appliedDir,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* The manifest read, resolved synchronously.
|
||||
*
|
||||
* Migration runs inside one transaction per stack, and better-sqlite3
|
||||
* transactions cannot await, so the read is performed before the transaction
|
||||
* opens and passed in.
|
||||
*/
|
||||
let manifestReader: (stackName: string, source: StackGitSource) => ManifestReadResult = () => null;
|
||||
type ManifestReadResult =
|
||||
| { manifestVersion: number; generation: { appliedDir: string }; resolvedRevision: { commitSha: string } }
|
||||
| { corrupt: string }
|
||||
| null;
|
||||
|
||||
export function primeMigrationManifests(
|
||||
read: (stackName: string, source: StackGitSource) => ManifestReadResult,
|
||||
): void {
|
||||
manifestReader = read;
|
||||
}
|
||||
|
||||
function readManifestSync(stackName: string, source: StackGitSource): ManifestReadResult {
|
||||
return manifestReader(stackName, source);
|
||||
}
|
||||
|
||||
/** Read every manifest up front, so the per-stack transaction stays synchronous. */
|
||||
export async function loadMigrationManifests(): Promise<void> {
|
||||
const manifestSvc = GitProjectManifestService.getInstance();
|
||||
const cache = new Map<string, ManifestReadResult>();
|
||||
for (const source of DatabaseService.getInstance().getGitSources()) {
|
||||
try {
|
||||
cache.set(source.stack_name, await manifestSvc.readManifest(source.stack_name, source.repo_url, source.branch));
|
||||
} catch (error) {
|
||||
cache.set(source.stack_name, { corrupt: error instanceof Error ? error.message : String(error) });
|
||||
}
|
||||
}
|
||||
primeMigrationManifests((stackName) => cache.get(stackName) ?? null);
|
||||
}
|
||||
|
||||
/**
|
||||
* Every reason this stack could not be fully described, as recorded evidence.
|
||||
*
|
||||
* A legacy applied commit that could not be proven appears here and nowhere
|
||||
* else. Putting it on a canonical pointer would assert that the stack is at
|
||||
* that commit under the current configuration, which is exactly what could not
|
||||
* be established.
|
||||
*/
|
||||
function collectLimitations(source: StackGitSource, trust: ManifestTrust): GitOpsEvidenceLimitation[] {
|
||||
const limitations: GitOpsEvidenceLimitation[] = [];
|
||||
const legacySha = source.last_applied_commit_sha;
|
||||
if (legacySha) {
|
||||
if (trust.kind === 'absent') limitations.push({ code: 'manifest_absent', detail: legacySha });
|
||||
if (trust.kind === 'corrupt') limitations.push({ code: 'manifest_corrupt', detail: legacySha });
|
||||
if (trust.kind === 'identity_invalid') limitations.push({ code: 'manifest_identity_invalid', detail: legacySha });
|
||||
if (trust.kind === 'commit_unresolved') {
|
||||
limitations.push({ code: 'manifest_commit_unresolved', detail: legacySha });
|
||||
}
|
||||
// Both commits are named: which record is right cannot be decided here, and
|
||||
// an operator reading one of them alone has no way to see the disagreement.
|
||||
if (trust.kind === 'commit_mismatch') {
|
||||
limitations.push({
|
||||
code: 'manifest_commit_mismatch',
|
||||
detail: `${legacySha} (recorded) vs ${trust.manifestCommitSha} (manifest)`,
|
||||
});
|
||||
}
|
||||
}
|
||||
// A pending pull proves nothing about the current repository or ref: the blob
|
||||
// predates any configuration change and carries no identity stamp.
|
||||
if (source.pending_commit_sha && source.pending_commit_sha !== legacySha) {
|
||||
limitations.push({ code: 'legacy_pending', detail: source.pending_commit_sha });
|
||||
}
|
||||
return limitations;
|
||||
}
|
||||
|
||||
function encodeLimitations(limitations: GitOpsEvidenceLimitation[]): string | null {
|
||||
let encoded: string | null = null;
|
||||
for (const limitation of limitations) {
|
||||
encoded = encodeGitOpsEvidenceLimitations(
|
||||
encoded ? JSON.parse(encoded) as GitOpsEvidenceLimitation[] : [],
|
||||
limitation.code,
|
||||
limitation,
|
||||
);
|
||||
}
|
||||
return encoded;
|
||||
}
|
||||
|
||||
function stackDirectoryPresent(stackName: string, nodeId: number): boolean {
|
||||
try {
|
||||
const base = path.resolve(FileSystemService.getInstance(nodeId).getBaseDir());
|
||||
const resolved = path.resolve(base, stackName);
|
||||
if (!resolved.startsWith(base + path.sep)) return false;
|
||||
return fs.statSync(resolved).isDirectory();
|
||||
} catch (error) {
|
||||
if ((error as NodeJS.ErrnoException).code === 'ENOENT') return false;
|
||||
// Cannot prove it is gone, so treat it as present: tombstoning a stack that
|
||||
// is actually there is the unrecoverable mistake.
|
||||
return true;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Bring Blueprints that predate the revision state model into it.
|
||||
*
|
||||
* Every Blueprint gets an application, an intent describing what it currently
|
||||
* asks for, and a candidate marked as coming from the legacy inline record.
|
||||
* None of that is an acknowledgement. The Blueprint revision and the
|
||||
* deployment's applied revision are carried as display only, because neither
|
||||
* proves a node is running the intent this pass just minted, and recording them
|
||||
* as agreement would report convergence nobody verified.
|
||||
*/
|
||||
export function migrateInlineBlueprints(): MigrationResult[] {
|
||||
const db = DatabaseService.getInstance();
|
||||
const results: MigrationResult[] = [];
|
||||
for (const blueprint of db.listBlueprints()) {
|
||||
try {
|
||||
results.push(migrateOneBlueprint(blueprint));
|
||||
} catch (error) {
|
||||
console.error(
|
||||
`[GitOps] Could not migrate the blueprint ${sanitizeForLog(blueprint.name)}; retrying next boot:`,
|
||||
error instanceof Error ? error.stack ?? error.message : String(error),
|
||||
);
|
||||
results.push({ stackName: blueprint.name, outcome: 'failed' });
|
||||
}
|
||||
}
|
||||
return results;
|
||||
}
|
||||
|
||||
function migrateOneBlueprint(blueprint: Blueprint): MigrationResult {
|
||||
const store = GitOpsStore.getInstance();
|
||||
const fingerprint = intentFingerprint(blueprint);
|
||||
const scope = `inline_blueprint:${blueprint.id}`;
|
||||
|
||||
const checkpoint = store.getMigrationCheckpoint(scope);
|
||||
if (
|
||||
checkpoint
|
||||
&& checkpoint.schema_version === MIGRATION_SCHEMA_VERSION
|
||||
&& checkpoint.fingerprint === fingerprint
|
||||
) {
|
||||
return { stackName: blueprint.name, outcome: 'skipped_current' };
|
||||
}
|
||||
|
||||
// A Blueprint created through the new path already describes itself, and its
|
||||
// rows were written with proof this pass does not have.
|
||||
if (store.getLiveBlueprintApplication(blueprint.id)) {
|
||||
store.upsertMigrationCheckpoint(scope, MIGRATION_SCHEMA_VERSION, fingerprint, Date.now());
|
||||
return { stackName: blueprint.name, outcome: 'skipped_live_application' };
|
||||
}
|
||||
|
||||
const at = Date.now();
|
||||
const envelope: EventEnvelope = {
|
||||
operationId: newGitOpsId(),
|
||||
actor: 'system:migration',
|
||||
trigger: 'migrate',
|
||||
at,
|
||||
};
|
||||
const applicationId = newGitOpsId();
|
||||
const intentId = newGitOpsId();
|
||||
|
||||
return DatabaseService.getInstance().getDb().transaction((): MigrationResult => {
|
||||
const tx = GitOpsTransitions.getInstance();
|
||||
tx.activateInlineBlueprint({
|
||||
application: {
|
||||
...blankInlineApplication(applicationId, blueprint.id, at),
|
||||
evidence_limitations_json: encodeLimitations(inlineApprovalLimitations(blueprint)),
|
||||
},
|
||||
envelope,
|
||||
});
|
||||
|
||||
tx.intentRevised({
|
||||
applicationId,
|
||||
intent: {
|
||||
id: intentId,
|
||||
application_id: applicationId,
|
||||
blueprint_id: blueprint.id,
|
||||
compose_content_sha256: createHash('sha256').update(blueprint.compose_content, 'utf8').digest('hex'),
|
||||
// Display only. A revision is not an acknowledgement: nothing here
|
||||
// proves a node is running what this intent describes.
|
||||
blueprint_revision: blueprint.revision,
|
||||
deploy_stack_name: blueprint.name,
|
||||
selector_json: JSON.stringify(blueprint.selector),
|
||||
pinned_node_id: blueprint.pinned_node_id,
|
||||
cordon_implications_json: JSON.stringify({ pinnedOverridesCordon: blueprint.pinned_node_id !== null }),
|
||||
rollout_strategy_json: JSON.stringify({ driftMode: blueprint.drift_mode, enabled: blueprint.enabled }),
|
||||
runtime_drift_policy: blueprint.drift_mode,
|
||||
stateful_policy_json: null,
|
||||
health_failure_rollback_policy_json: null,
|
||||
operation_id: envelope.operationId,
|
||||
actor: envelope.actor,
|
||||
created_at: at,
|
||||
},
|
||||
envelope,
|
||||
});
|
||||
|
||||
tx.rolloutCandidateOpened({
|
||||
applicationId,
|
||||
candidate: {
|
||||
id: newGitOpsId(),
|
||||
application_id: applicationId,
|
||||
intent_revision_id: intentId,
|
||||
compose_content_sha256: createHash('sha256').update(blueprint.compose_content, 'utf8').digest('hex'),
|
||||
accepted_generation_id: null,
|
||||
artifact_set_id: null,
|
||||
// Placement is not resolved here. Migration records what the Blueprint
|
||||
// asks for, never which nodes currently satisfy it.
|
||||
required_targets_json: JSON.stringify({ nodeIds: [] }),
|
||||
authoritative: 1,
|
||||
provenance: 'legacy_inline',
|
||||
operation_id: envelope.operationId,
|
||||
created_at: at,
|
||||
},
|
||||
envelope,
|
||||
});
|
||||
|
||||
store.upsertMigrationCheckpoint(scope, MIGRATION_SCHEMA_VERSION, fingerprint, at);
|
||||
return { stackName: blueprint.name, outcome: 'migrated_inline' };
|
||||
})();
|
||||
}
|
||||
|
||||
/**
|
||||
* Why an approval could not be carried across.
|
||||
*
|
||||
* An approval authorizes the intent it was given for. A Blueprint edited since
|
||||
* then, or never approved, has nothing this pass can record as authority, and
|
||||
* saying so is what stops the gap reading as an approval that is simply absent.
|
||||
*/
|
||||
function inlineApprovalLimitations(blueprint: Blueprint): GitOpsEvidenceLimitation[] {
|
||||
const { effectiveApproval } = evaluateEffectiveApproval(blueprint, []);
|
||||
if (effectiveApproval === 'approved') return [];
|
||||
return [{ code: 'blueprint_reapproval_required', detail: String(blueprint.id) }];
|
||||
}
|
||||
@@ -0,0 +1,101 @@
|
||||
/**
|
||||
* Node-side changes that move where Blueprints are allowed to run.
|
||||
*
|
||||
* A label is not a statement about any one Blueprint, but it changes which
|
||||
* nodes a selector matches, so it revises placement for whichever Blueprints
|
||||
* the change actually moved. That set is computed by comparing the desired
|
||||
* nodes before and after: a label nothing selects on moves nothing and records
|
||||
* nothing.
|
||||
*
|
||||
* A cordon goes through the same comparison and, as things stand, never moves
|
||||
* anything. It governs whether new placements may be made, not what a Blueprint
|
||||
* asks for, and the desired-node computation deliberately ignores it. The
|
||||
* comparison is still the right shape for it, so the caller gets a truthful
|
||||
* empty answer instead of a special case.
|
||||
*
|
||||
* As in the Blueprint producers, the desired-node computation is supplied by
|
||||
* the caller. The reconciler that knows how to do it reaches this layer, and
|
||||
* importing it back would close a module cycle.
|
||||
*/
|
||||
import { DatabaseService, type Blueprint } from '../DatabaseService';
|
||||
import { sanitizeForLog } from '../../utils/safeLog';
|
||||
import { GitOpsStore } from './store';
|
||||
import { GitOpsTransitions } from './transitions';
|
||||
import { candidateRowFor, envelopeFor, intentRowFor, recordableApplication } from './blueprintProducers';
|
||||
|
||||
/** Desired node ids per Blueprint id, as placement currently resolves them. */
|
||||
export type PlacementSnapshot = Map<number, number[]>;
|
||||
|
||||
/** Takes a snapshot of what every enabled Blueprint currently wants. */
|
||||
export type SnapshotPlacement = () => PlacementSnapshot;
|
||||
|
||||
function sameNodeSet(a: number[] | undefined, b: number[] | undefined): boolean {
|
||||
if (!a || !b) return a === b;
|
||||
if (a.length !== b.length) return false;
|
||||
const left = [...a].sort((x, y) => x - y);
|
||||
const right = [...b].sort((x, y) => x - y);
|
||||
return left.every((value, index) => value === right[index]);
|
||||
}
|
||||
|
||||
/**
|
||||
* Revise placement for every Blueprint whose desired node set actually moved.
|
||||
*
|
||||
* Comparing sets rather than reacting to the event is what keeps this honest.
|
||||
* Labelling a node no selector mentions changes nothing a Blueprint wants, and
|
||||
* minting an intent for it would invalidate every acknowledgement in the fleet
|
||||
* over an edit that moved nothing.
|
||||
*/
|
||||
export function recordPlacementShift(
|
||||
before: PlacementSnapshot,
|
||||
after: PlacementSnapshot,
|
||||
actor: string | null,
|
||||
trigger: string,
|
||||
): number[] {
|
||||
const db = DatabaseService.getInstance();
|
||||
const store = GitOpsStore.getInstance();
|
||||
const tx = GitOpsTransitions.getInstance();
|
||||
const moved: number[] = [];
|
||||
|
||||
for (const [blueprintId, desired] of after) {
|
||||
if (sameNodeSet(before.get(blueprintId), desired)) continue;
|
||||
const app = store.getLiveBlueprintApplication(blueprintId);
|
||||
// A Blueprint that predates the model has no application yet. Migration
|
||||
// brings it in rather than this path inventing a first intent for it.
|
||||
if (!recordableApplication(app)) continue;
|
||||
const blueprint = db.getBlueprint(blueprintId);
|
||||
if (!blueprint) {
|
||||
// Unlike the skip above, this one is a fault. A live application exists
|
||||
// for a Blueprint whose own row is gone, and with cascade off nothing
|
||||
// else will notice. The placement really did move, no intent or candidate
|
||||
// is minted for it, and the caller goes on to report that nothing moved.
|
||||
console.error(
|
||||
'[GitOps] Placement shift skipped: blueprint %s has a live application but no blueprint row.',
|
||||
sanitizeForLog(blueprintId),
|
||||
);
|
||||
continue;
|
||||
}
|
||||
|
||||
const envelope = envelopeFor(actor, trigger);
|
||||
const intent = intentRowFor(app.id, blueprint, envelope.operationId, actor, envelope.at);
|
||||
tx.intentRevised({ applicationId: app.id, intent, envelope });
|
||||
tx.rolloutCandidateOpened({
|
||||
applicationId: app.id,
|
||||
candidate: candidateRowFor(app.id, intent, desired, 'roster_change', envelope.operationId, envelope.at),
|
||||
envelope,
|
||||
});
|
||||
moved.push(blueprintId);
|
||||
}
|
||||
return moved;
|
||||
}
|
||||
|
||||
/** Placement as it currently resolves, for every Blueprint. */
|
||||
export function snapshotPlacementWith(
|
||||
desiredNodeIdsFor: (blueprint: Blueprint) => number[],
|
||||
blueprints: Blueprint[],
|
||||
): PlacementSnapshot {
|
||||
const snapshot: PlacementSnapshot = new Map();
|
||||
for (const blueprint of blueprints) {
|
||||
snapshot.set(blueprint.id, desiredNodeIdsFor(blueprint));
|
||||
}
|
||||
return snapshot;
|
||||
}
|
||||
@@ -0,0 +1,201 @@
|
||||
/**
|
||||
* Announcing transitions after they commit.
|
||||
*
|
||||
* A history row that is inserted and still present when the drain runs produces
|
||||
* one metric increment and, when a sink is installed, one `state-invalidate`
|
||||
* event. Both have to happen *after* the transaction that wrote the row, and
|
||||
* neither may happen for a transaction that rolled back, which is why nothing
|
||||
* here runs inline.
|
||||
*
|
||||
* The mechanism is a buffer drained on `setImmediate`. better-sqlite3 is fully
|
||||
* synchronous, so by the time a macrotask runs, the transaction that enqueued
|
||||
* the row has committed or rolled back, and so has any outer transaction
|
||||
* wrapping it. That matters: several producers wrap a handful of transitions in
|
||||
* one outer transaction, and a publisher that fired when the innermost one
|
||||
* returned would announce work that a later statement then discarded. Waiting
|
||||
* for the macrotask covers both nesting depths without having to detect which
|
||||
* one it is in.
|
||||
*
|
||||
* Rollback needs no detection either. The drain checks that each row is still
|
||||
* there before announcing it, so a discarded transaction publishes nothing on
|
||||
* its own. A replay publishes nothing for a different reason: the dedupe index
|
||||
* means no row was inserted, so nothing was ever enqueued.
|
||||
*
|
||||
* The event sink is injected rather than imported. Reaching into
|
||||
* NotificationService from inside the GitOps layer would close a module cycle
|
||||
* of exactly the kind that once made an imported constant evaluate as
|
||||
* `undefined` here, and injection also lets the tests observe events without
|
||||
* standing up the notification stack.
|
||||
*/
|
||||
import type Database from 'better-sqlite3';
|
||||
import { GitOpsMetricsService } from '../GitOpsMetricsService';
|
||||
import type { GitOpsHistoryStage, HistoryOutcome } from './history';
|
||||
import type { GitOpsTargetMode } from './types';
|
||||
|
||||
/**
|
||||
* The `state-invalidate` payload one committed transition produces.
|
||||
*
|
||||
* A type alias rather than an interface so it satisfies the broadcaster's
|
||||
* open envelope parameter: TypeScript infers an implicit index signature for
|
||||
* the former and not the latter.
|
||||
*/
|
||||
export type GitOpsInvalidateEvent = {
|
||||
type: 'state-invalidate';
|
||||
scope: 'gitops';
|
||||
/** The transition's stage, so a client can tell a fetch from a deploy. */
|
||||
action: GitOpsHistoryStage;
|
||||
applicationId: string;
|
||||
targetMode: GitOpsTargetMode;
|
||||
stackName: string | null;
|
||||
blueprintId: number | null;
|
||||
nodeId: number | null;
|
||||
ts: number;
|
||||
};
|
||||
|
||||
/**
|
||||
* Pins the requirement the alias above exists to satisfy.
|
||||
*
|
||||
* Without this, switching `type` to `interface` compiles here and fails at the
|
||||
* startup wiring in another module, as an index-signature complaint that says
|
||||
* nothing about the cause. The failure belongs at the declaration.
|
||||
*/
|
||||
type AssertsOpenEnvelope =
|
||||
GitOpsInvalidateEvent extends { type: string; [key: string]: unknown } ? true : never;
|
||||
const _openEnvelope: AssertsOpenEnvelope = true;
|
||||
void _openEnvelope;
|
||||
|
||||
export type GitOpsEventSink = (event: GitOpsInvalidateEvent) => void;
|
||||
|
||||
/** What the drain needs to know, captured while it is still typed. */
|
||||
interface PendingRow {
|
||||
db: Database.Database;
|
||||
id: string;
|
||||
stage: GitOpsHistoryStage;
|
||||
outcome: HistoryOutcome;
|
||||
applicationId: string;
|
||||
targetMode: GitOpsTargetMode;
|
||||
stackName: string | null;
|
||||
blueprintId: number | null;
|
||||
nodeId: number | null;
|
||||
at: number;
|
||||
}
|
||||
|
||||
let sink: GitOpsEventSink | null = null;
|
||||
let pending: PendingRow[] = [];
|
||||
let scheduled = false;
|
||||
let warnedUnannounced = false;
|
||||
|
||||
/**
|
||||
* Say once that transitions are committing with nobody to announce them to.
|
||||
*
|
||||
* A server that never installs the sink still counts every transition and
|
||||
* still writes every history row, so the only symptom is that no client ever
|
||||
* refreshes: the UI silently goes back to being as stale as it was before any
|
||||
* of this existed. That is precisely the kind of unwired producer this branch
|
||||
* has already shipped once, so it says so rather than being inferred from an
|
||||
* absence. Once, not per row: a boot migration would otherwise fill the log,
|
||||
* and the second occurrence tells a reader nothing the first did not.
|
||||
*/
|
||||
function warnUnannounced(): void {
|
||||
if (warnedUnannounced) return;
|
||||
warnedUnannounced = true;
|
||||
console.warn('[GitOps] Transitions are committing with no event sink installed; no client will be told about them.');
|
||||
}
|
||||
|
||||
/**
|
||||
* Install the broadcaster. Called once at startup, and with null by tests that
|
||||
* want the metrics side without the event side.
|
||||
*/
|
||||
export function setGitOpsEventSink(next: GitOpsEventSink | null): void {
|
||||
sink = next;
|
||||
}
|
||||
|
||||
/**
|
||||
* Queue one inserted history row for announcement.
|
||||
*
|
||||
* Called only where a row was genuinely inserted. The stage and outcome are
|
||||
* carried from the insert rather than read back, because they are already typed
|
||||
* there and re-reading them would turn a closed union into an unvalidated
|
||||
* column value.
|
||||
*/
|
||||
export function enqueueHistoryPublication(row: PendingRow): void {
|
||||
pending.push(row);
|
||||
if (scheduled) return;
|
||||
scheduled = true;
|
||||
setImmediate(drain);
|
||||
}
|
||||
|
||||
function drain(): void {
|
||||
scheduled = false;
|
||||
const batch = pending;
|
||||
pending = [];
|
||||
const metrics = GitOpsMetricsService.getInstance();
|
||||
|
||||
for (const row of batch) {
|
||||
if (!survived(row)) continue;
|
||||
metrics.record(row.stage, row.outcome);
|
||||
if (!sink) {
|
||||
warnUnannounced();
|
||||
continue;
|
||||
}
|
||||
announce(sink, row);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Hand one committed transition to the broadcaster.
|
||||
*
|
||||
* A throw is logged and swallowed: one client's broadcast must not cost the
|
||||
* rest of the batch their events, and none of this is worth failing a
|
||||
* committed transition over.
|
||||
*/
|
||||
function announce(to: GitOpsEventSink, row: PendingRow): void {
|
||||
try {
|
||||
to({
|
||||
type: 'state-invalidate',
|
||||
scope: 'gitops',
|
||||
action: row.stage,
|
||||
applicationId: row.applicationId,
|
||||
targetMode: row.targetMode,
|
||||
stackName: row.stackName,
|
||||
blueprintId: row.blueprintId,
|
||||
nodeId: row.nodeId,
|
||||
ts: row.at,
|
||||
});
|
||||
} catch (error) {
|
||||
console.error(
|
||||
'[GitOps] Could not announce %s for application %s:',
|
||||
row.stage, row.applicationId,
|
||||
error instanceof Error ? error.stack ?? error.message : String(error),
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether the row is still in the table.
|
||||
*
|
||||
* A missing row means its transaction rolled back, which is an ordinary
|
||||
* outcome and not worth logging. A failed *query* is different: it means the
|
||||
* check itself could not be made, so the row is treated as gone rather than
|
||||
* announced on the strength of a lookup that did not answer.
|
||||
*/
|
||||
function survived(row: PendingRow): boolean {
|
||||
try {
|
||||
return row.db.prepare('SELECT 1 FROM gitops_history WHERE id = ?').get(row.id) !== undefined;
|
||||
} catch (error) {
|
||||
console.error(
|
||||
'[GitOps] Could not confirm history row %s before announcing it:',
|
||||
row.id,
|
||||
error instanceof Error ? error.stack ?? error.message : String(error),
|
||||
);
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
/** Drop anything queued and uninstall the sink, so one test cannot reach the next. */
|
||||
export function resetGitOpsPublicationsForTests(): void {
|
||||
pending = [];
|
||||
sink = null;
|
||||
scheduled = false;
|
||||
warnedUnannounced = false;
|
||||
}
|
||||
@@ -0,0 +1,179 @@
|
||||
import type { Request } from 'express';
|
||||
import { checkPermission } from '../../middleware/permissions';
|
||||
import { isRecord } from './json';
|
||||
import type { GitOpsHistoryEvidenceFields, GitOpsRevisionProjection } from './types';
|
||||
|
||||
/**
|
||||
* What a caller must hold to read one GitOps row.
|
||||
*
|
||||
* `stack_read` is the narrow answer, used whenever a row can be tied to a stack
|
||||
* the caller may read. The other two are the fail-closed fallbacks for a row
|
||||
* whose audience cannot be narrowed, and they differ by what the row *is*:
|
||||
*
|
||||
* - `audit` for history entries, which are an audit trail. Auditing is what the
|
||||
* `system:audit` permission exists for, and the request audit log is already
|
||||
* gated on it, so an entry nobody can tie to a stack belongs to the same
|
||||
* audience rather than to Admin alone.
|
||||
* - `admin` for source rows, which are live Git configuration (repository,
|
||||
* ref, credentials policy, compose paths) rather than a record of events. An
|
||||
* auditing mandate does not imply reading the configuration of stacks that
|
||||
* have been deleted or never finished being created.
|
||||
*/
|
||||
export type GitOpsReadRequirement =
|
||||
| { readonly kind: 'admin' }
|
||||
| { readonly kind: 'audit' }
|
||||
| { readonly kind: 'stack_read'; readonly stackName: string };
|
||||
|
||||
const ADMIN: GitOpsReadRequirement = Object.freeze({ kind: 'admin' });
|
||||
const AUDIT: GitOpsReadRequirement = Object.freeze({ kind: 'audit' });
|
||||
|
||||
/**
|
||||
* The projection field the source-row classifier probes.
|
||||
*
|
||||
* Tied to the live projection variant so renaming that field fails the build
|
||||
* here. Without the tie, a rename would leave the classifier probing a key that
|
||||
* no longer exists, and every row would quietly fall to Admin: fail-closed, but
|
||||
* invisible, since nothing would error and no test that hand-builds a payload
|
||||
* would notice.
|
||||
*/
|
||||
const LIFECYCLE_KEY = 'lifecycleStatus' satisfies keyof Extract<
|
||||
GitOpsRevisionProjection,
|
||||
{ lifecycleStatus: unknown }
|
||||
>;
|
||||
|
||||
/**
|
||||
* The evidence a history entry must carry to be classified.
|
||||
*
|
||||
* Values stay `unknown` because they may have crossed an instance boundary and
|
||||
* carry no shape guarantee, but the *key names* are bound to the item type the
|
||||
* producer emits. Without that tie, renaming a field on the producer would
|
||||
* leave this probing keys that no longer exist: every row would degrade to the
|
||||
* audit bucket, fail-closed but silent, with nothing failing to compile and no
|
||||
* test noticing. The required (`-?`) mapping also makes the call site fail, not
|
||||
* just this function.
|
||||
*/
|
||||
type Evidence<T> = { [K in keyof T]-?: unknown };
|
||||
export type HistoryRowEvidence = Evidence<
|
||||
Pick<GitOpsHistoryEvidenceFields, 'stackName' | 'applicationLifecycleStatus' | 'stackResourcePresent'>
|
||||
>;
|
||||
|
||||
/**
|
||||
* Validate an owning instance's resource-existence claim.
|
||||
*
|
||||
* Only a real JSON boolean counts. A peer that omits the field, sends null, or
|
||||
* sends a string is treated as "not present", so an older or malformed instance
|
||||
* degrades to Admin rather than silently widening who may read its rows.
|
||||
*/
|
||||
export function normalizeStackResourcePresent(value: unknown): boolean {
|
||||
return value === true;
|
||||
}
|
||||
|
||||
function usableStackName(value: unknown): string | null {
|
||||
return typeof value === 'string' && value.length > 0 ? value : null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Lifecycle states whose source rows a stack grant can authorize.
|
||||
*
|
||||
* A deleted application no longer has a stack whose grant could authorize it,
|
||||
* and a creating one does not yet have a stack that survived. In both cases a
|
||||
* later application may hold the same stack name, so honouring a grant here
|
||||
* would let one application's rows be read through another's name.
|
||||
*
|
||||
* History rows do not use this. They require `active`, for the reason given on
|
||||
* `classifyHistoryRow`.
|
||||
*/
|
||||
function lifecycleAllowsStackRead(lifecycleStatus: unknown): boolean {
|
||||
return lifecycleStatus === 'active' || lifecycleStatus === 'detached';
|
||||
}
|
||||
|
||||
/**
|
||||
* Authorization for one `GET /api/git-sources` row.
|
||||
*
|
||||
* The projection is typed `unknown` rather than as a projection because the
|
||||
* hub will classify rows that arrived from another instance as parsed JSON,
|
||||
* which carries no guarantee of shape. No such caller exists yet; every current
|
||||
* one passes a locally derived projection.
|
||||
*
|
||||
* Only `active` is reachable here in practice, and deliberately so. The
|
||||
* projection comes from `projectStackRevision`, which resolves live Direct
|
||||
* applications only. Detach deletes the Git-source row in the same transaction
|
||||
* that tombstones the application, so a source row beside a detached
|
||||
* application is not a producible state anyway. The detached case is reported
|
||||
* through the stack-state surface, which never runs this classifier.
|
||||
*
|
||||
* Keep it that way. This function takes `stackName` from the Git-source row but
|
||||
* lifecycle from whatever the projection resolved, so widening that resolution
|
||||
* to reach another application silently changes who may read this row.
|
||||
*/
|
||||
export function classifySourceRow(input: {
|
||||
stackName: unknown;
|
||||
gitopsRevision: unknown;
|
||||
stackResourcePresent: unknown;
|
||||
}): GitOpsReadRequirement {
|
||||
const stackName = usableStackName(input.stackName);
|
||||
if (!stackName) return ADMIN;
|
||||
if (!isRecord(input.gitopsRevision)) return ADMIN;
|
||||
if (!lifecycleAllowsStackRead(input.gitopsRevision[LIFECYCLE_KEY])) return ADMIN;
|
||||
if (!normalizeStackResourcePresent(input.stackResourcePresent)) return ADMIN;
|
||||
return { kind: 'stack_read', stackName };
|
||||
}
|
||||
|
||||
/**
|
||||
* Authorization for one history row.
|
||||
*
|
||||
* Lifecycle comes from the owning application row rather than the entry's
|
||||
* `before`/`after`, because those record only the fields a transition moved:
|
||||
* most entries never mention lifecycle at all, so decoding them would send
|
||||
* nearly every row to Admin and leave operators unable to read the history of
|
||||
* their own stacks. Reading the application row also keeps authorization off
|
||||
* the audit payload entirely, so a corrupt delta cannot influence who may see
|
||||
* it.
|
||||
*/
|
||||
export function classifyHistoryRow(input: HistoryRowEvidence): GitOpsReadRequirement {
|
||||
const stackName = usableStackName(input.stackName);
|
||||
if (!stackName) return AUDIT;
|
||||
// Only a live application, never a `detached` one, even though a detached
|
||||
// application's files are usually still the stack standing at its name.
|
||||
//
|
||||
// A stack grant is a grant on whatever occupies that name today, so the
|
||||
// allowance is only sound while the detached application is still what
|
||||
// occupies it, and nothing here can establish that. A later Direct
|
||||
// application is visible in these tables, but a Blueprint deploying under the
|
||||
// same name records it as `deploy_stack_name` on its intent revision rather
|
||||
// than as an application `stack_name`, and a plain Compose stack recreated at
|
||||
// that name leaves no GitOps trace at all. Since the last case cannot be
|
||||
// detected in principle, the allowance cannot be made sound by detecting
|
||||
// harder, and a rule that holds only for the successors we happen to see is
|
||||
// worse than not having one.
|
||||
//
|
||||
// Detach therefore moves a stack's trail to the audit audience, the same
|
||||
// answer `deleted` and `creating` predecessors already get. A create still in
|
||||
// flight shows its own history through the scope exemption in
|
||||
// `helpers/gitopsHistoryPage.ts`, which never reaches this classifier.
|
||||
if (input.applicationLifecycleStatus !== 'active') return AUDIT;
|
||||
if (!normalizeStackResourcePresent(input.stackResourcePresent)) return AUDIT;
|
||||
return { kind: 'stack_read', stackName };
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether this caller satisfies a classifier's requirement.
|
||||
*
|
||||
* Exhaustive on purpose: a requirement this function does not recognize is
|
||||
* denied rather than falling through to the narrower stack check.
|
||||
*/
|
||||
export function satisfiesGitOpsRead(req: Request, requirement: GitOpsReadRequirement): boolean {
|
||||
switch (requirement.kind) {
|
||||
case 'admin':
|
||||
return req.user?.role === 'admin';
|
||||
case 'audit':
|
||||
return checkPermission(req, 'system:audit');
|
||||
case 'stack_read':
|
||||
return checkPermission(req, 'stack:read', 'stack', requirement.stackName);
|
||||
default: {
|
||||
const unrecognized: never = requirement;
|
||||
void unrecognized;
|
||||
return false;
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,62 @@
|
||||
import { GitOpsStore } from './store';
|
||||
|
||||
export type GitOpsRecoveryCapture = {
|
||||
gitops_generation_id: string | null;
|
||||
gitops_artifact_set_id: string | null;
|
||||
gitops_source_acceptance_ref: string | null;
|
||||
};
|
||||
|
||||
export const EMPTY_GITOPS_RECOVERY_CAPTURE: GitOpsRecoveryCapture = {
|
||||
gitops_generation_id: null,
|
||||
gitops_artifact_set_id: null,
|
||||
gitops_source_acceptance_ref: null,
|
||||
};
|
||||
|
||||
/**
|
||||
* Bind a rollback point to the generation that is actually deployed on this
|
||||
* target. A generation that was applied but never deployed is not rollback
|
||||
* identity, so only `deployed_generation_id` is read.
|
||||
*
|
||||
* The acceptance reference prefers the one recorded on the target and
|
||||
* otherwise falls back to the newest acceptance *of that same generation*, so
|
||||
* an acceptance belonging to a later generation can never be captured. Any
|
||||
* candidate that does not resolve against the deployed generation is stored as
|
||||
* null rather than as a reference the restore path would have to trust.
|
||||
*/
|
||||
export function captureGitOpsRecoveryBinding(stackName: string, nodeId: number): GitOpsRecoveryCapture {
|
||||
const store = GitOpsStore.getInstance();
|
||||
const application = store.getLiveDirectApplication(stackName);
|
||||
if (!application) return { ...EMPTY_GITOPS_RECOVERY_CAPTURE };
|
||||
const target = store.getTarget(application.id, nodeId);
|
||||
const generationId = target?.deployed_generation_id ?? null;
|
||||
if (!generationId) return { ...EMPTY_GITOPS_RECOVERY_CAPTURE };
|
||||
|
||||
let artifactSetId: string | null = null;
|
||||
if (target?.expected_artifact_set_id) {
|
||||
const artifact = store.getArtifactSet(target.expected_artifact_set_id);
|
||||
if (artifact && artifact.generation_id === generationId) {
|
||||
artifactSetId = artifact.id;
|
||||
}
|
||||
}
|
||||
|
||||
const expected = {
|
||||
kind: 'source_acceptance' as const,
|
||||
applicationId: application.id,
|
||||
generationId,
|
||||
};
|
||||
let sourceAcceptanceRef: string | null = null;
|
||||
if (target?.source_acceptance_ref && store.resolveApprovalRef(target.source_acceptance_ref, expected)) {
|
||||
sourceAcceptanceRef = target.source_acceptance_ref;
|
||||
} else {
|
||||
const newest = store.newestSourceAcceptanceId(application.id, generationId);
|
||||
if (newest && store.resolveApprovalRef(newest, expected)) {
|
||||
sourceAcceptanceRef = newest;
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
gitops_generation_id: generationId,
|
||||
gitops_artifact_set_id: artifactSetId,
|
||||
gitops_source_acceptance_ref: sourceAcceptanceRef,
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,103 @@
|
||||
// Upper bound so a caller cannot flood the service with a huge payload.
|
||||
// Generous compared to anything a real Git provider emits.
|
||||
export const MAX_REPO_URL_LENGTH = 2048;
|
||||
|
||||
export type RepoIdentity = { host: string; pathname: string };
|
||||
|
||||
export type ParseHttpsRepoUrlResult =
|
||||
| { ok: true; url: URL }
|
||||
| { ok: false; reason: 'not_https' | 'userinfo' | 'query' | 'fragment' | 'too_long' | 'invalid' };
|
||||
|
||||
export function parseHttpsRepoUrl(raw: string): ParseHttpsRepoUrlResult {
|
||||
const trimmed = raw.trim();
|
||||
if (trimmed.length === 0 || trimmed.length > MAX_REPO_URL_LENGTH) {
|
||||
return { ok: false, reason: trimmed.length > MAX_REPO_URL_LENGTH ? 'too_long' : 'invalid' };
|
||||
}
|
||||
let url: URL;
|
||||
try {
|
||||
url = new URL(trimmed);
|
||||
} catch {
|
||||
return { ok: false, reason: 'invalid' };
|
||||
}
|
||||
if (url.protocol !== 'https:') {
|
||||
return { ok: false, reason: 'not_https' };
|
||||
}
|
||||
if (url.username !== '' || url.password !== '') {
|
||||
return { ok: false, reason: 'userinfo' };
|
||||
}
|
||||
if (url.search !== '') {
|
||||
return { ok: false, reason: 'query' };
|
||||
}
|
||||
if (url.hash !== '') {
|
||||
return { ok: false, reason: 'fragment' };
|
||||
}
|
||||
return { ok: true, url };
|
||||
}
|
||||
|
||||
export function serializeRepoIdentity(url: URL): RepoIdentity {
|
||||
return { host: url.host, pathname: url.pathname };
|
||||
}
|
||||
|
||||
export type ParseLegacyRepoUrlResult =
|
||||
| { ok: true; url: URL }
|
||||
| { ok: false; reason: 'not_https' | 'too_long' | 'invalid' };
|
||||
|
||||
/**
|
||||
* Parse an operational repository URL that predates strict ingress.
|
||||
*
|
||||
* Legacy operational rows retain the original URL (with userinfo, query,
|
||||
* and fragment) because fetch still needs them. Migration derives the
|
||||
* storable identity by stripping those components instead of refusing the
|
||||
* stack. Everything strict ingress refuses for want of a recoverable
|
||||
* identity (non-HTTPS, unparseable, oversized) is refused here too.
|
||||
*/
|
||||
export function parseLegacyRepoUrl(raw: string): ParseLegacyRepoUrlResult {
|
||||
const trimmed = raw.trim();
|
||||
if (trimmed.length === 0 || trimmed.length > MAX_REPO_URL_LENGTH) {
|
||||
return { ok: false, reason: trimmed.length > MAX_REPO_URL_LENGTH ? 'too_long' : 'invalid' };
|
||||
}
|
||||
let url: URL;
|
||||
try {
|
||||
url = new URL(trimmed);
|
||||
} catch {
|
||||
return { ok: false, reason: 'invalid' };
|
||||
}
|
||||
if (url.protocol !== 'https:') {
|
||||
return { ok: false, reason: 'not_https' };
|
||||
}
|
||||
url.username = '';
|
||||
url.password = '';
|
||||
url.search = '';
|
||||
url.hash = '';
|
||||
return { ok: true, url };
|
||||
}
|
||||
|
||||
/**
|
||||
* Rebuild a storable repository URL from an identity.
|
||||
*
|
||||
* The secret-free guarantee comes from `parseHttpsRepoUrl` having already
|
||||
* rejected userinfo, query strings, and fragments at every ingress. This
|
||||
* function only reassembles what that check let through.
|
||||
*/
|
||||
export function secretFreeRepoUrl(identity: RepoIdentity): string {
|
||||
return `https://${identity.host}${identity.pathname}`;
|
||||
}
|
||||
|
||||
export function repoUrlRejectionMessage(raw: string): string | null {
|
||||
const parsed = parseHttpsRepoUrl(raw);
|
||||
if (parsed.ok) return null;
|
||||
switch (parsed.reason) {
|
||||
case 'too_long':
|
||||
return 'repo_url is too long';
|
||||
case 'not_https':
|
||||
return 'Only HTTPS repository URLs are supported';
|
||||
case 'userinfo':
|
||||
return 'Repository URL must not include userinfo';
|
||||
case 'query':
|
||||
return 'Repository URL must not include a query string';
|
||||
case 'fragment':
|
||||
return 'Repository URL must not include a fragment';
|
||||
default:
|
||||
return 'Repository URL is invalid';
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,450 @@
|
||||
/**
|
||||
* DatabaseService executes this at init, then separately seeds
|
||||
* gitops_schema_version, adds the gitops_* columns to
|
||||
* stack_update_recovery_generations, and adds deployed_generation_id to
|
||||
* health_gate_runs. Those three live outside this string because they alter
|
||||
* pre-existing tables rather than creating GitOps ones.
|
||||
*/
|
||||
|
||||
export const GITOPS_SCHEMA_SQL = `
|
||||
CREATE TABLE IF NOT EXISTS gitops_migration_checkpoints (
|
||||
scope TEXT PRIMARY KEY,
|
||||
schema_version INTEGER NOT NULL,
|
||||
fingerprint TEXT NOT NULL,
|
||||
migrated_at INTEGER NOT NULL
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS gitops_create_checkpoints (
|
||||
application_id TEXT PRIMARY KEY,
|
||||
stack_name TEXT NOT NULL,
|
||||
phase TEXT NOT NULL CHECK (phase IN (
|
||||
'pre_stack','stack_created','promoting','manifest_committed','pointers_committed'
|
||||
)),
|
||||
generation_id TEXT NULL,
|
||||
operation_id TEXT NOT NULL,
|
||||
repo_url TEXT NOT NULL,
|
||||
branch TEXT NOT NULL,
|
||||
compose_path TEXT NOT NULL,
|
||||
compose_paths_json TEXT NOT NULL,
|
||||
context_dir TEXT NULL,
|
||||
sync_env INTEGER NOT NULL DEFAULT 0,
|
||||
env_path TEXT NULL,
|
||||
auth_type TEXT NOT NULL,
|
||||
encrypted_token TEXT NULL,
|
||||
auto_apply_on_webhook INTEGER NOT NULL DEFAULT 0,
|
||||
auto_deploy_on_apply INTEGER NOT NULL DEFAULT 0,
|
||||
commit_sha TEXT NULL,
|
||||
applied_spec_json TEXT NULL,
|
||||
created_managed_root INTEGER NOT NULL DEFAULT 0,
|
||||
created_at INTEGER NOT NULL,
|
||||
updated_at INTEGER NOT NULL
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_gitops_create_ck_stack
|
||||
ON gitops_create_checkpoints(stack_name);
|
||||
CREATE INDEX IF NOT EXISTS idx_gitops_create_ck_phase
|
||||
ON gitops_create_checkpoints(phase);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS gitops_applications (
|
||||
id TEXT PRIMARY KEY,
|
||||
lifecycle_key TEXT NOT NULL,
|
||||
lifecycle_status TEXT NOT NULL CHECK (lifecycle_status IN (
|
||||
'active','creating','detached','deleted'
|
||||
)),
|
||||
target_mode TEXT NOT NULL CHECK (target_mode IN ('direct','inline_blueprint','blueprint')),
|
||||
stack_name TEXT NULL,
|
||||
blueprint_id INTEGER NULL,
|
||||
configured_repo_url TEXT NULL,
|
||||
repo_identity_json TEXT NULL,
|
||||
configured_ref TEXT NULL,
|
||||
compose_paths_json TEXT NULL,
|
||||
context_dir TEXT NULL,
|
||||
sync_env INTEGER NULL,
|
||||
env_path TEXT NULL,
|
||||
materialization_fingerprint TEXT NULL,
|
||||
desired_commit_sha TEXT NULL,
|
||||
fetched_commit_sha TEXT NULL,
|
||||
candidate_generation_id TEXT NULL,
|
||||
accepted_generation_id TEXT NULL,
|
||||
candidate_plan_blocked INTEGER NOT NULL DEFAULT 0,
|
||||
review_required INTEGER NOT NULL DEFAULT 0,
|
||||
artifact_set_id TEXT NULL,
|
||||
latest_artifact_set_id TEXT NULL,
|
||||
intent_revision_id TEXT NULL,
|
||||
rollout_candidate_id TEXT NULL,
|
||||
rollout_generation_id TEXT NULL,
|
||||
source_acceptance_ref TEXT NULL,
|
||||
placement_approval_ref TEXT NULL,
|
||||
rollout_authorization_ref TEXT NULL,
|
||||
legacy_combined_approval_ref TEXT NULL,
|
||||
preflight_fingerprint TEXT NULL,
|
||||
latest_operation_id TEXT NULL,
|
||||
active_operation_id TEXT NULL,
|
||||
active_operation_stage TEXT NULL CHECK (
|
||||
active_operation_stage IS NULL OR active_operation_stage IN (
|
||||
'fetch_started','apply_started','deploy_started','recovery_started'
|
||||
)
|
||||
),
|
||||
active_operation_at INTEGER NULL,
|
||||
active_generation_id TEXT NULL,
|
||||
pause_at INTEGER NULL,
|
||||
pause_reason TEXT NULL,
|
||||
partial_json TEXT NULL,
|
||||
failure_stage TEXT NULL CHECK (
|
||||
failure_stage IS NULL OR failure_stage IN (
|
||||
'fetch','validation','apply','create','recovery'
|
||||
)
|
||||
),
|
||||
failure_class TEXT NULL,
|
||||
failure_at INTEGER NULL,
|
||||
retry_at INTEGER NULL,
|
||||
retry_count INTEGER NOT NULL DEFAULT 0,
|
||||
suspended_at INTEGER NULL,
|
||||
recovery_ref TEXT NULL,
|
||||
recovery_phase TEXT NULL CHECK (
|
||||
recovery_phase IS NULL OR recovery_phase IN (
|
||||
'capturing','restoring','compensating','complete','failed'
|
||||
)
|
||||
),
|
||||
interruption_stage TEXT NULL CHECK (
|
||||
interruption_stage IS NULL OR interruption_stage IN (
|
||||
'fetch_started','apply_started','deploy_started','recovery_started'
|
||||
)
|
||||
),
|
||||
interruption_at INTEGER NULL,
|
||||
interruption_operation_id TEXT NULL,
|
||||
interruption_generation_id TEXT NULL,
|
||||
evidence_fresh_at INTEGER NULL,
|
||||
-- Why this row could not prove something, recorded at write time. Read-time
|
||||
-- limitations are derived; these are the ones only the writer knows.
|
||||
evidence_limitations_json TEXT NULL,
|
||||
created_at INTEGER NOT NULL,
|
||||
updated_at INTEGER NOT NULL,
|
||||
CHECK (target_mode != 'direct' OR (stack_name IS NOT NULL AND blueprint_id IS NULL)),
|
||||
CHECK (target_mode != 'inline_blueprint' OR (blueprint_id IS NOT NULL AND stack_name IS NULL AND configured_repo_url IS NULL)),
|
||||
CHECK (target_mode != 'blueprint' OR (blueprint_id IS NOT NULL AND stack_name IS NULL AND configured_repo_url IS NOT NULL))
|
||||
);
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS idx_gitops_app_active_direct
|
||||
ON gitops_applications(stack_name)
|
||||
WHERE lifecycle_status IN ('active','creating') AND target_mode = 'direct';
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS idx_gitops_app_active_blueprint_any
|
||||
ON gitops_applications(blueprint_id)
|
||||
WHERE lifecycle_status IN ('active','creating')
|
||||
AND target_mode IN ('inline_blueprint','blueprint');
|
||||
CREATE INDEX IF NOT EXISTS idx_gitops_app_lifecycle_key
|
||||
ON gitops_applications(lifecycle_key);
|
||||
CREATE INDEX IF NOT EXISTS idx_gitops_app_status
|
||||
ON gitops_applications(lifecycle_status);
|
||||
-- The two unique indexes above are partial on the live rows, so neither serves
|
||||
-- a lookup for a detached one. Without this the drift route's fallback is a
|
||||
-- full scan and a sort. Direct only: Blueprint retirement writes 'deleted',
|
||||
-- never 'detached', so the Blueprint equivalent would index an empty set.
|
||||
CREATE INDEX IF NOT EXISTS idx_gitops_app_detached_direct
|
||||
ON gitops_applications(stack_name, updated_at DESC)
|
||||
WHERE lifecycle_status = 'detached' AND target_mode = 'direct';
|
||||
|
||||
CREATE TABLE IF NOT EXISTS gitops_generations (
|
||||
id TEXT PRIMARY KEY,
|
||||
application_id TEXT NOT NULL,
|
||||
commit_sha TEXT NOT NULL,
|
||||
repo_url TEXT NOT NULL,
|
||||
configured_ref TEXT NOT NULL,
|
||||
repo_identity_json TEXT NOT NULL,
|
||||
manifest_version INTEGER NOT NULL,
|
||||
candidate_dir TEXT NOT NULL,
|
||||
applied_dir TEXT NOT NULL,
|
||||
expected_invocation_json TEXT NOT NULL,
|
||||
materialization_fingerprint TEXT NOT NULL,
|
||||
validation_ok INTEGER NOT NULL,
|
||||
plan_blocked INTEGER NOT NULL DEFAULT 0,
|
||||
change_plan_fingerprint TEXT NULL,
|
||||
operation_id TEXT NOT NULL,
|
||||
trigger TEXT NOT NULL,
|
||||
actor TEXT NULL,
|
||||
previous_generation_id TEXT NULL,
|
||||
redacted_limitations_json TEXT NOT NULL DEFAULT '[]',
|
||||
created_at INTEGER NOT NULL
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_gitops_gen_app_created
|
||||
ON gitops_generations(application_id, created_at);
|
||||
CREATE INDEX IF NOT EXISTS idx_gitops_gen_sha
|
||||
ON gitops_generations(commit_sha);
|
||||
CREATE INDEX IF NOT EXISTS idx_gitops_gen_op
|
||||
ON gitops_generations(operation_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_gitops_gen_repo_ref
|
||||
ON gitops_generations(repo_url, configured_ref);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS gitops_artifact_sets (
|
||||
id TEXT PRIMARY KEY,
|
||||
generation_id TEXT NOT NULL,
|
||||
evidence_version INTEGER NOT NULL,
|
||||
authoritative INTEGER NOT NULL DEFAULT 0,
|
||||
qualification TEXT NOT NULL CHECK (qualification IN (
|
||||
'unresolved','exact','qualified','stale','unavailable','local_build_unverified'
|
||||
)),
|
||||
evidence_json TEXT NOT NULL DEFAULT '{}',
|
||||
created_at INTEGER NOT NULL,
|
||||
UNIQUE (generation_id, evidence_version)
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_gitops_artifact_gen
|
||||
ON gitops_artifact_sets(generation_id, evidence_version);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS gitops_intent_revisions (
|
||||
id TEXT PRIMARY KEY,
|
||||
application_id TEXT NOT NULL,
|
||||
blueprint_id INTEGER NOT NULL,
|
||||
compose_content_sha256 TEXT NOT NULL,
|
||||
blueprint_revision INTEGER NOT NULL,
|
||||
deploy_stack_name TEXT NOT NULL,
|
||||
selector_json TEXT NOT NULL,
|
||||
pinned_node_id INTEGER NULL,
|
||||
cordon_implications_json TEXT NOT NULL DEFAULT '[]',
|
||||
rollout_strategy_json TEXT NOT NULL DEFAULT '{}',
|
||||
runtime_drift_policy TEXT NULL,
|
||||
stateful_policy_json TEXT NULL,
|
||||
health_failure_rollback_policy_json TEXT NULL,
|
||||
operation_id TEXT NOT NULL,
|
||||
actor TEXT NULL,
|
||||
created_at INTEGER NOT NULL
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_gitops_intent_app_created
|
||||
ON gitops_intent_revisions(application_id, created_at);
|
||||
CREATE INDEX IF NOT EXISTS idx_gitops_intent_blueprint
|
||||
ON gitops_intent_revisions(blueprint_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_gitops_intent_content
|
||||
ON gitops_intent_revisions(compose_content_sha256);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS gitops_rollout_candidates (
|
||||
id TEXT PRIMARY KEY,
|
||||
application_id TEXT NOT NULL,
|
||||
intent_revision_id TEXT NOT NULL,
|
||||
compose_content_sha256 TEXT NOT NULL,
|
||||
accepted_generation_id TEXT NULL,
|
||||
artifact_set_id TEXT NULL,
|
||||
required_targets_json TEXT NOT NULL,
|
||||
authoritative INTEGER NOT NULL DEFAULT 0,
|
||||
provenance TEXT NOT NULL CHECK (provenance IN (
|
||||
'intent_change','roster_change','legacy_inline'
|
||||
)),
|
||||
operation_id TEXT NOT NULL,
|
||||
created_at INTEGER NOT NULL
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_gitops_rollout_app
|
||||
ON gitops_rollout_candidates(application_id, created_at);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS gitops_approvals (
|
||||
id TEXT PRIMARY KEY,
|
||||
kind TEXT NOT NULL CHECK (kind IN (
|
||||
'source_acceptance','placement_approval','rollout_authorization','legacy_combined'
|
||||
)),
|
||||
authority TEXT NOT NULL CHECK (authority IN (
|
||||
'operator','configured_policy','legacy_combined'
|
||||
)),
|
||||
authoritative INTEGER NOT NULL DEFAULT 0,
|
||||
application_id TEXT NOT NULL,
|
||||
generation_id TEXT NULL,
|
||||
intent_revision_id TEXT NULL,
|
||||
artifact_set_id TEXT NULL,
|
||||
rollout_candidate_id TEXT NULL,
|
||||
rollout_generation_id TEXT NULL,
|
||||
source_acceptance_ref TEXT NULL,
|
||||
placement_approval_ref TEXT NULL,
|
||||
required_targets_json TEXT NULL,
|
||||
preflight_fingerprint TEXT NULL,
|
||||
fingerprint TEXT NULL,
|
||||
blast_json TEXT NULL,
|
||||
policy_provenance_json TEXT NULL,
|
||||
actor TEXT NULL,
|
||||
created_at INTEGER NOT NULL,
|
||||
CHECK (
|
||||
kind != 'source_acceptance' OR (
|
||||
authoritative = 1
|
||||
AND authority IN ('operator','configured_policy')
|
||||
AND generation_id IS NOT NULL
|
||||
)
|
||||
),
|
||||
CHECK (
|
||||
kind != 'placement_approval' OR (
|
||||
authoritative = 1
|
||||
AND authority IN ('operator','configured_policy')
|
||||
AND intent_revision_id IS NOT NULL
|
||||
AND blast_json IS NOT NULL
|
||||
)
|
||||
),
|
||||
CHECK (
|
||||
kind != 'rollout_authorization' OR (
|
||||
authoritative = 1
|
||||
AND authority IN ('operator','configured_policy')
|
||||
AND generation_id IS NOT NULL
|
||||
AND artifact_set_id IS NOT NULL
|
||||
AND intent_revision_id IS NOT NULL
|
||||
AND rollout_candidate_id IS NOT NULL
|
||||
AND source_acceptance_ref IS NOT NULL
|
||||
AND placement_approval_ref IS NOT NULL
|
||||
AND required_targets_json IS NOT NULL
|
||||
AND preflight_fingerprint IS NOT NULL
|
||||
)
|
||||
),
|
||||
CHECK (
|
||||
kind != 'legacy_combined' OR (
|
||||
authoritative = 0
|
||||
AND authority = 'legacy_combined'
|
||||
)
|
||||
)
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_gitops_approval_app
|
||||
ON gitops_approvals(application_id, created_at);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS gitops_target_current (
|
||||
application_id TEXT NOT NULL,
|
||||
node_id INTEGER NOT NULL,
|
||||
target_status TEXT NOT NULL CHECK (target_status IN ('active','tombstoned')),
|
||||
desired_generation_id TEXT NULL,
|
||||
candidate_generation_id TEXT NULL,
|
||||
applied_generation_id TEXT NULL,
|
||||
deployed_generation_id TEXT NULL,
|
||||
healthy_generation_id TEXT NULL,
|
||||
lkg_generation_id TEXT NULL,
|
||||
lkg_artifact_set_id TEXT NULL,
|
||||
lkg_unavailable_at INTEGER NULL,
|
||||
lkg_unavailable_reason TEXT NULL CHECK (
|
||||
lkg_unavailable_reason IS NULL OR lkg_unavailable_reason IN (
|
||||
'generation_missing','recovery_unretainable'
|
||||
)
|
||||
),
|
||||
expected_artifact_set_id TEXT NULL,
|
||||
latest_artifact_set_id TEXT NULL,
|
||||
observed_artifact_identity_json TEXT NULL,
|
||||
intent_revision_id TEXT NULL,
|
||||
rollout_candidate_id TEXT NULL,
|
||||
rollout_generation_id TEXT NULL,
|
||||
source_acceptance_ref TEXT NULL,
|
||||
placement_approval_ref TEXT NULL,
|
||||
rollout_authorization_ref TEXT NULL,
|
||||
legacy_combined_approval_ref TEXT NULL,
|
||||
legacy_applied_revision INTEGER NULL,
|
||||
connectivity TEXT NULL CHECK (
|
||||
connectivity IS NULL OR connectivity IN ('unknown','reachable','unreachable','stale')
|
||||
),
|
||||
latest_stage TEXT NULL,
|
||||
active_operation_id TEXT NULL,
|
||||
active_operation_stage TEXT NULL CHECK (
|
||||
active_operation_stage IS NULL OR active_operation_stage IN (
|
||||
'deploy_started','blueprint_deploy_started','blueprint_withdraw_started','recovery_started'
|
||||
)
|
||||
),
|
||||
active_operation_at INTEGER NULL,
|
||||
active_generation_id TEXT NULL,
|
||||
active_intent_revision_id TEXT NULL,
|
||||
active_rollout_candidate_id TEXT NULL,
|
||||
failure_stage TEXT NULL CHECK (
|
||||
failure_stage IS NULL OR failure_stage IN (
|
||||
'deploy','recovery','blueprint_deploy','blueprint_withdraw'
|
||||
)
|
||||
),
|
||||
failure_class TEXT NULL,
|
||||
failure_at INTEGER NULL,
|
||||
recovery_ref TEXT NULL,
|
||||
recovery_generation_id TEXT NULL,
|
||||
recovery_phase TEXT NULL CHECK (
|
||||
recovery_phase IS NULL OR recovery_phase IN (
|
||||
'capturing','restoring','compensating','complete','failed'
|
||||
)
|
||||
),
|
||||
interruption_stage TEXT NULL CHECK (
|
||||
interruption_stage IS NULL OR interruption_stage IN (
|
||||
'deploy_started','blueprint_deploy_started','blueprint_withdraw_started','recovery_started'
|
||||
)
|
||||
),
|
||||
interruption_at INTEGER NULL,
|
||||
interruption_operation_id TEXT NULL,
|
||||
interruption_generation_id TEXT NULL,
|
||||
interruption_intent_revision_id TEXT NULL,
|
||||
interruption_rollout_candidate_id TEXT NULL,
|
||||
pause_at INTEGER NULL,
|
||||
pause_reason TEXT NULL,
|
||||
retry_at INTEGER NULL,
|
||||
suspended_at INTEGER NULL,
|
||||
partial_json TEXT NULL,
|
||||
-- Why this target could not prove something, recorded at write time.
|
||||
evidence_limitations_json TEXT NULL,
|
||||
updated_at INTEGER NOT NULL,
|
||||
PRIMARY KEY (application_id, node_id),
|
||||
CHECK (
|
||||
(lkg_unavailable_at IS NULL AND lkg_unavailable_reason IS NULL)
|
||||
OR (lkg_unavailable_at IS NOT NULL AND lkg_unavailable_reason IS NOT NULL)
|
||||
),
|
||||
CHECK (
|
||||
lkg_unavailable_at IS NULL
|
||||
OR (lkg_generation_id IS NULL AND lkg_artifact_set_id IS NULL)
|
||||
)
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_gitops_target_status
|
||||
ON gitops_target_current(target_status);
|
||||
CREATE INDEX IF NOT EXISTS idx_gitops_target_node
|
||||
ON gitops_target_current(node_id);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS gitops_history (
|
||||
id TEXT PRIMARY KEY,
|
||||
created_at INTEGER NOT NULL,
|
||||
application_id TEXT NOT NULL,
|
||||
target_mode TEXT NOT NULL,
|
||||
lifecycle_key TEXT NOT NULL,
|
||||
stack_name TEXT NULL,
|
||||
blueprint_id INTEGER NULL,
|
||||
node_id INTEGER NULL,
|
||||
dedupe_target TEXT NOT NULL,
|
||||
repo_url TEXT NULL,
|
||||
configured_ref TEXT NULL,
|
||||
repo_identity_json TEXT NULL,
|
||||
commit_sha TEXT NULL,
|
||||
generation_id TEXT NULL,
|
||||
artifact_set_id TEXT NULL,
|
||||
intent_revision_id TEXT NULL,
|
||||
rollout_candidate_id TEXT NULL,
|
||||
rollout_generation_id TEXT NULL,
|
||||
source_acceptance_ref TEXT NULL,
|
||||
placement_approval_ref TEXT NULL,
|
||||
rollout_authorization_ref TEXT NULL,
|
||||
legacy_combined_approval_ref TEXT NULL,
|
||||
operation_id TEXT NOT NULL,
|
||||
stage TEXT NOT NULL,
|
||||
outcome TEXT NOT NULL CHECK (outcome IN (
|
||||
'committed','failed','skipped','superseded','recovered','unknown'
|
||||
)),
|
||||
trigger TEXT NOT NULL,
|
||||
actor TEXT NULL,
|
||||
before_json TEXT NOT NULL,
|
||||
after_json TEXT NOT NULL,
|
||||
required_targets_json TEXT NULL,
|
||||
validation_json TEXT NULL,
|
||||
per_target_results_json TEXT NULL,
|
||||
health_run_id TEXT NULL,
|
||||
health_snapshot_json TEXT NULL,
|
||||
invocation_observed_json TEXT NULL,
|
||||
recovery_ref TEXT NULL,
|
||||
redacted_reason_class TEXT NULL
|
||||
);
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS idx_gitops_history_dedupe
|
||||
ON gitops_history(application_id, operation_id, stage, dedupe_target);
|
||||
CREATE INDEX IF NOT EXISTS idx_gitops_history_app_created
|
||||
ON gitops_history(application_id, created_at DESC, id DESC);
|
||||
CREATE INDEX IF NOT EXISTS idx_gitops_history_sha ON gitops_history(commit_sha);
|
||||
CREATE INDEX IF NOT EXISTS idx_gitops_history_gen ON gitops_history(generation_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_gitops_history_artifact ON gitops_history(artifact_set_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_gitops_history_blueprint ON gitops_history(blueprint_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_gitops_history_rollout ON gitops_history(rollout_candidate_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_gitops_history_rollout_gen ON gitops_history(rollout_generation_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_gitops_history_node ON gitops_history(node_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_gitops_history_trigger ON gitops_history(trigger);
|
||||
CREATE INDEX IF NOT EXISTS idx_gitops_history_actor ON gitops_history(actor);
|
||||
CREATE INDEX IF NOT EXISTS idx_gitops_history_outcome ON gitops_history(outcome);
|
||||
CREATE INDEX IF NOT EXISTS idx_gitops_history_repo_ref
|
||||
ON gitops_history(repo_url, configured_ref);
|
||||
CREATE INDEX IF NOT EXISTS idx_gitops_history_stack_created
|
||||
ON gitops_history(stack_name, created_at DESC, id DESC);
|
||||
-- Serves the cross-stack history page, whose ordering and cursor are on
|
||||
-- (created_at, id) with no other filter. Every other index here leads with a
|
||||
-- different column, so without this one that route sorts the whole table.
|
||||
CREATE INDEX IF NOT EXISTS idx_gitops_history_created
|
||||
ON gitops_history(created_at DESC, id DESC);
|
||||
`;
|
||||
@@ -0,0 +1,815 @@
|
||||
import type Database from 'better-sqlite3';
|
||||
import { DatabaseService } from '../DatabaseService';
|
||||
import {
|
||||
decodeArtifactEvidenceJson,
|
||||
decodeGitOpsApprovedTargetEffectJson,
|
||||
decodeGitOpsJson,
|
||||
decodeGitOpsRequiredTargetsJson,
|
||||
GitOpsJsonError,
|
||||
isPreflightFingerprint,
|
||||
} from './json';
|
||||
import type {
|
||||
FutureRolloutAuthorizationBinding,
|
||||
GitOpsApplicationRow,
|
||||
GitOpsApprovalRow,
|
||||
GitOpsArtifactSetRow,
|
||||
GitOpsCreateCheckpointRow,
|
||||
GitOpsCreatePhase,
|
||||
GitOpsGenerationRow,
|
||||
GitOpsIntentRevisionRow,
|
||||
GitOpsRolloutCandidateRow,
|
||||
GitOpsTargetCurrentRow,
|
||||
ResolveApprovalExpected,
|
||||
} from './types';
|
||||
|
||||
export type LiveBlueprintApplication = {
|
||||
id: string;
|
||||
targetMode: GitOpsApplicationRow['target_mode'];
|
||||
lifecycleStatus: GitOpsApplicationRow['lifecycle_status'];
|
||||
};
|
||||
|
||||
export type AssertNoLiveBlueprintResult =
|
||||
| { ok: true }
|
||||
| { ok: false; existing: LiveBlueprintApplication };
|
||||
|
||||
export class GitOpsStore {
|
||||
private static instance: GitOpsStore | undefined;
|
||||
|
||||
static getInstance(): GitOpsStore {
|
||||
if (!GitOpsStore.instance) {
|
||||
GitOpsStore.instance = new GitOpsStore();
|
||||
}
|
||||
return GitOpsStore.instance;
|
||||
}
|
||||
|
||||
static resetForTests(): void {
|
||||
GitOpsStore.instance = undefined;
|
||||
}
|
||||
|
||||
private db(): Database.Database {
|
||||
return DatabaseService.getInstance().getDb();
|
||||
}
|
||||
|
||||
assertNoLiveBlueprintApplication(blueprintId: number): AssertNoLiveBlueprintResult {
|
||||
const row = this.db().prepare(
|
||||
`SELECT id, target_mode, lifecycle_status
|
||||
FROM gitops_applications
|
||||
WHERE blueprint_id = ?
|
||||
AND lifecycle_status IN ('active','creating')
|
||||
AND target_mode IN ('inline_blueprint','blueprint')
|
||||
LIMIT 1`,
|
||||
).get(blueprintId) as {
|
||||
id: string;
|
||||
target_mode: GitOpsApplicationRow['target_mode'];
|
||||
lifecycle_status: GitOpsApplicationRow['lifecycle_status'];
|
||||
} | undefined;
|
||||
if (!row) return { ok: true };
|
||||
return {
|
||||
ok: false,
|
||||
existing: {
|
||||
id: row.id,
|
||||
targetMode: row.target_mode,
|
||||
lifecycleStatus: row.lifecycle_status,
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
getApplication(id: string): GitOpsApplicationRow | undefined {
|
||||
return this.db().prepare('SELECT * FROM gitops_applications WHERE id = ?').get(id) as GitOpsApplicationRow | undefined;
|
||||
}
|
||||
|
||||
getLiveDirectApplication(stackName: string): GitOpsApplicationRow | undefined {
|
||||
return this.db().prepare(
|
||||
`SELECT * FROM gitops_applications
|
||||
WHERE stack_name = ? AND target_mode = 'direct' AND lifecycle_status IN ('active','creating')`,
|
||||
).get(stackName) as GitOpsApplicationRow | undefined;
|
||||
}
|
||||
|
||||
/**
|
||||
* The live application for a Blueprint, in either Blueprint mode.
|
||||
*
|
||||
* One query across both modes because a Blueprint owns at most one live
|
||||
* application whichever way it is delivered, and the unique live index
|
||||
* enforces exactly that.
|
||||
*/
|
||||
getLiveBlueprintApplication(blueprintId: number): GitOpsApplicationRow | undefined {
|
||||
return this.db().prepare(
|
||||
`SELECT * FROM gitops_applications
|
||||
WHERE blueprint_id = ?
|
||||
AND target_mode IN ('inline_blueprint','blueprint')
|
||||
AND lifecycle_status IN ('active','creating')`,
|
||||
).get(blueprintId) as GitOpsApplicationRow | undefined;
|
||||
}
|
||||
|
||||
/**
|
||||
* The most recently detached Direct application for a stack, if any.
|
||||
*
|
||||
* Consulted only after the live lookup misses. `applicationTombstoned` keeps
|
||||
* the configured identity and SHA pointers as frozen facts precisely so the
|
||||
* projection can still say what an application was, and `deriveSource` has a
|
||||
* `not_live` status for it, but neither could be reached while every entry
|
||||
* point filtered to the live rows.
|
||||
*
|
||||
* `detached` only, never `deleted`. A detached application's files are still
|
||||
* on disk and still describe that stack. A deleted one means the stack is
|
||||
* gone, so any directory of that name now belongs to something else, and
|
||||
* `readAuth` refuses stack-grant reads on deleted rows for the same
|
||||
* name-reuse reason.
|
||||
*
|
||||
* Newest first, because a stack name can be detached and reattached
|
||||
* repeatedly and only the latest detachment describes what was there last.
|
||||
* `rowid` breaks a tie rather than `id`, which is a random UUID and orders
|
||||
* arbitrarily; ties are reachable because a transaction stamps every row it
|
||||
* touches with one `envelope.at`.
|
||||
*/
|
||||
getDetachedDirectApplication(stackName: string): GitOpsApplicationRow | undefined {
|
||||
return this.db().prepare(
|
||||
`SELECT * FROM gitops_applications
|
||||
WHERE stack_name = ? AND target_mode = 'direct' AND lifecycle_status = 'detached'
|
||||
ORDER BY updated_at DESC, rowid DESC
|
||||
LIMIT 1`,
|
||||
).get(stackName) as GitOpsApplicationRow | undefined;
|
||||
}
|
||||
|
||||
/** Direct applications that never reached their success boundary. */
|
||||
listCreatingDirectApplications(): GitOpsApplicationRow[] {
|
||||
return this.db().prepare(
|
||||
`SELECT * FROM gitops_applications
|
||||
WHERE target_mode = 'direct' AND lifecycle_status = 'creating'
|
||||
ORDER BY created_at ASC`,
|
||||
).all() as GitOpsApplicationRow[];
|
||||
}
|
||||
|
||||
getGeneration(id: string): GitOpsGenerationRow | undefined {
|
||||
return this.db().prepare('SELECT * FROM gitops_generations WHERE id = ?').get(id) as GitOpsGenerationRow | undefined;
|
||||
}
|
||||
|
||||
getArtifactSet(id: string): GitOpsArtifactSetRow | undefined {
|
||||
return this.db().prepare('SELECT * FROM gitops_artifact_sets WHERE id = ?').get(id) as GitOpsArtifactSetRow | undefined;
|
||||
}
|
||||
|
||||
getIntentRevision(id: string): GitOpsIntentRevisionRow | undefined {
|
||||
return this.db().prepare('SELECT * FROM gitops_intent_revisions WHERE id = ?').get(id) as GitOpsIntentRevisionRow | undefined;
|
||||
}
|
||||
|
||||
getRolloutCandidate(id: string): GitOpsRolloutCandidateRow | undefined {
|
||||
return this.db().prepare('SELECT * FROM gitops_rollout_candidates WHERE id = ?').get(id) as GitOpsRolloutCandidateRow | undefined;
|
||||
}
|
||||
|
||||
getApproval(id: string): GitOpsApprovalRow | undefined {
|
||||
return this.db().prepare('SELECT * FROM gitops_approvals WHERE id = ?').get(id) as GitOpsApprovalRow | undefined;
|
||||
}
|
||||
|
||||
getTarget(applicationId: string, nodeId: number): GitOpsTargetCurrentRow | undefined {
|
||||
return this.db().prepare(
|
||||
'SELECT * FROM gitops_target_current WHERE application_id = ? AND node_id = ?',
|
||||
).get(applicationId, nodeId) as GitOpsTargetCurrentRow | undefined;
|
||||
}
|
||||
|
||||
listTargets(applicationId: string): GitOpsTargetCurrentRow[] {
|
||||
return this.db().prepare(
|
||||
'SELECT * FROM gitops_target_current WHERE application_id = ? ORDER BY node_id ASC',
|
||||
).all(applicationId) as GitOpsTargetCurrentRow[];
|
||||
}
|
||||
|
||||
/**
|
||||
* Live applications that were mid-operation, on the application row or on any
|
||||
* of their targets.
|
||||
*
|
||||
* Read at boot to reclassify work the previous process never finished. An
|
||||
* operation left open reports as still running forever, and offers no actions
|
||||
* while it does.
|
||||
*/
|
||||
listApplicationsWithOpenOperations(): GitOpsApplicationRow[] {
|
||||
return this.db().prepare(
|
||||
`SELECT a.* FROM gitops_applications a
|
||||
WHERE a.lifecycle_status IN ('active','creating')
|
||||
AND (
|
||||
a.active_operation_stage IS NOT NULL
|
||||
OR EXISTS (
|
||||
SELECT 1 FROM gitops_target_current t
|
||||
WHERE t.application_id = a.id AND t.active_operation_stage IS NOT NULL
|
||||
)
|
||||
)
|
||||
ORDER BY a.created_at ASC`,
|
||||
).all() as GitOpsApplicationRow[];
|
||||
}
|
||||
|
||||
/** Every live target on one node, across all applications. */
|
||||
listActiveTargetsForNode(nodeId: number): GitOpsTargetCurrentRow[] {
|
||||
return this.db().prepare(
|
||||
`SELECT * FROM gitops_target_current
|
||||
WHERE node_id = ? AND target_status = 'active'
|
||||
ORDER BY application_id ASC`,
|
||||
).all(nodeId) as GitOpsTargetCurrentRow[];
|
||||
}
|
||||
|
||||
newestSourceAcceptanceId(applicationId: string, generationId: string): string | null {
|
||||
const row = this.db().prepare(
|
||||
`SELECT id FROM gitops_approvals
|
||||
WHERE application_id = ? AND kind = 'source_acceptance' AND authoritative = 1 AND generation_id = ?
|
||||
ORDER BY created_at DESC, id DESC LIMIT 1`,
|
||||
).get(applicationId, generationId) as { id: string } | undefined;
|
||||
return row?.id ?? null;
|
||||
}
|
||||
|
||||
getMigrationCheckpoint(scope: string): { scope: string; schema_version: number; fingerprint: string } | undefined {
|
||||
return this.db().prepare(
|
||||
'SELECT scope, schema_version, fingerprint FROM gitops_migration_checkpoints WHERE scope = ?',
|
||||
).get(scope) as { scope: string; schema_version: number; fingerprint: string } | undefined;
|
||||
}
|
||||
|
||||
/**
|
||||
* Record that this scope has been migrated at this schema version and
|
||||
* configuration fingerprint.
|
||||
*
|
||||
* Replay is decided from the triple: an unchanged fingerprint skips, a
|
||||
* changed one re-runs the matrix. It never licenses upgrading an already
|
||||
* justified pointer to a stronger claim.
|
||||
*/
|
||||
upsertMigrationCheckpoint(scope: string, schemaVersion: number, fingerprint: string, at: number): void {
|
||||
this.db().prepare(
|
||||
`INSERT INTO gitops_migration_checkpoints (scope, schema_version, fingerprint, migrated_at)
|
||||
VALUES (?, ?, ?, ?)
|
||||
ON CONFLICT(scope) DO UPDATE SET
|
||||
schema_version=excluded.schema_version,
|
||||
fingerprint=excluded.fingerprint,
|
||||
migrated_at=excluded.migrated_at`,
|
||||
).run(scope, schemaVersion, fingerprint, at);
|
||||
}
|
||||
|
||||
/**
|
||||
* Persist an application's mutable columns without going through a
|
||||
* transition.
|
||||
*
|
||||
* Used only by migration, which builds a whole row from evidence rather than
|
||||
* moving one pointer at a time. Every other writer goes through the
|
||||
* transitions so the change lands in history.
|
||||
*/
|
||||
writeApplicationPointers(app: GitOpsApplicationRow): void {
|
||||
this.db().prepare(
|
||||
`UPDATE gitops_applications SET
|
||||
desired_commit_sha=?, fetched_commit_sha=?, accepted_generation_id=?,
|
||||
artifact_set_id=?, latest_artifact_set_id=?, evidence_limitations_json=?, updated_at=?
|
||||
WHERE id=?`,
|
||||
).run(
|
||||
app.desired_commit_sha, app.fetched_commit_sha, app.accepted_generation_id,
|
||||
app.artifact_set_id, app.latest_artifact_set_id, app.evidence_limitations_json,
|
||||
app.updated_at, app.id,
|
||||
);
|
||||
}
|
||||
|
||||
insertCreateCheckpoint(row: GitOpsCreateCheckpointRow): void {
|
||||
decodeGitOpsJson(row.compose_paths_json);
|
||||
this.db().prepare(
|
||||
`INSERT INTO gitops_create_checkpoints (
|
||||
application_id, stack_name, phase, generation_id, operation_id, repo_url, branch,
|
||||
compose_path, compose_paths_json, context_dir, sync_env, env_path, auth_type,
|
||||
encrypted_token, auto_apply_on_webhook, auto_deploy_on_apply, commit_sha,
|
||||
applied_spec_json, created_managed_root, created_at, updated_at
|
||||
) VALUES (${Array(21).fill('?').join(', ')})`,
|
||||
).run(
|
||||
row.application_id, row.stack_name, row.phase, row.generation_id, row.operation_id,
|
||||
row.repo_url, row.branch, row.compose_path, row.compose_paths_json, row.context_dir,
|
||||
row.sync_env, row.env_path, row.auth_type, row.encrypted_token, row.auto_apply_on_webhook,
|
||||
row.auto_deploy_on_apply, row.commit_sha, row.applied_spec_json, row.created_managed_root,
|
||||
row.created_at, row.updated_at,
|
||||
);
|
||||
}
|
||||
|
||||
getCreateCheckpoint(applicationId: string): GitOpsCreateCheckpointRow | undefined {
|
||||
return this.db().prepare(
|
||||
'SELECT * FROM gitops_create_checkpoints WHERE application_id = ?',
|
||||
).get(applicationId) as GitOpsCreateCheckpointRow | undefined;
|
||||
}
|
||||
|
||||
listCreateCheckpoints(): GitOpsCreateCheckpointRow[] {
|
||||
return this.db().prepare(
|
||||
'SELECT * FROM gitops_create_checkpoints ORDER BY created_at ASC',
|
||||
).all() as GitOpsCreateCheckpointRow[];
|
||||
}
|
||||
|
||||
/** Advance the phase, and optionally record facts the phase depends on. */
|
||||
updateCreateCheckpoint(
|
||||
applicationId: string,
|
||||
patch: {
|
||||
phase?: GitOpsCreatePhase;
|
||||
generationId?: string | null;
|
||||
commitSha?: string | null;
|
||||
appliedSpecJson?: string | null;
|
||||
createdManagedRoot?: number;
|
||||
},
|
||||
at: number,
|
||||
): void {
|
||||
const current = this.getCreateCheckpoint(applicationId);
|
||||
if (!current) throw new Error('create checkpoint not found');
|
||||
this.db().prepare(
|
||||
`UPDATE gitops_create_checkpoints SET
|
||||
phase=?, generation_id=?, commit_sha=?, applied_spec_json=?,
|
||||
created_managed_root=?, updated_at=?
|
||||
WHERE application_id=?`,
|
||||
).run(
|
||||
patch.phase ?? current.phase,
|
||||
patch.generationId === undefined ? current.generation_id : patch.generationId,
|
||||
patch.commitSha === undefined ? current.commit_sha : patch.commitSha,
|
||||
patch.appliedSpecJson === undefined ? current.applied_spec_json : patch.appliedSpecJson,
|
||||
patch.createdManagedRoot ?? current.created_managed_root,
|
||||
at,
|
||||
applicationId,
|
||||
);
|
||||
}
|
||||
|
||||
deleteCreateCheckpoint(applicationId: string): void {
|
||||
this.db().prepare('DELETE FROM gitops_create_checkpoints WHERE application_id = ?').run(applicationId);
|
||||
}
|
||||
|
||||
insertApproval(row: GitOpsApprovalRow): void {
|
||||
// Decode every JSON column the resolver will later read, so a malformed
|
||||
// payload aborts the write instead of persisting an approval that reads
|
||||
// back as absent. Rollout-authorization blast is reserved policy payload
|
||||
// and is never a node set, so it is only checked for well-formedness.
|
||||
if (row.required_targets_json !== null) decodeGitOpsRequiredTargetsJson(row.required_targets_json);
|
||||
if (row.blast_json !== null) {
|
||||
if (row.kind === 'placement_approval') decodeGitOpsApprovedTargetEffectJson(row.blast_json);
|
||||
else decodeGitOpsJson(row.blast_json);
|
||||
}
|
||||
if (row.policy_provenance_json !== null) decodeGitOpsJson(row.policy_provenance_json);
|
||||
this.db().prepare(
|
||||
`INSERT INTO gitops_approvals (
|
||||
id, kind, authority, authoritative, application_id, generation_id, intent_revision_id,
|
||||
artifact_set_id, rollout_candidate_id, rollout_generation_id, source_acceptance_ref,
|
||||
placement_approval_ref, required_targets_json, preflight_fingerprint, fingerprint,
|
||||
blast_json, policy_provenance_json, actor, created_at
|
||||
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`,
|
||||
).run(
|
||||
row.id, row.kind, row.authority, row.authoritative, row.application_id, row.generation_id,
|
||||
row.intent_revision_id, row.artifact_set_id, row.rollout_candidate_id, row.rollout_generation_id,
|
||||
row.source_acceptance_ref, row.placement_approval_ref, row.required_targets_json,
|
||||
row.preflight_fingerprint, row.fingerprint, row.blast_json, row.policy_provenance_json,
|
||||
row.actor, row.created_at,
|
||||
);
|
||||
}
|
||||
|
||||
insertApplication(row: GitOpsApplicationRow): void {
|
||||
this.db().prepare(
|
||||
`INSERT INTO gitops_applications (
|
||||
id, lifecycle_key, lifecycle_status, target_mode, stack_name, blueprint_id,
|
||||
configured_repo_url, repo_identity_json, configured_ref, compose_paths_json,
|
||||
context_dir, sync_env, env_path, materialization_fingerprint, desired_commit_sha,
|
||||
fetched_commit_sha, candidate_generation_id, accepted_generation_id,
|
||||
candidate_plan_blocked, review_required, artifact_set_id, latest_artifact_set_id,
|
||||
intent_revision_id, rollout_candidate_id, rollout_generation_id, source_acceptance_ref,
|
||||
placement_approval_ref, rollout_authorization_ref, legacy_combined_approval_ref,
|
||||
preflight_fingerprint, latest_operation_id, active_operation_id, active_operation_stage,
|
||||
active_operation_at, active_generation_id, pause_at, pause_reason, partial_json,
|
||||
failure_stage, failure_class, failure_at, retry_at, retry_count, suspended_at,
|
||||
recovery_ref, recovery_phase, interruption_stage, interruption_at,
|
||||
interruption_operation_id, interruption_generation_id, evidence_fresh_at,
|
||||
evidence_limitations_json, created_at, updated_at
|
||||
) VALUES (${Array(54).fill('?').join(', ')})`,
|
||||
).run(
|
||||
row.id, row.lifecycle_key, row.lifecycle_status, row.target_mode, row.stack_name, row.blueprint_id,
|
||||
row.configured_repo_url, row.repo_identity_json, row.configured_ref, row.compose_paths_json,
|
||||
row.context_dir, row.sync_env, row.env_path, row.materialization_fingerprint, row.desired_commit_sha,
|
||||
row.fetched_commit_sha, row.candidate_generation_id, row.accepted_generation_id,
|
||||
row.candidate_plan_blocked, row.review_required, row.artifact_set_id, row.latest_artifact_set_id,
|
||||
row.intent_revision_id, row.rollout_candidate_id, row.rollout_generation_id, row.source_acceptance_ref,
|
||||
row.placement_approval_ref, row.rollout_authorization_ref, row.legacy_combined_approval_ref,
|
||||
row.preflight_fingerprint, row.latest_operation_id, row.active_operation_id, row.active_operation_stage,
|
||||
row.active_operation_at, row.active_generation_id, row.pause_at, row.pause_reason, row.partial_json,
|
||||
row.failure_stage, row.failure_class, row.failure_at, row.retry_at, row.retry_count, row.suspended_at,
|
||||
row.recovery_ref, row.recovery_phase, row.interruption_stage, row.interruption_at,
|
||||
row.interruption_operation_id, row.interruption_generation_id, row.evidence_fresh_at,
|
||||
row.evidence_limitations_json, row.created_at, row.updated_at,
|
||||
);
|
||||
}
|
||||
|
||||
insertGeneration(row: GitOpsGenerationRow): void {
|
||||
this.db().prepare(
|
||||
`INSERT INTO gitops_generations (
|
||||
id, application_id, commit_sha, repo_url, configured_ref, repo_identity_json,
|
||||
manifest_version, candidate_dir, applied_dir, expected_invocation_json,
|
||||
materialization_fingerprint, validation_ok, plan_blocked, change_plan_fingerprint,
|
||||
operation_id, trigger, actor, previous_generation_id, redacted_limitations_json, created_at
|
||||
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`,
|
||||
).run(
|
||||
row.id, row.application_id, row.commit_sha, row.repo_url, row.configured_ref, row.repo_identity_json,
|
||||
row.manifest_version, row.candidate_dir, row.applied_dir, row.expected_invocation_json,
|
||||
row.materialization_fingerprint, row.validation_ok, row.plan_blocked, row.change_plan_fingerprint,
|
||||
row.operation_id, row.trigger, row.actor, row.previous_generation_id, row.redacted_limitations_json,
|
||||
row.created_at,
|
||||
);
|
||||
}
|
||||
|
||||
insertArtifactSet(row: GitOpsArtifactSetRow): void {
|
||||
const evidence = decodeArtifactEvidenceJson(row.evidence_json);
|
||||
if (evidence.kind !== row.qualification) {
|
||||
throw new GitOpsJsonError('artifact qualification must match evidence_json.kind');
|
||||
}
|
||||
if (row.authoritative !== 0) {
|
||||
throw new Error('artifact rows must be non-authoritative until qualification is accepted');
|
||||
}
|
||||
this.db().prepare(
|
||||
`INSERT INTO gitops_artifact_sets (
|
||||
id, generation_id, evidence_version, authoritative, qualification, evidence_json, created_at
|
||||
) VALUES (?, ?, ?, ?, ?, ?, ?)`,
|
||||
).run(
|
||||
row.id, row.generation_id, row.evidence_version, row.authoritative,
|
||||
row.qualification, row.evidence_json, row.created_at,
|
||||
);
|
||||
}
|
||||
|
||||
insertIntentRevision(row: GitOpsIntentRevisionRow): void {
|
||||
this.db().prepare(
|
||||
`INSERT INTO gitops_intent_revisions (
|
||||
id, application_id, blueprint_id, compose_content_sha256, blueprint_revision,
|
||||
deploy_stack_name, selector_json, pinned_node_id, cordon_implications_json,
|
||||
rollout_strategy_json, runtime_drift_policy, stateful_policy_json,
|
||||
health_failure_rollback_policy_json, operation_id, actor, created_at
|
||||
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`,
|
||||
).run(
|
||||
row.id, row.application_id, row.blueprint_id, row.compose_content_sha256, row.blueprint_revision,
|
||||
row.deploy_stack_name, row.selector_json, row.pinned_node_id, row.cordon_implications_json,
|
||||
row.rollout_strategy_json, row.runtime_drift_policy, row.stateful_policy_json,
|
||||
row.health_failure_rollback_policy_json, row.operation_id, row.actor, row.created_at,
|
||||
);
|
||||
}
|
||||
|
||||
insertRolloutCandidate(row: GitOpsRolloutCandidateRow): void {
|
||||
decodeGitOpsRequiredTargetsJson(row.required_targets_json);
|
||||
this.db().prepare(
|
||||
`INSERT INTO gitops_rollout_candidates (
|
||||
id, application_id, intent_revision_id, compose_content_sha256, accepted_generation_id,
|
||||
artifact_set_id, required_targets_json, authoritative, provenance, operation_id, created_at
|
||||
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`,
|
||||
).run(
|
||||
row.id, row.application_id, row.intent_revision_id, row.compose_content_sha256,
|
||||
row.accepted_generation_id, row.artifact_set_id, row.required_targets_json,
|
||||
row.authoritative, row.provenance, row.operation_id, row.created_at,
|
||||
);
|
||||
}
|
||||
|
||||
upsertTarget(row: GitOpsTargetCurrentRow): void {
|
||||
this.assertTargetInvariants(row);
|
||||
this.db().prepare(
|
||||
`INSERT INTO gitops_target_current (
|
||||
application_id, node_id, target_status, desired_generation_id, candidate_generation_id,
|
||||
applied_generation_id, deployed_generation_id, healthy_generation_id, lkg_generation_id,
|
||||
lkg_artifact_set_id, lkg_unavailable_at, lkg_unavailable_reason, expected_artifact_set_id,
|
||||
latest_artifact_set_id, observed_artifact_identity_json, intent_revision_id,
|
||||
rollout_candidate_id, rollout_generation_id, source_acceptance_ref, placement_approval_ref,
|
||||
rollout_authorization_ref, legacy_combined_approval_ref, legacy_applied_revision,
|
||||
connectivity, latest_stage, active_operation_id, active_operation_stage, active_operation_at,
|
||||
active_generation_id, active_intent_revision_id, active_rollout_candidate_id,
|
||||
failure_stage, failure_class, failure_at, recovery_ref, recovery_generation_id,
|
||||
recovery_phase, interruption_stage, interruption_at, interruption_operation_id,
|
||||
interruption_generation_id, interruption_intent_revision_id, interruption_rollout_candidate_id,
|
||||
pause_at, pause_reason, retry_at, suspended_at, partial_json, evidence_limitations_json, updated_at
|
||||
) VALUES (${Array(50).fill('?').join(', ')})
|
||||
ON CONFLICT(application_id, node_id) DO UPDATE SET
|
||||
target_status=excluded.target_status,
|
||||
desired_generation_id=excluded.desired_generation_id,
|
||||
candidate_generation_id=excluded.candidate_generation_id,
|
||||
applied_generation_id=excluded.applied_generation_id,
|
||||
deployed_generation_id=excluded.deployed_generation_id,
|
||||
healthy_generation_id=excluded.healthy_generation_id,
|
||||
lkg_generation_id=excluded.lkg_generation_id,
|
||||
lkg_artifact_set_id=excluded.lkg_artifact_set_id,
|
||||
lkg_unavailable_at=excluded.lkg_unavailable_at,
|
||||
lkg_unavailable_reason=excluded.lkg_unavailable_reason,
|
||||
expected_artifact_set_id=excluded.expected_artifact_set_id,
|
||||
latest_artifact_set_id=excluded.latest_artifact_set_id,
|
||||
observed_artifact_identity_json=excluded.observed_artifact_identity_json,
|
||||
intent_revision_id=excluded.intent_revision_id,
|
||||
rollout_candidate_id=excluded.rollout_candidate_id,
|
||||
rollout_generation_id=excluded.rollout_generation_id,
|
||||
source_acceptance_ref=excluded.source_acceptance_ref,
|
||||
placement_approval_ref=excluded.placement_approval_ref,
|
||||
rollout_authorization_ref=excluded.rollout_authorization_ref,
|
||||
legacy_combined_approval_ref=excluded.legacy_combined_approval_ref,
|
||||
legacy_applied_revision=excluded.legacy_applied_revision,
|
||||
connectivity=excluded.connectivity,
|
||||
latest_stage=excluded.latest_stage,
|
||||
active_operation_id=excluded.active_operation_id,
|
||||
active_operation_stage=excluded.active_operation_stage,
|
||||
active_operation_at=excluded.active_operation_at,
|
||||
active_generation_id=excluded.active_generation_id,
|
||||
active_intent_revision_id=excluded.active_intent_revision_id,
|
||||
active_rollout_candidate_id=excluded.active_rollout_candidate_id,
|
||||
failure_stage=excluded.failure_stage,
|
||||
failure_class=excluded.failure_class,
|
||||
failure_at=excluded.failure_at,
|
||||
recovery_ref=excluded.recovery_ref,
|
||||
recovery_generation_id=excluded.recovery_generation_id,
|
||||
recovery_phase=excluded.recovery_phase,
|
||||
interruption_stage=excluded.interruption_stage,
|
||||
interruption_at=excluded.interruption_at,
|
||||
interruption_operation_id=excluded.interruption_operation_id,
|
||||
interruption_generation_id=excluded.interruption_generation_id,
|
||||
interruption_intent_revision_id=excluded.interruption_intent_revision_id,
|
||||
interruption_rollout_candidate_id=excluded.interruption_rollout_candidate_id,
|
||||
pause_at=excluded.pause_at,
|
||||
pause_reason=excluded.pause_reason,
|
||||
retry_at=excluded.retry_at,
|
||||
suspended_at=excluded.suspended_at,
|
||||
partial_json=excluded.partial_json,
|
||||
evidence_limitations_json=excluded.evidence_limitations_json,
|
||||
updated_at=excluded.updated_at`,
|
||||
).run(
|
||||
row.application_id, row.node_id, row.target_status, row.desired_generation_id, row.candidate_generation_id,
|
||||
row.applied_generation_id, row.deployed_generation_id, row.healthy_generation_id, row.lkg_generation_id,
|
||||
row.lkg_artifact_set_id, row.lkg_unavailable_at, row.lkg_unavailable_reason, row.expected_artifact_set_id,
|
||||
row.latest_artifact_set_id, row.observed_artifact_identity_json, row.intent_revision_id,
|
||||
row.rollout_candidate_id, row.rollout_generation_id, row.source_acceptance_ref, row.placement_approval_ref,
|
||||
row.rollout_authorization_ref, row.legacy_combined_approval_ref, row.legacy_applied_revision,
|
||||
row.connectivity, row.latest_stage, row.active_operation_id, row.active_operation_stage, row.active_operation_at,
|
||||
row.active_generation_id, row.active_intent_revision_id, row.active_rollout_candidate_id,
|
||||
row.failure_stage, row.failure_class, row.failure_at, row.recovery_ref, row.recovery_generation_id,
|
||||
row.recovery_phase, row.interruption_stage, row.interruption_at, row.interruption_operation_id,
|
||||
row.interruption_generation_id, row.interruption_intent_revision_id, row.interruption_rollout_candidate_id,
|
||||
row.pause_at, row.pause_reason, row.retry_at, row.suspended_at, row.partial_json,
|
||||
row.evidence_limitations_json, row.updated_at,
|
||||
);
|
||||
}
|
||||
|
||||
resolveApprovalRef(id: string, expected: ResolveApprovalExpected): GitOpsApprovalRow | null {
|
||||
const row = this.getApproval(id);
|
||||
if (!row) return null;
|
||||
if (row.application_id !== expected.applicationId) return null;
|
||||
if (row.kind !== expected.kind) return null;
|
||||
// legacy_combined inverts the flag on purpose. It is a migration marker for
|
||||
// an approval made before source acceptance and placement were separable,
|
||||
// so it can never stand as proof for either one. Requiring
|
||||
// authoritative = 0 is what stops it being copied into a decomposed slot.
|
||||
if (expected.kind === 'legacy_combined') {
|
||||
if (row.authoritative !== 0 || row.authority !== 'legacy_combined') return null;
|
||||
return row;
|
||||
}
|
||||
if (row.authoritative !== 1) return null;
|
||||
if (row.authority !== 'operator' && row.authority !== 'configured_policy') return null;
|
||||
|
||||
if (expected.kind === 'source_acceptance') {
|
||||
if (!row.generation_id) return null;
|
||||
const generation = this.getGeneration(row.generation_id);
|
||||
if (!generation || generation.application_id !== expected.applicationId) return null;
|
||||
if (row.generation_id !== expected.generationId) return null;
|
||||
return row;
|
||||
}
|
||||
|
||||
if (expected.kind === 'placement_approval') {
|
||||
if (!row.intent_revision_id || row.intent_revision_id !== expected.intentRevisionId) return null;
|
||||
if (!row.blast_json) return null;
|
||||
let effect;
|
||||
try {
|
||||
effect = decodeGitOpsApprovedTargetEffectJson(row.blast_json);
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
if (!placementEffectCompatible(effect, expected.requiredNodeIds)) return null;
|
||||
return row;
|
||||
}
|
||||
|
||||
const reconstructed = this.reconstructAuthorizationBinding(row);
|
||||
if (!reconstructed) return null;
|
||||
if (!authorizationBindingsEqual(reconstructed, expected.binding)) return null;
|
||||
const source = this.resolveApprovalRef(reconstructed.sourceAcceptanceRef, {
|
||||
kind: 'source_acceptance',
|
||||
applicationId: expected.applicationId,
|
||||
generationId: expected.binding.acceptedGenerationId,
|
||||
});
|
||||
if (!source) return null;
|
||||
const placement = this.resolveApprovalRef(reconstructed.placementApprovalRef, {
|
||||
kind: 'placement_approval',
|
||||
applicationId: expected.applicationId,
|
||||
intentRevisionId: expected.binding.intentRevisionId,
|
||||
requiredNodeIds: expected.binding.requiredNodeIds,
|
||||
});
|
||||
if (!placement) return null;
|
||||
return row;
|
||||
}
|
||||
|
||||
reconstructAuthorizationBinding(row: GitOpsApprovalRow): FutureRolloutAuthorizationBinding | null {
|
||||
if (row.kind !== 'rollout_authorization') return null;
|
||||
if (
|
||||
!row.rollout_candidate_id
|
||||
|| !row.generation_id
|
||||
|| !row.artifact_set_id
|
||||
|| !row.intent_revision_id
|
||||
|| !row.source_acceptance_ref
|
||||
|| !row.placement_approval_ref
|
||||
|| !row.required_targets_json
|
||||
|| !row.preflight_fingerprint
|
||||
) {
|
||||
return null;
|
||||
}
|
||||
if (!isPreflightFingerprint(row.preflight_fingerprint)) return null;
|
||||
let required;
|
||||
try {
|
||||
required = decodeGitOpsRequiredTargetsJson(row.required_targets_json);
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
return {
|
||||
rolloutCandidateId: row.rollout_candidate_id,
|
||||
acceptedGenerationId: row.generation_id,
|
||||
artifactSetId: row.artifact_set_id,
|
||||
intentRevisionId: row.intent_revision_id,
|
||||
requiredNodeIds: required.nodeIds,
|
||||
sourceAcceptanceRef: row.source_acceptance_ref,
|
||||
placementApprovalRef: row.placement_approval_ref,
|
||||
preflightFingerprint: row.preflight_fingerprint,
|
||||
};
|
||||
}
|
||||
|
||||
currentAuthorizationBinding(app: GitOpsApplicationRow): FutureRolloutAuthorizationBinding | null {
|
||||
if (
|
||||
!app.rollout_candidate_id
|
||||
|| !app.accepted_generation_id
|
||||
|| !app.artifact_set_id
|
||||
|| !app.intent_revision_id
|
||||
|| !app.source_acceptance_ref
|
||||
|| !app.placement_approval_ref
|
||||
|| !app.preflight_fingerprint
|
||||
) {
|
||||
return null;
|
||||
}
|
||||
if (!isPreflightFingerprint(app.preflight_fingerprint)) return null;
|
||||
const candidate = this.getRolloutCandidate(app.rollout_candidate_id);
|
||||
if (!candidate || candidate.application_id !== app.id) return null;
|
||||
if (candidate.accepted_generation_id !== app.accepted_generation_id) return null;
|
||||
if (candidate.artifact_set_id !== app.artifact_set_id) return null;
|
||||
if (candidate.intent_revision_id !== app.intent_revision_id) return null;
|
||||
const generation = this.getGeneration(app.accepted_generation_id);
|
||||
if (!generation || generation.application_id !== app.id) return null;
|
||||
const artifact = this.getArtifactSet(app.artifact_set_id);
|
||||
if (!artifact || artifact.generation_id !== app.accepted_generation_id) return null;
|
||||
let required;
|
||||
try {
|
||||
required = decodeGitOpsRequiredTargetsJson(candidate.required_targets_json);
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
const source = this.resolveApprovalRef(app.source_acceptance_ref, {
|
||||
kind: 'source_acceptance',
|
||||
applicationId: app.id,
|
||||
generationId: app.accepted_generation_id,
|
||||
});
|
||||
if (!source) return null;
|
||||
const placement = this.resolveApprovalRef(app.placement_approval_ref, {
|
||||
kind: 'placement_approval',
|
||||
applicationId: app.id,
|
||||
intentRevisionId: app.intent_revision_id,
|
||||
requiredNodeIds: required.nodeIds,
|
||||
});
|
||||
if (!placement) return null;
|
||||
return {
|
||||
rolloutCandidateId: app.rollout_candidate_id,
|
||||
acceptedGenerationId: app.accepted_generation_id,
|
||||
artifactSetId: app.artifact_set_id,
|
||||
intentRevisionId: app.intent_revision_id,
|
||||
requiredNodeIds: required.nodeIds,
|
||||
sourceAcceptanceRef: app.source_acceptance_ref,
|
||||
placementApprovalRef: app.placement_approval_ref,
|
||||
preflightFingerprint: app.preflight_fingerprint,
|
||||
};
|
||||
}
|
||||
|
||||
private assertTargetInvariants(row: GitOpsTargetCurrentRow): void {
|
||||
if (row.lkg_artifact_set_id) {
|
||||
const artifact = this.getArtifactSet(row.lkg_artifact_set_id);
|
||||
if (!artifact || artifact.generation_id !== row.lkg_generation_id) {
|
||||
throw new Error('lkg_artifact_set_id must belong to lkg_generation_id');
|
||||
}
|
||||
}
|
||||
if (row.lkg_unavailable_at !== null && row.lkg_generation_id !== null) {
|
||||
throw new Error('lkg unavailability cannot coexist with an LKG generation');
|
||||
}
|
||||
this.assertArtifactPointer(row.expected_artifact_set_id, row.desired_generation_id);
|
||||
this.assertArtifactPointer(row.latest_artifact_set_id, row.desired_generation_id);
|
||||
// A pointer to a row that does not exist is rejected, not tolerated: the
|
||||
// write is where the bad reference is cheap to find. Tolerating it moves
|
||||
// the symptom to derivation, far from the transition that caused it.
|
||||
if (row.desired_generation_id) {
|
||||
const generation = this.getGeneration(row.desired_generation_id);
|
||||
if (!generation || generation.application_id !== row.application_id) {
|
||||
throw new Error('target desired generation must belong to the application');
|
||||
}
|
||||
}
|
||||
if (row.candidate_generation_id) {
|
||||
const generation = this.getGeneration(row.candidate_generation_id);
|
||||
if (!generation || generation.application_id !== row.application_id) {
|
||||
throw new Error('target candidate generation must belong to the application');
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private assertArtifactPointer(artifactSetId: string | null, desiredGenerationId: string | null): void {
|
||||
if (!artifactSetId) return;
|
||||
if (!desiredGenerationId) {
|
||||
throw new Error('artifact pointer requires desired_generation_id');
|
||||
}
|
||||
const artifact = this.getArtifactSet(artifactSetId);
|
||||
if (!artifact || artifact.generation_id !== desiredGenerationId) {
|
||||
throw new Error('target artifact pointer must belong to desired_generation_id');
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether an approved placement effect can authorize this required target set.
|
||||
*
|
||||
* This is a non-contradiction check, not a coverage check. A node the approval
|
||||
* places must be required, and a node it removes must not be, but a required
|
||||
* node absent from the effect is fine: it is already converged, so the approved
|
||||
* action list has nothing to say about it. An empty effect is therefore valid
|
||||
* against any required set. Do not tighten this into set equality; a converged
|
||||
* fleet legitimately produces no actions to approve.
|
||||
*/
|
||||
export function placementEffectCompatible(
|
||||
effect: Array<{ nodeId: number; outcome: 'place' | 'remove' }>,
|
||||
requiredNodeIds: readonly number[],
|
||||
): boolean {
|
||||
const required = new Set(requiredNodeIds);
|
||||
for (const entry of effect) {
|
||||
if (entry.outcome === 'place' && !required.has(entry.nodeId)) return false;
|
||||
if (entry.outcome === 'remove' && required.has(entry.nodeId)) return false;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
export function authorizationBindingsEqual(
|
||||
left: FutureRolloutAuthorizationBinding,
|
||||
right: FutureRolloutAuthorizationBinding,
|
||||
): boolean {
|
||||
if (left.rolloutCandidateId !== right.rolloutCandidateId) return false;
|
||||
if (left.acceptedGenerationId !== right.acceptedGenerationId) return false;
|
||||
if (left.artifactSetId !== right.artifactSetId) return false;
|
||||
if (left.intentRevisionId !== right.intentRevisionId) return false;
|
||||
if (left.sourceAcceptanceRef !== right.sourceAcceptanceRef) return false;
|
||||
if (left.placementApprovalRef !== right.placementApprovalRef) return false;
|
||||
if (left.preflightFingerprint !== right.preflightFingerprint) return false;
|
||||
if (left.requiredNodeIds.length !== right.requiredNodeIds.length) return false;
|
||||
for (let i = 0; i < left.requiredNodeIds.length; i += 1) {
|
||||
if (left.requiredNodeIds[i] !== right.requiredNodeIds[i]) return false;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
export function emptyTargetRow(
|
||||
applicationId: string,
|
||||
nodeId: number,
|
||||
now: number,
|
||||
): GitOpsTargetCurrentRow {
|
||||
return {
|
||||
application_id: applicationId,
|
||||
node_id: nodeId,
|
||||
target_status: 'active',
|
||||
desired_generation_id: null,
|
||||
candidate_generation_id: null,
|
||||
applied_generation_id: null,
|
||||
deployed_generation_id: null,
|
||||
healthy_generation_id: null,
|
||||
lkg_generation_id: null,
|
||||
lkg_artifact_set_id: null,
|
||||
lkg_unavailable_at: null,
|
||||
lkg_unavailable_reason: null,
|
||||
expected_artifact_set_id: null,
|
||||
latest_artifact_set_id: null,
|
||||
observed_artifact_identity_json: null,
|
||||
intent_revision_id: null,
|
||||
rollout_candidate_id: null,
|
||||
rollout_generation_id: null,
|
||||
source_acceptance_ref: null,
|
||||
placement_approval_ref: null,
|
||||
rollout_authorization_ref: null,
|
||||
legacy_combined_approval_ref: null,
|
||||
legacy_applied_revision: null,
|
||||
connectivity: null,
|
||||
latest_stage: null,
|
||||
active_operation_id: null,
|
||||
active_operation_stage: null,
|
||||
active_operation_at: null,
|
||||
active_generation_id: null,
|
||||
active_intent_revision_id: null,
|
||||
active_rollout_candidate_id: null,
|
||||
failure_stage: null,
|
||||
failure_class: null,
|
||||
failure_at: null,
|
||||
recovery_ref: null,
|
||||
recovery_generation_id: null,
|
||||
recovery_phase: null,
|
||||
interruption_stage: null,
|
||||
interruption_at: null,
|
||||
interruption_operation_id: null,
|
||||
interruption_generation_id: null,
|
||||
interruption_intent_revision_id: null,
|
||||
interruption_rollout_candidate_id: null,
|
||||
pause_at: null,
|
||||
pause_reason: null,
|
||||
retry_at: null,
|
||||
suspended_at: null,
|
||||
partial_json: null,
|
||||
evidence_limitations_json: null,
|
||||
updated_at: now,
|
||||
};
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,818 @@
|
||||
import type { ArtifactEvidenceJson, ObservedArtifactIdentity } from './json';
|
||||
import type { RepoIdentity } from './repoIdentity';
|
||||
|
||||
export type GitOpsTargetMode = 'direct' | 'inline_blueprint' | 'blueprint';
|
||||
export type GitOpsLifecycleStatus = 'active' | 'creating' | 'detached' | 'deleted';
|
||||
export type ArtifactQualification =
|
||||
| 'unresolved'
|
||||
| 'exact'
|
||||
| 'qualified'
|
||||
| 'stale'
|
||||
| 'unavailable'
|
||||
| 'local_build_unverified';
|
||||
|
||||
export type GitOpsApprovalKind =
|
||||
| 'source_acceptance'
|
||||
| 'placement_approval'
|
||||
| 'rollout_authorization'
|
||||
| 'legacy_combined';
|
||||
|
||||
export type GitOpsApprovalAuthority = 'operator' | 'configured_policy' | 'legacy_combined';
|
||||
|
||||
export type ApplicationActiveStage = 'fetch_started' | 'apply_started' | 'deploy_started' | 'recovery_started';
|
||||
export type TargetActiveStage =
|
||||
| 'deploy_started'
|
||||
| 'blueprint_deploy_started'
|
||||
| 'blueprint_withdraw_started'
|
||||
| 'recovery_started';
|
||||
export type RecoveryPhase = 'capturing' | 'restoring' | 'compensating' | 'complete' | 'failed';
|
||||
export type ApplicationFailureStage = 'fetch' | 'validation' | 'apply' | 'create' | 'recovery';
|
||||
export type TargetFailureStage = 'deploy' | 'recovery' | 'blueprint_deploy' | 'blueprint_withdraw';
|
||||
export type Connectivity = 'unknown' | 'reachable' | 'unreachable' | 'stale';
|
||||
export type LkgUnavailableReason = 'generation_missing' | 'recovery_unretainable';
|
||||
|
||||
export type GitOpsApplicationRow = {
|
||||
id: string;
|
||||
lifecycle_key: string;
|
||||
lifecycle_status: GitOpsLifecycleStatus;
|
||||
target_mode: GitOpsTargetMode;
|
||||
stack_name: string | null;
|
||||
blueprint_id: number | null;
|
||||
configured_repo_url: string | null;
|
||||
repo_identity_json: string | null;
|
||||
configured_ref: string | null;
|
||||
compose_paths_json: string | null;
|
||||
context_dir: string | null;
|
||||
sync_env: number | null;
|
||||
env_path: string | null;
|
||||
materialization_fingerprint: string | null;
|
||||
desired_commit_sha: string | null;
|
||||
fetched_commit_sha: string | null;
|
||||
candidate_generation_id: string | null;
|
||||
accepted_generation_id: string | null;
|
||||
candidate_plan_blocked: number;
|
||||
review_required: number;
|
||||
artifact_set_id: string | null;
|
||||
latest_artifact_set_id: string | null;
|
||||
intent_revision_id: string | null;
|
||||
rollout_candidate_id: string | null;
|
||||
rollout_generation_id: string | null;
|
||||
source_acceptance_ref: string | null;
|
||||
placement_approval_ref: string | null;
|
||||
rollout_authorization_ref: string | null;
|
||||
legacy_combined_approval_ref: string | null;
|
||||
preflight_fingerprint: string | null;
|
||||
latest_operation_id: string | null;
|
||||
active_operation_id: string | null;
|
||||
active_operation_stage: ApplicationActiveStage | null;
|
||||
active_operation_at: number | null;
|
||||
active_generation_id: string | null;
|
||||
pause_at: number | null;
|
||||
pause_reason: string | null;
|
||||
partial_json: string | null;
|
||||
failure_stage: ApplicationFailureStage | null;
|
||||
failure_class: string | null;
|
||||
failure_at: number | null;
|
||||
retry_at: number | null;
|
||||
retry_count: number;
|
||||
suspended_at: number | null;
|
||||
recovery_ref: string | null;
|
||||
recovery_phase: RecoveryPhase | null;
|
||||
interruption_stage: ApplicationActiveStage | null;
|
||||
interruption_at: number | null;
|
||||
interruption_operation_id: string | null;
|
||||
interruption_generation_id: string | null;
|
||||
evidence_fresh_at: number | null;
|
||||
/** Write-time record of what this row could not prove. See json.ts. */
|
||||
evidence_limitations_json: string | null;
|
||||
created_at: number;
|
||||
updated_at: number;
|
||||
};
|
||||
|
||||
/**
|
||||
* How far a create-from-Git operation got before it stopped.
|
||||
*
|
||||
* The phase is what startup uses to decide between finishing the create and
|
||||
* tearing it down, so it is advanced only after the durable write it names has
|
||||
* actually committed.
|
||||
*/
|
||||
export type GitOpsCreatePhase =
|
||||
| 'pre_stack'
|
||||
| 'stack_created'
|
||||
| 'promoting'
|
||||
| 'manifest_committed'
|
||||
| 'pointers_committed';
|
||||
|
||||
export type GitOpsCreateCheckpointRow = {
|
||||
application_id: string;
|
||||
stack_name: string;
|
||||
phase: GitOpsCreatePhase;
|
||||
generation_id: string | null;
|
||||
operation_id: string;
|
||||
repo_url: string;
|
||||
branch: string;
|
||||
compose_path: string;
|
||||
compose_paths_json: string;
|
||||
context_dir: string | null;
|
||||
sync_env: number;
|
||||
env_path: string | null;
|
||||
auth_type: string;
|
||||
encrypted_token: string | null;
|
||||
auto_apply_on_webhook: number;
|
||||
auto_deploy_on_apply: number;
|
||||
commit_sha: string | null;
|
||||
applied_spec_json: string | null;
|
||||
/**
|
||||
* 1 only when this operation observed the managed root as absent and then
|
||||
* created it. Deleting the whole root during cleanup requires that proof.
|
||||
*/
|
||||
created_managed_root: number;
|
||||
created_at: number;
|
||||
updated_at: number;
|
||||
};
|
||||
|
||||
export type GitOpsGenerationRow = {
|
||||
id: string;
|
||||
application_id: string;
|
||||
commit_sha: string;
|
||||
repo_url: string;
|
||||
configured_ref: string;
|
||||
repo_identity_json: string;
|
||||
manifest_version: number;
|
||||
candidate_dir: string;
|
||||
applied_dir: string;
|
||||
expected_invocation_json: string;
|
||||
materialization_fingerprint: string;
|
||||
validation_ok: number;
|
||||
plan_blocked: number;
|
||||
change_plan_fingerprint: string | null;
|
||||
operation_id: string;
|
||||
trigger: string;
|
||||
actor: string | null;
|
||||
previous_generation_id: string | null;
|
||||
redacted_limitations_json: string;
|
||||
created_at: number;
|
||||
};
|
||||
|
||||
export type GitOpsArtifactSetRow = {
|
||||
id: string;
|
||||
generation_id: string;
|
||||
evidence_version: number;
|
||||
authoritative: number;
|
||||
qualification: ArtifactQualification;
|
||||
evidence_json: string;
|
||||
created_at: number;
|
||||
};
|
||||
|
||||
export type GitOpsIntentRevisionRow = {
|
||||
id: string;
|
||||
application_id: string;
|
||||
blueprint_id: number;
|
||||
compose_content_sha256: string;
|
||||
blueprint_revision: number;
|
||||
deploy_stack_name: string;
|
||||
selector_json: string;
|
||||
pinned_node_id: number | null;
|
||||
cordon_implications_json: string;
|
||||
rollout_strategy_json: string;
|
||||
runtime_drift_policy: string | null;
|
||||
stateful_policy_json: string | null;
|
||||
health_failure_rollback_policy_json: string | null;
|
||||
operation_id: string;
|
||||
actor: string | null;
|
||||
created_at: number;
|
||||
};
|
||||
|
||||
export type GitOpsRolloutCandidateRow = {
|
||||
id: string;
|
||||
application_id: string;
|
||||
intent_revision_id: string;
|
||||
compose_content_sha256: string;
|
||||
accepted_generation_id: string | null;
|
||||
artifact_set_id: string | null;
|
||||
required_targets_json: string;
|
||||
authoritative: number;
|
||||
provenance: 'intent_change' | 'roster_change' | 'legacy_inline';
|
||||
operation_id: string;
|
||||
created_at: number;
|
||||
};
|
||||
|
||||
export type GitOpsApprovalRow = {
|
||||
id: string;
|
||||
kind: GitOpsApprovalKind;
|
||||
authority: GitOpsApprovalAuthority;
|
||||
authoritative: number;
|
||||
application_id: string;
|
||||
generation_id: string | null;
|
||||
intent_revision_id: string | null;
|
||||
artifact_set_id: string | null;
|
||||
rollout_candidate_id: string | null;
|
||||
rollout_generation_id: string | null;
|
||||
source_acceptance_ref: string | null;
|
||||
placement_approval_ref: string | null;
|
||||
required_targets_json: string | null;
|
||||
preflight_fingerprint: string | null;
|
||||
fingerprint: string | null;
|
||||
blast_json: string | null;
|
||||
policy_provenance_json: string | null;
|
||||
actor: string | null;
|
||||
created_at: number;
|
||||
};
|
||||
|
||||
export type GitOpsTargetCurrentRow = {
|
||||
application_id: string;
|
||||
node_id: number;
|
||||
target_status: 'active' | 'tombstoned';
|
||||
desired_generation_id: string | null;
|
||||
candidate_generation_id: string | null;
|
||||
applied_generation_id: string | null;
|
||||
deployed_generation_id: string | null;
|
||||
healthy_generation_id: string | null;
|
||||
lkg_generation_id: string | null;
|
||||
lkg_artifact_set_id: string | null;
|
||||
lkg_unavailable_at: number | null;
|
||||
lkg_unavailable_reason: LkgUnavailableReason | null;
|
||||
expected_artifact_set_id: string | null;
|
||||
latest_artifact_set_id: string | null;
|
||||
observed_artifact_identity_json: string | null;
|
||||
intent_revision_id: string | null;
|
||||
rollout_candidate_id: string | null;
|
||||
rollout_generation_id: string | null;
|
||||
source_acceptance_ref: string | null;
|
||||
placement_approval_ref: string | null;
|
||||
rollout_authorization_ref: string | null;
|
||||
legacy_combined_approval_ref: string | null;
|
||||
legacy_applied_revision: number | null;
|
||||
connectivity: Connectivity | null;
|
||||
latest_stage: string | null;
|
||||
active_operation_id: string | null;
|
||||
active_operation_stage: TargetActiveStage | null;
|
||||
active_operation_at: number | null;
|
||||
active_generation_id: string | null;
|
||||
active_intent_revision_id: string | null;
|
||||
active_rollout_candidate_id: string | null;
|
||||
failure_stage: TargetFailureStage | null;
|
||||
failure_class: string | null;
|
||||
failure_at: number | null;
|
||||
recovery_ref: string | null;
|
||||
recovery_generation_id: string | null;
|
||||
recovery_phase: RecoveryPhase | null;
|
||||
interruption_stage: TargetActiveStage | null;
|
||||
interruption_at: number | null;
|
||||
interruption_operation_id: string | null;
|
||||
interruption_generation_id: string | null;
|
||||
interruption_intent_revision_id: string | null;
|
||||
interruption_rollout_candidate_id: string | null;
|
||||
pause_at: number | null;
|
||||
pause_reason: string | null;
|
||||
retry_at: number | null;
|
||||
suspended_at: number | null;
|
||||
partial_json: string | null;
|
||||
/** Write-time record of what this target could not prove. See json.ts. */
|
||||
evidence_limitations_json: string | null;
|
||||
updated_at: number;
|
||||
};
|
||||
|
||||
export type GitOpsHistoryRow = {
|
||||
id: string;
|
||||
created_at: number;
|
||||
application_id: string;
|
||||
target_mode: GitOpsTargetMode;
|
||||
lifecycle_key: string;
|
||||
stack_name: string | null;
|
||||
blueprint_id: number | null;
|
||||
node_id: number | null;
|
||||
dedupe_target: string;
|
||||
repo_url: string | null;
|
||||
configured_ref: string | null;
|
||||
repo_identity_json: string | null;
|
||||
commit_sha: string | null;
|
||||
generation_id: string | null;
|
||||
artifact_set_id: string | null;
|
||||
intent_revision_id: string | null;
|
||||
rollout_candidate_id: string | null;
|
||||
rollout_generation_id: string | null;
|
||||
source_acceptance_ref: string | null;
|
||||
placement_approval_ref: string | null;
|
||||
rollout_authorization_ref: string | null;
|
||||
legacy_combined_approval_ref: string | null;
|
||||
operation_id: string;
|
||||
stage: string;
|
||||
outcome: 'committed' | 'failed' | 'skipped' | 'superseded' | 'recovered' | 'unknown';
|
||||
trigger: string;
|
||||
actor: string | null;
|
||||
before_json: string;
|
||||
after_json: string;
|
||||
required_targets_json: string | null;
|
||||
validation_json: string | null;
|
||||
per_target_results_json: string | null;
|
||||
health_run_id: string | null;
|
||||
health_snapshot_json: string | null;
|
||||
invocation_observed_json: string | null;
|
||||
recovery_ref: string | null;
|
||||
redacted_reason_class: string | null;
|
||||
};
|
||||
|
||||
export type FutureRolloutAuthorizationBinding = {
|
||||
readonly rolloutCandidateId: string;
|
||||
readonly acceptedGenerationId: string;
|
||||
readonly artifactSetId: string;
|
||||
readonly intentRevisionId: string;
|
||||
readonly requiredNodeIds: readonly number[];
|
||||
readonly sourceAcceptanceRef: string;
|
||||
readonly placementApprovalRef: string;
|
||||
readonly preflightFingerprint: string;
|
||||
};
|
||||
|
||||
export type ResolveApprovalExpected =
|
||||
| { kind: 'source_acceptance'; applicationId: string; generationId: string }
|
||||
| { kind: 'placement_approval'; applicationId: string; intentRevisionId: string; requiredNodeIds: readonly number[] }
|
||||
| { kind: 'rollout_authorization'; applicationId: string; binding: FutureRolloutAuthorizationBinding }
|
||||
| { kind: 'legacy_combined'; applicationId: string };
|
||||
|
||||
export type FutureGitOpsEvidence = {
|
||||
readonly applicationId: string;
|
||||
readonly source: Readonly<{ kind: 'source_superseded'; supersededGenerationId: string }> | null;
|
||||
readonly placement:
|
||||
| Readonly<{ kind: 'source_acceptance_pending'; candidateGenerationId: string }>
|
||||
| Readonly<{ kind: 'authorization_pending'; binding: FutureRolloutAuthorizationBinding }>
|
||||
| Readonly<{ kind: 'authorization_stale'; rolloutAuthorizationRef: string; bound: FutureRolloutAuthorizationBinding }>
|
||||
| Readonly<{ kind: 'preflight_blocked'; reason: string; binding: FutureRolloutAuthorizationBinding }>
|
||||
| null;
|
||||
readonly rollout: Readonly<{
|
||||
kind:
|
||||
| 'queued'
|
||||
| 'canary'
|
||||
| 'batch'
|
||||
| 'superseded'
|
||||
| 'fully_deployed_health_pending'
|
||||
| 'configuration_converged_artifact_qualified'
|
||||
| 'exactly_converged_healthy';
|
||||
rolloutGenerationId: string;
|
||||
}> | null;
|
||||
readonly targetRuntime: ReadonlyArray<Readonly<{
|
||||
nodeId: number;
|
||||
kind: 'rollout_artifact_drift';
|
||||
rolloutGenerationId: string;
|
||||
expectedIdentity: string;
|
||||
observedIdentity: string;
|
||||
freshnessAt: number;
|
||||
}>>;
|
||||
};
|
||||
|
||||
export type GitOpsApprovalRefs = {
|
||||
sourceAcceptanceRef: string | null;
|
||||
placementApprovalRef: string | null;
|
||||
rolloutAuthorizationRef: string | null;
|
||||
legacyCombinedApprovalRef: string | null;
|
||||
};
|
||||
|
||||
export type EvidenceSource = 'current' | 'future' | 'current_or_future' | 'not_applicable';
|
||||
|
||||
export type SourceIdentityFields = {
|
||||
configuredRepoUrl: string;
|
||||
repoIdentity: RepoIdentity;
|
||||
configuredRef: string;
|
||||
desiredCommitSha: string | null;
|
||||
fetchedCommitSha: string | null;
|
||||
candidateGenerationId: string | null;
|
||||
acceptedGenerationId: string | null;
|
||||
};
|
||||
|
||||
export type SourceFacet =
|
||||
| { status: 'not_applicable' }
|
||||
| (SourceIdentityFields & {
|
||||
status:
|
||||
| 'never_reconciled'
|
||||
| 'checking_fetching'
|
||||
| 'application_generation_accepted'
|
||||
| 'candidate_ready'
|
||||
| 'source_review_pending'
|
||||
| 'source_conflict_blocker'
|
||||
| 'source_reconcile_required';
|
||||
})
|
||||
| (SourceIdentityFields & { status: 'source_superseded'; supersededGenerationId: string })
|
||||
| (SourceIdentityFields & { status: 'applying'; activeOperationId: string; activeGenerationId: string })
|
||||
| (SourceIdentityFields & { status: 'source_retry_scheduled'; retryAt: number; retryCount: number })
|
||||
| (SourceIdentityFields & { status: 'source_suspended'; suspendedAt: number })
|
||||
| (SourceIdentityFields & {
|
||||
status: 'source_failed';
|
||||
failureStage: 'fetch' | 'validation' | 'apply' | 'create';
|
||||
failureClass: string;
|
||||
failureAt: number;
|
||||
retryAt: number | null;
|
||||
retryCount: number;
|
||||
})
|
||||
| (SourceIdentityFields & {
|
||||
status: 'source_unknown';
|
||||
interruptedStage: 'fetch_started' | 'apply_started';
|
||||
interruptedAt: number;
|
||||
interruptedOperationId: string | null;
|
||||
interruptedGenerationId: string | null;
|
||||
})
|
||||
| (SourceIdentityFields & {
|
||||
status: 'recovery_required';
|
||||
recoveryRef: string | null;
|
||||
recoveryGenerationId: string | null;
|
||||
})
|
||||
| (SourceIdentityFields & {
|
||||
status: 'recovery_failed';
|
||||
recoveryRef: string | null;
|
||||
recoveryGenerationId: string | null;
|
||||
failureClass: string;
|
||||
failureAt: number;
|
||||
})
|
||||
| (SourceIdentityFields & { status: 'not_live'; lifecycleStatus: 'detached' | 'deleted' });
|
||||
|
||||
export type ArtifactExpectedIdentity = {
|
||||
artifactSetId: string;
|
||||
evidenceVersion: number;
|
||||
qualification: ArtifactQualification;
|
||||
identity: string | null;
|
||||
};
|
||||
|
||||
export type ArtifactLatestEvidence = {
|
||||
artifactSetId: string;
|
||||
evidenceVersion: number;
|
||||
qualification: ArtifactQualification;
|
||||
identity: string | null;
|
||||
};
|
||||
|
||||
export type ArtifactFacet =
|
||||
| { status: 'not_applicable' }
|
||||
| {
|
||||
status: 'artifact_unresolved';
|
||||
generationId: string;
|
||||
expected: ArtifactExpectedIdentity | null;
|
||||
latestEvidence: null;
|
||||
limitation: 'artifact_pointer_missing';
|
||||
}
|
||||
| {
|
||||
status:
|
||||
| 'artifact_unresolved'
|
||||
| 'artifact_resolution_pending'
|
||||
| 'artifact_exact'
|
||||
| 'artifact_qualified'
|
||||
| 'artifact_stale'
|
||||
| 'artifact_unavailable'
|
||||
| 'artifact_local_build_unverified'
|
||||
| 'artifact_identity_changed';
|
||||
artifactSetId: string;
|
||||
generationId: string;
|
||||
evidenceVersion: number;
|
||||
qualification: ArtifactQualification;
|
||||
freshnessAt: number;
|
||||
expected: ArtifactExpectedIdentity | null;
|
||||
latestEvidence: ArtifactLatestEvidence;
|
||||
};
|
||||
|
||||
export type PlacementFacet =
|
||||
| { status: 'not_applicable' }
|
||||
| { status: 'unbound_direct' }
|
||||
| { status: 'unknown'; limitation: 'missing_intent' }
|
||||
| { status: 'source_acceptance_pending'; sourceAcceptanceRef: string | null; candidateGenerationId: string }
|
||||
| { status: 'placement_review_pending' }
|
||||
| { status: 'rollout_authorization_pending'; rolloutAuthorizationRef: null; binding: FutureRolloutAuthorizationBinding }
|
||||
| { status: 'rollout_authorization_stale'; rolloutAuthorizationRef: string; bound: FutureRolloutAuthorizationBinding }
|
||||
| { status: 'stateful_confirmation_required' }
|
||||
| { status: 'preflight_blocked'; reason: string; binding: FutureRolloutAuthorizationBinding }
|
||||
| { status: 'blueprint_bound'; completion: 'unknown' };
|
||||
|
||||
export type RolloutFacet =
|
||||
| { status: 'not_applicable' }
|
||||
| { status: 'rollout_not_executable'; rolloutCandidateId: string }
|
||||
| { status: 'rollout_queued'; rolloutGenerationId: string }
|
||||
| { status: 'canary_in_progress'; rolloutGenerationId: string }
|
||||
| { status: 'batch_in_progress'; rolloutGenerationId: string }
|
||||
| { status: 'rollout_paused'; pauseAt: number; pauseReason: string | null }
|
||||
| { status: 'partially_rolled_out'; partial: unknown }
|
||||
| { status: 'fully_deployed_health_pending'; rolloutGenerationId: string }
|
||||
| { status: 'configuration_converged_artifact_qualified'; rolloutGenerationId: string }
|
||||
| { status: 'exactly_converged_healthy'; rolloutGenerationId: string }
|
||||
| { status: 'rollout_superseded'; rolloutGenerationId: string }
|
||||
| { status: 'target_stale' }
|
||||
| { status: 'target_unreachable' }
|
||||
| { status: 'rollback_in_progress'; recoveryRef: string; recoveryGenerationId: string | null }
|
||||
| { status: 'rollback_partial_failed'; recoveryRef: string; recoveryGenerationId: string | null; failureClass: string; failureAt: number }
|
||||
| { status: 'recovery_required' }
|
||||
| { status: 'completion_unknown' };
|
||||
|
||||
export type RuntimeFacet =
|
||||
| {
|
||||
status:
|
||||
| 'tombstoned'
|
||||
| 'recovery_required'
|
||||
| 'deploying'
|
||||
| 'withdrawing'
|
||||
| 'failed_previous_workload_intact'
|
||||
| 'failed_after_mutation'
|
||||
| 'disk_invocation_drift'
|
||||
| 'rollout_artifact_drift'
|
||||
| 'runtime_artifact_drift'
|
||||
| 'artifact_verification_pending'
|
||||
| 'never_applied'
|
||||
| 'applied_not_deployed'
|
||||
| 'acknowledged_completion_unknown'
|
||||
| 'stale_acknowledgement'
|
||||
| 'pending_state_review'
|
||||
| 'evict_blocked'
|
||||
| 'drifted'
|
||||
| 'correcting'
|
||||
| 'fully_deployed_health_pending'
|
||||
| 'health_checking'
|
||||
| 'synced_and_healthy'
|
||||
| 'health_drift'
|
||||
| 'partially_rolled_out'
|
||||
| 'retry_scheduled';
|
||||
}
|
||||
| { status: 'paused'; pauseAt: number; pauseReason: string | null }
|
||||
| {
|
||||
status: 'recovery_failed';
|
||||
recoveryRef: string | null;
|
||||
recoveryGenerationId: string | null;
|
||||
failureClass: string;
|
||||
failureAt: number;
|
||||
}
|
||||
| {
|
||||
status: 'completion_unknown';
|
||||
interruptedStage: 'deploy_started' | 'blueprint_deploy_started' | 'blueprint_withdraw_started';
|
||||
interruptedAt: number;
|
||||
interruptedOperationId: string | null;
|
||||
interruptedGenerationId: string | null;
|
||||
interruptedIntentRevisionId: string | null;
|
||||
interruptedRolloutCandidateId: string | null;
|
||||
};
|
||||
|
||||
export type LkgFacet =
|
||||
| { status: 'none' }
|
||||
| { status: 'available'; generationId: string; artifactSetId: string | null }
|
||||
| { status: 'unavailable' }
|
||||
| { status: 'qualified'; generationId: string; artifactSetId: string };
|
||||
|
||||
export type HealthFacet =
|
||||
| { status: 'not_applicable' | 'unbound' }
|
||||
| { status: 'pending'; runId: string | null }
|
||||
| { status: 'checking'; runId: string; deployedGenerationId: string | null }
|
||||
| { status: 'passed'; runId: string; deployedGenerationId: string }
|
||||
| { status: 'failed'; runId: string; deployedGenerationId: string | null }
|
||||
| { status: 'unknown'; runId: string | null; limitation: 'health_unknown' };
|
||||
|
||||
export type FacetEvidenceSource = {
|
||||
source: Record<SourceFacet['status'], EvidenceSource>;
|
||||
artifact: Record<ArtifactFacet['status'], EvidenceSource>;
|
||||
placement: Record<PlacementFacet['status'], EvidenceSource>;
|
||||
rollout: Record<RolloutFacet['status'], EvidenceSource>;
|
||||
runtime: Record<RuntimeFacet['status'], EvidenceSource>;
|
||||
lkg: Record<LkgFacet['status'], EvidenceSource>;
|
||||
health: Record<HealthFacet['status'], EvidenceSource>;
|
||||
};
|
||||
|
||||
/**
|
||||
* What kind of evidence can produce each facet status.
|
||||
*
|
||||
* `current` means it derives from persisted rows. `future` means it can only
|
||||
* come from a rollout-evidence envelope that no producer emits yet, so the
|
||||
* derivers in this module must never return it. `current_or_future` marks the
|
||||
* statuses both paths can reach.
|
||||
*
|
||||
* The `Record` type makes coverage a compile error in both directions: a new
|
||||
* facet status without an entry here fails to compile, and so does an entry
|
||||
* whose status no longer exists.
|
||||
*/
|
||||
export const FACET_EVIDENCE_SOURCE: FacetEvidenceSource = {
|
||||
source: {
|
||||
not_applicable: 'not_applicable',
|
||||
never_reconciled: 'current',
|
||||
checking_fetching: 'current',
|
||||
applying: 'current',
|
||||
candidate_ready: 'current',
|
||||
source_review_pending: 'current',
|
||||
source_conflict_blocker: 'current',
|
||||
source_reconcile_required: 'current',
|
||||
application_generation_accepted: 'current',
|
||||
source_superseded: 'future',
|
||||
source_retry_scheduled: 'current',
|
||||
source_suspended: 'current',
|
||||
source_failed: 'current',
|
||||
source_unknown: 'current',
|
||||
recovery_required: 'current',
|
||||
recovery_failed: 'current',
|
||||
not_live: 'current',
|
||||
},
|
||||
artifact: {
|
||||
not_applicable: 'not_applicable',
|
||||
artifact_unresolved: 'current',
|
||||
artifact_resolution_pending: 'current',
|
||||
artifact_exact: 'current',
|
||||
artifact_qualified: 'current',
|
||||
artifact_stale: 'current',
|
||||
artifact_unavailable: 'current',
|
||||
artifact_local_build_unverified: 'current',
|
||||
artifact_identity_changed: 'current',
|
||||
},
|
||||
placement: {
|
||||
not_applicable: 'not_applicable',
|
||||
unbound_direct: 'current',
|
||||
unknown: 'current',
|
||||
placement_review_pending: 'current',
|
||||
stateful_confirmation_required: 'current',
|
||||
blueprint_bound: 'current',
|
||||
source_acceptance_pending: 'future',
|
||||
rollout_authorization_pending: 'future',
|
||||
rollout_authorization_stale: 'future',
|
||||
preflight_blocked: 'future',
|
||||
},
|
||||
rollout: {
|
||||
not_applicable: 'not_applicable',
|
||||
rollout_not_executable: 'current',
|
||||
rollout_paused: 'current',
|
||||
partially_rolled_out: 'current',
|
||||
target_stale: 'current',
|
||||
target_unreachable: 'current',
|
||||
rollback_in_progress: 'current',
|
||||
rollback_partial_failed: 'current',
|
||||
recovery_required: 'current',
|
||||
completion_unknown: 'current_or_future',
|
||||
rollout_queued: 'future',
|
||||
canary_in_progress: 'future',
|
||||
batch_in_progress: 'future',
|
||||
fully_deployed_health_pending: 'future',
|
||||
configuration_converged_artifact_qualified: 'future',
|
||||
exactly_converged_healthy: 'future',
|
||||
rollout_superseded: 'future',
|
||||
},
|
||||
runtime: {
|
||||
tombstoned: 'current',
|
||||
recovery_required: 'current',
|
||||
deploying: 'current',
|
||||
withdrawing: 'current',
|
||||
failed_previous_workload_intact: 'current',
|
||||
failed_after_mutation: 'current',
|
||||
disk_invocation_drift: 'current',
|
||||
runtime_artifact_drift: 'current',
|
||||
artifact_verification_pending: 'current',
|
||||
never_applied: 'current',
|
||||
applied_not_deployed: 'current',
|
||||
acknowledged_completion_unknown: 'current',
|
||||
stale_acknowledgement: 'current',
|
||||
pending_state_review: 'current',
|
||||
evict_blocked: 'current',
|
||||
drifted: 'current',
|
||||
correcting: 'current',
|
||||
fully_deployed_health_pending: 'current',
|
||||
health_checking: 'current',
|
||||
synced_and_healthy: 'current',
|
||||
health_drift: 'current',
|
||||
partially_rolled_out: 'current',
|
||||
retry_scheduled: 'current',
|
||||
paused: 'current',
|
||||
recovery_failed: 'current',
|
||||
completion_unknown: 'current',
|
||||
rollout_artifact_drift: 'future',
|
||||
},
|
||||
lkg: {
|
||||
none: 'current',
|
||||
available: 'current',
|
||||
unavailable: 'current',
|
||||
qualified: 'current',
|
||||
},
|
||||
health: {
|
||||
not_applicable: 'not_applicable',
|
||||
unbound: 'current',
|
||||
pending: 'current',
|
||||
checking: 'current',
|
||||
passed: 'current',
|
||||
failed: 'current',
|
||||
unknown: 'current',
|
||||
},
|
||||
};
|
||||
|
||||
export type GitOpsFacets = {
|
||||
source: SourceFacet;
|
||||
artifact: ArtifactFacet;
|
||||
placement: PlacementFacet;
|
||||
rollout: RolloutFacet;
|
||||
};
|
||||
|
||||
export type AuthoredInvocationIdentity = {
|
||||
composeFileOrder: string[];
|
||||
projectName: string | null;
|
||||
projectDirectory: string | null;
|
||||
envFileOrder: string[];
|
||||
};
|
||||
|
||||
export type GitOpsIdentityRef =
|
||||
| { kind: 'none' }
|
||||
| { kind: 'unknown' }
|
||||
| { kind: 'commit'; sha: string; repoUrl: string; ref: string }
|
||||
| { kind: 'generation'; id: string }
|
||||
| { kind: 'artifact_set'; id: string; qualification: ArtifactQualification; evidenceVersion: number }
|
||||
| { kind: 'runtime_artifact'; identity: string; observedAt: number | null }
|
||||
| { kind: 'intent'; id: string; composeContentSha256: string }
|
||||
| { kind: 'rollout_candidate'; id: string }
|
||||
| { kind: 'rollout_generation'; id: string }
|
||||
| { kind: 'invocation'; authored: AuthoredInvocationIdentity }
|
||||
| { kind: 'health_run'; runId: string; deployedGenerationId: string | null };
|
||||
|
||||
/**
|
||||
* Cross-instance evidence a history entry carries so any reader can decide who
|
||||
* may see it. Only the owning instance can answer any of these questions, so
|
||||
* they all travel with the row rather than being inferred by a reader that
|
||||
* holds none of that instance's state.
|
||||
*/
|
||||
export type GitOpsHistoryEvidenceFields = {
|
||||
stackName: string | null;
|
||||
applicationLifecycleStatus: GitOpsLifecycleStatus | null;
|
||||
stackResourcePresent: boolean;
|
||||
};
|
||||
|
||||
export type GitOpsLimitation = { code: string; message: string; evidence: unknown };
|
||||
export type GitOpsAvailableAction = 'fetch' | 'apply' | 'dismiss' | 'deploy' | 'approve_legacy' | 'none';
|
||||
|
||||
export type ConfiguredPolicy =
|
||||
| { kind: 'git_source'; autoApplyOnWebhook: boolean; autoDeployOnApply: boolean }
|
||||
| { kind: 'blueprint_drift'; driftMode: 'observe' | 'suggest' | 'enforce' }
|
||||
| null;
|
||||
|
||||
export type GitOpsDriftItem = {
|
||||
class: 'source' | 'managed_project' | 'invocation' | 'placement' | 'rollout' | 'runtime' | 'health';
|
||||
expected: GitOpsIdentityRef;
|
||||
observed: GitOpsIdentityRef;
|
||||
freshnessAt: number | null;
|
||||
owner: string;
|
||||
reason: string;
|
||||
configuredPolicy: ConfiguredPolicy;
|
||||
affectedTargets: Array<{ nodeId: number | null; stackName: string | null }>;
|
||||
action: GitOpsAvailableAction;
|
||||
};
|
||||
|
||||
export type GitOpsTargetProjection = {
|
||||
nodeId: number;
|
||||
stackName: string | null;
|
||||
desiredGenerationId: string | null;
|
||||
candidateGenerationId: string | null;
|
||||
appliedGenerationId: string | null;
|
||||
deployedGenerationId: string | null;
|
||||
healthyGenerationId: string | null;
|
||||
lkgGenerationId: string | null;
|
||||
lkgArtifactSetId: string | null;
|
||||
lkgUnavailableAt: number | null;
|
||||
lkgUnavailableReason: LkgUnavailableReason | null;
|
||||
expectedArtifactSetId: string | null;
|
||||
latestArtifactSetId: string | null;
|
||||
artifact: ArtifactFacet;
|
||||
observedArtifactIdentity: ObservedArtifactIdentity;
|
||||
intentRevisionId: string | null;
|
||||
rolloutCandidateId: string | null;
|
||||
rolloutGenerationId: string | null;
|
||||
approvals: GitOpsApprovalRefs;
|
||||
connectivity: Connectivity;
|
||||
legacyAppliedRevision: number | null;
|
||||
runtime: RuntimeFacet;
|
||||
health: HealthFacet;
|
||||
lkg: LkgFacet;
|
||||
tombstoned: boolean;
|
||||
};
|
||||
|
||||
export type GitOpsRevisionProjection =
|
||||
| {
|
||||
schemaVersion: 1;
|
||||
targetMode: 'not_applicable';
|
||||
applicationId: null;
|
||||
facets: null;
|
||||
targets: [];
|
||||
drift: [];
|
||||
/**
|
||||
* Why there is nothing to project, when the answer is not simply "no
|
||||
* application".
|
||||
*
|
||||
* Empty for the ordinary case: a stack or Blueprint the model has never
|
||||
* been asked about. Non-empty when the projection could not reach an
|
||||
* application it had reason to believe exists, which is a different fact
|
||||
* and must not read as the ordinary one.
|
||||
*
|
||||
* `readonly` because the shared frozen NOT_APPLICABLE_REVISION is this
|
||||
* variant: a caller pushing onto it would corrupt every later response,
|
||||
* and this keeps that a compile error rather than a runtime throw.
|
||||
*/
|
||||
limitations: readonly GitOpsLimitation[];
|
||||
availableActions: [];
|
||||
approvals: null;
|
||||
}
|
||||
| {
|
||||
schemaVersion: 1;
|
||||
targetMode: GitOpsTargetMode;
|
||||
applicationId: string;
|
||||
lifecycleStatus: GitOpsLifecycleStatus;
|
||||
stackName: string | null;
|
||||
blueprintId: number | null;
|
||||
rolloutGenerationId: string | null;
|
||||
approvals: GitOpsApprovalRefs;
|
||||
facets: GitOpsFacets;
|
||||
targets: GitOpsTargetProjection[];
|
||||
drift: GitOpsDriftItem[];
|
||||
limitations: GitOpsLimitation[];
|
||||
availableActions: GitOpsAvailableAction[];
|
||||
};
|
||||
|
||||
export type { ArtifactEvidenceJson, ObservedArtifactIdentity, RepoIdentity };
|
||||
@@ -94,7 +94,7 @@ export interface HealthGateReport {
|
||||
stack: string;
|
||||
id: string | null;
|
||||
status: HealthGateStatus | 'never-run';
|
||||
trigger: 'update' | 'deploy' | 'service_update' | 'service_restore' | null;
|
||||
trigger: 'update' | 'deploy' | 'service_update' | 'service_restore' | 'recovery' | null;
|
||||
reason: string | null;
|
||||
windowSeconds: number | null;
|
||||
startedAt: number | null;
|
||||
|
||||
Reference in New Issue
Block a user