mirror of
https://github.com/Studio-Saelix/sencho.git
synced 2026-08-28 19:27:41 +00:00
Anso
38ee4527b1
* feat(git): add GitOps revision store and Direct apply transitions
Adds the canonical GitOps revision state model: schema, store, history,
approval resolution, Direct apply transitions, and a first-cut read
projection. Also widens recovery capture with generation, artifact, and
source-acceptance bindings, rejects credentials and query strings in new
Git repository URLs, and keeps Blueprint and node-label routes on the hub.
* feat(git): add create-from-Git activation and teardown transitions
Adds the durable half of create-from-Git: the single transaction that persists
the application, resolved commit, generation, checkpoint, and candidate
together, and the teardown that tombstones a create which never reached its
success boundary. Adds the staging marker and operation-owned cleanup rules
that decide what a crashed create is allowed to delete.
* feat(git): make create-from-Git crash-safe
Wires the create path through the GitOps state model: the staging marker is
written before the candidate is built, the activation transaction persists the
application, generation, and checkpoint together, and the source row plus
accepted pointers now commit as one success boundary. A create that fails
before that boundary removes only what it staged, then tombstones itself.
The managed-area sweep no longer deletes a directory it cannot prove it owns,
counts in-flight creates as claimed, and is awaited at startup so an
interrupted create is settled before any mutation service runs.
* feat(git): settle interrupted create-from-Git at startup
Adds boot-time recovery for creates a previous process left in flight. A create
whose project was already committed to disk is finished, including its source
row and accepted pointers. Anything earlier has its files removed first and is
then tombstoned, so the stack name becomes usable again instead of staying
locked by a half-created application. A source row that outlived its
application is always preserved.
* fix(git): close create-from-Git ownership and reporting gaps
Removes two ways a failed create could destroy files it did not own: boot
recovery no longer deletes a stack directory the create never recorded making,
and the managed-root teardown is gated on having created that root. Cleanup
now rejects a path that is not a generation directory, so a malformed row
cannot widen the blast radius.
A create that fails during materialization now clears its staging marker, which
previously stayed behind and made the stack name uncreatable until restart.
Generation paths no longer resolve through a module cycle that left them
pointing at a directory that never existed.
Boot recovery restores the deploy spec and manifest cache a finished create
needs, settles each application independently so one bad row cannot strand the
others, and distinguishes a missing directory from one it could not read. A
create that succeeds now clears its checkpoint, and a failure after the success
boundary says the stack was created rather than reporting a bare error.
* fix(git): reap managed areas that nothing has ever claimed
The boot sweep now distinguishes a missing staging marker from an unreadable
one. A missing marker means no stack, no create, and no marker claims the
directory, which is the ordinary orphan a crashed stack deletion leaves behind,
so it is removed as it always was. An unreadable marker is evidence of a claim
that cannot be verified, so the area is preserved.
Without this an orphaned managed area survived every boot forever, since a
completed create deletes its marker and nothing else would ever claim it.
* feat(git): complete the Direct source and target transitions
Adds the remaining Direct events the state model owed: an invalid fetch that
advances the resolved commit without minting a candidate, a blocked candidate
that is visible but cannot apply, dismissal that leaves the running workload
alone, material configuration change that invalidates a staged candidate while
keeping accepted and applied pointers, the two deploy failure classes, and
application and target tombstones.
Widens the application update to every mutable column. It previously wrote
about half of them, so a transition assigning a pause, suspend, recovery, or
intent field would have type-checked, appeared in the history snapshot, and
been silently dropped at commit.
* feat(git): record Direct Git operations in the revision state
Wires fetch, apply, and detach through the state model. A pull records the
fetch and, when it produces a candidate, the generation behind it, marking a
blocked change plan as a blocker that cannot apply. An apply records the
acceptance that binds the generation to the workload, and closes its operation
on failure so a throw cannot leave the source reporting work in progress.
Detach tombstones the application and its target in the same transaction that
removes the source row.
Every producer is a no-op for a stack with no live application, so installs
whose Git stacks predate this model are untouched until migration runs.
* test(git): cover the Direct Git producers end to end
Adds a harness that stubs only the git transport and rollback capture, so
fetch, candidate materialization, change-plan classification, apply, and detach
all run for real and are asserted against the resulting revision state.
Covers the seam the transition-level tests could not: that a fetch advances the
resolved commit without moving the accepted generation, that the apply binds
the exact candidate the fetch recorded and its acceptance proves that
generation and no other, that detach tombstones while keeping configured
identity as a frozen fact, that a failed fetch closes its operation and leaves
every pointer alone, and that a stack with no application produces no writes.
* feat(git): bind the deployed generation from the Compose adapter
Makes ComposeService the sole producer of deploy events. Every deploy path
funnels through deployStack, so recording it there keeps one start and one
terminal row per mutation instead of each caller reporting its own.
A successful deploy binds the applied generation, a failure records the class
and leaves the deployed pointer alone. The class is conservative: once the
compose command is handed off we cannot prove the workload was untouched, and
claiming it was intact is the more dangerous error. A stack with no live
application, or one with nothing applied, has no deploy identity to record and
is skipped entirely.
* feat(git): promote healthy and last-known-good from health verdicts
A stack health run now records which generation it observed, and its verdict
drives the revision state. Promotion is narrow by design: healthy and
last-known-good move only when the run passed, watched the whole stack, and the
generation it watched is still deployed. A stale or service-scoped verdict
records history and moves nothing.
Last-known-good keeps the artifact expectation only when that expectation
belongs to the promoted generation. Otherwise the generation is still good and
its executable identity simply is not proven, so the pointer is left null
rather than borrowed from another generation.
beginStack now takes the deployed generation explicitly rather than reading
current state, because a verdict is only meaningful for the generation the run
actually observed. Update-path callers pass null until updateStack reports what
it deployed.
* feat(git): bind the deployed generation on the update path
Gives updateStack the same treatment deployStack has: it opens a deploy
operation at the point Compose is handed the mutation, closes it on both
outcomes, and reports the generation it bound. The orchestrator carries that
binding through, and all five update call sites now pass it to the health gate
instead of null, so a passed health run after an update promotes healthy and
last-known-good exactly as it does after a deploy.
The operation is opened at the compose call rather than at entry, so an update
that fails during capture or classification records nothing: it never touched
the workload, and reporting a deploy failure for it would make the projection
claim something that did not happen.
* feat(git): track linked stacks and material config changes
Linking an existing stack to a Git source now brings it into the revision
state: the application starts live with nothing desired or accepted, so the
projection asks for a fetch rather than claiming a state it has not observed.
Editing a source's material configuration invalidates its staged candidate in
the same transaction that writes the row and clears the pending pull. A
candidate built from a different repository, ref, or file set can no longer be
applied, and clearing pending without invalidating it would leave the model
offering an apply the operator cannot produce. Credential-only and policy-only
edits change nothing material and leave the candidate alone.
* feat(git): retire the application when a stack is deleted
Stack deletion now tombstones its GitOps application and targets in the same
transaction that commits the deletion. A deleted stack that kept a live
application would go on claiming the stack name and block re-creating it
through the unique live-application index.
The tombstone is driven from the deletion service rather than from inside
DatabaseService, so the store keeps its transitions and their history in one
place and the two modules do not form an import cycle.
* feat(git): retire node targets when a node is removed
Deleting a node now tombstones the GitOps targets that lived on it, in the same
transaction as the delete and while those rows still exist. Otherwise a
part-way failure would leave targets pointing at a node that is gone.
Applications stay live: a Direct application still describes a real stack, and
a Blueprint one may have targets on other nodes. Both the local and remote node
paths go through the same helper, so neither can skip the retirement.
* feat(git): add proof-bound recovery transitions
A restore moves a target back to an older generation, the one case where a
target and its application legitimately disagree about what is current. These
transitions decide what may move with it.
Pointers move only when the restore is provable: the recovery point named a
generation, that generation still exists, and it belongs to this application.
The artifact expectation comes from what the recovery point captured, never
from what the application expects now, and the acceptance is kept only if it
still proves the restored generation rather than borrowing one that authorized
a newer one. A last-known-good survives unless the generation behind it is gone
or turns out to belong elsewhere, and then the reason is recorded so the
projection can say unavailable rather than none.
An unproven restore is still recorded as a real operational recovery and moves
nothing, because there is no evidence to move pointers to.
* fix(git): close GitOps operations that cannot be recorded
Recording a transition still never fails the operation it describes, but a
rejected terminal event no longer leaves that operation open. A start that
never terminates reported work in flight for ever and offered no actions, and
because a new fetch refuses to open a second operation, one rejection silently
stopped the model tracking that stack until a restart.
Startup now reclassifies operations the previous process left open, which the
transition already supported but nothing called. An interrupted restore is
closed there too, since only the terminal recovery events clear it.
The pull records its fetch, generation, and candidate as one transaction, and
the apply verifies the candidate was built from the commit being applied.
Separately those two allowed an apply to accept a generation whose files were
never on disk while the projection reported the older commit as current.
Also: the candidate now records its own invocation rather than the one it
replaces, changing material configuration is refused while an operation is in
flight, a tombstoned target cannot be repopulated by a late health verdict, a
deploy handle is not returned when its start was not recorded, the mutation
handoff is marked only once Compose is genuinely about to receive it, and a
rejected tombstone fails one deletion rather than aborting the boot sweep.
* feat(git): record why unprovable evidence was dropped
The projection carried a limitations array that only read-time derivation ever
wrote to. When a transition cleared a pointer it could not prove, the result
was indistinguishable from that pointer never having existed, and in two cases
it made the target read healthier: a dropped artifact expectation silently
disables the runtime drift check, and an unproven restore left every pointer
agreeing with itself so the target reported as synced and healthy.
Adds a persisted, fail-closed evidence record on the application and target
rows. The transitions that drop an artifact expectation, a last-known-good
artifact, or a source acceptance now say why, an unproven restore is marked as
such, and the deriver folds all of it into the limitations the projection
already exposes. Clearing a code when the evidence becomes provable again is
part of the contract, so a stale doubt cannot outlive its cause.
This lands before the migration matrix because migration is the largest
producer of evidence that cannot be proven, and the plan requires those to
surface as bounded limitation evidence rather than as fabricated pointers.
* feat(git): migrate pre-existing Git stacks into the revision state
Git stacks created before this model had no application at all, so every
producer was a no-op for them and the projection could not describe them. They
are now brought in at boot.
The governing rule is that a canonical pointer is written only when the
evidence proves that exact generation under the repository and ref configured
now. A legacy applied commit is not that proof on its own: the manifest may be
gone, unreadable, or stamped for a repository the stack no longer points at. In
each of those the commit is kept as recorded limitation evidence and the
pointers stay null, so the stack asks for a fetch instead of asserting a state
nobody verified. Deployed, healthy, and last-known-good are never invented,
because a manifest proves what was materialized, not what is running, and no
source acceptance is written because nobody approved through the model.
Replay is decided by scope, schema version, and configuration fingerprint. A
changed fingerprint re-runs the matrix, but a stack that already has an
application is skipped rather than rebuilt, so migration cannot overwrite
pointers written with proof it does not have. A stack whose directory has
vanished migrates to a tombstone rather than claiming a name it cannot back.
* feat(git): record rollbacks in the revision state
A restore now opens a recovery in the model before any file moves, so a crash
mid-restore leaves a target that says what it was doing rather than one that
merely looks broken, and closes it on both outcomes. The failure is classified
by whether the files had already been restored, because only a failure before
that leaves the previous workload provably intact.
Pointers move only when the restore is provable: the recovery point named a
generation, that generation still exists, and the manifest actually restored
carries the same commit and manifest version. Anything less is still recorded
as a real recovery, it simply has nothing to bind, and the transition marks it
unproven rather than guessing.
The deployed pointer is never claimed here. This path drives Compose through a
callback that reports nothing back, so binding cannot be proven and applied
moves without it.
* feat(git): add the deferred rollout-state transitions
Retry scheduling, suspension, pause, and partial rollout complete the
transition store. They have no production writer by design, but implementing
them now means the deriver has no branch a writer cannot reach, and the shape a
future producer has to satisfy is pinned rather than inferred from the read
side.
None of them is a statement about health. A scheduled retry leaves the failure
that caused it visible, so a stack that keeps failing does not read as merely
busy. A suspended source keeps everything it had accepted, and an operation in
flight is interrupted rather than abandoned so it cannot report as running for
ever. A paused or partially rolled out target keeps whatever was deployed, and
the partial record never stands in for a deployed pointer.
* test(git): stop the Compose producer tests reading Docker from the host
Both deploy cases asserted that the compose command would reject, which was
only true on a workstation with no Docker daemon. On Linux the command
succeeded, the promise resolved, and the assertions failed.
The compose subprocess now reports an exit code each test chooses, so the
adapter is what decides the outcome rather than the machine. Only the verbs
that need a daemon and travel through spawn are answered; `config` still runs
for real, because a host with the CLI and no daemon parses compose files
exactly as CI does, and failing that call breaks every create in the file.
That also makes the bound path reachable, so the deploy test now covers it:
a failed compose leaves the deployed pointer alone and classifies
post-mutation, and a successful one binds the applied generation, returns it
for health to bind against, and clears the earlier failure.
* fix(git): contain every managed-area path at the call that uses it
A stack name reaches the managed root without passing through
`isValidStackName` on this path, so the marker, the cleanup, and the create's
root probe all built filesystem paths from unvalidated input. Each call now
resolves its target against the managed area and checks containment in its own
scope, which is also the form the security scan credits: it does not follow the
barrier through the shared `isPathWithinBase` helper, so the check has to sit
with the call it protects.
`cleanupUnclaimedManagedRoot` removes a whole root recursively and gets the
same check, even though it was not among the reported calls.
Clearing a staging marker can now fail, so the two boot-recovery branches that
dropped the checkpoint first were reordered. Losing the checkpoint while the
marker survives would report the area as settled and leave the stack name
uncreatable.
The deploy adapter's two failure logs built their format string from the stack
name; both now use a constant format with the name as an argument.
The marker and cleanup fixtures stood a bare temp directory in for a managed
root, so nothing exercised the invariant these checks enforce. Both now build a
real managed area over a scoped data directory.
* fix(git): refuse to claim a managed area whose marker cannot be read
A marker that exists but will not parse is still someone's claim, yet the write
path refused only a readable foreign marker and wrote straight over a corrupt
one, discarding the reason and logging nothing. A transient permission error, a
marker truncated by a crash between write and rename, or a full disk all read
as corrupt, and each one let a second create take ownership of an area the
first still owned. Every other path in the module preserves on corrupt; this
one now does too, and says why.
Clearing a settled create's staging marker can fail, and reporting that as a
retained create sent a reader looking for an unfinished create that had
finished. It reports `marker_retained` instead. The checkpoint is still kept
for the retry, so its encrypted token can outlive the create while the clear
keeps failing; that tradeoff is stated at the call, because the alternative
leaves a marker no later create can get past.
The reaper also computed why it was preserving an area and threw the reason
away, so a directory could survive every boot with nothing said about it.
The containment checks these paths rest on had no tests and were silently
deletable, so the out-of-area read, write, delete, and reap now have them.
* feat(git): bind and observe a recovered generation
A proven restore could never claim the deployed pointer, because the restore
path drives Compose through a callback that reported nothing back: a caller
that restored some other way resolved identically, so binding on a resolved
promise would have claimed a workload nobody launched. The Compose wrapper now
returns what it did, and only that answer binds.
With binding reachable, a bound recovery claims its health run inside the same
transaction as the pointers it describes, and arms the timer once that
transaction lands. Committing the two together is the point: a crash between
them would otherwise leave a restored workload nothing was watching. Anything
that stops the timer starting writes the run off immediately, so an observing
row never outlives the timer meant to watch it, and a reservation is never
armed across a restart.
The reservation is handed to the transition rather than reached for, because
the health gate reports its verdicts back through the transition store and
importing it there would close a module cycle.
Startup now finalizes each interrupted observation on its own instead of
sweeping them with one update, so the revision state hears a verdict for every
run rather than watching the rows change under it.
* feat(git): add the rollout-scoped rollback transitions
Completes the deferred-state set. The three aliases write the same recovery
columns under the same rules as the recovery events, differing only in
provenance: Direct Git recovery emits `recovery_*`, and a later rollout
producer emits these. Nothing in this PR writes them, which is why they are
tested directly rather than through a caller.
`partial` is the one failure class they add, for a rollback that reached some
targets and not others. Completing has no unproven variant on purpose: a
rollback nobody can bind to a generation has nothing to complete against, so
it refuses without one and refuses a generation another application owns.
Also covers the recovery health reservation end to end: reserving writes and
links the row without arming anything, a replay reuses the run rather than
opening a second observation of the same restore, arming inserts nothing and
supersedes a conflicting stack gate, an unarmed reservation is written off
once, and a reservation that outlived its process is finalized rather than
armed. The per-row startup sweep is pinned too, since one row whose write
fails leaving the others finalized is the entire reason it replaced a bulk
update.
* feat(git): add the Blueprint source and deployment transitions
The first slice of the Blueprint work: the store side, with no production
caller yet. The routes and the reconciler are wired to these next, and pinning
the contract first means a caller cannot quietly satisfy a different one.
What these enforce is that a terminal event names the request it answers. A
deploy records the intent and candidate it was launched for, and a terminal is
accepted only against that same request, matching stage and identity on one
side rather than either in isolation. Without that pairing a deploy could be
acknowledged out of an in-flight withdraw, or one request's intent stored
against another's candidate. An acknowledgement carries the candidate from the
matched request rather than from its own payload, for the same reason.
Releasing an operation now releases its identity with it, and a resolved
interruption is retired rather than left to match again. Both were leaking
through the shared helpers: a stale identity let a later start resurrect a
superseded intent as live, and a surviving interruption both reported
completion as unknown for ever and let a late acknowledgement regress the
target after two later deploys had succeeded. Fixing the helpers fixes the
Direct paths too.
A start refuses to displace an unrelated operation, which would otherwise
abandon it with no terminal event and no history saying so. Observations write
the latest stage, since the placement facet reads it to decide whether a
stateful deployment is waiting on confirmation, and history alone could never
reach a reader.
Minting is left to the caller. A no-op Blueprint edit must mint no intent,
because a fresh identity would invalidate acknowledgements that are still
accurate.
* feat(git): record Blueprint create, edit, pin, and delete
Wires the Blueprint routes to the revision state. Each producer writes the
Blueprint source row and its GitOps rows in one transaction, so an operator
never sees a Blueprint that exists with nothing describing what it means, or an
intent for a Blueprint that failed to save.
The question these answer is when an edit invalidates what the fleet already
acknowledged. Changing the name, the compose content, the selector, the drift
mode, or whether it is enabled changes what nodes run or where, so each mints a
new intent and candidate. Changing the description or the classification
changes how the Blueprint reads and nothing a node can observe, so it updates
the source row alone: minting there would make every acknowledgement report as
stale over a reworded sentence.
An edit is measured by value, not by which fields were submitted. The editor
sends every field on every save, and the source layer decides what to
invalidate from which keys are present, so only the keys that genuinely differ
are passed down. Without that the two disagreed: rewording a description
advanced the revision past the one the current intent describes and cleared the
approval, while this layer classified it as metadata and minted nothing.
Selectors compare by value too, so reordering a list that names the same nodes
is not a placement change.
That measurement also changes one behaviour: a save that alters nothing now
leaves the approval alone, where before any save cleared it. Nothing changed,
so nothing is invalidated.
Pinning revises placement the same way a selector edit does, and re-pinning the
node already pinned is not a change. Deleting tombstones the application and
its live targets so the Blueprint stops claiming its slot, and withdraws
nothing itself, because the route has already done that and doing it twice
would record removals that never happened.
The required node set is stored in a canonical order, so reordering it is not
mistaken for a placement change. A Blueprint that predates the model has no
application yet and is left for migration rather than given a first intent
here, which would claim a starting point for deployments nobody has reconciled.
Desired nodes are passed in rather than computed here: the reconciler that
knows how to compute them reaches this layer, and importing it back would close
a module cycle.
* feat(git): record the placement a label or cordon moves
A label and a cordon say nothing about any one Blueprint, but both change which
nodes a selector matches. Each now revises placement for whichever Blueprints
the change actually moved, by comparing the desired nodes either side of the
write rather than reacting to the event.
That comparison is the whole point. Labelling a node no selector mentions, or
cordoning one no Blueprint wanted, moves nothing, and minting an intent for it
would invalidate every acknowledgement in the fleet over an edit no node can
observe. The same comparison covers a Blueprint pinned to a cordoned node: a
cordon governs automatic placement only, so the pinned target still wants that
node and its set does not move, without needing a case of its own. Nodes are
compared as a set, so the same nodes returned in a different order is not a
change either.
Each route wraps its existing write and the recording in one transaction, so a
recording failure cannot leave a fleet selecting on a label nothing recorded.
The write itself is untouched, which keeps label validation and the per-node
limit where they already live rather than restating them in a producer.
Node deletion already retires its targets, so this adds nothing there.
* feat(git): record Blueprint deployments by what caused them
Every production write to a Blueprint deployment row now goes through one
funnel that records the cause. The cause is carried rather than inferred from
the resulting status, because several causes land on the same one: a deploy
that failed and a withdraw that failed both read `failed`, and they mean
opposite things about whether the deployment is still on the node.
Recording is skipped when the status did not move, so a reconciler tick that
re-asserts a state it already reported does not append a second event
describing the same fact. The write still happens either way.
A terminal answers the request the target says it was given, not whatever the
Blueprint currently wants: an acknowledgement matched against the current
intent would accept work for a revision that node was never sent. The first
deploy to a node creates its target, since a Blueprint application has no
targets until something is sent somewhere.
Recording never fails the deployment. The rollout already happened, and turning
a bookkeeping problem into a stuck rollout would be the worse outcome.
The four reconciler observations record what was seen and nothing else. Preview
cleanup stays outside this path deliberately: it reverses a projection nobody
deployed, so recording it would report removals that never happened.
* fix(git): record an Inline Blueprint coming into existence
Creating a Blueprint inserted its application row directly, so the history
began at the first intent and described an application nothing recorded coming
into existence. Every other application-creation path emits the activation
event, and the event's own definition covers this case: an Inline Blueprint
inserts the application, the first intent, and the first candidate.
Activation goes through a transition now, which also enforces the one thing the
direct insert could not: a Blueprint gets one live application, refused rather
than silently duplicated.
No target is created. A Blueprint application has no targets until something is
deployed somewhere, unlike a Direct one which always has the node its stack
lives on.
* fix(git): file withdraw failures as withdraw failures
Four of the five withdraw failure paths were recorded as failed deploys. Only
the thrown one was tagged correctly, and it is the rare case: a delete-lock
conflict, a network error, a remote 409, and a remote non-200 all return rather
than throw, and all four were filed under deploy.
That inverted the thing this funnel exists to prevent. Recording a deploy
failure clears the in-flight operation, so a routine "another operation is
already in progress" wiped the withdraw that had just started and left the
target reporting a failed deploy while the deployment was still on the node.
A withdraw refused because the on-disk stack is not ours is now its own cause
too, rather than borrowing the deploy-side name conflict.
The status guard no longer suppresses starts. A start writes the identity that
terminals are matched against, so skipping one because the row already read
`deploying` let a later acknowledgement answer a request that had been
superseded, reporting a node as converged on a revision it was not running. The
guard now covers observations only, which is where repetition is the reconciler
re-asserting a state it already reported. A newer deploy supersedes an older
one so a redeploy of a stuck request can take over; anything else in flight is
still refused rather than displaced with no terminal event.
A rejection is logged differently from an infrastructure error, because a
target the model keeps refusing holds its active slot and stops recording
anything further, and that is worth seeing.
* feat(git): migrate pre-existing Blueprints into the revision state
Blueprints that predate the model are brought in at boot: an application, an
intent describing what the Blueprint currently asks for, and a candidate marked
as coming from the legacy inline record.
None of it is an acknowledgement. The Blueprint revision is carried for display
and nothing else, because a revision number is not evidence that any node is
running what it names, and recording it as agreement would report a fleet as
converged on an intent nobody verified. No targets are created for the same
reason, and the candidate's required set is empty: migration records what is
asked for, never which nodes currently satisfy it.
An approval authorizes the intent it was given for, so a Blueprint that was
never approved, or edited since, is recorded as needing reapproval rather than
left blank. Blank would read the same as an approval that is merely absent.
Replay is decided by the same scope, schema version, and fingerprint the Git
migration uses, and a Blueprint the new path already described is skipped
outright, since its rows were written with proof this pass does not have.
Blueprints migrate in their own pass. Coupling them to the Git migration would
let one unreadable Git stack keep every Blueprint outside the model.
* fix(git): contain the blueprint stack-directory probe at its call
The ownership probe resolved a path from the blueprint name and checked it
through the shared helper, which the security scan does not credit as a
barrier. The check now sits with the call it protects, matching the form used
elsewhere for the same reason.
The path was already validated, so this changes no behaviour. It surfaced now
because an earlier commit in this branch rewrote the file's line endings, which
made every line read as new and brought a pre-existing call into the scan's
changed-code window.
* feat(git): read GitOps history and carry revision state on source reads
Adds the instance-local history API and the additive revision fields the
source reads have been missing.
`GET /api/git-sources` and `GET /api/stacks/:stackName/git-source` now
carry `gitopsRevision` and `stackResourcePresent`. Only the instance that
owns the files can say whether a stack directory is really there, so that
answer travels with the response rather than being inferred by a reader
that has never seen the filesystem. It is read through the strict stack
listing: the lenient one answers a failed directory read with an empty
list, which here would read as every stack having vanished and would turn
an unreadable disk into an empty audit trail.
Two history routes land with them: `GET /api/git-sources/history` across
stacks, and `GET /api/stacks/:stackName/git-source/history` for one. Both
page on `(created_at, id)` so rows written inside the same millisecond
survive a page boundary, and both bound how far one request will scan. A
malformed cursor or an unusable filter value is refused rather than
ignored, because answering "show me the failures" with everything reads
as an answer instead of a non-answer.
Authorization is decided per row. A row reduces to a stack read only when
it names a stack, its application is live, and the stack is present on
disk; anything unprovable stays with Admin. The cross-stack routes
classify every row, while the per-stack route is authorized whole by
name, so reading a stack shows its full history including entries from an
earlier application. That distinction is carried in the scope type rather
than a flag, so skipping the row classifier without pinning the query to
the authorized stack cannot be expressed.
History entries record the fields each transition moved rather than a
whole revision, so lifecycle for the classifier comes from the owning
application row. That keeps authorization off the recorded payload
entirely: an entry whose detail cannot be read still returns its identity,
stage, and outcome with a stated limitation, and the decode failure is
logged rather than passed over in silence.
* feat(git): let auditors read GitOps history they cannot tie to a stack
A history entry that cannot be tied to a readable stack fell to Admin,
which left the auditor role seeing exactly what a viewer sees on the one
surface it exists to read.
History entries are an audit trail: insert-only, and recording the actor,
trigger, stage, and outcome of every transition. The request audit log is
already gated on the audit permission, so an entry whose audience cannot
be narrowed now falls there too. Withholding it protected nothing in any
case, since the request log already records that a Git-source mutation
happened and who made it.
The source collection deliberately keeps its Admin fallback. Those rows
are live Git configuration, not a record of events, and a mandate to
audit does not extend to reading the repository, ref, and credentials
policy of stacks that were deleted or never finished being created.
The fallback is now part of the requirement type rather than a role
comparison, so each surface states which audience it falls back to and
the compiler requires every case to be answered.
* feat(git): correct remote node identities on GitOps reads
A remote instance numbers its own nodes and has never heard of the hub's
numbering, so every node id it reports is a statement in its own
namespace. Read straight through, a hub joining two nodes showed two
different machines as the same node.
Four GETs now take a second hop that buffers the response and corrects
those identities: the git-source list and detail, and both history
routes. Everything else keeps streaming exactly as before, which is the
reason this is a separate hop rather than a mode of the existing one:
logs, downloads, and event streams must never be buffered, and the
enumerated positions rewritten here do not appear in them.
Only JSON numbers are replaced. A null node is preserved, since "no node"
is a fact the remote is entitled to state and inventing one there would
claim a placement that does not exist. Strings, application ids, and
stack names travel untouched.
The hub re-authorizes what comes back. A remote authorized its rows for
the machine account the hub proxies with, which says nothing about the
person behind the request, so every row is classified again against the
signed-in user. To make that possible without the hub holding another
instance's database, each history entry now carries the lifecycle of its
owning application alongside the stack-presence answer already there.
Both are validated fail-closed, and a verdict a remote might volunteer is
ignored: the hub decides, the remote supplies evidence.
Asking one node for another node's history is refused before the hop
rather than forwarded, because the remote would answer about itself and
the page would read as an answer to a question nobody asked. When the
node asked for is the node being talked to, the hub translates that into
a flag the remote resolves to its own default node. Only the hub may set
that flag, and a remote honours it only on a proxied hop.
Every way this hop can end converges on one terminal answer: rewritten,
passed through, too large, undecodable, failed upstream, or abandoned by
the client. A body the hub could not read never borrows the remote's
success status.
* test(git): assert the unlinked git-source response field by field
The detach test compared the whole response to `{ linked: false }`, which
stopped holding once that route started carrying the additive revision
fields. The equivalent backend route test was updated when those fields
landed; this one was missed.
Asserted field by field rather than loosened to a partial match, so the
two new fields are actually checked: a detached stack has no application
to project while its directory is still on disk.
* feat(git): carry GitOps revisions on blueprint, node, and drift responses
Completes the additive JSON half of the revision-state read contract, so the
Blueprint, node, and drift surfaces report GitOps state through the same
projection the Git-source routes already return instead of leaving it
unreadable.
Blueprint list, detail, create, update, and pin carry gitopsRevision. Node-label
add, cordon, uncordon, and node delete carry gitopsRevisions, ordered by
blueprint id and covering only the Blueprints the mutation actually moved: an
edit no selector reacts to reports an empty list rather than invalidating the
whole catalog. Node deletion reads its Blueprint owners inside the deletion
transaction and before the tombstone, which is the last moment a target row can
be traced back to an application, and an orphaned target is logged rather than
being silently indistinguishable from a Direct one. Both DELETEs that answered
204 still answer 204.
The drift GET and re-check gain the same field alongside the existing ledger,
which is untouched; no GitOps class is written into stack_drift_findings.
Reads and mutations treat a projection fault differently, on purpose. On a read
the revision is part of the answer, so a fault surfaces. On a mutation the write
has already committed, so the revision is decoration and is built defensively:
letting it throw would answer a successful cordon or a completed node deletion
with a 500, sending the operator to retry a hard delete that already happened
and be told the node does not exist. It degrades to an empty list and logs.
Because both drift routes answer with that projection, the identity hop now
intercepts them too, matching its route table per method instead of rejecting
every non-GET. Left as they were, one object would carry the hub's node
numbering or the remote's depending only on which route asked for it. The
allowlist names a single mutation rather than opening a verb, and the hub's
collection filter still keys off the git-source paths alone, so drift payloads
are rewritten without being re-authorized: they are per-stack and already
authorized by name before the hop.
Tests cover the exact shapes, the preserved 204s, revision ordering, the
empty-list cases, node deletion reporting its tombstoned Blueprints and
surviving a projection failure, and the proxy's new route matching, drift
rewrite, and filter exclusion.
* fix(git): resolve every application a GitOps surface can own
Three gaps found while building the additive JSON, all the same shape: an id or
a sentinel stood in for a row, so "there is nothing here" and "the thing that
should be here is unreachable" collapsed into one answer nobody could tell
apart.
A Blueprint application is stored with stack_name NULL, so no lookup by stack
name could reach it, while the reconciler materializes every Blueprint as a
stack directory of that name. A stack's Drift tab and Git panel therefore
reported no GitOps at all for a stack GitOps was actively managing, while the
Blueprint page reported a live application for the same thing. Stack surfaces
now bridge the two through the deployment row for the asking node. The
deployment row is what makes it safe: Blueprint and stack names share one
namespace, so matching on name alone would let a Blueprint claim an unrelated
stack of the same name on a node it never targeted.
A retired application was unreachable for the same reason, because every entry
point filtered to the live rows. Tombstoning deliberately keeps the configured
identity and SHA pointers as frozen facts so the projection can still say what
an application was, and the source deriver has a not_live status waiting for
exactly that, but nothing could hand it one. A detached Blueprint or stack now
reports what it was instead of reading as one that never existed.
The not-applicable projection was typed so it could not carry a reason, which
made a row that vanished between resolving it and re-reading it by id identical
to a stack the model was never asked about. That one field widens to hold
limitations, and the missing case now says so with the id as evidence. Every
other missing-row path in the deriver already worked this way.
Also logs, rather than silently skipping, a live application whose blueprint row
is gone: with cascade off that is a real integrity fault, the placement did
move, and the response would otherwise report that nothing had.
Cordon and uncordon are deliberately left reporting nothing. A cordon governs
whether new placements may be made, not what a Blueprint asks for, and the
reconciler applies it only to new placements, so revising intent for it would
invalidate acknowledgements fleet-wide over a change that evicts nothing. Two
comments implied otherwise and now state the contract.
* fix(git): keep stack-state resolution out of the Git-source read path
Review of the previous commit found the fallback it added had reached further
than intended. The resolver is now split by the question it answers.
projectStackRevision answers "what Git source is attached to this stack" and
stays Direct-only. projectManagedStackRevision answers "what manages this stack"
and is what the drift routes use. The Git-source routes must not be answered
with another application's identity, and there was a second reason: the row
classifier takes the stack name from the Git-source row but the lifecycle from
whatever the projection resolved, so a Blueprint application arriving there
would have turned an Admin-only row into one any stack grant could read. The
Blueprint that unlocked it could be the very one holding name_conflict because
of that stack.
The Blueprint-to-stack bridge also needed the right predicate rather than a
present deployment row. name_conflict is written precisely when a stack of that
name already exists on the node and Sencho does not own it, so guarding only
against withdrawn admitted exactly the collision the bridge exists to prevent.
It now requires last_deployed_at, which proves this Blueprint wrote the
directory, and excludes name_conflict and withdrawn: the same predicate the
delete and withdraw paths use. Only a live Blueprint application qualifies,
since a retired one has no claim on the directory.
The detached fallback no longer resolves deleted applications. Detached means
the files are still on disk and still describe that stack; deleted means the
stack is gone, so any directory of that name now belongs to something else and
reporting the old repository and SHA against it would disclose one stack's Git
identity through another's name. The authorization side already refused deleted
rows for that reason, so the two now agree. Ordering breaks ties on rowid rather
than a random UUID, and two partial indexes cover the new lookups, which the
existing live-only unique indexes could not serve.
The not-applicable variant's limitations are readonly, so pushing onto the
shared frozen instance is a compile error again rather than a runtime throw.
Tests cover a Blueprint claiming a stack it never deployed onto and one it hit a
name conflict on, a detached Direct source projecting not_live, and a deleted
one staying unresolved for a reused name. Drift fixtures move to afterEach so a
failing assertion cannot leak state into the next test.
* fix(git): stop the stack resolver falling through proven ownership
A second review round found the resolution chain could answer with the wrong
application, and that part of the previous commit was reading a state nothing
writes.
The Blueprint bridge returned one value for two different facts. Once the
deployment row proved a Blueprint had written the directory, a missing
application row for it still answered "no Blueprint here", so the chain
continued to the detached-Direct lookup. A stack that once had Direct Git, was
detached, and whose directory a Blueprint later took over would then report the
old Direct application's repository, ref, and SHA as that directory's state.
The bridge now separates "not mine" from "mine, but broken": the second logs and
returns its own limitation, so the fault is visible and no unrelated identity is
offered in its place.
A Blueprint projection also carried the whole placement roster into a route
authorized by a grant on one stack name. Targets are now scoped to the node
being asked about, which is both the safe answer and the accurate one, since the
question is what manages this directory here.
The detached-Blueprint lookup, its index, and its fallback are removed. Blueprint
retirement writes deleted, never detached, and the only detached writer is the
Direct detach path, so the getter could never match and the index covered an
empty set. The test that seemed to cover it built the state by hand and was
green against something the product cannot produce; it now drives the real
delete route and asserts what that path actually writes.
The Git-source resolver goes back to live applications only. Detach removes the
source row in the same transaction that tombstones the application, so a source
row beside a detached one is not producible, and the route's row classifier
takes its stack name from the source row but its lifecycle from the projection.
Keeping that resolver narrow is what stops a resolution change from quietly
moving who may read a row. The comment saying so is restored and now explains
why it must stay true.
* refactor(git): share the GitOps application fixture across the route tests
Two test files carried near-identical 58-line GitOpsApplicationRow literals that
differed only in a repo URL nothing asserted on. Both now import one helper.
The drift tests also repeated a Blueprint seed and its activation across four
cases, and re-imported DatabaseService per test; those become local helpers and
one hoisted import, which lets the cleanup hook go synchronous. Node lookup
throws on a missing default instead of casting.
projectBlueprintRevisions is module-private again: its only caller is the
committed-projection wrapper beside it.
A third copy of the same fixture remains in git-source-routes.test.ts. It is
left alone deliberately: it pins timestamps at 1 rather than now, and that file
has ordering-sensitive history assertions that would need auditing first.
* feat(git): mirror the GitOps revision contract on the frontend
Adds the client-side read contract for the GitOps revision projection, with
no consumers yet.
types/gitops.ts is a hand-written mirror of the backend read contract, in the
same convention as the other domain type files: the frontend never imports
backend. The projection's two arms are named separately so a component can
take the live one as a prop without re-narrowing, and the absent arm keeps its
absent keys rather than nulls so reaching for a lifecycle status without
narrowing is a compile error. The limitation code stays an open string, since
the backend adds codes without a schema bump and an exhaustive switch would
silently stop rendering the newest ones.
lib/gitopsState.ts is the one place a facet status becomes words and a colour,
so a state reads the same in a sidebar tooltip, a panel banner and the drift
tab. Both maps are keyed on the closed status unions, so widening the mirror
without naming the new state fails this build.
Every line of copy states the condition the deriver actually tests, which is
not always what the status name suggests. recovery_required is a recovery
already running rather than one that is needed. candidate_ready is reached only
when review is not required, so it is ready to apply, not ready to review.
synced_and_healthy is also reached with the health gate switched off and never
compares the deployed generation against the accepted one, so it claims
neither. never_reconciled means nothing has been accepted; a fetch that
produced no materialization still records its commit.
pendingSourceStatus keys "an update is waiting" on the candidate pointer rather
than the status name, because source_reconcile_required is reachable both from
a stale candidate and from an accepted generation with no candidate at all.
Retirement is excluded first: tombstoning keeps the candidate pointer as a
frozen fact, so a stack detached mid-review still carries one.
absentFault separates the two facts the empty projection carries. Empty
limitations means the model was never asked about this stack and the right
rendering is nothing; a non-empty list means an application that was expected
could not be reached. A live application's limitations are caveats on state
that is being reported, not faults, so they are deliberately excluded.
* feat(git): name the Git source state on the stack and sidebar surfaces
The sidebar indicator and the anatomy source row both read a raw pending
commit pointer, so a candidate blocked by local conflicts, held for review, or
stale against the configuration in force all render identically to one that is
ready to apply. The projection distinguishes them; nothing was reading it.
refreshGitSourcePending now derives each stack's state from the projection and
the pending map carries that state rather than a boolean. Presence in the map
still means exactly what it meant before, because it is keyed on the candidate
pointer rather than the status name: source_reconcile_required is reachable
both from a stale candidate and from an accepted generation with no candidate,
and only the first is something to review. One narrow fallback remains, for a
row with no projection at all: a failed GitOps write is logged and swallowed
while the pending commit still commits, so the flat pointer is the only thing
that can answer there. Wherever a projection exists it is the sole authority.
The sidebar keeps one indicator, in the same slot, at the same size and colour,
with the same position in the priority ladder. Only the tooltip changes, so a
blocked plan now says so instead of reading as an ordinary update. A test
asserts the rendered indicator is identical across states, which is what keeps
the rendered sidebar unchanged.
The anatomy source row keeps its pulsing dot alone for the ordinary case and
adds the state word only for the states the dot cannot express. Inline rather
than in a tooltip, since this is the reason something is stuck.
* feat(git): surface derived GitOps state in the Git source and Drift panels
The Git source panel's pending banner had two things it could say, read from a
raw commit pointer and a single blocked boolean. It now names which of four
states the candidate is actually in, so a plan blocked by local conflicts, one
held for review, and one gone stale against the configuration in force stop
rendering as the same ordinary update. The short commit sha stays.
A new card above it reports an application the projection could not reach. That
case renders as nothing at all today, which is indistinguishable from a stack
the model was never asked about. Empty limitations stays silent, because that
is the ordinary answer for most stacks and a header over an empty block would
be worse than nothing.
The summary block gains one row naming the source state. It is the first place
the panel can show applying, retry scheduled, suspended, recovering, or a
detached source, none of which have a pending commit and so never reach the
banner. Both existing rows are untouched and the last applied commit stays a
display fact.
The revision lives in its own state rather than on the Git source type, because
the PUT that saves this panel answers with a bare source and no revision. A
save drops it instead of rendering a state the write has already moved.
The Drift tab gains a third block below the two it already has. The compose
versus runtime card and the since-last-deploy card keep their exact positions
and copy; the new block answers a different question, which generation this node
was asked to run and whether it got there. A Blueprint-owned stack shows its
target rows and no source card, since a Blueprint application has no Git source
and inventing one would be a claim the model never made. The drift class list
renders against the type and expects no rows, because nothing populates it yet.
* feat(git): declare revision fields on the blueprint and node clients
Types first. The blueprint list, detail, create, update and pin responses all
carry a GitOps revision, and the node label add, cordon, uncordon and delete
responses carry a list of them. None of that was modelled, so the fields were
being dropped silently and the next reader would have taken these for bare
records.
Most of it is declared and deliberately unread, with the reason written where
the type lives. Create, update and pin are followed immediately by a re-read of
the catalog or the detail that carries the same projection, so rendering the
mutation's copy would show the same fact twice with one of them stale. Cordon
and uncordon always answer with an empty list by design, because a cordon
governs whether new placements may be made rather than what a Blueprint asks
for, and the reconciler leaves existing deployments where they are. The comment
on that type is the point: the risk there is a future reader building a
consumer for a list that is never populated.
Two places do report something. Deleting a node and adding a node label both
re-place Blueprints, and both now say how many. The count only, and only when
it is not zero: an empty list means both that nothing moved and that the
projection faulted after the write had already committed, so it can never be
reported as the first.
Blueprint detail gains the same unreachable-application card the Git source
panel has. A Blueprint with no live application row is a fact nothing in the
product could previously express, which is a different problem from having two
ways to say the same thing.
* fix(git): close the gaps review found in the GitOps frontend consumers
Six defects, each verified against the code before being fixed.
The pending-map read assumed every /git-sources row carries a projection. That
route is proxied, so a node predating the model answers rows without one, and
dereferencing it threw inside the loop. The catch swallowed the throw, the map
was never set, and every stack's Git indicator froze at its previous value for
as long as that node was selected. The field is optional now and a row without
one falls through to the same branch as a row with nothing to project.
That fallback also branched on the arm alone, so a projection reporting an
unreachable application plus a stale commit pointer was reported as a candidate
ready to apply. It now excludes a faulted projection: a fault means an
application was expected and could not be read, so the pointer is not evidence
that anything is ready, and naming a state there would be a guess.
Detaching a Git source cleared the source but not the revision, and the pending
card is derived from the revision alone. The panel kept advertising a waiting
commit for a stack Git no longer managed, behind a Review button that silently
did nothing. A read that threw had the same problem across stacks: the panel is
reused, so stack A's state could render under stack B's header.
The panel also lost the flat-pointer fallback the sidebar keeps, so the two
surfaces disagreed: the sidebar would show an indicator and clicking through
showed no card. Both now apply the same rule.
Node delete parsed the success body unguarded. The delete has already committed
at that point, so a malformed body would have reported a completed deletion as
a failure and skipped the refresh, leaving the deleted node on screen.
Tests: five mutations that previously survived now fail. The zero-pixel claim
compared the wrong element and passed when the indicator vanished entirely; the
stale-read test asserted during the loading window, when the body is skeletons
regardless; the sidebar passthrough test proved only that an indicator existed,
not that the state reached it. Added coverage for the drift row, an unknown
node, the not-applicable source guard, the save clearing, the Review button, and
a live application carrying a caveat, which must not read as a fault.
The fixture set drops two source statuses whose identity defaults describe a
state they cannot be in, and gains a drift-item builder.
* refactor(git): share the GitOps fault card and live-facet derivation
The unreachable-application card was built inline on three surfaces, and the
copies had already drifted: the Git source panel used one icon while the Drift
tab and the Blueprint sheet used another, so the same failure rendered two
different ways. It is one component now, which also owns the state key and the
test id those surfaces assert on.
Deriving the live source facet was likewise repeated, along with the two
semantic decisions behind it: the absent arm has no facets, and a source facet
of not applicable means a Blueprint owns the stack rather than that something
is missing. Both now live next to pendingSourceStatus, which already encoded
the same exclusion.
The Git source panel's seven derived values collapse to three, with the
pending-commit rule extracted to a named function so the four cases read
straight through instead of as nested ternaries. Behaviour is unchanged in
every case.
The two flat-pointer fallbacks are deliberately not unified: they look alike
but the panel also treats a live application whose source facet is not
applicable as unanswered, and merging the predicates would change what that
case renders.
* feat(git): count and announce committed GitOps transitions
Every history row that is actually inserted now produces one metric
increment and one state-invalidate event, so the surfaces that read GitOps
state hear about a change instead of waiting for the next poll.
Announcement is buffered and drained on a macrotask rather than fired
inline. better-sqlite3 is synchronous, so waiting for the macrotask puts
the drain after the transaction that wrote the row, and after any outer
transaction wrapping it, without having to detect which nesting depth it
is in. The drain confirms each row is still present before announcing it,
so a rolled-back transaction says nothing on its own, and the insert
declines to queue a dedupe replay, so a retry says nothing either. The
broadcaster is injected at startup rather than imported, keeping the
GitOps layer free of a cycle back into the notification stack.
Counters are process-local and in-memory, and their keyspace is finite by
construction: history stages are now a closed union that the build
enforces, and outcomes were already a closed set. The payload names no
stack, node, repository, or actor, because a counter carrying those would
be an audit trail with no retention rules and no per-row authorization,
which is what the history routes provide. GET /api/gitops-metrics is
Admin-only and instance-local, so selecting a node answers with that
node's counters.
On the client, a gitops-scoped invalidate refreshes the derived state
through a 250ms trailing window, matching the existing stack refresh. One
operation commits several transitions in a row and a first-boot migration
commits a great many, so refetching per event would thrash the API for a
picture that only settles at the end.
* feat(git): show GitOps source state on the stack dashboards
The dashboards list every stack on a node, which is where a fleet-wide
reading of Git state is most useful and where, until now, a Git-backed
stack looked exactly like a local one. Each row that the model has
something to say about carries a chip naming its source state.
The chip reads from the same status vocabulary the panels and the sidebar
use, so a stack cannot be "pending update" in one place and something else
in another. It sits beside the stack name rather than replacing the source
column: the column says where the files come from, the chip says what
GitOps makes of them, and those are different facts. The label is a word
and the title is a whole sentence, so the state never rests on colour
alone; the phone rows keep the word on screen, since touch has no hover.
The join is by stack name, which is what the dashboards have, and its
source is the same proxied route the sidebar reads. A row without a
revision, a Blueprint-owned application, and a projection fault all leave
the stack unbadged rather than inventing a state for it. State arrives by
announcement rather than by poll, since it only moves on a transition, and
a node switch blanks the map first: stack names repeat across nodes, so a
slow answer for the node just left is discarded rather than allowed to
label the wrong stacks.
* feat(git): say which part of a GitOps state could not be proven
Eighteen conditions can qualify a live projection: a manifest that does
not match the branch configured now, an approval that could not be
restored after a recovery, an artifact record that has gone. Every one was
already recorded and none of them reached the operator, so a state with a
hole in it read exactly like one without.
Each now has operator wording under the state it qualifies, checked
against the site that emits it rather than against the code's name, since
several names describe something narrower or wider than the condition
actually tested. The stored messages stay out of the UI: they are written
for a log reader, and some are raw decoder errors.
The presentation is deliberately quiet. A caveat is not a failure: the
state above it is real, and one piece of evidence behind it is missing, so
the reader learns which part to distrust without being told the whole
thing is broken. Faults keep their own card, because those replace the
state rather than qualifying it, and the two arms are read through
separate helpers so neither can be rendered as the other.
A code with no wording names itself rather than vanishing, so a node
running ahead of this build degrades to something honest instead of
reporting full confidence in a state its own backend flagged.
* fix(git): record a stateful first placement and keep a half-built Blueprint editable
Two faults that today's product cannot reach, and that the Git-backed
Blueprint mode would reach immediately.
The live-application lookup answers with an application that is active or
still being created, because its other callers ask whether the Blueprint
already holds the live slot, where a half-built row counts. Three
producers passed that answer straight into transitions that accept only an
active application and reject anything else by throwing, inside the
caller's own transaction. A Blueprint edited or pinned while its
application was still being created would have failed with a server error
and rolled the Blueprint write back with it. The producers now narrow to
what they actually require, through one shared predicate, leaving the slot
check honest.
Separately, a stateful placement is held for operator review before
anything is deployed, so no target exists when that hold is recorded, and
the observation was dropped for want of one. The hold left no trace: no
history row, and nothing on the target to say the node had been asked to
hold anything. First contact now creates the target, exactly as the first
deploy does. Nothing else changes with it: no intent, no generation, no
operation, and connectivity stays unset, because a node that has only been
asked to hold something has not been contacted. Observations that follow a
deploy already have a target, so they are unaffected, and a drift or evict
report for a node nothing was ever sent to is still dropped.
* fix(git): keep the Git source panel's state after a save
The save answers with the source row and no revision, so the panel had
nothing to replace its copy with and dropped it. The result was a stack
that had just been reconfigured showing no GitOps state at all until the
panel was reopened, which reads as a stack the model knows nothing about
rather than one whose state has just moved.
It re-reads instead. Keeping the old copy was not an option either: a
material configuration change clears the staged candidate server side, so
the state genuinely has moved and the panel would have gone on offering a
commit that is no longer there. Only the server can say what replaced it.
The alternative was returning the revision on the save itself. That route
is proxied, and the identity hop rewrites node numbering for an enumerated
set of routes, so adding a revision to a response outside that set would
hand back a remote node's numbering unrewritten. Re-reading costs one
request on a low-frequency action and needs no change to the hop.
* docs(git): document what GitOps state means for an operator
A commit SHA says which files Sencho wrote. It does not say whether that
commit was accepted, whether something newer is waiting, or whether an
operation was interrupted halfway, and until now the docs had no words for
any of that.
Git Sources gains a source-state table and a section on what happens when
part of a state could not be proven, since a stack with missing evidence
looks identical to one with complete evidence unless the product says
otherwise. Drift Detection gains the gitops section and explains why it
answers a different question from the two signals above it: those compare
files and containers as they are now, this reports what has been proven
over time. The dashboard page describes the state chip and, importantly,
that it and the row tint are independent, so a healthy row carrying
"pending update" is not a contradiction.
The Blueprint page now separates its revision counter from GitOps state.
The counter labels the spec; it says nothing about whether a given node
has got there, and reading it as fleet truth is the mistake the note
prevents.
The tutorial's verification step names the source state it should show,
so a reader can tell the difference between files written and a commit
accepted.
Audited and left unchanged: health-gated updates and fleet federation.
Neither presents a SHA or a revision as canonical GitOps truth, so neither
needed correcting.
* docs(git): name which saves actually clear a staged commit
The note claimed saving the form always clears the staged commit and moves
the state to reconcile required. Only a change to what gets materialized
does that: the repository, the ref, the compose paths, the project
directory, or the env sync. Changing the token or the apply behavior
leaves a staged commit alone, because neither changes what would be
written, and telling an operator otherwise would have them pull again for
nothing.
* fix(git): say when a committed transition has nobody to announce it to
Two silences worth breaking, both found reviewing the error paths in this
slice.
A server that never installs the event sink still counts every transition
and still writes every history row, so the only symptom is that no client
ever refreshes and the UI is quietly as stale as it was before any of this
existed. It now says so, once rather than per row, since a boot migration
would otherwise fill the log with one fact.
The dashboard's source-state fetch dropped a non-ok response without a
word. A refusal there freezes every badge at its last value, which looks
exactly like a fleet where nothing has changed, so the status code is
logged rather than inferred from badges that stopped moving.
* fix(git): close the defects review found in the announcement slice
Five findings, two of them able to reach an operator.
The event sink was installed one line after the deletion reconcile, which
tombstones applications and targets and awaits inside its own loop. A
drain therefore landed while the sink was still absent, so those boot-time
transitions were counted and never announced, and the warning added for an
unwired sink would have fired on every boot with a prepared deletion
intent: the fastest way to teach an operator to ignore it when it means
something. The sink now precedes every reconcile and migration pass.
The first-placement target was created outside the transition that records
it. The observation runs in its own savepoint and can refuse, and the
caller deliberately lets the deployment commit whatever the record says,
so a refusal left an active target with no generation, no stage and no
history: a placement relationship the model never established, which the
delete path would later tombstone as if it were real. Both writes now
succeed or fail together.
The badge looked its status up in a map closed at compile time, behind two
casts, while the value arrives over a proxy from a node that may run a
newer vocabulary. An unmapped key dereferenced undefined inside a stack
row and would have taken the whole table down with it. The facet and its
status are now a discriminated union, the lookup is optional, and an
unrecognised status renders nothing, which is what the join already does
for a stack it has no state for.
The dashboard's source-state loop derived each row inside the shared
try, so one row in an unexpected shape abandoned the loop and froze every
badge at its last value with nothing on screen to say so. Each row is now
guarded on its own and the count of unreadable ones is logged.
Two tests claimed things they did not check: both asserted a word was on
screen with a matcher that also matches screen-reader-only text, so a
compact badge would have kept them green, and one compared a label by
containment where the shorter label is a prefix of the longer. Both now
assert the visible node by equality. Added coverage for the window event
the dashboard actually refetches on, an unknown status, a row that cannot
be read, node-switch blanking, and a caveat recorded twice. One fixture
built a status pair the deriver cannot emit and was corrected.
Also: the guard narrowing Blueprint producers to an active application
excludes nothing today, because every Blueprint-mode application is
inserted as active and only Direct mode can be creating. The guard stays,
since the getter's slot semantics and the transitions' requirement have
drifted apart once already, but its rationale and its tests now say
plainly that they pin a defensive guard rather than reachable behaviour.
* refactor(git): make three runtime guards visible to the compiler
Type review found the same shape three times: a correct guard whose
necessity was asserted in a comment the type system could not see. This
project does not set noUncheckedIndexedAccess, so each of these read as
dead code to anything that trusts the types, and the guards protect the
behaviours that matter most here.
The limitation copy map typed its values optional, so the fallback for an
unrecognised code is now something the compiler requires rather than
something a comment explains. The badge reads its status through partial
views of the two vocabularies, so a miss is a fact TypeScript derives:
deleting the guard that stops one unknown status taking down a stack list
now fails the build, where before it compiled. Neither needed a cast.
The event payload has to be a type alias rather than an interface, because
only the former gets an implicit index signature and the broadcaster takes
an open envelope. That requirement now fails at the declaration instead of
surfacing in the startup wiring as an unexplained index-signature error.
Also added the reverse of the copy-coverage check: a code retired from the
backend leaving stale wording behind was invisible, since the fallback
only fires for entries that are missing rather than ones that linger.
* fix(git): make the state cards survive an unknown status, and correct the copy
Comment review found the protection added for the dashboard chip was
claimed more widely than it held. Five other surfaces indexed the status
vocabularies directly and passed the result straight to a card that
dereferences it, so the exact input the comments said would take a list
down still threw in the Git source panel, the Drift tab, the sidebar
tooltip and the stack anatomy row. The card now takes an optional state
and renders nothing without one, which also means a new surface cannot
reintroduce the dereference by rendering a card the ordinary way, and the
remaining four read through the lookups.
Copy corrections, each against the site that emits it rather than the
name of the code:
The manifest-identity caveat said the manifest named a different
repository or branch. It also fires when the manifest has no identity
block at all, or identifies another node or another stack, so it now says
it does not identify this stack on this node from the repository
configured now. The last-known-good artifact caveat covered a mismatch but
not the row being gone, which is the other half of the same condition.
Doc corrections: a Blueprint-delivered stack was described as carrying the
dashboard chip and a source state on the Drift tab, and it carries
neither, because a Blueprint has no Git source of its own. Per-node
Blueprint state was attributed to the detail sheet's GitOps section, which
only reports what could not be read or proven; the Deployments table is
where per-node status lives. Clearing a staged commit was said to always
land on reconcile required, but a stack that has never had a commit
accepted lands on never reconciled. The caveat lines were described as
sitting under the state when they render above the form. And the pending
banner was still documented as webhook-only, which this branch changed:
it now appears for any staged commit and its heading is the source state.
Also corrected two claims in the code: the observation branch does not
skip for a Blueprint that migration brought in, because migration creates
no targets, and the only refusal reachable from that call site is a
tombstoned target.
* refactor(git): simplify the announcement slice without moving any behaviour
A simplification sweep over the new code, all of it behaviour-preserving
and re-checked by breaking each guard afterwards.
The drain now separates policy from mechanism: it decides whether a row
survived, counts it, and either warns or hands it to a small announce
helper that owns the envelope and its error handling. The row-to-state
pass in the dashboard hook lifts out of the fetch, so one function does
the request and another does the derivation. The caveat dedupe collapses
to a set, which preserves first-insertion order and so keeps the ordering
its own docstring promises. The badge renders one span with a conditional
class rather than two that differed only by class. The metrics service
drops an internal type that was byte-identical to the one it exports.
One assertion goes: the observation branch tested membership and then
re-asserted the same fact to index the stage map, and a type predicate
lets the compiler carry it instead.
Everything the sweep was told to leave alone is intact: the buffered
drain, the row-existence re-check, the injected sink, the one-shot
warning, the partial-view lookups and their guards, the optional copy map
with its fallback, and the generation counter with its per-row guard.
* fix(git): close audit findings in the GitOps revision model
Five corrections from the pre-merge audit of this branch.
Migration no longer certifies a commit it cannot prove. The manifest read
now carries its resolved commit, and a manifest that names a different
commit than the source row records as applied is refused: the applied
directory materializes one commit while the row names another, so
trusting them together would mint a generation pointing at files that are
not the ones it claims. A manifest that names no commit at all, which is
what adoption writes, is a separate answer rather than a disagreement,
because reporting it as one would name a commit the manifest does not
contain. Both are recorded as evidence and the projection asks for a
fetch.
Create cleanup proves containment against the real filesystem before it
deletes. Every check on these paths was lexical, so a symlink or Windows
junction above the target read as contained while the recursive delete
followed it out of the managed area. Resolution keeps "is not there"
apart from "could not be read": only a genuinely missing path lets the
walk climb to an ancestor, because treating an unreadable one as absent
would infer containment for the single path whose link status could not
be established. Both sides are resolved, so relocating the data directory
onto another volume keeps working, and an area that does not exist at all
is nothing to delete rather than a suspected escape. The marker write
takes the same barrier as the marker delete, so a link cannot be written
through and then refused on the way out, wedging the stack name.
Startup stops while an interrupted create is unresolved, matching the
restore reconcile above it. A create that could not be settled leaves a
stack directory the deploy path cannot tell apart from a finished one, so
the alternative to stopping is letting a scheduler, webhook or operator
act on a half-built stack. Only that blocks: once the staged directories
are gone the create is torn down, and a staging marker nobody could
unlink is reported rather than thrown, so one failed unlink is not the
difference between an instance that boots and one that does not.
Per-stack history no longer exposes a predecessor through a reused stack
name. The grant covers the application holding the name now, which is
what keeps a stack's own entries readable while it is still being
created; every other row on that name is classified per row, so entries
belonging to an application that held the name earlier need the audit
permission, as they already did across stacks. A detached predecessor is
still readable, which is the classifier's own standing decision about
detach rather than a gap here, and is now pinned by a test.
Blueprint observations reach a reader. The reconciler recorded state
review, evict blocked, drifted and correcting against the target and
nothing projected them, so a deployed Blueprint could report itself as
never applied. The runtime facet now projects all four, below the states
a live or failed mutation puts the target in and above the pointer
checks, and any later transition supersedes the observation. The map is
declared total over the stages the reconciler can record, so a stage
added without a projection fails the build rather than silently reading
as never applied again.
* fix(git): confine managed-area cleanup to each stack's own location
Cleanup proved only that a resolved path landed somewhere inside the
managed area. That is satisfied by every other stack and every other node
in it, so a link from one stack's generations directory into another's
passed the check while the delete took a generation belonging to someone
else. The check is now positional: the managed area is resolved once, so
relocating the data directory onto another volume still works, and every
segment below it must be reached without redirection. The create-path
sinks use it on the write as well as the delete, so a claim cannot be
written through a link and then refused by the hardened delete, and the
manifest service's pre-existing deletion sinks (generation pruning,
boot-sweep orphan reaping, detach staging and finalization, and
whole-area removal, including after a restored snapshot) run it before
their recursive deletes too. Each of those sinks also keeps a literal
containment comparison beside the positional check, because static
analysis credits only a comparison at the call and reports the delete
otherwise. A refusal names both paths in the log, because it can hold
the boot gate.
Boot recovery dropped the checkpoint of an application that had left the
creating state without first clearing its staging marker. That left a
claim on the stack name with nothing to retry it, and every later create
for that name was refused by a marker nothing could remove. The settled
branches now share one exit, so the marker ordering holds for all of them.
A detached application's history was readable on a stack grant because
its files are still the stack standing at that name. Nothing in these
tables can prove it still is: a Blueprint successor records the name off
the application row, and a plain Compose stack recreated at the name
leaves no trace at all, so no detection-based allowance can be made
sound. Detach now moves an application's trail to the audit audience
outright, the same answer deleted and creating predecessors get, while
reading the stack itself stays where it was.
* fix(gitops): close four audit findings in revision state and identity proxy
A1: derive runtime and health against target desired_generation_id
- deriveRuntime returns 'applied_not_deployed' when desired != deployed, so a
stale deployment stays deploy-pending instead of reading synced_and_healthy
- deriveHealth judges against desired_generation_id falling back to deployed
when null; control case proves null-desired rows unchanged
- collectRuntimeDrift emits the plan-pinned runtime drift item for exact/
qualified observation mismatches; equal/non-comparable observations emit
nothing; ordering pin keeps deploy question first
A2: stop upstream 304 responses from bypassing hub rewrite and reauthorization
- identity proxy answers no-store on every terminal (rewrite, 204, 304,
generated failures); validators (etag, last-modified, cache-control,
expires, vary) no longer forwarded
- conditional request headers (if-none-match, if-modified-since, if-match,
if-unmodified-since) stripped on the identity-hop branch of the shared
proxyReq handler; streaming hop untouched (optimistic-concurrency file writes
depend on If-Match/If-Unmodified-Since)
- supertest integration test drives the real middleware through a loopback
capture server and asserts if-none-match absent while accept survives
S1: migrate legacy query/userinfo URLs via secret-free path
- parseLegacyRepoUrl strips userinfo, query, fragment instead of refusing;
parseHttpsRepoUrl remains the strict gate for user-driven paths
- migrationDirectSourceIdentity uses the tolerant parser; operational
stack_git_sources.repo_url untouched; fingerprint convergence proven
- trusted-manifest-on-legacy-URL test covers the worst real-world instance
S2: emit runtime artifact drift for current evidence
- top-level drift array carries the seven-class runtime item; frontend
comments updated to reflect backend now emits runtime drift
All regression tests added and passing. CI green (7,374 backend + 2,790
frontend tests; only pre-existing Windows EBUSY teardown failure unrelated).
* test(frontend): fix DriftPanel drift item rendering test
Update test assertions to match the new backend drift item format.
The function now renders artifact_set expected
identities as 'artifact <id> · <qualification>' and runtime_artifact
observed identities as the raw identity string.
* fix(gitops): emit a runtime drift item for desired-versus-deployed mismatch
The projection reported applied_not_deployed with no entry in drift, so
the canonical drift list contradicted the runtime facet it travels with.
collectRuntimeDrift now emits the r27 generation-mismatch item for that
state: desired generation as expected, deployed generation as observed,
ComposeService as owner. The action mirrors what availableActions offers:
deploy, unless an application-level fetch or apply is in flight or a
recovery is in progress, which withhold deploying without removing the
fact of the mismatch. The item clears once the desired generation deploys.
The stale-deployment test pins the exact item shape, its stability across
re-derivation, its convergence removal, and the withheld-action case; the
ordering pin keeps the artifact observation suppressed while the deploy
question stands. Frontend fixture aligns its artifact-mismatch example
with the backend's none action.
* fix(gitops): keep generation drift visible across failure and pause states
The desired-versus-deployed drift item was keyed to the applied_not_deployed
runtime status, but paused, failed, recovering, interrupted, and in-flight
statuses all outrank the pointer comparison in the deriver. A failed deploy
of a new generation over a running older one therefore dropped the drift
report exactly while the old workload was still serving.
The item is now judged from the pointers themselves: known and different
means reported, whatever presentation status the target carries. Retired
targets are excluded because nothing can rebind them, so their surviving
pointer divergence would be permanently unresolvable noise. The action
follows what the payload offers: deploy only when no application-level
operation or recovery withholds it and availableActions lists deploying,
none otherwise.
The failed-redeploy regression test pins the exact item shape after a
pre-mutation deploy failure, its stability across re-derivation, continued
artifact suppression for the replaced workload, and convergence removal;
a second pin keeps retirement silent.
* fix(gitops): complete the available-action legality matrix
The action list transcribed only part of the approved rules. An
interrupted apply offered apply without checking that its recorded
generation was still the current candidate; deploy fired for any target
reading applied_not_deployed regardless of target mode, with no retry
for an interrupted Direct deploy and none of the Blueprint interruption
rules; approve_legacy sat in the type union with no producer, leaving a
pending Inline placement review permanently unactionable through the
projection; and drift items inherited whatever deploy recommendation any
sibling target earned.
Actions now follow the matrix. Deploy is decided per target through one
predicate shared by deriveActions and the drift items: a paused or
failed sibling can no longer inherit another target's legal deploy, a
Blueprint-mode divergence never advertises Direct deployment, an
interrupted Direct deploy retries against the generation still applied,
and an interrupted Blueprint deploy or withdraw repeats only while its
recorded intent revision and rollout candidate still equal what the
application requires, where an absent pair matches because inline
Blueprints carry no candidate until the later-phase producer lands.
Interrupted apply requires the recorded generation to still be the
current candidate and the source not to have been suspended meanwhile.
A reachable Inline placement review now offers approve_legacy.
Six tests pin the matrix: sibling isolation across failed and paused
targets, the Blueprint mode guard, vacuous and matched Blueprint retry
identities plus the superseded case, matching-versus-stale Direct deploy
and apply interruptions, and the legacy review action.
* fix(gitops): withhold apply retry while the candidate is blocked
The interrupted-apply retry checked that the recorded generation still
matched the current candidate and that the source was not suspended, but
a later classification can also block that candidate, and applyStarted
refuses a blocked one outright. The retry is offered only when every
precondition the transition enforces still holds.
* fix(gitops): prove apply preconditions and limit fetch to Direct
The interrupted-apply retry checked identity, suspension, and blockage
but never loaded the candidate generation, so apply could be recommended
for a row that was missing, owned by another application, or built from
a superseded materialization fingerprint; applyStarted refuses all
three, and the positive fixture itself described a state no transition
would accept. The gate now proves existence, ownership, and fingerprint
before offering apply. Its fixture is rebuilt around a transition-legal
candidate whose projected action is executed against applyStarted, with
negatives for every refusal including suspension.
Fetch was offered to any application whose source looked unreconciled,
but the approved rules reserve fetch for live Direct applications; a
Git-backed Blueprint divergence now advertises nothing of the sort.
Controls pin both modes against equivalent source state.
* fix(gitops): fail closed when a candidate row is missing or foreign
Ordinary candidate_ready checked a fingerprint only when the candidate
generation row existed, so a dangling id or one owned by another
application fell through to ready and offered an apply that
applyStarted would refuse on sight. The source now requires the row to
exist under this application before readiness, records a limitation
for the anomaly with its operator copy mirrored in the frontend, and
reports reconcile-required instead. The Blueprint retry test also pins
that a superseded rollout candidate alone, with the intent still
matching, suppresses deployment.
* fix(gitops): fail accepted-generation derivation closed
The acceptance branch compared fingerprint and sha only when the
accepted generation row loaded, so a dangling id or one owned by
another application fell through and reported
application_generation_accepted with no evidence behind it, while also
withholding fetch as the recovery. The source now requires the row to
exist under this application before any comparison, records an
accepted_generation_invalid limitation naming the pointer with its
operator copy and inventory entry mirrored on the frontend, and
reports reconcile-required instead. Five scenarios pin valid, missing,
foreign, fingerprint-mismatched, and sha-mismatched acceptances.
* fix(gitops): enforce canonical state in dismiss, reconcile, and pulls
- Route dismiss-pending through the canonical dismissed transition so a
refusal while an operation is in flight surfaces as 409 instead of
clearing staged state behind the model's back
- Treat tombstoned targets as authoritative in both reconciler decision
surfaces so ticks never redeploy onto a placement the model severed;
an explicit deploy revives the target and records the revival delta
- Keep application-level rows in node-scoped history pages so proxied
hub views do not read as if the application never came into being
- Expose repoIdentity and configuredRef on history items to match the
server-side identity filters
- Stand down when a pull resolves to exactly the live candidate (same
commit, source fingerprint, plan verdict); pulls after an acceptance
still open a fresh staging generation as the apply target
* fix(rbac): classify git-source manifest reads as stack:read
GET /stacks/:name/git-source/manifest had no rule in the hub route
classifier, so proxied requests were refused with 403 before reaching
the remote node even for callers holding stack read access.
* fix(gitops): surface an error toast when dismiss is refused as in-flight
Dismissing a pending Git update while a fetch or apply is still running
now returns 409 OPERATION_IN_FLIGHT, but the frontend handler had no
else branch for a non-2xx response, so the refusal was swallowed with
no operator feedback at all. Add the same error-toast pattern already
used by the sibling apply handler, and order the success toast before
its side effects so a downstream failure cannot invert the outcome.
410 lines
42 KiB
Plaintext
410 lines
42 KiB
Plaintext
---
|
|
title: "Blueprints"
|
|
description: "Fleet-wide compose templates that Sencho keeps in sync across the nodes you choose, with stateless or stateful classification, drift detection, and three response modes."
|
|
---
|
|
|
|
A **Blueprint** bundles Compose YAML with a node selector and a drift policy. Every minute Sencho compares the live state on each targeted node against the blueprint's desired revision, and either reports the drift, dispatches a notification, or auto-corrects, depending on the mode you picked. You declare a stack once; Sencho handles the distribution and the reconciliation loop.
|
|
|
|
Blueprints live under **Fleet · Deployments**.
|
|
|
|
<Note>
|
|
Blueprints are Sencho's Compose-first orchestration model. Creating, editing, withdrawing, and applying blueprints requires an admin role; operators and viewers can read the catalog and the detail sheet. Pinning a blueprint to a specific node lives under **Fleet · Federation** (admin role required).
|
|
</Note>
|
|
|
|
<Frame caption="Fleet · Deployments catalog with blueprint tiles, the All / Drifted / Observe / Suggest / Enforce filter chips, and the New Blueprint action in the top-right.">
|
|
<img src="/images/blueprint-model/catalog.png" alt="Blueprint catalog in Fleet Deployments" />
|
|
</Frame>
|
|
|
|
## Mental model
|
|
|
|
Three moving parts cooperate per blueprint.
|
|
|
|
1. **The declared spec.** A blueprint is a row in Sencho's database. It carries the compose YAML, the selector (labels or explicit node IDs), the drift policy, and a monotonic revision number. The revision auto-increments every time the compose changes.
|
|
2. **The reconciler.** A background loop on the controlling Sencho ticks every 60 seconds (with a 5-second initial delay after startup) and on demand after **Confirm Apply**. Each tick only mutates the fleet when the blueprint's current operational intent is approved. The tick resolves the selector, compares every desired-vs-live node, and runs only the place or remove outcomes the operator already confirmed.
|
|
3. **The executor.** Per-node deploy and withdraw run against the local Docker socket on local nodes, and through the authenticated proxy to `/api/blueprints/apply-local` and `/api/blueprints/withdraw-local` on remote nodes (older remotes fail closed until upgraded). Every blueprint deployment writes a `.blueprint.json` marker into the stack directory; the marker carries the blueprint ID, the revision, and the last-applied timestamp.
|
|
|
|
The marker is the trust root. If a directory by the blueprint's name already exists on a node and does not carry a matching marker, the reconciler refuses to touch it and surfaces a **Name conflict** on the deployment row. A Blueprint named `nginx` will never overwrite an existing user-authored `nginx` stack on any node.
|
|
|
|
<Note>
|
|
The revision number counts compose edits. It is a label on the spec, not a statement about the fleet: two nodes can both sit at revision 7 with one of them still deploying it. What Sencho has established per node is reported separately: the **Deployments** table on the detail sheet for each node's status, and the stack's Drift tab for its GitOps state. Where the two seem to disagree, the established state is the one that has been proven. The **GitOps** section of the detail sheet is narrower than either: it appears only when something about this Blueprint's own state could not be read or could not be proven.
|
|
</Note>
|
|
|
|
Statelessness vs statefulness is decided at author time by parsing the compose file. Stateless blueprints (no persistent volumes, or only `tmpfs` mounts) deploy and evict freely. Stateful blueprints (named volumes or bind mounts) get explicit operator-confirmation prompts on the first deploy to a fresh node and on eviction from any node. Blueprints with `external: true` volumes are classified as **unknown** and treated as stateful for safety.
|
|
|
|
Drift detection runs on every tick for every Active deployment regardless of policy. The policy only governs what Sencho does next: surface the drift silently, notify, or auto-redeploy.
|
|
|
|
## Key capabilities
|
|
|
|
**One declaration covers many nodes.** Pick nodes by label or by node ID. The selector set is re-resolved on every reconciliation tick. When a new node matches after a label change, the catalog shows **reapproval required** until you confirm the updated blast radius. Removing a label, removing a node, or shrinking the selector likewise needs confirmation before withdraw outcomes run (subject to the stateful-eviction safety rail).
|
|
|
|
**Drift detection always on.** Each tick compares the marker's revision against the live containers and the blueprint's current revision, then checks that all containers labeled with the compose project name are running. Drift is recorded on the deployment row whatever the policy is. **Observe** records it silently, **Suggest** also dispatches a notification, **Enforce** also redeploys.
|
|
|
|
**Stateful safety rails.** Stateful and state-unknown blueprints enter **Awaiting confirmation** on every fresh node before the first deploy, and **Evict blocked** when a node falls out of the selector while still hosting a stateful deployment. The reconciler never deploys empty volumes or destroys named volumes without a human acknowledging the action.
|
|
|
|
**Pin override and cordon respect.** Admins can pin a blueprint to a single node from **Fleet · Federation**. A pin replaces the selector entirely, deploys only to the pinned node, and overrides the cordon flag on that node. Cordoning a node otherwise prevents the reconciler from picking it for new placements; existing deployments on a cordoned node keep running and stay drift-checked.
|
|
|
|
**Vulnerability-policy participation.** Local blueprint deploys evaluate against the same pre-deploy policy gate that the per-stack deploy lane uses. If an enabled policy blocks one of the blueprint's image references, the deployment row moves to **Failed** and the stack is never written to disk. Remote blueprint deploys are routed through the remote node's stack deploy endpoint, so policy enforcement runs on the remote instance with that node's credentials and scanner state.
|
|
|
|
**Per-node, per-revision marker ownership.** Every blueprint deployment writes a `.blueprint.json` marker carrying the blueprint ID, revision, and last-applied timestamp. The reconciler refuses to touch any directory whose marker does not match its blueprint ID, so user-authored stacks and other blueprints can coexist on the same node without collision.
|
|
|
|
## Prerequisites
|
|
|
|
| Requirement | Detail |
|
|
|---|---|
|
|
| User role | **Admin** to create, edit, withdraw, accept, and apply. Operators and viewers can read the catalog and the detail sheet. Pinning requires admin. |
|
|
| Nodes | At least one node that the selector resolves to. Remote nodes need a healthy proxy connection; see [Multi-node management](/features/multi-node) and [Pilot Agent](/features/pilot-agent) for enrollment. |
|
|
| Compose YAML | Valid Compose YAML, 96 KiB or fewer. |
|
|
| Blueprint name | 1 to 64 characters matching `^[a-z0-9][a-z0-9_-]*$`. The name doubles as the stack directory on every targeted node. Rename is blocked while any non-withdrawn deployment or guard exists. |
|
|
| Selector | A `labels` or `nodes` selector with up to 200 entries per side. An empty resolved set is allowed but produces no deployments. |
|
|
| Compose directory | Per-node compose directory must be writable. Remote nodes must accept the controlling instance's bearer token; this is the same channel the rest of the fleet management already uses. |
|
|
|
|
## The catalog
|
|
|
|
The Deployments tab opens on a catalog of every blueprint configured on this instance. Each tile shows the name, an optional description, the classification chip (stateless, stateful, or unknown), the active-vs-targeted node count, and the current drift mode. Filter chips at the top break the catalog down by drift mode and let you isolate any tile that has drift.
|
|
|
|
<Frame caption="Catalog tiles. Each tile carries the classification chip, the active-over-targeted count, the selector summary, and the drift mode. The filter chips above the grid pivot the view.">
|
|
<img src="/images/blueprint-model/catalog.png" alt="Deployments tab catalog with classification chips and filter chips" />
|
|
</Frame>
|
|
|
|
The first time you visit the tab, the catalog is empty and a three-step explainer walks you through Author, Target, and Reconcile, with a **Create your first Blueprint** button at the bottom.
|
|
|
|
<Frame caption="Empty Deployments tab. Three numbered cards (01 Author, 02 Target, 03 Reconcile) explain the model before you create the first blueprint.">
|
|
<img src="/images/blueprint-model/empty-state.png" alt="Empty Deployments tab with three-step explainer" />
|
|
</Frame>
|
|
|
|
## Anatomy of a Blueprint
|
|
|
|
| Field | Purpose |
|
|
|---|---|
|
|
| **Name** | Used as the stack directory on every targeted node (`<COMPOSE_DIR>/<blueprint-name>/`). Lowercase letters, digits, hyphens, and underscores only. Rename is allowed only when every deployment is withdrawn (or there are none); live placements and guards block rename. |
|
|
| **Description** | Short prose for the catalog tile and the detail header. |
|
|
| **Compose** | Standard Compose YAML. The same content ships to every targeted node as `compose.yaml`. Sencho parses it on save and classifies the blueprint as stateless, stateful, or unknown. The YAML must parse successfully and stay under 96 KiB. |
|
|
| **Selector** | Either label expressions (any-of plus all-of) or a list of node IDs picked by hand. |
|
|
| **Drift policy** | Observe, Suggest, or Enforce. Drift detection always runs; only the response differs. |
|
|
| **Reconciler enabled** | Toggle the reconciliation loop without deleting the blueprint. The **Disable** button on the detail sheet flips this toggle off; **Enable** turns it back on. Useful when you want to pause auto-deploys without losing the configuration. |
|
|
|
|
Sencho writes a `.blueprint.json` marker into each targeted node's stack directory. The marker carries the blueprint ID, revision, and the timestamp of the last apply. The reconciler refuses to touch any directory that does not carry a matching marker, so a Blueprint named `nginx` will never overwrite an existing user-authored `nginx` stack on any node. When the directory exists without a matching marker, the deployment row enters the **Name conflict** status until you rename one of the two.
|
|
|
|
Before a local Blueprint deploy starts, Sencho applies the same pre-deploy [vulnerability scan](/features/vulnerability-scanning) policy gate used by standard stack deploys. If an enabled policy blocks one of the Blueprint's image references, the deployment row moves to **Failed** and the stack is not written to disk. Successful local Blueprint deploys also trigger the normal post-deploy scan. Remote Blueprint deploys are routed through the remote node's stack deploy endpoint, so policy enforcement runs on the remote instance with that node's credentials and scanner state.
|
|
|
|
## The editor
|
|
|
|
**New Blueprint** opens a single editor dialog with everything inline: name and description fields, a Monaco YAML editor with the classification banner sitting above it, the selector picker (Labels or Specific nodes), the drift policy cards, and the Reconciler enabled toggle.
|
|
|
|
<Frame caption="The Blueprint editor. Name and description fields, the classification banner pinned above the compose editor, and the Specific nodes selector below. Drift policy cards and the Reconciler enabled toggle sit further down in the same scrollable dialog.">
|
|
<img src="/images/blueprint-model/editor-dialog.png" alt="Blueprint editor dialog with name, description, classification banner, compose editor, and node selector" />
|
|
</Frame>
|
|
|
|
### Classification banner
|
|
|
|
The banner above the YAML editor shows what Sencho found when it parsed your compose file:
|
|
|
|
- **Stateless · portable** when no persistent volumes were detected, or only `tmpfs` mounts. Sencho can deploy and evict the blueprint freely on any node.
|
|
- **Stateful · pins to data** when named volumes or bind mounts were detected. Each node holds its own data; Sencho does not replicate volumes between nodes.
|
|
- **State unknown** when `external: true` volumes were detected. Sencho cannot prove portability and treats the blueprint as stateful for safety.
|
|
|
|
Click the signals toggle on the right of the banner (`1 SIGNAL`, `2 SIGNALS`, etc.) to expand the list of reasons Sencho classified the way it did, line by line.
|
|
|
|
<Frame caption="The classification banner expanded. The signals list shows the exact rule that fired for each detected mount, volume, or external reference.">
|
|
<img src="/images/blueprint-model/classification-banner.png" alt="Classification banner in the Stateful · pins to data variant with two signals expanded" />
|
|
</Frame>
|
|
|
|
### Selector
|
|
|
|
A **Labels** selector matches any node whose labels satisfy the expression. The picker is split into two pill rows: **Match nodes with ANY of these labels** (the OR set), and **AND ALSO require ALL of these** (the AND set). Either side may be empty. The dialog renders a one-line summary just below the pills, so you can see exactly which nodes the expression resolves to before you save.
|
|
|
|
A **Specific nodes** selector picks nodes by ID with a checkbox list. Use it when you want a one-off blueprint that runs only on a known node, or when no labels exist yet.
|
|
|
|
Add labels to nodes from **Settings · Infrastructure · Nodes**. Each node row carries a Labels column with a `+` button.
|
|
|
|
<Frame caption="Settings · Infrastructure · Nodes. The Labels column carries pills per node; the `+` button opens an inline add-label popover.">
|
|
<img src="/images/blueprint-model/node-labels.png" alt="Settings Infrastructure Nodes page with the Labels column and add-label popover" />
|
|
</Frame>
|
|
|
|
### Drift policy
|
|
|
|
The three policy cards in the editor map exactly to the three modes the reconciler runs against every tick:
|
|
|
|
| Card label | Reconciler behavior |
|
|
|---|---|
|
|
| **Observe** · Detect & display, no notifications | Drift surfaces in the deployment table; no notification, no auto-fix. |
|
|
| **Suggest** · Detect & notify, operator decides | Sencho dispatches a `blueprint_drift_detected` notification through your notification routes, if any. |
|
|
| **Enforce** · Detect & auto-fix, silent on success | Sencho re-deploys the blueprint silently when drift is detected. A `blueprint_drift_correction_failed` notification fires only when the auto-fix attempt fails. |
|
|
|
|
Even **Observe** keeps Sencho honest about what it found: the deployment row shows "drifted 3h ago: service caddy exited code 1". Silence would forfeit Sencho's authority over your fleet.
|
|
|
|
Enforce's auto-fix only ever redeploys the blueprint's *current* revision to correct container-level drift (a stopped or crashed container, for example). A compose edit never reaches that auto-fix path on a stateful or state-unknown blueprint: any revision change re-enters **Awaiting confirmation** on every targeted node first, regardless of drift mode, so a human always signs off before a new revision touches existing data. See [Lifecycle: status transitions](#lifecycle-status-transitions).
|
|
|
|
## The detail sheet
|
|
|
|
Click any tile in the catalog to open the blueprint detail sheet on the right. The sheet header shows the name, the meta line (`<selector> · <drift mode> · rev <n>`), the action bar (Apply now, Edit, Disable, Delete), the description, the deployment table, and a collapsible compose source.
|
|
|
|
<Frame caption="The detail sheet. Header carries the selector and revision; the action bar offers Apply now, Edit, Disable, and Delete; the deployment table lists one row per resolved node.">
|
|
<img src="/images/blueprint-model/detail-sheet.png" alt="Blueprint detail sheet showing header, action bar, and deployment table" />
|
|
</Frame>
|
|
|
|
The deployment table lists every node currently in the blueprint's resolved selector set. Each row carries the node name, the deployment status, the last-activity timestamp, a notes column (drift summary or the last error message), and the action available for that row.
|
|
|
|
Possible status values:
|
|
|
|
| Status | Meaning |
|
|
|---|---|
|
|
| **Pending** | The reconciler has acquired the deployment lock and is about to start. |
|
|
| **Awaiting confirmation** | A stateful blueprint reached a new node. The reconciler refuses to deploy until you click **Confirm deploy** in the action column. |
|
|
| **Deploying** | The compose action is in flight on this node. |
|
|
| **Active** | The container set matches the blueprint's revision. |
|
|
| **Drifted** | The reconciler detected a difference between the live state and the desired one. |
|
|
| **Correcting** | An Enforce-mode auto-fix is in flight. |
|
|
| **Failed** | The deployment or withdrawal errored. The notes column carries the message. |
|
|
| **Withdrawing** | A withdraw is in flight. |
|
|
| **Withdrawn** | The deployment was successfully removed; the row is about to be cleared. |
|
|
| **Evict blocked** | A node left the selector while a stateful deployment was active. The reconciler refuses to evict until you choose an explicit mode. |
|
|
| **Name conflict** | A directory by the blueprint's name already exists on the node and does not carry a matching `.blueprint.json` (missing, malformed, or another blueprint ID). The reconciler will not touch it. |
|
|
|
|
<Frame caption="A deployment row in Awaiting confirmation. The Confirm deploy action sits at the right; the notes column explains why the reconciler is waiting on the operator.">
|
|
<img src="/images/blueprint-model/detail-state-review.png" alt="Blueprint detail sheet with a deployment row in Awaiting confirmation state" />
|
|
</Frame>
|
|
|
|
## Lifecycle: status transitions
|
|
|
|
The common paths through the status enum.
|
|
|
|
- **Stateless first deploy.** No row → **Pending** → **Deploying** → **Active**. Drift detected later on Enforce flips to **Drifted** → **Correcting** → **Active**.
|
|
- **Stateful first deploy.** No row → **Awaiting confirmation**. Operator clicks **Confirm deploy** and picks **Deploy fresh** → **Deploying** → **Active**.
|
|
- **Compose change on a stateless deployment.** **Active** → **Deploying** on the next tick → **Active**.
|
|
- **Compose change on a stateful or unknown deployment.** **Active** → **Awaiting confirmation** on every targeted node so the operator can decide whether the new revision is safe for that node's local data. Operator confirms → **Deploying** → **Active**.
|
|
- **Node leaves a stateless selector.** **Active** → **Withdrawing** → **Withdrawn** and the row clears.
|
|
- **Node leaves a stateful selector.** **Active** → **Evict blocked**. Operator chooses **Snapshot, then evict** or **Evict and destroy data** → **Withdrawing** → **Withdrawn**.
|
|
- **Pre-existing stack collision.** New row → **Name conflict**. No further action without operator rename of the user-authored stack or of the blueprint itself.
|
|
- **Deploy or withdraw error.** Any in-flight status → **Failed** with the message in the notes column. The row stays put until the next tick or until the operator clicks **Apply now** to retry.
|
|
|
|
## Working with Blueprints
|
|
|
|
### Create
|
|
|
|
1. Go to **Fleet · Deployments**.
|
|
2. Click **New Blueprint**.
|
|
3. Fill in the name, description, compose YAML, selector, and drift policy.
|
|
4. Watch the classification banner update as you type. It tells you whether the blueprint is portable or pinned to data.
|
|
5. Click **Create blueprint**. The blueprint starts with approval **pending**. No fleet mutation runs until you confirm a rollout preview.
|
|
|
|
If the YAML is malformed or larger than 96 KiB, Sencho rejects the save before creating a Blueprint row.
|
|
|
|
### Apply on demand
|
|
|
|
The reconciler runs every minute, but only for blueprints whose current operational intent is **approved**. To authorize or re-authorize a rollout, click **Apply now** on the detail sheet (or **Retry** on a failed row). Sencho opens a rollout preview that lists place and remove outcomes, requirements, and health notes. **Confirm Apply** stores that approval and runs the confirmed executor actions. If some nodes fail (for example a remote host is unreachable) or an unmanaged same-name stack blocks placement, Confirm still records approval, but the response and toast report those per-node outcomes instead of a clean success. Changing the compose, selector, name, pin, drift mode, or enabled flag clears approval back to pending. Catalog tiles show **pending** or **reapproval required** when confirmation is needed again (for example after a label change adds a new target node).
|
|
|
|
Raw `POST /api/blueprints/:id/apply` clients must send `{ planFingerprint, actions }` from a fresh `GET .../preview`. A body without that confirmation is rejected.
|
|
|
|
### Edit
|
|
|
|
Click **Edit** on the detail sheet. Editing the compose bumps the revision. Stateless blueprints redeploy on the next reconciliation tick. Stateful and state-unknown blueprints re-enter **Awaiting confirmation** on every targeted node so the operator can decide whether the new revision is safe for each node's local data, whether or not the change actually touches a volume.
|
|
|
|
### Confirm a stateful first deploy
|
|
|
|
The first time a stateful blueprint reaches a node, the deployment row enters **Awaiting confirmation**. Click **Confirm deploy** in the action column to open the state review dialog. **Deploy fresh** creates empty named volumes on the target node and starts the stack with the default state from each container image. **Restore from snapshot** is reserved for the future Volume Migration feature and is currently disabled.
|
|
|
|
<Frame caption="State review dialog. Deploy fresh creates empty named volumes on the target node; Restore from snapshot is reserved for the future Volume Migration feature.">
|
|
<img src="/images/blueprint-model/state-review-dialog.png" alt="State review dialog with Deploy fresh enabled and Restore from snapshot disabled" />
|
|
</Frame>
|
|
|
|
### Withdraw a single deployment
|
|
|
|
In the deployment table, click **Withdraw** on the node's row. For stateless blueprints, Sencho confirms once and runs `docker compose down` plus a directory removal. For stateful blueprints, the eviction dialog asks how to handle the data on this node:
|
|
|
|
- **Snapshot, then evict (recommended)** captures the blueprint's compose definition into [Fleet · Snapshots](/features/fleet-backups) before running `docker compose down`. The volume bytes still leave the node when compose tears down the named volumes; the snapshot only preserves the YAML so you can redeploy it elsewhere.
|
|
- **Evict and destroy data** runs the eviction without a snapshot. Type the blueprint name to confirm.
|
|
- Manual **Evict and destroy** / **Snapshot, then evict** require a current approved remove outcome for that node, and the node must no longer be desired. Confirm a remove rollout after the node leaves the selector first; destructive Evict while the node is still desired, or without remove approval, returns a stale-guard error. Plain stateless **Withdraw** does not require remove approval.
|
|
|
|
<Frame caption="Stateful eviction dialog. Snapshot, then evict captures the compose YAML to Fleet · Snapshots; Evict and destroy data requires typing the blueprint name to confirm.">
|
|
<img src="/images/blueprint-model/eviction-dialog.png" alt="Stateful eviction dialog with Snapshot then evict and Evict and destroy data options" />
|
|
</Frame>
|
|
|
|
<Frame caption="Stateless eviction dialog. A single Withdraw deployment action; no data prompt because nothing persistent was detected.">
|
|
<img src="/images/blueprint-model/eviction-stateless.png" alt="Stateless eviction dialog with a single Withdraw deployment action" />
|
|
</Frame>
|
|
|
|
<Note>
|
|
**Snapshot, then evict** captures the compose definition only. Volume bytes are not shipped. The named volumes managed by this stack on the target node are removed by `docker compose down` just as with **Evict and destroy data**. To preserve data, capture volumes manually before withdrawing (see *Migrating stateful data between nodes* below).
|
|
</Note>
|
|
|
|
### Delete the blueprint
|
|
|
|
Click **Delete** on the detail sheet, then type the blueprint's name to confirm. Stateless deployments, and any deployment that never made it past **Awaiting confirmation**, are withdrawn for you as part of the delete. A stateful or state-unknown blueprint with a deployment that is live on a node refuses to delete; withdraw that deployment from the deployment table first, so you choose the snapshot-or-destroy path for its data, then delete the blueprint.
|
|
|
|
## Federation: pin a blueprint to a single node
|
|
|
|
Admins can pin a blueprint to a specific node from **Fleet · Federation**. A pinned blueprint deploys only to its pinned node, regardless of the configured selector, and overrides the cordon flag on that node. The Blueprint detail sheet shows a read-only `Pin` section when a pin is in place; pin management itself lives in the Federation tab.
|
|
|
|
<Frame caption="Fleet · Federation, Blueprints subsection. Each row shows the blueprint, its configured selector, the Pinned to dropdown, and the effective placement that the reconciler will use.">
|
|
<img src="/images/blueprint-model/federation-pin.png" alt="Federation tab pin policy table with one blueprint pinned and one unpinned" />
|
|
</Frame>
|
|
|
|
See [Fleet Federation](/features/fleet-federation) for the cordon and pin model.
|
|
|
|
## Cordoned nodes
|
|
|
|
Cordoning a node prevents the reconciler from selecting it for **new** placements and skips state-review provisioning on it. Existing deployments on a cordoned node keep running, and drift checks continue normally. Pin policy overrides cordon, so a blueprint pinned to a cordoned node still deploys.
|
|
|
|
## Notifications
|
|
|
|
Two notification events fire from the reconciler:
|
|
|
|
- `blueprint_drift_detected` (warning severity), in Observe and Suggest modes when drift is detected. In Observe the row updates silently in the table; in Suggest the notification is also dispatched through your notification routes.
|
|
- `blueprint_drift_correction_failed` (error severity), in Enforce mode when an auto-fix attempt fails.
|
|
|
|
Both events route through the standard alert pipeline. Configure delivery channels under [Notification routing](/features/alerts-notifications#notification-routing).
|
|
|
|
## Security and trust boundaries
|
|
|
|
**Who can do what.** The user role determines the available actions. Reading the catalog, the detail sheet, and the deployment status is open to every role. Creating, editing, withdrawing, accepting a stateful deploy, and applying on demand require the admin role. Pinning a blueprint requires the admin role.
|
|
|
|
**The marker file is the trust root.** The reconciler will only deploy into, modify, or withdraw a directory that carries a `.blueprint.json` marker whose blueprint ID matches. A pre-existing directory with no marker, or a marker referencing a different blueprint, surfaces as **Name conflict** and is never modified.
|
|
|
|
**Local vs remote policy enforcement.** Local blueprint deploys evaluate the pre-deploy vulnerability policy gate against the local scanner state and credentials before the compose file is written to disk. Remote blueprint deploys are routed through the remote node's standard `/api/stacks` and `/api/stacks/<name>/deploy` endpoints over the proxy, so the remote node enforces its own policy with its own scanner state. The controlling instance does not bypass remote policy.
|
|
|
|
**Remote-node call path.** Sencho's controlling instance reaches remote nodes through the same authenticated proxy used by the rest of the fleet management surfaces, carrying the remote node's bearer token. Remote nodes do not need to accept any inbound connection beyond the one they already accept for fleet operations.
|
|
|
|
**Audit.** Blueprint create, edit, delete, pin, withdraw, and accept are administrative actions and are recorded through the standard [Audit log](/features/audit-log) pipeline along with the rest of the admin surface.
|
|
|
|
## Limitations and non-goals
|
|
|
|
By design, Blueprints do not include:
|
|
|
|
- A distributed storage layer (no CSI, no Longhorn-style replication).
|
|
- Automatic volume migration between nodes.
|
|
- Per-node parameter overrides or templating (one compose, all nodes).
|
|
- Staged or canary rollouts.
|
|
- Versioning history with one-click rollback. To revert, paste the prior compose into the editor and save.
|
|
|
|
Concrete operational constraints:
|
|
|
|
- **Reconciler cadence.** The tick interval is 60 seconds (5-second initial delay after startup). Approved blueprints reconcile on that cadence. Use **Apply now** and **Confirm Apply** after a change that clears approval.
|
|
- **Compose size.** YAML must be 96 KiB or fewer. Split very large compose files into smaller blueprints, or move generated content out of the compose body.
|
|
- **Selector size.** A selector accepts up to 200 entries per side (200 `nodes.ids`, or up to 200 each in `labels.any` and `labels.all`).
|
|
- **Name.** 1 to 64 characters matching `^[a-z0-9][a-z0-9_-]*$`. Rename is allowed only when every deployment is withdrawn (or there are none). Live placements and guards must be resolved first; rename clears approval back to pending.
|
|
- **Snapshot semantics.** **Snapshot, then evict** captures the compose YAML only. Volume bytes are removed by `docker compose down` just as with **Evict and destroy data**.
|
|
- **Restore from snapshot.** Reserved for the future Volume Migration feature and currently disabled in the state review dialog.
|
|
|
|
These omissions keep Blueprints honest: a compose-native fleet primitive that distributes the file you already have to the nodes you choose.
|
|
|
|
## Migrating stateful data between nodes (manual)
|
|
|
|
Sencho's compose-native lane does not include automatic volume shipping. **Snapshot, then evict** is a compose-only safety net: it preserves the YAML so you can redeploy elsewhere, but it does not move data. To relocate a stateful Blueprint's data from node A to node B, do it by hand before withdrawing:
|
|
|
|
1. Stop the Blueprint deployment on node A from the deployment table. Use **Snapshot, then evict** so the compose YAML is parked in [Fleet · Snapshots](/features/fleet-backups) while you handle volumes.
|
|
2. Use your host tooling (`docker run --rm -v <volume>:/data busybox tar -czf - /data > snapshot.tar.gz`, or app-aware tooling such as `pg_basebackup`, `mysqldump`, or `mongodump`) to capture the volume on node A.
|
|
3. Transfer the artifact to node B and restore it into the named volume there.
|
|
4. Update the Blueprint's selector to include node B; click **Apply now**, review the preview, and **Confirm Apply**.
|
|
|
|
A future Volume Migration feature will automate this with app-aware backup tooling.
|
|
|
|
## Practical workflows
|
|
|
|
**Multi-node identical reverse proxy.** A stateless `caddy-edge` blueprint with a labels selector matching `any=[edge]` and drift mode **Enforce**. Adding a new edge node deploys the proxy automatically within one tick. Editing the compose to bump the Caddy image redeploys every edge node on the next tick without operator intervention. Drift caused by a manual `docker compose down` on one edge node is corrected silently within a minute.
|
|
|
|
**Single-node managed Postgres.** A stateful `pg-fleet` blueprint with a `nodes` selector pointing at one database node and drift mode **Suggest**. The first deploy enters **Awaiting confirmation** so the operator chooses **Deploy fresh**. Subsequent compose changes (image bump, config change) re-enter **Awaiting confirmation** on the same node so the operator can decide whether the new revision is safe for the existing volume. Drift on the running container fires a `blueprint_drift_detected` notification but never auto-redeploys.
|
|
|
|
**Pin a blueprint to a specific node despite the selector.** Open **Fleet · Federation**, find the blueprint in the pin policy table, and pick the target node from the **Pinned to** dropdown. The pin saves immediately but also clears approval, so nothing moves until you open the blueprint here in Deployments, click **Apply now**, and **Confirm Apply**. Once confirmed, the pin overrides the selector for that blueprint, deploys only to the pinned node, and also overrides cordon on that node. Useful for relocating a stateful service to a specific host without rewriting the selector. Clear the pin (and confirm again) to restore selector-driven placement.
|
|
|
|
**Observe-only audit blueprint.** A stateless monitoring stack (Vector, Promtail, a Prometheus exporter) with drift mode **Observe**. Drift is recorded silently in the deployment table; no notification fires and no auto-fix runs. Useful when you want Sencho to track placement and detect divergence on a low-signal stack without paging anyone.
|
|
|
|
## Troubleshooting
|
|
|
|
<AccordionGroup>
|
|
<Accordion title="A deployment row shows 'Name conflict' and refuses to deploy">
|
|
A directory by the blueprint's name already exists on that node without a matching `.blueprint.json` marker (missing, malformed, or owned by another blueprint). The most likely cause is a manually created stack with the same name. Resolution: rename either the existing stack or the blueprint, then click **Apply now** on the detail sheet to retry.
|
|
</Accordion>
|
|
<Accordion title="A stateful deployment is stuck in 'Awaiting confirmation'">
|
|
The reconciler will not auto-deploy a stateful blueprint to a node it has never run on. Click **Confirm deploy** on the row, then choose **Deploy fresh** in the dialog. Sencho will create empty named volumes and start the stack.
|
|
</Accordion>
|
|
<Accordion title="A deploy was blocked by vulnerability policy">
|
|
Open the **Security** page and review the active scan policies for the target node on the **Policies** tab. The Blueprint deployment row records the blocking policy and affected image count. Either fix the image, relax the policy, or deploy through an explicit admin-approved bypass on the stack surface. Blueprints do not silently bypass deploy enforcement.
|
|
</Accordion>
|
|
<Accordion title="Compose content was rejected on save">
|
|
Sencho accepts valid YAML up to 96 KiB. Split very large compose files into smaller blueprints or move generated content out of the Blueprint. Do not paste secrets into the compose body; use environment files or [Fleet Secrets](/features/fleet-secrets) where appropriate.
|
|
</Accordion>
|
|
<Accordion title="A remote node is offline or disconnected during apply">
|
|
The row moves to **Failed** with the remote error. Reconnect the node (see [Pilot Agent](/features/pilot-agent) and [Multi-node management](/features/multi-node)), verify whether the stack directory contains `compose.yaml` and `.blueprint.json`, then click **Apply now**. If the directory exists without a matching marker, Sencho treats it as a name conflict until you rename or remove the remote stack manually.
|
|
</Accordion>
|
|
<Accordion title="Remote apply or withdraw fails asking for an upgrade">
|
|
Blueprint apply and withdraw on a remote node require that node's Sencho build to expose the atomic node-local endpoints (`/api/blueprints/apply-local` and `/api/blueprints/withdraw-local`). Older remotes refuse the mutation with a failed deployment row rather than writing files without ownership checks. Upgrade the remote instance to a matching Sencho release, then retry **Apply now** or withdraw.
|
|
</Accordion>
|
|
<Accordion title="A Docker daemon or registry failure surfaced during apply">
|
|
The deployment row moves to **Failed** and records the Docker or registry error. Resolve the daemon, socket, registry credentials, rate limit, disk, or volume-permission issue on the affected node, then click **Apply now**. Watch that node's stack activity and security scan status after retry.
|
|
</Accordion>
|
|
<Accordion title="Drift is reported but never gets corrected">
|
|
Confirm the drift policy is **Enforce** and that **Reconciler enabled** is on. Open the detail sheet to check the row's status and the most recent drift summary. If the row is sitting in **Awaiting confirmation** instead of **Drifted**, that is expected: a compose edit on a stateful or state-unknown blueprint never auto-corrects under any drift mode, including Enforce. It always waits for an explicit **Confirm deploy** so a human signs off before the new revision touches that node's data. Enforce's auto-fix only applies to container-level drift (a stopped or crashed container) on a deployment that is already **Active**.
|
|
</Accordion>
|
|
<Accordion title="'Apply now' is greyed out">
|
|
**Apply now** requires the blueprint to be enabled. Open the detail sheet, click **Enable** to flip the reconciler back on, then click **Apply now**. The button is also disabled while the detail sheet is in edit mode; save or cancel the edit first.
|
|
</Accordion>
|
|
<Accordion title="Disabling a blueprint is rejected">
|
|
Blueprints with active or drifted deployments refuse to disable, because doing so would orphan the running containers without a follow-up plan. Withdraw each affected deployment first, then disable the blueprint.
|
|
</Accordion>
|
|
<Accordion title="The data is gone after 'Snapshot, then evict'">
|
|
Both eviction modes run `docker compose down`, which removes the named volumes managed by the stack on the target node. The snapshot stored in **Fleet · Snapshots** captures the compose definition only; the volume bytes are not included. To preserve data, capture the volume by hand before withdrawing (see *Migrating stateful data between nodes*). Bind mounts on the host filesystem are left in place by both eviction modes.
|
|
</Accordion>
|
|
<Accordion title="Eviction was aborted with 'Failed to capture compose snapshot before eviction'">
|
|
Sencho aborted the eviction because the pre-eviction compose snapshot could not be written to the database. The deployment is still in place; nothing was destroyed. Check that the database is reachable, then retry the eviction. If you accept data loss and want to evict regardless, use **Evict and destroy data** instead.
|
|
</Accordion>
|
|
<Accordion title="A pinned blueprint did not deploy to nodes in its selector">
|
|
By design. A pin replaces the selector entirely with the single pinned node, even if the selector resolves to other nodes. Either clear the pin in the Federation tab to restore selector-driven placement, or move the pin to a different node.
|
|
</Accordion>
|
|
<Accordion title="The reconciler took a minute to react to a change I just made">
|
|
The reconciler tick interval is one minute for approved blueprints. After editing the selector, the compose, or the drift policy, click **Apply now**, review the rollout preview, and **Confirm Apply**. The same path recovers a row in **Failed** status.
|
|
</Accordion>
|
|
</AccordionGroup>
|
|
|
|
## Common questions
|
|
|
|
<AccordionGroup>
|
|
<Accordion title="Do Blueprints replace my existing stacks?">
|
|
No. Stacks created in the per-stack lane (the **Stacks** sidebar) and Blueprints (the **Deployments** tab) coexist on every node. A Blueprint is identified by its `.blueprint.json` marker; the reconciler refuses to touch any directory that does not carry one with a matching blueprint ID, so user-authored stacks are safe from blueprint actions.
|
|
</Accordion>
|
|
<Accordion title="Is a Blueprint the same as a Stack?">
|
|
A Blueprint is a fleet-wide declaration: one compose YAML targeting a set of nodes. Each node where the blueprint resolves materializes as a Stack on that node, in the same compose directory layout the rest of the per-stack lane uses, with a `.blueprint.json` marker added. You can browse the materialized stack in the per-node Stacks view; the Blueprint is the source of truth that drives it.
|
|
</Accordion>
|
|
<Accordion title="How fast does drift get noticed?">
|
|
After you confirm the updated rollout, the next reconciler tick (or the Confirm Apply execution itself) places or removes as approved. The reconciler ticks every 60 seconds with a 5-second initial delay after startup. Within Enforce mode, drift correction for an already-approved place outcome begins on the same tick that detects the drift.
|
|
</Accordion>
|
|
<Accordion title="Can I roll back to a previous revision?">
|
|
Not through a one-click history. Blueprints intentionally do not keep a versioned revision history. To revert, paste the prior compose into the editor and save; the reconciler treats the change as a new revision and redeploys.
|
|
</Accordion>
|
|
<Accordion title="Does the snapshot in 'Snapshot, then evict' contain my database?">
|
|
No. The snapshot captures the compose YAML only, so you can redeploy the same definition elsewhere. Volume bytes (the database files, the cached state, the uploads directory) are removed by `docker compose down` along with the named volumes. To preserve data across an eviction, capture the volume by hand before withdrawing.
|
|
</Accordion>
|
|
<Accordion title="Why doesn't Sencho move volumes between nodes for me?">
|
|
Volume migration is app-aware. A Postgres datadir is not the same kind of artifact as a Redis RDB file or a MinIO bucket, and a generic `tar` of the volume is rarely safe while the container is running. A future Volume Migration feature will integrate app-aware backup tooling. Until then, the manual workflow in *Migrating stateful data between nodes* is the supported path.
|
|
</Accordion>
|
|
<Accordion title="If I cordon a node, does that uninstall the Blueprints on it?">
|
|
No. Cordoning only blocks new placements and skips state-review provisioning. Existing deployments on the cordoned node keep running and continue drift-checking. To uninstall a deployment from a cordoned node, withdraw it explicitly from the deployment table.
|
|
</Accordion>
|
|
<Accordion title="Can a Pilot-attached node host a Blueprint deployment?">
|
|
Yes. The reconciler dispatches deploys to remote nodes through the standard proxy regardless of whether the remote runs in Pilot Agent or Distributed API Proxy mode. The remote node enforces its own vulnerability policy and writes its own marker file, the same way a local deploy does.
|
|
</Accordion>
|
|
</AccordionGroup>
|
|
|
|
## Where Blueprints fits
|
|
|
|
<CardGroup cols={2}>
|
|
<Card title="Fleet Federation" icon="network-wired" href="/features/fleet-federation">
|
|
Cordon nodes to block new placements and pin blueprints to a specific node to override the selector.
|
|
</Card>
|
|
<Card title="Fleet Snapshots" icon="camera" href="/features/fleet-backups">
|
|
Where Snapshot-then-evict parks the compose YAML and where you redeploy it from.
|
|
</Card>
|
|
<Card title="Fleet Actions" icon="bolt" href="/features/fleet-actions">
|
|
Bulk operations on the running fleet. Blueprints sit on the desired-state lane; Fleet Actions sit on the imperative lane.
|
|
</Card>
|
|
<Card title="Atomic Deployments" icon="shield-check" href="/features/atomic-deployments">
|
|
Deploy safety on the per-stack lane. The same vulnerability policy gate runs before each Blueprint deploy.
|
|
</Card>
|
|
<Card title="Pilot Agent" icon="link" href="/features/pilot-agent">
|
|
Outbound-only remote mode. A Pilot-attached node is eligible as a Blueprint target the same way a Distributed API Proxy node is.
|
|
</Card>
|
|
<Card title="Alerts & Notifications" icon="bell" href="/features/alerts-notifications">
|
|
Where `blueprint_drift_detected` and `blueprint_drift_correction_failed` route. Configure channels per notification.
|
|
</Card>
|
|
</CardGroup>
|