mirror of
https://github.com/Studio-Saelix/sencho.git
synced 2026-08-28 19:27:41 +00:00
Anso
38ee4527b1
* feat(git): add GitOps revision store and Direct apply transitions
Adds the canonical GitOps revision state model: schema, store, history,
approval resolution, Direct apply transitions, and a first-cut read
projection. Also widens recovery capture with generation, artifact, and
source-acceptance bindings, rejects credentials and query strings in new
Git repository URLs, and keeps Blueprint and node-label routes on the hub.
* feat(git): add create-from-Git activation and teardown transitions
Adds the durable half of create-from-Git: the single transaction that persists
the application, resolved commit, generation, checkpoint, and candidate
together, and the teardown that tombstones a create which never reached its
success boundary. Adds the staging marker and operation-owned cleanup rules
that decide what a crashed create is allowed to delete.
* feat(git): make create-from-Git crash-safe
Wires the create path through the GitOps state model: the staging marker is
written before the candidate is built, the activation transaction persists the
application, generation, and checkpoint together, and the source row plus
accepted pointers now commit as one success boundary. A create that fails
before that boundary removes only what it staged, then tombstones itself.
The managed-area sweep no longer deletes a directory it cannot prove it owns,
counts in-flight creates as claimed, and is awaited at startup so an
interrupted create is settled before any mutation service runs.
* feat(git): settle interrupted create-from-Git at startup
Adds boot-time recovery for creates a previous process left in flight. A create
whose project was already committed to disk is finished, including its source
row and accepted pointers. Anything earlier has its files removed first and is
then tombstoned, so the stack name becomes usable again instead of staying
locked by a half-created application. A source row that outlived its
application is always preserved.
* fix(git): close create-from-Git ownership and reporting gaps
Removes two ways a failed create could destroy files it did not own: boot
recovery no longer deletes a stack directory the create never recorded making,
and the managed-root teardown is gated on having created that root. Cleanup
now rejects a path that is not a generation directory, so a malformed row
cannot widen the blast radius.
A create that fails during materialization now clears its staging marker, which
previously stayed behind and made the stack name uncreatable until restart.
Generation paths no longer resolve through a module cycle that left them
pointing at a directory that never existed.
Boot recovery restores the deploy spec and manifest cache a finished create
needs, settles each application independently so one bad row cannot strand the
others, and distinguishes a missing directory from one it could not read. A
create that succeeds now clears its checkpoint, and a failure after the success
boundary says the stack was created rather than reporting a bare error.
* fix(git): reap managed areas that nothing has ever claimed
The boot sweep now distinguishes a missing staging marker from an unreadable
one. A missing marker means no stack, no create, and no marker claims the
directory, which is the ordinary orphan a crashed stack deletion leaves behind,
so it is removed as it always was. An unreadable marker is evidence of a claim
that cannot be verified, so the area is preserved.
Without this an orphaned managed area survived every boot forever, since a
completed create deletes its marker and nothing else would ever claim it.
* feat(git): complete the Direct source and target transitions
Adds the remaining Direct events the state model owed: an invalid fetch that
advances the resolved commit without minting a candidate, a blocked candidate
that is visible but cannot apply, dismissal that leaves the running workload
alone, material configuration change that invalidates a staged candidate while
keeping accepted and applied pointers, the two deploy failure classes, and
application and target tombstones.
Widens the application update to every mutable column. It previously wrote
about half of them, so a transition assigning a pause, suspend, recovery, or
intent field would have type-checked, appeared in the history snapshot, and
been silently dropped at commit.
* feat(git): record Direct Git operations in the revision state
Wires fetch, apply, and detach through the state model. A pull records the
fetch and, when it produces a candidate, the generation behind it, marking a
blocked change plan as a blocker that cannot apply. An apply records the
acceptance that binds the generation to the workload, and closes its operation
on failure so a throw cannot leave the source reporting work in progress.
Detach tombstones the application and its target in the same transaction that
removes the source row.
Every producer is a no-op for a stack with no live application, so installs
whose Git stacks predate this model are untouched until migration runs.
* test(git): cover the Direct Git producers end to end
Adds a harness that stubs only the git transport and rollback capture, so
fetch, candidate materialization, change-plan classification, apply, and detach
all run for real and are asserted against the resulting revision state.
Covers the seam the transition-level tests could not: that a fetch advances the
resolved commit without moving the accepted generation, that the apply binds
the exact candidate the fetch recorded and its acceptance proves that
generation and no other, that detach tombstones while keeping configured
identity as a frozen fact, that a failed fetch closes its operation and leaves
every pointer alone, and that a stack with no application produces no writes.
* feat(git): bind the deployed generation from the Compose adapter
Makes ComposeService the sole producer of deploy events. Every deploy path
funnels through deployStack, so recording it there keeps one start and one
terminal row per mutation instead of each caller reporting its own.
A successful deploy binds the applied generation, a failure records the class
and leaves the deployed pointer alone. The class is conservative: once the
compose command is handed off we cannot prove the workload was untouched, and
claiming it was intact is the more dangerous error. A stack with no live
application, or one with nothing applied, has no deploy identity to record and
is skipped entirely.
* feat(git): promote healthy and last-known-good from health verdicts
A stack health run now records which generation it observed, and its verdict
drives the revision state. Promotion is narrow by design: healthy and
last-known-good move only when the run passed, watched the whole stack, and the
generation it watched is still deployed. A stale or service-scoped verdict
records history and moves nothing.
Last-known-good keeps the artifact expectation only when that expectation
belongs to the promoted generation. Otherwise the generation is still good and
its executable identity simply is not proven, so the pointer is left null
rather than borrowed from another generation.
beginStack now takes the deployed generation explicitly rather than reading
current state, because a verdict is only meaningful for the generation the run
actually observed. Update-path callers pass null until updateStack reports what
it deployed.
* feat(git): bind the deployed generation on the update path
Gives updateStack the same treatment deployStack has: it opens a deploy
operation at the point Compose is handed the mutation, closes it on both
outcomes, and reports the generation it bound. The orchestrator carries that
binding through, and all five update call sites now pass it to the health gate
instead of null, so a passed health run after an update promotes healthy and
last-known-good exactly as it does after a deploy.
The operation is opened at the compose call rather than at entry, so an update
that fails during capture or classification records nothing: it never touched
the workload, and reporting a deploy failure for it would make the projection
claim something that did not happen.
* feat(git): track linked stacks and material config changes
Linking an existing stack to a Git source now brings it into the revision
state: the application starts live with nothing desired or accepted, so the
projection asks for a fetch rather than claiming a state it has not observed.
Editing a source's material configuration invalidates its staged candidate in
the same transaction that writes the row and clears the pending pull. A
candidate built from a different repository, ref, or file set can no longer be
applied, and clearing pending without invalidating it would leave the model
offering an apply the operator cannot produce. Credential-only and policy-only
edits change nothing material and leave the candidate alone.
* feat(git): retire the application when a stack is deleted
Stack deletion now tombstones its GitOps application and targets in the same
transaction that commits the deletion. A deleted stack that kept a live
application would go on claiming the stack name and block re-creating it
through the unique live-application index.
The tombstone is driven from the deletion service rather than from inside
DatabaseService, so the store keeps its transitions and their history in one
place and the two modules do not form an import cycle.
* feat(git): retire node targets when a node is removed
Deleting a node now tombstones the GitOps targets that lived on it, in the same
transaction as the delete and while those rows still exist. Otherwise a
part-way failure would leave targets pointing at a node that is gone.
Applications stay live: a Direct application still describes a real stack, and
a Blueprint one may have targets on other nodes. Both the local and remote node
paths go through the same helper, so neither can skip the retirement.
* feat(git): add proof-bound recovery transitions
A restore moves a target back to an older generation, the one case where a
target and its application legitimately disagree about what is current. These
transitions decide what may move with it.
Pointers move only when the restore is provable: the recovery point named a
generation, that generation still exists, and it belongs to this application.
The artifact expectation comes from what the recovery point captured, never
from what the application expects now, and the acceptance is kept only if it
still proves the restored generation rather than borrowing one that authorized
a newer one. A last-known-good survives unless the generation behind it is gone
or turns out to belong elsewhere, and then the reason is recorded so the
projection can say unavailable rather than none.
An unproven restore is still recorded as a real operational recovery and moves
nothing, because there is no evidence to move pointers to.
* fix(git): close GitOps operations that cannot be recorded
Recording a transition still never fails the operation it describes, but a
rejected terminal event no longer leaves that operation open. A start that
never terminates reported work in flight for ever and offered no actions, and
because a new fetch refuses to open a second operation, one rejection silently
stopped the model tracking that stack until a restart.
Startup now reclassifies operations the previous process left open, which the
transition already supported but nothing called. An interrupted restore is
closed there too, since only the terminal recovery events clear it.
The pull records its fetch, generation, and candidate as one transaction, and
the apply verifies the candidate was built from the commit being applied.
Separately those two allowed an apply to accept a generation whose files were
never on disk while the projection reported the older commit as current.
Also: the candidate now records its own invocation rather than the one it
replaces, changing material configuration is refused while an operation is in
flight, a tombstoned target cannot be repopulated by a late health verdict, a
deploy handle is not returned when its start was not recorded, the mutation
handoff is marked only once Compose is genuinely about to receive it, and a
rejected tombstone fails one deletion rather than aborting the boot sweep.
* feat(git): record why unprovable evidence was dropped
The projection carried a limitations array that only read-time derivation ever
wrote to. When a transition cleared a pointer it could not prove, the result
was indistinguishable from that pointer never having existed, and in two cases
it made the target read healthier: a dropped artifact expectation silently
disables the runtime drift check, and an unproven restore left every pointer
agreeing with itself so the target reported as synced and healthy.
Adds a persisted, fail-closed evidence record on the application and target
rows. The transitions that drop an artifact expectation, a last-known-good
artifact, or a source acceptance now say why, an unproven restore is marked as
such, and the deriver folds all of it into the limitations the projection
already exposes. Clearing a code when the evidence becomes provable again is
part of the contract, so a stale doubt cannot outlive its cause.
This lands before the migration matrix because migration is the largest
producer of evidence that cannot be proven, and the plan requires those to
surface as bounded limitation evidence rather than as fabricated pointers.
* feat(git): migrate pre-existing Git stacks into the revision state
Git stacks created before this model had no application at all, so every
producer was a no-op for them and the projection could not describe them. They
are now brought in at boot.
The governing rule is that a canonical pointer is written only when the
evidence proves that exact generation under the repository and ref configured
now. A legacy applied commit is not that proof on its own: the manifest may be
gone, unreadable, or stamped for a repository the stack no longer points at. In
each of those the commit is kept as recorded limitation evidence and the
pointers stay null, so the stack asks for a fetch instead of asserting a state
nobody verified. Deployed, healthy, and last-known-good are never invented,
because a manifest proves what was materialized, not what is running, and no
source acceptance is written because nobody approved through the model.
Replay is decided by scope, schema version, and configuration fingerprint. A
changed fingerprint re-runs the matrix, but a stack that already has an
application is skipped rather than rebuilt, so migration cannot overwrite
pointers written with proof it does not have. A stack whose directory has
vanished migrates to a tombstone rather than claiming a name it cannot back.
* feat(git): record rollbacks in the revision state
A restore now opens a recovery in the model before any file moves, so a crash
mid-restore leaves a target that says what it was doing rather than one that
merely looks broken, and closes it on both outcomes. The failure is classified
by whether the files had already been restored, because only a failure before
that leaves the previous workload provably intact.
Pointers move only when the restore is provable: the recovery point named a
generation, that generation still exists, and the manifest actually restored
carries the same commit and manifest version. Anything less is still recorded
as a real recovery, it simply has nothing to bind, and the transition marks it
unproven rather than guessing.
The deployed pointer is never claimed here. This path drives Compose through a
callback that reports nothing back, so binding cannot be proven and applied
moves without it.
* feat(git): add the deferred rollout-state transitions
Retry scheduling, suspension, pause, and partial rollout complete the
transition store. They have no production writer by design, but implementing
them now means the deriver has no branch a writer cannot reach, and the shape a
future producer has to satisfy is pinned rather than inferred from the read
side.
None of them is a statement about health. A scheduled retry leaves the failure
that caused it visible, so a stack that keeps failing does not read as merely
busy. A suspended source keeps everything it had accepted, and an operation in
flight is interrupted rather than abandoned so it cannot report as running for
ever. A paused or partially rolled out target keeps whatever was deployed, and
the partial record never stands in for a deployed pointer.
* test(git): stop the Compose producer tests reading Docker from the host
Both deploy cases asserted that the compose command would reject, which was
only true on a workstation with no Docker daemon. On Linux the command
succeeded, the promise resolved, and the assertions failed.
The compose subprocess now reports an exit code each test chooses, so the
adapter is what decides the outcome rather than the machine. Only the verbs
that need a daemon and travel through spawn are answered; `config` still runs
for real, because a host with the CLI and no daemon parses compose files
exactly as CI does, and failing that call breaks every create in the file.
That also makes the bound path reachable, so the deploy test now covers it:
a failed compose leaves the deployed pointer alone and classifies
post-mutation, and a successful one binds the applied generation, returns it
for health to bind against, and clears the earlier failure.
* fix(git): contain every managed-area path at the call that uses it
A stack name reaches the managed root without passing through
`isValidStackName` on this path, so the marker, the cleanup, and the create's
root probe all built filesystem paths from unvalidated input. Each call now
resolves its target against the managed area and checks containment in its own
scope, which is also the form the security scan credits: it does not follow the
barrier through the shared `isPathWithinBase` helper, so the check has to sit
with the call it protects.
`cleanupUnclaimedManagedRoot` removes a whole root recursively and gets the
same check, even though it was not among the reported calls.
Clearing a staging marker can now fail, so the two boot-recovery branches that
dropped the checkpoint first were reordered. Losing the checkpoint while the
marker survives would report the area as settled and leave the stack name
uncreatable.
The deploy adapter's two failure logs built their format string from the stack
name; both now use a constant format with the name as an argument.
The marker and cleanup fixtures stood a bare temp directory in for a managed
root, so nothing exercised the invariant these checks enforce. Both now build a
real managed area over a scoped data directory.
* fix(git): refuse to claim a managed area whose marker cannot be read
A marker that exists but will not parse is still someone's claim, yet the write
path refused only a readable foreign marker and wrote straight over a corrupt
one, discarding the reason and logging nothing. A transient permission error, a
marker truncated by a crash between write and rename, or a full disk all read
as corrupt, and each one let a second create take ownership of an area the
first still owned. Every other path in the module preserves on corrupt; this
one now does too, and says why.
Clearing a settled create's staging marker can fail, and reporting that as a
retained create sent a reader looking for an unfinished create that had
finished. It reports `marker_retained` instead. The checkpoint is still kept
for the retry, so its encrypted token can outlive the create while the clear
keeps failing; that tradeoff is stated at the call, because the alternative
leaves a marker no later create can get past.
The reaper also computed why it was preserving an area and threw the reason
away, so a directory could survive every boot with nothing said about it.
The containment checks these paths rest on had no tests and were silently
deletable, so the out-of-area read, write, delete, and reap now have them.
* feat(git): bind and observe a recovered generation
A proven restore could never claim the deployed pointer, because the restore
path drives Compose through a callback that reported nothing back: a caller
that restored some other way resolved identically, so binding on a resolved
promise would have claimed a workload nobody launched. The Compose wrapper now
returns what it did, and only that answer binds.
With binding reachable, a bound recovery claims its health run inside the same
transaction as the pointers it describes, and arms the timer once that
transaction lands. Committing the two together is the point: a crash between
them would otherwise leave a restored workload nothing was watching. Anything
that stops the timer starting writes the run off immediately, so an observing
row never outlives the timer meant to watch it, and a reservation is never
armed across a restart.
The reservation is handed to the transition rather than reached for, because
the health gate reports its verdicts back through the transition store and
importing it there would close a module cycle.
Startup now finalizes each interrupted observation on its own instead of
sweeping them with one update, so the revision state hears a verdict for every
run rather than watching the rows change under it.
* feat(git): add the rollout-scoped rollback transitions
Completes the deferred-state set. The three aliases write the same recovery
columns under the same rules as the recovery events, differing only in
provenance: Direct Git recovery emits `recovery_*`, and a later rollout
producer emits these. Nothing in this PR writes them, which is why they are
tested directly rather than through a caller.
`partial` is the one failure class they add, for a rollback that reached some
targets and not others. Completing has no unproven variant on purpose: a
rollback nobody can bind to a generation has nothing to complete against, so
it refuses without one and refuses a generation another application owns.
Also covers the recovery health reservation end to end: reserving writes and
links the row without arming anything, a replay reuses the run rather than
opening a second observation of the same restore, arming inserts nothing and
supersedes a conflicting stack gate, an unarmed reservation is written off
once, and a reservation that outlived its process is finalized rather than
armed. The per-row startup sweep is pinned too, since one row whose write
fails leaving the others finalized is the entire reason it replaced a bulk
update.
* feat(git): add the Blueprint source and deployment transitions
The first slice of the Blueprint work: the store side, with no production
caller yet. The routes and the reconciler are wired to these next, and pinning
the contract first means a caller cannot quietly satisfy a different one.
What these enforce is that a terminal event names the request it answers. A
deploy records the intent and candidate it was launched for, and a terminal is
accepted only against that same request, matching stage and identity on one
side rather than either in isolation. Without that pairing a deploy could be
acknowledged out of an in-flight withdraw, or one request's intent stored
against another's candidate. An acknowledgement carries the candidate from the
matched request rather than from its own payload, for the same reason.
Releasing an operation now releases its identity with it, and a resolved
interruption is retired rather than left to match again. Both were leaking
through the shared helpers: a stale identity let a later start resurrect a
superseded intent as live, and a surviving interruption both reported
completion as unknown for ever and let a late acknowledgement regress the
target after two later deploys had succeeded. Fixing the helpers fixes the
Direct paths too.
A start refuses to displace an unrelated operation, which would otherwise
abandon it with no terminal event and no history saying so. Observations write
the latest stage, since the placement facet reads it to decide whether a
stateful deployment is waiting on confirmation, and history alone could never
reach a reader.
Minting is left to the caller. A no-op Blueprint edit must mint no intent,
because a fresh identity would invalidate acknowledgements that are still
accurate.
* feat(git): record Blueprint create, edit, pin, and delete
Wires the Blueprint routes to the revision state. Each producer writes the
Blueprint source row and its GitOps rows in one transaction, so an operator
never sees a Blueprint that exists with nothing describing what it means, or an
intent for a Blueprint that failed to save.
The question these answer is when an edit invalidates what the fleet already
acknowledged. Changing the name, the compose content, the selector, the drift
mode, or whether it is enabled changes what nodes run or where, so each mints a
new intent and candidate. Changing the description or the classification
changes how the Blueprint reads and nothing a node can observe, so it updates
the source row alone: minting there would make every acknowledgement report as
stale over a reworded sentence.
An edit is measured by value, not by which fields were submitted. The editor
sends every field on every save, and the source layer decides what to
invalidate from which keys are present, so only the keys that genuinely differ
are passed down. Without that the two disagreed: rewording a description
advanced the revision past the one the current intent describes and cleared the
approval, while this layer classified it as metadata and minted nothing.
Selectors compare by value too, so reordering a list that names the same nodes
is not a placement change.
That measurement also changes one behaviour: a save that alters nothing now
leaves the approval alone, where before any save cleared it. Nothing changed,
so nothing is invalidated.
Pinning revises placement the same way a selector edit does, and re-pinning the
node already pinned is not a change. Deleting tombstones the application and
its live targets so the Blueprint stops claiming its slot, and withdraws
nothing itself, because the route has already done that and doing it twice
would record removals that never happened.
The required node set is stored in a canonical order, so reordering it is not
mistaken for a placement change. A Blueprint that predates the model has no
application yet and is left for migration rather than given a first intent
here, which would claim a starting point for deployments nobody has reconciled.
Desired nodes are passed in rather than computed here: the reconciler that
knows how to compute them reaches this layer, and importing it back would close
a module cycle.
* feat(git): record the placement a label or cordon moves
A label and a cordon say nothing about any one Blueprint, but both change which
nodes a selector matches. Each now revises placement for whichever Blueprints
the change actually moved, by comparing the desired nodes either side of the
write rather than reacting to the event.
That comparison is the whole point. Labelling a node no selector mentions, or
cordoning one no Blueprint wanted, moves nothing, and minting an intent for it
would invalidate every acknowledgement in the fleet over an edit no node can
observe. The same comparison covers a Blueprint pinned to a cordoned node: a
cordon governs automatic placement only, so the pinned target still wants that
node and its set does not move, without needing a case of its own. Nodes are
compared as a set, so the same nodes returned in a different order is not a
change either.
Each route wraps its existing write and the recording in one transaction, so a
recording failure cannot leave a fleet selecting on a label nothing recorded.
The write itself is untouched, which keeps label validation and the per-node
limit where they already live rather than restating them in a producer.
Node deletion already retires its targets, so this adds nothing there.
* feat(git): record Blueprint deployments by what caused them
Every production write to a Blueprint deployment row now goes through one
funnel that records the cause. The cause is carried rather than inferred from
the resulting status, because several causes land on the same one: a deploy
that failed and a withdraw that failed both read `failed`, and they mean
opposite things about whether the deployment is still on the node.
Recording is skipped when the status did not move, so a reconciler tick that
re-asserts a state it already reported does not append a second event
describing the same fact. The write still happens either way.
A terminal answers the request the target says it was given, not whatever the
Blueprint currently wants: an acknowledgement matched against the current
intent would accept work for a revision that node was never sent. The first
deploy to a node creates its target, since a Blueprint application has no
targets until something is sent somewhere.
Recording never fails the deployment. The rollout already happened, and turning
a bookkeeping problem into a stuck rollout would be the worse outcome.
The four reconciler observations record what was seen and nothing else. Preview
cleanup stays outside this path deliberately: it reverses a projection nobody
deployed, so recording it would report removals that never happened.
* fix(git): record an Inline Blueprint coming into existence
Creating a Blueprint inserted its application row directly, so the history
began at the first intent and described an application nothing recorded coming
into existence. Every other application-creation path emits the activation
event, and the event's own definition covers this case: an Inline Blueprint
inserts the application, the first intent, and the first candidate.
Activation goes through a transition now, which also enforces the one thing the
direct insert could not: a Blueprint gets one live application, refused rather
than silently duplicated.
No target is created. A Blueprint application has no targets until something is
deployed somewhere, unlike a Direct one which always has the node its stack
lives on.
* fix(git): file withdraw failures as withdraw failures
Four of the five withdraw failure paths were recorded as failed deploys. Only
the thrown one was tagged correctly, and it is the rare case: a delete-lock
conflict, a network error, a remote 409, and a remote non-200 all return rather
than throw, and all four were filed under deploy.
That inverted the thing this funnel exists to prevent. Recording a deploy
failure clears the in-flight operation, so a routine "another operation is
already in progress" wiped the withdraw that had just started and left the
target reporting a failed deploy while the deployment was still on the node.
A withdraw refused because the on-disk stack is not ours is now its own cause
too, rather than borrowing the deploy-side name conflict.
The status guard no longer suppresses starts. A start writes the identity that
terminals are matched against, so skipping one because the row already read
`deploying` let a later acknowledgement answer a request that had been
superseded, reporting a node as converged on a revision it was not running. The
guard now covers observations only, which is where repetition is the reconciler
re-asserting a state it already reported. A newer deploy supersedes an older
one so a redeploy of a stuck request can take over; anything else in flight is
still refused rather than displaced with no terminal event.
A rejection is logged differently from an infrastructure error, because a
target the model keeps refusing holds its active slot and stops recording
anything further, and that is worth seeing.
* feat(git): migrate pre-existing Blueprints into the revision state
Blueprints that predate the model are brought in at boot: an application, an
intent describing what the Blueprint currently asks for, and a candidate marked
as coming from the legacy inline record.
None of it is an acknowledgement. The Blueprint revision is carried for display
and nothing else, because a revision number is not evidence that any node is
running what it names, and recording it as agreement would report a fleet as
converged on an intent nobody verified. No targets are created for the same
reason, and the candidate's required set is empty: migration records what is
asked for, never which nodes currently satisfy it.
An approval authorizes the intent it was given for, so a Blueprint that was
never approved, or edited since, is recorded as needing reapproval rather than
left blank. Blank would read the same as an approval that is merely absent.
Replay is decided by the same scope, schema version, and fingerprint the Git
migration uses, and a Blueprint the new path already described is skipped
outright, since its rows were written with proof this pass does not have.
Blueprints migrate in their own pass. Coupling them to the Git migration would
let one unreadable Git stack keep every Blueprint outside the model.
* fix(git): contain the blueprint stack-directory probe at its call
The ownership probe resolved a path from the blueprint name and checked it
through the shared helper, which the security scan does not credit as a
barrier. The check now sits with the call it protects, matching the form used
elsewhere for the same reason.
The path was already validated, so this changes no behaviour. It surfaced now
because an earlier commit in this branch rewrote the file's line endings, which
made every line read as new and brought a pre-existing call into the scan's
changed-code window.
* feat(git): read GitOps history and carry revision state on source reads
Adds the instance-local history API and the additive revision fields the
source reads have been missing.
`GET /api/git-sources` and `GET /api/stacks/:stackName/git-source` now
carry `gitopsRevision` and `stackResourcePresent`. Only the instance that
owns the files can say whether a stack directory is really there, so that
answer travels with the response rather than being inferred by a reader
that has never seen the filesystem. It is read through the strict stack
listing: the lenient one answers a failed directory read with an empty
list, which here would read as every stack having vanished and would turn
an unreadable disk into an empty audit trail.
Two history routes land with them: `GET /api/git-sources/history` across
stacks, and `GET /api/stacks/:stackName/git-source/history` for one. Both
page on `(created_at, id)` so rows written inside the same millisecond
survive a page boundary, and both bound how far one request will scan. A
malformed cursor or an unusable filter value is refused rather than
ignored, because answering "show me the failures" with everything reads
as an answer instead of a non-answer.
Authorization is decided per row. A row reduces to a stack read only when
it names a stack, its application is live, and the stack is present on
disk; anything unprovable stays with Admin. The cross-stack routes
classify every row, while the per-stack route is authorized whole by
name, so reading a stack shows its full history including entries from an
earlier application. That distinction is carried in the scope type rather
than a flag, so skipping the row classifier without pinning the query to
the authorized stack cannot be expressed.
History entries record the fields each transition moved rather than a
whole revision, so lifecycle for the classifier comes from the owning
application row. That keeps authorization off the recorded payload
entirely: an entry whose detail cannot be read still returns its identity,
stage, and outcome with a stated limitation, and the decode failure is
logged rather than passed over in silence.
* feat(git): let auditors read GitOps history they cannot tie to a stack
A history entry that cannot be tied to a readable stack fell to Admin,
which left the auditor role seeing exactly what a viewer sees on the one
surface it exists to read.
History entries are an audit trail: insert-only, and recording the actor,
trigger, stage, and outcome of every transition. The request audit log is
already gated on the audit permission, so an entry whose audience cannot
be narrowed now falls there too. Withholding it protected nothing in any
case, since the request log already records that a Git-source mutation
happened and who made it.
The source collection deliberately keeps its Admin fallback. Those rows
are live Git configuration, not a record of events, and a mandate to
audit does not extend to reading the repository, ref, and credentials
policy of stacks that were deleted or never finished being created.
The fallback is now part of the requirement type rather than a role
comparison, so each surface states which audience it falls back to and
the compiler requires every case to be answered.
* feat(git): correct remote node identities on GitOps reads
A remote instance numbers its own nodes and has never heard of the hub's
numbering, so every node id it reports is a statement in its own
namespace. Read straight through, a hub joining two nodes showed two
different machines as the same node.
Four GETs now take a second hop that buffers the response and corrects
those identities: the git-source list and detail, and both history
routes. Everything else keeps streaming exactly as before, which is the
reason this is a separate hop rather than a mode of the existing one:
logs, downloads, and event streams must never be buffered, and the
enumerated positions rewritten here do not appear in them.
Only JSON numbers are replaced. A null node is preserved, since "no node"
is a fact the remote is entitled to state and inventing one there would
claim a placement that does not exist. Strings, application ids, and
stack names travel untouched.
The hub re-authorizes what comes back. A remote authorized its rows for
the machine account the hub proxies with, which says nothing about the
person behind the request, so every row is classified again against the
signed-in user. To make that possible without the hub holding another
instance's database, each history entry now carries the lifecycle of its
owning application alongside the stack-presence answer already there.
Both are validated fail-closed, and a verdict a remote might volunteer is
ignored: the hub decides, the remote supplies evidence.
Asking one node for another node's history is refused before the hop
rather than forwarded, because the remote would answer about itself and
the page would read as an answer to a question nobody asked. When the
node asked for is the node being talked to, the hub translates that into
a flag the remote resolves to its own default node. Only the hub may set
that flag, and a remote honours it only on a proxied hop.
Every way this hop can end converges on one terminal answer: rewritten,
passed through, too large, undecodable, failed upstream, or abandoned by
the client. A body the hub could not read never borrows the remote's
success status.
* test(git): assert the unlinked git-source response field by field
The detach test compared the whole response to `{ linked: false }`, which
stopped holding once that route started carrying the additive revision
fields. The equivalent backend route test was updated when those fields
landed; this one was missed.
Asserted field by field rather than loosened to a partial match, so the
two new fields are actually checked: a detached stack has no application
to project while its directory is still on disk.
* feat(git): carry GitOps revisions on blueprint, node, and drift responses
Completes the additive JSON half of the revision-state read contract, so the
Blueprint, node, and drift surfaces report GitOps state through the same
projection the Git-source routes already return instead of leaving it
unreadable.
Blueprint list, detail, create, update, and pin carry gitopsRevision. Node-label
add, cordon, uncordon, and node delete carry gitopsRevisions, ordered by
blueprint id and covering only the Blueprints the mutation actually moved: an
edit no selector reacts to reports an empty list rather than invalidating the
whole catalog. Node deletion reads its Blueprint owners inside the deletion
transaction and before the tombstone, which is the last moment a target row can
be traced back to an application, and an orphaned target is logged rather than
being silently indistinguishable from a Direct one. Both DELETEs that answered
204 still answer 204.
The drift GET and re-check gain the same field alongside the existing ledger,
which is untouched; no GitOps class is written into stack_drift_findings.
Reads and mutations treat a projection fault differently, on purpose. On a read
the revision is part of the answer, so a fault surfaces. On a mutation the write
has already committed, so the revision is decoration and is built defensively:
letting it throw would answer a successful cordon or a completed node deletion
with a 500, sending the operator to retry a hard delete that already happened
and be told the node does not exist. It degrades to an empty list and logs.
Because both drift routes answer with that projection, the identity hop now
intercepts them too, matching its route table per method instead of rejecting
every non-GET. Left as they were, one object would carry the hub's node
numbering or the remote's depending only on which route asked for it. The
allowlist names a single mutation rather than opening a verb, and the hub's
collection filter still keys off the git-source paths alone, so drift payloads
are rewritten without being re-authorized: they are per-stack and already
authorized by name before the hop.
Tests cover the exact shapes, the preserved 204s, revision ordering, the
empty-list cases, node deletion reporting its tombstoned Blueprints and
surviving a projection failure, and the proxy's new route matching, drift
rewrite, and filter exclusion.
* fix(git): resolve every application a GitOps surface can own
Three gaps found while building the additive JSON, all the same shape: an id or
a sentinel stood in for a row, so "there is nothing here" and "the thing that
should be here is unreachable" collapsed into one answer nobody could tell
apart.
A Blueprint application is stored with stack_name NULL, so no lookup by stack
name could reach it, while the reconciler materializes every Blueprint as a
stack directory of that name. A stack's Drift tab and Git panel therefore
reported no GitOps at all for a stack GitOps was actively managing, while the
Blueprint page reported a live application for the same thing. Stack surfaces
now bridge the two through the deployment row for the asking node. The
deployment row is what makes it safe: Blueprint and stack names share one
namespace, so matching on name alone would let a Blueprint claim an unrelated
stack of the same name on a node it never targeted.
A retired application was unreachable for the same reason, because every entry
point filtered to the live rows. Tombstoning deliberately keeps the configured
identity and SHA pointers as frozen facts so the projection can still say what
an application was, and the source deriver has a not_live status waiting for
exactly that, but nothing could hand it one. A detached Blueprint or stack now
reports what it was instead of reading as one that never existed.
The not-applicable projection was typed so it could not carry a reason, which
made a row that vanished between resolving it and re-reading it by id identical
to a stack the model was never asked about. That one field widens to hold
limitations, and the missing case now says so with the id as evidence. Every
other missing-row path in the deriver already worked this way.
Also logs, rather than silently skipping, a live application whose blueprint row
is gone: with cascade off that is a real integrity fault, the placement did
move, and the response would otherwise report that nothing had.
Cordon and uncordon are deliberately left reporting nothing. A cordon governs
whether new placements may be made, not what a Blueprint asks for, and the
reconciler applies it only to new placements, so revising intent for it would
invalidate acknowledgements fleet-wide over a change that evicts nothing. Two
comments implied otherwise and now state the contract.
* fix(git): keep stack-state resolution out of the Git-source read path
Review of the previous commit found the fallback it added had reached further
than intended. The resolver is now split by the question it answers.
projectStackRevision answers "what Git source is attached to this stack" and
stays Direct-only. projectManagedStackRevision answers "what manages this stack"
and is what the drift routes use. The Git-source routes must not be answered
with another application's identity, and there was a second reason: the row
classifier takes the stack name from the Git-source row but the lifecycle from
whatever the projection resolved, so a Blueprint application arriving there
would have turned an Admin-only row into one any stack grant could read. The
Blueprint that unlocked it could be the very one holding name_conflict because
of that stack.
The Blueprint-to-stack bridge also needed the right predicate rather than a
present deployment row. name_conflict is written precisely when a stack of that
name already exists on the node and Sencho does not own it, so guarding only
against withdrawn admitted exactly the collision the bridge exists to prevent.
It now requires last_deployed_at, which proves this Blueprint wrote the
directory, and excludes name_conflict and withdrawn: the same predicate the
delete and withdraw paths use. Only a live Blueprint application qualifies,
since a retired one has no claim on the directory.
The detached fallback no longer resolves deleted applications. Detached means
the files are still on disk and still describe that stack; deleted means the
stack is gone, so any directory of that name now belongs to something else and
reporting the old repository and SHA against it would disclose one stack's Git
identity through another's name. The authorization side already refused deleted
rows for that reason, so the two now agree. Ordering breaks ties on rowid rather
than a random UUID, and two partial indexes cover the new lookups, which the
existing live-only unique indexes could not serve.
The not-applicable variant's limitations are readonly, so pushing onto the
shared frozen instance is a compile error again rather than a runtime throw.
Tests cover a Blueprint claiming a stack it never deployed onto and one it hit a
name conflict on, a detached Direct source projecting not_live, and a deleted
one staying unresolved for a reused name. Drift fixtures move to afterEach so a
failing assertion cannot leak state into the next test.
* fix(git): stop the stack resolver falling through proven ownership
A second review round found the resolution chain could answer with the wrong
application, and that part of the previous commit was reading a state nothing
writes.
The Blueprint bridge returned one value for two different facts. Once the
deployment row proved a Blueprint had written the directory, a missing
application row for it still answered "no Blueprint here", so the chain
continued to the detached-Direct lookup. A stack that once had Direct Git, was
detached, and whose directory a Blueprint later took over would then report the
old Direct application's repository, ref, and SHA as that directory's state.
The bridge now separates "not mine" from "mine, but broken": the second logs and
returns its own limitation, so the fault is visible and no unrelated identity is
offered in its place.
A Blueprint projection also carried the whole placement roster into a route
authorized by a grant on one stack name. Targets are now scoped to the node
being asked about, which is both the safe answer and the accurate one, since the
question is what manages this directory here.
The detached-Blueprint lookup, its index, and its fallback are removed. Blueprint
retirement writes deleted, never detached, and the only detached writer is the
Direct detach path, so the getter could never match and the index covered an
empty set. The test that seemed to cover it built the state by hand and was
green against something the product cannot produce; it now drives the real
delete route and asserts what that path actually writes.
The Git-source resolver goes back to live applications only. Detach removes the
source row in the same transaction that tombstones the application, so a source
row beside a detached one is not producible, and the route's row classifier
takes its stack name from the source row but its lifecycle from the projection.
Keeping that resolver narrow is what stops a resolution change from quietly
moving who may read a row. The comment saying so is restored and now explains
why it must stay true.
* refactor(git): share the GitOps application fixture across the route tests
Two test files carried near-identical 58-line GitOpsApplicationRow literals that
differed only in a repo URL nothing asserted on. Both now import one helper.
The drift tests also repeated a Blueprint seed and its activation across four
cases, and re-imported DatabaseService per test; those become local helpers and
one hoisted import, which lets the cleanup hook go synchronous. Node lookup
throws on a missing default instead of casting.
projectBlueprintRevisions is module-private again: its only caller is the
committed-projection wrapper beside it.
A third copy of the same fixture remains in git-source-routes.test.ts. It is
left alone deliberately: it pins timestamps at 1 rather than now, and that file
has ordering-sensitive history assertions that would need auditing first.
* feat(git): mirror the GitOps revision contract on the frontend
Adds the client-side read contract for the GitOps revision projection, with
no consumers yet.
types/gitops.ts is a hand-written mirror of the backend read contract, in the
same convention as the other domain type files: the frontend never imports
backend. The projection's two arms are named separately so a component can
take the live one as a prop without re-narrowing, and the absent arm keeps its
absent keys rather than nulls so reaching for a lifecycle status without
narrowing is a compile error. The limitation code stays an open string, since
the backend adds codes without a schema bump and an exhaustive switch would
silently stop rendering the newest ones.
lib/gitopsState.ts is the one place a facet status becomes words and a colour,
so a state reads the same in a sidebar tooltip, a panel banner and the drift
tab. Both maps are keyed on the closed status unions, so widening the mirror
without naming the new state fails this build.
Every line of copy states the condition the deriver actually tests, which is
not always what the status name suggests. recovery_required is a recovery
already running rather than one that is needed. candidate_ready is reached only
when review is not required, so it is ready to apply, not ready to review.
synced_and_healthy is also reached with the health gate switched off and never
compares the deployed generation against the accepted one, so it claims
neither. never_reconciled means nothing has been accepted; a fetch that
produced no materialization still records its commit.
pendingSourceStatus keys "an update is waiting" on the candidate pointer rather
than the status name, because source_reconcile_required is reachable both from
a stale candidate and from an accepted generation with no candidate at all.
Retirement is excluded first: tombstoning keeps the candidate pointer as a
frozen fact, so a stack detached mid-review still carries one.
absentFault separates the two facts the empty projection carries. Empty
limitations means the model was never asked about this stack and the right
rendering is nothing; a non-empty list means an application that was expected
could not be reached. A live application's limitations are caveats on state
that is being reported, not faults, so they are deliberately excluded.
* feat(git): name the Git source state on the stack and sidebar surfaces
The sidebar indicator and the anatomy source row both read a raw pending
commit pointer, so a candidate blocked by local conflicts, held for review, or
stale against the configuration in force all render identically to one that is
ready to apply. The projection distinguishes them; nothing was reading it.
refreshGitSourcePending now derives each stack's state from the projection and
the pending map carries that state rather than a boolean. Presence in the map
still means exactly what it meant before, because it is keyed on the candidate
pointer rather than the status name: source_reconcile_required is reachable
both from a stale candidate and from an accepted generation with no candidate,
and only the first is something to review. One narrow fallback remains, for a
row with no projection at all: a failed GitOps write is logged and swallowed
while the pending commit still commits, so the flat pointer is the only thing
that can answer there. Wherever a projection exists it is the sole authority.
The sidebar keeps one indicator, in the same slot, at the same size and colour,
with the same position in the priority ladder. Only the tooltip changes, so a
blocked plan now says so instead of reading as an ordinary update. A test
asserts the rendered indicator is identical across states, which is what keeps
the rendered sidebar unchanged.
The anatomy source row keeps its pulsing dot alone for the ordinary case and
adds the state word only for the states the dot cannot express. Inline rather
than in a tooltip, since this is the reason something is stuck.
* feat(git): surface derived GitOps state in the Git source and Drift panels
The Git source panel's pending banner had two things it could say, read from a
raw commit pointer and a single blocked boolean. It now names which of four
states the candidate is actually in, so a plan blocked by local conflicts, one
held for review, and one gone stale against the configuration in force stop
rendering as the same ordinary update. The short commit sha stays.
A new card above it reports an application the projection could not reach. That
case renders as nothing at all today, which is indistinguishable from a stack
the model was never asked about. Empty limitations stays silent, because that
is the ordinary answer for most stacks and a header over an empty block would
be worse than nothing.
The summary block gains one row naming the source state. It is the first place
the panel can show applying, retry scheduled, suspended, recovering, or a
detached source, none of which have a pending commit and so never reach the
banner. Both existing rows are untouched and the last applied commit stays a
display fact.
The revision lives in its own state rather than on the Git source type, because
the PUT that saves this panel answers with a bare source and no revision. A
save drops it instead of rendering a state the write has already moved.
The Drift tab gains a third block below the two it already has. The compose
versus runtime card and the since-last-deploy card keep their exact positions
and copy; the new block answers a different question, which generation this node
was asked to run and whether it got there. A Blueprint-owned stack shows its
target rows and no source card, since a Blueprint application has no Git source
and inventing one would be a claim the model never made. The drift class list
renders against the type and expects no rows, because nothing populates it yet.
* feat(git): declare revision fields on the blueprint and node clients
Types first. The blueprint list, detail, create, update and pin responses all
carry a GitOps revision, and the node label add, cordon, uncordon and delete
responses carry a list of them. None of that was modelled, so the fields were
being dropped silently and the next reader would have taken these for bare
records.
Most of it is declared and deliberately unread, with the reason written where
the type lives. Create, update and pin are followed immediately by a re-read of
the catalog or the detail that carries the same projection, so rendering the
mutation's copy would show the same fact twice with one of them stale. Cordon
and uncordon always answer with an empty list by design, because a cordon
governs whether new placements may be made rather than what a Blueprint asks
for, and the reconciler leaves existing deployments where they are. The comment
on that type is the point: the risk there is a future reader building a
consumer for a list that is never populated.
Two places do report something. Deleting a node and adding a node label both
re-place Blueprints, and both now say how many. The count only, and only when
it is not zero: an empty list means both that nothing moved and that the
projection faulted after the write had already committed, so it can never be
reported as the first.
Blueprint detail gains the same unreachable-application card the Git source
panel has. A Blueprint with no live application row is a fact nothing in the
product could previously express, which is a different problem from having two
ways to say the same thing.
* fix(git): close the gaps review found in the GitOps frontend consumers
Six defects, each verified against the code before being fixed.
The pending-map read assumed every /git-sources row carries a projection. That
route is proxied, so a node predating the model answers rows without one, and
dereferencing it threw inside the loop. The catch swallowed the throw, the map
was never set, and every stack's Git indicator froze at its previous value for
as long as that node was selected. The field is optional now and a row without
one falls through to the same branch as a row with nothing to project.
That fallback also branched on the arm alone, so a projection reporting an
unreachable application plus a stale commit pointer was reported as a candidate
ready to apply. It now excludes a faulted projection: a fault means an
application was expected and could not be read, so the pointer is not evidence
that anything is ready, and naming a state there would be a guess.
Detaching a Git source cleared the source but not the revision, and the pending
card is derived from the revision alone. The panel kept advertising a waiting
commit for a stack Git no longer managed, behind a Review button that silently
did nothing. A read that threw had the same problem across stacks: the panel is
reused, so stack A's state could render under stack B's header.
The panel also lost the flat-pointer fallback the sidebar keeps, so the two
surfaces disagreed: the sidebar would show an indicator and clicking through
showed no card. Both now apply the same rule.
Node delete parsed the success body unguarded. The delete has already committed
at that point, so a malformed body would have reported a completed deletion as
a failure and skipped the refresh, leaving the deleted node on screen.
Tests: five mutations that previously survived now fail. The zero-pixel claim
compared the wrong element and passed when the indicator vanished entirely; the
stale-read test asserted during the loading window, when the body is skeletons
regardless; the sidebar passthrough test proved only that an indicator existed,
not that the state reached it. Added coverage for the drift row, an unknown
node, the not-applicable source guard, the save clearing, the Review button, and
a live application carrying a caveat, which must not read as a fault.
The fixture set drops two source statuses whose identity defaults describe a
state they cannot be in, and gains a drift-item builder.
* refactor(git): share the GitOps fault card and live-facet derivation
The unreachable-application card was built inline on three surfaces, and the
copies had already drifted: the Git source panel used one icon while the Drift
tab and the Blueprint sheet used another, so the same failure rendered two
different ways. It is one component now, which also owns the state key and the
test id those surfaces assert on.
Deriving the live source facet was likewise repeated, along with the two
semantic decisions behind it: the absent arm has no facets, and a source facet
of not applicable means a Blueprint owns the stack rather than that something
is missing. Both now live next to pendingSourceStatus, which already encoded
the same exclusion.
The Git source panel's seven derived values collapse to three, with the
pending-commit rule extracted to a named function so the four cases read
straight through instead of as nested ternaries. Behaviour is unchanged in
every case.
The two flat-pointer fallbacks are deliberately not unified: they look alike
but the panel also treats a live application whose source facet is not
applicable as unanswered, and merging the predicates would change what that
case renders.
* feat(git): count and announce committed GitOps transitions
Every history row that is actually inserted now produces one metric
increment and one state-invalidate event, so the surfaces that read GitOps
state hear about a change instead of waiting for the next poll.
Announcement is buffered and drained on a macrotask rather than fired
inline. better-sqlite3 is synchronous, so waiting for the macrotask puts
the drain after the transaction that wrote the row, and after any outer
transaction wrapping it, without having to detect which nesting depth it
is in. The drain confirms each row is still present before announcing it,
so a rolled-back transaction says nothing on its own, and the insert
declines to queue a dedupe replay, so a retry says nothing either. The
broadcaster is injected at startup rather than imported, keeping the
GitOps layer free of a cycle back into the notification stack.
Counters are process-local and in-memory, and their keyspace is finite by
construction: history stages are now a closed union that the build
enforces, and outcomes were already a closed set. The payload names no
stack, node, repository, or actor, because a counter carrying those would
be an audit trail with no retention rules and no per-row authorization,
which is what the history routes provide. GET /api/gitops-metrics is
Admin-only and instance-local, so selecting a node answers with that
node's counters.
On the client, a gitops-scoped invalidate refreshes the derived state
through a 250ms trailing window, matching the existing stack refresh. One
operation commits several transitions in a row and a first-boot migration
commits a great many, so refetching per event would thrash the API for a
picture that only settles at the end.
* feat(git): show GitOps source state on the stack dashboards
The dashboards list every stack on a node, which is where a fleet-wide
reading of Git state is most useful and where, until now, a Git-backed
stack looked exactly like a local one. Each row that the model has
something to say about carries a chip naming its source state.
The chip reads from the same status vocabulary the panels and the sidebar
use, so a stack cannot be "pending update" in one place and something else
in another. It sits beside the stack name rather than replacing the source
column: the column says where the files come from, the chip says what
GitOps makes of them, and those are different facts. The label is a word
and the title is a whole sentence, so the state never rests on colour
alone; the phone rows keep the word on screen, since touch has no hover.
The join is by stack name, which is what the dashboards have, and its
source is the same proxied route the sidebar reads. A row without a
revision, a Blueprint-owned application, and a projection fault all leave
the stack unbadged rather than inventing a state for it. State arrives by
announcement rather than by poll, since it only moves on a transition, and
a node switch blanks the map first: stack names repeat across nodes, so a
slow answer for the node just left is discarded rather than allowed to
label the wrong stacks.
* feat(git): say which part of a GitOps state could not be proven
Eighteen conditions can qualify a live projection: a manifest that does
not match the branch configured now, an approval that could not be
restored after a recovery, an artifact record that has gone. Every one was
already recorded and none of them reached the operator, so a state with a
hole in it read exactly like one without.
Each now has operator wording under the state it qualifies, checked
against the site that emits it rather than against the code's name, since
several names describe something narrower or wider than the condition
actually tested. The stored messages stay out of the UI: they are written
for a log reader, and some are raw decoder errors.
The presentation is deliberately quiet. A caveat is not a failure: the
state above it is real, and one piece of evidence behind it is missing, so
the reader learns which part to distrust without being told the whole
thing is broken. Faults keep their own card, because those replace the
state rather than qualifying it, and the two arms are read through
separate helpers so neither can be rendered as the other.
A code with no wording names itself rather than vanishing, so a node
running ahead of this build degrades to something honest instead of
reporting full confidence in a state its own backend flagged.
* fix(git): record a stateful first placement and keep a half-built Blueprint editable
Two faults that today's product cannot reach, and that the Git-backed
Blueprint mode would reach immediately.
The live-application lookup answers with an application that is active or
still being created, because its other callers ask whether the Blueprint
already holds the live slot, where a half-built row counts. Three
producers passed that answer straight into transitions that accept only an
active application and reject anything else by throwing, inside the
caller's own transaction. A Blueprint edited or pinned while its
application was still being created would have failed with a server error
and rolled the Blueprint write back with it. The producers now narrow to
what they actually require, through one shared predicate, leaving the slot
check honest.
Separately, a stateful placement is held for operator review before
anything is deployed, so no target exists when that hold is recorded, and
the observation was dropped for want of one. The hold left no trace: no
history row, and nothing on the target to say the node had been asked to
hold anything. First contact now creates the target, exactly as the first
deploy does. Nothing else changes with it: no intent, no generation, no
operation, and connectivity stays unset, because a node that has only been
asked to hold something has not been contacted. Observations that follow a
deploy already have a target, so they are unaffected, and a drift or evict
report for a node nothing was ever sent to is still dropped.
* fix(git): keep the Git source panel's state after a save
The save answers with the source row and no revision, so the panel had
nothing to replace its copy with and dropped it. The result was a stack
that had just been reconfigured showing no GitOps state at all until the
panel was reopened, which reads as a stack the model knows nothing about
rather than one whose state has just moved.
It re-reads instead. Keeping the old copy was not an option either: a
material configuration change clears the staged candidate server side, so
the state genuinely has moved and the panel would have gone on offering a
commit that is no longer there. Only the server can say what replaced it.
The alternative was returning the revision on the save itself. That route
is proxied, and the identity hop rewrites node numbering for an enumerated
set of routes, so adding a revision to a response outside that set would
hand back a remote node's numbering unrewritten. Re-reading costs one
request on a low-frequency action and needs no change to the hop.
* docs(git): document what GitOps state means for an operator
A commit SHA says which files Sencho wrote. It does not say whether that
commit was accepted, whether something newer is waiting, or whether an
operation was interrupted halfway, and until now the docs had no words for
any of that.
Git Sources gains a source-state table and a section on what happens when
part of a state could not be proven, since a stack with missing evidence
looks identical to one with complete evidence unless the product says
otherwise. Drift Detection gains the gitops section and explains why it
answers a different question from the two signals above it: those compare
files and containers as they are now, this reports what has been proven
over time. The dashboard page describes the state chip and, importantly,
that it and the row tint are independent, so a healthy row carrying
"pending update" is not a contradiction.
The Blueprint page now separates its revision counter from GitOps state.
The counter labels the spec; it says nothing about whether a given node
has got there, and reading it as fleet truth is the mistake the note
prevents.
The tutorial's verification step names the source state it should show,
so a reader can tell the difference between files written and a commit
accepted.
Audited and left unchanged: health-gated updates and fleet federation.
Neither presents a SHA or a revision as canonical GitOps truth, so neither
needed correcting.
* docs(git): name which saves actually clear a staged commit
The note claimed saving the form always clears the staged commit and moves
the state to reconcile required. Only a change to what gets materialized
does that: the repository, the ref, the compose paths, the project
directory, or the env sync. Changing the token or the apply behavior
leaves a staged commit alone, because neither changes what would be
written, and telling an operator otherwise would have them pull again for
nothing.
* fix(git): say when a committed transition has nobody to announce it to
Two silences worth breaking, both found reviewing the error paths in this
slice.
A server that never installs the event sink still counts every transition
and still writes every history row, so the only symptom is that no client
ever refreshes and the UI is quietly as stale as it was before any of this
existed. It now says so, once rather than per row, since a boot migration
would otherwise fill the log with one fact.
The dashboard's source-state fetch dropped a non-ok response without a
word. A refusal there freezes every badge at its last value, which looks
exactly like a fleet where nothing has changed, so the status code is
logged rather than inferred from badges that stopped moving.
* fix(git): close the defects review found in the announcement slice
Five findings, two of them able to reach an operator.
The event sink was installed one line after the deletion reconcile, which
tombstones applications and targets and awaits inside its own loop. A
drain therefore landed while the sink was still absent, so those boot-time
transitions were counted and never announced, and the warning added for an
unwired sink would have fired on every boot with a prepared deletion
intent: the fastest way to teach an operator to ignore it when it means
something. The sink now precedes every reconcile and migration pass.
The first-placement target was created outside the transition that records
it. The observation runs in its own savepoint and can refuse, and the
caller deliberately lets the deployment commit whatever the record says,
so a refusal left an active target with no generation, no stage and no
history: a placement relationship the model never established, which the
delete path would later tombstone as if it were real. Both writes now
succeed or fail together.
The badge looked its status up in a map closed at compile time, behind two
casts, while the value arrives over a proxy from a node that may run a
newer vocabulary. An unmapped key dereferenced undefined inside a stack
row and would have taken the whole table down with it. The facet and its
status are now a discriminated union, the lookup is optional, and an
unrecognised status renders nothing, which is what the join already does
for a stack it has no state for.
The dashboard's source-state loop derived each row inside the shared
try, so one row in an unexpected shape abandoned the loop and froze every
badge at its last value with nothing on screen to say so. Each row is now
guarded on its own and the count of unreadable ones is logged.
Two tests claimed things they did not check: both asserted a word was on
screen with a matcher that also matches screen-reader-only text, so a
compact badge would have kept them green, and one compared a label by
containment where the shorter label is a prefix of the longer. Both now
assert the visible node by equality. Added coverage for the window event
the dashboard actually refetches on, an unknown status, a row that cannot
be read, node-switch blanking, and a caveat recorded twice. One fixture
built a status pair the deriver cannot emit and was corrected.
Also: the guard narrowing Blueprint producers to an active application
excludes nothing today, because every Blueprint-mode application is
inserted as active and only Direct mode can be creating. The guard stays,
since the getter's slot semantics and the transitions' requirement have
drifted apart once already, but its rationale and its tests now say
plainly that they pin a defensive guard rather than reachable behaviour.
* refactor(git): make three runtime guards visible to the compiler
Type review found the same shape three times: a correct guard whose
necessity was asserted in a comment the type system could not see. This
project does not set noUncheckedIndexedAccess, so each of these read as
dead code to anything that trusts the types, and the guards protect the
behaviours that matter most here.
The limitation copy map typed its values optional, so the fallback for an
unrecognised code is now something the compiler requires rather than
something a comment explains. The badge reads its status through partial
views of the two vocabularies, so a miss is a fact TypeScript derives:
deleting the guard that stops one unknown status taking down a stack list
now fails the build, where before it compiled. Neither needed a cast.
The event payload has to be a type alias rather than an interface, because
only the former gets an implicit index signature and the broadcaster takes
an open envelope. That requirement now fails at the declaration instead of
surfacing in the startup wiring as an unexplained index-signature error.
Also added the reverse of the copy-coverage check: a code retired from the
backend leaving stale wording behind was invisible, since the fallback
only fires for entries that are missing rather than ones that linger.
* fix(git): make the state cards survive an unknown status, and correct the copy
Comment review found the protection added for the dashboard chip was
claimed more widely than it held. Five other surfaces indexed the status
vocabularies directly and passed the result straight to a card that
dereferences it, so the exact input the comments said would take a list
down still threw in the Git source panel, the Drift tab, the sidebar
tooltip and the stack anatomy row. The card now takes an optional state
and renders nothing without one, which also means a new surface cannot
reintroduce the dereference by rendering a card the ordinary way, and the
remaining four read through the lookups.
Copy corrections, each against the site that emits it rather than the
name of the code:
The manifest-identity caveat said the manifest named a different
repository or branch. It also fires when the manifest has no identity
block at all, or identifies another node or another stack, so it now says
it does not identify this stack on this node from the repository
configured now. The last-known-good artifact caveat covered a mismatch but
not the row being gone, which is the other half of the same condition.
Doc corrections: a Blueprint-delivered stack was described as carrying the
dashboard chip and a source state on the Drift tab, and it carries
neither, because a Blueprint has no Git source of its own. Per-node
Blueprint state was attributed to the detail sheet's GitOps section, which
only reports what could not be read or proven; the Deployments table is
where per-node status lives. Clearing a staged commit was said to always
land on reconcile required, but a stack that has never had a commit
accepted lands on never reconciled. The caveat lines were described as
sitting under the state when they render above the form. And the pending
banner was still documented as webhook-only, which this branch changed:
it now appears for any staged commit and its heading is the source state.
Also corrected two claims in the code: the observation branch does not
skip for a Blueprint that migration brought in, because migration creates
no targets, and the only refusal reachable from that call site is a
tombstoned target.
* refactor(git): simplify the announcement slice without moving any behaviour
A simplification sweep over the new code, all of it behaviour-preserving
and re-checked by breaking each guard afterwards.
The drain now separates policy from mechanism: it decides whether a row
survived, counts it, and either warns or hands it to a small announce
helper that owns the envelope and its error handling. The row-to-state
pass in the dashboard hook lifts out of the fetch, so one function does
the request and another does the derivation. The caveat dedupe collapses
to a set, which preserves first-insertion order and so keeps the ordering
its own docstring promises. The badge renders one span with a conditional
class rather than two that differed only by class. The metrics service
drops an internal type that was byte-identical to the one it exports.
One assertion goes: the observation branch tested membership and then
re-asserted the same fact to index the stage map, and a type predicate
lets the compiler carry it instead.
Everything the sweep was told to leave alone is intact: the buffered
drain, the row-existence re-check, the injected sink, the one-shot
warning, the partial-view lookups and their guards, the optional copy map
with its fallback, and the generation counter with its per-row guard.
* fix(git): close audit findings in the GitOps revision model
Five corrections from the pre-merge audit of this branch.
Migration no longer certifies a commit it cannot prove. The manifest read
now carries its resolved commit, and a manifest that names a different
commit than the source row records as applied is refused: the applied
directory materializes one commit while the row names another, so
trusting them together would mint a generation pointing at files that are
not the ones it claims. A manifest that names no commit at all, which is
what adoption writes, is a separate answer rather than a disagreement,
because reporting it as one would name a commit the manifest does not
contain. Both are recorded as evidence and the projection asks for a
fetch.
Create cleanup proves containment against the real filesystem before it
deletes. Every check on these paths was lexical, so a symlink or Windows
junction above the target read as contained while the recursive delete
followed it out of the managed area. Resolution keeps "is not there"
apart from "could not be read": only a genuinely missing path lets the
walk climb to an ancestor, because treating an unreadable one as absent
would infer containment for the single path whose link status could not
be established. Both sides are resolved, so relocating the data directory
onto another volume keeps working, and an area that does not exist at all
is nothing to delete rather than a suspected escape. The marker write
takes the same barrier as the marker delete, so a link cannot be written
through and then refused on the way out, wedging the stack name.
Startup stops while an interrupted create is unresolved, matching the
restore reconcile above it. A create that could not be settled leaves a
stack directory the deploy path cannot tell apart from a finished one, so
the alternative to stopping is letting a scheduler, webhook or operator
act on a half-built stack. Only that blocks: once the staged directories
are gone the create is torn down, and a staging marker nobody could
unlink is reported rather than thrown, so one failed unlink is not the
difference between an instance that boots and one that does not.
Per-stack history no longer exposes a predecessor through a reused stack
name. The grant covers the application holding the name now, which is
what keeps a stack's own entries readable while it is still being
created; every other row on that name is classified per row, so entries
belonging to an application that held the name earlier need the audit
permission, as they already did across stacks. A detached predecessor is
still readable, which is the classifier's own standing decision about
detach rather than a gap here, and is now pinned by a test.
Blueprint observations reach a reader. The reconciler recorded state
review, evict blocked, drifted and correcting against the target and
nothing projected them, so a deployed Blueprint could report itself as
never applied. The runtime facet now projects all four, below the states
a live or failed mutation puts the target in and above the pointer
checks, and any later transition supersedes the observation. The map is
declared total over the stages the reconciler can record, so a stage
added without a projection fails the build rather than silently reading
as never applied again.
* fix(git): confine managed-area cleanup to each stack's own location
Cleanup proved only that a resolved path landed somewhere inside the
managed area. That is satisfied by every other stack and every other node
in it, so a link from one stack's generations directory into another's
passed the check while the delete took a generation belonging to someone
else. The check is now positional: the managed area is resolved once, so
relocating the data directory onto another volume still works, and every
segment below it must be reached without redirection. The create-path
sinks use it on the write as well as the delete, so a claim cannot be
written through a link and then refused by the hardened delete, and the
manifest service's pre-existing deletion sinks (generation pruning,
boot-sweep orphan reaping, detach staging and finalization, and
whole-area removal, including after a restored snapshot) run it before
their recursive deletes too. Each of those sinks also keeps a literal
containment comparison beside the positional check, because static
analysis credits only a comparison at the call and reports the delete
otherwise. A refusal names both paths in the log, because it can hold
the boot gate.
Boot recovery dropped the checkpoint of an application that had left the
creating state without first clearing its staging marker. That left a
claim on the stack name with nothing to retry it, and every later create
for that name was refused by a marker nothing could remove. The settled
branches now share one exit, so the marker ordering holds for all of them.
A detached application's history was readable on a stack grant because
its files are still the stack standing at that name. Nothing in these
tables can prove it still is: a Blueprint successor records the name off
the application row, and a plain Compose stack recreated at the name
leaves no trace at all, so no detection-based allowance can be made
sound. Detach now moves an application's trail to the audit audience
outright, the same answer deleted and creating predecessors get, while
reading the stack itself stays where it was.
* fix(gitops): close four audit findings in revision state and identity proxy
A1: derive runtime and health against target desired_generation_id
- deriveRuntime returns 'applied_not_deployed' when desired != deployed, so a
stale deployment stays deploy-pending instead of reading synced_and_healthy
- deriveHealth judges against desired_generation_id falling back to deployed
when null; control case proves null-desired rows unchanged
- collectRuntimeDrift emits the plan-pinned runtime drift item for exact/
qualified observation mismatches; equal/non-comparable observations emit
nothing; ordering pin keeps deploy question first
A2: stop upstream 304 responses from bypassing hub rewrite and reauthorization
- identity proxy answers no-store on every terminal (rewrite, 204, 304,
generated failures); validators (etag, last-modified, cache-control,
expires, vary) no longer forwarded
- conditional request headers (if-none-match, if-modified-since, if-match,
if-unmodified-since) stripped on the identity-hop branch of the shared
proxyReq handler; streaming hop untouched (optimistic-concurrency file writes
depend on If-Match/If-Unmodified-Since)
- supertest integration test drives the real middleware through a loopback
capture server and asserts if-none-match absent while accept survives
S1: migrate legacy query/userinfo URLs via secret-free path
- parseLegacyRepoUrl strips userinfo, query, fragment instead of refusing;
parseHttpsRepoUrl remains the strict gate for user-driven paths
- migrationDirectSourceIdentity uses the tolerant parser; operational
stack_git_sources.repo_url untouched; fingerprint convergence proven
- trusted-manifest-on-legacy-URL test covers the worst real-world instance
S2: emit runtime artifact drift for current evidence
- top-level drift array carries the seven-class runtime item; frontend
comments updated to reflect backend now emits runtime drift
All regression tests added and passing. CI green (7,374 backend + 2,790
frontend tests; only pre-existing Windows EBUSY teardown failure unrelated).
* test(frontend): fix DriftPanel drift item rendering test
Update test assertions to match the new backend drift item format.
The function now renders artifact_set expected
identities as 'artifact <id> · <qualification>' and runtime_artifact
observed identities as the raw identity string.
* fix(gitops): emit a runtime drift item for desired-versus-deployed mismatch
The projection reported applied_not_deployed with no entry in drift, so
the canonical drift list contradicted the runtime facet it travels with.
collectRuntimeDrift now emits the r27 generation-mismatch item for that
state: desired generation as expected, deployed generation as observed,
ComposeService as owner. The action mirrors what availableActions offers:
deploy, unless an application-level fetch or apply is in flight or a
recovery is in progress, which withhold deploying without removing the
fact of the mismatch. The item clears once the desired generation deploys.
The stale-deployment test pins the exact item shape, its stability across
re-derivation, its convergence removal, and the withheld-action case; the
ordering pin keeps the artifact observation suppressed while the deploy
question stands. Frontend fixture aligns its artifact-mismatch example
with the backend's none action.
* fix(gitops): keep generation drift visible across failure and pause states
The desired-versus-deployed drift item was keyed to the applied_not_deployed
runtime status, but paused, failed, recovering, interrupted, and in-flight
statuses all outrank the pointer comparison in the deriver. A failed deploy
of a new generation over a running older one therefore dropped the drift
report exactly while the old workload was still serving.
The item is now judged from the pointers themselves: known and different
means reported, whatever presentation status the target carries. Retired
targets are excluded because nothing can rebind them, so their surviving
pointer divergence would be permanently unresolvable noise. The action
follows what the payload offers: deploy only when no application-level
operation or recovery withholds it and availableActions lists deploying,
none otherwise.
The failed-redeploy regression test pins the exact item shape after a
pre-mutation deploy failure, its stability across re-derivation, continued
artifact suppression for the replaced workload, and convergence removal;
a second pin keeps retirement silent.
* fix(gitops): complete the available-action legality matrix
The action list transcribed only part of the approved rules. An
interrupted apply offered apply without checking that its recorded
generation was still the current candidate; deploy fired for any target
reading applied_not_deployed regardless of target mode, with no retry
for an interrupted Direct deploy and none of the Blueprint interruption
rules; approve_legacy sat in the type union with no producer, leaving a
pending Inline placement review permanently unactionable through the
projection; and drift items inherited whatever deploy recommendation any
sibling target earned.
Actions now follow the matrix. Deploy is decided per target through one
predicate shared by deriveActions and the drift items: a paused or
failed sibling can no longer inherit another target's legal deploy, a
Blueprint-mode divergence never advertises Direct deployment, an
interrupted Direct deploy retries against the generation still applied,
and an interrupted Blueprint deploy or withdraw repeats only while its
recorded intent revision and rollout candidate still equal what the
application requires, where an absent pair matches because inline
Blueprints carry no candidate until the later-phase producer lands.
Interrupted apply requires the recorded generation to still be the
current candidate and the source not to have been suspended meanwhile.
A reachable Inline placement review now offers approve_legacy.
Six tests pin the matrix: sibling isolation across failed and paused
targets, the Blueprint mode guard, vacuous and matched Blueprint retry
identities plus the superseded case, matching-versus-stale Direct deploy
and apply interruptions, and the legacy review action.
* fix(gitops): withhold apply retry while the candidate is blocked
The interrupted-apply retry checked that the recorded generation still
matched the current candidate and that the source was not suspended, but
a later classification can also block that candidate, and applyStarted
refuses a blocked one outright. The retry is offered only when every
precondition the transition enforces still holds.
* fix(gitops): prove apply preconditions and limit fetch to Direct
The interrupted-apply retry checked identity, suspension, and blockage
but never loaded the candidate generation, so apply could be recommended
for a row that was missing, owned by another application, or built from
a superseded materialization fingerprint; applyStarted refuses all
three, and the positive fixture itself described a state no transition
would accept. The gate now proves existence, ownership, and fingerprint
before offering apply. Its fixture is rebuilt around a transition-legal
candidate whose projected action is executed against applyStarted, with
negatives for every refusal including suspension.
Fetch was offered to any application whose source looked unreconciled,
but the approved rules reserve fetch for live Direct applications; a
Git-backed Blueprint divergence now advertises nothing of the sort.
Controls pin both modes against equivalent source state.
* fix(gitops): fail closed when a candidate row is missing or foreign
Ordinary candidate_ready checked a fingerprint only when the candidate
generation row existed, so a dangling id or one owned by another
application fell through to ready and offered an apply that
applyStarted would refuse on sight. The source now requires the row to
exist under this application before readiness, records a limitation
for the anomaly with its operator copy mirrored in the frontend, and
reports reconcile-required instead. The Blueprint retry test also pins
that a superseded rollout candidate alone, with the intent still
matching, suppresses deployment.
* fix(gitops): fail accepted-generation derivation closed
The acceptance branch compared fingerprint and sha only when the
accepted generation row loaded, so a dangling id or one owned by
another application fell through and reported
application_generation_accepted with no evidence behind it, while also
withholding fetch as the recovery. The source now requires the row to
exist under this application before any comparison, records an
accepted_generation_invalid limitation naming the pointer with its
operator copy and inventory entry mirrored on the frontend, and
reports reconcile-required instead. Five scenarios pin valid, missing,
foreign, fingerprint-mismatched, and sha-mismatched acceptances.
* fix(gitops): enforce canonical state in dismiss, reconcile, and pulls
- Route dismiss-pending through the canonical dismissed transition so a
refusal while an operation is in flight surfaces as 409 instead of
clearing staged state behind the model's back
- Treat tombstoned targets as authoritative in both reconciler decision
surfaces so ticks never redeploy onto a placement the model severed;
an explicit deploy revives the target and records the revival delta
- Keep application-level rows in node-scoped history pages so proxied
hub views do not read as if the application never came into being
- Expose repoIdentity and configuredRef on history items to match the
server-side identity filters
- Stand down when a pull resolves to exactly the live candidate (same
commit, source fingerprint, plan verdict); pulls after an acceptance
still open a fresh staging generation as the apply target
* fix(rbac): classify git-source manifest reads as stack:read
GET /stacks/:name/git-source/manifest had no rule in the hub route
classifier, so proxied requests were refused with 403 before reaching
the remote node even for callers holding stack read access.
* fix(gitops): surface an error toast when dismiss is refused as in-flight
Dismissing a pending Git update while a fetch or apply is still running
now returns 409 OPERATION_IN_FLIGHT, but the frontend handler had no
else branch for a non-2xx response, so the refusal was swallowed with
no operator feedback at all. Add the same error-toast pattern already
used by the sibling apply handler, and order the success toast before
its side effects so a downstream failure cannot invert the outcome.
3879 lines
168 KiB
TypeScript
3879 lines
168 KiB
TypeScript
import { Router, type Request, type Response, type NextFunction } from 'express';
|
|
import { z } from 'zod';
|
|
import path from 'path';
|
|
import os from 'os';
|
|
import crypto from 'crypto';
|
|
import zlib from 'zlib';
|
|
import { promises as fsp } from 'fs';
|
|
import * as tar from 'tar-stream';
|
|
import { inspect } from 'node:util';
|
|
import YAML from 'yaml';
|
|
import multer from 'multer';
|
|
import { FileSystemService } from '../services/FileSystemService';
|
|
import { StackFileRootsService, STACK_SOURCE_ROOT_ID, stackSourceFileRoot, type StackFileRoot } from '../services/StackFileRootsService';
|
|
import { FileRootGateway } from '../services/FileRootGateway';
|
|
import { ComposeService, getComposeRollbackInfo } from '../services/ComposeService';
|
|
import { StackUpdateOrchestrator, shortImageId, type OrchestratorResult } from '../services/StackUpdateOrchestrator';
|
|
import DockerController, { type BulkStackInfo } from '../services/DockerController';
|
|
import { DatabaseService, type StackDossierFields } from '../services/DatabaseService';
|
|
import { CacheService, type CacheFetchOutcome } from '../services/CacheService';
|
|
import {
|
|
UpdatePreviewService,
|
|
isAuthoritativeNegativePreview,
|
|
buildDetectionDisabledPreview,
|
|
} from '../services/UpdatePreviewService';
|
|
import { GitSourceService, GitSourceError, repoHost as gitRepoHost } from '../services/GitSourceService';
|
|
import { repoUrlRejectionMessage } from '../services/gitops/repoIdentity';
|
|
import { enforcePolicyPreDeploy } from '../services/PolicyEnforcement';
|
|
import { buildStackDriftReport, type DriftFindingKind, type StackDriftReport } from '../services/DriftDetectionService';
|
|
import { DriftLedgerService, type DriftTemporal } from '../services/DriftLedgerService';
|
|
import { ComposeDoctorService } from '../services/ComposeDoctorService';
|
|
import { RULE_IDS, isPreflightNoteFinding } from '../services/preflight/rules';
|
|
import { parseServiceImages, isPreflightAckActive } from '../utils/preflight-ack-filter';
|
|
import type { PreflightAckExpiryMode } from '../services/DatabaseService';
|
|
import { buildStackNetworkFacts } from '../services/network/composeNetworkInspector';
|
|
import { buildStorageInventory } from '../services/storage/inventory';
|
|
import { probeComposeDiscovery } from '../services/ComposeDiscoveryService';
|
|
import { buildEffectiveAnatomy } from '../services/effectiveAnatomy';
|
|
import { buildEffectiveServiceModel } from '../services/effectiveServiceModel';
|
|
import { buildEnvInventory } from '../services/EnvInventoryService';
|
|
import { buildStackLabelInventory } from '../services/LabelInventoryService';
|
|
import { labelInventoryOptionsFromRequest, requireRevealAdmin } from '../helpers/labelInventoryRequest';
|
|
import { EXPOSURE_INTENTS, type ExposureIntent } from '../services/network/types';
|
|
import { UpdateGuardService, SingleServiceUpdateReadinessError } from '../services/UpdateGuardService';
|
|
import { HealthGateService } from '../services/HealthGateService';
|
|
import { DeployedStackDeletionService } from '../services/DeployedStackDeletionService';
|
|
import { StackUpdateRecoveryService } from '../services/StackUpdateRecoveryService';
|
|
import { classifyFailure } from '../services/updateGuard/failureClassifier';
|
|
import { requirePermission, checkPermission } from '../middleware/permissions';
|
|
import { NotificationService, type NotificationCategory } from '../services/NotificationService';
|
|
import { StackOpLockService, type StackOpAction } from '../services/StackOpLockService';
|
|
import { StackOpMetricsService, type StackOpAction as StackMetricAction } from '../services/StackOpMetricsService';
|
|
import { FileExplorerMetricsService, type FileExplorerOp } from '../services/FileExplorerMetricsService';
|
|
import { isValidGitSourcePath, isValidStackName, isValidServiceName, isValidRelativeStackPath } from '../utils/validation';
|
|
import { normalizeBulkPaths, destWithinAnySource } from '../utils/bulkPaths';
|
|
import { getErrorMessage } from '../utils/errors';
|
|
import { isDebugEnabled } from '../utils/debug';
|
|
import { sanitizeForLog } from '../utils/safeLog';
|
|
import { logDebugTiming } from '../utils/requestTiming';
|
|
import { sendGitSourceError } from '../utils/gitSourceHttp';
|
|
import { buildPolicyGateOptions, runPolicyGate, triggerPostDeployScan, describePolicyBlock } from '../helpers/policyGate';
|
|
import { parseComposePreview, type ComposePreview } from '../helpers/composePreview';
|
|
import { filterContainersByComposeService } from '../helpers/composeServiceMatch';
|
|
import { invalidateNodeCaches } from '../helpers/cacheInvalidation';
|
|
import { invalidateFleetUpdateCache } from '../helpers/fleetUpdateCache';
|
|
import {
|
|
ImageUpdateService,
|
|
UPDATE_VERIFICATION_INCOMPLETE_WARNING,
|
|
} from '../services/ImageUpdateService';
|
|
import { parseComposeSelection, defaultEnvPath } from '../helpers/gitSourceSelection';
|
|
import {
|
|
applyFleetSnapshotFiles,
|
|
fleetSnapshotApplyConflictCode,
|
|
getCodedError,
|
|
selectFleetSnapshotApplyFiles,
|
|
} from '../helpers/applyFleetSnapshotFiles';
|
|
import { resolveStackEnvSources, discoverStackLocalEnvFiles } from '../helpers/envFileResolution';
|
|
import { STACK_STATUSES_CACHE_TTL_MS } from '../helpers/constants';
|
|
import { getTerminalWs, DEPLOY_SESSION_HEADER } from '../websocket/generic';
|
|
import {
|
|
isSelfStack,
|
|
isSelfStackByIdentity,
|
|
refuseIfSelfStack,
|
|
resolveSelfStackIdentity,
|
|
selfStackProtectedBulkResult,
|
|
UNRESOLVED_SELF_STACK_IDENTITY,
|
|
} from '../helpers/selfStackGuard';
|
|
import { getActiveCapabilities, STACK_DOWN_REMOVE_VOLUMES_CAPABILITY, SERVICE_SCOPED_UPDATE_CAPABILITY } from '../services/CapabilityRegistry';
|
|
import { ServiceUpdateRecoveryService } from '../services/ServiceUpdateRecoveryService';
|
|
import { classifyStackApiPath } from '../helpers/stackRouteAuth';
|
|
import { projectManagedStackRevision } from '../helpers/gitopsResponse';
|
|
import type { GitOpsRevisionProjection } from '../services/gitops/types';
|
|
|
|
// Authenticated users with edit permission can write arbitrarily large compose
|
|
// files. Refuse to YAML.parse anything beyond this bound so a malformed (or
|
|
// adversarial) file cannot exhaust heap during an env or service lookup.
|
|
const MAX_COMPOSE_PARSE_BYTES = 1_048_576; // 1 MiB
|
|
|
|
/**
|
|
* Diagnostic-mode log helper. Wraps console.log so non-essential informational
|
|
* lines (deploys completed, files saved, stacks created) stay silent in
|
|
* production and only appear when developer_mode is on. console.warn and
|
|
* console.error remain unconditional - they are operational signal, not noise.
|
|
*/
|
|
function dlog(...args: Parameters<typeof console.log>): void {
|
|
if (isDebugEnabled()) console.log(...args);
|
|
}
|
|
function notifyActionFailure(action: string, stackName: string, error: unknown, actor: string): void {
|
|
const message = getErrorMessage(error, `Failed to ${action} stack`);
|
|
NotificationService.getInstance()
|
|
.dispatchAlert('error', 'deploy_failure', message, { stackName, actor })
|
|
.catch(err => console.error('[Stacks] Failed to dispatch failure notification for %s:', sanitizeForLog(stackName), err));
|
|
}
|
|
|
|
function notifyActionSuccess(category: NotificationCategory, message: string, stackName: string, actor: string): void {
|
|
NotificationService.getInstance()
|
|
.dispatchAlert('info', category, message, { stackName, actor })
|
|
.catch(err => console.error('[Stacks] Failed to dispatch activity for %s:', sanitizeForLog(stackName), err));
|
|
}
|
|
|
|
const STACK_OP_PRESENT_PARTICIPLE: Record<StackOpAction, string> = {
|
|
deploy: 'deploying',
|
|
down: 'taking down',
|
|
restart: 'restarting',
|
|
stop: 'stopping',
|
|
start: 'starting',
|
|
update: 'updating',
|
|
rollback: 'rolling back',
|
|
backup: 'backing up',
|
|
delete: 'deleting',
|
|
git_apply: 'applying Git changes',
|
|
};
|
|
|
|
function linkStackUpdateRecoveryGate(recoveryId: string | null | undefined, healthGateId: string | null): void {
|
|
if (!recoveryId) return;
|
|
StackUpdateRecoveryService.getInstance().linkGateOrRetain(recoveryId, healthGateId);
|
|
}
|
|
|
|
function tryAcquireStackOpLock(
|
|
req: Request,
|
|
res: Response,
|
|
stackName: string,
|
|
action: StackOpAction,
|
|
): boolean {
|
|
const user = req.user?.username ?? 'system';
|
|
const result = StackOpLockService.getInstance().tryAcquire(req.nodeId, stackName, action, user);
|
|
if (!result.acquired) {
|
|
res.status(409).json({
|
|
error: `${stackName} is already ${STACK_OP_PRESENT_PARTICIPLE[result.existing.action]}`,
|
|
code: 'stack_op_in_progress',
|
|
inProgress: {
|
|
action: result.existing.action,
|
|
startedAt: result.existing.startedAt,
|
|
user: result.existing.user,
|
|
},
|
|
});
|
|
return false;
|
|
}
|
|
return true;
|
|
}
|
|
|
|
function releaseStackOpLock(req: Request, stackName: string): void {
|
|
StackOpLockService.getInstance().release(req.nodeId, stackName);
|
|
}
|
|
|
|
/** Root compose + blueprint marker on the stack-source root must not change (content or mode) while a lifecycle op holds the stack lock. */
|
|
const STACK_OP_LOCKED_ROOT_TRUST_FILES = new Set([
|
|
'compose.yaml',
|
|
'compose.yml',
|
|
'docker-compose.yaml',
|
|
'docker-compose.yml',
|
|
'.blueprint.json',
|
|
]);
|
|
|
|
function rejectIfStackOpBlocksRootTrustFileWrite(
|
|
req: Request,
|
|
res: Response,
|
|
stackName: string,
|
|
relPath: string,
|
|
root: StackFileRoot,
|
|
): boolean {
|
|
if (root.kind !== 'stack-source') return false;
|
|
const normalized = relPath.endsWith('/') ? relPath.slice(0, -1) : relPath;
|
|
if (normalized.includes('/')) return false;
|
|
const base = normalized.toLowerCase();
|
|
if (!STACK_OP_LOCKED_ROOT_TRUST_FILES.has(base)) return false;
|
|
const existing = StackOpLockService.getInstance().get(req.nodeId, stackName);
|
|
if (!existing) return false;
|
|
res.status(409).json({
|
|
error: `${stackName} is busy: another operation (${existing.action}) is already in progress`,
|
|
code: 'stack_op_in_progress',
|
|
inProgress: {
|
|
action: existing.action,
|
|
startedAt: existing.startedAt,
|
|
user: existing.user,
|
|
},
|
|
});
|
|
return true;
|
|
}
|
|
|
|
function stackFileEtag(mtimeMs: number): string {
|
|
return `W/"${Math.floor(mtimeMs)}"`;
|
|
}
|
|
|
|
function parseIfMatchMtime(raw: string | undefined): number | null {
|
|
if (!raw) return null;
|
|
const m = /(?:W\/)?"(\d+)"/.exec(raw);
|
|
if (!m) return null;
|
|
const value = Number(m[1]);
|
|
return Number.isFinite(value) ? value : null;
|
|
}
|
|
|
|
async function requireStackExists(nodeId: number, stackName: string, res: Response): Promise<boolean> {
|
|
if (!isValidStackName(stackName)) {
|
|
res.status(400).json({ error: 'Invalid stack name' });
|
|
return false;
|
|
}
|
|
const fsSvc = FileSystemService.getInstance(nodeId);
|
|
const stackDir = path.join(fsSvc.getBaseDir(), stackName);
|
|
try {
|
|
if (!(await fsSvc.hasComposeFile(stackDir))) {
|
|
res.status(404).json({ error: 'Stack not found' });
|
|
return false;
|
|
}
|
|
} catch {
|
|
res.status(404).json({ error: 'Stack not found' });
|
|
return false;
|
|
}
|
|
return true;
|
|
}
|
|
|
|
// Thin wrapper over the shared env-source resolver. Returns the absolute paths of
|
|
// the env files Compose would consult for this stack: existing declared `env_file:`
|
|
// paths (injection sources) plus the project interpolation source(s). Configured
|
|
// project env files are included as interpolation sources. The multi-file Git case
|
|
// and path validation live in resolveStackEnvSources so every consumer agrees.
|
|
export async function resolveAllEnvFilePaths(nodeId: number, stackName: string): Promise<string[]> {
|
|
const sources = await resolveStackEnvSources(nodeId, stackName);
|
|
const present = sources.envFiles.filter(f => f.existence === 'present' && f.resolvedPath);
|
|
const injection = present.filter(f => f.isInjectionSource).map(f => f.resolvedPath as string);
|
|
const interpolation = present.filter(f => f.isInterpolationSource && !f.isInjectionSource).map(f => f.resolvedPath as string);
|
|
// Dedup: interpolation sources may also appear as injection sources.
|
|
const seen = new Set(injection);
|
|
for (const p of interpolation) {
|
|
if (!seen.has(p)) {
|
|
seen.add(p);
|
|
injection.push(p);
|
|
}
|
|
}
|
|
return injection;
|
|
}
|
|
|
|
// Uploads spool to disk (not memory) so a 25 MB upload is never held in RAM.
|
|
// The temp dir lives under the OS temp root, deliberately outside COMPOSE_DIR and
|
|
// any browsable volume, so a running container never observes a half-written
|
|
// spool. SENCHO_UPLOAD_DIR relocates it (e.g. onto a larger volume).
|
|
const UPLOAD_TMP_DIR = process.env.SENCHO_UPLOAD_DIR
|
|
? path.resolve(process.env.SENCHO_UPLOAD_DIR)
|
|
: path.join(os.tmpdir(), 'sencho-uploads');
|
|
const upload = multer({
|
|
storage: multer.diskStorage({
|
|
destination: (_req, _file, cb) => {
|
|
fsp.mkdir(UPLOAD_TMP_DIR, { recursive: true })
|
|
.then(() => cb(null, UPLOAD_TMP_DIR))
|
|
.catch((err: Error) => cb(err, UPLOAD_TMP_DIR));
|
|
},
|
|
filename: (_req, _file, cb) => {
|
|
cb(null, `up-${process.pid}-${Date.now()}-${crypto.randomBytes(8).toString('hex')}`);
|
|
},
|
|
}),
|
|
limits: { fileSize: 25 * 1024 * 1024, files: 1 },
|
|
preservePath: true,
|
|
});
|
|
|
|
/**
|
|
* Best-effort cleanup of a spooled upload temp file; never throws (a failed
|
|
* cleanup must not turn a successful upload into an error). A persistent failure
|
|
* would silently grow the spool dir, so log it at diagnostic level rather than
|
|
* swallowing it blind.
|
|
*/
|
|
async function cleanupUploadTemp(req: Request): Promise<void> {
|
|
const tmp = req.file?.path;
|
|
if (!tmp) return;
|
|
// Canonical js/path-injection barrier inline with the unlink sink: the spool
|
|
// path is multer-generated within UPLOAD_TMP_DIR (a random filename), but
|
|
// static analysis taints req.file.*, so confirm containment before unlinking.
|
|
const baseResolved = path.resolve(UPLOAD_TMP_DIR);
|
|
const resolved = path.resolve(tmp);
|
|
if (!resolved.startsWith(baseResolved + path.sep)) return;
|
|
await fsp.unlink(resolved).catch((err: unknown) => {
|
|
logFileDiag('upload temp cleanup failed', { path: resolved, errorCode: fsErrorCode(err) });
|
|
});
|
|
}
|
|
|
|
function getRelPath(req: Request): string {
|
|
return typeof req.query.path === 'string' ? req.query.path : '';
|
|
}
|
|
|
|
export const stacksRouter = Router();
|
|
|
|
stacksRouter.use((req: Request, res: Response, next: NextFunction): void => {
|
|
const classified = classifyStackApiPath(req.method, `/stacks${req.path}`);
|
|
if (classified.kind === 'static') {
|
|
next();
|
|
return;
|
|
}
|
|
if (classified.kind === 'unknown-named') {
|
|
if (req.user?.role === 'admin') {
|
|
next();
|
|
return;
|
|
}
|
|
res.status(403).json({ error: 'Permission denied.', code: 'PERMISSION_DENIED' });
|
|
return;
|
|
}
|
|
if (!requirePermission(req, res, classified.action, 'stack', classified.stackName)) return;
|
|
next();
|
|
});
|
|
|
|
stacksRouter.param('stackName', (req, res, next, stackName) => {
|
|
if (typeof stackName !== 'string' || !isValidStackName(stackName)) {
|
|
res.status(400).json({ error: 'Invalid stack name' });
|
|
return;
|
|
}
|
|
next();
|
|
});
|
|
|
|
stacksRouter.get('/', async (req: Request, res: Response) => {
|
|
if (!requirePermission(req, res, 'stack:read')) return;
|
|
const startedAt = Date.now();
|
|
let outcome: 'ok' | 'error' = 'ok';
|
|
let count = 0;
|
|
try {
|
|
const stacks = await FileSystemService.getInstance(req.nodeId).getStacks();
|
|
count = stacks.length;
|
|
res.json(stacks);
|
|
} catch (error) {
|
|
outcome = 'error';
|
|
res.status(500).json({ error: 'Failed to fetch stacks' });
|
|
} finally {
|
|
logDebugTiming('[Stacks:debug]', {
|
|
route: 'GET /',
|
|
nodeId: req.nodeId,
|
|
count,
|
|
elapsedMs: Date.now() - startedAt,
|
|
outcome,
|
|
});
|
|
}
|
|
});
|
|
|
|
stacksRouter.get('/statuses', async (req: Request, res: Response) => {
|
|
if (!requirePermission(req, res, 'stack:read')) return;
|
|
const startedAt = Date.now();
|
|
let outcome: 'ok' | 'error' = 'ok';
|
|
let cacheOutcome: CacheFetchOutcome | null = null;
|
|
let dockerMs: number | null = null;
|
|
let enrichmentMs: number | null = null;
|
|
let count = 0;
|
|
try {
|
|
// Enrichment (git-source labels, self identity) is part of the cached
|
|
// payload so cache hits serve fully decorated statuses with no per-request
|
|
// work. The git label lookup failure still falls back to 'local' and must
|
|
// not take down the primary status payload.
|
|
const { value: result, outcome: fetchOutcome } = await CacheService.getInstance().getOrFetchWithMeta(
|
|
`stack-statuses:${req.nodeId}`,
|
|
STACK_STATUSES_CACHE_TTL_MS,
|
|
async () => {
|
|
const stacks = await FileSystemService.getInstance(req.nodeId).getStacks();
|
|
const stackNames = stacks.map((s: string) => s.replace(/\.(yml|yaml)$/, ''));
|
|
const dockerController = DockerController.getInstance(req.nodeId);
|
|
const dockerStartedAt = Date.now();
|
|
const bulkInfo = await dockerController.getBulkStackStatuses(stackNames);
|
|
dockerMs = Date.now() - dockerStartedAt;
|
|
const data: Record<string, BulkStackInfo> = {};
|
|
for (const stack of stacks) {
|
|
const name = stack.replace(/\.(yml|yaml)$/, '');
|
|
data[stack] = bulkInfo[name] ?? { status: 'unknown' };
|
|
}
|
|
const enrichmentStartedAt = Date.now();
|
|
let gitStackNames = new Set<string>();
|
|
let gitSourcesDegraded = false;
|
|
try {
|
|
gitStackNames = new Set(GitSourceService.getInstance().list().map((s) => s.stack_name));
|
|
} catch (sourceError) {
|
|
console.error(`Failed to load git sources for status labels on node ${req.nodeId}; defaulting to local:`, sourceError);
|
|
gitSourcesDegraded = true;
|
|
}
|
|
// Self-stack identity is resolved once per request instead of once per
|
|
// stack, so cache misses pay a single container-list call, not N.
|
|
const selfIdentity = stackNames.length > 0
|
|
? await resolveSelfStackIdentity()
|
|
: UNRESOLVED_SELF_STACK_IDENTITY;
|
|
const withSource: Record<string, BulkStackInfo & { source: 'local' | 'git'; isSelf: boolean }> = {};
|
|
const composeDir = FileSystemService.getInstance(req.nodeId).getBaseDir();
|
|
for (const [stack, info] of Object.entries(data)) {
|
|
const name = stack.replace(/\.(yml|yaml)$/, '');
|
|
withSource[stack] = {
|
|
...info,
|
|
source: gitStackNames.has(name) ? 'git' : 'local',
|
|
isSelf: isSelfStackByIdentity(selfIdentity, name, composeDir),
|
|
};
|
|
}
|
|
enrichmentMs = Date.now() - enrichmentStartedAt;
|
|
// The payload is flagged degraded when any enrichment source failed
|
|
// (Docker socket unreachable, git-source scan failure) so the route
|
|
// can refuse to let a mislabeled payload persist.
|
|
return { data: withSource, degraded: selfIdentity.degraded || gitSourcesDegraded };
|
|
},
|
|
);
|
|
cacheOutcome = fetchOutcome;
|
|
const { data, degraded } = result;
|
|
count = Object.keys(data).length;
|
|
// A degraded identity resolution (Docker socket unreachable) cannot be
|
|
// trusted to classify every stack, which un-gates destructive UI
|
|
// affordances on the Sencho stack itself, and a failed git-source scan
|
|
// mislabels every source badge as 'local'. Never let either mislabel
|
|
// persist for a full TTL: serve the live result, drop the cache entry,
|
|
// and let the next request re-resolve. Everything between the fetch and
|
|
// this invalidate is synchronous, so no concurrent reader can observe
|
|
// the degraded entry. Running outside Docker is not degraded (both
|
|
// identity sources legitimately resolve to null there), and an empty
|
|
// fleet skips resolution entirely; both are cached as-is.
|
|
if (fetchOutcome === 'computed' && count > 0 && degraded) {
|
|
CacheService.getInstance().invalidate(`stack-statuses:${req.nodeId}`);
|
|
}
|
|
res.json(data);
|
|
} catch (error) {
|
|
outcome = 'error';
|
|
console.error('Failed to fetch stack statuses:', error);
|
|
res.status(500).json({ error: 'Failed to fetch stack statuses' });
|
|
} finally {
|
|
logDebugTiming('[Stacks:debug]', {
|
|
route: 'GET /statuses',
|
|
nodeId: req.nodeId,
|
|
cacheOutcome,
|
|
count,
|
|
dockerMs,
|
|
enrichmentMs,
|
|
elapsedMs: Date.now() - startedAt,
|
|
outcome,
|
|
});
|
|
}
|
|
});
|
|
|
|
// Read-only scan of the compose directory for the guided first-import flow.
|
|
// Surfaces compose files that are not yet stacks (loose at the root, or one
|
|
// directory too deep) with a dry preview so a new user can land their first
|
|
// stack without reading the docs first. Performs no writes.
|
|
stacksRouter.get('/import/scan', async (req: Request, res: Response) => {
|
|
if (!requirePermission(req, res, 'stack:read')) return;
|
|
try {
|
|
const fsSvc = FileSystemService.getInstance(req.nodeId);
|
|
const raw = await fsSvc.findImportCandidates();
|
|
const candidates = raw.map((c) => {
|
|
let preview: ComposePreview;
|
|
if (c.oversized) {
|
|
preview = { services: [], warnings: [], parseError: 'Compose file is too large to preview.' };
|
|
} else if (c.content !== null) {
|
|
preview = parseComposePreview(c.content);
|
|
} else {
|
|
preview = { services: [], warnings: [], parseError: 'Could not read compose file.' };
|
|
}
|
|
return {
|
|
name: c.name,
|
|
composeFile: c.composeFile,
|
|
location: c.location,
|
|
status: c.status,
|
|
services: preview.services,
|
|
warnings: preview.warnings,
|
|
parseError: preview.parseError,
|
|
};
|
|
});
|
|
res.json({ composeDir: fsSvc.getBaseDir(), candidates });
|
|
} catch (error) {
|
|
console.error('Failed to scan compose directory:', error);
|
|
res.status(500).json({ error: 'Failed to scan compose directory' });
|
|
}
|
|
});
|
|
|
|
// Move a discovered import candidate into its own stack directory so Sencho
|
|
// picks it up. The single write path of the guided import flow: it relocates a
|
|
// loose or nested compose file on disk and never captures it into a store. The
|
|
// source is re-derived from a fresh scan and matched by location, so the client
|
|
// cannot point the move at an arbitrary path.
|
|
stacksRouter.post('/import/move', async (req: Request, res: Response) => {
|
|
if (!requirePermission(req, res, 'stack:create')) return;
|
|
const { location, name } = req.body as { location?: unknown; name?: unknown };
|
|
if (typeof location !== 'string' || !location) {
|
|
return res.status(400).json({ error: 'A candidate location is required' });
|
|
}
|
|
if (typeof name !== 'string' || !isValidStackName(name.trim())) {
|
|
return res.status(400).json({ error: 'Name must be alphanumeric, hyphens, or underscores only' });
|
|
}
|
|
const destName = name.trim();
|
|
try {
|
|
const fsSvc = FileSystemService.getInstance(req.nodeId);
|
|
const match = (await fsSvc.findImportCandidates()).find((c) => c.location === location);
|
|
if (!match) {
|
|
return res.status(404).json({ error: 'That compose file was not found. Rescan and try again.' });
|
|
}
|
|
await fsSvc.importCandidateIntoStack(match, destName);
|
|
invalidateNodeCaches(req.nodeId);
|
|
dlog(`[Stacks] Imported compose file into stack: ${sanitizeForLog(destName)}`);
|
|
res.json({ name: destName });
|
|
} catch (error) {
|
|
const code = (error as { code?: string })?.code;
|
|
// A destination that already exists (our own DEST_EXISTS) or that appeared
|
|
// between the existence check and the rename (EEXIST/ENOTEMPTY) is a clean
|
|
// conflict, not a server error.
|
|
if (code === 'DEST_EXISTS' || code === 'EEXIST' || code === 'ENOTEMPTY') {
|
|
return res.status(409).json({ error: `A stack named "${destName}" already exists` });
|
|
}
|
|
if (code === 'INVALID_PATH' || code === 'INVALID_STACK_NAME') {
|
|
return res.status(400).json({ error: 'Invalid path' });
|
|
}
|
|
// The candidate vanished between the scan above and the move (e.g. deleted
|
|
// on disk): treat it like a stale candidate rather than a server fault.
|
|
if (code === 'ENOENT') {
|
|
return res.status(404).json({ error: 'That compose file was not found. Rescan and try again.' });
|
|
}
|
|
console.error('Failed to import compose file into stack:', sanitizeForLog((error as Error)?.message ?? String(error)));
|
|
res.status(500).json({ error: 'Failed to move the compose file into place' });
|
|
}
|
|
});
|
|
|
|
type BulkLifecycleAction = 'start' | 'stop' | 'restart' | 'update';
|
|
const VALID_BULK_ACTIONS: ReadonlySet<BulkLifecycleAction> = new Set(['start', 'stop', 'restart', 'update']);
|
|
const BULK_PARALLELISM = 4;
|
|
const BULK_MAX_STACKS = 100;
|
|
|
|
interface BulkResultItem {
|
|
stackName: string;
|
|
ok: boolean;
|
|
error?: string;
|
|
code?: string;
|
|
/** Health gate run started for a successful update, when gating is enabled. */
|
|
healthGateId?: string | null;
|
|
}
|
|
|
|
async function runStackBulkOp(
|
|
req: Request,
|
|
stackName: string,
|
|
action: BulkLifecycleAction,
|
|
): Promise<BulkResultItem> {
|
|
if (!isValidStackName(stackName)) {
|
|
return { stackName, ok: false, error: 'Invalid stack name', code: 'invalid_name' };
|
|
}
|
|
if (!checkPermission(req, 'stack:deploy', 'stack', stackName)) {
|
|
return { stackName, ok: false, error: 'Permission denied', code: 'PERMISSION_DENIED' };
|
|
}
|
|
|
|
const fsSvc = FileSystemService.getInstance(req.nodeId);
|
|
const stackDir = path.join(fsSvc.getBaseDir(), stackName);
|
|
try {
|
|
if (!(await fsSvc.hasComposeFile(stackDir))) {
|
|
return { stackName, ok: false, error: 'Stack not found', code: 'not_found' };
|
|
}
|
|
} catch {
|
|
return { stackName, ok: false, error: 'Stack not found', code: 'not_found' };
|
|
}
|
|
|
|
if (action === 'update' || action === 'stop') {
|
|
if (await isSelfStack(stackName, fsSvc.getBaseDir())) {
|
|
return selfStackProtectedBulkResult(stackName);
|
|
}
|
|
}
|
|
|
|
const user = req.user?.username ?? 'system';
|
|
const lockAction: StackOpAction = action;
|
|
const lockResult = StackOpLockService.getInstance().tryAcquire(req.nodeId, stackName, lockAction, user);
|
|
if (!lockResult.acquired) {
|
|
return {
|
|
stackName,
|
|
ok: false,
|
|
error: `${stackName} is already ${STACK_OP_PRESENT_PARTICIPLE[lockResult.existing.action]}`,
|
|
code: 'stack_op_in_progress',
|
|
};
|
|
}
|
|
|
|
try {
|
|
if (action === 'update') {
|
|
const gate = await enforcePolicyPreDeploy(stackName, req.nodeId, buildPolicyGateOptions(req));
|
|
if (!gate.ok) {
|
|
return {
|
|
stackName,
|
|
ok: false,
|
|
error: describePolicyBlock(gate.policy, gate.violations, 'update'),
|
|
code: 'policy_blocked',
|
|
};
|
|
}
|
|
const atomic = true;
|
|
const orchResult = await StackUpdateOrchestrator.getInstance().execute(
|
|
{ nodeId: req.nodeId, stackName, target: { scope: 'stack' }, trigger: 'bulk', actor: user },
|
|
{ atomic, terminalWs: getTerminalWs(req.get(DEPLOY_SESSION_HEADER)) },
|
|
);
|
|
DatabaseService.getInstance().clearStackUpdateStatus(req.nodeId, stackName);
|
|
NotificationService.getInstance().broadcastEvent({
|
|
type: 'state-invalidate',
|
|
scope: 'image-updates',
|
|
nodeId: req.nodeId,
|
|
stackName,
|
|
action: 'stack-updated',
|
|
ts: Date.now(),
|
|
});
|
|
notifyActionSuccess('image_update_applied', `${stackName} updated`, stackName, user);
|
|
triggerPostDeployScan(stackName, req.nodeId).catch(err =>
|
|
console.error('[Security] Post-deploy scan failed for %s:', sanitizeForLog(stackName), err),
|
|
);
|
|
const healthGateId = HealthGateService.getInstance().beginStack(req.nodeId, stackName, 'update', req.user?.username ?? null, { deployedGenerationId: orchResult.kind === 'stack_compose_done' ? orchResult.deployedGenerationId : null });
|
|
const recoveryId = orchResult.kind === 'stack_compose_done' ? orchResult.recoveryId : null;
|
|
linkStackUpdateRecoveryGate(recoveryId, healthGateId);
|
|
return { stackName, ok: true, healthGateId };
|
|
} else {
|
|
const outcome = await containerActionForStack(req.nodeId, stackName, action);
|
|
if (outcome.kind === 'no-containers') {
|
|
return { stackName, ok: false, error: 'No containers found for this stack', code: 'no_containers' };
|
|
}
|
|
if (outcome.kind === 'error') {
|
|
if (action !== 'start') notifyActionFailure(action, stackName, new Error(outcome.message), user);
|
|
return { stackName, ok: false, error: outcome.message, code: 'op_failed' };
|
|
}
|
|
const meta = CONTAINER_ACTION_META[action];
|
|
notifyActionSuccess(meta.category, `${stackName} ${meta.pastTense}`, stackName, user);
|
|
}
|
|
return { stackName, ok: true };
|
|
} catch (err) {
|
|
if (action !== 'start') notifyActionFailure(action, stackName, err, user);
|
|
return { stackName, ok: false, error: getErrorMessage(err, `${action} failed`), code: 'op_failed' };
|
|
} finally {
|
|
StackOpLockService.getInstance().release(req.nodeId, stackName);
|
|
}
|
|
}
|
|
|
|
async function runWithBoundedParallelism<T, R>(
|
|
items: T[],
|
|
limit: number,
|
|
task: (item: T) => Promise<R>,
|
|
): Promise<R[]> {
|
|
const results: R[] = new Array(items.length);
|
|
let next = 0;
|
|
const worker = async (): Promise<void> => {
|
|
while (next < items.length) {
|
|
const idx = next;
|
|
next += 1;
|
|
results[idx] = await task(items[idx]);
|
|
}
|
|
};
|
|
const workerCount = Math.min(limit, items.length);
|
|
await Promise.all(Array.from({ length: workerCount }, () => worker()));
|
|
return results;
|
|
}
|
|
|
|
stacksRouter.post('/bulk', async (req: Request, res: Response) => {
|
|
const body = req.body as { action?: unknown; stackNames?: unknown } | undefined;
|
|
const action = body?.action;
|
|
const stackNames = body?.stackNames;
|
|
|
|
if (typeof action !== 'string' || !VALID_BULK_ACTIONS.has(action as BulkLifecycleAction)) {
|
|
return res.status(400).json({
|
|
error: `Invalid action. Must be one of: ${[...VALID_BULK_ACTIONS].join(', ')}`,
|
|
});
|
|
}
|
|
if (!Array.isArray(stackNames) || stackNames.length === 0) {
|
|
return res.status(400).json({ error: 'stackNames must be a non-empty array' });
|
|
}
|
|
if (stackNames.length > BULK_MAX_STACKS) {
|
|
return res.status(400).json({ error: `Bulk operations are limited to ${BULK_MAX_STACKS} stacks per request` });
|
|
}
|
|
if (!stackNames.every(s => typeof s === 'string')) {
|
|
return res.status(400).json({ error: 'stackNames must be an array of strings' });
|
|
}
|
|
|
|
const typedAction = action as BulkLifecycleAction;
|
|
const typedNames = Array.from(new Set(stackNames as string[]));
|
|
const denied = typedNames.some(name =>
|
|
isValidStackName(name) && !checkPermission(req, 'stack:deploy', 'stack', name));
|
|
if (denied) {
|
|
return res.status(403).json({ error: 'Permission denied.', code: 'PERMISSION_DENIED' });
|
|
}
|
|
|
|
const results = await runWithBoundedParallelism(
|
|
typedNames,
|
|
BULK_PARALLELISM,
|
|
name => runStackBulkOp(req, name, typedAction),
|
|
);
|
|
|
|
invalidateNodeCaches(req.nodeId);
|
|
const okCount = results.filter(r => r.ok).length;
|
|
console.log(
|
|
`[Stacks] Bulk ${sanitizeForLog(action)} completed: ${okCount}/${results.length} on node ${req.nodeId}`,
|
|
);
|
|
|
|
res.json({ action: typedAction, results });
|
|
});
|
|
|
|
// Read-only compose discovery for the sidebar empty state. stack:read only;
|
|
// never runs full environment diagnostics. Must register before /:stackName.
|
|
stacksRouter.get('/discovery', async (req: Request, res: Response) => {
|
|
if (!requirePermission(req, res, 'stack:read')) return;
|
|
try {
|
|
const probe = await probeComposeDiscovery(req.nodeId);
|
|
if (probe.readable) {
|
|
res.json({
|
|
composeDir: probe.composeDir,
|
|
readable: true,
|
|
discovery: probe.discovery,
|
|
});
|
|
} else {
|
|
res.json({
|
|
composeDir: probe.composeDir,
|
|
readable: false,
|
|
discovery: null,
|
|
error: probe.error,
|
|
});
|
|
}
|
|
} catch (error) {
|
|
console.error('Failed to probe compose discovery:', error);
|
|
res.status(500).json({ error: 'Failed to probe compose discovery' });
|
|
}
|
|
});
|
|
|
|
stacksRouter.get('/:stackName', async (req: Request, res: Response) => {
|
|
try {
|
|
const stackName = req.params.stackName as string;
|
|
const { content, mtimeMs } = await FileSystemService.getInstance(req.nodeId).getStackContentWithMtime(stackName);
|
|
res.setHeader('ETag', stackFileEtag(mtimeMs));
|
|
res.send(content);
|
|
} catch (error) {
|
|
res.status(500).json({ error: 'Failed to read stack' });
|
|
}
|
|
});
|
|
|
|
stacksRouter.put('/:stackName', async (req: Request, res: Response) => {
|
|
const stackName = req.params.stackName as string;
|
|
if (!requirePermission(req, res, 'stack:edit', 'stack', stackName)) return;
|
|
try {
|
|
const { content } = req.body;
|
|
if (typeof content !== 'string') {
|
|
console.error('Content is not a string, got:', typeof content);
|
|
return res.status(400).json({ error: 'Content must be a string' });
|
|
}
|
|
if (isDebugEnabled()) console.debug(`[Stacks:debug] Save starting`, { stackName: sanitizeForLog(stackName), bytes: content.length, nodeId: req.nodeId });
|
|
const expectedMtimeMs = parseIfMatchMtime(req.header('if-match'));
|
|
const result = await FileSystemService.getInstance(req.nodeId)
|
|
.saveStackContentIfUnchanged(stackName, content, expectedMtimeMs);
|
|
if (!result.ok) {
|
|
res.setHeader('ETag', stackFileEtag(result.currentMtimeMs));
|
|
return res.status(412).json({
|
|
error: `${stackName}'s compose file changed since you opened it.`,
|
|
code: 'stack_file_changed',
|
|
currentMtimeMs: result.currentMtimeMs,
|
|
currentContent: result.currentContent,
|
|
});
|
|
}
|
|
invalidateNodeCaches(req.nodeId);
|
|
StackFileRootsService.invalidate(req.nodeId, stackName);
|
|
dlog(`[Stacks] Compose file saved: ${sanitizeForLog(stackName)}`);
|
|
res.setHeader('ETag', stackFileEtag(result.mtimeMs));
|
|
res.json({ message: 'Stack saved successfully', mtimeMs: result.mtimeMs });
|
|
} catch (error) {
|
|
console.error('Failed to save stack:', sanitizeForLog(getErrorMessage(error, 'unknown')));
|
|
res.status(500).json({ error: 'Failed to save stack' });
|
|
}
|
|
});
|
|
|
|
stacksRouter.get('/:stackName/envs', async (req: Request, res: Response) => {
|
|
try {
|
|
const stackName = req.params.stackName as string;
|
|
const envPaths = await resolveAllEnvFilePaths(req.nodeId, stackName);
|
|
res.json({ envFiles: envPaths });
|
|
} catch (error) {
|
|
res.status(500).json({ error: 'Failed to resolve env files' });
|
|
}
|
|
});
|
|
|
|
stacksRouter.get('/:stackName/env', async (req: Request, res: Response) => {
|
|
try {
|
|
const stackName = req.params.stackName as string;
|
|
const requestedFile = req.query.file as string | undefined;
|
|
const envPaths = await resolveAllEnvFilePaths(req.nodeId, stackName);
|
|
|
|
let envPath: string | undefined = envPaths[0];
|
|
|
|
if (requestedFile) {
|
|
if (envPaths.includes(requestedFile)) {
|
|
envPath = requestedFile;
|
|
} else {
|
|
return res.status(400).json({ error: 'Requested env file not allowed' });
|
|
}
|
|
}
|
|
|
|
// Default path with no env files yet: reply 200 with an empty body and a
|
|
// header the frontend can read. This avoids surfacing a 404 for the
|
|
// legitimate "stack has no .env yet" case, which previous flows
|
|
// sometimes echoed back to the user as a confusing error string.
|
|
if (!envPath) {
|
|
res.setHeader('X-Env-Exists', 'false');
|
|
return res.send('');
|
|
}
|
|
|
|
const fsService = FileSystemService.getInstance(req.nodeId);
|
|
|
|
try {
|
|
await fsService.access(envPath);
|
|
} catch (e: unknown) {
|
|
const code = (e as NodeJS.ErrnoException)?.code;
|
|
if (code !== 'ENOENT') {
|
|
console.error('[Sencho] Unexpected error checking env file existence:', (e as Error).message);
|
|
}
|
|
// No env file at the resolved path. For an explicit ?file= query we
|
|
// surface a 404 (the caller asked for something specific). Otherwise
|
|
// treat it as the empty-stack case above.
|
|
if (requestedFile) {
|
|
return res.status(404).json({ error: 'Env file not found' });
|
|
}
|
|
res.setHeader('X-Env-Exists', 'false');
|
|
return res.send('');
|
|
}
|
|
|
|
try {
|
|
const content = await fsService.readFile(envPath, 'utf-8');
|
|
const mtimeMs = await fsService.statMtime(envPath);
|
|
if (mtimeMs !== null) res.setHeader('ETag', stackFileEtag(mtimeMs));
|
|
res.setHeader('X-Env-Exists', 'true');
|
|
return res.send(content);
|
|
} catch (e: unknown) {
|
|
// TOCTOU: the file existed at access() but vanished before readFile().
|
|
// Return the same friendly empty-body shape rather than a generic 500
|
|
// that the frontend would otherwise echo as an opaque error.
|
|
const code = (e as NodeJS.ErrnoException)?.code;
|
|
if (code === 'ENOENT' && !requestedFile) {
|
|
res.setHeader('X-Env-Exists', 'false');
|
|
return res.send('');
|
|
}
|
|
throw e;
|
|
}
|
|
} catch (error) {
|
|
console.error('Failed to read env file:', error);
|
|
res.status(500).json({ error: 'Failed to read env file' });
|
|
}
|
|
});
|
|
|
|
stacksRouter.put('/:stackName/env', async (req: Request, res: Response) => {
|
|
const stackName = req.params.stackName as string;
|
|
if (!requirePermission(req, res, 'stack:edit', 'stack', stackName)) return;
|
|
try {
|
|
const { content } = req.body;
|
|
if (typeof content !== 'string') {
|
|
return res.status(400).json({ error: 'Content must be a string' });
|
|
}
|
|
|
|
const requestedFile = req.query.file as string | undefined;
|
|
const envPaths = await resolveAllEnvFilePaths(req.nodeId, stackName);
|
|
|
|
let envPath = envPaths[0];
|
|
|
|
if (requestedFile) {
|
|
if (envPaths.includes(requestedFile)) {
|
|
envPath = requestedFile;
|
|
} else {
|
|
return res.status(400).json({ error: 'Requested env file not allowed' });
|
|
}
|
|
}
|
|
|
|
// No env file resolved: the stack has no .env yet and the editor only edits
|
|
// an existing env file. GET treats this same case as an empty 200; PUT cannot,
|
|
// since there is no resolved path to write. Reply with a clean, handled response
|
|
// instead of writing to an undefined path, which would otherwise surface as an opaque 500.
|
|
if (!envPath) {
|
|
return res.status(404).json({ error: 'No env file exists for this stack' });
|
|
}
|
|
|
|
const fsService = FileSystemService.getInstance(req.nodeId);
|
|
const expectedMtimeMs = parseIfMatchMtime(req.header('if-match'));
|
|
const result = await fsService.writeFileIfUnchanged(envPath, content, expectedMtimeMs);
|
|
if (!result.ok) {
|
|
res.setHeader('ETag', stackFileEtag(result.currentMtimeMs));
|
|
return res.status(412).json({
|
|
error: `${stackName}'s env file changed since you opened it.`,
|
|
code: 'stack_file_changed',
|
|
currentMtimeMs: result.currentMtimeMs,
|
|
currentContent: result.currentContent,
|
|
});
|
|
}
|
|
invalidateNodeCaches(req.nodeId);
|
|
StackFileRootsService.invalidate(req.nodeId, stackName);
|
|
const envFileName = path.basename(envPath);
|
|
dlog(`[Stacks] Env file saved: ${sanitizeForLog(stackName)}/${sanitizeForLog(envFileName)}`);
|
|
res.setHeader('ETag', stackFileEtag(result.mtimeMs));
|
|
res.json({ message: 'Env file saved successfully', mtimeMs: result.mtimeMs });
|
|
} catch (error) {
|
|
console.error('[Stacks] Failed to save env file:', error);
|
|
res.status(500).json({ error: 'Failed to save env file' });
|
|
}
|
|
});
|
|
|
|
// Project env files: per-stack ordered list of env files that serve as the Docker
|
|
// Compose project environment file(s) for ${VAR} interpolation. An empty array means
|
|
// "use the default .env behavior".
|
|
stacksRouter.get('/:stackName/project-env-files', async (req: Request, res: Response) => {
|
|
const stackName = req.params.stackName as string;
|
|
if (!requirePermission(req, res, 'stack:read', 'stack', stackName)) return;
|
|
try {
|
|
const files = DatabaseService.getInstance().getStackProjectEnvFiles(req.nodeId, stackName);
|
|
res.json({ envFiles: files });
|
|
} catch (error) {
|
|
console.error('[Stacks] Failed to read project env files:', error);
|
|
res.status(500).json({ error: 'Failed to read project env files' });
|
|
}
|
|
});
|
|
|
|
stacksRouter.put('/:stackName/project-env-files', async (req: Request, res: Response) => {
|
|
const stackName = req.params.stackName as string;
|
|
if (!requirePermission(req, res, 'stack:edit', 'stack', stackName)) return;
|
|
if (!(await requireStackExists(req.nodeId, stackName, res))) return;
|
|
try {
|
|
const { envFiles } = req.body as { envFiles?: unknown };
|
|
if (!Array.isArray(envFiles)) {
|
|
return res.status(400).json({ error: 'envFiles must be an array' });
|
|
}
|
|
const files = envFiles as string[];
|
|
const fsService = FileSystemService.getInstance(req.nodeId);
|
|
|
|
// Dedup while preserving order. Skip validation for duplicates: already
|
|
// confirmed on first occurrence and the file cannot change mid-handler.
|
|
const seen = new Set<string>();
|
|
const deduped: string[] = [];
|
|
for (const file of files) {
|
|
if (typeof file !== 'string' || file.length === 0) {
|
|
return res.status(400).json({ error: 'Each env file must be a non-empty string' });
|
|
}
|
|
if (seen.has(file)) continue;
|
|
if (path.isAbsolute(file) || file.includes('..') || file.startsWith('.git')) {
|
|
return res.status(400).json({ error: `Invalid env file path: "${file}"` });
|
|
}
|
|
if (file.includes('/') || file.includes('\\')) {
|
|
return res.status(400).json({ error: `Project env files must be at the stack root: "${file}"` });
|
|
}
|
|
if (!isValidRelativeStackPath(file)) {
|
|
return res.status(400).json({ error: `Invalid env file path: "${file}"` });
|
|
}
|
|
// Canonical js/path-injection barrier inline with the fs sink: resolve both
|
|
// the compose root and the stack directory from a single canonical root so
|
|
// containment is unambiguous even when the compose dir is a symlink. The
|
|
// pattern mirrors authoredComposeArgs.ts and every FileSystemService sink.
|
|
const baseResolved = path.resolve(fsService.getBaseDir());
|
|
const stackDir = path.resolve(baseResolved, stackName);
|
|
if (!stackDir.startsWith(baseResolved + path.sep)) {
|
|
return res.status(400).json({ error: `Stack directory escapes the compose directory: "${stackName}"` });
|
|
}
|
|
const safePath = path.resolve(stackDir, file);
|
|
if (!safePath.startsWith(baseResolved + path.sep)) {
|
|
return res.status(400).json({ error: `Env file path escapes the compose directory: "${file}"` });
|
|
}
|
|
try {
|
|
await fsService.access(safePath);
|
|
const stat = await fsp.stat(safePath);
|
|
if (!stat.isFile()) {
|
|
return res.status(400).json({ error: `Not a regular file: "${file}"` });
|
|
}
|
|
} catch (err) {
|
|
if ((err as NodeJS.ErrnoException).code === 'ENOENT') {
|
|
return res.status(400).json({ error: `Env file not found: "${file}"` });
|
|
}
|
|
return res.status(400).json({ error: `Cannot access env file "${file}"` });
|
|
}
|
|
seen.add(file);
|
|
deduped.push(file);
|
|
}
|
|
|
|
DatabaseService.getInstance().setStackProjectEnvFiles(req.nodeId, stackName, deduped);
|
|
res.json({ envFiles: deduped });
|
|
} catch (error) {
|
|
console.error('[Stacks] Failed to save project env files:', error);
|
|
res.status(500).json({ error: 'Failed to save project env files' });
|
|
}
|
|
});
|
|
|
|
// Candidates for project env file selection: stack-local env-like files.
|
|
stacksRouter.get('/:stackName/project-env-files/candidates', async (req: Request, res: Response) => {
|
|
const stackName = req.params.stackName as string;
|
|
if (!requirePermission(req, res, 'stack:read', 'stack', stackName)) return;
|
|
try {
|
|
const files = await discoverStackLocalEnvFiles(req.nodeId, stackName);
|
|
res.json({ envFiles: files });
|
|
} catch (error) {
|
|
console.error('[Stacks] Failed to discover project env file candidates:', error);
|
|
res.status(500).json({ error: 'Failed to discover project env file candidates' });
|
|
}
|
|
});
|
|
|
|
// Stack Dossier: operator-authored documentation persisted per (node, stack).
|
|
// All fields default to '' so a PUT is a full-document save (an omitted field
|
|
// clears it) and a GET for a stack with no dossier yet returns a clean blank.
|
|
const dossierField = (max: number) => z.string().max(max).default('');
|
|
const StackDossierUpdateSchema = z.object({
|
|
purpose: dossierField(1000),
|
|
owner: dossierField(1000),
|
|
access_urls: dossierField(2000),
|
|
static_ip: dossierField(255),
|
|
vlan: dossierField(255),
|
|
firewall_notes: dossierField(8000),
|
|
reverse_proxy_notes: dossierField(8000),
|
|
backup_notes: dossierField(8000),
|
|
upgrade_notes: dossierField(8000),
|
|
recovery_notes: dossierField(8000),
|
|
custom_notes: dossierField(8000),
|
|
});
|
|
const emptyDossierFields = (): StackDossierFields => StackDossierUpdateSchema.parse({});
|
|
|
|
stacksRouter.get('/:stackName/dossier', async (req: Request, res: Response) => {
|
|
const stackName = req.params.stackName as string;
|
|
if (!requirePermission(req, res, 'stack:read', 'stack', stackName)) return;
|
|
if (!(await requireStackExists(req.nodeId, stackName, res))) return;
|
|
try {
|
|
const row = DatabaseService.getInstance().getStackDossier(req.nodeId, stackName);
|
|
// No dossier yet: answer 200 with a blank document so the editor loads clean
|
|
// rather than forcing the client to special-case a 404.
|
|
res.json(row ?? { node_id: req.nodeId, stack_name: stackName, ...emptyDossierFields(), created_at: 0, updated_at: 0 });
|
|
} catch (error) {
|
|
console.error('[Stacks] Failed to read dossier:', sanitizeForLog(getErrorMessage(error, 'unknown')));
|
|
res.status(500).json({ error: 'Failed to read dossier' });
|
|
}
|
|
});
|
|
|
|
stacksRouter.put('/:stackName/dossier', async (req: Request, res: Response) => {
|
|
const stackName = req.params.stackName as string;
|
|
if (!requirePermission(req, res, 'stack:edit', 'stack', stackName)) return;
|
|
if (!(await requireStackExists(req.nodeId, stackName, res))) return;
|
|
const parsed = StackDossierUpdateSchema.safeParse(req.body);
|
|
if (!parsed.success) {
|
|
res.status(400).json({ error: parsed.error.issues[0]?.message ?? 'Invalid input' });
|
|
return;
|
|
}
|
|
try {
|
|
const row = DatabaseService.getInstance().upsertStackDossier(req.nodeId, stackName, parsed.data);
|
|
res.json(row);
|
|
} catch (error) {
|
|
console.error('[Stacks] Failed to save dossier:', sanitizeForLog(getErrorMessage(error, 'unknown')));
|
|
res.status(500).json({ error: 'Failed to save dossier' });
|
|
}
|
|
});
|
|
|
|
stacksRouter.post('/', async (req: Request, res: Response) => {
|
|
if (!requirePermission(req, res, 'stack:create')) return;
|
|
try {
|
|
const { stackName } = req.body;
|
|
if (!stackName || typeof stackName !== 'string') {
|
|
return res.status(400).json({ error: "Field 'stackName' is required and must be a string" });
|
|
}
|
|
if (!isValidStackName(stackName)) {
|
|
return res.status(400).json({ error: 'Stack name can only contain alphanumeric characters, hyphens, and underscores' });
|
|
}
|
|
try {
|
|
DeployedStackDeletionService.getInstance().assertNoBlockingDeletionIntent(req.nodeId, stackName);
|
|
} catch (guardError) {
|
|
res.status(409).json({ error: getErrorMessage(guardError, 'Stack deletion in progress'), code: 'stack_deletion_in_progress' });
|
|
return;
|
|
}
|
|
await FileSystemService.getInstance(req.nodeId).createStack(stackName);
|
|
invalidateNodeCaches(req.nodeId);
|
|
dlog(`[Stacks] Stack created: ${sanitizeForLog(stackName)}`);
|
|
res.json({ message: 'Stack created successfully', name: stackName });
|
|
} catch (error: unknown) {
|
|
const message = getErrorMessage(error, '');
|
|
if (message.includes('already exists')) {
|
|
return res.status(409).json({ error: 'Stack already exists' });
|
|
}
|
|
console.error('Failed to create stack:', error);
|
|
res.status(500).json({ error: 'Failed to create stack' });
|
|
}
|
|
});
|
|
|
|
stacksRouter.post('/from-git', async (req: Request, res: Response) => {
|
|
if (!requirePermission(req, res, 'stack:create')) return;
|
|
const fromGitStartedAt = Date.now();
|
|
const fromGitDiag = isDebugEnabled();
|
|
let fromGitStackName = '';
|
|
try {
|
|
const {
|
|
stack_name,
|
|
repo_url,
|
|
branch,
|
|
sync_env,
|
|
env_path,
|
|
auth_type,
|
|
token,
|
|
auto_apply_on_webhook,
|
|
auto_deploy_on_apply,
|
|
deploy_now,
|
|
skip_scan,
|
|
} = req.body ?? {};
|
|
fromGitStackName = typeof stack_name === 'string' ? stack_name : '';
|
|
|
|
if (typeof stack_name !== 'string' || !stack_name.trim()) {
|
|
return res.status(400).json({ error: 'stack_name is required' });
|
|
}
|
|
if (!isValidStackName(stack_name)) {
|
|
return res.status(400).json({ error: 'Stack name can only contain alphanumeric characters, hyphens, and underscores' });
|
|
}
|
|
if (typeof repo_url !== 'string' || !repo_url.trim()) {
|
|
return res.status(400).json({ error: 'repo_url is required' });
|
|
}
|
|
if (typeof branch !== 'string' || !branch.trim()) {
|
|
return res.status(400).json({ error: 'branch is required' });
|
|
}
|
|
const selection = parseComposeSelection(req.body);
|
|
if (!selection.ok) {
|
|
return res.status(400).json({ error: selection.error });
|
|
}
|
|
if (auto_apply_on_webhook !== undefined && typeof auto_apply_on_webhook !== 'boolean') {
|
|
return res.status(400).json({ error: 'auto_apply_on_webhook must be a boolean' });
|
|
}
|
|
if (auto_deploy_on_apply !== undefined && typeof auto_deploy_on_apply !== 'boolean') {
|
|
return res.status(400).json({ error: 'auto_deploy_on_apply must be a boolean' });
|
|
}
|
|
const resolvedAuthType = auth_type === 'token' ? 'token' : 'none';
|
|
const repoUrlError = repoUrlRejectionMessage(repo_url);
|
|
if (repoUrlError) {
|
|
return res.status(400).json({ error: repoUrlError });
|
|
}
|
|
if (branch.length > 256) {
|
|
return res.status(400).json({ error: 'branch is too long' });
|
|
}
|
|
if (typeof env_path === 'string' && env_path.length > 1024) {
|
|
return res.status(400).json({ error: 'env_path is too long' });
|
|
}
|
|
if (typeof token === 'string' && token.length > 8192) {
|
|
return res.status(400).json({ error: 'token is too long' });
|
|
}
|
|
if (typeof env_path === 'string' && env_path.trim() && !isValidGitSourcePath(env_path.trim())) {
|
|
return res.status(400).json({ error: 'env_path must be a relative repository file path' });
|
|
}
|
|
const autoApplyOnWebhook = auto_apply_on_webhook === true;
|
|
const autoDeployOnApply = auto_deploy_on_apply === true;
|
|
if (autoDeployOnApply && !requirePermission(req, res, 'stack:deploy', 'stack', stack_name)) return;
|
|
if (deploy_now === true && !requirePermission(req, res, 'stack:deploy', 'stack', stack_name)) return;
|
|
|
|
const stacks = await FileSystemService.getInstance(req.nodeId).getStacks();
|
|
if (stacks.includes(stack_name)) {
|
|
return res.status(409).json({ error: 'Stack already exists' });
|
|
}
|
|
|
|
const syncEnv = Boolean(sync_env);
|
|
const resolvedEnvPath = syncEnv
|
|
? defaultEnvPath(selection.value.composePaths[0], env_path)
|
|
: null;
|
|
|
|
if (fromGitDiag) {
|
|
dlog(
|
|
`[Stacks:diag] from-git start stack=${sanitizeForLog(stack_name)} nodeId=${req.nodeId ?? 'local'} host=${sanitizeForLog(gitRepoHost(repo_url))} branch=${sanitizeForLog(branch)} files=${selection.value.composePaths.length} envPath=${sanitizeForLog(resolvedEnvPath ?? 'none')} authType=${sanitizeForLog(resolvedAuthType)} autoApplyOnWebhook=${autoApplyOnWebhook} autoDeployOnApply=${autoDeployOnApply} deployNow=${deploy_now === true}`
|
|
);
|
|
}
|
|
|
|
const result = await GitSourceService.getInstance().createStackFromGit({
|
|
stackName: stack_name.trim(),
|
|
repoUrl: repo_url.trim(),
|
|
branch: branch.trim(),
|
|
composePaths: selection.value.composePaths,
|
|
contextDir: selection.value.contextDir,
|
|
syncEnv,
|
|
envPath: resolvedEnvPath,
|
|
authType: resolvedAuthType,
|
|
token: resolvedAuthType === 'token' && typeof token === 'string' && token !== '' ? token : null,
|
|
autoApplyOnWebhook,
|
|
autoDeployOnApply,
|
|
});
|
|
|
|
invalidateNodeCaches(req.nodeId);
|
|
|
|
let deployed = false;
|
|
let deployError: string | undefined;
|
|
if (deploy_now === true) {
|
|
const gate = await enforcePolicyPreDeploy(
|
|
stack_name,
|
|
req.nodeId,
|
|
buildPolicyGateOptions(req),
|
|
);
|
|
if (!gate.ok) {
|
|
deployError = describePolicyBlock(gate.policy, gate.violations);
|
|
} else {
|
|
try {
|
|
await ComposeService.getInstance(req.nodeId).deployStack(
|
|
stack_name,
|
|
undefined,
|
|
undefined,
|
|
req.deployContext ?? { source: 'from_git', actor: req.user?.username ?? null },
|
|
);
|
|
deployed = true;
|
|
invalidateNodeCaches(req.nodeId);
|
|
} catch (e) {
|
|
deployError = getErrorMessage(e, 'Deploy failed');
|
|
console.error(`[Stacks] Deploy after create-from-git failed for ${sanitizeForLog(stack_name)}:`, deployError);
|
|
}
|
|
}
|
|
}
|
|
|
|
dlog(`[Stacks] Stack created from Git: ${sanitizeForLog(stack_name)} at ${result.commitSha.slice(0, 7)}`);
|
|
if (fromGitDiag) {
|
|
dlog(
|
|
`[Stacks:diag] from-git ok stack=${sanitizeForLog(stack_name)} sha=${result.commitSha.slice(0, 7)} deployed=${deployed} envWritten=${result.envWritten} warnings=${result.warnings.length} elapsedMs=${Date.now() - fromGitStartedAt}`
|
|
);
|
|
}
|
|
res.json({
|
|
name: stack_name,
|
|
source: result.source,
|
|
commitSha: result.commitSha,
|
|
envWritten: result.envWritten,
|
|
warnings: result.warnings,
|
|
deployed,
|
|
deployError,
|
|
});
|
|
if (deployed && skip_scan !== true) {
|
|
triggerPostDeployScan(stack_name, req.nodeId).catch(err =>
|
|
console.error(`[Security] Post-deploy scan failed for ${sanitizeForLog(stack_name)}:`, err),
|
|
);
|
|
}
|
|
} catch (error) {
|
|
if (fromGitDiag) {
|
|
const code = error instanceof GitSourceError ? error.code : 'UNKNOWN';
|
|
dlog(
|
|
`[Stacks:diag] from-git fail stack=${sanitizeForLog(fromGitStackName)} code=${code} elapsedMs=${Date.now() - fromGitStartedAt}`
|
|
);
|
|
}
|
|
sendGitSourceError(res, error);
|
|
}
|
|
});
|
|
|
|
stacksRouter.delete('/:stackName', async (req: Request, res: Response) => {
|
|
const stackName = req.params.stackName as string;
|
|
if (!requirePermission(req, res, 'stack:delete', 'stack', stackName)) return;
|
|
if (await refuseIfSelfStack(req, res, stackName)) return;
|
|
const pruneVolumes = req.query.pruneVolumes === 'true';
|
|
const debug = isDebugEnabled();
|
|
const sanitizedName = sanitizeForLog(stackName);
|
|
if (debug) console.debug(`[Stacks:debug] Delete starting`, { stackName: sanitizedName, pruneVolumes, nodeId: req.nodeId });
|
|
|
|
const result = await DeployedStackDeletionService.getInstance().deleteDeployedStack({
|
|
nodeId: req.nodeId,
|
|
stackName,
|
|
pruneVolumes,
|
|
actor: req.user?.username ?? 'system',
|
|
});
|
|
|
|
if (!result.ok) {
|
|
if (result.code === 'lock_conflict') {
|
|
res.status(409).json({
|
|
error: result.error,
|
|
code: 'stack_op_in_progress',
|
|
action: result.existingAction ?? 'delete',
|
|
});
|
|
return;
|
|
}
|
|
if (result.code === 'fs_failed') {
|
|
res.status(500).json({
|
|
error: `${result.error}. Stack containers may have been stopped but on-disk files remain. Retry the delete or clean the files manually.`,
|
|
});
|
|
return;
|
|
}
|
|
res.status(500).json({ error: result.error });
|
|
return;
|
|
}
|
|
|
|
invalidateNodeCaches(req.nodeId);
|
|
dlog(`[Stacks] Stack deleted: ${sanitizeForLog(stackName)}`);
|
|
res.json({ success: true });
|
|
});
|
|
|
|
stacksRouter.get('/:stackName/containers', async (req: Request, res: Response) => {
|
|
const stackName = req.params.stackName as string;
|
|
if (!isValidStackName(stackName)) {
|
|
res.status(400).json({ error: 'Invalid stack name' });
|
|
return;
|
|
}
|
|
if (!requirePermission(req, res, 'stack:read', 'stack', stackName)) return;
|
|
const startedAt = Date.now();
|
|
let outcome: 'ok' | 'error' = 'ok';
|
|
let dockerMs: number | null = null;
|
|
let count = 0;
|
|
try {
|
|
const dockerController = DockerController.getInstance(req.nodeId);
|
|
const dockerStartedAt = Date.now();
|
|
const containers = await dockerController.getContainersByStack(stackName);
|
|
dockerMs = Date.now() - dockerStartedAt;
|
|
count = containers.length;
|
|
res.json(containers);
|
|
} catch (error) {
|
|
outcome = 'error';
|
|
console.error('[Stacks] Failed to fetch containers for %s:', sanitizeForLog(stackName), error);
|
|
res.status(500).json({ error: 'Failed to fetch containers' });
|
|
} finally {
|
|
logDebugTiming('[Stacks:debug]', {
|
|
route: 'GET /:stack/containers',
|
|
nodeId: req.nodeId,
|
|
count,
|
|
dockerMs,
|
|
elapsedMs: Date.now() - startedAt,
|
|
outcome,
|
|
});
|
|
}
|
|
});
|
|
|
|
stacksRouter.get('/:stackName/services', async (req: Request, res: Response) => {
|
|
const stackName = req.params.stackName as string;
|
|
if (!isValidStackName(stackName)) {
|
|
res.status(400).json({ error: 'Invalid stack name' });
|
|
return;
|
|
}
|
|
try {
|
|
const content = await FileSystemService.getInstance(req.nodeId).getStackContent(stackName);
|
|
if (content.length > MAX_COMPOSE_PARSE_BYTES) {
|
|
console.warn(`[Stacks] Compose for ${sanitizeForLog(stackName)} exceeds ${MAX_COMPOSE_PARSE_BYTES} bytes; refusing to parse services`);
|
|
res.status(413).json({ error: 'Compose file too large to parse' });
|
|
return;
|
|
}
|
|
const parsed = YAML.parse(content);
|
|
const services = parsed?.services ? Object.keys(parsed.services) : [];
|
|
res.json(services);
|
|
} catch (error) {
|
|
console.error('[Stacks] Failed to fetch services:', sanitizeForLog(getErrorMessage(error, 'unknown')));
|
|
res.status(500).json({ error: 'Failed to fetch services' });
|
|
}
|
|
});
|
|
|
|
/** A persisted finding as the Drift tab consumes it (no internal column names). */
|
|
interface DriftLedgerEntry {
|
|
service: string;
|
|
kind: DriftFindingKind;
|
|
message: string;
|
|
detectedAt: number;
|
|
resolvedAt: number | null;
|
|
}
|
|
|
|
/**
|
|
* Assemble the full Drift tab payload: the spatial report (compose vs runtime),
|
|
* the temporal overlay (source changed since last deploy), and the persisted
|
|
* ledger history. When `reconcile` is set, the current findings are persisted
|
|
* into the ledger (new ones recorded, cleared ones resolved) before the ledger
|
|
* is read back, so the returned history reflects the just-observed state.
|
|
*/
|
|
async function buildDriftPayload(
|
|
nodeId: number,
|
|
stackName: string,
|
|
reconcile: boolean,
|
|
): Promise<StackDriftReport & {
|
|
temporal: DriftTemporal;
|
|
ledger: DriftLedgerEntry[];
|
|
lastCheckedAt: number | null;
|
|
gitopsRevision: GitOpsRevisionProjection;
|
|
}> {
|
|
const report = await buildStackDriftReport(nodeId, stackName);
|
|
// Only the on-disk read is best-effort: an unreadable compose is already surfaced
|
|
// by the report as a parse error, so temporal degrades to neutral. computeTemporal
|
|
// runs outside the try so a real ledger fault (a DB or hashing error) surfaces as a
|
|
// 500 instead of being hidden behind a misleading "no baseline".
|
|
let content: string | null = null;
|
|
try {
|
|
content = await FileSystemService.getInstance(nodeId).getStackContent(stackName);
|
|
} catch {
|
|
// Unreadable compose: the report carries the parseError; temporal stays neutral.
|
|
}
|
|
const temporal: DriftTemporal = content !== null
|
|
? DriftLedgerService.getInstance().computeTemporal(nodeId, stackName, content)
|
|
: { hasBaseline: false, sourceChanged: false, renderedChanged: false };
|
|
if (reconcile) {
|
|
DriftLedgerService.getInstance().reconcile(nodeId, stackName, report);
|
|
}
|
|
// finding_type is a free-text column, but reconcile only ever writes a DriftFindingKind.
|
|
const ledger: DriftLedgerEntry[] = DatabaseService.getInstance()
|
|
.getRecentDriftFindings(nodeId, stackName, 20)
|
|
.map(r => ({
|
|
service: r.finding_type === 'managed-path-conflict' ? '' : r.service,
|
|
kind: r.finding_type as DriftFindingKind,
|
|
message: r.message,
|
|
detectedAt: r.detected_at,
|
|
resolvedAt: r.resolved_at,
|
|
}));
|
|
// The ledger reflects the last reconcile (re-check, deploy, or background scan),
|
|
// not this passive read, so surface when that was: the Drift tab labels the history
|
|
// "checked {time ago}" and a stale finding reads as history, not current truth.
|
|
const lastCheckedAt = DatabaseService.getInstance().getStackDossier(nodeId, stackName)?.last_drift_check_at ?? null;
|
|
// Additive and separate on purpose. The ledger above is the compose-versus-runtime
|
|
// record this tab has always shown; the revision carries the GitOps drift classes,
|
|
// which are derived state and are never written into stack_drift_findings.
|
|
//
|
|
// Deliberately not guarded, matching computeTemporal above: on a read, the
|
|
// revision is part of the answer rather than decoration on one, so a fault
|
|
// reading it surfaces as a 500 instead of a projection that quietly reports
|
|
// less state than exists. Mutation routes take the opposite side, because
|
|
// there the write has already committed and a decoration must not be able to
|
|
// report it as failed.
|
|
return { ...report, temporal, ledger, lastCheckedAt, gitopsRevision: projectManagedStackRevision(stackName, nodeId) };
|
|
}
|
|
|
|
stacksRouter.get('/:stackName/drift', async (req: Request, res: Response) => {
|
|
const stackName = req.params.stackName as string;
|
|
if (!(await requireStackExists(req.nodeId, stackName, res))) return;
|
|
try {
|
|
res.json(await buildDriftPayload(req.nodeId, stackName, false));
|
|
} catch (error) {
|
|
console.error('[Stacks] Failed to build drift report for %s:', sanitizeForLog(stackName),
|
|
sanitizeForLog(inspect(error, { depth: 4 })));
|
|
res.status(500).json({ error: 'Failed to build drift report' });
|
|
}
|
|
});
|
|
|
|
// Re-check is the one place a passive drift view becomes a ledger write: it
|
|
// reconciles the current findings into stack_drift_findings (recording newly
|
|
// detected and newly resolved ones) before returning the fresh payload, so the
|
|
// GET above can stay a side-effect-free read.
|
|
stacksRouter.post('/:stackName/drift/recheck', async (req: Request, res: Response) => {
|
|
const stackName = req.params.stackName as string;
|
|
if (!requirePermission(req, res, 'stack:read', 'stack', stackName)) return;
|
|
if (!(await requireStackExists(req.nodeId, stackName, res))) return;
|
|
try {
|
|
res.json(await buildDriftPayload(req.nodeId, stackName, true));
|
|
} catch (error) {
|
|
console.error('[Stacks] Failed to re-check drift for %s:', sanitizeForLog(stackName),
|
|
sanitizeForLog(inspect(error, { depth: 4 })));
|
|
res.status(500).json({ error: 'Failed to re-check drift' });
|
|
}
|
|
});
|
|
|
|
// Compose Doctor: GET returns the last stored preflight run (or a never-run
|
|
// sentinel); it is a side-effect-free read. Both routes auto-proxy to the active
|
|
// node, so a remote stack is preflighted on the node that actually owns it.
|
|
stacksRouter.get('/:stackName/preflight', async (req: Request, res: Response) => {
|
|
const stackName = req.params.stackName as string;
|
|
if (!requirePermission(req, res, 'stack:read', 'stack', stackName)) return;
|
|
if (!(await requireStackExists(req.nodeId, stackName, res))) return;
|
|
try {
|
|
res.json(ComposeDoctorService.getInstance().getLatest(req.nodeId, stackName));
|
|
} catch (error) {
|
|
console.error('[Stacks] Failed to load preflight for %s:', sanitizeForLog(stackName),
|
|
sanitizeForLog(inspect(error, { depth: 4 })));
|
|
res.status(500).json({ error: 'Failed to load preflight report' });
|
|
}
|
|
});
|
|
|
|
stacksRouter.get('/:stackName/missing-external-networks', async (req: Request, res: Response) => {
|
|
const stackName = req.params.stackName as string;
|
|
if (!requirePermission(req, res, 'stack:read', 'stack', stackName)) return;
|
|
if (!(await requireStackExists(req.nodeId, stackName, res))) return;
|
|
try {
|
|
const { resolveMissingExternalNetworks } = await import('../services/network/resolveMissingExternalNetworks');
|
|
res.json(await resolveMissingExternalNetworks(req.nodeId, stackName));
|
|
} catch (error) {
|
|
console.error('[Stacks] Failed to resolve missing external networks for %s:', sanitizeForLog(stackName),
|
|
sanitizeForLog(getErrorMessage(error, 'unknown')));
|
|
res.status(500).json({ error: 'Failed to check external networks' });
|
|
}
|
|
});
|
|
|
|
// Running preflight renders the effective model and stores the result, replacing
|
|
// any prior run. It is advisory and never blocks a deploy; stack:read is the
|
|
// correct gate since it mutates only the preflight tables, never the stack.
|
|
stacksRouter.post('/:stackName/preflight/run', async (req: Request, res: Response) => {
|
|
const stackName = req.params.stackName as string;
|
|
if (!requirePermission(req, res, 'stack:read', 'stack', stackName)) return;
|
|
if (!(await requireStackExists(req.nodeId, stackName, res))) return;
|
|
try {
|
|
const report = await ComposeDoctorService.getInstance().runPreflight(req.nodeId, stackName, req.user?.username ?? null);
|
|
res.json(report);
|
|
} catch (error) {
|
|
console.error('[Stacks] Failed to run preflight for %s:', sanitizeForLog(stackName),
|
|
sanitizeForLog(inspect(error, { depth: 4 })));
|
|
res.status(500).json({ error: 'Failed to run preflight' });
|
|
}
|
|
});
|
|
|
|
const PREFLIGHT_ACK_EXPIRY_MODES = new Set<PreflightAckExpiryMode>([
|
|
'forever', 'until_compose_change', 'days', 'until_image_change',
|
|
]);
|
|
const PREFLIGHT_RULE_ID_SET = new Set(RULE_IDS);
|
|
|
|
stacksRouter.get('/:stackName/preflight/acknowledgements', async (req: Request, res: Response) => {
|
|
const stackName = req.params.stackName as string;
|
|
if (!requirePermission(req, res, 'stack:read', 'stack', stackName)) return;
|
|
if (!(await requireStackExists(req.nodeId, stackName, res))) return;
|
|
try {
|
|
const run = DatabaseService.getInstance().getLatestPreflightRun(req.nodeId, stackName);
|
|
const ctx = {
|
|
renderedHash: run?.rendered_hash ?? null,
|
|
serviceImages: parseServiceImages(run?.service_images ?? null),
|
|
};
|
|
const now = Date.now();
|
|
const rows = DatabaseService.getInstance().getPreflightAcknowledgements(req.nodeId, stackName).map((ack) => ({
|
|
...ack,
|
|
active: isPreflightAckActive(ack, ctx, now),
|
|
}));
|
|
res.json(rows);
|
|
} catch (error) {
|
|
console.error('[Stacks] Failed to list preflight acknowledgements for %s:', sanitizeForLog(stackName),
|
|
sanitizeForLog(inspect(error, { depth: 4 })));
|
|
res.status(500).json({ error: 'Failed to load preflight acknowledgements' });
|
|
}
|
|
});
|
|
|
|
stacksRouter.post('/:stackName/preflight/acknowledgements', async (req: Request, res: Response) => {
|
|
const stackName = req.params.stackName as string;
|
|
if (!requirePermission(req, res, 'stack:edit', 'stack', stackName)) return;
|
|
if (!(await requireStackExists(req.nodeId, stackName, res))) return;
|
|
const body = req.body ?? {};
|
|
const ruleId = typeof body.ruleId === 'string' ? body.ruleId.trim() : '';
|
|
if (!PREFLIGHT_RULE_ID_SET.has(ruleId)) {
|
|
res.status(400).json({ error: 'ruleId must be a known Compose Doctor rule id' });
|
|
return;
|
|
}
|
|
if (isPreflightNoteFinding(ruleId)) {
|
|
res.status(400).json({ error: 'Informational notes cannot be acknowledged' });
|
|
return;
|
|
}
|
|
const serviceRaw = body.service == null || body.service === ''
|
|
? null
|
|
: String(body.service).trim();
|
|
if (serviceRaw !== null && !isValidServiceName(serviceRaw)) {
|
|
res.status(400).json({ error: 'service must be a valid service name' });
|
|
return;
|
|
}
|
|
const expiryMode = typeof body.expiryMode === 'string' ? body.expiryMode.trim() : 'forever';
|
|
if (!PREFLIGHT_ACK_EXPIRY_MODES.has(expiryMode as PreflightAckExpiryMode)) {
|
|
res.status(400).json({ error: 'expiryMode must be forever, until_compose_change, days, or until_image_change' });
|
|
return;
|
|
}
|
|
if (expiryMode === 'until_image_change' && !serviceRaw) {
|
|
res.status(400).json({ error: 'until_image_change requires a specific service' });
|
|
return;
|
|
}
|
|
const reason = typeof body.reason === 'string' ? body.reason.trim() : '';
|
|
if (reason.length > 2000) {
|
|
res.status(400).json({ error: 'reason is too long' });
|
|
return;
|
|
}
|
|
let expiresAt: number | null = null;
|
|
if (expiryMode === 'days') {
|
|
const days = Number(body.expiresInDays ?? 30);
|
|
if (!Number.isFinite(days) || days <= 0 || days > 3650) {
|
|
res.status(400).json({ error: 'expiresInDays must be between 1 and 3650' });
|
|
return;
|
|
}
|
|
expiresAt = Date.now() + Math.round(days * 86_400_000);
|
|
}
|
|
const run = DatabaseService.getInstance().getLatestPreflightRun(req.nodeId, stackName);
|
|
if (!run) {
|
|
res.status(400).json({ error: 'Run Compose Doctor before acknowledging a finding' });
|
|
return;
|
|
}
|
|
const serviceImages = parseServiceImages(run.service_images);
|
|
let anchorRenderedHash: string | null = null;
|
|
let anchorImageRef: string | null = null;
|
|
if (expiryMode === 'until_compose_change') {
|
|
if (!run.rendered_hash) {
|
|
res.status(400).json({ error: 'The latest preflight run has no compose fingerprint to anchor against' });
|
|
return;
|
|
}
|
|
anchorRenderedHash = run.rendered_hash;
|
|
}
|
|
if (expiryMode === 'until_image_change') {
|
|
const imageRef = serviceImages[serviceRaw!] ?? null;
|
|
if (!imageRef) {
|
|
res.status(400).json({ error: 'The latest preflight run has no image reference for that service' });
|
|
return;
|
|
}
|
|
anchorImageRef = imageRef;
|
|
}
|
|
try {
|
|
const ack = DatabaseService.getInstance().upsertPreflightAcknowledgement({
|
|
node_id: req.nodeId,
|
|
stack_name: stackName,
|
|
rule_id: ruleId,
|
|
service: serviceRaw,
|
|
reason,
|
|
expiry_mode: expiryMode as PreflightAckExpiryMode,
|
|
expires_at: expiresAt,
|
|
anchor_rendered_hash: anchorRenderedHash,
|
|
anchor_image_ref: anchorImageRef,
|
|
created_by: req.user?.username ?? 'unknown',
|
|
created_at: Date.now(),
|
|
});
|
|
res.status(201).json(ack);
|
|
} catch (error) {
|
|
console.error('[Stacks] Failed to create preflight acknowledgement for %s:', sanitizeForLog(stackName),
|
|
sanitizeForLog(inspect(error, { depth: 4 })));
|
|
res.status(500).json({ error: 'Failed to create preflight acknowledgement' });
|
|
}
|
|
});
|
|
|
|
stacksRouter.delete('/:stackName/preflight/acknowledgements/:id', async (req: Request, res: Response) => {
|
|
const stackName = req.params.stackName as string;
|
|
if (!requirePermission(req, res, 'stack:edit', 'stack', stackName)) return;
|
|
if (!(await requireStackExists(req.nodeId, stackName, res))) return;
|
|
const id = Number(req.params.id);
|
|
if (!Number.isFinite(id)) {
|
|
res.status(400).json({ error: 'Invalid acknowledgement id' });
|
|
return;
|
|
}
|
|
const existing = DatabaseService.getInstance().getPreflightAcknowledgement(id);
|
|
if (!existing || existing.node_id !== req.nodeId || existing.stack_name !== stackName) {
|
|
res.status(404).json({ error: 'Acknowledgement not found' });
|
|
return;
|
|
}
|
|
try {
|
|
DatabaseService.getInstance().deletePreflightAcknowledgement(id);
|
|
res.status(204).end();
|
|
} catch (error) {
|
|
console.error('[Stacks] Failed to delete preflight acknowledgement for %s:', sanitizeForLog(stackName),
|
|
sanitizeForLog(inspect(error, { depth: 4 })));
|
|
res.status(500).json({ error: 'Failed to delete preflight acknowledgement' });
|
|
}
|
|
});
|
|
|
|
// Compose Network Inspector: per-stack networking facts (network map, service
|
|
// membership, published ports/bindings, network_mode, extra_hosts, runtime
|
|
// drift) derived from the authored effective model + live snapshot. Read-only
|
|
// and advisory; auto-proxies to the active node. Never returns raw render
|
|
// stderr, env values, or label values.
|
|
stacksRouter.get('/:stackName/networking', async (req: Request, res: Response) => {
|
|
const stackName = req.params.stackName as string;
|
|
if (!requirePermission(req, res, 'stack:read', 'stack', stackName)) return;
|
|
if (!(await requireStackExists(req.nodeId, stackName, res))) return;
|
|
try {
|
|
res.json(await buildStackNetworkFacts(req.nodeId, stackName));
|
|
} catch (error) {
|
|
console.error('[Stacks] Failed to build networking facts for %s:', sanitizeForLog(stackName),
|
|
sanitizeForLog(inspect(error, { depth: 4 })));
|
|
res.status(500).json({ error: 'Failed to build networking facts' });
|
|
}
|
|
});
|
|
|
|
// Storage inventory: per-stack mount inventory (binds, named/anonymous volumes,
|
|
// tmpfs, docker socket; read-only vs read-write; host-path existence/type/owner)
|
|
// and a portability verdict derived from the effective model + within-stack
|
|
// host-path probes. Read-only and advisory; auto-proxies to the active node.
|
|
// Never returns raw render stderr or any environment value.
|
|
stacksRouter.get('/:stackName/storage', async (req: Request, res: Response) => {
|
|
const stackName = req.params.stackName as string;
|
|
if (!requirePermission(req, res, 'stack:read', 'stack', stackName)) return;
|
|
if (!(await requireStackExists(req.nodeId, stackName, res))) return;
|
|
try {
|
|
res.json(await buildStorageInventory(req.nodeId, stackName));
|
|
} catch (error) {
|
|
console.error('[Stacks] Failed to build storage inventory for %s:', sanitizeForLog(stackName),
|
|
sanitizeForLog(inspect(error, { depth: 4 })));
|
|
res.status(500).json({ error: 'Failed to build storage inventory' });
|
|
}
|
|
});
|
|
|
|
// Effective Stack Anatomy: structural facts (services, ports, volumes, networks,
|
|
// restart) from the fully-merged effective model, so a multi-file Git source's
|
|
// dossier and doc-drift reflect every override file, not just the root compose.
|
|
// Read-only and advisory; auto-proxies to the active node. The response carries
|
|
// only structural fields; resolved env, label, and command values are never
|
|
// extracted. A secret interpolated INTO a structural field still resolves into
|
|
// the payload, but is already readable at the same stack:read scope via the
|
|
// stack's files (see docs/features/environment-guardrails).
|
|
stacksRouter.get('/:stackName/effective-anatomy', async (req: Request, res: Response) => {
|
|
const stackName = req.params.stackName as string;
|
|
if (!requirePermission(req, res, 'stack:read', 'stack', stackName)) return;
|
|
if (!(await requireStackExists(req.nodeId, stackName, res))) return;
|
|
try {
|
|
res.json(await buildEffectiveAnatomy(req.nodeId, stackName));
|
|
} catch (error) {
|
|
console.error('[Stacks] Failed to build effective anatomy for %s:', sanitizeForLog(stackName),
|
|
sanitizeForLog(inspect(error, { depth: 4 })));
|
|
res.status(500).json({ error: 'Failed to build effective anatomy' });
|
|
}
|
|
});
|
|
|
|
// Effective Service Model: per-service facts (declared image, build presence,
|
|
// expected replica count, dependencies, healthcheck) that service-scoped
|
|
// update/restore key off of, from the fully-merged effective model. Read-only
|
|
// and advisory; auto-proxies to the active node. Never returns raw render
|
|
// stderr or any environment/label/command value.
|
|
stacksRouter.get('/:stackName/effective-services', async (req: Request, res: Response) => {
|
|
const stackName = req.params.stackName as string;
|
|
if (!requirePermission(req, res, 'stack:read', 'stack', stackName)) return;
|
|
if (!(await requireStackExists(req.nodeId, stackName, res))) return;
|
|
try {
|
|
res.json(await buildEffectiveServiceModel(req.nodeId, stackName));
|
|
} catch (error) {
|
|
console.error('[Stacks] Failed to build effective service model for %s:', sanitizeForLog(stackName),
|
|
sanitizeForLog(inspect(error, { depth: 4 })));
|
|
res.status(500).json({ error: 'Failed to build effective service model' });
|
|
}
|
|
});
|
|
|
|
// Environment inventory: per-stack env vars with their source, scope (Compose
|
|
// interpolation vs container injection), and status (present/missing/unused/
|
|
// duplicate/unpersisted), plus likely-secret classification. Read-only and
|
|
// advisory; auto-proxies to the active node. Names only: an env value is never
|
|
// read into the payload, so stack:read is the correct gate.
|
|
stacksRouter.get('/:stackName/env-inventory', async (req: Request, res: Response) => {
|
|
const stackName = req.params.stackName as string;
|
|
if (!requirePermission(req, res, 'stack:read', 'stack', stackName)) return;
|
|
if (!(await requireStackExists(req.nodeId, stackName, res))) return;
|
|
try {
|
|
res.json(await buildEnvInventory(req.nodeId, stackName));
|
|
} catch (error) {
|
|
console.error('[Stacks] Failed to build env inventory for %s:', sanitizeForLog(stackName),
|
|
sanitizeForLog(inspect(error, { depth: 4 })));
|
|
res.status(500).json({ error: 'Failed to build env inventory' });
|
|
}
|
|
});
|
|
|
|
// Docker/Compose label inventory: declared compose labels vs runtime container
|
|
// labels per service. Read-only; auto-proxies to the active node.
|
|
stacksRouter.get('/:stackName/label-inventory', async (req: Request, res: Response) => {
|
|
const stackName = req.params.stackName as string;
|
|
if (!requirePermission(req, res, 'stack:read', 'stack', stackName)) return;
|
|
if (!(await requireStackExists(req.nodeId, stackName, res))) return;
|
|
if (!requireRevealAdmin(req, res)) return;
|
|
try {
|
|
res.json(await buildStackLabelInventory(req.nodeId, stackName, labelInventoryOptionsFromRequest(req)));
|
|
} catch (error) {
|
|
console.error('[Stacks] Failed to build label inventory for %s:', sanitizeForLog(stackName),
|
|
sanitizeForLog(inspect(error, { depth: 4 })));
|
|
res.status(500).json({ error: 'Failed to build label inventory' });
|
|
}
|
|
});
|
|
|
|
// Exposure intent: the user's per-stack (service '') and per-service exposure
|
|
// classification, stored separately from generated facts so mismatches stay
|
|
// detectable. Rows are stored independently; precedence (a service row taking
|
|
// priority over the stack row, an absent service row inheriting the stack
|
|
// intent) is applied by the consumers that read these rows, not enforced here.
|
|
// Clearing a row (intent null) deletes it, returning that scope to unset.
|
|
const ExposurePutSchema = z.object({
|
|
service: z.string().max(255).optional().default(''),
|
|
intent: z.enum(EXPOSURE_INTENTS).nullable(),
|
|
});
|
|
|
|
function exposurePayload(nodeId: number, stackName: string): {
|
|
intents: { service: string; intent: ExposureIntent; updatedAt: number; updatedBy: string | null }[];
|
|
} {
|
|
return {
|
|
intents: DatabaseService.getInstance().getStackExposureIntents(nodeId, stackName).map(r => ({
|
|
service: r.service, intent: r.intent, updatedAt: r.updated_at, updatedBy: r.updated_by,
|
|
})),
|
|
};
|
|
}
|
|
|
|
stacksRouter.get('/:stackName/exposure', async (req: Request, res: Response) => {
|
|
const stackName = req.params.stackName as string;
|
|
if (!requirePermission(req, res, 'stack:read', 'stack', stackName)) return;
|
|
if (!(await requireStackExists(req.nodeId, stackName, res))) return;
|
|
try {
|
|
res.json(exposurePayload(req.nodeId, stackName));
|
|
} catch (error) {
|
|
console.error('[Stacks] Failed to read exposure intent for %s:', sanitizeForLog(stackName),
|
|
sanitizeForLog(getErrorMessage(error, 'unknown')));
|
|
res.status(500).json({ error: 'Failed to read exposure intent' });
|
|
}
|
|
});
|
|
|
|
stacksRouter.put('/:stackName/exposure', async (req: Request, res: Response) => {
|
|
const stackName = req.params.stackName as string;
|
|
if (!requirePermission(req, res, 'stack:edit', 'stack', stackName)) return;
|
|
if (!(await requireStackExists(req.nodeId, stackName, res))) return;
|
|
const parsed = ExposurePutSchema.safeParse(req.body);
|
|
if (!parsed.success) {
|
|
res.status(400).json({ error: 'Invalid exposure intent' });
|
|
return;
|
|
}
|
|
const { service, intent } = parsed.data;
|
|
try {
|
|
if (intent === null) {
|
|
DatabaseService.getInstance().deleteStackExposureIntent(req.nodeId, stackName, service);
|
|
} else {
|
|
DatabaseService.getInstance().setStackExposureIntent(req.nodeId, stackName, service, intent, req.user?.username ?? null);
|
|
}
|
|
res.json(exposurePayload(req.nodeId, stackName));
|
|
} catch (error) {
|
|
console.error('[Stacks] Failed to save exposure intent for %s:', sanitizeForLog(stackName),
|
|
sanitizeForLog(getErrorMessage(error, 'unknown')));
|
|
res.status(500).json({ error: 'Failed to save exposure intent' });
|
|
}
|
|
});
|
|
|
|
// Update guard: readiness reports computed on demand from existing stores
|
|
// (preflight runs, drift findings, backup slot, update preview, live Docker
|
|
// state). Node-scoped like preflight: a remote stack is evaluated on the node
|
|
// that owns it. Read-only, so stack:read is the correct gate.
|
|
stacksRouter.get('/:stackName/update-readiness', async (req: Request, res: Response) => {
|
|
const stackName = req.params.stackName as string;
|
|
const serviceName = typeof req.query.service === 'string' && req.query.service.length > 0 ? req.query.service : undefined;
|
|
if (!requirePermission(req, res, 'stack:read', 'stack', stackName)) return;
|
|
if (!(await requireStackExists(req.nodeId, stackName, res))) return;
|
|
try {
|
|
const report = await UpdateGuardService.getInstance().computeUpdateReadiness(req.nodeId, stackName, serviceName);
|
|
res.json(report);
|
|
} catch (error) {
|
|
if (error instanceof SingleServiceUpdateReadinessError) {
|
|
res.status(400).json({ error: error.message, code: 'service_update_single_service' });
|
|
return;
|
|
}
|
|
console.error('[Stacks] Failed to compute update readiness for %s:', sanitizeForLog(stackName),
|
|
sanitizeForLog(getErrorMessage(error, 'unknown')));
|
|
res.status(500).json({ error: 'Failed to compute update readiness' });
|
|
}
|
|
});
|
|
|
|
stacksRouter.get('/:stackName/rollback-readiness', async (req: Request, res: Response) => {
|
|
const stackName = req.params.stackName as string;
|
|
if (!requirePermission(req, res, 'stack:read', 'stack', stackName)) return;
|
|
if (!(await requireStackExists(req.nodeId, stackName, res))) return;
|
|
try {
|
|
const report = await UpdateGuardService.getInstance().computeRollbackReadiness(req.nodeId, stackName);
|
|
res.json(report);
|
|
} catch (error) {
|
|
console.error('[Stacks] Failed to compute rollback readiness for %s:', sanitizeForLog(stackName),
|
|
sanitizeForLog(getErrorMessage(error, 'unknown')));
|
|
res.status(500).json({ error: 'Failed to compute rollback readiness' });
|
|
}
|
|
});
|
|
|
|
// Post-update health gate result. `gateId` returns that specific run so a
|
|
// superseded gate still resolves to its terminal state; without it, the
|
|
// latest run (or a never-run sentinel).
|
|
stacksRouter.get('/:stackName/health-gate', async (req: Request, res: Response) => {
|
|
const stackName = req.params.stackName as string;
|
|
if (!requirePermission(req, res, 'stack:read', 'stack', stackName)) return;
|
|
if (!(await requireStackExists(req.nodeId, stackName, res))) return;
|
|
try {
|
|
const gateId = typeof req.query.gateId === 'string' && req.query.gateId.trim() ? req.query.gateId : undefined;
|
|
res.json(HealthGateService.getInstance().getReport(req.nodeId, stackName, gateId));
|
|
} catch (error) {
|
|
console.error('[Stacks] Failed to load health gate for %s:', sanitizeForLog(stackName),
|
|
sanitizeForLog(getErrorMessage(error, 'unknown')));
|
|
res.status(500).json({ error: 'Failed to load health gate' });
|
|
}
|
|
});
|
|
|
|
stacksRouter.post('/:stackName/deploy', async (req: Request, res: Response) => {
|
|
const stackName = req.params.stackName as string;
|
|
if (!requirePermission(req, res, 'stack:deploy', 'stack', stackName)) return;
|
|
if (!(await requireStackExists(req.nodeId, stackName, res))) return;
|
|
if (await refuseIfSelfStack(req, res, stackName)) return;
|
|
// Lock held below. All early-returns must stay inside the try so finally fires.
|
|
if (!tryAcquireStackOpLock(req, res, stackName, 'deploy')) return;
|
|
const t0 = Date.now();
|
|
let ok = false;
|
|
try {
|
|
if (!(await runPolicyGate(req, res, stackName, req.nodeId))) return;
|
|
const skipScan = req.body?.skip_scan === true;
|
|
const debug = isDebugEnabled();
|
|
const atomic = true;
|
|
if (debug) console.debug('[Stacks:debug] Deploy starting', { stackName, atomic, nodeId: req.nodeId });
|
|
const deployResult = await ComposeService.getInstance(req.nodeId).deployStack(
|
|
stackName,
|
|
getTerminalWs(req.get(DEPLOY_SESSION_HEADER)),
|
|
atomic,
|
|
req.deployContext ?? { source: 'manual', actor: req.user?.username ?? null },
|
|
);
|
|
invalidateNodeCaches(req.nodeId);
|
|
dlog(`[Stacks] Deploy completed: ${sanitizeForLog(stackName)}`);
|
|
if (debug) console.debug(`[Stacks:debug] Deploy finished in ${Date.now() - t0}ms`);
|
|
ok = true;
|
|
const healthGateId = HealthGateService.getInstance().beginStack(req.nodeId, stackName, 'deploy', req.user?.username ?? null, { deployedGenerationId: deployResult.deployedGenerationId });
|
|
linkStackUpdateRecoveryGate(deployResult.recoveryId, healthGateId);
|
|
res.json({ message: 'Deployed successfully', healthGateId });
|
|
notifyActionSuccess('deploy_success', `${stackName} deployed`, stackName, req.user?.username ?? 'system');
|
|
if (!skipScan) {
|
|
triggerPostDeployScan(stackName, req.nodeId).catch(err =>
|
|
console.error('[Security] Post-deploy scan failed for %s:', sanitizeForLog(stackName), err),
|
|
);
|
|
}
|
|
} catch (error: unknown) {
|
|
console.error('[Stacks] Deploy failed: %s', sanitizeForLog(stackName), error);
|
|
const { isMissingExternalNetworksError } = await import('../services/network/missingExternalNetworksError');
|
|
if (isMissingExternalNetworksError(error)) {
|
|
const status = error.kind === 'unavailable' ? 503 : error.kind === 'create_failed' ? 500 : 409;
|
|
if (!res.headersSent) {
|
|
res.status(status).json({
|
|
error: error.message,
|
|
code: error.code,
|
|
kind: error.kind,
|
|
networks: error.networks,
|
|
createdNames: error.createdNames,
|
|
remainingNames: error.remainingNames,
|
|
});
|
|
}
|
|
return;
|
|
}
|
|
const rollbackInfo = getComposeRollbackInfo(error);
|
|
const rolledBack = rollbackInfo?.rolledBack ?? false;
|
|
if (rolledBack) {
|
|
console.warn('[Stacks] Deploy failed, rolled back: %s', sanitizeForLog(stackName));
|
|
} else if (rollbackInfo?.attempted) {
|
|
console.warn('[Stacks] Deploy failed, rollback did not complete: %s', sanitizeForLog(stackName));
|
|
}
|
|
const message = getErrorMessage(error, 'Failed to deploy stack');
|
|
// ComposeRollbackError already carries the cause's message; see classifyFailure.
|
|
const failure = classifyFailure(message, { dockerUnavailable: isDockerUnavailableError(error) });
|
|
notifyActionFailure('deploy', stackName, error, req.user?.username ?? 'system');
|
|
if (!res.headersSent) {
|
|
if (isDockerUnavailableError(error)) {
|
|
res.status(503).json({ error: message, code: 'docker_unavailable', rolledBack, failure });
|
|
} else {
|
|
res.status(500).json({ error: message, rolledBack, failure });
|
|
}
|
|
}
|
|
} finally {
|
|
releaseStackOpLock(req, stackName);
|
|
StackOpMetricsService.getInstance().record(req.nodeId, 'deploy', Date.now() - t0, ok);
|
|
}
|
|
});
|
|
|
|
stacksRouter.post('/:stackName/down', async (req: Request, res: Response) => {
|
|
const stackName = req.params.stackName as string;
|
|
if (!requirePermission(req, res, 'stack:deploy', 'stack', stackName)) return;
|
|
if (!(await requireStackExists(req.nodeId, stackName, res))) return;
|
|
if (await refuseIfSelfStack(req, res, stackName)) return;
|
|
// Lock held below. All early-returns must stay inside the try so finally fires.
|
|
if (!tryAcquireStackOpLock(req, res, stackName, 'down')) return;
|
|
const t0 = Date.now();
|
|
let ok = false;
|
|
const removeVolumes = req.query.removeVolumes === 'true';
|
|
try {
|
|
if (removeVolumes && !getActiveCapabilities().includes(STACK_DOWN_REMOVE_VOLUMES_CAPABILITY)) {
|
|
res.status(400).json({ error: 'Volume removal is not supported on this node' });
|
|
return;
|
|
}
|
|
if (isDebugEnabled()) console.debug(`[Stacks:debug] Down starting`, { stackName: sanitizeForLog(stackName), nodeId: req.nodeId, removeVolumes });
|
|
await ComposeService.getInstance(req.nodeId).runDown(stackName, { removeVolumes }, getTerminalWs(req.get(DEPLOY_SESSION_HEADER)));
|
|
invalidateNodeCaches(req.nodeId);
|
|
dlog(`[Stacks] Down completed: ${sanitizeForLog(stackName)}`);
|
|
notifyActionSuccess(
|
|
'stack_taken_down',
|
|
`${stackName} taken down${removeVolumes ? ' (volumes removed)' : ''}`,
|
|
stackName,
|
|
req.user?.username ?? 'system',
|
|
);
|
|
ok = true;
|
|
res.json({ status: 'Command started' });
|
|
} catch (error: unknown) {
|
|
console.error('[Stacks] Down failed: %s', sanitizeForLog(stackName), error);
|
|
notifyActionFailure('down', stackName, error, req.user?.username ?? 'system');
|
|
if (!res.headersSent) {
|
|
if (isDockerUnavailableError(error)) {
|
|
res.status(503).json({ error: getErrorMessage(error, 'Docker daemon is unreachable'), code: 'docker_unavailable' });
|
|
} else {
|
|
res.status(500).json({ error: 'Failed to start command' });
|
|
}
|
|
}
|
|
} finally {
|
|
releaseStackOpLock(req, stackName);
|
|
StackOpMetricsService.getInstance().record(req.nodeId, 'down', Date.now() - t0, ok);
|
|
}
|
|
});
|
|
|
|
export type StackContainerAction = 'restart' | 'stop' | 'start';
|
|
|
|
const CONTAINER_ACTION_META: Record<StackContainerAction, { category: NotificationCategory; pastTense: string }> = {
|
|
restart: { category: 'stack_restarted', pastTense: 'restarted' },
|
|
stop: { category: 'stack_stopped', pastTense: 'stopped' },
|
|
start: { category: 'stack_started', pastTense: 'started' },
|
|
};
|
|
|
|
export type ContainerActionOutcome =
|
|
| { kind: 'ok'; count: number }
|
|
| { kind: 'no-containers' }
|
|
| { kind: 'docker-unavailable'; message: string }
|
|
| { kind: 'error'; message: string };
|
|
|
|
/**
|
|
* Returns true when the error looks like a Docker-engine reachability
|
|
* failure (socket gone, daemon down, connection refused). The string match
|
|
* is intentionally permissive: Dockerode wraps the underlying Node error
|
|
* differently depending on the transport (unix socket vs tcp vs ssh) and
|
|
* the OS, so a code-only check (`err.code === 'ECONNREFUSED'`) would miss
|
|
* some shapes.
|
|
*/
|
|
export function isDockerUnavailableError(error: unknown): boolean {
|
|
if (!error) return false;
|
|
const err = error as NodeJS.ErrnoException & { errno?: number };
|
|
if (err.code === 'ECONNREFUSED' || err.code === 'ENOENT' && /docker\.sock/i.test(err.message ?? '')) {
|
|
return true;
|
|
}
|
|
const message = String(err.message ?? '').toLowerCase();
|
|
return (
|
|
message.includes('econnrefused') ||
|
|
message.includes('cannot connect to the docker daemon') ||
|
|
message.includes('docker.sock') && (message.includes('connect') || message.includes('no such'))
|
|
);
|
|
}
|
|
|
|
export async function containerActionForStack(
|
|
nodeId: number,
|
|
stackName: string,
|
|
action: StackContainerAction,
|
|
): Promise<ContainerActionOutcome> {
|
|
try {
|
|
const dockerController = DockerController.getInstance(nodeId);
|
|
const containers = await dockerController.getContainersByStack(stackName);
|
|
if (!containers || containers.length === 0) return { kind: 'no-containers' };
|
|
const op =
|
|
action === 'restart' ? (id: string) => dockerController.restartContainer(id)
|
|
: action === 'stop' ? (id: string) => dockerController.stopContainer(id)
|
|
: (id: string) => dockerController.startContainer(id);
|
|
await Promise.all(containers.map(c => op(c.Id)));
|
|
return { kind: 'ok', count: containers.length };
|
|
} catch (error: unknown) {
|
|
if (isDockerUnavailableError(error)) {
|
|
return { kind: 'docker-unavailable', message: getErrorMessage(error, 'Docker daemon is unreachable') };
|
|
}
|
|
return { kind: 'error', message: getErrorMessage(error, `Failed to ${action} containers`) };
|
|
}
|
|
}
|
|
|
|
async function bulkContainerOp(
|
|
req: Request,
|
|
res: Response,
|
|
action: StackContainerAction,
|
|
): Promise<void> {
|
|
const stackName = req.params.stackName as string;
|
|
if (!requirePermission(req, res, 'stack:deploy', 'stack', stackName)) return;
|
|
if (!(await requireStackExists(req.nodeId, stackName, res))) return;
|
|
if (action === 'stop' && (await refuseIfSelfStack(req, res, stackName))) return;
|
|
// Lock held below. All early-returns must stay inside the try so finally fires.
|
|
if (!tryAcquireStackOpLock(req, res, stackName, action)) return;
|
|
const t0 = Date.now();
|
|
let ok = false;
|
|
try {
|
|
const titleCase = action.charAt(0).toUpperCase() + action.slice(1);
|
|
if (isDebugEnabled()) console.debug(`[Stacks:debug] ${titleCase} starting`, { stackName: sanitizeForLog(stackName), nodeId: req.nodeId });
|
|
const outcome = await containerActionForStack(req.nodeId, stackName, action);
|
|
|
|
if (outcome.kind === 'no-containers') {
|
|
res.status(404).json({ error: 'No containers found for this stack.' });
|
|
return;
|
|
}
|
|
if (outcome.kind === 'docker-unavailable') {
|
|
console.error('[Stacks] %s failed: docker unavailable for %s', sanitizeForLog(titleCase), sanitizeForLog(stackName));
|
|
if (action !== 'start') notifyActionFailure(action, stackName, new Error(outcome.message), req.user?.username ?? 'system');
|
|
res.status(503).json({ error: outcome.message, code: 'docker_unavailable' });
|
|
return;
|
|
}
|
|
if (outcome.kind === 'error') {
|
|
console.error('[Stacks] %s failed: %s %s', sanitizeForLog(titleCase), sanitizeForLog(stackName), sanitizeForLog(outcome.message));
|
|
if (action !== 'start') notifyActionFailure(action, stackName, new Error(outcome.message), req.user?.username ?? 'system');
|
|
res.status(500).json({ error: outcome.message });
|
|
return;
|
|
}
|
|
|
|
invalidateNodeCaches(req.nodeId);
|
|
dlog(`[Stacks] ${titleCase} completed: ${sanitizeForLog(stackName)} (${outcome.count} containers)`);
|
|
ok = true;
|
|
res.json({ success: true, message: `${titleCase} completed via Engine API.` });
|
|
const { category, pastTense } = CONTAINER_ACTION_META[action];
|
|
notifyActionSuccess(category, `${stackName} ${pastTense}`, stackName, req.user?.username ?? 'system');
|
|
} catch (error: unknown) {
|
|
console.error('[Stacks] %s threw unexpectedly: %s', sanitizeForLog(action), sanitizeForLog(stackName), error);
|
|
if (!res.headersSent) {
|
|
res.status(500).json({ error: getErrorMessage(error, `Failed to ${action} stack`) });
|
|
}
|
|
} finally {
|
|
releaseStackOpLock(req, stackName);
|
|
StackOpMetricsService.getInstance().record(req.nodeId, action as StackMetricAction, Date.now() - t0, ok);
|
|
}
|
|
}
|
|
|
|
stacksRouter.post('/:stackName/restart', (req, res) => bulkContainerOp(req, res, 'restart'));
|
|
stacksRouter.post('/:stackName/stop', (req, res) => bulkContainerOp(req, res, 'stop'));
|
|
stacksRouter.post('/:stackName/start', (req, res) => bulkContainerOp(req, res, 'start'));
|
|
|
|
type ServiceAction = 'start' | 'stop' | 'restart';
|
|
|
|
async function handleServiceAction(
|
|
req: Request,
|
|
res: Response,
|
|
action: ServiceAction,
|
|
): Promise<void> {
|
|
const stackName = req.params.stackName as string;
|
|
const serviceName = req.params.serviceName as string;
|
|
if (!requirePermission(req, res, 'stack:deploy', 'stack', stackName)) return;
|
|
if (action === 'stop' && await refuseIfSelfStack(req, res, stackName)) return;
|
|
if (!isValidServiceName(serviceName)) {
|
|
res.status(400).json({ error: 'Invalid service name' });
|
|
return;
|
|
}
|
|
try {
|
|
const dockerController = DockerController.getInstance(req.nodeId);
|
|
const all = await dockerController.getContainersByStack(stackName);
|
|
if (!all || all.length === 0) {
|
|
res.status(404).json({ error: 'No containers found for this stack.' });
|
|
return;
|
|
}
|
|
const matching = filterContainersByComposeService(all, serviceName);
|
|
if (matching.length === 0) {
|
|
res.status(404).json({ error: `Service '${serviceName}' not found in stack '${stackName}'.` });
|
|
return;
|
|
}
|
|
const op =
|
|
action === 'start'
|
|
? (id: string) => dockerController.startContainer(id)
|
|
: action === 'stop'
|
|
? (id: string) => dockerController.stopContainer(id)
|
|
: (id: string) => dockerController.restartContainer(id);
|
|
await Promise.all(matching.map(c => op(c.Id)));
|
|
invalidateNodeCaches(req.nodeId);
|
|
dlog(
|
|
`[Stacks] Service ${sanitizeForLog(action)} completed: ${sanitizeForLog(stackName)}/${sanitizeForLog(serviceName)} (${matching.length} containers)`,
|
|
);
|
|
res.json({
|
|
success: true,
|
|
message: `Service ${action} completed via Engine API.`,
|
|
count: matching.length,
|
|
});
|
|
} catch (error: unknown) {
|
|
console.error('[Stacks] Service %s failed: %s/%s', sanitizeForLog(action), sanitizeForLog(stackName), sanitizeForLog(serviceName), error);
|
|
res.status(500).json({ error: getErrorMessage(error, `Failed to ${action} service`) });
|
|
}
|
|
}
|
|
|
|
stacksRouter.post('/:stackName/services/:serviceName/restart', (req, res) =>
|
|
handleServiceAction(req, res, 'restart'));
|
|
stacksRouter.post('/:stackName/services/:serviceName/stop', (req, res) =>
|
|
handleServiceAction(req, res, 'stop'));
|
|
stacksRouter.post('/:stackName/services/:serviceName/start', (req, res) =>
|
|
handleServiceAction(req, res, 'start'));
|
|
|
|
/** Map an orchestrator `service_failed` code to an HTTP status. */
|
|
function serviceFailureStatus(code: string): number {
|
|
switch (code) {
|
|
case 'service_update_single_service':
|
|
case 'service_not_updatable':
|
|
case 'effective_model_render_failed':
|
|
case 'recovery_id_required':
|
|
return 400;
|
|
case 'service_not_found':
|
|
case 'recovery_not_found':
|
|
return 404;
|
|
case 'policy_blocked':
|
|
case 'recovery_not_restorable':
|
|
case 'recovery_claim_failed':
|
|
return 409;
|
|
default:
|
|
// Compose, retag, inspect, and replica-divergence failures are server-side.
|
|
return 500;
|
|
}
|
|
}
|
|
|
|
/** Send an orchestrator service result as an HTTP response. Returns success. */
|
|
function sendServiceResult(res: Response, result: OrchestratorResult, serviceName: string): boolean {
|
|
if (result.kind === 'service_done') {
|
|
res.json({
|
|
serviceName: result.serviceName,
|
|
healthGateId: result.healthGateId,
|
|
observing: result.observing,
|
|
recoveryId: result.recoveryId,
|
|
recoveryAvailable: result.recoveryAvailable,
|
|
...(result.previousImageId ? { previousImageId: result.previousImageId } : {}),
|
|
...(result.newImageId ? { newImageId: result.newImageId } : {}),
|
|
...(result.recheckWarning ? { recheckWarning: result.recheckWarning } : {}),
|
|
});
|
|
return true;
|
|
}
|
|
if (result.kind === 'service_failed') {
|
|
res.status(serviceFailureStatus(result.code)).json({
|
|
error: result.error,
|
|
code: result.code,
|
|
serviceName: result.serviceName ?? serviceName,
|
|
...(result.mutationStage ? { mutationStage: result.mutationStage } : {}),
|
|
...(result.recoveryId ? { recoveryId: result.recoveryId } : {}),
|
|
});
|
|
return false;
|
|
}
|
|
// Stack results never reach the service routes; treat defensively.
|
|
res.status(500).json({ error: 'Unexpected orchestrator result', code: 'unexpected_result' });
|
|
return false;
|
|
}
|
|
|
|
function requireServiceScopedUpdateCapability(res: Response): boolean {
|
|
if (getActiveCapabilities().includes(SERVICE_SCOPED_UPDATE_CAPABILITY)) return true;
|
|
res.status(400).json({ error: 'Service-scoped updates are not supported on this node', code: 'capability_unavailable' });
|
|
return false;
|
|
}
|
|
|
|
async function handleServiceScopedMutation(
|
|
req: Request,
|
|
res: Response,
|
|
options: {
|
|
lockAction: 'update' | 'rollback';
|
|
recoveryId?: string;
|
|
notifyFailureAction: 'update' | 'rollback';
|
|
failureCode: string;
|
|
failureMessage: string;
|
|
onSuccess: (
|
|
stackName: string,
|
|
serviceName: string,
|
|
actor: string,
|
|
meta: { previousImageId?: string | null; newImageId?: string | null },
|
|
) => void;
|
|
},
|
|
): Promise<void> {
|
|
const stackName = req.params.stackName as string;
|
|
const serviceName = req.params.serviceName as string;
|
|
if (!requirePermission(req, res, 'stack:deploy', 'stack', stackName)) return;
|
|
if (!(await requireStackExists(req.nodeId, stackName, res))) return;
|
|
if (await refuseIfSelfStack(req, res, stackName)) return;
|
|
if (!requireServiceScopedUpdateCapability(res)) return;
|
|
if (!tryAcquireStackOpLock(req, res, stackName, options.lockAction)) return;
|
|
|
|
const t0 = Date.now();
|
|
let ok = false;
|
|
try {
|
|
const result = await StackUpdateOrchestrator.getInstance().execute(
|
|
{
|
|
nodeId: req.nodeId,
|
|
stackName,
|
|
target: { scope: 'service', serviceName },
|
|
trigger: 'manual',
|
|
actor: req.user?.username ?? null,
|
|
},
|
|
{
|
|
policyOptions: buildPolicyGateOptions(req),
|
|
terminalWs: getTerminalWs(req.get(DEPLOY_SESSION_HEADER)),
|
|
...(options.recoveryId ? { recoveryId: options.recoveryId } : {}),
|
|
},
|
|
);
|
|
invalidateNodeCaches(req.nodeId);
|
|
ok = sendServiceResult(res, result, serviceName);
|
|
if (ok && result.kind === 'service_done') {
|
|
options.onSuccess(stackName, serviceName, req.user?.username ?? 'system', {
|
|
previousImageId: result.previousImageId,
|
|
newImageId: result.newImageId,
|
|
});
|
|
NotificationService.getInstance().broadcastEvent({
|
|
type: 'state-invalidate',
|
|
scope: 'image-updates',
|
|
nodeId: req.nodeId,
|
|
stackName,
|
|
action: 'stack-updated',
|
|
ts: Date.now(),
|
|
});
|
|
} else if (result.kind === 'service_failed') {
|
|
notifyActionFailure(
|
|
options.notifyFailureAction,
|
|
stackName,
|
|
new Error(result.error),
|
|
req.user?.username ?? 'system',
|
|
);
|
|
}
|
|
} catch (error: unknown) {
|
|
console.error(
|
|
'[Stacks] Service %s failed: %s/%s: %s',
|
|
sanitizeForLog(options.notifyFailureAction),
|
|
sanitizeForLog(stackName),
|
|
sanitizeForLog(serviceName),
|
|
sanitizeForLog(getErrorMessage(error, 'unknown')),
|
|
);
|
|
notifyActionFailure(options.notifyFailureAction, stackName, error, req.user?.username ?? 'system');
|
|
if (!res.headersSent) {
|
|
res.status(500).json({ error: getErrorMessage(error, options.failureMessage), code: options.failureCode });
|
|
}
|
|
} finally {
|
|
StackOpMetricsService.getInstance().record(req.nodeId, 'update', Date.now() - t0, ok, {
|
|
targetScope: 'service',
|
|
serviceName,
|
|
});
|
|
releaseStackOpLock(req, stackName);
|
|
}
|
|
}
|
|
|
|
stacksRouter.post('/:stackName/services/:serviceName/update', async (req: Request, res: Response) => {
|
|
await handleServiceScopedMutation(req, res, {
|
|
lockAction: 'update',
|
|
notifyFailureAction: 'update',
|
|
failureCode: 'service_update_failed',
|
|
failureMessage: 'Failed to update service',
|
|
onSuccess: (stackName, serviceName, actor, meta) => {
|
|
const from = shortImageId(meta.previousImageId);
|
|
const to = shortImageId(meta.newImageId);
|
|
const transition = from && to && from !== to ? ` (${from} -> ${to})` : '';
|
|
notifyActionSuccess(
|
|
'image_update_applied',
|
|
`${stackName}/${serviceName} updated${transition}`,
|
|
stackName,
|
|
actor,
|
|
);
|
|
},
|
|
});
|
|
});
|
|
|
|
/** Latest restorable service recovery snapshot (active, unexpired), for UI discovery outside Deploy Progress. */
|
|
stacksRouter.get('/:stackName/services/:serviceName/recovery', async (req: Request, res: Response) => {
|
|
const stackName = req.params.stackName as string;
|
|
const serviceName = req.params.serviceName as string;
|
|
if (!requirePermission(req, res, 'stack:deploy', 'stack', stackName)) return;
|
|
if (!(await requireStackExists(req.nodeId, stackName, res))) return;
|
|
if (!requireServiceScopedUpdateCapability(res)) return;
|
|
try {
|
|
const row = ServiceUpdateRecoveryService.getInstance().listActive(req.nodeId, stackName, serviceName)[0];
|
|
if (!row) {
|
|
res.json({ recovery: null });
|
|
return;
|
|
}
|
|
res.json({
|
|
recovery: {
|
|
id: row.id,
|
|
status: row.status,
|
|
healthGateId: row.health_gate_id,
|
|
expiresAt: row.expires_at,
|
|
createdAt: row.created_at,
|
|
majorityImageId: row.majority_image_id,
|
|
declaredImageRef: row.declared_image_ref,
|
|
},
|
|
});
|
|
} catch (error: unknown) {
|
|
console.error(
|
|
'[Stacks] Failed to list service recovery for %s/%s: %s',
|
|
sanitizeForLog(stackName),
|
|
sanitizeForLog(serviceName),
|
|
sanitizeForLog(getErrorMessage(error, 'unknown')),
|
|
);
|
|
res.status(500).json({ error: 'Failed to load service recovery', code: 'service_recovery_lookup_failed' });
|
|
}
|
|
});
|
|
|
|
stacksRouter.post('/:stackName/services/:serviceName/restore', async (req: Request, res: Response) => {
|
|
const recoveryId = typeof req.body?.recoveryId === 'string' ? req.body.recoveryId : '';
|
|
if (!recoveryId) {
|
|
res.status(400).json({ error: 'A recoveryId is required to restore a service.', code: 'recovery_id_required' });
|
|
return;
|
|
}
|
|
await handleServiceScopedMutation(req, res, {
|
|
lockAction: 'rollback',
|
|
recoveryId,
|
|
notifyFailureAction: 'rollback',
|
|
failureCode: 'service_restore_failed',
|
|
failureMessage: 'Failed to restore service',
|
|
onSuccess: (stackName, serviceName, actor, meta) => {
|
|
const from = shortImageId(meta.previousImageId);
|
|
const to = shortImageId(meta.newImageId);
|
|
const transition = from && to && from !== to ? ` (${from} -> ${to})` : '';
|
|
notifyActionSuccess(
|
|
'deploy_success',
|
|
`${stackName}/${serviceName} restored${transition}`,
|
|
stackName,
|
|
actor,
|
|
);
|
|
},
|
|
});
|
|
});
|
|
|
|
stacksRouter.get('/:stackName/update-preview', async (req: Request, res: Response) => {
|
|
const stackName = req.params.stackName as string;
|
|
try {
|
|
// Anatomy and other GET consumers must not contact registries while
|
|
// node-scoped detection is off.
|
|
if (!ImageUpdateService.isChecksEnabled()) {
|
|
res.json(buildDetectionDisabledPreview(stackName));
|
|
return;
|
|
}
|
|
// Read-only: sticky reconciliation lives on POST so UpdateGuard and other
|
|
// GET consumers never mutate persisted scanner state.
|
|
const preview = await UpdatePreviewService.getInstance().getPreview(req.nodeId, stackName);
|
|
res.json(preview);
|
|
} catch (error) {
|
|
console.error('[Stacks] Update preview failed: %s', sanitizeForLog(stackName), sanitizeForLog(getErrorMessage(error, 'unknown')));
|
|
res.status(500).json({ error: 'Failed to compute update preview' });
|
|
}
|
|
});
|
|
|
|
stacksRouter.post('/:stackName/update-preview', async (req: Request, res: Response) => {
|
|
const stackName = req.params.stackName as string;
|
|
try {
|
|
if (!ImageUpdateService.isChecksEnabled()) {
|
|
// No registry I/O and no sticky reconcile on a synthetic disabled preview.
|
|
res.json({ ...buildDetectionDisabledPreview(stackName), reconciled: false });
|
|
return;
|
|
}
|
|
// Snapshot write-generation watermarks before the read-only preview so a
|
|
// later clear can erase older confirmed/sticky rows without racing a
|
|
// scanner that reserved or rewrote the row after this observation.
|
|
const imageUpdates = ImageUpdateService.getInstance();
|
|
const db = DatabaseService.getInstance();
|
|
const observedMemoryGeneration = imageUpdates.peekStackWriteGeneration(req.nodeId, stackName);
|
|
const observedRowGeneration = db.getStackUpdateWriteGeneration(req.nodeId, stackName);
|
|
const preview = await UpdatePreviewService.getInstance().getPreview(req.nodeId, stackName);
|
|
let reconciled = false;
|
|
if (isAuthoritativeNegativePreview(preview)) {
|
|
const clearResult = await imageUpdates.commitPreviewClear(
|
|
req.nodeId,
|
|
stackName,
|
|
observedMemoryGeneration,
|
|
observedRowGeneration,
|
|
);
|
|
if (clearResult === 'cleared') {
|
|
reconciled = true;
|
|
invalidateFleetUpdateCache();
|
|
NotificationService.getInstance().broadcastEvent({
|
|
type: 'state-invalidate',
|
|
scope: 'image-updates',
|
|
nodeId: req.nodeId,
|
|
stackName,
|
|
action: 'update-status-reconciled',
|
|
ts: Date.now(),
|
|
});
|
|
}
|
|
}
|
|
res.json({ ...preview, reconciled });
|
|
} catch (error) {
|
|
console.error('[Stacks] Update preview reconcile failed: %s', sanitizeForLog(stackName), sanitizeForLog(getErrorMessage(error, 'unknown')));
|
|
res.status(500).json({ error: 'Failed to compute update preview' });
|
|
}
|
|
});
|
|
|
|
stacksRouter.post('/:stackName/update', async (req: Request, res: Response) => {
|
|
const stackName = req.params.stackName as string;
|
|
if (!requirePermission(req, res, 'stack:deploy', 'stack', stackName)) return;
|
|
if (!(await requireStackExists(req.nodeId, stackName, res))) return;
|
|
if (await refuseIfSelfStack(req, res, stackName)) return;
|
|
// Lock held below. All early-returns must stay inside the try so finally fires.
|
|
if (!tryAcquireStackOpLock(req, res, stackName, 'update')) return;
|
|
const t0 = Date.now();
|
|
let ok = false;
|
|
try {
|
|
if (!(await runPolicyGate(req, res, stackName, req.nodeId))) return;
|
|
const skipScan = req.body?.skip_scan === true;
|
|
const debug = isDebugEnabled();
|
|
const atomic = true;
|
|
if (debug) console.debug('[Stacks:debug] Update starting', { stackName, atomic, nodeId: req.nodeId });
|
|
const orchResult = await StackUpdateOrchestrator.getInstance().execute(
|
|
{ nodeId: req.nodeId, stackName, target: { scope: 'stack' }, trigger: 'manual', actor: req.user?.username ?? null },
|
|
{ atomic, terminalWs: getTerminalWs(req.get(DEPLOY_SESSION_HEADER)) },
|
|
);
|
|
// Health observation starts immediately after Compose; registry recheck is
|
|
// isolated so a verification failure cannot turn Compose success into 500.
|
|
ok = true;
|
|
const healthGateId = HealthGateService.getInstance().beginStack(req.nodeId, stackName, 'update', req.user?.username ?? null, { deployedGenerationId: orchResult.kind === 'stack_compose_done' ? orchResult.deployedGenerationId : null });
|
|
const recoveryId = orchResult.kind === 'stack_compose_done' ? orchResult.recoveryId : null;
|
|
linkStackUpdateRecoveryGate(recoveryId, healthGateId);
|
|
|
|
let recheckWarning: string | undefined;
|
|
try {
|
|
const recheck = await ImageUpdateService.getInstance().recheckStack(req.nodeId, stackName);
|
|
if (recheck.warning) recheckWarning = recheck.warning;
|
|
} catch (recheckErr) {
|
|
console.warn(
|
|
'[Stacks] Post-update recheck failed for %s: %s',
|
|
sanitizeForLog(stackName),
|
|
sanitizeForLog(getErrorMessage(recheckErr, 'unknown')),
|
|
);
|
|
recheckWarning = UPDATE_VERIFICATION_INCOMPLETE_WARNING;
|
|
}
|
|
|
|
invalidateFleetUpdateCache();
|
|
invalidateNodeCaches(req.nodeId);
|
|
NotificationService.getInstance().broadcastEvent({
|
|
type: 'state-invalidate',
|
|
scope: 'image-updates',
|
|
nodeId: req.nodeId,
|
|
stackName,
|
|
action: 'stack-updated',
|
|
ts: Date.now(),
|
|
});
|
|
dlog(`[Stacks] Update completed: ${sanitizeForLog(stackName)}`);
|
|
if (debug) console.debug(`[Stacks:debug] Update finished in ${Date.now() - t0}ms`);
|
|
res.json({
|
|
status: 'Update completed',
|
|
healthGateId,
|
|
...(recheckWarning ? { recheckWarning } : {}),
|
|
});
|
|
notifyActionSuccess('image_update_applied', `${stackName} updated`, stackName, req.user?.username ?? 'system');
|
|
if (!skipScan) {
|
|
triggerPostDeployScan(stackName, req.nodeId).catch(err =>
|
|
console.error('[Security] Post-deploy scan failed for %s:', sanitizeForLog(stackName), err),
|
|
);
|
|
}
|
|
} catch (error: unknown) {
|
|
console.error('[Stacks] Update failed: %s', sanitizeForLog(stackName), error);
|
|
const rollbackInfo = getComposeRollbackInfo(error);
|
|
const rolledBack = rollbackInfo?.rolledBack ?? false;
|
|
if (rolledBack) {
|
|
console.warn(`[Stacks] Update failed, rolled back: ${sanitizeForLog(stackName)}`);
|
|
} else if (rollbackInfo?.attempted) {
|
|
console.warn(`[Stacks] Update failed, rollback did not complete: ${sanitizeForLog(stackName)}`);
|
|
}
|
|
notifyActionFailure('update', stackName, error, req.user?.username ?? 'system');
|
|
if (!res.headersSent) {
|
|
const message = getErrorMessage(error, 'Failed to update');
|
|
// ComposeRollbackError already carries the cause's message; see classifyFailure.
|
|
const failure = classifyFailure(message, { dockerUnavailable: isDockerUnavailableError(error) });
|
|
if (isDockerUnavailableError(error)) {
|
|
res.status(503).json({ error: message, code: 'docker_unavailable', rolledBack, failure });
|
|
} else {
|
|
res.status(500).json({ error: message, rolledBack, failure });
|
|
}
|
|
}
|
|
} finally {
|
|
releaseStackOpLock(req, stackName);
|
|
StackOpMetricsService.getInstance().record(req.nodeId, 'update', Date.now() - t0, ok);
|
|
}
|
|
});
|
|
|
|
stacksRouter.post('/:stackName/rollback', async (req: Request, res: Response) => {
|
|
const stackName = req.params.stackName as string;
|
|
if (!requirePermission(req, res, 'stack:deploy', 'stack', stackName)) return;
|
|
if (!(await requireStackExists(req.nodeId, stackName, res))) return;
|
|
if (await refuseIfSelfStack(req, res, stackName)) return;
|
|
// Rollback restores files and re-deploys, so it must hold the same per-stack
|
|
// lock deploy/update use. Without it a rollback racing an in-flight deploy
|
|
// would mutate the compose files and run a second `docker compose up` against
|
|
// the same project. Lock held below: all early-returns stay inside the try so
|
|
// finally fires.
|
|
if (!tryAcquireStackOpLock(req, res, stackName, 'rollback')) return;
|
|
let revertRestore: (() => Promise<void>) | null = null;
|
|
try {
|
|
const { StackUpdateRecoveryService } = await import('../services/StackUpdateRecoveryService');
|
|
const recoverySvc = StackUpdateRecoveryService.getInstance();
|
|
const currentGen = recoverySvc.getCurrent(req.nodeId, stackName);
|
|
// Any current recovery row uses compensateWithCandidate. Policy is evaluated
|
|
// against the restored target inside compensate, not the live pre-restore project.
|
|
if (currentGen) {
|
|
dlog(`[Stacks] Rollback initiated via recovery generation: ${sanitizeForLog(stackName)}`);
|
|
try {
|
|
const rolledBack = await recoverySvc.compensateWithCandidate(
|
|
currentGen.id,
|
|
// Returns the Compose result rather than swallowing it, so a proven
|
|
// restore can bind its deployed pointer and open a health run.
|
|
(overridePath, invocation) => ComposeService.getInstance(req.nodeId).composeUpWithRecoveryOverride(
|
|
stackName,
|
|
overridePath,
|
|
getTerminalWs(req.get(DEPLOY_SESSION_HEADER)),
|
|
invocation,
|
|
),
|
|
buildPolicyGateOptions(req, { actor: req.user?.username ?? 'system' }),
|
|
);
|
|
if (!rolledBack) {
|
|
res.status(500).json({ error: 'Rollback restore did not complete.' });
|
|
notifyActionFailure('rollback', stackName, new Error('rollback restore did not complete'), req.user?.username ?? 'system');
|
|
return;
|
|
}
|
|
} catch (compError: unknown) {
|
|
const compCode = (compError as { code?: string }).code;
|
|
if (compCode === 'ROLLBACK_PROHIBITED') {
|
|
res.status(409).json({
|
|
error: (compError as Error).message || 'Rollback is prohibited for this generation',
|
|
code: 'ROLLBACK_PROHIBITED',
|
|
});
|
|
return;
|
|
}
|
|
if (compCode === 'HELD_IMAGE_MISSING') {
|
|
res.status(500).json({
|
|
error: (compError as Error).message || 'Held recovery image is missing.',
|
|
code: 'HELD_IMAGE_MISSING',
|
|
});
|
|
notifyActionFailure('rollback', stackName, compError, req.user?.username ?? 'system');
|
|
return;
|
|
}
|
|
if (compCode === 'RECOVERY_PROBE_FAILED') {
|
|
res.status(500).json({
|
|
error: 'Rollback restore completed but recovery probe failed.',
|
|
code: 'RECOVERY_PROBE_FAILED',
|
|
});
|
|
notifyActionFailure('rollback', stackName, new Error('recovery probe failed'), req.user?.username ?? 'system');
|
|
return;
|
|
}
|
|
throw compError;
|
|
}
|
|
invalidateNodeCaches(req.nodeId);
|
|
dlog(`[Stacks] Rollback completed: ${sanitizeForLog(stackName)}`);
|
|
res.json({ message: 'Stack rolled back from recovery generation.', recoveryId: currentGen.id });
|
|
notifyActionSuccess('deploy_success', `${stackName} rolled back`, stackName, req.user?.username ?? 'system');
|
|
return;
|
|
}
|
|
|
|
const fsSvc = FileSystemService.getInstance(req.nodeId);
|
|
const backupInfo = await fsSvc.getBackupInfo(stackName);
|
|
if (!backupInfo.exists) {
|
|
res.status(404).json({ error: 'No backup available for this stack.' });
|
|
return;
|
|
}
|
|
dlog(`[Stacks] Rollback initiated: ${sanitizeForLog(stackName)}`);
|
|
// Snapshot the current files before restoring so a policy gate that blocks
|
|
// the restored target can be undone: restoreStackFiles commits to disk, and
|
|
// without this a blocked rollback would leave disk rolled back while the
|
|
// deployed state is unchanged.
|
|
revertRestore = await fsSvc.snapshotStackFiles(stackName);
|
|
await fsSvc.restoreStackFiles(stackName);
|
|
if (!(await runPolicyGate(req, res, stackName, req.nodeId))) {
|
|
try {
|
|
await revertRestore();
|
|
} catch (revertError) {
|
|
console.error('[Stacks] Failed to revert files after a policy-blocked rollback: %s', sanitizeForLog(stackName), revertError);
|
|
// The 409 is already sent and the on-disk config now diverges from the
|
|
// running stack; surface it on the persistent alert feed so the operator
|
|
// can repair it rather than discovering it on the next deploy.
|
|
notifyActionFailure('rollback', stackName, revertError, req.user?.username ?? 'system');
|
|
}
|
|
return;
|
|
}
|
|
await ComposeService.getInstance(req.nodeId).deployStack(
|
|
stackName,
|
|
getTerminalWs(req.get(DEPLOY_SESSION_HEADER)),
|
|
false,
|
|
req.deployContext ?? { source: 'rollback', actor: req.user?.username ?? null },
|
|
);
|
|
invalidateNodeCaches(req.nodeId);
|
|
dlog(`[Stacks] Rollback completed: ${sanitizeForLog(stackName)}`);
|
|
res.json({ message: 'Stack rolled back: compose and env files restored.' });
|
|
notifyActionSuccess('deploy_success', `${stackName} rolled back`, stackName, req.user?.username ?? 'system');
|
|
} catch (error: unknown) {
|
|
console.error('[Stacks] Rollback failed: %s', sanitizeForLog(stackName), error);
|
|
const { isMissingExternalNetworksError } = await import('../services/network/missingExternalNetworksError');
|
|
if (isMissingExternalNetworksError(error)) {
|
|
if (revertRestore) {
|
|
try {
|
|
await revertRestore();
|
|
} catch (revertError) {
|
|
console.error('[Stacks] Failed to revert files after a network-blocked rollback: %s', sanitizeForLog(stackName), revertError);
|
|
notifyActionFailure('rollback', stackName, revertError, req.user?.username ?? 'system');
|
|
}
|
|
}
|
|
const status = error.kind === 'unavailable' ? 503 : error.kind === 'create_failed' ? 500 : 409;
|
|
if (!res.headersSent) {
|
|
res.status(status).json({
|
|
error: error.message,
|
|
code: error.code,
|
|
kind: error.kind,
|
|
networks: error.networks,
|
|
});
|
|
}
|
|
return;
|
|
}
|
|
const message = getErrorMessage(error, 'Rollback failed.');
|
|
notifyActionFailure('rollback', stackName, error, req.user?.username ?? 'system');
|
|
if (!res.headersSent) {
|
|
res.status(500).json({ error: message });
|
|
}
|
|
} finally {
|
|
releaseStackOpLock(req, stackName);
|
|
}
|
|
});
|
|
|
|
stacksRouter.get('/:stackName/backup', async (req: Request, res: Response) => {
|
|
try {
|
|
const stackName = req.params.stackName as string;
|
|
const fsSvc = FileSystemService.getInstance(req.nodeId);
|
|
const info = await fsSvc.getBackupInfo(stackName);
|
|
const { StackUpdateRecoveryService } = await import('../services/StackUpdateRecoveryService');
|
|
const currentGen = StackUpdateRecoveryService.getInstance().getCurrent(req.nodeId, stackName);
|
|
const recoveryAvailable = !!currentGen;
|
|
const exists = info.exists || recoveryAvailable;
|
|
const timestamp = currentGen?.created_at ?? info.timestamp;
|
|
res.json({
|
|
exists,
|
|
timestamp,
|
|
recoveryAvailable,
|
|
recoveryId: currentGen?.id ?? null,
|
|
hasGenerationContent: !!(currentGen && currentGen.content_path),
|
|
});
|
|
} catch (error: unknown) {
|
|
console.error('Failed to get backup info:', error);
|
|
const message = getErrorMessage(error, 'Failed to get backup info.');
|
|
res.status(500).json({ error: message });
|
|
}
|
|
});
|
|
|
|
stacksRouter.post('/:stackName/backup', async (req: Request, res: Response) => {
|
|
// Captures files, holds, and a recovery override, then hands off that
|
|
// generation as current without compose or a runtime probe. Exposed so a
|
|
// scheduled backup can run on a remote node through the proxy path, and so
|
|
// an operator can capture an on-demand snapshot.
|
|
const stackName = req.params.stackName as string;
|
|
if (!requirePermission(req, res, 'stack:deploy', 'stack', stackName)) return;
|
|
if (!(await requireStackExists(req.nodeId, stackName, res))) return;
|
|
// Handoff mutates the current generation, so hold the stack-op lock against
|
|
// a concurrent deploy/update/rollback. After a successful acquire, release
|
|
// in finally.
|
|
if (!tryAcquireStackOpLock(req, res, stackName, 'backup')) return;
|
|
try {
|
|
await StackUpdateRecoveryService.getInstance().captureCurrentBackup({
|
|
nodeId: req.nodeId,
|
|
stackName,
|
|
createdBy: req.user?.username ?? null,
|
|
});
|
|
dlog(`[Stacks] Backup completed: ${sanitizeForLog(stackName)}`);
|
|
res.json({ success: true });
|
|
} catch (error: unknown) {
|
|
console.error('[Stacks] Backup failed: %s', sanitizeForLog(stackName), error);
|
|
res.status(500).json({ error: getErrorMessage(error, 'Failed to back up stack files') });
|
|
} finally {
|
|
releaseStackOpLock(req, stackName);
|
|
}
|
|
});
|
|
|
|
function isFleetSnapshotFile(value: unknown): value is { filename: string; content: string } {
|
|
return typeof value === 'object' && value !== null
|
|
&& 'filename' in value && 'content' in value
|
|
&& typeof value.filename === 'string'
|
|
&& typeof value.content === 'string';
|
|
}
|
|
|
|
stacksRouter.post('/:stackName/fleet-snapshot-apply', async (req: Request, res: Response) => {
|
|
// Node-local capture-then-write used by hub Fleet snapshot restore.
|
|
const stackName = req.params.stackName as string;
|
|
if (!requirePermission(req, res, 'stack:edit', 'stack', stackName)) return;
|
|
const rawFiles = req.body?.files;
|
|
if (!Array.isArray(rawFiles) || !rawFiles.every(isFleetSnapshotFile)) {
|
|
res.status(400).json({ error: 'files must be an array of { filename, content }' });
|
|
return;
|
|
}
|
|
const files = selectFleetSnapshotApplyFiles(rawFiles);
|
|
if (files.length === 0) {
|
|
res.status(400).json({ error: 'files must include compose.yaml or .env' });
|
|
return;
|
|
}
|
|
try {
|
|
const result = await applyFleetSnapshotFiles({
|
|
nodeId: req.nodeId,
|
|
stackName,
|
|
files,
|
|
actor: req.user?.username ?? 'system',
|
|
});
|
|
res.json({ success: true, capturedGenerationId: result.capturedGenerationId });
|
|
} catch (error: unknown) {
|
|
const code = getCodedError(error)?.code;
|
|
if (code === 'INVALID_STACK_NAME' || code === 'INVALID_SNAPSHOT_FILES') {
|
|
res.status(400).json({ error: getErrorMessage(error, 'Invalid restore request'), code });
|
|
return;
|
|
}
|
|
const conflict = fleetSnapshotApplyConflictCode(error);
|
|
if (conflict) {
|
|
res.status(409).json({ error: getErrorMessage(error, 'Restore conflict'), code: conflict });
|
|
return;
|
|
}
|
|
console.error('[Stacks] Fleet snapshot apply failed: %s', sanitizeForLog(stackName), error);
|
|
res.status(500).json({ error: getErrorMessage(error, 'Failed to restore snapshot files') });
|
|
}
|
|
});
|
|
|
|
/**
|
|
* Returns the latest post-deploy scan attempt for this stack, or null if
|
|
* no scan has been attempted yet. Used by the editor UI to flag stacks
|
|
* whose latest deploy did not trigger a successful scan (Trivy missing,
|
|
* registry rejection, etc).
|
|
*/
|
|
stacksRouter.get('/:stackName/scan-status', (req: Request, res: Response): void => {
|
|
const stackName = req.params.stackName as string;
|
|
const row = DatabaseService.getInstance().getStackScanAttempt(req.nodeId, stackName);
|
|
if (!row) {
|
|
res.json({ status: null });
|
|
return;
|
|
}
|
|
res.json({
|
|
status: row.status,
|
|
attemptedAt: row.attempted_at,
|
|
errorMessage: row.error_message,
|
|
});
|
|
});
|
|
|
|
// ── File explorer endpoints ──
|
|
|
|
type FsErrorCode =
|
|
| 'INVALID_PATH'
|
|
| 'SYMLINK_ESCAPE'
|
|
| 'INVALID_ROOT'
|
|
| 'READONLY_ROOT'
|
|
| 'ROOT_UNAVAILABLE'
|
|
| 'UNSUPPORTED_ON_ROOT'
|
|
| 'IS_DIRECTORY'
|
|
| 'NOT_EMPTY'
|
|
| 'NOT_FOUND'
|
|
| 'TOO_LARGE'
|
|
| 'ALREADY_EXISTS'
|
|
| 'FILE_EXISTS'
|
|
| 'DIR_EXISTS'
|
|
| 'PROTECTED_FILE'
|
|
| 'LINK_CHMOD_UNSUPPORTED'
|
|
| 'EXDEV';
|
|
|
|
function sendFsError(
|
|
res: Response,
|
|
err: unknown,
|
|
fallback: string,
|
|
opts: { notFoundMessage?: string } = {},
|
|
): Response {
|
|
const e = err as NodeJS.ErrnoException & { code?: string };
|
|
if (e.code === 'INVALID_PATH' || e.code === 'SYMLINK_ESCAPE') {
|
|
return res.status(400).json({ error: e.message, code: e.code as FsErrorCode });
|
|
}
|
|
if (e.code === 'IS_DIRECTORY') {
|
|
return res.status(400).json({ error: e.message, code: e.code as FsErrorCode });
|
|
}
|
|
if (e.code === 'NOT_EMPTY') {
|
|
return res.status(409).json({ error: e.message, code: e.code as FsErrorCode });
|
|
}
|
|
if (e.code === 'PROTECTED_FILE') {
|
|
return res.status(409).json({ error: e.message, code: 'PROTECTED_FILE' satisfies FsErrorCode });
|
|
}
|
|
if (e.code === 'LINK_CHMOD_UNSUPPORTED') {
|
|
return res.status(409).json({ error: e.message, code: 'LINK_CHMOD_UNSUPPORTED' satisfies FsErrorCode });
|
|
}
|
|
if (e.code === 'EEXIST') {
|
|
return res.status(409).json({ error: e.message, code: 'ALREADY_EXISTS' satisfies FsErrorCode });
|
|
}
|
|
if (e.code === 'EXDEV') {
|
|
return res.status(409).json({ error: 'Cannot move across a storage boundary', code: 'EXDEV' satisfies FsErrorCode });
|
|
}
|
|
if (e.code === 'ENOTDIR') {
|
|
return res.status(400).json({ error: 'Target path is not a directory', code: 'INVALID_PATH' satisfies FsErrorCode });
|
|
}
|
|
if (e.code === 'ENOENT') {
|
|
return res.status(404).json({ error: opts.notFoundMessage ?? 'File not found', code: 'NOT_FOUND' });
|
|
}
|
|
if (e.code === 'UNSUPPORTED_ON_ROOT') {
|
|
return res.status(400).json({ error: e.message, code: 'UNSUPPORTED_ON_ROOT' satisfies FsErrorCode });
|
|
}
|
|
if (e.code === 'FILE_EXISTS') {
|
|
return res.status(409).json({ error: e.message, code: 'FILE_EXISTS' satisfies FsErrorCode });
|
|
}
|
|
// Helper-backed (named-volume) ops throw an ExecError carrying an HTTP status;
|
|
// honour it (403 permission-denied, 409 conflict, 413 too-large, 504 timeout).
|
|
// A 4xx message is a self-explanatory client error and is forwarded as-is; a
|
|
// 5xx is a server-side failure, so log the detail and return a clean message.
|
|
const status = (err as { status?: unknown }).status;
|
|
if (typeof status === 'number' && status >= 400 && status < 600) {
|
|
if (status >= 500) {
|
|
// Constant format string + sanitized args (the status is already carried by
|
|
// the HTTP response, so it is not repeated in the log).
|
|
console.error('[files] %s (helper failure): %s', sanitizeForLog(fallback), sanitizeForLog(e.message));
|
|
return res.status(status).json({ error: fallback });
|
|
}
|
|
if (status === 409 && /not empty/i.test(e.message)) {
|
|
return res.status(409).json({ error: e.message, code: 'NOT_EMPTY' satisfies FsErrorCode });
|
|
}
|
|
return res.status(status).json({ error: e.message });
|
|
}
|
|
console.error(`[files] ${fallback}:`, sanitizeForLog(e.message));
|
|
return res.status(500).json({ error: fallback });
|
|
}
|
|
|
|
const ROOT_ID_RE = /^[A-Za-z0-9:_-]{1,80}$/;
|
|
|
|
function readRootId(req: Request): string {
|
|
const raw = req.query.rootId;
|
|
return typeof raw === 'string' && raw ? raw : STACK_SOURCE_ROOT_ID;
|
|
}
|
|
|
|
/**
|
|
* Resolve the client's rootId to a server-derived root, enforcing browsability
|
|
* for reads and writability for writes. Sends the error response and returns
|
|
* null when the root is unknown, read-only, or not browsable, so the caller
|
|
* just does `if (!root) return;`. Writes resolve fresh (cache bypass) so a
|
|
* removed mount cannot be written through a stale allowlist.
|
|
*/
|
|
async function resolveRootForOp(
|
|
req: Request,
|
|
res: Response,
|
|
stackName: string,
|
|
mode: 'read' | 'write',
|
|
): Promise<StackFileRoot | null> {
|
|
const rootId = readRootId(req);
|
|
// Back-compat fast path: no rootId (or the stack-source root) is exactly the
|
|
// legacy behaviour. Return a synthetic stack-source root without touching the
|
|
// roots service, so plain stack-source ops never trigger a compose render.
|
|
if (rootId === STACK_SOURCE_ROOT_ID) {
|
|
return stackSourceFileRoot();
|
|
}
|
|
if (!ROOT_ID_RE.test(rootId)) {
|
|
res.status(400).json({ error: 'Invalid root', code: 'INVALID_ROOT' satisfies FsErrorCode });
|
|
return null;
|
|
}
|
|
let root: StackFileRoot;
|
|
try {
|
|
root = await StackFileRootsService.getInstance(req.nodeId).resolveRoot(stackName, rootId, { fresh: mode === 'write' });
|
|
} catch (err) {
|
|
if ((err as { code?: string }).code === 'INVALID_ROOT') {
|
|
res.status(400).json({ error: 'Unknown file root', code: 'INVALID_ROOT' satisfies FsErrorCode });
|
|
return null;
|
|
}
|
|
sendFsError(res, err, 'Failed to resolve file root');
|
|
return null;
|
|
}
|
|
if (mode === 'write' && !root.writable) {
|
|
res.status(403).json({ error: root.warning ?? 'This location is read-only.', code: 'READONLY_ROOT' satisfies FsErrorCode });
|
|
return null;
|
|
}
|
|
if (mode === 'read' && !root.browsable) {
|
|
res.status(400).json({ error: root.warning ?? 'This location cannot be browsed.', code: 'ROOT_UNAVAILABLE' satisfies FsErrorCode });
|
|
return null;
|
|
}
|
|
return root;
|
|
}
|
|
|
|
/** Drop the cached root allowlist after a stack-source mutation that can change declared mounts. */
|
|
function afterStackMutation(req: Request, stackName: string): void {
|
|
StackFileRootsService.invalidate(req.nodeId, stackName);
|
|
}
|
|
|
|
function logFileOperation(level: 'info' | 'warn', message: string, details: Record<string, unknown>): void {
|
|
const cleaned = Object.fromEntries(
|
|
Object.entries(details).map(([key, value]) => [key, sanitizeForLog(value)]),
|
|
);
|
|
const log = level === 'warn' ? console.warn : console.log;
|
|
log(`[Files] ${message}`, cleaned);
|
|
}
|
|
|
|
function fsErrorCode(err: unknown): string {
|
|
const code = (err as NodeJS.ErrnoException & { code?: unknown }).code;
|
|
return typeof code === 'string' ? code : 'UNKNOWN';
|
|
}
|
|
|
|
function logFileDiag(message: string, details: Record<string, unknown>): void {
|
|
if (DatabaseService.getInstance().getGlobalSettings().developer_mode !== '1') return;
|
|
const cleaned = Object.fromEntries(
|
|
Object.entries(details).map(([key, value]) => [key, sanitizeForLog(value)]),
|
|
);
|
|
console.debug(`[Files:diag] ${message}`, cleaned);
|
|
}
|
|
|
|
/**
|
|
* Records one file-explorer op into the in-process metrics service. Always
|
|
* called once per request from the route layer, regardless of success or
|
|
* failure, so the counts in the `/api/file-explorer-metrics` snapshot stay
|
|
* in step with the INFO log line emitted in the same handler.
|
|
*/
|
|
function recordFileOp(nodeId: number, op: FileExplorerOp, startedAt: number, ok: boolean): void {
|
|
FileExplorerMetricsService.getInstance().record(nodeId, op, Date.now() - startedAt, ok);
|
|
}
|
|
|
|
/**
|
|
* Emits the warn log, records the metric, and sends the JSON response for a
|
|
* mutation rejected with a known error code (e.g. DIR_EXISTS, FILE_EXISTS,
|
|
* PRECONDITION_FAILED). Centralizes the three-step shape so a future rejection
|
|
* site cannot skip the metric and drift from the log.
|
|
*/
|
|
function rejectFileMutation(
|
|
req: Request,
|
|
res: Response,
|
|
args: {
|
|
op: FileExplorerOp;
|
|
stack: string;
|
|
path: string;
|
|
startedAt: number;
|
|
status: number;
|
|
code: string;
|
|
body: Record<string, unknown>;
|
|
},
|
|
): Response {
|
|
logFileOperation('warn', 'mutate rejected', {
|
|
nodeId: req.nodeId,
|
|
op: args.op,
|
|
stack: args.stack,
|
|
path: args.path,
|
|
errorCode: args.code,
|
|
});
|
|
recordFileOp(req.nodeId, args.op, args.startedAt, false);
|
|
return res.status(args.status).json({ ...args.body, code: args.code });
|
|
}
|
|
|
|
function isSafeUploadFilename(rawName: string): boolean {
|
|
if (!rawName || rawName === '.' || rawName === '..') return false;
|
|
if (rawName.includes('\0') || rawName.includes('/') || rawName.includes('\\')) return false;
|
|
if (/^[a-zA-Z]:/.test(rawName) || path.isAbsolute(rawName)) return false;
|
|
return path.basename(rawName) === rawName;
|
|
}
|
|
|
|
const DIR_LIST_LIMIT = 1000;
|
|
|
|
stacksRouter.get('/:stackName/file-roots', async (req: Request, res: Response) => {
|
|
const stackName = req.params.stackName as string;
|
|
if (!requirePermission(req, res, 'stack:read', 'stack', stackName)) return;
|
|
try {
|
|
const roots = await StackFileRootsService.getInstance(req.nodeId).listRoots(stackName);
|
|
return res.json(roots);
|
|
} catch (err: unknown) {
|
|
return sendFsError(res, err, 'Failed to list file roots');
|
|
}
|
|
});
|
|
|
|
stacksRouter.get('/:stackName/files', async (req: Request, res: Response) => {
|
|
const stackName = req.params.stackName as string;
|
|
if (!requirePermission(req, res, 'stack:read', 'stack', stackName)) return;
|
|
const relPath = getRelPath(req);
|
|
if (relPath !== '' && !isValidRelativeStackPath(relPath)) {
|
|
return res.status(400).json({ error: 'Invalid path', code: 'INVALID_PATH' });
|
|
}
|
|
const root = await resolveRootForOp(req, res, stackName, 'read');
|
|
if (!root) return;
|
|
const startedAt = Date.now();
|
|
logFileDiag('list start', { stackName, relPath, nodeId: req.nodeId, rootKind: root.kind });
|
|
try {
|
|
const result = await FileRootGateway.getInstance(req.nodeId).listDir(root, stackName, relPath, DIR_LIST_LIMIT);
|
|
// Expose pagination context via headers; the JSON body stays
|
|
// FileEntry[] for backward compatibility with any direct API caller.
|
|
res.setHeader('X-Total-Count', String(result.total));
|
|
res.setHeader('X-Returned-Count', String(result.entries.length));
|
|
if (result.truncated) res.setHeader('X-Truncated', 'true');
|
|
logFileDiag('list complete', {
|
|
stackName,
|
|
relPath,
|
|
nodeId: req.nodeId,
|
|
returned: result.entries.length,
|
|
total: result.total,
|
|
truncated: result.truncated,
|
|
elapsedMs: Date.now() - startedAt,
|
|
});
|
|
recordFileOp(req.nodeId, 'list', startedAt, true);
|
|
return res.json(result.entries);
|
|
} catch (err: unknown) {
|
|
logFileOperation('warn', 'list failed', { nodeId: req.nodeId, errorCode: fsErrorCode(err) });
|
|
recordFileOp(req.nodeId, 'list', startedAt, false);
|
|
return sendFsError(res, err, 'Failed to list directory');
|
|
}
|
|
});
|
|
|
|
stacksRouter.get('/:stackName/files/content', async (req: Request, res: Response) => {
|
|
const stackName = req.params.stackName as string;
|
|
if (!requirePermission(req, res, 'stack:read', 'stack', stackName)) return;
|
|
const relPath = getRelPath(req);
|
|
if (!relPath) return res.status(400).json({ error: 'path query parameter is required', code: 'INVALID_PATH' });
|
|
if (!isValidRelativeStackPath(relPath)) {
|
|
return res.status(400).json({ error: 'Invalid path', code: 'INVALID_PATH' });
|
|
}
|
|
const forceText = req.query.force === 'text';
|
|
const root = await resolveRootForOp(req, res, stackName, 'read');
|
|
if (!root) return;
|
|
const startedAt = Date.now();
|
|
logFileDiag('read start', { stackName, relPath, nodeId: req.nodeId, forceText, rootKind: root.kind });
|
|
try {
|
|
const result = await FileRootGateway.getInstance(req.nodeId).read(root, stackName, relPath, forceText);
|
|
// ETag carries the opaque version token the matching PUT compares. For fs
|
|
// roots it is the weak ETag over the integer mtimeMs (unchanged); for helper
|
|
// roots it is a composite token. The body also carries `version` so the
|
|
// client round-trips it verbatim as If-Match.
|
|
res.setHeader('ETag', result.version);
|
|
logFileDiag('read complete', {
|
|
stackName,
|
|
relPath,
|
|
nodeId: req.nodeId,
|
|
binary: result.binary,
|
|
oversized: result.oversized,
|
|
size: result.size,
|
|
elapsedMs: Date.now() - startedAt,
|
|
});
|
|
recordFileOp(req.nodeId, 'read', startedAt, true);
|
|
return res.json(result);
|
|
} catch (err: unknown) {
|
|
logFileOperation('warn', 'read failed', { nodeId: req.nodeId, errorCode: fsErrorCode(err) });
|
|
recordFileOp(req.nodeId, 'read', startedAt, false);
|
|
return sendFsError(res, err, 'Failed to read file');
|
|
}
|
|
});
|
|
|
|
stacksRouter.get('/:stackName/files/download', async (req: Request, res: Response) => {
|
|
const stackName = req.params.stackName as string;
|
|
if (!requirePermission(req, res, 'stack:read', 'stack', stackName)) return;
|
|
const relPath = getRelPath(req);
|
|
if (!relPath) return res.status(400).json({ error: 'path query parameter is required', code: 'INVALID_PATH' });
|
|
if (!isValidRelativeStackPath(relPath)) {
|
|
return res.status(400).json({ error: 'Invalid path', code: 'INVALID_PATH' });
|
|
}
|
|
const root = await resolveRootForOp(req, res, stackName, 'read');
|
|
if (!root) return;
|
|
const startedAt = Date.now();
|
|
logFileDiag('download start', { stackName, relPath, nodeId: req.nodeId, rootKind: root.kind });
|
|
try {
|
|
const result = await FileRootGateway.getInstance(req.nodeId).download(root, stackName, relPath);
|
|
const setDownloadHeaders = (filename: string, size: number, mime: string): void => {
|
|
res.setHeader('Content-Type', mime);
|
|
res.setHeader('Content-Length', size);
|
|
const encodedFilename = encodeURIComponent(filename);
|
|
const safeFilename = filename.replace(/[\\"]/g, '');
|
|
res.setHeader('Content-Disposition', `attachment; filename="${safeFilename}"; filename*=UTF-8''${encodedFilename}`);
|
|
};
|
|
// Helper-backed (named-volume) downloads come back as a bounded buffer; send
|
|
// it directly rather than through the file-stream lifecycle below.
|
|
if (result.kind === 'buffer') {
|
|
setDownloadHeaders(result.filename, result.size, 'application/octet-stream');
|
|
recordFileOp(req.nodeId, 'download', startedAt, true);
|
|
return res.end(result.buffer);
|
|
}
|
|
setDownloadHeaders(result.filename, result.size, result.mime);
|
|
// Track download completion off both the file stream's lifecycle and the
|
|
// response close. Under the in-process supertest transport, request close
|
|
// events can race ahead of normal stream completion. End/error are the
|
|
// durable source signals; a close (source or response) counts as success
|
|
// only when the stream reports it read the full file, and as an abort
|
|
// otherwise.
|
|
let downloadRecorded = false;
|
|
const streamWithBytes = result.stream as typeof result.stream & { bytesRead?: number };
|
|
const hasReadFullFile = (): boolean => (
|
|
result.size === 0 ||
|
|
(typeof streamWithBytes.bytesRead === 'number' && streamWithBytes.bytesRead >= result.size)
|
|
);
|
|
let abortCleanupHandle: NodeJS.Immediate | null = null;
|
|
const clearAbortCleanup = (): void => {
|
|
if (!abortCleanupHandle) return;
|
|
clearImmediate(abortCleanupHandle);
|
|
abortCleanupHandle = null;
|
|
};
|
|
const recordDownloadOnce = (ok: boolean): void => {
|
|
if (downloadRecorded) return;
|
|
downloadRecorded = true;
|
|
clearAbortCleanup();
|
|
recordFileOp(req.nodeId, 'download', startedAt, ok);
|
|
};
|
|
result.stream.on('error', (streamErr) => {
|
|
console.error('[files] stream error:', sanitizeForLog(getErrorMessage(streamErr, 'unknown')));
|
|
if (!res.headersSent) {
|
|
res.removeHeader('Content-Length');
|
|
res.status(500).end();
|
|
} else {
|
|
res.destroy();
|
|
}
|
|
recordDownloadOnce(false);
|
|
});
|
|
result.stream.on('end', () => recordDownloadOnce(hasReadFullFile()));
|
|
result.stream.on('close', () => recordDownloadOnce(hasReadFullFile()));
|
|
res.on('close', () => {
|
|
if (downloadRecorded) return;
|
|
// This op measures a server-side file read, so once the source has
|
|
// streamed the whole file a response close is a successful completion.
|
|
// Record it here rather than waiting on the source stream's end/close,
|
|
// which can be dropped once the response consumer is gone, leaving the
|
|
// op unrecorded.
|
|
if (hasReadFullFile()) { recordDownloadOnce(true); return; }
|
|
// Otherwise response close is an abort signal. It can beat the source
|
|
// stream's final events in the in-process test transport, so give a
|
|
// same-turn clean source completion a chance to win before cleanup.
|
|
abortCleanupHandle = setImmediate(() => {
|
|
abortCleanupHandle = null;
|
|
if (downloadRecorded) return;
|
|
if (hasReadFullFile()) { recordDownloadOnce(true); return; }
|
|
recordDownloadOnce(false);
|
|
result.stream.destroy();
|
|
});
|
|
abortCleanupHandle.unref?.();
|
|
});
|
|
logFileDiag('download stream opened', { stackName, relPath, nodeId: req.nodeId, size: result.size, elapsedMs: Date.now() - startedAt });
|
|
result.stream.pipe(res);
|
|
return;
|
|
} catch (err: unknown) {
|
|
logFileOperation('warn', 'download failed', { nodeId: req.nodeId, errorCode: fsErrorCode(err) });
|
|
recordFileOp(req.nodeId, 'download', startedAt, false);
|
|
return sendFsError(res, err, 'Failed to download file');
|
|
}
|
|
});
|
|
|
|
type UploadStartedReq = Request & { _fileUploadStartedAt?: number; _fileUploadRoot?: StackFileRoot };
|
|
|
|
stacksRouter.post(
|
|
'/:stackName/files/upload',
|
|
// Authorize BEFORE multer touches the body, so an unauthorized caller or a
|
|
// read-only/non-existent root is rejected without ever spooling a temp file.
|
|
// The resolved root is stashed for the handler so it is not resolved twice.
|
|
async (req: Request, res: Response, next: NextFunction) => {
|
|
const stackName = req.params.stackName as string;
|
|
if (!requirePermission(req, res, 'stack:edit', 'stack', stackName)) return;
|
|
const root = await resolveRootForOp(req, res, stackName, 'write');
|
|
if (!root) return;
|
|
(req as UploadStartedReq)._fileUploadRoot = root;
|
|
next();
|
|
},
|
|
(req: Request, res: Response, next: NextFunction) => {
|
|
// Capture the time the upload entered the route so every downstream
|
|
// metric reports the same latency window: the body-transfer +
|
|
// parser-buffer time on success, plus the multer rejection branches
|
|
// below. Reading a fresh Date.now() after multer would hide the
|
|
// multipart upload's network and buffering cost from the histogram.
|
|
const startedAt = Date.now();
|
|
(req as UploadStartedReq)._fileUploadStartedAt = startedAt;
|
|
upload.single('file')(req, res, (err) => {
|
|
if (err && (err as multer.MulterError).code === 'LIMIT_FILE_SIZE') {
|
|
logFileOperation('warn', 'mutate rejected', {
|
|
nodeId: req.nodeId,
|
|
op: 'upload',
|
|
stack: req.params.stackName,
|
|
errorCode: 'TOO_LARGE',
|
|
});
|
|
recordFileOp(req.nodeId, 'upload', startedAt, false);
|
|
// diskStorage may have spooled a partial file before the limit fired.
|
|
void cleanupUploadTemp(req).finally(() =>
|
|
res.status(413).json({ error: 'File exceeds 25 MB limit', code: 'TOO_LARGE' }),
|
|
);
|
|
return;
|
|
}
|
|
if (err) {
|
|
logFileOperation('warn', 'upload failed', {
|
|
nodeId: req.nodeId,
|
|
op: 'upload',
|
|
stack: req.params.stackName,
|
|
errorCode: 'MULTER_ERROR',
|
|
});
|
|
recordFileOp(req.nodeId, 'upload', startedAt, false);
|
|
void cleanupUploadTemp(req).finally(() => res.status(500).json({ error: 'Upload failed' }));
|
|
return;
|
|
}
|
|
next();
|
|
});
|
|
},
|
|
async (req: Request, res: Response) => {
|
|
const stackName = req.params.stackName as string;
|
|
// The pre-multer middleware already authorized and resolved the root.
|
|
const root = (req as UploadStartedReq)._fileUploadRoot;
|
|
if (!root) {
|
|
await cleanupUploadTemp(req);
|
|
return res.status(500).json({ error: 'Upload failed' });
|
|
}
|
|
if (!req.file) {
|
|
return res.status(400).json({ error: 'No file provided' });
|
|
}
|
|
const relPath = getRelPath(req);
|
|
if (relPath !== '' && !isValidRelativeStackPath(relPath)) {
|
|
await cleanupUploadTemp(req);
|
|
return res.status(400).json({ error: 'Invalid path', code: 'INVALID_PATH' });
|
|
}
|
|
const originalName = req.file.originalname;
|
|
if (!isSafeUploadFilename(originalName)) {
|
|
await cleanupUploadTemp(req);
|
|
return res.status(400).json({ error: 'Invalid filename' });
|
|
}
|
|
const targetRelPath = relPath ? `${relPath}/${originalName}` : originalName;
|
|
if (rejectIfStackOpBlocksRootTrustFileWrite(req, res, stackName, targetRelPath, root)) {
|
|
await cleanupUploadTemp(req);
|
|
return;
|
|
}
|
|
const overwrite = String(req.query.overwrite) === '1';
|
|
// The multer wrapper stashed the route-entry timestamp on the request so
|
|
// the success path and the rejection paths share one window. Fall back to
|
|
// Date.now() defensively in case the wrapper was bypassed in a test.
|
|
const startedAt = (req as UploadStartedReq)._fileUploadStartedAt ?? Date.now();
|
|
logFileDiag('upload start', { stackName, relPath: targetRelPath, nodeId: req.nodeId, size: req.file.size, overwrite });
|
|
try {
|
|
const gateway = FileRootGateway.getInstance(req.nodeId);
|
|
const existing = await gateway.pathKind(root, stackName, targetRelPath);
|
|
if (existing === 'directory') {
|
|
// A directory can never be replaced by an upload; surface a distinct code
|
|
// so the UI does not offer a useless "Replace" button.
|
|
return rejectFileMutation(req, res, {
|
|
op: 'upload',
|
|
stack: stackName,
|
|
path: targetRelPath,
|
|
startedAt,
|
|
status: 409,
|
|
code: 'DIR_EXISTS',
|
|
body: {
|
|
error: `A folder named ${originalName} already exists in this folder. Rename the upload or remove the folder first.`,
|
|
},
|
|
});
|
|
}
|
|
if (existing === 'file' && !overwrite) {
|
|
// Real FS work ran (pathKind) and the operator was rejected; surface
|
|
// the conflict in metrics so a node with a hot overwrite-confirm
|
|
// pattern shows up in the snapshot rather than disappearing.
|
|
return rejectFileMutation(req, res, {
|
|
op: 'upload',
|
|
stack: stackName,
|
|
path: targetRelPath,
|
|
startedAt,
|
|
status: 409,
|
|
code: 'FILE_EXISTS',
|
|
body: {
|
|
error: `${originalName} already exists in this folder. Confirm to replace.`,
|
|
},
|
|
});
|
|
}
|
|
// Canonical js/path-injection barrier: the spool path is multer-generated
|
|
// within UPLOAD_TMP_DIR (a random filename), but static analysis taints
|
|
// req.file.*; confirm containment so the value handed to the gateway and
|
|
// FileSystemService streaming sinks is credited as safe.
|
|
const spoolBase = path.resolve(UPLOAD_TMP_DIR);
|
|
const tempPath = path.resolve(req.file.path);
|
|
if (!tempPath.startsWith(spoolBase + path.sep)) {
|
|
return res.status(400).json({ error: 'Upload failed' });
|
|
}
|
|
// Copy the spooled temp file into place (the spool survives; the finally
|
|
// removes it). The atomic exclusive create for the non-overwrite case means
|
|
// a file created by another writer after the pathKind check above is not
|
|
// silently clobbered (a race surfaces as FILE_EXISTS -> 409, same as the
|
|
// pre-emptive check). overwrite=true intentionally allows the clobber.
|
|
await gateway.writeFromTemp(root, stackName, targetRelPath, tempPath, !overwrite);
|
|
afterStackMutation(req, stackName);
|
|
logFileOperation('info', 'mutate', {
|
|
nodeId: req.nodeId,
|
|
op: 'upload',
|
|
stack: stackName,
|
|
path: targetRelPath,
|
|
bytes: req.file.size,
|
|
overwrite,
|
|
rootKind: root.kind,
|
|
backend: root.backend,
|
|
});
|
|
logFileDiag('upload timing', { stackName, relPath: targetRelPath, nodeId: req.nodeId, elapsedMs: Date.now() - startedAt });
|
|
recordFileOp(req.nodeId, 'upload', startedAt, true);
|
|
FileExplorerMetricsService.getInstance().recordUploadBytes(req.nodeId, req.file.size);
|
|
return res.status(204).send();
|
|
} catch (err: unknown) {
|
|
logFileOperation('warn', 'upload failed', {
|
|
nodeId: req.nodeId,
|
|
op: 'upload',
|
|
stack: stackName,
|
|
path: targetRelPath,
|
|
errorCode: fsErrorCode(err),
|
|
});
|
|
recordFileOp(req.nodeId, 'upload', startedAt, false);
|
|
return sendFsError(res, err, 'Failed to upload file', { notFoundMessage: 'Target directory not found' });
|
|
} finally {
|
|
// writeFromTemp streams (copies) the spool into place, so the temp file
|
|
// always remains and must be removed on every exit (success, conflict, error).
|
|
await cleanupUploadTemp(req);
|
|
}
|
|
},
|
|
);
|
|
|
|
stacksRouter.put('/:stackName/files/content', async (req: Request, res: Response) => {
|
|
const stackName = req.params.stackName as string;
|
|
if (!requirePermission(req, res, 'stack:edit', 'stack', stackName)) return;
|
|
const relPath = getRelPath(req);
|
|
if (!relPath) return res.status(400).json({ error: 'path query parameter is required', code: 'INVALID_PATH' });
|
|
if (!isValidRelativeStackPath(relPath)) {
|
|
return res.status(400).json({ error: 'Invalid path', code: 'INVALID_PATH' });
|
|
}
|
|
const { content } = req.body as { content?: unknown };
|
|
if (typeof content !== 'string') {
|
|
return res.status(400).json({ error: '"content" must be a string' });
|
|
}
|
|
const expectedVersion = req.header('if-match') || undefined;
|
|
const root = await resolveRootForOp(req, res, stackName, 'write');
|
|
if (!root) return;
|
|
if (rejectIfStackOpBlocksRootTrustFileWrite(req, res, stackName, relPath, root)) return;
|
|
const startedAt = Date.now();
|
|
logFileDiag('write start', { stackName, relPath, nodeId: req.nodeId, bytes: Buffer.byteLength(content, 'utf-8'), hasIfMatch: expectedVersion !== undefined, rootKind: root.kind });
|
|
try {
|
|
const result = await FileRootGateway.getInstance(req.nodeId).writeIfUnchanged(root, stackName, relPath, content, expectedVersion);
|
|
if (!result.ok) {
|
|
// Stale version: surface the current content + token so the client can
|
|
// show a "file changed elsewhere" diff and retry with the fresh version.
|
|
// Real work ran (the if-unchanged compare); record the attempted-and-
|
|
// rejected write so concurrent-edit patterns show in the snapshot.
|
|
res.setHeader('ETag', result.currentVersion);
|
|
return rejectFileMutation(req, res, {
|
|
op: 'write',
|
|
stack: stackName,
|
|
path: relPath,
|
|
startedAt,
|
|
status: 412,
|
|
code: 'PRECONDITION_FAILED',
|
|
body: {
|
|
error: 'File has been modified since you last read it. Reload to see the current version.',
|
|
currentMtimeMs: result.currentMtimeMs,
|
|
currentContent: result.currentContent,
|
|
currentVersion: result.currentVersion,
|
|
},
|
|
});
|
|
}
|
|
res.setHeader('ETag', result.version);
|
|
afterStackMutation(req, stackName);
|
|
logFileOperation('info', 'mutate', {
|
|
nodeId: req.nodeId,
|
|
op: 'write',
|
|
stack: stackName,
|
|
path: relPath,
|
|
bytes: Buffer.byteLength(content, 'utf-8'),
|
|
rootKind: root.kind,
|
|
backend: root.backend,
|
|
});
|
|
logFileDiag('write timing', { stackName, relPath, nodeId: req.nodeId, elapsedMs: Date.now() - startedAt });
|
|
recordFileOp(req.nodeId, 'write', startedAt, true);
|
|
return res.status(204).send();
|
|
} catch (err: unknown) {
|
|
logFileOperation('warn', 'write failed', {
|
|
nodeId: req.nodeId,
|
|
op: 'write',
|
|
stack: stackName,
|
|
path: relPath,
|
|
errorCode: fsErrorCode(err),
|
|
});
|
|
recordFileOp(req.nodeId, 'write', startedAt, false);
|
|
return sendFsError(res, err, 'Failed to write file');
|
|
}
|
|
});
|
|
|
|
stacksRouter.delete('/:stackName/files', async (req: Request, res: Response) => {
|
|
const stackName = req.params.stackName as string;
|
|
if (!requirePermission(req, res, 'stack:edit', 'stack', stackName)) return;
|
|
const relPath = getRelPath(req);
|
|
if (relPath === '') return res.status(400).json({ error: 'Path is required for delete' });
|
|
if (!isValidRelativeStackPath(relPath)) {
|
|
return res.status(400).json({ error: 'Invalid path', code: 'INVALID_PATH' });
|
|
}
|
|
const recursive = req.query.recursive === '1';
|
|
const root = await resolveRootForOp(req, res, stackName, 'write');
|
|
if (!root) return;
|
|
const startedAt = Date.now();
|
|
logFileDiag('delete start', { stackName, relPath, recursive, nodeId: req.nodeId, rootKind: root.kind });
|
|
try {
|
|
await FileRootGateway.getInstance(req.nodeId).deletePath(root, stackName, relPath, recursive);
|
|
afterStackMutation(req, stackName);
|
|
logFileOperation('info', 'mutate', {
|
|
nodeId: req.nodeId,
|
|
op: 'delete',
|
|
stack: stackName,
|
|
path: relPath,
|
|
recursive,
|
|
rootKind: root.kind,
|
|
backend: root.backend,
|
|
});
|
|
logFileDiag('delete timing', { stackName, relPath, recursive, nodeId: req.nodeId, elapsedMs: Date.now() - startedAt });
|
|
recordFileOp(req.nodeId, 'delete', startedAt, true);
|
|
return res.status(204).send();
|
|
} catch (err: unknown) {
|
|
logFileOperation('warn', 'delete failed', {
|
|
nodeId: req.nodeId,
|
|
op: 'delete',
|
|
stack: stackName,
|
|
path: relPath,
|
|
recursive,
|
|
errorCode: fsErrorCode(err),
|
|
});
|
|
recordFileOp(req.nodeId, 'delete', startedAt, false);
|
|
return sendFsError(res, err, 'Failed to delete path');
|
|
}
|
|
});
|
|
|
|
stacksRouter.post('/:stackName/files/folder', async (req: Request, res: Response) => {
|
|
const stackName = req.params.stackName as string;
|
|
if (!requirePermission(req, res, 'stack:edit', 'stack', stackName)) return;
|
|
const relPath = getRelPath(req);
|
|
if (relPath === '') return res.status(400).json({ error: 'Path is required to create a folder' });
|
|
if (!isValidRelativeStackPath(relPath)) {
|
|
return res.status(400).json({ error: 'Invalid path', code: 'INVALID_PATH' });
|
|
}
|
|
const root = await resolveRootForOp(req, res, stackName, 'write');
|
|
if (!root) return;
|
|
const startedAt = Date.now();
|
|
logFileDiag('mkdir start', { stackName, relPath, nodeId: req.nodeId, rootKind: root.kind });
|
|
try {
|
|
await FileRootGateway.getInstance(req.nodeId).mkdir(root, stackName, relPath);
|
|
afterStackMutation(req, stackName);
|
|
logFileOperation('info', 'mutate', {
|
|
nodeId: req.nodeId,
|
|
op: 'mkdir',
|
|
stack: stackName,
|
|
path: relPath,
|
|
rootKind: root.kind,
|
|
backend: root.backend,
|
|
});
|
|
logFileDiag('mkdir timing', { stackName, relPath, nodeId: req.nodeId, elapsedMs: Date.now() - startedAt });
|
|
recordFileOp(req.nodeId, 'mkdir', startedAt, true);
|
|
return res.status(204).send();
|
|
} catch (err: unknown) {
|
|
logFileOperation('warn', 'mkdir failed', {
|
|
nodeId: req.nodeId,
|
|
op: 'mkdir',
|
|
stack: stackName,
|
|
path: relPath,
|
|
errorCode: fsErrorCode(err),
|
|
});
|
|
recordFileOp(req.nodeId, 'mkdir', startedAt, false);
|
|
return sendFsError(res, err, 'Failed to create folder');
|
|
}
|
|
});
|
|
|
|
stacksRouter.patch('/:stackName/files/rename', async (req: Request, res: Response) => {
|
|
const stackName = req.params.stackName as string;
|
|
if (!requirePermission(req, res, 'stack:edit', 'stack', stackName)) return;
|
|
const { from, to } = req.body as { from?: unknown; to?: unknown };
|
|
if (typeof from !== 'string' || !from) {
|
|
return res.status(400).json({ error: '"from" must be a non-empty string' });
|
|
}
|
|
if (typeof to !== 'string' || !to) {
|
|
return res.status(400).json({ error: '"to" must be a non-empty string' });
|
|
}
|
|
if (!isValidRelativeStackPath(from)) {
|
|
return res.status(400).json({ error: 'Invalid source path', code: 'INVALID_PATH' });
|
|
}
|
|
if (!isValidRelativeStackPath(to)) {
|
|
return res.status(400).json({ error: 'Invalid destination path', code: 'INVALID_PATH' });
|
|
}
|
|
const root = await resolveRootForOp(req, res, stackName, 'write');
|
|
if (!root) return;
|
|
const startedAt = Date.now();
|
|
logFileDiag('rename start', { stackName, from, to, nodeId: req.nodeId, rootKind: root.kind });
|
|
try {
|
|
await FileRootGateway.getInstance(req.nodeId).rename(root, stackName, from, to);
|
|
afterStackMutation(req, stackName);
|
|
logFileOperation('info', 'mutate', {
|
|
nodeId: req.nodeId,
|
|
op: 'rename',
|
|
stack: stackName,
|
|
path: from,
|
|
toPath: to,
|
|
rootKind: root.kind,
|
|
backend: root.backend,
|
|
});
|
|
logFileDiag('rename timing', { stackName, from, to, nodeId: req.nodeId, elapsedMs: Date.now() - startedAt });
|
|
recordFileOp(req.nodeId, 'rename', startedAt, true);
|
|
return res.status(204).send();
|
|
} catch (err: unknown) {
|
|
logFileOperation('warn', 'rename failed', {
|
|
nodeId: req.nodeId,
|
|
op: 'rename',
|
|
stack: stackName,
|
|
path: from,
|
|
toPath: to,
|
|
errorCode: fsErrorCode(err),
|
|
});
|
|
recordFileOp(req.nodeId, 'rename', startedAt, false);
|
|
return sendFsError(res, err, 'Failed to rename');
|
|
}
|
|
});
|
|
|
|
stacksRouter.post('/:stackName/files/copy', async (req: Request, res: Response) => {
|
|
const stackName = req.params.stackName as string;
|
|
if (!requirePermission(req, res, 'stack:edit', 'stack', stackName)) return;
|
|
const { from, to } = req.body as { from?: unknown; to?: unknown };
|
|
if (typeof from !== 'string' || !from) {
|
|
return res.status(400).json({ error: '"from" must be a non-empty string' });
|
|
}
|
|
if (typeof to !== 'string' || !to) {
|
|
return res.status(400).json({ error: '"to" must be a non-empty string' });
|
|
}
|
|
if (!isValidRelativeStackPath(from)) {
|
|
return res.status(400).json({ error: 'Invalid source path', code: 'INVALID_PATH' });
|
|
}
|
|
if (!isValidRelativeStackPath(to)) {
|
|
return res.status(400).json({ error: 'Invalid destination path', code: 'INVALID_PATH' });
|
|
}
|
|
const root = await resolveRootForOp(req, res, stackName, 'write');
|
|
if (!root) return;
|
|
const startedAt = Date.now();
|
|
logFileDiag('copy start', { stackName, from, to, nodeId: req.nodeId, rootKind: root.kind });
|
|
try {
|
|
await FileRootGateway.getInstance(req.nodeId).copy(root, stackName, from, to);
|
|
afterStackMutation(req, stackName);
|
|
logFileOperation('info', 'mutate', {
|
|
nodeId: req.nodeId,
|
|
op: 'copy',
|
|
stack: stackName,
|
|
path: from,
|
|
toPath: to,
|
|
rootKind: root.kind,
|
|
backend: root.backend,
|
|
});
|
|
logFileDiag('copy timing', { stackName, from, to, nodeId: req.nodeId, elapsedMs: Date.now() - startedAt });
|
|
recordFileOp(req.nodeId, 'copy', startedAt, true);
|
|
return res.status(204).send();
|
|
} catch (err: unknown) {
|
|
logFileOperation('warn', 'copy failed', {
|
|
nodeId: req.nodeId,
|
|
op: 'copy',
|
|
stack: stackName,
|
|
path: from,
|
|
toPath: to,
|
|
errorCode: fsErrorCode(err),
|
|
});
|
|
recordFileOp(req.nodeId, 'copy', startedAt, false);
|
|
return sendFsError(res, err, 'Failed to copy');
|
|
}
|
|
});
|
|
|
|
// ── Bulk file operations (delete / move / download) ─────────────────────────
|
|
|
|
const MAX_BULK = 100; // selected paths accepted per bulk request
|
|
const MAX_ARCHIVE_ENTRIES = 5000; // files packed into one bulk-download archive
|
|
const MAX_ARCHIVE_BYTES = 1024 * 1024 * 1024; // 1 GiB uncompressed cap
|
|
|
|
/**
|
|
* Helper-backed named volumes are Linux containers (case-sensitive). Filesystem
|
|
* roots follow the host: Windows/macOS fold case, Linux does not.
|
|
*/
|
|
function rootCaseSensitive(root: StackFileRoot): boolean {
|
|
if (root.backend === 'helper') return true;
|
|
return process.platform !== 'win32' && process.platform !== 'darwin';
|
|
}
|
|
|
|
/** Validate a bulk path array; sends the 400 and returns null on any problem. */
|
|
function parseBulkPaths(value: unknown, res: Response): string[] | null {
|
|
if (!Array.isArray(value) || value.length === 0) {
|
|
res.status(400).json({ error: 'A non-empty list of paths is required' });
|
|
return null;
|
|
}
|
|
if (value.length > MAX_BULK) {
|
|
res.status(400).json({ error: `Select at most ${MAX_BULK} items at once`, code: 'TOO_MANY' });
|
|
return null;
|
|
}
|
|
const out: string[] = [];
|
|
for (const p of value) {
|
|
if (typeof p !== 'string' || p === '' || !isValidRelativeStackPath(p)) {
|
|
res.status(400).json({ error: 'Invalid path in selection', code: 'INVALID_PATH' satisfies FsErrorCode });
|
|
return null;
|
|
}
|
|
out.push(p);
|
|
}
|
|
return out;
|
|
}
|
|
|
|
/** A clean per-item failure message for a bulk result, mapping the opaque
|
|
* filesystem codes that carry no friendly message of their own. */
|
|
function bulkItemError(err: unknown): string {
|
|
const e = err as Error & { code?: string };
|
|
switch (e.code) {
|
|
case 'EXDEV': return 'Cannot move across a storage boundary';
|
|
case 'EEXIST': return 'A file or folder with that name already exists';
|
|
case 'ENOENT': return 'No longer exists';
|
|
case 'EISDIR': case 'ENOTDIR': return 'Path type changed';
|
|
default: return e.message || e.code || 'Operation failed';
|
|
}
|
|
}
|
|
|
|
function archiveTooLargeError(message: string): Error & { code: string } {
|
|
return Object.assign(new Error(message), { code: 'ARCHIVE_TOO_LARGE' });
|
|
}
|
|
|
|
/**
|
|
* Walk the normalized selection and return every file to pack, enforcing the
|
|
* entry and byte caps. Throws ARCHIVE_TOO_LARGE if either cap is exceeded (or an
|
|
* fs directory listing is truncated), so the caller can 413 before any archive
|
|
* byte is streamed. File sizes come from listDir; every directory is stat-ed once
|
|
* (the walk recurses into each), but individual files in a listing are not re-stat-ed.
|
|
*/
|
|
async function enumerateArchiveFiles(
|
|
gateway: FileRootGateway,
|
|
root: StackFileRoot,
|
|
stackName: string,
|
|
selection: string[],
|
|
): Promise<string[]> {
|
|
const files: string[] = [];
|
|
let totalBytes = 0;
|
|
const addFile = (relPath: string, size: number): void => {
|
|
files.push(relPath);
|
|
totalBytes += size;
|
|
if (files.length > MAX_ARCHIVE_ENTRIES) throw archiveTooLargeError('The selection has too many files to download');
|
|
if (totalBytes > MAX_ARCHIVE_BYTES) throw archiveTooLargeError('The selection is too large to download');
|
|
};
|
|
const visit = async (relPath: string): Promise<void> => {
|
|
const st = await gateway.stat(root, stackName, relPath);
|
|
if (st.type !== 'directory') {
|
|
gateway.assertArchivable(root, relPath, st);
|
|
addFile(relPath, st.size);
|
|
return;
|
|
}
|
|
// Request one over the remaining budget so a directory that would push us one
|
|
// entry past the cap is detected: the overflow entry reaches addFile (which
|
|
// throws on >), or for larger directories the fs listing reports truncated.
|
|
const remaining = MAX_ARCHIVE_ENTRIES - files.length + 1;
|
|
const { entries, truncated } = await gateway.listDir(root, stackName, relPath, remaining);
|
|
if (truncated) throw archiveTooLargeError('A selected folder has too many files to download');
|
|
for (const entry of entries) {
|
|
const childRel = `${relPath}/${entry.name}`;
|
|
if (entry.type === 'directory') await visit(childRel);
|
|
else {
|
|
gateway.assertArchivable(root, childRel, entry);
|
|
addFile(childRel, entry.size);
|
|
}
|
|
}
|
|
};
|
|
for (const p of selection) await visit(p);
|
|
return files;
|
|
}
|
|
|
|
stacksRouter.post('/:stackName/files/bulk-delete', async (req: Request, res: Response) => {
|
|
const stackName = req.params.stackName as string;
|
|
if (!requirePermission(req, res, 'stack:edit', 'stack', stackName)) return;
|
|
const parsed = parseBulkPaths((req.body as { paths?: unknown }).paths, res);
|
|
if (!parsed) return;
|
|
const root = await resolveRootForOp(req, res, stackName, 'write');
|
|
if (!root) return;
|
|
const normalized = normalizeBulkPaths(parsed, rootCaseSensitive(root));
|
|
const gateway = FileRootGateway.getInstance(req.nodeId);
|
|
const deleted: string[] = [];
|
|
const failed: { path: string; error: string }[] = [];
|
|
for (const relPath of normalized) {
|
|
const startedAt = Date.now();
|
|
try {
|
|
await gateway.deletePath(root, stackName, relPath, true);
|
|
deleted.push(relPath);
|
|
recordFileOp(req.nodeId, 'delete', startedAt, true);
|
|
} catch (err: unknown) {
|
|
failed.push({ path: relPath, error: bulkItemError(err) });
|
|
recordFileOp(req.nodeId, 'delete', startedAt, false);
|
|
}
|
|
}
|
|
// Partial-success: invalidate the roots cache if anything actually changed.
|
|
if (deleted.length > 0) afterStackMutation(req, stackName);
|
|
logFileOperation('info', 'mutate', { nodeId: req.nodeId, op: 'bulkDelete', stack: stackName, deleted: deleted.length, failed: failed.length, rootKind: root.kind, backend: root.backend });
|
|
return res.json({ deleted, failed });
|
|
});
|
|
|
|
stacksRouter.post('/:stackName/files/bulk-move', async (req: Request, res: Response) => {
|
|
const stackName = req.params.stackName as string;
|
|
if (!requirePermission(req, res, 'stack:edit', 'stack', stackName)) return;
|
|
const body = req.body as { from?: unknown; toDir?: unknown };
|
|
const parsed = parseBulkPaths(body.from, res);
|
|
if (!parsed) return;
|
|
if (typeof body.toDir !== 'string') {
|
|
return res.status(400).json({ error: '"toDir" must be a string (use "" for the root)' });
|
|
}
|
|
const toDir = body.toDir;
|
|
if (toDir !== '' && !isValidRelativeStackPath(toDir)) {
|
|
return res.status(400).json({ error: 'Invalid destination', code: 'INVALID_PATH' satisfies FsErrorCode });
|
|
}
|
|
const root = await resolveRootForOp(req, res, stackName, 'write');
|
|
if (!root) return;
|
|
const caseSensitive = rootCaseSensitive(root);
|
|
const normalized = normalizeBulkPaths(parsed, caseSensitive);
|
|
// Reject the whole request if the destination is one of the moved folders or
|
|
// sits inside one (which would move a folder into its own subtree).
|
|
if (destWithinAnySource(toDir, normalized, caseSensitive)) {
|
|
return res.status(400).json({ error: 'Cannot move the selection into itself', code: 'INVALID_PATH' satisfies FsErrorCode });
|
|
}
|
|
const gateway = FileRootGateway.getInstance(req.nodeId);
|
|
const moved: string[] = [];
|
|
const failed: { path: string; error: string }[] = [];
|
|
for (const fromRel of normalized) {
|
|
const startedAt = Date.now();
|
|
const name = fromRel.split('/').pop() as string;
|
|
const toRel = toDir ? `${toDir}/${name}` : name;
|
|
try {
|
|
await gateway.rename(root, stackName, fromRel, toRel);
|
|
moved.push(fromRel);
|
|
recordFileOp(req.nodeId, 'rename', startedAt, true);
|
|
} catch (err: unknown) {
|
|
failed.push({ path: fromRel, error: bulkItemError(err) });
|
|
recordFileOp(req.nodeId, 'rename', startedAt, false);
|
|
}
|
|
}
|
|
if (moved.length > 0) afterStackMutation(req, stackName);
|
|
logFileOperation('info', 'mutate', { nodeId: req.nodeId, op: 'bulkMove', stack: stackName, moved: moved.length, failed: failed.length, rootKind: root.kind, backend: root.backend });
|
|
return res.json({ moved, failed });
|
|
});
|
|
|
|
stacksRouter.get('/:stackName/files/bulk-download', async (req: Request, res: Response) => {
|
|
const stackName = req.params.stackName as string;
|
|
if (!requirePermission(req, res, 'stack:read', 'stack', stackName)) return;
|
|
const raw = req.query.path;
|
|
const list = Array.isArray(raw) ? raw : raw !== undefined ? [raw] : [];
|
|
const parsed = parseBulkPaths(list, res);
|
|
if (!parsed) return;
|
|
const root = await resolveRootForOp(req, res, stackName, 'read');
|
|
if (!root) return;
|
|
const gateway = FileRootGateway.getInstance(req.nodeId);
|
|
const normalized = normalizeBulkPaths(parsed, rootCaseSensitive(root));
|
|
const startedAt = Date.now();
|
|
|
|
// Prewalk + cap enforcement BEFORE any response header is sent, so a too-large
|
|
// selection fails as a clean 413 rather than a truncated archive.
|
|
let files: string[];
|
|
try {
|
|
files = await enumerateArchiveFiles(gateway, root, stackName, normalized);
|
|
} catch (err: unknown) {
|
|
recordFileOp(req.nodeId, 'download', startedAt, false);
|
|
const code = (err as { code?: string }).code;
|
|
if (code === 'ARCHIVE_TOO_LARGE') {
|
|
return res.status(413).json({ error: (err as Error).message, code: 'TOO_LARGE' });
|
|
}
|
|
if (code === 'ARCHIVE_UNSUPPORTED') {
|
|
return res.status(400).json({ error: (err as Error).message, code: 'UNSUPPORTED' });
|
|
}
|
|
return sendFsError(res, err, 'Failed to prepare download');
|
|
}
|
|
if (files.length === 0) {
|
|
recordFileOp(req.nodeId, 'download', startedAt, false);
|
|
return res.status(404).json({ error: 'Nothing to download', code: 'NOT_FOUND' satisfies FsErrorCode });
|
|
}
|
|
|
|
res.setHeader('Content-Type', 'application/gzip');
|
|
res.setHeader('Content-Disposition', `attachment; filename="${stackName}-files.tar.gz"`);
|
|
const pack = tar.pack();
|
|
const gzip = zlib.createGzip();
|
|
const onStreamError = (err: Error): void => {
|
|
logFileOperation('warn', 'bulk download stream error', { nodeId: req.nodeId, stack: stackName, errorCode: fsErrorCode(err) });
|
|
if (!res.writableEnded) res.destroy();
|
|
};
|
|
pack.on('error', onStreamError);
|
|
gzip.on('error', onStreamError);
|
|
// If the client aborts mid-download, stop fetching the remaining files (each
|
|
// helper-volume read is a container exec) instead of streaming into a dead pipe.
|
|
let aborted = false;
|
|
res.on('close', () => {
|
|
if (!res.writableEnded) {
|
|
aborted = true;
|
|
pack.destroy();
|
|
}
|
|
});
|
|
pack.pipe(gzip).pipe(res);
|
|
|
|
try {
|
|
for (const relPath of files) {
|
|
if (aborted) break;
|
|
const dl = await gateway.download(root, stackName, relPath);
|
|
if (dl.kind === 'buffer') {
|
|
await new Promise<void>((resolve, reject) => {
|
|
pack.entry({ name: relPath }, dl.buffer, (err) => (err ? reject(err) : resolve()));
|
|
});
|
|
} else {
|
|
await new Promise<void>((resolve, reject) => {
|
|
const entry = pack.entry({ name: relPath, size: dl.size }, (err) => (err ? reject(err) : resolve()));
|
|
dl.stream.on('error', reject);
|
|
entry.on('error', reject);
|
|
dl.stream.pipe(entry);
|
|
});
|
|
}
|
|
}
|
|
pack.finalize();
|
|
recordFileOp(req.nodeId, 'download', startedAt, true);
|
|
} catch (err: unknown) {
|
|
// Headers are already sent, so surface the failure by tearing the stream
|
|
// down rather than trying to change the status.
|
|
logFileOperation('warn', 'bulk download failed mid-stream', { nodeId: req.nodeId, stack: stackName, errorCode: fsErrorCode(err) });
|
|
recordFileOp(req.nodeId, 'download', startedAt, false);
|
|
pack.destroy();
|
|
if (!res.writableEnded) res.destroy();
|
|
}
|
|
});
|
|
|
|
stacksRouter.get('/:stackName/files/permissions', async (req: Request, res: Response) => {
|
|
const stackName = req.params.stackName as string;
|
|
if (!requirePermission(req, res, 'stack:read', 'stack', stackName)) return;
|
|
const relPath = getRelPath(req);
|
|
if (!relPath) return res.status(400).json({ error: 'path query parameter is required', code: 'INVALID_PATH' });
|
|
if (!isValidRelativeStackPath(relPath)) {
|
|
return res.status(400).json({ error: 'Invalid path', code: 'INVALID_PATH' });
|
|
}
|
|
const root = await resolveRootForOp(req, res, stackName, 'read');
|
|
if (!root) return;
|
|
const startedAt = Date.now();
|
|
logFileDiag('permissions read start', { stackName, relPath, nodeId: req.nodeId, rootKind: root.kind });
|
|
try {
|
|
const result = await FileRootGateway.getInstance(req.nodeId).getMode(root, stackName, relPath);
|
|
logFileDiag('permissions read complete', { stackName, relPath, nodeId: req.nodeId, mode: result.octal, elapsedMs: Date.now() - startedAt });
|
|
recordFileOp(req.nodeId, 'permissionsRead', startedAt, true);
|
|
return res.json(result);
|
|
} catch (err: unknown) {
|
|
logFileOperation('warn', 'permissions read failed', { nodeId: req.nodeId, errorCode: fsErrorCode(err) });
|
|
recordFileOp(req.nodeId, 'permissionsRead', startedAt, false);
|
|
return sendFsError(res, err, 'Failed to read permissions');
|
|
}
|
|
});
|
|
|
|
stacksRouter.put('/:stackName/files/permissions', async (req: Request, res: Response) => {
|
|
const stackName = req.params.stackName as string;
|
|
if (!requirePermission(req, res, 'stack:edit', 'stack', stackName)) return;
|
|
const relPath = getRelPath(req);
|
|
if (!relPath) return res.status(400).json({ error: 'path query parameter is required', code: 'INVALID_PATH' });
|
|
if (!isValidRelativeStackPath(relPath)) {
|
|
return res.status(400).json({ error: 'Invalid path', code: 'INVALID_PATH' });
|
|
}
|
|
const { mode } = req.body as { mode?: unknown };
|
|
if (typeof mode !== 'number') {
|
|
return res.status(400).json({ error: '"mode" must be a number' });
|
|
}
|
|
const root = await resolveRootForOp(req, res, stackName, 'write');
|
|
if (!root) return;
|
|
if (rejectIfStackOpBlocksRootTrustFileWrite(req, res, stackName, relPath, root)) return;
|
|
const startedAt = Date.now();
|
|
logFileDiag('chmod start', { stackName, relPath, nodeId: req.nodeId, mode, rootKind: root.kind });
|
|
try {
|
|
await FileRootGateway.getInstance(req.nodeId).chmod(root, stackName, relPath, mode);
|
|
afterStackMutation(req, stackName);
|
|
logFileOperation('info', 'mutate', {
|
|
nodeId: req.nodeId,
|
|
op: 'chmod',
|
|
stack: stackName,
|
|
path: relPath,
|
|
mode: mode.toString(8).padStart(3, '0'),
|
|
rootKind: root.kind,
|
|
backend: root.backend,
|
|
});
|
|
logFileDiag('chmod timing', { stackName, relPath, nodeId: req.nodeId, elapsedMs: Date.now() - startedAt });
|
|
recordFileOp(req.nodeId, 'chmod', startedAt, true);
|
|
return res.status(204).send();
|
|
} catch (err: unknown) {
|
|
logFileOperation('warn', 'chmod failed', {
|
|
nodeId: req.nodeId,
|
|
op: 'chmod',
|
|
stack: stackName,
|
|
path: relPath,
|
|
mode: mode.toString(8).padStart(3, '0'),
|
|
errorCode: fsErrorCode(err),
|
|
});
|
|
recordFileOp(req.nodeId, 'chmod', startedAt, false);
|
|
return sendFsError(res, err, 'Failed to set permissions');
|
|
}
|
|
});
|
|
|