fix(fleet): resolve self-update compose file access and improve completion detection (#441)

The self-update feature failed on remote nodes because SelfUpdateService
ran `docker compose -f <host_path>` inside the container, where the host
compose file path does not exist. The fix splits the update into two
steps: (1) pull the latest image directly via `docker pull`, and (2)
spawn a short-lived helper container that mounts the compose directory
from the host and runs `docker compose up --force-recreate`.

Additional changes:
- Use execFileSync/execFile with argument arrays instead of shell strings
  to eliminate shell injection surface from Docker label values
- Add Signal 4 completion detection: mark update as completed when the
  remote version matches the gateway version (with 15s elapsed guard)
- Extend early failure heuristic from 90s to 3 minutes for slow pulls
- Distinguish "node unreachable" from "node lacks self-update capability"
  in error messages; use silent skip in update-all to avoid res crashes
- Add requireAdmin guard to POST /api/system/update
- Handle comma-separated compose config file paths (multiple -f flags)
- Update fleet docs with self-update mechanism, troubleshooting entries
This commit is contained in:
Anso
2026-04-08 14:59:11 -04:00
committed by GitHub
parent 24011aea9e
commit 6fff2c2d35
7 changed files with 93 additions and 19 deletions
+7
View File
@@ -26,6 +26,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
* **dashboard:** show remote node badge on Recent Alerts card, matching the notification panel style, so users can identify which node generated each alert.
### Fixed
* **fleet:** resolve self-update "compose file not found" failure by using a helper container that mounts the compose directory from the host, eliminating the dependency on the compose file being accessible inside the main container.
* **fleet:** improve update completion detection with a new "version current" signal (fires when the remote reaches the gateway version) and extend the early failure heuristic from 90 seconds to 3 minutes for slower connections.
* **fleet:** distinguish between "node unreachable" and "node does not support self-update" error messages when triggering remote updates.
* **fleet:** add admin role requirement to the `/api/system/update` endpoint, preventing non-admin users from triggering self-updates.
## [0.40.0](https://github.com/AnsoCode/Sencho/compare/v0.39.6...v0.40.0) (2026-04-07)
+22 -4
View File
@@ -1095,7 +1095,8 @@ function scheduleLocalUpdate(res: Response, message: string): void {
});
}
app.post('/api/system/update', (_req: Request, res: Response): void => {
app.post('/api/system/update', (req: Request, res: Response): void => {
if (!requireAdmin(req, res)) return;
if (!SelfUpdateService.getInstance().isAvailable()) {
res.status(503).json({ error: 'Self-update unavailable. Sencho must be deployed via Docker Compose.' });
return;
@@ -1119,7 +1120,7 @@ interface UpdateTracker {
const updateTracker = new Map<number, UpdateTracker>();
const UPDATE_TIMEOUT_MS = 5 * 60 * 1000; // 5 minutes
const UPDATE_TIMEOUT_MSG = 'Node did not come back online within 5 minutes.';
const EARLY_FAIL_MS = 90 * 1000; // 90 seconds before declaring a probable pull failure
const EARLY_FAIL_MS = 180 * 1000; // 3 minutes before declaring a probable pull failure
function createTracker(
status: UpdateTracker['status'],
@@ -1323,8 +1324,18 @@ app.get('/api/fleet/update-status', async (_req: Request, res: Response): Promis
} else if (tracker.wasOffline && remoteOnline) {
// Signal 3: Node went offline and is back online (container was recreated)
updateTracker.set(node.id, { ...tracker, status: 'completed' });
} else if (
elapsed > 15_000 &&
isValidVersion(version) &&
gatewayValid &&
!semver.lt(version, gatewayVersion!)
) {
// Signal 4: Remote is now at or above gateway version (after minimum processing time).
// Catches fast restarts where the 5s polling interval misses the offline window
// and startedAt hasn't been observed to change yet.
updateTracker.set(node.id, { ...tracker, status: 'completed' });
} else if (elapsed > EARLY_FAIL_MS) {
// Heuristic: node never went offline and nothing changed after 90s
// Heuristic: node never went offline and nothing changed after 3 min
updateTracker.set(node.id, {
...tracker,
status: 'failed',
@@ -1430,8 +1441,12 @@ app.post('/api/fleet/nodes/:nodeId/update', async (req: Request, res: Response):
return;
}
// Check remote capabilities
// Check remote availability and capabilities
const meta = await fetchRemoteMeta(node.api_url, node.api_token);
if (!meta.online) {
res.status(503).json({ error: 'Remote node is unreachable. Verify the node is running and the API URL is correct.' });
return;
}
if (!meta.capabilities.includes('self-update')) {
res.status(503).json({ error: 'Remote node does not support self-update. It may need to be updated manually first.' });
return;
@@ -1491,6 +1506,9 @@ app.post('/api/fleet/update-all', async (req: Request, res: Response): Promise<v
const results = await Promise.allSettled(candidates.map(async (node) => {
const meta = await fetchRemoteMeta(node.api_url!, node.api_token!);
if (!meta.online) {
return { name: node.name, triggered: false };
}
if (!meta.capabilities.includes('self-update')) {
return { name: node.name, triggered: false };
}
+38 -14
View File
@@ -1,4 +1,4 @@
import { execSync, exec } from 'child_process';
import { execFileSync, execFile } from 'child_process';
import DockerController from './DockerController';
import { disableCapability } from './CapabilityRegistry';
@@ -6,6 +6,7 @@ interface ComposeContext {
workingDir: string;
configFiles: string;
serviceName: string;
imageName: string;
}
class SelfUpdateService {
@@ -47,16 +48,24 @@ class SelfUpdateService {
// Verify docker compose CLI is available inside the container
try {
execSync('docker compose version', { shell: '/bin/sh', stdio: 'pipe', timeout: 5000 });
execFileSync('docker', ['compose', 'version'], { stdio: 'pipe', timeout: 5000 });
} catch {
console.log('[SelfUpdate] docker compose CLI not available in container');
disableCapability('self-update');
return;
}
this.composeContext = { workingDir, configFiles, serviceName };
// Read the container's own image name for direct docker pull
const imageName = info.Config?.Image;
if (!imageName) {
console.log('[SelfUpdate] Could not determine container image name');
disableCapability('self-update');
return;
}
this.composeContext = { workingDir, configFiles, serviceName, imageName };
this.canSelfUpdate = true;
console.log(`[SelfUpdate] Ready - service="${serviceName}" in ${workingDir}`);
console.log(`[SelfUpdate] Ready - service="${serviceName}" image="${imageName}" in ${workingDir}`);
} catch (error) {
console.log('[SelfUpdate] Could not inspect own container - self-update unavailable:', (error as Error).message);
disableCapability('self-update');
@@ -79,15 +88,15 @@ class SelfUpdateService {
triggerUpdate(): void {
if (!this.composeContext) return;
const { configFiles, serviceName } = this.composeContext;
const { workingDir, configFiles, serviceName, imageName } = this.composeContext;
const env = { ...process.env, PATH: process.env.PATH || '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin' };
this.lastUpdateError = null;
console.log(`[SelfUpdate] Pulling latest image for ${serviceName}...`);
// Step 1: Pull latest image directly (no compose file needed)
console.log(`[SelfUpdate] Pulling latest image: ${imageName}...`);
try {
execSync(`docker compose -f ${configFiles} pull ${serviceName}`, {
execFileSync('docker', ['pull', imageName], {
env,
shell: '/bin/sh',
stdio: 'pipe',
timeout: 300_000, // 5 min max for pull
});
@@ -98,12 +107,27 @@ class SelfUpdateService {
return;
}
console.log(`[SelfUpdate] Recreating container for ${serviceName}... (last breath)`);
exec(`docker compose -f ${configFiles} up -d --force-recreate ${serviceName}`, {
env,
shell: '/bin/sh',
});
// Process will be killed by Docker during recreate, no code runs after this
// Step 2: Spawn a helper container to run docker compose recreate.
// The main container cannot access the compose file because the host path
// from Docker labels does not exist inside this container. The helper
// explicitly mounts the compose working directory from the host, so the
// compose file is accessible at the original path.
console.log(`[SelfUpdate] Spawning updater container... (last breath)`);
const fFlags = configFiles.split(',').flatMap(f => ['-f', f.trim()]);
const composeCmd = ['sleep 3 && docker compose', ...fFlags, 'up -d --force-recreate', serviceName].join(' ');
const args = [
'run', '--rm', '-d',
'--user', 'root',
'--entrypoint', 'sh',
'-v', '/var/run/docker.sock:/var/run/docker.sock',
'-v', `${workingDir}:${workingDir}:ro`,
'-w', workingDir,
imageName,
'-c', composeCmd,
];
execFile('docker', args, { env });
// Process will be killed by Docker during recreate; no code runs after this
}
}
+26 -1
View File
@@ -145,7 +145,11 @@ The modal shows:
- **Recheck** button to re-scan for available updates
- **Update All** button to trigger updates on all remote nodes that have a pending update
When you click **Update** on a remote node, Sencho sends the update command and the node restarts with the new version. If you update the local node, a reconnection overlay appears while your primary instance restarts.
When you click **Update** on a remote node, Sencho sends the update command to the remote instance. The remote pulls the latest Docker image, then spawns a short-lived helper container that performs the compose recreate. The node restarts with the new version, and the status badge transitions from "Updating" to "Updated" once the gateway detects the version change.
If you update the local node, a confirmation dialog appears first, then a reconnection overlay shows while your primary instance restarts.
**How self-update works:** Each Sencho instance reads its own Docker Compose labels to determine the image name, compose file path, and service name. It pulls the latest image directly, then spawns a helper container that mounts the compose directory from the host and runs `docker compose up --force-recreate`. This approach works regardless of the container's own volume mounts.
---
@@ -178,3 +182,24 @@ The **Update All** button only triggers updates on remote nodes that:
2. Support the self-update capability (requires running in Docker)
If a remote node's version is unresolvable ("unknown"), the **Update** button on its individual card will still be available, but **Update All** requires both versions to be known for a safe comparison.
### Update fails with "no such file or directory"
If a remote node update fails with an error mentioning a compose file path (e.g. "open /path/to/compose.yaml: no such file or directory"), the remote node is running an older Sencho version (prior to v0.42.0) that attempted to access the compose file directly inside the container, where the host path does not exist.
**Resolution:** Update the remote node manually once by SSH-ing into the host and running:
```bash
docker compose pull && docker compose up -d
```
After this one-time manual update, the node will have the fixed self-update mechanism and all future updates can be triggered from Fleet Overview.
### Update shows "Remote node is unreachable"
This means the gateway could not connect to the remote node's `/api/meta` endpoint. Check that:
- The remote node is powered on and running
- The API URL configured for this node is correct and reachable from the gateway
- The network allows traffic between the gateway and remote node on the configured port
- The remote node's Sencho container is healthy (`docker ps` should show it as running)
Binary file not shown.

After

Width:  |  Height:  |  Size: 102 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 100 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 34 KiB