gsadmin/sencho
1
0
Fork 0
mirror of https://github.com/Studio-Saelix/sencho.git synced 2026-08-21 15:46:43 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix: distinguish failed image-update checks from "up to date" (#1470)

Browse Source 
* fix: distinguish failed image-update checks from "up to date"

The image-update detector collapsed every failure (registry unreachable,
missing auth, rate limit, unresolved local digest) into hasUpdate:false and
dropped the captured reason, so a failed check was indistinguishable from a
current image and never raised a notification, even while a manual stack
update still pulled a newer image.

Detection now records a tri-state per stack (ok / partial / failed) with the
failure reason, exposed via a new GET /api/image-updates/detail (the boolean
GET / is unchanged so fleet aggregation is unaffected). A fully-failed check
preserves the last known has_update, so a transient outage neither erases a
real update nor flaps the notification state. The sidebar shows a muted
"couldn't check" indicator with the reason on hover, and the Update board
lists stacks whose check failed in a "could not be checked" advisory.

Detector hardening: the manifest digest lookup issues HEAD first (falling back
to GET) so it no longer draws down Docker Hub's anonymous pull-rate budget, and
local RepoDigest matching is normalized so official library/* images resolve
their digest instead of falling through to a silent "no update".

* fix: preserve confirmed updates through partial checks; tighten failure surfacing

Address review findings on the tri-state image-update detection:

- A partial check (some images errored) no longer erases a previously
  confirmed update; only a fully-ok check can lower has_update, so a single
  image's registry blip cannot drop the stack's update and re-fire the
  notification on recovery. Adds a regression test.
- The image-level catch stores getErrorMessage(e) rather than raw String(e),
  since that value surfaces verbatim in the sidebar tooltip and readiness
  advisory.
- useImageUpdates and the readiness detail fetch now log unexpected non-ok
  responses instead of silently leaving stale state.
- Remove an unused checkFailedCount derivation (the row indicator is driven by
  the checkStatus prop).
- Reword the recordStackCheckFailure docstring and the HEAD-first comment.
This commit is contained in:
Anso
2026-06-26 16:16:34 -04:00
committed by GitHub
parent e3b3c3b857
commit d9b7911f12
19 changed files with 820 additions and 38 deletions
Download Patch File Download Diff File Expand all files Collapse all files
backend/src/services/DatabaseService.ts
+82 -5
View File
@@ -24,6 +24,21 @@ export interface GlobalSetting {
value: string;
}
/**
* Per-stack image-update check outcome. 'ok' = every checkable image was
* reached; 'partial' = some checkable images errored; 'failed' = no checkable
* image could be reached (status undeterminable). Distinguishes a failed check
* from a confirmed "up to date".
*/
export type StackCheckStatus = 'ok' | 'partial' | 'failed';
export interface StackUpdateDetail {
hasUpdate: boolean;
checkStatus: StackCheckStatus;
lastError: string | null;
checkedAt: number;
}
export interface StackAlert {
id?: number;
stack_name: string;
@@ -920,6 +935,8 @@ export class DatabaseService {
node_id INTEGER NOT NULL DEFAULT 0,
stack_name TEXT NOT NULL,
has_update INTEGER DEFAULT 0,
check_status TEXT NOT NULL DEFAULT 'ok',
last_error TEXT,
checked_at INTEGER NOT NULL,
PRIMARY KEY (node_id, stack_name)
);
@@ -1548,6 +1565,15 @@ export class DatabaseService {
`);
}
// Tri-state image-update check outcome. Must run AFTER the composite-PK
// recreate above (that block recreates the table from the original four
// columns, so columns added earlier would be dropped). 'ok' = every
// checkable image was reached; the detector records 'failed'/'partial'
// plus a reason when registry checks could not determine status, so a
// failed check is no longer indistinguishable from "up to date".
maybeAddCol('stack_update_status', 'check_status', "TEXT NOT NULL DEFAULT 'ok'");
maybeAddCol('stack_update_status', 'last_error', 'TEXT');
// Drop legacy SSH/TLS columns from pre-0.7 databases (no longer read or written)
const legacyCols = ['host', 'port', 'ssh_port', 'ssh_user', 'ssh_password', 'ssh_key', 'tls_ca', 'tls_cert', 'tls_key'];
for (const col of legacyCols) {
@@ -3161,12 +3187,42 @@ export class DatabaseService {
// --- Stack Update Status ---
public upsertStackUpdateStatus(nodeId: number, stackName: string, hasUpdate: boolean, checkedAt: number): void {
public upsertStackUpdateStatus(
nodeId: number,
stackName: string,
hasUpdate: boolean,
checkedAt: number,
checkStatus: StackCheckStatus = 'ok',
lastError: string | null = null,
): void {
this.db.prepare(
`INSERT INTO stack_update_status (node_id, stack_name, has_update, checked_at)
VALUES (?, ?, ?, ?)
ON CONFLICT(node_id, stack_name) DO UPDATE SET has_update = excluded.has_update, checked_at = excluded.checked_at`
).run(nodeId, stackName, hasUpdate ? 1 : 0, checkedAt);
`INSERT INTO stack_update_status (node_id, stack_name, has_update, check_status, last_error, checked_at)
VALUES (?, ?, ?, ?, ?, ?)
ON CONFLICT(node_id, stack_name) DO UPDATE SET
has_update = excluded.has_update,
check_status = excluded.check_status,
last_error = excluded.last_error,
checked_at = excluded.checked_at`
).run(nodeId, stackName, hasUpdate ? 1 : 0, checkStatus, lastError, checkedAt);
}
/**
* Record a fully-failed check (no checkable image could be reached) without
* touching has_update, so a transient registry outage cannot erase a real
* update or flap the notification state. On an existing row it updates only
* check_status / last_error / checked_at, leaving has_update intact; a
* first-ever failed check inserts a row with has_update = 0 so the stack
* still appears with its failure reason.
*/
public recordStackCheckFailure(nodeId: number, stackName: string, lastError: string, checkedAt: number): void {
this.db.prepare(
`INSERT INTO stack_update_status (node_id, stack_name, has_update, check_status, last_error, checked_at)
VALUES (?, ?, 0, 'failed', ?, ?)
ON CONFLICT(node_id, stack_name) DO UPDATE SET
check_status = 'failed',
last_error = excluded.last_error,
checked_at = excluded.checked_at`
).run(nodeId, stackName, lastError, checkedAt);
}
public getStackUpdateStatus(nodeId?: number): Record<string, boolean> {
@@ -3180,6 +3236,27 @@ export class DatabaseService {
return result;
}
/**
* Rich per-stack update status (hasUpdate + check outcome + reason) for the
* sidebar/readiness UI. GET /api/image-updates stays the boolean map so the
* cross-version fleet aggregation contract is unaffected.
*/
public getStackUpdateDetail(nodeId: number): Record<string, StackUpdateDetail> {
const rows = this.db.prepare(
'SELECT stack_name, has_update, check_status, last_error, checked_at FROM stack_update_status WHERE node_id = ?'
).all(nodeId) as Array<{ stack_name: string; has_update: number; check_status: string | null; last_error: string | null; checked_at: number }>;
const result: Record<string, StackUpdateDetail> = {};
for (const row of rows) {
result[row.stack_name] = {
hasUpdate: row.has_update === 1,
checkStatus: (row.check_status === 'failed' || row.check_status === 'partial') ? row.check_status : 'ok',
lastError: row.last_error,
checkedAt: row.checked_at,
};
}
return result;
}
public clearStackUpdateStatus(nodeId: number, stackName: string): void {
this.db.prepare('DELETE FROM stack_update_status WHERE node_id = ? AND stack_name = ?').run(nodeId, stackName);
}
backend/src/services/ImageUpdateService.ts
+50 -7
View File
@@ -8,7 +8,7 @@ import { RegistryService } from './RegistryService';
import { NodeRegistry } from './NodeRegistry';
import { NotificationService } from './NotificationService';
import { sanitizeNotificationMessage } from '../utils/notificationMessage';
import { parseImageRef, getRemoteDigest } from './registry-api';
import { parseImageRef, getRemoteDigest, repoDigestMatchesRef } from './registry-api';
import { isDebugEnabled } from '../utils/debug';
import { getErrorMessage } from '../utils/errors';
import { sanitizeForLog } from '../utils/safeLog';
@@ -18,6 +18,13 @@ const BACKFILL_KEY = 'image_update_notifications_backfilled';
export interface ImageCheckResult {
hasUpdate: boolean;
error?: string;
/**
* The image is not registry-backed (locally built, or a bare digest ref
* with no resolvable tag), so update status is not applicable. Distinct
* from `error`: such an image must be excluded from a stack's pass/fail
* tally rather than counted as a failed or up-to-date check.
*/
notCheckable?: boolean;
}
/**
@@ -549,7 +556,9 @@ export class ImageUpdateService {
imageUpdateMap.set(imageRef, await this.checkImage(docker, imageRef));
} catch (e) {
console.error(`[ImageUpdateService] Error checking ${sanitizeForLog(imageRef)}:`, sanitizeForLog((e as Error)?.message ?? String(e)));
imageUpdateMap.set(imageRef, { hasUpdate: false, error: String(e) });
// getErrorMessage (not raw String(e)) because this value can surface
// verbatim in the sidebar tooltip / readiness advisory as lastError.
imageUpdateMap.set(imageRef, { hasUpdate: false, error: getErrorMessage(e, 'Update check failed') });
}
await sleep(ImageUpdateService.INTER_IMAGE_DELAY_MS);
}
@@ -566,7 +575,32 @@ export class ImageUpdateService {
let updatesFound = 0;
const newlyUpdated: string[] = [];
for (const [stackName, images] of stackImages) {
const hasUpdate = Array.from(images).some(img => imageUpdateMap.get(img)?.hasUpdate === true);
// Tally only checkable images: a not-checkable image (locally built,
// or a bare digest ref) is neither a pass nor a failure.
const checkable = Array.from(images)
.map(img => imageUpdateMap.get(img))
.filter((r): r is ImageCheckResult => !!r && !r.notCheckable);
const errored = checkable.filter(r => r.error !== undefined);
const confirmedHasUpdate = checkable.some(r => r.error === undefined && r.hasUpdate === true);
// Every checkable image failed: status is undeterminable. Preserve the
// last-known has_update so a transient registry outage neither erases a
// real update nor flaps the notification state.
if (checkable.length > 0 && errored.length === checkable.length) {
db.recordStackCheckFailure(nodeId, stackName, errored[0].error ?? 'Update check failed', now);
continue;
}
const checkStatus = errored.length > 0 ? 'partial' : 'ok';
const lastError = errored.length > 0 ? (errored[0].error ?? null) : null;
// Only a fully-ok check is authoritative enough to lower has_update to
// false. On a partial check some image could not be reached, so a
// previously confirmed update is preserved rather than erased (which
// would also re-fire the notification when that image recovers).
const hasUpdate = checkStatus === 'partial'
? (confirmedHasUpdate || previousState[stackName] === true)
: confirmedHasUpdate;
if (hasUpdate) {
updatesFound++;
// Notify only on state transition: was false/absent, now true
@@ -574,7 +608,7 @@ export class ImageUpdateService {
newlyUpdated.push(stackName);
}
}
db.upsertStackUpdateStatus(nodeId, stackName, hasUpdate, now);
db.upsertStackUpdateStatus(nodeId, stackName, hasUpdate, now, checkStatus, lastError);
}
// Dispatch notifications for stacks that newly have updates
@@ -629,7 +663,8 @@ export class ImageUpdateService {
public async checkImage(docker: DockerController, imageRef: string): Promise<ImageCheckResult> {
const parsed = parseImageRef(imageRef);
if (!parsed) return { hasUpdate: false };
// A bare digest ref (sha256:...) has no tag to track upstream; not applicable.
if (!parsed) return { hasUpdate: false, notCheckable: true };
if (isDebugEnabled()) {
console.log(`[ImageUpdateService] Checking ${imageRef}: registry=${parsed.registry} repo=${parsed.repo} tag=${parsed.tag}`);
@@ -647,11 +682,15 @@ export class ImageUpdateService {
const inspect = await withTimeout(docker.getDocker().getImage(imageRef).inspect(), ImageUpdateService.SOCKET_TIMEOUT_MS, 'inspect');
const repoDigests: string[] = inspect.RepoDigests ?? [];
// No RepoDigests at all: locally built / not registry-backed, so update
// status does not apply.
if (repoDigests.length === 0) return { hasUpdate: false, notCheckable: true };
for (const rd of repoDigests) {
if (!rd.includes('@sha256:')) continue;
const [, digest] = rd.split('@');
if (rd.includes(parsed.repo) || rd.includes(parsed.registry) || repoDigests.length === 1) {
if (repoDigestMatchesRef(rd, parsed) || repoDigests.length === 1) {
localDigest = digest;
break;
}
@@ -660,7 +699,11 @@ export class ImageUpdateService {
return { hasUpdate: false, error: `Failed to inspect local image "${imageRef}"` };
}
if (!localDigest) return { hasUpdate: false };
// RepoDigests were present but none resolved a usable digest: genuinely
// ambiguous, so surface it rather than silently call the image up to date.
if (!localDigest) {
return { hasUpdate: false, error: `Could not resolve a local registry digest for "${imageRef}"` };
}
const remoteDigest = await getRemoteDigest(parsed.registry, parsed.repo, parsed.tag, credentials);
if (!remoteDigest) {
backend/src/services/registry-api.ts
+57 -4
View File
@@ -50,8 +50,9 @@ export function parseImageRef(imageRef: string): ParsedRef | null {
return { registry, repo: rest, tag };
}
export function httpGet(
export function httpRequest(
url: string,
method: 'GET' | 'HEAD',
headers: Record<string, string> = {},
timeoutMs = 10000,
): Promise<HttpResult> {
@@ -63,7 +64,7 @@ export function httpGet(
settled = true;
fn();
};
const req = lib.get(url, { headers }, (res) => {
const req = lib.request(url, { method, headers }, (res) => {
let body = '';
res.on('data', (chunk: Buffer) => { body += chunk.toString(); });
res.on('end', () => finish(() => resolve({
@@ -79,9 +80,18 @@ export function httpGet(
req.destroy(err);
finish(() => reject(err));
});
req.end();
});
}
export function httpGet(
url: string,
headers: Record<string, string> = {},
timeoutMs = 10000,
): Promise<HttpResult> {
return httpRequest(url, 'GET', headers, timeoutMs);
}
export async function getAuthToken(
registry: string,
repo: string,
@@ -129,6 +139,31 @@ const MANIFEST_ACCEPT = [
'application/vnd.oci.image.manifest.v1+json',
].join(', ');
/** docker.io has three hostnames that all address the same registry. */
function canonicalRegistry(host: string): string {
if (host === 'docker.io' || host === 'index.docker.io' || host === 'registry-1.docker.io') {
return 'docker.io';
}
return host;
}
/**
* True when a local RepoDigest entry ("name@sha256:...") refers to the same
* registry + repository as the parsed image ref. Parses the name side through
* the same normalization as the image ref (Docker Hub's implicit `library/`
* namespace and default registry), replacing a fragile substring check that
* missed `library/*` official images: their RepoDigests read `nginx@sha256:...`,
* never `library/nginx@...`, so `name.includes('library/nginx')` was false.
*/
export function repoDigestMatchesRef(repoDigest: string, parsed: ParsedRef): boolean {
const at = repoDigest.indexOf('@');
if (at === -1) return false;
const parsedName = parseImageRef(repoDigest.slice(0, at));
if (!parsedName) return false;
return canonicalRegistry(parsedName.registry) === canonicalRegistry(parsed.registry)
&& parsedName.repo === parsed.repo;
}
export async function getRemoteDigest(
registry: string,
repo: string,
@@ -139,10 +174,28 @@ export async function getRemoteDigest(
const token = await getAuthToken(registry, repo, credentials);
const headers: Record<string, string> = { Accept: MANIFEST_ACCEPT };
if (token) headers['Authorization'] = `Bearer ${token}`;
const url = `https://${registry}/v2/${repo}/manifests/${tag}`;
const res = await httpGet(`https://${registry}/v2/${repo}/manifests/${tag}`, headers);
// HEAD first: the registry returns docker-content-digest without
// transferring the manifest body, so it does not draw down Docker Hub's
// anonymous pull-rate budget the way a GET does (a GET can self-inflict a
// 429). Fall back to GET only when the registry rejects HEAD (405/501) or
// omits the digest header on a 200. A 401/403/404/429/5xx HEAD returns
// null without a GET retry: the bearer token is fetched up-front, so a
// 401 here is a real auth failure, not a token-scope challenge to retry.
const head = await httpRequest(url, 'HEAD', headers);
if (head.statusCode === 200) {
const digest = head.headers['docker-content-digest'];
if (typeof digest === 'string') return digest;
} else if (head.statusCode !== 405 && head.statusCode !== 501) {
// 401/403/404/429/5xx: a GET would fail the same way. Report as unreachable.
return null;
}
const res = await httpRequest(url, 'GET', headers);
if (res.statusCode !== 200) return null;
return (res.headers['docker-content-digest'] as string) ?? null;
const digest = res.headers['docker-content-digest'];
return typeof digest === 'string' ? digest : null;
} catch {
return null;
}