test(pilot): in-process integration test and Playwright E2E (#983)

* test(pilot): in-process integration test for the tunnel handshake

Layered unit tests cover the protocol decoder, the bridge cap, the
manager, and the DB-layer enrollment lifecycle. None exercise the
glue: the WebSocket dispatch order in upgradeHandler.ts, the actual
hello / enroll_ack round-trip, the long-lived token swap. A
regression that moves /api/pilot/tunnel below the auth gate, breaks
the hello / enroll_ack ordering, or changes the token shape would not
be caught today.

Spin up a real http.Server with attachUpgrade wired and a real
ws.WebSocket client. Three tests:

  - Enroll-ack happy path: connect with an enrollment token, receive
    hello + ctrl enroll_ack, assert the manager has registered the
    tunnel.
  - Replay rejection on the wire: connect, complete enroll-ack, close,
    reconnect with the same enrollment token, assert HTTP 401 from the
    upgrade handshake.
  - Long-lived token reconnect: capture the enroll_ack token, close,
    reconnect with the long-lived pilot_tunnel JWT, assert hello but
    no enroll_ack and the tunnel is registered.

The test does not mock the agent side beyond the framing protocol.
The goal is to confirm the wires connect end-to-end, not to drive a
full request round-trip (which would require a synthetic agent bridge
and adds little marginal coverage over the existing bridge unit tests).

This file also establishes the in-process WS-pair test pattern for
any future WebSocket integration work; no such pattern existed in the
suite before.

* test(e2e): cover operator-side pilot-agent enrollment in Playwright

Backend integration is covered by pilot-tunnel-integration.test.ts
and the vitest enrollment suites. The browser-only surfaces (mode
selector default, enrollment dialog, docker run code block,
regenerate affordance on an existing pilot-mode node) had no
automated coverage; a typo in the mode label, a styling regression
that hid the docker-run code, or a backend response shape change
that broke the rendered command would all ship unchecked today.

Two specs reusing the existing loginAs helper and the same
Settings -> Nodes navigation pattern from nodes.spec.ts:

  - Create flow: open Add Node, switch type to Remote, confirm pilot
    mode is the default (api_url field stays absent), submit, assert
    the enrollment dialog renders a docker run command containing
    SENCHO_MODE=pilot, SENCHO_PRIMARY_URL=, and a JWT-shaped
    SENCHO_ENROLL_TOKEN=. Cleanup deletes the row.
  - Regenerate flow: create a pilot node, capture the first token,
    open the row's edit dialog, click Regenerate enrollment token,
    assert the second token differs from the first. Cleanup deletes
    the row.

Out of scope for this E2E: simulating an agent connecting to flip the
row to Online. The integration test covers the wire side; this spec
keeps focus on the operator-visible UI.

Each test name-suffixes with Date.now() so re-runs do not collide on
the UNIQUE name constraint and so leftover rows from a partial run
get distinct names instead of stacking on the same row.

* fix(pilot): address PR B code-review findings

Three code-review findings:

  - High: NodeManager row action buttons were icon-only with no
    accessible name. Added aria-label="Edit node",
    aria-label="Delete node", aria-label="Test connection" so the
    Playwright E2E can target them by role/name (currently the
    accessible name is rendered into a Radix tooltip portal that
    Playwright cannot resolve as the button's name).
  - High: Replaced UI-driven cleanup in the E2E with API-based
    deletion via page.request, run in both beforeEach (sweep
    leftovers) and afterEach (sweep this test's row even on failure).
    Targets every node whose name starts with the test prefix so a
    crashed previous run cannot affect the next.
  - Medium: Replaced two fixed 50ms sleeps in the integration test
    with vi.waitFor polling on hasActiveTunnel(nodeId) === false,
    eliminating the race between the WS close hop chain and the next
    test step on slow CI runners.

Plus two cleanup items:

  - The integration test's afterAll now removes 'tunnel-up' /
    'tunnel-down' listeners from the manager singleton so this
    file's runs do not leak listeners into other test files in the
    same Vitest worker.
  - Tightened the WS error swallowing in the rejection-path test:
    only swallow the expected "Unexpected server response" error
    shape; surface anything else (ECONNREFUSED etc.) instead of
    silently masking it.

No em dashes added (Directive 18 verified clean).

backend/src/__tests__/pilot-tunnel-integration.test.ts

@@ -0,0 +1,269 @@
+/**
+ * In-process integration test for the pilot reverse-tunnel handshake.
+ *
+ * Layered unit tests already cover the protocol decoder, the bridge cap,
+ * the manager, and the DB-layer enrollment lifecycle. None of them exercise
+ * the *glue*: the WebSocket dispatch order, the agent-side connect path,
+ * the actual hello / enroll_ack round-trip, the long-lived token swap. A
+ * regression that breaks the dispatch order in upgradeHandler.ts (e.g. a
+ * future refactor that puts /api/pilot/tunnel below the auth gate), or a
+ * change that breaks the enroll_ack frame ordering, would not be caught by
+ * any other test today.
+ *
+ * This test spins up a real http.Server with attachUpgrade wired up and a
+ * real ws.WebSocket client. It intentionally does NOT mock the agent side
+ * of the tunnel beyond the framing protocol; the goal is to confirm the
+ * wires connect end-to-end.
+ */
+import { describe, it, expect, beforeAll, afterAll, afterEach, vi } from 'vitest';
+import http from 'http';
+import crypto from 'crypto';
+import jwt from 'jsonwebtoken';
+import { WebSocket, WebSocketServer } from 'ws';
+import { setupTestDb, cleanupTestDb } from './helpers/setupTestDb';
+import { attachUpgrade } from '../websocket/upgradeHandler';
+import { decodeJsonFrame, type JsonFrame, PROTOCOL_VERSION } from '../pilot/protocol';
+import { PilotTunnelManager } from '../services/PilotTunnelManager';
+
+let tmpDir: string;
+let DatabaseService: typeof import('../services/DatabaseService').DatabaseService;
+
+let server: http.Server;
+let port: number;
+let pilotTunnelWss: WebSocketServer;
+let mainWss: WebSocketServer;
+let nodeId: number;
+
+beforeAll(async () => {
+    tmpDir = await setupTestDb();
+    ({ DatabaseService } = await import('../services/DatabaseService'));
+
+    server = http.createServer();
+    mainWss = new WebSocketServer({ noServer: true });
+    pilotTunnelWss = new WebSocketServer({ noServer: true });
+    attachUpgrade(server, { wss: mainWss, pilotTunnelWss });
+
+    await new Promise<void>((resolve, reject) => {
+        server.once('error', reject);
+        server.listen(0, '127.0.0.1', () => {
+            const addr = server.address();
+            if (!addr || typeof addr === 'string') {
+                reject(new Error('listen returned unexpected address'));
+                return;
+            }
+            port = addr.port;
+            resolve();
+        });
+    });
+
+    nodeId = DatabaseService.getInstance().addNode({
+        name: `pilot-integration-${Date.now()}`,
+        type: 'remote',
+        mode: 'pilot_agent',
+        compose_dir: '/tmp/x',
+        is_default: false,
+        api_url: '',
+        api_token: '',
+    });
+});
+
+afterAll(async () => {
+    const mgr = PilotTunnelManager.getInstance();
+    mgr.closeTunnel(nodeId);
+    // The manager is a process-singleton; any tunnel-up / tunnel-down
+    // listeners attached during this file's run survive into other test
+    // files in the same Vitest worker. Clear them so a sibling test that
+    // asserts listener counts or counts emissions is not polluted.
+    mgr.removeAllListeners('tunnel-up');
+    mgr.removeAllListeners('tunnel-down');
+    pilotTunnelWss.close();
+    mainWss.close();
+    await new Promise<void>((resolve) => server.close(() => resolve()));
+    cleanupTestDb(tmpDir);
+});
+
+afterEach(() => {
+    // Belt-and-braces: any test that left a tunnel open shouldn't leak into
+    // the next one. closeTunnel is a no-op if nothing is registered.
+    PilotTunnelManager.getInstance().closeTunnel(nodeId);
+});
+
+function mintEnrollToken(): string {
+    const db = DatabaseService.getInstance();
+    const jwtSecret = db.getGlobalSettings().auth_jwt_secret;
+    if (!jwtSecret) throw new Error('test DB has no auth_jwt_secret');
+    const ttlSeconds = 15 * 60;
+    const token = jwt.sign(
+        { scope: 'pilot_enroll', nodeId, enrollNonce: crypto.randomUUID() },
+        jwtSecret,
+        { expiresIn: ttlSeconds },
+    );
+    const hash = crypto.createHash('sha256').update(token).digest('hex');
+    db.createPilotEnrollment(nodeId, hash, Date.now() + ttlSeconds * 1000);
+    return token;
+}
+
+/**
+ * Open a real ws client to the test server's pilot tunnel endpoint and
+ * return a reader that surfaces decoded JSON frames in arrival order.
+ */
+async function openTunnel(token: string): Promise<{
+    ws: WebSocket;
+    nextJsonFrame: () => Promise<JsonFrame>;
+    closed: Promise<void>;
+}> {
+    const ws = new WebSocket(`ws://127.0.0.1:${port}/api/pilot/tunnel`, {
+        headers: {
+            Authorization: `Bearer ${token}`,
+            'x-sencho-agent-version': 'integration-test/1.0',
+        },
+    });
+
+    const queue: JsonFrame[] = [];
+    const waiters: Array<(frame: JsonFrame) => void> = [];
+    ws.on('message', (data, isBinary) => {
+        if (isBinary) return; // binary frames not exercised by this test
+        try {
+            const frame = decodeJsonFrame(data.toString());
+            const waiter = waiters.shift();
+            if (waiter) waiter(frame);
+            else queue.push(frame);
+        } catch {
+            // ignore malformed frames in tests
+        }
+    });
+
+    const closed = new Promise<void>((resolve) => ws.once('close', () => resolve()));
+
+    await new Promise<void>((resolve, reject) => {
+        ws.once('open', () => resolve());
+        ws.once('error', reject);
+    });
+
+    return {
+        ws,
+        nextJsonFrame: () =>
+            new Promise<JsonFrame>((resolve, reject) => {
+                if (queue.length) { resolve(queue.shift()!); return; }
+                const t = setTimeout(() => reject(new Error('timed out waiting for frame')), 5000);
+                waiters.push((f) => { clearTimeout(t); resolve(f); });
+            }),
+        closed,
+    };
+}
+
+describe('pilot tunnel handshake (in-process integration)', () => {
+    it('completes the enroll-ack token swap and registers the tunnel', async () => {
+        const enrollToken = mintEnrollToken();
+
+        const tunnel = await openTunnel(enrollToken);
+
+        // First frame: hello from primary.
+        const hello = await tunnel.nextJsonFrame();
+        expect(hello.t).toBe('hello');
+        if (hello.t !== 'hello') throw new Error('narrowing');
+        expect(hello.version).toBe(PROTOCOL_VERSION);
+        expect(hello.role).toBe('primary');
+
+        // Second frame: ctrl enroll_ack carrying the long-lived tunnel token.
+        const ack = await tunnel.nextJsonFrame();
+        expect(ack.t).toBe('ctrl');
+        if (ack.t !== 'ctrl') throw new Error('narrowing');
+        expect(ack.op).toBe('enroll_ack');
+        expect(ack.payload?.token).toBeTypeOf('string');
+        expect(ack.payload?.nodeId).toBe(nodeId);
+
+        // The manager records the tunnel as active.
+        // registerTunnel awaits bridge.start() before returning, so by the
+        // time we have the enroll_ack frame the registration has landed.
+        expect(PilotTunnelManager.getInstance().hasActiveTunnel(nodeId)).toBe(true);
+
+        tunnel.ws.close();
+        await tunnel.closed;
+    }, 10_000);
+
+    it('rejects a second connect with the now-consumed enrollment token', async () => {
+        const enrollToken = mintEnrollToken();
+
+        // First connect succeeds and consumes the row.
+        const first = await openTunnel(enrollToken);
+        await first.nextJsonFrame(); // hello
+        await first.nextJsonFrame(); // enroll_ack
+        first.ws.close();
+        await first.closed;
+        // Poll for the manager to actually drop the tunnel rather than
+        // sleeping a fixed window. The chain (ws close -> bridge
+        // onTunnelClose -> bridge.close -> 'closed' event -> manager
+        // delete) hops through several event-loop ticks.
+        await vi.waitFor(
+            () => expect(PilotTunnelManager.getInstance().hasActiveTunnel(nodeId)).toBe(false),
+            { timeout: 2000 },
+        );
+
+        // Second connect with the same enrollment token must fail at the
+        // upgrade handshake (HTTP 401 is sent before the WS upgrade).
+        const ws = new WebSocket(`ws://127.0.0.1:${port}/api/pilot/tunnel`, {
+            headers: {
+                Authorization: `Bearer ${enrollToken}`,
+                'x-sencho-agent-version': 'integration-test/1.0',
+            },
+        });
+        const result = await new Promise<{ kind: 'error' | 'open'; status?: number; error?: Error }>((resolve) => {
+            ws.on('unexpected-response', (_req, res) => {
+                resolve({ kind: 'error', status: res.statusCode });
+                res.destroy();
+            });
+            ws.on('open', () => resolve({ kind: 'open' }));
+            ws.on('error', (err) => {
+                // Reject-path triggers both 'unexpected-response' and 'error'
+                // ("Unexpected server response: 401"). We resolve from the
+                // unexpected-response handler; only escalate here if the
+                // message shape does not match the expected reject path,
+                // which would indicate a different failure (ECONNREFUSED, etc.).
+                if (!/Unexpected server response/.test(err.message)) {
+                    resolve({ kind: 'error', error: err });
+                }
+            });
+        });
+
+        expect(result.kind).toBe('error');
+        expect(result.status).toBe(401);
+    }, 10_000);
+
+    it('accepts a reconnect with the long-lived pilot_tunnel token', async () => {
+        // First flow: enroll and capture the long-lived token.
+        const enrollToken = mintEnrollToken();
+        const first = await openTunnel(enrollToken);
+        await first.nextJsonFrame(); // hello
+        const ack = await first.nextJsonFrame();
+        if (ack.t !== 'ctrl' || ack.op !== 'enroll_ack' || typeof ack.payload?.token !== 'string') {
+            throw new Error('expected enroll_ack with token');
+        }
+        const longLivedToken = ack.payload.token;
+        first.ws.close();
+        await first.closed;
+        await vi.waitFor(
+            () => expect(PilotTunnelManager.getInstance().hasActiveTunnel(nodeId)).toBe(false),
+            { timeout: 2000 },
+        );
+
+        // Reconnect with the long-lived token: no enrollment row needed,
+        // the upgrade handler accepts the pilot_tunnel scope directly.
+        const second = await openTunnel(longLivedToken);
+        const hello = await second.nextJsonFrame();
+        expect(hello.t).toBe('hello');
+
+        // No enroll_ack on a pilot_tunnel reconnect. Wait briefly to confirm
+        // no surprise frame arrives, then assert the tunnel is registered.
+        const surprise = Promise.race([
+            second.nextJsonFrame(),
+            new Promise<null>((resolve) => setTimeout(() => resolve(null), 200)),
+        ]);
+        await expect(surprise).resolves.toBeNull();
+
+        expect(PilotTunnelManager.getInstance().hasActiveTunnel(nodeId)).toBe(true);
+
+        second.ws.close();
+        await second.closed;
+    }, 10_000);
+});

e2e/pilot-agent-enrollment.spec.ts

@@ -0,0 +1,130 @@
+/**
+ * E2E coverage for the operator-side pilot-agent enrollment flow.
+ *
+ * Backend integration is covered by pilot-tunnel-integration.test.ts and
+ * the pilot-enrollment / pilot-enrollment-replay vitest suites. This file
+ * exercises the parts only the browser sees: the mode selector, the
+ * enrollment dialog, the docker run code block, and the regenerate
+ * affordance on an existing pilot-mode node.
+ *
+ * Out of scope: simulating an agent connecting to flip the row to Online.
+ * The integration test covers the wire side; the E2E focuses on what the
+ * operator clicks and reads.
+ */
+import { test, expect, type Page } from '@playwright/test';
+import { loginAs } from './helpers';
+
+const NAME_PREFIX = 'pilot-e2e-';
+
+/**
+ * API-based teardown: list all nodes whose name starts with the test prefix
+ * and DELETE them. Reuses the browser's auth cookie so we get the same
+ * permissions as the logged-in admin. Reliable in a way that "click the
+ * delete icon, then click confirm" never can be (animation timing,
+ * confirmation modal markup drift, and so on).
+ */
+async function deleteTestNodes(page: Page): Promise<void> {
+  const list = await page.request.get('/api/nodes');
+  if (!list.ok()) return;
+  const nodes = (await list.json()) as Array<{ id: number; name: string }>;
+  for (const n of nodes) {
+    if (n.name.startsWith(NAME_PREFIX)) {
+      await page.request.delete(`/api/nodes/${n.id}`).catch(() => undefined);
+    }
+  }
+}
+
+test.describe('Pilot Agent enrollment', () => {
+  test.beforeEach(async ({ page }) => {
+    await loginAs(page);
+    // Sweep any leftover test nodes BEFORE running so a previous failed
+    // run cannot affect this one.
+    await deleteTestNodes(page);
+    // Settings lives inside the User Profile Dropdown.
+    await page.getByRole('button', { name: /profile/i }).click();
+    await page.getByRole('button', { name: 'Settings', exact: true }).click();
+    await page.getByRole('button', { name: /^nodes$/i }).click();
+  });
+
+  test.afterEach(async ({ page }) => {
+    // Cleanup via the API so a UI assertion failure does not leave rows
+    // behind. Runs even when the test body throws.
+    await deleteTestNodes(page);
+  });
+
+  test('creating a pilot-agent node opens the enrollment dialog with a docker run command', async ({ page }) => {
+    const nodeName = `${NAME_PREFIX}create-${Date.now()}`;
+
+    const addBtn = page.getByRole('button', { name: /add node/i }).first();
+    if (!await addBtn.isVisible()) {
+      test.skip();
+      return;
+    }
+    await addBtn.click();
+    await expect(page.locator('#node-name')).toBeVisible({ timeout: 5_000 });
+
+    // Switch type to Remote. Pilot Agent is the default mode for remote
+    // nodes (NodeManager.tsx initializes formData.mode = 'pilot_agent'),
+    // so the mode combobox does not need to be touched.
+    await page.locator('#node-type').click();
+    await page.getByRole('option', { name: /remote/i }).click();
+
+    // Confirm pilot mode is selected and the proxy-only api_url field is
+    // NOT rendered. A regression that flipped the default would surface
+    // here as the api_url field becoming visible.
+    await expect(page.locator('#node-api-url')).toHaveCount(0);
+
+    await page.locator('#node-name').fill(nodeName);
+    await page.getByRole('dialog').getByRole('button', { name: /add node/i }).click();
+
+    // Enrollment modal opens. The docker run command must contain the
+    // SENCHO_MODE flag and a JWT-shaped Bearer token.
+    await expect(page.getByText(/Run this command on/i)).toBeVisible({ timeout: 10_000 });
+
+    const dockerCommand = page.locator('pre').filter({ hasText: /SENCHO_MODE=pilot/ });
+    await expect(dockerCommand).toBeVisible();
+
+    const cmd = await dockerCommand.innerText();
+    expect(cmd).toContain('SENCHO_MODE=pilot');
+    expect(cmd).toContain('SENCHO_PRIMARY_URL=');
+    expect(cmd).toMatch(/SENCHO_ENROLL_TOKEN=[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+/);
+  });
+
+  test('regenerating the enrollment token issues a fresh docker run command', async ({ page }) => {
+    const nodeName = `${NAME_PREFIX}regen-${Date.now()}`;
+
+    const addBtn = page.getByRole('button', { name: /add node/i }).first();
+    if (!await addBtn.isVisible()) {
+      test.skip();
+      return;
+    }
+    await addBtn.click();
+    await expect(page.locator('#node-name')).toBeVisible({ timeout: 5_000 });
+    await page.locator('#node-type').click();
+    await page.getByRole('option', { name: /remote/i }).click();
+    await page.locator('#node-name').fill(nodeName);
+    await page.getByRole('dialog').getByRole('button', { name: /add node/i }).click();
+
+    const firstCommand = page.locator('pre').filter({ hasText: /SENCHO_MODE=pilot/ });
+    await expect(firstCommand).toBeVisible({ timeout: 10_000 });
+    const firstText = await firstCommand.innerText();
+    const firstToken = firstText.match(/SENCHO_ENROLL_TOKEN=([A-Za-z0-9_.-]+)/)?.[1];
+    expect(firstToken).toBeTruthy();
+
+    // Close the enrollment dialog (Escape lands on the row view).
+    await page.keyboard.press('Escape');
+
+    // Open the row's edit dialog via the aria-labeled icon button.
+    const row = page.getByRole('row', { name: new RegExp(nodeName) }).first();
+    await expect(row).toBeVisible({ timeout: 5_000 });
+    await row.getByRole('button', { name: 'Edit node' }).click();
+    await page.getByRole('button', { name: /regenerate enrollment token/i }).click();
+
+    const secondCommand = page.locator('pre').filter({ hasText: /SENCHO_MODE=pilot/ });
+    await expect(secondCommand).toBeVisible({ timeout: 10_000 });
+    const secondText = await secondCommand.innerText();
+    const secondToken = secondText.match(/SENCHO_ENROLL_TOKEN=([A-Za-z0-9_.-]+)/)?.[1];
+    expect(secondToken).toBeTruthy();
+    expect(secondToken).not.toBe(firstToken);
+  });
+});

frontend/src/components/NodeManager.tsx

@@ -665,6 +665,7 @@ export function NodeManager() {
                             className="h-8 w-8"
                             onClick={() => testConnection(node)}
                             disabled={testing === node.id}
+                            aria-label="Test connection"
                           >
                             <Wifi className={`w-4 h-4 ${testing === node.id ? 'animate-pulse' : ''}`} />
                           </Button>
@@ -681,6 +682,7 @@ export function NodeManager() {
                             size="icon"
                             className="h-8 w-8"
                             onClick={() => openEditDialog(node)}
+                            aria-label="Edit node"
                           >
                             <Pencil className="w-4 h-4" />
                           </Button>
@@ -698,6 +700,7 @@ export function NodeManager() {
                               size="icon"
                               className="h-8 w-8 text-destructive hover:text-destructive"
                               onClick={() => { setDeletingNode(node); setDeleteOpen(true); }}
+                              aria-label="Delete node"
                             >
                               <Trash2 className="w-4 h-4" />
                             </Button>