From 707dbefa1378aa75fc0cabcf3e74592e2ebebb7b Mon Sep 17 00:00:00 2001 From: Ben Duncan <5178243+benduncan@users.noreply.github.com> Date: Sat, 31 Oct 2026 13:42:13 -0500 Subject: [PATCH] fix: bound the capacity describe reply by instance-type count, not free slots (#1368) * fix: bound the capacity describe reply by instance-type count, not free slots A node answering `DescribeInstanceTypes` with `capacity=true` sent one full copy of an instance type's description per free slot, so the reply grew with how much capacity the host had free. At 1344 bytes per slot the four-node production cluster, holding no guests, produced 874 slots and a reply over the 1 MB NATS maximum payload. Every node's reply was dropped, the gateway collected zero frames, and it returned an empty list as a success. The console reads an empty capacity list as no capacity and refuses every launch, which is how a cluster with hundreds of free vCPU reported "No compute available". The emptier the cluster, the more likely the failure, which is the opposite of what the shape should be. A node now returns each admissible type once with the number of instances of it the node can take, so its reply is bounded by the size of the instance-type catalogue and not by free capacity. On the production catalogue that is 84 KB for 62 types whatever the slot count, against 59 MB for the same reply on a large idle host under the old shape. The gateway sums the per-type counts across nodes and expands them on the last hop, because the client SDK drops response fields it does not model and the console displays the figure to the customer. A capacity fan-out that collected no frames now returns a retryable ServiceUnavailable. Reporting an empty list as a success is what turned a transport failure into a false statement about the cluster. Two tests pin the shape: the node reply for an idle 4096-vCPU host stays under the NATS payload limit, and a capacity fan-out with no responders does not return an empty success. * test: cover the console against a real cluster, and pin the capacity banner The console has never been exercised against a real control plane. `playwright.config.ts` pins `localhost:3101` and `:3102` and intercepts every API call, so the suite cannot be pointed at a cluster and cannot see a defect whose whole content is what the control plane actually sends. That is why an idle production cluster came to tell customers it had no compute: the capacity listing was dropped in transport, the empty result read as no capacity, and nothing in CI could reach the page that says so. `playwright.live.config.ts` runs the same console against a cluster named by `SPX_CONSOLE_URL`, signing in with `SPX_ACCESS_KEY_ID` and `SPX_SECRET_ACCESS_KEY`. It starts no server, mocks nothing, runs one worker because it launches real guests, and is never part of `pnpm e2e`. The suite signs in, asserts a cluster with free capacity offers instance types rather than reporting none, reads the per-type capacity figure, launches an instance and waits for `running`, reads its live metric graphs, terminates it in a `finally` block, and loads the nine listing pages a customer reaches from the sidebar. Three tests in the mocked suite pin the banner's own logic, which is cheap and catches the UI side of the same failure: the form offers what the capacity listing reports, it shows the free slot count for the selected type, and an account with genuinely no capacity is told so and cannot submit. The third passes on the current code and is the one that documents why the banner exists at all. * test: correct three live-console selectors found by running it against prod The first run of the live suite was 11 passed and 2 failed, both in the suite rather than in the console. The dashboard check looked for node status on `/`, where it is not shown. It now loads `/nodes` and asserts the `Resource usage by node` and `Instance type capacity` tables, which is where the daemon reports a node's own capacity. The OS tile's radio is `sr-only` behind the distribution logo, so `check()` failed its actionability test on the `` covering it. The click goes to the enclosing `