mirror of
https://github.com/PerpetualSoftware/pad.git
synced 2026-09-11 13:28:57 +00:00
088ba2f839
* fix(mcp): raise MCP per-token burst + classify 429 as ErrRateLimited (BUG-1430)
BUG-1409 reported an agent hitting "Pad backend 500s on parallel writes"
during workspace onboarding via remote MCP on Pad Cloud. Triage split
that umbrella into three children; this PR addresses BUG-1430 (the
parallel-writes symptom).
Root cause investigation showed the underlying write path is fine —
local SQLite handled 24 parallel item-create POSTs cleanly (busy_timeout
+ BEGIN IMMEDIATE + WAL serialize writers without errors). The most
plausible cause of the agent's "500 on parallel writes" report is the
MCP per-token rate limiter (burst 20, 60/min) rejecting requests 21-24
of an onboarding burst with HTTP 429, which the dispatcher's classifier
then collapsed into a generic ErrServerError envelope.
Changes:
- middleware_ratelimit.go: MCPPerToken burst 20 → 60. Sustained rate
unchanged at 60/min/token. Matches the general API limiter's burst-60
per-user cap so the MCP path no longer imposes a tighter ceiling than
the equivalent /api/v1 path. Comment expanded to record the rationale.
- internal/mcp/errors.go: add ErrRateLimited error code and an explicit
case http.StatusTooManyRequests in classifyHTTPStatusKind. 429s now
surface as a first-class rate-limited envelope with an actionable hint
pointing at Retry-After and the per-token cap, instead of landing in
the generic ErrServerError "other 4xx" bucket. Agents implementing
exponential backoff can switch on code without parsing free-form text.
- handlers_cloud.go: add slog.Error instrumentation to enforcePlanLimit
and enforceUserPlanLimit error paths. These are cloud-mode-only 500
candidates we couldn't exercise locally (local dev runs cloudMode=false);
the structured logs give operators a grep-able tag the next time the
symptom surfaces on real Pad Cloud, so we can rule the path in or out
empirically without another investigation pass.
- tests: bump iteration counts past the new burst (20 → 60), add 429
case to classifyHTTPStatus code-mapping table + envelope hint-shape
table.
Investigation context (full triage in BUG-1430):
- ../pad-cloud sidecar is NOT in the /api/v1 or /mcp request path
(nginx-router proxies those directly to pad backend).
- featureCount + advisory-lock contention on Postgres remain plausible
500 candidates under heavy bursts; the new logging is intended to
catch those if they fire.
Siblings BUG-1431 (status field placement) and BUG-1432 (tags field)
are tracked separately and not addressed here.
* fix(mcp): drop hardcoded cap from rate-limit hint per Codex review (round 1)
Codex round 1 [P2] caught that rateLimitHintFor's "the per-token cap is
60 req/min with a burst of 60" text was misleading: classifyHTTPStatusKind
handles 429s from the dispatcher's SYNTHESIZED /api/v1/... requests, which
come from the general API limiter (600/min, burst 60), the Search limiter
(30/min, burst 10), and potentially others — NOT the MCP per-token
limiter (which fires before the dispatcher runs and so never lands in
this classifier path).
Generalize the hint: point at Retry-After (which carries the correct
limiter-specific wait) and drop the cap from prose. Update the matching
test assertion to assert the generic shape ("burst-heavy" instead of
"60 req/min").
300 lines
12 KiB
Go
300 lines
12 KiB
Go
package mcp
|
|
|
|
// Tests for TASK-1077 / 1078 / 1079 — pin the uniform error envelope
|
|
// shape across every HTTP-dispatcher tool.
|
|
//
|
|
// Pre-fix some dispatchers (note, decide, project next, bulk-update)
|
|
// emitted plain-string errors via NewToolResultErrorf. Post-fix every
|
|
// error path goes through validationFailedResult / dispatcherErrorResult
|
|
// / upstreamHTTPErrorResult, which all wrap the standard envelope:
|
|
//
|
|
// {"error": {"code": "...", "message": "...", "hint": "..."}}
|
|
//
|
|
// This test parameterizes over each dispatcher's missing-required-input
|
|
// error path because that's the cheapest reproducible failure case
|
|
// (no need to mock a backend handler) and the path most agents will
|
|
// hit when learning the surface.
|
|
|
|
import (
|
|
"context"
|
|
"encoding/json"
|
|
"strings"
|
|
"testing"
|
|
|
|
"github.com/mark3labs/mcp-go/mcp"
|
|
|
|
"github.com/PerpetualSoftware/pad/internal/models"
|
|
)
|
|
|
|
// TestDispatcher_AllErrorsUseStructuredEnvelope walks every special-
|
|
// case dispatcher + each link command and asserts that calling them
|
|
// without the required workspace/ref/etc. produces a structured
|
|
// {error: {code, message, hint}} envelope — never a bare string.
|
|
//
|
|
// New dispatchers that emit errors via NewToolResultErrorf will fail
|
|
// this test, which is the regression gate TASK-1077 wants.
|
|
func TestDispatcher_AllErrorsUseStructuredEnvelope(t *testing.T) {
|
|
user := &models.User{ID: "u-1", Name: "Tester"}
|
|
d := &HTTPHandlerDispatcher{
|
|
Handler: errorHandler(t, "must not call backend"),
|
|
UserResolver: fixedUserResolver(user),
|
|
}
|
|
|
|
cases := []struct {
|
|
name string
|
|
cmdPath []string
|
|
input map[string]any
|
|
wantCode ErrorCode // expected envelope code
|
|
}{
|
|
// Missing workspace on every workspace-required dispatcher.
|
|
// The cmdPath list mirrors the special-case switch in
|
|
// HTTPHandlerDispatcher.Dispatch — adding a new entry there
|
|
// without a matching case here is a smell.
|
|
{"item update no workspace", []string{"item", "update"}, map[string]any{}, ErrValidationFailed},
|
|
{"item deps no workspace", []string{"item", "deps"}, map[string]any{}, ErrValidationFailed},
|
|
{"item related no workspace", []string{"item", "related"}, map[string]any{}, ErrValidationFailed},
|
|
{"item implemented-by no workspace", []string{"item", "implemented-by"}, map[string]any{}, ErrValidationFailed},
|
|
{"item bulk-update no workspace", []string{"item", "bulk-update"}, map[string]any{}, ErrValidationFailed},
|
|
{"item note no workspace", []string{"item", "note"}, map[string]any{}, ErrValidationFailed},
|
|
{"item decide no workspace", []string{"item", "decide"}, map[string]any{}, ErrValidationFailed},
|
|
{"project ready no workspace", []string{"project", "ready"}, map[string]any{}, ErrValidationFailed},
|
|
{"project stale no workspace", []string{"project", "stale"}, map[string]any{}, ErrValidationFailed},
|
|
{"project next no workspace", []string{"project", "next"}, map[string]any{}, ErrValidationFailed},
|
|
{"project standup no workspace", []string{"project", "standup"}, map[string]any{}, ErrValidationFailed},
|
|
{"project changelog no workspace", []string{"project", "changelog"}, map[string]any{}, ErrValidationFailed},
|
|
{"attachment list no workspace", []string{"attachment", "list"}, map[string]any{}, ErrValidationFailed},
|
|
{"attachment show no workspace", []string{"attachment", "show"}, map[string]any{}, ErrValidationFailed},
|
|
|
|
// Link commands — workspace required, then refs.
|
|
{"item block no workspace", []string{"item", "block"}, map[string]any{}, ErrValidationFailed},
|
|
{"item blocked-by no workspace", []string{"item", "blocked-by"}, map[string]any{}, ErrValidationFailed},
|
|
{"item unblock no workspace", []string{"item", "unblock"}, map[string]any{}, ErrValidationFailed},
|
|
|
|
// Library commands.
|
|
{"library activate no workspace", []string{"library", "activate"}, map[string]any{}, ErrValidationFailed},
|
|
}
|
|
|
|
for _, tc := range cases {
|
|
t.Run(tc.name, func(t *testing.T) {
|
|
ctx := WithDispatchInput(context.Background(), tc.input)
|
|
res, err := d.Dispatch(ctx, tc.cmdPath, nil)
|
|
if err != nil {
|
|
t.Fatalf("Dispatch protocol error: %v", err)
|
|
}
|
|
if res == nil {
|
|
t.Fatal("nil result")
|
|
}
|
|
if !res.IsError {
|
|
t.Errorf("expected IsError=true; result=%+v", res)
|
|
}
|
|
env := unwrapErrorEnvelope(t, res)
|
|
if env.Error.Code != tc.wantCode {
|
|
t.Errorf("code: got %q, want %q (full envelope: %+v)",
|
|
env.Error.Code, tc.wantCode, env.Error)
|
|
}
|
|
if env.Error.Message == "" {
|
|
t.Error("envelope missing Message")
|
|
}
|
|
if env.Error.Hint == "" {
|
|
t.Error("envelope missing Hint — agents need actionable recovery text (TASK-1079)")
|
|
}
|
|
// Hints should NOT be the literal "404 page not found" /
|
|
// "page not found" passthrough that triggered Bug 17.
|
|
if strings.EqualFold(env.Error.Hint, "404 page not found") ||
|
|
strings.EqualFold(env.Error.Hint, "page not found") {
|
|
t.Errorf("hint is the raw HTTP status text — not actionable (Bug 17 / TASK-1079): %q",
|
|
env.Error.Hint)
|
|
}
|
|
})
|
|
}
|
|
}
|
|
|
|
// TestClassifyHTTPStatus_KindAware pins TASK-1078's resource-kind
|
|
// classification — 404s split into item_not_found / unknown_workspace /
|
|
// not_found based on what the dispatcher was reading, instead of the
|
|
// pre-fix blanket item_not_found.
|
|
func TestClassifyHTTPStatus_KindAware(t *testing.T) {
|
|
cases := []struct {
|
|
name string
|
|
kind ResourceKind
|
|
wantCode ErrorCode
|
|
}{
|
|
{"item lookup → item_not_found", ResourceItem, ErrItemNotFound},
|
|
{"workspace lookup → unknown_workspace", ResourceWorkspace, ErrUnknownWorkspace},
|
|
{"collection lookup → not_found", ResourceCollection, ErrNotFound},
|
|
{"listing route → not_found", ResourceListing, ErrNotFound},
|
|
{"link target → not_found", ResourceLink, ErrNotFound},
|
|
{"attachment → not_found", ResourceAttachment, ErrNotFound},
|
|
}
|
|
for _, tc := range cases {
|
|
t.Run(tc.name, func(t *testing.T) {
|
|
res := classifyHTTPStatusKind(context.Background(),
|
|
"item show", "/api/v1/workspaces/foo/items/TASK-7",
|
|
404, []byte("404 page not found"), nil, tc.kind, "TASK-7")
|
|
env, ok := res.StructuredContent.(ErrorEnvelope)
|
|
if !ok {
|
|
t.Fatalf("expected ErrorEnvelope, got %T", res.StructuredContent)
|
|
}
|
|
if env.Error.Code != tc.wantCode {
|
|
t.Errorf("code: got %q, want %q", env.Error.Code, tc.wantCode)
|
|
}
|
|
})
|
|
}
|
|
}
|
|
|
|
// TestClassifyHTTPStatus_HintsAreActionable pins TASK-1079's hint
|
|
// improvements: hints reference the actual route + ref (or workspace
|
|
// slug) the dispatcher was reading, and never just dump the upstream
|
|
// "404 page not found" passthrough.
|
|
func TestClassifyHTTPStatus_HintsAreActionable(t *testing.T) {
|
|
cases := []struct {
|
|
name string
|
|
kind ResourceKind
|
|
ref string
|
|
route string
|
|
body []byte
|
|
mustContain []string
|
|
mustNotContainEq string // forbid this as the WHOLE hint
|
|
}{
|
|
{
|
|
name: "item 404 hint references the ref + recovery tools",
|
|
kind: ResourceItem,
|
|
ref: "TASK-7",
|
|
route: "/api/v1/workspaces/foo/items/TASK-7",
|
|
body: []byte("404 page not found"),
|
|
mustContain: []string{"TASK-7", "/api/v1/workspaces/foo/items/TASK-7", "pad_item"},
|
|
},
|
|
{
|
|
name: "collection 404 hint suggests the listing tool + names route",
|
|
kind: ResourceCollection,
|
|
ref: "tasks",
|
|
route: "/api/v1/workspaces/foo/collections/tasks",
|
|
body: []byte("404 page not found"),
|
|
mustContain: []string{"pad_collection list", "/api/v1/workspaces/foo/collections/tasks"},
|
|
},
|
|
{
|
|
name: "5xx hint suggests retry + flags as transient",
|
|
kind: ResourceItem, // kind doesn't matter for 5xx
|
|
ref: "TASK-7",
|
|
route: "/api/v1/workspaces/foo/items/TASK-7",
|
|
body: []byte(`{"error":{"message":"db down"}}`),
|
|
mustContain: []string{"transient", "retry", "500"},
|
|
},
|
|
{
|
|
// BUG-1430: 429 hint points at backoff + Retry-After so
|
|
// agents implementing backoff can adjust without parsing
|
|
// free-form prose. Cap is intentionally NOT named in the
|
|
// hint — classifyHTTPStatusKind handles synthesized
|
|
// /api/v1/... 429s which can come from several limiters
|
|
// with different sizing (Codex review #546 round 1 [P2]).
|
|
name: "429 hint suggests backoff + points at Retry-After",
|
|
kind: ResourceItem, // kind doesn't matter for 429
|
|
ref: "TASK-7",
|
|
route: "/api/v1/workspaces/foo/items/TASK-7",
|
|
body: []byte(`{"error":{"code":"rate_limited","message":"Too many requests."}}`),
|
|
mustContain: []string{"Rate-limited", "Retry-After", "burst-heavy"},
|
|
},
|
|
}
|
|
for _, tc := range cases {
|
|
t.Run(tc.name, func(t *testing.T) {
|
|
status := 404
|
|
if strings.Contains(tc.name, "5xx") {
|
|
status = 500
|
|
} else if strings.Contains(tc.name, "429") {
|
|
status = 429
|
|
}
|
|
res := classifyHTTPStatusKind(context.Background(),
|
|
"test cmd", tc.route, status, tc.body, nil, tc.kind, tc.ref)
|
|
env := res.StructuredContent.(ErrorEnvelope)
|
|
if env.Error.Hint == "" {
|
|
t.Fatal("hint is empty")
|
|
}
|
|
for _, want := range tc.mustContain {
|
|
if !strings.Contains(env.Error.Hint, want) {
|
|
t.Errorf("hint missing %q; got %q", want, env.Error.Hint)
|
|
}
|
|
}
|
|
// Forbid the bare "404 page not found" passthrough that
|
|
// triggered Bug 17.
|
|
if env.Error.Hint == "404 page not found" {
|
|
t.Errorf("hint is the raw HTTP status text passthrough (Bug 17 regression)")
|
|
}
|
|
})
|
|
}
|
|
}
|
|
|
|
// TestExtractUpstreamMessage covers each shape the helper handles —
|
|
// envelope happy path, chi default 404, and the safe-fallback cases.
|
|
//
|
|
// Codex review #387 round 1 caught that the pre-fix fallback returned
|
|
// the raw body verbatim, which would forward arbitrary upstream JSON
|
|
// (potentially including tokens / passwords / debug dumps) into the
|
|
// hint surface. Post-fix non-envelope bodies return "" — the hint
|
|
// surface omits the upstream-message clause entirely and operators
|
|
// debugging unstructured upstream errors check the pad container
|
|
// logs instead.
|
|
func TestExtractUpstreamMessage(t *testing.T) {
|
|
cases := []struct {
|
|
name string
|
|
in string
|
|
want string
|
|
}{
|
|
// Happy path — pad's writeError envelope shape.
|
|
{"envelope happy path", `{"error":{"message":"workspace not visible"}}`, "workspace not visible"},
|
|
{"envelope with sibling fields", `{"error":{"message":"db down"},"code":"db_err"}`, "db down"},
|
|
|
|
// chi's default NotFound body — known-zero-value, drop it.
|
|
{"chi 404 literal", "404 page not found", ""},
|
|
{"chi 404 with trailing newline", "404 page not found\n", ""},
|
|
|
|
// Empty / whitespace input → empty output.
|
|
{"empty", "", ""},
|
|
{"whitespace only", " ", ""},
|
|
|
|
// Safe fallbacks: anything we can't recognize as the structured
|
|
// envelope returns "" rather than leaking the raw body.
|
|
{"empty inner message", `{"error":{"message":""}}`, ""},
|
|
{"missing inner message field", `{"error":{}}`, ""},
|
|
{"unparseable", "not json at all", ""},
|
|
{"wrong shape", `{"unrelated":"shape"}`, ""},
|
|
{"html debug dump with token-like content", `<html>internal stack trace with token=abc</html>`, ""},
|
|
{"sensitive-looking JSON (NOT envelope shape)", `{"secret":"abc","token":"xyz"}`, ""},
|
|
}
|
|
for _, tc := range cases {
|
|
t.Run(tc.name, func(t *testing.T) {
|
|
if got := extractUpstreamMessage(tc.in); got != tc.want {
|
|
t.Errorf("extractUpstreamMessage(%q) = %q, want %q", tc.in, got, tc.want)
|
|
}
|
|
})
|
|
}
|
|
}
|
|
|
|
// unwrapErrorEnvelope extracts the structured ErrorEnvelope from a
|
|
// CallToolResult. Test helper — fails fast if the result isn't shaped
|
|
// like a structured envelope (which is the whole regression we're
|
|
// guarding against).
|
|
func unwrapErrorEnvelope(t *testing.T, res *mcp.CallToolResult) ErrorEnvelope {
|
|
t.Helper()
|
|
if res == nil {
|
|
t.Fatal("result is nil")
|
|
}
|
|
if env, ok := res.StructuredContent.(ErrorEnvelope); ok {
|
|
return env
|
|
}
|
|
// Fallback: parse the JSON content. Some helpers use
|
|
// NewErrorResult which sets StructuredContent directly; others
|
|
// might not have the typed struct populated — try to round-trip
|
|
// from the text content for robustness.
|
|
if len(res.Content) > 0 {
|
|
if tc, ok := res.Content[0].(mcp.TextContent); ok {
|
|
var env ErrorEnvelope
|
|
if err := json.Unmarshal([]byte(tc.Text), &env); err == nil && env.Error.Code != "" {
|
|
return env
|
|
}
|
|
t.Fatalf("text content isn't a parseable ErrorEnvelope: %s", tc.Text)
|
|
}
|
|
}
|
|
t.Fatalf("result has no parseable error envelope; result=%+v", res)
|
|
return ErrorEnvelope{}
|
|
}
|