mirror of
https://github.com/PerpetualSoftware/pad.git
synced 2026-10-26 14:30:13 +00:00
7825f3afc5
* fix(mcp): MCP's long-lived streams keep alive and take a stream admission slot (BUG-3835) An idle MCP event stream made no write, so #2182's write deadline never fired on it: a client that stopped reading but held TCP open pinned a handler goroutine and its session until a notification happened to be written, and a signed-in client could open any number of them. - A ': keepalive' comment at the /events cadence on any response the transport commits as 200 text/event-stream, written through the same bounded writer. mcp-go writes each SSE event with one Write, so a mutex at Write boundaries keeps keepalives between events. No keepalive after a JSON-RPC response (the final event of a POST stream) or after the handler returns. mcp-go's own heartbeat is not used: it sends ping requests every client must answer. - The shared streamAdmission gate, keyed by streamPrincipal, taken at request entry for the two long-lived doors: GET /mcp and a subscriptions/listen POST. A refusal is the same plain 429 sse_limit_exceeded with Retry-After 5, sent before the transport runs. A tool call upgraded to SSE mid-call is not counted: a refusal there would land after its side effects. Listen is detected by decoding the whole body exactly as mcp-go does (encoding/json into a "method" field), not by a fixed-size peek that key order could evade. - Docs, config, the startup log and the metric help say the budget now covers MCP's long-lived streams. Credential expiry for /mcp streams is BUG-3837. Claude-Session: https://claude.ai/code/session_01Rj4CMyK2EqxerMihbGs3Kr * docs(mcp): the keepalive's shutdown window, the event-store premise and the chatgpt mount, stated precisely (BUG-3835) Codex r1 (Medium, proposed declined): a keepalive tick between the transport's return and end() can add one comment frame; it is still inside mcpBoundWrites, which joins the keepalive goroutine before returning, so net/http never sees a write after the handler. The comments now say so. Codex r1 (Low): GET /mcp/chatgpt is admitted too; the docs named only /mcp. Hunter r1: the one-Write premise holds only without an event store (writeSSEEventRaw writes twice); stated at the premise. Detection of a stalled idle client takes up to about three keepalive intervals; stated in docs/api.md. Claude-Session: https://claude.ai/code/session_01Rj4CMyK2EqxerMihbGs3Kr * fix(mcp): a panicking transport stops the keepalive; MCP streams are stated long-lived without a credential re-check (BUG-3835) Hunter r2 (Low-Medium): the keepalive shutdown ran after next.ServeHTTP without defer, so a panicking transport left it running past the handler, with chi's Recoverer swallowing the panic. Now deferred; pinned by TestMCP_BUG3835_PanickingTransportStopsTheKeepalive (red when not deferred). Hunter r2 (Medium): docs/api.md and two code comments said no MCP route is long-lived. The MCP transport's GET and subscriptions/listen streams are, and do not yet re-check their credential; that is BUG-3837. Claude-Session: https://claude.ai/code/session_01Rj4CMyK2EqxerMihbGs3Kr * docs(mcp): the keepalive catches dead peers; a live non-reader is bounded by the budget, not detected (BUG-3835) r3 hunter Medium: with default socket buffers a comment every 30s takes days to fill them, so the "caught within about three keepalive intervals" claim held only in the shrunk-buffer tests. api.md, the file header comment and the stalled-reader test comment now say what the keepalive catches and what the budget bounds. Codex r3 Low: deployment.md's stream gauge and checklist now include MCP's long-lived streams. The end()/ended comments state the one-frame window between the transport's return and end(). Claude-Session: https://claude.ai/code/session_01Rj4CMyK2EqxerMihbGs3Kr * fix(mcp): the admission body read is in the body census; dead-peer detection is stated as TCP gives it (BUG-3835) r4 blind hunter: - M1: mcpLongLivedRequest reads the request body and was not in the body reader census, so TestEveryRequestBodyReaderIsAccountedFor failed. It is now listed, with the reason and the second-buffer cost the lead accepted. - M3: a half-open peer accepts keepalive writes into the socket buffer until TCP gives up on it; "within one keepalive plus the write deadline" held only in the shrunk-buffer tests. api.md and the header comment now say so. - L3: the panicking-transport test waits for its first keepalive, read under the writer's mutex, instead of sleeping a fixed 10ms (M7 still red). Claude-Session: https://claude.ai/code/session_01Rj4CMyK2EqxerMihbGs3Kr * fix(mcp): admission reads a POST body only when its protocol header is modern; r6 fixes (BUG-3835) Pad serves the legacy MCP protocol versions only, and mcp-go serves a subscriptions/listen only on a modern-era POST whose Mcp-Protocol-Version header is modern and matches _meta. So a POST without a modern header is never a listen, and admission no longer reads its whole body a second time (r6 hunter M1, gate approved by the lead). The docs, the metrics help, the startup log and the census reason say listen is counted once the modern era is served. api.md states the keepalive outlives a proxy idle timeout until BUG-3837, the deployment note says the per-user bucket is shared with the browser's /events, and a test pins isFinalSSEEvent against the real transport's response frame. Claude-Session: https://claude.ai/code/session_01Rj4CMyK2EqxerMihbGs3Kr * fix(mcp): admission looks for a listen only on a protocol version the transport serves; r7 fixes (BUG-3835) The listen gate now also requires the header's version to be one the mount's transport serves, read through mcp-go's SupportsProtocolVersion (forwarded through the ChatGPT wrapper). A client probing an unserved modern version at a full budget reaches mcp-go and gets the 400 it falls back on, not a 429 (r7 ruling). Pad serves the legacy versions only, so no POST body is read for admission today, which the docs now say. New tests pin that mcp-go never serves a listen without a modern header (with a control that it serves a well-formed one), that an unserved modern version is not read or counted, and that both production mounts report their served versions. Claude-Session: https://claude.ai/code/session_01Rj4CMyK2EqxerMihbGs3Kr * test(mcp): the served-version gate is checked against the transports pad server mounts (BUG-3835) r8 L2 (lead ruling): internal/mcp's wiring test builds its own transports, so a shim around either production binding would silently send admission back to counting every modern version. Server.MCPMountServesVersion exposes the gate's read of the transport the server holds, and a cmd/pad test drives wireMCP and asserts both /mcp and /mcp/chatgpt report their versions. Shim mutants on each binding turn it red. Claude-Session: https://claude.ai/code/session_012QxKpjqXTpU84auY5zZ8FR * docs(mcp): the bus and mcpBoundWrites comments name the MCP stream slot and keepalive (BUG-3835) r5 hunter 3 and hunter 6's doc half, accepted by the lead (16:04Z): internal/events/bus.go said streamAdmission is acquired by "both endpoints", falsified once MCP's long-lived streams acquire it too, and mcpBoundWrites' doc comment did not mention the admission slot or the keepalive. Comment-only. Claude-Session: https://claude.ai/code/session_012QxKpjqXTpU84auY5zZ8FR * docs(api): the stream budget's 429 is stated once, and the MCP keepalive has its own paragraph (BUG-3835) r5 hunter 2, accepted by the lead (16:04Z): the admission paragraph stated the 429 refusal twice, the second time after the keepalive text it no longer fitted. Docs-only. Claude-Session: https://claude.ai/code/session_012QxKpjqXTpU84auY5zZ8FR
544 lines
27 KiB
Go
544 lines
27 KiB
Go
package server
|
|
|
|
import (
|
|
"go/ast"
|
|
"go/types"
|
|
"path/filepath"
|
|
"sort"
|
|
"strings"
|
|
"sync"
|
|
"testing"
|
|
|
|
"golang.org/x/tools/go/packages"
|
|
)
|
|
|
|
// bodyReaderSelectors are the members of net/http.Request that read the
|
|
// request BODY, or expose what parsing it produced. FormValue, PostFormValue,
|
|
// ParseForm, ParseMultipartForm and FormFile parse it; MultipartReader streams
|
|
// it; PostForm and MultipartForm hold the parsed result.
|
|
var bodyReaderSelectors = map[string]bool{
|
|
"Body": true, "FormValue": true, "PostFormValue": true, "ParseForm": true,
|
|
"ParseMultipartForm": true, "MultipartForm": true, "PostForm": true,
|
|
"MultipartReader": true, "FormFile": true,
|
|
}
|
|
|
|
var (
|
|
censusLoadMu sync.Mutex
|
|
censusLoadCache = map[string]*packages.Package{}
|
|
)
|
|
|
|
// loadCensusPackage type-checks the one package in dir, once per test binary:
|
|
// the reader census and the handoff census ask different questions of the same
|
|
// load, and a go/packages load of this package is several seconds.
|
|
func loadCensusPackage(t *testing.T, dir string) *packages.Package {
|
|
t.Helper()
|
|
censusLoadMu.Lock()
|
|
defer censusLoadMu.Unlock()
|
|
if p, ok := censusLoadCache[dir]; ok {
|
|
return p
|
|
}
|
|
cfg := &packages.Config{
|
|
Mode: packages.NeedName | packages.NeedFiles | packages.NeedSyntax |
|
|
packages.NeedTypes | packages.NeedTypesInfo | packages.NeedImports |
|
|
packages.NeedDeps | packages.NeedModule,
|
|
Dir: dir,
|
|
Tests: false,
|
|
}
|
|
pkgs, err := packages.Load(cfg, ".")
|
|
if err != nil {
|
|
t.Fatalf("load %s: %v", dir, err)
|
|
}
|
|
if len(pkgs) != 1 {
|
|
t.Fatalf("load %s: want 1 package, got %d", dir, len(pkgs))
|
|
}
|
|
if len(pkgs[0].Errors) > 0 {
|
|
t.Fatalf("load %s: %v", dir, pkgs[0].Errors)
|
|
}
|
|
censusLoadCache[dir] = pkgs[0]
|
|
return pkgs[0]
|
|
}
|
|
|
|
// bodyReadSite is one selector expression that reaches a request body.
|
|
type bodyReadSite struct {
|
|
file string // base name
|
|
fn string // enclosing top-level function, "Recv.Method" for methods
|
|
sel string // the member selected
|
|
line int
|
|
}
|
|
|
|
// key is the unit the accounting table is written in: one function's use of
|
|
// one member. Line numbers are not part of it, so an unrelated edit above a
|
|
// reader does not churn the table; a SECOND use of the same member in the
|
|
// same function changes that key's count instead.
|
|
func (s bodyReadSite) key() string { return s.file + "::" + s.fn + "::" + s.sel }
|
|
|
|
var (
|
|
bodyCensusMu sync.Mutex
|
|
bodyCensusCache = map[string][]bodyReadSite{}
|
|
)
|
|
|
|
// collectRequestBodyReads type-checks the package in dir and returns every
|
|
// selector whose RESOLVED object is one of net/http.Request's body readers
|
|
// (BUG-2820).
|
|
//
|
|
// This asks the type checker, not the spelling. The parser it replaces matched
|
|
// the literal `http.Request` in a signature and then names bound to it, so a
|
|
// request reached through a struct field, a context value, a type alias, an
|
|
// embedded field, an import alias, a dot import, a `var` declaration, a
|
|
// call-derived alias, a named result or a range binding was invisible. Here
|
|
// a selector counts when go/types resolves it to the SAME object that
|
|
// looking the member up on net/http.Request yields, which is what every one of
|
|
// those forms has in common. Promotion through embedding resolves to the same
|
|
// field or method, and an alias is the same type, so none needs a special
|
|
// case. A same-named member of another type (a Response's Body, a struct's
|
|
// own Body field) resolves to a different object and is not counted.
|
|
func collectRequestBodyReads(t *testing.T, dir string) []bodyReadSite {
|
|
t.Helper()
|
|
bodyCensusMu.Lock()
|
|
defer bodyCensusMu.Unlock()
|
|
if cached, ok := bodyCensusCache[dir]; ok {
|
|
return cached
|
|
}
|
|
|
|
pkg := loadCensusPackage(t, dir)
|
|
|
|
httpPkg := findImportedPackage(pkg, "net/http")
|
|
if httpPkg == nil {
|
|
t.Fatalf("%s does not import net/http, so there is no request type to look for", dir)
|
|
}
|
|
reqObj := httpPkg.Types.Scope().Lookup("Request")
|
|
if reqObj == nil {
|
|
t.Fatal("net/http.Request not found")
|
|
}
|
|
reqType := reqObj.Type()
|
|
want := map[types.Object]string{}
|
|
for name := range bodyReaderSelectors {
|
|
obj, _, _ := types.LookupFieldOrMethod(reqType, true, nil, name)
|
|
if obj == nil {
|
|
t.Fatalf("net/http.Request has no member %q; the selector list is stale", name)
|
|
}
|
|
want[obj] = name
|
|
}
|
|
|
|
var sites []bodyReadSite
|
|
for _, f := range pkg.Syntax {
|
|
file := filepath.Base(pkg.Fset.File(f.Pos()).Name())
|
|
for _, decl := range f.Decls {
|
|
fn := declName(decl)
|
|
ast.Inspect(decl, func(n ast.Node) bool {
|
|
sel, ok := n.(*ast.SelectorExpr)
|
|
if !ok {
|
|
return true
|
|
}
|
|
selection, ok := pkg.TypesInfo.Selections[sel]
|
|
if !ok {
|
|
return true
|
|
}
|
|
if name, hit := want[selection.Obj()]; hit {
|
|
sites = append(sites, bodyReadSite{
|
|
file: file, fn: fn, sel: name,
|
|
line: pkg.Fset.Position(sel.Pos()).Line,
|
|
})
|
|
}
|
|
return true
|
|
})
|
|
}
|
|
}
|
|
sort.Slice(sites, func(i, j int) bool {
|
|
if sites[i].file != sites[j].file {
|
|
return sites[i].file < sites[j].file
|
|
}
|
|
return sites[i].line < sites[j].line
|
|
})
|
|
bodyCensusCache[dir] = sites
|
|
return sites
|
|
}
|
|
|
|
func findImportedPackage(pkg *packages.Package, path string) *packages.Package {
|
|
seen := map[*packages.Package]bool{}
|
|
var walk func(*packages.Package) *packages.Package
|
|
walk = func(p *packages.Package) *packages.Package {
|
|
if seen[p] {
|
|
return nil
|
|
}
|
|
seen[p] = true
|
|
if p.PkgPath == path {
|
|
return p
|
|
}
|
|
for _, imp := range p.Imports {
|
|
if found := walk(imp); found != nil {
|
|
return found
|
|
}
|
|
}
|
|
return nil
|
|
}
|
|
return walk(pkg)
|
|
}
|
|
|
|
// declName names a top-level declaration: "Func", "Recv.Method", or "var"/
|
|
// "const"/"type" for a GenDecl (a reader inside a package-level initializer
|
|
// is still a reader, and still has to be accounted for).
|
|
func declName(d ast.Decl) string {
|
|
switch x := d.(type) {
|
|
case *ast.FuncDecl:
|
|
if x.Recv != nil && len(x.Recv.List) > 0 {
|
|
recv := x.Recv.List[0].Type
|
|
if star, ok := recv.(*ast.StarExpr); ok {
|
|
recv = star.X
|
|
}
|
|
if idx, ok := recv.(*ast.IndexExpr); ok {
|
|
recv = idx.X
|
|
}
|
|
if id, ok := recv.(*ast.Ident); ok {
|
|
return id.Name + "." + x.Name.Name
|
|
}
|
|
}
|
|
return x.Name.Name
|
|
case *ast.GenDecl:
|
|
return x.Tok.String()
|
|
}
|
|
return "?"
|
|
}
|
|
|
|
// TestBodyReaderCensusSeesTheFormsTheParserMissed is the census's control: a
|
|
// fixture package with one function per form the old parser could not see, and
|
|
// three that must NOT count. A census that went blind to any form fails here,
|
|
// by name, rather than by a count drifting in the production table.
|
|
func TestBodyReaderCensusSeesTheFormsTheParserMissed(t *testing.T) {
|
|
sites := collectRequestBodyReads(t, filepath.Join("testdata", "bodyreaders"))
|
|
found := map[string]int{}
|
|
for _, s := range sites {
|
|
found[s.fn]++
|
|
}
|
|
for _, fn := range []string{
|
|
// The forms the parser could not see (BUG-2820).
|
|
"StructField", "ContextValue", "TypeAlias", "EmbeddedField", "EmbeddedMethod",
|
|
"ImportAlias", "DotImport", "VarDecl", "CallDerived", "NamedResult",
|
|
"RangeBinding", "MethodValue",
|
|
// The forms the parser's own controls pinned.
|
|
"DirectParam", "ClosureParam", "LocalAlias", "ValueCopy", "ClosureValueParam",
|
|
"ReassignedWithContext", "MixedShortDecl", "InsideSwitch", "AfterIfInitShadow",
|
|
} {
|
|
if found[fn] != 1 {
|
|
t.Errorf("%s: census counted %d body readers, want 1", fn, found[fn])
|
|
}
|
|
}
|
|
for _, fn := range []string{"NotARequest", "HeaderOnly", "ResponseBody",
|
|
"ShadowedByClosureParam", "ReboundInNestedBlock", "CommentOnly"} {
|
|
if found[fn] != 0 {
|
|
t.Errorf("%s is not a request body reader, but the census counted %d", fn, found[fn])
|
|
}
|
|
}
|
|
}
|
|
|
|
func sitesByKey(sites []bodyReadSite) map[string]int {
|
|
out := map[string]int{}
|
|
for _, s := range sites {
|
|
out[s.key()]++
|
|
}
|
|
return out
|
|
}
|
|
|
|
// TestEveryRequestBodyReaderIsAccountedFor is the completeness claim for the
|
|
// request-body rule (BUG-2803): a JSON body reaches the store through
|
|
// decodeJSON / decodeJSONWithLimit, which refuse a decoded NUL and apply the
|
|
// size cap, so every OTHER reader of a request body has to be a deliberate,
|
|
// written-down decision.
|
|
//
|
|
// Since BUG-2820 the readers come from the TYPE CHECKER
|
|
// (collectRequestBodyReads), not a parser. The parser matched the spelling
|
|
// `http.Request` and names bound to it, so a request reached through a struct
|
|
// field, a context value, a type alias or an embedded field was invisible, and
|
|
// its accounting was per FILE with a reader count to stop an entry absorbing a
|
|
// new reader. Both workarounds are gone: a reader is found wherever its type
|
|
// says it is one, and the table below is per CALL SITE, keyed by
|
|
// file::function::member, with the number of uses of that member in that
|
|
// function. A new reader anywhere, including a second one in an accounted
|
|
// function, changes the table and fails here until someone writes it down.
|
|
//
|
|
// Run against the tree when this was built, the type checker found exactly the
|
|
// readers the parser had found, file for file (measured, not assumed): the
|
|
// blind spots were real but nothing in the package was using them yet. The
|
|
// fixture control, TestBodyReaderCensusSeesTheFormsTheParserMissed, is what
|
|
// shows the census would see one.
|
|
//
|
|
// It asserts BOTH directions: an unaccounted site fails, and an accounted site
|
|
// that no longer exists fails, so the table cannot rot into stale excuses.
|
|
func TestEveryRequestBodyReaderIsAccountedFor(t *testing.T) {
|
|
t.Parallel()
|
|
// WHY each file reads a body. Every site below must name a file here.
|
|
bodyReaderFileWhy := map[string]string{
|
|
"middleware_request_text.go": "the chokepoint itself: readBodyForDecode reads the body under the caller's cap so bodyDecodesNUL can scan it",
|
|
"handlers_import_bundle.go": "tar.gz bundle import — streams the body through gzip, and its pad-export.json is checked with bodyDecodesNUL before ImportWorkspace",
|
|
"handlers_attachments.go": "multipart upload — the body is binary blob content, not text, and must NOT be scanned for text validity",
|
|
"app_api_attachments.go": "app upload (TASK-3401 U6c) — a raw binary body with a required Content-Length, handed to appstore.StageUpload, which bounds it with a LimitReader at the declared size plus one; blob content, never JSON",
|
|
"artifact_import.go": "raw artifact TEXT (not JSON) — checked with bindableText, the same predicate ValidatePath and ValidateQuery apply",
|
|
"handlers_cloud.go": "bodyHasCloudSecret PEEKS at the body and restores the first 64 KiB of it — a larger body loses its tail, a bound that file documents and accepts; the real decode still happens through decodeJSON downstream",
|
|
"middleware_mcp_audit.go": "audit capture — parses the body ITSELF and binds the decoded method / params.name to mcp_audit_log.tool_name, so it is a second READER, not a pass-through. That the MCP dispatcher decodes the body again is true and says nothing about what this middleware persists — the earlier rationale here made exactly that mistake and certified it safe (codex round 20). parseMCPRequestBody now runs both caller-derived returns through sanitiseStoredText",
|
|
"middleware_mcp_body_limit.go": "the MCP mounts' body cap (BUG-3534). It decodes nothing and stores nothing: a declared length is compared against the limit without a read, and a chunked body is read up to limit+1 bytes only to measure it, then replayed byte for byte to the transport (or refused 413). It exists because the transport (mcp-go) reads the whole body itself and has no cap, which this census cannot see because that read is in a dependency",
|
|
"mcp_stream_keepalive.go": "the MCP stream admission check (BUG-3835) reads a POST body, already bounded by limitMCPBody, only to learn whether it names subscriptions/listen, decoded as mcp-go decodes it; a fixed-size peek would let a listen whose method follows a large params skip the budget. It stores nothing and echoes nothing, and replays every byte (or the same read error) to the transport. It reads only a POST whose Mcp-Protocol-Version header is a modern version the mount's transport serves, the only POST mcp-go may serve as a listen, so while pad serves the legacy versions no POST is read; such a POST would pay a second whole-body buffer, ruled acceptable by the lead on BUG-3835",
|
|
"middleware_mcp_scope.go": "the /mcp insufficient_scope pre-check (TASK-2308) PEEKS at up to 1 MiB of a POST body, decodes only method, params.name and the exact arguments.action, and restores every byte for mcp-go. The decoded values are compared against the catalog's action table and NOTHING is stored or echoed, so the NUL rule and the size cap have nothing to protect here; a body past the bound is passed on unread",
|
|
"handlers_tokens.go": "guards on r.Body != nil && r.ContentLength != 0, then decodes THROUGH decodeJSON — so the body is read by the chokepoint, which applies the cap and the NUL rule. The earlier reason here said it never reads the body, which was simply false (codex round 29): a wrong reason in this list is the same defect as a missing entry, since both let a reader pass as reviewed",
|
|
"app_tokens.go": "the install-client token rules (TASK-3394) read the same FORM-encoded OAuth body handleOAuthToken/handleOAuthIntrospect read, behind ValidateFormBody (BUG-2811), to classify the client exactly as fosite will",
|
|
"app_delegated_consent.go": "the delegated app sign-in (TASK-3399) is the OAuth authorize/decide doors' form: appSignInAvailable parses it for client_id and decideAppConsent reads app_access, both on routes wrapped in ValidateFormBody (BUG-2811) like the rest of handlers_oauth.go; a form-encoded body, never JSON",
|
|
"handlers_oauth.go": "the OAuth handlers read FORM-encoded bodies (r.Form/FormValue). Every POST route they serve is wrapped in ValidateFormBody (BUG-2811), which applies bindableText to the body before these reads. The transport rules cover the query half of r.Form; ValidateFormBody covers the body half",
|
|
"middleware_form_body.go": "the form-body chokepoint itself: ValidateFormBody reads a form-encoded body up to net/http's own ParseForm cap, checks it with validQueryText, and hands the same bytes (or the same read error) back to the handler (BUG-2811)",
|
|
"handlers_watches.go": "guards on r.Body != nil && r.ContentLength != 0, then decodes THROUGH decodeJSON — the closing-round-4 fix for the chunked-body drop; the one reader expression is the nil check itself, and the body bytes flow through the chokepoint",
|
|
"handlers_item_lease.go": "guards on r.Body != nil && r.ContentLength != 0, then decodes THROUGH decodeJSON — same shape as handlers_watches.go; the one reader expression is the nil check, and the body (optional holder/ttl_seconds) flows through the chokepoint, so the caller-text holder gets BUG-2803's NUL refusal before it can reach items.lease_holder",
|
|
"import_read_deadline.go": "a PASS-THROUGH, not a reader: it replaces r.Body with a wrapper whose Read arms the connection deadline and delegates, byte for byte, to the original body (BUG-3184). Nothing in the file inspects the bytes; the import route's two decoders (decodeJSONWithLimit, and the bundle path's bodyDecodesNUL on pad-export.json) read them afterwards, unchanged",
|
|
}
|
|
|
|
// Every call site, counted. MEASURED from the census, not written from
|
|
// expectation.
|
|
bodyReaderSites := map[string]int{
|
|
"artifact_import.go::parseArtifactRequest::Body": 1,
|
|
"handlers_attachments.go::Server.handleUploadAttachment::Body": 2,
|
|
"app_api_attachments.go::Server.appUploadAttachment::Body": 1,
|
|
"handlers_attachments.go::Server.handleUploadAttachment::MultipartForm": 2,
|
|
"handlers_attachments.go::Server.handleUploadAttachment::ParseMultipartForm": 1,
|
|
"handlers_attachments.go::Server.handleUploadAttachment::FormFile": 1,
|
|
"handlers_attachments.go::multipartValues::MultipartForm": 2,
|
|
"handlers_cloud.go::hasCloudSecretMarker::Body": 1,
|
|
"handlers_cloud.go::bodyHasCloudSecret::Body": 2,
|
|
// BUG-3475: +2 for the transport tracker, which wraps the raw body
|
|
// BENEATH the MaxBytesReader, so every read still passes the cap.
|
|
"handlers_import_bundle.go::Server.handleImportWorkspaceBundle::Body": 5,
|
|
"handlers_item_lease.go::Server.resolveLeaseRequest::Body": 1,
|
|
"middleware_form_body.go::ValidateFormBody::Body": 6,
|
|
"handlers_oauth.go::Server.handleOAuthAuthorize::ParseForm": 1,
|
|
"handlers_oauth.go::Server.handleOAuthAuthorizeDecide::ParseForm": 1,
|
|
"handlers_oauth.go::Server.handleOAuthAuthorizeDecide::FormValue": 1,
|
|
"handlers_oauth.go::Server.parseConsentPayload::FormValue": 4,
|
|
"handlers_oauth.go::Server.parseConsentPayload::PostForm": 2,
|
|
// TASK-3394: parsed as fosite parses it (multipart included), so the
|
|
// client classified for the install-client rules is fosite's client.
|
|
"handlers_oauth.go::Server.handleOAuthToken::ParseMultipartForm": 1,
|
|
"app_delegated_consent.go::Server.appSignInAvailable::ParseForm": 1,
|
|
"app_delegated_consent.go::Server.decideAppConsent::PostForm": 1,
|
|
// TASK-3399 adds four: the install client's grant type and the code
|
|
// or refresh token its ownership is checked by before fosite runs.
|
|
"handlers_oauth.go::Server.handleOAuthToken::PostForm": 5,
|
|
"handlers_oauth.go::Server.handleOAuthIntrospect::ParseForm": 1,
|
|
"app_tokens.go::Server.prepareInstallTokenRequest::PostForm": 4,
|
|
"app_tokens.go::requestClientID::PostForm": 1,
|
|
"handlers_oauth.go::Server.handleOAuthRevoke::ParseForm": 1,
|
|
"handlers_oauth.go::Server.handleOAuthRevoke::PostForm": 1,
|
|
"handlers_oauth.go::Server.validateConsentCSRFToken::FormValue": 1,
|
|
"handlers_tokens.go::Server.handleRotateUserToken::Body": 1,
|
|
"handlers_watches.go::Server.handleCreateWatch::Body": 1,
|
|
"import_read_deadline.go::Server.withImportReadDeadline::Body": 2,
|
|
"middleware_mcp_audit.go::Server.MCPAuditLog::Body": 5,
|
|
"middleware_mcp_audit.go::Server.emitMCPAuditDenied::Body": 2,
|
|
"middleware_mcp_scope.go::Server.MCPInsufficientScope::Body": 5,
|
|
"middleware_mcp_body_limit.go::Server.limitMCPBody::Body": 7,
|
|
"mcp_stream_keepalive.go::mcpLongLivedRequest::Body": 7,
|
|
"middleware_request_text.go::readBodyForDecode::Body": 4,
|
|
}
|
|
|
|
sites := collectRequestBodyReads(t, ".")
|
|
// The census must have found something: a load that silently resolved no
|
|
// request type would pass the comparisons below with an empty table.
|
|
if len(sites) < 20 {
|
|
t.Fatalf("census found only %d body readers in the package; the load or the type lookup is broken", len(sites))
|
|
}
|
|
got := sitesByKey(sites)
|
|
for key, n := range got {
|
|
want, ok := bodyReaderSites[key]
|
|
switch {
|
|
case !ok:
|
|
t.Errorf("%s reads the request body %d time(s) and is not accounted for. A body carrying JSON must go "+
|
|
"through decodeJSON/decodeJSONWithLimit, which apply BUG-2803's NUL refusal and the size cap. If this "+
|
|
"reader is legitimate, add it to bodyReaderSites (and its file to bodyReaderFileWhy) WITH the reason.", key, n)
|
|
case n != want:
|
|
t.Errorf("%s has %d body reader(s) but was reviewed with %d. A reader was added or removed in that "+
|
|
"function since it was justified: re-read it and update the count.", key, n, want)
|
|
}
|
|
file := strings.SplitN(key, "::", 2)[0]
|
|
if _, ok := bodyReaderFileWhy[file]; !ok {
|
|
t.Errorf("%s has no reason in bodyReaderFileWhy", file)
|
|
}
|
|
}
|
|
for key := range bodyReaderSites {
|
|
if _, ok := got[key]; !ok {
|
|
t.Errorf("%s is accounted for but no longer reads a request body; remove the entry so it cannot "+
|
|
"cover a future reader", key)
|
|
}
|
|
}
|
|
for file := range bodyReaderFileWhy {
|
|
live := false
|
|
for key := range got {
|
|
if strings.HasPrefix(key, file+"::") {
|
|
live = true
|
|
}
|
|
}
|
|
if !live {
|
|
t.Errorf("bodyReaderFileWhy names %s, which no longer reads a request body; remove it", file)
|
|
}
|
|
}
|
|
}
|
|
|
|
// requestHandoff is a call that passes a request to a function OUTSIDE this
|
|
// module, which may read the body where no selector here does.
|
|
type requestHandoff struct {
|
|
file, fn, callee string
|
|
line int
|
|
}
|
|
|
|
func (h requestHandoff) key() string { return h.file + "::" + h.fn + "::" + h.callee }
|
|
|
|
// collectRequestHandoffs returns every call in the package in dir that passes a
|
|
// value of type *net/http.Request or net/http.Request to a function or method
|
|
// declared outside this module. The body-reader census sees selectors; a
|
|
// library that reads the body on our behalf (fosite parsing an OAuth form) has
|
|
// no selector here, so it is counted as a HANDOFF and classified by callee.
|
|
//
|
|
// What it cannot see, stated: a call through an INTERFACE (http.Handler's
|
|
// ServeHTTP, which is how the router and every middleware chain dispatch)
|
|
// resolves to the interface method, not to whatever implements it. Those are
|
|
// classified as dispatch below, not as readers.
|
|
func collectRequestHandoffs(t *testing.T, dir string) []requestHandoff {
|
|
t.Helper()
|
|
pkg := loadCensusPackage(t, dir)
|
|
module := pkg.Module
|
|
if module == nil {
|
|
t.Fatalf("%s has no module information", dir)
|
|
}
|
|
reqType := findImportedPackage(pkg, "net/http").Types.Scope().Lookup("Request").Type()
|
|
|
|
var out []requestHandoff
|
|
for _, f := range pkg.Syntax {
|
|
file := filepath.Base(pkg.Fset.File(f.Pos()).Name())
|
|
for _, decl := range f.Decls {
|
|
fn := declName(decl)
|
|
ast.Inspect(decl, func(n ast.Node) bool {
|
|
call, ok := n.(*ast.CallExpr)
|
|
if !ok {
|
|
return true
|
|
}
|
|
var id *ast.Ident
|
|
switch x := call.Fun.(type) {
|
|
case *ast.SelectorExpr:
|
|
id = x.Sel
|
|
case *ast.Ident:
|
|
id = x
|
|
}
|
|
if id == nil {
|
|
return true
|
|
}
|
|
callee := pkg.TypesInfo.ObjectOf(id)
|
|
if callee == nil || callee.Pkg() == nil {
|
|
return true
|
|
}
|
|
path := callee.Pkg().Path()
|
|
if path == module.Path || strings.HasPrefix(path, module.Path+"/") {
|
|
return true
|
|
}
|
|
for _, a := range call.Args {
|
|
at := pkg.TypesInfo.TypeOf(a)
|
|
if ptr, ok := at.(*types.Pointer); ok {
|
|
at = ptr.Elem()
|
|
}
|
|
if at != nil && types.Identical(at, reqType) {
|
|
out = append(out, requestHandoff{file: file, fn: fn,
|
|
callee: qualifiedCallee(callee), line: pkg.Fset.Position(call.Pos()).Line})
|
|
break
|
|
}
|
|
}
|
|
return true
|
|
})
|
|
}
|
|
}
|
|
return out
|
|
}
|
|
|
|
// TestEveryRequestHandoffIsClassified extends the accounting past this
|
|
// package's own selectors (BUG-2820): a request handed to third-party code
|
|
// must be classified by what that code does with the body. A new callee fails
|
|
// here until someone decides. Body-READING callees are pinned per call site,
|
|
// like the readers above; the rest are pinned by callee.
|
|
func TestEveryRequestHandoffIsClassified(t *testing.T) {
|
|
notReaders := map[string]string{
|
|
"github.com/go-chi/chi/v5.URLParam": "reads the route context, not the body",
|
|
"net/http.NotFound": "writes a 404; reads nothing of the request body",
|
|
"net/http.Redirect": "reads the URL to build the Location header",
|
|
"net/http.ServeContent": "reads range and conditional HEADERS",
|
|
"github.com/gorilla/websocket.IsWebSocketUpgrade": "reads headers",
|
|
"github.com/gorilla/websocket.Upgrader.Upgrade": "hijacks the connection after checking headers; a GET upgrade has no body",
|
|
"net/http.Handler.ServeHTTP": "interface DISPATCH: the router and middleware chains passing the request on to the next handler, whose own reads are counted where they are written. Which handler runs is not resolvable statically",
|
|
"github.com/go-chi/chi/v5.Mux.ServeHTTP": "dispatch into the router, as above",
|
|
// TASK-3452: the Cloud tutorial catalog and poster fetch. The request is
|
|
// OUTBOUND, built by this package with no body; an inbound server
|
|
// request cannot be passed here at all (Client.Do refuses one with
|
|
// RequestURI set), so no caller's body can reach this callee.
|
|
"net/http.Client.Do": "sends an outbound request this package built (no body); never receives an inbound request",
|
|
}
|
|
// Body readers by delegation: fosite parses the OAuth form body itself.
|
|
// Each POST handler here sits behind ValidateFormBody (BUG-2811), so the
|
|
// body fosite parses has already been checked; counted per call site
|
|
// rather than hidden behind the handler file's own ParseForm calls.
|
|
readers := map[string]int{
|
|
"handlers_oauth.go::Server.handleOAuthAuthorize::github.com/ory/fosite.OAuth2Provider.NewAuthorizeRequest": 1,
|
|
"handlers_oauth.go::Server.handleOAuthAuthorizeDecide::github.com/ory/fosite.OAuth2Provider.NewAuthorizeRequest": 1,
|
|
"handlers_oauth.go::Server.handleOAuthToken::github.com/ory/fosite.OAuth2Provider.NewAccessRequest": 1,
|
|
"handlers_oauth.go::Server.handleOAuthRevoke::github.com/ory/fosite.OAuth2Provider.NewRevocationRequest": 1,
|
|
"handlers_oauth.go::Server.handleOAuthIntrospect::github.com/ory/fosite.OAuth2Provider.NewIntrospectionRequest": 1,
|
|
// TASK-3394: fosite's own bearer extraction (header, then the form's
|
|
// access_token, parsing multipart), so the caller is read exactly as
|
|
// NewIntrospectionRequest will. Shared since BUG-3398 by the
|
|
// install-caller refusal and the token-ownership check.
|
|
"app_tokens.go::Server.introspectionCallerClientID::github.com/ory/fosite.AccessTokenFromRequest": 1,
|
|
}
|
|
|
|
handoffs := collectRequestHandoffs(t, ".")
|
|
if len(handoffs) < 10 {
|
|
t.Fatalf("found only %d handoffs; the census is broken", len(handoffs))
|
|
}
|
|
got := map[string]int{}
|
|
seenCallee := map[string]bool{}
|
|
for _, h := range handoffs {
|
|
seenCallee[h.callee] = true
|
|
if _, ok := notReaders[h.callee]; ok {
|
|
continue
|
|
}
|
|
got[h.key()]++
|
|
}
|
|
for key, n := range got {
|
|
if want, ok := readers[key]; !ok {
|
|
t.Errorf("%s hands a request to third-party code %d time(s) and is not classified. Decide whether that "+
|
|
"callee reads the body: if not, add the callee to notReaders with the reason; if it does, add this site "+
|
|
"to readers and make sure the body it reads is covered", key, n)
|
|
} else if n != want {
|
|
t.Errorf("%s: %d handoff(s), classified with %d", key, n, want)
|
|
}
|
|
}
|
|
for key := range readers {
|
|
if _, ok := got[key]; !ok {
|
|
t.Errorf("%s is classified as a reader but no longer exists; remove it", key)
|
|
}
|
|
}
|
|
for callee := range notReaders {
|
|
if !seenCallee[callee] {
|
|
t.Errorf("notReaders names %s, which nothing calls with a request any more; remove it", callee)
|
|
}
|
|
}
|
|
}
|
|
|
|
// qualifiedCallee names a function "pkg/path.Name" and a method
|
|
// "pkg/path.Recv.Name", so an interface method (http.Handler's ServeHTTP) and a
|
|
// concrete one (chi's Mux) are told apart.
|
|
func qualifiedCallee(obj types.Object) string {
|
|
name := obj.Pkg().Path() + "."
|
|
if fn, ok := obj.(*types.Func); ok {
|
|
if sig, ok := fn.Type().(*types.Signature); ok && sig.Recv() != nil {
|
|
rt := sig.Recv().Type()
|
|
if ptr, ok := rt.(*types.Pointer); ok {
|
|
rt = ptr.Elem()
|
|
}
|
|
if named, ok := rt.(*types.Named); ok {
|
|
name += named.Obj().Name() + "."
|
|
}
|
|
}
|
|
}
|
|
return name + obj.Name()
|
|
}
|