fix(server,store,items): coerce field values to their declared types server-side (BUG-2850)

The write doors disagreed about what `key=value` means. The CLI has coerced
by schema type since BUG-1125, and local stdio MCP inherits that by shelling
out to the binary — but the remote /mcp transport builds its field map in
ingestFieldKVP with `dst[key] = val`, so every value arrives as a string.
validateFieldType then correctly refuses a string for a declared number or
json field, and the net effect was that an MCP agent on that transport could
not write those fields AT ALL: every attempt a 400, not a mis-typed value.

Measured before writing anything (repro table on BUG-2850's trail): CLI and
stdio MCP store 42 and an array; the HTTP door 400s on both; an UNDECLARED
key is stored as a string on every door.

items.CoerceFields(fields, schema) converts strings to the declared type —
number via ParseFloat (NaN/±Inf refused, because json.Marshal cannot encode
them and the ignored downstream error would silently drop the whole payload),
json/multi_select via Unmarshal, checkbox via ParseBool — and is applied
immediately before every Validate* call.

Three deliberate non-behaviours, each with a test:
- A value that will not parse is left as the string for the validator, so the
  existing "must be a number" error still fires. Coercion invents no error
  path, and cannot turn a currently-PASSING write into a failure.
- Non-string values pass through untouched; an int stays an int.
- Text-typed fields holding "42" stay strings. Coercing anything that parses
  would retype real data while fixing the bug.

Not folded into ValidateFields, though that would be the single call site: a
function named Validate that mutates its input is a trap, and two callers
re-marshal the map they pass.

THE POPULATION IS 8 CALL SITES, and finding them took two sweeps. The first
was scoped to internal/server and found 7; the copy path validates in
internal/store (items_cross_workspace_copy.go), which only a repo-wide sweep
sees. The preflight and the store-side copy now carry cross-references to
each other: the preflight exists to PREDICT the copy, they live in different
packages, and that is exactly how they would drift unnoticed.

The undeclared-key half of BUG-2850 is untouched and marked as a decision
point in CoerceFields — refuse/warn/keep is with Dave. A test pins today's
keep behaviour so the ruling lands as a deliberate change.

The CLI's parseFieldFlag deliberately STAYS: it is why two of four doors are
correct today, and removing it alongside its replacement would put all four
at risk of one mistake. Retiring it is a follow-up.

Claude-Session: https://claude.ai/code/session_011Q4b1iHtJtSyMs7BA2ySxo
This commit is contained in:
xarmian
2026-09-02 23:30:26 +00:00
parent 70099c2724
commit ae793e6fa6
7 changed files with 427 additions and 0 deletions
+102
View File
@@ -0,0 +1,102 @@
package items
import (
"encoding/json"
"math"
"strconv"
"github.com/PerpetualSoftware/pad/internal/models"
)
// CoerceFields converts STRING field values to the type the collection schema
// declares, so every write door stores the same native type for the same input
// (BUG-2850).
//
// The doors did not agree. The CLI has coerced by schema type since BUG-1125
// (cmd/pad/cmd_item.go::parseFieldFlag), and local stdio MCP inherits that by
// shelling out to the binary — but the remote /mcp transport builds its field
// map in ingestFieldKVP, which does `dst[key] = val` unconditionally, so every
// value arrives as a string. validateFieldType then correctly refuses a string
// for a declared number/json field, and the net effect was that an MCP agent on
// that transport could not write those fields AT ALL: every attempt a 400.
// Typing belongs to the server, keyed on the schema, so the doors cannot drift
// again.
//
// WHAT THIS DELIBERATELY DOES NOT DO:
//
// - It does not report errors. A value that will not coerce is left as the
// string and handed to the validator, which already produces the right
// message ("field %q must be a number"). Coercion never invents an error
// path, which is also why it cannot turn a currently-PASSING write into a
// failure — the only inputs whose behaviour changes are ones that are
// 400ing today.
// - It does not touch non-string values. A caller already sending 42 keeps
// sending 42; this is not a re-typing pass over well-formed input.
// - It does not touch keys the schema does not declare. Those are stored as
// given, silently, which is the OTHER half of BUG-2850 — see the decision
// point below.
//
// It returns a new map rather than mutating in place: two of the call sites
// re-marshal the map they pass, and a function that quietly rewrote their input
// would change what gets stored from behind a name that does not say so.
func CoerceFields(fields map[string]any, schema models.CollectionSchema) map[string]any {
if len(fields) == 0 {
return fields
}
byKey := make(map[string]models.FieldDef, len(schema.Fields))
for i := range schema.Fields {
byKey[schema.Fields[i].Key] = schema.Fields[i]
}
out := make(map[string]any, len(fields))
for k, v := range fields {
def, declared := byKey[k]
if !declared {
// DECISION POINT — BUG-2850 undeclared-key disposition (with Dave:
// refuse / warn / keep). Today's behaviour is KEEP, which is what
// silently stores materials_cost="42" on a collection that never
// declared it. Refuse becomes a returned issue here; warn becomes a
// collected key. Deliberately isolated to this branch so the choice
// drops in without touching the coercion above it.
out[k] = v
continue
}
out[k] = coerceValue(def, v)
}
return out
}
// coerceValue converts one string value to its declared type, or returns it
// unchanged when it is not a string or will not parse.
func coerceValue(def models.FieldDef, v any) any {
s, ok := v.(string)
if !ok {
return v
}
switch def.Type {
case "json", "multi_select":
var parsed any
if err := json.Unmarshal([]byte(s), &parsed); err == nil {
return parsed
}
case "number":
if f, err := strconv.ParseFloat(s, 64); err == nil {
// Reject NaN / ±Inf rather than storing them: encoding/json cannot
// marshal either, and the downstream json.Marshal(fields) error is
// ignored, so a non-finite float silently drops the ENTIRE fields
// payload. Falling through leaves the string for the validator,
// which says "must be a number" — the same reasoning as the CLI's
// guard in parseFieldFlag (BUG-1125).
if !math.IsNaN(f) && !math.IsInf(f, 0) {
return f
}
}
case "checkbox":
if b, err := strconv.ParseBool(s); err == nil {
return b
}
}
// text, url, select, date, relation — a string is already the right type.
// Anything that did not parse above also lands here, on purpose.
return s
}
+138
View File
@@ -0,0 +1,138 @@
package items
import (
"testing"
"github.com/PerpetualSoftware/pad/internal/models"
)
// The schema every case below is coerced against.
func coerceSchema() models.CollectionSchema {
return models.CollectionSchema{Fields: []models.FieldDef{
{Key: "cost", Type: "number"},
{Key: "spec", Type: "json"},
{Key: "tags", Type: "multi_select"},
{Key: "done", Type: "checkbox"},
{Key: "note", Type: "text"},
{Key: "due", Type: "date"},
}}
}
// The defect this exists for: the remote /mcp door builds its field map with
// `dst[key] = val`, so a declared number field arrives as the STRING "42" and
// validateFieldType refuses it — the field is unwritable on that transport
// (BUG-2850). Every assertion here is about the native TYPE that reaches the
// store, because "the write succeeded" is what the CLI door already did.
func TestCoerceFieldsTypesDeclaredStrings(t *testing.T) {
in := map[string]any{
"cost": "42",
"spec": `[{"name":"a"}]`,
"tags": `["x","y"]`,
"done": "true",
}
out := CoerceFields(in, coerceSchema())
if got, ok := out["cost"].(float64); !ok || got != 42 {
t.Fatalf("cost: want float64(42), got %[1]T(%[1]v)", out["cost"])
}
if _, ok := out["spec"].([]any); !ok {
t.Fatalf("spec: want []any, got %[1]T(%[1]v)", out["spec"])
}
if _, ok := out["tags"].([]any); !ok {
t.Fatalf("tags: want []any, got %[1]T(%[1]v)", out["tags"])
}
if got, ok := out["done"].(bool); !ok || !got {
t.Fatalf("done: want bool(true), got %[1]T(%[1]v)", out["done"])
}
}
// Types that are already strings must stay strings. A "coerce everything that
// parses" implementation would turn a text field holding "42" into a number and
// corrupt data that was never broken — this is the guard against fixing the bug
// by over-reaching.
func TestCoerceFieldsLeavesStringTypedFieldsAlone(t *testing.T) {
out := CoerceFields(map[string]any{
"note": "42",
"due": "2026-09-02",
}, coerceSchema())
if got, ok := out["note"].(string); !ok || got != "42" {
t.Fatalf("note: want string(\"42\"), got %[1]T(%[1]v)", out["note"])
}
if got, ok := out["due"].(string); !ok || got != "2026-09-02" {
t.Fatalf("due: want the date string, got %[1]T(%[1]v)", out["due"])
}
}
// A value that will not parse is handed to the validator UNCHANGED, so the
// existing "must be a number" error still fires. Coercion must not invent an
// error path, and must not swallow a bad value into something plausible.
func TestCoerceFieldsLeavesUnparseableValuesForTheValidator(t *testing.T) {
out := CoerceFields(map[string]any{
"cost": "not-a-number",
"spec": "{definitely not json",
}, coerceSchema())
if got, ok := out["cost"].(string); !ok || got != "not-a-number" {
t.Fatalf("cost: want the original string, got %[1]T(%[1]v)", out["cost"])
}
if err := ValidateFields(out, coerceSchema()); err == nil {
t.Fatal("expected the validator to still refuse the un-coercible value")
}
}
// NaN and ±Inf parse as floats and then cannot be marshalled: encoding/json
// fails, the downstream json.Marshal(fields) error is ignored, and the ENTIRE
// fields payload is silently dropped. They must fall through as strings so the
// validator refuses them loudly instead.
func TestCoerceFieldsRefusesNonFiniteNumbers(t *testing.T) {
for _, raw := range []string{"NaN", "Inf", "-Inf", "+Inf"} {
out := CoerceFields(map[string]any{"cost": raw}, coerceSchema())
if f, ok := out["cost"].(float64); ok {
t.Fatalf("%q was coerced to float64(%v); non-finite values must stay strings", raw, f)
}
}
}
// Non-string values pass through untouched. This is not a re-typing pass over
// well-formed input — a caller already sending 42 keeps sending 42, and an
// int must not become a float64 behind their back.
func TestCoerceFieldsPassesNonStringsThrough(t *testing.T) {
in := map[string]any{"cost": 42, "spec": []any{"already", "parsed"}, "done": true}
out := CoerceFields(in, coerceSchema())
if got, ok := out["cost"].(int); !ok || got != 42 {
t.Fatalf("cost: want int(42) untouched, got %[1]T(%[1]v)", out["cost"])
}
if _, ok := out["spec"].([]any); !ok {
t.Fatalf("spec: want []any untouched, got %T", out["spec"])
}
if got, ok := out["done"].(bool); !ok || !got {
t.Fatalf("done: want bool(true) untouched, got %[1]T(%[1]v)", out["done"])
}
}
// Keys the schema does not declare are left exactly as they arrived. This is
// TODAY'S behaviour and the other half of BUG-2850 — the disposition (refuse /
// warn / keep) is a product decision still open. The test pins what the code
// does so the decision, when it lands, is a deliberate change to a stated
// behaviour rather than a silent one.
func TestCoerceFieldsLeavesUndeclaredKeysUntouched(t *testing.T) {
out := CoerceFields(map[string]any{"materials_cost": "42"}, coerceSchema())
if got, ok := out["materials_cost"].(string); !ok || got != "42" {
t.Fatalf("undeclared key: want the string untouched, got %[1]T(%[1]v)", out["materials_cost"])
}
}
// CoerceFields returns a NEW map. Two call sites re-marshal the map they pass
// in, so a function that quietly rewrote its argument would change what gets
// stored from behind a name that does not say so.
func TestCoerceFieldsDoesNotMutateItsInput(t *testing.T) {
in := map[string]any{"cost": "42"}
_ = CoerceFields(in, coerceSchema())
if got, ok := in["cost"].(string); !ok || got != "42" {
t.Fatalf("input was mutated: cost is now %[1]T(%[1]v)", in["cost"])
}
}
+8
View File
@@ -744,6 +744,8 @@ func (e *itemCreateError) Error() string { return e.message }
// Returns the created item (Ref/Slug populated by the store) or an
// *itemCreateError with a status hint.
func (s *Server) createItemChecked(r *http.Request, workspaceID string, coll *models.Collection, schema models.CollectionSchema, input models.ItemCreate, fieldMap map[string]any, parentValue string) (*models.Item, *itemCreateError) {
// Coerce strings to their declared types before validating (BUG-2850).
fieldMap = items.CoerceFields(fieldMap, schema)
if err := items.ValidateFields(fieldMap, schema); err != nil {
return nil, &itemCreateError{http.StatusBadRequest, "validation_error", err.Error()}
}
@@ -1184,6 +1186,8 @@ func (s *Server) handleUpdateItem(w http.ResponseWriter, r *http.Request) {
return
}
// Coerce strings to their declared types before validating (BUG-2850).
fieldMap = items.CoerceFields(fieldMap, schema)
if err := items.ValidateFields(fieldMap, schema); err != nil {
writeError(w, http.StatusBadRequest, "validation_error", err.Error())
return
@@ -1340,6 +1344,8 @@ func (s *Server) handleUpdateItem(w http.ResponseWriter, r *http.Request) {
return
}
// Coerce strings to their declared types before validating (BUG-2850).
patchMap = items.CoerceFields(patchMap, schema)
if err := items.ValidatePartialFields(patchMap, schema); err != nil {
writeError(w, http.StatusBadRequest, "validation_error", err.Error())
return
@@ -2267,6 +2273,8 @@ func (s *Server) handleMoveItem(w http.ResponseWriter, r *http.Request) {
// `missing_required_fields` code and message shape, because CLI and
// web callers key off it; genuinely invalid VALUES get their own
// `invalid_fields` code rather than being mislabelled as missing.
// Coerce strings to their declared types before validating (BUG-2850).
result.Fields = items.CoerceFields(result.Fields, items.SchemaForMigratedFields(targetSchema))
if issues := items.ValidateFieldsDetailed(result.Fields, items.SchemaForMigratedFields(targetSchema)); len(issues) > 0 {
var missing, invalid []string
for _, iss := range issues {
+4
View File
@@ -495,6 +495,8 @@ func (s *Server) bulkFieldUpdate(r *http.Request, workspaceID string, item *mode
fieldMap[k] = v
}
// Coerce strings to their declared types before validating (BUG-2850).
fieldMap = items.CoerceFields(fieldMap, schema)
if err := items.ValidateFields(fieldMap, schema); err != nil {
return nil, &bulkOpError{message: err.Error(), code: "validation_error"}
}
@@ -677,6 +679,8 @@ func (s *Server) bulkMoveCollection(r *http.Request, workspaceID string, item *m
// against the TARGET schema — MigrateFields validates migrated
// values but an override can smuggle in a value the target schema
// doesn't allow (e.g. a status not in the target's options).
// Coerce strings to their declared types before validating (BUG-2850).
result.Fields = items.CoerceFields(result.Fields, items.SchemaForMigratedFields(targetSchema))
if err := items.ValidateFields(result.Fields, items.SchemaForMigratedFields(targetSchema)); err != nil {
return nil, &bulkOpError{message: err.Error(), code: "validation_error"}
}
@@ -644,6 +644,11 @@ func (s *Server) handleCopyItemPreflight(w http.ResponseWriter, r *http.Request)
// ValidateFieldsDetailed injects any remaining schema defaults into
// `final` in place, so a key that appears only afterwards has no
// origin entry and is reported as "default".
// Coerce strings to their declared types before validating (BUG-2850).
// MUST match the store-side copy (items_cross_workspace_copy.go): the
// preflight exists to PREDICT what the copy does, so a coercion on one
// side only would make it report a field as failing that the copy accepts.
final = items.CoerceFields(final, items.SchemaForMigratedFields(targetSchema))
issues := items.ValidateFieldsDetailed(final, items.SchemaForMigratedFields(targetSchema))
// DR-12's other half: an override whose VALUE is invalid is rejected,
@@ -0,0 +1,165 @@
package server
import (
"encoding/json"
"net/http"
"testing"
"github.com/PerpetualSoftware/pad/internal/models"
)
// BUG-2850 at the HTTP door — the door that was broken.
//
// The remote /mcp transport builds its field map in ingestFieldKVP with
// `dst[key] = val`, so every value reaches this handler as a STRING. Before
// the fix, validateFieldType then refused a string for a declared number or
// json field and the write 400'd: an MCP agent on that transport could not
// write those fields at all. The CLI and local stdio MCP were unaffected —
// they coerce by schema before the request is built — which is why this
// reproduced only against Pad Cloud.
//
// These assert the stored NATIVE TYPE, not that the request succeeded. A test
// that only checked for 201 would pass on an implementation that stored the
// string, which is the shape the reporter described.
func TestItemFieldsCoercedFromStringsAtTheHTTPDoor(t *testing.T) {
t.Parallel()
srv := testServer(t)
sessionToken := bootstrapFirstUser(t, srv, "admin@example.com", "Admin")
rr := doRequestWithCookie(srv, "POST", "/api/v1/workspaces",
map[string]string{"name": "Coercion Test"}, sessionToken)
if rr.Code != http.StatusCreated {
t.Fatalf("create ws: expected 201, got %d: %s", rr.Code, rr.Body.String())
}
var ws models.Workspace
parseJSON(t, rr, &ws)
// A collection whose schema declares the two types the bug made unwritable.
schema := `{"fields":[{"key":"cost","type":"number"},{"key":"spec","type":"json"},{"key":"note","type":"text"}]}`
rr = doRequestWithCookie(srv, "POST", "/api/v1/workspaces/"+ws.Slug+"/collections",
map[string]interface{}{"name": "Jobs", "schema": schema}, sessionToken)
if rr.Code != http.StatusCreated {
t.Fatalf("create collection: expected 201, got %d: %s", rr.Code, rr.Body.String())
}
var coll models.Collection
parseJSON(t, rr, &coll)
// Exactly what ingestFieldKVP produces: every value a string.
rr = doRequestWithHeaders(srv, "POST",
"/api/v1/workspaces/"+ws.Slug+"/collections/"+coll.Slug+"/items",
map[string]interface{}{
"title": "from the remote mcp door",
"fields": `{"cost":"42","spec":"[{\"name\":\"a\"}]","note":"42"}`,
},
map[string]string{"Authorization": "Bearer " + sessionToken},
)
if rr.Code != http.StatusCreated {
t.Fatalf("expected 201, got %d: %s", rr.Code, rr.Body.String())
}
var created models.Item
parseJSON(t, rr, &created)
stored := map[string]any{}
if err := json.Unmarshal([]byte(created.Fields), &stored); err != nil {
t.Fatalf("stored fields are not JSON: %v (%s)", err, created.Fields)
}
if got, ok := stored["cost"].(float64); !ok || got != 42 {
t.Fatalf("cost: want a JSON number, got %[1]T(%[1]v) — stored blob: %s", stored["cost"], created.Fields)
}
if _, ok := stored["spec"].([]any); !ok {
t.Fatalf("spec: want a JSON array, got %[1]T(%[1]v) — stored blob: %s", stored["spec"], created.Fields)
}
// The text field holding "42" must STAY a string. Fixing the bug by
// coercing anything that parses would silently retype real data.
if got, ok := stored["note"].(string); !ok || got != "42" {
t.Fatalf("note: want the string \"42\" untouched, got %[1]T(%[1]v)", stored["note"])
}
}
// The same door on UPDATE, which is a separate call site and would not have
// been covered by the create test above (BUG-2850 wires eight sites).
func TestItemFieldsCoercedFromStringsOnUpdate(t *testing.T) {
t.Parallel()
srv := testServer(t)
sessionToken := bootstrapFirstUser(t, srv, "admin@example.com", "Admin")
rr := doRequestWithCookie(srv, "POST", "/api/v1/workspaces",
map[string]string{"name": "Coercion Update"}, sessionToken)
if rr.Code != http.StatusCreated {
t.Fatalf("create ws: expected 201, got %d: %s", rr.Code, rr.Body.String())
}
var ws models.Workspace
parseJSON(t, rr, &ws)
schema := `{"fields":[{"key":"cost","type":"number"}]}`
rr = doRequestWithCookie(srv, "POST", "/api/v1/workspaces/"+ws.Slug+"/collections",
map[string]interface{}{"name": "Jobs", "schema": schema}, sessionToken)
if rr.Code != http.StatusCreated {
t.Fatalf("create collection: expected 201, got %d: %s", rr.Code, rr.Body.String())
}
var coll models.Collection
parseJSON(t, rr, &coll)
rr = doRequestWithHeaders(srv, "POST",
"/api/v1/workspaces/"+ws.Slug+"/collections/"+coll.Slug+"/items",
map[string]interface{}{"title": "job", "fields": `{}`},
map[string]string{"Authorization": "Bearer " + sessionToken})
if rr.Code != http.StatusCreated {
t.Fatalf("create item: expected 201, got %d: %s", rr.Code, rr.Body.String())
}
var item models.Item
parseJSON(t, rr, &item)
rr = doRequestWithHeaders(srv, "PATCH",
"/api/v1/workspaces/"+ws.Slug+"/items/"+item.Slug,
map[string]interface{}{"fields": `{"cost":"99.5"}`},
map[string]string{"Authorization": "Bearer " + sessionToken})
if rr.Code != http.StatusOK {
t.Fatalf("update: expected 200, got %d: %s", rr.Code, rr.Body.String())
}
var updated models.Item
parseJSON(t, rr, &updated)
stored := map[string]any{}
if err := json.Unmarshal([]byte(updated.Fields), &stored); err != nil {
t.Fatalf("stored fields are not JSON: %v (%s)", err, updated.Fields)
}
if got, ok := stored["cost"].(float64); !ok || got != 99.5 {
t.Fatalf("cost: want a JSON number, got %[1]T(%[1]v) — stored blob: %s", stored["cost"], updated.Fields)
}
}
// A value that will not coerce must still be REFUSED, with the validator's
// existing message. Coercion removes the cases where a correct value arrived
// in the wrong clothes; it must not start accepting wrong values.
func TestUncoercibleFieldValueStillRefused(t *testing.T) {
t.Parallel()
srv := testServer(t)
sessionToken := bootstrapFirstUser(t, srv, "admin@example.com", "Admin")
rr := doRequestWithCookie(srv, "POST", "/api/v1/workspaces",
map[string]string{"name": "Coercion Refusal"}, sessionToken)
if rr.Code != http.StatusCreated {
t.Fatalf("create ws: expected 201, got %d: %s", rr.Code, rr.Body.String())
}
var ws models.Workspace
parseJSON(t, rr, &ws)
schema := `{"fields":[{"key":"cost","type":"number"}]}`
rr = doRequestWithCookie(srv, "POST", "/api/v1/workspaces/"+ws.Slug+"/collections",
map[string]interface{}{"name": "Jobs", "schema": schema}, sessionToken)
if rr.Code != http.StatusCreated {
t.Fatalf("create collection: expected 201, got %d: %s", rr.Code, rr.Body.String())
}
var coll models.Collection
parseJSON(t, rr, &coll)
rr = doRequestWithHeaders(srv, "POST",
"/api/v1/workspaces/"+ws.Slug+"/collections/"+coll.Slug+"/items",
map[string]interface{}{"title": "bad", "fields": `{"cost":"not-a-number"}`},
map[string]string{"Authorization": "Bearer " + sessionToken})
if rr.Code != http.StatusBadRequest {
t.Fatalf("expected 400 for an un-coercible number, got %d: %s", rr.Code, rr.Body.String())
}
}
@@ -1084,6 +1084,11 @@ func migrateCopyFields(sourceFieldsJSON, sourceSchemaJSON, targetSchemaJSON stri
}
migrated.Fields[k] = v
}
// Coerce strings to their declared types before validating (BUG-2850).
// MUST match the preflight (handlers_items_copy_preflight.go) — see the
// note there; these two live in different PACKAGES, which is exactly how
// they would drift unnoticed.
migrated.Fields = items.CoerceFields(migrated.Fields, items.SchemaForMigratedFields(targetSchema))
if err := items.ValidateFields(migrated.Fields, items.SchemaForMigratedFields(targetSchema)); err != nil {
return nil, nil, &FieldValidationError{Err: err}
}