xarmian
4532eecffd
* chore(eval): TASK-3119 U2 gates in one command; decision.BuildCommentState -gates runs three passes through the production question path: A U0's population on the U2a item state (links in, trail out), gated on precision not dropping below U0's on any convention that fired; B comment-borne breaks, each comment asked alone with its item's title and collection, gated on precision >= 0.9; C a measurement of 12+ questions in one call (MaxPerCall 12). BuildCommentState is the comment subject's one serialisation, shared by the harness now and the runner in U2b. Comments are clipped at the body bound (16k runes), not the trail's 2k. Claude-Session: https://claude.ai/code/session_01Q9WssCjFuHjSfLABoW4tmS * feat(decision): the conventions set asks about the item without its comment trail, 12 questions a call (TASK-3119 U2a) The conventions state carried the last 10 comments, clipped at 2,000 runes each: on docapp that was most of every call's input (15.7k tokens for one write across 3 calls), 720 items had a clipped comment, and a break past the clip was invisible anyway. Comments get their own subject in U2b. - QuestionSet.NoTrail leaves the trail out; the conventions set sets it. The state is exactly what the U2 gates measured (recent_trail [], links in): precision held at 100% on every convention that fires, recall rose (1286 40->80%, 2693 70->90%), 0 truncated states, ~954 input tokens per call. - MaxPerCall 5 -> 12, from measure C: up to 13 questions in one call gave identical precision and recall, 0 of 116 answers moved by more than 0.2. - A comment write no longer changes this set's state, so it costs no call. - Every conventions answer computed before this reads not-current until the item's next write (the state bytes changed): a fleet-wide re-ask, by design. - CLAUDE.md: the registered sets (it still said attention was the only one, since U1a) and the per-set state. Claude-Session: https://claude.ai/code/session_01Q9WssCjFuHjSfLABoW4tmS
Decision-provider evals
The scripts behind PLAN-3114's go/no-go decision. They measure the decision
provider (typesafe.ai Jev) against human-set labels in the docapp workspace,
so a model pin change (internal/decision.DefaultModel) can be measured
against a recorded baseline instead of argued about.
| script | what it asks | ground truth |
|---|---|---|
eval.py |
A. field recovery: severity/component (bugs), priority/effort (tasks), impact (ideas), as Choice questions. B. collection classification: bug, task or idea, with the collection hidden. | the human-set field / the real collection |
eval2.py |
C. CONVE-4 (conventional commits) as a Noul over 200 real commit subjects plus up to 60 mutated negatives. D. needs_human_decision / blocked as Nouls over human-tasks (positives) vs 70 random tasks (negatives). |
a regex (C); the collection the item lives in (D) |
attention/ |
TASK-3137's replay of D through the production state builder; see its RESULTS.md. |
as D |
Both Python scripts call https://api.typesafe.ai/v1/systemone directly
(standard library only) and send item titles and bodies there, which is the
same disclosure the product makes when the provider is enabled.
Running
export TYPESAFE_API_KEY=... # read from the environment, never printed
cd scripts/decision-eval
# 1. regenerate the corpora (below); they must sit NEXT TO the scripts
# 2.
python3 eval.py --dry-run # builds the jobs and prints one request; no API call
python3 eval.py --model jev-1.13.0 # A + B; writes results-<model>.jsonl
python3 eval2.py # C + D; pinned to jev-1.13.0; writes results-decision.json
The scripts read TYPESAFE_API_KEY, the vendor's name. Pad itself reads
PAD_TYPESAFE_API_KEY (see the README's Decision provider section).
They are deliberately separate, so an eval run never borrows a server's
credentials by accident.
eval.py defaults to --model jev-latest, which is what the day-73 A/B run
used. jev-latest moves without notice, so pass the pinned model to compare
like with like. eval2.py has no --dry-run: its code runs at the top level,
so running it at all reads the key and makes the API calls.
Regenerating the corpora
The corpora are exports of the docapp workspace and are never committed
(.gitignore). From a checkout linked to the docapp workspace:
pad item list bugs --all --full --limit 1000 --format json > bugs.json
pad item list tasks --all --full --limit 1000 --format json > tasks.json
pad item list ideas --all --full --limit 1000 --format json > ideas.json
pad item list human-tasks --all --full --limit 1000 --format json > human-tasks.json
pad collection list --format json > collections.json
git log --format=%s -n 400 > commits.txt
--all is required, because the day-73 corpora include done and closed items,
and --full supplies the bodies. A regenerated corpus is today's workspace,
not day 73's: at day 73 there were 557 bugs, 1000 tasks (capped by
--limit), 433 ideas, 70 human-tasks and 400 commit subjects, and the
workspace has grown since. The random draws are seeded (73), but over
different populations, so compare RATES, not item-level answers.
Day-73 baseline (2026-09-19)
A, B and D were recomputed from the saved result files with the scripts' own summary logic, and match the figures recorded on PLAN-3114. C's results were not saved; its figures are from PLAN-3114's trail.
A. field recovery (jev-latest, 360 items; exact = the human's value, adjacent = within one level)
| field | n | exact | adjacent | conf >= 0.8 correct | majority baseline |
|---|---|---|---|---|---|
| bugs.severity | 114 | 42.1% | 86.8% | 15 / 27 | 43.0% (low) |
| bugs.component | 112 | 42.0% | 42.0% | 41 / 83 | 21.4% (web) |
| tasks.priority | 120 | 56.7% | 95.8% | 24 / 36 | 45.0% (medium) |
| tasks.effort | 33 | 51.5% | 97.0% | 4 / 6 | 45.5% (m) |
| ideas.impact | 120 | 56.7% | 92.5% | 26 / 41 | 47.5% (medium) |
B. collection classification (jev-latest): 278 / 360 = 77.2% overall; 217 / 250 = 86.8% where confidence >= 0.8.
C. CONVE-4 Noul (jev-1.13.0, 260 subjects): AUC 0.971; at threshold 0.9, precision 98.8% and recall 92.3%.
D. needs_human_decision Noul (jev-1.13.0, 70 + 70 items, title + body)
| AUC | P / R at 0.5 | P / R at 0.7 | P / R at 0.9 |
|---|---|---|---|
| 0.939 | 93.0% / 75.7% | 94.4% / 48.6% | 100.0% / 21.4% |
What those numbers led to is on PLAN-3114: subjective scales are not worth automating (tranche 2 chips only at >= 0.8); objective Nouls are. TASK-3137 re-ran D through the production state builder.