xarmian
deec62235e
* wip(decision): conventions set (TASK-3119 U1a) * chore(eval): executable-conventions eval through the production question path (TASK-3119 U0) Measures, before any chip exists, how often the decision provider says a hand-labelled item breaks a convention at the 0.9 threshold: per convention precision and recall (all cases and real-only), the false 'Breaks' rate of a behaviour convention, and the calls and tokens spent. State is decision.BuildItemState, the question a decision.Noul worded as the U1 contract says, the model decision.DefaultModel. The hand labels were posted to TASK-3119 before any score existed; the trail comment carrying them is dropped from every state, and the dry run refuses a state that still carries them. Reads TYPESAFE_API_KEY only. Claude-Session: https://claude.ai/code/session_01Q9WssCjFuHjSfLABoW4tmS * chore(eval): -links and -only for the conventions eval (TASK-3119 U1a) -links builds each state as the conventions set does, with the item's links (from the dump: the parent link only; blocking links are not in pad item show's JSON). -only runs one convention. Used for the lead's U1a gate: CONVE-1286 rerun with links. Claude-Session: https://claude.ai/code/session_01Q9WssCjFuHjSfLABoW4tmS * fix(decision): liveness before EACH split call (TASK-3119 U1a, codex r1) A split set makes several provider calls; the liveness check ran once before them, so an item or workspace deleted between calls was still sent in the later ones. It now runs immediately before every call. A test deletes the item from inside the first call and the second is not sent; checking only once fails it. Claude-Session: https://claude.ai/code/session_01Q9WssCjFuHjSfLABoW4tmS * fix(decision): decision_check has no default; allow-list the eval's decode (TASK-3119 U1a) A default "on" was written into every new convention: imports gained a decision_check field (artifact import parity) and an activated built-in no longer matched its seed (TASK-3462 would read every one as diverged). Absent already reads as on, so the field carries no default, in the template and both migrations. The conventions eval's readJSON is a raw decode the reserved-field guard counts; allow-listed with its reason, like the attention harness. Claude-Session: https://claude.ai/code/session_01Q9WssCjFuHjSfLABoW4tmS
Decision-provider evals
The scripts behind PLAN-3114's go/no-go decision. They measure the decision
provider (typesafe.ai Jev) against human-set labels in the docapp workspace,
so a model pin change (internal/decision.DefaultModel) can be measured
against a recorded baseline instead of argued about.
| script | what it asks | ground truth |
|---|---|---|
eval.py |
A. field recovery: severity/component (bugs), priority/effort (tasks), impact (ideas), as Choice questions. B. collection classification: bug, task or idea, with the collection hidden. | the human-set field / the real collection |
eval2.py |
C. CONVE-4 (conventional commits) as a Noul over 200 real commit subjects plus up to 60 mutated negatives. D. needs_human_decision / blocked as Nouls over human-tasks (positives) vs 70 random tasks (negatives). |
a regex (C); the collection the item lives in (D) |
attention/ |
TASK-3137's replay of D through the production state builder; see its RESULTS.md. |
as D |
Both Python scripts call https://api.typesafe.ai/v1/systemone directly
(standard library only) and send item titles and bodies there, which is the
same disclosure the product makes when the provider is enabled.
Running
export TYPESAFE_API_KEY=... # read from the environment, never printed
cd scripts/decision-eval
# 1. regenerate the corpora (below); they must sit NEXT TO the scripts
# 2.
python3 eval.py --dry-run # builds the jobs and prints one request; no API call
python3 eval.py --model jev-1.13.0 # A + B; writes results-<model>.jsonl
python3 eval2.py # C + D; pinned to jev-1.13.0; writes results-decision.json
The scripts read TYPESAFE_API_KEY, the vendor's name. Pad itself reads
PAD_TYPESAFE_API_KEY (see the README's Decision provider section).
They are deliberately separate, so an eval run never borrows a server's
credentials by accident.
eval.py defaults to --model jev-latest, which is what the day-73 A/B run
used. jev-latest moves without notice, so pass the pinned model to compare
like with like. eval2.py has no --dry-run: its code runs at the top level,
so running it at all reads the key and makes the API calls.
Regenerating the corpora
The corpora are exports of the docapp workspace and are never committed
(.gitignore). From a checkout linked to the docapp workspace:
pad item list bugs --all --full --limit 1000 --format json > bugs.json
pad item list tasks --all --full --limit 1000 --format json > tasks.json
pad item list ideas --all --full --limit 1000 --format json > ideas.json
pad item list human-tasks --all --full --limit 1000 --format json > human-tasks.json
pad collection list --format json > collections.json
git log --format=%s -n 400 > commits.txt
--all is required, because the day-73 corpora include done and closed items,
and --full supplies the bodies. A regenerated corpus is today's workspace,
not day 73's: at day 73 there were 557 bugs, 1000 tasks (capped by
--limit), 433 ideas, 70 human-tasks and 400 commit subjects, and the
workspace has grown since. The random draws are seeded (73), but over
different populations, so compare RATES, not item-level answers.
Day-73 baseline (2026-09-19)
A, B and D were recomputed from the saved result files with the scripts' own summary logic, and match the figures recorded on PLAN-3114. C's results were not saved; its figures are from PLAN-3114's trail.
A. field recovery (jev-latest, 360 items; exact = the human's value, adjacent = within one level)
| field | n | exact | adjacent | conf >= 0.8 correct | majority baseline |
|---|---|---|---|---|---|
| bugs.severity | 114 | 42.1% | 86.8% | 15 / 27 | 43.0% (low) |
| bugs.component | 112 | 42.0% | 42.0% | 41 / 83 | 21.4% (web) |
| tasks.priority | 120 | 56.7% | 95.8% | 24 / 36 | 45.0% (medium) |
| tasks.effort | 33 | 51.5% | 97.0% | 4 / 6 | 45.5% (m) |
| ideas.impact | 120 | 56.7% | 92.5% | 26 / 41 | 47.5% (medium) |
B. collection classification (jev-latest): 278 / 360 = 77.2% overall; 217 / 250 = 86.8% where confidence >= 0.8.
C. CONVE-4 Noul (jev-1.13.0, 260 subjects): AUC 0.971; at threshold 0.9, precision 98.8% and recall 92.3%.
D. needs_human_decision Noul (jev-1.13.0, 70 + 70 items, title + body)
| AUC | P / R at 0.5 | P / R at 0.7 | P / R at 0.9 |
|---|---|---|---|
| 0.939 | 93.0% / 75.7% | 94.4% / 48.6% | 100.0% / 21.4% |
What those numbers led to is on PLAN-3114: subjective scales are not worth automating (tranche 2 chips only at >= 0.8); objective Nouls are. TASK-3137 re-ran D through the production state builder.