xarmian
7512f5500b
* docs(decision): README decision-provider section, and the day-73 eval scripts with their recorded baseline (TASK-3123) Housekeeping for PLAN-3114 tranche 1; no behaviour change. - README: a Decision provider (optional) section covering what uses it, that answers are asynchronous, configuration precedence (config file, then the admin setting, then the environment; PAD_DECISION_PROVIDER=none as the operator off-switch), and exactly what leaves the box and where. - scripts/decision-eval/eval.py and eval2.py: the day-73 scripts, as run (standard library, raw systemone API, key read from TYPESAFE_API_KEY). A read and a pattern scan found no key or token literal. - scripts/decision-eval/README.md: what each script measures, how to run it, how to regenerate the corpora, and the day-73 baseline. A, B and D were recomputed from the saved result files with the scripts' own summary logic and match PLAN-3114 (D AUC 0.939; the B >=0.8 bucket 86.8%). C comes from PLAN-3114's trail. eval.py --dry-run over the saved corpora queues the same 360 items the day-73 run did. - scripts/decision-eval/.gitignore: the corpora are private workspace exports and the results embed their content, so neither can be committed; neither was. The CHANGELOG line is queued as DOC-3242 (draft highlights for the release after v0.15.0, per PLAYB-1160), since releases take their changelog from goreleaser and a highlights DOC. Claude-Session: https://claude.ai/code/session_016PXRzXhokr61CLbbu3Txj9 * docs(decision): the README section states what off, the triggers, server info and disabling actually do (TASK-3123, codex round 1) Codex fact-checked every claim against the code, and four were too broad. Off is not "every surface unchanged": the decisions list is empty and playbook match answers 404. Jobs are also queued by restore and by comment edit and delete (verified at items.go:614/3207/3437/5459 and comments.go:133/196/416), and a closed item's job is dropped at the tick. pad server info cannot see the admin setting. Disabling stops new calls without a restart, but an in-flight request may finish. The scripts README now says eval2.py runs at the top level. Claude-Session: https://claude.ai/code/session_016PXRzXhokr61CLbbu3Txj9
Decision-provider evals
The scripts behind PLAN-3114's go/no-go decision. They measure the decision
provider (typesafe.ai Jev) against human-set labels in the docapp workspace,
so a model pin change (internal/decision.DefaultModel) can be measured
against a recorded baseline instead of argued about.
| script | what it asks | ground truth |
|---|---|---|
eval.py |
A. field recovery: severity/component (bugs), priority/effort (tasks), impact (ideas), as Choice questions. B. collection classification: bug, task or idea, with the collection hidden. | the human-set field / the real collection |
eval2.py |
C. CONVE-4 (conventional commits) as a Noul over 200 real commit subjects plus up to 60 mutated negatives. D. needs_human_decision / blocked as Nouls over human-tasks (positives) vs 70 random tasks (negatives). |
a regex (C); the collection the item lives in (D) |
attention/ |
TASK-3137's replay of D through the production state builder; see its RESULTS.md. |
as D |
Both Python scripts call https://api.typesafe.ai/v1/systemone directly
(standard library only) and send item titles and bodies there, which is the
same disclosure the product makes when the provider is enabled.
Running
export TYPESAFE_API_KEY=... # read from the environment, never printed
cd scripts/decision-eval
# 1. regenerate the corpora (below); they must sit NEXT TO the scripts
# 2.
python3 eval.py --dry-run # builds the jobs and prints one request; no API call
python3 eval.py --model jev-1.13.0 # A + B; writes results-<model>.jsonl
python3 eval2.py # C + D; pinned to jev-1.13.0; writes results-decision.json
The scripts read TYPESAFE_API_KEY, the vendor's name. Pad itself reads
PAD_TYPESAFE_API_KEY (see the README's Decision provider section).
They are deliberately separate, so an eval run never borrows a server's
credentials by accident.
eval.py defaults to --model jev-latest, which is what the day-73 A/B run
used. jev-latest moves without notice, so pass the pinned model to compare
like with like. eval2.py has no --dry-run: its code runs at the top level,
so running it at all reads the key and makes the API calls.
Regenerating the corpora
The corpora are exports of the docapp workspace and are never committed
(.gitignore). From a checkout linked to the docapp workspace:
pad item list bugs --all --full --limit 1000 --format json > bugs.json
pad item list tasks --all --full --limit 1000 --format json > tasks.json
pad item list ideas --all --full --limit 1000 --format json > ideas.json
pad item list human-tasks --all --full --limit 1000 --format json > human-tasks.json
pad collection list --format json > collections.json
git log --format=%s -n 400 > commits.txt
--all is required, because the day-73 corpora include done and closed items,
and --full supplies the bodies. A regenerated corpus is today's workspace,
not day 73's: at day 73 there were 557 bugs, 1000 tasks (capped by
--limit), 433 ideas, 70 human-tasks and 400 commit subjects, and the
workspace has grown since. The random draws are seeded (73), but over
different populations, so compare RATES, not item-level answers.
Day-73 baseline (2026-09-19)
A, B and D were recomputed from the saved result files with the scripts' own summary logic, and match the figures recorded on PLAN-3114. C's results were not saved; its figures are from PLAN-3114's trail.
A. field recovery (jev-latest, 360 items; exact = the human's value, adjacent = within one level)
| field | n | exact | adjacent | conf >= 0.8 correct | majority baseline |
|---|---|---|---|---|---|
| bugs.severity | 114 | 42.1% | 86.8% | 15 / 27 | 43.0% (low) |
| bugs.component | 112 | 42.0% | 42.0% | 41 / 83 | 21.4% (web) |
| tasks.priority | 120 | 56.7% | 95.8% | 24 / 36 | 45.0% (medium) |
| tasks.effort | 33 | 51.5% | 97.0% | 4 / 6 | 45.5% (m) |
| ideas.impact | 120 | 56.7% | 92.5% | 26 / 41 | 47.5% (medium) |
B. collection classification (jev-latest): 278 / 360 = 77.2% overall; 217 / 250 = 86.8% where confidence >= 0.8.
C. CONVE-4 Noul (jev-1.13.0, 260 subjects): AUC 0.971; at threshold 0.9, precision 98.8% and recall 92.3%.
D. needs_human_decision Noul (jev-1.13.0, 70 + 70 items, title + body)
| AUC | P / R at 0.5 | P / R at 0.7 | P / R at 0.9 |
|---|---|---|---|
| 0.939 | 93.0% / 75.7% | 94.4% / 48.6% | 100.0% / 21.4% |
What those numbers led to is on PLAN-3114: subjective scales are not worth automating (tranche 2 chips only at >= 0.8); objective Nouls are. TASK-3137 re-ran D through the production state builder.