mirror of
https://github.com/PerpetualSoftware/pad.git
synced 2026-10-27 06:47:07 +00:00
xarmian
7512f5500b
* docs(decision): README decision-provider section, and the day-73 eval scripts with their recorded baseline (TASK-3123) Housekeeping for PLAN-3114 tranche 1; no behaviour change. - README: a Decision provider (optional) section covering what uses it, that answers are asynchronous, configuration precedence (config file, then the admin setting, then the environment; PAD_DECISION_PROVIDER=none as the operator off-switch), and exactly what leaves the box and where. - scripts/decision-eval/eval.py and eval2.py: the day-73 scripts, as run (standard library, raw systemone API, key read from TYPESAFE_API_KEY). A read and a pattern scan found no key or token literal. - scripts/decision-eval/README.md: what each script measures, how to run it, how to regenerate the corpora, and the day-73 baseline. A, B and D were recomputed from the saved result files with the scripts' own summary logic and match PLAN-3114 (D AUC 0.939; the B >=0.8 bucket 86.8%). C comes from PLAN-3114's trail. eval.py --dry-run over the saved corpora queues the same 360 items the day-73 run did. - scripts/decision-eval/.gitignore: the corpora are private workspace exports and the results embed their content, so neither can be committed; neither was. The CHANGELOG line is queued as DOC-3242 (draft highlights for the release after v0.15.0, per PLAYB-1160), since releases take their changelog from goreleaser and a highlights DOC. Claude-Session: https://claude.ai/code/session_016PXRzXhokr61CLbbu3Txj9 * docs(decision): the README section states what off, the triggers, server info and disabling actually do (TASK-3123, codex round 1) Codex fact-checked every claim against the code, and four were too broad. Off is not "every surface unchanged": the decisions list is empty and playbook match answers 404. Jobs are also queued by restore and by comment edit and delete (verified at items.go:614/3207/3437/5459 and comments.go:133/196/416), and a closed item's job is dropped at the tick. pad server info cannot see the admin setting. Disabling stops new calls without a restart, but an in-flight request may finish. The scripts README now says eval2.py runs at the top level. Claude-Session: https://claude.ai/code/session_016PXRzXhokr61CLbbu3Txj9
90 lines
4.6 KiB
Markdown
90 lines
4.6 KiB
Markdown
# Decision-provider evals
|
|
|
|
The scripts behind PLAN-3114's go/no-go decision. They measure the decision
|
|
provider (typesafe.ai Jev) against human-set labels in the docapp workspace,
|
|
so a model pin change (`internal/decision.DefaultModel`) can be measured
|
|
against a recorded baseline instead of argued about.
|
|
|
|
| script | what it asks | ground truth |
|
|
|---|---|---|
|
|
| `eval.py` | **A.** field recovery: severity/component (bugs), priority/effort (tasks), impact (ideas), as Choice questions. **B.** collection classification: bug, task or idea, with the collection hidden. | the human-set field / the real collection |
|
|
| `eval2.py` | **C.** CONVE-4 (conventional commits) as a Noul over 200 real commit subjects plus up to 60 mutated negatives. **D.** `needs_human_decision` / `blocked` as Nouls over human-tasks (positives) vs 70 random tasks (negatives). | a regex (C); the collection the item lives in (D) |
|
|
| `attention/` | TASK-3137's replay of D through the production state builder; see its `RESULTS.md`. | as D |
|
|
|
|
Both Python scripts call `https://api.typesafe.ai/v1/systemone` directly
|
|
(standard library only) and send item titles and bodies there, which is the
|
|
same disclosure the product makes when the provider is enabled.
|
|
|
|
## Running
|
|
|
|
```bash
|
|
export TYPESAFE_API_KEY=... # read from the environment, never printed
|
|
cd scripts/decision-eval
|
|
# 1. regenerate the corpora (below); they must sit NEXT TO the scripts
|
|
# 2.
|
|
python3 eval.py --dry-run # builds the jobs and prints one request; no API call
|
|
python3 eval.py --model jev-1.13.0 # A + B; writes results-<model>.jsonl
|
|
python3 eval2.py # C + D; pinned to jev-1.13.0; writes results-decision.json
|
|
```
|
|
|
|
The scripts read **`TYPESAFE_API_KEY`**, the vendor's name. Pad itself reads
|
|
**`PAD_TYPESAFE_API_KEY`** (see the README's *Decision provider* section).
|
|
They are deliberately separate, so an eval run never borrows a server's
|
|
credentials by accident.
|
|
|
|
`eval.py` defaults to `--model jev-latest`, which is what the day-73 A/B run
|
|
used. `jev-latest` moves without notice, so pass the pinned model to compare
|
|
like with like. `eval2.py` has no `--dry-run`: its code runs at the top level,
|
|
so running it at all reads the key and makes the API calls.
|
|
|
|
## Regenerating the corpora
|
|
|
|
The corpora are exports of the docapp workspace and are **never committed**
|
|
(`.gitignore`). From a checkout linked to the docapp workspace:
|
|
|
|
```bash
|
|
pad item list bugs --all --full --limit 1000 --format json > bugs.json
|
|
pad item list tasks --all --full --limit 1000 --format json > tasks.json
|
|
pad item list ideas --all --full --limit 1000 --format json > ideas.json
|
|
pad item list human-tasks --all --full --limit 1000 --format json > human-tasks.json
|
|
pad collection list --format json > collections.json
|
|
git log --format=%s -n 400 > commits.txt
|
|
```
|
|
|
|
`--all` is required, because the day-73 corpora include done and closed items,
|
|
and `--full` supplies the bodies. **A regenerated corpus is today's workspace,
|
|
not day 73's**: at day 73 there were 557 bugs, 1000 tasks (capped by
|
|
`--limit`), 433 ideas, 70 human-tasks and 400 commit subjects, and the
|
|
workspace has grown since. The random draws are seeded (`73`), but over
|
|
different populations, so compare RATES, not item-level answers.
|
|
|
|
## Day-73 baseline (2026-09-19)
|
|
|
|
A, B and D were recomputed from the saved result files with the scripts' own
|
|
summary logic, and match the figures recorded on PLAN-3114. C's results were
|
|
not saved; its figures are from PLAN-3114's trail.
|
|
|
|
**A. field recovery** (`jev-latest`, 360 items; exact = the human's value, adjacent = within one level)
|
|
|
|
| field | n | exact | adjacent | conf >= 0.8 correct | majority baseline |
|
|
|---|---|---|---|---|---|
|
|
| bugs.severity | 114 | 42.1% | 86.8% | 15 / 27 | 43.0% (low) |
|
|
| bugs.component | 112 | 42.0% | 42.0% | 41 / 83 | 21.4% (web) |
|
|
| tasks.priority | 120 | 56.7% | 95.8% | 24 / 36 | 45.0% (medium) |
|
|
| tasks.effort | 33 | 51.5% | 97.0% | 4 / 6 | 45.5% (m) |
|
|
| ideas.impact | 120 | 56.7% | 92.5% | 26 / 41 | 47.5% (medium) |
|
|
|
|
**B. collection classification** (`jev-latest`): 278 / 360 = 77.2% overall; 217 / 250 = 86.8% where confidence >= 0.8.
|
|
|
|
**C. CONVE-4 Noul** (`jev-1.13.0`, 260 subjects): AUC 0.971; at threshold 0.9, precision 98.8% and recall 92.3%.
|
|
|
|
**D. `needs_human_decision` Noul** (`jev-1.13.0`, 70 + 70 items, title + body)
|
|
|
|
| AUC | P / R at 0.5 | P / R at 0.7 | P / R at 0.9 |
|
|
|---|---|---|---|
|
|
| 0.939 | 93.0% / 75.7% | 94.4% / 48.6% | 100.0% / 21.4% |
|
|
|
|
What those numbers led to is on PLAN-3114: subjective scales are not worth
|
|
automating (tranche 2 chips only at >= 0.8); objective Nouls are. TASK-3137
|
|
re-ran D through the production state builder.
|