xarmian
e598bc4f64
* feat(decision): effort, effort_medium and high_stakes on the attention set, staged; one-shot backfill (TASK-3921)
- attention set gains effort (Score), effort_medium (Choice) and high_stakes
(Noul) in the same call; needs_human_decision's text is unchanged.
- QuestionSet.Unsurfaced names the staged keys: WorkspaceFlags and
SurfacedDecisions skip them (no flag, no state read, no counter); the
decisions list and endpoint still return every row.
- attention chips only render keys that have a label.
- store: per-set enqueue body extracted (owedDecisionJobTx), OweDecisionJobs
(chunked, under the installed debounce) and ItemsMissingDecisionKey.
- Runner.BackfillEffort owes the attention set once for eligible items with a
needs_human_decision answer and no effort answer, marker
effort_decisions_backfilled written after every chunk; hooked at the end of
reconfigureDecisionsLocked (boot and later admin enable); makes no provider call.
- eval harness: titlebody now sends the live set; RESULTS.md rows pending.
- docs/api.md and tests updated.
Claude-Session: https://claude.ai/code/session_018oQ5EErQKHtBJK8yLpcDNF
* test(decision-eval): keep titlebody as the day-73 control, add titlebody-live; record the TASK-3957 and TASK-3921 runs
- titlebody again sends the day-73 pair (needs_human_decision + blocked), so it
stays comparable with its 09-22 row as the model-drift control.
- titlebody-live sends the live attention set on the same {title, body} state:
the TASK-3921 gate (needs_human_decision AUC 0.936, within .02 of 0.939).
- Every run prints the question keys it sends, before -dry too, so a row's
call shape is in its own log.
- RESULTS.md: rows for the one-question call (TASK-3957) and the
four-question call (TASK-3921).
Claude-Session: https://claude.ai/code/session_018oQ5EErQKHtBJK8yLpcDNF
* fix(decision): effort backfill lifecycle, staged keys off item reads, dry-run guard (TASK-3921; #r1 F1-F4)
- F1: startEffortBackfill checks stopped and does bg.Add under one
decisionTick.mu hold; BackfillEffort/EffortBackfillCandidates take a ctx
checked before each candidate read and each chunk; cancel returns ctx.Err()
and never writes the marker. stopDecisionTick cancels the run.
- F2: reconfigure cancels a running backfill; only a non-nil runner starts a
new one. A per-run generation replaces the atomic latch so a cancelled run
cannot clobber its successor.
- F3: item show / --agent / MCP item get use Runner.ItemReadDecisions, which
drops the set's Unsurfaced keys; the decisions endpoint stays raw. docs/api.md
updated.
- F4: the file dry run refuses a path with a -wal/-shm sibling or under
$HOME/.pad (refuseLiveDatabase, tested without the env var).
Claude-Session: https://claude.ai/code/session_018oQ5EErQKHtBJK8yLpcDNF
* test(decision): a chunk write that fails in the store writes no marker (TASK-3921; mutant M6)
The two failure tests fail the per-chunk test hook, so the path where
OweDecisionJobs itself returns an error was never exercised: a mutant that
ignores that error and marks anyway survived. The new test breaks the
decision_jobs table from the hook so the error comes back through the store,
asserts no marker, then restores the table and re-runs to completion.
Claude-Session: https://claude.ai/code/session_018oQ5EErQKHtBJK8yLpcDNF
* refactor(decision): drop the pre-marker cancel check; a cancel after the last chunk may mark (TASK-3921; M5, lead ruling)
Mutant M5 (that check removed) survived every test, and could not do
otherwise: a cancel can land there only after every chunk was owed, and the
marker means every candidate was ENQUEUED, not answered, so marking then is
true. The check is removed rather than kept as a guard no instrument can
turn red. The doc comments on BackfillEffort, cancelEffortBackfill and
startEffortBackfill now say a cancel before the last chunk writes no
marker and one after it may.
Claude-Session: https://claude.ai/code/session_018oQ5EErQKHtBJK8yLpcDNF
* docs(api): a backfill chunk in flight when decisions are disabled may still commit (TASK-3921; r2 L1, lead ruling bdce2116)
docs/api.md said that with no provider "no job row is ever written". That is
false for one window: a disable that lands while the one-shot effort
backfill is inside a chunk's write lets that chunk commit, and its jobs wait
until a provider returns. Ruled docs-only. The disable test's comment now
says it covers a backfill parked before its first chunk, not that window.
Sweep for the claim (CONVE-23): grep of docs, internal, cmd, scripts and
skills (.go, .md) for "no job row", "job row is", "no job is", "no jobs
are written/owed/enqueued" and "never written/owed/enqueued". Two hits are
about this mechanism: api.md:72 (fixed) and the test comment (narrowed).
Unchanged, still true:
- store/decisions.go:27 is about the WRITE DOORS, which still enqueue
nothing without a resolver.
- runner_test.go:238 is about the nil-provider runner, which never starts a
backfill.
- effort_backfill_lifecycle_test.go:128 covers a run parked before its
first chunk.
The other hits are unrelated mechanisms.
Claude-Session: https://claude.ai/code/session_018oQ5EErQKHtBJK8yLpcDNF
* fix(decision): pace the effort backfill on drain; dry run opens a copy; docs eligibility (TASK-3921; r3 M1, L2, L4)
- M1: the backfill owes one tick pass of items per chunk (20 default, wired
from the tick limit) and owes chunk k+1 only once chunk k's own jobs have
drained: none claimable now, by ClaimDecisionJobs' own predicate
(new Store.CountClaimableDecisionJobs). Debounced jobs do not hold it.
The wait polls at the tick interval, honours ctx, and a cancel writes
no marker; the marker still follows the last chunk being owed.
- L2: TestEffortBackfillDryRunOnFile copies the named file under GOTMPDIR
and opens only the copy.
- L4: docs/api.md states the real eligibility and the drain pacing.
Claude-Session: https://claude.ai/code/session_018oQ5EErQKHtBJK8yLpcDNF
* docs(decision): the Unsurfaced comment names ItemReadDecisions (TASK-3921; r4 L2)
The field's comment listed WorkspaceFlags and SurfacedDecisions as the
readers that skip staged keys, but not ItemReadDecisions, which item show
(and so --agent and MCP item get) has used since the r1 F3 fix. Comment only.
Claude-Session: https://claude.ai/code/session_018oQ5EErQKHtBJK8yLpcDNF
* test(decision): break the table before the LAST chunk so the store-failure test pins M6 (TASK-3921; lead ruling)
Since the drain pacing (51c545850), the store-failure test broke the
decision_jobs table before chunk 1 of 3. Under mutant M6 (a failed chunk
write ignored) the drain wait that follows read the same broken table and
failed, so no marker was written by a different mechanism and the test stayed
green: an end-state assertion another path also produced (CONVE-12). Breaking
the table before the last chunk leaves no drain wait after it (the marker does
not wait for the last chunk to drain), so only the chunk's own error check can
keep the marker unwritten.
Claude-Session: https://claude.ai/code/session_018oQ5EErQKHtBJK8yLpcDNF
Decision-provider evals
The scripts behind PLAN-3114's go/no-go decision. They measure the decision
provider (typesafe.ai Jev) against human-set labels in the docapp workspace,
so a model pin change (internal/decision.DefaultModel) can be measured
against a recorded baseline instead of argued about.
| script | what it asks | ground truth |
|---|---|---|
eval.py |
A. field recovery: severity/component (bugs), priority/effort (tasks), impact (ideas), as Choice questions. B. collection classification: bug, task or idea, with the collection hidden. | the human-set field / the real collection |
eval2.py |
C. CONVE-4 (conventional commits) as a Noul over 200 real commit subjects plus up to 60 mutated negatives. D. needs_human_decision / blocked as Nouls over human-tasks (positives) vs 70 random tasks (negatives). |
a regex (C); the collection the item lives in (D) |
attention/ |
TASK-3137's replay of D through the production state builder; see its RESULTS.md. |
as D |
Both Python scripts call https://api.typesafe.ai/v1/systemone directly
(standard library only) and send item titles and bodies there, which is the
same disclosure the product makes when the provider is enabled.
Running
export TYPESAFE_API_KEY=... # read from the environment, never printed
cd scripts/decision-eval
# 1. regenerate the corpora (below); they must sit NEXT TO the scripts
# 2.
python3 eval.py --dry-run # builds the jobs and prints one request; no API call
python3 eval.py --model jev-1.13.0 # A + B; writes results-<model>.jsonl
python3 eval2.py # C + D; pinned to jev-1.13.0; writes results-decision.json
The scripts read TYPESAFE_API_KEY, the vendor's name. Pad itself reads
PAD_TYPESAFE_API_KEY (see the README's Decision provider section).
They are deliberately separate, so an eval run never borrows a server's
credentials by accident.
eval.py defaults to --model jev-latest, which is what the day-73 A/B run
used. jev-latest moves without notice, so pass the pinned model to compare
like with like. eval2.py has no --dry-run: its code runs at the top level,
so running it at all reads the key and makes the API calls.
Regenerating the corpora
The corpora are exports of the docapp workspace and are never committed
(.gitignore). From a checkout linked to the docapp workspace:
pad item list bugs --all --full --limit 1000 --format json > bugs.json
pad item list tasks --all --full --limit 1000 --format json > tasks.json
pad item list ideas --all --full --limit 1000 --format json > ideas.json
pad item list human-tasks --all --full --limit 1000 --format json > human-tasks.json
pad collection list --format json > collections.json
git log --format=%s -n 400 > commits.txt
--all is required, because the day-73 corpora include done and closed items,
and --full supplies the bodies. A regenerated corpus is today's workspace,
not day 73's: at day 73 there were 557 bugs, 1000 tasks (capped by
--limit), 433 ideas, 70 human-tasks and 400 commit subjects, and the
workspace has grown since. The random draws are seeded (73), but over
different populations, so compare RATES, not item-level answers.
Day-73 baseline (2026-09-19)
A, B and D were recomputed from the saved result files with the scripts' own summary logic, and match the figures recorded on PLAN-3114. C's results were not saved; its figures are from PLAN-3114's trail.
A. field recovery (jev-latest, 360 items; exact = the human's value, adjacent = within one level)
| field | n | exact | adjacent | conf >= 0.8 correct | majority baseline |
|---|---|---|---|---|---|
| bugs.severity | 114 | 42.1% | 86.8% | 15 / 27 | 43.0% (low) |
| bugs.component | 112 | 42.0% | 42.0% | 41 / 83 | 21.4% (web) |
| tasks.priority | 120 | 56.7% | 95.8% | 24 / 36 | 45.0% (medium) |
| tasks.effort | 33 | 51.5% | 97.0% | 4 / 6 | 45.5% (m) |
| ideas.impact | 120 | 56.7% | 92.5% | 26 / 41 | 47.5% (medium) |
B. collection classification (jev-latest): 278 / 360 = 77.2% overall; 217 / 250 = 86.8% where confidence >= 0.8.
C. CONVE-4 Noul (jev-1.13.0, 260 subjects): AUC 0.971; at threshold 0.9, precision 98.8% and recall 92.3%.
D. needs_human_decision Noul (jev-1.13.0, 70 + 70 items, title + body)
| AUC | P / R at 0.5 | P / R at 0.7 | P / R at 0.9 |
|---|---|---|---|
| 0.939 | 93.0% / 75.7% | 94.4% / 48.6% | 100.0% / 21.4% |
What those numbers led to is on PLAN-3114: subjective scales are not worth automating (tranche 2 chips only at >= 0.8); objective Nouls are. TASK-3137 re-ran D through the production state builder.