* chore(eval): TASK-3119 U2 gates in one command; decision.BuildCommentState
-gates runs three passes through the production question path:
A U0's population on the U2a item state (links in, trail out), gated on
precision not dropping below U0's on any convention that fired;
B comment-borne breaks, each comment asked alone with its item's title
and collection, gated on precision >= 0.9;
C a measurement of 12+ questions in one call (MaxPerCall 12).
BuildCommentState is the comment subject's one serialisation, shared by
the harness now and the runner in U2b. Comments are clipped at the body
bound (16k runes), not the trail's 2k.
Claude-Session: https://claude.ai/code/session_01Q9WssCjFuHjSfLABoW4tmS
* feat(decision): the conventions set asks about the item without its comment trail, 12 questions a call (TASK-3119 U2a)
The conventions state carried the last 10 comments, clipped at 2,000 runes
each: on docapp that was most of every call's input (15.7k tokens for one
write across 3 calls), 720 items had a clipped comment, and a break past the
clip was invisible anyway. Comments get their own subject in U2b.
- QuestionSet.NoTrail leaves the trail out; the conventions set sets it. The
state is exactly what the U2 gates measured (recent_trail [], links in):
precision held at 100% on every convention that fires, recall rose (1286
40->80%, 2693 70->90%), 0 truncated states, ~954 input tokens per call.
- MaxPerCall 5 -> 12, from measure C: up to 13 questions in one call gave
identical precision and recall, 0 of 116 answers moved by more than 0.2.
- A comment write no longer changes this set's state, so it costs no call.
- Every conventions answer computed before this reads not-current until the
item's next write (the state bytes changed): a fleet-wide re-ask, by design.
- CLAUDE.md: the registered sets (it still said attention was the only one,
since U1a) and the per-set state.
Claude-Session: https://claude.ai/code/session_01Q9WssCjFuHjSfLABoW4tmS
* wip(decision): conventions set (TASK-3119 U1a)
* chore(eval): executable-conventions eval through the production question path (TASK-3119 U0)
Measures, before any chip exists, how often the decision provider says a
hand-labelled item breaks a convention at the 0.9 threshold: per
convention precision and recall (all cases and real-only), the false
'Breaks' rate of a behaviour convention, and the calls and tokens spent.
State is decision.BuildItemState, the question a decision.Noul worded as
the U1 contract says, the model decision.DefaultModel.
The hand labels were posted to TASK-3119 before any score existed; the
trail comment carrying them is dropped from every state, and the dry run
refuses a state that still carries them. Reads TYPESAFE_API_KEY only.
Claude-Session: https://claude.ai/code/session_01Q9WssCjFuHjSfLABoW4tmS
* chore(eval): -links and -only for the conventions eval (TASK-3119 U1a)
-links builds each state as the conventions set does, with the item's
links (from the dump: the parent link only; blocking links are not in
pad item show's JSON). -only runs one convention. Used for the lead's
U1a gate: CONVE-1286 rerun with links.
Claude-Session: https://claude.ai/code/session_01Q9WssCjFuHjSfLABoW4tmS
* fix(decision): liveness before EACH split call (TASK-3119 U1a, codex r1)
A split set makes several provider calls; the liveness check ran once
before them, so an item or workspace deleted between calls was still
sent in the later ones. It now runs immediately before every call. A
test deletes the item from inside the first call and the second is not
sent; checking only once fails it.
Claude-Session: https://claude.ai/code/session_01Q9WssCjFuHjSfLABoW4tmS
* fix(decision): decision_check has no default; allow-list the eval's decode (TASK-3119 U1a)
A default "on" was written into every new convention: imports gained a
decision_check field (artifact import parity) and an activated built-in
no longer matched its seed (TASK-3462 would read every one as diverged).
Absent already reads as on, so the field carries no default, in the
template and both migrations.
The conventions eval's readJSON is a raw decode the reserved-field guard
counts; allow-listed with its reason, like the attention harness.
Claude-Session: https://claude.ai/code/session_01Q9WssCjFuHjSfLABoW4tmS
* docs(decision): README decision-provider section, and the day-73 eval scripts with their recorded baseline (TASK-3123)
Housekeeping for PLAN-3114 tranche 1; no behaviour change.
- README: a Decision provider (optional) section covering what uses it,
that answers are asynchronous, configuration precedence (config file,
then the admin setting, then the environment; PAD_DECISION_PROVIDER=none
as the operator off-switch), and exactly what leaves the box and where.
- scripts/decision-eval/eval.py and eval2.py: the day-73 scripts, as run
(standard library, raw systemone API, key read from TYPESAFE_API_KEY). A
read and a pattern scan found no key or token literal.
- scripts/decision-eval/README.md: what each script measures, how to run
it, how to regenerate the corpora, and the day-73 baseline. A, B and D
were recomputed from the saved result files with the scripts' own
summary logic and match PLAN-3114 (D AUC 0.939; the B >=0.8 bucket
86.8%). C comes from PLAN-3114's trail. eval.py --dry-run over the saved
corpora queues the same 360 items the day-73 run did.
- scripts/decision-eval/.gitignore: the corpora are private workspace
exports and the results embed their content, so neither can be
committed; neither was.
The CHANGELOG line is queued as DOC-3242 (draft highlights for the
release after v0.15.0, per PLAYB-1160), since releases take their
changelog from goreleaser and a highlights DOC.
Claude-Session: https://claude.ai/code/session_016PXRzXhokr61CLbbu3Txj9
* docs(decision): the README section states what off, the triggers, server info and disabling actually do (TASK-3123, codex round 1)
Codex fact-checked every claim against the code, and four were too broad.
Off is not "every surface unchanged": the decisions list is empty and
playbook match answers 404. Jobs are also queued by restore and by comment
edit and delete (verified at items.go:614/3207/3437/5459 and
comments.go:133/196/416), and a closed item's job is dropped at the tick.
pad server info cannot see the admin setting. Disabling stops new calls
without a restart, but an in-flight request may finish. The scripts README
now says eval2.py runs at the top level.
Claude-Session: https://claude.ai/code/session_016PXRzXhokr61CLbbu3Txj9
* chore(decision-eval): attention threshold eval through the production state builder (TASK-3137)
The instrument and its recorded results. The pre-registered rule is met by
no threshold on the production state, so AttentionThreshold is unchanged;
the ablation shows the 70/70 labels are historical while production asks
about open items in the present. See RESULTS.md.
Claude-Session: https://claude.ai/code/session_01ByMjAhvdWgbAWWeGE9jwWd
* chore(decision-eval): replay each item to its last open moment (TASK-3137)
The lead-ruled instrument: BuildItemState over the item as of just before
its first logged terminal transition (r1), and at filing (r0), with the same
140 labels. The pre-registered rule (P >= 90%, R >= 60% on r1) is met by no
threshold, so AttentionThreshold is unchanged. RESULTS.md records the
finding: collection-membership labels cannot see the transient human waits
that a task at its last open moment really has.
Claude-Session: https://claude.ai/code/session_01ByMjAhvdWgbAWWeGE9jwWd
* test(models): classify the eval tool's generic decode site (TASK-3137)
The BUG-2685 guard found the new scripts/decision-eval readJSON helper as an
unclassified interface-destination decode. It decodes item, comment, history
and activity dumps, never a collection definition.
Claude-Session: https://claude.ai/code/session_01ByMjAhvdWgbAWWeGE9jwWd
xarmian