mirror of
https://github.com/PerpetualSoftware/pad.git
synced 2026-10-26 22:37:03 +00:00
104f809e67f45afb67f34fbcc511af1ab81da32d
6 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 xarmian
|
e598bc4f64 |
feat(decision): effort, effort_medium and high_stakes on the attention set, staged; drain-paced backfill (TASK-3921) (#2236)
* feat(decision): effort, effort_medium and high_stakes on the attention set, staged; one-shot backfill (TASK-3921)
- attention set gains effort (Score), effort_medium (Choice) and high_stakes
(Noul) in the same call; needs_human_decision's text is unchanged.
- QuestionSet.Unsurfaced names the staged keys: WorkspaceFlags and
SurfacedDecisions skip them (no flag, no state read, no counter); the
decisions list and endpoint still return every row.
- attention chips only render keys that have a label.
- store: per-set enqueue body extracted (owedDecisionJobTx), OweDecisionJobs
(chunked, under the installed debounce) and ItemsMissingDecisionKey.
- Runner.BackfillEffort owes the attention set once for eligible items with a
needs_human_decision answer and no effort answer, marker
effort_decisions_backfilled written after every chunk; hooked at the end of
reconfigureDecisionsLocked (boot and later admin enable); makes no provider call.
- eval harness: titlebody now sends the live set; RESULTS.md rows pending.
- docs/api.md and tests updated.
Claude-Session: https://claude.ai/code/session_018oQ5EErQKHtBJK8yLpcDNF
* test(decision-eval): keep titlebody as the day-73 control, add titlebody-live; record the TASK-3957 and TASK-3921 runs
- titlebody again sends the day-73 pair (needs_human_decision + blocked), so it
stays comparable with its 09-22 row as the model-drift control.
- titlebody-live sends the live attention set on the same {title, body} state:
the TASK-3921 gate (needs_human_decision AUC 0.936, within .02 of 0.939).
- Every run prints the question keys it sends, before -dry too, so a row's
call shape is in its own log.
- RESULTS.md: rows for the one-question call (TASK-3957) and the
four-question call (TASK-3921).
Claude-Session: https://claude.ai/code/session_018oQ5EErQKHtBJK8yLpcDNF
* fix(decision): effort backfill lifecycle, staged keys off item reads, dry-run guard (TASK-3921; #r1 F1-F4)
- F1: startEffortBackfill checks stopped and does bg.Add under one
decisionTick.mu hold; BackfillEffort/EffortBackfillCandidates take a ctx
checked before each candidate read and each chunk; cancel returns ctx.Err()
and never writes the marker. stopDecisionTick cancels the run.
- F2: reconfigure cancels a running backfill; only a non-nil runner starts a
new one. A per-run generation replaces the atomic latch so a cancelled run
cannot clobber its successor.
- F3: item show / --agent / MCP item get use Runner.ItemReadDecisions, which
drops the set's Unsurfaced keys; the decisions endpoint stays raw. docs/api.md
updated.
- F4: the file dry run refuses a path with a -wal/-shm sibling or under
$HOME/.pad (refuseLiveDatabase, tested without the env var).
Claude-Session: https://claude.ai/code/session_018oQ5EErQKHtBJK8yLpcDNF
* test(decision): a chunk write that fails in the store writes no marker (TASK-3921; mutant M6)
The two failure tests fail the per-chunk test hook, so the path where
OweDecisionJobs itself returns an error was never exercised: a mutant that
ignores that error and marks anyway survived. The new test breaks the
decision_jobs table from the hook so the error comes back through the store,
asserts no marker, then restores the table and re-runs to completion.
Claude-Session: https://claude.ai/code/session_018oQ5EErQKHtBJK8yLpcDNF
* refactor(decision): drop the pre-marker cancel check; a cancel after the last chunk may mark (TASK-3921; M5, lead ruling)
Mutant M5 (that check removed) survived every test, and could not do
otherwise: a cancel can land there only after every chunk was owed, and the
marker means every candidate was ENQUEUED, not answered, so marking then is
true. The check is removed rather than kept as a guard no instrument can
turn red. The doc comments on BackfillEffort, cancelEffortBackfill and
startEffortBackfill now say a cancel before the last chunk writes no
marker and one after it may.
Claude-Session: https://claude.ai/code/session_018oQ5EErQKHtBJK8yLpcDNF
* docs(api): a backfill chunk in flight when decisions are disabled may still commit (TASK-3921; r2 L1, lead ruling bdce2116)
docs/api.md said that with no provider "no job row is ever written". That is
false for one window: a disable that lands while the one-shot effort
backfill is inside a chunk's write lets that chunk commit, and its jobs wait
until a provider returns. Ruled docs-only. The disable test's comment now
says it covers a backfill parked before its first chunk, not that window.
Sweep for the claim (CONVE-23): grep of docs, internal, cmd, scripts and
skills (.go, .md) for "no job row", "job row is", "no job is", "no jobs
are written/owed/enqueued" and "never written/owed/enqueued". Two hits are
about this mechanism: api.md:72 (fixed) and the test comment (narrowed).
Unchanged, still true:
- store/decisions.go:27 is about the WRITE DOORS, which still enqueue
nothing without a resolver.
- runner_test.go:238 is about the nil-provider runner, which never starts a
backfill.
- effort_backfill_lifecycle_test.go:128 covers a run parked before its
first chunk.
The other hits are unrelated mechanisms.
Claude-Session: https://claude.ai/code/session_018oQ5EErQKHtBJK8yLpcDNF
* fix(decision): pace the effort backfill on drain; dry run opens a copy; docs eligibility (TASK-3921; r3 M1, L2, L4)
- M1: the backfill owes one tick pass of items per chunk (20 default, wired
from the tick limit) and owes chunk k+1 only once chunk k's own jobs have
drained: none claimable now, by ClaimDecisionJobs' own predicate
(new Store.CountClaimableDecisionJobs). Debounced jobs do not hold it.
The wait polls at the tick interval, honours ctx, and a cancel writes
no marker; the marker still follows the last chunk being owed.
- L2: TestEffortBackfillDryRunOnFile copies the named file under GOTMPDIR
and opens only the copy.
- L4: docs/api.md states the real eligibility and the drain pacing.
Claude-Session: https://claude.ai/code/session_018oQ5EErQKHtBJK8yLpcDNF
* docs(decision): the Unsurfaced comment names ItemReadDecisions (TASK-3921; r4 L2)
The field's comment listed WorkspaceFlags and SurfacedDecisions as the
readers that skip staged keys, but not ItemReadDecisions, which item show
(and so --agent and MCP item get) has used since the r1 F3 fix. Comment only.
Claude-Session: https://claude.ai/code/session_018oQ5EErQKHtBJK8yLpcDNF
* test(decision): break the table before the LAST chunk so the store-failure test pins M6 (TASK-3921; lead ruling)
Since the drain pacing (
|
||
|
xarmian
|
b1785f8952 |
fix(decision): retire text-derived blocked, waiting_on_external and conventions_comments; keep a pending answer on the dashboard (TASK-3915, TASK-3924) (#2229)
* fix(decision): retire text-derived blocked, waiting_on_external and conventions_comments; keep a pending answer on the dashboard (TASK-3915, TASK-3924) IDEA-3906 half B, units 1 and 3. - The attention set asks one question, needs_human_decision. The text-derived blocked Noul is gone: blocked is the dependency graph's alone, and the dashboard no longer appends a "its text says it is blocked" entry. waiting_on_external flagged nothing on 10-17 and goes too. - conventions_comments is no longer registered in production. It took 42% of provider calls on 10-17 for 7 chip-level answers. The set definition and the PerComment machinery stay, since tests and the conventions eval gates use them. The web drops the comment-card chips and their store. - Stored answers to the retired keys and set need no migration: no registered question carries their keys, so they read current=false and the brief counts them as stale questions. - WorkspaceFlags pending tolerance: an above-threshold answer whose item has moved on is kept while a job is owed for it with no recorded failure and the set still applies. Without that, the dashboard entry vanishes for the whole wait for the re-ask, which TASK-3916's debounce will stretch to an hour. - The day-73 titlebody eval control keeps the retired blocked question text locally, so it still re-runs that shape. - docs/api.md and README updated. Claude-Session: https://claude.ai/code/session_018oQ5EErQKHtBJK8yLpcDNF * test(decision): a claimed re-ask still counts as pending; rewording one question keeps the others (TASK-3915 r1) r1 hunter Lows, all FIXED: the per-key rewording rule had lost its coverage with the one-question set, so a two-question test set keeps it; a claimed-but-in-flight job leg is added to the pending test; the ProductionRegistry comment's run-on join is split. Claude-Session: https://claude.ai/code/session_018oQ5EErQKHtBJK8yLpcDNF |
||
|
xarmian
|
4532eecffd |
feat(decision): conventions ask about the item without its comment trail, 12 questions a call (TASK-3119 U2a) (#1871)
* chore(eval): TASK-3119 U2 gates in one command; decision.BuildCommentState -gates runs three passes through the production question path: A U0's population on the U2a item state (links in, trail out), gated on precision not dropping below U0's on any convention that fired; B comment-borne breaks, each comment asked alone with its item's title and collection, gated on precision >= 0.9; C a measurement of 12+ questions in one call (MaxPerCall 12). BuildCommentState is the comment subject's one serialisation, shared by the harness now and the runner in U2b. Comments are clipped at the body bound (16k runes), not the trail's 2k. Claude-Session: https://claude.ai/code/session_01Q9WssCjFuHjSfLABoW4tmS * feat(decision): the conventions set asks about the item without its comment trail, 12 questions a call (TASK-3119 U2a) The conventions state carried the last 10 comments, clipped at 2,000 runes each: on docapp that was most of every call's input (15.7k tokens for one write across 3 calls), 720 items had a clipped comment, and a break past the clip was invisible anyway. Comments get their own subject in U2b. - QuestionSet.NoTrail leaves the trail out; the conventions set sets it. The state is exactly what the U2 gates measured (recent_trail [], links in): precision held at 100% on every convention that fires, recall rose (1286 40->80%, 2693 70->90%), 0 truncated states, ~954 input tokens per call. - MaxPerCall 5 -> 12, from measure C: up to 13 questions in one call gave identical precision and recall, 0 of 116 answers moved by more than 0.2. - A comment write no longer changes this set's state, so it costs no call. - Every conventions answer computed before this reads not-current until the item's next write (the state bytes changed): a fleet-wide re-ask, by design. - CLAUDE.md: the registered sets (it still said attention was the only one, since U1a) and the per-set state. Claude-Session: https://claude.ai/code/session_01Q9WssCjFuHjSfLABoW4tmS |
||
|
xarmian
|
deec62235e |
feat(decision): executable conventions, the server half (TASK-3119 U1a) (#1859)
* wip(decision): conventions set (TASK-3119 U1a) * chore(eval): executable-conventions eval through the production question path (TASK-3119 U0) Measures, before any chip exists, how often the decision provider says a hand-labelled item breaks a convention at the 0.9 threshold: per convention precision and recall (all cases and real-only), the false 'Breaks' rate of a behaviour convention, and the calls and tokens spent. State is decision.BuildItemState, the question a decision.Noul worded as the U1 contract says, the model decision.DefaultModel. The hand labels were posted to TASK-3119 before any score existed; the trail comment carrying them is dropped from every state, and the dry run refuses a state that still carries them. Reads TYPESAFE_API_KEY only. Claude-Session: https://claude.ai/code/session_01Q9WssCjFuHjSfLABoW4tmS * chore(eval): -links and -only for the conventions eval (TASK-3119 U1a) -links builds each state as the conventions set does, with the item's links (from the dump: the parent link only; blocking links are not in pad item show's JSON). -only runs one convention. Used for the lead's U1a gate: CONVE-1286 rerun with links. Claude-Session: https://claude.ai/code/session_01Q9WssCjFuHjSfLABoW4tmS * fix(decision): liveness before EACH split call (TASK-3119 U1a, codex r1) A split set makes several provider calls; the liveness check ran once before them, so an item or workspace deleted between calls was still sent in the later ones. It now runs immediately before every call. A test deletes the item from inside the first call and the second is not sent; checking only once fails it. Claude-Session: https://claude.ai/code/session_01Q9WssCjFuHjSfLABoW4tmS * fix(decision): decision_check has no default; allow-list the eval's decode (TASK-3119 U1a) A default "on" was written into every new convention: imports gained a decision_check field (artifact import parity) and an activated built-in no longer matched its seed (TASK-3462 would read every one as diverged). Absent already reads as on, so the field carries no default, in the template and both migrations. The conventions eval's readJSON is a raw decode the reserved-field guard counts; allow-listed with its reason, like the attention harness. Claude-Session: https://claude.ai/code/session_01Q9WssCjFuHjSfLABoW4tmS |
||
|
xarmian
|
7512f5500b |
docs(decision): README decision-provider section, and the day-73 eval scripts with their recorded baseline (TASK-3123) (#1593)
* docs(decision): README decision-provider section, and the day-73 eval scripts with their recorded baseline (TASK-3123) Housekeeping for PLAN-3114 tranche 1; no behaviour change. - README: a Decision provider (optional) section covering what uses it, that answers are asynchronous, configuration precedence (config file, then the admin setting, then the environment; PAD_DECISION_PROVIDER=none as the operator off-switch), and exactly what leaves the box and where. - scripts/decision-eval/eval.py and eval2.py: the day-73 scripts, as run (standard library, raw systemone API, key read from TYPESAFE_API_KEY). A read and a pattern scan found no key or token literal. - scripts/decision-eval/README.md: what each script measures, how to run it, how to regenerate the corpora, and the day-73 baseline. A, B and D were recomputed from the saved result files with the scripts' own summary logic and match PLAN-3114 (D AUC 0.939; the B >=0.8 bucket 86.8%). C comes from PLAN-3114's trail. eval.py --dry-run over the saved corpora queues the same 360 items the day-73 run did. - scripts/decision-eval/.gitignore: the corpora are private workspace exports and the results embed their content, so neither can be committed; neither was. The CHANGELOG line is queued as DOC-3242 (draft highlights for the release after v0.15.0, per PLAYB-1160), since releases take their changelog from goreleaser and a highlights DOC. Claude-Session: https://claude.ai/code/session_016PXRzXhokr61CLbbu3Txj9 * docs(decision): the README section states what off, the triggers, server info and disabling actually do (TASK-3123, codex round 1) Codex fact-checked every claim against the code, and four were too broad. Off is not "every surface unchanged": the decisions list is empty and playbook match answers 404. Jobs are also queued by restore and by comment edit and delete (verified at items.go:614/3207/3437/5459 and comments.go:133/196/416), and a closed item's job is dropped at the tick. pad server info cannot see the admin setting. Disabling stops new calls without a restart, but an in-flight request may finish. The scripts README now says eval2.py runs at the top level. Claude-Session: https://claude.ai/code/session_016PXRzXhokr61CLbbu3Txj9 |
||
|
xarmian
|
3b7665a469 |
chore(decision-eval): attention threshold eval — no threshold passes the pre-registered rule (TASK-3137) (#1424)
* chore(decision-eval): attention threshold eval through the production state builder (TASK-3137) The instrument and its recorded results. The pre-registered rule is met by no threshold on the production state, so AttentionThreshold is unchanged; the ablation shows the 70/70 labels are historical while production asks about open items in the present. See RESULTS.md. Claude-Session: https://claude.ai/code/session_01ByMjAhvdWgbAWWeGE9jwWd * chore(decision-eval): replay each item to its last open moment (TASK-3137) The lead-ruled instrument: BuildItemState over the item as of just before its first logged terminal transition (r1), and at filing (r0), with the same 140 labels. The pre-registered rule (P >= 90%, R >= 60% on r1) is met by no threshold, so AttentionThreshold is unchanged. RESULTS.md records the finding: collection-membership labels cannot see the transient human waits that a task at its last open moment really has. Claude-Session: https://claude.ai/code/session_01ByMjAhvdWgbAWWeGE9jwWd * test(models): classify the eval tool's generic decode site (TASK-3137) The BUG-2685 guard found the new scripts/decision-eval readJSON helper as an unclassified interface-destination decode. It decodes item, comment, history and activity dumps, never a collection definition. Claude-Session: https://claude.ai/code/session_01ByMjAhvdWgbAWWeGE9jwWd |