mirror of
https://github.com/PerpetualSoftware/pad.git
synced 2026-10-24 21:43:26 +00:00
feat(decision): conventions ask about the item without its comment trail, 12 questions a call (TASK-3119 U2a) (#1871)
* chore(eval): TASK-3119 U2 gates in one command; decision.BuildCommentState -gates runs three passes through the production question path: A U0's population on the U2a item state (links in, trail out), gated on precision not dropping below U0's on any convention that fired; B comment-borne breaks, each comment asked alone with its item's title and collection, gated on precision >= 0.9; C a measurement of 12+ questions in one call (MaxPerCall 12). BuildCommentState is the comment subject's one serialisation, shared by the harness now and the runner in U2b. Comments are clipped at the body bound (16k runes), not the trail's 2k. Claude-Session: https://claude.ai/code/session_01Q9WssCjFuHjSfLABoW4tmS * feat(decision): the conventions set asks about the item without its comment trail, 12 questions a call (TASK-3119 U2a) The conventions state carried the last 10 comments, clipped at 2,000 runes each: on docapp that was most of every call's input (15.7k tokens for one write across 3 calls), 720 items had a clipped comment, and a break past the clip was invisible anyway. Comments get their own subject in U2b. - QuestionSet.NoTrail leaves the trail out; the conventions set sets it. The state is exactly what the U2 gates measured (recent_trail [], links in): precision held at 100% on every convention that fires, recall rose (1286 40->80%, 2693 70->90%), 0 truncated states, ~954 input tokens per call. - MaxPerCall 5 -> 12, from measure C: up to 13 questions in one call gave identical precision and recall, 0 of 116 answers moved by more than 0.2. - A comment write no longer changes this set's state, so it costs no call. - Every conventions answer computed before this reads not-current until the item's next write (the state bytes changed): a fleet-wide re-ask, by design. - CLAUDE.md: the registered sets (it still said attention was the only one, since U1a) and the per-set state. Claude-Session: https://claude.ai/code/session_01Q9WssCjFuHjSfLABoW4tmS
This commit is contained in:
xarmian
@@ -108,7 +108,7 @@ REST API at `/api/v1/`. Key endpoints:
|
||||
- `POST /workspaces/{ws}/attachments/{id}/attach {item}` — bind an UNATTACHED attachment (item_id NULL, what `pad attachment upload -` leaves) and its variants to an item (TASK-2247). Share pages serve only attachments the shared item (or an item in the shared collection) owns, so an embedded `-` upload renders as a placeholder there until it is attached. Never a move: an attached row answers 409 when the caller can see its owner, otherwise the shared attachment 404. The caller must be able to see and edit the item, and must have uploaded the attachment or be a workspace owner; restricted members and guests cannot read unattached rows, so they get the 404. Advertised as `attachment_attach` in `GET /server/capabilities`; the CLI (`pad attachment attach <id> <item-ref>`) refuses against a server without it. Not on MCP (`pad_attachment` has no upload)
|
||||
- `GET/POST /workspaces/{ws}/items/{slug}/reminders` — item reminders (IDEA-2641). POST arms one; `remind_at` is an RFC3339 **instant** and a bare date is refused, not read as midnight
|
||||
- `PATCH/DELETE /workspaces/{ws}/reminders/{id}`, `POST /workspaces/{ws}/reminders/{id}/ack` — re-arm (clears both fire marks), disarm, acknowledge. Permission is the ITEM's; the reminder has no separate owner
|
||||
- `GET /workspaces/{ws}/items/{slug}/decisions` — the item's latest typed-decision answer per question (PLAN-3114 / TASK-3117), each with `current: true` only when it was computed from the item's PRESENT state (a sha256 of the exact state bytes sent to the provider — title, collection, fields, bounded body, last 10 comments) AND for the question as registered now under the model pinned now — rewording a question or changing the model re-asks. A third condition since TASK-3118: the set must still APPLY to the item (its collection is covered and the set's eligibility predicate accepts the item as it stands). A collection schema edit can make an item terminal without changing a byte of its state, and its answers then read `current: false`. Evaluation is asynchronous: item create / update / move and comment create enqueue a coalesced `decision_jobs` row in their own transaction and a server tick does the provider call, so nothing on a write path waits on the network. System fan-out rewrites (title-rename cascade, field migration, attachment remap) deliberately do NOT enqueue — their items read `current: false` until the next direct write — and neither does workspace import, whose items have no decisions until their next direct write. With no provider configured the list is empty and no job row is ever written. The single-item GET (and `pad item show --format json` / `--agent`) carries the same list as `decisions` when non-empty. The one registered set is `attention` (TASK-3118): three Nouls (`needs_human_decision`, `blocked`, `waiting_on_external`), asked only about OPEN items in NON-SYSTEM collections. A job owed for a terminal item is dropped at the tick without a provider call, and reopening the item re-enqueues it
|
||||
- `GET /workspaces/{ws}/items/{slug}/decisions` — the item's latest typed-decision answer per question (PLAN-3114 / TASK-3117), each with `current: true` only when it was computed from the item's PRESENT state (a sha256 of the exact state bytes sent to the provider — title, collection, fields, bounded body, and per set: the last 10 comments for `attention`, the parent and blocking links and NO comments for `conventions`) AND for the question as registered now under the model pinned now — rewording a question or changing the model re-asks. A third condition since TASK-3118: the set must still APPLY to the item (its collection is covered and the set's eligibility predicate accepts the item as it stands). A collection schema edit can make an item terminal without changing a byte of its state, and its answers then read `current: false`. Evaluation is asynchronous: item create / update / move and comment create enqueue a coalesced `decision_jobs` row in their own transaction and a server tick does the provider call, so nothing on a write path waits on the network. System fan-out rewrites (title-rename cascade, field migration, attachment remap) deliberately do NOT enqueue — their items read `current: false` until the next direct write — and neither does workspace import, whose items have no decisions until their next direct write. With no provider configured the list is empty and no job row is ever written. The single-item GET (and `pad item show --format json` / `--agent`) carries the same list as `decisions` when non-empty. Two sets are registered. `attention` (TASK-3118): three Nouls (`needs_human_decision`, `blocked`, `waiting_on_external`), asked only about OPEN items in NON-SYSTEM collections. A job owed for a terminal item is dropped at the tick without a provider call, and reopening the item re-enqueues it. `conventions` (TASK-3119): one Noul per active `always` convention whose `decision_check` is not `off` and whose body is at most 4 KB, keyed `conv:<ref>`, asked about items in non-system collections, at most 12 per provider call; the web shows a "Possibly breaks CONVE-N" chip at p >= 0.9 and nothing below it. Its state carries no comment trail (U2a), so a comment write changes nothing it reads and costs it no call. Each evaluation that reaches the provider logs one `decision evaluation` line with its calls and tokens
|
||||
- `POST /workspaces/{ws}/playbooks/match {text}` — which ACTIVE playbook (or `none`) free text asks for, answered by the decision provider synchronously (TASK-3120; `pad playbook match -- "<text>"`, `pad_playbook.action: match`). 404 `decision_provider_unavailable` with no provider, so callers fall back to slug/trigger routing. Each call that reaches the provider spends from its own bucket, 30/min per user, burst 5, charged immediately before the provider call, so an unconfigured 404, a 400 for the text, a too-many-playbooks 400 or a zero-playbook answer consumes nothing (a request-too-large 400 is raised inside that call and has spent its token); over it is `429` with `Retry-After` (TASK-3141). No env knob by design: tuning belongs in the instance-admin decision-provider setting
|
||||
- `GET /workspaces/{ws}/dashboard` — computed project overview (active items, plans, attention, blockers). Also carries `pending_reminders`: fired-but-unacknowledged reminders, which is the delivery path on any instance with no webhook configured. With a decision provider, `attention` also carries `needs_human` entries (the item's latest `needs_human_decision` answer is >= 0.7), and a text-derived `blocked` entry for an item whose `blocked` answer is >= 0.7 and which the dependency graph has NOT already flagged — the graph stays primary. Only CURRENT answers count, the same rule the decisions endpoint applies (the item's present state, the question as registered now, the model pinned now), so an item edited or commented since its last evaluation drops off until the tick re-asks. The state check costs one item-plus-trail read per ABOVE-threshold item, never per workspace item. A failing provider (any owed `attention` job with a recorded error) marks the section `attention.decisions` in `degraded_sections`. Text-derived `blocked` entries therefore also reach `pad project stale` (which filters on type); `needs_human` does not
|
||||
- `GET /workspaces/{ws}/activity` — workspace activity feed (enriched with item titles + change details)
|
||||
|
||||
@@ -0,0 +1,38 @@
|
||||
package decision
|
||||
|
||||
import (
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
)
|
||||
|
||||
// TASK-3119 U2: a comment is asked about on its own (the "comment subject"),
|
||||
// with its item's title and collection for context, instead of riding in
|
||||
// every item state as part of a trail that was clipped at 2,000 runes per
|
||||
// comment. This is that state's ONE serialisation, shared by the eval
|
||||
// harness (which measured it before it shipped) and the runner.
|
||||
|
||||
// maxCommentSubjectRunes bounds the comment. It is the body's bound, not the
|
||||
// trail's: a comment asked about alone has the request to itself. Census at
|
||||
// the time (the :7777 DB): 20 of 23,618 comments exceed it.
|
||||
const maxCommentSubjectRunes = maxStateBodyRunes
|
||||
|
||||
// CommentState is the state a comment subject is asked about. Like
|
||||
// ItemState, part of the hash: a change here re-asks every comment.
|
||||
type CommentState struct {
|
||||
Title string `json:"title"`
|
||||
Collection string `json:"collection"`
|
||||
Comment string `json:"comment"`
|
||||
}
|
||||
|
||||
// BuildCommentState builds and serialises one comment's state.
|
||||
func BuildCommentState(itemTitle, collectionSlug, body string) (BuiltState, error) {
|
||||
clipped, truncated := clipRunes(body, maxCommentSubjectRunes)
|
||||
b, err := json.Marshal(CommentState{Title: itemTitle, Collection: collectionSlug, Comment: clipped})
|
||||
if err != nil {
|
||||
return BuiltState{}, fmt.Errorf("decision comment state: marshal: %w", err)
|
||||
}
|
||||
sum := sha256.Sum256(b)
|
||||
return BuiltState{Bytes: b, Hash: hex.EncodeToString(sum[:]), Truncated: truncated}, nil
|
||||
}
|
||||
@@ -0,0 +1,46 @@
|
||||
package decision
|
||||
|
||||
import (
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// The comment subject's state is exactly title, collection and the comment,
|
||||
// hashed over the bytes that are sent.
|
||||
func TestBuildCommentState_ShapeAndHash(t *testing.T) {
|
||||
st, err := BuildCommentState("Set up staging", "tasks", "the password is hunter2")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
want := `{"title":"Set up staging","collection":"tasks","comment":"the password is hunter2"}`
|
||||
if string(st.Bytes) != want {
|
||||
t.Fatalf("state = %s\nwant %s", st.Bytes, want)
|
||||
}
|
||||
sum := sha256.Sum256(st.Bytes)
|
||||
if st.Hash != hex.EncodeToString(sum[:]) || st.Truncated {
|
||||
t.Fatalf("hash/truncated = %s/%v", st.Hash, st.Truncated)
|
||||
}
|
||||
}
|
||||
|
||||
// A comment is clipped at the BODY bound, not the trail's 2,000 runes: a
|
||||
// credential at rune 2,786 of a long status update must reach the provider.
|
||||
func TestBuildCommentState_ClipsAtTheBodyBound(t *testing.T) {
|
||||
long := strings.Repeat("é", maxStateCommentRunes+800) + " SECRET"
|
||||
st, err := BuildCommentState("t", "tasks", long)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if st.Truncated || !strings.Contains(string(st.Bytes), "SECRET") {
|
||||
t.Fatalf("a %d-rune comment was clipped (truncated=%v)", maxStateCommentRunes+807, st.Truncated)
|
||||
}
|
||||
over := strings.Repeat("a", maxCommentSubjectRunes+1)
|
||||
st, err = BuildCommentState("t", "tasks", over)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !st.Truncated {
|
||||
t.Fatal("a comment over the bound was not reported truncated")
|
||||
}
|
||||
}
|
||||
@@ -38,8 +38,12 @@ const (
|
||||
// logged by ref.
|
||||
conventionMaxBodyBytes = 4096
|
||||
|
||||
// conventionsPerCall bounds the questions in one provider call.
|
||||
conventionsPerCall = 5
|
||||
// conventionsPerCall bounds the questions in one provider call. 12, from
|
||||
// the U2 gates' measure C (TASK-3119 trail): up to 13 questions in ONE
|
||||
// call gave identical precision and recall to one question per call on
|
||||
// every convention, and moved 0 of 116 answers by more than 0.2. The
|
||||
// state is sent once per call, so fewer calls is most of the saving.
|
||||
conventionsPerCall = 12
|
||||
|
||||
// decisionCheckField is the conventions schema field that switches a
|
||||
// convention's check off (value "off"); absent or anything else is on.
|
||||
@@ -61,6 +65,7 @@ func ConventionsSet() QuestionSet {
|
||||
Name: ConventionsSetName,
|
||||
Resolve: resolveConventions,
|
||||
WithLinks: true,
|
||||
NoTrail: true,
|
||||
MaxPerCall: conventionsPerCall,
|
||||
// User collections only: a convention or playbook is not work an
|
||||
// item-content rule is about.
|
||||
|
||||
@@ -123,7 +123,7 @@ func TestConventions_ResolveScope(t *testing.T) {
|
||||
// stored, and the observer sees each call.
|
||||
func TestConventions_SplitsCallsAndStoresEveryAnswer(t *testing.T) {
|
||||
fx := newConvFixture(t)
|
||||
for i := 0; i < 7; i++ {
|
||||
for i := 0; i < 13; i++ {
|
||||
fx.convention(t, "Rule", `{"status":"active","trigger":"always"}`, "rule "+string(rune('a'+i)))
|
||||
}
|
||||
item := fx.task(t, "A task")
|
||||
@@ -134,10 +134,11 @@ func TestConventions_SplitsCallsAndStoresEveryAnswer(t *testing.T) {
|
||||
if len(fx.f.requests) != 2 {
|
||||
t.Fatalf("provider calls = %d, want 2", len(fx.f.requests))
|
||||
}
|
||||
// MaxPerCall 12 (U2 measure C): 13 questions split 12 + 1.
|
||||
sizes := []int{len(fx.f.requests[0].Questions), len(fx.f.requests[1].Questions)}
|
||||
sort.Ints(sizes)
|
||||
if sizes[0] != 2 || sizes[1] != 5 {
|
||||
t.Fatalf("questions per call = %v, want [2 5]", sizes)
|
||||
if sizes[0] != 1 || sizes[1] != 12 {
|
||||
t.Fatalf("questions per call = %v, want [1 12]", sizes)
|
||||
}
|
||||
if strings.Join(fx.usage, ",") != "conventions,conventions" {
|
||||
t.Fatalf("usage observer saw %v", fx.usage)
|
||||
@@ -152,8 +153,8 @@ func TestConventions_SplitsCallsAndStoresEveryAnswer(t *testing.T) {
|
||||
current++
|
||||
}
|
||||
}
|
||||
if current != 7 {
|
||||
t.Fatalf("current conventions answers = %d, want 7", current)
|
||||
if current != 13 {
|
||||
t.Fatalf("current conventions answers = %d, want 13", current)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -294,7 +295,7 @@ func TestConventions_CurrencyFollowsTheConventionAndTheLinks(t *testing.T) {
|
||||
// deleted while the first call is in flight is not sent in the second.
|
||||
func TestConventions_DeleteBetweenSplitCallsStopsTheRest(t *testing.T) {
|
||||
fx := newConvFixture(t)
|
||||
for i := 0; i < 7; i++ {
|
||||
for i := 0; i < 13; i++ {
|
||||
fx.convention(t, "Rule", `{"status":"active","trigger":"always"}`, "rule "+string(rune('a'+i)))
|
||||
}
|
||||
item := fx.task(t, "A task")
|
||||
@@ -351,7 +352,7 @@ func TestConventions_EvaluationLogsItsSpend(t *testing.T) {
|
||||
fx := newConvFixture(t)
|
||||
var buf bytes.Buffer
|
||||
fx.r.logger = slog.New(slog.NewJSONHandler(&buf, nil))
|
||||
for i := 0; i < 7; i++ {
|
||||
for i := 0; i < 13; i++ {
|
||||
fx.convention(t, "Rule", `{"status":"active","trigger":"always"}`, "SECRET-RULE-"+string(rune('a'+i)))
|
||||
}
|
||||
item, err := fx.s.CreateItem(fx.ws.ID, fx.tasks.ID, models.ItemCreate{Title: "SECRET-TITLE", Fields: `{"status":"open"}`, Content: "SECRET-BODY"})
|
||||
@@ -366,10 +367,10 @@ func TestConventions_EvaluationLogsItsSpend(t *testing.T) {
|
||||
t.Fatalf("spend lines = %d, want 1: %s", len(lines), buf.String())
|
||||
}
|
||||
got := lines[0]
|
||||
// Two calls (5 + 2 questions) at the fake's 10 in / 5 out each.
|
||||
// Two calls (12 + 1 questions) at the fake's 10 in / 5 out each.
|
||||
want := map[string]any{
|
||||
"level": "INFO", "set": ConventionsSetName, "workspace_id": fx.ws.ID,
|
||||
"item_ref": item.Ref, "item_id": item.ID, "questions": float64(7), "calls": float64(2),
|
||||
"item_ref": item.Ref, "item_id": item.ID, "questions": float64(13), "calls": float64(2),
|
||||
"input_tokens": float64(20), "output_tokens": float64(10), "truncated": false, "outcome": "stored",
|
||||
}
|
||||
if item.Ref == "" {
|
||||
@@ -416,3 +417,46 @@ func TestConventions_FailedEvaluationLogsItsSpend(t *testing.T) {
|
||||
t.Fatalf("the spend line carries the provider's error text: %s", buf.String())
|
||||
}
|
||||
}
|
||||
|
||||
// TASK-3119 U2a: the conventions set asks about the ITEM, so its state carries
|
||||
// no comment trail (comments get their own subject in U2b). Two consequences,
|
||||
// both pinned: no comment text reaches the provider in this set's state, and
|
||||
// a comment write does not change the state, so it costs no conventions call.
|
||||
func TestConventions_StateHasNoTrail_AndACommentCostsNoCall(t *testing.T) {
|
||||
fx := newConvFixture(t)
|
||||
fx.convention(t, "Rule", `{"status":"active","trigger":"always"}`, "rule")
|
||||
item := fx.task(t, "A task")
|
||||
if _, err := fx.s.CreateComment(fx.ws.ID, item.ID, "", models.CommentCreate{Author: "wren", Body: "TRAIL-SENTINEL one"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if called, err := fx.r.Evaluate(context.Background(), item.ID, ConventionsSetName); err != nil || !called {
|
||||
t.Fatalf("evaluate: called=%v err=%v", called, err)
|
||||
}
|
||||
if len(fx.f.rawBodies) != 1 {
|
||||
t.Fatalf("provider calls = %d, want 1", len(fx.f.rawBodies))
|
||||
}
|
||||
body := fx.f.rawBodies[0]
|
||||
if strings.Contains(body, "TRAIL-SENTINEL") {
|
||||
t.Fatalf("a comment reached the conventions state: %s", body)
|
||||
}
|
||||
if !strings.Contains(body, `"recent_trail":[]`) || !strings.Contains(body, `"links":[]`) {
|
||||
t.Fatalf(`state is not the U2a shape (empty trail, links present): %s`, body)
|
||||
}
|
||||
|
||||
// A further comment changes nothing this set reads.
|
||||
if _, err := fx.s.CreateComment(fx.ws.ID, item.ID, "", models.CommentCreate{Author: "wren", Body: "TRAIL-SENTINEL two"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if called, err := fx.r.Evaluate(context.Background(), item.ID, ConventionsSetName); err != nil || called {
|
||||
t.Fatalf("re-evaluate after a comment: called=%v err=%v, want no call", called, err)
|
||||
}
|
||||
ds, err := fx.r.Decisions(item.ID)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, d := range ds {
|
||||
if d.QuestionSet == ConventionsSetName && !d.Current {
|
||||
t.Fatalf("a comment made the conventions answer not current: %+v", d)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -64,6 +64,15 @@ type QuestionSet struct {
|
||||
// builder's bytes, and so every other set's state hashes, are unchanged.
|
||||
WithLinks bool
|
||||
|
||||
// NoTrail leaves the comment trail out of the state (TASK-3119 U2a). The
|
||||
// conventions set asks about the ITEM: a trail is discussion, it was
|
||||
// clipped at 2,000 runes per comment (so a break past that was invisible
|
||||
// anyway), and it was most of every call's input. Comments are asked about
|
||||
// one at a time instead (U2b). The U2 gates measured this state against
|
||||
// U0's labels: precision held at 100%, recall rose on two conventions, and
|
||||
// no state was truncated.
|
||||
NoTrail bool
|
||||
|
||||
// MaxPerCall bounds the questions sent in one provider call; a larger
|
||||
// set is split into several calls. Zero sends them all in one.
|
||||
MaxPerCall int
|
||||
@@ -408,7 +417,7 @@ func (r *Runner) State(itemID string) (*models.Item, BuiltState, error) {
|
||||
}
|
||||
|
||||
// stateFor builds the item's current state as qs sees it: with its links
|
||||
// when qs is WithLinks.
|
||||
// when qs is WithLinks, and without the comment trail when qs is NoTrail.
|
||||
func (r *Runner) stateFor(itemID string, qs QuestionSet) (*models.Item, BuiltState, error) {
|
||||
item, err := r.store.GetItem(itemID)
|
||||
if err != nil {
|
||||
@@ -417,9 +426,12 @@ func (r *Runner) stateFor(itemID string, qs QuestionSet) (*models.Item, BuiltSta
|
||||
if item == nil || item.DeletedAt != nil {
|
||||
return nil, BuiltState{}, ErrItemGone
|
||||
}
|
||||
comments, err := r.store.RecentComments(itemID, RecentTrailWindow)
|
||||
if err != nil {
|
||||
return nil, BuiltState{}, err
|
||||
var comments []models.Comment
|
||||
if !qs.NoTrail {
|
||||
comments, err = r.store.RecentComments(itemID, RecentTrailWindow)
|
||||
if err != nil {
|
||||
return nil, BuiltState{}, err
|
||||
}
|
||||
}
|
||||
if !qs.WithLinks {
|
||||
st, err := BuildItemState(item, comments)
|
||||
|
||||
@@ -0,0 +1,382 @@
|
||||
package main
|
||||
|
||||
// TASK-3119 U2 gates, one command (-gates). Three passes, all through the
|
||||
// production question path and model:
|
||||
//
|
||||
// A U0's labelled population, re-scored on the U2a ITEM state: links in,
|
||||
// comment trail OUT. Bar: on every convention that fired in U0,
|
||||
// precision does not drop below U0's.
|
||||
// B Comment-borne breaks (comments.json, labels posted to TASK-3119 before
|
||||
// any score): each comment asked about ALONE, with its item's title and
|
||||
// collection (decision.BuildCommentState, the U2b subject). Bar:
|
||||
// precision >= 0.9 overall, and it must fire at least once.
|
||||
// C Measurement, not a gate: each population item asked every question of
|
||||
// batch12.json (the 12 conventions docapp asked before the behaviour
|
||||
// rules were switched off) plus its labelled one, in ONE call, on the A
|
||||
// state. Says whether MaxPerCall can grow from 5 to 12 without moving
|
||||
// the answers, and what one such call costs.
|
||||
//
|
||||
// -gates -dry-run builds every state and question and calls nothing.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/PerpetualSoftware/pad/internal/decision"
|
||||
)
|
||||
|
||||
type commentCase struct {
|
||||
Convention string `json:"convention"`
|
||||
ID string `json:"id"`
|
||||
Violates bool `json:"violates"`
|
||||
Title string `json:"title"`
|
||||
Collection string `json:"collection"`
|
||||
Comment string `json:"comment"`
|
||||
}
|
||||
|
||||
// u0Baseline is U0's result for each convention that FIRED (TASK-3119 trail:
|
||||
// results.txt, and the -links run for CONVE-1286, which is how the set asks
|
||||
// it). A convention that never fired in U0 has no precision to hold.
|
||||
var u0Baseline = map[string]struct{ precision, recall float64 }{
|
||||
"CONVE-2": {1.00, 0.50},
|
||||
"CONVE-1285": {1.00, 0.75},
|
||||
"CONVE-1286": {1.00, 0.40},
|
||||
"CONVE-2693": {1.00, 0.70},
|
||||
}
|
||||
|
||||
const commentPrecisionBar = 0.9
|
||||
|
||||
type gateJob struct {
|
||||
state []byte
|
||||
truncated bool
|
||||
questions map[string]decision.Question
|
||||
}
|
||||
|
||||
type gateResult struct {
|
||||
answers map[string]decision.Answer
|
||||
usage decision.Usage
|
||||
truncated bool
|
||||
err error
|
||||
}
|
||||
|
||||
type scored struct {
|
||||
convention, id string
|
||||
violates bool
|
||||
constructed bool
|
||||
p float64
|
||||
err bool
|
||||
}
|
||||
|
||||
func runGates(dump string, pop []member, convs map[string]string, p decision.Provider, workers int) int {
|
||||
var cases []commentCase
|
||||
mustRead(filepath.Join(dump, "comments.json"), &cases)
|
||||
var batch12 []string
|
||||
mustRead(filepath.Join(dump, "batch12.json"), &batch12)
|
||||
for _, c := range cases {
|
||||
mustBody(convs, c.Convention)
|
||||
}
|
||||
for _, ref := range batch12 {
|
||||
mustBody(convs, ref)
|
||||
}
|
||||
|
||||
// Pass A: one question per call, as U0 asked.
|
||||
aJobs := make([]gateJob, len(pop))
|
||||
for i, m := range pop {
|
||||
st, trunc, err := state(dump, m)
|
||||
if err != nil {
|
||||
fmt.Fprintf(os.Stderr, "A %s %s: %v\n", m.Convention, m.Ref, err)
|
||||
return 1
|
||||
}
|
||||
mustNotCarryLabels(st, m.Ref)
|
||||
// The point of pass A: the U2a item state, links in and trail out.
|
||||
if !strings.Contains(string(st), `"recent_trail":[]`) || !strings.Contains(string(st), `"links":`) {
|
||||
fmt.Fprintf(os.Stderr, "A %s: not the U2a item state (trail must be empty, links present)\n", m.Ref)
|
||||
return 1
|
||||
}
|
||||
aJobs[i] = gateJob{st, trunc, map[string]decision.Question{"v": question(mustBody(convs, m.Convention))}}
|
||||
}
|
||||
// Pass B: each comment alone.
|
||||
bJobs := make([]gateJob, len(cases))
|
||||
for i, c := range cases {
|
||||
st, err := decision.BuildCommentState(c.Title, c.Collection, c.Comment)
|
||||
if err != nil {
|
||||
fmt.Fprintf(os.Stderr, "B %s: %v\n", c.ID, err)
|
||||
return 1
|
||||
}
|
||||
bJobs[i] = gateJob{st.Bytes, st.Truncated, map[string]decision.Question{"v": question(convs[c.Convention])}}
|
||||
}
|
||||
// Pass C: one call per item, every batch12 question plus its labelled ones.
|
||||
byItem := map[string][]int{}
|
||||
var itemOrder []string
|
||||
for i, m := range pop {
|
||||
if _, ok := byItem[m.Ref]; !ok {
|
||||
itemOrder = append(itemOrder, m.Ref)
|
||||
}
|
||||
byItem[m.Ref] = append(byItem[m.Ref], i)
|
||||
}
|
||||
cJobs := make([]gateJob, len(itemOrder))
|
||||
maxQ := 0
|
||||
for j, ref := range itemOrder {
|
||||
qs := map[string]decision.Question{}
|
||||
for _, c := range batch12 {
|
||||
qs[c] = question(convs[c])
|
||||
}
|
||||
for _, i := range byItem[ref] {
|
||||
qs[pop[i].Convention] = question(convs[pop[i].Convention])
|
||||
}
|
||||
if len(qs) > maxQ {
|
||||
maxQ = len(qs)
|
||||
}
|
||||
first := aJobs[byItem[ref][0]]
|
||||
cJobs[j] = gateJob{first.state, first.truncated, qs}
|
||||
}
|
||||
|
||||
if p == nil {
|
||||
fmt.Printf("gates dry run: A %d cases, B %d comment cases, C %d items (up to %d questions per call). No calls made.\n",
|
||||
len(aJobs), len(bJobs), len(cJobs), maxQ)
|
||||
fmt.Printf("A first state %d bytes; B first state %s\n", len(aJobs[0].state), bJobs[0].state)
|
||||
return 0
|
||||
}
|
||||
|
||||
start := time.Now()
|
||||
aRes := ask(p, aJobs, workers)
|
||||
bRes := ask(p, bJobs, workers)
|
||||
cRes := ask(p, cJobs, workers)
|
||||
|
||||
fmt.Printf("TASK-3119 U2 gates: model %s, threshold %.1f, %s wall\n", decision.DefaultModel, threshold, time.Since(start).Round(time.Second))
|
||||
|
||||
// Gate 1.
|
||||
aScored := make([]scored, len(pop))
|
||||
for i, m := range pop {
|
||||
aScored[i] = scored{m.Convention, m.Ref, m.Violates, m.Constructed, aRes[i].answers["v"].Noul, aRes[i].err != nil}
|
||||
}
|
||||
fmt.Printf("\nGATE 1: U0's population on the item state WITHOUT the trail (links in)\n")
|
||||
fmt.Printf("%-11s %4s %4s %4s %4s %-9s %-9s %-18s %s\n", "convention", "tp", "fp", "fn", "tn", "precision", "recall", "U0 prec/recall", "verdict")
|
||||
gate1 := true
|
||||
for _, c := range conventionsOf(pop) {
|
||||
tp, fp, fn, tn := confusion(aScored, c)
|
||||
base, fired := u0Baseline[c]
|
||||
verdict := "not gated (did not fire in U0)"
|
||||
baseStr := "-"
|
||||
if fired {
|
||||
baseStr = fmt.Sprintf("%.0f%% / %.0f%%", 100*base.precision, 100*base.recall)
|
||||
switch {
|
||||
case tp+fp == 0:
|
||||
verdict = "WARN: no longer fires (precision undefined; recall 0)"
|
||||
case float64(tp)/float64(tp+fp) < base.precision:
|
||||
verdict = "FAIL: precision dropped"
|
||||
gate1 = false
|
||||
default:
|
||||
verdict = "PASS"
|
||||
}
|
||||
}
|
||||
fmt.Printf("%-11s %4d %4d %4d %4d %-9s %-9s %-18s %s\n", c, tp, fp, fn, tn, ratio(tp, tp+fp), ratio(tp, tp+fn), baseStr, verdict)
|
||||
}
|
||||
fmt.Printf("gate 1: %s\n", passFail(gate1))
|
||||
|
||||
// Gate 2.
|
||||
bScored := make([]scored, len(cases))
|
||||
for i, c := range cases {
|
||||
bScored[i] = scored{c.Convention, c.ID, c.Violates, false, bRes[i].answers["v"].Noul, bRes[i].err != nil}
|
||||
}
|
||||
tp, fp, fn, tn := confusion(bScored, "")
|
||||
gate2 := tp+fp > 0 && float64(tp)/float64(tp+fp) >= commentPrecisionBar
|
||||
fmt.Printf("\nGATE 2: comment-borne breaks, each comment asked alone\n")
|
||||
fmt.Printf("all: tp %d fp %d fn %d tn %d precision %s recall %s (bar: precision >= %.0f%%, fires at least once)\n",
|
||||
tp, fp, fn, tn, ratio(tp, tp+fp), ratio(tp, tp+fn), 100*commentPrecisionBar)
|
||||
for _, c := range conventionsOfCases(cases) {
|
||||
ctp, cfp, cfn, ctn := confusion(bScored, c)
|
||||
fmt.Printf(" %-11s tp %d fp %d fn %d tn %d precision %s recall %s\n", c, ctp, cfp, cfn, ctn, ratio(ctp, ctp+cfp), ratio(ctp, ctp+cfn))
|
||||
}
|
||||
fmt.Printf("gate 2: %s\n", passFail(gate2))
|
||||
|
||||
// Pass C.
|
||||
cScored := make([]scored, len(pop))
|
||||
for j, ref := range itemOrder {
|
||||
for _, i := range byItem[ref] {
|
||||
m := pop[i]
|
||||
cScored[i] = scored{m.Convention, m.Ref, m.Violates, m.Constructed, cRes[j].answers[m.Convention].Noul, cRes[j].err != nil}
|
||||
}
|
||||
}
|
||||
fmt.Printf("\nMEASURE C: up to %d questions in ONE call per item (MaxPerCall 12), against pass A's one question per call\n", maxQ)
|
||||
fmt.Printf("%-11s %-22s %-22s\n", "convention", "batched prec/recall", "single prec/recall")
|
||||
batchHolds := true
|
||||
for _, c := range conventionsOf(pop) {
|
||||
btp, bfp, bfn, _ := confusion(cScored, c)
|
||||
stp, sfp, sfn, _ := confusion(aScored, c)
|
||||
fmt.Printf("%-11s %-22s %-22s\n", c, ratio(btp, btp+bfp)+" / "+ratio(btp, btp+bfn), ratio(stp, stp+sfp)+" / "+ratio(stp, stp+sfn))
|
||||
if btp+bfp > 0 && stp+sfp > 0 && float64(btp)/float64(btp+bfp) < float64(stp)/float64(stp+sfp) {
|
||||
batchHolds = false
|
||||
}
|
||||
}
|
||||
moved := 0
|
||||
for i := range pop {
|
||||
if d := cScored[i].p - aScored[i].p; d > 0.2 || d < -0.2 {
|
||||
moved++
|
||||
}
|
||||
}
|
||||
fmt.Printf("batched precision holds on every convention: %v; cases whose p moved by more than 0.2: %d of %d\n", batchHolds, moved, len(pop))
|
||||
|
||||
fmt.Printf("\nSPEND\n")
|
||||
for _, x := range []struct {
|
||||
name string
|
||||
res []gateResult
|
||||
}{{"A", aRes}, {"B", bRes}, {"C", cRes}} {
|
||||
calls, errs, in, out, trunc := spend(x.res)
|
||||
avg := 0
|
||||
if calls > 0 {
|
||||
avg = in / calls
|
||||
}
|
||||
fmt.Printf("pass %s: %d calls (%d errors), %d input / %d output tokens, %d input per call, %d truncated states\n", x.name, calls, errs, in, out, avg, trunc)
|
||||
}
|
||||
fmt.Println("money: read the delta on the typesafe.ai dashboard for this window; the API reports tokens, not price.")
|
||||
|
||||
fmt.Println("\nPER CASE (pass, convention, case, label, p; C repeats A's cases batched)")
|
||||
for _, x := range []struct {
|
||||
name string
|
||||
rows []scored
|
||||
}{{"A", aScored}, {"B", bScored}, {"C", cScored}} {
|
||||
for _, r := range x.rows {
|
||||
lab := "complies"
|
||||
if r.violates {
|
||||
lab = "VIOLATES"
|
||||
}
|
||||
if r.constructed {
|
||||
lab += "[C]"
|
||||
}
|
||||
errMark := ""
|
||||
if r.err {
|
||||
errMark = " ERR"
|
||||
}
|
||||
fmt.Printf(" %s %-11s %-12s %-12s %.3f%s\n", x.name, r.convention, r.id, lab, r.p, errMark)
|
||||
}
|
||||
}
|
||||
for _, x := range [][]gateResult{aRes, bRes, cRes} {
|
||||
for _, r := range x {
|
||||
if r.err != nil {
|
||||
fmt.Fprintf(os.Stderr, "provider error: %v\n", r.err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if gate1 && gate2 {
|
||||
fmt.Println("\nRESULT: PASS (gates 1 and 2)")
|
||||
return 0
|
||||
}
|
||||
fmt.Println("\nRESULT: FAIL")
|
||||
return 1
|
||||
}
|
||||
|
||||
func ask(p decision.Provider, jobs []gateJob, workers int) []gateResult {
|
||||
out := make([]gateResult, len(jobs))
|
||||
sem := make(chan struct{}, workers)
|
||||
var wg sync.WaitGroup
|
||||
for i, j := range jobs {
|
||||
wg.Add(1)
|
||||
go func(i int, j gateJob) {
|
||||
defer wg.Done()
|
||||
sem <- struct{}{}
|
||||
defer func() { <-sem }()
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 90*time.Second)
|
||||
defer cancel()
|
||||
ans, usage, err := p.Ask(ctx, json.RawMessage(j.state), j.questions)
|
||||
out[i] = gateResult{ans, usage, j.truncated || usage.StateTruncated, err}
|
||||
}(i, j)
|
||||
}
|
||||
wg.Wait()
|
||||
return out
|
||||
}
|
||||
|
||||
// confusion counts at the threshold; convention "" means every row. A row
|
||||
// whose call failed is left out, and counted in the spend's errors.
|
||||
func confusion(rows []scored, convention string) (tp, fp, fn, tn int) {
|
||||
for _, r := range rows {
|
||||
if r.err || (convention != "" && r.convention != convention) {
|
||||
continue
|
||||
}
|
||||
hit := r.p >= threshold
|
||||
switch {
|
||||
case hit && r.violates:
|
||||
tp++
|
||||
case hit:
|
||||
fp++
|
||||
case r.violates:
|
||||
fn++
|
||||
default:
|
||||
tn++
|
||||
}
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
func spend(res []gateResult) (calls, errs, in, out, trunc int) {
|
||||
for _, r := range res {
|
||||
if r.err != nil {
|
||||
errs++
|
||||
continue
|
||||
}
|
||||
calls++
|
||||
in += r.usage.InputTokens
|
||||
out += r.usage.OutputTokens
|
||||
if r.truncated {
|
||||
trunc++
|
||||
}
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
func conventionsOf(pop []member) []string {
|
||||
seen := map[string]bool{}
|
||||
var out []string
|
||||
for _, m := range pop {
|
||||
if !seen[m.Convention] {
|
||||
seen[m.Convention] = true
|
||||
out = append(out, m.Convention)
|
||||
}
|
||||
}
|
||||
sort.Strings(out)
|
||||
return out
|
||||
}
|
||||
|
||||
func conventionsOfCases(cases []commentCase) []string {
|
||||
seen := map[string]bool{}
|
||||
var out []string
|
||||
for _, c := range cases {
|
||||
if !seen[c.Convention] {
|
||||
seen[c.Convention] = true
|
||||
out = append(out, c.Convention)
|
||||
}
|
||||
}
|
||||
sort.Strings(out)
|
||||
return out
|
||||
}
|
||||
|
||||
func mustBody(convs map[string]string, ref string) string {
|
||||
b, ok := convs[ref]
|
||||
if !ok || b == "" {
|
||||
fmt.Fprintf(os.Stderr, "no body for %s in conventions.json\n", ref)
|
||||
os.Exit(1)
|
||||
}
|
||||
return b
|
||||
}
|
||||
|
||||
func mustNotCarryLabels(st []byte, ref string) {
|
||||
if strings.Contains(string(st), "HAND LABELS") {
|
||||
fmt.Fprintf(os.Stderr, "%s: the state carries the hand labels\n", ref)
|
||||
os.Exit(1)
|
||||
}
|
||||
}
|
||||
|
||||
func passFail(ok bool) string {
|
||||
if ok {
|
||||
return "PASS"
|
||||
}
|
||||
return "FAIL"
|
||||
}
|
||||
@@ -87,7 +87,15 @@ func question(body string) decision.Question {
|
||||
// that matters.
|
||||
var withLinks bool
|
||||
|
||||
// noTrail builds the item state WITHOUT its comment trail, as the conventions
|
||||
// set's item subject does from TASK-3119 U2a (the comments are asked about
|
||||
// one at a time instead, see gates.go).
|
||||
var noTrail bool
|
||||
|
||||
func build(item *models.Item, comments []models.Comment, parentRef, parentTitle string) (decision.BuiltState, error) {
|
||||
if noTrail {
|
||||
comments = nil
|
||||
}
|
||||
if !withLinks {
|
||||
return decision.BuildItemState(item, comments)
|
||||
}
|
||||
@@ -142,6 +150,7 @@ func main() {
|
||||
workers := flag.Int("workers", 6, "concurrent provider calls")
|
||||
links := flag.Bool("links", false, "build states with the item's links, as the conventions set does (TASK-3119 U1a)")
|
||||
only := flag.String("only", "", "run only this convention (e.g. CONVE-1286)")
|
||||
gates := flag.Bool("gates", false, "run the TASK-3119 U2 gates (item state without trail, comment subjects, 12-question batches) and print PASS/FAIL")
|
||||
flag.Parse()
|
||||
withLinks = *links
|
||||
if *dump == "" {
|
||||
@@ -162,6 +171,15 @@ func main() {
|
||||
pop = kept
|
||||
}
|
||||
|
||||
if *gates {
|
||||
withLinks, noTrail = true, true
|
||||
var p decision.Provider
|
||||
if !*dry {
|
||||
p = mustProvider()
|
||||
}
|
||||
os.Exit(runGates(*dump, pop, convs, p, *workers))
|
||||
}
|
||||
|
||||
if *dry {
|
||||
for _, m := range pop {
|
||||
if _, ok := convs[m.Convention]; !ok {
|
||||
@@ -185,16 +203,7 @@ func main() {
|
||||
return
|
||||
}
|
||||
|
||||
key := os.Getenv("TYPESAFE_API_KEY")
|
||||
if key == "" {
|
||||
fmt.Fprintln(os.Stderr, "TYPESAFE_API_KEY is not set")
|
||||
os.Exit(2)
|
||||
}
|
||||
p, err := decision.New(decision.Config{Provider: decision.ProviderTypesafe, APIKey: key, Model: decision.DefaultModel})
|
||||
if err != nil {
|
||||
fmt.Fprintln(os.Stderr, err)
|
||||
os.Exit(1)
|
||||
}
|
||||
p := mustProvider()
|
||||
|
||||
rows := make([]row, len(pop))
|
||||
sem := make(chan struct{}, *workers)
|
||||
@@ -290,6 +299,21 @@ func report(rows []row, took time.Duration) {
|
||||
}
|
||||
}
|
||||
|
||||
// mustProvider reads the key from TYPESAFE_API_KEY and never prints it.
|
||||
func mustProvider() decision.Provider {
|
||||
key := os.Getenv("TYPESAFE_API_KEY")
|
||||
if key == "" {
|
||||
fmt.Fprintln(os.Stderr, "TYPESAFE_API_KEY is not set")
|
||||
os.Exit(2)
|
||||
}
|
||||
p, err := decision.New(decision.Config{Provider: decision.ProviderTypesafe, APIKey: key, Model: decision.DefaultModel})
|
||||
if err != nil {
|
||||
fmt.Fprintln(os.Stderr, err)
|
||||
os.Exit(1)
|
||||
}
|
||||
return p
|
||||
}
|
||||
|
||||
func ratio(a, b int) string {
|
||||
if b == 0 {
|
||||
return "n/a"
|
||||
|
||||
Reference in New Issue
Block a user