mirror of
https://github.com/PerpetualSoftware/pad.git
synced 2026-09-10 15:05:40 +00:00
cfda4463e8
* feat(cmdhelp): tests + golden contract + drift validator (TASK-938)
The verification layer that turns cmdhelp v0.1 from "implementation"
into "stable contract." Three categories of tests, all running in
`go test ./...`:
1. Schema validation (cmdhelp.schema.json as CI gate)
- internal/cmdhelp/schema_test.go — synthetic tree's emitted JSON
validates after static walk, after dynamic resolution, and after
a no-workspace fallback.
- cmd/pad/cmdhelp_real_test.go — the REAL pad cobra tree's emitted
JSON validates against the published schema. Future regressions
caught: types outside the closed vocabulary, non-numeric exit_code
keys, flag names violating propertyNames, malformed cmdhelp_version.
2. Drift-prevention contract (spec §6 / §11 Q5)
- internal/cmdhelp/example_validation.go — ValidateExamples walks
every example's `cmd` string, tokenizes with shellSplit, resolves
non-flag tokens against the live cobra tree, and asserts every
--flag exists on the resolved command (or any ancestor for
persistent / inherited flags). Negate-flag form (`--no-cache`)
is recognized via the negation rule from spec §5.3.
- shellSplit handles double/single quotes, backslash escape, and
stops at unquoted pipeline boundaries (|, ;, &, >, <) so the
validator only checks the first command in a pipeline.
- ValidateBoolArity asserts no bool flag appears in valued form
(--flag=value) anywhere in its examples (spec §5.3).
- cmd/pad/cmdhelp_real_test.go runs both validators against the
real pad tree as CI gates.
- Negative tests in internal/cmdhelp/example_validation_test.go
prove the validator catches: typo'd flag (--priorty), unknown
command path, valued-form bool flag.
3. Capabilities form equivalence (spec §8)
- cmd/pad/cmdhelp_real_test.go — both forms (help --capabilities
and --cmdhelp-capabilities fallback) produce byte-identical
output. Side-effect-free guarantee verified by passing garbage
args alongside the fallback flag.
Refactors enabling the tests:
- cmd/pad/main.go: extract newRootCmd() so tests can build the real
cobra tree without running it. main() body shrinks to two lines.
- cmd/pad/main.go: extract handleCmdhelpCapabilitiesFallback() so the
fallback's side-effect-free contract is directly assertable instead
of requiring a subprocess.
Parser improvements driven by real-pad-tree drift findings:
- parseExamplesFromLong: strip same-line `# comment` annotations so
`pad foo --bar # one item's attachments` doesn't pollute Examples.
stripCommentIndex is quote-aware (# inside "..." or '...' is literal).
- main.go (github cmd): the Long had annotations on example lines
separated only by spaces (no `#`), which was malformed input. Fixed
to use `#` separators — caught by the drift validator on first run.
New deps:
- github.com/santhosh-tekuri/jsonschema/v6 v6.0.2 — Go JSON Schema
validator supporting draft 2020-12 (matches the cmdhelp schema's $schema).
New helpers in internal/cmdhelp/:
- FindAndCompileSchema(startDir) walks up to locate
schema/cmdhelp.schema.json and returns a compiled schema. Reusable
by any consumer that wants to validate cmdhelp documents.
End-to-end on real binary:
- pad help --format json → 100 commands, schema-valid.
- All examples in pad's emitted output resolve against the live tree
(zero drift findings).
- pad help --capabilities byte-identical to pad --cmdhelp-capabilities.
- Adding a typo'd flag in any cobra Long block in cmd/pad MUST break
TestRealPadTree_ExampleDriftValidator. Verified by the negative
TestValidateExamples_DetectsTypoFlag.
make check clean. All 53 cmdhelp + cmd/pad tests pass.
Parent: PLAN-930.
* fix(cmdhelp): pass full token stream to cobra.Find per Codex review (round 1)
Codex round 1 caught: ValidateExamples stopped collecting the command
path at the first flag, so an example like
pad --workspace foo item create task --priority high
resolved to root, not `item create`. That meant `--priority` was
checked against root's flag set (where it doesn't exist) — false
positive — AND the validator silently missed any command-path drift
after a leading root flag.
Cobra's own Find walks the full token stream and uses each command's
flag definitions to skip flag/value pairs while matching subcommand
names. Pass tokens[1:] directly to root.Find — let cobra handle the
interleaving correctly.
New test:
- TestValidateExamples_FlagBeforeSubcommandResolvesToCorrectTarget —
flag-before-subcommand resolves to the leaf and accepts leaf flags.
219 lines
7.1 KiB
Go
219 lines
7.1 KiB
Go
package cmdhelp
|
|
|
|
import (
|
|
"fmt"
|
|
"strings"
|
|
|
|
"github.com/spf13/cobra"
|
|
)
|
|
|
|
// ValidateExamples is the contractual mechanism by which cmdhelp v0.1
|
|
// prevents documentation drift (spec §6, §11 Q5). For every example in
|
|
// every command of `doc`:
|
|
//
|
|
// 1. Tokenize the `cmd` string with a shell-aware splitter.
|
|
// 2. Resolve the non-flag tokens against `root`'s live cobra command
|
|
// tree. Failure means the example references a command path that
|
|
// doesn't exist (typo, deleted command, drift).
|
|
// 3. Verify every `--flag` token resolves to a flag on the resolved
|
|
// command — local, persistent, or inherited from any parent.
|
|
//
|
|
// Returns a slice of error strings, one per finding, with the offending
|
|
// command path + example index for fast triage. Empty slice means clean.
|
|
//
|
|
// This is intentionally a function not a test so cmd/pad's own test
|
|
// suite (which has access to the real cobra tree) can call it.
|
|
// Callers must pass `root` so flag-tree walks can include persistent
|
|
// flags from the binary root.
|
|
func ValidateExamples(doc *Document, root *cobra.Command) []string {
|
|
if doc == nil || root == nil {
|
|
return nil
|
|
}
|
|
var findings []string
|
|
|
|
for path, cmd := range doc.Commands {
|
|
for i, ex := range cmd.Examples {
|
|
if ferrs := validateExample(path, i, ex, doc, root); len(ferrs) > 0 {
|
|
findings = append(findings, ferrs...)
|
|
}
|
|
}
|
|
}
|
|
return findings
|
|
}
|
|
|
|
func validateExample(path string, idx int, ex Example, doc *Document, root *cobra.Command) []string {
|
|
tokens, err := shellSplit(ex.Cmd)
|
|
if err != nil {
|
|
return []string{fmt.Sprintf("%s example[%d]: cannot tokenize %q: %v", path, idx, ex.Cmd, err)}
|
|
}
|
|
if len(tokens) == 0 {
|
|
return []string{fmt.Sprintf("%s example[%d]: empty cmd string", path, idx)}
|
|
}
|
|
|
|
// Token 0 should be the binary's name. Examples that begin with
|
|
// something else (e.g. `cat ~/foo.json | jq`) are documentation
|
|
// snippets showing related output, not pad invocations the
|
|
// validator can check. Skip them rather than fail — the drift
|
|
// contract is for pad-invocation drift specifically.
|
|
binary := root.Name()
|
|
if tokens[0] != binary {
|
|
return nil
|
|
}
|
|
|
|
// Pass the full post-binary token stream to cobra's Find. Cobra
|
|
// knows each command's flag set and skips flag/value pairs while
|
|
// matching subcommand names — so examples that interleave flags
|
|
// with subcommands (e.g. `pad --workspace foo item create task`)
|
|
// resolve to `item create`, not the root.
|
|
target, _, ferr := root.Find(tokens[1:])
|
|
if ferr != nil || target == nil {
|
|
return []string{fmt.Sprintf("%s example[%d]: command path doesn't resolve: %s", path, idx, ex.Cmd)}
|
|
}
|
|
|
|
// Validate every --flag/-f against target's flag tree.
|
|
// Note: flags can appear before the subcommand path on cobra
|
|
// (e.g. `pad --workspace foo item create ...`), so scan all tokens.
|
|
var findings []string
|
|
skipNext := false
|
|
for _, t := range tokens[1:] {
|
|
if skipNext {
|
|
skipNext = false
|
|
continue
|
|
}
|
|
if !strings.HasPrefix(t, "-") {
|
|
continue
|
|
}
|
|
// Strip leading dashes and any =value suffix.
|
|
name := strings.TrimLeft(t, "-")
|
|
if eq := strings.IndexByte(name, '='); eq >= 0 {
|
|
name = name[:eq]
|
|
}
|
|
if name == "" {
|
|
continue // bare "--" terminator
|
|
}
|
|
// Walk target up to root, accept the flag if any level has it.
|
|
// Boolean flags don't consume the next token; non-bool flags
|
|
// do — but we only need the name check for drift detection,
|
|
// so don't bother with the value-consumption walk except to
|
|
// note that the "next token" might be a value rather than
|
|
// another flag (no special handling needed here).
|
|
if !flagExists(target, name) {
|
|
// Try negate-flag form: --no-<rest>.
|
|
if strings.HasPrefix(name, "no-") {
|
|
stripped := strings.TrimPrefix(name, "no-")
|
|
if flagExists(target, stripped) {
|
|
continue
|
|
}
|
|
}
|
|
findings = append(findings, fmt.Sprintf("%s example[%d]: unknown flag --%s in %q", path, idx, name, ex.Cmd))
|
|
}
|
|
}
|
|
return findings
|
|
}
|
|
|
|
// flagExists reports whether `name` is a known flag on cmd or any of
|
|
// its ancestors (covers persistent / inherited flags). Both long and
|
|
// short forms are checked — pflag's Lookup treats single-character
|
|
// names as shorthand and ShorthandLookup as the lookup for them.
|
|
func flagExists(cmd *cobra.Command, name string) bool {
|
|
for c := cmd; c != nil; c = c.Parent() {
|
|
if c.Flags().Lookup(name) != nil {
|
|
return true
|
|
}
|
|
if c.PersistentFlags().Lookup(name) != nil {
|
|
return true
|
|
}
|
|
if len(name) == 1 {
|
|
if c.Flags().ShorthandLookup(name) != nil {
|
|
return true
|
|
}
|
|
if c.PersistentFlags().ShorthandLookup(name) != nil {
|
|
return true
|
|
}
|
|
}
|
|
}
|
|
return false
|
|
}
|
|
|
|
// shellSplit tokenizes a command line in a POSIX-ish way: whitespace
|
|
// separates tokens, double-quotes group, single-quotes group (no
|
|
// expansion inside), backslash escapes the next rune outside single
|
|
// quotes. Stops at the first unquoted pipeline boundary (`|`, `;`,
|
|
// `&`, `>`, `<`) so the validator only checks the first command in a
|
|
// pipeline. Not a full shell — no $vars, no globbing.
|
|
//
|
|
// Sufficient for cmdhelp `examples[].cmd` strings, which are expected
|
|
// to be runnable invocations of one CLI command (possibly piped to
|
|
// another tool — that other tool is the user's shell, not pad's).
|
|
func shellSplit(s string) ([]string, error) {
|
|
var tokens []string
|
|
var cur strings.Builder
|
|
var inDQuote, inSQuote, escape bool
|
|
flush := func() {
|
|
if cur.Len() > 0 {
|
|
tokens = append(tokens, cur.String())
|
|
cur.Reset()
|
|
}
|
|
}
|
|
for _, r := range s {
|
|
switch {
|
|
case escape:
|
|
cur.WriteRune(r)
|
|
escape = false
|
|
case r == '\\' && !inSQuote:
|
|
escape = true
|
|
case r == '"' && !inSQuote:
|
|
inDQuote = !inDQuote
|
|
case r == '\'' && !inDQuote:
|
|
inSQuote = !inSQuote
|
|
case (r == ' ' || r == '\t') && !inDQuote && !inSQuote:
|
|
flush()
|
|
case (r == '|' || r == ';' || r == '&' || r == '>' || r == '<') && !inDQuote && !inSQuote:
|
|
// Pipeline boundary — return what we have so far. The rest
|
|
// of the string belongs to a different command (or the
|
|
// shell), which is out of scope for cmdhelp validation.
|
|
flush()
|
|
return tokens, nil
|
|
default:
|
|
cur.WriteRune(r)
|
|
}
|
|
}
|
|
if inDQuote || inSQuote {
|
|
return nil, fmt.Errorf("unterminated quote")
|
|
}
|
|
if escape {
|
|
return nil, fmt.Errorf("trailing backslash")
|
|
}
|
|
flush()
|
|
return tokens, nil
|
|
}
|
|
|
|
// ValidateBoolArity asserts that every bool-typed flag in doc obeys
|
|
// spec §5.3: bool flags MUST be presence switches. They MUST NOT
|
|
// appear in `--flag=value` form anywhere in their command's examples
|
|
// (those should be declared as enum-typed instead).
|
|
//
|
|
// Returns one finding per violation. Empty slice means clean.
|
|
func ValidateBoolArity(doc *Document) []string {
|
|
if doc == nil {
|
|
return nil
|
|
}
|
|
var findings []string
|
|
for path, cmd := range doc.Commands {
|
|
for fname, f := range cmd.Flags {
|
|
if f.Type != "bool" {
|
|
continue
|
|
}
|
|
needle := "--" + fname + "="
|
|
for i, ex := range cmd.Examples {
|
|
if strings.Contains(ex.Cmd, needle) {
|
|
findings = append(findings,
|
|
fmt.Sprintf("%s example[%d]: bool flag --%s appears in valued form (--%s=...) — spec §5.3 requires bool flags to be presence-only; declare as enum: [\"true\",\"false\"] for valued booleans",
|
|
path, i, fname, fname))
|
|
}
|
|
}
|
|
}
|
|
}
|
|
return findings
|
|
}
|