Files
pad/internal/cmdhelp/json.go
T
xarmian cfda4463e8 feat(cmdhelp): tests + golden contract + drift validator (TASK-938) (#332)
* feat(cmdhelp): tests + golden contract + drift validator (TASK-938)

The verification layer that turns cmdhelp v0.1 from "implementation"
into "stable contract." Three categories of tests, all running in
`go test ./...`:

1. Schema validation (cmdhelp.schema.json as CI gate)
   - internal/cmdhelp/schema_test.go — synthetic tree's emitted JSON
     validates after static walk, after dynamic resolution, and after
     a no-workspace fallback.
   - cmd/pad/cmdhelp_real_test.go — the REAL pad cobra tree's emitted
     JSON validates against the published schema. Future regressions
     caught: types outside the closed vocabulary, non-numeric exit_code
     keys, flag names violating propertyNames, malformed cmdhelp_version.

2. Drift-prevention contract (spec §6 / §11 Q5)
   - internal/cmdhelp/example_validation.go — ValidateExamples walks
     every example's `cmd` string, tokenizes with shellSplit, resolves
     non-flag tokens against the live cobra tree, and asserts every
     --flag exists on the resolved command (or any ancestor for
     persistent / inherited flags). Negate-flag form (`--no-cache`)
     is recognized via the negation rule from spec §5.3.
   - shellSplit handles double/single quotes, backslash escape, and
     stops at unquoted pipeline boundaries (|, ;, &, >, <) so the
     validator only checks the first command in a pipeline.
   - ValidateBoolArity asserts no bool flag appears in valued form
     (--flag=value) anywhere in its examples (spec §5.3).
   - cmd/pad/cmdhelp_real_test.go runs both validators against the
     real pad tree as CI gates.
   - Negative tests in internal/cmdhelp/example_validation_test.go
     prove the validator catches: typo'd flag (--priorty), unknown
     command path, valued-form bool flag.

3. Capabilities form equivalence (spec §8)
   - cmd/pad/cmdhelp_real_test.go — both forms (help --capabilities
     and --cmdhelp-capabilities fallback) produce byte-identical
     output. Side-effect-free guarantee verified by passing garbage
     args alongside the fallback flag.

Refactors enabling the tests:
- cmd/pad/main.go: extract newRootCmd() so tests can build the real
  cobra tree without running it. main() body shrinks to two lines.
- cmd/pad/main.go: extract handleCmdhelpCapabilitiesFallback() so the
  fallback's side-effect-free contract is directly assertable instead
  of requiring a subprocess.

Parser improvements driven by real-pad-tree drift findings:
- parseExamplesFromLong: strip same-line `# comment` annotations so
  `pad foo --bar # one item's attachments` doesn't pollute Examples.
  stripCommentIndex is quote-aware (# inside "..." or '...' is literal).
- main.go (github cmd): the Long had annotations on example lines
  separated only by spaces (no `#`), which was malformed input. Fixed
  to use `#` separators — caught by the drift validator on first run.

New deps:
- github.com/santhosh-tekuri/jsonschema/v6 v6.0.2 — Go JSON Schema
  validator supporting draft 2020-12 (matches the cmdhelp schema's $schema).

New helpers in internal/cmdhelp/:
- FindAndCompileSchema(startDir) walks up to locate
  schema/cmdhelp.schema.json and returns a compiled schema. Reusable
  by any consumer that wants to validate cmdhelp documents.

End-to-end on real binary:
- pad help --format json → 100 commands, schema-valid.
- All examples in pad's emitted output resolve against the live tree
  (zero drift findings).
- pad help --capabilities byte-identical to pad --cmdhelp-capabilities.
- Adding a typo'd flag in any cobra Long block in cmd/pad MUST break
  TestRealPadTree_ExampleDriftValidator. Verified by the negative
  TestValidateExamples_DetectsTypoFlag.

make check clean. All 53 cmdhelp + cmd/pad tests pass.

Parent: PLAN-930.

* fix(cmdhelp): pass full token stream to cobra.Find per Codex review (round 1)

Codex round 1 caught: ValidateExamples stopped collecting the command
path at the first flag, so an example like
  pad --workspace foo item create task --priority high
resolved to root, not `item create`. That meant `--priority` was
checked against root's flag set (where it doesn't exist) — false
positive — AND the validator silently missed any command-path drift
after a leading root flag.

Cobra's own Find walks the full token stream and uses each command's
flag definitions to skip flag/value pairs while matching subcommand
names. Pass tokens[1:] directly to root.Find — let cobra handle the
interleaving correctly.

New test:
- TestValidateExamples_FlagBeforeSubcommandResolvesToCorrectTarget —
  flag-before-subcommand resolves to the leaf and accepts leaf flags.
2026-05-01 07:17:58 -04:00

543 lines
16 KiB
Go

package cmdhelp
import (
"encoding/json"
"fmt"
"io"
"regexp"
"strings"
"time"
"github.com/spf13/cobra"
"github.com/spf13/pflag"
)
// CapabilityLine returns the single-line capability bit a conforming
// CLI emits in response to `<cmd> help --capabilities` (or the
// `--cmdhelp-capabilities` fallback form). Per spec §8:
//
// - Format: `cmdhelp/<MAJOR>.<MINOR>: <comma-separated formats>`
// - Single-line, parseable, stable across releases.
// - The format list MUST include every supported format, including
// the mandatory `text`. Order is not significant.
//
// Pass the formats this binary actually supports. The helper does not
// imply any default set so different binaries can advertise different
// surfaces (e.g. a future cmdhelp v0.2 might add formats).
func CapabilityLine(formats []string) string {
return fmt.Sprintf("cmdhelp/%s: %s", Version, strings.Join(formats, ", "))
}
// Options configure a Build/EmitJSON call. All fields are optional except
// Binary, which is used as the document's `binary` key.
type Options struct {
// Binary is the CLI binary name as invoked on the command line
// (e.g. "pad"). Defaults to root.Name() when empty.
Binary string
// Version is the implementation's own software version, independent
// of cmdhelp's wire-format version. Free-form.
Version string
// Homepage is the project's canonical URL. Optional.
Homepage string
// MaxDepth caps the walk at N levels below `target`. Pass a negative
// value (-1 is conventional) for unlimited depth. 0 emits only the
// target itself; 1 emits the target plus its direct subcommands; N
// emits N levels of subcommands.
//
// Note that Go's zero value (0) means "target only". Callers that
// want the unlimited default MUST pass -1 (or any negative value)
// explicitly — see cmd/pad/help_cmdhelp.go for the canonical wiring.
MaxDepth int
// Now overrides time.Now for the markdown emitter's YAML frontmatter
// `generated:` field. Useful for snapshot tests that need a stable
// timestamp. Leave nil to use the real wall clock (UTC).
Now func() time.Time
// Resolver, when non-nil, splices live workspace facts into the
// emitted Document after the static walk completes (spec §7). Pass
// nil to emit a purely static document — useful when no workspace
// is detected, when running outside a pad install, or in tests.
Resolver *Resolver
}
// EmitJSON walks the command tree below `target`, builds a cmdhelp v0.1
// Document, and writes it to w as indented JSON terminated by a newline.
//
// `root` MUST be the root cobra command (used to derive global flags and
// command-path keys relative to the binary). `target` is where the user
// requested help; pass root for both when the user runs `<cmd> help`.
func EmitJSON(target, root *cobra.Command, opts Options, w io.Writer) error {
doc := Build(target, root, opts)
enc := json.NewEncoder(w)
enc.SetIndent("", " ")
return enc.Encode(doc)
}
// Build returns a cmdhelp Document without serializing it. Callers that
// need to inspect or further mutate the document (tests, dynamic enum
// resolvers in TASK-936) should use this and serialize themselves.
func Build(target, root *cobra.Command, opts Options) *Document {
binary := opts.Binary
if binary == "" {
binary = root.Name()
}
doc := &Document{
CmdhelpVersion: Version,
Binary: binary,
Version: opts.Version,
Summary: firstLine(root.Short),
Homepage: opts.Homepage,
Commands: map[string]Command{},
}
if globals := collectFlags(root.PersistentFlags()); len(globals) > 0 {
doc.GlobalFlags = globals
}
walk(target, root, doc, 0, opts.MaxDepth)
// Splice dynamic workspace facts after the static walk so callers
// can inspect Build's output as either pre- or post-resolution.
opts.Resolver.Apply(doc)
return doc
}
func walk(cur, root *cobra.Command, doc *Document, depth, maxDepth int) {
if cur == nil || cur.Hidden {
return
}
// Skip the help command itself — it documents the cmdhelp surface,
// not a regular command, and cobra installs it on every root.
if cur.Name() == "help" && cur.Parent() != nil && cur.Parent() == root {
return
}
// The root binary itself is described by top-level fields (binary,
// summary, version, etc.) — don't also emit it as a command entry.
if cur != root {
path := commandPath(cur, root)
if path != "" {
doc.Commands[path] = buildCommand(cur)
}
}
// `depth` is how many levels below the initial target we currently are.
// `maxDepth` is the number of additional levels of descendants to emit;
// children of target sit at depth=1, grandchildren at depth=2, etc.
// Per spec §4, --depth=0 means "subcommand list" (immediate children of
// target), so we recurse while depth+1 <= maxDepth+1 — i.e. while
// depth <= maxDepth. maxDepth < 0 disables the cap.
if maxDepth >= 0 && depth > maxDepth {
return
}
for _, sub := range cur.Commands() {
walk(sub, root, doc, depth+1, maxDepth)
}
}
func commandPath(cmd, root *cobra.Command) string {
full := cmd.CommandPath()
rootPath := root.CommandPath()
trimmed := strings.TrimPrefix(full, rootPath)
return strings.TrimSpace(trimmed)
}
func buildCommand(cmd *cobra.Command) Command {
out := Command{
Summary: firstLine(cmd.Short),
}
// Long-form description: only emit if it adds something beyond Short.
if longTrimmed := strings.TrimSpace(cmd.Long); longTrimmed != "" && longTrimmed != cmd.Short && firstLine(longTrimmed) != cmd.Short {
out.Description = longTrimmed
}
out.Args = parseArgs(cmd)
if flags := collectFlags(cmd.LocalFlags()); len(flags) > 0 {
out.Flags = flags
}
// Examples come from cobra's dedicated Example field when set.
// Many CLIs (pad included, historically) embed an "Examples:"
// section in Long instead. Fall back to parsing that block so
// existing commands populate examples in cmdhelp output without a
// Long → Example migration. Long stays the canonical Long; the
// extracted examples become the Examples field, leaving room for
// commands to migrate to cobra's structured field on their own
// schedule.
out.Examples = parseExamples(cmd.Example)
if len(out.Examples) == 0 {
out.Examples = parseExamplesFromLong(cmd.Long)
}
return out
}
// examplesHeaderRE matches a stand-alone "Examples:" or "Example:"
// section header in a cobra Long string. Anchored to a line on its own
// (not anything that just contains the word "Examples"), so prose like
// "See the Examples section below" doesn't trigger the fallback.
var examplesHeaderRE = regexp.MustCompile(`^\s*Examples?:\s*$`)
// parseExamplesFromLong extracts the indented invocation lines that
// follow an "Examples:" header in a cobra Long string. Returns nil
// when no such section exists or no example lines are found.
//
// The block ends at:
// - a blank line after at least one example was collected, OR
// - an unindented line after at least one example was collected, OR
// - the end of Long.
//
// Comment lines (starting with `#`) inside the block are dropped.
// Used as a fallback in buildCommand when cmd.Example is empty.
func parseExamplesFromLong(long string) []Example {
if long == "" {
return nil
}
lines := strings.Split(long, "\n")
startIdx := -1
for i, line := range lines {
if examplesHeaderRE.MatchString(line) {
startIdx = i + 1
break
}
}
if startIdx < 0 {
return nil
}
var examples []Example
for i := startIdx; i < len(lines); i++ {
line := lines[i]
trimmed := strings.TrimSpace(line)
// Blank line: end of block if we already collected something.
// Skip otherwise (the section may start with a blank line).
if trimmed == "" {
if len(examples) > 0 {
break
}
continue
}
// Comment lines are dropped from the example set.
if strings.HasPrefix(trimmed, "#") {
continue
}
// Unindented line after we started collecting → end of block.
// Detected as: the trimmed text equals the original (no leading
// whitespace was stripped).
if line == trimmed && len(examples) > 0 {
break
}
// Strip a trailing same-line comment (" # blah") so it doesn't
// pollute the recorded example. Pad's existing Long blocks use
// this idiom occasionally, e.g.
// pad attachment list --item TASK-5 # one item's attachments
if hashIdx := stripCommentIndex(trimmed); hashIdx >= 0 {
trimmed = strings.TrimRight(trimmed[:hashIdx], " \t")
}
if trimmed == "" {
continue
}
examples = append(examples, Example{Cmd: trimmed})
}
if len(examples) == 0 {
return nil
}
return examples
}
// argRE matches positional arg placeholders in a cobra Use string:
//
// <name> → required
// [name] → optional
// <name>... → required + repeatable (variadic)
// [name]... → optional + repeatable (variadic)
// [a|b|c] → optional alternation (cobra ValidArgs convention) — enum-typed
// <a|b|c> → required alternation — enum-typed
//
// Cobra-conventional placeholders ([flags], [options], [command]) and
// embedded flag-like fragments ([--status X]) are filtered downstream
// in parseArgs.
var argRE = regexp.MustCompile(`<([^<>]+)>(\.\.\.)?|\[([^\[\]]+)\](\.\.\.)?`)
func parseArgs(cmd *cobra.Command) []Arg {
matches := argRE.FindAllStringSubmatch(cmd.Use, -1)
args := make([]Arg, 0, len(matches))
for _, m := range matches {
var inner, ellipsis string
var required bool
switch {
case m[1] != "":
inner = m[1]
ellipsis = m[2]
required = true
case m[3] != "":
inner = m[3]
ellipsis = m[4]
required = false
default:
continue
}
inner = strings.TrimSpace(inner)
if inner == "" {
continue
}
// Filter cobra-conventional placeholders that are not real args.
lower := strings.ToLower(inner)
if !required && (lower == "flags" || lower == "options" || lower == "command") {
continue
}
// Filter embedded flag-like fragments such as `[--status X]`.
if strings.HasPrefix(inner, "-") {
continue
}
// Alternation: <a|b|c> or [a|b|c] → enum-typed positional.
if strings.Contains(inner, "|") {
parts := strings.Split(inner, "|")
values := make([]interface{}, 0, len(parts))
for _, p := range parts {
p = strings.TrimSpace(p)
if p != "" {
values = append(values, p)
}
}
if len(values) > 0 {
args = append(args, Arg{
Name: "value", // synthesized; cobra Use doesn't carry a name here
Type: "enum",
Required: required,
Enum: values,
Repeatable: ellipsis != "",
})
continue
}
}
// Plain <name> / [name]. Reject anything that looks like prose
// (spaces, punctuation that wouldn't be in an arg identifier).
if !validArgName(inner) {
continue
}
args = append(args, Arg{
Name: inner,
Type: "string",
Required: required,
Repeatable: ellipsis != "",
})
}
// If cobra's ValidArgs is set and we have at least one positional,
// attach those values as the first arg's enum. ValidArgs is the
// authoritative machine-readable form (Use is for humans), so this
// covers the case where Use says `[shell]` but the allowed values
// only live on the cobra struct.
if len(cmd.ValidArgs) > 0 && len(args) > 0 && args[0].Type == "string" {
values := make([]interface{}, len(cmd.ValidArgs))
for i, v := range cmd.ValidArgs {
// cobra ValidArgs entries can carry shell-completion descriptions
// after a tab; strip them so the enum carries just the values.
if tab := strings.IndexByte(v, '\t'); tab >= 0 {
v = v[:tab]
}
values[i] = v
}
args[0].Type = "enum"
args[0].Enum = values
}
if len(args) == 0 {
return nil
}
return args
}
// validArgName returns true if s looks like an ordinary identifier-style
// arg name. Used to reject embedded flag fragments and prose that the
// regex would otherwise capture from idiosyncratic Use strings.
func validArgName(s string) bool {
if s == "" {
return false
}
for _, r := range s {
switch {
case r == '-' || r == '_' || r == '.' || r == '/':
case r >= 'a' && r <= 'z':
case r >= 'A' && r <= 'Z':
case r >= '0' && r <= '9':
default:
return false
}
}
return true
}
func collectFlags(fs *pflag.FlagSet) map[string]Flag {
if fs == nil {
return nil
}
out := map[string]Flag{}
fs.VisitAll(func(f *pflag.Flag) {
// Skip cobra's auto-installed --help flag and any explicitly
// hidden flag. These are noise in machine-readable output.
if f.Hidden || f.Name == "help" {
return
}
out[f.Name] = buildFlag(f)
})
if len(out) == 0 {
return nil
}
return out
}
func buildFlag(f *pflag.Flag) Flag {
typ, repeatable := mapPflagType(f.Value.Type())
flag := Flag{
Type: typ,
Description: f.Usage,
Repeatable: repeatable,
}
if f.DefValue != "" && !isZeroDefault(f.DefValue, typ, repeatable) {
flag.Default = f.DefValue
}
return flag
}
// isZeroDefault returns true when DefValue is the conventional zero for
// the flag's type. Suppressing zero defaults keeps the emitted document
// small and avoids encoding "" / "0" / "false" / "[]" everywhere.
func isZeroDefault(def, typ string, repeatable bool) bool {
if repeatable && (def == "[]" || def == "" || def == "[" || def == "]") {
return true
}
switch typ {
case "string":
return def == ""
case "int", "float":
return def == "0" || def == "0.0"
case "bool":
return def == "false"
case "duration":
return def == "0s" || def == "0"
}
return false
}
// mapPflagType translates pflag Value.Type() strings to the cmdhelp v0.1
// type vocabulary (spec §5.1) and reports whether the flag is repeatable.
//
// pflag exposes a wider type space than cmdhelp; we map related types
// down to the closed cmdhelp set. Slice/array variants become the scalar
// type with repeatable=true. Unknown/unmapped types fall back to "string"
// so emission never fails on an exotic flag.
func mapPflagType(t string) (cmdhelpType string, repeatable bool) {
switch t {
case "string":
return "string", false
case "int", "int8", "int16", "int32", "int64",
"uint", "uint8", "uint16", "uint32", "uint64",
"count":
return "int", false
case "float32", "float64":
return "float", false
case "bool":
return "bool", false
case "duration":
return "duration", false
case "stringSlice", "stringArray", "stringToString":
return "string", true
case "intSlice":
return "int", true
case "boolSlice":
return "bool", true
case "ip", "ipMask", "ipNet", "ipSlice":
return "string", t == "ipSlice"
case "bytesHex", "bytesBase64":
return "string", false
default:
return "string", false
}
}
// parseExamples turns cobra's Example field (a single multiline string)
// into individual Example entries. Each non-empty, non-comment line is
// taken as one runnable invocation; comment lines (`# ...`) are dropped.
//
// The convention in pad's existing commands is two-space-indented lines
// like:
//
// pad item create task "Fix" --priority high
//
// We trim leading/trailing whitespace per line.
func parseExamples(example string) []Example {
example = strings.TrimSpace(example)
if example == "" {
return nil
}
var examples []Example
for _, line := range strings.Split(example, "\n") {
line = strings.TrimSpace(line)
if line == "" || strings.HasPrefix(line, "#") {
continue
}
examples = append(examples, Example{Cmd: line})
}
if len(examples) == 0 {
return nil
}
return examples
}
// firstLine returns the first line of s with surrounding whitespace
// trimmed. Useful for collapsing a multi-line Long string into a Summary.
func firstLine(s string) string {
if i := strings.IndexByte(s, '\n'); i >= 0 {
return strings.TrimSpace(s[:i])
}
return strings.TrimSpace(s)
}
// stripCommentIndex returns the position of an unquoted `#` that begins
// a same-line comment in s, or -1 if no such comment exists. Comments
// preceded by an unquoted `#` and at least one whitespace separator are
// considered comments; `#abc` glued to a token is left alone (could be
// a literal). Quote handling matches shell semantics: `#` inside double
// or single quotes is a literal character, not a comment.
func stripCommentIndex(s string) int {
var inDQuote, inSQuote, escape bool
for i, r := range s {
switch {
case escape:
escape = false
case r == '\\' && !inSQuote:
escape = true
case r == '"' && !inSQuote:
inDQuote = !inDQuote
case r == '\'' && !inDQuote:
inSQuote = !inSQuote
case r == '#' && !inDQuote && !inSQuote:
// Require whitespace before # (or # at start) to avoid eating
// literal #s glued to a token.
if i == 0 {
return i
}
prev := s[i-1]
if prev == ' ' || prev == '\t' {
return i
}
}
}
return -1
}