MCP rule authoring: deterministic context, validate_rule, and session dry-run (#124)

* feat(backend): dry-run endpoint for draft analyze rules (rule authoring)

POST /api/rules/analyze/dryrun evaluates a DRAFT analyze rule against one
session's events and returns a full diagnostic trace: per-condition
matched/evidence (incl. per-eventType occurrence counts), per-factor
confidence breakdown, verdict, and the would-be MatchedConditions map for
{{token}} interpolation preview. Persists nothing: no RuleResult row, no
session-status change, no notifications.

- RuleEngine.DryRun.cs: DryRunRuleAsync / DryRunEvaluate share the exact
  evaluators (EvaluatePrecondition/EvaluateCondition/EvaluateConfidenceFactor)
  with the production path but evaluate ALL conditions instead of breaking on
  the first failed required one; parity with EvaluateRule is pinned by tests.
- DryRunAnalyzeRuleFunction: TenantAdminOrGlobalReader (read-in-effect POST),
  GA/GlobalReader cross-tenant via session tenant resolution, 1 MB body cap,
  structural draft validation (unknown source/operator, correlation without
  joinField, invalid regex, non-evaluable confidenceFactor DSL) with
  actionable error messages.
- EndpointAccessPolicyCatalog: register the new route.
- RuleEngineDryRunTests: trace semantics, production parity matrix,
  side-effect freedom (VerifyNoOtherCalls), draft validation matrix.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(mcp): rule-authoring resources — guide, schemas, guardrails

Deterministic authoring context for customer AIs, replacing chunked
search_docs retrieval for rule creation: get_resource (and the MCP-protocol
resources) now serve

- rule_authoring_guide: hand-written, complete authoring knowledge (collector
  types, triggers, condition sources, operators, confidence model incl. the
  exact factor-condition DSL shapes, {{token}} interpolation order, guardrail
  matching semantics, pitfalls, validate->dry-run workflow)
- rule_schemas: the JSON Schemas (2020-12) for gather + analyze rules
- rule_guardrails: the on-device collection allowlists

Schemas + guardrails are generated into rule-authoring.generated.ts by a new
combine.js emitter (same pattern as the web guardrails mirror; rules/ stays
the single source of truth and CI guardrails-in-sync guards the file).
rule-authoring-drift.test.ts additionally pins generated == repo source,
guide enums == schema enums, and the get_resource size cap.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(mcp): validate_rule + test_analyze_rule tools and author-rule prompt

Closes the authoring loop for customer AIs: draft -> validate -> dry-run
against a real session -> iterate -> create in the portal.

- validate_rule (MCP-local, no backend call): ajv 2020-12 validation against
  the baked schemas, guardrail pre-flight mirroring the agent matching
  semantics (hive-stripped segment-bounded registry/file prefixes,
  whitespace-bounded WMI, exact commands, /-bounded event log channels, hard
  blocks), and semantic lint for schema-valid-but-dead rules (unreachable
  threshold, non-evaluable factor DSL, unknown eventTypes, unanchored
  not_regex allowlists, unresolvable {{tokens}}, trigger cross-fields,
  outputEventType conventions/collisions).
- test_analyze_rule: calls the new POST /api/rules/analyze/dryrun and adds
  the {{token}}-interpolated explanation/remediation preview via the
  existing interpolate-rule-template.
- author-rule MCP prompt guiding scenario -> resources -> draft -> validate
  -> dry-run.
- Both tools are visible to every role (backend enforces
  TenantAdminOrGlobalReader on the dryrun route); catalog snapshot updated.
- deps: ajv + ajv-formats promoted to direct dependencies.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(okf): rule authoring surface concept doc

docs/rules/rule-authoring-surface.md: the AI-assisted authoring stack
(deterministic MCP resources via combine.js baking, MCP-local validate_rule
layers, the dry-run trace twin with its parity + side-effect invariants, and
what is deliberately absent); registered in index.md, logged 2026-08-05.

Customer docs (rules/ai-assisted-rule-authoring.md + SUMMARY + FAQ entry) are
committed in the autopilotmonitor-docs repository separately.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Oliver Kieselbach
2026-08-05 17:24:28 +02:00
committed by GitHub
parent b4e409783e
commit 3c1229a301
23 changed files with 3486 additions and 10 deletions
+42
View File
@@ -161,3 +161,45 @@ ${flatArray(guardrails.blockedEventLogChannels)}
} else {
console.warn('guardrails.json not found — skipping guardrails generation');
}
// ── MCP rule-authoring module generation ────────────────────────────────────
// The MCP server exposes the rule schemas + guardrails as get_resource content
// so a customer's AI can author rules against the real contract instead of
// retrieval fragments. The server's tsconfig (rootDir: src, no resolveJsonModule)
// and the Docker build (rules/schema not copied) rule out importing the JSON
// directly — so we bake it into a TS module, exactly like the web mirror above.
// CI's guardrails-in-sync job re-runs this script and fails on any diff, which
// makes drift between rules/* and this generated module impossible to merge.
const mcpAuthoringOutput = path.resolve(
rulesRoot, '..', 'src', 'McpServer', 'autopilot-monitor-mcp', 'src', 'rule-authoring.generated.ts'
);
{
const guardrails = JSON.parse(fs.readFileSync(guardrailsPath, 'utf8'));
const gatherSchema = JSON.parse(fs.readFileSync(path.join(rulesRoot, 'schema', 'gather-rule.schema.json'), 'utf8'));
const analyzeSchema = JSON.parse(fs.readFileSync(path.join(rulesRoot, 'schema', 'analyze-rule.schema.json'), 'utf8'));
const ts = `/**
* AUTO-GENERATED from rules/guardrails.json + rules/schema/*.schema.json — DO NOT EDIT.
* Run: node rules/scripts/combine.js
*
* Consumed by the MCP server's rule-authoring surface (get_resource +
* validate_rule): the JSON Schemas are the validation contract, the guardrails
* are the agent-side collection allowlists. Single source of truth is rules/;
* the CI guardrails-in-sync job guards this file against drift.
*/
/** JSON Schema (2020-12) for gather rules — rules/schema/gather-rule.schema.json verbatim. */
export const GATHER_RULE_SCHEMA: Record<string, unknown> = ${JSON.stringify(gatherSchema, null, 2)};
/** JSON Schema (2020-12) for analyze rules — rules/schema/analyze-rule.schema.json verbatim. */
export const ANALYZE_RULE_SCHEMA: Record<string, unknown> = ${JSON.stringify(analyzeSchema, null, 2)};
/** Gather-rule collection guardrails — rules/guardrails.json verbatim. */
export const RULE_GUARDRAILS = ${JSON.stringify(guardrails, null, 2)} as const;
`;
fs.writeFileSync(mcpAuthoringOutput, ts, 'utf8');
console.log('rule-authoring.generated.ts: schemas + guardrails baked for MCP server');
}