docs: record the 2026-09-29 conformance baseline

Update request and EC2 error-code counts after the suite artefacts were removed, describe how the sweep now reaches its targets, and point at the plan that traces each finding.
This commit is contained in:
Julian Sommer
2026-09-29 11:16:26 +10:00
parent bfad1dc004
commit ba2621ded5
+14 -11
View File
@@ -17,13 +17,11 @@ Twelve responses are reported separately as `errors_unmodelled`: four IAM and ei
EC2's model declares no operation errors, so EC2 uses a separate checked-in catalog curated from AWS's official EC2 error-code reference. It contains every documented common and server error plus the documented action-specific codes Spinifex currently emits. The validator checks the EC2 XML envelope, catalog membership, and AWS's documented 4xx client / 5xx server classification.
The same baseline examined 42 EC2 error responses and exposed 16 violations. The
`InsufficientInstanceCapacity` status-class defect has since been fixed — that
code now returns 503, matching the other capacity errors — leaving 15 open:
The same baseline examined 42 EC2 error responses and exposed 16 violations. The `InsufficientInstanceCapacity` status-class defect has since been fixed — that code now returns 503, matching the other capacity errors. The 2026-09-29 run examined 50 EC2 error responses, and all 23 of its violations are the one open defect, which more tests now exercise:
| Finding | Count | Classification | Implementation cause |
|---|---:|---|---|
| EC2 authorization failures return `AccessDenied`, which is absent from the EC2 catalog; EC2 documents `UnauthorizedOperation` | 15 | Defect | The shared policy evaluator returns the Query/IAM-oriented `AccessDenied` code for EC2 requests. |
| EC2 authorization failures return `AccessDenied`, which is absent from the EC2 catalog; EC2 documents `UnauthorizedOperation` | 23 | Defect | The shared policy evaluator returns the Query/IAM-oriented `AccessDenied` code for EC2 requests. |
Six additional production-referenced values are deliberately not accepted by the catalog because the chosen EC2 reference does not list them: `ExpiredToken`, `IamInstanceProfileAlreadyAssociated`, `InvalidIamInstanceProfile.NotFound`, `NoSuchAssociation`, `NoSuchEntity`, and `RequestEntityTooLarge`. A runtime occurrence remains a visible violation until it is either backed by an authoritative EC2 source or replaced with a documented EC2 code.
@@ -41,19 +39,24 @@ A rejection that succeeds is always a finding. A refusal counts as a pass only w
Request findings and undeclared errors block only for services listed under `promotedRequestServices` in `conformance-promoted-services.json`, which starts empty; response promotion does not carry over.
The 2026-09-28 baseline sent 3986 requests across 368 operations: 794 pass, 396 findings, 101 undeclared errors, 2695 inconclusive. None of it has been checked against AWS, so each finding below is a lead to verify before any fix.
The 2026-09-29 baseline sent 4047 requests across 371 operations: 980 pass, 506 findings, 96 undeclared errors, 2465 inconclusive. None of it has been checked against AWS, so each finding below is a lead to verify before any fix. Every finding is traced to its cause, and grouped into fix items, in the mulga plan `docs/development/bugs/aws-integration-conformance-findings.md`.
| Finding | Count | Notes |
|---|---:|---|
| A request breaking a model constraint is accepted | 336 | IAM 116, ECS 63, ELBv2 48, ECR 41, EC2 34, ACM 15, RDS 7, STS 6, EKS 6. Examples: IAM `List*` accept out-of-range `MaxItems` and `Marker`; IAM, ELBv2 and ECR accept a tag without `Key`; EC2 `Describe*` accept out-of-range `MaxResults`; ECS `CreateCapacityProvider` accepts every enum and range break inside `autoScalingGroupProvider`; STS `GetSessionToken` accepts a `SerialNumber` and `TokenCode` below their minimum length. |
| A model-valid request is refused as invalid | 60 | Mostly deliberate limitations that say so: RDS `ModifyDBInstance` members, STS session policies, tags and MFA, IAM `PermissionsBoundary`, ECS `availabilityZoneRebalancing`. Others return a generic message whose cause is unresolved (ECS, EC2, EKS), or the model leaves out a value AWS itself requires, such as an RDS `AllocatedStorage` below the engine minimum. |
| A refusal uses an undeclared error code | 101 | ECR (89) and EKS (1) answer with `InvalidParameterValueException`; both models declare `InvalidParameterException`. IAM (11) answers `CreateGroup` and `Put*Policy` constraint breaks with `InvalidInput`, which those operations do not declare. |
| A request breaking a model constraint is accepted | 432 | ECS 116, IAM 112, ACM 56, ELBv2 48, ECR 47, EC2 34, RDS 7, STS 6, EKS 6. Examples: ACM, ELBv2 and IAM accept a tag without `Key`; EC2 `Describe*` accept out-of-range `MaxResults`; ECS `CreateCapacityProvider` accepts every enum and range break inside `autoScalingGroupProvider` and `managedInstancesProvider`; ACM `RequestCertificate` accepts every break inside `DomainValidationOptions`; STS `GetSessionToken` accepts out-of-range `DurationSeconds` and a `SerialNumber` and `TokenCode` below their minimum length. |
| A model-valid request is refused as invalid | 74 | RDS 42, ECS 10, STS 10, EC2 8, EKS 2, IAM 2. Mostly deliberate limitations that say so: RDS `CreateDBInstance`, `ModifyDBInstance` and `RestoreDBInstanceFromDBSnapshot` members, STS session policies, tags and MFA, ECS `availabilityZoneRebalancing`. Others return a generic message whose cause is unresolved (ECS task and task-definition operations, EC2 `Modify*Attribute`), or add a member next to a seeded one that AWS may refuse in combination, such as EC2 `CreateCapacityReservation` `AvailabilityZoneId` beside `AvailabilityZone`. |
| A refusal uses an undeclared error code | 96 | ECR (90) and EKS (2) answer with `InvalidParameterValueException`; both models declare `InvalidParameterException`. IAM (4) answers `CreateGroup` and `Put*Policy` name-length breaks with `InvalidInput`, which those operations do not declare. |
How the sweep reaches its targets:
- Models mark conditionally required members optional. The members AWS requires anyway are seeded into every request for the operations listed in `conditionallyRequired` (`internal/awsmodel/request_hints.go`). For any other operation, an acceptance request refused with `MissingParameter` or `InvalidParameterCombination` is inconclusive, and so is every later refusal for that operation once its required-only request is refused.
- Members whose model shape accepts values AWS refuses, such as ARNs, IDs, zones, policy documents, certificates and ECR manifests, take hinted values of the right form (`request_hints.go`).
- A pattern Go's regexp cannot compile, because it uses lookahead, is sampled through a hand-written RE2 equivalent (`re2Equivalents` in `internal/awsmodel/pattern.go`). An operation whose required member has a pattern with no equivalent is listed as `UNTESTED`; none is today.
- The runner creates an IAM role, managed policy, OIDC provider, instance profile, group and user, an ECS cluster and an ECR repository per run, and every operation except a create or delete names them. Other services have no fixture, so their tag, update and describe cases still end in not-found.
Known limits of the sweep:
- Models mark conditionally required members optional, so an acceptance request refused with `MissingParameter` or `InvalidParameterCombination` is inconclusive, and so is every later refusal for that operation once its required-only request is refused.
- Filter names and pagination tokens cannot be generated validly, so `Filters`, `Marker` and `NextToken` get rejection requests only, and a refusal of one is inconclusive.
- ACM `RequestCertificate` sends no requests: its `DomainName` pattern uses lookahead, which Go's regexp cannot sample. The report lists such operations as `UNTESTED`.
- Constraints no request breaks are counted as `unbroken_constraints`: required URI labels, maximum lengths above the generated limit (such as IAM policy documents), map entry counts, and patterns no value of a valid length breaks.
- Most inconclusives are resource lookups that run before validation (`NoSuchEntity`, `*NotFound`), EC2 requests missing a conditionally required member (`MissingParameter`), EC2 subjects no daemon-lite answers (`InternalError`), and EKS operations whose backends are absent (`ServiceUnavailableException`, `NotImplementedException`).
- Most inconclusives are resource lookups that run before validation (`NoSuchEntity`, `*NotFound`), ECS's bare `InvalidParameterException` for a missing resource, EC2 requests missing a conditionally required member (`MissingParameter`), EC2 subjects no daemon-lite answers (`InternalError`), RDS `ServerInternal`, and EKS operations whose backends are absent (`ServiceUnavailableException`, `NotImplementedException`).
- The EC2 query parser ignores parameters it does not know, which no generated request can reveal because every member it sends is modelled.