Adopt the release train: promote the soaked candidate, not the branch tip

Stable promotions built whatever the dispatch branch was at that second. The resolver checked that HEAD descends from the promoted release candidate but never that its content matches, so v6.4.0 shipped 64 changed files, including product code, that v6.4.0-rc.12 had not soaked. Every v6 version was mapped to main, which now moves every few minutes under the autonomous maintainer, so each fix to a candidate brought everything landed since and stable was never an exact soaked commit. Five of six stable minor releases shipped under version-bound owner exceptions that waived the soak. From v6.5.0 the release train applies (RELEASE_PROMOTION_POLICY.md, "Release Train"): a two-week train sized to measured velocity, a release/v6.N branch per train declared in the control plane so the workflow refuses a dispatch from anywhere else, a stable promotion that may differ from its candidate only in release metadata unless hotfix_exception names active customer harm, and a seven day soak for minor releases. The 6.4.x line stays on main so the v6.4.3-rc.1 candidate already prepared there is unaffected. The gap is registered as coverage gap release-train-exact-candidate-promotion.
2026-09-10 02:25:56 +00:00 · 2026-09-01 22:56:16 +01:00
parent 62003caaea
commit df7ad9be43
6 changed files with 244 additions and 1 deletions
@@ -17,6 +17,11 @@
      "version_prefix": "6.3.2",
      "prerelease_branch": "release/v6.3.2",
      "stable_branch": "release/v6.3.2"
+    },
+    {
+      "version_prefix": "6.5.",
+      "prerelease_branch": "release/v6.5",
+      "stable_branch": "release/v6.5"
    }
  ],
  "profiles": [
@@ -233,6 +233,53 @@ TLS-unverified receipts leave the claim at `implemented` or
 6. Failed prereleases are fixed forward and replaced with a new prerelease. They are never
   promoted as-is to `stable`.

+## Release Train
+
+Adopted 2026-09-01 under the delivery contract
+(`pulse-dev-infra/docs/delivery-contract.md`). The train exists so that a
+stable release is an exact soaked candidate rather than the tip of a branch
+that keeps moving, and so that the always-running maintainer can release
+without the other lanes changing the candidate underneath it.
+
+1. A minor train runs every two weeks. The first candidate is cut on a
+   fixed day and time, Tuesday 09:00 Europe/London, and general availability
+   is the Tuesday one week later; the next train's candidate is cut the
+   Tuesday after that. The first train is v6.5.0: `v6.5.0-rc.1` on
+   2026-09-08, general availability on 2026-09-15, then `v6.6.0-rc.1` on
+   2026-09-22. The cadence follows measured velocity, not preference: at the
+   70 to 150 commits a day `main` received in the week before adoption, a
+   four-week train would put two to four thousand commits into every user
+   upgrade, and a two-week train halves that while keeping a full seven day
+   soak. The release steward reviews the cadence against velocity after
+   every third train and records the decision here.
+2. Each train has its own branch, `release/v6.N`, created from `main` at cut
+   time and declared in `docs/release-control/control_plane.json` so the
+   release workflow refuses a dispatch from any other branch. `main` is never
+   frozen. A fix for something found in the candidate is backported to the
+   release branch through a pull request; each backport produces the next
+   `rc.N` and restarts the soak. After general availability the branch is
+   the patch line for that train.
+3. General availability promotes the candidate's content. The resolver
+   refuses a stable promotion whose tree differs from the promoted candidate
+   in anything but release metadata (version, chart, compose, release notes,
+   upgrade guide, release records) unless `hotfix_exception` names active
+   customer harm. The v6.4.0 promotion, which shipped 64 changed files that
+   `v6.4.0-rc.12` had not soaked, is the case this rule prevents.
+4. A minor release (`X.Y.0`) requires a seven day soak of the promoted
+   candidate; patch releases keep the 72 hour minimum. Patch releases are for
+   a named regression or security issue only.
+5. "Soaked clean" means all of: the soak has elapsed since the candidate's
+   release was published; no open issue labelled `affects-<candidate
+   version>` is at high or critical severity; the maintainer's dogfood
+   instance and the demo server ran the candidate for the whole soak without
+   an incident; and preview-channel telemetry, where it exists, shows no
+   elevated failure rate. The release steward names this evidence in the
+   packet.
+6. Version-bound owner exceptions waived the soak for v6.0.0, v6.1.0,
+   v6.2.0, v6.3.0, and v6.4.0. They remain recorded and bounded; the train
+   does not continue the practice. An exception requires active customer
+   harm and is recorded in the release notes.
+
 ## Paid Pro Artifact Lineage

 1. Customer-facing private Pulse Pro archives and private Pulse Pro Docker images
@@ -9943,6 +9943,38 @@
          "kind": "file"
        }
      ]
+    },
+    {
+      "id": "release-train-exact-candidate-promotion",
+      "summary": "Stable promotions built whatever the dispatch branch was at that second: the resolver checked that HEAD descends from the promoted RC but not that its content matches, so v6.4.0 shipped 64 changed files that v6.4.0-rc.12 never soaked, and every v6 version was mapped to main, which moves every few minutes under the autonomous maintainer. The release train needs a branch per train declared in the control plane, a resolver rule that refuses stable content beyond release metadata without a recorded hotfix reason, and a seven day soak for minor releases so the two-week cadence carries a real soak.",
+      "owner": "project-owner",
+      "status": "triaged",
+      "recorded_at": "2026-09-01",
+      "lane_ids": [
+        "L1"
+      ],
+      "subsystem_ids": [
+        "deployment-installability"
+      ],
+      "proposed_resolution": "lane-expansion",
+      "coverage_impact": 5,
+      "evidence": [
+        {
+          "repo": "pulse",
+          "path": "docs/release-control/control_plane.json",
+          "kind": "file"
+        },
+        {
+          "repo": "pulse",
+          "path": "docs/release-control/v6/internal/RELEASE_PROMOTION_POLICY.md",
+          "kind": "file"
+        },
+        {
+          "repo": "pulse",
+          "path": "scripts/release_control/resolve_release_promotion.py",
+          "kind": "file"
+        }
+      ]
    }
  ],
  "candidate_lanes": [
@@ -4953,3 +4953,19 @@ boundary. `scripts/check_workflow_trust.py`,
 `scripts/tests/test_workflow_trust.py`, and
 `scripts/installtests/build_release_assets_test.go` pin the policy and the
 release-workflow integration.
+
+### Stable promotion ships the soaked candidate, not the branch tip
+
+The promotion resolver now compares the promoted release candidate's commit
+with the dispatch head. A stable promotion may differ from its candidate only
+in release metadata: `VERSION`, the Helm chart version and README, the compose
+default image, `docs/RELEASE_NOTES.md`, `docs/UPGRADE_v6.md` and its shipped
+docs mirror, `docs/releases/`, release-control records, this contract, and
+`status.json`. Any other path refuses the promotion unless `hotfix_exception`
+names active customer harm, because that content was never soaked. Minor
+releases (`X.Y.0`) additionally require a seven day soak; patches keep 72
+hours. `docs/release-control/control_plane.json` declares `release/v6.5` for
+the first release train so the workflow refuses a v6.5 dispatch from any other
+branch. `scripts/release_control/resolve_release_promotion_test.py` pins the
+allowlist, the drift refusal, the hotfix path, and the minor soak;
+`release_promotion_policy_test.py` pins the policy's Release Train section.
@@ -24,6 +24,28 @@ SEMVER_PUBLISHED_PRERELEASE_RE = re.compile(
 )
 SEMVER_RC_RE = re.compile(r"^(\d+)\.(\d+)\.(\d+)-rc\.(\d+)$")
 MIN_PRERELEASE_OBSERVATION_HOURS = 24
+# Release train (RELEASE_PROMOTION_POLICY.md, "Release Train"): a patch keeps
+# the 72 hour candidate soak; a minor release soaks its candidate for a week.
+MIN_STABLE_SOAK_HOURS = 72
+MIN_MINOR_STABLE_SOAK_HOURS = 168
+RELEASE_TRAIN_MIN_VERSION = (6, 5, 0)
+# Paths a stable promotion may change relative to its promoted candidate.
+# Everything else is content the candidate never soaked, so the resolver
+# refuses it unless hotfix_exception names active customer harm.
+RELEASE_METADATA_PATH_RE = re.compile(
+    r"^(?:"
+    r"VERSION"
+    r"|deploy/helm/pulse/(?:Chart\.yaml|README\.md)"
+    r"|docker-compose\.yml"
+    r"|docs/RELEASE_NOTES\.md"
+    r"|docs/UPGRADE_v6\.md"
+    r"|frontend-modern/public/docs/(?:RELEASE_NOTES|UPGRADE_v6)\.md"
+    r"|docs/releases/.+"
+    r"|docs/release-control/v6/internal/records/.+"
+    r"|docs/release-control/v6/internal/status\.json"
+    r"|docs/release-control/v6/internal/subsystems/deployment-installability\.md"
+    r")$"
+)
 WINDOWS_AUTHENTICODE_AVAILABLE = False
 WINDOWS_AUTHENTICODE_STANDING_UNSIGNED_MIN_VERSION = (6, 3, 2)
 WINDOWS_AUTHENTICODE_UNAVAILABLE_REASON = (
@@ -492,11 +514,37 @@ def resolve_metadata(
            promoted_tag_ts = tag_created_unix_fn(promoted_from_tag)
            soak_hours_value = int((now_unix_fn() - promoted_tag_ts) / 3600)
            soak_hours = str(soak_hours_value)
+            # The release train governs v6.5.0 and later. Earlier lines shipped
+            # under the previous regime and their recorded exceptions stand.
+            train_governed = bool(stable_version and stable_version >= RELEASE_TRAIN_MIN_VERSION)
+            candidate_content_drift: list[str] = []
+            if train_governed:
+                candidate_content_drift = [
+                    path
+                    for path in changed_paths_fn(promoted_from_tag)
+                    if not RELEASE_METADATA_PATH_RE.match(path)
+                ]

            if hotfix_exception:
                if not hotfix_reason:
                    raise ValueError("hotfix_reason is required when hotfix_exception is true.")
-            elif soak_hours_value < 72:
+            elif candidate_content_drift:
+                shown = ", ".join(candidate_content_drift[:10])
+                if len(candidate_content_drift) > 10:
+                    shown += f", and {len(candidate_content_drift) - 10} more"
+                raise ValueError(
+                    f"Stable promotion {tag} would ship content that {promoted_from_tag} never soaked "
+                    f"({len(candidate_content_drift)} paths beyond release metadata: {shown}). "
+                    "Cut another release candidate from the release branch, or use hotfix_exception "
+                    "with a concrete active-customer-harm reason."
+                )
+            elif train_governed and not stable_patch and soak_hours_value < MIN_MINOR_STABLE_SOAK_HOURS:
+                raise ValueError(
+                    f"Minor stable promotion {tag} has only {soak_hours_value} hours of prerelease soak since "
+                    f"{promoted_from_tag}; the release train requires {MIN_MINOR_STABLE_SOAK_HOURS} hours "
+                    "(seven days) for a minor release unless hotfix_exception is true."
+                )
+            elif soak_hours_value < MIN_STABLE_SOAK_HOURS:
                raise ValueError(
                    f"Stable promotion {tag} has only {soak_hours_value} hours of prerelease soak since {promoted_from_tag}; minimum is 72 hours unless hotfix_exception is true."
                )
@@ -790,5 +790,100 @@ class ResolveReleasePromotionTest(unittest.TestCase):
            )


+class ReleaseTrainPromotionTest(unittest.TestCase):
+    """The release train: a stable ships its soaked candidate, and minors soak a week."""
+
+    def promote(self, version: str, **overrides):
+        promoted = f"{version}-rc.1"
+        arguments = dict(
+            version=version,
+            promoted_from_tag_input=promoted,
+            rollback_version_input="6.4.1",
+            ga_date_input="",
+            v5_eos_date_input="",
+            hotfix_exception=False,
+            hotfix_reason_input="",
+            release_notes_input="",
+            tag_exists_fn=lambda tag: tag in {f"v{promoted}", "v6.4.1"},
+            tag_commit_fn=lambda tag: "abc123",
+            head_descends_from_fn=lambda commit: commit == "abc123",
+            tag_created_unix_fn=lambda tag: 100,
+            now_unix_fn=lambda: 100 + (168 * 3600),
+            changed_paths_fn=lambda base_tag: [
+                "VERSION",
+                "deploy/helm/pulse/Chart.yaml",
+                "docs/RELEASE_NOTES.md",
+                "docs/releases/RELEASE_NOTES_v6.5.0.md",
+                "docs/release-control/v6/internal/status.json",
+            ],
+        )
+        arguments.update(overrides)
+        return resolver.resolve_metadata(**arguments)
+
+    def test_release_metadata_paths_are_the_only_allowed_drift(self) -> None:
+        for path in (
+            "VERSION",
+            "deploy/helm/pulse/Chart.yaml",
+            "deploy/helm/pulse/README.md",
+            "docker-compose.yml",
+            "docs/RELEASE_NOTES.md",
+            "docs/UPGRADE_v6.md",
+            "frontend-modern/public/docs/UPGRADE_v6.md",
+            "docs/releases/V6_CHANGELOG_v6.5.0.md",
+            "docs/release-control/v6/internal/records/v6.5.0-ga.md",
+            "docs/release-control/v6/internal/status.json",
+            "docs/release-control/v6/internal/subsystems/deployment-installability.md",
+        ):
+            with self.subTest(path=path):
+                self.assertIsNotNone(resolver.RELEASE_METADATA_PATH_RE.match(path))
+        for path in (
+            "internal/api/router.go",
+            "frontend-modern/src/App.tsx",
+            ".github/workflows/create-release.yml",
+            "docs/TRUENAS.md",
+            "docs/release-control/v6/internal/RELEASE_PROMOTION_POLICY.md",
+            "scripts/install.sh",
+        ):
+            with self.subTest(path=path):
+                self.assertIsNone(resolver.RELEASE_METADATA_PATH_RE.match(path))
+
+    def test_minor_promotion_ships_exactly_the_soaked_candidate(self) -> None:
+        metadata = self.promote("6.5.0")
+        self.assertEqual(metadata["promoted_from_tag"], "v6.5.0-rc.1")
+        self.assertEqual(metadata["soak_hours"], "168")
+
+    def test_content_the_candidate_never_soaked_is_refused(self) -> None:
+        drift = [
+            "VERSION",
+            "internal/api/router.go",
+            "frontend-modern/src/features/proxmox/ProxmoxBackupServersTable.tsx",
+        ]
+        with self.assertRaisesRegex(ValueError, "never soaked.*2 paths beyond release metadata"):
+            self.promote("6.5.0", changed_paths_fn=lambda base_tag: drift)
+        with self.assertRaisesRegex(ValueError, "never soaked"):
+            self.promote("6.5.1", changed_paths_fn=lambda base_tag: drift)
+
+    def test_hotfix_exception_still_requires_a_reason_for_drift(self) -> None:
+        drift = ["VERSION", "internal/api/router.go"]
+        with self.assertRaisesRegex(ValueError, "hotfix_reason is required"):
+            self.promote("6.5.1", hotfix_exception=True, changed_paths_fn=lambda base_tag: drift)
+        metadata = self.promote(
+            "6.5.1",
+            hotfix_exception=True,
+            hotfix_reason_input="Active customer harm: agents cannot re-enrol after upgrade.",
+            changed_paths_fn=lambda base_tag: drift,
+            now_unix_fn=lambda: 100 + (2 * 3600),
+        )
+        self.assertEqual(metadata["hotfix_exception"], "true")
+
+    def test_minor_releases_soak_seven_days_and_patches_seventy_two_hours(self) -> None:
+        with self.assertRaisesRegex(ValueError, "release train requires 168 hours"):
+            self.promote("6.5.0", now_unix_fn=lambda: 100 + (100 * 3600))
+        metadata = self.promote("6.5.1", now_unix_fn=lambda: 100 + (73 * 3600))
+        self.assertEqual(metadata["soak_hours"], "73")
+        with self.assertRaisesRegex(ValueError, "minimum is 72 hours"):
+            self.promote("6.5.1", now_unix_fn=lambda: 100 + (71 * 3600))
+
+
 if __name__ == "__main__":
    unittest.main()