Merge pull request #1825 from rcourtman/topic/release-train

Adopt the release train: promote the soaked candidate, not the branch tip

docs/release-control/control_plane.json

@@ -17,6 +17,11 @@
       "version_prefix": "6.3.2",
       "prerelease_branch": "release/v6.3.2",
       "stable_branch": "release/v6.3.2"
+    },
+    {
+      "version_prefix": "6.5.",
+      "prerelease_branch": "release/v6.5",
+      "stable_branch": "release/v6.5"
     }
   ],
   "profiles": [

docs/release-control/v6/internal/RELEASE_PROMOTION_POLICY.md

@@ -233,6 +233,53 @@ TLS-unverified receipts leave the claim at `implemented` or
 6. Failed prereleases are fixed forward and replaced with a new prerelease. They are never
    promoted as-is to `stable`.
 
+## Release Train
+
+Adopted 2026-09-01 under the delivery contract
+(`pulse-dev-infra/docs/delivery-contract.md`). The train exists so that a
+stable release is an exact soaked candidate rather than the tip of a branch
+that keeps moving, and so that the always-running maintainer can release
+without the other lanes changing the candidate underneath it.
+
+1. A minor train runs every two weeks. The first candidate is cut on a
+   fixed day and time, Tuesday 09:00 Europe/London, and general availability
+   is the Tuesday one week later; the next train's candidate is cut the
+   Tuesday after that. The first train is v6.5.0: `v6.5.0-rc.1` on
+   2026-09-08, general availability on 2026-09-15, then `v6.6.0-rc.1` on
+   2026-09-22. The cadence follows measured velocity, not preference: at the
+   70 to 150 commits a day `main` received in the week before adoption, a
+   four-week train would put two to four thousand commits into every user
+   upgrade, and a two-week train halves that while keeping a full seven day
+   soak. The release steward reviews the cadence against velocity after
+   every third train and records the decision here.
+2. Each train has its own branch, `release/v6.N`, created from `main` at cut
+   time and declared in `docs/release-control/control_plane.json` so the
+   release workflow refuses a dispatch from any other branch. `main` is never
+   frozen. A fix for something found in the candidate is backported to the
+   release branch through a pull request; each backport produces the next
+   `rc.N` and restarts the soak. After general availability the branch is
+   the patch line for that train.
+3. General availability promotes the candidate's content. The resolver
+   refuses a stable promotion whose tree differs from the promoted candidate
+   in anything but release metadata (version, chart, compose, release notes,
+   upgrade guide, release records) unless `hotfix_exception` names active
+   customer harm. The v6.4.0 promotion, which shipped 64 changed files that
+   `v6.4.0-rc.12` had not soaked, is the case this rule prevents.
+4. A minor release (`X.Y.0`) requires a seven day soak of the promoted
+   candidate; patch releases keep the 72 hour minimum. Patch releases are for
+   a named regression or security issue only.
+5. "Soaked clean" means all of: the soak has elapsed since the candidate's
+   release was published; no open issue labelled `affects-<candidate
+   version>` is at high or critical severity; the maintainer's dogfood
+   instance and the demo server ran the candidate for the whole soak without
+   an incident; and preview-channel telemetry, where it exists, shows no
+   elevated failure rate. The release steward names this evidence in the
+   packet.
+6. Version-bound owner exceptions waived the soak for v6.0.0, v6.1.0,
+   v6.2.0, v6.3.0, and v6.4.0. They remain recorded and bounded; the train
+   does not continue the practice. An exception requires active customer
+   harm and is recorded in the release notes.
+
 ## Paid Pro Artifact Lineage
 
 1. Customer-facing private Pulse Pro archives and private Pulse Pro Docker images

docs/release-control/v6/internal/status.json

@@ -9943,6 +9943,38 @@
           "kind": "file"
         }
       ]
+    },
+    {
+      "id": "release-train-exact-candidate-promotion",
+      "summary": "Stable promotions built whatever the dispatch branch was at that second: the resolver checked that HEAD descends from the promoted RC but not that its content matches, so v6.4.0 shipped 64 changed files that v6.4.0-rc.12 never soaked, and every v6 version was mapped to main, which moves every few minutes under the autonomous maintainer. The release train needs a branch per train declared in the control plane, a resolver rule that refuses stable content beyond release metadata without a recorded hotfix reason, and a seven day soak for minor releases so the two-week cadence carries a real soak.",
+      "owner": "project-owner",
+      "status": "triaged",
+      "recorded_at": "2026-09-01",
+      "lane_ids": [
+        "L1"
+      ],
+      "subsystem_ids": [
+        "deployment-installability"
+      ],
+      "proposed_resolution": "lane-expansion",
+      "coverage_impact": 5,
+      "evidence": [
+        {
+          "repo": "pulse",
+          "path": "docs/release-control/control_plane.json",
+          "kind": "file"
+        },
+        {
+          "repo": "pulse",
+          "path": "docs/release-control/v6/internal/RELEASE_PROMOTION_POLICY.md",
+          "kind": "file"
+        },
+        {
+          "repo": "pulse",
+          "path": "scripts/release_control/resolve_release_promotion.py",
+          "kind": "file"
+        }
+      ]
     }
   ],
   "candidate_lanes": [

docs/release-control/v6/internal/subsystems/deployment-installability.md

@@ -4971,3 +4971,19 @@ boundary. `scripts/check_workflow_trust.py`,
 `scripts/tests/test_workflow_trust.py`, and
 `scripts/installtests/build_release_assets_test.go` pin the policy and the
 release-workflow integration.
+
+### Stable promotion ships the soaked candidate, not the branch tip
+
+The promotion resolver now compares the promoted release candidate's commit
+with the dispatch head. A stable promotion may differ from its candidate only
+in release metadata: `VERSION`, the Helm chart version and README, the compose
+default image, `docs/RELEASE_NOTES.md`, `docs/UPGRADE_v6.md` and its shipped
+docs mirror, `docs/releases/`, release-control records, this contract, and
+`status.json`. Any other path refuses the promotion unless `hotfix_exception`
+names active customer harm, because that content was never soaked. Minor
+releases (`X.Y.0`) additionally require a seven day soak; patches keep 72
+hours. `docs/release-control/control_plane.json` declares `release/v6.5` for
+the first release train so the workflow refuses a v6.5 dispatch from any other
+branch. `scripts/release_control/resolve_release_promotion_test.py` pins the
+allowlist, the drift refusal, the hotfix path, and the minor soak;
+`release_promotion_policy_test.py` pins the policy's Release Train section.

scripts/release_control/resolve_release_promotion.py

@@ -24,6 +24,28 @@ SEMVER_PUBLISHED_PRERELEASE_RE = re.compile(
 )
 SEMVER_RC_RE = re.compile(r"^(\d+)\.(\d+)\.(\d+)-rc\.(\d+)$")
 MIN_PRERELEASE_OBSERVATION_HOURS = 24
+# Release train (RELEASE_PROMOTION_POLICY.md, "Release Train"): a patch keeps
+# the 72 hour candidate soak; a minor release soaks its candidate for a week.
+MIN_STABLE_SOAK_HOURS = 72
+MIN_MINOR_STABLE_SOAK_HOURS = 168
+RELEASE_TRAIN_MIN_VERSION = (6, 5, 0)
+# Paths a stable promotion may change relative to its promoted candidate.
+# Everything else is content the candidate never soaked, so the resolver
+# refuses it unless hotfix_exception names active customer harm.
+RELEASE_METADATA_PATH_RE = re.compile(
+    r"^(?:"
+    r"VERSION"
+    r"|deploy/helm/pulse/(?:Chart\.yaml|README\.md)"
+    r"|docker-compose\.yml"
+    r"|docs/RELEASE_NOTES\.md"
+    r"|docs/UPGRADE_v6\.md"
+    r"|frontend-modern/public/docs/(?:RELEASE_NOTES|UPGRADE_v6)\.md"
+    r"|docs/releases/.+"
+    r"|docs/release-control/v6/internal/records/.+"
+    r"|docs/release-control/v6/internal/status\.json"
+    r"|docs/release-control/v6/internal/subsystems/deployment-installability\.md"
+    r")$"
+)
 WINDOWS_AUTHENTICODE_AVAILABLE = False
 WINDOWS_AUTHENTICODE_STANDING_UNSIGNED_MIN_VERSION = (6, 3, 2)
 WINDOWS_AUTHENTICODE_UNAVAILABLE_REASON = (
@@ -492,11 +514,37 @@ def resolve_metadata(
             promoted_tag_ts = tag_created_unix_fn(promoted_from_tag)
             soak_hours_value = int((now_unix_fn() - promoted_tag_ts) / 3600)
             soak_hours = str(soak_hours_value)
+            # The release train governs v6.5.0 and later. Earlier lines shipped
+            # under the previous regime and their recorded exceptions stand.
+            train_governed = bool(stable_version and stable_version >= RELEASE_TRAIN_MIN_VERSION)
+            candidate_content_drift: list[str] = []
+            if train_governed:
+                candidate_content_drift = [
+                    path
+                    for path in changed_paths_fn(promoted_from_tag)
+                    if not RELEASE_METADATA_PATH_RE.match(path)
+                ]
 
             if hotfix_exception:
                 if not hotfix_reason:
                     raise ValueError("hotfix_reason is required when hotfix_exception is true.")
-            elif soak_hours_value < 72:
+            elif candidate_content_drift:
+                shown = ", ".join(candidate_content_drift[:10])
+                if len(candidate_content_drift) > 10:
+                    shown += f", and {len(candidate_content_drift) - 10} more"
+                raise ValueError(
+                    f"Stable promotion {tag} would ship content that {promoted_from_tag} never soaked "
+                    f"({len(candidate_content_drift)} paths beyond release metadata: {shown}). "
+                    "Cut another release candidate from the release branch, or use hotfix_exception "
+                    "with a concrete active-customer-harm reason."
+                )
+            elif train_governed and not stable_patch and soak_hours_value < MIN_MINOR_STABLE_SOAK_HOURS:
+                raise ValueError(
+                    f"Minor stable promotion {tag} has only {soak_hours_value} hours of prerelease soak since "
+                    f"{promoted_from_tag}; the release train requires {MIN_MINOR_STABLE_SOAK_HOURS} hours "
+                    "(seven days) for a minor release unless hotfix_exception is true."
+                )
+            elif soak_hours_value < MIN_STABLE_SOAK_HOURS:
                 raise ValueError(
                     f"Stable promotion {tag} has only {soak_hours_value} hours of prerelease soak since {promoted_from_tag}; minimum is 72 hours unless hotfix_exception is true."
                 )

scripts/release_control/resolve_release_promotion_test.py

@@ -790,5 +790,100 @@ class ResolveReleasePromotionTest(unittest.TestCase):
             )
 
 
+class ReleaseTrainPromotionTest(unittest.TestCase):
+    """The release train: a stable ships its soaked candidate, and minors soak a week."""
+
+    def promote(self, version: str, **overrides):
+        promoted = f"{version}-rc.1"
+        arguments = dict(
+            version=version,
+            promoted_from_tag_input=promoted,
+            rollback_version_input="6.4.1",
+            ga_date_input="",
+            v5_eos_date_input="",
+            hotfix_exception=False,
+            hotfix_reason_input="",
+            release_notes_input="",
+            tag_exists_fn=lambda tag: tag in {f"v{promoted}", "v6.4.1"},
+            tag_commit_fn=lambda tag: "abc123",
+            head_descends_from_fn=lambda commit: commit == "abc123",
+            tag_created_unix_fn=lambda tag: 100,
+            now_unix_fn=lambda: 100 + (168 * 3600),
+            changed_paths_fn=lambda base_tag: [
+                "VERSION",
+                "deploy/helm/pulse/Chart.yaml",
+                "docs/RELEASE_NOTES.md",
+                "docs/releases/RELEASE_NOTES_v6.5.0.md",
+                "docs/release-control/v6/internal/status.json",
+            ],
+        )
+        arguments.update(overrides)
+        return resolver.resolve_metadata(**arguments)
+
+    def test_release_metadata_paths_are_the_only_allowed_drift(self) -> None:
+        for path in (
+            "VERSION",
+            "deploy/helm/pulse/Chart.yaml",
+            "deploy/helm/pulse/README.md",
+            "docker-compose.yml",
+            "docs/RELEASE_NOTES.md",
+            "docs/UPGRADE_v6.md",
+            "frontend-modern/public/docs/UPGRADE_v6.md",
+            "docs/releases/V6_CHANGELOG_v6.5.0.md",
+            "docs/release-control/v6/internal/records/v6.5.0-ga.md",
+            "docs/release-control/v6/internal/status.json",
+            "docs/release-control/v6/internal/subsystems/deployment-installability.md",
+        ):
+            with self.subTest(path=path):
+                self.assertIsNotNone(resolver.RELEASE_METADATA_PATH_RE.match(path))
+        for path in (
+            "internal/api/router.go",
+            "frontend-modern/src/App.tsx",
+            ".github/workflows/create-release.yml",
+            "docs/TRUENAS.md",
+            "docs/release-control/v6/internal/RELEASE_PROMOTION_POLICY.md",
+            "scripts/install.sh",
+        ):
+            with self.subTest(path=path):
+                self.assertIsNone(resolver.RELEASE_METADATA_PATH_RE.match(path))
+
+    def test_minor_promotion_ships_exactly_the_soaked_candidate(self) -> None:
+        metadata = self.promote("6.5.0")
+        self.assertEqual(metadata["promoted_from_tag"], "v6.5.0-rc.1")
+        self.assertEqual(metadata["soak_hours"], "168")
+
+    def test_content_the_candidate_never_soaked_is_refused(self) -> None:
+        drift = [
+            "VERSION",
+            "internal/api/router.go",
+            "frontend-modern/src/features/proxmox/ProxmoxBackupServersTable.tsx",
+        ]
+        with self.assertRaisesRegex(ValueError, "never soaked.*2 paths beyond release metadata"):
+            self.promote("6.5.0", changed_paths_fn=lambda base_tag: drift)
+        with self.assertRaisesRegex(ValueError, "never soaked"):
+            self.promote("6.5.1", changed_paths_fn=lambda base_tag: drift)
+
+    def test_hotfix_exception_still_requires_a_reason_for_drift(self) -> None:
+        drift = ["VERSION", "internal/api/router.go"]
+        with self.assertRaisesRegex(ValueError, "hotfix_reason is required"):
+            self.promote("6.5.1", hotfix_exception=True, changed_paths_fn=lambda base_tag: drift)
+        metadata = self.promote(
+            "6.5.1",
+            hotfix_exception=True,
+            hotfix_reason_input="Active customer harm: agents cannot re-enrol after upgrade.",
+            changed_paths_fn=lambda base_tag: drift,
+            now_unix_fn=lambda: 100 + (2 * 3600),
+        )
+        self.assertEqual(metadata["hotfix_exception"], "true")
+
+    def test_minor_releases_soak_seven_days_and_patches_seventy_two_hours(self) -> None:
+        with self.assertRaisesRegex(ValueError, "release train requires 168 hours"):
+            self.promote("6.5.0", now_unix_fn=lambda: 100 + (100 * 3600))
+        metadata = self.promote("6.5.1", now_unix_fn=lambda: 100 + (73 * 3600))
+        self.assertEqual(metadata["soak_hours"], "73")
+        with self.assertRaisesRegex(ValueError, "minimum is 72 hours"):
+            self.promote("6.5.1", now_unix_fn=lambda: 100 + (71 * 3600))
+
+
 if __name__ == "__main__":
     unittest.main()