ci: add schedule-failure-issue composite action and wire nightly alerts (#4671)

Scheduled workflow failures previously went unnoticed (15 consecutive red
weekly s3-tests sweeps, silent perf nightly failures). Add a reusable
composite action that opens a tracking issue titled
"[scheduled-failure] <workflow name>" — or appends a comment to the
existing open one (dedupe key = workflow name) — with the run URL, run
attempt, and failed job names, labeled "infrastructure".

Wire it into the scheduled paths of e2e-s3tests, mint, fuzz (nightly) and
performance-ab via a final alert-on-failure job gated by
"always() && github.event_name == 'schedule' &&
contains(needs.*.result, 'failure')", with issues:write scoped to that
job only. e2e-s3tests and mint previously had no permissions key; they now
get workflow-level contents:read, reducing every other job's token.

Add a dispatch-only drill workflow that forces a job failure and runs the
action through the exact consumer wiring, for end-to-end verification.
The ci-7 e2e nightly does not exist yet; it is recorded as a pending
consumer in the action README.

Refs: rustfs/backlog#1149 (ci-8), rustfs/backlog#1155

.github/workflows/e2e-s3tests.yml

@@ -109,6 +109,11 @@ concurrency:
   group: ${{ github.workflow }}-${{ github.ref }}-${{ github.event.inputs['test-mode'] || 'single' }}
   cancel-in-progress: true
 
+# Only alert-on-failure needs more than read access; it declares its own
+# job-level `issues: write`.
+permissions:
+  contents: read
+
 defaults:
   run:
     shell: bash
@@ -325,3 +330,22 @@ jobs:
         with:
           name: s3tests-${{ env.TEST_MODE }}
           path: artifacts/**
+
+  alert-on-failure:
+    name: Alert on scheduled failure
+    needs: [s3tests]
+    # `always()` is required: without it this job is skipped when a needed
+    # job fails. Alerts only for scheduled runs (backlog#1149 ci-8) — manual
+    # dispatch failures are already watched by a human.
+    if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+    permissions:
+      contents: read
+      issues: write
+    steps:
+      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
+      - name: Open or update failure-tracking issue
+        uses: ./.github/actions/schedule-failure-issue
+        with:
+          github-token: ${{ secrets.GITHUB_TOKEN }}

.github/workflows/fuzz.yml

@@ -229,3 +229,25 @@ jobs:
             fuzz/corpus/${{ matrix.target }}/**
           if-no-files-found: ignore
           retention-days: 30
+
+  # ──────────────────────────────────────────────────────────────
+  # Nightly alerting: open/update a tracking issue on failure.
+  # ──────────────────────────────────────────────────────────────
+  alert-on-failure:
+    name: Alert on scheduled failure
+    needs: [fuzz-build, nightly-fuzz-corpus]
+    # `always()` is required: without it this job is skipped when a needed
+    # job fails. Alerts only for scheduled (nightly) runs (backlog#1149
+    # ci-8); PR and manual dispatch failures are already watched by a human.
+    if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+    permissions:
+      contents: read
+      issues: write
+    steps:
+      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
+      - name: Open or update failure-tracking issue
+        uses: ./.github/actions/schedule-failure-issue
+        with:
+          github-token: ${{ secrets.GITHUB_TOKEN }}

.github/workflows/mint.yml

@@ -98,6 +98,11 @@ concurrency:
   group: ${{ github.workflow }}-${{ github.ref }}
   cancel-in-progress: true
 
+# Only alert-on-failure needs more than read access; it declares its own
+# job-level `issues: write`.
+permissions:
+  contents: read
+
 defaults:
   run:
     shell: bash
@@ -238,3 +243,22 @@ jobs:
         with:
           name: mint
           path: artifacts/**
+
+  alert-on-failure:
+    name: Alert on scheduled failure
+    needs: [mint]
+    # `always()` is required: without it this job is skipped when a needed
+    # job fails. Alerts only for scheduled runs (backlog#1149 ci-8) — manual
+    # dispatch failures are already watched by a human.
+    if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+    permissions:
+      contents: read
+      issues: write
+    steps:
+      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
+      - name: Open or update failure-tracking issue
+        uses: ./.github/actions/schedule-failure-issue
+        with:
+          github-token: ${{ secrets.GITHUB_TOKEN }}

.github/workflows/performance-ab.yml

@@ -217,3 +217,22 @@ jobs:
             exit "$status"
           fi
           echo "warp A/B budget gate passed."
+
+  alert-on-failure:
+    name: Alert on scheduled failure
+    needs: [warp-ab]
+    # `always()` is required: without it this job is skipped when a needed
+    # job fails. Alerts only for scheduled (nightly) runs (backlog#1149
+    # ci-8); PR and manual dispatch failures are already watched by a human.
+    if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+    permissions:
+      contents: read
+      issues: write
+    steps:
+      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
+      - name: Open or update failure-tracking issue
+        uses: ./.github/actions/schedule-failure-issue
+        with:
+          github-token: ${{ secrets.GITHUB_TOKEN }}

.github/workflows/schedule-failure-alert-drill.yml

@@ -0,0 +1,62 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Manual drill for the scheduled-failure alerting path (backlog#1149 ci-8).
+#
+# Forces a job failure and then runs .github/actions/schedule-failure-issue
+# through the exact `needs` + `always() && contains(needs.*.result,
+# 'failure')` wiring used by the real consumers (e2e-s3tests, mint, fuzz,
+# performance-ab). Dispatch it twice to verify both the issue-creation and
+# the dedupe-comment paths, then close the resulting
+# "[scheduled-failure] Schedule Failure Alert Drill" issue.
+#
+# The run itself is expected to end red (the forced failure); only the
+# alert-on-failure job result matters.
+
+name: Schedule Failure Alert Drill
+
+on:
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+jobs:
+  forced-failure:
+    name: Forced failure
+    runs-on: ubuntu-latest
+    timeout-minutes: 5
+    steps:
+      - name: Fail on purpose
+        run: |
+          echo "Deliberate failure so alert-on-failure exercises the real consumer wiring."
+          exit 1
+
+  alert-on-failure:
+    name: Alert on scheduled failure
+    needs: [forced-failure]
+    # Mirrors the consumer wiring, minus the schedule-event guard (this
+    # workflow is dispatch-only by design).
+    if: always() && contains(needs.*.result, 'failure')
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+    permissions:
+      contents: read
+      issues: write
+    steps:
+      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
+      - name: Open or update failure-tracking issue
+        uses: ./.github/actions/schedule-failure-issue
+        with:
+          github-token: ${{ secrets.GITHUB_TOKEN }}