From 4310b55238c96bef47b4be99ee2a248cf1ad6709 Mon Sep 17 00:00:00 2001 From: Zhengchao An Date: Fri, 10 Jul 2026 20:53:44 +0800 Subject: [PATCH] ci: add schedule-failure-issue composite action and wire nightly alerts (#4671) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Scheduled workflow failures previously went unnoticed (15 consecutive red weekly s3-tests sweeps, silent perf nightly failures). Add a reusable composite action that opens a tracking issue titled "[scheduled-failure] " — or appends a comment to the existing open one (dedupe key = workflow name) — with the run URL, run attempt, and failed job names, labeled "infrastructure". Wire it into the scheduled paths of e2e-s3tests, mint, fuzz (nightly) and performance-ab via a final alert-on-failure job gated by "always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')", with issues:write scoped to that job only. e2e-s3tests and mint previously had no permissions key; they now get workflow-level contents:read, reducing every other job's token. Add a dispatch-only drill workflow that forces a job failure and runs the action through the exact consumer wiring, for end-to-end verification. The ci-7 e2e nightly does not exist yet; it is recorded as a pending consumer in the action README. Refs: rustfs/backlog#1149 (ci-8), rustfs/backlog#1155 --- .../actions/schedule-failure-issue/README.md | 74 +++++++++++++ .../actions/schedule-failure-issue/action.yml | 104 ++++++++++++++++++ .github/workflows/e2e-s3tests.yml | 24 ++++ .github/workflows/fuzz.yml | 22 ++++ .github/workflows/mint.yml | 24 ++++ .github/workflows/performance-ab.yml | 19 ++++ .../schedule-failure-alert-drill.yml | 62 +++++++++++ 7 files changed, 329 insertions(+) create mode 100644 .github/actions/schedule-failure-issue/README.md create mode 100644 .github/actions/schedule-failure-issue/action.yml create mode 100644 .github/workflows/schedule-failure-alert-drill.yml diff --git a/.github/actions/schedule-failure-issue/README.md b/.github/actions/schedule-failure-issue/README.md new file mode 100644 index 000000000..d2e780f69 --- /dev/null +++ b/.github/actions/schedule-failure-issue/README.md @@ -0,0 +1,74 @@ +# schedule-failure-issue + +Opens (or updates) a GitHub issue when a **scheduled** workflow run fails. +This is the single alerting mechanism for all timed pipelines +(rustfs/backlog#1149 ci-8, arbitration G3): scheduled workflows must consume +this action instead of inventing their own notification paths. + +## Behavior + +- Issue title: `[scheduled-failure] ` — the workflow name is + the dedupe key. +- If an **open** issue with that exact title exists, the failure is appended + as a comment; otherwise a new issue is created (labeled `infrastructure` by + default). Closing the issue resets the cycle: the next failure opens a + fresh one. +- The issue body / comment includes the run URL, run attempt, event, ref and + the names of the failed (or timed-out/cancelled) jobs of the current run + attempt. +- Requires `gh` and `jq` on the runner (both preinstalled on GitHub-hosted + runners such as `ubuntu-latest`). + +## Usage + +Add a final job to the workflow. `always()` is required — without it the job +is skipped when a needed job fails; `contains(needs.*.result, 'failure')` +makes it act only when something actually failed. Scope `issues: write` to +this job only; no other job may gain permissions. + +```yaml + alert-on-failure: + name: Alert on scheduled failure + needs: [] + if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure') + runs-on: ubuntu-latest + timeout-minutes: 10 + permissions: + contents: read + issues: write + steps: + - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7 + - name: Open or update failure-tracking issue + uses: ./.github/actions/schedule-failure-issue + with: + github-token: ${{ secrets.GITHUB_TOKEN }} +``` + +Notes: + +- Upstream jobs that are *cancelled* (e.g. a manually cancelled run) do not + trigger the alert — a human was already looking. Job `timeout-minutes` + expiry surfaces as `failure` and does alert. +- Manual `workflow_dispatch` runs never alert; a human is watching those. + +## Current consumers + +- `.github/workflows/e2e-s3tests.yml` (weekly full s3-tests sweep) +- `.github/workflows/mint.yml` (weekly multi-SDK mint run) +- `.github/workflows/fuzz.yml` (nightly fuzz corpus jobs) +- `.github/workflows/performance-ab.yml` (nightly warp A/B gate) +- **Pending:** the ci-7 e2e nightly-full workflow does not exist yet + (backlog#1149 ci-7). When it lands, wire it up with the snippet above. + +## Drill + +`.github/workflows/schedule-failure-alert-drill.yml` is a manual-only +workflow that forces a job failure and runs this action through the exact +consumer wiring. Use it to verify the alert path end to end: + +```bash +gh workflow run schedule-failure-alert-drill.yml -R rustfs/rustfs --ref +``` + +Run it twice to verify both paths (issue creation, then comment dedupe), and +close the `[scheduled-failure] Schedule Failure Alert Drill` issue afterwards. diff --git a/.github/actions/schedule-failure-issue/action.yml b/.github/actions/schedule-failure-issue/action.yml new file mode 100644 index 000000000..60e938690 --- /dev/null +++ b/.github/actions/schedule-failure-issue/action.yml @@ -0,0 +1,104 @@ +# Copyright 2024 RustFS Team +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +name: "Schedule Failure Issue" +description: >- + Open (or update) a tracking issue when a scheduled workflow run fails. + Dedupes by workflow name: if an open issue titled + "[scheduled-failure] " already exists, the failure is + appended as a comment; otherwise a new issue is created. This is the + single alerting mechanism for all scheduled pipelines (backlog#1149 ci-8). + +inputs: + github-token: + description: >- + Token with issues:write on the repository. Pass secrets.GITHUB_TOKEN + from a job that declares `permissions: issues: write` (scope the + permission to the alert job only, never workflow-wide). + required: true + workflow-name: + description: "Workflow name used for the issue title (the dedupe key)." + required: false + default: ${{ github.workflow }} + label: + description: >- + Label applied when a new issue is created. Must already exist in the + repository; if applying it fails, the issue is created without a label. + Set to an empty string to skip labeling. + required: false + default: "infrastructure" + +runs: + using: "composite" + steps: + - name: Open or update failure-tracking issue + shell: bash + env: + GH_TOKEN: ${{ inputs.github-token }} + WORKFLOW_NAME: ${{ inputs.workflow-name }} + ISSUE_LABEL: ${{ inputs.label }} + run: | + set -euo pipefail + + title="[scheduled-failure] ${WORKFLOW_NAME}" + run_url="${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}" + + # Failed job names for this run attempt. The alert job runs while the + # run as a whole is still in progress, so inspect the jobs that have + # already completed with a non-success conclusion. + failed_jobs="$(gh api \ + "repos/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}/attempts/${GITHUB_RUN_ATTEMPT}/jobs" \ + --paginate \ + --jq '.jobs[] + | select(.conclusion == "failure" or .conclusion == "timed_out" or .conclusion == "cancelled") + | "- `\(.name)` (\(.conclusion))"')" + if [ -z "${failed_jobs}" ]; then + failed_jobs="- (failed job not recorded yet — see the run page)" + fi + + body="$(cat <