mirror of
https://github.com/rustfs/rustfs.git
synced 2026-07-27 16:48:58 +00:00
ci: add schedule-failure-issue composite action and wire nightly alerts (#4671)
Scheduled workflow failures previously went unnoticed (15 consecutive red weekly s3-tests sweeps, silent perf nightly failures). Add a reusable composite action that opens a tracking issue titled "[scheduled-failure] <workflow name>" — or appends a comment to the existing open one (dedupe key = workflow name) — with the run URL, run attempt, and failed job names, labeled "infrastructure". Wire it into the scheduled paths of e2e-s3tests, mint, fuzz (nightly) and performance-ab via a final alert-on-failure job gated by "always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')", with issues:write scoped to that job only. e2e-s3tests and mint previously had no permissions key; they now get workflow-level contents:read, reducing every other job's token. Add a dispatch-only drill workflow that forces a job failure and runs the action through the exact consumer wiring, for end-to-end verification. The ci-7 e2e nightly does not exist yet; it is recorded as a pending consumer in the action README. Refs: rustfs/backlog#1149 (ci-8), rustfs/backlog#1155
This commit is contained in:
Zhengchao An
@@ -0,0 +1,74 @@
|
||||
# schedule-failure-issue
|
||||
|
||||
Opens (or updates) a GitHub issue when a **scheduled** workflow run fails.
|
||||
This is the single alerting mechanism for all timed pipelines
|
||||
(rustfs/backlog#1149 ci-8, arbitration G3): scheduled workflows must consume
|
||||
this action instead of inventing their own notification paths.
|
||||
|
||||
## Behavior
|
||||
|
||||
- Issue title: `[scheduled-failure] <workflow name>` — the workflow name is
|
||||
the dedupe key.
|
||||
- If an **open** issue with that exact title exists, the failure is appended
|
||||
as a comment; otherwise a new issue is created (labeled `infrastructure` by
|
||||
default). Closing the issue resets the cycle: the next failure opens a
|
||||
fresh one.
|
||||
- The issue body / comment includes the run URL, run attempt, event, ref and
|
||||
the names of the failed (or timed-out/cancelled) jobs of the current run
|
||||
attempt.
|
||||
- Requires `gh` and `jq` on the runner (both preinstalled on GitHub-hosted
|
||||
runners such as `ubuntu-latest`).
|
||||
|
||||
## Usage
|
||||
|
||||
Add a final job to the workflow. `always()` is required — without it the job
|
||||
is skipped when a needed job fails; `contains(needs.*.result, 'failure')`
|
||||
makes it act only when something actually failed. Scope `issues: write` to
|
||||
this job only; no other job may gain permissions.
|
||||
|
||||
```yaml
|
||||
alert-on-failure:
|
||||
name: Alert on scheduled failure
|
||||
needs: [<the jobs to watch>]
|
||||
if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 10
|
||||
permissions:
|
||||
contents: read
|
||||
issues: write
|
||||
steps:
|
||||
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
|
||||
- name: Open or update failure-tracking issue
|
||||
uses: ./.github/actions/schedule-failure-issue
|
||||
with:
|
||||
github-token: ${{ secrets.GITHUB_TOKEN }}
|
||||
```
|
||||
|
||||
Notes:
|
||||
|
||||
- Upstream jobs that are *cancelled* (e.g. a manually cancelled run) do not
|
||||
trigger the alert — a human was already looking. Job `timeout-minutes`
|
||||
expiry surfaces as `failure` and does alert.
|
||||
- Manual `workflow_dispatch` runs never alert; a human is watching those.
|
||||
|
||||
## Current consumers
|
||||
|
||||
- `.github/workflows/e2e-s3tests.yml` (weekly full s3-tests sweep)
|
||||
- `.github/workflows/mint.yml` (weekly multi-SDK mint run)
|
||||
- `.github/workflows/fuzz.yml` (nightly fuzz corpus jobs)
|
||||
- `.github/workflows/performance-ab.yml` (nightly warp A/B gate)
|
||||
- **Pending:** the ci-7 e2e nightly-full workflow does not exist yet
|
||||
(backlog#1149 ci-7). When it lands, wire it up with the snippet above.
|
||||
|
||||
## Drill
|
||||
|
||||
`.github/workflows/schedule-failure-alert-drill.yml` is a manual-only
|
||||
workflow that forces a job failure and runs this action through the exact
|
||||
consumer wiring. Use it to verify the alert path end to end:
|
||||
|
||||
```bash
|
||||
gh workflow run schedule-failure-alert-drill.yml -R rustfs/rustfs --ref <branch>
|
||||
```
|
||||
|
||||
Run it twice to verify both paths (issue creation, then comment dedupe), and
|
||||
close the `[scheduled-failure] Schedule Failure Alert Drill` issue afterwards.
|
||||
@@ -0,0 +1,104 @@
|
||||
# Copyright 2024 RustFS Team
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
name: "Schedule Failure Issue"
|
||||
description: >-
|
||||
Open (or update) a tracking issue when a scheduled workflow run fails.
|
||||
Dedupes by workflow name: if an open issue titled
|
||||
"[scheduled-failure] <workflow name>" already exists, the failure is
|
||||
appended as a comment; otherwise a new issue is created. This is the
|
||||
single alerting mechanism for all scheduled pipelines (backlog#1149 ci-8).
|
||||
|
||||
inputs:
|
||||
github-token:
|
||||
description: >-
|
||||
Token with issues:write on the repository. Pass secrets.GITHUB_TOKEN
|
||||
from a job that declares `permissions: issues: write` (scope the
|
||||
permission to the alert job only, never workflow-wide).
|
||||
required: true
|
||||
workflow-name:
|
||||
description: "Workflow name used for the issue title (the dedupe key)."
|
||||
required: false
|
||||
default: ${{ github.workflow }}
|
||||
label:
|
||||
description: >-
|
||||
Label applied when a new issue is created. Must already exist in the
|
||||
repository; if applying it fails, the issue is created without a label.
|
||||
Set to an empty string to skip labeling.
|
||||
required: false
|
||||
default: "infrastructure"
|
||||
|
||||
runs:
|
||||
using: "composite"
|
||||
steps:
|
||||
- name: Open or update failure-tracking issue
|
||||
shell: bash
|
||||
env:
|
||||
GH_TOKEN: ${{ inputs.github-token }}
|
||||
WORKFLOW_NAME: ${{ inputs.workflow-name }}
|
||||
ISSUE_LABEL: ${{ inputs.label }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
|
||||
title="[scheduled-failure] ${WORKFLOW_NAME}"
|
||||
run_url="${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}"
|
||||
|
||||
# Failed job names for this run attempt. The alert job runs while the
|
||||
# run as a whole is still in progress, so inspect the jobs that have
|
||||
# already completed with a non-success conclusion.
|
||||
failed_jobs="$(gh api \
|
||||
"repos/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}/attempts/${GITHUB_RUN_ATTEMPT}/jobs" \
|
||||
--paginate \
|
||||
--jq '.jobs[]
|
||||
| select(.conclusion == "failure" or .conclusion == "timed_out" or .conclusion == "cancelled")
|
||||
| "- `\(.name)` (\(.conclusion))"')"
|
||||
if [ -z "${failed_jobs}" ]; then
|
||||
failed_jobs="- (failed job not recorded yet — see the run page)"
|
||||
fi
|
||||
|
||||
body="$(cat <<EOF
|
||||
Scheduled run of **${WORKFLOW_NAME}** failed.
|
||||
|
||||
- Run: ${run_url} (attempt ${GITHUB_RUN_ATTEMPT})
|
||||
- Event: \`${GITHUB_EVENT_NAME}\`
|
||||
- Ref: \`${GITHUB_REF_NAME}\` @ \`${GITHUB_SHA}\`
|
||||
|
||||
Failed jobs:
|
||||
${failed_jobs}
|
||||
EOF
|
||||
)"
|
||||
|
||||
# Dedupe by exact title among OPEN issues (a closed issue means the
|
||||
# earlier breakage was resolved; a new failure gets a fresh issue).
|
||||
# GitHub search strips the [] characters, so search loosely and match
|
||||
# the exact title with jq.
|
||||
existing="$(gh issue list --repo "${GITHUB_REPOSITORY}" --state open --limit 100 \
|
||||
--search "\"scheduled-failure\" in:title" --json number,title \
|
||||
| jq -r --arg title "${title}" 'map(select(.title == $title)) | (.[0].number // empty)')"
|
||||
|
||||
if [ -n "${existing}" ]; then
|
||||
echo "Appending comment to existing open issue #${existing}"
|
||||
gh issue comment "${existing}" --repo "${GITHUB_REPOSITORY}" --body "${body}"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
echo "Creating new issue: ${title}"
|
||||
if [ -n "${ISSUE_LABEL}" ]; then
|
||||
if gh issue create --repo "${GITHUB_REPOSITORY}" \
|
||||
--title "${title}" --body "${body}" --label "${ISSUE_LABEL}"; then
|
||||
exit 0
|
||||
fi
|
||||
echo "::warning::issue creation with label '${ISSUE_LABEL}' failed (missing label?); retrying without a label"
|
||||
fi
|
||||
gh issue create --repo "${GITHUB_REPOSITORY}" --title "${title}" --body "${body}"
|
||||
@@ -109,6 +109,11 @@ concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.ref }}-${{ github.event.inputs['test-mode'] || 'single' }}
|
||||
cancel-in-progress: true
|
||||
|
||||
# Only alert-on-failure needs more than read access; it declares its own
|
||||
# job-level `issues: write`.
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
defaults:
|
||||
run:
|
||||
shell: bash
|
||||
@@ -325,3 +330,22 @@ jobs:
|
||||
with:
|
||||
name: s3tests-${{ env.TEST_MODE }}
|
||||
path: artifacts/**
|
||||
|
||||
alert-on-failure:
|
||||
name: Alert on scheduled failure
|
||||
needs: [s3tests]
|
||||
# `always()` is required: without it this job is skipped when a needed
|
||||
# job fails. Alerts only for scheduled runs (backlog#1149 ci-8) — manual
|
||||
# dispatch failures are already watched by a human.
|
||||
if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 10
|
||||
permissions:
|
||||
contents: read
|
||||
issues: write
|
||||
steps:
|
||||
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
|
||||
- name: Open or update failure-tracking issue
|
||||
uses: ./.github/actions/schedule-failure-issue
|
||||
with:
|
||||
github-token: ${{ secrets.GITHUB_TOKEN }}
|
||||
|
||||
@@ -229,3 +229,25 @@ jobs:
|
||||
fuzz/corpus/${{ matrix.target }}/**
|
||||
if-no-files-found: ignore
|
||||
retention-days: 30
|
||||
|
||||
# ──────────────────────────────────────────────────────────────
|
||||
# Nightly alerting: open/update a tracking issue on failure.
|
||||
# ──────────────────────────────────────────────────────────────
|
||||
alert-on-failure:
|
||||
name: Alert on scheduled failure
|
||||
needs: [fuzz-build, nightly-fuzz-corpus]
|
||||
# `always()` is required: without it this job is skipped when a needed
|
||||
# job fails. Alerts only for scheduled (nightly) runs (backlog#1149
|
||||
# ci-8); PR and manual dispatch failures are already watched by a human.
|
||||
if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 10
|
||||
permissions:
|
||||
contents: read
|
||||
issues: write
|
||||
steps:
|
||||
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
|
||||
- name: Open or update failure-tracking issue
|
||||
uses: ./.github/actions/schedule-failure-issue
|
||||
with:
|
||||
github-token: ${{ secrets.GITHUB_TOKEN }}
|
||||
|
||||
@@ -98,6 +98,11 @@ concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
# Only alert-on-failure needs more than read access; it declares its own
|
||||
# job-level `issues: write`.
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
defaults:
|
||||
run:
|
||||
shell: bash
|
||||
@@ -238,3 +243,22 @@ jobs:
|
||||
with:
|
||||
name: mint
|
||||
path: artifacts/**
|
||||
|
||||
alert-on-failure:
|
||||
name: Alert on scheduled failure
|
||||
needs: [mint]
|
||||
# `always()` is required: without it this job is skipped when a needed
|
||||
# job fails. Alerts only for scheduled runs (backlog#1149 ci-8) — manual
|
||||
# dispatch failures are already watched by a human.
|
||||
if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 10
|
||||
permissions:
|
||||
contents: read
|
||||
issues: write
|
||||
steps:
|
||||
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
|
||||
- name: Open or update failure-tracking issue
|
||||
uses: ./.github/actions/schedule-failure-issue
|
||||
with:
|
||||
github-token: ${{ secrets.GITHUB_TOKEN }}
|
||||
|
||||
@@ -217,3 +217,22 @@ jobs:
|
||||
exit "$status"
|
||||
fi
|
||||
echo "warp A/B budget gate passed."
|
||||
|
||||
alert-on-failure:
|
||||
name: Alert on scheduled failure
|
||||
needs: [warp-ab]
|
||||
# `always()` is required: without it this job is skipped when a needed
|
||||
# job fails. Alerts only for scheduled (nightly) runs (backlog#1149
|
||||
# ci-8); PR and manual dispatch failures are already watched by a human.
|
||||
if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 10
|
||||
permissions:
|
||||
contents: read
|
||||
issues: write
|
||||
steps:
|
||||
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
|
||||
- name: Open or update failure-tracking issue
|
||||
uses: ./.github/actions/schedule-failure-issue
|
||||
with:
|
||||
github-token: ${{ secrets.GITHUB_TOKEN }}
|
||||
|
||||
@@ -0,0 +1,62 @@
|
||||
# Copyright 2024 RustFS Team
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
# Manual drill for the scheduled-failure alerting path (backlog#1149 ci-8).
|
||||
#
|
||||
# Forces a job failure and then runs .github/actions/schedule-failure-issue
|
||||
# through the exact `needs` + `always() && contains(needs.*.result,
|
||||
# 'failure')` wiring used by the real consumers (e2e-s3tests, mint, fuzz,
|
||||
# performance-ab). Dispatch it twice to verify both the issue-creation and
|
||||
# the dedupe-comment paths, then close the resulting
|
||||
# "[scheduled-failure] Schedule Failure Alert Drill" issue.
|
||||
#
|
||||
# The run itself is expected to end red (the forced failure); only the
|
||||
# alert-on-failure job result matters.
|
||||
|
||||
name: Schedule Failure Alert Drill
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
forced-failure:
|
||||
name: Forced failure
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 5
|
||||
steps:
|
||||
- name: Fail on purpose
|
||||
run: |
|
||||
echo "Deliberate failure so alert-on-failure exercises the real consumer wiring."
|
||||
exit 1
|
||||
|
||||
alert-on-failure:
|
||||
name: Alert on scheduled failure
|
||||
needs: [forced-failure]
|
||||
# Mirrors the consumer wiring, minus the schedule-event guard (this
|
||||
# workflow is dispatch-only by design).
|
||||
if: always() && contains(needs.*.result, 'failure')
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 10
|
||||
permissions:
|
||||
contents: read
|
||||
issues: write
|
||||
steps:
|
||||
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
|
||||
- name: Open or update failure-tracking issue
|
||||
uses: ./.github/actions/schedule-failure-issue
|
||||
with:
|
||||
github-token: ${{ secrets.GITHUB_TOKEN }}
|
||||
Reference in New Issue
Block a user