gsadmin/rustfs
1
0
Fork 0
mirror of https://github.com/rustfs/rustfs.git synced 2026-09-06 03:59:14 +00:00
Code Issues Packages Projects Releases Wiki Activity

ci(functional): retry chain handoffs and alert on stall (#7023)

Browse Source 
The repository_dispatch handoff step was continue-on-error with a single
attempt: if the call failed (token lacking contents:write, transient API
error), the chain stalled silently while every job stayed green.

Each handoff now retries 3x and, if all attempts fail, files an alert
issue in rustfs/backlog with the exact recovery command before exiting 1
(still continue-on-error, so suite workflows themselves never fail).
This commit is contained in:
hector
2026-09-01 23:11:25 +08:00
committed by GitHub
parent cebe57a2f0
commit bba934723a
7 changed files with 266 additions and 49 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.github/workflows/rustfs-kms-test.yml
+38 -7
View File
@@ -340,21 +340,52 @@ jobs:
- name: "Continue functional chain (next: Tier)"
# Only chain-triggered runs forward to the next suite; standalone
# workflow_dispatch runs stop after their own cleanup.
# workflow_dispatch runs stop after their own cleanup. A failed
# handoff must never pass silently: it retries, then files an alert
# issue in rustfs/backlog so a stalled chain is visible.
if: ${{ always() && github.event_name == 'repository_dispatch' }}
continue-on-error: true
env:
GH_TOKEN: ${{ secrets.PF_TESTING_GH_TOKEN }}
run: |
set -euo pipefail
if [ -z "${GH_TOKEN:-}" ]; then
set -uo pipefail
if [ -z "${{GH_TOKEN:-}}" ]; then
echo "PF_TESTING_GH_TOKEN is not configured; cannot dispatch the next suite" >&2
exit 1
fi
echo "Dispatching next functional suite: Tier"
gh api --method POST repos/rustfs/rustfs/dispatches \
-f event_type='rustfs-chain-tier' \
-F 'client_payload[from_suite]=kms'
DISPATCHED=0
for attempt in 1 2 3; do
if gh api --method POST repos/rustfs/rustfs/dispatches \
-f event_type='rustfs-chain-tier' \
-F 'client_payload[from_suite]=kms'; then
echo "dispatched next suite Tier (attempt ${{attempt}})"
DISPATCHED=1
break
fi
echo "dispatch attempt ${{attempt}} failed; retrying in ${{attempt}}0s" >&2
sleep "${{attempt}}0"
done
if [ "${{DISPATCHED:-0}}" -ne 1 ]; then
echo "ERROR: functional chain stalled: could not dispatch Tier after 3 attempts" >&2
TITLE="[functional][chain] stalled after kms (run ${{GITHUB_RUN_ID}})"
BODY_FILE="$(mktemp)"
{
echo "The functional chain could not hand off from **kms** to **Tier** after 3 attempts."
echo ""
echo "- Failed suite job: ${{GITHUB_SERVER_URL}}/${{GITHUB_REPOSITORY}}/actions/runs/${{GITHUB_RUN_ID}}"
echo "- Expected next event: `rustfs-chain-tier`"
echo "- Likely cause: PF_TESTING_GH_TOKEN lacks contents:write on rustfs/rustfs, or the GitHub API was unavailable."
echo "- Recovery: re-dispatch manually with"
echo " ```"
echo " gh api --method POST repos/rustfs/rustfs/dispatches -f event_type='rustfs-chain-tier'"
echo " ```"
} > "${{BODY_FILE}}"
gh issue create -R rustfs/backlog --title "${{TITLE}}" \
--body-file "${{BODY_FILE}}" --label functional-test \
|| gh issue create -R rustfs/backlog --title "${{TITLE}}" --body-file "${{BODY_FILE}}" \
|| echo "could not file the stall alert issue either; check the token" >&2
exit 1
fi
- name: Notify on failure
if: failure()