mirror of
https://github.com/rustfs/rustfs.git
synced 2026-10-31 08:16:35 +00:00
Chris
b06923beaf
* test(e2e): add scanner/heal topology protection matrix Add the S00 protection baseline for the Scanner/Heal program (rustfs/backlog#2873, program rustfs/backlog#2872): seven scanner_heal_topology_ cases in the distributed e2e lane that pin pool expansion, rebalance and decommission behavior before the runtime changes. The fixture seeds a deterministic inventory (empty, inline, external, multipart, special and nested keys; a versioned bucket with an explicit null version, three history versions and a delete marker per key; an Object Lock bucket with legal hold and COMPLIANCE retention), records every version with its requested and listed version id, length and SHA-256, verifies the manifest through every node, reads physical placement from xl.meta on the pool drives, and reads the strict durability mode back from the admin API and the node logs. Fault cases hold the mover at proven windows through file barriers that compile only with the e2e-test-hooks feature: before the multipart target commit and before source cleanup (shared by decommission and rebalance), plus the existing pool-metadata before_publish cutpoint. An unreached window fails the case. DistCluster gains capture_logs_under so a case can read node startup events without a global environment variable. docs/testing/scanner-heal-topology-gates.md maps the gates to the tests and records the timeout basis. * test(e2e): keep the barrier failure message static The error(format!) ratchet rejects new formatted io::Error::other call sites in ecstore. The reached observation already records the stage and object, so the injected failure needs no per-call detail. * test(e2e): require e2e-test-hooks for topology fault cases The four scanner_heal_topology_ fault cases inject at server barriers that exist only with the e2e-test-hooks feature. Against a default server they waited for the barrier timeout and reported it as not reached. Check the verified binary's features first so the missing prerequisite fails immediately and is named. * ci(e2e): build the distributed lane with e2e-test-hooks Every other e2e lane builds and runs the fault-test hooks server. The distributed lane alone used the default server, so the topology fault cases could never reach their barriers there. Build and run with the same feature so the receipt matches, and update the documented local commands. The upgrade cases spawn the pinned previous release directly and do not compare feature sets. * test(e2e): keep topology artifacts under RUSTFS_E2E_LOG_DIR The topology fixture captured node logs, barrier observations and evidence in a temp dir removed on drop, so the distributed workflow uploaded nothing for these cases. Fall back to RUSTFS_E2E_LOG_DIR, which the workflow uploads, when no explicit evidence directory is set. * test(e2e): remove retired pools before restart; race the pre-ACK start The queue case restarted with decommissioned pools 0 and 1 still on the command line, which the server rightly refuses. Add remove_single_node_pools and restart without them, then expect the remaining pools renumbered to 0 and 1. The pre-ACK case now races the start request against the barrier. A start that returns first, or stays in flight past the documented 60 s budget, is reported as such instead of as an unreached barrier, and the evidence records the reach latency. capture_logs_under now overrides a log path assigned from RUSTFS_E2E_LOG_DIR, so the seed node logs into the case directory too. * test(e2e): bound fault-window waits by the movement budget On Linux CI a busy target capacity gate delayed the first fault object by about two minutes, past the 120 s wait borrowed from the server's release poll. Wait for a movement window within the 240 s movement budget instead, and fail at once when the decommission or rebalance ends without entering the window, so an uninjected fault is never counted. * test(e2e): bound status probes while waiting for a fault window The admin decommission status request can wait on a mover held at a test barrier until the barrier's own 120 s release timeout. The window wait issued that request each loop, so a reached barrier went unnoticed until the server timed it out and failed the decommission, which surfaced as a false resume failure. Bound each status probe to 5 s and treat a slow probe as still running. * test(e2e): retry a busy decommission start A decommission start that meets a busy movement gate can answer a retryable 503 instead of waiting. The harness treated any non-proof failure as fatal. Share one start helper that retries 503 within the case's movement budget and the fleet-proof 500 for 45 s, and keeps every other status fatal. The pre-ACK case retries 503 within its window budget.
220 lines
8.7 KiB
YAML
220 lines
8.7 KiB
YAML
# Copyright 2024 RustFS Team
|
|
#
|
|
# Licensed under the Apache License, Version 2.0 (the "License");
|
|
# you may not use this file except in compliance with the License.
|
|
# You may obtain a copy of the License at
|
|
#
|
|
# http://www.apache.org/LICENSE-2.0
|
|
#
|
|
# Unless required by applicable law or agreed to in writing, software
|
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
# See the License for the specific language governing permissions and
|
|
# limitations under the License.
|
|
|
|
# 4-node 4-disk distributed e2e lane.
|
|
#
|
|
# Each selected test starts a real localhost cluster via
|
|
# `RustFSTestClusterEnvironment` (4 processes; 4 drives per node unless the
|
|
# case is a two-site 4-node 1-drive pair or a 4-node upgrade). Membership is
|
|
# `[profile.e2e-distributed]` in `.config/nextest.toml`. Storage-sensitive PRs,
|
|
# nightly runs, and manual dispatches all execute the same fail-closed suite.
|
|
# Upgrade cases download the same pinned previous release as e2e-upgrade.yml.
|
|
#
|
|
# Isolated pool filesystems: expand/decommission/rebalance cases require
|
|
# independent `statfs` capacity. This job runs on GitHub-hosted
|
|
# `ubuntu-latest` because the self-hosted `sm-standard-4` ARC pods cannot
|
|
# create filesystems: `mount -o loop` fails with ENOENT (no
|
|
# `/dev/loop-control`), and `mount -t tmpfs` fails with "cannot mount tmpfs
|
|
# read-only" (no `CAP_SYS_ADMIN` in the initial namespace). The same reason
|
|
# `uring-integration` and `e2e-s3tests.yml` left that label. The prepare
|
|
# step mounts four 1 GiB tmpfs instances and exports `RUSTFS_E2E_POOL_ROOTS`.
|
|
|
|
name: e2e-distributed
|
|
|
|
on:
|
|
pull_request:
|
|
paths:
|
|
- "Cargo.lock"
|
|
- "Cargo.toml"
|
|
- ".config/nextest.toml"
|
|
- ".config/e2e-distributed-selection.txt"
|
|
- ".github/workflows/e2e-distributed.yml"
|
|
- "crates/audit/**"
|
|
- "crates/common/**"
|
|
- "crates/config/**"
|
|
- "crates/e2e_test/**"
|
|
- "crates/ecstore/**"
|
|
- "crates/filemeta/**"
|
|
- "crates/heal/**"
|
|
- "crates/iam/**"
|
|
- "crates/lock/**"
|
|
- "crates/madmin/**"
|
|
- "crates/notify/**"
|
|
- "crates/replication/**"
|
|
- "crates/s3-client/**"
|
|
- "crates/s3-ops/**"
|
|
- "crates/s3-types/**"
|
|
- "crates/scanner/**"
|
|
- "crates/storage-api/**"
|
|
- "crates/utils/**"
|
|
- "rustfs/**"
|
|
workflow_dispatch:
|
|
inputs:
|
|
filter:
|
|
description: "Optional nextest -E filter (default: the whole e2e-distributed profile)"
|
|
required: false
|
|
default: ""
|
|
schedule:
|
|
# 05:53 UTC nightly — clear of e2e-nightly (04:29) and ODM interop (05:23).
|
|
- cron: "53 5 * * *"
|
|
|
|
permissions:
|
|
contents: read
|
|
|
|
concurrency:
|
|
group: ${{ github.workflow }}-${{ github.ref }}
|
|
cancel-in-progress: ${{ github.event_name != 'schedule' }}
|
|
|
|
jobs:
|
|
distributed:
|
|
if: github.repository == 'rustfs/rustfs'
|
|
name: Distributed 4-node 4-disk e2e
|
|
# GitHub-hosted VM: loop and tmpfs mounts work here. sm-standard-4 is an
|
|
# ARC pod and rejects both (`mount -o loop` ENOENT, tmpfs "read-only").
|
|
runs-on: ubuntu-latest
|
|
timeout-minutes: 180
|
|
env:
|
|
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
|
|
NO_PROXY: 127.0.0.1,localhost
|
|
HTTP_PROXY: ""
|
|
HTTPS_PROXY: ""
|
|
# Pinned previous release used by distributed::upgrade_test (same pin as e2e-upgrade.yml).
|
|
UPGRADE_SOURCE_VERSION: 1.0.0-rc.2
|
|
UPGRADE_SOURCE_ASSET: rustfs-linux-x86_64-gnu-v1.0.0-rc.2.zip
|
|
UPGRADE_SOURCE_SHA256: 7c789386bf85278f865b8e0d359bf4edb84d5aa408cc3fa54a18c25ca74cd6e7
|
|
steps:
|
|
- name: Checkout repository
|
|
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
|
|
with:
|
|
persist-credentials: false
|
|
|
|
- name: Setup Rust environment
|
|
uses: ./.github/actions/setup
|
|
with:
|
|
rust-version: stable
|
|
# Dedicated key: ubuntu-latest and sm-standard-4 share runner.os, so
|
|
# a shared key would mix VM and ARC pod target/ artifacts.
|
|
cache-shared-key: ci-e2e-distributed-hosted
|
|
cache-save-if: ${{ github.ref == 'refs/heads/main' }}
|
|
install-build-packaging-tools: 'false'
|
|
|
|
- name: Prepare isolated filesystems for pool movement
|
|
run: |
|
|
set -euo pipefail
|
|
mount_base="${RUNNER_TEMP}/rustfs-e2e-pools"
|
|
mkdir -p "${mount_base}"
|
|
roots=()
|
|
for pool in 0 1 2 3; do
|
|
mountpoint="${mount_base}/pool-${pool}"
|
|
mkdir -p "${mountpoint}"
|
|
# Sized tmpfs reports a distinct st_dev and independent 1G
|
|
# statfs capacity. Requires a VM runner (ubuntu-latest).
|
|
if ! sudo mount -t tmpfs -o size=1G,nosuid,nodev,mode=1777 tmpfs "${mountpoint}"; then
|
|
echo "tmpfs mount failed on $(uname -a)" >&2
|
|
findmnt || true
|
|
grep Cap /proc/self/status || true
|
|
exit 1
|
|
fi
|
|
sudo chmod 1777 "${mountpoint}"
|
|
roots+=("${mountpoint}")
|
|
done
|
|
printf -v joined_roots '%s:' "${roots[@]}"
|
|
echo "RUSTFS_E2E_POOL_ROOTS=${joined_roots%:}" >> "${GITHUB_ENV}"
|
|
findmnt --noheadings --output TARGET,SOURCE,FSTYPE,SIZE --target "${roots[0]}"
|
|
findmnt --noheadings --output TARGET,SOURCE,FSTYPE,SIZE --target "${roots[1]}"
|
|
findmnt --noheadings --output TARGET,SOURCE,FSTYPE,SIZE --target "${roots[2]}"
|
|
findmnt --noheadings --output TARGET,SOURCE,FSTYPE,SIZE --target "${roots[3]}"
|
|
|
|
- name: Download pinned previous release
|
|
env:
|
|
SOURCE_DIR: ${{ runner.temp }}/rustfs-upgrade-source
|
|
run: |
|
|
set -euo pipefail
|
|
mkdir -p "$SOURCE_DIR"
|
|
archive="$SOURCE_DIR/$UPGRADE_SOURCE_ASSET"
|
|
curl --fail --location --retry 3 --output "$archive" \
|
|
"https://github.com/${GITHUB_REPOSITORY}/releases/download/${UPGRADE_SOURCE_VERSION}/${UPGRADE_SOURCE_ASSET}"
|
|
echo "$UPGRADE_SOURCE_SHA256 $archive" | sha256sum --check --strict
|
|
unzip -q "$archive" -d "$SOURCE_DIR"
|
|
chmod +x "$SOURCE_DIR/rustfs"
|
|
test -x "$SOURCE_DIR/rustfs"
|
|
echo "RUSTFS_UPGRADE_SOURCE_BINARY=$SOURCE_DIR/rustfs" >> "$GITHUB_ENV"
|
|
|
|
- name: Build rustfs binary
|
|
run: |
|
|
python3 scripts/e2e_binary.py build --bins --features e2e-test-hooks
|
|
|
|
- name: Verify distributed e2e membership
|
|
env:
|
|
NEXTEST_LISTING: ${{ runner.temp }}/rustfs-e2e-distributed-list.json
|
|
run: |
|
|
cargo nextest list --profile e2e-distributed -p e2e_test --message-format json > "${NEXTEST_LISTING}"
|
|
python3 ./scripts/check_test_wiring.py --check-profile e2e-distributed "${NEXTEST_LISTING}"
|
|
|
|
- name: Run distributed 4-node e2e suite
|
|
env:
|
|
RUSTFS_E2E_LOG_DIR: ${{ runner.temp }}/rustfs-e2e-distributed-logs
|
|
FILTER: ${{ inputs.filter }}
|
|
run: |
|
|
set -euo pipefail
|
|
rm -f target/nextest/e2e-distributed/junit.xml
|
|
if [ -n "${FILTER}" ]; then
|
|
python3 scripts/e2e_binary.py run --features e2e-test-hooks -- cargo nextest run --profile e2e-distributed -p e2e_test -E "${FILTER}"
|
|
else
|
|
python3 scripts/e2e_binary.py run --features e2e-test-hooks -- cargo nextest run --profile e2e-distributed -p e2e_test --no-tests=fail
|
|
fi
|
|
test -s target/nextest/e2e-distributed/junit.xml
|
|
|
|
- name: Upload distributed e2e diagnostics
|
|
if: always()
|
|
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
|
|
with:
|
|
name: e2e-distributed-${{ github.run_number }}
|
|
path: |
|
|
target/nextest/e2e-distributed/junit.xml
|
|
${{ runner.temp }}/rustfs-e2e-distributed-list.json
|
|
${{ runner.temp }}/rustfs-e2e-distributed-logs/
|
|
retention-days: 7
|
|
if-no-files-found: warn
|
|
|
|
- name: Unmount isolated pool filesystems
|
|
if: always()
|
|
run: |
|
|
set -euo pipefail
|
|
mount_base="${RUNNER_TEMP}/rustfs-e2e-pools"
|
|
for pool in 0 1 2 3; do
|
|
mountpoint="${mount_base}/pool-${pool}"
|
|
if mountpoint --quiet "${mountpoint}"; then
|
|
sudo umount "${mountpoint}"
|
|
fi
|
|
done
|
|
|
|
alert-on-failure:
|
|
name: Alert on scheduled failure
|
|
needs: [distributed]
|
|
if: github.repository == 'rustfs/rustfs' && (always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure'))
|
|
runs-on: ubuntu-latest
|
|
timeout-minutes: 10
|
|
permissions:
|
|
contents: read
|
|
issues: write
|
|
steps:
|
|
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
|
|
with:
|
|
persist-credentials: false
|
|
- name: Open or update failure-tracking issue
|
|
uses: ./.github/actions/schedule-failure-issue
|
|
with:
|
|
github-token: ${{ secrets.BACKLOG_ISSUE_TOKEN }}
|