Files
rustfs/.github/workflows/ci.yml
houseme 1553dc3f62 Address P2 follow-ups from the 2026-07-10..12 merged-PR review (backlog#1210-1220) (#4783)
* fix(obs): open cleaner compression source with O_NOFOLLOW

The compressor opened the source log via File::open, which follows a
symlink at the final path component. Between the scanner selecting a
regular file and this open, an attacker with write access to the log
directory could swap the entry for a symlink (TOCTOU) pointing at, say,
/etc/shadow, whose contents would then be copied into an archive. Open
the source with O_NOFOLLOW on Unix so such a swap fails with ELOOP; the
temp/archive path already refused symlinks, this closes the source side.

Refs rustfs/backlog#1210
Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(obs): recompress instead of trusting leftover cleaner archives

archive_header_ok only checked the first 2-4 magic bytes before treating
an existing .gz/.zst as a completed prior result and letting the caller
delete the source log. A file with valid magic but a truncated or forged
body passes that check, so an attacker with write access to the log
directory (or a crashed prior run) could plant such a stub and make the
cleaner delete the real log without ever producing a usable archive —
silent audit-data loss.

Chosen fix: stop trusting cross-process leftovers entirely and always
recompress the source in this pass, rather than fully decoding every
leftover to validate it. Full-decode validation would add real CPU cost
and decode-bug surface for a rare crash-recovery case; the existing
atomic create_new+rename already overwrites whatever sits at the archive
path (a planted symlink is replaced, never followed) with a freshly
written, fsync'd archive, so a partial/forged leftover can never gate
source deletion. This is the lowest-regression option.

Refs rustfs/backlog#1211
Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(object-data-cache): cap memory-gate reservation at cache growth headroom

The memory gate subtracts `admitted_since_refresh` from the snapshot's
available bytes so a burst arriving faster than the 5 s refresh cannot
over-allocate. That counter is GROSS: it only rolls over on the refresh and
never rolls back when a fill is later evicted, cancelled, or loses the
invalidation race. Under sustained high-throughput churn (net footprint flat
and far below `max_capacity`) the raw counter balloons past the memory the
cache actually holds, so `effective_available` collapses and the gate reports
false memory pressure — skipping the hottest fills with SkippedMemoryPressure
until the next 5 s refresh. This only lowers hit rate; it never returns wrong
data and self-heals each refresh.

Fix direction 1 (minimal regression): cap the reservation deduction at the
cache's own growth headroom (`max_capacity - weighted_size()`) instead of
letting the unbounded gross counter shrink the system-available budget. The
cache can never hold more than `max_capacity`, so a burst adds at most that
headroom of real memory before moka evicts to stay bounded (net-zero churn
beyond that point) — capping the deduction there keeps the reservation honest
without treating gross churn as growth. Chosen over net-accounting (direction
2, releasing bytes on every failure/cancel/eviction path) because that only
plugs the leak on failed fills and would not address the core defect: churn of
*successful* insert/evict fills over the 5 s window. It also touches only the
gate plus one call site rather than every failure path in moka_backend.

The cap only ever raises `effective_available`, so real memory pressure (a low
snapshot at refresh) still suppresses fills; when the cache is at capacity the
headroom is 0 and the deduction vanishes, correctly reflecting net-zero churn.
`MokaBackend` now stores `max_capacity` and passes the live headroom into
`allows_fill`. Adds targeted gate tests: gross churn far above headroom no
longer falsely suppresses, yet the reservation still bounds a burst while the
cache can genuinely grow.

Refs rustfs/backlog#1212
Co-Authored-By: heihutu <heihutu@gmail.com>

* test(ecstore): assert native O_DIRECT path runs in uring read test

uring_preserves_o_direct_for_eligible_reads only compared bytes through
LocalDisk::read_file_mmap_copy. On a filesystem that rejects O_DIRECT the
read silently degrades to the buffered StdBackend fallback and the byte
check still passes, so the test could go green without the native
read_at_direct path ever executing -- a vacuous pass.

Add a per-disk native_direct_reads counter on UringBackend, incremented
only when pread_uring_direct completes, and rebuild the test to drive a
real UringBackend's pread_bytes and assert the counter is non-zero (every
eligible read went through the native tier). When io_uring or O_DIRECT is
unavailable on the host filesystem (restricted CI runners, tmpfs), the
test skips loudly via eprintln instead of asserting a tautology, while
still checking byte-correctness on whatever tier served the read.

The counter also gives a gray release a positive signal that the O_DIRECT
tier is serving reads, not just a fallback count.

Refs rustfs/backlog#1213
Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(ecstore): warn + count read-time EINVAL on native O_DIRECT reads

classify_direct_read_error is only reached from the read side: the
O_DIRECT open in pread_uring_direct already succeeded (an open-time
refusal is handled earlier as DirectOpenError::ODirectRefused). So an
EINVAL/EOPNOTSUPP arriving here is a read-time error on an fd the kernel
accepted for O_DIRECT -- far more likely an alignment bug in the aligned
read path than an unsupported filesystem. The old code latched the disk's
native path off with only a once-per-disk debug trace, hiding a potential
correctness regression behind a silent buffered-read downgrade.

Diagnostics only: the fallback behaviour is unchanged (the native path is
still latched off and the caller still reads via StdBackend). This adds a
rustfs_io_uring_direct_read_einval_total counter and promotes the
once-per-disk trace from debug to warn so an operator can see an alignment
regression instead of an unexplained latency/CPU shift.

Refs rustfs/backlog#1214
Co-Authored-By: heihutu <heihutu@gmail.com>

* docs(ecstore): document data-blocks-first default and its tail-latency cost

DEFAULT_RUSTFS_GET_DATA_BLOCKS_FIRST_READER_SETUP is true and must stay
true: deferred-parity is the deliberate, already-rolled-out full-object
GET default from backlog#1159/#923. Flipping it back to false in code
would silently revert that rollout for every deployment that has not set
the env var, so this commit only documents -- no behaviour change.

The added notes explain what data-blocks-first does (schedule data shards
up front, engage parity lazily on a missing/corrupt data shard), the known
trade-off (parity is engaged late, so a slow-but-not-dead data drive
raises GET p99 because the faster parity shards are not raced against it
until a data shard is declared missing), and the operational rollback
switch (RUSTFS_GET_DATA_BLOCKS_FIRST_READER_SETUP=false), which is
intentionally an env override rather than a code default change.

No metric was added: the low-risk observability hook for "slow data drive
engaged deferred parity" would live at the deferred-stripe engage point,
which is out of this file's scope; this change stays documentation-only to
avoid touching the hot GET path.

Refs rustfs/backlog#1215
Co-Authored-By: heihutu <heihutu@gmail.com>

* docs(ecstore): document wide-directory walk stall hazard and tuning

list_dir enumerates a whole directory in one os::read_dir call (count =
-1), and the walk caller bounds that entire enumeration with the per-read
stall budget (default 5s) as if it were a single read. For a wide, flat
prefix -- one directory holding millions of immediate children -- a single
readdir can exceed the budget on a healthy disk, trip DiskError::Timeout,
and surface as a ListObjects 500 quorum failure though the drive is fine
(a #2999 sub-class).

This is documented, not rewritten: turning the one-shot readdir into a
streaming/batched enumeration that refreshes the stall deadline between
chunks is an architecture-level change with high regression surface
(ordering, the count contract, quorum merge) and belongs in a separate
follow-up. The supported mitigation today is operational, so the comments
point wide-directory deployments at RUSTFS_DRIVE_WALKDIR_STALL_TIMEOUT_SECS
and the high-latency drive-timeout profile, which widen the budget with no
code change. Notes were added at list_dir, the scan_dir call site, and
get_drive_walkdir_stall_timeout. No behaviour change.

Refs rustfs/backlog#1216
Co-Authored-By: heihutu <heihutu@gmail.com>

* docs(ecstore): document consumer-peek vs producer-stall coupling

In list_path_raw the consumer's peek_timeout is drawn from the same source
and same value (walkdir_stall_timeout, default 5s) as the producer-side
walk stall budget, but the two measure different things: the producer
stall bounds a single drive read, while the consumer peek bounds the gap
between two ADJACENT entries arriving from a reader. Because they share a
value, the consumer cannot wait meaningfully longer for the next entry
than the producer is allowed to spend producing one. Walking a region
dense with non-listable internal items can make a HEALTHY drive miss the
budget between visible entries; the consumer then declares it stalled and
detaches it, dropping a good drive from the merge and capping the "large
prefix succeeds" guarantee.

Documented, not decoupled: giving the consumer peek an independent,
strictly-larger budget would cut these false detaches but equally delays
detaching a genuinely dead drive and shifts listing tail-latency
semantics, so it wants soak data before changing the default. The comment
records the invariant any such follow-up must keep -- consumer peek >=
producer stall, never stricter -- so it can never fail a drive before the
producer would. No behaviour change.

Refs rustfs/backlog#1217
Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(io-metrics): add time-based trigger for low-IOPS latency percentiles

Percentiles were recomputed only every 128 IOs and seeded to 0, so a
low-traffic deployment exported p95/p99 = 0/stale for a long time after
startup. Add a 10s wall-clock trigger alongside the count throttle so the
first recompute can fire before 128 samples accrue. Hot-path per-op mean
update is unchanged.

Refs rustfs/backlog#1218
Co-Authored-By: heihutu <heihutu@gmail.com>

* test(e2e): cover codec-streaming parity under fault injection and NoSuchKey

The codec-streaming compat A/B previously ran only against a healthy
4-disk EC set with successful full GETs: the DiskFaultHarness was
constructed but never faulted, the error path was untested, and the
range assertion silently compared legacy-vs-legacy (ranges always fall
back to the duplex path), overstating what it proved.

Add two genuinely-failable scenarios reusing the existing harness and
fixtures:

- Parity reconstruction A/B: take one data disk offline and re-run the
  full object matrix on both phases while the EC 2+2 set rebuilds each
  large object from the surviving shards. Assert codec == legacy
  byte-for-byte (sha256) and header-for-header, and assert the codec
  phase served the reconstructed objects with zero duplex-pipe fallback
  (the reader gate is drive-health-independent, so the codec fast path
  is really exercised through reconstruction).
- NoSuchKey negative path: compare the HTTP status + S3 error code of a
  missing-key GET across the legacy and codec phases and require them to
  be identical (404/NoSuchKey), guarding against the codec env
  perturbing the error path.

Also clarify the range-phase comment so it is not misread as
codec-range correctness coverage: both sides are served by the same
legacy range path, so the assertion only proves ranges keep working and
keep falling back to legacy with the gates open.

Verified: cargo check/--no-run pass and the test passes locally
(1 passed; dup_codec=0 confirms the codec path ran).

Refs rustfs/backlog#1219
Co-Authored-By: heihutu <heihutu@gmail.com>

* ci(ecstore): exercise native O_DIRECT read path on an ext4 loopback

The uring-integration leg ran on the runner's default TMPDIR, which may sit
on tmpfs/overlayfs where open(O_DIRECT) fails and the native read_at_direct
path silently latches off to the aligned StdBackend fallback. Mount a
dedicated ext4 loopback and point TMPDIR at it so the real io_uring dep
(bumped git->0.1.0->0.2.0->0.2.1) and the native O_DIRECT read path are
actually covered rather than validated only by signature diffing.

Refs rustfs/backlog#1220
Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-12 16:03:28 +00:00

632 lines
25 KiB
YAML

# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
name: Continuous Integration
on:
push:
branches: [ main ]
paths-ignore:
- "**.md"
- "docs/**"
- "deploy/**"
- "scripts/dev_*.sh"
- "scripts/probe.sh"
- "LICENSE*"
- ".gitignore"
- ".dockerignore"
- "README*"
- "**/*.png"
- "**/*.jpg"
- "**/*.svg"
- ".github/workflows/build.yml"
- ".github/workflows/docker.yml"
- ".github/workflows/audit.yml"
pull_request:
types: [ opened, synchronize, reopened, closed ]
branches: [ main ]
# Keep this list in sync with the `paths` list in ci-docs-only.yml, which
# reports the required "Test and Lint" check for PRs skipped here.
paths-ignore:
- "**.md"
- "docs/**"
- "deploy/**"
- "scripts/dev_*.sh"
- "scripts/probe.sh"
- "LICENSE*"
- ".gitignore"
- ".dockerignore"
- "README*"
- "**/*.png"
- "**/*.jpg"
- "**/*.svg"
- ".github/workflows/build.yml"
- ".github/workflows/docker.yml"
- ".github/workflows/audit.yml"
merge_group:
types: [ checks_requested ]
schedule:
- cron: "0 0 * * 0" # Weekly on Sunday at midnight UTC
workflow_dispatch:
permissions:
contents: read
# Concurrency groups are scoped per event so different triggers never cancel
# each other: PR pushes cancel the previous run of that PR, main pushes keep
# latest-wins semantics among themselves, and scheduled runs always complete
# (a shared group used to let every merge kill the weekly scheduled run).
concurrency:
group: ${{ github.workflow }}-${{ github.event_name }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: ${{ github.event_name != 'schedule' }}
env:
CARGO_TERM_COLOR: always
RUST_BACKTRACE: 1
jobs:
cancel-closed-pr-runs:
name: Cancel Closed PR Runs
if: github.event_name == 'pull_request' && github.event.action == 'closed'
runs-on: ubuntu-latest
steps:
- name: Explain cancellation run
run: echo "PR closed; this run only cancels older runs in the same concurrency group."
typos:
name: Typos
if: github.event_name != 'pull_request' || github.event.action != 'closed'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- name: Typos check with custom config file
uses: crate-ci/typos@37bb98842b0d8c4ffebdb75301a13db0267cef89 # master
# Fast, compile-free checks that fail early so contributors get feedback in
# ~1 minute instead of waiting for the full test job.
quick-checks:
name: Quick Checks
if: github.event_name != 'pull_request' || github.event.action != 'closed'
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- name: Install ripgrep
run: sudo apt-get update && sudo apt-get install -y ripgrep
- name: Install Rust toolchain
uses: dtolnay/rust-toolchain@29eef336d9b2848a0b548edc03f92a220660cdb8 # stable
with:
components: rustfmt
- name: Check code formatting
run: cargo fmt --all --check
- name: Check unsafe code allowances
run: ./scripts/check_unsafe_code_allowances.sh
- name: Check layered dependencies
run: ./scripts/check_layer_dependencies.sh
- name: Check architecture migration rules
run: ./scripts/check_architecture_migration_rules.sh
- name: Check tokio io-uring feature guard
run: ./scripts/check_no_tokio_io_uring.sh
- name: Check extension schema boundaries
run: ./scripts/check_extension_schema_boundaries.sh
- name: Check body-cache whitelist guard
run: ./scripts/check_body_cache_whitelist.sh
- name: Check no planning docs committed
run: ./scripts/check_no_planning_docs.sh
test-and-lint:
name: Test and Lint
if: github.event_name != 'pull_request' || github.event.action != 'closed'
runs-on: sm-standard-4
timeout-minutes: 60
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: stable
cache-shared-key: ci-test
github-token: ${{ secrets.GITHUB_TOKEN }}
cache-save-if: ${{ github.ref == 'refs/heads/main' }}
# Clippy runs before the test pass: lint failures are the most common
# CI-only breakage and should surface in minutes, not after 20+ minutes
# of tests.
- name: Run clippy lints
run: cargo clippy --all-targets -- -D warnings
- name: Run tests
run: |
cargo nextest run --profile ci --all --exclude e2e_test
cargo test --all --doc
- name: Upload test junit report
if: always()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
with:
name: junit-test-and-lint-${{ github.run_number }}
path: target/nextest/ci/junit.xml
retention-days: 3
if-no-files-found: ignore
# Explicit gate for migration-critical suites. These tests already ran in
# the full nextest pass above; a single filtered nextest invocation keeps
# the named gate without rebuilding or re-running them one package at a time.
#
# The gate selects tests by name substring (data_movement / rebalance /
# decommission / source_cleanup / delete_marker), so renames can silently
# thin it. The script owns the filter expression and first verifies the
# selected-test count against the committed floor in
# .config/migration-gate-floor.txt before running the gate; renames or
# removals must update that file consciously (see the script header).
# Kept on the default profile (no --profile ci): a second --profile ci run
# would clobber target/nextest/ci/junit.xml, and none of these tests are
# quarantined so they gain nothing from the ci profile's retry overrides.
- name: Run rebalance/decommission migration proofs
run: ./scripts/check_migration_gate_count.sh
# Dedicated serial lane for the ILM / lifecycle integration tests. These tests
# drive the object layer through process-global singletons (the GLOBAL_ENV
# ECStore, the global tier-config manager, background-expiry workers) and bind
# fixed ports (9002 for the scanner suite, 9003 for the app suite), so they are
# marked #[ignore] to keep them out of the default parallel `cargo nextest run
# --all` pass. Run them here explicitly with `--run-ignored ignored-only` and
# `-j1` so no two of them share global state or race for a port at once.
# serial_test's #[serial] does NOT serialize across nextest's process-per-test
# boundary (see .config/nextest.toml); `-j1` is what actually serializes them.
# See rustfs/backlog#1148 (ilm-1) and #1155.
test-ilm-integration-serial:
name: ILM Integration (serial)
if: github.event_name != 'pull_request' || github.event.action != 'closed'
runs-on: sm-standard-4
timeout-minutes: 45
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: stable
cache-shared-key: ci-ilm-serial
github-token: ${{ secrets.GITHUB_TOKEN }}
cache-save-if: ${{ github.ref == 'refs/heads/main' }}
# Three scanner tests fail on main independently of this lane (restore of
# a transitioned multipart object, noncurrent transition/expiry after an
# immediate compensation transition); they keep #[ignore] with a backlog
# reference and are excluded here by name until fixed (rustfs/backlog#1148).
- name: Run ignored ILM integration tests serially
run: |
cargo nextest run -j1 --run-ignored ignored-only \
-p rustfs-scanner -p rustfs \
-E '(binary(lifecycle_integration_test) or (package(rustfs) and test(lifecycle_transition_api_test))) and not (test(test_transition_and_restore_flows) or test(test_noncurrent_expiry_still_works_after_immediate_compensation_transition) or test(test_noncurrent_transition_still_works_after_immediate_compensation_transition))'
test-and-lint-rio-v2:
name: Test and Lint (rio-v2)
if: github.event_name != 'pull_request' || github.event.action != 'closed'
runs-on: sm-standard-4
timeout-minutes: 60
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: stable
cache-shared-key: ci-test-rio-v2
github-token: ${{ secrets.GITHUB_TOKEN }}
cache-save-if: ${{ github.ref == 'refs/heads/main' }}
- name: Run rio-v2 clippy lints
run: cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings
- name: Run rio-v2 feature tests
run: |
cargo nextest run -p rustfs -p rustfs-ecstore --features rio-v2
cargo test -p rustfs --doc --features rio-v2
test-and-lint-protocols:
name: "Test and Lint (${{ matrix.features.name }})"
if: github.event_name != 'pull_request' || github.event.action != 'closed'
runs-on: sm-standard-4
timeout-minutes: 60
strategy:
fail-fast: false
matrix:
features:
- name: swift
flags: "--features swift"
- name: sftp
flags: "--features sftp"
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: stable
cache-shared-key: ci-test-${{ matrix.features.name }}
github-token: ${{ secrets.GITHUB_TOKEN }}
cache-save-if: ${{ github.ref == 'refs/heads/main' }}
- name: Run clippy with ${{ matrix.features.name }}
run: |
cargo clippy -p rustfs -p rustfs-protocols --all-targets ${{ matrix.features.flags }} -- -D warnings
- name: Run tests with ${{ matrix.features.name }}
run: |
cargo nextest run -p rustfs -p rustfs-protocols ${{ matrix.features.flags }}
build-rustfs-debug-binary:
name: Build RustFS Debug Binary
if: github.event_name != 'pull_request' || github.event.action != 'closed'
runs-on: sm-standard-4
timeout-minutes: 30
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: stable
cache-shared-key: ci-rustfs-debug-binary
cache-save-if: ${{ github.ref == 'refs/heads/main' }}
github-token: ${{ secrets.GITHUB_TOKEN }}
- name: Build debug binary
run: cargo build -p rustfs --bins
- name: Upload debug binary
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
with:
name: rustfs-debug-binary
path: target/debug/rustfs
if-no-files-found: error
retention-days: 1
build-rustfs-debug-binary-rio-v2:
name: Build RustFS Debug Binary (rio-v2)
if: github.event_name != 'pull_request' || github.event.action != 'closed'
runs-on: sm-standard-4
timeout-minutes: 30
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: stable
cache-shared-key: ci-rustfs-debug-binary-rio-v2
cache-save-if: ${{ github.ref == 'refs/heads/main' }}
github-token: ${{ secrets.GITHUB_TOKEN }}
- name: Build debug binary with rio-v2
run: cargo build -p rustfs --bins --features rio-v2
- name: Upload debug binary
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
with:
name: rustfs-debug-binary-rio-v2
path: target/debug/rustfs
if-no-files-found: error
retention-days: 1
uring-integration:
name: io_uring Integration (real)
# GitHub-hosted ubuntu-latest runs a recent kernel with io_uring and, unlike
# a container, applies no seccomp filter that would block io_uring_setup — so
# the probe succeeds and the tests exercise the real UringBackend/FdCache/
# latch paths instead of the StdBackend fallback (rustfs/backlog#1179). The
# self-hosted sm-standard runners cannot guarantee this.
runs-on: ubuntu-latest
timeout-minutes: 30
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: stable
cache-shared-key: ci-uring
github-token: ${{ secrets.GITHUB_TOKEN }}
cache-save-if: ${{ github.ref == 'refs/heads/main' }}
- name: Install build dependencies
run: sudo apt-get update && sudo apt-get install -y protobuf-compiler
# ext4 supports O_DIRECT; the runner's default TMPDIR may sit on tmpfs or
# overlayfs, where open(O_DIRECT) returns EINVAL/EOPNOTSUPP and the native
# read_at_direct path silently latches off to the aligned StdBackend
# fallback. A dedicated ext4 loopback guarantees the native O_DIRECT read
# path is actually exercised, not just the fallback (rustfs/backlog#1220).
- name: Prepare ext4 loopback for O_DIRECT coverage
run: |
set -euo pipefail
dd if=/dev/zero of=/tmp/rustfs-odirect.img bs=1M count=1024
mkfs.ext4 -q -F /tmp/rustfs-odirect.img
sudo mkdir -p /mnt/rustfs-odirect
sudo mount -o loop /tmp/rustfs-odirect.img /mnt/rustfs-odirect
sudo chmod 1777 /mnt/rustfs-odirect
mount | grep rustfs-odirect
# RUSTFS_URING_TESTS_MUST_RUN=1 makes the io_uring tests fail instead of
# silently skipping if io_uring is unavailable, so this leg can never pass
# vacuously. RUSTFS_IO_URING_READ_ENABLE=true selects the UringBackend.
# TMPDIR points every tempfile-based test fixture at the ext4 loopback so
# eligible reads keep O_DIRECT and the native path is covered end to end.
- name: Run ecstore io_uring integration tests on real io_uring
env:
RUSTFS_IO_URING_READ_ENABLE: "true"
RUSTFS_URING_TESTS_MUST_RUN: "1"
TMPDIR: /mnt/rustfs-odirect
run: cargo test -p rustfs-ecstore uring_ -- --test-threads=1 --nocapture
e2e-tests:
name: End-to-End Tests
needs: [ build-rustfs-debug-binary ]
runs-on: sm-standard-2
timeout-minutes: 30
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
# Full setup with dependency caching: the smoke-suite step below
# compiles the e2e_test crate, which pulls in most of the workspace.
# Without a restored cache this was a cold multi-GB build on every run.
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: stable
cache-shared-key: ci-e2e
github-token: ${{ secrets.GITHUB_TOKEN }}
cache-save-if: ${{ github.ref == 'refs/heads/main' }}
# Download after the cache restore so the freshly built binary from the
# build job always wins over anything restored into target/debug.
- name: Download debug binary
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
with:
name: rustfs-debug-binary
path: target/debug
- name: Make binary executable
run: chmod +x ./target/debug/rustfs
# PR smoke subset of the in-repo e2e suite (backlog#1149 ci-4). The
# profile.e2e-smoke default-filter in .config/nextest.toml is the single
# wiring mechanism for e2e tests in CI — extend that filter instead of
# adding new e2e jobs here. Each test spawns its own rustfs server on a
# random port and reuses the downloaded debug binary above.
- name: Run e2e smoke suite
run: cargo nextest run --profile e2e-smoke -p e2e_test
- name: Install s3s-e2e test tool
uses: taiki-e/cache-cargo-install-action@7447f04c51f2ba27ca35e7f1e28fab848c5b3ba7 # v2
with:
tool: s3s-e2e
git: https://github.com/s3s-project/s3s.git
rev: 62cb4a71dd759a6ec56b64c4c42fcc183a2c6a52
- name: Run end-to-end tests
run: |
s3s-e2e --version
RUN_ROOT="${RUNNER_TEMP}/rustfs-e2e-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}-${GITHUB_JOB}"
mkdir -p "${RUN_ROOT}"
RUSTFS_TEST_PORT="$(python3 -c 'import socket; s=socket.socket(); s.bind(("127.0.0.1", 0)); print(s.getsockname()[1]); s.close()')"
RUSTFS_TEST_PORT="${RUSTFS_TEST_PORT}" \
RUSTFS_TEST_LOG="${RUN_ROOT}/rustfs.log" \
./scripts/e2e-run.sh ./target/debug/rustfs "${RUN_ROOT}/data"
- name: Upload test logs
if: failure()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
with:
name: e2e-test-logs-${{ github.run_number }}
path: ${{ runner.temp }}/rustfs-e2e-*/rustfs.log
retention-days: 3
e2e-tests-rio-v2:
name: End-to-End Tests (rio-v2)
needs: [ build-rustfs-debug-binary-rio-v2 ]
runs-on: sm-standard-2
timeout-minutes: 30
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- name: Clean up previous test run
run: |
rm -rf /tmp/rustfs
rm -f /tmp/rustfs.log
- name: Download debug binary
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
with:
name: rustfs-debug-binary-rio-v2
path: target/debug
- name: Make binary executable
run: chmod +x ./target/debug/rustfs
- name: Setup Rust toolchain for s3s-e2e installation
uses: dtolnay/rust-toolchain@29eef336d9b2848a0b548edc03f92a220660cdb8 # stable
- name: Install s3s-e2e test tool
uses: taiki-e/cache-cargo-install-action@7447f04c51f2ba27ca35e7f1e28fab848c5b3ba7 # v2
with:
tool: s3s-e2e
git: https://github.com/s3s-project/s3s.git
rev: 62cb4a71dd759a6ec56b64c4c42fcc183a2c6a52
- name: Run end-to-end tests
run: |
s3s-e2e --version
./scripts/e2e-run.sh ./target/debug/rustfs /tmp/rustfs
- name: Upload test logs
if: failure()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
with:
name: e2e-test-logs-rio-v2-${{ github.run_number }}
path: /tmp/rustfs.log
retention-days: 3
s3-implemented-tests:
name: S3 Implemented Tests
needs: [ build-rustfs-debug-binary, e2e-tests ]
runs-on: sm-standard-4
timeout-minutes: 60
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- name: Download debug binary
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
with:
name: rustfs-debug-binary
path: target/debug
- name: Make binary executable
run: chmod +x ./target/debug/rustfs
- name: Run implemented s3-tests
run: |
RUN_ROOT="${RUNNER_TEMP}/rustfs-s3tests-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}-${GITHUB_JOB}"
mkdir -p "${RUN_ROOT}"
S3_PORT="$(python3 -c 'import socket; s=socket.socket(); s.bind(("127.0.0.1", 0)); print(s.getsockname()[1]); s.close()')"
DEPLOY_MODE=binary \
RUSTFS_BINARY=./target/debug/rustfs \
TEST_MODE=single \
MAXFAIL=0 \
XDIST=4 \
S3_PORT="${S3_PORT}" \
DATA_ROOT="${RUN_ROOT}" \
S3TESTS_CONF=artifacts/s3tests-single/s3tests.conf \
./scripts/s3-tests/run.sh
- name: Upload s3 test artifacts
if: always()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
with:
name: s3tests-implemented-${{ github.run_number }}
path: artifacts/s3tests-single/**
if-no-files-found: ignore
retention-days: 3
# Dedicated lifecycle behavior lane (backlog#1148 ilm-10, master plan #1155).
#
# These ceph/s3-tests cases assert that objects/versions/uploads are actually
# removed by the background scanner and the stale-multipart cleanup loop, so
# they need a debug-accelerated day plus an enabled scanner. That configuration
# cannot be applied to the s3-implemented-tests lane above because a global
# RUSTFS_ILM_DEBUG_DAY_SECS also shrinks the x-amz-expiration header those tests
# assert on (test_lifecycle_expiration_header_*). Hence a separate server here.
#
# RUSTFS_ILM_DEBUG_DAY_SECS MUST stay equal to the s3-tests lc_debug_interval
# default (10s). test_lifecycle_expiration polls the Days=1 rule with a
# 4*lc_interval window while the Days=5 rule fires at 5*debug_day; observing
# the intermediate 4-object plateau requires 4*lc_interval < 5*debug_day, i.e.
# debug_day > 8. A smaller value lets the Days=5 rule fire inside the Days=1
# poll window so the count collapses straight to 2 and the test fails (#4764).
#
# RUSTFS_DATA_USAGE_UPDATE_DIR_CYCLES=1 is also required. A compacted directory
# is only re-descended (and its objects re-evaluated for ILM) once every
# DATA_USAGE_UPDATE_DIR_CYCLES scanner cycles (default 16). At the accelerated
# RUSTFS_SCANNER_CYCLE=2 that is ~32s between evaluations, so a Days=1 object
# that becomes due at debug_day (10s) is not actually expired until the next
# ~32s boundary (~42s), landing just past the 4*lc_interval (40s) poll window
# and making the count stall at 6 (#4767). Forcing every-cycle re-descent
# evaluates ILM within ~2s of the due time, well inside the poll window.
s3-lifecycle-behavior-tests:
name: S3 Lifecycle Behavior Tests
needs: [ build-rustfs-debug-binary ]
runs-on: sm-standard-4
timeout-minutes: 30
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- name: Download debug binary
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
with:
name: rustfs-debug-binary
path: target/debug
- name: Make binary executable
run: chmod +x ./target/debug/rustfs
- name: Run lifecycle behavior s3-tests
run: |
RUN_ROOT="${RUNNER_TEMP}/rustfs-s3tests-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}-${GITHUB_JOB}"
mkdir -p "${RUN_ROOT}"
S3_PORT="$(python3 -c 'import socket; s=socket.socket(); s.bind(("127.0.0.1", 0)); print(s.getsockname()[1]); s.close()')"
DEPLOY_MODE=binary \
RUSTFS_BINARY=./target/debug/rustfs \
TEST_MODE=single \
MAXFAIL=0 \
XDIST=0 \
IMPLEMENTED_TESTS_FILE=scripts/s3-tests/lifecycle_behavior_tests.txt \
RUSTFS_ILM_DEBUG_DAY_SECS=10 \
RUSTFS_ILM_PROCESS_TIME=1 \
RUSTFS_SCANNER_ENABLED=true \
RUSTFS_SCANNER_CYCLE=2 \
RUSTFS_DATA_USAGE_UPDATE_DIR_CYCLES=1 \
RUSTFS_SCANNER_START_DELAY_SECS=0 \
RUSTFS_API_STALE_UPLOADS_CLEANUP_INTERVAL=2s \
S3_PORT="${S3_PORT}" \
DATA_ROOT="${RUN_ROOT}" \
S3TESTS_CONF=artifacts/s3tests-single/s3tests.conf \
./scripts/s3-tests/run.sh
- name: Upload s3 test artifacts
if: always()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
with:
name: s3tests-lifecycle-behavior-${{ github.run_number }}
path: artifacts/s3tests-single/**
if-no-files-found: ignore
retention-days: 3