diff --git a/.github/workflows/README.md b/.github/workflows/README.md index e5492068d..913058a61 100644 --- a/.github/workflows/README.md +++ b/.github/workflows/README.md @@ -7,7 +7,7 @@ This repository is public. For a public repository, GitHub's standard hosted runners (`ubuntu-latest`, `windows-latest`, `macos-latest`) cost nothing and have no minute cap, whatever plan the organisation is on. GitHub bills only two things: larger hosted runners, which have custom labels and are not used here, and standard runners in private repositories. -Every other label in this repository is capacity RustFS pays for by the hour whether or not a job is running well: +Every other label in this repository is capacity RustFS pays for. ARC pods and lab machines accrue cost while the machine exists. A Ubicloud label is metered for the minutes a job holds its VM: | Label | Paid by | What it is | Use it for | |---|---|---|---| @@ -15,7 +15,7 @@ Every other label in this repository is capacity RustFS pays for by the hour whe | `windows-latest`, `macos-latest` | nobody | GitHub-hosted standard runners | Windows and macOS targets | | `sm-standard-2` | RustFS | ARC pod, Ubuntu 24.04, minimal image: no `gh`, no C toolchain; Docker only on the `dind-` label | only a job that a hosted runner cannot run (see the placement rule) | | `sm-standard-4` | RustFS | ARC pod, Ubuntu 24.04, 14 CPUs and 28 GB (the label is not the core count), large disk | workspace-scale Cargo builds | -| `sm-standard-4-arm` | RustFS | ARC pod, aarch64; reached through the `build.yml` matrix, so `actionlint.yaml` does not list it | native aarch64 release builds | +| `ubicloud-standard-16-arm` | RustFS | Ubicloud VM, alias of `ubicloud-standard-16-arm-ubuntu-2404`: Ubuntu 24.04 arm64, 16 vCPU, 48 GB, 300 GB disk. Reached through the `build.yml` matrix, so `actionlint.yaml` does not list it | native aarch64 release builds (`linux-aarch64-musl`, `linux-aarch64-gnu`) | | `dind-sm-standard-2` | RustFS | ARC pod with Docker-in-Docker | image builds that need `buildx` and a registry push | | `smoke-testing`, `pf-testing` | RustFS | long-lived lab machines | the packaged functional chain and performance suites | @@ -40,7 +40,7 @@ A hosted runner is also the only choice for a few things the pods cannot do: `mo - Every `shared-key` has exactly one writer. `cache-warm.yml` owns `ci-dev`, `ci-feat-rio`, `ci-feat-proto` and `ci-uring`; every other `./.github/actions/setup` call states `cache-save-if` explicitly, and `scripts/security/check_cache_save_if.sh` fails the build when one does not. - All readers of a key run on the same image as its writer. Keys that live on hosted runners say so in their name (`ci-uring` is warmed on `ubuntu-latest`; `ci-e2e-distributed-hosted`, `e2e-upgrade-server-hosted`, `e2e-upgrade-tests-hosted`). -- Moving a job between pools therefore means renaming its key. The new key is empty until a run from `main` (a scheduled run, or a manual dispatch of that workflow from `main`) saves it; until then that lane does a cold build. +- Moving a job between pools therefore means renaming its key. The new key is empty until a run from `main` (a scheduled run, or a manual dispatch of that workflow from `main`) saves it; until then that lane does a cold build. The `build.yml` aarch64 release legs append `-ubicloud-arm` for this reason. `cache-warm.yml` does not write those keys. - `CARGO_BUILD_JOBS` and every other `CARGO*`, `CC*`, `CFLAGS*`, `CXX*`, `CMAKE*` and `RUST*` variable is hashed into the key. `ci.yml` sets `CARGO_BUILD_JOBS` at step level for that reason; promoting it to job level rotates the key for every lane that shares it. ## Where self-hosted minutes leak @@ -62,7 +62,7 @@ gh api "repos/rustfs/rustfs/actions/runs//jobs?per_page=100" \ --jq '.jobs[] | "\(.conclusion)\t\(.labels | join(","))\t\(.started_at)\t\(.completed_at)\t\(.name)"' ``` -Only rows whose label starts with `sm-`, `dind-`, `smoke-` or `pf-` cost anything. At the October 2026 review, a full pull-request run of `ci.yml` spent about 150 self-hosted minutes, a third of it in jobs that never compiled; treat that as the scale of what a placement mistake costs, not as a current figure. +Only rows whose label starts with `sm-`, `dind-`, `ubicloud-`, `smoke-` or `pf-` cost anything. At the October 2026 review, a full pull-request run of `ci.yml` spent about 150 self-hosted minutes, a third of it in jobs that never compiled; treat that as the scale of what a placement mistake costs, not as a current figure. ## Checklist for a new or moved job diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index 41941fbcf..980878898 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -181,12 +181,18 @@ jobs: selected="all" fi - + # linux-aarch64-* need a real aarch64 host (placement rule: hardware) + # so the packaged binary and console smoke execute the artifact. + # ubicloud-standard-16-arm is that host: Ubuntu 24.04 arm64, alias of + # ubicloud-standard-16-arm-ubuntu-2404 (16 vCPU, 48 GB). linux-x86_64-* stay on the + # ARC sm-standard-4 pool. GNU rows keep cross=true so cargo-zigbuild + # pins the glibc 2.31 floor on the same architecture; musl stays a + # native cargo build. all='{"include":[ {"target_id":"linux-x86_64-musl","os":"sm-standard-4","target":"x86_64-unknown-linux-musl","cross":false,"platform":"linux","rustflags":"","features":"","glibc":""}, - {"target_id":"linux-aarch64-musl","os":"sm-standard-4-arm","target":"aarch64-unknown-linux-musl","cross":false,"platform":"linux","rustflags":"","features":"","glibc":""}, + {"target_id":"linux-aarch64-musl","os":"ubicloud-standard-16-arm","target":"aarch64-unknown-linux-musl","cross":false,"platform":"linux","rustflags":"","features":"","glibc":""}, {"target_id":"linux-x86_64-gnu","os":"sm-standard-4","target":"x86_64-unknown-linux-gnu","cross":true,"platform":"linux","rustflags":"","features":"pyroscope","glibc":"2.31"}, - {"target_id":"linux-aarch64-gnu","os":"sm-standard-4-arm","target":"aarch64-unknown-linux-gnu","cross":true,"platform":"linux","rustflags":"","features":"pyroscope","glibc":"2.31"}, + {"target_id":"linux-aarch64-gnu","os":"ubicloud-standard-16-arm","target":"aarch64-unknown-linux-gnu","cross":true,"platform":"linux","rustflags":"","features":"pyroscope","glibc":"2.31"}, {"target_id":"macos-aarch64","os":"macos-latest","target":"aarch64-apple-darwin","cross":false,"platform":"macos","rustflags":"","features":"pyroscope","glibc":""}, {"target_id":"windows-x86_64","os":"windows-latest","target":"x86_64-pc-windows-msvc","cross":false,"platform":"windows","rustflags":"","features":"","glibc":""} ]}' @@ -244,7 +250,11 @@ jobs: with: rust-version: stable target: ${{ matrix.target }} - cache-shared-key: build-${{ matrix.target }}${{ matrix.glibc && format('-glibc-{0}', matrix.glibc) || '' }} + # rust-cache keys include OS and architecture but not the runner image. + # The Ubicloud aarch64 legs therefore use their own suffix so they + # cannot restore a cache written on a different arm64 image. + # x86_64, macOS, and Windows keys stay unsuffixed. + cache-shared-key: build-${{ matrix.target }}${{ matrix.glibc && format('-glibc-{0}', matrix.glibc) || '' }}${{ matrix.os == 'ubicloud-standard-16-arm' && '-ubicloud-arm' || '' }} # main only. A cache saved on refs/tags/X is scoped to that tag: no # other tag, no main run and no PR can restore it, so every release # cycle wrote up to 12 entries of 1-2GB (preview tag plus final tag, @@ -764,9 +774,9 @@ jobs: exit 0 fi - # The self-hosted Linux runners do not ship the aws CLI (GitHub-hosted - # images did). Install the official CLI bundle on demand so R2 uploads - # survive a fresh runner without relying on a distro package. + # ARC pods (the x86_64 legs) do not ship the aws CLI. Ubicloud arm64 + # images follow the GitHub runner image and often do. Install the + # official CLI bundle when it is missing so R2 uploads work on either. if ! command -v aws >/dev/null 2>&1; then echo "aws CLI not found on runner; installing..." if command -v apt-get >/dev/null 2>&1; then