mirror of
https://github.com/rustfs/rustfs.git
synced 2026-08-01 19:12:14 +00:00
ed2e11ee2e28af4ea4e2a91d7f81b826ee380104
13 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Zhengchao An
|
2b31bda6d1 |
ci: clear the checkout token in every job that does not push (#5547)
actions/checkout writes its token into .git/config as an http extraheader, where it stays for the rest of the job. That matters more here than usual: pull_request jobs run on self-hosted runners and execute the PR's own build.rs, proc-macros and tests, any of which can read that file. Test and Lint holds actions: write on top of that, so its token can cancel runs and delete the Actions caches the whole pipeline now depends on — it was given persist-credentials: false when that permission was added, and this extends the same treatment to the other 45 checkouts. Two are exempt because the token IS the credential the job needs. helm-package's publish job pushes to rustfs/helm with it; clearing it would break chart publishing. nix-flake-update is exempt pending verification: it passes FLAKE_UPDATE_TOKEN to create-pull-request directly rather than reusing .git/config, so it very likely does not need persistence, but that is unproven and a broken weekly bot is not worth the guess. Both carry a persist-credentials-exempt comment saying which. scripts/security/check_persist_credentials.sh requires every checkout to either clear its credentials or carry that comment, so the decision stays visible in review rather than being an omission nobody notices. Refs: rustfs/backlog#1598, rustfs/backlog#1602 |
||
|
Zhengchao An
|
428fde069d |
ci: drop the docker layer cache and stop installing unused tooling (#5544)
Two cleanups, neither changing what is built or published. docker.yml loses its type=gha layer cache. The image build compiles nothing — it downloads a release zip and runs apk/apt — so the cache could only save the minute or two those take, against a real correctness problem: with RELEASE=latest the binary URL is resolved by curl inside a RUN layer, and the layer key does not include what it resolved to. A rebuild at the same RELEASE value (a dispatch with version=latest, or a re-run of the same version) would hit the old layer and ship the previous release's binary. mode=max also drew on the same repo-wide 10GB Actions cache quota the Rust lanes are contending for. Only RELEASE is passed as a build-arg now; it is the sole one the Dockerfiles declare besides TARGETARCH. BUILDTIME, VERSION, BUILD_TYPE, REVISION and CHANNEL were read by no stage, and BUILDTIME's $(date ...) was a literal string in the YAML block rather than a substitution. The DOCKER_CHANNEL computation that fed CHANNEL evaluated to "release" down every branch and is gone with it. BUILD_DATE and VCS_REF stay unset even though the Dockerfiles declare them: supplying them would change the published image labels. The setup composite stops installing tools its callers do not use. protobuf- compiler comes out of the apt list entirely — setup-protoc installs 34.1 into the tool cache and prepends it to PATH, so the apt build was shadowed on every run and never used; the same duplicate install is removed from the io_uring lane. musl-tools, zip and unzip move behind install-build-packaging-tools, off for the CI and coverage lanes and left on for build.yml, whose native musl leg needs musl-gcc and whose packaging steps need zip. cargo-nextest and the rustfmt/clippy components move behind install-test-tools, off only for build.yml, which runs no tests and no lints; coverage and the nightly replication lane keep them because both invoke nextest. The action's github-token input is deleted along with its 20 call sites. Its runs block never referenced it — setup-protoc uses github.token directly — so it was 20 places handing a token to something that ignored it. Refs: rustfs/backlog#1598, rustfs/backlog#1600, rustfs/backlog#1603 |
||
|
Zhengchao An
|
6c99d4fe22 |
ci: stop the weekly flake.lock bot from running the whole pipeline (#5539)
nix-flake-update opens a PR every Sunday that changes flake.lock and nothing else. flake.lock is consumed only by Nix packaging — cargo never reads it — yet it was in no paths filter, so each of those PRs ran the full Continuous Integration pipeline and the merge then ran Build and Release too. Added to all four lists that have to agree: ci.yml's push and pull_request paths-ignore, build.yml's push paths-ignore, and ci-docs-only.yml's paths. The cron stays. flake.lock still has consumers — anyone running `nix build` or `nix develop` — so stopping the updates would remove the workflow's output, not just its CI cost. Those four lists drifting is a silent failure, so scripts/check_ci_paths_sync.sh now asserts the pair that matters: ci.yml's pull_request paths-ignore must equal ci-docs-only.yml's paths. An entry present only in the first means a PR touching those files triggers neither workflow, nobody reports "Test and Lint" or "Quick Checks", and the PR waits on a required check forever. It also asserts both companion job names still exist, since renaming one produces the same hang. The push list is not compared: no required check is reported for push events. Also in this cleanup: - nix-flake-update's GITHUB_TOKEN drops to contents: read. The branch push and the pull request are both made by update-flake-lock with the FLAKE_UPDATE_TOKEN PAT, so the write scopes were an unused repo-write credential on an unattended weekly job. - build.yml's build_docker dispatch input is documented as advisory. docker.yml triggers on workflow_run and requires the triggering event to be a tag push, so a manual dispatch never reaches it whatever this input says. - The nine disabled workflow files get a banner saying so. Their disabled_manually state lives in GitHub's UI and is invisible when reading the file, which has already misled one audit into treating dead workflows as live. Refs: rustfs/backlog#1598, rustfs/backlog#1603 |
||
|
Zhengchao An
|
e2b2bdcc34 |
ci: bound every job's runtime and stop pasting inputs into shell (#5537)
Three hardening changes with no effect on what any workflow produces. Declare timeout-minutes on the 25 jobs that lacked it. GitHub's default is 360 minutes, and this repository has a history of runners stalling intermittently (#5394) plus a measured 9m57s plain `git checkout` under node-level I/O contention, so one wedged job could hold a runner for six hours out of a pool of roughly 15-21. Budgets follow what the jobs actually do: 10 minutes for echo-only and guard-script jobs, 30 for anything calling the GitHub API, uploading release assets or pushing over the network. scripts/security/check_job_timeouts.sh keeps it that way, checking only jobs that declare runs-on so reusable-workflow callers are not flagged. Pass workflow inputs and workflow_run fields through env instead of `${{ }}` interpolation in run blocks. A git ref name may contain `$(...)` — any string without a space is a legal tag — and interpolation pastes it into the script where bash evaluates it. The worst instance was helm-package's final commit message: it is built from the triggering tag name inside the job that holds the cross-repository push token with rustfs/helm already checked out. Also converted in build.yml, docker.yml and performance-ab.yml; the last is currently disabled, but a disabled workflow can be re-enabled. Not touched: helm-package's `contains(head_branch, '.')` tag test, since GitHub expressions have no regex and this repository's tags carry no `v` prefix, so rewriting the condition would change which builds publish a chart. Give audit.yml a scheduled-failure alert and run it daily. A scheduled cargo-deny failure usually means the dependency tree just matched a newly published RustSec advisory — the most important signal this workflow produces, and until now it was visible only to whoever happened to open the Actions tab. coverage.yml and e2e-replication-nightly.yml already use this ci-8 mechanism. The cron moves from weekly to daily so a new advisory against an unchanged tree surfaces within a day instead of seven; the check list is untouched, since splitting it into a light daily run and a weekly full run would create runs where sources, bans and licenses go unverified. Refs: rustfs/backlog#1598, rustfs/backlog#1602 |
||
|
Zhengchao An
|
4310b55238 |
ci: add schedule-failure-issue composite action and wire nightly alerts (#4671)
Scheduled workflow failures previously went unnoticed (15 consecutive red weekly s3-tests sweeps, silent perf nightly failures). Add a reusable composite action that opens a tracking issue titled "[scheduled-failure] <workflow name>" — or appends a comment to the existing open one (dedupe key = workflow name) — with the run URL, run attempt, and failed job names, labeled "infrastructure". Wire it into the scheduled paths of e2e-s3tests, mint, fuzz (nightly) and performance-ab via a final alert-on-failure job gated by "always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')", with issues:write scoped to that job only. e2e-s3tests and mint previously had no permissions key; they now get workflow-level contents:read, reducing every other job's token. Add a dispatch-only drill workflow that forces a job failure and runs the action through the exact consumer wiring, for end-to-end verification. The ci-7 e2e nightly does not exist yet; it is recorded as a pending consumer in the action README. Refs: rustfs/backlog#1149 (ci-8), rustfs/backlog#1155 |
||
|
Zhengchao An
|
2d1323d2f0 |
ci(fuzz): narrow PR triggers and cover all fuzz targets (#4664)
Re-enable the Fuzz workflow (disabled_manually) with two changes that address the congestion that likely caused it to be disabled: - Narrow the PR trigger paths to fuzz/**, scripts/fuzz/** and fuzz.yml only. The broad crate paths (crates/ecstore|filemeta|utils) queued a ~45min fuzz-build on nearly every PR; coverage of those crates moves to the nightly schedule, which fuzzes against main. - Expand the smoke matrix, nightly matrix, run.sh default set and the fuzz-build staging/upload steps from 3 to all 5 buildable targets: archive_extract, bucket_validation, local_metadata, path_containment, policy_ingress. archive_extract and policy_ingress were previously in no matrix. The two *_storage_api.rs files are `mod` submodules of their parent targets (bucket_validation, path_containment), not standalone bins, so fuzz/Cargo.toml registers exactly 5 fuzz bins. Smoke jobs run one target per parallel matrix job (-max_total_time=60), so wall-clock stays per-target, well under the 15min budget for a fuzz-only PR. A TODO(ci-8) hook marks where scheduled-failure alerting will attach on the nightly job. Refs rustfs/backlog#1149 (ci-9, absorbed sec-13), rustfs/backlog#1155 |
||
|
houseme
|
00536da80c |
refactor(obs): make dial9 telemetry opt-in and actually record events (#4663)
* refactor(obs): make dial9 telemetry opt-in and actually record events
The dial9 Tokio-runtime profiler was disabled by default, yet every build
paid for it, and enabling it produced trace files with no events in them.
Recorded empty traces
---------------------
`build_traced_runtime` called `TracedRuntime::builder()...build(..)`, but dial9
only starts recording in `build_and_start*`. `build` still returns a live guard
whose `is_enabled()` reports true, and still creates and seals segment files —
they just contain a header and no events. It also skipped `with_trace_path`, so
the background worker driving the segment pipeline was never spawned.
Measured on the new smoke example: 310 bytes of bare segment header, against
5640 bytes for the same workload once recording actually starts.
Switch to `with_trace_path(..).build_and_start(..)`.
Cost was unconditional
----------------------
`--cfg tokio_unstable` was a global `[build] rustflags` entry and `rustfs-obs`
depended on `dial9-tokio-telemetry` unconditionally, so all builds depended on
Tokio's non-semver API. Worse, an environment `RUSTFLAGS` replaces (never
appends to) the config-file value, so any caller exporting their own RUSTFLAGS
silently dropped the flag — the long comment in build.yml was a scar from that.
dial9 is now an opt-in feature (`dial9`, plus `dial9-s3` and `dial9-taskdump`),
the global rustflag is gone, and `crates/obs/build.rs` fails the compile if the
feature is on without the flag. Telemetry builds go through `make build-profiling`.
Metrics that could not lie
--------------------------
`rustfs_dial9_{events_total,bytes_written_total,rotations_total,cpu_overhead_percent}`
were hard-coded to zero — a Counter pinned at 0 reads as "nothing happened".
Removed. `rustfs_dial9_enabled` was sourced from the environment, so it read 1
even when the traced runtime failed and the process fell back to a standard
runtime; it is replaced by `rustfs_dial9_supported` (compile-time),
`rustfs_dial9_configured` (intent) and `rustfs_dial9_active_sessions` (reality).
No `writer_healthy` gauge is exported: dial9's `RotatingWriter` can enter its
`Finished` state and stop writing, but exposes no way to observe that, so the
gauge could only ever be hard-coded to 1. Documented as a known gap instead.
Final events were lost
----------------------
The `TelemetryGuard` lived in a `static OnceLock`, which is never dropped, so
buffered events were never flushed at exit. `build_tokio_runtime` now returns
the guard and `run_process` drops it before any exit path.
Also
----
- `disk_usage_bytes` was a `read_dir` + per-file `stat` on the metrics
collection path. It is now sampled by a background task into an atomic.
- `SAMPLING_RATE`/`S3_BUCKET`/`S3_PREFIX` were parsed, warned about, and
discarded. S3 upload is now wired to dial9's `with_s3_uploader` behind
`dial9-s3`; `SAMPLING_RATE` has no upstream equivalent and is removed.
- Wire `with_task_dumps` (async backtraces of stalled tasks), configurable via
`RUSTFS_RUNTIME_DIAL9_TASK_DUMP_{ENABLED,IDLE_THRESHOLD_MS}`.
- Split `telemetry/dial9.rs` into `config`/`state`/`enabled`/`disabled`; the
stub keeps the public API identical so callers need no `#[cfg]`.
- Drop four print-only examples and the manual test bin that exercised the
removed `init_session` scaffolding.
Verified: cargo check/clippy/test across default, `dial9`, and `dial9-s3`;
build.rs correctly rejects `dial9` without `--cfg tokio_unstable`;
`make pre-commit` passes.
Co-Authored-By: heihutu <heihutu@gmail.com>
* docs(obs): document dial9 as an on-demand profiler
scripts/run.sh advertised a `SAMPLING_RATE` knob that was never passed to dial9,
and claimed "CPU overhead < 5% (with sampling rate 1.0)" and "lower values reduce
CPU overhead" on the strength of it. The knob is gone; the guidance built on it
had to go too.
Replace it with what is actually true: dial9 needs a `make build-profiling`
binary, its disk budget evicts oldest-first (so a high poll rate can overwrite
the incident you are chasing), and it cannot be toggled without a restart.
Add docs/operations/dial9-runtime-profiling.md covering the build variants, an
investigation walkthrough, the configuration table, how to read the three
supported/configured/active_sessions gauges against each other, and the upstream
gap that makes writer death only indirectly observable.
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(obs): add a dial9 smoke example that proves events are recorded
The bug this guards against is invisible to every existing signal: with
`build` instead of `build_and_start`, dial9 creates the trace file, seals
segments, and reports `TelemetryGuard::is_enabled() == true` — it simply
records no events. Only the segment's byte count tells the two apart.
Measured on this workload: 5640 bytes when recording, 310 bytes (a bare
segment header) when not. The example asserts >= 2048 bytes, and was verified
to fail with the `build` call restored.
Also correct the comment on the `is_enabled` check in `finish_traced_runtime`.
It claimed to catch "recording silently off"; it does not. It only rejects the
inert guard a lenient config yields after a build failure. Recording is
guaranteed by `build_and_start`, not by that check.
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(rustfs): accept Unsupported runtime telemetry capability
A binary built without the `dial9` feature now reports the runtime-telemetry
capability as `Unsupported` rather than `Disabled`. The distinction matters to
operators: `Disabled` implies the capability can be switched on by setting an
environment variable, which is not true here — telemetry needs a rebuild.
Widen the assertion and pin the new semantics: when `dial9::is_supported()` is
false, the state must be exactly `Unsupported`.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(obs): drop the dial9-s3 feature, its TLS stack is vulnerable
CI's Dependency Review and `cargo deny` both reject the branch: dial9's
`worker-s3` feature depends on aws-sdk-s3-transfer-manager 0.1.3, which pins
aws-smithy-http-client onto hyper-rustls 0.24 and rustls-webpki 0.101.7. That
webpki carries RUSTSEC-2026-0098, -0099 and -0104.
0.1.3 is the latest release of the transfer manager, and 1.2.0 the latest of the
smithy client, so there is nothing to upgrade to. Cargo's feature unification can
add features but cannot drop a transitive dependency, so it cannot be worked
around from here either — the rest of the workspace already resolves to the safe
rustls-webpki 0.103 / hyper-rustls 0.27.
Remove the `dial9-s3` feature and the `with_s3_uploader` wiring. The two S3
environment variables stay parsed and warned about, now naming the real reason
rather than a missing build feature. Trace segments are collected from the output
directory instead. Tracked as D9-14 in rustfs/backlog#1157.
With this, Cargo.lock is byte-identical to main: the PR no longer touches the
dependency graph at all.
Also correct the `dial9-taskdump` documentation. It claimed the feature "compiles
to a no-op elsewhere"; in fact `tokio/taskdump` raises a `compile_error!` on any
target other than linux/{aarch64,x86,x86_64}. Verified by trying to build it on
macOS, which is how the claim was found to be wrong.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
|
||
|
Zhengchao An
|
5f1759eb3c | ci: cancel PR workflows on close (#4323) | ||
|
majinghe
|
4542d4060f | ci: replace ubicloud runner with self host runner on k8s (#4245) | ||
|
Zhengchao An
|
7ae9697d42 | docs: close final architecture audit gaps (#3966) | ||
|
houseme
|
0a00d8d500 |
perf(server): lighten internode data-plane stack (#3735)
* refactor(server): split internode dispatch scaffold
* test(server): cover internode dispatch prefix split
* refactor(server): name internode stack boundaries
* perf(server): skip internode request logging layer
* perf(server): skip internode trace layer
* perf(server): use lite internode request context
* feat(metrics): track internode rpc duration
* feat(ecstore): add put object stage summary logs
* test(metrics): update internode descriptor expectations
* fix(server): tighten internode path matching
* fix(pr): address review follow-up comments
* style(ecstore): simplify commit tail duration field
* refactor(ecstore): group put stage summary fields
* refactor(ecstore): inline put stage summary log
* fix(s3): return storage class for object attributes
* merge: sync latest main and resolve object attributes conflict
* fmt
* fix(server): remove duplicate rpc imports
* build(deps): bump memmap2 for RUSTSEC-2026-0186
* fix(s3select): align object_store with datafusion
* chore(deps): prune workspace dependencies
* perf(fuzz): optimize CI runtime with build/run split and matrix parallelization
Separate fuzz harness compilation from execution to eliminate redundant
builds across targets. Introduce matrix-based parallel execution for
PR smoke and nightly fuzz jobs.
Changes:
- Split CI workflow into `fuzz-build` (compile once) and matrix run jobs
(`pr-fuzz-smoke`, `nightly-fuzz-corpus`) that execute targets in parallel
- Add `BUILD_ONLY` mode to run_ci_targets.sh / run_nightly_targets.sh
- Add run_single_target.sh for matrix jobs (no build phase)
- Optimize `local_metadata` fuzz target: reduce prefix iterations from
8-10 (4 functions each) to 5 critical prefixes (parser-only), cutting
per-iteration cost by ~3-5x
- Move archive path validation (`validate_extract_relative_path`,
`normalize_extract_entry_key`) from `rustfs` to `rustfs-utils::path`,
eliminating `rustfs` binary crate dependency from fuzz harness
- Remove `rustfs` from fuzz/Cargo.toml (drops significant transitive deps)
- Add unit tests for archive path validation in rustfs-utils
- Update fuzz/README.md with new workflow and script documentation
Expected CI improvement: PR smoke wall-clock from ~120min (frequent
timeout) to ~40min; nightly from ~180min to ~60min.
* refactor(fuzz): consolidate scripts and fix prefix test alignment
Replace three duplicated shell scripts (run_ci_targets.sh,
run_nightly_targets.sh, run_single_target.sh) with a single
parameterized run.sh that supports BUILD_ONLY, SKIP_BUILD, and
MAX_TOTAL_TIME environment variables.
Fix local_metadata fuzz target prefix testing: replace always-true
'len > 0' guard with lengths aligned to xl.meta binary layout
(4/5/8/12 bytes for magic+version+header fields). Remove redundant
empty-slice test.
Hoist RUSTFLAGS to workflow top-level env to eliminate per-job
duplication. Update README with unified script documentation.
Net: -118 lines, zero functionality loss.
* perf(fuzz): optimize CI runtime with build/run split and matrix parallelization
Restructure fuzz CI workflow to eliminate redundant compilation and
run targets in parallel via matrix strategy.
Workflow changes:
- Split into fuzz-build (compile once) and matrix run jobs
- PR smoke: 3 targets parallel, 60s each, timeout 30min (was 120min)
- Nightly: 3 targets parallel, 300s each, timeout 60min (was 180min)
- Pass compiled harness via actions/artifact between jobs
- Hoist RUSTFLAGS to workflow top-level env
Script consolidation:
- Replace 3 duplicated scripts with single parameterized run.sh
- Supports BUILD_ONLY, SKIP_BUILD, MAX_TOTAL_TIME env vars
Target optimizations:
- Remove rustfs binary crate from fuzz dependencies (was pulling
979 transitive deps); move archive path validation to rustfs-utils
- Optimize local_metadata: reduce prefix iterations from 8-10x4
calls to 5 prefixes with parser-only (no decompress), aligned
with xl.meta binary layout (4/5/8/12 bytes)
- Add unit tests for archive path validation in rustfs-utils
- Update fuzz/README.md with unified script documentation
Expected: PR smoke wall-clock from ~120min (frequent timeout)
to ~40min; nightly from ~180min to ~60min.
* fix(rpc): resolve internode metrics via app context
* fmt
* ci: speed up fuzz smoke artifact restore
---------
Signed-off-by: houseme <housemecn@gmail.com>
|
||
|
houseme
|
9e6d0c7292 | chore(deps): refresh workflow actions and crate pins (#3696) | ||
|
houseme
|
32aca6d835 | feat(fuzz): scaffold cargo-fuzz harness for smoke targets (#3537) |