mirror of https://github.com/rustfs/rustfs.git synced 2026-08-16 01:48:21 +00:00

Files

T

唐小鸭 e11ce2f132 fix(site-replication): route every state RMW through the locked transaction (#6097 )

* fix(site-replication): route every state RMW through the locked transaction

P1-15 PR2 (rustfs/backlog#1796, batch B2 of rustfs/backlog#1675), the
follow-up promised by rustfs/rustfs#5882.

PR1 left ~26 read-modify-write call sites on
config/site-replication/state.json in the pre-transaction shape: a
process-local mutex around load / mutate / save, each IO taking its own
object lock. Nothing held a distributed lock across the whole sequence, so
two nodes of one site still lost each other's updates, and the transitional
mutex kept the old shape available to copy.

Every remaining RMW now runs inside update_site_replication_state;
read-only sites use load_site_replication_state, whose object read comes
with the object-level read lock. SITE_REPLICATION_STATE_LOCK and its owner
helper are gone, together with their architecture-guard allowlist entry and
inventory row.

The multi-stage flows (add / edit / peer join / peer edit / remove / rotate)
keep their updated_at and pending-id CAS, but the CAS now runs inside the
transaction that writes, against the state that transaction loaded. Peer
probes, IAM work and fan-outs run between transactions and hold no lock at
all — the add no longer blocks every writer of the site across its peer join
round trips, and it re-checks the precondition right after the capability
probes so the common race is rejected before any IAM write or remote join.
When the add's commit CAS still fails, the error says the peers may already
be joined and that re-running the add reconverges. The add adopts only the
fields it computed (exhaustive destructure — adding a state field is a
compile error until classified); fields owned by writers that do not bump
updated_at keep their freshly loaded values.

Ordering of peer-edit deliveries now rests on the generation fence landed in
PR1 rather than on a guard that could never order two nodes: the add's
finalize fan-out carries the generation allocated in its commit. An accepted
peer join PRESERVES the applied-generation high-water marks — join fan-outs
are routine (adds and rotations both deliver SRPeerJoin to existing peers),
so wiping them would let stalled older edits land after any join; the
unilateral-removal rejoin misfence that a wipe would have patched is
pre-existing since the fence landed and needs an epoch in the fence instead.

The rotation handler now takes the lifecycle guard: the background
service-account reconciler runs its repair under a lifecycle try-acquire,
and its pending-rotation precheck is only sound if a rotation cannot start
mid-repair — an exclusion the removed process mutex used to provide as a
side effect.

update_site_replication_state_when_changed adds persist-or-skip so ack
markers and pending-clearing paths stop rewriting the object on a miss —
load-bearing, because the shared persist helper clears the whole object for
a ≤1-peer pending-free state — and save_site_replication_state is now
cfg(test): the pre-P1-15 shape can no longer be written in production code.

No on-disk format change.

Verification: cargo nextest run -p rustfs -E
'test(/admin::handlers::site_replication::/)' (181 passed); site-replication
dual/three-node e2e (13 passed); cargo clippy -p rustfs --all-targets -D
warnings; make pre-commit. Mutation checks: dropping the state-object lock
from the boundary reds the separate-node concurrency tests; flipping a
persist-or-skip miss to a persist reds
test_missed_pending_clear_must_not_rewrite_the_state_object. Reviewed by
three independent adversarial passes (correctness/concurrency,
security/compatibility, simplicity/test-coverage); their confirmed findings
are folded in.

* fix(site-replication): serialize peer-join admission around its IAM write

Review follow-up (overtrue): two joins accepted by the same node could
interleave as "A checks a stale snapshot and pauses reading its body, B
applies secret B and commits, A resumes, overwrites IAM with secret A, and
A's commit is refused as superseded" — the persisted state advertised B's
contract while IAM only accepted A's secret, failing every peer
control-plane call. The pre-P1-15 process mutex serialized same-node joins
end to end; removing it dropped that exclusion.

admit_peer_join now runs the staleness check, the IAM upsert and the state
commit under the lifecycle guard, with the authoritative pre-check taken
against a load under that guard BEFORE IAM changes anything. The closing
transaction still re-checks staleness: the guard is process-local (exactly
as far as the old mutex reached) and the state-object lock arbitrates joins
accepted by different nodes. The body is fully read before the guard so a
stalling sender cannot block add/remove/rotate/reconciler.

The IAM step is injected, and the gated-body regression test reproduces the
review's ordering: join A is held mid-IAM while a newer join B arrives; B
must wait at the guard, and both IAM order and the final persisted state end
on B. Mutation-verified: removing the lifecycle guard from admit_peer_join
turns the test red.

Verification: cargo nextest run -p rustfs -E
'test(/admin::handlers::site_replication::/)' (182 passed);
site-replication dual/three-node e2e (13 passed); cargo clippy -p rustfs
--all-targets -D warnings; make pre-commit.

* fix(site-replication): fence peer-join admission across nodes

Review follow-up (overtrue, round 2): the lifecycle guard only serializes
joins within one process. Node A could pass the staleness check for an
older T1, node B write secret B to IAM and commit a newer T2, and node A
then overwrite IAM with secret A while its own state commit is refused as
superseded — state advertising T2's contract while IAM only accepts A's
secret.

The admission (staleness check -> IAM upsert -> state commit) now also runs
under a distributed join-admission lock, a namespace-lock key with no
backing object, following the repair execution lock's pattern — including
its nesting of config-object locks (admission -> state), and delegating
crash safety to the lock subsystem's lease expiry instead of a hand-rolled
TTL. The staleness check runs against a load taken inside the lock, before
IAM changes anything, so a superseded join exits without touching IAM. The
closing transaction keeps its re-check for defence in depth and for
old-version nodes that do not take the admission lock during a rolling
upgrade (that mixed-version window keeps today's behavior and closes when
the upgrade completes).

admit_peer_join_across_nodes is the admission minus the process-local
lifecycle guard — exactly what a second node runs — and the new
separate-nodes regression test drives it directly with join A gated
mid-IAM: join B must wait at the distributed lock, and both the IAM write
order and the final persisted state end on B. Mutation-verified: removing
the admission lock turns the test red while the same-node test (which
drives the full admit_peer_join) stays green.

Verification: cargo nextest run -p rustfs -E
'test(/admin::handlers::site_replication::/)' (183 passed);
site-replication dual/three-node e2e (13 passed); cargo clippy -p rustfs
--all-targets -D warnings; make pre-commit.

2026-08-14 22:13:37 +08:00

README.md

scripts/ index

Authoritative inventory of everything under scripts/ (backlog#1153 infra-13). One row per top-level entry; subdirectories get one row each and keep their own READMEs. The test-layer map that ties the major runners together lives in docs/testing/README.md.

Statuses

ci-gate — wired into CI, release, or image-build pipelines. Do not move, rename, or change flags without updating the wiring listed in the last column.
dev-tool — run by humans: local dev loops, runbooks, validation harnesses. Kept working, not wired into CI.
archived — one-shot scripts whose investigation/issue is finished, moved to scripts/archive/. Unmaintained reference material: never wire into CI, and expect bit-rot. To resurrect one, move it back and give it an index row here.

Adding a script? Add an index row in the same PR. Issue-scoped scripts (run_issueNNN_*, validate_issue_NNN_*) are expected to be archived when their issue closes.

Repository & CI gates

Entry	Status	Purpose	Wiring / docs
`check_architecture_migration_rules.sh`	ci-gate	Architecture-boundary anti-regression guard	ci.yml Quick Checks; `make pre-commit`
`check_body_cache_whitelist.sh`	ci-gate	Keeps the app-layer body-cache eligibility gate fail-closed	ci.yml Quick Checks
`check_doc_paths.sh`	ci-gate	Fails when instruction/architecture docs reference repo paths that no longer exist	`make pre-commit` / `pre-pr`
`check_extension_schema_boundaries.sh`	ci-gate	Extension-schema crate boundary guard	ci.yml Quick Checks; `make pre-commit`
`check_layer_dependencies.sh`	ci-gate	Crate-layering DAG guard (reads `layer-dependency-baseline.txt`)	ci.yml Quick Checks
`check_logging_guardrails.sh`	ci-gate	Blocks legacy logging patterns from returning	`make pre-commit` / `pre-pr`
`check_migration_gate_count.sh`	ci-gate	Migration-critical test gate with committed count floor (`.config/migration-gate-floor.txt`)	ci.yml Test and Lint; `docs/testing/README.md`
`check_no_planning_docs.sh`	ci-gate	Blocks committed planning-type documents	ci.yml Quick Checks; `make pre-commit`
`check_no_tokio_io_uring.sh`	ci-gate	Keeps tokio's io-uring backend disabled	ci.yml Quick Checks
`check_s3s_footprint.sh`	ci-gate	Lower-only ratchet freezing the direct s3s surface ahead of the s3gate migration	ci.yml Quick Checks; `make pre-commit`
`check_unsafe_code_allowances.sh`	ci-gate	Unsafe-code allowance ledger guard	ci.yml Quick Checks
`layer-dependency-baseline.txt`	ci-gate (data)	Committed baseline consumed by `check_layer_dependencies.sh`	arch-checks skill
`static.sh`	ci-gate	Static-build helper executed inside image builds	`Dockerfile.source`, `Dockerfile.decommission-local`
`helm_chart_version.sh`	ci-gate	Keeps the Helm chart version in sync with the release	helm-package.yml
`test_helm_templates.sh`	ci-gate	Helm template rendering test	helm-package.yml

Test & e2e runners

Entry	Status	Purpose	Wiring / docs
`e2e-run.sh`	ci-gate	Boots a rustfs server and runs the `s3s-e2e` black-box conformance tool against it	ci.yml `e2e-tests` jobs; `docs/testing/README.md`
`run_ecstore_validation_suite.sh`	dev-tool	ecstore black-box validation suite (`quick`/`full`/`destructive`/`fuzz` profiles)	`docs/testing/README.md`, `docs/testing/ecstore-validation-suite-design.md`
`run_e2e_tests.sh`	dev-tool	Local `e2e_test` crate runner (starts a server, applies filters, cleans up)	`crates/e2e_test/README.md`
`run.sh`	dev-tool	Local rustfs startup wrapper	`make e2e-server`; Justfile
`run.ps1`	dev-tool	Windows counterpart of `run.sh`	—
`probe.sh`	dev-tool	Probe-style e2e run	`make probe-e2e`
`run_scanner_validation_harness.sh`	dev-tool	Scanner validation harness	`docs/operations/scanner-benchmark-runbook.md`
`test_scanner_validation_harness.sh`	dev-tool	Self-test for the scanner validation harness	—
`test_build_rustfs_options.sh`	dev-tool	Shell test for rustfs build-option wiring	`make test` (script-tests)
`test_entrypoint_credentials.sh`	dev-tool	Container entrypoint credential-handling test	`make test` (script-tests)
`test_helm_chart_version.sh`	dev-tool	Test for `helm_chart_version.sh`	—
`windows-sftp-listener-smoke.sh`	dev-tool	Confirms `rustfs.exe --features sftp` binds an SFTP listener on Windows	—

Benchmark & performance harnesses

Entry	Status	Purpose	Wiring / docs
`run_hotpath_warp_ab.sh`	ci-gate	Linux warp A/B rig for the hotpath series	performance-ab.yml (scheduled); `docs/operations/hotpath-warp-ab-runbook.md`
`hotpath_warp_ab_gate.sh`	dev-tool	Relative-budget gate evaluated over the warp A/B results	used by `run_hotpath_warp_ab.sh`; hotpath runbook
`run_internode_grpc_ab_bench.sh`	dev-tool	One-click A/B driver for the internode gRPC optimization stages	`docs/operations/internode-grpc-benchmark-runbook.md`
`run_internode_transport_baseline.sh`	dev-tool	Internode transport baseline runner	internode runbook; `crates/io-metrics/README.md`
`run_four_node_cluster_failover_bench.sh`	dev-tool	Four-node cluster failover benchmark	`docker/compose/README.md`; internode runbook
`run_object_batch_bench.sh`	dev-tool	Batch object benchmark runner (warp/s3bench)	internode + scanner runbooks
`run_object_batch_bench_enhanced.sh`	dev-tool	Enhanced batch benchmark runner; hub used by the smoke rigs	hotpath runbook
`run_pinned_paired_abba_bench.sh`	dev-tool	Pinned RustFS/MinIO paired ABBA benchmark orchestrator for backlog#1432	`test_pinned_paired_abba_bench.sh`
`run_get_codec_streaming_smoke.sh`	dev-tool	Local GET benchmark harness for the codec streaming read path	`docs/testing/ecstore-validation-suite-design.md`
`run_get_1mib_abba_stage_metrics.sh`	dev-tool	Exact-1MiB isolated-host GET ABBA/stage-metrics harness for backlog#1434	`test_get_1mib_abba_stage_metrics.sh`
`run_gt1g_get_http_matrix.sh`	dev-tool	>1 GiB GET HTTP matrix	`docs/testing/ecstore-validation-suite-design.md`
`run_gt1g_multipart_put_matrix.sh`	dev-tool	>1 GiB multipart PUT matrix	`docs/testing/ecstore-validation-suite-design.md`
`sample_remote_rustfs_rss.sh`	dev-tool	Remote RustFS PID CPU/RSS TSV sampler for hotpath profiling runs	`test_sample_remote_rustfs_rss.sh`; backlog#1647
`summarize_samply_profile_symbols.py`	dev-tool	Offline samply `profile.json.gz` + `.syms.json` function-level hotpath summarizer	`test_summarize_samply_profile_symbols.py`; backlog#1647
`run_scanner_benchmarks.sh`	dev-tool (disposition pending)	Scanner performance benchmark runner. Contains a hardcoded stale path; disposition owned by backlog perf-10 — do not fix, move, or delete it here	—

Local development & operations

Entry	Status	Purpose	Wiring / docs
`dev_clear.sh`	dev-tool	Local dev cleanup. `scripts/dev_*.sh` is a CI paths-filter glob — keep the naming	ci.yml/build.yml paths filters
`dev_deploy.sh`	dev-tool	Copy a built binary to dev servers	`make deploy` (`.config/make/deploy.mak`); Justfile
`dev_rustfs.sh`	dev-tool	Local dev run loop	—
`dev_rustfs.env`	dev-tool (data)	Env presets for the dev scripts	—
`restart_local_single_node_multidisk_rustfs.sh`	dev-tool	Restart a local single-node multi-disk instance	—
`inspect_dashboard.sh`	dev-tool	Sanity-checks the Grafana dashboard JSON	`.docker/observability`
`notify.sh`	dev-tool	Starts a local webhook receiver for notify-target development	—
`manual_transition_debug.sh`	dev-tool	Log/metrics helper for manual transition troubleshooting	—
`manual_transition_journal_audit.sh`	dev-tool	Journal + metrics + log audit for manual transition jobs	—
`manual_transition_mixed_rollout_matrix.sh`	dev-tool	Matrix generator for mixed-version rollout phases	—
`manual_transition_mixed_rollout_runbook.sh`	dev-tool	Reusable mixed-version rollout runbook generator (external run)	—
`manual_transition_mixed_version_docker_harness.sh`	dev-tool	Dedicated #1508 Docker harness for old/new manual-transition rollout evidence with strict/baseline/blocked result classification	`test_manual_transition_runbooks.sh`
`monitor_manual_transition_ci.sh`	dev-tool	CI workflow/status watcher for manual transition follow-up monitoring	—
`manual_transition_soak_matrix.sh`	dev-tool	Matrix generator for nightly stress windows	—
`manual_transition_nightly_stress_runbook.sh`	dev-tool	Nightly stress entrypoint with failure snapshot templates	—
`install-flatc.sh`	dev-tool	Local flatc installer (macOS)	—
`install-protoc.sh`	dev-tool	Local protoc installer (macOS/Linux)	—
`makefile-header.sh`	dev-tool	Generates the `## —— section ——` header lines used in `.config/make/*.mak`	—
`tls_gen.md`	dev-tool (doc)	Notes on generating local TLS certificates	—

Subdirectories

Entry	Status	Purpose	Wiring / docs
`fuzz/`	ci-gate	Unified cargo-fuzz runner and helpers for the `fuzz/` sub-workspace	fuzz.yml; `fuzz/README.md`
`s3-tests/`	ci-gate	ceph/s3-tests compatibility harness (allow-lists, patches, report tooling)	ci.yml; e2e-s3tests.yml; `scripts/s3-tests/README.md`
`security/`	ci-gate	Workflow-pin enforcement and release supply-chain asset generation	audit.yml; build.yml
`table-catalog/`	dev-tool	S3-Tables / pyiceberg validation suite	`docs/architecture/s3-tables-support-matrix.md`
`test/`	dev-tool	Manual operational validation runbooks (decommission, tier lifecycle), paired `.sh` + `.md`	—
`archive/`	archived	Retired one-shot scripts (see below)	—

Archived (scripts/archive/)

Moved 2026-07 (backlog#1153 infra-13) after a whole-tree reference census: each entry had zero references from CI, Makefiles, docs, or code — or was referenced only by other scripts in this same archived set. Reasons:

Entry	Was
`validate_issue_785_list_objects.sh`	One-shot issue validation (list-objects series)
`validate_issue_786_list_objects.sh`	One-shot issue validation (list-objects series)
`validate_issue_787_list_quorum.sh`	One-shot issue validation (list-quorum)
`validate_issue_841_list_objects_observability.sh`	One-shot issue validation (list observability)
`validate_issue_1365_docker.sh`	One-shot issue validation (docker repro)
`validate_issue_2723_site_replication.sh`	One-shot issue validation (site replication)
`validate_issue_3031_docker.sh`	One-shot issue validation (docker repro)
`run_issue712_deeper_zero_copy_put_with_capture.sh`	One-shot perf capture for backlog#712
`run_issue797_local_4node_16disk_ab.sh`	One-shot 4-node/16-disk A/B for backlog#797
`run_issue_2573_acceptance.sh`	One-shot acceptance run for issue #2573
`run_issue_2941_perf_capture.sh`	One-shot perf capture for issue #2941
`run_put_large_stage_breakdown.sh`	backlog#706 large-PUT stage breakdown (family)
`run_put_large_stage_breakdown_with_capture.sh`	backlog#706 one-shot wrapper (family)
`run_put_large_tuning_matrix.sh`	backlog#706 tuning matrix (family)
`collect_put_large_stage_breakdown_artifacts.sh`	backlog#706 artifact collector (family)
`analyze_put_service_metrics_deltas.py`	backlog#706 metrics-delta analyzer (family)
`README-stress-test.md`	GET-optimization one-shot suite doc
`stress-test-get-optimization.sh`	GET-optimization one-shot stress test
`quick-validate-get-optimization.sh`	GET-optimization one-shot validation
`benchmark-sf-optimization.sh`	GET-optimization one-shot benchmark
`prepare_gt1g_get_test_objects.sh`	>1 GiB GET investigation one-shot fixture prep
`run_gt1g_multipart_put_server_path_focus.sh`	>1 GiB PUT investigation one-shot focus run
`run_get_metrics_gate_smoke.sh`	One-shot GET metrics-gate smoke
`run_listobjects_verified_bench.sh`	One-shot verified list-objects bench
`run_object_batch_bench_abc.sh`	One-shot capacity/object profile A/B/C controller
`run_object_data_cache_bench.sh`	One-shot GET bench for the object-data-cache rollout gate
`setup-test-binaries.sh`	One-shot Docker-build test binary fixture
`test.sh`	Ancient manual `mc` bucket smoke scratchpad
`test_policy.json`	Orphaned IAM policy fixture (hardcoded test bucket)