mirror of
https://github.com/rustfs/rustfs.git
synced 2026-11-20 23:15:23 +00:00
9e57f83885
* perf(ecstore): reduce allocations for bounded bitrot frames * test(s3): cover dynamic block layouts over HTTP * perf: measure quantized range read shape * chore: checkpoint quantized read metrics draft Save work in progress for collaboration. Counts omit retired readers and transient or cancelled hedges. Frame semantics, async recorder coverage and disabled-metrics overhead remain unresolved; these counters are not ready for performance conclusions. Validation: rustfmt and ecstore check passed; io-metrics 120, quantized 4 and hashed-read 3 tests passed. No new live or performance validation. * perf(ecstore): reduce dynamic layout read overhead * fix(ecstore): avoid prefetching conservative short range tails * perf(ecstore): compact deferred shard reader state * perf(ecstore): include shard checksums in file buffer cap * fix(ecstore): preserve bounded quantum frames during heal * perf(ecstore): reuse owned quantized frame batches * perf(ecstore): batch q reads and retain shared copy framing * fix(ecstore): preserve typed quantized read errors * perf(ecstore): avoid zero-fill and serial copies in local batch reads The local batch reader zero-initialized every window at a nonzero offset before reading it, and copied every shard serially inside one blocking task. Two-segment q-window range reads (cross-shard, cross-block) pay both costs twice per request. Read each window with a positioned read into uninitialized Vec capacity, and dispatch each shard separately when every window is at least 512 KiB, where copies dominate blocking-pool round trips. Smaller windows keep one coalesced dispatch. * perf(ecstore): keep readahead pipelined for local batch windows A single window-sized pread on cold pages waits on each kernel readahead chunk in turn. Seek to the window start and fill the buffer with read_to_end over a length-limited reader instead: it still reads into uninitialized spare capacity, but its growing sequential reads keep readahead pipelined. Warm windows cost the same. * Revert "perf(ecstore): keep readahead pipelined for local batch windows" This reverts commitc252fca525. On the ESSD-backed cloud footprint, the growing read_to_end reads were slower than one window-sized positioned read for cold windows at nonzero offsets (cross-block -27% on 4 MiB layouts, cross-shard -21% on 64 KiB layouts versus the previous batch reader). The local benchmark that motivated the change ran on a WSL virtual disk and did not represent the target storage. Keep the single positioned read into spare capacity and the per-shard dispatch for large windows. * perf(ecstore): choose the batch read method by window size Measured on ESSD-backed cloud storage, a cold window of about 1 MiB is fastest as one positioned read, which completes as a single device request, while windows of 2 MiB and more are faster with growing sequential reads that keep readahead pipelined (one large read waits on each readahead chunk in turn). Use one positioned read up to 1.5 MiB and a seek plus length-limited read_to_end above it. Both paths read into uninitialized spare capacity; per-shard dispatch for large windows is unchanged. * perf(ecstore): read local batch windows with an exact readahead hint Kernel readahead sizes its first request from the read size. A cold 16 KiB window at offset 0 pulled 64-128 KiB, and a single 2 MiB read was split into serial readahead chunks that cached almost twice the window, which is why neither the single positioned read nor growing sequential reads won for every window size on block storage. Announce each window with POSIX_FADV_WILLNEED, which submits exactly the window at once, and then fill the exact spare capacity with positioned reads. This replaces the size-based choice between the two read methods. The hint is advisory and its failure does not fail the read. A residency test checks that cold reads cache only their window; it failed with the hint disabled (a 5-page window cached 32 pages). * perf(ecstore): use half-shard read quantum at 1 MiB EC layout * revert(release): defer certification rollout for 1.0.5 (#9254) Reverte5563e6004only on release/1.0.5. Its trusted verifier workflow and immutable-candidate integration are not available yet. Restore the previous publication path under the approved release policy; retain the completed product regression and preview acceptance requirements. * fix(ecstore): preserve legacy full-shard framing for default 1 MiB layout * perf(ecstore): enable direct-memory GET and coalesce cross-shard range reads for bounded-quantum blocks * chore(ecstore): remove unused schedule_bitrot_reader_task wrapper * fix(ecstore): run delete-recovery discovery off the metadata lane Bucket-wide delete-recovery discovery starts with the server and walks the whole bucket tree in pages of up to 4096 entries, one lstat per entry. Each page ran on the per-disk namespace queue's foreground lane, which has a single worker and also serves GET metadata reads. On a cold tree a page holds that worker for hundreds of milliseconds, so every GET on the disk waits behind it until the walk finishes. Run the page on the queue's sync lane instead, as the delete-transaction inspection already does for the same reason. The namespace lease and the page bounds are unchanged. The regression test occupies the foreground lane and requires a discovery page to complete within ten seconds; it times out without this change. * test(ecstore): align read_appending rollback assertions on error * fix: align multipart and EC read paths with current workspace APIs --------- Co-authored-by: Xiaoyang Han <lux1an@qq.com> Co-authored-by: Chris An <anzhengchao@gmail.com>
scripts/ index
Authoritative inventory of everything under scripts/ (backlog#1153
infra-13). One row per top-level entry; subdirectories get one row each and
keep their own READMEs. The test-layer map that ties the major runners
together lives in docs/testing/README.md.
Statuses
- ci-gate — wired into CI, release, or image-build pipelines. Do not move, rename, or change flags without updating the wiring listed in the last column.
- dev-tool — run by humans: local dev loops, runbooks, validation harnesses. Kept working, not wired into CI.
- archived — one-shot scripts whose investigation/issue is finished, moved
to
scripts/archive/. Unmaintained reference material: never wire into CI, and expect bit-rot. To resurrect one, move it back and give it an index row here.
Adding a script? Add an index row in the same PR. Issue-scoped scripts
(run_issueNNN_*, validate_issue_NNN_*) are expected to be archived when
their issue closes.
Repository & CI gates
| Entry | Status | Purpose | Wiring / docs |
|---|---|---|---|
check_architecture_migration_rules.sh |
ci-gate | Architecture-boundary anti-regression guard | ci.yml Quick Checks; make pre-commit |
check_body_cache_whitelist.sh |
ci-gate | Keeps the app-layer body-cache eligibility gate fail-closed | ci.yml Quick Checks |
check_doc_paths.sh |
ci-gate | Fails when instruction/architecture docs reference repo paths that no longer exist | make pre-commit / pre-pr |
check_embedded_secrets.sh |
ci-gate | Repo-wide scan blocking committed private key material and provider credential literals | ci.yml Quick Checks; make pre-commit / pre-pr |
check_extension_schema_boundaries.sh |
ci-gate | Extension-schema crate boundary guard | ci.yml Quick Checks; make pre-commit |
check_layer_dependencies.sh |
ci-gate | Crate-layering DAG guard (reads layer-dependency-baseline.txt) |
ci.yml Quick Checks |
check_logging_guardrails.sh |
ci-gate | Blocks legacy logging patterns from returning | make pre-commit / pre-pr |
download_rc.sh |
ci-gate | Downloads and verifies the RustFS CLI release binary for container builds | Dockerfile, Dockerfile.glibc, Dockerfile.source, Dockerfile.decommission-local |
check_migration_gate_count.sh |
ci-gate | Migration-critical test gate with committed count floor (.config/migration-gate-floor.txt) |
ci.yml Test and Lint; docs/testing/README.md |
check_no_planning_docs.sh |
ci-gate | Blocks committed planning-type documents | ci.yml Quick Checks; make pre-commit |
check_no_tokio_io_uring.sh |
ci-gate | Keeps tokio's io-uring backend disabled | ci.yml Quick Checks |
check_s3s_footprint.sh |
ci-gate | Lower-only s3s surface ratchet: a pull request (--base <sha>, from Quick Checks) may not rise above its base revision; the baseline is a ceiling everywhere (a drop passes and prints the line to lower) ahead of the gateway migration (reads .config/s3s-footprint-baseline.txt; s3_error_lines leaves out lines whose file imports the macro from rustfs-s3-types, rule in the script header); --mode allowlist fails on any Rust file referencing s3s outside .config/s3s-edge-allowlist.txt, --dry-run lists them |
ci.yml Quick Checks; make pre-commit; make script-tests (--self-test) |
check_test_retirements.py |
ci-gate | Every test a migration slice retires (.config/test-retirements/<slice>.txt, rustfs/backlog#2765) has five fields, is listed once, and names a replacement that exists (a test function, a differential step or an e2e function) |
ci.yml Quick Checks; make pre-commit; make script-tests (--self-test) |
check_test_count_ratchet.sh |
ci-gate | Per-crate test-count floor and #[ignore] ceiling against .config/test-count-baseline.txt; no crate may fall below its tests row or rise above its ignored row (extra tests and new crates pass with a note) |
ci.yml Quick Checks; make pre-commit |
guard_source_set.sh |
ci-gate (library) | Sourced helpers for text guards that must survive a mechanical file split (rustfs/backlog#2765): guard_source_set resolves a required file plus the split directory (a missing required file fails), module_relative_rg rewrites a super:: rule for each file's module depth |
Sourced by check_architecture_migration_rules.sh and check_logging_guardrails.sh; contract tests test_guard_source_set.sh; end-to-end split probe test_split_guard_rules.sh (both make script-tests) |
check_unsafe_code_allowances.sh |
ci-gate | Unsafe-code allowance ledger guard | ci.yml Quick Checks |
layer-dependency-baseline.txt |
ci-gate (data) | Committed baseline consumed by check_layer_dependencies.sh |
Architecture guard troubleshooting |
static.sh |
ci-gate | Static-build helper executed inside image builds | Dockerfile.source, Dockerfile.decommission-local |
helm_chart_version.sh |
dev-tool | Normalizes chart/app versions for installation references | rustfs-release-version-bump skill |
test_helm_templates.sh |
ci-gate | Helm template rendering test | helm-package.yml |
Test & e2e runners
| Entry | Status | Purpose | Wiring / docs |
|---|---|---|---|
repro_rejoin_readiness_docker.py |
dev-tool | Fixed-source Docker HTTPS rejoin and acknowledged-overwrite diagnostic; retains first failure | Guide |
test_repro_rejoin_readiness_docker.py |
ci-gate | Provenance, native architecture, ownership and failure-preservation guards | make script-tests; same guide |
diagnose_scanner_enumeration_restart.py |
dev-tool | Strict fixed raw-entry-budget scanner-worker restart diagnostic | Checkpoint fixture |
prepare_replacement_migration.py |
dev-tool | Prepares digest-bound schema 5/6 replacement maintenance approvals | Replacement recovery |
test_prepare_replacement_migration.py |
dev-tool | Verifies maintenance approval scope, publication, and stopped-writer assertion | Python unittest; same runbook |
test_diagnose_scanner_enumeration_restart.py |
dev-tool | Driver report validation and positive convergence oracle tests | Python unittest; same guide |
test_object_generation_protocol_model.py |
dev-tool | Bounded single-slot, four-voter promise/accept design model and recovery/retirement examples; no production runtime evidence | python3 scripts/test_object_generation_protocol_model.py; Generation contract |
junit_diff.py |
dev-tool | Case-by-case pass/fail/skip diff of two JUnit XML reports (reference vs candidate stack); exit 1 on any difference or stale --allow entry, 2 on unreadable input |
python3 scripts/junit_diff.py a.xml b.xml; backlog T3.1 |
test_junit_diff.py |
ci-gate | Contract tests and fixtures (scripts/tests/junit_diff/) for junit_diff.py |
make script-tests |
dual_stack.py |
ci-gate | The both-stack lanes: stack (which stack the servers' own start-up lines show), preflight (does the binary start on the gateway stack), summary (per-profile junit_diff.py and the dated summary) |
python3 scripts/dual_stack.py summary --help; backlog T3.1 |
test_dual_stack.py |
ci-gate | Contract tests for dual_stack.py |
make script-tests |
mint_stack_diff.py |
dev-tool | Mint gate: every case legacy passes passes on gateway and no new gateway failure; a side without a result never satisfies it | python3 scripts/mint_stack_diff.py <legacy-leg-dir> <gateway-leg-dir>; backlog T3.3 |
test_mint_stack_diff.py |
ci-gate | Contract tests for mint_stack_diff.py |
make script-tests |
acceptance_lane.py |
ci-gate | Shared reader for the two gateway-acceptance checkers: the startup record, the stack the server's own log names, and recorded exit codes | imported by the two scripts below; tested by their --self-test |
check_conformance_rulings.py |
ci-gate | Judges the gateway stack's conformance report against .config/conformance-rulings.toml (an unruled failure, a ruled case failing differently, a stale ruling, or a skip on the server's own answer fails), reports the legacy run as an unjudged reference, and diffs the two |
.github/workflows/gateway-acceptance.yml; make script-tests (--self-test); backlog T3.4 |
compat_matrix_report.py |
ci-gate | Judges one stack's gateway client matrix manifest and gates on every legacy pass cell passing on the gateway stack |
.github/workflows/gateway-acceptance.yml; make script-tests (--self-test); backlog T3.5 |
s3-tests/reference_parity.py |
ci-gate | Plans the reference parity targets, lists a run's failures for the single rerun, and compares every candidate's s3-tests JUnit against the pinned reference per node id with a tolerated-difference file | .github/workflows/reference-parity.yml; backlog T3.2 |
s3-tests/test_reference_parity.py |
ci-gate | Contract tests for reference_parity.py |
make script-tests |
test_reference_binary.py |
ci-gate | Contract tests for .github/actions/reference-binary/reference_binary.py (pin resolution and verified unpacking) |
make script-tests |
e2e-run.sh |
ci-gate | Boots a rustfs server and runs the s3s-e2e black-box conformance tool against it |
ci.yml e2e-tests jobs; docs/testing/README.md |
run_ecstore_validation_suite.sh |
dev-tool | ecstore black-box validation suite (quick/full/destructive/fuzz profiles) |
docs/testing/README.md, docs/testing/ecstore-validation-suite-design.md |
run_e2e_tests.sh |
dev-tool | Local e2e_test crate runner (starts a server, applies filters, cleans up) |
crates/e2e_test/README.md |
gen_bucket_config_goldens.sh |
dev-tool | One-shot generator for the 13-family persisted bucket-config XML byte goldens; refuses to overwrite without --force and records the generating commit |
crates/ecstore/tests/fixtures/bucket-config-goldens/README.md; crates/ecstore/tests/bucket_config_goldens.rs |
run.sh |
dev-tool | Local rustfs startup wrapper | make e2e-server; Justfile |
run.ps1 |
dev-tool | Windows counterpart of run.sh |
— |
probe.sh |
dev-tool | Probe-style e2e run | make probe-e2e |
run_scanner_validation_harness.sh |
dev-tool | Scanner validation harness | docs/operations/scanner-benchmark-runbook.md |
scanner_memory_acceptance.py |
dev-tool | Owned S3 scanner-memory and scanner-off burst acceptance phases | Memory acceptance |
test_scanner_memory_acceptance.py |
dev-tool | Dataset, ownership, telemetry and acceptance-oracle tests | Python unittest; same guide |
check_test_wiring.py |
dev-tool | Validates test wiring and assembles one or more measured Scanner/Heal release descriptors into a single release evidence bundle | .config/scanner-heal-required-tests.json; check_test_wiring.py --self-test |
run_scanner_heal_evidence_case.sh |
dev-tool | Runs one Scanner/Heal release-evidence registry case and checks the produced receipt/oracle | .config/scanner-heal-required-tests.json; check_test_wiring.py --check-scanner-heal |
run_scanner_heal_authority_evidence.py |
dev-tool | Assembles measured Scanner/Heal G01 root/quota authority release descriptors from operator-collected authority artifacts | .config/scanner-heal-required-tests.json; test_scanner_heal_authority_evidence.sh |
run_scanner_heal_checkpoint_crash_evidence.py |
dev-tool | Assembles measured Scanner/Heal G02/R-E checkpoint and restart release descriptors from scanner restart diagnostic reports | diagnose_scanner_enumeration_restart.py; test_scanner_heal_checkpoint_crash_evidence.sh |
run_scanner_heal_g14_multiset_evidence.py |
dev-tool | Assembles measured Scanner/Heal G14 same-window EC8+4 multi-set/multi-pool release descriptors from e2e case directories or an operator-collected proof | .config/scanner-heal-required-tests.json; test_scanner_heal_g14_multiset_evidence.sh |
run_scanner_heal_g09_upgrade_evidence.sh |
dev-tool | Runs the G09 mixed-version and rollback upgrade E2E lanes against a pinned previous release and verifies the raw evidence artifacts | docs/testing/ci-gates.md; .github/workflows/e2e-upgrade.yml; test_scanner_heal_g09_upgrade_evidence.sh |
run_scanner_heal_linux_evidence_plan.py |
dev-tool | Writes the unified Scanner/Heal Linux release-evidence execution manifest and can run lightweight preflight checks without producing measured evidence | docs/testing/ci-gates.md; test_scanner_heal_linux_evidence_plan.sh |
run_scanner_heal_scoped_ack_evidence.py |
dev-tool | Assembles measured Scanner/Heal G03 scoped ACK publication and mixed-peer fallback release descriptors | .config/scanner-heal-required-tests.json; test_scanner_heal_scoped_ack_evidence.sh |
run_scanner_heal_legacy_rollback_evidence.py |
dev-tool | Assembles measured Scanner/Heal R-L legacy source-conflict, migration-gap, and source-retirement release descriptors | .config/scanner-heal-required-tests.json; test_scanner_heal_legacy_rollback_evidence.sh |
run_scanner_heal_mrf_evidence.py |
dev-tool | Assembles measured Scanner/Heal G07/G08/P4 MRF release descriptors from W13 raw artifacts | .config/scanner-heal-required-tests.json; test_scanner_heal_w13_mrf_evidence.sh |
run_scanner_heal_scheduler_pressure_evidence.py |
dev-tool | Assembles measured Scanner/Heal G10/P1/P3 scheduler-pressure release descriptors from a completed measured ABBA run, recovery-window proof, and profile artifacts | docs/operations/scanner-benchmark-runbook.md; test_scanner_heal_scheduler_pressure_evidence.sh |
run_scanner_heal_status_outcome_probe.py |
dev-tool | Normalizes live Scanner/Heal status/outcome observations into the measured G05/G06/R-D raw artifacts consumed by the descriptor producer | .config/scanner-heal-required-tests.json; test_scanner_heal_status_outcome_evidence.sh |
run_scanner_heal_status_outcome_evidence.py |
dev-tool | Assembles measured Scanner/Heal G05/G06/R-D status-and-outcome release descriptors from same-run status, compatibility, and disposition artifacts | .config/scanner-heal-required-tests.json; test_scanner_heal_status_outcome_evidence.sh |
run_scanner_heal_maintenance_evidence.py |
dev-tool | Assembles measured Scanner/Heal G11/G13 maintenance-producer release descriptors from operator-collected proof JSON | .config/scanner-heal-required-tests.json; test_scanner_heal_maintenance_evidence.sh |
run_scanner_heal_w13_mrf_evidence.sh |
dev-tool | Runs the W13 durable MRF replay lanes and writes G07/G08/P4 bundle-ready evidence descriptors | docs/testing/ci-gates.md; test_scanner_heal_w13_mrf_evidence.sh |
run_scanner_heal_w16_recovery_evidence.sh |
dev-tool | Runs the W16 recovery-intent and quota authority lanes and writes G04/G12 bundle-ready evidence descriptors | docs/testing/ci-gates.md; test_scanner_heal_w16_recovery_evidence.sh |
test_scanner_validation_harness.sh |
dev-tool | Self-test for the scanner validation harness | — |
test_scanner_heal_checkpoint_crash_evidence.sh |
dev-tool | Shell self-test for the Scanner/Heal checkpoint/crash evidence assembler | — |
test_scanner_heal_authority_evidence.sh |
dev-tool | Shell self-test for the Scanner/Heal G01 authority evidence assembler | — |
test_scanner_heal_scoped_ack_evidence.sh |
dev-tool | Shell self-test for the Scanner/Heal G03 scoped ACK evidence assembler | — |
test_scanner_heal_legacy_rollback_evidence.sh |
dev-tool | Shell self-test for the Scanner/Heal R-L legacy rollback evidence assembler | — |
test_scanner_heal_g14_multiset_evidence.sh |
dev-tool | Shell self-test for the Scanner/Heal G14 multi-set/multi-pool evidence assembler | — |
test_scanner_heal_scheduler_pressure_evidence.sh |
dev-tool | Shell self-test for the Scanner/Heal scheduler-pressure evidence assembler | — |
test_scanner_heal_status_outcome_evidence.sh |
dev-tool | Shell self-test for the Scanner/Heal status-and-outcome evidence assembler | — |
test_scanner_heal_maintenance_evidence.sh |
dev-tool | Shell self-test for the Scanner/Heal G11/G13 maintenance evidence assembler | — |
test_scanner_heal_g09_upgrade_evidence.sh |
dev-tool | Shell self-test for the Scanner/Heal G09 upgrade evidence runner | — |
test_scanner_heal_w16_recovery_evidence.sh |
dev-tool | Shell self-test for the Scanner/Heal W16 recovery evidence runner | — |
test_scanner_heal_w13_mrf_evidence.sh |
dev-tool | Shell self-test for the Scanner/Heal W13 MRF evidence runner | — |
scanner_abba.py |
dev-tool | Scanner/heal ABBA orchestration and evidence gates via run_scanner_validation_harness.sh --abba |
docs/operations/scanner-benchmark-runbook.md |
scanner_client_trace.py |
dev-tool | Reconcile client request ledgers and calculate per-window exact sample percentiles | scanner_abba.py |
test_scanner_abba.py |
dev-tool | Synthetic ABBA adapter and failure-path tests | test_scanner_validation_harness.sh |
test_scanner_client_trace.py |
dev-tool | Client ledger reconciliation and fail-closed tests | test_scanner_validation_harness.sh |
test_summarize_scanner_heal_perf.py |
dev-tool | Scanner/heal summary and provenance tests | test_scanner_validation_harness.sh |
test_scanner_heal_scheduler_pressure_evidence.py |
dev-tool | Scheduler-pressure evidence fail-closed tests | test_scanner_heal_scheduler_pressure_evidence.sh |
test_scanner_heal_linux_evidence_plan.py |
dev-tool | Linux evidence plan coverage tests | test_scanner_heal_linux_evidence_plan.sh |
test_build_rustfs_options.sh |
dev-tool | Shell test for rustfs build-option wiring | make test (script-tests) |
test_entrypoint_credentials.sh |
dev-tool | Container entrypoint credential and bundled bootstrap-default test | make test (script-tests) |
test_helm_chart_version.sh |
dev-tool | Test for helm_chart_version.sh |
— |
test_package_service_scripts.sh |
ci-gate | Verifies DEB/RPM install, upgrade, removal, and service restart scriptlets | audit.yml workflow-pin-report |
test_package_versions.sh |
ci-gate | Exact-output and fail-closed tests for DEB/RPM package version normalization | audit.yml workflow-pin-report; package.yml RPM build |
windows-sftp-listener-smoke.sh |
dev-tool | Confirms rustfs.exe --features sftp binds an SFTP listener on Windows |
— |
Benchmark & performance harnesses
| Entry | Status | Purpose | Wiring / docs |
|---|---|---|---|
run_hotpath_warp_ab.sh |
dev-tool | Linux warp A/B rig for the hotpath series (quick local A/B) | docs/operations/hotpath-warp-ab-runbook.md |
run_hotpath_warp_abba.sh |
dev-tool | Formal ABBA warp runner (baseline/candidate interleaved, --rounds >= 3) |
docs/operations/hotpath-warp-ab-runbook.md |
hotpath_warp_ab_gate.sh |
dev-tool | Relative-budget gate evaluated over the warp A/B results | used by run_hotpath_warp_ab.sh; hotpath runbook |
run_internode_grpc_ab_bench.sh |
dev-tool | One-click A/B driver for the internode gRPC optimization stages | docs/operations/internode-grpc-benchmark-runbook.md |
run_internode_transport_baseline.sh |
dev-tool | Internode transport baseline runner | internode runbook; crates/io-metrics/README.md |
check_internode_rpc_timeout_boundary.py |
dev-tool | Verifies under-budget success, over-budget deadlines and post-stall recovery from recorded internode RPC probe reports | internode gRPC benchmark runbook; test_internode_rpc_timeout_boundary.py |
run_four_node_cluster_failover_bench.sh |
dev-tool | Four-node cluster failover benchmark with configurable host ports and optional server-only image build | docker/compose/README.md; internode runbook |
run_object_batch_bench.sh |
dev-tool | Batch object benchmark runner with PUT-specific Warp summary extraction | internode + scanner runbooks |
test_object_batch_bench.sh |
dev-tool | Regression test for Warp PUT summary extraction | scripts/test_object_batch_bench.sh |
wait_storage_quiescence.py |
dev-tool | Fail-closed dirty/writeback and PSI stability gate before storage benchmarks | python3 -m unittest scripts/test_wait_storage_quiescence.py |
run_object_batch_bench_enhanced.sh |
dev-tool | Enhanced batch benchmark runner; hub used by the smoke rigs | hotpath runbook |
run_pinned_paired_abba_bench.sh |
dev-tool | Pinned RustFS/MinIO paired ABBA benchmark orchestrator for backlog#1432 | test_pinned_paired_abba_bench.sh |
prepare_conditional_read_baseline.py |
dev-tool | Installs only the matching conditional-read benchmark harness in a clean detached baseline | docs/testing/conditional-read-benchmark.md |
run_conditional_read_abba.py |
dev-tool | Serial ordinary/hit/miss GET ABBA with cache/slow-disk controls and drift/regression gates | docs/testing/conditional-read-benchmark.md |
test_conditional_read_abba.py |
dev-tool | Parser, drift rejection, and throughput/lock regression tests for conditional-read evidence | run_conditional_read_abba.py |
run_get_codec_streaming_smoke.sh |
dev-tool | Local GET benchmark harness for the codec streaming read path | docs/testing/ecstore-validation-suite-design.md |
run_get_1mib_abba_stage_metrics.sh |
dev-tool | Exact-1MiB isolated-host GET ABBA/stage-metrics harness for backlog#1434 | test_get_1mib_abba_stage_metrics.sh |
issue_2007_coalescer_prometheus_report.py |
dev-tool | Read-only Prometheus report for GET metadata coalescer delay cost validation; usage in the module docstring | test_issue_2007_coalescer_prometheus_report.sh |
prometheus_metrics_1649_smoke.py |
dev-tool | Read-only Prometheus instant-query smoke check for the backlog#1649 metric dimensions, required labels, and retired series; usage in the module docstring | --self-test |
run_gt1g_get_http_matrix.sh |
dev-tool | >1 GiB GET HTTP matrix | docs/testing/ecstore-validation-suite-design.md |
run_gt1g_multipart_put_matrix.sh |
dev-tool | >1 GiB multipart PUT matrix | docs/testing/ecstore-validation-suite-design.md |
sample_remote_rustfs_rss.sh |
dev-tool | Remote RustFS PID CPU/RSS TSV sampler for hotpath profiling runs | test_sample_remote_rustfs_rss.sh; backlog#1647 |
summarize_samply_profile_symbols.py |
dev-tool | Offline samply profile.json.gz + .syms.json function-level hotpath summarizer |
test_summarize_samply_profile_symbols.py; backlog#1647 |
run_scanner_benchmarks.sh |
dev-tool (disposition pending) | Scanner performance benchmark runner. Contains a hardcoded stale path; disposition owned by backlog perf-10 — do not fix, move, or delete it here | — |
Local development & operations
| Entry | Status | Purpose | Wiring / docs |
|---|---|---|---|
dev_clear.sh |
dev-tool | Local dev cleanup. scripts/dev_*.sh is a CI paths-filter glob — keep the naming |
ci.yml/build.yml paths filters |
dev_deploy.sh |
dev-tool | Copy a built binary to dev servers | make deploy (.config/make/deploy.mak); Justfile |
dev_rustfs.sh |
dev-tool | Local dev run loop | — |
dev_rustfs.env |
dev-tool (data) | Env presets for the dev scripts | — |
restart_local_single_node_multidisk_rustfs.sh |
dev-tool | Restart a local single-node multi-disk instance | — |
inspect_dashboard.sh |
dev-tool | Sanity-checks the Grafana dashboard JSON | .docker/observability |
notify.sh |
dev-tool | Starts a local webhook receiver for notify-target development | — |
manual_transition_debug.sh |
dev-tool | Log/metrics helper for manual transition troubleshooting | — |
manual_transition_journal_audit.sh |
dev-tool | Journal + metrics + log audit for manual transition jobs | — |
manual_transition_mixed_rollout_matrix.sh |
dev-tool | Matrix generator for mixed-version rollout phases | — |
manual_transition_mixed_rollout_runbook.sh |
dev-tool | Reusable mixed-version rollout runbook generator (external run) | — |
manual_transition_mixed_version_docker_harness.sh |
dev-tool | Dedicated #1508 Docker harness for old/new manual-transition rollout evidence with strict/baseline/blocked result classification | test_manual_transition_runbooks.sh |
monitor_manual_transition_ci.sh |
dev-tool | CI workflow/status watcher for manual transition follow-up monitoring | — |
manual_transition_soak_matrix.sh |
dev-tool | Matrix generator for nightly stress windows | — |
manual_transition_nightly_stress_runbook.sh |
dev-tool | Nightly stress entrypoint with failure snapshot templates | — |
install-flatc.sh |
dev-tool | Local flatc installer (macOS) | — |
install-protoc.sh |
dev-tool | Local protoc installer (macOS/Linux) | — |
cargo_publish_workspace.sh |
dev-tool | Generates the workspace crate publish order, checks existing registry versions, and optionally runs ordered cargo publish dry-runs or publication |
docs/operations/cargo-publish-workspace.md |
makefile-header.sh |
dev-tool | Generates the ## —— section —— header lines used in .config/make/*.mak |
— |
tls_gen.md |
dev-tool (doc) | Notes on generating local TLS certificates | — |
Subdirectories
| Entry | Status | Purpose | Wiring / docs |
|---|---|---|---|
codemods/ |
dev-tool | Scripted, idempotent source migrations for the gateway migration (rustfs/backlog#2734); s3_error_to_s3_types.py moves s3s error-type paths onto rustfs-s3-types per crate closure and refuses the s3s edge allowlist; split_bucket_usecase.py splits rustfs/src/app/bucket_usecase.rs into one file per configuration family (preparatory refactor P2, rustfs/backlog#2765) and stops on any item its tables do not assign |
make script-tests (--self-test); usage in each script's docstring |
fuzz/ |
ci-gate | Unified cargo-fuzz runner and helpers for the fuzz/ sub-workspace |
fuzz.yml; fuzz/README.md |
release/ |
ci-gate | Release creation and DEB/RPM version-normalization helpers | build.yml; package.yml; manual-main-packages.yml |
s3-tests/ |
ci-gate | ceph/s3-tests compatibility harness (allow-lists, patches, report tooling) | ci.yml; e2e-s3tests.yml; scripts/s3-tests/README.md |
security/ |
ci-gate | Workflow-pin enforcement and release supply-chain asset generation | audit.yml; build.yml |
table-catalog/ |
dev-tool | S3-Tables / pyiceberg validation suite | docs/architecture/s3-tables-support-matrix.md |
test/ |
dev-tool | Manual operational validation runbooks (decommission, tier lifecycle), paired .sh + .md |
— |
archive/ |
archived | Retired one-shot scripts (see below) | — |
Archived (scripts/archive/)
Moved 2026-07 (backlog#1153 infra-13) after a whole-tree reference census: each entry had zero references from CI, Makefiles, docs, or code — or was referenced only by other scripts in this same archived set. Reasons:
| Entry | Was |
|---|---|
validate_issue_785_list_objects.sh |
One-shot issue validation (list-objects series) |
validate_issue_786_list_objects.sh |
One-shot issue validation (list-objects series) |
validate_issue_787_list_quorum.sh |
One-shot issue validation (list-quorum) |
validate_issue_841_list_objects_observability.sh |
One-shot issue validation (list observability) |
validate_issue_1365_docker.sh |
One-shot issue validation (docker repro) |
validate_issue_2723_site_replication.sh |
One-shot issue validation (site replication) |
validate_issue_3031_docker.sh |
One-shot issue validation (docker repro) |
run_issue712_deeper_zero_copy_put_with_capture.sh |
One-shot perf capture for backlog#712 |
run_issue797_local_4node_16disk_ab.sh |
One-shot 4-node/16-disk A/B for backlog#797 |
run_issue_2573_acceptance.sh |
One-shot acceptance run for issue #2573 |
run_issue_2941_perf_capture.sh |
One-shot perf capture for issue #2941 |
run_put_large_stage_breakdown.sh |
backlog#706 large-PUT stage breakdown (family) |
run_put_large_stage_breakdown_with_capture.sh |
backlog#706 one-shot wrapper (family) |
run_put_large_tuning_matrix.sh |
backlog#706 tuning matrix (family) |
collect_put_large_stage_breakdown_artifacts.sh |
backlog#706 artifact collector (family) |
analyze_put_service_metrics_deltas.py |
backlog#706 metrics-delta analyzer (family) |
README-stress-test.md |
GET-optimization one-shot suite doc |
stress-test-get-optimization.sh |
GET-optimization one-shot stress test |
quick-validate-get-optimization.sh |
GET-optimization one-shot validation |
benchmark-sf-optimization.sh |
GET-optimization one-shot benchmark |
prepare_gt1g_get_test_objects.sh |
>1 GiB GET investigation one-shot fixture prep |
run_gt1g_multipart_put_server_path_focus.sh |
>1 GiB PUT investigation one-shot focus run |
run_get_metrics_gate_smoke.sh |
One-shot GET metrics-gate smoke |
run_object_batch_bench_abc.sh |
One-shot capacity/object profile A/B/C controller |
run_object_data_cache_bench.sh |
One-shot GET bench for the object-data-cache rollout gate |
setup-test-binaries.sh |
One-shot Docker-build test binary fixture |
test.sh |
Ancient manual mc bucket smoke scratchpad |
test_policy.json |
Orphaned IAM policy fixture (hardcoded test bucket) |