mirror of
https://github.com/rustfs/rustfs.git
synced 2026-09-08 13:06:00 +00:00
houseme
4c2a0cdf9a
* fix(scanner): remove unused digest import Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * feat(scanner): add raw page owner index (#7375) * feat(scanner): add raw page owner index Add a serializable raw enumeration page owner index for scanner resume work. The index exposes unsupported, building, and ready states, validates committed page identity by recomputing digests, and uses generation checks for CAS-style page commits. Focused tests cover small-budget restart progress, page digest/source drift rejection, corrupt deserialized state, CAS failure, precommit crash, empty sources, and invalid entry boundaries. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * feat(scanner): persist raw page owner resume state (#7379) Wire the scanner raw enumeration partial-cache writer to the raw page owner index so interrupted bucket walks can retain validated page-builder state across scanner restarts. Keep complete owner sources terminal-only, add partial-source ingestion for in-progress raw directory reads, and validate the persisted page index through bucket checkpoint preparation. Co-authored-by: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(scanner): fence segment producer observations (#7381) Require the segment observation fixture to carry source, incarnation, key-format, baseline, process epoch, generation-window, gap, overflow, and producer-coverage proof before accepting a narrowed proposal. Keep the diagnostic path fixture-only and remove its ordinary stderr output. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): isolate pool metadata read probes (#7367) Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(heal): cover MRF crash successor matrix (#7369) * test(heal): cover MRF crash successor matrix Add process-boundary MRF replay coverage for the successor snapshot window after a retained startup journal is flushed but before cleanup deletes it. Extend the mixed authoritative/legacy reader fixture with a scoped v2 journal epoch to pin the no-merge contract. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(heal): cover service-kill MRF replay (#7380) Add a Unix process fixture that waits after publishing the pending MRF successor snapshot, then is terminated by the parent before restart replay. Co-authored-by: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(heal): cover transport-lost start receipts (#7371) Add gRPC transport fault fixtures for heal-control start admission. The tests distinguish pre-admission transport loss from post-admission response loss, then verify exact envelope retries reuse the canonical receipt while fresh forceStart requests create distinct tasks. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(scanner): add crash-restart heal evidence case (#7370) * test(scanner): add crash-restart heal evidence case Add a distinct W21 background target crash case to the scanner/heal evidence registry and oracle path. Keep the existing restart lane on graceful process restart, keep the crash lane on hard kill, and make the wiring checker reject evidence/oracle mismatches. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(scanner): support older Python wiring checks Let the scanner/heal evidence wiring checker run under Python 3.9/3.10 by falling back to tomli and chunked SHA-256 hashing when the Python 3.11 standard APIs are unavailable. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix(scanner): reject stale raw page source seeds (#7382) Do not prefill a resumed raw page owner with previously indexed entries when starting a new raw directory observation pass. The next pass must observe the same prefix again before the page index can advance; otherwise the index is discarded fail-closed. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix(scanner): defer raw page revalidation until observed (#7384) A resumed raw page owner index must not prefill entries from older cache state, but it also must not discard a valid multi-entry index before the current raw directory pass has observed enough entries to prove identity. Track the persisted index floor and only run the strict owner identity check once the current pass reaches that floor. Co-authored-by: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: zhi22915 <qiuzgang@gmail.com>
scripts/ index
Authoritative inventory of everything under scripts/ (backlog#1153
infra-13). One row per top-level entry; subdirectories get one row each and
keep their own READMEs. The test-layer map that ties the major runners
together lives in docs/testing/README.md.
Statuses
- ci-gate — wired into CI, release, or image-build pipelines. Do not move, rename, or change flags without updating the wiring listed in the last column.
- dev-tool — run by humans: local dev loops, runbooks, validation harnesses. Kept working, not wired into CI.
- archived — one-shot scripts whose investigation/issue is finished, moved
to
scripts/archive/. Unmaintained reference material: never wire into CI, and expect bit-rot. To resurrect one, move it back and give it an index row here.
Adding a script? Add an index row in the same PR. Issue-scoped scripts
(run_issueNNN_*, validate_issue_NNN_*) are expected to be archived when
their issue closes.
Repository & CI gates
| Entry | Status | Purpose | Wiring / docs |
|---|---|---|---|
check_architecture_migration_rules.sh |
ci-gate | Architecture-boundary anti-regression guard | ci.yml Quick Checks; make pre-commit |
check_body_cache_whitelist.sh |
ci-gate | Keeps the app-layer body-cache eligibility gate fail-closed | ci.yml Quick Checks |
check_doc_paths.sh |
ci-gate | Fails when instruction/architecture docs reference repo paths that no longer exist | make pre-commit / pre-pr |
check_embedded_secrets.sh |
ci-gate | Repo-wide scan blocking committed private key material and provider credential literals | ci.yml Quick Checks; make pre-commit / pre-pr |
check_extension_schema_boundaries.sh |
ci-gate | Extension-schema crate boundary guard | ci.yml Quick Checks; make pre-commit |
check_layer_dependencies.sh |
ci-gate | Crate-layering DAG guard (reads layer-dependency-baseline.txt) |
ci.yml Quick Checks |
check_logging_guardrails.sh |
ci-gate | Blocks legacy logging patterns from returning | make pre-commit / pre-pr |
check_migration_gate_count.sh |
ci-gate | Migration-critical test gate with committed count floor (.config/migration-gate-floor.txt) |
ci.yml Test and Lint; docs/testing/README.md |
check_no_planning_docs.sh |
ci-gate | Blocks committed planning-type documents | ci.yml Quick Checks; make pre-commit |
check_no_tokio_io_uring.sh |
ci-gate | Keeps tokio's io-uring backend disabled | ci.yml Quick Checks |
check_s3s_footprint.sh |
ci-gate | Lower-only ratchet freezing the direct s3s surface ahead of the s3gate migration | ci.yml Quick Checks; make pre-commit |
check_unsafe_code_allowances.sh |
ci-gate | Unsafe-code allowance ledger guard | ci.yml Quick Checks |
layer-dependency-baseline.txt |
ci-gate (data) | Committed baseline consumed by check_layer_dependencies.sh |
arch-checks skill |
static.sh |
ci-gate | Static-build helper executed inside image builds | Dockerfile.source, Dockerfile.decommission-local |
helm_chart_version.sh |
ci-gate | Keeps the Helm chart version in sync with the release | helm-package.yml |
test_helm_templates.sh |
ci-gate | Helm template rendering test | helm-package.yml |
Test & e2e runners
| Entry | Status | Purpose | Wiring / docs |
|---|---|---|---|
diagnose_scanner_enumeration_restart.py |
dev-tool | Strict fixed raw-entry-budget scanner-worker restart diagnostic | Checkpoint fixture |
test_diagnose_scanner_enumeration_restart.py |
dev-tool | Driver report validation and positive convergence oracle tests | Python unittest; same guide |
e2e-run.sh |
ci-gate | Boots a rustfs server and runs the s3s-e2e black-box conformance tool against it |
ci.yml e2e-tests jobs; docs/testing/README.md |
run_ecstore_validation_suite.sh |
dev-tool | ecstore black-box validation suite (quick/full/destructive/fuzz profiles) |
docs/testing/README.md, docs/testing/ecstore-validation-suite-design.md |
run_e2e_tests.sh |
dev-tool | Local e2e_test crate runner (starts a server, applies filters, cleans up) |
crates/e2e_test/README.md |
run.sh |
dev-tool | Local rustfs startup wrapper | make e2e-server; Justfile |
run.ps1 |
dev-tool | Windows counterpart of run.sh |
— |
probe.sh |
dev-tool | Probe-style e2e run | make probe-e2e |
run_scanner_validation_harness.sh |
dev-tool | Scanner validation harness | docs/operations/scanner-benchmark-runbook.md |
test_scanner_validation_harness.sh |
dev-tool | Self-test for the scanner validation harness | — |
scanner_abba.py |
dev-tool | Scanner/heal ABBA orchestration and evidence gates via run_scanner_validation_harness.sh --abba |
docs/operations/scanner-benchmark-runbook.md |
test_scanner_abba.py |
dev-tool | Synthetic ABBA adapter and failure-path tests | test_scanner_validation_harness.sh |
test_build_rustfs_options.sh |
dev-tool | Shell test for rustfs build-option wiring | make test (script-tests) |
test_entrypoint_credentials.sh |
dev-tool | Container entrypoint credential-handling test | make test (script-tests) |
test_helm_chart_version.sh |
dev-tool | Test for helm_chart_version.sh |
— |
test_package_versions.sh |
ci-gate | Exact-output and fail-closed tests for DEB/RPM package version normalization | audit.yml workflow-pin-report; package.yml RPM build |
windows-sftp-listener-smoke.sh |
dev-tool | Confirms rustfs.exe --features sftp binds an SFTP listener on Windows |
— |
Benchmark & performance harnesses
| Entry | Status | Purpose | Wiring / docs |
|---|---|---|---|
run_hotpath_warp_ab.sh |
dev-tool | Linux warp A/B rig for the hotpath series (quick local A/B) | docs/operations/hotpath-warp-ab-runbook.md |
run_hotpath_warp_abba.sh |
ci-gate | Formal ABBA warp runner (baseline/candidate interleaved, --rounds >= 3); the runner performance-ab.yml executes |
performance-ab.yml (scheduled); docs/operations/hotpath-warp-ab-runbook.md |
hotpath_warp_ab_gate.sh |
dev-tool | Relative-budget gate evaluated over the warp A/B results | used by run_hotpath_warp_ab.sh; hotpath runbook |
run_internode_grpc_ab_bench.sh |
dev-tool | One-click A/B driver for the internode gRPC optimization stages | docs/operations/internode-grpc-benchmark-runbook.md |
run_internode_transport_baseline.sh |
dev-tool | Internode transport baseline runner | internode runbook; crates/io-metrics/README.md |
run_four_node_cluster_failover_bench.sh |
dev-tool | Four-node cluster failover benchmark | docker/compose/README.md; internode runbook |
run_object_batch_bench.sh |
dev-tool | Batch object benchmark runner (warp/s3bench) | internode + scanner runbooks |
run_object_batch_bench_enhanced.sh |
dev-tool | Enhanced batch benchmark runner; hub used by the smoke rigs | hotpath runbook |
run_pinned_paired_abba_bench.sh |
dev-tool | Pinned RustFS/MinIO paired ABBA benchmark orchestrator for backlog#1432 | test_pinned_paired_abba_bench.sh |
run_get_codec_streaming_smoke.sh |
dev-tool | Local GET benchmark harness for the codec streaming read path | docs/testing/ecstore-validation-suite-design.md |
run_get_1mib_abba_stage_metrics.sh |
dev-tool | Exact-1MiB isolated-host GET ABBA/stage-metrics harness for backlog#1434 | test_get_1mib_abba_stage_metrics.sh |
issue_2007_coalescer_prometheus_report.py |
dev-tool | Read-only Prometheus report for GET metadata coalescer delay cost validation; usage in the module docstring | test_issue_2007_coalescer_prometheus_report.sh |
prometheus_metrics_1649_smoke.py |
dev-tool | Read-only Prometheus instant-query smoke check for the backlog#1649 metric dimensions, required labels, and retired series; usage in the module docstring | --self-test |
run_gt1g_get_http_matrix.sh |
dev-tool | >1 GiB GET HTTP matrix | docs/testing/ecstore-validation-suite-design.md |
run_gt1g_multipart_put_matrix.sh |
dev-tool | >1 GiB multipart PUT matrix | docs/testing/ecstore-validation-suite-design.md |
sample_remote_rustfs_rss.sh |
dev-tool | Remote RustFS PID CPU/RSS TSV sampler for hotpath profiling runs | test_sample_remote_rustfs_rss.sh; backlog#1647 |
summarize_samply_profile_symbols.py |
dev-tool | Offline samply profile.json.gz + .syms.json function-level hotpath summarizer |
test_summarize_samply_profile_symbols.py; backlog#1647 |
run_scanner_benchmarks.sh |
dev-tool (disposition pending) | Scanner performance benchmark runner. Contains a hardcoded stale path; disposition owned by backlog perf-10 — do not fix, move, or delete it here | — |
Local development & operations
| Entry | Status | Purpose | Wiring / docs |
|---|---|---|---|
dev_clear.sh |
dev-tool | Local dev cleanup. scripts/dev_*.sh is a CI paths-filter glob — keep the naming |
ci.yml/build.yml paths filters |
dev_deploy.sh |
dev-tool | Copy a built binary to dev servers | make deploy (.config/make/deploy.mak); Justfile |
dev_rustfs.sh |
dev-tool | Local dev run loop | — |
dev_rustfs.env |
dev-tool (data) | Env presets for the dev scripts | — |
restart_local_single_node_multidisk_rustfs.sh |
dev-tool | Restart a local single-node multi-disk instance | — |
inspect_dashboard.sh |
dev-tool | Sanity-checks the Grafana dashboard JSON | .docker/observability |
notify.sh |
dev-tool | Starts a local webhook receiver for notify-target development | — |
manual_transition_debug.sh |
dev-tool | Log/metrics helper for manual transition troubleshooting | — |
manual_transition_journal_audit.sh |
dev-tool | Journal + metrics + log audit for manual transition jobs | — |
manual_transition_mixed_rollout_matrix.sh |
dev-tool | Matrix generator for mixed-version rollout phases | — |
manual_transition_mixed_rollout_runbook.sh |
dev-tool | Reusable mixed-version rollout runbook generator (external run) | — |
manual_transition_mixed_version_docker_harness.sh |
dev-tool | Dedicated #1508 Docker harness for old/new manual-transition rollout evidence with strict/baseline/blocked result classification | test_manual_transition_runbooks.sh |
monitor_manual_transition_ci.sh |
dev-tool | CI workflow/status watcher for manual transition follow-up monitoring | — |
manual_transition_soak_matrix.sh |
dev-tool | Matrix generator for nightly stress windows | — |
manual_transition_nightly_stress_runbook.sh |
dev-tool | Nightly stress entrypoint with failure snapshot templates | — |
install-flatc.sh |
dev-tool | Local flatc installer (macOS) | — |
install-protoc.sh |
dev-tool | Local protoc installer (macOS/Linux) | — |
makefile-header.sh |
dev-tool | Generates the ## —— section —— header lines used in .config/make/*.mak |
— |
tls_gen.md |
dev-tool (doc) | Notes on generating local TLS certificates | — |
Subdirectories
| Entry | Status | Purpose | Wiring / docs |
|---|---|---|---|
fuzz/ |
ci-gate | Unified cargo-fuzz runner and helpers for the fuzz/ sub-workspace |
fuzz.yml; fuzz/README.md |
release/ |
ci-gate | Release creation and DEB/RPM version-normalization helpers | build.yml; package.yml |
s3-tests/ |
ci-gate | ceph/s3-tests compatibility harness (allow-lists, patches, report tooling) | ci.yml; e2e-s3tests.yml; scripts/s3-tests/README.md |
security/ |
ci-gate | Workflow-pin enforcement and release supply-chain asset generation | audit.yml; build.yml |
table-catalog/ |
dev-tool | S3-Tables / pyiceberg validation suite | docs/architecture/s3-tables-support-matrix.md |
test/ |
dev-tool | Manual operational validation runbooks (decommission, tier lifecycle), paired .sh + .md |
— |
archive/ |
archived | Retired one-shot scripts (see below) | — |
Archived (scripts/archive/)
Moved 2026-07 (backlog#1153 infra-13) after a whole-tree reference census: each entry had zero references from CI, Makefiles, docs, or code — or was referenced only by other scripts in this same archived set. Reasons:
| Entry | Was |
|---|---|
validate_issue_785_list_objects.sh |
One-shot issue validation (list-objects series) |
validate_issue_786_list_objects.sh |
One-shot issue validation (list-objects series) |
validate_issue_787_list_quorum.sh |
One-shot issue validation (list-quorum) |
validate_issue_841_list_objects_observability.sh |
One-shot issue validation (list observability) |
validate_issue_1365_docker.sh |
One-shot issue validation (docker repro) |
validate_issue_2723_site_replication.sh |
One-shot issue validation (site replication) |
validate_issue_3031_docker.sh |
One-shot issue validation (docker repro) |
run_issue712_deeper_zero_copy_put_with_capture.sh |
One-shot perf capture for backlog#712 |
run_issue797_local_4node_16disk_ab.sh |
One-shot 4-node/16-disk A/B for backlog#797 |
run_issue_2573_acceptance.sh |
One-shot acceptance run for issue #2573 |
run_issue_2941_perf_capture.sh |
One-shot perf capture for issue #2941 |
run_put_large_stage_breakdown.sh |
backlog#706 large-PUT stage breakdown (family) |
run_put_large_stage_breakdown_with_capture.sh |
backlog#706 one-shot wrapper (family) |
run_put_large_tuning_matrix.sh |
backlog#706 tuning matrix (family) |
collect_put_large_stage_breakdown_artifacts.sh |
backlog#706 artifact collector (family) |
analyze_put_service_metrics_deltas.py |
backlog#706 metrics-delta analyzer (family) |
README-stress-test.md |
GET-optimization one-shot suite doc |
stress-test-get-optimization.sh |
GET-optimization one-shot stress test |
quick-validate-get-optimization.sh |
GET-optimization one-shot validation |
benchmark-sf-optimization.sh |
GET-optimization one-shot benchmark |
prepare_gt1g_get_test_objects.sh |
>1 GiB GET investigation one-shot fixture prep |
run_gt1g_multipart_put_server_path_focus.sh |
>1 GiB PUT investigation one-shot focus run |
run_get_metrics_gate_smoke.sh |
One-shot GET metrics-gate smoke |
run_listobjects_verified_bench.sh |
One-shot verified list-objects bench |
run_object_batch_bench_abc.sh |
One-shot capacity/object profile A/B/C controller |
run_object_data_cache_bench.sh |
One-shot GET bench for the object-data-cache rollout gate |
setup-test-binaries.sh |
One-shot Docker-build test binary fixture |
test.sh |
Ancient manual mc bucket smoke scratchpad |
test_policy.json |
Orphaned IAM policy fixture (hardcoded test bucket) |