* fix(replication): close the GA blocker set from backlog#2366 (#7503)
* fix(replication): close GA blockers from backlog#2366
Implements the P1 set from the pre-GA replication audit:
- Replication rule tag filters now require every And.Tag to match, replacing
the s3s OR semantics with a local AND matcher that fails closed on a
malformed tag.
- A replicated group membership change no longer writes the group status, so
a membership update carrying the default Enabled status cannot silently
re-enable a disabled group on the peer.
- A successful IAM import schedules one collapsed full-IAM snapshot per remote
peer instead of leaving the imported entities local-only.
- A pending endpoint refresh is redriven by the heavyweight reconcile tick,
carries its own ilm-expiry override, and no longer blocks a remove that
drops every unacknowledged peer.
- Site metrics expose local replication failure totals and rolling windows;
node-level counters no longer report a constructed zero.
- set/remove-remote-target notify peer metadata caches before returning, so a
follow-up put-bucket-replication on another node sees the target.
- Adds the site-replication operations runbook, a docs index, a replication
support boundary section, and the Replication changelog section.
* fix(site-replication): resume only a locally driven endpoint refresh
The peer-side edit handler journals a pending endpoint refresh with an empty
`remote_peers` map and commits it inside the same request through
`apply_internal_peer_edit`. The reconcile tick could not tell that journal
from the coordinator's own: with no required peers it reads as complete on
sight, so the tick committed it with `edit_state` - losing the local-name
sync - and cleared it under the request that owned it, whose commit then
reported the refresh as changed and denied the coordinator the peer
acknowledgement it was waiting for.
Resume now runs only for a journal that carries the fan-out topology. A
receiver's journal stays for the coordinator to redrive with the same
refresh id, which is the path that already recovers it.
* fix(site-replication): keep an explicit disabled group status on a snapshot
Skipping the group-status write whenever an item carries members stopped a
membership change from re-enabling a disabled group, but it also silenced the
full-IAM snapshot, which always sends members together with the sender's real
status. A peer that did not have the group yet created it through
`GroupInfo::new` - enabled - so a bootstrap, a repair, or the snapshot an IAM
import now schedules handed every member of a frozen group live access there.
The madmin wire maps an unset `groupStatus` to Enabled, so only Enabled can be
a default. Disabled is always explicit and is applied again.
* fix(site-replication): schedule the import snapshot without recording a failure
`import-iam` reused the failure-recording path to queue its full-IAM
snapshot. That raises `retry_count` on every call, so three imports - the
normal shape of a bulk migration done one archive at a time - escalated a
healthy peer to `retryStats.failed` with the scheduling note shown as
`lastError`, which is exactly the signal the runbook tells operators to
repair. A full retry queue also turned a completed import into a 503.
Scheduling now only ensures the collapsed entry exists, and a failure to
schedule is logged instead of failing the request: the entities are already
imported and the reconcile pass still closes the gap.
* fix(admin): stop reporting replication failures as retries
`retries` is the minio-go counter for redeliveries, and mc prints it as such.
Filling it with the failure count claimed a redelivery that never happens: a
failed object is not retried by an event today, it waits for the scanner heal
pass. `errors` keeps the failure counters; `retries` stays zero until there is
a real redelivery to count, and the runbook now says so.
* perf(site-replication): aggregate failure windows without cloning bucket stats
`site_metrics_snapshot` went through `get_all`, which clones every bucket's
stats, and then scanned each target's sample deque twice. That deque is
bounded only by the one-hour window, so an unreachable target under load -
the case an operator polls this endpoint for - made every
`mc admin replicate status` copy the whole backlog and hold the read lock
against the failure path while doing it.
It now folds under the read lock and takes both windows in one walk. The
`max` against the serialized `last_minute` / `last_hour` snapshots is dropped:
those are stamped onto per-bucket clones elsewhere and are always zero in this
node-local cache.
* fix(site-replication): reject a conflicting ilm-expiry override on a re-run
The commit now reads the ilm-expiry override back out of the pending refresh
journal, so a second edit that asks for a different value had it dropped while
the request still reported success. Re-running without the flag keeps pinning
the recorded value - that is the documented way to redrive a stuck refresh -
but an explicit different value is now rejected instead of ignored.
* fix(admin): do not fail a remote-target write on a peer reload error
set/remove-remote-target propagated the peer metadata reload error, so a
target that was already persisted and live on this node reported a 5xx to the
client whenever one peer could not be reached. Every S3 bucket-config write
path treats that reload as best effort and only warns; these two admin
handlers now do the same, and the reason is logged with the bucket and action.
* fix(site-replication): undo every bucket a cut-short refresh rewrote
When a remove accepted on another node clears the refresh journal mid-pass,
only the bucket holding the lock at that moment had its restored target
undone. The buckets rewritten earlier in the same pass kept a target pointing
at the removed peer whenever the remove's own cleanup had already walked past
them. The undo now covers every bucket this pass rewrote, attempting all of
them so one failure does not strand the rest.
* fix(site-replication): keep replay running while an endpoint refresh is pending
A pending endpoint refresh took the whole heavyweight pass with it, so a peer
that never came back froze IAM and bucket replay to every healthy peer too -
the stall this journal's resume path was meant to end. The refresh arm now
drains the retry queue before returning; it replays per-peer deliveries
against the endpoints currently committed in state, so it is unaffected by the
edit in flight. Bucket wiring reconciliation still waits, because it rewrites
the very targets the refresh is changing, and the runbook now says so.
* test(e2e): cover the AND semantics of a two-tag replication filter
The acceptance matrix only had a single-tag rule, which matches under both AND
and OR semantics and therefore proved nothing about the filter this fix
changed. It now also carries a two-tag `And` rule - the shape
`mc replicate add --tags "k1=v1&k2=v2"` writes - and asserts that an object
with one of the two tags is not admitted while an object with both is.
No new test function, so the nightly selection digest is unchanged.
* refactor(site-replication): fold the refresh state-change error into one constructor
The endpoint-refresh work added three `s3_error!` invocation lines, which the
s3s footprint ratchet is meant to prevent. Five copies of the same
concurrent-change error now share one constructor, so the surface nets one
line smaller than main; the baseline is retightened to match.
* fix(site-replication): report a peer whose IAM snapshot waits for a repair
An escalated snapshot entry records a deletion a snapshot cannot replay, so
only a repair settles it and the marker must survive. Scheduling an import
snapshot therefore leaves that peer's entry alone - and now says so, instead
of returning success while nothing was scheduled for it.
* docs(operations): state the group-status and escalation convergence limits
Two boundaries the fixes in this branch make load-bearing: a membership change
never carries an enable, so a group disabled on one site only has to be
re-enabled there explicitly; and a peer holding an escalated IAM entry does
not receive a scheduled snapshot, including the one a bulk import schedules,
until a repair settles it.
* fix(ci): bind performance runs to selected inputs (#7512)
* test(scanner): add G09 upgrade evidence runner
Add a reusable Scanner/Heal G09 runner for Linux PR-head validation. The script downloads the pinned previous release, builds the current checkout, runs the mixed-version and rollback upgrade E2E lanes, and validates the measured raw artifacts before they can be consumed by the release bundle gate.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: 唐小鸭 <tangtang1251@qq.com>
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Authoritative inventory of everything under scripts/ (backlog#1153
infra-13). One row per top-level entry; subdirectories get one row each and
keep their own READMEs. The test-layer map that ties the major runners
together lives in docs/testing/README.md.
Statuses
ci-gate — wired into CI, release, or image-build pipelines. Do not move,
rename, or change flags without updating the wiring listed in the last
column.
dev-tool — run by humans: local dev loops, runbooks, validation
harnesses. Kept working, not wired into CI.
archived — one-shot scripts whose investigation/issue is finished, moved
to scripts/archive/. Unmaintained reference material: never
wire into CI, and expect bit-rot. To resurrect one, move it back and give it
an index row here.
Adding a script? Add an index row in the same PR. Issue-scoped scripts
(run_issueNNN_*, validate_issue_NNN_*) are expected to be archived when
their issue closes.
Repository & CI gates
Entry
Status
Purpose
Wiring / docs
check_architecture_migration_rules.sh
ci-gate
Architecture-boundary anti-regression guard
ci.yml Quick Checks; make pre-commit
check_body_cache_whitelist.sh
ci-gate
Keeps the app-layer body-cache eligibility gate fail-closed
ci.yml Quick Checks
check_doc_paths.sh
ci-gate
Fails when instruction/architecture docs reference repo paths that no longer exist
make pre-commit / pre-pr
check_embedded_secrets.sh
ci-gate
Repo-wide scan blocking committed private key material and provider credential literals
ci.yml Quick Checks; make pre-commit / pre-pr
check_extension_schema_boundaries.sh
ci-gate
Extension-schema crate boundary guard
ci.yml Quick Checks; make pre-commit
check_layer_dependencies.sh
ci-gate
Crate-layering DAG guard (reads layer-dependency-baseline.txt)
ci.yml Quick Checks
check_logging_guardrails.sh
ci-gate
Blocks legacy logging patterns from returning
make pre-commit / pre-pr
check_migration_gate_count.sh
ci-gate
Migration-critical test gate with committed count floor (.config/migration-gate-floor.txt)
ci.yml Test and Lint; docs/testing/README.md
check_no_planning_docs.sh
ci-gate
Blocks committed planning-type documents
ci.yml Quick Checks; make pre-commit
check_no_tokio_io_uring.sh
ci-gate
Keeps tokio's io-uring backend disabled
ci.yml Quick Checks
check_s3s_footprint.sh
ci-gate
Lower-only ratchet freezing the direct s3s surface ahead of the s3gate migration
ci.yml Quick Checks; make pre-commit
check_unsafe_code_allowances.sh
ci-gate
Unsafe-code allowance ledger guard
ci.yml Quick Checks
layer-dependency-baseline.txt
ci-gate (data)
Committed baseline consumed by check_layer_dependencies.sh
arch-checks skill
static.sh
ci-gate
Static-build helper executed inside image builds
Dockerfile.source, Dockerfile.decommission-local
helm_chart_version.sh
ci-gate
Keeps the Helm chart version in sync with the release
Enhanced batch benchmark runner; hub used by the smoke rigs
hotpath runbook
run_pinned_paired_abba_bench.sh
dev-tool
Pinned RustFS/MinIO paired ABBA benchmark orchestrator for backlog#1432
test_pinned_paired_abba_bench.sh
run_get_codec_streaming_smoke.sh
dev-tool
Local GET benchmark harness for the codec streaming read path
docs/testing/ecstore-validation-suite-design.md
run_get_1mib_abba_stage_metrics.sh
dev-tool
Exact-1MiB isolated-host GET ABBA/stage-metrics harness for backlog#1434
test_get_1mib_abba_stage_metrics.sh
issue_2007_coalescer_prometheus_report.py
dev-tool
Read-only Prometheus report for GET metadata coalescer delay cost validation; usage in the module docstring
test_issue_2007_coalescer_prometheus_report.sh
prometheus_metrics_1649_smoke.py
dev-tool
Read-only Prometheus instant-query smoke check for the backlog#1649 metric dimensions, required labels, and retired series; usage in the module docstring
--self-test
run_gt1g_get_http_matrix.sh
dev-tool
>1 GiB GET HTTP matrix
docs/testing/ecstore-validation-suite-design.md
run_gt1g_multipart_put_matrix.sh
dev-tool
>1 GiB multipart PUT matrix
docs/testing/ecstore-validation-suite-design.md
sample_remote_rustfs_rss.sh
dev-tool
Remote RustFS PID CPU/RSS TSV sampler for hotpath profiling runs
Moved 2026-07 (backlog#1153 infra-13) after a whole-tree reference census:
each entry had zero references from CI, Makefiles, docs, or code — or was
referenced only by other scripts in this same archived set. Reasons:
Entry
Was
validate_issue_785_list_objects.sh
One-shot issue validation (list-objects series)
validate_issue_786_list_objects.sh
One-shot issue validation (list-objects series)
validate_issue_787_list_quorum.sh
One-shot issue validation (list-quorum)
validate_issue_841_list_objects_observability.sh
One-shot issue validation (list observability)
validate_issue_1365_docker.sh
One-shot issue validation (docker repro)
validate_issue_2723_site_replication.sh
One-shot issue validation (site replication)
validate_issue_3031_docker.sh
One-shot issue validation (docker repro)
run_issue712_deeper_zero_copy_put_with_capture.sh
One-shot perf capture for backlog#712
run_issue797_local_4node_16disk_ab.sh
One-shot 4-node/16-disk A/B for backlog#797
run_issue_2573_acceptance.sh
One-shot acceptance run for issue #2573
run_issue_2941_perf_capture.sh
One-shot perf capture for issue #2941
run_put_large_stage_breakdown.sh
backlog#706 large-PUT stage breakdown (family)
run_put_large_stage_breakdown_with_capture.sh
backlog#706 one-shot wrapper (family)
run_put_large_tuning_matrix.sh
backlog#706 tuning matrix (family)
collect_put_large_stage_breakdown_artifacts.sh
backlog#706 artifact collector (family)
analyze_put_service_metrics_deltas.py
backlog#706 metrics-delta analyzer (family)
README-stress-test.md
GET-optimization one-shot suite doc
stress-test-get-optimization.sh
GET-optimization one-shot stress test
quick-validate-get-optimization.sh
GET-optimization one-shot validation
benchmark-sf-optimization.sh
GET-optimization one-shot benchmark
prepare_gt1g_get_test_objects.sh
>1 GiB GET investigation one-shot fixture prep
run_gt1g_multipart_put_server_path_focus.sh
>1 GiB PUT investigation one-shot focus run
run_get_metrics_gate_smoke.sh
One-shot GET metrics-gate smoke
run_listobjects_verified_bench.sh
One-shot verified list-objects bench
run_object_batch_bench_abc.sh
One-shot capacity/object profile A/B/C controller
run_object_data_cache_bench.sh
One-shot GET bench for the object-data-cache rollout gate
setup-test-binaries.sh
One-shot Docker-build test binary fixture
test.sh
Ancient manual mc bucket smoke scratchpad
test_policy.json
Orphaned IAM policy fixture (hardcoded test bucket)
houseme
857584b3c2