chore: update star history

.agents/references/adversarial-validation.md

@@ -1,33 +0,0 @@
-# Adversarial Review Shape
-
-Use when root `AGENTS.md` triggers adversarial validation or for a substantial PR
-review. Paths below are repository-relative. The root finding standard and
-completion rule apply; selecting a lens does not require finding a defect.
-
-Risk and review shape:
-
-- **Exempt:** documentation, comments, formatting, or typos with no runtime,
-  build, test, or agent-execution effect.
-- **Mechanical:** renames, moves, test/tooling-only changes, and agent-rule
-  changes. Run correctness and simplicity lenses.
-- **Standard:** localized behavior changes. Run one integrated final-diff pass
-  covering correctness, simplicity, and test coverage; add only domain lenses
-  matched by the diff.
-- **High risk / substantial PR review:** high risk includes locking,
-  erasure/quorum/heal, replication, multipart, RPC, lifecycle/tiering,
-  persistence/fsync, IAM/KMS/auth, cryptography, on-disk/on-wire formats, and
-  S3-visible semantics. Cover all applicable lenses using exactly two
-  independent reviewers when delegation is explicitly authorized. Split the
-  lenses between them. Otherwise perform two fresh sequential passes.
-- **Outbound client defaults:** what `TargetClient`, `PutObjectOptions`, or
-  the remote SDK configuration sends to every replication or migration target
-  is high risk for every target class even when the change fixes one. Follow
-  the SOP in `docs/postmortems/2026-09-03-replication-checksum-default-regression.md`:
-  run the outbound target matrix, document each new env knob in the same PR,
-  and list verified and unverified target classes in the PR Impact section.
-
-Available domain lenses are security, concurrency/durability, compatibility,
-and performance. Select `.agents/skills/adversarial-validation/SKILL.md` for an
-explicit adversarial request, a high-risk change, or a substantial PR review;
-then read only its matching role references. A routine standard pass does not
-load the playbook unless the reviewer needs a RustFS-specific probe.

.agents/references/implementation.md

@@ -1,72 +0,0 @@
-# Implementation Rules
-
-Applies when changing code or running artifact-heavy work. Paths below
-are repository-relative. Read only the relevant sections during read-only review.
-
-## Worktree and Disk Hygiene
-
-- Start implementation from the latest `origin/main` and confirm the requested
-  change is not already present.
-- An existing clean, isolated task worktree is sufficient. Create another
-  worktree only when the current checkout is shared, dirty with unrelated work,
-  or belongs to another task.
-- Never commit from a shared checkout.
-- Use a task-specific branch named `<type>/<topic>`, such as `fix/...`,
-  `feat/...`, `test/...`, or `docs/...`, unless the user specifies a name.
-- Do not include agent, tool, contributor, account, or organization names in
-  branch names.
-- Push to the user-requested remote or the repository's configured push remote.
-  Do not hard-code or infer a remote from an account name.
-- Check free space before artifact-heavy builds, tests, coverage, or downloads.
-  Re-check before a broad gate when space is tight.
-- Remove only task-owned temporary/build artifacts. Never delete another task's
-  worktree or uncommitted data.
-- At handoff, mention disk or cleanup details only when they affected execution
-  or artifacts/worktrees remain intentionally.
-
-## Change Style
-
-- Preserve existing control flow unless changing it is required for correctness.
-- Prefer a direct local edit over new files, wrappers, managers, or speculative
-  abstractions.
-- Add a helper only when it removes current duplication, names a real domain
-  boundary, or isolates a non-trivial invariant.
-- Remove an in-scope path superseded by the change. If compatibility requires it,
-  adapt at the boundary to one canonical core and use the repository's
-  `RUSTFS_COMPAT_TODO` policy.
-- Comments explain non-obvious invariants or reasons. Do not narrate code or
-  record change history.
-- Mention unrelated problems when useful; do not fix them in a narrow task.
-
-## Reuse and Boundary Rules
-
-- Before adding helpers, constants, fixtures, or wrappers, search the touched
-  crate, the domain-owning crate, `crates/utils`, `crates/common`, and relevant
-  direct dependencies.
-- Reuse requires matching semantics: normalization, error types, deadlines,
-  durability, and compatibility must fit the call site. A narrowly named local
-  helper is better than forced reuse with different semantics.
-- Validate untrusted input at its trust boundary, then trust the validated type.
-  Values crossing disk, RPC, persistence, or version boundaries remain
-  untrusted at every consumer.
-- Re-check boundary values immediately before destructive actions such as
-  delete, overwrite, or quorum decisions.
-- Every new branch needs a concrete triggering input/state. For decoded or peer
-  data, corruption and mixed-version input are valid triggers.
-- Required values must return a typed error when absent or corrupt; do not use a
-  default that converts corruption into a plausible result.
-- Attach error context once where it is actionable. Do not erase typed errors
-  below aggregation or quorum layers.
-
-## Rust Boundaries
-
-- Production `unwrap`/`expect` must follow a type guarantee or checked invariant; explain only non-obvious guarantees.
-- Public APIs do not return `Result<_, String>`. Library APIs use domain errors unless intentional erasure is part of the boundary contract; expose a stored inner error through `Error::source()`.
-- Numeric conversions must not silently truncate. Use fallible conversions for untrusted values; validate finiteness, sign, and range before float-to-integer conversion. Clamp/saturate only when required by the domain.
-- Do not add crate-root `#![allow(dead_code)]`.
-
-## Naming
-
-Use Rust API naming: `SCREAMING_SNAKE_CASE` constants/statics, `snake_case`
-functions/variables, and `PascalCase` types. Do not rename unrelated existing
-violations.

.agents/references/pull-requests.md

@@ -1,51 +0,0 @@
-# Git and Pull Request Rules
-
-Applies to commits, pushes, PRs, and issue/discussion actions. Paths below are
-repository-relative. User authorization and root `AGENTS.md` still govern scope.
-
-## Final PR Preflight
-
-Before creating or updating a PR, reuse completed review and verification:
-
-- Verify the actual base (normally `origin/main`) and the complete task diff,
-  including file names and whitespace. Exclude secrets, logs, generated
-  artifacts, and unrelated edits. Retain an existing PR's base unless requested.
-- Confirm the final diff passed the root verification tier; fix task-owned
-  failures and run missing scoped checks. Report unresolved required checks or
-  authority without expanding the task. Do not start another general review.
-- Keep the English Conventional Commit title at most 72 characters. Use the
-  template headings, actual checks, material risks, and rollback notes.
-- Immediately before writing to GitHub, confirm the head and task diff are
-  unchanged. Rerun only checks invalidated by edits or relevant state changes.
-
-## Pull Request Lifecycle
-
-- Creating or updating a PR includes one immediate snapshot of checks,
-  mergeability, reviews, and unresolved threads.
-- Unless the user explicitly requests monitoring, a release workflow requires
-  it, or an automation already owns it, hand off after the PR is open with the
-  current state and next event to watch. Do not delay ordinary handoff with
-  fixed quiet-period sleeps.
-- For requested monitoring, use event-driven or bounded waits. Report only state
-  changes, actionable failures, or a meaningful prolonged delay.
-- Investigate failures/comments before changing code. Fix task-attributable
-  issues, rerun affected verification, push, reply or resolve the thread, then
-  resume the requested monitor.
-- Never merge without required reviewer approval or explicit authority.
-- After an observed merge, verify the commit reached the base, then clean the
-  task worktree/branch when safe. Preserve unmerged work for closed PRs unless
-  deletion was explicitly authorized.
-
-## Git and PR Baseline
-
-- Follow Conventional Commits; keep the subject at most 72 characters.
-- Source comments, commits, PR titles, and PR bodies are in English.
-- Keep every heading from `.github/pull_request_template.md`; use `N/A` where
-  needed and include commands actually run.
-- Use `--body-file` for multiline `gh pr create`/`gh pr edit` content.
-- PR/issue/discussion content must not contain the literal sequence `\n` or
-  hard-wrapped prose paragraphs.
-- Do not include local absolute paths or tool-specific labels/prefixes in GitHub
-  content.
-- Resolve review threads after the underlying issue is fixed. If declining a
-  suggestion, reply with a short evidence-based reason.

.agents/skills/adversarial-validation/SKILL.md

@@ -1,48 +0,0 @@
----
-name: adversarial-validation
-description: Review RustFS diffs or designs for explicit adversarial requests, high-risk changes under the repository review policy, or substantial PR reviews. Skip ordinary questions, diagnosis, planning, status, routine low-risk implementation, and prose with no execution effect.
----
-
-# RustFS Adversarial Validation
-
-Use the [repository risk tiers and review shape](../../references/adversarial-validation.md). This skill
-routes a review to RustFS-specific probes without loading unrelated domains.
-
-## Select Lenses
-
-Read only the references required by the diff:
-
-| Lens | When to read |
-|---|---|
-| [Correctness](references/correctness.md) | Every non-exempt adversarial review |
-| [Simplicity](references/simplicity.md) | Mechanical/standard changes and production growth |
-| [Test coverage](references/test-coverage.md) | Behavior or test changes |
-| [Security](references/security.md) | Authn/authz, IAM, RPC trust, paths, secrets, parsing, browser, encryption |
-| [Concurrency/durability](references/concurrency-durability.md) | Async shared state, locks, storage commit, cancellation, persisted queues |
-| [Compatibility](references/compatibility.md) | S3 surface, MinIO interop, metadata, wire/disk formats, mixed versions |
-| [Performance](references/performance.md) | Request/object hot paths, allocation, blocking work, fsync, fan-out |
-
-Do not read all references as a precaution. A path name alone is insufficient;
-the changed behavior must touch the lens's domain.
-
-For a dedicated security audit or advisory analysis, use
-`security-advisory-lessons` instead of loading it automatically during every
-adversarial review.
-
-## Review Protocol
-
-1. Freeze the exact final diff/head (or the design under review) and list the
-   selected lenses.
-2. Run the review shape required by the repository risk tier.
-3. For each selected lens, either report a concrete finding or a null verdict
-   naming the attacks performed.
-4. Apply root `AGENTS.md`'s finding standard. Test each candidate against callers,
-   existing coverage, and invariants before accepting it; an adversarial role
-   does not have to produce a defect.
-5. Fix or rebut supported findings with code-path, test, or invariant evidence.
-6. After a non-trivial edit, rerun only lenses affected by that edit against the
-   new exact diff.
-
-Do not turn a null verdict into a long checklist. Record concise evidence that
-the relevant failure classes were attacked, then stop under the root completion
-rule. Keep the required per-lens verdicts for high-risk PRs.

.agents/skills/adversarial-validation/references/compatibility.md

@@ -1,38 +0,0 @@
-# Compatibility Lens
-
-- Internal metadata uses `metadata_compat` helpers for dual RustFS/MinIO keys,
-  including mixed casing and removal of both twins.
-- Binary UUID metadata treats absent, empty, and nil as no value. Unversioned
-  remote tiers receive no `versionId`; versioned purge requests retain the real
-  version ID.
-- `xl.meta` changes preserve supported header/meta versions, recompute
-  signatures, decode legacy fixtures, and remain readable by old RustFS/MinIO.
-- Foreign/corrupt metadata validates parallel array lengths and missing fields;
-  it returns a decode error rather than indexing, panicking, or fabricating data.
-- Do not “correct” byte-for-byte MinIO ports without legacy fixture evidence.
-  Bitrot framing, shard math, distribution, and inline prefixes are contracts.
-- Client-visible metadata/events strip both internal prefixes
-  case-insensitively.
-- Proto fields are appended, never reused/renumbered; FlatBuffers tables extend
-  compatibly and absent new fields fail closed where authorization/quorum is
-  involved.
-- Replay real client request shapes and exact pagination boundaries for S3
-  handler changes.
-- Bucket metadata/IAM/config parsing remains compatible with pinned real MinIO
-  fixtures and encrypted migration data.
-- Compatibility shims use `RUSTFS_COMPAT_TODO(<task-id>)`, have a removal
-  condition, and default toward reading old data safely.
-
-## Outbound targets
-
-- A change to what the replication or migration client sends by default
-  (checksum policy, payload framing, headers, version-id addressing) is judged
-  against every target class, not the one it fixes. Name each target-side rule
-  the current default satisfies — checksum required with Object Lock
-  parameters, `aws-chunked` decoding, version-id adoption, ETag equals content
-  MD5 — and show which cell of
-  `crates/e2e_test/src/replication_target_matrix_test.rs` covers each.
-- A test that asserts the fix ("no trailer header") is not evidence; the
-  matrix cell that asserts the target accepted and stored the object is.
-- Every new environment escape hatch appears in
-  `docs/operations/replication-outbound-transport.md` in the same diff.

.agents/skills/adversarial-validation/references/concurrency-durability.md

@@ -1,27 +0,0 @@
-# Concurrency and Durability Lens
-
-- For every changed lock, enumerate overlapping lock sets and construct the
-  ABBA interleaving. Multiple-lock order must be documented and consistent.
-- Mark guard lifetimes and every `.await`, disk,
-  and RPC call; estimate contention/timeouts. Tokio read/write guards across
-  `.await` need bounded hold time; long-lived reads can wedge writers (#4195).
-  Keep `std::sync::Mutex` holds brief and never across `.await`.
-- Use direct `fetch_*` for unconditional atomic read-modify-write and
-  `compare_exchange` for conditional updates.
-- Object commits remain fenced if the distributed lock is lost after shard
-  writes and before metadata rename.
-- For write/rename changes, trace `write tmp -> sync tmp -> rename -> sync parent
-  -> sync required ancestors`; simulate a crash after each step and honor the
-  configured durability gate.
-- Multi-disk fan-out counts every result. Quorum-minus-one cannot become success;
-  heal remains best-effort per target where that is the established contract.
-- At every new cancellable await between mutation and cleanup/commit, drop the
-  future and inspect leftover files, counters, permits, and replay state.
-- Multipart operations on the same upload ID are serialized where required;
-  abort/complete/list races cannot delete parts before durable commit.
-- Post-commit cleanup is best-effort, retry-safe, and cannot fail an already
-  committed write or delete the last surviving copy.
-- Persisted read-modify-write uses serialization/CAS. Queue replay is crash-safe
-  and duplicate delivery has an idempotency contract.
-- Streaming reconstruction failures after partial output surface as errors, not
-  successful EOF.

.agents/skills/adversarial-validation/references/correctness.md

@@ -1,31 +0,0 @@
-# Correctness Lens
-
-Attack the changed behavior, not every subsystem in the repository.
-
-- Trace new error paths to the caller. Inject the ignored/wildcard variants and
-  verify they cannot become success, not-found, or a plausible default.
-- Exercise zero/empty/missing, maximum, and exact-boundary inputs for every
-  changed count, size, index, page limit, or optional value.
-- For aggregation/quorum changes, test exactly quorum and quorum-minus-one with
-  mixed disk errors and nil/placeholder entries.
-- For listing/pagination, test `n == max`, `n == max + 1`, delimiter folding,
-  continuation markers, and object/prefix name collisions.
-- For EC/read/streaming changes, inject failure after partial output and verify
-  the client receives an error rather than a clean truncated body. Assert exact
-  bytes and length.
-- For multipart/object commits, fail before/after rename and cleanup; committed
-  data must remain readable and pre-commit cleanup must not destroy parts.
-- Bound recursion over untrusted or persisted input; handle corrupted/cyclic
-  tree and cache traversals safely.
-- For version/index ordering, test `len - 1`, `len`, equal timestamps, missing
-  versions, and deterministic tie-breaking.
-- For directory-object behavior, trace `__XLDIR__` at the store layer; branches
-  below the layer that sees trailing slashes are dead.
-- For binary UUID metadata, absent, empty, and nil all mean no value. Never send
-  nil/empty `versionId` to an unversioned tier.
-- For agent rules/skill routers, test a trigger matrix covering ordinary
-  inquiry, low-risk implementation, explicit review, high-risk code, PR
-  creation, release, and post-PR monitoring. Each case must select only the
-  intended workflow and retain required safety/authorization boundaries.
-
-Null verdicts name only the probes relevant to the diff.

.agents/skills/adversarial-validation/references/performance.md

@@ -1,20 +0,0 @@
-# Performance Lens
-
-- For added clones/allocations on request/object/block paths, quantify copied
-  data and frequency. Recommend borrowing, move, `Bytes`/`Arc`, `Cow`, or
-  capacity reservation only for a concrete repeated cost.
-- Route every new sync/flush through the durability-mode and bucket override
-  gates; mode `none` must not pay the new fsync.
-- Keep blocking filesystem/CPU work off async runtime threads, but do not split
-  one small operation into many `spawn_blocking` round trips.
-- Measure lock hold time across I/O and compare acquisition order for ABBA.
-- Keep cleanup, extra stat/rename, and diagnostics out of the PUT commit critical
-  section when they need not be there.
-- Detect per-item serial I/O/RPC in batch APIs and accidental quadratic scans;
-  use a gate or bounded concurrency when the concrete fan-out warrants it.
-- Count buffer growth and byte copies in EC/bitrot paths; preserve pool gauge
-  balance and avoid repeated metadata decode/fetch per object.
-- Repetitive success logs stay at `trace`; metrics/instrumentation on hot paths
-  require an existing gate.
-- Claims of no impact on PUT/GET/commit/erasure paths need relevant benchmark or
-  A/B evidence, especially for 4 KiB objects.

.agents/skills/adversarial-validation/references/security.md

@@ -1,31 +0,0 @@
-# Security Lens
-
-Use `security-advisory-lessons` only for a dedicated advisory/security audit.
-For an ordinary matched diff, attack these boundaries:
-
-- Admin routes: route registration, whitelist, handler authn, and the exact
-  `AdminAction` must agree. Read-only diagnostics still require admin authz.
-- IAM/service accounts: treat parent, claims, keys, groups, status, and policy
-  names as attacker-controlled; prove ownership/root authority before writes.
-- Protocol frontends: every changed/sibling command authorizes the matching S3
-  action before reaching storage.
-- Secrets/signatures: use constant-time comparison, normalize public failures,
-  keep RPC/root/STS keys independent, and fail closed when secrets are absent.
-- RPC: bind signatures to the exact method/path and timestamp; reject replay,
-  stale, malformed, truncated, and invalid-enum payloads without panic.
-- Paths/object/archive entries: reject traversal, absolute/platform escapes,
-  and normalization differences between authz and storage.
-- Copy/multipart/presigned POST: enforce source, destination, version-aware
-  actions, copy-source conditions, and every signed policy condition.
-- Logging/errors: never expose credentials, tokens, expected signatures, raw
-  secret-bearing input, or merged configs—including via `Debug` and parse errors.
-- Untrusted serde: reject unknown fields where compatible and validate
-  security-critical defaults/ranges before numeric conversion.
-- SSE/browser/CORS/trusted proxy: inspect stored ciphertext and wrapper order;
-  isolate user content; never reflect credentialed arbitrary origins or trust
-  forwarded identity from direct clients.
-- Object Lock: unreadable/fabricated/unparsable metadata fails closed across
-  foreground, lifecycle, scanner, and force-delete paths.
-
-Security findings distinguish unauthenticated compromise from a
-low-privileged authenticated bypass.

.agents/skills/adversarial-validation/references/simplicity.md

@@ -1,22 +0,0 @@
-# Simplicity Lens
-
-- Compare the production diff with the smallest equivalent local edit. Fewer
-  lines alone are not evidence; the replacement must preserve correctness,
-  compatibility, readability, and real boundaries.
-- Search the touched crate, domain owner, `crates/utils`, `crates/common`, and
-  relevant dependencies for each new helper, constant, wrapper, or fixture.
-- Reject forced reuse when normalization, error, deadline, or durability
-  semantics differ.
-- Require a concrete trigger for every new defensive branch. Keep boundary
-  checks for disk/RPC/version data and checks immediately before destructive
-  actions.
-- Flag one-caller helpers only when they merely forward or split a short linear
-  flow without adding domain naming, invariant isolation, or useful context.
-- Ensure a replacement removes the superseded in-scope path or keeps one
-  canonical core behind a documented compatibility adapter.
-- Remove narration/change-history comments; preserve concise safety, lock,
-  durability, and compatibility invariants.
-- Treat tests, fixtures, generated code, and documentation separately from
-  production growth. Do not optimize away meaningful regression coverage.
-
-A finding must include a concrete smaller design, not a style preference.

.agents/skills/adversarial-validation/references/test-coverage.md

@@ -1,26 +0,0 @@
-# Test-Coverage Lens
-
-- For every behavior claim, name the focused test/check that fails if the
-  changed hunk is reverted. If none is practical, require the reason and
-  residual risk.
-- Confirm tests exercise the real production path and distinguish the intended
-  behavior from the named regression. A success, `is_err()`, or no-panic check
-  can be sufficient when that is the actual contract; require exact values,
-  bytes, state, or error variants when those distinctions matter to the change.
-- For new flags/modes, verify each branch and ask which test fails if the branch
-  is inverted.
-- For new error propagation, inject the failure and assert the caller observes
-  it; mentally replacing `?`/`return Err` with success must break a test.
-- Streaming GET tests assert the complete body and length under degraded reads.
-- Disk/wire-format tests use pinned foreign/legacy fixtures; same-code
-  round-trips are insufficient for compatibility.
-- Concurrency tests use readiness polling, isolate global state, and avoid fixed
-  sleeps or unrealistically short timeouts. Use nextest groups when process-level
-  serialization is required.
-- Internal metadata tests assert both RustFS and MinIO keys, not only read-back
-  through a helper that prefers one key.
-- Avoid duplicate tests of the same production path and poison-value class.
-  Boundary companions are distinct coverage: `n == max` vs `max + 1`, and
-  absent vs empty vs nil UUID.
-- A focused test proves only the targets/features it builds. Add compilation or
-  Clippy only for uncovered changed targets.

.agents/skills/issue-triage/SKILL.md

@@ -1,13 +0,0 @@
----
-name: issue-triage
-description: Verify issue completion against current implementation and merged work. Use for issue triage; comment, close, or change labels only when authorized.
----
-# Issue Triage
-
-- Resolve the issue repository separately from the implementation repository: `rustfs/backlog` tracks work in `rustfs/rustfs`. Pass `--repo` explicitly to GitHub queries.
-- Read the issue body, relevant comments, linked PRs, checklist, and sub-issues with `gh issue view`. Search explicit links, the full issue URL, qualified references, and subject keywords; an empty search result does not prove implementation is absent.
-- Fetch the implementation base and inspect its current code. For each merged candidate, verify `git merge-base --is-ancestor <merge-commit> <remote>/<base>`; a title, commit message, or merged state alone is insufficient. A local checkout may be stale or on another branch.
-- Verify every checklist/sub-issue before declaring completion. For batches, paginate the entire requested issue scope, exclude PR entries, and filter by author only when requested.
-- Recommend closing only when all requested behavior is present or evidence shows the issue is superseded. Otherwise name what remains. A status request is read-only; reuse existing authority for comments/closing/labels, and follow [Git and PR rules](../../references/pull-requests.md) before posting. Use Chinese for `rustfs/backlog`, existing repository labels only, and a closing comment naming the verified behavior and PRs.
-
-Report the issue, current state, implementation evidence, remaining work, verdict, and action actually taken. Use a table for batches. Write multiline comments via `--body-file`; when closing, post the prepared comment before closing without an inline multiline body.

.agents/skills/plugin-contract-guard/SKILL.md

@@ -1,83 +0,0 @@
----
-name: plugin-contract-guard
-description: Guard changes to target-plugin manifests, extension schemas, admin catalog/instance contracts, secret redaction, and external-plugin install policy. Use when a diff changes those contracts in crates/targets, crates/extension-schema, or admin plugin/extension handlers; path membership alone, comments, and unrelated runtime internals do not trigger it.
----
-
-# Plugin & Extension Contract Guard
-
-The "plugin system" spans four surfaces that must stay consistent:
-
-| Surface | Location |
-|---|---|
-| Manifests & registry | `crates/targets/src/{manifest,plugin}.rs` |
-| Install/enable planning (control plane) | `crates/targets/src/control_plane.rs` |
-| Extension schemas | `crates/extension-schema/src/lib.rs`, `crates/targets/src/catalog/extension.rs` |
-| Admin API contract | `rustfs/src/admin/plugin_contract.rs`, `handlers/{plugins_catalog,plugins_instances,extensions,target_descriptor}.rs` |
-
-## Hard invariants (verify before merging)
-
-1. **Secrets have one source of truth.** Secret config keys are declared only
-   in the plugin manifest (`TargetPluginManifest.secret_fields`,
-   `crates/targets/src/manifest.rs`) and flow to admin via
-   `AdminTargetSpec.secret_fields`. Never add a hand-maintained per-service
-   secret table in a handler; if redaction misses a field, fix the manifest.
-
-2. **Redaction must round-trip.** Instance GET responses replace secret values
-   with `***redacted***` (`REDACTED_SECRET_VALUE` in `plugins_instances.rs`).
-   Instance PUT restores the stored secret when it receives that placeholder
-   back (`restore_redacted_secret_values`). Any new read or write path for
-   target config must keep both halves: redact on the way out, restore the
-   placeholder on the way in. The placeholder literal must never be persisted.
-
-3. **Fixtures never reach production responses.**
-   `example_external_webhook_plugin()` (`crates/targets/src/catalog/mod.rs`)
-   is a test/demo fixture for control-plane planning tests. Production
-   catalog/extension handlers must not include it; regression tests
-   (`plugin_catalog_never_exposes_example_or_external_fixtures`,
-   `extension_catalog_never_exposes_example_or_external_fixtures`) enforce it.
-
-4. **External plugin flow is planning-only and deny-by-default.**
-   `plan_external_target_plugin_action` returns decisions, it executes
-   nothing. `TargetPluginExternalFlowGate::default()` is fully closed and
-   `TargetPluginInstallPolicy::default().allowed_download_hosts` is empty —
-   keep it that way; tests opt in via explicit policies. Install validation
-   requires https, an allowlisted host, a full 64-hex-char sha256 digest,
-   signature and provenance URIs, and an artifact matching the host
-   `target_triple`.
-
-5. **Custom target types must not collide.** Unknown target types get an
-   interned unique `custom:<type>` plugin id (`custom_plugin_id` in
-   `manifest.rs`). Custom plugins with secrets must register via
-   `TargetPluginDescriptor::with_manifest` and declare `secret_fields`;
-   `::new` derives a manifest with no secrets.
-
-## Changing the admin JSON contract
-
-- Shapes are locked twice in `plugin_contract.rs` tests: insta snapshots
-  (`rustfs/src/admin/snapshots/`) plus literal `json!` assertions. Update
-  both deliberately; a shape change is a console-facing API change.
-- Field naming is `snake_case`, except discovery blocks
-  (`runtimeCapabilities`, `clusterSnapshot`, `extensionsCatalog`) which are
-  camelCase **by cross-endpoint convention** (same shape in `system.rs`,
-  `console.rs`, `pools.rs`). Do not "fix" that inconsistency locally.
-- Contract types deliberately duplicate `rustfs_targets` types
-  (anti-corruption layer). Add a `From` impl; do not serialize internal
-  types directly.
-
-## Handler conventions
-
-- Every new admin plugin/extension route needs authorization at the top of
-  `call` and an `include_str!` guard test asserting it (repo-wide pattern —
-  see `plugin_instance_handlers_require_admin_authorization_contract`).
-- Reads use `GetBucketTargetAction` (instances) or `ServerInfoAdminAction`
-  (catalogs); writes use `SetBucketTargetAction`.
-- Refresh persisted module switches once per request
-  (`refresh_persisted_module_switches`), then evaluate the sync
-  `module_disabled_block_reason` per domain — do not re-read the store per
-  domain or per instance.
-
-## Generic bounds
-
-Event-payload generics use the `PluginEvent` blanket trait
-(`crates/targets/src/plugin.rs`). Do not respell
-`Send + Sync + 'static + Clone + Serialize + DeserializeOwned`.

.agents/skills/pr-review/SKILL.md

@@ -1,20 +0,0 @@
----
-name: pr-review
-description: Review a GitHub PR using its exact base/head and repository risk policy. Use for requested PR reviews; publish or fix only within conversation authorization.
----
-# PR Review
-
-1. Resolve the PR repository and remote; the current checkout may belong elsewhere. Read the PR purpose and linked issues, then fetch the actual base and PR head:
-
-   ```bash
-   gh pr view <N> --repo <owner/repo> --json title,author,state,body,baseRefName,headRefName,baseRefOid,headRefOid
-   git fetch <repo-remote> <baseRefName> refs/pull/<N>/head
-   git diff <baseRefOid>...<headRefOid>
-   ```
-
-   Record both SHAs; refresh if either moved during fetching. Follow relevant [implementation boundaries](../../references/implementation.md).
-2. Apply the [risk policy](../../references/adversarial-validation.md). Ordinary reviews use the root finding standard directly. For explicit adversarial, substantial, or high-risk reviews, use [the domain probes](../adversarial-validation/SKILL.md). File count does not authorize delegation; use exactly two independent reviewers only when authorized, otherwise two fresh sequential passes for that risk tier. Reviewers do not spawn further agents.
-3. Check `gh pr checks <N> --repo <owner/repo>`. Investigate failures when relevant to a finding or a requested CI/readiness diagnosis. Distinguish pending, unavailable, pre-existing, flaky, and PR-caused states using evidence; a code review does not authorize unrelated repairs.
-4. Report the reviewed SHAs, risk tier, supported findings or `No findings`, material verification limits, check state, and verdict. P0/P1 findings block approval. Severity: P0 = demonstrated data loss, security breach, remote crash, or deadlock; P1 = correctness, compatibility, or material hot-path regression; P2 = a concrete maintainability defect; P3 = optional style, only when requested.
-5. When posting is authorized, follow [Git and PR rules](../../references/pull-requests.md), recheck the remote head, and review any delta before submitting. Use `gh pr review` with `--approve`, `--request-changes`, or `--comment` and `--body-file`; use [the API example](references/posting.md) for authorized inline comments. Reuse existing authorization.
-6. Follow the repository PR lifecycle and explicit monitoring scope. Fetch new heads before comparing to the reviewed SHA; never rely on an unfetched `origin/pull/<N>/head`. Push fixes or resolve threads only when authorized; check `maintainerCanModify` before a fork push.

.agents/skills/pr-review/references/posting.md

@@ -1,22 +0,0 @@
-# Inline PR Review Submission
-
-Read only when an inline review is authorized. Recheck the PR head before posting and bind the review to the reviewed commit.
-
-For inline comments on specific lines, use the GitHub API:
-```bash
-cat > /tmp/pr_review.json <<'EOF'
-{
-  "commit_id": "<reviewed-head-sha>",
-  "body": "review body",
-  "event": "REQUEST_CHANGES",
-  "comments": [
-    {
-      "path": "crates/foo/src/bar.rs",
-      "line": 42,
-      "body": "finding description"
-    }
-  ]
-}
-EOF
-gh api --method POST /repos/{owner}/{repo}/pulls/<N>/reviews --input /tmp/pr_review.json
-```

.agents/skills/rustfs-release-publish/SKILL.md

@@ -1,42 +0,0 @@
----
-name: rustfs-release-publish
-description: Publish a RustFS version through Console, release-branch CI, preview acceptance, human confirmation, and final publication. Use only for an explicit release request.
----
-# RustFS Release
-
-Require the exact final SemVer target before version edits or publication. If unspecified, inspect current releases/tags and ask with concrete candidates; do not guess the channel or bump. A named bump type resolves to an exact version that still needs confirmation. Report the confirmed target verbatim; resolve any later mismatch before proceeding. Independent read-only preflight may continue.
-
-## Invariants
-
-- Cargo stores `<target>`, never `-preview.N`. Before publication, installation defaults retain the previous available deliverable. Final and preview binaries embed their own tag names, so version/asset names differ despite identical source.
-- Use annotated tags without `v`. Preview names are exactly `<target>-preview.<digits>`; choose the next unused iteration after fetching tags. Preview Releases are prereleases, never Latest; publish only versioned assets/exact Docker tags, never latest assets, R2, Helm, moving Docker channels, or installation/banner updates.
-- `release/<target>` owns the candidate. Retain its history, backport only reviewed fixes, and never merge main wholesale or force-push it. Pin `PREVIEW_HASH` to its exact commit with complete passing CI. Preview and final tags both point there; never tag a later main/branch HEAD or insert another bump.
-- Phase failures block dependents. Keep preview Releases until final publication and notes generation finish; CI removes Releases while retaining tags. Never move an existing preview or final tag.
-
-## Read by Phase
-
-Read only the current phase's procedure; retain completed evidence across turns. Follow root verification, Git/PR, and authorization rules.
-
-| Phase | Procedure |
-|---|---|
-| 0 | Clean tree, fetch main/tags, verify GitHub access, then complete the mandatory [Console release gate](references/console-gate.md). A green Console build alone is insufficient. |
-| 1–2 | [Prepare/reuse the release candidate and run exact-SHA branch CI](references/release-candidate.md). Source version changes use `rustfs-release-version-bump` with authorized delivery scope. |
-| 3 | [Verify the preview build, assets, channels, and notes](references/preview-release.md). |
-| 4–5 | [Run the downloaded binary, Console CRUD, and latest-rc matrix](references/preview-acceptance.md). Every check must pass. |
-| Confirmation | Complete the gate below and end the turn. |
-| 6 | After fresh confirmation, [publish and verify the final tag](references/final-release.md) at `PREVIEW_HASH`. |
-| 7 | [Maintain milestones, installation references, and the website](references/post-release-updates.md) only after complete publication. |
-
-## Human Confirmation
-
-After Phases 3–5 pass, recheck that the release branch still equals `PREVIEW_HASH`. Report the target, branch, preview tag/hash, branch CI run, preview Release URL, Console result, and rc matrix. Explicitly ask whether to publish the final tag and end the turn without creating/pushing it.
-
-Only a new affirmative user reply to that exact report authorizes Phase 6. The original release request, earlier confirmation, silence, or automated follow-up does not count. Confirmation binds the target, preview tag, and hash; a changed value or repeated/failed acceptance cycle invalidates it. Revalidate and obtain fresh confirmation before continuing.
-
-## Recovery
-
-- Before the final tag exists, land fixes on main, backport to the candidate, verify target versions, and restart branch CI/acceptance. Use a new preview iteration once a preview was tagged; main may advance independently.
-- Once the final tag exists, retry publication jobs for that exact tag. A source fix requires a newly confirmed target, not retagging or a newer preview. Phase 7 retries recheck artifact availability and superseding releases.
-- If abandoned after the bump merged, report the unpublished version on main and record whether to revert or let the next release overwrite it. Deleting candidate branches or preview tags requires an explicit decision.
-
-Report completed phase results and incomplete work with evidence: Console previous/latest tags, release need, hash and run/Release URLs; target, branch CI/SHA, previews and hash, confirmation state; preview/final assets, latest-channel state, Console/rc results and cleanup; both milestone states, installation PR checks/merge state, and website text/link/PR/deployment state. Keep artifact publication separate from follow-up delivery; do not call an open PR a live update. Report deviations and their authorization.

.agents/skills/rustfs-release-publish/references/console-gate.md

@@ -1,58 +0,0 @@
-# Console Release Gate
-
-Read during Phase 0, before changing RustFS version files or tags. Follow the parent skill's release scope and authorization rules.
-
-### Console release gate
-
-Complete this gate before changing any RustFS version file or creating any RustFS tag. RustFS `build.yml` downloads the asset returned by `repos/rustfs/console/releases/latest`, so a successful Console build alone is insufficient.
-
-1. Read the latest published Console tag and compare it with Console `main`:
-
-```bash
-CONSOLE_REPO="rustfs/console"
-CONSOLE_LATEST=$(gh api "repos/${CONSOLE_REPO}/releases/latest" --jq .tag_name)
-gh api "repos/${CONSOLE_REPO}/compare/${CONSOLE_LATEST}...main" \
-  --jq '{status, ahead_by, behind_by, commits: [.commits[] | {sha, message: .commit.message}]}'
-```
-
-- `ahead_by == 0`: no merged Console change is waiting for release. Still verify the current latest asset using step 4, then continue to Phase 1.
-- `ahead_by > 0` and `behind_by == 0`: publish Console before continuing. Report the merged commits and select the next unused `vX.Y.Z` tag. Default to the next patch version when the changes are fixes or backward-compatible UI work; stop for confirmation if a minor/major bump is plausible.
-- Any diverged history or `behind_by > 0`: stop and resolve the Console release baseline explicitly. Do not guess a range or publish RustFS.
-
-2. Clone/fetch `rustfs/console` into a scratch directory and record its exact `main` commit. Before creating a tag, check for a `v*` tag or Release workflow already associated with that hash. If one is in progress, wait for it instead of creating another version:
-
-```bash
-CONSOLE_SCRATCH=$(mktemp -d)
-gh repo clone "$CONSOLE_REPO" "$CONSOLE_SCRATCH/console"
-git -C "$CONSOLE_SCRATCH/console" fetch origin main --tags
-CONSOLE_HASH=$(git -C "$CONSOLE_SCRATCH/console" rev-parse origin/main)
-git -C "$CONSOLE_SCRATCH/console" tag --points-at "$CONSOLE_HASH" 'v*'
-gh run list -R "$CONSOLE_REPO" --workflow release.yml --commit "$CONSOLE_HASH" --limit 5
-```
-
-If no release exists or is running for `CONSOLE_HASH`, create the selected annotated tag at that exact hash and push it:
-
-```bash
-git -C "$CONSOLE_SCRATCH/console" tag -a "<console-tag>" -m "Release <console-tag>" "$CONSOLE_HASH"
-git -C "$CONSOLE_SCRATCH/console" push origin "<console-tag>"
-```
-
-Console tags include the `v` prefix. Pushing the tag triggers `.github/workflows/release.yml` (`🚀 Release`). Remove `CONSOLE_SCRATCH` after the gate completes.
-
-3. Find the exact tag run and wait for completion:
-
-```bash
-gh run list -R "$CONSOLE_REPO" --workflow release.yml --branch "<console-tag>" --limit 1
-gh run watch -R "$CONSOLE_REPO" "<console-run-id>" --exit-status
-```
-
-4. Block until the published Release is non-draft, the latest endpoint returns the expected tag, and `rustfs-console-<console-tag>.zip` is uploaded, non-empty, and carries a `sha256:` digest:
-
-```bash
-gh release view -R "$CONSOLE_REPO" "<console-tag>" --json isDraft,isPrerelease,assets,url
-test "$(gh api "repos/${CONSOLE_REPO}/releases/latest" --jq .tag_name)" = "<console-tag>"
-test "$(gh api "repos/${CONSOLE_REPO}/releases/tags/<console-tag>" \
-  --jq '[.assets[] | select(.name == "rustfs-console-<console-tag>.zip" and .state == "uploaded" and .size > 0 and (.digest | startswith("sha256:")))] | length')" -eq 1
-```
-
-Treat a missing/mismatched asset, digest, latest tag, or failed/cancelled workflow as BLOCKED. Do not start Phase 1 until the Console gate passes. Record `CONSOLE_TAG`, `CONSOLE_HASH`, Console run URL, and Release URL for the final report.

.agents/skills/rustfs-release-publish/references/final-release.md

@@ -1,22 +0,0 @@
-# Final Release Publication
-
-Read only after fresh user confirmation of the preview acceptance report. The [parent confirmation gate](../SKILL.md#human-confirmation) applies.
-
-## Phase 6 — Publish the final tag on the validated commit
-
-No second source version bump. Before creating the final tag, recheck the remote `release/<target>` HEAD equals `PREVIEW_HASH`. If it moved after confirmation, the approval no longer applies; validate and obtain confirmation for a new preview. The final tag goes on the exact commit the preview validated:
-
-```bash
-git fetch origin --tags
-git rev-parse "<preview-tag>^{commit}"          # must equal PREVIEW_HASH — abort if not
-git tag -a "<target>" -m "Release <target>" "$PREVIEW_HASH"
-git push origin "<target>"
-```
-
-- CI rebuilds from the same source; the only changed input is the tag name, so the binary now self-reports `<target>`.
-- Verify the final tag's complete publication path: all matrix and release jobs green; `gh release view "<target>"` shows the full versioned and `-latest` asset set plus checksums, SBOM, and provenance; Docker and Helm workflows succeed; `latest.json` points to `<target>`. A stable target must have `isPrerelease=false` and `isLatest=true`. An alpha/beta/rc target must have `isPrerelease=true`; GitHub does not permit prereleases to be Latest, but the project `latest.json` still advances to the final non-preview target.
-- Verify the final Release body contains `## What's Changed` and a Full Changelog link. When `PREVIOUS_DELIVERABLE` exists, the link MUST be `https://github.com/rustfs/rustfs/compare/<PREVIOUS_DELIVERABLE>...<target>` and the baseline MUST equal the preview Release baseline; for example, both `1.0.0-beta.12-preview.1` and `1.0.0-beta.12` compare from `1.0.0-beta.11`.
-- Verify the preview cleanup: `cleanup-preview-releases` must succeed, `gh release view "<preview-tag>"` must then report `release not found` for every preview iteration of this target, and `git rev-parse "<preview-tag>^{commit}"` must still resolve to `PREVIEW_HASH` (the tag is kept). If the job failed, delete the leftover Releases manually with `gh release delete "<preview-tag>" --yes` and report it.
-- Optionally spot-check `./rustfs --version` from a final-tag artifact — it must report `<target>`.
-
-Never pass `--cleanup-tag` when removing preview Releases. Remove them manually only if `cleanup-preview-releases` failed, after final publication and notes generation. Record deleted Releases and surviving tags.

.agents/skills/rustfs-release-publish/references/post-release-updates.md

@@ -1,40 +0,0 @@
-# Post-release Milestone, Installation, and Website Updates
-
-Read only after the final non-preview tag has passed Phase 6. Tag creation or a green binary build alone is insufficient: the release assets, source archive, container images, Helm package, and applicable latest channels must be available. Preview tags never enter this phase.
-
-## Version milestones
-
-1. Query all pages of both open and closed milestones in the RustFS release repository, matching titles exactly to version strings without a `v` prefix. A failed or incomplete query is not evidence that a milestone is absent; resolve the lookup before making changes.
-2. Close the milestone matching the published `<target>` if it is open; an already closed milestone needs no change. If it is missing, report the gap and continue checking the next version. Do not create a substitute for the released milestone or close/reassign its issues or PRs.
-3. Use an explicitly planned next version when provided; otherwise, for a stable `X.Y.Z`, use `X.Y.(Z+1)`. For example, after `1.0.1` is published, close milestone `1.0.1` and ensure milestone `1.0.2` exists. For an alpha/beta/rc target with a trailing numeric counter, increment that counter within the same channel unless a different next version was specified. If the next version cannot be determined, ask for that version and report this follow-up as blocked. Creating a planning milestone does not confirm the target of a future release or bypass the semver gate.
-4. Reuse an existing next-version milestone, whether open or closed; do not reopen it or create a duplicate. Only when the complete query confirms it is absent, create an open milestone with the exact next-version title. Do not invent a due date or description. After an uncertain creation result, query again before retrying.
-5. Read back both milestones and report their titles, URLs, and actual states, including whether the next milestone was reused or created. A failed close, create, or verification leaves milestone maintenance incomplete; retry only the unfinished operation after refreshing live state. Continue independent installation and website work without moving or republishing the release tag.
-
-## Installation references
-
-1. Re-read the published deliverable state before edits or a retry. If a newer release has superseded this target, do not downgrade installation references or the website announcement. Report the superseding release and leave those defaults intact.
-2. Invoke `rustfs-release-version-bump` with stage `post-release`, the exact target, and the authorized delivery scope. Prepare the follow-up on current main; do not edit or move either validated release tag. Cargo versions may already be preparing the next target and must remain untouched.
-3. Verify the exact Docker image manifest, Release download links, RPM source archive, and Helm repository entry before publishing the installation PR. The chart workflow overrides chart/app versions from the final tag, pins the source checkout to that release, and waits for its default Docker image before packaging.
-4. Report the installation PR and its merge state separately from the completed release. Reuse existing merge authorization and repository checks; do not infer new merge authority from this reference.
-
-## rustfs.com top banner
-
-Use the existing announcement configuration from [website PR #112](https://github.com/rustfs/rustfs.com/pull/112). Do not add a second banner or redesign the header.
-
-1. In an isolated checkout of `rustfs/rustfs.com`, read its current `AGENTS.md`, PR template if present, `data/announcement.ts`, and the consuming header component. Search for an existing announcement PR for this target before creating another.
-2. Update `homeAnnouncement` in `data/announcement.ts` after the release and installation availability checks pass. Keep `enabled: true`, `badge: 'New'`, and the existing component. For a stable release, use:
-
-   ```ts
-   export const homeAnnouncement = {
-     enabled: true,
-     badge: 'New',
-     message: 'RustFS <target> is now available! Please',
-     linkText: 'upgrade and try',
-     href: '/download',
-   } as const
-   ```
-
-   Replace `<target>` with the exact published version. Reserve GA wording for the actual first stable release. For a final alpha/beta/rc tag, explicitly call it a prerelease; use its GitHub Release URL if `/download` does not expose that target. Never announce a `-preview.N` build here.
-3. Use the site's current package manager and required checks. For this config-only change, type-check and lint the changed file, build the site, and inspect the generated homepage for the version, visible link text, and destination. Confirm `/download` actually offers the announced version before using that link.
-4. Commit, push, and create or update the website PR when included in the authorized release scope. Include the published RustFS Release URL and checks run. Keep merge and production-deployment authority separate; reuse permission already given, but this reference grants none by itself.
-5. After an authorized merge/deployment, verify the live `https://rustfs.com/` banner and its destination. If only the PR is ready, report that state and URL; do not report the banner as deployed. A website failure does not invalidate or move the published release tag: retry this phase after resolving the failure.

.agents/skills/rustfs-release-publish/references/preview-acceptance.md

@@ -1,52 +0,0 @@
-# Preview Artifact Acceptance
-
-Read after Phase 3 succeeds. Complete every check below before the parent skill's manual confirmation gate. These checks cover the downloaded artifact, embedded Console, and latest rc client.
-
-## Phase 4 — Run the artifact locally, verify the console
-
-Work inside the session scratchpad directory; never leave stray data dirs.
-
-```bash
-gh release download "<preview-tag>" -p "rustfs-macos-aarch64-v<preview-tag>.zip" -D "$SCRATCH"
-cd "$SCRATCH" && unzip -o rustfs-*.zip
-./rustfs --version        # must report the PREVIEW TAG (build::TAG), not the Cargo.toml version, plus expected short SHA
-mkdir -p data
-RUSTFS_ACCESS_KEY=rustfsadmin RUSTFS_SECRET_KEY=rustfsadmin ./rustfs ./data
-```
-
-Defaults: S3 endpoint `:9000`, embedded console `:9001`.
-
-Checks (all must pass):
-
-- `./rustfs --version` reports the preview tag name and the short SHA of `PREVIEW_HASH`. Reporting `<target>` without the `-preview.N` suffix means the build did not embed the tag — treat as FAIL and investigate before proceeding.
-- `curl -fsS http://localhost:9000/health/ready` returns ready.
-- Startup log shows the embedded console being served (this was the regression that `fix(release): require embedded console assets` guards).
-- Open `http://localhost:9001` in the browser: login with `rustfsadmin`/`rustfsadmin`; dashboard renders without JS console errors; create a bucket, upload a file, download it back (byte-identical), delete the object and bucket. Keep the server running for Phase 5.
-
-## Phase 5 — Validate with the latest rc client
-
-`rc` is the RustFS CLI client from <https://github.com/rustfs/cli>.
-
-- Ensure the latest release is installed: compare `rc --version` against `gh api repos/rustfs/cli/releases/latest --jq .tag_name`; update via `brew upgrade rustfs/tap/rc` (or download the release binary).
-- Point it at the preview server and run the command matrix, recording PASS/FAIL per command:
-
-```bash
-rc alias set preview http://localhost:9000 rustfsadmin rustfsadmin
-rc ls preview/
-rc mb preview/rel-check
-rc cp <local-file> preview/rel-check/
-rc stat preview/rel-check/<file>
-rc cat preview/rel-check/<file>          # matches source
-rc cp preview/rel-check/<file> ./out && cmp <local-file> ./out
-rc cp -r <local-dir>/ preview/rel-check/dir/
-rc find preview/rel-check --name "*"
-rc share download preview/rel-check/<file> --expire 1h   # presigned URL fetchable via curl
-rc rm preview/rel-check/<file> && rc rm -r --force preview/rel-check/dir
-rc rb preview/rel-check
-rc admin user list preview/
-rc admin user add preview/ relcheckuser relchecksecret12
-rc admin user remove preview/ relcheckuser
-rc alias remove preview
-```
-
-- Any FAIL blocks the release. Afterwards stop the server and delete the scratch data directory.

.agents/skills/rustfs-release-publish/references/preview-release.md

@@ -1,24 +0,0 @@
-# Preview Release Verification
-
-Read for Phase 3 after the candidate passed exact-SHA CI. The [parent release invariants](../SKILL.md) apply.
-
-## Preview channels
-
-- Use `<target>-preview.N` for every target, e.g. `1.0.0-beta.10-preview.3` or `1.1.0-preview.1`.
-- The canonical suffix is exactly `-preview.<digits>`. `build.yml` recognizes it before alpha/beta/rc classification and routes it to the preview-only path; any other tag containing `-preview` fails closed instead of being treated as a release.
-- A preview Release MUST be published with `isPrerelease=true` and `isLatest=false`. Any `*-latest` preview asset or preview-triggered `latest.json`, R2, Helm, or moving Docker channel tag is a pipeline failure. Docker images may use the exact preview version tag, including its existing variant suffixes.
-- Preview Releases are cleaned up by the `cleanup-preview-releases` job after `publish-release` succeeds for the deliverable tag. It deletes every Release whose tag is exactly `<target>-preview.<digits>` and never passes `--cleanup-tag`, so the tags survive.
-
-## Phase 3 — CI and preview Release verification
-
-- Find and watch the tag build: `gh run list --workflow build.yml --branch "<preview-tag>" --limit 1` then `gh run watch <run-id>`. Every build matrix target must succeed (linux x86_64/aarch64 × musl/gnu, macos-aarch64, windows-x86_64).
-- Confirm the preview tag resolves to the Phase 2 CI-validated `PREVIEW_HASH`. If the release branch advances during acceptance, repeat Phases 2–5 with a new preview tag.
-- Confirm the Release publication jobs (`create-release`, `upload-release-assets`, and `publish-release`) succeed while `update-latest-version` is skipped.
-- Verify `gh release view "<preview-tag>" --json isPrerelease,assets,url`: `isPrerelease` must be `true`, and the Release must contain all 6 versioned platform zips, checksums, SBOM, and provenance with no `-latest` assets. Confirm `gh api repos/{owner}/{repo}/releases/latest --jq .tag_name` does not return `<preview-tag>`.
-- Record `PREVIOUS_DELIVERABLE`, selected from published Releases by `publishedAt` after excluding the current tag and every `-preview.N` tag. Verify `gh release view "<preview-tag>" --json body --jq .body` contains `## What's Changed` and, when `PREVIOUS_DELIVERABLE` exists, `**Full Changelog**: https://github.com/rustfs/rustfs/compare/<PREVIOUS_DELIVERABLE>...<preview-tag>`. For a repository with no previous deliverable, verify a Full Changelog link exists and record the GitHub baseline fallback.
-- Confirm Helm is skipped. If preview Docker images are published, confirm they use only exact preview tags (and variant suffixes), with no `latest`, `alpha`, `beta`, or `rc` channel updates. Preview validation covers the built RustFS binaries, embedded console, and rc compatibility; it does not authorize advancing any installation default.
-
-## Release notes
-
-- When a previous deliverable exists, GitHub Release notes for the preview and final tags MUST use it as their shared comparison baseline: the most recently published non-preview Release before the target. Internal `-preview.N` Releases are explicitly excluded from that selection, even when they point at the same commit as the final tag — cleanup runs after the notes are generated, so the preview Release is still present and would otherwise be picked as the baseline. If no previous deliverable exists, omit `previous_tag_name` and record that GitHub's default baseline fallback was used.
-- Generated Release notes carry a workflow-management marker so retries can repair them. Before manually curating a generated body, remove that marker; unmarked non-placeholder notes are preserved by later workflow runs.

.agents/skills/rustfs-release-publish/references/release-candidate.md

@@ -1,37 +0,0 @@
-# Release Candidate Preparation
-
-Read for Phases 1–2. The [parent release invariants](../SKILL.md) and Console gate apply.
-
-## Candidate Resume Preflight
-
-- `git status --short` clean; `git fetch origin main --tags`.
-- `gh auth status` works; confirm you can view `gh release list -L 3`.
-- Confirm the exact final target version with the user if not explicit.
-- Check whether `release/<target>` already exists before creating it; a resumed release reuses the existing branch and its history. For an in-flight release that already has a preview tag but no branch, anchor the new branch to that tag's commit only after verifying its target versions and prior acceptance evidence.
-
-## Phase 1 — Source version bump to the final target (once)
-
-- If `release/<target>` exists, verify its Cargo.toml and workspace members in Cargo.lock read `<target>` and skip the source bump, even if main has advanced. If the branch versions differ, stop and resolve the mismatch without downgrading main.
-- If no branch exists but an unfinished preview tag does, verify its commit has `<target>` in both version files and skip the bump; Phase 2 anchors the branch to that commit. Stop if those files disagree. Otherwise, if main already has `<target>` in both files, skip the bump. Installation references intentionally retain the previous published deliverable during preview; if they were advanced to an unavailable target, restore that baseline before choosing the next candidate.
-- Only when neither the release branch, an unfinished preview tag, nor current main already has `<target>`: if main is at a later version, select and verify a historical commit containing `<target>` rather than downgrading main; block if none exists. Otherwise invoke the `rustfs-release-version-bump` skill with stage `prepare`, the final `<target>` (NOT a preview version), and full GitHub flow (commit/push/PR).
-- If a bump PR was needed, get it merged into main and record its merge commit. For a new release branch when main already has `<target>`, identify the main commit that introduced those versions. Phase 2 chooses a release candidate containing that commit; an existing release branch keeps its own history and a newer main HEAD is not required.
-
-Do not wait for the moving latest main HEAD to become green. The release-branch CI in Phase 2 validates the chosen commit before a preview tag is created.
-
-## Phase 2 — Pin and validate the release candidate, then publish the preview tag
-
-- Use `release/<target>`. On the first attempt, create it from a selected main commit containing the version bump and intended changes; it need not be the latest main HEAD. On a restart, retain the existing branch and backport only reviewed fixes from main. Verify the branch's Cargo versions still equal `<target>`. If adopting an existing preview, run this branch CI before continuing its acceptance; reuse the tag only when it points to the validated SHA.
-- The current `ci.yml` runs automatically on main pushes, not release-branch pushes. Dispatch it explicitly with `gh workflow run ci.yml --ref "release/<target>"`. Locate that dispatch run, verify its `head_sha` equals the release branch's remote HEAD, and require the full expected CI job set to pass, including `Quick Checks`, `Test and Lint`, and `End-to-End Tests (full merge gate)`. Treat a missing or skipped required lane as incomplete. A successful PR check or CI run for a different SHA does not qualify. If the branch advances or a run is cancelled, validate the new SHA again. Do not tag a candidate with failing or incomplete CI.
-- Prefer protecting `release/*` against force pushes and deletion and requiring reviewed backport PRs. Check live rulesets rather than assuming the main ruleset covers release branches; report a missing release-branch rule as a gap.
-- Set `PREVIEW_HASH` to the CI-validated release-branch commit and recheck the remote branch HEAD immediately before tagging. Report the branch, commit, and CI run URL; do not derive `PREVIEW_HASH` from a later `origin/main` fetch. When adopting an existing preview tag at this SHA, skip tag creation and continue Phase 3.
-
-```bash
-git tag -a "<preview-tag>" -m "Release <preview-tag>" "$PREVIEW_HASH"
-git push origin "<preview-tag>"
-```
-
-Pushing the tag triggers `.github/workflows/build.yml` ("Build and Release"); `docker.yml` chains off it via `workflow_run`.
-
-The preview run builds versioned artifacts and publishes them in a GitHub prerelease. Docker may publish exact preview image tags. Latest-channel, R2, and Helm publication must be skipped; installation references and the website announcement stay unchanged.
-
-On a restart after a backport, re-run release-branch CI before setting the new `PREVIEW_HASH`. Use the next unused preview iteration when the validated commit differs from an existing preview tag; never move an existing tag.

.agents/skills/rustfs-release-version-bump/SKILL.md

@@ -1,24 +0,0 @@
----
-name: rustfs-release-version-bump
-description: Prepare an exact RustFS Cargo version or align installation references after publication. Use for requested version bumps or the release workflow.
----
-# Release Version Files
-
-Require an exact target; ask if missing or ambiguous. Reject targets containing `-preview`: those are tag-only. Infer `post-release` only from an explicit request or parent workflow; otherwise use `prepare`. Delivery defaults to local edit/verify; commit, push, PR, and merge follow existing authorization and [Git rules](../../references/pull-requests.md).
-
-| Stage | Files and constraints |
-|---|---|
-| `prepare` | Only `Cargo.toml` and `Cargo.lock`: workspace package, internal dependency, and workspace member versions become `<target>`. External dependencies stay unchanged. |
-| `post-release` | Only `README.md`, `README_ZH.md`, `flake.nix`, `helm/rustfs/Chart.yaml`, and `rustfs.spec`. Cargo may already target the next release; leave it untouched. |
-
-Before installation edits, verify the exact non-preview Release, downloadable assets/source archive, pullable `rustfs/rustfs:<target>` image manifest, and published Helm chart. Block on missing artifacts. Do not downgrade defaults if a newer deliverable superseded the target. Preview preparation retains the previous published installation references; follow-up commits never replace either validated tag.
-
-Packaging policy:
-
-- Docker image tags have no `v` prefix. Derive chart/app versions with `scripts/helm_chart_version.sh <target>` (`beta.N -> 0.N.0`; other versions unchanged). Helm CI derives versions from the final tag, so no early chart bump is required.
-- RPM `Version` is numeric; `Release` is the prerelease suffix or `1` for stable. Expanded `Source0` and the unpack directory must resolve to the exact target including its suffix. Add the changelog entry using current git identity, date, and target; never copy an example identity/date.
-- Changing these mappings requires explicit confirmation.
-
-Verification follows root tiers: `prepare` validates Cargo metadata with the updated lockfile and checks every workspace/internal version with no unrelated lockfile updates. `post-release` renders Helm with the verified image, runs `scripts/test_helm_chart_version.sh` for chart version edits, and checks README, Nix, and RPM references. Run `git diff --check`; report unresolved required checks as `BLOCKED` without unrelated fixes.
-
-For authorized commits, keep stages separate: `chore(release): prepare <target>` or `chore(release): align installation references for <target>`. Report target/stage, changed files, verification, material uncertainty, and actual commit/push/PR state.

.agents/skills/rustfs-release-version-bump/agents/openai.yaml

@@ -1,4 +0,0 @@
-interface:
-  display_name: "RustFS Release Bump"
-  short_description: "Prepare RustFS release branches like PR #2957."
-  default_prompt: "Use $rustfs-release-version-bump to prepare and verify an exact RustFS release-version bump."

.agents/skills/security-advisory-lessons/SKILL.md

@@ -1,41 +0,0 @@
----
-name: security-advisory-lessons
-description: Perform a dedicated RustFS security/advisory review for authn/authz, IAM, RPC trust, paths, secrets, browser isolation, encryption, Object Lock, or other security boundaries. Use only when the user requests a security/advisory review or an adversarial review explicitly escalates to the full advisory map; do not auto-load solely because code touches a sensitive path.
----
-
-# RustFS Security Advisory Lessons
-
-Use this skill as the deep security lens. For a normal adversarial review with a
-matched security surface, the concise security reference under
-`adversarial-validation` is sufficient.
-
-## Workflow
-
-1. Freeze the exact diff/head and identify the changed trust boundaries.
-2. Inspect the headings in [advisory-patterns.md](references/advisory-patterns.md),
-   then read the matching sections. Read the full map only for a broad security
-   audit. Useful headings are
-   auth/admin, IAM/STS/OIDC, policy/plugins, S3/copy/multipart, protocols, paths,
-   secrets/logging/RPC, browser/CORS/proxy, SSE, Object Lock, and serde.
-3. Trace unauthenticated, low-privilege, wrong-action/owner/bucket, malformed,
-   and default-config cases. Security decisions must fail closed.
-4. Require a focused negative regression test for the bypass/exploit form, not
-   only the intended success path. State residual risk when a test is impractical.
-5. Report proven vulnerabilities separately from defense-in-depth hardening.
-
-When advisory currentness matters, fetch the live inventory instead of treating
-the reference as a status mirror:
-
-```bash
-gh api repos/rustfs/rustfs/security-advisories --paginate \
-  --jq '.[] | {ghsa_id,state,severity,summary,updated_at}'
-```
-
-Fetch an individual advisory only when the live summary indicates a new or
-changed lesson.
-
-## Finding Standard
-
-Each finding includes severity, `file:line`, attacker prerequisites, concrete
-input/path, impact, smallest safe fix, and a regression check. Do not exaggerate
-unauthenticated impact when the actual issue requires authenticated low privilege.

.agents/skills/security-advisory-lessons/agents/openai.yaml

@@ -1,4 +0,0 @@
-interface:
-  display_name: "Security Advisory Lessons"
-  short_description: "Apply advisory lessons in reviews."
-  default_prompt: "Use $security-advisory-lessons for a dedicated RustFS security review grounded in past advisories."

.agents/skills/security-advisory-lessons/references/advisory-patterns.md

@@ -1,155 +0,0 @@
-# RustFS Advisory Pattern Map
-
-This file is a lesson map, not an advisory inventory mirror. It keeps durable security patterns distilled from RustFS GitHub Security Advisories.
-
-When current advisory state, severity, URLs, or full text matters, fetch it live:
-
-```bash
-gh api repos/rustfs/rustfs/security-advisories --paginate \
-  --jq '.[] | {ghsa_id,state,severity,summary,updated_at}'
-gh api repos/rustfs/rustfs/security-advisories/<GHSA_ID>
-```
-
-Update this file only when an advisory adds or changes a reusable lesson, affected surface, validation pattern, or regression-test expectation. Do not update it for state-only, URL-only, count-only, or timestamp-only changes.
-
-## Pattern Index
-
-### Admin authorization and route exposure
-
-- `GHSA-pfcq-4gjr-6gjm`: notification target endpoints accepted authenticated users but skipped admin authorization. Lesson: distinguish authn from authz; admin target CRUD must call the operation-specific admin authorization path.
-- `GHSA-mm2q-qcmx-gw4w`: `ListServiceAccount` used `UpdateServiceAccountAdminAction`, while update lacked target ownership checks. Lesson: exact action constants and ownership checks are both required; information disclosure can chain into secret rotation and takeover.
-- `GHSA-vcwh-pff9-64cc`: `ImportIam` checked `ExportIAMAction` for an import/write operation. Lesson: every admin handler must authorize the action it actually performs.
-- `GHSA-jqmc-mg33-v45g` and `GHSA-8784-9m7f-c6p6`: `/profile/cpu` and `/profile/memory` were whitelisted from auth and allowed expensive diagnostics plus path disclosure. Lesson: profiling/debug endpoints need admin auth, opt-in, rate limits, and non-sensitive responses.
-- `GHSA-f5cv-v44x-2xgf`: `/rustfs/admin/v3/metrics` accepted any authenticated IAM user and skipped admin authorization. Lesson: read-only metrics and diagnostic admin endpoints still require an operation-specific admin action check.
-- `GHSA-796f-j7xp-hwf4`: `/rustfs/admin/v3/list-remote-targets` checked only that credentials existed and leaked replication target credentials. Lesson: replication target reads are privileged admin operations, and stored remote credentials require strict authz plus response redaction review.
-- `GHSA-xp32-gxq2-3v52`: console license metadata endpoint was public and exposed subject and expiration fields. Lesson: management metadata endpoints should require admin auth or return only coarse public status.
-
-### IAM import, service accounts, and privilege boundaries
-
-- `GHSA-566f-q62r-wcr8`: `ImportIam` accepted attacker-controlled service account `parent`, `claims`, `accessKey`, and `secretKey`, enabling persistent backdoor accounts under root. Lesson: imported IAM payloads are untrusted data and must be validated against privilege boundaries.
-- `GHSA-3495-h8r9-gfqg`: `ExportIAM` wrote regular-user and service-account secret keys into exported ZIP data. Lesson: IAM export is a credential-disclosure boundary; redact, seal, or strictly justify every exported secret before treating export permission as safe.
-- `GHSA-5354-r3w2-34m8`: `AddServiceAccount` checked `CreateServiceAccountAdminAction` but trusted caller-supplied `target_user`, allowing service accounts under the root parent. Lesson: service-account create paths must validate parent ownership or root/admin authority, not only the create action.
-- `GHSA-xgr5-qc6w-vcg9`: `deny_only=true` skipped allow checks and let restricted service accounts mint unrestricted children. Lesson: deny-only logic must never become implicit allow for privilege creation.
-- `GHSA-mm2q-qcmx-gw4w`: leaked service account access keys plus update-without-ownership formed an escalation chain. Lesson: service-account identifiers are security-sensitive because update APIs consume them.
-
-### STS, OIDC, and federation flows
-
-- `GHSA-5qfg-mf7r-jp3w` and `GHSA-3473-5353-xhwh`: `AssumeRoleWithWebIdentity` was reachable through unauthenticated `POST /` routing and could issue temporary credentials from crafted web identity input. Lesson: every STS route needs explicit SigV4 or trusted identity-provider validation before role assumption, and unauthenticated exemptions must be narrowed to the exact action with uniform failure responses.
-- `GHSA-jxrr-r6pv-h958`: unsigned JWT issuer data was decoded before verification to select an OIDC provider, and distinguishable failures could expose provider configuration. Lesson: web-identity routing may be unauthenticated, but pre-verification claims are untrusted routing hints; bound and rate-limit the request, normalize public errors, and verify signature, issuer, audience, and expiration before issuing credentials.
-- `GHSA-ccrv-v8v9-ch9q`, `GHSA-48rf-7j3q-3hfv`, and `GHSA-xvfh-7c9g-hpw2`: service-account-controlled material could self-sign JWT session tokens with forged policy claims, and missing `exp` was accepted for service-account tokens. Lesson: session tokens must be signed by a trusted issuer/key path, enforce required claims and expiration, and reject self-signed or principal-controlled tokens.
-- `GHSA-9pjf-w3c2-m32r`, `GHSA-4x2q-cpx9-9h26`, and `GHSA-xvpm-p3f7-34c3`: public OIDC authorize/callback flows trusted request `Host` or forwarded scheme when building credential-bearing redirects. Lesson: OIDC redirects must use configured allowlisted origins and trusted-proxy handling; never derive the post-login credential destination from direct client headers.
-- `GHSA-m479-9x88-94w6`, `GHSA-frwq-mfqx-83p8`, `GHSA-q9q8-rf9r-fg9f`, and `GHSA-j5c2-hhf7-6gf5`: OIDC validation accepted attacker-controlled discovery URLs because hostname checks rejected only literal forbidden IPs, allowing DNS rebinding SSRF. Lesson: outbound federation URL validation must resolve and classify hostnames at the connection boundary and reject loopback, private, link-local, and rebound addresses.
-
-### IAM policy conditions and external policy plugins
-
-- `GHSA-6r96-hmgc-726c`: request headers collided with lowercase server-derived condition keys such as `userid`, `groups`, `versionid`, and JWT/LDAP claims. Lesson: never let caller-controlled headers append to or replace server-derived policy context; reserve trusted condition keys and keep intentional request-header keys separate.
-- `GHSA-v9cp-qfw9-9pfp`: quantified negated string conditions applied negation after aggregation, transposing `ForAllValues` and `ForAnyValue` semantics. Lesson: push negation into the per-value predicate for quantified operators and test partially overlapping multi-value sets.
-- `GHSA-5w8r-p896-6vq2`: OPA policy mode skipped `ExistingObjectTag/*` loading, so tagged objects looked untagged to external policies. Lesson: external authorization plugins need the same object-tag and request context as built-in policy evaluation before they decide.
-
-### S3 object actions, copy, multipart, and upload policy validation
-
-- `GHSA-g8w9-qw9q-fghr` and `GHSA-xm99-m3gq-83g8`: a valid presigned `PutObject` accepted extra `x-amz-*` headers omitted from `SignedHeaders`; `x-amz-copy-source` could turn that upload capability into a cross-bucket read performed as the signer. Lesson: a presigned URL is a bounded capability; inspect all received security-sensitive headers and reject unsigned ones before selecting the S3 operation or reaching storage.
-- `GHSA-3ppv-fx5m-m749`: explicit `versionId` reads and copy sources authorized `s3:GetObject` instead of `s3:GetObjectVersion`. Lesson: version-specific object access must select version-specific actions for direct reads, `CopyObject`, and `UploadPartCopy`, with tests proving the backend is not reached on denial.
-- `GHSA-gjph-xcgw-4p7w`: `PutObject` and browser POST accepted a caller-supplied `versionId` and replaced an unlocked historical version with only `s3:PutObject`. Lesson: only replication-authorized writes may choose an existing version ID; ordinary uploads must not rewrite version history.
-- `GHSA-x298-9x87-fvjq`: anonymous `ListObjectVersions` fell back to `ListBucket` and returned before public-access-block gates. Lesson: compatibility fallbacks must converge on the same post-authorization checks as direct grants, especially `RestrictPublicBuckets` and anonymous data-plane denies.
-- `GHSA-mx42-j6wv-px98`: `UploadPartCopy` missed source authorization and allowed cross-bucket object exfiltration. Lesson: multipart copy must enforce the same source and destination contract as `CopyObject`.
-- `GHSA-wfxj-ph3v-7mjf`: `UploadPartCopy` checked source and destination independently but missed destination copy-source policy constraints. Lesson: source read and destination write checks are not sufficient when policy constrains allowed copy sources.
-- `GHSA-w5fh-f8xh-5x3p`: presigned POST accepted uploads without enforcing signed policy conditions. Lesson: parse and enforce all POST policy constraints server-side, including size, key prefix, and content type.
-
-### Protocol frontends and IAM parity
-
-- `GHSA-3g29-xff2-92vp`: FTP `RETR` and `SIZE`/`MDTM` read paths authenticated the user but skipped IAM before calling storage. Lesson: non-HTTP protocol frontends must enforce the same per-operation authorization as the S3 API before backend access.
-- `GHSA-g3vq-vv42-f647`: FTPS `MKD` called `create_bucket` without checking `s3:CreateBucket`. Lesson: protocol command handlers need action-specific checks even when sibling handlers already authorize correctly.
-- `GHSA-3p3x-734c-h5vx`: FTPS and WebDAV compared secret keys with early-return string equality, while FTPS also returned distinguishable invalid-user and invalid-password failures. Lesson: password-style protocol auth needs constant-time secret comparison, indistinguishable failures where practical, and rate limiting.
-- `GHSA-rhg6-jrf2-29c4`: WebDAV, FTP/FTPS, and SFTP used identity-policy checks with empty conditions and bypassed S3 bucket-policy explicit denies and condition-key denies. Lesson: alternate listeners must evaluate the same effective policy and trustworthy request context before storage access.
-
-### Filesystem paths and object key traversal
-
-- `GHSA-pq29-69jg-9mxc`: RPC `read_file_stream` joined untrusted paths under a volume directory without canonical boundary checks. Lesson: `PathBuf::join` plus length checks are not path security.
-- `GHSA-8r6f-hmq2-28rg`: object keys containing traversal sequences bypassed bucket/object authorization when mapped to filesystem paths. Lesson: reject traversal at object-key parsing and verify final storage paths remain under the expected bucket/key root.
-- `GHSA-f4vq-9ffr-m8m3`: Snowball auto-extract accepted archive entries such as `../victim-bucket/object`, authorized the raw attacker-bucket path, then storage path cleaning crossed bucket boundaries. Lesson: archive entries become object keys and need traversal rejection plus consistent authz/storage normalization before writes.
-- `GHSA-g7fq-6xjm-mxf4`: versioned `DeleteObjects` accepted dot-segment keys, authorized the literal prefix, then wrote delete markers under a resolved sibling prefix. Lesson: validate each batch key before authorization and storage normalization, using the same object-name rules as single delete.
-
-### Secrets, defaults, and cryptographic misuse
-
-- `GHSA-j59h-h7q5-q348`, `GHSA-3wm5-wpm5-hmfm`, `GHSA-6wc8-xm48-qhmx`, `GHSA-9gf3-jx4p-4xxf`, `GHSA-63xc-c3w3-m2cf`, and `GHSA-ch63-6q4v-hwp5`: RustFS shipped known default root credentials that could authenticate to S3, admin APIs, IAM, KMS, console, and token-signing surfaces. Lesson: root credentials must be operator-provided or generated per install; known defaults and warnings are not acceptable for network-reachable deployments.
-- `GHSA-h956-rh7x-ppgj`: gRPC used the hard-coded token `rustfs rpc` on both client and server. Lesson: source-visible shared tokens are authentication bypasses.
-- `GHSA-r5qv-rc46-hv8q`: internode RPC HMAC secret fell back to the public default `rustfsadmin`. Lesson: RPC/internode auth must fail closed instead of silently using public defaults.
-- `GHSA-75fx-qg6f-8rm7` and `GHSA-68cw-96m3-h2cf`: internode RPC secrets were derivable from known root credentials, making raw storage RPC signatures forgeable when explicit RPC secrets were unset. Lesson: RPC auth keys must be independent random secrets, never derived from S3 root credentials, and raw storage RPC should not share the public S3 listener without an internode-only boundary.
-- `GHSA-m77q-r63m-pj89`: STS JWTs used the root secret key as the shared token signing key, allowing token forgery when the root secret was known. Lesson: STS signing keys need key separation, rotation, and key IDs; do not reuse root credentials for JWT/HMAC signing.
-- `GHSA-923g-jp7v-f97f`: license verification embedded a production RSA private key and used private-key decryption as authenticity. Lesson: ship verifying/public keys only and use real signature verification.
-
-### Sensitive logging and debug output
-
-- `GHSA-r54g-49rx-98cr`: STS credentials were logged at info level. Lesson: generated credentials must never be logged in plaintext.
-- `GHSA-8cm2-h255-v749`: debug logs leaked session tokens, secret keys, JWT claims, and raw STS response bodies. Lesson: redaction must cover custom `Debug` implementations and dependency response-body logging.
-- `GHSA-333v-68xh-8mmq`: invalid RPC signature logging included the shared HMAC secret and expected signature. Lesson: error paths often leak secrets; never log raw secrets or derived authenticators.
-
-### RPC input validation and panic safety
-
-- `GHSA-gw2x-q739-qhcr`: malformed gRPC `GetMetrics` payloads reached `unwrap()` on deserialization and caused remote DoS. Lesson: every network/RPC deserialization failure returns an error, not a panic.
-- `GHSA-h956-rh7x-ppgj` and `GHSA-r5qv-rc46-hv8q`: weak RPC auth increased reachability of otherwise internal handlers. Lesson: panic bugs become more severe when internode auth is weak or defaulted.
-- `GHSA-c667-rgrv-99vj`: NodeService authentication signed the service prefix instead of the concrete generated method path, so valid metadata for one RPC could be replayed to another method during the timestamp window. Lesson: RPC HMAC payloads must bind exact gRPC method path, HTTP method surrogate, timestamp, and secret.
-
-### Browser, CORS, and console isolation
-
-- `GHSA-v9fg-3cr2-277j`: object preview rendered attacker-controlled HTML in a same-origin iframe, exposing console credentials stored in `localStorage`. Lesson: user content must be origin-isolated from the console and protected with `nosniff`, CSP, and strict content-type handling.
-- `GHSA-7gcx-wg4x-q9x6`: an incomplete preview fix reintroduced extension-based PDF detection and bypassed the sandboxed fallback for attacker-controlled content. Lesson: browser-surface fixes need regression tests for alternate viewers and file-type branches, and preview trust must come from validated content type plus sandboxing rather than object names.
-- `GHSA-x5xv-223c-8vm7`: default CORS reflected arbitrary origins with credentials. Lesson: never combine reflected origins with `Access-Control-Allow-Credentials: true`; default should be fail-closed.
-
-### Trusted proxy and source IP conditions
-
-- `GHSA-fc6g-2gcp-2qrq`: `aws:SourceIp` trusted client-supplied `X-Forwarded-For` or `X-Real-IP`. Lesson: forwarded IP headers are valid only behind configured trusted proxies; direct clients use socket peer IP.
-
-### SSE and on-disk storage invariants
-
-- `GHSA-xrrf-67jm-3c2r`: SSE metadata reported encryption while reader composition bypassed `EncryptReader` and stored plaintext. Lesson: test actual bytes on disk and wrapper order, not only API metadata.
-- `GHSA-wqmc-vjgv-jrpw`: SSE-C persisted and listed the plaintext MD5 as the ETag, allowing list-only users to confirm or brute-force low-entropy content and correlate equal plaintext across keys. Lesson: encrypted-object ETags must not expose deterministic plaintext fingerprints; protect them with per-object key material and keep listing, conditional, copy, and multipart semantics consistent.
-
-### Object Lock and retention invariants
-
-- `GHSA-j548-9grx-fh4f`: Object Lock enforcement treated unreadable, fabricated, or unparsable bucket metadata as absent configuration and allowed retained objects to be deleted or expired. Lesson: retention must fail closed unless Object Lock absence is authoritative, and every delete, lifecycle, scanner, force-delete, and default-retention path needs the same state distinction.
-- `GHSA-28w4-rp2p-636w` and `GHSA-c7hx-vr97-45rg`: an authorized `DeleteBucket` with `x-rustfs-force-delete: true` removed active COMPLIANCE-retained versions even though ordinary version deletion was denied. Lesson: bucket force-delete must enforce retention across every version, including against root/owner callers.
-
-### Serde deserialization and input validation
-
-- Reject unknown fields in untrusted S3 API XML/JSON, lifecycle, policy, and replication input where compatibility permits. Check the current type and supported payload fixtures; do not infer the repository's current coverage from an older audit. Where extra fields are part of the compatibility contract, validate security-critical values explicitly.
-- `#[serde(default)]` on security-critical fields silently accepts missing values as zero/empty. Lesson: when a field has security implications (retention days, permissions, limits), validate the deserialized value explicitly rather than relying on defaults.
-- Integer fields deserialized from user input and cast with `as` (e.g., `i32 as u32`) can wrap negative values to large positives. Lesson: validate ranges before casting; use `try_into()` with a typed error, or clamp only when the domain explicitly requires saturation.
-- XML config typos (e.g., `"NoncurentDays"` instead of `"NoncurrentDays"`) are silently accepted when `deny_unknown_fields` is absent. Lesson: strict deserialization prevents silent misconfiguration that could cause data loss or unexpected retention behavior.
-
-## Useful Search Seeds
-
-Use these targeted searches when a diff touches security-sensitive code:
-
-```bash
-rg -n "validate_admin_request|check_permissions|AdminAction::|deny_only|is_allowed" rustfs crates
-rg -n "authorize_operation|FtpsDriver|SftpDriver|RETR|MKD|SIZE|MDTM|CreateBucket|GetObject|HeadObject" crates/protocols rustfs
-rg -n "UploadPartCopy|upload_part_copy|CompleteMultipart|PostObject|presign|SignedHeaders|x-amz-copy-source|collect_signed_headers|content-length-range|starts-with" rustfs crates
-rg -n "ListBucketVersions|GetObjectVersion|versionId|VersionId|ExistingObjectTag|ForAllValues|ForAnyValue|POLICY_PLUGIN|opa" rustfs crates
-rg -n "normalize_extract_entry_key|Snowball|auto-extract|PathBuf::join|canonicalize|\\.\\.|x-forwarded-for|x-real-ip|SourceIp" rustfs crates
-rg -n "DEFAULT_SECRET|DEFAULT_ACCESS|TEST_PRIVATE_KEY|rustfs rpc|RUSTFS_RPC_SECRET" rustfs crates
-rg -n "TONIC_RPC_PREFIX|verify_rpc_signature|check_auth|NodeServiceServer|x-rustfs-signature" rustfs crates
-rg -n "debug!|trace!|info!|error!|\\?resp|\\?merged_config|session_token|secret_key" rustfs crates
-rg -n "HashReader|EncryptReader|try_resolve_etag|ETag|SSE|server-side encryption|Access-Control-Allow-Credentials|Origin" rustfs crates
-rg -n "ObjectLock|object_lock|retention|COMPLIANCE|GOVERNANCE|delete_prefix|lifecycle|scanner" rustfs crates
-rg -n "deny_unknown_fields|serde.default|as u32|as usize|as i32" rustfs crates
-```
-
-## Minimum Regression Test Expectations
-
-- Authz fixes: include unauthenticated, valid low-privilege, wrong-action, correct-action, owner, non-owner, and root/admin cases as applicable.
-- Protocol frontend authz fixes: include denied `RETR`, `SIZE`/`MDTM`, `MKD`, bucket probe, and sibling allowed-operation cases; test bucket-policy explicit denies and condition-key denies on each enabled listener against an S3 control, and assert denied paths do not reach storage.
-- IAM fixes: include import/update/list service-account cases with attacker-controlled parent, claims, access key, secret key, and policy.
-- Copy/upload fixes: include cross-bucket, cross-user, source-denied, destination-denied, copy-source-condition, and multipart completion cases.
-- Presigned upload fixes: include a valid presign with extra unsigned tagging, redirect, storage-class, and cross-bucket copy-source headers; require rejection before source or destination storage access, and verify explicitly signed equivalents still work.
-- Version-action fixes: include historical UUID, explicit current version, `null`, range, partNumber, presigned, STS/session, service-account, anonymous bucket-policy, copy source, and multipart-copy source cases.
-- Version-write fixes: a `PutObject`-only principal must not replace an existing version through `PutObject` or browser POST `?versionId=`; verify replication-authorized writes and versioned, suspended, and unversioned buckets separately.
-- Policy-condition fixes: include reserved-key header collisions, missing keys, partially overlapping multi-value sets, plugin mode, and built-in policy mode.
-- Path fixes: include encoded traversal, absolute path, nested traversal, archive entries with `..`, valid object keys that resemble traversal text but should be rejected, and canonical bucket/prefix boundary checks; test per-key `DeleteObjects` errors and prefix-scoped denials on versioned buckets.
-- Logging fixes: assert redacted output for structs and response bodies that may contain credentials.
-- IAM export fixes: assert exported archives omit plaintext user and service-account secrets unless the format deliberately encrypts or seals them.
-- RPC auth fixes: include captured metadata replay across two concrete methods, stale timestamps, wrong path, wrong method surrogate, wrong secret, and valid same-method calls.
-- Browser/CORS fixes: assert no credentials on reflected/default origins, correct behavior for explicit allowlists, and no same-origin script execution for previewed object content.
-- SSE fixes: inspect stored bytes and verify API metadata, read-back behavior, and on-disk ciphertext together; for ETags, upload equal plaintext under different keys and assert list-only callers cannot derive or correlate the plaintext digest, including multipart objects.
-- Object Lock fixes: include unreadable metadata, fabricated metadata defaults, unparsable config, confirmed absent config, COMPLIANCE/GOVERNANCE retention, lifecycle expiry, scanner sweeps, and force-delete paths; an authorized force-delete must preserve active COMPLIANCE versions.

.agents/skills/tier-debug/SKILL.md

@@ -1,37 +0,0 @@
----
-name: tier-debug
-description: Debug ILM tiering / lifecycle transition issues — NoSuchVersion on tier GET, restore failures, xl.meta inspection, remote-tier versionId tracing. Use when investigating tiered/transitioned objects, warm backends, or transition metadata.
----
-
-# Tier / ILM Debugging
-
-Playbook: [docs/operations/tier-ilm-debugging.md](../../../docs/operations/tier-ilm-debugging.md).
-Read the section matching the symptom: metadata/`xl.meta`, runtime versionId,
-manual jobs, or retained-record recovery. Read the local-first expiry invariant
-before changing cleanup ordering. Before a reconcile/disposition action, read
-its entire procedure and retain its exact-evidence and confirmation gates.
-
-Quick moves:
-
-```bash
-# Inspect transition metadata on disk (one xl.meta per erasure shard disk)
-cargo run -p rustfs-filemeta --example dump_fileinfo -- "/path/to/{bucket}/{object}/xl.meta"
-
-# Trace what versionId is sent to the remote tier
-RUST_LOG=rustfs_ecstore::bucket::lifecycle=debug ./target/debug/rustfs …
-```
-
-Interpretation:
-
-- `transition_ver_id: <none>` → correct for an unversioned tier bucket; no
-  `versionId` must be sent on tier GET/DELETE.
-- `transition_ver_id: 00000000-…` (nil) → corrupt legacy write-back; readers
-  must filter it out, never send it.
-- Empty-string `transitioned-versionID` metadata under both
-  `x-rustfs-internal-*` and `x-minio-internal-*` keys → object went to an
-  unversioned tier bucket.
-
-Code entry points: `crates/ecstore/src/bucket/lifecycle/bucket_lifecycle_ops.rs`
-(ILM actions), `crates/ecstore/src/services/tier/` (warm backends),
-`crates/filemeta/src/filemeta/version.rs` (metadata read/write + regression
-tests).

.cargo/config.toml

@@ -1,33 +0,0 @@
-# Copyright 2024 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-# RustFS Cargo configuration
-
-# NOTE: `--cfg tokio_unstable` is deliberately NOT set here.
-#
-# It used to be a global `[build] rustflags` entry so that the (default-off)
-# `dial9` telemetry feature could compile. That made every build depend on
-# Tokio's unstable, non-semver API, and it broke silently whenever a caller
-# exported their own RUSTFLAGS — an environment RUSTFLAGS replaces the value
-# from this file rather than appending to it.
-#
-# The flag is now scoped to telemetry builds, which opt in explicitly:
-#
-#     make build-profiling
-#     RUSTFLAGS="--cfg tokio_unstable" cargo build --features dial9
-#
-# `crates/obs/build.rs` fails the compile if the `dial9` feature is on without
-# the flag, so the two can no longer drift apart unnoticed.
-#
-# For CPU profiling, add `-C force-frame-pointers=yes` to that RUSTFLAGS value.

.claude/skills

@@ -1 +0,0 @@
-../.agents/skills

.codex/config.toml

@@ -1,5 +0,0 @@
-# Bound individual tool outputs retained in context; retrieve relevant ranges
-# from task-owned log files when more evidence is needed.
-# https://learn.chatgpt.com/docs/config-file/config-reference
-
-tool_output_token_limit = 4000

.config/coverage-baselines.toml

@@ -1,20 +0,0 @@
-# Report-only calibration baseline from https://github.com/rustfs/rustfs/actions/runs/29394996173.
-# Update counts only with a linked coverage run and a reviewed explanation.
-phase = "report-only"
-allowed_drop_percentage_points = 1.0
-
-[crates."crates/iam"]
-covered = 5149
-count = 8131
-
-[crates."crates/kms"]
-covered = 2950
-count = 4200
-
-[crates."crates/policy"]
-covered = 4636
-count = 5464
-
-[crates."crates/crypto"]
-covered = 469
-count = 494

.config/e2e-distributed-selection.txt

@@ -1,2 +0,0 @@
-sha256-linux=233bd7a68777eda1f044b6cb77128056be2b6492583c0dfd114b21e98a6102ef
-sha256-darwin=233bd7a68777eda1f044b6cb77128056be2b6492583c0dfd114b21e98a6102ef

.config/e2e-full-selection.txt

@@ -1,2 +0,0 @@
-sha256-darwin=98dff0961c6eb21d6128bfd2d4a6100a624ddf1fb7db05dba360a5d3d0a310c2
-sha256-linux=890f5925b87e1bf3710c356253f3b2276d901e156df9ccc04ec9315374de6d22

.config/e2e-nightly-selection.txt

@@ -1,2 +0,0 @@
-sha256-darwin=efe68675f4676a0d2901242c5815b1f7af5cdec8231f7cccd408f6dacf5c75db
-sha256-linux=003ed6480935a0a95680cc061be143722fbd37bd477b8ff98607b1a064f3aef1

.config/e2e-odm-interop-selection.txt

@@ -1 +0,0 @@
-sha256=87c05c46d611ea7ed3feb5f7276bda8e5a0f70d72165d305d73a457907e7ba79

.config/e2e-protocols-selection.txt

@@ -1,2 +0,0 @@
-sha256-darwin=a2cb67d91eb0d093928371d6e84a84b2dd41b2ba975a28094e81879bd65838ad
-sha256-linux=82db0561cf9b111f11782d647d0859c9d2a44c5c68481b5e39c48505828bd392

.config/e2e-repl-nightly-selection.txt

@@ -1 +0,0 @@
-sha256=e4ee3898b840a8374cb02c64342cc15f8d7ecd1becaf4b4a2a4657290f9e7ff1

.config/e2e-smoke-selection.txt

@@ -1 +0,0 @@
-sha256=ba8f392b617cf9f2338142233a417e43927b28096635e628233ce8a60cf49d10

.config/ecstore-required-tests.json

@@ -1,392 +0,0 @@
-{
-  "lane": "ci/test-and-lint",
-  "tests": [
-    {
-      "invariant": "write-quorum",
-      "suite": "rustfs-ecstore",
-      "name": "set_disk::ops::object::inline_put_commit_path_tests::inline_put_direct_commit_accepts_exact_quorum_and_rejects_quorum_minus_one"
-    },
-    {
-      "invariant": "metadata-rollback",
-      "suite": "rustfs-ecstore",
-      "name": "set_disk::core::io_primitives::tests::write_unique_file_info_reverts_metadata_when_write_quorum_fails"
-    },
-    {
-      "invariant": "stale-writer",
-      "suite": "rustfs-ecstore",
-      "name": "set_disk::ops::object::put_object_tmp_cleanup_tests::put_object_no_lock_aborts_after_outer_namespace_lock_loss"
-    },
-    {
-      "invariant": "range-body",
-      "suite": "rustfs-ecstore",
-      "name": "set_disk::ops::object::transition_upload_integrity_tests::transitioned_compressed_object_range_get_returns_plaintext_slice"
-    },
-    {
-      "invariant": "multipart-cancellation",
-      "suite": "rustfs-ecstore",
-      "name": "set_disk::ops::multipart::tests::cancelled_complete_keeps_upload_lock_through_tail_cleanup"
-    },
-    {
-      "invariant": "list-uncommitted-version",
-      "suite": "rustfs-filemeta",
-      "name": "metacache::tests::resolve_with_write_quorum_slack_keeps_partial_latest_hidden_during_merge"
-    },
-    {
-      "invariant": "minio-object-fixture",
-      "suite": "rustfs-filemeta",
-      "name": "filemeta::test::parses_real_minio_object_xlmeta"
-    },
-    {
-      "invariant": "corrupt-part-arrays",
-      "suite": "rustfs-filemeta",
-      "name": "filemeta::test::crc_valid_but_part_arrays_corrupt_into_fileinfo_errors_not_panics"
-    },
-    {
-      "invariant": "odm-source-contract-s3",
-      "suite": "rustfs",
-      "name": "on_demand_migration::source_client::tests::s3_backend_satisfies_the_shared_backend_contract"
-    },
-    {
-      "invariant": "odm-source-contract-azure",
-      "suite": "rustfs",
-      "name": "on_demand_migration::azure::tests::azure_backend_satisfies_the_shared_backend_contract"
-    },
-    {
-      "invariant": "odm-source-contract-gcs",
-      "suite": "rustfs",
-      "name": "on_demand_migration::gcs::tests::gcs_native_backend_satisfies_the_shared_backend_contract"
-    },
-    {
-      "invariant": "rollback-inspection",
-      "suite": "rustfs-ecstore",
-      "name": "set_disk::core::io_primitives::tests::rename_rollback_incomplete_preserves_overwrite_data_dirs_and_staging"
-    },
-    {
-      "invariant": "rollback-closed-inspection-channel",
-      "suite": "rustfs-ecstore",
-      "name": "set_disk::core::io_primitives::tests::rename_rollback_closed_heal_receiver_preserves_recovery_material"
-    },
-    {
-      "invariant": "rollback-cancellation",
-      "suite": "rustfs-ecstore",
-      "name": "set_disk::core::io_primitives::tests::rename_rollback_incomplete_cancelled_task_is_not_success"
-    },
-    {
-      "invariant": "rollback-completion-equivalence",
-      "suite": "rustfs-ecstore",
-      "name": "set_disk::core::io_primitives::tests::rename_rollback_incomplete_matches_early_ack_and_full_wait_after_reopen"
-    },
-    {
-      "invariant": "rollback-physical-owner",
-      "suite": "rustfs-ecstore",
-      "name": "set_disk::core::io_primitives::tests::rename_rollback_children_keep_namespace_ownership_after_coordinator_panic"
-    },
-    {
-      "invariant": "late-ack-retains-commit",
-      "suite": "rustfs-ecstore",
-      "name": "set_disk::core::io_primitives::tests::rename_data_early_ack_post_mutation_tail_error_never_rolls_back_commit"
-    },
-    {
-      "invariant": "tail-drained-cas",
-      "suite": "rustfs-ecstore",
-      "name": "set_disk::ops::object::put_object_tmp_cleanup_tests::tail_drained_put_waits_for_tail_and_allows_immediate_cas"
-    },
-    {
-      "invariant": "seeded-put-tail-pressure",
-      "suite": "rustfs-ecstore",
-      "name": "set_disk::ops::object::put_object_tmp_cleanup_tests::seeded_put_completion_pressure_drains_every_real_tail"
-    },
-    {
-      "invariant": "borrowed-lock-tail",
-      "suite": "rustfs-ecstore",
-      "name": "set_disk::ops::object::put_object_tmp_cleanup_tests::no_lock_put_waits_for_rename_tail_under_outer_guard"
-    },
-    {
-      "invariant": "cancelled-writer-owner",
-      "suite": "rustfs-ecstore",
-      "name": "set_disk::ops::object::put_object_tmp_cleanup_tests::tail_drained_put_owned_commit_survives_waiter_cancellation"
-    },
-    {
-      "invariant": "get-body-reconstruction",
-      "suite": "rustfs-ecstore",
-      "name": "ecstore_validation_blackbox::blackbox_get_restores_body_after_one_shard_file_is_removed"
-    },
-    {
-      "invariant": "range-reconstruction",
-      "suite": "rustfs-ecstore",
-      "name": "ecstore_validation_blackbox::blackbox_range_read_restores_exact_slice_with_one_offline_disk"
-    },
-    {
-      "invariant": "multipart-abort-linearization",
-      "suite": "rustfs-ecstore",
-      "name": "set_disk::ops::multipart::tests::abort_and_complete_linearize_for_plain_sse_and_legacy_layouts"
-    },
-    {
-      "invariant": "multipart-write-quorum",
-      "suite": "rustfs-ecstore",
-      "name": "set_disk::ops::multipart::tests::abort_enforces_delete_write_quorum_boundary"
-    },
-    {
-      "invariant": "multipart-physical-owner",
-      "suite": "rustfs-ecstore",
-      "name": "set_disk::ops::multipart::tests::complete_multipart_timeout_keeps_namespace_owner_until_physical_publication"
-    },
-    {
-      "invariant": "restore-abort-cleanup",
-      "suite": "rustfs-ecstore",
-      "name": "set_disk::ops::object::transition_commit_failure_tests::multipart_restore_aborts_every_post_create_failure"
-    },
-    {
-      "invariant": "legacy-metadata",
-      "suite": "rustfs-filemeta",
-      "name": "filemeta::test::test_issue_2288_legacy_xlmeta_compatibility"
-    },
-    {
-      "invariant": "legacy-nil-version",
-      "suite": "rustfs-filemeta",
-      "name": "filemeta::test::test_into_fileinfo_reads_legacy_nil_uuid_inline_key"
-    },
-    {
-      "invariant": "commit-phase-schedule",
-      "suite": "rustfs-ecstore",
-      "name": "set_disk::core::io_primitives::tests::commit_fault_schedule_keeps_multiple_disk_phases_independent"
-    },
-    {
-      "invariant": "commit-phase-ownership",
-      "suite": "rustfs-ecstore",
-      "name": "set_disk::core::io_primitives::tests::commit_fault_schedule_rejects_duplicate_checkpoint_without_disarming_owner"
-    },
-    {
-      "invariant": "metadata-observation-all-four-slot-states-and-arrival-orders-preserve-reduction",
-      "suite": "rustfs-ecstore",
-      "name": "set_disk::core::metadata_quorum::tests::metadata_observation_all_four_slot_states_and_arrival_orders_preserve_reduction"
-    },
-    {
-      "invariant": "metadata-observation-mixed-current-identity-arrivals-preserve-selection",
-      "suite": "rustfs-ecstore",
-      "name": "set_disk::core::metadata_quorum::tests::metadata_observation_mixed_current_identity_arrivals_preserve_selection"
-    },
-    {
-      "invariant": "metadata-observation-pending-newer-version-and-same-time-different-directory-force-full-wait",
-      "suite": "rustfs-ecstore",
-      "name": "set_disk::core::metadata_quorum::tests::metadata_observation_pending_newer_version_and_same_time_different_directory_force_full_wait"
-    },
-    {
-      "invariant": "metadata-observation-duplicate-shard-cannot-supply-a-read-reserve",
-      "suite": "rustfs-ecstore",
-      "name": "set_disk::core::metadata_quorum::tests::metadata_observation_duplicate_shard_cannot_supply_a_read_reserve"
-    },
-    {
-      "invariant": "metadata-observation-null-versions-markers-and-invalid-success-keep-their-meaning",
-      "suite": "rustfs-ecstore",
-      "name": "set_disk::core::metadata_quorum::tests::metadata_observation_null_versions_markers_and_invalid_success_keep_their_meaning"
-    },
-    {
-      "invariant": "metadata-cache-real-get-and-range-preserve-body-across-read-modes",
-      "suite": "rustfs-ecstore",
-      "name": "set_disk::read::metadata_cache_tests::metadata_cache_real_get_and_range_preserve_body_across_read_modes"
-    },
-    {
-      "invariant": "late-metadata-refresh-rejects-changed-generation-and-mapped-shards",
-      "suite": "rustfs-ecstore",
-      "name": "set_disk::read::metadata_cache_tests::late_metadata_refresh_rejects_changed_generation_and_mapped_shards"
-    },
-    {
-      "invariant": "late-full-get-rejects-changed-object-generation",
-      "suite": "rustfs-ecstore",
-      "name": "set_disk::prepared_get_object_metadata_tests::non_inline_two_phase_get_rejects_generation_change_before_late_refresh"
-    },
-    {
-      "invariant": "list-objects-shared-corpus-static-namespace",
-      "suite": "rustfs-ecstore",
-      "name": "store::list_objects::differential_tests::list_objects_shared_corpus_static_namespace"
-    },
-    {
-      "invariant": "list-objects-shared-corpus-cross-subdirectory-limit",
-      "suite": "rustfs-ecstore",
-      "name": "store::list_objects::differential_tests::list_objects_shared_corpus_cross_subdirectory_limit"
-    },
-    {
-      "invariant": "list-objects-shared-corpus-max-and-max-plus-one",
-      "suite": "rustfs-ecstore",
-      "name": "store::list_objects::differential_tests::list_objects_shared_corpus_max_and_max_plus_one"
-    },
-    {
-      "invariant": "list-objects-shared-corpus-fixed-seed-marker-and-prefix",
-      "suite": "rustfs-ecstore",
-      "name": "store::list_objects::differential_tests::list_objects_shared_corpus_fixed_seed_marker_and_prefix"
-    },
-    {
-      "invariant": "list-objects-shared-corpus-offline-corrupt-and-minority-stale",
-      "suite": "rustfs-ecstore",
-      "name": "store::list_objects::differential_tests::list_objects_shared_corpus_offline_corrupt_and_minority_stale"
-    },
-    {
-      "invariant": "list-objects-shared-corpus-versions-delete-markers-and-free-versions",
-      "suite": "rustfs-ecstore",
-      "name": "store::list_objects::differential_tests::list_objects_shared_corpus_versions_delete_markers_and_free_versions"
-    },
-    {
-      "invariant": "list-objects-shared-corpus-orphan-cleanup-requires-authoritative-evidence",
-      "suite": "rustfs-ecstore",
-      "name": "store::list_objects::differential_tests::list_objects_shared_corpus_orphan_cleanup_requires_authoritative_evidence"
-    },
-    {
-      "invariant": "list-objects-shared-corpus-consumer-drop-joins-real-disk-producer",
-      "suite": "rustfs-ecstore",
-      "name": "store::list_objects::differential_tests::list_objects_shared_corpus_consumer_drop_joins_real_disk_producer"
-    },
-    {
-      "invariant": "list-objects-shared-corpus-cancelled-quorum-walk-releases-senders-and-disk-lock",
-      "suite": "rustfs-ecstore",
-      "name": "store::list_objects::differential_tests::list_objects_shared_corpus_cancelled_quorum_walk_releases_senders_and_disk_lock"
-    },
-    {
-      "invariant": "list-objects-shared-corpus-enabled-index-contracts",
-      "suite": "rustfs-ecstore",
-      "name": "store::list_objects::differential_tests::list_objects_shared_corpus_enabled_index_contracts"
-    },
-    {
-      "invariant": "borrowed-context-boundary",
-      "suite": "rustfs-ecstore",
-      "name": "set_disk::ops::object::put_object_tmp_cleanup_tests::borrowed_write_context_rejects_wrong_namespace_and_cache_flag"
-    },
-    {
-      "invariant": "borrowed-put-cancellation",
-      "suite": "rustfs-ecstore",
-      "name": "set_disk::ops::object::put_object_tmp_cleanup_tests::borrowed_tail_drained_put_keeps_owner_after_caller_cancel"
-    },
-    {
-      "invariant": "borrowed-mpu-cancellation",
-      "suite": "rustfs-ecstore",
-      "name": "set_disk::ops::multipart::tests::borrowed_complete_keeps_namespace_owner_after_caller_cancel"
-    },
-    {
-      "invariant": "pool-cas-owner",
-      "suite": "rustfs-ecstore",
-      "name": "core::pools::pools_tests::pool_meta_cas_context_retains_actual_owner_for_exact_target"
-    },
-    {
-      "invariant": "pool-cas-released-owner",
-      "suite": "rustfs-ecstore",
-      "name": "core::pools::pools_tests::pool_meta_cas_rejects_released_owner"
-    },
-    {
-      "invariant": "copy-tail-drained",
-      "suite": "rustfs-ecstore",
-      "name": "store::object::tests::borrowed_copy_tail_drained_waits_for_rename_tail"
-    },
-    {
-      "invariant": "catalog-borrowed-publication",
-      "suite": "rustfs",
-      "name": "admin::handlers::table_catalog::tests::commit_backend_borrowed_put_preserves_owner_authorization_and_precondition"
-    },
-    {
-      "invariant": "directory-marker-quorum",
-      "suite": "rustfs-filemeta",
-      "name": "metacache::tests::resolve_directory_marker_and_prefix_preserve_quorum_in_both_orders"
-    },
-    {
-      "invariant": "logical-prefix-limit",
-      "suite": "rustfs-ecstore",
-      "name": "store::list_objects::test::list_path_gather_results_counts_common_prefixes_before_page_limit"
-    },
-    {
-      "invariant": "quota-same-name-owner-and-disk-authority",
-      "suite": "rustfs-ecstore",
-      "name": "bucket::metadata_sys::tests::quota_snapshot_reuses_same_name_owner_and_rereads_disk_authority"
-    },
-    {
-      "invariant": "quota-snapshot-scope-and-owner",
-      "suite": "rustfs-ecstore",
-      "name": "bucket::metadata_sys::tests::quota_snapshot_rejects_wrong_scope_incarnation_and_transaction_guard"
-    },
-    {
-      "invariant": "quota-snapshot-lifecycle-and-incarnation",
-      "suite": "rustfs-ecstore",
-      "name": "bucket::metadata_sys::tests::quota_snapshot_rejects_lifecycle_loss_during_read_and_changed_disk_incarnation"
-    },
-    {
-      "invariant": "marker-purge-receipt-reopen",
-      "suite": "rustfs-ecstore",
-      "name": "set_disk::ops::object::marker_purge_receipt_tests::quorum_marker_purge_receipt_survives_reopen_with_stale_member"
-    },
-    {
-      "invariant": "external-minio-shards",
-      "suite": "rustfs-ecstore::pinned_erasure_fixtures_test",
-      "name": "pinned_minio_shards_decode_and_reconstruct_plaintext"
-    },
-    {
-      "invariant": "external-legacy-shards",
-      "suite": "rustfs-ecstore::pinned_erasure_fixtures_test",
-      "name": "pinned_legacy_shards_decode_and_reconstruct_plaintext"
-    },
-    {
-      "invariant": "external-shard-bitrot",
-      "suite": "rustfs-ecstore::pinned_erasure_fixtures_test",
-      "name": "pinned_erasure_shards_reject_bitrot_corruption"
-    },
-    {
-      "invariant": "external-shard-quorum-codec",
-      "suite": "rustfs-ecstore::pinned_erasure_fixtures_test",
-      "name": "pinned_erasure_shards_reject_insufficient_quorum_and_wrong_codec"
-    }
-  ],
-  "fixtures": [
-    {
-      "path": "crates/filemeta/tests/fixtures/minio/object_large_bin.xlmeta.hex",
-      "sha256": "e8093767806d701e639b48d023190e858fbc4cde69bcfd83c22af8cba8452ce5",
-      "source": "MinIO RELEASE.2025-07-23T15-54-02Z; crates/ecstore/tests/fixtures/minio/README.md"
-    },
-    {
-      "path": "crates/filemeta/tests/fixtures/minio/object_small_txt.xlmeta.hex",
-      "sha256": "2a415ad3a3be5a9440035d4026ff880e0e8c1ec1701be9f4e077734e8dce03da",
-      "source": "MinIO RELEASE.2025-07-23T15-54-02Z; crates/ecstore/tests/fixtures/minio/README.md"
-    },
-    {
-      "path": "crates/filemeta/tests/fixtures/minio/object_versioned_txt.xlmeta.hex",
-      "sha256": "7f21f50c326dd8b0228deb6dbdb7052b3d0a3f8ee6c85d43486f0e6bb7a97261",
-      "source": "MinIO RELEASE.2025-07-23T15-54-02Z; crates/ecstore/tests/fixtures/minio/README.md"
-    },
-    {
-      "path": "crates/ecstore/tests/fixtures/minio/bucket_metadata.blob.hex",
-      "sha256": "f2b6e260aff106adf6039feb1c645686e84e75404ff725491fb18668be5db203",
-      "source": "MinIO RELEASE.2025-07-23T15-54-02Z; crates/ecstore/tests/fixtures/minio/README.md"
-    },
-    {
-      "path": "crates/ecstore/tests/fixtures/minio/bucket_metadata_full.xlmeta.hex",
-      "sha256": "3b6de589519c08a1614c8bd409bb8199c17d42043861b07bce513075e6fbfc12",
-      "source": "MinIO RELEASE.2025-07-23T15-54-02Z; crates/ecstore/tests/fixtures/minio/README.md"
-    },
-    {
-      "path": "crates/filemeta/tests/fixtures/issue_2288_legacy_xlmeta.hex",
-      "sha256": "b3e8d32f9950a38b5dc71f679519cc830a2b2a807e8b0b39f78840b238d8eacf",
-      "source": "Legacy xl.meta captured for rustfs/backlog#2288; filemeta::test::test_issue_2288_legacy_xlmeta_compatibility"
-    },
-    {
-      "path": "crates/filemeta/tests/fixtures/issue_2265_legacy_meta_v2_object.hex",
-      "sha256": "89dd7ec92d16e54b5d31e0c49c9edffff8a7aa58890aaeff59bc2ff84da0a62f",
-      "source": "Legacy metadata v2 object captured for rustfs/backlog#2265; crates/filemeta/src/test_data.rs"
-    },
-    {
-      "path": "crates/ecstore/tests/fixtures/list_namespace_keys.json",
-      "sha256": "5afddb60565deba0938e617470255563f0e15a15fab699645c59bb7ccc84898e",
-      "source": "Shared static LIST namespace corpus; ECStore differential tests and ODM list_through_static_namespace_boundary_matrix"
-    },
-    {
-      "path": "crates/ecstore/tests/fixtures/erasure-shards/minio/manifest.json",
-      "sha256": "c398f8dcb76903f6cf8eee07931c76592895ced201ff9d59ed4ff9b0ead2b13c",
-      "source": "MinIO RELEASE.2025-07-23T15-54-02Z; crates/ecstore/tests/fixtures/erasure-shards/README.md"
-    },
-    {
-      "path": "crates/ecstore/tests/fixtures/erasure-shards/legacy/manifest.json",
-      "sha256": "dce397d5b2f42748eb5091b4355bb9d133a50cbba003626fafac28095dbe540b",
-      "source": "RustFS 1.0.0-alpha.37; crates/ecstore/tests/fixtures/erasure-shards/README.md"
-    },
-    {
-      "path": "crates/ecstore/tests/fixtures/erasure-shards/plaintext.bin",
-      "sha256": "b5a83332327961a31b65a8731c64559fd04ff8ac9f6124022201dbbdba5ffa01",
-      "source": "Independent fixed SHA256 counter-stream oracle; crates/ecstore/tests/fixtures/erasure-shards/README.md"
-    }
-  ]
-}

.config/functional-script-revision.txt

@@ -1 +0,0 @@
-27e9584a2d776db9416d0edfd040d524d48b5d4d

.config/make/build-docker-buildx-dev.mak

@@ -1,64 +0,0 @@
-## —— Development/Source builds using direct buildx commands ---------------------------------------
-
-.PHONY: docker-dev
-docker-dev: ## Build dev multi-arch image (cannot load locally)
-	@echo "🏗️ Building multi-architecture development Docker images with buildx..."
-	@echo "💡 This builds from source code and is intended for local development and testing"
-	@echo "⚠️  Multi-arch images cannot be loaded locally, use docker-dev-push to push to registry"
-	$(DOCKER_CLI) buildx build \
-		--platform linux/amd64,linux/arm64 \
-		--file $(DOCKERFILE_SOURCE) \
-		--tag rustfs:source-latest \
-		--tag rustfs:dev-latest \
-		.
-
-.PHONY: docker-dev-local
-docker-dev-local: ## Build dev single-arch image (local load)
-	@echo "🏗️ Building single-architecture development Docker image for local use..."
-	@echo "💡 This builds from source code for the current platform and loads locally"
-	$(DOCKER_CLI) buildx build \
-		--file $(DOCKERFILE_SOURCE) \
-		--tag rustfs:source-latest \
-		--tag rustfs:dev-latest \
-		--load \
-		.
-
-.PHONY: docker-dev-push
-docker-dev-push: ## Build and push multi-arch development image # e.g (make docker-dev-push REGISTRY=xxx)
-	@if [ -z "$(REGISTRY)" ]; then \
-		echo "❌ Error: Please specify registry, example: make docker-dev-push REGISTRY=ghcr.io/username"; \
-		exit 1; \
-	fi
-	@echo "🚀 Building and pushing multi-architecture development Docker images..."
-	@echo "💡 Pushing to registry: $(REGISTRY)"
-	$(DOCKER_CLI) buildx build \
-		--platform linux/amd64,linux/arm64 \
-		--file $(DOCKERFILE_SOURCE) \
-		--tag $(REGISTRY)/rustfs:source-latest \
-		--tag $(REGISTRY)/rustfs:dev-latest \
-		--push \
-		.
-
-.PHONY: dev-env-start
-dev-env-start: ## Start development container environment
-	@echo "🚀 Starting development environment..."
-	$(DOCKER_CLI) buildx build \
-		--file $(DOCKERFILE_SOURCE) \
-		--tag rustfs:dev \
-		--load \
-		.
-	$(DOCKER_CLI) stop $(CONTAINER_NAME) 2>/dev/null || true
-	$(DOCKER_CLI) rm $(CONTAINER_NAME) 2>/dev/null || true
-	$(DOCKER_CLI) run -d --name $(CONTAINER_NAME) \
-		-p 9010:9010 -p 9000:9000 \
-		-v $(shell pwd):/workspace \
-		-it rustfs:dev
-
-.PHONY: dev-env-stop
-dev-env-stop: ## Stop development container environment
-	@echo "🛑 Stopping development environment..."
-	$(DOCKER_CLI) stop $(CONTAINER_NAME) 2>/dev/null || true
-	$(DOCKER_CLI) rm $(CONTAINER_NAME) 2>/dev/null || true
-
-.PHONY: dev-env-restart
-dev-env-restart: dev-env-stop dev-env-start ## Restart development container environment

.config/make/build-docker-buildx-production.mak

@@ -1,41 +0,0 @@
-## —— Production builds using docker buildx (for CI/CD and production) -----------------------------
-
-.PHONY: docker-buildx
-docker-buildx: ## Build production multi-arch image (no push)
-	@echo "🏗️ Building multi-architecture production Docker images with buildx..."
-	./docker-buildx.sh
-
-.PHONY: docker-buildx-push
-docker-buildx-push: ## Build and push production multi-arch image
-	@echo "🚀 Building and pushing multi-architecture production Docker images with buildx..."
-	./docker-buildx.sh --push
-
-.PHONY: docker-buildx-version
-docker-buildx-version: ## Build and version production multi-arch image # e.g (make docker-buildx-version VERSION=v1.0.0)
-	@if [ -z "$(VERSION)" ]; then \
-		echo "❌ Error: Please specify version, example: make docker-buildx-version VERSION=v1.0.0"; \
-		exit 1; \
-	fi
-	@echo "🏗️ Building multi-architecture production Docker images (version: $(VERSION))..."
-	./docker-buildx.sh --release $(VERSION)
-
-.PHONY: docker-buildx-push-version
-docker-buildx-push-version: ## Build and version and push production multi-arch image # e.g (make docker-buildx-push-version VERSION=v1.0.0)
-	@if [ -z "$(VERSION)" ]; then \
-		echo "❌ Error: Please specify version, example: make docker-buildx-push-version VERSION=v1.0.0"; \
-		exit 1; \
-	fi
-	@echo "🚀 Building and pushing multi-architecture production Docker images (version: $(VERSION))..."
-	./docker-buildx.sh --release $(VERSION) --push
-
-.PHONY: docker-buildx-production-local
-docker-buildx-production-local: ## Build production single-arch image locally
-	@echo "🏗️ Building single-architecture production Docker image locally..."
-	@echo "💡 Alternative to docker-buildx.sh for local testing"
-	$(DOCKER_CLI) buildx build \
-		--file $(DOCKERFILE_PRODUCTION) \
-		--tag rustfs:production-latest \
-		--tag rustfs:latest \
-		--load \
-		--build-arg RELEASE=latest \
-		.

.config/make/build-docker-production.mak

@@ -1,16 +0,0 @@
-## —— Single Architecture Docker Builds (Traditional) ----------------------------------------------
-
-.PHONY: docker-build-production
-docker-build-production: ## Build single-arch production image
-	@echo "🏗️ Building single-architecture production Docker image..."
-	@echo "💡 Consider using 'make docker-buildx-production-local' for multi-arch support"
-	$(DOCKER_CLI) build -f $(DOCKERFILE_PRODUCTION) -t rustfs:latest .
-
-.PHONY: docker-build-source
-docker-build-source: ## Build single-arch source image
-	@echo "🏗️ Building single-architecture source Docker image..."
-	@echo "💡 Consider using 'make docker-dev-local' for multi-arch support"
-	DOCKER_BUILDKIT=1 $(DOCKER_CLI) build \
-		--build-arg BUILDKIT_INLINE_CACHE=1 \
-		-f $(DOCKERFILE_SOURCE) -t rustfs:source .
-

.config/make/build-docker.mak

@@ -1,22 +0,0 @@
-## —— Docker-based build (alternative approach) ----------------------------------------------------
-
-# Usage: make BUILD_OS=ubuntu22.04 build-docker
-# Output: target/ubuntu22.04/release/rustfs
-
-.PHONY: build-docker
-build-docker: SOURCE_BUILD_IMAGE_NAME = rustfs-$(BUILD_OS):v1
-build-docker: SOURCE_BUILD_CONTAINER_NAME = rustfs-$(BUILD_OS)-build
-build-docker: BUILD_CMD = /root/.cargo/bin/cargo build --release --bin rustfs --target-dir /root/s3-rustfs/target/$(BUILD_OS)
-build-docker: ## Build using Docker container # e.g (make build-docker BUILD_OS=ubuntu22.04)
-	@echo "🐳 Building RustFS using Docker ($(BUILD_OS))..."
-	$(DOCKER_CLI) buildx build -t $(SOURCE_BUILD_IMAGE_NAME) -f $(DOCKERFILE_SOURCE) --load .
-	$(DOCKER_CLI) run --rm --name $(SOURCE_BUILD_CONTAINER_NAME) -v $(shell pwd):/root/s3-rustfs -it $(SOURCE_BUILD_IMAGE_NAME) $(BUILD_CMD)
-
-.PHONY: docker-inspect-multiarch
-docker-inspect-multiarch: ## Check image architecture support
-	@if [ -z "$(IMAGE)" ]; then \
-		echo "❌ Error: Please specify image, example: make docker-inspect-multiarch IMAGE=rustfs/rustfs:latest"; \
-		exit 1; \
-	fi
-	@echo "🔍 Inspecting multi-architecture image: $(IMAGE)"
-	docker buildx imagetools inspect $(IMAGE)

.config/make/build.mak

@@ -1,74 +0,0 @@
-## —— Local Native Build using build-rustfs.sh script (Recommended) --------------------------------
-
-.PHONY: build
-build: ## Build RustFS binary (includes console by default)
-	@echo "🔨 Building RustFS using build-rustfs.sh script..."
-	./build-rustfs.sh
-
-.PHONY: build-dev
-build-dev: ## Build RustFS in Development mode
-	@echo "🔨 Building RustFS in development mode..."
-	./build-rustfs.sh --dev
-
-.PHONY: build-musl
-build-musl: ## Build x86_64 musl version
-	@echo "🔨 Building rustfs for x86_64-unknown-linux-musl..."
-	@echo "💡 On macOS/Windows, use 'make build-docker' or 'make docker-dev' instead"
-	./build-rustfs.sh --platform x86_64-unknown-linux-musl
-
-.PHONY: build-gnu
-build-gnu: ## Build x86_64 GNU version
-	@echo "🔨 Building rustfs for x86_64-unknown-linux-gnu..."
-	@echo "💡 On macOS/Windows, use 'make build-docker' or 'make docker-dev' instead"
-	./build-rustfs.sh --platform x86_64-unknown-linux-gnu
-
-.PHONY: build-musl-arm64
-build-musl-arm64: ## Build aarch64 musl version
-	@echo "🔨 Building rustfs for aarch64-unknown-linux-musl..."
-	@echo "💡 On macOS/Windows, use 'make build-docker' or 'make docker-dev' instead"
-	./build-rustfs.sh --platform aarch64-unknown-linux-musl
-
-.PHONY: build-gnu-arm64
-build-gnu-arm64: ## Build aarch64 GNU version
-	@echo "🔨 Building rustfs for aarch64-unknown-linux-gnu..."
-	@echo "💡 On macOS/Windows, use 'make build-docker' or 'make docker-dev' instead"
-	./build-rustfs.sh --platform aarch64-unknown-linux-gnu
-
-
-## —— Profiling build (dial9 Tokio runtime telemetry) ------------------------------------------
-
-# dial9 hooks Tokio's unstable runtime instrumentation, so it needs
-# `--cfg tokio_unstable`. That flag is deliberately absent from
-# .cargo/config.toml: it is not free, and release binaries do not carry it.
-# Setting RUSTFLAGS here replaces (never appends to) the config-file value, and
-# crates/obs/build.rs fails the build if the feature and the flag disagree.
-#
-# There are no task-dump or S3-upload features — see the notes in
-# crates/obs/Cargo.toml for why.
-DIAL9_FEATURES ?= dial9
-DIAL9_RUSTFLAGS ?= --cfg tokio_unstable
-
-.PHONY: build-profiling
-build-profiling: ## Build RustFS with dial9 Tokio runtime telemetry (diagnostic builds only)
-	@echo "🔬 Building RustFS with dial9 telemetry (features: $(DIAL9_FEATURES))..."
-	@echo "⚠️  Diagnostic build: telemetry writes trace segments to disk continuously."
-	RUSTFLAGS="$(DIAL9_RUSTFLAGS)" cargo build --release --bin rustfs --features $(DIAL9_FEATURES)
-
-.PHONY: build-cross-all
-build-cross-all: core-deps ## Build binaries for all architectures
-	@echo "🔧 Building all target architectures..."
-	@echo "💡 On macOS/Windows, use 'make docker-dev' for reliable multi-arch builds"
-	@echo "🔨 Generating protobuf code..."
-	cargo run --bin gproto || true
-
-	@echo "🔨 Building rustfs for x86_64-unknown-linux-musl..."
-	./build-rustfs.sh --platform x86_64-unknown-linux-musl
-
-	@echo "🔨 Building rustfs for x86_64-unknown-linux-gnu..."
-	./build-rustfs.sh --platform x86_64-unknown-linux-gnu
-
-	@echo "🔨 Building rustfs for aarch64-unknown-linux-musl..."
-	./build-rustfs.sh --platform aarch64-unknown-linux-musl
-
-	@echo "🔨 Building rustfs for aarch64-unknown-linux-gnu..."
-	./build-rustfs.sh --platform aarch64-unknown-linux-gnu

.config/make/check.mak

@@ -1,26 +0,0 @@
-## —— Check and Inform Dependencies ----------------------------------------------------------------
-
-# Fatal check
-# Checks all required dependencies and exits with error if not found
-# (e.g., cargo, rustfmt)
-check-%:
-	@command -v $* >/dev/null 2>&1 || { \
-		echo >&2 "❌ '$*' is not installed."; \
-		exit 1; \
-	}
-
-# Warning-only check
-# Checks for optional dependencies and issues a warning if not found
-warn-%:
-	@command -v $* >/dev/null 2>&1 || { \
-		echo >&2 "⚠️ '$*' is not installed."; \
-	}
-
-# For checking dependencies use check-<dep-name> or warn-<dep-name>
-#
-# NOTE: cargo-nextest is a HARD dependency of `make test`, gated inside the
-# test recipe itself (with a RUSTFS_ALLOW_CARGO_TEST_FALLBACK=1 escape hatch)
-# rather than a warn-only prerequisite here — see .config/make/tests.mak.
-.PHONY: core-deps fmt-deps
-core-deps: check-cargo ## Check core dependencies
-fmt-deps: check-rustfmt ## Check lint and formatting dependencies

.config/make/coverage.mak

@@ -1,26 +0,0 @@
-## —— Coverage --------------------------------------------------------------------------------------
-
-# Local equivalent of the weekly coverage workflow (.github/workflows/coverage.yml,
-# backlog#1153 infra-5): same measurement scope (--workspace --exclude e2e_test,
-# nextest `ci` profile) and the same per-crate table. Slow — the instrumented
-# build cannot reuse your normal target cache and then runs the whole suite.
-# Doctests are not measured (needs nightly). Outputs land in target/llvm-cov/.
-.PHONY: coverage
-coverage: core-deps ## Workspace line coverage (cargo-llvm-cov + nextest; slow, writes target/llvm-cov/)
-	@if ! command -v cargo-llvm-cov >/dev/null 2>&1; then \
-		echo >&2 "❌ cargo-llvm-cov is required for 'make coverage' but was not found."; \
-		echo >&2 "   Install it with:"; \
-		echo >&2 "     cargo install cargo-llvm-cov --locked"; \
-		echo >&2 "     rustup component add llvm-tools-preview"; \
-		exit 1; \
-	fi
-	@if ! command -v cargo-nextest >/dev/null 2>&1; then \
-		echo >&2 "❌ cargo-nextest is required for 'make coverage' (see 'make test')."; \
-		echo >&2 "   Install it with: cargo install cargo-nextest --locked"; \
-		exit 1; \
-	fi
-	NEXTEST_PROFILE=ci cargo llvm-cov nextest --workspace --exclude e2e_test --no-report
-	@mkdir -p target/llvm-cov
-	cargo llvm-cov report --lcov --output-path target/llvm-cov/lcov.info
-	cargo llvm-cov report --json --output-path target/llvm-cov/coverage.json
-	$(RUSTFS_PYTHON_BIN) scripts/coverage_per_crate.py target/llvm-cov/coverage.json

.config/make/deploy.mak

@@ -1,6 +0,0 @@
-## —— Deploy using dev_deploy.sh script ------------------------------------------------------------
-
-.PHONY: deploy-dev
-deploy-dev: build-musl ## Deploy to dev server
-	@echo "🚀 Deploying to dev server: $${IP}"
-	./scripts/dev_deploy.sh $${IP}

.config/make/help.mak

@@ -1,38 +0,0 @@
-## —— Help, Help Build and Help Docker -------------------------------------------------------------
-
-
-.PHONY: help
-help: ## Shows This Help Menu
-	echo -e "$$HEADER"
-	grep -E '(^[a-zA-Z0-9_-]+:.*?## .*$$)|(^## )' $(MAKEFILE_LIST) | sed 's/^[^:]*://g' | awk 'BEGIN {FS = ":.*?## | #"} /^## / {printf "\n${green}%s${reset}\n", $$0; next} {printf "${cyan}%-30s${reset} ${white}%s${reset} ${green}%s${reset}\n", $$1, $$2, $$3}'
-
-.PHONY: help-build
-help-build: ## Shows RustFS build help
-	@echo ""
-	@echo "💡 build-rustfs.sh script provides more options, smart detection and binary verification"
-	@echo ""
-	@echo "🔧 Direct usage of build-rustfs.sh script:"
-	@echo ""
-	@echo "	./build-rustfs.sh --help                                # View script help"
-	@echo "	./build-rustfs.sh --no-console                          # Build without console resources"
-	@echo "	./build-rustfs.sh --force-console-update                # Force update console resources"
-	@echo "	./build-rustfs.sh --dev                                 # Development mode build"
-	@echo "	./build-rustfs.sh --sign                                # Sign binary files"
-	@echo "	./build-rustfs.sh --platform x86_64-unknown-linux-gnu   # Specify target platform"
-	@echo "	./build-rustfs.sh --skip-verification                   # Skip binary verification"
-	@echo ""
-
-.PHONY: help-docker
-help-docker: ## Shows docker environment and suggestion help
-	@echo ""
-	@echo "📋 Environment Variables:"
-	@echo "	REGISTRY                 Image registry address (required for push)"
-	@echo "	DOCKERHUB_USERNAME       Docker Hub username"
-	@echo "	DOCKERHUB_TOKEN          Docker Hub access token"
-	@echo "	GITHUB_TOKEN             GitHub access token"
-	@echo ""
-	@echo "💡 Suggestions:"
-	@echo "	Production use:          Use docker-buildx* commands (based on precompiled binaries)"
-	@echo "	Local development:       Use docker-dev* commands (build from source)"
-	@echo "	Development environment: Use dev-env-* commands to manage dev containers"
-	@echo ""

.config/make/lint-fmt.mak

@@ -1,102 +0,0 @@
-## —— Code quality and Formatting ------------------------------------------------------------------
-
-.NOTPARALLEL: fix
-
-.PHONY: fmt
-fmt: core-deps fmt-deps ## Format code
-	@echo "🔧 Formatting code..."
-	cargo fmt --all
-
-.PHONY: fmt-check
-fmt-check: core-deps fmt-deps ## Check code formatting
-	@echo "📝 Checking code formatting..."
-	cargo fmt --all --check
-
-.PHONY: clippy-check
-clippy-check: core-deps ## Run clippy checks
-	@echo "🔍 Running clippy checks..."
-	cargo clippy --all-targets -- -D warnings
-
-.PHONY: clippy-fix
-clippy-fix: core-deps ## Apply clippy fixes
-	@echo "🔧 Applying clippy fixes..."
-	cargo clippy --fix --allow-dirty
-
-.PHONY: fix
-fix: fmt clippy-fix ## Format code and apply clippy fixes
-
-.PHONY: quick-check
-quick-check: core-deps ## Run fast workspace compilation check
-	@echo "🔨 Running fast compilation check..."
-	cargo check --workspace --exclude e2e_test
-
-.PHONY: unsafe-code-check
-unsafe-code-check: ## Check unsafe_code allowances have SAFETY comments
-	@echo "🔒 Checking unsafe_code allowances..."
-	./scripts/check_unsafe_code_allowances.sh
-
-.PHONY: architecture-migration-check
-architecture-migration-check: ## Check architecture migration guardrails
-	@echo "🏗️ Checking architecture migration guardrails..."
-	./scripts/check_architecture_migration_rules.sh
-
-.PHONY: logging-guardrails-check
-logging-guardrails-check: ## Check logging guardrails for redaction and noise regressions
-	@echo "🪵 Checking logging guardrails..."
-	./scripts/check_logging_guardrails.sh
-
-.PHONY: error-other-ratchet-check
-error-other-ratchet-check: ## Check the ecstore ::other(format!) quorum-bucketing ratchet stays shrink-only
-	@echo "🪣 Checking error other(format!) ratchet..."
-	./scripts/check_error_other_format_ratchet.sh
-
-.PHONY: tokio-io-uring-check
-tokio-io-uring-check: ## Check tokio io-uring runtime feature stays removed
-	@echo "🚫 Checking tokio io-uring feature guard..."
-	./scripts/check_no_tokio_io_uring.sh
-
-.PHONY: extension-schema-check
-extension-schema-check: ## Check extension-schema stays a lightweight contract crate
-	@echo "🧩 Checking extension schema boundaries..."
-	./scripts/check_extension_schema_boundaries.sh
-
-.PHONY: body-cache-whitelist-check
-body-cache-whitelist-check: ## Check the body-cache eligibility gate stays a fail-closed allow-list
-	@echo "🧱 Checking body-cache whitelist guard..."
-	./scripts/check_body_cache_whitelist.sh
-
-.PHONY: s3s-footprint-check
-s3s-footprint-check: ## Check the s3s dependency footprint ratchet stays frozen
-	@echo "📦 Checking s3s footprint ratchet..."
-	./scripts/check_s3s_footprint.sh
-
-.PHONY: fips-wording-check
-fips-wording-check: ## Check docs and crates/kms do not over-claim crypto capabilities
-	@echo "📣 Checking cryptographic capability wording guard..."
-	./scripts/check_fips_wording.sh
-
-.PHONY: embedded-secrets-check
-embedded-secrets-check: ## Check no private key material or credential literal is committed
-	@echo "🔑 Checking embedded secret material guard..."
-	./scripts/check_embedded_secrets.sh
-
-.PHONY: offline-enrollment-e2e-check
-offline-enrollment-e2e-check: core-deps ## Build and exercise the dedicated offline enrollment E2E root
-	@echo "🔐 Checking the offline enrollment E2E root boundary..."
-	./scripts/check_offline_enrollment_e2e.sh
-
-.PHONY: test-wiring-check
-test-wiring-check: ## Check tests stay registered and selected by their intended runners
-	@echo "🧪 Checking test wiring..."
-	$(RUSTFS_PYTHON_BIN) ./scripts/check_test_wiring.py
-	$(RUSTFS_PYTHON_BIN) ./scripts/ci_gate.py --check-workflow
-
-.PHONY: log-analyzer-rules-check
-log-analyzer-rules-check: core-deps ## Check log-analyzer rule anchors still exist verbatim in source
-	@echo "🩺 Checking log-analyzer rule anchors..."
-	./scripts/check_log_analyzer_rules.sh
-
-.PHONY: compilation-check
-compilation-check: core-deps ## Run compilation check
-	@echo "🔨 Running compilation check..."
-	cargo check --all-targets

.config/make/pre-commit.mak

@@ -1,32 +0,0 @@
-## —— Pre Commit Checks ----------------------------------------------------------------------------
-
-.NOTPARALLEL: pre-commit pre-pr dev-check
-
-.PHONY: setup-hooks
-setup-hooks: ## Install the configured pre-commit hooks
-	@echo "🔧 Setting up git hooks..."
-	pre-commit validate-config
-	pre-commit install
-	@echo "✅ Git hooks setup complete!"
-
-.PHONY: doc-paths-check
-doc-paths-check: ## Check that instruction/architecture docs reference existing file paths
-	@echo "📄 Checking doc path references..."
-	./scripts/check_doc_paths.sh
-
-.PHONY: planning-docs-check
-planning-docs-check: ## Check that no planning-type documents are committed
-	@echo "📄 Checking for committed planning docs..."
-	./scripts/check_no_planning_docs.sh
-
-.PHONY: pre-commit
-pre-commit: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check error-other-ratchet-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check s3s-footprint-check fips-wording-check embedded-secrets-check test-wiring-check doc-paths-check planning-docs-check quick-check ## Run fast pre-commit checks without clippy/full tests
-	@echo "✅ All pre-commit checks passed!"
-
-.PHONY: pre-pr
-pre-pr: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check error-other-ratchet-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check s3s-footprint-check fips-wording-check embedded-secrets-check test-wiring-check doc-paths-check planning-docs-check log-analyzer-rules-check offline-enrollment-e2e-check clippy-check test ## Run full pre-PR checks with clippy and tests
-	@echo "✅ All pre-PR checks passed!"
-
-.PHONY: dev-check
-dev-check: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check error-other-ratchet-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check s3s-footprint-check fips-wording-check embedded-secrets-check test-wiring-check doc-paths-check planning-docs-check quick-check ## Run fast local development checks
-	@echo "✅ Fast development checks passed!"

.config/make/tests.mak

@@ -1,113 +0,0 @@
-## —— Tests and e2e test ---------------------------------------------------------------------------
-
-TEST_THREADS ?= 1
-
-# cargo-nextest is a HARD dependency of `make test`.
-#
-# nextest changes test semantics vs plain `cargo test`: it runs every test in
-# its own process (so serial_test's #[serial] mutex does not serialize across
-# tests) and it is the only runner that honours .config/nextest.toml
-# [test-groups] (e.g. the ecstore-serial-flaky serialization guard). CI runs
-# nextest (.github/actions/setup/action.yml installs it), so a silent fallback
-# to `cargo test` would run with different serialization behaviour than CI and
-# mask (or invent) flakes.
-#
-# Installing cargo-nextest:
-#   cargo install cargo-nextest --locked          # from source
-#   # or a prebuilt binary (faster) via the taiki-e installer / get.nexte.st:
-#   #   https://nexte.st/docs/installation/
-#
-# Escape hatch: set RUSTFS_ALLOW_CARGO_TEST_FALLBACK=1 to run the plain
-# `cargo test` fallback anyway. Results are NOT authoritative — semantics
-# differ from CI and .config/nextest.toml test-groups will NOT apply.
-
-.PHONY: script-tests
-script-tests: ## Run shell script tests
-	@echo "Running script tests..."
-	trap 'status=$$?; printf "ERROR: script-tests failed (exit %s): %s\n" "$$status" "$$BASH_COMMAND" >&2; exit "$$status"' ERR
-	./scripts/test_build_rustfs_options.sh
-	./scripts/test_docker_runtime_timezone.sh
-	./scripts/test_entrypoint_credentials.sh
-	./scripts/test_internode_grpc_ab_bench.sh
-	./scripts/test_object_batch_bench_enhanced.sh
-	./scripts/test_package_service_scripts.sh
-	./scripts/test_hotpath_warp_ab_gate.sh
-	./scripts/test_hotpath_warp_abba.sh
-	./scripts/test_scanner_validation_harness.sh
-	./scripts/test_scanner_heal_checkpoint_crash_evidence.sh
-	./scripts/test_scanner_heal_authority_evidence.sh
-	./scripts/test_scanner_heal_scoped_ack_evidence.sh
-	./scripts/test_scanner_heal_legacy_rollback_evidence.sh
-	./scripts/test_scanner_heal_g14_multiset_evidence.sh
-	./scripts/test_scanner_heal_scheduler_pressure_evidence.sh
-	./scripts/test_scanner_heal_status_outcome_evidence.sh
-	./scripts/test_scanner_heal_maintenance_evidence.sh
-	./scripts/test_scanner_heal_w13_mrf_evidence.sh
-	./scripts/test_scanner_heal_w16_recovery_evidence.sh
-	./scripts/test_exact_1mib_handoff_abba.sh
-	./scripts/test_pinned_paired_abba_bench.sh
-	./scripts/test_manual_transition_runbooks.sh
-	./scripts/test_fuzz_runner.sh
-	./scripts/test_python_bin.sh
-	./scripts/check_embedded_secrets.sh --self-test
-	$(RUSTFS_PYTHON_BIN) ./scripts/check_test_wiring.py --self-test
-	$(RUSTFS_PYTHON_BIN) ./scripts/test_e2e_binary.py
-	$(RUSTFS_PYTHON_BIN) ./scripts/test_migration_gate_evidence.py
-	$(RUSTFS_PYTHON_BIN) ./scripts/ci_gate.py --self-test
-	$(RUSTFS_PYTHON_BIN) ./scripts/check_security_coverage.py --self-test
-	$(RUSTFS_PYTHON_BIN) ./scripts/check_scheduled_validation_freshness.py --self-test
-	$(RUSTFS_PYTHON_BIN) ./scripts/test_security_workflow.py
-	$(RUSTFS_PYTHON_BIN) ./scripts/test_nightly_candidate.py
-	$(RUSTFS_PYTHON_BIN) ./scripts/test_functional_chain.py
-	$(RUSTFS_PYTHON_BIN) ./scripts/test_functional_chain_health.py
-	$(RUSTFS_PYTHON_BIN) ./scripts/test_ci_timing_report.py
-	$(RUSTFS_PYTHON_BIN) ./scripts/s3-tests/test_report_compat.py
-	$(RUSTFS_PYTHON_BIN) ./scripts/s3-tests/test_runner_tools.py
-	bash -n ./scripts/validate_object_data_cache_cold_stampede.sh
-	$(RUSTFS_PYTHON_BIN) ./scripts/check_object_data_cache_follower_samples.py --self-test
-	./scripts/validate_object_data_cache_cold_stampede.sh --self-test
-	./scripts/run_scanner_heal_evidence_case.sh --self-test
-
-.PHONY: test
-test: core-deps script-tests ## Run all tests (needs cargo-nextest; RUSTFS_ALLOW_CARGO_TEST_FALLBACK=1 to override)
-	@echo "🧪 Running tests..."
-	@if command -v cargo-nextest >/dev/null 2>&1; then \
-		cargo nextest run --all --exclude e2e_test; \
-	elif [ "$${RUSTFS_ALLOW_CARGO_TEST_FALLBACK:-0}" = "1" ]; then \
-		echo >&2 "⚠️  ============================================================================"; \
-		echo >&2 "⚠️  cargo-nextest NOT found — running the 'cargo test' fallback (opt-in)."; \
-		echo >&2 "⚠️  TEST SEMANTICS DIFFER FROM CI; results are NOT authoritative:"; \
-		echo >&2 "⚠️    * nextest runs each test in its own process; 'cargo test' does not,"; \
-		echo >&2 "⚠️      so serial_test #[serial] serialization behaves differently."; \
-		echo >&2 "⚠️    * .config/nextest.toml [test-groups] will NOT apply (e.g. the"; \
-		echo >&2 "⚠️      ecstore-serial-flaky group), so load-sensitive tests may flake here"; \
-		echo >&2 "⚠️      but pass on CI (or vice versa)."; \
-		echo >&2 "⚠️  Install cargo-nextest and re-run before trusting these results."; \
-		echo >&2 "⚠️  ============================================================================"; \
-		cargo test --workspace --exclude e2e_test -- --nocapture --test-threads="$(TEST_THREADS)"; \
-	else \
-		echo >&2 "❌ cargo-nextest is required for 'make test' but was not found."; \
-		echo >&2 ""; \
-		echo >&2 "   RustFS tests run under cargo-nextest (process-per-test isolation)."; \
-		echo >&2 "   CI runs nextest and .config/nextest.toml [test-groups] only take effect"; \
-		echo >&2 "   under nextest. Plain 'cargo test' has different serialization semantics"; \
-		echo >&2 "   and is NOT a faithful substitute."; \
-		echo >&2 ""; \
-		echo >&2 "   Install it with either:"; \
-		echo >&2 "     cargo install cargo-nextest --locked"; \
-		echo >&2 "   or a prebuilt binary (faster) — see https://nexte.st/docs/installation/"; \
-		echo >&2 ""; \
-		echo >&2 "   To run the plain 'cargo test' fallback anyway (results NOT authoritative;"; \
-		echo >&2 "   serialization semantics differ from CI), re-run with:"; \
-		echo >&2 "     RUSTFS_ALLOW_CARGO_TEST_FALLBACK=1 make test"; \
-		exit 1; \
-	fi
-	cargo test --all --doc
-
-.PHONY: e2e-server
-e2e-server: ## Run e2e-server tests
-	sh $(shell pwd)/scripts/run.sh
-
-.PHONY: probe-e2e
-probe-e2e: ## Probe e2e tests
-	sh $(shell pwd)/scripts/probe.sh

.config/migration-gate-floor.txt

@@ -1,11 +0,0 @@
-# Committed floor for the number of tests selected by the migration-critical
-# CI gate (see scripts/check_migration_gate_count.sh, backlog#1153 infra-12).
-#
-# The floor equals the exact count of rustfs-ecstore --lib tests, with the
-# test-util feature enabled, matching the gate filter (name substrings:
-# data_movement, rebalance, decommission, source_cleanup, delete_marker) at
-# the time this file was last updated.
-# CI fails if the selected count drops below this number, so renames or
-# removals that thin the gate must update this file in the same PR.
-# Adding tests does not require a bump, but bumping keeps the guard tight.
-946

.config/nextest.toml

@@ -1,861 +0,0 @@
-# nextest configuration for RustFS.
-#
-experimental = ["setup-scripts"]
-
-# Serialize the ecstore tests that share the process-wide disk registry or
-# exercise a multi-disk commit handoff across nextest process boundaries.
-#
-#   * store::bucket::tests::bucket_delete_* share process/global state (disk
-#     registry, lock client) and race make_bucket into InsufficientWriteQuorum
-#     when run concurrently with other ecstore tests.
-#   * bucket_lifecycle_ops::tests::concurrent_resend_same_part_commits_one_generation
-#     uses the shared multipart fixture and a deterministic uploadId-lock
-#     handoff, so it must not overlap another process mutating that fixture.
-#   * bucket::metadata_sys::tests::concurrent_config_writes_from_separate_nodes_do_not_lose_writes
-#     uses the shared transaction lock and must not overlap other ecstore tests.
-#
-# serial_test's #[serial] attribute does NOT serialize these across runs:
-# nextest executes each test in its own process, where the in-process
-# serial_test mutex has no effect. A nextest test-group with max-threads = 1 is
-# the mechanism that actually serializes across nextest's process boundary.
-#
-# ---------------------------------------------------------------------------
-# Profiles
-# ---------------------------------------------------------------------------
-# The `default` profile is what local `cargo nextest run` uses. It NEVER
-# retries: a red test locally means a real failure to investigate, not noise to
-# paper over. The `ci` profile (below) is the strict CI gate: global
-# retries = 0 so a new race's first occurrence is never masked, plus a
-# narrowly-scoped quarantine list (retries = 2) for tests with a tracked OPEN
-# flake issue. Flake policy lives in docs/testing/README.md.
-
-[test-groups]
-ecstore-serial-flaky = { max-threads = 1 }
-embedded-test-ports = { max-threads = 1 }
-e2e-vault = { max-threads = 1 }
-
-# Reliability / fault-injection e2e tests each spawn a single-node 4-disk RustFS
-# server and manipulate its disk directories at runtime (crates/e2e_test:
-# reliability_disk_fault_test, degraded_read_eof_regression_test / dist-13, and
-# replacement_privileged_e2e_test when explicitly run as root on Linux). They
-# are correct in isolation but resource-heavy; serialize them under nextest's
-# process boundary (serial_test's #[serial] does not cross it) so several 4-disk
-# servers never run at once. The e2e-full merge/main lane picks these up;
-# they are deliberately NOT in the fast PR `e2e-smoke` filter.
-e2e-reliability = { max-threads = 1 }
-e2e-inline-boundaries = { max-threads = 1 }
-e2e-cluster-nightly = { max-threads = 1 }
-
-# Deep async storage futures are composed into tests across several crates.
-# Keep the test stack bounded but above libtest's 2 MiB default.
-[scripts.setup.ecstore-base-stack]
-command = ['sh', '-c', 'echo RUST_MIN_STACK=4194304 >> "$NEXTEST_ENV"']
-
-# These exact regression scenarios build deep async storage futures that exceed
-# libtest's 2 MiB spawned-thread stack on Linux. Give only their test processes
-# the same 32 MiB stack already used by the crate's dedicated large-stack tests.
-[scripts.setup.ecstore-large-stack]
-command = ['sh', '-c', 'echo RUST_MIN_STACK=33554432 >> "$NEXTEST_ENV"']
-
-# The serial ILM selection builds the same deep storage futures in both the
-# lifecycle transition module and scanner integration binary. Different tests
-# in each have overflowed first across otherwise unrelated CI runs.
-[scripts.setup.lifecycle-large-stack]
-command = ['sh', '-c', 'echo RUST_MIN_STACK=33554432 >> "$NEXTEST_ENV"']
-
-# --- default profile (local): serialize the flaky groups, never retry --------
-# These real-disk progress oracles exercise fsync, quorum IO, and crash/replay
-# boundaries. Reserve the run's capacity so unrelated fixtures cannot starve
-# their existing deadlines and typed-failure assertions (rustfs/backlog#2706).
-# Keep each test's internal concurrency and zero-retry policy unchanged.
-[[profile.default.overrides]]
-filter = 'package(rustfs) & test(/^connect::diagnostics::schedule::tests::/)'
-threads-required = "num-test-threads"
-
-[[profile.default.overrides]]
-filter = 'package(rustfs-ecstore) & (test(/^set_disk::tests::streaming_get_blocks_concurrent_/) | test(=store::object::tests::retired_marker_converges_after_bucket_recreation_on_sixteen_disks))'
-threads-required = "num-test-threads"
-
-[[profile.default.overrides]]
-filter = 'package(rustfs-heal) & (binary(mrf_partial_write_test) | (binary(mrf_pipeline_test) & test(/^journal_replay_survives_/)) | test(=admin_selector_survives_sigkill_with_original_scope_and_receipts))'
-threads-required = "num-test-threads"
-
-[[profile.default.overrides]]
-filter = 'package(rustfs-scanner) & (test(/^scanner::backlog::tests::native_(writer|retirement)_/) | test(=scanner_folder::tests::mrf_ownership::mrf_ownership_manager_completion_preserves_scanner_pending) | test(/^scanner_io::tests::service_cohort::/))'
-threads-required = "num-test-threads"
-
-[[profile.default.scripts]]
-filter = 'package(rustfs-ecstore) & test(/^(bucket::lifecycle::bucket_lifecycle_ops::tests::manual_transition_worker_result_recovery_marks_unknown_for_corrupt_marker|services::rebalance::entry::tests::real_rebalance_run_fence_loss_blocks_multipart_publication|store::init::tests::(batch_transitioned_delete_uses_free_version_per_item|decommission_entry_(allows_free_version_consumed_before_source_lock|rejects_subquorum_free_version_conflict_and_retains_source|skips_cleanup_only_marker_when_free_version_is_present)|dispatched_tier_delete_recovery_(checks_later_pool_then_commits_after_source_removal|finds_directory_source_on_encoded_set|retains_journal_on_source_metadata_error)|force_tier_remove_blocks_on_physical_free_version_hidden_by_other_pool|legacy_unknown_transition_delete_falls_back_for_single_batch_and_blocks_prefix|multi_pool_(recursive_prefix_rejects_legacy_or_hidden_merge_loser_before_delete|same_remote_tuple_(batch|single)_delete_waits_for_all_sources|same_tuple_recursive_prefix_uses_one_journal_owner|transitioned_delete_persists_one_free_version_per_remote_tuple)|recursive_prefix_partial_(pool|set)_failure_keeps_prepared_cleanup_owners|restored_transitioned_delete_uses_free_version_as_cleanup_owner|stable_transitioned_recursive_prefix_delete_uses_journal_owners|suspended_null_transition_delete_uses_free_version_as_sole_owner|tier_mutation_peer_handler_applies_prepare_commit_and_abort_idempotently|transition_response_loss_persists_unknown_outcome_for_provider_recovery|transition_transaction_recovery_(drops_record_after_confirmed_local_commit|keeps_cleanup_pending_local_commit)|transitioned_delete_(free_version_replays_after_store_restart|local_quorum_failure_rolls_back_without_cleanup_owner|uses_free_version_as_cleanup_owner)|versioned_delete_marker_keeps_transitioned_source_and_remote_object|versioned_explicit_transition_delete_preserves_other_version_then_allows_bucket_delete))$/)'
-setup = 'ecstore-large-stack'
-
-[[profile.default.scripts]]
-filter = 'package(rustfs-ecstore) | package(rustfs-s3select-api) | package(rustfs-scanner) | (package(rustfs) & test(/^(app::multipart_usecase::tests::concurrent_completions_share_durable_bucket_quota_reservations|app::object::delete::tests::compressed_delete_requests_update_observed_usage_without_releasing_quota_floor|app::object::internal_put::tests::internal_multipart_roundtrip_completes_and_abort_leaves_nothing|app::object::restore::tests::execute_restore_object_maps_failures_to_typed_s3_errors|storage::access::tests::(delete_object_access_captures_authorized_bucket_incarnation|copy_operations_reject_recreated_source_bucket_after_authorization|request_slot_keeps_bucket_policy_bound_to_its_store))$/))'
-setup = 'ecstore-base-stack'
-
-# These handler/store fixtures and the relocated-pool GET compose deep storage
-# futures in libtest threads; on Linux they overflow the default 2 MiB stack.
-[[profile.default.scripts]]
-filter = 'package(rustfs) & (test(/^admin::handlers::table_catalog::tests::/) | test(/^table_catalog::store::tests::/) | test(=app::object::get::tests::execute_get_object_resumes_from_relocated_pool_without_splicing_body))'
-setup = 'ecstore-base-stack'
-
-[[profile.default.scripts]]
-filter = 'binary(lifecycle_integration_test) | (package(rustfs) & test(/^app::lifecycle_transition_api_test::/))'
-setup = 'lifecycle-large-stack'
-
-# The pool metadata placement regression composes the full erasure-heal
-# startup future and overflows the default Linux test-thread stack.
-[[profile.default.scripts]]
-filter = 'package(rustfs-heal) & test(=replacement_pool_metadata_follows_real_two_set_placement)'
-setup = 'ecstore-large-stack'
-
-# These real-disk scanner fixtures assert quiet-workload publication and
-# durable MRF admission. Reserve the pool while their bounded CAS/fsync
-# handoffs run so unrelated disk fanouts cannot consume that fixture budget.
-[[profile.default.overrides]]
-filter = 'package(rustfs-scanner) & (test(=scanner_folder::tests::mrf_ownership::mrf_ownership_manager_completion_preserves_scanner_pending) | test(=scanner_io::tests::multi_pool_scanner_cycle_publishes_combined_usage) | test(=scanner_io::tests::pending_put_commit_keeps_scanner_walk_live_without_authoritative_usage) | test(=scanner_folder::tests::mrf_ownership::checkpoint_fixture_metadata_heal_reaches_budgeted_tail))'
-threads-required = "num-test-threads"
-
-[[profile.default.overrides]]
-filter = 'package(rustfs-ecstore) & (test(concurrent_resend_same_part_commits_one_generation) | test(concurrent_config_writes_from_separate_nodes_do_not_lose_writes) | test(/^store::bucket::tests::bucket_delete_(mark_delete|purge_removes|default_s3_delete)/))'
-test-group = 'ecstore-serial-flaky'
-
-# Serialize the multipart crash-consistency scenarios (dist-2, backlog#1150):
-# each spawns a 4-disk hermetic erasure set and drives full staged-upload +
-# commit + GET cycles — the same cross-disk-commit IO shape that made
-# concurrent_resend load-sensitive. Preventive serialization only, no retries.
-# The matching ci-profile override is after [profile.ci].
-[[profile.default.overrides]]
-filter = 'package(rustfs-ecstore) & test(/^set_disk::ops::multipart::tests::crash_consistency::/)'
-test-group = 'ecstore-serial-flaky'
-
-# Serialize the heal result-report tests. Every test in the module builds a
-# real-disk (TempDir-backed) hermetic erasure set and drives MiB-scale writes
-# plus deep-scan heal — the same load-sensitive cross-disk IO shape as the
-# crash_consistency scenarios above. Under a heavily parallel run a single
-# disk's IO can fail while write quorum still holds, which flips per-disk
-# readback and aggregate-outcome assertions nondeterministically (different
-# tests each round; all pass standalone). Preventive serialization only, no
-# retries. The matching ci-profile override is after [profile.ci].
-[[profile.default.overrides]]
-filter = 'package(rustfs-ecstore) & test(/^set_disk::ops::heal::heal_result_report_tests::/)'
-test-group = 'ecstore-serial-flaky'
-
-# Serialize the replacement bucket metadata heal test. It builds a real-disk
-# hermetic erasure set, writes bucket metadata, then asserts byte-identical
-# xl.meta readback after a rejected delayed repair. Under parallel load,
-# concurrent ecstore tests mutate the process-wide disk registry and alter
-# the serialized timestamp/disk-identity fields, flipping the assertion.
-# Preventive serialization only, no retries.
-[[profile.default.overrides]]
-filter = 'package(rustfs-ecstore) & test(replacement_bucket_metadata_respects_pool_ownership_and_required_records)'
-test-group = 'ecstore-serial-flaky'
-
-# Serialize the metadata-cache generation-retirement pair. Both carry
-# #[serial(metadata_cache_invalidation_probe)] — a no-op across nextest's
-# process boundary — and assert get_object_metadata_cache generation
-# semantics on a 4-disk hermetic set, the same load-sensitive shape that
-# forced the transition matrix tests into this group. Preventive
-# serialization only, no retries. The matching ci-profile override is after
-# [profile.ci].
-[[profile.default.overrides]]
-filter = 'package(rustfs-ecstore) & test(retires_cached_snapshot)'
-test-group = 'ecstore-serial-flaky'
-
-# The production-handler relocation regression builds an isolated 8-disk,
-# 2-pool store and commits a 72 MiB multipart object. Keep that cross-disk IO
-# from overlapping the ecstore commit fixtures above.
-[[profile.default.overrides]]
-filter = 'package(rustfs) & test(execute_get_object_resumes_from_relocated_pool_without_splicing_body)'
-test-group = 'ecstore-serial-flaky'
-
-# Embedded integration-test binaries discover an ephemeral port and release
-# the probe listener before RustFS binds it. Serialize that cross-process
-# TOCTOU window; retries would only hide real startup failures.
-[[profile.default.overrides]]
-filter = 'package(rustfs) & (binary(/^embedded.*_test$/) | binary(admin_diagnostic_capability_e2e))'
-test-group = 'embedded-test-ports'
-
-# Registration, inventory delivery, and real drive/object probes include
-# filesystem or HTTP IO in short deadlines. Reserve capacity so sibling test
-# processes cannot exhaust those budgets. Keep deadlines and assertions intact.
-# The short HTTP fixtures are tracked in rustfs/backlog#2714.
-[[profile.default.overrides]]
-filter = 'package(rustfs) & (binary(/^connect_registration$/) | binary(connect_inventory) | binary(connect_perf_drive) | (binary(connect_perf_client) & test(=configured_proxy_and_direct_endpoint_failures_are_distinct)) | (binary(connect_perf_object) & test(/^(real_rustfs_endpoint_and_production_cli_support_bounded_get_and_put|cleanup_uses_a_budget_independent_of_the_measurement_window|oversized_response_is_rejected_without_buffering_it)$/)))'
-threads-required = "num-test-threads"
-
-# This cancellation probe hashes the serving executable before acquiring its
-# collector lease. Keep competing test processes out of its 30s startup budget.
-[[profile.default.overrides]]
-filter = 'package(rustfs) & test(=connect::diagnostics::trace_runtime::tests::local_runtime_profile_cancellation_waits_for_lease_release)'
-threads-required = "num-test-threads"
-
-# These real IAM/state fixtures keep production lock-acquisition limits while
-# peers complete durable IO. Reserve capacity from unrelated test processes;
-# the shared fixture's durable_iam_state_ prefix keeps the whole family together.
-[[profile.default.overrides]]
-filter = 'package(rustfs) & test(/^admin::handlers::site_replication::tests::durable_iam_state_/)'
-threads-required = "num-test-threads"
-
-# Real target-repair and rebalance-stop fixtures commit durable metadata while
-# peers wait on production locks or bounded cancellation checks. Reserve IO
-# capacity for each fixture; its internal races and deadlines remain intact.
-[[profile.default.overrides]]
-filter = 'package(rustfs) & test(/^admin::handlers::(replication::target_repair_tests::|rebalance::rebalance_handler_tests::real_admin_stop_)/)'
-threads-required = "num-test-threads"
-
-# Reserve capacity for these short heartbeat TLS/fsync probes (rustfs#8129).
-# Their existing deadlines, assertions, and retry policy remain in force.
-[[profile.default.overrides]]
-filter = 'package(rustfs) & binary(connect_heartbeat) & test(/^(restart_replays_pending_request_then_advances_sequence|restart_replays_exact_legacy_capabilities_with_and_without_jobs)$/)'
-threads-required = "num-test-threads"
-
-# Serialize the durable manual-transition checkpoint test across nextest's
-# process boundary; it mutates bucket lifecycle metadata and is not quarantined.
-[[profile.default.overrides]]
-filter = 'package(rustfs-ecstore) & test(manual_transition_page_checkpoint_persists_durable_job_progress)'
-test-group = 'ecstore-serial-flaky'
-
-# Serialize the transition matrix tests. They build a 4-disk hermetic erasure
-# set, populate the get_object_metadata_cache, and assert generation lifecycle
-# semantics. serial_test's #[serial] has no effect across nextest's process
-# boundary, so concurrent execution races the shared metadata-cache generation
-# counter and causes spurious "metadata read should publish the generation"
-# panics. Preventive serialization, no retries.
-[[profile.default.overrides]]
-filter = 'package(rustfs-ecstore) & test(set_disk::transition_matrix_tests::)'
-test-group = 'ecstore-serial-flaky'
-
-# The durable ILM decommission regressions build isolated multi-pool stores and
-# deliberately take source or target disks offline while checking fencing.
-[[profile.default.overrides]]
-filter = 'package(rustfs-ecstore) & (test(decommission_migrates_and_verifies_registered_durable_ilm_records) | test(decommission_durable_ilm_target_read_error_is_not_masked_by_peer_success) | test(decommission_durable_ilm_terminal_receipt_recovers_failed_source_cleanup) | test(decommission_durable_ilm_receipt_pagination_fails_closed_on_second_page) | test(decommission_durable_ilm_recovery_keeps_multiple_active_sources))'
-test-group = 'ecstore-serial-flaky'
-
-# Decommission entry and marker/barrier tests share process-wide fault hooks and
-# deterministic commit barriers. Keep the whole init decommission family in one
-# nextest group; serial_test alone cannot isolate separate test processes.
-[[profile.default.overrides]]
-filter = 'package(rustfs-ecstore) & test(/^store::init::tests::(decommission_.*|suspended_.*decommission.*)$/)'
-test-group = 'ecstore-serial-flaky'
-
-# The tier overwrite cleanup-owner matrix restarts a real 4-disk store and
-# drives free-version recovery through lifecycle expiry workers. Reserve the
-# full test budget so competing real-store fixtures cannot starve the walk's
-# filesystem progress deadlines. Keep the production deadlines and no retries.
-[[profile.default.overrides]]
-filter = 'package(rustfs-ecstore) & test(tier_overwrite_put_and_self_copy_recover_persisted_cleanup_owners)'
-test-group = 'ecstore-serial-flaky'
-threads-required = "num-test-threads"
-
-# Serialize the bucket-incarnation / lifecycle-fence tests. They drive
-# init_bucket_metadata_sys and bucket_metadata_sys_of, i.e. process-global
-# OnceLock state that serial_test's #[serial] cannot protect across nextest's
-# process boundary, and they delete+recreate buckets — the same shape that
-# raced into InsufficientWriteQuorum in backlog#937. Preventive only, no
-# retries. The matching ci-profile override is after [profile.ci].
-[[profile.default.overrides]]
-filter = 'package(rustfs-ecstore) & (test(bucket_delete_waits_for_config_mutation_fence) | test(stale_config_request_cannot_mutate_a_recreated_bucket) | test(disk_incarnation_read_detects_stale_cache_until_peer_reload) | test(lifecycle_expiry_fails_closed_on_corrupt_object_lock_metadata) | test(expiry_configs_are_resolved_from_the_owning_store))'
-test-group = 'ecstore-serial-flaky'
-
-# Serialize the default-off dst-dir fsync group-commit tests. They use
-# process-global test hooks/registry to deterministically freeze fsync batches;
-# no retries, just one at a time under nextest too.
-[[profile.default.overrides]]
-filter = 'package(rustfs-ecstore) & test(dst_dir_fsync_group_commit)'
-test-group = 'ecstore-serial-flaky'
-
-# Serialize the 4-disk reliability / degraded-read e2e tests (see the
-# e2e-reliability test-group note above). The matching ci-profile override is at
-# the end of the file, after [profile.ci] is declared.
-[[profile.default.overrides]]
-filter = 'package(e2e_test) & test(/^(reliability_disk_fault|degraded_read_eof_regression|replacement_privileged_e2e)_test::/)'
-test-group = 'e2e-reliability'
-
-[[profile.default.overrides]]
-filter = 'package(e2e_test) & test(/^inline_fast_path_cluster_test::/)'
-test-group = 'e2e-inline-boundaries'
-
-# 4-node 4-drive distributed Actions suite: each case starts four rustfs
-# processes and up to sixteen data directories. Serialize across nextest's
-# process boundary so several 4x4 clusters never overlap.
-[[profile.default.overrides]]
-filter = 'package(e2e_test) & test(/^distributed::/)'
-test-group = 'e2e-cluster-nightly'
-
-# Vault KMS tests share the fixed dev-server port 8200. serial_test's #[serial]
-# does not cross nextest process boundaries, so keep every Vault-backed test in
-# one group.
-[[profile.default.overrides]]
-filter = 'package(e2e_test) & (test(/^kms::kms_vault_test::/) | test(/^kms::kms_rekey_sweep_test::/) | test(/^kms::configured_roundtrip_test::test_configured_vault_kms_admin_and_versioned_cleanup$/))'
-test-group = 'e2e-vault'
-
-# These server-heavy smoke cases perform 1k+ sequential S3 writes or run the
-# scanner and a remote tier server together. Reserve the full test budget so
-# unrelated E2E processes cannot starve their storage and polling deadlines.
-[[profile.default.overrides]]
-filter = 'package(e2e_test) & (test(/^list_objects_v2_pagination_test::tests::(test_list_objects_v2_delimiter_small_page_traverses_all|test_list_objects_v2_max_keys_above_limit_returns_token|test_list_objects_v2_maxkeys_above_limit_with_delimiter)$/) | test(=reliant::tiering::test_hermetic_transition_restore_failure_expiry_and_retry))'
-threads-required = "num-test-threads"
-
-# This four-disk, 65-member rollback probe already drives up to 32 concurrent
-# durable deletions. Reserve this nextest run's capacity for its progress oracle.
-[[profile.default.overrides]]
-filter = 'package(rustfs-ecstore) & test(=store::init::tests::dispatch_manifest_rollback_bounded_concurrency_reaches_tail_behind_slow_member)'
-threads-required = "num-test-threads"
-
-# ---------------------------------------------------------------------------
-# ci profile — the strict CI gate (ci.yml `cargo nextest run --profile ci`)
-# ---------------------------------------------------------------------------
-[profile.ci]
-# Strict: a new race must fail on its first occurrence, never be retried away.
-retries = 0
-# Report every failure in one run instead of bailing on the first.
-fail-fast = false
-
-[profile.ci.junit]
-# Emitted to target/nextest/ci/junit.xml; uploaded as a CI artifact.
-# Tests that pass only after a quarantine retry are marked `flaky` here — that
-# marker is the observable signal the flake policy is built around.
-path = "junit.xml"
-
-# Match the local durable-progress reservations. All members, deadlines,
-# assertions, and internal concurrency remain enabled without retries.
-[[profile.ci.overrides]]
-filter = 'package(rustfs) & test(/^connect::diagnostics::schedule::tests::/)'
-threads-required = "num-test-threads"
-
-[[profile.ci.overrides]]
-filter = 'package(rustfs-ecstore) & (test(/^set_disk::tests::streaming_get_blocks_concurrent_/) | test(=store::object::tests::retired_marker_converges_after_bucket_recreation_on_sixteen_disks))'
-threads-required = "num-test-threads"
-
-[[profile.ci.overrides]]
-filter = 'package(rustfs-heal) & (binary(mrf_partial_write_test) | (binary(mrf_pipeline_test) & test(/^journal_replay_survives_/)) | test(=admin_selector_survives_sigkill_with_original_scope_and_receipts))'
-threads-required = "num-test-threads"
-
-[[profile.ci.overrides]]
-filter = 'package(rustfs-scanner) & (test(/^scanner::backlog::tests::native_(writer|retirement)_/) | test(=scanner_folder::tests::mrf_ownership::mrf_ownership_manager_completion_preserves_scanner_pending) | test(/^scanner_io::tests::service_cohort::/))'
-threads-required = "num-test-threads"
-
-[[profile.ci.scripts]]
-filter = 'package(rustfs-ecstore) & test(/^(bucket::lifecycle::bucket_lifecycle_ops::tests::manual_transition_worker_result_recovery_marks_unknown_for_corrupt_marker|services::rebalance::entry::tests::real_rebalance_run_fence_loss_blocks_multipart_publication|store::init::tests::(batch_transitioned_delete_uses_free_version_per_item|decommission_entry_(allows_free_version_consumed_before_source_lock|rejects_subquorum_free_version_conflict_and_retains_source|skips_cleanup_only_marker_when_free_version_is_present)|dispatched_tier_delete_recovery_(checks_later_pool_then_commits_after_source_removal|finds_directory_source_on_encoded_set|retains_journal_on_source_metadata_error)|force_tier_remove_blocks_on_physical_free_version_hidden_by_other_pool|legacy_unknown_transition_delete_falls_back_for_single_batch_and_blocks_prefix|multi_pool_(recursive_prefix_rejects_legacy_or_hidden_merge_loser_before_delete|same_remote_tuple_(batch|single)_delete_waits_for_all_sources|same_tuple_recursive_prefix_uses_one_journal_owner|transitioned_delete_persists_one_free_version_per_remote_tuple)|recursive_prefix_partial_(pool|set)_failure_keeps_prepared_cleanup_owners|restored_transitioned_delete_uses_free_version_as_cleanup_owner|stable_transitioned_recursive_prefix_delete_uses_journal_owners|suspended_null_transition_delete_uses_free_version_as_sole_owner|tier_mutation_peer_handler_applies_prepare_commit_and_abort_idempotently|transition_response_loss_persists_unknown_outcome_for_provider_recovery|transition_transaction_recovery_(drops_record_after_confirmed_local_commit|keeps_cleanup_pending_local_commit)|transitioned_delete_(free_version_replays_after_store_restart|local_quorum_failure_rolls_back_without_cleanup_owner|uses_free_version_as_cleanup_owner)|versioned_delete_marker_keeps_transitioned_source_and_remote_object|versioned_explicit_transition_delete_preserves_other_version_then_allows_bucket_delete))$/)'
-setup = 'ecstore-large-stack'
-
-[[profile.ci.scripts]]
-filter = 'package(rustfs-ecstore) | package(rustfs-s3select-api) | package(rustfs-scanner) | (package(rustfs) & test(/^(app::multipart_usecase::tests::concurrent_completions_share_durable_bucket_quota_reservations|app::object::delete::tests::compressed_delete_requests_update_observed_usage_without_releasing_quota_floor|app::object::internal_put::tests::internal_multipart_roundtrip_completes_and_abort_leaves_nothing|app::object::restore::tests::execute_restore_object_maps_failures_to_typed_s3_errors|storage::access::tests::(delete_object_access_captures_authorized_bucket_incarnation|copy_operations_reject_recreated_source_bucket_after_authorization|request_slot_keeps_bucket_policy_bound_to_its_store))$/))'
-setup = 'ecstore-base-stack'
-
-[[profile.ci.scripts]]
-filter = 'package(rustfs) & (test(/^admin::handlers::table_catalog::tests::/) | test(/^table_catalog::store::tests::/) | test(=app::object::get::tests::execute_get_object_resumes_from_relocated_pool_without_splicing_body))'
-setup = 'ecstore-base-stack'
-
-[[profile.ci.scripts]]
-filter = 'binary(lifecycle_integration_test) | (package(rustfs) & test(/^app::lifecycle_transition_api_test::/))'
-setup = 'lifecycle-large-stack'
-
-# The pool metadata placement regression composes the full erasure-heal
-# startup future and overflows the default Linux test-thread stack.
-[[profile.ci.scripts]]
-filter = 'package(rustfs-heal) & test(=replacement_pool_metadata_follows_real_two_set_placement)'
-setup = 'ecstore-large-stack'
-
-# These real-disk scanner fixtures assert quiet-workload publication and
-# durable MRF admission. Reserve the pool while their bounded CAS/fsync
-# handoffs run so unrelated disk fanouts cannot consume that fixture budget.
-[[profile.ci.overrides]]
-filter = 'package(rustfs-scanner) & (test(=scanner_folder::tests::mrf_ownership::mrf_ownership_manager_completion_preserves_scanner_pending) | test(=scanner_io::tests::multi_pool_scanner_cycle_publishes_combined_usage) | test(=scanner_io::tests::pending_put_commit_keeps_scanner_walk_live_without_authoritative_usage) | test(=scanner_folder::tests::mrf_ownership::checkpoint_fixture_metadata_heal_reaches_budgeted_tail))'
-threads-required = "num-test-threads"
-
-# ===========================================================================
-# QUARANTINE — flaky tests granted retries = 2 under the ci profile ONLY.
-#
-# RULES (enforced by review, see docs/testing/README.md):
-#   * Every entry MUST link exactly one OPEN issue tracking the flake.
-#   * An entry stays until the issue is fixed (test made robust) or the test is
-#     deleted — 30-day policy. No entry may exist without a live issue link.
-#
-# Each entry also re-declares the `ecstore-serial-flaky` test-group so the
-# serialization holds under the ci profile (nextest evaluates a named
-# profile's own overrides list, not the default profile's).
-# ===========================================================================
-
-# QUARANTINE: OPEN backlog#937 — store::bucket::tests::bucket_delete_* race
-# make_bucket into InsufficientWriteQuorum via shared global state under load.
-[[profile.ci.overrides]]
-filter = 'package(rustfs-ecstore) & test(/^store::bucket::tests::bucket_delete_(mark_delete|purge_removes|default_s3_delete)/)'
-test-group = 'ecstore-serial-flaky'
-retries = 2
-
-# Keep deterministic ECStore write handoffs isolated across nextest processes.
-[[profile.ci.overrides]]
-filter = 'package(rustfs-ecstore) & (test(concurrent_resend_same_part_commits_one_generation) | test(concurrent_config_writes_from_separate_nodes_do_not_lose_writes))'
-test-group = 'ecstore-serial-flaky'
-
-# Serialize the relocated-pool GET resume regression under the ci profile too
-# (see the matching default-profile override near the top). No longer a
-# quarantine: the fixture race (rustfs#6701/rustfs#6703) was fixed by #6707,
-# which made the staging tolerate quorum-tolerated disk gaps; only the 8-disk
-# cross-disk-IO serialization remains.
-[[profile.ci.overrides]]
-filter = 'package(rustfs) & test(execute_get_object_resumes_from_relocated_pool_without_splicing_body)'
-test-group = 'ecstore-serial-flaky'
-
-# Serialize the 4-disk reliability / degraded-read e2e tests under the ci
-# profile too (see the e2e-reliability test-group note near the top). Not a
-# quarantine: no retries, just single-threaded so several 4-disk servers never
-# run concurrently when e2e-full runs the suite.
-[[profile.ci.overrides]]
-filter = 'package(e2e_test) & test(/^(reliability_disk_fault|degraded_read_eof_regression|replacement_privileged_e2e)_test::/)'
-test-group = 'e2e-reliability'
-
-# Serialize the multipart crash-consistency scenarios under the ci profile too
-# (see the matching default-profile override near the top). Not a quarantine:
-# no retries, just serialized 4-disk cross-disk-commit IO.
-[[profile.ci.overrides]]
-filter = 'package(rustfs-ecstore) & test(/^set_disk::ops::multipart::tests::crash_consistency::/)'
-test-group = 'ecstore-serial-flaky'
-
-# Serialize the heal result-report tests under the ci profile too (see the
-# matching default-profile override near the top). Not a quarantine: no
-# retries, just serialized real-disk heal IO.
-[[profile.ci.overrides]]
-filter = 'package(rustfs-ecstore) & test(/^set_disk::ops::heal::heal_result_report_tests::/)'
-test-group = 'ecstore-serial-flaky'
-
-# Serialize the replacement bucket metadata heal test under the ci profile too.
-[[profile.ci.overrides]]
-filter = 'package(rustfs-ecstore) & test(replacement_bucket_metadata_respects_pool_ownership_and_required_records)'
-test-group = 'ecstore-serial-flaky'
-
-# Serialize the metadata-cache generation-retirement pair under the ci
-# profile too (see the matching default-profile override near the top). Not a
-# quarantine: no retries.
-[[profile.ci.overrides]]
-filter = 'package(rustfs-ecstore) & test(retires_cached_snapshot)'
-test-group = 'ecstore-serial-flaky'
-
-# Match the default-profile embedded test isolation without quarantining or
-# retrying failures in CI.
-[[profile.ci.overrides]]
-filter = 'package(rustfs) & (binary(/^embedded.*_test$/) | binary(admin_diagnostic_capability_e2e))'
-test-group = 'embedded-test-ports'
-
-[[profile.ci.overrides]]
-filter = 'package(rustfs) & (binary(/^connect_registration$/) | binary(connect_inventory) | binary(connect_perf_drive) | (binary(connect_perf_client) & test(=configured_proxy_and_direct_endpoint_failures_are_distinct)) | (binary(connect_perf_object) & test(/^(real_rustfs_endpoint_and_production_cli_support_bounded_get_and_put|cleanup_uses_a_budget_independent_of_the_measurement_window|oversized_response_is_rejected_without_buffering_it)$/)))'
-threads-required = "num-test-threads"
-
-# Preserve the cancellation probe's startup budget and assertions in CI too.
-[[profile.ci.overrides]]
-filter = 'package(rustfs) & test(=connect::diagnostics::trace_runtime::tests::local_runtime_profile_cancellation_waits_for_lease_release)'
-threads-required = "num-test-threads"
-
-# Keep the same IAM/state fixture capacity reservation in CI without changing the
-# production lock deadline, internal concurrency, assertions, or retry policy.
-[[profile.ci.overrides]]
-filter = 'package(rustfs) & test(/^admin::handlers::site_replication::tests::durable_iam_state_/)'
-threads-required = "num-test-threads"
-
-# Match the real admin fixture capacity reservation without retries or changes
-# to production lock deadlines, cancellation barriers, or assertions.
-[[profile.ci.overrides]]
-filter = 'package(rustfs) & test(/^admin::handlers::(replication::target_repair_tests::|rebalance::rebalance_handler_tests::real_admin_stop_)/)'
-threads-required = "num-test-threads"
-
-# Reserve capacity for these short heartbeat TLS/fsync probes (rustfs#8129).
-# Their existing deadlines, assertions, and retry policy remain in force.
-[[profile.ci.overrides]]
-filter = 'package(rustfs) & binary(connect_heartbeat) & test(/^(restart_replays_pending_request_then_advances_sequence|restart_replays_exact_legacy_capabilities_with_and_without_jobs)$/)'
-threads-required = "num-test-threads"
-
-# Serialize the durable manual-transition checkpoint test under the ci profile
-# too. No retries: failures stay visible.
-[[profile.ci.overrides]]
-filter = 'package(rustfs-ecstore) & test(manual_transition_page_checkpoint_persists_durable_job_progress)'
-test-group = 'ecstore-serial-flaky'
-
-# Serialize the transition matrix tests under the ci profile too (see the
-# matching default-profile override near the top). No retries.
-[[profile.ci.overrides]]
-filter = 'package(rustfs-ecstore) & test(set_disk::transition_matrix_tests::)'
-test-group = 'ecstore-serial-flaky'
-
-[[profile.ci.overrides]]
-filter = 'package(rustfs-ecstore) & (test(decommission_migrates_and_verifies_registered_durable_ilm_records) | test(decommission_durable_ilm_target_read_error_is_not_masked_by_peer_success) | test(decommission_durable_ilm_terminal_receipt_recovers_failed_source_cleanup) | test(decommission_durable_ilm_receipt_pagination_fails_closed_on_second_page) | test(decommission_durable_ilm_recovery_keeps_multiple_active_sources))'
-test-group = 'ecstore-serial-flaky'
-
-[[profile.ci.overrides]]
-filter = 'package(rustfs-ecstore) & test(/^store::init::tests::(decommission_.*|suspended_.*decommission.*)$/)'
-test-group = 'ecstore-serial-flaky'
-
-[[profile.ci.overrides]]
-filter = 'package(rustfs-ecstore) & test(tier_overwrite_put_and_self_copy_recover_persisted_cleanup_owners)'
-test-group = 'ecstore-serial-flaky'
-threads-required = "num-test-threads"
-
-# Serialize the bucket-incarnation / lifecycle-fence tests under the ci profile
-# too (see the matching default-profile override near the top). No retries.
-[[profile.ci.overrides]]
-filter = 'package(rustfs-ecstore) & (test(bucket_delete_waits_for_config_mutation_fence) | test(stale_config_request_cannot_mutate_a_recreated_bucket) | test(disk_incarnation_read_detects_stale_cache_until_peer_reload) | test(lifecycle_expiry_fails_closed_on_corrupt_object_lock_metadata) | test(expiry_configs_are_resolved_from_the_owning_store))'
-test-group = 'ecstore-serial-flaky'
-
-[[profile.ci.overrides]]
-filter = 'package(rustfs-ecstore) & test(dst_dir_fsync_group_commit)'
-test-group = 'ecstore-serial-flaky'
-
-# ---------------------------------------------------------------------------
-# e2e-smoke profile — PR smoke subset of the e2e_test crate (backlog#1149 ci-4)
-# ---------------------------------------------------------------------------
-# PR smoke subset of the e2e_test crate (backlog#1149 ci-4). This profile is
-# the single wiring mechanism for e2e tests in CI: other suites join by
-# extending this filter (or a sibling profile), never by adding ad-hoc e2e
-# jobs to ci.yml. Admission criteria (see crates/e2e_test/README.md): fast,
-# single-node topology, no external dependencies (no awscurl / Vault / fixed
-# ports / pre-started server), no #[ignore].
-#
-# Each e2e test spawns its own rustfs server on a random port with an isolated
-# temp dir (crates/e2e_test/src/common.rs), so the subset is parallel-safe.
-#
-# Replication failure harness (backlog#1147 repl-8): the first clause admits
-# its four in-process fake-target self-tests. They bind random loopback ports,
-# use no external service, and finish in under a second.
-#
-# Replication PR subset (backlog#1147 repl-1): the second clause admits the 20
-# FAST bucket-replication tests from replication_extension_test — the
-# target-registration / replication-check / list / remove / delete admin paths
-# that validate config synchronously and never wait for asynchronous
-# replication convergence. Each spawns its own single-node rustfs server(s) on
-# random ports (source, plus an independent single-node target for the pair
-# checks — NOT a cluster), so the subset stays parallel-safe and single-digit
-# seconds. The data-plane tests that poll for convergence and all
-# `_real_dual_node` / `_real_single_node`
-# site-replication tests run in the [profile.e2e-repl-nightly] lane below, NOT
-# here. This allowlist is the single source of truth for the PR/nightly split:
-# the nightly profile derives its set as "the replication module MINUS this
-# allowlist", so any new replication test lands in nightly by default (never
-# silently unrun) until it is explicitly blessed as fast here. Keep the two
-# regexes byte-identical. The committed profile selection digests make changes
-# visible in CI; list current membership with `cargo nextest list -p e2e_test
-# --profile <profile>` (platform-dependent; see docs/testing/README.md).
-# HISTORY (2026-07-11): the 20 fast tests were briefly pulled out of this lane
-# (#4724) because they set a loopback (127.0.0.1) replication target that the
-# SSRF egress guard rejected on every PR after repl-1 (#4712). That is fixed —
-# the guard now honours an off-by-default opt-in and this suite's source servers
-# set it (RUSTFS_REPLICATION_ALLOW_LOOPBACK_TARGET) — so the allowlist below is
-# restored.
-#
-# Security negative-auth subset (backlog#1151 sec-5): the three attacker-facing
-# S3 auth-rejection suites join the first clause above by module name —
-# presigned_negative (sec-2), negative_sigv4 (sec-1, header SigV4), and
-# admin_auth (sec-4, admin gate + root-credential lifecycle). All three use
-# RustFSTestEnvironment on a random port and are parallel-safe, so they meet the
-# smoke admission criteria unchanged. This is the wiring step that makes those
-# merged suites actually execute on every PR (they were dead until listed here).
-# A rename that drops any of them out of this filter would silently thin the
-# security gate with no CI signal, so scripts/check_security_smoke_count.sh owns
-# a count-floor guard over exactly this subset (infra-12 mechanism, floor in
-# .config/security-smoke-floor.txt), invoked from the e2e-tests job in ci.yml.
-# NOT here by topology: the GHSA-3p3x FTPS/WebDAV constant-time e2e
-# (protocols::test_protocol_core_suite) binds fixed ports and needs the
-# ftps,webdav features, so it cannot join this random-port, default-feature
-# profile; its GHSA-r5qv sibling is a unit test that already runs in the
-# test-and-lint `--all --exclude e2e_test` pass. See
-# docs/testing/security-regressions.md for the full CI-execution map.
-#
-# ILM tiering main path (backlog#1148 ilm-7): the `reliant::tiering::` clause
-# admits the hermetic transition e2e. Like the fast replication pair checks it
-# spawns a second independent single-node server (the cold RustFS tier), not a
-# cluster, so it keeps the lane's parallel-safe / no-external-dependency
-# properties. The RustFS warm backend has no loopback guard (that guard is
-# replication-only), so it needs no opt-in env for its 127.0.0.1 tier target.
-#
-# Disk compression (backlog#1848): the `compression` module joins the smoke
-# lane so the multipart disk-compression roundtrips (restored after
-# rustfs/rustfs#5169 disabled them) have PR-lane signal, not just merge-gate.
-# Single-node servers on random ports with isolated temp dirs — meets the
-# admission criteria unchanged.
-#
-# On-demand migration GA (backlog#2163 ODM-16): three named cases join the
-# lane, one per user-visible contract of the feature — a GET miss that pulls
-# the object and persists it locally, a HEAD miss that answers from the source
-# and stores nothing, and the admin config/status pair that must redact the
-# source secret. Each spawns one single-node rustfs server plus the in-process
-# fake S3 source (`fake_s3_target`, already in the first clause), so they meet
-# the admission criteria unchanged; measured at 15.8 s / 15.8 s / 15.9 s, which
-# is entirely the shared server startup and overlaps the lane's other tests.
-# The rest of `on_demand_migration::{get_basic,interaction,backfill,
-# harness_self}_test` stays in e2e-full and the fault / concurrency /
-# real-source modules stay in e2e-nightly; this is an allowlist, not a module
-# clause, so a new ODM test never lands here silently.
-#
-# Scanner authoritative usage publication (backlog#2213): data_usage_test is
-# the PR-lane e2e coverage for scanner usage snapshots consumed by quota and
-# admin surfaces. It uses the same single-node, random-port, isolated-temp-dir
-# fixture as the existing smoke modules.
-[profile.e2e-smoke]
-default-filter = """
-    package(e2e_test) & (
-        test(/^(delete_marker_migration_semantics|version_id_regression|list_objects_v2_pagination|list_object_versions_regression|list_objects_duplicates|list_buckets_double_slash|list_buckets_auth|list_buckets_iam_filter|leading_slash_key|special_chars|create_bucket_region|delete_objects_versioning|head_object_consistency|head_object_range|copy_object_metadata|copy_object_tagging|copy_source_invalid_date|content_encoding|compression|multipart_storage_class|storage_class_capability|ssec_copy|anonymous_access|bucket_policy_check|presigned_negative|negative_sigv4|admin_auth|notification_webhook|tls_hot_reload|console_smoke|admin_iam_crud|admin_pools|sts_query_compat|data_usage)_test::|^fake_s3_target::/)
-        | test(/^replication_extension_test::(test_replication_check_succeeds_with_remote_target|test_replication_check_rejects_target_without_object_lock|test_set_remote_target_rejects_unversioned_source_bucket|test_replication_check_rejects_unversioned_source_bucket|test_replication_check_rejects_missing_replication_config|test_replication_check_rejects_invalid_bucket|test_set_remote_target_rejects_same_bucket_on_same_deployment|test_set_remote_target_rejects_unversioned_target_bucket|test_set_remote_target_update_requires_arn|test_set_remote_target_update_rejects_missing_target|test_set_remote_target_rejects_invalid_target_url|test_set_remote_target_rejects_self_signed_https_target_without_skip_tls_verify|test_set_remote_target_rejects_private_ca_https_target_without_ca_cert_pem|test_list_remote_targets_rejects_empty_bucket|test_list_remote_targets_rejects_invalid_bucket|test_remove_remote_target_rejects_missing_target|test_remove_remote_target_rejects_missing_arn|test_remove_remote_target_rejects_invalid_bucket|test_remove_remote_target_rejects_target_used_by_replication|test_delete_bucket_replication_removes_remote_target)$/)
-        | test(/^reliant::lifecycle::/)
-        | test(/^reliant::tiering::/)
-        | test(/^upgrade_compatibility_test::upgrade_write_readiness_tests::/)
-        | test(/^on_demand_migration::(get_basic_test::(get_miss_pulls_inline_and_serves_locally_afterwards|head_miss_answers_from_the_source_without_persisting)|interaction_test::test_odm_admin_config_is_redacted_and_status_counts_match_the_source)$/)
-    )
-"""
-fail-fast = false
-
-[profile.e2e-smoke.junit]
-path = "junit.xml"
-
-# Bound the whole pagination fixture, including startup, seeding, and cleanup;
-# its LIST deadline begins only after the seed objects have been written.
-[[profile.e2e-smoke.overrides]]
-filter = 'package(e2e_test) & test(/^list_objects_v2_pagination_test::tests::(test_list_objects_v2_delimiter_small_page_traverses_all|test_list_objects_v2_max_keys_above_limit_returns_token|test_list_objects_v2_maxkeys_above_limit_with_delimiter)$/)'
-slow-timeout = { period = "60s", terminate-after = 2, grace-period = "10s" }
-threads-required = "num-test-threads"
-
-[[profile.e2e-smoke.overrides]]
-filter = 'package(e2e_test) & test(=reliant::tiering::test_hermetic_transition_restore_failure_expiry_and_retry)'
-threads-required = "num-test-threads"
-
-# ---------------------------------------------------------------------------
-# e2e-repl-nightly profile — scheduled full replication e2e lane (repl-1)
-# ---------------------------------------------------------------------------
-# backlog#1147 repl-1 (deps: ci-4). Runs the SLOW / cross-process replication
-# tests that are unfit for the per-PR e2e-smoke gate:
-#
-#   * 2 remote-target TLS validation tests.
-#   * 15 bucket-replication data-plane/helper tests — they PUT/delete objects
-#     and poll until source and target converge; two replicate over HTTPS,
-#     six pin SSE replication contracts (managed SSE-S3/SSE-KMS re-encrypt on
-#     the target incl. multipart and the resync path, SSE-C and
-#     target-without-KMS stay fail-closed), and one guards event/history
-#     observers.
-#   * 13 `_real_dual_node` site-replication tests — each spawns TWO full rustfs
-#     servers and drives the cross-process site-replication control plane.
-#   * 1 `_real_three_node` site-replication test.
-#   * 1 `_real_single_node` service-account round-trip test.
-#
-# The set is defined as "everything in replication_extension_test that is NOT
-# in the e2e-smoke PR allowlist above" (the negated clause is byte-identical to
-# the allowlist), so a newly added replication test automatically runs here
-# until it is explicitly promoted to the fast PR subset — no replication test
-# is ever silently left out of CI.
-#
-# replication_target_matrix_test (the outbound target matrix: every object
-# shape against every remote-target failure mode the fake target models) runs
-# here in full; its expectation table pins known-red cells to open issues.
-#
-# #[serial] does NOT serialize under nextest (process-per-test; see the file
-# header). These tests need no cross-test serialization: each spawns its own
-# server(s) on random ports with isolated temp dirs, so they are parallel-safe
-# by construction — the same property the e2e-smoke subset relies on. If load
-# on the runner surfaces a real flake, quarantine the specific test with an
-# OPEN issue link (ci-10 / backlog#937 policy), never blanket-retry or exclude.
-#
-# Wired by .github/workflows/e2e-replication-nightly.yml (schedule +
-# workflow_dispatch), which builds the rustfs binary once, installs awscurl so
-# the STS dual-node test actually exercises its path (the test fails when
-# awscurl is absent), and routes scheduled failures
-# through .github/actions/schedule-failure-issue (ci-8). Explicit division of
-# labor with e2e-full: these tests run only in the consolidated nightly
-# workflow, not in the merge/main lane.
-[profile.e2e-repl-nightly]
-default-filter = """
-    package(e2e_test)
-    & (test(/^replication_extension_test::/) | test(/^replication_target_matrix_test::/))
-    & !test(/^replication_extension_test::(test_replication_check_succeeds_with_remote_target|test_replication_check_rejects_target_without_object_lock|test_set_remote_target_rejects_unversioned_source_bucket|test_replication_check_rejects_unversioned_source_bucket|test_replication_check_rejects_missing_replication_config|test_replication_check_rejects_invalid_bucket|test_set_remote_target_rejects_same_bucket_on_same_deployment|test_set_remote_target_rejects_unversioned_target_bucket|test_set_remote_target_update_requires_arn|test_set_remote_target_update_rejects_missing_target|test_set_remote_target_rejects_invalid_target_url|test_set_remote_target_rejects_self_signed_https_target_without_skip_tls_verify|test_set_remote_target_rejects_private_ca_https_target_without_ca_cert_pem|test_list_remote_targets_rejects_empty_bucket|test_list_remote_targets_rejects_invalid_bucket|test_remove_remote_target_rejects_missing_target|test_remove_remote_target_rejects_missing_arn|test_remove_remote_target_rejects_invalid_bucket|test_remove_remote_target_rejects_target_used_by_replication|test_delete_bucket_replication_removes_remote_target)$/)
-"""
-fail-fast = false
-
-[profile.e2e-repl-nightly.junit]
-# Emitted to target/nextest/e2e-repl-nightly/junit.xml; uploaded by the nightly
-# workflow as the failure-triage artifact.
-path = "junit.xml"
-
-# ---------------------------------------------------------------------------
-# e2e-nightly profile — destructive multi-process cluster fault domains
-# ---------------------------------------------------------------------------
-# These eight modules are deliberately outside e2e-full's merge budget. Each
-# starts a real multi-process or multi-disk topology and exercises node/disk
-# loss, quorum, cleanup, notification fan-in, or admin-timeout behavior. The
-# consolidated nightly workflow runs them serially to avoid resource
-# starvation; failures are never retried.
-#
-# heal_erasure_disk_rebuild_test also runs in e2e-full so core heal rebuild
-# regressions are caught no later than the merge/main lane. It remains here for
-# nightly serial coverage with the other cluster fault domains.
-#
-# On-demand migration (backlog#2158 ODM-11) joins by the second clause: the
-# fault matrix waits out the 30 s circuit-breaker window, the concurrency
-# matrix drives 100-deep bursts, and the real-source cases start a second
-# (loop guard: a third) RustFS process. They are too slow or too heavy for
-# the merge budget; `on_demand_migration::{get_basic,interaction}_test` stay
-# in e2e-full, which excludes exactly these three modules.
-[profile.e2e-nightly]
-default-filter = """
-    package(e2e_test)
-    & (
-        test(/^(admin_timeout_regression_test|cluster_concurrency_test|cluster_multidrive_pool_test|degraded_listing_availability_test|heal_erasure_disk_rebuild_test|namespace_lock_quorum_test|object_lambda_test|stale_multipart_cleanup_cluster_test)::/)
-        | test(/^on_demand_migration::(concurrency_test|fault_test|real_source_test)::/)
-    )
-"""
-fail-fast = false
-
-[profile.e2e-nightly.junit]
-path = "junit.xml"
-
-[[profile.e2e-nightly.overrides]]
-filter = 'package(e2e_test)'
-test-group = 'e2e-cluster-nightly'
-
-# The EC8+4 multi-set heal proof deliberately uploads a larger workload so the
-# background root-heal pass can be interrupted after targeting the replacement
-# drive's erasure slots. Keep the extended budget scoped to this proof case.
-[[profile.e2e-nightly.overrides]]
-filter = 'package(e2e_test) & test(=heal_erasure_disk_rebuild_test::tests::test_cluster_root_heal_recovers_ec84_shards_across_multi_set_after_background_target_restart)'
-slow-timeout = { period = "120s", terminate-after = 12, grace-period = "10s" }
-
-# ---------------------------------------------------------------------------
-# e2e-distributed profile — 4-node 4-disk Actions suite
-# ---------------------------------------------------------------------------
-# Storage-sensitive PR / nightly / dispatch lane owned by
-# .github/workflows/e2e-distributed.yml.
-# Each case starts four rustfs processes (and for site replication, two
-# clusters). Upgrade cases also require RUSTFS_UPGRADE_SOURCE_BINARY.
-# Serialized via e2e-cluster-nightly with no retries.
-[profile.e2e-distributed]
-default-filter = 'package(e2e_test) & test(/^distributed::/)'
-fail-fast = false
-# Decommission / rebalance cases poll for up to 180s with little stdout.
-slow-timeout = { period = "120s", terminate-after = 6 }
-
-[profile.e2e-distributed.junit]
-path = "junit.xml"
-
-[[profile.e2e-distributed.overrides]]
-filter = 'package(e2e_test)'
-test-group = 'e2e-cluster-nightly'
-
-# ---------------------------------------------------------------------------
-# e2e-odm-interop profile — on-demand migration provider interop lane (ODM-20)
-# ---------------------------------------------------------------------------
-# backlog#2167. Report-only, scheduled, never a required check; wired by
-# .github/workflows/on-demand-migration-interop.yml.
-#
-# The four cases in `on_demand_migration::interop_test` take their source from
-# the environment (`RUSTFS_ODM_INTEROP_*`, documented on the constants in
-# `crates/e2e_test/src/on_demand_migration/common.rs`), so the same bodies run
-# against the in-process fake source locally and against a MinIO container or a
-# real cloud provider in the lane. The cloud jobs narrow this profile with their
-# own `-E` filter to the three-case minimum (GET miss, HEAD miss, merged list
-# pagination) and pass `--no-tests=fail` so a rename cannot silently select
-# nothing; the MinIO job runs the whole profile, backfill included.
-#
-# These cases are deliberately absent from every other lane: without an
-# interop source they only re-prove what `get_basic_test` and
-# `list_through_test` already cover in e2e-smoke and e2e-full. The committed
-# selection digest is the guard against a rename dropping one of them.
-[profile.e2e-odm-interop]
-default-filter = 'package(e2e_test) & test(/^on_demand_migration::interop_test::/)'
-fail-fast = false
-
-[profile.e2e-odm-interop.junit]
-# Emitted to target/nextest/e2e-odm-interop/junit.xml; the lane uploads it and
-# reconciles it against the per-case JSON report entries.
-path = "junit.xml"
-
-# ---------------------------------------------------------------------------
-# e2e-protocols profile — serial protocol lane
-# ---------------------------------------------------------------------------
-# The suite owns fixed ports, so the nightly workflow runs this exact profile
-# with one nextest worker.
-[profile.e2e-protocols]
-default-filter = 'package(e2e_test) & test(/^protocols::/)'
-fail-fast = false
-
-[profile.e2e-protocols.junit]
-path = "junit.xml"
-
-# ---------------------------------------------------------------------------
-# e2e-full profile — merge-gate full single-node e2e lane (backlog#1149 ci-5)
-# ---------------------------------------------------------------------------
-# The merge gate (ci.yml `e2e-full` job: push main + merge_group +
-# workflow_dispatch). Runs the user-visible KMS, object-lock, multipart-auth,
-# quota, checksum, encryption,
-# security-boundary, ... — that the fast PR `e2e-smoke` subset deliberately
-# skips. Budget <= 45 min; authority for the suite count is `cargo nextest list
-# --profile e2e-full -p e2e_test` (platform-dependent; see docs/testing/README.md).
-#
-# The filter is "the whole e2e_test crate MINUS the sets owned by other lanes":
-#   * protocols:: — FTPS/SFTP/WebDAV, run from the dedicated protocol profile
-#     with one worker because the suite owns fixed ports.
-#   * cluster suites that spin up a RustFSTestClusterEnvironment
-#     (cluster_concurrency, cluster_multidrive_pool, stale_multipart_cleanup_cluster,
-#     namespace_lock_quorum, admin_timeout_regression, object_lambda) — too
-#     heavy for the merge budget; they run in the e2e-nightly serial
-#     cluster-fault lane. heal_erasure_disk_rebuild is intentionally not
-#     excluded here because backlog#2213 promotes core heal rebuild coverage to
-#     this merge/main lane while retaining nightly coverage.
-#   * distributed:: — 4-node 4-disk Actions suite (S3, lock, versioning,
-#     replication, quota, observability, expand/decommission/rebalance, site
-#     replication, chaos, upgrade history/IAM). Owns [profile.e2e-distributed] and
-#     .github/workflows/e2e-distributed.yml.
-#   * on_demand_migration::interop_test — the ODM-20 provider interoperability
-#     cases, which are meaningless without a source: they run in the dedicated
-#     [profile.e2e-odm-interop] lane below, where the workflow points them at a
-#     MinIO container or a real cloud provider. Excluding them here also keeps
-#     this profile's committed selection digest stable.
-#   * replication_extension_test — repl-1 already splits it into the PR
-#     `e2e-smoke` (20 fast) and `e2e-repl-nightly` (56 slow) lanes and reserves
-#     it for those, so e2e-full does not double-run it.
-#   * #[ignore]d tests — nextest skips them by default (no --run-ignored); the
-#     manual-localhost:9000 reliant tests are ci-13's migration.
-#
-# Each e2e test spawns its own single-node rustfs server on a random port with
-# an isolated temp dir (crates/e2e_test/src/common.rs), so the set is
-# parallel-safe — the same property e2e-smoke relies on. The exceptions are the
-# 4-disk reliability / degraded-read fault-injection tests, multi-node heal
-# clusters, and the fixed-port Vault tests, all serialized below.
-[profile.e2e-full]
-default-filter = """
-    package(e2e_test)
-    & !test(/^protocols::/)
-    & !test(/^(admin_timeout_regression_test|cluster_concurrency_test|cluster_multidrive_pool_test|degraded_listing_availability_test|namespace_lock_quorum_test|object_lambda_test|stale_multipart_cleanup_cluster_test)::/)
-    & !test(/^distributed::/)
-    & !test(/^replication_extension_test::/)
-    & !test(/^replication_target_matrix_test::/)
-    & !test(/^on_demand_migration::(concurrency_test|fault_test|interop_test|real_source_test)::/)
-    & !test(/^upgrade_compatibility_test::upgrade_write_readiness_tests::/)
-"""
-fail-fast = false
-
-[profile.e2e-full.junit]
-# Emitted to target/nextest/e2e-full/junit.xml; uploaded by the e2e-full job.
-path = "junit.xml"
-
-# Serialize the 4-disk reliability / degraded-read e2e tests under e2e-full too
-# (see the e2e-reliability test-group note near the top of this file). Not a
-# quarantine: no retries, just single-threaded so several 4-disk servers never
-# run concurrently.
-[[profile.e2e-full.overrides]]
-filter = 'package(e2e_test) & test(/^(reliability_disk_fault|degraded_read_eof_regression|replacement_privileged_e2e)_test::/)'
-test-group = 'e2e-reliability'
-
-[[profile.e2e-full.overrides]]
-filter = 'package(e2e_test) & test(/^inline_fast_path_cluster_test::/)'
-test-group = 'e2e-inline-boundaries'
-
-# Heal interruption cases start whole multi-node clusters. Bound their overlap
-# just as the nightly lane does, without changing membership or retry policy.
-[[profile.e2e-full.overrides]]
-filter = 'package(e2e_test) & test(/^heal_erasure_disk_rebuild_test::/)'
-test-group = 'e2e-cluster-nightly'
-
-[[profile.e2e-full.overrides]]
-filter = 'package(e2e_test) & (test(/^kms::kms_vault_test::/) | test(/^kms::kms_rekey_sweep_test::/) | test(/^kms::configured_roundtrip_test::test_configured_vault_kms_admin_and_versioned_cleanup$/))'
-test-group = 'e2e-vault'
-
-# Keep the storage-heavy smoke cases isolated in the full suite too. They
-# start multiple server fixtures and issue many sequential S3 operations.
-[[profile.e2e-full.overrides]]
-filter = 'package(e2e_test) & (test(/^list_objects_v2_pagination_test::tests::(test_list_objects_v2_delimiter_small_page_traverses_all|test_list_objects_v2_max_keys_above_limit_returns_token|test_list_objects_v2_maxkeys_above_limit_with_delimiter)$/) | test(=reliant::tiering::test_hermetic_transition_restore_failure_expiry_and_retry))'
-threads-required = "num-test-threads"

.config/scanner-heal-required-tests.json

@@ -1,449 +0,0 @@
-{
-  "schema": 2,
-  "cases": {
-    "background-target-restart": {
-      "gate": "G14",
-      "task": "W21",
-      "lane": "e2e-nightly",
-      "suite": "e2e_test",
-      "name": "heal_erasure_disk_rebuild_test::tests::test_cluster_root_heal_recovers_remote_shards_after_background_target_restart",
-      "oracle": "background-target-restart.json",
-      "evidence": "process-restart",
-      "unclean_shutdown_marker": false,
-      "min_objects": 9,
-      "max_objects": 65,
-      "topology": {"nodes": 4, "drives_per_node": 1},
-      "erasure": {"data_blocks": 2, "parity_blocks": 2},
-      "scope": "Target process restart, exact unversioned S3 bodies and replacement-disk shards; not power loss or EC8+4."
-    },
-    "background-target-crash": {
-      "gate": "G14",
-      "task": "W21",
-      "lane": "e2e-nightly",
-      "suite": "e2e_test",
-      "name": "heal_erasure_disk_rebuild_test::tests::test_cluster_root_heal_recovers_remote_shards_after_background_target_crash",
-      "oracle": "background-target-crash.json",
-      "evidence": "process-crash-restart",
-      "unclean_shutdown_marker": true,
-      "min_objects": 9,
-      "max_objects": 65,
-      "topology": {"nodes": 4, "drives_per_node": 1},
-      "erasure": {"data_blocks": 2, "parity_blocks": 2},
-      "scope": "Target process killed during partial background rebuild, real unclean-shutdown marker, exact unversioned S3 bodies and replacement-disk shards; not power loss or EC8+4."
-    },
-    "ec84-target-drive-restart": {
-      "gate": "G14",
-      "task": "W20/W21",
-      "lane": "e2e-distributed",
-      "suite": "e2e_test",
-      "name": "distributed::heal_test::three_node_four_drive_ec8_4_root_heal_rebuilds_replaced_drive_after_restart",
-      "oracle": "ec84-target-drive-restart.json",
-      "evidence": "process-restart",
-      "unclean_shutdown_marker": false,
-      "min_objects": 5,
-      "max_objects": 5,
-      "topology": {"nodes": 3, "drives_per_node": 4},
-      "erasure": {"data_blocks": 8, "parity_blocks": 4},
-      "erasure_set_drive_count": 12,
-      "scope": "3-node x 4-drive single-set EC8+4, graceful target restart, preformatted replacement drive, exact unversioned S3 bodies and physical target shards; not mixed-version, multi-pool or long-window ABBA."
-    },
-    "background-target-restart-ec8-4": {
-      "gate": "G14",
-      "task": "W21",
-      "lane": "e2e-nightly",
-      "suite": "e2e_test",
-      "name": "heal_erasure_disk_rebuild_test::tests::test_cluster_root_heal_recovers_ec84_shards_after_background_target_restart",
-      "oracle": "background-target-restart-ec8-4.json",
-      "evidence": "process-restart",
-      "unclean_shutdown_marker": false,
-      "min_objects": 9,
-      "max_objects": 65,
-      "topology": {"nodes": 3, "drives_per_node": 4},
-      "erasure": {"data_blocks": 8, "parity_blocks": 4},
-      "erasure_set_drive_count": 12,
-      "scope": "Target process restart during partial background rebuild on a single 3x4 EC8+4 set; exact unversioned S3 bodies and replacement-drive shards; not power loss, multi-set, or multi-pool."
-    },
-    "background-target-crash-ec8-4": {
-      "gate": "G14",
-      "task": "W21",
-      "lane": "e2e-nightly",
-      "suite": "e2e_test",
-      "name": "heal_erasure_disk_rebuild_test::tests::test_cluster_root_heal_recovers_ec84_shards_after_background_target_crash",
-      "oracle": "background-target-crash-ec8-4.json",
-      "evidence": "process-crash-restart",
-      "unclean_shutdown_marker": true,
-      "min_objects": 9,
-      "max_objects": 65,
-      "topology": {"nodes": 3, "drives_per_node": 4},
-      "erasure": {"data_blocks": 8, "parity_blocks": 4},
-      "erasure_set_drive_count": 12,
-      "scope": "Target process killed during partial background rebuild on a single 3x4 EC8+4 set; real unclean-shutdown marker, exact unversioned S3 bodies and replacement-drive shards; not power loss, multi-set, or multi-pool."
-    },
-    "background-target-restart-ec8-4-multi-set": {
-      "gate": "G14",
-      "task": "W21",
-      "lane": "e2e-nightly",
-      "suite": "e2e_test",
-      "name": "heal_erasure_disk_rebuild_test::tests::test_cluster_root_heal_recovers_ec84_shards_across_multi_set_after_background_target_restart",
-      "oracle": "background-target-restart-ec8-4-multi-set.json",
-      "evidence": "process-restart",
-      "unclean_shutdown_marker": false,
-      "min_objects": 9,
-      "max_objects": 65,
-      "topology": {"nodes": 3, "drives_per_node": 8},
-      "erasure": {"data_blocks": 8, "parity_blocks": 4},
-      "erasure_set_drive_count": 12,
-      "sets": 2,
-      "pools": 1,
-      "scope": "Target process restart during partial background rebuild on a 3x8 EC8+4 layout with two erasure sets in one pool; exact unversioned S3 bodies and replacement-drive shards; not power loss or multi-pool."
-    },
-    "background-target-crash-ec8-4-multi-pool": {
-      "gate": "G14",
-      "task": "W21",
-      "lane": "e2e-nightly",
-      "suite": "e2e_test",
-      "name": "heal_erasure_disk_rebuild_test::tests::test_cluster_root_heal_recovers_ec84_shards_across_multi_pool_after_background_target_crash",
-      "oracle": "background-target-crash-ec8-4-multi-pool.json",
-      "evidence": "process-crash-restart",
-      "unclean_shutdown_marker": true,
-      "min_objects": 9,
-      "max_objects": 65,
-      "topology": {"nodes": 3, "drives_per_node": 12},
-      "erasure": {"data_blocks": 8, "parity_blocks": 4},
-      "erasure_set_drive_count": 12,
-      "sets": 3,
-      "pools": 3,
-      "outage_target_manifest_required": false,
-      "scope": "Target process crash during partial background rebuild on three single-node EC8+4 pools; exact baseline S3 bodies and replacement-drive shards, with down-window outage PUT refusal recorded and a deferred post-rejoin outage object verified through S3 when the full target pool was offline."
-    }
-  },
-  "release_lanes": {
-    "single-set-restart": {
-      "status": "implemented",
-      "cases": ["background-target-restart", "background-target-crash"],
-      "covers": ["four-node one-drive topology", "unversioned objects", "target restart/crash"]
-    },
-    "authority-coverage": {
-      "status": "pending",
-      "gates": ["G01", "G12"],
-      "requires": ["root authority coverage", "quota authority coverage"]
-    },
-    "checkpoint-and-crash": {
-      "status": "pending",
-      "gates": ["G02", "G04", "R-E"],
-      "requires": ["bounded checkpoint progress", "boundary crash matrix", "fixed-budget restart evidence"]
-    },
-    "status-and-outcome": {
-      "status": "pending",
-      "gates": ["G05", "G06", "R-D"],
-      "requires": ["per-object outcomes", "legacy status clients", "manager/event/ledger disposition"]
-    },
-    "mrf-responsibility": {
-      "status": "pending",
-      "gates": ["G07", "G08", "P4"],
-      "requires": ["durable MRF responsibility", "disk-full and replica-loss matrix", "MRF replay cost"]
-    },
-    "mixed-version-rollback": {
-      "status": "pending",
-      "gates": ["G03", "G09", "R-L"],
-      "requires": ["mixed-version peers", "rollback payloads", "crash-safe source retirement"]
-    },
-    "scheduler-pressure": {
-      "status": "pending",
-      "gates": ["G10", "P1", "P2", "P3"],
-      "requires": ["bounded scheduling", "foreground latency and throughput", "two-hour pressure evidence"]
-    },
-    "maintenance-producers": {
-      "status": "pending",
-      "gates": ["G11", "G13"],
-      "requires": ["complete producer coverage", "quorum-minus-one and remount matrix"]
-    },
-    "ec8-4-multiset": {
-      "status": "pending",
-      "gates": ["G14"],
-      "requires": ["3x4 EC8+4 topology", "multi-set coverage", "multi-pool coverage"]
-    }
-  },
-  "release_requirements": [
-    {
-      "gate": "G01",
-      "task": "W02/W04",
-      "lane": "authority-coverage",
-      "status": "pending",
-      "description": "Complete root and quota authority coverage",
-      "requires": ["root authority evidence", "quota authority evidence"],
-      "evidence_fields": [
-        "root_authority_evidence",
-        "quota_authority_evidence"
-      ]
-    },
-    {
-      "gate": "G02",
-      "task": "W03",
-      "lane": "checkpoint-and-crash",
-      "status": "pending",
-      "description": "Bounded checkpoint progress and independent version inventory",
-      "requires": ["bounded checkpoint oracle", "independent version inventory"],
-      "evidence_fields": [
-        "bounded_checkpoint_oracle",
-        "independent_version_inventory"
-      ]
-    },
-    {
-      "gate": "G03",
-      "task": "W17/W18",
-      "lane": "mixed-version-rollback",
-      "status": "pending",
-      "description": "Exact scoped ACK with durable publication and mixed peers",
-      "requires": ["durable scoped ACK publication", "mixed-peer evidence"],
-      "evidence_fields": [
-        "durable_root_publication_proof",
-        "scoped_ack_request_identity",
-        "participating_peer_capability_snapshot",
-        "mixed_peer_ack_fallback_oracle"
-      ]
-    },
-    {
-      "gate": "G04",
-      "task": "W03/W15/W16",
-      "lane": "checkpoint-and-crash",
-      "status": "pending",
-      "description": "Crash at every cache, root, floor and intent boundary",
-      "requires": ["cache boundary crash evidence", "root/floor/intent crash evidence"],
-      "evidence_fields": [
-        "cache_boundary_crash_evidence",
-        "root_floor_intent_crash_evidence"
-      ]
-    },
-    {
-      "gate": "G05",
-      "task": "W06/W07",
-      "lane": "status-and-outcome",
-      "status": "pending",
-      "description": "Per-object outcomes and bounded terminal retention",
-      "requires": ["per-object outcome oracle", "terminal retention bounds"],
-      "evidence_fields": [
-        "per_object_outcome_oracle",
-        "terminal_retention_bounds"
-      ]
-    },
-    {
-      "gate": "G06",
-      "task": "W06/W08/W23",
-      "lane": "status-and-outcome",
-      "status": "pending",
-      "description": "Concurrent status, legacy clients and truncation",
-      "requires": ["concurrent status evidence", "legacy client compatibility", "truncation behavior"],
-      "evidence_fields": [
-        "concurrent_status_evidence",
-        "legacy_client_compatibility",
-        "truncation_behavior"
-      ]
-    },
-    {
-      "gate": "G07",
-      "task": "W12/W13/W14",
-      "lane": "mrf-responsibility",
-      "status": "pending",
-      "description": "Durable MRF responsibility at every commit boundary",
-      "requires": ["MRF responsibility oracle", "commit-boundary crash matrix"],
-      "evidence_fields": [
-        "mrf_responsibility_oracle",
-        "commit_boundary_crash_matrix"
-      ]
-    },
-    {
-      "gate": "G08",
-      "task": "W12/W13/W14",
-      "lane": "mrf-responsibility",
-      "status": "pending",
-      "description": "MRF capacity, disk-full and replica-loss matrix",
-      "requires": ["MRF capacity evidence", "disk-full matrix", "replica-loss matrix"],
-      "evidence_fields": [
-        "mrf_capacity_evidence",
-        "disk_full_matrix",
-        "replica_loss_matrix"
-      ]
-    },
-    {
-      "gate": "G09",
-      "task": "W13/W18/W23",
-      "lane": "mixed-version-rollback",
-      "status": "pending",
-      "description": "Actual mixed-version reader/writer and rollback payloads",
-      "requires": ["mixed-version reader evidence", "mixed-version writer evidence", "rollback payload evidence"],
-      "evidence_fields": [
-        "mixed_version_reader_evidence",
-        "mixed_version_writer_evidence",
-        "rollback_payload_evidence"
-      ]
-    },
-    {
-      "gate": "G10",
-      "task": "W05/W09/W10/W11",
-      "lane": "scheduler-pressure",
-      "status": "pending",
-      "description": "Bounded scheduling and pressure recovery",
-      "requires": ["scheduler bound evidence", "pressure recovery evidence"],
-      "evidence_fields": [
-        "scheduler_bound_evidence",
-        "pressure_recovery_evidence"
-      ]
-    },
-    {
-      "gate": "G11",
-      "task": "W04/W19/W24",
-      "lane": "maintenance-producers",
-      "status": "pending",
-      "description": "Maintenance and complete producer coverage",
-      "requires": [
-        "maintenance producer matrix",
-        "complete producer inventory",
-        "segment activation preflight"
-      ],
-      "evidence_fields": [
-        "maintenance_producer_matrix",
-        "complete_producer_inventory",
-        "segment_activation_preflight"
-      ]
-    },
-    {
-      "gate": "G12",
-      "task": "W02/W15/W16",
-      "lane": "authority-coverage",
-      "status": "pending",
-      "description": "Both quota paths during reset and settlement",
-      "requires": ["reset quota-path evidence", "settlement quota-path evidence"],
-      "evidence_fields": [
-        "reset_quota_path_evidence",
-        "settlement_quota_path_evidence"
-      ]
-    },
-    {
-      "gate": "G13",
-      "task": "W07/W14",
-      "lane": "maintenance-producers",
-      "status": "pending",
-      "description": "Quorum-minus-one, unknown disks, remount, Object Lock, dry-run, grace and commit tail",
-      "requires": ["quorum-minus-one matrix", "unknown-disk/remount matrix", "Object Lock dry-run grace evidence"],
-      "evidence_fields": [
-        "quorum_minus_one_matrix",
-        "unknown_disk_remount_matrix",
-        "object_lock_dry_run_grace_evidence"
-      ]
-    },
-    {
-      "gate": "G14",
-      "task": "W20/W21",
-      "lane": "ec8-4-multiset",
-      "status": "pending",
-      "description": "Same-window field evidence with 3x4 EC8+4 and multi-set/pool coverage",
-      "requires": [
-        "same-window field evidence",
-        "3x4 EC8+4 evidence",
-        "multi-set evidence",
-        "multi-pool evidence",
-        "distributed segment invalidation evidence"
-      ],
-      "evidence_fields": [
-        "same_window_field_evidence",
-        "ec8_4_evidence",
-        "multi_set_evidence",
-        "multi_pool_evidence",
-        "distributed_segment_invalidation_evidence"
-      ]
-    },
-    {
-      "gate": "P1",
-      "task": "W20",
-      "lane": "scheduler-pressure",
-      "status": "pending",
-      "description": "Measured cold-walk share and foreground latency/throughput",
-      "requires": ["cold-walk share measurement", "foreground latency/throughput measurement"],
-      "evidence_fields": [
-        "cold_walk_share_measurement",
-        "foreground_latency_throughput_measurement",
-        "profile_evidence"
-      ]
-    },
-    {
-      "gate": "P2",
-      "task": "W20/W24",
-      "lane": "scheduler-pressure",
-      "status": "pending",
-      "description": "Measured post-stop convergence and cold segment reuse",
-      "requires": ["post-stop convergence measurement", "cold segment reuse measurement"],
-      "evidence_fields": [
-        "post_stop_convergence_measurement",
-        "cold_segment_reuse_measurement"
-      ]
-    },
-    {
-      "gate": "P3",
-      "task": "W20",
-      "lane": "scheduler-pressure",
-      "status": "pending",
-      "description": "Measured two-hour pressure/heal capacity and recovery window",
-      "requires": ["two-hour pressure measurement", "heal capacity measurement", "recovery-window measurement"],
-      "evidence_fields": [
-        "two_hour_pressure_measurement",
-        "heal_capacity_measurement",
-        "recovery_window_measurement"
-      ]
-    },
-    {
-      "gate": "P4",
-      "task": "W20",
-      "lane": "mrf-responsibility",
-      "status": "pending",
-      "description": "Measured MRF scale and replay cost with retained responsibility",
-      "requires": ["MRF scale measurement", "MRF replay-cost measurement", "retained responsibility evidence", "cleanup/GC soak evidence"],
-      "evidence_fields": [
-        "mrf_scale_measurement",
-        "mrf_replay_cost_measurement",
-        "retained_responsibility_evidence",
-        "mrf_cleanup_gc_soak_evidence"
-      ]
-    },
-    {
-      "gate": "R-E",
-      "task": "W03/W05",
-      "lane": "checkpoint-and-crash",
-      "status": "pending",
-      "description": "Fixed-budget real process restart through enumeration and classification",
-      "requires": ["fixed-budget restart evidence", "enumeration evidence", "classification evidence"],
-      "evidence_fields": [
-        "fixed_budget_restart_evidence",
-        "enumeration_evidence",
-        "classification_evidence"
-      ]
-    },
-    {
-      "gate": "R-D",
-      "task": "W07/W14",
-      "lane": "status-and-outcome",
-      "status": "pending",
-      "description": "Manager-to-event-to-ledger exact disposition, including grace",
-      "requires": ["manager disposition evidence", "event disposition evidence", "ledger disposition evidence", "grace handling"],
-      "evidence_fields": [
-        "manager_disposition_evidence",
-        "event_disposition_evidence",
-        "ledger_disposition_evidence",
-        "grace_handling"
-      ]
-    },
-    {
-      "gate": "R-L",
-      "task": "W13/W14",
-      "lane": "mixed-version-rollback",
-      "status": "pending",
-      "description": "Legacy source conflicts, migration gaps and crash-safe source retirement",
-      "requires": ["legacy source-conflict evidence", "migration-gap evidence", "crash-safe source retirement evidence"],
-      "evidence_fields": [
-        "legacy_source_conflict_evidence",
-        "migration_gap_evidence",
-        "crash_safe_source_retirement_evidence"
-      ]
-    }
-  ]
-}

.config/security-smoke-floor.txt

@@ -1,12 +0,0 @@
-# Committed floor for the number of security negative-auth tests selected by the
-# e2e-smoke PR profile (see scripts/check_security_smoke_count.sh, backlog#1151
-# sec-5).
-#
-# The floor equals the exact count of e2e_test cases whose name starts with a
-# security module prefix (negative_sigv4_test, presigned_negative_test,
-# admin_auth_test) that the [profile.e2e-smoke] default-filter in
-# .config/nextest.toml selects, at the time this file was last updated. CI fails
-# if the selected count drops below this number, so a rename or removal that
-# thins the security smoke gate must update this file in the same PR.
-# Adding tests does not require a bump, but bumping keeps the guard tight.
-28

.docker/README.md

@@ -1,131 +0,0 @@
-# RustFS Docker Infrastructure
-
-This directory contains the complete Docker infrastructure for building, deploying, and monitoring RustFS. It provides ready-to-use configurations for development, testing, and production-grade observability.
-
-## 📂 Directory Structure
-
-| Directory | Description | Status |
-| :--- | :--- | :--- |
-| **[`observability/`](observability/README.md)** | **[RECOMMENDED]** Full-stack observability (Prometheus, Grafana, Tempo, Loki). | ✅ Production-Ready |
-| **[`compose/`](compose/README.md)** | Specialized setups (e.g., 4-node distributed cluster testing). | ⚠️ Testing Only |
-| **[`mqtt/`](mqtt/README.md)** | EMQX Broker configuration for MQTT integration testing. | 🧪 Development |
-| **[`openobserve-otel/`](openobserve-otel/README.md)** | Alternative lightweight observability stack using OpenObserve. | 🔄 Alternative |
-
----
-
-## 📄 Root Directory Files
-
-The following files in the project root are essential for Docker operations:
-
-### Build Scripts & Dockerfiles
-
-| File | Description | Usage |
-| :--- | :--- | :--- |
-| **`docker-buildx.sh`** | **Multi-Arch Build Script**<br>Automates building and pushing Docker images for `amd64` and `arm64`. Supports release and dev channels. | `./docker-buildx.sh --push` |
-| **`Dockerfile`** | **Production Image (Alpine)**<br>Lightweight image using musl libc. Downloads pre-built binaries from GitHub Releases. | `docker build -t rustfs:latest .` |
-| **`Dockerfile.glibc`** | **Production Image (Ubuntu)**<br>Standard image using glibc. Useful if you need specific dynamic libraries. | `docker build -f Dockerfile.glibc .` |
-| **`Dockerfile.source`** | **Development Image**<br>Builds RustFS from source code. Includes build tools. Ideal for local development and CI. | `docker build -f Dockerfile.source .` |
-
-### Docker Compose Configurations
-
-| File | Description | Usage |
-| :--- | :--- | :--- |
-| **`docker-compose.yml`** | **Main Development Setup**<br>Comprehensive setup with profiles for development, observability, and proxying. | `docker compose up -d`<br>`docker compose --profile observability up -d` |
-| **`docker-compose-simple.yml`** | **Quick Start Setup**<br>Minimal configuration running a single RustFS instance with 4 volumes. Perfect for first-time users. | `docker compose -f docker-compose-simple.yml up -d` |
-
----
-
-## 🌟 Observability Stack (Recommended)
-
-Located in: [`.docker/observability/`](observability/README.md)
-
-We provide a comprehensive, industry-standard observability stack designed for deep insights into RustFS performance. This is the recommended setup for both development and production monitoring.
-
-### Components
-- **Metrics**: Prometheus (Collection) + Grafana (Visualization)
-- **Traces**: Tempo (Storage) + Jaeger (UI)
-- **Logs**: Loki
-- **Ingestion**: OpenTelemetry Collector
-
-### Key Features
-- **Full Persistence**: All metrics, logs, and traces are saved to Docker volumes, ensuring no data loss on restarts.
-- **Correlation**: Seamlessly jump between Logs, Traces, and Metrics in Grafana.
-- **High Performance**: Optimized configurations for batching, compression, and memory management.
-
-### Quick Start
-```bash
-cd .docker/observability
-docker compose up -d
-```
-
----
-
-## 🧪 Specialized Environments
-
-Located in: [`.docker/compose/`](compose/README.md)
-
-These configurations are tailored for specific testing scenarios that require complex topologies.
-
-### Distributed Cluster (4-Nodes)
-Simulates a real-world distributed environment with 4 RustFS nodes running locally.
-```bash
-docker compose -f .docker/compose/docker-compose.cluster.yaml up -d
-```
-
-### Integrated Observability Test
-A self-contained environment running 4 RustFS nodes alongside the full observability stack. Useful for end-to-end testing of telemetry.
-```bash
-docker compose -f .docker/compose/docker-compose.observability.yaml up -d
-```
-
----
-
-## 📡 MQTT Integration
-
-Located in: [`.docker/mqtt/`](mqtt/README.md)
-
-Provides an EMQX broker for testing RustFS MQTT features.
-
-### Quick Start
-```bash
-cd .docker/mqtt
-docker compose up -d
-```
-- **Dashboard**: [http://localhost:18083](http://localhost:18083) (Default: `admin` / `public`)
-- **MQTT Port**: `1883`
-
----
-
-## 👁️ Alternative: OpenObserve
-
-Located in: [`.docker/openobserve-otel/`](openobserve-otel/README.md)
-
-For users preferring a lightweight, all-in-one solution, we support OpenObserve. It combines logs, metrics, and traces into a single binary and UI.
-
-### Quick Start
-```bash
-cd .docker/openobserve-otel
-docker compose up -d
-```
-
----
-
-## 🔧 Common Operations
-
-### Cleaning Up
-To stop all containers and remove volumes (**WARNING**: deletes all persisted data):
-```bash
-docker compose down -v
-```
-
-### Viewing Logs
-To follow logs for a specific service:
-```bash
-docker compose logs -f [service_name]
-```
-
-### Checking Status
-To see the status of all running containers:
-```bash
-docker compose ps
-```

.docker/compose/README.md

@@ -1,102 +0,0 @@
-# Specialized Docker Compose Configurations
-
-This directory contains specialized Docker Compose configurations for specific testing scenarios.
-
-## ⚠️ Important Note
-
-**For Observability:**
-We **strongly recommend** using the new, fully integrated observability stack located in `../observability/`. It provides a production-ready setup with Prometheus, Grafana, Tempo, Loki, and OpenTelemetry Collector, all with persistent storage and optimized configurations.
-
-The `docker-compose.observability.yaml` in this directory is kept for legacy reference or specific minimal testing needs but is **not** the primary recommended setup.
-
-## 📁 Configuration Files
-
-### Cluster Testing
-
-- **`docker-compose.cluster.yaml`**
-  - **Purpose**: Simulates a 4-node RustFS distributed cluster.
-  - **Use Case**: Testing distributed storage logic, consensus, and failover.
-  - **Nodes**: 4 RustFS instances.
-  - **Storage**: Uses local HTTP endpoints.
-
-### Legacy / Minimal Observability
-
-- **`docker-compose.observability.yaml`**
-  - **Purpose**: A minimal observability setup.
-  - **Status**: **Deprecated**. Please use `../observability/docker-compose.yml` instead.
-
-## 🚀 Usage Examples
-
-### Cluster Testing
-
-To start a 4-node cluster for distributed testing:
-
-```bash
-# From project root
-docker compose -f .docker/compose/docker-compose.cluster.yaml up -d
-```
-
-### Script-Based 4-Node Validation (Recommended)
-
-Use the local validation script when you need local-source image build, failover checks,
-and benchmark workflow in one command:
-
-```bash
-# Default mode: WAIT_PROBE_MODE=service
-# This avoids false negatives where /health/ready remains 503 locally
-# while the service path is already available.
-./scripts/run_four_node_cluster_failover_bench.sh
-```
-
-Strict mode is available when you explicitly want `/health/ready == 200` as the gate:
-
-```bash
-WAIT_PROBE_MODE=ready ./scripts/run_four_node_cluster_failover_bench.sh
-```
-
-### Profiling + Trace Validation
-
-The profiling-focused 4-node compose keeps profiling enabled and points RustFS
-to an OTLP/HTTP collector endpoint:
-
-```bash
-docker compose -f .docker/compose/docker-compose.cluster.local-build.profiling-amd64.yml up -d
-```
-
-Important behavior notes:
-
-- `RUSTFS_OBS_ENDPOINT` is the OTLP/HTTP base URL. RustFS automatically sends
-  traces to `/v1/traces`, metrics to `/v1/metrics`, and logs to `/v1/logs`.
-- Startup usually produces logs and metrics first. That does not guarantee
-  visible traces yet.
-- Trace data becomes obvious only after real HTTP/S3/gRPC requests hit RustFS.
-- `RUSTFS_OBS_LOGGER_LEVEL=info` keeps the top-level request span but filters
-  many nested `debug` spans. If Tempo/Jaeger looks sparse, retry with
-  `RUSTFS_OBS_LOGGER_LEVEL=debug` before suspecting the collector.
-
-Minimal trace verification flow:
-
-```bash
-# 1. Start the profiling compose with richer span visibility.
-RUSTFS_OBS_LOGGER_LEVEL=debug \
-docker compose -f .docker/compose/docker-compose.cluster.local-build.profiling-amd64.yml up -d
-
-# 2. Generate real request traffic after startup.
-curl -I http://127.0.0.1:9000/health
-curl -I http://127.0.0.1:9000/health/ready
-
-# 3. Then inspect Tempo or Jaeger.
-# Grafana: http://localhost:3000
-# Jaeger:  http://localhost:16686
-```
-
-If logs and metrics are present but traces are sparse, the most common cause is
-"no real request traffic yet" or "`info` level filtered nested spans", not an
-OTLP routing failure.
-
-### (Deprecated) Minimal Observability
-
-```bash
-# From project root
-docker compose -f .docker/compose/docker-compose.observability.yaml up -d
-```

.docker/compose/docker-compose.cluster.local-build.profiling-amd64.yml

@@ -1,240 +0,0 @@
-# Copyright 2024 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-# Profiling-first 4-node local-build compose.
-#
-# Goals:
-# - force linux/amd64 runtime/build on Apple Silicon hosts;
-# - enable RustFS built-in CPU profiling;
-# - keep all tuning knobs host-overridable via env.
-#
-# Observability notes:
-# - `RUSTFS_OBS_ENDPOINT` is the OTLP/HTTP base URL. RustFS appends
-#   `/v1/traces`, `/v1/metrics`, and `/v1/logs` automatically.
-# - Logs and metrics usually appear during startup. Traces mainly appear after
-#   real HTTP/S3/gRPC requests create spans.
-# - `RUSTFS_OBS_LOGGER_LEVEL=info` keeps the top-level request trace span but
-#   filters many `debug`-level nested spans. Use `debug` when validating trace
-#   richness rather than collector reachability.
-
-services:
-  node1:
-    platform: ${RUSTFS_DOCKER_PLATFORM:-linux/amd64}
-    image: ${RUSTFS_IMAGE:-rustfs/rustfs:local-4node}
-    build:
-      context: ../..
-      dockerfile: Dockerfile.source
-    hostname: node1
-    environment:
-      - RUSTFS_VOLUMES=http://node{1...4}:9000/data/rustfs{0...3}
-      - RUSTFS_ADDRESS=:9000
-      - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfs-cluster-admin}
-      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfs-cluster-secret}
-      - RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
-      - OTEL_RESOURCE_ATTRIBUTES=${OTEL_RESOURCE_ATTRIBUTES:-rustfs.cluster.id=rustfs-dev}
-      # `info` is enough for startup logs/metrics. Use `debug` if Tempo/Jaeger
-      # should show richer nested spans during request-path verification.
-      - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
-      - RUSTFS_OBS_PROFILING_ENDPOINT=${RUSTFS_OBS_PROFILING_ENDPOINT:-http://host.docker.internal:4040}
-      - RUSTFS_OBS_PROFILING_EXPORT_ENABLED=${RUSTFS_OBS_PROFILING_EXPORT_ENABLED:-true}
-      - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
-      - RUSTFS_ENABLE_PROFILING=${RUSTFS_ENABLE_PROFILING:-true}
-      - RUSTFS_PROF_CPU_MODE=${RUSTFS_PROF_CPU_MODE:-continuous}
-      - RUSTFS_PROF_CPU_FREQ=${RUSTFS_PROF_CPU_FREQ:-99}
-      - RUSTFS_PROF_OUTPUT_DIR=${RUSTFS_PROF_OUTPUT_DIR:-/tmp/rustfs-profiles}
-      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=${RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS:-48}
-      - RUSTFS_OBJECT_IO_BUFFER_SIZE=${RUSTFS_OBJECT_IO_BUFFER_SIZE:-262144}
-      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=${RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD:-6}
-      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=${RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD:-12}
-      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=${RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY:-8}
-      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=${RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE:-8388608}
-      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=${RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES:-25165824}
-      - RUSTFS_RUNTIME_WORKER_THREADS=${RUSTFS_RUNTIME_WORKER_THREADS:-12}
-      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=${RUSTFS_RUNTIME_MAX_BLOCKING_THREADS:-512}
-      - RUSTFS_ALLOCATOR_RECLAIM_ENABLED=${RUSTFS_ALLOCATOR_RECLAIM_ENABLED:-false}
-    extra_hosts:
-      - "host.docker.internal:host-gateway"
-    volumes:
-      - node1_data_0:/data/rustfs0
-      - node1_data_1:/data/rustfs1
-      - node1_data_2:/data/rustfs2
-      - node1_data_3:/data/rustfs3
-    ports:
-      - "9000:9000"
-    networks:
-      - rustfs-cluster-net
-
-  node2:
-    platform: ${RUSTFS_DOCKER_PLATFORM:-linux/amd64}
-    image: ${RUSTFS_IMAGE:-rustfs/rustfs:local-4node}
-    build:
-      context: ../..
-      dockerfile: Dockerfile.source
-    hostname: node2
-    environment:
-      - RUSTFS_VOLUMES=http://node{1...4}:9000/data/rustfs{0...3}
-      - RUSTFS_ADDRESS=:9000
-      - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfs-cluster-admin}
-      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfs-cluster-secret}
-      - RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
-      - OTEL_RESOURCE_ATTRIBUTES=${OTEL_RESOURCE_ATTRIBUTES:-rustfs.cluster.id=rustfs-dev}
-      # `info` is enough for startup logs/metrics. Use `debug` if Tempo/Jaeger
-      # should show richer nested spans during request-path verification.
-      - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
-      - RUSTFS_OBS_PROFILING_ENDPOINT=${RUSTFS_OBS_PROFILING_ENDPOINT:-http://host.docker.internal:4040}
-      - RUSTFS_OBS_PROFILING_EXPORT_ENABLED=${RUSTFS_OBS_PROFILING_EXPORT_ENABLED:-true}
-      - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
-      - RUSTFS_ENABLE_PROFILING=${RUSTFS_ENABLE_PROFILING:-true}
-      - RUSTFS_PROF_CPU_MODE=${RUSTFS_PROF_CPU_MODE:-continuous}
-      - RUSTFS_PROF_CPU_FREQ=${RUSTFS_PROF_CPU_FREQ:-99}
-      - RUSTFS_PROF_OUTPUT_DIR=${RUSTFS_PROF_OUTPUT_DIR:-/tmp/rustfs-profiles}
-      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=${RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS:-48}
-      - RUSTFS_OBJECT_IO_BUFFER_SIZE=${RUSTFS_OBJECT_IO_BUFFER_SIZE:-262144}
-      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=${RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD:-6}
-      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=${RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD:-12}
-      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=${RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY:-8}
-      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=${RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE:-8388608}
-      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=${RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES:-25165824}
-      - RUSTFS_RUNTIME_WORKER_THREADS=${RUSTFS_RUNTIME_WORKER_THREADS:-12}
-      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=${RUSTFS_RUNTIME_MAX_BLOCKING_THREADS:-512}
-      - RUSTFS_ALLOCATOR_RECLAIM_ENABLED=${RUSTFS_ALLOCATOR_RECLAIM_ENABLED:-false}
-    extra_hosts:
-      - "host.docker.internal:host-gateway"
-    volumes:
-      - node2_data_0:/data/rustfs0
-      - node2_data_1:/data/rustfs1
-      - node2_data_2:/data/rustfs2
-      - node2_data_3:/data/rustfs3
-    ports:
-      - "9001:9000"
-    networks:
-      - rustfs-cluster-net
-
-  node3:
-    platform: ${RUSTFS_DOCKER_PLATFORM:-linux/amd64}
-    image: ${RUSTFS_IMAGE:-rustfs/rustfs:local-4node}
-    build:
-      context: ../..
-      dockerfile: Dockerfile.source
-    hostname: node3
-    environment:
-      - RUSTFS_VOLUMES=http://node{1...4}:9000/data/rustfs{0...3}
-      - RUSTFS_ADDRESS=:9000
-      - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfs-cluster-admin}
-      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfs-cluster-secret}
-      - RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
-      - OTEL_RESOURCE_ATTRIBUTES=${OTEL_RESOURCE_ATTRIBUTES:-rustfs.cluster.id=rustfs-dev}
-      # `info` is enough for startup logs/metrics. Use `debug` if Tempo/Jaeger
-      # should show richer nested spans during request-path verification.
-      - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
-      - RUSTFS_OBS_PROFILING_ENDPOINT=${RUSTFS_OBS_PROFILING_ENDPOINT:-http://host.docker.internal:4040}
-      - RUSTFS_OBS_PROFILING_EXPORT_ENABLED=${RUSTFS_OBS_PROFILING_EXPORT_ENABLED:-true}
-      - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
-      - RUSTFS_ENABLE_PROFILING=${RUSTFS_ENABLE_PROFILING:-true}
-      - RUSTFS_PROF_CPU_MODE=${RUSTFS_PROF_CPU_MODE:-continuous}
-      - RUSTFS_PROF_CPU_FREQ=${RUSTFS_PROF_CPU_FREQ:-99}
-      - RUSTFS_PROF_OUTPUT_DIR=${RUSTFS_PROF_OUTPUT_DIR:-/tmp/rustfs-profiles}
-      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=${RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS:-48}
-      - RUSTFS_OBJECT_IO_BUFFER_SIZE=${RUSTFS_OBJECT_IO_BUFFER_SIZE:-262144}
-      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=${RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD:-6}
-      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=${RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD:-12}
-      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=${RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY:-8}
-      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=${RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE:-8388608}
-      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=${RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES:-25165824}
-      - RUSTFS_RUNTIME_WORKER_THREADS=${RUSTFS_RUNTIME_WORKER_THREADS:-12}
-      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=${RUSTFS_RUNTIME_MAX_BLOCKING_THREADS:-512}
-      - RUSTFS_ALLOCATOR_RECLAIM_ENABLED=${RUSTFS_ALLOCATOR_RECLAIM_ENABLED:-false}
-    extra_hosts:
-      - "host.docker.internal:host-gateway"
-    volumes:
-      - node3_data_0:/data/rustfs0
-      - node3_data_1:/data/rustfs1
-      - node3_data_2:/data/rustfs2
-      - node3_data_3:/data/rustfs3
-    ports:
-      - "9002:9000"
-    networks:
-      - rustfs-cluster-net
-
-  node4:
-    platform: ${RUSTFS_DOCKER_PLATFORM:-linux/amd64}
-    image: ${RUSTFS_IMAGE:-rustfs/rustfs:local-4node}
-    build:
-      context: ../..
-      dockerfile: Dockerfile.source
-    hostname: node4
-    environment:
-      - RUSTFS_VOLUMES=http://node{1...4}:9000/data/rustfs{0...3}
-      - RUSTFS_ADDRESS=:9000
-      - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfs-cluster-admin}
-      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfs-cluster-secret}
-      - RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
-      - OTEL_RESOURCE_ATTRIBUTES=${OTEL_RESOURCE_ATTRIBUTES:-rustfs.cluster.id=rustfs-dev}
-      # `info` is enough for startup logs/metrics. Use `debug` if Tempo/Jaeger
-      # should show richer nested spans during request-path verification.
-      - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
-      - RUSTFS_OBS_PROFILING_ENDPOINT=${RUSTFS_OBS_PROFILING_ENDPOINT:-http://host.docker.internal:4040}
-      - RUSTFS_OBS_PROFILING_EXPORT_ENABLED=${RUSTFS_OBS_PROFILING_EXPORT_ENABLED:-true}
-      - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
-      - RUSTFS_ENABLE_PROFILING=${RUSTFS_ENABLE_PROFILING:-true}
-      - RUSTFS_PROF_CPU_MODE=${RUSTFS_PROF_CPU_MODE:-continuous}
-      - RUSTFS_PROF_CPU_FREQ=${RUSTFS_PROF_CPU_FREQ:-99}
-      - RUSTFS_PROF_OUTPUT_DIR=${RUSTFS_PROF_OUTPUT_DIR:-/tmp/rustfs-profiles}
-      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=${RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS:-48}
-      - RUSTFS_OBJECT_IO_BUFFER_SIZE=${RUSTFS_OBJECT_IO_BUFFER_SIZE:-262144}
-      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=${RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD:-6}
-      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=${RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD:-12}
-      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=${RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY:-8}
-      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=${RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE:-8388608}
-      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=${RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES:-25165824}
-      - RUSTFS_RUNTIME_WORKER_THREADS=${RUSTFS_RUNTIME_WORKER_THREADS:-12}
-      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=${RUSTFS_RUNTIME_MAX_BLOCKING_THREADS:-512}
-      - RUSTFS_ALLOCATOR_RECLAIM_ENABLED=${RUSTFS_ALLOCATOR_RECLAIM_ENABLED:-false}
-    extra_hosts:
-      - "host.docker.internal:host-gateway"
-    volumes:
-      - node4_data_0:/data/rustfs0
-      - node4_data_1:/data/rustfs1
-      - node4_data_2:/data/rustfs2
-      - node4_data_3:/data/rustfs3
-    ports:
-      - "9003:9000"
-    networks:
-      - rustfs-cluster-net
-
-volumes:
-  node1_data_0:
-  node1_data_1:
-  node1_data_2:
-  node1_data_3:
-  node2_data_0:
-  node2_data_1:
-  node2_data_2:
-  node2_data_3:
-  node3_data_0:
-  node3_data_1:
-  node3_data_2:
-  node3_data_3:
-  node4_data_0:
-  node4_data_1:
-  node4_data_2:
-  node4_data_3:
-
-networks:
-  rustfs-cluster-net:
-    driver: bridge

.docker/compose/docker-compose.cluster.local-build.yml

@@ -1,224 +0,0 @@
-# Copyright 2024 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-services:
-  node1:
-    image: ${RUSTFS_IMAGE:-rustfs/rustfs:local-4node}
-    build:
-      context: ../..
-      dockerfile: Dockerfile.source
-    hostname: node1
-    environment:
-      - RUSTFS_VOLUMES=${RUSTFS_VOLUMES:-http://node{1...4}:9000/data/rustfs{0...3}}
-      - RUSTFS_ADDRESS=:9000
-      - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin-local}
-      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfssecret-local}
-      - RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
-      - OTEL_RESOURCE_ATTRIBUTES=${OTEL_RESOURCE_ATTRIBUTES:-rustfs.cluster.id=rustfs-dev}
-      - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
-      - RUSTFS_OBS_USE_STDOUT=${RUSTFS_OBS_USE_STDOUT:-false}
-      - RUSTFS_OBS_LOG_STDOUT_ENABLED=${RUSTFS_OBS_LOG_STDOUT_ENABLED:-false}
-      - RUSTFS_ISSUE3031_DIAG_ENABLE=${RUSTFS_ISSUE3031_DIAG_ENABLE:-false}
-      - RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT:-5}
-      - RUSTFS_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_LOCK_ACQUIRE_TIMEOUT:-5}
-      - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
-      # Internode gRPC optimization knobs (grpc-optimization P0-P3). Defaults match the binary
-      # defaults, so leaving these unset is a no-op; the A/B driver exports them to toggle a stage.
-      - RUSTFS_INTERNODE_RPC_TCP_NODELAY=${RUSTFS_INTERNODE_RPC_TCP_NODELAY:-true}
-      - RUSTFS_INTERNODE_RPC_HTTP2_STREAM_WINDOW_SIZE=${RUSTFS_INTERNODE_RPC_HTTP2_STREAM_WINDOW_SIZE:-1048576}
-      - RUSTFS_INTERNODE_RPC_HTTP2_CONN_WINDOW_SIZE=${RUSTFS_INTERNODE_RPC_HTTP2_CONN_WINDOW_SIZE:-2097152}
-      - RUSTFS_INTERNODE_RPC_MAX_MESSAGE_SIZE=${RUSTFS_INTERNODE_RPC_MAX_MESSAGE_SIZE:-104857600}
-      - RUSTFS_INTERNODE_CHANNEL_ISOLATION=${RUSTFS_INTERNODE_CHANNEL_ISOLATION:-false}
-      - RUSTFS_INTERNODE_BULK_CHANNELS=${RUSTFS_INTERNODE_BULK_CHANNELS:-2}
-      - RUSTFS_INTERNODE_RPC_MSGPACK_ONLY=${RUSTFS_INTERNODE_RPC_MSGPACK_ONLY:-false}
-      - RUSTFS_INTERNODE_PREWARM=${RUSTFS_INTERNODE_PREWARM:-false}
-      - RUSTFS_INTERNODE_OFFLINE_BYPASS=${RUSTFS_INTERNODE_OFFLINE_BYPASS:-false}
-      - RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS=${RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS:-5}
-      - RUSTFS_INTERNODE_OFFLINE_FAILURE_THRESHOLD=${RUSTFS_INTERNODE_OFFLINE_FAILURE_THRESHOLD:-3}
-      - RUSTFS_INTERNODE_IDEMPOTENT_READ_RETRIES=${RUSTFS_INTERNODE_IDEMPOTENT_READ_RETRIES:-1}
-    extra_hosts:
-      - "host.docker.internal:host-gateway"
-    volumes:
-      - node1_data_0:/data/rustfs0
-      - node1_data_1:/data/rustfs1
-      - node1_data_2:/data/rustfs2
-      - node1_data_3:/data/rustfs3
-    ports:
-      - "9000:9000"
-    networks:
-      - rustfs-cluster-net
-
-  node2:
-    image: ${RUSTFS_IMAGE:-rustfs/rustfs:local-4node}
-    build:
-      context: ../..
-      dockerfile: Dockerfile.source
-    hostname: node2
-    environment:
-      - RUSTFS_VOLUMES=${RUSTFS_VOLUMES:-http://node{1...4}:9000/data/rustfs{0...3}}
-      - RUSTFS_ADDRESS=:9000
-      - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin-local}
-      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfssecret-local}
-      - RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
-      - OTEL_RESOURCE_ATTRIBUTES=${OTEL_RESOURCE_ATTRIBUTES:-rustfs.cluster.id=rustfs-dev}
-      - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
-      - RUSTFS_OBS_USE_STDOUT=${RUSTFS_OBS_USE_STDOUT:-false}
-      - RUSTFS_OBS_LOG_STDOUT_ENABLED=${RUSTFS_OBS_LOG_STDOUT_ENABLED:-false}
-      - RUSTFS_ISSUE3031_DIAG_ENABLE=${RUSTFS_ISSUE3031_DIAG_ENABLE:-false}
-      - RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT:-5}
-      - RUSTFS_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_LOCK_ACQUIRE_TIMEOUT:-5}
-      - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
-      # Internode gRPC optimization knobs (grpc-optimization P0-P3). Defaults match the binary
-      # defaults, so leaving these unset is a no-op; the A/B driver exports them to toggle a stage.
-      - RUSTFS_INTERNODE_RPC_TCP_NODELAY=${RUSTFS_INTERNODE_RPC_TCP_NODELAY:-true}
-      - RUSTFS_INTERNODE_RPC_HTTP2_STREAM_WINDOW_SIZE=${RUSTFS_INTERNODE_RPC_HTTP2_STREAM_WINDOW_SIZE:-1048576}
-      - RUSTFS_INTERNODE_RPC_HTTP2_CONN_WINDOW_SIZE=${RUSTFS_INTERNODE_RPC_HTTP2_CONN_WINDOW_SIZE:-2097152}
-      - RUSTFS_INTERNODE_RPC_MAX_MESSAGE_SIZE=${RUSTFS_INTERNODE_RPC_MAX_MESSAGE_SIZE:-104857600}
-      - RUSTFS_INTERNODE_CHANNEL_ISOLATION=${RUSTFS_INTERNODE_CHANNEL_ISOLATION:-false}
-      - RUSTFS_INTERNODE_BULK_CHANNELS=${RUSTFS_INTERNODE_BULK_CHANNELS:-2}
-      - RUSTFS_INTERNODE_RPC_MSGPACK_ONLY=${RUSTFS_INTERNODE_RPC_MSGPACK_ONLY:-false}
-      - RUSTFS_INTERNODE_PREWARM=${RUSTFS_INTERNODE_PREWARM:-false}
-      - RUSTFS_INTERNODE_OFFLINE_BYPASS=${RUSTFS_INTERNODE_OFFLINE_BYPASS:-false}
-      - RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS=${RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS:-5}
-      - RUSTFS_INTERNODE_OFFLINE_FAILURE_THRESHOLD=${RUSTFS_INTERNODE_OFFLINE_FAILURE_THRESHOLD:-3}
-      - RUSTFS_INTERNODE_IDEMPOTENT_READ_RETRIES=${RUSTFS_INTERNODE_IDEMPOTENT_READ_RETRIES:-1}
-    extra_hosts:
-      - "host.docker.internal:host-gateway"
-    volumes:
-      - node2_data_0:/data/rustfs0
-      - node2_data_1:/data/rustfs1
-      - node2_data_2:/data/rustfs2
-      - node2_data_3:/data/rustfs3
-    ports:
-      - "9001:9000"
-    networks:
-      - rustfs-cluster-net
-
-  node3:
-    image: ${RUSTFS_IMAGE:-rustfs/rustfs:local-4node}
-    build:
-      context: ../..
-      dockerfile: Dockerfile.source
-    hostname: node3
-    environment:
-      - RUSTFS_VOLUMES=${RUSTFS_VOLUMES:-http://node{1...4}:9000/data/rustfs{0...3}}
-      - RUSTFS_ADDRESS=:9000
-      - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin-local}
-      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfssecret-local}
-      - RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
-      - OTEL_RESOURCE_ATTRIBUTES=${OTEL_RESOURCE_ATTRIBUTES:-rustfs.cluster.id=rustfs-dev}
-      - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
-      - RUSTFS_OBS_USE_STDOUT=${RUSTFS_OBS_USE_STDOUT:-false}
-      - RUSTFS_OBS_LOG_STDOUT_ENABLED=${RUSTFS_OBS_LOG_STDOUT_ENABLED:-false}
-      - RUSTFS_ISSUE3031_DIAG_ENABLE=${RUSTFS_ISSUE3031_DIAG_ENABLE:-false}
-      - RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT:-5}
-      - RUSTFS_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_LOCK_ACQUIRE_TIMEOUT:-5}
-      - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
-      # Internode gRPC optimization knobs (grpc-optimization P0-P3). Defaults match the binary
-      # defaults, so leaving these unset is a no-op; the A/B driver exports them to toggle a stage.
-      - RUSTFS_INTERNODE_RPC_TCP_NODELAY=${RUSTFS_INTERNODE_RPC_TCP_NODELAY:-true}
-      - RUSTFS_INTERNODE_RPC_HTTP2_STREAM_WINDOW_SIZE=${RUSTFS_INTERNODE_RPC_HTTP2_STREAM_WINDOW_SIZE:-1048576}
-      - RUSTFS_INTERNODE_RPC_HTTP2_CONN_WINDOW_SIZE=${RUSTFS_INTERNODE_RPC_HTTP2_CONN_WINDOW_SIZE:-2097152}
-      - RUSTFS_INTERNODE_RPC_MAX_MESSAGE_SIZE=${RUSTFS_INTERNODE_RPC_MAX_MESSAGE_SIZE:-104857600}
-      - RUSTFS_INTERNODE_CHANNEL_ISOLATION=${RUSTFS_INTERNODE_CHANNEL_ISOLATION:-false}
-      - RUSTFS_INTERNODE_BULK_CHANNELS=${RUSTFS_INTERNODE_BULK_CHANNELS:-2}
-      - RUSTFS_INTERNODE_RPC_MSGPACK_ONLY=${RUSTFS_INTERNODE_RPC_MSGPACK_ONLY:-false}
-      - RUSTFS_INTERNODE_PREWARM=${RUSTFS_INTERNODE_PREWARM:-false}
-      - RUSTFS_INTERNODE_OFFLINE_BYPASS=${RUSTFS_INTERNODE_OFFLINE_BYPASS:-false}
-      - RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS=${RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS:-5}
-      - RUSTFS_INTERNODE_OFFLINE_FAILURE_THRESHOLD=${RUSTFS_INTERNODE_OFFLINE_FAILURE_THRESHOLD:-3}
-      - RUSTFS_INTERNODE_IDEMPOTENT_READ_RETRIES=${RUSTFS_INTERNODE_IDEMPOTENT_READ_RETRIES:-1}
-    extra_hosts:
-      - "host.docker.internal:host-gateway"
-    volumes:
-      - node3_data_0:/data/rustfs0
-      - node3_data_1:/data/rustfs1
-      - node3_data_2:/data/rustfs2
-      - node3_data_3:/data/rustfs3
-    ports:
-      - "9002:9000"
-    networks:
-      - rustfs-cluster-net
-
-  node4:
-    image: ${RUSTFS_IMAGE:-rustfs/rustfs:local-4node}
-    build:
-      context: ../..
-      dockerfile: Dockerfile.source
-    hostname: node4
-    environment:
-      - RUSTFS_VOLUMES=${RUSTFS_VOLUMES:-http://node{1...4}:9000/data/rustfs{0...3}}
-      - RUSTFS_ADDRESS=:9000
-      - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin-local}
-      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfssecret-local}
-      - RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
-      - OTEL_RESOURCE_ATTRIBUTES=${OTEL_RESOURCE_ATTRIBUTES:-rustfs.cluster.id=rustfs-dev}
-      - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
-      - RUSTFS_OBS_USE_STDOUT=${RUSTFS_OBS_USE_STDOUT:-false}
-      - RUSTFS_OBS_LOG_STDOUT_ENABLED=${RUSTFS_OBS_LOG_STDOUT_ENABLED:-false}
-      - RUSTFS_ISSUE3031_DIAG_ENABLE=${RUSTFS_ISSUE3031_DIAG_ENABLE:-false}
-      - RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT:-5}
-      - RUSTFS_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_LOCK_ACQUIRE_TIMEOUT:-5}
-      - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
-      # Internode gRPC optimization knobs (grpc-optimization P0-P3). Defaults match the binary
-      # defaults, so leaving these unset is a no-op; the A/B driver exports them to toggle a stage.
-      - RUSTFS_INTERNODE_RPC_TCP_NODELAY=${RUSTFS_INTERNODE_RPC_TCP_NODELAY:-true}
-      - RUSTFS_INTERNODE_RPC_HTTP2_STREAM_WINDOW_SIZE=${RUSTFS_INTERNODE_RPC_HTTP2_STREAM_WINDOW_SIZE:-1048576}
-      - RUSTFS_INTERNODE_RPC_HTTP2_CONN_WINDOW_SIZE=${RUSTFS_INTERNODE_RPC_HTTP2_CONN_WINDOW_SIZE:-2097152}
-      - RUSTFS_INTERNODE_RPC_MAX_MESSAGE_SIZE=${RUSTFS_INTERNODE_RPC_MAX_MESSAGE_SIZE:-104857600}
-      - RUSTFS_INTERNODE_CHANNEL_ISOLATION=${RUSTFS_INTERNODE_CHANNEL_ISOLATION:-false}
-      - RUSTFS_INTERNODE_BULK_CHANNELS=${RUSTFS_INTERNODE_BULK_CHANNELS:-2}
-      - RUSTFS_INTERNODE_RPC_MSGPACK_ONLY=${RUSTFS_INTERNODE_RPC_MSGPACK_ONLY:-false}
-      - RUSTFS_INTERNODE_PREWARM=${RUSTFS_INTERNODE_PREWARM:-false}
-      - RUSTFS_INTERNODE_OFFLINE_BYPASS=${RUSTFS_INTERNODE_OFFLINE_BYPASS:-false}
-      - RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS=${RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS:-5}
-      - RUSTFS_INTERNODE_OFFLINE_FAILURE_THRESHOLD=${RUSTFS_INTERNODE_OFFLINE_FAILURE_THRESHOLD:-3}
-      - RUSTFS_INTERNODE_IDEMPOTENT_READ_RETRIES=${RUSTFS_INTERNODE_IDEMPOTENT_READ_RETRIES:-1}
-    extra_hosts:
-      - "host.docker.internal:host-gateway"
-    volumes:
-      - node4_data_0:/data/rustfs0
-      - node4_data_1:/data/rustfs1
-      - node4_data_2:/data/rustfs2
-      - node4_data_3:/data/rustfs3
-    ports:
-      - "9003:9000"
-    networks:
-      - rustfs-cluster-net
-
-volumes:
-  node1_data_0:
-  node1_data_1:
-  node1_data_2:
-  node1_data_3:
-  node2_data_0:
-  node2_data_1:
-  node2_data_2:
-  node2_data_3:
-  node3_data_0:
-  node3_data_1:
-  node3_data_2:
-  node3_data_3:
-  node4_data_0:
-  node4_data_1:
-  node4_data_2:
-  node4_data_3:
-
-networks:
-  rustfs-cluster-net:
-    driver: bridge

.docker/compose/docker-compose.cluster.perf-round1-linux-32c64g-latency-stable.override.yml

@@ -1,56 +0,0 @@
-services:
-  node1:
-    environment:
-      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=56
-      - RUSTFS_OBJECT_IO_BUFFER_SIZE=131072
-      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=10
-      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=20
-      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=10
-      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
-      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=25165824
-      - RUSTFS_RUNTIME_WORKER_THREADS=16
-      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=768
-      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=420
-      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=50
-
-  node2:
-    environment:
-      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=56
-      - RUSTFS_OBJECT_IO_BUFFER_SIZE=131072
-      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=10
-      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=20
-      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=10
-      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
-      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=25165824
-      - RUSTFS_RUNTIME_WORKER_THREADS=16
-      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=768
-      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=420
-      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=50
-
-  node3:
-    environment:
-      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=56
-      - RUSTFS_OBJECT_IO_BUFFER_SIZE=131072
-      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=10
-      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=20
-      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=10
-      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
-      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=25165824
-      - RUSTFS_RUNTIME_WORKER_THREADS=16
-      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=768
-      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=420
-      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=50
-
-  node4:
-    environment:
-      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=56
-      - RUSTFS_OBJECT_IO_BUFFER_SIZE=131072
-      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=10
-      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=20
-      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=10
-      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
-      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=25165824
-      - RUSTFS_RUNTIME_WORKER_THREADS=16
-      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=768
-      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=420
-      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=50

.docker/compose/docker-compose.cluster.perf-round1-linux-32c64g-throughput-max.override.yml

@@ -1,56 +0,0 @@
-services:
-  node1:
-    environment:
-      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=96
-      - RUSTFS_OBJECT_IO_BUFFER_SIZE=524288
-      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=16
-      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=32
-      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=24
-      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=33554432
-      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=50331648
-      - RUSTFS_RUNTIME_WORKER_THREADS=24
-      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=1536
-      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=600
-      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=80
-
-  node2:
-    environment:
-      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=96
-      - RUSTFS_OBJECT_IO_BUFFER_SIZE=524288
-      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=16
-      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=32
-      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=24
-      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=33554432
-      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=50331648
-      - RUSTFS_RUNTIME_WORKER_THREADS=24
-      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=1536
-      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=600
-      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=80
-
-  node3:
-    environment:
-      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=96
-      - RUSTFS_OBJECT_IO_BUFFER_SIZE=524288
-      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=16
-      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=32
-      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=24
-      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=33554432
-      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=50331648
-      - RUSTFS_RUNTIME_WORKER_THREADS=24
-      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=1536
-      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=600
-      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=80
-
-  node4:
-    environment:
-      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=96
-      - RUSTFS_OBJECT_IO_BUFFER_SIZE=524288
-      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=16
-      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=32
-      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=24
-      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=33554432
-      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=50331648
-      - RUSTFS_RUNTIME_WORKER_THREADS=24
-      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=1536
-      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=600
-      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=80

.docker/compose/docker-compose.cluster.perf-round1-linux-32c64g.override.yml

@@ -1,56 +0,0 @@
-services:
-  node1:
-    environment:
-      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=64
-      - RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
-      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=12
-      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=24
-      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=16
-      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=16777216
-      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=33554432
-      - RUSTFS_RUNTIME_WORKER_THREADS=20
-      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=1024
-      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
-      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=60
-
-  node2:
-    environment:
-      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=64
-      - RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
-      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=12
-      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=24
-      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=16
-      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=16777216
-      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=33554432
-      - RUSTFS_RUNTIME_WORKER_THREADS=20
-      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=1024
-      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
-      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=60
-
-  node3:
-    environment:
-      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=64
-      - RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
-      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=12
-      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=24
-      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=16
-      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=16777216
-      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=33554432
-      - RUSTFS_RUNTIME_WORKER_THREADS=20
-      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=1024
-      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
-      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=60
-
-  node4:
-    environment:
-      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=64
-      - RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
-      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=12
-      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=24
-      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=16
-      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=16777216
-      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=33554432
-      - RUSTFS_RUNTIME_WORKER_THREADS=20
-      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=1024
-      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
-      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=60

.docker/compose/docker-compose.cluster.perf-round1-linux.override.yml

@@ -1,56 +0,0 @@
-services:
-  node1:
-    environment:
-      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=48
-      - RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
-      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=8
-      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=16
-      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=12
-      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
-      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=25165824
-      - RUSTFS_RUNTIME_WORKER_THREADS=12
-      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=512
-      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
-      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=40
-
-  node2:
-    environment:
-      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=48
-      - RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
-      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=8
-      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=16
-      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=12
-      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
-      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=25165824
-      - RUSTFS_RUNTIME_WORKER_THREADS=12
-      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=512
-      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
-      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=40
-
-  node3:
-    environment:
-      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=48
-      - RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
-      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=8
-      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=16
-      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=12
-      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
-      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=25165824
-      - RUSTFS_RUNTIME_WORKER_THREADS=12
-      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=512
-      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
-      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=40
-
-  node4:
-    environment:
-      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=48
-      - RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
-      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=8
-      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=16
-      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=12
-      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
-      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=25165824
-      - RUSTFS_RUNTIME_WORKER_THREADS=12
-      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=512
-      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
-      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=40

.docker/compose/docker-compose.cluster.perf-round1.override.yml

@@ -1,56 +0,0 @@
-services:
-  node1:
-    environment:
-      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=32
-      - RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
-      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=6
-      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=12
-      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=8
-      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
-      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=16777216
-      - RUSTFS_RUNTIME_WORKER_THREADS=6
-      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=256
-      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
-      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=30
-
-  node2:
-    environment:
-      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=32
-      - RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
-      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=6
-      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=12
-      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=8
-      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
-      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=16777216
-      - RUSTFS_RUNTIME_WORKER_THREADS=6
-      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=256
-      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
-      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=30
-
-  node3:
-    environment:
-      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=32
-      - RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
-      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=6
-      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=12
-      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=8
-      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
-      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=16777216
-      - RUSTFS_RUNTIME_WORKER_THREADS=6
-      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=256
-      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
-      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=30
-
-  node4:
-    environment:
-      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=32
-      - RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
-      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=6
-      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=12
-      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=8
-      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
-      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=16777216
-      - RUSTFS_RUNTIME_WORKER_THREADS=6
-      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=256
-      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
-      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=30

.docker/compose/docker-compose.cluster.yaml

@@ -1,82 +0,0 @@
-# Copyright 2024 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-services:
-  node0:
-    image: rustfs/rustfs:latest # Replace with your image name and label
-    container_name: node0
-    hostname: node0
-    environment:
-      - RUSTFS_VOLUMES=http://node{0...3}:9000/data/rustfs{0...3}
-      - RUSTFS_ADDRESS=0.0.0.0:9000
-      - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin-local}
-      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfssecret-local}
-    platform: linux/amd64
-    ports:
-      - "9000:9000" # Map port 9001 of the host to port 9000 of the container
-    volumes:
-      - ../../target/x86_64-unknown-linux-gnu/release/rustfs:/app/rustfs
-    command: "/app/rustfs"
-
-  node1:
-    image: rustfs/rustfs:latest
-    container_name: node1
-    hostname: node1
-    environment:
-      - RUSTFS_VOLUMES=http://node{0...3}:9000/data/rustfs{0...3}
-      - RUSTFS_ADDRESS=0.0.0.0:9000
-      - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin-local}
-      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfssecret-local}
-    platform: linux/amd64
-    ports:
-      - "9001:9000" # Map port 9002 of the host to port 9000 of the container
-    volumes:
-      - ../../target/x86_64-unknown-linux-gnu/release/rustfs:/app/rustfs
-    command: "/app/rustfs"
-
-  node2:
-    image: rustfs/rustfs:latest
-    container_name: node2
-    hostname: node2
-    environment:
-      - RUSTFS_VOLUMES=http://node{0...3}:9000/data/rustfs{0...3}
-      - RUSTFS_ADDRESS=0.0.0.0:9000
-      - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin-local}
-      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfssecret-local}
-    platform: linux/amd64
-    ports:
-      - "9002:9000" # Map port 9003 of the host to port 9000 of the container
-    volumes:
-      - ../../target/x86_64-unknown-linux-gnu/release/rustfs:/app/rustfs
-    command: "/app/rustfs"
-
-  node3:
-    image: rustfs/rustfs:latest
-    container_name: node3
-    hostname: node3
-    environment:
-      - RUSTFS_VOLUMES=http://node{0...3}:9000/data/rustfs{0...3}
-      - RUSTFS_ADDRESS=0.0.0.0:9000
-      - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin-local}
-      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfssecret-local}
-    platform: linux/amd64
-    ports:
-      - "9003:9000" # Map port 9004 of the host to port 9000 of the container
-    volumes:
-      - ../../target/x86_64-unknown-linux-gnu/release/rustfs:/app/rustfs
-    command: "/app/rustfs"

.docker/compose/docker-compose.observability.yaml

@@ -1,286 +0,0 @@
-# Copyright 2024 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-services:
-  # --- Observability Stack ---
-
-  tempo-init:
-    image: busybox:latest
-    command: [ "sh", "-c", "chown -R 10001:10001 /var/tempo" ]
-    volumes:
-      - tempo-data:/var/tempo
-    user: root
-    networks:
-      - rustfs-network
-    restart: "no"
-
-  tempo:
-    image: grafana/tempo:2.10.5
-    user: "10001"
-    command: [ "-config.file=/etc/tempo.yaml" ]
-    volumes:
-      - ../../.docker/observability/tempo.yaml:/etc/tempo.yaml:ro
-      - tempo-data:/var/tempo
-    ports:
-      - "3200:3200" # tempo
-      - "4317"      # otlp grpc
-      - "4318"      # otlp http
-      - "7946"      # memberlist
-    restart: unless-stopped
-    networks:
-      - rustfs-network
-    depends_on:
-      tempo-init:
-        condition: service_completed_successfully
-    healthcheck:
-      test: [ "CMD", "/tempo", "-version" ]
-      interval: 10s
-      timeout: 5s
-      retries: 3
-      start_period: 15s
-
-  otel-collector:
-    image: otel/opentelemetry-collector-contrib:latest
-    environment:
-      - TZ=Asia/Shanghai
-    volumes:
-      - ../../.docker/observability/otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml:ro
-    ports:
-      - "1888:1888"   # pprof
-      - "8888:8888"   # Prometheus metrics for Collector
-      - "8889:8889"   # Prometheus metrics for application indicators
-      - "13133:13133" # health check
-      - "4317:4317"   # OTLP gRPC
-      - "4318:4318"   # OTLP HTTP
-      - "55679:55679" # zpages
-    networks:
-      - rustfs-network
-    depends_on:
-      - tempo
-      - jaeger
-      - prometheus
-      - loki
-    restart: unless-stopped
-    healthcheck:
-      test: [ "CMD", "/otelcol-contrib", "--version" ]
-      interval: 10s
-      timeout: 5s
-      retries: 3
-
-  jaeger:
-    image: jaegertracing/jaeger:latest
-    environment:
-      - TZ=Asia/Shanghai
-      - SPAN_STORAGE_TYPE=badger
-      - BADGER_EPHEMERAL=false
-      - BADGER_DIRECTORY_VALUE=/badger/data
-      - BADGER_DIRECTORY_KEY=/badger/key
-      - COLLECTOR_OTLP_ENABLED=true
-    volumes:
-      - ../../.docker/observability/jaeger.yaml:/etc/jaeger/config.yml:ro
-      - jaeger-data:/badger
-    ports:
-      - "16686:16686" # Web UI
-      - "14269:14269" # Admin/Metrics
-      - "4317"        # otlp grpc
-      - "4318"        # otlp http
-    command: [ "--config", "/etc/jaeger/config.yml" ]
-    networks:
-      - rustfs-network
-    restart: unless-stopped
-    healthcheck:
-      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:14269" ]
-      interval: 10s
-      timeout: 5s
-      retries: 3
-      start_period: 15s
-
-  prometheus:
-    image: prom/prometheus:latest
-    environment:
-      - TZ=Asia/Shanghai
-    volumes:
-      - ../../.docker/observability/prometheus.yml:/etc/prometheus/prometheus.yml:ro
-      - ../../.docker/observability/prometheus-rules:/etc/prometheus/rules:ro
-      - prometheus-data:/prometheus
-    ports:
-      - "9090:9090"
-    command:
-      - '--config.file=/etc/prometheus/prometheus.yml'
-      - '--web.enable-otlp-receiver'
-      - '--web.enable-remote-write-receiver'
-      - '--enable-feature=promql-experimental-functions'
-      - '--storage.tsdb.path=/prometheus'
-      - '--storage.tsdb.retention.time=30d'
-    networks:
-      - rustfs-network
-    restart: unless-stopped
-    healthcheck:
-      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:9090/-/healthy" ]
-      interval: 10s
-      timeout: 5s
-      retries: 3
-
-  loki:
-    image: grafana/loki:latest
-    environment:
-      - TZ=Asia/Shanghai
-    volumes:
-      - ../../.docker/observability/loki.yaml:/etc/loki/loki.yaml:ro
-      - loki-data:/loki
-    ports:
-      - "3100:3100"
-    command: -config.file=/etc/loki/loki.yaml
-    networks:
-      - rustfs-network
-    restart: unless-stopped
-    healthcheck:
-      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:3100/ready" ]
-      interval: 15s
-      timeout: 10s
-      retries: 5
-      start_period: 60s
-
-  pyroscope:
-    image: grafana/pyroscope:latest
-    ports:
-      - "4040:4040"
-    command:
-      - -self-profiling.disable-push=true
-    networks:
-      - rustfs-network
-    restart: unless-stopped
-
-  grafana:
-    image: grafana/grafana:latest
-    ports:
-      - "3000:3000" # Web UI
-    environment:
-      - GF_SECURITY_ADMIN_PASSWORD=admin
-      - GF_SECURITY_ADMIN_USER=admin
-      - TZ=Asia/Shanghai
-      - GF_INSTALL_PLUGINS=grafana-pyroscope-datasource
-      - GF_DASHBOARDS_DEFAULT_HOME_DASHBOARD_PATH=/var/lib/grafana/dashboards/home.json
-    networks:
-      - rustfs-network
-    volumes:
-      - ../../.docker/observability/grafana/provisioning:/etc/grafana/provisioning:ro
-      - ../../.docker/observability/grafana/dashboards:/var/lib/grafana/dashboards:ro
-      - grafana-data:/var/lib/grafana
-    depends_on:
-      - prometheus
-      - tempo
-      - loki
-    restart: unless-stopped
-    healthcheck:
-      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:3000/api/health" ]
-      interval: 10s
-      timeout: 5s
-      retries: 3
-
-  # --- RustFS Cluster ---
-
-  node1:
-    build:
-      context: ../..
-      dockerfile: Dockerfile.source
-    container_name: node1
-    environment:
-      - RUSTFS_VOLUMES=http://node{1...4}:9000/root/data/target/volume/test{1...4}
-      - RUSTFS_ADDRESS=:9000
-      - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_OBS_ENDPOINT=http://otel-collector:4318
-      - OTEL_RESOURCE_ATTRIBUTES=${OTEL_RESOURCE_ATTRIBUTES:-rustfs.cluster.id=rustfs-dev}
-      - RUSTFS_OBS_LOGGER_LEVEL=debug
-    platform: linux/amd64
-    ports:
-      - "9001:9000"
-    networks:
-      - rustfs-network
-    depends_on:
-      - otel-collector
-
-  node2:
-    build:
-      context: ../..
-      dockerfile: Dockerfile.source
-    container_name: node2
-    environment:
-      - RUSTFS_VOLUMES=http://node{1...4}:9000/root/data/target/volume/test{1...4}
-      - RUSTFS_ADDRESS=:9000
-      - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_OBS_ENDPOINT=http://otel-collector:4318
-      - OTEL_RESOURCE_ATTRIBUTES=${OTEL_RESOURCE_ATTRIBUTES:-rustfs.cluster.id=rustfs-dev}
-      - RUSTFS_OBS_LOGGER_LEVEL=debug
-    platform: linux/amd64
-    ports:
-      - "9002:9000"
-    networks:
-      - rustfs-network
-    depends_on:
-      - otel-collector
-
-  node3:
-    build:
-      context: ../..
-      dockerfile: Dockerfile.source
-    container_name: node3
-    environment:
-      - RUSTFS_VOLUMES=http://node{1...4}:9000/root/data/target/volume/test{1...4}
-      - RUSTFS_ADDRESS=:9000
-      - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_OBS_ENDPOINT=http://otel-collector:4318
-      - OTEL_RESOURCE_ATTRIBUTES=${OTEL_RESOURCE_ATTRIBUTES:-rustfs.cluster.id=rustfs-dev}
-      - RUSTFS_OBS_LOGGER_LEVEL=debug
-    platform: linux/amd64
-    ports:
-      - "9003:9000"
-    networks:
-      - rustfs-network
-    depends_on:
-      - otel-collector
-
-  node4:
-    build:
-      context: ../..
-      dockerfile: Dockerfile.source
-    container_name: node4
-    environment:
-      - RUSTFS_VOLUMES=http://node{1...4}:9000/root/data/target/volume/test{1...4}
-      - RUSTFS_ADDRESS=:9000
-      - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_OBS_ENDPOINT=http://otel-collector:4318
-      - OTEL_RESOURCE_ATTRIBUTES=${OTEL_RESOURCE_ATTRIBUTES:-rustfs.cluster.id=rustfs-dev}
-      - RUSTFS_OBS_LOGGER_LEVEL=debug
-    platform: linux/amd64
-    ports:
-      - "9004:9000"
-    networks:
-      - rustfs-network
-    depends_on:
-      - otel-collector
-
-volumes:
-  prometheus-data:
-  tempo-data:
-  loki-data:
-  jaeger-data:
-  grafana-data:
-
-networks:
-  rustfs-network:
-    driver: bridge
-    name: "network_rustfs_config"
-    driver_opts:
-      com.docker.network.enable_ipv6: "true"

.docker/mqtt/README.md

@@ -1,30 +0,0 @@
-# MQTT Broker (EMQX)
-
-This directory contains the configuration for running an EMQX MQTT broker, which can be used for testing RustFS's MQTT integration.
-
-## 🚀 Quick Start
-
-To start the EMQX broker:
-
-```bash
-docker compose up -d
-```
-
-## 📊 Access
-
-- **Dashboard**: [http://localhost:18083](http://localhost:18083)
-- **Default Credentials**: `admin` / `public`
-- **MQTT Port**: `1883`
-- **WebSocket Port**: `8083`
-
-## 🛠️ Configuration
-
-The `docker-compose.yml` file sets up a single-node EMQX instance.
-
-- **Persistence**: Data is not persisted by default (for testing).
-- **Network**: Uses the default bridge network.
-
-## 📝 Notes
-
-- This setup is intended for development and testing purposes.
-- For production deployments, please refer to the official [EMQX Documentation](https://www.emqx.io/docs/en/latest/).

.docker/mqtt/config/emqx.conf

@@ -1,37 +0,0 @@
-# 节点配置
-node.name = "emqx@127.0.0.1"
-node.cookie = "aBcDeFgHiJkLmNoPqRsTuVwXyZ012345"
-node.data_dir = "/opt/emqx/data"
-
-# 日志配置
-log.console = {level = info, enable = true}
-log.file = {path = "/opt/emqx/log/emqx.log", enable = true, level = info}
-
-# MQTT TCP 监听器
-listeners.tcp.default = {bind = "0.0.0.0:1883", max_connections = 1000000, enable = true}
-
-# MQTT SSL 监听器
-listeners.ssl.default = {bind = "0.0.0.0:8883", enable = false}
-
-# MQTT WebSocket 监听器
-listeners.ws.default = {bind = "0.0.0.0:8083", enable = true}
-
-# MQTT WebSocket SSL 监听器
-listeners.wss.default = {bind = "0.0.0.0:8084", enable = false}
-
-# 管理控制台
-dashboard.listeners.http = {bind = "0.0.0.0:18083", enable = true}
-
-# HTTP API
-management.listeners.http = {bind = "0.0.0.0:8081", enable = true}
-
-# 认证配置
-authentication = [
-  {enable = true, mechanism = password_based, backend = built_in_database, user_id_type = username}
-]
-
-# 授权配置
-authorization.sources = [{type = built_in_database, enable = true}]
-
-# 持久化消息存储
-message.storage.backend = built_in_database

.docker/mqtt/config/vm.args

@@ -1,9 +0,0 @@
--name emqx@127.0.0.1
--setcookie aBcDeFgHiJkLmNoPqRsTuVwXyZ012345
-+P 2097152
-+t 1048576
-+zdbbl 32768
--kernel inet_dist_listen_min 6000
--kernel inet_dist_listen_max 6100
--smp enable
--mnesia dir "/opt/emqx/data/mnesia"

.docker/mqtt/docker-compose-more.yml

@@ -1,74 +0,0 @@
-# Copyright 2024 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-services:
-  emqx:
-    image: emqx/emqx:latest
-    container_name: emqx
-    restart: unless-stopped
-    environment:
-      - EMQX_NODE__NAME=emqx@127.0.0.1
-      - EMQX_NODE__COOKIE=aBcDeFgHiJkLmNoPqRsTuVwXyZ012345
-      - EMQX_NODE__DATA_DIR=/opt/emqx/data
-      - EMQX_LOG__CONSOLE__LEVEL=info
-      - EMQX_LOG__CONSOLE__ENABLE=true
-      - EMQX_LOG__FILE__PATH=/opt/emqx/log/emqx.log
-      - EMQX_LOG__FILE__LEVEL=info
-      - EMQX_LOG__FILE__ENABLE=true
-      - EMQX_LISTENERS__TCP__DEFAULT__BIND=0.0.0.0:1883
-      - EMQX_LISTENERS__TCP__DEFAULT__MAX_CONNECTIONS=1000000
-      - EMQX_LISTENERS__TCP__DEFAULT__ENABLE=true
-      - EMQX_LISTENERS__SSL__DEFAULT__BIND=0.0.0.0:8883
-      - EMQX_LISTENERS__SSL__DEFAULT__ENABLE=false
-      - EMQX_LISTENERS__WS__DEFAULT__BIND=0.0.0.0:8083
-      - EMQX_LISTENERS__WS__DEFAULT__ENABLE=true
-      - EMQX_LISTENERS__WSS__DEFAULT__BIND=0.0.0.0:8084
-      - EMQX_LISTENERS__WSS__DEFAULT__ENABLE=false
-      - EMQX_DASHBOARD__LISTENERS__HTTP__BIND=0.0.0.0:18083
-      - EMQX_DASHBOARD__LISTENERS__HTTP__ENABLE=true
-      - EMQX_MANAGEMENT__LISTENERS__HTTP__BIND=0.0.0.0:8081
-      - EMQX_MANAGEMENT__LISTENERS__HTTP__ENABLE=true
-      - EMQX_AUTHENTICATION__1__ENABLE=true
-      - EMQX_AUTHENTICATION__1__MECHANISM=password_based
-      - EMQX_AUTHENTICATION__1__BACKEND=built_in_database
-      - EMQX_AUTHENTICATION__1__USER_ID_TYPE=username
-      - EMQX_AUTHORIZATION__SOURCES__1__TYPE=built_in_database
-      - EMQX_AUTHORIZATION__SOURCES__1__ENABLE=true
-    ports:
-      - "1883:1883"    # MQTT TCP
-      - "8883:8883"    # MQTT SSL
-      - "8083:8083"    # MQTT WebSocket
-      - "8084:8084"    # MQTT WebSocket SSL
-      - "18083:18083"  # Web 管理控制台
-      - "8081:8081"    # HTTP API
-    volumes:
-      - ./data:/opt/emqx/data
-      - ./log:/opt/emqx/log
-      - ./config:/opt/emqx/etc
-    networks:
-      - mqtt-net
-    healthcheck:
-      test: [ "CMD", "/opt/emqx/bin/emqx_ctl", "status" ]
-      interval: 30s
-      timeout: 10s
-      retries: 3
-    logging:
-      driver: "json-file"
-      options:
-        max-size: "100m"
-        max-file: "3"
-
-networks:
-  mqtt-net:
-    driver: bridge

.docker/mqtt/docker-compose.yml

@@ -1,29 +0,0 @@
-# Copyright 2024 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-services:
-  emqx:
-    image: emqx/emqx:latest
-    container_name: emqx
-    ports:
-      - "1883:1883"
-      - "8083:8083"
-      - "8084:8084"
-      - "8883:8883"
-      - "18083:18083"
-    restart: unless-stopped
-
-networks:
-  default:
-    driver: bridge

.docker/nginx/nginx.conf

@@ -1,93 +0,0 @@
-# Copyright 2024 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-worker_processes auto;
-pid /var/run/nginx.pid;
-
-events {
-    worker_connections 1024;
-}
-
-http {
-    include /etc/nginx/mime.types;
-    default_type application/octet-stream;
-
-    log_format main '$remote_addr - $remote_user [$time_local] "$request" '
-                    '$status $body_bytes_sent "$http_referer" '
-                    '"$http_user_agent" "$http_x_forwarded_for"';
-
-    access_log /var/log/nginx/access.log main;
-    error_log /var/log/nginx/error.log warn;
-
-    sendfile on;
-    keepalive_timeout 65;
-
-    # RustFS Server Block
-    server {
-        listen 80;
-        server_name localhost;
-
-        # Redirect HTTP to HTTPS (optional, uncomment if SSL is configured)
-        # return 301 https://$host$request_uri;
-
-        location / {
-            proxy_pass http://rustfs:9000;
-            proxy_set_header Host $host;
-            proxy_set_header X-Real-IP $remote_addr;
-            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
-            proxy_set_header X-Forwarded-Proto $scheme;
-
-            # S3 specific headers
-            proxy_set_header X-Amz-Date $http_x_amz_date;
-            proxy_set_header Authorization $http_authorization;
-
-            # Disable buffering for large uploads
-            proxy_request_buffering off;
-            client_max_body_size 0;
-        }
-
-        location /rustfs/console {
-            proxy_pass http://rustfs:9001;
-            proxy_set_header Host $host;
-            proxy_set_header X-Real-IP $remote_addr;
-            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
-            proxy_set_header X-Forwarded-Proto $scheme;
-        }
-    }
-
-    # SSL Configuration (Example)
-    # server {
-    #     listen 443 ssl;
-    #     server_name localhost;
-    #
-    #     ssl_certificate /etc/nginx/ssl/server.crt;
-    #     ssl_certificate_key /etc/nginx/ssl/server.key;
-    #
-    #     # Restrict to modern TLS versions and ciphers. Operators copying this
-    #     # example must keep at least these directives — without them, nginx
-    #     # may negotiate older protocol versions that have known weaknesses.
-    #     ssl_protocols TLSv1.2 TLSv1.3;
-    #     ssl_prefer_server_ciphers on;
-    #     ssl_ciphers 'ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305';
-    #     ssl_session_timeout 1d;
-    #     ssl_session_cache shared:SSL:10m;
-    #     ssl_session_tickets off;
-    #     # add_header Strict-Transport-Security "max-age=63072000" always;
-    #
-    #     location / {
-    #         proxy_pass http://rustfs:9000;
-    #         ...
-    #     }
-    # }
-}

.docker/observability/.gitignore

@@ -1,5 +0,0 @@
-jaeger-data/*
-loki-data/*
-prometheus-data/*
-tempo-data/*
-grafana-data/*

.docker/observability/README.md

@@ -1,225 +0,0 @@
-# RustFS Observability Stack
-
-This directory contains the comprehensive observability stack for RustFS, designed to provide deep insights into application performance, logs, and traces.
-
-## Components
-
-The stack is composed of the following best-in-class open-source components:
-
-- **Prometheus** (v2.53.1): The industry standard for metric collection and alerting.
-- **Grafana** (v11.1.0): The leading platform for observability visualization.
-- **Loki** (v3.1.0): A horizontally-scalable, highly-available, multi-tenant log aggregation system.
-- **Tempo** (v2.5.0): A high-volume, minimal dependency distributed tracing backend.
-- **Jaeger** (v1.59.0): Distributed tracing system (configured as a secondary UI/storage).
-- **OpenTelemetry Collector** (v0.104.0): A vendor-agnostic implementation for receiving, processing, and exporting telemetry data.
-
-By default, this stack uses Tempo in single-binary mode and does not require Kafka/Redpanda.
-If you want the Kafka-backed HA Tempo path, use `docker-compose-example-for-rustfs.yml` together with `docker-compose-tempo-ha-override.yml`.
-
-## Architecture
-
-1. **Telemetry Collection**: Applications send OTLP (OpenTelemetry Protocol) data (Metrics, Logs, Traces) to the **OpenTelemetry Collector**.
-2. **Processing & Exporting**: The Collector processes the data (batching, memory limiting) and exports it to the respective backends:
-    - **Traces** -> **Tempo** (Primary) & **Jaeger** (Secondary/Optional)
-    - **Metrics** -> **Prometheus** (via scraping the Collector's exporter)
-    - **Logs** -> **Loki**
-3. **Visualization**: **Grafana** connects to all backends (Prometheus, Tempo, Loki, Jaeger) to provide a unified dashboard experience.
-
-## Features
-
-- **Full Persistence**: All data (Metrics, Logs, Traces) is persisted to Docker volumes, ensuring no data loss on restart.
-- **Correlation**: Seamless navigation between Metrics, Logs, and Traces in Grafana.
-  - Jump from a Metric spike to relevant Traces.
-  - Jump from a Trace to relevant Logs.
-- **High Performance**: Optimized configurations for batching, compression, and memory management.
-- **Standardized Protocols**: Built entirely on OpenTelemetry standards.
-
-## GET Performance Optimization Dashboards
-
-Three pre-built Grafana dashboards are included for monitoring RustFS GET performance optimization rollout:
-
-### Available Dashboards
-
-| Dashboard | File | Description |
-|-----------|------|-------------|
-| **GET Rollout Health** | `grafana-get-rollout-health.json` | Monitors optimization rollout: latency by reader path, early-stop hit rate, codec streaming usage, pipeline failures |
-| **GET Data Integrity** | `grafana-get-data-integrity.json` | Monitors data safety: bitrot verify failures, decode errors, short reads, shard read outcomes |
-| **GET Resource Impact** | `grafana-get-resource-impact.json` | Monitors resource usage: concurrent requests, IO queue utilization, disk permit wait, RSS trend |
-| **Object Data Cache** | `grafana-object-data-cache.json` | Monitors the GET body cache (`rustfs_object_data_cache_*`): hit ratio, lookup/plan/fill outcomes, fill duration quantiles, hit vs fill throughput, entries/weighted bytes, inflight fills, memory-pressure skips, invalidations, and size-class breakdowns |
-
-### Storage Metrics
-
-Storage panels require `prometheus-rules/rustfs-storage.yml` and the cluster resource attribute. See the [storage metrics guide](../../docs/operations/storage-metrics.md) for ownership, observer selection, freshness, and rolling upgrades.
-
-### Prometheus Alert Rules
-
-The file `prometheus-rules/rustfs-get-optimization-alerts.yaml` contains pre-configured alerting rules:
-
-| Alert | Severity | Condition |
-|-------|----------|-----------|
-| `GetP99Regression` | Critical | GET p99 latency > 2x baseline for 10m |
-| `PipelineFailureSpike` | Critical | Pipeline failure rate > 5x baseline for 5m |
-| `BitrotMismatchSpike` | Critical | Bitrot mismatch rate > 3x baseline for 5m |
-| `EarlyStopInsufficientQuorum` | Warning | Early-stop insufficient quorum rate > 0.1/s for 5m |
-| `CodecStreamingFallbackSpike` | Warning | Codec streaming fallback > 10x baseline for 10m |
-| `IoQueueSaturation` | Warning | IO queue utilization > 90% for 5m |
-
-The file `prometheus-rules/rustfs-kms-alerts.yml` contains alerting rules for the KMS backend operation metrics. Thresholds are conservative defaults pending staging baseline calibration; response procedures live in `docs/operations/kms-observability-runbook.md`, and the matching dashboard is `deploy/observability/grafana/rustfs-kms-observability.json`.
-
-| Alert | Severity | Condition |
-|-------|----------|-----------|
-| `KmsBackendFatalErrors` | Critical | Fatal (non-retryable) attempt failures > 0 for 5m |
-| `KmsBackendHighErrorRate` | Critical | Non-success operation ratio > 5% for 10m (with traffic guard) |
-| `KmsBackendP99LatencyHigh` | Warning | Operation p99 duration (incl. retries) > 2s for 10m |
-| `KmsBackendAttemptFailureSpike` | Warning | Attempt failure rate > 0.5/s for 10m |
-| `KmsBackendRetryBudgetExhausted` | Warning | budget_exhausted / deadline_exceeded outcomes > 0.05/s for 10m |
-
-### Enabling Alert Rules
-
-Add the alert rules file to your Prometheus configuration:
-
-```yaml
-# prometheus.yml
-rule_files:
-  - "/etc/prometheus/rules/*.yml"
-
-# Or mount the file in docker-compose.yml:
-# volumes:
-#   - ./prometheus-rules:/etc/prometheus/rules
-```
-
-### Dashboard Usage
-
-The dashboards are automatically provisioned when Grafana starts. They use the `${DS_PROMETHEUS}` datasource variable, so you need a Prometheus datasource configured in Grafana.
-
-Key panels to monitor during optimization rollout:
-
-1. **GET Latency by Reader Path** - Compare `codec_streaming` vs `legacy_duplex` latency
-2. **Early-Stop Hit Rate** - Verify early-stop is triggering effectively
-3. **Pipeline Failure Rate** - Detect any new failure modes introduced by optimizations
-4. **Bitrot Verify Failures** - Ensure data integrity is maintained
-
-## Quick Start
-
-### Prerequisites
-
-- Docker
-- Docker Compose
-
-### Deploy
-
-Run the following command to start the entire stack:
-
-```bash
-docker compose up -d
-```
-
-### High Availability Tempo
-
-The default `docker-compose.yml` is the single-node stack.
-If you need the Kafka-backed HA Tempo configuration, start it with:
-
-```bash
-docker compose -f docker-compose-example-for-rustfs.yml -f docker-compose-tempo-ha-override.yml up -d
-```
-
-### Access Dashboards
-
-| Service        | URL                                              | Credentials       | Description                    |
-| :------------- | :----------------------------------------------- | :---------------- | :----------------------------- |
-| **Grafana**    | [http://localhost:3000](http://localhost:3000)   | `admin` / `admin` | Main visualization hub.        |
-| **Prometheus** | [http://localhost:9090](http://localhost:9090)   | -                 | Metric queries and status.     |
-| **Jaeger UI**  | [http://localhost:16686](http://localhost:16686) | -                 | Secondary trace visualization. |
-| **Tempo**      | [http://localhost:3200](http://localhost:3200)   | -                 | Tempo status/metrics.          |
-
-## Configuration
-
-### Data Persistence
-
-Data is stored in the following Docker volumes:
-
-- `prometheus-data`: Prometheus metrics
-- `tempo-data`: Tempo traces (WAL and Blocks)
-- `loki-data`: Loki logs (Chunks and Rules)
-- `jaeger-data`: Jaeger traces (Badger DB)
-
-To clear all data:
-
-```bash
-docker compose down -v
-```
-
-### Customization
-
-- **Prometheus**: Edit `prometheus.yml` to add scrape targets or alerting rules.
-- **Grafana**: Dashboards and datasources are provisioned from the `grafana/` directory.
-- **Collector**: Edit `otel-collector-config.yaml` to modify pipelines, processors, or exporters.
-
-### Verifying RustFS Traces
-
-When RustFS points `RUSTFS_OBS_ENDPOINT` at this stack, treat the value as the
-OTLP/HTTP base URL, for example:
-
-```bash
-export RUSTFS_OBS_ENDPOINT=http://host.docker.internal:4318
-```
-
-RustFS automatically expands that base URL to:
-
-- `/v1/traces`
-- `/v1/metrics`
-- `/v1/logs`
-
-Important behavior notes:
-
-- Logs and metrics usually appear during startup, so seeing those two signals
-  first is expected.
-- The OpenTelemetry bridge sends `tracing` fields as log attributes. Loki stores
-  those attributes as structured metadata, and the Collector also mirrors the
-  common troubleshooting fields into the log line so simple line filters can
-  find them.
-- Visible trace data usually requires real HTTP/S3/gRPC request traffic after
-  startup, because request-path spans are created on demand.
-- `RUSTFS_OBS_LOGGER_LEVEL=info` keeps the top-level request span but filters
-  many nested `debug` spans. If Tempo or Jaeger looks sparse, retry with
-  `RUSTFS_OBS_LOGGER_LEVEL=debug` before suspecting collector or Tempo issues.
-
-Minimal validation flow:
-
-```bash
-# 1. Start this observability stack.
-docker compose up -d
-
-# 2. Start RustFS with OTLP/HTTP export and richer span visibility.
-export RUSTFS_OBS_ENDPOINT=http://host.docker.internal:4318
-export RUSTFS_OBS_LOGGER_LEVEL=debug
-
-# 3. Generate real request traffic.
-curl -I http://127.0.0.1:9000/health
-curl -I http://127.0.0.1:9000/health/ready
-
-# 4. Inspect Grafana or Jaeger.
-# Grafana: http://localhost:3000
-# Jaeger:  http://localhost:16686
-```
-
-For a structured RustFS log such as an inter-node RPC authentication failure,
-the Loki line now includes fields such as `event`, `component`, `subsystem`,
-`failure_reason`, `rpc_service`, `rpc_method`, and `expected_audience`. Useful
-LogQL checks:
-
-```logql
-{service_name="RustFS"} |= "RPC signature verification failed"
-{service_name="RustFS"} |= "failure_reason="
-{service_name="RustFS"} | failure_reason != ""
-```
-
-If logs and metrics are present but traces are sparse, the most common cause is
-"no real request traffic yet" or "`info` level filtered nested spans", not an
-OTLP routing failure.
-
-## Troubleshooting
-
-- **Service Health**: Check the health of services using `docker compose ps`.
-- **Logs**: View logs for a specific service using `docker compose logs -f <service_name>`.
-- **Otel Collector**: Check `http://localhost:13133` for health status and `http://localhost:1888/debug/pprof/` for profiling.

.docker/observability/README_ZH.md

@@ -1,211 +0,0 @@
-# RustFS 可观测性技术栈
-
-本目录包含 RustFS 的全面可观测性技术栈，旨在提供对应用程序性能、日志和追踪的深入洞察。
-
-## 组件
-
-该技术栈由以下一流的开源组件组成：
-
-- **Prometheus** (v2.53.1): 行业标准的指标收集和告警工具。
-- **Grafana** (v11.1.0): 领先的可观测性可视化平台。
-- **Loki** (v3.1.0): 水平可扩展、高可用、多租户的日志聚合系统。
-- **Tempo** (v2.5.0): 高吞吐量、最小依赖的分布式追踪后端。
-- **Jaeger** (v1.59.0): 分布式追踪系统（配置为辅助 UI/存储）。
-- **OpenTelemetry Collector** (v0.104.0): 接收、处理和导出遥测数据的供应商无关实现。
-
-默认情况下，这套技术栈使用 Tempo 单二进制模式，不依赖 Kafka/Redpanda。
-如果需要基于 Kafka 的 HA Tempo 路径，请使用 `docker-compose-example-for-rustfs.yml` 配合 `docker-compose-tempo-ha-override.yml`。
-
-## 架构
-
-1.  **遥测收集**: 应用程序将 OTLP (OpenTelemetry Protocol) 数据（指标、日志、追踪）发送到 **OpenTelemetry Collector**。
-2.  **处理与导出**: Collector 处理数据（批处理、内存限制）并将其导出到相应的后端：
-    -   **追踪** -> **Tempo** (主要) & **Jaeger** (辅助/可选)
-    -   **指标** -> **Prometheus** (通过抓取 Collector 的导出器)
-    -   **日志** -> **Loki**
-3.  **可视化**: **Grafana** 连接到所有后端（Prometheus, Tempo, Loki, Jaeger），提供统一的仪表盘体验。
-
-## 特性
-
--   **完全持久化**: 所有数据（指标、日志、追踪）都持久化到 Docker 卷，确保重启后无数据丢失。
--   **关联性**: 在 Grafana 中实现指标、日志和追踪之间的无缝导航。
-    -   从指标峰值跳转到相关追踪。
-    -   从追踪跳转到相关日志。
--   **高性能**: 针对批处理、压缩和内存管理进行了优化配置。
--   **标准化协议**: 完全基于 OpenTelemetry 标准构建。
-
-## GET 性能优化仪表盘
-
-包含三个预构建的 Grafana 仪表盘，用于监控 RustFS GET 性能优化发布：
-
-### 可用仪表盘
-
-| 仪表盘 | 文件 | 描述 |
-|--------|------|------|
-| **GET 发布健康度** | `grafana-get-rollout-health.json` | 监控优化发布：按 reader path 的延迟、early-stop 命中率、codec streaming 使用率、pipeline 失败率 |
-| **GET 数据完整性** | `grafana-get-data-integrity.json` | 监控数据安全：bitrot 校验失败、decode 错误、short read、shard 读取结果 |
-| **GET 资源影响** | `grafana-get-resource-impact.json` | 监控资源使用：并发请求数、IO 队列利用率、disk permit 等待、RSS 趋势 |
-| **对象数据缓存** | `grafana-object-data-cache.json` | 监控 GET body 缓存（`rustfs_object_data_cache_*`）：命中率、查找/规划/填充结果、填充耗时分位、命中 vs 填充吞吐、条目数/加权字节、在途填充、内存压力拒绝、失效、按尺寸档拆分 |
-
-### Prometheus 告警规则
-
-文件 `prometheus-rules/rustfs-get-optimization-alerts.yaml` 包含预配置的告警规则：
-
-| 告警 | 级别 | 条件 |
-|------|------|------|
-| `GetP99Regression` | 严重 | GET p99 延迟 > 2x 基线，持续 10 分钟 |
-| `PipelineFailureSpike` | 严重 | Pipeline 失败率 > 5x 基线，持续 5 分钟 |
-| `BitrotMismatchSpike` | 严重 | Bitrot 不匹配率 > 3x 基线，持续 5 分钟 |
-| `EarlyStopInsufficientQuorum` | 警告 | Early-stop quorum 不足率 > 0.1/s，持续 5 分钟 |
-| `CodecStreamingFallbackSpike` | 警告 | Codec streaming 回退 > 10x 基线，持续 10 分钟 |
-| `IoQueueSaturation` | 警告 | IO 队列利用率 > 90%，持续 5 分钟 |
-
-文件 `prometheus-rules/rustfs-kms-alerts.yml` 包含 KMS 后端操作指标的告警规则。阈值为保守默认值，待 staging 基线校准；响应流程见 `docs/operations/kms-observability-runbook.md`，配套仪表盘为 `deploy/observability/grafana/rustfs-kms-observability.json`。
-
-| 告警 | 级别 | 条件 |
-|------|------|------|
-| `KmsBackendFatalErrors` | 严重 | fatal（不可重试）尝试失败 > 0，持续 5 分钟 |
-| `KmsBackendHighErrorRate` | 严重 | 非 success 操作占比 > 5%，持续 10 分钟（含流量下限保护） |
-| `KmsBackendP99LatencyHigh` | 警告 | 操作 p99 耗时（含重试）> 2s，持续 10 分钟 |
-| `KmsBackendAttemptFailureSpike` | 警告 | 尝试失败率 > 0.5/s，持续 10 分钟 |
-| `KmsBackendRetryBudgetExhausted` | 警告 | budget_exhausted / deadline_exceeded 结果 > 0.05/s，持续 10 分钟 |
-
-### 启用告警规则
-
-在 Prometheus 配置中添加告警规则文件：
-
-```yaml
-# prometheus.yml
-rule_files:
-  - "/etc/prometheus/rules/*.yml"
-
-# 或在 docker-compose.yml 中挂载文件：
-# volumes:
-#   - ./prometheus-rules:/etc/prometheus/rules
-```
-
-### 仪表盘使用
-
-仪表盘在 Grafana 启动时自动预置。它们使用 `${DS_PROMETHEUS}` 数据源变量，因此需要在 Grafana 中配置 Prometheus 数据源。
-
-优化发布期间需要关注的关键面板：
-
-1. **GET 延迟按 Reader Path** - 对比 `codec_streaming` vs `legacy_duplex` 延迟
-2. **Early-Stop 命中率** - 验证 early-stop 是否有效触发
-3. **Pipeline 失败率** - 检测优化引入的新故障模式
-4. **Bitrot 校验失败** - 确保数据完整性
-
-## 快速开始
-
-### 前置条件
-
--   Docker
--   Docker Compose
-
-### 部署
-
-运行以下命令启动整个技术栈：
-
-```bash
-docker compose up -d
-```
-
-### Tempo 高可用模式
-
-默认的 `docker-compose.yml` 对应单机栈。
-如果需要基于 Kafka 的 HA Tempo 配置，请使用：
-
-```bash
-docker compose -f docker-compose-example-for-rustfs.yml -f docker-compose-tempo-ha-override.yml up -d
-```
-
-### 访问仪表盘
-
-| 服务 | URL | 凭据 | 描述 |
-| :--- | :--- | :--- | :--- |
-| **Grafana** | [http://localhost:3000](http://localhost:3000) | `admin` / `admin` | 主要可视化中心。 |
-| **Prometheus** | [http://localhost:9090](http://localhost:9090) | - | 指标查询和状态。 |
-| **Jaeger UI** | [http://localhost:16686](http://localhost:16686) | - | 辅助追踪可视化。 |
-| **Tempo** | [http://localhost:3200](http://localhost:3200) | - | Tempo 状态/指标。 |
-
-## 配置
-
-### 数据持久化
-
-数据存储在以下 Docker 卷中：
-
--   `prometheus-data`: Prometheus 指标
--   `tempo-data`: Tempo 追踪 (WAL 和 Blocks)
--   `loki-data`: Loki 日志 (Chunks 和 Rules)
--   `jaeger-data`: Jaeger 追踪 (Badger DB)
-
-要清除所有数据：
-
-```bash
-docker compose down -v
-```
-
-### 自定义
-
--   **Prometheus**: 编辑 `prometheus.yml` 以添加抓取目标或告警规则。
--   **Grafana**: 仪表盘和数据源从 `grafana/` 目录预置。
--   **Collector**: 编辑 `otel-collector-config.yaml` 以修改管道、处理器或导出器。
-
-### 验证 RustFS Trace
-
-当 RustFS 将 `RUSTFS_OBS_ENDPOINT` 指向这套技术栈时，应将该值视为
-OTLP/HTTP 的基础 URL，例如：
-
-```bash
-export RUSTFS_OBS_ENDPOINT=http://host.docker.internal:4318
-```
-
-RustFS 会自动在该基础 URL 后补全：
-
-- `/v1/traces`
-- `/v1/metrics`
-- `/v1/logs`
-
-需要注意：
-
-- 启动阶段通常会先看到日志和指标，因此“先有日志/指标、后有 trace”是正常现象。
-- OpenTelemetry bridge 会把 `tracing` 字段作为日志 attributes 发送。Loki 会将这些 attributes 存为 structured metadata，同时 Collector 会把常用排障字段镜像进日志行，方便用简单的行内容过滤直接查到。
-- 可见的 trace 数据通常依赖启动后的真实 HTTP/S3/gRPC 请求流量，因为请求路径上的 span 是按需创建的。
-- `RUSTFS_OBS_LOGGER_LEVEL=info` 会保留顶层请求 span，但会过滤掉很多 `debug` 级别的嵌套 span。
-  如果 Tempo 或 Jaeger 中的 trace 看起来很稀疏，建议先改成 `RUSTFS_OBS_LOGGER_LEVEL=debug`，再判断是否是 collector 或 Tempo 问题。
-
-最小验证流程：
-
-```bash
-# 1. 启动本目录下的可观测性技术栈。
-docker compose up -d
-
-# 2. 以 OTLP/HTTP 导出方式启动 RustFS，并提高 span 可见性。
-export RUSTFS_OBS_ENDPOINT=http://host.docker.internal:4318
-export RUSTFS_OBS_LOGGER_LEVEL=debug
-
-# 3. 产生真实请求流量。
-curl -I http://127.0.0.1:9000/health
-curl -I http://127.0.0.1:9000/health/ready
-
-# 4. 到 Grafana 或 Jaeger 中检查。
-# Grafana: http://localhost:3000
-# Jaeger:  http://localhost:16686
-```
-
-对于 RustFS 结构化日志，例如节点间 RPC 鉴权失败，Loki 日志行现在会包含 `event`、`component`、`subsystem`、`failure_reason`、`rpc_service`、`rpc_method`、`expected_audience` 等字段。常用 LogQL 检查：
-
-```logql
-{service_name="RustFS"} |= "RPC signature verification failed"
-{service_name="RustFS"} |= "failure_reason="
-{service_name="RustFS"} | failure_reason != ""
-```
-
-如果日志和指标已经正常，但 trace 仍然稀疏，最常见的原因通常是
-“还没有真实请求流量”或“`info` 级别过滤了嵌套 span”，而不是 OTLP 路由失败。
-
-## 故障排除
-
--   **服务健康**: 使用 `docker compose ps` 检查服务健康状况。
--   **日志**: 使用 `docker compose logs -f <service_name>` 查看特定服务的日志。
--   **Otel Collector**: 检查 `http://localhost:13133` 获取健康状态，检查 `http://localhost:1888/debug/pprof/` 进行性能分析。

.docker/observability/docker-compose-example-for-rustfs.yml

@@ -1,269 +0,0 @@
-# Copyright 2024 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-services:
-  rustfs:
-    security_opt:
-      - "no-new-privileges:true"
-    image: rustfs/rustfs:latest
-    container_name: rustfs-server
-    ports:
-      - "9000:9000" # S3 API port
-      - "9001:9001" # Console port
-    environment:
-      - RUSTFS_VOLUMES=/data/rustfs
-      - RUSTFS_ADDRESS=0.0.0.0:9000
-      - RUSTFS_CONSOLE_ADDRESS=0.0.0.0:9001
-      - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_CORS_ALLOWED_ORIGINS=*
-      - RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS=*
-      - RUSTFS_ACCESS_KEY=rustfsadmin
-      - RUSTFS_SECRET_KEY=rustfsadmin
-      - RUSTFS_OBS_LOGGER_LEVEL=info
-      - RUSTFS_OBS_ENDPOINT=http://otel-collector:4318
-      - OTEL_RESOURCE_ATTRIBUTES=${OTEL_RESOURCE_ATTRIBUTES:-rustfs.cluster.id=rustfs-dev}
-      - RUSTFS_OBS_PROFILING_ENDPOINT=http://pyroscope:4040
-    volumes:
-      - rustfs-data:/data/rustfs
-    networks:
-      - otel-network
-    restart: unless-stopped
-    healthcheck:
-      test:
-        [
-          "CMD",
-          "sh",
-          "-c",
-          "curl -f http://127.0.0.1:9000/health && curl -f http://127.0.0.1:9001/rustfs/console/health",
-        ]
-      interval: 30s
-      timeout: 10s
-      retries: 3
-      start_period: 40s
-    depends_on:
-      otel-collector:
-        condition: service_started
-
-  rustfs-init:
-    image: alpine
-    container_name: rustfs-init
-    volumes:
-      - rustfs-data:/data
-    networks:
-      - otel-network
-    command: >
-      sh -c "
-        chown -R 10001:10001 /data &&
-        echo 'Volume Permissions fixed' &&
-        exit 0
-      "
-    restart: no
-
-  # --- Tracing ---
-
-  tempo:
-    image: grafana/tempo:latest
-    container_name: tempo
-    command: [ "-config.file=/etc/tempo.yaml" ]
-    volumes:
-      - ./tempo.yaml:/etc/tempo.yaml:ro
-      - tempo-data:/var/tempo
-    ports:
-      - "3200:3200" # tempo
-      - "4317" # otlp grpc
-      - "4318" # otlp http
-    networks:
-      - otel-network
-    restart: unless-stopped
-    healthcheck:
-      test: [ "CMD", "/tempo", "-version" ]
-      interval: 10s
-      timeout: 5s
-      retries: 3
-      start_period: 15s
-
-  redpanda:
-    image: redpandadata/redpanda:latest # for tempo ingest
-    container_name: redpanda
-    ports:
-      - "9092:9092"
-    networks:
-      - otel-network
-    restart: unless-stopped
-    command: >
-      redpanda start --overprovisioned
-      --mode=dev-container
-      --kafka-addr=PLAINTEXT://0.0.0.0:9092
-      --advertise-kafka-addr=PLAINTEXT://redpanda:9092
-
-  jaeger:
-    image: jaegertracing/jaeger:latest
-    container_name: jaeger
-    environment:
-      - SPAN_STORAGE_TYPE=badger
-      - BADGER_EPHEMERAL=false
-      - BADGER_DIRECTORY_VALUE=/badger/data
-      - BADGER_DIRECTORY_KEY=/badger/key
-      - COLLECTOR_OTLP_ENABLED=true
-    volumes:
-      - ./jaeger.yaml:/etc/jaeger/config.yml
-      - jaeger-data:/badger
-    ports:
-      - "16686:16686" # Web UI
-      - "14269:14269" # Admin/Metrics
-      - "4317" # otlp grpc
-      - "4318" # otlp http
-    command: [ "--config", "/etc/jaeger/config.yml" ]
-    networks:
-      - otel-network
-    restart: unless-stopped
-    healthcheck:
-      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:14269" ]
-      interval: 10s
-      timeout: 5s
-      retries: 3
-      start_period: 15s
-
-  # --- Metrics ---
-
-  prometheus:
-    image: prom/prometheus:latest
-    container_name: prometheus
-    volumes:
-      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
-      - prometheus-data:/prometheus
-    ports:
-      - "9090:9090"
-    command:
-      - "--config.file=/etc/prometheus/prometheus.yml"
-      - "--web.enable-otlp-receiver" # Enable OTLP
-      - "--web.enable-remote-write-receiver" # Enable remote write
-      - "--enable-feature=promql-experimental-functions" # Enable info()
-      - "--storage.tsdb.retention.time=30d"
-    restart: unless-stopped
-    networks:
-      - otel-network
-    healthcheck:
-      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:9090/-/healthy" ]
-      interval: 10s
-      timeout: 5s
-      retries: 3
-
-  # --- Logging ---
-
-  loki:
-    image: grafana/loki:latest
-    container_name: loki
-    volumes:
-      - ./loki.yaml:/etc/loki/loki.yaml:ro
-      - loki-data:/loki
-    ports:
-      - "3100:3100"
-    command: -config.file=/etc/loki/loki.yaml
-    networks:
-      - otel-network
-    restart: unless-stopped
-    healthcheck:
-      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:3100/ready" ]
-      interval: 15s
-      timeout: 10s
-      retries: 5
-      start_period: 60s
-
-  # --- Collection ---
-
-  otel-collector:
-    image: otel/opentelemetry-collector-contrib:latest
-    volumes:
-      - ./otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml:ro
-    ports:
-      - "1888:1888" # pprof
-      - "8888:8888" # Prometheus metrics for Collector
-      - "8889:8889" # Prometheus metrics for application indicators
-      - "13133:13133" # health check
-      - "4317:4317" # OTLP gRPC
-      - "4318:4318" # OTLP HTTP
-      - "55679:55679" # zpages
-    networks:
-      - otel-network
-    restart: unless-stopped
-    depends_on:
-      - tempo
-      - jaeger
-      - prometheus
-      - loki
-    healthcheck:
-      test: [ "CMD", "/otelcol-contrib", "--version" ]
-      interval: 10s
-      timeout: 5s
-      retries: 3
-
-  # --- Profiles ---
-
-  pyroscope:
-    image: grafana/pyroscope:latest
-    container_name: pyroscope
-    ports:
-      - "4040:4040"
-    command:
-      - -self-profiling.disable-push=true
-    networks:
-      - otel-network
-    restart: unless-stopped
-
-  # --- Visualization ---
-
-  grafana:
-    image: grafana/grafana:latest
-    container_name: grafana
-    ports:
-      - "3000:3000"
-    environment:
-      - GF_SECURITY_ADMIN_PASSWORD=admin
-      - GF_SECURITY_ADMIN_USER=admin
-    volumes:
-      - ./grafana/provisioning:/etc/grafana/provisioning:ro
-      - ./grafana/dashboards:/etc/grafana/dashboards:ro
-      - grafana-data:/var/lib/grafana
-    networks:
-      - otel-network
-    restart: unless-stopped
-    depends_on:
-      - prometheus
-      - tempo
-      - loki
-    healthcheck:
-      test:
-        [ "CMD", "wget", "--spider", "-q", "http://localhost:3000/api/health" ]
-      interval: 10s
-      timeout: 5s
-      retries: 3
-
-volumes:
-  rustfs-data:
-  tempo-data:
-  jaeger-data:
-  prometheus-data:
-  loki-data:
-  grafana-data:
-
-networks:
-  otel-network:
-    driver: bridge
-    name: "network_otel"
-    ipam:
-      config:
-        - subnet: 172.28.0.0/16
-    driver_opts:
-      com.docker.network.enable_ipv6: "true"

.docker/observability/docker-compose-tempo-ha-override.yml

@@ -1,62 +0,0 @@
-# Copyright 2024 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-# Docker Compose override file for High Availability Tempo setup
-#
-# Usage:
-# docker-compose -f docker-compose-example-for-rustfs.yml \
-#                -f docker-compose-tempo-ha-override.yml up
-
-services:
-  # Override Tempo to use high-availability configuration
-  tempo:
-    volumes:
-      - ./tempo-ha.yaml:/etc/tempo.yaml:ro
-      - tempo-data:/var/tempo
-    ports:
-      - "3200:3200"   # Tempo HTTP
-      - "4317:4317"   # OTLP gRPC
-      - "4318:4318"   # OTLP HTTP
-      - "7946:7946"   # Memberlist
-      - "14250:14250" # Jaeger gRPC
-      - "14268:14268" # Jaeger Thrift HTTP
-      - "9411:9411"   # Zipkin
-    environment:
-      - TEMPO_MEMBERLIST_BIND_PORT=7946
-    healthcheck:
-      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:3200/ready" ]
-      interval: 10s
-      timeout: 5s
-      retries: 5
-      start_period: 30s
-    depends_on:
-      - redpanda
-
-volumes:
-  tempo-data:
-    driver: local
-    driver_opts:
-      type: tmpfs
-      device: tmpfs
-      o: "size=4g"  # Allocate 4GB tmpfs for Tempo data (adjust based on your needs)
-
-# Network configuration remains the same
-# networks:
-#   otel-network:
-#     driver: bridge
-#     name: "network_otel"
-#     ipam:
-#       config:
-#         - subnet: 172.28.0.0/16
-

.docker/observability/docker-compose.yml

@@ -1,211 +0,0 @@
-# Copyright 2024 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-services:
-
-  # --- Tracing ---
-  tempo:
-    image: grafana/tempo:2.10.5
-    container_name: tempo
-    command: [ "-config.file=/etc/tempo.yaml" ]
-    volumes:
-      - ./tempo.yaml:/etc/tempo.yaml:ro
-      - tempo-data:/var/tempo
-    ports:
-      - "3200:3200" # tempo
-      - "4317" # otlp grpc
-      - "4318" # otlp http
-      - "7946" # memberlist
-    networks:
-      - otel-network
-    restart: unless-stopped
-    healthcheck:
-      test: [ "CMD", "/tempo", "-version" ]
-      interval: 10s
-      timeout: 5s
-      retries: 3
-      start_period: 15s
-
-  vulture:
-    image: grafana/tempo-vulture:latest
-    restart: always
-    command:
-      [
-        "-prometheus-listen-address=:8080",
-        "-tempo-query-url=http://tempo:3200",
-        "-tempo-push-url=http://tempo:4317",
-      ]
-    depends_on:
-      - tempo
-
-  jaeger:
-    image: jaegertracing/jaeger:latest
-    container_name: jaeger
-    environment:
-      - SPAN_STORAGE_TYPE=badger
-      - BADGER_EPHEMERAL=false
-      - BADGER_DIRECTORY_VALUE=/badger/data
-      - BADGER_DIRECTORY_KEY=/badger/key
-      - COLLECTOR_OTLP_ENABLED=true
-    volumes:
-      - ./jaeger.yaml:/etc/jaeger/config.yml
-      - jaeger-data:/badger
-    ports:
-      - "16686:16686" # Web UI
-      - "18888:8888" # Metrics
-      - "4317" # otlp grpc
-      - "4318" # otlp http
-    command: [ "--config", "/etc/jaeger/config.yml" ]
-    networks:
-      - otel-network
-    restart: unless-stopped
-    healthcheck:
-      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:8888/metrics" ]
-      interval: 10s
-      timeout: 5s
-      retries: 3
-      start_period: 15s
-
-  # --- Metrics ---
-
-  prometheus:
-    image: prom/prometheus:latest
-    container_name: prometheus
-    volumes:
-      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
-      - ./prometheus-rules:/etc/prometheus/rules:ro
-      - prometheus-data:/prometheus
-    ports:
-      - "9090:9090"
-    command:
-      - "--config.file=/etc/prometheus/prometheus.yml"
-      - "--web.enable-otlp-receiver" # Enable OTLP
-      - "--web.enable-remote-write-receiver" # Enable remote write
-      - "--enable-feature=promql-experimental-functions" # Enable info()
-      - "--storage.tsdb.retention.time=30d"
-    restart: unless-stopped
-    networks:
-      - otel-network
-    healthcheck:
-      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:9090/-/healthy" ]
-      interval: 10s
-      timeout: 5s
-      retries: 3
-
-  # --- Logging ---
-
-  loki:
-    image: grafana/loki:latest
-    container_name: loki
-    volumes:
-      - ./loki.yaml:/etc/loki/loki.yaml:ro
-      - loki-data:/loki
-    ports:
-      - "3100:3100"
-    command: -config.file=/etc/loki/loki.yaml
-    networks:
-      - otel-network
-    restart: unless-stopped
-    healthcheck:
-      test: [ "CMD", "/usr/bin/loki", "--version" ]
-      interval: 15s
-      timeout: 10s
-      retries: 5
-      start_period: 60s
-
-  # --- Collection ---
-
-  otel-collector:
-    image: otel/opentelemetry-collector-contrib:latest
-    volumes:
-      - ./otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml:ro
-    ports:
-      - "1888:1888" # pprof
-      - "8888:8888" # Prometheus metrics for Collector
-      - "8889:8889" # Prometheus metrics for application indicators
-      - "13133:13133" # health check
-      - "4317:4317" # OTLP gRPC
-      - "4318:4318" # OTLP HTTP
-      - "55679:55679" # zpages
-    networks:
-      - otel-network
-    restart: unless-stopped
-    depends_on:
-      - tempo
-      - jaeger
-      - prometheus
-      - loki
-    healthcheck:
-      test: [ "CMD", "/otelcol-contrib", "--version" ]
-      interval: 10s
-      timeout: 5s
-      retries: 3
-
-  # --- Profiles ---
-
-  pyroscope:
-    image: grafana/pyroscope:latest
-    container_name: pyroscope
-    ports:
-      - "4040:4040"
-    command:
-      - -self-profiling.disable-push=true
-    networks:
-      - otel-network
-    restart: unless-stopped
-
-  # --- Visualization ---
-
-  grafana:
-    image: grafana/grafana:latest
-    container_name: grafana
-    ports:
-      - "3000:3000"
-    environment:
-      - GF_SECURITY_ADMIN_PASSWORD=admin
-      - GF_SECURITY_ADMIN_USER=admin
-    volumes:
-      - ./grafana/provisioning:/etc/grafana/provisioning:ro
-      - ./grafana/dashboards:/etc/grafana/dashboards:ro
-      - grafana-data:/var/lib/grafana
-    networks:
-      - otel-network
-    restart: unless-stopped
-    depends_on:
-      - prometheus
-      - tempo
-      - loki
-    healthcheck:
-      test:
-        [ "CMD", "wget", "--spider", "-q", "http://localhost:3000/api/health" ]
-      interval: 10s
-      timeout: 5s
-      retries: 3
-
-volumes:
-  tempo-data:
-  jaeger-data:
-  prometheus-data:
-  loki-data:
-  grafana-data:
-
-networks:
-  otel-network:
-    driver: bridge
-    name: "network_otel"
-    ipam:
-      config:
-        - subnet: 172.28.0.0/16
-    driver_opts:
-      com.docker.network.enable_ipv6: "true"

.docker/observability/grafana/dashboards/grafana-get-data-integrity.json

@@ -1,500 +0,0 @@
-{
-  "annotations": {
-    "list": []
-  },
-  "editable": true,
-  "fiscalYearStartMonth": 0,
-  "graphTooltip": 1,
-  "id": null,
-  "links": [],
-  "liveNow": false,
-  "panels": [
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "ops",
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 20,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "thresholdsStyle": {
-              "mode": "line+area"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "transparent",
-                "value": null
-              },
-              {
-                "color": "red",
-                "value": 1
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 0,
-        "y": 0
-      },
-      "id": 1,
-      "options": {
-        "reduceOptions": {
-          "values": false,
-          "calcs": ["lastNotNull"],
-          "fields": ""
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        },
-        "legend": {
-          "displayMode": "table",
-          "placement": "bottom",
-          "calcs": ["mean", "max", "sum"]
-        }
-      },
-      "title": "Bitrot Verify Failures",
-      "type": "timeseries",
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(rate(rustfs_io_get_object_pipeline_failures_total{stage=\"bitrot_verify\"}[$__rate_interval])) by (path, reason)",
-          "legendFormat": "bitrot_fail {{path}} / {{reason}}",
-          "refId": "A"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(rate(rustfs_io_get_object_pipeline_failures_total{reason=\"bitrot_verify_failed\"}[$__rate_interval])) by (path, stage)",
-          "legendFormat": "verify_failed {{path}} / {{stage}}",
-          "refId": "B"
-        }
-      ]
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "s",
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 10,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 12,
-        "y": 0
-      },
-      "id": 2,
-      "options": {
-        "reduceOptions": {
-          "values": false,
-          "calcs": ["lastNotNull"],
-          "fields": ""
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        },
-        "legend": {
-          "displayMode": "table",
-          "placement": "bottom",
-          "calcs": ["mean", "max"]
-        }
-      },
-      "title": "Bitrot Verify Duration (p50 / p95 / p99)",
-      "type": "timeseries",
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "histogram_quantile(0.50, sum(rate(rustfs_io_get_object_shard_bitrot_verify_duration_seconds_bucket[$__rate_interval])) by (le))",
-          "legendFormat": "p50",
-          "refId": "A"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "histogram_quantile(0.95, sum(rate(rustfs_io_get_object_shard_bitrot_verify_duration_seconds_bucket[$__rate_interval])) by (le))",
-          "legendFormat": "p95",
-          "refId": "B"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "histogram_quantile(0.99, sum(rate(rustfs_io_get_object_shard_bitrot_verify_duration_seconds_bucket[$__rate_interval])) by (le))",
-          "legendFormat": "p99",
-          "refId": "C"
-        }
-      ]
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "ops",
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 20,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "thresholdsStyle": {
-              "mode": "line+area"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "transparent",
-                "value": null
-              },
-              {
-                "color": "red",
-                "value": 1
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 0,
-        "y": 8
-      },
-      "id": 3,
-      "options": {
-        "reduceOptions": {
-          "values": false,
-          "calcs": ["lastNotNull"],
-          "fields": ""
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        },
-        "legend": {
-          "displayMode": "table",
-          "placement": "bottom",
-          "calcs": ["mean", "max", "sum"]
-        }
-      },
-      "title": "Decode Errors",
-      "type": "timeseries",
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(rate(rustfs_io_get_object_pipeline_failures_total{reason=\"decode_error\"}[$__rate_interval])) by (path, stage)",
-          "legendFormat": "decode_error {{path}} / {{stage}}",
-          "refId": "A"
-        }
-      ]
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "ops",
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 20,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "thresholdsStyle": {
-              "mode": "line+area"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "transparent",
-                "value": null
-              },
-              {
-                "color": "red",
-                "value": 1
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 12,
-        "y": 8
-      },
-      "id": 4,
-      "options": {
-        "reduceOptions": {
-          "values": false,
-          "calcs": ["lastNotNull"],
-          "fields": ""
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        },
-        "legend": {
-          "displayMode": "table",
-          "placement": "bottom",
-          "calcs": ["mean", "max", "sum"]
-        }
-      },
-      "title": "Short Reads",
-      "type": "timeseries",
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(rate(rustfs_io_get_object_pipeline_failures_total{reason=\"short_read\"}[$__rate_interval])) by (path, stage)",
-          "legendFormat": "short_read {{path}} / {{stage}}",
-          "refId": "A"
-        }
-      ]
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "ops",
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 10,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 0,
-        "y": 16
-      },
-      "id": 5,
-      "options": {
-        "reduceOptions": {
-          "values": false,
-          "calcs": ["lastNotNull"],
-          "fields": ""
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        },
-        "legend": {
-          "displayMode": "table",
-          "placement": "bottom",
-          "calcs": ["mean", "max"]
-        }
-      },
-      "title": "Shard Read Outcomes by Role",
-      "type": "timeseries",
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(rate(rustfs_io_get_object_shard_read_total[$__rate_interval])) by (path, role, outcome)",
-          "legendFormat": "{{path}} {{role}}/{{outcome}}",
-          "refId": "A"
-        }
-      ]
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "ops",
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 10,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 12,
-        "y": 16
-      },
-      "id": 6,
-      "options": {
-        "reduceOptions": {
-          "values": false,
-          "calcs": ["lastNotNull"],
-          "fields": ""
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        },
-        "legend": {
-          "displayMode": "table",
-          "placement": "bottom",
-          "calcs": ["mean", "max"]
-        }
-      },
-      "title": "Request Result Status Rate",
-      "type": "timeseries",
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(rate(rustfs_io_get_object_request_results_total[$__rate_interval])) by (status)",
-          "legendFormat": "{{status}}",
-          "refId": "A"
-        }
-      ]
-    }
-  ],
-  "refresh": "30s",
-  "schemaVersion": 38,
-  "style": "dark",
-  "tags": ["rustfs", "get-optimization", "data-integrity"],
-  "templating": {
-    "list": [
-      {
-        "current": {
-          "selected": false,
-          "text": "Prometheus",
-          "value": "Prometheus"
-        },
-        "hide": 0,
-        "includeAll": false,
-        "multi": false,
-        "name": "DS_PROMETHEUS",
-        "options": [],
-        "query": "prometheus",
-        "refresh": 1,
-        "regex": "",
-        "skipUrlSync": false,
-        "type": "datasource"
-      }
-    ]
-  },
-  "time": {
-    "from": "now-1h",
-    "to": "now"
-  },
-  "timepicker": {},
-  "timezone": "",
-  "title": "RustFS GET Data Integrity",
-  "uid": "rustfs-get-data-integrity",
-  "version": 1
-}

.docker/observability/grafana/dashboards/grafana-get-performance-attribution.json

@@ -1,716 +0,0 @@
-{
-  "annotations": {
-    "list": []
-  },
-  "editable": true,
-  "fiscalYearStartMonth": 0,
-  "graphTooltip": 1,
-  "id": null,
-  "links": [],
-  "liveNow": false,
-  "panels": [
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "ops",
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 10,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 0,
-        "y": 0
-      },
-      "id": 1,
-      "options": {
-        "legend": {
-          "calcs": [
-            "mean",
-            "max"
-          ],
-          "displayMode": "table",
-          "placement": "bottom"
-        },
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ],
-          "fields": "",
-          "values": false
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum by (strategy, mode) (rate(rustfs_io_get_object_reader_setup_strategy_total[$__rate_interval])) or vector(0)",
-          "legendFormat": "{{strategy}} / {{mode}}",
-          "refId": "A"
-        }
-      ],
-      "title": "GET Reader Setup Strategy Rate",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "ops",
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 10,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 12,
-        "y": 0
-      },
-      "id": 2,
-      "options": {
-        "legend": {
-          "calcs": [
-            "mean",
-            "max"
-          ],
-          "displayMode": "table",
-          "placement": "bottom"
-        },
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ],
-          "fields": "",
-          "values": false
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum by (path, strategy, size_bucket) (rate(rustfs_io_get_object_reader_setup_strategy_by_size_total[$__rate_interval])) or vector(0)",
-          "legendFormat": "{{path}} / {{strategy}} / {{size_bucket}}",
-          "refId": "A"
-        }
-      ],
-      "title": "GET Reader Setup Strategy by Size",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "none",
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 10,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 0,
-        "y": 8
-      },
-      "id": 3,
-      "options": {
-        "legend": {
-          "calcs": [
-            "mean",
-            "max"
-          ],
-          "displayMode": "table",
-          "placement": "bottom"
-        },
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ],
-          "fields": "",
-          "values": false
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum by (strategy, mode) (rate(rustfs_io_get_object_reader_setup_scheduled_sum[$__rate_interval])) / clamp_min(sum by (strategy, mode) (rate(rustfs_io_get_object_reader_setup_scheduled_count[$__rate_interval])), 1) or vector(0)",
-          "legendFormat": "scheduled {{strategy}} / {{mode}}",
-          "refId": "A"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum by (strategy, mode) (rate(rustfs_io_get_object_reader_setup_attempted_sum[$__rate_interval])) / clamp_min(sum by (strategy, mode) (rate(rustfs_io_get_object_reader_setup_attempted_count[$__rate_interval])), 1) or vector(0)",
-          "legendFormat": "attempted {{strategy}} / {{mode}}",
-          "refId": "B"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum by (strategy, mode) (rate(rustfs_io_get_object_reader_setup_ready_sum[$__rate_interval])) / clamp_min(sum by (strategy, mode) (rate(rustfs_io_get_object_reader_setup_ready_count[$__rate_interval])), 1) or vector(0)",
-          "legendFormat": "ready {{strategy}} / {{mode}}",
-          "refId": "C"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum by (strategy, mode) (rate(rustfs_io_get_object_reader_setup_deferred_sum[$__rate_interval])) / clamp_min(sum by (strategy, mode) (rate(rustfs_io_get_object_reader_setup_deferred_count[$__rate_interval])), 1) or vector(0)",
-          "legendFormat": "deferred {{strategy}} / {{mode}}",
-          "refId": "D"
-        }
-      ],
-      "title": "GET Reader Setup Fanout Average",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "none",
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 10,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 12,
-        "y": 8
-      },
-      "id": 4,
-      "options": {
-        "legend": {
-          "calcs": [
-            "mean",
-            "max"
-          ],
-          "displayMode": "table",
-          "placement": "bottom"
-        },
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ],
-          "fields": "",
-          "values": false
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum by (path, strategy, size_bucket) (rate(rustfs_io_get_object_reader_setup_scheduled_by_size_sum[$__rate_interval])) / clamp_min(sum by (path, strategy, size_bucket) (rate(rustfs_io_get_object_reader_setup_scheduled_by_size_count[$__rate_interval])), 1) or vector(0)",
-          "legendFormat": "scheduled {{path}} / {{strategy}} / {{size_bucket}}",
-          "refId": "A"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum by (path, strategy, size_bucket) (rate(rustfs_io_get_object_reader_setup_ready_by_size_sum[$__rate_interval])) / clamp_min(sum by (path, strategy, size_bucket) (rate(rustfs_io_get_object_reader_setup_ready_by_size_count[$__rate_interval])), 1) or vector(0)",
-          "legendFormat": "ready {{path}} / {{strategy}} / {{size_bucket}}",
-          "refId": "B"
-        }
-      ],
-      "title": "GET Reader Setup Fanout by Size",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "s",
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 10,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 0,
-        "y": 16
-      },
-      "id": 5,
-      "options": {
-        "legend": {
-          "calcs": [
-            "mean",
-            "max"
-          ],
-          "displayMode": "table",
-          "placement": "bottom"
-        },
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ],
-          "fields": "",
-          "values": false
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "histogram_quantile(0.95, sum by (path, stage, size_bucket, le) (rate(rustfs_io_get_object_stage_duration_seconds_by_size_bucket{stage=~\"reader_setup|reader_setup_schedule|reader_setup_wait_quorum|reader_setup_drop_pending|reader_task_file_open|reader_task_reader_construction|reader_task_bitrot_reader_init\"}[$__rate_interval]))) or vector(0)",
-          "legendFormat": "p95 {{path}} / {{stage}} / {{size_bucket}}",
-          "refId": "A"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "histogram_quantile(0.99, sum by (path, stage, size_bucket, le) (rate(rustfs_io_get_object_stage_duration_seconds_by_size_bucket{stage=~\"reader_setup|reader_setup_schedule|reader_setup_wait_quorum|reader_setup_drop_pending|reader_task_file_open|reader_task_reader_construction|reader_task_bitrot_reader_init\"}[$__rate_interval]))) or vector(0)",
-          "legendFormat": "p99 {{path}} / {{stage}} / {{size_bucket}}",
-          "refId": "B"
-        }
-      ],
-      "title": "GET Hot Stage Latency by Size",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "s",
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 10,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 12,
-        "y": 16
-      },
-      "id": 6,
-      "options": {
-        "legend": {
-          "calcs": [
-            "mean",
-            "max"
-          ],
-          "displayMode": "table",
-          "placement": "bottom"
-        },
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ],
-          "fields": "",
-          "values": false
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "histogram_quantile(0.95, sum by (path, stage, le) (rate(rustfs_io_get_object_stage_duration_seconds_bucket{stage=~\"request_context|first_byte|response_handoff|full_body|body_build|output_poll|output_lock_wait|reader_open_mmap_copy_success|reader_open_mmap_copy_fallback|reader_open_stream|reader_stream_first_read|reader_mmap_blocking_wait|reader_mmap_blocking_task|reader_mmap_file_open|reader_mmap_map|reader_mmap_copy_buffer|reader_mmap_direct_read_copy\"}[$__rate_interval]))) or vector(0)",
-          "legendFormat": "p95 {{path}} / {{stage}}",
-          "refId": "A"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "histogram_quantile(0.99, sum by (path, stage, le) (rate(rustfs_io_get_object_stage_duration_seconds_bucket{stage=~\"request_context|first_byte|response_handoff|full_body|body_build|output_poll|output_lock_wait|reader_open_mmap_copy_success|reader_open_mmap_copy_fallback|reader_open_stream|reader_stream_first_read|reader_mmap_blocking_wait|reader_mmap_blocking_task|reader_mmap_file_open|reader_mmap_map|reader_mmap_copy_buffer|reader_mmap_direct_read_copy\"}[$__rate_interval]))) or vector(0)",
-          "legendFormat": "p99 {{path}} / {{stage}}",
-          "refId": "B"
-        }
-      ],
-      "title": "GET Reader, Handler and Body Latency",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "ops",
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 10,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 0,
-        "y": 24
-      },
-      "id": 7,
-      "options": {
-        "legend": {
-          "calcs": [
-            "mean",
-            "max"
-          ],
-          "displayMode": "table",
-          "placement": "bottom"
-        },
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ],
-          "fields": "",
-          "values": false
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum by (subpath) (rate(rustfs_io_get_object_direct_memory_subpath_total[$__rate_interval])) or vector(0)",
-          "legendFormat": "{{subpath}}",
-          "refId": "A"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum by (subpath, size_bucket) (rate(rustfs_io_get_object_direct_memory_subpath_by_size_total[$__rate_interval])) or vector(0)",
-          "legendFormat": "{{subpath}} / {{size_bucket}}",
-          "refId": "B"
-        }
-      ],
-      "title": "GET Direct-Memory Subpath Rate",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "ops",
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 10,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 12,
-        "y": 24
-      },
-      "id": 8,
-      "options": {
-        "legend": {
-          "calcs": [
-            "mean",
-            "max"
-          ],
-          "displayMode": "table",
-          "placement": "bottom"
-        },
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ],
-          "fields": "",
-          "values": false
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum by (path, stage, kind) (rate(rustfs_io_get_object_mmap_page_faults_total[$__rate_interval])) or vector(0)",
-          "legendFormat": "mmap {{path}} / {{stage}} / {{kind}}",
-          "refId": "A"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum by (path, stage, kind) (rate(rustfs_io_get_object_direct_read_page_faults_total[$__rate_interval])) or vector(0)",
-          "legendFormat": "direct-read {{path}} / {{stage}} / {{kind}}",
-          "refId": "B"
-        }
-      ],
-      "title": "GET mmap vs Direct-Read Page Fault Rate",
-      "type": "timeseries"
-    }
-  ],
-  "refresh": "30s",
-  "schemaVersion": 38,
-  "style": "dark",
-  "tags": [
-    "rustfs",
-    "get",
-    "performance",
-    "attribution"
-  ],
-  "templating": {
-    "list": [
-      {
-        "current": {
-          "selected": false,
-          "text": "Prometheus",
-          "value": "Prometheus"
-        },
-        "hide": 0,
-        "includeAll": false,
-        "multi": false,
-        "name": "DS_PROMETHEUS",
-        "options": [],
-        "query": "prometheus",
-        "refresh": 1,
-        "regex": "",
-        "skipUrlSync": false,
-        "type": "datasource"
-      }
-    ]
-  },
-  "time": {
-    "from": "now-1h",
-    "to": "now"
-  },
-  "timezone": "",
-  "title": "RustFS GET Performance Attribution",
-  "uid": "rustfs-get-performance-attribution",
-  "version": 1,
-  "weekStart": ""
-}

.docker/observability/grafana/dashboards/grafana-get-resource-impact.json

@@ -1,525 +0,0 @@
-{
-  "annotations": {
-    "list": []
-  },
-  "editable": true,
-  "fiscalYearStartMonth": 0,
-  "graphTooltip": 1,
-  "id": null,
-  "links": [],
-  "liveNow": false,
-  "panels": [
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "none",
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 20,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 0,
-        "y": 0
-      },
-      "id": 1,
-      "options": {
-        "reduceOptions": {
-          "values": false,
-          "calcs": ["lastNotNull"],
-          "fields": ""
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        },
-        "legend": {
-          "displayMode": "table",
-          "placement": "bottom",
-          "calcs": ["mean", "max"]
-        }
-      },
-      "title": "Concurrent GET Requests",
-      "type": "timeseries",
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "rustfs_io_get_object_concurrent_requests",
-          "legendFormat": "concurrent GETs",
-          "refId": "A"
-        }
-      ]
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "percent",
-          "min": 0,
-          "max": 100,
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 20,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "thresholdsStyle": {
-              "mode": "line+area"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "transparent",
-                "value": null
-              },
-              {
-                "color": "yellow",
-                "value": 70
-              },
-              {
-                "color": "red",
-                "value": 90
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 12,
-        "y": 0
-      },
-      "id": 2,
-      "options": {
-        "reduceOptions": {
-          "values": false,
-          "calcs": ["lastNotNull"],
-          "fields": ""
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        },
-        "legend": {
-          "displayMode": "table",
-          "placement": "bottom",
-          "calcs": ["mean", "max"]
-        }
-      },
-      "title": "IO Queue Utilization",
-      "type": "timeseries",
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "rustfs_io_queue_utilization_percent",
-          "legendFormat": "queue utilization %",
-          "refId": "A"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "rustfs_io_queue_permits_in_use",
-          "legendFormat": "permits in use",
-          "refId": "B"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "rustfs_io_queue_permits_available",
-          "legendFormat": "permits available",
-          "refId": "C"
-        }
-      ]
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "s",
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 10,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 0,
-        "y": 8
-      },
-      "id": 3,
-      "options": {
-        "reduceOptions": {
-          "values": false,
-          "calcs": ["lastNotNull"],
-          "fields": ""
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        },
-        "legend": {
-          "displayMode": "table",
-          "placement": "bottom",
-          "calcs": ["mean", "max"]
-        }
-      },
-      "title": "Disk Permit Wait Duration (p50 / p95 / p99)",
-      "type": "timeseries",
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "histogram_quantile(0.50, sum(rate(rustfs_io_disk_permit_wait_duration_seconds_bucket[$__rate_interval])) by (le))",
-          "legendFormat": "p50",
-          "refId": "A"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "histogram_quantile(0.95, sum(rate(rustfs_io_disk_permit_wait_duration_seconds_bucket[$__rate_interval])) by (le))",
-          "legendFormat": "p95",
-          "refId": "B"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "histogram_quantile(0.99, sum(rate(rustfs_io_disk_permit_wait_duration_seconds_bucket[$__rate_interval])) by (le))",
-          "legendFormat": "p99",
-          "refId": "C"
-        }
-      ]
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "bytes",
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 15,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 12,
-        "y": 8
-      },
-      "id": 4,
-      "options": {
-        "reduceOptions": {
-          "values": false,
-          "calcs": ["lastNotNull"],
-          "fields": ""
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        },
-        "legend": {
-          "displayMode": "table",
-          "placement": "bottom",
-          "calcs": ["lastNotNull"]
-        }
-      },
-      "title": "RSS Trend (Process Memory)",
-      "type": "timeseries",
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "process_resident_memory_bytes{job=~\"rustfs.*\"}",
-          "legendFormat": "RSS {{instance}}",
-          "refId": "A"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "rustfs_process_memory_bytes",
-          "legendFormat": "RustFS memory {{instance}}",
-          "refId": "B"
-        }
-      ]
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "ops",
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 10,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 0,
-        "y": 16
-      },
-      "id": 5,
-      "options": {
-        "reduceOptions": {
-          "values": false,
-          "calcs": ["lastNotNull"],
-          "fields": ""
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        },
-        "legend": {
-          "displayMode": "table",
-          "placement": "bottom",
-          "calcs": ["mean", "max"]
-        }
-      },
-      "title": "IO Queue Operations by Priority",
-      "type": "timeseries",
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(rate(rustfs_io_queue_operations[$__rate_interval])) by (operation, priority)",
-          "legendFormat": "{{operation}} / {{priority}}",
-          "refId": "A"
-        }
-      ]
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "ops",
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 15,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 12,
-        "y": 16
-      },
-      "id": 6,
-      "options": {
-        "reduceOptions": {
-          "values": false,
-          "calcs": ["lastNotNull"],
-          "fields": ""
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        },
-        "legend": {
-          "displayMode": "table",
-          "placement": "bottom",
-          "calcs": ["sum"]
-        }
-      },
-      "title": "IO Queue Congestion Events",
-      "type": "timeseries",
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(rate(rustfs_io_queue_congestion_total[$__rate_interval]))",
-          "legendFormat": "congestion events/s",
-          "refId": "A"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(rate(rustfs_io_get_object_timeout_total[$__rate_interval])) by (stage)",
-          "legendFormat": "timeout {{stage}}",
-          "refId": "B"
-        }
-      ]
-    }
-  ],
-  "refresh": "30s",
-  "schemaVersion": 38,
-  "style": "dark",
-  "tags": ["rustfs", "get-optimization", "resource-impact"],
-  "templating": {
-    "list": [
-      {
-        "current": {
-          "selected": false,
-          "text": "Prometheus",
-          "value": "Prometheus"
-        },
-        "hide": 0,
-        "includeAll": false,
-        "multi": false,
-        "name": "DS_PROMETHEUS",
-        "options": [],
-        "query": "prometheus",
-        "refresh": 1,
-        "regex": "",
-        "skipUrlSync": false,
-        "type": "datasource"
-      }
-    ]
-  },
-  "time": {
-    "from": "now-1h",
-    "to": "now"
-  },
-  "timepicker": {},
-  "timezone": "",
-  "title": "RustFS GET Resource Impact",
-  "uid": "rustfs-get-resource-impact",
-  "version": 1
-}

.docker/observability/grafana/dashboards/grafana-get-rollout-health.json

@@ -1,530 +0,0 @@
-{
-  "annotations": {
-    "list": []
-  },
-  "editable": true,
-  "fiscalYearStartMonth": 0,
-  "graphTooltip": 1,
-  "id": null,
-  "links": [],
-  "liveNow": false,
-  "panels": [
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "s",
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 10,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "axisLabel": "",
-            "axisColorMode": "text",
-            "scaleDistribution": {
-              "type": "linear"
-            },
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 0,
-        "y": 0
-      },
-      "id": 1,
-      "options": {
-        "reduceOptions": {
-          "values": false,
-          "calcs": ["lastNotNull"],
-          "fields": ""
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        },
-        "legend": {
-          "displayMode": "table",
-          "placement": "bottom",
-          "calcs": ["mean", "max"]
-        }
-      },
-      "title": "GET Latency by Reader Path (p50 / p95 / p99)",
-      "type": "timeseries",
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "histogram_quantile(0.50, sum(rate(rustfs_io_get_object_stage_duration_seconds_bucket{stage=\"total\"}[$__rate_interval])) by (le, path))",
-          "legendFormat": "p50 {{path}}",
-          "refId": "A"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "histogram_quantile(0.95, sum(rate(rustfs_io_get_object_stage_duration_seconds_bucket{stage=\"total\"}[$__rate_interval])) by (le, path))",
-          "legendFormat": "p95 {{path}}",
-          "refId": "B"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "histogram_quantile(0.99, sum(rate(rustfs_io_get_object_stage_duration_seconds_bucket{stage=\"total\"}[$__rate_interval])) by (le, path))",
-          "legendFormat": "p99 {{path}}",
-          "refId": "C"
-        }
-      ]
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "percentunit",
-          "min": 0,
-          "max": 1,
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 15,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              },
-              {
-                "color": "yellow",
-                "value": 0.3
-              },
-              {
-                "color": "red",
-                "value": 0.7
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 12,
-        "y": 0
-      },
-      "id": 2,
-      "options": {
-        "reduceOptions": {
-          "values": false,
-          "calcs": ["lastNotNull"],
-          "fields": ""
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        },
-        "legend": {
-          "displayMode": "table",
-          "placement": "bottom",
-          "calcs": ["mean", "max"]
-        }
-      },
-      "title": "Early-Stop Hit Rate",
-      "type": "timeseries",
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(rate(rustfs_io_get_object_metadata_early_stop_total{decision=\"hit\"}[$__rate_interval])) by (path) / clamp_min(sum(rate(rustfs_io_get_object_metadata_early_stop_total[$__rate_interval])) by (path), 1)",
-          "legendFormat": "hit rate {{path}}",
-          "refId": "A"
-        }
-      ]
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "percentunit",
-          "min": 0,
-          "max": 1,
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 15,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 0,
-        "y": 8
-      },
-      "id": 3,
-      "options": {
-        "reduceOptions": {
-          "values": false,
-          "calcs": ["lastNotNull"],
-          "fields": ""
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        },
-        "legend": {
-          "displayMode": "table",
-          "placement": "bottom",
-          "calcs": ["mean", "max"]
-        }
-      },
-      "title": "Codec Streaming Usage Rate",
-      "type": "timeseries",
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(rate(rustfs_io_get_object_codec_streaming_decision_total[$__rate_interval])) by (decision) / clamp_min(sum(rate(rustfs_io_get_object_codec_streaming_decision_total[$__rate_interval])), 1)",
-          "legendFormat": "codec_streaming {{decision}}",
-          "refId": "A"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(rate(rustfs_io_get_object_stream_strategy_total[$__rate_interval])) by (strategy) / clamp_min(sum(rate(rustfs_io_get_object_stream_strategy_total[$__rate_interval])), 1)",
-          "legendFormat": "stream_strategy {{strategy}}",
-          "refId": "B"
-        }
-      ]
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "ops",
-          "custom": {
-            "drawStyle": "bars",
-            "fillOpacity": 80,
-            "lineWidth": 0,
-            "pointSize": 5,
-            "showPoints": "never",
-            "spanNulls": false,
-            "stacking": {
-              "mode": "normal",
-              "group": "A"
-            },
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              },
-              {
-                "color": "red",
-                "value": 1
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 12,
-        "y": 8
-      },
-      "id": 4,
-      "options": {
-        "reduceOptions": {
-          "values": false,
-          "calcs": ["lastNotNull"],
-          "fields": ""
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        },
-        "legend": {
-          "displayMode": "table",
-          "placement": "bottom",
-          "calcs": ["sum"]
-        }
-      },
-      "title": "Pipeline Failure Rate by Stage",
-      "type": "timeseries",
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(rate(rustfs_io_get_object_pipeline_failures_total[$__rate_interval])) by (stage, reason)",
-          "legendFormat": "{{stage}} / {{reason}}",
-          "refId": "A"
-        }
-      ]
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "reqps",
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 10,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 0,
-        "y": 16
-      },
-      "id": 5,
-      "options": {
-        "reduceOptions": {
-          "values": false,
-          "calcs": ["lastNotNull"],
-          "fields": ""
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        },
-        "legend": {
-          "displayMode": "table",
-          "placement": "bottom",
-          "calcs": ["mean", "max"]
-        }
-      },
-      "title": "GET Request Rate by Path",
-      "type": "timeseries",
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(rate(rustfs_io_get_object_reader_path_total[$__rate_interval])) by (path)",
-          "legendFormat": "{{path}}",
-          "refId": "A"
-        }
-      ]
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "ops",
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 10,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 12,
-        "y": 16
-      },
-      "id": 6,
-      "options": {
-        "reduceOptions": {
-          "values": false,
-          "calcs": ["lastNotNull"],
-          "fields": ""
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        },
-        "legend": {
-          "displayMode": "table",
-          "placement": "bottom",
-          "calcs": ["mean", "max"]
-        }
-      },
-      "title": "GET Total Latency (p50 / p95 / p99)",
-      "type": "timeseries",
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "histogram_quantile(0.50, sum(rate(rustfs_io_get_object_total_duration_seconds_bucket[$__rate_interval])) by (le))",
-          "legendFormat": "p50",
-          "refId": "A"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "histogram_quantile(0.95, sum(rate(rustfs_io_get_object_total_duration_seconds_bucket[$__rate_interval])) by (le))",
-          "legendFormat": "p95",
-          "refId": "B"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "histogram_quantile(0.99, sum(rate(rustfs_io_get_object_total_duration_seconds_bucket[$__rate_interval])) by (le))",
-          "legendFormat": "p99",
-          "refId": "C"
-        }
-      ]
-    }
-  ],
-  "refresh": "30s",
-  "schemaVersion": 38,
-  "style": "dark",
-  "tags": ["rustfs", "get-optimization"],
-  "templating": {
-    "list": [
-      {
-        "current": {
-          "selected": false,
-          "text": "Prometheus",
-          "value": "Prometheus"
-        },
-        "hide": 0,
-        "includeAll": false,
-        "multi": false,
-        "name": "DS_PROMETHEUS",
-        "options": [],
-        "query": "prometheus",
-        "refresh": 1,
-        "regex": "",
-        "skipUrlSync": false,
-        "type": "datasource"
-      }
-    ]
-  },
-  "time": {
-    "from": "now-1h",
-    "to": "now"
-  },
-  "timepicker": {},
-  "timezone": "",
-  "title": "RustFS GET Rollout Health",
-  "uid": "rustfs-get-rollout-health",
-  "version": 1
-}

.docker/observability/grafana/dashboards/grafana-put-performance-attribution.json

@@ -1,534 +0,0 @@
-{
-  "annotations": {
-    "list": []
-  },
-  "editable": true,
-  "fiscalYearStartMonth": 0,
-  "graphTooltip": 1,
-  "id": null,
-  "links": [],
-  "liveNow": false,
-  "panels": [
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "ops",
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 10,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 0,
-        "y": 0
-      },
-      "id": 1,
-      "options": {
-        "legend": {
-          "calcs": [
-            "mean",
-            "max"
-          ],
-          "displayMode": "table",
-          "placement": "bottom"
-        },
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ],
-          "fields": "",
-          "values": false
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum by (path, eager_status, size_bucket, buffer_bucket, large_concurrency_tuning) (rate(rustfs_s3_put_object_diagnostics_total[$__rate_interval])) or vector(0)",
-          "legendFormat": "{{path}} / {{eager_status}} / {{size_bucket}} / {{buffer_bucket}} / large={{large_concurrency_tuning}}",
-          "refId": "A"
-        }
-      ],
-      "title": "PUT Diagnostics Decision Rate",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "ops",
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 10,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 12,
-        "y": 0
-      },
-      "id": 2,
-      "options": {
-        "legend": {
-          "calcs": [
-            "mean",
-            "max"
-          ],
-          "displayMode": "table",
-          "placement": "bottom"
-        },
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ],
-          "fields": "",
-          "values": false
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum by (path) (rate(rustfs_s3_put_object_path_total[$__rate_interval])) or vector(0)",
-          "legendFormat": "{{path}}",
-          "refId": "A"
-        }
-      ],
-      "title": "PUT Path Rate",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "ms",
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 10,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 0,
-        "y": 8
-      },
-      "id": 3,
-      "options": {
-        "legend": {
-          "calcs": [
-            "mean",
-            "max"
-          ],
-          "displayMode": "table",
-          "placement": "bottom"
-        },
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ],
-          "fields": "",
-          "values": false
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "histogram_quantile(0.95, sum by (stage, le) (rate(rustfs_s3_put_object_stage_duration_ms_bucket{stage=~\"ingress_prepare|set_disk_writer_setup|set_disk_encode|set_disk_rename|set_disk_old_data_cleanup|multipart_ingress_prepare|multipart_set_disk_writer_setup|multipart_set_disk_encode|multipart_complete_tail\"}[$__rate_interval]))) or vector(0)",
-          "legendFormat": "p95 {{stage}}",
-          "refId": "A"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "histogram_quantile(0.99, sum by (stage, le) (rate(rustfs_s3_put_object_stage_duration_ms_bucket{stage=~\"ingress_prepare|set_disk_writer_setup|set_disk_encode|set_disk_rename|set_disk_old_data_cleanup|multipart_ingress_prepare|multipart_set_disk_writer_setup|multipart_set_disk_encode|multipart_complete_tail\"}[$__rate_interval]))) or vector(0)",
-          "legendFormat": "p99 {{stage}}",
-          "refId": "B"
-        }
-      ],
-      "title": "PUT Hot Stage Latency",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "bytes",
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 10,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 12,
-        "y": 8
-      },
-      "id": 4,
-      "options": {
-        "legend": {
-          "calcs": [
-            "mean",
-            "max"
-          ],
-          "displayMode": "table",
-          "placement": "bottom"
-        },
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ],
-          "fields": "",
-          "values": false
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "histogram_quantile(0.50, sum by (path, size_bucket, le) (rate(rustfs_s3_put_object_selected_buffer_size_bytes_bucket[$__rate_interval]))) or vector(0)",
-          "legendFormat": "p50 {{path}} / {{size_bucket}}",
-          "refId": "A"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "histogram_quantile(0.95, sum by (path, size_bucket, le) (rate(rustfs_s3_put_object_selected_buffer_size_bytes_bucket[$__rate_interval]))) or vector(0)",
-          "legendFormat": "p95 {{path}} / {{size_bucket}}",
-          "refId": "B"
-        }
-      ],
-      "title": "PUT Selected Buffer Size",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "percentunit",
-          "min": 0,
-          "max": 1,
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 15,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 0,
-        "y": 16
-      },
-      "id": 5,
-      "options": {
-        "legend": {
-          "calcs": [
-            "mean",
-            "max"
-          ],
-          "displayMode": "table",
-          "placement": "bottom"
-        },
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ],
-          "fields": "",
-          "values": false
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(rate(rustfs_s3_put_object_zero_copy_eligible_total[$__rate_interval])) / clamp_min(sum(rate(rustfs_s3_put_object_total[$__rate_interval])), 1) or vector(0)",
-          "legendFormat": "eligible / total",
-          "refId": "A"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(rate(rustfs_s3_put_object_zero_copy_enabled_total[$__rate_interval])) / clamp_min(sum(rate(rustfs_s3_put_object_total[$__rate_interval])), 1) or vector(0)",
-          "legendFormat": "enabled / total",
-          "refId": "B"
-        }
-      ],
-      "title": "PUT Zero-Copy Eligibility Ratio",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "ops",
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 10,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 12,
-        "y": 16
-      },
-      "id": 6,
-      "options": {
-        "legend": {
-          "calcs": [
-            "mean",
-            "max"
-          ],
-          "displayMode": "table",
-          "placement": "bottom"
-        },
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ],
-          "fields": "",
-          "values": false
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(rate(rustfs_s3_put_object_total[$__rate_interval])) or vector(0)",
-          "legendFormat": "PUT total",
-          "refId": "A"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum by (stage) (rate(rustfs_system_storage_erasure_write_quorum_failures_total[$__rate_interval])) or vector(0)",
-          "legendFormat": "write quorum failure {{stage}}",
-          "refId": "B"
-        }
-      ],
-      "title": "PUT Throughput and Quorum Failure Signals",
-      "type": "timeseries"
-    }
-  ],
-  "refresh": "30s",
-  "schemaVersion": 38,
-  "style": "dark",
-  "tags": [
-    "rustfs",
-    "put",
-    "performance",
-    "attribution"
-  ],
-  "templating": {
-    "list": [
-      {
-        "current": {
-          "selected": false,
-          "text": "Prometheus",
-          "value": "Prometheus"
-        },
-        "hide": 0,
-        "includeAll": false,
-        "multi": false,
-        "name": "DS_PROMETHEUS",
-        "options": [],
-        "query": "prometheus",
-        "refresh": 1,
-        "regex": "",
-        "skipUrlSync": false,
-        "type": "datasource"
-      }
-    ]
-  },
-  "time": {
-    "from": "now-1h",
-    "to": "now"
-  },
-  "timezone": "",
-  "title": "RustFS PUT Performance Attribution",
-  "uid": "rustfs-put-performance-attribution",
-  "version": 1,
-  "weekStart": ""
-}

.docker/observability/grafana/provisioning/dashboards/dashboard.yml

@@ -1,25 +0,0 @@
-# Copyright 2024 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-apiVersion: 1
-
-providers:
-  - name: "default"
-    orgId: 1
-    folder: ""
-    type: file
-    disableDeletion: false
-    updateIntervalSeconds: 10
-    options:
-      path: /etc/grafana/dashboards

.docker/observability/grafana/provisioning/datasources/datasources.yaml

@@ -1,98 +0,0 @@
-# Copyright 2024 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-apiVersion: 1
-
-datasources:
-  - name: Prometheus
-    type: prometheus
-    uid: prometheus
-    url: http://prometheus:9090
-    access: proxy
-    isDefault: true
-    editable: false
-    jsonData:
-      httpMethod: GET
-      exemplarTraceIdDestinations:
-        - name: trace_id
-          datasourceUid: tempo
-
-  - name: Tempo
-    type: tempo
-    uid: tempo
-    access: proxy
-    url: http://tempo:3200
-    isDefault: false
-    editable: false
-    jsonData:
-      httpMethod: GET
-      serviceMap:
-        datasourceUid: prometheus
-      tracesToLogs:
-        datasourceUid: loki
-        tags: [ 'job', 'instance', 'pod', 'namespace', 'service.name' ]
-        mappedTags: [ { key: 'service.name', value: 'app' } ]
-        spanStartTimeShift: '-1h'
-        spanEndTimeShift: '1h'
-        filterByTraceID: true
-        filterBySpanID: false
-      tracesToMetrics:
-        datasourceUid: prometheus
-        tags: [ { key: 'service.name' }, { key: 'job' } ]
-        queries:
-          - name: 'Service-Level Latency'
-            query: 'sum(rate(traces_spanmetrics_latency_bucket{$$__tags}[5m])) by (le)'
-          - name: 'Service-Level Calls'
-            query: 'sum(rate(traces_spanmetrics_calls_total{$$__tags}[5m]))'
-          - name: 'Service-Level Errors'
-            query: 'sum(rate(traces_spanmetrics_calls_total{status_code="ERROR", $$__tags}[5m]))'
-      nodeGraph:
-        enabled: true
-
-  - name: Loki
-    type: loki
-    uid: loki
-    url: http://loki:3100
-    basicAuth: false
-    isDefault: false
-    editable: false
-    jsonData:
-      derivedFields:
-        - datasourceUid: tempo
-          matcherRegex: 'trace_id=(\w+)'
-          name: 'TraceID'
-          url: '$${__value.raw}'
-
-  - name: Jaeger
-    type: jaeger
-    uid: jaeger
-    url: http://jaeger:16686
-    access: proxy
-    isDefault: false
-    editable: false
-    jsonData:
-      tracesToLogs:
-        datasourceUid: loki
-        tags: [ 'job', 'instance', 'pod', 'namespace', 'service.name' ]
-        mappedTags: [ { key: 'service.name', value: 'app' } ]
-        spanStartTimeShift: '1s'
-        spanEndTimeShift: '-1s'
-        filterByTraceID: true
-        filterBySpanID: false
-
-  - name: Pyroscope
-    type: grafana-pyroscope-datasource
-    url: http://pyroscope:4040
-    jsonData:
-      minStep: '15s'

.docker/observability/jaeger.yaml

@@ -1,74 +0,0 @@
-# Copyright 2024 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-service:
-  extensions: [jaeger_storage, jaeger_query]
-  pipelines:
-    traces:
-      receivers: [otlp]
-      processors: [batch]
-      exporters: [jaeger_storage_exporter, spanmetrics]
-    metrics/spanmetrics:
-      receivers: [spanmetrics]
-      exporters: [prometheus]
-  telemetry:
-    resource:
-      service.name: jaeger
-    metrics:
-      level: detailed
-      readers:
-        - pull:
-            exporter:
-              prometheus:
-                host: 0.0.0.0
-                port: 8888
-    logs:
-      level: DEBUG
-
-extensions:
-  jaeger_query:
-    storage:
-      traces: some_storage
-      metrics: some_metrics_storage
-  jaeger_storage:
-    backends:
-      some_storage:
-        memory:
-          max_traces: 100000
-    metric_backends:
-      some_metrics_storage:
-        prometheus:
-          endpoint: http://prometheus:9090
-          normalize_calls: true
-          normalize_duration: true
-
-connectors:
-  spanmetrics:
-
-receivers:
-  otlp:
-    protocols:
-      grpc:
-        endpoint: "0.0.0.0:4317"
-      http:
-        endpoint: "0.0.0.0:4318"
-
-processors:
-  batch:
-
-exporters:
-  jaeger_storage_exporter:
-    trace_storage: some_storage
-  prometheus:
-    endpoint: "0.0.0.0:8889"

.docker/observability/loki.yaml

@@ -1,63 +0,0 @@
-# Copyright 2024 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-auth_enabled: false
-
-server:
-  http_listen_port: 3100
-  grpc_listen_port: 9095
-  log_level: info
-  grpc_server_max_concurrent_streams: 1000
-
-common:
-  instance_addr: 127.0.0.1
-  path_prefix: /loki
-  storage:
-    filesystem:
-      chunks_directory: /loki/chunks
-      rules_directory: /loki/rules
-  replication_factor: 1
-  ring:
-    kvstore:
-      store: inmemory
-
-query_range:
-  results_cache:
-    cache:
-      embedded_cache:
-        enabled: true
-        max_size_mb: 100
-
-schema_config:
-  configs:
-    - from: 2020-10-24
-      store: tsdb
-      object_store: filesystem
-      schema: v13
-      index:
-        prefix: index_
-        period: 24h
-
-limits_config:
-  reject_old_samples: true
-  reject_old_samples_max_age: 168h
-  allow_structured_metadata: true
-  max_line_size: 256KB
-
-pattern_ingester:
-  enabled: true
-  metric_aggregation:
-    loki_address: localhost:3100
-
-frontend:
-  encoding: protobuf

.docker/observability/otel-collector-config.yaml

@@ -1,132 +0,0 @@
-# Copyright 2024 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-receivers:
-  otlp:
-    protocols:
-      grpc:
-        endpoint: 0.0.0.0:4317
-      http:
-        endpoint: 0.0.0.0:4318
-
-processors:
-  batch:
-    timeout: 1s
-    send_batch_size: 1024
-  memory_limiter:
-    check_interval: 1s
-    limit_mib: 1024
-    spike_limit_mib: 256
-  transform/logs:
-    error_mode: ignore
-    log_statements:
-      - context: log
-        statements:
-          - set(attributes["message"], body.string) where IsString(body)
-          - set(attributes["log.body"], body.string) where IsString(body)
-          - set(body, Concat([body, " event=", attributes["event"]], "")) where IsString(body) and attributes["event"] != nil
-          - set(body, Concat([body, " component=", attributes["component"]], "")) where IsString(body) and attributes["component"] != nil
-          - set(body, Concat([body, " subsystem=", attributes["subsystem"]], "")) where IsString(body) and attributes["subsystem"] != nil
-          - set(body, Concat([body, " state=", attributes["state"]], "")) where IsString(body) and attributes["state"] != nil
-          - set(body, Concat([body, " result=", attributes["result"]], "")) where IsString(body) and attributes["result"] != nil
-          - set(body, Concat([body, " reason=", attributes["reason"]], "")) where IsString(body) and attributes["reason"] != nil
-          - set(body, Concat([body, " failure_reason=", attributes["failure_reason"]], "")) where IsString(body) and attributes["failure_reason"] != nil
-          - set(body, Concat([body, " rpc_path=", attributes["rpc_path"]], "")) where IsString(body) and attributes["rpc_path"] != nil
-          - set(body, Concat([body, " rpc_service=", attributes["rpc_service"]], "")) where IsString(body) and attributes["rpc_service"] != nil
-          - set(body, Concat([body, " rpc_method=", attributes["rpc_method"]], "")) where IsString(body) and attributes["rpc_method"] != nil
-          - set(body, Concat([body, " expected_audience=", attributes["expected_audience"]], "")) where IsString(body) and attributes["expected_audience"] != nil
-          - set(body, Concat([body, " peer_addr=", attributes["peer_addr"]], "")) where IsString(body) and attributes["peer_addr"] != nil
-          - set(body, Concat([body, " replay_scope_bootstrap_allowed=", attributes["replay_scope_bootstrap_allowed"]], "")) where IsString(body) and attributes["replay_scope_bootstrap_allowed"] != nil
-          - set(body, Concat([body, " error=", attributes["error"]], "")) where IsString(body) and attributes["error"] != nil
-          - set(body, Concat([body, " exception_message=", attributes["exception.message"]], "")) where IsString(body) and attributes["exception.message"] != nil
-
-exporters:
-  otlp/tempo:
-    endpoint: "tempo:4317"
-    tls:
-      insecure: true
-    compression: gzip
-    retry_on_failure:
-      enabled: true
-      initial_interval: 1s
-      max_interval: 30s
-      max_elapsed_time: 300s
-    sending_queue:
-      enabled: true
-      num_consumers: 10
-      queue_size: 5000
-
-  otlp/jaeger:
-    endpoint: "jaeger:4317"
-    tls:
-      insecure: true
-    compression: gzip
-    retry_on_failure:
-      enabled: true
-      initial_interval: 1s
-      max_interval: 30s
-      max_elapsed_time: 300s
-    sending_queue:
-      enabled: true
-      num_consumers: 10
-      queue_size: 5000
-
-  prometheus:
-    endpoint: "0.0.0.0:8889"
-    send_timestamps: true
-    metric_expiration: 5m
-    resource_to_telemetry_conversion:
-      enabled: true
-
-  otlphttp/loki:
-    endpoint: "http://loki:3100/otlp"
-    tls:
-      insecure: true
-    compression: gzip
-
-extensions:
-  health_check:
-    endpoint: 0.0.0.0:13133
-  pprof:
-    endpoint: 0.0.0.0:1888
-  zpages:
-    endpoint: 0.0.0.0:55679
-
-service:
-  extensions: [ health_check, pprof, zpages ]
-  pipelines:
-    traces:
-      receivers: [ otlp ]
-      processors: [ memory_limiter, batch ]
-      exporters: [ otlp/tempo, otlp/jaeger ]
-    metrics:
-      receivers: [ otlp ]
-      processors: [ batch ]
-      exporters: [ prometheus ]
-    logs:
-      receivers: [ otlp ]
-      processors: [ batch, transform/logs ]
-      exporters: [ otlphttp/loki ]
-  telemetry:
-    logs:
-      level: "info"
-      encoding: "json"
-    metrics:
-      level: "normal"
-      readers:
-        - pull:
-            exporter:
-              prometheus:
-                host: '0.0.0.0'
-                port: 8888

.docker/observability/prometheus-rules/rustfs-dashboard.yml

@@ -1,53 +0,0 @@
-groups:
-  - name: rustfs-dashboard
-    interval: 30s
-    rules:
-      - record: rustfs:http_server_requests:rate5m
-        expr: sum by (job) (rate(rustfs_http_server_requests_total[5m]))
-
-      - record: rustfs:http_server_request_duration_seconds:p50_5m
-        expr: histogram_quantile(0.50, sum by (le, job) (rate(rustfs_http_server_request_duration_seconds_bucket[5m])))
-      - record: rustfs:http_server_request_duration_seconds:p95_5m
-        expr: histogram_quantile(0.95, sum by (le, job) (rate(rustfs_http_server_request_duration_seconds_bucket[5m])))
-      - record: rustfs:http_server_request_duration_seconds:p99_5m
-        expr: histogram_quantile(0.99, sum by (le, job) (rate(rustfs_http_server_request_duration_seconds_bucket[5m])))
-
-      - record: rustfs:http_server_response_body_size_bytes:p50_5m
-        expr: histogram_quantile(0.50, sum by (le, job) (rate(rustfs_http_server_response_body_size_bytes_bucket[5m])))
-      - record: rustfs:http_server_response_body_size_bytes:p95_5m
-        expr: histogram_quantile(0.95, sum by (le, job) (rate(rustfs_http_server_response_body_size_bytes_bucket[5m])))
-      - record: rustfs:http_server_response_body_size_bytes:p99_5m
-        expr: histogram_quantile(0.99, sum by (le, job) (rate(rustfs_http_server_response_body_size_bytes_bucket[5m])))
-
-      - record: rustfs:log_cleaner_runs:rate15m
-        expr: sum by (job) (rate(rustfs_log_cleaner_runs_total[15m]))
-      - record: rustfs:log_cleaner_failure_ratio:rate5m
-        expr: sum by (job) (rate(rustfs_log_cleaner_run_failures_total[5m])) / clamp_min(sum by (job) (rate(rustfs_log_cleaner_runs_total[5m])), 1e-9)
-      - record: rustfs:log_cleaner_rotation_failure_ratio:rate5m
-        expr: sum by (job) (rate(rustfs_log_cleaner_rotation_failures_total[5m])) / clamp_min(sum by (job) (rate(rustfs_log_cleaner_rotation_total[5m])), 1e-9)
-      - record: rustfs:log_cleaner_rotation_duration_seconds:p95_5m
-        expr: histogram_quantile(0.95, sum by (le, job) (rate(rustfs_log_cleaner_rotation_duration_seconds_bucket[5m])))
-      - record: rustfs:log_cleaner_compress_duration_seconds:p95_5m
-        expr: histogram_quantile(0.95, sum by (le, job) (rate(rustfs_log_cleaner_compress_duration_seconds_bucket[5m])))
-
-      - record: rustfs:scanner_objects_scanned:rate5m
-        expr: sum by (job) (rate(rustfs_scanner_objects_scanned_total[5m]))
-      - record: rustfs:scanner_directories_scanned:rate5m
-        expr: sum by (job) (rate(rustfs_scanner_directories_scanned_total[5m]))
-      - record: rustfs:scanner_buckets_scanned:rate5m
-        expr: sum by (job) (rate(rustfs_scanner_buckets_scanned_total[5m]))
-      - record: rustfs:scanner_cycles_success:rate5m
-        expr: sum by (job) (rate(rustfs_scanner_cycles_total{result="success"}[5m]))
-
-      - record: rustfs:log_chain_op_event_mismatch:rate5m
-        expr: sum by (job) (rate(rustfs_log_chain_op_event_mismatch_total[5m]))
-
-      - alert: RustFSLogChainOpEventMismatchDetected
-        expr: rustfs:log_chain_op_event_mismatch:rate5m > 0
-        for: 10m
-        labels:
-          severity: warning
-          component: s3-log-chain
-        annotations:
-          summary: "RustFS log-chain op/event mismatch detected"
-          description: "job={{ $labels.job }} has non-zero rustfs_log_chain_op_event_mismatch_total rate for more than 10m. Check s3 op/event mapping changes."

.docker/observability/prometheus-rules/rustfs-get-optimization-alerts.yaml

@@ -1,260 +0,0 @@
-# =============================================================================
-# RustFS GET Optimization — Prometheus Alerting Rules
-# =============================================================================
-#
-# Import into Prometheus:
-#   1. Copy this file to your Prometheus rules directory
-#   2. Add to prometheus.yml:
-#        rule_files:
-#          - "prometheus-alert-rules.yaml"
-#   3. Validate: promtool check rules prometheus-alert-rules.yaml
-#   4. Reload:   curl -X POST http://localhost:9090/-/reload
-#
-# All metric names match those registered in crates/io-metrics/src/lib.rs
-# and documented in crates/ecstore/src/diagnostics/get.rs.
-#
-# Baseline comparison uses "offset 1d" — adjust to "offset 7d" for weekly
-# seasonality if your traffic pattern varies by day of week.
-# =============================================================================
-
-groups:
-  # ==========================================================================
-  # Critical alerts — immediate action required
-  # ==========================================================================
-  - name: rustfs-get-optimization-critical
-    interval: 30s
-    rules:
-      # ------------------------------------------------------------------
-      # 1. GetP99Regression
-      #    GET p99 latency exceeds 2x the baseline (same time yesterday)
-      #    sustained for 10 minutes.
-      #    Action: Roll back the GET optimization immediately.
-      # ------------------------------------------------------------------
-      - alert: GetP99Regression
-        expr: |
-          histogram_quantile(0.99,
-            sum(rate(rustfs_io_get_object_total_duration_seconds_bucket[5m])) by (le)
-          )
-          >
-          2
-          *
-          histogram_quantile(0.99,
-            sum(rate(rustfs_io_get_object_total_duration_seconds_bucket[5m] offset 1d)) by (le)
-          )
-        for: 10m
-        labels:
-          severity: critical
-          team: rustfs-storage
-          area: get-optimization
-        annotations:
-          summary: "GET p99 latency regression detected (>2x baseline for 10m)"
-          description: >-
-            The 99th-percentile GET object latency is {{ $value | humanizeDuration }}
-            which is more than double the baseline measured 24 hours ago.
-            This indicates a severe performance regression introduced by
-            a recent GET optimization change.
-          runbook_url: "https://internal.wiki/runbooks/rustfs/get-p99-regression"
-          action: >
-            1. Verify the regression is not caused by external factors
-               (disk health, network, load spike).
-            2. If confirmed optimization-related, roll back:
-               - Set RUSTFS_GET_CODEC_STREAMING=0
-               - Set RUSTFS_GET_METADATA_EARLY_STOP=0
-               - Restart affected nodes.
-            3. Collect flamegraphs and open a P0 incident.
-
-      # ------------------------------------------------------------------
-      # 2. PipelineFailureSpike
-      #    Pipeline failure rate exceeds 5x the baseline sustained for
-      #    5 minutes.  Covers all failure reasons: bitrot_mismatch,
-      #    decode_error, downstream_closed, io, read_quorum, timeout, etc.
-      #    Action: Investigate pipeline health and roll back if needed.
-      # ------------------------------------------------------------------
-      - alert: PipelineFailureSpike
-        expr: |
-          sum(rate(rustfs_io_get_object_pipeline_failures_total[5m]))
-          >
-          5
-          *
-          sum(rate(rustfs_io_get_object_pipeline_failures_total[5m] offset 1d))
-        for: 5m
-        labels:
-          severity: critical
-          team: rustfs-storage
-          area: get-optimization
-        annotations:
-          summary: "GET pipeline failure rate spike (>5x baseline for 5m)"
-          description: >-
-            The GET pipeline failure rate is {{ $value | printf "%.2f" }}/s,
-            more than 5x the baseline from 24 hours ago.
-            Failure reasons may include: bitrot_mismatch, decode_error,
-            downstream_closed, io, range_or_length_invalid, read_quorum,
-            short_read, timeout, unknown.
-          runbook_url: "https://internal.wiki/runbooks/rustfs/pipeline-failure-spike"
-          action: >
-            1. Check Grafana "GET Data Integrity" dashboard for failure
-               breakdown by reason label.
-            2. If decode_error or bitrot_mismatch dominates, stop
-               optimization and investigate data integrity.
-            3. If io or timeout dominates, check disk and network health.
-            4. Roll back optimization if failures persist.
-
-      # ------------------------------------------------------------------
-      # 3. BitrotMismatchSpike
-      #    Bitrot verification mismatch rate exceeds 3x baseline for
-      #    5 minutes.  This is a data-integrity signal — shard checksums
-      #    do not match after read.
-      #
-      #    The "bitrot_mismatch" reason is recorded on the
-      #    rustfs_io_get_object_pipeline_failures_total counter when a
-      #    StorageError::FileCorrupt or DiskError::FileCorrupt /
-      #    DiskError::PartMissingOrCorrupt is classified during the GET
-      #    pipeline (see classify_storage_error / classify_disk_error in
-      #    crates/ecstore/src/diagnostics/get.rs).
-      #
-      #    Action: Stop optimization, investigate data integrity urgently.
-      # ------------------------------------------------------------------
-      - alert: BitrotMismatchSpike
-        expr: |
-          sum(rate(rustfs_io_get_object_pipeline_failures_total{reason="bitrot_mismatch"}[5m]))
-          >
-          3
-          *
-          sum(rate(rustfs_io_get_object_pipeline_failures_total{reason="bitrot_mismatch"}[5m] offset 1d))
-        for: 5m
-        labels:
-          severity: critical
-          team: rustfs-storage
-          area: get-optimization
-        annotations:
-          summary: "Bitrot mismatch rate spike (>3x baseline for 5m)"
-          description: >-
-            The rate of pipeline failures classified as bitrot_mismatch is
-            {{ $value | printf "%.2f" }}/s, more than 3x the baseline from
-            24 hours ago.  This indicates shard checksum verification
-            failures (FileCorrupt / PartMissingOrCorrupt) which may point
-            to data corruption introduced by the GET optimization pipeline
-            (e.g., incorrect decode, buffer reuse bug).
-          runbook_url: "https://internal.wiki/runbooks/rustfs/bitrot-mismatch-spike"
-          action: >
-            1. Immediately disable codec streaming:
-               RUSTFS_GET_CODEC_STREAMING=0
-            2. Run "mc admin scan" on affected buckets to verify on-disk
-               integrity independent of the GET path.
-            3. Compare xl.meta checksums across erasure shards.
-            4. If corruption confirmed, initiate data recovery from parity.
-            5. Do NOT re-enable optimization until root cause is identified.
-
-  # ==========================================================================
-  # Warning alerts — investigation needed
-  # ==========================================================================
-  - name: rustfs-get-optimization-warning
-    interval: 30s
-    rules:
-      # ------------------------------------------------------------------
-      # 4. EarlyStopInsufficientQuorum
-      #    The metadata early-stop path is hitting "insufficient_quorum"
-      #    at a rate above 0.1/s for 5 minutes.  This means too many
-      #    disks are failing to return valid metadata in time.
-      #    Action: Check disk health and metadata fanout latency.
-      # ------------------------------------------------------------------
-      - alert: EarlyStopInsufficientQuorum
-        expr: |
-          sum(rate(rustfs_io_get_object_metadata_early_stop_total{reason="insufficient_quorum"}[5m]))
-          > 0.1
-        for: 5m
-        labels:
-          severity: warning
-          team: rustfs-storage
-          area: get-optimization
-        annotations:
-          summary: "Early-stop insufficient quorum rate elevated (>0.1/s for 5m)"
-          description: >-
-            The metadata early-stop path is returning "insufficient_quorum"
-            at {{ $value | printf "%.3f" }}/s.  This means the bounded
-            metadata fanout cannot gather enough valid responses before
-            the quorum deadline, suggesting disk or network issues.
-          runbook_url: "https://internal.wiki/runbooks/rustfs/early-stop-quorum"
-          action: >
-            1. Check disk health: mc admin info --json | jq '.disks'
-            2. Review rustfs_io_get_object_metadata_response_total by
-               outcome (error, timeout, disk_not_found) in Grafana.
-            3. Check rustfs_io_disk_permit_wait_duration_seconds for
-               I/O scheduler saturation.
-            4. If disks are healthy, consider increasing the early-stop
-               timeout or temporarily disabling early-stop.
-
-      # ------------------------------------------------------------------
-      # 5. CodecStreamingFallbackSpike
-      #    The codec streaming fallback rate is >10x baseline for 10
-      #    minutes.  This means the optimized codec streaming path is
-      #    being bypassed much more often than expected.
-      #    Action: Check fallback reasons and object eligibility.
-      # ------------------------------------------------------------------
-      - alert: CodecStreamingFallbackSpike
-        expr: |
-          sum(rate(rustfs_io_get_object_codec_streaming_fallback_total[5m]))
-          >
-          10
-          *
-          sum(rate(rustfs_io_get_object_codec_streaming_fallback_total[5m] offset 1d))
-        for: 10m
-        labels:
-          severity: warning
-          team: rustfs-storage
-          area: get-optimization
-        annotations:
-          summary: "Codec streaming fallback rate spike (>10x baseline for 10m)"
-          description: >-
-            The codec streaming fallback rate is {{ $value | printf "%.2f" }}/s,
-            more than 10x the baseline from 24 hours ago.  Fallback reasons
-            are labeled by "reason" — check Grafana for breakdown.
-            Common reasons: object too small, multipart not supported,
-            unsupported erasure layout, feature flag disabled.
-          runbook_url: "https://internal.wiki/runbooks/rustfs/codec-fallback-spike"
-          action: >
-            1. Query by reason label:
-               sum by (reason) (rate(rustfs_io_get_object_codec_streaming_fallback_total[5m]))
-            2. If dominated by a single reason, investigate why that
-               condition became more frequent (e.g., workload change,
-               configuration drift).
-            3. Cross-reference with rustfs_io_get_object_reader_path_total
-               to verify the fallback path (legacy_duplex) is healthy.
-            4. If fallback is expected (e.g., workload shifted to small
-               objects), update the alert baseline.
-
-      # ------------------------------------------------------------------
-      # 6. IoQueueSaturation
-      #    I/O queue utilization exceeds 90% for 5 minutes.  High
-      #    utilization causes disk permit wait latency to increase and
-      #    can cascade into pipeline timeouts.
-      #    Action: Check disk load and consider reducing concurrency.
-      # ------------------------------------------------------------------
-      - alert: IoQueueSaturation
-        expr: |
-          rustfs_io_queue_utilization_percent > 90
-        for: 5m
-        labels:
-          severity: warning
-          team: rustfs-storage
-          area: get-optimization
-        annotations:
-          summary: "I/O queue utilization >90% for 5m"
-          description: >-
-            The I/O queue utilization is {{ $value | printf "%.1f" }}%,
-            sustained above 90% for 5 minutes.  This indicates the disk
-            I/O scheduler is near saturation, which will increase
-            rustfs_io_disk_permit_wait_duration_seconds and may trigger
-            pipeline timeouts.
-          runbook_url: "https://internal.wiki/runbooks/rustfs/io-queue-saturation"
-          action: >
-            1. Check disk I/O metrics (iostat, node_exporter) for
-               individual disk saturation.
-            2. Review rustfs_io_queue_permits_in_use vs
-               rustfs_io_queue_permits_available for permit exhaustion.
-            3. Check rustfs_io_starvation_events for priority starvation.
-            4. If GET optimization increased concurrency, consider:
-               - Reducing RUSTFS_GET_PIPELINE_PARALLELISM
-               - Lowering RUSTFS_IO_QUEUE_PERMITS
-            5. If caused by background operations (ILM, healing), throttle
-               those before adjusting GET concurrency.

.docker/observability/prometheus-rules/rustfs-kms-alerts.yml

@@ -1,251 +0,0 @@
-# Copyright 2024 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-# =============================================================================
-# RustFS KMS backend — Prometheus alerting rules
-# =============================================================================
-#
-# Metric source: the KMS operation-policy choke point in
-# crates/kms/src/policy.rs, except KmsKeyRotationOverdue, which reads the
-# label-less key-lifecycle gauge published by the deletion worker's sweep
-# (crates/kms/src/deletion_worker.rs). All label values are bounded static
-# strings (operation, op_class, outcome, error_class, backend, scope); key
-# identifiers, key material, and tokens never appear in labels.
-#
-# Response procedures: docs/operations/kms-observability-runbook.md
-#
-# IMPORTANT — threshold status: every numeric threshold below is a
-# conservative default chosen without a production baseline. Calibrate against
-# a staging baseline before relying on these alerts for paging, and prefer
-# loosening over tightening until the baseline exists. Formal SLO targets are
-# deliberately not encoded here (see rustfs/backlog#1584).
-#
-# NOTE: prometheus.yml loads /etc/prometheus/rules/*.yml — keep the .yml
-# extension or the file is silently ignored by the docker-compose stack.
-#
-# Validate: promtool check rules rustfs-kms-alerts.yml
-# =============================================================================
-
-groups:
-  # ==========================================================================
-  # Critical alerts — immediate action required
-  # ==========================================================================
-  - name: rustfs-kms-critical
-    interval: 30s
-    rules:
-      # ------------------------------------------------------------------
-      # 1. KmsBackendFatalErrors
-      #    Any attempt failure classified as fatal (non-retryable): auth
-      #    or permission errors, malformed requests, missing keys. The
-      #    policy never retries these, so even a low rate means real
-      #    operations are failing right now.
-      # ------------------------------------------------------------------
-      - alert: KmsBackendFatalErrors
-        expr: |
-          sum by (operation) (rate(rustfs_kms_backend_attempt_failures_total{error_class="fatal"}[5m])) > 0
-        for: 5m
-        labels:
-          severity: critical
-          component: kms
-        annotations:
-          summary: "KMS backend fatal errors on operation {{ $labels.operation }}"
-          description: >-
-            Attempt failures classified as fatal are occurring at
-            {{ $value | printf "%.3f" }}/s on operation
-            {{ $labels.operation }}. Fatal failures are not retried:
-            each one is a KMS backend call that failed permanently
-            (authentication, permissions, malformed request, or a
-            missing key/version).
-          runbook_url: "https://github.com/rustfs/rustfs/blob/main/docs/operations/kms-observability-runbook.md#kmsbackendfatalerrors"
-
-      # ------------------------------------------------------------------
-      # 2. KmsBackendHighErrorRate
-      #    Sustained share of operations terminating without success
-      #    (fatal, budget/deadline exhaustion, admission backpressure,
-      #    or an open circuit). The cancelled outcome is excluded because
-      #    shutdowns legitimately produce it.
-      #    The traffic guard keeps a single failure on a near-idle
-      #    cluster from firing the alert.
-      #    Threshold: 5% for 10m — conservative default, calibrate
-      #    against a staging baseline.
-      # ------------------------------------------------------------------
-      - alert: KmsBackendHighErrorRate
-        expr: |
-          (
-            sum(rate(rustfs_kms_backend_operations_total{outcome!~"success|cancelled"}[5m]))
-            /
-            clamp_min(sum(rate(rustfs_kms_backend_operations_total[5m])), 1e-9)
-          ) > 0.05
-          and
-          sum(rate(rustfs_kms_backend_operations_total[5m])) > 0.02
-        for: 10m
-        labels:
-          severity: critical
-          component: kms
-        annotations:
-          summary: "KMS backend non-success ratio above 5% for 10m"
-          description: >-
-            {{ $value | humanizePercentage }} of KMS backend operations
-            are terminating in fatal, budget_exhausted,
-            deadline_exceeded, backpressure_timeout,
-            backpressure_rejected, or circuit_open. Object encryption
-            and decryption paths depending on the KMS are degraded or
-            failing.
-          runbook_url: "https://github.com/rustfs/rustfs/blob/main/docs/operations/kms-observability-runbook.md#kmsbackendhigherrorrate"
-
-  # ==========================================================================
-  # Warning alerts — investigation needed
-  # ==========================================================================
-  - name: rustfs-kms-warning
-    interval: 30s
-    rules:
-      # ------------------------------------------------------------------
-      # 3. KmsBackendP99LatencyHigh
-      #    p99 wall-clock duration of whole operations (attempts plus
-      #    backoff) is sustained above 2 seconds. Because the histogram
-      #    includes retries, a high p99 usually means the retry policy
-      #    is absorbing backend failures, not that every call is slow.
-      #    Threshold: 2s for 10m — conservative default, calibrate
-      #    against a staging baseline.
-      # ------------------------------------------------------------------
-      - alert: KmsBackendP99LatencyHigh
-        expr: |
-          histogram_quantile(0.99,
-            sum by (le) (rate(rustfs_kms_backend_operation_duration_seconds_bucket[5m]))
-          ) > 2
-        for: 10m
-        labels:
-          severity: warning
-          component: kms
-        annotations:
-          summary: "KMS backend operation p99 latency above 2s for 10m"
-          description: >-
-            The 99th-percentile KMS backend operation duration is
-            {{ $value | humanizeDuration }}, including retries and
-            backoff. Encryption and decryption latency is leaking into
-            S3 request latency.
-          runbook_url: "https://github.com/rustfs/rustfs/blob/main/docs/operations/kms-observability-runbook.md#kmsbackendp99latencyhigh"
-
-      # ------------------------------------------------------------------
-      # 4. KmsBackendAttemptFailureSpike
-      #    Aggregate attempt-failure rate (all error classes) sustained
-      #    above an absolute floor. An absolute threshold is used instead
-      #    of an offset-1d baseline ratio because fresh deployments have
-      #    no baseline and an empty offset vector would keep a ratio
-      #    alert from ever firing; switch to a baseline-relative form
-      #    (see rustfs-get-optimization-alerts.yaml for the pattern)
-      #    once a stable staging baseline exists.
-      #    Threshold: 0.5/s for 10m — conservative default, calibrate
-      #    against a staging baseline.
-      # ------------------------------------------------------------------
-      - alert: KmsBackendAttemptFailureSpike
-        expr: |
-          sum(rate(rustfs_kms_backend_attempt_failures_total[5m])) > 0.5
-        for: 10m
-        labels:
-          severity: warning
-          component: kms
-        annotations:
-          summary: "KMS backend attempt failures above 0.5/s for 10m"
-          description: >-
-            KMS backend attempts are failing at
-            {{ $value | printf "%.2f" }}/s across all error classes.
-            The retry policy may still be masking these from callers —
-            check the error-class breakdown before it stops absorbing
-            them.
-          runbook_url: "https://github.com/rustfs/rustfs/blob/main/docs/operations/kms-observability-runbook.md#kmsbackendattemptfailurespike"
-
-      # ------------------------------------------------------------------
-      # 5. KmsBackendRetryBudgetExhausted
-      #    Operations are running out of retry budget (budget_exhausted)
-      #    or operation deadline (deadline_exceeded). These surface to
-      #    callers as failed KMS operations even though every individual
-      #    failure was retryable — the backend is unhealthy for longer
-      #    than the policy can bridge.
-      #    Threshold: 0.05/s for 10m — conservative default, calibrate
-      #    against a staging baseline.
-      # ------------------------------------------------------------------
-      - alert: KmsBackendRetryBudgetExhausted
-        expr: |
-          sum by (outcome) (rate(rustfs_kms_backend_operations_total{outcome=~"budget_exhausted|deadline_exceeded"}[5m])) > 0.05
-        for: 10m
-        labels:
-          severity: warning
-          component: kms
-        annotations:
-          summary: "KMS backend operations exhausting retry budget ({{ $labels.outcome }})"
-          description: >-
-            KMS backend operations are terminating as
-            {{ $labels.outcome }} at {{ $value | printf "%.3f" }}/s.
-            Retryable failures are outlasting the retry budget, so
-            callers are seeing hard failures.
-          runbook_url: "https://github.com/rustfs/rustfs/blob/main/docs/operations/kms-observability-runbook.md#kmsbackendretrybudgetexhausted"
-
-      # ------------------------------------------------------------------
-      # 6. KmsBackendCircuitOpen
-      #    Direct circuit-state signal, independent of operation traffic.
-      #    A transient open can recover on its first half-open probe; alert
-      #    only when the circuit remains open or half-open for one minute.
-      # ------------------------------------------------------------------
-      - alert: KmsBackendCircuitOpen
-        expr: |
-          rustfs_kms_backend_circuit_open > 0
-        for: 1m
-        labels:
-          severity: warning
-          component: kms
-        annotations:
-          summary: "KMS backend circuit open ({{ $labels.backend }}/{{ $labels.scope }})"
-          description: >-
-            The KMS backend circuit for {{ $labels.backend }} scope
-            {{ $labels.scope }} has remained open or half-open for one
-            minute. Operations in this scope can terminate as
-            circuit_open until the half-open probe succeeds or returns
-            a non-retryable failure.
-          runbook_url: "https://github.com/rustfs/rustfs/blob/main/docs/operations/kms-observability-runbook.md#kmsbackendcircuitopen"
-
-      # ------------------------------------------------------------------
-      # 7. KmsKeyRotationOverdue
-      #    The least recently rotated usable key has gone more than 400
-      #    days without a rotation (measured from creation for keys with
-      #    no recorded rotation). Direct gauge state published by the
-      #    deletion worker's sweep, so no traffic guard applies; the
-      #    one-hour hold only bridges scrape gaps. The worker runs only
-      #    on backends with the schedule_deletion capability, so on the
-      #    Static backend the series never exists and this alert cannot
-      #    fire — that backend cannot rotate either; see the rotation
-      #    driver matrix in docs/operations/kms-backend-security.md.
-      #    Threshold: 400 days — conservative default sitting above a
-      #    one-year rotation policy. Align it with the rotation period
-      #    your compliance policy requires, and with
-      #    RUSTFS_KMS_ROTATION_MAX_AGE_SECS so the per-key rotation_due
-      #    verdict and this aggregate alert agree.
-      # ------------------------------------------------------------------
-      - alert: KmsKeyRotationOverdue
-        expr: |
-          rustfs_kms_oldest_key_rotation_age_seconds > (400 * 86400)
-        for: 1h
-        labels:
-          severity: warning
-          component: kms
-        annotations:
-          summary: "Oldest KMS key unrotated for more than 400 days"
-          description: >-
-            The least recently rotated usable KMS key was last rotated
-            {{ $value | humanizeDuration }} ago (measured from creation
-            for keys with no recorded rotation). List keys through the
-            admin API and read rotation_due / rotation_due_reason for
-            the per-key verdict; an "unsupported" reason means the
-            backend cannot rotate at all.
-          runbook_url: "https://github.com/rustfs/rustfs/blob/main/docs/operations/kms-observability-runbook.md#kmskeyrotationoverdue"

.docker/observability/prometheus-rules/rustfs-storage.yml

@@ -1,575 +0,0 @@
-# Preserve OTLP timestamps in the Collector. Apply timestamp() directly to
-# each raw selector, before label rewriting; functions such as label_replace()
-# replace the evaluation timestamp and would make a cached value look new.
-# Compare publication times at Prometheus millisecond precision so points
-# published and exported within the same millisecond are not withheld.
-groups:
-  - name: rustfs-storage-snapshots
-    interval: 15s
-    rules:
-      - record: rustfs:storage_snapshot:fresh
-        expr: |
-          rustfs_storage_snapshot_last_success_timestamp_seconds{rustfs_cluster_id!="",collection_scope=~"local|cluster"}
-          and ((time() - rustfs_storage_snapshot_last_success_timestamp_seconds{rustfs_cluster_id!="",collection_scope=~"local|cluster"}) <= rustfs_storage_snapshot_max_age_seconds{rustfs_cluster_id!="",collection_scope=~"local|cluster"})
-          and ((time() - rustfs_storage_snapshot_last_success_timestamp_seconds{rustfs_cluster_id!="",collection_scope=~"local|cluster"}) >= 0)
-          and (rustfs_storage_snapshot_max_age_seconds{rustfs_cluster_id!="",collection_scope=~"local|cluster"} > 0)
-      - record: rustfs:storage:current
-        labels:
-          source_metric: rustfs_cluster_buckets_total
-        expr: |
-          rustfs_cluster_buckets_total{rustfs_cluster_id!="",collection_scope="cluster"}
-          and (timestamp(rustfs_cluster_buckets_total{rustfs_cluster_id!="",collection_scope="cluster"})
-            >= on (rustfs_cluster_id, observer, collection_scope, job, instance) group_left()
-            (floor(rustfs:storage_snapshot:fresh * 1000) / 1000))
-      - record: rustfs:storage:current
-        labels:
-          source_metric: rustfs_cluster_capacity_free_bytes
-        expr: |
-          rustfs_cluster_capacity_free_bytes{rustfs_cluster_id!="",collection_scope="cluster"}
-          and (timestamp(rustfs_cluster_capacity_free_bytes{rustfs_cluster_id!="",collection_scope="cluster"})
-            >= on (rustfs_cluster_id, observer, collection_scope, job, instance) group_left()
-            (floor(rustfs:storage_snapshot:fresh * 1000) / 1000))
-      - record: rustfs:storage:current
-        labels:
-          source_metric: rustfs_cluster_capacity_missing_drives
-        expr: |
-          rustfs_cluster_capacity_missing_drives{rustfs_cluster_id!="",collection_scope="cluster"}
-          and (timestamp(rustfs_cluster_capacity_missing_drives{rustfs_cluster_id!="",collection_scope="cluster"})
-            >= on (rustfs_cluster_id, observer, collection_scope, job, instance) group_left()
-            (floor(rustfs:storage_snapshot:fresh * 1000) / 1000))
-      - record: rustfs:storage:current
-        labels:
-          source_metric: rustfs_cluster_capacity_raw_total_bytes
-        expr: |
-          rustfs_cluster_capacity_raw_total_bytes{rustfs_cluster_id!="",collection_scope="cluster"}
-          and (timestamp(rustfs_cluster_capacity_raw_total_bytes{rustfs_cluster_id!="",collection_scope="cluster"})
-            >= on (rustfs_cluster_id, observer, collection_scope, job, instance) group_left()
-            (floor(rustfs:storage_snapshot:fresh * 1000) / 1000))
-      - record: rustfs:storage:current
-        labels:
-          source_metric: rustfs_cluster_capacity_stale_drives
-        expr: |
-          rustfs_cluster_capacity_stale_drives{rustfs_cluster_id!="",collection_scope="cluster"}
-          and (timestamp(rustfs_cluster_capacity_stale_drives{rustfs_cluster_id!="",collection_scope="cluster"})
-            >= on (rustfs_cluster_id, observer, collection_scope, job, instance) group_left()
-            (floor(rustfs:storage_snapshot:fresh * 1000) / 1000))
-      - record: rustfs:storage:current
-        labels:
-          source_metric: rustfs_cluster_capacity_usable_total_bytes
-        expr: |
-          rustfs_cluster_capacity_usable_total_bytes{rustfs_cluster_id!="",collection_scope="cluster"}
-          and (timestamp(rustfs_cluster_capacity_usable_total_bytes{rustfs_cluster_id!="",collection_scope="cluster"})
-            >= on (rustfs_cluster_id, observer, collection_scope, job, instance) group_left()
-            (floor(rustfs:storage_snapshot:fresh * 1000) / 1000))
-      - record: rustfs:storage:current
-        labels:
-          source_metric: rustfs_cluster_capacity_used_bytes
-        expr: |
-          rustfs_cluster_capacity_used_bytes{rustfs_cluster_id!="",collection_scope="cluster"}
-          and (timestamp(rustfs_cluster_capacity_used_bytes{rustfs_cluster_id!="",collection_scope="cluster"})
-            >= on (rustfs_cluster_id, observer, collection_scope, job, instance) group_left()
-            (floor(rustfs:storage_snapshot:fresh * 1000) / 1000))
-      - record: rustfs:storage:current
-        labels:
-          source_metric: rustfs_cluster_drive_capacity_observation_age_seconds
-        expr: |
-          rustfs_cluster_drive_capacity_observation_age_seconds{rustfs_cluster_id!="",collection_scope="cluster"}
-          and (timestamp(rustfs_cluster_drive_capacity_observation_age_seconds{rustfs_cluster_id!="",collection_scope="cluster"})
-            >= on (rustfs_cluster_id, observer, collection_scope, job, instance) group_left()
-            (floor(rustfs:storage_snapshot:fresh * 1000) / 1000))
-      - record: rustfs:storage:current
-        labels:
-          source_metric: rustfs_cluster_drive_capacity_observation_state
-        expr: |
-          rustfs_cluster_drive_capacity_observation_state{rustfs_cluster_id!="",collection_scope="cluster"}
-          and (timestamp(rustfs_cluster_drive_capacity_observation_state{rustfs_cluster_id!="",collection_scope="cluster"})
-            >= on (rustfs_cluster_id, observer, collection_scope, job, instance) group_left()
-            (floor(rustfs:storage_snapshot:fresh * 1000) / 1000))
-      - record: rustfs:storage:current
-        labels:
-          source_metric: rustfs_cluster_drive_free_bytes
-        expr: |
-          rustfs_cluster_drive_free_bytes{rustfs_cluster_id!="",collection_scope="cluster"}
-          and (timestamp(rustfs_cluster_drive_free_bytes{rustfs_cluster_id!="",collection_scope="cluster"})
-            >= on (rustfs_cluster_id, observer, collection_scope, job, instance) group_left()
-            (floor(rustfs:storage_snapshot:fresh * 1000) / 1000))
-      - record: rustfs:storage:current
-        labels:
-          source_metric: rustfs_cluster_drive_offline_duration_seconds
-        expr: |
-          rustfs_cluster_drive_offline_duration_seconds{rustfs_cluster_id!="",collection_scope="cluster"}
-          and (timestamp(rustfs_cluster_drive_offline_duration_seconds{rustfs_cluster_id!="",collection_scope="cluster"})
-            >= on (rustfs_cluster_id, observer, collection_scope, job, instance) group_left()
-            (floor(rustfs:storage_snapshot:fresh * 1000) / 1000))
-      - record: rustfs:storage:current
-        labels:
-          source_metric: rustfs_cluster_drive_present
-        expr: |
-          rustfs_cluster_drive_present{rustfs_cluster_id!="",collection_scope="cluster"}
-          and (timestamp(rustfs_cluster_drive_present{rustfs_cluster_id!="",collection_scope="cluster"})
-            >= on (rustfs_cluster_id, observer, collection_scope, job, instance) group_left()
-            (floor(rustfs:storage_snapshot:fresh * 1000) / 1000))
-      - record: rustfs:storage:current
-        labels:
-          source_metric: rustfs_cluster_drive_runtime_state
-        expr: |
-          rustfs_cluster_drive_runtime_state{rustfs_cluster_id!="",collection_scope="cluster"}
-          and (timestamp(rustfs_cluster_drive_runtime_state{rustfs_cluster_id!="",collection_scope="cluster"})
-            >= on (rustfs_cluster_id, observer, collection_scope, job, instance) group_left()
-            (floor(rustfs:storage_snapshot:fresh * 1000) / 1000))
-      - record: rustfs:storage:current
-        labels:
-          source_metric: rustfs_cluster_drive_total_bytes
-        expr: |
-          rustfs_cluster_drive_total_bytes{rustfs_cluster_id!="",collection_scope="cluster"}
-          and (timestamp(rustfs_cluster_drive_total_bytes{rustfs_cluster_id!="",collection_scope="cluster"})
-            >= on (rustfs_cluster_id, observer, collection_scope, job, instance) group_left()
-            (floor(rustfs:storage_snapshot:fresh * 1000) / 1000))
-      - record: rustfs:storage:current
-        labels:
-          source_metric: rustfs_cluster_drive_used_bytes
-        expr: |
-          rustfs_cluster_drive_used_bytes{rustfs_cluster_id!="",collection_scope="cluster"}
-          and (timestamp(rustfs_cluster_drive_used_bytes{rustfs_cluster_id!="",collection_scope="cluster"})
-            >= on (rustfs_cluster_id, observer, collection_scope, job, instance) group_left()
-            (floor(rustfs:storage_snapshot:fresh * 1000) / 1000))
-      - record: rustfs:storage:current
-        labels:
-          source_metric: rustfs_cluster_erasure_set_data_shards
-        expr: |
-          rustfs_cluster_erasure_set_data_shards{rustfs_cluster_id!="",collection_scope="cluster"}
-          and (timestamp(rustfs_cluster_erasure_set_data_shards{rustfs_cluster_id!="",collection_scope="cluster"})
-            >= on (rustfs_cluster_id, observer, collection_scope, job, instance) group_left()
-            (floor(rustfs:storage_snapshot:fresh * 1000) / 1000))
-      - record: rustfs:storage:current
-        labels:
-          source_metric: rustfs_cluster_erasure_set_healing_drives_count
-        expr: |
-          rustfs_cluster_erasure_set_healing_drives_count{rustfs_cluster_id!="",collection_scope="cluster"}
-          and (timestamp(rustfs_cluster_erasure_set_healing_drives_count{rustfs_cluster_id!="",collection_scope="cluster"})
-            >= on (rustfs_cluster_id, observer, collection_scope, job, instance) group_left()
-            (floor(rustfs:storage_snapshot:fresh * 1000) / 1000))
-      - record: rustfs:storage:current
-        labels:
-          source_metric: rustfs_cluster_erasure_set_health
-        expr: |
-          rustfs_cluster_erasure_set_health{rustfs_cluster_id!="",collection_scope="cluster"}
-          and (timestamp(rustfs_cluster_erasure_set_health{rustfs_cluster_id!="",collection_scope="cluster"})
-            >= on (rustfs_cluster_id, observer, collection_scope, job, instance) group_left()
-            (floor(rustfs:storage_snapshot:fresh * 1000) / 1000))
-      - record: rustfs:storage:current
-        labels:
-          source_metric: rustfs_cluster_erasure_set_online_drives_count
-        expr: |
-          rustfs_cluster_erasure_set_online_drives_count{rustfs_cluster_id!="",collection_scope="cluster"}
-          and (timestamp(rustfs_cluster_erasure_set_online_drives_count{rustfs_cluster_id!="",collection_scope="cluster"})
-            >= on (rustfs_cluster_id, observer, collection_scope, job, instance) group_left()
-            (floor(rustfs:storage_snapshot:fresh * 1000) / 1000))
-      - record: rustfs:storage:current
-        labels:
-          source_metric: rustfs_cluster_erasure_set_overall_health
-        expr: |
-          rustfs_cluster_erasure_set_overall_health{rustfs_cluster_id!="",collection_scope="cluster"}
-          and (timestamp(rustfs_cluster_erasure_set_overall_health{rustfs_cluster_id!="",collection_scope="cluster"})
-            >= on (rustfs_cluster_id, observer, collection_scope, job, instance) group_left()
-            (floor(rustfs:storage_snapshot:fresh * 1000) / 1000))
-      - record: rustfs:storage:current
-        labels:
-          source_metric: rustfs_cluster_erasure_set_overall_write_quorum
-        expr: |
-          rustfs_cluster_erasure_set_overall_write_quorum{rustfs_cluster_id!="",collection_scope="cluster"}
-          and (timestamp(rustfs_cluster_erasure_set_overall_write_quorum{rustfs_cluster_id!="",collection_scope="cluster"})
-            >= on (rustfs_cluster_id, observer, collection_scope, job, instance) group_left()
-            (floor(rustfs:storage_snapshot:fresh * 1000) / 1000))
-      - record: rustfs:storage:current
-        labels:
-          source_metric: rustfs_cluster_erasure_set_parity
-        expr: |
-          rustfs_cluster_erasure_set_parity{rustfs_cluster_id!="",collection_scope="cluster"}
-          and (timestamp(rustfs_cluster_erasure_set_parity{rustfs_cluster_id!="",collection_scope="cluster"})
-            >= on (rustfs_cluster_id, observer, collection_scope, job, instance) group_left()
-            (floor(rustfs:storage_snapshot:fresh * 1000) / 1000))
-      - record: rustfs:storage:current
-        labels:
-          source_metric: rustfs_cluster_erasure_set_read_health
-        expr: |
-          rustfs_cluster_erasure_set_read_health{rustfs_cluster_id!="",collection_scope="cluster"}
-          and (timestamp(rustfs_cluster_erasure_set_read_health{rustfs_cluster_id!="",collection_scope="cluster"})
-            >= on (rustfs_cluster_id, observer, collection_scope, job, instance) group_left()
-            (floor(rustfs:storage_snapshot:fresh * 1000) / 1000))
-      - record: rustfs:storage:current
-        labels:
-          source_metric: rustfs_cluster_erasure_set_read_quorum
-        expr: |
-          rustfs_cluster_erasure_set_read_quorum{rustfs_cluster_id!="",collection_scope="cluster"}
-          and (timestamp(rustfs_cluster_erasure_set_read_quorum{rustfs_cluster_id!="",collection_scope="cluster"})
-            >= on (rustfs_cluster_id, observer, collection_scope, job, instance) group_left()
-            (floor(rustfs:storage_snapshot:fresh * 1000) / 1000))
-      - record: rustfs:storage:current
-        labels:
-          source_metric: rustfs_cluster_erasure_set_read_tolerance
-        expr: |
-          rustfs_cluster_erasure_set_read_tolerance{rustfs_cluster_id!="",collection_scope="cluster"}
-          and (timestamp(rustfs_cluster_erasure_set_read_tolerance{rustfs_cluster_id!="",collection_scope="cluster"})
-            >= on (rustfs_cluster_id, observer, collection_scope, job, instance) group_left()
-            (floor(rustfs:storage_snapshot:fresh * 1000) / 1000))
-      - record: rustfs:storage:current
-        labels:
-          source_metric: rustfs_cluster_erasure_set_size
-        expr: |
-          rustfs_cluster_erasure_set_size{rustfs_cluster_id!="",collection_scope="cluster"}
-          and (timestamp(rustfs_cluster_erasure_set_size{rustfs_cluster_id!="",collection_scope="cluster"})
-            >= on (rustfs_cluster_id, observer, collection_scope, job, instance) group_left()
-            (floor(rustfs:storage_snapshot:fresh * 1000) / 1000))
-      - record: rustfs:storage:current
-        labels:
-          source_metric: rustfs_cluster_erasure_set_write_health
-        expr: |
-          rustfs_cluster_erasure_set_write_health{rustfs_cluster_id!="",collection_scope="cluster"}
-          and (timestamp(rustfs_cluster_erasure_set_write_health{rustfs_cluster_id!="",collection_scope="cluster"})
-            >= on (rustfs_cluster_id, observer, collection_scope, job, instance) group_left()
-            (floor(rustfs:storage_snapshot:fresh * 1000) / 1000))
-      - record: rustfs:storage:current
-        labels:
-          source_metric: rustfs_cluster_erasure_set_write_quorum
-        expr: |
-          rustfs_cluster_erasure_set_write_quorum{rustfs_cluster_id!="",collection_scope="cluster"}
-          and (timestamp(rustfs_cluster_erasure_set_write_quorum{rustfs_cluster_id!="",collection_scope="cluster"})
-            >= on (rustfs_cluster_id, observer, collection_scope, job, instance) group_left()
-            (floor(rustfs:storage_snapshot:fresh * 1000) / 1000))
-      - record: rustfs:storage:current
-        labels:
-          source_metric: rustfs_cluster_erasure_set_write_tolerance
-        expr: |
-          rustfs_cluster_erasure_set_write_tolerance{rustfs_cluster_id!="",collection_scope="cluster"}
-          and (timestamp(rustfs_cluster_erasure_set_write_tolerance{rustfs_cluster_id!="",collection_scope="cluster"})
-            >= on (rustfs_cluster_id, observer, collection_scope, job, instance) group_left()
-            (floor(rustfs:storage_snapshot:fresh * 1000) / 1000))
-      - record: rustfs:storage:current
-        labels:
-          source_metric: rustfs_cluster_health_drives_count
-        expr: |
-          rustfs_cluster_health_drives_count{rustfs_cluster_id!="",collection_scope="cluster"}
-          and (timestamp(rustfs_cluster_health_drives_count{rustfs_cluster_id!="",collection_scope="cluster"})
-            >= on (rustfs_cluster_id, observer, collection_scope, job, instance) group_left()
-            (floor(rustfs:storage_snapshot:fresh * 1000) / 1000))
-      - record: rustfs:storage:current
-        labels:
-          source_metric: rustfs_cluster_health_drives_offline_count
-        expr: |
-          rustfs_cluster_health_drives_offline_count{rustfs_cluster_id!="",collection_scope="cluster"}
-          and (timestamp(rustfs_cluster_health_drives_offline_count{rustfs_cluster_id!="",collection_scope="cluster"})
-            >= on (rustfs_cluster_id, observer, collection_scope, job, instance) group_left()
-            (floor(rustfs:storage_snapshot:fresh * 1000) / 1000))
-      - record: rustfs:storage:current
-        labels:
-          source_metric: rustfs_cluster_health_drives_online_count
-        expr: |
-          rustfs_cluster_health_drives_online_count{rustfs_cluster_id!="",collection_scope="cluster"}
-          and (timestamp(rustfs_cluster_health_drives_online_count{rustfs_cluster_id!="",collection_scope="cluster"})
-            >= on (rustfs_cluster_id, observer, collection_scope, job, instance) group_left()
-            (floor(rustfs:storage_snapshot:fresh * 1000) / 1000))
-      - record: rustfs:storage:current
-        labels:
-          source_metric: rustfs_cluster_objects_total
-        expr: |
-          rustfs_cluster_objects_total{rustfs_cluster_id!="",collection_scope="cluster"}
-          and (timestamp(rustfs_cluster_objects_total{rustfs_cluster_id!="",collection_scope="cluster"})
-            >= on (rustfs_cluster_id, observer, collection_scope, job, instance) group_left()
-            (floor(rustfs:storage_snapshot:fresh * 1000) / 1000))
-      - record: rustfs:storage:current
-        labels:
-          source_metric: rustfs_node_disk_free_bytes
-        expr: |
-          rustfs_node_disk_free_bytes{rustfs_cluster_id!="",collection_scope="local"}
-          and (timestamp(rustfs_node_disk_free_bytes{rustfs_cluster_id!="",collection_scope="local"})
-            >= on (rustfs_cluster_id, observer, collection_scope, job, instance) group_left()
-            (floor(rustfs:storage_snapshot:fresh * 1000) / 1000))
-      - record: rustfs:storage:current
-        labels:
-          source_metric: rustfs_node_disk_total_bytes
-        expr: |
-          rustfs_node_disk_total_bytes{rustfs_cluster_id!="",collection_scope="local"}
-          and (timestamp(rustfs_node_disk_total_bytes{rustfs_cluster_id!="",collection_scope="local"})
-            >= on (rustfs_cluster_id, observer, collection_scope, job, instance) group_left()
-            (floor(rustfs:storage_snapshot:fresh * 1000) / 1000))
-      - record: rustfs:storage:current
-        labels:
-          source_metric: rustfs_node_disk_used_bytes
-        expr: |
-          rustfs_node_disk_used_bytes{rustfs_cluster_id!="",collection_scope="local"}
-          and (timestamp(rustfs_node_disk_used_bytes{rustfs_cluster_id!="",collection_scope="local"})
-            >= on (rustfs_cluster_id, observer, collection_scope, job, instance) group_left()
-            (floor(rustfs:storage_snapshot:fresh * 1000) / 1000))
-      - record: rustfs:storage:current
-        labels:
-          source_metric: rustfs_system_drive_api_calls_total
-        expr: |
-          rustfs_system_drive_api_calls_total{rustfs_cluster_id!="",collection_scope="local"}
-          and (timestamp(rustfs_system_drive_api_calls_total{rustfs_cluster_id!="",collection_scope="local"})
-            >= on (rustfs_cluster_id, observer, collection_scope, job, instance) group_left()
-            (floor(rustfs:storage_snapshot:fresh * 1000) / 1000))
-      - record: rustfs:storage:current
-        labels:
-          source_metric: rustfs_system_drive_api_latency_by_api_micros
-        expr: |
-          rustfs_system_drive_api_latency_by_api_micros{rustfs_cluster_id!="",collection_scope="local"}
-          and (timestamp(rustfs_system_drive_api_latency_by_api_micros{rustfs_cluster_id!="",collection_scope="local"})
-            >= on (rustfs_cluster_id, observer, collection_scope, job, instance) group_left()
-            (floor(rustfs:storage_snapshot:fresh * 1000) / 1000))
-      - record: rustfs:storage:current
-        labels:
-          source_metric: rustfs_system_drive_api_latency_micros
-        expr: |
-          rustfs_system_drive_api_latency_micros{rustfs_cluster_id!="",collection_scope="local"}
-          and (timestamp(rustfs_system_drive_api_latency_micros{rustfs_cluster_id!="",collection_scope="local"})
-            >= on (rustfs_cluster_id, observer, collection_scope, job, instance) group_left()
-            (floor(rustfs:storage_snapshot:fresh * 1000) / 1000))
-      - record: rustfs:storage:current
-        labels:
-          source_metric: rustfs_system_drive_availability_errors_total
-        expr: |
-          rustfs_system_drive_availability_errors_total{rustfs_cluster_id!="",collection_scope="local"}
-          and (timestamp(rustfs_system_drive_availability_errors_total{rustfs_cluster_id!="",collection_scope="local"})
-            >= on (rustfs_cluster_id, observer, collection_scope, job, instance) group_left()
-            (floor(rustfs:storage_snapshot:fresh * 1000) / 1000))
-      - record: rustfs:storage:current
-        labels:
-          source_metric: rustfs_system_drive_capacity_observation_age_seconds
-        expr: |
-          rustfs_system_drive_capacity_observation_age_seconds{rustfs_cluster_id!="",collection_scope="local"}
-          and (timestamp(rustfs_system_drive_capacity_observation_age_seconds{rustfs_cluster_id!="",collection_scope="local"})
-            >= on (rustfs_cluster_id, observer, collection_scope, job, instance) group_left()
-            (floor(rustfs:storage_snapshot:fresh * 1000) / 1000))
-      - record: rustfs:storage:current
-        labels:
-          source_metric: rustfs_system_drive_capacity_observation_state
-        expr: |
-          rustfs_system_drive_capacity_observation_state{rustfs_cluster_id!="",collection_scope="local"}
-          and (timestamp(rustfs_system_drive_capacity_observation_state{rustfs_cluster_id!="",collection_scope="local"})
-            >= on (rustfs_cluster_id, observer, collection_scope, job, instance) group_left()
-            (floor(rustfs:storage_snapshot:fresh * 1000) / 1000))
-      - record: rustfs:storage:current
-        labels:
-          source_metric: rustfs_system_drive_count
-        expr: |
-          rustfs_system_drive_count{rustfs_cluster_id!="",collection_scope="local"}
-          and (timestamp(rustfs_system_drive_count{rustfs_cluster_id!="",collection_scope="local"})
-            >= on (rustfs_cluster_id, observer, collection_scope, job, instance) group_left()
-            (floor(rustfs:storage_snapshot:fresh * 1000) / 1000))
-      - record: rustfs:storage:current
-        labels:
-          source_metric: rustfs_system_drive_deletes_total
-        expr: |
-          rustfs_system_drive_deletes_total{rustfs_cluster_id!="",collection_scope="local"}
-          and (timestamp(rustfs_system_drive_deletes_total{rustfs_cluster_id!="",collection_scope="local"})
-            >= on (rustfs_cluster_id, observer, collection_scope, job, instance) group_left()
-            (floor(rustfs:storage_snapshot:fresh * 1000) / 1000))
-      - record: rustfs:storage:current
-        labels:
-          source_metric: rustfs_system_drive_free_bytes
-        expr: |
-          rustfs_system_drive_free_bytes{rustfs_cluster_id!="",collection_scope="local"}
-          and (timestamp(rustfs_system_drive_free_bytes{rustfs_cluster_id!="",collection_scope="local"})
-            >= on (rustfs_cluster_id, observer, collection_scope, job, instance) group_left()
-            (floor(rustfs:storage_snapshot:fresh * 1000) / 1000))
-      - record: rustfs:storage:current
-        labels:
-          source_metric: rustfs_system_drive_free_inodes
-        expr: |
-          rustfs_system_drive_free_inodes{rustfs_cluster_id!="",collection_scope="local"}
-          and (timestamp(rustfs_system_drive_free_inodes{rustfs_cluster_id!="",collection_scope="local"})
-            >= on (rustfs_cluster_id, observer, collection_scope, job, instance) group_left()
-            (floor(rustfs:storage_snapshot:fresh * 1000) / 1000))
-      - record: rustfs:storage:current
-        labels:
-          source_metric: rustfs_system_drive_healing
-        expr: |
-          rustfs_system_drive_healing{rustfs_cluster_id!="",collection_scope="local"}
-          and (timestamp(rustfs_system_drive_healing{rustfs_cluster_id!="",collection_scope="local"})
-            >= on (rustfs_cluster_id, observer, collection_scope, job, instance) group_left()
-            (floor(rustfs:storage_snapshot:fresh * 1000) / 1000))
-      - record: rustfs:storage:current
-        labels:
-          source_metric: rustfs_system_drive_health
-        expr: |
-          rustfs_system_drive_health{rustfs_cluster_id!="",collection_scope="local"}
-          and (timestamp(rustfs_system_drive_health{rustfs_cluster_id!="",collection_scope="local"})
-            >= on (rustfs_cluster_id, observer, collection_scope, job, instance) group_left()
-            (floor(rustfs:storage_snapshot:fresh * 1000) / 1000))
-      - record: rustfs:storage:current
-        labels:
-          source_metric: rustfs_system_drive_info
-        expr: |
-          rustfs_system_drive_info{rustfs_cluster_id!="",collection_scope="local"}
-          and (timestamp(rustfs_system_drive_info{rustfs_cluster_id!="",collection_scope="local"})
-            >= on (rustfs_cluster_id, observer, collection_scope, job, instance) group_left()
-            (floor(rustfs:storage_snapshot:fresh * 1000) / 1000))
-      - record: rustfs:storage:current
-        labels:
-          source_metric: rustfs_system_drive_io_errors_total
-        expr: |
-          rustfs_system_drive_io_errors_total{rustfs_cluster_id!="",collection_scope="local"}
-          and (timestamp(rustfs_system_drive_io_errors_total{rustfs_cluster_id!="",collection_scope="local"})
-            >= on (rustfs_cluster_id, observer, collection_scope, job, instance) group_left()
-            (floor(rustfs:storage_snapshot:fresh * 1000) / 1000))
-      - record: rustfs:storage:current
-        labels:
-          source_metric: rustfs_system_drive_offline_count
-        expr: |
-          rustfs_system_drive_offline_count{rustfs_cluster_id!="",collection_scope="local"}
-          and (timestamp(rustfs_system_drive_offline_count{rustfs_cluster_id!="",collection_scope="local"})
-            >= on (rustfs_cluster_id, observer, collection_scope, job, instance) group_left()
-            (floor(rustfs:storage_snapshot:fresh * 1000) / 1000))
-      - record: rustfs:storage:current
-        labels:
-          source_metric: rustfs_system_drive_offline_duration_seconds
-        expr: |
-          rustfs_system_drive_offline_duration_seconds{rustfs_cluster_id!="",collection_scope="local"}
-          and (timestamp(rustfs_system_drive_offline_duration_seconds{rustfs_cluster_id!="",collection_scope="local"})
-            >= on (rustfs_cluster_id, observer, collection_scope, job, instance) group_left()
-            (floor(rustfs:storage_snapshot:fresh * 1000) / 1000))
-      - record: rustfs:storage:current
-        labels:
-          source_metric: rustfs_system_drive_online_count
-        expr: |
-          rustfs_system_drive_online_count{rustfs_cluster_id!="",collection_scope="local"}
-          and (timestamp(rustfs_system_drive_online_count{rustfs_cluster_id!="",collection_scope="local"})
-            >= on (rustfs_cluster_id, observer, collection_scope, job, instance) group_left()
-            (floor(rustfs:storage_snapshot:fresh * 1000) / 1000))
-      - record: rustfs:storage:current
-        labels:
-          source_metric: rustfs_system_drive_perc_util
-        expr: |
-          rustfs_system_drive_perc_util{rustfs_cluster_id!="",collection_scope="local"}
-          and (timestamp(rustfs_system_drive_perc_util{rustfs_cluster_id!="",collection_scope="local"})
-            >= on (rustfs_cluster_id, observer, collection_scope, job, instance) group_left()
-            (floor(rustfs:storage_snapshot:fresh * 1000) / 1000))
-      - record: rustfs:storage:current
-        labels:
-          source_metric: rustfs_system_drive_present
-        expr: |
-          rustfs_system_drive_present{rustfs_cluster_id!="",collection_scope="local"}
-          and (timestamp(rustfs_system_drive_present{rustfs_cluster_id!="",collection_scope="local"})
-            >= on (rustfs_cluster_id, observer, collection_scope, job, instance) group_left()
-            (floor(rustfs:storage_snapshot:fresh * 1000) / 1000))
-      - record: rustfs:storage:current
-        labels:
-          source_metric: rustfs_system_drive_reads_await
-        expr: |
-          rustfs_system_drive_reads_await{rustfs_cluster_id!="",collection_scope="local"}
-          and (timestamp(rustfs_system_drive_reads_await{rustfs_cluster_id!="",collection_scope="local"})
-            >= on (rustfs_cluster_id, observer, collection_scope, job, instance) group_left()
-            (floor(rustfs:storage_snapshot:fresh * 1000) / 1000))
-      - record: rustfs:storage:current
-        labels:
-          source_metric: rustfs_system_drive_reads_kb_per_sec
-        expr: |
-          rustfs_system_drive_reads_kb_per_sec{rustfs_cluster_id!="",collection_scope="local"}
-          and (timestamp(rustfs_system_drive_reads_kb_per_sec{rustfs_cluster_id!="",collection_scope="local"})
-            >= on (rustfs_cluster_id, observer, collection_scope, job, instance) group_left()
-            (floor(rustfs:storage_snapshot:fresh * 1000) / 1000))
-      - record: rustfs:storage:current
-        labels:
-          source_metric: rustfs_system_drive_reads_per_sec
-        expr: |
-          rustfs_system_drive_reads_per_sec{rustfs_cluster_id!="",collection_scope="local"}
-          and (timestamp(rustfs_system_drive_reads_per_sec{rustfs_cluster_id!="",collection_scope="local"})
-            >= on (rustfs_cluster_id, observer, collection_scope, job, instance) group_left()
-            (floor(rustfs:storage_snapshot:fresh * 1000) / 1000))
-      - record: rustfs:storage:current
-        labels:
-          source_metric: rustfs_system_drive_runtime_state
-        expr: |
-          rustfs_system_drive_runtime_state{rustfs_cluster_id!="",collection_scope="local"}
-          and (timestamp(rustfs_system_drive_runtime_state{rustfs_cluster_id!="",collection_scope="local"})
-            >= on (rustfs_cluster_id, observer, collection_scope, job, instance) group_left()
-            (floor(rustfs:storage_snapshot:fresh * 1000) / 1000))
-      - record: rustfs:storage:current
-        labels:
-          source_metric: rustfs_system_drive_scanning
-        expr: |
-          rustfs_system_drive_scanning{rustfs_cluster_id!="",collection_scope="local"}
-          and (timestamp(rustfs_system_drive_scanning{rustfs_cluster_id!="",collection_scope="local"})
-            >= on (rustfs_cluster_id, observer, collection_scope, job, instance) group_left()
-            (floor(rustfs:storage_snapshot:fresh * 1000) / 1000))
-      - record: rustfs:storage:current
-        labels:
-          source_metric: rustfs_system_drive_timeout_errors_total
-        expr: |
-          rustfs_system_drive_timeout_errors_total{rustfs_cluster_id!="",collection_scope="local"}
-          and (timestamp(rustfs_system_drive_timeout_errors_total{rustfs_cluster_id!="",collection_scope="local"})
-            >= on (rustfs_cluster_id, observer, collection_scope, job, instance) group_left()
-            (floor(rustfs:storage_snapshot:fresh * 1000) / 1000))
-      - record: rustfs:storage:current
-        labels:
-          source_metric: rustfs_system_drive_total_bytes
-        expr: |
-          rustfs_system_drive_total_bytes{rustfs_cluster_id!="",collection_scope="local"}
-          and (timestamp(rustfs_system_drive_total_bytes{rustfs_cluster_id!="",collection_scope="local"})
-            >= on (rustfs_cluster_id, observer, collection_scope, job, instance) group_left()
-            (floor(rustfs:storage_snapshot:fresh * 1000) / 1000))
-      - record: rustfs:storage:current
-        labels:
-          source_metric: rustfs_system_drive_total_inodes
-        expr: |
-          rustfs_system_drive_total_inodes{rustfs_cluster_id!="",collection_scope="local"}
-          and (timestamp(rustfs_system_drive_total_inodes{rustfs_cluster_id!="",collection_scope="local"})
-            >= on (rustfs_cluster_id, observer, collection_scope, job, instance) group_left()
-            (floor(rustfs:storage_snapshot:fresh * 1000) / 1000))
-      - record: rustfs:storage:current
-        labels:
-          source_metric: rustfs_system_drive_used_bytes
-        expr: |
-          rustfs_system_drive_used_bytes{rustfs_cluster_id!="",collection_scope="local"}
-          and (timestamp(rustfs_system_drive_used_bytes{rustfs_cluster_id!="",collection_scope="local"})
-            >= on (rustfs_cluster_id, observer, collection_scope, job, instance) group_left()
-            (floor(rustfs:storage_snapshot:fresh * 1000) / 1000))
-      - record: rustfs:storage:current
-        labels:
-          source_metric: rustfs_system_drive_used_inodes
-        expr: |
-          rustfs_system_drive_used_inodes{rustfs_cluster_id!="",collection_scope="local"}
-          and (timestamp(rustfs_system_drive_used_inodes{rustfs_cluster_id!="",collection_scope="local"})
-            >= on (rustfs_cluster_id, observer, collection_scope, job, instance) group_left()
-            (floor(rustfs:storage_snapshot:fresh * 1000) / 1000))
-      - record: rustfs:storage:current
-        labels:
-          source_metric: rustfs_system_drive_waiting_io
-        expr: |
-          rustfs_system_drive_waiting_io{rustfs_cluster_id!="",collection_scope="local"}
-          and (timestamp(rustfs_system_drive_waiting_io{rustfs_cluster_id!="",collection_scope="local"})
-            >= on (rustfs_cluster_id, observer, collection_scope, job, instance) group_left()
-            (floor(rustfs:storage_snapshot:fresh * 1000) / 1000))
-      - record: rustfs:storage:current
-        labels:
-          source_metric: rustfs_system_drive_writes_await
-        expr: |
-          rustfs_system_drive_writes_await{rustfs_cluster_id!="",collection_scope="local"}
-          and (timestamp(rustfs_system_drive_writes_await{rustfs_cluster_id!="",collection_scope="local"})
-            >= on (rustfs_cluster_id, observer, collection_scope, job, instance) group_left()
-            (floor(rustfs:storage_snapshot:fresh * 1000) / 1000))
-      - record: rustfs:storage:current
-        labels:
-          source_metric: rustfs_system_drive_writes_kb_per_sec
-        expr: |
-          rustfs_system_drive_writes_kb_per_sec{rustfs_cluster_id!="",collection_scope="local"}
-          and (timestamp(rustfs_system_drive_writes_kb_per_sec{rustfs_cluster_id!="",collection_scope="local"})
-            >= on (rustfs_cluster_id, observer, collection_scope, job, instance) group_left()
-            (floor(rustfs:storage_snapshot:fresh * 1000) / 1000))
-      - record: rustfs:storage:current
-        labels:
-          source_metric: rustfs_system_drive_writes_per_sec
-        expr: |
-          rustfs_system_drive_writes_per_sec{rustfs_cluster_id!="",collection_scope="local"}
-          and (timestamp(rustfs_system_drive_writes_per_sec{rustfs_cluster_id!="",collection_scope="local"})
-            >= on (rustfs_cluster_id, observer, collection_scope, job, instance) group_left()
-            (floor(rustfs:storage_snapshot:fresh * 1000) / 1000))
-      - record: rustfs:storage:current
-        labels:
-          source_metric: rustfs_system_drive_writes_total
-        expr: |
-          rustfs_system_drive_writes_total{rustfs_cluster_id!="",collection_scope="local"}
-          and (timestamp(rustfs_system_drive_writes_total{rustfs_cluster_id!="",collection_scope="local"})
-            >= on (rustfs_cluster_id, observer, collection_scope, job, instance) group_left()
-            (floor(rustfs:storage_snapshot:fresh * 1000) / 1000))

.docker/observability/prometheus.yml

@@ -1,84 +0,0 @@
-# Copyright 2024 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-global:
-  scrape_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
-  evaluation_interval: 15s
-  external_labels:
-    cluster: 'rustfs-dev'  # Label to identify the cluster
-    replica: '1'  # Replica identifier
-
-rule_files:
-  - /etc/prometheus/rules/*.yml
-
-scrape_configs:
-  - job_name: 'otel-collector'
-    static_configs:
-      - targets: [ 'otel-collector:8888' ]  # Scrape metrics from Collector
-    scrape_interval: 10s
-
-  - job_name: 'rustfs-app-metrics'
-    static_configs:
-      - targets: [ 'otel-collector:8889' ]  # Application indicators
-    scrape_interval: 15s
-    metric_relabel_configs:
-      - source_labels: [ __name__ ]
-        regex: 'go_.*'
-        action: drop  # Drop Go runtime metrics if not needed
-
-  - job_name: 'tempo'
-    static_configs:
-      - targets: [ 'tempo:3200' ]  # Scrape metrics from Tempo
-
-  - job_name: 'jaeger'
-    static_configs:
-      - targets: [ 'jaeger:14269' ]  # Jaeger admin port (14269 is standard for admin/metrics)
-
-  - job_name: 'loki'
-    static_configs:
-      - targets: [ 'loki:3100' ]
-
-  - job_name: 'prometheus'
-    static_configs:
-      - targets: [ 'localhost:9090' ]
-
-  - job_name: 'vulture'
-    static_configs:
-      - targets:
-          - 'vulture:8080'
-
-otlp:
-  promote_resource_attributes:
-    - service.instance.id
-    - service.name
-    - service.namespace
-    - cloud.availability_zone
-    - cloud.region
-    - container.name
-    - deployment.environment.name
-    - k8s.cluster.name
-    - k8s.container.name
-    - k8s.cronjob.name
-    - k8s.daemonset.name
-    - k8s.deployment.name
-    - k8s.job.name
-    - k8s.namespace.name
-    - k8s.pod.name
-    - k8s.replicaset.name
-    - k8s.statefulset.name
-  translation_strategy: NoUTF8EscapingWithSuffixes
-
-storage:
-  tsdb:
-    out_of_order_time_window: 30m

.docker/observability/prometheus.yml.bak-issue2007-node-exporter-20260823T172248Z

@@ -1,84 +0,0 @@
-# Copyright 2024 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-global:
-  scrape_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
-  evaluation_interval: 15s
-  external_labels:
-    cluster: 'rustfs-dev'  # Label to identify the cluster
-    replica: '1'  # Replica identifier
-
-rule_files:
-  - /etc/prometheus/rules/*.yml
-
-scrape_configs:
-  - job_name: 'otel-collector'
-    static_configs:
-      - targets: [ 'otel-collector:8888' ]  # Scrape metrics from Collector
-    scrape_interval: 10s
-
-  - job_name: 'rustfs-app-metrics'
-    static_configs:
-      - targets: [ 'otel-collector:8889' ]  # Application indicators
-    scrape_interval: 15s
-    metric_relabel_configs:
-      - source_labels: [ __name__ ]
-        regex: 'go_.*'
-        action: drop  # Drop Go runtime metrics if not needed
-
-  - job_name: 'tempo'
-    static_configs:
-      - targets: [ 'tempo:3200' ]  # Scrape metrics from Tempo
-
-  - job_name: 'jaeger'
-    static_configs:
-      - targets: [ 'jaeger:14269' ]  # Jaeger admin port (14269 is standard for admin/metrics)
-
-  - job_name: 'loki'
-    static_configs:
-      - targets: [ 'loki:3100' ]
-
-  - job_name: 'prometheus'
-    static_configs:
-      - targets: [ 'localhost:9090' ]
-
-  - job_name: 'vulture'
-    static_configs:
-      - targets:
-          - 'vulture:8080'
-
-otlp:
-  promote_resource_attributes:
-    - service.instance.id
-    - service.name
-    - service.namespace
-    - cloud.availability_zone
-    - cloud.region
-    - container.name
-    - deployment.environment.name
-    - k8s.cluster.name
-    - k8s.container.name
-    - k8s.cronjob.name
-    - k8s.daemonset.name
-    - k8s.deployment.name
-    - k8s.job.name
-    - k8s.namespace.name
-    - k8s.pod.name
-    - k8s.replicaset.name
-    - k8s.statefulset.name
-  translation_strategy: NoUTF8EscapingWithSuffixes
-
-storage:
-  tsdb:
-    out_of_order_time_window: 30m

.docker/observability/redpanda-console.yaml

@@ -1,3 +0,0 @@
-kafka:
-  brokers:
-    - redpanda:9092