Initial plan

.agents/skills/adversarial-validation/SKILL.md

@@ -1,45 +0,0 @@
----
-name: adversarial-validation
-description: Review a final RustFS diff adversarially when the user requests adversarial review, the root AGENTS.md classifies the change as high risk, or a substantial PR is being reviewed. Do not use for ordinary questions, diagnosis, planning, status, documentation-only work, or routine low-risk implementation.
----
-
-# RustFS Adversarial Validation
-
-Use the risk tier and review shape defined in the root `AGENTS.md`. This skill
-routes a review to RustFS-specific probes without loading unrelated domains.
-
-## Select Lenses
-
-Read only the references required by the diff:
-
-| Lens | When to read |
-|---|---|
-| [Correctness](references/correctness.md) | Every non-exempt adversarial review |
-| [Simplicity](references/simplicity.md) | Mechanical/standard changes and production growth |
-| [Test coverage](references/test-coverage.md) | Behavior or test changes |
-| [Security](references/security.md) | Authn/authz, IAM, RPC trust, paths, secrets, parsing, browser, encryption |
-| [Concurrency/durability](references/concurrency-durability.md) | Async shared state, locks, storage commit, cancellation, persisted queues |
-| [Compatibility](references/compatibility.md) | S3 surface, MinIO interop, metadata, wire/disk formats, mixed versions |
-| [Performance](references/performance.md) | Request/object hot paths, allocation, blocking work, fsync, fan-out |
-
-Do not read all references as a precaution. A path name alone is insufficient;
-the changed behavior must touch the lens's domain.
-
-For a dedicated security audit or advisory analysis, use
-`security-advisory-lessons` instead of loading it automatically during every
-adversarial review.
-
-## Review Protocol
-
-1. Freeze the exact final diff/head and list the selected lenses.
-2. Run the review shape required by root `AGENTS.md`.
-3. For each selected lens, either report a concrete finding or a null verdict
-   naming the attacks performed.
-4. A finding needs `file:line`, a triggering input/state/interleaving, the wrong
-   outcome, and a focused fix or missing regression check.
-5. Fix or rebut every finding with code-path, test, or invariant evidence.
-6. After a non-trivial edit, rerun only lenses affected by that edit against the
-   new exact diff.
-
-Do not turn a null verdict into a long checklist. Record concise evidence that
-the relevant failure classes were attacked.

.agents/skills/adversarial-validation/references/compatibility.md

@@ -1,24 +0,0 @@
-# Compatibility Lens
-
-- Internal metadata uses `metadata_compat` helpers for dual RustFS/MinIO keys,
-  including mixed casing and removal of both twins.
-- Binary UUID metadata treats absent, empty, and nil as no value. Unversioned
-  remote tiers receive no `versionId`; versioned purge requests retain the real
-  version ID.
-- `xl.meta` changes preserve supported header/meta versions, recompute
-  signatures, decode legacy fixtures, and remain readable by old RustFS/MinIO.
-- Foreign/corrupt metadata validates parallel array lengths and missing fields;
-  it returns a decode error rather than indexing, panicking, or fabricating data.
-- Do not “correct” byte-for-byte MinIO ports without legacy fixture evidence.
-  Bitrot framing, shard math, distribution, and inline prefixes are contracts.
-- Client-visible metadata/events strip both internal prefixes
-  case-insensitively.
-- Proto fields are appended, never reused/renumbered; FlatBuffers tables extend
-  compatibly and absent new fields fail closed where authorization/quorum is
-  involved.
-- Replay real client request shapes and exact pagination boundaries for S3
-  handler changes.
-- Bucket metadata/IAM/config parsing remains compatible with pinned real MinIO
-  fixtures and encrypted migration data.
-- Compatibility shims use `RUSTFS_COMPAT_TODO(<task-id>)`, have a removal
-  condition, and default toward reading old data safely.

.agents/skills/adversarial-validation/references/concurrency-durability.md

@@ -1,23 +0,0 @@
-# Concurrency and Durability Lens
-
-- For every changed lock, enumerate overlapping lock sets and construct the
-  ABBA interleaving. Multiple-lock order must be documented and consistent.
-- Mark guard lifetimes and every `.await`, disk, and RPC call inside them.
-  Estimate contention and timeout behavior under concurrent requests.
-- Object commits remain fenced if the distributed lock is lost after shard
-  writes and before metadata rename.
-- For write/rename changes, trace `write tmp -> sync tmp -> rename -> sync parent
-  -> sync required ancestors`; simulate a crash after each step and honor the
-  configured durability gate.
-- Multi-disk fan-out counts every result. Quorum-minus-one cannot become success;
-  heal remains best-effort per target where that is the established contract.
-- At every new cancellable await between mutation and cleanup/commit, drop the
-  future and inspect leftover files, counters, permits, and replay state.
-- Multipart operations on the same upload ID are serialized where required;
-  abort/complete/list races cannot delete parts before durable commit.
-- Post-commit cleanup is best-effort, retry-safe, and cannot fail an already
-  committed write or delete the last surviving copy.
-- Persisted read-modify-write uses serialization/CAS. Queue replay is crash-safe
-  and duplicate delivery has an idempotency contract.
-- Streaming reconstruction failures after partial output surface as errors, not
-  successful EOF.

.agents/skills/adversarial-validation/references/correctness.md

@@ -1,29 +0,0 @@
-# Correctness Lens
-
-Attack the changed behavior, not every subsystem in the repository.
-
-- Trace new error paths to the caller. Inject the ignored/wildcard variants and
-  verify they cannot become success, not-found, or a plausible default.
-- Exercise zero/empty/missing, maximum, and exact-boundary inputs for every
-  changed count, size, index, page limit, or optional value.
-- For aggregation/quorum changes, test exactly quorum and quorum-minus-one with
-  mixed disk errors and nil/placeholder entries.
-- For listing/pagination, test `n == max`, `n == max + 1`, delimiter folding,
-  continuation markers, and object/prefix name collisions.
-- For EC/read/streaming changes, inject failure after partial output and verify
-  the client receives an error rather than a clean truncated body. Assert exact
-  bytes and length.
-- For multipart/object commits, fail before/after rename and cleanup; committed
-  data must remain readable and pre-commit cleanup must not destroy parts.
-- For version/index ordering, test `len - 1`, `len`, equal timestamps, missing
-  versions, and deterministic tie-breaking.
-- For directory-object behavior, trace `__XLDIR__` at the store layer; branches
-  below the layer that sees trailing slashes are dead.
-- For binary UUID metadata, absent, empty, and nil all mean no value. Never send
-  nil/empty `versionId` to an unversioned tier.
-- For agent rules/skill routers, test a trigger matrix covering ordinary
-  inquiry, low-risk implementation, explicit review, high-risk code, PR
-  creation, release, and post-PR monitoring. Each case must select only the
-  intended workflow and retain required safety/authorization boundaries.
-
-Null verdicts name only the probes relevant to the diff.

.agents/skills/adversarial-validation/references/performance.md

@@ -1,20 +0,0 @@
-# Performance Lens
-
-- For added clones/allocations on request/object/block paths, quantify copied
-  data and frequency. Recommend borrowing, move, `Bytes`/`Arc`, `Cow`, or
-  capacity reservation only for a concrete repeated cost.
-- Route every new sync/flush through the durability-mode and bucket override
-  gates; mode `none` must not pay the new fsync.
-- Keep blocking filesystem/CPU work off async runtime threads, but do not split
-  one small operation into many `spawn_blocking` round trips.
-- Measure lock hold time across I/O and compare acquisition order for ABBA.
-- Keep cleanup, extra stat/rename, and diagnostics out of the PUT commit critical
-  section when they need not be there.
-- Detect per-item serial I/O/RPC in batch APIs and accidental quadratic scans;
-  use a gate or bounded concurrency when the concrete fan-out warrants it.
-- Count buffer growth and byte copies in EC/bitrot paths; preserve pool gauge
-  balance and avoid repeated metadata decode/fetch per object.
-- Repetitive success logs stay at `trace`; metrics/instrumentation on hot paths
-  require an existing gate.
-- Claims of no impact on PUT/GET/commit/erasure paths need relevant benchmark or
-  A/B evidence, especially for 4 KiB objects.

.agents/skills/adversarial-validation/references/security.md

@@ -1,31 +0,0 @@
-# Security Lens
-
-Use `security-advisory-lessons` only for a dedicated advisory/security audit.
-For an ordinary matched diff, attack these boundaries:
-
-- Admin routes: route registration, whitelist, handler authn, and the exact
-  `AdminAction` must agree. Read-only diagnostics still require admin authz.
-- IAM/service accounts: treat parent, claims, keys, groups, status, and policy
-  names as attacker-controlled; prove ownership/root authority before writes.
-- Protocol frontends: every changed/sibling command authorizes the matching S3
-  action before reaching storage.
-- Secrets/signatures: use constant-time comparison, normalize public failures,
-  keep RPC/root/STS keys independent, and fail closed when secrets are absent.
-- RPC: bind signatures to the exact method/path and timestamp; reject replay,
-  stale, malformed, truncated, and invalid-enum payloads without panic.
-- Paths/object/archive entries: reject traversal, absolute/platform escapes,
-  and normalization differences between authz and storage.
-- Copy/multipart/presigned POST: enforce source, destination, version-aware
-  actions, copy-source conditions, and every signed policy condition.
-- Logging/errors: never expose credentials, tokens, expected signatures, raw
-  secret-bearing input, or merged configs—including via `Debug` and parse errors.
-- Untrusted serde: reject unknown fields where compatible and validate
-  security-critical defaults/ranges before numeric conversion.
-- SSE/browser/CORS/trusted proxy: inspect stored ciphertext and wrapper order;
-  isolate user content; never reflect credentialed arbitrary origins or trust
-  forwarded identity from direct clients.
-- Object Lock: unreadable/fabricated/unparsable metadata fails closed across
-  foreground, lifecycle, scanner, and force-delete paths.
-
-Security findings distinguish unauthenticated compromise from a
-low-privileged authenticated bypass.

.agents/skills/adversarial-validation/references/simplicity.md

@@ -1,22 +0,0 @@
-# Simplicity Lens
-
-- Compare the production diff with the smallest equivalent local edit. Fewer
-  lines alone are not evidence; the replacement must preserve correctness,
-  compatibility, readability, and real boundaries.
-- Search the touched crate, domain owner, `crates/utils`, `crates/common`, and
-  relevant dependencies for each new helper, constant, wrapper, or fixture.
-- Reject forced reuse when normalization, error, deadline, or durability
-  semantics differ.
-- Require a concrete trigger for every new defensive branch. Keep boundary
-  checks for disk/RPC/version data and checks immediately before destructive
-  actions.
-- Flag one-caller helpers only when they merely forward or split a short linear
-  flow without adding domain naming, invariant isolation, or useful context.
-- Ensure a replacement removes the superseded in-scope path or keeps one
-  canonical core behind a documented compatibility adapter.
-- Remove narration/change-history comments; preserve concise safety, lock,
-  durability, and compatibility invariants.
-- Treat tests, fixtures, generated code, and documentation separately from
-  production growth. Do not optimize away meaningful regression coverage.
-
-A finding must include a concrete smaller design, not a style preference.

.agents/skills/adversarial-validation/references/test-coverage.md

@@ -1,24 +0,0 @@
-# Test-Coverage Lens
-
-- For every behavior claim, name the focused test/check that fails if the
-  changed hunk is reverted. If none is practical, require the reason and
-  residual risk.
-- Confirm tests exercise the real production path and assert returned values,
-  exact bytes, stored state, or the specific error variant—not only success,
-  `is_err()`, or no panic.
-- For new flags/modes, verify each branch and ask which test fails if the branch
-  is inverted.
-- For new error propagation, inject the failure and assert the caller observes
-  it; mentally replacing `?`/`return Err` with success must break a test.
-- Streaming GET tests assert the complete body and length under degraded reads.
-- Disk/wire-format tests use pinned foreign/legacy fixtures; same-code
-  round-trips are insufficient for compatibility.
-- Concurrency tests use readiness polling, isolate global state, and avoid fixed
-  sleeps or unrealistically short timeouts. Use nextest groups when process-level
-  serialization is required.
-- Internal metadata tests assert both RustFS and MinIO keys, not only read-back
-  through a helper that prefers one key.
-- Boundary companions are distinct coverage: `n == max` vs `max + 1`, and
-  absent vs empty vs nil UUID.
-- A focused test proves only the targets/features it builds. Add compilation or
-  Clippy only for uncovered changed targets.

.agents/skills/arch-checks/SKILL.md

@@ -1,64 +0,0 @@
----
-name: arch-checks
-description: Resolve failures from the repository's architecture guard scripts — check_layer_dependencies.sh, check_architecture_migration_rules.sh, check_unsafe_code_allowances.sh, check_logging_guardrails.sh, check_doc_paths.sh. Use when make pre-commit / pre-pr or CI fails on one of these checks.
----
-
-# Architecture Guard Checks
-
-All five run in `make pre-commit` / `make pre-pr` and in CI. Fix the cause;
-never weaken a check to get green.
-
-## `check_layer_dependencies.sh` — layer DAG in `rustfs/src`
-
-Enforces `composition (server, startup/init) → interface (admin,
-storage/ecfs, storage/s3_api) → app → infra`; no upward imports. Server source
-files are composition roots, while imports of their exported HTTP contracts
-are classified as interface dependencies. Known legacy violations live in
-`scripts/layer-dependency-baseline.txt`.
-
-Dedicated `*_test.rs` and `tests/` modules are outside this production guard.
-Inline `#[cfg(test)]` imports remain checked under their source file's layer;
-move architecture-crossing test scaffolding into a dedicated test module.
-
-- **New violation**: restructure your change so the dependency points
-  downward (move the shared type/function to the lower layer).
-- **You legitimately removed a baseline entry**: run
-  `./scripts/check_layer_dependencies.sh --update-baseline` and commit the
-  shrunken baseline. Never add new entries to the baseline to make a new
-  violation pass.
-
-## `check_architecture_migration_rules.sh` — required doc sections
-
-Asserts that the core docs under `docs/architecture/` (overview,
-crate-boundaries, runtime-lifecycle, readiness-matrix,
-storage-control-data-plane, global-state-crate-split-plan,
-ecstore-module-split-plan, …) still contain specific headings and exact
-source lines. If it fails after a doc edit, you reworded or removed a
-guarded line — restore the wording or update the script deliberately in the
-same PR, with rationale.
-
-## `check_unsafe_code_allowances.sh`
-
-Every `#[allow(unsafe_code)]` needs a `SAFETY:` comment within a few lines.
-Write the actual safety argument; don't add a placeholder.
-
-## `check_logging_guardrails.sh`
-
-A fixed list of security-sensitive files (auth, IAM, KMS, admin handlers…)
-is scanned for logging violations. If you created a new sensitive file,
-consider adding it to the script's `checked_files` list.
-
-## `check_doc_paths.sh`
-
-Instruction/architecture docs (`AGENTS.md`, `CLAUDE.md`, `ARCHITECTURE.md`,
-`docs/architecture/*.md`) must not reference repo file paths that no longer
-exist. If your refactor moved code, update the docs that point at it — the
-error message lists `doc -> stale-path` pairs.
-
-## `check_no_planning_docs.sh`
-
-Planning-type documents must not be committed (see AGENTS.md "Sources of
-Truth"). The guard fails if anything is tracked under `docs/superpowers/` —
-`.gitignore` already ignores it, but `git add -f` bypasses that, so this closes
-the hole. Fix by removing the listed file(s) with `git rm`; keep the plan or
-spec in the issue tracker or a local worktree instead.

.agents/skills/code-change-verification/SKILL.md

@@ -1,82 +0,0 @@
----
-name: code-change-verification
-description: Review a commit, PR, or merged patch when the user requests ordinary code-change verification. Do not combine with adversarial-validation; use that skill instead for explicitly adversarial, substantial, or high-risk RustFS reviews.
----
-
-# Code Change Verification
-
-Use this skill for an ordinary requested review. If the root policy or user calls
-for adversarial validation, use `adversarial-validation` instead of running both.
-
-## Quick Start
-
-1. Read the scope: commit, PR, patch, or file list.
-2. Map each changed area by risk and user impact.
-3. Inspect each risky change in context.
-4. Report findings first, ordered by severity.
-5. Close with residual risks and verification recommendations.
-
-## Core Workflow
-
-### 1) Scope and assumptions
-- Confirm change source (diff, commit, PR, files), target branch, language/runtime, and version.
-- If context is missing, state assumptions before deeper analysis.
-- Focus only on requested scope; avoid reviewing unrelated files.
-
-### 2) Risk map
-- Prioritize in this order:
-  - Data correctness and user-visible behavior
-  - API/contract compatibility
-  - Security and authz/authn boundaries
-  - Concurrency and lifecycle correctness
-  - Performance and resource usage
-- Give higher priority to stateful paths, migration logic, defaults, and error handling.
-
-### 3) Evidence-based inspection
-- Read each modified hunk with neighboring context.
-- Trace call paths and call-site expectations.
-- Check for:
-  - invariant breaks and missing guards
-  - unchecked assumptions and null/empty/error-path handling
-  - stale tests, fixtures, and configs
-  - hidden coupling to shared helpers/constants/features
-- If a point is uncertain, mark it as an open question instead of guessing.
-
-#### Rust-specific checks (apply to all Rust changes)
-
-Run the full checklist in [rust-code-quality](../rust-code-quality/SKILL.md) — the canonical Rust review checklist for the unwrap/casting/cloning/locking/recursion/error-type/serde/test rules and the reuse-and-necessity checks (duplicated helpers, defensive branches without a nameable trigger, redundant error wrapping). Do not restate those rules here; carry its P0–P3 ratings over unchanged and use this skill's output format.
-
-### 4) Findings-first output
-- Order findings by severity:
-  - P0: critical failure, security breach, or data loss risk
-  - P1: high-impact regression
-  - P2: medium risk correctness gap
-  - P3: low risk/quality debt
-- For each finding include:
-  - Severity
-  - `path:line` reference
-  - concise issue statement
-  - impact and likely failure mode
-  - specific fix or mitigation
-  - validation step to confirm
-- If no issues exist, explicitly state `No findings` and why.
-
-### 5) Close
-- Report assumptions and unknowns.
-- Suggest targeted checks (tests, canary checks, logs/metrics, migration validation).
-
-## Output Template
-
-1. Findings
-2. No findings (if applicable)
-3. Assumptions / Unknowns
-4. Recommended verification steps
-
-## Finding Template
-
-- `[P1] Missing timeout for downstream call`
-  - Location: `path/to/file.rs:123`
-  - Issue: ...
-  - Impact: ...
-  - Fix suggestion: ...
-  - Validation: ...

.agents/skills/code-change-verification/agents/openai.yaml

@@ -1,4 +0,0 @@
-interface:
-  display_name: "Code Change Verification"
-  short_description: "Prioritize risks and verify code changes before merge."
-  default_prompt: "Use $code-change-verification for an ordinary requested diff review with prioritized findings."

.agents/skills/plugin-contract-guard/SKILL.md

@@ -1,83 +0,0 @@
----
-name: plugin-contract-guard
-description: Invariants and change procedure for the target-plugin / extension system — plugin manifests, admin plugin/extension catalog and instance APIs, secret redaction, external-plugin install policy. Use when editing crates/targets (manifest, plugin, control_plane, catalog, runtime), crates/extension-schema, or rustfs/src/admin plugin_contract.rs / plugins_*.rs / extensions.rs / target_descriptor.rs.
----
-
-# Plugin & Extension Contract Guard
-
-The "plugin system" spans four surfaces that must stay consistent:
-
-| Surface | Location |
-|---|---|
-| Manifests & registry | `crates/targets/src/{manifest,plugin}.rs` |
-| Install/enable planning (control plane) | `crates/targets/src/control_plane.rs` |
-| Extension schemas | `crates/extension-schema/src/lib.rs`, `crates/targets/src/catalog/extension.rs` |
-| Admin API contract | `rustfs/src/admin/plugin_contract.rs`, `handlers/{plugins_catalog,plugins_instances,extensions,target_descriptor}.rs` |
-
-## Hard invariants (verify before merging)
-
-1. **Secrets have one source of truth.** Secret config keys are declared only
-   in the plugin manifest (`TargetPluginManifest.secret_fields`,
-   `crates/targets/src/manifest.rs`) and flow to admin via
-   `AdminTargetSpec.secret_fields`. Never add a hand-maintained per-service
-   secret table in a handler; if redaction misses a field, fix the manifest.
-
-2. **Redaction must round-trip.** Instance GET responses replace secret values
-   with `***redacted***` (`REDACTED_SECRET_VALUE` in `plugins_instances.rs`).
-   Instance PUT restores the stored secret when it receives that placeholder
-   back (`restore_redacted_secret_values`). Any new read or write path for
-   target config must keep both halves: redact on the way out, restore the
-   placeholder on the way in. The placeholder literal must never be persisted.
-
-3. **Fixtures never reach production responses.**
-   `example_external_webhook_plugin()` (`crates/targets/src/catalog/mod.rs`)
-   is a test/demo fixture for control-plane planning tests. Production
-   catalog/extension handlers must not include it; regression tests
-   (`plugin_catalog_never_exposes_example_or_external_fixtures`,
-   `extension_catalog_never_exposes_example_or_external_fixtures`) enforce it.
-
-4. **External plugin flow is planning-only and deny-by-default.**
-   `plan_external_target_plugin_action` returns decisions, it executes
-   nothing. `TargetPluginExternalFlowGate::default()` is fully closed and
-   `TargetPluginInstallPolicy::default().allowed_download_hosts` is empty —
-   keep it that way; tests opt in via explicit policies. Install validation
-   requires https, an allowlisted host, a full 64-hex-char sha256 digest,
-   signature and provenance URIs, and an artifact matching the host
-   `target_triple`.
-
-5. **Custom target types must not collide.** Unknown target types get an
-   interned unique `custom:<type>` plugin id (`custom_plugin_id` in
-   `manifest.rs`). Custom plugins with secrets must register via
-   `TargetPluginDescriptor::with_manifest` and declare `secret_fields`;
-   `::new` derives a manifest with no secrets.
-
-## Changing the admin JSON contract
-
-- Shapes are locked twice in `plugin_contract.rs` tests: insta snapshots
-  (`rustfs/src/admin/snapshots/`) plus literal `json!` assertions. Update
-  both deliberately; a shape change is a console-facing API change.
-- Field naming is `snake_case`, except discovery blocks
-  (`runtimeCapabilities`, `clusterSnapshot`, `extensionsCatalog`) which are
-  camelCase **by cross-endpoint convention** (same shape in `system.rs`,
-  `console.rs`, `pools.rs`). Do not "fix" that inconsistency locally.
-- Contract types deliberately duplicate `rustfs_targets` types
-  (anti-corruption layer). Add a `From` impl; do not serialize internal
-  types directly.
-
-## Handler conventions
-
-- Every new admin plugin/extension route needs authorization at the top of
-  `call` and an `include_str!` guard test asserting it (repo-wide pattern —
-  see `plugin_instance_handlers_require_admin_authorization_contract`).
-- Reads use `GetBucketTargetAction` (instances) or `ServerInfoAdminAction`
-  (catalogs); writes use `SetBucketTargetAction`.
-- Refresh persisted module switches once per request
-  (`refresh_persisted_module_switches`), then evaluate the sync
-  `module_disabled_block_reason` per domain — do not re-read the store per
-  domain or per instance.
-
-## Generic bounds
-
-Event-payload generics use the `PluginEvent` blanket trait
-(`crates/targets/src/plugin.rs`). Do not respell
-`Send + Sync + 'static + Clone + Serialize + DeserializeOwned`.

.agents/skills/pr-creation-checker/SKILL.md

@@ -1,46 +0,0 @@
----
-name: pr-creation-checker
-description: Perform the final RustFS PR preflight and draft compliant English title/body metadata immediately before creating or updating a PR. Do not use during implementation or as a second general code review.
----
-
-# PR Creation Checker
-
-Use this skill only at the PR boundary. Reuse completed diff review and
-verification evidence; do not reread the repository or rerun equivalent checks.
-
-## Preflight
-
-1. Confirm the branch is based on current `origin/main` and contains only the
-   intended task diff.
-2. Inspect `git diff --stat`, `git diff --check`, and changed file names for
-   secrets, logs, generated artifacts, or unrelated edits.
-3. Confirm the checks selected by root `AGENTS.md` passed on the final diff.
-   Do not replace focused behavioral tests with a generic gate or rerun checks
-   already covered by an unchanged umbrella run.
-4. Read `.github/pull_request_template.md`. Consult `Makefile`, `.config/make/`,
-   or CI only when the required command/current gate is uncertain.
-5. Return `BLOCKED` for an unclean scope, missing required evidence, failed
-   required checks, or non-compliant metadata.
-
-## Metadata
-
-- Title: English Conventional Commit, at most 72 characters, with no tool
-  prefix.
-- Body: English, exact template headings, `N/A` where needed, concise rationale,
-  actual verification commands, and material risks/rollback notes.
-- Use repository-relative paths; never include local absolute paths.
-- Keep prose paragraphs on one logical line and never include the literal
-  sequence `\n`.
-- Use a temporary body file with `gh pr create --body-file` or
-  `gh pr edit --body-file`; never pass multiline Markdown inline.
-
-## Output
-
-- Status: `READY` or `BLOCKED`.
-- Title.
-- Complete PR body.
-- Verification commands and results.
-- Risks or `N/A`.
-
-Immediately before the GitHub write, repeat only the five preflight checks above
-against the final head.

.agents/skills/pr-creation-checker/agents/openai.yaml

@@ -1,4 +0,0 @@
-interface:
-  display_name: "PR Creation Checker"
-  short_description: "Draft RustFS-ready PRs with checks, template, and blockers."
-  default_prompt: "Use $pr-creation-checker for final PR preflight and compliant English title/body metadata."

.agents/skills/rust-code-quality/SKILL.md

@@ -1,123 +0,0 @@
----
-name: rust-code-quality
-description: Run a focused Rust quality review when the user requests one, when reviewing a Rust PR/commit, or when another selected review workflow delegates Rust-specific checks. Do not auto-load for every implementation edit.
----
-
-# Rust Code Quality Gate
-
-Use this skill for a dedicated Rust review to cover rules that `cargo clippy`
-does not catch.
-
-## Quick Start
-
-1. Identify changed `.rs` files.
-2. Run automated checks on changed files.
-3. Run manual review checklist on the diff.
-4. Resolve or rebut every finding with evidence; P0/P1 findings cannot be deferred.
-
-## Automated Checks
-
-Use these searches to find candidates in changed `.rs` files. Inspect syntax,
-`#[cfg(test)]` scope, and the changed hunk before reporting a finding; text
-filters do not reliably distinguish production code from tests.
-
-```bash
-# 1. unwrap/expect candidates
-rg -n '\.unwrap\(\)|\.expect\(' <changed-files>
-
-# 2. Silent type truncation via `as` cast
-rg -n ' as (u8|u16|u32|u64|usize|i8|i16|i32|i64|isize)\b' <changed-files>
-
-# 3. String as error type
-rg -n 'Result<.*String>' <changed-files>
-
-# 4. Box<dyn Error> in public APIs
-rg -n 'Box<dyn.*Error' <changed-files>
-
-# 5. println/eprintln in production
-rg -n 'println!\|eprintln!' <changed-files>
-
-# 6. Ordering::Relaxed usage (verify each is intentional)
-rg -n 'Ordering::Relaxed' <changed-files>
-
-# 7. Default substituted for a possibly-required value (judge each: is the value optional by domain?)
-rg -n 'unwrap_or_default\(\)|unwrap_or\(' <changed-files>
-```
-
-## Manual Review Checklist
-
-For the Rust diff under review, verify:
-
-### Error Handling
-- [ ] Every production `unwrap()` or `expect()` is infallible by type or a checked invariant; explain only non-obvious invariants, using an existing type, a useful `expect` message, or a concise comment
-- [ ] No `Result<_, String>` in public API signatures
-- [ ] Public library APIs use domain errors unless deliberate error erasure at a boundary is part of the contract
-- [ ] `Error::source()` is overridden when inner error is stored
-- [ ] Error messages are actionable without exposing secret input
-
-### Type Safety
-- [ ] No silent `as` truncation (negative→unsigned, large→small)
-- [ ] Fallible numeric conversions use `TryFrom`/`try_into()` and return a typed error; clamp or saturate only when the domain explicitly requires it
-- [ ] Floating-point to integer conversion validates finiteness, sign, and range before conversion
-
-### Concurrency
-- [ ] Lock acquisition order is documented when multiple locks are used, and matches every other call site taking any overlapping subset (ABBA check)
-- [ ] No `tokio::sync` lock guard (read or write) held across `.await` without bounded hold time — long-lived read guards wedge writers (#4195)
-- [ ] Atomic read-modify-write uses the direct `fetch_*` operation when possible; use `compare_exchange` only for conditional updates
-- [ ] `std::sync::Mutex` in async context is held only briefly, never across `.await`
-
-### Memory and Performance
-- [ ] On an identified hot path, report cloning or allocation only with a concrete per-request/per-object cost or benchmark signal
-- [ ] Prefer borrowing, moving, `Bytes`/`Arc`, or capacity reservation only when it reduces that cost without obscuring ownership or APIs
-
-### Recursion Safety
-- [ ] Recursion over untrusted, persisted, or otherwise unbounded input has a depth limit or uses iterative traversal
-- [ ] Tree/cache traversals handle corrupted/cyclic input safely
-
-### Testing
-- [ ] Tests have an observable failure criterion; delegated assertions, `#[should_panic]`, snapshot/property checks, and meaningful `Result` failures do not need a redundant `assert!`
-- [ ] Use `expect` only when its message improves failure diagnosis; do not add boilerplate to self-evident test setup
-- [ ] Test volume and line count are never treated as production-code growth
-
-### Serde
-- [ ] Structs from untrusted input have `#[serde(deny_unknown_fields)]`
-- [ ] `#[serde(default)]` not used on security-critical fields without validation
-
-### Code Hygiene
-- [ ] No `#![allow(dead_code)]` at crate root
-- [ ] No camelCase statics or Hungarian notation
-- [ ] New string literals don't duplicate existing constants
-
-### Reuse and Necessity
-- [ ] No new helper duplicates `crates/utils`, `crates/common`, the touched crate, the likely domain-owning crate, a relevant direct dependency, or plain std/tokio behavior; reused helpers match the call site's semantics
-- [ ] No branch without a nameable concrete trigger; no re-validation of what a validated upstream layer on the same path already guarantees (Cross-Cutting Domain Invariant patterns and pre-destructive-action re-checks are load-bearing — keep them)
-- [ ] Error context attached once where actionable, not re-wrapped at every hop; no typed→generic error conversion below aggregation/quorum layers
-- [ ] Comments avoid narration and change history while completely stating non-obvious lock, `SAFETY`, durability, compatibility, and unwrap invariants
-- [ ] No near-duplicate test pinning the same code path and poison-value class as an existing test (boundary companions — n==max vs max+1, absent/empty/nil UUID — are never near-duplicates)
-
-## Severity Classification
-
-- **P0 (Block merge)**: demonstrated data loss, security breach, remote crash, or deadlock
-- **P1 (Must fix)**: concrete correctness, compatibility, or material hot-path regression
-- **P2 (Should fix)**: avoidable duplication or maintainability issue with a concrete simpler replacement
-- **P3 (Nice to fix)**: local style or clarity issue with no behavioral risk
-
-## Output Template
-
-```
-## Rust Code Quality Report
-
-### Automated Scan
-- unwrap/expect candidates inspected: N
-- numeric-cast candidates inspected: N
-- error-type candidates inspected: N
-- output-macro candidates inspected: N
-
-### Findings
-- [P1] `path:line` — description
-  - Fix: ...
-  - Validation: ...
-
-### Verdict
-PASS / BLOCKED (list blocking findings)
-```

.agents/skills/rustfs-logging-governance/SKILL.md

@@ -1,34 +0,0 @@
----
-name: rustfs-logging-governance
-description: Add or review RustFS `tracing` events with the repository field shape, level policy, privacy boundaries, and guardrails. Use when a change adds or edits a tracing macro/instrumentation site or the logging guardrail script.
----
-
-# RustFS Logging Governance
-
-Apply this skill only to changed logging sites; do not turn a local log edit into
-a broad logging cleanup.
-
-## Workflow
-
-1. Read the changed function/module context and classify the site as lifecycle,
-   request/hot path, fallback, external fetch, or summary.
-2. Match neighboring structured events and reuse existing `EVENT_*`,
-   `LOG_COMPONENT_*`, and `LOG_SUBSYSTEM_*` constants.
-3. Put stable fields first (`event`, `component`, `subsystem`, `state`/`result`,
-   then context) and a short label last.
-4. Select the level by operational meaning:
-   - `error`: behavior/security-affecting failure;
-   - `warn`: degraded/fallback/operator-actionable state;
-   - `info`: low-frequency lifecycle/mode change;
-   - `debug`: targeted diagnostics;
-   - `trace`: repetitive request/object/shard success paths.
-5. Never log secrets, tokens, auth headers, credential payloads, raw
-   attacker-controlled bodies, or merged config dumps. Error strings and
-   `Debug` output are log surfaces too.
-6. Prefer one aggregate summary over inventories or startup banners.
-7. Run `./scripts/check_logging_guardrails.sh` and the checks selected by root
-   `AGENTS.md`.
-
-Read [logging-governance.md](references/logging-governance.md) only for a broad
-logging audit, event-model migration, or guardrail expansion. Ordinary single-
-site edits do not require the full workspace scope map.

.agents/skills/rustfs-logging-governance/agents/openai.yaml

@@ -1,4 +0,0 @@
-interface:
-  display_name: "RustFS Logging Governance"
-  short_description: "Standardize RustFS logs with structured events and guardrails."
-  default_prompt: "Use $rustfs-logging-governance to standardize or review RustFS logging, reduce noise, and update guardrails."

.agents/skills/rustfs-logging-governance/references/logging-governance.md

@@ -1,62 +0,0 @@
-# Logging Audit and Migration Reference
-
-Read this reference only for a broad logging audit, an event-model migration,
-or a change to `scripts/check_logging_guardrails.sh`. Use `Cargo.toml` for the
-current workspace/crate list instead of maintaining one here.
-
-## Audit by Operational Role
-
-- Server/protocol/admin: lifecycle, authorization failures, request boundaries,
-  and degraded subsystems; avoid normal request success at `info`.
-- Storage/heal/scanner/capacity: integrity failures and aggregate lifecycle;
-  avoid per-object, per-shard, and folder iteration noise.
-- IAM/policy/credentials/KMS/crypto: safe identifiers and enforcement results;
-  never emit secrets, claims, payloads, or expected authenticators.
-- Notify/audit/targets: target lifecycle and batch/backpressure summaries; avoid
-  per-event success logs.
-- Locking/concurrency/I/O foundations: contention anomalies and state changes;
-  prefer metrics for high-frequency worker/permit signals.
-- Shared type/schema crates: log at the operational caller boundary unless the
-  crate itself owns the failure context.
-
-## Event Shape
-
-Prefer stable fields in this order when available:
-
-1. `event`
-2. `component`
-3. `subsystem`
-4. `state` or `result`
-5. stable context such as mode, duration, reason, counts, safe identifiers, or
-   capacity/permit values
-6. short message label
-
-Reuse the module's constants and neighboring field names. Do not create aliases
-for the same concept.
-
-## Patterns to Retire
-
-- sentence-style lifecycle announcements;
-- startup banners and checklist lines;
-- repetitive success logs at `info`/`debug`;
-- raw inventories when an aggregate count is sufficient;
-- fallback prose with values embedded in the message;
-- `?value`/`Debug` output for credential-bearing or attacker-controlled data;
-- logging a parse input when the malformed input may itself be a secret.
-
-## Guardrail Changes
-
-When expanding `scripts/check_logging_guardrails.sh`:
-
-1. Add only files/patterns intentionally migrated in the same change.
-2. Keep patterns concrete and grep-friendly.
-3. Do not encode a style that remains valid elsewhere as a global ban.
-4. Run the guardrail script and the root validation tier.
-5. Treat the script as a floor; manually verify level, field shape, and privacy.
-
-Useful search seeds for the changed surface:
-
-```bash
-rg -n 'error!|warn!|info!|debug!|trace!|#\[instrument' <changed-paths>
-rg -n '\?[^,)]|secret|token|credential|authorization|merged_config' <changed-paths>
-```

.agents/skills/rustfs-release-publish/SKILL.md

@@ -1,243 +0,0 @@
----
-name: rustfs-release-publish
-description: "Run the end-to-end RustFS console gate, version bump, preview validation, human confirmation, and final-tag publication pipeline. Use only when the user explicitly asks to release or publish a RustFS version (发版/发布)."
----
-# RustFS Release Publish (preview-validated pipeline)
-
-This skill orchestrates a full release. It wraps `rustfs-release-version-bump` (which only edits version files and opens the PR) with a mandatory preview-tag validation loop before the final tag is published.
-
-Core design: **version files never carry a `-preview.N` suffix**. The preview suffix exists only in tag names. A preview tag creates a visible GitHub Release marked Prerelease and uploads versioned assets, but it never becomes GitHub Latest and never updates `*-latest`, `latest.json`, R2, Docker, or Helm channels. This works because the binary self-reports the git tag it was built from (`build::TAG` via shadow_rs, see `rustfs/src/config/cli.rs` `SHORT_VERSION`), and `build.yml` derives artifact names and preview classification from the tag name — Cargo.toml's version is only a no-tag fallback. Therefore the preview tag and the final tag can (and MUST) point at the exact same commit: what you validated is byte-for-byte the source that ships.
-
-Pipeline shape:
-
-```
-check console main against its latest Release
-  -> if ahead: publish console -> wait for Release asset + latest API
-  -> bump RustFS version files to <target> (final version, ONE commit) -> merge
-  -> tag <preview-tag> at that commit -> CI green
-  -> verify preview Release assets -> run binary locally + console checks
-  -> validate with latest rc client
-  -> report preview acceptance results -> STOP for explicit human confirmation
-  -> tag <target> at the SAME commit (zero delta) -> re-verify CI/release
-```
-
-On validation failure: fix lands on main via normal PR (version files are already at `<target>`, no new bump PR), then tag `<preview-tag N+1>` at the new main commit and restart from Phase 2.
-
-## Required inputs
-
-- Final target version, for example `1.0.0-beta.10`.
-- Preview iteration `N` (default: next unused preview tag for that target; check with `git tag -l '<target>-preview.*'` after `git fetch --tags`).
-
-If the target version is missing or ambiguous, stop and ask before doing anything (see the semver gate below).
-
-## Semver gate — confirm the target version before touching anything
-
-Versions follow [SemVer 2.0.0](https://semver.org/). Precedence reminder:
-
-```
-1.0.0-alpha < 1.0.0-alpha.1 < 1.0.0-beta.2 < 1.0.0-beta.11 < 1.0.0-rc.1 < 1.0.0 < 1.0.1 < 1.1.0 < 2.0.0
-```
-
-Numeric prerelease identifiers compare numerically (`beta.9 < beta.10`), not lexically — see [semver.org spec item 11](https://semver.org/#spec-item-11). Preview tags are internal validation tags layered on top of the target's prerelease channel — they are never themselves a deliverable version and never appear in version files.
-
-Rules:
-
-- A request like "发个版" / "release the next version" without an exact version string is ALWAYS ambiguous. Derive the current latest tag (`git tag --sort=-v:refname | head`), then ask the user to choose via AskUserQuestion with concrete candidates, e.g. from `1.0.0-beta.10`: next prerelease `1.0.0-beta.11`, promote to `1.0.0-rc.1`, promote to stable `1.0.0`. Never guess between these — they have very different meanings (channel promotion vs. iteration) and different CI classification consequences.
-- After a stable `X.Y.Z` exists, the next version must state which component bumps: patch `X.Y.(Z+1)` for fixes only, minor `X.(Y+1).0` for backward-compatible features, major `(X+1).0.0` for breaking changes. If the user names a bump type but not a number, compute it from the latest stable tag and echo the exact resulting version back for confirmation.
-- Echo the final confirmed version string verbatim in your first status report; every later phase must use exactly that string. If at any point the user's wording and the confirmed version diverge, stop and re-confirm.
-
-## Preview tag naming
-
-- Use `<target>-preview.N` for every target, e.g. `1.0.0-beta.10-preview.3` or `1.1.0-preview.1`.
-- The canonical suffix is exactly `-preview.<digits>`. `build.yml` recognizes it before alpha/beta/rc classification and routes it to the preview-only path; any other tag containing `-preview` fails closed instead of being treated as a release.
-- A preview Release MUST be published with `isPrerelease=true` and `isLatest=false`. Any `*-latest` preview asset or preview-triggered `latest.json`, R2, Docker, or Helm publication is a pipeline failure.
-
-## Hard rules
-
-- Version files (Cargo.toml, Cargo.lock, README, flake.nix, Chart.yaml, rustfs.spec) are bumped ONCE, directly to `<target>`. Never write a `-preview.N` suffix into any version file. If `rustfs-release-version-bump` is ever asked for a `-preview` version, that is a pipeline bug — stop.
-- Preview Release assets are versioned and intentionally visible on the Releases page. Do not label them Latest or use them to update any latest distribution channel.
-- Tags have no `v` prefix. Always annotated: `git tag -a <tag> -m "Release <tag>"`.
-- The final tag MUST point at exactly `PREVIEW_HASH` — the commit the validated preview tag points at. Never tag current `main` HEAD (commits merged after validation are unvalidated), and never create an extra version-bump commit between preview and final.
-- When a previous deliverable exists, GitHub Release notes for the preview and final tags MUST use it as their shared comparison baseline: the most recently published non-preview Release before the target. Internal `-preview.N` Releases are explicitly excluded from that selection, even when they point at the same commit as the final tag. If no previous deliverable exists, omit `previous_tag_name` and record that GitHub's default baseline fallback was used.
-- Generated Release notes carry a workflow-management marker so retries can repair them. Before manually curating a generated body, remove that marker; unmarked non-placeholder notes are preserved by later workflow runs.
-- Phases run in order; a failure in any phase blocks everything after it. After the fix lands on main, restart from Phase 2 with the next preview iteration against the new `origin/main` hash — do not resume mid-pipeline against a stale hash.
-- Completing preview acceptance does not authorize the final tag. After Phases 3–5 pass, report the acceptance evidence and stop until the user explicitly confirms continuation. The original release request, an earlier confirmation, silence, or an automated follow-up does not satisfy this gate.
-- Confirmation is scoped to the reported `<target>`, `<preview-tag>`, and `PREVIEW_HASH`. A failed or repeated acceptance cycle, including any new preview iteration, invalidates prior confirmation and requires a new one.
-- If the release is abandoned after Phase 1 merged, main's version files claim a version that was never tagged. Either revert the bump PR or leave it to be overwritten by the next release — but tell the user explicitly and record the decision.
-- User-facing status updates in Chinese; commits, PR titles/bodies, and tag messages in English. No hard-wrapping in commit messages, PR bodies, or documentation prose — one logical line per sentence/paragraph, let soft wrap handle display.
-
-## Phase 0 — Preflight
-
-- `git status --short` clean; `git fetch origin main --tags`.
-- `gh auth status` works; confirm you can view `gh release list -L 3`.
-- Confirm the exact final target version with the user if not explicit.
-
-### Console release gate
-
-Complete this gate before changing any RustFS version file or creating any RustFS tag. RustFS `build.yml` downloads the asset returned by `repos/rustfs/console/releases/latest`, so a successful Console build alone is insufficient.
-
-1. Read the latest published Console tag and compare it with Console `main`:
-
-```bash
-CONSOLE_REPO="rustfs/console"
-CONSOLE_LATEST=$(gh api "repos/${CONSOLE_REPO}/releases/latest" --jq .tag_name)
-gh api "repos/${CONSOLE_REPO}/compare/${CONSOLE_LATEST}...main" \
-  --jq '{status, ahead_by, behind_by, commits: [.commits[] | {sha, message: .commit.message}]}'
-```
-
-- `ahead_by == 0`: no merged Console change is waiting for release. Still verify the current latest asset using step 4, then continue to Phase 1.
-- `ahead_by > 0` and `behind_by == 0`: publish Console before continuing. Report the merged commits and select the next unused `vX.Y.Z` tag. Default to the next patch version when the changes are fixes or backward-compatible UI work; stop for confirmation if a minor/major bump is plausible.
-- Any diverged history or `behind_by > 0`: stop and resolve the Console release baseline explicitly. Do not guess a range or publish RustFS.
-
-2. Clone/fetch `rustfs/console` into a scratch directory and record its exact `main` commit. Before creating a tag, check for a `v*` tag or Release workflow already associated with that hash. If one is in progress, wait for it instead of creating another version:
-
-```bash
-CONSOLE_SCRATCH=$(mktemp -d)
-gh repo clone "$CONSOLE_REPO" "$CONSOLE_SCRATCH/console"
-git -C "$CONSOLE_SCRATCH/console" fetch origin main --tags
-CONSOLE_HASH=$(git -C "$CONSOLE_SCRATCH/console" rev-parse origin/main)
-git -C "$CONSOLE_SCRATCH/console" tag --points-at "$CONSOLE_HASH" 'v*'
-gh run list -R "$CONSOLE_REPO" --workflow release.yml --commit "$CONSOLE_HASH" --limit 5
-```
-
-If no release exists or is running for `CONSOLE_HASH`, create the selected annotated tag at that exact hash and push it:
-
-```bash
-git -C "$CONSOLE_SCRATCH/console" tag -a "<console-tag>" -m "Release <console-tag>" "$CONSOLE_HASH"
-git -C "$CONSOLE_SCRATCH/console" push origin "<console-tag>"
-```
-
-Console tags include the `v` prefix. Pushing the tag triggers `.github/workflows/release.yml` (`🚀 Release`). Remove `CONSOLE_SCRATCH` after the gate completes.
-
-3. Find the exact tag run and wait for completion:
-
-```bash
-gh run list -R "$CONSOLE_REPO" --workflow release.yml --branch "<console-tag>" --limit 1
-gh run watch -R "$CONSOLE_REPO" "<console-run-id>" --exit-status
-```
-
-4. Block until the published Release is non-draft, the latest endpoint returns the expected tag, and `rustfs-console-<console-tag>.zip` is uploaded, non-empty, and carries a `sha256:` digest:
-
-```bash
-gh release view -R "$CONSOLE_REPO" "<console-tag>" --json isDraft,isPrerelease,assets,url
-test "$(gh api "repos/${CONSOLE_REPO}/releases/latest" --jq .tag_name)" = "<console-tag>"
-test "$(gh api "repos/${CONSOLE_REPO}/releases/tags/<console-tag>" \
-  --jq '[.assets[] | select(.name == "rustfs-console-<console-tag>.zip" and .state == "uploaded" and .size > 0 and (.digest | startswith("sha256:")))] | length')" -eq 1
-```
-
-Treat a missing/mismatched asset, digest, latest tag, or failed/cancelled workflow as BLOCKED. Do not start Phase 1 until the Console gate passes. Record `CONSOLE_TAG`, `CONSOLE_HASH`, Console run URL, and Release URL for the final report.
-
-## Phase 1 — Version bump to the final target (once)
-
-- If main's version files already read `<target>` (e.g. this is a restart after a failed preview), verify with `rg -n "<target>" Cargo.toml rustfs.spec helm/rustfs/Chart.yaml` and skip to Phase 2.
-- Otherwise invoke the `rustfs-release-version-bump` skill with the final `<target>` (NOT a preview version), full GitHub flow (commit/push/PR).
-- Get the PR merged into main. Record the resulting main commit:
-
-```bash
-git fetch origin main
-PREVIEW_HASH=$(git rev-parse origin/main)   # must contain the bump PR
-```
-
-`PREVIEW_HASH` is the single source of truth for the rest of the pipeline — report it to the user and reuse it verbatim in Phases 2 and 6. Both the preview tag and the final tag will point at it.
-
-## Phase 2 — Publish the preview tag
-
-```bash
-git tag -a "<preview-tag>" -m "Release <preview-tag>" "$PREVIEW_HASH"
-git push origin "<preview-tag>"
-```
-
-Pushing the tag triggers `.github/workflows/build.yml` ("Build and Release"); `docker.yml` chains off it via `workflow_run`.
-
-The preview run builds versioned artifacts and publishes them in a GitHub prerelease. Its latest-channel, R2, Docker, and Helm jobs must be skipped. Those publication paths run only after the final tag is pushed.
-
-On a restart (N+1), refresh `PREVIEW_HASH=$(git rev-parse origin/main)` first — it must contain the fix — and re-report it.
-
-## Phase 3 — CI and preview Release verification
-
-- Find and watch the tag build: `gh run list --workflow build.yml --branch "<preview-tag>" --limit 1` then `gh run watch <run-id>`. Every build matrix target must succeed (linux x86_64/aarch64 × musl/gnu, macos-aarch64, windows-x86_64).
-- Confirm the Release publication jobs (`create-release`, `upload-release-assets`, and `publish-release`) succeed while `update-latest-version` is skipped.
-- Verify `gh release view "<preview-tag>" --json isPrerelease,assets,url`: `isPrerelease` must be `true`, and the Release must contain all 6 versioned platform zips, checksums, SBOM, and provenance with no `-latest` assets. Confirm `gh api repos/{owner}/{repo}/releases/latest --jq .tag_name` does not return `<preview-tag>`.
-- Record `PREVIOUS_DELIVERABLE`, selected from published Releases by `publishedAt` after excluding the current tag and every `-preview.N` tag. Verify `gh release view "<preview-tag>" --json body --jq .body` contains `## What's Changed` and, when `PREVIOUS_DELIVERABLE` exists, `**Full Changelog**: https://github.com/rustfs/rustfs/compare/<PREVIOUS_DELIVERABLE>...<preview-tag>`. For a repository with no previous deliverable, verify a Full Changelog link exists and record the GitHub baseline fallback.
-- Confirm preview-triggered Docker and Helm jobs are skipped. Preview validation covers the built RustFS binaries, embedded console, and rc compatibility; Docker image construction and Helm publication are deferred to the final tag because the Dockerfiles consume GitHub Release assets.
-
-## Phase 4 — Run the artifact locally, verify the console
-
-Work inside the session scratchpad directory; never leave stray data dirs.
-
-```bash
-gh release download "<preview-tag>" -p "rustfs-macos-aarch64-v<preview-tag>.zip" -D "$SCRATCH"
-cd "$SCRATCH" && unzip -o rustfs-*.zip
-./rustfs --version        # must report the PREVIEW TAG (build::TAG), not the Cargo.toml version, plus expected short SHA
-mkdir -p data
-RUSTFS_ACCESS_KEY=rustfsadmin RUSTFS_SECRET_KEY=rustfsadmin ./rustfs ./data
-```
-
-Defaults: S3 endpoint `:9000`, embedded console `:9001`.
-
-Checks (all must pass):
-
-- `./rustfs --version` reports the preview tag name and the short SHA of `PREVIEW_HASH`. Reporting `<target>` without the `-preview.N` suffix means the build did not embed the tag — treat as FAIL and investigate before proceeding.
-- `curl -fsS http://localhost:9000/health/ready` returns ready.
-- Startup log shows the embedded console being served (this was the regression that `fix(release): require embedded console assets` guards).
-- Open `http://localhost:9001` in the browser: login with `rustfsadmin`/`rustfsadmin`; dashboard renders without JS console errors; create a bucket, upload a file, download it back (byte-identical), delete the object and bucket. Keep the server running for Phase 5.
-
-## Phase 5 — Validate with the latest rc client
-
-`rc` is the RustFS CLI client from <https://github.com/rustfs/cli>.
-
-- Ensure the latest release is installed: compare `rc --version` against `gh api repos/rustfs/cli/releases/latest --jq .tag_name`; update via `brew upgrade rustfs/tap/rc` (or download the release binary).
-- Point it at the preview server and run the command matrix, recording PASS/FAIL per command:
-
-```bash
-rc alias set preview http://localhost:9000 rustfsadmin rustfsadmin
-rc ls preview/
-rc mb preview/rel-check
-rc cp <local-file> preview/rel-check/
-rc stat preview/rel-check/<file>
-rc cat preview/rel-check/<file>          # matches source
-rc cp preview/rel-check/<file> ./out && cmp <local-file> ./out
-rc cp -r <local-dir>/ preview/rel-check/dir/
-rc find preview/rel-check --name "*"
-rc share download preview/rel-check/<file> --expire 1h   # presigned URL fetchable via curl
-rc rm preview/rel-check/<file> && rc rm -r --force preview/rel-check/dir
-rc rb preview/rel-check
-rc admin user list preview/
-rc admin user add preview/ relcheckuser relchecksecret12
-rc admin user remove preview/ relcheckuser
-rc alias remove preview
-```
-
-- Any FAIL blocks the release. Afterwards stop the server and delete the scratch data directory.
-
-### Manual confirmation gate
-
-After every Phase 3–5 check passes, report the target, preview tag, `PREVIEW_HASH`, preview Release URL, console result, and rc matrix, then explicitly ask the user whether to publish the final tag. End the turn without creating or pushing `<target>`.
-
-Continue to Phase 6 only after a new user reply explicitly confirms the reported target, preview tag, and commit. A clear affirmative reply to that exact report, such as `确认继续`, is sufficient; if the reply is ambiguous or any reported value changed, ask again.
-
-## Phase 6 — Publish the final tag on the validated commit
-
-No second version bump, no release branch. The final tag goes on the exact commit the preview validated:
-
-```bash
-git fetch origin --tags
-git rev-parse "<preview-tag>^{commit}"          # must equal PREVIEW_HASH — abort if not
-git tag -a "<target>" -m "Release <target>" "$PREVIEW_HASH"
-git push origin "<target>"
-```
-
-- CI rebuilds from the same source; the only changed input is the tag name, so the binary now self-reports `<target>`.
-- Verify the final tag's complete publication path: all matrix and release jobs green; `gh release view "<target>"` shows the full versioned and `-latest` asset set plus checksums, SBOM, and provenance; Docker and Helm workflows succeed; `latest.json` points to `<target>`. A stable target must have `isPrerelease=false` and `isLatest=true`. An alpha/beta/rc target must have `isPrerelease=true`; GitHub does not permit prereleases to be Latest, but the project `latest.json` still advances to the final non-preview target.
-- Verify the final Release body contains `## What's Changed` and a Full Changelog link. When `PREVIOUS_DELIVERABLE` exists, the link MUST be `https://github.com/rustfs/rustfs/compare/<PREVIOUS_DELIVERABLE>...<target>` and the baseline MUST equal the preview Release baseline; for example, both `1.0.0-beta.12-preview.1` and `1.0.0-beta.12` compare from `1.0.0-beta.11`.
-- Optionally spot-check `./rustfs --version` from a final-tag artifact — it must report `<target>`.
-
-## Output contract
-
-Always report:
-
-- Console gate result: previous/latest Console tags, whether merged changes required a release, `CONSOLE_HASH`, and Console run/Release URLs when a release was published.
-- Target version, preview tag(s) used, `PREVIEW_HASH` (which both tags point at).
-- Manual confirmation gate status (`WAITING_FOR_CONFIRMATION` or `CONFIRMED`) and its exact target, preview tag, and `PREVIEW_HASH`.
-- Per-phase result (PASS/FAIL/BLOCKED) with key evidence: preview and final Release URLs, preview `isPrerelease`/`isLatest` state, final latest-channel state, console check results, and the rc command matrix.
-- Any deviation from this pipeline and why the user approved it.

.agents/skills/rustfs-release-version-bump/SKILL.md

@@ -1,120 +0,0 @@
----
-name: rustfs-release-version-bump
-description: "Prepare the version-file and release-asset bump for an exact RustFS alpha/beta/stable target, with verification and optional commit/push/PR delivery. Use for an explicit version bump or when invoked by the release-publish workflow."
----
-# RustFS Release Version Bump
-
-Use this skill to publish a RustFS release (alpha, beta, or stable) with a minimal, auditable diff and a complete ship flow (`edit -> verify -> commit -> push -> PR`).
-
-Validated baseline: release pattern used in PR `#2957`.
-
-## Required inputs
-
-- Exact target version, for example `1.0.0-beta.4`.
-- Delivery scope:
-- Local only (`edit/verify`).
-- Local + git (`commit/push`).
-- Full GitHub flow (`commit/push/PR`).
-
-If target version is missing or ambiguous, stop and ask before editing.
-
-Reject any target version containing `-preview`: preview identifiers are tag-only (see `rustfs-release-publish`) and must never be written into version files. If asked for one, stop and point to the release pipeline instead of editing.
-
-## Read before editing
-
-- `AGENTS.md` (root and nearest path-specific files).
-- `.github/pull_request_template.md`.
-- Current branch status and diff against `origin/main`.
-
-## Default release file scope
-
-Treat the following file list as the default checklist for each release bump:
-
-- `Cargo.toml`
-- `Cargo.lock`
-- `README.md`
-- `README_ZH.md`
-- `flake.nix`
-- `helm/rustfs/Chart.yaml`
-- `rustfs.spec`
-
-Only drop a file when the current repository release process clearly no longer requires it.
-
-## Hard release policy
-
-- Docker doc tags use `<version>` (for example `rustfs/rustfs:1.0.0-beta.4`), not `v<version>`.
-- Helm chart version mapping follows `beta.N -> 0.N.0`.
-- `rustfs.spec` `Release` uses prerelease suffix only (for example `beta.4`).
-- Do not change these rules without explicit confirmation.
-
-## Step-by-step workflow
-
-1. Confirm intent and isolate scope
-- Confirm target version string exactly.
-- Confirm whether user requested local-only or full GitHub flow.
-- Inspect current branch and ensure only release-related files are touched for this task.
-
-2. Update workspace versions
-- Bump `[workspace.package].version` in `Cargo.toml`.
-- Bump internal workspace crate dependency versions in `Cargo.toml`.
-- Update `Cargo.lock` so workspace package versions match target version.
-- Re-scan for partial leftovers.
-
-3. Update release assets
-- `README.md` and `README_ZH.md`: update versioned Docker examples to target version.
-- `flake.nix`: update package version to target version.
-- `helm/rustfs/Chart.yaml`:
-- `appVersion` = target version.
-- `version` follows chart mapping rule, for example:
-- `1.0.0-beta.3` -> `0.3.0`
-- `1.0.0-beta.4` -> `0.4.0`
-- `rustfs.spec`:
-- Set `Release` to prerelease suffix (example `beta.4`).
-- Add/update top changelog entry with exact format:
-- `* Thu May 20 2026 houseme <housemecn@gmail.com>`
-- `- Update RPM package to RustFS 1.0.0-beta.4`
-- Changelog identity and time must come from current environment:
-- `git config --get user.name`
-- `git config --get user.email`
-- `date '+%a %b %d %Y'`
-- Changelog version text must match target release version exactly.
-
-4. Verify before shipping
-- Run:
-- `make pre-commit`
-- If `make pre-commit` fails, return `BLOCKED` with root cause and do not silently widen scope to fix unrelated issues unless user asks.
-
-5. Commit strategy
-- Preferred split when both parts changed:
-- `chore(release): prepare <version>` for `Cargo.toml` and `Cargo.lock`.
-- `chore(release): align release assets for <version>` for docs and packaging files.
-- If user asks for one commit, use one commit.
-- Stage only intended release files; do not include unrelated working tree changes.
-
-6. Push and PR
-- Push branch:
-- `git push -u origin <branch>` (first push), or `git push` (tracking already exists).
-- Create PR with template headings unchanged:
-- `gh pr create --base main --head <branch> --title ... --body-file ...`
-- PR title/body must be English.
-- Use `N/A` for non-applicable template sections.
-- Include verification commands and any `BLOCKED` reason clearly.
-
-## Recommended check commands
-
-- `git status --short --branch`
-- `git diff --name-only origin/main...HEAD`
-- `git diff --stat origin/main...HEAD`
-- `rg -n "<old_version>|<new_version>" Cargo.toml Cargo.lock README.md README_ZH.md flake.nix helm/rustfs/Chart.yaml rustfs.spec`
-- `make pre-commit`
-
-## Output contract
-
-When using this skill, always report:
-
-- Target version.
-- Files changed.
-- Any assumptions or uncertainties requiring confirmation.
-- Verification result (`PASSED` or `BLOCKED`) with key evidence.
-- Commit message(s) used.
-- Push status and PR URL when GitHub flow is requested.

.agents/skills/rustfs-release-version-bump/agents/openai.yaml

@@ -1,4 +0,0 @@
-interface:
-  display_name: "RustFS Release Bump"
-  short_description: "Prepare RustFS release branches like PR #2957."
-  default_prompt: "Use $rustfs-release-version-bump to prepare and verify an exact RustFS release-version bump."

.agents/skills/security-advisory-lessons/SKILL.md

@@ -1,40 +0,0 @@
----
-name: security-advisory-lessons
-description: Perform a dedicated RustFS security/advisory review for authn/authz, IAM, RPC trust, paths, secrets, browser isolation, encryption, Object Lock, or other security boundaries. Use only when the user requests a security/advisory review or an adversarial review explicitly escalates to the full advisory map; do not auto-load solely because code touches a sensitive path.
----
-
-# RustFS Security Advisory Lessons
-
-Use this skill as the deep security lens. For a normal adversarial review with a
-matched security surface, the concise security reference under
-`adversarial-validation` is sufficient.
-
-## Workflow
-
-1. Freeze the exact diff/head and identify the changed trust boundaries.
-2. Read [advisory-patterns.md](references/advisory-patterns.md), then apply only
-   the matching sections. Useful headings are
-   auth/admin, IAM/STS/OIDC, policy/plugins, S3/copy/multipart, protocols, paths,
-   secrets/logging/RPC, browser/CORS/proxy, SSE, Object Lock, and serde.
-3. Trace unauthenticated, low-privilege, wrong-action/owner/bucket, malformed,
-   and default-config cases. Security decisions must fail closed.
-4. Require a focused negative regression test for the bypass/exploit form, not
-   only the intended success path. State residual risk when a test is impractical.
-5. Report proven vulnerabilities separately from defense-in-depth hardening.
-
-When advisory currentness matters, fetch the live inventory instead of treating
-the reference as a status mirror:
-
-```bash
-gh api repos/rustfs/rustfs/security-advisories --paginate \
-  --jq '.[] | {ghsa_id,state,severity,summary,updated_at}'
-```
-
-Fetch an individual advisory only when the live summary indicates a new or
-changed lesson.
-
-## Finding Standard
-
-Each finding includes severity, `file:line`, attacker prerequisites, concrete
-input/path, impact, smallest safe fix, and a regression check. Do not exaggerate
-unauthenticated impact when the actual issue requires authenticated low privilege.

.agents/skills/security-advisory-lessons/agents/openai.yaml

@@ -1,4 +0,0 @@
-interface:
-  display_name: "Security Advisory Lessons"
-  short_description: "Apply advisory lessons in reviews."
-  default_prompt: "Use $security-advisory-lessons for a dedicated RustFS security review grounded in past advisories."

.agents/skills/security-advisory-lessons/references/advisory-patterns.md

@@ -1,147 +0,0 @@
-# RustFS Advisory Pattern Map
-
-This file is a lesson map, not an advisory inventory mirror. It keeps durable security patterns distilled from RustFS GitHub Security Advisories.
-
-When current advisory state, severity, URLs, or full text matters, fetch it live:
-
-```bash
-gh api repos/rustfs/rustfs/security-advisories --paginate \
-  --jq '.[] | {ghsa_id,state,severity,summary,updated_at}'
-gh api repos/rustfs/rustfs/security-advisories/<GHSA_ID>
-```
-
-Update this file only when an advisory adds or changes a reusable lesson, affected surface, validation pattern, or regression-test expectation. Do not update it for state-only, URL-only, count-only, or timestamp-only changes.
-
-## Pattern Index
-
-### Admin authorization and route exposure
-
-- `GHSA-pfcq-4gjr-6gjm`: notification target endpoints accepted authenticated users but skipped admin authorization. Lesson: distinguish authn from authz; admin target CRUD must call the operation-specific admin authorization path.
-- `GHSA-mm2q-qcmx-gw4w`: `ListServiceAccount` used `UpdateServiceAccountAdminAction`, while update lacked target ownership checks. Lesson: exact action constants and ownership checks are both required; information disclosure can chain into secret rotation and takeover.
-- `GHSA-vcwh-pff9-64cc`: `ImportIam` checked `ExportIAMAction` for an import/write operation. Lesson: every admin handler must authorize the action it actually performs.
-- `GHSA-jqmc-mg33-v45g` and `GHSA-8784-9m7f-c6p6`: `/profile/cpu` and `/profile/memory` were whitelisted from auth and allowed expensive diagnostics plus path disclosure. Lesson: profiling/debug endpoints need admin auth, opt-in, rate limits, and non-sensitive responses.
-- `GHSA-f5cv-v44x-2xgf`: `/rustfs/admin/v3/metrics` accepted any authenticated IAM user and skipped admin authorization. Lesson: read-only metrics and diagnostic admin endpoints still require an operation-specific admin action check.
-- `GHSA-796f-j7xp-hwf4`: `/rustfs/admin/v3/list-remote-targets` checked only that credentials existed and leaked replication target credentials. Lesson: replication target reads are privileged admin operations, and stored remote credentials require strict authz plus response redaction review.
-- `GHSA-xp32-gxq2-3v52`: console license metadata endpoint was public and exposed subject and expiration fields. Lesson: management metadata endpoints should require admin auth or return only coarse public status.
-
-### IAM import, service accounts, and privilege boundaries
-
-- `GHSA-566f-q62r-wcr8`: `ImportIam` accepted attacker-controlled service account `parent`, `claims`, `accessKey`, and `secretKey`, enabling persistent backdoor accounts under root. Lesson: imported IAM payloads are untrusted data and must be validated against privilege boundaries.
-- `GHSA-3495-h8r9-gfqg`: `ExportIAM` wrote regular-user and service-account secret keys into exported ZIP data. Lesson: IAM export is a credential-disclosure boundary; redact, seal, or strictly justify every exported secret before treating export permission as safe.
-- `GHSA-5354-r3w2-34m8`: `AddServiceAccount` checked `CreateServiceAccountAdminAction` but trusted caller-supplied `target_user`, allowing service accounts under the root parent. Lesson: service-account create paths must validate parent ownership or root/admin authority, not only the create action.
-- `GHSA-xgr5-qc6w-vcg9`: `deny_only=true` skipped allow checks and let restricted service accounts mint unrestricted children. Lesson: deny-only logic must never become implicit allow for privilege creation.
-- `GHSA-mm2q-qcmx-gw4w`: leaked service account access keys plus update-without-ownership formed an escalation chain. Lesson: service-account identifiers are security-sensitive because update APIs consume them.
-
-### STS, OIDC, and federation flows
-
-- `GHSA-5qfg-mf7r-jp3w` and `GHSA-3473-5353-xhwh`: `AssumeRoleWithWebIdentity` was reachable through unauthenticated `POST /` routing and could issue temporary credentials from crafted web identity input. Lesson: every STS route needs explicit SigV4 or trusted identity-provider validation before role assumption, and unauthenticated exemptions must be narrowed to the exact action with uniform failure responses.
-- `GHSA-jxrr-r6pv-h958`: unsigned JWT issuer data was decoded before verification to select an OIDC provider, and distinguishable failures could expose provider configuration. Lesson: web-identity routing may be unauthenticated, but pre-verification claims are untrusted routing hints; bound and rate-limit the request, normalize public errors, and verify signature, issuer, audience, and expiration before issuing credentials.
-- `GHSA-ccrv-v8v9-ch9q`, `GHSA-48rf-7j3q-3hfv`, and `GHSA-xvfh-7c9g-hpw2`: service-account-controlled material could self-sign JWT session tokens with forged policy claims, and missing `exp` was accepted for service-account tokens. Lesson: session tokens must be signed by a trusted issuer/key path, enforce required claims and expiration, and reject self-signed or principal-controlled tokens.
-- `GHSA-9pjf-w3c2-m32r`, `GHSA-4x2q-cpx9-9h26`, and `GHSA-xvpm-p3f7-34c3`: public OIDC authorize/callback flows trusted request `Host` or forwarded scheme when building credential-bearing redirects. Lesson: OIDC redirects must use configured allowlisted origins and trusted-proxy handling; never derive the post-login credential destination from direct client headers.
-- `GHSA-m479-9x88-94w6`, `GHSA-frwq-mfqx-83p8`, `GHSA-q9q8-rf9r-fg9f`, and `GHSA-j5c2-hhf7-6gf5`: OIDC validation accepted attacker-controlled discovery URLs because hostname checks rejected only literal forbidden IPs, allowing DNS rebinding SSRF. Lesson: outbound federation URL validation must resolve and classify hostnames at the connection boundary and reject loopback, private, link-local, and rebound addresses.
-
-### IAM policy conditions and external policy plugins
-
-- `GHSA-6r96-hmgc-726c`: request headers collided with lowercase server-derived condition keys such as `userid`, `groups`, `versionid`, and JWT/LDAP claims. Lesson: never let caller-controlled headers append to or replace server-derived policy context; reserve trusted condition keys and keep intentional request-header keys separate.
-- `GHSA-v9cp-qfw9-9pfp`: quantified negated string conditions applied negation after aggregation, transposing `ForAllValues` and `ForAnyValue` semantics. Lesson: push negation into the per-value predicate for quantified operators and test partially overlapping multi-value sets.
-- `GHSA-5w8r-p896-6vq2`: OPA policy mode skipped `ExistingObjectTag/*` loading, so tagged objects looked untagged to external policies. Lesson: external authorization plugins need the same object-tag and request context as built-in policy evaluation before they decide.
-
-### S3 object actions, copy, multipart, and upload policy validation
-
-- `GHSA-3ppv-fx5m-m749`: explicit `versionId` reads and copy sources authorized `s3:GetObject` instead of `s3:GetObjectVersion`. Lesson: version-specific object access must select version-specific actions for direct reads, `CopyObject`, and `UploadPartCopy`, with tests proving the backend is not reached on denial.
-- `GHSA-x298-9x87-fvjq`: anonymous `ListObjectVersions` fell back to `ListBucket` and returned before public-access-block gates. Lesson: compatibility fallbacks must converge on the same post-authorization checks as direct grants, especially `RestrictPublicBuckets` and anonymous data-plane denies.
-- `GHSA-mx42-j6wv-px98`: `UploadPartCopy` missed source authorization and allowed cross-bucket object exfiltration. Lesson: multipart copy must enforce the same source and destination contract as `CopyObject`.
-- `GHSA-wfxj-ph3v-7mjf`: `UploadPartCopy` checked source and destination independently but missed destination copy-source policy constraints. Lesson: source read and destination write checks are not sufficient when policy constrains allowed copy sources.
-- `GHSA-w5fh-f8xh-5x3p`: presigned POST accepted uploads without enforcing signed policy conditions. Lesson: parse and enforce all POST policy constraints server-side, including size, key prefix, and content type.
-
-### Protocol frontends and IAM parity
-
-- `GHSA-3g29-xff2-92vp`: FTP `RETR` and `SIZE`/`MDTM` read paths authenticated the user but skipped IAM before calling storage. Lesson: non-HTTP protocol frontends must enforce the same per-operation authorization as the S3 API before backend access.
-- `GHSA-g3vq-vv42-f647`: FTPS `MKD` called `create_bucket` without checking `s3:CreateBucket`. Lesson: protocol command handlers need action-specific checks even when sibling handlers already authorize correctly.
-- `GHSA-3p3x-734c-h5vx`: FTPS and WebDAV compared secret keys with early-return string equality, while FTPS also returned distinguishable invalid-user and invalid-password failures. Lesson: password-style protocol auth needs constant-time secret comparison, indistinguishable failures where practical, and rate limiting.
-
-### Filesystem paths and object key traversal
-
-- `GHSA-pq29-69jg-9mxc`: RPC `read_file_stream` joined untrusted paths under a volume directory without canonical boundary checks. Lesson: `PathBuf::join` plus length checks are not path security.
-- `GHSA-8r6f-hmq2-28rg`: object keys containing traversal sequences bypassed bucket/object authorization when mapped to filesystem paths. Lesson: reject traversal at object-key parsing and verify final storage paths remain under the expected bucket/key root.
-- `GHSA-f4vq-9ffr-m8m3`: Snowball auto-extract accepted archive entries such as `../victim-bucket/object`, authorized the raw attacker-bucket path, then storage path cleaning crossed bucket boundaries. Lesson: archive entries become object keys and need traversal rejection plus consistent authz/storage normalization before writes.
-
-### Secrets, defaults, and cryptographic misuse
-
-- `GHSA-j59h-h7q5-q348`, `GHSA-3wm5-wpm5-hmfm`, `GHSA-6wc8-xm48-qhmx`, `GHSA-9gf3-jx4p-4xxf`, `GHSA-63xc-c3w3-m2cf`, and `GHSA-ch63-6q4v-hwp5`: RustFS shipped known default root credentials that could authenticate to S3, admin APIs, IAM, KMS, console, and token-signing surfaces. Lesson: root credentials must be operator-provided or generated per install; known defaults and warnings are not acceptable for network-reachable deployments.
-- `GHSA-h956-rh7x-ppgj`: gRPC used the hard-coded token `rustfs rpc` on both client and server. Lesson: source-visible shared tokens are authentication bypasses.
-- `GHSA-r5qv-rc46-hv8q`: internode RPC HMAC secret fell back to the public default `rustfsadmin`. Lesson: RPC/internode auth must fail closed instead of silently using public defaults.
-- `GHSA-75fx-qg6f-8rm7` and `GHSA-68cw-96m3-h2cf`: internode RPC secrets were derivable from known root credentials, making raw storage RPC signatures forgeable when explicit RPC secrets were unset. Lesson: RPC auth keys must be independent random secrets, never derived from S3 root credentials, and raw storage RPC should not share the public S3 listener without an internode-only boundary.
-- `GHSA-m77q-r63m-pj89`: STS JWTs used the root secret key as the shared token signing key, allowing token forgery when the root secret was known. Lesson: STS signing keys need key separation, rotation, and key IDs; do not reuse root credentials for JWT/HMAC signing.
-- `GHSA-923g-jp7v-f97f`: license verification embedded a production RSA private key and used private-key decryption as authenticity. Lesson: ship verifying/public keys only and use real signature verification.
-
-### Sensitive logging and debug output
-
-- `GHSA-r54g-49rx-98cr`: STS credentials were logged at info level. Lesson: generated credentials must never be logged in plaintext.
-- `GHSA-8cm2-h255-v749`: debug logs leaked session tokens, secret keys, JWT claims, and raw STS response bodies. Lesson: redaction must cover custom `Debug` implementations and dependency response-body logging.
-- `GHSA-333v-68xh-8mmq`: invalid RPC signature logging included the shared HMAC secret and expected signature. Lesson: error paths often leak secrets; never log raw secrets or derived authenticators.
-
-### RPC input validation and panic safety
-
-- `GHSA-gw2x-q739-qhcr`: malformed gRPC `GetMetrics` payloads reached `unwrap()` on deserialization and caused remote DoS. Lesson: every network/RPC deserialization failure returns an error, not a panic.
-- `GHSA-h956-rh7x-ppgj` and `GHSA-r5qv-rc46-hv8q`: weak RPC auth increased reachability of otherwise internal handlers. Lesson: panic bugs become more severe when internode auth is weak or defaulted.
-- `GHSA-c667-rgrv-99vj`: NodeService authentication signed the service prefix instead of the concrete generated method path, so valid metadata for one RPC could be replayed to another method during the timestamp window. Lesson: RPC HMAC payloads must bind exact gRPC method path, HTTP method surrogate, timestamp, and secret.
-
-### Browser, CORS, and console isolation
-
-- `GHSA-v9fg-3cr2-277j`: object preview rendered attacker-controlled HTML in a same-origin iframe, exposing console credentials stored in `localStorage`. Lesson: user content must be origin-isolated from the console and protected with `nosniff`, CSP, and strict content-type handling.
-- `GHSA-7gcx-wg4x-q9x6`: an incomplete preview fix reintroduced extension-based PDF detection and bypassed the sandboxed fallback for attacker-controlled content. Lesson: browser-surface fixes need regression tests for alternate viewers and file-type branches, and preview trust must come from validated content type plus sandboxing rather than object names.
-- `GHSA-x5xv-223c-8vm7`: default CORS reflected arbitrary origins with credentials. Lesson: never combine reflected origins with `Access-Control-Allow-Credentials: true`; default should be fail-closed.
-
-### Trusted proxy and source IP conditions
-
-- `GHSA-fc6g-2gcp-2qrq`: `aws:SourceIp` trusted client-supplied `X-Forwarded-For` or `X-Real-IP`. Lesson: forwarded IP headers are valid only behind configured trusted proxies; direct clients use socket peer IP.
-
-### SSE and on-disk storage invariants
-
-- `GHSA-xrrf-67jm-3c2r`: SSE metadata reported encryption while reader composition bypassed `EncryptReader` and stored plaintext. Lesson: test actual bytes on disk and wrapper order, not only API metadata.
-
-### Object Lock and retention invariants
-
-- `GHSA-j548-9grx-fh4f`: Object Lock enforcement treated unreadable, fabricated, or unparsable bucket metadata as absent configuration and allowed retained objects to be deleted or expired. Lesson: retention must fail closed unless Object Lock absence is authoritative, and every delete, lifecycle, scanner, force-delete, and default-retention path needs the same state distinction.
-
-### Serde deserialization and input validation
-
-- No `#[serde(deny_unknown_fields)]` found across the entire codebase. Lesson: all structs deserialized from untrusted input (S3 API XML/JSON, lifecycle rules, bucket policies, replication configs) should have `#[serde(deny_unknown_fields)]` to reject malformed or adversarial payloads.
-- `#[serde(default)]` on security-critical fields silently accepts missing values as zero/empty. Lesson: when a field has security implications (retention days, permissions, limits), validate the deserialized value explicitly rather than relying on defaults.
-- Integer fields deserialized from user input and cast with `as` (e.g., `i32 as u32`) can wrap negative values to large positives. Lesson: validate ranges before casting; use `try_into()` or clamp.
-- XML config typos (e.g., `"NoncurentDays"` instead of `"NoncurrentDays"`) are silently accepted when `deny_unknown_fields` is absent. Lesson: strict deserialization prevents silent misconfiguration that could cause data loss or unexpected retention behavior.
-
-## Useful Search Seeds
-
-Use these targeted searches when a diff touches security-sensitive code:
-
-```bash
-rg -n "validate_admin_request|check_permissions|AdminAction::|deny_only|is_allowed" rustfs crates
-rg -n "authorize_operation|FtpsDriver|SftpDriver|RETR|MKD|SIZE|MDTM|CreateBucket|GetObject|HeadObject" crates/protocols rustfs
-rg -n "UploadPartCopy|upload_part_copy|CompleteMultipart|PostObject|content-length-range|starts-with" rustfs crates
-rg -n "ListBucketVersions|GetObjectVersion|versionId|VersionId|ExistingObjectTag|ForAllValues|ForAnyValue|POLICY_PLUGIN|opa" rustfs crates
-rg -n "normalize_extract_entry_key|Snowball|auto-extract|PathBuf::join|canonicalize|\\.\\.|x-forwarded-for|x-real-ip|SourceIp" rustfs crates
-rg -n "DEFAULT_SECRET|DEFAULT_ACCESS|TEST_PRIVATE_KEY|rustfs rpc|RUSTFS_RPC_SECRET" rustfs crates
-rg -n "TONIC_RPC_PREFIX|verify_rpc_signature|check_auth|NodeServiceServer|x-rustfs-signature" rustfs crates
-rg -n "debug!|trace!|info!|error!|\\?resp|\\?merged_config|session_token|secret_key" rustfs crates
-rg -n "HashReader|EncryptReader|SSE|server-side encryption|Access-Control-Allow-Credentials|Origin" rustfs crates
-rg -n "ObjectLock|object_lock|retention|COMPLIANCE|GOVERNANCE|delete_prefix|lifecycle|scanner" rustfs crates
-rg -n "deny_unknown_fields|serde.default|as u32|as usize|as i32" rustfs crates
-```
-
-## Minimum Regression Test Expectations
-
-- Authz fixes: include unauthenticated, valid low-privilege, wrong-action, correct-action, owner, non-owner, and root/admin cases as applicable.
-- Protocol frontend authz fixes: include denied `RETR`, `SIZE`/`MDTM`, `MKD`, bucket probe, and sibling allowed-operation cases, and assert denied paths do not reach the storage backend.
-- IAM fixes: include import/update/list service-account cases with attacker-controlled parent, claims, access key, secret key, and policy.
-- Copy/upload fixes: include cross-bucket, cross-user, source-denied, destination-denied, copy-source-condition, and multipart completion cases.
-- Version-action fixes: include historical UUID, explicit current version, `null`, range, partNumber, presigned, STS/session, service-account, anonymous bucket-policy, copy source, and multipart-copy source cases.
-- Policy-condition fixes: include reserved-key header collisions, missing keys, partially overlapping multi-value sets, plugin mode, and built-in policy mode.
-- Path fixes: include encoded traversal, absolute path, nested traversal, archive entries with `..`, valid object keys that resemble traversal text but should be rejected, and canonical bucket/prefix boundary checks.
-- Logging fixes: assert redacted output for structs and response bodies that may contain credentials.
-- IAM export fixes: assert exported archives omit plaintext user and service-account secrets unless the format deliberately encrypts or seals them.
-- RPC auth fixes: include captured metadata replay across two concrete methods, stale timestamps, wrong path, wrong method surrogate, wrong secret, and valid same-method calls.
-- Browser/CORS fixes: assert no credentials on reflected/default origins, correct behavior for explicit allowlists, and no same-origin script execution for previewed object content.
-- SSE fixes: inspect stored bytes and verify API metadata, read-back behavior, and on-disk ciphertext together.
-- Object Lock fixes: include unreadable metadata, fabricated metadata defaults, unparsable config, confirmed absent config, COMPLIANCE/GOVERNANCE retention, lifecycle expiry, scanner sweeps, and force-delete paths.

.agents/skills/test-coverage-improver/SKILL.md

@@ -1,66 +0,0 @@
----
-name: test-coverage-improver
-description: Run project coverage checks, rank high-risk gaps, and propose high-impact tests to improve regression confidence for changed and critical code paths before release.
----
-
-# Test Coverage Improver
-
-Use this skill when you need a prioritized, risk-aware plan to improve tests from coverage results.
-
-## Usage assumptions
-- Focus scope is either changed lines/files, a module, or the whole repository.
-- Coverage artifact must be generated or provided in a supported format.
-- If required context is missing, call out assumptions explicitly before proposing work.
-
-## Workflow
-
-1. Define scope and baseline
-   - Confirm target language, framework, and branch.
-   - Confirm whether the scope is changed files only or full-repo.
-
-2. Produce coverage snapshot
-   - Rust: `cargo llvm-cov` (or `cargo tarpaulin`) with existing repo config.
-   - JavaScript/TypeScript: `npm test -- --coverage` and read `coverage/coverage-final.json`.
-   - Python: `pytest --cov=<pkg> --cov-report=json` and read `coverage.json`.
-   - Collect total, per-file, and changed-line coverage.
-
-3. Rank highest-risk gaps
-   - Prioritize changed code, branch coverage gaps, and low-confidence boundaries.
-   - Apply the risk rubric in [coverage-prioritization.md](references/coverage-prioritization.md).
-   - Keep shortlist to 5–8 gaps.
-   - For each gap, capture: file, lines, uncovered branches, and estimated risk score.
-
-4. Propose high-impact tests
-   - For each shortlisted gap, output:
-     - Intent and expected behavior.
-     - Normal, edge, and failure scenarios.
-     - Assertions and side effects to verify.
-     - Setup needs (fixtures, mocks, integration dependencies).
-     - Estimated effort (`S/M/L`).
-
-5. Close with validation plan
-   - State which gaps remain after proposals.
-   - Provide concrete verification command and acceptance threshold.
-   - List assumptions or blockers (environment, fixtures, flaky dependencies).
-
-## Output template
-
-### Coverage Snapshot
-- total / branch coverage
-- changed-file coverage
-- top missing regions by size
-
-### Top Gaps (ranked)
-- `path:line-range` | risk score | why critical
-
-### Test Proposals
-- `path:line-range`
-  - Test name
-  - scenarios
-  - assertions
-  - effort
-
-### Validation Plan
-- command
-- pass criteria
-- remaining risk

.agents/skills/test-coverage-improver/agents/openai.yaml

@@ -1,4 +0,0 @@
-interface:
-  display_name: "Test Coverage Improver"
-  short_description: "Find top uncovered risk areas and propose high-impact tests."
-  default_prompt: "Run coverage checks, identify largest gaps, and recommend highest-impact test cases to improve risk coverage."

.agents/skills/test-coverage-improver/references/coverage-prioritization.md

@@ -1,25 +0,0 @@
-# Coverage Gap Prioritization Guide
-
-Use this rubric for each uncovered area.
-
-Score = (Criticality × 2) + CoverageDebt + (Volatility × 0.5)
-
-- Criticality:
-  - 5: authz/authn, data-loss, payment/consistency path
-  - 4: state mutation, cache invalidation, scheduling
-  - 3: error handling + fallbacks in user-visible flows
-  - 2: parsing/format conversion paths
-  - 1: logging-only or low-impact utilities
-
-- CoverageDebt:
-  - 0: 0–5 uncovered lines
-  - 1: 6–20 uncovered lines
-  - 2: 21–40 uncovered lines
-  - 3: 41+ uncovered lines
-
-- Volatility:
-  - 1: stable legacy code with few recent edits
-  - 2: changed in last 2 releases
-  - 3: touched in last 30 days or currently in active PR
-
-Sort by score descending, then by business impact.

.agents/skills/tier-debug/SKILL.md

@@ -1,34 +0,0 @@
----
-name: tier-debug
-description: Debug ILM tiering / lifecycle transition issues — NoSuchVersion on tier GET, restore failures, xl.meta inspection, remote-tier versionId tracing. Use when investigating tiered/transitioned objects, warm backends, or transition metadata.
----
-
-# Tier / ILM Debugging
-
-Full playbook: [docs/operations/tier-ilm-debugging.md](../../../docs/operations/tier-ilm-debugging.md)
-— read it before changing tier code.
-
-Quick moves:
-
-```bash
-# Inspect transition metadata on disk (one xl.meta per erasure shard disk)
-cargo run -p rustfs-filemeta --example dump_fileinfo -- "/path/to/{bucket}/{object}/xl.meta"
-
-# Trace what versionId is sent to the remote tier
-RUST_LOG=rustfs_ecstore::bucket::lifecycle=debug ./target/debug/rustfs …
-```
-
-Interpretation:
-
-- `transition_ver_id: <none>` → correct for an unversioned tier bucket; no
-  `versionId` must be sent on tier GET/DELETE.
-- `transition_ver_id: 00000000-…` (nil) → corrupt legacy write-back; readers
-  must filter it out, never send it.
-- Empty-string `transitioned-versionID` metadata under both
-  `x-rustfs-internal-*` and `x-minio-internal-*` keys → object went to an
-  unversioned tier bucket.
-
-Code entry points: `crates/ecstore/src/bucket/lifecycle/bucket_lifecycle_ops.rs`
-(ILM actions), `crates/ecstore/src/services/tier/` (warm backends),
-`crates/filemeta/src/filemeta/version.rs` (metadata read/write + regression
-tests).

.cargo/config.toml

@@ -1,33 +0,0 @@
-# Copyright 2024 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-# RustFS Cargo configuration
-
-# NOTE: `--cfg tokio_unstable` is deliberately NOT set here.
-#
-# It used to be a global `[build] rustflags` entry so that the (default-off)
-# `dial9` telemetry feature could compile. That made every build depend on
-# Tokio's unstable, non-semver API, and it broke silently whenever a caller
-# exported their own RUSTFLAGS — an environment RUSTFLAGS replaces the value
-# from this file rather than appending to it.
-#
-# The flag is now scoped to telemetry builds, which opt in explicitly:
-#
-#     make build-profiling
-#     RUSTFLAGS="--cfg tokio_unstable" cargo build --features dial9
-#
-# `crates/obs/build.rs` fails the compile if the `dial9` feature is on without
-# the flag, so the two can no longer drift apart unnoticed.
-#
-# For CPU profiling, add `-C force-frame-pointers=yes` to that RUSTFLAGS value.

.claude/skills

@@ -1 +0,0 @@
-../.agents/skills

.config/coverage-baselines.toml

@@ -1,20 +0,0 @@
-# Report-only calibration baseline from https://github.com/rustfs/rustfs/actions/runs/29394996173.
-# Update counts only with a linked coverage run and a reviewed explanation.
-phase = "report-only"
-allowed_drop_percentage_points = 1.0
-
-[crates."crates/iam"]
-covered = 5149
-count = 8131
-
-[crates."crates/kms"]
-covered = 2950
-count = 4200
-
-[crates."crates/policy"]
-covered = 4636
-count = 5464
-
-[crates."crates/crypto"]
-covered = 469
-count = 494

.config/e2e-full-selection.txt

@@ -1,2 +0,0 @@
-sha256-darwin=d6aa36cfaae2c4d8590482c7e47138c5965b335b34a75f50d11ffc3366e9021e
-sha256-linux=c8315465f50c194faee36141cdbb1e15e59271e524d948564a69e2d5eb408f2a

.config/e2e-nightly-selection.txt

@@ -1 +0,0 @@
-sha256=9b9bc336b43b70d0e06e0adb5455bf035bb18945d85d60936eb6fe4d48e0e680

.config/e2e-protocols-selection.txt

@@ -1,2 +0,0 @@
-sha256-darwin=55534a97fbd376f64c8f6c341d319017d11ff77cad6da8629a1a7f6a874e0315
-sha256-linux=c06fb8c19aed6f388b9dc61cb8251b7a44f8561a9bf764ad2b9e635598f8dc17

.config/e2e-repl-nightly-selection.txt

@@ -1 +0,0 @@
-sha256=655a3f3c1d042e694339d15caba7580518320322d1bac0f09450b37e6c09e2e7

.config/e2e-smoke-selection.txt

@@ -1 +0,0 @@
-sha256=294350518743cac8d7c41880a2835216e4b697908d7b0b1bc92b62816d94c59d

.config/make/build-docker-buildx-dev.mak

@@ -1,64 +0,0 @@
-## —— Development/Source builds using direct buildx commands ---------------------------------------
-
-.PHONY: docker-dev
-docker-dev: ## Build dev multi-arch image (cannot load locally)
-	@echo "🏗️ Building multi-architecture development Docker images with buildx..."
-	@echo "💡 This builds from source code and is intended for local development and testing"
-	@echo "⚠️  Multi-arch images cannot be loaded locally, use docker-dev-push to push to registry"
-	$(DOCKER_CLI) buildx build \
-		--platform linux/amd64,linux/arm64 \
-		--file $(DOCKERFILE_SOURCE) \
-		--tag rustfs:source-latest \
-		--tag rustfs:dev-latest \
-		.
-
-.PHONY: docker-dev-local
-docker-dev-local: ## Build dev single-arch image (local load)
-	@echo "🏗️ Building single-architecture development Docker image for local use..."
-	@echo "💡 This builds from source code for the current platform and loads locally"
-	$(DOCKER_CLI) buildx build \
-		--file $(DOCKERFILE_SOURCE) \
-		--tag rustfs:source-latest \
-		--tag rustfs:dev-latest \
-		--load \
-		.
-
-.PHONY: docker-dev-push
-docker-dev-push: ## Build and push multi-arch development image # e.g (make docker-dev-push REGISTRY=xxx)
-	@if [ -z "$(REGISTRY)" ]; then \
-		echo "❌ Error: Please specify registry, example: make docker-dev-push REGISTRY=ghcr.io/username"; \
-		exit 1; \
-	fi
-	@echo "🚀 Building and pushing multi-architecture development Docker images..."
-	@echo "💡 Pushing to registry: $(REGISTRY)"
-	$(DOCKER_CLI) buildx build \
-		--platform linux/amd64,linux/arm64 \
-		--file $(DOCKERFILE_SOURCE) \
-		--tag $(REGISTRY)/rustfs:source-latest \
-		--tag $(REGISTRY)/rustfs:dev-latest \
-		--push \
-		.
-
-.PHONY: dev-env-start
-dev-env-start: ## Start development container environment
-	@echo "🚀 Starting development environment..."
-	$(DOCKER_CLI) buildx build \
-		--file $(DOCKERFILE_SOURCE) \
-		--tag rustfs:dev \
-		--load \
-		.
-	$(DOCKER_CLI) stop $(CONTAINER_NAME) 2>/dev/null || true
-	$(DOCKER_CLI) rm $(CONTAINER_NAME) 2>/dev/null || true
-	$(DOCKER_CLI) run -d --name $(CONTAINER_NAME) \
-		-p 9010:9010 -p 9000:9000 \
-		-v $(shell pwd):/workspace \
-		-it rustfs:dev
-
-.PHONY: dev-env-stop
-dev-env-stop: ## Stop development container environment
-	@echo "🛑 Stopping development environment..."
-	$(DOCKER_CLI) stop $(CONTAINER_NAME) 2>/dev/null || true
-	$(DOCKER_CLI) rm $(CONTAINER_NAME) 2>/dev/null || true
-
-.PHONY: dev-env-restart
-dev-env-restart: dev-env-stop dev-env-start ## Restart development container environment

.config/make/build-docker-buildx-production.mak

@@ -1,41 +0,0 @@
-## —— Production builds using docker buildx (for CI/CD and production) -----------------------------
-
-.PHONY: docker-buildx
-docker-buildx: ## Build production multi-arch image (no push)
-	@echo "🏗️ Building multi-architecture production Docker images with buildx..."
-	./docker-buildx.sh
-
-.PHONY: docker-buildx-push
-docker-buildx-push: ## Build and push production multi-arch image
-	@echo "🚀 Building and pushing multi-architecture production Docker images with buildx..."
-	./docker-buildx.sh --push
-
-.PHONY: docker-buildx-version
-docker-buildx-version: ## Build and version production multi-arch image # e.g (make docker-buildx-version VERSION=v1.0.0)
-	@if [ -z "$(VERSION)" ]; then \
-		echo "❌ Error: Please specify version, example: make docker-buildx-version VERSION=v1.0.0"; \
-		exit 1; \
-	fi
-	@echo "🏗️ Building multi-architecture production Docker images (version: $(VERSION))..."
-	./docker-buildx.sh --release $(VERSION)
-
-.PHONY: docker-buildx-push-version
-docker-buildx-push-version: ## Build and version and push production multi-arch image # e.g (make docker-buildx-push-version VERSION=v1.0.0)
-	@if [ -z "$(VERSION)" ]; then \
-		echo "❌ Error: Please specify version, example: make docker-buildx-push-version VERSION=v1.0.0"; \
-		exit 1; \
-	fi
-	@echo "🚀 Building and pushing multi-architecture production Docker images (version: $(VERSION))..."
-	./docker-buildx.sh --release $(VERSION) --push
-
-.PHONY: docker-buildx-production-local
-docker-buildx-production-local: ## Build production single-arch image locally
-	@echo "🏗️ Building single-architecture production Docker image locally..."
-	@echo "💡 Alternative to docker-buildx.sh for local testing"
-	$(DOCKER_CLI) buildx build \
-		--file $(DOCKERFILE_PRODUCTION) \
-		--tag rustfs:production-latest \
-		--tag rustfs:latest \
-		--load \
-		--build-arg RELEASE=latest \
-		.

.config/make/build-docker-production.mak

@@ -1,16 +0,0 @@
-## —— Single Architecture Docker Builds (Traditional) ----------------------------------------------
-
-.PHONY: docker-build-production
-docker-build-production: ## Build single-arch production image
-	@echo "🏗️ Building single-architecture production Docker image..."
-	@echo "💡 Consider using 'make docker-buildx-production-local' for multi-arch support"
-	$(DOCKER_CLI) build -f $(DOCKERFILE_PRODUCTION) -t rustfs:latest .
-
-.PHONY: docker-build-source
-docker-build-source: ## Build single-arch source image
-	@echo "🏗️ Building single-architecture source Docker image..."
-	@echo "💡 Consider using 'make docker-dev-local' for multi-arch support"
-	DOCKER_BUILDKIT=1 $(DOCKER_CLI) build \
-		--build-arg BUILDKIT_INLINE_CACHE=1 \
-		-f $(DOCKERFILE_SOURCE) -t rustfs:source .
-

.config/make/build-docker.mak

@@ -1,22 +0,0 @@
-## —— Docker-based build (alternative approach) ----------------------------------------------------
-
-# Usage: make BUILD_OS=ubuntu22.04 build-docker
-# Output: target/ubuntu22.04/release/rustfs
-
-.PHONY: build-docker
-build-docker: SOURCE_BUILD_IMAGE_NAME = rustfs-$(BUILD_OS):v1
-build-docker: SOURCE_BUILD_CONTAINER_NAME = rustfs-$(BUILD_OS)-build
-build-docker: BUILD_CMD = /root/.cargo/bin/cargo build --release --bin rustfs --target-dir /root/s3-rustfs/target/$(BUILD_OS)
-build-docker: ## Build using Docker container # e.g (make build-docker BUILD_OS=ubuntu22.04)
-	@echo "🐳 Building RustFS using Docker ($(BUILD_OS))..."
-	$(DOCKER_CLI) buildx build -t $(SOURCE_BUILD_IMAGE_NAME) -f $(DOCKERFILE_SOURCE) --load .
-	$(DOCKER_CLI) run --rm --name $(SOURCE_BUILD_CONTAINER_NAME) -v $(shell pwd):/root/s3-rustfs -it $(SOURCE_BUILD_IMAGE_NAME) $(BUILD_CMD)
-
-.PHONY: docker-inspect-multiarch
-docker-inspect-multiarch: ## Check image architecture support
-	@if [ -z "$(IMAGE)" ]; then \
-		echo "❌ Error: Please specify image, example: make docker-inspect-multiarch IMAGE=rustfs/rustfs:latest"; \
-		exit 1; \
-	fi
-	@echo "🔍 Inspecting multi-architecture image: $(IMAGE)"
-	docker buildx imagetools inspect $(IMAGE)

.config/make/build.mak

@@ -1,74 +0,0 @@
-## —— Local Native Build using build-rustfs.sh script (Recommended) --------------------------------
-
-.PHONY: build
-build: ## Build RustFS binary (includes console by default)
-	@echo "🔨 Building RustFS using build-rustfs.sh script..."
-	./build-rustfs.sh
-
-.PHONY: build-dev
-build-dev: ## Build RustFS in Development mode
-	@echo "🔨 Building RustFS in development mode..."
-	./build-rustfs.sh --dev
-
-.PHONY: build-musl
-build-musl: ## Build x86_64 musl version
-	@echo "🔨 Building rustfs for x86_64-unknown-linux-musl..."
-	@echo "💡 On macOS/Windows, use 'make build-docker' or 'make docker-dev' instead"
-	./build-rustfs.sh --platform x86_64-unknown-linux-musl
-
-.PHONY: build-gnu
-build-gnu: ## Build x86_64 GNU version
-	@echo "🔨 Building rustfs for x86_64-unknown-linux-gnu..."
-	@echo "💡 On macOS/Windows, use 'make build-docker' or 'make docker-dev' instead"
-	./build-rustfs.sh --platform x86_64-unknown-linux-gnu
-
-.PHONY: build-musl-arm64
-build-musl-arm64: ## Build aarch64 musl version
-	@echo "🔨 Building rustfs for aarch64-unknown-linux-musl..."
-	@echo "💡 On macOS/Windows, use 'make build-docker' or 'make docker-dev' instead"
-	./build-rustfs.sh --platform aarch64-unknown-linux-musl
-
-.PHONY: build-gnu-arm64
-build-gnu-arm64: ## Build aarch64 GNU version
-	@echo "🔨 Building rustfs for aarch64-unknown-linux-gnu..."
-	@echo "💡 On macOS/Windows, use 'make build-docker' or 'make docker-dev' instead"
-	./build-rustfs.sh --platform aarch64-unknown-linux-gnu
-
-
-## —— Profiling build (dial9 Tokio runtime telemetry) ------------------------------------------
-
-# dial9 hooks Tokio's unstable runtime instrumentation, so it needs
-# `--cfg tokio_unstable`. That flag is deliberately absent from
-# .cargo/config.toml: it is not free, and release binaries do not carry it.
-# Setting RUSTFLAGS here replaces (never appends to) the config-file value, and
-# crates/obs/build.rs fails the build if the feature and the flag disagree.
-#
-# There are no task-dump or S3-upload features — see the notes in
-# crates/obs/Cargo.toml for why.
-DIAL9_FEATURES ?= dial9
-DIAL9_RUSTFLAGS ?= --cfg tokio_unstable
-
-.PHONY: build-profiling
-build-profiling: ## Build RustFS with dial9 Tokio runtime telemetry (diagnostic builds only)
-	@echo "🔬 Building RustFS with dial9 telemetry (features: $(DIAL9_FEATURES))..."
-	@echo "⚠️  Diagnostic build: telemetry writes trace segments to disk continuously."
-	RUSTFLAGS="$(DIAL9_RUSTFLAGS)" cargo build --release --bin rustfs --features $(DIAL9_FEATURES)
-
-.PHONY: build-cross-all
-build-cross-all: core-deps ## Build binaries for all architectures
-	@echo "🔧 Building all target architectures..."
-	@echo "💡 On macOS/Windows, use 'make docker-dev' for reliable multi-arch builds"
-	@echo "🔨 Generating protobuf code..."
-	cargo run --bin gproto || true
-
-	@echo "🔨 Building rustfs for x86_64-unknown-linux-musl..."
-	./build-rustfs.sh --platform x86_64-unknown-linux-musl
-
-	@echo "🔨 Building rustfs for x86_64-unknown-linux-gnu..."
-	./build-rustfs.sh --platform x86_64-unknown-linux-gnu
-
-	@echo "🔨 Building rustfs for aarch64-unknown-linux-musl..."
-	./build-rustfs.sh --platform aarch64-unknown-linux-musl
-
-	@echo "🔨 Building rustfs for aarch64-unknown-linux-gnu..."
-	./build-rustfs.sh --platform aarch64-unknown-linux-gnu

.config/make/check.mak

@@ -1,26 +0,0 @@
-## —— Check and Inform Dependencies ----------------------------------------------------------------
-
-# Fatal check
-# Checks all required dependencies and exits with error if not found
-# (e.g., cargo, rustfmt)
-check-%:
-	@command -v $* >/dev/null 2>&1 || { \
-		echo >&2 "❌ '$*' is not installed."; \
-		exit 1; \
-	}
-
-# Warning-only check
-# Checks for optional dependencies and issues a warning if not found
-warn-%:
-	@command -v $* >/dev/null 2>&1 || { \
-		echo >&2 "⚠️ '$*' is not installed."; \
-	}
-
-# For checking dependencies use check-<dep-name> or warn-<dep-name>
-#
-# NOTE: cargo-nextest is a HARD dependency of `make test`, gated inside the
-# test recipe itself (with a RUSTFS_ALLOW_CARGO_TEST_FALLBACK=1 escape hatch)
-# rather than a warn-only prerequisite here — see .config/make/tests.mak.
-.PHONY: core-deps fmt-deps
-core-deps: check-cargo ## Check core dependencies
-fmt-deps: check-rustfmt ## Check lint and formatting dependencies

.config/make/coverage.mak

@@ -1,26 +0,0 @@
-## —— Coverage --------------------------------------------------------------------------------------
-
-# Local equivalent of the weekly coverage workflow (.github/workflows/coverage.yml,
-# backlog#1153 infra-5): same measurement scope (--workspace --exclude e2e_test,
-# nextest `ci` profile) and the same per-crate table. Slow — the instrumented
-# build cannot reuse your normal target cache and then runs the whole suite.
-# Doctests are not measured (needs nightly). Outputs land in target/llvm-cov/.
-.PHONY: coverage
-coverage: core-deps ## Workspace line coverage (cargo-llvm-cov + nextest; slow, writes target/llvm-cov/)
-	@if ! command -v cargo-llvm-cov >/dev/null 2>&1; then \
-		echo >&2 "❌ cargo-llvm-cov is required for 'make coverage' but was not found."; \
-		echo >&2 "   Install it with:"; \
-		echo >&2 "     cargo install cargo-llvm-cov --locked"; \
-		echo >&2 "     rustup component add llvm-tools-preview"; \
-		exit 1; \
-	fi
-	@if ! command -v cargo-nextest >/dev/null 2>&1; then \
-		echo >&2 "❌ cargo-nextest is required for 'make coverage' (see 'make test')."; \
-		echo >&2 "   Install it with: cargo install cargo-nextest --locked"; \
-		exit 1; \
-	fi
-	NEXTEST_PROFILE=ci cargo llvm-cov nextest --workspace --exclude e2e_test --no-report
-	@mkdir -p target/llvm-cov
-	cargo llvm-cov report --lcov --output-path target/llvm-cov/lcov.info
-	cargo llvm-cov report --json --output-path target/llvm-cov/coverage.json
-	python3 scripts/coverage_per_crate.py target/llvm-cov/coverage.json

.config/make/deploy.mak

@@ -1,6 +0,0 @@
-## —— Deploy using dev_deploy.sh script ------------------------------------------------------------
-
-.PHONY: deploy-dev
-deploy-dev: build-musl ## Deploy to dev server
-	@echo "🚀 Deploying to dev server: $${IP}"
-	./scripts/dev_deploy.sh $${IP}

.config/make/help.mak

@@ -1,38 +0,0 @@
-## —— Help, Help Build and Help Docker -------------------------------------------------------------
-
-
-.PHONY: help
-help: ## Shows This Help Menu
-	echo -e "$$HEADER"
-	grep -E '(^[a-zA-Z0-9_-]+:.*?## .*$$)|(^## )' $(MAKEFILE_LIST) | sed 's/^[^:]*://g' | awk 'BEGIN {FS = ":.*?## | #"} /^## / {printf "\n${green}%s${reset}\n", $$0; next} {printf "${cyan}%-30s${reset} ${white}%s${reset} ${green}%s${reset}\n", $$1, $$2, $$3}'
-
-.PHONY: help-build
-help-build: ## Shows RustFS build help
-	@echo ""
-	@echo "💡 build-rustfs.sh script provides more options, smart detection and binary verification"
-	@echo ""
-	@echo "🔧 Direct usage of build-rustfs.sh script:"
-	@echo ""
-	@echo "	./build-rustfs.sh --help                                # View script help"
-	@echo "	./build-rustfs.sh --no-console                          # Build without console resources"
-	@echo "	./build-rustfs.sh --force-console-update                # Force update console resources"
-	@echo "	./build-rustfs.sh --dev                                 # Development mode build"
-	@echo "	./build-rustfs.sh --sign                                # Sign binary files"
-	@echo "	./build-rustfs.sh --platform x86_64-unknown-linux-gnu   # Specify target platform"
-	@echo "	./build-rustfs.sh --skip-verification                   # Skip binary verification"
-	@echo ""
-
-.PHONY: help-docker
-help-docker: ## Shows docker environment and suggestion help
-	@echo ""
-	@echo "📋 Environment Variables:"
-	@echo "	REGISTRY                 Image registry address (required for push)"
-	@echo "	DOCKERHUB_USERNAME       Docker Hub username"
-	@echo "	DOCKERHUB_TOKEN          Docker Hub access token"
-	@echo "	GITHUB_TOKEN             GitHub access token"
-	@echo ""
-	@echo "💡 Suggestions:"
-	@echo "	Production use:          Use docker-buildx* commands (based on precompiled binaries)"
-	@echo "	Local development:       Use docker-dev* commands (build from source)"
-	@echo "	Development environment: Use dev-env-* commands to manage dev containers"
-	@echo ""

.config/make/lint-fmt.mak

@@ -1,101 +0,0 @@
-## —— Code quality and Formatting ------------------------------------------------------------------
-
-.NOTPARALLEL: fix
-
-.PHONY: fmt
-fmt: core-deps fmt-deps ## Format code
-	@echo "🔧 Formatting code..."
-	cargo fmt --all
-
-.PHONY: fmt-check
-fmt-check: core-deps fmt-deps ## Check code formatting
-	@echo "📝 Checking code formatting..."
-	cargo fmt --all --check
-
-.PHONY: clippy-check
-clippy-check: core-deps ## Run clippy checks
-	@echo "🔍 Running clippy checks..."
-	cargo clippy --all-targets -- -D warnings
-
-.PHONY: clippy-fix
-clippy-fix: core-deps ## Apply clippy fixes
-	@echo "🔧 Applying clippy fixes..."
-	cargo clippy --fix --allow-dirty
-
-.PHONY: fix
-fix: fmt clippy-fix ## Format code and apply clippy fixes
-
-.PHONY: quick-check
-quick-check: core-deps ## Run fast workspace compilation check
-	@echo "🔨 Running fast compilation check..."
-	cargo check --workspace --exclude e2e_test
-
-.PHONY: unsafe-code-check
-unsafe-code-check: ## Check unsafe_code allowances have SAFETY comments
-	@echo "🔒 Checking unsafe_code allowances..."
-	./scripts/check_unsafe_code_allowances.sh
-
-.PHONY: architecture-migration-check
-architecture-migration-check: ## Check architecture migration guardrails
-	@echo "🏗️ Checking architecture migration guardrails..."
-	./scripts/check_architecture_migration_rules.sh
-
-.PHONY: logging-guardrails-check
-logging-guardrails-check: ## Check logging guardrails for redaction and noise regressions
-	@echo "🪵 Checking logging guardrails..."
-	./scripts/check_logging_guardrails.sh
-
-.PHONY: error-other-ratchet-check
-error-other-ratchet-check: ## Check the ecstore ::other(format!) quorum-bucketing ratchet stays shrink-only
-	@echo "🪣 Checking error other(format!) ratchet..."
-	./scripts/check_error_other_format_ratchet.sh
-
-.PHONY: tokio-io-uring-check
-tokio-io-uring-check: ## Check tokio io-uring runtime feature stays removed
-	@echo "🚫 Checking tokio io-uring feature guard..."
-	./scripts/check_no_tokio_io_uring.sh
-
-.PHONY: extension-schema-check
-extension-schema-check: ## Check extension-schema stays a lightweight contract crate
-	@echo "🧩 Checking extension schema boundaries..."
-	./scripts/check_extension_schema_boundaries.sh
-
-.PHONY: body-cache-whitelist-check
-body-cache-whitelist-check: ## Check the body-cache eligibility gate stays a fail-closed allow-list
-	@echo "🧱 Checking body-cache whitelist guard..."
-	./scripts/check_body_cache_whitelist.sh
-
-.PHONY: s3s-footprint-check
-s3s-footprint-check: ## Check the s3s dependency footprint ratchet stays frozen
-	@echo "📦 Checking s3s footprint ratchet..."
-	./scripts/check_s3s_footprint.sh
-
-.PHONY: fips-wording-check
-fips-wording-check: ## Check docs and crates/kms do not over-claim crypto capabilities
-	@echo "📣 Checking cryptographic capability wording guard..."
-	./scripts/check_fips_wording.sh
-
-.PHONY: embedded-secrets-check
-embedded-secrets-check: ## Check no private key material or credential literal is committed
-	@echo "🔑 Checking embedded secret material guard..."
-	./scripts/check_embedded_secrets.sh
-
-.PHONY: offline-enrollment-e2e-check
-offline-enrollment-e2e-check: core-deps ## Build and exercise the dedicated offline enrollment E2E root
-	@echo "🔐 Checking the offline enrollment E2E root boundary..."
-	./scripts/check_offline_enrollment_e2e.sh
-
-.PHONY: test-wiring-check
-test-wiring-check: ## Check tests stay registered and selected by their intended runners
-	@echo "🧪 Checking test wiring..."
-	python3 ./scripts/check_test_wiring.py
-
-.PHONY: log-analyzer-rules-check
-log-analyzer-rules-check: core-deps ## Check log-analyzer rule anchors still exist verbatim in source
-	@echo "🩺 Checking log-analyzer rule anchors..."
-	./scripts/check_log_analyzer_rules.sh
-
-.PHONY: compilation-check
-compilation-check: core-deps ## Run compilation check
-	@echo "🔨 Running compilation check..."
-	cargo check --all-targets

.config/make/pre-commit.mak

@@ -1,31 +0,0 @@
-## —— Pre Commit Checks ----------------------------------------------------------------------------
-
-.NOTPARALLEL: pre-commit pre-pr dev-check
-
-.PHONY: setup-hooks
-setup-hooks: ## Set up git hooks
-	@echo "🔧 Setting up git hooks..."
-	chmod +x .git/hooks/pre-commit
-	@echo "✅ Git hooks setup complete!"
-
-.PHONY: doc-paths-check
-doc-paths-check: ## Check that instruction/architecture docs reference existing file paths
-	@echo "📄 Checking doc path references..."
-	./scripts/check_doc_paths.sh
-
-.PHONY: planning-docs-check
-planning-docs-check: ## Check that no planning-type documents are committed
-	@echo "📄 Checking for committed planning docs..."
-	./scripts/check_no_planning_docs.sh
-
-.PHONY: pre-commit
-pre-commit: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check error-other-ratchet-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check s3s-footprint-check fips-wording-check embedded-secrets-check test-wiring-check doc-paths-check planning-docs-check quick-check ## Run fast pre-commit checks without clippy/full tests
-	@echo "✅ All pre-commit checks passed!"
-
-.PHONY: pre-pr
-pre-pr: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check error-other-ratchet-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check s3s-footprint-check fips-wording-check embedded-secrets-check test-wiring-check doc-paths-check planning-docs-check log-analyzer-rules-check offline-enrollment-e2e-check clippy-check test ## Run full pre-PR checks with clippy and tests
-	@echo "✅ All pre-PR checks passed!"
-
-.PHONY: dev-check
-dev-check: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check error-other-ratchet-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check s3s-footprint-check fips-wording-check embedded-secrets-check test-wiring-check doc-paths-check planning-docs-check quick-check ## Run fast local development checks
-	@echo "✅ Fast development checks passed!"

.config/make/tests.mak

@@ -1,89 +0,0 @@
-## —— Tests and e2e test ---------------------------------------------------------------------------
-
-TEST_THREADS ?= 1
-
-# cargo-nextest is a HARD dependency of `make test`.
-#
-# nextest changes test semantics vs plain `cargo test`: it runs every test in
-# its own process (so serial_test's #[serial] mutex does not serialize across
-# tests) and it is the only runner that honours .config/nextest.toml
-# [test-groups] (e.g. the ecstore-serial-flaky serialization guard). CI runs
-# nextest (.github/actions/setup/action.yml installs it), so a silent fallback
-# to `cargo test` would run with different serialization behaviour than CI and
-# mask (or invent) flakes.
-#
-# Installing cargo-nextest:
-#   cargo install cargo-nextest --locked          # from source
-#   # or a prebuilt binary (faster) via the taiki-e installer / get.nexte.st:
-#   #   https://nexte.st/docs/installation/
-#
-# Escape hatch: set RUSTFS_ALLOW_CARGO_TEST_FALLBACK=1 to run the plain
-# `cargo test` fallback anyway. Results are NOT authoritative — semantics
-# differ from CI and .config/nextest.toml test-groups will NOT apply.
-
-.PHONY: script-tests
-script-tests: ## Run shell script tests
-	@echo "Running script tests..."
-	./scripts/test_build_rustfs_options.sh
-	./scripts/test_docker_runtime_timezone.sh
-	./scripts/test_entrypoint_credentials.sh
-	./scripts/test_internode_grpc_ab_bench.sh
-	./scripts/test_object_batch_bench_enhanced.sh
-	./scripts/test_hotpath_warp_ab_gate.sh
-	./scripts/test_hotpath_warp_abba.sh
-	./scripts/test_exact_1mib_handoff_abba.sh
-	./scripts/test_pinned_paired_abba_bench.sh
-	./scripts/test_manual_transition_runbooks.sh
-	./scripts/test_fuzz_runner.sh
-	./scripts/check_embedded_secrets.sh --self-test
-	python3 ./scripts/check_test_wiring.py --self-test
-	python3 ./scripts/check_security_coverage.py --self-test
-	python3 ./scripts/check_scheduled_validation_freshness.py --self-test
-	python3 ./scripts/s3-tests/test_report_compat.py
-	bash -n ./scripts/validate_object_data_cache_cold_stampede.sh
-	python3 ./scripts/check_object_data_cache_follower_samples.py --self-test
-	./scripts/validate_object_data_cache_cold_stampede.sh --self-test
-
-.PHONY: test
-test: core-deps script-tests ## Run all tests (needs cargo-nextest; RUSTFS_ALLOW_CARGO_TEST_FALLBACK=1 to override)
-	@echo "🧪 Running tests..."
-	@if command -v cargo-nextest >/dev/null 2>&1; then \
-		cargo nextest run --all --exclude e2e_test; \
-	elif [ "$${RUSTFS_ALLOW_CARGO_TEST_FALLBACK:-0}" = "1" ]; then \
-		echo >&2 "⚠️  ============================================================================"; \
-		echo >&2 "⚠️  cargo-nextest NOT found — running the 'cargo test' fallback (opt-in)."; \
-		echo >&2 "⚠️  TEST SEMANTICS DIFFER FROM CI; results are NOT authoritative:"; \
-		echo >&2 "⚠️    * nextest runs each test in its own process; 'cargo test' does not,"; \
-		echo >&2 "⚠️      so serial_test #[serial] serialization behaves differently."; \
-		echo >&2 "⚠️    * .config/nextest.toml [test-groups] will NOT apply (e.g. the"; \
-		echo >&2 "⚠️      ecstore-serial-flaky group), so load-sensitive tests may flake here"; \
-		echo >&2 "⚠️      but pass on CI (or vice versa)."; \
-		echo >&2 "⚠️  Install cargo-nextest and re-run before trusting these results."; \
-		echo >&2 "⚠️  ============================================================================"; \
-		cargo test --workspace --exclude e2e_test -- --nocapture --test-threads="$(TEST_THREADS)"; \
-	else \
-		echo >&2 "❌ cargo-nextest is required for 'make test' but was not found."; \
-		echo >&2 ""; \
-		echo >&2 "   RustFS tests run under cargo-nextest (process-per-test isolation)."; \
-		echo >&2 "   CI runs nextest and .config/nextest.toml [test-groups] only take effect"; \
-		echo >&2 "   under nextest. Plain 'cargo test' has different serialization semantics"; \
-		echo >&2 "   and is NOT a faithful substitute."; \
-		echo >&2 ""; \
-		echo >&2 "   Install it with either:"; \
-		echo >&2 "     cargo install cargo-nextest --locked"; \
-		echo >&2 "   or a prebuilt binary (faster) — see https://nexte.st/docs/installation/"; \
-		echo >&2 ""; \
-		echo >&2 "   To run the plain 'cargo test' fallback anyway (results NOT authoritative;"; \
-		echo >&2 "   serialization semantics differ from CI), re-run with:"; \
-		echo >&2 "     RUSTFS_ALLOW_CARGO_TEST_FALLBACK=1 make test"; \
-		exit 1; \
-	fi
-	cargo test --all --doc
-
-.PHONY: e2e-server
-e2e-server: ## Run e2e-server tests
-	sh $(shell pwd)/scripts/run.sh
-
-.PHONY: probe-e2e
-probe-e2e: ## Probe e2e tests
-	sh $(shell pwd)/scripts/probe.sh

.config/migration-gate-floor.txt

@@ -1,10 +0,0 @@
-# Committed floor for the number of tests selected by the migration-critical
-# CI gate (see scripts/check_migration_gate_count.sh, backlog#1153 infra-12).
-#
-# The floor equals the exact count of rustfs-ecstore --lib tests matching the
-# gate filter (name substrings: data_movement, rebalance, decommission,
-# source_cleanup, delete_marker) at the time this file was last updated.
-# CI fails if the selected count drops below this number, so renames or
-# removals that thin the gate must update this file in the same PR.
-# Adding tests does not require a bump, but bumping keeps the guard tight.
-571

.config/nextest.toml

@@ -1,493 +0,0 @@
-# nextest configuration for RustFS.
-#
-experimental = ["setup-scripts"]
-
-# Serialize the ecstore tests that share the process-wide disk registry or
-# exercise a multi-disk commit handoff across nextest process boundaries.
-#
-#   * store::bucket::tests::bucket_delete_* share process/global state (disk
-#     registry, lock client) and race make_bucket into InsufficientWriteQuorum
-#     when run concurrently with other ecstore tests.
-#   * bucket_lifecycle_ops::tests::concurrent_resend_same_part_commits_one_generation
-#     uses the shared multipart fixture and a deterministic uploadId-lock
-#     handoff, so it must not overlap another process mutating that fixture.
-#   * bucket::metadata_sys::tests::concurrent_config_writes_from_separate_nodes_do_not_lose_writes
-#     uses the shared transaction lock and must not overlap other ecstore tests.
-#
-# serial_test's #[serial] attribute does NOT serialize these across runs:
-# nextest executes each test in its own process, where the in-process
-# serial_test mutex has no effect. A nextest test-group with max-threads = 1 is
-# the mechanism that actually serializes across nextest's process boundary.
-#
-# ---------------------------------------------------------------------------
-# Profiles
-# ---------------------------------------------------------------------------
-# The `default` profile is what local `cargo nextest run` uses. It NEVER
-# retries: a red test locally means a real failure to investigate, not noise to
-# paper over. The `ci` profile (below) is the strict CI gate: global
-# retries = 0 so a new race's first occurrence is never masked, plus a
-# narrowly-scoped quarantine list (retries = 2) for tests with a tracked OPEN
-# flake issue. Flake policy lives in docs/testing/README.md.
-
-[test-groups]
-ecstore-serial-flaky = { max-threads = 1 }
-embedded-test-ports = { max-threads = 1 }
-e2e-vault = { max-threads = 1 }
-
-# Reliability / fault-injection e2e tests each spawn a single-node 4-disk RustFS
-# server and manipulate its disk directories at runtime (crates/e2e_test:
-# reliability_disk_fault_test, degraded_read_eof_regression_test / dist-13, and
-# replacement_privileged_e2e_test when explicitly run as root on Linux). They
-# are correct in isolation but resource-heavy; serialize them under nextest's
-# process boundary (serial_test's #[serial] does not cross it) so several 4-disk
-# servers never run at once. The e2e-full merge/main lane picks these up;
-# they are deliberately NOT in the fast PR `e2e-smoke` filter.
-e2e-reliability = { max-threads = 1 }
-e2e-inline-boundaries = { max-threads = 1 }
-e2e-cluster-nightly = { max-threads = 1 }
-
-# These exact regression scenarios build deep async storage futures that exceed
-# libtest's 2 MiB spawned-thread stack on Linux. Give only their test processes
-# the same 32 MiB stack already used by the crate's dedicated large-stack tests.
-[scripts.setup.ecstore-large-stack]
-command = ['sh', '-c', 'echo RUST_MIN_STACK=33554432 >> "$NEXTEST_ENV"']
-
-# The serial ILM selection builds the same deep storage futures in both the
-# lifecycle transition module and scanner integration binary. Different tests
-# in each have overflowed first across otherwise unrelated CI runs.
-[scripts.setup.lifecycle-large-stack]
-command = ['sh', '-c', 'echo RUST_MIN_STACK=33554432 >> "$NEXTEST_ENV"']
-
-# --- default profile (local): serialize the flaky groups, never retry --------
-[[profile.default.scripts]]
-filter = 'package(rustfs-ecstore) & test(/^(bucket::lifecycle::bucket_lifecycle_ops::tests::manual_transition_worker_result_recovery_marks_unknown_for_corrupt_marker|services::rebalance::entry::tests::real_rebalance_run_fence_loss_blocks_multipart_publication|store::init::tests::(decommission_entry_(allows_free_version_consumed_before_source_lock|rejects_subquorum_free_version_conflict_and_retains_source|skips_cleanup_only_marker_when_free_version_is_present)|prepared_tier_delete_recovery_(checks_later_pool_then_commits_after_source_removal|finds_directory_source_on_encoded_set|retains_journal_on_source_metadata_error)|tier_mutation_peer_handler_applies_prepare_commit_and_abort_idempotently|transition_response_loss_persists_unknown_outcome_for_provider_recovery|transition_transaction_recovery_(drops_record_after_confirmed_local_commit|keeps_cleanup_pending_local_commit)))$/)'
-setup = 'ecstore-large-stack'
-
-[[profile.default.scripts]]
-filter = 'binary(lifecycle_integration_test) | (package(rustfs) & test(/^app::lifecycle_transition_api_test::/))'
-setup = 'lifecycle-large-stack'
-
-[[profile.default.overrides]]
-filter = 'package(rustfs-ecstore) & (test(concurrent_resend_same_part_commits_one_generation) | test(concurrent_config_writes_from_separate_nodes_do_not_lose_writes) | test(/^store::bucket::tests::bucket_delete_(mark_delete|purge_removes|default_s3_delete)/))'
-test-group = 'ecstore-serial-flaky'
-
-# Serialize the multipart crash-consistency scenarios (dist-2, backlog#1150):
-# each spawns a 4-disk hermetic erasure set and drives full staged-upload +
-# commit + GET cycles — the same cross-disk-commit IO shape that made
-# concurrent_resend load-sensitive. Preventive serialization only, no retries.
-# The matching ci-profile override is after [profile.ci].
-[[profile.default.overrides]]
-filter = 'package(rustfs-ecstore) & test(/^set_disk::ops::multipart::tests::crash_consistency::/)'
-test-group = 'ecstore-serial-flaky'
-
-# The production-handler relocation regression builds an isolated 8-disk,
-# 2-pool store and commits a 72 MiB multipart object. Keep that cross-disk IO
-# from overlapping the ecstore commit fixtures above.
-[[profile.default.overrides]]
-filter = 'package(rustfs) & test(execute_get_object_resumes_from_relocated_pool_without_splicing_body)'
-test-group = 'ecstore-serial-flaky'
-
-# Embedded integration-test binaries discover an ephemeral port and release
-# the probe listener before RustFS binds it. Serialize that cross-process
-# TOCTOU window; retries would only hide real startup failures.
-[[profile.default.overrides]]
-filter = 'package(rustfs) & (binary(/^embedded.*_test$/) | binary(admin_diagnostic_capability_e2e))'
-test-group = 'embedded-test-ports'
-
-# Serialize the durable manual-transition checkpoint test across nextest's
-# process boundary; it mutates bucket lifecycle metadata and is not quarantined.
-[[profile.default.overrides]]
-filter = 'package(rustfs-ecstore) & test(manual_transition_page_checkpoint_persists_durable_job_progress)'
-test-group = 'ecstore-serial-flaky'
-
-# The durable ILM decommission regressions build isolated multi-pool stores and
-# deliberately take source or target disks offline while checking fencing.
-[[profile.default.overrides]]
-filter = 'package(rustfs-ecstore) & (test(decommission_migrates_and_verifies_registered_durable_ilm_records) | test(decommission_durable_ilm_target_read_error_is_not_masked_by_peer_success) | test(decommission_durable_ilm_terminal_receipt_recovers_failed_source_cleanup) | test(decommission_durable_ilm_receipt_pagination_fails_closed_on_second_page) | test(decommission_durable_ilm_recovery_keeps_multiple_active_sources))'
-test-group = 'ecstore-serial-flaky'
-
-# Serialize the bucket-incarnation / lifecycle-fence tests. They drive
-# init_bucket_metadata_sys and bucket_metadata_sys_of, i.e. process-global
-# OnceLock state that serial_test's #[serial] cannot protect across nextest's
-# process boundary, and they delete+recreate buckets — the same shape that
-# raced into InsufficientWriteQuorum in backlog#937. Preventive only, no
-# retries. The matching ci-profile override is after [profile.ci].
-[[profile.default.overrides]]
-filter = 'package(rustfs-ecstore) & (test(bucket_delete_waits_for_config_mutation_fence) | test(stale_config_request_cannot_mutate_a_recreated_bucket) | test(disk_incarnation_read_detects_stale_cache_until_peer_reload) | test(lifecycle_expiry_fails_closed_on_corrupt_object_lock_metadata) | test(expiry_configs_are_resolved_from_the_owning_store))'
-test-group = 'ecstore-serial-flaky'
-
-# Serialize the default-off dst-dir fsync group-commit tests. They use
-# process-global test hooks/registry to deterministically freeze fsync batches;
-# no retries, just one at a time under nextest too.
-[[profile.default.overrides]]
-filter = 'package(rustfs-ecstore) & test(dst_dir_fsync_group_commit)'
-test-group = 'ecstore-serial-flaky'
-
-# Serialize the 4-disk reliability / degraded-read e2e tests (see the
-# e2e-reliability test-group note above). The matching ci-profile override is at
-# the end of the file, after [profile.ci] is declared.
-[[profile.default.overrides]]
-filter = 'package(e2e_test) & test(/^(reliability_disk_fault|degraded_read_eof_regression|replacement_privileged_e2e)_test::/)'
-test-group = 'e2e-reliability'
-
-[[profile.default.overrides]]
-filter = 'package(e2e_test) & test(/^inline_fast_path_cluster_test::/)'
-test-group = 'e2e-inline-boundaries'
-
-# Vault KMS tests share the fixed dev-server port 8200. serial_test's #[serial]
-# does not cross nextest process boundaries, so keep every Vault-backed test in
-# one group.
-[[profile.default.overrides]]
-filter = 'package(e2e_test) & (test(/^kms::kms_vault_test::/) | test(/^kms::kms_rekey_sweep_test::/) | test(/^kms::configured_roundtrip_test::test_configured_vault_kms_admin_and_versioned_cleanup$/))'
-test-group = 'e2e-vault'
-
-# ---------------------------------------------------------------------------
-# ci profile — the strict CI gate (ci.yml `cargo nextest run --profile ci`)
-# ---------------------------------------------------------------------------
-[profile.ci]
-# Strict: a new race must fail on its first occurrence, never be retried away.
-retries = 0
-# Report every failure in one run instead of bailing on the first.
-fail-fast = false
-
-[profile.ci.junit]
-# Emitted to target/nextest/ci/junit.xml; uploaded as a CI artifact.
-# Tests that pass only after a quarantine retry are marked `flaky` here — that
-# marker is the observable signal the flake policy is built around.
-path = "junit.xml"
-
-[[profile.ci.scripts]]
-filter = 'package(rustfs-ecstore) & test(/^(bucket::lifecycle::bucket_lifecycle_ops::tests::manual_transition_worker_result_recovery_marks_unknown_for_corrupt_marker|services::rebalance::entry::tests::real_rebalance_run_fence_loss_blocks_multipart_publication|store::init::tests::(decommission_entry_(allows_free_version_consumed_before_source_lock|rejects_subquorum_free_version_conflict_and_retains_source|skips_cleanup_only_marker_when_free_version_is_present)|prepared_tier_delete_recovery_(checks_later_pool_then_commits_after_source_removal|finds_directory_source_on_encoded_set|retains_journal_on_source_metadata_error)|tier_mutation_peer_handler_applies_prepare_commit_and_abort_idempotently|transition_response_loss_persists_unknown_outcome_for_provider_recovery|transition_transaction_recovery_(drops_record_after_confirmed_local_commit|keeps_cleanup_pending_local_commit)))$/)'
-setup = 'ecstore-large-stack'
-
-[[profile.ci.scripts]]
-filter = 'binary(lifecycle_integration_test) | (package(rustfs) & test(/^app::lifecycle_transition_api_test::/))'
-setup = 'lifecycle-large-stack'
-
-# ===========================================================================
-# QUARANTINE — flaky tests granted retries = 2 under the ci profile ONLY.
-#
-# RULES (enforced by review, see docs/testing/README.md):
-#   * Every entry MUST link exactly one OPEN issue tracking the flake.
-#   * An entry stays until the issue is fixed (test made robust) or the test is
-#     deleted — 30-day policy. No entry may exist without a live issue link.
-#
-# Each entry also re-declares the `ecstore-serial-flaky` test-group so the
-# serialization holds under the ci profile (nextest evaluates a named
-# profile's own overrides list, not the default profile's).
-# ===========================================================================
-
-# QUARANTINE: OPEN backlog#937 — store::bucket::tests::bucket_delete_* race
-# make_bucket into InsufficientWriteQuorum via shared global state under load.
-[[profile.ci.overrides]]
-filter = 'package(rustfs-ecstore) & test(/^store::bucket::tests::bucket_delete_(mark_delete|purge_removes|default_s3_delete)/)'
-test-group = 'ecstore-serial-flaky'
-retries = 2
-
-# Keep deterministic ECStore write handoffs isolated across nextest processes.
-[[profile.ci.overrides]]
-filter = 'package(rustfs-ecstore) & (test(concurrent_resend_same_part_commits_one_generation) | test(concurrent_config_writes_from_separate_nodes_do_not_lose_writes))'
-test-group = 'ecstore-serial-flaky'
-
-# QUARANTINE: OPEN rustfs#4690 — walk_dir stall-budget accounting test depends
-# on producer/consumer timing windows that stretch past the budget on loaded
-# CI runners (regression test for rustfs#4644; failed on a zero-Rust-diff PR).
-[[profile.ci.overrides]]
-filter = 'package(rustfs-ecstore) & test(walk_dir_does_not_charge_consumer_backpressure_to_the_stall_budget)'
-retries = 2
-
-# Serialize the relocated-pool GET resume regression under the ci profile too
-# (see the matching default-profile override near the top). No longer a
-# quarantine: the fixture race (rustfs#6701/rustfs#6703) was fixed by #6707,
-# which made the staging tolerate quorum-tolerated disk gaps; only the 8-disk
-# cross-disk-IO serialization remains.
-[[profile.ci.overrides]]
-filter = 'package(rustfs) & test(execute_get_object_resumes_from_relocated_pool_without_splicing_body)'
-test-group = 'ecstore-serial-flaky'
-
-# Serialize the 4-disk reliability / degraded-read e2e tests under the ci
-# profile too (see the e2e-reliability test-group note near the top). Not a
-# quarantine: no retries, just single-threaded so several 4-disk servers never
-# run concurrently when e2e-full runs the suite.
-[[profile.ci.overrides]]
-filter = 'package(e2e_test) & test(/^(reliability_disk_fault|degraded_read_eof_regression|replacement_privileged_e2e)_test::/)'
-test-group = 'e2e-reliability'
-
-# Serialize the multipart crash-consistency scenarios under the ci profile too
-# (see the matching default-profile override near the top). Not a quarantine:
-# no retries, just serialized 4-disk cross-disk-commit IO.
-[[profile.ci.overrides]]
-filter = 'package(rustfs-ecstore) & test(/^set_disk::ops::multipart::tests::crash_consistency::/)'
-test-group = 'ecstore-serial-flaky'
-
-# Match the default-profile embedded test isolation without quarantining or
-# retrying failures in CI.
-[[profile.ci.overrides]]
-filter = 'package(rustfs) & (binary(/^embedded.*_test$/) | binary(admin_diagnostic_capability_e2e))'
-test-group = 'embedded-test-ports'
-
-# Serialize the durable manual-transition checkpoint test under the ci profile
-# too. No retries: failures stay visible.
-[[profile.ci.overrides]]
-filter = 'package(rustfs-ecstore) & test(manual_transition_page_checkpoint_persists_durable_job_progress)'
-test-group = 'ecstore-serial-flaky'
-
-[[profile.ci.overrides]]
-filter = 'package(rustfs-ecstore) & (test(decommission_migrates_and_verifies_registered_durable_ilm_records) | test(decommission_durable_ilm_target_read_error_is_not_masked_by_peer_success) | test(decommission_durable_ilm_terminal_receipt_recovers_failed_source_cleanup) | test(decommission_durable_ilm_receipt_pagination_fails_closed_on_second_page) | test(decommission_durable_ilm_recovery_keeps_multiple_active_sources))'
-test-group = 'ecstore-serial-flaky'
-
-# Serialize the bucket-incarnation / lifecycle-fence tests under the ci profile
-# too (see the matching default-profile override near the top). No retries.
-[[profile.ci.overrides]]
-filter = 'package(rustfs-ecstore) & (test(bucket_delete_waits_for_config_mutation_fence) | test(stale_config_request_cannot_mutate_a_recreated_bucket) | test(disk_incarnation_read_detects_stale_cache_until_peer_reload) | test(lifecycle_expiry_fails_closed_on_corrupt_object_lock_metadata) | test(expiry_configs_are_resolved_from_the_owning_store))'
-test-group = 'ecstore-serial-flaky'
-
-[[profile.ci.overrides]]
-filter = 'package(rustfs-ecstore) & test(dst_dir_fsync_group_commit)'
-test-group = 'ecstore-serial-flaky'
-
-# ---------------------------------------------------------------------------
-# e2e-smoke profile — PR smoke subset of the e2e_test crate (backlog#1149 ci-4)
-# ---------------------------------------------------------------------------
-# PR smoke subset of the e2e_test crate (backlog#1149 ci-4). This profile is
-# the single wiring mechanism for e2e tests in CI: other suites join by
-# extending this filter (or a sibling profile), never by adding ad-hoc e2e
-# jobs to ci.yml. Admission criteria (see crates/e2e_test/README.md): fast,
-# single-node topology, no external dependencies (no awscurl / Vault / fixed
-# ports / pre-started server), no #[ignore].
-#
-# Each e2e test spawns its own rustfs server on a random port with an isolated
-# temp dir (crates/e2e_test/src/common.rs), so the subset is parallel-safe.
-#
-# Replication failure harness (backlog#1147 repl-8): the first clause admits
-# its four in-process fake-target self-tests. They bind random loopback ports,
-# use no external service, and finish in under a second.
-#
-# Replication PR subset (backlog#1147 repl-1): the second clause admits the 20
-# FAST bucket-replication tests from replication_extension_test — the
-# target-registration / replication-check / list / remove / delete admin paths
-# that validate config synchronously and never wait for asynchronous
-# replication convergence. Each spawns its own single-node rustfs server(s) on
-# random ports (source, plus an independent single-node target for the pair
-# checks — NOT a cluster), so the subset stays parallel-safe and single-digit
-# seconds. The data-plane tests that poll for convergence and all
-# `_real_dual_node` / `_real_single_node`
-# site-replication tests run in the [profile.e2e-repl-nightly] lane below, NOT
-# here. This allowlist is the single source of truth for the PR/nightly split:
-# the nightly profile derives its set as "the replication module MINUS this
-# allowlist", so any new replication test lands in nightly by default (never
-# silently unrun) until it is explicitly blessed as fast here. Keep the two
-# regexes byte-identical. The committed profile selection digests make changes
-# visible in CI; current counts live in docs/testing/e2e-suite-inventory.md.
-# HISTORY (2026-07-11): the 20 fast tests were briefly pulled out of this lane
-# (#4724) because they set a loopback (127.0.0.1) replication target that the
-# SSRF egress guard rejected on every PR after repl-1 (#4712). That is fixed —
-# the guard now honours an off-by-default opt-in and this suite's source servers
-# set it (RUSTFS_REPLICATION_ALLOW_LOOPBACK_TARGET) — so the allowlist below is
-# restored.
-#
-# Security negative-auth subset (backlog#1151 sec-5): the three attacker-facing
-# S3 auth-rejection suites join the first clause above by module name —
-# presigned_negative (sec-2), negative_sigv4 (sec-1, header SigV4), and
-# admin_auth (sec-4, admin gate + root-credential lifecycle). All three use
-# RustFSTestEnvironment on a random port and are parallel-safe, so they meet the
-# smoke admission criteria unchanged. This is the wiring step that makes those
-# merged suites actually execute on every PR (they were dead until listed here).
-# A rename that drops any of them out of this filter would silently thin the
-# security gate with no CI signal, so scripts/check_security_smoke_count.sh owns
-# a count-floor guard over exactly this subset (infra-12 mechanism, floor in
-# .config/security-smoke-floor.txt), invoked from the e2e-tests job in ci.yml.
-# NOT here by topology: the GHSA-3p3x FTPS/WebDAV constant-time e2e
-# (protocols::test_protocol_core_suite) binds fixed ports and needs the
-# ftps,webdav features, so it cannot join this random-port, default-feature
-# profile; its GHSA-r5qv sibling is a unit test that already runs in the
-# test-and-lint `--all --exclude e2e_test` pass. See
-# docs/testing/security-regressions.md for the full CI-execution map.
-#
-# ILM tiering main path (backlog#1148 ilm-7): the `reliant::tiering::` clause
-# admits the hermetic transition e2e. Like the fast replication pair checks it
-# spawns a second independent single-node server (the cold RustFS tier), not a
-# cluster, so it keeps the lane's parallel-safe / no-external-dependency
-# properties. The RustFS warm backend has no loopback guard (that guard is
-# replication-only), so it needs no opt-in env for its 127.0.0.1 tier target.
-#
-# Disk compression (backlog#1848): the `compression` module joins the smoke
-# lane so the multipart disk-compression roundtrips (restored after
-# rustfs/rustfs#5169 disabled them) have PR-lane signal, not just merge-gate.
-# Single-node servers on random ports with isolated temp dirs — meets the
-# admission criteria unchanged.
-[profile.e2e-smoke]
-default-filter = """
-    package(e2e_test) & (
-        test(/^(delete_marker_migration_semantics|version_id_regression|list_objects_v2_pagination|list_object_versions_regression|list_objects_duplicates|list_buckets_double_slash|list_buckets_auth|list_buckets_iam_filter|leading_slash_key|special_chars|create_bucket_region|delete_objects_versioning|head_object_consistency|head_object_range|copy_object_metadata|copy_object_tagging|copy_source_invalid_date|content_encoding|compression|multipart_storage_class|storage_class_capability|ssec_copy|anonymous_access|bucket_policy_check|presigned_negative|negative_sigv4|admin_auth|notification_webhook|tls_hot_reload|console_smoke|admin_iam_crud|admin_pools|sts_query_compat)_test::|^fake_s3_target::/)
-        | test(/^replication_extension_test::(test_replication_check_succeeds_with_remote_target|test_replication_check_rejects_target_without_object_lock|test_set_remote_target_rejects_unversioned_source_bucket|test_replication_check_rejects_unversioned_source_bucket|test_replication_check_rejects_missing_replication_config|test_replication_check_rejects_invalid_bucket|test_set_remote_target_rejects_same_bucket_on_same_deployment|test_set_remote_target_rejects_unversioned_target_bucket|test_set_remote_target_update_requires_arn|test_set_remote_target_update_rejects_missing_target|test_set_remote_target_rejects_invalid_target_url|test_set_remote_target_rejects_self_signed_https_target_without_skip_tls_verify|test_set_remote_target_rejects_private_ca_https_target_without_ca_cert_pem|test_list_remote_targets_rejects_empty_bucket|test_list_remote_targets_rejects_invalid_bucket|test_remove_remote_target_rejects_missing_target|test_remove_remote_target_rejects_missing_arn|test_remove_remote_target_rejects_invalid_bucket|test_remove_remote_target_rejects_target_used_by_replication|test_delete_bucket_replication_removes_remote_target)$/)
-        | test(/^reliant::lifecycle::/)
-        | test(/^reliant::tiering::/)
-    )
-"""
-fail-fast = false
-
-[profile.e2e-smoke.junit]
-path = "junit.xml"
-
-# The pagination boundary cases can stall when a server/listing regression
-# prevents the continuation request from completing. Keep the timeout scoped
-# to those known failure modes so legitimate lifecycle/tiering waits retain
-# their test-level timing budget.
-[[profile.e2e-smoke.overrides]]
-filter = 'package(e2e_test) & test(/^list_objects_v2_pagination_test::tests::(test_list_objects_v2_delimiter_small_page_traverses_all|test_list_objects_v2_max_keys_above_limit_returns_token|test_list_objects_v2_maxkeys_above_limit_with_delimiter)$/)'
-slow-timeout = { period = "60s", terminate-after = 2, grace-period = "10s" }
-
-# ---------------------------------------------------------------------------
-# e2e-repl-nightly profile — scheduled full replication e2e lane (repl-1)
-# ---------------------------------------------------------------------------
-# backlog#1147 repl-1 (deps: ci-4). Runs the SLOW / cross-process replication
-# tests that are unfit for the per-PR e2e-smoke gate:
-#
-#   * 2 remote-target TLS validation tests.
-#   * 15 bucket-replication data-plane/helper tests — they PUT/delete objects
-#     and poll until source and target converge; two replicate over HTTPS,
-#     six pin SSE replication contracts (managed SSE-S3/SSE-KMS re-encrypt on
-#     the target incl. multipart and the resync path, SSE-C and
-#     target-without-KMS stay fail-closed), and one guards event/history
-#     observers.
-#   * 13 `_real_dual_node` site-replication tests — each spawns TWO full rustfs
-#     servers and drives the cross-process site-replication control plane.
-#   * 1 `_real_three_node` site-replication test.
-#   * 1 `_real_single_node` service-account round-trip test.
-#
-# The set is defined as "everything in replication_extension_test that is NOT
-# in the e2e-smoke PR allowlist above" (the negated clause is byte-identical to
-# the allowlist), so a newly added replication test automatically runs here
-# until it is explicitly promoted to the fast PR subset — no replication test
-# is ever silently left out of CI.
-#
-# #[serial] does NOT serialize under nextest (process-per-test; see the file
-# header). These tests need no cross-test serialization: each spawns its own
-# server(s) on random ports with isolated temp dirs, so they are parallel-safe
-# by construction — the same property the e2e-smoke subset relies on. If load
-# on the runner surfaces a real flake, quarantine the specific test with an
-# OPEN issue link (ci-10 / backlog#937 policy), never blanket-retry or exclude.
-#
-# Wired by .github/workflows/e2e-replication-nightly.yml (schedule +
-# workflow_dispatch), which builds the rustfs binary once, installs awscurl so
-# the STS dual-node test actually exercises its path (the test fails when
-# awscurl is absent), and routes scheduled failures
-# through .github/actions/schedule-failure-issue (ci-8). Explicit division of
-# labor with e2e-full: these tests run only in the consolidated nightly
-# workflow, not in the merge/main lane.
-[profile.e2e-repl-nightly]
-default-filter = """
-    package(e2e_test)
-    & test(/^replication_extension_test::/)
-    & !test(/^replication_extension_test::(test_replication_check_succeeds_with_remote_target|test_replication_check_rejects_target_without_object_lock|test_set_remote_target_rejects_unversioned_source_bucket|test_replication_check_rejects_unversioned_source_bucket|test_replication_check_rejects_missing_replication_config|test_replication_check_rejects_invalid_bucket|test_set_remote_target_rejects_same_bucket_on_same_deployment|test_set_remote_target_rejects_unversioned_target_bucket|test_set_remote_target_update_requires_arn|test_set_remote_target_update_rejects_missing_target|test_set_remote_target_rejects_invalid_target_url|test_set_remote_target_rejects_self_signed_https_target_without_skip_tls_verify|test_set_remote_target_rejects_private_ca_https_target_without_ca_cert_pem|test_list_remote_targets_rejects_empty_bucket|test_list_remote_targets_rejects_invalid_bucket|test_remove_remote_target_rejects_missing_target|test_remove_remote_target_rejects_missing_arn|test_remove_remote_target_rejects_invalid_bucket|test_remove_remote_target_rejects_target_used_by_replication|test_delete_bucket_replication_removes_remote_target)$/)
-"""
-fail-fast = false
-
-[profile.e2e-repl-nightly.junit]
-# Emitted to target/nextest/e2e-repl-nightly/junit.xml; uploaded by the nightly
-# workflow as the failure-triage artifact.
-path = "junit.xml"
-
-# ---------------------------------------------------------------------------
-# e2e-nightly profile — destructive multi-process cluster fault domains
-# ---------------------------------------------------------------------------
-# These seven modules are deliberately outside e2e-full's merge budget. Each
-# starts a real multi-process or multi-disk topology and exercises node/disk
-# loss, quorum, cleanup, notification fan-in, or admin-timeout behavior. The
-# consolidated nightly workflow runs them serially to avoid resource
-# starvation; failures are never retried.
-[profile.e2e-nightly]
-default-filter = """
-    package(e2e_test)
-    & test(/^(admin_timeout_regression_test|cluster_concurrency_test|cluster_multidrive_pool_test|heal_erasure_disk_rebuild_test|namespace_lock_quorum_test|object_lambda_test|stale_multipart_cleanup_cluster_test)::/)
-"""
-fail-fast = false
-
-[profile.e2e-nightly.junit]
-path = "junit.xml"
-
-[[profile.e2e-nightly.overrides]]
-filter = 'package(e2e_test)'
-test-group = 'e2e-cluster-nightly'
-
-# ---------------------------------------------------------------------------
-# e2e-protocols profile — serial protocol lane
-# ---------------------------------------------------------------------------
-# The suite owns fixed ports, so the nightly workflow runs this exact profile
-# with one nextest worker.
-[profile.e2e-protocols]
-default-filter = 'package(e2e_test) & test(/^protocols::/)'
-fail-fast = false
-
-[profile.e2e-protocols.junit]
-path = "junit.xml"
-
-# ---------------------------------------------------------------------------
-# e2e-full profile — merge-gate full single-node e2e lane (backlog#1149 ci-5)
-# ---------------------------------------------------------------------------
-# The merge gate (ci.yml `e2e-full` job: push main + merge_group +
-# workflow_dispatch). Runs the user-visible KMS, object-lock, multipart-auth,
-# quota, checksum, encryption,
-# security-boundary, ... — that the fast PR `e2e-smoke` subset deliberately
-# skips. Budget <= 45 min; authority for the suite count is `cargo nextest list
-# --profile e2e-full` (see docs/testing/e2e-suite-inventory.md).
-#
-# The filter is "the whole e2e_test crate MINUS the sets owned by other lanes":
-#   * protocols:: — FTPS/SFTP/WebDAV, run from the dedicated protocol profile
-#     with one worker because the suite owns fixed ports.
-#   * the 7 cluster suites that spin up a RustFSTestClusterEnvironment
-#     (cluster_concurrency, cluster_multidrive_pool, stale_multipart_cleanup_cluster,
-#     namespace_lock_quorum, heal_erasure_disk_rebuild, admin_timeout_regression,
-#     object_lambda) — too heavy for the merge budget; they run in the
-#     e2e-nightly serial cluster-fault lane.
-#   * replication_extension_test — repl-1 already splits it into the PR
-#     `e2e-smoke` (20 fast) and `e2e-repl-nightly` (56 slow) lanes and reserves
-#     it for those, so e2e-full does not double-run it.
-#   * #[ignore]d tests — nextest skips them by default (no --run-ignored); the
-#     manual-localhost:9000 reliant tests are ci-13's migration.
-#
-# Each e2e test spawns its own single-node rustfs server on a random port with
-# an isolated temp dir (crates/e2e_test/src/common.rs), so the set is
-# parallel-safe — the same property e2e-smoke relies on. The exceptions are the
-# 4-disk reliability / degraded-read fault-injection tests and the fixed-port
-# Vault tests, both serialized below.
-# KNOWN-FAILURE EXCLUSIONS (characterization run 29381309848, 2026-07-15:
-# 341 ran / 32 failed on the suites' first automated run ever). Deterministic
-# product failures cannot be quarantined away with retries, so each family is
-# excluded here with its tracking issue, under the same discipline as the
-# ci-profile quarantine (docs/testing/README.md): every entry MUST cite one
-# OPEN issue, and the fixing PR MUST delete the exclusion. The passing
-# negative-path siblings of each family stay in as regression guards.
-#   * rustfs#4843 — over-limit archive entry paths hard-reject the whole
-#     archive even under ignore-errors semantics.
-[profile.e2e-full]
-default-filter = """
-    package(e2e_test)
-    & !test(/^protocols::/)
-    & !test(/^(admin_timeout_regression_test|cluster_concurrency_test|cluster_multidrive_pool_test|heal_erasure_disk_rebuild_test|namespace_lock_quorum_test|object_lambda_test|stale_multipart_cleanup_cluster_test)::/)
-    & !test(/^replication_extension_test::/)
-    & !test(/^multipart_auth_test::test_signed_put_object_extract_skips_invalid_entry_when_ignore_errors_enabled$/)
-    & !test(/^snowball_auto_extract_test::tests::snowball_auto_extract_(ignores_invalid_entries_when_requested|supports_standard_headers_with_combined_extract_options)$/)
-"""
-fail-fast = false
-
-[profile.e2e-full.junit]
-# Emitted to target/nextest/e2e-full/junit.xml; uploaded by the e2e-full job.
-path = "junit.xml"
-
-# Serialize the 4-disk reliability / degraded-read e2e tests under e2e-full too
-# (see the e2e-reliability test-group note near the top of this file). Not a
-# quarantine: no retries, just single-threaded so several 4-disk servers never
-# run concurrently.
-[[profile.e2e-full.overrides]]
-filter = 'package(e2e_test) & test(/^(reliability_disk_fault|degraded_read_eof_regression|replacement_privileged_e2e)_test::/)'
-test-group = 'e2e-reliability'
-
-[[profile.e2e-full.overrides]]
-filter = 'package(e2e_test) & test(/^inline_fast_path_cluster_test::/)'
-test-group = 'e2e-inline-boundaries'
-
-[[profile.e2e-full.overrides]]
-filter = 'package(e2e_test) & (test(/^kms::kms_vault_test::/) | test(/^kms::kms_rekey_sweep_test::/) | test(/^kms::configured_roundtrip_test::test_configured_vault_kms_admin_and_versioned_cleanup$/))'
-test-group = 'e2e-vault'

.config/security-smoke-floor.txt

@@ -1,12 +0,0 @@
-# Committed floor for the number of security negative-auth tests selected by the
-# e2e-smoke PR profile (see scripts/check_security_smoke_count.sh, backlog#1151
-# sec-5).
-#
-# The floor equals the exact count of e2e_test cases whose name starts with a
-# security module prefix (negative_sigv4_test, presigned_negative_test,
-# admin_auth_test) that the [profile.e2e-smoke] default-filter in
-# .config/nextest.toml selects, at the time this file was last updated. CI fails
-# if the selected count drops below this number, so a rename or removal that
-# thins the security smoke gate must update this file in the same PR.
-# Adding tests does not require a bump, but bumping keeps the guard tight.
-18

.docker/README.md

@@ -1,131 +1,261 @@
-# RustFS Docker Infrastructure
+# RustFS Docker Images
 
-This directory contains the complete Docker infrastructure for building, deploying, and monitoring RustFS. It provides ready-to-use configurations for development, testing, and production-grade observability.
+This directory contains Docker configuration files and supporting infrastructure for building and running RustFS container images.
 
-## 📂 Directory Structure
+## 📁 Directory Structure
 
-| Directory | Description | Status |
-| :--- | :--- | :--- |
-| **[`observability/`](observability/README.md)** | **[RECOMMENDED]** Full-stack observability (Prometheus, Grafana, Tempo, Loki). | ✅ Production-Ready |
-| **[`compose/`](compose/README.md)** | Specialized setups (e.g., 4-node distributed cluster testing). | ⚠️ Testing Only |
-| **[`mqtt/`](mqtt/README.md)** | EMQX Broker configuration for MQTT integration testing. | 🧪 Development |
-| **[`openobserve-otel/`](openobserve-otel/README.md)** | Alternative lightweight observability stack using OpenObserve. | 🔄 Alternative |
-
----
-
-## 📄 Root Directory Files
-
-The following files in the project root are essential for Docker operations:
-
-### Build Scripts & Dockerfiles
-
-| File | Description | Usage |
-| :--- | :--- | :--- |
-| **`docker-buildx.sh`** | **Multi-Arch Build Script**<br>Automates building and pushing Docker images for `amd64` and `arm64`. Supports release and dev channels. | `./docker-buildx.sh --push` |
-| **`Dockerfile`** | **Production Image (Alpine)**<br>Lightweight image using musl libc. Downloads pre-built binaries from GitHub Releases. | `docker build -t rustfs:latest .` |
-| **`Dockerfile.glibc`** | **Production Image (Ubuntu)**<br>Standard image using glibc. Useful if you need specific dynamic libraries. | `docker build -f Dockerfile.glibc .` |
-| **`Dockerfile.source`** | **Development Image**<br>Builds RustFS from source code. Includes build tools. Ideal for local development and CI. | `docker build -f Dockerfile.source .` |
-
-### Docker Compose Configurations
-
-| File | Description | Usage |
-| :--- | :--- | :--- |
-| **`docker-compose.yml`** | **Main Development Setup**<br>Comprehensive setup with profiles for development, observability, and proxying. | `docker compose up -d`<br>`docker compose --profile observability up -d` |
-| **`docker-compose-simple.yml`** | **Quick Start Setup**<br>Minimal configuration running a single RustFS instance with 4 volumes. Perfect for first-time users. | `docker compose -f docker-compose-simple.yml up -d` |
-
----
-
-## 🌟 Observability Stack (Recommended)
-
-Located in: [`.docker/observability/`](observability/README.md)
-
-We provide a comprehensive, industry-standard observability stack designed for deep insights into RustFS performance. This is the recommended setup for both development and production monitoring.
-
-### Components
-- **Metrics**: Prometheus (Collection) + Grafana (Visualization)
-- **Traces**: Tempo (Storage) + Jaeger (UI)
-- **Logs**: Loki
-- **Ingestion**: OpenTelemetry Collector
-
-### Key Features
-- **Full Persistence**: All metrics, logs, and traces are saved to Docker volumes, ensuring no data loss on restarts.
-- **Correlation**: Seamlessly jump between Logs, Traces, and Metrics in Grafana.
-- **High Performance**: Optimized configurations for batching, compression, and memory management.
-
-### Quick Start
-```bash
-cd .docker/observability
-docker compose up -d
+```
+rustfs/
+├── Dockerfile           # Production image (Alpine + pre-built binaries)
+├── Dockerfile.source    # Development image (Debian + source build)
+├── docker-buildx.sh     # Multi-architecture build script
+├── Makefile             # Build automation with simplified commands
+└── .docker/             # Supporting infrastructure
+    ├── observability/   # Monitoring and observability configs
+    ├── compose/         # Docker Compose configurations
+    ├── mqtt/            # MQTT broker configs
+    └── openobserve-otel/ # OpenObserve + OpenTelemetry configs
 ```
 
----
+## 🎯 Image Variants
 
-## 🧪 Specialized Environments
+### Core Images
 
-Located in: [`.docker/compose/`](compose/README.md)
+| Image | Base OS | Build Method | Size | Use Case |
+|-------|---------|--------------|------|----------|
+| `production` (default) | Alpine 3.18 | GitHub Releases | Smallest | Production deployment |
+| `source` | Debian Bookworm | Source build | Medium | Custom builds with cross-compilation |
+| `dev` | Debian Bookworm | Development tools | Large | Interactive development |
 
-These configurations are tailored for specific testing scenarios that require complex topologies.
+## 🚀 Usage Examples
+
+### Quick Start (Production)
 
-### Distributed Cluster (4-Nodes)
-Simulates a real-world distributed environment with 4 RustFS nodes running locally.
 ```bash
-docker compose -f .docker/compose/docker-compose.cluster.yaml up -d
+# Default production image (Alpine + GitHub Releases)
+docker run -p 9000:9000 rustfs/rustfs:latest
+
+# Specific version
+docker run -p 9000:9000 rustfs/rustfs:1.2.3
 ```
 
-### Integrated Observability Test
-A self-contained environment running 4 RustFS nodes alongside the full observability stack. Useful for end-to-end testing of telemetry.
+### Complete Tag Strategy Examples
+
 ```bash
-docker compose -f .docker/compose/docker-compose.observability.yaml up -d
+# Stable Releases
+docker run rustfs/rustfs:1.2.3           # Main version (production)
+docker run rustfs/rustfs:1.2.3-production # Explicit production variant
+docker run rustfs/rustfs:1.2.3-source   # Source build variant
+docker run rustfs/rustfs:latest          # Latest stable
+
+# Prerelease Versions
+docker run rustfs/rustfs:1.3.0-alpha.2  # Specific alpha version
+docker run rustfs/rustfs:alpha           # Latest alpha
+docker run rustfs/rustfs:beta            # Latest beta
+docker run rustfs/rustfs:rc              # Latest release candidate
+
+# Development Versions
+docker run rustfs/rustfs:dev             # Latest main branch development
+docker run rustfs/rustfs:dev-13e4a0b     # Specific commit
+docker run rustfs/rustfs:dev-latest      # Latest development
+docker run rustfs/rustfs:main-latest     # Main branch latest
 ```
 
----
+### Development Environment
 
-## 📡 MQTT Integration
-
-Located in: [`.docker/mqtt/`](mqtt/README.md)
-
-Provides an EMQX broker for testing RustFS MQTT features.
-
-### Quick Start
 ```bash
-cd .docker/mqtt
-docker compose up -d
-```
-- **Dashboard**: [http://localhost:18083](http://localhost:18083) (Default: `admin` / `public`)
-- **MQTT Port**: `1883`
+# Quick setup using Makefile (recommended)
+make docker-dev-local     # Build development image locally
+make dev-env-start        # Start development container
 
----
+# Manual Docker commands
+docker run -it -v $(pwd):/workspace -p 9000:9000 rustfs/rustfs:latest-dev
 
-## 👁️ Alternative: OpenObserve
+# Build from source locally
+docker build -f Dockerfile.source -t rustfs:custom .
 
-Located in: [`.docker/openobserve-otel/`](openobserve-otel/README.md)
-
-For users preferring a lightweight, all-in-one solution, we support OpenObserve. It combines logs, metrics, and traces into a single binary and UI.
-
-### Quick Start
-```bash
-cd .docker/openobserve-otel
-docker compose up -d
+# Development with hot reload
+docker-compose up rustfs-dev
 ```
 
----
+## 🏗️ Build Arguments and Scripts
 
-## 🔧 Common Operations
+### Using Makefile Commands (Recommended)
+
+The easiest way to build images using simplified commands:
 
-### Cleaning Up
-To stop all containers and remove volumes (**WARNING**: deletes all persisted data):
 ```bash
-docker compose down -v
+# Development images (build from source)
+make docker-dev-local             # Build for local use (single arch)
+make docker-dev                   # Build multi-arch (for CI/CD)
+make docker-dev-push REGISTRY=xxx # Build and push to registry
+
+# Production images (using pre-built binaries)
+make docker-buildx                # Build multi-arch production images
+make docker-buildx-push           # Build and push production images
+make docker-buildx-version VERSION=v1.0.0  # Build specific version
+
+# Development environment
+make dev-env-start                # Start development container
+make dev-env-stop                 # Stop development container
+make dev-env-restart              # Restart development container
+
+# Help
+make help-docker                  # Show all Docker-related commands
 ```
 
-### Viewing Logs
-To follow logs for a specific service:
+### Using docker-buildx.sh (Advanced)
+
+For direct script usage and advanced scenarios:
+
 ```bash
-docker compose logs -f [service_name]
+# Build latest version for all architectures
+./docker-buildx.sh
+
+# Build and push to registry
+./docker-buildx.sh --push
+
+# Build specific version
+./docker-buildx.sh --release v1.2.3
+
+# Build and push specific version
+./docker-buildx.sh --release v1.2.3 --push
 ```
 
-### Checking Status
-To see the status of all running containers:
+### Manual Docker Builds
+
+All images support dynamic version selection:
+
 ```bash
-docker compose ps
+# Build production image with latest release
+docker build --build-arg RELEASE="latest" -t rustfs:latest .
+
+# Build from source with specific target
+docker build -f Dockerfile.source \
+  --build-arg TARGETPLATFORM="linux/amd64" \
+  -t rustfs:source .
+
+# Development build
+docker build -f Dockerfile.source -t rustfs:dev .
 ```
+
+## 🔧 Binary Download Sources
+
+### Unified GitHub Releases
+
+The production image downloads from GitHub Releases for reliability and transparency:
+
+- ✅ **production** → GitHub Releases API with automatic latest detection
+- ✅ **Checksum verification** → SHA256SUMS validation when available
+- ✅ **Multi-architecture** → Supports amd64 and arm64
+
+### Source Build
+
+The source variant compiles from source code with advanced features:
+
+- 🔧 **Cross-compilation** → Supports multiple target platforms via `TARGETPLATFORM`
+- ⚡ **Build caching** → sccache for faster compilation
+- 🎯 **Optimized builds** → Release optimizations with LTO and symbol stripping
+
+## 📋 Architecture Support
+
+All variants support multi-architecture builds:
+
+- **linux/amd64** (x86_64)
+- **linux/arm64** (aarch64)
+
+Architecture is automatically detected during build using Docker's `TARGETARCH` build argument.
+
+## 🔐 Security Features
+
+- **Checksum Verification**: Production image verifies SHA256SUMS when available
+- **Non-root User**: All images run as user `rustfs` (UID 1000)
+- **Minimal Runtime**: Production image only includes necessary dependencies
+- **Secure Defaults**: No hardcoded credentials or keys
+
+## 🛠️ Development Workflow
+
+### Quick Start with Makefile (Recommended)
+
+```bash
+# 1. Start development environment
+make dev-env-start
+
+# 2. Your development container is now running with:
+#    - Port 9000 exposed for RustFS
+#    - Port 9010 exposed for admin console
+#    - Current directory mounted as /workspace
+
+# 3. Stop when done
+make dev-env-stop
+```
+
+### Manual Development Setup
+
+```bash
+# Build development image from source
+make docker-dev-local
+
+# Or use traditional Docker commands
+docker build -f Dockerfile.source -t rustfs:dev .
+
+# Run with development tools
+docker run -it -v $(pwd):/workspace -p 9000:9000 rustfs:dev bash
+
+# Or use docker-compose for complex setups
+docker-compose up rustfs-dev
+```
+
+### Common Development Tasks
+
+```bash
+# Build and test locally
+make build                        # Build binary natively
+make docker-dev-local            # Build development Docker image
+make test                        # Run tests
+make fmt                         # Format code
+make clippy                      # Run linter
+
+# Get help
+make help                        # General help
+make help-docker                 # Docker-specific help
+make help-build                  # Build-specific help
+```
+
+## 🚀 CI/CD Integration
+
+The project uses GitHub Actions for automated multi-architecture Docker builds:
+
+### Automated Builds
+
+- **Tags**: Automatic builds triggered on version tags (e.g., `v1.2.3`)
+- **Main Branch**: Development builds with `dev-latest` and `main-latest` tags
+- **Pull Requests**: Test builds without registry push
+
+### Build Variants
+
+Each build creates three image variants:
+
+- `rustfs/rustfs:v1.2.3` (production - Alpine-based)
+- `rustfs/rustfs:v1.2.3-source` (source build - Debian-based)
+- `rustfs/rustfs:v1.2.3-dev` (development - Debian-based with tools)
+
+### Manual Builds
+
+Trigger custom builds via GitHub Actions:
+
+```bash
+# Use workflow_dispatch to build specific versions
+# Available options: latest, main-latest, dev-latest, v1.2.3, dev-abc123
+```
+
+## 📦 Supporting Infrastructure
+
+The `.docker/` directory contains supporting configuration files:
+
+- **observability/** - Prometheus, Grafana, OpenTelemetry configs
+- **compose/** - Multi-service Docker Compose setups
+- **mqtt/** - MQTT broker configurations
+- **openobserve-otel/** - Log aggregation and tracing setup
+
+See individual README files in each subdirectory for specific usage instructions.

.docker/compose/README.md

@@ -1,102 +1,80 @@
-# Specialized Docker Compose Configurations
+# Docker Compose Configurations
 
-This directory contains specialized Docker Compose configurations for specific testing scenarios.
-
-## ⚠️ Important Note
-
-**For Observability:**
-We **strongly recommend** using the new, fully integrated observability stack located in `../observability/`. It provides a production-ready setup with Prometheus, Grafana, Tempo, Loki, and OpenTelemetry Collector, all with persistent storage and optimized configurations.
-
-The `docker-compose.observability.yaml` in this directory is kept for legacy reference or specific minimal testing needs but is **not** the primary recommended setup.
+This directory contains specialized Docker Compose configurations for different use cases.
 
 ## 📁 Configuration Files
 
-### Cluster Testing
+This directory contains specialized Docker Compose configurations and their associated Dockerfiles, keeping related files organized together.
 
-- **`docker-compose.cluster.yaml`**
-  - **Purpose**: Simulates a 4-node RustFS distributed cluster.
-  - **Use Case**: Testing distributed storage logic, consensus, and failover.
-  - **Nodes**: 4 RustFS instances.
-  - **Storage**: Uses local HTTP endpoints.
+### Main Configuration (Root Directory)
 
-### Legacy / Minimal Observability
+- **`../../docker-compose.yml`** - **Default Production Setup**
+  - Complete production-ready configuration
+  - Includes RustFS server + full observability stack
+  - Supports multiple profiles: `dev`, `observability`, `cache`, `proxy`
+  - Recommended for most users
 
-- **`docker-compose.observability.yaml`**
-  - **Purpose**: A minimal observability setup.
-  - **Status**: **Deprecated**. Please use `../observability/docker-compose.yml` instead.
+### Specialized Configurations
+
+- **`docker-compose.cluster.yaml`** - **Distributed Testing**
+  - 4-node cluster setup for testing distributed storage
+  - Uses local compiled binaries
+  - Simulates multi-node environment
+  - Ideal for development and cluster testing
+
+- **`docker-compose.observability.yaml`** - **Observability Focus**
+  - Specialized setup for testing observability features
+  - Includes OpenTelemetry, Jaeger, Prometheus, Loki, Grafana
+  - Uses `../../Dockerfile.source` for builds
+  - Perfect for observability development
 
 ## 🚀 Usage Examples
 
+### Production Setup
+
+```bash
+# Start main service
+docker-compose up -d
+
+# Start with development profile
+docker-compose --profile dev up -d
+
+# Start with full observability
+docker-compose --profile observability up -d
+```
+
 ### Cluster Testing
 
-To start a 4-node cluster for distributed testing:
-
 ```bash
-# From project root
-docker compose -f .docker/compose/docker-compose.cluster.yaml up -d
+# Build and start 4-node cluster (run from project root)
+cd .docker/compose
+docker-compose -f docker-compose.cluster.yaml up -d
+
+# Or run directly from project root
+docker-compose -f .docker/compose/docker-compose.cluster.yaml up -d
 ```
 
-### Script-Based 4-Node Validation (Recommended)
-
-Use the local validation script when you need local-source image build, failover checks,
-and benchmark workflow in one command:
+### Observability Testing
 
 ```bash
-# Default mode: WAIT_PROBE_MODE=service
-# This avoids false negatives where /health/ready remains 503 locally
-# while the service path is already available.
-./scripts/run_four_node_cluster_failover_bench.sh
+# Start observability-focused environment (run from project root)
+cd .docker/compose
+docker-compose -f docker-compose.observability.yaml up -d
+
+# Or run directly from project root
+docker-compose -f .docker/compose/docker-compose.observability.yaml up -d
 ```
 
-Strict mode is available when you explicitly want `/health/ready == 200` as the gate:
+## 🔧 Configuration Overview
 
-```bash
-WAIT_PROBE_MODE=ready ./scripts/run_four_node_cluster_failover_bench.sh
-```
+| Configuration | Nodes | Storage | Observability | Use Case |
+|---------------|-------|---------|---------------|----------|
+| **Main** | 1 | Volume mounts | Full stack | Production |
+| **Cluster** | 4 | HTTP endpoints | Basic | Testing |
+| **Observability** | 4 | Local data | Advanced | Development |
 
-### Profiling + Trace Validation
+## 📝 Notes
 
-The profiling-focused 4-node compose keeps profiling enabled and points RustFS
-to an OTLP/HTTP collector endpoint:
-
-```bash
-docker compose -f .docker/compose/docker-compose.cluster.local-build.profiling-amd64.yml up -d
-```
-
-Important behavior notes:
-
-- `RUSTFS_OBS_ENDPOINT` is the OTLP/HTTP base URL. RustFS automatically sends
-  traces to `/v1/traces`, metrics to `/v1/metrics`, and logs to `/v1/logs`.
-- Startup usually produces logs and metrics first. That does not guarantee
-  visible traces yet.
-- Trace data becomes obvious only after real HTTP/S3/gRPC requests hit RustFS.
-- `RUSTFS_OBS_LOGGER_LEVEL=info` keeps the top-level request span but filters
-  many nested `debug` spans. If Tempo/Jaeger looks sparse, retry with
-  `RUSTFS_OBS_LOGGER_LEVEL=debug` before suspecting the collector.
-
-Minimal trace verification flow:
-
-```bash
-# 1. Start the profiling compose with richer span visibility.
-RUSTFS_OBS_LOGGER_LEVEL=debug \
-docker compose -f .docker/compose/docker-compose.cluster.local-build.profiling-amd64.yml up -d
-
-# 2. Generate real request traffic after startup.
-curl -I http://127.0.0.1:9000/health
-curl -I http://127.0.0.1:9000/health/ready
-
-# 3. Then inspect Tempo or Jaeger.
-# Grafana: http://localhost:3000
-# Jaeger:  http://localhost:16686
-```
-
-If logs and metrics are present but traces are sparse, the most common cause is
-"no real request traffic yet" or "`info` level filtered nested spans", not an
-OTLP routing failure.
-
-### (Deprecated) Minimal Observability
-
-```bash
-# From project root
-docker compose -f .docker/compose/docker-compose.observability.yaml up -d
-```
+- Always ensure you have built the required binaries before starting cluster tests
+- The main configuration is sufficient for most use cases
+- Specialized configurations are for specific testing scenarios

.docker/compose/docker-compose.cluster.local-build.profiling-amd64.yml

@@ -1,236 +0,0 @@
-# Copyright 2024 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-# Profiling-first 4-node local-build compose.
-#
-# Goals:
-# - force linux/amd64 runtime/build on Apple Silicon hosts;
-# - enable RustFS built-in CPU profiling;
-# - keep all tuning knobs host-overridable via env.
-#
-# Observability notes:
-# - `RUSTFS_OBS_ENDPOINT` is the OTLP/HTTP base URL. RustFS appends
-#   `/v1/traces`, `/v1/metrics`, and `/v1/logs` automatically.
-# - Logs and metrics usually appear during startup. Traces mainly appear after
-#   real HTTP/S3/gRPC requests create spans.
-# - `RUSTFS_OBS_LOGGER_LEVEL=info` keeps the top-level request trace span but
-#   filters many `debug`-level nested spans. Use `debug` when validating trace
-#   richness rather than collector reachability.
-
-services:
-  node1:
-    platform: ${RUSTFS_DOCKER_PLATFORM:-linux/amd64}
-    image: ${RUSTFS_IMAGE:-rustfs/rustfs:local-4node}
-    build:
-      context: ../..
-      dockerfile: Dockerfile.source
-    hostname: node1
-    environment:
-      - RUSTFS_VOLUMES=http://node{1...4}:9000/data/rustfs{0...3}
-      - RUSTFS_ADDRESS=:9000
-      - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfs-cluster-admin}
-      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfs-cluster-secret}
-      - RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
-      # `info` is enough for startup logs/metrics. Use `debug` if Tempo/Jaeger
-      # should show richer nested spans during request-path verification.
-      - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
-      - RUSTFS_OBS_PROFILING_ENDPOINT=${RUSTFS_OBS_PROFILING_ENDPOINT:-http://host.docker.internal:4040}
-      - RUSTFS_OBS_PROFILING_EXPORT_ENABLED=${RUSTFS_OBS_PROFILING_EXPORT_ENABLED:-true}
-      - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
-      - RUSTFS_ENABLE_PROFILING=${RUSTFS_ENABLE_PROFILING:-true}
-      - RUSTFS_PROF_CPU_MODE=${RUSTFS_PROF_CPU_MODE:-continuous}
-      - RUSTFS_PROF_CPU_FREQ=${RUSTFS_PROF_CPU_FREQ:-99}
-      - RUSTFS_PROF_OUTPUT_DIR=${RUSTFS_PROF_OUTPUT_DIR:-/tmp/rustfs-profiles}
-      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=${RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS:-48}
-      - RUSTFS_OBJECT_IO_BUFFER_SIZE=${RUSTFS_OBJECT_IO_BUFFER_SIZE:-262144}
-      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=${RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD:-6}
-      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=${RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD:-12}
-      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=${RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY:-8}
-      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=${RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE:-8388608}
-      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=${RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES:-25165824}
-      - RUSTFS_RUNTIME_WORKER_THREADS=${RUSTFS_RUNTIME_WORKER_THREADS:-12}
-      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=${RUSTFS_RUNTIME_MAX_BLOCKING_THREADS:-512}
-      - RUSTFS_ALLOCATOR_RECLAIM_ENABLED=${RUSTFS_ALLOCATOR_RECLAIM_ENABLED:-false}
-    extra_hosts:
-      - "host.docker.internal:host-gateway"
-    volumes:
-      - node1_data_0:/data/rustfs0
-      - node1_data_1:/data/rustfs1
-      - node1_data_2:/data/rustfs2
-      - node1_data_3:/data/rustfs3
-    ports:
-      - "9000:9000"
-    networks:
-      - rustfs-cluster-net
-
-  node2:
-    platform: ${RUSTFS_DOCKER_PLATFORM:-linux/amd64}
-    image: ${RUSTFS_IMAGE:-rustfs/rustfs:local-4node}
-    build:
-      context: ../..
-      dockerfile: Dockerfile.source
-    hostname: node2
-    environment:
-      - RUSTFS_VOLUMES=http://node{1...4}:9000/data/rustfs{0...3}
-      - RUSTFS_ADDRESS=:9000
-      - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfs-cluster-admin}
-      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfs-cluster-secret}
-      - RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
-      # `info` is enough for startup logs/metrics. Use `debug` if Tempo/Jaeger
-      # should show richer nested spans during request-path verification.
-      - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
-      - RUSTFS_OBS_PROFILING_ENDPOINT=${RUSTFS_OBS_PROFILING_ENDPOINT:-http://host.docker.internal:4040}
-      - RUSTFS_OBS_PROFILING_EXPORT_ENABLED=${RUSTFS_OBS_PROFILING_EXPORT_ENABLED:-true}
-      - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
-      - RUSTFS_ENABLE_PROFILING=${RUSTFS_ENABLE_PROFILING:-true}
-      - RUSTFS_PROF_CPU_MODE=${RUSTFS_PROF_CPU_MODE:-continuous}
-      - RUSTFS_PROF_CPU_FREQ=${RUSTFS_PROF_CPU_FREQ:-99}
-      - RUSTFS_PROF_OUTPUT_DIR=${RUSTFS_PROF_OUTPUT_DIR:-/tmp/rustfs-profiles}
-      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=${RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS:-48}
-      - RUSTFS_OBJECT_IO_BUFFER_SIZE=${RUSTFS_OBJECT_IO_BUFFER_SIZE:-262144}
-      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=${RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD:-6}
-      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=${RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD:-12}
-      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=${RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY:-8}
-      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=${RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE:-8388608}
-      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=${RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES:-25165824}
-      - RUSTFS_RUNTIME_WORKER_THREADS=${RUSTFS_RUNTIME_WORKER_THREADS:-12}
-      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=${RUSTFS_RUNTIME_MAX_BLOCKING_THREADS:-512}
-      - RUSTFS_ALLOCATOR_RECLAIM_ENABLED=${RUSTFS_ALLOCATOR_RECLAIM_ENABLED:-false}
-    extra_hosts:
-      - "host.docker.internal:host-gateway"
-    volumes:
-      - node2_data_0:/data/rustfs0
-      - node2_data_1:/data/rustfs1
-      - node2_data_2:/data/rustfs2
-      - node2_data_3:/data/rustfs3
-    ports:
-      - "9001:9000"
-    networks:
-      - rustfs-cluster-net
-
-  node3:
-    platform: ${RUSTFS_DOCKER_PLATFORM:-linux/amd64}
-    image: ${RUSTFS_IMAGE:-rustfs/rustfs:local-4node}
-    build:
-      context: ../..
-      dockerfile: Dockerfile.source
-    hostname: node3
-    environment:
-      - RUSTFS_VOLUMES=http://node{1...4}:9000/data/rustfs{0...3}
-      - RUSTFS_ADDRESS=:9000
-      - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfs-cluster-admin}
-      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfs-cluster-secret}
-      - RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
-      # `info` is enough for startup logs/metrics. Use `debug` if Tempo/Jaeger
-      # should show richer nested spans during request-path verification.
-      - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
-      - RUSTFS_OBS_PROFILING_ENDPOINT=${RUSTFS_OBS_PROFILING_ENDPOINT:-http://host.docker.internal:4040}
-      - RUSTFS_OBS_PROFILING_EXPORT_ENABLED=${RUSTFS_OBS_PROFILING_EXPORT_ENABLED:-true}
-      - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
-      - RUSTFS_ENABLE_PROFILING=${RUSTFS_ENABLE_PROFILING:-true}
-      - RUSTFS_PROF_CPU_MODE=${RUSTFS_PROF_CPU_MODE:-continuous}
-      - RUSTFS_PROF_CPU_FREQ=${RUSTFS_PROF_CPU_FREQ:-99}
-      - RUSTFS_PROF_OUTPUT_DIR=${RUSTFS_PROF_OUTPUT_DIR:-/tmp/rustfs-profiles}
-      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=${RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS:-48}
-      - RUSTFS_OBJECT_IO_BUFFER_SIZE=${RUSTFS_OBJECT_IO_BUFFER_SIZE:-262144}
-      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=${RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD:-6}
-      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=${RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD:-12}
-      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=${RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY:-8}
-      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=${RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE:-8388608}
-      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=${RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES:-25165824}
-      - RUSTFS_RUNTIME_WORKER_THREADS=${RUSTFS_RUNTIME_WORKER_THREADS:-12}
-      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=${RUSTFS_RUNTIME_MAX_BLOCKING_THREADS:-512}
-      - RUSTFS_ALLOCATOR_RECLAIM_ENABLED=${RUSTFS_ALLOCATOR_RECLAIM_ENABLED:-false}
-    extra_hosts:
-      - "host.docker.internal:host-gateway"
-    volumes:
-      - node3_data_0:/data/rustfs0
-      - node3_data_1:/data/rustfs1
-      - node3_data_2:/data/rustfs2
-      - node3_data_3:/data/rustfs3
-    ports:
-      - "9002:9000"
-    networks:
-      - rustfs-cluster-net
-
-  node4:
-    platform: ${RUSTFS_DOCKER_PLATFORM:-linux/amd64}
-    image: ${RUSTFS_IMAGE:-rustfs/rustfs:local-4node}
-    build:
-      context: ../..
-      dockerfile: Dockerfile.source
-    hostname: node4
-    environment:
-      - RUSTFS_VOLUMES=http://node{1...4}:9000/data/rustfs{0...3}
-      - RUSTFS_ADDRESS=:9000
-      - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfs-cluster-admin}
-      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfs-cluster-secret}
-      - RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
-      # `info` is enough for startup logs/metrics. Use `debug` if Tempo/Jaeger
-      # should show richer nested spans during request-path verification.
-      - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
-      - RUSTFS_OBS_PROFILING_ENDPOINT=${RUSTFS_OBS_PROFILING_ENDPOINT:-http://host.docker.internal:4040}
-      - RUSTFS_OBS_PROFILING_EXPORT_ENABLED=${RUSTFS_OBS_PROFILING_EXPORT_ENABLED:-true}
-      - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
-      - RUSTFS_ENABLE_PROFILING=${RUSTFS_ENABLE_PROFILING:-true}
-      - RUSTFS_PROF_CPU_MODE=${RUSTFS_PROF_CPU_MODE:-continuous}
-      - RUSTFS_PROF_CPU_FREQ=${RUSTFS_PROF_CPU_FREQ:-99}
-      - RUSTFS_PROF_OUTPUT_DIR=${RUSTFS_PROF_OUTPUT_DIR:-/tmp/rustfs-profiles}
-      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=${RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS:-48}
-      - RUSTFS_OBJECT_IO_BUFFER_SIZE=${RUSTFS_OBJECT_IO_BUFFER_SIZE:-262144}
-      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=${RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD:-6}
-      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=${RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD:-12}
-      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=${RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY:-8}
-      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=${RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE:-8388608}
-      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=${RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES:-25165824}
-      - RUSTFS_RUNTIME_WORKER_THREADS=${RUSTFS_RUNTIME_WORKER_THREADS:-12}
-      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=${RUSTFS_RUNTIME_MAX_BLOCKING_THREADS:-512}
-      - RUSTFS_ALLOCATOR_RECLAIM_ENABLED=${RUSTFS_ALLOCATOR_RECLAIM_ENABLED:-false}
-    extra_hosts:
-      - "host.docker.internal:host-gateway"
-    volumes:
-      - node4_data_0:/data/rustfs0
-      - node4_data_1:/data/rustfs1
-      - node4_data_2:/data/rustfs2
-      - node4_data_3:/data/rustfs3
-    ports:
-      - "9003:9000"
-    networks:
-      - rustfs-cluster-net
-
-volumes:
-  node1_data_0:
-  node1_data_1:
-  node1_data_2:
-  node1_data_3:
-  node2_data_0:
-  node2_data_1:
-  node2_data_2:
-  node2_data_3:
-  node3_data_0:
-  node3_data_1:
-  node3_data_2:
-  node3_data_3:
-  node4_data_0:
-  node4_data_1:
-  node4_data_2:
-  node4_data_3:
-
-networks:
-  rustfs-cluster-net:
-    driver: bridge

.docker/compose/docker-compose.cluster.local-build.yml

@@ -1,220 +0,0 @@
-# Copyright 2024 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-services:
-  node1:
-    image: ${RUSTFS_IMAGE:-rustfs/rustfs:local-4node}
-    build:
-      context: ../..
-      dockerfile: Dockerfile.source
-    hostname: node1
-    environment:
-      - RUSTFS_VOLUMES=${RUSTFS_VOLUMES:-http://node{1...4}:9000/data/rustfs{0...3}}
-      - RUSTFS_ADDRESS=:9000
-      - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin-local}
-      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfssecret-local}
-      - RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
-      - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
-      - RUSTFS_OBS_USE_STDOUT=${RUSTFS_OBS_USE_STDOUT:-false}
-      - RUSTFS_OBS_LOG_STDOUT_ENABLED=${RUSTFS_OBS_LOG_STDOUT_ENABLED:-false}
-      - RUSTFS_ISSUE3031_DIAG_ENABLE=${RUSTFS_ISSUE3031_DIAG_ENABLE:-false}
-      - RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT:-5}
-      - RUSTFS_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_LOCK_ACQUIRE_TIMEOUT:-5}
-      - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
-      # Internode gRPC optimization knobs (grpc-optimization P0-P3). Defaults match the binary
-      # defaults, so leaving these unset is a no-op; the A/B driver exports them to toggle a stage.
-      - RUSTFS_INTERNODE_RPC_TCP_NODELAY=${RUSTFS_INTERNODE_RPC_TCP_NODELAY:-true}
-      - RUSTFS_INTERNODE_RPC_HTTP2_STREAM_WINDOW_SIZE=${RUSTFS_INTERNODE_RPC_HTTP2_STREAM_WINDOW_SIZE:-1048576}
-      - RUSTFS_INTERNODE_RPC_HTTP2_CONN_WINDOW_SIZE=${RUSTFS_INTERNODE_RPC_HTTP2_CONN_WINDOW_SIZE:-2097152}
-      - RUSTFS_INTERNODE_RPC_MAX_MESSAGE_SIZE=${RUSTFS_INTERNODE_RPC_MAX_MESSAGE_SIZE:-104857600}
-      - RUSTFS_INTERNODE_CHANNEL_ISOLATION=${RUSTFS_INTERNODE_CHANNEL_ISOLATION:-false}
-      - RUSTFS_INTERNODE_BULK_CHANNELS=${RUSTFS_INTERNODE_BULK_CHANNELS:-2}
-      - RUSTFS_INTERNODE_RPC_MSGPACK_ONLY=${RUSTFS_INTERNODE_RPC_MSGPACK_ONLY:-false}
-      - RUSTFS_INTERNODE_PREWARM=${RUSTFS_INTERNODE_PREWARM:-false}
-      - RUSTFS_INTERNODE_OFFLINE_BYPASS=${RUSTFS_INTERNODE_OFFLINE_BYPASS:-false}
-      - RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS=${RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS:-5}
-      - RUSTFS_INTERNODE_OFFLINE_FAILURE_THRESHOLD=${RUSTFS_INTERNODE_OFFLINE_FAILURE_THRESHOLD:-3}
-      - RUSTFS_INTERNODE_IDEMPOTENT_READ_RETRIES=${RUSTFS_INTERNODE_IDEMPOTENT_READ_RETRIES:-1}
-    extra_hosts:
-      - "host.docker.internal:host-gateway"
-    volumes:
-      - node1_data_0:/data/rustfs0
-      - node1_data_1:/data/rustfs1
-      - node1_data_2:/data/rustfs2
-      - node1_data_3:/data/rustfs3
-    ports:
-      - "9000:9000"
-    networks:
-      - rustfs-cluster-net
-
-  node2:
-    image: ${RUSTFS_IMAGE:-rustfs/rustfs:local-4node}
-    build:
-      context: ../..
-      dockerfile: Dockerfile.source
-    hostname: node2
-    environment:
-      - RUSTFS_VOLUMES=${RUSTFS_VOLUMES:-http://node{1...4}:9000/data/rustfs{0...3}}
-      - RUSTFS_ADDRESS=:9000
-      - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin-local}
-      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfssecret-local}
-      - RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
-      - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
-      - RUSTFS_OBS_USE_STDOUT=${RUSTFS_OBS_USE_STDOUT:-false}
-      - RUSTFS_OBS_LOG_STDOUT_ENABLED=${RUSTFS_OBS_LOG_STDOUT_ENABLED:-false}
-      - RUSTFS_ISSUE3031_DIAG_ENABLE=${RUSTFS_ISSUE3031_DIAG_ENABLE:-false}
-      - RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT:-5}
-      - RUSTFS_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_LOCK_ACQUIRE_TIMEOUT:-5}
-      - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
-      # Internode gRPC optimization knobs (grpc-optimization P0-P3). Defaults match the binary
-      # defaults, so leaving these unset is a no-op; the A/B driver exports them to toggle a stage.
-      - RUSTFS_INTERNODE_RPC_TCP_NODELAY=${RUSTFS_INTERNODE_RPC_TCP_NODELAY:-true}
-      - RUSTFS_INTERNODE_RPC_HTTP2_STREAM_WINDOW_SIZE=${RUSTFS_INTERNODE_RPC_HTTP2_STREAM_WINDOW_SIZE:-1048576}
-      - RUSTFS_INTERNODE_RPC_HTTP2_CONN_WINDOW_SIZE=${RUSTFS_INTERNODE_RPC_HTTP2_CONN_WINDOW_SIZE:-2097152}
-      - RUSTFS_INTERNODE_RPC_MAX_MESSAGE_SIZE=${RUSTFS_INTERNODE_RPC_MAX_MESSAGE_SIZE:-104857600}
-      - RUSTFS_INTERNODE_CHANNEL_ISOLATION=${RUSTFS_INTERNODE_CHANNEL_ISOLATION:-false}
-      - RUSTFS_INTERNODE_BULK_CHANNELS=${RUSTFS_INTERNODE_BULK_CHANNELS:-2}
-      - RUSTFS_INTERNODE_RPC_MSGPACK_ONLY=${RUSTFS_INTERNODE_RPC_MSGPACK_ONLY:-false}
-      - RUSTFS_INTERNODE_PREWARM=${RUSTFS_INTERNODE_PREWARM:-false}
-      - RUSTFS_INTERNODE_OFFLINE_BYPASS=${RUSTFS_INTERNODE_OFFLINE_BYPASS:-false}
-      - RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS=${RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS:-5}
-      - RUSTFS_INTERNODE_OFFLINE_FAILURE_THRESHOLD=${RUSTFS_INTERNODE_OFFLINE_FAILURE_THRESHOLD:-3}
-      - RUSTFS_INTERNODE_IDEMPOTENT_READ_RETRIES=${RUSTFS_INTERNODE_IDEMPOTENT_READ_RETRIES:-1}
-    extra_hosts:
-      - "host.docker.internal:host-gateway"
-    volumes:
-      - node2_data_0:/data/rustfs0
-      - node2_data_1:/data/rustfs1
-      - node2_data_2:/data/rustfs2
-      - node2_data_3:/data/rustfs3
-    ports:
-      - "9001:9000"
-    networks:
-      - rustfs-cluster-net
-
-  node3:
-    image: ${RUSTFS_IMAGE:-rustfs/rustfs:local-4node}
-    build:
-      context: ../..
-      dockerfile: Dockerfile.source
-    hostname: node3
-    environment:
-      - RUSTFS_VOLUMES=${RUSTFS_VOLUMES:-http://node{1...4}:9000/data/rustfs{0...3}}
-      - RUSTFS_ADDRESS=:9000
-      - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin-local}
-      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfssecret-local}
-      - RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
-      - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
-      - RUSTFS_OBS_USE_STDOUT=${RUSTFS_OBS_USE_STDOUT:-false}
-      - RUSTFS_OBS_LOG_STDOUT_ENABLED=${RUSTFS_OBS_LOG_STDOUT_ENABLED:-false}
-      - RUSTFS_ISSUE3031_DIAG_ENABLE=${RUSTFS_ISSUE3031_DIAG_ENABLE:-false}
-      - RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT:-5}
-      - RUSTFS_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_LOCK_ACQUIRE_TIMEOUT:-5}
-      - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
-      # Internode gRPC optimization knobs (grpc-optimization P0-P3). Defaults match the binary
-      # defaults, so leaving these unset is a no-op; the A/B driver exports them to toggle a stage.
-      - RUSTFS_INTERNODE_RPC_TCP_NODELAY=${RUSTFS_INTERNODE_RPC_TCP_NODELAY:-true}
-      - RUSTFS_INTERNODE_RPC_HTTP2_STREAM_WINDOW_SIZE=${RUSTFS_INTERNODE_RPC_HTTP2_STREAM_WINDOW_SIZE:-1048576}
-      - RUSTFS_INTERNODE_RPC_HTTP2_CONN_WINDOW_SIZE=${RUSTFS_INTERNODE_RPC_HTTP2_CONN_WINDOW_SIZE:-2097152}
-      - RUSTFS_INTERNODE_RPC_MAX_MESSAGE_SIZE=${RUSTFS_INTERNODE_RPC_MAX_MESSAGE_SIZE:-104857600}
-      - RUSTFS_INTERNODE_CHANNEL_ISOLATION=${RUSTFS_INTERNODE_CHANNEL_ISOLATION:-false}
-      - RUSTFS_INTERNODE_BULK_CHANNELS=${RUSTFS_INTERNODE_BULK_CHANNELS:-2}
-      - RUSTFS_INTERNODE_RPC_MSGPACK_ONLY=${RUSTFS_INTERNODE_RPC_MSGPACK_ONLY:-false}
-      - RUSTFS_INTERNODE_PREWARM=${RUSTFS_INTERNODE_PREWARM:-false}
-      - RUSTFS_INTERNODE_OFFLINE_BYPASS=${RUSTFS_INTERNODE_OFFLINE_BYPASS:-false}
-      - RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS=${RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS:-5}
-      - RUSTFS_INTERNODE_OFFLINE_FAILURE_THRESHOLD=${RUSTFS_INTERNODE_OFFLINE_FAILURE_THRESHOLD:-3}
-      - RUSTFS_INTERNODE_IDEMPOTENT_READ_RETRIES=${RUSTFS_INTERNODE_IDEMPOTENT_READ_RETRIES:-1}
-    extra_hosts:
-      - "host.docker.internal:host-gateway"
-    volumes:
-      - node3_data_0:/data/rustfs0
-      - node3_data_1:/data/rustfs1
-      - node3_data_2:/data/rustfs2
-      - node3_data_3:/data/rustfs3
-    ports:
-      - "9002:9000"
-    networks:
-      - rustfs-cluster-net
-
-  node4:
-    image: ${RUSTFS_IMAGE:-rustfs/rustfs:local-4node}
-    build:
-      context: ../..
-      dockerfile: Dockerfile.source
-    hostname: node4
-    environment:
-      - RUSTFS_VOLUMES=${RUSTFS_VOLUMES:-http://node{1...4}:9000/data/rustfs{0...3}}
-      - RUSTFS_ADDRESS=:9000
-      - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin-local}
-      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfssecret-local}
-      - RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
-      - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
-      - RUSTFS_OBS_USE_STDOUT=${RUSTFS_OBS_USE_STDOUT:-false}
-      - RUSTFS_OBS_LOG_STDOUT_ENABLED=${RUSTFS_OBS_LOG_STDOUT_ENABLED:-false}
-      - RUSTFS_ISSUE3031_DIAG_ENABLE=${RUSTFS_ISSUE3031_DIAG_ENABLE:-false}
-      - RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT:-5}
-      - RUSTFS_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_LOCK_ACQUIRE_TIMEOUT:-5}
-      - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
-      # Internode gRPC optimization knobs (grpc-optimization P0-P3). Defaults match the binary
-      # defaults, so leaving these unset is a no-op; the A/B driver exports them to toggle a stage.
-      - RUSTFS_INTERNODE_RPC_TCP_NODELAY=${RUSTFS_INTERNODE_RPC_TCP_NODELAY:-true}
-      - RUSTFS_INTERNODE_RPC_HTTP2_STREAM_WINDOW_SIZE=${RUSTFS_INTERNODE_RPC_HTTP2_STREAM_WINDOW_SIZE:-1048576}
-      - RUSTFS_INTERNODE_RPC_HTTP2_CONN_WINDOW_SIZE=${RUSTFS_INTERNODE_RPC_HTTP2_CONN_WINDOW_SIZE:-2097152}
-      - RUSTFS_INTERNODE_RPC_MAX_MESSAGE_SIZE=${RUSTFS_INTERNODE_RPC_MAX_MESSAGE_SIZE:-104857600}
-      - RUSTFS_INTERNODE_CHANNEL_ISOLATION=${RUSTFS_INTERNODE_CHANNEL_ISOLATION:-false}
-      - RUSTFS_INTERNODE_BULK_CHANNELS=${RUSTFS_INTERNODE_BULK_CHANNELS:-2}
-      - RUSTFS_INTERNODE_RPC_MSGPACK_ONLY=${RUSTFS_INTERNODE_RPC_MSGPACK_ONLY:-false}
-      - RUSTFS_INTERNODE_PREWARM=${RUSTFS_INTERNODE_PREWARM:-false}
-      - RUSTFS_INTERNODE_OFFLINE_BYPASS=${RUSTFS_INTERNODE_OFFLINE_BYPASS:-false}
-      - RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS=${RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS:-5}
-      - RUSTFS_INTERNODE_OFFLINE_FAILURE_THRESHOLD=${RUSTFS_INTERNODE_OFFLINE_FAILURE_THRESHOLD:-3}
-      - RUSTFS_INTERNODE_IDEMPOTENT_READ_RETRIES=${RUSTFS_INTERNODE_IDEMPOTENT_READ_RETRIES:-1}
-    extra_hosts:
-      - "host.docker.internal:host-gateway"
-    volumes:
-      - node4_data_0:/data/rustfs0
-      - node4_data_1:/data/rustfs1
-      - node4_data_2:/data/rustfs2
-      - node4_data_3:/data/rustfs3
-    ports:
-      - "9003:9000"
-    networks:
-      - rustfs-cluster-net
-
-volumes:
-  node1_data_0:
-  node1_data_1:
-  node1_data_2:
-  node1_data_3:
-  node2_data_0:
-  node2_data_1:
-  node2_data_2:
-  node2_data_3:
-  node3_data_0:
-  node3_data_1:
-  node3_data_2:
-  node3_data_3:
-  node4_data_0:
-  node4_data_1:
-  node4_data_2:
-  node4_data_3:
-
-networks:
-  rustfs-cluster-net:
-    driver: bridge

.docker/compose/docker-compose.cluster.perf-round1-linux-32c64g-latency-stable.override.yml

@@ -1,56 +0,0 @@
-services:
-  node1:
-    environment:
-      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=56
-      - RUSTFS_OBJECT_IO_BUFFER_SIZE=131072
-      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=10
-      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=20
-      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=10
-      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
-      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=25165824
-      - RUSTFS_RUNTIME_WORKER_THREADS=16
-      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=768
-      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=420
-      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=50
-
-  node2:
-    environment:
-      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=56
-      - RUSTFS_OBJECT_IO_BUFFER_SIZE=131072
-      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=10
-      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=20
-      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=10
-      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
-      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=25165824
-      - RUSTFS_RUNTIME_WORKER_THREADS=16
-      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=768
-      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=420
-      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=50
-
-  node3:
-    environment:
-      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=56
-      - RUSTFS_OBJECT_IO_BUFFER_SIZE=131072
-      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=10
-      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=20
-      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=10
-      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
-      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=25165824
-      - RUSTFS_RUNTIME_WORKER_THREADS=16
-      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=768
-      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=420
-      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=50
-
-  node4:
-    environment:
-      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=56
-      - RUSTFS_OBJECT_IO_BUFFER_SIZE=131072
-      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=10
-      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=20
-      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=10
-      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
-      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=25165824
-      - RUSTFS_RUNTIME_WORKER_THREADS=16
-      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=768
-      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=420
-      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=50

.docker/compose/docker-compose.cluster.perf-round1-linux-32c64g-throughput-max.override.yml

@@ -1,56 +0,0 @@
-services:
-  node1:
-    environment:
-      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=96
-      - RUSTFS_OBJECT_IO_BUFFER_SIZE=524288
-      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=16
-      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=32
-      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=24
-      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=33554432
-      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=50331648
-      - RUSTFS_RUNTIME_WORKER_THREADS=24
-      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=1536
-      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=600
-      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=80
-
-  node2:
-    environment:
-      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=96
-      - RUSTFS_OBJECT_IO_BUFFER_SIZE=524288
-      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=16
-      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=32
-      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=24
-      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=33554432
-      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=50331648
-      - RUSTFS_RUNTIME_WORKER_THREADS=24
-      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=1536
-      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=600
-      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=80
-
-  node3:
-    environment:
-      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=96
-      - RUSTFS_OBJECT_IO_BUFFER_SIZE=524288
-      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=16
-      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=32
-      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=24
-      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=33554432
-      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=50331648
-      - RUSTFS_RUNTIME_WORKER_THREADS=24
-      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=1536
-      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=600
-      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=80
-
-  node4:
-    environment:
-      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=96
-      - RUSTFS_OBJECT_IO_BUFFER_SIZE=524288
-      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=16
-      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=32
-      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=24
-      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=33554432
-      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=50331648
-      - RUSTFS_RUNTIME_WORKER_THREADS=24
-      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=1536
-      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=600
-      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=80

.docker/compose/docker-compose.cluster.perf-round1-linux-32c64g.override.yml

@@ -1,56 +0,0 @@
-services:
-  node1:
-    environment:
-      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=64
-      - RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
-      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=12
-      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=24
-      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=16
-      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=16777216
-      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=33554432
-      - RUSTFS_RUNTIME_WORKER_THREADS=20
-      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=1024
-      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
-      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=60
-
-  node2:
-    environment:
-      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=64
-      - RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
-      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=12
-      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=24
-      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=16
-      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=16777216
-      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=33554432
-      - RUSTFS_RUNTIME_WORKER_THREADS=20
-      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=1024
-      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
-      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=60
-
-  node3:
-    environment:
-      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=64
-      - RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
-      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=12
-      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=24
-      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=16
-      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=16777216
-      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=33554432
-      - RUSTFS_RUNTIME_WORKER_THREADS=20
-      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=1024
-      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
-      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=60
-
-  node4:
-    environment:
-      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=64
-      - RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
-      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=12
-      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=24
-      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=16
-      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=16777216
-      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=33554432
-      - RUSTFS_RUNTIME_WORKER_THREADS=20
-      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=1024
-      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
-      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=60

.docker/compose/docker-compose.cluster.perf-round1-linux.override.yml

@@ -1,56 +0,0 @@
-services:
-  node1:
-    environment:
-      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=48
-      - RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
-      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=8
-      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=16
-      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=12
-      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
-      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=25165824
-      - RUSTFS_RUNTIME_WORKER_THREADS=12
-      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=512
-      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
-      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=40
-
-  node2:
-    environment:
-      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=48
-      - RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
-      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=8
-      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=16
-      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=12
-      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
-      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=25165824
-      - RUSTFS_RUNTIME_WORKER_THREADS=12
-      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=512
-      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
-      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=40
-
-  node3:
-    environment:
-      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=48
-      - RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
-      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=8
-      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=16
-      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=12
-      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
-      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=25165824
-      - RUSTFS_RUNTIME_WORKER_THREADS=12
-      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=512
-      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
-      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=40
-
-  node4:
-    environment:
-      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=48
-      - RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
-      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=8
-      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=16
-      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=12
-      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
-      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=25165824
-      - RUSTFS_RUNTIME_WORKER_THREADS=12
-      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=512
-      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
-      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=40

.docker/compose/docker-compose.cluster.perf-round1.override.yml

@@ -1,56 +0,0 @@
-services:
-  node1:
-    environment:
-      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=32
-      - RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
-      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=6
-      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=12
-      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=8
-      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
-      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=16777216
-      - RUSTFS_RUNTIME_WORKER_THREADS=6
-      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=256
-      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
-      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=30
-
-  node2:
-    environment:
-      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=32
-      - RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
-      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=6
-      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=12
-      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=8
-      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
-      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=16777216
-      - RUSTFS_RUNTIME_WORKER_THREADS=6
-      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=256
-      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
-      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=30
-
-  node3:
-    environment:
-      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=32
-      - RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
-      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=6
-      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=12
-      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=8
-      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
-      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=16777216
-      - RUSTFS_RUNTIME_WORKER_THREADS=6
-      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=256
-      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
-      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=30
-
-  node4:
-    environment:
-      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=32
-      - RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
-      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=6
-      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=12
-      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=8
-      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
-      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=16777216
-      - RUSTFS_RUNTIME_WORKER_THREADS=6
-      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=256
-      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
-      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=30

.docker/compose/docker-compose.cluster.yaml

@@ -21,8 +21,8 @@ services:
       - RUSTFS_VOLUMES=http://node{0...3}:9000/data/rustfs{0...3}
       - RUSTFS_ADDRESS=0.0.0.0:9000
       - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin-local}
-      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfssecret-local}
+      - RUSTFS_ACCESS_KEY=rustfsadmin
+      - RUSTFS_SECRET_KEY=rustfsadmin
     platform: linux/amd64
     ports:
       - "9000:9000" # Map port 9001 of the host to port 9000 of the container
@@ -38,8 +38,8 @@ services:
       - RUSTFS_VOLUMES=http://node{0...3}:9000/data/rustfs{0...3}
       - RUSTFS_ADDRESS=0.0.0.0:9000
       - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin-local}
-      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfssecret-local}
+      - RUSTFS_ACCESS_KEY=rustfsadmin
+      - RUSTFS_SECRET_KEY=rustfsadmin
     platform: linux/amd64
     ports:
       - "9001:9000" # Map port 9002 of the host to port 9000 of the container
@@ -55,8 +55,8 @@ services:
       - RUSTFS_VOLUMES=http://node{0...3}:9000/data/rustfs{0...3}
       - RUSTFS_ADDRESS=0.0.0.0:9000
       - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin-local}
-      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfssecret-local}
+      - RUSTFS_ACCESS_KEY=rustfsadmin
+      - RUSTFS_SECRET_KEY=rustfsadmin
     platform: linux/amd64
     ports:
       - "9002:9000" # Map port 9003 of the host to port 9000 of the container
@@ -72,8 +72,8 @@ services:
       - RUSTFS_VOLUMES=http://node{0...3}:9000/data/rustfs{0...3}
       - RUSTFS_ADDRESS=0.0.0.0:9000
       - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin-local}
-      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfssecret-local}
+      - RUSTFS_ACCESS_KEY=rustfsadmin
+      - RUSTFS_SECRET_KEY=rustfsadmin
     platform: linux/amd64
     ports:
       - "9003:9000" # Map port 9004 of the host to port 9000 of the container

.docker/compose/docker-compose.observability.yaml

@@ -13,183 +13,62 @@
 # limitations under the License.
 
 services:
-  # --- Observability Stack ---
-
-  tempo-init:
-    image: busybox:latest
-    command: [ "sh", "-c", "chown -R 10001:10001 /var/tempo" ]
-    volumes:
-      - tempo-data:/var/tempo
-    user: root
-    networks:
-      - rustfs-network
-    restart: "no"
-
-  tempo:
-    image: grafana/tempo:2.10.5
-    user: "10001"
-    command: [ "-config.file=/etc/tempo.yaml" ]
-    volumes:
-      - ../../.docker/observability/tempo.yaml:/etc/tempo.yaml:ro
-      - tempo-data:/var/tempo
-    ports:
-      - "3200:3200" # tempo
-      - "4317"      # otlp grpc
-      - "4318"      # otlp http
-      - "7946"      # memberlist
-    restart: unless-stopped
-    networks:
-      - rustfs-network
-    depends_on:
-      tempo-init:
-        condition: service_completed_successfully
-    healthcheck:
-      test: [ "CMD", "/tempo", "-version" ]
-      interval: 10s
-      timeout: 5s
-      retries: 3
-      start_period: 15s
-
   otel-collector:
-    image: otel/opentelemetry-collector-contrib:latest
+    image: otel/opentelemetry-collector-contrib:0.129.1
     environment:
       - TZ=Asia/Shanghai
     volumes:
-      - ../../.docker/observability/otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml:ro
+      - ../../.docker/observability/otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml
     ports:
-      - "1888:1888"   # pprof
-      - "8888:8888"   # Prometheus metrics for Collector
-      - "8889:8889"   # Prometheus metrics for application indicators
-      - "13133:13133" # health check
-      - "4317:4317"   # OTLP gRPC
-      - "4318:4318"   # OTLP HTTP
-      - "55679:55679" # zpages
+      - 1888:1888
+      - 8888:8888
+      - 8889:8889
+      - 13133:13133
+      - 4317:4317
+      - 4318:4318
+      - 55679:55679
     networks:
       - rustfs-network
-    depends_on:
-      - tempo
-      - jaeger
-      - prometheus
-      - loki
-    restart: unless-stopped
-    healthcheck:
-      test: [ "CMD", "/otelcol-contrib", "--version" ]
-      interval: 10s
-      timeout: 5s
-      retries: 3
-
   jaeger:
-    image: jaegertracing/jaeger:latest
+    image: jaegertracing/jaeger:2.8.0
     environment:
       - TZ=Asia/Shanghai
-      - SPAN_STORAGE_TYPE=badger
-      - BADGER_EPHEMERAL=false
-      - BADGER_DIRECTORY_VALUE=/badger/data
-      - BADGER_DIRECTORY_KEY=/badger/key
-      - COLLECTOR_OTLP_ENABLED=true
-    volumes:
-      - ../../.docker/observability/jaeger.yaml:/etc/jaeger/config.yml:ro
-      - jaeger-data:/badger
     ports:
-      - "16686:16686" # Web UI
-      - "14269:14269" # Admin/Metrics
-      - "4317"        # otlp grpc
-      - "4318"        # otlp http
-    command: [ "--config", "/etc/jaeger/config.yml" ]
+      - "16686:16686"
+      - "14317:4317"
+      - "14318:4318"
     networks:
       - rustfs-network
-    restart: unless-stopped
-    healthcheck:
-      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:14269" ]
-      interval: 10s
-      timeout: 5s
-      retries: 3
-      start_period: 15s
-
   prometheus:
-    image: prom/prometheus:latest
+    image: prom/prometheus:v3.4.2
     environment:
       - TZ=Asia/Shanghai
     volumes:
-      - ../../.docker/observability/prometheus.yml:/etc/prometheus/prometheus.yml:ro
-      - ../../.docker/observability/prometheus-rules:/etc/prometheus/rules:ro
-      - prometheus-data:/prometheus
+      - ../../.docker/observability/prometheus.yml:/etc/prometheus/prometheus.yml
     ports:
       - "9090:9090"
-    command:
-      - '--config.file=/etc/prometheus/prometheus.yml'
-      - '--web.enable-otlp-receiver'
-      - '--web.enable-remote-write-receiver'
-      - '--enable-feature=promql-experimental-functions'
-      - '--storage.tsdb.path=/prometheus'
-      - '--storage.tsdb.retention.time=30d'
     networks:
       - rustfs-network
-    restart: unless-stopped
-    healthcheck:
-      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:9090/-/healthy" ]
-      interval: 10s
-      timeout: 5s
-      retries: 3
-
   loki:
-    image: grafana/loki:latest
+    image: grafana/loki:3.5.1
     environment:
       - TZ=Asia/Shanghai
     volumes:
-      - ../../.docker/observability/loki.yaml:/etc/loki/loki.yaml:ro
-      - loki-data:/loki
+      - ../../.docker/observability/loki-config.yaml:/etc/loki/local-config.yaml
     ports:
       - "3100:3100"
-    command: -config.file=/etc/loki/loki.yaml
+    command: -config.file=/etc/loki/local-config.yaml
     networks:
       - rustfs-network
-    restart: unless-stopped
-    healthcheck:
-      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:3100/ready" ]
-      interval: 15s
-      timeout: 10s
-      retries: 5
-      start_period: 60s
-
-  pyroscope:
-    image: grafana/pyroscope:latest
-    ports:
-      - "4040:4040"
-    command:
-      - -self-profiling.disable-push=true
-    networks:
-      - rustfs-network
-    restart: unless-stopped
-
   grafana:
-    image: grafana/grafana:latest
+    image: grafana/grafana:12.0.2
     ports:
       - "3000:3000" # Web UI
     environment:
       - GF_SECURITY_ADMIN_PASSWORD=admin
-      - GF_SECURITY_ADMIN_USER=admin
       - TZ=Asia/Shanghai
-      - GF_INSTALL_PLUGINS=grafana-pyroscope-datasource
-      - GF_DASHBOARDS_DEFAULT_HOME_DASHBOARD_PATH=/var/lib/grafana/dashboards/home.json
     networks:
       - rustfs-network
-    volumes:
-      - ../../.docker/observability/grafana/provisioning:/etc/grafana/provisioning:ro
-      - ../../.docker/observability/grafana/dashboards:/var/lib/grafana/dashboards:ro
-      - grafana-data:/var/lib/grafana
-    depends_on:
-      - prometheus
-      - tempo
-      - loki
-    restart: unless-stopped
-    healthcheck:
-      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:3000/api/health" ]
-      interval: 10s
-      timeout: 5s
-      retries: 3
-
-  # --- RustFS Cluster ---
 
   node1:
     build:
@@ -200,15 +79,13 @@ services:
       - RUSTFS_VOLUMES=http://node{1...4}:9000/root/data/target/volume/test{1...4}
       - RUSTFS_ADDRESS=:9000
       - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_OBS_ENDPOINT=http://otel-collector:4318
+      - RUSTFS_OBS_ENDPOINT=http://otel-collector:4317
       - RUSTFS_OBS_LOGGER_LEVEL=debug
     platform: linux/amd64
     ports:
-      - "9001:9000"
+      - "9001:9000" # Map port 9001 of the host to port 9000 of the container
     networks:
       - rustfs-network
-    depends_on:
-      - otel-collector
 
   node2:
     build:
@@ -219,15 +96,13 @@ services:
       - RUSTFS_VOLUMES=http://node{1...4}:9000/root/data/target/volume/test{1...4}
       - RUSTFS_ADDRESS=:9000
       - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_OBS_ENDPOINT=http://otel-collector:4318
+      - RUSTFS_OBS_ENDPOINT=http://otel-collector:4317
       - RUSTFS_OBS_LOGGER_LEVEL=debug
     platform: linux/amd64
     ports:
-      - "9002:9000"
+      - "9002:9000" # Map port 9002 of the host to port 9000 of the container
     networks:
       - rustfs-network
-    depends_on:
-      - otel-collector
 
   node3:
     build:
@@ -238,15 +113,13 @@ services:
       - RUSTFS_VOLUMES=http://node{1...4}:9000/root/data/target/volume/test{1...4}
       - RUSTFS_ADDRESS=:9000
       - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_OBS_ENDPOINT=http://otel-collector:4318
+      - RUSTFS_OBS_ENDPOINT=http://otel-collector:4317
       - RUSTFS_OBS_LOGGER_LEVEL=debug
     platform: linux/amd64
     ports:
-      - "9003:9000"
+      - "9003:9000" # Map port 9003 of the host to port 9000 of the container
     networks:
       - rustfs-network
-    depends_on:
-      - otel-collector
 
   node4:
     build:
@@ -257,22 +130,13 @@ services:
       - RUSTFS_VOLUMES=http://node{1...4}:9000/root/data/target/volume/test{1...4}
       - RUSTFS_ADDRESS=:9000
       - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_OBS_ENDPOINT=http://otel-collector:4318
+      - RUSTFS_OBS_ENDPOINT=http://otel-collector:4317
       - RUSTFS_OBS_LOGGER_LEVEL=debug
     platform: linux/amd64
     ports:
-      - "9004:9000"
+      - "9004:9000" # Map port 9004 of the host to port 9000 of the container
     networks:
       - rustfs-network
-    depends_on:
-      - otel-collector
-
-volumes:
-  prometheus-data:
-  tempo-data:
-  loki-data:
-  jaeger-data:
-  grafana-data:
 
 networks:
   rustfs-network:

.docker/mqtt/README.md

@@ -1,30 +0,0 @@
-# MQTT Broker (EMQX)
-
-This directory contains the configuration for running an EMQX MQTT broker, which can be used for testing RustFS's MQTT integration.
-
-## 🚀 Quick Start
-
-To start the EMQX broker:
-
-```bash
-docker compose up -d
-```
-
-## 📊 Access
-
-- **Dashboard**: [http://localhost:18083](http://localhost:18083)
-- **Default Credentials**: `admin` / `public`
-- **MQTT Port**: `1883`
-- **WebSocket Port**: `8083`
-
-## 🛠️ Configuration
-
-The `docker-compose.yml` file sets up a single-node EMQX instance.
-
-- **Persistence**: Data is not persisted by default (for testing).
-- **Network**: Uses the default bridge network.
-
-## 📝 Notes
-
-- This setup is intended for development and testing purposes.
-- For production deployments, please refer to the official [EMQX Documentation](https://www.emqx.io/docs/en/latest/).

.docker/nginx/nginx.conf

@@ -1,93 +0,0 @@
-# Copyright 2024 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-worker_processes auto;
-pid /var/run/nginx.pid;
-
-events {
-    worker_connections 1024;
-}
-
-http {
-    include /etc/nginx/mime.types;
-    default_type application/octet-stream;
-
-    log_format main '$remote_addr - $remote_user [$time_local] "$request" '
-                    '$status $body_bytes_sent "$http_referer" '
-                    '"$http_user_agent" "$http_x_forwarded_for"';
-
-    access_log /var/log/nginx/access.log main;
-    error_log /var/log/nginx/error.log warn;
-
-    sendfile on;
-    keepalive_timeout 65;
-
-    # RustFS Server Block
-    server {
-        listen 80;
-        server_name localhost;
-
-        # Redirect HTTP to HTTPS (optional, uncomment if SSL is configured)
-        # return 301 https://$host$request_uri;
-
-        location / {
-            proxy_pass http://rustfs:9000;
-            proxy_set_header Host $host;
-            proxy_set_header X-Real-IP $remote_addr;
-            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
-            proxy_set_header X-Forwarded-Proto $scheme;
-
-            # S3 specific headers
-            proxy_set_header X-Amz-Date $http_x_amz_date;
-            proxy_set_header Authorization $http_authorization;
-
-            # Disable buffering for large uploads
-            proxy_request_buffering off;
-            client_max_body_size 0;
-        }
-
-        location /rustfs/console {
-            proxy_pass http://rustfs:9001;
-            proxy_set_header Host $host;
-            proxy_set_header X-Real-IP $remote_addr;
-            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
-            proxy_set_header X-Forwarded-Proto $scheme;
-        }
-    }
-
-    # SSL Configuration (Example)
-    # server {
-    #     listen 443 ssl;
-    #     server_name localhost;
-    #
-    #     ssl_certificate /etc/nginx/ssl/server.crt;
-    #     ssl_certificate_key /etc/nginx/ssl/server.key;
-    #
-    #     # Restrict to modern TLS versions and ciphers. Operators copying this
-    #     # example must keep at least these directives — without them, nginx
-    #     # may negotiate older protocol versions that have known weaknesses.
-    #     ssl_protocols TLSv1.2 TLSv1.3;
-    #     ssl_prefer_server_ciphers on;
-    #     ssl_ciphers 'ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305';
-    #     ssl_session_timeout 1d;
-    #     ssl_session_cache shared:SSL:10m;
-    #     ssl_session_tickets off;
-    #     # add_header Strict-Transport-Security "max-age=63072000" always;
-    #
-    #     location / {
-    #         proxy_pass http://rustfs:9000;
-    #         ...
-    #     }
-    # }
-}

.docker/observability/.gitignore

@@ -1,5 +0,0 @@
-jaeger-data/*
-loki-data/*
-prometheus-data/*
-tempo-data/*
-grafana-data/*

.docker/observability/README.md

@@ -1,221 +1,109 @@
-# RustFS Observability Stack
+# Observability
 
-This directory contains the comprehensive observability stack for RustFS, designed to provide deep insights into application performance, logs, and traces.
+This directory contains the observability stack for the application. The stack is composed of the following components:
 
-## Components
+- Prometheus v3.2.1
+- Grafana 11.6.0
+- Loki 3.4.2
+- Jaeger 2.4.0
+- Otel Collector 0.120.0 # 0.121.0 remove loki
 
-The stack is composed of the following best-in-class open-source components:
+## Prometheus
 
-- **Prometheus** (v2.53.1): The industry standard for metric collection and alerting.
-- **Grafana** (v11.1.0): The leading platform for observability visualization.
-- **Loki** (v3.1.0): A horizontally-scalable, highly-available, multi-tenant log aggregation system.
-- **Tempo** (v2.5.0): A high-volume, minimal dependency distributed tracing backend.
-- **Jaeger** (v1.59.0): Distributed tracing system (configured as a secondary UI/storage).
-- **OpenTelemetry Collector** (v0.104.0): A vendor-agnostic implementation for receiving, processing, and exporting telemetry data.
+Prometheus is a monitoring and alerting toolkit. It scrapes metrics from instrumented jobs, either directly or via an
+intermediary push gateway for short-lived jobs. It stores all scraped samples locally and runs rules over this data to
+either aggregate and record new time series from existing data or generate alerts. Grafana or other API consumers can be
+used to visualize the collected data.
 
-By default, this stack uses Tempo in single-binary mode and does not require Kafka/Redpanda.
-If you want the Kafka-backed HA Tempo path, use `docker-compose-example-for-rustfs.yml` together with `docker-compose-tempo-ha-override.yml`.
+## Grafana
 
-## Architecture
+Grafana is a multi-platform open-source analytics and interactive visualization web application. It provides charts,
+graphs, and alerts for the web when connected to supported data sources.
 
-1. **Telemetry Collection**: Applications send OTLP (OpenTelemetry Protocol) data (Metrics, Logs, Traces) to the **OpenTelemetry Collector**.
-2. **Processing & Exporting**: The Collector processes the data (batching, memory limiting) and exports it to the respective backends:
-    - **Traces** -> **Tempo** (Primary) & **Jaeger** (Secondary/Optional)
-    - **Metrics** -> **Prometheus** (via scraping the Collector's exporter)
-    - **Logs** -> **Loki**
-3. **Visualization**: **Grafana** connects to all backends (Prometheus, Tempo, Loki, Jaeger) to provide a unified dashboard experience.
+## Loki
 
-## Features
+Loki is a horizontally-scalable, highly-available, multi-tenant log aggregation system inspired by Prometheus. It is
+designed to be very cost-effective and easy to operate. It does not index the contents of the logs, but rather a set of
+labels for each log stream.
 
-- **Full Persistence**: All data (Metrics, Logs, Traces) is persisted to Docker volumes, ensuring no data loss on restart.
-- **Correlation**: Seamless navigation between Metrics, Logs, and Traces in Grafana.
-  - Jump from a Metric spike to relevant Traces.
-  - Jump from a Trace to relevant Logs.
-- **High Performance**: Optimized configurations for batching, compression, and memory management.
-- **Standardized Protocols**: Built entirely on OpenTelemetry standards.
+## Jaeger
 
-## GET Performance Optimization Dashboards
+Jaeger is a distributed tracing system released as open source by Uber Technologies. It is used for monitoring and
+troubleshooting microservices-based distributed systems, including:
 
-Three pre-built Grafana dashboards are included for monitoring RustFS GET performance optimization rollout:
+- Distributed context propagation
+- Distributed transaction monitoring
+- Root cause analysis
+- Service dependency analysis
+- Performance / latency optimization
 
-### Available Dashboards
+## Otel Collector
 
-| Dashboard | File | Description |
-|-----------|------|-------------|
-| **GET Rollout Health** | `grafana-get-rollout-health.json` | Monitors optimization rollout: latency by reader path, early-stop hit rate, codec streaming usage, pipeline failures |
-| **GET Data Integrity** | `grafana-get-data-integrity.json` | Monitors data safety: bitrot verify failures, decode errors, short reads, shard read outcomes |
-| **GET Resource Impact** | `grafana-get-resource-impact.json` | Monitors resource usage: concurrent requests, IO queue utilization, disk permit wait, RSS trend |
-| **Object Data Cache** | `grafana-object-data-cache.json` | Monitors the GET body cache (`rustfs_object_data_cache_*`): hit ratio, lookup/plan/fill outcomes, fill duration quantiles, hit vs fill throughput, entries/weighted bytes, inflight fills, memory-pressure skips, invalidations, and size-class breakdowns |
+The OpenTelemetry Collector offers a vendor-agnostic implementation on how to receive, process, and export telemetry
+data. It removes the need to run, operate, and maintain multiple agents/collectors in order to support open-source
+observability data formats (e.g. Jaeger, Prometheus, etc.) sending to one or more open-source or commercial back-ends.
 
-### Prometheus Alert Rules
+## How to use
 
-The file `prometheus-rules/rustfs-get-optimization-alerts.yaml` contains pre-configured alerting rules:
+To deploy the observability stack, run the following command:
 
-| Alert | Severity | Condition |
-|-------|----------|-----------|
-| `GetP99Regression` | Critical | GET p99 latency > 2x baseline for 10m |
-| `PipelineFailureSpike` | Critical | Pipeline failure rate > 5x baseline for 5m |
-| `BitrotMismatchSpike` | Critical | Bitrot mismatch rate > 3x baseline for 5m |
-| `EarlyStopInsufficientQuorum` | Warning | Early-stop insufficient quorum rate > 0.1/s for 5m |
-| `CodecStreamingFallbackSpike` | Warning | Codec streaming fallback > 10x baseline for 10m |
-| `IoQueueSaturation` | Warning | IO queue utilization > 90% for 5m |
+- docker latest version
 
-The file `prometheus-rules/rustfs-kms-alerts.yml` contains alerting rules for the KMS backend operation metrics. Thresholds are conservative defaults pending staging baseline calibration; response procedures live in `docs/operations/kms-observability-runbook.md`, and the matching dashboard is `deploy/observability/grafana/rustfs-kms-observability.json`.
+```bash
+docker compose -f docker-compose.yml -f docker-compose.override.yml up -d
+```
 
-| Alert | Severity | Condition |
-|-------|----------|-----------|
-| `KmsBackendFatalErrors` | Critical | Fatal (non-retryable) attempt failures > 0 for 5m |
-| `KmsBackendHighErrorRate` | Critical | Non-success operation ratio > 5% for 10m (with traffic guard) |
-| `KmsBackendP99LatencyHigh` | Warning | Operation p99 duration (incl. retries) > 2s for 10m |
-| `KmsBackendAttemptFailureSpike` | Warning | Attempt failure rate > 0.5/s for 10m |
-| `KmsBackendRetryBudgetExhausted` | Warning | budget_exhausted / deadline_exceeded outcomes > 0.05/s for 10m |
+- docker compose v2.0.0 or before
 
-### Enabling Alert Rules
+```bash
+docke-compose -f docker-compose.yml -f docker-compose.override.yml up -d
+```
 
-Add the alert rules file to your Prometheus configuration:
+To access the Grafana dashboard, navigate to `http://localhost:3000` in your browser. The default username and password
+are `admin` and `admin`, respectively.
+
+To access the Jaeger dashboard, navigate to `http://localhost:16686` in your browser.
+
+To access the Prometheus dashboard, navigate to `http://localhost:9090` in your browser.
+
+## How to stop
+
+To stop the observability stack, run the following command:
+
+```bash
+docker compose -f docker-compose.yml -f docker-compose.override.yml down
+```
+
+## How to remove data
+
+To remove the data generated by the observability stack, run the following command:
+
+```bash
+docker compose -f docker-compose.yml -f docker-compose.override.yml down -v
+```
+
+## How to configure
+
+To configure the observability stack, modify the `docker-compose.override.yml` file. The file contains the following
 
 ```yaml
-# prometheus.yml
-rule_files:
-  - "/etc/prometheus/rules/*.yml"
+services:
+  prometheus:
+    environment:
+      - PROMETHEUS_CONFIG_FILE=/etc/prometheus/prometheus.yml
+    volumes:
+      - ./prometheus.yml:/etc/prometheus/prometheus.yml
 
-# Or mount the file in docker-compose.yml:
-# volumes:
-#   - ./prometheus-rules:/etc/prometheus/rules
+  grafana:
+    environment:
+      - GF_SECURITY_ADMIN_PASSWORD=admin
+    volumes:
+      - ./grafana/provisioning:/etc/grafana/provisioning
 ```
 
-### Dashboard Usage
+The `prometheus` service mounts the `prometheus.yml` file to `/etc/prometheus/prometheus.yml`. The `grafana` service
+mounts the `grafana/provisioning` directory to `/etc/grafana/provisioning`. You can modify these files to configure the
+observability stack.
 
-The dashboards are automatically provisioned when Grafana starts. They use the `${DS_PROMETHEUS}` datasource variable, so you need a Prometheus datasource configured in Grafana.
 
-Key panels to monitor during optimization rollout:
 
-1. **GET Latency by Reader Path** - Compare `codec_streaming` vs `legacy_duplex` latency
-2. **Early-Stop Hit Rate** - Verify early-stop is triggering effectively
-3. **Pipeline Failure Rate** - Detect any new failure modes introduced by optimizations
-4. **Bitrot Verify Failures** - Ensure data integrity is maintained
-
-## Quick Start
-
-### Prerequisites
-
-- Docker
-- Docker Compose
-
-### Deploy
-
-Run the following command to start the entire stack:
-
-```bash
-docker compose up -d
-```
-
-### High Availability Tempo
-
-The default `docker-compose.yml` is the single-node stack.
-If you need the Kafka-backed HA Tempo configuration, start it with:
-
-```bash
-docker compose -f docker-compose-example-for-rustfs.yml -f docker-compose-tempo-ha-override.yml up -d
-```
-
-### Access Dashboards
-
-| Service        | URL                                              | Credentials       | Description                    |
-| :------------- | :----------------------------------------------- | :---------------- | :----------------------------- |
-| **Grafana**    | [http://localhost:3000](http://localhost:3000)   | `admin` / `admin` | Main visualization hub.        |
-| **Prometheus** | [http://localhost:9090](http://localhost:9090)   | -                 | Metric queries and status.     |
-| **Jaeger UI**  | [http://localhost:16686](http://localhost:16686) | -                 | Secondary trace visualization. |
-| **Tempo**      | [http://localhost:3200](http://localhost:3200)   | -                 | Tempo status/metrics.          |
-
-## Configuration
-
-### Data Persistence
-
-Data is stored in the following Docker volumes:
-
-- `prometheus-data`: Prometheus metrics
-- `tempo-data`: Tempo traces (WAL and Blocks)
-- `loki-data`: Loki logs (Chunks and Rules)
-- `jaeger-data`: Jaeger traces (Badger DB)
-
-To clear all data:
-
-```bash
-docker compose down -v
-```
-
-### Customization
-
-- **Prometheus**: Edit `prometheus.yml` to add scrape targets or alerting rules.
-- **Grafana**: Dashboards and datasources are provisioned from the `grafana/` directory.
-- **Collector**: Edit `otel-collector-config.yaml` to modify pipelines, processors, or exporters.
-
-### Verifying RustFS Traces
-
-When RustFS points `RUSTFS_OBS_ENDPOINT` at this stack, treat the value as the
-OTLP/HTTP base URL, for example:
-
-```bash
-export RUSTFS_OBS_ENDPOINT=http://host.docker.internal:4318
-```
-
-RustFS automatically expands that base URL to:
-
-- `/v1/traces`
-- `/v1/metrics`
-- `/v1/logs`
-
-Important behavior notes:
-
-- Logs and metrics usually appear during startup, so seeing those two signals
-  first is expected.
-- The OpenTelemetry bridge sends `tracing` fields as log attributes. Loki stores
-  those attributes as structured metadata, and the Collector also mirrors the
-  common troubleshooting fields into the log line so simple line filters can
-  find them.
-- Visible trace data usually requires real HTTP/S3/gRPC request traffic after
-  startup, because request-path spans are created on demand.
-- `RUSTFS_OBS_LOGGER_LEVEL=info` keeps the top-level request span but filters
-  many nested `debug` spans. If Tempo or Jaeger looks sparse, retry with
-  `RUSTFS_OBS_LOGGER_LEVEL=debug` before suspecting collector or Tempo issues.
-
-Minimal validation flow:
-
-```bash
-# 1. Start this observability stack.
-docker compose up -d
-
-# 2. Start RustFS with OTLP/HTTP export and richer span visibility.
-export RUSTFS_OBS_ENDPOINT=http://host.docker.internal:4318
-export RUSTFS_OBS_LOGGER_LEVEL=debug
-
-# 3. Generate real request traffic.
-curl -I http://127.0.0.1:9000/health
-curl -I http://127.0.0.1:9000/health/ready
-
-# 4. Inspect Grafana or Jaeger.
-# Grafana: http://localhost:3000
-# Jaeger:  http://localhost:16686
-```
-
-For a structured RustFS log such as an inter-node RPC authentication failure,
-the Loki line now includes fields such as `event`, `component`, `subsystem`,
-`failure_reason`, `rpc_service`, `rpc_method`, and `expected_audience`. Useful
-LogQL checks:
-
-```logql
-{service_name="RustFS"} |= "RPC signature verification failed"
-{service_name="RustFS"} |= "failure_reason="
-{service_name="RustFS"} | failure_reason != ""
-```
-
-If logs and metrics are present but traces are sparse, the most common cause is
-"no real request traffic yet" or "`info` level filtered nested spans", not an
-OTLP routing failure.
-
-## Troubleshooting
-
-- **Service Health**: Check the health of services using `docker compose ps`.
-- **Logs**: View logs for a specific service using `docker compose logs -f <service_name>`.
-- **Otel Collector**: Check `http://localhost:13133` for health status and `http://localhost:1888/debug/pprof/` for profiling.

.docker/observability/README_ZH.md

@@ -1,211 +1,27 @@
-# RustFS 可观测性技术栈
+## 部署可观测性系统
 
-本目录包含 RustFS 的全面可观测性技术栈，旨在提供对应用程序性能、日志和追踪的深入洞察。
+OpenTelemetry Collector 提供了一个厂商中立的遥测数据处理方案，用于接收、处理和导出遥测数据。它消除了为支持多种开源可观测性数据格式（如
+Jaeger、Prometheus 等）而需要运行和维护多个代理/收集器的必要性。
 
-## 组件
+### 快速部署
 
-该技术栈由以下一流的开源组件组成：
-
-- **Prometheus** (v2.53.1): 行业标准的指标收集和告警工具。
-- **Grafana** (v11.1.0): 领先的可观测性可视化平台。
-- **Loki** (v3.1.0): 水平可扩展、高可用、多租户的日志聚合系统。
-- **Tempo** (v2.5.0): 高吞吐量、最小依赖的分布式追踪后端。
-- **Jaeger** (v1.59.0): 分布式追踪系统（配置为辅助 UI/存储）。
-- **OpenTelemetry Collector** (v0.104.0): 接收、处理和导出遥测数据的供应商无关实现。
-
-默认情况下，这套技术栈使用 Tempo 单二进制模式，不依赖 Kafka/Redpanda。
-如果需要基于 Kafka 的 HA Tempo 路径，请使用 `docker-compose-example-for-rustfs.yml` 配合 `docker-compose-tempo-ha-override.yml`。
-
-## 架构
-
-1.  **遥测收集**: 应用程序将 OTLP (OpenTelemetry Protocol) 数据（指标、日志、追踪）发送到 **OpenTelemetry Collector**。
-2.  **处理与导出**: Collector 处理数据（批处理、内存限制）并将其导出到相应的后端：
-    -   **追踪** -> **Tempo** (主要) & **Jaeger** (辅助/可选)
-    -   **指标** -> **Prometheus** (通过抓取 Collector 的导出器)
-    -   **日志** -> **Loki**
-3.  **可视化**: **Grafana** 连接到所有后端（Prometheus, Tempo, Loki, Jaeger），提供统一的仪表盘体验。
-
-## 特性
-
--   **完全持久化**: 所有数据（指标、日志、追踪）都持久化到 Docker 卷，确保重启后无数据丢失。
--   **关联性**: 在 Grafana 中实现指标、日志和追踪之间的无缝导航。
-    -   从指标峰值跳转到相关追踪。
-    -   从追踪跳转到相关日志。
--   **高性能**: 针对批处理、压缩和内存管理进行了优化配置。
--   **标准化协议**: 完全基于 OpenTelemetry 标准构建。
-
-## GET 性能优化仪表盘
-
-包含三个预构建的 Grafana 仪表盘，用于监控 RustFS GET 性能优化发布：
-
-### 可用仪表盘
-
-| 仪表盘 | 文件 | 描述 |
-|--------|------|------|
-| **GET 发布健康度** | `grafana-get-rollout-health.json` | 监控优化发布：按 reader path 的延迟、early-stop 命中率、codec streaming 使用率、pipeline 失败率 |
-| **GET 数据完整性** | `grafana-get-data-integrity.json` | 监控数据安全：bitrot 校验失败、decode 错误、short read、shard 读取结果 |
-| **GET 资源影响** | `grafana-get-resource-impact.json` | 监控资源使用：并发请求数、IO 队列利用率、disk permit 等待、RSS 趋势 |
-| **对象数据缓存** | `grafana-object-data-cache.json` | 监控 GET body 缓存（`rustfs_object_data_cache_*`）：命中率、查找/规划/填充结果、填充耗时分位、命中 vs 填充吞吐、条目数/加权字节、在途填充、内存压力拒绝、失效、按尺寸档拆分 |
-
-### Prometheus 告警规则
-
-文件 `prometheus-rules/rustfs-get-optimization-alerts.yaml` 包含预配置的告警规则：
-
-| 告警 | 级别 | 条件 |
-|------|------|------|
-| `GetP99Regression` | 严重 | GET p99 延迟 > 2x 基线，持续 10 分钟 |
-| `PipelineFailureSpike` | 严重 | Pipeline 失败率 > 5x 基线，持续 5 分钟 |
-| `BitrotMismatchSpike` | 严重 | Bitrot 不匹配率 > 3x 基线，持续 5 分钟 |
-| `EarlyStopInsufficientQuorum` | 警告 | Early-stop quorum 不足率 > 0.1/s，持续 5 分钟 |
-| `CodecStreamingFallbackSpike` | 警告 | Codec streaming 回退 > 10x 基线，持续 10 分钟 |
-| `IoQueueSaturation` | 警告 | IO 队列利用率 > 90%，持续 5 分钟 |
-
-文件 `prometheus-rules/rustfs-kms-alerts.yml` 包含 KMS 后端操作指标的告警规则。阈值为保守默认值，待 staging 基线校准；响应流程见 `docs/operations/kms-observability-runbook.md`，配套仪表盘为 `deploy/observability/grafana/rustfs-kms-observability.json`。
-
-| 告警 | 级别 | 条件 |
-|------|------|------|
-| `KmsBackendFatalErrors` | 严重 | fatal（不可重试）尝试失败 > 0，持续 5 分钟 |
-| `KmsBackendHighErrorRate` | 严重 | 非 success 操作占比 > 5%，持续 10 分钟（含流量下限保护） |
-| `KmsBackendP99LatencyHigh` | 警告 | 操作 p99 耗时（含重试）> 2s，持续 10 分钟 |
-| `KmsBackendAttemptFailureSpike` | 警告 | 尝试失败率 > 0.5/s，持续 10 分钟 |
-| `KmsBackendRetryBudgetExhausted` | 警告 | budget_exhausted / deadline_exceeded 结果 > 0.05/s，持续 10 分钟 |
-
-### 启用告警规则
-
-在 Prometheus 配置中添加告警规则文件：
-
-```yaml
-# prometheus.yml
-rule_files:
-  - "/etc/prometheus/rules/*.yml"
-
-# 或在 docker-compose.yml 中挂载文件：
-# volumes:
-#   - ./prometheus-rules:/etc/prometheus/rules
-```
-
-### 仪表盘使用
-
-仪表盘在 Grafana 启动时自动预置。它们使用 `${DS_PROMETHEUS}` 数据源变量，因此需要在 Grafana 中配置 Prometheus 数据源。
-
-优化发布期间需要关注的关键面板：
-
-1. **GET 延迟按 Reader Path** - 对比 `codec_streaming` vs `legacy_duplex` 延迟
-2. **Early-Stop 命中率** - 验证 early-stop 是否有效触发
-3. **Pipeline 失败率** - 检测优化引入的新故障模式
-4. **Bitrot 校验失败** - 确保数据完整性
-
-## 快速开始
-
-### 前置条件
-
--   Docker
--   Docker Compose
-
-### 部署
-
-运行以下命令启动整个技术栈：
+1. 进入 `.docker/observability` 目录
+2. 执行以下命令启动服务：
 
 ```bash
-docker compose up -d
+docker compose -f docker-compose.yml  up -d
 ```
 
-### Tempo 高可用模式
+### 访问监控面板
 
-默认的 `docker-compose.yml` 对应单机栈。
-如果需要基于 Kafka 的 HA Tempo 配置，请使用：
+服务启动后，可通过以下地址访问各个监控面板：
 
-```bash
-docker compose -f docker-compose-example-for-rustfs.yml -f docker-compose-tempo-ha-override.yml up -d
+- Grafana: `http://localhost:3000` (默认账号/密码：`admin`/`admin`)
+- Jaeger: `http://localhost:16686`
+- Prometheus: `http://localhost:9090`
+
+## 配置可观测性
+
+```shell
+export RUSTFS_OBS_ENDPOINT="http://localhost:4317" # OpenTelemetry Collector 地址
 ```
-
-### 访问仪表盘
-
-| 服务 | URL | 凭据 | 描述 |
-| :--- | :--- | :--- | :--- |
-| **Grafana** | [http://localhost:3000](http://localhost:3000) | `admin` / `admin` | 主要可视化中心。 |
-| **Prometheus** | [http://localhost:9090](http://localhost:9090) | - | 指标查询和状态。 |
-| **Jaeger UI** | [http://localhost:16686](http://localhost:16686) | - | 辅助追踪可视化。 |
-| **Tempo** | [http://localhost:3200](http://localhost:3200) | - | Tempo 状态/指标。 |
-
-## 配置
-
-### 数据持久化
-
-数据存储在以下 Docker 卷中：
-
--   `prometheus-data`: Prometheus 指标
--   `tempo-data`: Tempo 追踪 (WAL 和 Blocks)
--   `loki-data`: Loki 日志 (Chunks 和 Rules)
--   `jaeger-data`: Jaeger 追踪 (Badger DB)
-
-要清除所有数据：
-
-```bash
-docker compose down -v
-```
-
-### 自定义
-
--   **Prometheus**: 编辑 `prometheus.yml` 以添加抓取目标或告警规则。
--   **Grafana**: 仪表盘和数据源从 `grafana/` 目录预置。
--   **Collector**: 编辑 `otel-collector-config.yaml` 以修改管道、处理器或导出器。
-
-### 验证 RustFS Trace
-
-当 RustFS 将 `RUSTFS_OBS_ENDPOINT` 指向这套技术栈时，应将该值视为
-OTLP/HTTP 的基础 URL，例如：
-
-```bash
-export RUSTFS_OBS_ENDPOINT=http://host.docker.internal:4318
-```
-
-RustFS 会自动在该基础 URL 后补全：
-
-- `/v1/traces`
-- `/v1/metrics`
-- `/v1/logs`
-
-需要注意：
-
-- 启动阶段通常会先看到日志和指标，因此“先有日志/指标、后有 trace”是正常现象。
-- OpenTelemetry bridge 会把 `tracing` 字段作为日志 attributes 发送。Loki 会将这些 attributes 存为 structured metadata，同时 Collector 会把常用排障字段镜像进日志行，方便用简单的行内容过滤直接查到。
-- 可见的 trace 数据通常依赖启动后的真实 HTTP/S3/gRPC 请求流量，因为请求路径上的 span 是按需创建的。
-- `RUSTFS_OBS_LOGGER_LEVEL=info` 会保留顶层请求 span，但会过滤掉很多 `debug` 级别的嵌套 span。
-  如果 Tempo 或 Jaeger 中的 trace 看起来很稀疏，建议先改成 `RUSTFS_OBS_LOGGER_LEVEL=debug`，再判断是否是 collector 或 Tempo 问题。
-
-最小验证流程：
-
-```bash
-# 1. 启动本目录下的可观测性技术栈。
-docker compose up -d
-
-# 2. 以 OTLP/HTTP 导出方式启动 RustFS，并提高 span 可见性。
-export RUSTFS_OBS_ENDPOINT=http://host.docker.internal:4318
-export RUSTFS_OBS_LOGGER_LEVEL=debug
-
-# 3. 产生真实请求流量。
-curl -I http://127.0.0.1:9000/health
-curl -I http://127.0.0.1:9000/health/ready
-
-# 4. 到 Grafana 或 Jaeger 中检查。
-# Grafana: http://localhost:3000
-# Jaeger:  http://localhost:16686
-```
-
-对于 RustFS 结构化日志，例如节点间 RPC 鉴权失败，Loki 日志行现在会包含 `event`、`component`、`subsystem`、`failure_reason`、`rpc_service`、`rpc_method`、`expected_audience` 等字段。常用 LogQL 检查：
-
-```logql
-{service_name="RustFS"} |= "RPC signature verification failed"
-{service_name="RustFS"} |= "failure_reason="
-{service_name="RustFS"} | failure_reason != ""
-```
-
-如果日志和指标已经正常，但 trace 仍然稀疏，最常见的原因通常是
-“还没有真实请求流量”或“`info` 级别过滤了嵌套 span”，而不是 OTLP 路由失败。
-
-## 故障排除
-
--   **服务健康**: 使用 `docker compose ps` 检查服务健康状况。
--   **日志**: 使用 `docker compose logs -f <service_name>` 查看特定服务的日志。
--   **Otel Collector**: 检查 `http://localhost:13133` 获取健康状态，检查 `http://localhost:1888/debug/pprof/` 进行性能分析。

.docker/observability/docker-compose-example-for-rustfs.yml

@@ -1,268 +0,0 @@
-# Copyright 2024 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-services:
-  rustfs:
-    security_opt:
-      - "no-new-privileges:true"
-    image: rustfs/rustfs:latest
-    container_name: rustfs-server
-    ports:
-      - "9000:9000" # S3 API port
-      - "9001:9001" # Console port
-    environment:
-      - RUSTFS_VOLUMES=/data/rustfs
-      - RUSTFS_ADDRESS=0.0.0.0:9000
-      - RUSTFS_CONSOLE_ADDRESS=0.0.0.0:9001
-      - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_CORS_ALLOWED_ORIGINS=*
-      - RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS=*
-      - RUSTFS_ACCESS_KEY=rustfsadmin
-      - RUSTFS_SECRET_KEY=rustfsadmin
-      - RUSTFS_OBS_LOGGER_LEVEL=info
-      - RUSTFS_OBS_ENDPOINT=http://otel-collector:4318
-      - RUSTFS_OBS_PROFILING_ENDPOINT=http://pyroscope:4040
-    volumes:
-      - rustfs-data:/data/rustfs
-    networks:
-      - otel-network
-    restart: unless-stopped
-    healthcheck:
-      test:
-        [
-          "CMD",
-          "sh",
-          "-c",
-          "curl -f http://127.0.0.1:9000/health && curl -f http://127.0.0.1:9001/rustfs/console/health",
-        ]
-      interval: 30s
-      timeout: 10s
-      retries: 3
-      start_period: 40s
-    depends_on:
-      otel-collector:
-        condition: service_started
-
-  rustfs-init:
-    image: alpine
-    container_name: rustfs-init
-    volumes:
-      - rustfs-data:/data
-    networks:
-      - otel-network
-    command: >
-      sh -c "
-        chown -R 10001:10001 /data &&
-        echo 'Volume Permissions fixed' &&
-        exit 0
-      "
-    restart: no
-
-  # --- Tracing ---
-
-  tempo:
-    image: grafana/tempo:latest
-    container_name: tempo
-    command: [ "-config.file=/etc/tempo.yaml" ]
-    volumes:
-      - ./tempo.yaml:/etc/tempo.yaml:ro
-      - tempo-data:/var/tempo
-    ports:
-      - "3200:3200" # tempo
-      - "4317" # otlp grpc
-      - "4318" # otlp http
-    networks:
-      - otel-network
-    restart: unless-stopped
-    healthcheck:
-      test: [ "CMD", "/tempo", "-version" ]
-      interval: 10s
-      timeout: 5s
-      retries: 3
-      start_period: 15s
-
-  redpanda:
-    image: redpandadata/redpanda:latest # for tempo ingest
-    container_name: redpanda
-    ports:
-      - "9092:9092"
-    networks:
-      - otel-network
-    restart: unless-stopped
-    command: >
-      redpanda start --overprovisioned
-      --mode=dev-container
-      --kafka-addr=PLAINTEXT://0.0.0.0:9092
-      --advertise-kafka-addr=PLAINTEXT://redpanda:9092
-
-  jaeger:
-    image: jaegertracing/jaeger:latest
-    container_name: jaeger
-    environment:
-      - SPAN_STORAGE_TYPE=badger
-      - BADGER_EPHEMERAL=false
-      - BADGER_DIRECTORY_VALUE=/badger/data
-      - BADGER_DIRECTORY_KEY=/badger/key
-      - COLLECTOR_OTLP_ENABLED=true
-    volumes:
-      - ./jaeger.yaml:/etc/jaeger/config.yml
-      - jaeger-data:/badger
-    ports:
-      - "16686:16686" # Web UI
-      - "14269:14269" # Admin/Metrics
-      - "4317" # otlp grpc
-      - "4318" # otlp http
-    command: [ "--config", "/etc/jaeger/config.yml" ]
-    networks:
-      - otel-network
-    restart: unless-stopped
-    healthcheck:
-      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:14269" ]
-      interval: 10s
-      timeout: 5s
-      retries: 3
-      start_period: 15s
-
-  # --- Metrics ---
-
-  prometheus:
-    image: prom/prometheus:latest
-    container_name: prometheus
-    volumes:
-      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
-      - prometheus-data:/prometheus
-    ports:
-      - "9090:9090"
-    command:
-      - "--config.file=/etc/prometheus/prometheus.yml"
-      - "--web.enable-otlp-receiver" # Enable OTLP
-      - "--web.enable-remote-write-receiver" # Enable remote write
-      - "--enable-feature=promql-experimental-functions" # Enable info()
-      - "--storage.tsdb.retention.time=30d"
-    restart: unless-stopped
-    networks:
-      - otel-network
-    healthcheck:
-      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:9090/-/healthy" ]
-      interval: 10s
-      timeout: 5s
-      retries: 3
-
-  # --- Logging ---
-
-  loki:
-    image: grafana/loki:latest
-    container_name: loki
-    volumes:
-      - ./loki.yaml:/etc/loki/loki.yaml:ro
-      - loki-data:/loki
-    ports:
-      - "3100:3100"
-    command: -config.file=/etc/loki/loki.yaml
-    networks:
-      - otel-network
-    restart: unless-stopped
-    healthcheck:
-      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:3100/ready" ]
-      interval: 15s
-      timeout: 10s
-      retries: 5
-      start_period: 60s
-
-  # --- Collection ---
-
-  otel-collector:
-    image: otel/opentelemetry-collector-contrib:latest
-    volumes:
-      - ./otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml:ro
-    ports:
-      - "1888:1888" # pprof
-      - "8888:8888" # Prometheus metrics for Collector
-      - "8889:8889" # Prometheus metrics for application indicators
-      - "13133:13133" # health check
-      - "4317:4317" # OTLP gRPC
-      - "4318:4318" # OTLP HTTP
-      - "55679:55679" # zpages
-    networks:
-      - otel-network
-    restart: unless-stopped
-    depends_on:
-      - tempo
-      - jaeger
-      - prometheus
-      - loki
-    healthcheck:
-      test: [ "CMD", "/otelcol-contrib", "--version" ]
-      interval: 10s
-      timeout: 5s
-      retries: 3
-
-  # --- Profiles ---
-
-  pyroscope:
-    image: grafana/pyroscope:latest
-    container_name: pyroscope
-    ports:
-      - "4040:4040"
-    command:
-      - -self-profiling.disable-push=true
-    networks:
-      - otel-network
-    restart: unless-stopped
-
-  # --- Visualization ---
-
-  grafana:
-    image: grafana/grafana:latest
-    container_name: grafana
-    ports:
-      - "3000:3000"
-    environment:
-      - GF_SECURITY_ADMIN_PASSWORD=admin
-      - GF_SECURITY_ADMIN_USER=admin
-    volumes:
-      - ./grafana/provisioning:/etc/grafana/provisioning:ro
-      - ./grafana/dashboards:/etc/grafana/dashboards:ro
-      - grafana-data:/var/lib/grafana
-    networks:
-      - otel-network
-    restart: unless-stopped
-    depends_on:
-      - prometheus
-      - tempo
-      - loki
-    healthcheck:
-      test:
-        [ "CMD", "wget", "--spider", "-q", "http://localhost:3000/api/health" ]
-      interval: 10s
-      timeout: 5s
-      retries: 3
-
-volumes:
-  rustfs-data:
-  tempo-data:
-  jaeger-data:
-  prometheus-data:
-  loki-data:
-  grafana-data:
-
-networks:
-  otel-network:
-    driver: bridge
-    name: "network_otel"
-    ipam:
-      config:
-        - subnet: 172.28.0.0/16
-    driver_opts:
-      com.docker.network.enable_ipv6: "true"

.docker/observability/docker-compose-tempo-ha-override.yml

@@ -1,62 +0,0 @@
-# Copyright 2024 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-# Docker Compose override file for High Availability Tempo setup
-#
-# Usage:
-# docker-compose -f docker-compose-example-for-rustfs.yml \
-#                -f docker-compose-tempo-ha-override.yml up
-
-services:
-  # Override Tempo to use high-availability configuration
-  tempo:
-    volumes:
-      - ./tempo-ha.yaml:/etc/tempo.yaml:ro
-      - tempo-data:/var/tempo
-    ports:
-      - "3200:3200"   # Tempo HTTP
-      - "4317:4317"   # OTLP gRPC
-      - "4318:4318"   # OTLP HTTP
-      - "7946:7946"   # Memberlist
-      - "14250:14250" # Jaeger gRPC
-      - "14268:14268" # Jaeger Thrift HTTP
-      - "9411:9411"   # Zipkin
-    environment:
-      - TEMPO_MEMBERLIST_BIND_PORT=7946
-    healthcheck:
-      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:3200/ready" ]
-      interval: 10s
-      timeout: 5s
-      retries: 5
-      start_period: 30s
-    depends_on:
-      - redpanda
-
-volumes:
-  tempo-data:
-    driver: local
-    driver_opts:
-      type: tmpfs
-      device: tmpfs
-      o: "size=4g"  # Allocate 4GB tmpfs for Tempo data (adjust based on your needs)
-
-# Network configuration remains the same
-# networks:
-#   otel-network:
-#     driver: bridge
-#     name: "network_otel"
-#     ipam:
-#       config:
-#         - subnet: 172.28.0.0/16
-

.docker/observability/docker-compose.yml

@@ -14,198 +14,93 @@
 
 services:
 
-  # --- Tracing ---
+  tempo-init:
+    image: busybox:latest
+    command: ["sh", "-c", "chown -R 10001:10001 /var/tempo"]
+    volumes:
+      - ./tempo-data:/var/tempo
+    user: root
+    networks:
+      - otel-network
+    restart: "no"
+
   tempo:
-    image: grafana/tempo:2.10.5
-    container_name: tempo
-    command: [ "-config.file=/etc/tempo.yaml" ]
+    image: grafana/tempo:latest
+    user: "10001" # The container must be started with root to execute chown in the script
+    command: [ "-config.file=/etc/tempo.yaml" ] # This is passed as a parameter to the entry point script
     volumes:
       - ./tempo.yaml:/etc/tempo.yaml:ro
-      - tempo-data:/var/tempo
+      - ./tempo-data:/var/tempo
     ports:
       - "3200:3200" # tempo
-      - "4317" # otlp grpc
-      - "4318" # otlp http
-      - "7946" # memberlist
-    networks:
-      - otel-network
-    restart: unless-stopped
-    healthcheck:
-      test: [ "CMD", "/tempo", "-version" ]
-      interval: 10s
-      timeout: 5s
-      retries: 3
-      start_period: 15s
-
-  vulture:
-    image: grafana/tempo-vulture:latest
-    restart: always
-    command:
-      [
-        "-prometheus-listen-address=:8080",
-        "-tempo-query-url=http://tempo:3200",
-        "-tempo-push-url=http://tempo:4317",
-      ]
-    depends_on:
-      - tempo
-
-  jaeger:
-    image: jaegertracing/jaeger:latest
-    container_name: jaeger
-    environment:
-      - SPAN_STORAGE_TYPE=badger
-      - BADGER_EPHEMERAL=false
-      - BADGER_DIRECTORY_VALUE=/badger/data
-      - BADGER_DIRECTORY_KEY=/badger/key
-      - COLLECTOR_OTLP_ENABLED=true
-    volumes:
-      - ./jaeger.yaml:/etc/jaeger/config.yml
-      - jaeger-data:/badger
-    ports:
-      - "16686:16686" # Web UI
-      - "18888:8888" # Metrics
-      - "4317" # otlp grpc
-      - "4318" # otlp http
-    command: [ "--config", "/etc/jaeger/config.yml" ]
-    networks:
-      - otel-network
-    restart: unless-stopped
-    healthcheck:
-      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:8888/metrics" ]
-      interval: 10s
-      timeout: 5s
-      retries: 3
-      start_period: 15s
-
-  # --- Metrics ---
-
-  prometheus:
-    image: prom/prometheus:latest
-    container_name: prometheus
-    volumes:
-      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
-      - ./prometheus-rules:/etc/prometheus/rules:ro
-      - prometheus-data:/prometheus
-    ports:
-      - "9090:9090"
-    command:
-      - "--config.file=/etc/prometheus/prometheus.yml"
-      - "--web.enable-otlp-receiver" # Enable OTLP
-      - "--web.enable-remote-write-receiver" # Enable remote write
-      - "--enable-feature=promql-experimental-functions" # Enable info()
-      - "--storage.tsdb.retention.time=30d"
+      - "24317:4317" # otlp grpc
     restart: unless-stopped
     networks:
       - otel-network
-    healthcheck:
-      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:9090/-/healthy" ]
-      interval: 10s
-      timeout: 5s
-      retries: 3
-
-  # --- Logging ---
-
-  loki:
-    image: grafana/loki:latest
-    container_name: loki
-    volumes:
-      - ./loki.yaml:/etc/loki/loki.yaml:ro
-      - loki-data:/loki
-    ports:
-      - "3100:3100"
-    command: -config.file=/etc/loki/loki.yaml
-    networks:
-      - otel-network
-    restart: unless-stopped
-    healthcheck:
-      test: [ "CMD", "/usr/bin/loki", "--version" ]
-      interval: 15s
-      timeout: 10s
-      retries: 5
-      start_period: 60s
-
-  # --- Collection ---
 
   otel-collector:
-    image: otel/opentelemetry-collector-contrib:latest
+    image: otel/opentelemetry-collector-contrib:0.129.1
+    environment:
+      - TZ=Asia/Shanghai
     volumes:
-      - ./otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml:ro
+      - ./otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml
     ports:
-      - "1888:1888" # pprof
-      - "8888:8888" # Prometheus metrics for Collector
-      - "8889:8889" # Prometheus metrics for application indicators
-      - "13133:13133" # health check
-      - "4317:4317" # OTLP gRPC
-      - "4318:4318" # OTLP HTTP
-      - "55679:55679" # zpages
+      - "1888:1888"
+      - "8888:8888"
+      - "8889:8889"
+      - "13133:13133"
+      - "4317:4317"
+      - "4318:4318"
+      - "55679:55679"
     networks:
       - otel-network
-    restart: unless-stopped
-    depends_on:
-      - tempo
-      - jaeger
-      - prometheus
-      - loki
-    healthcheck:
-      test: [ "CMD", "/otelcol-contrib", "--version" ]
-      interval: 10s
-      timeout: 5s
-      retries: 3
-
-  # --- Profiles ---
-
-  pyroscope:
-    image: grafana/pyroscope:latest
-    container_name: pyroscope
+  jaeger:
+    image: jaegertracing/jaeger:2.8.0
+    environment:
+      - TZ=Asia/Shanghai
     ports:
-      - "4040:4040"
-    command:
-      - -self-profiling.disable-push=true
+      - "16686:16686"
+      - "14317:4317"
+      - "14318:4318"
+    networks:
+      - otel-network
+  prometheus:
+    image: prom/prometheus:v3.4.2
+    environment:
+      - TZ=Asia/Shanghai
+    volumes:
+      - ./prometheus.yml:/etc/prometheus/prometheus.yml
+    ports:
+      - "9090:9090"
+    networks:
+      - otel-network
+  loki:
+    image: grafana/loki:3.5.1
+    environment:
+      - TZ=Asia/Shanghai
+    volumes:
+      - ./loki-config.yaml:/etc/loki/local-config.yaml
+    ports:
+      - "3100:3100"
+    command: -config.file=/etc/loki/local-config.yaml
     networks:
       - otel-network
-    restart: unless-stopped
-
-  # --- Visualization ---
-
   grafana:
-    image: grafana/grafana:latest
-    container_name: grafana
+    image: grafana/grafana:12.0.2
     ports:
-      - "3000:3000"
+      - "3000:3000"  # Web UI
+    volumes:
+      - ./grafana-datasources.yaml:/etc/grafana/provisioning/datasources/datasources.yaml
     environment:
       - GF_SECURITY_ADMIN_PASSWORD=admin
-      - GF_SECURITY_ADMIN_USER=admin
-    volumes:
-      - ./grafana/provisioning:/etc/grafana/provisioning:ro
-      - ./grafana/dashboards:/etc/grafana/dashboards:ro
-      - grafana-data:/var/lib/grafana
+      - TZ=Asia/Shanghai
     networks:
       - otel-network
-    restart: unless-stopped
-    depends_on:
-      - prometheus
-      - tempo
-      - loki
-    healthcheck:
-      test:
-        [ "CMD", "wget", "--spider", "-q", "http://localhost:3000/api/health" ]
-      interval: 10s
-      timeout: 5s
-      retries: 3
 
-volumes:
-  tempo-data:
-  jaeger-data:
-  prometheus-data:
-  loki-data:
-  grafana-data:
 
 networks:
   otel-network:
     driver: bridge
-    name: "network_otel"
-    ipam:
-      config:
-        - subnet: 172.28.0.0/16
+    name: "network_otel_config"
     driver_opts:
       com.docker.network.enable_ipv6: "true"

.docker/observability/grafana-datasources.yaml

@@ -0,0 +1,32 @@
+apiVersion: 1
+
+datasources:
+  - name: Prometheus
+    type: prometheus
+    uid: prometheus
+    access: proxy
+    orgId: 1
+    url: http://prometheus:9090
+    basicAuth: false
+    isDefault: false
+    version: 1
+    editable: false
+    jsonData:
+      httpMethod: GET
+  - name: Tempo
+    type: tempo
+    access: proxy
+    orgId: 1
+    url: http://tempo:3200
+    basicAuth: false
+    isDefault: true
+    version: 1
+    editable: false
+    apiVersion: 1
+    uid: tempo
+    jsonData:
+      httpMethod: GET
+      serviceMap:
+        datasourceUid: prometheus
+      streamingEnabled:
+        search: true

.docker/observability/grafana/dashboards/grafana-get-data-integrity.json

@@ -1,500 +0,0 @@
-{
-  "annotations": {
-    "list": []
-  },
-  "editable": true,
-  "fiscalYearStartMonth": 0,
-  "graphTooltip": 1,
-  "id": null,
-  "links": [],
-  "liveNow": false,
-  "panels": [
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "ops",
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 20,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "thresholdsStyle": {
-              "mode": "line+area"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "transparent",
-                "value": null
-              },
-              {
-                "color": "red",
-                "value": 1
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 0,
-        "y": 0
-      },
-      "id": 1,
-      "options": {
-        "reduceOptions": {
-          "values": false,
-          "calcs": ["lastNotNull"],
-          "fields": ""
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        },
-        "legend": {
-          "displayMode": "table",
-          "placement": "bottom",
-          "calcs": ["mean", "max", "sum"]
-        }
-      },
-      "title": "Bitrot Verify Failures",
-      "type": "timeseries",
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(rate(rustfs_io_get_object_pipeline_failures_total{stage=\"bitrot_verify\"}[$__rate_interval])) by (path, reason)",
-          "legendFormat": "bitrot_fail {{path}} / {{reason}}",
-          "refId": "A"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(rate(rustfs_io_get_object_pipeline_failures_total{reason=\"bitrot_verify_failed\"}[$__rate_interval])) by (path, stage)",
-          "legendFormat": "verify_failed {{path}} / {{stage}}",
-          "refId": "B"
-        }
-      ]
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "s",
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 10,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 12,
-        "y": 0
-      },
-      "id": 2,
-      "options": {
-        "reduceOptions": {
-          "values": false,
-          "calcs": ["lastNotNull"],
-          "fields": ""
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        },
-        "legend": {
-          "displayMode": "table",
-          "placement": "bottom",
-          "calcs": ["mean", "max"]
-        }
-      },
-      "title": "Bitrot Verify Duration (p50 / p95 / p99)",
-      "type": "timeseries",
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "histogram_quantile(0.50, sum(rate(rustfs_io_get_object_shard_bitrot_verify_duration_seconds_bucket[$__rate_interval])) by (le))",
-          "legendFormat": "p50",
-          "refId": "A"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "histogram_quantile(0.95, sum(rate(rustfs_io_get_object_shard_bitrot_verify_duration_seconds_bucket[$__rate_interval])) by (le))",
-          "legendFormat": "p95",
-          "refId": "B"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "histogram_quantile(0.99, sum(rate(rustfs_io_get_object_shard_bitrot_verify_duration_seconds_bucket[$__rate_interval])) by (le))",
-          "legendFormat": "p99",
-          "refId": "C"
-        }
-      ]
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "ops",
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 20,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "thresholdsStyle": {
-              "mode": "line+area"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "transparent",
-                "value": null
-              },
-              {
-                "color": "red",
-                "value": 1
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 0,
-        "y": 8
-      },
-      "id": 3,
-      "options": {
-        "reduceOptions": {
-          "values": false,
-          "calcs": ["lastNotNull"],
-          "fields": ""
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        },
-        "legend": {
-          "displayMode": "table",
-          "placement": "bottom",
-          "calcs": ["mean", "max", "sum"]
-        }
-      },
-      "title": "Decode Errors",
-      "type": "timeseries",
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(rate(rustfs_io_get_object_pipeline_failures_total{reason=\"decode_error\"}[$__rate_interval])) by (path, stage)",
-          "legendFormat": "decode_error {{path}} / {{stage}}",
-          "refId": "A"
-        }
-      ]
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "ops",
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 20,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "thresholdsStyle": {
-              "mode": "line+area"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "transparent",
-                "value": null
-              },
-              {
-                "color": "red",
-                "value": 1
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 12,
-        "y": 8
-      },
-      "id": 4,
-      "options": {
-        "reduceOptions": {
-          "values": false,
-          "calcs": ["lastNotNull"],
-          "fields": ""
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        },
-        "legend": {
-          "displayMode": "table",
-          "placement": "bottom",
-          "calcs": ["mean", "max", "sum"]
-        }
-      },
-      "title": "Short Reads",
-      "type": "timeseries",
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(rate(rustfs_io_get_object_pipeline_failures_total{reason=\"short_read\"}[$__rate_interval])) by (path, stage)",
-          "legendFormat": "short_read {{path}} / {{stage}}",
-          "refId": "A"
-        }
-      ]
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "ops",
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 10,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 0,
-        "y": 16
-      },
-      "id": 5,
-      "options": {
-        "reduceOptions": {
-          "values": false,
-          "calcs": ["lastNotNull"],
-          "fields": ""
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        },
-        "legend": {
-          "displayMode": "table",
-          "placement": "bottom",
-          "calcs": ["mean", "max"]
-        }
-      },
-      "title": "Shard Read Outcomes by Role",
-      "type": "timeseries",
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(rate(rustfs_io_get_object_shard_read_total[$__rate_interval])) by (path, role, outcome)",
-          "legendFormat": "{{path}} {{role}}/{{outcome}}",
-          "refId": "A"
-        }
-      ]
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "ops",
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 10,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 12,
-        "y": 16
-      },
-      "id": 6,
-      "options": {
-        "reduceOptions": {
-          "values": false,
-          "calcs": ["lastNotNull"],
-          "fields": ""
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        },
-        "legend": {
-          "displayMode": "table",
-          "placement": "bottom",
-          "calcs": ["mean", "max"]
-        }
-      },
-      "title": "Request Result Status Rate",
-      "type": "timeseries",
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(rate(rustfs_io_get_object_request_results_total[$__rate_interval])) by (status)",
-          "legendFormat": "{{status}}",
-          "refId": "A"
-        }
-      ]
-    }
-  ],
-  "refresh": "30s",
-  "schemaVersion": 38,
-  "style": "dark",
-  "tags": ["rustfs", "get-optimization", "data-integrity"],
-  "templating": {
-    "list": [
-      {
-        "current": {
-          "selected": false,
-          "text": "Prometheus",
-          "value": "Prometheus"
-        },
-        "hide": 0,
-        "includeAll": false,
-        "multi": false,
-        "name": "DS_PROMETHEUS",
-        "options": [],
-        "query": "prometheus",
-        "refresh": 1,
-        "regex": "",
-        "skipUrlSync": false,
-        "type": "datasource"
-      }
-    ]
-  },
-  "time": {
-    "from": "now-1h",
-    "to": "now"
-  },
-  "timepicker": {},
-  "timezone": "",
-  "title": "RustFS GET Data Integrity",
-  "uid": "rustfs-get-data-integrity",
-  "version": 1
-}

.docker/observability/grafana/dashboards/grafana-get-performance-attribution.json

@@ -1,716 +0,0 @@
-{
-  "annotations": {
-    "list": []
-  },
-  "editable": true,
-  "fiscalYearStartMonth": 0,
-  "graphTooltip": 1,
-  "id": null,
-  "links": [],
-  "liveNow": false,
-  "panels": [
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "ops",
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 10,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 0,
-        "y": 0
-      },
-      "id": 1,
-      "options": {
-        "legend": {
-          "calcs": [
-            "mean",
-            "max"
-          ],
-          "displayMode": "table",
-          "placement": "bottom"
-        },
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ],
-          "fields": "",
-          "values": false
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum by (strategy, mode) (rate(rustfs_io_get_object_reader_setup_strategy_total[$__rate_interval])) or vector(0)",
-          "legendFormat": "{{strategy}} / {{mode}}",
-          "refId": "A"
-        }
-      ],
-      "title": "GET Reader Setup Strategy Rate",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "ops",
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 10,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 12,
-        "y": 0
-      },
-      "id": 2,
-      "options": {
-        "legend": {
-          "calcs": [
-            "mean",
-            "max"
-          ],
-          "displayMode": "table",
-          "placement": "bottom"
-        },
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ],
-          "fields": "",
-          "values": false
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum by (path, strategy, size_bucket) (rate(rustfs_io_get_object_reader_setup_strategy_by_size_total[$__rate_interval])) or vector(0)",
-          "legendFormat": "{{path}} / {{strategy}} / {{size_bucket}}",
-          "refId": "A"
-        }
-      ],
-      "title": "GET Reader Setup Strategy by Size",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "none",
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 10,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 0,
-        "y": 8
-      },
-      "id": 3,
-      "options": {
-        "legend": {
-          "calcs": [
-            "mean",
-            "max"
-          ],
-          "displayMode": "table",
-          "placement": "bottom"
-        },
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ],
-          "fields": "",
-          "values": false
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum by (strategy, mode) (rate(rustfs_io_get_object_reader_setup_scheduled_sum[$__rate_interval])) / clamp_min(sum by (strategy, mode) (rate(rustfs_io_get_object_reader_setup_scheduled_count[$__rate_interval])), 1) or vector(0)",
-          "legendFormat": "scheduled {{strategy}} / {{mode}}",
-          "refId": "A"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum by (strategy, mode) (rate(rustfs_io_get_object_reader_setup_attempted_sum[$__rate_interval])) / clamp_min(sum by (strategy, mode) (rate(rustfs_io_get_object_reader_setup_attempted_count[$__rate_interval])), 1) or vector(0)",
-          "legendFormat": "attempted {{strategy}} / {{mode}}",
-          "refId": "B"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum by (strategy, mode) (rate(rustfs_io_get_object_reader_setup_ready_sum[$__rate_interval])) / clamp_min(sum by (strategy, mode) (rate(rustfs_io_get_object_reader_setup_ready_count[$__rate_interval])), 1) or vector(0)",
-          "legendFormat": "ready {{strategy}} / {{mode}}",
-          "refId": "C"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum by (strategy, mode) (rate(rustfs_io_get_object_reader_setup_deferred_sum[$__rate_interval])) / clamp_min(sum by (strategy, mode) (rate(rustfs_io_get_object_reader_setup_deferred_count[$__rate_interval])), 1) or vector(0)",
-          "legendFormat": "deferred {{strategy}} / {{mode}}",
-          "refId": "D"
-        }
-      ],
-      "title": "GET Reader Setup Fanout Average",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "none",
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 10,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 12,
-        "y": 8
-      },
-      "id": 4,
-      "options": {
-        "legend": {
-          "calcs": [
-            "mean",
-            "max"
-          ],
-          "displayMode": "table",
-          "placement": "bottom"
-        },
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ],
-          "fields": "",
-          "values": false
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum by (path, strategy, size_bucket) (rate(rustfs_io_get_object_reader_setup_scheduled_by_size_sum[$__rate_interval])) / clamp_min(sum by (path, strategy, size_bucket) (rate(rustfs_io_get_object_reader_setup_scheduled_by_size_count[$__rate_interval])), 1) or vector(0)",
-          "legendFormat": "scheduled {{path}} / {{strategy}} / {{size_bucket}}",
-          "refId": "A"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum by (path, strategy, size_bucket) (rate(rustfs_io_get_object_reader_setup_ready_by_size_sum[$__rate_interval])) / clamp_min(sum by (path, strategy, size_bucket) (rate(rustfs_io_get_object_reader_setup_ready_by_size_count[$__rate_interval])), 1) or vector(0)",
-          "legendFormat": "ready {{path}} / {{strategy}} / {{size_bucket}}",
-          "refId": "B"
-        }
-      ],
-      "title": "GET Reader Setup Fanout by Size",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "s",
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 10,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 0,
-        "y": 16
-      },
-      "id": 5,
-      "options": {
-        "legend": {
-          "calcs": [
-            "mean",
-            "max"
-          ],
-          "displayMode": "table",
-          "placement": "bottom"
-        },
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ],
-          "fields": "",
-          "values": false
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "histogram_quantile(0.95, sum by (path, stage, size_bucket, le) (rate(rustfs_io_get_object_stage_duration_seconds_by_size_bucket{stage=~\"reader_setup|reader_setup_schedule|reader_setup_wait_quorum|reader_setup_drop_pending|reader_task_file_open|reader_task_reader_construction|reader_task_bitrot_reader_init\"}[$__rate_interval]))) or vector(0)",
-          "legendFormat": "p95 {{path}} / {{stage}} / {{size_bucket}}",
-          "refId": "A"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "histogram_quantile(0.99, sum by (path, stage, size_bucket, le) (rate(rustfs_io_get_object_stage_duration_seconds_by_size_bucket{stage=~\"reader_setup|reader_setup_schedule|reader_setup_wait_quorum|reader_setup_drop_pending|reader_task_file_open|reader_task_reader_construction|reader_task_bitrot_reader_init\"}[$__rate_interval]))) or vector(0)",
-          "legendFormat": "p99 {{path}} / {{stage}} / {{size_bucket}}",
-          "refId": "B"
-        }
-      ],
-      "title": "GET Hot Stage Latency by Size",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "s",
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 10,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 12,
-        "y": 16
-      },
-      "id": 6,
-      "options": {
-        "legend": {
-          "calcs": [
-            "mean",
-            "max"
-          ],
-          "displayMode": "table",
-          "placement": "bottom"
-        },
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ],
-          "fields": "",
-          "values": false
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "histogram_quantile(0.95, sum by (path, stage, le) (rate(rustfs_io_get_object_stage_duration_seconds_bucket{stage=~\"request_context|first_byte|response_handoff|full_body|body_build|output_poll|output_lock_wait|reader_open_mmap_copy_success|reader_open_mmap_copy_fallback|reader_open_stream|reader_stream_first_read|reader_mmap_blocking_wait|reader_mmap_blocking_task|reader_mmap_file_open|reader_mmap_map|reader_mmap_copy_buffer|reader_mmap_direct_read_copy\"}[$__rate_interval]))) or vector(0)",
-          "legendFormat": "p95 {{path}} / {{stage}}",
-          "refId": "A"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "histogram_quantile(0.99, sum by (path, stage, le) (rate(rustfs_io_get_object_stage_duration_seconds_bucket{stage=~\"request_context|first_byte|response_handoff|full_body|body_build|output_poll|output_lock_wait|reader_open_mmap_copy_success|reader_open_mmap_copy_fallback|reader_open_stream|reader_stream_first_read|reader_mmap_blocking_wait|reader_mmap_blocking_task|reader_mmap_file_open|reader_mmap_map|reader_mmap_copy_buffer|reader_mmap_direct_read_copy\"}[$__rate_interval]))) or vector(0)",
-          "legendFormat": "p99 {{path}} / {{stage}}",
-          "refId": "B"
-        }
-      ],
-      "title": "GET Reader, Handler and Body Latency",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "ops",
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 10,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 0,
-        "y": 24
-      },
-      "id": 7,
-      "options": {
-        "legend": {
-          "calcs": [
-            "mean",
-            "max"
-          ],
-          "displayMode": "table",
-          "placement": "bottom"
-        },
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ],
-          "fields": "",
-          "values": false
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum by (subpath) (rate(rustfs_io_get_object_direct_memory_subpath_total[$__rate_interval])) or vector(0)",
-          "legendFormat": "{{subpath}}",
-          "refId": "A"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum by (subpath, size_bucket) (rate(rustfs_io_get_object_direct_memory_subpath_by_size_total[$__rate_interval])) or vector(0)",
-          "legendFormat": "{{subpath}} / {{size_bucket}}",
-          "refId": "B"
-        }
-      ],
-      "title": "GET Direct-Memory Subpath Rate",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "ops",
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 10,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 12,
-        "y": 24
-      },
-      "id": 8,
-      "options": {
-        "legend": {
-          "calcs": [
-            "mean",
-            "max"
-          ],
-          "displayMode": "table",
-          "placement": "bottom"
-        },
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ],
-          "fields": "",
-          "values": false
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum by (path, stage, kind) (rate(rustfs_io_get_object_mmap_page_faults_total[$__rate_interval])) or vector(0)",
-          "legendFormat": "mmap {{path}} / {{stage}} / {{kind}}",
-          "refId": "A"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum by (path, stage, kind) (rate(rustfs_io_get_object_direct_read_page_faults_total[$__rate_interval])) or vector(0)",
-          "legendFormat": "direct-read {{path}} / {{stage}} / {{kind}}",
-          "refId": "B"
-        }
-      ],
-      "title": "GET mmap vs Direct-Read Page Fault Rate",
-      "type": "timeseries"
-    }
-  ],
-  "refresh": "30s",
-  "schemaVersion": 38,
-  "style": "dark",
-  "tags": [
-    "rustfs",
-    "get",
-    "performance",
-    "attribution"
-  ],
-  "templating": {
-    "list": [
-      {
-        "current": {
-          "selected": false,
-          "text": "Prometheus",
-          "value": "Prometheus"
-        },
-        "hide": 0,
-        "includeAll": false,
-        "multi": false,
-        "name": "DS_PROMETHEUS",
-        "options": [],
-        "query": "prometheus",
-        "refresh": 1,
-        "regex": "",
-        "skipUrlSync": false,
-        "type": "datasource"
-      }
-    ]
-  },
-  "time": {
-    "from": "now-1h",
-    "to": "now"
-  },
-  "timezone": "",
-  "title": "RustFS GET Performance Attribution",
-  "uid": "rustfs-get-performance-attribution",
-  "version": 1,
-  "weekStart": ""
-}

.docker/observability/grafana/dashboards/grafana-get-resource-impact.json

@@ -1,525 +0,0 @@
-{
-  "annotations": {
-    "list": []
-  },
-  "editable": true,
-  "fiscalYearStartMonth": 0,
-  "graphTooltip": 1,
-  "id": null,
-  "links": [],
-  "liveNow": false,
-  "panels": [
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "none",
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 20,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 0,
-        "y": 0
-      },
-      "id": 1,
-      "options": {
-        "reduceOptions": {
-          "values": false,
-          "calcs": ["lastNotNull"],
-          "fields": ""
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        },
-        "legend": {
-          "displayMode": "table",
-          "placement": "bottom",
-          "calcs": ["mean", "max"]
-        }
-      },
-      "title": "Concurrent GET Requests",
-      "type": "timeseries",
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "rustfs_io_get_object_concurrent_requests",
-          "legendFormat": "concurrent GETs",
-          "refId": "A"
-        }
-      ]
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "percent",
-          "min": 0,
-          "max": 100,
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 20,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "thresholdsStyle": {
-              "mode": "line+area"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "transparent",
-                "value": null
-              },
-              {
-                "color": "yellow",
-                "value": 70
-              },
-              {
-                "color": "red",
-                "value": 90
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 12,
-        "y": 0
-      },
-      "id": 2,
-      "options": {
-        "reduceOptions": {
-          "values": false,
-          "calcs": ["lastNotNull"],
-          "fields": ""
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        },
-        "legend": {
-          "displayMode": "table",
-          "placement": "bottom",
-          "calcs": ["mean", "max"]
-        }
-      },
-      "title": "IO Queue Utilization",
-      "type": "timeseries",
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "rustfs_io_queue_utilization_percent",
-          "legendFormat": "queue utilization %",
-          "refId": "A"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "rustfs_io_queue_permits_in_use",
-          "legendFormat": "permits in use",
-          "refId": "B"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "rustfs_io_queue_permits_available",
-          "legendFormat": "permits available",
-          "refId": "C"
-        }
-      ]
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "s",
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 10,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 0,
-        "y": 8
-      },
-      "id": 3,
-      "options": {
-        "reduceOptions": {
-          "values": false,
-          "calcs": ["lastNotNull"],
-          "fields": ""
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        },
-        "legend": {
-          "displayMode": "table",
-          "placement": "bottom",
-          "calcs": ["mean", "max"]
-        }
-      },
-      "title": "Disk Permit Wait Duration (p50 / p95 / p99)",
-      "type": "timeseries",
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "histogram_quantile(0.50, sum(rate(rustfs_io_disk_permit_wait_duration_seconds_bucket[$__rate_interval])) by (le))",
-          "legendFormat": "p50",
-          "refId": "A"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "histogram_quantile(0.95, sum(rate(rustfs_io_disk_permit_wait_duration_seconds_bucket[$__rate_interval])) by (le))",
-          "legendFormat": "p95",
-          "refId": "B"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "histogram_quantile(0.99, sum(rate(rustfs_io_disk_permit_wait_duration_seconds_bucket[$__rate_interval])) by (le))",
-          "legendFormat": "p99",
-          "refId": "C"
-        }
-      ]
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "bytes",
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 15,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 12,
-        "y": 8
-      },
-      "id": 4,
-      "options": {
-        "reduceOptions": {
-          "values": false,
-          "calcs": ["lastNotNull"],
-          "fields": ""
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        },
-        "legend": {
-          "displayMode": "table",
-          "placement": "bottom",
-          "calcs": ["lastNotNull"]
-        }
-      },
-      "title": "RSS Trend (Process Memory)",
-      "type": "timeseries",
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "process_resident_memory_bytes{job=~\"rustfs.*\"}",
-          "legendFormat": "RSS {{instance}}",
-          "refId": "A"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "rustfs_process_memory_bytes",
-          "legendFormat": "RustFS memory {{instance}}",
-          "refId": "B"
-        }
-      ]
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "ops",
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 10,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 0,
-        "y": 16
-      },
-      "id": 5,
-      "options": {
-        "reduceOptions": {
-          "values": false,
-          "calcs": ["lastNotNull"],
-          "fields": ""
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        },
-        "legend": {
-          "displayMode": "table",
-          "placement": "bottom",
-          "calcs": ["mean", "max"]
-        }
-      },
-      "title": "IO Queue Operations by Priority",
-      "type": "timeseries",
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(rate(rustfs_io_queue_operations[$__rate_interval])) by (operation, priority)",
-          "legendFormat": "{{operation}} / {{priority}}",
-          "refId": "A"
-        }
-      ]
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "ops",
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 15,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 12,
-        "y": 16
-      },
-      "id": 6,
-      "options": {
-        "reduceOptions": {
-          "values": false,
-          "calcs": ["lastNotNull"],
-          "fields": ""
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        },
-        "legend": {
-          "displayMode": "table",
-          "placement": "bottom",
-          "calcs": ["sum"]
-        }
-      },
-      "title": "IO Queue Congestion Events",
-      "type": "timeseries",
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(rate(rustfs_io_queue_congestion_total[$__rate_interval]))",
-          "legendFormat": "congestion events/s",
-          "refId": "A"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(rate(rustfs_io_get_object_timeout_total[$__rate_interval])) by (stage)",
-          "legendFormat": "timeout {{stage}}",
-          "refId": "B"
-        }
-      ]
-    }
-  ],
-  "refresh": "30s",
-  "schemaVersion": 38,
-  "style": "dark",
-  "tags": ["rustfs", "get-optimization", "resource-impact"],
-  "templating": {
-    "list": [
-      {
-        "current": {
-          "selected": false,
-          "text": "Prometheus",
-          "value": "Prometheus"
-        },
-        "hide": 0,
-        "includeAll": false,
-        "multi": false,
-        "name": "DS_PROMETHEUS",
-        "options": [],
-        "query": "prometheus",
-        "refresh": 1,
-        "regex": "",
-        "skipUrlSync": false,
-        "type": "datasource"
-      }
-    ]
-  },
-  "time": {
-    "from": "now-1h",
-    "to": "now"
-  },
-  "timepicker": {},
-  "timezone": "",
-  "title": "RustFS GET Resource Impact",
-  "uid": "rustfs-get-resource-impact",
-  "version": 1
-}

.docker/observability/grafana/dashboards/grafana-get-rollout-health.json

@@ -1,530 +0,0 @@
-{
-  "annotations": {
-    "list": []
-  },
-  "editable": true,
-  "fiscalYearStartMonth": 0,
-  "graphTooltip": 1,
-  "id": null,
-  "links": [],
-  "liveNow": false,
-  "panels": [
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "s",
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 10,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "axisLabel": "",
-            "axisColorMode": "text",
-            "scaleDistribution": {
-              "type": "linear"
-            },
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 0,
-        "y": 0
-      },
-      "id": 1,
-      "options": {
-        "reduceOptions": {
-          "values": false,
-          "calcs": ["lastNotNull"],
-          "fields": ""
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        },
-        "legend": {
-          "displayMode": "table",
-          "placement": "bottom",
-          "calcs": ["mean", "max"]
-        }
-      },
-      "title": "GET Latency by Reader Path (p50 / p95 / p99)",
-      "type": "timeseries",
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "histogram_quantile(0.50, sum(rate(rustfs_io_get_object_stage_duration_seconds_bucket{stage=\"total\"}[$__rate_interval])) by (le, path))",
-          "legendFormat": "p50 {{path}}",
-          "refId": "A"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "histogram_quantile(0.95, sum(rate(rustfs_io_get_object_stage_duration_seconds_bucket{stage=\"total\"}[$__rate_interval])) by (le, path))",
-          "legendFormat": "p95 {{path}}",
-          "refId": "B"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "histogram_quantile(0.99, sum(rate(rustfs_io_get_object_stage_duration_seconds_bucket{stage=\"total\"}[$__rate_interval])) by (le, path))",
-          "legendFormat": "p99 {{path}}",
-          "refId": "C"
-        }
-      ]
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "percentunit",
-          "min": 0,
-          "max": 1,
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 15,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              },
-              {
-                "color": "yellow",
-                "value": 0.3
-              },
-              {
-                "color": "red",
-                "value": 0.7
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 12,
-        "y": 0
-      },
-      "id": 2,
-      "options": {
-        "reduceOptions": {
-          "values": false,
-          "calcs": ["lastNotNull"],
-          "fields": ""
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        },
-        "legend": {
-          "displayMode": "table",
-          "placement": "bottom",
-          "calcs": ["mean", "max"]
-        }
-      },
-      "title": "Early-Stop Hit Rate",
-      "type": "timeseries",
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(rate(rustfs_io_get_object_metadata_early_stop_total{decision=\"hit\"}[$__rate_interval])) by (path) / clamp_min(sum(rate(rustfs_io_get_object_metadata_early_stop_total[$__rate_interval])) by (path), 1)",
-          "legendFormat": "hit rate {{path}}",
-          "refId": "A"
-        }
-      ]
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "percentunit",
-          "min": 0,
-          "max": 1,
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 15,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 0,
-        "y": 8
-      },
-      "id": 3,
-      "options": {
-        "reduceOptions": {
-          "values": false,
-          "calcs": ["lastNotNull"],
-          "fields": ""
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        },
-        "legend": {
-          "displayMode": "table",
-          "placement": "bottom",
-          "calcs": ["mean", "max"]
-        }
-      },
-      "title": "Codec Streaming Usage Rate",
-      "type": "timeseries",
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(rate(rustfs_io_get_object_codec_streaming_decision_total[$__rate_interval])) by (decision) / clamp_min(sum(rate(rustfs_io_get_object_codec_streaming_decision_total[$__rate_interval])), 1)",
-          "legendFormat": "codec_streaming {{decision}}",
-          "refId": "A"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(rate(rustfs_io_get_object_stream_strategy_total[$__rate_interval])) by (strategy) / clamp_min(sum(rate(rustfs_io_get_object_stream_strategy_total[$__rate_interval])), 1)",
-          "legendFormat": "stream_strategy {{strategy}}",
-          "refId": "B"
-        }
-      ]
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "ops",
-          "custom": {
-            "drawStyle": "bars",
-            "fillOpacity": 80,
-            "lineWidth": 0,
-            "pointSize": 5,
-            "showPoints": "never",
-            "spanNulls": false,
-            "stacking": {
-              "mode": "normal",
-              "group": "A"
-            },
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              },
-              {
-                "color": "red",
-                "value": 1
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 12,
-        "y": 8
-      },
-      "id": 4,
-      "options": {
-        "reduceOptions": {
-          "values": false,
-          "calcs": ["lastNotNull"],
-          "fields": ""
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        },
-        "legend": {
-          "displayMode": "table",
-          "placement": "bottom",
-          "calcs": ["sum"]
-        }
-      },
-      "title": "Pipeline Failure Rate by Stage",
-      "type": "timeseries",
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(rate(rustfs_io_get_object_pipeline_failures_total[$__rate_interval])) by (stage, reason)",
-          "legendFormat": "{{stage}} / {{reason}}",
-          "refId": "A"
-        }
-      ]
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "reqps",
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 10,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 0,
-        "y": 16
-      },
-      "id": 5,
-      "options": {
-        "reduceOptions": {
-          "values": false,
-          "calcs": ["lastNotNull"],
-          "fields": ""
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        },
-        "legend": {
-          "displayMode": "table",
-          "placement": "bottom",
-          "calcs": ["mean", "max"]
-        }
-      },
-      "title": "GET Request Rate by Path",
-      "type": "timeseries",
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(rate(rustfs_io_get_object_reader_path_total[$__rate_interval])) by (path)",
-          "legendFormat": "{{path}}",
-          "refId": "A"
-        }
-      ]
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "ops",
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 10,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 12,
-        "y": 16
-      },
-      "id": 6,
-      "options": {
-        "reduceOptions": {
-          "values": false,
-          "calcs": ["lastNotNull"],
-          "fields": ""
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        },
-        "legend": {
-          "displayMode": "table",
-          "placement": "bottom",
-          "calcs": ["mean", "max"]
-        }
-      },
-      "title": "GET Total Latency (p50 / p95 / p99)",
-      "type": "timeseries",
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "histogram_quantile(0.50, sum(rate(rustfs_io_get_object_total_duration_seconds_bucket[$__rate_interval])) by (le))",
-          "legendFormat": "p50",
-          "refId": "A"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "histogram_quantile(0.95, sum(rate(rustfs_io_get_object_total_duration_seconds_bucket[$__rate_interval])) by (le))",
-          "legendFormat": "p95",
-          "refId": "B"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "histogram_quantile(0.99, sum(rate(rustfs_io_get_object_total_duration_seconds_bucket[$__rate_interval])) by (le))",
-          "legendFormat": "p99",
-          "refId": "C"
-        }
-      ]
-    }
-  ],
-  "refresh": "30s",
-  "schemaVersion": 38,
-  "style": "dark",
-  "tags": ["rustfs", "get-optimization"],
-  "templating": {
-    "list": [
-      {
-        "current": {
-          "selected": false,
-          "text": "Prometheus",
-          "value": "Prometheus"
-        },
-        "hide": 0,
-        "includeAll": false,
-        "multi": false,
-        "name": "DS_PROMETHEUS",
-        "options": [],
-        "query": "prometheus",
-        "refresh": 1,
-        "regex": "",
-        "skipUrlSync": false,
-        "type": "datasource"
-      }
-    ]
-  },
-  "time": {
-    "from": "now-1h",
-    "to": "now"
-  },
-  "timepicker": {},
-  "timezone": "",
-  "title": "RustFS GET Rollout Health",
-  "uid": "rustfs-get-rollout-health",
-  "version": 1
-}

.docker/observability/grafana/dashboards/grafana-put-performance-attribution.json

@@ -1,534 +0,0 @@
-{
-  "annotations": {
-    "list": []
-  },
-  "editable": true,
-  "fiscalYearStartMonth": 0,
-  "graphTooltip": 1,
-  "id": null,
-  "links": [],
-  "liveNow": false,
-  "panels": [
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "ops",
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 10,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 0,
-        "y": 0
-      },
-      "id": 1,
-      "options": {
-        "legend": {
-          "calcs": [
-            "mean",
-            "max"
-          ],
-          "displayMode": "table",
-          "placement": "bottom"
-        },
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ],
-          "fields": "",
-          "values": false
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum by (path, eager_status, size_bucket, buffer_bucket, large_concurrency_tuning) (rate(rustfs_s3_put_object_diagnostics_total[$__rate_interval])) or vector(0)",
-          "legendFormat": "{{path}} / {{eager_status}} / {{size_bucket}} / {{buffer_bucket}} / large={{large_concurrency_tuning}}",
-          "refId": "A"
-        }
-      ],
-      "title": "PUT Diagnostics Decision Rate",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "ops",
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 10,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 12,
-        "y": 0
-      },
-      "id": 2,
-      "options": {
-        "legend": {
-          "calcs": [
-            "mean",
-            "max"
-          ],
-          "displayMode": "table",
-          "placement": "bottom"
-        },
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ],
-          "fields": "",
-          "values": false
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum by (path) (rate(rustfs_s3_put_object_path_total[$__rate_interval])) or vector(0)",
-          "legendFormat": "{{path}}",
-          "refId": "A"
-        }
-      ],
-      "title": "PUT Path Rate",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "ms",
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 10,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 0,
-        "y": 8
-      },
-      "id": 3,
-      "options": {
-        "legend": {
-          "calcs": [
-            "mean",
-            "max"
-          ],
-          "displayMode": "table",
-          "placement": "bottom"
-        },
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ],
-          "fields": "",
-          "values": false
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "histogram_quantile(0.95, sum by (stage, le) (rate(rustfs_s3_put_object_stage_duration_ms_bucket{stage=~\"ingress_prepare|set_disk_writer_setup|set_disk_encode|set_disk_rename|set_disk_old_data_cleanup|multipart_ingress_prepare|multipart_set_disk_writer_setup|multipart_set_disk_encode|multipart_complete_tail\"}[$__rate_interval]))) or vector(0)",
-          "legendFormat": "p95 {{stage}}",
-          "refId": "A"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "histogram_quantile(0.99, sum by (stage, le) (rate(rustfs_s3_put_object_stage_duration_ms_bucket{stage=~\"ingress_prepare|set_disk_writer_setup|set_disk_encode|set_disk_rename|set_disk_old_data_cleanup|multipart_ingress_prepare|multipart_set_disk_writer_setup|multipart_set_disk_encode|multipart_complete_tail\"}[$__rate_interval]))) or vector(0)",
-          "legendFormat": "p99 {{stage}}",
-          "refId": "B"
-        }
-      ],
-      "title": "PUT Hot Stage Latency",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "bytes",
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 10,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 12,
-        "y": 8
-      },
-      "id": 4,
-      "options": {
-        "legend": {
-          "calcs": [
-            "mean",
-            "max"
-          ],
-          "displayMode": "table",
-          "placement": "bottom"
-        },
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ],
-          "fields": "",
-          "values": false
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "histogram_quantile(0.50, sum by (path, size_bucket, le) (rate(rustfs_s3_put_object_selected_buffer_size_bytes_bucket[$__rate_interval]))) or vector(0)",
-          "legendFormat": "p50 {{path}} / {{size_bucket}}",
-          "refId": "A"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "histogram_quantile(0.95, sum by (path, size_bucket, le) (rate(rustfs_s3_put_object_selected_buffer_size_bytes_bucket[$__rate_interval]))) or vector(0)",
-          "legendFormat": "p95 {{path}} / {{size_bucket}}",
-          "refId": "B"
-        }
-      ],
-      "title": "PUT Selected Buffer Size",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "percentunit",
-          "min": 0,
-          "max": 1,
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 15,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 0,
-        "y": 16
-      },
-      "id": 5,
-      "options": {
-        "legend": {
-          "calcs": [
-            "mean",
-            "max"
-          ],
-          "displayMode": "table",
-          "placement": "bottom"
-        },
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ],
-          "fields": "",
-          "values": false
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(rate(rustfs_s3_put_object_zero_copy_eligible_total[$__rate_interval])) / clamp_min(sum(rate(rustfs_s3_put_object_total[$__rate_interval])), 1) or vector(0)",
-          "legendFormat": "eligible / total",
-          "refId": "A"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(rate(rustfs_s3_put_object_zero_copy_enabled_total[$__rate_interval])) / clamp_min(sum(rate(rustfs_s3_put_object_total[$__rate_interval])), 1) or vector(0)",
-          "legendFormat": "enabled / total",
-          "refId": "B"
-        }
-      ],
-      "title": "PUT Zero-Copy Eligibility Ratio",
-      "type": "timeseries"
-    },
-    {
-      "datasource": {
-        "type": "prometheus",
-        "uid": "${DS_PROMETHEUS}"
-      },
-      "fieldConfig": {
-        "defaults": {
-          "unit": "ops",
-          "custom": {
-            "drawStyle": "line",
-            "lineInterpolation": "smooth",
-            "fillOpacity": 10,
-            "pointSize": 5,
-            "lineWidth": 1,
-            "showPoints": "auto",
-            "spanNulls": false,
-            "thresholdsStyle": {
-              "mode": "off"
-            }
-          },
-          "thresholds": {
-            "mode": "absolute",
-            "steps": [
-              {
-                "color": "green",
-                "value": null
-              }
-            ]
-          }
-        },
-        "overrides": []
-      },
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 12,
-        "y": 16
-      },
-      "id": 6,
-      "options": {
-        "legend": {
-          "calcs": [
-            "mean",
-            "max"
-          ],
-          "displayMode": "table",
-          "placement": "bottom"
-        },
-        "reduceOptions": {
-          "calcs": [
-            "lastNotNull"
-          ],
-          "fields": "",
-          "values": false
-        },
-        "tooltip": {
-          "mode": "multi",
-          "sort": "desc"
-        }
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum(rate(rustfs_s3_put_object_total[$__rate_interval])) or vector(0)",
-          "legendFormat": "PUT total",
-          "refId": "A"
-        },
-        {
-          "datasource": {
-            "type": "prometheus",
-            "uid": "${DS_PROMETHEUS}"
-          },
-          "expr": "sum by (stage) (rate(rustfs_system_storage_erasure_write_quorum_failures_total[$__rate_interval])) or vector(0)",
-          "legendFormat": "write quorum failure {{stage}}",
-          "refId": "B"
-        }
-      ],
-      "title": "PUT Throughput and Quorum Failure Signals",
-      "type": "timeseries"
-    }
-  ],
-  "refresh": "30s",
-  "schemaVersion": 38,
-  "style": "dark",
-  "tags": [
-    "rustfs",
-    "put",
-    "performance",
-    "attribution"
-  ],
-  "templating": {
-    "list": [
-      {
-        "current": {
-          "selected": false,
-          "text": "Prometheus",
-          "value": "Prometheus"
-        },
-        "hide": 0,
-        "includeAll": false,
-        "multi": false,
-        "name": "DS_PROMETHEUS",
-        "options": [],
-        "query": "prometheus",
-        "refresh": 1,
-        "regex": "",
-        "skipUrlSync": false,
-        "type": "datasource"
-      }
-    ]
-  },
-  "time": {
-    "from": "now-1h",
-    "to": "now"
-  },
-  "timezone": "",
-  "title": "RustFS PUT Performance Attribution",
-  "uid": "rustfs-put-performance-attribution",
-  "version": 1,
-  "weekStart": ""
-}

.docker/observability/grafana/provisioning/dashboards/dashboard.yml

@@ -1,25 +0,0 @@
-# Copyright 2024 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-apiVersion: 1
-
-providers:
-  - name: "default"
-    orgId: 1
-    folder: ""
-    type: file
-    disableDeletion: false
-    updateIntervalSeconds: 10
-    options:
-      path: /etc/grafana/dashboards

.docker/observability/grafana/provisioning/datasources/datasources.yaml

@@ -1,98 +0,0 @@
-# Copyright 2024 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-apiVersion: 1
-
-datasources:
-  - name: Prometheus
-    type: prometheus
-    uid: prometheus
-    url: http://prometheus:9090
-    access: proxy
-    isDefault: true
-    editable: false
-    jsonData:
-      httpMethod: GET
-      exemplarTraceIdDestinations:
-        - name: trace_id
-          datasourceUid: tempo
-
-  - name: Tempo
-    type: tempo
-    uid: tempo
-    access: proxy
-    url: http://tempo:3200
-    isDefault: false
-    editable: false
-    jsonData:
-      httpMethod: GET
-      serviceMap:
-        datasourceUid: prometheus
-      tracesToLogs:
-        datasourceUid: loki
-        tags: [ 'job', 'instance', 'pod', 'namespace', 'service.name' ]
-        mappedTags: [ { key: 'service.name', value: 'app' } ]
-        spanStartTimeShift: '-1h'
-        spanEndTimeShift: '1h'
-        filterByTraceID: true
-        filterBySpanID: false
-      tracesToMetrics:
-        datasourceUid: prometheus
-        tags: [ { key: 'service.name' }, { key: 'job' } ]
-        queries:
-          - name: 'Service-Level Latency'
-            query: 'sum(rate(traces_spanmetrics_latency_bucket{$$__tags}[5m])) by (le)'
-          - name: 'Service-Level Calls'
-            query: 'sum(rate(traces_spanmetrics_calls_total{$$__tags}[5m]))'
-          - name: 'Service-Level Errors'
-            query: 'sum(rate(traces_spanmetrics_calls_total{status_code="ERROR", $$__tags}[5m]))'
-      nodeGraph:
-        enabled: true
-
-  - name: Loki
-    type: loki
-    uid: loki
-    url: http://loki:3100
-    basicAuth: false
-    isDefault: false
-    editable: false
-    jsonData:
-      derivedFields:
-        - datasourceUid: tempo
-          matcherRegex: 'trace_id=(\w+)'
-          name: 'TraceID'
-          url: '$${__value.raw}'
-
-  - name: Jaeger
-    type: jaeger
-    uid: jaeger
-    url: http://jaeger:16686
-    access: proxy
-    isDefault: false
-    editable: false
-    jsonData:
-      tracesToLogs:
-        datasourceUid: loki
-        tags: [ 'job', 'instance', 'pod', 'namespace', 'service.name' ]
-        mappedTags: [ { key: 'service.name', value: 'app' } ]
-        spanStartTimeShift: '1s'
-        spanEndTimeShift: '-1s'
-        filterByTraceID: true
-        filterBySpanID: false
-
-  - name: Pyroscope
-    type: grafana-pyroscope-datasource
-    url: http://pyroscope:4040
-    jsonData:
-      minStep: '15s'

.docker/observability/jaeger-config.yaml

@@ -0,0 +1,112 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+service:
+  extensions: [ jaeger_storage, jaeger_query, remote_sampling, healthcheckv2 ]
+  pipelines:
+    traces:
+      receivers: [ otlp, jaeger, zipkin ]
+      processors: [ batch, adaptive_sampling ]
+      exporters: [ jaeger_storage_exporter ]
+  telemetry:
+    resource:
+      service.name: jaeger
+    metrics:
+      level: detailed
+      readers:
+        - pull:
+            exporter:
+              prometheus:
+                host: 0.0.0.0
+                port: 8888
+    logs:
+      level: debug
+    # TODO Initialize telemetry tracer once OTEL released new feature.
+    # https://github.com/open-telemetry/opentelemetry-collector/issues/10663
+
+extensions:
+  healthcheckv2:
+    use_v2: true
+    http:
+
+  # pprof:
+  #   endpoint: 0.0.0.0:1777
+  # zpages:
+  #   endpoint: 0.0.0.0:55679
+
+  jaeger_query:
+    storage:
+      traces: some_store
+      traces_archive: another_store
+    ui:
+      config_file: ./cmd/jaeger/config-ui.json
+      log_access: true
+    # The maximum duration that is considered for clock skew adjustments.
+    # Defaults to 0 seconds, which means it's disabled.
+    max_clock_skew_adjust: 0s
+    grpc:
+      endpoint: 0.0.0.0:16685
+    http:
+      endpoint: 0.0.0.0:16686
+
+  jaeger_storage:
+    backends:
+      some_store:
+        memory:
+          max_traces: 1000000
+      another_store:
+        memory:
+          max_traces: 1000000
+    metric_backends:
+      some_metrics_storage:
+        prometheus:
+          endpoint: http://prometheus:9090
+          normalize_calls: true
+          normalize_duration: true
+
+  remote_sampling:
+    # You can either use file or adaptive sampling strategy in remote_sampling
+    # file:
+    #   path: ./cmd/jaeger/sampling-strategies.json
+    adaptive:
+      sampling_store: some_store
+      initial_sampling_probability: 0.1
+    http:
+    grpc:
+
+receivers:
+  otlp:
+    protocols:
+      grpc:
+      http:
+
+  jaeger:
+    protocols:
+      grpc:
+      thrift_binary:
+      thrift_compact:
+      thrift_http:
+
+  zipkin:
+
+processors:
+  batch:
+  # Adaptive Sampling Processor is required to support adaptive sampling.
+  # It expects remote_sampling extension with `adaptive:` config to be enabled.
+  adaptive_sampling:
+
+exporters:
+  jaeger_storage_exporter:
+    trace_storage: some_store
+

.docker/observability/jaeger.yaml

@@ -1,74 +0,0 @@
-# Copyright 2024 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-service:
-  extensions: [jaeger_storage, jaeger_query]
-  pipelines:
-    traces:
-      receivers: [otlp]
-      processors: [batch]
-      exporters: [jaeger_storage_exporter, spanmetrics]
-    metrics/spanmetrics:
-      receivers: [spanmetrics]
-      exporters: [prometheus]
-  telemetry:
-    resource:
-      service.name: jaeger
-    metrics:
-      level: detailed
-      readers:
-        - pull:
-            exporter:
-              prometheus:
-                host: 0.0.0.0
-                port: 8888
-    logs:
-      level: DEBUG
-
-extensions:
-  jaeger_query:
-    storage:
-      traces: some_storage
-      metrics: some_metrics_storage
-  jaeger_storage:
-    backends:
-      some_storage:
-        memory:
-          max_traces: 100000
-    metric_backends:
-      some_metrics_storage:
-        prometheus:
-          endpoint: http://prometheus:9090
-          normalize_calls: true
-          normalize_duration: true
-
-connectors:
-  spanmetrics:
-
-receivers:
-  otlp:
-    protocols:
-      grpc:
-        endpoint: "0.0.0.0:4317"
-      http:
-        endpoint: "0.0.0.0:4318"
-
-processors:
-  batch:
-
-exporters:
-  jaeger_storage_exporter:
-    trace_storage: some_storage
-  prometheus:
-    endpoint: "0.0.0.0:8889"

.docker/observability/loki-config.yaml

@@ -11,21 +11,22 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
+
 auth_enabled: false
 
 server:
   http_listen_port: 3100
-  grpc_listen_port: 9095
-  log_level: info
+  grpc_listen_port: 9096
+  log_level: debug
   grpc_server_max_concurrent_streams: 1000
 
 common:
   instance_addr: 127.0.0.1
-  path_prefix: /loki
+  path_prefix: /tmp/loki
   storage:
     filesystem:
-      chunks_directory: /loki/chunks
-      rules_directory: /loki/rules
+      chunks_directory: /tmp/loki/chunks
+      rules_directory: /tmp/loki/rules
   replication_factor: 1
   ring:
     kvstore:
@@ -38,6 +39,9 @@ query_range:
         enabled: true
         max_size_mb: 100
 
+limits_config:
+  metric_aggregation_enabled: true
+
 schema_config:
   configs:
     - from: 2020-10-24
@@ -48,16 +52,26 @@ schema_config:
         prefix: index_
         period: 24h
 
-limits_config:
-  reject_old_samples: true
-  reject_old_samples_max_age: 168h
-  allow_structured_metadata: true
-  max_line_size: 256KB
-
 pattern_ingester:
   enabled: true
   metric_aggregation:
     loki_address: localhost:3100
 
+ruler:
+  alertmanager_url: http://localhost:9093
+
 frontend:
   encoding: protobuf
+
+# By default, Loki will send anonymous, but uniquely-identifiable usage and configuration
+# analytics to Grafana Labs. These statistics are sent to https://stats.grafana.org/
+#
+# Statistics help us better understand how Loki is used, and they show us performance
+# levels for most users. This helps us prioritize features and documentation.
+# For more information on what's sent, look at
+# https://github.com/grafana/loki/blob/main/pkg/analytics/stats.go
+# Refer to the buildReport method to see what goes into a report.
+#
+# If you would like to disable reporting, uncomment the following lines:
+#analytics:
+#  reporting_enabled: false

.docker/observability/otel-collector-config.yaml

@@ -15,118 +15,67 @@
 receivers:
   otlp:
     protocols:
-      grpc:
+      grpc: # OTLP gRPC 接收器
         endpoint: 0.0.0.0:4317
-      http:
+      http: # OTLP HTTP 接收器
         endpoint: 0.0.0.0:4318
 
 processors:
-  batch:
-    timeout: 1s
-    send_batch_size: 1024
+  batch: # 批处理处理器，提升吞吐量
+    timeout: 5s
+    send_batch_size: 1000
   memory_limiter:
     check_interval: 1s
-    limit_mib: 1024
-    spike_limit_mib: 256
-  transform/logs:
-    error_mode: ignore
-    log_statements:
-      - context: log
-        statements:
-          - set(attributes["message"], body.string) where IsString(body)
-          - set(attributes["log.body"], body.string) where IsString(body)
-          - set(body, Concat([body, " event=", attributes["event"]], "")) where IsString(body) and attributes["event"] != nil
-          - set(body, Concat([body, " component=", attributes["component"]], "")) where IsString(body) and attributes["component"] != nil
-          - set(body, Concat([body, " subsystem=", attributes["subsystem"]], "")) where IsString(body) and attributes["subsystem"] != nil
-          - set(body, Concat([body, " state=", attributes["state"]], "")) where IsString(body) and attributes["state"] != nil
-          - set(body, Concat([body, " result=", attributes["result"]], "")) where IsString(body) and attributes["result"] != nil
-          - set(body, Concat([body, " reason=", attributes["reason"]], "")) where IsString(body) and attributes["reason"] != nil
-          - set(body, Concat([body, " failure_reason=", attributes["failure_reason"]], "")) where IsString(body) and attributes["failure_reason"] != nil
-          - set(body, Concat([body, " rpc_path=", attributes["rpc_path"]], "")) where IsString(body) and attributes["rpc_path"] != nil
-          - set(body, Concat([body, " rpc_service=", attributes["rpc_service"]], "")) where IsString(body) and attributes["rpc_service"] != nil
-          - set(body, Concat([body, " rpc_method=", attributes["rpc_method"]], "")) where IsString(body) and attributes["rpc_method"] != nil
-          - set(body, Concat([body, " expected_audience=", attributes["expected_audience"]], "")) where IsString(body) and attributes["expected_audience"] != nil
-          - set(body, Concat([body, " peer_addr=", attributes["peer_addr"]], "")) where IsString(body) and attributes["peer_addr"] != nil
-          - set(body, Concat([body, " replay_scope_bootstrap_allowed=", attributes["replay_scope_bootstrap_allowed"]], "")) where IsString(body) and attributes["replay_scope_bootstrap_allowed"] != nil
-          - set(body, Concat([body, " error=", attributes["error"]], "")) where IsString(body) and attributes["error"] != nil
-          - set(body, Concat([body, " exception_message=", attributes["exception.message"]], "")) where IsString(body) and attributes["exception.message"] != nil
+    limit_mib: 512
 
 exporters:
-  otlp/tempo:
-    endpoint: "tempo:4317"
+  otlp/traces: # OTLP 导出器，用于跟踪数据
+    endpoint: "jaeger:4317"  # Jaeger 的 OTLP gRPC 端点
+    tls:
+      insecure: true  # 开发环境禁用 TLS，生产环境需配置证书
+  otlp/tempo: # OTLP 导出器，用于跟踪数据
+    endpoint: "tempo:4317"  # tempo 的 OTLP gRPC 端点
+    tls:
+      insecure: true  # 开发环境禁用 TLS，生产环境需配置证书
+  prometheus: # Prometheus 导出器，用于指标数据
+    endpoint: "0.0.0.0:8889"  # Prometheus 刮取端点
+    namespace: "rustfs"  # 指标前缀
+    send_timestamps: true  # 发送时间戳
+    # enable_open_metrics: true
+  otlphttp/loki: # Loki 导出器，用于日志数据
+    # endpoint: "http://loki:3100/otlp/v1/logs"
+    endpoint: "http://loki:3100/otlp/v1/logs"
     tls:
       insecure: true
-    compression: gzip
-    retry_on_failure:
-      enabled: true
-      initial_interval: 1s
-      max_interval: 30s
-      max_elapsed_time: 300s
-    sending_queue:
-      enabled: true
-      num_consumers: 10
-      queue_size: 5000
-
-  otlp/jaeger:
-    endpoint: "jaeger:4317"
-    tls:
-      insecure: true
-    compression: gzip
-    retry_on_failure:
-      enabled: true
-      initial_interval: 1s
-      max_interval: 30s
-      max_elapsed_time: 300s
-    sending_queue:
-      enabled: true
-      num_consumers: 10
-      queue_size: 5000
-
-  prometheus:
-    endpoint: "0.0.0.0:8889"
-    send_timestamps: true
-    metric_expiration: 5m
-    resource_to_telemetry_conversion:
-      enabled: true
-
-  otlphttp/loki:
-    endpoint: "http://loki:3100/otlp"
-    tls:
-      insecure: true
-    compression: gzip
-
 extensions:
   health_check:
-    endpoint: 0.0.0.0:13133
   pprof:
-    endpoint: 0.0.0.0:1888
   zpages:
-    endpoint: 0.0.0.0:55679
-
 service:
-  extensions: [ health_check, pprof, zpages ]
+  extensions: [ health_check, pprof, zpages ]  # 启用扩展
   pipelines:
     traces:
       receivers: [ otlp ]
-      processors: [ memory_limiter, batch ]
-      exporters: [ otlp/tempo, otlp/jaeger ]
+      processors: [ memory_limiter,batch ]
+      exporters: [ otlp/traces,otlp/tempo ]
     metrics:
       receivers: [ otlp ]
       processors: [ batch ]
       exporters: [ prometheus ]
     logs:
       receivers: [ otlp ]
-      processors: [ batch, transform/logs ]
+      processors: [ batch ]
       exporters: [ otlphttp/loki ]
   telemetry:
     logs:
-      level: "info"
-      encoding: "json"
+      level: "info"  # Collector 日志级别
     metrics:
-      level: "normal"
+      level: "detailed" # 可以是 basic, normal, detailed
       readers:
-        - pull:
+        - periodic:
             exporter:
-              prometheus:
-                host: '0.0.0.0'
-                port: 8888
+              otlp:
+                protocol: http/protobuf
+                endpoint: http://otel-collector:4318
+
+

.docker/observability/prometheus-rules/rustfs-dashboard.yml

@@ -1,53 +0,0 @@
-groups:
-  - name: rustfs-dashboard
-    interval: 30s
-    rules:
-      - record: rustfs:http_server_requests:rate5m
-        expr: sum by (job) (rate(rustfs_http_server_requests_total[5m]))
-
-      - record: rustfs:http_server_request_duration_seconds:p50_5m
-        expr: histogram_quantile(0.50, sum by (le, job) (rate(rustfs_http_server_request_duration_seconds_bucket[5m])))
-      - record: rustfs:http_server_request_duration_seconds:p95_5m
-        expr: histogram_quantile(0.95, sum by (le, job) (rate(rustfs_http_server_request_duration_seconds_bucket[5m])))
-      - record: rustfs:http_server_request_duration_seconds:p99_5m
-        expr: histogram_quantile(0.99, sum by (le, job) (rate(rustfs_http_server_request_duration_seconds_bucket[5m])))
-
-      - record: rustfs:http_server_response_body_size_bytes:p50_5m
-        expr: histogram_quantile(0.50, sum by (le, job) (rate(rustfs_http_server_response_body_size_bytes_bucket[5m])))
-      - record: rustfs:http_server_response_body_size_bytes:p95_5m
-        expr: histogram_quantile(0.95, sum by (le, job) (rate(rustfs_http_server_response_body_size_bytes_bucket[5m])))
-      - record: rustfs:http_server_response_body_size_bytes:p99_5m
-        expr: histogram_quantile(0.99, sum by (le, job) (rate(rustfs_http_server_response_body_size_bytes_bucket[5m])))
-
-      - record: rustfs:log_cleaner_runs:rate15m
-        expr: sum by (job) (rate(rustfs_log_cleaner_runs_total[15m]))
-      - record: rustfs:log_cleaner_failure_ratio:rate5m
-        expr: sum by (job) (rate(rustfs_log_cleaner_run_failures_total[5m])) / clamp_min(sum by (job) (rate(rustfs_log_cleaner_runs_total[5m])), 1e-9)
-      - record: rustfs:log_cleaner_rotation_failure_ratio:rate5m
-        expr: sum by (job) (rate(rustfs_log_cleaner_rotation_failures_total[5m])) / clamp_min(sum by (job) (rate(rustfs_log_cleaner_rotation_total[5m])), 1e-9)
-      - record: rustfs:log_cleaner_rotation_duration_seconds:p95_5m
-        expr: histogram_quantile(0.95, sum by (le, job) (rate(rustfs_log_cleaner_rotation_duration_seconds_bucket[5m])))
-      - record: rustfs:log_cleaner_compress_duration_seconds:p95_5m
-        expr: histogram_quantile(0.95, sum by (le, job) (rate(rustfs_log_cleaner_compress_duration_seconds_bucket[5m])))
-
-      - record: rustfs:scanner_objects_scanned:rate5m
-        expr: sum by (job) (rate(rustfs_scanner_objects_scanned_total[5m]))
-      - record: rustfs:scanner_directories_scanned:rate5m
-        expr: sum by (job) (rate(rustfs_scanner_directories_scanned_total[5m]))
-      - record: rustfs:scanner_buckets_scanned:rate5m
-        expr: sum by (job) (rate(rustfs_scanner_buckets_scanned_total[5m]))
-      - record: rustfs:scanner_cycles_success:rate5m
-        expr: sum by (job) (rate(rustfs_scanner_cycles_total{result="success"}[5m]))
-
-      - record: rustfs:log_chain_op_event_mismatch:rate5m
-        expr: sum by (job) (rate(rustfs_log_chain_op_event_mismatch_total[5m]))
-
-      - alert: RustFSLogChainOpEventMismatchDetected
-        expr: rustfs:log_chain_op_event_mismatch:rate5m > 0
-        for: 10m
-        labels:
-          severity: warning
-          component: s3-log-chain
-        annotations:
-          summary: "RustFS log-chain op/event mismatch detected"
-          description: "job={{ $labels.job }} has non-zero rustfs_log_chain_op_event_mismatch_total rate for more than 10m. Check s3 op/event mapping changes."

.docker/observability/prometheus-rules/rustfs-get-optimization-alerts.yaml

@@ -1,260 +0,0 @@
-# =============================================================================
-# RustFS GET Optimization — Prometheus Alerting Rules
-# =============================================================================
-#
-# Import into Prometheus:
-#   1. Copy this file to your Prometheus rules directory
-#   2. Add to prometheus.yml:
-#        rule_files:
-#          - "prometheus-alert-rules.yaml"
-#   3. Validate: promtool check rules prometheus-alert-rules.yaml
-#   4. Reload:   curl -X POST http://localhost:9090/-/reload
-#
-# All metric names match those registered in crates/io-metrics/src/lib.rs
-# and documented in crates/ecstore/src/diagnostics/get.rs.
-#
-# Baseline comparison uses "offset 1d" — adjust to "offset 7d" for weekly
-# seasonality if your traffic pattern varies by day of week.
-# =============================================================================
-
-groups:
-  # ==========================================================================
-  # Critical alerts — immediate action required
-  # ==========================================================================
-  - name: rustfs-get-optimization-critical
-    interval: 30s
-    rules:
-      # ------------------------------------------------------------------
-      # 1. GetP99Regression
-      #    GET p99 latency exceeds 2x the baseline (same time yesterday)
-      #    sustained for 10 minutes.
-      #    Action: Roll back the GET optimization immediately.
-      # ------------------------------------------------------------------
-      - alert: GetP99Regression
-        expr: |
-          histogram_quantile(0.99,
-            sum(rate(rustfs_io_get_object_total_duration_seconds_bucket[5m])) by (le)
-          )
-          >
-          2
-          *
-          histogram_quantile(0.99,
-            sum(rate(rustfs_io_get_object_total_duration_seconds_bucket[5m] offset 1d)) by (le)
-          )
-        for: 10m
-        labels:
-          severity: critical
-          team: rustfs-storage
-          area: get-optimization
-        annotations:
-          summary: "GET p99 latency regression detected (>2x baseline for 10m)"
-          description: >-
-            The 99th-percentile GET object latency is {{ $value | humanizeDuration }}
-            which is more than double the baseline measured 24 hours ago.
-            This indicates a severe performance regression introduced by
-            a recent GET optimization change.
-          runbook_url: "https://internal.wiki/runbooks/rustfs/get-p99-regression"
-          action: >
-            1. Verify the regression is not caused by external factors
-               (disk health, network, load spike).
-            2. If confirmed optimization-related, roll back:
-               - Set RUSTFS_GET_CODEC_STREAMING=0
-               - Set RUSTFS_GET_METADATA_EARLY_STOP=0
-               - Restart affected nodes.
-            3. Collect flamegraphs and open a P0 incident.
-
-      # ------------------------------------------------------------------
-      # 2. PipelineFailureSpike
-      #    Pipeline failure rate exceeds 5x the baseline sustained for
-      #    5 minutes.  Covers all failure reasons: bitrot_mismatch,
-      #    decode_error, downstream_closed, io, read_quorum, timeout, etc.
-      #    Action: Investigate pipeline health and roll back if needed.
-      # ------------------------------------------------------------------
-      - alert: PipelineFailureSpike
-        expr: |
-          sum(rate(rustfs_io_get_object_pipeline_failures_total[5m]))
-          >
-          5
-          *
-          sum(rate(rustfs_io_get_object_pipeline_failures_total[5m] offset 1d))
-        for: 5m
-        labels:
-          severity: critical
-          team: rustfs-storage
-          area: get-optimization
-        annotations:
-          summary: "GET pipeline failure rate spike (>5x baseline for 5m)"
-          description: >-
-            The GET pipeline failure rate is {{ $value | printf "%.2f" }}/s,
-            more than 5x the baseline from 24 hours ago.
-            Failure reasons may include: bitrot_mismatch, decode_error,
-            downstream_closed, io, range_or_length_invalid, read_quorum,
-            short_read, timeout, unknown.
-          runbook_url: "https://internal.wiki/runbooks/rustfs/pipeline-failure-spike"
-          action: >
-            1. Check Grafana "GET Data Integrity" dashboard for failure
-               breakdown by reason label.
-            2. If decode_error or bitrot_mismatch dominates, stop
-               optimization and investigate data integrity.
-            3. If io or timeout dominates, check disk and network health.
-            4. Roll back optimization if failures persist.
-
-      # ------------------------------------------------------------------
-      # 3. BitrotMismatchSpike
-      #    Bitrot verification mismatch rate exceeds 3x baseline for
-      #    5 minutes.  This is a data-integrity signal — shard checksums
-      #    do not match after read.
-      #
-      #    The "bitrot_mismatch" reason is recorded on the
-      #    rustfs_io_get_object_pipeline_failures_total counter when a
-      #    StorageError::FileCorrupt or DiskError::FileCorrupt /
-      #    DiskError::PartMissingOrCorrupt is classified during the GET
-      #    pipeline (see classify_storage_error / classify_disk_error in
-      #    crates/ecstore/src/diagnostics/get.rs).
-      #
-      #    Action: Stop optimization, investigate data integrity urgently.
-      # ------------------------------------------------------------------
-      - alert: BitrotMismatchSpike
-        expr: |
-          sum(rate(rustfs_io_get_object_pipeline_failures_total{reason="bitrot_mismatch"}[5m]))
-          >
-          3
-          *
-          sum(rate(rustfs_io_get_object_pipeline_failures_total{reason="bitrot_mismatch"}[5m] offset 1d))
-        for: 5m
-        labels:
-          severity: critical
-          team: rustfs-storage
-          area: get-optimization
-        annotations:
-          summary: "Bitrot mismatch rate spike (>3x baseline for 5m)"
-          description: >-
-            The rate of pipeline failures classified as bitrot_mismatch is
-            {{ $value | printf "%.2f" }}/s, more than 3x the baseline from
-            24 hours ago.  This indicates shard checksum verification
-            failures (FileCorrupt / PartMissingOrCorrupt) which may point
-            to data corruption introduced by the GET optimization pipeline
-            (e.g., incorrect decode, buffer reuse bug).
-          runbook_url: "https://internal.wiki/runbooks/rustfs/bitrot-mismatch-spike"
-          action: >
-            1. Immediately disable codec streaming:
-               RUSTFS_GET_CODEC_STREAMING=0
-            2. Run "mc admin scan" on affected buckets to verify on-disk
-               integrity independent of the GET path.
-            3. Compare xl.meta checksums across erasure shards.
-            4. If corruption confirmed, initiate data recovery from parity.
-            5. Do NOT re-enable optimization until root cause is identified.
-
-  # ==========================================================================
-  # Warning alerts — investigation needed
-  # ==========================================================================
-  - name: rustfs-get-optimization-warning
-    interval: 30s
-    rules:
-      # ------------------------------------------------------------------
-      # 4. EarlyStopInsufficientQuorum
-      #    The metadata early-stop path is hitting "insufficient_quorum"
-      #    at a rate above 0.1/s for 5 minutes.  This means too many
-      #    disks are failing to return valid metadata in time.
-      #    Action: Check disk health and metadata fanout latency.
-      # ------------------------------------------------------------------
-      - alert: EarlyStopInsufficientQuorum
-        expr: |
-          sum(rate(rustfs_io_get_object_metadata_early_stop_total{reason="insufficient_quorum"}[5m]))
-          > 0.1
-        for: 5m
-        labels:
-          severity: warning
-          team: rustfs-storage
-          area: get-optimization
-        annotations:
-          summary: "Early-stop insufficient quorum rate elevated (>0.1/s for 5m)"
-          description: >-
-            The metadata early-stop path is returning "insufficient_quorum"
-            at {{ $value | printf "%.3f" }}/s.  This means the bounded
-            metadata fanout cannot gather enough valid responses before
-            the quorum deadline, suggesting disk or network issues.
-          runbook_url: "https://internal.wiki/runbooks/rustfs/early-stop-quorum"
-          action: >
-            1. Check disk health: mc admin info --json | jq '.disks'
-            2. Review rustfs_io_get_object_metadata_response_total by
-               outcome (error, timeout, disk_not_found) in Grafana.
-            3. Check rustfs_io_disk_permit_wait_duration_seconds for
-               I/O scheduler saturation.
-            4. If disks are healthy, consider increasing the early-stop
-               timeout or temporarily disabling early-stop.
-
-      # ------------------------------------------------------------------
-      # 5. CodecStreamingFallbackSpike
-      #    The codec streaming fallback rate is >10x baseline for 10
-      #    minutes.  This means the optimized codec streaming path is
-      #    being bypassed much more often than expected.
-      #    Action: Check fallback reasons and object eligibility.
-      # ------------------------------------------------------------------
-      - alert: CodecStreamingFallbackSpike
-        expr: |
-          sum(rate(rustfs_io_get_object_codec_streaming_fallback_total[5m]))
-          >
-          10
-          *
-          sum(rate(rustfs_io_get_object_codec_streaming_fallback_total[5m] offset 1d))
-        for: 10m
-        labels:
-          severity: warning
-          team: rustfs-storage
-          area: get-optimization
-        annotations:
-          summary: "Codec streaming fallback rate spike (>10x baseline for 10m)"
-          description: >-
-            The codec streaming fallback rate is {{ $value | printf "%.2f" }}/s,
-            more than 10x the baseline from 24 hours ago.  Fallback reasons
-            are labeled by "reason" — check Grafana for breakdown.
-            Common reasons: object too small, multipart not supported,
-            unsupported erasure layout, feature flag disabled.
-          runbook_url: "https://internal.wiki/runbooks/rustfs/codec-fallback-spike"
-          action: >
-            1. Query by reason label:
-               sum by (reason) (rate(rustfs_io_get_object_codec_streaming_fallback_total[5m]))
-            2. If dominated by a single reason, investigate why that
-               condition became more frequent (e.g., workload change,
-               configuration drift).
-            3. Cross-reference with rustfs_io_get_object_reader_path_total
-               to verify the fallback path (legacy_duplex) is healthy.
-            4. If fallback is expected (e.g., workload shifted to small
-               objects), update the alert baseline.
-
-      # ------------------------------------------------------------------
-      # 6. IoQueueSaturation
-      #    I/O queue utilization exceeds 90% for 5 minutes.  High
-      #    utilization causes disk permit wait latency to increase and
-      #    can cascade into pipeline timeouts.
-      #    Action: Check disk load and consider reducing concurrency.
-      # ------------------------------------------------------------------
-      - alert: IoQueueSaturation
-        expr: |
-          rustfs_io_queue_utilization_percent > 90
-        for: 5m
-        labels:
-          severity: warning
-          team: rustfs-storage
-          area: get-optimization
-        annotations:
-          summary: "I/O queue utilization >90% for 5m"
-          description: >-
-            The I/O queue utilization is {{ $value | printf "%.1f" }}%,
-            sustained above 90% for 5 minutes.  This indicates the disk
-            I/O scheduler is near saturation, which will increase
-            rustfs_io_disk_permit_wait_duration_seconds and may trigger
-            pipeline timeouts.
-          runbook_url: "https://internal.wiki/runbooks/rustfs/io-queue-saturation"
-          action: >
-            1. Check disk I/O metrics (iostat, node_exporter) for
-               individual disk saturation.
-            2. Review rustfs_io_queue_permits_in_use vs
-               rustfs_io_queue_permits_available for permit exhaustion.
-            3. Check rustfs_io_starvation_events for priority starvation.
-            4. If GET optimization increased concurrency, consider:
-               - Reducing RUSTFS_GET_PIPELINE_PARALLELISM
-               - Lowering RUSTFS_IO_QUEUE_PERMITS
-            5. If caused by background operations (ILM, healing), throttle
-               those before adjusting GET concurrency.

.docker/observability/prometheus-rules/rustfs-kms-alerts.yml

@@ -1,251 +0,0 @@
-# Copyright 2024 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-# =============================================================================
-# RustFS KMS backend — Prometheus alerting rules
-# =============================================================================
-#
-# Metric source: the KMS operation-policy choke point in
-# crates/kms/src/policy.rs, except KmsKeyRotationOverdue, which reads the
-# label-less key-lifecycle gauge published by the deletion worker's sweep
-# (crates/kms/src/deletion_worker.rs). All label values are bounded static
-# strings (operation, op_class, outcome, error_class, backend, scope); key
-# identifiers, key material, and tokens never appear in labels.
-#
-# Response procedures: docs/operations/kms-observability-runbook.md
-#
-# IMPORTANT — threshold status: every numeric threshold below is a
-# conservative default chosen without a production baseline. Calibrate against
-# a staging baseline before relying on these alerts for paging, and prefer
-# loosening over tightening until the baseline exists. Formal SLO targets are
-# deliberately not encoded here (see rustfs/backlog#1584).
-#
-# NOTE: prometheus.yml loads /etc/prometheus/rules/*.yml — keep the .yml
-# extension or the file is silently ignored by the docker-compose stack.
-#
-# Validate: promtool check rules rustfs-kms-alerts.yml
-# =============================================================================
-
-groups:
-  # ==========================================================================
-  # Critical alerts — immediate action required
-  # ==========================================================================
-  - name: rustfs-kms-critical
-    interval: 30s
-    rules:
-      # ------------------------------------------------------------------
-      # 1. KmsBackendFatalErrors
-      #    Any attempt failure classified as fatal (non-retryable): auth
-      #    or permission errors, malformed requests, missing keys. The
-      #    policy never retries these, so even a low rate means real
-      #    operations are failing right now.
-      # ------------------------------------------------------------------
-      - alert: KmsBackendFatalErrors
-        expr: |
-          sum by (operation) (rate(rustfs_kms_backend_attempt_failures_total{error_class="fatal"}[5m])) > 0
-        for: 5m
-        labels:
-          severity: critical
-          component: kms
-        annotations:
-          summary: "KMS backend fatal errors on operation {{ $labels.operation }}"
-          description: >-
-            Attempt failures classified as fatal are occurring at
-            {{ $value | printf "%.3f" }}/s on operation
-            {{ $labels.operation }}. Fatal failures are not retried:
-            each one is a KMS backend call that failed permanently
-            (authentication, permissions, malformed request, or a
-            missing key/version).
-          runbook_url: "https://github.com/rustfs/rustfs/blob/main/docs/operations/kms-observability-runbook.md#kmsbackendfatalerrors"
-
-      # ------------------------------------------------------------------
-      # 2. KmsBackendHighErrorRate
-      #    Sustained share of operations terminating without success
-      #    (fatal, budget/deadline exhaustion, admission backpressure,
-      #    or an open circuit). The cancelled outcome is excluded because
-      #    shutdowns legitimately produce it.
-      #    The traffic guard keeps a single failure on a near-idle
-      #    cluster from firing the alert.
-      #    Threshold: 5% for 10m — conservative default, calibrate
-      #    against a staging baseline.
-      # ------------------------------------------------------------------
-      - alert: KmsBackendHighErrorRate
-        expr: |
-          (
-            sum(rate(rustfs_kms_backend_operations_total{outcome!~"success|cancelled"}[5m]))
-            /
-            clamp_min(sum(rate(rustfs_kms_backend_operations_total[5m])), 1e-9)
-          ) > 0.05
-          and
-          sum(rate(rustfs_kms_backend_operations_total[5m])) > 0.02
-        for: 10m
-        labels:
-          severity: critical
-          component: kms
-        annotations:
-          summary: "KMS backend non-success ratio above 5% for 10m"
-          description: >-
-            {{ $value | humanizePercentage }} of KMS backend operations
-            are terminating in fatal, budget_exhausted,
-            deadline_exceeded, backpressure_timeout,
-            backpressure_rejected, or circuit_open. Object encryption
-            and decryption paths depending on the KMS are degraded or
-            failing.
-          runbook_url: "https://github.com/rustfs/rustfs/blob/main/docs/operations/kms-observability-runbook.md#kmsbackendhigherrorrate"
-
-  # ==========================================================================
-  # Warning alerts — investigation needed
-  # ==========================================================================
-  - name: rustfs-kms-warning
-    interval: 30s
-    rules:
-      # ------------------------------------------------------------------
-      # 3. KmsBackendP99LatencyHigh
-      #    p99 wall-clock duration of whole operations (attempts plus
-      #    backoff) is sustained above 2 seconds. Because the histogram
-      #    includes retries, a high p99 usually means the retry policy
-      #    is absorbing backend failures, not that every call is slow.
-      #    Threshold: 2s for 10m — conservative default, calibrate
-      #    against a staging baseline.
-      # ------------------------------------------------------------------
-      - alert: KmsBackendP99LatencyHigh
-        expr: |
-          histogram_quantile(0.99,
-            sum by (le) (rate(rustfs_kms_backend_operation_duration_seconds_bucket[5m]))
-          ) > 2
-        for: 10m
-        labels:
-          severity: warning
-          component: kms
-        annotations:
-          summary: "KMS backend operation p99 latency above 2s for 10m"
-          description: >-
-            The 99th-percentile KMS backend operation duration is
-            {{ $value | humanizeDuration }}, including retries and
-            backoff. Encryption and decryption latency is leaking into
-            S3 request latency.
-          runbook_url: "https://github.com/rustfs/rustfs/blob/main/docs/operations/kms-observability-runbook.md#kmsbackendp99latencyhigh"
-
-      # ------------------------------------------------------------------
-      # 4. KmsBackendAttemptFailureSpike
-      #    Aggregate attempt-failure rate (all error classes) sustained
-      #    above an absolute floor. An absolute threshold is used instead
-      #    of an offset-1d baseline ratio because fresh deployments have
-      #    no baseline and an empty offset vector would keep a ratio
-      #    alert from ever firing; switch to a baseline-relative form
-      #    (see rustfs-get-optimization-alerts.yaml for the pattern)
-      #    once a stable staging baseline exists.
-      #    Threshold: 0.5/s for 10m — conservative default, calibrate
-      #    against a staging baseline.
-      # ------------------------------------------------------------------
-      - alert: KmsBackendAttemptFailureSpike
-        expr: |
-          sum(rate(rustfs_kms_backend_attempt_failures_total[5m])) > 0.5
-        for: 10m
-        labels:
-          severity: warning
-          component: kms
-        annotations:
-          summary: "KMS backend attempt failures above 0.5/s for 10m"
-          description: >-
-            KMS backend attempts are failing at
-            {{ $value | printf "%.2f" }}/s across all error classes.
-            The retry policy may still be masking these from callers —
-            check the error-class breakdown before it stops absorbing
-            them.
-          runbook_url: "https://github.com/rustfs/rustfs/blob/main/docs/operations/kms-observability-runbook.md#kmsbackendattemptfailurespike"
-
-      # ------------------------------------------------------------------
-      # 5. KmsBackendRetryBudgetExhausted
-      #    Operations are running out of retry budget (budget_exhausted)
-      #    or operation deadline (deadline_exceeded). These surface to
-      #    callers as failed KMS operations even though every individual
-      #    failure was retryable — the backend is unhealthy for longer
-      #    than the policy can bridge.
-      #    Threshold: 0.05/s for 10m — conservative default, calibrate
-      #    against a staging baseline.
-      # ------------------------------------------------------------------
-      - alert: KmsBackendRetryBudgetExhausted
-        expr: |
-          sum by (outcome) (rate(rustfs_kms_backend_operations_total{outcome=~"budget_exhausted|deadline_exceeded"}[5m])) > 0.05
-        for: 10m
-        labels:
-          severity: warning
-          component: kms
-        annotations:
-          summary: "KMS backend operations exhausting retry budget ({{ $labels.outcome }})"
-          description: >-
-            KMS backend operations are terminating as
-            {{ $labels.outcome }} at {{ $value | printf "%.3f" }}/s.
-            Retryable failures are outlasting the retry budget, so
-            callers are seeing hard failures.
-          runbook_url: "https://github.com/rustfs/rustfs/blob/main/docs/operations/kms-observability-runbook.md#kmsbackendretrybudgetexhausted"
-
-      # ------------------------------------------------------------------
-      # 6. KmsBackendCircuitOpen
-      #    Direct circuit-state signal, independent of operation traffic.
-      #    A transient open can recover on its first half-open probe; alert
-      #    only when the circuit remains open or half-open for one minute.
-      # ------------------------------------------------------------------
-      - alert: KmsBackendCircuitOpen
-        expr: |
-          rustfs_kms_backend_circuit_open > 0
-        for: 1m
-        labels:
-          severity: warning
-          component: kms
-        annotations:
-          summary: "KMS backend circuit open ({{ $labels.backend }}/{{ $labels.scope }})"
-          description: >-
-            The KMS backend circuit for {{ $labels.backend }} scope
-            {{ $labels.scope }} has remained open or half-open for one
-            minute. Operations in this scope can terminate as
-            circuit_open until the half-open probe succeeds or returns
-            a non-retryable failure.
-          runbook_url: "https://github.com/rustfs/rustfs/blob/main/docs/operations/kms-observability-runbook.md#kmsbackendcircuitopen"
-
-      # ------------------------------------------------------------------
-      # 7. KmsKeyRotationOverdue
-      #    The least recently rotated usable key has gone more than 400
-      #    days without a rotation (measured from creation for keys with
-      #    no recorded rotation). Direct gauge state published by the
-      #    deletion worker's sweep, so no traffic guard applies; the
-      #    one-hour hold only bridges scrape gaps. The worker runs only
-      #    on backends with the schedule_deletion capability, so on the
-      #    Static backend the series never exists and this alert cannot
-      #    fire — that backend cannot rotate either; see the rotation
-      #    driver matrix in docs/operations/kms-backend-security.md.
-      #    Threshold: 400 days — conservative default sitting above a
-      #    one-year rotation policy. Align it with the rotation period
-      #    your compliance policy requires, and with
-      #    RUSTFS_KMS_ROTATION_MAX_AGE_SECS so the per-key rotation_due
-      #    verdict and this aggregate alert agree.
-      # ------------------------------------------------------------------
-      - alert: KmsKeyRotationOverdue
-        expr: |
-          rustfs_kms_oldest_key_rotation_age_seconds > (400 * 86400)
-        for: 1h
-        labels:
-          severity: warning
-          component: kms
-        annotations:
-          summary: "Oldest KMS key unrotated for more than 400 days"
-          description: >-
-            The least recently rotated usable KMS key was last rotated
-            {{ $value | humanizeDuration }} ago (measured from creation
-            for keys with no recorded rotation). List keys through the
-            admin API and read rotation_due / rotation_due_reason for
-            the per-key verdict; an "unsupported" reason means the
-            backend cannot rotate at all.
-          runbook_url: "https://github.com/rustfs/rustfs/blob/main/docs/operations/kms-observability-runbook.md#kmskeyrotationoverdue"

.docker/observability/prometheus.yml

@@ -13,72 +13,16 @@
 # limitations under the License.
 
 global:
-  scrape_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
-  evaluation_interval: 15s
-  external_labels:
-    cluster: 'rustfs-dev'  # Label to identify the cluster
-    replica: '1'  # Replica identifier
-
-rule_files:
-  - /etc/prometheus/rules/*.yml
+  scrape_interval: 5s  # 刮取间隔
 
 scrape_configs:
   - job_name: 'otel-collector'
     static_configs:
-      - targets: [ 'otel-collector:8888' ]  # Scrape metrics from Collector
-    scrape_interval: 10s
-
-  - job_name: 'rustfs-app-metrics'
+      - targets: [ 'otel-collector:8888' ]  # 从 Collector 刮取指标
+  - job_name: 'otel-metrics'
     static_configs:
-      - targets: [ 'otel-collector:8889' ]  # Application indicators
-    scrape_interval: 15s
-    metric_relabel_configs:
-      - source_labels: [ __name__ ]
-        regex: 'go_.*'
-        action: drop  # Drop Go runtime metrics if not needed
-
+      - targets: [ 'otel-collector:8889' ]  # 应用指标
   - job_name: 'tempo'
     static_configs:
-      - targets: [ 'tempo:3200' ]  # Scrape metrics from Tempo
-
-  - job_name: 'jaeger'
-    static_configs:
-      - targets: [ 'jaeger:14269' ]  # Jaeger admin port (14269 is standard for admin/metrics)
-
-  - job_name: 'loki'
-    static_configs:
-      - targets: [ 'loki:3100' ]
-
-  - job_name: 'prometheus'
-    static_configs:
-      - targets: [ 'localhost:9090' ]
-
-  - job_name: 'vulture'
-    static_configs:
-      - targets:
-          - 'vulture:8080'
-
-otlp:
-  promote_resource_attributes:
-    - service.instance.id
-    - service.name
-    - service.namespace
-    - cloud.availability_zone
-    - cloud.region
-    - container.name
-    - deployment.environment.name
-    - k8s.cluster.name
-    - k8s.container.name
-    - k8s.cronjob.name
-    - k8s.daemonset.name
-    - k8s.deployment.name
-    - k8s.job.name
-    - k8s.namespace.name
-    - k8s.pod.name
-    - k8s.replicaset.name
-    - k8s.statefulset.name
-  translation_strategy: NoUTF8EscapingWithSuffixes
-
-storage:
-  tsdb:
-    out_of_order_time_window: 30m
+      - targets: [ 'tempo:3200' ]
+      

.docker/observability/prometheus.yml.bak-issue2007-node-exporter-20260823T172248Z

@@ -1,84 +0,0 @@
-# Copyright 2024 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-global:
-  scrape_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
-  evaluation_interval: 15s
-  external_labels:
-    cluster: 'rustfs-dev'  # Label to identify the cluster
-    replica: '1'  # Replica identifier
-
-rule_files:
-  - /etc/prometheus/rules/*.yml
-
-scrape_configs:
-  - job_name: 'otel-collector'
-    static_configs:
-      - targets: [ 'otel-collector:8888' ]  # Scrape metrics from Collector
-    scrape_interval: 10s
-
-  - job_name: 'rustfs-app-metrics'
-    static_configs:
-      - targets: [ 'otel-collector:8889' ]  # Application indicators
-    scrape_interval: 15s
-    metric_relabel_configs:
-      - source_labels: [ __name__ ]
-        regex: 'go_.*'
-        action: drop  # Drop Go runtime metrics if not needed
-
-  - job_name: 'tempo'
-    static_configs:
-      - targets: [ 'tempo:3200' ]  # Scrape metrics from Tempo
-
-  - job_name: 'jaeger'
-    static_configs:
-      - targets: [ 'jaeger:14269' ]  # Jaeger admin port (14269 is standard for admin/metrics)
-
-  - job_name: 'loki'
-    static_configs:
-      - targets: [ 'loki:3100' ]
-
-  - job_name: 'prometheus'
-    static_configs:
-      - targets: [ 'localhost:9090' ]
-
-  - job_name: 'vulture'
-    static_configs:
-      - targets:
-          - 'vulture:8080'
-
-otlp:
-  promote_resource_attributes:
-    - service.instance.id
-    - service.name
-    - service.namespace
-    - cloud.availability_zone
-    - cloud.region
-    - container.name
-    - deployment.environment.name
-    - k8s.cluster.name
-    - k8s.container.name
-    - k8s.cronjob.name
-    - k8s.daemonset.name
-    - k8s.deployment.name
-    - k8s.job.name
-    - k8s.namespace.name
-    - k8s.pod.name
-    - k8s.replicaset.name
-    - k8s.statefulset.name
-  translation_strategy: NoUTF8EscapingWithSuffixes
-
-storage:
-  tsdb:
-    out_of_order_time_window: 30m

.docker/observability/redpanda-console.yaml

@@ -1,3 +0,0 @@
-kafka:
-  brokers:
-    - redpanda:9092

.docker/observability/tempo-data/.gitignore

@@ -0,0 +1 @@
+*

.docker/observability/tempo-ha.yaml

@@ -1,286 +0,0 @@
-# Copyright 2024 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-# High Availability Tempo Configuration for docker-compose-example-for-rustfs.yml
-# Features:
-# - Distributed architecture with multiple components
-# - Kafka-based ingestion for fault tolerance
-# - Replication factor of 3 for data resilience
-# - Query frontend for load balancing
-# - Metrics generation from traces
-# - WAL for durability
-
-partition_ring_live_store: true
-stream_over_http_enabled: true
-
-server:
-  http_listen_port: 3200
-  http_server_read_timeout: 30s
-  http_server_write_timeout: 30s
-  grpc_server_max_recv_msg_size: 4194304  # 4MB
-  grpc_server_max_send_msg_size: 4194304
-  log_level: info
-  log_format: json
-
-# Memberlist configuration for distributed mode
-memberlist:
-  node_name: tempo
-  bind_port: 7946
-  join_members:
-    - tempo:7946
-  retransmit_factor: 4
-  node_timeout: 15s
-  retransmit_interval: 300ms
-  dead_node_reclaim_time: 30s
-
-# Distributor configuration - receives traces and routes to ingesters
-distributor:
-  ingester_write_path_enabled: true
-  kafka_write_path_enabled: true
-  rate_limit_bytes: 10MB
-  rate_limit_enabled: true
-  receivers:
-    otlp:
-      protocols:
-        grpc:
-          endpoint: "0.0.0.0:4317"
-          max_concurrent_streams: 0
-          max_receive_message_size: 4194304
-        http:
-          endpoint: "0.0.0.0:4318"
-          cors:
-            allowed_origins:
-              - "*"
-            max_age: 86400
-    jaeger:
-      protocols:
-        grpc:
-          endpoint: "0.0.0.0:14250"
-        thrift_http:
-          endpoint: "0.0.0.0:14268"
-    zipkin:
-      endpoint: "0.0.0.0:9411"
-  ring:
-    kvstore:
-      store: memberlist
-    heartbeat_timeout: 5s
-    replication_factor: 3
-    heartbeat_interval: 5s
-
-# Ingester configuration - stores traces and querying
-ingester:
-  lifecycler:
-    address: tempo
-    ring:
-      kvstore:
-        store: memberlist
-      replication_factor: 3
-      max_cache_freshness_per_sec: 10s
-      heartbeat_interval: 5s
-      heartbeat_timeout: 5s
-    num_tokens: 128
-    tokens_file_path: /var/tempo/tokens.json
-    claim_on_rollout: true
-  trace_idle_period: 20s
-  max_block_bytes: 10_000_000
-  max_block_duration: 10m
-  chunk_size_bytes: 1_000_000
-  chunk_encoding: snappy
-  wal:
-    checkpoint_duration: 5s
-    max_wal_blocks: 4
-  metrics:
-    enabled: true
-    level: block
-    target_info_duration: 15m
-
-# WAL configuration for data durability
-wal:
-  checkpoint_duration: 5s
-  flush_on_shutdown: true
-  path: /var/tempo/wal
-
-# Kafka ingestion configuration - for high throughput scenarios
-ingest:
-  enabled: true
-  kafka:
-    brokers: [ redpanda:9092 ]
-    topic: tempo-ingest
-    encoding: protobuf
-    consumer_group: tempo-ingest-consumer
-    session_timeout: 10s
-    rebalance_timeout: 1m
-    partition: auto
-    verbosity: 2
-
-# Query frontend configuration - distributed querying
-query_frontend:
-  compression: gzip
-  downstream_url: http://localhost:3200
-  log_queries_longer_than: 5s
-  cache_uncompressed_bytes: 100MB
-  max_outstanding_requests_per_tenant: 100
-  max_query_length: 48h
-  max_query_lookback: 30d
-  default_result_cache_ttl: 1m
-  result_cache:
-    cache:
-      enable_fifocache: true
-      default_validity: 1m
-  rf1_after: "1999-01-01T00:00:00Z"
-  mcp_server:
-    enabled: true
-
-# Querier configuration - queries traces
-querier:
-  frontend_worker:
-    frontend_address: localhost:3200
-    grpc_client_config:
-      max_recv_msg_size: 104857600
-  max_concurrent_queries: 20
-  max_metric_bytes_per_trace: 1MB
-
-# Query scheduler configuration - for distributed querying
-query_scheduler:
-  use_scheduler_ring: false
-
-# Metrics generator configuration - generates metrics from traces
-metrics_generator:
-  enabled: true
-  registry:
-    enabled: true
-    external_labels:
-      source: tempo
-      cluster: rustfs-docker-ha
-      environment: production
-  storage:
-    path: /var/tempo/generator/wal
-    remote_write:
-      - url: http://prometheus:9090/api/v1/write
-        send_exemplars: true
-        resource_to_telemetry_conversion:
-          enabled: true
-  processor:
-    batch:
-      timeout: 10s
-      send_batch_size: 1024
-    memory_limiter:
-      check_interval: 5s
-      limit_mib: 512
-      spike_limit_mib: 128
-  processors:
-    - span-metrics
-    - local-blocks
-    - service-graphs
-  generate_native_histograms: both
-
-# Backend worker configuration
-backend_worker:
-  backend_scheduler_addr: localhost:3200
-  compaction:
-    block_retention: 24h
-    compacted_block_retention: 1h
-  ring:
-    kvstore:
-      store: memberlist
-
-# Backend scheduler configuration
-backend_scheduler:
-  enabled: true
-  provider:
-    compaction:
-      compaction:
-        block_retention: 24h
-        compacted_block_retention: 1h
-        concurrency: 25
-        v2_out_path: /var/tempo/blocks/compaction
-
-# Storage configuration - local backend with proper retention
-storage:
-  trace:
-    backend: local
-    wal:
-      path: /var/tempo/wal
-      checkpoint_duration: 5s
-      flush_on_shutdown: true
-    local:
-      path: /var/tempo/blocks
-      bloom_filter_false_positive: 0.05
-      bloom_shift: 4
-      index:
-        downsample_bytes: 1000000
-        page_size_bytes: 0
-        cache_size_bytes: 0
-    pool:
-      max_workers: 400
-      queue_depth: 10000
-
-# Compactor configuration - manages block compaction
-compactor:
-  compaction:
-    block_retention: 168h  # 7 days
-    compacted_block_retention: 1h
-    concurrency: 25
-    v2_out_path: /var/tempo/blocks/compaction
-    shard_count: 32
-    max_block_bytes: 107374182400  # 100GB
-    max_compaction_objects: 6000000
-    max_time_per_tenant: 5m
-    block_size_bytes: 107374182400
-  ring:
-    kvstore:
-      store: memberlist
-    heartbeat_interval: 5s
-    heartbeat_timeout: 5s
-
-# Limits configuration - rate limiting and quotas
-limits:
-  max_traces_per_user: 10000
-  max_bytes_per_trace: 10485760  # 10MB
-  max_search_bytes_per_trace: 0
-  forgiving_oversize_traces: true
-  rate_limit_bytes: 10MB
-  rate_limit_enabled: true
-  ingestion_burst_size_bytes: 20MB
-  ingestion_rate_limit_bytes: 10MB
-  max_bytes_per_second: 10485760
-  metrics_generator_max_active_series: 10000
-  metrics_generator_max_churned_series: 10000
-  metrics_generator_forta_out_of_order_ttl: 5m
-
-# Override configuration
-overrides:
-  defaults:
-    metrics_generator:
-      processors:
-        - span-metrics
-        - local-blocks
-        - service-graphs
-      generate_native_histograms: both
-      max_active_series: 10000
-      max_churned_series: 10000
-
-# Usage reporting configuration
-usage_report:
-  reporting_enabled: false
-
-# Tracing configuration for debugging
-tracing:
-  enabled: true
-  jaeger:
-    sampler:
-      name: probabilistic
-      param: 0.1
-    reporter_log_spans: false
-

.docker/observability/tempo.yaml

@@ -1,72 +1,55 @@
-# Copyright 2024 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
 stream_over_http_enabled: true
-
 server:
   http_listen_port: 3200
   log_level: info
 
-memberlist:
-  node_name: tempo
-  bind_port: 7946
-  join_members:
-    - tempo:7946
+query_frontend:
+  search:
+    duration_slo: 5s
+    throughput_bytes_slo: 1.073741824e+09
+    metadata_slo:
+      duration_slo: 5s
+      throughput_bytes_slo: 1.073741824e+09
+  trace_by_id:
+    duration_slo: 5s
 
 distributor:
   receivers:
     otlp:
       protocols:
         grpc:
-          endpoint: "0.0.0.0:4317"
-        http:
-          endpoint: "0.0.0.0:4318"
+          endpoint: "tempo:4317"
 
 ingester:
-  max_block_duration: 5m
+  max_block_duration: 5m # cut the headblock when this much time passes. this is being set for demo purposes and should probably be left alone normally
+
+compactor:
+  compaction:
+    block_retention: 1h # overall Tempo trace retention. set for demo purposes
 
 metrics_generator:
   registry:
     external_labels:
       source: tempo
       cluster: docker-compose
-  traces_storage:
-    path: /var/tempo/generator/traces
   storage:
     path: /var/tempo/generator/wal
     remote_write:
       - url: http://prometheus:9090/api/v1/write
         send_exemplars: true
-
-query_frontend:
-  rf1_after: "1999-01-01T00:00:00Z"
-  mcp_server:
-    enabled: true
+  traces_storage:
+    path: /var/tempo/generator/traces
 
 storage:
   trace:
-    backend: local
+    backend: local # backend configuration to use
     wal:
       path: /var/tempo/wal # where to store the wal locally
     local:
-      path: /var/tempo/blocks # where to store the traces locally
+      path: /var/tempo/blocks
 
 overrides:
   defaults:
     metrics_generator:
-      processors: [ "span-metrics", "service-graphs", "local-blocks" ]
-      generate_native_histograms: both
-
-usage_report:
-  reporting_enabled: false
+      processors: [ service-graphs, span-metrics, local-blocks ] # enables metrics generator
+      generate_native_histograms: both

.docker/openobserve-otel/README.md

@@ -5,57 +5,71 @@
 
 English | [中文](README_ZH.md)
 
-This directory contains the configuration for an **alternative** observability stack using OpenObserve.
+This directory contains the configuration files for setting up an observability stack with OpenObserve and OpenTelemetry
+Collector.
 
-## ⚠️ Note
+### Overview
 
-For the **recommended** observability stack (Prometheus, Grafana, Tempo, Loki), please see `../observability/`.
+This setup provides a complete observability solution for your applications:
 
-## 🌟 Overview
+- **OpenObserve**: A modern, open-source observability platform for logs, metrics, and traces.
+- **OpenTelemetry Collector**: Collects and processes telemetry data before sending it to OpenObserve.
 
-OpenObserve is a lightweight, all-in-one observability platform that handles logs, metrics, and traces in a single binary. This setup is ideal for:
-- Resource-constrained environments.
-- Quick setup and testing.
-- Users who prefer a unified UI.
+### Setup Instructions
 
-## 🚀 Quick Start
+1. **Prerequisites**:
+    - Docker and Docker Compose installed
+    - Sufficient memory resources (minimum 2GB recommended)
 
-### 1. Start Services
+2. **Starting the Services**:
+   ```bash
+   cd .docker/openobserve-otel
+   docker compose -f docker-compose.yml  up -d
+   ```
+
+3. **Accessing the Dashboard**:
+    - OpenObserve UI: http://localhost:5080
+    - Default credentials:
+        - Username: root@rustfs.com
+        - Password: rustfs123
+
+### Configuration
+
+#### OpenObserve Configuration
+
+The OpenObserve service is configured with:
+
+- Root user credentials
+- Data persistence through a volume mount
+- Memory cache enabled
+- Health checks
+- Exposed ports:
+    - 5080: HTTP API and UI
+    - 5081: OTLP gRPC
+
+#### OpenTelemetry Collector Configuration
+
+The collector is configured to:
+
+- Receive telemetry data via OTLP (HTTP and gRPC)
+- Collect logs from files
+- Process data in batches
+- Export data to OpenObserve
+- Manage memory usage
+
+### Integration with Your Application
+
+To send telemetry data from your application, configure your OpenTelemetry SDK to send data to:
+
+- OTLP gRPC: `localhost:4317`
+- OTLP HTTP: `localhost:4318`
+
+For example, in a Rust application using the `rustfs-obs` library:
 
 ```bash
-cd .docker/openobserve-otel
-docker compose up -d
+export RUSTFS_OBS_ENDPOINT=http://localhost:4317
+export RUSTFS_OBS_SERVICE_NAME=yourservice
+export RUSTFS_OBS_SERVICE_VERSION=1.0.0
+export RUSTFS_OBS_ENVIRONMENT=development
 ```
 
-### 2. Access Dashboard
-
-- **URL**: [http://localhost:5080](http://localhost:5080)
-- **Username**: `root@rustfs.com`
-- **Password**: `rustfs123`
-
-## 🛠️ Configuration
-
-### OpenObserve
-
-- **Persistence**: Data is persisted to a Docker volume.
-- **Ports**:
-    - `5080`: HTTP API and UI
-    - `5081`: OTLP gRPC
-
-### OpenTelemetry Collector
-
-- **Receivers**: OTLP (gRPC `4317`, HTTP `4318`)
-- **Exporters**: Sends data to OpenObserve.
-
-## 🔗 Integration
-
-Configure your application to send OTLP data to the collector:
-
-- **Endpoint**: `http://localhost:4318` (HTTP) or `localhost:4317` (gRPC)
-
-Example for RustFS:
-
-```bash
-export RUSTFS_OBS_ENDPOINT=http://localhost:4318
-export RUSTFS_OBS_SERVICE_NAME=rustfs-node-1
-```

.docker/openobserve-otel/README_ZH.md

@@ -5,57 +5,71 @@
 
 [English](README.md) | 中文
 
-本目录包含使用 OpenObserve 的**替代**可观测性技术栈配置。
+## 中文
 
-## ⚠️ 注意
+本目录包含搭建 OpenObserve 和 OpenTelemetry Collector 可观测性栈的配置文件。
 
-对于**推荐**的可观测性技术栈（Prometheus, Grafana, Tempo, Loki），请参阅 `../observability/`。
+### 概述
 
-## 🌟 概览
+此设置为应用程序提供了完整的可观测性解决方案：
 
-OpenObserve 是一个轻量级、一体化的可观测性平台，在一个二进制文件中处理日志、指标和追踪。此设置非常适合：
-- 资源受限的环境。
-- 快速设置和测试。
-- 喜欢统一 UI 的用户。
+- **OpenObserve**：现代化、开源的可观测性平台，用于日志、指标和追踪。
+- **OpenTelemetry Collector**：收集和处理遥测数据，然后将其发送到 OpenObserve。
 
-## 🚀 快速开始
+### 设置说明
 
-### 1. 启动服务
+1. **前提条件**：
+    - 已安装 Docker 和 Docker Compose
+    - 足够的内存资源（建议至少 2GB）
+
+2. **启动服务**：
+   ```bash
+   cd .docker/openobserve-otel
+   docker compose -f docker-compose.yml  up -d
+   ```
+
+3. **访问仪表板**：
+    - OpenObserve UI：http://localhost:5080
+    - 默认凭据：
+        - 用户名：root@rustfs.com
+        - 密码：rustfs123
+
+### 配置
+
+#### OpenObserve 配置
+
+OpenObserve 服务配置：
+
+- 根用户凭据
+- 通过卷挂载实现数据持久化
+- 启用内存缓存
+- 健康检查
+- 暴露端口：
+    - 5080：HTTP API 和 UI
+    - 5081：OTLP gRPC
+
+#### OpenTelemetry Collector 配置
+
+收集器配置为：
+
+- 通过 OTLP（HTTP 和 gRPC）接收遥测数据
+- 从文件中收集日志
+- 批处理数据
+- 将数据导出到 OpenObserve
+- 管理内存使用
+
+### 与应用程序集成
+
+要从应用程序发送遥测数据，将 OpenTelemetry SDK 配置为发送数据到：
+
+- OTLP gRPC:`localhost:4317`
+- OTLP HTTP:`localhost:4318`
+
+例如，在使用 `rustfs-obs` 库的 Rust 应用程序中：
 
 ```bash
-cd .docker/openobserve-otel
-docker compose up -d
-```
-
-### 2. 访问仪表盘
-
-- **URL**: [http://localhost:5080](http://localhost:5080)
-- **用户名**: `root@rustfs.com`
-- **密码**: `rustfs123`
-
-## 🛠️ 配置
-
-### OpenObserve
-
-- **持久化**: 数据持久化到 Docker 卷。
-- **端口**:
-    - `5080`: HTTP API 和 UI
-    - `5081`: OTLP gRPC
-
-### OpenTelemetry Collector
-
-- **接收器**: OTLP (gRPC `4317`, HTTP `4318`)
-- **导出器**: 将数据发送到 OpenObserve。
-
-## 🔗 集成
-
-配置您的应用程序将 OTLP 数据发送到收集器：
-
-- **端点**: `http://localhost:4318` (HTTP) 或 `localhost:4317` (gRPC)
-
-RustFS 示例：
-
-```bash
-export RUSTFS_OBS_ENDPOINT=http://localhost:4318
-export RUSTFS_OBS_SERVICE_NAME=rustfs-node-1
-```
+export RUSTFS_OBS_ENDPOINT=http://localhost:4317
+export RUSTFS_OBS_SERVICE_NAME=yourservice
+export RUSTFS_OBS_SERVICE_VERSION=1.0.0
+export RUSTFS_OBS_ENVIRONMENT=development
+```

.docker/test/compat/README.md

@@ -1,91 +0,0 @@
-# Rio compatibility compose files
-
-These compose files prepare 4-node, 4-disk clusters for rio/rio-v2 storage format compatibility checks. All disks are bind-mounted under `.docker/compat/data` so the on-disk files remain available on the host.
-
-## Clusters
-
-```bash
-docker compose -f .docker/compat/docker-compose.rustfs-beta5.yml up -d --build
-docker compose -f .docker/compat/docker-compose.minio.yml up -d
-docker compose -f .docker/compat/docker-compose.rustfs-rio-v2.yml up -d --build
-```
-
-Default API endpoints:
-
-- RustFS `1.0.0-beta.5`: `http://127.0.0.1:9100`
-- MinIO: `http://127.0.0.1:9200`
-- current main with `rio-v2`: `http://127.0.0.1:9300`
-
-## Reading old datasets with rio-v2
-
-Stop the writer cluster before mounting its disks into the rio-v2 cluster.
-
-```bash
-docker compose -f .docker/compat/docker-compose.rustfs-beta5.yml down
-RUSTFS_RIO_V2_DATASET=./data/rustfs-beta5 \
-  docker compose -f .docker/compat/docker-compose.rustfs-rio-v2.yml up -d --build
-
-docker compose -f .docker/compat/docker-compose.minio.yml down
-RUSTFS_RIO_V2_DATASET=./data/minio \
-  docker compose -f .docker/compat/docker-compose.rustfs-rio-v2.yml up -d --build
-```
-
-## 200G object mix
-
-Use the same bucket/object matrix against the beta5 and MinIO endpoints, then read it back through the rio-v2 endpoint. A practical 200G mix is:
-
-- 1 KiB x 1024
-- 1 MiB x 1024
-- 64 MiB x 512
-- 1 GiB x 64
-- 8 GiB x 12
-- 6 GiB x 1
-
-Compression is enabled by default for RustFS and MinIO. Server-side KMS/SSE settings are intentionally left to environment variables or mounted key directories so real key material is not committed. For SSE-C cases, run the clusters with TLS because MinIO requires HTTPS for SSE-C.
-
-## High-concurrency write/read stress
-
-Use `run_rw_compat_stress.sh` to generate a manifest on an old endpoint, then verify the same objects through the rio-v2 endpoint after mounting the old disks.
-
-```bash
-.docker/compat/run_rw_compat_stress.sh \
-  --mode write \
-  --endpoint http://127.0.0.1:9100 \
-  --access-key rustfsadmin \
-  --secret-key rustfsadmin \
-  --bucket compat-beta5 \
-  --concurrency 96
-
-RUSTFS_RIO_V2_DATASET=./data/rustfs-beta5 \
-  docker compose -f .docker/compat/docker-compose.rustfs-rio-v2.yml up -d --build
-
-.docker/compat/run_rw_compat_stress.sh \
-  --mode verify \
-  --endpoint http://127.0.0.1:9300 \
-  --access-key rustfsadmin \
-  --secret-key rustfsadmin \
-  --bucket compat-beta5 \
-  --concurrency 96 \
-  --manifest target/compat/rw-stress-YYYYmmdd-HHMMSS/manifest.csv
-```
-
-For encrypted datasets, add `--encryption sse-s3`, `--encryption sse-kms --sse-kms-key-id <key-id>`, or `--encryption sse-c --sse-c-key-file <raw-32-byte-key-file>` to both the write and verify commands.
-
-## 5 GiB encrypted compatibility run
-
-The `5g` profile covers 1 KiB, 1 MiB, 16 MiB, 64 MiB, and 1 GiB objects and totals exactly 5 GiB. Generate `compat-key.key` under `.docker/compat/kms/rustfs-compat`, enable local KMS on both RustFS clusters, then use the same encryption arguments while writing with beta5 and verifying with rio-v2. Set non-default local test credentials first because distributed listeners reject the built-in default credentials.
-
-```bash
-export COMPAT_ACCESS_KEY='<non-default-access-key>'
-export COMPAT_SECRET_KEY='<non-default-secret-key>'
-
-RUSTFS_ACCESS_KEY="$COMPAT_ACCESS_KEY" RUSTFS_SECRET_KEY="$COMPAT_SECRET_KEY" \
-  RUSTFS_KMS_ENABLE=true RUSTFS_KMS_ALLOW_INSECURE_DEV_DEFAULTS=true \
-  docker compose -f .docker/compat/docker-compose.rustfs-beta5.yml up -d
-
-.docker/compat/run_rw_compat_stress.sh \
-  --mode write --endpoint http://127.0.0.1:9100 \
-  --access-key "$COMPAT_ACCESS_KEY" --secret-key "$COMPAT_SECRET_KEY" \
-  --bucket compat-beta5-sse-s3 --profile 5g --concurrency 16 \
-  --encryption sse-s3
-```

.docker/test/compat/docker-compose.minio.yml

@@ -1,88 +0,0 @@
-# Copyright 2024 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0.
-
-x-minio-env: &minio-env
-  MINIO_ROOT_USER: "${MINIO_ROOT_USER:-minioadmin}"
-  MINIO_ROOT_PASSWORD: "${MINIO_ROOT_PASSWORD:-minioadmin}"
-  MINIO_COMPRESSION_ENABLE: "${MINIO_COMPRESSION_ENABLE:-on}"
-  MINIO_COMPRESSION_ALLOW_ENCRYPTION: "${MINIO_COMPRESSION_ALLOW_ENCRYPTION:-on}"
-  MINIO_COMPRESSION_EXTENSIONS: "${MINIO_COMPRESSION_EXTENSIONS:-.txt,.log,.csv,.json,.tar,.xml,.bin}"
-  MINIO_COMPRESSION_MIME_TYPES: "${MINIO_COMPRESSION_MIME_TYPES:-text/*,application/json,application/xml,binary/octet-stream}"
-
-x-minio-node: &minio-node
-  image: "${MINIO_IMAGE:-quay.io/minio/minio:latest}"
-  command: server --console-address ":9001" "http://minio{1...4}:9000/data/disk{1...4}"
-  environment: *minio-env
-  networks:
-    - minio-compat-net
-  restart: unless-stopped
-
-services:
-  minio-permission-helper:
-    image: alpine:3.23
-    command: sh -c "mkdir -p /compat-data && chown -R 1000:1000 /compat-data"
-    volumes:
-      - ./data/minio:/compat-data
-    restart: "no"
-
-  minio1:
-    <<: *minio-node
-    hostname: minio1
-    depends_on:
-      minio-permission-helper:
-        condition: service_completed_successfully
-    volumes:
-      - ./data/minio/node1/disk1:/data/disk1
-      - ./data/minio/node1/disk2:/data/disk2
-      - ./data/minio/node1/disk3:/data/disk3
-      - ./data/minio/node1/disk4:/data/disk4
-    ports:
-      - "${MINIO_API_PORT:-9200}:9000"
-      - "${MINIO_CONSOLE_PORT:-9201}:9001"
-
-  minio2:
-    <<: *minio-node
-    hostname: minio2
-    depends_on:
-      minio-permission-helper:
-        condition: service_completed_successfully
-    volumes:
-      - ./data/minio/node2/disk1:/data/disk1
-      - ./data/minio/node2/disk2:/data/disk2
-      - ./data/minio/node2/disk3:/data/disk3
-      - ./data/minio/node2/disk4:/data/disk4
-    ports:
-      - "${MINIO_NODE2_PORT:-9202}:9000"
-
-  minio3:
-    <<: *minio-node
-    hostname: minio3
-    depends_on:
-      minio-permission-helper:
-        condition: service_completed_successfully
-    volumes:
-      - ./data/minio/node3/disk1:/data/disk1
-      - ./data/minio/node3/disk2:/data/disk2
-      - ./data/minio/node3/disk3:/data/disk3
-      - ./data/minio/node3/disk4:/data/disk4
-    ports:
-      - "${MINIO_NODE3_PORT:-9203}:9000"
-
-  minio4:
-    <<: *minio-node
-    hostname: minio4
-    depends_on:
-      minio-permission-helper:
-        condition: service_completed_successfully
-    volumes:
-      - ./data/minio/node4/disk1:/data/disk1
-      - ./data/minio/node4/disk2:/data/disk2
-      - ./data/minio/node4/disk3:/data/disk3
-      - ./data/minio/node4/disk4:/data/disk4
-    ports:
-      - "${MINIO_NODE4_PORT:-9204}:9000"
-
-networks:
-  minio-compat-net:
-    driver: bridge

.docker/test/compat/docker-compose.rustfs-beta5.yml

@@ -1,104 +0,0 @@
-# Copyright 2024 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0.
-
-x-rustfs-env: &rustfs-env
-  RUSTFS_VOLUMES: "http://rustfs-beta5-node{1...4}:9000/data/disk{1...4}"
-  RUSTFS_ADDRESS: ":9000"
-  RUSTFS_CONSOLE_ADDRESS: ":9001"
-  RUSTFS_CONSOLE_ENABLE: "true"
-  RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS: "*"
-  RUSTFS_ACCESS_KEY: "${RUSTFS_ACCESS_KEY:-rustfsadmin}"
-  RUSTFS_SECRET_KEY: "${RUSTFS_SECRET_KEY:-rustfsadmin}"
-  RUSTFS_OBS_LOGGER_LEVEL: "${RUSTFS_OBS_LOGGER_LEVEL:-info}"
-  RUSTFS_OBS_LOG_DIRECTORY: "/logs"
-  RUSTFS_COMPRESSION_ENABLED: "${RUSTFS_COMPRESSION_ENABLED:-true}"
-  RUSTFS_COMPRESSION_EXTENSIONS: "${RUSTFS_COMPRESSION_EXTENSIONS:-.txt,.log,.csv,.json,.tar,.xml,.bin}"
-  RUSTFS_COMPRESSION_MIME_TYPES: "${RUSTFS_COMPRESSION_MIME_TYPES:-text/*,application/json,application/xml,binary/octet-stream}"
-  RUSTFS_KMS_ENABLE: "${RUSTFS_KMS_ENABLE:-false}"
-  RUSTFS_KMS_BACKEND: "${RUSTFS_KMS_BACKEND:-local}"
-  RUSTFS_KMS_KEY_DIR: "${RUSTFS_KMS_KEY_DIR:-/kms}"
-  RUSTFS_KMS_DEFAULT_KEY_ID: "${RUSTFS_KMS_DEFAULT_KEY_ID:-compat-key}"
-  RUSTFS_KMS_ALLOW_INSECURE_DEV_DEFAULTS: "${RUSTFS_KMS_ALLOW_INSECURE_DEV_DEFAULTS:-false}"
-  RUSTFS_UNSAFE_BYPASS_DISK_CHECK: "${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}"
-
-x-rustfs-node: &rustfs-node
-  image: "${RUSTFS_BETA5_IMAGE:-rustfs/rustfs:1.0.0-beta.5}"
-  build:
-    context: ../..
-    dockerfile: Dockerfile
-    args:
-      RELEASE: "1.0.0-beta.5"
-  environment: *rustfs-env
-  depends_on:
-    rustfs-beta5-permission-helper:
-      condition: service_completed_successfully
-  networks:
-    - rustfs-beta5-net
-  restart: unless-stopped
-
-services:
-  rustfs-beta5-permission-helper:
-    image: alpine:3.23
-    command: sh -c "mkdir -p /compat-data /kms && chown -R 10001:10001 /compat-data /kms"
-    volumes:
-      - ./data/rustfs-beta5:/compat-data
-      - ./kms/rustfs-compat:/kms
-    restart: "no"
-
-  rustfs-beta5-node1:
-    <<: *rustfs-node
-    hostname: rustfs-beta5-node1
-    volumes:
-      - ./data/rustfs-beta5/node1/disk1:/data/disk1
-      - ./data/rustfs-beta5/node1/disk2:/data/disk2
-      - ./data/rustfs-beta5/node1/disk3:/data/disk3
-      - ./data/rustfs-beta5/node1/disk4:/data/disk4
-      - ./data/rustfs-beta5/logs/node1:/logs
-      - ./kms/rustfs-compat:/kms
-    ports:
-      - "${RUSTFS_BETA5_API_PORT:-9100}:9000"
-      - "${RUSTFS_BETA5_CONSOLE_PORT:-9101}:9001"
-
-  rustfs-beta5-node2:
-    <<: *rustfs-node
-    hostname: rustfs-beta5-node2
-    volumes:
-      - ./data/rustfs-beta5/node2/disk1:/data/disk1
-      - ./data/rustfs-beta5/node2/disk2:/data/disk2
-      - ./data/rustfs-beta5/node2/disk3:/data/disk3
-      - ./data/rustfs-beta5/node2/disk4:/data/disk4
-      - ./data/rustfs-beta5/logs/node2:/logs
-      - ./kms/rustfs-compat:/kms
-    ports:
-      - "${RUSTFS_BETA5_NODE2_PORT:-9102}:9000"
-
-  rustfs-beta5-node3:
-    <<: *rustfs-node
-    hostname: rustfs-beta5-node3
-    volumes:
-      - ./data/rustfs-beta5/node3/disk1:/data/disk1
-      - ./data/rustfs-beta5/node3/disk2:/data/disk2
-      - ./data/rustfs-beta5/node3/disk3:/data/disk3
-      - ./data/rustfs-beta5/node3/disk4:/data/disk4
-      - ./data/rustfs-beta5/logs/node3:/logs
-      - ./kms/rustfs-compat:/kms
-    ports:
-      - "${RUSTFS_BETA5_NODE3_PORT:-9103}:9000"
-
-  rustfs-beta5-node4:
-    <<: *rustfs-node
-    hostname: rustfs-beta5-node4
-    volumes:
-      - ./data/rustfs-beta5/node4/disk1:/data/disk1
-      - ./data/rustfs-beta5/node4/disk2:/data/disk2
-      - ./data/rustfs-beta5/node4/disk3:/data/disk3
-      - ./data/rustfs-beta5/node4/disk4:/data/disk4
-      - ./data/rustfs-beta5/logs/node4:/logs
-      - ./kms/rustfs-compat:/kms
-    ports:
-      - "${RUSTFS_BETA5_NODE4_PORT:-9104}:9000"
-
-networks:
-  rustfs-beta5-net:
-    driver: bridge

.docker/test/compat/docker-compose.rustfs-rio-v2.yml

@@ -1,115 +0,0 @@
-# Copyright 2024 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0.
-
-x-rustfs-rio-v2-env: &rustfs-rio-v2-env
-  RUSTFS_VOLUMES: "http://rustfs-rio-v2-node{1...4}:9000/data/disk{1...4}"
-  RUSTFS_ADDRESS: ":9000"
-  RUSTFS_CONSOLE_ADDRESS: ":9001"
-  RUSTFS_CONSOLE_ENABLE: "true"
-  RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS: "*"
-  RUSTFS_ACCESS_KEY: "${RUSTFS_ACCESS_KEY:-rustfsadmin}"
-  RUSTFS_SECRET_KEY: "${RUSTFS_SECRET_KEY:-rustfsadmin}"
-  RUSTFS_OBS_LOGGER_LEVEL: "${RUSTFS_OBS_LOGGER_LEVEL:-info}"
-  RUSTFS_OBS_LOG_DIRECTORY: "/logs"
-  RUSTFS_COMPRESSION_ENABLED: "${RUSTFS_COMPRESSION_ENABLED:-true}"
-  RUSTFS_COMPRESSION_EXTENSIONS: "${RUSTFS_COMPRESSION_EXTENSIONS:-.txt,.log,.csv,.json,.tar,.xml,.bin}"
-  RUSTFS_COMPRESSION_MIME_TYPES: "${RUSTFS_COMPRESSION_MIME_TYPES:-text/*,application/json,application/xml,binary/octet-stream}"
-  RUSTFS_KMS_ENABLE: "${RUSTFS_KMS_ENABLE:-false}"
-  RUSTFS_KMS_BACKEND: "${RUSTFS_KMS_BACKEND:-local}"
-  RUSTFS_KMS_KEY_DIR: "${RUSTFS_KMS_KEY_DIR:-/kms}"
-  RUSTFS_KMS_DEFAULT_KEY_ID: "${RUSTFS_KMS_DEFAULT_KEY_ID:-compat-key}"
-  RUSTFS_KMS_ALLOW_INSECURE_DEV_DEFAULTS: "${RUSTFS_KMS_ALLOW_INSECURE_DEV_DEFAULTS:-false}"
-  RUSTFS_UNSAFE_BYPASS_DISK_CHECK: "${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}"
-
-x-rustfs-rio-v2-node: &rustfs-rio-v2-node
-  image: "${RUSTFS_RIO_V2_IMAGE:-rustfs/rustfs:compat-rio-v2}"
-  entrypoint:
-    - /bin/sh
-    - -c
-    - |
-      until getent hosts rustfs-rio-v2-node1 >/dev/null &&
-        getent hosts rustfs-rio-v2-node2 >/dev/null &&
-        getent hosts rustfs-rio-v2-node3 >/dev/null &&
-        getent hosts rustfs-rio-v2-node4 >/dev/null; do
-        sleep 1
-      done
-      exec /entrypoint.sh /usr/bin/rustfs
-  build:
-    context: ../..
-    dockerfile: Dockerfile.source
-    args:
-      RUSTFS_BUILD_FEATURES: "rio-v2"
-  environment: *rustfs-rio-v2-env
-  depends_on:
-    rustfs-rio-v2-permission-helper:
-      condition: service_completed_successfully
-  networks:
-    - rustfs-rio-v2-net
-  restart: unless-stopped
-
-services:
-  rustfs-rio-v2-permission-helper:
-    image: alpine:3.23
-    command: sh -c "mkdir -p /compat-data /kms && chown -R 10001:10001 /compat-data /kms"
-    volumes:
-      - ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}:/compat-data
-      - ./kms/rustfs-compat:/kms
-    restart: "no"
-
-  rustfs-rio-v2-node1:
-    <<: *rustfs-rio-v2-node
-    hostname: rustfs-rio-v2-node1
-    volumes:
-      - ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/node1/disk1:/data/disk1
-      - ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/node1/disk2:/data/disk2
-      - ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/node1/disk3:/data/disk3
-      - ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/node1/disk4:/data/disk4
-      - ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/logs/node1:/logs
-      - ./kms/rustfs-compat:/kms
-    ports:
-      - "${RUSTFS_RIO_V2_API_PORT:-9300}:9000"
-      - "${RUSTFS_RIO_V2_CONSOLE_PORT:-9301}:9001"
-
-  rustfs-rio-v2-node2:
-    <<: *rustfs-rio-v2-node
-    hostname: rustfs-rio-v2-node2
-    volumes:
-      - ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/node2/disk1:/data/disk1
-      - ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/node2/disk2:/data/disk2
-      - ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/node2/disk3:/data/disk3
-      - ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/node2/disk4:/data/disk4
-      - ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/logs/node2:/logs
-      - ./kms/rustfs-compat:/kms
-    ports:
-      - "${RUSTFS_RIO_V2_NODE2_PORT:-9302}:9000"
-
-  rustfs-rio-v2-node3:
-    <<: *rustfs-rio-v2-node
-    hostname: rustfs-rio-v2-node3
-    volumes:
-      - ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/node3/disk1:/data/disk1
-      - ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/node3/disk2:/data/disk2
-      - ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/node3/disk3:/data/disk3
-      - ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/node3/disk4:/data/disk4
-      - ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/logs/node3:/logs
-      - ./kms/rustfs-compat:/kms
-    ports:
-      - "${RUSTFS_RIO_V2_NODE3_PORT:-9303}:9000"
-
-  rustfs-rio-v2-node4:
-    <<: *rustfs-rio-v2-node
-    hostname: rustfs-rio-v2-node4
-    volumes:
-      - ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/node4/disk1:/data/disk1
-      - ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/node4/disk2:/data/disk2
-      - ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/node4/disk3:/data/disk3
-      - ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/node4/disk4:/data/disk4
-      - ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/logs/node4:/logs
-      - ./kms/rustfs-compat:/kms
-    ports:
-      - "${RUSTFS_RIO_V2_NODE4_PORT:-9304}:9000"
-
-networks:
-  rustfs-rio-v2-net:
-    driver: bridge